sometimes exceeding human performance. However, this is not generally the
case, as [178] show that reward models trained online together with a policy
may not be effective when a completely new policy is trained.

####  5.2.7 Combining Different Feedback Types

An additional extension that is of interest is how the learning process can
incorporate different sources of feedback. So far, three options have been
considered in the literature: First, the combination of demonstrations and
preferences in a model-free approach [119] or a passive-active preference-
based reward learning approach [202, 29]. Second, the combination of pairwise
preferences and ordinal labels for actions in order to ensure safety-aware
learning [60]. And third, the combination of pairwise preferences,
demonstrations, and ratings for reducing the cognitive load for the human
labeler [146].

###  5.3 Evaluating Learned Reward Functions

A central question when it comes to learning the reward function is how to
evaluate the learned reward function and how reward functions can be compared
with each other. For this purpose, a couple of different approaches are
available.

Rollout Method

In inverse RL, a common method for evaluation is the rollout method [194, 80,
119, 40]. In this approach, one first learns an optimal policy for the learned
reward function and then estimates the value of this policy for online
trajectory rollouts using the known ground-truth reward function. This
approach can be transferred to RLHF as well. In many cases, however, the true
reward function is unknown, and in safety-critical areas such as medicine or
autonomous driving, such online rollouts cannot be executed.

Off-policy Evaluations

As a possible alternative, so-called off-policy evaluations, which estimate
the value of the optimal policy on the basis of an available data set, have
been established. For coping with biases or large variances due to policy
mismatch, approaches using importance sampling [208], regression- or
classification-based methods [201, 153, 120], or combinations of these [128]
have been proposed. The problem with these approaches, however, is that the
traces of the explicit sources of error through policy learning or reward
learning are blurred.

Distance Functions

Yet another alternative, which has been advanced in the seminal paper by [98],
is using a suitable distance function for reward functions. Suitable here
means that two reward functions, which differ only by certain transformations
such as potential shaping [195] or positive scaling, should have zero distance
if these transformations do not change the policy ranking with regard to the
expected return. For this purpose, [98] present a pseudometric, called
Equivalent-Policy Invariant Comparison (EPIC) distance, that is determined in
three steps: First, mapping two reward functions to a so-called
_canonicalization form_ that is invariant to transformations of the latter
kind. Second, normalizing these canonicalization forms by means of a specific
weighted $L_{2}$-norm whose weights are determined by a distribution over the
transitions. Finally, the EPIC distance is the weighted $L_{2}$-norm distance
of the normalized canonicalization forms.

Even if some attractive properties, above all a Lipschitz continuity in terms
of the EPIC distance of two reward functions for the difference of the value
functions of the induced optimal policies is shown, this distance has its
shortcomings. One of these is that the canonicalization form used by EPIC
distance does not encode sufficient knowledge about the underlying transition
dynamics, which might lead to unreliable distances when evaluating reward
functions on physically non-realizable transitions. To this end, [276] propose
the Dynamics-Aware Reward Distance (DARD), which uses a slightly different
form of canonicalization but restricts the evaluation of the reward functions
to transitions that are approximately physically feasible.

Recently, EPIC-like distances [124] and STAndardised Reward Comparison (STARC)
metrics [240], which are entire classes of pseudometrics on the space of all
reward functions were proposed that generalize the three-step approach
underlying the EPIC distance (and DARD) by parameterizing each of the steps.
Specifically, the canonicalization function in the first step, the
normalization in the second, and the metric in the third step are kept
variable. If these three functional parameters fulfill certain requirements,
then the resulting distance has some appealing properties, e.g., being a
pseudometric that is zero if and only if the two reward functions induce the
same ordering of policies or imply upper and lower bounds on value function
differences.

Visual and Human Inspection

For an evaluation by visual inspection, [123] propose a method for
preprocessing reward functions by transforming them into simpler but
equivalent reward functions for better interpretability. Related to this and
the rollout method, the quality of the reward function learned can also be
evaluated by a human (or expert) by examining the behavior of the agent on the
target task.

###  5.4 Reward Model Inputs

Besides the feedback type, another factor is the modality of the reward model
input data. This usually consists of the agent’s observations and actions.
Observations can range from true state to high dimensional inputs (e.g.,
images), while actions can range from discrete finite actions to continuous
actions.

For instance, many typical RL benchmarks are in the continuous control domain
(e.g., MuJoCo simulated robotics tasks) with true state representations and
simple discrete actions. In such problems, [58] train reward models from these
inputs.

When no compact state representation is available, raw images are often used
in control tasks, which makes the learning of rewards more challenging since
the setting becomes partially observable and the reward function is generally
not Markov with respect to the observations. In such cases, the conventional
trick of approximating a true state with a sequence of frames is often
employed. This approach is used, for instance, by [58] to train reward models
on the Atari benchmark suite.

More recently, many applications of RLHF are in the natural language
processing (NLP) domain. In these settings, the policy takes natural language
as both input and output while the reward model takes it as input (e.g.,
[200]). Naturally, more complex scenarios (e.g., with both language and vision
inputs [2]) have also been studied.

###  5.5 Increasing Feedback Efficiency

Maximizing feedback efficiency is vital in RLHF due to the high cost of human
feedback. This section delves into methods that enhance learning from limited
human feedback. We discuss methods that leverage prior offline data, methods
that use (partially unlabelled) data more efficiently, and methods that aim to
gather more informative data.

####  5.5.1 Using Prior Data

There are often large amounts of prior data available at little or no
additional cost. While this data generally was generated for other tasks, many
basic human preferences are the same for various tasks and can often even be
extracted from completely unsupervised data such as text corpora. By
leveraging this prior data, we can greatly reduce the amount of feedback
necessary to learn the current task’s objective. We explore various methods,
including foundation models, meta- and transfer learning, pertaining, and
reward model initialization.

##### Meta- and Transfer Learning

Meta- and transfer learning techniques in reward model training exploit the
commonalities across different objectives, facilitating quick adaptation to
new tasks. [219] develop a broadly applicable meta-reward model, pre-trained
on a diverse set of tasks to capture a wide range of preference patterns,
enabling efficient adaptation to new tasks with fewer examples. [278] use a
similar meta-learning approach to build a goal classifier across multiple
visuomotor tasks. Closely related to these meta-learning approaches, [113]
integrate few-shot learning principles, optimizing their approach for
scenarios where only a few examples are available for adapting to new tasks.
In the domain of transfer learning, [169] explore zero-shot transfer of
preferences, a method that enables adapting preferences without additional
data from the new task. In a different vein, but closely related to meta- and
transfer learning, [179] tackle the lifelong inverse RL problem, focusing on
inferring reward functions for multiple tasks over time, which involves
knowledge transfer between tasks. Collectively, these studies underscore the
potential of meta- and transfer learning in enhancing the efficiency and
applicability of reward models in RLHF.

##### Leveraging Foundation Models

Foundation models, i.e., models trained on large amounts of often unlabeled
data, can acquire significant knowledge about basic human preferences. A
language model trained to predict the next token in a text corpus, for
example, may learn to complete the sentence ‘Frank was mad that his vacuum
robot broke the vase,’ thereby learning that humans prefer non-destructive
behavior. These learned preferences can then be leveraged in RLHF approaches.
[70] exemplify such an approach by training a success detector using a pre-
trained vision-language model (Flamingo). Their approach utilizes a data set
of trajectories with binary success labels, employing a non-interactive
training method.

##### Reward Model Initialization

It is often beneficial to initialize the reward model with parameters from a
model trained on a related task. This strategy is particularly common in
language model fine-tuning, where self-supervised pretraining is a common
practice. In such scenarios, it becomes logical to use these pre-trained
models for initializing not just the policy but also the reward model. This
methodology is adopted by [9] and [200]. Specifically, [200] use a pretrained
language model for the reward model, opting for a smaller model relative to
the policy to mitigate unstable learning. Notably, while they apply supervised
fine-tuning to the policy before the RLHF phase, the reward model is
initialized directly from the language model without any preliminary fine-
tuning. This approach’s applicability extends beyond language models to other
areas. A notable example is [2], who, in the control domain, begin by training
a policy through contrastive self-supervised learning and behavioral cloning.
They then add an MLP head to the policy for the prediction of cumulative
rewards.

##### Preference Model Pretraining

Preference-model pretraining [9, 12] leverages prior offline data to pretrain
the preference model before training it on policy samples. [9] note that in
the case of language models, noisy preference data can be readily obtained
from sources such as rated Reddit comments, preferred Stack Overflow answers,
and reverted Wikipedia edits. They leverage this as a pretraining step to
increase data efficiency. This is in addition to regular language model
pretraining, as discussed in the previous paragraph. A similar approach could
be applied to control in case prior data and some means of inferring
preferences, such as human corrections, are available. Even if no inferred
preferences are available, [259] show that it can be beneficial to pre-train
the preference model to predict close to constant reward on an initial set of
trajectories. This avoids excessive fitting of the policy to random
initialization differences in the reward function.

##### Supervised Representation Learning

A compact representation that captures all relevant information for human
preferences while minimizing noise can greatly enhance preference learning
efficiency. It may also generalize better than a representation learned end-
to-end as part of the preference learning task, which may contain spurious
correlations. [36] address this by proposing the learning of features through
explicit human feedback using feature traces. Feature traces (see Section
3.2.9) involve human labelers explicitly teaching relevant features one by one
by demonstrating behavior in which the feature monotonically increases or
decreases. This method directly aligns the learned representation with human-
identified features, enhancing preference learning efficiency but requiring
detailed human input. However, feature traces require labelers to be able to
identify and articulate relevant features, which can be challenging. [34]
offer an alternative approach with their Similarity-based Implicit
Representation Learning (SIRL) method. SIRL learns representations from
similarity queries (see Section 3.2.10), where human labelers provide feedback
on whether behaviors are similar or different concerning the features that
matter to them. This method captures a broader range of human notions of
similarity without needing explicit feature knowledge, thus reducing the
cognitive load on human labelers. In summary, while both approaches emphasize
human feedback’s centrality in representation learning, they differ in their
methods of gathering this feedback. The feature traces used by [36] require
specific feature knowledge, whereas SIRL used by [34] utilizes more intuitive
similarity assessments, potentially offering a more user-friendly way to
capture human preferences.

These diverse methods of utilizing prior data demonstrate the potential for
enhancing data efficiency in RLHF, enabling more effective learning from
limited human feedback.

####  5.5.2 Using Data More Efficiently

Beyond the application of prior data, several techniques can enhance the
efficiency of data utilization in training processes. This section will
discuss a range of such methods, including self-supervised and semi-supervised
training, as well as the integration of inductive biases and data augmentation
strategies. These approaches are designed to make the most of the available
human interactions and improve the final performance of RLHF models.

##### Self-Supervised Auxiliary Tasks

Self-supervised training enhances data efficiency in reward model training by
using unannotated data to capture information about the task. This technique
extends beyond the scope of pretraining methods, as discussed in the prior
section, to incorporating concurrent auxiliary tasks to maximize the utility
of available data. A prevalent technique, as applied by [2], [39], and [180],
involves adding self-supervised losses to enhance representation learning for
rewards. [2] implement a contrastive task where the reward network
differentiates between observations that are consistent between multiple
modalities and those that are not, blending this with preference learning loss
and behavioral cloning. [39] add multiple auxiliary tasks such as inverse and
forward dynamics modeling, temporal distance prediction, and variational
autoencoder training. Similarly, [180] use the self-predictive representations
technique [229] to learn state representations that encode environmental
dynamics, enabling a linear model to anticipate successor states, thereby
forming an efficient basis for preference learning and significantly boosting
sample efficiency. However, auxiliary losses for better representation
learning are not the only approach to leverage self-supervised training. An
alternate approach by [260] involves identifying important states using
attention weights from a world model transformer and state importance
estimates based on a preference predicting transformer. These estimates can
aid credit assignment for observed preferences, further optimizing the
training process.

##### Semi-Supervised Training

Semi-supervised training, blending labeled and unlabeled data, can leverage
the unlabeled data to glean information about the task and the environment.
This is most commonly done by generating pseudo-labels for the unlabeled data,
either by leveraging model predictions or by making assumptions. The first
approach is utilized by [44] and [285], which use generative models and GAN-
based methods to mimic human preferences. Similarly, [203] expand their data
set with high-confidence unlabeled samples based on the preference predictor’s
evaluations. The second strategy, making assumptions to augment data, is
showcased by [293]. They generate preference data by assuming that (i) human-
written examples are better than model-written examples, (ii) human-written
and model-written examples are indistinguishable amongst themselves, and (iii)
generations of later model iterations are better than those of earlier ones.

##### Data Augmentation

Data augmentation focuses on creating additional examples from existing
labeled data. Temporal augmentation is particularly effective in RLHF,
involving trajectory data. This is exemplified by [40, 203] who base their
augmentation on the premise that preferences for complete trajectories can be
extrapolated to cropped segments, allowing the generation of multiple
derivative pairs from a single labeled trajectory pair. [203] additionally
explore state modifications, such as random re-scaling and Gaussian noise
addition, finding temporal cropping to be the most effective, with noise
sometimes negatively impacting performance. In a similar vein, [258] focus on
augmenting trajectories by concentrating on changing elements in observations
and perturbing the other parts, based on the premise that movement indicates
importance in image-based observations. Complementing these methods, [2]
employ augmentation by randomly altering instructions and language responses,
thus creating artificial examples of non-preferred behavior. These diverse
data augmentation methods collectively enhance the training data set,
contributing to the increased robustness and efficacy of RLHF models.

####  5.5.3 Gathering Better Data

In addition to leveraging unlabeled data and using labels more efficiently,
sample efficiency can be further increased by collecting more informative
samples in the first place. This can either be achieved by selecting more
informative samples from the experience buffer or by generating more
informative experiences. While selecting informative samples from the
experience buffer is addressed under active learning (see Section 4.1.1), this
section focuses on generating more informative experiences.

While we are not aware of many works in this area, one possible approach
involves steering the agent’s exploration towards regions of the state space
where human feedback would be most beneficial. [162] implement this by
employing intrinsic motivation, driven by the estimated uncertainty of the
reward model, to guide the agent’s exploration. This highlights the potential
of not just using data more efficiently but also generating data in a more
targeted manner.

##  6 Theory

The field of RLHF has recently made some progress in terms of theoretical
results, which we will discuss in this section. First, we consider the
contributions where the goal is to learn a provable (near) optimal policy both
in an online and offline fashion or even in a way that falls in between. Then,
we discuss and highlight recent contributions related to different theoretical
aspects of RLHF, such as its relation to the standard reward-based RL. Tables
5 and 6 provide a concise overview of the results for the online or offline
policy learning setting. Here, $\mathcal{N}_{\mathcal{F}}(\epsilon,d)$ denotes
the $\epsilon$-covering number of a set $\mathcal{F}$ under some metric
$d$333The $\epsilon$-covering number is the minimum integer $N$ such that
there exists a subset $\mathcal{F}^{\prime}\subset\mathcal{F}$ with
$|\mathcal{F}^{\prime}|=N$, and for any $f\in\mathcal{F}$, there exists some
$f^{\prime}\in\mathcal{F}^{\prime}$ satisfying $d(f,f^{\prime})\leq\epsilon$..
It is worth mentioning that all works have two standard assumptions, namely
that the reward function is bounded and that the ground-truth reward, human
feedback model, or transition dynamic is an element of the considered model
space, respectively.

###  6.1 Policy Learning

In the literature focusing on theoretical results, there is a distinction
(similar to the distinction made in standard RL) between an offline and online
setting. In the former, learning is based on a given fixed data set, usually
previously collected through an interaction with the environment. In contrast,
in the online environment, one interacts directly with the environment to
learn from real-time feedback and continuously updates one’s strategies based
on the feedback received, allowing the agent to learn and adapt as it engages
with the environment.

Table 5: An overview of approaches, their assumptions, goals, and properties
for online policy learning. $\widetilde{\mathcal{O}}$ is used to hide
$\log$-factors. $T$ is the number of iterations of the respective algorithm.
Algorithm
(Reference) |  Algorithmic
approach |  Assumptions |  Target(s) and goal(s) of learner |  Theoretical guarantee(s)
---|---|---|---|---
Dueling Posterior Sampling (DPS)
[196] |  Leveraging Posterior Sampling from dueling bandits |  Linear link function, tabular MDP |  Bayes regret minimization w.r.t. optimal policy based on trajectory comparison feedback |  Asymptotic regret rate:
$\mathcal{O}\left(|\mathcal{S}|\sqrt{|\mathcal{A}|T\log(|\mathcal{A}|)}\right)$
Logistic Preference Reinforcement Learning (LPbRL)
[225] |  Leveraging MaxInP from contextual dueling bandits |  Logistic link function, tabular MDP, linear rewards & $d$-dimensional feature embedding of trajectories |  Expected regret minimization w.r.t. optimal policy based on trajectory comparison feedback |  Transition dynamics:
1\. Known
$\widetilde{\mathcal{O}}\left(|\mathcal{S}|Hd\sqrt{T\log(T)}\right)$
2\. Unknown
$\widetilde{\mathcal{O}}((\sqrt{d}+H^{2}+|\mathcal{S}|)\sqrt{dT}\newline
\hphantom{\widetilde{\mathcal{O}}(}+\sqrt{|\mathcal{S}||\mathcal{A}|TH})$
Preference-based Optimistic Planning (PbOP)
[52] |  Leveraging MaxInP; general function approximation |  General human feedback model class $\mathcal{F}_{\mathbb{T}}$ and general transition dynamic class $\mathcal{F}_{\mathbb{P}}$ with finite $l_{2}$-norm $\rho$-Eluder dimension $d_{\mathbb{T}}^{(2)}(\rho)$ and $d_{\mathbb{P}}^{(2)}(\rho)$, respectively |  High probability regret minimization w.r.t. optimal policy based on trajectory comparison feedback |  $\widetilde{\mathcal{O}}\left(\sqrt{d_{\mathbb{P}}(\tfrac{1}{T})HT\log\left(\mathcal{N}_{\mathcal{F}_{\mathbb{P}}}\left(\tfrac{1}{T},d\right)\right)}\right)\newline +\widetilde{\mathcal{O}}\left(\sqrt{d_{\mathbb{T}}(\tfrac{1}{T})T\log\left(\mathcal{N}_{\mathcal{F}_{\mathbb{T}}}\left(\tfrac{1}{T},d\right)\right)}\right)$
$d$ being the $\ell$-infinity norm $\|\cdot\|_{\infty}$
Preference-based Policy Search (PPS)
[280] |  Dynamic programming, policy search, $(\epsilon,\delta)$-PAC black-box dueling bandit algorithm and simulator |  Uniform dependence of policy preference probabilities on value function differences, tabular MDP, $(\epsilon,\delta)$-PAC dueling bandit algorithm with $\Psi(K,\varepsilon,\delta)\varepsilon^{-\alpha}$ sample complexity for $K$ arms |  $(\epsilon,\delta)$-PAC for optimal policy based on trajectory comparison feedback |  Simulator step bound
$\mathcal{O}\left(\frac{H^{\alpha+1}|\mathcal{S}|\Psi(|\mathcal{A}|,\varepsilon/H,\delta/|\mathcal{S}|)}{\varepsilon^{\alpha}}\right)$
Sample complexity bound
$\mathcal{O}\left(\frac{H^{\alpha}|\mathcal{S}|\Psi(|\mathcal{A}|,\varepsilon/H,\delta/|\mathcal{S}|)}{\varepsilon^{\alpha}}\right)$
Preference-based Exploration & Policy Search (PEPS)
[280] |  Similar to PPS, instead of simulator using an auxiliary synthetic reward function |  Same as PPS and stochastic triangle inequality of trajectory comparisons preferences |  $(\epsilon,\delta)$-PAC for optimal policy based on trajectory comparison feedback |  Step complexity bound
$\widetilde{\mathcal{O}}\left(\frac{H^{\alpha+1}|\mathcal{S}|^{2}\Psi(|\mathcal{A}|,\varepsilon/H,\delta/|\mathcal{S}|)}{\varepsilon^{\alpha+1}}\right)$
Comparison complexity bound
$\mathcal{O}\left(\frac{H^{\alpha}|\mathcal{S}|\Psi(|\mathcal{A}|,\varepsilon/H,\delta/|\mathcal{S}|)}{\varepsilon^{\alpha}}\right)$
UCBVI-Planning
[145] |  Optimistic least-squares value iteration, maximum information gain, value iteration based on pessimistic expected value function estimation |  Binary rewards for state-action pairs based on human response model $f\in\mathcal{F}_{H}$ with bounded noise $\Delta>0$, compliant and tabular/linear MDP with dimension $d$ |  $(\epsilon,\delta)$-PAC for optimal policy based on binary state-action reward feedback |  Tabular MDP:
${\mathcal{O}}\Bigg{(}\frac{H^{4}|\mathcal{S}||\mathcal{A}|\log\left(\frac{H|\mathcal{S}||\mathcal{A}|}{\epsilon\delta}\right)}{\epsilon^{2}}$
$\hphantom{\mathcal{O}\Big{(}}+\frac{H^{3}|\mathcal{S}|^{2}|\mathcal{A}|\log\left(\frac{H|\mathcal{S}||\mathcal{A}|}{\epsilon\delta}\right)}{\epsilon}\Bigg{)}$
Linear MDP:
${\mathcal{O}}\left(\frac{|\mathcal{A}|^{2}d^{5}d_{\mathcal{F}_{H}}H^{4}\log\left(\frac{H|\mathcal{S}||\mathcal{A}|}{\epsilon\delta\Delta}\right)}{\epsilon^{2}}\right)$

Preference-based & Randomized Least-Squares Value Iteration (PR-LSVI)
[274] |  Least-squares value iteration with perturbed state-action-wise reward model |  General differentiable link function $\Phi$, linear MDP, linear rewards with $d$-dimensional feature embedding of trajectories |  Expected regret minimization w.r.t. optimal policy and/or low trajectory comparison feedback complexity steered by $\epsilon\in[0,1]$ |  Expected regret bound:
$\widetilde{\mathcal{O}}\Big{(}\epsilon Td^{1/2}+\sqrt{T}\cdot
d^{3}H^{5/2}\gamma$
$\hphantom{\mathcal{O}\Big{(}}+d^{17/2}H^{11/2}\gamma^{3}\Big{)}$
Comparison complexity bound:
$\widetilde{\mathcal{O}}\Big{(}d^{4}{(\kappa+R_{\mathrm{max}})}^{2}/\epsilon^{2}\Big{)}$
$\kappa=\inf_{x\in[-R_{\mathrm{max}},R_{\mathrm{max}}]}\Phi^{\prime}(x)$

##### Online Learning

The first work dealing with the question of theoretical guarantees for
learning an optimal policy from trajectory comparison feedback (see Section
3.2) in an online manner is by [196]. It laid the foundation for a paradigm
subsequently embraced by many subsequent research endeavors: Adapting learning
algorithms from the dueling or preference-based bandit literature [284, 247,
21] to the underlying situation with additional states. The preference-based
bandit problem can be viewed as a preference-based RL problem with one state,
so state transition dynamics must be considered accordingly for a fruitful
adaptation. It is worth mentioning that [122] used a quite similar idea before
for feedback in the form of corrections (see Section 3.2) by resorting to the
coactive learning setting [233]. Assuming the existence of a ground-truth
context-trajectory scoring function and that the user’s feedback is
informative, the Preference Perceptron algorithm by [233] is used and analyzed
in terms of its cumulative regret.

[196] suggest the Dueling Posterior Sampling (DPS), which is an adaptation of
the self-sparring algorithm [246]. It takes a Bayesian perspective on the
problem and defines a Dirichlet prior on the dynamics and a Gaussian prior on
the rewards that are subsequently updated, while the trajectories to be
compared by the human labeler are chosen based on their (posterior)
probability of being optimal444The latter probability is assessed by posterior
sampling; a commonly used technique in the bandit literature used by so-called
Thompson Sampling strategies, see [152] for more details.. Assuming a linear
link function (see Section 5.1.4) as well as a tabular MDP, it is shown that
DPS is (i) consistent, i.e., converges in distribution to the optimal policy,
and (ii) achieves an asymptotic expected regret bound (see Table 5).

[280] combine dynamic programming and policy search with a black-box
preference-based bandit algorithm for each state to design routines that
return an almost optimal (a.k.a. $\varepsilon$-optimal) policy with high
probability555This is a so-called PAC learning setting [254] in which the goal
of finding the/an optimal object is relaxed to finding a “good enough” object,
usually specified by some distance measure on the object domain.  The first
routine, called Preference-based Policy Search (PPS), requires access to a
simulator, while the second routine, called Preference-based Exploration and
Policy Search (PEPS), gets rid of this requirement by exploring the state
space by means of an auxiliary synthetic reward function. By assuming that the
probability of one policy dominating another policy is bounded uniformly over
all states from below by a multiplicative of their value function, they show
generic upper bounds for both routines on the number of pairwise trajectory
comparisons (see Table 5). If these dominance probabilities have even more
structural properties, such as fulfilling stochastic transitivity or
stochastic triangle inequality (see [106, 21]), then these upper bounds can be
further refined.

A follow-up work by [225] assumes a feature embedding of trajectories that
gives rise to a feature embedding of policies and adapts the MaxInP algorithm
[224] for contextual dueling bandits by essentially viewing the policy
embeddings as the contexts. More precisely, assuming a logistic link function
(see Section 5.1.4), confidence sets for the expected scores of the policies
are constructed based on the maximum likelihood estimate (MLE), and the two
policies with the highest uncertainty in terms of maximal variance are used to
sample a trajectory, respectively, to be compared. In this way, the logistic
preference-based reinforcement learning (LPbRL) is derived and also extended
to the case of unknown dynamics by taking the uncertainty regarding the
dynamics into account when constructing the confidence sets. For both cases,
i.e., known or unknown dynamics, upper bounds on the regret of LPbRL are shown
(see Table 5).

In contrast to previous work that all considers tabular MDPs, [52] consider
the case of a general unknown human feedback model and unknown dynamics each
from function classes with a finite Eluder dimension666Roughly speaking, the
Eluder dimension of a function class refers to the number of worst-case errors
one must make to identify an unknown function from that class. [220]. They
propose and analyze the Preference-based Optimistic Planning (PbOP) algorithm,
which essentially follows a similar design as LPbRL but uses least-square
estimates for the human feedback model and transitions dynamics along with
confidence sets based on them. Moreover, they derive lower bounds for the
regret of any learning algorithm by reducing the once-per-episode-feedback RL
problem [49] to the PbRL problem. Finally, they extend their analysis to the
case of $K$-wise comparisons, where one obtains all $\binom{K}{2}$ pairwise
comparisons for $K$ many queried trajectories. In essence, the regret term
coming from the human feedback model class improves by a factor of $\sqrt{K}$.

[274] consider a similar learning scenario as [225] but with the additional
objective to keep the number of queries of trajectory comparisons low, which
is a combination of two competing objectives also studied in the bandit
literature [67]. For this purpose, they suggest the Preference-based and
Randomized Least-Squares Value Iteration (PR-LSVI) algorithm, which combines
least-squares value iteration with a perturbed state-action-based reward model
with Gaussian noise for regret minimization; a similar idea to CoLSTIM
suggested for contextual dueling bandits [23]. More specifically, in each time
step, the policy maximizing the value function of the perturbed state-action-
based reward model and the policy maximizing the later in the previous time
steps are “played”. By sampling trajectories of these two policies and
computing their expected absolute reward difference (based on the perturbed
state-action-based reward model) as a measure of uncertainty, preference
feedback for these two trajectories is queried if the uncertainty exceeds a
certain threshold. Moreover, they also suggest a posterior sampling
counterpart of this algorithm, the Preference-based Thompson Sampling (PbTS)
algorithm, and analyze it in terms of Bayesian quantities.

Table 6: Overview of approaches, their assumptions, goals, and properties for
offline policy learning with a data set of size $n$. $\widetilde{\mathcal{O}}$
is used to hide $\log$-factors. $R_{\mathrm{max}}$ is a bound on the reward.
Algorithm
(Reference) |  Algorithmic
approach |  Assumptions |  Target(s) and goal(s) of learner |  Theoretical guarantee(s)
---|---|---|---|---
Pessimistic MLE
[294] |  Greedy policy for pessimistic expected value function estimation |  Logistic link function, linear reward function for a state-pair feature embedding with some regularity assumptions on weights, known transition dynamics |  High probability bound for the performance gap based on trajectory-based (and action-based) feedback |
$\mathcal{O}\left(e^{2HR_{\mathrm{max}}}\sqrt{\frac{d+\log(1/\delta)}{n}}\right)$
oFfline ReinforcemEnt lEarning with HumAN feeDback (FREEHAND)
[286] |  Greedy policy for pessimistic expected value function estimation |  General differentiable link function $\Phi$, general bounded reward function class $\mathcal{F}_{r}$ and general transition dynamic class $\mathcal{F}_{\mathbb{P}}$ |  High probability bound for the performance gap based on trajectory-based (and action-based) feedback |  Transition dynamics:
1\. Known
$\mathcal{O}\left(\sqrt{\frac{C_{r}^{2}\kappa^{2}\log(\mathcal{N}_{\mathcal{F}_{r}}(1/N,|\cdot|)/\delta)}{n}}\right)$
2\. Unknown
$\mathcal{O}\left(\sqrt{\frac{C_{r}^{2}\kappa^{2}\log(\mathcal{N}_{\mathcal{F}_{r}}(1/N,|\cdot|)/\delta)}{n}}\right)+\mathcal{O}\left(R_{\mathrm{max}}\sqrt{\frac{C_{P}^{2}\kappa^{2}\log(\mathcal{N}_{\mathcal{F}_{\mathbb{P}}}(1/N,|\cdot|)/\delta)}{n}}\right)$
$\kappa=\inf_{x\in[-R_{\mathrm{max}},R_{\mathrm{max}}]}\Phi^{\prime}(x)$
$C_{r}$, $C_{P}$ reward and transition concentrability coefficient
Dynamic-Choice-Pessimistic-Policy-Optimization (DCPPO)
[161] |  Value iteration based on pessimistic expected value function estimation |  Dynamic discrete choice model, linear MDP, linear reward function for a state-pair feature embedding with some regularity assumptions on weights, known model class entailing the value and reward function of the dynamic discrete choice model |  High probability bound for the performance gap based on action-based feedback |  Linear model class:
$\mathcal{O}\left(|\mathcal{A}|d^{3/2}H^{2}e^{H}\sqrt{\frac{\log(dHn/\delta)}{n}}\right)$
RKHS model class with different eigenvalue decay:
$\mathcal{O}\left(\tilde{d}He^{H}|\mathcal{A}|\sqrt{\mu\log(nR_{\mathrm{max}}H/\delta)}\right)$
$\mu\text{-finite spectrum, }$
$\mathcal{O}\left(\tilde{d}He^{H}|\mathcal{A}|\sqrt{{(\log(nR_{\mathrm{max}}H)/\delta)}^{1+1/\mu}}\right)$
$\mu\text{-exponential decay, }$
$\mathcal{O}\left(\tilde{d}He^{H}|\mathcal{A}|{(nR_{\mathrm{max}})}^{\kappa^{*}}\sqrt{\log(nR_{\mathrm{max}}H/\delta)}\right)$
$\mu\text{-polynomial decay,}$
$\kappa^{*}=\frac{d+1}{2(\mu+d)}+\frac{1}{\mu(1-2\tau)-1}$,
$\tilde{d}=$population effective $\cdot$ sampling effective dimension
LCBVI-Tabular-Offline
[145] |  Maximum information gain for reward querying, value iteration based on pessimistic expected value function estimation for policy learning |  Binary rewards for state-action pairs based on human response model with bounded noise, compliant and tabular MDP |  High probability bound for the performance gap based on binary state-action reward feedback |  Linear model class:
$\mathcal{O}\Big{(}H\sqrt{|\mathcal{S}|\log(|\mathcal{S}||\mathcal{A}|Hn/\delta)}$
$\hphantom{\Big{(}}\cdot\mathbb{E}_{\pi^{*}}\left[\sum^{H}_{h=1}{(N_{h}(s_{h},a_{h})+1)}^{-1/2}\right]\Big{)}$
$N_{h}$ are numbers of visit time

##### Offline Learning

[294] study the performance of a greedy policy trained from a data set
consisting of trajectory pairs along with the observed preference that is
assumed to be generated by means of a Bradley-Terry model with linear rewards.
For this purpose, different results with respect to the MLE of the Bradley-
Terry model for different feedback scenarios are derived that are quite of
independent interest. In particular, they show concentration inequalities of
the MLE for trajectory-based comparison feedback and additionally its
asymptotic normality for action-based comparison feedback that also holds for
$K$-wise comparisons. Based on these, it is shown that the greedy policy using
the MLE in the case of action-based feedback might fail while using a
pessimistic MLE leads to minimax-rates with respect to the performance
gap777The expected difference between the optimal value function and the value
function of the used policy.. The latter is also shown to be true in the case
of trajectory-based feedback. Technically, the pessimistic MLE is realized by
taking the policy that has the largest pessimistic expected value function,
i.e., the lowest realization of the value function within a hyperparameter-
dependent confidence region around the MLE. Further results of independent
interest are the inferred theoretical guarantees for maximum entropy inverse
RL [295] and action-based inverse RL algorithms [193].

The simple model assumptions underlying [294] were then replaced by more
sophisticated assumptions in some subsequent work. The linear reward
assumption has been replaced by more general reward function classes by [286]
and [161]. In addition, [286] also consider more general unknown human
feedback models and construct the confidence regions for the pessimistic
approach directly from the log-likelihood function. The resulting approach,
called FREEHAND, is analyzed in terms of its performance gap, for which some
problem-dependent coefficients, the per-step, per-trajectory, and transition
concentrability coefficient, are introduced. On the basis of a lower bound, it
is shown that the per-trajectory concentrability coefficient should naturally
appear in the bound on the performance gap. Moreover, the concentrability
coefficient is shown to be upper bounded by the constant appearing in the
special case of linear rewards considered by [294]. Finally, it is worth
mentioning that both trajectory-based and action-based comparison feedback are
considered.

Assuming a dynamic discrete choice model [221] underlying the given data set
of observed trajectories (without explicitly observed preferences), [161]
suggest the Dynamic-Choice-Pessimistic-Policy-Optimization (DCPPO) algorithm.
It first estimates the reward model using this assumption and then learns a
policy in a (pessimistic) value iteration manner from the estimated reward
model. In the case of a linear MDP and a known model class that entails both
the value and the reward function of the dynamic discrete choice model, DCPPO
is analyzed with respect to its performance gap. This is done for the case of
a linear function model class as well as a subset of a reproducing kernel
Hilbert space (RKHS) as the model class.

Focusing on the estimation of the weight parameter in the Bradley-Terry model
for the action-based feedback under label differential privacy conditions
[72], [55] analyze two estimation procedures, MLE and stochastic gradient
descent (SGD), under similar assumptions as in [294]. In both cases, the cost
of ensuring label differential privacy is a multiplicative factor.

Reward collapse, a term introduced by [243], describes the issue when rank-
based training methods for LLMs lead to the same reward distribution
regardless of the prompts used in the final training steps. The authors show
that this occurs because the rank-based approach does not adequately account
for prompt-related information. To address this problem, the authors propose a
family of utility functions as well as an optimization method that
successfully creates prompt-dependent reward distributions, effectively
mitigating the collapse of rewards during training.

##### Blending Online and Offline Learning

[145] study the problem of optimal policy learning from critique feedback (see
Section 3.2), i.e., binary rewards for state-action pairs, with as few queries
to the human as possible. They assume an underlying ground-truth human
feedback model that leads to a positive evaluation for a state-action pair if
it exceeds a specific threshold evaluated at that pair. In addition, the
learning process consists of two phases: First, exploring the environment in
an unsupervised manner, and then querying user feedback in an active reward
learning phase to learn the human feedback model. This learning process is
again analyzed in two variants: Either the exploration phase was performed
externally, and a data set consisting of trajectories is provided (offline),
or this data set is actively collected itself (online). For both variants, an
active learning algorithm is proposed that essentially selects query points
(state-action pairs) that provide the most information gain given the points
already designated to be queried. For the online variant, an exploration
strategy based on optimistic least-squares value iteration [129] is also
introduced for tabular or linear MDPs. In both variants, policy learning is
carried out by a pessimistic value iteration with the empirical transitions
and the estimated reward function, resulting in UCBVI-Planning (online) and
LCBVI-Tabular-Offline (offline). Under the assumption of bounded noise [177]
or low-noise assumption [147, 105], bounds on the performance gap of both
algorithms are derived.

The question of the ideal experimental design for RLHF is addressed by [287],
in particular, how to separate the process of data acquisition (e.g.,
trajectories to be evaluated) from the process of retrieving human feedback to
avoid constantly involving humans in the training loop. Assuming linear
rewards, the Bradley-Terry model and either a transition oracle (e.g.,
available for tabular or low-rank MDPs) or a linear MDP they suggest the
expeRimental dEsiGn for queryIng huMan prEference (REGIME) algorithm that
first samples exploratory trajectories indented to be as informative as
possible for learning the reward via MLE and then applies a greedy policy
based on the reward learned by the latter. They explicitly show that REGIME
requires less human feedback to be queried in order to output an
$\epsilon$-optimal policy at the end than the approach by [225].

###  6.2 Preference-Based vs. Reward-Based Learning

There have been some theoretical analyses regarding the question in how far,
or if at all, preference-based feedback in the form of trajectory comparisons
is more s more suitable compared to numerical feedback. [127] suggest a human
rating model for this purpose in the numerical feedback case and analyze the
LCB algorithm [130] in order to compare it with the pessimistic MLE [294]. It
is shown that under specific assumptions, LCB has a constant performance gap,
while the preference-based pessimistic MLE under similar assumptions has a
similar bound as in Table 6.

[262] provide reduction-based algorithms that can directly utilize state-of-
the-art results in reward-based RL for RLHF with utility-based and general
state-action and trajectory-based comparison feedback. They show, in general,
how theoretical results of the underlying standard RL algorithm can be
translated to theoretical results for the resulting preference-based RL
algorithm. For some special cases, such as MDPs with finite Eluder dimension
and utility-based preference feedback, the theoretical guarantees are
explicitly derived using state-of-the-art RL algorithms that are qualitatively
similar to explicit preference-based RL algorithms.

##  7 Applications and Benchmarks

The field of RLHF has advanced significantly in the last few years, with
increasing interest driven by prominent applications. First and foremost are
applications to large language models, exemplified by ChatGPT [197]. This
section starts by providing a sample of such applications, showcasing how this
technology is being utilized in fields as varied as robotics, language
processing, image generation, and more. We will also delve into libraries that
provide foundational support for RLHF research, enabling researchers and
practitioners to experiment with and refine a range of approaches. We then
explore a spectrum of benchmarks that have been developed to standardize and
simplify the evaluation of new approaches, offering insights into their
performance in different settings. Finally, and closely related to those
benchmarks, we will discuss common evaluation practices.

###  7.1 Applications

RLHF finds applications across various domains, showcasing its versatility in
addressing complex and nuanced tasks. The most prominent application is
ChatGPT [197], which is an example of an application in the domain of language
models. Beyond that, however, applications extend across diverse domains such
as control tasks, generative models, and recommender systems. This section
provides an overview of notable works applying RLHF in different areas.

##### Control and Interactive Environments

There is a long history of using control environments as benchmark tasks for
RL. In addition to the breadth of available environments, control applications
are of particular interest because tasks are often hard to specify. [58]
demonstrated the effectiveness RLHF in games as well as simulated continuous
control tasks, matching the performance of RL agents trained on ground-truth
rewards with a fraction of the feedback. Extending to robotics, [68] trained a
reward model for diverse tasks with a single robot, achieving human-like
behavior. [149] applied RLHF for precise robot-to-human handovers. Similarly,
[2] used RLHF in the Playhouse simulator, a platform for sensorimotor task
training, and [182] showcase an application in the context of the MineRL
Basalt competition for Minecraft tasks.

##### Generative Models in Language and Imaging

Generative models, i.e., models that generate new data instead of just
predicting labels, can be framed as an RL setting in which a policy assembles
the output through its actions. In the context of language models, this means
that the language model is interpreted as a policy with tokens as actions.
Using this reframing, we can use RLHF approaches to fine-tune generative
models to produce preferred outputs. ChatGPT [197] and GPT-4 [198] are prime
examples of language models fine-tuned using RLHF. These applications build on
earlier work, such as by [200], [296] and [96]. This method extends to text
summarization [87, 88, 244], dialogue summarization [50], and question
answering [191]. In image generation, [155] and [279] demonstrate the use of
reward modeling for text-to-image tasks, while [204] and [137] explore RLHF
applications in broader computer vision tasks.

##### Recommender Systems

In the context of recommender systems, [282] have shown the potential of RLHF
in optimizing for long-term engagement. Although it is, in principle, possible
to algorithmically evaluate policies in this domain, these rewards are sparse.
To combat this, [282] use RLHF to distill sparse, global feedback into a dense
reward model.

These diverse applications underscore RLHF’s adaptability and its growing
importance in various technological domains, paving the way for innovative
solutions and enhanced human-computer interactions.

###  7.2 Supporting Libraries

Several libraries have emerged that aim to provide a toolset for implementing
and experimenting with RLHF and reward learning algorithms, contributing to
the ease and efficiency of research and development. One notable example is
the imitation library [100]. It encompasses a collection of imitation and
reward learning algorithms, including those introduced in the seminal work by
[58]. Two other libraries, APReL [32] and POLAR [253], focus on the Bayesian
setting. [32] provide a specialized framework for preference-based reward
learning with a focus on Bayesian methods. Meanwhile, [253] introduce a
framework designed for Bayesian reward learning from multiple modalities,
including pairwise preferences, in MATLAB. Finally, the domain of language
model fine-tuning, the trlX library [46] offers a toolkit specifically
designed for language model training. It specializes in the fine-tuning of
transformer-based language models, treating the language model as the policy
in an RLHF setup.

Due to the many interacting components and the human element in RLHF research,
implementing new ideas and running experiments can be quite challenging. The
discussed libraries reduce this challenge and make RLHF research more
approachable to many researchers.

###  7.3 Benchmarks

Due to the difficulty of reproducible evaluations without a ground-truth
objective and with humans in the loop, benchmarks play an important role in
the advancement and evaluation of RLHF approaches. Several benchmarks have
been proposed, each focusing on different applications and challenges.

One such benchmark is B-Pref [156], which focuses on control tasks with
synthetic feedback. B-Pref aims to provide simulated human feedback that
captures some irrationalities, thereby coming closer to evaluation with real
human feedback than other approaches. At the same time, by relying entirely on
synthetic feedback, the results are reproducible and cost-effective to
generate. In a similar vein, [79] propose a set of environments designed to
diagnose common problems in reward learning. These environments help in
identifying and addressing the typical challenges that arise in RLHF
scenarios.

A more application-driven benchmark with a complex environment is given by the
MineRL BASALT competition [231, 182]. The competition proposes the challenge
of solving tasks defined by natural language descriptions in Minecraft based
on human feedback. Writing hand-engineered reward functions is very
challenging in that setting, which makes it a good benchmark for methods based
on human feedback. The competition is method-agnostic in principle, also
considering non-RL approaches such as behavioral cloning. It is also agnostic
for the feedback type, which may include demonstrations, comparisons, and
others. The initial data set consists of demonstrations, however. The final
evaluation is performed by humans through pairwise comparisons.

In the domain of language modeling, Truthful QA [165] serves as a benchmark
that measures the truthfulness of models. Also, in the context of language
models, [215] introduce a set of pre-trained reward models, learned from human
feedback, as benchmarks. These models serve as reference points for evaluating
new RLHF techniques against established standards.

Together, these benchmarks provide a diverse and comprehensive suite of tests
that drive the development and refinement of RLHF methods, ensuring they are
robust, effective, and capable of handling a wide range of real-world
scenarios.

###  7.4 Evaluation

Evaluating RLHF poses unique challenges, particularly in scenarios without
clear ground-truth task specifications. Evaluations generally focus on either
the learned policy or the reward model, each shedding light on different
aspects of system performance.

##### Policy Evaluation

Assessing learned behavior is crucial for the evaluation of an RLHF system. In
domains with ground-truth rewards, these can be used for policy evaluation
[58]. However, many RLHF applications lack this clarity. [200], for instance,
evaluate the quality of language model responses by having labelers rate the
output quality on a test set of prompts, highlighting the significance of
human judgment in assessing model outputs. [122] use direct Likert-scale
scores for evaluations, including self-assessments by trainers and cross-
evaluations by others. [171] extend this with a Likert-scale survey and free-
form participant comments, comparing evaluations based on known true rewards
with subjective experiences. Moreover, [2] employ a multi-stage evaluation
scheme that includes scripted probe tasks, a standardized test suite evaluated
by humans, and full interactive assessments, demonstrating the need for
diverse and thorough evaluation methodologies in RLHF.

##### Reward Model Evaluation

Direct reward model evaluation complements policy assessment. While reward
model accuracy is a more direct measure of preference-learning success, the
ultimate goal is inducing effective policies. A perfectly accurate reward
model is often not necessary to induce a good policy, which is the actual goal
of RLHF. Therefore, both evaluation methods are ideally used in combination.
[122] also use a ranking loss method for test sets of trajectories, compared
against expert evaluations with known Likert-scores. This approach provides
quantitative measures of the reward model’s fidelity. In addition, [265]
compare parameter-based and reward-based evaluation measures for learned
reward functions, identifying strengths and weaknesses in both methods and
contributing to a more nuanced understanding of reward model assessment in
RLHF. These approaches provide a quantitative measure of the reward model’s
accuracy in reflecting human preferences and expert judgments. For a detailed
discussion of reward model evaluation, also refer to Section 5.3.

Policy- and reward model evaluation both offer insights into the performance
of an RLHF approach. Ideally both measures should be combined to enable quick
iteration and give insights into both the preference learning performance as
well as the quality of the learned behavior.

##  8 Discussion and Conclusion

In this survey, we have provided an overview of the current state of RLHF,
highlighting its evolution from PbRL and examining its broad applications
across various domains like control, natural language processing, and computer
vision. While our survey captures the current state and many significant
trends and advancements in RLHF, we acknowledge the rapid expansion of this
field and the inevitable limitations in covering every extension and
application in depth. We will discuss some of these extensions, open
questions, and conclusions in this section.

We have specifically focused on RLHF methods where a reward function is
learned online from human feedback. There have been some recent works that are
outside of this scope and yet propose promising new methods to learn human-
aligned objectives. One of them is learning reward functions offline, as
recently investigated by [232]. An alternative approach is to learn objectives
from a pre-trained AI system instead of human feedback. This has been termed
RL from AI feedback (RLAIF) and leverages foundation models as a source of
preference [13]. Alternatively, instead of a reward function, it is possible
to train a policy directly [6, 41] or to learn a $Q$-function from preference
feedback [114, 53].

Most work on RLHF implicitly assumes that tasks can be specified by
maximization of expected accumulated scalar rewards. This assumption, called
the _reward hypothesis_ [236], is under active debate [150, 255, 37, 238] in
the RL community. Recent approaches in RLHF are, for instance, considering
more complex objective functions, such as multi-objective frameworks involving
non-linear aggregation of expected accumulated vector rewards [210].

Many more extensions of RLHF are inspired by revisiting classic RL topics
under the RLHF lens. This is exemplified by studies on exploration [162],
reward feature learning [135], reward design [175], reward shaping [277],
multi-task RL [200, 2, 189], hindsight experience replay [291],
interpretability [26], safe RL [63, 60], and fair RL [235]. As discussed in
Section 4.2, the intersection of RLHF with HCI also offers a fertile ground
for future research, especially for refining feedback mechanisms. It is
crucial to keep human psychology in mind when designing these systems and to
learn from other related fields that already studied such issues extensively.

In addition to those extensions, current RLHF methods also have challenges and
limitations to be aware of. [45] offer a thorough analysis of these issues and
limitations, highlighting the practical constraints of current approaches.
Adding to that, from a theoretical perspective, a primary challenge lies in
further relaxing underlying assumptions. This requires striking a delicate
balance: On the one hand, ensuring the assumptions are not overly restrictive
to encompass a broad range of practical use cases, and on the other,
maintaining the feasibility of theoretical guarantees for computationally
efficient algorithms. Key questions in this context are whether it is possible
to design algorithms that do not need to actively maintain a policy space and
eliminate sub-optimal policies nor rely on a computation oracle. Recent work
such as [262] or [274] give hope that this may be possible.

Although RLHF has significantly contributed to the advancements in LLMs and
other areas of machine learning, it is a domain still in its infancy with many
unanswered questions and inherent limitations. Despite and because of these
challenges, it is ripe for further advancements in theory and practice,
hopefully resulting in even more robust algorithms making more efficient use
of human feedback. It remains intriguing to what extent RLHF will continue to
shape the fields of natural language processing, RL, robotics, AI alignment,
and beyond in the future.

##### Acknowledgements

This publication was supported by LMUexcellent, funded by the Federal Ministry
of Education and Research (BMBF) and the Free State of Bavaria under the
Excellence Strategy of the Federal Government and the Länder as well as by the
Hightech Agenda Bavaria. This work has also been supported in part by the
program of National Natural Science Foundation of China (No. 62176154) and a
collaborative project funded by NetEase.

## References

  * [1] Youssef Abdelkareem, Shady Shehata and Fakhri Karray  “Advances in Preference-based Reinforcement Learning: A Review”  In _Proceedings of IEEE International Conference on Systems, Man, and Cybernetics (SMC)_ , 2022, pp. 2527–2532  DOI: 10.1109/SMC53654.2022.9945333
  * [2] Josh Abramson, Arun Ahuja, Federico Carnevale, Petko Georgiev, Alex Goldin, Alden Hung, Jessica Landon, Jirka Lhotka, Timothy Lillicrap, Alistair Muldal, George Powell, Adam Santoro, Guy Scully, Sanjana Srivastava, Tamara Glehn, Greg Wayne, Nathaniel Wong, Chen Yan and Rui Zhu  “Improving Multimodal Interactive Agents with Reinforcement Learning from Human Feedback”, 2022  arXiv:2211.11602
  * [3] Riad Akrour, Marc Schoenauer and Michele Sebag  “Preference-Based Policy Learning”  In _Proceedings of Machine Learning and Knowledge Discovery in Databases (ECML PKDD)_ Springer, 2011, pp. 12–27  DOI: 10.1007/978-3-642-23780-5_11
  * [4] Mayer Alvo and Philip L.H. Yu  “Statistical Methods for Ranking Data”  Springer, 2014  DOI: 10.1007/978-1-4939-1471-5
  * [5] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman and Dan Mané  “Concrete Problems in AI Safety”, 2016  arXiv:1606.06565
  * [6] Gaon An, Junhyeok Lee, Xingdong Zuo, Norio Kosaka, Kyung-Min Kim and Hyun Oh Song  “Direct Preference-based Policy Optimization without Reward Modeling”, 2023  URL: https://openreview.net/forum?id=FkAwlqBuyO
  * [7] Saurabh Arora and Prashant Doshi  “A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress”  In _Artificial Intelligence_ 297, 2021, pp. 103500  DOI: 10.1016/j.artint.2021.103500
  * [8] Christian Arzate Cruz and Takeo Igarashi  “A Survey on Interactive Reinforcement Learning: Design Principles and Open Challenges”  In _Proceedings of the ACM Designing Interactive Systems Conference (DIS)_ Association for Computing Machinery, 2020, pp. 1195–1209  DOI: 10.1145/3357236.3395525
  * [9] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah and Jared Kaplan  “A General Language Assistant as a Laboratory for Alignment”, 2021  arXiv:2112.00861
  * [10] Müslüm Atas, Alexander Felfernig, Seda Polat-Erdeniz, Andrei Popescu, Thi Ngoc Trang Tran and Mathias Uta  “Towards Psychology-Aware Preference Construction in Recommender Systems: Overview and Research Issues”  In _Journal of Intelligent Information Systems_ 57.3, 2021, pp. 467–489  DOI: 10.1007/s10844-021-00674-5
  * [11] Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko and Rémi Munos  “A General Theoretical Paradigm to Understand Learning from Human Preferences”, 2023  arXiv:2310.12036
  * [12] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann and Jared Kaplan  “Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback”, 2022  arXiv:2204.05862
  * [13] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown and Jared Kaplan  “Constitutional AI: Harmlessness from AI Feedback”, 2022  arXiv:2212.08073
  * [14] Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew and Igor Mordatch  “Emergent Tool Use From Multi-Agent Autocurricula”  In _Proceedings of International Conference on Learning Representations (ICLR)_ , 2020  URL: https://openreview.net/forum?id=SkxpxJBKwS
  * [15] Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick and Christopher Summerfield  “Fine-Tuning Language Models to Find Agreement among Humans with Diverse Preferences”  In _Advances in Neural Information Processing Systems (NeurIPS)_ 35, 2022, pp. 38176–38189  URL: https://proceedings.neurips.cc/paper/2022/hash/f978c8f3b5f399cae464e85f72e28503-Abstract-Conference.html
  * [16] Peter Barnett, Rachel Freedman, Justin Svegliato and Stuart Russell  “Active Reward Learning from Multiple Teachers”, 2023
  * [17] Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila and Francisco Herrera  “Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI”  In _Information Fusion_ 58, 2020, pp. 82–115  DOI: 10.1016/j.inffus.2019.12.012
  * [18] Chandrayee Basu, Erdem Bıyık, Zhixun He, Mukesh Singhal and Dorsa Sadigh  “Active Learning of Reward Dynamics from Hierarchical Queries”  In _Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2019, pp. 120–127  DOI: 10.1109/IROS40897.2019.8968522
  * [19] Chandrayee Basu, Mukesh Singhal and Anca D. Dragan  “Learning from Richer Human Guidance: Augmenting Comparison-Based Learning with Feature Queries”  In _Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI)_ Association for Computing Machinery, 2018, pp. 132–140  DOI: 10.1145/3171221.3171284
  * [20] Chandrayee Basu, Qian Yang, David Hungerman, Mukesh Singhal and Anca D. Dragan  “Do You Want Your Autonomous Car To Drive Like You?”  In _Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI)_ Association for Computing Machinery, 2017, pp. 417–425  DOI: 10.1145/2909824.3020250
  * [21] Viktor Bengs, Róbert Busa-Fekete, Adil El Mesaoudi-Paul and Eyke Hüllermeier  “Preference-Based Online Learning with Dueling Bandits: A Survey”  In _Journal of Machine Learning Research_ 22.7, 2021, pp. 1–108  URL: http://jmlr.org/papers/v22/18-546.html
  * [22] Viktor Bengs and Eyke Hüllermeier  “Preselection Bandits”  In _Proceedings of the International Conference on Machine Learning (ICML)_ PMLR, 2020, pp. 778–787  URL: https://proceedings.mlr.press/v119/bengs20a.html
  * [23] Viktor Bengs, Aadirupa Saha and Eyke Hüllermeier  “Stochastic Contextual Dueling Bandits under Linear Stochastic Transitivity Models”  In _Proceedings of the International Conference on Machine Learning (ICML)_ PMLR, 2022, pp. 1764–1786  URL: https://proceedings.mlr.press/v162/bengs22a.html
  * [24] Tom Bewley, Jonathan Lawry and Arthur Richards  “Learning Interpretable Models of Aircraft Handling Behaviour by Reinforcement Learning from Human Feedback”, 2023  arXiv:2305.16924
  * [25] Tom Bewley, Jonathan Lawry, Arthur Richards, Rachel Craddock and Ian Henderson  “Reward Learning with Trees: Methods and Evaluation”, 2022  arXiv:2210.01007
  * [26] Tom Bewley and Freddy Lécué  “Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions”  In _Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS)_ IFAAMAS, 2022, pp. 118–126  URL: https://www.ifaamas.org/Proceedings/aamas2022/pdfs/p118.pdf
  * [27] Adam Bignold, Francisco Cruz, Matthew E. Taylor, Tim Brys, Richard Dazeley, Peter Vamplew and Cameron Foale  “A Conceptual Framework for Externally-Influenced Agents: An Assisted Reinforcement Learning Review”  In _Journal of Ambient Intelligence and Humanized Computing_ , 2021, pp. 3621–3644  DOI: 10.1007/s12652-021-03489-y
  * [28] Erdem Bıyık, Nicolas Huynh, Mykel Kochenderfer and Dorsa Sadigh  “Active Preference-Based Gaussian Process Regression for Reward Learning”  In _Proceedings of Robotics: Science and Systems (RSS)_ 16, 2020  URL: http://www.roboticsproceedings.org/rss16/p041.html
  * [29] Erdem Bıyık, Dylan P. Losey, Malayandi Palan, Nicholas C. Landolfi, Gleb Shevchuk and Dorsa Sadigh  “Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences”  In _The International Journal of Robotics Research_ 41.1 SAGE Publications Ltd STM, 2022, pp. 45–67  DOI: 10.1177/02783649211041652
  * [30] Erdem Bıyık, Malayandi Palan, Nicholas C. Landolfi, Dylan P. Losey and Dorsa Sadigh  “Asking Easy Questions: A User-Friendly Approach to Active Reward Learning”  In _Proceedings of the Conference on Robot Learnin (CoRL)_ PMLR, 2020, pp. 1177–1190  URL: https://proceedings.mlr.press/v100/b-iy-ik20a.html
  * [31] Erdem Bıyık and Dorsa Sadigh  “Batch Active Preference-Based Learning of Reward Functions”  In _Proceedings of the Conference on Robot Learning (CoRL)_ PMLR, 2018, pp. 519–528  URL: https://proceedings.mlr.press/v87/biyik18a.html
  * [32] Erdem Bıyık, Aditi Talati and Dorsa Sadigh  “APReL: A Library for Active Preference-based Reward Learning Algorithms”  In _Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI)_ , 2022, pp. 613–617  DOI: 10.1109/HRI53351.2022.9889650
  * [33] Andreea Bobu, Andrea Bajcsy, Jaime F. Fisac, Sampada Deglurkar and Anca D. Dragan  “Quantifying Hypothesis Space Misspecification in Learning From Human–Robot Demonstrations and Physical Corrections”  In _IEEE Transactions on Robotics_ 36.3, 2020, pp. 835–854  DOI: 10.1109/TRO.2020.2971415
  * [34] Andreea Bobu, Yi Liu, Rohin Shah, Daniel S. Brown and Anca D. Dragan  “SIRL: Similarity-based Implicit Representation Learning”  In _Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI)_ Association for Computing Machinery, 2023, pp. 565–574  DOI: 10.1145/3568162.3576989
  * [35] Andreea Bobu, Dexter R.. Scobee, Jaime F. Fisac, S. Sastry and Anca D. Dragan  “LESS Is More: Rethinking Probabilistic Models of Human Behavior”  In _Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI)_ Association for Computing Machinery, 2020, pp. 429–437  DOI: 10.1145/3319502.3374811
  * [36] Andreea Bobu, Marius Wiggert, Claire Tomlin and Anca D Dragan  “Inducing Structure in Reward Learning by Learning Features”  In _The International Journal of Robotics Research_ 41.5 SAGE Publications Ltd STM, 2022, pp. 497–518  DOI: 10.1177/02783649221078031
  * [37] Michael Bowling, John D. Martin, David Abel and Will Dabney  “Settling the Reward Hypothesis”, 2022  arXiv:2212.10420
  * [38] Ralph Allan Bradley and Milton E. Terry  “Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons”  In _Biometrika_ 39.3/4 [Oxford University Press, Biometrika Trust], 1952, pp. 324–345  DOI: 10.2307/2334029
  * [39] Daniel S. Brown, Russell Coleman, Ravi Srinivasan and Scott Niekum  “Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences”  In _Proceedings of the International Conference on Machine Learning (ICML)_ PMLR, 2020, pp. 1165–1177  URL: https://proceedings.mlr.press/v119/brown20a.html
  * [40] Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan and Scott Niekum  “Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations”  In _Proceedings of the International Conference on Machine Learning (ICML)_ PMLR, 2019, pp. 783–792  URL: https://proceedings.mlr.press/v97/brown19a.html
  * [41] Róbert Busa-Fekete, Balázs Szörényi, Paul Weng, Weiwei Cheng and Eyke Hüllermeier  “Preference-Based Reinforcement Learning: Evolutionary Direct Policy Search Using a Preference-Based Racing Algorithm”  In _Machine Learning_ 97.3, 2014, pp. 327–351  DOI: 10.1007/s10994-014-5458-8
  * [42] Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, Oleg Sushkov, David Barker, Jonathan Scholz, Misha Denil, Nando Freitas and Ziyu Wang  “Scaling Data-Driven Robotics with Reward Sketching and Batch Reinforcement Learning”  In _Proceedings of Robotics: Science and Systems (RSS)_ 16, 2020  DOI: 10.15607/RSS.2020.XVI.076
  * [43] Haoyang Cao, Samuel Cohen and Lukasz Szpruch  “Identifiability in Inverse Reinforcement Learning”  In _Advances in Neural Information Processing Systems (NeurIPS)_ 34 Curran Associates, Inc., 2021, pp. 12362–12373  URL: https://proceedings.neurips.cc/paper/2021/hash/671f0311e2754fcdd37f70a8550379bc-Abstract.html
  * [44] Zehong Cao, KaiChiu Wong and Chin-Teng Lin  “Weak Human Preference Supervision for Deep Reinforcement Learning”  In _IEEE Transactions on Neural Networks and Learning Systems_ 32.12, 2021, pp. 5369–5378  DOI: 10.1109/TNNLS.2021.3084198
  * [45] Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bıyık, Anca Dragan, David Krueger, Dorsa Sadigh and Dylan Hadfield-Menell  “Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback”, 2023  arXiv:2307.15217
  * [46] Louis Castricato, Alex Havrilla, Shahbuland Matiana, Duy V. Phung, Aman Tiwari, Jonathan Tow and Maksym Zhuravinsky  “trlX: A Scalable Framework for RLHF”, 2023  Zenodo  DOI: 10.5281/zenodo.8076391
  * [47] Manuela Cattelan  “Models for Paired Comparison Data: A Review with Emphasis on Dependent Data”  In _Statistical Science_ 27.3 Institute of Mathematical Statistics, 2012, pp. 412–433  DOI: 10.1214/12-STS396
  * [48] Lawrence Chan, Andrew Critch and Anca Dragan  “Human Irrationality: Both Bad and Good for Reward Inference”, 2021  arXiv:2111.06956
  * [49] Niladri Chatterji, Aldo Pacchiano, Peter Bartlett and Michael Jordan  “On the Theory of Reinforcement Learning with Once-per-Episode Feedback”  In _Advances in Neural Information Processing Systems (NeurIPS)_ 34 Curran Associates, Inc., 2021, pp. 3401–3412  URL: https://proceedings.neurips.cc/paper/2021/hash/1bf2efbbe0c49b9f567c2e40f645279a-Abstract.html
  * [50] Jiaao Chen, Mohan Dodda and Diyi Yang  “Human-in-the-Loop Abstractive Dialogue Summarization”  In _Findings of the Association for Computational Linguistics (ACL)_ Association for Computational Linguistics, 2023, pp. 9176–9190  DOI: 10.18653/v1/2023.findings-acl.584
  * [51] Li Chen, Marco Gemmis, Alexander Felfernig, Pasquale Lops, Francesco Ricci and Giovanni Semeraro  “Human Decision Making and Recommender Systems”  In _ACM Transactions on Interactive Intelligent Systems_ 3.3, 2013, pp. 17:1–17:7  DOI: 10.1145/2533670.2533675
  * [52] Xiaoyu Chen, Han Zhong, Zhuoran Yang, Zhaoran Wang and Liwei Wang  “Human-in-the-Loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximation”  In _Proceedings of the International Conference on Machine Learning (ICML)_ PMLR, 2022, pp. 3773–3793  URL: https://proceedings.mlr.press/v162/chen22ag.html
  * [53] Weiwei Cheng, Johannes Fürnkranz, Eyke Hüllermeier and Sang-Hyeun Park  “Preference-Based Policy Iteration: Leveraging Preference Learning for Reinforcement Learning”  In _Proceedings of Machine Learning and Knowledge Discovery in Databases (ECML PKDD)_ Springer, 2011, pp. 312–327  DOI: 10.1007/978-3-642-23780-5_30
  * [54] Jaedeug Choi and Kee-eung Kim  “Nonparametric Bayesian Inverse Reinforcement Learning for Multiple Reward Functions”  In _Advances in Neural Information Processing Systems (NIPS)_ 25 Curran Associates, Inc., 2012, pp. 314–322  URL: https://proceedings.neurips.cc/paper/2012/hash/140f6969d5213fd0ece03148e62e461e-Abstract.html
  * [55] Sayak Ray Chowdhury and Xingyu Zhou  “Differentially Private Reward Estimation from Preference Based Feedback”, 2023  URL: https://openreview.net/forum?id=TqzYmBPSGC
  * [56] Paul Christiano  “Semi-Supervised Reinforcement Learning”, 2016  Medium  URL: https://ai-alignment.com/semi-supervised-reinforcement-learning-cf7d5375197f
  * [57] Paul Christiano  “Thoughts on the Impact of RLHF Research”, 2023  URL: https://www.alignmentforum.org/posts/vwu4kegAEZTBtpT6p/thoughts-on-the-impact-of-rlhf-research
  * [58] Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg and Dario Amodei  “Deep Reinforcement Learning from Human Preferences”  In _Advances in Neural Information Processing Systems (NIPS)_ 30 Curran Associates, Inc., 2017, pp. 4299–4307  URL: https://proceedings.neurips.cc/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html
  * [59] Jack Clark and Dario Amodei  “Faulty Reward Functions in the Wild”, 2016  OpenAI Blog  URL: https://openai.com/blog/faulty-reward-functions/
  * [60] Ryan Cosner, Maegan Tucker, Andrew Taylor, Kejun Li, Tamas Molnar, Wyatt Ubelacker, Anil Alan, Gabor Orosz, Yisong Yue and Aaron Ames  “Safety-Aware Preference-Based Learning for Safety-Critical Control”  In _Proceedings of the Annual Learning for Dynamics and Control Conference (L4DC)_ PMLR, 2022, pp. 1020–1033  URL: https://proceedings.mlr.press/v168/cosner22a.html
  * [61] Yuchen Cui and Scott Niekum  “Active Reward Learning from Critiques”  In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , 2018, pp. 6907–6914  DOI: 10.1109/ICRA.2018.8460854
  * [62] Yuchen Cui, Qiping Zhang, Brad Knox, Alessandro Allievi, Peter Stone and Scott Niekum  “The EMPATHIC Framework for Task Learning from Implicit Human Feedback”  In _Proceedings of the Conference on Robot Learning (CoRL)_ PMLR, 2021, pp. 604–626  URL: https://proceedings.mlr.press/v155/cui21a.html
  * [63] Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang and Yaodong Yang  “Safe RLHF: Safe Reinforcement Learning from Human Feedback”, 2023  arXiv:2310.12773
  * [64] Christian Daniel, Malte Viering, Jan Metz, Oliver Kroemer and Jan Peters  “Active Reward Learning”  In _Proceedings of Robotics: Science and Systems (RSS)_ 10, 2014  URL: http://www.roboticsproceedings.org/rss10/p31.html
  * [65] Oliver Daniels-Koch and Rachel Freedman  “The Expertise Problem: Learning from Specialized Feedback”, 2022  URL: https://openreview.net/forum?id=I7K975-H1Mg
  * [66] Brett Day, Ian J. Bateman, Richard T. Carson, Diane Dupont, Jordan J. Louviere, Sanae Morimoto, Riccardo Scarpa and Paul Wang  “Ordering Effects and Choice Set Awareness in Repeat-Response Stated Preference Studies”  In _Journal of Environmental Economics and Management_ 63.1, 2012, pp. 73–91  DOI: 10.1016/j.jeem.2011.09.001
  * [67] Rémy Degenne, Thomas Nedelec, Clement Calauzenes and Vianney Perchet  “Bridging the Gap between Regret Minimization and Best Arm Identification, with Application to A/B Tests”  In _Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS)_ PMLR, 2019, pp. 1988–1996  URL: https://proceedings.mlr.press/v89/degenne19a.html
  * [68] Zihan Ding, Yuanpei Chen, Allen Z. Ren, Shixiang Shane Gu, Hao Dong and Chi Jin  “Learning a Universal Human Prior for Dexterous Manipulation from Human Preference”, 2023  arXiv:2304.04602
  * [69] Zibin Dong, Yifu Yuan, Jianye Hao, Fei Ni, Yao Mu, Yan Zheng, Yujing Hu, Tangjie Lv, Changjie Fan and Zhipeng Hu  “AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model”, 2023  arXiv:2310.02054
  * [70] Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando Freitas and Serkan Cabi  “Vision-Language Models as Success Detectors”, 2023  arXiv:2303.07280
  * [71] Krishnamurthy Dvijotham and Emanuel Todorov  “Inverse Optimal Control with Linearly-Solvable MDPs”  In _Proceedings of the International Conference on Machine Learning (ICML)_ Omnipress, 2010, pp. 335–342  URL: https://icml.cc/Conferences/2010/papers/571.pdf
  * [72] Cynthia Dwork  “Differential Privacy: A Survey of Results”  In _Proceedings of Theory and Applications of Models of Computation (TAMC)_ Springer, 2008, pp. 1–19  DOI: 10.1007/978-3-540-79228-4_1
  * [73] Joseph Early, Tom Bewley, Christine Evers and Sarvapali Ramchurn  “Non-Markovian Reward Modelling from Trajectory Labels via Interpretable Multiple Instance Learning”  In _Advances in Neural Information Processing Systems (NeurIPS)_ 35, 2022, pp. 27652–27663  URL: https://proceedings.neurips.cc/paper/2022/hash/b157cfde6794e93b2353b9712bbd45a5-Abstract-Conference.html
  * [74] André Eberhard, Houssam Metni, Georg Fahland, Alexander Stroh and Pascal Friederich  “Actively Learning Costly Reward Functions for Reinforcement Learning”, 2022  URL: https://openreview.net/forum?id=eFHNEv6G9fF
  * [75] Tesca Fitzgerald, Pallavi Koppol, Patrick Callaghan, Russell Quinlan Jun Hei Wong, Reid Simmons, Oliver Kroemer and Henny Admoni  “INQUIRE: INteractive Querying for User-aware Informative REasoning”  In _Proceedings of the Conference on Robot Learning (CoRL)_ PMLR, 2023, pp. 2241–2250  URL: https://proceedings.mlr.press/v205/fitzgerald23a.html
  * [76] Floyd J. Fowler  “Survey Research Methods”  SAGE Publications, 2013
  * [77] Rachel Freedman, Rohin Shah and Anca Dragan  “Choice Set Misspecification in Reward Inference”  In _Proceedings of the Workshop on Artificial Intelligence Safety 2020 Co-Located with the 29th International Joint Conference on Artificial Intelligence and the 17th Pacific Rim International Conference on Artificial Intelligence_ 2640 CEUR, 2021  URL: https://ceur-ws.org/Vol-2640/#paper_14
  * [78] Rachel Freedman, Justin Svegliato, Kyle Wray and Stuart Russell  “Active Teacher Selection for Reinforcement Learning from Human Feedback”, 2023  arXiv:2310.15288
  * [79] Pedro Freire, Adam Gleave, Sam Toyer and Stuart Russell  “DERAIL: Diagnostic Environments for Reward And Imitation Learning”, 2020
  * [80] Justin Fu, Katie Luo and Sergey Levine  “Learning Robust Rewards with Adverserial Inverse Reinforcement Learning”  In _Proceedings of International Conference on Learning Representations (ICLR)_ , 2018  URL: https://openreview.net/forum?id=rkHywl-A-
  * [81] Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang and Sergey Levine  “Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition”  In _Advances in Neural Information Processing Systems (NIPS)_ 31 Curran Associates, Inc., 2018, pp. 8547–8556  URL: https://proceedings.neurips.cc/paper/2018/hash/c9319967c038f9b923068dabdf60cfe3-Abstract.html
  * [82] Scott Fujimoto, Herke Hoof and David Meger  “Addressing Function Approximation Error in Actor-Critic Methods”  In _Proceedings of the International Conference on Machine Learning (ICML)_ PMLR, 2018, pp. 1587–1596  URL: https://proceedings.mlr.press/v80/fujimoto18a.html
  * [83] Johannes Fürnkranz, Eyke Hüllermeier, Weiwei Cheng and Sang-Hyeun Park  “Preference-Based Reinforcement Learning: A Formal Framework and a Policy Iteration Algorithm”  In _Machine Learning_ 89.1, 2012, pp. 123–156  DOI: 10.1007/s10994-012-5313-8
  * [84] R. Furr  “Psychometrics: An Introduction”  SAGE Publications, 2021
  * [85] Iason Gabriel  “Artificial Intelligence, Values, and Alignment”  In _Minds and Machines_ 30.3, 2020, pp. 411–437  DOI: 10.1007/s11023-020-09539-2
  * [86] Leo Gao, John Schulman and Jacob Hilton  “Scaling Laws for Reward Model Overoptimization”  In _Proceedings of the International Conference on Machine Learning (ICML)_ PMLR, 2023, pp. 10835–10866  URL: https://proceedings.mlr.press/v202/gao23h.html
  * [87] Yang Gao, Christian M. Meyer and Iryna Gurevych  “APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning”  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ Association for Computational Linguistics, 2018, pp. 4120–4130  DOI: 10.18653/v1/D18-1445
  * [88] Yang Gao, Christian M. Meyer and Iryna Gurevych  “Preference-Based Interactive Multi-Document Summarisation”  In _Information Retrieval Journal_ 23.6, 2020, pp. 555–585  DOI: 10.1007/s10791-019-09367-8
  * [89] Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum and Marc G. Bellemare  “DeepMDP: Learning Continuous Latent Space Models for Representation Learning”  In _Proceedings of the International Conference on Machine Learning (ICML)_ PMLR, 2019, pp. 2170–2179  URL: https://proceedings.mlr.press/v97/gelada19a.html
  * [90] Hans-Otto Georgii  “Gibbs Measures and Phase Transitions”  Walter de Gruyter, 2011
  * [91] Gaurav R. Ghosal, Matthew Zurek, Daniel S. Brown and Anca D. Dragan  “The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 37.5, 2023, pp. 5983–5992  DOI: 10.1609/aaai.v37i5.25740
  * [92] Hugo Gilbert, Olivier Spanjaard, Paolo Viappiani and Paul Weng  “Reducing the Number of Queries in Interactive Value Iteration”  In _Proceedings of Algorithmic Decision Theory (ADT)_ Springer International Publishing, 2015, pp. 139–152  DOI: 10.1007/978-3-319-23114-3_9
  * [93] Hugo Gilbert and Paul Weng  “Quantile Reinforcement Learning”, 2016
  * [94] Hugo Gilbert, Paul Weng and Yan Xu  “Optimizing Quantiles in Preference-Based Markov Decision Processes”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 31.1, 2017, pp. 3569–3575  DOI: 10.1609/aaai.v31i1.11026
  * [95] Hugo Gilbert, Bruno Zanuttini, Paolo Viappiani, Paul Weng and Esther Nicart  “Model-Free Reinforcement Learning with Skew-Symmetric Bilinear Utilities”  In _Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI)_ , 2016, pp. 252–261  URL: http://auai.org/uai2016/proceedings/papers/91.pdf
  * [96] Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Soňa Mokrá, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, Iason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks and Geoffrey Irving  “Improving Alignment of Dialogue Agents via Targeted Human Judgements”, 2022  arXiv:2209.14375
  * [97] Claire Glanois, Paul Weng, Matthieu Zimmer, Dong Li, Tianpei Yang, Jianye Hao and Wulong Liu  “A Survey on Interpretable Reinforcement Learning”, 2022  arXiv:2112.13112
  * [98] Adam Gleave, Michael D. Dennis, Shane Legg, Stuart Russell and Jan Leike  “Quantifying Differences in Reward Functions”  In _Proceedings of International Conference on Learning Representations (ICLR)_ , 2022  URL: https://iclr.cc/virtual/2021/poster/3348
  * [99] Adam Gleave and Geoffrey Irving  “Uncertainty Estimation for Language Reward Models”, 2022  arXiv:2203.07472
  * [100] Adam Gleave, Mohammad Taufeeque, Juan Rocamonde, Erik Jenner, Steven H. Wang, Sam Toyer, Maximilian Ernestus, Nora Belrose, Scott Emmons and Stuart Russell  “Imitation: Clean Imitation Learning Implementations”, 2022  arXiv:2211.11972
  * [101] Lin Guan, Mudit Verma, Sihang Guo, Ruohan Zhang and Subbarao Kambhampati  “Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021, pp. 21885–21897  URL: https://proceedings.neurips.cc/paper/2021/hash/b6f8dc086b2d60c5856e4ff517060392-Abstract.html
  * [102] Faruk Gul, Paulo Natenzon and Wolfgang Pesendorfer  “Random Choice as Behavioral Optimization”  In _Econometrica_ 82.5, 2014, pp. 1873–1912  DOI: 10.3982/ECTA10621
  * [103] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel and Sergey Levine  “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor”  In _Proceedings of the International Conference on Machine Learning (ICML)_ PMLR, 2018, pp. 1861–1870  URL: https://proceedings.mlr.press/v80/haarnoja18b.html
  * [104] Soheil Habibian, Ananth Jonnavittula and Dylan P. Losey  “Here’s What I’ve Learned: Asking Questions That Reveal Reward Learning”  In _ACM Transactions on Human-Robot Interaction_ 11.4, 2022, pp. 40:1–40:28  DOI: 10.1145/3526107
  * [105] Björn Haddenhorst, Viktor Bengs, Jasmin Brandt and Eyke Hüllermeier  “Testification of Condorcet Winners in Dueling Bandits”  In _Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI)_ PMLR, 2021, pp. 1195–1205  URL: https://proceedings.mlr.press/v161/haddenhorst21a.html
  * [106] Björn Haddenhorst, Eyke Hüllermeier and Martin Kolb  “Generalized Transitivity: A Systematic Comparison of Concepts with an Application to Preferences in the Babington Smith Model”  In _International Journal of Approximate Reasoning_ 119, 2020, pp. 373–407  DOI: 10.1016/j.ijar.2020.01.007
  * [107] Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel and Stuart Russell  “The Off-Switch Game”  In _Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)_ International Joint Conferences on Artificial Intelligence Organization, 2017, pp. 220–227  DOI: 10.24963/ijcai.2017/32
  * [108] Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell and Anca Dragan  “Inverse Reward Design”  In _Advances in Neural Information Processing Systems (NIPS)_ 30 Curran Associates, Inc., 2017, pp. 6765–6774  URL: https://proceedings.neurips.cc/paper/2017/hash/32fdab6559cdfa4f167f8c31b9199643-Abstract.html
  * [109] Danijar Hafner, Timothy Lillicrap, Jimmy Ba and Mohammad Norouzi  “Dream to Control: Learning Behaviors by Latent Imagination”  In _Proceedings of International Conference on Learning Representations (ICLR)_ , 2020  URL: https://openreview.net/forum?id=S1lOTC4tDS
  * [110] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba and Timothy Lillicrap  “Mastering Diverse Domains through World Models”, 2023  arXiv:2301.04104
  * [111] Thilo Hagendorff and Sarah Fabi  “Methodological Reflections for AI Alignment Research Using Human Feedback”, 2022  arXiv:2301.06859
  * [112] Jerry Zhi-Yang He and Anca D. Dragan  “Assisted Robust Reward Design”  In _Proceedings of the Conference on Robot Learning (CoRL)_ PMLR, 2022, pp. 1234–1246  URL: https://proceedings.mlr.press/v164/he22a.html
  * [113] Donald Joseph Hejna and Dorsa Sadigh  “Few-Shot Preference Learning for Human-in-the-Loop RL”  In _Proceedings of the Conference on Robot Learning (CoRL)_ PMLR, 2023, pp. 2014–2025  URL: https://proceedings.mlr.press/v205/iii23a.html
  * [114] Joey Hejna and Dorsa Sadigh  “Inverse Preference Learning: Preference-based RL without a Reward Function”, 2023  URL: https://openreview.net/forum?id=ut9y3udeAo
  * [115] Matteo Hessel, Joseph Modayil, Hado Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar and David Silver  “Rainbow: Combining Improvements in Deep Reinforcement Learning”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 32.1, 2018, pp. 3215–3222  DOI: 10.1609/aaai.v32i1.11796
  * [116] Matthew Hoffman, Eric Brochu and Nando Freitas  “Portfolio Allocation for Bayesian Optimization”  In _Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI)_ AUAI Press, 2011, pp. 327–336  DOI: 10.5555/3020548.3020587
  * [117] Rachel Holladay, Shervin Javdani, Anca Dragan and Siddhartha Srinivasa  “Active Comparison Based Learning Incorporating User Uncertainty and Noise”, 2016
  * [118] Jie Huang, Jiangshan Hao, Rongshun Juan, Randy Gomez, Keisuke Nakamura and Guangliang Li  “GAN-Based Interactive Reinforcement Learning from Demonstration and Human Evaluative Feedback”  In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , 2023, pp. 4991–4998  DOI: 10.1109/ICRA48891.2023.10160939
  * [119] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg and Dario Amodei  “Reward Learning from Human Preferences and Demonstrations in Atari”  In _Advances in Neural Information Processing Systems (NIPS)_ 31 Curran Associates, Inc., 2018, pp. 8022–8034  URL: https://proceedings.neurips.cc/paper/2018/hash/8cbe9ce23f42628c98f80fa0fac8b19a-Abstract.html
  * [120] Alexander Irpan, Kanishka Rao, Konstantinos Bousmalis, Chris Harris, Julian Ibarz and Sergey Levine  “Off-Policy Evaluation via Off-Policy Classification”  In _Advances in Neural Information Processing Systems (NeurIPS)_ 32 Curran Associates, Inc., 2019, pp. 5438–5449  URL: https://proceedings.neurips.cc/paper/2019/hash/b5b03f06271f8917685d14cea7c6c50a-Abstract.html
  * [121] Charles Isbell, Christian R. Shelton, Michael Kearns, Satinder Singh and Peter Stone  “A Social Reinforcement Learning Agent”  In _Proceedings of the International Conference on Autonomous Agents (AGENTS)_ Association for Computing Machinery, 2001, pp. 377–384  DOI: 10.1145/375735.376334
  * [122] Ashesh Jain, Shikhar Sharma, Thorsten Joachims and Ashutosh Saxena  “Learning Preferences for Manipulation Tasks from Online Coactive Feedback”  In _The International Journal of Robotics Research_ 34.10 SAGE Publications Ltd STM, 2015, pp. 1296–1313  DOI: 10.1177/0278364915581193
  * [123] Erik Jenner and Adam Gleave  “Preprocessing Reward Functions for Interpretability”, 2021
  * [124] Erik Jenner, Joar Max Viktor Skalse and Adam Gleave  “A General Framework for Reward Function Distances”, 2022  URL: https://openreview.net/forum?id=Hn21kZHiCK
  * [125] Hong Jun Jeon, Smitha Milli and Anca Dragan  “Reward-Rational (Implicit) Choice: A Unifying Formalism for Reward Learning”  In _Advances in Neural Information Processing Systems (NeurIPS)_ 33 Curran Associates, Inc., 2020, pp. 4415–4426  URL: https://proceedings.neurips.cc/paper/2020/hash/2f10c1578a0706e06b6d7db6f0b4a6af-Abstract.html
  * [126] Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Kwan Yee Ng, Juntao Dai, Xuehai Pan, Aidan O’Gara, Yingshan Lei, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang, Yizhou Wang, Song-Chun Zhu, Yike Guo and Wen Gao  “AI Alignment: A Comprehensive Survey”, 2023  arXiv:2310.19852
  * [127] Xiang Ji, Huazheng Wang, Minshuo Chen, Tuo Zhao and Mengdi Wang  “Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problems”, 2023  arXiv:2307.12975
  * [128] Nan Jiang and Lihong Li  “Doubly Robust Off-policy Value Evaluation for Reinforcement Learning”  In _Proceedings of the International Conference on Machine Learning (ICML)_ PMLR, 2016, pp. 652–661  URL: https://proceedings.mlr.press/v48/jiang16.html
  * [129] Chi Jin, Zhuoran Yang, Zhaoran Wang and Michael I. Jordan  “Provably Efficient Reinforcement Learning with Linear Function Approximation”  In _Proceedings of the Conference on Learning Theory (COLT)_ PMLR, 2020, pp. 2137–2143  URL: https://proceedings.mlr.press/v125/jin20a.html
  * [130] Ying Jin, Zhuoran Yang and Zhaoran Wang  “Is Pessimism Provably Efficient for Offline RL?”  In _Proceedings of the International Conference on Machine Learning (ICML)_ PMLR, 2021, pp. 5084–5096  URL: https://proceedings.mlr.press/v139/jin21e.html
  * [131] Gregory Kahn, Pieter Abbeel and Sergey Levine  “LaND: Learning to Navigate From Disengagements”  In _IEEE Robotics and Automation Letters_ 6.2, 2021, pp. 1872–1879  DOI: 10.1109/LRA.2021.3060404
  * [132] Akansha Kalra and Daniel S. Brown  “Interpretable Reward Learning via Differentiable Decision Trees”, 2022  URL: https://openreview.net/forum?id=3bk40MsYjet
  * [133] Akansha Kalra and Daniel S. Brown  “Can Differentiable Decision Trees Learn Interpretable Reward Functions?”, 2023  arXiv:2306.13004
  * [134] Yachen Kang, Diyuan Shi, Jinxin Liu, Li He and Donglin Wang  “Beyond Reward: Offline Preference-guided Policy Optimization”  In _Proceedings of the International Conference on Machine Learning (ICML)_ PMLR, 2023, pp. 15753–15768  URL: https://proceedings.mlr.press/v202/kang23b.html
  * [135] Sydney M. Katz, Amir Maleki, Erdem Bıyık and Mykel J. Kochenderfer  “Preference-Based Learning of Reward Function Features”, 2021  arXiv:2103.02727
  * [136] Timo Kaufmann, Sarah Ball, Jacob Beck, Frauke Kreuter and Eyke Hüllermeier  “On the Challenges and Practices of Reinforcement Learning from Real Human Feedback”, 2023
  * [137] Hadi Kazemi, Fariborz Taherkhani and Nasser M. Nasrabadi  “Preference-Based Image Generation”  In _Proceedings of IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , 2020, pp. 3393–3402  DOI: 10.1109/WACV45572.2020.9093406
  * [138] Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel and Kimin Lee  “Preference Transformer: Modeling Human Preferences Using Transformers for RL”  In _Proceedings of International Conference on Learning Representations (ICLR)_ , 2023  URL: https://openreview.net/forum?id=Peot1SFDX0
  * [139] Kuno Kim, Shivam Garg, Kirankumar Shiragur and Stefano Ermon  “Reward Identification in Inverse Reinforcement Learning”  In _Proceedings of the International Conference on Machine Learning (ICML)_ PMLR, 2021, pp. 5496–5505  URL: https://proceedings.mlr.press/v139/kim21c.html
  * [140] Jan H. Kirchner, Logan Smith, Jacques Thibodeau, Kyle McDonell and Laria Reynolds  “Researching Alignment Research: Unsupervised Analysis”, 2022  arXiv:2206.02841
  * [141] W. Knox  “Learning from Human-generated Reward”, 2012  URL: https://repositories.lib.utexas.edu/items/20b9e8a1-a78d-4844-816f-3c0b0a4c848a
  * [142] W. Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt and Peter Stone  “Reward (Mis)Design for Autonomous Driving”  In _Artificial Intelligence_ 316, 2023, pp. 103829  DOI: 10.1016/j.artint.2022.103829
  * [143] W. Knox, Stephane Hatgis-Kessell, Serena Booth, Scott Niekum, Peter Stone and Alessandro Allievi  “Models of Human Preference for Learning Reward Functions”, 2022  arXiv:2206.02231
  * [144] W. Knox and Peter Stone  “TAMER: Training an Agent Manually via Evaluative Reinforcement”  In _Proceedings of the IEEE International Conference on Development and Learning (ICDL)_ , 2008, pp. 292–297  DOI: 10.1109/DEVLRN.2008.4640845
  * [145] Dingwen Kong and Lin Yang  “Provably Feedback-Efficient Reinforcement Learning via Active Reward Learning”  In _Advances in Neural Information Processing Systems (NeurIPS)_ 35, 2022, pp. 11063–11078  URL: https://proceedings.neurips.cc/paper/2022/hash/476c289f685e27936aa089e9d53a4213-Abstract-Conference.html
  * [146] Pallavi Koppol, Henny Admoni and Reid Simmons  “Iterative Interactive Reward Learning”, 2020
  * [147] Anna Korba, Stéphan Clemencon and Eric Sibony  “A Learning Theory of Ranking Aggregation”  In _Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS)_ PMLR, 2017, pp. 1001–1010  URL: https://proceedings.mlr.press/v54/korba17a.html
  * [148] Samantha Krening and Karen M. Feigh  “Interaction Algorithm Effect on Human Experience with Reinforcement Learning”  In _ACM Transactions on Human-Robot Interaction_ 7.2, 2018, pp. 16:1–16:22  DOI: 10.1145/3277904
  * [149] Andras Kupcsik, David Hsu and Wee Sun Lee  “Learning Dynamic Robot-to-Human Object Handover from Human Feedback”  In _Robotics Research: Volume 1_ , Springer Proceedings in Advanced Robotics  Springer International Publishing, 2018, pp. 161–176  DOI: 10.1007/978-3-319-51532-8_10
  * [150] Nathan Lambert  “Reward Is Not Enough”, 2021  Democratizing Automation  URL: https://robotic.substack.com/p/reward-is-not-enough
  * [151] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel and Aravind Srinivas  “Reinforcement Learning with Augmented Data”  In _Advances in Neural Information Processing Systems (NeurIPS)_ 33 Curran Associates, Inc., 2020, pp. 19884–19895  URL: https://proceedings.neurips.cc/paper/2020/hash/e615c82aba461681ade82da2da38004a-Abstract.html
  * [152] Tor Lattimore and Csaba Szepesvári  “Bandit Algorithms”  Cambridge University Press, 2020  DOI: 10.1017/9781108571401
  * [153] Hoang Le, Cameron Voloshin and Yisong Yue  “Batch Policy Learning under Constraints”  In _Proceedings of the International Conference on Machine Learning (ICML)_ PMLR, 2019, pp. 3703–3712  URL: https://proceedings.mlr.press/v97/le19a.html
  * [154] Kimin Lee, Kibok Lee, Jinwoo Shin and Honglak Lee  “Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning”  In _Proceedings of International Conference on Learning Representations (ICLR)_ , 2020  URL: https://openreview.net/forum?id=HJgcvJBFvB
  * [155] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh and Shixiang Shane Gu  “Aligning Text-to-Image Models Using Human Feedback”, 2023  arXiv:2302.12192
  * [156] Kimin Lee, Laura Smith, Anca Dragan and Pieter Abbeel  “B-Pref: Benchmarking Preference-Based Reinforcement Learning”, 2021  URL: https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/d82c8d1619ad8176d665453cfb2e55f0-Abstract-round1.html
  * [157] Kimin Lee, Laura M. Smith and Pieter Abbeel  “PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training”  In _Proceedings of the International Conference on Machine Learning (ICML)_ PMLR, 2021, pp. 6152–6163  URL: https://proceedings.mlr.press/v139/lee21i.html
  * [158] Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter J. Bentley, Samuel Bernard, Guillaume Beslon, David M. Bryson, Nick Cheney, Patryk Chrabaszcz, Antoine Cully, Stephane Doncieux, Fred C. Dyer, Kai Olav Ellefsen, Robert Feldt, Stephan Fischer, Stephanie Forrest, Antoine Fŕenoy, Christian Gagńe, Leni Le Goff, Laura M. Grabowski, Babak Hodjat, Frank Hutter, Laurent Keller, Carole Knibbe, Peter Krcah, Richard E. Lenski, Hod Lipson, Robert MacCurdy, Carlos Maestre, Risto Miikkulainen, Sara Mitri, David E. Moriarty, Jean-Baptiste Mouret, Anh Nguyen, Charles Ofria, Marc Parizeau, David Parsons, Robert T. Pennock, William F. Punch, Thomas S. Ray, Marc Schoenauer, Eric Schulte, Karl Sims, Kenneth O. Stanley, François Taddei, Danesh Tarapore, Simon Thibault, Richard Watson, Westley Weimer and Jason Yosinski  “The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities”  In _Artificial Life_ 26.2, 2020, pp. 274–306  DOI: 10.1162/artl_a_00319
  * [159] Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini and Shane Legg  “Scalable Agent Alignment via Reward Modeling: A Research Direction”, 2018  arXiv:1811.07871
  * [160] Mengxi Li, Alper Canberk, Dylan P. Losey and Dorsa Sadigh  “Learning Human Objectives from Sequences of Physical Corrections”  In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , 2021, pp. 2877–2883  DOI: 10.1109/ICRA48506.2021.9560829
  * [161] Zihao Li, Zhuoran Yang and Mengdi Wang  “Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism”, 2023  URL: https://openreview.net/forum?id=gxM2AUFMsK
  * [162] Xinran Liang, Katherine Shu, Kimin Lee and Pieter Abbeel  “Reward Uncertainty for Exploration in Preference-based Reinforcement Learning”  In _Proceedings of International Conference on Learning Representations (ICLR)_ , 2022  URL: https://openreview.net/forum?id=OWZVD-l-ZrC
  * [163] “The Construction of Preference”  Cambridge University Press, 2006  DOI: 10.1017/CBO9780511618031
  * [164] Jinying Lin, Zhen Ma, Randy Gomez, Keisuke Nakamura, Bo He and Guangliang Li  “A Review on Interactive Reinforcement Learning From Human Social Feedback”  In _IEEE Access_ 8, 2020, pp. 120757–120765  DOI: 10.1109/ACCESS.2020.3006254
  * [165] Stephanie Lin, Jacob Hilton and Owain Evans  “TruthfulQA: Measuring How Models Mimic Human Falsehoods”  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL)_ Association for Computational Linguistics, 2022, pp. 3214–3252  DOI: 10.18653/v1/2022.acl-long.229
  * [166] Yijiong Lin, Jiancong Huang, Matthieu Zimmer, Yisheng Guan, Juan Rojas and Paul Weng  “Invariant Transform Experience Replay: Data Augmentation for Deep Reinforcement Learning”  In _IEEE Robotics and Automation Letters_ 5.4, 2020, pp. 6615–6622  DOI: 10.1109/LRA.2020.3013937
  * [167] David Lindner and Mennatallah El-Assady  “Humans Are Not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learning”, 2022
  * [168] David Lindner, Matteo Turchetta, Sebastian Tschiatschek, Kamil Ciosek and Andreas Krause  “Information Directed Reward Learning for Reinforcement Learning”  In _Advances in Neural Information Processing Systems (NeurIPS)_ 34 Curran Associates, Inc., 2021, pp. 3850–3862  URL: https://proceedings.neurips.cc/paper/2021/hash/1fa6269f58898f0e809575c9a48747ef-Abstract.html
  * [169] Runze Liu, Yali Du, Fengshuo Bai, Jiafei Lyu and Xiu Li  “Zero-Shot Preference Learning for Offline RL via Optimal Transport”, 2023  arXiv:2306.03615
  * [170] Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, Zihao Wu, Lin Zhao, Dajiang Zhu, Xiang Li, Ning Qiang, Dingang Shen, Tianming Liu and Bao Ge  “Summary of ChatGPT-Related Research and Perspective towards the Future of Large Language Models”  In _Meta-Radiology_ 1.2, 2023, pp. 100017  DOI: 10.1016/j.metrad.2023.100017
  * [171] Dylan P. Losey, Andrea Bajcsy, Marcia K. O’Malley and Anca D. Dragan  “Physical Interaction as Communication: Learning Robot Objectives Online from Human Corrections”  In _The International Journal of Robotics Research_ 41.1 SAGE Publications Ltd STM, 2022, pp. 20–44  DOI: 10.1177/02783649211050958
  * [172] Dylan P. Losey and Marcia K. O’Malley  “Including Uncertainty When Learning from Human Corrections”  In _Proceedings of the Conference on Robot Learning (CoRL)_ PMLR, 2018, pp. 123–132  URL: http://proceedings.mlr.press/v87/losey18a.html
  * [173] R. Luce  “Individual Choice Behavior”, Individual Choice Behavior  John Wiley, 1959, pp. xii153
  * [174] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson and Tim Rocktäschel  “A Survey of Reinforcement Learning Informed by Natural Language”  In _Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)_ International Joint Conferences on Artificial Intelligence Organization, 2019, pp. 6309–6317  DOI: 10.24963/ijcai.2019/880
  * [175] Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan and Anima Anandkumar  “Eureka: Human-Level Reward Design via Coding Large Language Models”, 2023  arXiv:2310.12931
  * [176] Monika Mandl, Alexander Felfernig, Erich Teppan and Monika Schubert  “Consumer Decision Making in Knowledge-Based Recommendation”  In _Journal of Intelligent Information Systems_ 37.1, 2011, pp. 1–22  DOI: 10.1007/s10844-010-0134-3
  * [177] Pascal Massart and Élodie Nédélec  “Risk Bounds for Statistical Learning”  In _The Annals of Statistics_ 34.5 Institute of Mathematical Statistics, 2006, pp. 2326–2366  DOI: 10.1214/009053606000000786
  * [178] Lev E. McKinney, Yawen Duan, David Krueger and Adam Gleave  “On The Fragility of Learned Reward Functions”, 2022  URL: https://openreview.net/forum?id=9gj9vXfeS-y
  * [179] Jorge Mendez, Shashank Shivkumar and Eric Eaton  “Lifelong Inverse Reinforcement Learning”  In _Advances in Neural Information Processing Systems (NIPS)_ 31 Curran Associates, Inc., 2018, pp. 4507–4518  URL: https://papers.nips.cc/paper/2018/hash/2d969e2cee8cfa07ce7ca0bb13c7a36d-Abstract.html
  * [180] Katherine Metcalf, Miguel Sarabia and Barry-John Theobald  “Rewards Encoding Environment Dynamics Improves Preference-based Reinforcement Learning”, 2022  arXiv:2211.06527
  * [181] Yannick Metz, David Lindner, Raphaël Baur, Daniel A. Keim and Mennatallah El-Assady  “RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedback”, 2023  URL: https://openreview.net/forum?id=JvkZtzJBFQ
  * [182] Stephanie Milani, Anssi Kanervisto, Karolis Ramanauskas, Sander Schulhoff, Brandon Houghton, Sharada Mohanty, Byron Galbraith, Ke Chen, Yan Song, Tianze Zhou, Bingquan Yu, He Liu, Kai Guan, Yujing Hu, Tangjie Lv, Federico Malato, Florian Leopold, Amogh Raut, Ville Hautamäki, Andrew Melnik, Shu Ishida, João Henriques, Robert Klassert, Walter Laurito, Lucas Cazzonelli, Cedric Kulbach, Nicholas Popovic, Marvin Schweizer, Ellen Novoseller, Vinicius Goecks, Nicholas Waytowich, David Watkins, Josh Miller and Rohin Shah  “Towards Solving Fuzzy Tasks with Human Feedback: A Retrospective of the MineRL BASALT 2022 Competition”  In _Proceedings of the NeurIPS 2022 Competitions Track_ PMLR, 2022, pp. 171–188  URL: https://proceedings.mlr.press/v220/milani22a.html
  * [183] Stephanie Milani, Nicholay Topin, Manuela Veloso and Fei Fang  “Explainable Reinforcement Learning: A Survey and Comparative Review”  In _ACM Computing Surveys_ , 2023  DOI: 10.1145/3616864
  * [184] Smitha Milli and Anca D. Dragan  “Literal or Pedagogic Human? Analyzing Human Model Misspecification in Objective Learning”  In _Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI)_ PMLR, 2020, pp. 925–934  URL: https://proceedings.mlr.press/v115/milli20a.html
  * [185] Sören Mindermann, Rohin Shah, Adam Gleave and Dylan Hadfield-Menell  “Active Inverse Reward Design”, 2018
  * [186] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg and Demis Hassabis  “Human-Level Control through Deep Reinforcement Learning”  In _Nature_ 518.7540 Nature Publishing Group, 2015, pp. 529–533  DOI: 10.1038/nature14236
  * [187] Weiyan Mu and Shifeng Xiong  “On Huber’s Contaminated Model”  In _Journal of Complexity_ 77, 2023, pp. 101745  DOI: 10.1016/j.jco.2023.101745
  * [188] Vivek Myers, Erdem Bıyık, Nima Anari and Dorsa Sadigh  “Learning Multimodal Rewards from Rankings”  In _Proceedings of the Conference on Robot Learning (CoRL)_ PMLR, 2022, pp. 342–352  URL: https://proceedings.mlr.press/v164/myers22a.html
  * [189] Vivek Myers, Erdem Bıyık and Dorsa Sadigh  “Active Reward Learning from Online Preferences”  In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , 2023, pp. 7511–7518  DOI: 10.1109/ICRA48891.2023.10160439
  * [190] Anis Najar and Mohamed Chetouani  “Reinforcement Learning With Human Advice: A Survey”  In _Frontiers in Robotics and AI_ 8, 2021, pp. 584075  DOI: 10.3389/frobt.2021.584075
  * [191] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess and John Schulman  “WebGPT: Browser-assisted Question-Answering with Human Feedback”, 2022  arXiv:2112.09332
  * [192] Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor and Peter Stone  “Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey”  In _Journal of Machine Learning Research_ 21.181, 2020, pp. 1–50  URL: http://jmlr.org/papers/v21/20-212.html
  * [193] Gergely Neu and Csaba Szepesvári  “Training Parsers by Inverse Reinforcement Learning”  In _Machine Learning_ 77.2, 2009, pp. 303–337  DOI: 10.1007/s10994-009-5110-1
  * [194] Andrew Y. Ng, Daishi Harada and Stuart J. Russell  “Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping”  In _Proceedings of the International Conference on Machine Learning (ICML)_ Morgan Kaufmann Publishers Inc., 1999, pp. 278–287
  * [195] Andrew Y. Ng and Stuart J. Russell  “Algorithms for Inverse Reinforcement Learning”  In _Proceedings of the International Conference on Machine Learning (ICML)_ Morgan Kaufmann Publishers Inc., 2000, pp. 663–670
  * [196] Ellen Novoseller, Yibing Wei, Yanan Sui, Yisong Yue and Joel Burdick  “Dueling Posterior Sampling for Preference-Based Reinforcement Learning”  In _Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI)_ PMLR, 2020, pp. 1029–1038  URL: https://proceedings.mlr.press/v124/novoseller20a.html
  * [197] OpenAI  “ChatGPT: Optimizing Language Models for Dialogue”, 2022  URL: https://openai.com/blog/chatgpt
  * [198] OpenAI  “GPT-4 Technical Report”, 2023  URL: https://cdn.openai.com/papers/gpt-4.pdf
  * [199] Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J. Bagnell, Pieter Abbeel and Jan Peters  “An Algorithmic Perspective on Imitation Learning”  In _Foundations and Trends® in Robotics_ 7.1-2 Now Publishers, Inc., 2018, pp. 1–179  DOI: 10.1561/2300000053
  * [200] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike and Ryan Lowe  “Training Language Models to Follow Instructions with Human Feedback”  In _Advances in Neural Information Processing Systems (NeurIPS)_ 35, 2022, pp. 27730–27744  URL: https://proceedings.neurips.cc/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html
  * [201] Cosmin Paduraru  “Off-Policy Evaluation in Markov Decision Processes”, 2013  URL: https://escholarship.mcgill.ca/concern/theses/p8418r74h
  * [202] Malayandi Palan, Gleb Shevchuk, Nicholas Charles Landolfi and Dorsa Sadigh  “Learning Reward Functions by Integrating Human Demonstrations and Preferences”  In _Proceedings of Robotics: Science and Systems (RSS)_ 15, 2019  URL: http://www.roboticsproceedings.org/rss15/p23.html
  * [203] Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel and Kimin Lee  “SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning”  In _Proceedings of International Conference on Learning Representations (ICLR)_ , 2022  URL: https://openreview.net/forum?id=TfhfZLQ2EJO
  * [204] André Susano Pinto, Alexander Kolesnikov, Yuge Shi, Lucas Beyer and Xiaohua Zhai  “Tuning Computer Vision Models With Task Rewards”  In _Proceedings of the International Conference on Machine Learning (ICML)_ PMLR, 2023, pp. 33229–33239  URL: https://proceedings.mlr.press/v202/susano-pinto23a.html
  * [205] R.. Plackett  “The Analysis of Permutations”  In _Journal of the Royal Statistical Society. Series C (Applied Statistics)_ 24.2 [Wiley, Royal Statistical Society], 1975, pp. 193–202  DOI: 10.2307/2346567
  * [206] Alina Pommeranz, Joost Broekens, Pascal Wiggers, Willem-Paul Brinkman and Catholijn M. Jonker  “Designing Interfaces for Explicit Preference Elicitation: A User-Centered Investigation of Preference Representation and Elicitation Process”  In _User Modeling and User-Adapted Interaction_ 22.4, 2012, pp. 357–397  DOI: 10.1007/s11257-011-9116-6
  * [207] Benjamin Poole and Minwoo Lee  “Towards Intrinsic Interactive Reinforcement Learning”, 2022  arXiv:2112.01575
  * [208] Doina Precup, Richard S. Sutton and Satinder P. Singh  “Eligibility Traces for Off-Policy Policy Evaluation”  In _Proceedings of the International Conference on Machine Learning (ICML)_ Morgan Kaufmann Publishers Inc., 2000, pp. 759–766
  * [209] Erika Puiutta and Eric M… Veith  “Explainable Reinforcement Learning: A Survey”  In _Proceedings of Machine Learning and Knowledge Extraction (CD-MAKE)_ Springer International Publishing, 2020, pp. 77–95  DOI: 10.1007/978-3-030-57321-8_5
  * [210] Junqi Qian, Paul Weng and Chenmien Tan  “Learning Rewards to Optimize Global Performance Metrics in Deep Reinforcement Learning”  In _Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS)_ IFAAMAS, 2023, pp. 1951–1960  DOI: 10.5555/3545946.3598864
  * [211] Yunpeng Qing, Shunyu Liu, Jie Song, Huiqiong Wang and Mingli Song  “A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges”, 2023  arXiv:2211.06665
  * [212] Mattia Racca, Antti Oulasvirta and Ville Kyrki  “Teacher-Aware Active Robot Learning”  In _Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI)_ , 2019, pp. 335–343  DOI: 10.1109/HRI.2019.8673300
  * [213] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon and Chelsea Finn  “Direct Preference Optimization: Your Language Model Is Secretly a Reward Model”, 2023  URL: https://openreview.net/forum?id=HPuSIXJaa9
  * [214] Matthew Rahtz, Vikrant Varma, Ramana Kumar, Zachary Kenton, Shane Legg and Jan Leike  “Safe Deep RL in 3D Environments Using Human Feedback”, 2022  arXiv:2201.08102
  * [215] Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi and Yejin Choi  “Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization”  In _Proceedings of International Conference on Learning Representations (ICLR)_ , 2023  URL: https://openreview.net/forum?id=8aHzds2uUyB
  * [216] Siddharth Reddy, Anca Dragan, Sergey Levine, Shane Legg and Jan Leike  “Learning Human Objectives by Evaluating Hypothetical Behavior”  In _Proceedings of the International Conference on Machine Learning (ICML)_ PMLR, 2020, pp. 8020–8029  URL: https://proceedings.mlr.press/v119/reddy20a.html
  * [217] Kevin Regan and Craig Boutilier  “Regret-Based Reward Elicitation for Markov Decision Processes”  In _Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI)_ AUAI Press, 2009, pp. 444–451  URL: https://dl.acm.org/doi/10.5555/1795114.1795166
  * [218] Kevin Regan and Craig Boutilier  “Robust Online Optimization of Reward-Uncertain MDPs”  In _Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)_ AAAI Press, 2011, pp. 2165–2171  DOI: 10.5591/978-1-57735-516-8/IJCAI11-361
  * [219] Zhizhou Ren, Anji Liu, Yitao Liang, Jian Peng and Jianzhu Ma  “Efficient Meta Reinforcement Learning for Preference-based Fast Adaptation”  In _Advances in Neural Information Processing Systems (NeurIPS)_ 35, 2022, pp. 15502–15515  URL: https://papers.nips.cc/paper_files/paper/2022/hash/63b2b056f48653b7cff0d8d233c96a4d-Abstract-Conference.html
  * [220] Daniel Russo and Benjamin Van Roy  “Eluder Dimension and the Sample Complexity of Optimistic Exploration”  In _Advances in Neural Information Processing Systems (NIPS)_ 26 Curran Associates, Inc., 2013, pp. 2256–2264  URL: https://papers.nips.cc/paper/2013/hash/41bfd20a38bb1b0bec75acf0845530a7-Abstract.html
  * [221] John Rust  “Optimal Replacement of GMC Bus Engines: An Empirical Model of Harold Zurcher”  In _Econometrica_ 55.5 [Wiley, Econometric Society], 1987, pp. 999–1033  DOI: 10.2307/1911259
  * [222] “Using Discrete Choice Experiments to Value Health and Health Care” 11, The Economics of Non-Market Goods and Resources  Springer Netherlands, 2008  DOI: 10.1007/978-1-4020-5753-3
  * [223] Dorsa Sadigh, Anca Dragan, Shankar Sastry and Sanjit Seshia  “Active Preference-Based Learning of Reward Functions”  In _Proceedings of Robotics: Science and Systems (RSS)_ 13, 2017  URL: http://www.roboticsproceedings.org/rss13/p53.html
  * [224] Aadirupa Saha  “Optimal Algorithms for Stochastic Contextual Preference Bandits”  In _Advances in Neural Information Processing Systems (NeurIPS)_ 34 Curran Associates, Inc., 2021, pp. 30050–30062  URL: https://proceedings.neurips.cc/paper/2021/hash/fc3cf452d3da8402bebb765225ce8c0e-Abstract.html
  * [225] Aadirupa Saha, Aldo Pacchiano and Jonathan Lee  “Dueling RL: Reinforcement Learning with Trajectory Preferences”  In _Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS)_ PMLR, 2023, pp. 6263–6289  URL: https://proceedings.mlr.press/v206/saha23a.html
  * [226] William Saunders, Girish Sastry, Andreas Stuhlmüller and Owain Evans  “Trial without Error: Towards Safe Reinforcement Learning via Human Intervention”  In _Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS)_ IFAAMAS, 2018, pp. 2067–2069  DOI: 10.5555/3237383.3238074
  * [227] Marc Schoenauer, Riad Akrour, Michele Sebag and Jean-Christophe Souplet  “Programming by Feedback”  In _Proceedings of the International Conference on Machine Learning (ICML)_ PMLR, 2014, pp. 1503–1511  URL: https://proceedings.mlr.press/v32/schoenauer14.html
  * [228] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford and Oleg Klimov  “Proximal Policy Optimization Algorithms”, 2017  arXiv:1707.06347
  * [229] Max Schwarzer, Ankesh Anand, Rishab Goel, R. Hjelm, Aaron Courville and Philip Bachman  “Data-Efficient Reinforcement Learning with Self-Predictive Representations”  In _Proceedings of International Conference on Learning Representations (ICLR)_ , 2022  URL: https://openreview.net/forum?id=uCQfPZwRaUu
  * [230] Burr Settles  “Active Learning”  Morgan & Claypool Publishers, 2012
  * [231] Rohin Shah, Cody Wild, Steven H. Wang, Neel Alex, Brandon Houghton, William Guss, Sharada Mohanty, Anssi Kanervisto, Stephanie Milani, Nicholay Topin, Pieter Abbeel, Stuart Russell and Anca Dragan  “The MineRL BASALT Competition on Learning from Human Feedback”, 2021  arXiv:2107.01969
  * [232] Daniel Shin, Anca Dragan and Daniel S. Brown  “Benchmarks and Algorithms for Offline Preference-Based Reward Learning”  In _Transactions on Machine Learning Research_ , 2023  URL: https://openreview.net/forum?id=TGuXXlbKsn
  * [233] Pannaga Shivaswamy and Thorsten Joachims  “Online Structured Prediction via Coactive Learning”  In _Proceedings of the International Conference on Machine Learning (ICML)_ Omnipress, 2012, pp. 59–66  URL: http://icml.cc/2012/papers/717.pdf
  * [234] Pannaga Shivaswamy and Thorsten Joachims  “Coactive Learning”  In _Journal of Artificial Intelligence Research_ 53, 2015, pp. 1–40  DOI: 10.1613/jair.4539
  * [235] Umer Siddique, Abhinav Sinha and Yongcan Cao  “Fairness in Preference-based Reinforcement Learning”, 2023  URL: https://openreview.net/forum?id=ColATVnkEl
  * [236] David Silver, Satinder Singh, Doina Precup and Richard S. Sutton  “Reward Is Enough”  In _Artificial Intelligence_ 299, 2021, pp. 103535  DOI: 10.1016/j.artint.2021.103535
  * [237] Avi Singh, Larry Yang, Chelsea Finn and Sergey Levine  “End-To-End Robotic Reinforcement Learning without Reward Engineering”  In _Proceedings of Robotics: Science and Systems (RSS)_ 15, 2019  URL: http://www.roboticsproceedings.org/rss15/p73.html
  * [238] Joar Max Viktor Skalse and Alessandro Abate  “The Reward Hypothesis Is False”, 2022  URL: https://openreview.net/forum?id=5l1NgpzAfH
  * [239] Joar Max Viktor Skalse and Alessandro Abate  “Misspecification in Inverse Reinforcement Learning”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 37.12, 2023, pp. 15136–15143  DOI: 10.1609/aaai.v37i12.26766
  * [240] Joar Max Viktor Skalse, Lucy Farnik, Sumeet Ramesh Motwani, Erik Jenner, Adam Gleave and Alessandro Abate  “STARC: A General Framework For Quantifying Differences Between Reward Functions”, 2023  arXiv:2309.15257
  * [241] Joar Max Viktor Skalse, Matthew Farrugia-Roberts, Stuart Russell, Alessandro Abate and Adam Gleave  “Invariance in Policy Optimisation and Partial Identifiability in Reward Learning”  In _Proceedings of the International Conference on Machine Learning (ICML)_ PMLR, 2023, pp. 32033–32058  URL: https://proceedings.mlr.press/v202/skalse23a.html
  * [242] Joar Max Viktor Skalse, Nikolaus Howe, Dmitrii Krasheninnikov and David Krueger  “Defining and Characterizing Reward Gaming”  In _Advances in Neural Information Processing Systems (NeurIPS)_ 35, 2022, pp. 9460–9471  URL: https://proceedings.neurips.cc/paper_files/paper/2022/hash/3d719fee332caa23d5038b8a90e81796-Abstract-Conference.html
  * [243] Ziang Song, Tianle Cai, Jason D. Lee and Weijie J. Su  “Reward Collapse in Aligning Large Language Models: A Prompt-Aware Approach to Preference Rankings”, 2023  URL: https://openreview.net/forum?id=dpWxK6aqIK
  * [244] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei and Paul F Christiano  “Learning to Summarize with Human Feedback”  In _Advances in Neural Information Processing Systems (NeurIPS)_ 33 Curran Associates, Inc., 2020, pp. 3008–3021  URL: https://proceedings.neurips.cc/paper/2020/hash/1f89885d556929e98d3ef9b86448f951-Abstract.html
  * [245] Hiroaki Sugiyama, Toyomi Meguro and Yasuhiro Minami  “Preference-Learning Based Inverse Reinforcement Learning for Dialog Control”  In _Proceedings of Interspeech_ ISCA, 2012, pp. 222–225  DOI: 10.21437/Interspeech.2012-72
  * [246] Yanan Sui, Vincent Zhuang, Joel W. Burdick and Yisong Yue  “Multi-Dueling Bandits with Dependent Arms”  In _Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI)_ AUAI Press, 2017  URL: http://auai.org/uai2017/proceedings/papers/155.pdf
  * [247] Yanan Sui, Masrour Zoghi, Katja Hofmann and Yisong Yue  “Advancements in Dueling Bandits”  In _Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)_ International Joint Conferences on Artificial Intelligence Organization, 2018, pp. 5502–5510  DOI: 10.24963/ijcai.2018/776
  * [248] Richard S. Sutton and Andrew G. Barto  “Reinforcement Learning: An Introduction”, Adaptive Computation and Machine Learning Series  The MIT Press, 2018
  * [249] Louis Leon Thurstone  “A Law of Comparative Judgment”  In _Psychological Review_ 34 Psychological Review Company, 1927, pp. 273–286  DOI: 10.1037/h0070288
  * [250] Jeremy Tien, Jerry Zhi-Yang He, Zackory Erickson, Anca Dragan and Daniel S. Brown  “Causal Confusion and Reward Misidentification in Preference-Based Reward Learning”  In _Proceedings of International Conference on Learning Representations (ICLR)_ , 2023  URL: https://openreview.net/forum?id=R0Xxvr_X3ZA
  * [251] Kenneth E. Train  “Discrete Choice Methods with Simulation”  Cambridge University Press, 2009  DOI: 10.1017/CBO9780511805271
  * [252] Thi Ngoc Trang Tran, Alexander Felfernig and Nava Tintarev  “Humanized Recommender Systems: State-of-the-art and Research Issues”  In _ACM Transactions on Interactive Intelligent Systems_ 11.2, 2021, pp. 9:1–9:41  DOI: 10.1145/3446906
  * [253] Maegan Tucker, Kejun Li, Yisong Yue and Aaron D. Ames  “POLAR: Preference Optimization and Learning Algorithms for Robotics”, 2022  arXiv:2208.04404
  * [254] L.. Valiant  “A Theory of the Learnable”  In _Communications of the ACM_ 27.11, 1984, pp. 1134–1142  DOI: 10.1145/1968.1972
  * [255] Peter Vamplew, Benjamin J. Smith, Johan Källström, Gabriel Ramos, Roxana Rădulescu, Diederik M. Roijers, Conor F. Hayes, Fredrik Heintz, Patrick Mannion, Pieter J.. Libin, Richard Dazeley and Cameron Foale  “Scalar Reward Is Not Enough: A Response to Silver, Singh, Precup and Sutton (2021)”  In _Autonomous Agents and Multi-Agent Systems_ 36.2, 2022, pp. 41  DOI: 10.1007/s10458-022-09575-5
  * [256] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser and Illia Polosukhin  “Attention Is All You Need”  In _Advances in Neural Information Processing Systems (NIPS)_ 30 Curran Associates, Inc., 2017, pp. 5998–6008  URL: https://papers.nips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
  * [257] Mudit Verma, Siddhant Bhambri and Subbarao Kambhampati  “Exploiting Unlabeled Data for Feedback Efficient Human Preference Based Reinforcement Learning”, 2023
  * [258] Mudit Verma and Subbarao Kambhampati  “A State Augmentation Based Approach to Reinforcement Learning from Human Preferences”, 2023
  * [259] Mudit Verma and Subbarao Kambhampati  “Data Driven Reward Initialization for Preference Based Reinforcement Learning”, 2023
  * [260] Mudit Verma and Katherine Metcalf  “Symbol Guided Hindsight Priors for Reward Learning from Human Preferences”, 2022  arXiv:2210.09151
  * [261] Haoran Wang, Qiuye Jin, Shiman Li, Siyu Liu, Manning Wang and Zhijian Song  “A Comprehensive Survey on Deep Active Learning and Its Applications in Medical Image Analysis”, 2023  arXiv:2310.14230
  * [262] Yuanhao Wang, Qinghua Liu and Chi Jin  “Is RLHF More Difficult than Standard RL? A Theoretical Perspective”, 2023  URL: https://openreview.net/forum?id=sxZLrBqg50