# A Survey of Reinforcement Learning
from Human Feedback

Timo Kaufmann
LMU Munich, MCML Munich
Munich, Germany
<EMAIL_ADDRESS>&Paul Weng
Duke Kunshan University
Kunshan, China
<EMAIL_ADDRESS>&Viktor Bengs
LMU Munich, MCML Munich
Munich, Germany
<EMAIL_ADDRESS>&Eyke Hüllermeier
LMU Munich, MCML Munich
Munich, Germany
<EMAIL_ADDRESS>
(December 2023)

###### Abstract

Reinforcement learning from human feedback (RLHF) is a variant of
reinforcement learning (RL) that learns from human feedback instead of relying
on an engineered reward function. Building on prior work on the related
setting of preference-based reinforcement learning (PbRL), it stands at the
intersection of artificial intelligence and human-computer interaction. This
positioning offers a promising avenue to enhance the performance and
adaptability of intelligent systems while also improving the alignment of
their objectives with human values. The training of large language models
(LLMs) has impressively demonstrated this potential in recent years, where
RLHF played a decisive role in targeting the model’s capabilities toward human
objectives. This article provides a comprehensive overview of the fundamentals
of RLHF, exploring the intricate dynamics between machine agents and human
input. While recent focus has been on RLHF for LLMs, our survey adopts a
broader perspective, examining the diverse applications and wide-ranging
impact of the technique. We delve into the core principles that underpin RLHF,
shedding light on the symbiotic relationship between algorithms and human
feedback, and discuss the main research trends in the field. By synthesizing
the current landscape of RLHF research, this article aims to provide
researchers as well as practitioners with a comprehensive understanding of
this rapidly growing field of research.

_Keywords_ Preference-Based Reinforcement Learning $\cdot$ Human Feedback
$\cdot$ Preference Learning $\cdot$ Interactive Learning $\cdot$ Reward
Learning $\cdot$ Reward Modeling $\cdot$ Agent Alignment $\cdot$ Human-in-the-
Loop $\cdot$ Human-Computer Interaction $\cdot$ Human-Compatible AI

###### Contents

  1. 1 Introduction
    1. 1.1 Why Human Feedback
    2. 1.2 The Origins of Reinforcement Learning from Human Feedback
    3. 1.3 Scope of the Survey
    4. 1.4 Prior Surveys
    5. 1.5 Outline
  2. 2 Preliminaries
    1. 2.1 Reinforcement Learning
    2. 2.2 Preference-Based MDPs
    3. 2.3 Reward Learning
    4. 2.4 Reinforcement Learning from Human Feedback
    5. 2.5 Active Learning
  3. 3 Feedback
    1. 3.1 Attributes of Feedback Types
    2. 3.2 Common Classes
    3. 3.3 Initializations
  4. 4 Label Collection
    1. 4.1 Active Learning
    2. 4.2 Challenges of Human Labeling
  5. 5 Reward Model Training
    1. 5.1 Human Feedback Model
    2. 5.2 Utility Learning
    3. 5.3 Evaluating Learned Reward Functions
    4. 5.4 Reward Model Inputs
    5. 5.5 Increasing Feedback Efficiency
  6. 6 Theory
    1. 6.1 Policy Learning
    2. 6.2 Preference-Based vs. Reward-Based Learning
  7. 7 Applications and Benchmarks
    1. 7.1 Applications
    2. 7.2 Supporting Libraries
    3. 7.3 Benchmarks
    4. 7.4 Evaluation
  8. 8 Discussion and Conclusion

##  1 Introduction

In reinforcement learning (RL), an agent traditionally navigates through an
environment and attempts to make optimal actions or decisions through a
process of trial and error. Whether a decision is optimal or not is determined
solely by reward signals. These signals have to be defined manually based on
measurements of the agent’s performance, ensuring that the learning agent
receives the necessary signals to learn the correct behavior. Designing a
reward function by hand, however, is challenging. Success is hard to formally
define and measure in many applications. Beyond that, a sparse signal of
success may not be well suited for agent learning – resulting in the need for
_reward shaping_ [194], where the reward signal is transformed into one that
is more suitable for learning. This often makes the reward signal more
susceptible to spurious correlations, however – behaviors that are rewarded
because they are usually correlated with the true objective but are not
valuable in themselves. This ultimately cumulates in the issue of _reward
hacking_ [242], where learning agents exploit reward-specific loopholes to
achieve undesired outcomes while still generating high rewards.

In response to these challenges, reinforcement learning from human feedback
(RLHF) has emerged as a practically meaningful alternative that introduces a
critical human-in-the-loop component to the standard RL learning paradigm. In
a nutshell, RLHF differs from RL in that the objective is defined and
iteratively refined by the human in the loop instead of being specified ahead
of time. This approach not only has the potential to overcome the limitations
and issues of classical RL methods but also has potential benefits for agent
alignment, where the agent’s learning goals are more closely aligned with
human values, promoting ethically sound and socially responsible AI systems.

RLHF has seen a number of successful applications, advances in methodology,
and theoretical insights since the last comparable survey [270]. The
applications range from the domain of large language models (LLMs) [197] over
image generation [155], continuous control [58] and games [119] to robotics
[113]. At the same time, there have been a lot of developments of the
methodology since the last comparable survey [270]. Examples of methodological
developments include using data augmentation and semi-supervised learning
approaches to improve sample complexity [203], using meta-learning to quickly
adapt the learned preferences to new tasks [219], fusing multiple feedback
types [202], using self-supervised representation learning to increase
feedback efficiency [180], actively synthesizing hypothetical behaviors for
queries [216], and optimizing queries for ease of answer [30]. Finally, there
have been some achievements with regard to theoretical results for the field
of RLHF, providing new insights but also new questions for the fundamental
mathematical problems underlying the modeling of the learning scenario in
RLHF.

In this survey, we, therefore, discuss the current state of affairs with
regard to the ongoing research in RLHF, classify the current approaches as
well as concisely describe their main characteristics, and give a brief
overview of the application areas.

###  1.1 Why Human Feedback

In conventional RL, the agent’s objective is defined by a reward function
which it aims to maximize [248]. Specifying this reward function can be
challenging, particularly in complex domains: What would be a suitable reward
function for a robot assisting humans in a household environment or for
autonomous vehicles navigating through a busy urban environment? Moreover,
even well-defined reward functions can lead to unexpected behaviors due to
distributional shifts or over-optimization, raising practical and safety
concerns. Learning the agent’s objective from human feedback circumvents
reward engineering challenges and fosters robust training, with the reward
function dynamically refined and adjusted to distributional shifts as the
agent learns.

##### Feedback vs. Demonstrations

The field of inverse RL aims to infer reward functions from human
demonstrations [7]. While this can partially resolve reward engineering
challenges, it faces inherent difficulties: (i) It is generally not possible
to robustly identify rewards from demonstrations [43], (ii) it is only
applicable in scenarios where good demonstrations can be obtained, (iii) it
struggles to outperform the demonstrator, and (iv) humans often do not
demonstrate the behavior they would prefer a machine to adopt [20].
Interactive feedback, by contrast, can use active queries to differentiate
between human preferences and irrelevant noise, is much easier to provide than
demonstrations, does not require near-optimal performance from the human
evaluators, and elicits preferences on the behavior that a human would prefer
from the machine. Interactive feedback can also be used to complement
demonstrations, in which case it can be used to shape and refine capabilities
learned through initial training, like behavioral cloning, thereby preventing
overfitting to demonstrated behavior [2].

##### Avoiding Reward Engineering

Reward engineering in RL presents significant challenges, as accurately
specifying reward functions is notoriously difficult [5, 142]. These
challenges can be mitigated by utilizing human feedback, which enables
training agents for tasks that are hard to define manually and helps avoid
safety issues arising from misaligned rewards [242]. Safety issues related to
a misalignment between the agent’s and the human’s objectives are studied as
the AI alignment problem [85], in particular agent alignment and value
alignment [140]. Although the effectiveness of RLHF in resolving these
alignment issues is debated [57], it presents a promising approach to enhance
alignment [159].

Excessive optimization for poorly specified rewards often leads to unintended
behaviors. Agents may exploit simulation flaws for higher rewards [158, 14] or
engage in _reward hacking_ [242], where the behavior maximizes the specified
reward but deviates from the intended objective. This is evident in cases
where agents focus on intermediate rewards without achieving the actual goal
[59] or prematurely exit a game to avoid negative rewards [226]. The root of
these issues is that the reward function does not properly reflect the actual
learning task. While these issues may seem trivial in game-like environments,
their implications are far more serious in safety-critical contexts such as
healthcare and autonomous driving. In these settings, it is crucial to prevent
misaligned reward functions from leading to harmful outcomes, like a care
robot causing injury or a self-driving car jeopardizing road safety.

###  1.2 The Origins of Reinforcement Learning from Human Feedback

Learning behavior from human feedback has long been studied as a subfield of
RL, but methods and terminology have evolved over time. Early methods, as
discussed in more detail by [141], focused on learning directly from human
rewards [121, 144]. This survey, however, focuses on more indirect approaches
of inferring the objective from human feedback.

Reinforcement learning from human feedback (RLHF) in its modern guise has its
origin in the setting of preference-based reinforcement learning (PbRL) as
introduced independently by [3] and [53]. The original idea of preference-
based reinforcement learning (PbRL) is to infer the objective from qualitative
feedback, such as pairwise preferences between behaviors or between actions
given states, instead of quantitiative feedback in the form of numerical
rewards. The term RLHF was coined as an alternative later on [9, 200, 197],
though initially referring to the same concept of learning behavior from
relative feedback.

Table 1: Feedback types classified as belonging to PbRL, SSRL, and RLHF as defined in this survey. Feedback Type | PbRL | SSRL | RLHF
---|---|---|---
Binary trajectory comparisons | ✓ | ✗ | ✓
Trajectory rankings | ✓ | ✗ | ✓
State preferences | ✓ | ✗ | ✓
Action preferences | ✓ | ✗ | ✓
Binary critique | ✗ | ✓ | ✓
Scalar feedback | ✗ | ✓ | ✓
Corrections | ✗ | ✗ | ✓
Action advice | ✗ | ✗ | ✓
Implicit feedback | ✗ | ✗ | ✓
Natural language | ✗ | ✗ | ✓

Disentangling PbRL and RLHF is challenging due to their overlapping use in the
literature. For instance, [58] themselves are using the term PbRL, yet are
often cited as a seminal reference for RLHF [65, 200]. This indicates the
interchangeability of these terms. Practically, RLHF is often associated with
reward modeling and deep RL, while PbRL is often linked to direct policy
optimization in traditional RL settings. This is underlined by [125], who
characterize PbRL as limited to direct policy learning from preferences. This
is in contrast with other sources, however, who include reward learning within
the scope of RLHF [58, 270].

Despite the overlapping and sometimes conflicting usage, RLHF is increasingly
viewed as a generalization of PbRL. While both involve human feedback to
define RL objectives, PbRL primarily focuses on relative feedback, such as
binary comparisons and rankings. RLHF not only includes these aspects but also
extends to a wider range of feedback types [181]. Table 1 gives an exemplary
overview of our interpretation of these terms.

Another concept, semi-supervised reinforcement learning (SSRL), introduced by
[56] and discussed by [5], refers to an RL setting where an agent receives
feedback on a subset of its experiences. The initial discussions of SSRL
focused on absolute feedback on subsets of the agent’s experiences, making the
concept complementary to PbRL. In contrast to PbRL and RLHF, the term SSRL
seems to be used less in the recent literature.

In our work, we adopt the viewpoint that RLHF is a broader category that
encompasses various approaches where human feedback is used to define the
objective of an RL agent. In this definition, RLHF encompasses both PbRL and
SSRL. As the definitions and distinctions between these terms are not
universally agreed upon, these distinctions are based on our interpretation of
the current predominant usage of these terms in the literature.

###  1.3 Scope of the Survey

This section outlines the criteria guiding our selection of approaches in the
realm of RLHF. We focus on works that rely on a reward model as the sole
source of information about the objective. This reward model should then be
learned in an interactive, online, scalable, and asynchronous manner. The
following will describe each of these criteria in more detail.

Reward Modeling

We focus on approaches that learn a reward model from human feedback and then
use this model to train a policy. Although it is possible to directly optimize
a policy from human feedback [270], thereby performing RLHF without reward
learning, this approach has been practiced only rarely so far. The
decomposition into reward learning and policy training offers many conceptual
and practical benefits. Among those benefits are the direct applicability of
supervised learning techniques for the reward model and the possibility of
evaluating the reward model in isolation. In addition to that, the
decomposition naturally leads to a form of semi-supervised learning, enabling
the agent to use labeled episodes for reward model training while leveraging
unlabelled episodes to refine its behavior and explore the environment. Note,
however, that there is a recent trend in the direction of direct policy
learning in the domain of language model fine-tuning (see Section 5.2.3).

Human Defined

While there are many approaches that include humans in the RL loop, in this
survey, we focus on approaches where human feedback is the only source of
truth about the objective. This excludes approaches to reward shaping, feature
engineering, and other forms of human guidance that are supplementary to a
given objective.

Interactive and Online

We also put an emphasis on providing feedback in an interactive, online
manner. This excludes imitation learning, learning from demonstration, and
pure inverse RL. While we do not directly cover inverse RL in this survey,
combinations of inverse RL methods with interactive improvements of the reward
function are in scope and employed by some of the surveyed methods. See
Sections 3.3 and 5.5.1 for a discussion of those approaches.

Scalable and Asynchronous

We focus on works in which the human is included in the loop, but the agent is
not blocked by the human’s feedback, and the human does not need to be present
continuously. This distinguishes RLHF from more direct methods of
incorporating a human into the RL loop, and we believe that this is key for
practicality and efficiency.

In addition to these criteria, we mainly focus on works published after 2017
since earlier works are surveyed by [270]. Nevertheless, some works from this
period are revisited from time to time in order to elaborate on certain
concepts that are still state of the art or have significantly shaped it.
Exceptions are made if the used methods are of interest for RLHF approaches.

###  1.4 Prior Surveys

Based on the criteria mentioned in the previous section, we will first
differentiate our survey from other surveys in marginally related subject
areas sharing the common theme of human-in-the-loop RL. Then, we will describe
the differences between our survey and previous surveys or survey-like
articles that exist within the RLHF field.

####  1.4.1 Human-in-the-Loop RL

Table 2: An overview of prior surveys of human-in-the-loop RL.  ✓ indicates that the criterion is a main focus of the survey,  (✓) indicates that the criterion is partially addressed, while ✗ indicates that the criterion is not covered.  Reference |  Topic |  Reward Modelling |  Human Defined |  Interactive and Online |  Scalable and Async.
---|---|---|---|---|---
[275] |  HITL machine learning. |  ✗ |  ✗ |  ✗ |  ✗
[190] |  RL with human advice. |  (✓) |  (✓) |  ✓ |  (✓)
[164] |  Social feedback. |  ✗ |  (✓) |  ✓ |  ✗
[207] |  Human brain signals. |  ✗ |  ✓ |  ✓ |  ✗
[8] |  Interactive RL for HCI. |  ✗ |  ✗ |  ✓ |  ✗
[199] |  Imitation learning. |  ✗ |  ✓ |  ✗ |  ✗
[7] |  Inverse RL. |  ✓ |  ✓ |  ✗ |  ✓
[27] |  Assisted RL. |  ✗ |  ✗ |  (✓) |  ✗
[174] |  Language-informed RL. |  ✗ |  ✗ |  ✗ |  ✗
[290] |  Human guidance. |  ✗ |  ✓ |  ✓ |  ✗
[126] |  AI Alignment. |  ✓ |  ✗ |  ✓ |  ✓
[170] |  RLHF for LLMs. |  ✗ |  ✗ |  ✗ |  ✗
Ours |  RLHF. |  ✓ |  ✓ |  ✓ |  ✓

Human participation in machine learning, particularly in guiding machine
learners, is a much-studied scenario. This field, commonly referred to as
human-in-the-loop (HITL) machine learning, can be further divided into
subfields based on various criteria, e.g., the ones detailed in Section 1.3.
Prior surveys of these subfields are compiled in Table 2 and briefly
summarized in the following.

Human-in-the-Loop

[275] survey HITL machine learning in general. They also cover some
applications of RLHF (for LLMs in particular) but do not give a detailed
overview. Similarly broad in scope, [190] study the setting of RL with human
advice, which they define as ‘teaching signals that can be communicated by the
teacher to the learning system without executing the task.’ While this setting
subsumes RLHF, the broad generality limits the depth in which their survey can
cover RLHF approaches.

Interactive RL

RLHF can be considered a sub-field of interactive RL, which studies RL
algorithms that learn in interaction with humans. This interaction can take
the form of feedback defining an objective, resulting in the RLHF setting, but
can also, e.g., be used to drive exploration or speed up the agent’s learning
process.

[8] survey interactive RL from an human-computer interaction (HCI) viewpoint,
exploring various ways humans can influence RL agents, with a particular focus
on reward definition based on human feedback, without a predefined
environmental reward function. Due to the breadth of their survey, they do not
cover many works in this area. The survey by [164] centers on interactive RL
using human social cues, like gestures and spoken language, but does not cover
the reward modeling aspect. Similarly, the study by [207] examines RL with
direct feedback from human brain signals, such as through brain-computer
interfaces, also not focusing on reward modeling.

Demonstrations

Learning from demonstrations, in the form of behavior cloning [199] and
inverse RL [7], shares the goal of RLHF to learn behavior from human input. In
contrast to RLHF, however, it requires demonstrations of the desired behavior
instead of feedback, and these demonstrations are usually not provided
interactively and online. This limits their applications and also their final
performance due to the availability of near-optimal demonstrations.
Nonetheless, imitation and demonstration can be a useful component of an RLHF
system but are not the main focus of this survey. However, we will discuss the
intersection between these fields in some parts whenever necessary.

Assisted RL

[27] review the field of assisted RL, where an agent may receive external
information (for example, from a human) that aids it in action selection.
While updates to the reward function are one of the possible effects of advice
in this setting (in addition to action selection or modifications of the
agent’s internal state), it is usually assumed that an initial reward function
is given and the extent of the updates is limited to reward shaping or
supplementary reward signals. In contrast to RLHF, the external information
does not define the task but only helps the agent in achieving it. Closely
related to this, [174] survey RL assisted by natural language. In addition to
this assistance setting, they also discuss approaches that infer a language-
conditioned reward function. However, they discuss this setting rather briefly
and use techniques from inverse RL and not RLHF.

Guidance

In their survey on human guidance, [290] delve into various aspects related to
RLHF. Although they touch on aspects such as reward learning, it is not the
primary emphasis of their work. Instead, their main focus lies on exploring
more immediate approaches that do not involve the learning of a reward model.

AI Alignment

[126] provide a general overview of AI alignment, i.e., the challenge of
aligning the objectives of an intelligent system with those of its human
operators. This survey covers RLHF in some detail. As AI alignment is a very
broad field, however, the article nevertheless does not go into as much depth
on the topic of RLHF as we do here.

Applications

[170] give an overview of current applications of RLHF methods for LLMs such
as ChatGPT and GPT-4. Even though it currently enjoys a lot of attention, it
is only a specific application area for RLHF. Our survey adopts a broader
perspective, examining the diverse applications and impact of RLHF
encompassing application areas beyond LLMs.

####  1.4.2 PbRL and RLHF

Table 3: An overview of prior RLHF-specific surveys and articles with substantial review components.  ✓ indicates that the aspect is addressed,  (✓) indicates that the aspect is partially addressed, while ✗ indicates that the aspect is not covered.  Reference (Focus) |  Beyond Comparisons |  Label Collection |  RM Training |  Theory |  App. and Benchmarks
---|---|---|---|---|---
[270] (preference-based RL). |  ✗ |  (✓) |  (✓) |  ✗ |  (✓)
[1] (recent advances of PbRL). |  ✗ |  ✗ |  (✓) |  (✓) |  (✓)
[125] (reward-rational implicit choice). |  ✓ |  ✗ |  (✓) |  ✗ |  ✗
[181] (feedback types). |  ✓ |  ✓ |  ✗ |  ✗ |  ✗
[45] (open issues in RLHF). |  ✓ |  ✗ |  ✓ |  ✗ |  (✓)
Ours (fundamentals, recent advances, and trends). |  ✓ |  ✓ |  ✓ |  ✓ |  ✓

There have been previous surveys or survey-like articles that are closely
related to RLHF. Table 3 gives a brief overview of how these articles differ
from ours, which we will explain in more detail below.

Preference-Based RL

Previous surveys in the domain of RLHF often focus on PbRL, where feedback is
limited to binary preferences (see Section 1.2). An illustrative example of
this is the survey by [270], which is a direct precursor to our work. In
contrast to our work, they concentrate on binary preferences for trajectories
and primarily survey methods that learn policies without deriving a reward
model. Since then, the reward-modeling approach has become dominant in the
field, and other approaches have extended RLHF to new feedback types. [1] give
another more recent literature review of PbRL. While this review focuses on
reward modeling and includes some recent work, it is far less comprehensive
than our review, as many aspects are only touched upon and partly overlap with
those of [270].

Feedback Types

Although not a survey per se, [125] propose reward-rational implicit choice as
a unifying framework to comprehend many previous studies in PbRL and RLHF. To
illustrate the generality of their approach, they overview different feedback
types used in previous work and explain how they fit into their framework.
Similarly, it is not strictly a survey, [181] propose a common framework for
studying user interaction and interface design for multiple feedback types. As
part of their work, they provide a classification of feedback types and a
brief overview of RLHF approaches. Nevertheless, many facets of RLHF are not
dealt with at all in those studies, as they are not primarily survey articles.
In contrast to [125, 181], our survey has a more extensive coverage going
beyond their study of feedback types and also discussing more recent work.

Open Problems

[45] provide a detailed overview of the open questions and limitations of RLHF
with a particular focus on aspects of security, governance, and transparency.
In their article, reward modeling is also covered, as is human feedback, which
goes beyond preference comparisons, but other aspects, such as theoretical
approaches or an overview of existing benchmarks, are not included. Thus, it
can be seen as a supplementary article that is ideal for further reading once
being familiarized with the topic through our survey.

All in all, our survey can be seen as the canonical continuation of [270],
which examines the evolution of the field of PbRL to the more modern and
general field of RLHF. This includes a thorough description of the basics as
well as an in-depth discussion of current advances and trends in the field.

###  1.5 Outline

In the next section, we begin with an introduction to the basics by revisiting
the most important concepts from the standard RL setting, which are also
naturally important in RLHF (Section 2). We then dive into the RLHF topic by
outlining the most studied scenario of reward model learning from pairwise
preferences. Using this introductory and illustrative example scenario, we
explain the basic framework of the RLHF alongside its three main components of
(human) feedback, label collection (feedback acquisition), and reward model
learning. These three main components will essentially form the structure of
our survey. In Section 3, we turn our attention to the human feedback
component and provide an overview of the different types of feedback as well
as their key attributes. The important concepts in terms of label collection
are then explained in Section 4, followed by learning the reward model in
Section 5. Section 6 is devoted to an overview of recent progress on the
theoretical side of RLHF, including approaches involving a theoretical
guarantee, as well as theoretical insights into the relationship between
standard RL and RLHF. Finally, Section 7 highlights some interesting practical
applications of RLHF and the existing benchmarks before Section 8 concludes
the survey by pointing out some possible avenues for future work.

##  2 Preliminaries

In this section, we recall the basic setting and the most important concepts
of RL and RLHF. In the course of this review, we will fix the notation that
will be used throughout the survey. We first introduce what is probably the
most studied RLHF scenario, i.e., learning a reward model from binary
trajectory comparisons. On the basis of this introductory and illustrative
example scenario, we explain the basic framework of RLHF with its main
components and briefly discuss the respective roles of these components in the
learning process. We will also briefly touch on active learning, which has a
strong connection to the feedback collection component.

##### Notations

For any integer $n\in\mathbb{N}$, we denote by $[n]$ the set
$\\{1,2,\ldots,n\\}$. For any set $S$, $\Delta(S)$ denotes the set of
probability distributions over $S$. We use $\mathbb{P}(E)$ for denoting the
probability of some event $E$, while $\mathbb{E}[X]$ is used to denote the
expected value of a random variable $X$. In some cases, we will write
$\mathbb{E}_{P}[\cdot]$ or similar variants to emphasize that the distribution
for the expected value is governed by the probability distribution
$P\in\Delta(S)$. Moreover, we will write $X\sim P$ if a random variable $X$ is
distributed according to a probability distribution $P$.

###  2.1 Reinforcement Learning

Reinforcement learning (RL) [248] is the setting of learning behavior from
rewarded interaction with an environment. Such a learning environment is
formalized as an Markov decision process (MDP), which is a model for
sequential decision-making. In an MDP, an agent iteratively observes its
current state, takes an action that causes the transition to a new state, and
finally receives a reward that depends on the action’s effectiveness.
Formally, an MDP is defined as a tuple
$({\mathcal{S}},{\mathcal{A}},{P},{R},d_{0},\gamma)$ where

  * •

${\mathcal{S}}$ is a set of states (the _state space_),

  * •

${\mathcal{A}}$ is a set of actions (the _action space_),

  * •

${P}:{\mathcal{S}}\times{\mathcal{A}}\to\Delta({\mathcal{S}})$ is a transition
function (the _transition dynamics_),

  * •

${R}:{\mathcal{S}}\times{\mathcal{A}}\to\mathbb{R}$ is a reward function,

  * •

$d_{0}\in\Delta({\mathcal{S}})$ is a distribution over initial states,

  * •

and $\gamma\in[0,1]$ is a discount factor.

The transition function ${P}$ defines the dynamics of the environment: For any
state $s$ and action $a$, the value ${P}(s,a)(s^{\prime})$, also sometimes
denoted ${P}(s^{\prime}\mid s,a)$, is the probability of reaching the state
$s^{\prime}$ after executing $a$ in $s$. In light of this, we will also refer
to the transition function sometimes simply as the _transition dynamics._ For
a given state and action, the transition is conditionally independent of all
previous states and actions, which is known as the _Markov property_ and the
reason for the naming as an MDP. The value ${R}(s,a)\in\mathbb{R}$ provides an
immediate evaluation after performing action $a$ in state $s$, which is also
called the (instantaneous) reward. It is also quite possible that the
instantaneous reward is $0$ for some states, and one only receives a reward in
specific states, for example, in so-called _terminal_ states for which the
transition function is zero. When both the state space ${\mathcal{S}}$ and the
action space ${\mathcal{A}}$ are finite, we call the MDP a _tabular_ MDP.

In an MDP, an _$H$ -step trajectory_ $\tau$ is a sequence of
$H\in\mathbb{N}\backslash\\{0\\}$ pairs of state-action ending in a terminal
state. Formally, it is given by $\tau=(s_{0},a_{0},s_{1},a_{1},\ldots,s_{H})$.
Given $t_{0}\geq 0$ and $H^{\prime}\leq H$, we can define a _segment_
$\sigma=(s_{t_{0}},a_{t_{0}},s_{t_{0}+1},a_{t_{0}+1},\dots,s_{H^{\prime}}$),
which refers to a continuous sequence of steps within a larger trajectory. A
trajectory $\tau$’s _return_ ${R}(\tau)$ is the accumulated (discounted)
rewards collected along this trajectory:

$\displaystyle{R}(\tau)=\sum_{h=0}^{H-1}\gamma^{h}{R}(s_{h},a_{h})\,\text{.}$
(1)

Note that we here use the same notation for the return and the reward
function, but both have different signatures (trajectory vs. state-action
pair). We can also define return $R(\sigma)$ of a segment $\sigma$ in a
similar manner. The return is well defined even if the horizon $H$ is infinite
as long as $\gamma<1$. If the MDP is a tabular MDP and any trajectory has
finite length, i.e., $H$ is necessarily finite, we call the MDP _finite,_ and
otherwise _infinite._

ptEnvironmentDynamicsReward FunctionAgentAction $a_{t}$State $s_{t+1}$Reward
$r_{t+1}$ Figure 1: The standard RL setting.

A _policy_ specifies how to select actions in a state, either
deterministically or stochastically. In the former case, a policy is simply a
mapping $\pi:{\mathcal{S}}\to{\mathcal{A}}$ from states to actions, while in
the latter, it is a mapping $\pi:{\mathcal{S}}\to\Delta({\mathcal{A}})$ from
states to probability distributions over actions. Since the deterministic case
is a special case of the stochastic one, we assume the latter case in the
following.

The basic interaction loop is depicted in Fig. 1: The agent chooses an action
$a_{t}\sim\pi(s_{t})$ based on its policy and the current state. As a
consequence, the environment transitions into the new state $s_{t+1}\sim
P(s_{t},a_{t})$, governed by the transition dynamics. The agent observes this
new state and the reward $r_{t+1}\sim R(s,a)$, after which this interaction
cycle is restarted.

In this setting, the RL agent aims at learning a policy that maximizes the
expected return

$J(\pi)=\mathbb{E}_{d_{0},{P},\pi}[{R}(\tau)]\,\text{,}$

where the expectation is with respect to policy $\pi$, transition function
${P}$, and initial distribution $d_{0}$. To solve this problem, two families
of RL approaches have been considered: _model-based_ RL and _model-free_ RL.
The methods in the first family learn a model (i.e., ${P},{R}$) of the
underlying MDP to help solve the RL problem, while the methods in the second
directly try to obtain a good policy without learning an MDP model. The second
family can be further decomposed into two main categories: _value-based_
methods and _policy search_ methods. In deep RL, both value functions and
policies are approximated with neural networks.

Value-based methods (e.g., DQN and its variants [186, 115]) aim at learning
the $Q$-function $Q^{*}$ of an optimal policy. The $Q$-function of a policy
$\pi$ is defined by:

$\displaystyle
Q_{\pi}(s,a)=R(s,a)+\gamma\mathbb{E}_{{P},\pi}\left[\sum_{h=0}^{H-1}\gamma^{h}{R}(s_{h},a_{h})\right]\,\text{,}$

where in the expectation, $s_{0}\sim d_{0}$, $a_{0}\sim\pi(\cdot\mid s_{0})$,
and $a_{h}\sim\pi(\cdot\mid s_{h})$ as well as $s_{h}\sim{P}(\cdot\mid
s_{h-1},a_{h-1})$ for $h\in[H]$. A policy can be naturally designed from a
$Q$-function by choosing an action in a greedy manner in each state:
$\pi(s)=\arg\max_{a}Q(s,a)$. Note that for a deterministic optimal policy
$\pi^{*}$ it holds that $J(\pi^{*})=\mathbb{E}_{d_{0}}[Q^{*}(s,\pi^{*}(s))]$.

Similar to the action-value function $Q$, we can also define the state-value
function

$\displaystyle
V_{\pi}(s)=\gamma\mathbb{E}_{{P},\pi}\left[\sum_{h=0}^{H-1}\gamma^{h}{R}(s_{h},a_{h})\,|\,s_{0}=s\right]\,\text{.}$

Its value for some state $s$ is the expected return when starting in that
state and then always using the policy $\pi$. It is related to the
$Q$-function by means of

$V_{\pi}(s)=\mathbb{E}_{a\sim\pi(s)}\left[Q_{\pi}(s,a)\right]$

for any state $s\in{\mathcal{S}}$.

In contrast, policy search methods directly aim at finding a good policy in
some parametrized policy space. The most data-efficient algorithms in this
class of methods follow an actor-critic scheme where both an actor (i.e., a
policy) and a critic (i.e., usually its $Q$-value function) are learned at the
same time. Typical representative methods here are PPO [228], TD3 [82], or SAC
[103].

RL algorithms can further be classified as either _on-policy_ or _off-policy_.
In an on-policy algorithm, such as PPO, only the recently generated
transitions are used for training. In contrast, in an off-policy algorithm,
such as DQN (or its variants), TD3, or SAC, the agent can be updated with
transitions not necessarily generated by its current policy. While on-policy
training is usually more stable, off-policy training enables more data-
efficient learning by reusing samples from a replay buffer that stores past
transitions.

###  2.2 Preference-Based MDPs

ptEnvironmentDynamicsReward ModelAgentAction $a_{t}$State $s_{t+1}$Reward
$\tilde{r}_{t+1}$LabelerptQuery $q_{i}$ptLabel $l_{i}$ Figure 2: RLHF with
reward modeling.

In contrast to standard RL as described in the previous section, RLHF does not
assume that a reward signal is available. It instead assumes the existence of
an _oracle_ (e.g., _human labeler_) that can provide information about the
reward in a specific indirect manner. More precisely, in the RLHF, the agent
can make queries $q_{i}$ to the oracle, which in practice means asking for
human feedback, and in response, the agent receives a label $l_{i}$, which in
general gives a hint about the reward. In principle, the query can be made
asynchronously to the actual conventional RL cycle. See Fig. 2 for an
illustration.

In the most common setting, the oracle can compare two (segments of)
trajectories, but various other cases have been considered, as we shall see
later on. For the former case, RLHF is based on the setting of preference-
based MDPs [94, 270], which can be defined as an MDP model without reward
function, but where comparisons of trajectories are available.

###  2.3 Reward Learning

RLHF approaches can be divided into two categories, depending on whether a
utility-based approach is used for reward modeling or an alternative criterion
that is detached from a utility concept is used [95, 93, 270]. Most works fall
into the first category, on which this overview focuses. Such approaches
assume a human-dependent utility function that can be used as a reward
function in order to apply standard RL methods. Next, we will describe the
commonly used approach for reward learning for the common setting of binary
trajectory comparisons.

The prevalent approach to learning a utility function from observations of
pairwise comparisons is based on the Bradley-Terry model [38], which
stipulates a probabilistic model for the oracle (human labeler):

$\displaystyle\mathbb{P}(\tau_{1}\succ\tau_{2})=\frac{1}{1+\exp({R}(\tau_{2})-{R}(\tau_{1}))}\,\text{,}$

where $\succ$ means “preferred to” and ${R}(\tau)$ corresponds to the utility
(i.e., return in the context of RL) of $\tau$. Note that this utility function
is a kind of surrogate function for the true reward function, which is
(tacitly) assumed to induce the same optimal policy as the true reward
function. For a given data set
$\mathcal{D}=\\{\tau^{i}_{1}\succ\tau^{i}_{2}\mid i\in[N]\\}$, a utility
function ${R}_{\psi}$ parameterized by $\psi$ can then be learned by the
maximum likelihood principle (or equivalently using a cross-entropy loss):

$\displaystyle\max_{\psi}\prod_{i=1}^{N}\frac{1}{1+\exp({R}_{\psi}(\tau^{i}_{2})-{R}_{\psi}(\tau^{i}_{1}))}\,\text{.}$
(2)

In the context of RL, since
${R}_{\psi}(\tau)=\sum_{h=0}^{H-1}\gamma^{h}R_{\psi}(s_{h},a_{h})$, (2) can
then directly be used to train a function approximator (e.g., single or
ensemble of neural network) to approximate $R$.

This entire modeling approach accommodates the case of a noisy or unreliable
oracle, in which case the Bradley-Terry model can be understood as the
generative model of the answers from the oracle (or labels provided by the
human labeler). When the oracle is reliable, more direct methods based on
preference elicitation to recover the reward function have been studied [217,
218, 264, 92, 223]. In this survey, we will focus on the general case where
the oracle may be noisy.

Note that in contrast to the typical way of preference learning, the learned
reward function is used to train an RL agent and not directly to compare
trajectories. This discrepancy in the objective function in the reward
learning part and how the learned rewards are used may lead to suboptimal
policies [168].

###  2.4 Reinforcement Learning from Human Feedback

In the RLHF setting as illustrated in Fig. 2, the learning agent needs to
solve an RL task without having access to a reward function. To this end, the
agent usually simultaneously learns an approximation of the reward function
(via the assumed utility function) and an RL policy. Therefore, a generic RLHF
algorithm consists of repeating two phases: (1) reward learning and (2) RL
training. The first phase can itself be decomposed into two main steps: (i)
generate queries to ask the oracle, (ii) train a reward function approximator
with the answers provided by the oracle. The RL training part is more
conventional and is usually directly based on running a deep RL algorithm
using the currently trained reward function approximator.

Algorithm 1 Generic RLHF Algorithm in an Actor-Critic Scheme

1: Initialize parameters $\theta$ (policy), $\phi$ (critic), and $\psi$
(reward)

2: Initialize replay buffer $\mathcal{B}$ with randomly-generated trajectories

3: for $i=1,\ldots,N$ do

4: // Reward learning

5: Generate queries from $\mathcal{B}$

6: Update $\mathcal{D}$ with answers to queries from the oracle

7: Update $\psi$ using $\mathcal{D}$ (e.g., to maximize Eq. 2)

8: // RL training

9: Update $\mathcal{B}$ with new trajectories generated with $\pi_{\theta}$

10: Update $\theta$ (actor) using $\mathcal{B}$ and ${R}_{\psi}$

11: Update $\phi$ (critic) using $\mathcal{B}$ and ${R}_{\psi}$

12: end for

This basic generic algorithm is summarized in Algorithm 1, where an off-policy
actor-critic scheme is assumed to be used for the RL training part, but other
RL policy learning approaches can, of course, also be used here. For an on-
policy algorithm, such as PPO [228], only the recently generated transitions
are used for training. For a DQN-like algorithm, lines 9 to 11 would be
replaced by a loop where transitions are generated by a behavior policy based
on the current estimate of the $Q$-function (e.g., $\varepsilon$-greedy
algorithm) and the $Q$ network is updated using mini-batches of transitions
sampled from the replay buffer $\mathcal{D}$.

An efficient RLHF algorithm needs to overcome several difficulties which are
specific to this setting:

  * •

The oracle may provide various types of feedback (see Section 3). The
questions of what information some given feedback provides and how observed
feedback can be exploited need to be answered (see Section 5).

  * •

Informative queries need to be generated to minimize the efforts of the
oracle, which is crucial when it is a human (see Section 4). Active learning
techniques (see next subsection) can be adapted to face this challenge.

  * •

The RL agent is actually trained in a non-stationary environment since the
reward approximator is concurrently updated. The RL training part needs,
therefore, to account for this factor (e.g., using non-vanishing learning
rates).

  * •

There is also the question of how the agent’s performance can be meaningfully
evaluated, especially if the reward function is not known (see Section 7).

  * •

Collecting feedback directly from humans introduces its own challenges, such
as the question of a suitable user interface and the associated issues of
delay between query and feedback observation, or the feedback variability and
reliability (see Section 4). This may explain why many studies evaluate novel
RLHF algorithms with simulated feedback.

A standard RL algorithm can be run in the RL training part, as done in most
previous work in RLHF (although this may not be the best approach). This
suggests that any improvements in a standard deep RL method (e.g., auxiliary
losses [89], planning in learned model [109], curriculum learning [192], or
data augmentation [151, 154, 166]) may potentially be transferred to the RLHF
setting. In addition, most previous work in RLHF directly uses trajectories
stored in replay buffer $\mathcal{D}$ to synthesize queries. An interesting
research direction to explore in RLHF would be to specifically generate
trajectories in order to be able to synthesize more informative queries
(instead of only generating trajectories that are beneficial for RL training).
This would lead to tackle a novel exploration-exploitation dilemma: Shall we
visit state-action pairs that may be bad but may help better learn the reward
function, or shall we visit state-action pairs that we currently think are
good? This is further discussed in Section 5.5.3.

In RLHF, since the oracle is a human or a group of humans, reducing the number
of queries is crucial to limit the labeling cost. Therefore, the reward
learning part requires techniques similar to those proposed in active
learning, which we recall next.

###  2.5 Active Learning

In active learning [230], the task is to strategically select data points for
labeling to minimize the amount of labeled data required to achieve a desired
level of performance, particularly valuable in scenarios like RLHF where
labeling is costly. Unlike batch learning, where labeled data is
predetermined, active learning empowers the learner to actively select the
most informative unlabeled instances for labeling, maximizing the learning
process with limited labeled data. We will only briefly introduce the active
learning task here and then discuss the strategies for creating informative
queries considered thus far in Section 4.

For RLHF with pairwise comparisons, this setting can be formally described as
follows. Suppose there is a set of $N$ pairs of trajectories
$\\{(\tau_{1}^{i},\tau_{2}^{i})\mid i=1,\ldots,N\\}$, where each pair
$(\tau_{1}^{i},\tau_{2}^{i})$ can be interpreted as an unlabeled instance. To
efficiently learn a reward function to explain observed pairwise comparisons,
an agent can select a set of unlabeled pairs (possibly a singleton) to query
an oracle to obtain their labels.

At a high level, the main idea in active learning is to query data points to
quickly reduce the epistemic (i.e., reducible) uncertainty about the
predictions of the learned model, although other aspects can be important,
such as the representativeness of the queried data points (see [261] for a
survey). Two main representations are considered to describe this epistemic
uncertainty: either using an ensemble of models or using a Bayesian
representation. In both cases, a first basic approach selects queries using
uncertainty-based criteria in order to focus on instances with high prediction
uncertainty as measured by, e.g., variance or entropy computed over
predictions. In contrast to the first approach, where the selection criteria
are instance-based, a second approach considers criteria that may depend on
all instances. Possible options are, for instance, expected model change,
expected error reduction, or density-weighted uncertainty-based criteria.
Here, the weights in the expectation or density allow us to take into account
the distributional information about the instances and, therefore, to focus on
the higher-density regions.

##  3 Feedback

Feedback mechanisms are fundamental to the success of any RL system. In the
standard setting as described in Section 2.1, the RL agents expect feedback in
the form of scalar immediate rewards. These rewards are most commonly
determined by a hand-engineered reward function, which can be used to evaluate
any state-action combination. As discussed in Section 1.1, it is desirable to
allow humans to refine behavior interactively through feedback instead of
requiring them to pre-specify a reward function.

While a human could, in principle, assign rewards to each of the agent’s
actions directly, thereby taking the role of the reward function, this is
usually impractical for multiple reasons. The main challenge is the human
effort required to provide rewards on a sufficiently regular basis, i.e., at
least once per episode. In addition to that, directly integrating human
rewards into the RL loop would require these rewards immediately, which
impedes the learning pace while waiting for human feedback. Finally, the
standard RL setting expects numeric state-action rewards, which is challenging
to provide in a consistent manner.

In contrast to directly rewarding each of the agent’s actions, RLHF as
discussed in this survey (see Section 1.3) harnesses indirect and asynchronous
feedback methods. Such methods avoid the challenges of immediate numeric
rewards and are also better aligned with human interaction patterns, resulting
in improved learning progress and human user experience.

A feedback type is a kind of interaction in which a human conveys some
information about their preferences. Examples include pairwise comparisons and
direct critiques. This section is concerned with the many ways in which human
feedback can be expressed and used. Several previous works have already
studied and sorted feedback types by listing the most common ones [125] and
discussing their attributes and dimensions [181, 167]. The attributes and
classes described in this section build upon this prior work and can be
considered a synthesis and extension of it.

The remainder of this section will start by discussing relevant attributes of
feedback types that can be used to classify them (Section 3.1). We will then
discuss common classes and examples of interactive feedback types (Section
3.2) as well as some non-interactive types that can serve as initializations
(Section 3.3).

###  3.1 Attributes of Feedback Types

Feedback types may differ on many dimensions, some of which relate to the way
feedback is given (arity, involvement), others to the form of the query
instance it is given on (granularity, abstraction), and yet others to features
of the human interaction (intent, explicitness). The attributes we discuss in
this section are based on the framework proposed by [181]. We have adjusted
and added terminology where it aids clarity, generalized the distinction
between relative and absolute feedback to ‘arity’, and added the categories of
co-generative involvement and literal intent. We have also systematically
analyzed a set of exemplary feedback types with respect to these dimensions
(see the next section), extending the initial work in that direction by [181]
who consider a smaller set of abstract classes. Furthermore, we systematically
analyze a set of exemplary feedback types in the next section, expanding on
the initial examination of a smaller range of abstract classes by [181]. In
the following, we will introduce each of the six attributes in more detail.

Arity

This attribute describes whether a single instance is evaluated in isolation
(_unary_) or relative to other instances (_binary_ , _$n$ -ary_). Unary
feedback is often convenient for detailed and descriptive feedback but lacks
any grounding and therefore puts a great burden on the human to provide
consistent feedback. Non-unary feedback always has an implicit grounding but
relates to the instances being comparable. While $n$-ary feedback, such as
rankings, can provide more information than binary feedback, it also puts a
higher cognitive burden on the labeler.

Involvement

The labeler may either passively _observe_ an instance, actively _generate_
it, or coactively participate in its generation (_co-generation_). Passive
involvement poses the smallest challenge to the labelers since it does not
require the ability to demonstrate the task. It can also easily be directed at
the most informative examples with active learning techniques. Unfortunately,
passive feedback often cannot match the information density of active
feedback. It is, therefore, common to combine both types to first initialize
the reward model from (possibly very suboptimal) active feedback and then
refine it from passive feedback. Between these two extremes is co-generative
feedback, in which a human can share control with the agent. This can be less
demanding than active feedback and makes it possible to direct the human’s
attention to the most informative samples, but it is still more taxing than
purely passive involvement.

Granularity

Feedback may also differ on the granularity of the instances being evaluated.
This ranges from whole _episode_ recordings over partial _segments_ to
feedback on individual _steps_ (i.e., states, actions, or state-action pairs).
A more coarse-grained granularity has the advantage of giving humans more
context and getting feedback for larger sections of behavior but also poses
credit assignment problems. Finer-grained feedback is much easier to learn
from and conveys more information, but is often impractical or tedious for
humans to provide. Note that we only classify a type of feedback as “episode”
granularity if it _requires_ entire episodes. If it is compatible with partial
segments as well, we classify it as “segment” even if the discussed source
paper uses entire episodes.

Abstraction

This describes whether feedback is given directly on raw _instances_ , e.g.,
behavior recordings (see granularity) or on abstract _features_ of the
instances. While feature-level information can be easier to learn from,
extracting useful features is challenging. In some contexts, it may also be
harder for a human to make abstract judgments rather than more intuitive
instance-level judgments. Note that this always refers to the level of
abstraction that the user sees, which may differ from the features used as
inputs for the reward model. Types of feedback that depend on active
generation (see involvement), such as improvements, generally work on a raw
instance level.

Intent

The assumed human intent can be important for feedback processing. A human may
be _evaluative_ , _instructive_ , or _descriptive_ in their explicit feedback,
while they are generally _literal_ in their implicit feedback. Evaluative,
instructive, and descriptive feedback is pedagogical in nature, aiming to
teach a reward function, whereas literal feedback is a byproduct of a human
actor’s efforts to optimize the reward function directly. While evaluative,
instructive, and literal feedback is generally given within the context of a
particular query instance, descriptive feedback can describe the task more
generally, e.g., through a partial reward function.

The distinction between literal and pedagogical feedback was introduced by
[184]. They argue that humans with pedagogical intent (i.e., evaluative,
instructive, or descriptive) may act differently compared to humans with
literal intent. Even though they find that assuming the (wrong) literal intent
can still lead to better reward inference, it still indicates that it can be
important to know this intent to choose the right human model (see Section
5.1.5).

Explicitness

Humans may communicate _explicitly_ for the purposes of feedback or
_implicitly_ as a side-effect of actions directed at other purposes. While
explicit information is easiest to learn from, implicit information is often
much more readily available and can possibly communicate more detailed
preferences.

###  3.2 Common Classes

Even though the concrete types of feedback used in the literature are rarely
exactly the same, they can generally be sorted into a set of common classes.
We will describe a selection of those classes, their defining attributes, and
examples of concrete instances in the literature in the following.

Table 4: An overview of the common classes and their defining attributes. When an attribute is not specified, this means that it is not a defining feature of the class and may vary in different instantiations.  Class | Granularity | Involvement | Arity | Abstraction | Intent | Explicitness
---|---|---|---|---|---|---
Critique | – | Observed | Unary | – | Evaluative | Explicit
Comparisons | – | Observed | 2+ | – | Evaluative | Explicit
Inter-Temporal | Segment | Observed | Unary | – | Evaluative | Explicit
Task Desc. | Episode | Observed | – | Feature | Descriptive | Explicit
Social Behavior | Segment | Observed | Unary | Instance | Literal | Implicit
Improvements | Episode | Co-generative | Unary | Instance | – | –
E-Stops | Episode | Observed | Unary | Instance | Literal | Implicit
Importance | – | Observed | – | – | Descriptive | Explicit
Feature Traces | Segment | Active | Unary | Instance | Descriptive | Explicit
Similarity Queries | – | Observed | Ternary | – | Descriptive | Explicit

Table 4 gives an overview of the classes and their attributes as described in
Section 3.1.

####  3.2.1 Critique

Critique is arguably the most direct type of feedback. In this setting, the
human expresses their preference by directly critiquing an instance of agent
behavior, often in the form of binary feedback. Note that critique, as
considered in this survey, is distinct from directly supplying a reward signal
since it is given in an asynchronous and indirect manner (see Section 1.3).
The defining features of critique are that the human passively observes the
behavior (involvement), gives feedback on a single instance (arity), and does
so explicitly (explicitness) with an evaluative intent. The feedback may be
given for any granularity and on any level of abstraction.

There are many examples of critique feedback in the literature. [277] employ
binary feedback on individual state and action pairs. Although they learn a
shaping reward that complements an environment reward signal, the same
technique could be used without environment reward. [118] extend this to
multi-label feedback, allowing the user to distinguish between a regular good
or bad action and a terminal action that achieves the goal or fails to do so.
They map these classes to scalar values and then learn a reward model by
regression. [263] present an approach to learning a reward model from noisy
critiques in the form of human physiological signals (brain signals) using
active querying. In contrast to this action-level feedback, [81, 237] rely on
binary outcome success labels. [81] introduce the basic approach, which [237]
extend by moving to an off-policy setting and including online queries,
thereby reducing the reliance on many positive examples by interactively
correcting false positives.

In addition to learning the main reward function, critique can also be used
for safety evaluation. [60] train a secondary reward model focused on safety
from binary action critiques. This is in addition to the main reward model,
which is trained from comparisons in their approach. Note that this secondary
safety model could, in principle, be trained with any of the feedback types
discussed here, using methods identical to the ones used for reward learning.

####  3.2.2 Comparisons

Binary comparisons and rankings are among the most common types of feedback.
The defining features of comparisons are that the human passively observes the
behavior (involvement), gives relative feedback on multiple instances (arity),
and does so explicitly (explicitness) with an evaluative intent. It is most
commonly given on a segment (granularity), but other granularities are
possible in principle [60]. Similarly, comparisons are commonly requested on
an instance level (abstraction), but this is not a requirement.

Comparisons were first used for direct policy learning [3, 53], but were later
extended to the reward-learning setting [273, 58]. The most common setting
[58] relies on pairwise comparisons of trajectory segments, but comparisons of
individual states or actions were also considered in early PbRL works [83].

This basic setting has been extended and modified in various ways. To reduce
noise in the labels, it is common to extend the binary choice by giving the
labelers the option to avoid the hard choice and instead indicate
incomparability, uncertainty, or perceived similarity [117]. The most common
interpretation of this intermediate option is “equally preferable” [169],
e.g., as if each trajectory had an equal probability of being preferred in the
preference predictor. [119] instead provide two intermediate options: The
previously discussed “equally preferable” and an “incomparable” option which
results in simply omitting the query from the data set. In contrast to this,
[257] explicitly state that they do not allow for the equally preferred
option, arguing that these cases are rare enough not to matter very much.
Another line of research suggests that more precision in the expression of
pairwise preferences, such as softening the hard binary choice to scalar
feedback indicating the strength of a preference [266], can be beneficial for
preference learning. Other extensions change the pairwise setting to choices
among larger choice sets [296] or even full rankings [188, 39, 200]. [296]
note that for language tasks, a larger choice set can amortize the time needed
for a labeler to get acquainted with the context necessary to understand a
query. [18] propose to use hierarchical queries, i.e., a sequence of pairwise
comparisons that build up on each other.

Pairwise comparisons provide many benefits over absolute ratings, such as
reduced bias, inconsistencies, and subjectivity [283], but also convey
relatively little information per label. [250] study the weaknesses of
pairwise-comparison-based reward learning. Since the amount of information
provided for each label is small, these models are prone to causal confusion,
i.e., misattributing reward to noise and misidentifying the reward function.

####  3.2.3 Inter-Temporal Feedback

One limitation of trajectory comparisons is that they require a set of roughly
comparable trajectories. In many real-world environments, starting conditions
or even the agent’s current task may vary between episodes. In these cases, it
is hard for human labelers to compare trajectories from these episodes,
limiting the usefulness of comparison feedback. One way to remedy this
limitation is to provide feedback within a single trajectory. Instead of
comparing a set of instances with each other as in regular comparative
feedback, inter-temporal feedback conveys relative judgments over different
states in time within a single instance. The defining features of inter-
temporal feedback are that it is given explicitly (explicitness) on segment
(granularity) while passively observing (involvement) a single instance
(arity) of the agent’s behavior with evaluative intent. It is most commonly
given on raw instances, but any level of abstraction is possible in principle.
There are two main ways to convey this feedback: _Reward sketching_ and
_inter-temporal preferences_.

Reward sketching, as introduced by [42], involves users sketching a visual
representation of the reward function over time. This type of feedback, which
can be given by sketching a graph with the mouse while watching a behavior
recording, provides intuitive, per-timestep reward annotations. [214] also
adopted this approach, referring to it as “one of the highest-bandwidth
feedback mechanisms currently available”.

Inter-temporal preferences were introduced by [2]. In this setting, humans
give feedback on multiple points of a trajectory, indicating whether an agent
makes progress towards or regresses from a goal. This is then interpreted as
preferences relative to the other labeled and unlabelled points. The authors
note that one potential downside of this feedback type is that labelers may
tend to give preferences on short-term actions that are easy to judge, failing
to communicate long-horizon preferences. [61] propose a similar type of
feedback, in which humans segment a trajectory into good and bad parts. This
makes it possible to derive many state-action labels from few segmentation
points.

####  3.2.4 Task Descriptions

Task descriptions consist of direct, concept-level descriptions of the task
the agent is supposed to complete. This is most commonly conveyed in the form
of information about the reward function, i.e., by proxy rewards or reward
queries. Descriptive feedback does not generally refer to any particular
behavior instance but instead gives global direction for the entire task.
However, in line with our selection criteria (Section 1.3), we only consider
task descriptions that are interactive and online within the context of one or
multiple observations to fill holes in the initial description. The defining
features of task descriptions are that the labeler passively observes the
agent’s behavior (involvement) and gives feedback explicitly (explicitness) on
a feature-level (abstraction) with descriptive intent (intent). Descriptive
feedback may be given with respect to a single or multiple instances (arity),
although it generally refers to multiple instances. It is most commonly given
on an episode (granularity), but other granularities are possible in
principle.

Task descriptions are most commonly provided in the form of incomplete reward
functions. [112] demonstrate this using proxy reward functions, which are
preliminary reward functions that might not cover all edge cases, to guide the
agent toward learning the actual reward function. Alternatively, [185, 108]
suggest querying about the reward function. They allow users to choose from a
set of understandable, linear proxy rewards or to specify which features are
more critical in the linear reward structure.

####  3.2.5 Social Behavior

Humans give rich implicit social feedback in the form of facial reactions and
gestures when interacting with agents. The defining attributes of this type of
feedback are that it is given implicitly (explicitness) on passively observed
(involvement) segments (granularity) with respect to a single instance (arity)
and literal intent.

[62] propose a framework to learn reward functions from such social behavior.
They suggest a two-phase training setup. In the first phase, they ground the
implicit feedback by use of incentives, i.e., they incentivize humans to have
a known objective. After learning a mapping from feedback to reward, they use
regular RL techniques to learn a policy. Note that the learned reward function
can be seen as conditional on human implicit feedback and, therefore, they
require a human in the loop throughout training.

####  3.2.6 Improvements

Improvements are a form of feedback in which the human improves on the agent’s
behavior, either by intervening as the agent acts or by providing a corrected
behavior after the agent acts. To improve an episode, it is usually necessary
to observe the entire episode (granularity) at the instance level
(abstraction). In this type of feedback, the human both observes and
demonstrates behavior, resulting in co-generative involvement. Improvements
generally relate to a single reference trajectory being improved (unary
arity), although an improvement could also be interpreted as a binary
comparison between the improved and the non-improved trajectory. Improvements
are most commonly provided explicitly with instructive intent.

We distinguish between post-facto improvements, calling them _corrections_ ,
and improvements made while the agent is acting, calling them _interventions_.
The key difference is that the uncorrected trajectory is available in the case
of corrections, while it can only be estimated in the case of interventions.

Interventions can be considered to be an instance of the shared autonomy
setting since the agent and the user share autonomy to reach a common goal.
This setting is studied by [2], who ask humans to intercede on agent failure.
They use this to collect targeted demonstrations where the agent is the
weakest and to identify challenging situations for their evaluations. The
gathered data is then used for behavior cloning and reward model training. The
fact that a correction occurred is not directly used as feedback. In contrast
to this [171] observe that the interventions themselves are intentional and
therefore carry information about the true objective. [172] propose to
incorporate uncertainty for active learning and risk-sensitive deployment in
this setting. [160] further extend the intervention setting to learn from a
sequence of correlated physical corrections without needing to wait until the
trajectory is completed.

The correction case is closely related to the setting of coactive learning
[234], in which a learning system repeatedly proposes a solution which a user
may correct to reach a common goal. [122] treat corrections as a demonstration
while [125] propose (but do not evaluate) an alternative interpretation of
inferring implicit preferences from comparisons by assuming the corrected
trajectory is preferred over the original one.

####  3.2.7 E-Stops

Emergency stops (e-stops) [91] are an active type of feedback. In this type of
feedback, the human may intervene with the agent’s behavior by stopping it,
i.e., they may choose to stop the agent’s current trajectory at any point.
This is closely related to interventions but, in contrast to those, does not
suggest an alternative action. The defining features of e-stops are that the
human both passively observes the agent’s behavior (involvement), gives
absolute feedback on a single instance (arity) on the instance level
(abstraction) and does so implicitly (explicitness) as a side-effect of
regular interaction. The intent is literal due to the implicit nature. For the
purposes of intervention, the human usually observes the full episode
(granularity). Due to the small amount of infrequent information they provide,
e-stops should only be considered as a supplementary feedback type.

E-stops are primarily intended to prevent bad behavior and only implicitly
convey information about the correct behavior. This interaction and the
arising incentives have been formalized in the form of the “off-switch game”
by [107]. [125] propose to interpret this as a form of reward-rational
feedback, where the ‘off’ choice maps to the trajectory with the robot
remaining still after the off switch has been triggered. [131] demonstrate
that a robot can learn to navigate using such feedback.

####  3.2.8 Importance

Another form of supplementary feedback may come in the form of importance
labels, communicating which parts of the observation are important for the
objective. Its defining features are that the importance information itself
does not contribute towards generating behavior samples (observed
involvement), is of descriptive intent, and is given explicitly
(explicitness). Granularity, arity, and abstraction may vary depending on the
primary feedback type. Since importance feedback needs a base task with
respect to which the importance is defined, it cannot be used on its own but
is rather a supplementary type of feedback.

One way to convey this information is by labeling salient parts of a visual
input. This is explored by [101], who augment pairwise comparisons with
manually annotated visual saliency maps, informing the algorithm which parts
of the visual input contributed to the decision. They leverage these
annotations for data augmentation by assuming that random perturbations to
irrelevant (non-salient) regions do not impact the human preferences. [19]
take an even more direct approach by combining comparative feedback with
direct feature queries, i.e., asking the user which feature is important for
inferring the reward.

####  3.2.9 Feature Traces

While many approaches either rely on hard-coded features or learn a model
entirely end-to-end, it is also possible to actively elicit new features from
human feedback. Feature traces were proposed by [36] as an approach to
actively learn new relevant features. This type of feedback relies on a human
operator to demonstrate a behavior in which a certain feature of interest,
such as the distance to a sensitive object, monotonically increases or
decreases. They make it possible to extend the set of features once the
current set can no longer adequately explain the human feedback supplied
through another type of feedback. The defining characteristics of feature
traces are that they are of descriptive (intent) and explicitly (explicitness)
given in an active manner (involvement) for a single (arity) segment
(granularity) on an instance-level abstraction.

Feature traces are strongly related to inter-temporal preferences (Section
3.2.3) since both types rely on changes in feature- or reward values in the
course of a single trajectory. [36] propose to learn from feature traces by
leveraging a Bradley-Terry model to learn the feature values, similar to other
approaches that use such a model to learn reward values. Similar to importance
feedback, feature traces rely on another type of feedback to actually make use
of the learned features and is, therefore, a purely supplementary form of
feedback. For instance, [36] use intervention feedback to train a reward model
on the set of features derived using feature traces.

####  3.2.10 Similarity Queries

Similarity queries are a feedback type aimed at learning a representation
conforming to a notion of similarity and difference in the trajectory space.
That aim is closely aligned with that of feature queries, though the actual
queries are more similar to comparisons. The queries consist of triples of
trajectories, with one anchor and two alternatives, for which the human has to
decide which pair is more similar. Responses to similarity queries are given
on observed behavior (involvement) with ternary arity, descriptive intent, and
explicit feedback, while the granularity and abstraction may vary. This type
of feedback was first introduced by [34], who used it to learn representations
for reward learning. Similar to feature traces and importance, this is a
supplementary type of feedback.

###  3.3 Initializations

Some modes of communicating reward functions are not interactive nor online
and, therefore, do not directly fit within the scope of this survey (Section
1.3). Since these are often used to initialize a reward function for later
refinement with some of the previously discussed interactive feedback types,
they are still worth mentioning.

Initializations are most commonly given by examples of successful task
completions, either in the form of terminal or goal states [278], expert
demonstrations [80, 2], labeled demonstrations [70], or ranked demonstrations
[40]. Since this is not the main focus of our survey, we refer to the
literature on inverse RL [7] for further details.

##  4 Label Collection

In this section, we explain how preference data can be collected for training
a reward model. We first overview how queries can be generated for a given
query type, then how the query type itself can be selected, and finally
discuss issues arising in such human-computer interaction.

###  4.1 Active Learning

Following the terminology used in Bayesian active learning, we call the
criterion to measure the quality of a query _acquisition function_ , which is
used to select which queries are presented to a human labeler. We first review
the various acquisition functions used in RLHF for query selection and then
discuss the extension to the choice of feedback type.

####  4.1.1 Acquisition Function

One core problem that needs to be tackled in RLHF is that of learning about
the human’s preferences. This problem shares some similarities with the active
learning setting since the agent can actively query a human teacher about
those preferences. However, in contrast to standard active learning, which
usually assumes a supervised learning setting, in RLHF, the agent needs to
solve this problem in the context of RL. This means that it can both influence
the distribution of the data (i.e., the transitions) and decide which data
should be labeled.

As the RL agent is trained and its learned policy changes, the trajectories it
generates will naturally evolve. Most work directly uses the trajectories
obtained during RL training for preferences learning. Alternatively, the agent
can also generate trajectories specifically to be used for querying (not
necessarily for RL training), possibly with a learned transition model to
control the sampling cost (e.g., [216]). This kind of active generation of
queries can possibly lead to more informative ones.

In order to efficiently learn a suitable reward model, the agent must generate
and select queries (Line 5 from Algorithm 1) so that it can quickly learn a
good strategy using those queries. This selection is performed via a
criterion, usually called _acquisition function_ , which allows the queries to
be compared. An efficient acquisition function may need to include various
factors, such as: uncertainty, on-policy data, query simplicity, trajectory
quality, query diversity, and query cost, which will be discussed one by one
in the following. As a side remark, as noted by [104], interestingly, the
queries asked by an RL agent also reveal its current reward learning stage.

Uncertainty

This corresponds to how uncertain the agent is about the ground-truth reward
function. We are particularly interested in epistemic uncertainty in this
setting, i.e., uncertainty that can be reduced by additional queries. This is
usually one of the most important aspects to consider when deciding which
query to ask. This uncertainty is usually either represented as a probability
distribution (i.e., belief) in Bayesian approaches or using an ensemble of
reward networks to approximate this belief.

With a belief representation, various classic Bayesian acquisition functions
have been considered, such as the _probability of improvement_ or the
_expected improvement_ (e.g., [64]). However, they require maintaining a
distribution over reward functions (e.g., using Gaussian processes [64]) and,
therefore, may not be suitable for more complex domains due to the
computational complexity. Thus, simpler alternative criteria have been
considered, such as _information gain_ [185, 28] or _variance_. Although
computationally heavy, _volume removal_ (i.e., the minimum volume of the
hypothesis set removed by an answer to a query) has also been studied [223,
19, 18].

When using an ensemble instead of a direct belief representation, these
criteria for epistemic uncertainty reduction correspond to measures of
disagreement within the ensemble. Previous criteria could possibly be applied,
but one popular candidate is the _variance_ of the ensemble outputs [157, 180,
99].

Recently, the _average entropy_ of the ensemble outputs (assuming a Bradley-
Terry model for the human answers) has also been used [157, 156, 203].
However, note that it does not quantify epistemic uncertainty but rather the
uncertainty in the human’s answers, as provided by the response model of the
human. Therefore, this criterion may not be suitable in the RLHF setting since
it amounts to focusing on the queries for which an answer is expected to be
the most random (according to the Bradley-Terry model). By definition of this
model, the segments in the pairwise comparisons are the most similar in terms
of returns and are, therefore, the hardest to answer for the human.

In addition to epistemic uncertainty, one may also take utilities (i.e.,
returns or expected returns in RL) into account to select queries. In a
Bayesian setting, this leads to acquisition functions such as _expected value
of information_ [189] or _information gain over return differences_ [168],
while in a non-Bayesian setting, the notion of _regret_ , which measures the
difference of performance between a policy optimal for the ground-truth reward
function and a policy optimal for a learned reward function, can be used
[268].

On-Policy Data

Only focusing on uncertainty is likely insufficient or inefficient in RL. In
particular, it may be important to favor more on-policy trajectories to
guarantee the relevance of the generated queries. Indeed, improving reward
learning in state-action regions that may never be visited with the current
policy would lead to wasteful queries [168]. One simple approach to ensure
that the data is more on-policy is by favoring more recently-generated
trajectories [74].

Query Simplicity

Selecting queries only based on their informativeness may lead to queries that
are hard to answer for a human, which is, for example, the case for the
average entropy. The ease of answering a query is important to alleviate the
cognitive load of the human oracle. Some work specifically takes this aspect
into account, for instance, by considering the similarity of consecutive
queries [212] or the information gain. For this latter criterion, [30] show
that in contrast to volume removal, it naturally leads to queries that are
easier to answer for a human because information gain can be increased when
the uncertainty in the human answer is lower.

Trajectory Quality

Most approaches directly use the trajectories generated during RL training.
Especially early in training, these can be very bad with respect to the
ground-truth reward function. In addition to that, they can be irrelevant or
even contradictory for a given task [135]. Building queries on such
trajectories may lead to unnatural queries for a human to respond to, such as
comparing a very bad trajectory with an irrelevant one. [135] measure
trajectory quality by optimizing over sampled reward functions.

Query Diversity

When asking many queries (in sequence or in batch), the diversity of the
queries becomes especially crucial to avoid asking redundant queries. Most
work follows a very myopic approach: Queries are often selected from a
randomly generated set of potential queries, and sequences of queries are not
really coordinated. Few work [31, 267] specifically tackles the former point,
but the latter point is rarely considered, which may be due to its
computational intractability. Indeed, planning ahead a sequence of queries
would amount to solving a sequential decision-making problem under uncertainty
over a combinatorial action space (i.e., the set of possible queries).

Query Cost

The cost of generating queries may also be an important factor if the
interaction of the human is live since it may not be practical to let the
human wait before showing any queries [31]. In that case, it may be more
important to quickly show some relatively good queries instead of computing
the most informative ones.

Although this factor may not translate directly into an acquisition function,
it may influence the choice of the acquisition function and its implementation
in a given problem.

Since various different acquisition functions have been considered, some
effort [157, 156] has been made to compare them. Generally speaking,
uncertainty-based criteria (e.g., variance or average entropy) seem to often
perform better empirically compared to random selection, a query diversity-
based criterion alone or combined with an uncertainty-based criterion. In
addition, [74] empirically observed that a variance-based criterion performs
better than a selection method only based on trajectory recency. Surprisingly,
random selection has been shown to perform competitively in some cases [58,
119]. Thus, a better understanding of which acquisition function should be
preferred in which situation or domain is still an open question.

In addition, combinations of different criteria have naturally also been
evaluated. For instance, [216] use four acquisition functions (high
uncertainty, high novelty, high reward, low reward) in parallel. This approach
has also been validated in a 3D environment [214]. A more sophisticated
approach consists of considering a portfolio of acquisition functions and
learning to select them using a multi-armed bandit approach [116].

Various extensions to the basic setting have also been investigated. In the
context of multiple human labelers, the issue of selecting reliable teachers
to query arises [65]. Assuming all teachers have the same preferences, this
can be modeled by incorporating a rationality coefficient $\beta$ into a
Bradley-Terry model and estimating this factor:

$\displaystyle\max_{\theta}\prod_{i=1}^{N}\frac{1}{1+\exp(\beta({R}_{\psi}(\tau^{i}_{2})-{R}_{\psi}(\tau^{i}_{1})))}\,\text{,}$
(3)

where a higher $\beta$ corresponds to a more reliable human (see Section
5.1.2). The setting in which this assumption does not hold, i.e., the
labelers’ reward functions differ (a setting already considered in inverse RL
[54]), has also been studied recently [281, 69, 188, 15]. Interestingly, a
noisy oracle may provide more information than a completely reliable oracle
because the frequency of erroneous answers given by the former is related to
how much a segment is preferred to the other one [280, 48]. In contrast, only
a binary preorder over segments can be inferred from the answers of a reliable
and deterministic oracle, which may not be enough to recover the true reward
function.

####  4.1.2 Adaptive Choice of Feedback Type

In addition to selecting queries within a given feedback type, it is also
possible to actively select the feedback type itself [75]. The best choice of
feedback type can depend on many factors, such as human rationality as well as
task-dependent factors, some of which may change during the labeling process.
[91] formalize this setting as one in which we try to select a feedback design
(or feedback type) $x$ out of the space of possible designs $\mathcal{X}$ such
that the expected information gain over the distribution of reward functions
is maximized for the next human response. Concretely, the goal is to choose a
feedback design by means of

$x=\operatornamewithlimits{argmax}_{x\in\mathcal{X}}\mathbb{E}_{c_{h}\sim
P(c_{h}\mid x)}\big{[}D_{\mathit{KL}}\big{(}\mathbb{P}(\theta\mid
c_{h},x)\mid\mathbb{P}(\theta)\big{)}\big{]}\,\text{,}$

where $c_{h}$ is the human response to a query defined by $x$ and
$\mathbb{P}(\theta)$ is the prior distribution over reward functions.

[91] find that the most informative feedback type depends on the (type-
dependent) rationality of the human labeler (see Section 4.2.1). More
precisely, it is shown that the most informative feedback depends on the
rationality factor, e.g., while demonstrations are more informative than
comparisons when the human is highly rational, comparisons should be preferred
in less-rational settings. Given that this rationality might change due to
factors such as fatigue or an individual labeler’s capabilities, this suggests
that adaptively adjusting the feedback type during the labeling process may be
worthwhile. Further study of this relationship is a promising area for future
work.

###  4.2 Challenges of Human Labeling

This section explores the label collection process, which follows after query
selection. This task intersects with several related disciplines, especially
within the social sciences, as it encompasses the design of interactions to
facilitate informative query responses. A prominent field in this area is
psychometrics [84], which focuses on measuring psychological attributes,
including preferences. Similarly, survey research [76] is dedicated to
developing techniques for gathering information from individuals via surveys.
Human-computer interaction plays a significant role as well, investigating the
design of user interfaces tailored for preference elicitation [206]. Moreover,
preference label collection is also necessary for discrete choice experiments
within health economics [222], where it is used for the assessment of service
values.

####  4.2.1 Psychology-Aware Preference Elicitation

Understanding human psychology is essential for effective preference
elicitation in RLHF systems. Human decision-making is complex, often diverging
from traditional rational choice models due to cognitive, social, and
emotional factors. This complexity is exemplified by phenomena like fatigue,
which can affect the reliability of choices based on the order of queries.
This section investigates these phenomena, exploring how constructive
preferences, biases, framing effects, and social interactions shape the
observed choices. Recognizing and addressing these psychological underpinnings
is key to developing more accurate and reliable systems. In this section, we
will discuss various psychological phenomena, such as cognitive biases and
response biases, and related effects (fallacies, biases, heuristics,
psychological phenomena impacting decision-making processes), which may
falsify labels by adding systematic bias or noise.

Preference learning methods typically assume the existence of inherent, stable
preferences that can be elicited through querying. Contrary to this
assumption, psychological research, such as the work by [163], indicates that
preferences are often constructed during the elicitation process and may vary
with the method of elicitation or over time. This suggests that the feedback
type not only affects elicitation’s effectiveness but also shapes preferences.
Systematic biases, noise, and other psychological factors may influence
observed choices, challenging the traditional models of human choice used to
infer latent utilities (see Section 5.1). The elicitation method, query
presentation, and context thus play a critical role in shaping measured
preferences, compounded by cognitive biases and irrationalities.

The influence of psychological phenomena on preference learning has been well-
documented in the literature, especially within the context of explicit
preference elicitation for recommender systems. For instance, [252] provide a
thorough discussion of the relationship between psychology and recommender
systems. Similarly, [10] review how preference construction is influenced by
cognitive biases, personality traits, and emotional states in recommender
systems, discussing effects like serial position, framing, anchoring, choice
overload, and preference visibility. In a more specialized discussion, [176]
focus on cognitive biases in the context of consumer decision-making and its
interaction with recommender systems. [176] specifically address cognitive
biases in consumer decision-making in interaction with recommender systems.
Finally, [136] link these psychological aspects to RLHF, discussing the common
practice of using synthetic instead of real human feedback for algorithm
evaluation and highlighting the limitations of that approach. They further
discuss challenges posed by real human feedback, many of which are related to
the concepts discussed in the following paragraphs, as well as the
opportunities provided by integrating psychological insights into RLHF
systems.

Constructive preferences are closely related to _framing effects_ , which
refer to changes in elicited preferences based on how tasks or alternatives
are described, even when these descriptions are essentially equivalent. For
example, presenting a choice as a loss versus a gain can lead to different
decisions despite identical outcomes. Moreover, _serial position effects_ ,
commonly known as primacy and recency effects, also play a significant role.
These effects describe the tendency for the beginning and end of an experience
to influence subjective experience disproportionately. This phenomenon becomes
particularly relevant in scenarios like video choices, where the initial or
concluding segments might disproportionately affect preferences. [10] discuss
both of these effects in the context of recommender systems.

_Ordering effects_ pose another challenge in preference elicitation, where the
sequence of queries can affect responses. [66] outline several factors
contributing to these effects: institutional learning, changing preferences,
and varying levels of cognitive effort. Institutional learning involves
gaining familiarity with the task and feedback type, which can enhance
labelers’ expertise and, consequently, the accuracy of their responses.
However, due to the constructive nature of preferences, their preferences may
evolve during the elicitation process, leading to changing preferences. This
evolution might also be influenced by _anchoring effects_ , where previously
seen instances bias responses. Furthermore, cognitive effort levels can
fluctuate due to factors like fatigue or boredom. This is closely related to
_choice overload_ , a form of fatigue from excessive choices, as discussed by
[10] and bounded rationality, as explored by [51]. In such scenarios, labelers
might opt out of making a choice when overwhelmed by options. Bounded
rationality refers to the limitations in human decision-making capabilities,
particularly when processing large amounts of information, which aligns with
the concept of choice overload. To address these challenges, studies like [30]
and [288] propose methods to reduce cognitive effort in responding to queries.
[30] focus on posing queries that are straightforward for humans to answer,
while [288] enhance the human evaluation process by presenting queries in a
user-friendly format.

When multiple labelers collaborate on the same task in preference elicitation,
as is studied, e.g., by [16] and [65], this may lead to another set of biases
if they have the opportunity to exchange information. This exchange may either
be direct or indirect through observing the system’s predictions, which are
based on the other labeler’s feedback. Such interactions can affect their
preferences through several mechanisms, as identified by [10]: anchoring
effects, transfer of emotional states, and conflict avoidance. _Anchoring
effects_ , for instance, occur when a labeler’s choices are influenced by the
knowledge of others’ preferences or system predictions, a phenomenon also
discussed under the term _preference visibility_. This bias can lead labelers
to align their preferences with the anchors they are exposed to, which is a
significant consideration in recommender systems. Understanding these biases
is crucial for designing RLHF systems that mitigate the influence of labeler
interactions on preference construction.

The effects previously discussed stem from systemic biases in preference
expression. In addition to these biases, choices may also be affected by
noise. This is commonly discussed under the term stochastic rationality, where
an agent’s behavior is rational with respect to an unobserved random state.
The reward-rational implicit choice framework, as introduced by [125],
addresses this by integrating a rationality factor $\beta$ into the human
choice model (see Eq. 3). This factor’s impact has been further examined by
[91] through synthetic experiments and user studies, demonstrating that
accurately estimating this type-dependent rationality coefficient can enhance
learning performance and guide feedback type selection (see Section 4.1.2).
However, a practical method for estimating this factor remains a challenge.
While [91] use calibration feedback with a known latent utility function for
estimation, such an approach is not feasible for most tasks. In a related
study, [65] investigate a scenario with multiple teachers, focusing on the
agent’s ability to select the most knowledgeable or rational teacher.
Therefore, developing more advanced methods to estimate this factor, along
with understanding its variability due to factors like fatigue or other
ordering effects, presents a vital area for future research in preference
elicitation

Incorporating psychological insights into the preference-learning components
of RLHF systems is essential for optimizing their efficacy. A key area of
focus should be research aimed at mitigating biases and harnessing cognitive
aspects of preference formation. For instance, designing user interfaces that
minimize framing effects and developing algorithms that account for ordering
and serial positioning are crucial steps. In this realm, [181] propose a
configurable user interface, called RLHF-Blender, for studying various
feedback modalities and their combinations. Additionally, the study by [148]
on the impact of feedback type, such as binary critiques versus action advice,
on task performance and labeler satisfaction highlights the significant role
of feedback type in preference elicitation. Furthermore, the work of [206] in
user-interaction design underlines the importance of having an expressive
feedback type to increase user engagement.

The integration of these research findings into RLHF systems points to a clear
need for a more multidisciplinary approach. Drawing insights from related
fields like behavioral economics and psychology can provide valuable
methodologies and perspectives. Addressing irrational choice patterns and
enhancing the quality of human feedback remain critical challenges. As we
continue to develop and refine these systems, the focus should be on creating
robust frameworks that align learning processes with human behavior,
effectively managing the inherent complexity and variability of human
feedback.

####  4.2.2 Importance of Researcher-Labeler Agreement

High-quality labels are important for the final policy in an RLHF process.
Early work on fine-tuning language models using RLHF noticed a mismatch
between the researcher’s goals and the (paid) labeler’s actual labels
(researcher-labeler disagreement). [296] note that researchers agreed with
each other about $60\%$ of the time (on 4-way comparisons, where random choice
would result in $25\%$ agreement), while agreeing with labelers only $38\%$ or
$46\%$ of the time (depending on the task). [244] attempt to reduce these
disagreements by maintaining a hands-on relationship with the labelers and
thereby ensuring high researcher-labeler agreement. Concretely, they provide
on-boarding with detailed instructions, keep an open channel of communication
between researchers and labelers, and give feedback to the labelers. They
evaluate the researcher-labeler agreement and reach an agreement rate of
$77\%\pm 2\%$.

Due to the inherently subjective nature of the task (researchers agreed with
each other in $73\%\pm 4\%$ of the cases), a much higher agreement rate cannot
be expected. [200] also report the agreement rates on a different task
(instruction fine-tuning instead of summarization) and find that labelers
agree with each other in $72.6\pm 1.5\%$ of the time, after a screening
procedure that, amongst others, selects labelers that agree with researcher
labels.

The importance of quality does not trump the importance of quantity, however.
Indeed, [244] note that excluding low-confidence samples from the data set
generally did not help with reward model training. This indicates that even
though quality is important, a larger quantity is still generally better.

The scale of the labeled data set required for effective training and
refinement varies widely, impacting the quality of the resulting models.
Studies have shown a broad range in data set sizes, from tens of labels in
smaller studies [122] to hundreds in more complex scenarios [58]. Larger-scale
applications may require thousands [101] or even millions of labels [2], each
bringing its own challenges in ensuring label accuracy and consistency. This
variability in data set size underscores the need for rigorous label quality
control measures across different scales. In smaller data sets, each label
carries more weight, making accuracy and precision critical. Conversely, in
larger data sets, the challenge lies in maintaining consistency and mitigating
systematic biases that might emerge from the sheer volume of data.

Similarly, the labeling setting varies in the surveyed works, from author-
provided feedback [138], over small in-person studies [135], to larger remote
studies [138]. Each setting provides unique challenges to ensure high-quality
labels.

Various works have suggested measures to improve label quality. [111] discuss
the possible failure modes of the labeling task in more detail, for example,
discussing systematic biases and conflicting motivation, and propose concrete
changes to the training and evaluation methodology to alleviate these. [96]
suggest providing labelers with multiple natural language rules and collecting
preference labels for each rule individually to improve label quality. This is
related to [13], who propose to generate feedback automatically based on such
a set of rules and a language model.

##  5 Reward Model Training

In this section, we delve deeper into the aspect of reward model learning,
which we briefly touched on in Section 2.3. We will discuss various aspects
associated with the topic, such as the different reward model inputs or model
architectures considered hitherto or the various possibilities regarding the
training process.

###  5.1 Human Feedback Model

The basic premise underlying the majority of approaches in RLHF is that human
feedback is directly related to the reward function to be learned. To this
end, the human feedback must first be captured in a sound mathematical
framework that establishes the connection to the reward function. On a high
level, one can break down (almost) all feedback types in Section 3.2 to a
choice scenario: The human chooses one specific feedback option (label) from
an available (possibly infinite) pool of possible feedback options (choice
sets)111This point of view goes back to the work of [125].. Here, the query
that is made specifies the explicit contents of the choice set, e.g., if the
query is to compare two trajectories, then the choice set consists of all
possible outcomes for these two trajectories.

Assuming that human choices are not always optimal, one obtains a fruitful
mathematical framework when focusing on the probability

$\displaystyle\mathbb{P}\left(\mbox{$c$ is
chosen}\,|\,\mathcal{C}\right)\,\text{,}$ (4)

where $\mathcal{C}$ is the set of possible choices and $c\in\mathcal{C}$ one
explicit choice. For the RLHF scenario, where the agent asks queries $q_{i}$
and the human gives labels $l_{i}$ as feedback (see Section 2.4), the choice
set is specified by a function of the query. Formally, $\mathcal{C}=m(q)$ for
some mapping $m$ that maps a query $q$ to the set of all possible candidate
labels extractable from $q$ for the specific feedback type. For example, if
the query is to rank a finite number of trajectories, then the choice set is
the set of all possible rankings that can occur for the trajectories involved.

With this view, we can therefore place (4) in the RLHF context and write

$\displaystyle\mathbb{P}\left(\mbox{label $l$ is provided}\,|\,m(q)\right)$
(5)

for the probability that a human labeler returns a label $l$ from all possible
candidate labels that can be extracted from a given query $q$ One could also
recover the noiseless scenario if the latter probability distribution is
degenerated for all possible candidate label sets.

####  5.1.1 Boltzmann Distribution

Human choice models as in (4) have been studied for a long time in various
scientific fields such as psychology [249], economics [251] or behavioral
science [47]. Accordingly, there are many different choice models to resort to
for (5), which, in some cases, are the same models, just under different
names. A popular class of such human choice models assumes every choice option
$c$ to be equipped with a (latent) utility $u_{c}$, which the human perceives
in a perturbed way. This perturbation is modeled by means of perturbation
random variables $\epsilon_{c}$ that perturb the utility in an additive way,
so that (4) becomes

$\displaystyle\mathbb{P}\left(\mbox{$c$ is
chosen}\,|\,\mathcal{C}\right)=\mathbb{P}\left(c=\operatornamewithlimits{argmax}_{c\in\mathcal{C}}\
u_{c}+\epsilon_{c}\right)\,\text{.}$ (6)

The translation for the RLHF setting for (5) is then accordingly

$\displaystyle\mathbb{P}\left(\mbox{label $l$ is
provided}\,|\,m(q)\right)=\mathbb{P}\left(l=\operatornamewithlimits{argmax}_{l\in
m(q)}\ u_{l}+\epsilon_{l}\right)\,\text{,}$ (7)

and we shall now stick to the RLHF translation from now on. These
probabilities depend on the specific distributional assumptions that are made
on the perturbation variables that only for specific cases lead to a closed-
form of the right-hand sides of the latter displays. When stipulating a
standard Gumbel distribution for the perturbations, one always obtains a
closed form that is proportional to the exponential utility of the provided
label:

$\displaystyle\mathbb{P}\left(\mbox{label $l$ is
provided}\,|\,m(q)\right)\propto\exp(u_{l})\,\text{.}$ (8)

This is known as the _Boltzmann distribution_ that also appears in a perhaps
slightly modified version in various different subfields of machine learning
and statistics. When restricting to discrete (choice) sets for $m(q)$, this
distribution is also known as the multinomial logit model [251] or Gibbs
measure [90], and as the Bradley-Terry model [38] when the choice sets consist
of pairs. All of these also have a link to the Plackett-Luce model [173, 205],
which is a probability distribution on the space of total orders or rankings
(see [4] for details).

This model is often used for various reasons. A particularly compelling reason
is the closed analytic form, which in turn makes it possible to obtain a
closed form for the gradient with respect to the utilities. Another reason is
that this model satisfies Luce’s axiom of choice [205], which requires the
probability of choosing an option from a pool of choice options not being
affected by the presence or absence of other options in the pool. In this way,
coherent decision-making is ensured, which, however, might be challenged as
humans are likely not making fully rational decisions (see Section 4.2.1).
[125] show that the usage of the Boltzmann distribution is justified by the
principle of maximum entropy. More precisely, they show that it is the maximum
entropy distribution over choices for a so-called satisficing human decision
maker, i.e., one who is making in expectation a choice with an optimal reward
up to some slack $\epsilon>0$.

To build the bridge between reward learning and the modeling of human
feedback, the Boltzmann distribution can be used by assuming that the
utilities can be represented as a function of the reward function, usually
featuring the return of a trajectory. More specifically, one assumes a
_grounding function_ $\psi$ that maps choice options (or labels) to the set of
distributions over trajectories and sets the utility of a label $l$ as

$\displaystyle u_{l}:=\mathbb{E}_{\tau\sim\psi(l)}[R(\tau)]\,\text{.}$ (9)

Table 1 in [125] provides an overview of the different grounding functions
that lead to a specific feedback type. It is worth noting that one can also
easily find a grounding function for the feedback type of a (partial) order
over trajectories as considered, for instance, by [188]. Moreover, one can
generalize this modeling approach by using (partial) segments instead of
trajectories.

Although this general human feedback model has been much in use and shown to
be useful for the sake of human alignment, it is not without its critics (see
[167] or Section 3.2.1 in [45]). This has led to different adaptions of the
general model based on the Boltzmann distribution that will be discussed in
the following. Moreover, we will also concisely review other human feedback
models that have been in use besides the Boltzmann distribution, discuss
relevant work on the consequences or robustness of human feedback model
misspecification, and highlight contributions on varying the standard
assumptions on the nature of the human feedback.

####  5.1.2 Human-Specific Rationality Coefficient

The Boltzmann distribution in (8) can be extended by a rationality coefficient
$\beta\in[0,\infty)$ that reflects the precision of the human labeler222This
can be achieved by multiplying the utilities $u_{l}$ with $\beta$.:

$\displaystyle\begin{split}&\mathbb{P}\left(\mbox{label $l$ is
provided}\,|\,m(q)\right)=\mathbb{P}\left(l=\operatornamewithlimits{argmax}_{l\in
m(q)}\ \beta\,u_{l}+\epsilon_{l}\right)\propto\exp(\beta\cdot
u_{l})\,\text{.}\end{split}$ (10)

The higher $\beta$, the more (10) resembles a pointmass distribution modeling
a highly rational human labeler (decision-maker) that is always able to
identify the option with highest utility, while the lower $\beta$, the more
(10) resembles a uniform distribution modeling a highly irrational human
labeler (decision-maker) acting purely at random. Without this extension, the
commonly considered Boltzmann distribution (or Bradley-Terry model in the
common case of pairwise comparisons) in (8) assumes a rationality coefficient
of $1$. [91] show in their experiments that the estimation of this coefficient
can indeed positively influence reward learning. For the estimation, however,
a calibration reward function is needed, as the rationality coefficient is
otherwise not identifiable [22]. Similar findings are shown by [65], who model
the rationality coefficient as a query-dependent function that might differ
for the human labelers (see Section 5.1.6).

Another alternative to the rationality coefficient for representing irrational
humans is achieved by introducing a query-independent error probability [58].
To be more precise, it is assumed that the human labeler only adheres to the
Boltzmann distribution in (8) in 90% of cases and otherwise makes completely
random decisions. This formulation is similar to Huber’s contaminated model
[187].

####  5.1.3 Alternative Utility Notions

[143] show that the Boltzmann model does not generally lead to an identifiable
reward function using (9) by presenting three concrete scenarios for which
identification is not possible. The root cause of the non-identifiability is
the usage of a trajectory’s return as the utility in (9). They, therefore,
suggest using a trajectory’s regret as an alternative, which provably leads to
identifiable rewards.

A trajectory’s regret is the negated sum of the optimal policy’s advantage
over each state-action pair in the trajectory. Empirically, it has been shown
that this modification improves the alignment of the learned strategy with
human preferences. The downside of this alternative is that regret depends on
the unknown optimal policy. Recently, it has also been suggested to consider
$Q$-values of a human policy as the utilities [189], while [117] used
differences of cost functions that depend on the available choice set and the
human’s uncertainty.

####  5.1.4 Human Feedback Models Beyond Boltzmann

While the human feedback model based on the Boltzmann distribution is the most
widely used model nowadays, other models have also been considered in the
literature. In particular, for the probability in (4) other models such as the
Thurstone model [269, 149, 28], the ridge-noise model [227], the binary model
[245] or mixed forms thereof [273] have been considered. Of these models, only
the Thurstone model [249] has a similar interpretation as the Boltzmann
distribution based on perturbed utilities, only differing in the distribution
of the perturbance random variables.

##### Link functions

Another possibility, which is particularly popular in theoretical work on RLHF
(see Section 6), is the use of other functions on the right-hand sides of Eq.
8 than the exponential function. The concept is primarily used for pairwise
comparisons of trajectories. It essentially states that the probability of the
result of a pairwise comparison between two trajectories is the difference of
their utility values under a so-called _link function_. More specifically, let
$q=\\{\tau_{1},\tau_{2}\\}$ be the query to compare the trajectories
$\tau_{1}$ and $\tau_{2}$, then, assuming a link function
$\Phi:\mathbb{R}\to[0,1]$, one models the probability in (5) for $l$
representing a preference for $\tau_{1}$ as

$\displaystyle\mathbb{P}\left(\mbox{label $l$ is provided}\,|\,m(q)\right)$
$\displaystyle=\mathbb{P}\left(\tau_{1}\succ\tau_{2}\,|\,m(\\{\tau_{1},\tau_{2}\\})\right)=\Phi(u_{\tau_{1}}-u_{\tau_{2}})\,\text{.}$
(11)

For $l$ representing a preference for $\tau_{2}$, one proceeds similarly. The
minimal assumptions on the link functions are that

  * (i)

it is (strictly) monotonically increasing to take into account that
trajectories with higher utilities will also have a higher chance to be
picked;

  * (ii)

$\Phi(x)=1-\Phi(-x)$ to ensure that
$\mathbb{P}\left(\tau_{1}\succ\tau_{2}\,|\,m(\\{\tau_{1},\tau_{2}\\})\right)=1-\mathbb{P}\left(\tau_{1}\prec\tau_{2}\,|\,m(\\{\tau_{1},\tau_{2}\\})\right)$.

Note that the second property implies $\Phi(x)=1/2$ so that trajectories with
the same utility also have the same chance of being selected. Any cumulative
distribution function of a symmetric continuous random variable fulfills these
two conditions. The two most common link functions that both fulfill the
conditions are the linear link function given by

$\Phi(x)=\max\\{0,\min\\{1,1/2\cdot(1+x)\\}\\}$

and the logistic link function given by

$\Phi(x)=\frac{1}{1+\exp(-x)}\,\text{.}$

Both are cumulative distribution functions: The linear link function is the
cumulative distribution function of a continuous uniform distribution on
$[0,1]$. In contrast, the logistic link function is the cumulative
distribution function of a logistic distribution with location parameter 0 and
scale parameter 1. Moreover, both are intensively studied in theoretical
approaches (see Section 6.1), and the latter leads to (8) (when restricted on
pairwise comparisons) and is a special case of the softmax function.

##### Two-Staged Choice Model

[35] propose the Limiting Errors due to Similar Selections (LESS) model that
is inspired by the attribute rule model suggested by [102]. It assumes a
feature map for trajectories and a (similarity) function mapping trajectory
features and trajectories to integers and uses a two-stage process for
modeling the human feedback (or choice): First, choosing a trajectory feature
according to the Boltzmann distribution and then a trajectory with the
(logarithmic) similarity functions as the utilities within the Boltzmann
distribution. Their experiments show that this model can capture human
feedback more appropriately than the standard Boltzmann distribution.

##### Generative Model

[2] evaluate the usage of a generative model for learning from human
preferences. More specifically, instead of assuming some underlying utility as
in the Bradley-Terry model, they attempt to train a model to generate the
human feedback (inter-temporal preferences in this case, see Section 3.2.3)
and directly interpret this feedback as reward. However, they found that this
empirically does not perform as well as the inter-temporal Bradley-Terry
model.

####  5.1.5 Misspecification

The human feedback model may be misspecified in various ways. [184]
investigate the problem of misspecifying the nature of human feedback that can
be either literal or pedagogical. The former means that the human gives
targeted feedback for solving the actual RL problem, while the latter means
that the human gives targeted feedback that is deemed helpful for the learner.
They show theoretically and empirically that the case of a learner assuming a
pedagogical with an actual literal human always performs worse than the
reversed case, i.e., a learner assuming a literal with an actual pedagogical
human.

A related question is studied by [77], namely, what if the learner makes
incorrect assumptions about the choices from which the human selects its
feedback? They consider different types of such choice set misspecification
and show that depending on the type of misspecification, the performances
might vary drastically, even leading to no losses at all in specific cases.

In the field of inverse RL, the general question of the robustness of reward
learning in terms of a misspecified human feedback model is theoretically
investigated by [239]. It turns out that the optimality model is not robust
with respect to any misspecification, the Boltzmann model is robust for quite
a range of specific misspecifications, and the degree of robustness of the
maximal causal entropy model lies between the latter two. Even though these
results are primarily derived for inverse RL, they also have similar immediate
implications for RLHF.

####  5.1.6 Diverse Preferences

One potential issue with the RLHF framework is that it does not specify whose
preferences to align to. In practice, it is common to aggregate preference
feedback from multiple users and learn a single reward model to try to predict
the mean preference. In contrast, [15] investigate this question more
explicitly and proposes to learn multiple reward functions, which can then be
aggregated in arbitrary manners and even be utilized to find consensus among
people with different preferences. [188] consider the case of multiple humans
that provide the feedback and use a mixture model of Plackett-Luce models to
represent the feedback more accurately. With a stronger focus on the active
retrieval of human feedback, [78] model the problem of selecting a suitable
human labeler as a variant of the multi-armed bandit problem [152] that they
call hidden-utility bandit. In this variant, the agent has in each decision
round the choice between two things: (i) drawing a bandit arm and receiving a
hidden arm-dependent utility and observing an item, or (ii) asking a human for
a preference for two pairs of items but incurring a human-specific query cost.
The feedback mechanism of an individual human is modeled via a Boltzmann
distribution with a known individual rationality coefficient. The same
modeling of human feedback is also considered by [16], who, however, use a
Bayesian approach to determine which person should be queried in order to
obtain the most informative feedback in expectation. [65] model the
rationality coefficient as a query-dependent function that might differ for
the human labelers.

####  5.1.7 Relaxation of the Markov Assumption

Most works assume that the human feedback is given based on a latent Markovian
reward model, i.e., the return of a trajectory $\tau$ decomposes into a sum of
independent rewards over state-action pairs (see (1)). [73] relax this
assumption by dropping the need for the Markov property, such that the
instantaneous reward might depend on hidden states. Similarly, [138] avoid the
Markov assumption by utilizing a transformer as the preference model. A
similar effect may be achieved by learning a state representation with a
recurrent network in which the rewards are Markov, similar to the approach
taken by [110], but we are not aware of any work exploring this. [2] work in a
non-Markovian setting as well by utilizing memory-augmented networks for both
the policy and the reward model.

###  5.2 Utility Learning

After choosing a human model to relate feedback to utilities, we can use the
observed feedback to recover the latent utilities. This utility learning can
be reduced to a standard supervised learning problem and, therefore, is
commonly solved with the techniques of empirical risk minimization or Bayesian
approaches, both of which will be discussed in the following.

####  5.2.1 Risk Minimization

The most prevalent variant for learning the reward function, already been
presented in Section 2.3, is a special case of empirical risk minimization.
The general approach of empirical risk minimization for reward function
learning, assuming an underlying human feedback model with utilities
$u_{l_{i}}$ as in (9), is to find the minimizer of

$\displaystyle\mathcal{L}(R;\mathcal{D})=\sum_{i=1}^{N}\ell\left(u_{l_{i}}(R),m(q_{i})\right)\,\text{,}$
(12)

where $\mathcal{D}={\\{(l_{i},q_{i})\\}}_{i=1}^{N}$ is the given data set of
observed label and query pairs and $\ell:\mathbb{R}\times\mathcal{Q}$ is a
suitable loss function, where $\mathcal{Q}$ denotes the set of all possible
label sets. To provide an example, let us consider the common case of pairwise
trajectory comparisons. Here the grounding function $\psi$ is the identity,
the queries are pairs of trajectories $q_{i}=\\{\tau_{1}^{i},\tau_{2}^{i}\\}$
and the labels
$l_{i}\in\\{\tau_{1}^{i}\succ\tau_{2}^{i},\tau_{1}^{i}\prec\tau_{2}^{i}\\}=m(q_{i})$
are the human’s preference over the two trajectories. For a given query
$q_{i}$, we then obtain (2) as a special case of (12) by using the loss
function:

$\displaystyle\ell(u_{l_{i}}(R),m(q_{i}))$
$\displaystyle=-\log\left(\frac{1}{1+\exp(u_{l_{i}}(R)-u_{m(q_{i})\setminus
l_{i}}(R))}\right)$
$\displaystyle=-\log\left(\frac{1}{1+\exp(\mathbb{E}_{\tau\sim\psi(l_{i})}[R(\tau)]-\mathbb{E}_{\tau\sim\psi(m(q_{i})\setminus
l_{i})}[R(\tau)])}\right)\,\text{.}$

This is the negative log-likelihood for the Boltzmann distribution for the
observational pair $(l_{i},m(q_{i}))$.

For the entire learning process, a model class $\mathcal{R}$ is assumed for
the reward function $R$. This model class is usually a parameterized class of
functions, such as, for example, the class of linear reward functions [135]

$\mathcal{R}=\\{R_{\bm{\theta}}(s,a)=\bm{\theta}^{\top}\bm{\phi}(s,a)\,|\,(s,a)\in\mathcal{S}\times\mathcal{A},\bm{\theta}\in\mathbb{R}^{d}\\},$

where $\bm{\phi}:\mathcal{S}\times\mathcal{A}\to\mathbb{R}^{d}$ is some state-
action feature mapping. This entails that good features are known in advance
such that rewards can be expressed as a linear combination of those features.
Using such linear models may lead to reward model misspecification. Studying
this setting, [33] propose to adapt the hyperparameter $\beta$ in Eq. 3 to
account for this issue.

Since the assumption of a linear reward model may be too strong in practice,
most recent work is based on non-linear models, especially using
differentiable models, but other cases have been investigated as well. In the
former case, for instance, decision trees have been considered to learn an
interpretable reward model (see Section 5.2.5). In the latter case, simple
multilayer perceptron (MLP) has naturally been considered, but more recent
deep learning architectures are more commonly used in the recent literature.
Thus, especially with partially observable domains, the reward network may be
composed of a state-action encoder followed by fully connected layers. For
instance, [2] combine ResNet blocks for image processing, a learnable
embedding table, a multi-modal transformer, LSTMs, and MLPs. Besides, [138]
utilize a Transformer-based architecture [256], motivated by the observation
that rewards are often non-Markovian.

In addition to the usual empirical risk in (12), it is also typical, as in
supervised machine learning, to add a regularization function to prevent
overfitting:

$\displaystyle\mathcal{L}(R_{\bm{\theta}};\mathcal{D})=\sum_{i=1}^{N}\ell(u_{c_{i}}(R_{\bm{\theta}}),\mathcal{C}_{i})+\lambda_{r}(\bm{\theta})\,\text{,}$
(13)

where $\lambda_{r}:\Theta\to\mathbb{R}_{+}$ is a regularization function
defined on the parameter space $\Theta$ of the underlying reward model class.
For instance, [58] simply use L2 regularization and also considers dropout in
some domains.

The main supervised loss to train the reward model can also be augmented with
additional losses corresponding to auxiliary tasks to avoid overfitting and
improve generalizability. For instance, [2] use a behavior cloning loss and
add a policy head to the reward network, thereby preventing that the reward
model drifts too far from its initialization from the policy.

####  5.2.2 Bayesian Approach

As is generally the case in supervised machine learning, there is also the
variant of using Bayesian modeling for learning a target object instead of the
(empirical) minimization of a loss function. To this end, one starts with a
prior distribution $\rho$ over the parameter space of the reward function that
is updated in light of the data set $\mathcal{D}$ by means of Bayes theorem:

$\displaystyle\mathbb{P}(\mathbf{\theta}\,|\,\mathcal{D})\propto
L_{\mathbf{\theta}}(\mathcal{D})\cdot\rho(\mathbf{\theta})\,\text{,}$

where
$L_{\mathbf{\theta}}(\mathcal{D})=\prod_{i=1}^{N}\mathbb{P}\left(\mbox{$l_{i}$
is provided}\,|\,m(q_{i}),{\mathbf{\theta}}\right)$ is the likelihood of the
data under the assumed human feedback model with reward function
$R_{\mathbf{\theta}}$. Such an approach is used for pairwise trajectory
comparisons, for instance, by [227] for the noisy-ridge model or by [223] for
the Boltzmann distribution as the human feedback model. In inverse RL, such
Bayesian approaches have been considered as well (see Section 4.3 in [7]).

Instead of assuming that the reward functions are parameterized, one can use
the reward functions directly as a parameter class and use a prior
distribution over them. This could, for example, be a Gaussian process as
initially considered by [149] for pairwise trajectory comparisons and adapted
in later works [28, 60]. Here, again, it is worth mentioning that such
considerations have also been made in inverse RL before (see Section 4.3 in
[7]).

####  5.2.3 Direct Methods

Note that the two-step approach of utility learning and policy optimization is
not the only viable path to learning a policy from human feedback. While the
previous two subsections have discussed the case in which we learn a reward
function from observed preferences by assuming a human feedback model, an
emerging branch of the literature is concerned with circumventing the reward-
learning step and using preferences directly to address the actual RL problem.
Concrete approaches are DPO [213], SLiC-HF [292], OPPO [134], or DPPO [6].
[11] introduce an objective called $\Psi$-preference optimization ($\Psi$PO)
that unifies the objective functions in DPO and RLHF. More specifically, for a
specific instantiation of $\Psi$, the objective in $\Psi$PO recovers the
latter two.

It is worth noting that approaches for directly learning the policy from
preferences have been considered in the past as well [269, 83, 271, 272, 41].
In Sections 3.2.1 and 3.2.2 in the survey by [270], these methods are
explained in more detail.

####  5.2.4 Partial Identifiability

A crucial question when it comes to learning the reward function is whether
the reward function can be identified at all. If two reward functions induce
exactly the same human feedback model, the reward function is called partially
identifiable or ambiguous. [241] study this topic for the Boltzmann
distribution as the underlying human feedback model when demonstrations
(inverse RL) or pairwise trajectory preferences are given as feedback. For
demonstrations, this question has also been examined in other works [195, 71,
139, 43].

####  5.2.5 Interpretability

The field of explainable artificial intelligence (XAI) has emerged in recent
years to improve the transparency and explainability of models or even to
enable them in the first place. Roughly speaking, the aim is to resort to more
interpretable methods or provide explanations for both experts and non-
experts, shedding light on why a certain input in a (black box) model leads to
a certain result. Explanations can take different forms, as can the ways to
ensure the transparency of models, and for a detailed overview, we refer to
[17]. It is worth noting that the field has grown so extensively over the
years that even dedicated overviews for the field of interpretable and
explainable RL are by now available [209, 97, 211, 183].

For the branch of RLHF, the existing works are quite sparse and mostly limited
to using tree models as transparent and explainable models for learning the
reward function [26, 25, 132, 24, 133]. Another way to realize explainability
within RLHF suggested by [289] is to learn simultaneously the reward function
and the importance of states using a weight network. Assuming that for (long)
trajectories, only a few states are important for the preference outcome,
their framework can be used to select samples for explainability purposes.
Moreover, a perturbation analysis is suggested to evaluate explanations in a
quantitative manner using the learned state importance weights.

####  5.2.6 Online Improvements

[58] demonstrate that it is important to improve the reward model online,
which is also confirmed by [86] as otherwise, the issue of overoptimization of
an imperfect reward model may occur. [2] give an example of this: They attempt
to fine-tune an agent initialized with behavioral cloning with an engineered
reward function and find that it fails to generalize and actually worsens the
performance. They also compare RLHF with a reward model trained offline with
iterative improvement and find that this leads to better performance, even