**footnotetext: These authors contributed equally to this work.

# Benchmarking Multi-Agent Preference-based
Reinforcement Learning for Human-AI Teaming

Siddhant Bhambri∗
Arizona State University
Tempe, AZ
USA
<EMAIL_ADDRESS>
&Mudit Verma∗
Arizona State University
Tempe, AZ
USA
<EMAIL_ADDRESS>
Anil Murthy
Arizona State University
Tempe, AZ
USA
<EMAIL_ADDRESS>
&Subbarao Kambhampati
Arizona State University
Tempe, AZ
USA
<EMAIL_ADDRESS>

###### Abstract

Preference-based Reinforcement Learning (PbRL) is an active area of research,
and has made significant strides in single-agent actor and in observer human-
in-the-loop scenarios. However, its application within the co-operative multi-
agent RL frameworks, where humans actively participate and express preferences
for agent behavior, remains largely uncharted. We consider a two-agent (Human-
AI) cooperative setup where both the agents are rewarded according to human’s
reward function for the team. However, the agent does not have access to it,
and instead, utilizes preference-based queries to elicit its objectives and
human’s preferences for the robot in the human-robot team. We introduce the
notion of Human-Flexibility, i.e. whether the human partner is amenable to
multiple team strategies, with a special case being Specified Orchestration
where the human has a single team policy in mind (most constrained case). We
propose a suite of domains to study PbRL for Human-AI cooperative setup which
explicitly require forced cooperation. Adapting state-of-the-art single-agent
PbRL algorithms to our two-agent setting, we conduct a comprehensive
benchmarking study across our domain suite. Our findings highlight the
challenges associated with high degree of Human-Flexibility and the limited
access to the human’s envisioned policy in PbRL for Human-AI cooperation.
Notably, we observe that PbRL algorithms exhibit effective performance
exclusively in the case of Specified Orchestration which can be seen as an
upper bound PbRL performance for future research.

_K_ eywords Multi-Agent $\cdot$ Human-AI Teaming $\cdot$ Preference-based
Reinforcement Learning

##  1 Introduction

Modeling co-operation for Human-AI Teaming scenarios has been a long-standing
research challenge. With the increase in the pivotal roles AI agents play in
day-to-day human activities [1, 2, 3], modeling agents with teaming
capabilities from the human-in-the-loop’s perspective has become all the more
relevant. This perspective can include what the human-in-the-loop’s
expectations are on the AI agent’s policy, what preferences they have on the
AI agent’s behavior, or even what their assumptions are on their own policy.
Typically, co-operative multi-agent teaming requires all the acting agents to
understand each other’s strategies (via explicit communication or implicit
deduction from past behavior) and come up with compatible behavior that boosts
team performance, and if possible, achieve their personal goals. In this work,
we consider the case of a two-agent (Human-AI) cooperative team. While multi-
agent cooperation (and even two-agent cooperation) is hard [4], the problem is
even more challenging when one of the agents is a human [5, 6, 7]. In
particular, it becomes crucial for the AI robot to take into account the
cognitive limitations of humans, the preferences humans hold concerning the
robot’s goals and strategies, and to align its policies with those of humans
to ensure compatibility with policies that humans deem achievable.

Preference-based Reinforcement Learning has been used in several complex
domains like robotic manipulation and locomotion to incorporate human
preferences on AI agent’s behavior [8]. The initial conceptualization of PbRL
focuses on single-agent dynamics with a human in the loop observer available
to provide feedback on agent queries [9]. In this work, we provide a first
exploration of a more general view of PbRL i.e. in human-AI team setting where
the human-in-the-loop co-exists in the same environment as the AI agent and
performs actions for achieving the team goal.

The fundamental approach to account for human preferences has been to learn a
reward function by querying the human-in-the-loop for their feedback over a
pair of agent trajectories. Such preference elicitation becomes more
challenging at various levels in a multi-agent scenario. For example, multi-
agent (with human-in-the-loop) can span multiple humans or multiple AI agents,
which can be cooperative, competitive or act as a swarm system. Moreover,
taking human preferences into consideration can be from an external human
observing a human-AI team or from the human actively participating in the
team, etc. Also, the preferences can themselves be about arbitrary aspects of
interaction or behavior between the multiple agents. As a first exploration of
Preference-based RL in a multi-agent setting, we constrain our problem to a
two-agent cooperative Human-AI team where the human-in-the-loop provides their
preferences on the team behavior. Other settings can be explored in future
works.

In this work, we specifically look at human-AI teaming as a co-operative game
referred to as Human-AI Team PbRL, adopted from [10], with the assumption that
the human agent defines the team reward and uses it to compute feasible team
strategies, but the AI agent does not have access to it. This reward function
is representative of the human’s preferences on the joint behavior of the two
agents. We propose a suite of domains, comprising of variants of Highway
domain [11] and adapted MuJoCo locomotion domains [12], that requires forced
cooperation in a two-agent setting with one agent assumed to be a human agent.
As a first exploration of PbRL in Human-AI team setup, we adapt existing
state-of-the-art single agent PbRL methods (i.e. RUNE [13], SURF [14] and
PEBBLE [15]). Operating under this setting, we argue along two aspects
particular of human-AI teaming : Human Flexibility and Access to Human Policy.
Human Flexibility, informally, describes how rigid the human agent is with
respect to using a specific team strategy. Since the human is acting in the
shared world towards a team goal, the robot must find a policy for itself
which belongs to set of team strategies considered feasible by the human
agent. Otherwise, either the human agent would have to adapt (which can be
costlier than having an AI agent adapt) or the team would suffer from sub-
optimal performance. Furthermore, the size of this feasible set can be very
small or even singleton owing to cognitive limitations of human in computing
several strategies or simply due to their preference over team behavior.
Another important aspect is the agent’s access to human policy during learning
from human preferences. If the agent has zero access to the human’s policy (as
selected by the human agent to execute), it has an additional challenge of
‘imagining’ human actions when it queries the human for their preference on
the team behavior. On the other hand, the problem can be simplified to a
single-agent PbRL setting if the agent has complete knowledge of the human
policy and that the human has a single policy in their set of feasible team
strategies. We call this as Specified Orchestration, which requires maximal
information regarding human behavior and arguably the simplest case for the AI
agent, therefore, can be treated as a loose upper bound to the performance of
Human-AI PbRL algorithms.

We summarize the contributions of this work as follows:

  1. 1.

This work serves as a first exploration of Preference-based Reinforcement
Learning beyond a single-agent setting.

  2. 2.

We propose a suite of two-agent domains which require forced cooperation
between the two agents to achieve the team objective. The domains are packaged
with human’s feasible set of policies (computed through an oracular approach)
and can be used for future research in Human-AI team PbRL.

  3. 3.

We adopt and modify state-of-the-art single-agent PbRL algorithms for our
setting and benchmark their performance in policy-learning on the proposed
domains.

  4. 4.

We consider specific aspects of Human-AI team affecting Human-AI PbRL, i.e.
Human flexibility in opting for a team strategy and Agent’s access to human
Policy, and perform a comprehensive analysis. In particular, we find the case
of Specified Orchestration (where the agent has complete access to human’s
selected policy and human has a singleton set of team strategies in mind) as a
useful upper-bound where the PbRL algorithms exhibit effective performance.

We begin with discussing an example of active human-AI teaming with required
co-operation, and analyze the assumptions on the flexibility of the human-in-
the-loop in Section 2. Next, we give an overview of the relevant literature in
Section 3, followed by describing the preliminaries for this work in Section
4. Section 5 provides a formal framework under which we operate, and defines
the notions of Human Flexibility, Agent’s access to Human policy and Specified
Orchestration. We describe the proposed domain suite in Section 6.1 and
provide an empirical investigation of PbRL for the Human-AI team setup in
Section 6. Finally, we conclude with our discussion on the learnings from this
work motivating PbRL for human-AI teaming problems in Section 7. An appendix
has also been attached as supplementary that includes additional details on
the experiments and results, for the ease of reproducibility of our work for
future research.

##  2 Active Human-AI Teaming Example

Figure 1: Two Arm Peg-In-Hole Environment: A representative example for Human-
AI Team PbRL Cooperation Game.

Let us consider the example of the Baxter robot, a classic bi-manual robot as
shown in Figure 1. There are two robotic arms where one is holding a peg while
the other is holding a square plate with a hole in it. Assume that the arm
holding the peg is being controlled by the human agent. The human agent has
already decided on a policy that they will be moving the peg to a specified
location, and insert the peg according to their policy. While there may exist
multiple corresponding robot policies that successfully allow for team
success, the human agent has specific preferences on the robot behavior as
well. For example, the human wants the robot to hold the plate horizontally,
reach the location where the peg would be in time and the robot arm should
slowly approach that location for safety. The joint task for the two agents is
to place the peg in the hole (square plate) successfully without collisions in
accordance with the human preference.

The agent is unaware of its goals (holding the plate horizontal at some
location), the human preference on how to achieve that goal (manner of
reaching the location) and the human partner policy (where the location of peg
would be and the trajectory of the human controlled arm to avoid collisions).
Therefore, it must utilize some form of communication to gather information
about these aspects. Such communication may happen through explicit language-
based interaction (which may require an expert understanding of the internal
working of the robot, and a robust method for reward function specification
that is not prone to reward-hacking), through demonstrations (but in this
scenario the human agent holds a peg whereas the robot agent holds the plate
which makes providing a demonstration to teach the robot non-trivial) or
through binary feedback on agent’s queries (PbRL). In this work we explore the
latter case of studying PbRL, where the agent generates behavior trajectories
with imagined actions of the human and itself and queries a pair of these
behaviors for human feedback. The human agent views the queried trajectory
(typically presented as a video) pair and specifies which they prefer more,
i.e. which pair achieves the team objective according to them. Recall that the
human agent has pre-computed a policy for themselves and has an expectation
from the AI partner to complete any remainder objectives, therefore the
preference labels are given according to how well the AI-agent is conforming
to the human-expected policy. Naturally, the key challenges for the AI agent
are to realize the human partner’s chosen policy (such that the queried
behaviors show reasonable imagined human agent actions), and how large the
team-behavior solution set is based on the human’s internal reward function
(for example, the human may find it acceptable to have the plate be placed
somewhere in a large region at any angle).

##  3 Related Work

Preference-based Reinforcement Learning: Preference-based RL has become a
popular paradigm for human-in-the-loop settings where the human is queried for
binary feedback by the AI agent to learn a reward function [16, 17, 18, 19,
20, 21, 22]. These methods have not only shown to be effective in teaching an
AI agent to accomplish a task where rewards may be otherwise hard to specify
such as in the case of tacit knowledge tasks, but also to incorporate the
human-in-the-loop’s preferences on how the task needs to be achieved [17].
However, PbRL methods have only been seen in single-agent settings where the
AI agent is acting in the environment, and the human is only interacting with
the system as and when asked for query feedback [23, 24, 25, 26]. Having
realized the potential of PbRL frameworks, we see its applicability in human-
AI teaming scenarios where both the human and the AI agent act in the
environment to accomplish a team task, and the human may want to specify their
preferences through binary feedback distilled into the agent’s reward
function. Closest to our research are prior works in multi-agent Inverse
Reinforcement Learning (IRL) [27, 28, 29]. We note that IRL and PbRL have key
distinctions in assuming expertise of the human in the loop and the modality
of the feedback. [30] were one of the first to tackle the multi-agent IRL
problem with a central controller. Several works have attempted to study
multi-agent IRL for homogeneous agent systems [31, 32, 33, 34] whereas our
focus is on a heterogeneous system with a two-player Human-AI cooperative
team. More recently, [10] proposed the Cooperative Inverse RL formulation to
highlight the limitation of classic formulation which does not allow humans to
provide teaching demonstrations. In contrast, our focus is on allowing the AI
agent to make explicit queries to the human agent to elicit preferred
behaviors. Further, while IRL based setups are dependent on expert-
demonstrations by the human agent, such demonstrations may be infeasible (in
our Peg-In-Hole example, the human does not control the arm with the square
plate), or the human maybe incapable. PbRL, on the other hand, makes it easier
for the human agent to provide a binary feedback over trajectories generated
by the agent.

Multi-Agent Teaming : Deep Reinforcement Learning has also been used in multi-
agent settings [35, 36, 37] such as non-competitive games [38, 39], zero-sum
[34], general-sum games [33, 40] and other collaborative environments where
rewards are not shared across agents [41, 42]. More recently several works
have investigated the challenges in Human-AI teams but commonly assume robot’s
access to the team reward function [6, 43, 44, 45]. Prior works have also
studied joint-decision making in a two-agent collaboration setting [46, 47,
48, 49] but do not consider inverse RL or PbRL setting.

##  4 Preliminaries

Preference-based Reinforcement Learning :  In traditional PbRL setups, an
agent $A$ interacts with the environment $\mathcal{E}$ with a missing reward
function $\mathcal{R}$. It assumes the underlying problem as a MDP
$\mathcal{M}=\langle\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R},\gamma\rangle$.
The agent $A$ may not have access to this reward system $\mathcal{R}$ as it
may either be hidden [50], hard to specify as seen in tacit-control tasks
[17], or otherwise may not be able to capture the goal completely as seen in
human-in-the-loop scenarios [51].

The problem of learning the unknown reward function $r_{\psi}$, in single-
agent settings, is formulated as a supervised learning problem [17]. A
classifier is trained on limited human feedback over agent trajectories.
Formally, each agent trajectory is a segment $\sigma$, a sequence of agent
observations and actions
$\\{(s_{k},a_{k}),(s_{k+1},a_{k+1}),\cdot\cdot\cdot(s_{k+N-1},a_{k+N-1})\\}$
where $N$ represents the horizon. The human-in-the-loop, i.e., the teacher,
gives a feedback $y\in\\{0,1\\}$ to the agent when queried with a pair of
trajectories ($\sigma_{1},\sigma_{2}$), and this feedback is stored as a label
along with the two segments in a trajectory buffer $\mathcal{D}$ as
($\sigma_{1},\sigma_{2},y$). Following the Bradley-Terry preference model
[52], the preference predictor using the reward function $r_{\psi}$ is
learned:

$P_{\psi}[\sigma_{0}\succ\sigma_{1}]=\frac{\exp\Bigl{(}\sum_{t}\hat{r_{\psi}}(s_{t}^{0},a_{t}^{0})\Bigl{)}}{\sum_{i\in\\{0,1\\}}\exp\Bigl{(}\sum_{t}\hat{r_{\psi}}(s_{t}^{i},a_{t}^{i})\Bigl{)}}$
(1)

where, $[\sigma_{0}\succ\sigma_{1}]$ implies that trajectory segment
$\sigma_{0}$ is preferred over trajectory segment $\sigma_{1}$ by the human
teacher. Given the trajectory buffer $\mathcal{D}$, the reward function
$r_{\psi}$ is learned by training on minimizing the following Cross-Entropy
loss:

$\mathcal{L}_{CE}=-\displaystyle\mathop{\mathbb{E}}_{\begin{subarray}{c}(\sigma_{0},\sigma_{1},y)\sim\mathcal{D}\end{subarray}}\Bigr{[}y(0)\text{log}P_{\psi}[\sigma_{0}\succ\sigma_{1}]+y(1)\text{log}P_{\psi}[\sigma_{1}\succ\sigma_{0}]\Bigr{]}$
(2)

##  5 Preference-based RL for Human-AI Teaming

Preference based RL has been predominantly studied in single-actor human-
observer setting where the human provides feedback on the agent’s behavior but
does not act in the shared environment. The aim of this work is to explore
PbRL for human-AI team scenario. As discussed in Section 1, there can be
multiple interpretations of multi-agent (or a two-agent human-AI team) PbRL.
We consider the setup where the human and the AI agents are a cooperative team
and the role of the robot is to achieve goals as set by the human agent.
Having no knowledge of its objectives, the robot relies on a reward learning
process through Preference-based Reinforcement Learning to elicit and learn
about the human preferences over the team behavior. We formalize our Human-AI
Team PbRL as a special case of the Cooperative Inverse Reinforcement Learning
formulation [10] in section 5.

Further, we discuss the two critical aspects for Human-AI team i.e. Human
Flexibility over team behavior 5.2 and assumption on agent’s access to human
policy 5.3. We discuss a special case with assumed knowledge about human’s
policy and the fact that human admits a single team behavior as an optimal
solution, referred to as Specified Orchestration. Since we adapt PbRL methods
developed for single-actor human-observer setups for our first exploration of
Human-AI team PbRL, we discuss the design decisions in Section 6.2.

###  5.1 Human-AI PbRL Cooperation Game

[10] introduces the problem of Cooperative Inverse Reinforcement Learning but
studies it for Apprenticeship Learning. For our Human-AI Team PbRL setting, we
modify their formulation as follows:

######  Definition 5.1.

A Human-AI PbRL cooperation game, as a special case of a Cooperative Inverse
Reinforcement Learning game, $M$ is a two-player Markov game with identical
payoffs between a human $H$ and a robot $R$. The game is defined as a tuple
$M=(S,\\{{\mathcal{A}^{H},\mathcal{A}^{R}}\\},T(\cdot|\cdot,\cdot,\cdot),R_{\theta}(\cdot,\cdot,\cdot),\gamma)$
with the following definitions:

  * •

$S$ a set of world states: $s\in S$;

  * •

$\mathcal{A}^{H}$ a set of actions for H : $a^{H}\in\mathcal{A}^{H}$;

  * •

$\mathcal{A}^{R}$ a set of actions for R : $a^{R}\in\mathcal{A}^{R}$;

  * •

$T(\cdot|\cdot,\cdot,\cdot)$ a conditional distribution on the next world
state, given previous state and action for both agents, i.e.,
$T(s^{\prime}|s,a^{H},a^{R})$;

  * •

$R_{\theta}(\cdot,\cdot,\cdot)$ a reward function that maps world states,
joint actions to reward values according to H’s preference on the team
objective, H’s objectives, R’s objectives and preference over R’s behavior;

  * •

$\gamma$ a discount factor : $\gamma\in[0,1]$

The reward function $R$ is set by the human agent defining the team objectives
and human-agent acceptable policy pairs $(\pi_{H},\pi_{R})$. The game begins
at the initial state $s_{o}\in S$. The human agent pre-computes acceptable
solutions to the game with assumed knowledge on AI partner’s capabilities.
Suppose the computed solution set is
$\Pi=(\pi_{H},\pi_{R})_{0},\pi_{H},\pi_{R})_{1}\cdots\pi_{H},\pi_{R})_{n}$.
For simplicity, suppose the solution set is singleton. The human takes an
action following their policy $\pi_{H}$. The robot takes a random action in
$\mathcal{A}^{R}$. As the game proceeds the robot builds a bank of
trajectories from past experiences. Following PbRL setup, the agent samples
two trajectories from this bank and queries the human agent for their
feedback. The human agent uses their internal reward function $R$ to provide
their preference label. A behavior pair where the shown human actions align
with the true human actions from $\pi_{H}$ and the shown robot actions align
with human expected $\pi_{R}$ such that resultant return of the trajectory is
higher than the alternate is preferred.

While the policies can be arbitrary functions of history to next action as
$\pi_{H}:[\mathcal{A}^{H}\times\mathcal{A}^{R}\times
S]*\rightarrow\mathcal{A}^{H}$ (and similarly for robot R), we restrict the
policy to be defined on states (thereby assuming the dynamics, reward function
and the optimal policy pair is Markovian with respect to the world state).

###  5.2 Flexibility of the Human-in-the-Loop

It is important to note that for our human-AI teaming PbRL setup, we only
consider the case where the AI agent’s access to the underlying reward
function is restricted. That is, the agent does not experience a sampled
reward from the environment, and instead, depends on an explicit reward
learning procedure (through human feedback) to realize its objectives in the
team. Assume the set $\Pi=\\{\Pi_{1},\Pi_{2},\cdots\Pi_{n}\\}$ where
$\Pi_{i}=(\pi_{i}^{H},\pi_{i}^{R})$ be the possible solutions of the
Cooperation Game defined in Section 5.1 as can be computed by the human. We
define human flexibility over their policy, i.e.
$\\{\pi_{i}^{H}\\}_{\\{i=1\cdots n\\}}$ as :

######  Definition 5.2.

Given $\Pi$ solution set, the human in the loop is said to be
$\epsilon$-H-flexible if

$\left|\\{\pi_{k}^{H}\,|\,k=1\,\text{to}\,n\\}\right|=\epsilon-1$

The definition of H-flexible (5.2) human in the loop helps us recognize the
rigidity of the human agent over a strategy for the two agents. A large
$\epsilon\rightarrow|\Pi|$ implies that the human is willing to execute any
one of the policies in $\Pi_{H}$ subsequently providing the robot to converge
to a larger solution space (policies $\pi_{k}^{R}$). The nature and
requirements of the cooperation for the given team-goal can be helpful in
determining the values of $\epsilon$. For instance, $\epsilon=0$ is useful
when considering human cognitive limitations, thereby the human is able to
come up with a single policy for themselves.

When $\epsilon>1$ then the human agent has a choice to make in selecting which
policy $\pi_{k}^{H}$ they will follow. In this work, we assume that the AI
agent’s objective is to be conformant to all the human policies. This choice
is motivated from how prior works in Human-AI cooperation have attempted to
learn the robot’s policy [53]. Other alternatives are: the agent is compatible
to at least one of the human policies, a subset of the human policies etc.
exist which would also impact the training regime of the RL agent (i.e. which
policy is used by the human agent when the Robot is learning its policy?).
Following prior works [53, 54] in learning a response policy, during training
we randomly sample the human’s policy which is used for an entire episode.

###  5.3 Access to Human Policy

Based on the Human-AI Team PbRL cooperative game describe in Section 5.1, the
human and the robot take actions in a shared environment. This implies that
for the duration of training, in addition to the preference feedback, the
framework assumes the human to sample their optimal action $a_{h}$ for a world
state $s$. That is, the agent gains indirect access to the human policy by
virtue of the shared world state and the fact that human-AI coexist in the
shared world. While this is the default operating assumption through this
work, it has higher requirements from the human agent as compared to single-
agent PbRL. For example, in single-agent PbRL the human is required to be
present in the loop during agent’s training procedure and provide feedback,
whereas in the current setup, in addition to the preference feedback the human
is also required to follow their policy throughout the training process (which
can be millions of steps).

We can reinterpret the Human-AI Team PbRL cooperative game as follows: the
agent has access to an environment simulator which can simulate the agent’s
actions and the human’s actions. Now, the agent uses the policy $\pi_{R}$
being learned and an imagined human policy $\pi_{R}^{h}$ to generate team
behaviors. These behaviors can be stored in a trajectory buffer and queried to
the human actor. The advantage of this re-interpretation is that while the
human agent is present in the loop they are not required to execute millions
of actions during agent’s training. The disadvantage is, the agent has to
model the human’s policy in addition to the human’s preferences. This re-
interpretation of the cooperative game can enable us to study the performance
of the PbRL agent when it has budgeted access to human actions. Full access is
equivalent to the default interpretation, 50% access is when the agent can
query for the human’s actions for the first half of the training process and
0% access implies the agent cannot query human’s actions at all. In this work,
the agent defaults to using random actions as imagined human actions when it
exhausts the human-action query budget. Techniques like behavior-cloning [6]
can be leveraged to independently model the human’s policy along with the PbRL
step for modeling human preferences in future work.

###  5.4 Specified Orchestration

Recall our Two Arm Peg-In-Hole example in Section 2. It is clear that since
the human agent defines the team objective through reward function $R$, they
are restricting the possible robot behaviors to achieve the team goal. For
example, the robot arm should approach very slowly towards the human
controlled arm because of human’s preference for safety. It becomes extremely
challenging if the agent does not have access to the human’s policy. Further,
as described in Section 5.2, if the human is switching their policy in the
feasible team behavior set during the agent’s training process, the agent now
has an additional objective of learning a conformant response policy that is
compatible with all of the policies in $\epsilon$-Human Flexible solution set.
Our empirical results in sections 6 show that the agent’s performance in
policy learning is significantly diminished in the absence of these
assumptions.

######  Definition 5.3.

Specified Orchestration is a special case of the Human-AI Team Cooperative
game $M=(S,\mathcal{A},\mathcal{T},\mathcal{R},\gamma)$ where the human agent
(H) is 1-H-flexible with the team behavior solution $(\pi_{H},\pi_{R})$ and
$\pi_{H}$ is accessible by robot R.

Note that this is a special case of the cooperative game where we have
specific assumptions on the human’s preference on the team behavior and access
to human’s actions. While these assumptions are quite expensive to meet, they
play an important role in benchmarking Human-AI Team PbRL algorithms.
Specified Orchestration provides a loose upper-bound performance by assuming
away other challenging aspects of Human-AI interaction except preference
elicitation, reward learning and subsequent policy learning.

##  6 Benchmarking Human-AI Team PbRL

In this section we will discuss the proposed benchmark suite (Section 6.1) and
our experiments showcasing performance of Human-AI Team PbRL across the two
critical aspects (as discussed in Sections 6.4 and 6.3) i.e. impact of human
flexibility on team behavior and the impact of providing access to human
policy to the PbRL agent.

Further, our discussion will highlight the challenges associated with using
PbRL for Human-AI team cooperative game and empirically answer the following
questions :

  1. 1.

What is the relative performance when the human partner can choose from
multiple possible policies to follow, for accomplishing the team task?

  2. 2.

In Human-AI team PbRL, what is the performance of the adapted single-agent
PbRL algorithms with maximal assumptions on access to human policy and
$\epsilon=1$-H-flexible human (Specified Orchestration)?

  3. 3.

How do PbRL algorithms perform with a budgeted access and no access to human’s
policy?

Question (1) supports our discussion on Human Flexibility, (2) highlights the
case where PbRL algorithms are effective making Specified Orchestration a
loose upper bound performance for studying Human-AI team PbRL performance.
Question (3) discusses the special case of limiting agent’s access to human
policy thereby highlighting the need for an explicit modeling of human policy.

In the subsequent sections we first study (Q2) which provides us with an upper
bound performance of PbRL algorithms followed by (Q3) where we find that the
adapted PbRL algorithms fail without access to human policy. Finally, we study
(Q1) under full-access to human policy with a $\epsilon>1$-H-flexible solution
set.

We refer the readers to the supplementary material (Sections A, B, and C) for
detailed discussion on algorithms, the human policy and preference oracles,
domain construction and the respective hyperparameters used for the
experiments.

Figure 2: MA Highway domain - human and the AI agent cars are shown one behind
the other. Figure 3: MA MuJoCo domains - (from L to R) Cheetah, Ant, Walker,
Swimmer, & Hopper. The action space for the joints in each of the domains has
been split between the human and the AI agent. Algorithm 1 Adapting Single
Agent PbRL for Human-AI Team

Input: feedback frequency $K$, # queries per feedback session $M$, human
policy set: $|\Pi|$, agent access budget: $b$

Initialize parameters of $Q_{\theta}$ and $\hat{r}_{\psi}$

Initialize a dataset of preferences $D_{h}\leftarrow\varnothing$

// EXPLORATION PHASE

$D_{\tau},\pi_{\phi}\leftarrow$ EXPLORE() in [15]

POLICY LEARNING PHASE

for each iteration do

// REWARD LEARNING PHASE

if new episode then

choose human policy $\pi_{H}\in|\Pi|$

end if

if iteration % $K$ == 0 then

for m in 1…$M$ do

$(\sigma^{0},\sigma^{1})\sim$ SAMPLE() in [15]

Query human partner for $y$

end for

for each gradient step do

Sample mini-batch $\\{{(\sigma^{0},\sigma^{1},y)}_{j}\\}^{D_{h}}_{j=1}\sim
D_{h}$

end for

Relabel entire replay buffer $D_{\tau}$ using $\hat{r}_{\psi}$

end if

REINFORCEMENT LEARNING PHASE

for each time-step $t$ do

if time-step $t\leq$ budget $b$ then

sample $a_{h}\sim\pi_{H}$

else

randomly sample $a_{h}\sim\mathcal{A}^{H}$

end if

Collect $s_{t+1}$ by taking $a_{t}\sim\pi_{\phi}(a_{t}|s_{t})$ and $a_{H}$

Store transitions $D_{\tau}\leftarrow
D_{\tau}\\{(s_{t},a_{t},{\color[rgb]{1,0,0}a_{h}},s_{t+1},\hat{r}_{\psi}(s_{t}))\\}$

end for

for each gradient step do

Sample random mini-batch $\\{{\tau_{j}}\\}^{D_{\tau}}_{j=1}\sim D_{\tau}$

Optimize $L^{SAC}_{critic}$ and $L^{SAC}_{actor}$ w.r.t. $\theta$ and $\phi$,
respectively [15]

end for

end for

###  6.1 Evaluation Domains

####  6.1.1 Multi-Agent Highway Orchestration Task:

We first design a set of three multi-agent environments built on top of the
Highway environment [11]. As shown in Figure 2, we construct a three lane
highway where the human car (in orange) is on the top-most lane, and the AI
agent (in purple) has to overtake and merge back into the human car’s lane by
increasing its speed and avoiding collisions with any other car in the scene.
The human prefers that the AI agent car overtakes from the right-most (bottom-
most in the figure) lane and not from the middle lane. Similarly, we construct
two more scenarios. In the second one, both the cars are in the right lane and
the human preference is for the AI agent car to overtake from the left-most
lane. In the third domain, both cars are in the middle lane and the AI agent
can only overtake from the left lane. In the results shown in Figure 4, these
environments are referred to as MA Highway - Right Overtake, MA Highway - Left
Overtake, and MA Highway - Middle Overtake respectively.

####  6.1.2 Multi-Agent Locomotion Orchestration Tasks:

We present a more complex domain suite, which necessitates cooperation, to
study Human-AI team PbRL cooperation game. Following prior works in
constructing multi-agent environments [55] in continuous control domains, we
modify the MuJoCo-based locomotion environments [56] for the human-AI teaming
tasks. The locomotion tasks include a robot attempting to move (for example by
a walk, run or crawl). Such tasks are essentially tacit knowledge robotic
simulation tasks for which reward specification can be challenging [14, 9].
Moreover, single-agent variant of these environments have been used as
benchmark domains by single-agent PbRL works [14, 13, 15], further motivating
their use.

We construct multi-agent settings for Cheetah, Ant, Walker, Swimmer & Hopper
environments, as shown in Figure 3. In each environment, the human agent
controls the orange highlighted joints action space, while the rest of the
joint angle controls are for the AI agent highlighted in purple. For example,
in the MA-Walker, the human controls the walker’s right leg joints to walk
forward, while the AI agent controls the left leg joints.

Figure 4: Learning curves on MA Highway (rows 1 to 3) and MA MuJoCo - Cheetah
(row 4): comparing (from L to R) (a) 0% access to $\pi_{H}$, (b) partial
access to $\pi_{H}$, and (c) human flexibility on multiple $\pi_{H}$, with
Specified Orchestration case that assumes a single $\pi_{H}$ and complete
access to it; as measured on the episodic returns. The solid lines and shaded
regions represent the mean and standard deviation, respectively, across three
runs.

((a))

((b))

((c))

((d))

Figure 5: Learning curves on MA MuJoCo domains: comparing PbRL algorithms
extended to the multi-agent setting under Specified Orchestration, as measured
on the episodic returns. The solid lines and shaded regions represent the mean
and standard deviation, respectively, across three runs.

###  6.2 Adapting Single Agent PbRL to Human-AI Team setting

We present our modified PbRL algorithm for the human-AI team setting, which
uses the PEBBLE algorithm [15] as a backbone. Note, we make the same
modifications for experiments using RUNE [13] and SURF [14] as these
algorithms also use PEBBLE as their PbRL backbone. We do not require any
change to the model architectures (including reward approximator and policy
approximator). The updates are highlighted in red (see Algorithm 1) and can be
easily incorporated into any online single agent PbRL algorithm.

Specifically, we assume a human policy set $|\Pi|$ from which the human agent
can select a single (or multiple) policies for themselves, and the AI agent
has the access to the human policy during training with a budget $b$. When a
new episode begins, we uniformly sample a human policy $\pi_{H}$ from $|\Pi|$.
Note, if the human only has a single policy, i.e., in the case of Specified
Orchestration, then we use the same $\pi_{H}$ for every episode. During the
policy learning phase for the AI agent if the agent has remaining budget to
request actions from human policy, we sample $a_{H}$ from $\pi_{H}$, or else,
the AI agent assumes a random action on behalf of the human. Again, in the
case of Specified Orchestration, the budget $b$ allows the AI agent to access
human actions till the end of the training. As per our re-interpretation of
the Human-AI team PbRL Cooperation game in Section 5.1, this is equivalent to
the assumption that human is acting in the shared world with the agent.

Following past works in Preference based Reinforcement Learning, we use a
simulated oracular human policy to provide the actions on behalf of the human
agent. As suggested in [22] we use a scripted preference labeling function
that provides preference labels on behalf of the human agent to the PbRL
agent. We train SAC [57] by assuming centralized controller for human and AI
agent using underlying task reward available with the environment package for
MuJoCo [12] and handcrafted reward function for Highway. We take the actions
corresponding to the joint angles under control of the human as human actions
and rest for the AI agent’s actions expected by the human as part of the
compatible policy pair. Preference labels are generated based on hand-crafted
rewards for MA Highway and environment packaged rewards (which serve as team
reward) for MA MuJoCo. We refer the readers to the appendix for further
details.

###  6.3 Impact of Access to Human Policy

####  6.3.1 Specified Orchestration - complete access to $\pi_{H}$:

To answer Question (2) and to support our discussion on Specified
Orchestration, we report the performance of all three PbRL algorithms on the
three MA Highway, and five MA MuJoCo environments in Figures 4 and 5. We note
from each of the plots, that PEBBLE [15], RUNE [13] and SURF [14] show the
optimal returns. From Figure 5, we also note that RUNE consistently performs
better than SURF and PEBBLE in these multi-agent settings.

####  6.3.2 Restricting Access to Human Policy:

To answer Question (3), we first investigate the case where we restrict the AI
agent’s budget to access the human policy $\pi_{H}$ to 50% for MA Highway
domains, and 10% for MA MuJoCo - Cheetah. We allocate this entire budget in
the first half of the training. For example, in the case of MA MuJoCo -
Cheetah domain, the AI agent has access to $\pi_{H}$ up-till 100k steps, and
assumes random actions sampled for the remaining training steps. We report the
results in the second column of Figure 4. Surprisingly, we note that the
performance plummets significantly as soon as the human agent’s actions are
not available for all the domains.

####  6.3.3 No Access to Human policy:

In reference to Question (3), we report the performance of PEBBLE [15], RUNE
[13], and SURF [14] on the three MA Highway and MA MuJoCo - Cheetah domains in
first column of Figure 4. None of the algorithms have any access to $\pi_{H}$,
and thus, have to ‘imagine’, i.e., assume random actions for the human agent.
In all the four domains and three algorithms, we note that the performance is
significantly worse as compared to the case when the AI agent has complete
access to $\pi_{H}$. Utilizing explicit modeling of human policy is a natural
next step and an important future work, but it beyond the scope of the first
exploration of PbRL in a Multi-agent setting.

####  6.3.4 Observed Challenges:

From the experiments with 0% and partially restricted access to the human
partner policy, we observe that reward learning is severely impacted. Since
Specified Orchestration assumes maximal information on the human partner’s
policy and preferences, the team performance is optimal. This highlights the
potential of using Specified Orchestration as a means to obtain an loose upper
bound on the Human-AI team PbRL cooperation game performance. Further, unless
the AI agent explicitly models human policy (and preferences) with a
restricted access, reward learning along with accomplishing the team task
stays a challenging problem.

###  6.4 Impact of Human Flexibility

####  6.4.1 Multiple Human policies:

We observe that each of three PbRL approaches perform the best in the case
where the agent had complete access to $\pi_{H}$. To study the impact of Human
Flexibility (our Q1) on PbRL performance we operate under the assumption that
agent has complete access to the human policy (i.e. human executes their
policy during agent’s training)

We assume that for each episode of learning, the human agent can sample 1-out-
of-N (3, in our case) policies to provide the actions to the AI agent. From
the results shown in the third column of Figure 4, we note that the
performance comparatively drops as in the case where we assumed a single
$\pi_{H}$ and complete access to it. The decline in performance is amplified
in the more complex case of MA MuJoCo Cheetah where cooperation is forced
amongst the two agents. We conclude that the team performance is the highest
in the case of Specified Orchestration, where the human agent has a single
$\pi_{H}$ and the AI agent has complete access to the same.

####  6.4.2 Observed Challenges:

From the experiments with multiple $\pi_{H}$ available to the AI agent during
the training phase, we conclude what while it is a cognitively challenging
task for a human agent to compute multiple policies and also provide feedback
in this case, reward learning is also negatively impacted as seen from our
empirical analysis.

##  7 Conclusion & Future Work

We perform a first exploration of Preference-based Reinforcement Learning for
Human-AI team setup. We introduce a Human-AI PbRL Cooperation game as a
special case of Cooperative Inverse Reinforcement Learning [10] and argue
across two critical dimensions, namely - AI agent’s Access to Human Policy and
Human Flexibility. In our heterogeneous Human-AI cooperative setup, both the
agents are rewarded according to human’s reward function for the team
comprising of human’s objectives for the team, sub-objectives for themselves
and the robot, and human preferences on team behavior. For investigating the
performance of PbRL approaches, we modify the existing state-of-the-art
single-agent PbRL algorithms (namely, PEBBLE [15], SURF [14] & RUNE [13]). To
this end, we benchmark PbRL algorithms on our benchmark suite consisting of
simpler Highway domains and more complex Robot locomotion.

We note that all three PbRL algorithms gave optimal returns only when the
human partner has a preference for a single policy pair (team behavior) and
that they have complete access to human partner’s policy (termed as Specified
Orchestration). We discuss how Specified Orchestration can be useful to
benchmark future works in Human-AI team PbRL as a loose upper bound on
algorithm performance. To our surprise, partial access to $\pi_{H}$ also had a
drop in performance as soon as the agent’s access budget was exhausted.
Subsequently, we also observe poor performance when the AI agent does not have
access to human partner’s policy. We conclude that there is a need for AI
agents to explicitly model partner human policy (along with their
preferences). Furthermore, utilizing these observations, we assumed complete
access to $\pi_{H}$, and ablated in the case where the human partner is
flexible, i.e., has multiple acceptable team strategies. Counter-intuitively,
we find that when the human is more flexible, the agent’s performance
declines. This is primarily because we constrain our AI agent to learn a
conformant policy (compatible with all human policies).

From our benchmarking results, we conclude that Specified Orchestration, while
assuming the maximal amount of information on the human, shows notable
performance across all domains. However, this can also be seen as one of the
key limitations as it may not always be the case that the human can easily
compute and provide their policy to the AI agent (or be available to act
during the training procedure of the AI agent). Secondly, while this approach
allows for optimal team performance of an AI agent with a specific human, it
may be hard to generalize the learnt reward function (and hence, the learnt
policy) to multiple human partners. Lastly, as in the case of single-agent
PbRL works where human is just an observer, the problem of expensive human
interaction with the AI agent is carried over to our setting. To conclude, our
work provides researchers with a first exploration of PbRL beyond single-agent
setup, towards human-AI teaming.

##  8 Societal Impact

This paper focuses on motivating the use of Preference-based Reinforcement
Learning (PbRL) for human-AI teaming settings. We argue on two critical
aspects of such teaming settings in this work, namely - Human Flexibility and
AI agent’s Access to Human Policy. The empirical benchmarking includes
experiments across a suite of domains with simulated human oracles. Since this
work uses simulated human oracle policies and preferences for the experiments,
it does not bear any direct ethical implications. However, we note that once
such studies advance to real-world teaming problems, ethical considerations
with respect to privacy, adherence to the intended preference, consent,
transparency, etc. will come in place.

Specifically, with the use of PbRL methods in real-world settings, we
highlight that the human partner may experience unexpected and unwanted
scenarios while interacting with the AI agent. This may include dealing with
irrational or erratic behavior of the AI agent. This may further lead to
frustration and fatigue for the human which may be undesirable. Moreover,
trust on the AI agent can also play a crucial role in the effectiveness of
modeling a human-AI teaming problem. For example, if the human partner does
not trust the AI agent for task completion or its ability to learn their
preferences (due to any possible reason), the human may find it difficult to
give feedback that can be useful for the AI agent’s training. Hence,
generalizing PbRL for real-world human-AI teaming settings may require design
decisions that can encompass these aspects to benefit the human partner agent.

## References

  * [1] Sandra Wachter, Brent Mittelstadt, and Luciano Floridi.  Transparent, explainable, and accountable ai for robotics.  Science robotics, 2(6):eaan6080, 2017.
  * [2] Alexander Maedche, Christine Legner, Alexander Benlian, Benedikt Berger, Henner Gimpel, Thomas Hess, Oliver Hinz, Stefan Morana, and Matthias Söllner.  Ai-based digital assistants: Opportunities, threats, and research perspectives.  Business & Information Systems Engineering, 61:535–544, 2019.
  * [3] Yoav Mintz and Ronit Brodie.  Introduction to artificial intelligence in medicine.  Minimally Invasive Therapy & Allied Technologies, 28(2):73–81, 2019.
  * [4] Yali Du.  Cooperative multi-agent learning in a complex world: challenges and solutions.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 15436–15436, 2023.
  * [5] Kobi Gal and Barbara J Grosz.  Multi-agent systems: Technical & ethical challenges of functioning in a mixed group.  Daedalus, 151(2):114–126, 2022.
  * [6] Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan.  On the utility of learning about humans for human-ai coordination.  Advances in neural information processing systems, 32, 2019.
  * [7] Sage Bergerson.  Multi-agent inverse reinforcement learning: Suboptimal demonstrations and alternative solution concepts.  arXiv preprint arXiv:2109.01178, 2021.
  * [8] Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz.  A survey of preference-based reinforcement learning methods.  Journal of Machine Learning Research, 18(136):1–46, 2017.
  * [9] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  Advances in neural information processing systems, 30, 2017.
  * [10] Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan.  Cooperative inverse reinforcement learning.  Advances in neural information processing systems, 29, 2016.
  * [11] Edouard Leurent.  An environment for autonomous driving decision-making.  https://github.com/eleurent/highway-env, 2018.
  * [12] Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez-Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis.  Gymnasium, March 2023.
  * [13] Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel.  Reward uncertainty for exploration in preference-based reinforcement learning.  arXiv preprint arXiv:2205.12401, 2022.
  * [14] Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee.  Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning.  arXiv preprint arXiv:2203.10050, 2022.
  * [15] Kimin Lee, Laura Smith, and Pieter Abbeel.  Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training.  arXiv preprint arXiv:2106.05091, 2021.
  * [16] Johannes Fürnkranz, Eyke Hüllermeier, Weiwei Cheng, and Sang-Hyeun Park.  Preference-based reinforcement learning: a formal framework and a policy iteration algorithm.  Machine learning, 89:123–156, 2012.
  * [17] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  Advances in neural information processing systems, 30, 2017.
  * [18] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei.  Reward learning from human preferences and demonstrations in atari.  Advances in neural information processing systems, 31, 2018.
  * [19] Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg.  Scalable agent alignment via reward modeling: a research direction.  arXiv preprint arXiv:1811.07871, 2018.
  * [20] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano.  Learning to summarize with human feedback.  Advances in Neural Information Processing Systems, 33:3008–3021, 2020.
  * [21] Tom Bewley and Freddy Lecue.  Interpretable preference-based reinforcement learning with tree-structured reward functions.  arXiv preprint arXiv:2112.11230, 2021.
  * [22] Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel.  B-pref: Benchmarking preference-based reinforcement learning.  arXiv preprint arXiv:2111.03026, 2021.
  * [23] Kimin Lee, Laura Smith, and Pieter Abbeel.  Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training.  arXiv preprint arXiv:2106.05091, 2021.
  * [24] Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee.  Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning.  arXiv preprint arXiv:2203.10050, 2022.
  * [25] Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel.  Reward uncertainty for exploration in preference-based reinforcement learning.  arXiv preprint arXiv:2205.12401, 2022.
  * [26] Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel.  B-pref: Benchmarking preference-based reinforcement learning.  arXiv preprint arXiv:2111.03026, 2021.
  * [27] Saurabh Arora and Prashant Doshi.  A survey of inverse reinforcement learning: Challenges, methods and progress.  Artificial Intelligence, 297:103500, 2021.
  * [28] Stephen Adams, Tyler Cody, and Peter A Beling.  A survey of inverse reinforcement learning.  Artificial Intelligence Review, 55(6):4307–4346, 2022.
  * [29] David V Pynadath and Milind Tambe.  The communicative multiagent team decision problem: Analyzing teamwork theories and models.  Journal of artificial intelligence research, 16:389–423, 2002.
  * [30] Sriraam Natarajan, Gautam Kunapuli, Kshitij Judah, Prasad Tadepalli, Kristian Kersting, and Jude Shavlik.  Multi-agent inverse reinforcement learning.  In 2010 ninth international conference on machine learning and applications, pages 395–400. IEEE, 2010.
  * [31] Kenneth Bogert and Prashant Doshi.  Scaling expectation-maximization for inverse reinforcement learning to multiple robots under occlusion.  In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, pages 522–529, 2017.
  * [32] Kenneth Bogert and Prashant Doshi.  Multi-robot inverse reinforcement learning under occlusion with interactions.  In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems, pages 173–180, 2014.
  * [33] Xiaomin Lin, Stephen C Adams, and Peter A Beling.  Multi-agent inverse reinforcement learning for certain general-sum stochastic games.  Journal of Artificial Intelligence Research, 66:473–502, 2019.
  * [34] Xiaomin Lin, Peter A Beling, and Randy Cogill.  Multiagent inverse reinforcement learning for two-person zero-sum games.  IEEE Transactions on Games, 10(1):56–68, 2017.
  * [35] Thanh Thi Nguyen, Ngoc Duy Nguyen, and Saeid Nahavandi.  Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications.  IEEE transactions on cybernetics, 50(9):3826–3839, 2020.
  * [36] Yali Du, Bo Liu, Vincent Moens, Ziqi Liu, Zhicheng Ren, Jun Wang, Xu Chen, and Haifeng Zhang.  Learning correlated communication topology in multi-agent reinforcement learning.  In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pages 456–464, 2021.
  * [37] Jingqing Ruan, Yali Du, Xuantang Xiong, Dengpeng Xing, Xiyun Li, Linghui Meng, Haifeng Zhang, Jun Wang, and Bo Xu.  Gcs: graph-based coordination strategy for multi-agent reinforcement learning.  arXiv preprint arXiv:2201.06257, 2022.
  * [38] Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch.  Multi-agent actor-critic for mixed cooperative-competitive environments.  Advances in neural information processing systems, 30, 2017.
  * [39] Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer.  Cooperative multi-agent control using deep reinforcement learning.  In Autonomous Agents and Multiagent Systems: AAMAS 2017 Workshops, Best Papers, São Paulo, Brazil, May 8-12, 2017, Revised Selected Papers 16, pages 66–83. Springer, 2017.
  * [40] Adam Lerer and Alexander Peysakhovich.  Learning existing social conventions via observationally augmented self-play.  In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 107–114, 2019.
  * [41] Joel Z Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel.  Multi-agent reinforcement learning in sequential social dilemmas.  arXiv preprint arXiv:1702.03037, 2017.
  * [42] Edward Hughes, Joel Z Leibo, Matthew Phillips, Karl Tuyls, Edgar Dueñez-Guzman, Antonio García Castañeda, Iain Dunning, Tina Zhu, Kevin McKee, Raphael Koster, et al.  Inequity aversion improves cooperation in intertemporal social dilemmas.  Advances in neural information processing systems, 31, 2018.
  * [43] DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett.  Collaborating with humans without human data.  Advances in Neural Information Processing Systems, 34:14502–14515, 2021.
  * [44] Paul Tylkin, Goran Radanovic, and David C Parkes.  Learning robust helpful behaviors in two-player cooperative atari environments.  In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pages 1686–1688, 2021.
  * [45] Patrick M Pilarski, Andrew Butcher, Michael Johanson, Matthew M Botvinick, Andrew Bolt, and Adam SR Parker.  Learned human-agent decision-making, communication and joint action in a virtual reality environment.  arXiv preprint arXiv:1905.02691, 2019.
  * [46] Jacob W Crandall, Mayada Oudah, Tennom, Fatimah Ishowo-Oloko, Sherief Abdallah, Jean-François Bonnefon, Manuel Cebrian, Azim Shariff, Michael A Goodrich, and Iyad Rahwan.  Cooperating with machines.  Nature communications, 9(1):233, 2018.
  * [47] Christos Dimitrakakis, David C Parkes, Goran Radanovic, and Paul Tylkin.  Multi-view decision processes: the helper-ai problem.  Advances in neural information processing systems, 30, 2017.
  * [48] Goran Radanovic, Rati Devidze, David Parkes, and Adish Singla.  Learning to collaborate in markov decision processes.  In International Conference on Machine Learning, pages 5261–5270. PMLR, 2019.
  * [49] Stefanos Nikolaidis, Swaprava Nath, Ariel D. Procaccia, and Siddhartha S. Srinivasa.  Game-theoretic modeling of human adaptation in human-robot collaboration.  CoRR, abs/1701.07790, 2017.
  * [50] Andrew Y Ng, Stuart Russell, et al.  Algorithms for inverse reinforcement learning.  In Icml, volume 1, page 2, 2000.
  * [51] Michael A Goodrich, Alan C Schultz, et al.  Human–robot interaction: a survey.  Foundations and Trends® in Human–Computer Interaction, 1(3):203–275, 2008.
  * [52] Ralph Allan Bradley and Milton E Terry.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  Biometrika, 39(3/4):324–345, 1952.
  * [53] DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett.  Collaborating with humans without human data.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 14502–14515. Curran Associates, Inc., 2021.
  * [54] Chao Yu, Jiaxuan Gao, Weilin Liu, Botian Xu, Hao Tang, Jiaqi Yang, Yu Wang, and Yi Wu.  Learning zero-shot cooperation with humans, assuming humans are biased.  arXiv preprint arXiv:2302.01605, 2023.
  * [55] Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson.  Facmac: Factored multi-agent centralised policy gradients.  Advances in Neural Information Processing Systems, 34:12208–12221, 2021.
  * [56] Emanuel Todorov, Tom Erez, and Yuval Tassa.  Mujoco: A physics engine for model-based control.  In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033. IEEE, 2012.
  * [57] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In International conference on machine learning, pages 1861–1870. PMLR, 2018.
  * [58] Nuttapong Chentanez, Andrew Barto, and Satinder Singh.  Intrinsically motivated reinforcement learning.  Advances in neural information processing systems, 17, 2004.
  * [59] Andrew G Barto.  Intrinsic motivation and reinforcement learning.  Intrinsically motivated learning in natural and artificial systems, pages 17–47, 2013.
  * [60] David Abel, Will Dabney, Anna Harutyunyan, Mark K Ho, Michael Littman, Doina Precup, and Satinder Singh.  On the expressivity of markov reward.  Advances in Neural Information Processing Systems, 34:7799–7812, 2021.
  * [61] Jürgen Schmidhuber.  Formal theory of creativity, fun, and intrinsic motivation (1990–2010).  IEEE transactions on autonomous mental development, 2(3):230–247, 2010.
  * [62] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.  Openai gym, 2016.
  * [63] Sam Michael Devlin and Daniel Kudenko.  Dynamic potential-based reward shaping.  In Proceedings of the 11th international conference on autonomous agents and multiagent systems, pages 433–440. IFAAMAS, 2012.
  * [64] Andrew Y Ng, Daishi Harada, and Stuart Russell.  Policy invariance under reward transformations: Theory and application to reward shaping.  In Icml, volume 99, pages 278–287. Citeseer, 1999.
  * [65] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.

##  Appendix A PbRL Algorithm

###  A.1 PEBBLE Algorithm

PEBBLE is a PbRL algorithm that comprises of two key elements: pre-training
and relabeling experience buffer. To gather a wide range of experiences,
PEBBLE starts by using intrinsic motivation [58, 59, 60, 61] to pre-train the
policy, which optimizes the policy to increase the state entropy in order to
explore the environment effectively. Afterwards, PEBBLE uses the SAC
algorithm, a state-of-the-art off-policy RL algorithm, to further train the
policy. To ensure stability in the learning process, PEBBLE relabels all
experiences in the buffer when the reward model is updated.

###  A.2 Modified PEBBLE Algorithm for varying Access to Human Policy
experiments

Algorithm 2 Adapting Single Agent PbRL for Human-AI Team

Input: feedback frequency $K$, # queries per feedback session $M$, human
policy set: $|\Pi|={\pi_{H}}$, agent access budget: $b\sim[0,1]$

Initialize parameters of $Q_{\theta}$ and $\hat{r}_{\psi}$

Initialize a dataset of preferences $D_{h}\leftarrow\varnothing$

// EXPLORATION PHASE

$D_{\tau},\pi_{\phi}\leftarrow$ EXPLORE() in [15]

POLICY LEARNING PHASE

for each iteration do

// REWARD LEARNING PHASE

if new episode then

choose human policy $\pi_{H}$

end if

if iteration % $K$ == 0 then

for m in 1…$M$ do

$(\sigma^{0},\sigma^{1})\sim$ SAMPLE() in [15]

Query human partner for $y$

end for

for each gradient step do

Sample mini-batch $\\{{(\sigma^{0},\sigma^{1},y)}_{j}\\}^{D_{h}}_{j=1}\sim
D_{h}$

end for

Relabel entire replay buffer $D_{\tau}$ using $\hat{r}_{\psi}$

end if

REINFORCEMENT LEARNING PHASE

for each time-step $t$ do

if time-step $t\leq$ budget $b$ then

sample $a_{h}\sim\pi_{H}$

else

randomly sample $a_{h}\sim\mathcal{A}^{H}$

end if

Collect $s_{t+1}$ by taking $a_{t}\sim\pi_{\phi}(a_{t}|s_{t})$ and $a_{H}$

Store transitions $D_{\tau}\leftarrow
D_{\tau}\\{(s_{t},a_{t},{\color[rgb]{1,0,0}a_{h}},s_{t+1},\hat{r}_{\psi}(s_{t}))\\}$

end for

for each gradient step do

Sample random mini-batch $\\{{\tau_{j}}\\}^{D_{\tau}}_{j=1}\sim D_{\tau}$

Optimize $L^{SAC}_{critic}$ and $L^{SAC}_{actor}$ w.r.t. $\theta$ and $\phi$,
respectively [15]

end for

end for

###  A.3 Modified PEBBLE Algorithm for Human Flexibility experiments

Algorithm 3 Adapting Single Agent PbRL for Human-AI Team

Input: feedback frequency $K$, # queries per feedback session $M$, human
policy set: $|\Pi|$, agent access budget: $b={1}$

Initialize parameters of $Q_{\theta}$ and $\hat{r}_{\psi}$

Initialize a dataset of preferences $D_{h}\leftarrow\varnothing$

// EXPLORATION PHASE

$D_{\tau},\pi_{\phi}\leftarrow$ EXPLORE() in [15]

POLICY LEARNING PHASE

for each iteration do

// REWARD LEARNING PHASE

if new episode then

choose human policy $\pi_{H}\in|\Pi|$

end if

if iteration % $K$ == 0 then

for m in 1…$M$ do

$(\sigma^{0},\sigma^{1})\sim$ SAMPLE() in [15]

Query human partner for $y$

end for

for each gradient step do

Sample mini-batch $\\{{(\sigma^{0},\sigma^{1},y)}_{j}\\}^{D_{h}}_{j=1}\sim
D_{h}$

end for

Relabel entire replay buffer $D_{\tau}$ using $\hat{r}_{\psi}$

end if

REINFORCEMENT LEARNING PHASE

for each time-step $t$ do

sample $a_{h}\sim\pi_{H}$

Collect $s_{t+1}$ by taking $a_{t}\sim\pi_{\phi}(a_{t}|s_{t})$ and $a_{H}$

Store transitions $D_{\tau}\leftarrow
D_{\tau}\\{(s_{t},a_{t},{\color[rgb]{1,0,0}a_{h}},s_{t+1},\hat{r}_{\psi}(s_{t}))\\}$

end for

for each gradient step do

Sample random mini-batch $\\{{\tau_{j}}\\}^{D_{\tau}}_{j=1}\sim D_{\tau}$

Optimize $L^{SAC}_{critic}$ and $L^{SAC}_{actor}$ w.r.t. $\theta$ and $\phi$,
respectively [15]

end for

end for

Table 1: MA MuJoCO - Action Space

Environment | Human’s Action Space | AI Agent’s Action Space
---|---|---
Walker | Right Leg Joints | Left Leg Joints
Hopper | Thigh Rotor | Leg Rotor, Foot Rotor
HalfCheetah | Front Rotor | Back Rotor
Ant | Front Left Leg, Right Back Leg | Front Right Leg, Back Leg
Swimmer | First Rotor | Second Rotor

##  Appendix B Experiment Details

###  B.1 State Space and Action Space

We use the available MuJoCo implementation packaged by OpenAI Gym [62] to
instantiate our environments. We use the default state space representation
given by the package, and the same observation space is shared by both, the
human and the AI agents. The details about the action space are given in Table
1.

For MA Highway, we use the Highway environment packages provided by [11] to
create our custom multi-agent environments and the associated tasks. We use
the continuous action space consisting of (steering, throttle) for both the
agent cars.

###  B.2 Reward Architecture and Embedding Space

Following the implementation of [15, 13, 14] we implement reward model via a
neural network and bound the final output using a $\tanh$ activation function
: [-1, 1]. For all the experiments, the reward model has three hidden layers
with 256 neurons each followed by the prediction layer (with one neuron). We
use the ADAM optimizer for training the SAC actor-critic as well as the reward
model. The hyperparameters used for PEBBLE, RUNE & SURF are given in Tables 3,
4, and 5.

The input to the reward model is (state, action) tuple. As used previously
[14], we stack the state and action vectors and treat them as a single input
for the reward model.

###  B.3 Human Policy & Preference Oracles

We assume an oracle scripted human who uses the underlying reward to correctly
provide the binary feedback for their preferences. The feedback is given as
follows :

$y(\tau_{0},\tau_{1})=\begin{cases}0,&\text{if}\quad\sum_{i}\tilde{R_{h}}(\tau_{0})>\sum_{i}\tilde{R_{h}}(\tau_{1})\\\
1,&\text{if}\quad\sum_{i}\tilde{R_{h}}(\tau_{0})<\sum_{i}\tilde{R_{h}}(\tau_{1})\\\
\end{cases}$ (3)

where, $\tilde{R_{h}}$ is the environment reward being used as the human
preference reward function and $\tau_{0},\tau_{1}$ are the queried trajectory
pairs. Note that we work under the setting that the preference feedback is
binary and therefore if the trajectory returns are equal we uniformly pick a
preferred trajectory. This does not pose any problems with our chosen
benchmark domains as the underlying reward is dense and shaped [63, 64].

To construct the human policy oracles, we train a SAC agent treating each
environment as a single-agent environment, and retrieve the oracle policy for
the human’s action space from this SAC agent at the time of training the AI
agent.

The hand-crafted rewards for MA Highway - Right Overtake have been shown in
Table 2. Similarly, we construct the rewards for MA Highway - Middle & Left
Overtake domains.

Table 2: MA Highway - Right Overtake rewards. (x1,y1) is the position of AI agent and (x2,y2) is the position of the Human agent in the environment. Condition | Position | Return
---|---|---
Same Lane | x1 <x2 | -0.5
y1 >= y2 - 2 & y1 <= y2 + 2 | x1 >x2 | 1
Adjacent Lane | x1 <= x2 - 10 | -0.25
y1 >= 2& y2 <= 6 | x1 >= x2 + 10 | 0.5
| else | -1
Overtaking Lane | x1 <= x2 | 0.25
y1 >= 6 & y2 <= 10: | x1 >= x2 | 0.25

###  B.4 Implementation, Code and Compute

We use the publicly available implementation of [15, 13, 14] for the
implementation of SAC, PEBBLE, RUNE & SURF algorithms. Our implementation of
extending these PbRL algorithms to the multi-agent setting can be found in the
code given in the supplementary. All the experiments were run on an Intel(R)
Xeon(R) Gold 6258R CPU @ 2.70GHz, with Quadro RTX 8000 GPU.

##  Appendix C Hyperparameters

Table 3: PEBBLE Hyperparameters

Hyperparameter | Value | Hyperparameter | Value
---|---|---|---
Initial temperature | 0.1 | Hidden units per each layer | 1024
Segment length | 30 (MA Highway) | # of layers | 2
| 50 (MA MuJoCo) | Optimizer | Adam [65]
Learning rate | 0.0003 | Batch Size | 1024(DMControl)
Critic target update freq | 2 | Critic EMA $\tau$ | 0.005
$\beta_{1},\beta_{2}$ | (0.9, 0.999) | Discount $\gamma$ | 0.99
Frequency of feedback | 500 (MA Highway) | Maximum budget / | 800/25 (MA Highway)
| 10000 (MA MuJoCo) | # of queries per session | 1400/14 (MA MuJoCo)
|  | # of pre-training steps | 1000 (MA Highway)
# of ensemble models $N_{en}$ | 3 |  | 5000 (MA MuJoCo)

Table 4: SURF Hyperparameters Hyperparameter | Value
---|---
Unlabeled batch ratio $\mu$ | 4
Threshold $\tau$ | 0.95
Segment length | 15 (MA Highway)
| 50 (MA MuJoCo)
Loss weight $\lambda$ | 1
Table 5: RUNE Hyperparameters Hyperparameter | Value
---|---
Segment length | 20 (MA Highway)
| 50 (MA MuJoCo)