#  On the Utility of Accounting for Human Beliefs about AI Behavior in Human-
AI Collaboration

Guanghui Yu Robert Kasumba Chien-Ju Ho William Yeoh

###### Abstract

To enable effective human-AI collaboration, merely optimizing AI performance
while ignoring humans is not sufficient. Recent research has demonstrated that
designing AI agents to account for human behavior leads to improved
performance in human-AI collaboration. However, a limitation of most existing
approaches is their assumption that human behavior is static, irrespective of
AI behavior. In reality, humans may adjust their action plans based on their
observations of AI behavior. In this paper, we address this limitation by
enabling a collaborative AI agent to consider the beliefs of its human
partner, i.e., what the human partner thinks the AI agent is doing, and design
its action plan to facilitate easier collaboration with its human partner.
Specifically, we developed a model of human beliefs that accounts for how
humans reason about the behavior of their AI partners. Based on this belief
model, we then developed an AI agent that considers both human behavior and
human beliefs in devising its strategy for working with humans. Through
extensive real-world human-subject experiments, we demonstrated that our
belief model more accurately predicts humans’ beliefs about AI behavior.
Moreover, we showed that our design of AI agents that accounts for human
beliefs enhances performance in human-AI collaboration.

##  1 Introduction

The potential for human-AI collaboration is immense and spans various domains.
In healthcare, AI systems can identify diagnoses that might be overlooked by
human professionals (Cheng et al., 2016; Mobadersany et al., 2018). In
industrial manufacturing, robots work alongside human workers to enhance
efficiency and safety (Bauer et al., 2008; Sherwani et al., 2020). In workflow
productivity, virtual assistants can generate drafts for humans to refine and
finalize (Wilson and Daugherty, 2018). However, despite significant
improvements in AI performance over the past decade, designing AI agents to
optimize the overall performance of human-AI collaboration remains a
challenge.

In particular, optimizing the AI system in isolation is not sufficient to
enhance the performance of human-AI collaboration. Both the human and the AI
agent need to _coordinate_ by inferring the goals and intentions of their
counterpart and taking complementary actions. For example, in AI-assisted
decision-making, researchers have demonstrated that instead of optimizing AI
by itself, training AI to focus on improving in areas where humans typically
struggle can significantly enhance the performance of human-AI teams (Bansal
et al., 2021; Wilder et al., 2021). In human-AI collaboration, Carroll et al.
(2019) show that incorporating models of human behavior into the training of
the AI agent leads to higher collaborative performance compared to training
the agent to play with themselves through self-play (Silver et al., 2018).
These studies highlight the importance of integrating human behavior into the
design of collaborative AI. However, a key limitation of these research
efforts is that they mostly assume that human behavior remains static,
irrespective of the actions and behavior of the AI counterpart. In practice,
humans may modify their behavior in response to their beliefs about what AI
agents intend to do based on their observations of AI behavior.

Figure 1: Example task.

In this work, we argue that, in addition to incorporating human behavior,
designing an AI agent that account for humans’ beliefs about AI behavior could
significantly improve their collaborative performance. Consider a simple
illustrative example in Figure 1, where the human and AI agents need to go to
both goals $\\{G1,G2\\}$ to complete the task. Assume that the human prefers
$G2$ over $G1$ because it requires fewer steps. When designing a collaborative
AI that accounts for human behavior only, the AI would choose to go to $G1$,
leading to lower overall collaborative performance. However, if the human
would adjust their behavior based on their belief about AI behavior (e.g.,
they will avoid going to the same goal that the AI is going towards), we could
incorporate this knowledge into the design of the AI. For example, the AI can
choose to go to $G2$, anticipating that the human will account for that and go
to $G1$, leading to improved overall performance. This example underscores the
potential benefits of incorporating human beliefs about AI behavior in AI
design. Meanwhile, it also highlights the importance of developing accurate
models of human beliefs and incorporating them into the AI design.

In this work, we aim to examine the benefits of incorporating human beliefs
about AI behavior in the design of AI to improve human-AI collaboration. Our
work focuses on the setting where the human and AI agents need to coordinate
their goals (i.e., they need to complete different subtasks in order to
accomplish the overall task). More formally, we formulate the human-AI
collaboration environment as a multi-player goal-oriented Markov decision
process (MDP). We begin our investigation by developing models of human
beliefs regarding AI behavior. This modeling effort extends the level-$k$
reasoning framework (Stahl II and Wilson, 1994) to account for suboptimal
human behavior. Specifically, we first develop a _behavioral level-0_ model
that assumes agents take actions without considering the behavior of other
agents. We then enhance the belief model by introducing a _behavioral level-1_
model, which assumes humans interpret the behavior of another player as if the
other agent adheres to the _behavioral level-0_ model. With the models of
human behavior and beliefs established, we proceed to develop collaborative AI
agents that incorporate different assumptions about human models.

To examine our proposed approaches, we conduct extensive human-subject
experiments on two environments. To develop and assess the model of human
behavior (i.e., the behavioral level-$0$ model), we first train models of
human behavior using behavioral cloning on real-world human behavior and
evaluate their performance. For the development and assessment of the model of
human beliefs (i.e., the behavioral level-$1$ model), we conduct experiments
that present each participant with a trace of behavior by an agent and ask
them to infer the agent’s goal, examining whether our model leads to accurate
predictions of human inference on the agent’s goal. Moreover, based on the
belief model, we also explore the possibility of developing AI policies such
that humans can more easily infer the goals of AI from its actions. Finally,
utilizing the developed models of human behavior and beliefs, we conduct a
final set of human-subject experiments that pair each human participant with
an AI agent in a two-player coordination game. We assess the human-AI
collaborative performance for different designs of AI agents. Overall, our
results demonstrate the effectiveness of our developed model of human behavior
and beliefs. Furthermore, we show that AI agents accounting for models of
human behavior and beliefs achieve better collaborative performance with
humans, compared to AI agents that do not consider human beliefs or those that
disregard both human behavior and beliefs.

Contributions. The main contributions of our work can be summarized as
follows:

  * $\bullet$

We develop models of human beliefs, inspired by the level-$k$ framework, that
account for human suboptimality. Moreover, we have designed AI agents that can
take explicable action plans, making it easier for humans to infer AI goals
from observing AI actions.

  * $\bullet$

We have designed collaborative AI agents, considering both models of human
behavior and human beliefs about AI behavior, aiming to optimize the
performance of human-AI collaboration.

  * $\bullet$

Through extensive human-subject experiments with 1,690 participants, we
demonstrate that our belief models are effective and can help us design AI
policies that are explicable to humans. Moreover, our design of collaborative
AI that incorporates human beliefs about AI behavior improves the
collaborative performance when paired with real-world human participants. The
results demonstrate both the importance of incorporating human beliefs about
AI behavior in the design of collaborative AI and the effectiveness of our
approach.

##  2 Related Work

Our work joins the recent growing research in human-AI collaborations (Carroll
et al., 2019; Jaderberg et al., 2019; Bansal et al., 2021; Wilder et al.,
2021). Bansal et al. (2021) demonstrate that optimizing AI in isolation may
lead to suboptimal performance for human-AI collaboration. Wilder et al.
(2021) show that training AI systems to complement humans, performing better
in areas where humans struggle, leads to improved collaborative performance.
Carroll et al. (2019) illustrate that incorporating a human model, learned
from human data, into the training of AI results in enhanced performance when
these AI systems work with real humans. Additionally, Jaderberg et al. (2019)
use population-based reinforcement learning to improve the robustness of
trained AI agents. Our work extends this line of work by proposing to
incorporate not only human behavior but also human beliefs of AI behavior into
the design of collaborative AI.

From a technical perspective, our work involves understanding and modeling
humans in decision-making. There has been a significant amount of work in the
literature on modeling human behavior. For example, Inverse Reinforcement
Learning (IRL)(Ng et al., 2000; Abbeel and Ng, 2004; Ramachandran and Amir,
2007) aims to infer the reward functions in Markov Decision Processes (MDPs)
through observing demonstrations of the optimal policy. If the demonstrator is
a human being, the demonstrations could be noisy or contain behavioral biases.
Studies (Evans et al., 2016; Shah et al., 2019; Hughes et al., 2020; Zhi-Xuan
et al., 2020) have aimed to incorporate human behavioral biases in the
inference process and infer both the rewards and biases simultaneously.
Imitation Learning (Hussein et al., 2017) also aims at developing models that
can mimic human behavior from demonstrations. Recently, there have also been
an increasing amount of research efforts that incorporates human models in
computational and machine learning frameworks Kleinberg and Oren (2014); Tang
and Ho (2019, 2021); Masters et al. (2021a, b); Yu and Ho (2022); Yu et al.
(2023); Feng et al. (2024).

From the perspective of modeling human beliefs over others’ behavior, this has
been discussed in the literature of level-$k$ reasoning (Stahl II and Wilson,
1994; Gill and Prowse, 2016) in economics and theory of mind (Premack and
Woodruff, 1978; Westby and Riedl, 2023). As a few examples, Yu et al. (2006)
have found that real human behavior is close to level-$1$ and level-$2$
reasoning models in cooperative games. Albrecht and Stone (2018) survey works
on autonomous agents modeling the beliefs and intentions of other agents. They
distinguish methods for modeling stationary or changing agent behaviors. The
belief model includes theory of mind, recursive reasoning models, plan
recognition, partially observable Markov decision processes (POMDPs), and
others. Baker et al. (2011) propose a Bayesian theory of mind to describe
human modeling of joint belief of state in a partially observable MDP and
conduct human experiments in (Baker et al., 2009, 2017). A similar human
modeling approach is adopted by Wu et al. (2021) and applied to the Overcooked
experiment. Their idea is to split delivering a dish into several subtasks,
such as picking up an ingredient or putting an ingredient into a pot, and the
player is inferring others’ subtasks and selecting their own corresponding
subtask before taking actions.

##  3 Problem Formulation and Methods

In this section, we first formulate the human-AI collaboration framework as a
multi-player goal-oriented Markov decision process. We then outline our
methods for modeling human behavior and beliefs about AI behavior. Finally, we
present our approaches for integrating human behavior and beliefs into the
development of collaborative AI agents.

###  3.1 Decision-Making Environment

We formulate the human-AI cooperative decision-making environment as a multi-
agent Markov decision process (MDP), represented by $W=\langle
S,\alpha,{A_{i}|_{i\in[\alpha]}},R,P\rangle$. $S$ is a finite set of states;
$\alpha$ is a finite set of players; $A_{i}$ is the action set available to
player $i$. $P:S\times A_{1}\times\dots\times A_{|\alpha|}\times
S\rightarrow[0,1]$ is the transition function that determines the next state
given all players’ joint actions. $R:S\times A_{1}\times\dots\times
A_{|\alpha|}\rightarrow\mathbb{R}$ is the joint reward function assigned to
all players given their joint actions. Note that in this work we consider the
cooperative setting. Therefore, our formulation only incorporates a single
reward function $R$ for all players, though this can be easily extended.

While our formulation could address cases with multiple human and AI players,
in this work, we focus on a two-player cooperative game, i.e.,
$\alpha=\\{1,2\\}$, where one player is a human and the other is the AI.
During the decision-making process, neither the human nor the AI knows the
other player’s next action or future plans, and they cannot communicate
directly. However, they can observe each other’s past actions, enabling them
to infer about the other player and modify their own actions accordingly.

Goal-oriented MDP. In this work, we focus on the setting with goal-oriented
MDP, where there is a set of goals $G=\\{g_{1},\dots,g_{k}\\}\subseteq S$,
which are subset of states that are terminal states, i.e.,
$P(g|g,a)=1~{}\forall g\in G,a\in A$. Moreover, the decision-making agent only
receives rewards when arriving at one of the goal states.

###  3.2 Modeling Human Behavior and Human Beliefs about AI Behavior

Our models are motivated by the level-$k$ framework (Stahl II and Wilson,
1994) in economics. In particular, we start by considering humans as level-$0$
agents that do not account for others’ behavior in the environment. Differing
from the literature, we address the natural situation that level-$0$ agents do
not behave optimally, and we call this model _behavioral level- $0$_ agents.
To account for human beliefs,111In this work, _human beliefs_ refer to humans’
belief about the goal of the AI agent. we model humans as an extension of the
level-$1$ agents, that assume other agents are _behavioral_ level-$0$
agents222Our model can iteratively progress to higher level-$k$ agents.
However, we focus on the case with $k\leq 1$. and update their beliefs in a
Bayesian manner based on the observations of others’ behavior.

Modeling human behavior. We first model human behavior under the assumption
that humans do not consider other players in the environment (or that they
consider other players as a part of the environment without strategically
responding). In this case, a human behavior model can be represented as
$H:W\rightarrow\Pi$, mapping a given environment $w\in W$ to a policy
$\pi=H(w)$. We give two examples of human behavior models utilized in our work
below.

  * $\bullet$

_Standard model._ First consider the standard human behavior model in MDPs, in
which the goal of the human is to maximize the expected cumulative reward, and
their policy only depends on the current state. The model can be represented
by $\pi(a|s)$, indicating the probability of choosing action $a$ at state $s$.
For the standard model that assumes decision optimality, humans choose actions
maximizing the Q-function, where $Q(s,a)$ indicates the expected cumulative
reward if the player takes action $a$ in state $s$ and follows policy $\pi$.
$Q(s,a)$ could be calculated by standard reinforcement learning techniques
such as value iteration or Q-learning.

  * $\bullet$

_Behavioral level- $0$ model._ We also consider the case that we can learn
human behavioral models from their historical behavioral traces through
behavioral cloning (Pomerleau, 1988; Torabi et al., 2018). Behavioral cloning
is one of the imitation learning approaches, which learns a policy from human
demonstration by building a map from states to actions with supervised
learning methods Bain and Sammut (1995). We build a fully connected neural
network, where the input is the state encoding, and output is the probability
over action space, and train the model with standard gradient descent method
with cross-entropy loss.

Modeling human beliefs. We now describe how we develop models for human
beliefs. As summarized earlier, we consider cases where human decision-makers
assume that other agents in the environments are behavioral level-$0$ agents.
In our discussion, we also describe this belief model as _behavioral level-
$1$ agents_.

More specifically, we utilize Bayesian inference to model the human belief
updating process, as in Equation (1), where $\lambda(g)$ represents the prior
distribution of goals, and $Pr(s_{t},a_{t}|g)$ denotes the probability of
observing $(s_{t},a_{t})$ given the goal, according to the policy model
$\pi(a|s,g)$. This policy represents how humans perceive the actions of other
players. If a human believes the other agent is following the standard model,
then the policy is derived from the optimal policy,
$\pi(a|s,g)\propto\exp(\beta Q(s,a|g))$ 333$\beta\geq 0$ controls the level of
optimality. When $\beta=\infty$, $\pi(a|s,g)=1$ if
$a=argmax_{a^{\prime}}Q(s,a^{\prime}|g)$ and $\pi(a|s,g)=0$ otherwise. ; if,
however, humans believe the agent follows a data-driven model, then the policy
will be the output of the human model, $\pi=H(w|g)$.

$\displaystyle
B(g|(s,a)_{1:t})\propto\lambda(g)Pr((s,a)_{1:t}|g)=\lambda(g)\prod_{1:t}\pi(a_{i}|s_{i},g)P(s_{i}|s_{i-1},a_{i-1})$
(1)

###  3.3 Designing Collaborative AI Agents

To illustrate the effectiveness of incorporating models of human beliefs, we
train different AI agents that work with humans in human-AI collaboration via
simulations and real-world human subject experiments.

Training methodology. The main idea of our training method is through self-
play. We incorporate the models of humans and have AI teammates play with the
agents specified by the human models through simulated plays. We use proximal
policy optimization (PPO) to train collaborative AI agents.

Collaborative AI agents. Using the methodology above, we have designed several
AI agents based on different assumptions of the behavior of human
counterparts.

  * $\bullet$

_Assuming humans are optimal._ We first train an AI agent that learns to
collaborate with itself through self-play. This is equivalent to assuming the
human counterpart is acting optimally.

  * $\bullet$

_Assuming humans are behavioral level- $0$ agents._ We next train an AI agent
that assumes the human is a behavioral level-$0$ agent, using behavioral
cloning to train the behavioral model.

  * $\bullet$

_Incorporating models of human behavior and beliefs._ Finally we also train an
AI agent that incorporates both the models of human behavior and beliefs into
the design of AI agents.

The details of the setup and implementations are included in the appendix.

##  4 Experiments

To evaluate our approaches, we have conducted three sets of experiments,
consisting of both simulations and human-subject experiments. For our human-
subject experiments, we have recruited in total $1,690$ participants from
Amazon Mechanical Turk (MTurk) for multiple sets of experiments. The
experiments are approved by the IRB of our institution. Workers were paid $1
base payments with the potential for bonus payments in some experiments. The
average hourly rate is approximately $\$14$ across all our experiments.

###  4.1 Experiment Environment: Grid World with Two Players and Multiple
Goals

Our experiments are conducted in a grid world environment with two players and
multiple goals. We have conducted two variations of the experiments. In the
first variation, we designed the environments such that the two players are
not playing in the same grid world. However, each of them has access to the
full information of the environment and the actions of the other player. This
variation abstracts away the interdependency of agent actions, meaning the
agents’ actions do not influence each other directly. This allows us to focus
on how humans and AI reason about each other’s goals and intentions. Note that
we obtained qualitatively similar results in this first variation and the
second, more complicated variation of environments described next. For
simplicity, we have included the setup and results of the first variation in
the appendix. Below, we describe the experiment environment and results of our
second variation.

More specifically, as shown in Figure 2, in our second variation of the
environment, the grid world is 8 by 8 in size, containing the positions of
both players and four possible goals. The players can choose to move {Up,
Down, Right, Left} or stay in the current grid. They can see each other’s
positions and take actions simultaneously. Each player can navigate to two of
the four goals. In particular, the human player can only reach one of the two
goals colored blue, and the AI player can only reach one of the goals colored
green. When both agents reach the same type of goal (both reaching a "star" or
both reaching a "triangle"), they earn positive points. However, they will not
earn points if they reach different goals or if they collide into each other
(move into the same position). We set the maximum number of actions to be 20.

(a) The interface for experiment 1 & 3.

(b) The interface for experiment 2.

Figure 2: The interface for our human-subject experiment. In experiment 1,
each participant is playing with an AI agent. The participants in experiment 1
are told what their goal is and only need to focus on reaching the goal
without colliding with the AI agent. In experiment 2, each participant is
provided traces of the behavior by other agents and is asked to infer which
goal one agent is trying to reach. Finally, in experiment 3, the participants
are not told which goal to reach, and they need to make decisions based on
their beliefs over AI behavior. In experiments 1 and 3, the participants can
only receive bonus rewards by reaching the same type of goals (star or
triangle) as the AI agent.

In the first experiment, we recruit real-world humans to play against another
pre-defined AI agent. We then examine the effectiveness of utilizing
behavioral cloning as a model to mimic human behavior. Note that using
behavioral cloning to model human behavior is extensively adopted in the
literature, and the purpose of the first experiment serves as the foundation
to develop our human belief models and the design of collaborative AI agents.

To evaluate our proposed approaches in modeling human beliefs and designing
cooperative AI, we have designed and conducted additional two sets of
experiments. In the second set of experiments, we provide participants with
different traces of actions from other agents, and ask participants to infer
the goal of the agents. This experiment helps us evaluate whether our belief
model leads to better predictions of human beliefs over others’ behavior.
Afterwards, in the third experiment, we conduct experiments similar to the
first experiment. We team up AI agents and human players to examine the team
performance of our design of collaborative AI when working with real-world
human participants. But unlike the first experiment, we don’t reveal any
information about goals the players are trying to reach.

###  4.2 Experiment 1: Training and Evaluating Human Behavior Models

In our first experiment, our goal is to examine the effectiveness of utilizing
behavioral cloning to model human behavior. Note that his approach has been
proposed and examined in various domains Carroll et al. (2019). Our main
purpose to conduct this experiment is to ensure whether this method works well
in our setting, as our proposed approaches in developing human belief models
and collaborative AI are built on top of models of human behavior.

Experiment setup. We recruited 190 workers from Amazon Mechanical Turk. Each
recruited worker was asked to play 30 navigation games within a grid world
with a pre-defined AI model, as shown in Figure 2a. To collect diverse data,
we designed three AI models to play with humans: random AI (selecting a random
goal and taking the shortest path), self-play AI (trained using self-play),
and optimal AI (optimized via joint optimization).

Note that in this experiment, our goal is to develop a model of human behavior
that outputs a sequence of actions given a goal. Therefore, in experiment 1,
we provided additional instructions to participants about which goal they
should reach in each round, and the AI models controlled the other player to
move towards the same type of goals. In our later experiment 3, we did not
provide these instructions, and humans needed to infer the goal of the AI
agent.

Evaluation of data-driven behavior models. We divided the collected data of
human actions into three sets: training, validation, and testing. The training
set comprised data from 152 workers, including approximately 136,000 instances
of user decisions, while the validation and testing sets each contained data
from 19 workers, amounting to around 17,000 instances of user decisions each.
We employed a 4-layer Multilayer Perceptron (MLP) model, where the input is
the current environment layout, and the output predicts the next human action.
We fine-tuned the hyperparameters, such as learning rate, hidden layer size,
and L2 penalty, based on validation errors.

Our results suggest that human behavior is noisy and is significantly away
from being optimal. Even when human participants are provided with suggested
goals during data collection, there is only a $55.0\%$ chance that both
players will reach the same type of goals across all treatments. Besides,
there is a notable chance of two players colliding (about $15.8\%$) or ending
in different types of goals (about $25.6\%$). We compared the performance of
our learned model to a model predicated on optimal agent behavior, defined as
taking the shortest path to the goal. The training, validation, and test
accuracies of both models are presented in Table 1. These results clearly
reveal that human behavior deviates significantly from the assumed optimality.
This deviation highlights the importance of incorporating a realistic model of
human behavior into human-AI cooperation frameworks.

Table 1: The prediction accuracy for human behavior assuming optimal behavior and using data-driven model in Experiment 1. | Training Accuracy | Validation Accuracy | Testing Accuracy
---|---|---|---
Assuming Optimal Behavior | 0.4498 | 0.4327 | 0.4459
Data-Driven Model | 0.8547 | 0.7831 | 0.7899

###  4.3 Experiment 2: Evaluating Human Belief Models

We next examine our model of human beliefs. Specifically, we investigate
whether we can design AI behavior such that it is easier for humans to infer
the goal of the AI agent.

In our experiments with a simplified environment, as included in the appendix,
we directly examined how accurately our belief model can infer human beliefs
by comparing the predictions of our models with the direct solicitation of
users’ inferences about others’ behavior. The results suggest that our model
is more accurate in predicting human beliefs compared to baselines. However,
human beliefs are generally very noisy, and if we randomly draw a behavior
trace from another agent and ask humans to predict the goal, human prediction
accuracy is close to random guessing. Therefore, we next shift our focus to
whether AI agents can design their action plans to make it easier for humans
to infer the goal. In this more complicated environment, we have directly
examined this follow-up question of designing AI policies that are explicable
to humans.

Experiment setup. The experiment setup is presented in Figure 2b. In addition
to player positions and goal positions, we also display behavioral traces,
which are sequences of actions taken by previous players. For each
participant, we show them the traces of two players and ask the participant to
infer which one of the two goals those agents are trying to reach.

We compare two belief models. The first one is the standard level-1 model:
humans assume the other agent is a level-0 agent that takes the optimal
decision (i.e., taking the shortest path towards the goal). The second one is
our proposed _behavioral level-1_ model: humans assume the other agent is also
a suboptimal decision-maker, taking actions in the same way as themselves. In
our implementation, we leverage the data-driven behavioral model derived from
Experiment 1 as the user behavioral model. More concretely, we construct a
belief model using Bayesian inference, as described in Section 3.2, using the
corresponding human model of behavior.

Explicable AI policy. As mentioned earlier, humans generally struggle to infer
the goals of other agents from behavioral traces randomly drawn from history
(see the appendix for detailed results). Therefore, in this experiment,
instead of displaying a randomly drawn historical behavioral trace, we display
the behavioral trace of an AI agent that aims to make its behavior
_explicable_. In particular, based on the developed belief models, we train AI
agents to not only achieve their goals but also to maximize the likelihood
that humans can accurately infer these goals from their behavior. Our
implementation utilizes self-play and awards an additional bonus when the goal
inference aligns with the predictions of a human belief model, in addition to
the rewards for goal achievement. The bonus is proportional to the log
likelihood of belief model inference $log(\pi(a|s,g))$. By conducting
experiments with this explicable AI policy, we simultaneously evaluate whether
our belief models are accurate and whether our design of explicable AI policy
is effective.

Figure 3: The simulation results in Experiment 2.

Simulations. We design the AI agents to incorporate the belief models of
standard level-1 agents and behavioral level-1 agents, enabling them to adopt
policies that simplify the task of goal inference for humans. For each
environment with a given goal, we generated the behavioral traces of both AI
policies. The goal is to examine whether the explicable AI with our belief
model makes it easier for humans to infer the goal of the AI. We first
conducted simulations assuming humans are behavioral level-1 agent. The
results are shown in Figure 3. The explicable AI equipped with the belief
model of behavioral level-1 agents leads to behavioral traces that are easier
to infer. Note that this result is not surprising is not surprising since we
assume humans infer the goal following the beliefs of behavioral level-1
agent. The results provide evidence that our design of explicable AI is
effective.

Figure 4: The results of human-subject experiments in Experiment 2.

Human-subject experiments. We next conducted human-subject experiments to
assess whether our approach results in actions that make it easier for real
humans to infer the goal. We recruited 200 workers from Amazon Mechanical
Turk, randomly assigning them to one of two treatments. Each participant was
tasked with identifying the goals of the player in 30 different scenarios. The
length of actions is drawn from the range 4 to 8. To incentivize effort,
participants were awarded a $\$0.03$ bonus for each correctly identified goal.
The results, as shown in Figure 4, demonstrate that humans are better at
inferring the goal of the explicable AI agent, that assumes humans are
behavioral level-1 agents. The difference is statistical significant with
$p>0.0001$. This comparison demonstrates that when coupled with the more
accurate human belief models, we can indeed design AI policy to be explicable,
i.e., making it easier for humans to infer AI goals from observing AI actions.

###  4.4 Experiment 3: Evaluating the Design of Collaborative AI Agents

We now examine our design of collaborative AI agents. As described in Section
3.3, we train collaborative AI agents using three different corresponding
human models and utilize both simulations and human-subject experiments to
examine the effectiveness of our design.

Experiment setup. The setup of our Experiment 3 is the same as Experiment 1,
shown in Figure 2a. But human participants will not receive any hint about
which goal their teammate or themselves are suggested to reach.

Simulations. We first run simulations to evaluate the performance of human-AI
teams with different design of collaborative AI. The evaluation is based on
$10,000$ randomly generated environments. However, we further filter out cases
where the distance between the two goals for the same player is smaller than 3
to encourage models to adjust their behavior based on the inference of their
teammate’s actions. We examine the performance of three designs of
collaborative AI using the methodology described in Section 3.3:

  * $\bullet$

_Self-Play_ is the AI that is trained assuming they are playing with itself.

  * $\bullet$

_Behavior-AI_ is the AI that assumes the human partner is the behavioral
level-$0$ agent.

  * $\bullet$

_Belief-AI_ is the AI that assumes the human partner is the behavioral
level-$1$ agent.

We partner the collaborative AI agents with the three simulated human agents
and measure the collaborative performance. The simulation results, shown in
Table 2, highlight the importance of selecting appropriate human models for
training collaborative AI. When an AI agent is paired with the human model
that is used to train the AI, the collaborative performance is better compared
to pairing with other models.

Table 2: Simulation results of collaborative performance over 10k testing cases in experiment 3. Column players are different AI agents, and row players are different simulated human models. Human Model | AI Agent
---|---
Self-play | Behavior-AI | Belief-AI
Self-play AI | 0.6245 | 0.5250 | 0.6177
Behavior Model | 0.4902 | 0.6334 | 0.5755
Behavior&Belief | 0.6411 | 0.6574 | 0.7675
Figure 5: The average human-AI performance in our human-subject experiment in
Experiment 3.

Human-subject experiments. To examine the performance of our collaborative AI
design when pairing with real humans, we recruited 200 workers and divided
them into three treatment groups, each interacting with a different design of
AI. Participants were tasked with playing 30 games, preceded by three tutorial
games designed to familiarize them with the gameplay. Participants can receive
a $0.05 bonus payment for reaching the same goal with the AI agent for each of
the 30 tasks.

Figure 5 presents the average collaborative reward, highlighting that AI
trained with models of human behavior and beliefs achieved the highest
collaborative performance when working with real humans. This outcome not only
validates our design of AI but also suggests that real humans’ actions align
with the behavior and belief models in this environment. Statistical analysis
shows significant differences in the performance of AI models paired with
humans, with $p<0.0001$ for comparisons between self-play and AI accounting
for human behaviors, and a $p$ value of $0.0014$ for comparisons between AI
accounting for human behaviors versus accounting for both behavior and belief.
These results demonstrate that incorporating human beliefs into the deign of
AI agents enhances collaborative performance when working with real humans.

##  5 Conclusion

Our work explores the impact of incorporating human behavior and beliefs into
the design of collaborative AI, aiming to improve collaboration between humans
and AI. By developing models that account for human beliefs regarding AI
actions and integrating these models into AI design, we have observed improved
performance in human-AI collaboration. Our approach, validated through
simulations and human-subject experiments, demonstrates that AI agents
designed with an understanding of human behavior and beliefs can be more
effective in working with humans. This suggests a potential path forward for
creating AI systems that are better aligned with human partners, making
collaborative tasks more efficient and intuitive.

Generalization and limitations. We have demonstrated the effectiveness of our
approach through human-subject experiments in commonly used grid world
environments with varying levels of complexity. However, as with most human-
subject studies, our findings are limited to the chosen environments.
Moreover, our approach leverages historical data to construct models of human
behavior and beliefs. This approach implicitly assumes that both human
behavior and beliefs remain unchanged over time. Therefore, it is important to
improve the development of human models and examine their generalizability to
other environments.

Societal impacts. Our approach highlights the importance of incorporating not
only human behavior but also human beliefs about AI behavior in the design of
AI for human-AI collaboration. We believe this will become increasingly
important as AI capabilities grow. However, while our research focuses on
enhancing human-AI collaboration, there is a potential for the same methods to
be used negatively, such as designing AI that intentionally sabotages human
utility. One approach to mitigate potential negative impacts is to increase
the transparency of AI models, enabling humans to develop appropriate reliance
when working with AI agents.

## Acknowledgements

This work is supported in part by J.P. Morgan Faculty Research Award and a
Global Incubator Seed Grant from McDonnell International Scholars Academy.

## References

  * Abbeel and Ng [2004] Pieter Abbeel and Andrew Y Ng.  Apprenticeship learning via inverse reinforcement learning.  In _Proceedings of the International Conference on Machine learning_ , page 1, 2004.
  * Albrecht and Stone [2018] Stefano V Albrecht and Peter Stone.  Autonomous agents modelling other agents: A comprehensive survey and open problems.  _Artificial Intelligence_ , 258:66–95, 2018.
  * Bain and Sammut [1995] Michael Bain and Claude Sammut.  A framework for behavioural cloning.  In _Proceedings of Machine Intelligence 15_ , pages 103–129, 1995.
  * Baker et al. [2011] Chris Baker, Rebecca Saxe, and Joshua Tenenbaum.  Bayesian theory of mind: Modeling joint belief-desire attribution.  In _Proceedings of the Annual Meeting of the Cognitive Science Society_ , volume 33, 2011.
  * Baker et al. [2009] Chris L Baker, Rebecca Saxe, and Joshua B Tenenbaum.  Action understanding as inverse planning.  _Cognition_ , 113(3):329–349, 2009.
  * Baker et al. [2017] Chris L Baker, Julian Jara-Ettinger, Rebecca Saxe, and Joshua B Tenenbaum.  Rational quantitative attribution of beliefs, desires and percepts in human mentalizing.  _Nature Human Behaviour_ , 1(4):1–10, 2017.
  * Bansal et al. [2021] Gagan Bansal, Besmira Nushi, Ece Kamar, Eric Horvitz, and Daniel S Weld.  Is the most accurate AI the best teammate? optimizing AI for teamwork.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 11405–11414, 2021.
  * Bauer et al. [2008] Andrea Bauer, Dirk Wollherr, and Martin Buss.  Human–robot collaboration: a survey.  _International Journal of Humanoid Robotics_ , 5(01):47–66, 2008.
  * Carroll et al. [2019] Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan.  On the utility of learning about humans for human-AI coordination.  In _Proceedings of the Advances in Neural Information Processing Systems_ , 2019.
  * Cheng et al. [2016] Yu Cheng, Fei Wang, Ping Zhang, and Jianying Hu.  Risk prediction with electronic health records: A deep learning approach.  In _Proceedings of the SIAM International Conference on Data Mining_ , pages 432–440, 2016.
  * Evans et al. [2016] Owain Evans, Andreas Stuhlmüller, and Noah Goodman.  Learning the preferences of ignorant, inconsistent agents.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2016.
  * Feng et al. [2024] Yiding Feng, Chien-Ju Ho, and Wei Tang.  Rationality-robust information design: Bayesian persuasion under quantal response.  In _Proceedings of the ACM-SIAM Symposium on Discrete Algorithms_ , 2024.
  * Gill and Prowse [2016] David Gill and Victoria Prowse.  Cognitive ability, character skills, and learning to play equilibrium: A level-k analysis.  _Journal of Political Economy_ , 124(6):1619–1676, 2016.
  * Hughes et al. [2020] Dana Hughes, Akshat Agarwal, Yue Guo, and Katia Sycara.  Inferring non-stationary human preferences for human-agent teams.  In _Proceedings of the IEEE International Conference on Robot and Human Interactive Communication_ , 2020.
  * Hussein et al. [2017] Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne.  Imitation learning: A survey of learning methods.  _ACM Computing Surveys_ , 50(2):1–35, 2017.
  * Jaderberg et al. [2019] Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al.  Human-level performance in 3d multiplayer games with population-based reinforcement learning.  _Science_ , 364(6443):859–865, 2019.
  * Kleinberg and Oren [2014] Jon Kleinberg and Sigal Oren.  Time-inconsistent planning: a computational problem in behavioral economics.  In _Proceedings of the ACM Conference on Economics and Computation_ , pages 547–564, 2014.
  * Masters et al. [2021a] Peta Masters, Michael Kirley, and Wally Smith.  Extended goal recognition: a planning-based model for strategic deception.  In _Proceedings of the International Conference on Autonomous Agents and Multiagent Systems_ , 2021a.
  * Masters et al. [2021b] Peta Masters, Wally Smith, and Michael Kirley.  Extended goal recognition: Lessons from magic.  _Frontiers in Artificial Intelligence_ , 4, 2021b.
  * Mobadersany et al. [2018] Pooya Mobadersany, Safoora Yousefi, Mohamed Amgad, David A Gutman, Jill S Barnholtz-Sloan, José E Velázquez Vega, Daniel J Brat, and Lee AD Cooper.  Predicting cancer outcomes from histology and genomics using convolutional networks.  _Proceedings of the National Academy of Sciences_ , 115(13):E2970–E2979, 2018.
  * Ng et al. [2000] Andrew Y Ng, Stuart J Russell, et al.  Algorithms for inverse reinforcement learning.  In _Proceedings of the International Conference on Machine Learning_ , 2000.
  * Pomerleau [1988] Dean A Pomerleau.  Alvinn: An autonomous land vehicle in a neural network.  In _Proceedings of the Advances in Neural Information Processing Systems_ , 1988.
  * Premack and Woodruff [1978] David Premack and Guy Woodruff.  Does the chimpanzee have a theory of mind?  _Behavioral and Brain Sciences_ , 1(4):515–526, 1978.
  * Ramachandran and Amir [2007] Deepak Ramachandran and Eyal Amir.  Bayesian inverse reinforcement learning.  In _Proceedings of the International Joint Conference on Artificial Intelligence_ , 2007.
  * Shah et al. [2019] Rohin Shah, Noah Gundotra, Pieter Abbeel, and Anca Dragan.  On the feasibility of learning, rather than assuming, human biases for reward inference.  In _Proceedings of the International Conference on Machine Learning_ , 2019.
  * Sherwani et al. [2020] Fahad Sherwani, Muhammad Mujtaba Asad, and Babul Salam Kader K Ibrahim.  Collaborative robots and industrial revolution 4.0 (IR 4.0).  In _Proceedings of the International Conference on Emerging Trends in Smart Technologies_ , pages 1–5, 2020.
  * Silver et al. [2018] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al.  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  _Science_ , 362(6419):1140–1144, 2018.
  * Stahl II and Wilson [1994] Dale O Stahl II and Paul W Wilson.  Experimental evidence on players’ models of other players.  _Journal of Economic Behavior & Organization_, 25(3):309–327, 1994.
  * Tang and Ho [2019] Wei Tang and Chien-Ju Ho.  Bandit learning with biased human feedback.  In _Proceedings of the International Conference on Autonomous Agents and Multiagent Systems_ , 2019.
  * Tang and Ho [2021] Wei Tang and Chien-Ju Ho.  On the bayesian rational assumption in information design.  In _Proceedings of the AAAI Conference on Human Computation and Crowdsourcing_ , 2021.
  * Torabi et al. [2018] Faraz Torabi, Garrett Warnell, and Peter Stone.  Behavioral cloning from observation.  _arXiv preprint arXiv:1805.01954_ , 2018.
  * Westby and Riedl [2023] Samuel Westby and Christoph Riedl.  Collective intelligence in human-AI teams: A Bayesian theory of mind approach.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 6119–6127, 2023.
  * Wilder et al. [2021] Bryan Wilder, Eric Horvitz, and Ece Kamar.  Learning to complement humans.  In _Proceedings of the International Conference on International Joint Conferences on Artificial Intelligence_ , pages 1526–1533, 2021.
  * Wilson and Daugherty [2018] H James Wilson and Paul R Daugherty.  Collaborative intelligence: Humans and AI are joining forces.  _Harvard Business Review_ , 96(4):114–123, 2018.
  * Wu et al. [2021] Sarah A Wu, Rose E Wang, James A Evans, Joshua B Tenenbaum, David C Parkes, and Max Kleiman-Weiner.  Too many cooks: Bayesian inference for coordinating multi-agent collaboration.  _Topics in Cognitive Science_ , 13(2):414–432, 2021.
  * Yu and Ho [2022] Guanghui Yu and Chien-Ju Ho.  Environment design for biased decision makers.  In _Proceedings of the International Joint Conference on Artificial Intelligence_ , 2022.
  * Yu et al. [2023] Guanghui Yu, Wei Tang, Saumik Narayanan, and Chien-Ju Ho.  Encoding human behavior in information design through deep learning.  In _Proceedings of the Advances in Neural Information Processing Systems_ , 2023.
  * Yu et al. [2006] Jianming Yu, Gael Pressoir, William H Briggs, Irie Vroh Bi, Masanori Yamasaki, John F Doebley, Michael D McMullen, Brandon S Gaut, Dahlia M Nielsen, James B Holland, et al.  A unified mixed-model method for association mapping that accounts for multiple levels of relatedness.  _Nature Genetics_ , 38(2):203–208, 2006.
  * Zhi-Xuan et al. [2020] Tan Zhi-Xuan, Jordyn Mann, Tom Silver, Josh Tenenbaum, and Vikash Mansinghka.  Online bayesian goal inference for boundedly rational planning agents.  In _Proceedings of the Advances in Neural Information Processing Systems_ , 2020.

##  Appendix A Additional Experiments

Besides experiments in Section 4, we conduct additional experiments in a
simple grid world environment with multiple goals to illustrate our study of
humans’ beliefs about AI behavior.

###  A.1 Experiment Environment: Grid World with Multiple Goals

We use grid worlds of size of $6\times 6$ in both simulations and human
experiments. Similar to the environment setup in Section 4.1, the grid world
contains a start position, two goal positions, and some blocked positions that
the player cannot enter. The player needs to move from the start position
towards one of the goal positions. The player can choose to move {Up, Down,
Right, Left}. The player will get a positive reward upon reaching the goal,
and we set the maximum number of actions to be 20.

We first recruit participants to engage in a single-player game and record
their behavior. We then build human behavior model using behavioral cloning.
To evaluate our proposed approaches in modeling human beliefs and designing
cooperative AI, we conducted additional two sets of experiments. In the second
set of experiments, we provide participants with different traces of actions
from other agents, again in a single-player game, and ask participants to
infer the goal of the agents. This experiment helps us evaluate whether our
belief model leads to better predictions of human beliefs over others’
behavior. Afterwards, we conduct a two-player game in the third experiment,
where humans are paired with different AI agents to examine the team
performance of our design of collaborative AI.

The interfaces of our experiments can be seen in Figure 6a, 6b, and 6c. The
detailed descriptions of the experiment and interfaces are included in
Appendix C.

(a) Experiment 4.

(b) Experiment 5.

(c) Experiment 6.

Figure 6: Human-subject experiment interfaces. In Experiment 4, each
participant is asked to control the player to move to the goal (red star). In
Experiment 5, each participant is provided a trace of the behavior by another
agent, and is asked to infer which goal the agent is trying to reach. In
Experiment 6, each participant is playing with an AI agent in separate
environments. The participants only receive bonus rewards by reaching the same
goal (red star or green triangle) as the AI agent.

###  A.2 Experiment 4: Training and Evaluating Human Behavior Models in
Single-Player MDP

Similar to Experiment 1, we recruited 200 workers from Amazon Mechanical Turk.
Each recruited worker was asked to play 15 navigation games within a grid
world, as shown in Figure 6a. Our goal is to leverage the collected data to
create a data-driven model of human behavior. We divided the collected data of
human actions into three sets: training, validation, and testing. The training
set comprised data from 160 workers, including approximately 70,000 instances
of user decisions, while the validation and testing sets each contained data
from 20 workers, amounting to around 8,800 instances of user decisions each.
The model and parameter tuning were similar to what we did in Experiment 1.

Evaluation of data-driven behavior models. The training, validation, and test
accuracies of assuming optimal behavior and behavioral models are presented in
Table 3. Results show that data-driven model could predict human behavior more
accurately than assuming humans are optimal.

Table 3: The prediction accuracy for human behavior for different human models in Experiment 4. | Training Accuracy | Validation Accuracy | Testing Accuracy
---|---|---|---
Assuming Optimal Behavior | 0.7266 | 0.6964 | 0.7131
Data-Driven Model | 0.9189 | 0.8136 | 0.8422

###  A.3 Experiment 5: Evaluating Human Belief Models

We developed human belief models (behavioral level-1) similar to previous
experiments. We examine whether belief inference using behavioral model aligns
with real humans, and whether we can design AI behavior such that it is easier
for humans to infer the goal of the AI agent.

Experiment setup. The experiment setup is presented in Figure 6b. The grid
world contains a starting position and two goal positions. For each
participant, we show them a trace of behavior from another agent and ask the
participant to infer which of the two goal the agent is trying to reach.

Experiment 5.1: Examining the belief models. We recruited 200 workers from
Amazon Mechanical Turk to compare standard level-$1$ and behavioral level-$1$
model. Each worker was asked to infer the goal for 25 behavioral traces. We
compared the performance of two belief models and the worker accuracy, as
shown in Table 4. As we can see from the table, humans are generally poor at
inferring the goal of other agents: their accuracy only reaches $59.37\%$ in
inferring the goal of the other agent. Moreover, the behavioral level-$1$
model, which accounts for the human behavior model in the belief model,
captures human beliefs better than the standard level-$1$ model, which assumes
human behavior is optimal.

Table 4: Performance comparison between Bayesian inference framework assuming standard model and human behavior model.  | Consistency to Human Predictions | Cross Entropy Loss
---|---|---
Standard level-$1$ | 0.4977 | 0.9284
Behavioral level-$1$ | 0.5764 | 0.7506
True goals | 0.5937 | 0.6631
Figure 7: Human evaluations of belief inference accuracy regrading AI goals.

Experiment 5.2: Developing explicable AI policy. As demonstrated in Experiment
5.1, humans generally struggle to infer the goals of other agents based on
others’ behavior. To design AI agents which makes inference of their goals
easier by humans, we train AI agents to maximize the likelihood that humans
can accurately infer these goals from their behavior.

We recruited 400 workers from Amazon Mechanical Turk, randomly assigning them
to one of two treatments, to assess the behavior of AI agents. Each
participant was tasked with identifying the goals of the player in 30
different scenarios. Participants were awarded a $\$0.03$ bonus for each
correctly identified goal. Figure 7 displays the results of human evaluation.
The findings indicate that humans achieve higher accuracy in inferring the
correct goals of AI agents when employing Behavioral level-$1$ model.

###  A.4 Experiment 6: Evaluating the Design of Collaborative AI Agents

Utilizing developed models of human behavior and beliefs, we follow the same
methodology in Experiment 3: train different collaborative AI, pair them with
different human models, and examine the collaborative performance in
simulations and human subject experiments (via recruiting 300 workers).

The setup of our Experiment 6 is shown in Figure 6c. The goal for the human-AI
team is for both agents to reach the same goal in their own environments (both
reaching “red star” or both reaching “green triangle”) within a time limit.
The team will not get points if they reach different goals or one of the
players fails to reach any goal.

Simulations results are shown in Table 8. Simulations indicate that
collaborative performance is highest when an AI agent is paired with the human
model used to train it. Figure 8 presents the average collaborative reward in
human-subject experiments. Statistical analysis revealed significant
differences in the performance of AI models when paired with human
participants, with $p$-values of $0.0166$ for comparisons between the
treatments _self-play_ and _Behavior-AI_ , and $p<0.0001$ for _Behavior-AI_
versus _Belief-AI_. These results demonstrate that incorporating human beliefs
into the deign of AI agents enhances collaborative performance when working
with real humans.

tableSimulation results of human-AI collaborative rewards in Experiment 6.
Columns players are different AI agents, and row players are different
simulated human models.  Human Model AI Agent Self-play Behavior-AI Belief-AI
Self-play AI 0.7828 0.4780 0.6164 Behavior 0.5926 0.7584 0.4552
Behavior&Belief 0.6919 0.7268 0.7813

Figure 8: Average collaborative reward in human-subject experiment 6.

##  Appendix B Implementation Details

All our experiments are conducted on a cluster of 40 CPU cores (Intel Xeon
Gold 6148 CPU $@$ 2.40GHz), 2 GPUs (NVIDIA Tesla V100 SXM2 32GB), and a
maximal memory of 80GB. For completeness, we provide the details of the
implementation and the tuning of hyperparameters below.

State encoding of grid world. For a grid world of size $N\times N$ and
contains $k$ different objects, we use $k$ channels to encode the environment
state. Each channel is a binary matrix of size $N\times N$, indicating whether
it contains specific objects. For example, in a single-player environment with
one goal (environment in Experiment 4), we encode space, blocks, the start
position, and goal positions as four channels. Thus the input state is of size
$4\times 6\times 6$ before flattening. The state encoding is used as input of
our behavioral cloning models and AI agents.

Behavioral cloning for grid world of size $6\times 6$. We construct a neural
network with four fully connected layers to model human behavior. The input is
the state encoding, represented as a matrix of dimensions $4\times 6\times 6$,
and the output is the probability of taking various actions. The network
parameters are initialized using the Glorot uniform initializer. We employ the
Adam optimizer and cross-entropy loss to optimize the neural network
parameters. The batch size is set to $1024$, and the maximum number of
training epochs is set to $1000$. Training is halted early if the validation
loss begins to increase. The hyperparameters are adjusted based on the
performance on the validation dataset. The number of nodes for each hidden
layer is tuned within the range $\\{64,128,256,512,1024\\}$, the initial
learning rate within the range $\\{0.001,0.003,0.01,0.03,0.1\\}$, and the L2
penalty within the range $\\{0,0.01,0.03,0.1\\}$. Each training process takes
$10$ to $30$ minutes depends on hyperparameters. In our reported results, we
choose hidden layer size of $512$, learning rate $0.01$ and L2 penalty $0.01$.

Behavioral cloning for grid world of size $8\times 8$. The neural network
structure mirrors that used for the grid world of size $6\times 6$. The input
is the state encoding, represented as a matrix with dimensions $8\times
8\times 8$, and the output is the probability of taking various actions. The
initialization, optimizer, loss function, batch size and training process are
consistent with those used in the grid world of size $6\times 6$. The maximum
of training epochs is set to $3000$. The hyperparameters are adjusted based on
the performance on the validation dataset. The number of nodes for each hidden
layer is tuned within the range $\\{1024,2048,4096\\}$, the initial learning
rate within the range $\\{0.001,0.003,0.01,0.03,0.1,0.3\\}$, and the L2
penalty within the range $\\{0,0.01,0.1\\}$. In our reported results, we
choose hidden layer size of $2048$, learning rate $0.01$ and L2 penalty
$0.01$.

AI agent training for grid world of size $6\times 6$. We utilize the Proximal
Policy Optimization (PPO) and actor-critic framework for our AI model. Both
the actor and critic networks employ four fully connected layers, maintained
at the same size. The input is the state encoding of a two-player game, which
includes two game states and the index vector (the input vector length is
$290$). The critic network outputs a single value, while the actor network
produces a distribution over the action space. The batch size is set to $256$,
and the maximum number of training episodes is limited to $1000$. The discount
factor is set to $0.99$. For PPO optimization, the clipping parameter
$\epsilon$ is fixed at $0.2$, and the number of PPO epochs is $10$. The number
of nodes for each hidden layer is tuned within the range
$\\{64,128,256,512,1024\\}$, and the initial learning rate is adjusted within
the range of $\\{0.001,0.003,0.01,0.03,0.1\\}$. Each training process takes
about $2$ to $6$ hours depends on hyperparameters. In comparisons between the
self-play AI and the AI trained to play with a human model, we use the same
model size (hidden layer size of $256$) and the same initial learning rate of
$0.01$.

AI agent training for grid world of size $8\times 8$. The model structure
mirrors that of the grid world of size $6\times 6$. The input size is adjusted
to accommodate the new state representation (the input vector length is
$514$). The training batch size, discounting factor, maximum number of
training epochs and clipping parameters remain unchanged. The number of nodes
for each hidden layer is tuned within the range $\\{256,512,1024,2048\\}$, and
the initial learning rate is adjusted within the range of
$\\{0.01,0.03,0.1,0.3,0.5\\}$. In comparisons between the self-play AI and the
AI trained to play with a human model, we consistently use the same model size
(hidden layer size of $512$) and the same initial learning rate of $0.1$.

##  Appendix C Human Experiment Details

We include more details about human-subject experiments. The experiment
interface is shown in Figure 9. We conduct four human-subject experiments, and
recruited 1690 workers from Amazon Mechanical Turk in total. Table 5 lists the
number of workers, the number of tasks per worker and the number of treatments
in each experiment. Table 6 contains the demographic information of all the
workers.

(a) Experiment 1 & 3 interface: humans play games with AI agents in grid
worlds of size $8\times 8$.

(b) Experiment 2 interface: humans infer goals of other players in grid worlds
of size $8\times 8$.

(c) Experiment 4 interface: humans play single-player navigation games in grid
worlds of size $6\times 6$.

(d) Experiment 5 interface: humans infer goals of other players in grid worlds
of size $6\times 6$.

(e) Experiment 6 interface: humans play games with AI agents in grid worlds of
size $6\times 6$.

Figure 9: Human-subject experiment interfaces.  Table 5: Number of workers in each human-subject experiment. | Workers | Tasks per Worker | Treatments | Base Payment | Bonus per Task
---|---|---|---|---|---
Experiment 1 | 190 | 30 | 3 | $1.00 | $0.05
Experiment 2 | 200 | 30 | 2 | $1.00 | $0.03
Experiment 3 | 200 | 30 | 3 | $1.00 | $0.05
Experiment 4 | 200 | 15 | 1 | $1.00 | 0
Experiment 5.1 | 200 | 25 | 1 | $1.00 | 0
Experiment 5.2 | 400 | 30 | 2 | $1.00 | $0.03
Experiment 6 | 300 | 20 | 3 | $1.50 | $0.05
Table 6: Demographic information of all the participants in our experiments. Group | Category | Number
---|---|---
Age | 20 to 29 | 775
30 to 39 | 642
40 to 49 | 146
50 or older | 127
Gender | Female | 547
Male | 1118
Other | 25
Race / Ethnicity | Caucasian | 1511
Black or African-American | 82
American Indian/Alaskan Native | 32
Asian or Asian-American | 28
Spanish/Hispanic | 7
Other | 30
Education | High school degree | 60
Some college credit, no degree | 42
Associate’s degree | 39
Bachelor’s degree | 1217
Graduate’s degree | 293
Other | 39