# Latent Emission-Augmented Perspective-Taking (LEAPT)
for Human-Robot Interaction

Kaiqi Chen1, Jing Yu Lim1, Kingsley Kuan1, and Harold Soh1,2 This research is
supported by the National Research Foundation Singapore and DSO National
Laboratories under the AI Singapore Programme (AISG Award No:
AISG2-RP-2020-016).1Authors are with the Dept. of Computer Science, National
University of Singapore. {kaiqi, jy_lim, kingsley,
<EMAIL_ADDRESS>Systems Institute, National University of
Singapore

###### Abstract

Perspective-taking is the ability to perceive or understand a situation or
concept from another individual’s point of view, and is crucial in daily human
interactions. Enabling robots to perform perspective-taking remains an
unsolved problem; existing approaches that use deterministic or handcrafted
methods are unable to accurately account for uncertainty in partially-
observable settings. This work proposes to address this limitation via a deep
world model that enables a robot to perform both perception and conceptual
perspective taking, i.e., the robot is able to infer what a human sees and
believes. The key innovation is a decomposed multi-modal latent state space
model able to generate and augment fictitious observations/emissions.
Optimizing the ELBO that arises from this probabilistic graphical model
enables the learning of uncertainty in latent space, which facilitates
uncertainty estimation from high-dimensional observations. We tasked our model
to predict human observations and beliefs on three partially-observable HRI
tasks. Experiments show that our method significantly outperforms existing
baselines and is able to infer visual observations available to other agent
and their internal beliefs.

##  I INTRODUCTION

This work focuses on the problem of perspective taking, which is the ability
to take another agent’s point of view, either visually or conceptually. For
example, consider the scenario in Fig. 1 where a robot and a human are
aligning a peg and hole to assemble a table. As the robot is holding the
table, it cannot see the peg and hole. To collaborate effectively, the robot
should reason about the human’s visual perspective and infer that the human is
able to see the peg and hole (despite not knowing what he actually sees). It
can then query the human for relevant information.

Figure 1: Perspective-Taking Example. (A) As the robot (green) is holding the
table, it cannot see the peg and hole (purple), which is clearly visible to
the human (light blue). The robot considers the perspective of the human and
reasons about what the human can observe, despite not knowing what he actually
sees. (B) Human belief inference. In brief, the robot’s world model (self-
model) is used to sample possible environment states and predict human
observations, and the human model is used to infer human belief. By leveraging
the inference network and perspective taking-model, we can infer human belief
under each sampled environment state.

Enabling robots to perform such perspective-taking is challenging, especially
when the environment is partially-observable. Prior works on perspective-
taking in Human-Robot Interaction (HRI) focus mainly on hand-crafted models
(e.g., [1, 2, 3, 4, 5]) or learning deterministic models in fully-observable
environments [6]. Unfortunately, handcrafted models do not easily scale to
complex real-world environments with high-dimensional observations, and learnt
deterministic models do not capture uncertainty in the beliefs or world state.

In this paper, we take a step towards addressing this gap and propose a
perspective-taking approach based on deep latent state-space world models.
Given only partial observations (e.g., camera images), our Latent Emission-
Augmented Perspective-Taking (LEAPT) is able to infer what another agent
observes and knows. This is achieved by sampling a set of possible world
states from the robot’s underlying world or “self” model (Fig.1.B.). Using
these sampled states along with human pose information, we predict potential
human observations, which are then used to infer human beliefs using a
modified self-model [7].

We find that the self-model requires careful design to be effective — a
standard latent state-space model with a single time-dependent latent variable
did not properly capture uncertainty in the world state (resulting in samples
that were blurry “averages” of training observations). To address this issue,
we impose an additional model structure by decomposing the latent state into
two variables that represent (i) information that the robot can fully observe
($s_{t}^{R}$) and (ii) information that is not revealed by the robot’s
observations ($h_{t}^{R}$). By conditioning $h_{t}^{R}$ on $s_{t}^{R}$, we
explicitly train the model to generate possible task states given only partial
information. Moreover, by minimizing the Kullback-Leibler (KL) divergence
between task-complete observations (available during training) and the robot’s
estimation in latent space, the model learns to properly estimate uncertainty
about the state.

Experiments on various tasks in a simulated domain show that our model can
better estimate the visual perceptions and beliefs of other agents compared to
a deterministic approach and a standard latent-state space model.
Specifically, we show our model enables robots to model false beliefs and
better estimate another agent’s (and it’s own) beliefs during interaction and
communication. We also report on a human-subject study using Virtual Reality
(VR), which shows that LEAPT is better at predicting the human’s belief about
the world.

In summary, this paper makes the following contributions:

  * •

LEAPT — a perspective-taking approach based on a latent state-space model —
for partially-observable environments;

  * •

A decomposed model structure and corresponding ELBO training loss that
influences the model to better capture uncertainty over possible world states;

  * •

Empirical results that show that LEAPT outperforms competing methods and
ablated variants.

##  II Background & Related Work

Perspective-taking [8, 9, 10] is considered essential for many useful social
skills in human-human interaction, such as understanding social dynamics and
relationships. Within the context of robotics, perspective-taking can also
occur during human-robot interaction (HRI) [1, 11]. In the following, we
briefly mention prior works on enabling artificial agents to perform
visuospatial persepective taking and mentalizing (reasoning about another
agent’s mental states).

Cognitive Architectures. To enable efficient HRI, earlier works proposed
cognitive architectures using hand-crafted rules and perspective-taking models
[1, 2, 3, 4, 5]. Although these models can be effective, they are generally
time-consuming to create and do not lend themselves easily to scenarios with
high-dimensional observations (e.g., images). Potentially, modern physics
simulators [12, 13, 14] could be embedded within the cognitive architecture of
the robot to perform perspective-taking. However, uncertainty estimation
within such a setup remains computationally expensive — one would need to
either simulate multiple possible observations and track beliefs over time.

ML-based Models. An alternative approach is to leverage machine learning (ML)
to predict other agents’ perspectives [6, 15] and apply planning [6] (or
reinforcement learning [15]) to generate robot behaviors. For example, in a
hide-and-seek task [6], a robot hider predicts what the seeker sees using
deterministic deep neural networks and then applies planning to avoid the
seeker. Recent work learned a Q function conditioned on both ego-agent and
other agents observations in a dominant-and-subordinate-monkey task [15].
Unlike LEAPT, these works use deterministic models or assume access to other
agents’ observations.

Agent Modeling and Communication in Multi-Agent Systems Our work is also
related to opponent modeling in a multi-agent system. In this line of
research, one approach is to optimize policies using a latent embedding that
encodes the (predicted) trajectories of other agents [16, 17, 18]. These
methods either employ deterministic approaches when deployed [17, 18] or model
uncertainty in the observation space [16]. In the field of human-robot
communication, recent works modify generated observations and track human
belief using learned models [7, 19]. Unlike LEAPT, these methods assume that
the robot is able to fully observe the environment at test time. LEAPT is
designed to operate in _partially-observable environments_ for tasks where
visual and conceptual (belief) perspective-taking is helpful.

##  III Latent Emission-Augmented Perspective-Taking (LEAPT)

Objective. We aim to to enable a robot to perform:

  * •

_Visual perspective-taking_ : estimate what another agent observes at a given
time;

  * •

_Conceptual perspective-taking_ : Infer what the agent believes given the
observations they have made so far, which is a form of a set of latent states.

We assume that the robot has access to its own on-board sensors and knows the
other agent’s pose. In addition, during training, the robot has access to a
world-view that can provide task-complete information; this information is
_not_ available during test time.

In this section, we describe our main contribution: the Latent Emission-
Augmented Perspective-Taking (LEAPT) model. At its core, LEAPT uses a multi-
modal latent state-space model (MSSM) [20, 7] that is modified by decomposing
the latent state variable. We begin with a discussion of the standard MSSM and
its limitations, which we then address by factorizing the latent state
representation. This enables the robot to (i) better track its own uncertainty
over the world state, and (ii) track the beliefs of other agents. We then
illustrate how this model can be used for both visual and conceptual (belief)
perspective taking via sampling.

Figure 2: The LEAPT Decomposed Latent State-Space Model. Circle nodes
represent random variables. The latent variables are decomposed into two
parts, $z_{t}=[h_{t},s_{t}]$. Inference networks $q_{\phi}$ are shown using
blue dotted line arrows; we see that $q(s_{t}|x_{t}^{1:M})$ computes $s_{t}$
(the part of the world that robot can fully observe), whilst $p(h_{1}|s_{1})$
and $p(h_{t}|s_{t-1,t},h_{t-1})$ generates the part that robot cannot observe.

###  III-A LEAPT Latent State-Space Model

Standard MSSM World Model. Our robot maintains a world model — called the
_self-model_ — on which it can perform inference and planning. We build upon
the MSSM [20, 7], which captures how the environment changes over time given
actions taken by the robot. In general, the robot is unable to directly
perceive the entire world state and we distinguish between two kinds of
observations:

  * •

Ego observations $x^{m}_{t}$ at time $t$ for $m=1,\dots,M$ sensory modalities,
which are the agent’s observations from its own perspective (e.g., a camera
mounted on the robot’s arm) and can be accessed in both training and testing;

  * •

Task-Complete observations $y^{k}_{t}$ at time $t$ for $k=1,\dots,K$
modalities, which contains all sufficient information about world state to
complete the task and _can only be accessed during training_.

Similar to standard state-space models, the MSSM assumes Markovian transitions
that are only dependent on the current state and the action $a_{t}$ taken by
the agent.

Limitations of the standard MSSM. One can train the MSSM using variational
inference by optimizing the evidence lower bound (ELBO):

$\displaystyle\mathcal{L}_{e}=$
$\displaystyle\sum_{t=1}^{T}\displaystyle\mathop{\mathbb{E}}_{q_{\phi}(z_{t})}\left[\sum_{m=1}^{M}\log
p_{\theta}(x^{m}_{t}|z_{t})+\sum_{k=1}^{K}\log
p_{\theta}(y^{k}_{t}|z_{t})\right]$
$\displaystyle-\sum_{t=2}^{T}\displaystyle\mathop{\mathbb{E}}_{q_{\phi}(z_{t-1})}\left[\mathbb{D}_{\mathrm{KL}}\left[q_{\phi}(z_{t})\|p_{\theta}(z_{t}|z_{t-1},a_{t-1})\right]\right]$
$\displaystyle-\displaystyle\mathop{\mathbb{E}}_{q_{\phi}(z_{1})}\left[\mathbb{D}_{\mathrm{KL}}\left[q_{\phi}(z_{1})\|p_{\theta}(z_{1})\right]\right]$
(1)

where the variational distribution $q_{\phi}(z_{t})$ is shorthand for
$q(z_{t}|u_{\phi}(x_{1:t}^{1:M},y_{1:t}^{1:K}))$. This variational
distribution can be modeled using an inference network $u_{\phi}$ that takes
as input observations (_both_ ego observations $x_{1:t}^{1:M}$ and task-
complete observations $y_{1:t}^{1:K}$) and outputs a distribution over
$z_{t}$. In theory, once trained, $q_{\phi}(z_{t})$ can be used to sample
state $z_{t}$ given only agent’s _ego_ observations $x^{1:M}_{1:t}$, i.e., the
$z_{t}$’s will be used to enable perspective taking.

The problem lies in using this inference network during test time when the
task-complete observations are missing. One simple solution is to randomly
drop the task-complete observations during training [20, 7, 21] by zero-ing
out $y_{1:t}^{1:M}$. Unfortunately, our preliminary experiments results showed
that this approach led to poor variational distributions: $q_{\phi}(z_{t})$
was ‘narrow’ (over-confident) when $x_{1:t}^{1:M}$ did not contain sufficient
information of the world state. The image samples from $q_{\phi}(z_{t})$
tended to be blurry “averages” of the data the model was trained with (see Fig
7 for examples).

One possible reason for this undesirable behavior is that the MSSM is trained
to encode $y^{1:K}_{t}$ into $z_{t}$ via the reconstruction term
$\mathop{\mathbb{E}}_{q_{\phi}(z_{t})}\left[\log
p_{\theta}(y^{k}_{t}|z_{t})\right]$.

If we drop $y_{1:t}^{1:K}$, $q(z_{t}|u_{\phi}(x_{1:t}^{1:M}))$ may lack
sufficient information to generate a sample representative of the world state.
In such cases, the model learns to “hedge” by generating (decoding) the mean
observation, which corresponds to a narrow $q_{\phi}(z_{t})$ centered around a
specific latent state.

Decomposed Latent State. As a remedy, we propose to restructure the MSSM to
have a more explicit generation process. This modified structure is
illustrated in Fig.2. Intuitively, we would like the model to learn how to
generate task-complete observations _only_ using ego observations. As such, we
decompose the latent state into two disjoint parts, $z_{t}=[s_{t},h_{t}]$,
where $s_{t}$ represents the observable part of the world and $h_{t}$
represents information that the robot cannot observe directly. By conditioning
$h_{t}$ on $s_{t}$ in $p(h_{t}|g_{\theta}([s,h,a]_{t-1},s_{t}))$, the model
explicitly represents our desired computation using a neural network
$g_{\theta}$.

Given this decomposed latent structure, the joint distribution factorizes as:

$\displaystyle p_{\theta}($ $\displaystyle
x^{1:M}_{1:T},y^{1:K}_{1:T},s_{1:T},h_{1:T},a_{1:T-1})=$ $\displaystyle
p_{\theta}(h_{1}|s_{1})p(s_{1})\prod_{t=2}^{T}\left[\prod_{m=1}^{M}p_{\theta}(x^{m}_{t}|s_{t})\right]\left[\prod_{k=1}^{K}p_{\theta}(y^{k}_{t}|s_{t},h_{t})\right]$
$\displaystyle
p_{\theta}(s_{t}|s_{t-1},h_{t-1},a_{t-1})p_{\theta}(h_{t}|s_{t-1,t},h_{t-1},a_{t-1})$
(2)

where each of the factorized distributions is modeled using deep neural
networks parameterized by $\theta$:

$\displaystyle\mathrm{\textrm{Transition:}}\
p_{\theta}([s,h]_{t}|[s,h,a]_{t-1})$
$\displaystyle\quad\quad=p(s_{t}|f_{\theta}([s,h,a]_{t-1}))p(h_{t}|g_{\theta}([s,h,a]_{t-1},s_{t}))$
(3) $\displaystyle\mathrm{\textrm{Ego Observations:}}\
p_{\theta}(x^{m}_{t}|s_{t})=p(x^{m}_{t}|d^{m}_{x_{\theta}}(s_{t}))$ (4)
$\displaystyle\mathrm{\textrm{Task-complete Observations:}}\
p_{\theta}(y^{k}_{t}|[s,h]_{t})$
$\displaystyle\quad\quad=p(y^{k}_{t}|d^{k}_{y_{\theta}}([s,h]_{t}))$ (5)

Model Training. Similar to the basic MSSM, we optimize the ELBO, but with two
variational distributions $q(s_{t}|v_{\psi}(x_{1:t}^{1:M}))$ and
$q(h_{t}|l_{\eta}(y_{1:t}^{1:K}))$ over the latent state variables $s_{t}$ and
$h_{t}$. Note that $v_{\psi}$ and $l_{\eta}$ are two different inference
networks. For notational simplicity, we will drop the explicit dependence on
the conditioning variables and write the ELBO as,

$\displaystyle\mathcal{L}_{e}$
$\displaystyle=\sum_{t=1}^{T}\Big{(}\displaystyle\mathop{\mathbb{E}}_{q_{\psi}(s_{t})}\left[\sum_{m=1}^{M}\log
p_{\theta}(x^{m}_{t}|s_{t})\right]$
$\displaystyle+\displaystyle\mathop{\mathbb{E}}_{q_{\psi}(s_{t})q_{\eta}(h_{t})}\left[\sum_{k=1}^{K}\log
p_{\theta}(y^{k}_{t}|[s,h]_{t})\right]\Big{)}$
$\displaystyle-\sum_{t=2}^{T}\Big{(}\displaystyle\mathop{\mathbb{E}}_{q_{\psi}(s_{t-1})q_{\eta}(h_{t-1})}\left[\mathbb{D}_{\mathrm{KL}}\left[q_{\psi}(s_{t})\|p_{\theta}(s_{t}|[s,h,a]_{t-1})\right]\right]$
$\displaystyle-\displaystyle\mathop{\mathbb{E}}_{q_{\psi}(s_{t-1})q_{\eta}(h_{t-1})}\left[\mathbb{D}_{\mathrm{KL}}\left[q_{\eta}(h_{t})\|p_{\theta}(h_{t}|[s,h,a]_{t-1},s_{t})\right]\right]\Big{)}$
$\displaystyle-\displaystyle\mathop{\mathbb{E}}_{q_{\psi}(s_{1})}\left[\mathbb{D}_{\mathrm{KL}}\left[q_{\eta}(h_{1})\|p_{\theta}(h_{1}|s_{1})\right]\right]$
$\displaystyle-\mathbb{D}_{\mathrm{KL}}\left[q_{\psi}(s_{1})\|p_{\theta}(s_{1})\right]$
(6)

In contrast to the MSSM, we don’t need to discard $y^{1:K}_{1:t}$ during the
training process. Instead, we substitute $q_{\eta}(h_{t})$ with
$p_{\theta}(h_{t}|[s,h,a]_{t-1},s_{t})$ during the testing phase (or with
$p_{\theta}(h_{1}|s_{1})$ when $t=1$). This ELBO variant enables
$q_{\eta}(h_{t})$ to convert $y^{1:K}_{1:t}$ into a latent distribution, such
as a Gaussian or Categorical distribution. By minimizing
$\mathbb{D}_{\mathrm{KL}}\left[q_{\eta}(h_{t})\|p_{\theta}(h_{t}|[s,h,a]_{t-1},s_{t})\right]$
and
$\mathbb{D}_{\mathrm{KL}}\left[q_{\eta}(h_{1})\|p_{\theta}(h_{1}|s_{1})\right]$,
$p_{\theta}(h_{t}|[s,h,a]_{t-1},s_{t})$ and $p_{\theta}(h_{1}|s_{1})$ are
trained to generate potential $h_{t}$ using observations $s_{1:t}$. In a
nutshell, this latent state-space model decomposition captures task-complete
uncertainty at the latent level. By carefully setting $q$ and $p$, e.g.,
Gaussian, we can calculate these KL divergence terms exactly. We found that
this decomposition resolves the issue of suboptimal latent state estimations
and reconstructions, enabling us to create a self-model that produces credible
samples. Please refer to Fig. 7 and 6 for examples.

###  III-B Visual Perspective-Taking

We use sampled task-complete observations $y^{1:K}_{t}$ to generate relevant
observations from different perspectives. Specifically, we train a visual
perspective-taking model $\hat{x}^{1:M}_{t}=d_{\chi}(y^{1:K}_{t},\omega_{t})$,
parameterized by $\chi$, that produces observations given an agent’s pose
$\omega_{t}$ at time $t$ and $y^{1:K}_{t}$.

Training can be performed with data gathered by the robot R during roll-outs
in the environment. Given trajectories of the form
$\left\\{(x^{1:M,\textrm{R}}_{t},y^{1:K}_{t},\omega_{t}^{\textrm{R}})\right\\}_{t=1}^{T}$
collected by the robot, we learn $\chi$ by minimizing the following loss:

$\displaystyle\operatorname*{arg\,min}_{\chi}\ $
$\displaystyle\mathcal{L}(\chi)=\sum_{t=1}^{T}\sum_{m=1}^{M}\left(\hat{x}^{m}_{t}-x^{m}_{t}\right)^{2}$

Once trained, we can use the decomposed latent state space model and
$d_{\chi}$ to obtain samples of another agent’s observations. The process is
straightforward: given robot observations $x^{1:M,\textrm{R}}_{t}$ and the
agent’s pose $\omega_{t}^{\textrm{H}}$, we sample a task-complete observation
$y^{1:K}_{t}$ via

  1. 1.

Sample the robot’s belief state $s_{t}^{\textrm{R}}\sim
q_{\psi}(s_{t}|(\hat{x}^{1:M,\textrm{R}}_{t}))$ and $h_{t}^{\textrm{R}}\sim
p_{\theta}(h_{t}|[s^{\textrm{R}},h^{\textrm{R}},a]_{t-1},s^{\textrm{R}}_{t})$
with $h_{1}^{\textrm{R}}\sim p_{\theta}(h_{1}|s^{\textrm{R}}_{1})$.

  2. 2.

Sample the task-complete observation $\hat{y}^{1:K}_{t}\sim
p_{\theta}(y^{1:K}_{t}|s_{t}^{\textrm{R}},h_{t}^{\textrm{R}})$.

  3. 3.

Given human position, generate human observations
$\hat{x}^{1:M,\textrm{H}}_{t}=d_{\chi}(\hat{y}^{1:K}_{t},\omega_{t}^{\textrm{H}})$

As an aside, one may be curious why we did not directly predict $x^{m}_{t}$
using $h_{t}^{\textrm{R}}$ and $s_{t}^{\textrm{R}}$. Our preliminary
experiments indicated that this direct approach caused the model to disregard
the poses $\omega_{t}^{\textrm{H}}$, resulting in poor samples. This behavior
stems from training the perspective-taking model on tuples of _robot_ pose and
_robot_ ego observations $x^{1:M,\textrm{R}}_{t}$. By including the robot’s
latent states $h_{t}^{\textrm{R}}$ and $s_{t}^{\textrm{R}}$ (derived from
$x^{1:M,\textrm{R}}_{t}$) the model is able to make predictions directly from
them, overshadowing the importance of the agent poses. As a result,
substituting $\omega_{t}^{\textrm{R}}$ with human pose
$\omega_{t}^{\textrm{H}}$ during test time leads to the failure of the
perspective-taking model in generating accurate observations from the other
agent’s perspective.

###  III-C Conceptual (Belief) Perspective-Taking

Next, we turn our attention to how the self-model can be used to estimate
human belief, which is a form of conceptual perspective taking. We follow
prior work [7] and use a variant of the robot’s self-model. By coupling the
human model and robot’s self-model together as shown in Fig.1.B., we sample
belief states in the human model:

  1. 1.

Generate the human observations $\hat{x}^{1:M,\textrm{H}}_{1:t}$ by visual
perspective-taking.

  2. 2.

Sample the human’s belief state $s_{t}^{\textrm{H}}\sim
q_{\psi}(s_{t}|(\hat{x}^{1:M,\textrm{H}}_{1:t}))$ and $h_{t}^{\textrm{H}}\sim
p_{\theta}(h_{t}|[s^{\textrm{H}},h^{\textrm{H}},a]_{t-1},s^{\textrm{H}}_{t})$
with $h_{1}^{\textrm{H}}\sim p_{\theta}(h_{1}|s^{\textrm{H}}_{1})$.

Unlike existing approaches, LEAPT can capture a (sample-based) distribution
over possible observations and beliefs. This enables the robot to perform
epistemic reasoning, i.e., the robot can infer that the other agent is aware
of some information even though the robot doesn’t know exactly what that
information is. We will see examples of this ability in the experiments.

##  IV Simulated Human Experiments

Figure 3: Experimental domains. (A) A False-Belief test, where the robot
_cannot_ see what is inside the box but the human can. The human first
observes the position of the drill (in the left or right box) and then walks
away. Then, unobserved by the human, the robot will either switch the box or
not. The robot’s task is to infer whether the human would believe that the
objects remain in their original position. (B) In the Fetch-Tool task, a human
is sitting in front of his workstation and observes a randomly chosen object
on his computer screen. There are six different types of objects, which are
characterized by their color and type. The robot’s task is to reason about
human uncertainty and what he sees without observing the object, using only
communication information from the human. However, communication of the object
has three levels of ambiguity: type, colour and both the type and colour of
the object. Once the robot figures out what the human sees, it can
subsequently fetch the appropriate tool to assemble the object. (C) In the
Table-Assembly, the robot’s task is to fit the peg of a table piece into a
hole with guidance from the human. However, the hole may be occluded depending
on the agent’s position. As the robot is holding on to the table, only the
human is able to observe the relative distance between the peg and hole. The
robot’s task is to reason whether the human can see the hole so that the robot
can move the table to match it. (D) Our VR setup in the human-subject
experiments used an Occulus Quest headset.

In the following section, we describe experiments with simulated humans on a
False-Belief test, a Fetch-Tool task, and a Table-Assembly task (Fig. 3.).
Using simulated humans allowed us to compare the learned models against ground
truth observations and beliefs. Experiments with real human subjects are
described in the next section.

###  IV-A Experimental Setup

Domains. Fig. 3. summarizes the three domains used in our experiments. All
domains involve two agents: a robot and a human. The observations of the
environment (both ego and task-complete) are images. We assume the agents can
communicate via specific speech symbols in the Fetch-Tool and Table-Assembly
tasks (these are simply modeled as observations as in [7]). We assume the
robot can always observe the human’s pose.

Simulated Humans. We assume that the human is a Bayes-rational agent that
updates their beliefs using ego-centric observations. For example, in the
False-Belief test, the simulated human will believe that the drill remains in
the same position when they walk away. Specifically, the simulated human’s
belief is a Bernoulli distribution representing where the drill is (left or
right box). In the Fetch-Tool task, the simulated human belief is a
Categorical distribution over which object they see on the screen (six
classes). In the Table-Assembly task, we use a Gaussian distribution over the
relative distance between the peg and hole.

Compared Methods. In total, we compare four different perspective-taking
methods:

  * •

Baseline-S: This baseline uses the standard MSSM and is trained with the ELBO
in Eq. III-A. We use bi-directional GRU as the main component of the inference
network.

  * •

Baseline-D: This baseline is similar to Baseline-S except that the latent
space is deterministic.

  * •

LEAPT-GRU: Our decomposed latent state space model with a GRU-based inference
network. The model is trained with the ELBO in Eq. III-A.

  * •

LEAPT-Transformer: This model is the same as the LEAPT-GRU, except we use a
Transformer-based [22] inference network.

All the models are trained on the same data and epochs within each domain
(False-Belief: 30 trajectories with length 5, 100 epochs; Fetch-Tool: 100
trajectories with length 5, 400 epochs; Table-Assembly: 300 trajectories with
length 10, 700 epochs). The trajectories used for training are generated by
taking random actions. At every time step $t$, the inference time is less than
0.05s for all models on a NVIDIA GeForce RTX 2080 Ti. Source code is available
at https://github.com/clear-nus//perspective-taking-dmssm.

Evaluation We first evaluate whether the robot maintains a correct belief
about the world. Then, we evaluate the robot’s ability to perform visual and
belief perspective-taking, i.e., infer what the other agent sees and believes.
As the direct evaluation of latent states is challenging, we utilize pre-
trained classifiers or regression models on the samples $x$ and $y$ that are
decoded from the latent states. Denote the predicted labels as $c(x)$ or
$c(y)$. The exact predicted quantities was task-dependent: i) In the False-
Belief test, we predict where the drill is; ii) In the Fetch-Tool task, we
predict the object present on the screen (Fig. 3.B.); iii) In the Table-
Assembly task, we predict the relative position between human and table and
the relative distance between peg and hole (Fig. 3.C.). We then compare the
distribution of these predictions to a specified ground-truth distribution
(assuming a Bayes-rational agent) via KL divergence (lower is better).

_Evaluation of robot’s belief:_ We first decode $\hat{y}^{\textrm{R}}$ from a
set of the robot’s latent states. Then, we apply classifiers/regression models
on $\hat{y}^{\textrm{R}}$ to obtain predictions $c(\hat{y}_{t}^{\textrm{R}})$.
The quality of the robot’s belief is measured through the KL divergence
$\mathbb{D}_{\mathrm{KL}}\left[\hat{p}(c(\hat{y}_{t}^{\textrm{R}})|x_{1:t}^{\textrm{R}})\|p(c({y}_{t}^{\textrm{R}})|x_{1:t}^{\textrm{R}})\right]$
between the empirical distribution, $\hat{p}$, and the ground truth
distribution $p$ (representing what the robot should know about world given
our experiment parameters).

_Evaluation of visual perspective-taking:_ We generate human observations
$\hat{x}_{t}^{\textrm{H}}$ via our visual perspective-taking method (Sec.
III-B). Then, we predict labels $c(\hat{x}_{t}^{\textrm{H}})$ and compute the
KL divergence
$\mathbb{D}_{\mathrm{KL}}\left[\hat{p}(c(\hat{x}_{t}^{\textrm{H}})|x_{1:t}^{\textrm{R}})\|p(c({x}_{t}^{\textrm{H}})|x_{1:t}^{\textrm{R}})\right]$.
The distribution $p$ varies according to the amount of information available,
e.g., in the Fetch-Tool task, if the robot has yet to communicate with the
human, $p(c({x}_{t}^{\textrm{H}})|x_{1:t}^{\textrm{R}})$ is uniform over all
possible objects. After receiving a message that the object is brown, the
ground truth distribution shrinks to equal probabilities across brown objects.

_Evaluation of belief perspective-taking:_ We begin by sampling from the human
belief distribution (via belief perspective-taking in Sec. III-C). Then, we
generate labels $c(y_{t}^{\textrm{H}})$ and compute the Conditional KL (Cond
KL) measure via
$\mathbb{E}_{x_{1:t}^{\textrm{H}}}\left[\mathbb{D}_{\mathrm{KL}}\left[\hat{p}(c(\hat{y}_{t}^{\textrm{H}})|x_{1:t}^{\textrm{H}})\|p(c({y}_{t}^{\textrm{H}})|x_{1:t}^{\textrm{H}})\right]\right]$,
where $x_{1:t}^{\textrm{H}}$ is sampled using visual perspective-taking.

Figure 4: Evaluation of (visual & conceptual) perspective-taking in three
tasks using KL divergence as a measure (lower is better). In most cases,
LEAPT-Transformer predicts the (simulated) human’s observation and belief more
accurately compared to the other methods. Figure 5: Accuracy of human belief
prediction on the False-Belief Test. (A) boxes not switched by robot (B) boxes
switched by robot.

###  IV-B Results and Analysis

Due to space considerations, we highlight key results from our experiments.
Additional plots are available in our online appendix:
https://github.com/clear-nus//perspective-taking-dmssm..

Overall, LEAPT-Transformer maintains more accurate beliefs about the world
compared to the baselines when given only ego observations. We see in the
first row of Fig. 4 that the KL divergence tends to be smallest for the LEAPT
variants in the False-Belief and Table-Assembly tasks. The sampled images from
the baselines tended to be blurry, leading to poor identification of task-
relevant features. For Fetch-Tool, the LEAPT-Transformer achieved better
scores in the initial time-steps, but the baselines performed well when all
needed information was available (at time-step 3). The LEAPT-Transformer state
distribution was overly broad at the last time-step, but it generated the best
samples overall over the course of the interaction.

The LEAPT-Transformer achieves better visual perspective-taking given the
robot’s beliefs. The second row of Fig. 4 shows better scores by the LEAPT-
Transformer, except (again) for the last time step of the Fetch-Tool task. In
general, the quality of the generated samples (both diversity and visual
clarity) was positively correlated with better robot beliefs111For the False-
Belief task, note the human doesn’t return to the boxes so the generated
observations are of a different area.. The baselines tended to produce poor
quality images, with sharp images only appearing when all task information was
provided (See Fig. 7 for examples).

The LEAPT models better infer the other agent’s beliefs compared to the
baselines. This can be seen in the last row of Fig. 4, where the KL scores are
overall better than the baselines. For the False-Belief test, a more
interpretable result is shown in Fig. 5 where we ask the robot to classify
where the human thinks the object to be. When the boxes are switched, the
baselines fail to capture that the human holds a false belief.

For the fetch tool task, a 2D Principal Components Analysis (PCA) plot (Fig.6)
suggest that the baseline models do not learn an appropriate latent space. It
incorrectly maps different partial observations to the same cluster of latent
points. On the other hand, LEAPT-Transformer correctly captures the
distribution of latent states: with more communication coming from the human,
the states converge.

Figure 6: The visualization of estimated human belief (latent states) in the
fetch tool task (dimension reduced to two by PCA). Unlike the baselines, the
LEAPT-Transformer maintains a reasonable distribution over possible objects
when provided with communication. Figure 7: Fetch-Tool qualitative results:
Baseline-S vs. LEAPT-Transformer. For each model, we sample 10 human
observations and 10 human latent states (we decode latent states into image
observations and apply a pre-trained classifier to obtain the estimated human
beliefs). The simulated ground truth always shows the correct object. Compared
to the baseline, the LEAPT model belief distributions are consistent with the
information available at each stage of the communication.

Fig.7 illustrates the ability of the LEAPT-Transformer to predict human belief
given partial information. At the start of the interaction, without any
information, the LEAPT-Transformer predicts a broad distribution over possible
human observations and beliefs over all objects. After receiving communication
that the object’s color is brown, the uncertainty over human observations and
human belief shrinks to the brown cabinet and table. Thereafter, it receives
communication that the object type is a table, allowing it to determine the
object in question and the distributions shrink to what the human actually
sees. In contrast, the Baseline-S model consistently outputs a single image
(Fig.7) which is often blurry, failing to correctly model uncertainty under
partial observations (similar results were obtained for Baseline-D).

Figure 8: Human-subject experiment results: KL divergence between predicted
and actual human belief. The scores were averaged over the episode length.
Overall, the LEAPT-Transformer achieves the best performance (lowest KL
divergence).

##  V Human-Subject Experiments

In this section, we describe preliminary human-subject experiments designed to
test the viability of LEAPT when interacting with real humans. A total of 12
participants (mean age = 22.2, 3 females, 9 males) were recruited from the
university community. The participants were asked to perform the Fetch-Tool
and the Table-Assembly tasks using a VR setup (Fig.3.D). In the Fetch-Tool
task, the participants sat in front of the (simulated) workstation, and the
robot was situated on the other side. The robot queried about the type or
color of the item shown on the screen, and the participants would respond
accordingly. In the Table-Assembly task, the participants stood in front of
the table, where they could see the peg and hole. The robot was on the other
side, gripping the table, which visually occluded the peg and hole. The robot
queried the participants on the relative distance of the hole from the peg and
moved the table accordingly until the participants confirmed that the peg was
aligned with the hole.

Evaluation Our evaluations are similar to the simulated human experiments
described in the previous section. From the robot observations and human
communication collected during the human-subject experiments, we use our
models to derive the estimated empirical human belief. We then evaluate it
with the ground truth distribution set based on the information provided in
human communication. For instance, in the Fetch-Tool task, when a human
communicated that they saw a table, the ground truth distribution was set as
uniform over all kinds of tables. We measure the KL divergence
$\mathbb{D}_{\mathrm{KL}}\left[p(c(\hat{y}_{t}^{\textrm{H}})|x_{1:t}^{\textrm{R}})\|p(c({y}_{t}^{\textrm{H}})|x_{1:t}^{\textrm{R}})\right]$,
where $c$ is a classifier/regression model.

Results. Overall, the LEAPT-Transformer attained the best performance. We
compared the methods using repeated measures one-way ANOVA, followed by
selected pairwise t-tests with adjusted $\alpha=0.008$ using Bonferroni
correction. Fig.8. shows that LEAPT-Transformer outperforms the baselines
significantly in the Fetch-Tool task ($F_{3,33}=966.324,p<0.001$; LEAPT-
Transformer vs. Baseline-D: $t(11)=9.209,p<0.001$ and LEAPT-Transformer vs.
Baseline-S: $t(11)=19.78,p<0.001$) and the Table-Assembly task
($F_{3,33}=7098.428,p<0.001$; LEAPT-Transformer vs. Baseline-D:
$t(11)=140.874,p<0.001$ and LEAPT-Transformer vs. Baseline-S:
$t(11)=52.906,p<0.001$).

##  VI Conclusions

This paper introduces LEAPT, a framework that empowers robots to conduct
visual and conceptual perspective-taking in partially-observable environments.
The key innovation is a specially-designed latent multi-modal state-space
model, which enables more consistent beliefs to be maintained when
observations are limited. Our experiments on three tasks show that LEAPT leads
to better performance compared to deterministic approaches and standard
latent-state space models when estimating the visual perceptions and beliefs
of other agents.

Moving forward, we plan to apply LEAPT to other robot tasks and in real-world
environments. In addition, LEAPT currently exhibits several limitations that
would make for interesting future work. First, LEAPT requires human poses
during testing. To address this, an alternative approach could involve an
additional distribution over the poses given partial observations of the human
and a learnt dynamics model. Second, the model needs to be trained with
sufficient views of the world to be effective. Inadequate training data may
impede the model’s ability to handle complex environments. Third, the human
model in LEAPT is based on the robot self-model, thus, falls short in
capturing certain aspects of the human, such as trust [23, 24, 25] and
emotions [26]. Lastly, communication in LEAPT is based on simple, pre-defined
sentence templates and we can integrate large language models (e.g., [27, 28])
to enhance LEAPT’s communication ability.

## References

  * [1] J. G. Trafton, N. L. Cassimatis, M. D. Bugajska, D. P. Brock, F. E. Mintz, and A. C. Schultz, “Enabling effective human-robot interaction using perspective-taking in robots,” _IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans_ , vol. 35, no. 4, pp. 460–470, 2005.
  * [2] M. Berlin, J. Gray, A. L. Thomaz, and C. Breazeal, “Perspective taking: An organizing principle for learning in human-robot interaction,” in _AAAI_ , vol. 2, 2006, pp. 1444–1450.
  * [3] C. Breazeal, M. Berlin, A. Brooks, J. Gray, and A. L. Thomaz, “Using perspective taking to learn from ambiguous demonstrations,” _Robotics and autonomous systems_ , vol. 54, no. 5, pp. 385–393, 2006.
  * [4] M. Johnson and Y. Demiris, “Perceptual perspective taking and action recognition,” _International Journal of Advanced Robotic Systems_ , vol. 2, no. 4, p. 32, 2005.
  * [5] G. Milliez, M. Warnier, A. Clodic, and R. Alami, “A framework for endowing an interactive robot with reasoning capabilities about perspective-taking and belief management,” in _The 23rd IEEE international symposium on robot and human interactive communication_. IEEE, 2014, pp. 1103–1109.
  * [6] B. Chen, Y. Hu, R. Kwiatkowski, S. Song, and H. Lipson, “Visual perspective taking for opponent behavior modeling,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 13 678–13 685.
  * [7] K. Chen, J. Fong, and H. Soh, “Mirror: Differentiable deep social projection for assistive human-robot communication,” in _Proceedings of Robotics: Science and Systems_ , June 2022.
  * [8] N. Newcombe and J. Huttenlocher, “Children’s early ability to solve perspective-taking problems.” _Developmental psychology_ , vol. 28, no. 4, p. 635, 1992.
  * [9] A. Pearson, D. Ropar, and A. F. de C. Hamilton, “A review of visual perspective taking in autism spectrum disorder,” _Frontiers in human neuroscience_ , vol. 7, p. 652, 2013.
  * [10] R. A. Ryskin, A. S. Benjamin, J. Tullis, and S. Brown-Schmidt, “Perspective-taking in comprehension, production, and memory: An individual differences approach.” _Journal of Experimental Psychology: General_ , vol. 144, no. 5, p. 898, 2015.
  * [11] X. Zhao and B. F. Malle, “Spontaneous perspective taking toward robots: The unique impact of humanlike appearance,” _Cognition_ , vol. 224, p. 105076, 2022.
  * [12] C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martín-Martín, C. Wang, G. Levine, M. Lingelbach, J. Sun, _et al._ , “Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation,” in _Conference on Robot Learning_. PMLR, 2023, pp. 80–93.
  * [13] R. Gong, J. Huang, Y. Zhao, H. Geng, X. Gao, Q. Wu, W. Ai, Z. Zhou, D. Terzopoulos, S.-C. Zhu, _et al._ , “Arnold: A benchmark for language-grounded task learning with continuous states in realistic 3d scenes,” _arXiv preprint arXiv:2304.04321_ , 2023.
  * [14] Y.-W. Chao, C. Paxton, Y. Xiang, W. Yang, B. Sundaralingam, T. Chen, A. Murali, M. Cakmak, and D. Fox, “Handoversim: A simulation framework and benchmark for human-to-robot object handovers,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 6941–6947.
  * [15] A. Labash, J. Aru, T. Matiisen, A. Tampuu, and R. Vicente, “Perspective taking in deep reinforcement learning agents,” _Frontiers in Computational Neuroscience_ , vol. 14, p. 69, 2020.
  * [16] G. Papoudakis and S. V. Albrecht, “Variational autoencoders for opponent modeling in multi-agent systems,” _arXiv preprint arXiv:2001.10829_ , 2020\.
  * [17] G. Papoudakis, F. Christianos, and S. Albrecht, “Agent modelling under partial observability for deep reinforcement learning,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 19 210–19 222, 2021.
  * [18] N. Rabinowitz, F. Perbet, F. Song, C. Zhang, S. A. Eslami, and M. Botvinick, “Machine theory of mind,” in _International conference on machine learning_. PMLR, 2018, pp. 4218–4227.
  * [19] S. Reddy, S. Levine, and A. Dragan, “Assisted perception: optimizing observations to communicate state,” in _Conference on Robot Learning_. PMLR, 2021, pp. 748–764.
  * [20] K. Chen, Y. Lee, and H. Soh, “Multi-modal mutual information (mummi) training for robust self-supervised deep reinforcement learning,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2021.
  * [21] M. Wu and N. Goodman, “Multimodal generative models for scalable weakly-supervised learning,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [22] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [23] M. Chen, S. Nikolaidis, H. Soh, D. Hsu, and S. Srinivasa, “Planning with trust for human-robot collaboration,” in _Proceedings of the 2018 ACM/IEEE international conference on human-robot interaction_ , 2018, pp. 307–315.
  * [24] J. Lee, J. Fong, B. C. Kok, and H. Soh, “Getting to know one another: Calibrating intent, capabilities and trust for human-robot collaboration,” p. 6296–6303, 2020.
  * [25] B. C. Kok and H. Soh, “Trust in robots: Challenges and opportunities,” _Current Robotics Reports_ , vol. 1, pp. 297–309, 2020.
  * [26] D. C. Ong, H. Soh, J. Zaki, and N. D. Goodman, “Applying probabilistic programming to affective computing,” _IEEE Transactions on Affective Computing_ , vol. 12, no. 2, pp. 306–317, 2019.
  * [27] B. Zhang and H. Soh, “Large language models as zero-shot human models for human-robot interaction,” 2023. [Online]. Available: https://arxiv.org/abs/2303.03548
  * [28] Y. Xie, C. Yu, T. Zhu, J. Bai, Z. Gong, and H. Soh, “Translating natural language to planning goals with large-language models,” 2023. [Online]. Available: https://arxiv.org/abs/2302.05128

##  VII Appendix

###  VII-A Experiment Observations

Figure 9: Sally-Anne environment observations. (A) Full observations. Drill at
the bottom denotes that left of the robot and vice versa. (B) Robot partial
observation. It can only observe if the boxes are switched or not. (C) Human
partial observations. Human can see the location of the item when he is in
front of the boxes. Figure 10: Fetch tool environment full observations. (A)
Full observations. (B) Robot observations, the robot is looking at the table
and does not observe the object. (C) Human observations. Figure 11: Table
assembly environment observations. (A) Full observation which shows the top-
down view of the position of hole (square) and table (circle). (B) Robot’s
partial observation. (C) Human partial observations.