#  Communicating Inferred Goals with Passive Augmented Reality and Active
Haptic Feedback

James F. Mullen Jr, Josh Mosier, Sounak Chakrabarti, Anqi Chen, Tyler White,
and Dylan P. Losey The student authors were equal members of a senior design
team.The authors are with the Collaborative Robotics Lab, Department of
Mechanical Engineering, Virginia Tech, Blacksburg, VA 24060 USA (e-mail:
<EMAIL_ADDRESS>losey@vt.edu).

###### Abstract

Robots learn as they interact with humans. Consider a human teleoperating an
assistive robot arm: as the human guides and corrects the arm’s motion, the
robot gathers information about the human’s desired task. But how does the
human know what their robot has inferred? Today’s approaches often focus on
conveying intent: for instance, using legible motions or gestures to indicate
what the robot is planning. However, closing the loop on robot inference
requires more than just revealing the robot’s current policy: the robot should
also display the alternatives it thinks are likely, and prompt the human
teacher when additional guidance is necessary. In this paper we propose a
multimodal approach for communicating robot inference that combines both
passive and active feedback. Specifically, we leverage information-rich
augmented reality to passively visualize what the robot has inferred, and
attention-grabbing haptic wristbands to actively prompt and direct the human’s
teaching. We apply our system to shared autonomy tasks where the robot must
infer the human’s goal in real-time. Within this context, we integrate passive
and active modalities into a single algorithmic framework that determines when
and which type of feedback to provide. Combining both passive and active
feedback experimentally outperforms single modality baselines; during an in-
person user study, we demonstrate that our integrated approach increases how
efficiently humans teach the robot while simultaneously decreasing the amount
of time humans spend interacting with the robot. Videos here:
https://youtu.be/swq_u4iIP-g

###### Index Terms:

Haptics and Haptic Interfaces, Virtual Reality and Interfaces, Intention
Recognition

##  I Introduction

Figure 1: Human teleoperating a $7$-DoF robot arm. (Top) The robot believes it
should place the cup on a shelf, and passively visualizes the likely options
in augmented reality. (Middle) But the robot is still uncertain about the
right shelf. To elicit informative feedback, the robot actively signals the
haptic wristband, and prompts the human to guide the robot either up or down.
(Bottom) The human responds by moving down. In augmented reality, the robot
shows that it has inferred the bottom shelf is the human’s goal.

Imagine teaching an assistive robot arm how to put away a coffee cup (see Fig.
1). You want the robot to place this cup upright on the lower shelf — as the
robot moves toward the shelves, you correct its mistakes and guide the robot
towards your desired goal. The robot gathers information in real-time from
your corrections, updating its understanding of how and where it should carry
the coffee cup. But how do you know what the robot has inferred? Ideally you
would like to know when you have taught the robot enough, so that you can step
away and do something else. But without feedback you are stuck watching the
robot throughout the entire task, unsure whether it fully understands your
goal, or what corrections you should provide to remove any remaining
confusion!

Existing research enables robots to convey their intent to nearby humans [1,
2, 3, 4, 5]. These prior works study a spectrum of different modalities,
including legible motions, language, gaze, gestures, and displays. Each new
modality offers a separate channel for providing intent feedback to the human.
But communicating inferred behavior requires more than just intent: the robot
learner must indicate what it thinks it should do, display its confidence that
this inferred behavior is correct, and even prompt additional human teaching
when unsure. As robot learners move from communicating their immediate intent
to proactively involving human teachers, we propose that robots synthesize
multiple modalities to close-the-loop and make it clear what they do and do
not know.

Selecting the right modalities often depends on the current user and task. But
— whichever modalities we choose — our central hypothesis is that combining
passive and active feedback will better bring the human into the learning loop
than systems which only leverage a single feedback type:

Robots should _passively_ convey what they have inferred, and _actively_
prompt the human when teaching is required.

Specifically, in this paper we integrate augmented reality displays and
wearable haptic feedback to develop a multimodal interface that passively
communicates learning and actively elicits teaching. We apply our system to
shared autonomy settings [6, 7, 8]. Here the robot arm knows a discrete set of
possible goals, and infers the desired goal from the human’s teleoperation
inputs. As shown in Fig. 1, our interface passively displays the different
ways to complete the task that the robot thinks are likely, and actively
prompts the human when further teaching is required. We conduct user studies
to test how this feedback system affects interactions between human teachers
and robot learners. When compared to single modality baselines, we find that
the combination of passive and active feedback i) prevents users from
continuing to teach the robot after it has already learned what they want, ii)
reduces the amount of time users spend monitoring the robot, and iii) improves
user teaching so that the robot infers what the human wants from fewer
interactions.

Overall, our proposed system is a step towards transparent robot learning and
accelerated human teaching. We make the following contributions:

Combining Multiple Feedback Modalities. Unlike related works that only
consider a single modality, we integrate an existing augmented reality head-
mounted display with a novel haptic wristband to convey multimodal information
and proactive alerts about robot inference to human teachers.

Converting Robot Inference to Passive and Active Feedback. Specifically in the
context of shared autonomy, we formulate an approach that maps robot inference
to intuitive multimodal cues. The resulting algorithm determines when to
provide which type of feedback, identifies critical states where additional
human teaching is especially important, and prompts the most informative
direction of human input.

Conducting a User Study. We compare passive and active feedback to single-
modality visual, haptic, and augmented reality baselines across an in-person
user study. Participants traded-off between teaching the robot arm and
performing a distractor task. We find that our approach reduces the amount of
time a user must spend focused on the robot learner, and also improves the
user’s teaching efficiency.

##  II Related Work

We build on prior work towards explainable robot learners. This includes
research that generally communicates robot intent to nearby humans, research
that specifically focuses on augmented reality and haptic displays, and
research that reveals what the robot is inferring during shared autonomy.

Communicating Robot Intent. For humans and robots to seamlessly collaborate it
must be clear to the human what the robot is trying to do [1]. Recent works
have developed a variety of different modalities that robots leverage to
convey their intent. For instance, robots can use legible motions to make
their goals clear to human onlookers [2, 9], or incorporate natural language
to communicate their objective [3, 10]. Non-verbal communication such as
gestures [5, 11] or gaze [4] are also expressive channels for indicating where
the robot plans to go. Besides these implicit cues, robots employ explicit
indicators in the form of projections [12, 13] — where they render 2D images
onto the environment — or on-board lights [14] — which are functionally
similar to turn signals on cars.

But while many distinct modalities are available for conveying intent, we
emphasize that this is different from communicating robot inference. Think of
our motivating example: communicating the robot’s intent is equivalent to
showing the final cup location and orientation that the robot thinks is most
likely and currently intends to reach. Yet this alone is not enough to make
the robot’s learning transparent: we may also need to highlight the
alternative goals and trajectories the robot is reasoning over, or ask the
human for additional feedback to determine which goal is correct.

Augmented Reality and Haptics. In order to convey detailed and attention
grabbing signals about robot inference we turn to augmented reality and haptic
feedback. Augmented reality head-mounted displays have previously been used to
facilitate human-robot interaction [15, 16]. Most relevant here are [17, 18,
19, 20], where the authors leverage augmented reality to convey the robot’s
internal state, visualize where the robot plans to go, and establish
bidirectional communication with the human. This visual overlay succeeds when
users are looking directly at the robot — but when users look away, these
graphics are no longer in view. We explore settings where the human must split
their attention between the robot and other tasks. Accordingly, we incorporate
haptic wristbands that humans wear without encumbering their hands. Recent
works have designed similar wristbands to provide tactile feedback for
augmented reality [21, 22].

Overall, today’s research on augmented reality and haptic feedback either
develops these two modalities separately, or else utilizes haptics to make
interactions with the virtual world seem more realistic. By contrast, we
combine the two modalities into a single framework to close-the-loop during
robot inference. Both active and passive feedback are important because they
serve complementary roles: we leverage augmented reality to visualize the
high-dimensional behaviors the robot has inferred, and apply haptics to spark
and direct user inputs when additional teaching is required.

Revealing Robot Inference. Creating learners that are interpretable by human
users is an active area of study [23]. For instance, in [24] the robot
displays informative examples of its learned behavior on a computer screen,
enabling users to extrapolate the robot’s underlying objective. Understanding
robot learning becomes especially challenging when that learning consists of
networks with thousands of weights. However, shared autonomy requires only
low-dimensional inference: here the robot is inferring the human’s desired
goal from a discrete set of options [6, 7, 8]. Related work has taken
advantage of this structure to visualize the robot’s inferred goal using
augmented reality [25, 26].

Figure 2: Our haptic wristband for delivering active feedback. This device can
both gently squeeze the wearer’s wrist and render localized vibration
patterns. Left: Exploded view of the electronics housing. The microcontroller
(1) sits over the battery (2) and squeeze mechanism (3). Three motor drivers
responsible for controlling vibration (4) are accessible behind the front
panel. Middle-Left: Fully assembled haptic device with cord (5). Middle-Right:
Vibrotactors (6) are held in place by printed casings (7). Right: To prompt
the human we squeeze their wrist and activate multiple vibrotactors in a pre-
defined pattern. For instance, powering the left and right vibrotactors
encourages the human to guide the robot either right or left.

Importantly, this related research provides purely passive feedback to the
user: i.e., while the user watches the robot they see overlays indicating
where the robot thinks it should go. But we are also interested in active
feedback which influences the human to provide new and informative
demonstrations. We envision a system that prompts the human when the robot is
confused, and even suggests to the human the direction of input that would
best reduce the robot’s uncertainty about the human’s desired goal.

##  III Developing Multimodal Interfaces for Communicating Robot Inference

We seek transparent robot learners that involve humans in the teaching
process. Our hypothesis is that we can better the human into the learning loop
with multimodal feedback — both in the form of passive visualizations, which
display the goals that the robot has inferred, and active prompts, which
direct the human towards effective teaching inputs. In this section we develop
complementary interfaces for each type of feedback. We particularly focus on
the design of a custom haptic wristband for eliciting human teaching.

###  III-A Passive Visualization: Augmented Reality

We start by leveraging an existing augmented reality head-mounted display
(Microsoft HoloLens) to overlay graphics onto the real-world environment.
These visualizations can passively convey complex information — for example,
as the human looks at the robot arm, they see high-dimensional trajectories
that the robot has learned from their demonstrations. But how should we design
these visualizations to intuitively convey what the robot does and does not
know?

In order to create user-friendly visualizations we take inspiration from
successful prototypes in prior work [18, 25, 26]. After implementing these
prototypes across short pilot tests — where humans teleoperated the robot arm
to perform simple shared autonomy tasks — we arrived at a design similar to
NavPoints [17]. As shown in Fig. 1, here the robot augments the environment by
displaying the position and orientation of end-effector waypoints along its
likely trajectories. The position of each waypoint is marked with a sphere,
while its orientation is denoted by an arrow.

This visualization design is suitable for shared autonomy tasks where the
robot reasons over multiple possible goals. Returning to our motivating
example, imagine the robot is unsure whether it should place the cup on the
top shelf or the bottom shelf: our design visualizes this uncertainty through
two trajectories starting at the robot’s end-effector and going to both of the
shelves. The starting point of every trajectory is affixed to the robot’s end-
effector and moves with that robot. When the robot updates its understanding
of the human’s goal, we now vary the color and transparency of each trajectory
to passively signal what the robot has inferred: our algorithm for mapping
inference to this augmented reality visualization is introduced in Section IV.

###  III-B Active Prompts: Haptic Wristband

To provide haptic feedback we designed and built a low-cost, wireless, and
lightweight wristband (see Fig. 2). Users wear this device just like a watch.
The wristband provides two separate types of active haptic signals: gently
squeezing the human’s wrist and vibrating in patterns distributed around the
band. Together, these two alerts convey low-dimensional and targeted
information. For instance, when the robot needs additional guidance, the
device lightly squeezes the human’s wrist and prompts the human by vibrating
in the direction where user inputs would be most informative.

Squeeze Feedback. The haptic wristband generates squeeze forces through a cord
and reel system. This system is controlled by a high torque N$20$ DC motor
which sits centrally in the main housing of the device (refer to Fig. 2). When
we actuate the DC motor in one direction, the cord is retracted into the
central housing equally from both sides; actuating the motor in the opposite
direction releases the cord and relaxes the band. The cord is routed through
the components of the haptic device and is never in direct contact with the
wearer’s skin. To make this device more comfortable, we normally keep the band
in a relaxed state, and only squeeze on the human’s wrist at short intervals
to provide notifications.

Vibrotactile Feedback. Localized vibrations are output by six cylindrical ERM
motors that are evenly spaced around our flexible 3D printed wristband. Each
ERM motor (i.e., each vibrotactor) sits rigidly in a resin-printed housing
which transfers vibrations directly onto the wearer’s skin. We created a
compliant accordion-style wristband so that the band fits varying wrist shapes
while isolating vibrations from adjacent vibrotactors. Activating multiple
vibrotactors in sequence creates patterns which carry additional information:
for example, we can activate the top and bottom vibrotactors to indicate to
the human to guide the robot either up or down.

##  IV Mapping Robot Inference to
Passive and Active Feedback

So far we have worked on two feedback modalities in parallel: we created
visual designs for existing augmented reality displays and developed a
complementary haptic wristband. In this section we combine these two feedback
mechanisms into a single formulation for communicating robot inference within
shared autonomy settings. We derive a decision rule for providing each type of
feedback — augmented reality is used for passive, information rich
visualizations of goals the robot thinks are likely, while the haptic
wristband is used to actively signal the user and elicit additional teaching.

Setting. We start by returning to our example from Fig. 1. Here the human is
teleoperating a robot arm, and the robot is inferring the human’s goal to
autonomously assist the human. Let $s\in\mathcal{S}$ be the robot’s state
(e.g., end-effector position), let $a\in\mathcal{A}$ be the robot’s action
(e.g., end-effector velocity), and let transition function $T(s,a)$ capture
the robot’s dynamics. We consider systems with continuous state and action
spaces. The human has in mind a goal $g\in\mathcal{G}$ that they want the
robot arm to reach (e.g., placing the cup on the lower shelf). Their reward
function $R_{g}(s,a)$ depends upon this chosen goal. The robot seeks to
maximize the human’s cumulative reward subject to uncertainty over the human’s
goal111This is an instance of a partially observable Markov decision process
where $g$ is the unknown state and Eq. (4) is the observation model [7]..

Consistent with prior work on shared autonomy [6, 7, 8], we assume that the
set of possible goals $\mathcal{G}$ is discrete and the robot knows
$\mathcal{G}$ a priori. In practice, these goals can be anywhere in the
robot’s workspace: the only constraint is that the robot must know the
possible goal locations $\mathcal{G}$ prior to interaction. But while the
robot knows the human’s candidate goals, it does not know the human’s goal
during the current task. Instead, the robot maintains a belief $b$ over the
set of possible goals. This belief is a probability distribution, i.e.,
$b(g)=1$ indicates that the robot is completely convinced that
$g\in\mathcal{G}$ is the human’s current goal.

Without any insight from the human the robot would have no way of updating $b$
and inferring the preferred goal. But the robot is not performing this task
alone — both the human and robot share control over the robot’s motion. The
human applies joystick inputs to teleoperate the robot arm, so that
$a_{h}\in\mathcal{A}$ is the human’s commanded action. The robot linearly
blends this human input with its own autonomous assistance
$a_{r}\in\mathcal{A}$ to get the overall action [6, 8]:

$a=(1-\alpha)\cdot a_{h}+\alpha\cdot a_{r}$ (1)

where $\alpha\in[0,1]$ arbitrates control between human and robot. During our
user study we select $\alpha=0.4$ so that the human’s actions are given more
weight than the robot’s assistance.

If action $a_{h}$ results from the human’s teleoperation input, how do we
obtain $a_{r}$? The robot selects this autonomous action to assist the human
based on what it has inferred so far. Assuming that the robot will fully
observe the human’s desired goal at the next timestep, the optimal autonomous
action at state $s$ given belief $b$ becomes [7]:

$a_{r}=\text{arg}\max_{a\in\mathcal{A}}\leavevmode\nobreak\
\sum_{g\in\mathcal{G}}b(g)\cdot Q_{g}(s,a)$ (2)

Here $Q_{g}(s,a)$ is the cumulative reward of taking action $a$ in state $s$
and then optimally completing the task to reach goal $g$. We use
$Q_{g}(s,a)=-\|T(s,a)-g\|^{2}$ in our user studies to compute this in closed
form and perform inference at $1$ kHz. Intuitively, Equation (2) encourages
the robot to assist across all likely goals. As the robot becomes increasingly
confident in a particular goal $g$, it leverages Equation (2) to move directly
towards that goal.

###  IV-A Conveying Inference with Passive Visualizations

The robot infers the human’s preferred goal by observing their teleoperation
commands $a_{h}$. We propose to utilize augmented reality head-mounted
displays to visualize what the robot is inferring. Augmented reality is
suitable here because it offers a user-friendly and intuitive way to convey
dense, high-dimensional information (e.g., trajectories).

Belief Conditioning. Jumping back to our motivating example, imagine that you
are trying to teach the robot arm. At the start of the task the robot has a
uniform belief $b$ over different ways to place the cup on the shelves. But
you do not know this — perhaps you think you need to teach the robot to move
towards the shelves in the first place. To reveal what the robot already
knows, here the visualization should show multiple, equally likely
trajectories reaching for the shelves. As the robot starts moving to the
shelves, you begin to correct its behavior, and the robot updates it’s belief
to place higher likelihood on your desired goal. Now the visualization should
change: instead of showing equal trajectories for every goal, the robot should
emphasize the trajectories that correspond to your likely goal(s). Put another
way, the robot’s visualization not only depends on where the robot and goals
are located, but it also depends on the robot’s belief.

Inference. In order to display the robot’s inference we must map beliefs to
visualizations. Before we do this, however, we first need to determine how the
robot infers the human’s goal. Here we apply Bayesian inference to extract the
human’s goal from their action inputs. Assuming that the human’s inputs
$a_{h}$ are conditionally independent, we reach:

$b^{t+1}(g)=P(g\mid\mathcal{D})\propto P(a_{h}^{t}\mid s^{t},g)\cdot b^{t}(g)$
(3)

where $\mathcal{D}=\\{(s^{1},a_{h}^{1}),\ldots(s^{t},a_{h}^{t})\\}$ is the set
of human inputs and $t$ is the current timestep. Notice that Equation (3) only
depends on human actions and not the autonomous assistance — this prevents the
robot from incorrectly learning from itself through a feedback loop. In
practice, to evaluate Equation (3) we need the likelihood function
$P(a_{h}\mid s,g)$, which models how probable it is that the human will choose
action $a_{h}$ given that their goal is $g$ and the robot is in state $s$.
Within shared autonomy, this likelihood function commonly follows the
Boltzmann-rational model [8]:

$P(a_{h}\mid s,g)=\frac{e^{\,\beta Q_{g}(s,a_{h})}}{\int e^{\,\beta
Q_{g}(s,a)}\leavevmode\nobreak\ da}\approx\frac{e^{\,\beta
Q_{g}(s,a_{h})}}{e^{\,\beta Q_{g}(s,a_{g})}}$ (4)

On the far right side we use Laplace’s method to approximate the denominator,
so that $a_{g}$ is the optimal action to reach goal $g$. The constant
$\beta\geq 0$ is a tunable hyperparameter which determines how sensitive the
robot is to each human input. When $\beta\rightarrow 0$ the robot treats the
human’s actions as random, and when $\beta\rightarrow\infty$ the robot models
the human as perfectly rational. We used $\beta=0.1$ for our user study.

Visual Mapping. Combining Equations (3) and (4) we have a way to update the
robot’s belief. Our final step is to map this changing belief into an
augmented reality visualization. Although there are an infinite number of
valid possibilities, we draw from prior work to identify a natural mapping
that requires minimal user interpretation [17, 18, 25].

Specifically, we map the robot’s inferred belief over each goal to the color
and transparency of the corresponding waypoints (see Fig. 1). When the robot
is sure that a given $g$ is not the human’s desired goal, i.e.,
$b(g)\rightarrow 0$, the trajectory moving to that goal (and the goal itself)
are rendered as completely transparent. Conversely, as $b(g)\rightarrow 1$ and
the robot becomes more confident that $g$ is the human’s goal, we linearly
interpolate the waypoint color between a dull gray and vibrant orange. This
mapping intuitively highlights goals the robot thinks are likely while
simultaneously hiding goals that the human does not want.

###  IV-B Eliciting Teaching with Active Prompts

We hypothesize that our augmented reality feedback will reveal the robot’s
inference while the human is watching the robot (and can see the visual
overlays). But what happens when the human looks away to do something else? So
far the feedback has been purely passive — providing information only when the
human seeks it out. Next we turn to active haptic feedback that purposely
alerts the human and guides their teaching, even if they are not concentrating
on the robot. Wearable haptics is suitable here because it offers direct,
attention grabbing signals without encumbering the human.

We break this problem down into two parts: i) determining when to provide
haptic prompts and ii) selecting which haptic prompts will elicit useful human
teaching.

Figure 3: Example robot and human behavior during the Avoiding task. The robot
is trying to drop a strawberry on the plate, but does not know how to avoid
the obstacle. When users receive feedback on the GUI (top row), they are more
tentative, and spend extra time watching and correcting the robot. This leads
to higher interaction time, lower teaching efficiency, and lower scores on the
distractor task. By contrast, humans with AR+Haptic feedback (bottom row) are
free to focus on the distractor task. When the robot needs help they get an
active haptic notification, and passively observe how the robot learns from
their corrections. This multimodal feedback enables the human to efficiently
teach the robot without having to constantly monitor its behavior.

When to Prompt. In our motivating example you are trying to teach the robot to
place a coffee cup upright on the lower shelf. At the beginning of the task
you notice that the robot is moving towards the shelves — and since the robot
is going in the right direction, you turn away to focus on something else. The
robot still needs to figure out whether it should place the cup on the top
shelf or the lower shelf; but when is the right time to ask for your feedback?
At the start of the task the shelves are far away, and the trajectories to
both the top and bottom shelf are very similar. However, as the robot
approaches the shelves, the trajectories for the two likely goals diverge, and
the robot needs to make a decision (should I move up or down?). Intuitively,
we want to alert the human at important decision points.

Drawing from prior research on critical states [27] and inverse reinforcement
learning [28], we define these decision points as states $s$ where taking the
robot’s autonomous action $a_{r}$ will produce a much worse result than acting
optimally:

$\mathcal{C}(s)=\sum_{g\in\mathcal{G}}b(g)\cdot\big{(}Q_{g}(s,a_{g})-Q_{g}(s,a_{r})\big{)}$
(5)

Here $\mathcal{C}$ is a measure of how critical the state $s$ is, and if
$\mathcal{C}$ exceeds some threshold $\sigma$ then the robot is in a critical
state. Recall that $a_{g}$ is the optimal action for goal $g$ (e.g., moving in
a straight line from $s$ towards $g$). Our choice of Equation (5) highlights
states where multiple likely goals require very different actions, and the
robot needs more information in order to select the correct action.

Which Prompt to Use. We keep track of Equation (5) during the task, and use
the haptic wristband to alert the human when $C(s)>\sigma$. This prompt brings
the human’s attention back to the robot — but what feedback should the human
teacher provide when they receive an alert? In our running example, the robot
has reached the shelves and is unsure whether the cup belongs on the upper or
lower shelf. Here the robot would learn the most from an input $a_{h}$ which
moves the robot either up or down. More generally, we want to encourage the
user to teleoperate the robot in the direction that best removes uncertainty
over the human’s goal.

Recall that our haptic device has vibrotactors distributed around the wearer’s
wrist (see Fig. 2). By actuating these vibrotactors in patterns we render
direction-based prompts to the human [29]. Let $\mathcal{U}$ be the set of
prompts we can render: for instance, $u_{z}\in\mathcal{U}$ alternates between
actuating the top and bottom vibrotactors to prompt the user to teleoperate
the robot up or down. Given that the robot has reached a critical state, which
stimuli $u\in\mathcal{U}$ should we send? Intuitively, we search for the
stimulus $u$ which — if the human responds — provides the most information
about the human’s unknown goal. Formally, we elicit the direction of human
teaching that greedily maximizes the robot’s expected information gain:

$u^{*}(s)=\text{arg}\max_{u\in\mathcal{U}}\leavevmode\nobreak\ I(g,a_{h}\mid
u,s,b)$ (6)

Here $I$ is mutual information and $a_{h}$ is the human’s response to prompt
$u$. When solving for $u^{*}$ we assume that the human will comply with the
robot’s prompt — so that if the robot chooses $u_{z}$, the human will input
either $a_{h}=up$ or $a_{h}=down$. Expanding Equation (6), we obtain [30]:

$u^{*}(s)=\text{arg}\max_{u\in\mathcal{U}}\sum_{a_{h}\in
u}\sum_{g\in\mathcal{G}}b(g)P(a_{h}\mid u,s,g)\\\
\cdot\log_{2}\frac{P(a_{h}\mid
u,s,g)}{\sum_{g^{\prime}\in\mathcal{G}}b(g^{\prime})P(a_{h}\mid u,s,g)}$ (7)

The key to evaluating Equation (7) is modeling the likelihood function
$P(a_{h}\mid u,s,g)$, which captures how likely it is that the human will
respond with action $a_{h}$ to prompt $u$. In other words, if we signal prompt
$u_{z}$, what is the probability the human applies $a_{h}=up$ instead of
$a_{h}=down$? Just like before, we leverage the Boltzmann-rational model:

$P(a_{h}\mid u,s,g)=\frac{e^{\beta Q_{g}(s,a_{h})}}{\sum_{a\in
u}\leavevmode\nobreak\ e^{\beta Q_{g}(s,a)}}$ (8)

Comparing Equation (8) to Equation (4), the only difference is our assumption
that the human will respond by moving the robot in one of the two discrete
directions suggested by $u$. We emphasize that this assumption is purely used
to solve for the optimal prompt, and is not imposed in our user study.

Haptic Mapping. Putting it all together, Equation (5) lets us know when to
provide haptic prompts, and Equations (7) and (8) tell us which direction-
based prompts will elicit the most information. Within the context of our
motivating example: as the robot nears the shelves it realizes that it needs
more information, and alerts the user by vibrating the haptic wristband on the
top and bottom — encouraging to the human to teleoperate the robot either up
or down.

Figure 4: Objective results from our user study. The study included four
tasks, and here we report the aggregated results across all tasks. Asterisks
denote statistically significant comparisons ($p<.01$) and error bars show
standard error. We found that AR+Haptic reduced interaction time and increased
teaching efficiency as compared to the alternatives. There was no significant
difference in distractor score between AR+Haptic and Haptic ($p=0.65$).

##  V User Study

To test how passive and active multimodal feedback closes the loop during
robot inference, we conducted a user study motivated by assistive robotics.
Participants interacted with a 2-DoF joystick to teleoperate a 7-DoF robot.
The robot shared autonomy with the participant: as the robot inferred the
human’s goal in real-time during the task, it leveraged different types of
feedback to communicate what it had inferred back to the user. We designed
experiments where participants had to trade-off between teaching on the robot
and focusing on a distractor task.

Independent Variables. We compared four different feedback conditions:

  * •

A graphical user interface (GUI) on the computer screen

  * •

Passive augmented reality feedback (AR)

  * •

Active haptic feedback (Haptic)

  * •

Augmented reality + haptic feedback (AR+Haptic)

Within the GUI baseline we displayed a graphical user interface on the same
computer screen as the distractor task. This GUI provided both active and
passive feedback: we displayed the robot’s belief over each goal, and printed
a text alert when the robot reached a critical state. The AR and Haptic
baselines are suitable for purely passive or active feedback. In all cases
where we used Haptic to provide active cues the participants responded by
teaching the robot. Similarly, participants with AR feedback were confident
that they understood the robot’s intent (see Fig. 5). Finally, the AR+Haptic
condition is our proposed approach from Section IV that combines both passive
and active feedback.

Experimental Setup. Participants completed four tasks with every feedback
condition. In the Placing task (see Fig. 1) participants taught the robot to
place a coffee cup on the shelf. There were $8$ candidate goals the robot
considered: the front or back of the top shelf, the front or back of the
bottom shelf, and putting the cup down either upright or on its side. In the
Avoiding task (see Fig. 3) users adjusted the robot’s motion to avoid an
obstacle. The robot initially moved directly towards this obstacle and was
unsure if it needed to avoid it, and if so, which way it should go around.
Here the robot had $4$ discrete candidate goals: moving through, to the left,
to the right, or above the obstacle. Both the Sorting task and the Forgetting
task started in the same way with the robot carrying a bottle of glue towards
the shelves. The robot had $4$ candidate goals corresponding to different
locations on the shelf. In the Sorting task the user taught the robot to sort
the glue on the lower shelf. By contrast, half-way through the Forgetting task
the robot automatically became confused about the human’s goal (i.e., it
forgot what it had learned), so that it required additional teaching.

During all four tasks users divided their attention between teaching the robot
and playing a distractor game. Within this game participants tried to track a
moving target with their mouse, and accrued points when their mouse was inside
the target. Because the game was played on a screen orthogonal to the robot,
users could not focus on both the game and robot at the same time. The set of
haptic cues $\mathcal{U}$ included three options: i) alternating the top and
bottom vibrotactors, ii) alternating the left and right vibrotactors, and iii)
actuating all vibrotactors in a circular pattern.

Participants and Procedure. We recruited ten participants from the Virginia
Tech community to take part in our study (4 female, average age $24.3\pm 3.0$
years). All subjects provided informed written consent prior to the
experiment. Half of the participants had previous experience interacting with
a robot arm, and two participants had used augmented reality before.
Participants teleoperated the robot for up to five minutes to familiarize
themselves with each feedback method. We used a within-subjects study design:
every participant interacted with all four feedback conditions. To mitigate
the confounding effect of participants improving over time, we counterbalanced
the order of the feedback conditions.

Dependent Measures – Objective. We recorded the amount of time users spent
teaching the robot with the joystick (Interaction Time), as well as the total
score users achieved on the distractor game (Distractor Score). We also
measured the user’s Teaching Efficiency. To compute this we recorded $b(g)$,
the robot’s learned belief in the human’s desired goal at the end of the task.
Recall that $b(g)$ is a scalar, where $b(g)\rightarrow 1$ indicates that the
robot correctly inferred the human’s goal. Following [31], we then divided
this scalar by the user’s interaction time (i.e., the amount of time they
teleoperated the robot). Intuitively, a high Teaching Efficiency indicates
that the human accurately taught the robot their desired goal through a
smaller number of well-timed corrections.

Dependent Measures – Subjective. Participants filled out a 7-point Likert
scale survey after each method. Questions were grouped into five multi-item
scales (see Fig. 5): did the user understand the robot’s intent, did the user
know when the robot needed more teaching (prompt), did the robot make it clear
what type of teaching to provide, did the robot reveal what it knew and did
not know, and did the user prefer this current feedback method to the
alternatives.

Figure 5: Subjective results from our user study. Colors are consistent with
Fig. 4, and higher ratings indicate agreement (i.e., conveys intent, prompts
for help). Overall, participants preferred teaching the robot with multimodal
AR+Haptic feedback. We note that some users reported the AR device to be
uncomfortable, which may skew their perception of this condition.

Hypotheses. We had three hypotheses in this user study:

> H1. _Users with passive visual feedback ( AR) will minimize interaction
> time._

> H2. _Users with active haptic feedback ( Haptic) will maximize score on the
> distractor task._

> H3. _Users with multimodal feedback ( AR+Haptic) will maximize teaching
> efficiency._

Results – Objective. An example interaction is illustrated in Fig. 3, and our
aggregate results across all tasks are plotted in Fig. 4. When interpreting
these results, remember that we want to minimize interaction time while
maximizing teaching efficiency and distractor score.

Because AR displays what the robot has inferred, we anticipated that with AR
participants would know when the robot understood their goal, and would only
interact as much as needed. More specifically, we anticipated that AR would
minimize interaction time. This turned out not to be true: only in the
Avoiding task did AR reduce the interaction time. When looking at the
aggregated results, post hoc analysis shows no significant differences in
interaction time between AR and GUI ($p=.55$) or AR and Haptic ($p=.93$).

Unlike passive AR, active Haptic feedback grabs the human’s attention when
teaching is required. Since this method provides alerts, participants can
focus on the distractor game until they receive a prompt. To test H2, we
compared the distractor score with and without Haptic. We found that in either
the Haptic or AR+Haptic condition participants scored statistically
significantly higher than with GUI or AR ($p<.01$). This suggests that active
feedback reduces the amount of time users spent monitoring the robot.

Thus far we have looked at the results of AR and Haptic individually; what
happens when we combine these into multimodal AR+Haptic feedback? Inspecting
Fig. 4, participants using AR+Haptic had lower interaction time and higher
teaching efficiency than the alternatives ($p<.01$). These results support H3.
We conclude that the synthesis of passive and active feedback was more
effective for communicating robot inference than either method in isolation.

Results – Subjective. The results from our 7-point Likert scale survey are
listed in Fig. 5. We first tested the reliability of all five scales using
Cronbach’s alpha, and found every scale to be reliable ($\alpha$ > 0.7).
Accordingly, we grouped the participants’ responses within each scale into a
single combined score, and ran a one-way repeated measures ANOVA. Post-hoc
tests revealed that users preferred AR+Haptic to the alternatives ($p<.05$).
One user commented that “the combination of tactile and visual feedback made
this method the easiest to control the robot.”

Limitations. During our user study the AR was implemented on a Microsoft
HoloLens. Several participants indicated that they disliked wearing this
device for prolonged periods: “the HoloLens is uncomfortable to wear.” We
recognize that user comfort may be a confounding factor that negatively
affected AR results. In addition, our results indicate that providing passive
and active feedback on the GUI was roughly on par with passive feedback in AR.
This suggests that a GUI+Haptic system (with passive visual feedback and
active haptic prompts) could potentially achieve similar performance as
AR+Haptic. However, here the GUI must be located in a position where human
teachers can readily view it while interacting with the robot arm. Our future
work will explore intuitive GUI design for GUI+Haptic systems.

##  VI Conclusion

From the human’s perspective robot inference is often a black box. Without any
feedback, humans have no way of knowing what their robot has inferred or
whether it is confused. Accordingly, we developed a multimodal augmented
reality and haptic interface that combines passive and active feedback. We
derived a decision rule for mapping robot inference to each type of feedback
during shared autonomy tasks. Our user study demonstrates that this passive
and active approach takes a step towards bringing the human into the loop
during robot inference.

## References

  * [1] T. Hellström and S. Bensch, “Understandable robots – What, why, and how,” _Journal of Behavioral Robotics_ , vol. 9, pp. 110–123, 2018.
  * [2] A. D. Dragan, K. C. Lee, and S. S. Srinivasa, “Legibility and predictability of robot motion,” in _ACM/IEEE Int. Conf. on Human-Robot Interaction_ , 2013, pp. 301–308.
  * [3] S. Tellex, R. Knepper, A. Li, D. Rus, and N. Roy, “Asking for help using inverse semantics,” in _Robotics: Science and Systems_ , 2014.
  * [4] H. Admoni and S. Srinivasa, “Predicting user intent through eye gaze for shared autonomy,” in _AAAI_ , 2016.
  * [5] E. Cha, Y. Kim, T. Fong, and M. J. Mataric, “A survey of nonverbal signaling methods for non-humanoid robots,” _Foundations and Trends in Robotics_ , vol. 6, no. 4, pp. 211–323, 2018.
  * [6] S. Jain and B. Argall, “Probabilistic human intent recognition for shared autonomy in assistive robotics,” _ACM Transactions on Human-Robot Interaction_ , vol. 9, no. 1, pp. 1–23, 2019.
  * [7] S. Javdani, H. Admoni, S. Pellegrinelli, S. S. Srinivasa, and J. A. Bagnell, “Shared autonomy via hindsight optimization for teleoperation and teaming,” _The Int. Journal of Robotics Research_ , vol. 37, no. 7, pp. 717–742, 2018\.
  * [8] A. D. Dragan and S. S. Srinivasa, “A policy-blending formalism for shared control,” _The Int. Journal of Robotics Research_ , vol. 32, no. 7, pp. 790–805, 2013.
  * [9] D. Szafir, B. Mutlu, and T. Fong, “Communication of intent in assistive free flyers,” in _ACM/IEEE Int. Conf. on Human-Robot Interaction_ , 2014, pp. 358–365.
  * [10] R. A. Knepper, C. I. Mavrogiannis, J. Proft, and C. Liang, “Implicit communication in a joint action,” in _ACM/IEEE Int. Conf. on Human-Robot Interaction_ , 2017, pp. 283–292.
  * [11] M. J. Gielniak and A. L. Thomaz, “Generating anticipation in robot motion,” in _IEEE Int. Symp. on Robot and Human Interactive Communication_ , 2011, pp. 449–454.
  * [12] R. S. Andersen, O. Madsen, T. B. Moeslund, and H. B. Amor, “Projecting robot intentions into human environments,” in _IEEE Int. Symp. on Robot and Human Interactive Communication_ , 2016.
  * [13] K. Baraka, S. Rosenthal, and M. Veloso, “Enhancing human understanding of a mobile robot’s state and actions using expressive lights,” in _IEEE Int. Symp. on Robot and Human Interactive Communication_ , 2016, pp. 652–657.
  * [14] T. Weng, L. Perlmutter, S. Nikolaidis, S. Srinivasa, and M. Cakmak, “Robot object referencing through legible situated projections,” in _Int. Conf. on Robotics and Automation_ , 2019, pp. 8004–8010.
  * [15] R. Newbury, A. Cosgun, T. Crowley-Davis, W. P. Chan, T. Drummond, and E. Croft, “Visualizing robot intent for object handovers with augmented reality,” _arXiv preprint arXiv:2103.04055_ , 2021.
  * [16] K. C. Hoang, W. P. Chan, S. Lay, A. Cosgun, and E. Croft, “Virtual barriers in augmented reality for safe and effective human-robot cooperation in manufacturing,” _arXiv preprint arXiv:2104.05211_ , 2021.
  * [17] M. Walker, H. Hedayati, J. Lee, and D. Szafir, “Communicating robot motion intent with augmented reality,” in _ACM/IEEE Int. Conf. on Human-Robot Interaction_ , 2018, pp. 316–324.
  * [18] E. Rosen, D. Whitney, E. Phillips, G. Chien, J. Tompkin, G. Konidaris, and S. Tellex, “Communicating and controlling robot arm motion intent through mixed-reality head-mounted displays,” _The Int. Journal of Robotics Research_ , vol. 38, no. 12-13, pp. 1513–1526, 2019.
  * [19] F. Muhammad, A. Hassan, A. Cleaver, and J. Sinapov, “Creating a shared reality with robots,” in _ACM/IEEE Int. Conf. on Human-Robot Interaction_ , 2019, pp. 614–615.
  * [20] K. Chandan, V. Kudalkar, X. Li, and S. Zhang, “ARROCH: Augmented reality for robots collaborating with a human,” in _Int. Conf. on Robotics and Automation_ , 2021.
  * [21] E. Pezent, A. Israr, M. Samad, S. Robinson, P. Agarwal, H. Benko, and N. Colonnese, “Tasbi: Multisensory squeeze and vibrotactile wrist haptics for augmented and virtual reality,” in _IEEE World Haptics Conf._ , 2019\.
  * [22] M. Aggravi, F. Pausé, P. R. Giordano, and C. Pacchierotti, “Design and evaluation of a wearable haptic device for skin stretch, pressure, and vibrotactile stimuli,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 3, pp. 2166–2173, 2018.
  * [23] A. B. Arrieta, N. Díaz-Rodríguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, S. García, S. Gil-López, D. Molina, and R. Benjamins, “Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible ai,” _Information Fusion_ , vol. 58, pp. 82–115, 2020.
  * [24] S. H. Huang, D. Held, P. Abbeel, and A. D. Dragan, “Enabling robots to communicate their objectives,” _Autonomous Robots_ , vol. 43, no. 2, pp. 309–326, 2019.
  * [25] C. Brooks and D. Szafir, “Visualization of intended assistance for acceptance of shared control,” in _IEEE/RSJ Int. Conf. on Intelligent Robots and Systems_ , 2020.
  * [26] M. Zolotas, J. Elsdon, and Y. Demiris, “Head-mounted augmented reality for explainable robotic wheelchair assistance,” in _IEEE/RSJ Int. Conf. on Intelligent Robots and Systems_ , 2018.
  * [27] S. H. Huang, K. Bhatia, P. Abbeel, and A. D. Dragan, “Establishing appropriate trust via critical states,” in _IEEE/RSJ Int. Conf. on Intelligent Robots and Systems_ , 2018, pp. 3929–3936.
  * [28] D. Ramachandran and E. Amir, “Bayesian inverse reinforcement learning,” in _IJCAI_ , 2007, pp. 2586–2591.
  * [29] Y. Che, A. M. Okamura, and D. Sadigh, “Efficient and trustworthy social navigation via explicit and implicit robot–human communication,” _IEEE Trans. on Robotics_ , vol. 36, no. 3, pp. 692–707, 2020.
  * [30] E. Bıyık, D. P. Losey, M. Palan, N. C. Landolfi, G. Shevchuk, and D. Sadigh, “Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences,” _The Int. Journal of Robotics Research_ , 2021.
  * [31] A. Sena and M. Howard, “Quantifying teaching behavior in robot learning from demonstration,” _The International Journal of Robotics Research_ , vol. 39, no. 1, pp. 54–72, 2020.