# Towards Physically Safe Reinforcement Learning under Supervision

Yinan Zhang1, Devin Balkcom2 and Haoxiang Li3 1Department of Computer Science,
Dartmouth College, Hanover, NH 03755, USA.
<EMAIL_ADDRESS>of Computer Science, Dartmouth
College, Hanover, NH 03755, USA<EMAIL_ADDRESS>Research,
Adobe System Inc., San Jose, CA 95110, USA<EMAIL_ADDRESS>
###### Abstract

This paper addresses the question of how a previously available control policy
$\pi_{s}$ can be used as a supervisor to more quickly and safely train a new
learned control policy $\pi_{L}$ for a robot. A weighted average of the
supervisor and learned policies is used during trials, with a heavier weight
initially on the supervisor, in order to allow safe and useful physical trials
while the learned policy is still ineffective. During the process, the weight
is adjusted to favor the learned policy. As weights are adjusted, the learned
network must compensate so as to give safe and reasonable outputs under the
different weights. A pioneer network is introduced that pre-learns a policy
that performs similarly to the current learned policy under the planned next
step for new weights; this pioneer network then replaces the currently learned
network in the next set of trials. Experiments in OpenAI Gym demonstrate the
effectiveness of the proposed method.

##  I Introduction

In the early stages of training, outputs from a learning policy can be
unreasonable and lead to catastrophic failures. Motivated by the twin goals of
better data efficiency and fewer failures in learning-based control, we
propose an algorithm to leverage a possibly imperfect external supervisor
policy to help accelerate the learning, and introduce the concept of a pioneer
policy to realize safe progressive updates of the learner policy. In our
framework, the training signals come from both the interactions with the
environment and the supervisor policy, and the goal is that eventually the
learner policy surpasses or at least matches the performance of the supervisor
policy.

Consider a problem of robot control in which a traditional manually designed
control policy is available and effective. However, new un-modeled conditions
may arise: a drone may encounter shifting wind patterns or pick up an unsteady
load for manipulation, or a self-driving vehicle may encounter unfamiliar
lighting conditions or slick roads. The goal of this paper is to use a
previously available control policy $\pi_{s}$ to more safely and effectively
learn a new control policy $\pi_{L}$ using deep reinforcement learning. Once a
policy is learned, it may in turn be used as a supervisor to bootstrap the
learning of better policies as training data becomes available.

Define $\pi_{c}$ to be the current policy at a particular time, and compute it
as a weighted linear combination of $\pi_{s}$ and $\pi_{L}$:

$\pi_{c}=k\pi_{s}+(1-k)\pi_{L}.$ (1)

Initially, $k$ is set close to $1$, so the resulting control policy is
dominated by the supervisor. Trials are conducted and $\pi_{L}$ is learned
with the objective of improving $\pi_{c}$ with regards to the current task.
Once a target score has been achieved, $k$ is reduced by some percent, for
example, 4%.

Notice that if $k$ is reduced suddenly, the policy $\pi_{c}$ also changes
suddenly at any point where $\pi_{s}$ and $\pi_{L}$ indicate different
controls. This may cause a robot to take surprising and perhaps dangerous
actions. To avoid this issue, during training of $\pi_{L}$, we also train a
pioneer policy $\pi_{p}$ simultaneously, but for a different weighting of the
supervisor $k^{\prime}$: $k^{\prime}\pi_{s}+(1-k^{\prime})\pi_{p}$. In order
resolve the discontinuity, the pioneer network is trained with the goal that:

$\pi_{c}=k^{\prime}\pi_{s}+(1-k^{\prime})\pi_{p}.$ (2)

Once $\pi_{p}$ and $\pi_{L}$ both reach desired objectives, then $k$ is
updated and $\pi_{L}$ is replaced by the pioneer network $\pi_{p}$.

(a) LunarLanderContinuous-v2 environment

(b) InvertedPendulum-v1 environment

(c) HalfCheetah-v1 environment

(d) Reacher-v1 environment

Figure 1: OpenAI gym environments used as experiment tasks.

##  II Related Work

Recent progress in deep learning has led to several success stories in various
domains [13, 8, 9]. Nevertheless, several practical issues for learning-based
control remain under-explored, including efficiency and safety of the training
process.

Supervised learning. Supervised learning is the machine learning task of
inferring a function from labeled training data [14], finding a mapping
between the input object and a desired output value while minimizing an error
signal. Linear regression and multivariate linear regression [19], for
instance, aim at finding a linear relationship between variables. Logistic
regression [4, 7] considers specifically mapping the independent input
variables to a binary output, while Naive Bayes classifiers [15, 17] map the
input to multiple categories. Support Vector Machines (SVM) [22, 1] are
efficient large-margin classifiers for both linear and non-linear
classifications. Recently, deep neural networks have been widely and
successfully adopted for many supervised learning tasks.

Supervised learning of an effective control policy requires a large number of
state-action pairs, which are impractical to manually annotate. Imitation
learning, which is generally supervised learning, works with an expert policy
to query optimal action given a state. To sample state-action pairs to train a
policy with supervised learning, one widely used method is DAGGER [18]. DAGGER
iteratively collects trajectories using the current policy and then queries
the expert policy at states along the trajectories for optimal actions to
augment the training data to update the policy. DAGGER cannot produce a policy
better than the expert policy. Further, in practice, it is not always possible
to obtain an expert policy. In our work, we assume access to a supervisor
policy, which may not be optimal, but provides reasonable performance. In the
current work, the trained policy may provide better performance than the
supervisor.

Reinforcement learning and Deep reinforcement learning. Reinforcement learning
(RL) is often applied to problems involving decision making and maximizing
feedback rewards as a performance signal. Markov decision processes (MDP) [16]
provide a framework for modeling decision making. Watkins [24] proposed the
Q-learning algorithm as a model-free technique for solving finite MDP
problems. Q-learning works by learning a state-action value function that
predicts the expected reward given an action at a state.

Deep neural networks have been introduced to reinforcement learning (DRL) for
training policies from end-to-end to solve complex problems. A recent break
through is the DQN from Minh et al. [13, 12], in which neural networks are
used to approximate the Q function for high dimensional state space. Levine et
al. [9] developed an end-to-end DRL method that maps raw image input to robot
motor controls. Silver et al. [20] combined tree search and DQN to beat the
best human Go player. For continuous control, Lillicrap [10] combined deep
Q-learning with an actor-critic framework to learn a deterministic policy
(DDPG).

Most of the DRL methods require a huge amount of training data to find a good
policy. As the community is aware of this issue, many methods have been
proposed to accelerate training. Popular techniques include adopting the
advantage function from Schulman et al., adding auxiliary tasks [5] and
asynchronous methods [11]. These methods are complementary to our proposed
solution, with which we can potentially further accelerate the RL training.

We approach this problem by introducing a supervisor policy in training. We
assume the existence of a reasonably good policy as a supervisor during
training. The learner policy learns from both the RL training signals and the
supervisor policy. We observe significant acceleration of the learning process
with respect to learning without a supervisor. A relevant work from Hester et
al. [3] also propose to add supervised loss in RL training. However, their
method does not address the safety issue in RL training. Although the leaner
policy improves quickly in generally in that work, there are sometimes
unexpected behaviors, which may lead to catastrophic failures of the overall
system.

Safe reinforcement learning. Safe reinforcement learning highlights the
importance of ensuring a reasonable system performance while searching for a
new and improved policy [23]. We refer readers to [2] for a more comprehensive
literature survey.

Quite recently, Kahn et al. [6], proposed an uncertainty measure used to
control the speed of the robot during learning to avoid collision. For some
tasks, e.g., stabilizing a quadrotor, a smaller magnitude of actions may not
make the training safe. Our framework is general, and not task-specific.

##  III Background

In this section, we review the mathematical background of Q-learning and the
technique of applying an artificial neural network to represent the actor-
value function, including its extension to a continuous action space.
Techniques discussed in this section will be applied in the next section where
we combine a supervisor with a learning actor.

In a standard reinforcement learning setup, an agent interacts with an
environment $E$ in discrete time-steps. Let $s_{t}$ be the observation of the
environment at time $t$, $a_{t}$ be the action taken, and $r_{t}$ be a scalar
reward feedback. We consider the action to be real valued
$a_{t}\in\mathbb{R}^{N}$ and the environment is fully observed.

A policy $\pi:S\to\mathcal{P}(A)$ maps the state space $S$ to a probability
distribution over the action space $A$. A policy determines the behavior of an
agent. In the continuous action space, an actor function $\mu:S\to A$ is a
policy that deterministically maps a state to a specific action. The reward of
an action taken at a state is described by $r(s_{t},a_{t})$. The sum of
discounted future rewards $R_{t}=\sum^{T}_{i=t}\gamma^{(}i-t)r(s_{t},a_{t})$
is the return from a state, where $\gamma\in[0,1]$ is a constant discount
factor.

The action-value function, or Q function, describes the expected return in
state $s_{t}$ after taking an action $a_{t}$ and thereafter under policy
$\pi$:

$Q^{\pi}=\mathbb{E}[R_{t}|s_{t},a_{t}].$ (3)

Q-learning [24] uses a greedy policy to determine the action under current
state that maximizes the return (the Q value).

The Bellman equation, as a necessary condition for optimality, is widely used
to represent Q function in a recursive manner:

$Q^{\pi}=\mathbb{E}_{r_{t},s_{t+1}}[r(s_{t},a_{t})+\gamma\mathbb{E}_{a_{t+1}}[Q^{\pi}(s_{t+1},a_{t+1})]].$
(4)

If both $S$ and $A$ are discrete, the Q function, a map of $S$ to $A$ can be
described using a table. When $S$ is continuous, Deep Q networks (DQN)
construct a network to represent the Q function. Let $\theta^{Q}$ be the
parameter of $Q^{\pi}$ network. We optimize $\theta^{Q}$ by minimizing the
loss:

$L(\theta^{Q})=\mathbb{E}[(Q(s_{t},a_{t}|\theta^{Q})-y_{t})^{2}],$ (5)

where

$y_{t}=r(s_{t},a_{t})+\gamma Q^{\pi}(s_{t+1},\pi(s_{t+1}))$ (6)

is the observed return. Practically, the dependence on $y_{t}$ on parameters
$\theta^{Q}$ is frequently ignored; we do the same. By iteratively updating
the parameters, DQN predicts the return more and more accurately, thus better
actions will be chosen.

DQN works only for discrete action spaces. In the case where actions are from
a continuous space, we construct a new neural network to represent a
deterministic actor function $\mu:S\to A$. With the deterministic actor
function, we reduce the inner expectation and modify the Bellman equation (4)
as follows:

$Q^{\mu}=\mathbb{E}_{r_{t},s_{t+1}}[r(s_{t},a_{t})+\gamma
Q^{\mu}(s_{t+1},\mu(s_{t+1}))].$ (7)

Lillicrap et al. [10] used an actor-critic approach to optimize an actor-value
function and actor policy, based on the Deterministic Policy Gradient (DPG)
method by Silver et al. [21].

The Q function as a critic network is still optimized by minimizing the loss
defined in Equation (5). Let $\theta^{\mu}$ be the parameters of the actor
network. Silver et al. [21] proved that the actor network can be updated by
applying the policy gradient as in equation (8):

$\nabla_{\theta^{\mu}}J\approx\mathbb{E}_{s_{t}}[\nabla_{\theta^{\mu}}Q(s_{t},a|\theta^{Q})|_{a=\mu(s_{t}|\theta^{\mu})}].$
(8)

Following the chain rule, we have:

$\nabla_{\theta^{\mu}}J\approx\mathbb{E}_{s_{t}}[\nabla_{a}Q(s_{t},a|\theta^{Q})|_{a=\mu(s_{t}|\theta^{\mu})}\nabla_{\theta^{\mu}}\mu(s_{t}|\theta^{\mu})].$
(9)

In practice, direct implementation of equation 5 and 9 with neural networks
has proven unstable in many cases. Lillicrap et al. [10] addressed this
problem by adding target networks for both critic and actor as inspired by
Minh et al. [12]. The parameters $\theta^{\prime}$ of target networks are
updated by slowly tracking the learned networks:
$\theta^{\prime}=\tau\theta+(1-\tau)\theta^{\prime}$, where $\tau\ll 1$ is a
scalar factor.

Algorithm 1 Supervised deep reinforcement learning

1:Randomly initialize critic $Q(s,a|\theta^{Q})$, actor
$\mu_{a}(s|\theta^{\mu})$ and pioneer $\mu_{p}(s|\theta^{\mu}_{t_{2}})$
networks.

2:Initialize target network $Q^{\prime}$ and $\mu_{a}^{\prime}$ with
parameters $\theta^{Q^{\prime}}\leftarrow\theta^{Q}$ and
$\theta^{\mu^{\prime}}\leftarrow\theta^{\mu}$

3:Initialize replay buffer $R$ and $R_{p}$

4:Initialize pioneer buffer threshold $r_{p}$

5:Initialize $k\leftarrow 1$

6:for episode = 1 to M do

7: Initialize a random process $\mathcal{N}$ for exploration.

8: Initialize temporary buffer $R_{e}$

9: Reset environment and receive initial observation state $s_{1}$.

10: for t = 1, T do

11: Select action $a_{t}=\mu(s|\theta^{\mu})$ and execute

12: Observe reward $r_{t}$ and new state $s_{t+1}$

13: Store transition $(s_{t},a_{t},r_{t},s_{t+1})$ in $R$ and $R_{e}$

14: Sample N transitions $(s_{i},a_{i},r_{i},s_{i+1})$ from $R$

15: Set $y_{i}=r_{i}+\gamma
Q^{\prime}(s_{i+1},\mu^{\prime}(s_{i+1})|\theta^{Q^{\prime}})$

16: Update critic by minimizing loss:

17: $L=\frac{1}{N}\sum_{i}(y_{i}-Q(s_{i},a_{i}|\theta^{Q}))$

18: Update the actor using sampled policy gradient:

19:
$\nabla_{\theta^{\mu}}J\approx\frac{1}{N}\\{(1-k)\sum_{i}\nabla_{a}Q(s_{i},a_{i})\nabla_{\theta^{\mu}}\mu_{a}(s_{i})$

20: $+k\sum_{i}[\mu(s_{i})-a_{i}]\\}$

21: Update the target networks:

22: $\theta^{Q^{\prime}}=\tau\theta^{Q}+(1-\tau)\theta^{Q^{\prime}}$

23: $\theta^{\mu^{\prime}}=\tau\theta^{\mu}+(1-\tau)\theta^{\mu^{\prime}}$

24: if episode total reward $\geq r_{p}$ then

25: Move transitions from $R_{e}$ to $R_{p}$

26: Copy the learning network to pioneer:
$\theta^{\mu}_{t_{2}}\leftarrow\theta^{\mu}$

27: Sample transaction $(s_{j},a_{j},r_{j},s_{j+1})$ from $R_{p}$

28: Update the pioneer policy by applying gradient:

29:
$\nabla_{\theta^{\mu}_{t_{2}}}J_{p}=\sum_{j}[\mu_{p}(s_{j}|\theta^{\mu}_{t_{2}})-a_{j}]$

30: Empty $R_{e}$

31: Decrease $k$ if a target score is achieved.

32: Re-initialize the learning network
$\theta^{\mu}\leftarrow\theta^{\mu}_{t_{2}}$ if $k$ is decreased.

33: Increase $r_{p}$

##  IV Our method

Figure 2: The framework: based on the actor-critic framework, we introduce a
supervisor policy, a learning policy, and a pioneer policy. The combined
action $\mu_{c}$ is executed. The pioneer policy is trained such that
$\mu_{c}^{\prime}$ outputs similar results as $\mu_{c}$ with an updated weight
$k$. The learning policy is then replaced by $\mu_{p}$.

Learning from scratch using the algorithms discussed above is impractical in
many real world tasks. The large number of trials and errors can be
destructive for agents like unmanned aerial vehicle and many other robots. In
a real world learning process, however, supervisors significantly improve the
performance of a learner and reduce the number of failures.

Inspired by real world learning experience, we introduce a supervisor into the
reinforcement learning process. This section gives detailed explanation of how
to combine a supervisor policy and a learning network to train the network and
perform tasks at the same time. We slowly reduce and eventually remove the
supervisor contribution throughout the learning process.

Let supervisor $\mu_{s}:S\to A$ be a deterministic actor function,
$\mu_{a}:S\to A$ be our learning policy. We combine the two policies plus some
exploration as a new policy, called the combined policy:

$\mu(s)=k\mu_{s}(s)+(1-k)\mu_{a}(s)+\mathcal{N}_{t}$ (10)

where the combination factor $k\in[0,1]$ is a scalar and $\mathcal{N}_{t}$ is
a noise process that diminishes as time increases.

In our proposed algorithm, the critic function $Q(s,a|\theta^{Q})$ is also
modeled by an artificial neural network with parameters $\theta^{Q}$, the
actor network $\mu_{a}(s|\theta^{\mu})$ is modeled by another multi-layer
neural network, where $\theta^{\mu}$ is the parameters of the actor network.
Because $\mu_{s}$ is a fixed policy, $\theta^{\mu}$ is also the parameters of
the combined actor network $\mu(s|\theta^{\mu})$.

###  IV-A Improving the combined actor

Assuming $k$ is a constant factor, we consider how to improve the combined
policy during learning.

The critic network parameters $\theta^{Q}$ are optimized by minimizing the
loss as defined in equation (5). The actor network parameters are updated by
applying gradient defined in equation (8). But after applying the chain rule,
because $\mu$ is a combined policy, equation (8) becomes:

$\begin{split}\nabla_{\theta^{\mu}}J_{a}&\approx\mathbb{E}_{s_{t}}[\nabla_{\theta^{\mu}}Q(s_{t},a|\theta^{Q})|_{a=\mu(s_{t}|\theta^{\mu})}]\\\
&=\mathbb{E}_{s_{t}}[\nabla_{a}Q(s_{t},a|\theta^{Q})|_{a=\mu(s_{t}|\theta^{\mu})}\nabla_{\theta^{\mu}}\mu(s_{t}|\theta^{\mu})]\\\
&=\mathbb{E}_{s_{t}}[\nabla_{a}Q(s_{t},a|\theta^{Q})|_{a=\mu(s_{t}|\theta^{\mu})}\cdot\nabla_{\theta^{\mu}}\mu_{a}(s_{t}|\theta^{\mu})]\\\
&~{}~{}~{}\cdot(1-k)\end{split}$ (11)

Equation (11) considers only the performance improvement of the combined
policy. To better train the learning policy, we also want the learning policy
to behave as closely as possible to the supervisor. The parameters
$\theta^{\mu}$ of the learning network can also be updated by minimizing the
loss:

$L(\theta^{\mu})=\frac{1}{2}\mathbb{E}[(\mu_{a}(s|\theta^{\mu})-\mu_{s}(s))^{2}],$
(12)

whose corresponding gradient is:

$\nabla_{\theta^{\mu}}J_{s}=\mathbb{E}[\mu_{a}(s|\theta^{\mu})-\mu_{s}(s)].$
(13)

In order to improve the performance of a combined network and learn from the
supervisor at the same time, we apply a combined gradient to the learning
actor network, as defined in the following equation

$\nabla_{\theta^{\mu}}J=\nabla_{\theta^{\mu}}J_{a}+\lambda\nabla_{\theta^{\mu}}J_{s},$
(14)

where $\lambda$ is a scalar factor.

###  IV-B Reducing supervision

We now consider reducing the contribution of the supervisor to our combined
policy.

Let $k_{t_{1}}$ be the value of $k$ at time $t_{1}$. We choose $k_{t_{2}}$ to
be no larger than $k_{t_{1}}$ for any $t_{2}\geq t_{1}$. Notice that when
$k\to 0$, the combined actor is the learning actor. However, equation (14)
prevents the learning actor from being much better than the supervisor if
$\lambda$ is a constant number. Assuming the learning actor has learned the
supervisor policy, we set $\lambda\leftarrow k$ so the learning network can
improve without relying on the supervisor.

In practice, we update the $k$ value after achiving a target score for one or
more epochs, so the combined policy has more time to be improved and stablize.
Let $\mu_{t_{1}}$ be the combined policy at time $t_{1}$, $k_{t_{1}}$ be the
$k$ value at time $t_{1}$. Assuming that at time $t_{2}>t_{1}$, $\mu_{t_{1}}$
is well trained and $\mu_{t_{2}}\neq\mu_{t_{1}}$, shifting the combined policy
from $\mu_{t_{1}}$ to $\mu_{t_{2}}$ can result in bad performance.

We address the problem by adding a copy of the learning network
$\mu_{p}(s|\theta^{\mu}_{t_{2}})$, named the pioneer network, representing the
learning actor at time $t_{2}$, where $\theta^{\mu}_{t_{2}}$ are the
parameters. Knowing the value of $k_{t_{2}}$ and parameters
$\theta^{\mu}_{t_{1}}$ of the learning actor at time $t_{1}$, the combined
policy at the time-step $t_{2}$ is known:

$\mu_{p}(s_{t_{2}}|\theta^{\mu}_{t_{2}})=k_{t_{2}}\mu_{s}(s_{t_{2}})+(1-k_{t_{2}})\mu_{a}(s_{t_{2}}|\theta^{\mu}_{t_{2}})+\mathcal{N}_{t+1}.$
(15)

Before shifting the combined policy from $\mu_{t_{1}}$ to $\mu_{t_{2}}$, we
optimize the pioneer network such that it behaves as similarly to previous
combined policies as possible. Parameters $\theta^{\mu}_{t_{2}}$ are updated
by minimizing

$L(\theta^{\mu}_{t_{2}})=\frac{1}{2}\mathbb{E}[(\mu_{p}(s)-a)^{2}]$ (16)

where $a$ is the output by previous combined policies. This is equivalent to
applying gradients

$\nabla_{\theta^{\mu}_{t_{2}}}J_{p}=\mathbb{E}[\mu_{p}(s)-a]$ (17)

to the pioneer network.

Applying gradients (17) using randomly sampled previous transitions requires
previous combined policies to have stably good performance. In practice, this
requirement is not always satisfied. We use a priority replay buffer to store
and sample state-action pairs with high returns.

In our implementation, we adapt the idea of using target networks from
Lillicrap et al. [10] and Minh et al. [12], to prevent actor and critic
networks from divergence. Algorithm 1 is the pseudo-code of our algorithm.

When the combination factor $k$ gets to zero, our method is reduced to DDPG
algorithm. So the learning policy is guaranteed to improve its performance
over more trails.

##  V Experiments

In this section, we test our method under several OpenAI Gym
[brockman2016openai] environments and discuss the impact of pioneer network
and supervisors with different performances. Our primary environment is
LunarLanderContinuous-v2, which operates a landing agent by setting main
engine and side engine forces to land on a pad centered at $(0,0)$. In this
environment, if an episode’s total return is less than 0, the agent has
crashed, if the return is larger than 200 the agent is landed successfully,
otherwise, the agent has landed but not in the desired range.

###  V-A Pioneer network

We also performed an experiment to test the difference of adding a pioneer
network. We first trained a supervisor using DDPG method. After 1500 episodes
of training, the supervisor performance is relatively stable. Its 100-episode
average return is above 200. Our learning network has the exact same structure
as the supervisor. We reduced the combination factor $k$ by $6\%$ for every 4
episodes. After 300 episodes, the supervisor contribution was less than $1\%$.

Figure 3: Comparison of the combined actor performances with and without the
pioneer network. The x-axis is the number of episodes and the y-axis is the
total reward in one episode. The pink curve shows the performance with the
pioneer network while the blue curve is without the pioneer. Data smoothed for
visual purposes.

Figure 3 shows the comparison of the combined actor performance with and
without the pioneer network. Without a pioneer network, the combined actor
performance drops sharply when the combination factor $k$ is less than 0.47.
Then performance improves throughout the remainder of training. With the
pioneer network, the performance is much more stable (average reward larger
than 200) and the learning process is much faster than when learning from
scratch. This indicates the pioneer network significantly improves the
stability of the learning process.

Although there are still crashes (reward below 0) with the pioneer added, we
suspect this is caused by the imperfect supervisor; running the supervisor
alone also leads to some crashes.

###  V-B Supervisor impact

The next experiment is for testing the combined actor performance under
different supervisors. Our supervisors are trained as in Section V-A. We pick
two supervisors: one (bad supervisor) trained for 1000 episodes whose
100-episode average performance is around 100 and one (good supervisor)
trained for 1500 episodes whose 100-episode average performance is above 200.
We enable the pioneer network in both experiments and reduce the combination
factor $k$ by $6\%$ for every 4 episodes.

Figure 4: Comparison of combined actor performance trained under supervision
of both a good (pink) and bad (green) policies. Data smoothed for
visualization.

Figure 4 shows the comparison of learning from good and bad supervisors. When
learning from the good supervisor, the combined actor performance is stably
good (rewards mostly above 200). However, with a bad supervisor, after 300
episodes, the rewards are between 130 and 160 and improve slowly.

Figure 5: Network performance under the supervision of a bad supervisor. The
blue curve is the performance of the supervisor alone and the green curve is
the performance of the combined actor.

Figure 5 shows the comparison of our combined actor and the supervisor. The
combined actor performs better than the supervisor alone. In 300 episodes, the
supervisor crashed 54 times and successfully landed 30 times, while the
combined actor crashed only 23 times and successfully landed 37 times.

Figure 5 indicates our method is learning and improving the performance of a
given supervisor.

Figure 6 shows comparison of performances in the first 340 episodes for
different methods. The DDPG algorithm achieved no successful landing, our
method with a good supervisor is consistently landing with success, and
without pioneer network, the performance over the training process is very
unstable.

Figure 6: Performance curves in the first 340 episodes for the DDPG method
(orange), our proposed method with a good supervisor (pink), our method
without pioneer network (blue) and our method using a bad performance
supervisor (green).

###  V-C Mujoco environments

We also ran several Mujoco environments to test our algorithm in different
continuous control tasks under supervisors with different performance
qualities.

For the InvertedPendulum-v1(Figure 1(b)) environment the goal is to swing up a
pendulum and make it stand as long as possible. Getting a reward of higher
than 950 is considered “solved”. We also use DDPG algorithm to train two
supervisors: a bad one capable of getting rewards around 255 and a good one
capable of getting rewards over 950. We decrease the combination factor by
$10\%$ for every 5 episodes until supervisor is contributing less than $1\%$.

The result of training with a good supervisor is shown in Figure 7. With a
good supervisor, our combined policy gets the highest rewards (1000) over all
trails.

Figure 7: Training to stand a pendulum with a good supervisor. Our method is
achieving rewards of 1000 over the whole process.

However, training with a bad supervisor is only slightly better than using the
supervisor alone in this environment. We believe the reason is that, in this
specific task, the distribution of states under a good supervisor policy is
very concentrated in a small region where the pendulum is close to stand. A
bad supervisor generates states distributed much wider in the state space; it
seems a lot of effort is spent on states that are less important. This
experiment indicates that if a supervisor policy is not good enough, the
learning policy will not improve significantly. The next experiment on
HalfCheetah-v1 environment uses a supervisor that is close to solve the task.

The goal for the HalfCheetah-v1(Figure 1(c)) environment is to control actions
of a 2D cheetah robot and keep it running as long as possible. Getting a
reward of higher than 4800 is considered “solved”. We trained using a bad
supervisor that gains rewards around 4300. This supervisor was trained for
over 5000 epoches, and not able to improve for 10000 more epoches, indicating
a neural network not complicated enough. Performance is shown in Figure 8

Figure 8: Training to run a 2D cheetah robot with a bad supervisor. Our
combined actor is graduately performing better than the supervisor and finally
achieved a target score of 4800.

We also run our method on Reacher-v1 (Figure 1(d)) environment. The goal for
this environment is to control a 2R arm to reach and stay in a goal position.
Getting a reward of higher than -3.75 is considered “solved”. We used a
supervisor policy that gains on average rewards of -5, which is still far from
solving the task in this environment. Our combined actor is getting similar
performance. See Figure 9. Considering that training this supervisor took over
10000 episodes. Having a similar near-solution policy trained in 350 episodes
is not bad. Our explination for this experiment is the same as in the
InvertedPendulum-v1 experiment, if a supervisor is too far from acheiving a
good score, the learning policy will also have hard time to make good
improvement.

Figure 9: Reacher-v1 environment. Our combined actor (light blue) is
performing similar as the supervisor (dark blue).

##  VI Conclusion

In this work, we introduced an algorithm to safely and quickly learn from a
supervisor policy and eventually improve performance. We combined a supervisor
with a learning network, and gradually decay the contribution of supervisor to
force the learning network to learn from previous executions. By introducing a
pioneer network, we are able to stabilize the combined policy performance over
the whole learning process.

The proposed method can be considered as an efficient way of reducing
exploration and exploitation on state-action pair space. Simulated experiments
proved the efficiency of the algorithm; next steps include applying this
algorithm to real-world robot systems.

## References

  * [1] Corinna Cortes and Vladimir Vapnik.  Support-vector networks.  Machine learning, 20(3):273–297, 1995.
  * [2] Javier Garcıa and Fernando Fernández.  A comprehensive survey on safe reinforcement learning.  Journal of Machine Learning Research, 16(1):1437–1480, 2015.
  * [3] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Andrew Sendonaris, Gabriel Dulac-Arnold, Ian Osband, John Agapiou, et al.  Learning from demonstrations for real world reinforcement learning.  arXiv preprint arXiv:1704.03732, 2017.
  * [4] David W Hosmer Jr, Stanley Lemeshow, and Rodney X Sturdivant.  Applied logistic regression, volume 398.  John Wiley & Sons, 2013.
  * [5] Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu.  Reinforcement learning with unsupervised auxiliary tasks.  arXiv preprint arXiv:1611.05397, 2016.
  * [6] Gregory Kahn, Adam Villaflor, Vitchyr Pong, Pieter Abbeel, and Sergey Levine.  Uncertainty-aware reinforcement learning for collision avoidance.  arXiv preprint arXiv:1702.01182, 2017.
  * [7] David G Kleinbaum and Mitchel Klein.  Analysis of matched data using logistic regression.  In Logistic regression, pages 389–428. Springer, 2010.
  * [8] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In Advances in Neural Information Processing Systems 25, pages 1097–1105. 2012.
  * [9] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel.  End-to-end training of deep visuomotor policies.  Journal of Machine Learning Research, 17(39):1–40, 2016.
  * [10] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  arXiv preprint arXiv:1509.02971, 2015.
  * [11] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.  Asynchronous methods for deep reinforcement learning.  In International Conference on Machine Learning, pages 1928–1937, 2016.
  * [12] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing atari with deep reinforcement learning.  arXiv preprint arXiv:1312.5602, 2013.
  * [13] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  Nature, 518(7540):529–533, 2015.
  * [14] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar.  Foundations of machine learning.  MIT press, 2012.
  * [15] Kevin P Murphy.  Naive bayes classifiers.  University of British Columbia, 2006.
  * [16] Martin L Puterman.  Markov decision processes: discrete stochastic dynamic programming.  John Wiley & Sons, 2014.
  * [17] Irina Rish.  An empirical study of the naive bayes classifier.  In IJCAI 2001 workshop on empirical methods in artificial intelligence, volume 3, pages 41–46. IBM, 2001.
  * [18] Stéphane Ross, Geoffrey J Gordon, and Drew Bagnell.  A reduction of imitation learning and structured prediction to no-regret online learning.  In International Conference on Artificial Intelligence and Statistics, pages 627–635, 2011.
  * [19] George AF Seber and Alan J Lee.  Linear regression analysis, volume 936.  John Wiley & Sons, 2012.
  * [20] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  Nature, 529(7587):484–489, 2016.
  * [21] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller.  Deterministic policy gradient algorithms.  In Proceedings of the 31st International Conference on Machine Learning (ICML-14), pages 387–395, 2014.
  * [22] Johan AK Suykens and Joos Vandewalle.  Least squares support vector machine classifiers.  Neural processing letters, 9(3):293–300, 1999.
  * [23] Philip S Thomas.  Safe reinforcement learning.  2015\.
  * [24] Christopher John Cornish Hellaby Watkins.  Learning from delayed rewards.  PhD thesis, King’s College, Cambridge, 1989.


#  EVE: Enabling Anyone to Train Robot using Augmented Reality

Jun Wang♣ Chun-Cheng Chang♣ Jiafei Duan♣ Dieter Fox♣♡ Ranjay Krishna♣♠
♣University of Washington ♡NVIDIA ♠Allen Institute for Artificial Intelligence

(2024)

###### Abstract.

The increasing affordability of robot hardware is accelerating the integration
of robots into everyday activities. However, training a robot to automate a
task typically requires physical robots and expensive demonstration data from
trained human annotators. Consequently, only those with access to physical
robots produce demonstrations to train robots. To mitigate this issue, we
introduce EVE, an iOS app that enables everyday users to train robots using
intuitive augmented reality visualizations without needing a physical robot.
With EVE, users can collect demonstrations by specifying waypoints with their
hands, visually inspecting the environment for obstacles, modifying existing
waypoints, and verifying collected trajectories. In a user study ($N=14$,
$D=30$) consisting of three common tabletop tasks, EVE outperformed three
state-of-the-art interfaces in success rate and was comparable to kinesthetic
teaching—physically moving a real robot—in completion time, usability, motion
intent communication, enjoyment, and preference ($mean_{p}=0.30$). We conclude
by enumerating limitations and design considerations for future AR-based
demonstration collection systems for robotics.

augmented reality, demonstration collection, imitation learning

††copyright: acmcopyright††journalyear: 2024††doi:
XXXXXXX.XXXXXXX††conference: ; April, 2024; ††ccs: Human-centered computing
Mixed / augmented reality††ccs: Computer systems organization Robotics

##  1\. Introduction

Figure 1. EVE enables everyday users to collect data to train a real robot
using intuitive augmented reality (AR) visualizations without needing a
physical robot. Using our interface, the user can move the AR robot by setting
waypoints with the hand, visually inspect the real world to avoid obstacles,
modify the collected trajectory by reverting to a previous waypoint, and
verify the collected trajectory by replaying the task with the AR robot. See
supplementary materials for the videos of real-world deployment.

The decreasing costs of robot components and systems are accelerating the
integration of robots into everyday life. With companies offering robotic arms
at a few hundred dollars, portable, lightweight, and mobile robots are being
trained to assist individuals with a wide range of everyday tasks (Elephant
Robotics, 2023; Robot, 2023; Shaw et al., 2023a). However, training a robot to
automate a task often requires expensive manually curated human
demonstrations. Annotators are hired and trained to kinesthetically move a
robot’s joints and gripper to a sequence of configurations to accomplish a
task. For example, to train a robot to clean a table, annotators move the
gripper towards a cloth, close the gripper to grasp the cloth, and then begin
moving it around to wipe the table.

As such, there are no mechanisms for everyday users, who are considering
purchasing a robot, to train robots without first purchasing an expensive
robot and learning to move bulky machinery to produce effective
demonstrations. Consequently, only a limited number of roboticists and
developers can determine the range of robotic applications.

Recently, a prototype, called AR2-D2, demonstrated the possibility of training
robots without the need for a physical robot (Duan et al., 2023). AR2-D2
enables users to capture videos of themselves manipulating objects in various
environments using smartphones or tablets. Furthermore, it has proven that the
quality of data gathered through AR visualization is comparable with data
collected using traditional approaches with real robots. However, AR2-D2 has
only been evaluated on simple, short-horizon tasks such as pressing a computer
mouse or picking up a plastic bowl. Ecologically valid user tasks, such as
cleaning a table, require longer interactions between the robot and its
environment. Additionally, the system’s usability has not been analyzed, which
is crucial for understanding the usability and utility for every user.

We introduce EVE, an iOS app that allows everyday users to train robots with
human-centric AR visualizations and features, even if they do not have access
to a robot. To collect demonstrations, users can specify waypoints for the
trajectory with their fingers or make the AR robot track the hand in real
time. During collection, users can visually inspect the real world to avoid
obstacles, modify the collected trajectory by reverting to a previous
waypoint, and verify the collected trajectory by replaying the task with the
AR robot.

To understand the opportunities and challenges of AR-based demonstration
collection, we conducted a formative study with 10 participants with varying
experience in robotics, using AR2-D2 as the baseline system and three new AR
visualizations. We identified six usability challenges users faced and
implemented seven additional features to improve EVE’s accuracy, flexibility,
and verifiability.

We conducted a user study with 14 participants to compare EVE with state-of-
the-art demonstration collection interfaces on three typical tabletop tasks.
EVE achieved the highest task success rate and performed comparably to
kinesthetic teaching in task completion time, system usability, motion intent
communication, user enjoyment, and overall preference ($mean_{p}=0.30$).
Participants customized their workflows with multiple visualizations and
features to collect demonstrations for complex, long-horizon tasks.

In summary, our contributions include:

  * •

EVE, an iOS app that enables everyday users to train robots using intuitive AR
visualizations and features

  * •

Formative study that identifies opportunities and challenges for improving the
usability of various demonstration collection interfaces

  * •

User study that demonstrates how people use diverse collection interfaces

  * •

A discussion on designing future AR-based demonstration collection interfaces

##  2\. Related Work

###  2.1. Human Demonstrations Collection for Robotics

Imitation learning enables robots to effectively automate tasks by learning
from human demonstrations (Zare et al., 2023; Wang et al., 2023; Shridhar et
al., 2022). A common approach for collecting demonstration data includes
kinesthetic teaching, where a human physically guides the robot through a
desired trajectory (Argall et al., 2009). Despite the direct haptic feedback,
this mechanism can be slow and tedious (Osentoski et al., 2010). Another
popular collection interface is teleoperation, where the user controls the
robot remotely using specialized controllers, such as 3D space mouse (Zhu et
al., 2022), VR controllers(Ebert et al., 2021; Brohan et al., 2022; Seo et
al., 2023), smartphones(Mandlekar et al., 2018), and haptic devices(Shaw et
al., 2023b; Toedtheide et al., 2023), which can be costly and challenging to
use due to high latency and lack of intuitiveness (Akgün and Subramanian,
2011). More recent efforts also involve capturing human expert demonstrations
with physical robots using specialized hardware like ALOHA(Zhao et al., 2023),
GELLO(Wu et al., 2023), DexCap(Wang et al., 2024), and UMI(Chi et al., 2024).
However, these studies often prioritize data collection efficiency and model
training with only a few exceptions, including minimum studies (Kapadia et
al., 2017; Szafir and Szafir, 2021) on examining user preference and
intuitiveness in their systems.

Prior works compared kinesthetic teaching and teleoperation on simple tasks
such as manipulating cups (Fischer et al., 2016; Akgün and Subramanian, 2011).
The common findings are that kinesthetic teaching is preferred over
teleoperation based on collection time, ease of use, and accuracy of
demonstrations. In contrast to this work, our paper compared the performance
of traditional collection interfaces as well as state-of-the-art AR-based
collection interfaces. Furthermore, we focus on three realistic tabletop tasks
with varying levels of difficulty, leading to a more comprehensive comparison
between different collection interfaces.

Most closely related to our work is AR2-D2, a system for collecting
demonstrations that do not require skilled demonstrators and real robots and
resituating demonstration collection outside the lab setting (Duan et al.,
2023). It is an iOS app that people can use to project AR robots into the real
world and collect waypoint information with their hands. The user study
demonstrates that AR2-D2 is intuitive to use and requires no training in
contrast to four other common demonstration collection interfaces. However,
AR2-D2 did not experiment with different user interface designs and only
offered one way to control and visualize the robot. In addition, it only
allows the collection of waypoint data, which is not useful for training the
robot to do dynamic tasks such as tossing a ball. In contrast, EVE focuses on
the human-centric aspect of the demonstration collection process by providing
three new AR visualizations and seven new features derived from the formative
study. Furthermore, EVE enables the collection of both waypoint data and
trajectory data, allowing users to select suitable collection methods based on
task types.

###  2.2. Augmented Reality and Robotics

Traditional approaches to improving human-robot interaction, such as through
robots’ movements (Dragan et al., 2013; Mainprice et al., 2010), gestural
motion (Huang and Mutlu, 2013; Chidambaram et al., 2012), and gaze outputs
(Mutlu et al., 2006; Kalegina et al., 2018), are limited due to the challenge
of presenting expressive physical feedback that goes beyond internal
capabilities. In contrast, AR provides visual feedback in one’s line of sight,
opening up exciting new opportunities for human-robot interaction research. A
recent taxonomy of augmented reality and robotics has identified five purposes
and benefits of visual augmentation (Suzuki et al., 2022). We focus on
analyzing works related to motion intent communication since our goal is to
improve the usability of the system by making it easier for the user to
understand the goal of the AR robot.

AR interfaces improve a robot’s motion intent communication through spatial
information. Prior works have experimented with different designs based on
factors such as path, waypoints, velocity, arrival times, etc. (Walker et al.,
2018; Zollmann et al., 2014; Rosen et al., 2017). Most relevant to our work is
Rosen et al., who visualized the robotic arm’s motion intent by overlaying the
expected robot poses with a mixed reality head-mounted display. They evaluated
the performance of the visualization against 2D displays and no visualizations
and found that the system allows a human to more quickly (-62% in time) and
accurately (+16% in accuracy) tell where the robot is going to move. In
addition to visualizing the robot’s motion intent, they iterated on the system
to allow the robot to replan an intended trajectory with MoveIt motion planner
(mov, 2023) with the same start and end points, with the final trajectory
chosen by the human. This has been shown to improve task completion time,
precision, accuracy, usability, and task load compared to the baseline
systems. Unlike this work, our application is in the form of an iOS app, and
the AR visualization is applied with a phone or a tablet. This increases the
potential to enable more people to collect demonstrations for training robots
compared to wearable displays. Furthermore, instead of projecting the entire
pose of the robot in AR, we move the AR robot directly to the goal pose due to
the lack of safety concerns. Similar to how they allow the user to replan the
path, we offer a redo function that allows users to reselect the trajectory
they desire.

##  3\. EVE Prototype 1

Figure 2. Prototype 1 of EVE includes three new AR visualizations. AR
kinesthetic teaching enables the robot to track the user’s hand in real-time.
Path history displays trajectory along collected waypoints. Invisible robot
allows users to toggle the robot body off and only view a cylinder that shows
the robot’s end effector position and the gripper state (green = open, red =
close).

While prior work has compared common demonstration collection interfaces and
various AR visualizations, there is a lack of research investigating user
preferences for AR-based demonstration collection systems. Since AR2-D2 only
provides one way to control and visualize the robot’s movement, we developed
the first EVE prototype with three new AR visualizations for the formative
user study: AR kinesthetic teaching, path history, and invisible robot (Figure
2). Below, we describe the design and implementation of the new AR
visualizations.

###  3.1. AR Kinesthetic Teaching (AR-KT)

Prior work suggests that kinesthetic teaching outperforms teleoperation in
system usability, collection time, and task success rate (Duan et al., 2023).
This is primarily due to the real-time feedback and intuitiveness when
manually guiding the robot with the hand. Inspired by these findings, we
implemented AR-based kinesthetic teaching (AR-KT). When AR-KT is activated,
the AR robot tracks the movement of the user’s hand and adjusts the position
of its end effector to match the location of the user’s hand. To further
enhance the naturalness of the robot’s motion, we employed inverse kinematics
to calculate the joint transformations to make the robot move in synchrony
with the user’s hand.

###  3.2. Path History

Trajectory visualization is a popular method to improve motion intent
communication, as it allows users to preview the robot’s plan before
execution. Previous research has shown that visualizing proposed robot motion
using a mixed reality head-mounted display led to a 16% increase in accuracy
and a 62% decrease in task completion time compared to a 2D display when
labeling robot arm motions as either colliding or non-colliding with objects
(Rosen et al., 2017). Inspired by these findings, we incorporated a virtual
line trajectory using Unity’s line renderer component. The system displayed a
straight green line between the robot’s end effector coordinate and the user’s
hand coordinate to minimize the occlusion of the AR robot and objects. By
utilizing the Jacobian matrix to relate joint velocities to end effector
velocity, applying a damped least squares approach to compute joint angle
updates, and iteratively updating the joint angles, our inverse kinematics
implementation enables the end effector to move in approximately straight-line
motions towards the desired goal pose. Thus, linear interpolation sufficiently
approximates the end effector’s actual trajectory. To support real-time
control and flexibility, we included a redo button that reverts the robot to
the last saved waypoint and removes the corresponding trajectory. We opted for
a redo functionality instead of projecting the robot’s goal pose before
movement because our entire system is AR-based, ensuring safety and efficiency
when moving the robot directly to the goal pose.

###  3.3. Invisible Robot

The AR robot’s 1:1 scale with the real Franka Panda robot often occludes the
surroundings. To address this, the invisible robot visualization renders the
AR robot invisible and displays a cylinder at the end effector’s position. The
cylinder’s color changes based on the gripper’s state: green when the gripper
is open and red when the gripper is closed (see Figure 2). The color provides
users with intuitive visual feedback about the gripper’s current state without
obstructing their view of the environment.

##  4\. formative study

Figure 3. User comments about the six demonstration collection interfaces used
in the formative study.

While prior work has compared various demonstration collection interfaces,
there is a lack of research on demonstration collection with AR. Thus, our
formative study examines the experiences and needs of end users with AR-based
collection interfaces.

###  4.1. Method

Participants. Since EVE aims to support everyday users, we recruited 10
participants via snowball sampling. Participants were screened via a
demographic questionnaire, which asked about their demographics and
backgrounds, including age, gender, area of expertise, familiarity with
robotics (5-point Likert scale), familiarity with demonstration collection
interfaces (5-point Likert scale), and tasks they prefer the robot to help
them with. We followed best practices from the HCI Guidelines for Gender
Equity and Inclusivity (Scheuerman et al., 2020) to collect gender-related
information. Participants ($\mu=20.7$ years old, $\sigma=1.7$ years), denoted
as F01-F10, represented seven different areas of expertise and varying levels
of familiarity in robotics and demonstration collection (Table A). The
participants’ familiarity with robotics ranged from unfamiliar to somewhat
familiar: one person was unfamiliar, five were somewhat unfamiliar, three had
neutral familiarity, and one was somewhat familiar. Regarding their
familiarity with the demonstration collection, six participants were
unfamiliar, and four were somewhat unfamiliar.

Procedure. The formative study took place on a university campus and lasted 50
minutes. We obtained the Institutional Review Board’s approval before the
formative study, and the consent form was taken in person. All sessions were
voice recorded for post hoc analysis. Because we were interested in candid
reactions, we did not tell participants that we created the additional AR
visualizations. Each participant was compensated with food for the study.

The formative study consisted of a two-part demonstration collection with six
demonstration collection interfaces. The researchers began by demonstrating
how to use each interface. Then, participants were given a few minutes to try
each interface and ask any technical questions. Afterward, participants were
given up to five minutes to try each interface by pushing, picking, and
placing various cubes. Since we are investigating the user experience with
various collection interfaces, we did not design the task to simulate a real-
world scenario.

In part one, participants used kinesthetic teaching, teleoperation with an HTC
Vive VR controller, and the AR2-D2 system. After the participants had tried
data collection with the three state-of-the-art collection interfaces, we
conducted a semi-structured interview about their experience, focusing on what
features they liked and disliked about each interface. Although part one does
not provide direct insight into an AR-based demonstration collection system,
we included it to provide participants with a more comprehensive experience
with robot data collection. In part two, participants performed the same task
with AR-KT, path history, and invisible robot visualizations. Aside from
discussing the same questions in part one, we brainstormed additional features
for the AR-based collection interface. Subsequently, participants completed a
survey for System Usability Scale (SUS) scores and a form to rank the motion
intent communication, user enjoyment, and preference for the AR2-D2 system and
the new AR visualizations. Finally, we explained to the participant how the
participant’s data would be used to investigate the design for AR-based
demonstration collection interfaces and answered any relevant questions.

Analysis. We analyzed two sources of data: interview transcripts and the post-
task questionnaires. For the qualitative data, we used reflexive thematic
coding to analyze the usability challenges faced by the participants with the
current AR-based demonstration collection system. The first author, who
facilitated all user study sessions, created an initial codebook by reviewing
study transcripts. The entire team then collaboratively iterated on the
codebook while checking for bias and coverage. With a final codebook
consisting of 15 codes, the team discussed the resulting themes. While the
formative study focuses on participants’ reactions to AR-based collection
interfaces, we summarized and selected several participants’ quotes regarding
what they liked and disliked about each collection interface (Figure 3). For
SUS scores, we converted survey responses, which are on a scale of 0-40 when
summed, to a range between 0-100(Brooke, 1996)111(((Q1 + Q3 + Q5 + Q7 + Q9) -
5) + (25 - (Q2 + Q4 + Q6 + Q8 + Q10))) * 2.5. We then conducted a Friedman
test as an omnibus test with an appropriate number of Wilcoxon signed-rank
tests corrected with Holm’s sequential Bonferroni procedure for statistical
significance. See Table 1 for a summary of quantitative results from the post-
task questionnaires.

|  AR2-D2 |  AR-KT |  Path History |  Invisible Robot
---|---|---|---|---
Usability (SUS) |  66.3 (20.3) |  67.3 (12.3) |  70.0 (13.0) |  68.3 (15.7)
Motion Intent (rank) |  2.9 (1.2) |  2.4 (1.0) |  1.9 (0.9) |  2.8 (1.3)
Enjoyment (rank) |  3.5 (1.0)^ |  2.4 (1.0) |  2.0 (0.7)^ |  2.1 (1.3)
Preference (rank) |  3.5 (1.0)^ |  2.7 (0.8) |  1.5 (0.7)^ |  2.3 (1.1)
|  |  |  |
Table 1. Mean and standard deviation of usability, motion intent
communication, enjoyment, and preference for different AR visualizations.
Usability is rated on a scale from 0 to 100, with higher scores indicating
better usability. Motion intent communication, user enjoyment, and overall
user preference are ranked from 1 to 4, with lower ranks indicating better
performance. For statistical significance, one caret (^) is $p<0.05$.

###  4.2. Findings

A Friedman test found no statistically significant differences between the
four AR visualizations in the SUS scores ($\chi^{2}(3)=2.6$, $p=0.5$) and
rankings of motion intent communication ($\chi^{2}(3)=3.7$, $p=0.3$). The path
history visualization significantly outperformed the baseline AR2-D2 system in
user enjoyment ($p=0.0059$) and preference ($p=0.0059$). Notably, the path
history visualization achieved the highest median scores and rankings across
all measured variables: SUS ($\mu=70.0$, $\sigma=13.0$), motion intent
communication ($\mu=1.9$, $\sigma=0.9$), user enjoyment ($\mu=2.0$,
$\sigma=0.7$), and preference ($\mu=1.5$, $\sigma=0.7$). These findings
suggest that participants found the path history visualization the most
usable, enjoyable, and preferable among the four tested AR visualizations,
even though it did not significantly outperform the others in all aspects.
Interestingly, these results contradicted our hypotheses. Given the benefits
of direct manipulation and real-time visual feedback, we had expected the AR-
KT visualization to achieve the highest scores on system usability and motion
intent communication compared to other visualizations. The contradiction may
be due to the AR robot’s inability to accurately track the hand when it moves
quickly. We also anticipated that the invisible robot visualization would
receive the lowest scores on system usability and motion intent communication
due to the lack of joint movement information. The contradiction could be
attributed to the approximate straight-line motion of the AR robot, which
allows users to easily predict the movement of the joints.

In addition, we distilled the challenges users faced with the current AR
visualizations into the following six based on the semi-structured interviews:

  1. C1

Ambiguous spatial position of the hand and the robot: Four participants (F03,
F04, F08, F10) expressed difficulty interpreting the 3D environment via a 2D
iPad screen. The lack of depth perception made it challenging for users to
accurately gauge the relative positions of their hands and the AR robot. This
ambiguity led to inaccuracies in the collected demonstrations. F03 also
complained about the hand ”passing through” the robot, resulting in an
unnatural collection.

  2. C2

Lack of knowledge of joint constraints: Five participants (F04, F05, F06, F08,
F10) reported a lack of information about the joint limits. This problem,
coupled with the lack of depth perception via the iPad screen, causes the
robot to move unexpectedly when the specified point with the hand is out of
the joint limit.

  3. C3

Imprecise trajectory visualization: The path history visualization displays a
straight green line between the robot’s end effector coordinate and the user’s
hand coordinate. Although three participants (F05, F06, F10) have asserted
that the trajectory visualization aids their understanding of the robot’s
motion intent, three other participants (F02, F04, F07) have proposed to
include a more accurate trajectory.

  4. C4

Absence of feedback regarding the collection efficacy: Five participants (F01,
F04, F08, F09, F10) complained about the lack of collision detection, which
prevented them from verifying whether the collected trajectory was feasible.
F01 claimed that while AR is ”useful and fun,” it does not make sense for the
AR robot to pass through all the obstacles.

  5. C5

Obstructive robot design: The AR robot’s 1:1 scale correspondence with the
physical Franka Panda robot frequently resulted in the occlusion of relevant
objects within the scene. Four participants (F01, F05, F07, F09) reported that
the AR robot’s size significantly impeded the data collection. Specifically,
F01 expressed frustration regarding the robot’s obstruction of the line, while
F09 noted that the robot’s presence obscured a substantial portion of the
iPad’s display.

  6. C6

Inconsistent hand tracking: Inconsistencies in hand tracking emerged as a
significant concern among participants. Five individuals (F01, F02, F03, F06,
F08) highlighted the presence of inaccuracies and latency in the hand tracking
system employed in the AR environment. Furthermore, F02 noted that the gray
line renderer, utilized in the AR2-D2 condition to visualize the tracked
coordinates of the user’s fingers, proved to be visually distracting.

##  5\. EVE Prototype 2

Figure 4. Overview of evaluation user study and prototype 2 of EVE. _Left:_
the study involved three common tabletop tasks: toggling a switch, sorting
food, and sweeping the table. These tasks were utilized to evaluate the user
study with various baseline interfaces. To ensure a fair comparison of
performance across these interfaces, the positions of all target objects and
desired actions were fixed. _Right:_ this section presents a sequence of
frames from a sampled rollout performed by users during the evaluation. It
illustrates how participants utilized the visualization and features of EVE
prototype 2 to successfully complete the required demonstrations for the given
task.

Informed by formative study findings and our own experiences using EVE, we
created a second EVE prototype with seven additional system changes to address
the usability challenges. We describe these advancements below:

###  5.1. Hand Position Projection

To communicate the position of the tracked hand pose in 3D space clearly (C1),
a green line is generated on the surface. We have tried to attach the green
line directly to the hand but found increased visual distraction and no
improvement in position interpretation. To enhance motion intent communication
and reduce potential collisions, the length and width of the projected line
were adjusted to match the dimensions of the robot’s end effector. This
improved representation of the hand position also helps prevent users from
specifying coordinates that are clearly beyond the robot’s joint limits (C2).

###  5.2. Joint Constraints Signifier

To address user concerns about joint constraints in AR2-D2 and EVE prototype 1
(C2), we followed the joint space limits provided in the Franka Emika Panda
robot specifications to define a range of reachable coordinates based on the
instantiation point of the AR robot. When the user’s hand is positioned
outside the reachable range, all the app interface buttons turn red (otherwise
white) and are deactivated, preventing the user from changing the robot’s
state.

###  5.3. Realistic Trajectory

Three participants (F02, F04, F07) indicated that the inaccurate, straight-
line trajectory visualization did not effectively communicate the robot’s
motion intent ((C3)). In response, the system was modified to display the
precise, curved trajectory of the end effector. Moreover, Path History is now
the default visualization for EVE, as it demonstrated superior performance
compared to other AR visualizations in the formative user study.

###  5.4. Trajectory Replay

The absence of feedback regarding collision detection and path feasibility
significantly hinders the system’s usability (C4). However, planning a
trajectory for the AR robot to avoid physical obstacles often requires
constructing a depth map of the world, which can be computationally expensive
for the iPad. After brainstorming potential solutions with the formative study
participants, we added a replay feature demonstrating the collected trajectory
and changes in the gripper state using a partially transparent robot. This
feature makes it easier to determine whether the robot collided with a
physical object during collection. This implementation was also inspired by
some participants’ preferences (F02, F03) for the replay feature available in
kinesthetic teaching. To integrate trajectory visualization and enable replay
for AR-KT, we experimented with various numbers of waypoints and decided to
use 20 waypoints for the trajectory collected with AR-KT.

###  5.5. Dynamic Camera View

To address the issue of the AR robot occupying a significant portion of the
screen (C5), a formative study participant (F05) proposed enabling the user to
move the camera to inspect the 3D world. However, collecting demonstrations
from multiple camera angles is not ideal because it is crucial to ensure that
the hand-eye calibration between the robot action and the 3D voxels is aligned
for a successful transfer to real-world deployment. To address this, we save
the initial camera extrinsic values after the robot has been instantiated and
make the buttons disappear when the current extrinsic values differ from the
saved values by more than a small threshold. This approach allows the user to
freely inspect the scene from multiple angles while ensuring the demonstration
is collected from a consistent camera angle. This feature also addresses the
lack of information about joint constraints (C2) by allowing users to visually
inspect the robot’s state and avoid specifying potentially unreachable
coordinates.

###  5.6. Accurate Gripper Control

In AR2-D2, a gray line is rendered between the thumb and index fingers to
enable users to toggle the gripper opening by adjusting the distance between
the fingers. Due to limitations in the iPad camera and ARKit’s
HandTrackingProvider, hand tracking is inconsistent, and toggling the gripper
state is extremely unpredictable (C6), as reported by five participants from
the formative study (F01, F02, F03, F06, F08). To maintain the system’s
ubiquity, we did not modify the hardware. Instead, we addressed the issue of
the flashing gray line renderer by removing it and adding a button to toggle
the gripper opening accurately. This change ensures a more reliable and
predictable user experience when controlling the gripper state.

###  5.7. Robot Instantiation Reselection

Our experience using the system and the feedback from one participant (F09)
suggest that the inability to change the instantiation point of the robot
interferes with the system’s usability, as we need to quit and restart the app
every time we instantiate the robot at an undesired place. We changed the
robot instantiation process into two steps to address this issue. First, a
green cylinder appears on the screen, signifying where the robot will spawn.
The user is then prompted with two choices: 1) instantiate the robot at the
position indicated by the cylinder, or 2) reselect a position for the green
cylinder. The ability to reselect the instantiation point for the AR robot
greatly increases the system’s usability.

##  6\. evaluation user study

We conducted a user study to evaluate the effectiveness of EVE compared to
baseline interfaces for three common tabletop tasks. These tasks were designed
based on the results of an extensive survey from BEHAVIOR-1K (Li et al., 2024)
asking participants, ”What do you want robots to do for you?” We included
obstacles in all tasks to simulate real-world environments where tables are
often cluttered. The three tasks were as follows:

  * •

Toggle switch: Participants guided the robot to push a switch and turn on a
light bulb.

  * •

Sort food: Participants were required to grab a melon and place it inside a
yellow bowl, and then grab a steak and put it into a basket. This task
simulated the sorting of different items into specific locations.

  * •

Sweep table: Participants used a sponge to sweep beans to a location marked by
tape. This task simulated the cleaning of debris from a table.

The setups for these tasks are illustrated in Figure 4.

###  6.1. Method

Participants. To ensure a diverse area of expertise, we recruited 14
participants via snowball sampling. Participants were screened using the same
demographic questionnaire from the formative study. We followed best practices
from the HCI Guidelines for Gender Equity and Inclusivity to collect gender-
related information. Participants ($\mu=22.6$ years old, $\sigma=2.6$ years),
denoted as E01-E14, represented 11 different areas of expertise and varying
levels of familiarity in robotics and demonstration collection (Table A). The
participants’ familiarity with robotics ranged from unfamiliar to familiar:
three people were unfamiliar, six people were somewhat unfamiliar, three
people had neutral familiarity, and two people were familiar. Regarding their
familiarity with the demonstration collection, eight people were unfamiliar,
three people were somewhat unfamiliar, two people had neutral familiarity, and
one person was somewhat familiar. Each participant performed one to three
tasks, resulting in a total of 30 collections (10 collections per task). None
of the participants participated in the formative study.

Procedure. The in-person laboratory study took place on a university campus,
and each task lasted 40 minutes. The consent form was taken in person. Because
we were interested in candid reactions, we did not tell participants that we
created EVE. Each participant was compensated with food for the laboratory
study.

We began by providing a short tutorial for all the collection interfaces
(i.e., kinesthetic teaching, teleoperation with an HTC Vive VR controller,
AR2-D2, and EVE) and informed them of the goal of the task(s). We did not
demonstrate how to accomplish the task(s) because we did not want to influence
the trajectory collected by the participants. After a 30-second practice
session before each collection, participants were asked to complete the task
in three minutes. We recorded each collection’s task success rate and
remaining time (three minutes - collection time). We allowed the participant
to retry the collection if a collision occurred and the collection time had
not reached three minutes. After the participants had completed the collection
with all the collection interfaces, we asked them to complete the SUS survey
and a form to rank the motion intent communication, user enjoyment, and
overall preference for all collection interfaces.

Analysis. We analyzed the results from the SUS survey and ranking forms with
the same method used for the formative study result analysis. See Table 2 for
a summary of quantitative results from the post-task questionnaires. We
calculated the overall task success rate and the mean and standard deviation
of the remaining time for each collection interface.

###  6.2. Findings

EVE achieved the highest task success rate and performed comparably to
kinesthetic teaching in task completion time, system usability, motion intent
communication, user enjoyment, and overall user preference ($mean_{p}=0.30$).
Kinesthetic teaching attained the highest SUS score and ranked the highest in
motion intent, user enjoyment, and overall preference. However, there was no
statistically significant difference between EVE and kinesthetic teaching,
suggesting that EVE may be a viable substitute for collecting trajectory data.
This might be because of the ability to clearly inspect the real world before
moving the robot, a feature not present in AR2-D2 and teleoperation. Our
results confirm the findings of previous work (Wrede et al., 2013; Mandlekar
et al., 2018), which showed that kinesthetic teaching outperforms VR
controllers in all measurements. EVE outperformed AR2-D2 in both demonstration
collection success rate and remaining time. In all tasks performed by the
participants, EVE surpassed AR2-D2 in SUS score, motion intent, enjoyment, and
overall preference ranking.

Figure 5. The mean success rate (%) for the evaluation user study, which encompassed three tasks with 10 trials each ($D=10$), indicates that the EVE achieved a 100% success rate across all tasks. This performance surpasses that of the second-highest interface, kinesthetic teaching, by an average margin of 10% across the tasks. Figure 6. The average remaining time in seconds and its standard deviation taken to successfully collected one demonstration for each tasks. EVE’s performance in terms of time efficiency is comparable to that of kinesthetic teaching, with a narrow average difference of 5.1 seconds across three tasks. Overall |  Kinesthetic Teaching |  Teleoperation |  AR2-D2 |  EVE
---|---|---|---|---
Usability (SUS) |  86.0 (17.0) |  27.3 (17.8)* |  59.0 (20.5)* |  77.8 (17.0)
Motion Intent (rank) |  1.4 (0.7) |  3.3 (0.8)* |  3.4 (0.7)* |  1.9 (0.8)
Enjoyment (rank) |  1.5 (0.8) |  3.2 (1.0)* |  3.3 (0.7)* |  2.0 (0.8)
Preference (rank) |  1.7 (0.9) |  3.4 (0.9)* |  3.1 (0.7)* |  1.8 (0.7)
|  |  |  |
Toggle Switch |  Kinesthetic Teaching |  Teleoperation |  AR2-D2 |  EVE
Usability (SUS) |  83.8 (18.0) |  31.3 (21.5)* |  66.0 (20.2)* |  82.8 (13.9)
Motion Intent (rank) |  1.4 (0.7) |  3.4 (1.0) |  3.2 (0.6)* |  2.0 (0.8)
Enjoyment (rank) |  2.0 (1.1) |  3.0 (1.2) |  3.1 (0.9)* |  1.9 (1.0)
Preference (rank) |  2.2 (1.0) |  3.3 (1.1) |  2.9 (0.9)* |  1.6 (0.8)
|  |  |  |
Sort Food |  Kinesthetic Teaching |  Teleoperation |  AR2-D2 |  EVE
Usability (SUS) |  88.0 (17.1)* |  26.5 (15.5)* |  54.0 (20.6)* |  71.0 (17.5)
Motion Intent (rank) |  1.3 (0.5) |  3.1 (0.7) |  3.7 (0.5)* |  1.9 (0.7)
Enjoyment (rank) |  1.2 (0.4)* |  3.1 (1.1) |  3.5 (0.5)* |  2.2 (0.6)
Preference (rank) |  1.2 (0.4) |  3.5 (0.7)* |  3.4 (0.5)* |  1.9 (0.6)
|  |  |  |
Sweep Table |  Kinesthetic Teaching |  Teleoperation |  AR2-D2 |  EVE
Usability (SUS) |  86.3 (17.5) |  24.3 (17.1)* |  56.8 (20.9)* |  80.0 (18.7)
Motion Intent (rank) |  1.6 (0.8) |  3.3 (0.8) |  3.3 (0.8)* |  1.8 (0.8)
Enjoyment (rank) |  1.4 (0.7) |  3.5 (0.8)* |  3.2 (0.6)* |  1.9 (0.7)
Preference (rank) |  1.6 (0.8) |  3.5 (0.8)* |  3.1 (0.7)* |  1.8 (0.8)
|  |  |  |
Table 2. Mean and standard deviation of usability, motion intent
communication, enjoyment, and overall preference for different demonstration
collection interfaces. Usability is rated on a scale from 0 to 100, with
higher scores indicating better usability. Motion intent communication,
enjoyment, and overall preference are ranked from 1 to 4, with lower ranks
indicating better performance. One asterisk (*) indicates statistical
significance ($p<0.05$) when compared to the EVE system.

Below are some notable comments from users based on their observations of
different collection interfaces used in three tasks during the evaluation
study:

Toggle Switch. This task involves operating an augmented robot gripper to
navigate around potential obstacles and accurately activate a switch. In
kinesthetic teaching, users appreciated the method’s simplicity and precise
effector control, facilitating movement to the desired location. However, this
system does not support the modification of previously established waypoints.
For teleoperation and AR2-D2, many users reported failures, attributing them
to collisions with scene distractors and the systems’ inability to provide a
comprehensive scene view due to robotic arm obstructions. Specifically,
teleoperation users struggled with accurately positioning the robot end-
effector to engage the switch, hindered by a lack of haptic feedback.
Conversely, in our EVE system, users found the AR-KT features significantly
improved control during the final positioning of the gripper, and the
invisible robot mode provided a clear view to assess potential collisions with
distractors.

Sort Food. This task required users to control a robot to perform a series of
long-horizon actions, including picking up and placing multiple objects. In
kinesthetic teaching, users found it straightforward to maneuver the robot for
pick-and-place actions, praising its intuitiveness and ease of use. However,
in teleoperation, the robot’s obstruction of the end-effector’s view
complicated precise control, leading to difficulty in achieving the ideal
grasp pose and frequent minor errors, often resulting in objects being in
ungraspable states. In AR2-D2, users struggled due to inadequate visualization
of potential collisions between the robot arm and table distractors.
Conversely, the EVE system was well-received, with users appreciating the path
history for tracking long-horizon actions and the redo feature for correcting
waypoints. The AR-KT functionality was highlighted for its precise control,
allowing for accurate object placement and enhanced gripper manipulation.

Sweep Table. In this task, users needed precise control to grasp onto the
sponge and execute a series of sweeping actions. Kinesthetic teaching was
favored for its simplicity, allowing users to manipulate the robot’s end-
effector directly. For the teleoperation system, users encountered
difficulties due to latency and the precision required to control the end-
effector with a VR controller. They reported struggling to maintain a
consistent direction of movement when controlling the end-effector, often
leading to collisions between the end-effector and the table. With AR2-D2, the
challenge was also in using fixed forward control to linearly move the end-
effector along one axis, compounded by unreliable tracking of the desired
gripper state. Conversely, the EVE system provided enhanced precision through
its AR-KT features, offering accurate gripper control that ensured the gripper
maintained the state desired by the user.

##  7\. discussion

In this section, we reflect on our findings from the development and
evaluation of EVE. We also discuss future opportunities for research exploring
AR-based demonstration collection interfaces for robotics.

###  7.1. Implications

Personalized Robot Training in Everyday Environments. By allowing users to
easily capture videos and manipulate objects in various locations outside of
traditional laboratory settings, EVE enables individuals to expose robots to a
wide range of objects and environments that are relevant to their specific
needs and preferences. As the cost of robot hardware decreases and household
robots become more prevalent, EVE puts the power of robot training directly
into the hands of users, allowing them to collect personalized demonstrations
for specific household tasks and fine-tune their robots to perform these tasks
effectively in their unique home environments.

Integrating with Other Types of Robots. EVE allows users to quickly adapt the
system to work with different robot platforms. Because each component of the
program is separated, by replacing the robot prefab in Unity with a different
robot model, such as the UR5e, users can collect demonstrations with a new AR
robot. Furthermore, since the waypoint data collected by EVE is a standard
input to multiple motion planners, EVE enables the training of manipulation
policies for different robots without modifying the collected data. This
flexibility empowers individuals to extend the applicability of EVE to a wide
range of robotic systems, making robot training accessible to a broader
audience.

###  7.2. Limitations

Multi-View Data Collection. EVE is limited to single-viewpoint data collection
to maintain accurate hand-eye calibration between the robot’s actions and the
corresponding 3D voxel representation, which is essential for successful real-
world deployment. However, multi-view data collection could be helpful in
scenarios where objects are occluded from one perspective but visible from
another, or when the demonstrator needs to maintain a more comfortable posture
by collecting trajectories from different angles. Future work could explore
the use of simultaneous localization and mapping (SLAM) techniques (Durrant-
Whyte and Bailey, 2006) to track the iPad LiDAR’s pose and build a 3D map of
the workspace, allowing the demonstrator to collect trajectories relative to
this 3D map.

Bimanual Robotic Manipulation. Many real-world tasks, such as washing dishes,
ironing clothes, and assembling furniture, require bimanual manipulation.
However, collecting demonstrations for bimanual tasks using EVE is challenging
due to the inability to press buttons during collection. Future research could
explore the use of mixed reality head-mounted displays for demonstration
collection. With the increasing availability of novel head-mounted displays,
collections using these devices might become more affordable and ubiquitous.
Alternative input modalities, such as gaze and speech, could be used to
trigger different functions instead of clicking buttons.

Collection for Mobile Robot. The real world also involves tasks that require
mobile robots, such as cleaning the floor and watering plants. Extending EVE
to support demonstration collection for mobile robots presents an exciting
direction for future work. To achieve this, researchers can integrate a 3D
model representing the mobile robot into Unity and develop functionalities to
control the robot’s movement through intuitive touch gestures. The system can
then leverage the device’s camera and inertial measurement unit (IMU) sensors
to construct a detailed map of the environment while simultaneously tracking
the robot’s pose, enabling data collection for mobile robots.

Hardware. The current system’s hand-tracking capabilities are not fully
optimized due to hardware constraints. The LiDAR technology employed in the
iPad occasionally results in inaccuracies in the end effector’s movement and
insufficient anchoring of the robot base on the surface. To address this issue
and improve tracking accuracy and robustness, future work can implement sensor
fusion techniques by combining data from multiple sensors, such as the iPad’s
camera, inertial measurement unit (IMU), and LiDAR scanner.

##  8\. conclusion

We introduced EVE, a system that democratizes robot data collection with
human-centric AR visualizations. Informed by our formative study ($N=10$), EVE
enables end users to collect, change, and verify valuable trajectory data to
train robot manipulation for personalized tasks. Our evaluation user study
($N=14,D=30$) demonstrated improved task success rate, completion time, system
usability, motion intent communication, user enjoyment, and user preference
against state-of-the-art collection interfaces. We hope EVE will catalyze
future work in AR-based demonstration collection for robotics and expedite the
incorporation of robots into daily lives.

## References

  * (1)
  * mov (2023) 2023\.  MoveIt Motion Planning Framework.  https://moveit.picknik.ai/main/doc/concepts/motion_planning.html.  Accessed: 2023-04-03.
  * Akgün and Subramanian (2011) Barış Akgün and Kaushik Subramanian. 2011.  Robot Learning from Demonstration : Kinesthetic Teaching vs . Teleoperation.  https://api.semanticscholar.org/CorpusID:45150299
  * Argall et al. (2009) Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. 2009.  A survey of robot learning from demonstration.  _Robotics and Autonomous Systems_ 57, 5 (2009), 469–483.
  * Brohan et al. (2022) Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. 2022\.  Rt-1: Robotics transformer for real-world control at scale.  _arXiv preprint arXiv:2212.06817_ (2022).
  * Brooke (1996) John Brooke. 1996.  SUS: A ”quick and dirty” usability scale.  _Usability evaluation in industry_ 189, 3 (1996), 189–194.
  * Chi et al. (2024) Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. 2024.  Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots.  _arXiv preprint arXiv:2402.10329_ (2024).
  * Chidambaram et al. (2012) Vijay Chidambaram, Yueh-Hsuan Chiang, and Bilge Mutlu. 2012.  Designing persuasive robots: how robots might persuade people using vocal and nonverbal cues. In _Proceedings of the Seventh Annual ACM/IEEE International Conference on Human-Robot Interaction_ (Boston, Massachusetts, USA) _(HRI ’12)_. Association for Computing Machinery, New York, NY, USA, 293–300.  https://doi.org/10.1145/2157689.2157798
  * Dragan et al. (2013) Anca D. Dragan, Kenton C. T. Lee, and Siddhartha S. Srinivasa. 2013.  Legibility and predictability of robot motion.  _2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_ (2013), 301–308.  https://api.semanticscholar.org/CorpusID:15202367
  * Duan et al. (2023) Jiafei Duan, Yi Ru Wang, Mohit Shridhar, Dieter Fox, and Ranjay Krishna. 2023.  AR2-D2: Training a Robot Without a Robot.  (2023).
  * Durrant-Whyte and Bailey (2006) Hugh Durrant-Whyte and Tim Bailey. 2006.  Simultaneous Localisation and Mapping (SLAM): Part I The Essential Algorithms.  _IEEE Robotics & Automation Magazine_ 13, 2 (2006), 99–110.  https://doi.org/10.1109/MRA.2006.1638022
  * Ebert et al. (2021) Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine. 2021.  Bridge data: Boosting generalization of robotic skills with cross-domain datasets.  _arXiv preprint arXiv:2109.13396_ (2021).
  * Elephant Robotics (2023) Elephant Robotics. 2023.  mechArm Pi: The Most Compact 6-Axis Robot Arm.  https://shop.elephantrobotics.com/collections/mecharm/products/mecharm.  Accessed on [insert date of access].
  * Fischer et al. (2016) Kerstin Fischer, Franziska Kirstein, Lars Christian Jensen, Norbert Krüger, Kamil Kukliński, Maria Vanessa aus der Wieschen, and Thiusius Rajeeth Savarimuthu. 2016.  A Comparison of Types of Robot Control for Programming by Demonstration. In _The Eleventh ACM/IEEE International Conference on Human Robot Interaction_ (Christchurch, New Zealand) _(HRI ’16)_. IEEE Press, 213–220.
  * Huang and Mutlu (2013) Chien-Ming Huang and Bilge Mutlu. 2013.  Modeling and Evaluating Narrative Gestures for Humanlike Robots. In _Robotics: Science and Systems_.  https://api.semanticscholar.org/CorpusID:13278469
  * Kalegina et al. (2018) Alisa Kalegina, Grace Schroeder, Aidan Allchin, Keara Berlin, and Maya Cakmak. 2018.  Characterizing the Design Space of Rendered Robot Faces. In _Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction_ (Chicago, IL, USA) _(HRI ’18)_. Association for Computing Machinery, New York, NY, USA, 96–104.  https://doi.org/10.1145/3171221.3171286
  * Kapadia et al. (2017) Rishi Kapadia, Sam Staszak, Lisa Jian, and Ken Goldberg. 2017.  EchoBot: Facilitating data collection for robot learning with the Amazon echo. In _2017 13th IEEE Conference on Automation Science and Engineering (CASE)_. IEEE, 159–165.
  * Li et al. (2024) Chengshu Li, Ruohan Zhang, J. Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martín, Chen Wang, Gabrael Levine, Wensi Ai, Benjamin Martinez, Hang Yin, Michael Lingelbach, Minjune Hwang, Ayano Hiranaka, Sujay S. Garlanka, Arman Aydin, Sharon Lee, Jiankai Sun, Mona Anvari, Manasi Sharma, Dhruva Bansal, Samuel Hunter, Kyu-Young Kim, Alan Lou, Caleb R. Matthews, Ivan Villa-Renteria, Jerry Huayang Tang, Claire Tang, Fei Xia, Yunzhu Li, Silvio Savarese, Hyowon Gweon, C. Karen Liu, Jiajun Wu, Fei-Fei Li, and Salesforce Research. 2024.  BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation.  https://api.semanticscholar.org/CorpusID:268520018
  * Mainprice et al. (2010) Jim Mainprice, Emrah Akin Sisbot, Thierry Siméon, and Rachid Alami. 2010.  Planning Safe and Legible Hand-over Motions for Human-Robot Interaction.  https://api.semanticscholar.org/CorpusID:6846660
  * Mandlekar et al. (2018) Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, et al. 2018\.  Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In _Conference on Robot Learning_. PMLR, 879–893.
  * Mutlu et al. (2006) Bilge Mutlu, Jodi Forlizzi, and Jessica K. Hodgins. 2006.  A Storytelling Robot: Modeling and Evaluation of Human-like Gaze Behavior.  _2006 6th IEEE-RAS International Conference on Humanoid Robots_ (2006), 518–523.  https://api.semanticscholar.org/CorpusID:18092365
  * Osentoski et al. (2010) Sarah Osentoski, Christopher Crick, Greg Jay, and O. C. Jenkins. 2010.  Crowdsourcing for closed loop control. In _Proc. of the NIPS Workshop on Computational Social Science and the Wisdom of Crowds_. NIPS, 4–7.
  * Robot (2023) Hello Robot. 2023.  Hello Robot: Open Source Mobile Manipulator for AI & Robotics.  https://hello-robot.com/.  Accessed on [insert date of access].
  * Rosen et al. (2017) Eric Rosen, David Whitney, Elizabeth Phillips, Gary Chien, James Tompkin, George Konidaris, and Stefanie Tellex. 2017.  Communicating Robot Arm Motion Intent Through Mixed Reality Head-mounted Displays.  arXiv:1708.03655 [cs.RO]
  * Scheuerman et al. (2020) Morgan Klaus Scheuerman, Katta Spiel, Oliver L Haimson, Foad Hamidi, and Stacy M Branham. 2020.  HCI Guidelines for Gender Equity and Inclusivity. In _UMBC Faculty Collection_. UMBC.
  * Seo et al. (2023) Mingyo Seo, Steve Han, Kyutae Sim, Seung Hyeon Bang, Carlos Gonzalez, Luis Sentis, and Yuke Zhu. 2023.  Deep Imitation Learning for Humanoid Loco-manipulation through Human Teleoperation. In _2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids)_. IEEE, 1–8.
  * Shaw et al. (2023a) Kenneth Shaw, Ananye Agarwal, and Deepak Pathak. 2023a.  Leap hand: Low-cost, efficient, and anthropomorphic hand for robot learning.  _arXiv preprint arXiv:2309.06440_ (2023).
  * Shaw et al. (2023b) Kenneth Shaw, Shikhar Bahl, and Deepak Pathak. 2023b.  Videodex: Learning dexterity from internet videos. In _Conference on Robot Learning_. PMLR, 654–665.
  * Shridhar et al. (2022) Mohit Shridhar, Lucas Manuelli, and Dieter Fox. 2022.  Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation.  arXiv:2209.05451 [cs.RO]
  * Suzuki et al. (2022) Ryo Suzuki, Adnan Karim, Tian Xia, Hooman Hedayati, and Nicolai Marquardt. 2022.  Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic Interfaces. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 553, 33 pages.  https://doi.org/10.1145/3491102.3517719
  * Szafir and Szafir (2021) Daniel Szafir and Danielle Albers Szafir. 2021.  Connecting human-robot interaction and data visualization. In _Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction_. 281–292.
  * Toedtheide et al. (2023) Alexander Toedtheide, Xiao Chen, Hamid Sadeghian, Abdeldjallil Naceri, and Sami Haddadin. 2023.  A force-sensitive exoskeleton for teleoperation: An application in elderly care robotics. In _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 12624–12630.
  * Walker et al. (2018) Michael Walker, Hooman Hedayati, Jennifer Lee, and Daniel Szafir. 2018.  Communicating Robot Motion Intent with Augmented Reality. In _Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction_ (Chicago, IL, USA) _(HRI ’18)_. Association for Computing Machinery, New York, NY, USA, 316–324.  https://doi.org/10.1145/3171221.3171253
  * Wang et al. (2023) Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu, and Anima Anandkumar. 2023.  MimicPlay: Long-Horizon Imitation Learning by Watching Human Play.  arXiv:2302.12422 [cs.RO]
  * Wang et al. (2024) Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang, Li Fei-Fei, and C Karen Liu. 2024.  DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation.  _arXiv preprint arXiv:2403.07788_ (2024).
  * Wrede et al. (2013) Sebastian Wrede, Christian Emmerich, Ricarda Grünberg, Arne Nordmann, Agnes Swadzba, and Jochen Steil. 2013.  A user study on kinesthetic teaching of redundant robots in task and configuration space.  _Journal of Human-Robot Interaction_ 2, 1 (2013), 56–81.
  * Wu et al. (2023) Philipp Wu, Yide Shentu, Zhongke Yi, Xingyu Lin, and Pieter Abbeel. 2023.  Gello: A general, low-cost, and intuitive teleoperation framework for robot manipulators.  _arXiv preprint arXiv:2309.13037_ (2023).
  * Zare et al. (2023) Maryam Zare, Parham M. Kebria, Abbas Khosravi, and Saeid Nahavandi. 2023.  A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges.  arXiv:2309.02473 [cs.LG]
  * Zhao et al. (2023) Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. 2023.  Learning fine-grained bimanual manipulation with low-cost hardware.  _arXiv preprint arXiv:2304.13705_ (2023).
  * Zhu et al. (2022) Yifeng Zhu, Abhishek Joshi, Peter Stone, and Yuke Zhu. 2022.  Viola: Imitation learning for vision-based manipulation with object proposal priors.  _arXiv preprint arXiv:2210.11339_ (2022).
  * Zollmann et al. (2014) Stefanie Zollmann, Christof Hoppe, Tobias Langlotz, and Gerhard Reitmayr. 2014.  FlyAR: Augmented Reality Supported Micro Aerial Vehicle Navigation.  _IEEE Transactions on Visualization and Computer Graphics_ 20, 4 (2014), 560–568.  https://doi.org/10.1109/TVCG.2014.24

##  Appendix A Study Participants Demographics

# | f | e |  Area of Expertise | Familiarity (Robotics) |  Familiarity (Demonstration Collection) | Tasks
---|---|---|---|---|---|---
1 | F01 | - |  Marketing | Somewhat unfamiliar |  Somewhat unfamiliar | Chores, remind about deadlines
2 | F02 | - |  Math | Neutral |  Somewhat unfamiliar | Chores
3 | F03 | - |  Food Systems | Somewhat unfamiliar |  Unfamiliar | Cooking
4 | F04 | - |  Applied Math | Unfamiliar |  Unfamiliar | Cooking, driving
5 | F05 | - |  Math | Somewhat unfamiliar |  Unfamiliar | Manage files
6 | F06 | - |  Music | Neutral |  Unfamiliar | Grab packages, sort folders
7 | F07 | - |  Applied Math | Somewhat familiar |  Unfamiliar | Fetch items
8 | F08 | - |  Computer Science | Somewhat unfamiliar |  Somewhat unfamiliar | Cooking, driving, grab packages
9 | F09 | - |  Statistics | Somewhat unfamiliar |  Unfamiliar | Chores, teaching
10 | F10 | - |  Computer Science | Neutral |  Somewhat unfamiliar | Assemble car components
11 | - | E01 |  Music | Unfamiliar |  Unfamiliar | Grab packages
12 | - | E02 |  Electrical Engineering | Somewhat unfamiliar |  Unfamiliar | Chores
13 | - | E03 |  Statistics | Unfamiliar |  Unfamiliar | Fetch items
14 | - | E04 |  Business | Somewhat unfamiliar |  Somewhat unfamiliar | Chores
15 | - | E05 |  Computer Science | Somewhat unfamiliar |  Unfamiliar | Fold clothes
16 | - | E06 |  Electrical Engineering | Neutral |  Somewhat unfamiliar | Cooking
17 | - | E07 |  Education | Somewhat unfamiliar |  Unfamiliar | Teaching
18 | - | E08 |  Informatics | Somewhat unfamiliar |  Unfamiliar | Grab packages
19 | - | E09 |  Genetics | Neutral |  Unfamiliar | Cleaning
20 | - | E10 |  Computer Science | Neutral |  Neutral | Sort folders
21 | - | E11 |  Informatics | Somewhat unfamiliar |  Somewhat unfamiliar | Grab packages
22 | - | E12 |  Math | Unfamiliar |  Unfamiliar | Teaching
23 | - | E13 |  Robotics | Familiar |  Neutral | Fold clothes
24 | - | E14 |  Mechanical Engineering | Familiar |  Somewhat familiar | Factory inspection, fetch items
Table 3. Participant demographics in formative and evaluation study. In total,
there were 24 participants overall (#). f denotes the participant IDs for the
formative study, e denotes the participant IDs for the evaluation study.