#  Approach to Evaluating Characteristics of Multichannel Loss System with
FCFD Preempted Priority Discipline

Tatashev A.G.
Department of Higher Mathematics
Moscow Automobile and Road Construction
State Technical University (MADI)
Moscow, Leningradsky avenue, 64, Russia
<EMAIL_ADDRESS>
Seleznjev O.V.
Umea University, Sweden
<EMAIL_ADDRESS>
Yashina M.V.
Department of Higher Mathematics
Moscow Automobile and Road Construction
State Technical University (MADI)
Moscow, Leningradsky avenue, 64, Russia
<EMAIL_ADDRESS>

###### Abstract

In the paper, we consider a multichannel loss preemptive priority system with
a Poisson input and general service time distribution depending on the
priority of job. Jobs of the same priority are preempted according with First
Come First Displaced (FCFD) protocol. Approximate formulas are obtained for
the loss probability of a prescribed priority job and some other
characteristics of the system. It particular cases, the obtained formulas are
exact.

_Keywords_ Queueing systems $\cdot$ loss multichannel system $\cdot$ loss
probability $\cdot$ first come — last displaced protocol (FCLD); $\cdot$ first
come — first displaced protocol (FCFD) $\cdot$ limited processor sharing
$\cdot$ approximate formulas

##  1 Introduction

Exact results for characteristics of queueing systems with priority
disciplines are known for one-channel system and systems in that service time
is distributed exponentially with parameter independent of job priority [1]–
[7]. Approximate approaches to evaluate characteristics of multichannel
priority waiting queueing systems with general service time distributions was
proposed in [8]– [11]. In [12], an approximate approach is proposed to
evaluate characteristics of a loss multichannel preemptive priority system
such that, if a job arrives to the system and all servers are processing jobs
of the same priority or higher, then the arriving job is lost. If all servers
are busy and at least one job with a lower priority is serviced, then the job
of the lowest priority among the priorities of jobs in service is preempted
and lost such that the preempted job arrived later than the other jobs of this
priority (Last Come, First Displaced protocol, LCFD).

This paper proposes an approximate approach to evaluate the loss probability
and some other characteristics of a loss multichannel preemptive priority
system such that, if a job arrives to the system, and all servers are
processing jobs of the higher priority, then the arriving job is lost. If all
servers are busy and at least one job with the same or a lower priority is
serviced, then a job of the lowest priority among the priorities of jobs in
service is lost such that this job arrived earlier than the other jobs of this
priority (First Come, First Displaced protocol, FCFD). The FCFD disciplines
are useful when the importance of call decreases with the elapsing time [13].
In the case of one-channel and in the case of exponential service time
distribution, the proposed formulas are exact. The formulas are also exact if,
with prescribed probabilities, the service time equals 0 or is distributed
exponentially with parameter independent of the job priority. Section 2
describes the considered system. In Section 3, an approach is proposed to
evaluate the probability that a job of a prescribed priority is lost at the
arrival moment. Section 4 proposes approach to evaluate the probability that a
accepted job of prescribed priority is lost due to preemption. An approximate
formula for the loss probability is proposed in Section 5. In Section 6, a
numeric example is presented.

##  2 System Description

We use the following notations: $v_{i}$ is the average sojourn time for a job
of the priority $i;$ $w_{i}$ is the average waiting time including the service
interruptions for the priority $i$ job; $p_{i}$ is the probability that the
service of priority $i$ class job does not start immediately; $u_{i}$ is the
average time before the start of the priority $i$ job service provided this
time is not equal to 0; $h_{i}$ is the average number of the priority $i$ job
preemptions $i=1,\dots,N;$ $g_{i}$ is the duration of a service interruption
interval for the priority $i$ job, $i=2,\dots,N;$ $\Lambda_{i}$ is the total
arrival rate of priority-classes no lower than $i:$
$\Lambda_{i}=\lambda_{1}+\dots+\lambda_{i};$ $R_{i}$ is the load due to
priority-classes no lower than $i:$ $R_{i}=(\lambda_{1}+\dots+\lambda_{i})/m,$
$i=1,\dots,N.$

Denote by $c_{i}$ the probability of non-zero waiting for $M/G/m$ system
computed by well-known Erlang’s formula for a waiting system with the arrival
load $R_{i}:$

$c_{i}=\frac{(mR_{i})^{m}}{m!(1-R_{i})\sum\limits_{k=0}^{m-1}\frac{(mR_{i})^{k}}{k!}+(mR_{i})^{m}}.$

The following equalities are true:

$w_{1}=p_{1}u_{1},$ $None$ $w_{i}=p_{i}u_{i}+h_{i}g_{i},\ i=2,\dots,N,$ $None$
$v_{i}=w_{i}+b_{i},\ i=1,\dots,N,$ $None$
$h_{i}=\frac{\Lambda_{i}(p_{i}-p_{i-1})}{\lambda_{i}},\ i=1,\dots,N.$ $None$

The proof of (4) is similar to the proof of an analogous statement for a
preemptive priority system such that, in this system, the service distribution
is exponential with average value independent of the priority class. The proof
of (4) is the following. The probability that all servers are busy by jobs of
priority-classes not lower $i$ and there is at least one job of priority-class
$i$ equals the difference of the probability that all servers are busy by jobs
of priorities not lower than $i$ and the probability that all servers are busy
by jobs fo priorities not lower $i-1,$ and therefore this probability is
$p_{i}-p_{i-1}.$ Hence the average number of preemptions of the priority $i$
per a time unit is equal to $\Lambda_{i-1}(p_{i}-p_{i-1}).$ From this, taking
into account that the average number of arriving priority-class $i$ jobs per a
time unit is equal to $\lambda_{i},$ one gets (1).

##  3 Evaluation of probability that job is lost at its arrival moment

Suppose $\Lambda_{i}$ is the total arrival rate of the priority $i$ and
higher, $\Lambda_{i}=\sum\limits_{j=1}^{i}\lambda_{j};$ $R_{i}$ is the
arriving load due to priority $i$ and higher;
$R_{i}=\sum\limits_{j=1}^{i}\lambda_{j}b_{j};$ $\beta(s)$ is the
Laplace–Stieltjes of distribution $B_{i}(x),$
$\beta_{i}(s)=\int\limits_{0}^{\infty}e^{-sx}dB_{i}(x);$ $q_{i}$ is the
probability that a job is lost at its arrival moment, $r_{i}$ is the
probability that a job of the $j$th priority is lost provided that this job
was accepted for the service; $\gamma_{i}$ is the loss probability for a job
of $i$th priority, $j=1,\dots,i.$

Let the considered queuing system be called the system $S.$ Let us describe
auxiliary $m$-channel queueing systems $S_{i},$ $i=1,\dots,N.$ There are $N$
Poisson arrival processes of different priorities, and the rate of the $i$th
priority processes is equal to $\lambda_{j},$ $j=1,\dots,N.$ If there are less
than $m$ jobs in the system, then they are serviced as usual. If there are
$m-1$ jobs in the system, and a new job arrives, then the arriving job starts
to be serviced at a rate increased by $m$ times, and the service of the other
jobs stops until their number becomes less than $m.$ If there are $m$ jobs in
the system and a new job arrives, and the priority of the serviced job is
higher than the priority of the arriving job, then the arriving job is lost.
If the priority of the arriving job is not lower, then the priority of the
serviced job, then the arriving job starts to be serviced at a rate increased
by $m$ times, and the job that was in service is lost.

Let us introduce the queuing system $S_{i}^{\prime},$ $i=1,\dots,N.$ This
system is a loss one-channel system with a Poisson input and the preemptive
priority discipline. The rate of the arrival process for the $i$th priority
job is equal to $\lambda_{j},$ $j=1,\dots,i.$ The service time distribution
for a job of the $j$th priority is $B_{j}(mx),$ i.e., the service rate
increases by $m$ times.

Suppose $p_{ki}$ is the stationary probability that there are $k$ jobs in the
system $S_{i},$ $k=0,1,\dots,m;$ $p_{ki}^{\prime}$ is the stationary
probability that there are $k$ jobs in the system $S_{i}^{\prime},$ $k=0,1;$
$g_{i}$ is the average duration of the busy period of the system
$S_{i}^{\prime}.$ We have

$p_{0i}^{\prime}=\frac{1/\Lambda_{i}}{g_{i}+\frac{1}{\Lambda_{i}}},\
p_{1i}^{\prime}=\frac{g_{i}}{g_{i}+\frac{1}{\Lambda_{i}}},\ i=1,\dots,N.$

Let us obtain the value $g_{i}.$ Denote by $d_{j}$ the average duration of a
busy period of the system $S_{i}^{\prime}$ provided that this period starts
with servicing a job of the $j$th priority, $j=1,\dots,i.$ The probability
that a busy period of the system $S_{i}^{\prime}$ starts with servicing a job
of the $j$th probability is $\lambda_{j}/\Lambda_{i}.$ Using the total
probability formula, we get

$g_{i}=\frac{1}{\Lambda_{i}}\sum\limits_{j=1}^{i}\lambda_{j}d_{j}.$ $None$

The probability that during the service time of a job of $j$th priority, no
job of a priority not lower than $i$ is
$\int\limits_{0}^{\infty}e^{-\Lambda_{i}x}dB_{j}(mx),$ or
$\beta_{j}\left(\frac{\Lambda_{j}}{m}\right).$ If the the $j$th priority job
is preemptied, then, with probability $\lambda_{s}/\Lambda_{j},$ the
preemption is due the arrival of the $s$th priority job, and the average
remaining time of the busy period equals $g_{s},$ $s=1,\dots,j.$ The average
service time of the $j$th priority job including service interruption time
equals $\int\limits_{0}^{\infty}(1-B_{j}(mx))e^{-\Lambda_{j}x}dx,$ or
$\left(1-\beta_{i}\left(\frac{\Lambda_{j}}{m}\right)\right)/\Lambda_{j}.$
Hence,

$d_{i}=\left(1-\beta_{j}\left(\frac{\Lambda_{j}}{m}\right)\right)\left(\frac{1}{\Lambda_{j}}+g_{j}\right),\
j=1,\dots,N.$ $None$

Using (1), (2), we get

$g_{i}=\frac{1}{1-\frac{\lambda_{i}}{\Lambda_{i}}\left(1-\beta_{i}\left(\frac{\Lambda_{i}}{m}\right)\right)}\left(\frac{1}{\Lambda_{i}}\sum\limits_{j=1}^{i-1}\lambda_{j}d_{j}+\frac{\lambda_{i}}{\Lambda_{i}^{2}}\left(1-\beta_{i}\left(\frac{\Lambda_{i}}{m}\right)\right)\right),\
i=1,\dots,N.$ $None$

Using recurrent procedure based on formulas (2), (3), we can compute the value
$g_{i}.$

If the time intervals during that there $m$ jobs in the system $S_{i},$ then
this system is equivalent to the usual queueng system with arrival load equal
to $R_{i}.$ Therefore, $p_{ki}=\frac{R_{i}^{k}}{k!}p_{0i},$ $k=0,1,\dots,m-1.$
Let $\nu_{i}(t),$ $\nu_{i}^{\prime}(t)$ be the number of jobs at time $t$ in
the systems $S_{i}$ and $S_{i}^{\prime}$ respectively. If the intervals are
excluded such that, in the system $S_{i},$ there are less than $m-1$ jobs,
then $\nu_{i}(t)-m+1$ and $\nu_{i}^{\prime}(t)$ are equivalent stochastic
processes. Therefore
$\frac{p_{mi}}{p_{m-1,i}}=\frac{p_{1i}^{\prime}}{p_{0i}},$ and hence,
$p_{mi}=\frac{R_{i}^{m-1}\Lambda_{i}g_{i}p_{i0}}{(m-1)!}.$

From the normalizing condition $\sum\limits_{k=0}^{m}p_{ki}=1,$ we get

$p_{ik}=\frac{R_{i}^{k}}{k!\left(\sum\limits_{k=0}^{m-1}\frac{R_{i}^{k}}{k!}+\frac{R_{i}^{m-1}\Lambda_{i}g_{i}}{(m-1)!}\right)},\
k=0,1,\dots,m-1,$
$p_{mi}=\frac{R_{i}^{m-1}\Lambda_{i}g_{i}}{(m-1)!\sum\limits_{k=0}^{m-1}\frac{R_{i}^{k}}{k!}+R_{i}^{m-1}\Lambda_{i}g_{i}}.$
$None$

Denote by $q_{i}$ the probability that a job is lost at its arrival moment for
the system $S.$ Assume that $p_{m,i-1}$ is the approximate value of this
probability. Since the jobs arrive according to a Poisson process, the value
$q_{i}$ is also equal to the probability that, in the system $S_{i-1},$ there
are $m$ jobs of the priorities no lower than $i-1.$ Denote by $c_{i}$ the
value $p_{mi}.$ The equality $q_{i}=c_{i-1}$ is exact in the following cases.
1\. If $m=1,$ then the system $S_{i}$ is equivalent to the system $S$ if it is
assumed that only jobs of priority not lower than $i,$ and therefore the
equality $q_{i}=c_{i}$ holds. 2\. Let the service time is distributed
exponentially with parameter $\mu$ not depending on the priority of job. The
presence of lower priorities in the system $S$ does not affect the servicing
of higher priority jobs. Since the service time is distributed exponentially,
then remaining service time is also distributed exponentially with the same
parameter. Thus the probability that, in the system $S,$ there are no jobs of
the priorities lower than $i$ is the same as the probability that, in related
non-priority system with arriving load $R_{i}$ a job is lost, i.e., according
to the first Erlang formula

$q_{i}=\frac{R_{i}^{m}}{m!\sum\limits_{k=0}^{m}\frac{R_{i}^{k}}{k!}}.$

In this case, $g_{i}=\frac{1}{m\mu}=\frac{R_{i}^{m}}{\Lambda_{i}m},$ then (4)
may be rewritten as

$p_{mi}=\frac{R_{i}^{m}}{m!\sum\limits_{k=0}^{m}\frac{R_{i}^{k}}{k!}}.$

Thus, $q_{i}=c_{i-1}.$ 3\. The equality $q_{i}=c_{i-1}$ is also holds if, with
a probability depending on the priority of job, the service time is equal to
0, and, with the additional priority, the service time is distributed
exponentially with parameter independent of the priority.

##  4 Evaluation of probability that accepted job will be preempted

Denote by $r_{i}$ the probability that a prescribed priority job accepted to
service will be preempted.

Let us prove the following equality

$r_{i}=\frac{\Lambda_{i}(q_{i}-q_{i-1})}{\lambda_{i}(1-q_{i})}.$ $None$

An arriving job of the priority $i$ or higher preempts a job of the $i$th
priority if there are $m$ jobs in the system, and at least one of these jobs
is a job of the $j$th priority. Therefore, with the probability
$q_{i}-q_{i-1},$ the system is in the state such that an arriving job of the
priority not lower than $i$ preempts a job of the $i$th priority. Hence the
average number of interruptions of servicing the $i$th priority jobs per a
time unit equals $\Lambda(q_{i}-q_{i-1})r_{i}.$ On the other hand, this
average number equals $\lambda(q_{i}-q_{i-1}).$ Thus we have (5).

Replacing $q_{i}$ with $c_{i}$ in (5), we get the following approximate
formula

$r_{i}=\frac{\Lambda_{i}(c_{i}-c_{i-1})}{\lambda_{i}(1-c_{i})},\ i=1,\dots,N.$
$None$

##  5 Evaluation of loss probability

We have the following equality

$\gamma_{i}=q_{i}+(1-q_{i})r_{i}.$ $None$

Replacing $q_{i}$ with $c_{i-1}$ in (7) and using (6), we get

$\gamma_{i}=c_{i-1}+\frac{\Lambda_{i}}{\lambda_{i}}(c_{i}-c_{i-1}).$ $None$

Thus we propose to compute the loss probability for a job of a prescribed
priority according to formula (8), where

$c_{i}=\frac{R_{i}^{m-1}\Lambda_{i}g_{i}}{(m-1)!\sum\limits_{k=0}^{m-1}\frac{R_{i}^{k}}{k!}+R_{i}^{m-1}\Lambda_{i}g_{i}},$
$\Lambda_{i}=\sum\limits_{j=1}^{i}\lambda_{j},\
R_{i}=\sum\limits_{j=1}^{i}\lambda_{j}b_{j},$
$g_{i}=\frac{1}{1-\frac{\lambda_{i}}{\Lambda_{i}}\left(1-\beta_{i}\left(\frac{\Lambda_{i}}{m}\right)\right)}\left(\frac{1}{\Lambda_{i}}\sum\limits_{j=1}^{i-1}\lambda_{j}d_{j}+\frac{\lambda_{i}}{\Lambda_{i}^{2}}\left(1-\beta_{i}\left(\frac{\Lambda_{i}}{m}\right)\right)\right),$
$d_{i}=\left(1-\beta_{j}\left(\frac{\Lambda_{j}}{m}\right)\right)\left(\frac{1}{\Lambda_{j}}+g_{j}\right),\
j=1,\dots,N.$

Remark 1. Note that, for the related multichannel loss system with LCFD
preemptied priority discipline an approximate approach is proposed in [12]
such that, according to this approach, the loss probability $\gamma_{i}$ for a
job of the $i$th priority is computed as

$\gamma_{i}=c_{i}+\frac{\Lambda_{i-1}}{\lambda_{i}}(c_{i}-c_{i-1}),\
m=1,\dots,N,$
$c_{i}=\frac{R_{i}^{m-1}\Lambda_{i}g_{i}}{(m-1)!\sum\limits_{k=0}^{m-1}\frac{R_{i}^{k}}{k!}+R_{i}^{m-1}\Lambda_{i}g_{i}},$
$\Lambda_{i}=\sum\limits_{j=1}^{i}\lambda_{j},\
R_{i}=\sum\limits_{j=1}^{i}\lambda_{j}b_{j},\
g_{i}=\frac{1}{\Lambda_{i}}\sum\limits_{j=1}^{i}\lambda_{j}d_{j},$
$d_{j}=\left(1-\beta_{j}\left(\frac{\Lambda_{j-1}}{m}\right)\right),\
j=2,3,\dots,N,$ $d_{1}=\frac{b_{1}}{m}.$

##  6 Estimating accuracy of approach

If the job service time is exponential with parameter that may depend on
priority class, then the values of the loss probability for a job of
prescribed priority is the same for the FCFD preemptive discipline and the
LCFD preemptive discipline. Assume that the job service time priority is
exponential with average value depending on priority-class:

$B_{i}(x)=1-e^{-\mu_{i}x},\ i=1,2,3,$

$N=3,$ $m=2,$ $\mu_{1}=10,$ $\mu_{2}=5,$ $\mu_{3}=2,$
$\lambda_{1}=\lambda_{2}=\lambda_{3}=1.$ Then the value of the loss
probabilities computed according to the proposed approach are

$\gamma_{1}=0.0045,\ \gamma_{2}=0.060,\ q_{3}=0.36.$

The values obtained by simulation are [12]

$\gamma_{1}=0.0045,\ \gamma_{2}=0.064,\ \gamma_{3}=0.32.$

##  7 Conclusion

Approximate approach is proposed to evaluate the following characteristics of
a multichannel loss preemptive priority system: the loss probability of a
prescribed probability job; the loss probability of a prescribed probability
job at the the arrival moment; the probability that a job of prescribed
priority is lost due to preemption. In particular cases the formulas are
exact. The accuracy of the approximate approach was estimated by simulation.

## References

  * [1] Jaiswal N.K. Priority queues. Vol. 50 of Mathematics and Science Engineering. Academic Press, New York and London, 1968.
  * [2] Brosh I. Preemptive priority assignment in multichannel systems. Operations Research, 1969, vol. 17, pp. 526–535.
  * [3] M. Segal, A multiserver system with preemptive priorities, Opns. Res., 18 (1970), 316–323,  DOI: 10.1287/opre.18.2.316
  * [4] Tatashev A.G. Calculation of the distribution of the waiting time in a multiple-channel queueing system with fixed priorities, Engineering Cybernetics, 22 (1984), 59–62. (Originally published in Tekhnicheskaya Kibernetika), 6 (1983), 163–166.
  * [5] Zeltyn S., Feldman Z., Wasserkrug S. Waiting and sojourn times in a multi-server queue with mixed priorities, Queueing Systems, 61 (2009), 305–328,  DOI:10.1007/s11134-009-9110-4
  * [6] Takagi H. Unified and refined analysis of the response time and waiting time in the M/M/m FCFC preemptive-resume priority queue. Journal of Industrial and Management Optimization, vol. 13, no. 4, 2017, pp. 1945–1973.  DOI:10.3934/jimo.2017026
  * [7] Takagi H. Analysis of the response and waiting times in the M/M/m LCFC preemptive-resume priority queue. International Journal of Pure and Applied Mathematics, vol. 109, no. 2, 2016, pp. 325–370. DOI: 10.12732/ijpam.v109i2.12
  * [8] Buzen J.P., Bondi A.B. The response times of priority classes under preemptive resume in M/M/m queues, Operations Research, 31 (1983), 456–465.
  * [9] Maslova E.A., Tatashev A.G. Approximate method for preemptive priority service in a multichannel system. Automatic Control and Computer Science, 1992, vol. 26, no 1, pp. 10–15. Translated from Avtomatika i Vychislitelnaya Tekhnika, 1992, no. 1, pp. 13–18.
  * [10] Tatashev A. G., Seleznjev O. V., Yashina M. V. Approximate formulas for characteristics of multichannel LIFO preemptive-resume priority queueing system. arXiv:2206.09263 [math.PR]– 2022.
  * [11] Alencar M.S., Tatashev A.G., Yashina M.V.. Stochastic estimation of on-time message delivery in a packet-switched network with messages of different priorities. 2020 International Conference on Engineering Management of Communication and Technology (EMCTECH), 2020, pp. 1-4,
DOI: 10.1109/EMCTECH49634.2020.9261511.

  * [12] Maslova E.A., Tatashev A.G. Approximate calculation of priority service characteristics in a multiserver system with losses. Automatic Control and Computer Science, 1992, vol. 26, no 4, pp. 31–35.
  * [13] Katchner L. Loss system with displacing priorities. Sixth Int. Teletraffic Cong., pp. 224/1-224-8, Munich 1970.


capbtabboxtable[][]

# Eliciting Compatible Demonstrations for Multi-Human Imitation Learning

Kanishk Gandhi, Siddharth Karamcheti, Madeline Liao, Dorsa Sadigh
Department of Computer Science, Stanford University
{kanishk.gandhi, skaramcheti, madelineliao<EMAIL_ADDRESS>
###### Abstract

Imitation learning from human-provided demonstrations is a strong approach for
learning policies for robot manipulation. While the ideal dataset for
imitation learning is homogenous and low-variance – reflecting a single,
optimal method for performing a task – natural human behavior has a great deal
of heterogeneity, with several optimal ways to demonstrate a task. This
multimodality is inconsequential to human users, with task variations
manifesting as subconscious choices; for example, reaching down, then across
to grasp an object, versus reaching across, then down. Yet, this mismatch
presents a problem for interactive imitation learning, where sequences of
users improve on a policy by iteratively collecting new, possibly conflicting
demonstrations. To combat this problem of demonstrator incompatibility, this
work designs an approach for 1) measuring the compatibility of a new
demonstration given a base policy, and 2) actively eliciting more compatible
demonstrations from new users. Across two simulation tasks requiring long-
horizon, dexterous manipulation and a real-world “food plating” task with a
Franka Emika Panda arm, we show that we can both identify incompatible
demonstrations via post-hoc filtering, and apply our compatibility measure to
actively elicit compatible demonstrations from new users, leading to improved
task success rates across simulated and real environments.111Additional videos
& results: https://sites.google.com/view/eliciting-demos-corl22/home

> Keywords: Interactive Imitation Learning, Active Demonstration Elicitation,
> Human-Robot Interaction

##  1 Introduction

Interactive imitation learning [1, 2, 3] from a pool of human demonstrators is
a scalable approach for learning multi-task policies for robotic manipulation
[4, 5, 6]. Yet, such approaches have a critical problem, especially in the
low-to-moderate data regime: data from multiple human demonstrators often have
conflicting modes, where two users provide opposing behaviors for a single
task – behaviors that manifest as subconscious, random choices. For example,
consider the nut-on-peg task in Fig. 1: one user (in orange) approaches the
nut by moving across the table, then down, while the other user (blue) reaches
down, then across.

Training on aggregated batches of data in series – starting with a base
policy, adding small amounts of data from new users, and retraining the policy
after each batch – is common in interactive imitation formulations [2, 3];
unfortunately, when we add a small number of conflicting demonstrations during
the interaction phase, the retrained policy attempts to cover both the base
demonstrations and the new set. This leads to incongruent overfitting, where a
policy – even one equipped to learn multimodal behaviors [7, 8, 9] – tries to
fit the base set for most of a trajectory, but overfits to the new set for a
small subset of the state space, often with catastrophic failure modes.

Figure 1: Visualization of two users (orange and blue) inserting a nut onto a
round peg. Both demonstrators approach the round nut, pick it up and insert it
into the peg in very different ways, introducing heterogenous data that is
incompatible with the base policy, hurting performance when retrained.

To mitigate this problem, this work tackles two questions: 1) how can we
measure the compatibility between a new demonstrator and an existing policy,
and 2) how can we use this measure to actively elicit better demonstrations
from a new user?

While our approach for measuring and eliciting compatible demonstrations
during online collection is novel, prior work has studied the impact of
suboptimal demonstrations on learning. Most relevant, Mandlekar et al. [10]
introduce RoboMimic, a suite of simulated manipulation tasks that consist of
demonstrations collected from 6 humans of varying qualities (Worse, Okay, and
Better). This work and recent followups [11] show how policies degrade in the
presence of heterogenous data with multimodal behaviors, finding that training
imitation learning policies on data from a single demonstrator can often
exceed the performance of training on equivalent amounts of data from multiple
demonstrators. A related body of work on offline policy learning propose
approaches for overcoming suboptimality by using extra annotations, usually in
the form of rewards or returns for each demonstration. These methods seek to
reweight [12, 13, 14], or directly filter out demonstrations [15, 16, 17].
Yet, these are all post-hoc approaches that operate after demonstrations have
been collected. None of these prior approaches target the root cause:
informing users of the compatibility of their demonstrations, as they collect
data in the first place.

In contrast, our proposed approach targets this root cause by first _learning_
a fine-grained measure of demonstrator compatibility subject to an initial
base policy, and then _operationalizing_ this measure to inform users while
they collect demonstrations. Our compatibility measure is derived by sorting
individual state-action pairs in the data based on their likelihood and
novelty under the base policy. By plotting each state-action along a 2D “map”
with these two properties as the axes, we are able to learn visual boundaries
denoting “aligned and compatible” demonstrations (those with high likelihood
and low novelty) to those that are grossly incompatible (low likelihood and
low novelty) – an approach inspired by prior work in dataset interpretability
[18, 19]. From these maps, we identify a scoring function and set of
thresholds to define our demonstrator compatibility measure, which we then
cheaply and efficiently deploy during active data collection. Crucially, with
this fine-grained measure, users with “incompatible” demonstrations are given
rich feedback as to where in their demonstration the incompatibility arose
coupled with visualizations of compatible demonstrations from the base
policy’s training set, guiding their behavior moving forward.

Our results across two simulated tasks from prior work [10, 20] and a real-
world “food-plating” task with a Franka Emika Panda arm show that our
compatibility measure enables the elicitation of more compatible
demonstrations – demonstrations that, when added to the base dataset, lead to
a boost of up to 25% in success rate relative to a 20% decrease in performance
when collecting data naively, without feedback.

##  2 Related Work

Identifying & Learning from Diverse Demonstrations. Our definition of
diversity subsumes both heterogeneity – where demonstrators capture multiple
distinct, but equally optimal behaviors – and suboptimality – where users
provide demonstrations that are poor relative to some objective reward
function. On examining the effect of learning from these diverse
demonstrations, Mandlekar et al. [10] found that imitation learning models
trained on demonstrations from a single “proficient” human achieved higher
success rates than models trained on significantly larger dataset sourced from
a mixture of human demonstrators with varying optimality. Gopalan et al. [21]
further show that without prompting users with videos or specific subtasks,
only a third of demonstrators provide compatible demonstrations. These works
motivate our approach by showing that in naive demonstration collection,
heterogeneity and suboptimality are prevalent; instead, we need a direct way
to intervene during collection, actively inform users, and guide them to
provide compatible demonstrations.

Other work studies learning from diverse demonstrations. Several approaches
[7, 8, 9, 22, 23] model the multimodality of diverse demonstrations directly,
usually by explicitly handling each mode of the data. Unfortunately, these
approaches fail in the sequential, interactive imitation learning setting that
we study, where we gradually introduce new modes and retrain after each
collection. A separate line of work tackles learning with suboptimal data
through the use of offline reinforcement learning [12, 13, 14], filtered
imitation [11, 15, 16, 17], or inverse reinforcement learning [24, 25, 26].
These methods use extra reward annotations for each demonstration, and
generally try to reweight or selectively remove suboptimal data. Unlike these
approaches, this work tackles the root cause of the problem – how to collect
compatible demonstrations from humans in the first place, removing the need to
filter out 10s – 100s of painstakingly collected demonstrations.

Interactive Imitation Learning. Most of the work in interactive imitation
learning attempts to address the problem of covariate shift endemic to
policies trained via behavioral cloning. Ross et al. [1] introduce DAgger, a
method for iteratively collecting demonstrations by relabeling the states
visited by a policy interactively, teaching the policy to recover. Later work
builds on DAgger, primarily focusing on reducing the number of expert labels
requested during training. These variants identify different measures such as
safety and uncertainty to use to query the expert limiting the amount of
supervision a user needs to provide [2, 20, 27, 28, 29, 30]. The focus of
these works is on learning from a single user rather than from multiple users
demonstrating heterogenous behaviors.

Our proposed approach presents a novel interactive imitation learning method
that learns to actively elicit demonstrations from a pool of multiple users
during the course of data collection, with the express goal of guiding users
towards compatible, optimal demonstrations.

##  3 Problem Setting

We consider sequential decision making tasks, modeled as a Markov Decision
Process (MDP) with the following components: the state space $\mathcal{S}$,
action space $\mathcal{A}$, transition function $\mathcal{T}$, reward function
$R$, and discount factor $\gamma$. In this work, we assume sparse rewards only
provided on task completion. A state
$s=(o_{\text{grounded}},o_{\text{proprio}})\in\mathcal{S}$ is comprised of a
grounded observation (either coordinates/poses of objects in the environment,
or an RGB visual observation) and the robot’s proprioceptive state. An action
$a\in\mathcal{A}$ is a continuous vector of robot joint actions. We assume
access to a small dataset of trajectories
$\mathcal{D}_{\text{base}}=\\{\tau_{1},\tau_{2}\ldots\tau_{N}\\}$ where each
trajectory $\tau_{i}$ is a sequence of states and actions,
$\tau_{i}=\\{(s_{1},a_{1})\ldots(s_{T},a_{T})\\}$. We train an initial policy
$\pi_{\text{base}}$ on this dataset via behavioral cloning.

Our approach has two components: 1) developing a measure of the compatibility
of a new demonstration with an existing base policy, and 2) building a method
for actively eliciting compatible demonstrations from users. For the first
component, we learn a measure
$\mathcal{M}(\mathcal{D}_{\text{base}},(s_{\text{new}},a_{\text{new}}))\in[0,1]$
that defines a compatibility at the granuality of a state-action pair. For the
second component, we use our fine-grained measure $\mathcal{M}$ to provide
rich feedback to users about their demonstrations, in addition to deciding
whether to accept/reject a new demonstration $\tau_{\text{new}}$. The set of
new demonstrations the user collects comprises $\mathcal{D}_{\text{new}}$ –
after each collection step, we train a new policy $\pi_{\text{new}}$ on
$\mathcal{D}_{\text{base}}\cup\mathcal{D}_{\text{new}}$.

Under our definition of compatibility (and the measure $\mathcal{M}$ we
derive), we hope to guide users to provide new datasets
$\mathcal{D}_{\text{new}}$ such that $\pi_{\text{new}}$ will have as high of
an expected return as possible, and minimally, a higher expected return than
$\pi_{\text{base}}$. In other words, our definition of compatibility asks that
new data should only help, not hurt performance relative to the initial set.

##  4 Learning to Measure Compatibility in Multi-Human Demonstrations

Figure 2: Plots, or “maps” of the likelihood and novelty values for
demonstrations recorded by the blue compatible operator (a), and the orange
incompatible operator with respect to the base policy $\pi_{\text{base}}$. The
gray border indicates a filter with our chosen thresholds ($\lambda$, $\eta$).
The orange operator performs far more low-likelihood actions in familiar
states, earning a lower compatibility score.

We first derive a general compatibility measure $\mathcal{M}$ given a base set
of demonstrations, then evaluate our measure through a series of case studies
grounded in real, user-provided demonstration data.

###  4.1 Estimating Compatibility and Identifying Good Demonstrations

An idealized compatibility measure $\mathcal{M}$ has one role – estimating the
performance of a policy $\pi_{\text{base}}$ that is retrained on the union of
a known base dataset $\mathcal{D}_{\text{base}}$ and a new dataset
$\mathcal{D}_{\text{new}}$. Crucially, $\mathcal{M}$ needs to operate at a
granular level (ideally at the level of individual state-action pairs),
without incurring the cost of retraining and evaluating $\pi_{\text{base}}$ on
the new dataset. Phrased this way, there is a clear connection to pool-based
active learning [31], informing a choice of a plausible metrics that could
help predict downstream success. While many metrics could work, we choose two
easy-to-compute metrics that lend themselves well to interpretability: the
likelihood of actions $a_{\text{new}}$ in $\mathcal{D}_{\text{new}}$ under
$\pi_{\text{base}}$, measured by the negative mean squared error between
actions predicted by $\pi_{\text{base}}(s_{\text{new}})$ and $a_{\text{new}}$,
as well as the novelty of a given state, measured by the standard deviation in
the predicted actions under the base policy. To turn these two metrics into a
single measure, we borrow from interpretability and active learning literature
[18, 19]; we plot, or “map” on a 2D plane the likelihood and novelty values
for each trajectory in our dataset, with the goal of identifying two
thresholds $(\lambda,\eta)$ on likelihood and novelty – thresholds that allow
us to identify a given trajectory based on its compatibility.

The final piece is setting the thresholds for novelty ($\eta$) and likelihood
($\lambda$). While this can be done in many ways – often just by looking at
the “mapped” dataset on the 2D plane [18] – in this work, we assume the
ability to obtain or collect a handful of a priori incompatible demonstrations
that we can use to build contrast sets to regress a threshold. We define
compatibility
$\mathcal{M}(\mathcal{D}_{\text{base}},(s_{\text{new}},a_{\text{new}}))\in[0,1]$
as parameterized by thresholds $(\lambda,\eta)$:

$\mathcal{M}=\begin{cases}1-\text{min}\left(\frac{(\pi_{\text{base}}(s_{\text{new}})-a_{\text{new}})^{2}}{\lambda},1\right)&\text{if
novelty$(s_{\text{new}})<\eta$}\\\ 1&\text{otherwise}.\end{cases}$

A state-action pair is compatible
($\mathcal{M}(\mathcal{D}_{\text{base}},(s_{\text{new}},a_{\text{new}}))=1$)
if the likelihood of $a_{\text{new}}$ under the base policy is high, or the
novelty of $s_{\text{new}}$ is $>\eta$; incompatibility is defined smoothly on
the inverse interval with a state-action pair being completely incompatible if
($\mathcal{M}(\mathcal{D}_{\text{base}},(s_{\text{new}},a_{\text{new}}))=0$)
when $(\pi_{\text{base}}(s_{\text{new}})-a_{\text{new}})^{2}\geq\lambda$.
Intuitively, this measure mirrors what one expects when collecting new data –
in states where the policy is confident about what actions to take, reject
other actions, and in new states where the policy is uncertain, more, diverse
data is good. The following subsection validates this measure against
demonstration data collected by real users on a series of tasks, showing how
we can use $\mathcal{M}$ as a filter to boost policy performance in the
presence of a large dataset of heterogenous demonstrations.

###  4.2 Case Studies and Experiments

To validate our measure $\mathcal{M}$, we consider three tasks in simulation,
each with a set of demonstrations collected by multiple operators. For each
task, we choose one operator to initialize the base set of demos
$\mathcal{D}_{\text{base}}$ and use trajectories from another operator to form
$\mathcal{D}_{\text{new}}$. For a more detailed summary of the tasks datasets,
and policy training procedure, please consult the Supplementary Material.

Square Nut [10]: The goal of the task is to place a square nut onto a square
peg. We use the environment and demonstrations collected by Mandlekar et al.
[10], consisting of 200 demonstrations from a proficient operator, and 50
demonstrations each from 6 operators of varying qualities. We initialize
$\mathcal{D}_{\text{base}}$ with 50 demonstrations from the proficient
operator, and use demonstrations from each of the 4 more efficient operators
as the different $\mathcal{D}_{\text{new}}$.

Round Nut [10, 20]: Similar to the prior task, the goal is to place a round
nut onto a round peg (as in Fig. 1). For this task, we collected 30
demonstrations from a proficient operator and 30 demonstrations each from 3
other operators – where the data provided for one operator is taken directly
from Hoque et al. [20]. We initialize $\mathcal{D}_{\text{base}}$ using the
proficient operator’s demonstrations, and we randomly sample 10 demonstrations
from each operator to get different $\mathcal{D}_{\text{new}}$.

Hammer Placement [32]: Here the goal is to open a drawer and place a hammer
inside (see Fig. 3). We collected 25 demonstrations from a proficient operator
to initialize the $\mathcal{D}_{\text{base}}$ and 5 demonstrations from each
of 4 other operators to get different $\mathcal{D}_{\text{new}}$.

Policy Architecture, Training, and Evaluation. We train an ensemble of 5 MLP
policies using coordinate-based state observations and the robot’s
proprioceptive state to predict 7-DoF joint actions. Models are trained for
1000 epochs, and evaluated every 200 epochs; we select our policy by taking
the checkpoint with the highest success rate averaged over 50 rollouts.

Case Study: Naïve Training. To establish baseline performance, we compare the
performance of policies trained on different $\mathcal{D}_{\text{new}}$ given
by different operators to identify meaningful markers. The results in Table 1
show a large difference in the success rate of $\pi_{\text{new}}$ depending on
which operator provided $\mathcal{D}_{\text{new}}$. For example, on Square
Nut, sampling $\mathcal{D}_{\text{new}}$ from Operator 1 (the same operator
who provided $\mathcal{D}_{\text{base}}$) leads to an improvement of 15.6%
over the base policy, while sampling from Operator 4 leads to a reduction of
11.4%. Similarly, Operator 1 (again, the operator who provided
$\mathcal{D}_{\text{base}}$) in Round Nut (the blue line, Fig. 1) almost
doubles the performance of the base policy while Operator 4 (orange line in
Fig. 1) halves it. In general, training on data with incompatible
demonstrations can degrade policy performance by seemingly arbitrary amounts.

Experiment: Compatibility-Driven Filtering. Finally, we evaluate the ability
of our measure $\mathcal{M}$ to identify compatible demonstrations from fixed
pools of demonstration data. We first note the result of “mapping” the various
demonstrators to provide some additional visual intuition, grounded in the
case study results; Fig. 2 shows the 2D “maps” for Operator 1 (orange) and
Operator 4 (blue). These plots (see the Supplemental Material for plots of all
operators), present a key insight: incompatible operators provide more
unlikely actions for familiar, or less-novel, states. Closing the loop and
using our measure $\mathcal{M}$ to filter out incompatible demonstrations, we
see strong results: Table 1 shows that removing incompatible samples leads to
performance that is at least as good, if not much better than the base policy.
For instance, we see an improvement of 6.7% for Operator 1 (the most
compatible), on the Square Nut task and of 10% for Operator 4 (the least
compatible).

Operator | Square Nut | Round Nut | Hammer Placement
---|---|---|---
Naive |  $\mathcal{M}$-Filtered | Naive |  $\mathcal{M}$-Filtered | Naive |  $\mathcal{M}$-Filtered
Base Operator | 38.7 (2.1) | - | 13.3 (2.3) | - | 24.7 (6.1) | -
Operator 1 | 54.3 (1.5) | 61.0 (4.4) | 26.7 (11.7) | 32.0 (12.2) | 38.0 (2.0) | 39.7 (4.6)
Operator 2 | 40.3 (5.1) | 42.0 (2.0) | 22.0 (7.2) | 26.7 (5.0) | 33.3 (3.1) | 32.7 (6.4)
Operator 3 | 37.3 (2.1) | 42.7 (0.6) | 17.3 (4.6) | 18.0 (13.9) | 8.0 (0.0) | 12.0 (0.0)
Operator 4 | 27.3 (3.5) | 37.3 (2.1) | 7.3 (4.6) | 13.3 (1.2) | 4.0 (0.0) | 4.0 (0.0)
Table 1: Success rates (mean/std across 3 training runs) for policies trained
on $\mathcal{D}_{\text{new}}$ from different operators. $\mathcal{M}$-Filtered
denotes filtering by compatibility scores.

##  5 Actively Eliciting Compatible Demonstrations

Figure 3: The three phases of our active elicitation interface spanning the
initial prompting phase (a), the subsequent demonstration phase with live
feedback (b), and finally, the feedback phase (c).

Our pipeline starts traditionally: operators are provided a task specification
(e.g., “place the square nut on the square peg”) and are given three episodes
of practice with the robot. We then enter our active elicitation process (see
Fig. 3 for a full overview of our interface). First, is prompting; each
operator is shown 5 rollouts from $\mathcal{D}_{\text{base}}$ to understand
the style of the base operator. Second is demonstration, where the operator is
tasked with providing a demonstration similar to that which they were just
shown. Critically, as they record their demonstration, we use our measure
$\mathcal{M}$ to provide online feedback – a green indicator for compatible
actions, and a red indicator for incompatible actions.

After demonstrating, begins the feedback phase; if the number of incompatible
state-action pairs – $(s_{\text{new}},a_{\text{new}})$ pairs with
$\mathcal{M}(\mathcal{D}_{\text{base}},(s_{\text{new}},a_{\text{new}}))=0$ –
exceeds 5% of the accepted demonstration length, then the demonstration is
rejected. Rather than a binary decision, we take the opportunity to provide
rich feedback: using a simple sliding window, we identify 3 candidates with
the highest average incompatibility from the rejected demonstration. We show
this to the user, along with a demonstration from $\mathcal{D}_{\text{base}}$
retrieved based on state-similarity to the 3 candidates. With this contrastive
feedback, we find that users are able to more effectively pinpoint where in
the demonstration they went wrong, and learn how to provide more compatible
demonstrations moving forward. We continue this process until a desired number
of demonstrations have been recorded. Finally, $\pi_{\text{base}}$ is updated
to $\pi_{\text{new}}$ every time a batch of demonstrations is collected from a
demonstrator.

###  5.1 User Studies – Simulation

We build a browser interface for users to provide teleoperated demonstrations
for two tasks in simulation: Round Nut and Hammer Placement. We conduct user
studies in two conditions, naive and informed. Under the naive condition,
users get to practice the controls during 3 practice episodes, before
collection demonstrations without any additional prompting. Under the informed
condition, users are provided the active elicitation interface above.

Round Nut. $\mathcal{D}_{\text{base}}$ consists of 30 trajectories collected
by a proficient operator. The users are randomly assigned to one of the two
conditions, and are asked to collect 10 successful demonstrations
$\mathcal{D}_{\text{new}}$. We conduct user studies for this task with $n=16$
participants.

Hammer Placement. $\mathcal{D}_{\text{base}}$ consists of 25 trajectories
collected by a proficient operator. For this task, we perform a longitudinal
study with $n=4$ participants, where users are first asked to collect 5
demonstrations in the naive condition and then collect 5 demonstrations in the
informed condition. This allows us to measure the effect of the interface in
eliciting demonstrations within subjects.

Results. For both tasks, we find that policies trained on demonstrations
collected via active elicitation consistently outperform those from the naive
condition. Our studies indicate that naive participants find it difficult to
provide compatible demonstrations, worsening the base policy on average. In
contrast, informed demonstrators improve the base policy, beating the naive
condition by 6.1% on Round Nut and 11% on Hammer Placement. From our
longitudinal studies on the latter, we find that informed demonstrations beat
naive demonstrations from the same user for 3/4 operators, clearly indicating
that active elicitation is a strong approach for interactive imitation. We
also find that the length of trajectories recorded by informed demonstrators
is closer to those of the base demonstrator (see Supplementary). To test if
the high frequency feedback from our interface discouraged users from
providing corrections, we also perform experiments with collecting
interactive, human-gated (Hg-DAgger [2]) demonstrations. Informed data
collection outperforms naive data collection for 3 out of 4 users even in this
setting and informed users intervene at about the same rate as informed ones
(see supplementary for details).

###  5.2 User Studies – Real Robot

Food Plating. As depicted in Fig. 4, the task requires the Franka Emika Panda
arm to lift the pink pan from a “stove” and guide it over a plate, where it is
responsible for transferring a fried egg from the pan to the plate without
dropping it on the table. The location of the plate varies across episodes,
and the robot has to learn to move and tilt the pan in a way that the egg
gently slides onto the plate. $\mathcal{D}_{\text{base}}$ consists of 15
demonstrations collected by a single, proficient operator that maintains a
consistent strategy of lifting the pan and tilting it forward (“vertically
plating”) to transfer the egg.

Policy. We train an ensemble of 5 policies that predict 7-DoF joint actions
from visual observations (RGB images) and proprioceptive states. We use a
ResNet-34 backbone pretrained on ImageNet [33] to encode the visual
observations, followed by a single MLP to predict actions from the
concatenated ResNet embedding and proprioceptive state (see the Supplementary
Material for additional details). We select our final model based on the best
validation loss (MSE) after 20 epochs of training.

Experimental Setup. We build an analog of our browser interface for active
elicitation in a console UI. Unlike simulation, users provide demonstrations
kinesthetically by physically moving the robot arm. Before collecting
demonstrations, users are time to explore the controls of the robot and
practice moving it. We conduct a within-subjects study with $n=4$ users who
first perform the naive collection of 5 demonstrations, then collect 5
demonstrations using the active elicitation interface.

Results. Training on demonstrations collected with active elicitation
outperform the base policy by 25% and the naive condition by 55%.
Additionally, similar to simulation, we see that naive demonstration
collection often hurts policy performance.

Task | Base | Naïve | Naïve + Filtered | Informed
---|---|---|---|---
Round Nut | 13.3 (2.3) | 9.6 (4.6) | 9.7 (4.2) | 15.7 (6.0)
Hammer Placement | 24.7 (6.1) | 20.8 (15.7) | 22.0 (15.5) | 31.8 (16.3)
[Real] Food Plating | 60.0 | 30.0 (17.3) | - | 85.0 (9.6)
Table 2: Success rates (mean/std across users) for the user studies evaluating
both naive and informed demonstration collection against base users.

##  6 Discussion

Driving this work has been a simple story; in § 4 we performed a thorough
analysis and set of experiments on pre-existing datasets to show that
heterogeneity is a very real problem in multi-human imitation learning
datasets, one that can harm policy performance in arbitrary ways. Through this
section, building up to § 4.1 we were able to profile these existing datasets,
and regress a straightforward compatibility measure $\mathcal{M}$ that hinged
on two attainable features of the base policy – the likelihood and novelty of
a new state-action pair under the existing model parameters. Given these case
studies and our proposed measure, we turned to active elicitation in § 5,
building an interface to guide users towards producing more compatible
demonstrations, as depicted in Fig. 3.

On our simulated tasks (Table 2), we showed that an uninformed collection
procedure – no prompting, no fine-grained feedback – leads to unequivocally
worse performance than the base policy downstream, due to the introduced
heterogeneous data. However, with our active elicitation and prompting
behavior, we see the opposite – more data leads to significant improvements
over the base policy, by a broad margin. This story is only further confirmed
with our experiments on a real robot, as we had users provide demonstrations
kinesthetically, showing more extreme results. On the seemingly simple “food
plating” task, uninformed demonstration collection led to a broad spectrum of
behaviors (e.g., the sideways plating depicted in Fig. 4 vs. the idealized
vertical plating of the base policy) driving the success rate down almost 30%
relative to the base policy. In contrast, the same users – with just 1-2
episodes of feedback – were able to learn to provide compatible
demonstrations, with their “informed” demonstrations leading to an absolute
policy success rate of 85%. Not only is the active elicitation procedure
highly effective, but it is fundamentally cheap, generalizable across various
control schemes (e.g., the discrete keyboard control of simulation vs. the
kinesthetic teaching in the real world), and is broadly applicable across
various types of manipulation tasks.

Limitations. We make several assumptions in this work that may impede future
applications. Our biggest assumption is that the preferences reflected in the
initial dataset are the “ground-truth” to aspire to. In settings where users
have specific, conflicting preferences about how to accomplish a task, our
approach falls short. Consider the age-old question of when to pour milk into
a cereal bowl – namely, before or after the cereal has made it to the bowl.
Under our approach, given a base dataset of cereal being poured into the bowl
first, every subsequent user trying to teach our system – despite their
preferences – will be guided towards a “cereal-first” approach. This paper
focuses on tasks that do not have this quality – tasks where the order in
which one picks up a nut, or opens a cupboard is inconsequential – but we hope
that future work expands on our initial formulation, and allows for cases
where users with strong preferences can override the default demonstration
set. In cases where users do have strong preferences, we are instead able to
allow users to make an informed choice about whether they want to spend the
time collection more demonstrations to reflect their preferences. Compared to
the status quo where users just don’t know if their demonstrations are
incompatible in the first place, our approach provides them with more feedback
to make an informed choice when it comes to demonstration collection.

The second limitation stems from our assumption of access to the initial
dataset of demonstrations for training the base policy. Access to this data is
necessary for policy retraining, as well as for developing our compatibility
measure $\mathcal{M}$. One could imagine scenarios where (due to privacy or
release feasibility) this initial dataset cannot be made available. While we
could obtain reasonable proxies to use in our method (e.g., by rolling out
several times in an environment to collect a new dataset), performance of the
learned policy would likely suffer. Luckily, methods for continual learning
and model interpretability – reaching within the black-box – are growing,
presenting a starting point for how to proceed in cases where initial datasets
are not available.

Looking Forward. A pivotal idea of this work is breaking the traditional
influence pattern in human-robot interaction, where a human influences a robot
(by providing demonstrations, feedback, preferences). There is power in going
the other way – allowing the robot to guide and prompt humans as to the type
of data needed to improve. Future work will build upon our Hg-DAgger results,
refining our interface for active elicitation for more responsive interactive
imitation algorithms. Other work might consider language, providing users with
higher-level feedback (“no, reach across first!”) for more effective guidance.

Figure 4: Evaluation trajectories for the Informed (a) & Naive (b) conditions.
In (b), the user provides conflicting demos (a “sideways tilt,” moving
laterally and rotating sideways) compared to the initial demos (“vertical
plating”). The resulting policy moves sideways and forward before failing.

##  7 Conclusion

We introduce a two-phase approach for multi-human interactive imitation
learning that 1) derives a fine-grained measure of demonstrator compatibility
with an existing policy, and 2) develops an interface for actively eliciting
compatible demonstrations from new users. Through our user studies on two
diverse simulation environments, and a real-world Franka Emika Panda arm, we
demonstrate the importance of demonstrator compatibility. We show the
inefficiency of naïve, unprompted demonstration collection and more
importantly, we show that spending time to prompt users with fine-grained
feedback about the demonstrations they provide is extremely useful, boosting
the performance of the task policy by up to 25% absolute success rate from
just 5 demonstrations.

#### Acknowledgments

This project was sponsored by NSF Awards 19417222, 2132847, and 2218760, the
Office of Naval Research (ONR), and the Air Force Office of Scientific
Research (AFOSR). Toyota Research Institute (“TRI”) additionally provided
funded funds to support this work. Siddharth Karamcheti is further grateful to
be supported by the Open Philanthropy Project AI Fellowship.

## References

  * Ross et al. [2011] S. Ross, G. Gordon, and A. Bagnell.  A reduction of imitation learning and structured prediction to no-regret online learning.  In _Artificial Intelligence and Statistics (AISTATS)_ , 2011.
  * Kelly et al. [2019] M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. J. Kochenderfer.  HG-DAgger: Interactive imitation learning with human experts.  In _International Conference on Robotics and Automation (ICRA)_ , pages 8077–8083, 2019.
  * Spencer et al. [2020] J. Spencer, S. Choudhury, M. Barnes, M. Schmittle, M. Chiang, P. J. Ramadge, and S. S. Srinivasa.  Learning from interventions: Human-robot interaction as both explicit and implicit feedback.  In _Robotics: Science and Systems (RSS)_ , 2020.
  * Jang et al. [2021] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn.  BC-Z: Zero-shot task generalization with robotic imitation learning.  In _Conference on Robot Learning (CoRL)_ , 2021.
  * Ebert et al. [2022] F. Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine.  Bridge data: Boosting generalization of robotic skills with cross-domain datasets.  In _Robotics: Science and Systems (RSS)_ , 2022.
  * Ahn et al. [2022] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. J. Joshi, R. C. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Rettinghouse, D. M. Reyes, P. Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, and M. Yan.  Do as I can, not as I say: Grounding language in robotic affordances.  _arXiv preprint arXiv:2204.01691_ , 2022.
  * Bishop [1994] C. M. Bishop.  Mixture density networks.  In _NCRG Report_ , 1994.
  * Lynch et al. [2019] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet.  Learning latent plans from play.  _arXiv preprint arXiv:1903.01973_ , 2019.
  * Florence et al. [2021] P. R. Florence, C. Lynch, A. Zeng, O. Ramirez, A. Wahid, L. Downs, A. S. Wong, J. Lee, I. Mordatch, and J. Tompson.  Implicit behavioral cloning.  In _Conference on Robot Learning (CoRL)_ , 2021.
  * Mandlekar et al. [2021] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martín.  What matters in learning from offline human demonstrations for robot manipulation.  In _Conference on Robot Learning (CoRL)_ , 2021.
  * Beliaev et al. [2022] M. Beliaev, A. Shih, S. Ermon, D. Sadigh, and R. Pedarsani.  Imitation learning by estimating expertise of demonstrators.  In _International Conference on Machine Learning (ICML)_ , 2022.
  * Peng et al. [2019] X. B. Peng, A. Kumar, G. H. Zhang, and S. Levine.  Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.  _arXiv preprint arXiv:abs/1910.00177_ , 2019.
  * Wang et al. [2020] Z. Wang, A. Novikov, K. Zolna, J. T. Springenberg, S. E. Reed, B. Shahriari, N. Siegel, J. Merel, C. Gulcehre, N. M. O. Heess, and N. de Freitas.  Critic regularized regression.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Yang et al. [2022] M. Yang, S. Levine, and O. Nachum.  Trail: Near-optimal imitation learning with suboptimal data.  In _International Conference on Learning Representations (ICLR)_ , 2022.
  * Emmons et al. [2022] S. Emmons, B. Eysenbach, I. Kostrikov, and S. Levine.  Rvs: What is essential for offline RL via supervised learning?  In _International Conference on Learning Representations (ICLR)_ , 2022.
  * Grigsby and Qi [2021] J. Grigsby and Y. Qi.  A closer look at advantage-filtered behavioral cloning in high-noise datasets.  _arXiv preprint arXiv:2110.04698_ , 2021.
  * Zhang et al. [2021] S. Zhang, Z. Cao, D. Sadigh, and Y. Sui.  Confidence-aware imitation learning from demonstrations with varying optimality.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021.
  * Swayamdipta et al. [2020] S. Swayamdipta, R. Schwartz, N. Lourie, Y. Wang, H. Hajishirzi, N. A. Smith, and Y. Choi.  Dataset cartography: Mapping and diagnosing datasets with training dynamics.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , 2020\.
  * Karamcheti et al. [2021] S. Karamcheti, R. Krishna, L. Fei-Fei, and C. D. Manning.  Mind your outliers! investigating the negative impact of outliers on active learning for visual question answering.  In _Association for Computational Linguistics (ACL)_ , 2021.
  * Hoque et al. [2021] R. Hoque, A. Balakrishna, E. R. Novoseller, A. Wilcox, D. S. Brown, and K. Goldberg.  ThriftyDAgger: Budget-aware novelty and risk gating for interactive imitation learning.  In _Conference on Robot Learning (CoRL)_ , 2021.
  * Gopalan et al. [2022] N. Gopalan, N. Moorman, M. Natarajan, M. C. Gombolay, and Georgia.  Negative result for learning from demonstration: Challenges for end-users teaching robots with task and motion planning abstractions.  In _Robotics: Science and Systems (RSS)_ , 2022.
  * Hausman et al. [2017] K. Hausman, Y. Chebotar, S. Schaal, G. S. Sukhatme, and J. J. Lim.  Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017.
  * Kuefler and Kochenderfer [2018] A. Kuefler and M. J. Kochenderfer.  Burn-in demonstrations for multi-modal imitation learning.  In _International Conference on Autonomous Agents and Multiagent Systems (AAMAS)_ , 2018.
  * Brown et al. [2019] D. S. Brown, W. Goo, P. Nagarajan, and S. Niekum.  Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations.  In _International Conference on Machine Learning (ICML)_ , 2019.
  * Chen et al. [2020] L. Chen, R. R. Paleja, and M. C. Gombolay.  Learning from suboptimal demonstration via self-supervised reward regression.  In _Conference on Robot Learning (CoRL)_ , 2020.
  * Myers et al. [2021] V. Myers, E. Biyik, N. Anari, and D. Sadigh.  Learning multimodal rewards from rankings.  In _Conference on Robot Learning (CoRL)_ , 2021.
  * Kim and Pineau [2013] B. Kim and J. Pineau.  Maximum mean discrepancy imitation learning.  In _Robotics: Science and Systems (RSS)_ , 2013.
  * Menda et al. [2019] K. Menda, K. Driggs-Campbell, and M. J. Kochenderfer.  Ensembledagger: A Bayesian approach to safe imitation learning.  In _International Conference on Intelligent Robots and Systems (IROS)_ , pages 5041–5048, 2019.
  * Zhang and Cho [2017] J. Zhang and K. Cho.  Query-efficient imitation learning for end-to-end autonomous driving.  In _Association for the Advancement of Artificial Intelligence (AAAI)_ , 2017.
  * Hoque et al. [2021] R. Hoque, A. Balakrishna, C. Putterman, M. Luo, D. S. Brown, D. Seita, B. Thananjeyan, E. R. Novoseller, and K. Goldberg.  LazyDAgger: Reducing context switching in interactive imitation learning.  _IEEE 17th International Conference on Automation Science and Engineering (CASE)_ , 3:502–509, 2021.
  * Settles [2009] B. Settles.  Active learning literature survey.  Technical report, University of Wisconsin, Madison, 2009.
  * Zhu et al. [2022] Y. Zhu, P. Stone, and Y. Zhu.  Bottom-up skill discovery from unsegmented demonstrations for long-horizon robot manipulation.  _IEEE Robotics and Automation Letters (RA-L)_ , 7:4126–4133, 2022.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * Ba et al. [2016] J. L. Ba, J. R. Kiros, and G. E. Hinton.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * Srivastava et al. [2014] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research (JMLR)_ , 15(1):1929–1958, 2014.
  * Kingma and Ba [2015] D. Kingma and J. Ba.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations (ICLR)_ , 2015.
  * Hochreiter and Schmidhuber [1997] S. Hochreiter and J. Schmidhuber.  Long short-term memory.  _Neural Computation_ , 9(8):1735–1780, 1997.
  * Lin et al. [2021] Y. Lin, A. S. Wang, G. Sutanto, A. Rai, and F. Meier.  Polymetis.  https://facebookresearch.github.io/fairo/polymetis/, 2021.
  * Hendrycks and Gimpel [2016] D. Hendrycks and K. Gimpel.  Gaussian error linear units (gelus).  _arXiv preprint arXiv:1606.08415_ , 2016.
  * Loshchilov and Hutter [2019] I. Loshchilov and F. Hutter.  Decoupled weight decay regularization.  In _International Conference on Learning Representations (ICLR)_ , 2019.
  * Hart [2006] S. G. Hart.  NASA task load index (nasa-tlx); 20 years later.  In _Proceedings of the Human Factors and Ergonomics Society Annual Meeting_ , volume 50, pages 904–908, 2006.

We provide further details, experiments, and descriptions of the attached
media, to reinforce the results and conclusions from the main body of our
paper. For a more fluid viewing experience please look through our project
website, where videos (and corresponding descriptions) are side-by-side:
https://sites.google.com/view/eliciting-demos-corl22/home.

##  Appendix A Additional Dataset Details

Square Nut [10]. The state space for this task is similar to Mandlekar et al.
[10]. We use a proprioceptive state consisting of the robot’s end-effector
position (3-DoF), end-effector rotation as a quaternion (4-DoF), the gripper
position (2-DoF) and a coordinate-based state representation for encoding
object positions and poses (14-dim). We use the original data from Mandlekar
et al. [10], using a 3DConnexion SpaceMouse for 6-DoF teleoperation. The
horizon is set to 500 steps.

We randomly sample 50 demos from the proficient operator [10] to initialize
the base dataset. Operators 1 through 4 are Better OP 1, Better OP 2, Okay OP
1 and Okay OP 2 from the Robomimic multi-human dataset.

Round Nut [10, 20]. The state space for this task is the same as Hoque et al.
[20]; complete robot proprioception states and object states are included. The
data was collected using a keyboard. The horizon is set to 400 steps.

Hammer Placement [32]. The state space for this task is the complete robot
proprioception state and object. The base demonstrations include 20
demonstration collected by the proficient demonstrator and 5 demonstrations
collected by the demonstrator using interactive interventions like in Kelly et
al. [2]. These interactive on-policy demos help us in learning a decent base
policy. Data was collected with a keyboard. The horizon is set to 175 steps.

##  Appendix B Policy Training & Other Implementation Details

Architecture. We train an ensemble of 5-MLPs. Each MLP has 2 hidden layers, a
hidden size of 1024. We use ReLU activations, LayerNorm [34] and a dropout of
0.5 [35] between the hidden layers.

Training. We train the models using an ADAM [36] optimizer with a learning
rate of 1e-3 for 1000 epochs with a batch size of 512. The models are trained
to reduce the mean squared error between the ground truth actions and the
predicted actions.

Evaluation. The models are evaluated for 50 rollouts for their respective
maximum horizons or till the task is completed. Checkpoints are evaluated at
every 200 epochs. We also evaluate the checkpoint with the best validation
loss.

Compatibility Thresholds. We use the thresholds detailed in Table 3 to compute
the compatibility score $\mathcal{M}$ for the new demonstrations
$\mathcal{D}_{\text{new}}$. Likelihood is measured using a negative mean
squared error between the actions predicted by $\pi_{\text{base}}$ and the
provided actions $a_{\text{new}}$. The novelty of a state is measured by the
standard deviation in the predicted actions from the ensemble policy. To
select these thresholds, we assume access to a compatible and an incompatible
trajectory in addition to the base demonstrations. We regress these thresholds
from a 2D compatibility map of likelihood vs novelty.

Parameter | Square Nut | Round Nut | Hammer Placement
---|---|---|---
Novelty $\eta$ | 0.05 | 0.05 | 0.06
Likelihood $\lambda$ | 0.4 | 0.35 | 0.35
Table 3: Thresholds for novelty (standard deviation of predicted actions) and
likelihood (mean squared error between predicted actions and provided
actions). The standard deviation and the MSE of actions were averaged across
the dimensions of the action space.

##  Appendix C Baseline Results

Operator | Round Nut | Hammer Placement
---|---|---
5-MLP | MDN | RNN | 5-MLP | MDN | RNN
Base | 13.3 (2.3) | 8.0 (4.0) | 14.7 (2.3) | 24.7 (6.1) | 11.3 (1.2) | 43.3 (13.3)
Operator 1 | 26.7 (11.7) | 29.3 (9.5) | 31.3 (8.3) | 38.0 (2.0) | 30.7 (15.5) | 30.0 (8.0)
Operator 2 | 22.0 (7.2) | 11.3 (3.1) | 15.3 (3.1) | 33.3 (3.1) | 12.0 (3.5) | 24.7 (3.1)
Operator 3 | 17.3 (4.6) | 10.7 (7.6) | 4.7 (3.1) | 8.0 (0.0) | 12.0 (5.3) | 48.0 (15.6)
Operator 4 | 7.3 (4.6) | 4.7 (3.1) | 13.3 (2.3) | 4.0 (0.0) | 6.7 (2.3) | 8.7 (5.0)
Table 4: Success rates on Round Nut and Hammer Placement (mean/std across 3 training runs) for policies trained on $\mathcal{D}_{\text{new}}$ from different operators using different models. Operator | Square Nut
---|---
5-MLP | MDN | RNN
Base | 38.7 (2.1) | 23.3 (1.2) | 30.7 (1.2)
Operator 1 | 54.3 (1.5) | 27.3 (8.3) | 31.3 (1.2)
Operator 2 | 40.3 (5.1) | 15.3 (6.1) | 10.7 (1.2)
Operator 3 | 37.3 (2.1) | 12.0 (2.0) | 10.0 (2.0)
Operator 4 | 27.3 (3.5) | 10.0 (0.0) | 10.7 (3.1)
Table 5: Success rates on Square Nut (mean/std across 3 training runs) for
policies trained on $\mathcal{D}_{\text{new}}$ from different operators using
different models.

###  C.1 Mixture Density Network (MDN)

Architecture. We train a Mixture Density Network (MDN) with 2 components
corresponding to the 2 operators in the aggregated dataset
$\mathcal{D}_{\text{base}}\cup\mathcal{D}_{\text{new}}$. The MDN is modelled
as an MLP with 2 hidden layers and a hidden size of 1024. We use ReLU
activations, LayerNorm [34] and a dropout of 0.5 [35] between the hidden
layers.

Training. We train the model using an ADAM [36] optimizer with a learning rate
of 1e-4 for 1000 epochs with a batch size of 512. The models are trained to
maximize the log likelihood of the expert actions.

Evaluation. The models are evaluated for 50 rollouts for their respective
maximum horizons or till the task is completed. Checkpoints are evaluated at
every 200 epochs. We also evaluate the checkpoint with the best validation
loss. We use a low-noise evaluation scheme similar to Mandlekar et al. [10],
setting the scale of the Gaussian components to 1e-4 during the evaluation
phase.

Results and Discussion. From the results in Table 4 and Table 5, we find that
using an MDN is worse, in general, compared to an ensemble of MLPs. The trends
of operators being compatible to varying degrees with the base dataset holds
even when using an MDN. Further, when we aggregate demonstrations from
multiple users, it is difficult to pre-define the number of modes (one mode
per user) for the MDN. Thus, trying to model multiple modes using an MDN does
not help mitigate the lack of compatibility between
$\mathcal{D}_{\text{base}}$ and $\mathcal{D}_{\text{new}}$. We also find that
the uncertainty estimates in the MDN are not calibrated and tend to collapse
to a constant value, making it difficult to use for active elicitation (as we
have no metric to tell novel states apart from familiar ones).

###  C.2 Recurrent Neural Network (RNN)

Architecture. We train an ensemble of 5 LSTM [37] models with two layers and
512 hidden units.

Training. We train the models using an ADAM [36] optimizer with a learning
rate of 1e-3 for 1000 epochs with a batch size of 512. The models are trained
to reduce the mean squared error between the ground truth actions and the
predicted actions.

Evaluation. The models are evaluated for 50 rollouts for their respective
maximum horizons or till the task is completed. Checkpoints are evaluated at
every 200 epochs. We also evaluate the checkpoint with the best validation
loss.

Results and Discussion. From the results in Table 4 and Table 5, we find that
the ensemble of RNNs, in general, is comparable to an ensemble of MLPs.
Similar to an ensemble of MLPs and MDNs, the trends are quite consistent,
albeit a couple of exceptions (e.g., Operator 3 in Hammer Placement). The
added computational load of using a sequential model does not mitigate the
problem of incompatibility between the demonstrations from different users. We
prefer to use an ensemble of MLPs (for their lower computational load) to test
the validity of our compatibility metric and demonstrations elicitation
procedure. Our procedure can easily be extended to sequential models.

##  Appendix D Real-World Robot Task Details: Food Plating

Hardware Details. We use a Franka Emika Panda arm for our experiments. We use
a RealSense camera to record visual observations (as RGB images). For control,
we predict 7-DoF joint actions and use the Polymetis library [38] for low-
level impedance control. We keep the gripper of the Panda arm in a fixed
position grasping the pan throughout the task.

Policy Architecture. We train an ensemble of 5 visually-conditioned policies.
We use a ResNet34 backbone pretrained on Imagenet [33] to encode the visual
observations, keeping the ResNet weights frozen. The robot proprioceptive
state consists of the end effector position and pose (as a quaternion),
concatenated with the visual embeddings and passed through an MLP to predict
the actions. The MLP consists of two hidden layers with a hidden size of 64
and GELU [39] activations.

Active Elicitation. If a demonstration is rejected, we provide corrective
feedback to demonstrators after the demo has been recorded. We show a video of
the incompatible parts of the trajectory, retrieve and play the closest expert
demo to the rejected one. For the retrieval of corrective demos, we look at
the similarity of demos in the state space. This is done by measuring the L2
distance of the ResNet embeddings. This isn’t a perfect measure and that lots
of other work tries to solve this problem; we choose ResNet features to be
expedient.

Policy Training. We train the ensemble of visual policies for 20 epochs with a
batch size of 512. The model is trained to minimize the mean squared error
(MSE) between predicted and recorded actions. We use an AdamW optimizer [40]
with a learning rate of 1e-3.

Evaluation. For evaluation, we choose five points for the location of the
plate and evaluate each policy for 5 rollouts.

##  Appendix E Additional Results on Active Elicitation

Figure 5: Results of our post-study survey. All responses are collected on a
5-point Likert scale (1: Strongly Disagree, 5: Strongly Agree).

Round Nut. We collect data from 16 users (age = $23.7\pm 1.7$, 11 males, 5
females). Each user is either assigned to the naive or informed condition.

Hammer Placement. We collect data from 4 users (age = $23.2\pm 0.9$, 4 males).
Each user performs the naive condition first and then the informed condition.

[Real] Food Plating. We collect data from 4 users (age = $23.0\pm 1.1$, 1
male, 3 females). Each user performs the naive condition first and then the
informed condition.

Post Study Survey. We asked users to rate their experience in collecting the
demonstrations by asking them questions related to mental demand, task
difficulty and task comprehension on a 5-point Likert scale. These questions
were inspired from Hart [41].

Figure 6: (a) and (c) show 2D “maps” of demonstrations collected from an
informed operator and a naive operator respectively. (b) shows the success
rates on using the two sets of demonstrations to train a policy.

Discussion. From the post-study survey Fig. 5, we find that our method is
slightly more difficult to understand (+0.37; averaged across 3 tasks),
requires marginally more mental demand (+0.30; averaged across 3 tasks), and
is a little more difficult to succeed at (+0.20; averaged across 3 tasks). We
find that this marginal increase in difficulty and effort in performing the
task lead to the collection of significantly better demonstrations. For
instance, we see in Fig. 6 that the informed operator, using our active
elicitation interface, is much better at giving more compatible demonstrations
that the naive operator. This is reflected in the success rates acheived by
the corresponding policies (25.3 v/s 7.3).

Trajectory Lengths. In Table 6, we present the average trajectory lengths for
demonstrations collected by the base user, naïve users, and informed users. We
find that informed users tend to be more optimal in providing demonstrations
while also providing demos of a similar style to the base user. For
demonstrations with the real food plating task, the base user’s style requires
longer trajectories on average compared to a naïve user’s style. Our active
elicitation procedure is able to bring the average trajectory length of an
informed user closer to that of the base user. So, we are able to elicit
behavior that matches a style, not solely optimizing for shorter trajectories.

Task | Base | Naïve | Informed
---|---|---|---
Round Nut | 87.3 | 95.9 (12.5) | 88.9 (6.8)
Hammer Placement | 174.6 | 185.5 (34.3) | 174 (8.7)
Real: Food Plating | 306.5 | 263.7 (10.26) | 278.3 (8.9)
Table 6: Average trajectory lengths for demonstrations collected using active
elicitation and naïve collection.

##  Appendix F Active Elicitation with Human-Gated (HG) DAgger

Operator | Naïve | Informed
---|---|---
Base | 13.3 (2.3) | -
Operator 1 | 24 (3.5) | 25.3 (5.0)
Operator 2 | 18 (7.2) | 23.3 (4.2)
Operator 3 | 31.3 (9.9) | 21.3 (2.3)
Operator 4 | 29.3 (5.8) | 32.7 (7.0)
Table 7: Success rates (mean/std across 3 random seeds) for user studies
evaluating both naive and informed demonstration collection using HG-DAgger
against base users for the Round Nut task.

Procedure. We use the same interface as described in Section 5 to collect
demonstrations interactively using Human-Gated DAgger [2]. Users were asked to
help a robot complete the Round Nut task successfully five times. They were
instructed to intervene and help the robot when they thought the robot was
stuck or was making a mistake in completing the task. They were also told to
give control back to the robot when they thought the robot could complete the
task successfully.

$\mathcal{D}_{\text{base}}$ consists of 30 trajectories collected by a
proficient operator. For this task, we perform a longitudinal study with $n=3$
participants, where users are first asked to complete 5 demonstrations in the
naive condition and then 5 demonstrations in the informed condition. This
allows us to measure the effect of the interface in eliciting demonstrations
within subjects.

Results and Discussion. Informed elicitation works better for three out of
four operators (see Table 7) but the gains are lower compared to the condition
where we collect complete trajectories. Further, we observe that none of the
conditions result in a policy that is worse than the base policy. We find that
the base policy is quite good and only requires intervention in a few
“critical states” like picking the nut up or inserting the nut into the peg.
Further, the results also show that the high frequency feedback to the users
from our interface does not discourage them from intervening and providing
corrections. Our results our limited by the number of users we test in this
condition and also by the task that we consider. Future work will address how
active elicitation might help in interactive imitation learning across more
users and more diverse tasks.

##  Appendix G Operator-wise Success Rates

Table 8: Success rates (mean/std across 3 random seeds) for different
operators.

Operator | Success Rates
---|---
Base | 13.3 (2.3)
Naïve 1 | 16 (3.5)
Naïve 2 | 7.3 (4.6)
Naïve 3 | 6.7 (1.2)
Naïve 4 | 8.0 (2.0)
Naïve 5 | 13.3 (4.2)
Naïve 6 | 7.3 (3.1)
Naïve 7 | 4.7 (1.2)
Naïve 8 | 13.3 (2.3)
Informed 1 | 25.3 (1.2)
Informed 2 | 20.0 (2.0)
Informed 3 | 18.0 (3.5)
Informed 4 | 11.3 (2.3)
Informed 5 | 16.0 (3.5)
Informed 6 | 5.3 (1.2)
Informed 7 | 15.3 (1.2)
Informed 8 | 14.0 (3.5)
(a) Round Nut

Operator | Naïve | Informed
---|---|---
Base | 24.7 (6.1) | -
Operator 1 | 8.0 (0.0) | 28.0 (6.0)
Operator 2 | 33.3 (3.1) | 52.7 (10.1)
Operator 3 | 38.0 (2.0) | 35.3 (2.3)
Operator 4 | 4.0 (0.0) | 11.3 (2.3)
(b) Hammer Placement


circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] (294.50,146.33) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (350.67,154.99)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] (406.84,165.89) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (463.00,168.48)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] ( 69.84,143.42) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (126.00,142.18)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] (182.17,147.60) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (238.34,136.11)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] (294.50,138.41) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (350.67,167.02)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] (406.84,178.78) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (463.00,181.57)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] ( 69.84,190.66) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (126.00,188.18)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] (182.17,187.39) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (238.34,184.36)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] (294.50,184.98) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (350.67,181.63)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] (406.84,176.17) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (463.00,174.42)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] ( 69.84,193.81) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (126.00,191.73)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] (182.17,189.57) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (238.34,186.44)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] (294.50,184.91) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (350.67,181.92)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] (406.84,178.40) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (463.00,174.25)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] ( 69.84,189.45) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (126.00,187.75)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] (182.17,185.88) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (238.34,182.48)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] (294.50,183.40) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (350.67,176.69)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] (406.84,166.50) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (463.00,163.57)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] ( 69.84,187.57) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (126.00,185.03)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] (182.17,188.89) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (238.34,184.80)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] (294.50,183.62) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (350.67,182.11)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] (406.84,173.56) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (463.00,168.25)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] ( 69.84,194.47) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (126.00,192.32)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] (182.17,191.67) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (238.34,190.26)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] (294.50,188.83) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (350.67,183.66)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] (406.84,178.64) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (463.00,177.05)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] ( 69.84,205.30) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (126.00,204.84)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] (182.17,214.75) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (238.34,208.56)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] (294.50,205.37) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (350.67,188.39)
circle ( 1.53); [draw=drawColor,line width= 0.4pt,line join=round,line
cap=round,fill=fillColor] (406.84,206.76) circle ( 1.53); [draw=drawColor,line
width= 0.4pt,line join=round,line cap=round,fill=fillColor] (463.00,206.67)
circle ( 1.53); fillColorRGB0,0,255 [fill=fillColor] ( 69.84,165.69) circle (
2.50); [fill=fillColor] (126.00,165.23) circle ( 2.50); [fill=fillColor]
(182.17,164.12) circle ( 2.50); [fill=fillColor] (238.34,163.58) circle (
2.50); [fill=fillColor] (294.50,162.98) circle ( 2.50); [fill=fillColor]
(350.67,169.60) circle ( 2.50); [fill=fillColor] (406.84,162.17) circle (
2.50); [fill=fillColor] (463.00,161.02) circle ( 2.50); [] ( 50.18, 29.33)
rectangle (482.66,233.33); [clip] ( 0.00, 0.00) rectangle (498.66,249.33);
drawColorgray0.40 [draw=drawColor,line width= 0.5pt,line join=round] ( 50.18,
29.33) – ( 50.18,233.33); [clip] ( 0.00, 0.00) rectangle (498.66,249.33);
drawColorgray0.13 [text=drawColor,anchor=base east,inner sep=0pt, outer
sep=0pt, scale= 1.00] at ( 45.68, 40.26) -8; [text=drawColor,anchor=base
east,inner sep=0pt, outer sep=0pt, scale= 1.00] at ( 45.68, 87.57) -6;
[text=drawColor,anchor=base east,inner sep=0pt, outer sep=0pt, scale= 1.00] at
( 45.68,134.87) -4; [text=drawColor,anchor=base east,inner sep=0pt, outer
sep=0pt, scale= 1.00] at ( 45.68,182.17) -2; [text=drawColor,anchor=base
east,inner sep=0pt, outer sep=0pt, scale= 1.00] at ( 45.68,229.47) 0; [clip] (
0.00, 0.00) rectangle (498.66,249.33); drawColorgray0.40 [draw=drawColor,line
width= 0.5pt,line join=round] ( 47.68, 43.71) – ( 50.18, 43.71);
[draw=drawColor,line width= 0.5pt,line join=round] ( 47.68, 91.01) – ( 50.18,
91.01); [draw=drawColor,line width= 0.5pt,line join=round] ( 47.68,138.31) – (
50.18,138.31); [draw=drawColor,line width= 0.5pt,line join=round] (
47.68,185.61) – ( 50.18,185.61); [draw=drawColor,line width= 0.5pt,line
join=round] ( 47.68,232.92) – ( 50.18,232.92); [clip] ( 0.00, 0.00) rectangle
(498.66,249.33); drawColorgray0.40 [draw=drawColor,line width= 0.5pt,line
join=round] ( 50.18, 29.33) – (482.66, 29.33); [clip] ( 0.00, 0.00) rectangle
(498.66,249.33); drawColorgray0.40 [draw=drawColor,line width= 0.5pt,line
join=round] ( 69.84, 26.83) – ( 69.84, 29.33); [draw=drawColor,line width=
0.5pt,line join=round] (182.17, 26.83) – (182.17, 29.33); [draw=drawColor,line
width= 0.5pt,line join=round] (294.50, 26.83) – (294.50, 29.33);
[draw=drawColor,line width= 0.5pt,line join=round] (406.84, 26.83) – (406.84,
29.33); [clip] ( 0.00, 0.00) rectangle (498.66,249.33); drawColorgray0.13
[text=drawColor,anchor=base,inner sep=0pt, outer sep=0pt, scale= 1.00] at (
69.84, 17.94) 1998; [text=drawColor,anchor=base,inner sep=0pt, outer sep=0pt,
scale= 1.00] at (182.17, 17.94) 2000; [text=drawColor,anchor=base,inner
sep=0pt, outer sep=0pt, scale= 1.00] at (294.50, 17.94) 2002;
[text=drawColor,anchor=base,inner sep=0pt, outer sep=0pt, scale= 1.00] at
(406.84, 17.94) 2004; [clip] ( 0.00, 0.00) rectangle (498.66,249.33);
drawColorgray0.27 [text=drawColor,rotate= 90.00,anchor=base,inner sep=0pt,
outer sep=0pt, scale= 1.28] at ( 24.85,131.33) $\ln($Theil TFP$)$;