# PyECLOUD and build-up simulations at CERN

G. Iadarola(1),(2) and G. Rumolo(1)
(1) CERN  Geneva  Switzerland  (2) Università di Napoli “Federico II  Naples
Italy

###### Abstract

PyECLOUD is a newly developed code for the simulation of the electron cloud
(EC) build-up in particle accelerators. Almost entirely written in Python, it
is mostly based on the physical models already used in the ECLOUD code but,
thanks to the implementation of new optimized algorithms, it exhibits a
significantly improved performance in accuracy, speed, reliability and
flexibility. Such new features of PyECLOUD have been already broadly exploited
to study EC observations in the Large Hadron Collider (LHC) and its injector
chain as well as for the extrapolation to high luminosity upgrade scenarios.

##  1 INTRODUCTION

The analysis of the electron cloud (EC) observations in the Large Hadron
Collider (LHC) and its injectors have raised new challenges for the EC build-
up simulations. On one hand, for a correct understanding of machine
observations it is often necessary to deal with beams with thousands of
bunches and with non idealities like non uniform bunch populations and bunch
lengths along the train. On the other hand, the demand for extensive
parametric scans gives quite stringent requirements in terms of speed and
reliability.

CERN has a long experience in the EC build-up simulation, mostly carried out
with the ECLOUD code, developed and maintained at CERN since 1997 [1, 2, 3,
4]. Unfortunately, due to its not modular structure and to the programming
language (FORTRAN 77), this code would need a deep reorganization and a
serious upgrade to enable it to fulfill the aforementioned requirements.

Therefore we have decided to write a fully reorganized code, in a more modern
and powerful language, considering that the initial effort would be
compensated by a significantly increased efficiency in future developments and
debugging. The new code has been called PyECLOUD, since it is almost entirely
written in Python and is largely based on the physical models of the ECLOUD
code. Nevertheless, several features and implementations have been modified,
in some cases completely redesigned, with respect to ECLOUD, with substantial
improvements in terms of reliability, accuracy, speed and usage flexibility.

##  2 PyECLOUD

Like ECLOUD, PyECLOUD is a 2D macroparticle (MP) code, where the electrons are
grouped in MPs in order to achieve a reasonable computational burden.

The dynamics of the MP system is simulated following the flow diagram sketched
in Fig. 1.

Figure 1: Flowchart representing PyECLOUD main loop. Figure 2: Top: evolution
of the number of electrons in the beam pipe for an LHC type beam with 25ns
bunch spacing in the SPS (2 trains of 72 bunches); middle: evolution of the
reference MP size; bottom: evolution of the number of MPs, the regeneration
threshold is highlighted in red.

At each time step, seed electrons, due to residual gas ionization and/or to
synchrotron radiation induced photoemission from the chamber walls, are
generated with a number consistent with the passing beam slice and with
positions and momenta determined by theoretical or empirical models.
Then the electric field acting on each MP is evaluated: the field of the beam
is precomputed on a suitable rectangular grid, loaded from file and obtained
at each MP location by a linear (4 points) interpolation; the space charge
contribution of the electron system itself is calculated by a classical
Particle in Cell (PIC) algorithm, where the finite difference method is
employed to solve the Poisson equation with perfectly conducting boundary
conditions on the beam chamber.
Once the total electric field at each MP location is known, MP positions and
momenta are updated by integrating the dynamics equation; at this stage the
presence of an externally applied dipolar magnetic field can also be taken
into account.
At each time step, a certain number of MPs can hit the wall. In these cases a
proper model of the secondary emission process is applied to generate charge,
energy and angle of the emitted electrons. According to the size of the
emitted charge, a rescaling of the impinging MP can be performed or new MPs
can be emitted.

###  2.1 MP size management

One of the peculiarities of the EC build-up process is the fact that, due to
the multipacting effect, the number of electrons can spread several orders of
magnitude during the passage of the bunch train (see Fig. 2 \- top). As a
consequence, it is impossible to choose a MP size which is suitable for the
entire simulation, allowing both a satisfactory description of the phenomena
and a computationally affordable number of MPs. The MP size management in
PyECLOUD has been significantly improved with respect to ECLOUD and will be
briefly described in this subsection.

MP sizes are not enforced throughout the simulation process but are determined
step by step by “decisions” taken during the execution. For this purpose a
target MP size $N_{ref}$, dynamically adapted during the simulation, is
employed to control the number of electrons per MP. In particular:

  * •

The size of MPs generated by seed mechanisms is exactly $N_{ref}$;

  * •

When a MP hits the wall, it is simply rescaled according to the Secondary
Electron Yield (SEY) if the emitted charge is below $1.5N_{ref}$, otherwise
”true” secondary MPs are generated so that the resulting MP size is as close
as possible to $N_{ref}$;

  * •

Once per bunch passage, a cleaning procedure is performed, which deletes the
MPs with charge lower than $10^{-4}N_{ref}$.

$N_{ref}$ is changed whenever the total number of MPs becomes larger than a
certain threshold defined in the input file(typical value $\sim 10^{5}$),
which means that the computational burden has become too high. When this
happens, a regeneration of the set of MPs is applied, by the following
procedure (see Fig. 2):

  * •

Each MP is assigned to a cell of a uniform grid in the 5-D phase space
$(x,y,v_{x},v_{y},v_{z})$ obtaining the phase space distribution of the
electron gas distributed on the mesh points;

  * •

The new $N_{ref}$ is chosen in order to get a target number of MPs (typically
5-10 times smaller than the regeneration threshold), which still allows for an
accurate simulation but with a more reasonable computational effort;

  * •

A new set of MPs, having the new reference size, is generated according to the
computed distribution.

The preservation of the entire phase space is very important in EC build-up
simulation since the dynamics imparted by passing bunches generates very
distinctive velocity distributions at the different time steps and the
conservation of few specific moments would not guarantee a sufficient
accuracy.
Several numerical test have shown that the errors on the total charge and the
total energy which are introduced by this procedure, are about 1% at the first
time step after the regeneration and they become even smaller at the first
bunch passage after the regeneration.

Figure 3: Electron cloud build-up simulations for different time steps in ECLOUD (top) and PyECLOUD (bottom). Simulated case: SPS MBB bending magnet, 26GeV, two trains of 72 bunches with 225ns gap, 25ns bunch spacing, 1.1$\cdot 10^{11}$ protons per bunch. | Processing time
---|---
Time step [ps] | ECLOUD | PyECLOUD
200 | 29 min | 12 min
100 | 1 h 27 min | 13 min
50 | 1 h 45 min | 24 min
25 | 3 h 7 min | 40 min
12 | 4 h 15 min | 1 h 6 min
Table 1: Computation time required by ECLOUD and PyECLOUD for the simualtions
in Fig. 3.

###  2.2 Performances

The passage from ECLOUD to PyECLOUD had a significant impact on the
performances both in terms of accuracy and of computational efficiency. Fig. 3
shows a comparison between the two codes in terms of convergence properties
with respect to the time step which is chosen for the simulation. While in
ECLOUD it is quite difficult to achieve a good convergence, PyECLOUD gives a
good estimate of the total number of electrons in the chamber already for
quite large time steps ($\sim 0.1$ns) while a satisfactory convergence is
obtained for a time step of the order of $25$ps.

For the same test cases the simulation time required by the two codes is
reported in Tab. 1 showing that the improvements introduced in PyECLOUD had
also a positive impact on the code efficiency.

Furthermore, the new code has been designed in order to offer an increased
usage flexibility, allowing to deal with irregular beam structures e.g. non
uniform bunch intensity and/or bunch length along the bunch train, irregular
bunch spacings and bunch profiles.

Thanks to these new features, PyECLOUD has been already largely exploited at
CERN for several EC simulation studies for the LHC and its injector chain [5,
6, 7, 8]. In particular, as described in detail in [9], PyECLOUD simulations
have been used to reconstruct the evolution of the SEY of the chambers in the
LHC arcs, from the measurement of the heat load deposited on the beam screen
of the cryogenic magnets. The new code also allows us to estimate the bunch by
bunch energy loss due to the interaction of the beam with the EC and to export
the electron distribution seen by each bunch. The first feature allowed us to
benchmark the results against bunch by bunch stable phase measurements [10]
while the second was used, together with HEADTAIL simulations, to analyze the
instabilities observed in the LHC with 25ns bunch spacing [11].

Simulation studies have also addressed the EC formation in the common vacuum
chambers of the LHC. An example of this kind of application will be described
in the following section in order to show the capability of PyECLOUD to deal
with beams made of thousands of bunches with irregular spacings.

##  3 EC build up in LHC common vacuum chambers

Common vacuum chambers having 800mm diameter are installed on both sides of
the ALICE experiments in the Long Straight Section 2 (LSS2) of the LHC. During
2011 operation with 50ns bunch spacing an important pressure rise was noticed
in these chambers with a significant impact on the background observed by the
ALICE experiment.
The analysis of the pressure data has shown that a severe pressure increase is
observed only when the two rings of the LHC are completely filled. Fig. 4
shows the pressure evolution during a proton-proton physics fill (with 50ns
spacing) in which the injection from the SPS of the last two trains of 144
bunches was delayed by about one hour. It can be noticed that the pressure
rise appears already at the injection energy (450GeV), but only after the last
two injections have taken place.

Figure 4: Pressure evolution in the 800mm chambers near the ALICE experiment
of the LHC during a proton physics fill. The total beam intensity is the two
rings is also included.

PyECLOUD simulations have been run in order to investigate if the EC formation
in these chambers could explain this pecular behavior. The presence of both
counter-rotating beams in the chamber had to be taken into account since it
determines different “hybrid” bunch spacings at the different sections of the
$\sim$30m long vacuum chambers when both beams are circulating in the LHC (see
Fig. 5).
In particular the two beam configurations in Fig. 6 have been simulated, which
correspond to the beam patterns in the two rings at the moments indicated by
(a) and (b) in Fig. 4. The results at a certain section of the considered
vacuum pipes are shown in Fig. 7.
It can be noticed that, in the configuration of Fig. 6a, both beams present a
gap of about one quarter of the length of the ring. Probably due to the quite
large radius, the EC can develop only when both beams are passing in the
chamber, while a decay of the number of electrons is observed when only one
beam is passing and the gap is long enough to allow a complete reset of the EC
between subsequent turns.
On the other hand, after the injection of the last two trains from the SPS,
the layout of the beam in the two rings looks like the one in Fig. 6b where no
large gap is present in any of the two beams. As a consequence a complete
decay of the EC between subsequent turns is not possible anymore. In fact, a
memory effect is observed between turns with a strong enhancement of the EC
activity and, as consequence, of the electron stimulated gas desorption
leading to the observed pressure rise.

Figure 5: Hybrid bunch spacings which are observed at different sections of
the 800mm common chambers. Figure 6: Filling pattern of the two beams in the
LHC before (a) and after (b) the injections of the last two trains. Beam 1
(blue) is clockwise rotating, beam 2 (red) counterclockwise. The position the
straight section 2 (ALICE) is highlighted Figure 7: PyECLOUD simulation of the
electron cloud build-up in the 800mm common chambers for the filling patterns
in Fig. 6.

##  4 CONCLUSIONS

PyECLOUD is a new EC build-up simulation code, which has been developed at
CERN for a reliable and efficient analysis of the EC observations in LHC as
well as for EC studies related to the high luminosity upgrade of the LHC and
its injectors.

Based on the physical models already implemented in ECLOUD, the new code
features several improvements in terms of algorithms and implementation (e.g.
a new MP size management method) with a significant impact on speed and
convergence properties. Moreover the code is explicitly designed to deal with
irregular beam structures (e.g. different bunch intensities and bunch lengths
along the train, arbitrary spacings and profiles) in order to allow for an
accurate analysis of EC observations in CERN accelerators.

Several EC studies for the LHC and its injectors have been already carried out
with the new code giving very encouraging indications on the reliability of
the models and numerical solutions.

##  5 ACKNOWLEDGMENTS

The authors would like to express their gratefulness to G. Arduini, H.
Bartosik, C. Bhat, V. Baglin, R. De Maria, O. Dominguez, M. Driss Mensi, J.
Esteban-Muller, K. Li, H. Maury Cuna, G. Miano, E. Méétral, H. Neupert, G.
Papotti, E. Shaposhnikova, M. Taborelli, L. Tavian, C. Y. Vallgren, and F.
Zimmermann for the support they provided in the code development, the
simulation work and the machine data analysis as well as for valuable comments
and discussions.

## References

  * [1] F. Zimmermann, “A Simulation Study of Electron-Cloud Instability and Beam-Induced Multipacting in the LHC”, CERN LHC Project Report 95 (February 1997)
  * [2] F. Zimmermann, “Electron Cloud Simulations for SPS and LHC”, Proc. Chamonix X, 17-21 January 2000, CERN-SL-2000-007 DI (2000), p. 136
  * [3] G. Rumolo and F. Zimmermann, “Practical User Guide for ECloud”, CERN-SL-Note-2002-016.
  * [4] G. Rumolo et al.,“Simulation of the electron-cloud build up and its consequences on heat load, beam stability, and diagnostics”, Phys. Rev. ST Accel. Beams 4 (2001), 1
  * [5] G. Iadarola and G. Rumolo, “Electron Cloud in the CERN accelerators (PS, SPS, LHC)”, elsewhere in these proceedings
  * [6] H. Maury Cuna et al., “Simulation of electron-cloud heat load for the cold arcs of the Large Hadron Collider”, Proceedings of the 3rd International Particle Accelerator Conference 2012, 20 - 25 May 2012, New Orleans, USA
  * [7] F. Caspers et al., “Comparison between electron cloud build-up measurements and simulations at the CERN PS”, Proceedings of the 3rd International Particle Accelerator Conference 2012, 20 - 25 May 2012, New Orleans, USA
  * [8] C. Bhat et al., “E-Cloud dependence on the Bunch Profile - An Experiment in the PS and an Extension to the LHC”, elsewhere in these proceedings
  * [9] G. Iadarola and G. Rumolo, “Electron cloud simulations with PyECLOUD”, Proceedings of the 11th International Computational Accelerator Physics Conference (ICAP), 19-24 August 2012, Rostock-Warnemünde, Germany
  * [10] J. Esteban Muller et al., “Synchronous Phase Shift at LHC”, elsewhere in these proceedings
  * [11] H. Bartosik et al., “Benchmarking HEADTAIL with e-cloud observations with LHC 25ns beam”, elsewhere in these proceedings


# Learning Online from Corrective Feedback: A Meta-Algorithm for Robotics

Matt Schmittle
Paul G. Allen School of Computer Science
University of Washington United States
<EMAIL_ADDRESS>
&Sanjiban Choudhury
Paul G. Allen School of Computer Science
University of Washington United States
<EMAIL_ADDRESS>
Siddhartha S. Srinivasa
Paul G. Allen School of Computer Science
University of Washington United States
<EMAIL_ADDRESS>

###### Abstract

A key challenge in Imitation Learning (IL) is that optimal state actions
demonstrations are difficult for the teacher to provide. For example in
robotics, providing kinesthetic demonstrations on a robotic manipulator
requires the teacher to control multiple degrees of freedom at once. The
difficulty of requiring optimal state action demonstrations limits the space
of problems where the teacher can provide quality feedback. As an alternative
to state action demonstrations, the teacher can provide corrective feedback
such as their preferences or rewards. Prior work has created algorithms
designed to learn from specific types of noisy feedback, but across teachers
and tasks different forms of feedback may be required. Instead we propose that
in order to learn from a diversity of scenarios we need to learn from a
_variety of feedback_. To learn from a variety of feedback we make the
following insight: the teacher’s cost function is latent and we can model a
stream of feedback as a stream of loss functions. We then use any online
learning algorithm to minimize the sum of these losses. With this insight we
can learn from a diversity of feedback that is weakly correlated with the
teacher’s true cost function. We unify prior work into a general corrective
feedback meta-algorithm and show that regardless of feedback we can obtain the
same regret bounds. We demonstrate our approach by learning to perform a
household navigation task on a robotic racecar platform. Our results show that
our approach can learn quickly from a variety of noisy feedback.

> Keywords: Corrective Feedback, Online-Learning

##  1 Introduction

Imitation learning (IL) encompasses algorithms that leverage expert feedback
to train dynamical systems. Key to its appeal is the observation [1, 2, 3]
that it is often easier for experts to provide insight via demonstrations or
feedback rather than to hand-tune reward functions. Additionally, even when
reward functions are pre-specified, IL demonstrates greater sample efficiency
[4, 2] than pure reinforcement learning (RL), which is critical for robotics
[5, 6] where real-world execution can be dangerous or expensive.

A key challenge to IL, however, lies in the details of how the expert
interacts with the system. Most IL algorithms expect optimal state-action
feedback. But, giving that feedback might be challenging, even for experts.
For example, Muxfeldt et al. [7] ran a user study on the viability of
kinesthetic demonstrations using a manipulator for an assembly task and found
minimal correlation between a person’s performance at manual assembly (no
robot) and their performance through kinesthetic teaching.

Prior work has addressed the difficulty in providing demonstrations by
alternatively using corrective feedback. Corrective feedback does not require
the teacher to be able to provide a full demonstration but instead adjustments
during robot execution. For example, instead of providing kinesthetic
demonstrations Bajcsy et al. [8] simply physically pushes the robot during
execution to the desired behavior. Even simplier, preference learning only
asks the user to choose between two proposed actions [9]. While these methods
provide a simple and effective option for human teachers, people are
different. For each person and task there may be a different ideal feedback
that properly balances ease and informativeness. Jain et al. [10] highlights
this by having users train a manipulator on a two tasks with two forms of
feedback, ranking and physical re-orientation. They find the ratio of type of
feedback the users use differs across both users and tasks. Our observation is
thus, in order to have an algorithm that can learn from different teachers and
different tasks, it needs to be able to learn from a variety of noisy
feedback.

Figure 1: The user has a desired latent policy where the car drives left
through the cones to the blue X. The user can communicate their latent policy
through a variety of noisy channels: a demonstration, online steering
corrections, and/or verbal feedback to name a few. If we can model each
channel with an associated loss function then we can update our autonomous
policy through online learning.

Building on the above observation we have the following key insight: The
desired human policy is latent. Each feedback reveals some information about
the latent policy. We can model a stream of feedback as a stream of loss
functions. We wish to find a policy that has low loss on the sum of all loss
functions. This naturally fits into the framework of online learning. This
insight is illustrated in Fig. 1. Using this insight we create the Corrective
Feedback Meta-Algorithm, which unifies prior work into an approach that can
learn from a variety of noisy feedback. Specifically, our approach has the
following properties:

  1. 1.

We can learn from a variety of feedback that needs only a mapping from
feedback to loss.

  2. 2.

We can learn from noisy feedback.

  3. 3.

With few assumptions we can bound regret in terms of the teacher’s latent
loss.

  4. 4.

Prior work fits into our approach and benefits from our latent regret bounds.

We demonstrate the our approach on a small scale autonomous racecar MuSHR [11]
performing a household navigation task.

Prior Work | Noisy | Feedback Type
---|---|---
Feedback
Ross et al. [12] | $\times$ | Actions
Shivaswamy and Joachims [13] | $\checkmark$ | Actions*
Bajcsy et al. [8] | $\times$ | Trajectory Perturbations
Losey and O’Malley [14] | $\checkmark$ | Trajectory Perturbations
Christiano et al. [9] | $\checkmark$ | Binary Preferences
MacGlashan et al. [15] | $\times$ | Reward
Arumugam et al. [16] | $\times$ | Reward
Crammer and Gentile [17] | $\checkmark$ | Binary Correct/Incorrect Feedback
Cui and Niekum [18] | $\checkmark$ | Good/Bad Labelling Parts of Trajectories
Table 1: Taxonomy of related works categorized by two characteristics: whether
they model suboptimal or noisy feedback, and the feedback type. While many
algorithms model noisy feedback, none explicitly allow for multiple forms of
feedback. *Coactive Learning requires that the feedback can be converted into
action level feedback.

##  2 Related Work

Table 1 shows related work as a taxonomy. Each approach is designed for a
specific form of user feedback, and some approaches model user feedback as
noisy. Below we give a brief summary of each approach. We also note that our
approach does not fall under active learning [19] as the robot never queries
the teacher. Instead, the teacher corrects when they deem necessary.

DAgger (Ross et al. [12]) collects action level feedback from the expert for
every step and continually aggregates that feedback into its dataset for
retraining. This is a very effective approach when the user can provide
noiseless action level feedback. Additionally, it has been shown to be
effective for robotics.

Coactive Learning (Shivaswamy and Joachims [13]) is very similar to our
approach. We use a similar $\alpha$-informative noise model (see Section 3).
Coactive learning has been used successfully in robotics [10, 14]. We show in
Section 5 how Coactive Learning fits into our framework.

Bajcsy et al. [8] used trajectory perturbations to train a robotic manipulator
to follow human preferred trajectories. Perturbations were done through
physically pushing the manipulator. This work takes a Bayesian approach to
updating the model parameters. They arrive at a similar update rule as ours
and Coactive Learning.

Losey and O’Malley [14] builds on Coactive Learning by modeling uncertainty
over the user’s preferences via a Kalman filter. Similar to Bajcsy et al. [8],
they use trajectory perturbations but differ in that corrections are not
applied as the robot is executing, but after execution.

Christiano et al. [9] uses a form of preference learning [20] where they
present two options to the user to decide between. From repeated queries, they
build a reward function. In their feedback model, they assume that the human
provides a random response 10% of the time. In Section 5, We show how a simple
version of this preference learning approach can fit into our framework.

COACH/DeepCOACH (MacGlashan et al. [15], Arumugam et al. [16]) ask the user
for a reward signal and incorporate that reward using actor critic methods.
Crammer and Gentile [17] similarly use a simple correct/incorrect feedback in
multi-class classification. These methods highlight how human preferred
behaviors can be learned from simple feedback.

Cui and Niekum [18] takes an interesting approach to feedback and has the user
label parts of the trajectory as good or bad. These trajectory critiques are
used to create a reward function. They model noise by a confidence factor that
represents how confident they can be in the user’s feedback being accurate.
They recommend adjusting this value based on how consistent the feedback is
with previous feedback.

##  3 Corrective Feedback Meta-Algorithm

Assumption 1

Figure 2: Our first assumption, the pseudo regret upperbounds the latent
regret*. This is possible because we can make the pseudo loss have a higher
curvature than the true loss. *Similar to the true latent regret except regret
is relative to $\underset{a}{\operatorname*{arg\,min}}\;\Delta_{t}(a)$ instead
of $a^{*}$.

We present the Corrective Feedback Meta-Algorithm as an online interaction
between a learner and a teacher. At round $t$, the learner observes the state
$s_{t}\in\mathcal{S}$ and must select an action $a_{t}\in\mathcal{A}$. We
model the teacher’s preference via a loss function
$\overline{\Delta}_{t}:\mathcal{A}\rightarrow\mathbb{R}^{+}$ that assigns a
cost to each action. If the learner had access to this cost function, they
would select an optimal action $a_{t}^{*}$ via:

$a_{t}^{*}=\operatorname*{arg\,min}_{a}\overline{\Delta}_{t}(a)$ (1)

However, the teacher cannot provide $\overline{\Delta}_{t}(a)$. Instead the
teacher is compelled to provide feedback $z_{t}$ which we can convert into a
_pseudo-loss_ $\Delta_{t}(a)$ which the learner can observe. The pseudo-loss
is mismatched with the teacher’s true loss - the question is how does this
mismatch affect the regret bound w.r.t $\overline{\Delta}$? We generalize the
following assumptions from [13] regarding $\Delta_{t}(a)$.

We can assume the teacher’s true loss is bounded and since we have flexibility
in constructing the pseudo-loss using $z_{t}$ we can assume that the curvature
is much higher than the true loss. With a higher curvature, the pseudo-regret
upper bounds the regret relative to the minimum of the pseudo loss. This is
seen visually in Fig. 2.

$\underbrace{\Delta_{t}(a_{t})-\min_{a}\Delta_{t}(a)}_{\text{pseudo
regret}}\geq\underbrace{\overline{\Delta}_{t}(a_{t})-\overline{\Delta}_{t}(\operatorname*{arg\,min}_{a}\Delta_{t}(a))}_{\text{learner
versus the best pseudo action}}$ (2)

Additionally, we make the $\alpha$-informative assumption. It says - the best
action according to the _pseudo loss_ may not be the best action according to
the _teacher’s loss_ , but is definitely a step in the right direction.

$\underbrace{\overline{\Delta}_{t}(a_{t})-\overline{\Delta}_{t}(\operatorname*{arg\,min}_{a}\Delta_{t}(a))}_{\text{learner
versus the best pseudo action
}}\geq\underbrace{\alpha\left(\overline{\Delta}_{t}(a_{t})-\overline{\Delta}_{t}(a^{*}_{t})\right)}_{\text{learner
versus the best latent action}}$ (3)

Assumption 2

Figure 3: $\alpha$-informative assumption, the lower $\alpha$ is the more mis-
aligned the pseudo loss is with the true loss.

$\alpha$ represents the amount of noise in the teacher’s feedback, or more
specifically how the mis-aligned the pseudo loss is with the true loss, see
Fig. 3. When $\alpha=1$, the pseudo loss has no noise and the minimum of the
pseudo loss is the minimum of the true loss. As $\alpha$ approaches 0, the
pseudo loss becomes more and more mis-aligned with the true loss. Note, this
parameter is purely for theoretical analysis and cannot be calculated in
practice.

These assumptions place no requirements on the type of feedback, requiring
only that it can be converted to a pseudo loss. We show in section 4 how these
assumptions can be used to bound regret on the teacher’s latent loss.

##  4 Theoretical Analysis

We bound latent regret using assumptions 2 and 3. Assume we have a policy
class that maps weights $w$ and features $\phi_{t}(.)$ to actions, i.e.,
$a_{t}=\operatorname*{arg\,max}_{a}\pi(\phi_{t}(a),w)$. We can transform the
pseudo-loss over actions to a loss over weights

$\Delta_{t}(w)=\Delta_{t}(\operatorname*{arg\,max}_{a}\pi(\phi_{t}(a),w))$ (4)

$\Delta_{t}(w)$ is non-convex and problematic to deal with. Hence, we derive a
convex surrogate loss $\ell_{t}(w)\geq\Delta_{t}(w)$. Now we can bound regret
via online learning. Here are the steps:

  1. 1.

Use _any online learning algorithm_ to bound regret on surrogate loss

$\displaystyle\sum_{t=1}^{T}\ell_{t}(w_{t})-\min_{w^{*}}\sum_{t=1}^{T}\ell_{t}(w^{*})\leq\mathcal{O}(\sqrt{T})$
(5)
$\displaystyle\sum_{t=1}^{T}\ell_{t}(w_{t})\leq\min_{w^{*}}\sum_{t=1}^{T}\ell_{t}(w^{*})+\mathcal{O}(\sqrt{T})$

Let’s assume we have an expressive function class so
$\min_{w^{*}}\sum_{t=1}^{T}\ell_{t}(w^{*})=0$ or at the most
$\mathcal{O}(\sqrt{T})$.

  2. 2.

We can bound the pseudo loss on actions selected by the surrogate loss

$\displaystyle\sum_{t=1}^{T}\Delta_{t}(a_{t})=\sum_{t=1}^{T}\Delta_{t}(w_{t})\leq\sum_{t=1}^{T}\ell_{t}(w_{t})\leq\mathcal{O}(\sqrt{T})$
(6)

For simplicity, we can zero bias the vector $\min_{a}\Delta_{t}(a)=0$.

  3. 3.

Combining the above with assumptions (2) and (3), we get the following bound
on true regret.

$\displaystyle\sum_{t=1}^{T}\alpha(\overline{\Delta}_{t}(a_{t})-\overline{\Delta}_{t}(a^{*}))\leq\sum_{t=1}^{T}\Delta_{t}(a_{t})\leq\mathcal{O}(\sqrt{T})$
(7)
$\displaystyle\sum_{t=1}^{T}(\overline{\Delta}_{t}(a_{t})-\overline{\Delta}_{t}(a^{*}))\leq\frac{1}{\alpha}\mathcal{O}(\sqrt{T})$

This can hold for any choice of latent loss, function classes, and surrogate
functions that follow the assumptions (2) and (3).

##  5 Handling a Variety of Feedback

We demonstrate how our approach works with arbitrary feedback, and how
Coactive Learning and Preference Learning fit into our meta-algorithm. Note,
the presented cases serve as examples and are not representative of all the
function classes, feedback, and loss functions that can be handled within this
approach.

###  5.1 Linear Models & Arbitrary Feedback

We first apply our approach to linear models with arbitrary feedback. We
consider policies of the form:

$\displaystyle\pi_{w}(s)=\operatorname*{arg\,max}_{a\in\mathcal{A}}w^{\intercal}\phi(s,a)$

where $\mathcal{A}$ is a discrete set of $k$ actions, $s$ is a state, $w$ is a
weight vector, and $\phi$ is the feature vector. The policy may receive a
correction $z_{t}\in\mathcal{Z}$ from the user. For example, Dagger [12]
considers corrections in the form of alternative actions, although we also
consider more general forms of feedback. The feedback is indicative of the
quality of various actions. Recall the pseudo-loss: $\Delta_{t}(a)$. For the
remainder of this section, we will rewrite this to depend on the current state
instead of time $\Delta(s,a)$. To incorporate $z_{t}$, we will depend on the
following _user model_ :

  1. 1.

If the user explicitly provides a correction $z$, we define
$\Delta(s,a)=\xi(a,z)$ where $\xi$ is a user-defined similarity function
between the action and feedback spaces.

  2. 2.

If the action $a_{t}$ is “good enough”, the user does not correct, i.e.
$z=\varnothing$. We will model this by not applying a loss.

When feedback is given, we have a vector of losses $\Delta(s,\cdot)$ for all
actions. We upper bound this pseudo loss with the following surrogate convex
loss over weights called the _generalized hinge loss_ [21].

$\ell(w)=\max_{i=1,\dots,k}\Delta(s,a_{i})+w^{\intercal}(\phi(s,\hat{a})-\phi(s,a_{i}))$
(8)

where $\hat{a}=\operatorname*{arg\,min}_{a\in\mathcal{A}}\Delta(s,a)$ or the
best action according to the pseudo-loss. The subgradient of the loss is

$\nabla_{w}\ell(w)=\phi(s,\hat{a})-\phi(s,a_{\text{sel}})$ (9)

Where $a_{\text{sel}}$ is the action that achieves the maximum in (8). Since
this fits into the our approach, we enjoy the same regret bounds as in
equation (LABEL:eq:cum_regret).

###  5.2 Coactive Learning

We show that Coactive Learning also fits into our approach. Specifically, we
are looking at the Preference Perceptron algorithm from Shivaswamy and
Joachims [13] not considering the slack variable. The perceptron policy class
is defined as follows:

$\pi_{w}(s)=\operatorname*{arg\,max}_{a\in\mathcal{A}}w^{\intercal}\phi(s,a)$
(10)

They use a latent utility function that they are trying to maximize:

$\overline{\Delta}(s,a)=w_{*}^{T}\phi(s,a)$ (11)

where $w_{*}$ is the unknown teacher weights. Note,
$\pi_{w_{*}}(s)=\operatorname*{arg\,max}_{a\in\mathcal{A}}\overline{\Delta}(s,a)$.
Since the teacher’s actions (feedback) are a noisy representation of this
policy, we can define a pseudo 0-1 loss that we will minimize.

$\Delta(s,a)=\mathbb{I}(a\neq\hat{a})$ (12)

This upper bounds the negated objective:
$\pi_{w_{*}}(s)=\operatorname*{arg\,min}_{a\in\mathcal{A}}-\overline{\Delta}(s,a)$
where $\hat{a}$ is the expert provided action. Finally, we can upper bound
this loss with a convex surrogate.

$\Delta(s,a)\leq\ell(w)=1+w^{\intercal}(\phi(s,a)-\phi(s,\hat{a})))$ (13)

Note, this breaks our assumption
$\min_{w^{*}}\sum_{t=1}^{T}\ell_{t}(w^{*})\leq\mathcal{O}(\sqrt{T})$ and is
now bounded by $\mathcal{O}(T)$. This produces a latent regret bound of the
following:

$\sum_{t=1}^{T}(\overline{\Delta}_{t}(a_{t})-\overline{\Delta}_{t}(a^{*}))\leq\frac{1}{\alpha}\mathcal{O}(T+\sqrt{T})$
(14)

We can take advantage of this being a linear policy to tighten this vacuous
bound. If we take the derivative of $\ell_{t}(w)$ with respect to $w$, we
obtain the Coactive update rule.

$w_{t+1}=w_{t}+\phi(s,\hat{a})-\phi(s,a)$ (15)

Then, we can follow the Coactive Learning proof to obtain their regret bounds,
which are the same worst case as our latent regret, $\mathcal{O}(\sqrt{T})$.

###  5.3 Preference Learning

We show that preference learning also fits into our approach. We focus on the
simplest case, where the teacher is presented with two actions: the learner’s
choice, and another random choice. The teacher must choose which action they
prefer. Given a teacher’s choice, we can construct a pseudo-loss as follows:

$\Delta(s,a)=\left\lVert\phi(s,a_{p})-\phi(s,a)\right\rVert-\left\lVert\phi(s,a_{np})-\phi(s,a)\right\rVert$
(16)

where $a_{p}$ is the preferred action and $a_{np}$ is the not-preferred
action. This promotes actions that are similar to $a_{p}$ and simultaneously
dissimilar from $a_{np}$. Using this pseudo loss, we can use the generalized
hinge loss as in 5.1 and recover the same regret bounds.

Avoid Stairs $|$ Tolstoy

Avoid Stairs & Chairs $|$ Corozal

Avoid Doors $|$ Newfields

Figure 4: Our semantically labelled testing environments and teacher
trajectory. The gray line is the fixed path that the robot should follow.

##  6 Experimental Evaluation

To demonstrate our approach we perform simulated household navigation
experiments with the MuSHR Racecar Platform [11]. The task is to follow fixed
paths throughout various households per the teacher’s preferences. At each
timestep, the learner must choose from 64 actions represented as 1 meter long
trajectories from which a model predictive controller will calculate a
steering angle and velocity. For each trajectory, the feature function for our
linear policy $\phi(s,a)$ provides:

  * •

Sum of distances to obstacles along the trajectory.

  * •

Sum of distances to semantically labelled objects: stairs, chairs, and doors.

  * •

Sum of cross track and along track error for the trajectory relative to a
reference point on the path 1 meter ahead.

  * •

Relative lateral displacement from end of trajectory to robot. Trajectories
going straight are 0, ones to the right are positive, and ones to the left are
negative.

We use 3 environments from the Gibson Simulator [22]: Tolstoy, Newfields, and
Corozal. While we don’t allow the robot to choose trajectories in collision ,
if the robot gets into an unrecoverable state and collides, then we reset it
to the starting point maintaining its current weights.

We train from three programmatic teachers, one that wants to avoid doors,
another that wants to avoid stairs and stay to the right of the path, and
another that wants to avoid chairs and stairs. We choose these behaviors
because we expect a user would want the robot to avoid places from which
people could suddenly move from and lateral position is important for hallways
where it is convention to stay to a particular side. Programmatic teachers
allow us to calculate latent loss as a metric for learning. All teachers also
want the robot to follow the path provided. See Fig. 4 for each map and
teacher’s desired behavior.

We use several forms of feedback. We use online gradient descent to update our
linear policy and note that other optimization methods could learn with fewer
corrections. For all forms of feedback, we upperbound each pseudo loss with
the generalized hinge loss. Specifically, we use:

  * •

Action: Feedback is the desired teacher action from the 64 potential actions.
We use a pseudo loss where the teacher’s action is 0 and the others are 100.

  * •

Preference: The teacher is presented with two actions: the learner’s choice
and another random action. They choose which action they prefer more. We use
the pseudo loss described in Equation 16.

  * •

Semantic: This rough form of feedback is intended to simulate a teacher
telling the robot to “avoid doors” etc. Given the teacher’s and learner’s
actions, we determine if the teacher’s action features prefer/avoid an action
closer/further from semantic objects as compared to the learner. If they
prefer the object, then all actions that prefer the object more than the
learner’s choice are weighted 0 and all others 100 (visa-Versa for avoiding
objects). We also include the path (via cross track error) as a semantic
object to represent the teacher saying “stay on path”. Our pseudo loss is the
sum of all the semantic losses (doors, stairs, chairs, and path).

We Learn From a Variety of Feedback

Figure 5: Top Row: Latent loss over time. Note how all policies converge to a
low loss except for Avoid Stairs & Chairs. This is due to the teacher
threshold for correcting being too high. Bottom Row: Number of corrections
over time. Note how the order of convergence is different across tasks.

###  6.1 Learning from a Variety of Feedback

To demonstrate how our approach can learn from a variety of corrective
feedback, we test each form of feedback in three environments, where each
environment employs a different teacher as shown in Fig. 4. For comparison, we
also run behavior cloning trained on 50 samples of action feedback. Our
metrics for learning are the latent loss: $w^{T}_{*}(\phi(s,a)-\phi(s,a^{*}))$
and the number of corrections. The teacher corrects whenever the latent loss
is above 1.0. The value of this threshold translates to how picky the teacher
is. We investigate how setting this parameter affects convergence in Section
6.3. For each feedback in each environment, we conduct 10 trials. We use a
learning rate of 0.01. As learning progresses, we expected to see the latent
loss decline and the number of corrections to stop growing. Results are shown
in Fig. 5.

The first row of the results show the latent loss over time. In Avoid Doors,
all methods of feedback converge near zero in 5000 steps. Behavior cloning
remains constant because it does not receive additional training from the
initial first 50 datapoints. In Avoid Stairs & Stay Right and Avoid Stairs &
Chairs, we see that all methods converge with a latent loss around 20. This is
due to the teacher threshold being too high for the task. We know the
threshold is too high because below the latent loss plot, we see the number of
corrections over time flattening out, indicating that the teacher is
satisfied. Without corrections, these models cannot improve.

Looking at the number of corrections over time (second row), we see that
action feedback converges with the least number of corrections consistently,
suggesting this is the most informative form of feedback. We also note that
each task requires a different number of corrections, because the path and
environment are different between each. Interestingly, semantic and preference
feedback do not have a clear order across tasks. This finding shows that a
particular form of feedback is not always the best choice across tasks and
environments highlighting the need for a variety of feedback types.

###  6.2 Learning from Noisy Feedback

We Learn Under Noisy Feedback

Figure 6: Left is the latent loss over time for four different degrees of
noise and right is the associated number of corrections. Notice how with more
noise, more corrections are supplied.

0.0

1.0

2.0

Figure 7: Paths taken after training with three different standard deviations
of noise. Little difference in the quality of the run is seen between
$\sigma=0.0$ and $\sigma=1.0$ but at $\sigma=2.0$ the learner takes an
alternate path to the end, not following the task.

To test our approach under noisy teacher feedback, we performed an experiment
where we add Gaussian noise to the teacher’s feedback. To calculate each form
of feedback, we convert the latent loss into $\Delta(s,a)$ as described above.
We add Gaussian noise to the latent loss, which creates both noisy feedback
and a noisy frequency of corrections. We test with four standard deviations,
0.0 or no noise, 0.25, 0.5, and 1.0. We ran 10 trials on the Newfields
environment with the teacher Avoid Doors and a latent threshold of 1. We used
action feedback for this experiment.

As shown on the left in Fig. 6, our approach learns from noisy feedback.
Unsurprisingly with more noise, the latent loss will increase. To get a sense
of how the latent loss scales with more noise we performed a second experiment
in the same environment where we increased the noise and measured the latent
loss after 5000 steps. On the right in Fig. 6 we see how more corrections are
given with more noise. This is because noisier corrections provide the learner
with a less accurate loss to update from. To give context to these loss values
we provide paths taken by learner’s after training for three different noise
values in Fig. 7. While the latent loss between $\sigma=0.0$ and $\sigma=1.0$
is different, qualitatively the learner still does a good job at the task with
$\sigma=1.0$. At $\sigma=2.0$ the learner does not care for the doors at all,
but simply follows the path.

###  6.3 Effect of Different Teacher Thresholds

A human teacher’s threshold for correcting is unknown and different for each
task. Thus, we wanted to explore how different numerical thresholds affect the
results. We test 5 thresholds: 0.0, 0.25, 0.5, 0.75, and 1.0. A threshold of
0.0 means that the teacher will correct every time the learner chooses a
different action than the teacher’s. In contrast, a higher threshold can
result in the learner not converging to the desired policy. We test these
thresholds in the Corozal environment with the Avoid Stairs & Chairs teacher.
We run each threshold for 10 trials and present quantitative and qualitative
results for each in Fig. 8 and Fig. 9.

Different Thresholds Result in Different Behaviors

0.0

0.25

0.5

1.0

Figure 8: Learner’s paths given different correction thresholds. From left to
right, as the threshold increases the behavior gets further from the desired
behavior avoid stairs and doors.

Lower Thresholds Result in Lower Loss & More Corrections

Figure 9: Left: Latent loss for Avoid Stairs & Chairs for different latent
thresholds. Right: The corresponding number of corrections over time for each
threshold.

We see qualitatively that lower thresholds result in a behavior most similar
to the teacher’s desired behavior from Fig. 4. As the threshold increases, the
learner does not care to avoid the stairs or the chair and just focuses on
following the line. Quantitatively, we see a similar trend, lower thresholds
result in lower loss and more corrections. Ideally, a human corrector will
want to be in the middle ground, in our case, around 0.25 - 0.5 so they can
recover their desired policy while also not having to make too many
corrections.

##  7 Discussion & Future Work

In this work, we present the Corrective Feedback Meta-Algorithm, which can
learn from a variety of noisy feedback. For different tasks, teachers, and
environments, different forms of feedback will be necessary. This work builds
on prior work and creates a straightforward approach to encompass a variety of
feedback. Our approach is built off of the insight that the teacher’s policy
is latent and that their feedback can be modelled as a stream of loss
functions. We demonstrated this approach on three simulated household
navigation tasks with different teachers in each. Our findings show that our
method learns from both a variety of feedback and noisy feedback. While our
method can learn successfully, we also highlight that teacher’s who provide
more feedback obtain policies with lower loss.

We see two interesting directions for future work. First is to apply this
approach to a real world robotics scenario, with human teachers providing
corrections. This would provide insights regarding when user’s correct, and
what types of corrections users use. Second, we would like to investigate how
we can learn mappings from feedback to pseudo loss functions from prior
experience instead of encoding each feedback type manually. This would
significantly scale the variety of feedback the teacher could provide.

#### Acknowledgments

This work was (partially) funded by the National Institute of Health R01
(#R01EB019335), National Science Foundation CPS (#1544797), National Science
Foundation NRI (#1637748), the Office of Naval Research, the RCTA, Amazon, and
Honda Research Institute USA.

## References

  * Bagnell [2015] J. A. D. Bagnell.  An invitation to imitation.  Technical Report CMU-RI-TR-15-08, Carnegie Mellon University, Pittsburgh, PA, March 2015.
  * Osa et al. [2018] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, and J. Peters.  An algorithmic perspective on imitation learning.  _CoRR_ , abs/1811.06711, 2018.  URL http://arxiv.org/abs/1811.06711.
  * Billard and Grollman [2012] A. Billard and D. Grollman.  _Imitation Learning in Robots_ , pages 1494–1496.  Springer US, Boston, MA, 2012.  ISBN 978-1-4419-1428-6.  doi:10.1007/978-1-4419-1428-6_758.  URL https://doi.org/10.1007/978-1-4419-1428-6_758.
  * Sun et al. [2017] W. Sun, A. Venkatraman, G. J. Gordon, B. Boots, and J. A. Bagnell.  Deeply aggrevated: Differentiable imitation learning for sequential prediction.  _CoRR_ , abs/1703.01030, 2017.  URL http://arxiv.org/abs/1703.01030.
  * Kober and Peters [2009] J. Kober and J. Peters.  Learning motor primitives for robotics.  In _2009 IEEE International Conference on Robotics and Automation_ , pages 2112–2118, 2009.
  * Coates et al. [2009] A. Coates, P. Abbeel, and A. Y. Ng.  Apprenticeship learning for helicopter control.  _Commun. ACM_ , 52(7):97–105, July 2009.  ISSN 0001-0782.  doi:10.1145/1538788.1538812.  URL https://doi.org/10.1145/1538788.1538812.
  * Muxfeldt et al. [2014] A. Muxfeldt, J.-H. Kluth, and D. Kubus.  Kinesthetic teaching in assembly operations – a user study.  In D. Brugali, J. F. Broenink, T. Kroeger, and B. A. MacDonald, editors, _Simulation, Modeling, and Programming for Autonomous Robots_ , pages 533–544, Cham, 2014. Springer International Publishing.  ISBN 978-3-319-11900-7.
  * Bajcsy et al. [2017] A. Bajcsy, D. P. Losey, M. K. O’Malley, and A. D. Dragan.  Learning robot objectives from physical human interaction.  In S. Levine, V. Vanhoucke, and K. Goldberg, editors, _Proceedings of the 1st Annual Conference on Robot Learning_ , volume 78 of _Proceedings of Machine Learning Research_ , pages 217–226. PMLR, 13–15 Nov 2017.
  * Christiano et al. [2017] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei.  Deep reinforcement learning from human preferences.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_ , pages 4299–4307. Curran Associates, Inc., 2017.
  * Jain et al. [2015] A. Jain, S. Sharma, T. Joachims, and A. Saxena.  Learning preferences for manipulation tasks from online coactive feedback.  _The International Journal of Robotics Research_ , 34(10):1296–1313, 2015.  doi:10.1177/0278364915581193.  URL https://doi.org/10.1177/0278364915581193.
  * Srinivasa et al. [2019] S. S. Srinivasa, P. Lancaster, J. Michalove, M. Schmittle, C. Summers, M. Rockett, J. R. Smith, S. Chouhury, C. Mavrogiannis, and F. Sadeghi.  MuSHR: A low-cost, open-source robotic racecar for education and research.  _CoRR_ , abs/1908.08031, 2019.
  * Ross et al. [2011] S. Ross, G. Gordon, and D. Bagnell.  A reduction of imitation learning and structured prediction to no-regret online learning.  In G. Gordon, D. Dunson, and M. Dudík, editors, _Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics_ , volume 15 of _Proceedings of Machine Learning Research_ , pages 627–635, Fort Lauderdale, FL, USA, 11–13 Apr 2011. PMLR.
  * Shivaswamy and Joachims [2015] P. Shivaswamy and T. Joachims.  Coactive learning.  _Journal of Artificial Intelligence Research_ , 53:1–40, 05 2015.  doi:10.1613/jair.4539.
  * Losey and O’Malley [2018] D. P. Losey and M. K. O’Malley.  Including uncertainty when learning from human corrections.  In _2nd Annual Conference on Robot Learning, CoRL 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings_ , volume 87 of _Proceedings of Machine Learning Research_ , pages 123–132. PMLR, 2018\.  URL http://proceedings.mlr.press/v87/losey18a.html.
  * MacGlashan et al. [2017] J. MacGlashan, M. K. Ho, R. T. Loftin, B. Peng, D. L. Roberts, M. E. Taylor, and M. L. Littman.  Interactive learning from policy-dependent human feedback.  _CoRR_ , abs/1701.06049, 2017.  URL http://arxiv.org/abs/1701.06049.
  * Arumugam et al. [2019] D. Arumugam, J. K. Lee, S. Saskin, and M. L. Littman.  Deep reinforcement learning from policy-dependent human feedback.  _CoRR_ , abs/1902.04257, 2019.  URL http://arxiv.org/abs/1902.04257.
  * Crammer and Gentile [2011] K. Crammer and C. Gentile.  Multiclass classification with bandit feedback using adaptive regularization.  In _Proceedings of the 28th International Conference on International Conference on Machine Learning_ , ICML’11, page 273–280, Madison, WI, USA, 2011. Omnipress.  ISBN 9781450306195.
  * Cui and Niekum [2018] Y. Cui and S. Niekum.  Active reward learning from critiques.  _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 6907–6914, 2018.
  * Tong and Dept [2001] S. Tong and S. U. C. S. Dept.  _Active learning: theory and applications_.  Stanford University, 2001.  URL https://books.google.com/books?id=LZ5FAQAAIAAJ.
  * Frnkranz and Hllermeier [2010] J. Frnkranz and E. Hllermeier.  _Preference Learning_.  Springer-Verlag, Berlin, Heidelberg, 1st edition, 2010.  ISBN 3642141242.
  * Shalev-Shwartz and Ben-David [2014] S. Shalev-Shwartz and S. Ben-David.  _Understanding machine learning : from theory to algorithms_.  2014\.  ISBN 9781107057135 1107057132.  URL http://www.worldcat.org/search?qt=worldcat_org_all&q=9781107057135.
  * Xia et al. [2018] F. Xia, A. R. Zamir, Z.-Y. He, A. Sax, J. Malik, and S. Savarese.  Gibson env: real-world perception for embodied agents.  In _Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on_. IEEE, 2018.