# Intervention Aided Reinforcement Learning for Safe and Practical Policy
Optimization in Navigation

Fan Wang  {wang.fan, zhoubo01, chenke06, zhangxi13, lijiangyong01,
<EMAIL_ADDRESS>Bo Zhou  {wang.fan, zhoubo01, chenke06, zhangxi13,
lijiangyong01<EMAIL_ADDRESS>Ke Chen  {wang.fan, zhoubo01, chenke06,
zhangxi13, lijiangyong01<EMAIL_ADDRESS>Tingxiang Fan
<EMAIL_ADDRESS>Xi Zhang  {wang.fan, zhoubo01, chenke06,
zhangxi13, lijiangyong01<EMAIL_ADDRESS>Jiangyong Li  {wang.fan,
zhoubo01, chenke06, zhangxi13, lijiangyong01<EMAIL_ADDRESS>Hao Tian
{wang.fan, zhoubo01, chenke06, zhangxi13, lijiangyong01<EMAIL_ADDRESS>Jia Pan<EMAIL_ADDRESS>
###### Abstract

Combining deep neural networks with reinforcement learning has shown great
potential in the next-generation intelligent control. However, there are
challenges in terms of safety and cost in practical applications. In this
paper, we propose the Intervention Aided Reinforcement Learning (IARL)
framework, which utilizes human intervened robot-environment interaction to
improve the policy. We used the Unmanned Aerial Vehicle (UAV) as the test
platform. We built neural networks as our policy to map sensor readings to
control signals on the UAV. Our experiment scenarios cover both simulation and
reality. We show that our approach substantially reduces the human
intervention and improves the performance in autonomous navigation111A visual
demonstration of the learned policy is available at Youtube; Our code is based
on the PaddlePaddle Reinforcement Learning library(PARL), which is released in
Github, at the same time it ensures safety and keeps training cost acceptable.

> Keywords: Imitation Learning, Reinforcement Learning, Visual Navigation

##  1 Introduction

In contrast to the intense studies of deep Reinforcement Learning(RL) in games
and simulations [1], employing deep RL to real world robots remains
challenging, especially in high risk scenarios. Though there has been some
progresses in RL based control in realistic robotics [2, 3, 4, 5], most of
those previous works does not specifically deal with the safety concerns in
the RL training process. For majority of high risk scenarios in real world,
deep RL still suffer from bottlenecks both in cost and safety. As an example,
collisions are extremely dangerous for UAV, while RL training requires
thousands of times of collisions. Other works contributes to building
simulation environments and bridging the gap between reality and simulation
[4, 5]. However, building such simulation environment is arduous, not to
mention that the gap can not be totally made up.

To address the safety issue in real-world RL training, we present the
Intervention Aided Reinforcement Learning (IARL) framework. Intervention is
commonly used in many automatic control systems in real world for safety
insurance. It is also regarded as an important evaluation criteria for
autonomous navigation systems, e.g. the disengagement ratio in autonomous
driving222For details, see: Autonomous Vehicle Disengagement Reports. In this
work, instead of using disruptive events such as collisions as feedbacks, we
try to optimize the policy by avoiding human intervention. Though utilizing
human intervention to avoid fatal mistakes in RL training has been applied to
Atari games recently [6], our work is featured in real world applications.
Besides, we propose a more general framework to take human intervention into
account: We define the expression of the intervention as a combination of a
unknown classifier deciding when to seize control and a reference control
policy; We redefine the behavior policy as blending of the policy to be
optimized and the intervention; Then we try to reduce the probability of
intervention and learn from the reference policy at the same time.

In this work we are mainly interested in navigating an Unmanned Aerial Vehicle
(UAV) in cluttered and unstructured environments. The agent is required to
approach a specified goal at certain velocity without collision. Compared with
traditional navigation, the proposed navigation policy is based on a more end-
to-end architecture. The policy model takes multi-modal observations including
3D depth image, supersonic sensors, velocity, direction to the goal etc. as
input and performs attitude control. We compared IARL with different learning
algorithms including classic imitation learning and reinforcement learning in
simulation, and tested IARL in the realistic UAV navigation task. We
demonstrate that IARL learn to avoid collision nearly perfectly compared with
the other baselines, even though it has never seen a single collision in the
training process. In the meantime, IARL substantially reduces the human
intervention more than the other methods. Furthermore, our method can be
applied to various autonomous control systems in reality such as autonomous
driving cars.

##  2 Related Works

There has been abundant works on visual navigation system[7], among which our
work is closely related to map-less reactive visual navigation systems [8, 9].
Traditional reactive controllers either rely on artificially designed rules,
such as boundary following, which highly adapts to specific obstacles [10, 8],
or use empirical models such as Artificial Potential Fields (APF [11, 12]),
which builds virtual potential fields for collision avoidance. The above works
requires artificial tuning of a lot of hyper-parameters and hand-written
rules. Some of those methods show competitive performance in specific
scenarios, but they are not easy to be extended to unstructured scenarios.

Imitation Learning takes an important role in intelligent controllers in
recent years, where policy is learned by reducing the difference between
itself and a reference policy(typically the human demonstration). Among them,
the supervised learning (or behavior cloning) generates the training
trajectories directly from the reference policy[13, 14]. It is relatively
simple, but the performance is usually unstable due to the inconsistency
between the training and testing trajectories. Ross et al. proposed the Data
Aggregation(DAgger [15]) as an improvement to supervised learning, by
generating trajectories with the current policy. Their subsequent work [16]
successfully employed DAgger to collision-free navigation on an UAV. Yet,
DAgger remains expensive, and it requires a tricky labeling process that is
hard to operate in practice.

Our work is also related to recent progresses in reinforcement learning,
especially the deterministic policy gradient dealing with continuous control
[17, 18, 19]. We use Proximal Policy Optimization(PPO) [19] with General
Advantage Estimation [20](GAE) as baseline, which has been widely accepted for
its robustness and effectiveness. However, most RL algorithms can not be
directly deployed to the high risk scenarios due to the safety issue, thus we
made the comparison in simulation only. In addition, we also use the trick of
combining imitation learning and reinforcement learning to accelerate the
training process, which is closely related to the previous work of [21, 22].

##  3 Algorithms

###  3.1 Backgrounds

In this part we introduce the notations as well as the representations of
different benchmarking algorithms. A Markov Decision Process(MDP) is described
by $(\mathcal{S},\mathcal{A},\mathcal{R},\mathcal{P})$, where $\mathcal{S}$
and $\mathcal{A}$ represent the state and action space, $\mathcal{R}$
represents the rewards, which is typically the real number field, and
$\mathcal{P}$ represents the transition probability. We use the notation
$t\in[0,T]$ to represent the time step, where $T$ is the length of an episode.
$s_{t}\in\mathcal{S}$, $a_{t}\in\mathcal{A}$ and $r_{t}\in\mathcal{R}$ denote
the specific state, action and reward at step $t$. We use the notation $\mu$
to represent the _deterministic_ policy that we want to optimize. We have
$a_{t}=\mu_{\theta}(s_{t})$, with $\theta$ representing the trainable
parameters to be optimized. A more general _stochastic_ policy is defined as a
distribution over $\mathcal{S}\times\mathcal{A}$. Usually a Gaussian
distribution $\mathcal{N}$ is used for stochastic policy in continuous action
space, which can be written as
$\pi_{\theta}(a|s_{t})=\mathcal{N}(\mu_{\theta}(s_{t}),\sigma)$. $\sigma$ can
either be a hyper-parameter or be optimized all-together with the other
trainable parameters.

Imitation Learning. The behavior cloning is a straightforward approach in
imitation learning. If we use the mean square error for the target
function(loss function), it can be written as

$L^{SL}(\theta)\propto\mathbb{E}_{\pi^{ref}}[\sum_{t\leq
T}(\mu_{\theta}(s_{t})-a^{ref}_{t})^{2}],\vspace*{-.1in}$ (1)

where $a^{ref}_{t}\in\pi^{ref}$ denotes the reference action, the notation
$\mathbb{E}_{\pi}$ represent the expectation over trajectories that follow
policy $\pi$(the behavior policy). Usually the data collection and the policy
updating are separate and independent in such supervised learning process,
which leads to inconsistency between training and testing. The DAgger proposed
in [15] substitutes the behavior policy $\pi^{ref}$ with $\pi_{\theta}$ to
alleviate the discrepancy. We omit the loss function, as it is similar to
Equation. 1.

Policy Gradient. We use the similar formulations of policy gradients as
Schulman et al. [20], where the target function $L_{PG}$ is written as:

$L^{PG}(\theta)\propto-\mathbb{E}_{\pi_{\theta}}[\sum_{t\leq
T}\Psi_{t}\log{\pi_{\theta}(a_{t}|s_{t})}].$ (2)

There are different expressions for $\Psi_{t}$ in Equation 2.
$\Psi_{t}=\sum_{0\leq\tau\leq T}r_{\tau}$ gives the REINFORCE [23]. The Actor-
Critic method [24] uses one-step temporal difference error that is given by
$\Psi_{t}^{(1)}=r_{t}+\gamma V^{\pi}_{\phi}(s_{t+1})-V^{\pi}_{\phi}(s_{t})$,
where $V^{\pi}_{\phi}$ represents the discounted value function on policy
$\pi$, and $\gamma$ denotes the discount factor. The GAE([20] is represented
by
$\Psi_{t}^{GAE}\propto\sum_{t\leq\tau<T}(\gamma\lambda)^{\tau-t}\Psi_{\tau}^{(1)}$.

The calculation of $\Psi_{t}$ mentioned above requires on-policy generation of
trajectories. In case external intervention is introduced, the policy
optimization in Equation 2 can not be directly applied. In the next subsection
we reformulate the policy optimization paradigm with intervention.

###  3.2 Intervention Aided Reinforcement Learning

We assume that the expert in the IARL system is represented by
$(\pi^{ref},\mathcal{G})$, where $\mathcal{G}(s)\in[0,1]$ is the possibility
of intervening the system when the expert observes the agent being at state
$s$. Notice that both $\pi^{ref}$ and $\mathcal{G}$ is decided by the expert
himself, which is not known in advance to the agent. The expert samples a
decision $g_{t}\in\\{0,1\\}$ according to $\mathcal{G}(s_{t})$ at time $t$,
determining whether the agent is intervened eventually. We write the actual
action as $a^{mix}_{t}=g_{t}\cdot a^{ref}_{t}+(1-g_{t})\cdot a_{t}$ with
$g_{t}\sim Bernoulli(\mathcal{G}(s_{t}))$. $a^{mix}_{t}$ is regarded to be
sampled from the mixed policy of the experts and the agent, which is denoted
as $\pi^{mix}_{\theta}$ and given by:

$\pi^{mix}_{\theta}(a_{t}|s_{t})=\mathcal{G}(s_{t})\pi^{ref}(a_{t}|s_{t})+(1-\mathcal{G}(s_{t}))\pi_{\theta}(a_{t}|s_{t}).$
(3)

Note that the behavior policy (the policy under which the data is collected)
in IARL is $\pi^{mix}_{\theta}$, not $\pi_{\theta}$. In order to approximate
$\Psi_{t}$ on-policy, we optimize $\theta$ through the mixed policy
$\pi^{mix}_{\theta}$ instead of $\pi_{\theta}$.

In order to prevent intervention, we also reshape the reward function in IARL.
We treat intervention as failures, and reshape the reward by punishing the
intervention $g_{t}$ with

$r^{mix}_{t}=r_{t}-g_{t}\cdot b,$ (4)

where $b$ is a predefined hyper-parameter. Notice that all the collisions have
been prevented by the expert intervention, thus the agent learns to avoid
collisions only from avoiding the intervention. By replacing $r_{t}$ with
$r^{mix}_{t}$ and $\pi_{\theta}$ with $\pi^{mix}_{\theta}$, the derivative of
policy loss in Equation 2 to $\theta$ is written as Equation 5. Note that any
derivation of the reference policy $\pi^{ref}$ vanishes, and we use $K(s,a)$
to replace
$(1-\mathcal{G}(s))\frac{\pi_{\theta}(a|s)}{\pi^{mix}_{\theta}(a|s)}$.

$\begin{split}&\frac{\partial
L^{PG}(\theta)}{\partial(\theta)}\propto-\mathbb{E}_{\pi^{mix}_{\theta}}[\sum_{t\leq
T}\Psi_{t}\frac{K(s_{t},a^{mix}_{t})}{\pi_{\theta}(a^{mix}_{t}|s_{t})}\frac{\partial\pi_{\theta}(a^{mix}_{t}|s_{t})}{\partial(\theta)}]\end{split}$
(5)

We can see that $K(s,a)\in[0,1]$ and $K(s,a)=0$ if $\mathcal{G}(s)=1$, and
$K(s,a)=1$ if $\mathcal{G}(s)=0$. $K(s,a)$ can be interpreted as a “mask” that
blocks the gradient to $\pi_{\theta}$ when $\mathcal{G}(s)=1$. For those
states that $\mathcal{G}(s)=1$, the action is intervened by $\pi^{ref}$ and
the trajectory is not determined by $\pi_{\theta}$, thus the gradient
vanishes. However, $\mathcal{G}(s_{t})$ as well as $\pi^{mix}_{\theta}$ cannot
be directly approximated, and thus $K(s_{t},a^{mix}_{t})$ is not known
exactly. We turn to use a simple approximation of
$K(s_{t},a^{mix}_{t})\approx(1-g_{t})$. The error of the approximation is
small when $g(s_{t})$ is close to $0$ or $1$; it indeed may result in large
errors in other cases, but these errors turns out to be acceptable in our
experiments. We rewrite the loss of IARL as:

$\begin{split}&L^{\text{IARL}}(\theta)\propto-\mathbb{E}_{\pi^{mix}_{\theta}}[\sum_{g_{t}=0}\Psi_{t}\log{\pi_{\theta}(s_{t},a^{mix}_{t})}],\end{split}$
(6)

In Equation 6 the summation is running over steps satisfying $g_{t}=0$ only,
while the steps with $g_{t}=1$ are completely “masked”. Notice that the value
function still backup through all states to give $\Psi_{t}$, thus Equation 6
tends to avoid the intervention beforehand. However, Equation 6 does not
improve the performance of the policy $\pi_{\theta}$ once the intervention has
taken place (e.g., when $\mathcal{G}(s_{t})$ is close to 1). This is harmful
to the robustness of policy $\pi_{\theta}$, as $\pi_{\theta}$ is likely to
malfunction in the states of “intervened zone”. To alleviate the defects, we
introduce an additional imitation loss into Equation 6. The imitation loss
matches the optimizing policy $\pi_{\theta}$ to the reference policy
$\pi^{ref}$, which leads to:

$\begin{split}&L^{\text{IARL-
IL}}(\theta)\propto-\mathbb{E}_{\pi^{mix}_{\theta}}[\sum_{g_{t}=0}\Psi_{t}\log{\pi_{\theta}(a^{mix}_{t}|s_{t})}+\alpha\sum_{g_{t}=1}||\mu_{\theta}(s_{t})-a^{ref}_{t}||^{2}].\end{split}$
(7)

An intuitive explanation of Equation 7 is that the first term in the right
side avoids intervention beforehand, and the second term imitates the
reference policy $\pi^{ref}$ in the “intervened zone”. The hyper-parameter
$\alpha$ balances the importance of the two parts. The training process of
IARL is presented in Algorithm 1. It can be easily extended to PPO, we leave
those details in the supplements. We call Equation 6 IARL (No Imitation) and
Equation 7 IARL (Imitation).

Algorithm 1 Intervention Aided Reinforcement Learning

1:Initialize $\phi$ for value function $V_{\phi}$, $\theta$ for policy
$\pi_{\theta}$ and $\pi^{mix}_{\theta}$.

2:for iterations = 1,2,3,… until convergence do

3: for episodes = 1,2,3,… until enough data collected do

4: for t = 1,2,3,… until finished do

5: Observe $s_{t}$, receive $g_{t}$ from the expert

6: if $g_{t}=1$ then receive $a^{ref}_{t}$ from the expert,
$a^{mix}_{t}=a^{ref}_{t}$

7: else $a^{mix}_{t}\sim\pi_{\theta}$

8: Take action $a^{mix}_{t}$

9: Receive $r_{t}$, push the record $(s_{t},a^{mix}_{t},r_{t},g_{t})$ to
memory

10: Optimize $\theta$ by minimizing $L^{IARL}$ for several epochs

11: Update $V_{\phi}$ using Temporal Difference or Monte Carlo estimate

##  4 System Setup

In order to validate IARL, we build the UAVs and the control systems in both
simulation and reality. The configurations in both environments are similar to
each other. In this section we briefly introduce the architecture of the
control system and the neural network structure of the decision model(the
policy).

###  4.1 Control Architecture

The overall control system includes the decision model to be optimized, two
assistant Proportional-Integral-Derivative (PID) controllers and an underlying
double closed loop PID controller. The input to our system includes image
flows captured by binocular stereo vision, supersonic sensors in its three
directions (left, right and front) and Inertial Measurement Unit(IMU).

The command of pitch and roll angle of the UAV is decided by the policy model.
We use two additional controller to deal with the yaw and the thrust, which
keeps the drone at a fixed height and heading angle. The central controller
sends the full attitude commands to the underlying control module, which
controls to the attenuators of the motor. The remote controller is able to
override those commands at any time if a switch is turned on by the human
expert, which corresponds to the $g$ in Algorithm 1. A sketch of the
controller architecture can be found in Figure 1. The output of the policy
model is calculated at a frequency of 10Hz, and the underlying controllers
exports at the frequency of 400Hz.

Figure 1: Diagram of the architecture of the controllers on the drone.

###  4.2 Model Structure

Our model does not use raw signals as the input directly. Instead, we
introduce several preprocessing steps. The IMU and stereo cameras are combined
and preprocessed to generate velocities and positions as well as the depth
information. We use an open-source SDK 333The boteye SDK:
https://github.com/baidu/boteye. We also calculate the velocity and the
direction to the goal from raw signals and use them as the observation,
details of the features used can be found in the supplements.

The policy network $\pi_{\theta}$ and value network $V_{\phi}$ share the same
model structure but have different output and independent parameters. For each
frame of depth image we apply four convolution-pooling layers. The hidden
representation is then concatenated with the other features. Traditional
reactive controller such as APF typically uses only observation of one single
frame. However, the real system is more close to a Partially Observable MDP
due to the noisy sensor signals. It is found that multiple frames of
observations are helpful to reducing the noise and making up the missing
information in each single observation. We build a model that is similar to
the architecture reported by Hausknecht et al. [25]. The LSTM unit [26] is
applied to encode the observations from last 2.5 seconds in order. We apply a
tanh layer to restrict the output to $[-1,+1]$, which corresponds to the
pitch/roll angle of $\pm 0.1rad$. The structure of the model is shown in
Figure 2. We also investigated the performance of two other different models.
We used a fixed dataset collected from the simulation using reference policy,
which contains 1 million frames of training data and 50,000 frames of testing
data. The mean square error(MSE) performance of different structures are
compared: The single observation (2D convolution + MLP) gives MSE = $0.02+$;
Stacking multiple frames (3D convolution + MLP) gives MSE = $0.0066$;
Recurrent networks (2D convolution + LSTM) gives MSE = $0.0044$. The test
result demonstrates the superiority of the proposed model structure.

Figure 2: A sketch of the policy model structure.

##  5 Experiments

###  5.1 Simulation Environments

Simulator. We use the Robot Operation System (ROS) and Gazebo 7 [27] for
simulation. The drone is simulated using the open-source hector-quad-rotor
package on ROS [28] . Some of the kinetic hyper-parameters are re-adjusted to
match our drone in reality. An Intel Realsense sensor is attached to the
simulated drone to represent the stereo vision. In the simulation, the
velocity and position are directly acquired from the simulator instead of
being calculated from the raw observation.

Environments. The simulation region is $10m\times 10m$ in scale. The drone is
required to start from the start point of ($x=5$m, $y=-1$m) and find its way
to the goal on top of the region ($y=11$m). The height and yaw angle of the
UAV is kept unchanged. We manually designed three kinds of obstacles in Gazebo
7, including $1m\times 3m$ walls, $3m\times 3m$ walls, and buckets of $0.7m$
diameter and $1.5m$ height. We randomly scattered a certain amount of
obstacles in the region, the scenarios without a valid inner path between the
starting point and the goal is dropped. 100 different scenarios were finally
generated with different random seeds. 90 of them were treated as training
scenes, while the left 10 were the testing scenes.

Expert simulation. Training with real human operators in simulation is a
relatively expensive choice. In order to further reduce the cost, we build a
simulated expert, which gives $\mathcal{G}$ and $\pi^{ref}$ automatically. We
use ground truth information provided by Gazebo 7 to simulate the expert,
which is hidden from the agents. The details of experts simulation are left in
the supplements. The simulated expert help to avoid around $95\%$ collision
during IARL training. In order to make a fair comparison, we want the IARL
agent to only learn from intervention. Thus we dropped the left $5\%$ collided
episodes in the IARL training process.

Reward. The basic reward function for RL and IARL is set to
$r=\max(5+20v_{d}-50v^{2},0.0)$, where $v_{d}$ is the projection of the
velocity in the direction of the goal. The term of $20v_{d}$ encourages the
drone toward the desired goal. The term $50v^{2}$ is to slow down the drone to
avoid risky accelerations. The maximum reward shall be achieved if the drone
moves at the speed of $0.20$m/s towards the goal. The collision with any
obstacle triggers the failure of an episode, which terminates the episode and
gives additional $-25$ punishment. If no collision is detected, the episode
terminates when arriving the goal or reaching at most $600$ steps($60$
seconds). In IARL, we use $b=5$ for the intervention punishment such that
$r^{mix}=r-5g_{t}$.

Evaluation. We perform two kinds of tests for each comparing method and each
test scenario: The _Normal Test_ evaluates $\pi_{\theta}$ by removing the
external interference $\pi^{ref}$, $R_{\text{NT}}=\sum r_{t}$ is used as the
criteria. The _Intervention Aided Test_ keeps the external interference
$\pi^{ref}$. $R_{\text{IAT}}=\sum r^{mix}_{t}$ is used as the criteria.
Another criteria in _Intervention Aided Test_ is called the intervention rate
(IR), defined by $\textbf{IR}=(\sum_{t}g_{t})/(\sum_{t}1)$. This evaluation
metric is similar to the disengagement rate evaluation in autonomous vehicle.
All our evaluation results are averaged among the $10$ test scenarios.

###  5.2 Methods for comparison

SL. We collect 1 million frames under policy $\pi^{ref}$ in 90 training
scenarios in advance. The policy is optimized with Adam and Equation 1 for 50
epochs. The test is performed only once.

DAgger. The behavior policy is set to $\pi_{\theta}$. The $\pi^{ref}$ is
recorded during the training process. For every 50,000 frames, the data is
collected to aggregate the current dataset. The aggregated dataset is used to
train the policy $\pi$ using Equation 1. The optimization includes 20 epochs
of training (one iteration). After each iteration we perform tests using
$\pi_{\theta}$. We run this process until 1 million frames.

RL. The RL group uses PPO ([19]) with GAE ([20]), and use collision as its
feedback. Every iteration contains 10,000 frames of data and 20 epochs of
training. The test is performed after each training epoch. We ran the training
process up to 1 million frames.

IARL. The policy is optimized using Algorithm 1, the training trajectories
include no collision. Both IARL(No Imitation) and IARL(Imitation) are tested.
The periods of training and testing is kept equal to the RL group.

###  5.3 Simulation Results and Discussions

We compare each of the four methods in both _Normal Test_ and _Intervention
Aided Test_. The performance against trained frames is plotted in Figure 3.
Several remarks can be made from the results. The IARL(No Imitation) group
shows completely different performance with different evaluation methods: In
the _Intervention Aided Test_ where the reward and the behavior policy of the
training and the testing are coherent, it achieves acceptable performance in
its rewards and IR; In _Normal Test_ where the intervention is removed, the
IARL(No Imitation) group collides frequently with obstacles and results in a
low performance score. On the other side, the IARL(Imitation) surpasses all
the other groups in each test, especially in the IR evaluation.

We count the average collision rate (the collided cases out of 10 test
scenarios) over the last 5 evaluations in the _Normal Test_ , Tab. 1 presents
the result. The IARL(Imitation) reduces its collision to 0 steadily. While the
other groups suffer from occasional collision. Notice that in the
_Intervention Aided Test_ the value of IR never dropped to 0, as we found that
the intervention might overreact in some cases. In other words, the expert
intervenes even when the drone could have avoided collision all by itself.

We also plot the trajectories of RL, IARL(Imitation) and DAgger in Figure 4. A
close look at the differences between the IARL and the other two groups shows
that the IARL agent is more likely to choose a conservative trajectory. In
other words, it is more likely to keep enough distance away from the
obstacles, while the RL one pursues more risky trajectories for higher
rewards. Besides, we can see that the DAgger and the IARL agent successfully
explore a path out of a blind alley(the trajectories in the middle figure),
where the RL agent fails to escape(it stops, without any collision).
Nevertheless, overstepping the local optimum by exploring around is nearly
impossible for reactive controllers that relies on single frame observation
only.

(a)

(b)

(c)

Figure 3: Comparison of SL, DAgger, RL, IARL(No Imitation) and IARL(Imitation). (a) averaged reward $R_{\text{NT}}$ in the _Normal Test_. (b) averaged reward $R_{\text{IAT}}$ in the _Intervention Aided Test_. (c) averaged IR in the _Intervention Aided Test_. Table 1: Average collision in _Normal Test_ in different groups Method | SL | DAgger | RL | IARL(No Imitation) | IARL(Imitation)
---|---|---|---|---|---
Average Collision | 40% | 24% | 22.5% | 42% | 0%
Figure 4: Comparison of trajectories of IARL(Imitation)(Yellow Cubes), RL(Red
Spheres), DAgger(Blue Crosses) in three different testing scenarios.

###  5.4 Experiments in Reality

Environments. The drone in real world is a X type quad-rotor helicopter with
38cm size and 1.6kg weight(Figure 5). It is equipped with Nvidia Jetson TX2 as
the onboard computer, stereo vision, supersonic sensors, IMU and barometer.
All computations are completed on board. For simplicity, the environment is
composed of two kinds of different obstacles, which are built from foam
boards, including trigonometric and square pillars(Figure 5).

Start-up Policy. In order to validate IARL in reality with more acceptable
cost, we first collected 100,000 frames of expert manipulation in different
cluttered scenarios. We then took advantage of the policy and the value
network trained in simulation. We fine-tuned the policy on the labeled data
for certain epochs. This model is further used as the start-up policy for IARL
training.

IARL Training. During the training process, a human operator keeps watching on
the drone and intervening with remote controller when necessary based on his
own judgment. The training environments are manually readjusted randomly after
several passes. Each iteration(training round) includes 5000 frames of
data(around 20 passes) and 20 epochs of training. Extra 2 different scenarios
are built and used for test. Similar to the evaluation in the simulations, we
apply the _Intervention Aided Test_ in the two test scenarios every 20,000
frames, but the _Normal Test_ is not available in reality due to the safety
issue.

Results. The IR in each test round is plotted in Figure 6. At the same time,
we plot the average pass time(defined as the total time consumed from the
start point to the goal) and average intervention time(defined as the length
of time that $g_{t}=1$). It is worth mentioning that the overall performance
is not as good as the simulation(where IR dropped below $1\%$). There can be
two reasons: Firstly, the signals acquired from real sensors are much more
noisy, especially the depth image; Secondly, there are considerable delay
between the observation and the decision making, which is mainly caused by the
stereo matching and visual odometer. The delay can go up to 0.3s or 0.4s. Even
with those disadvantages, we can see that our model learned to substantially
reduce the IR and the intervention time.

Figure 5: A close-up image of the Drone(left) and three frames during the
test. The picture above is captured by external camera, the image below shows
the corresponding input and output of the decision model.

(a)

(b)

Figure 6: Different evaluation criteria against the training frames in the
IARL training. (a) average pass time and average intervention time. (b) IR.

##  6 Conclusion

In this paper, we propose the IARL framework that improves the policy by
avoiding and imitating the external intervention at the same time. We
successfully deploy our method to UAV platform to achieve map-less collision-
free navigation in the unstructured environments. The experimental results
show that IARL achieves satisfying collision avoidance and low intervention
rate, and ensures safety in the training process. By looking into the future
of this technique, we anticipate that IARL will serve as an effective policy
optimization metric in various high risk real-world scenarios.

## References

  * Duan et al. [2016] Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel.  Benchmarking deep reinforcement learning for continuous control.  In _International Conference on Machine Learning_ , pages 1329–1338, 2016.
  * Kober et al. [2013] J. Kober, J. A. Bagnell, and J. Peters.  Reinforcement learning in robotics: A survey.  _IJRR_ , 32(11):1238–1274, 2013.
  * Levine et al. [2016] S. Levine, C. Finn, T. Darrell, and P. Abbeel.  End-to-end training of deep visuomotor policies.  _Journal of Machine Learning Research_ , 17(39):1–40, 2016.
  * Rusu et al. [2016] A. A. Rusu, M. Vecerik, T. Rothörl, N. Heess, R. Pascanu, and R. Hadsell.  Sim-to-real robot learning from pixels with progressive nets.  _arXiv preprint arXiv:1610.04286_ , 2016.
  * Zhu et al. [2017] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi.  Target-driven visual navigation in indoor scenes using deep reinforcement learning.  In _ICRA_ , pages 3357–3364, 2017.
  * Saunders et al. [2018] W. Saunders, G. Sastry, A. Stuhlmueller, and O. Evans.  Trial without error: Towards safe reinforcement learning via human intervention.  In _Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems_ , pages 2067–2069. International Foundation for Autonomous Agents and Multiagent Systems, 2018.
  * Bonin-Font et al. [2008] F. Bonin-Font, A. Ortiz, and G. Oliver.  Visual navigation for mobile robots: A survey.  _Journal of intelligent and robotic systems_ , 53(3):263, 2008.
  * Oleynikova et al. [2015] H. Oleynikova, D. Honegger, and M. Pollefeys.  Reactive avoidance using embedded stereo vision for mav flight.  In _ICRA_ , pages 50–56, 2015.
  * Syre Wiig et al. [2017] M. Syre Wiig, K. Y. Pettersen, and T. R. Krogstad.  A reactive collision avoidance algorithm for vehicles with underactuated dynamics.  In _CDC_ , pages 1452–1459, 2017.
  * Matveev et al. [2013] A. S. Matveev, M. C. Hoy, and A. V. Savkin.  The problem of boundary following by a unicycle-like robot with rigidly mounted sensors.  _Robotics and Autonomous Systems_ , 61(3):312–327, 2013.
  * Lacroix et al. [1998] S. Lacroix, S. Fleury, H. Haddad, M. Khatib, F. Ingrand, G. Bauzil, M. Herrb, C. Lemaire, and R. Chatila.  Reactive navigation in outdoor environments.  In _ICRA_ , pages 1232–1237, 1998.
  * Vadakkepat et al. [2000] P. Vadakkepat, K. C. Tan, and W. Ming-Liang.  Evolutionary artificial potential fields and their application in real time robot path planning.  In _Evolutionary Computation_ , volume 1, pages 256–263, 2000.
  * Liu et al. [2017] G.-H. Liu, A. Siravuru, S. Prabhakar, M. Veloso, and G. Kantor.  Learning end-to-end multimodal sensor policies for autonomous navigation.  _arXiv:1705.10422_ , 2017.
  * Bojarski et al. [2016] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, et al.  End to end learning for self-driving cars.  _arXiv:1604.07316_ , 2016.
  * Ross et al. [2011] S. Ross, G. Gordon, and D. Bagnell.  A reduction of imitation learning and structured prediction to no-regret online learning.  In _AISTATS_ , pages 627–635, 2011.
  * Ross et al. [2013] S. Ross, N. Melik-Barkhudarov, K. S. Shankar, A. Wendel, D. Dey, J. A. Bagnell, and M. Hebert.  Learning monocular reactive uav control in cluttered natural environments.  In _ICRA_ , pages 1765–1772, 2013.
  * Lillicrap et al. [2015] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra.  Continuous control with deep reinforcement learning.  _arXiv:1509.02971_ , 2015.
  * Schulman et al. [2015] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz.  Trust region policy optimization.  In _ICML_ , pages 1889–1897, 2015.
  * Schulman et al. [2017] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov.  Proximal policy optimization algorithms.  _arXiv:1707.06347_ , 2017.
  * Schulman et al. [2015] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel.  High-dimensional continuous control using generalized advantage estimation.  _arXiv:1506.02438_ , 2015.
  * Rajeswaran et al. [2017] A. Rajeswaran, V. Kumar, A. Gupta, J. Schulman, E. Todorov, and S. Levine.  Learning complex dexterous manipulation with deep reinforcement learning and demonstrations.  _arXiv preprint arXiv:1709.10087_ , 2017.
  * Levine and Koltun [2013] S. Levine and V. Koltun.  Guided policy search.  In _International Conference on Machine Learning_ , pages 1–9, 2013\.
  * Williams [1992] R. J. Williams.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  In _Reinforcement Learning_ , pages 5–32. Springer, 1992.
  * Sutton and Barto [1998] R. S. Sutton and A. G. Barto.  _Reinforcement learning: An introduction_ , volume 1.  MIT press Cambridge, 1998.
  * Hausknecht and Stone [2015] M. Hausknecht and P. Stone.  Deep recurrent q-learning for partially observable mdps.  _arXiv:1507.06527_ , 2015.
  * Hochreiter and Schmidhuber [1997] S. Hochreiter and J. Schmidhuber.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780, 1997.
  * Koenig and Howard [2004] N. Koenig and A. Howard.  Design and use paradigms for gazebo, an open-source multi-robot simulator.  In _IROS_ , volume 3, pages 2149–2154, 2004.
  * Meyer et al. [2012] J. Meyer, A. Sendobry, S. Kohlbrecher, U. Klingauf, and O. von Stryk.  Comprehensive simulation of quadrotor uavs using ros and gazebo.  In _SIMPAR_ , page to appear, 2012.

##  S7 Supplementary Materials

###  S7.1 The Value Function Approximation

In IARL, the approximation of value function $V_{\phi}(s)$ is optimized by
minimizing the mean square error loss in Equation 8.

$L^{V}(\phi)=\mathbb{E}_{\pi^{mix}_{\theta}}[\sum_{t}(V_{\phi}(s_{t})-\hat{V}_{t})^{2}]$
(8)

There are different estimation metrics for $\hat{V}_{t}$. For example,
considering the IARL paradigm, $\hat{V}_{t}=\sum_{\tau\geq
t}\gamma^{\tau-t}r^{mix}_{\tau}$ represents the Monte Carlo(MC) estimation,
and $\hat{V}_{t}=\gamma V_{\phi^{\prime}}(s_{t+1})+r^{mix}_{t}$ represents the
temporal difference estimation, where $\phi^{\prime}$ is the parameters of the
target network. In this work we use n-step return estimation shown in Equation
9.

$\hat{V}_{t}=\sum_{t\leq\tau\leq
T}\gamma^{\tau-t}r^{mix}_{\tau}+\gamma^{T-t+1}V(s_{T+1})$ (9)

The difference between Equation 9 and the MC estimation is that we backup the
value function of the last state $s_{T+1}$, which is to avoid the possible
bias brought by the finite horizon.

###  S7.2 Extending IARL(Imitation) to PPO

The policy loss of PPO based IARL is written as Equation 10. $\theta^{\prime}$
denotes the parameter $\theta$ from the last iteration. $\beta$ is dynamically
controlled by another hyper-parameter $d_{targ}$. For more details please
refer to [19].

$\begin{split}&L^{IARL-IL-
PPO}=-\mathbb{E}_{\pi^{mix}_{\theta^{\prime}}}[\sum_{g_{t}=0}\Psi_{t}\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta^{\prime}}(a_{t}|s_{t})}-\alpha\sum_{g_{t}=1}(a^{ref}_{t}-\mu_{\theta}(s_{t}))^{2}\\\
&\qquad-\beta\sum_{g_{t}=0}KL(\pi_{\theta^{\prime}}(\cdot|s_{t})||\pi_{\theta}(\cdot|s_{t}))]]\end{split}$
(10)

###  S7.3 Features of Observation

At each time step, the policy and value networks take 25 frames of observation
and encode the observations using LSTM unit. Each frame of observation is
composed of the features listed in Table. S2

Table S2: Features in each frame of observation Description |  Dimension
---|---
Depth Images, acquired from stereo matching |  $45\times 30$
Velocities in UAV body coordinate system, acquired from Visual Odometer, discretized independently in x and y directions |  $20\times 2$
Distance reported by supersonic sensors, discretized |  $20\times 3$
Angle of direction to the goal, discretized |  $20$

###  S7.4 Details of The Expert Simulation

Simulating experts in Gazebo 7 includes $\pi^{ref}$ and $\mathcal{G}$, which
represent the expert’s policy and intervention probability respectively.

$\pi^{ref}$: For each new scenario, we discretized the map into $300\times
300$ grids. We restrict that the UAV moves towards the neighboring 8 grids.
The cost of moving to the neighboring grid is defined as the distance to the
center of the grid. We then mask all the grids that intersect with the
obstacles, and ran Dijkstra’s algorithm on the discretized map to acquire the
cost to the goal for each grid. During each time step of the training or
testing process, starting from the current position, we plan a trajectory by
setting magnitude of velocity to 0.20m/s and following the direction to the
grid of lowest cost. We used the next three planned positions on the current
planned trajectory as the target. We defined the loss as the mean square error
between the predicted positions and the target positions. We applied the Model
Predictive Control to give $\mu^{ref}$.

$\mathcal{G}$: We collect a small amount of data in advance to build the
$\mathcal{G}$ simulator. Driving the UAV with a RL pre-trained policy(but not
yet converged, i.e. an imperfect policy), we required the experts to keep UAV
safe by intervening the UAV when necessary using keyboard in Gazebo7
simulation. We collect 50,000 frames of data. Each sample has the following
features

  * •

The roll and pitch angle of the UAV

  * •

The velocity of the UAV in its body coordinate system

  * •

Top-10 nearest obstacles, represented by its obstacle type, and the position
of the point that is closest to the UAV. The position is calculated in the UAV
body coordinate system.

Each sample is labeled with $g_{t}$. We train a Gradient Boosting Decision
Trees(GBDT) model with cross-entropy loss, which gives the $\mathcal{G}$.

###  S7.5 Hyper-parameters

Table S3: List of the hyper-parameters in RL and IARL Hyper-parameter |  Value |  Description
---|---|---
$\gamma$ |  0.96 |  Discount factor in RL, IARL
$\lambda$ |  0.98 |  Exponential decay weight in generalized advantage estimation(GAE, [20]) in RL and IARL
Learning rate(policy) |  1.0e-4 |  Adam optimizer
Learning rate(value) |  1.0e-3 |  Adam optimizer
Mini-batch size(value and policy) |  32 |  Mini-batch size in RL, IARL and SL
Iteration size |  1.0e-3 |  Collect this many data before training in each iteration in RL and IARL
Action repeat |  4 |  Repeat each action selected by the agent this many times in RL and IARL
$d_{targ}$ |  0.003 |  $d_{targ}$ in PPO([19]), for RL and IARL
$\alpha$ |  2.0 |  The ratio of imitation loss and policy loss in IARL
$\sigma^{2}$ |  1.8 |  The initial exploration noise in policy $\pi_{\theta}$, $\sigma$ is optimized together with $\theta$ during the training process in RL and IARL.