# Zero-Shot Uncertainty-Aware Deployment of Simulation Trained Policies on
Real-World Robots

Krishan Rana
QUT Centre for Robotics, Brisbane
<EMAIL_ADDRESS>&Vibhavari Dasagi
QUT Centre for Robotics, Brisbane
<EMAIL_ADDRESS>&Jesse Haviland
QUT Centre for Robotics, Brisbane
<EMAIL_ADDRESS>&Ben Talbot
QUT Centre for Robotics, Brisbane
<EMAIL_ADDRESS>&Michael Milford
QUT Centre for Robotics, Brisbane
<EMAIL_ADDRESS>&Niko Sünderhauf
QUT Centre for Robotics, Brisbane
<EMAIL_ADDRESS>
###### Abstract

While deep reinforcement learning (RL) agents have demonstrated incredible
potential in attaining dexterous behaviours for robotics, they tend to make
errors when deployed in the real world due to mismatches between the training
and execution environments. In contrast, the classical robotics community have
developed a range of controllers that can safely operate across most states in
the real world given their explicit derivation. These controllers however lack
the dexterity required for complex tasks given limitations in analytical
modelling and approximations. In this paper, we propose Bayesian Controller
Fusion (BCF), a novel uncertainty-aware deployment strategy that combines the
strengths of deep RL policies and traditional handcrafted controllers. In this
framework, we can perform zero-shot sim-to-real transfer, where our
uncertainty based formulation allows the robot to reliably act within out-of-
distribution states by leveraging the handcrafted controller while gaining the
dexterity of the learned system otherwise. We show promising results on two
real-world continuous control tasks, where BCF outperforms both the standalone
policy and controller, surpassing what either can achieve independently. A
supplementary video demonstrating our system is provided at
https://bit.ly/bcf_deploy.

##  1 Introduction

As the adoption of autonomous robotic systems increases around us, there is a
need for the controllers driving them to exhibit the level of sophistication
required to operate in our everyday unstructured environments. Recent advances
in reinforcement learning (RL) coupled with deep neural networks as function
approximators, have shown impressive results across a range of complex robotic
control tasks including dexterous in-hand manipulation [1], quadrupedal
locomotion [10], and targeted throwing [9]. Nevertheless, the lack of safety
guarantees in deep RL-based controllers limits their usability for real-world
robotics [15]. This comes as a result of the black-box nature of neural
network policies and their inability to reliably deal with out-of-distribution
states, particularly seen when simulation-trained models are transferred to
the real world.

On the contrary, the classical robotics community have yielded numerous
controllers and algorithms for a range of real-world physical systems (from
mobile robots to humanoids) that allow us to reliably and safely deploy
robotic agents in the real world. These include classical feedback controllers
[25], trajectory generators [16] and behaviour trees [5]. This is attributed
to their explicit analytic derivation and known models, allowing for control
theoretic guarantees which make them suitable for real-world deployment. They
however can be highly suboptimal when applied to complex tasks, due to
limitations in analytical modelling and approximations.

A promising direction for the future of robot control is in combining the
complementary strengths of these different control mechanisms in order to
address their respective limitations. Such approaches have been observed in
neuroscience, as the underlying control strategy used by biological systems.
The dual-process theory of decision making [6] proposes that multiple
different neural controllers are involved when controlling action selection in
biological systems. Lee et al. [23] provide evidence for this theory based on
the human brain and show the existence of an arbitration mechanism that
determines the extent to which the different neural controllers govern
behaviours. The arbitrator bases its selection on specific performance
measures exhibited by each controller, exploiting their respective strengths
in a given state.

We draw inspiration from this observation and present Bayesian Controller
Fusion (BCF), a hybrid control strategy that combines the respective strengths
of deep RL and traditional handcrafted controllers (control priors) for safe
real-world deployment. We formulate the final policy as a Bayesian composition
of these two controllers, where the controller output for each system captures
their respective epistemic uncertainty in a given state as shown in Figure 1.
This allows BCF to naturally arbitrate control between the two systems based
on their confidence to act. This has important implications during real-world
deployment, where we gain the dexterity of the learned system in states that
it has generalised to while relying on the risk-averse behaviours of the
handcrafted system in out-of-distribution states for safe operation.
Importantly, our method learns to control a real robot in joint space to
complete a given task with no on-robot time (zero-shot), even though the
learned policy may not be perfectly transferable from simulation to the real
world.

We demonstrate our approach on two continuous control, robotic tasks involving
reactive navigation on a mobile robot and a manipulability maximising reacher
on a robotic arm. We show how BCF allows us to reliably transfer a simulation
trained policy to the real world while gaining significant performance
improvements from the RL component when compared to the handcrafted system
alone. We see this as a promising and practical avenue to bringing simulation-
trained RL controllers to safely operate in the real world.

Figure 1: Bayesian Controller Fusion (BCF): a hybrid control strategy for safe
deployment on real robotic systems. We derive uncertainty-aware action outputs
for each controller and compose these outputs to better inform the action
selection process.

##  2 Related Work

Safe sim-to-real transfer has been an active research area, particularly in
robotics where the cost of training robots directly in the real world is high.
Many prior works have focused on developing realistic simulation environments
that represent the real world as close as possible [37, 32] or directly build
a training environment from real-world data [4]. While such approaches attempt
to provide realistic environments, there are still a wide range of states and
variations that are not captured by the training environment. Several works
have generated robust policies using domain randomisation, where the agent is
exposed to a wide range of environmental variations allowing it to generalise
to changing environments. In the case of physical interaction with the
environment, dynamics randomisation of simulated robots has also been used to
capture the intricacies of real-world robots and their environments [30].
Recent works have also utilised a meta-RL approach where the agent’s dynamics
are adapted online in the real world [2, 27]. While all these approaches do
produce increasingly robust policies for real-world operation, they still fail
to capture the vast range of potential states and physical intricacies that
the agent may encounter in the real world, limiting their safe operational
space.

As opposed to attempting to replicate the real world within the simulation
environment, recent works have explored the ability of the agent to reason
about the current state and utilise this as a proxy for decision making.
Osband et al. [28] and Gal and Ghahramani [7] first explored the idea of state
uncertainty estimation from neural networks to assist exploration during
training. Kahn et al. [20] extended these ideas to the robotics setting to
enable the agent to predict its state uncertainty as it acts within the real
world. This allowed it to move slowly to avoid high-speed collisions while
increasing its velocity within parts of the space where it had greater
confidence in its actions. In our work, we explore an approach to avoid
collisions altogether by switching to a safer controller. Other works learn a
predictive model for catastrophic states using supervised learning [24], as
well as a "backup policy" to return the agent from a critical state to a safe
regime [13]. The operation of the robot is however constrained to the states
that the predictor was trained on. As opposed to predictive models, we rely on
epistemic uncertainty estimates directly from an ensemble of trained policies
to identify unknown states. This removes the need to learn from labelled
states or the restriction of operation within specific domains. More closely
related to our work, Garcia and Fernández [8] utilise a distance-based risk
estimator to identify out-of-distribution states and utilise this to switch
between the learned policy and safe baseline controller. The switching however
is abrupt and can result in jittery behaviours unsuitable for controlling real
robots. In contrast, we formulate our controller as a composition of
stochastic controllers allowing the resulting hybrid controller to smoothly
interpolate between behaviours.

A growing area of interest is the combination of learning with traditional
control strategies given their inherent safety guarantees. Bansal et al. [3]
decouple the control from the perception module and learn an obstacle-free
way-point finder that can be used by a low-level optimal controller for
navigation. They show that this works well in the sim-to-real setting given
the decomposition, however, the system is heavily reliant on the accuracy of
the way-point prediction model. In our work we consider the uncertainty of our
trained model to better inform the decision process. Rana et al. [31] leverage
the Residual RL framework [18, 33] to learn a dexterous reactive navigation
controller and utilise uncertainty estimates of the residual policy to govern
whether its action output is used to augment the behaviours of the underlying
controller in the real world or not. The abrupt switching behaviour however
resulted in noisy control signals not ideal for continuous control tasks.
Julian et al. [19] learn a range of skill competencies for a task and use a
model predictive control (MPC) approach to forward simulate each of these
skills in the simulation environment before executing the best action in the
real world. While such hybrid controllers enable continuous and safe operation
of learned policies in the real world, they come with a considerable
computation overhead that can be detrimental to real-time operation. Our
Bayesian fusion formulation allows us to directly leverage uncertainty
estimates to govern the best action for execution between controllers at a
given state, allowing for reactive and real-time control.

##  3 Approach

We introduce Bayesian Controller Fusion (BCF), a hybrid control strategy that
composes stochastic action outputs from two separate control mechanisms: an RL
policy $\pi(a|s)$, and a control prior $\psi(a|s)$. These outputs are
formulated as distributions over actions, where each distribution captures the
relative state uncertainty for the system to act. The Bayesian composition of
these two outputs forms our hybrid policy $\phi(a|s)$.

Figure 1 illustrates our hybrid control strategy that composes the outputs
from the learned policy $\pi(a|s)$ and control prior $\psi(a|s)$. The
uncertainty-aware compositional policy $\phi(a|s)$ allows for the safe
deployment of learned controllers. In states of high uncertainty, the
compositional distribution naturally biases towards the reliable, risk-averse
and potentially suboptimal behaviours suggested by the control prior. In
states of lower uncertainty, it biases towards the policy, allowing the agent
to exploit the optimal behaviours discovered by it. This is reminiscent of the
arbitration mechanism suggested by [23] for behavioural control in the human
brain, where the most confident controller assumes control in a given
situation. This dual-control perspective provides a reliable strategy for
bringing RL to real-world robotics, where generalisation to all states is near
impossible and the presence of a risk-averse control prior serves as a
reliable fallback.

###  3.1 Method

Given a policy, $\pi$ and control prior, $\psi$, we can obtain two independent
estimates of an executable action, $a$, in a given state. In a Bayesian
context, we can utilise the normalised product to fuse these estimates under
the assumption of a uniform prior, $p(a)$:

$\displaystyle p\left(a\mid
s,\mathbf{\theta}_{\pi},\mathbf{\theta}_{\psi}\right)=\frac{p\left(s,\mathbf{\theta}_{\pi},\mathbf{\theta}_{\psi}\mid
a\right)p(a)}{p\left(s,\mathbf{\theta}_{\pi},\mathbf{\theta}_{\psi}\right)}.$
(1)

We assume Gaussian distributional outputs from each system and represent
$\pi(a|s)\approx p\left(a\mid s,\theta_{\pi}\right)$ and $\psi(a|s)\approx
p\left(a\mid s,\theta_{\psi}\right)$, where,
$\theta_{\pi}=\\{[\mu_{\pi_{1}},...,\mu_{\pi_{n}}]^{\intercal},[\sigma_{\pi_{1}},...,\sigma_{\pi_{n}}]^{\intercal}\\}$
and
$\theta_{\psi}=\\{[\mu_{\psi_{1}},...,\mu_{\psi_{n}}]^{\intercal},[\sigma_{\psi_{1}},...,\sigma_{\psi_{n}}]^{\intercal}\\}$
denote the distribution parameters for the policy and control prior outputs
respectively and $n$ is the dimensionality of the action space. We drop the
state $s$, to simplify notation.

Assuming statistical independence of $p\left(\mathbf{\theta}_{\pi}\mid
a\right)$ and $p\left(\mathbf{\theta}_{\psi}\mid a\right)$, we can expand our
likelihood estimate, $p\left(\mathbf{\theta}_{\pi},\mathbf{\theta}_{\psi}\mid
a\right)$, as follows:

$\displaystyle p\left(\mathbf{\theta}_{\pi},\mathbf{\theta}_{\psi}\mid
a\right)$ $\displaystyle=p\left(\mathbf{\theta}_{\pi}\mid
a\right)p\left(\mathbf{\theta}_{\psi}\mid a\right)$
$\displaystyle=\frac{p\left(a\mid\mathbf{\theta}_{\pi}\right)p\left(\mathbf{\theta}_{\pi}\right)}{p(a)}\frac{p\left(a\mid\mathbf{\theta}_{\psi}\right)p\left(\mathbf{\theta}_{\psi}\right)}{p(a)}.$
(2)

Substituting this result back into (1), we can simplify the fusion as a
normalised product of the respective action distributions from each control
mechanism:

$\displaystyle
p\left(a\mid\mathbf{\theta}_{\pi},\mathbf{\theta}_{\psi}\right)=\eta\underbrace{p\left(a\mid\mathbf{\theta}_{\pi}\right)}_{\begin{subarray}{c}\text{Policy}\end{subarray}}\underbrace{p\left(a\mid\mathbf{\theta}_{\psi}\right)}_{\begin{subarray}{c}\text{Control}\\\
\text{Prior}\end{subarray}},$ (3)

where,

$\displaystyle\eta=\frac{p\left(\mathbf{\theta}_{\pi}\right)p\left(\mathbf{\theta}_{\psi}\right)}{p\left(\mathbf{\theta}_{\pi},\mathbf{\theta}_{\psi}\right)p(a).}$
(4)

The composite distribution
$p\left(a\mid\mathbf{\theta}_{\pi},\mathbf{\theta}_{\psi}\right)$ forms our
hybrid policy output $\phi(a|s)$. As we approximate the distributional output
from each system to be univariate Gaussian for each action, the composite
distribution $\phi(a|s)$ will also be univariate Gaussian
$\phi(a|s)\sim\mathcal{N}(\mu_{\phi},\sigma^{2}_{\phi})$. As a result, we can
compute the corresponding mean $\mu_{\phi}$ and variance $\sigma^{2}_{\phi}$
for each action as follows:

$\mu_{\phi}=\frac{\mu_{\pi}\sigma_{\psi}^{2}+\mu_{\psi}\sigma_{\pi}^{2}}{\sigma_{\psi}^{2}+\sigma_{\pi}^{2}},$
(5)

$\sigma_{\phi}^{2}=\frac{\sigma^{2}_{\pi}\sigma_{\psi}^{2}}{\sigma_{\psi}^{2}+\sigma_{\pi}^{2}},$
(6)

where this expansion implicitly handles the normalisation constant $\eta$.

###  3.2 Components

In order to leverage our proposed approach in practice, we describe the
derivation of the distributional action outputs for each system below and
provide the complete BCF algorithm for combining these systems in Algorithm 1.

####  3.2.1 Uncertainty-Aware Policy

We leverage stochastic RL algorithms that output each action as an independent
Gaussian
$\pi^{\prime}(a|s)\sim\mathcal{N}(\mu_{\pi^{\prime}},\sigma_{\pi^{\prime}}^{2})$
where $\mu_{\pi^{\prime}}$ denotes the mean and $\sigma^{2}_{\pi^{\prime}}$
denotes the corresponding variance. This distribution is optimised to reflect
the action which would both maximise the returns from a given state, as well
as the entropy [12]. Such exploration distributions tend to be risk seeking
and do not capture the state uncertainty of the agent. The latter is a key
component required for our BCF formulation. To attain an uncertainty-aware
distribution, we leverage epistemic uncertainty estimation techniques
suggested in the computer vision literature based on ensemble learning [22].
We train an ensemble of $M$ agents to form a uniformly weighted Gaussian
mixture model, and combine these predictions into a single univariate Gaussian
whose mean and variance are respectively the mean, $\mu_{\pi}(s)$ and
variance, $\sigma^{2}_{\pi}(s)$ of the mixture,
$p(a|s,\theta_{\pi})=M^{-1}\sum_{m=1}^{M}p\left(a|s,\theta_{\pi^{\prime}_{m}}\right)$
as described in [22]. The mean and variance of the mixture
$M^{-1}\sum\mathcal{N}\left(\mu_{\pi^{\prime}_{m}}(s),\sigma_{\pi^{\prime}_{m}}^{2}(s)\right)$
are given by:

$\mu_{\pi}(s)=M^{-1}\sum_{m}\mu_{\pi^{\prime}_{m}}(s)$ (7)

$\sigma_{\pi}^{2}(s)=M^{-1}\sum_{m}\left(\sigma_{\pi^{\prime}_{m}}^{2}(s)+\mu_{\pi^{\prime}_{m}}^{2}(s)\right)-\mu_{\pi}^{2}(s)$
(8)

The empirical variance, $\sigma_{\pi}^{2}(s)$, of the resulting output
distribution, $p(a|s,\theta_{\pi})$ approximates a measure of the policy’s
epistemic uncertainty in a given state for a particular action. This allows
for a broader distribution when presented with unknown states and a tighter
distribution in familiar states. This plays an important role within our BCF
formulation as described previously. Note that we fuse the distributional
polices as opposed to just their means to prevent collapse to a deterministic
system once they converge. This allows the agent to continue to explore
alternate actions and identify better solutions.

####  3.2.2 Control Prior

In order to incorporate the inherently deterministic control priors developed
by the robotics community within our stochastic RL framework, we require a
distributional action output that captures its uncertainty to act in a given
state. As the uncertainty is state-centric, we empirically derive this action
distribution by propagating noise (provided by the known sensor model
variance, $\sigma^{2}_{\text{model}}$) from the sensor measurements through to
the action outputs using Monte Carlo (MC) sampling. By computing the mean,
$\mu_{\psi}$ and variance, $\sigma_{\psi}$ of the outputs, the distributional
action output, $\mathcal{N}(\mu_{\psi},\sigma_{\psi}^{2})$ for a given state,
$s$ is given by:

$\mu_{\psi}(s)=N^{-1}\sum_{n}a_{\psi}(s_{MC_{n}}),\hskip
20.0pts_{MC}\sim\mathcal{N}(s,\sigma_{\text{model}}^{2}),$ (9)

$\sigma_{\psi}^{2}(s)=N^{-1}\sum_{n}(a_{\psi}(s_{MC_{n}})-\mu_{\psi}(s))^{2},$
(10)

where $a_{\psi}(\cdot)$ denotes a deterministic action output from the control
prior for a given MC sample and $N$ is the number of sampled states. Given the
inherent robustness to noise of most control priors, we additionally set a
minimum possible standard deviation for the distribution. This prevents the
control prior distribution from collapsing to a deterministic value, rendering
the policy useless within the BCF formulation. The resulting variance for the
control prior distribution is defined as:

$\displaystyle\sigma_{\psi}^{2}(s)=\max\left(N^{-1}\sum_{n}(a_{\psi}(s_{MC_{n}})-\mu_{\psi}(s))^{2},\sigma_{d}^{2}(s)\right).$
(11)

The choice of $\sigma^{2}_{d}$ is left as a hyper-parameter for the user to
set based on the specific controller used and its optimality towards solving
the task.

1 Given: Ensemble of M policies
($[\pi^{\prime}_{1},\pi^{\prime}_{2}...\pi^{\prime}_{M}]$), control prior
($\pi_{\psi}$) and variance ($\sigma_{d}^{2}$)

Input: State $\textit{s}_{t}$

Output: Action $\textit{a}_{t}$

2 Approximate the policy ensemble predictions as a unimodal Gaussian
$\pi(\cdot|s_{t})\sim\mathcal{N}(\mu_{\pi},\sigma_{\pi}^{2})$ described in
Equations (7) and (8)

3

4Compute the control prior action distribution
$\psi(\cdot|s_{t})\sim\mathcal{N}\left(\mu_{\psi},\sigma_{\psi}^{2}\right)$ as
given in Equations (11) and (9)

5

6Compute the composite distribution
$\phi(\cdot|s_{t})\sim\mathcal{N}(\mu_{\phi},\sigma^{2}_{\phi})$

7

$\phi(\cdot|s_{t})=\eta(\pi(\cdot|s_{t})\cdot\psi(\cdot|s_{t}))$ as given in
Equations (5) and (6)

8

9Select action $\textit{a}_{t}$ from the distribution $\phi(\cdot|s_{t})$

10

return _$\textit{a}_{t}$_

Algorithm 1 Bayesian Controller Fusion

Given the formulation for the distributional outputs from each system, we
present the complete BCF algorithm for governing action selection both during
training and deployment in Algorithm 1.

Figure 2: Simulation training environments and real world deployment
environments for (a) PointGoal Navigation and (b) Maximum Manipulability
Reacher tasks. Note the stark discrepancy in obstacle profiles for the
navigation task between the simulation environment and real world
environments.

##  4 Experiments

In this section, we assess the ability of BCF to reliably control real-world
robots with a simulation trained policy and without any on-robot fine-tuning.
We evaluate our approach on two continuous control robotics tasks: target
driven navigation, and a manipulability maximising reacher task. We provide a
detailed description of these tasks in Appendix A.1. We additionally compare
BCF to its individual learned and handcrafted components in isolation in order
to understand its ability to exploit their respective strengths. We provide
details of the evaluated systems below.

  1. 1.

Control Prior: The deterministic classical controller derived using analytic
methods.

  2. 2.

SAC: RL policy trained using vanilla Soft Actor Critic (SAC) [11].

  3. 3.

BCF: Our proposed hybrid control strategy that combines uncertainty aware
outputs from the control prior and the learned RL policy. The agent was
trained using SAC and Algorithm 1 for action selection during exploration.

Note that all the policies used in this evaluation were trained to convergence
in the simulation environments. We provide a detailed account for each task in
the following sections.

###  4.1 PointGoal Navigation

In this experiment, we examine whether BCF could overcome the limitations of
an existing reactive navigation controller, in this case, an Artificial
Potential Fields (APF) controller [34, 21], while leveraging this control
prior to safely deal with out-of-distribution states that the policy could
fail in. The APF controller used exhibited suboptimal oscillatory behaviours
particularly in between obstacles.

Table 1: Evaluation of PointGoal Navigation in the Real World

| Trajectory 1 | Trajectory 2
---|---|---
Method |  | Distance Travelled
---
(meters)
| Actuation Time
---
(seconds)
| Distance Travelled
---
(meters)
| Actuation Time
---
(seconds)
Control Prior | 42.3 | 274 | 35.3 | 277
SAC | Fail | Fail | Fail | Fail
Move-Base | 62.6 | 263 | 35.8 | 258
BCF | 41.2 | 135 | 30.4 | 117

Figure 3: Trajectories taken by the real robot for different start and goal
locations in a cluttered office environment with long narrow corridors. The
trajectory was considered unsuccessful if a collision occurred. The trajectory
taken by BCF is colour coded to represent the uncertainty in the linear
velocity of the trained policy. We illustrate the behaviour of the fused
distributions at key areas along the trajectory. The symbols 1 and 2 indicate
the start locations for each trajectory and G indicates the corresponding goal
locations.

####  4.1.1 Evaluation

We utilise a GuiaBot mobile robot which is equipped with a 180∘ laser scanner,
matching that used in the simulation environment. The velocity outputs from
the policies are scaled to a maximum of
$0.25\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$ before execution on the robot at
a rate of 100 Hz. The system was deployed in a cluttered indoor office space
that was previously mapped using the laser scanner. We utilise the ROS AMCL
package to localise the robot within this map and extract the necessary state
inputs for the policy network and control prior. Despite having a global map,
the agent is only provided with global pose information with no additional
information about its operational space. The environment also contained
clutter which was unaccounted for in the mapping process. To enable large
traversals through the office space, we utilise a global planner to generate
target sub-goals, for our reactive agents to navigate towards. We report the
distance travelled by each controller and compare them to the distance
travelled by a fine-tuned ROS move-base controller. This controller is not
necessarily the optimal solution but serves as a practical example of a
commonly used controller on the Guiabot.

The evaluation was conducted on two different trajectories indicated as
Trajectory 1 and 2 in Figure 3 and Table 1. Trajectory 1 consisted of a lab
space with multiple obstacles, tight turns, and dynamic human subjects along
the trajectory, while Trajectory 2 consisted of narrow corridors never seen by
the robot during training. We terminated a trajectory once a collision
occurred and marked the run as a failed attempt. We summarise the results in
Table 1.

Across both trajectories, the standalone SAC agent failed to complete a
trajectory without any collisions, exhibiting sporadic reversing behaviours in
out-of-distribution states. We can attribute these behaviours to its poor
generalisation in such states, given the discrepancies in obstacle profiles
seen during training in simulation and those encountered in the real world as
shown in Figure 2 (a). The control prior was capable of completing all
trajectories however required excessive actuation times. We can attribute this
to its inefficient oscillatory motion when moving through passageways and in
between obstacles. BCF was successful across both trajectories exhibiting the
lowest actuation times across all methods. This indicates its ability to
exploit the optimal behaviours learned by the agent while ensuring it did not
act sporadically when presented with out-of-distribution states. It also
demonstrates superior results when compared with the fine-tuned ROS move-base
controller.

To gain a better understanding of the reasons for BCF’s success when compared
to the control prior and SAC agent acting in isolation, we examine the
trajectories taken by these systems as shown in Figure 3. The trajectory
attained using BCF is colour-coded to illustrate the uncertainty of the
policy’s actions as given by the outputs of the ensemble. We draw the readers
attention to the region marked A which exhibits higher values of policy
uncertainty. The composition of the respective distributions at this region is
shown within the orange ring. Given the higher policy uncertainty at this
point, the resulting composite distribution was biased more towards the
control prior which displayed greater certainty, allowing the robot to
progress beyond this point safely. We note here that this is the particular
region that the SAC agent failed as shown in Figure 3 (c). The purple ring at
region C illustrates a region of low policy uncertainty with the composite
distribution biased closer towards the policy. Comparing the performance
benefit over the control prior gained in such a case, we draw the readers
attention to regions B and D which show the path profile taken by the
respective agents. The dense darker path shown by the control prior indicates
regions of high oscillatory behaviour and significant time spent at a given
location. On the other hand, we see that BCF does not exhibit this and attains
a smoother trajectory which is attributed to the learned policy having higher
precedence in these regions, stabilising the oscillatory effects of the
control prior. This illustrates the ability of BCF to exploit the relative
strengths of each component throughout deployment.

###  4.2 Maximum Manipulability Reacher

We evaluate the ability of BCF to build upon the basic structure provided by a
Resolved Rate Motion Controller (RRMC) [35], for reaching on a 7 DoF arm
robot, in order to learn a more complex manipulability maximising reaching
controller. While RRMC provides the policy with the basic knowledge to reach a
goal, the agent has to learn how to modify the individual velocities of each
joint in order to maximise the manipulability of the controller. More details
on the tasks are provided in Appendix A.1.2.

Table 2: Evaluation of Maximum Manipulability Reacher in the Real World

Method | Average Manipulability | Average Final Manipulability | Success Rate
---|---|---|---
Control Prior | 0.0629 $\pm$ 0.00926 | 0.0658 $\pm$ 0.0165 | 98.2%
SAC | 0.0803 $\pm$ 0.00514 | 0.07812 $\pm$ 0.0150 | 78.6%
BCF | 0.0836 $\pm$ 0.0156 | 0.0889 $\pm$ 0.0177 | 98.2%

Figure 4: Manipulability and uncertainty curves for known and out-of-
distribution goals for the reacher task, deployed on a real robot. The red
cross indicates a failed trajectory.

####  4.2.1 Evaluation

To ensure that the simulation trained policies could be transferred directly
to a real robot, we matched the coordinate frames of the PyRep simulator [17]
used with the real Franka Emika Panda robot setup shown in Figure 2. The state
and action space were matched with that used in the training environment, with
the actions all scaled down to a maximum of
$1.74\text{\,}\mathrm{rad}\text{\,}{\mathrm{s}}^{-1}$ before publishing them
to the robot at a rate of 100 Hz. The robot was trained with a subset of goals
randomly sampled from the positive x-axis region of its workspace as shown in
Figure 2. We classify goal states sampled from outside this region as out-of-
distribution states.

Table 2 shows the results obtained when evaluating the agent on a random set
of ten different goals sampled from the robot’s entire workspace. We report
the average manipulability across the entire trajectory for all sampled goals
as well as the average final manipulability attained at the end of all
trajectories. We additionally indicate the success rate for each controller to
reach the given goals. In all cases, BCF attains the highest manipulability
and success rate surpassing both the control prior and SAC policy. It
additionally illustrates its ability to deal with higher dimensional action
spaces.

To better understand how BCF attains successful trajectories when compared to
a standalone SAC policy, we take a closer look at the individual trajectories
taken by the robot for goals sampled from both the known and out-of-
distribution regions. From each region, we sampled 3 goals and show the
corresponding manipulability curves of the robot across the trajectory in
Figure 4. For each goal, we additionally plot the policy ensemble uncertainty
estimate used by BCF across the trajectory as indicated by the red curves. As
shown in Figure 4 (a), for the known goals, BCF and the SAC agent both attain
similar performances, maximising the manipulability of the agent across the
trajectory. This is in stark contrast to the control prior which exhibits poor
performance. Note here that while the control prior exhibits poor performance
with regard to manipulability, it is still successful in completing the
reaching task at hand without any failures. It is interesting to note the high
uncertainty of the ensemble at the start of a trajectory which quickly drops
to a significantly lower value. The high uncertainty is a result of the
multiple possible trajectories that the robot could take at the start, which
quickly narrows down once the robot begins to move. Note that once the policy
ensemble exhibits a lower uncertainty, the corresponding performance of BCF
closely resembles that of the standalone SAC agent, indicating that BCF does
not cripple the optimality of the learned policy.

When evaluating the agents on out-of-distribution goals as shown in Figure 4
(b), BCF plays an important role in ensuring that the robot can successfully
and safely complete the task. Note the higher levels of uncertainty across
these trajectories when compared to the known goals case. In all these cases,
the standalone SAC agent fails to successfully complete a trajectory,
frequently self-colliding or exhibiting random sporadic behaviours. We
indicate these failed trajectories with a red cross in Figure 4 (b). BCF is
seen to closely follow the behaviours of the control prior in states of high
uncertainty, averting it from such catastrophic failures. While the composite
control strategy works well to ensure the safety of the robot, the higher
reliance of the system on the control prior results in suboptimal behaviour
with regards to manipulability. We provide a supplementary video to
demonstrate these behaviours 111Video demonstration:
https://bit.ly/bcf_deploy. The trade-off between task optimality versus the
safety of the robot is an interesting dilemma that BCF attempts to balance
naturally. The fixed variance, $\sigma_{d}^{2}$ chosen for the prior
controller could serve as a tuning parameter to allow the user to control this
trade-off at deployment. A smaller variance would bias the resulting
controller more strongly towards the control prior yielding more conservative
and suboptimal actions; whereas a larger variance would allow for close to
optimal behaviours at the expense of the robots safety. We leave the
exploration of this idea to future work.

##  5 Discussion and Future Work

Building on the large body of work already developed by the robotics community
can greatly help accelerate the use of RL based systems, allowing us to
develop better controllers for robots as they move towards solving tasks in
the real world. The ideas presented in this paper demonstrate a strategy that
closely couples traditional controllers with learned systems, exploiting the
strengths of each approach in order to attain more reliable and robust
behaviours in the zero-shot sim-to-real setting. We see this as a promising
step towards safely bringing reinforcement learning to real-world robotics.

Across two robotics tasks for navigation and reaching, we show that BCF can
safely deal with out-of-distribution states in the sim-to-real setting without
any fine-tuning, succeeding where a typical standalone policy would fail,
while attaining the optimality of the learned behaviours in known states. In
the navigation domain, we overcome the inefficient oscillatory motion of an
existing reactive navigation controller, decreasing the overall actuation time
during real-world navigation runs by 50.7%. For the reaching task, we show
that our hybrid controller achieves the highest success rate, and improves the
manipulability of an existing reaching controller by 34.9%, a controller
typically difficult to attain using analytical approaches.

While the uncertainty-based compositional policy we derive using BCF does
train with the control prior in the loop, the policy is not directly aware of
the control prior’s presence. This could impact its overall ability to work in
synergy with the control prior at deployment. In future work, we propose to
incorporate the control prior in the Q-value update or alternatively learn a
gating parameter to better inform the fusion process. This should allow the
hybrid controller to operate on more complex tasks as well as interpolate
across multiple behaviours. We are also interested in exploring alternative
state uncertainty estimation techniques for both the control prior and RL that
are faster than the sampling based approaches used in this work. This includes
work from the supervised learning literature for out-of-distribution detection
and distance-based uncertainty estimation techniques [26].

## References

  * Andrychowicz et al. [2020] OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojciech Zaremba.  Learning dexterous in-hand manipulation.  _The International Journal of Robotics Research_ , 39(1):3–20, 2020.  doi: 10.1177/0278364919887447.
  * Arndt et al. [2020] Karol Arndt, Murtaza Hazara, Ali Ghadirzadeh, and Ville Kyrki.  Meta reinforcement learning for sim-to-real domain adaptation.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 2725–2731. IEEE, 2020.
  * Bansal et al. [2020] Somil Bansal, Varun Tolani, Saurabh Gupta, Jitendra Malik, and Claire Tomlin.  Combining optimal control and learning for visual navigation in novel environments.  In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, _Proceedings of the Conference on Robot Learning_ , volume 100 of _Proceedings of Machine Learning Research_ , pages 420–429. PMLR, 30 Oct–01 Nov 2020.
  * Bruce et al. [2018] Jacob Bruce, Niko Suenderhauf, Piotr Mirowski, Raia Hadsell, and Michael Milford.  Learning deployable navigation policies at kilometer scale from a single traversal.  In A Dragan, J Peters, A Billard, and J Morimoto, editors, _Proceedings of Machine Learning Research (PMLR), Volume 87: Conference on Robot Learning 2018_ , pages 346–361. Proceedings of Machine Learning Research, 2018.
  * Colledanchise and Ögren [2016] Michele Colledanchise and Petter Ögren.  How behavior trees modularize hybrid control systems and generalize sequential behavior compositions, the subsumption architecture, and decision trees.  _IEEE Transactions on robotics_ , 33(2):372–389, 2016.
  * Dickinson and Balleine [2002] Anthony Dickinson and Bernard Balleine.  The role of learning in the operation of motivational systems.  _Stevens’ handbook of experimental psychology_ , 2002.
  * Gal and Ghahramani [2016] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _international conference on machine learning_ , pages 1050–1059, 2016.
  * Garcia and Fernández [2012] Javier Garcia and Fernando Fernández.  Safe exploration of state and action spaces in reinforcement learning.  _Journal of Artificial Intelligence Research_ , 45:515–564, 2012.
  * Ghadirzadeh et al. [2017] Ali Ghadirzadeh, Atsuto Maki, Danica Kragic, and Mårten Björkman.  Deep predictive policy training using reinforcement learning.  In _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 2351–2358, 2017.  doi: 10.1109/IROS.2017.8206046.
  * Haarnoja et al. [2018a] Tuomas Haarnoja, Sehoon Ha, Aurick Zhou, Jie Tan, George Tucker, and Sergey Levine.  Learning to walk via deep reinforcement learning.  _arXiv preprint arXiv:1812.11103_ , 2018a.
  * Haarnoja et al. [2018b] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  _arXiv preprint 1801.01290_ , 2018b.
  * Haarnoja et al. [2019] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine.  Soft actor-critic algorithms and applications, 2019.
  * Hans et al. [2008] Alexander Hans, Daniel Schneegaß, Anton Maximilian Schäfer, and Steffen Udluft.  Safe exploration for reinforcement learning.  In _ESANN_ , pages 143–148. Citeseer, 2008.
  * Haviland and Corke [2021] Jesse Haviland and Peter Corke.  A purely-reactive manipulability-maximising motion controller.  _arXiv preprint arXiv:2002.11901_ , 2021.
  * Ibarz et al. [2021] Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, and Sergey Levine.  How to train your robot with deep reinforcement learning: lessons we have learned.  _The International Journal of Robotics Research_ , 40(4-5):698–721, Jan 2021.  ISSN 1741-3176.  doi: 10.1177/0278364920987859.
  * Ijspeert [2008] Auke Jan Ijspeert.  Central pattern generators for locomotion control in animals and robots: a review.  _Neural networks_ , 21(4):642–653, 2008.
  * James et al. [2019] Stephen James, Marc Freese, and Andrew J. Davison.  Pyrep: Bringing v-rep to deep robot learning.  _arXiv preprint arXiv:1906.11176_ , 2019.
  * Johannink et al. [2018] Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow, and Sergey Levine.  Residual reinforcement learning for robot control.  _arXiv preprint arXiv:1812.03201_ , 2018.
  * Julian et al. [2020] Ryan C Julian, Eric Heiden, Zhanpeng He, Hejia Zhang, Stefan Schaal, Joseph J Lim, Gaurav S Sukhatme, and Karol Hausman.  Scaling simulation-to-real transfer by learning a latent space of robot skills.  _The International Journal of Robotics Research_ , 39(10-11):1259–1278, 2020.  doi: 10.1177/0278364920944474.
  * Kahn et al. [2017] Gregory Kahn, Adam Villaflor, Vitchyr Pong, Pieter Abbeel, and Sergey Levine.  Uncertainty-aware reinforcement learning for collision avoidance.  _arXiv preprint arXiv:1702.01182_ , 2017.
  * Koren and Borenstein [1991] Yoram Koren and Johann Borenstein.  Potential field methods and their inherent limitations for mobile robot navigation.  In _Proceedings. 1991 IEEE International Conference on Robotics and Automation_ , pages 1398–1404. IEEE, 1991.
  * Lakshminarayanan et al. [2017] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  In _Advances in Neural Information Processing Systems_ , pages 6402–6413, 2017.
  * Lee et al. [2014] Sang Wan Lee, Shinsuke Shimojo, and John P O’Doherty.  Neural computations underlying arbitration between model-based and model-free learning.  _Neuron_ , 81(3):687–699, 2014.
  * Lipton et al. [2016] Zachary C Lipton, Jianfeng Gao, Lihong Li, Jianshu Chen, and Li Deng.  Combating deep reinforcement learning’s sisyphean curse with intrinsic fear.  2016\.
  * Maxwell [1868] James Clerk Maxwell.  I. on governors.  _Proceedings of the Royal Society of London_ , (16):270–283, 1868.
  * Miller et al. [2021] Dimity Miller, Niko Sunderhauf, Michael Milford, and Feras Dayoub.  Class anchor clustering: A loss for distance-based open set recognition.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , pages 3570–3578, January 2021.
  * Murooka et al. [2020] Takayuki Murooka, Masashi Hamaya, Felix von Drigalski, Kazutoshi Tanaka, Yoshihisa Ijiri, and Yutaro Konta.  Exi-net: Explicitly/implicitly conditioned network for multiple environment sim-to-real transfer.  In _CoRL_ , 2020.
  * Osband et al. [2018] Ian Osband, John Aslanides, and Albin Cassirer.  Randomized prior functions for deep reinforcement learning.  _arXiv preprint arXiv:1806.03335_ , 2018.
  * Patel and Sobh [2015] Sarosh Patel and Tarek Sobh.  Manipulator performance measures-a comprehensive literature survey.  _Journal of Intelligent & Robotic Systems_, 77(3-4):547–570, 2015.
  * Peng et al. [2018] Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel.  Sim-to-real transfer of robotic control with dynamics randomization.  In _2018 IEEE international conference on robotics and automation (ICRA)_ , pages 3803–3810. IEEE, 2018.
  * Rana et al. [2020] Krishan Rana, Ben Talbot, Vibhavari Dasagi, Michael Milford, and Niko Sünderhauf.  Residual reactive navigation: Combining classical and learned navigation strategies for deployment in unknown environments.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 11493–11499. IEEE, 2020.
  * Rusu et al. [2017] Andrei A Rusu, Matej Večerík, Thomas Rothörl, Nicolas Heess, Razvan Pascanu, and Raia Hadsell.  Sim-to-real robot learning from pixels with progressive nets.  In _Conference on Robot Learning_ , pages 262–270. PMLR, 2017.
  * Silver et al. [2018] Tom Silver, Kelsey Allen, Josh Tenenbaum, and Leslie Kaelbling.  Residual policy learning.  _arXiv preprint arXiv:1812.06298_ , 2018.
  * Warren [1989] Charles W Warren.  Global path planning using artificial potential fields.  In _Proceedings, 1989 International Conference on Robotics and Automation_ , pages 316–321. Ieee, 1989.
  * Whitney [1969] D. E. Whitney.  Resolved motion rate control of manipulators and human prostheses.  _IEEE Transactions on Man-Machine Systems_ , 10(2):47–53, June 1969.  ISSN 2168-2860.  doi: 10.1109/TMMS.1969.299896.
  * Yoshikawa [1985] Tsuneo Yoshikawa.  Manipulability of Robotic Mechanisms.  _The International Journal of Robotics Research_ , 4(2):3–9, 1985.  doi: 10.1177/027836498500400201.
  * Zhang et al. [2015] Fangyi Zhang, Jürgen Leitner, Michael Milford, Ben Upcroft, and Peter Corke.  Towards vision-based deep reinforcement learning for robotic motion control.  _arXiv preprint arXiv:1511.03791_ , 2015.

##  Appendix A Appendix

###  A.1 Task Description

####  A.1.1 PointGoal Navigation

The objective of this task is to navigate a robot from a start location to a
goal location in the shortest time possible, while avoiding obstacles along
the way. We utilise the training environment provided by [31], which consists
of five arenas with different configurations of obstacles. The goal and start
location of the robot are randomised at the start of every episode, each
placed on the extreme opposite ends of the arena (see Figure 2 (a)). This sets
the long horizon nature of the task. As we focus on the sparse reward setting,
we define $r(s_{t},a_{t},s_{t+1})=1$ if
$d_{\text{target}}<d_{\text{threshold}}$ and $r(s_{t},a_{t},s_{t+1})=0$
otherwise, where $d_{\text{target}}$ is the distance between the agent and the
goal and $d_{\text{threshold}}$ is a set threshold. The action $a_{t}$
consists of two continuous values: linear velocity $\nu_{\text{nav}}\in[-1,1]$
and angular velocity $\omega_{\text{nav}}\in[-1,1]$. We assume that the robot
can localise itself within a global map in order to determine its relative
position to a goal location. The 180∘ laser scan range data is divided into 15
bins and concatenated to the robot’s angle. The overall state $s_{t}$ of the
environment is comprised of:

  * •

The binned laser scan data $l_{\text{bin}}\in\mathbb{R}^{15}$,

  * •

The pose error between the robot’s pose and the goal location
$e_{t}\in\mathbb{R}^{2}$,

  * •

The previous executed linear and angular velocity $a_{t-1}\in\mathbb{R}^{2}$,

for a total of 19 dimensions. The length of each episode is set to a maximum
of 500 steps and does not terminate once the goal is achieved.

####  A.1.2 Manipulability Maximising Reacher

The objective of this task is to actuate each joint of a manipulator within a
closed-loop velocity controller such that the end-effector moves towards and
reaches the goal point, while the manipulability of the manipulator is
maximised. The manipulability index describes how easily the manipulator can
achieve any arbitrary velocity. The ability of the manipulator to achieve an
arbitrary end-effector velocity is a function of the manipulator Jacobian.
While there are many methods that seek to summarise this, the manipulability
index proposed by [36] is the most used and accepted within the robotics
community [29]. Utilising Jacobian based indices in existing controllers have
several limitations, require greater engineering effort than simple inverse
kinematics based reaching systems, and precise tuning in order to ensure the
system is operational [14]. We explore the use of RL to learn such behaviours
by leveraging simple reaching controllers as priors. We utilise the PyRep
simulation environment [17], with the Franka Emika Panda as our manipulator as
shown in Figure 2 (b). For this task, we generate a random initial joint
configuration and random end-effector goal pose. We use a sparse goal reward,
$r(s_{t},a_{t},s_{t+1})=1$ if $e_{t}<e_{\text{threshold}}$ and
$r(s_{t},a_{t},s_{t+1})=m$ otherwise, where $e_{t}$ is the spatial
translational error between the end-effector and the goal,
$e_{\text{threshold}}$ is a set threshold and

$m=\sqrt{\mbox{det}\left(J(q)J(q)^{\top}\right)}\ \in[0,\infty)$ (12)

is the manipulability of the robot at the particular joint configuration $q$
where $J(q)$ is the manipulator Jacobian. The action space consists of the
manipulator joint velocities $\dot{\it{q}}\in[-1,1]^{n}$, where the values are
continuous, and $n$ is the number of joints within the manipulator. In this
work, the manipulator used consists of 7 joints. The state, $s_{t}$, of the
environment is comprised of:

  * •

The joint coordinate vector $\it{q}\in\mathbb{R}^{7}$,

  * •

The joint velocity vector $\dot{\it{q}}\in\mathbb{R}^{7}$,

  * •

The translation error between the manipulator’s end-effector and the goal
$e_{t}(\it{q})\in\mathbb{R}^{3}$,

  * •

The end-effector translation vector $e_{p}(\it{q})\in\mathbb{R}^{3}$,

for a total of 20 dimensions. Similar to the navigation task, the episode
length for this task in fixed at 1000 steps and only terminates at the end of
the episode.