# Protective Policy Transfer

Wenhao Yu1∗ C. Karen Liu2 and Greg Turk1 1Georgia Institute of Technology,
Atlanta, Georgia, USA<EMAIL_ADDRESS>University,
Stanford, California, USA<EMAIL_ADDRESS>at Robotics at
Google

###### Abstract

Being able to transfer existing skills to new situations is a key capability
when training robots to operate in unpredictable real-world environments. A
successful transfer algorithm should not only minimize the number of samples
that the robot needs to collect in the new environment, but also prevent the
robot from damaging itself or the surrounding environment during the transfer
process. In this work, we introduce a policy transfer algorithm for adapting
robot motor skills to novel scenarios while minimizing serious failures. Our
algorithm trains two control policies in the training environment: a task
policy that is optimized to complete the task of interest, and a protective
policy that is dedicated to keep the robot from unsafe events (e.g. falling to
the ground). To decide which policy to use during execution, we learn a safety
estimator model in the training environment that estimates a continuous safety
level of the robot. When used with a set of thresholds, the safety estimator
becomes a classifier for switching between the protective policy and the task
policy. We evaluate our approach on four simulated robot locomotion problems
and a 2D navigation problem and show that our method can achieve successful
transfer to notably different environments while taking the robot’s safety
into consideration.

##  I Introduction

When performing motor skills in a novel environment, humans tend to act more
conservatively at the beginning to avoid potential dangers and gradually take
more agile movements to improve our performance as we get more familiar with
the situation. This capability of temporarily shifting the focus between task
completion and self protection allows us to transfer our existing skills to
scenarios we have not seen before in a protective way and is crucial for us to
survive in the ever-changing and unpredictable real-world environment.

This work aims to develop computational tools that allow a robot to also adapt
its motor skills to a new scenario safely. We focus on the practical problem
of legged locomotion because legged robots are often prone to falling to the
ground and damaging themselves due to the fast movements and intricate balance
that is involved. Being able to transfer legged locomotion skills safely to
new environments is a crucial step towards deploying robots in real-world
applications.

There has been extensive research in transferring control policies from one
domain to another [1, 2, 3, 4] as well as incorporating safety in robotic
control [5, 6, 7, 8]. However, few of them address _both safety and transfer_.
Existing methods in safety usually considers learning a policy that satisfies
certain safety constraints such as velocity limit and obstacle avoidance.
However, the optimized policies usually do not transfer easily to novel
situations. On the other hand, current transfer learning algorithms often
require additional data collection to adapt the policy to new environments,
but do not consider the safety issue during the process of adaptation.

In this work, we introduce a transfer learning algorithm that takes the safety
of the robot into consideration. We assume that the robot is first trained in
a simulated environment, where it can explore different actions and
potentially dangerous states. Our transfer learning algorithm consists of four
components. We first train a task policy $\pi_{task}$ to achieve optimal
performance for the task. Based on states visited by $\pi_{task}$, we train a
protective policy $\pi_{protect}$ that protects the robot from unsafe states
such as falling on the ground. To determine which policy to use during
execution, we learn a safety estimator model $\psi$ that estimates a
continuous safety level of the robot. Training of these models is performed in
the source environment. Finally, we develop an adaptation procedure to
transfer the learned models from source to target environment with at most two
unsafe trials.

Figure 1: The four tasks used in our experiments: (a) Hopper, (b) HalfCheetah,
(c) Walker2D, and (d) 3D Quadruped. Our algorithm trains policies in the
source environments (Top row) and transfer them to target environments while
minimizing damage to the robot (Bottom row).

We evaluate our method of transferring locomotion skills to novel environments
for four different robots, as shown in Figure 1. For each robot, we design a
variety of testing scenarios inspired by prior work in sim-to-real transfer
[1], which creates non-trivial gaps between the training and testing
environments of the robot. We demonstrate that our method can successfully
complete the task while minimizing the number of failed trials in novels
environments, while baseline methods do not achieve successful transfer or
require a higher number of failed trials.

##  II Related Work

Transferring skills learned in a source environment such as computer
simulation to novel real-world scenarios is an important direction for
creating robots that can be deployed in real-world applications. Recent
advances in transfer learning for reinforcement learning have demonstrated
some success in deploying simulation-trained policies to real physical robots
[1, 9, 2, 3, 10, 11, 12, 13]. One of the key strategies to achieve successful
policy transfer is to improve the simulation accuracy by performing system
identification [1, 14, 11, 12, 13, 15]. For example, Tan et al. identified key
aspects to the discrepancy between simulated and real robot dynamics such as
actuator dynamics and sensor latency [1]. By properly measuring these factors
and modeling them in the computer simulation, they achieved successful sim-to-
real transfer for a real quadruped robot.

Complementary to improving the simulation fidelity, researchers have also
developed algorithms to train policies that can work for a large variety of
scenarios. One class of these algorithms is to fine-tune the simulation-
trained controller with a limited amount of data on the real-robot [9, 2, 3,
16, 17]. For example, Yu et al. proposed to learn a parameterized controller
with a latent space in simulation and optimize the latent input directly on
the real hardware [9]. Allowing controllers to be fine-tuned using real-world
data greatly improves their adaptation performance in novel situations.
However, the safety of the robot during real-world experiments is not
considered in existing methods. As a result, extra supervision is needed from
the experimenter. Our method improved upon the prior methods in this category
by considering the safety of the robot during adaptation.

Researchers have also proposed policy optimization methods that takes the
safety of the robot into consideration. Some of existing methods enforce the
safety of the robot using a model-based approach [18, 19, 20, 8, 21, 7, 22].
Given an initial safe but sub-optimal policy, Garcia et al. [18] and
Berkenkamp et al. [8] proposed exploration schemes that are provably safe
around the current policy and used them to gradually expand the operation
region for the policy. Despite being able to provide guarantees on the safety,
these methods usually assume the availability of the system dynamics, which
may not be true in real-world problems.

Alternatively, researchers have proposed model-free methods for learning
policies that satisfy certain safety constraints [5, 6, 23, 24, 25, 26, 27].
For example, Achiam et al. introduced a general DRL algorithm, Constrained
Policy Optimization (CPO), that solves a constrained MDP problem [5]. They
demonstrated CPO on simulated continuous control tasks with state constraints.
Eysenbach et al. proposed to jointly train a policy to complete the task and a
policy to reset the robot to the initial state when the robot is in an unsafe
state [27]. Although these methods can obtain controllers that respects safety
constraints in the limit, during the learning process the robot may still be
exposed to risky states.

In addition to enforcing general safety constraints, researchers have also
investigated more task-dependent safety problems. For example, for humanoid
robots, researchers have devised specialized algorithms to reduce the damage
they receives during falling [28, 29, 30, 31]. Though effective in handling
the falling scenario, these methods may not generalize to other types of
failure modes.

##  III Methods

###  III-A Training a task policy

We first train a task policy $\pi_{task}$ that performs the desired motor
skill using deep reinforcement learning (DRL). We formulate the motor skill
learning problem as a Markov Decision Process (MDP),
$\mathcal{M}=(\lx@glossaries@gls@link{main}{state_{s}pace}{\leavevmode\mathcal{S}},\lx@glossaries@gls@link{main}{action_{s}pace}{\leavevmode\mathcal{A}},\lx@glossaries@gls@link{main}{reward_{f}unction}{\leavevmode\mathcal{R}},\lx@glossaries@gls@link{main}{transition}{\leavevmode\mathcal{T}},\lx@glossaries@gls@link{main}{init_{d}istribution}{\leavevmode
p_{0}},\gamma)$, where S is the state space, A is the action space,
$\lx@glossaries@gls@link{main}{reward_{f}unction}{\leavevmode\mathcal{R}}:\lx@glossaries@gls@link{main}{state_{s}pace}{\leavevmode\mathcal{S}}\times\lx@glossaries@gls@link{main}{action_{s}pace}{\leavevmode\mathcal{A}}\mapsto\mathbb{R}$
is the reward function,
$\lx@glossaries@gls@link{main}{transition}{\leavevmode\mathcal{T}}:\lx@glossaries@gls@link{main}{state_{s}pace}{\leavevmode\mathcal{S}}\times\lx@glossaries@gls@link{main}{action_{s}pace}{\leavevmode\mathcal{A}}\mapsto\lx@glossaries@gls@link{main}{state_{s}pace}{\leavevmode\mathcal{S}}$
is the transition function, $p_{0}$ is the initial state distribution and
$\gamma$ is a discount factor. In DRL, a policy $\pi$ represented by a neural
network is searched to maximize the expected return:

$J_{task}=\mathbb{E}_{\tau=(\lx@glossaries@gls@link{main}{state}{\leavevmode\mathbf{s}}_{0},\lx@glossaries@gls@link{main}{action}{\leavevmode\mathbf{a}}_{0},\dots,\lx@glossaries@gls@link{main}{state}{\leavevmode\mathbf{s}}_{T})}\sum_{t=0}^{T}\gamma^{t}\lx@glossaries@gls@link{main}{reward_{f}unction}{\leavevmode\mathcal{R}}(\lx@glossaries@gls@link{main}{state}{\leavevmode\mathbf{s}}_{t},\lx@glossaries@gls@link{main}{action}{\leavevmode\mathbf{a}}_{t}),$
(1)

where
$\lx@glossaries@gls@link{main}{state}{\leavevmode\mathbf{s}}_{0}\sim\lx@glossaries@gls@link{main}{init_{d}istribution}{\leavevmode
p_{0}}$,
$\lx@glossaries@gls@link{main}{action}{\leavevmode\mathbf{a}}_{t}\sim\pi(\lx@glossaries@gls@link{main}{state}{\leavevmode\mathbf{s}}_{t})$
and
$\lx@glossaries@gls@link{main}{state}{\leavevmode\mathbf{s}}_{t+1}=\lx@glossaries@gls@link{main}{transition}{\leavevmode\mathcal{T}}(\lx@glossaries@gls@link{main}{state}{\leavevmode\mathbf{s}}_{t},\lx@glossaries@gls@link{main}{action}{\leavevmode\mathbf{a}}_{t})$.
In practice, we often only have access to a partial observation
$\lx@glossaries@gls@link{main}{observation}{\leavevmode\mathbf{o}}\in\lx@glossaries@gls@link{main}{observation_{s}pace}{\leavevmode\mathcal{O}}$
of the robot’s state, where O is the observation space. Thus the policies we
train would become
$\pi:\lx@glossaries@gls@link{main}{observation_{s}pace}{\leavevmode\mathcal{O}}\mapsto\lx@glossaries@gls@link{main}{action_{s}pace}{\leavevmode\mathcal{A}}$.
we use Proximal Policy Optimization (PPO) [32] to optimize the motor skill
policies, though our algorithm is agnostic to the specific choice of DRL
algorithm. During training of $\pi_{task}$, we randomize the dynamics of the
simulated robot, also known as Dynamics Randomization (DR) [14], to improve
its robustness. However, as we show in our experiments, using DR alone is not
sufficient to achieve successful transfer.

###  III-B Training a protective policy

To prevent potential damage to the robot, we train a protective policy
$\pi_{protect}$ that is dedicated to keeping the robot within a set of safe
states $\mathbf{S}_{safe}$. The boundary of $\mathbf{S}_{safe}$ can be
precisely defined by general inequality constraints on state variables. For
example, we can reuse the inequality constraints defined for early termination
conditions as the boundary of $\mathbf{S}_{safe}$. Concretely, in a locomotion
problem, a state where the robot falls to the ground will terminate the
rollout and is considered outside of $\mathbf{S}_{safe}$. To train
$\pi_{protect}$, we design a reward function:
$\lx@glossaries@gls@link{main}{protect_{r}eward_{f}unction}{\leavevmode\mathcal{R}_{protect}}(\lx@glossaries@gls@link{main}{state}{\leavevmode\mathbf{s}},\lx@glossaries@gls@link{main}{action}{\leavevmode\mathbf{a}})=w_{alive}r_{alive}+w_{action}r_{action}+w_{task}r_{task}$,
where $r_{alive}$ is a constant reward given to the robot,
$\lx@glossaries@gls@link{main}{state}{\leavevmode\mathbf{s}}_{t}\in\mathbf{S}_{safe}$,
$r_{action}=||\lx@glossaries@gls@link{main}{action}{\leavevmode\mathbf{a}}||^{2}_{2}$
is a regularization term for limiting the magnitude of the actions, and
$r_{task}=\min(\lx@glossaries@gls@link{main}{task_{r}eward_{f}unction}{\leavevmode\mathcal{R}_{task}}(\lx@glossaries@gls@link{main}{state}{\leavevmode\mathbf{s}},\lx@glossaries@gls@link{main}{action}{\leavevmode\mathbf{a}}),0)$
discourages the policy from exhibiting behaviors that hurt the desired task.
$w_{alive}$, $w_{action}$, $w_{task}$ modulate the relative importance of the
reward terms. Note that $r_{task}$ is always negative and only penalizes the
agent when it is acting in opposition to the task, e.g. if the robot walks
backward when the task is to move forward. We find this helps us train
protective policies that are more compatible with the task policies. Section
IV-A describes the weights and the $\mathbf{S}_{safe}$ for different problems
demonstrated in this work.

$\pi_{protect}$ needs to work for all states that the robot might encounter
during adaptation. However, we do not know beforehand what states the real
robot is going to visit. In our work, we approximate this state distribution
by running $\pi_{task}$ with added noise in the source environment to collect
a set of states $\mathbf{S}$ that are relevant to the task. We then use
$\mathbf{S}$ as the initial state distribution when training $\pi_{protect}$.

An important assumption we make is that $\pi_{protect}$ not only minimizes the
potential damage to the robot, but also generalizes to novel environments
better than $\pi_{task}$. For the locomotion tasks, this is a reasonable
assumption because $\pi_{protect}$ is specialized to prevent the robot from
falling over and often learns to take the robot to stable poses such as
standing upright, which are usually less sensitive to different dynamics. We
validate this assumption in Section IV-F. We do note that for other transfer
problems, this assumption may not always hold. However, our proposed framework
of protective policy transfer is agnostic to how $\pi_{protect}$ is obtained
as long as it satisfies the generalization assumption.

###  III-C Training a one-step safety estimator

Algorithm 1 Learning OSSE

1:Randomly initialize weights $\psi$ for OSSE

2:Obtain the value function $V_{protect}$ of $\pi_{protect}$

3:Run $\pi_{task}$ for $L$ rollouts and store states in $\mathbf{S}$

4:Run $\pi_{protect}$ for $M$ rollouts from states drawn from $\mathbf{S}$

5:Store the generated states
$\lx@glossaries@gls@link{main}{state}{\leavevmode\mathbf{s}}_{t}$ to a buffer
$\mathbf{B}$

6:for each $\lx@glossaries@gls@link{main}{state}{\leavevmode\mathbf{s}}_{t}$
in $\mathbf{B}$ do

7: Run $\pi_{task}$ from
$\lx@glossaries@gls@link{main}{state}{\leavevmode\mathbf{s}}_{t}$ and get
$(\lx@glossaries@gls@link{main}{observation}{\leavevmode\mathbf{o}}_{t},\lx@glossaries@gls@link{main}{observation}{\leavevmode\mathbf{o}}_{t+1})$

8: Add $\lx@glossaries@gls@link{main}{observation}{\leavevmode\mathbf{o}}_{t}$
to training inputs $T_{input}$

9: Add
$V_{protect}(\lx@glossaries@gls@link{main}{observation}{\leavevmode\mathbf{o}}_{t+1})$
to training labels $T_{label}$

10:end for

11:Optimize $\psi$ with $T_{input}$ and $T_{label}$

12:return $\psi$

$\pi_{task}$ outputs actions that optimize the task reward but do not transfer
to unseen environments. On the other hand, $\pi_{protect}$ generates actions
that keep the robot safe and can generalize to novel scenarios, yet do not
accomplish the desired task. Can we combine them to achieve protective policy
transfer in the target environment? Naively interpolating the actions from the
two policies is likely to result in suboptimal behaviors that are neither
functional nor safe. In this work, we propose to combine the task and
protective policies by switching between $\pi_{task}$ and $\pi_{protect}$
based on an estimation of how “safe” the robot is: we use $\pi_{task}$ if the
robot is sufficiently safe, and use $\pi_{protect}$ otherwise. A natural way
to perform this estimation is to use the value function $V_{protect}$ from
$\pi_{protect}$ as it measures the expected return of the protective policy.
However, $V_{protect}$ may not be sufficient for ensuring safety during
transfer. In particular, because $V_{protect}$ only considers actions from
$\pi_{protect}$ and does not take $\pi_{task}$ into consideration, a state
that is considered safe by $V_{protect}$ may not be safe anymore after taking
an action from $\pi_{task}$. This will cause over-estimation of the safety for
the robot, leading to unsafe behaviors. To address this issue, we train a
predictive model that computes the safety estimation for a state assuming that
we take one step from $\pi_{task}$ and then follow $\pi_{protect}$. We call
this new model the one-step safety estimator (OSSE). A more detailed
description of the algorithm can be found in Algorithm 1.

###  III-D Protective adaptation to testing environment

The trained OSSE provides a continuous estimation
$\psi(\lx@glossaries@gls@link{main}{observation}{\leavevmode\mathbf{o}})$ for
the safety-level of the robot. To make a binary decision for which policy to
use, we further threshold the OSSE predictions to produce a binary output. We
find that using a single threshold to perform the policy selection often leads
to oscillations between the two policies, causing the robot to go to states
that are unfamiliar to either policies. To mitigate this issue, we define two
thresholds $\kappa_{task}$ and $\kappa_{protect}$ and classify the robot into
one of two operation modes: protect and task. The operation mode of the robot
is updated by the OSSE prediction and the thresholds: if the robot is in the
protect mode and
$\psi(\lx@glossaries@gls@link{main}{observation}{\leavevmode\mathbf{o}})>\kappa_{protect}$,
it means the robot is in a sufficiently safe state and we switch to task mode.
Similarly, if the robot is in the task mode and
$\psi(\lx@glossaries@gls@link{main}{observation}{\leavevmode\mathbf{o}})<\kappa_{task}$,
then the robot is not safe anymore and should switch to protect mode. The
resulting policy as well as the operation model update rule is shown as
follows:

$\pi_{combine}(\lx@glossaries@gls@link{main}{observation}{\leavevmode\mathbf{o}},\texttt{m})=\begin{cases}\pi_{protect}(\lx@glossaries@gls@link{main}{observation}{\leavevmode\mathbf{o}}),&\text{if
}\texttt{m}=\texttt{protect}\\\
\pi_{task}(\lx@glossaries@gls@link{main}{observation}{\leavevmode\mathbf{o}}),&\text{if
}\texttt{m}=\texttt{task},\\\ \end{cases}$ (2)

$\texttt{m}=\begin{cases}\texttt{protect},&\text{if
}\psi(\lx@glossaries@gls@link{main}{observation}{\leavevmode\mathbf{o}})<\kappa_{task}\text{
\ and \ }\texttt{m}=\texttt{task}\\\ \texttt{task},&\text{if
}\psi(\lx@glossaries@gls@link{main}{observation}{\leavevmode\mathbf{o}})>\kappa_{protect}\text{
\ and \ }\texttt{m}=\texttt{protect}.\end{cases}$ (3)

Algorithm 2 Protective adaptation algorithm

1:Input: $\pi_{task}$, $\pi_{protect}$, $\psi$, $\Delta$, $\kappa_{min}$

2:Initialize $\kappa_{task}=1.0$, $\kappa_{protect}=1.0$,
$\kappa_{task}^{*}=1.0$, $\kappa_{protect}^{*}=1.0$, $R^{*}=-\infty$

3:while $\kappa_{task}>\kappa_{min}$ do

4: Evaluate performance $R$ of $\pi_{combine}$

5: if $R>R^{*}$ then

6: $\kappa_{task}^{*}=\kappa_{task},R^{*}=R$

7: end if

8: if robot is unsafe then

9: $\kappa_{task}=\kappa_{task}+\Delta$

10: break

11: end if

12: $\kappa_{task}=\kappa_{task}-\Delta$

13:end while

14:while $\kappa_{protect}>\kappa_{task}$ do

15: Evaluate performance $R$ of $\pi_{combine}$

16: if $R>R^{*}$ then

17: $\kappa_{protect}^{*}=\kappa_{protect},R^{*}=R$

18: end if

19: if robot is unsafe then

20: $\kappa_{protect}=\kappa_{protect}+\Delta$

21: break

22: end if

23: $\kappa_{protect}=\kappa_{protect}-\Delta$

24:end while

25:return $\kappa_{task}^{*}$, $\kappa_{protect}^{*}$

Manually specifying the values for $\kappa_{protect}$ and $\kappa_{task}$ can
be challenging: for different control problems and robots, the optimal
thresholds can be different. One possible way to find the thresholds is to
optimize them in the source environment. However, this will likely lead to a
policy that exclusively uses the task policy since it maximizes the task
reward in the source environment. We propose a protective adaptation algorithm
that automatically finds the values for $\kappa_{protect}$ and $\kappa_{task}$
with at most two unsafe trials in the target environment. Our core idea is to
start with the most conservative combination (using $\pi_{protect}$ only) and
gradually relax the thresholds to allow the task policy to take actions when
the robot is safe. Specifically, we first set
$\kappa_{protect}=\kappa_{task}=1$ and gradually reduce $\kappa_{task}$ until
the robot encounters the first unsafe trial. This step finds the boundary
where $\pi_{task}$ stops transfer to the target environment and needs
assistance from $\pi_{protect}$. We then perform a second phase of search
where we fix the optimized $\kappa_{task}$ and gradually reduce
$\kappa_{protect}$ until the resulting policy leads the robot into an unsafe
state. After the search, we will return the $\kappa_{task}$ and
$\kappa_{protect}$ that achieve the best task performance. Algorithm 2
describes our protective adaptation algorithm in greater details.

##  IV Results

###  IV-A Experimental Setup

We evaluate our algorithm on four challenging locomotion problems with
simulated legged robots, as shown in Figure 1. The first three robots: Hopper,
HalfCheetah, and Walker2D, are designed based on OpenAI Gym [33]. For the
fourth example, we use a 3D quadruped robot, the Unitree A1 (Figure 1(d)). A1
is equipped with $12$ motors: two at each hip joint and one for each knee
joint. The motors are controlled by Proportional Derivative (PD) controllers
with desired motor angles $\bar{\mathbf{q}}$ as input. We train locomotion
policies that output $\bar{\mathbf{q}}$ and constrain the change of each
dimension in $\bar{\mathbf{q}}$ between consecutive timesteps to be smaller
than $0.2$ for encouraging smooth motion. In all examples, the policies
observe the IMU sensor data, motor angles, and motor velocities of the robot.

TABLE I: Task reward functions.

| Hopper | HalfCheetah | Walker2D | A1
---|---|---|---|---
$v_{max}$ | $\infty$ | $\infty$ | $\infty$ | 2.5
$w_{vel}$ | 1 | 1 | 1 | 3
$w_{action}$ | 0.03 | 0.5 | 0.05 | 0.025
$w_{knee}$ | 0.5 | 0.5 | 0 | 0
$w_{hip}$ | - | - | - | 1
$w_{dev}$ | - | - | - | 5

We design the following task reward function tor training locomotion skills:
$\lx@glossaries@gls@link{main}{task_{r}eward_{f}unction}{\leavevmode\mathcal{R}_{task}}(\lx@glossaries@gls@link{main}{state}{\leavevmode\mathbf{s}},\lx@glossaries@gls@link{main}{action}{\leavevmode\mathbf{a}})=1+w_{vel}\min(\dot{x},v_{max})-w_{action}||\lx@glossaries@gls@link{main}{action}{\leavevmode\mathbf{a}}||_{2}^{2}-w_{knee}r_{knee}-w_{hip}|q_{hip}|-w_{dev}|y|$,
where $v_{max}$ caps the velocity reward and prevents the robot from
exploiting unrealistic movements to achieve fast moving speed, $r_{knee}$ is a
binary reward denoting whether the knee joint of the robot is at the limit,
$q_{hip}$ is the motor angle for the hip abduction and adduction, which
discourages the A1 robot to walk with its legs spread or crossed, and $y$ is
the deviation in the left-right direction. Table I lists the rewards
parameters used in our experiments. For training the protective policy
$\pi_{protect}$ in all tasks, we use
$\lx@glossaries@gls@link{main}{protect_{r}eward_{f}unction}{\leavevmode\mathcal{R}_{protect}}(\lx@glossaries@gls@link{main}{state}{\leavevmode\mathbf{s}},\lx@glossaries@gls@link{main}{action}{\leavevmode\mathbf{a}})=4.0+0.3||\lx@glossaries@gls@link{main}{action}{\leavevmode\mathbf{a}}||^{2}_{2}-0.2\min(\lx@glossaries@gls@link{main}{task_{r}eward_{f}unction}{\leavevmode\mathcal{R}_{task}}(\lx@glossaries@gls@link{main}{state}{\leavevmode\mathbf{s}},\lx@glossaries@gls@link{main}{action}{\leavevmode\mathbf{a}}),0)$.

TABLE II: Domain Randomization Setting.

| Hopper | HalfCheetah | Walker2D | A1
---|---|---|---|---
Friction | [0.2, 1.0] | [0.4, 1.5] | [0.2, 1.0] | [0.4, 1.5]
Restitution | [0.0, 0.3] | [0.0, 2.0] | [0.0, 0.8] | [0.0, 0.8]
Mass | [2.0, 10.0]kg | [60, 150]% | [2.0, 10.0]kg | [60, 150]%
Joint damping | [0.5, 3.0] | [70, 130]% | [0.01, 0.2] | -
Joint Stiffness | - | [70, 130]% | - | -
Torque limit | [100, 300]Nm | [60,150]% | - | [15,25]Nm
Position gain | - | - | - | [60, 150]%

Figure 2: Example sequences of our approach for the Hopper (top) and A1
(bottom) robots. Green bodynode means that $\pi_{task}$ is being used while
red bodynodes means $\pi_{protect}$ is being used.

During training, we perform dynamics randomization [14] to improve the
robustness of the policies. Table II summarizes the parameters and the range
we use for domain randomization for all four tasks. Note that for HalfCheetah
and A1 we randomize some parameters using percentage of the default values
instead of absolute values. This is to avoid un-physical settings during
randomization for these robots.

Figure 3: Comparison with baseline methods for Hopper (left), HalfCheetah
(middle), and A1 robot (right) in performance and the number of unsafe trials
during transfer. We train each policy for five random seeds (one standard
deviation shown as shaded area). DR and DR-RE have zero failed trial count
since they do not perform adaptation. Our method and NO-OSSE both allow two
failed trials.

We define the robot to be safe when it is not falling to the ground or losing
balance. Concretely, we specify the safety state set based on three
conditions: 1) the robot is allowed to touch the ground only with its feet, 2)
the height of the robot should be higher than $h_{min}$, and 3) the roll,
pitch, and yaw ($\psi$, $\theta$, $\phi$) of the robot base should not exceed
$\psi_{max},\theta_{max},\phi_{max}$. For Hopper, we set $h_{min}=0.75$ and
$\theta_{max}=0.8$. For Walker2D, we set $h_{min}=0.8$ and $\theta_{max}=1.0$.
For HalfCheetah, we set $h_{min}=0.0$ and $\theta_{max}=1.3$. And for A1, we
have $h_{min}=0.1$, $\psi_{max}=0.4$, $\theta_{max}=0.6$, and
$\phi_{max}=1.0$. These conditions are also used as the terminal condition
during training, which is commonly done in locomotion learning.

We use Proximal Policy Optimization (PPO) [32] for training both $\pi_{task}$
and $\pi_{protect}$. We represent both policies using neural networks with two
fully-connected layers, each with 64 neurons. In our implementation, we use an
ensemble of three models for both $V_{protect}$ and OSSE, where each model is
represented by a three-layer neural network with $256$, $128$, and $64$
neurons respectively. The final estimated safety level of OSSE is computed as
the average of the three models in the ensemble. We use tanh activation for
all models. For each method, we run five random seeds and report mean and one
standard deviation for the performance. Figure 2 illustrates some of the
results by our algorithm for the Hopper and A1 robot in the testing
environment. The results can be better seen in the supplementary video.

###  IV-B Transfer tasks

We design a variety of testing environments for each robot to evaluate the
ability of our algorithm to transfer to new situations. For Hopper, Walker2d,
and HalfCheetah, we train the policies in one physics simulator, DART [34],
and transfer them to a different simulator, MuJoCo [35]. As discussed in prior
work, the discrepancies between DART and MuJoCo presents significant challenge
in transferring policies from one to the other [36]. Furthermore, we add
latency of $8$ms, $50$ms, and $16$ms to the testing environments of the three
robots while no latency is added during training.

For the A1 robot, we build the simulated robot in DART and create two gaps
between the training and testing environments. First, we introduce a power
limit to each motor in the testing environments. The power for each robot is
measured by $\tau\dot{q}$, where $\tau$ is the torque applied by the motor.
This effectively creates a speed-dependent torque limit. As shown in prior
work [1, 15], mismatch in the actuator modeling is one of the major sources of
the ”Reality Gap”. Second, the terrain of the testing environment is modified
to be a sequence of steps (0.5m $\times$ 0.03m) going upward, while the
training environment uses a flat ground.

To better evaluate the transfer performance of different algorithms, we
further vary the physical properties during the evaluation in addition to the
gaps introduced above. For the Hopper task, we vary the torso mass between
$[2.0,7.0]$ kg. For the HalfCheetah task in MuJoCo, we vary the restitution
coefficient of the ground in $[0.5,1.0]$. For the Walker2D task, we vary the
mass of one foot in $[2.0,15.0]$ kg. And for the A1 robot, we vary the power
limit in $[50,100]$.

TABLE III: Performance of different methods for all examples. Numbers in
parenthesis is the average number of failed trials during adaptation.

| Ours | DR | DR-RE | NO-OSSE | CPO | Safe-Bayes
---|---|---|---|---|---|---
Hopper | 1700.5 | 402.0 | 681.1 | 1081.4 | 966.4 | 2092.8 (7)
HalfCheetah | 1402.2 | 290.2 | 538.1 | 1389.6 | 193.2 | 1536.9 (9.2)
Walker2D | 1656.4 | 1000.4 | 1339.1 | 1383.4 | 626.4 | 1947 (9.3)
A1 | 1281.6 | 475.18 | 550.9 | 1245.6 | 361.3 | 1615.75 (10.3)

###  IV-C Baseline methods

We compare our method to four baseline methods.

1) Dynamics Randomization (DR): a commonly used sim-to-real technique where a
robust policy is trained with randomized dynamics.

2) Dynamics Randomization with Reward Engineering (DR-RE): we increase the
alive bonus reward by four times to encourage more robust behaviors.

3) Safe-Bayes: we train $\pi_{task}$, $\pi_{protect}$, and OSSE the same way
as in our algorithm and apply Bayesian Optimization to search for the
thresholds $\kappa_{safe}$ and $\kappa_{task}$ instead of using our protective
adaptation algorithm.

4) Constrained Policy Optimization (CPO): a safe reinforcement learning
algorithm by Achiam et al. , which solves a task policy that satisfies the
defined safety constraints [5].

We further ablate our proposed method by using the $V_{protect}$ instead of
OSSE for estimating the safety level of the robot, which is denoted as NO-
OSSE.

###  IV-D Does our method outperform baseline methods?

We report the mean task performance of different algorithms in the target
environments, as summarized in Table III. Since the policies are tested on
different dynamics parameters during testing, we further plot the policy
performance with respect to the varied parameters in Figure 3 and Figure 4. As
shown in these results, our proposed method achieves notably better task
performance than DR, DR-RE, and CPO. This shows the importance of adapting the
policy in a novel testing environment. We note that it is possible to fine-
tune these policies in the testing environment directly using DRL algorithm.
However, we did not perform this test because 1) the fine-tuning process will
likely introduce many unsafe trials and 2) prior work demonstrated that fine-
tuning a DR policy is not very effective when the environment gap is large
[36]. Meanwhile, applying Bayesian Optimization to find $\kappa_{task}$ and
$\kappa_{safe}$ (Safe-Bayes) can in general achieve better task performance
than our algorithm. This suggests that the shifting between task performance
and robot protection is not simply monotonic as our algorithm hypothesize.
However, as shown in Figure 3 and Table III, Safe-Bayes results in a higher
number of failed trials during adaptation. In our experiments, we allow at
most $20$ trials during Bayesian Optimization in Safe-Bayes. We have also
examined Safe-Bayes using different sample allowance and find that our method
achieves better performance when the same budget of unsafe trials is allowed.

In addition, when the OSSE model is replaced with $V_{protect}$ (NO-OSSE), it
results in worse performance for Hopper and Walker2D, but comparable results
for HalfCheetah and A1. We hypothesize that this is because when the robot is
less stable, it will rely more heavily on taking actions from $\pi_{protect}$
to remain safe. As a result, a state that has high score from $V_{protect}$
may not be safe when we take an action from $\pi_{task}$, leading to lower
task reward. On the other hand, for robots that are structurally more stable,
$V_{protect}$ can provide a more reasonable estimation of the robot’s safety,
leading to similar performance between using OSSE and $V_{protect}$.

Figure 4: Comparison of our method and the baseline methods for the Walker2D
example. Figure 5: Visualization of one rollout of Hopper task where the blue
solid curve represents the policy selection and the dashed curves are the
scaled torso pitch and forward velocity of the robot.

###  IV-E Does our transfer scheme make reasonable policy assignment?

In this section, we examine the policy assignment strategy that emerges from
our method. We apply our method on the MuJoCo Hopper with torso weight of 2.5
kg and record the observations and the policy selection. We then train a
logistic regression model to predict the policy selection from the
observations. The logistic model shows that torso pitch and forward velocity
of the Hopper are the two most predictive features for the policy assignment
(Figure 5 (a)). For better visualization, we scale down the values for torso
pitch and forward velocity. Our policy tends to choose $\pi_{protect}$ when
the forward velocity is high and is tilting forward. This makes sense because
the combination of high forward velocity and forward tilting indicates that
the robot might fall forward.

Figure 2 shows some examples of our trained policies transferred to the
testing environments. The resulting scheduling can be better seen in the
supplementary video.

###  IV-F Generalization Comparison between $\pi_{task}$ and $\pi_{protect}$

TABLE IV: Generalization comparison for task and protective policy.

| Normalized rollout length | Return transfer ratio
---|---|---
| Hopper | Cheetah | Walker | Hopper | Cheetah | Walker
$\pi_{task}$ | 0.15 | 0.3 | 0.17 | 0.29 | 0.14 | 0.39
$\pi_{protect}$ | 0.75 | 1.0 | 0.97 | 0.99 | 0.94 | 0.95

A key assumption we make in our method is that $\pi_{protect}$ can generalize
to new scenarios better than $\pi_{task}$. To validate this, we use two
metrics to compare $\pi_{protect}$ and $\pi_{task}$. The first one is how long
the policy keeps the robot safe. To compute this metric, we run both policies
for $200$ rollouts and compute the average rollout lengths in the testing
environments. We normalize the lengths using the maximal rollout length
(1000). The results can be found in the left of Table IV. The second metric we
use is the ratio of the policy return between testing and training
environments, where the return corresponds to the training reward of the
policies. We report the average of return ratio over $200$ rollouts. The
results is shown in the right half of Table IV. We sample source and target
environment from the same variations described in Section IV-A when measuring
both metrics. We can see that $\pi_{protect}$ achieves better performance in
both metrics. This is likely because $\pi_{protect}$ is trained to specialize
in staying within safe states. As a result, $\pi_{protect}$ learns to take the
robot to stable states such as standing upright, which is less sensitive to
different dynamics.

##  V Discussion and Conclusion

We have presented a transfer learning algorithm for protectively adapting a
control policy to novel scenarios. Our algorithm trains three models in the
source environment: a task policy that optimizes the task reward, a protective
policy that protects the robot form unsafe states, and a safety estimation
model for assessing the safety-level of the robot. To transfer these models to
novel testing scenarios, we introduce a protective adaptation algorithm that
minimizes the unsafe trials during adaptation. We demonstrate that our
algorithm can overcome large modeling errors for four different simulated
transfer problems and achieve better performance than baseline methods in
safely adapting the policy to new environments.

Our algorithm achieves protective policy transfer at the cost of exhibiting
conservative behaviors with suboptimal task performance. An important future
direction is thus to improve the task performance of our algorithm while
keeping the unsafe trials low during transfer. Another interesting future
direction is to extend our approach to more diverse control problems such as
navigation or manipulation.

## References

  * [1] J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke, “Sim-to-real: Learning agile locomotion for quadruped robots,” in _Proceedings of Robotics: Science and Systems_ , Pittsburgh, Pennsylvania, June 2018.
  * [2] W. Yu, J. Tan, Y. Bai, E. Coumans, and S. Ha, “Learning fast adaptation with meta strategy optimization,” _arXiv preprint arXiv:1909.12995_ , 2019.
  * [3] X. B. Peng, E. Coumans, T. Zhang, T.-W. Lee, J. Tan, and S. Levine, “Learning agile robotic locomotion skills by imitating animals,” _arXiv preprint arXiv:2004.00784_ , 2020.
  * [4] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in _Intelligent Robots and Systems (IROS), 2017 IEEE/RSJ International Conference on_. IEEE, 2017, pp. 23–30.
  * [5] J. Achiam, D. Held, A. Tamar, and P. Abbeel, “Constrained policy optimization,” in _Proceedings of the 34th International Conference on Machine Learning-Volume 70_. JMLR. org, 2017, pp. 22–31.
  * [6] C. Tessler, D. J. Mankowitz, and S. Mannor, “Reward constrained policy optimization,” in _International Conference on Learning Representations_ , 2019. [Online]. Available: https://openreview.net/forum?id=SkfrvsA9FX
  * [7] A. D. Ames, S. Coogan, M. Egerstedt, G. Notomista, K. Sreenath, and P. Tabuada, “Control barrier functions: Theory and applications,” in _2019 18th European Control Conference (ECC)_. IEEE, 2019, pp. 3420–3431.
  * [8] F. Berkenkamp, M. Turchetta, A. Schoellig, and A. Krause, “Safe model-based reinforcement learning with stability guarantees,” in _Advances in neural information processing systems_ , 2017, pp. 908–918.
  * [9] W. Yu, V. C. Kumar, G. Turk, and C. K. Liu, “Sim-to-real transfer for biped locomotion,” _arXiv preprint arXiv:1903.01390_ , 2019.
  * [10] A. Cully, J. Clune, D. Tarapore, and J.-B. Mouret, “Robots that can adapt like animals,” _Nature_ , vol. 521, no. 7553, p. 503, 2015.
  * [11] I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas _et al._ , “Solving rubik’s cube with a robot hand,” _arXiv preprint arXiv:1910.07113_ , 2019.
  * [12] J. Siekmann, S. Valluri, J. Dao, F. Bermillo, H. Duan, A. Fern, and J. Hurst, “Learning Memory-Based Control for Human-Scale Bipedal Locomotion,” in _Proceedings of Robotics: Science and Systems_ , Corvalis, Oregon, USA, July 2020.
  * [13] Z. Xie, P. Clary, J. Dao, P. Morais, J. Hurst, and M. van de Panne, “Learning locomotion skills for cassie: Iterative design and sim-to-real,” ser. Proceedings of Machine Learning Research, L. P. Kaelbling, D. Kragic, and K. Sugiura, Eds., vol. 100. PMLR, 30 Oct–01 Nov 2020, pp. 317–329. [Online]. Available: http://proceedings.mlr.press/v100/xie20a.html
  * [14] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-to-real transfer of robotic control with dynamics randomization,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 1–8.
  * [15] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter, “Learning agile and dynamic motor skills for legged robots,” _Science Robotics_ , vol. 4, no. 26, p. eaau5872, 2019.
  * [16] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, N. Ratliff, and D. Fox, “Closing the sim-to-real loop: Adapting simulation randomization with real world experience,” in _2019 International Conference on Robotics and Automation (ICRA)_ , 2019, pp. 8973–8979.
  * [17] J. Yang, B. Petersen, H. Zha, and D. Faissol, “Single episode policy transfer in reinforcement learning,” in _International Conference on Learning Representations_ , 2020. [Online]. Available: https://openreview.net/forum?id=rJeQoCNYDS
  * [18] J. Garcia and F. Fernández, “Safe exploration of state and action spaces in reinforcement learning,” _Journal of Artificial Intelligence Research_ , vol. 45, pp. 515–564, 2012.
  * [19] A. Hans, D. Schneegaß, A. M. Schäfer, and S. Udluft, “Safe exploration for reinforcement learning.” in _ESANN_ , 2008, pp. 143–148.
  * [20] T. J. Perkins and A. G. Barto, “Lyapunov design for safe reinforcement learning,” _Journal of Machine Learning Research_ , vol. 3, no. Dec, pp. 803–832, 2002.
  * [21] A. Aswani, H. Gonzalez, S. S. Sastry, and C. Tomlin, “Provably safe and robust learning-based model predictive control,” _Automatica_ , vol. 49, no. 5, pp. 1216–1226, 2013.
  * [22] R. Cheng, G. Orosz, R. M. Murray, and J. W. Burdick, “End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 3387–3395.
  * [23] Y. Zhang, Q. Vuong, and K. W. Ross, “First order optimization in policy space for constrained deep reinforcement learning,” _arXiv preprint arXiv:2002.06506_ , 2020.
  * [24] J. Schreiter, D. Nguyen-Tuong, M. Eberts, B. Bischoff, H. Markert, and M. Toussaint, “Safe exploration for active learning with gaussian processes,” in _Joint European conference on machine learning and knowledge discovery in databases_. Springer, 2015, pp. 133–149.
  * [25] F. Berkenkamp, A. P. Schoellig, and A. Krause, “Safe controller optimization for quadrotors with gaussian processes,” in _2016 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2016, pp. 491–496.
  * [26] J. Fan and W. Li, “Safety-guided deep reinforcement learning via online gaussian process estimation,” _arXiv preprint arXiv:1903.02526_ , 2019.
  * [27] B. Eysenbach, S. Gu, J. Ibarz, and S. Levine, “Leave no trace: Learning to reset for safe and autonomous reinforcement learning,” _arXiv preprint arXiv:1711.06782_ , 2017.
  * [28] S. Ha and C. K. Liu, “Multiple contact planning for minimizing damage of humanoid falls,” in _2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2015, pp. 2761–2767.
  * [29] K. Fujiwara, F. Kanehiro, S. Kajita, K. Kaneko, K. Yokoi, and H. Hirukawa, “Ukemi: Falling motion control to minimize damage to biped humanoid robot,” in _IEEE/RSJ international conference on Intelligent robots and systems_ , vol. 3. IEEE, 2002, pp. 2521–2526.
  * [30] K. Fujiwara, S. Kajita, K. Harada, K. Kaneko, M. Morisawa, F. Kanehiro, S. Nakaoka, and H. Hirukawa, “Towards an optimal falling motion for a humanoid robot,” in _2006 6th IEEE-RAS International Conference on Humanoid Robots_. IEEE, 2006, pp. 524–529.
  * [31] V. C. Kumar, S. Ha, and C. K. Liu, “Learning a unified control policy for safe falling,” in _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2017, pp. 3940–3947.
  * [32] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * [33] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” _arXiv preprint arXiv:1606.01540_ , 2016.
  * [34] J. Lee, M. X. Grey, S. Ha, T. Kunz, S. Jain, Y. Ye, S. S. Srinivasa, M. Stilman, and C. K. Liu, “Dart: Dynamic animation and robotics toolkit,” _The Journal of Open Source Software_ , vol. 3, no. 22, p. 500, 2018.
  * [35] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in _Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on_. IEEE, 2012, pp. 5026–5033.
  * [36] W. Yu, C. K. Liu, and G. Turk, “Policy transfer with strategy optimization,” in _International Conference on Learning Representations_ , 2019. [Online]. Available: https://openreview.net/forum?id=H1g6osRcFQ