##  1 Introduction

Bipedal robots have the capability to cover a span of terrains that humans can
traverse. This makes bipeds an appealing locomotion option for robots in human
environments. It is impractical, however, to design a single controller that
works over all required environments, particularly if all conditions are not
known a priori. For robots to be able to perform a number of diverse
behaviours, it is desirable for a control suite to be modular, where any
number of behaviours can be added with minimal retraining. Deep Reinforcement
Learning (DRL) is an appealing method for developing visuo-motor behaviours
for legged platforms. However, such DRL policies are usually trained in the
same on the terrain they are expected to operate, though it is unlikely that
the agent would have access to a complete set of terrain conditions during
policy training before deployment. Controllers for dynamic platforms are
typically developed and tuned with harnesses and treadmills safety harnesses
and human supervisors in order to minimise potential damage to the robot and
the environment. These restrictions limit the development of controllers over
an exhaustive collection of terrain conditions. Furthermore, it is preferable
to not re-design modifying controllers that have already been tuned , whereas
is costly, for example DRL methods typically require retraining if the target
conditions are evolved to included new terrains new terrains are introduced.
We propose training a Setup Policysetup policy to transition between two pre-
trained policies targeting different terrains. A modular control suite for a
dynamic platform requires careful design. Simply switching between controllers
may result in the robot falling over if controllers are switched when the
robot is not in a safe region for the subsequent controller. Figure 1 shows
our primary contribution: a Setup Policy setup policy that prepares the robot
for the necessary controller to traverse the upcoming terrain obstacle. With
our method, we can transition between existing behaviours, allowing pre-
trained DRL policies to be combined.

[width=]images/Fig1.png

Figure : As the robot approaches a difficult terrain artifact, it must move
into position for target policy $j$$i$. The trajectory of the walking policy
$i$ $j$ (shown with a blue line), does not intersect with the trajectory of
the target policy $j$ $i$ (green line). Our setup policy provides a trajectory
that prepares the robot for the upcoming target policy (orange line).

Our contributions are as follows:

  * •

Using pre-trained policies, we develop a Setup Policysetup policies that
significantly improves improve the success rate from transitioning from a
default walking policy to a target policy (from 1.5$\%$ success rate without
Setup Policies setup policies to 82$\%$ for a difficult 0.70.5m jump). The
setup policy also learns when to switch to the target policy.

  * •

We introduce a novel reward, called Advantage Weighted Target Value, based on
the performance of the subsequent guiding the robot towards the target policy.

  * •

We show that we can use our setup policies with several difficult terrain
types, allowing for a modular control suite, combining policies behaviours
that have been trained separately without any retraining of low level
policies.

##  2 Related Work

Deep reinforcement learning (DRL) has demonstrated impressive results for
locomotion tasks in recent works [schulman_proximal_2017,
heess_emergence_2017, peng_deeploco_2017, peng_deepmimic_2018,
xie_allsteps_2020][heess_emergence_2017, peng_deeploco_2017,
peng_deepmimic_2018, xie_allsteps_2020]. Typically DRL policies optimise a
single reward function, as new environments and behaviours are introduced
costly retraining is required. Single policies have demonstrated locomotion
over various terrains for simple 2D cases [song_recurrent_2018], and
impressive quadruped behaviours [lee_learning_2020, rudin_learning_2021],
however, for many scenarios multiple policies are often required. Furthermore,
developing a single policy to perform multiple behaviours can degrade the
performance of each behaviour [lee_robust_2019]. Hierarchical reinforcement
learning (HRL) offers flexibility through architecture, involving by training
all segments of the hierarchy concurrently [sutton_between_1999, bacon_option-
critic_2018, frans_meta_2018, peng_deeploco_2017], or training parts of the
hierarchy separately [merel_hierarchical_2019], [lee_composing_2019]. When
trained together, HRL can improve task level outcomes, such as steering and
object tracking [peng_deeploco_2017], or improve learning efficiency by
reusing low level skills across multiple high level tasks [frans_meta_2018].
When trained separately, low level controllers can be refined efficiently
using prior knowledge, such as by utilising motion capture data
[peng_deepmimic_2018, merel_hierarchical_2019, peng_mcp_2019] or behavioural
cloning [strudel_learning_2020]. For robotics applications it may be difficult
to develop controllers for multiple behaviours simultaneously. Using pre-
trained primitives can break up a large difficult problem into smaller
solvable sub-problems [schaal_dynamic_2006]. Using pre-trained policies
requires a suitable handling of transitions between behaviours. Faloutsos et
al. [faloutsos_composable_2001] learns pre and post-conditions for each
controller, such that switching only occurs when these conditions have been
satisfied. DeepMimic by Peng et al. [peng_deepmimic_2018] combines pre-trained
behaviours learned from motion capture, with reliance on a phase variable to
determine when one behaviour has been completed. Policy sketches introduce a
hierarchical method that uses task specific policies, with each task performed
in sequence [andreas_modular_2017]. CompILE uses soft boundaries between task
segments [kipf_compile_2019]. Work by Peng et al. [peng_terrain-adaptive_2016]
trains several actor-critic control policies, modulated by the highest critic
value in an a given state. For these examples there must be a reasonable state
overlap between sequential controllers. Other methods that combine pre-trained
behaviours learn latent representations of skills [pertsch_accelerating_2020]
or primitives [ha_distilling_2020], enabling interpolation in the latent
space. From an offline dataset of experience, Pertsch et al
[pertsch_accelerating_2020] are . [pertsch_accelerating_2020] were able to
combine low level controllers in manipulation tasks and locomotion task for a
multi-legged agent. Ha et al. [ha_distilling_2020] utilise motion capture to
learn latent representations of primitives, then use model predictive control
for the high level navigation of to navigate with a high dimensional humanoid.
The FeUdal approach learns a master policy that modulates low-level policies
using a learned goal signal [vezhnevets_feudal_2017]. Work by Peng et al.
[peng_mcp_2019] combines pre-trained policies using a gating function that
learns a multiplicative combination of actions. In our previous work
[tidd_learning_2021], we learn when to switch between low level primitives for
a simulated biped using data collected by randomly switching between
behaviours. Interpolation between behaviours yields natural transitions
[xu_hierarchical_2020], however in . Yang et al. [yang_multi_2020] learn a
gating neural network to blend several separate expert neural network policies
to perform trotting, steering, and fall recovery in real-world experiments
with a quadruped. Da et al. [da_supervised_2017] use supervised learning to
train a control policy for a biped from several manually derived controllers
to perform periodic and transitional gaits on flat and sloped ground. In each
of these approaches, experience from all behaviours must be available during
training. For dynamic platforms, where separate controllers occupy different
subsets of the state, changing behaviours may not result in stable behaviour
result in instability if there is no overlap between controllers. Lee et al.
[lee_composing_2019] learn a proximity predictor to train a transition policy
to get guide an agent to the initial state required by the next controller.
Locomotion experiments are demonstrated with a 2D planar walker, where
learning occurs by training with all terrain conditions. Our setup policies
also use an intermediate controller, however we We show that our method
performs more reliably (with a higher success rate)while performing complex
behaviours with a 3D biped, despite only , despite training with experience
from a single terrain obstacle for each behaviourtype. Our setup policies
learn to transition between complex behaviours with a 3D biped in a simulation
environment using a depth image to perceive the terrain.

##  3 Method

In this section we describe the problem we are solving and our method for
training setup policies.

[][width=]images/Fig2.png [][width=]images/Fig3.png

Figure : a) shows the modular pipeline. A terrain classifier selects which
module should be utilised, and the torque from the output of the selected
module is applied to the robot, returning robot state and depth image. b) Each
module is pre-trained on a single terrain type. The Target Policy target
policy has been designed to traverse a specific terrain condition. The Setup
Policy setup policy guides the trajectory of the robot from a default walking
controller to the Target Policytarget policy, also learning when to switch to
the Target Policytarget policy.

###  Problem Description

We investigate composing controllers for a dynamic platform with little narrow
or no state overlap between adjacent behaviours. An example of where this
occurs when a bipedal robot performs a jump over a large block. In this
scenario, the robot walks up to the block using a walking policy and then
performs a jump with a terrain-specific Target Policytarget policy, where the
behaviour involves jumping with both feet. Fig 1 shows that the trajectory of
the walk policy does not intersect with the trajectory of the Target
Policytarget policy, therefore we require an intermediate policy. While it may
be possible to improve the robustness of the target policies after they have
been trained (i.e. to include states visited by the walk policy), we consider
the case where target behaviours and value functions are provided as a black
box and are therefore immutable. Our agent has 12 torque controlled actuators,
simulated in PybulletPyBullet [coumans_pybullet_2020]. We select which terrain
module to employ using an oracle terrain classifier (Fig 3a). From a default
walking policy, we utilise a Setup Policy setup policy that prepares the robot
for the Target Policy target policy (Fig 3b)). When to switch from the Setup
Policy to the Target Policy setup policy to the target policy is also a
learned output of the Setup Policysetup policy. We first study the traversal
of a single difficult terrain artifact that requires walking on a flat surface
and performing a jump. We then consider several diverse terrain types (gaps,
hurdles, stairs, and stepsstepping stones). The state provided to each policy
is $s_{t}=[rs_{t},I_{t}]$, where $rs_{t}$ is the robot state and $I_{t}$ is
the perception input at time $t$. Robot state:
$rs_{t}=[J_{t},Jv_{t},c_{t},c_{t-1},v_{CoM,t},\omega_{CoM,t},\\\
\theta_{CoM,t},\phi_{CoM,t},h_{CoM,t},s_{right},s_{left}]$, where $J_{t}$ are
the joint positions in radians, $Jv_{t}$ are the joint velocities in rad/s,
$c_{t}$ and $c_{t-1}$ are the current and previous contact information of each
foot , respectively (four Boolean contact points per foot), $v_{CoM,t}$ and
$\omega_{CoM,t}$ are the linear and angular velocities of the body Centre of
Mass (CoM)robot body, $\theta_{CoM,t}$ and $\phi_{CoM,t}$ are the pitch and
roll angles of the CoM robot body and $h_{CoM,t}$ is the height of the robot
from the walking surface. $s_{right}$ and $s_{left}$ are Boolean indicators of
which foot is in the swing phase, and is updated when the current swing foot
makes contact with the ground. All angles except joint angles are represented
in the world coordinate frameRobot body rotations are provided as Euler
angles. In total there are $51$ elements to the robot state, which is
normalised by subtracting the mean and dividing by the standard deviation for
each variable (statistics are collected as an aggregate during training).
Perception: Perception is a depth sensor mounted to the robot base, with a
resolution of $[48,48,1]$, and field of view of 60 degrees. Each pixel is a
continuous value scaled between $0-1$, measuring a distance between 0.25 and
$\SI{2}{\metre}$ from the robot base. The sensor moves with the body in
translation and yaw, we provide an artificial gimbal to keep the sensor fixed
in roll and pitch. The sensor is pointed at the robot’s feet (downwards 60
degrees so the feet are visible) and covers at least two steps in front of the
robot [zaytsev_two_2015]. We reduce the sampling rate of the perception to
$\SI{20}{\hertz}$ (reducing computational complexity), where the simulator
operates at $\SI{120}{\hertz}$.

###  Deep Reinforcement Learning for Continuous Control

We consider our task to be a Markov Decision Process $\text{MDP}$, defined by
tuple $\\{\mathcal{S},\mathcal{A},R,\mathcal{P},\gamma\\}$
$\\{\mathcal{S},\mathcal{A},\mathcal{R},\mathcal{P},\gamma\\}$ where
$s_{t}\in\mathcal{S}$, $a_{t}\in\mathcal{A}$, $r_{t}\in R$
$r_{t}\in\mathcal{R}$ are state, action and reward observed at time $t$,
$\mathcal{P}$ is an unknown transition probability from $s_{t}$ to $s_{t+1}$
taking action $a_{t}$, and applying discount factor $\gamma$. The goal of
reinforcement learning is to maximise the sum of future rewards
$R=\sum_{t=0}^{T}\gamma^{t}r_{t}$, where $r_{t}$ is provided by the
environment at time $t$. Actions are sampled from a deep neural network policy
$a_{t}\sim\pi_{\theta}(s_{t})$, where $a_{t}$ is a torque applied to each
joint the joint commands. Each policy is trained with Proximal Policy
Optimisation (PPO) [schulman_proximal_2017]. We use the implementation of PPO
from OpenAI Baselines [dhariwal_openai_2017].

###  Training Setup Policies

For each target terrain type we train a Setup Policy setup policy with the
purpose of bridging the trajectories of a default walking policy and the
policy for behaviour for traversing the upcoming terraincondition. The
algorithm for training each Setup Policy setup policy is shown in Algorithm 4.
Default Policy: The Default Policy default policy is a pre-trained policy
designed to walk on a flat surface. When training a Setup Policy setup policy
we always begin with the Default Policy default policy until the terrain
artifact is detected by an oracle. Once the terrain has been traversed, we
switch back to the Default Policy default policy when a termination criteria
is reached ($\tau_{\theta}$, defined below). We initialise the Setup Policy
setup policy from the default walking policy such that the robot continues to
walk towards the obstacle while learning how to prepare for the Target
Policytarget policy. Setup Policy: When terrain type $i$ is detected, we
switch immediately from the Default Policy to the Setup Policy default policy
to the setup policy for terrain type $i$, denoted as $\pi_{\phi}^{i}$. While
training we are given access to an oracle terrain detector that indicates when
the robot is a fixed distance from the terrain artifact. The Setup Policy
setup policy outputs joint torques $a_{i}$, and switch condition
$\tau_{\phi}$. If the switch condition is met, we immediately switch to the
Target Policytarget policy. Target Policy: The Target Policy target policy for
terrain type $i$, denoted as $\pi_{\theta}^{i}$, is trained with curriculum
learning following the method outlined in our previous prior work
[tidd_guided_2020]. We also define a termination condition $\tau_{\theta}$
that denotes switching from the Target Policy target policy back to the
Default Policydefault policy. For the the termination condition to be reached,
the robot must be have successfully traversed the terrain artifact,
successfully traverse the terrain obstacle and have both feet in contact with
the groundplane, and a positive forward velocity $vx$. This condition is the
same for all artifact types. Setup Policy Reward: We design a reward function
to motivate the Setup Policy setup policy to transition the robot from states
visited by the Default Policy default policy to states required by the Target
Policytarget policy. We note the value function of the Target Policy,
$\hat{V}(s_{t})=\sum_{t=0}^{T}\gamma^{t}r_{t}$target policy,
$V^{\pi_{\theta}^{i}}(s_{t})=\sum_{t=0}^{T}\gamma^{t}r_{t}$, (where $r_{t}$ is
the original reward afforded by the environment for the Target Policytarget
policy), provides an estimate of what return we can expect if we run the
Target Policy target policy from $s_{t}$. However, value functions are
notoriously over optimistic over-optimistic for states that have not been
visited [haarnoja_soft_2018] (such as those we might experience by running the
Default Policydefault policy). The advantage is a zero centered estimate of
the effect of running action $a_{t}\sim\pi_{\theta}^{i}(s_{t})$, given by the
advantage for the Target Policycalculation of how much better or worse policy
$\pi$ performs compared to the value function prediction ${V}^{\pi}(s_{t})$:

$\DIFadd{A^{\pi}(s_{t},a_{t})=R_{t}-V^{\pi}(s_{t})}$ (1)

where $R_{t}$ is sum of future rewards from time $t$ running policy $\pi$. We
estimate the advantage using the temporal difference (TD) error with the value
function of the target policy $\pi_{\theta}^{i}$:

$\hat{A}\DIFaddbegin\DIFadd{{}^{\pi_{\theta}^{i}}}\DIFaddend(s_{t},a_{t})=r_{t}+\gamma\DIFdelbegin\DIFdelend\DIFaddbegin\DIFadd{V^{\pi_{\theta}^{i}}}\DIFaddend(s\DIFdelbegin\DIFdel{{}_{t+1)}}\DIFdelend\DIFaddbegin\DIFadd{{}_{t+1})}\DIFaddend-\DIFdelbegin\DIFdelend\DIFaddbegin\DIFadd{V^{\pi_{\theta}^{i}}}\DIFaddend(s_{t})$
(2)

where $r_{t}$ is the Target Policy reward. (details can be found in our
previous work [tidd_guided_2020]). target policy reward. Using advantage
$\hat{A}(s_{t},a_{t})$ as a confidence for
$\hat{A}^{\pi_{\theta}^{i}}(s_{t},a_{t})$ as an indication of the accuracy of
$\hat{V}(s_{t})$$V^{\pi_{\theta}^{i}}(s_{t})$, we define the reward from the
Advantage Weighted Target Value:

$\hat{r}_{t}=(1-\text{min}(\alpha\hat{A}\DIFaddbegin\DIFadd{{}^{\pi_{\theta}^{i}}}\DIFaddend(s_{t},a_{t})^{2},1))\cdot\beta\hat{V}\DIFaddbegin\DIFadd{{}^{\pi_{\theta}^{i}}}\DIFaddend(s_{t})\DIFaddbegin\DIFaddend$
(3)

Where where $\alpha$ and $\beta$ are scaling factors are tuned empirically
(set to 0.15 and 0.01 respectively). The Target Policy value function
$\hat{V}(s_{t})$ target policy value function $V^{\pi_{\theta}^{i}}(s_{t})$
has learned the value for states where the Target Policy target policy has
collected experience. Outside of these states, for example when the robot is
activating a different policy, we expect the reward provided by the
environment $r_{t}$, and the next state $s_{t+1}$ will not be consistent with
what is expected by the Target Policy target policy value function. Therefore,
the advantage $\hat{A}(s_{t},a_{t})$ $\hat{A}^{\pi_{\theta}^{i}}(s_{t},a_{t})$
will not be close to zero, and $\hat{r}_{t}$ becomes small. Intuitively, for
states where the Target Policy value function $\hat{V}(s_{t})$ target policy
value function $V^{\pi_{\theta}^{i}}(s_{t})$ is accurate, the Target Policy
advantage $\hat{A}$ target policy advantage $\hat{A}^{\pi_{\theta}^{i}}$ will
be close to zero, thus the Setup Policy setup policy will learn to maximise
$\hat{V}$$V^{\pi_{\theta}^{i}}$. States where $\hat{A}$
$\hat{A}^{\pi_{\theta}^{i}}$ is far from zero will reduce the effect of an
overconfident $\hat{V}$$V^{\pi_{\theta}^{i}}$. Extended Reward: One issue with
training Setup Policies setup policies is the delayed effect of actions, where
once switched to the Target target policy, the Setup Policy setup policy is no
longer acting on the environment or receiving a reward. Once the Setup Policy
setup policy has transitioned to the Target Policytarget policy, we must
provide the Setup Policy setup policy with information regarding the
performance of the robot since switching. Our method for solving this problem
is to provide the additional reward obtained after switching, by including an
extended reward in the rollout buffer of the Setup Policysetup policy. We add
this extended reward to the last element of the Setup Policy bufferreward
received running the setup policy:

$\hat{r}_{final}=\hat{r}_{final}+\hat{r}_{t}$ (4)

where $\hat{r}_{final}$ is the final reward entry received by the Setup
Policysetup policy before switching, and $\hat{r_{t}}$ is the reward received
at time $t$, after the Setup Policy setup policy has transitioned to the
Target Policytarget policy. For algorithmic stability, we clear all but the
last entry of the buffer after each training update. The last reward entry of
the buffer becomes $\hat{r}_{final}$ for environment steps that continue after
training. The procedure for training Setup Policies setup policies is provided
in Algorithm 4. Note that the termination variable $\tau_{t}$ (line 7) is
either $\tau_{\phi}$ if the Current Policy is the Setup Policycurrent policy
is the setup policy, or $\tau_{\theta}$ if the Current Policy is the Target
Policycurrent policy is the target policy. Setup Before Switching [1] Load
Target Policy $\pi_{\theta}^{i}$. Load Default Policy $\pi_{\theta}^{j}$
Initialise Setup Policy from $\pi_{\phi}^{i}$ from the Default Policy
Initialise Buffer Current Policy $\leftarrow$ Default Policy
$iteration=1,2,\ldots$ not $done$ $a_{t},\tau_{t}\sim$ Current Policy$(s_{t})$
$\tau$ is $\tau_{\phi}$ or $\tau_{\theta}$
$s_{t+1},r_{t},done=env.step(a_{t})$ Terrain detected
$\hat{A}(s_{t},a_{t})=r_{t}+\gamma\hat{V}(s_{t+1})-\hat{V}(s_{t})$
$\hat{A}^{\pi_{\theta}^{i}}(s_{t},a_{t})=r_{t}+\gamma
V^{\pi_{\theta}^{i}}(s_{t+1})-V^{\pi_{\theta}^{i}}(s_{t})$
$\hat{r}_{t}=(1-\alpha\hat{A}(s_{t},a_{t})^{2})\cdot\beta\hat{V}(s_{t})$
$\hat{r}_{t}=(1-\text{min}(\alpha\hat{A}^{\pi_{\theta}^{i}}(s_{t},a_{t})^{2},1))\cdot\beta
V^{\pi_{\theta}^{i}}(s_{t})$ Current Policy == Default Policy Current Policy
$\leftarrow$ Setup Policy $\pi_{\phi}^{i}$ $\tau_{t}\leftarrow\tau_{\phi}$
Current Policy == Setup Policy Store $s_{t}$, $a_{t}$, $\hat{r}_{t}$,
$\tau_{\phi}$ into Buffer Buffer Full PPO Training Update Clear buffer (except
last entry) $\tau_{\phi}$ Current Policy $\leftarrow$ Target Policy
$\pi_{\theta}^{i}$ $\tau_{t}\leftarrow\tau_{\theta}$ $\tau_{\theta}$ Current
Policy $\leftarrow$ Default Policy $\pi_{\theta}^{j}$ Current Policy != Setup
Policy $\hat{r}_{final}=\hat{r}_{final}+\hat{r}_{t}$ $s_{t}=s_{t+1}$

##  4 Experiments

We evaluate our method with a biped in simulation. Initial evaluations are
performed with a single difficult terrain (Sec 4, Sec 4, Sec 4), before adding
more behaviours in Sec 4. The difficult terrain is a sample of flat terrain
with a single block 70cm 50 cm high, and 30cm 30 cm in length (in the forward
direction of the robot), an example is shown in Fig. 1. We perform training
and evaluation on a single terrain sample, starting the robot from a random
$x$ (forward direction) uniformly sampled from $(0.0,2.2)\si{\meter}$ (0.0,
2.2) $\si{\meter}$ before the terrain artifact, $y$ (strafe) uniformly sampled
from $(-0.6,0.6)\si{\meter}$, and heading (-0.6, 0.6) $\si{\meter}$, and
$heading$ (yaw) uniformly sampled from $(-0.3,0.3)\si{\radian}$(-0.3, 0.3)
$\si{\radian}$. All experiments using a single terrain type are evaluated by
pausing training and performing 100 evaluation runs, and recording both the
Success percentage and Distance percentage. Success % is defined as the
average success rate to reach a goal location several meters past the terrain
artifact. Distance % is the average percentage of total terrain distance the
agent successfully traversed of the terraincovered.

###  Effect of Using Initialisation and Extended Reward

We perform an ablation to study the effect of initialising the Setup Policy
setup policy from the walking policy, and the effect of extending the reward
after the Setup Policy setup policy has transitioned to the Target
Policytarget policy. In all experiments the Default Policy default policy is
used until the terrain is detected, then the Setup Policy setup policy is
activated until the termination signal $\tau_{\phi}$ indicates switching to
the Target Policytarget policy, and $\tau_{\theta}$ then results in switching
back to the Default Policydefault policy.

  * [leftmargin=*]

  * •

Without Initialisation: We evaluate our method without initialising the Setup
Policy from the Default Walk Policysetup policy from the default walking
policy, i.e. the Setup Policy setup policy is randomly initialised.

  * •

Without Extended Reward: We evaluate our method without including the reward
after the Setup Policy setup policy has switched to the Target Policytarget
policy.

  * •

Full Method: Our method uses a Setup Policy setup policy initialised from the
Default Walk Policydefault walk policy, and receives reward signal after the
Setup Policy setup policy has transitioned to the Target Policytarget policy.

min width=0.8  Success % Distance % Without Initialisation 38.8 73.2 Without
Extended Reward 12.3 58.3 Full Method 82.2 96.1

Table : Ablation study for Initialising initialising setup policies from Walk
Policy a default walk policy, and Extended Rewardsreceiving an extended
reward, when for switching from a walking policy to a jump policy on a single
terrain sample.

We can see that initialising the Setup Policy setup policy with the default
walking policy and extending the reward improves learning outcomes as shown in
Table 4, where our method achieves 82.2% success compared to 38.8% (without
initialisation from the default walking policy ) and 12.3% (without extended
reward)without receiving the extended reward.

###  Setup Policy Reward Choice

For training the setup policy, we investigate several options for the reward
provided at each timestep. For each reward type we evaluate with three
different random seeds, results are shown in Fig 4.

[width=]images/results.png

Figure : We investigate several options for the reward function used by the
Setup Policy to train a setup policy on the difficult jump terrain. We show
the percentage of successful runs success rate (robot reaches several meters
after the jump) from during training for three different random seeds.

  * [leftmargin=*]

  * •

Original: The reward afforded by the environment that is used to train the
Target Policytarget policy: the reward encourages following an expert
motion(See [tidd_guided_2020] for details of the reward function).

  * •

Constant: At each timestep a constant reward of 1.5 is given.

  * •

Target Torque: We encourage the setup policy to match the output of the Target
Policy torque target policy by minimising the errorbetween the Setup Policy
and the Target Policy:
$\exp[-2.0\cdot(\pi_{\phi}(s_{t})-\pi_{\theta}(s_{t}))^{2}]$.

  * •

Target Value: We evaluate the effect of using the Target Policy target policy
value function as the reward for the Setup Policysetup policy. This is not the
same as the original reward, as it is not provided by the environment, but by
$\beta\hat{V}(s_{t})$$\beta V^{\pi_{\theta}^{i}}(s_{t})$, where $\beta$ is
scaling factor 0.01.

  * •

Advantage Weighted Target Value: We evaluate the reward based on the Target
Policy advantage $\hat{A}$ and value function $\hat{V}$:
$(1-\text{min}(\alpha\hat{A}(s_{t},a_{t})^{2},1))\cdot\beta\hat{V}(s_{t})$,
where $\alpha$ and $\beta$ are scaling factors 0.15 and 0.01
respectivelyintroduced in Equation 3.

It can be seen from Fig 4 that a reward using Advantage Weighted Target Value
performs better than other reward choices, achieving the highest average
success rate (78.7%). Using the Target Value target value reaches a success
rate of 63.9%. This disparity, and the large variation in the Target Value
target value (green line), shows the effect of over estimation bias often seen
with value functions, validating our idea that weighting the value by the
advantage reduces the effect of this bias when training Setup Policiessetup
policies.

###  Comparisons

We compare our method with several others, including end to end methods,
without using Setup Policiessetup policies, and a method that uses proximity
prediction to train a Transition Policytransition policy [lee_composing_2019].

min width=0.8  Success% Distance% Single Policy 0.0 11.3 Single Policy With
Curriculum 0.0 3.8 No Setup Policy 1.5 51.3 Learn When to Switch 0.0 44.2
Proximity Prediction 51.3 82.4 Setup Policy (Ours) 82.2 96.1

Table : Success and average distance of the total length covered when
switching from a walking policy to a jump policy on a single terrain sample.

  * [leftmargin=*]

[width=]images/all.png

Figure : Setup policies allow us enable a biped to transition between
behaviours to traverse several complex visuo-motor behaviours for traversing a
sequence of diverse terrains.

  * •

Single Policy: We train a single end to end policy on the terrain sample.

  * •

Single Policy With Curriculum: We utilise curriculum learning to train a
single end to end policy on the terrain sample (using the method defined
outlined in [tidd_guided_2020]).

  * •

No Setup Policy: We switch to the Target Policy from the Default Policy target
policy from the default policy as soon as the terrain oracle detects the next
terrain.

  * •

Learn When to Switch: We collect data by switching from the Default Policy to
the Target Policy default policy to the target policy at random distances from
the Jump. We use supervised learning to learn when the robot is in an a
suitable state to switch. Details of this method is found in our previous work
[tidd_learning_2021].

  * •

Proximity Prediction: We follow the method defined by Lee et al.
[lee_composing_2019] to train Transition Policies (one for each behaviour)
using a proximity predictor function $P(s_{t})$. $P(s_{t})$ outputs a
continuous value that indicates how close the agent is to a configuration that
resulted in successful traversal. $P_{(}s_{t})$ $P(s_{t})$ is trained using
supervised learning from success and failure buffers. The reward used to train
the Transition Policy transition policy is the dense reward created by
$P(s_{t+1})-P(s_{t})$, encouraging the agent to move closer to a successful
switching configurationconfiguration that results in successful switching. For
accurate comparison we initialise the transition policy with weights from the
walk policy, and utilise the terrain oracle for policy selection (the paper
refers to a rule-based meta-policy in place of a terrain oracle
[lee_composing_2019]).

  * •

Setup Policy (Ours): We follow Algorithm 4 to train Setup Policiessetup
policies.

We can see from Table 4 that our method for developing Setup Policies setup
policies performs the best (82.2% success rate), compared to other methods. A
single policy was unable to traverse the difficult jump terrain, even after
with an extended learning time and with curriculum learning. The poor
performance of Single Policy With Curriculum was the result of the robot not
progressing through the curriculum, and as a consequence is unable to move
without assistive forces. We show that Setup Policies setup policies are
necessary for this problem, with No Setup Policy unable to successfully
traverse the terrain (1.5% success). Learning When to Switch also performed
poorly as there are very few states that overlap between the Default Policy
and the Target Policy default policy and the target policy for the switch
estimator to learn. The Proximity Prediction method was able to successfully
traverse the jump 51.3% of the time. It is worth noting that this method
required approximately three times more training episodes than our method to
reach the provided results, despite also being initialised from the Default
Policydefault policy. These experiments show that for the difficult jump
terrain, we require a specialised Target Policy target policy dedicated to
learning the complex jumping behaviour. We show that this specialised policy
has a trajectory that does not overlap with the Default Policydefault policy,
and thus Setup Policies setup policies are required. Our method for learning a
Setup Policy method achieves setup policies achieved the highest performance
on this difficult task.

###  Multiple Terrain Types

For our final experiment we train Setup Policies setup policies for several
additional terrain types. In total we now have modules for jumps, gaps,
hurdles, stairs, and stepsstepping stones. Fig 4 shows a sequence with each
terrain type. We follow the pipeline shown in Fig 3a), where a terrain oracle
determines which module should be selected. On selection, the Setup Policy
setup policy is employed until it determines when to switch to the Target
Policythe robot reaches a switch state ($\tau_{\phi}$), then switching to the
target policy. Termination criteria $\tau_{\theta}$ (introduced in Sec 3)
determines when to switch back to the Default Policy for the Jump Policy
default policy for the jump obstacle only. All other policies resemble the
behaviour of the Default Policy default policy on flat terrain, i.e. we
continue using the Target Policy target policy after the terrain has been
traversed for Gaps, Hurdles, Stairs, and Stepsgaps, hurdles, stairs, and
stepping stones, until the oracle determines the terrain has changed. We
evaluate with 7 randomly selected terrains types in sequence a sequence of
each of the 5 terrain types, randomly shuffled before each run. We perform
1000 runs, with a total of 7000 5000 various terrain types to be traversed. In
Table 4 we show the average percentage of total distance travelled and success
rate with and without using Setup Policiessetup policies. A successful run in
this scenario is defined as traversing the last terrain in the sequence. Table
4 shows the failure count for each terrain type.

min width=0.8  Success % Distance % Without Setup Policies 6.8 1.9  37.1 36.3
With Setup Policies 26.471.2 53.280.2

Table : Success rate and percent percentage of the total distance terrain
length travelled for from 1000 episodes of 7 randomly selected all 5 terrain
types, randomly shuffled each episode.

min width=0.9  Jump Gap Hurdle Stairs Steps Without Setup Policies 710 782  90
52  4336 2031 9086 With Setup Policies 42036 7048 73 43  36 65  144 96

Table : Number of failures by terrain type from 1000 episodes of 7 randomly
selected all 5 terrain types, randomly shuffled each episode.

min width=0.8  Success % Distance % Without Setup Policies 68.2 78.1 With
Setup Policies 76.7 84.6

Table : Success rate and percentage of the total terrain length travelled from
1000 episodes of 4 terrain types (without the jump terrain), randomly shuffled
each episode

min width=0.82  Gap Hurdle Stairs Steps Without Setup Policies 74 41 64 142
With Setup Policies 44 51 49 90

Table : Number of failures by terrain type from 1000 episodes of 4 terrain
types (without the jump terrain), randomly shuffled each episode.

We can see from Table 4 that Setup Policies setup policies improve the success
rate compared to switching policies without the intermediate policy (26.471.2%
success compared to 6.81.9%). Table 4 provides a breakdown of the number of
times the robot failed on each terrain. From Table 4 we can see that Setup
Policies setup policies are most critical for traversing the difficult jump
terrain (reducing the failures from 710 to 420782 to 36), though we also see
improvements for gap terrain. These results show the highest failure rate on
the multi terrain setup is on the jump terrain, and that there is clear room
for improvement on this terrain . While the results show that without setup
policies the hurdle, stairs, and steps terrain appear to have less failures
than running setup policies, we note that these terrains are visited far less
times when running without setup policies due to significant failure with the
jumps terrain To investigate further, we exclude the jump obstacle from the
terrain sequence and still observe a significant benefit using setup policies.
Table 4 shows the performance increase from 68.2% to 76.7% of successful
traversals for 4 terrain types (excluding the jump obstacle). The stepping
stone obstacle had the highest failure rate of the remaining terrain types,
investigation is required to improve the setup policies for this terrain
(Table 4). Despite the clear need for Setup Policiessetup policies, we
attribute the relatively low success rate (26.4%) failures to out of
distribution states, i.e. the robot visits states in the terrain sequence that
it does not experience when training the Setup Policy. For example, if the
robot was on a gap, and using the gaps behaviour, then switches to the jumps
Setup Policy to prepare for an upcoming jump, it will not necessarily be in a
state that has been seen during training. Training Setup Policies setup policy
on the individual terrain types. Training setup policies for wider range of
states will be investigated in future work. Furthermore, the performance of
behaviours on each individual terrain type could be analysed to improve the
generalisation of this method to new behaviours.

##  5 Conclusion

It is common practice for legged robots to have separate locomotion policies
to traverse different terrain conditions. We propose Setup Policies that
allows setup policies that enable smooth transition from the trajectory of one
locomotion policy to a Target Policy for the trajectory of a target policy for
traversing difficult terrain with a dynamic biped. We use deep reinforcement
learning to learn the Setup Policies setup policies for which we introduce a
novel reward based on the Advantage Weighted Target Value, utilising the
scaled value prediction of the Target Policy target policy as the reward for
the Setup Policysetup policy. In simulation experiments for transitioning from
a walking policy to a jumping policy, we show using an in-between Setup Policy
setup policy yields a higher success rate compared to using a single policy
(0$\%$ success rate), or the two policies without the Setup Policy setup
policy (from 1.5$\%$ success rate without Setup Policies setup policies to
82$\%$). We further show that our method is scalable to other terrain types,
and demonstrate the effectiveness of the approach on a sequence of difficult
terrain conditions, improving the success rate from 6.81.9$\%$ without Setup
Policies to 26.4setup policies to 71.2$\%$. A limitation of our method is that
the Setup Policies setup policies are trained from a default walking policy.
We would like to improve the scope of the Setup Policies setup policies such
that a wider range of states are funnelled toward the Target Policytowards the
target policy. If we allowed further training of the Target Policiestarget
policies, our training method could be used to include more states and improve
robustness. This will be considered The idea of utilising the advantage
(estimated by the temporal difference (TD) error) as a confidence metric can
be applied generally for region of attraction (RoA) expansion, safe
reinforcement learning, and also for blending several behaviours for
performing complex composite tasks. These ideas will be explored in future
work. A significant assumption in this work is that we have access to a
terrain oracle, removing this dependency will also be investigated in future
workrequired as we consider experiments in the real world.