#  Walk These Ways: Tuning Robot Control for Generalization with Multiplicity
of Behavior

Gabriel B. Margolis Pulkit Agrawal
Improbable AI Lab, Massachusetts Institute of Technology
{gmargo<EMAIL_ADDRESS>

###### Abstract

Learned locomotion policies can rapidly adapt to diverse environments similar
to those experienced during training but lack a mechanism for fast tuning when
they fail in an out-of-distribution test environment. This necessitates a slow
and iterative cycle of reward and environment redesign to achieve good
performance on a new task. As an alternative, we propose learning a single
policy that encodes a structured family of locomotion strategies that solve
training tasks in different ways, resulting in Multiplicity of Behavior (MoB).
Different strategies generalize differently and can be chosen in real-time for
new tasks or environments, bypassing the need for time-consuming retraining.
We release a fast, robust open-source MoB locomotion controller, Walk These
Ways, that can execute diverse gaits with variable footswing, posture, and
speed, unlocking diverse downstream tasks: crouching, hopping, high-speed
running, stair traversal, bracing against shoves, rhythmic dance, and more.
Video and code release: https://gmargo11.github.io/walk-these-ways/

Figure 1:  Multiplicity of Behavior (MoB) enables a human to tune a single
quadruped policy trained on flat ground to diverse unseen environments. Top
row: A low-frequency gait fails to sprint on slippery terrain (Gait 2; inset)
but tuning it to high frequency results in success (Gait 1). However, a low
frequency and high footswing height are necessary for stair traversal (Gait 2;
middle image). A low footswing and wide stance (Gait 3) makes the robot robust
to leg shoves, but Gait 1, which succeeded at sprinting, fails. Tuning gait
thus aids in generalizing to different tasks. Bottom row: Examples of other
behaviors enabled by our controller.

> Keywords: Locomotion, Reinforcement Learning, Task Specification

##  1 Introduction

Recent works have established that quadruped locomotion controllers trained
with reinforcement learning in simulation can successfully be transferred to
traverse challenging natural terrains [1, 2, 3]. Adaptation to diverse
terrains is accomplished by estimating terrain properties from sensory
observations that are then used by the controller (i.e., online system
identification). The success of this paradigm relies on two assumptions: a
priori modeling of environment parameters that can vary during deployment and
the ability to estimate these parameters from sensory observations. To bypass
the first assumption, one possibility is to widely vary a large set of
environment parameters during training. However, this creates a hard learning
problem due to creation of challenging or infeasible locomotion scenarios. To
simplify learning, typically the designer chooses a subset of parameters that
are randomized in a carefully restricted range. Even in this setup, additional
measures such as a learning curriculum and reward shaping are necessary for
successful learning in simulation.

As a result of these practical restrictions on the expressiveness of the
simulation, quite often the robot encounters scenarios during deployment that
were not modeled during training. For instance, if the robot is only presented
with flat ground and terrain geometry is not varied during training, it may
fail to traverse non-flat terrains such as stairs. In such a case, it is
common to tweak the training environments or the reward functions and re-train
the policy. This iterative loop of re-training and real-world testing is
tedious. To make things worse, in some scenarios such iteration is
insufficient because it is not possible to accurately model or sense important
environment properties. For example, thick bushes are both hard to simulate
due to compliance and hard to sense because depth sensors do not distinguish
them from walls. Thus, the robot may attempt to climb over thick bushes like a
rock or move through them with an overly conservative gait that leaves the
robot stuck.

The examples above illustrate that even for the most advanced sim-to-real
systems, the real world offers new challenges. We broadly refer to scenarios
that can be simulated but are not anticipated during training and the
situations which cannot be simulated or identified from sensory observations
as out-of-distribution cases. We present a framework for policy learning that
enables improved performance in out-of-distribution scenarios under some
assumptions detailed below. Our key insight is that given a task, there are
multiple equally good solutions (i.e., under-specification [4]) that have
equivalent training performance but can generalize in different ways. For
instance, the task of walking on flat ground only imposes a constraint on the
velocity of robot’s body, but not on how the legs should move, or high should
the torso be above the ground, etc. Consider two different walking behaviors:
crouch where the robot keeps its torso close to the ground and stomp where the
torso is high and also the legs have a high foot swing. While both crouch and
stomp succeed at walking on flat ground, their generalization to out-of-
distribution scenarios is different: with crouch the robot can traverse under
obstacles but not stairs, whereas with stomp it can climb over curbs/stairs
but not move under obstacles (Figure 1).

Out of the many possible locomotion behaviors that succeed in the training
environment, typical reinforcement learning formulations result in a policy
that only embodies one solution and therefore expresses a single generalizing
bias. To facilitate generalization to diverse scenarios, we propose a
technique, Multiplicity of Behavior (MoB), that given the same observation
history and a small set of behavior parameters outputs different walking
behaviors. When faced with an unseen scenario, one can test different
behaviors by varying these parameters, which affords much quicker iteration
than re-training a new policy and facilitates collection of online
demonstrations by a human pilot.

The utility of MoB depends on the assumption that some subset of behaviors
successful in the restricted training environment will also succeed in the
out-of-distribution target environment. To demonstrate that this is true in a
meaningful sense, we chose an extreme case: train a single policy for
quadrupedal walking only on flat ground and evaluate on non-flat terrains and
new tasks. We show that a human operator can tune behaviors in real-time to
enable successful locomotion in the presence of diverse unseen terrains and
dynamics including uneven ground, stairs, external shoves, and constrained
spaces (Figure 1). The same tuning mechanisms can be used to compose behaviors
to perform new tasks such as payload manipulation and rhythmic dance (Figure
1).

This work contributes a robust low-level quadruped controller that can execute
diverse structured behaviors, which we we hope will be a useful building block
for future locomotion applications. It can serve as a platform for collecting
quadruped demonstrations for diverse tasks which is enabled by an
interpretable high-level control interface. Furthermore, our controller
showcases MoB as a practical tool for out-of-distribution generalization.
While our implementation of MoB leverages expert knowledge in the domain of
locomotion, in general, the technique of learning multiple methods of
achieving goals to facilitate generalization is a promising approach with
potential for broad application in sim-to-real reinforcement learning for
robotics.

Figure 2: A learned controller transitions between classical structured gaits:
trotting, pronking, pacing, and bounding in place at alternating frequencies
$2$Hz and $4$Hz. Images show the robot achieving contact phases of each gait,
with stance feet highlighted in red. Black shading in the bottom plot reflects
the timing reference variables $\textbf{t}_{t}$ for each foot; colored bars
report the contact states measured by foot sensors. As we later expose,
diverse behaviors can facilitate novel downstream capabilities.

##  2 Background

Auxiliary Rewards. Locomotion gaits learned using only the task reward (e.g.
velocity tracking) have not been shown to successfully transfer to the real
world. It is necessary to train using auxiliary rewards that bias the robot to
maintain a particular contact schedule, action smoothness, energy consumption,
or foot clearance [1, 2, 3, 5, 6, 7], to compensate for the sim-to-real gap.
Such auxiliary rewards can be interpreted as biases for generalization. For
example, foot clearance enables the robot to be robust when the terrain in the
real world is more uneven, or the robot’s body sinks more in the real world
than in simulation [7]. If an agent fails on a real-world task, it is common
practice to manually tune auxiliary rewards to encourage the emergence of
successful real-world behavior. However, such tuning is tedious because it
requires repeated iterations of training and deployment. Furthermore, such
tuning is task-specific and must be repeated for new tasks commanded to the
agent. The difficulty of designing a single set of auxiliary rewards that
promote generalization in diverse set of downstream tasks is illustrated in
the top row insets of Figure 1: each shows an instance where a fixed auxiliary
reward would lead to failure for one task, despite working well for another.

Learning Locomotion with Parameterized Style. Closely related recent work on
bipeds has explicitly included gait parameters in the task specification
through parameter-augmented auxiliary reward terms, including tracking rewards
for contact patterns [8] and target foot placements [9]. [8] used gait-
dependent reward terms in bipedal locomotion to control the timing offset
between the two feet and the duration of the swing phase. The reward structure
of [8] inspired ours, but due to its small space of command parameters, did
not propose and would not support application to compositional tasks or out-
of-distribution environments. To similar effect, other works have imposed
parameterized style through libraries of reference gaits. [10, 11] generated a
library of reference trajectories and trained a goal-conditioned policy to
imitate them. [12] demonstrated that a small discrete set of motion styles can
be learned simultaneously from a reference trajectory library.

Learning with Diversity Objectives.  Several prior methods aim to
automatically learn a collection of high-performing and diverse behaviors.
Quality diversity (QD) methods [13, 14, 15] learn a library of diverse
policies by enforcing a novelty objective defined among trajectories. They
typically perform this optimization using evolutionary strategies. QD has
demonstrated benefits including improved optimization performance and reuse of
skills for online adaptation [13, 14]. Another line of work uses unsupervised
objectives for skill discovery in RL, towards improving optimization [16] or
out-of-distribution generalization [17, 18]. Unsupervised diversity approaches
hold promise, but they have not yet scaled to real robotic platforms and do
not produce an grounded interface of parameters for guiding behavior.

Hierarchical Control Using Gait Parameters. Several works have learned a high-
level policy to accomplish downstream tasks by modulating the gait parameters
of a low-level model-based controller. This approach has been previously
applied to energy minimization [19] and vision-guided foot placement [20, 21].
These works relied on a model-predictive low-level controller to execute the
different gaits in absence of a gait-conditioned learned controller. The
tradeoffs between model-based control and reinforcement learning are well
discussed in prior literature [3, 22]. Our work enables revisiting
hierarchical approaches with learning at both the high and low levels.

##  3 Method

To obtain MoB, we train a conditional policy
$\pi(\cdot|\textbf{c}_{t},\textbf{b}_{t})$ that achieves tasks specified by
the command ($\textbf{c}_{t}$) in multiple ways that result from different
choices of behavior parameters, $\textbf{b}_{t}$. The question arises of how
to define $\textbf{b}_{t}$. We could learn behaviors using an unsupervised
diversity metric, but these behaviors might not be useful [4] and are not
human tunable. To overcome these issues, we leverage human intuition about
useful behavior parameters ($\textbf{b}_{t}$) corresponding to gait properties
like foot swing motion, body posture, and contact schedule [6, 7, 8, 23].
During training, the agent receives a combination of task rewards (for
velocity tracking), fixed auxiliary rewards (to promote sim-to-real transfer
and stable motion), and finally augmented auxiliary rewards (that encourage
locomotion in the desired style). During deployment in a novel environment, a
human operator can tune behavior of the policy by changing its input
$\textbf{b}_{t}$.

Term | Equation | Weight
---|---|---
$r_{v^{{\textrm{cmd}}}_{x,y}}$: xy velocity tracking | $\exp\\{{-{|\textbf{v}_{xy}-\textbf{v}^{\text{cmd}}_{xy}|^{2}}/{\sigma_{vxy}}}\\}$ | $0.02$
$r_{\omega^{{\textrm{cmd}}}_{z}}$: yaw velocity tracking | $\exp\\{{-{(\boldsymbol{\omega}_{z}-\boldsymbol{\omega}_{z}^{{\textrm{cmd}}})^{2}}/{\sigma_{\omega z}}}\\}$ | $0.01$
$r_{c^{{\textrm{cmd}}}_{f}}$: swing phase tracking (force) | $\sum_{\textrm{foot}}[1-C^{{\textrm{cmd}}}_{\textrm{foot}}(\boldsymbol{\theta}^{{\textrm{cmd}}},t)]\exp\\{{-{|\textbf{f}^{\textrm{foot}}|^{2}}/{\sigma_{cf}}}\\}$ | $-0.08$
$r_{c^{{\textrm{cmd}}}_{v}}$: stance phase tracking (velocity) | $\sum_{\textrm{foot}}[C^{{\textrm{cmd}}}_{\textrm{foot}}(\boldsymbol{\theta}^{{\textrm{cmd}}},t)]\exp\\{{-{|\textbf{v}^{\textrm{foot}}_{xy}|^{2}}/{\sigma_{cv}}}\\}$ | $-0.08$
$r_{\boldsymbol{h}_{z}^{{\textrm{cmd}}}}$: body height tracking | $(\boldsymbol{h_{z}}-\boldsymbol{h}_{z}^{{\textrm{cmd}}})^{2}$ | $-0.2$
$r_{\boldsymbol{\phi}^{{\textrm{cmd}}}}$: body pitch tracking | $(\boldsymbol{\phi}-\boldsymbol{\phi}^{{\textrm{cmd}}})^{2}$ | $-0.1$
$r_{\boldsymbol{s}_{y}^{{\textrm{cmd}}}}$: raibert heuristic footswing tracking | $(\textbf{p}_{x,y,\textrm{foot}}^{f}-\textbf{p}_{x,y,\textrm{foot}}^{f,\text{cmd}}(\boldsymbol{s}_{y}^{{\textrm{cmd}}}))^{2}$ | $-0.2$
$r_{\boldsymbol{h}_{z}^{f,{\textrm{cmd}}}}$: footswing height tracking | $\sum_{\textrm{foot}}(\boldsymbol{h}_{z,\textrm{foot}}^{f}-\boldsymbol{h}_{z}^{f,{\textrm{cmd}}})^{2}C^{{\textrm{cmd}}}_{\textrm{foot}}(\boldsymbol{\theta}^{{\textrm{cmd}}},t)$ | $-0.6$
z velocity | $\textbf{v}_{z}^{2}$ | $-410-4$
roll-pitch velocity | $|\boldsymbol{\omega}_{xy}|^{2}$ | $-210-5$
foot slip | $|\textbf{v}^{\textrm{foot}}_{xy}|^{2}$ | $-810-4$
thigh/calf collision | $\mathbbm{1}_{\text{collision}}$ | $-0.02$
joint limit violation | $\mathbbm{1}_{q_{i}>q_{max}||q_{i}<q_{min}}$ | $-0.2$
joint torques | $|\boldsymbol{\tau}|^{2}$ | $-210-5$
joint velocities | $|\dot{\textbf{q}}|^{2}$ | $-210-5$
joint accelerations | $|\ddot{\textbf{q}}|^{2}$ | $-510-9$
action smoothing | $|\textbf{a}_{t-1}-\textbf{a}_{t}|^{2}$ | $-210-3$
action smoothing, 2nd order | $|\textbf{a}_{t-2}-2\textbf{a}_{t-1}+\textbf{a}_{t}|^{2}$ | $-210-3$
Table 1: Reward structure: task rewards,  augmented auxiliary rewards, and
fixed auxiliary rewards.

TaskAugmented AuxiliaryFixed Auxiliary

###  3.1 Task Structure for MoB

Task Specification. We consider the task of omnidirectional velocity tracking.
This task is specified by a $3$-dimensional command vector
$\textbf{c}_{t}=[\textbf{v}_{x}^{{\textrm{cmd}}},\textbf{v}_{y}^{{\textrm{cmd}}},\boldsymbol{\omega}_{z}^{{\textrm{cmd}}}]$
where $\textbf{v}_{x}^{{\textrm{cmd}}},\textbf{v}_{y}^{{\textrm{cmd}}}$ are
the desired linear velocities in the body-frame x- and y- axes, and
$\boldsymbol{\omega}_{z}^{{\textrm{cmd}}}$ is the desired angular velocity in
the yaw axis.

Behavior Specification. We parameterize the style of task completion by an
$8$-dimensional vector of behavior parameters, $\textbf{b}_{t}$:

$\textbf{b}_{t}=[\boldsymbol{\theta}_{1}^{{\textrm{cmd}}},\boldsymbol{\theta}_{2}^{{\textrm{cmd}}},\boldsymbol{\theta}_{3}^{{\textrm{cmd}}},\boldsymbol{f}^{{\textrm{cmd}}},\boldsymbol{h}_{z}^{{\textrm{cmd}}},\boldsymbol{\phi}^{{\textrm{cmd}}},\boldsymbol{s}_{y}^{{\textrm{cmd}}},\boldsymbol{h}_{z}^{f,{\textrm{cmd}}}].$

$\boldsymbol{\theta}^{{\textrm{cmd}}}=(\boldsymbol{\theta}_{1}^{{\textrm{cmd}}},\boldsymbol{\theta}_{2}^{{\textrm{cmd}}},\boldsymbol{\theta}_{3}^{{\textrm{cmd}}})$
are the timing offsets between pairs of feet. These express gaits including
pronking ($\boldsymbol{\theta}^{{\textrm{cmd}}}=(0.0,0,0)$), trotting
($\boldsymbol{\theta}^{{\textrm{cmd}}}=(0.5,0,0)$), bounding,
($\boldsymbol{\theta}^{{\textrm{cmd}}}=(0,0.5,0)$), pacing
($\boldsymbol{\theta}^{{\textrm{cmd}}}=(0,0,0.5)$), as well as their
continuous interpolations such as galloping
($\boldsymbol{\theta}^{{\textrm{cmd}}}=(0.25,0,0)$). Taken together, the
parameters $\boldsymbol{\theta}^{{\textrm{cmd}}}$ can express all two-beat
quadrupedal contact patterns; Figure 2 provides a visual illustration.
$\boldsymbol{f}^{{\textrm{cmd}}}$ is the stepping frequency expressed in
$\text{\,}\mathrm{H}\mathrm{z}$. As an example, commanding
$\boldsymbol{f}^{{\textrm{cmd}}}=$3\text{\,}\mathrm{H}\mathrm{z}$$ will result
in each foot making contact three times per second.
$\boldsymbol{h}_{z}^{{\textrm{cmd}}}$ is the body height command;
$\boldsymbol{\phi}^{{\textrm{cmd}}}$ is the body pitch command.
$\boldsymbol{s}_{y}^{{\textrm{cmd}}}$ is the foot stance width command;
$\boldsymbol{h}_{z}^{f,{\textrm{cmd}}}$ is the footswing height command.

Reward function. All reward terms are listed in Table 1. Task rewards for body
velocity tracking are defined as functions of the command vector
$\textbf{c}_{t}$. Auxiliary rewards are used constrain the quadruped’s motion
for various reasons. “Fixed” auxiliary rewards are independent of behavior
parameters ($\textbf{b}_{t}$) and encourage stability and smoothness across
all gaits for better sim-to-real transfer. During training, one concern is
that the robot might abandon its task or choose an early termination when the
task reward is overwhelmed by penalties from the auxiliary objectives. To
resolve this, as in [7], we force the total reward to be a positive linear
function of the task reward by computing it as
$r_{\textrm{task}}\exp{(c_{\textrm{aux}}r_{\textrm{aux}})}$ where
$r_{\textrm{task}}$ is the sum of (positive) task reward terms and
$r_{\textrm{aux}}$ is the sum of (negative) auxiliary reward terms (we use
$c_{\textrm{aux}}=0.02$). This way, the agent is always rewarded for progress
towards the task, more when auxiliary objectives are satisfied and less when
they are not.

For MoB, we define augmented auxiliary rewards as functions of the behavior
vector $\textbf{b}_{t}$. We designed these rewards to increase when the
realized behavior matches $\textbf{b}_{t}$, and to not conflict with the task
reward. This required some careful design of the reward structure. For
example, when implementing stance width as a behavior parameter, a naive
approach would be to simply reward a constant desired distance between left
and right feet. However, this penalizes the robot during fast turning tasks
requiring relative lateral motion of the feet. To avoid this, we implement the
Raibert Heuristic, which suggests the necessary kinematic motion of the feet
to achieve a particular body velocity and contact schedule [23, 24]. The
Raibert Heuristic computes the desired foot position in the ground plane,
$\boldsymbol{p}_{x,y,\textrm{foot}}^{f,\text{cmd}}(\boldsymbol{s}_{y}^{{\textrm{cmd}}})$,
as an adjustment to the baseline stance width to make it consistent with the
desired contact schedule and body velocity. To define the desired contact
schedule, function
$C^{{\textrm{cmd}}}_{\textrm{foot}}(\boldsymbol{\theta}^{{\textrm{cmd}}},t)$
computes the desired contact state of each foot from the phase and timing
variable, as described in [8], with details given in the appendix.

###  3.2 Learning Diversified Locomotion

Task and Behavior Sampling In order to learn graceful online transitions
between behaviors, we resample the desired task and behavior within each
training episode. To enable the robot to both run and spin fast, we sample
task
$\textbf{c}_{t}=(\textbf{v}_{x}^{{\textrm{cmd}}},\textbf{v}_{y}^{{\textrm{cmd}}},\boldsymbol{\omega}_{z}^{{\textrm{cmd}}})$
using the grid adaptive curriculum strategy from [3]. Then, we need to sample
a target behavior $\textbf{b}_{t}$. First, we sample
$(\boldsymbol{\theta}_{1}^{{\textrm{cmd}}},\boldsymbol{\theta}_{2}^{{\textrm{cmd}}},\boldsymbol{\theta}_{3}^{{\textrm{cmd}}})$
as one of the symmetric quadrupedal contact patterns (pronking, trotting,
bounding, or pacing) which are known as more stable and which we found a
sufficient basis for diverse useful gaits. Then, the remaining command
parameters
$(\textbf{v}_{y}^{{\textrm{cmd}}},\boldsymbol{f}^{{\textrm{cmd}}},\boldsymbol{h}_{z}^{{\textrm{cmd}}},\boldsymbol{\phi}^{{\textrm{cmd}}},\boldsymbol{h}_{z}^{f,{\textrm{cmd}}},\boldsymbol{s}_{y}^{{\textrm{cmd}}})$
are sampled independently and uniformly. Their ranges are given in Table 5.

Policy Input. The input to the policy is a $30$-step history of observations
$\textbf{o}_{t-H...t}$, commands $\textbf{c}_{t-H...t}$, behaviors
$\textbf{b}_{t-H...t}$, previous actions $\textbf{a}_{t-H-1...t-1}$, and
timing reference variables $\textbf{t}_{t-H...t}$. The observation space
$\textbf{o}_{t}$ consists of joint positions and velocities
$\textbf{q}_{t},\dot{\textbf{q}}_{t}$ (measured by joint encoders) and the
gravity vector in the body frame $\textbf{g}_{t}$ (measured by accelerometer).
The timing reference variables $\textbf{t}_{t}=[\sin(2\pi
t^{\textrm{FR}}),\sin(2\pi t^{\textrm{FL}}),\sin(2\pi
t^{\textrm{RR}}),\sin(2\pi t^{\textrm{RL}})]$ are computed from the offset
timings of each foot:
$[t^{\textrm{FR}},t^{\textrm{FL}},t^{\textrm{RR}},t^{\textrm{RL}}]=[t+\boldsymbol{\theta}_{2}^{{\textrm{cmd}}}+\boldsymbol{\theta}_{3}^{{\textrm{cmd}}},t+\boldsymbol{\theta}_{1}^{{\textrm{cmd}}}+\boldsymbol{\theta}_{3}^{{\textrm{cmd}}},t+\boldsymbol{\theta}_{1}^{{\textrm{cmd}}},t+\boldsymbol{\theta}_{2}^{{\textrm{cmd}}}]$,
where $t$ is a counter variable that advances from $0$ to $1$ during each gait
cycle and ${}^{\textrm{FR}}$, ${}^{\textrm{FL}}$, ${}^{\textrm{RR}}$,
${}^{\textrm{RL}}$ are the four feet. This form is adapted from [8] to express
quadrupedal gaits.

Policy Architecture. Our policy body is an MLP with hidden layer sizes
$[512,256,128]$ and ELU activations. Besides the above, the policy input also
includes estimated domain parameters: the velocity of the robot body and the
ground friction, which are predicted from the observation history using
supervised learning in the manner of [7]. The estimator module is an MLP with
hidden layer sizes $[256,128]$ and ELU activations. We did not analyze the
impact of this estimation on performance but found it useful for visualizing
deployments.

Action Space. The action $\textbf{a}_{t}$ consists of position targets for
each of the twelve joints. A zero action corresponds to the nominal joint
position, $\hat{\textbf{q}}$. The position targets are tracked using a
proportional-derivative controller with $k_{p}=20,k_{d}=0.5$.

###  3.3 Design Choices for Sim-to-Real Transfer

Domain Randomization. For better sim-to-real transfer, we train a policy that
is robust to a range of robot’s body mass, motor strength, joint position
calibration, the ground friction and restitution, and the orientation and
magnitude of gravity. As we are interested in studying out-of-distribution
generalization, we only train on flat ground without any randomization of
terrain geometry. This choice also simplified training. The randomization
ranges of all parameters are in Appendix A.

Latency and Actuator Modeling. Directly identifying invariant properties
avoids overly conservative behavior from unnecessary domain randomization. We
perform system identification to reduce the sim-to-real gap in the robot
dynamics. Following [22], we train an actuator network to capture the non-
ideal relationship between PD error and realized torque. Separately, we
identify a latency of around $20\text{\,}\mathrm{ms}$ in our system and model
this as a constant action delay during simulation.

Gait | $0.0\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$ | $1.0\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$ | $2.0\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$ | $3.0\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$
---|---|---|---|---
Trotting | $9_{\pm 1}$ | $24_{\pm 1}$ | $53_{\pm 5}$ | $98_{\pm 9}$
Pronking | $32_{\pm 1}$ | $43_{\pm 2}$ | $68_{\pm 5}$ | $112_{\pm 5}$
Pacing | $13_{\pm 3}$ | $25_{\pm 2}$ | $55_{\pm 3}$ | $99_{\pm 6}$
Bounding | $22_{\pm 2}$ | $39_{\pm 4}$ | $78_{\pm 5}$ | $127_{\pm 35}$
Gait-free Baseline | $17_{\pm 5}$ | $35_{\pm 5}$ | $64_{\pm 10}$ | $102_{\pm 14}$
Trotting ($\boldsymbol{f}^{{\textrm{cmd}}}=$2\text{\,}\mathrm{Hz}$$) | $11_{\pm 2}$ | $25_{\pm 1}$ | $55_{\pm 4}$ | $104_{\pm 8}$
Trotting ($\boldsymbol{f}^{{\textrm{cmd}}}=$3\text{\,}\mathrm{Hz}$$) | $9_{\pm 1}$ | $24_{\pm 1}$ | $53_{\pm 5}$ | $98_{\pm 9}$
Trotting ($\boldsymbol{f}^{{\textrm{cmd}}}=$4\text{\,}\mathrm{Hz}$$) | $9_{\pm 1}$ | $26_{\pm 0}$ | $60_{\pm 4}$ | $114_{\pm 12}$
Trotting ($\boldsymbol{h}_{z}^{{\textrm{cmd}}}=$20\text{\,}\mathrm{cm}$$) | $9_{\pm 1}$ | $26_{\pm 1}$ | $56_{\pm 3}$ | $102_{\pm 8}$
Trotting ($\boldsymbol{h}_{z}^{{\textrm{cmd}}}=$30\text{\,}\mathrm{cm}$$) | $9_{\pm 1}$ | $24_{\pm 1}$ | $53_{\pm 5}$ | $98_{\pm 9}$
Trotting ($\boldsymbol{h}_{z}^{{\textrm{cmd}}}=$40\text{\,}\mathrm{cm}$$) | $10_{\pm 1}$ | $23_{\pm 1}$ | $52_{\pm 4}$ | $95_{\pm 9}$

[table]A table beside a figure

Table 2: Behavior tuning enables interventional studies on the relationship
between gait properties and performance criteria within a single policy. Here,
we illustrate how power consumption varies across speeds for common
quadrupedal gaits and for a baseline policy without gait constraint. Several
structured gaits surpass the efficiency of unconstrained gait across all
speeds.

###  3.4 Materials

Simulator and Learning Algorithm. We define our training environment in the
Isaac Gym simulator [6, 25]. We train policies using Proximal Policy
Optimization [26]; details in Appendix A.

Hardware. We deploy our controller in the real world on the Unitree Go1 Edu
robot [27]. An onboard Jetson TX2 NX computer runs our trained policy. We
implement an interface based on Lightweight Communications and Marshalling
(LCM) [28] to pass sensor data, motor commands, and joystick state between our
code and the low-level control SDK provided by Unitree. For both training and
deployment, the control frequency is 50Hz.

Gait-free Baseline. To understand the impact of MoB on performance, we compare
our controller to a baseline velocity-tracking controller (the “gait-free
baseline”). The gait-free baseline is trained by the method above, but
excludes all augmented auxiliary rewards (Table 1). Therefore, it only learns
one solution to the training environment and its actions are independent of
behavior parameters $\textbf{b}_{t}$.

##  4 Experimental Results

###  4.1 Sim-to-Real Transfer and Gait Switching

We deploy the controller learned in simulation in the real world and first
evaluate its performance on flat ground similar to the training environment.
To start, we demonstrate generating and switching between structured gaits
that are well-known in the locomotion community. Figure 2 shows torques and
contact states during transition between trotting, pronking, bounding, and
pacing while alternating $\boldsymbol{f}^{{\textrm{cmd}}}$ between
$2\text{\,}\mathrm{Hz}$ and $4\text{\,}\mathrm{Hz}$. We find that all gait
parameters are consistently tracked after sim-to-real transfer. Videos
(i)-(iv) on the project website visualize the different gaits obtained by
modulating each parameter in $\textbf{b}_{t}$ individually.

Gait | $r_{v^{{\textrm{cmd}}}_{x,y}}$ | $r_{\omega^{{\textrm{cmd}}}_{z}}$ | $r_{c^{{\textrm{cmd}}}_{f}}$ | $r_{c^{{\textrm{cmd}}}_{v}}$ | Survival
---|---|---|---|---|---
Trotting | $0.80_{\pm 0.01}^{(0.95)}$ | $0.76_{\pm 0.00}^{(0.89)}$ | $0.95_{\pm 0.00}^{(0.97)}$ | $0.98_{\pm 0.00}^{(0.98)}$ | $0.88_{\pm 0.01}^{(1.00)}$
Pronking | $0.84_{\pm 0.01}^{(0.94)}$ | $0.77_{\pm 0.01}^{(0.85)}$ | $0.96_{\pm 0.00}^{(0.96)}$ | $0.97_{\pm 0.00}^{(0.98)}$ | $0.82_{\pm 0.02}^{(1.00)}$
Pacing | $0.76_{\pm 0.01}^{(0.91)}$ | $0.76_{\pm 0.01}^{(0.81)}$ | $0.94_{\pm 0.00}^{(0.96)}$ | $0.98_{\pm 0.00}^{(0.98)}$ | $0.87_{\pm 0.02}^{(1.00)}$
Bounding | $0.80_{\pm 0.01}^{(0.88)}$ | $0.73_{\pm 0.01}^{(0.86)}$ | $0.94_{\pm 0.00}^{(0.96)}$ | $0.98_{\pm 0.00}^{(0.98)}$ | $0.82_{\pm 0.01}^{(1.00)}$
Gait-free | $0.81_{\pm 0.03}^{(0.96)}$ | $0.74_{\pm 0.06}^{(0.92)}$ | – | – | $0.83_{\pm 0.01}^{(1.00)}$

Table 3: Zero-shot generalization to platform terrain (visualized right).
Pacing and trotting yield the best survival time in out-of-distribution
deployment, outperforming the gait-free baseline. Pronking attains the best
velocity tracking performance, with similar survival time to the baseline. We
report the fraction of maximum episodic reward. Superscript reports
performance in flat training environment with no platforms. Subscript reports
standard deviation across three random seeds.

###  4.2 Leveraging MoB for Generalization

Tuning for New Tasks. After training using a generic locomotion objective, one
might wish to tune a controller’s behavior to optimize a new metric in the
original environment. MoB facilitates this if some subset of learned behaviors
outperform the gait-free policy by the new task metric.

Energy efficiency (simulated): We consider the task of minimizing the
mechanical power consumption ($\text{\,}\mathrm{J}\mathrm{/}\mathrm{s}$)
measured by summing the product of joint velocity and torque at each of the 12
motors: $\sum_{i}\max(\boldsymbol{\tau}_{i}\dot{\textbf{q}}_{i},0)$. As shown
in Figure 2, several choices of the contact schedule
$\boldsymbol{\theta}^{{\textrm{cmd}}}$, height
$\boldsymbol{h}_{z}^{{\textrm{cmd}}}$, and frequency
$\boldsymbol{f}^{{\textrm{cmd}}}$ outperform the gait-free policy in this
unseen metric, consuming less energy across all speeds. Therefore, tuning the
behavior parameters can facilitate tuning performance on a new objective
(energy efficiency) in the original training environment.

Payload manipulation: We experiment with another task where the robot is
required to transport a ball from one place to another, then bend its body so
the ball is deposited on the ground. A gait-free policy couldn’t do this;
while many possible body posture profiles are valid to solve the training
environment, the gait-free policy will simply converge to one at random. In a
real-world experiment, we demonstrate how MoB-enabled body posture control can
be repurposed for teleoperated payload manipulation. The operator pilots the
robot to the delivery location with the body level, then pitches the body
backward, modulating $\boldsymbol{\phi}^{{\textrm{cmd}}}$, to dump the payload
(Figure 1, bottom row, second from left).

Tuning for New Environments. In the previous section, we showed that MoB can
support repurposement to novel tasks in the training environment. In the real
world, there is also always a long tail of environments that are not modeled
in training. We demonstrate the behavior of our controller trained only on
flat ground in the presence of challenging non-flat terrains and disturbances
such as curbs, bushes, hanging obstacles, and shoves. Successes suggest that
the space of policies learned by MoB contains some behaviors that transfer
better than the baseline to unseen environments.

Platform terrain (simulated): We evaluate the performance of our robot in
traversing randomly positioned platforms with height up to
$16\text{\,}\mathrm{cm}$ (Table 3). We report two metrics: mean reward and
mean survival time as a fraction of the maximum episode length
($10\text{\,}\mathrm{s}$). For each metric, we find that by modulating the
contact schedule $\boldsymbol{\theta}^{{\textrm{cmd}}}$ or footswing height
$\boldsymbol{h}_{z}^{f,{\textrm{cmd}}}$, we can outperform the gait-free
policy. (Figure 3, Appendix Figure 10). Therefore, it is possible to improve
performance in an out-of-distribution terrain by modulating the parameters of
the MoB policy.

Climbing over curbs: Previous works demonstrating blind obstacle traversal on
a quadruped [1, 2] learned a foot-trapping reflex where the robot first trips,
then raises its foot over the obstacle. In contrast, with the help of a human
pilot, our gait-conditioned policy with high footswing command enables fast
and smooth obstacle traversal without tripping, despite training on a simpler
flat-ground terrain. In the real world, we demonstrate that modulating
footswing height $\boldsymbol{h}_{z}^{f,{\textrm{cmd}}}$ enables our
controller to climb smoothly across stairs and curbs (Figure 1, bottom left).

Hacking through thick bushes: Extremely thick bushes pose a methodological
challenge for state-of-the-art perceptive locomotion controllers. They are
hard to simulate, because they comply against the whole body of the robot, and
they are hard to sense as they are indistinguishable from solid obstacles to
depth sensors. Therefore, prior works would either attempt to climb over
bushes as obstacles or fall back on a robust proprioceptive controller that is
unaware of the semantic context. In contrast, by modulating the gait
parameters for gait frequency $\boldsymbol{f}^{{\textrm{cmd}}}$ and footswing
height $\boldsymbol{h}_{z}^{f,{\textrm{cmd}}}$ in situ, a human operator can
guide our system quickly through challenging brush.

Navigating confined spaces: Consider the scenario where the robot needs to go
under a bar. The gait-free baseline cannot accomplish this; in the absence of
such constraints during training, it will converge to a fixed body height
profile. Our system with MoB can navigate confined spaces through modulation
of the body height $\boldsymbol{h}_{z}^{{\textrm{cmd}}}$. In a real-world
example, the robot was able to crawl under a $22\text{\,}\mathrm{cm}$ bar; the
robot body thickness is $13\text{\,}\mathrm{cm}$, leaving
$9\text{\,}\mathrm{cm}$ of clearance beneath the robot.

Anticipative bracing against shoves: Widening the stance can make a quadruped
more robust to hard shoves, but absent a perception module to anticipate a
human kick, the robot would always need to walk in a widened stance to be
prepared for a shove. This interferes with performance in other tasks like
running efficiently, so learned locomotion controllers without MoB often
provide incentive to keep the feet nominally below the hips [2, 7]. With MoB,
widening the stance width $\boldsymbol{s}_{y}^{{\textrm{cmd}}}$ in the pilot’s
anticipation of a shove enables the robot to remain upright.

Exploring the Speed and Range of Behavior Adaptation. If we want to use our
low-level controller for high-level tasks, one thing we might desire is to
switch between gaits even at high speeds, which might be useful for
applications such as parkour. Another is to transition between diverse gaits
with precise timing for a synchronous task like dancing.

Agile forward leap: As a demonstration of gait transitions at high speed, we
modulate contact schedule, velocity, and gait frequency at to encode an agile
forward leap (Figure 3). The robot first accelerates to a target speed of
$3\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$ at a trot while increasing its step
frequency from $2\text{\,}\mathrm{Hz}$ to $4\text{\,}\mathrm{Hz}$, then
switches to pronking at $2\text{\,}\mathrm{Hz}$ for one second, then
decelerates to a standstill while trotting. During the leap phase, the
distance from the location of the robot’s front feet at takeoff to the
location of the hind feet upon landing is $60\text{\,}\mathrm{cm}$.

Choreographed dance: To demonstrate precisely timed transition between diverse
gaits, we program a sequence of gait parameters to generate a dance routine
synchronized to a jazz song with a tempo of 90 bpm. At this tempo,
combinations of phases $0$, $0.25$, and $0.5$ with frequencies of
$1.5\text{\,}\mathrm{Hz}$ and $3\text{\,}\mathrm{Hz}$ yield eighth, quarter,
half, and full beat gaps between consecutive footsteps. We also modulate body
height and velocity in time with the music. An assistant script procedurally
generates gait parameters, fed into the controller in open-loop fashion.

Figure 3: We demonstrate that behavior transitions can be performed even in
quick sequence at high speed for synthesis of agile maneuvers. Emulating gap
crossing on flat ground, we show that this can facilitate crossing a gap wider
than the robot’s body length in a single leap.

##  5 Discussion and Limitations

Our experiments show that the benefits of adding MoB can come at a cost to in-
distribution task performance, specifically limiting the robot’s flat-ground
sprinting performance (Table 4, appendix). Heat maps reveal that our behavior
parameterization is restrictive for combinations of high linear and angular
velocity. To quantify and control the tradeoff between task performance and
reward shaping is an interesting future direction, for which some prior
methods have been proposed [29].

MoB confers a single learned policy a structured and controllable space of
diverse locomotion behaviors for each state and task in the training
distribution. This yields a set of ‘knobs’ to tune the performance of motor
skills in unseen test environments. The system currently requires a human
pilot to manually tune its behavior. In the future, the autonomy of our system
could be extended by automating behavior selection using imitation from real-
world human demonstrations, or by using a hierarchical learning approach to
automatically self-tune the controller during deployment.

#### Acknowledgments

We thank the members of the Improbable AI lab for the helpful discussions and
feedback on the paper. We are grateful to MIT Supercloud and the Lincoln
Laboratory Supercomputing Center for providing HPC resources. This research
was supported by the DARPA Machine Common Sense Program, the MIT-IBM Watson AI
Lab, and the National Science Foundation under Cooperative Agreement
PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental
Interactions, http://iaifi.org/). This research was also sponsored by the
United States Air Force Research Laboratory and the United States Air Force
Artificial Intelligence Accelerator and was accomplished under Cooperative
Agreement Number FA8750-19-2-1000. The views and conclusions contained in this
document are those of the authors and should not be interpreted as
representing the official policies, either expressed or implied, of the United
States Air Force or the U.S. Government. The U.S. Government is authorized to
reproduce and distribute reprints for Government purposes, notwithstanding any
copyright notation herein.

Author Contributions

  * •

Gabriel B. Margolis concieved, designed, and implemented the controller, ran
all experiments, and played the primary role in paper writing.

  * •

Pulkit Agrawal advised the project and contributed to its conceptual
development, experimental design, positioning, and writing.

## References

  * Lee et al. [2020] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter.  Learning quadrupedal locomotion over challenging terrain.  _Sci. Robot._ , 5(47):eabc5986, Oct. 2020.  doi:10.1126/scirobotics.abc5986.
  * Kumar et al. [2021] A. Kumar, Z. Fu, D. Pathak, and J. Malik.  RMA: Rapid motor adaptation for legged robots.  In _Proc. Robot.: Sci. and Syst. (RSS)_ , Virtual, July 2021.  doi:10.48550/arXiv.2107.04034.
  * Margolis et al. [2022] G. B. Margolis, G. Yang, K. Paigwar, T. Chen, and P. Agrawal.  Rapid locomotion via reinforcement learning.  _Proc. Robot.: Sci. and Syst. (RSS)_ , June 2022.  doi:10.48550/arXiv.2205.02824.
  * Agrawal [2022] P. Agrawal.  The task specification problem.  In _Conference on Robot Learning_ , pages 1745–1751. PMLR, 2022.
  * Fu et al. [2021] Z. Fu, A. Kumar, J. Malik, and D. Pathak.  Minimizing energy consumption leads to the emergence of gaits in legged robots.  In _Proc. Conf. Robot Learn. (CoRL)_ , pages 928–937, London, UK, Nov. 2021.  doi:10.48550/arXiv.2111.01674.
  * Rudin et al. [2021] N. Rudin, D. Hoeller, P. Reist, and M. Hutter.  Learning to walk in minutes using massively parallel deep reinforcement learning.  In _Proc. Conf. Robot Learn. (CoRL)_ , pages 91–100, London, UK, Nov. 2021.  doi:10.48550/arXiv.2109.11978.
  * Ji et al. [2022] G. Ji, J. Mun, H. Kim, and J. Hwangbo.  Concurrent training of a control policy and a state estimator for dynamic and robust legged locomotion.  _IEEE Robot. Automat. Lett. (RA-L)_ , 7(2):4630 – 4637, Apr. 2022.  doi:10.1109/LRA.2022.3151396.
  * Siekmann et al. [2021] J. Siekmann, Y. Godse, A. Fern, and J. Hurst.  Sim-to-real learning of all common bipedal gaits via periodic reward composition.  In _Proc. IEEE Int. Conf. Robot. Automat. (ICRA)_ , pages 7309–7315, Xi’an, China, June 2021.  doi:10.1109/ICRA48506.2021.9561814.
  * Duan et al. [2022] H. Duan, A. Malik, J. Dao, A. Saxena, K. Green, J. Siekmann, A. Fern, and J. Hurst.  Sim-to-real learning of footstep-constrained bipedal dynamic walking.  In _Proc. IEEE Int. Conf. Robot. Automat. (ICRA)_ , Philadelphia, USA, May 2022.  doi:10.1109/ICRA46639.2022.9812015.
  * Li et al. [2021] Z. Li, X. Cheng, X. B. Peng, P. Abbeel, S. Levine, G. Berseth, and K. Sreenath.  Reinforcement learning for robust parameterized locomotion control of bipedal robots.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 2811–2817. IEEE, 2021.
  * Shao et al. [2021] Y. Shao, Y. Jin, X. Liu, W. He, H. Wang, and W. Yang.  Learning free gait transition for quadruped robots via phase-guided controller.  _IEEE Robot. Automat. Lett. (RA-L)_ , 7(2):1230–1237, 2021.
  * Vollenweider et al. [2022] E. Vollenweider, M. Bjelonic, V. Klemm, N. Rudin, J. Lee, and M. Hutter.  Advanced skills through multiple adversarial motion priors in reinforcement learning.  _arXiv preprint arXiv:2203.14912_ , 2022.
  * Mouret and Clune [2015] J.-B. Mouret and J. Clune.  Illuminating search spaces by mapping elites.  _arXiv preprint arXiv:1504.04909_ , 2015.
  * Cully et al. [2015] A. Cully, J. Clune, D. Tarapore, and J.-B. Mouret.  Robots that can adapt like animals.  _Nature_ , 521(7553):503–507, 2015.
  * Lim et al. [2022] B. Lim, L. Grillotti, L. Bernasconi, and A. Cully.  Dynamics-aware quality-diversity for efficient learning of skill repertoires.  In _2022 International Conference on Robotics and Automation (ICRA)_ , pages 5360–5366. IEEE, 2022.
  * Eysenbach et al. [2018] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine.  Diversity is all you need: Learning skills without a reward function.  _arXiv preprint arXiv:1802.06070_ , 2018.
  * Kumar et al. [2020] S. Kumar, A. Kumar, S. Levine, and C. Finn.  One solution is not all you need: Few-shot extrapolation via structured maxent rl.  _Advances in Neural Information Processing Systems_ , 33:8198–8210, 2020.
  * Gaya et al. [2021] J.-B. Gaya, L. Soulier, and L. Denoyer.  Learning a subspace of policies for online adaptation in reinforcement learning.  _arXiv preprint arXiv:2110.05169_ , 2021.
  * Yang et al. [2022] Y. Yang, T. Zhang, E. Coumans, J. Tan, and B. Boots.  Fast and efficient locomotion via learned gait transitions.  In _Proc. Conf. Robot Learn. (CoRL)_ , pages 773–783, Nov. 2022.  doi:10.48550/arXiv.2104.04644.
  * Margolis et al. [2021] G. B. Margolis, T. Chen, K. Paigwar, X. Fu, D. Kim, S. Kim, and P. Agrawal.  Learning to jump from pixels.  In _Proc. Conf. Robot Learn. (CoRL)_ , pages 1025–1034, London, UK, Nov. 2021.  doi:10.48550/arXiv.2110.15344.
  * Yu et al. [2021] W. Yu, D. Jain, A. Escontrela, A. Iscen, P. Xu, E. Coumans, S. Ha, J. Tan, and T. Zhang.  Visual-locomotion: Learning to walk on complex terrains with vision.  In _Proc. Conf. Robot Learn. (CoRL)_ , Oct. 2021.
  * Hwangbo et al. [2019] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter.  Learning agile and dynamic motor skills for legged robots.  _Sci. Robot._ , 4(26):aau5872, Jan. 2019.  doi:10.1126/scirobotics.aau5872.
  * Kim et al. [2019] D. Kim, J. Di Carlo, B. Katz, G. Bledt, and S. Kim.  Highly dynamic quadruped locomotion via whole-body impulse control and model predictive control.  _arXiv preprint_ , 2019.  doi:10.48550/arXiv.1909.06586.
  * Raibert [1986] M. H. Raibert.  _Legged Robots That Balance_.  MIT press, 1986.
  * Makoviychuk et al. [2021] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al.  Isaac Gym: High performance GPU-based physics simulation for robot learning.  _arXiv preprint_ , 2021.  doi:10.48550/arXiv.2108.10470.
  * Schulman et al. [2017] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov.  Proximal policy optimization algorithms.  _arXiv preprint_ , 2017.  doi:10.48550/arXiv.1707.06347.
  * [27] Unitree Robotics, Go1, 2022, https://www.unitree.com/products/go1, [Online; accessed Jun. 2022].
  * Huang et al. [2010] A. S. Huang, E. Olson, and D. C. Moore.  LCM: Lightweight communications and marshalling.  In _2010 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 4057–4062. IEEE, 2010.
  * Chen et al. [2022] E. Chen, H. Zhang-Wei, J. Pajarinen, and P. Agrawal.  Redeeming intrinsic rewards via constrained optimization.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022.

Ablation | $r_{v^{{\textrm{cmd}}}_{x,y}}$ | $r_{\omega^{{\textrm{cmd}}}_{z}}$ | $r_{c^{{\textrm{cmd}}}_{f}}$ | $r_{c^{{\textrm{cmd}}}_{v}}$
---|---|---|---|---
Trotting | $0.92_{\pm 0.01}$ | $0.70_{\pm 0.04}$ | $0.98_{\pm 0.00}$ | $0.95_{\pm 0.00}$
Pronking | $0.85_{\pm 0.01}$ | $0.64_{\pm 0.05}$ | $0.98_{\pm 0.00}$ | $0.94_{\pm 0.00}$
Pacing | $0.83_{\pm 0.02}$ | $0.66_{\pm 0.04}$ | $0.96_{\pm 0.01}$ | $0.94_{\pm 0.01}$
Bounding | $0.88_{\pm 0.01}$ | $0.63_{\pm 0.05}$ | $0.96_{\pm 0.01}$ | $0.95_{\pm 0.00}$
Gait-free Baseline | $0.94_{\pm 0.02}$ | $0.76_{\pm 0.01}$ | – | –

Table 4: Removing gait constraints results in improved velocity tracking task
performance on flat ground. Heat maps (right) break down the mean task reward
for each velocity command, revealing that the gait-free approach is most
beneficial for combinations of high linear and angular velocity.

##  Appendix A Training Details

The ranges used for domain and command randomization are provided in Table 5.
The hyperparameters used for PPO are provided in Table 6. The hyperparameters
used in the curriculum are provided in Table 7.

Figure 6 illustrates data flow during training. The curriculum engine first
samples from a Gaussian distribution centered at one of the four main gaits
(trotting, pronking, bounding, pacing); then it samples velocity commands from
a grid distribution according to the method of [3]; then finally it samples
the stepping frequency and body height uniformly. The policy and simulator are
rolled out, and the reward is computed as a function of the gait parameters.
Then the curriculum is updated if the episodic reward meets the thresholds
given in Table 7.

Term Minimum Maximum Units Payload Mass $-1.0$ $3.0$ $\text{\,}\mathrm{kg}$
Motor Strength $90$ $110$ $\%$ Joint Calibration $-0.02$ $0.02$
$\text{\,}\mathrm{rad}$ Ground Friction $0.40$ $1.00$ – Ground Restitution
$0.00$ $1.00$ – Gravity Offset $-1.0$ $1.0$
$\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}^{2}$
$\textbf{v}_{x}^{{\textrm{cmd}}}$ – –
$\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$ $\textbf{v}_{y}^{{\textrm{cmd}}}$
$-0.6$ $0.6$ $\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$
$\boldsymbol{\omega}_{z}^{{\textrm{cmd}}}$ – –
$\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$ $\boldsymbol{f}^{{\textrm{cmd}}}$
$1.5$ $4.0$ $\text{\,}\mathrm{Hz}$
$\boldsymbol{\theta}_{1}^{{\textrm{cmd}}},\boldsymbol{\theta}_{2}^{{\textrm{cmd}}},\boldsymbol{\theta}_{3}^{{\textrm{cmd}}}$
$0.0$ $1.0$ – $\boldsymbol{h}_{z}^{{\textrm{cmd}}}$ $0.10$ $0.45$
$\text{\,}\mathrm{m}$ $\boldsymbol{\phi}^{{\textrm{cmd}}}$ $-0.4$ $0.4$
$\text{\,}\mathrm{rad}$ $\boldsymbol{s}_{y}^{{\textrm{cmd}}}$ $0.05$ $0.45$
$\text{\,}\mathrm{m}$ $\boldsymbol{h}_{z}^{f,{\textrm{cmd}}}$ $0.03$ $0.25$
$\text{\,}\mathrm{m}$

Table 5: Randomization ranges for dynamics parameters (top) and commands
(bottom) during training.
$\textbf{v}_{x}^{{\textrm{cmd}}},\boldsymbol{\omega}_{z}^{{\textrm{cmd}}}$ are
adapted according to a curriculum.

Figure 4: Pronking and trotting gaits are easier to learn and tend to dominate
pacing and bounding early in training. However, when discovered, pacing and
bounding gaits can yield good performance and later become preferred for some
downstream tasks (Section 4.2).

##  Appendix B Teleoperation Interface

Figure 6 illustrates the mapping from remote control inputs to gait parameters
used during teleoperation. The front bumpers on the top toggle between control
modes to accommodate mixing our large number of gait parameters. Preprogrammed
sequences such as dancing and leaping can be assigned to the rear bumpers.

Figure 5: Controller mapping. Mapping of remote control inputs to gait
parameters during robot teleoperation. The user can change between gaits at
any time. Continuous interpolation between contact patterns is supported by
our policy, but not mapped here. Lateral velocity is also supported by the
controller but excluded from the mapping.

Figure 6: Training architecture. The policy computes the action as a function
of the gait parameters and state. The simulator computes the reward and state
as a function of the gait parameters and actions. The curriculum engine
periodically resamples gait parameters based on the reward.

##  Appendix C Extended Performance Analysis

Impact of Gait Frequency on High-speed Running. We evaluate them impact of
gait frequency on performance of the robot at high speeds. Figure 10 reports
our result that higher gait frequency is necessary to yield good tracking
performance for higher-speed running.

Impact of Footswing Height on Platform Terrain Performance. We evaluate them
impact of footswing height on performance of the robot on the out-of-
distribution platform terrain. Figure 10 reports our result that higher swing
heights yield improved platform traversal, outperforming the gait-free policy.

Flat Ground Velocity Heatmaps for More Gaits. We provide velocity heatmaps in
Table 11 for pronking, pacing, and bounding gaits to supplement the trotting
and gait-free heatmaps provided in Table 4.

Forward and Backward Locomotion. During evaluation in the random platforms
environment, we found that walking backward leads to fewer failures than
walking forward. Figure 8 illustrates this phenomenon by plotting the mean
failure rate of each gait at each test velocity. Possible explanations include
(i) recovery strategies that are dependent on knee orientation and (ii) the
weight distribution of the robot.

Real-world Robustness Demonstrations. We conducted several hours of real-world
testing of different gaits across a variety of laboratory and outdoor
environments. A selection of this footage is available on the project website.
The robot was able to traverse down stairs, up and down granular and slippery
terrain, and to respond to external perturbations. To provide insight into the
robot’s disturbance response, we plot the joint torques, contact states, and
learned state estimate of the robot in Figure 7. The robot trots at two
different frequencies and is shoved twice by the operator, once from each
side. The state estimator correctly predicts the direction of the lateral
velocity increase (orange), and adapts the joint torques and contact schedule
to correct.

##  Appendix D Contact Schedule Parameterization

At each control timestep, we compute the desired contact from the gait
parameters $\boldsymbol{\theta}^{{\textrm{cmd}}}$ as follows. First, we
increment the global timing variable $t$ by
$\frac{\boldsymbol{f}^{{\textrm{cmd}}}}{f_{\pi}}$ where
$\boldsymbol{f}^{{\textrm{cmd}}}$ is the commanded stepping frequency and
$f_{\pi}$ is the control frequency. Then, we compute separate timing variables
for each foot, clipped between $0$ and $1$:

$[t^{\textrm{FR}},t^{\textrm{FL}},t^{\textrm{RR}},t^{\textrm{RL}}]=\text{clip}([t+\boldsymbol{\theta}_{2}^{{\textrm{cmd}}}+\boldsymbol{\theta}_{3}^{{\textrm{cmd}}},t+\boldsymbol{\theta}_{1}^{{\textrm{cmd}}}+\boldsymbol{\theta}_{3}^{{\textrm{cmd}}},t+\boldsymbol{\theta}_{1}^{{\textrm{cmd}}},t+\boldsymbol{\theta}_{2}^{{\textrm{cmd}}}],0,1)$

From these, we can directly compute the desired contact states:

$C^{{\textrm{cmd}}}_{\textrm{foot}}(t^{\textrm{foot}}(\boldsymbol{\theta}^{{\textrm{cmd}}},t))=\Phi(t^{\textrm{foot}},\sigma)*(1-\Phi(t^{\textrm{foot}}-0.5,\sigma))+\Phi(t^{\textrm{foot}}-1,\sigma)*(1-\Phi(t^{\textrm{foot}}-1.5,\sigma))$

where $\Phi(x;\sigma)$ is the cumulative density function of the normal
distribution:

$\Phi(x;\sigma)=\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}(\frac{x}{\sigma})^{2}}$

which is an approximation to the Von Mises distribution used in [8] to form a
smooth transition between stance and swing.

Figure 7: Shove Robustness Test. Joint torques (top), contact states (middle),
and velocity estimate (bottom) during trotting in the laboratory setting.
Dashed boxes indicate shove events. The robot was first shoved to the right
during trotting at low frequency, and then shoved to the left during trotting
at high frequency. The learned state estimator correctly infers the change in
lateral velocity and adjusts the joint torques and foot contacts to stabilize
the robot.  Figure 8: Forward vs Backward Walking on Platforms. Time to
failure for different gaits and velocities in the random platforms environment
(zero-shot test). The temperature bar unit is the mean fraction of a 20s
episode elapsed before failure, between zero and one. The gait frequency is
3Hz. The pacing gait boasts the longest mean survival time, possibly due to
higher footswings or more practice with recovery strategies during training.
Interestingly, almost all failures occur during forward locomotion, and the
robot is much more robust when moving backward. Possible explanations include
(i) recovery strategies that are dependent on knee orientation and (ii) the
weight distribution of the robot.

Hyperparameter Value discount factor 0.99 GAE parameter 0.95 # timesteps per
rollout 21 # epochs per rollout 5 # minibatches per epoch 4 entropy bonus
($\alpha_{2}$) 0.01 value loss coefficient ($\alpha_{1}$) 1.0 clip range 0.2
reward normalization yes learning rate 1e-3 # environments 4096 # total
timesteps 2.58B optimizer Adam

Table 6: PPO hyperparameters.

Parameter Value Units $\textbf{v}_{x}^{{\textrm{cmd}}}$ initial [-1.0, 1.0]
$\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$
$\boldsymbol{\omega}_{z}^{{\textrm{cmd}}}$ initial [-1.0, 1.0]
$\text{\,}\mathrm{rad}\mathrm{/}\mathrm{s}$ $\textbf{v}_{x}^{{\textrm{cmd}}}$
max [-3.0, 3.0] $\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$
$\boldsymbol{\omega}_{z}^{{\textrm{cmd}}}$ max [-5.0, 5.0]
$\text{\,}\mathrm{rad}\mathrm{/}\mathrm{s}$ $\textbf{v}_{x}^{{\textrm{cmd}}}$
bin size 0.5 $\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$
$\boldsymbol{\omega}_{z}^{{\textrm{cmd}}}$ bin size 0.5
$\text{\,}\mathrm{rad}\mathrm{/}\mathrm{s}$ $r_{v^{{\textrm{cmd}}}_{x,y}}$
threshold 0.8 – $r_{\omega^{{\textrm{cmd}}}_{z}}$ threshold 0.7 –
$r_{c^{{\textrm{cmd}}}_{f}}$ threshold 0.95 – $r_{c^{{\textrm{cmd}}}_{v}}$
threshold 0.95 –

Table 7: Curriculum parameters.

Figure 9: Footswing Height vs Robustness: Impact of footswing height on time
to failure on the platform terrain (Section 4.2). Increased footswing height
yields better generalization to uneven terrain from flat terrain compared to
the gait-free policy.

Figure 10: Frequency vs Speed: Impact of trotting frequency on flat-ground
velocity tracking reward across speeds (Section 4.2). Enforcing low frequency
(2Hz) makes high speeds less attainable. The gait-free policy offers slightly
better performance at the lowest and highest speeds.

Figure 11: Flat Ground Velocity Tracking Heatmaps: We provide heatmaps as in
Table 4 for the other major gaits: pronking, pacing, and bounding. In all
cases, the policy forgoes performance on the training task of flat-ground
velocity tracking to achieve diversity that will help accomplish new tasks.
However, in-distribution task performance is maintained well for the lower
range of speeds.