Majid Khadiv, Munich Institute of Robotics and Machine Intelligence (MIRMI),
Technical University of Munich (TUM), Georg-Brauchle-Ring 60-62, Munich 80992,
Germany.

# Learning-based legged locomotion; state of the art and future perspectives

Sehoon Ha11affiliationmark:  Joonho Lee 22affiliationmark:  Michiel van de
Panne 33affiliationmark:  Zhaoming Xie 44affiliationmark:  Wenhao Yu
55affiliationmark:  Majid Khadiv66affiliationmark:  authors are listed
alphabetically (except the last author).
11affiliationmark:  Georgia Institute of Technology, USA
22affiliationmark:  Neuromeka, Korea
33affiliationmark:  University of British Columbia, Canada
44affiliationmark:  The AI Institute, USA
55affiliationmark:  Google DeepMind, USA
66affiliationmark:  Technical University of Munich, Germany
<EMAIL_ADDRESS>
###### Abstract

Legged locomotion holds the premise of universal mobility, a critical
capability for many real-world robotic applications. Both model-based and
learning-based approaches have advanced the field of legged locomotion in the
past three decades. In recent years, however, a number of factors have
dramatically accelerated progress in learning-based methods, including the
rise of deep learning, rapid progress in simulating robotic systems, and the
availability of high-performance and affordable hardware. This article aims to
give a brief history of the field, to summarize recent efforts in learning
locomotion skills for quadrupeds, and to provide researchers new to the area
with an understanding of the key issues involved. With the recent
proliferation of humanoid robots, we further outline the rapid rise of
analogous methods for bipedal locomotion. We conclude with a discussion of
open problems as well as related societal impact.

###### keywords:

Learning locomotion skills, quadrupedal locomotion, Reinforcement learning for
locomotion

##  1 Introduction

Legged robots are complex systems with highly nonlinear, hybrid, and
inherently unstable dynamics. Early demonstrations in the 1980s showed that
simple feedback mechanisms could achieve robust dynamic locomotion for systems
with one, two, and four legs Raibert (1984, 1986). This set the stage for more
than forty years of research on legged locomotion. In particular, we have
witnessed explosive progress in quadruped locomotion over the past ten years
(Figure 1). There are multiple reasons for this rapid progress, including
reliable and affordable hardware, significant improvements in simulation
environments, and scalable learning algorithms for high-dimensional continuous
control problems. There has been further recent success with similar
approaches for bipedal and humanoid robots. We believe this is therefore an
opportune time to broadly summarize the efforts to date as well as to reflect
on future research directions. This article aims to provide current and new
researchers with a big-picture view of how the field has evolved to date. This
review article may also provide a useful foundation for courses covering
_learning-based legged locomotion_. We note that while many learning-based
approaches can be applied more generically to arbitrary robot tasks, e.g.,
including manipulation, we focus specifically on the challenges related to
learning legged locomotion.

Figure 1: Examples of notable learned behaviors executed in the real world.

To place the remainder of the survey in a relevant context, we now provide a
historical perspective of the key elements that have enabled the rapid
advances seen for learning-based legged locomotion, with a focus on
quadrupeds. This includes the evolution of the hardware, the physics-based
simulators required for learning, and the most common methods and algorithms
applied to learning-based control. We also briefly situate this survey itself
with respect to other recent surveys.

###  1.1 Hardware

Hardware for quadrupedal locomotion has evolved considerably over the past few
decades in terms of both capabilities and cost (Figure 2). Torque-controlled
quadruped robots traditionally use one of three actuation mechanisms:
hydraulic, electric motors with torque sensors, and series elastic actuators.
The legged robots from the Leg Lab Raibert (1986) and later the success of the
Bigdog project Raibert et al. (2008) introduced hydraulics as an attractive
actuation mechanism for quadrupedal locomotion. In particular, the large
power-to-weight ratio enables hydraulically-actuated quadruped robots to
perform highly dynamic motions and carry large payloads Doi et al. (2006);
Semini (2010); Rong et al. (2012); Junyao et al. (2013); Semini et al. (2016).
However, hydraulic systems are expensive and require specialized expertise for
design, maintenance, and repair.

Electric motors are the most common choice for actuation mechanism of
quadruped robots Buehler et al. (1998, 1999). To provide sufficient torque at
the output of the joints with an electric motor, a gearing mechanism with a
high ratio (such as harmonic drive) was traditionally required. However, the
large static friction introduced by the gearbox renders the joints non-
backdrivable which precludes output torque control by simply controlling the
motor current. Classically, two modifications were introduced to enable joint
torque control, namely using springs Hutter et al. (2012, 2016) or joint
torque/force sensors BDI (2015) after the gearbox. However, the former design
limits the control bandwidth of the joint, and the latter is sensitive to
large impact forces that could damage the gearbox or saturate (and damage) the
force-torque sensor.

Using custom-made high torque-density actuators, the next generation of
quadrupeds could control their interaction through direct joint current
control with a low gear ratio and without a need for springs and torque
sensors in the robot structure Seok et al. (2014). This ability, sometimes
called proprioceptive actuation Wensing et al. (2017), revolutionized the
field of quadrupedal locomotion. High torque density, high-bandwidth force
control, and the ability to mitigate impacts through backdrivability are some
of the most interesting features of these actuators. Thanks to these features,
proprioceptive actuators have become the dominant paradigm in the design of
legged robots, and a wide variety of quadrupeds have been developed based on
this concept Katz et al. (2019); Grimminger et al. (2020); Shin et al. (2022).
In particular, the open-source initiatives from academia Kau et al. (2019);
Grimminger et al. (2020) as well as inexpensive hardware from industry
Deeprobotics (2021); Unitree (2021); CyberDog (2021) have dramatically
accelerated experimental progress in quadrupedal locomotion. Among the many
different robots, the Unitree A1 and Go1/Go2 are among the most popular
choices for showcasing learned quadrupedal behaviors in the real world, due to
their price and performance.

Bipedal robots and humanoids are currently seeing resurgent interest, given
that much of our world is designed around the human form factor. The hardware
is following a similar evolutionary path as that seen for quadrupedal robots.
Early torque-controlled humanoids often used either hydraulic actuators Smith
(2016); Schaal (2018); Nelson et al. (2018), electrical motors with torque
sensors Englsberger et al. (2014), or series elastic actuators Hubicki et al.
(2016); Abate (2018); Ahn et al. (2019). A more complete summary of humanoid
hardware and software can be found in Goswami and Vadakkepat (2018). Recently,
we are observing the same trend towards humanoids with proprioceptive
actuators Chignoli et al. (2021); Zhu (2023). As with quadrupeds, these
systems will become more affordable and capable. Towards the end of this
survey, we revisit the key differences and opportunities of learned bipedal
locomotion as compared to that for quadrupeds.

Figure 2: Evolution of quadruped hardware over time

###  1.2 Simulators

The availability of highly efficient numerical algorithms for computing
forward and inverse dynamics of articulated rigid body systems gave the
robotics community with the necessary tools to simulate high-dimensional
systems Featherstone (1987). These algorithms can solve forward and inverse
dynamics problems for a high-dimensional articulated system such as a full
humanoid robot in a fraction of a millisecond Carpentier et al. (2019), if
contacts do not exist. However, the bigger challenge for simulating legged
locomotion is to include contact interactions.

Early simulation environments sometimes modeled contact using springs and
dampers, also known as penalty-based methods Marhefka and Orin (1996); Yamane
and Nakamura (2006); Schaal (2009). While being able to simulate compliant
contacts, these models by default result in large penetrations that do not
accurately reflect reality. To address this requires requires increasing the
stiffness of the system and therefore using small simulation time steps that
result in slow simulations. To avoid this _stiff_ numerical behavior, more
recent simulators rely on rigid body contact models, which can be either
elastic or inelastic. In this approach, contact is handled via a
complementarity condition together with friction cone constraints Smith et al.
(2005); Todorov et al. (2012); Coumans and Bai (2016); Lee et al. (2018b);
Hwangbo et al. (2018); Makoviychuk et al. (2021). Access to fast simulation
environments has been a key enabler for the success of deep reinforcement
learning (DRL) for locomotion. GPU-friendly algorithms, e.g., Makoviychuk et
al. (2021) has enabled fast training of complex robotics tasks on a single GPU
Kim et al. (2021). Importantly, it is equally relevant to be able to simulate
the sensors of a legged robot, including robot-mounted cameras, and thus
simulators have also seen advances in this regard.

###  1.3 Control and Learning Algorithms

Early methods to control quadruped robots commonly relied on the use of
simplified template models Blickhan (1989); Papadopoulos and Buehler (2000);
Kajita et al. (2001); Geyer et al. (2002) and simple-yet-effective heuristic-
based strategies Raibert (1986); Pratt et al. (1997). With the introduction of
biologically-inspired approaches such as the central pattern generators (CPG)
Ijspeert (2001), a large body of work employed CPGs for generating oscillatory
and cyclic behaviors for quadrupeds Buchli et al. (2006); Buchli and Ijspeert
(2008); Ijspeert (2008); Spröwitz et al. (2013); Barasuol et al. (2013). CPG-
like parameterizations are still adopted in several learning algorithms to
expedite the training process Miki et al. (2022a); Bellegarda and Ijspeert
(2022); Ruppert and Badri-Spröwitz (2022) or to explain the motion of
biological systems using robots Shafiee et al. (2023). However, the enforced
structure in these approaches limits the range of locomotion behaviors they
can express, which can lead to sub-optimal performance.

Currently, the two dominant approaches to controlling legged robots in multi-
contact scenarios are optimal control (OC) and reinforcement learning (RL)
methods. OC uses a forward model of the system dynamics to help solve for a
locally optimal time-indexed control policy by minimizing a performance cost,
typically over a finite future horizon Raković and Levine (2018). In contrast,
RL solves for a state-indexed optimal policy by maximizing the expected reward
based on collected samples from rolling out a control policy Sutton and Barto
(2018). The common ground between the two approaches is that they ultimately
estimate an optimal policy for a desired task, as described via a cost or
reward function. In recent years, both OC and RL have shown great success in
the control of quadrupeds.

Early work on the use of OC for quadrupedal locomotion cast the problem as a
convex optimization problem, most typically as a quadratic program, either
through using linear dynamics over a finite future time horizon Kalakrishnan
et al. (2011); Park et al. (2015); Di Carlo et al. (2018); Bledt and Kim
(2020) or for the current instant in time, acting as an inverse dynamics
controller Buchli et al. (2009); Mistry et al. (2010); Righetti et al. (2013);
Hutter et al. (2014); Focchi et al. (2017). To enable planning and control for
multi-contact behaviors through a holistic optimal control framework with
contact, several formulations have been proposed, which are based on
differential dynamic programming (DDP) with relaxed contact Tassa et al.
(2012), contact-invariant optimization Mordatch et al. (2012), contact-
implicit optimization Posa et al. (2014), mixed integer convex optimization
Deits and Tedrake (2014); Aceituno-Cabezas et al. (2017), and phase-based
parameterization of the end-effector trajectories Winkler et al. (2018). While
all of these approaches have shown impressive results in simulation, and some
variants have become real-time capable for model predictive control (MPC)
Neunert et al. (2016); Farshidian et al. (2017); Neunert et al. (2018), their
application in the real world has been limited. Most recent efforts to
implement MPC on legged robots have focused on separating contact planning and
whole-body motion generation and can achieve impressive behaviors on real
hardware Bledt and Kim (2020); Li et al. (2021a); Mastalli et al. (2023);
Grandia et al. (2023); Meduri et al. (2023).

While very successful, the use of optimal control and optimization-based
control algorithms Wensing et al. (2023) comes with multiple challenges.
Solving a high-dimensional non-convex optimization problem Wensing et al.
(2023) in real-time is computationally expensive Sleiman et al. (2021); Ponton
et al. (2021); Mastalli et al. (2023); Grandia et al. (2023); Meduri et al.
(2023). Dealing with uncertainties, especially in contact interaction is
difficult Tassa and Todorov (2011); Drnach and Zhao (2021); Hammoud et al.
(2021); Gazar et al. (2023). The estimation and control problems are separated
through the certainty equivalence principle, and direct inclusion of different
sensor modalities, such as vision, into control policy is prohibitively
difficult. These issues motivated efforts towards alternative learning-based
approaches for the control of legged robots.

The use of reinforcement learning for generating stable locomotion patterns
has a history of more than two decades Hornby et al. (2000); Kohl and Stone
(2004). Early works commonly use hand-crafted policies with a few free
parameters that are then tuned using RL on the hardware. Subsequent work
employs the notion of the Poincare map to ensure the cyclic stability of the
gaits Tedrake et al. (2005); Morimoto et al. (2005). These approaches have
enabled a humanoid robot to walk Morimoto and Atkeson (2009), but their
underlying function approximation has limited expressivity, which limits their
application. Later reinforcement learning for quadrupeds mostly use the
framework of stochastic optimal control, e.g., path integral policy
improvement (PI2) Theodorou et al. (2010); Fankhauser et al. (2013).

Thanks to the success of deep learning in the past decade LeCun et al. (2015),
much of the focus of reinforcement learning has shifted to DRL. Since DRL
approaches commonly require a large number of samples (trial-and-error
experience with the system being controlled) to learn control policies, it is
common practice to train the policies in simulation and then transfer them to
the real world Tan et al. (2018); Hwangbo et al. (2019). In contrast with MPC,
DRL approaches make extensive use of offline simulation of the world to learn
a policy. Once learned, the online computation for the deployed policy
consists of a simple forward pass through the neural network (often referred
to as ‘inference’) at each control time step, which is fast and efficient to
compute. Furthermore, while training policies using DRL, robustness to many
types of uncertainty can be achieved by adding noise during training to the
relevant robot parameters and environments Lee et al. (2020); Bogdanovic et
al. (2022). Finally, it is simple to directly incorporate arbitrary sensory
data as input to the policy, including high-dimensional modalities such as
vision. This largely eliminates the need for a separate state estimation
module Miki et al. (2022a); Agarwal et al. (2023), as state estimation becomes
an implicit part of the control policy. These unique features have led to the
success of DRL in controlling quadruped robots on various terrains and under a
wide range of conditions.

###  1.4 Related Survey Papers

Several existing survey papers have some overlap with the scope of our survey.
The most relevant is Zhang et al. (2022) which briefly surveys the use of DRL
for quadrupedal locomotion. Another recent survey paper Bao et al. (2024)
reviews the progress of DRL for bipedal locomotion. In comparison, this paper
aims to provide a more comprehensive overview of the methods and complexities
in this area. We cover not only DRL but also other types of learning for
quadrupedal locomotion, such as imitation-based learning methods. We provide a
historical perspective and further describe how learning-based approaches
complement the rich literature on model-based control methods for legged
locomotion. We further discuss ongoing advances and open directions.
Complementary to our survey, Wensing et al. (2023) focuses on optimal control
methods for locomotion, while Darvish et al. (2023) reviews works on tele-
operation of humanoid robots. The survey provided in Ibarz et al. (2021) has a
focus on DRL for manipulation.

Advances in simulators and simulation environments have been key to enabling
deep learning for robotics. Numerous high-performance open-source simulators
are now available and are accelerating progress, e.g., Coumans and Bai (2016);
Makoviychuk et al. (2021); MJX (2023). Multiple recent works have extensively
reviewed the state of the art in this area and provide detailed comparisons,
e.g., Collins et al. (2021); Kim et al. (2021); Liu and Negrut (2021).

##  2 Theoretical background

Figure 3: Taxonomy of learning algorithms. Popular algorithms are highlighted
in color. On-policy algorithms (red), such as TRPO or PPO, have been the most
frequent choice for legged locomotion. Other off-policy, gradient-free, or
imitation learning algorithms (orange) are also selected for sample efficiency
or better styles. For more details, please refer to the background section.

###  2.1 Markov Decision Process and Reinforcement Learning

For many decades, machine learning (ML) has demonstrated promise for solving
complex problems across a wide range of domains including computer vision,
natural language processing, finance, environmental science, and many more. In
robotics, RL Sutton and Barto (2018) stands out as a popular choice because
the control of robots can naturally be viewed as a sequential decision-making
problem. RL is one of the ML paradigms that seeks to develop an intelligent
agent in dynamic environments to maximize cumulative rewards.

Typically, RL models the control problem using the Markov decision processes
(MDP), which is defined as a tuple $(\mathcal{S},\mathcal{A},p,r,\gamma)$ of
the state space $\mathcal{S}$, action space $\mathcal{A}$, deterministic
$\mathbf{s}_{t+1}=f(\mathbf{s}_{t},\mathbf{a}_{t})$ or stochastic transition
function $p({\mathbf{s}_{t+1}}|{\mathbf{s}_{t}},\and)$, reward function
$r({\mathbf{s}_{t}},\and,{\mathbf{s}_{t+1}})$, and a discount factor $\gamma$.
By executing a policy $\pi(\and|{\mathbf{s}_{t}})$ in a stochastic
environment, we can generate a trajectory of state and actions
$\tau=(\mathbf{s}_{0},\mathbf{a}_{0},\mathbf{s}_{1},\mathbf{a}_{1},\ldots)$.
We denote the trajectory distribution induced by $\pi$ as
$\rho_{\pi}(\tau)=p(\mathbf{s}_{0})\prod_{t}\pi_{t}(\mathbf{a}_{t}|\mathbf{s}_{t})p(\mathbf{s}_{t+1}|\mathbf{s}_{t},\mathbf{a}_{t})$.
Our goal is to find the optimal policy $\pi^{*}$ that maximizes the sum of
expected returns:

$J(\pi)=\operatorname{\mathbb{E}}_{\tau\sim\rho_{\pi}}\left[\sum_{t=0}^{T}r({\mathbf{s}_{t}},\and)\right].$
(1)

In basic movement scenarios that involve navigation over a flat ground plane,
the MDP state is given by the state of the robot alone, i.e., the information
needed to describe position and velocities of all the links of the robot. In
more interesting and complex scenarios, the MDP state should encapsulate both
the state of the robot and the state of the surrounding environment. In
practice, many variations of MDPs are commonly used in order to accommodate
diverse scenarios. For example, partially-observable MDPs (PoMDPs) capture the
need to make control decisions using incomplete observations instead of the
full MDP state, as defined by the observation space $\mathcal{O}$ and the
observation function $o(\mathbf{o}_{t}|\mathbf{s}_{t})$. The PoMDP formulation
is common in robotics, where a robot camera can only observe part of the world
at any given time, for example. The available observations are sometimes
directly referred as the state, $\mathbf{s}_{t}$, which is a slight abuse of
notation which simnply allows policies to always be described as performing a
state-to-action mapping, even when the actual mapping being used is from
observations-to-actions.

Researchers have developed various RL algorithms to address MDP problems.
Early algorithms can be broadly categorized into value-function and policy
iteration methods. Value-function methods, exemplified by Q-learning Watkins
and Dayan (1992), SARSA Rummery and Niranjan (1994), or (Fitted) Value
Iteration Bellman (1966); Boyan and Moore (1994), aim to estimate the expected
values of states or state-action pairs, which implicitly define the optimal
policy. Conversely, policy iteration methods, whether gradient-based Howard
(1960) or gradient-free Hansen et al. (2003), frame the given MDP as an
optimization problem and seek to find the best parameters from a policy
standpoint. These algorithms have found successful applications in diverse
robotics problems, including legged locomotion.

Despite promising results, these early algorithms encounter challenges in
solving complex, large-scale control problems, mainly due to scalability and
convergence issues. Nonetheless, the established theoretical advancements
demonstrate significant performance enhancements when combined with deep
learning, which opens a new era of research.

###  2.2 Deep Reinforcement Learning

Deep learning LeCun et al. (2015); Goodfellow et al. (2016) stands out as one
of the most significant breakthroughs in ML. Artificial neural networks,
composed of layers of neurons, are recognized as universal function
approximators that can learn any multi-variate functions Hornik (1991);
Cybenko (1989). Various researchers have demonstrated that neural networks can
effectively solve complex regression problems by processing vast amounts of
data if they are provided with appropriate network architectures and learning
frameworks.

The pioneering work of Mnih et al. (2015) demonstrated that RL can also be
combined with deep learning to achieve human-level performance for Atari
games. This achievement was made possible by introducing replay buffers and
target networks to stabilize learning. Following Deep Q Network (DQN), several
innovations have emerged to improve convergence and sample efficiency Wang et
al. (2016); Van Hasselt et al. (2016); Hessel et al. (2018). While many
algorithms addressed simple discrete actions, robotic control usually requires
continuous action spaces to generate motor commands. Deep deterministic policy
gradient (DDPG) Lillicrap et al. (2015) was one of the first algorithms
supporting continuous actions, followed by numerous other RL algorithms and
variations, e.g., Fujimoto et al. (2018); Haarnoja et al. (2018b); Mnih et al.
(2016); Abdolmaleki et al. (2018); Schulman et al. (2015, 2017). Model-based
RL holds the premise of achieving the best sample efficiency by utilizing the
learned world model of the given MDP either in state space Nagabandi et al.
(2018); Feinberg et al. (2018); Buckman et al. (2018); Chua et al. (2018);
Janner et al. (2019) or image space Ha and Schmidhuber (2018); Hafner et al.
(2019b, a). Nevertheless, recent model-free approaches Chen et al. (2021);
Hiraoka et al. (2021) have also shown competitive sample efficiency. Besides
sample efficiency, there have been numerous efforts to improve the scalability
of RL algorithms for both gradient-based methods Espeholt et al. (2018);
Barth-Maron et al. (2018); Wijmans et al. (2019) and gradient-free approaches
Mania et al. (2018).

In the domain of learning-based locomotion, on-policy model-free algorithms
like trust region policy optimization (TRPO) Schulman et al. (2015) and
proximal policy optimization (PPO) Schulman et al. (2017) are often preferred
choices. This preference arises from a pursuit of optimal-and-robust
performance while being less concerned with sample efficiency. PPO has been a
particularly popular choice in the legged locomotion community due to its
excellent convergence and adaptability to diverse policy architectures. Table
1 provides a detailed taxonomy of the most common RL algorithms, and
identifies those most commonly used for learning legged locomotion.

It is important here to clarify one common misconception when the term _model-
free_ is used in the context of DRL in robotics. While none of the model-free
DRL algorithms need a model of physics of the world _per se_ , in practice
these policies are trained in simulation environments that are physics models.
This means that application of model-free DRL methods relies heavily on models
(simulation environments) that are developed using first principles. Hence,
the term _model-free_ in the context of DRL in robotics should be taken with a
grain of salt.

Table 1: Common algorithms and use-cases for legged locomotion. A popular option has been on-policy RL, such as TRPO or PPO, due to their convergence to the best performing policies. However, other algorithms have been used for other reasons, such as better explotation, sample efficiency, or scalability. |  Example Papers
---|---
Reinforcement Learning |  Off-policy Learning | DDPG |  Bogdanovic et al. (2020); Huang et al. (2022)
SAC |  Haarnoja et al. (2019); Ha et al. (2020); Smith et al. (2023)
On-policy Learning | TRPO |  Hwangbo et al. (2019); Lee et al. (2020); Yang et al. (2022b)
PPO |  Iscen et al. (2018); Tan et al. (2018); Peng et al. (2020); Kumar et al. (2021); Margolis et al. (2021); Rudin et al. (2022b); Xie et al. (2022); Miki et al. (2022a); Zhuang et al. (2023); Agarwal et al. (2023); Liu et al. (2024)
Model-based | Dreamer |  Wu et al. (2023b)
| Gradient-free | ARS |  Yu et al. (2020, 2021)
Imitation Learning | Behavior Cloning |  Lee et al. (2020); Kumar et al. (2021); Liu et al. (2024); Reske et al. (2021)
GAIL |  Escontrela et al. (2022)

###  2.3 Behavior Cloning and Imitation Learning

As we discussed in the previous subsection, the main goal of RL is to find
control policies that maximize the cumulative reward, in a model-based or
model-free fashion. Either ways, we assume that the reward function is given
to us and the algorithm collects data to learn a model (model-based), or a
value/policy function (model-free). While this has shown to very promising in
practice, it requires an imense amount of reward engineering, in particular
for problems with sparse reward.

Learning to imitate actions from an expert human demonstration or an existing
expert policy Pomerleau (1988) can be seen as a remedy to this problem.
However, naively mimicking the actions of the expert often fails due to the
compounding nature of errors in control problems and the resulting
distribution mismatches between the states visited during training and those
seen at runtime. To mitigate this issue, researchers have explored alternative
approaches for developing robust policies, such as dataset aggregation
(DAgger) Ross et al. (2011) or generative adversarial imitation learning
(GAIL) Ho and Ermon (2016). One notable usage of IL in learning-based
locomotion is privileged learning Chen et al. (2020), which learns a capable
teacher policy with ground-truth information first and copies the behaviors
into a student policy with a realistic sensor configuration. We will revisit
this particular strategy later, given its importance.

##  3 Components of MDP for Locomotion

As mentioned in the previous section, RL algorithms typically frame sequential
decision-making problems as MDPs. In this section, we will provide a brief
summary of common practices in MDP formulation for legged robot control
problems, including dynamics, states/observations, reward, and action spaces.

###  3.1 Dynamics

A legged robot can be described as a (typically deterministic) dynamical
system, with the equations of motion given by
$\dot{\mathbf{s}}=f(\mathbf{s},\mathbf{a})$, where $\mathbf{s}$ is the state
of the robot and the environment, and $\mathbf{a}$ is the action that the
robot can take. To formulate it as an MDP, we discretize the dynamics with
respect to time,
$\mathbf{s}_{t+1}=\mathbf{s}_{t}+f(\mathbf{s}_{t},\mathbf{a}_{t})dt$, with
$dt$ being the discretization step. In MDP notation, this is modeled using a
stochastic transition function $p({\mathbf{s}_{t+1}}|{\mathbf{s}_{t}},\and)$
having the probability mass lumped at ${\mathbf{s}_{t+1}}$. The state-
transition data tuples are generated from the dynamics,
$({\mathbf{s}_{t}},\and,\mathbf{r}_{t},{\mathbf{s}_{t+1}})$, and used for the
RL algorithms.

####  3.1.1 Simulators

Since DRL requires a large number of samples to train a policy, it is common
practice to learn policies in simulation and then directly deploy these
policies in the real world. As noted earlier, fast and accurate simulators
have a played a critical role for recent advances in learning locomotion
behaviors.

State-of-the-art simulators used in robotics commonly rely on a rigid contact
(elastic or inelastic) assumption. They model contact using a complementarity
condition together with friction cone constraints (linear or nonlinear)
Todorov et al. (2012); Coumans and Bai (2016); Hwangbo et al. (2018);
Makoviychuk et al. (2021). Also, there has been a tremendous effort towards
making these simulation environments differentiable Geilinger et al. (2020);
Le Lidec et al. (2021); Howell et al. (2022) which would make it possible to
perform efficient system identification through contact or to directly update
the parameters of a policy via backpropagation through time. Multiple recent
works have extensively compared the simulation environments and we refer the
readers to those works for a complete survey Collins et al. (2021); Kim et al.
(2021); Liu and Negrut (2021) and to Lidec et al. (2023) for an extensive
survey of the multitude of methods used by simulators to solve for (rigid)
contact in robotics. The GPU-friendly nature of Isaac Gym Makoviychuk et al.
(2021) allows for collecting state-transition data at a rate of nearly one
million per second, makes it especially interesting for DRL Kim et al. (2021).
Relatedly, the new version of MuJoCo includes MuJoCo XLA (MJX) which makes it
GPU-compatible via the JAX framework MJX (2023). The trend of recent simulator
usage for learning-based locomotion is illustrated in Figure 4, which captures
early-developed communities for PyBullet and MuJoCo and also highlights the
recent rise of Isaac Gym.

Relying on a rigid contact model that disallows any compliance or penetration
is a limitation when dealing with tasks where interacting with a compliant
environment is essential. Recently, researchers in both fields of locomotion
and in-hand manipulation have looked into more advanced contact models to
better capture the complexity of compliant interactions Khadiv et al. (2019);
Masterjohn et al. (2022); Choi et al. (2023). For instance, Khadiv et al.
(2019) proposes a nonlinear compliant contact model that can handle both rigid
and soft interaction scenarios. In Masterjohn et al. (2022), the idea of
pressure field contact patches is re-visited to foster their use with
available velocity-level time steppers which enable simulation of compliant
interactions with real-time rates. In Choi et al. (2023), on the other hand,
an analytical nonlinear point contact model is used. To match the behavior of
the model with the real-world data, the authors randomized the stiffness and
damping ratio of the terrain such that the distribution of the simulation
results matched the experimental distributions. To enable interaction with a
wide range of environments and objects, a crucial component for learning-based
algorithms will be reliable and fast simulation environments that can model a
wide range of physical interactions.

####  3.1.2 Real robot

Another possible choice for generating motion data is to directly use the real
robot to record the reward Saggar et al. (2007) or the resulting transitions
$\\{\mathbf{s}_{t},\mathbf{a}_{t},\mathbf{s}_{t+1}\\}$ Haarnoja et al. (2019);
Choi and Kim (2019); Yang et al. (2020). This approach can potentially
guarantee performance in the real-world environment by avoiding the need to
build a model of the system and ensuring the fidelity of the data being used
to train our ML model. However, collecting such data can be costly, which
greatly limits its scalability. In addition, safety becomes a more important
issue, especially for legged robots, because their underactuated base make
them vulnerable to falls and also do not offer reliable mechanisms for
resetting to a known initial state to begin every new episode. Therefore,
additional mechanisms are required to mitigate the safety issues, such as
safety-aware RL Ha et al. (2020); Yang et al. (2022b), automatic resetting
Luck et al. (2017); Ha et al. (2018), failure recovery Smith et al. (2022), or
simulation-based pre-training Smith et al. (2023).

Figure 4: A rough estimation of the usage of various simulators for locomotion
learning. The numbers are simply collected using a google scholar search with
the following keywords ”simulator name” ”legged” ”robot” ”hardware”
”learning”. For the year 2024, as the numbers are for the first 5 months, we
simply multiplied the numbers by 12/5.

###  3.2 Observation

One crucial aspect of MDPs is the observations and how to infer the states
from them. The observation space is comprised of noisy sensory measurements
that give information about the state of the robot and the environment. In the
following we categorize the body of works in the field of locomotion based on
the types of observations they use for learning policies.

####  3.2.1 Proprioception

The set of sensors that provide information about the internal state of the
robot is called proprioception, like IMU, joint encoders, or contact sensors.
In legged robotics, raw measurements from these sensors are typically not
directly used due to potential noise and inaccuracies. Instead, a state
estimator is typically used to estimate crucial states of the robot such as
base pose, base twist, joint position, and velocity. Such states are often
referred to as _proprioceptive states_ in DRL works, including Lee et al.
(2020); Nahrendra et al. (2023); Yang et al. (2021); Yu et al. (2023b).

Recently, Ji et al. (2022a) and Lee et al. (2024) have used sequences of IMU
measurements and joint encoder data directly, instead of relying on pre-
estimated velocities and poses. Specifically, Ji et al. (2022a) implemented an
RNN-based state estimator, diverging from traditional model-based estimators,
to directly compute the base velocity and contact states. On the other hand,
Lee et al. (2024) employed a privileged training strategy, effectively
utilizing the raw sensor data sequences to enhance the robustness.

The emerging research in legged robotics highlights the pivotal roles of joint
positions, base pose (often represented by the gravity vector), and base
linear and angular velocities in developing robust locomotion skills Yu et al.
(2023b). A significant body of work, including Lee et al. (2020), Ji et al.
(2022a), and Haarnoja et al. (2019) emphasize the importance of leveraging a
history of proprioceptive states and joint command information. This approach
addresses the non-Markovian characteristics of robotic systems that stem from
hardware latencies and partial observations.

In particular, historical data of proprioceptive states provide critical
insights into foot-terrain interactions and external disturbances, as
demonstrated by Lee et al. (2020) and Ji et al. (2022a). By utilizing
kinematic information collected over time, robots are equipped to make more
informed decisions. This capability significantly enhances their performance,
enabling robustness over unstructured, real-world terrains.

####  3.2.2 Exteroception

The sensory layer of the robot that provides information about the surrounding
environments is normally referred to as exteroception. These information are
in particular vital when the robot moves on non-flat environments.
Conventional approaches in legged robotics often rely on explicit mapping
techniques to pre-process the measurements before they are used by the
controllers. For example, methods like elevation mapping Miki et al. (2022b)
or voxel mapping Oleynikova et al. (2017); Besselmann et al. (2021) are
commonly employed. Early works on perceptive locomotion Miki et al. (2022a);
Xie et al. (2022) adopted elevation mapping, where height values around the
robot were sampled and used as observation to the policy.

Recent works removed such explicit mapping and instead used raw sensory
readings directly as input to the policy, such as depth images or point
clouds. This shift is driven by the necessity to effectively handle highly
dynamic situations such as parkour Zhuang et al. (2023) or obstacle avoidance
Yang et al. (2021), and tasks requiring high-resolution perception, such as
stepping stone scenarios Duan et al. (2022); Zhang et al. (2023). This
approach not only facilitates more responsive and accurate locomotion in
complex environments, but also reduces the likelihood of errors and
inaccuracies that often arise from mapping failures.

RGB data can also be used for a more evolved perception of complex scenes that
go beyond mere geometric information, enabling sidewalk navigation and
obstacle avoidance Sorokin et al. (2022). This modality includes a richer
context, recognizing and responding to elements that cannot be captured
through geometry alone, such as textures and colors. Semantics of the scenes
can also be learned for more efficient navigation. For instance, Yang et al.
(2023c) utilize semantic information from images to adapt the locomotion gaits
and speed of a quadrupedal robot to handle different terrains. Margolis et al.
(2023) further augment RGB data with proprioceptive data for better
traversability estimation.

Instead of feeding the sensory values directly into the policy, a compressed
representation of them can be learned. For example, Hoeller et al. (2021);
Yang et al. (2023b) use unsupervised learning to obtain a latent space that
can compress and reconstruct the original camera images, enabling learning
policies that can navigate complex terrains automatically.

####  3.2.3 Task-related inputs (Goals)

Apart from proprioception and exteroception, other information specific to the
robot’s task, including command inputs like velocity and pose, or more complex
data representations such as learned task embeddings can be included as input
to the policy. Note that these inputs can be seen more as goals than
observations, but to not overload the sections in the paper we cover them
under observation.

Velocity commands are frequently used to accurately control locomotion. Pose
commands, specifying particular position or orientation targets, are also
employed Rudin et al. (2022a). In addition to direct command inputs, some
approaches involve the use of learned task embeddings, which could be a latent
representation of a specific reference motion Peng et al. (2022) or any latent
space that can direct the low-level behavior Haarnoja et al. (2018a).

For robotic systems employing structured action spaces, such as those based on
CPGs or trajectory parameters, task-related information can also include
specific details about the desired phase or trajectory patterns Iscen et al.
(2018); Lee et al. (2020).

###  3.3 Reward

The reward function plays a crucial role in delivering the desired behavior by
the RL algorithm, as it defines the objective of the agent. A common approach
is to formulate the reward function as a linear combination of various reward
and penalty terms, denoted as
$r(\mathbf{s}_{t},\mathbf{s}_{t+1},\mathbf{s}_{t})=\sum_{i}c_{i}r_{i}(\mathbf{s}_{t},\mathbf{s}_{t+1},\mathbf{a}_{t})$,
while it is also possible to use multiplicative combination Kim et al. (2022).
There are various methods to define each reward component.

####  3.3.1 Manual reward shaping

Manual reward shaping involves defining each reward component $r_{i}$ and
tuning each weight $c_{i}$ by the engineer (hence a.k.a. reward engineering).
Some examples of these components are velocity tracking, pose tracking, and
other regularization terms. Also, normally the physical constraints of the
system is added as a soft cost to this function, e.g., limiting the magnitude
of joint velocity, acceleration, and base pose during locomotion.

In general, there exist no general set of rules one can follow to define the
components of the reward function. One common practice is to use bounded
functions for stable training. This is often done by simply clipping or
applying exponential kernels. For instance, exponential functions such as
$exp(-c\lvert\lvert e\rvert\rvert^{2})$ or $exp(-c\lvert\lvert
e\rvert\rvert)$, with $e$ representing the error term and $c$ being the
shaping coefficient, are frequently employed for tracking tasks Lee et al.
(2020); Rudin et al. (2022b); Duan et al. (2022). A set of commonly used
reward components are summarized in Table 2.

Table 2: Common rewards used for training quadruped locomotion. An asterisk ($*$) indicates a target quantity. The function $\phi(x):=\exp(-k\lvert\lvert x\rvert\rvert^{2})$, where $k$ is a scaling factor, is used to define bounded rewards. Here, $e_{z}^{b}$ represents the z-axis of the world in the base frame, $q$ denotes the joint angle, $\tau$ represents the joint torque, and $a_{t}$ is the action. Reward functions can be similarly defined using different norms or element-wise operations.  Horizontal velocity tracking | $\phi(v_{xy}^{*}-v_{xy})$
---|---
Yaw rate tracking | $\phi(\omega_{z}^{*}-\omega_{z})$
Base z motion |  $-v_{z}^{2}$ or $\phi(v_{z})$
Roll and pitch rate |  $-\lvert\lvert\omega_{xy}\rvert\rvert^{2}$ or $\phi(\omega_{xy})$
orientation | $-\lvert\lvert[0,0,1]^{T}-e_{z}^{b}\rvert\rvert^{2}$
Joint velocities | $-\sum_{i\in\text{joints}}\|\dot{q_{i}}\|^{2}$
Joint accelerations | $-\sum_{i\in\text{joints}}\|\ddot{q_{i}}\|^{2}$
Joint torques | $-\sum_{i\in\text{joints}}\|{\tau_{i}}\|^{2}$
Joint mechanical power | $-\sum_{i\in\text{joints}}({\tau_{i}}*\dot{q_{i}})^{2}$
Action rate | $-\|\mathbf{a}_{t}-\mathbf{a}_{t-1}\|^{2}$
Action smoothness | $-\|\mathbf{a}_{t}-2\mathbf{a}_{t-1}+\mathbf{a}_{t-2}\|^{2}$

####  3.3.2 Imitation reward

We usually have a rough idea of how a legged robot should move due to the
biologically inspired nature of these robots. These behaviors cab be obtained
from motion capture data of humans and and animals (or online videos). Reward
signals can be designed using this information, which greatly reduces the
engineering effort required.

There exist many works Peng et al. (2018, 2020); Han et al. (2023); Yang et
al. (2023a) that leverage dog motion capture data to enable quadrupedal robots
to learn animal-like motions, which is often approached by traditional model-
based control approaches Kang et al. (2021); Li et al. (2023d). In this motion
imitation problem, the reward is designed to simply imitate the given
reference trajectory, which can be obtained by either retargeting motion
capture data of a real canine Peng et al. (2020); Klipfel et al. (2023),
model-based controller Reske et al. (2021), or solving trajectory optimization
Kim et al. (2023b).

Instead of simply tracking the given reference motion, a set of motion clips
can be used to define the style of legged robot movements, such as gait
frequency and base motions. Generative adversarial imitation learning (GAIL)
Ho and Ermon (2016) simultaneously trains a discriminator along with a policy,
which distinguishes whether the motion is from the existing database or
generated by a policy. Then, the learned discriminator can serve as a generic
motion prior to realistic or stylistic movements, which can be co-optimized
with downstream task rewards. The concept of adversarial motion prior as a
substitute for complex reward terms have been used in multiple recent works
Escontrela et al. (2022); Wu et al. (2023a); Vollenweider et al. (2023); Li et
al. (2023b, c).

###  3.4 Action Space

Action space plays an important role in the performance of learned controllers
for robotics, especially in the case of RL where exploration happens in the
action space and the appropriate choice of action space could greatly improve
exploration efficiency.

####  3.4.1 Low-Level Joint Commands

Most works in quadrupedal learning use joint target position as the action
space (PD policy). For each joint of the robot and a given action value $a(s)$
for that joint, the motor will attempt to generate a torque of
$\tau=k_{p}(a(s)-\theta)-k_{d}\dot{\theta}$ to move the joint ($k_{p}$ and
$k_{d}$ are the gains, while $\theta$ and $\dot{\theta}$ are the measured
joint angles and velocities). In addition to its practicality, another reason
this is the default choice may be attributed to Peng and Van De Panne (2017),
where they show using joint position target as action space performs the best
for physics-based character animation tasks, compared to other choices such as
torque and velocity. Also, Bogdanovic et al. (2020) systematically showed on a
one-legged hopping task that varying the gains of the controller as a function
of states can improve the performance, when contact is highly uncertain.

More recently, Chen et al. (2023) also demonstrates successful learned
quadrupedal behaviors without imposing any structure in the action space and
directly outputting torque (torque policy). The main benefit of this approach
is that the policy function is not limited by the choice of the imposed
structure as in the PD policy case. However, in this case one needs to
increase the frequency of policy evaluation at run-time (e.g., 1 KHz), whereas
the PD policy can be updated much slower (e.g., 50 Hz) with torque still being
produced at 1 KHz. Furthermore, it is argued in Bogdanovic et al. (2020) that
the sim-to-real transfer for the PD policy is easier. It is important to note
that, as it is also explained in Hwangbo et al. (2019), the PD policy is
different from position control traditionally used in robotics. For position
control, the controller tracks a _desired time-indexed trajectory_ with large
gains. However, no desired trajectory is tracked by the PD policy (note that
the desired velocity is zero) and the target position is never achieved during
the motion. In fact, both the torque policy and PD policy are used for torque
control on the real robot.

####  3.4.2 Structured Action Spaces

Prior knowledge of how the robot should move can also be embedded to the
action space via the use of residual RL Johannink et al. (2019). In this
approach, an open-loop reference control signal $\hat{\mathbf{a}}_{t}$ is
provided, and policies are learned to generate feedback signals, which are
then added to the reference. For example, in Iscen et al. (2018), reference
signals $\hat{\mathbf{a}}_{t}$ is a sinusoidal wave over time that encourages
the policy to generate desired gaits such as trotting and bounding, and the
sum $\mathbf{a}_{t}=\hat{\mathbf{a}}_{t}+\pi(\mathbf{s}_{t})$ are used as the
joint target for the low-level PD policy.

One can also learn to modulate parameters of high-level motion primitives. For
example, Bellegarda and Ijspeert (2022) learns to modulate the intrinsic
oscillator amplitude and phases of a central pattern generator. Xie et al.
(2022); Margolis et al. (2021) learns to output the desired center of mass
accelerations, which are then used to generate motor commands using the single
rigid body model. We will further detail these approaches in the future
sections when talking about the combination of control and learning
algorithms.

##  4 Learning Frameworks

In the previous section, we discussed the common formulation of MDPs for
legged locomotion. Solving the MDP is a highly challenging non-linear
optimization problem and thus there has been extensive research to efficiently
solve this problem. In this section, we discuss popular choices of learning
frameworks researchers have explored to solve the MDP for locomotion policies.

###  4.1 End-to-end Learning

The most straightforward approach would be to treat the given MDP as a
monolithic formulation and solve everything end-to-end using DRL algorithms.
Among diverse choices, the most popular DRL algorithm for solving legged
locomotion is an on-policy algorithm, such as TRPO Schulman et al. (2015) or
PPO Schulman et al. (2017). These algorithms are designed to take conservative
update steps by constraining the parameter changes within the _trust region_
defined in the vicinity of the current policy. As a result, the on-policy
algorithms provide a robust learning framework that can reliably find high-
performing policies at the end, which makes them suitable for challenging
locomotion problems. However, off-policy algorithms, such as DDPG Lillicrap et
al. (2015) or SAC Haarnoja et al. (2018b), have also been adopted by many
researchers, particularly when the sample efficiency becomes a bottleneck of
the problem.

End-to-end learning works well when the initial policy can effectively explore
states that provide learning signals. However, for more challenging tasks,
such as learning a robust locomotion policy under large disturbance, the
initial policy may not be able to obtain any useful reward signal and thus
make it difficult for end-to-end methods to work well. In such cases, adding
more problem-specific structures to the problem can be helpful, e.g.,
curriculum and hierarchical learning.

Figure 5: Illustration of popular learning frameworks: (a) basic learning, (b)
curriculum learning, (c) hierarchical learning, and (d) privileged learning.

###  4.2 Curriculum Learning

Similar to the way schools create progressively challenging learning
milestones to enhance the educational outcomes, researchers have also explored
a similar idea known as curriculum Learning (CL) for handling more difficult
problems in robot learning. CL can be applied at different aspects of the
learning framework. For example, Heess et al. (2017); Xie et al. (2020b);
Rudin et al. (2022b); Margolis et al. (2024) trained virtual and real agents
in increasingly challenging environments and obtained policies that can handle
environments that basic learning cannot achieve. CL has also been applied to
improve robustness of the policies by imposing increasing amount of
disturbances and randomness throughout the training Akkaya et al. (2019). CL
has also been used to learn policies that can satisfy desired constraints by
starting with a soft constraint and gradually tightening it Zhuang et al.
(2023).

###  4.3 Hierarchical Learning

Another common strategy to handle more challenging learning problems,
especially ones with long horizons like navigation, is to decompose the
problem into hierarchies. Typically, the problem is decomposed into high-level
tasks and low-level skills, where the action space of the high-level task will
be part of the input of the low-level skills, and each level is learned
separately.

Hierarchical learning is adopted in several learning-based control works in
the computer graphics literature. The decomposition of the problem can be
based on human intuition, for example, in Peng et al. (2017), a high-level
policy will prescribe the desired footsteps to accomplish tasks such as
navigation and soccer dribbling, while the low-level policy will take the
desired footsteps as input to generate joint level control to follow the
footsteps. The output of the high-level policy can also be a learned latent
space. For example, Peng et al. (2022); Won et al. (2022); Zhu et al. (2023)
present systems where the low-level policies are trained to reproduce diverse
motion capture data, driven by a learned latent representation. High-level
policies are then trained to manipulate these latent spaces to accomplish
high-level tasks.

Similar frameworks are adopted in several quadrupedal robot locomotion works.
Han et al. (2023) learns low-level policies using the vector quantized
variational encoder (VQ-VAE) Van Den Oord et al. (2017). This allows a
quadrupedal robot to reproduce a variety of motion capture data of animals and
a high-level policy to manipulate the latent space of the VQ-VAE to handle
tasks like navigation and multi-agent chase tag game. Ji et al. (2022b)
divides a soccer shooting task into an end-effector motion tracking task and
an end-effector trajectory planning task, where the output of the trajectory
planning is used as input for the motion tracking policy. The two tasks are
learned separately to accomplish quadruped soccer shooting. Ji et al. (2023)
similarly learns task and recovery policies for soccer dribbling.

There are also hierarchical architectures that combine a model-based control
approach and a learning-based approach, where a control policy can either be
learned or manually designed. We leave the discussion to a later section in
the paper.

###  4.4 Privileged Learning

Robotic tasks in the real world are inherently partially observable, leading
to challenges in achieving high-performance locomotion. For instance,
accurately measuring or estimating factors like payload weight or friction
coefficient is nearly impossible, but crucial for the policy to be robust or
adaptive.

To address these challenges, existing approaches often involve constructing a
belief state from a history of available measurements. In DRL, this is
typically accomplished by stacking a sequence of previous observations
Haarnoja et al. (2018b); Hwangbo et al. (2019) or using models with memory
such as RNN or TCN. However, training a complex neural network policy with
such a large input space from scratch using RL can be time-consuming and
challenging.

Chen et al. (2020) proposes a strategy called _learning by cheating_ , which
was renamed to privileged learning in the context of quadrupedal locomotion
Lee et al. (2020). This approach leverages fully observable MDP in simulation
to train a _teacher_ policy, which is then distilled into a _student_ policy
based on sequence models like RNN or TCN. Specifically, the teacher has access
to privileged information, such as noiseless, rich simulated state of the
agent (e.g., friction coefficient). Chen et al. (2020) collected human
demonstrations in simulation, while Lee et al. (2020) trained a teacher policy
for quadrupedal locomotion using noiseless robot states and terrain
information as privileged information. At test-time, the student policy has no
access to such information but instead processes a history of measurements
available from the robot.

Lee et al. (2020) first demonstrated the effectiveness of this approach for
quadrupedal locomotion, enabling rough terrain traversal that outperformed
conventional optimization-based approaches. By imitating the fully-observing
teacher policy, the student policy constructs an internal representation of
the world that conveys locomotion-related information. This idea of combining
RL and imitation learning has been adopted by subsequent works to enhance
robustness or utilize more complex input modalities such as raw images or
voxel maps Miki et al. (2022a); Haarnoja et al. (2024); Agarwal et al. (2023);
Miki et al. (2024); Lee et al. (2024).

##  5 Sim-to-Real Transfer

The promise of computer simulations to efficiently generate large amounts of
training data creates a great synergy with modern deep learning algorithms
that are capable of absorbing large-scale datasets and often require abundant
data sources to achieve effective learning. However, a major challenge for
policies that are trained using simulation data is whether they will perform
well on real hardware due to the discrepancies between computer simulation and
real-world robot dynamics, also known as the sim-to-real gap. It is not
practical to construct a simulation model to accurately represent every aspect
of the real world due to its complexity. For example, motors can heat up over
time, showing a different behavior given the same command input, non-
deterministic delays exist due to communication of different parts of the
hardware, and the feet of the robots are usually made of soft materials
instead of the rigid body model commonly used in most simulators. Furthermore,
a policy trained entirely in simulation can take advantage of some aspects of
the simulation that are not present in reality, resulting in direct failure
upon direct deployment. In this section, we review some of the techniques
commonly used to combat the sim-to-real gap.

###  5.1 Good System Design

A naive implementation of the environment for locomotion tasks often leads to
undesirable behaviors that take advantage of the peculiarities of the
simulators. For example, a reward that encourages forward walking motion for
the commonly used MuJoCo humanoid often generates control policies that
resemble bang-bang control, with the joints oscillating at high frequency,
resulting in behaviors that glide forward. Such a bang-bang control strategy
is not suitable for hardware and sim-to-real is doomed to fail. A good design
of the overall system helps to constrain the policies to operate in the same
distribution as the simulator and to combat the sim-to-real gap.

##### Reward Design

The design of the reward function can directly affect the sim-to-real
performance. In the previous sections, we described the most common reward
terms, but we reiterate some important elements here. To avoid jittery motion,
the acceleration of the joints is often penalized; a reward for foot air time
is often used to avoid foot-dragging behavior; foot impact penalty is used to
avoid stomping behaviors Makoviychuk et al. (2021). Maintaining a good balance
between task rewards such as velocity tracking and regularization rewards
makes the reward shaping problem a laborious task.

##### Observation and Action Space Design

The choice of observation and action space also plays an important role. In
Xie et al. (2021), experiments identify several key factors that enable direct
sim-to-real transfer for the Laikago robot, without randomization or
adaptation techniques commonly used in other works. In particular, a low
proportional gain for the joint PD controller that allows for compliant
behavior greatly reduces sim-to-real gap, and the use of a state estimator
that allows for using body velocity as part of the observation allows for
rejecting velocity perturbation.

##### Domain Knowledge

Domain knowledge can also be used to improve the system design. For example,
in Xie et al. (2020a), symmetry constraints are used to promote the left-right
symmetry of bipedal walking policies, greatly improving the motion quality and
as a result greatly improving sim-to-real performance. Domain knowledge can
also be used in the reward function to promote certain behaviors, for example,
the use of motion capture data in the reward function can promote the style of
the motions Peng et al. (2020); Han et al. (2023) or using CPGs as a structure
in the policy Lee et al. (2020); Bellegarda and Ijspeert (2022); Iscen et al.
(2018) can lead to more desirable gait patterns.

With good system design, a few works demonstrated sim-to-real success for
legged robot locomotion without dynamics randomization, e.g., Smith et al.
(2023) demonstrates successful sim-to-real transfer for the A1 quadruped for
highly dynamics motion such as jumping and walking on hind legs. Xie et al.
(2020a); Dao et al. (2020) successfully show sim-to-real for the bipedal robot
Cassie. These examples demonstrate that a good design can sometimes help
bridge the sim-to-real gap, without a need for any further strategies.

###  5.2 System Identification

Even with good system design, inaccurate modeling or unmodelled dynamics in
the physical system can cause direct sim-to-real to fail. To improve the
fidelity of the simulator, system identification can be performed. A dominant
source of modeling errors comes from the actuator dynamics. In simulation,
motors can apply arbitrary torque profiles that a policy commands, while a
physical actuator often produces less accurate torque profiles due to the
limited bandwidth as well as limited tracking accuracy of the underlying motor
controller. One of the earliest successful sim-to-real transfer for
quadrupedal locomotion used an analytic actuator model Tan et al. (2018) for a
Miniature quadruped. Later Hwangbo et al. (2019) used a fully black-box model
(neural network) to learn an actuator model, which was then used in the
simulation to train locomotion policies.

Another important source of the sim-to-real gap is the discrepancy between the
contact model in simulation and the real-world interactions. Most of the
existing simulators use rigid contact, while in reality there always exist
some deformations in the interaction. However, it is quite challenging to
simulate deformable terrains or those with complicated shapes, which often
involves prohibitively expensive finite-element methods. One notable success
in using a more complicated contact model is Choi et al. (2023), which
develops a deformable terrain model to simulate diverse terrains, ranging from
very soft beach sand to hard asphalt. Training with this model boosts the
performance of the policy handling similar terrains in the real world compared
to training with a rigid terrain model.

###  5.3 Domain Randomization

In addition to better system design and more accurate simulation, another
important strategy to mitigate sim-to-real gap is to improve the
generalization capabilities of the trained policies. One important approach to
improve policy generalization is domain randomization, where the system
parameters are randomized during training. This is similar to robust control,
where a controller is optimized to perform robustly under a bounded set of
system parameters and uncertainties Dorato (1987). A fundamental assumption
behind domain randomizatioin is that if we generate control policies that can
handle sufficiently diverse training environments, it is more likely that this
policy also work in the real world, even though it has never seen it during
training. Another way to interpret this approach is that if we have a control
policy that works for an ensemble of different environments, the real world
will hopefully fall into one of these environments, hence the policy will work
in the real world.

Early works in using domain randomization for robot learning applied the idea
to the problem of manipulation to enable more robust perception modules Tobin
et al. (2017). In the legged locomotion field, Tan et al. (2018) were amongst
the first to adopt the idea of domain randomization. Specifically, they
randomized key dynamic parameters that are important for obtaining robust
policies such as robot mass, friction coefficient, motor strength, and
latency. This approach has since been widely adopted in follow-up works and
combined with newly developed techniques such as feed-forward trajectories
Iscen et al. (2018); Lee et al. (2020); Miki et al. (2022a), motion imitation
Peng et al. (2020), task-space control Bellegarda et al. (2022), and policy
distillation Caluwaerts et al. (2023); Kumar et al. (2021) to further improve
legged locomotion learning in different scenarios.

The aforementioned approach in domain randomization largely focused on
bridging the physics gap between the simulation and the real world, where the
resulting policies are often blind to their environments. As these techniques
mature, recent works in legged locomotion explored randomizing also the
simulated visual perception parameters such as camera intrinsics and
extrinsics, and noise models to achieve reliable vision-based legged
locomotion on highly unstructured terrains Yu et al. (2021); Miki et al.
(2022a); Margolis and Agrawal (2023); Zhuang et al. (2023); Agarwal et al.
(2023); Cheng et al. (2023c). Notably, Truong et al. (2023) reported that
higher visual fidelity does not always lead to better performance due to slow
simulation speed.

###  5.4 Domain Adaptation

Like domain randomization, domain adaptation aims to develop generalizable
policies that can cover real-world environments, with the main difference
being that domain adaptation explicitly identifies the current scenario that
the policy is operating within and adjusts the policy behavior to be optimal
for the current scenario (similar to adaptive control). As a result, when
successfully trained, domain adaptation-based policies can often achieve
better performance and cover a wider range of scenarios than domain
randomization techniques. Figure 6 demonstrates common components in domain
adaptation algorithms.

One class of domain adaptation algorithms is to explicitly identify the
parameters of the environment, which are used as input to the control policy
or during policy training Yu et al. (2017); Chebotar et al. (2019); Yu et al.
(2019); Muratore et al. (2021); Lee et al. (2018a). To identify the
environment parameters, Yu et al. (2017) learned models, Chebotar et al.
(2019) optimized trajectory matching loss, and Yu et al. (2019) directly
optimized task performance. Although explicitly identifying the environmental
parameters enables an intuitive interpretation of the adaptation process, it
is difficult to extend this idea to cases where a moderate number of
environment parameters need to be adapted.

To mitigate this issue, researchers have opted for implicitly representing the
environment parameters during the adaptation Yu et al. (2020); Kumar et al.
(2021); Peng et al. (2020); Lee et al. (2020), where the high-dimensional
environment parameters are compressed into a low-dimensional latent
representation. There is a variety of approaches for obtaining a good latent
representation. For example, Peng et al. (2020) propose an optimization-based
approach to find the latent environment representation for an estimated
advantage value during policy adaptation. Kumar et al. (2021) and Lee et al.
(2020) on the other hand, directly learn the latent environment representation
during policy training, and later on train a separate system identification
module to predict the latent representation during inference time. Lee et al.
(2022) leverage ideas from the representation learning field and used
predictive information to acquire a state representation that captures key
environment dynamic information. Due to the ability to handle larger amounts
of environment variations, this class of domain adaptation approaches has been
more widely applied to real-legged systems and has become one of the most
popular classes of methods in recent years.

Figure 6: Common components in domain adaptation algorithms.

##  6 Combining Control and Learning

While both RL and OC aim at finding the optimal policy for a desired task,
they have complementary features that attract researchers to devise frameworks
that benefit from the best of both worlds. For instance, as model-based
approaches use the knowledge of dynamics and constraints in the policy design,
they transfer our knowledge of the world among different tasks. Furthermore,
they can explicitly take constraints into account which makes them suitable
for safety-critical cases. On the other hand, learning-based approaches can
randomize the parts of the model with uncertainties and deliver a robust
policy without much overhead in the policy design. Furthermore, the online
computation is at a minimum, as one only needs to perform inference during
execution.

This line of works can be split into four general categories; 1) learning some
parameters of model-based controllers, 2) learning high-level policies to set
commands for a model-based controller, 3) leveraging learning to improve the
generalization and reduce online computation of model-based controllers, 4)
Using model-based control to make learning more sample-efficient.

###  6.1 Learning Control Parameters

There are certain parameters in model-based control policies that directly
affect the performance and robustness of the controller. Normally tuning these
parameters is done manually on the real system by the control designer.
However, it is always desired to automate this process. To do that, Ponton et
al. (2014); Heijmink et al. (2017) use PI2 to learn optimal gains for tracking
a desired trajectory. Pandala et al. (2022) learns the boundaries of the
uncertainty set that is then used to design a robust MPC controller.

To efficiently learn the control parameters, some recent approaches rely on
Bayesian optimization (BO). Yeganegi et al. (2019, 2021) learn the cost
weights of the trajectory optimization model and an MPC controller for
humanoid locomotion, respectively. Marco et al. (2021) learn the joint
impedance to track trajectories from a trajectory optimization module, while
simultaneously learning the failure constraints. Yang et al. (2022a) use BO to
learn the control parameters of a hybrid-zero-dynamics-based controller. As BO
can only handle a small number of decision variables, Yuan et al. (2019);
Sarmadi et al. (2023) first perform dimensionality reduction and then apply BO
to the low-dimensional latent space. To make the exploration in the real world
safe, Widmer et al. (2023) apply a safe BO framework to tune both the gait and
feedback control parameters in the real world.

###  6.2 Learning a High-level policy

A more sophisticated version of simply learning a few parameters of a model-
based controller is to learn a policy that works in concert with a model-based
controller. For instance, finding foothold locations for locomotion, given a
general map of the environment, is a highly challenging task for model-based
controllers. Or model-based controllers are sensitive to the precision of
their underlying model, hence a residual policy can correct for this model
discrepancy.

Villarreal et al. (2020); Byun et al. (2021); Xu et al. (2022); Omar et al.
(2023) learn a contact planner that, given the terrain map, finds safe
foothold locations and passes them to an MPC module to optimize ground
reaction forces. Gangapurwala et al. (2022) train an RL policy that generates
desired foot locations which are tracked using a model-based whole-body
controller. Taouil et al. (2023) develop a footstep location planner for a
given low-level controller that is conditioned on a desired velocity. Ravi et
al. (2024) leverage the power of diffusion models in handling multi-modal
datasets to learn in a supervised fashion a policy that outputs contact plans
for MPC. Gangapurwala et al. (2021) leverage DRL to learn a feedback
trajectory corrector to update the output of trajectory optimization that is
then stabilized using a whole-body controller. Yang et al. (2022c) learn a
high-level policy to transition between different gaits. Viereck and Righetti
(2021); Xie et al. (2022) learn a centroidal trajectory planner that generates
desired forces and body trajectories that are fed to an inverse dynamics
controller for tracking.

###  6.3 Learning for Efficient Model-based Control

While considering the model in the structure of the policy can result in a
better generalization, it comes with the cost of more online computational
burden. For instance, a model-based controller needs to plan online for a
walking motion in real-time. However, it seems much more efficient to simply
cache the solutions in an offline phase, e.g., using a function approximator,
to reduce the online computation. In this vein, Carius et al. (2020); Reske et
al. (2021) learn the control Hamiltonian function, whereas Wang et al. (2022);
Viereck et al. (2022); Kovalev et al. (2023) learn the value function of the
optimal control problem. Lembono et al. (2020); Dantec et al. (2021) use a
large dataset from trajectory optimization to learn warm-starts for quick
resolutions of the nonlinear program in the MPC. Yang et al. (2020); Bechtle
et al. (2021); Anne et al. (2021); Li et al. (2021b); Wu et al. (2023b) learn
a forward model that is then used within a model-based module to control the
robot. Kwon et al. (2020) learn the solution map of the trajectory
optimization problem to enable reactive planning in real-time.

###  6.4 Model-based Control for Efficient Learning

The fourth category leverages tools from model-based control to efficiently
learn locomotion skills. For instance, one can use trajectory optimization to
guide the RL through high-reward parts of state space. The final policy in
these approaches is a neural network policy that sends actions to the joins.

Within this line of research, Gangapurwala et al. (2020) use optimal control
to guide and constrain the exploration of the DRL algorithm. Tsounis et al.
(2020) leverage trajectory optimization to evaluate gait transitions in the
high-level controller in a hierarchical DRL setting. Green et al. (2021)
leverage the power of reduced-order models for planning and learn a residual
policy to compensate for the discrepancy between the reduced and full robot
models. Bellegarda et al. (2020); Bogdanovic et al. (2022); Brakel et al.
(2022); Kang et al. (2023) use optimal control to generate demonstrations to
guide DRL. Similarly, Jenelten et al. (2024) interactively queries a
trajectory optimization module to provide demonstrations for the DRL policy in
different initial and final configurations. Fuchioka et al. (2023) investigate
the use of feed-forward inputs from trajectory optimization to improve the RL
learning efficiency and sim-to-real transfer. Khadiv et al. (2023) take a
behavioral cloning approach to learn optimal policy from MPC directly in
sensor space. Youm et al. (2023) use a similar approach, but improves the
cloned policy using another DRL phase like Bogdanovic et al. (2022).

##  7 From Quadrupeds to Bipeds

Works using RL for bipedal locomotion dates back to as early as 2005, when
policy gradient Tedrake et al. (2005) was used to train a small biped to walk
in under 20 minutes. Schuitema et al. (2010) use RL to train a 2D biped to
walk in 2010. Hester et al. (2010) applied model-based RL to learn soccer
penalty kicks for a NAO humanoid. These early works focused on learning
directly on the physical robots and therefore were only applied to simplified
biped, e.g., a small biped with large feet or a biped with motion restricted
to a 2D plane.

From early 2000 to the DARPA Robotics Challenge (2015) (DRC) in 2015 , the
dominant approach to control humanoid robots was OC or heuristic methods with
simplified dynamics models. Early works on the use of OC for humanoids
resorted to the simplified linear models of the dynamics to enable linear MPC
Kajita et al. (2003); Wieber (2006). However, for multi-contact behaviors
(making contact with the environment using both feet and arm), later works
focused on solving the holistic problem using the robot full dynamics with
contact Tassa et al. (2012); Mordatch et al. (2012); Lengagne et al. (2013);
Posa et al. (2014). These holistic approaches showed little success through a
model predictive control fashion in real-world humanoid control Koenemann et
al. (2015). In DRC 2015, most teams used a combination of simplified models
planning together with full-body inverse dynamics/kinematics Feng et al.
(2015); Kuindersma et al. (2016); Johnson et al. (2017). However, this contest
demonstrated that humanoid technology is by no measure close to be deployed
for real-world problems and a paradigm-shift is required Murphy (2015);
Atkeson et al. (2018).

As DRL achieved success in physics-based character animation Peng et al.
(2018) and quadrupedal locomotion Tan et al. (2018); Hwangbo et al. (2019),
researchers started to use DRL algorithms and physics simulation to train
walking policies for the bipedal robot Cassie and successfully transferred
them to the hardware Xie et al. (2020a). Techniques such as domain
randomization, more modern neural network architectures, and smarter reward
design enabled dynamic and versatile behaviors that are robust to real-world
perturbation, e.g., Siekmann et al. (2021); Radosavovic et al. (2024); Li et
al. (2024). More recent works explored how to incorporate perception to enable
challenging terrain navigation, such as stepping stones Duan et al. (2022) and
uneven terrains Duan et al. (2023). In addition, more expressive learning-
based motions are recently shown on humanoid robotsCheng et al. (2024).

In the years 2023 and 2024, we witness an explosion of new companies starting
to develop humanoid robots, most of which have bipedal forms. The highlight of
the exhibition section of the 2024 International Conference in Robotics and
Automation (ICRA) was an immense increase in the number of electrically-
actuated humanoids. In particular, Unitree has unveiled its new humanoid G1
with an incredibly low price, on par with manipulator arms. This marks as the
starting point of a new era where many research laboratories will have access
to humanoid robots. We have seen this trend in the past decade that enabled
exponential progress in the fields of drones and quadrupedal locomotion.

Parallel to the humanoid efforts in industry, researches have developed open-
source Daneshmand et al. (2021) and affordable Liu et al. (2022a) miniature
bipedal platforms. These miniature robots are much easier to handle than full
humanoid robots and could potentially impact the rate of progress of research
on learning-based bipedal locomotion. For instance, Haarnoja et al. (2024)
develop a hierarchical system to enable learning of soccer plays on the
miniature Robotis OP3 humanoid. Recently, the miniature robot from LimX has
shown impressively robust bipedal locomotion behaviors in the wild Dynamics
(2024). With affordable human-size humanoids becoming widely available, one
would expect lessons learned from quadrupedal learning works to be transferred
to humanoid robots, while also additional techniques are to be developed to
handle the difficulties with balancing bipeds with respect to quadrupeds.

##  8 Unsolved Problems and Research Frontiers

The significant progress enabled by learning-based control opens the door to
new challenges. These include autonomous skill discovery, traversing ever-more
challenging environments, the use of differentiable simulators, integration of
locomotion with other goals such as manipulation, and controlling hybrid
wheeled-legged locomotion. The need to understand and mitigate safety issues
becomes ever more important with wider deployment of legged robots. New
learning pipelines enabled by foundation models might be leveraged for use in
robot control, as well as large language models to help with both natural
language task specification as well as planning. We now outline these
directions.

###  8.1 Unsupervised Skill Discovery

DRL often requires labor-intensive reward engineering of domain experts to
obtain desirable behaviors. Instead, unsupervised skill discovery can
potentially reduce the burden of repetitive reward engineering by learning a
set of useful skills based on intrinsic motivation. Typically, unsupervised
skill discovery aims to learn a skill-conditioned policy that executes diverse
and discernable skills, as discussed in Eysenbach et al. (2018).

Researchers have applied this idea of unsupervised skill discovery to learn
diverse locomotion skills Sharma et al. (2020); Cheng et al. (2023a). For
example, Schwarke et al. (2023) successfully used this approach to learn loco-
manipulation skills for a real quadrupedal robot. Once the skills are
obtained, they can be leveraged for downstream tasks via model-predictive
control Sharma et al. (2020) or hierarchical RL Eysenbach et al. (2018). A
similar idea has been also applied to learn skills from unstructured motions
from expert policies Li et al. (2023a) or even from human motion clips Li et
al. (2023c).

###  8.2 Traversing Challenging Environments

After the success of DRL in generating highly robust behaviors Hwangbo et al.
(2019); Lee et al. (2020); Bogdanovic et al. (2022), recently there has been
an increasing interest in learning locomotion skills in challenging
environments, e.g., performing parkour Zhuang et al. (2023); Cheng et al.
(2023c); Hoeller et al. (2023), traversing stepping stones Duan et al. (2022);
Zhang et al. (2023); Ravi et al. (2024), moving in confined environments Xu et
al. (2024); Chane-Sane et al. (2024) (see Fig. 1 for a few examples). This is
an interesting direction that can greatly benefit from the power of deep
learning to include perception in the control loop. While it has been argued
that DRL-based controllers may struggle with traversing highly constrained
environments Grandia et al. (2023), recent efforts have shown the opposite
Zhang et al. (2023). However, it remains an interesting question to be
answered; to what extent DRL approaches can handle safety-critical situations?

###  8.3 Differentiable Simulators

To improve the sample efficiency of learning-based approaches for legged
locomotion, there has been an increasing interest in the use of differentiable
simulators Schwarke et al. (2024). The main idea in these approaches is to
better exploit the structure available in the simulation environments, for
instance through the use of gradients obtained either analytically Werling et
al. (2021); Howell et al. (2022) or using automatic differentiation Degrave et
al. (2019); Hu et al. (2019); Freeman et al. (2021) or even finite difference
Todorov et al. (2012); MJX (2023). However, the main problem with this
approach is poor local minima due to contact. One interesting approach to
overcome this issue is the use of randomized smoothing techniques Suh et al.
(2022); Lidec et al. (2022); Pang et al. (2023). As more and more
differentiable simulators become available, this under-explored area in the
field can attract more researchers.

###  8.4 Loco-manipulation

When manipulating objects by a legged robot, it becomes essential to take the
object affordances, dynamics, and constraints into account in the planning and
control phases. Given the affordances of the objects and the environment
Gibson (1977), the robot needs to simultaneously decide on both locomotion and
manipulation aspects. There have been recent efforts on the use of DRL for
loco-manipulation tasks in the real world. In general, there are two ways to
think about loco-manipulation using quadrupeds. The first one is to use the
robot’s body and feet to perform some pushing and pressing tasks. The second
way is to add an arm to the robot and use the gripper to grasp objects and
perform more sophisticated loco-manipulation behaviors.

Within the first category, Ji et al. (2022b); Huang et al. (2023); Jeon et al.
(2023); Arm et al. (2024) propose a hierarchical framework to use the robot
body and feet to generate interesting loco-manipulation behaviors. Kim et al.
(2022) use teleoperation of human motion and employs DRL to re-target the
motions to realizable quadrupedal loco-manipulation behaviors. Cheng et al.
(2023b) employ curriculum learning to learn different skills in simulation and
then build a behavior tree to sequence the skills.

Using only feet and body for manipulation is limited to simple loco-
manipulation skills, as the robot is not able to grip/grasp
objects/environments. To perform more sophisticated object manipulation, it is
now common to mount a manipulator on quadrupeds. To perform loco-manipulation
using quadruped with an arm, Fu et al. (2023) use a privileged learning
framework to train a policy that simultaneously deals with locomotion and
manipulation. Liu et al. (2024) develops a hierarchical framework, using a
high-level policy that takes visual information as input and provides end-
effector commands for a learned low-level controller. However, these
approaches are limited to simple manipulation tasks with a quadruped and
cannot reason about the complex loco-manipulation tasks. Recently, Kumar et
al. (2023) proposed to use a library of loco-manipulation skill primitives and
to compose them and generate a long sequence of loco-manipulation system.
However, the manipulation part of the framework is still basic (such as
opening a door). Using learning-based algorithms to perform a long sequence of
complex loco-manipulation tasks is an interesting future frontier.

###  8.5 Hybrid Wheeled-legged Locomotion

A wheeled-legged robot is a variation of legged robots designed to enhance
efficiency and endurance. Unlike traditional point-foot legged robots, the
actuated wheels allow for an additional mode of locomotion, enabling them to
drive without stepping in the longitudinal direction.

Achieving both efficiency and stability requires a locomotion controller
capable of transitioning between walking and driving locomotion modes. In
conventional model-based approaches, this mode switching often relies on
operator commands or heuristics Bjelonic et al. (2019, 2021). However, gait
generation for wheeled-legged robots is not straightforward, because there is
no such systems in the nature. This makes heuristic approaches with periodic
patterns such as CPG impractical.

While this research field is till very young, recent studies have explored
training locomotion policies in an end-to-end manner using DRL. Lee et al. Lee
et al. (2024) applied an approach similar to Miki et al. (2022a) without CPGs
to a wheeled-legged system, and demonstrated adaptive gait transitions and
locomotion over rough terrains. Similarly, Chamorro et al. (2024) and Cui et
al. (2021) demonstrated wheeled-legged bipeds traversing uneven terrains,
albeit without gait transitions. This combined modality has a lot to offer and
it seems to be an interesting direction for more research.

###  8.6 Safety

With continuous advancements in robot locomotion, they will start to be
deployed in more diverse locations such as home, office, plants, or in the
wild. As such, safety becomes a critical topic as we want to make sure the
robots do not damage themselves during deployment nor do they damage the
environment or any human being nearby. One possible way to tackle safety in
robot locomotion is to formulate it as a constrained optimization problem.
Indeed, researchers in safe reinforcement learning have developed a variety of
safe RL algorithms that aims to train control policies without violating
safety constraints Thananjeyan et al. (2021); Hsu et al. (2023); Achiam et al.
(2017); Liu et al. (2022b), where they focus on theoretical properties of the
algorithms and are applied to synthetic tasks. More recently, researchers have
started to apply these methods to legged robots and demonstrated safe learning
in the real-world Yang et al. (2022b). Constrained policy optimization
approaches have also been used to achieve safer and more precise behaviors for
locomotion in complex environments Kim et al. (2023c); Lee et al. (2023); Xu
et al. (2024). It is anticipated that this direction of research will continue
to thrive with the more advanced legged robot capabilities and hardware.

Another way to ensure safety, specifically in deployment, is to use safety
filters, which have been extensively studied in the control community
Wabersich et al. (2023). These filters can be designed using Hamilton-jacobi
(HJ) reachability, control barrier functions (CBF), and predictive methods for
uncertain systems. Variations of algorithms with and without access to the
robot dynamics have been developed. However, their application to locomotion
has been limited Kim et al. (2023a); He et al. (2024). More research on this
under-explored area in the future can facilitate certification of this
technology for the real-world safety-critical applications.

###  8.7 Foundation Models

Learning locomotion skills and empowering robots with mobility is a key
milestone towards building general-purpose robots that are truly useful.
However, a notable gap still exists in terms of enabling robots to generalize
their behaviors to arbitrary objects, environments, and interactions with
humans. The recent progress in training large-capacity pre-trained models with
web-scale data (foundation models) has demonstrated impressive common-sense
reasoning, learning, and perception capabilities without specialized training
Achiam et al. (2023); Touvron et al. (2023); Team et al. (2023). Such
capabilities are critical for creating general-purpose robots and have thus
led to a plethora of works that combines robotics and foundation models in
planning, reasoning, navigation, and simulation of robotic systems Kira
(2022). We refer readers to several existing survey papers summarizing these
efforts Hu et al. (2023); Zhou et al. (2023); Firoozi et al. (2023); Xiao et
al. (2023) and focus on discussing ones applied to robot locomotion here.

A common approach in combining foundation models with legged robots applies it
for high-level planning Lykov et al. (2024); Xu et al. (2023). In these
methods, a foundation model is used to interpret the environment and task, and
plans a sequence of skills that the robot should perform to achieve a longer
horizon task. However, these methods usually do not have fine control of the
robots’ low-level behavior such as locomotion gaits. This is due to the fact
that pre-trained foundation models are usually not trained with robotic data
and cannot directly generate low-level robot actions. To leverage foundation
models to also work with low-level locomotion control, researchers have
designed different interface including foot contact patterns Tang et al.
(2023), and reward functions Yu et al. (2023a); Liang et al. (2024); Ma et al.
(2024) that allows foundation models to directly interact with robot’s low-
level controllers to perform tasks such as hopping happily, or high-five with
a person. Alternatively, one may also fine-tune a large foundation model to
directly output the low-level robot action using large amount of robot data.
This has been explored in the manipulation domain and shown promising
generalization capabilities to novel tasks Brohan et al. (2023). Applying this
idea to robot locomotion is a promising idea to obtain generalist robot
locomotion controllers.

##  9 Societal Impact

Ever-more capable legged robots brings significant potential for both good and
bad. This includes the potential to be deployed in many humanitarian
applications, tackling dull-and-repetitive tasks in warehouse and agriculture
operations, factory inspection, and performing jobs that are dangerous for
humans, e.g., search and rescue and firefighting. At the same time, these
robots have the potential to be used in multiple destructive applications. We
believe that our community should play an active role in the discussions
related to the regulations at this stage of development toward general-purpose
robots (which are mostly legged systems) and remain alerted about the
potential risks of this technology.

Here, we aim to only list some immediate issues related to the use of general-
purpose robots in the real world. While being preliminary, we believe
initiatives from the robotics research community into these topics can have
several positive impacts. In particular, we urge the audience of this paper to
read the excellent article on the ethics of artificial intelligence and
robotics Müller (2020). As well-said in the article: _there is a tendency for
businesses, the military, and some public administrations to “just talk” and
do some “ethics washing” in order to preserve a good public image and continue
as before. Actual policy is not just an implementation of ethical theory, but
subject to societal power structures—and the agents that do have the power
will push against anything that restricts them. There is thus a significant
risk that regulation will remain toothless in the face of economical and
political power._ Within such a power structure, academia and researchers can
constitute a powerful entity that raises awareness about the new technologies
and their implications, and makes sure that the right policies are
implemented.

One dangerous application is to use general-purpose robots as lethal
autonomous weapon systems Righetti et al. (2018). Unfortunately, we have seen
such activities towards weaponizing quadrupeds in recent years. Such systems
would raise multiple important ethical and societal issues. From an ethical
point of view, this question arises: should an autonomous system that _does
not suffer_ or _has no feelings_ itself have the right to decide to kill a
person? on the legal side, who would be responsible (and punished) for the
potential mistakes of an autonomous weapon system? Also, from the human rights
perspective, autonomous weapons would not be aligned with international
humanitarian law. Human Rights Watch and Harvard Law School’s International
Human Rights Clinic (IHRC) notes Docherty (2012): _… such revolutionary
weapons would not be consistent with international humanitarian law and would
increase the risk of death or injury to civilians during armed conflict. A
preemptive prohibition on their development and use is needed._. A recent open
letter letter (2022) signed by multiple companies (Agility Robotics,
ANYbotics, Clearpath Robotics, Open Robotics, Unitree, Boston Dynamics) aligns
with this view and condemns any use of weaponized mobile robots. Activities of
this kind, together with practical measures, can help to safe guard the
technology of legged robots in the future.

Another impact of developing general-purpose robots in the near future is the
possible human jobs lost due to new automation abilities Pham et al. (2018).
Replacing humans in health-threatening jobs, e.g., firefighting, search and
rescue in the wild, dangerous industrial work, and care-giving in pandemics
Shen et al. (2020), can be of significant benefit. However, the problem arises
with the potential to replace human workers by robots for many other job
categories, and the impact on the lives of displaced workers. An alternative
that is regularly advocated is to use robots to take over the physical work
while humans use their cognitive capabilities. Human-robot collaboration is
another scenario that is advocated over replacing humans. While human history
has seen significant shifts due to mechanization before, as with agriculture,
for example, past outcomes and adaptations may be a poor predictor of future
outcomes on these issues. A recent study spanning different cultures and
industries suggests that increased exposure to robots can lead to increased
job insecurity Yam et al. (2023). With rapidly advancing technology, it will
be important for policy makers to consider the impact of robotics on the
society and economy of the future.

Environmental concerns currently see minimal discussion in the robotics
community. As deep learning is increasingly used with models of increasing
size, it is important to discuss the carbon footprint of training and using
these massive models. This issue is already discussed in the machine learning
community, and it is important that the robotics community also thinks about
how to mitigate the rapid and unsustainable growth of compute requirements.
Another more nuanced environmental issue is that of the life-cycle of the
robot itself: what happens at the end of a robot’s operational life? Currently
legged robots and humanoids are not contributing significantly in this area
and the question is more immediate concern for industrial robots and
autonomous cars. However, the adoption of quadrupeds and humanoids on a larger
scale means that we may eventually see large numbers of these robots that
require disposal or recycling when they are no longer functional. It is
therefore incumbent on us to think about this issue sooner rather than later.

We have only touched on only some of the immediate risks related to general-
purpose robotic systems. This list is by no means complete. It does not take
other aspects of the problem into account that are related to the software of
an embodied AI, including enabling unwanted surveillance, biases in decision
making, possible amplification of existing inequities, and more.

##  10 Conclusions

Learning-based methods are at the heart of so many of the advances seen in
legged locomotion control. In this article, we summarized the core issues and
methods for quadrupedal locomotion learning, as well as touching on recent
related methods for bipedal locomotion. While it is impossible to be fully
comprehensive, we hope that this survey provides a valuable framework for
understanding recent progress along with ample points of reference. We have
further outlined unsolved problems and future directions that, because of
their difficulty and broad nature, will provide significant challenges for
years to come. However, it is reasonable to expect the rapid pace of
innovation to continue, yielding ever-more capable legged locomotion for
quadrupeds and bipeds.

## References

  * Abate [2018] A. M. Abate.  Mechanical design for robot locomotion.  _Graduate Thesis_ , 2018.
  * Abdolmaleki et al. [2018] A. Abdolmaleki, J. T. Springenberg, Y. Tassa, R. Munos, N. Heess, and M. Riedmiller.  Maximum a posteriori policy optimisation.  _arXiv preprint arXiv:1806.06920_ , 2018.
  * Aceituno-Cabezas et al. [2017] B. Aceituno-Cabezas, C. Mastalli, H. Dai, M. Focchi, A. Radulescu, D. G. Caldwell, J. Cappelletto, J. C. Grieco, G. Fernández-López, and C. Semini.  Simultaneous contact, gait, and motion planning for robust multilegged locomotion via mixed-integer convex optimization.  _IEEE Robotics and Automation Letters_ , 3(3):2531–2538, 2017.
  * Achiam et al. [2017] J. Achiam, D. Held, A. Tamar, and P. Abbeel.  Constrained policy optimization.  In _International conference on machine learning_ , pages 22–31. PMLR, 2017.
  * Achiam et al. [2023] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Agarwal et al. [2023] A. Agarwal, A. Kumar, J. Malik, and D. Pathak.  Legged locomotion in challenging terrains using egocentric vision.  In _Conference on Robot Learning_ , pages 403–415. PMLR, 2023.
  * Ahn et al. [2019] J. Ahn, D. Kim, S. Bang, N. Paine, and L. Sentis.  Control of a high performance bipedal robot using viscoelastic liquid cooled actuators.  In _2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids)_ , pages 146–153. IEEE, 2019.
  * Akkaya et al. [2019] I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al.  Solving rubik’s cube with a robot hand.  _arXiv preprint arXiv:1910.07113_ , 2019.
  * Anne et al. [2021] T. Anne, J. Wilkinson, and Z. Li.  Meta-learning for fast adaptive locomotion with uncertainties in environments and robot dynamics.  In _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 4568–4575. IEEE, 2021.
  * Arm et al. [2024] P. Arm, M. Mittal, H. Kolvenbach, and M. Hutter.  Pedipulate: Enabling manipulation skills using a quadruped robot’s leg.  _arXiv preprint arXiv:2402.10837_ , 2024.
  * Atkeson et al. [2018] C. G. Atkeson, P. B. Benzun, N. Banerjee, D. Berenson, C. P. Bove, X. Cui, M. DeDonato, R. Du, S. Feng, P. Franklin, et al.  What happened at the darpa robotics challenge finals.  _The DARPA robotics challenge finals: Humanoid robots to the rescue_ , pages 667–684, 2018.
  * Bao et al. [2024] L. Bao, J. Humphreys, T. Peng, and C. Zhou.  Deep reinforcement learning for bipedal locomotion: A brief survey.  _arXiv preprint arXiv:2401.16889_ , 2024.
  * Barasuol et al. [2013] V. Barasuol, J. Buchli, C. Semini, M. Frigerio, E. R. De Pieri, and D. G. Caldwell.  A reactive controller framework for quadrupedal locomotion on challenging terrain.  In _2013 IEEE International Conference on Robotics and Automation_ , pages 2554–2561. IEEE, 2013.
  * Barth-Maron et al. [2018] G. Barth-Maron, M. W. Hoffman, D. Budden, W. Dabney, D. Horgan, D. Tb, A. Muldal, N. Heess, and T. Lillicrap.  Distributed distributional deterministic policy gradients.  _arXiv preprint arXiv:1804.08617_ , 2018.
  * BDI [2015] BDI.  Spot specifications, 2015.  URL https://support.bostondynamics.com/s/article/Robot-specifications.
  * Bechtle et al. [2021] S. Bechtle, B. Hammoud, A. Rai, F. Meier, and L. Righetti.  Leveraging forward model prediction error for learning control.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 4445–4451. IEEE, 2021.
  * Bellegarda and Ijspeert [2022] G. Bellegarda and A. Ijspeert.  Cpg-rl: Learning central pattern generators for quadruped locomotion.  _IEEE Robotics and Automation Letters_ , 7(4):12547–12554, 2022.
  * Bellegarda et al. [2020] G. Bellegarda, C. Nguyen, and Q. Nguyen.  Robust quadruped jumping via deep reinforcement learning.  _arXiv preprint arXiv:2011.07089_ , 2020.
  * Bellegarda et al. [2022] G. Bellegarda, Y. Chen, Z. Liu, and Q. Nguyen.  Robust high-speed running for quadruped robots via deep reinforcement learning.  In _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 10364–10370. IEEE, 2022.
  * Bellman [1966] R. Bellman.  Dynamic programming.  _science_ , 153(3731):34–37, 1966.
  * Besselmann et al. [2021] M. G. Besselmann, L. Puck, L. Steffen, A. Roennau, and R. Dillmann.  Vdb-mapping: a high resolution and real-time capable 3d mapping framework for versatile mobile robots.  In _2021 IEEE 17th International Conference on Automation Science and Engineering (CASE)_ , pages 448–454. IEEE, 2021.
  * Bjelonic et al. [2019] M. Bjelonic, C. D. Bellicoso, Y. de Viragh, D. Sako, F. D. Tresoldi, F. Jenelten, and M. Hutter.  Keep rollin’—whole-body motion control and planning for wheeled quadrupedal robots.  _IEEE Robotics and Automation Letters_ , 4(2):2116–2123, 2019.
  * Bjelonic et al. [2021] M. Bjelonic, R. Grandia, O. Harley, C. Galliard, S. Zimmermann, and M. Hutter.  Whole-body mpc and online gait sequence generation for wheeled-legged robots.  In _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 8388–8395. IEEE, 2021.
  * Bledt and Kim [2020] G. Bledt and S. Kim.  Extracting legged locomotion heuristics with regularized predictive control.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 406–412. IEEE, 2020.
  * Blickhan [1989] R. Blickhan.  The spring-mass model for running and hopping.  _Journal of biomechanics_ , 22(11-12):1217–1227, 1989.
  * Bogdanovic et al. [2020] M. Bogdanovic, M. Khadiv, and L. Righetti.  Learning variable impedance control for contact sensitive tasks.  _IEEE Robotics and Automation Letters_ , 5(4):6129–6136, 2020.
  * Bogdanovic et al. [2022] M. Bogdanovic, M. Khadiv, and L. Righetti.  Model-free reinforcement learning for robust locomotion using demonstrations from trajectory optimization.  _Frontiers in Robotics and AI_ , 9, 2022.
  * Boyan and Moore [1994] J. Boyan and A. Moore.  Generalization in reinforcement learning: Safely approximating the value function.  _Advances in neural information processing systems_ , 7, 1994.
  * Brakel et al. [2022] P. Brakel, S. Bohez, L. Hasenclever, N. Heess, and K. Bousmalis.  Learning coordinated terrain-adaptive locomotion by imitating a centroidal dynamics planner.  In _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 10335–10342. IEEE, 2022.
  * Brohan et al. [2023] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al.  Rt-2: Vision-language-action models transfer web knowledge to robotic control.  _arXiv preprint arXiv:2307.15818_ , 2023.
  * Buchli and Ijspeert [2008] J. Buchli and A. J. Ijspeert.  Self-organized adaptive legged locomotion in a compliant quadruped robot.  _Autonomous Robots_ , 25:331–347, 2008.
  * Buchli et al. [2006] J. Buchli, F. Iida, and A. J. Ijspeert.  Finding resonance: Adaptive frequency oscillators for dynamic legged locomotion.  In _2006 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 3903–3909. IEEE, 2006.
  * Buchli et al. [2009] J. Buchli, M. Kalakrishnan, M. Mistry, P. Pastor, and S. Schaal.  Compliant quadruped locomotion over rough terrain.  In _2009 IEEE/RSJ international conference on Intelligent robots and systems_ , pages 814–820. IEEE, 2009.
  * Buckman et al. [2018] J. Buckman, D. Hafner, G. Tucker, E. Brevdo, and H. Lee.  Sample-efficient reinforcement learning with stochastic ensemble value expansion.  _Advances in neural information processing systems_ , 31, 2018.
  * Buehler et al. [1998] M. Buehler, R. Battaglia, A. Cocosco, G. Hawker, J. Sarkis, and K. Yamazaki.  Scout: A simple quadruped that walks, climbs, and runs.  In _Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No. 98CH36146)_ , volume 2, pages 1707–1712. IEEE, 1998.
  * Buehler et al. [1999] M. Buehler, A. Cocosco, K. Yamazaki, and R. Battaglia.  Stable open loop walking in quadruped robots with stick legs.  In _Proceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No. 99CH36288C)_ , volume 3, pages 2348–2353. IEEE, 1999\.
  * Byun et al. [2021] J. W. Byun, D. Youm, S. Jeon, J. Hwangbo, and H.-W. Park.  Learning footstep planning for the quadrupedal locomotion with model predictive control.  In _International Conference on Robot Intelligence Technology and Applications_ , pages 35–43. Springer, 2021.
  * Caluwaerts et al. [2023] K. Caluwaerts, A. Iscen, J. C. Kew, W. Yu, T. Zhang, D. Freeman, K.-H. Lee, L. Lee, S. Saliceti, V. Zhuang, et al.  Barkour: Benchmarking animal-level agility with quadruped robots.  _arXiv preprint arXiv:2305.14654_ , 2023.
  * Carius et al. [2020] J. Carius, F. Farshidian, and M. Hutter.  Mpc-net: A first principles guided policy search.  _IEEE Robotics and Automation Letters_ , 5(2):2897–2904, 2020.
  * Carpentier et al. [2019] J. Carpentier, G. Saurel, G. Buondonno, J. Mirabel, F. Lamiraux, O. Stasse, and N. Mansard.  The pinocchio c++ library: A fast and flexible implementation of rigid body dynamics algorithms and their analytical derivatives.  In _2019 IEEE/SICE International Symposium on System Integration (SII)_ , pages 614–619. IEEE, 2019.
  * Chamorro et al. [2024] S. Chamorro, V. Klemm, M. d. l. I. Valls, C. Pal, and R. Siegwart.  Reinforcement learning for blind stair climbing with legged and wheeled-legged robots.  _arXiv preprint arXiv:2402.06143_ , 2024.
  * Chane-Sane et al. [2024] E. Chane-Sane, P.-A. Leziart, T. Flayols, O. Stasse, P. Souères, and N. Mansard.  Cat: Constraints as terminations for legged locomotion reinforcement learning.  _arXiv preprint arXiv:2403.18765_ , 2024.
  * Chebotar et al. [2019] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, N. Ratliff, and D. Fox.  Closing the sim-to-real loop: Adapting simulation randomization with real world experience.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pages 8973–8979. IEEE, 2019.
  * Chen et al. [2020] D. Chen, B. Zhou, V. Koltun, and P. Krähenbühl.  Learning by cheating.  In _Conference on Robot Learning_ , pages 66–75. PMLR, 2020.
  * Chen et al. [2023] S. Chen, B. Zhang, M. W. Mueller, A. Rai, and K. Sreenath.  Learning torque control for quadrupedal locomotion.  In _2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids)_ , pages 1–8. IEEE, 2023.
  * Chen et al. [2021] X. Chen, C. Wang, Z. Zhou, and K. Ross.  Randomized ensembled double q-learning: Learning fast without a model.  _arXiv preprint arXiv:2101.05982_ , 2021.
  * Cheng et al. [2023a] J. Cheng, M. Vlastelica, P. Kolev, C. Li, and G. Martius.  Learning diverse skills for local navigation under multi-constraint optimality.  _arXiv preprint arXiv:2310.02440_ , 2023a.
  * Cheng et al. [2023b] X. Cheng, A. Kumar, and D. Pathak.  Legs as manipulator: Pushing quadrupedal agility beyond locomotion.  _arXiv preprint arXiv:2303.11330_ , 2023b.
  * Cheng et al. [2023c] X. Cheng, K. Shi, A. Agarwal, and D. Pathak.  Extreme parkour with legged robots.  _arXiv preprint arXiv:2309.14341_ , 2023c.
  * Cheng et al. [2024] X. Cheng, Y. Ji, J. Chen, R. Yang, G. Yang, and X. Wang.  Expressive whole-body control for humanoid robots.  _arXiv preprint arXiv:2402.16796_ , 2024.
  * Chignoli et al. [2021] M. Chignoli, D. Kim, E. Stanger-Jones, and S. Kim.  The mit humanoid robot: Design, motion planning, and control for acrobatic behaviors.  In _2020 IEEE-RAS 20th International Conference on Humanoid Robots (Humanoids)_ , pages 1–8. IEEE, 2021.
  * Choi and Kim [2019] S. Choi and J. Kim.  Trajectory-based probabilistic policy gradient for learning locomotion behaviors.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pages 1–7. IEEE, 2019.
  * Choi et al. [2023] S. Choi, G. Ji, J. Park, H. Kim, J. Mun, J. H. Lee, and J. Hwangbo.  Learning quadrupedal locomotion on deformable terrain.  _Science Robotics_ , 8(74):eade2256, 2023.
  * Chua et al. [2018] K. Chua, R. Calandra, R. McAllister, and S. Levine.  Deep reinforcement learning in a handful of trials using probabilistic dynamics models.  _Advances in neural information processing systems_ , 31, 2018.
  * Collins et al. [2021] J. Collins, S. Chand, A. Vanderkop, and D. Howard.  A review of physics simulators for robotic applications.  _IEEE Access_ , 9:51416–51431, 2021.
  * Coumans and Bai [2016] E. Coumans and Y. Bai.  Pybullet, a python module for physics simulation for games, robotics and machine learning, 2016.
  * Cui et al. [2021] L. Cui, S. Wang, J. Zhang, D. Zhang, J. Lai, Y. Zheng, Z. Zhang, and Z.-P. Jiang.  Learning-based balance control of wheel-legged robots.  _IEEE Robotics and Automation Letters_ , 6(4):7667–7674, 2021.
  * Cybenko [1989] G. Cybenko.  Approximation by superpositions of a sigmoidal function.  _Mathematics of control, signals and systems_ , 2(4):303–314, 1989.
  * CyberDog [2021] CyberDog.  Xiaomi launches cyberdog – an open source quadruped robot companion, 2021.  URL https://www.mi.com/global/discover/article?id=2069.
  * Daneshmand et al. [2021] E. Daneshmand, M. Khadiv, F. Grimminger, and L. Righetti.  Variable horizon mpc with swing foot dynamics for bipedal walking control.  _IEEE Robotics and Automation Letters_ , 6(2):2349–2356, 2021.
  * Dantec et al. [2021] E. Dantec, R. Budhiraja, A. Roig, T. Lembono, G. Saurel, O. Stasse, P. Fernbach, S. Tonneau, S. Vijayakumar, S. Calinon, et al.  Whole body model predictive control with a memory of motion: Experiments on a torque-controlled talos.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 8202–8208. IEEE, 2021.
  * Dao et al. [2020] J. Dao, H. Duan, K. Green, J. Hurst, and A. Fern.  Learning to walk without dynamics randomization.  In _2nd Workshop on Closing the Reality Gap in Sim2Real Transfer for Robotics_ , page 16, 2020.
  * DARPA Robotics Challenge [2015] D. DARPA Robotics Challenge.  The 2015 darpa robotics challenge finals, 2015.  URL https://www.youtube.com/watch?v=8P9geWwi9e0.
  * Darvish et al. [2023] K. Darvish, L. Penco, J. Ramos, R. Cisneros, J. Pratt, E. Yoshida, S. Ivaldi, and D. Pucci.  Teleoperation of humanoid robots: A survey.  _IEEE Transactions on Robotics_ , 39(3):1706–1727, 2023.
  * Deeprobotics [2021] Deeprobotics.  quadruped series from deep robotics, 2021.  URL https://www.deeprobotics.cn/en/index/product1.html.
  * Degrave et al. [2019] J. Degrave, M. Hermans, J. Dambre, et al.  A differentiable physics engine for deep learning in robotics.  _Frontiers in neurorobotics_ , 13:406386, 2019.
  * Deits and Tedrake [2014] R. Deits and R. Tedrake.  Footstep planning on uneven terrain with mixed-integer convex optimization.  In _2014 IEEE-RAS international conference on humanoid robots_ , pages 279–286. IEEE, 2014.
  * Di Carlo et al. [2018] J. Di Carlo, P. M. Wensing, B. Katz, G. Bledt, and S. Kim.  Dynamic locomotion in the mit cheetah 3 through convex model-predictive control.  In _2018 IEEE/RSJ international conference on intelligent robots and systems (IROS)_ , pages 1–9. IEEE, 2018.
  * Docherty [2012] B. L. Docherty.  _Losing humanity: The case against killer robots_.  Human Rights Watch, 2012.
  * Doi et al. [2006] T. Doi, R. Hodoshima, Y. Fukuda, S. Hirose, T. Okamoto, and J. Mori.  Development of quadruped walking robot titan xi for steep slopes-slope map generation and map information application.  _Journal of Robotics and mechatronics_ , 18(3):318–324, 2006.
  * Dorato [1987] P. Dorato.  A historical review of robust control.  _IEEE Control Systems Magazine_ , 7(2):44–47, 1987.
  * Drnach and Zhao [2021] L. Drnach and Y. Zhao.  Robust trajectory optimization over uncertain terrain with stochastic complementarity.  _IEEE Robotics and Automation Letters_ , 6(2):1168–1175, 2021.
  * Duan et al. [2022] H. Duan, A. Malik, M. S. Gadde, J. Dao, A. Fern, and J. Hurst.  Learning dynamic bipedal walking across stepping stones.  In _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 6746–6752. IEEE, 2022.
  * Duan et al. [2023] H. Duan, B. Pandit, M. S. Gadde, B. J. van Marum, J. Dao, C. Kim, and A. Fern.  Learning vision-based bipedal locomotion for challenging terrain.  _arXiv preprint arXiv:2309.14594_ , 2023.
  * Dynamics [2024] L. Dynamics.  Limx dynamics’ biped robot p1 conquers the wild based on reinforcement learning, 2024.  URL https://www.youtube.com/watch?v=UpNid_rWDnI.
  * Englsberger et al. [2014] J. Englsberger, A. Werner, C. Ott, B. Henze, M. A. Roa, G. Garofalo, R. Burger, A. Beyer, O. Eiberger, K. Schmid, et al.  Overview of the torque-controlled humanoid robot toro.  In _2014 IEEE-RAS International Conference on Humanoid Robots_ , pages 916–923. IEEE, 2014.
  * Escontrela et al. [2022] A. Escontrela, X. B. Peng, W. Yu, T. Zhang, A. Iscen, K. Goldberg, and P. Abbeel.  Adversarial motion priors make good substitutes for complex reward functions.  In _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 25–32. IEEE, 2022.
  * Espeholt et al. [2018] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al.  Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures.  In _International conference on machine learning_ , pages 1407–1416. PMLR, 2018.
  * Eysenbach et al. [2018] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine.  Diversity is all you need: Learning skills without a reward function.  _arXiv preprint arXiv:1802.06070_ , 2018.
  * Fankhauser et al. [2013] P. Fankhauser, M. Hutter, C. Gehring, M. Bloesch, M. A. Hoepflinger, and R. Siegwart.  Reinforcement learning of single legged locomotion.  In _2013 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 188–193. IEEE, 2013.
  * Farshidian et al. [2017] F. Farshidian, E. Jelavic, A. Satapathy, M. Giftthaler, and J. Buchli.  Real-time motion planning of legged robots: A model predictive control approach. in 2017 ieee-ras 17th international conference on humanoid robotics (humanoids), 2017.
  * Featherstone [1987] R. Featherstone.  Robot dynamics algorithms.  _The Springer International Series in Engineering and Computer Science_ , pages 1–211, 1987.
  * Feinberg et al. [2018] V. Feinberg, A. Wan, I. Stoica, M. I. Jordan, J. E. Gonzalez, and S. Levine.  Model-based value estimation for efficient model-free reinforcement learning.  _arXiv preprint arXiv:1803.00101_ , 2018.
  * Feng et al. [2015] S. Feng, E. Whitman, X. Xinjilefu, and C. G. Atkeson.  Optimization-based full body control for the darpa robotics challenge.  _Journal of field robotics_ , 32(2):293–312, 2015\.
  * Firoozi et al. [2023] R. Firoozi, J. Tucker, S. Tian, A. Majumdar, J. Sun, W. Liu, Y. Zhu, S. Song, A. Kapoor, K. Hausman, et al.  Foundation models in robotics: Applications, challenges, and the future.  _arXiv preprint arXiv:2312.07843_ , 2023.
  * Focchi et al. [2017] M. Focchi, A. Del Prete, I. Havoutis, R. Featherstone, D. G. Caldwell, and C. Semini.  High-slope terrain locomotion for torque-controlled quadruped robots.  _Autonomous Robots_ , 41:259–272, 2017.
  * Freeman et al. [2021] C. D. Freeman, E. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem.  Brax–a differentiable physics engine for large scale rigid body simulation.  _arXiv preprint arXiv:2106.13281_ , 2021.
  * Fu et al. [2023] Z. Fu, X. Cheng, and D. Pathak.  Deep whole-body control: learning a unified policy for manipulation and locomotion.  In _Conference on Robot Learning_ , pages 138–149. PMLR, 2023.
  * Fuchioka et al. [2023] Y. Fuchioka, Z. Xie, and M. Van de Panne.  Opt-mimic: Imitation of optimized trajectories for dynamic quadruped behaviors.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 5092–5098. IEEE, 2023.
  * Fujimoto et al. [2018] S. Fujimoto, H. Hoof, and D. Meger.  Addressing function approximation error in actor-critic methods.  In _International conference on machine learning_ , pages 1587–1596. PMLR, 2018.
  * Gangapurwala et al. [2020] S. Gangapurwala, A. Mitchell, and I. Havoutis.  Guided constrained policy optimization for dynamic quadrupedal robot locomotion.  _IEEE Robotics and Automation Letters_ , 5(2):3642–3649, 2020.
  * Gangapurwala et al. [2021] S. Gangapurwala, M. Geisert, R. Orsolino, M. Fallon, and I. Havoutis.  Real-time trajectory adaptation for quadrupedal locomotion using deep reinforcement learning.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 5973–5979. IEEE, 2021.
  * Gangapurwala et al. [2022] S. Gangapurwala, M. Geisert, R. Orsolino, M. Fallon, and I. Havoutis.  Rloc: Terrain-aware legged locomotion using reinforcement learning and optimal control.  _IEEE Transactions on Robotics_ , 38(5):2908–2927, 2022.
  * Gazar et al. [2023] A. Gazar, M. Khadiv, A. Del Prete, and L. Righetti.  Multi-contact stochastic predictive control for legged robots with contact locations uncertainty.  _arXiv preprint arXiv:2309.04469_ , 2023.
  * Geilinger et al. [2020] M. Geilinger, D. Hahn, J. Zehnder, M. Bächer, B. Thomaszewski, and S. Coros.  Add: Analytically differentiable dynamics for multi-body systems with frictional contact.  _ACM Transactions on Graphics (TOG)_ , 39(6):1–15, 2020.
  * Geyer et al. [2002] H. Geyer, R. Blickhan, and A. Seyfarth.  Natural dynamics of spring-like running: Emergence of selfstability.  In _5th International conference on climbing and walking robots_ , volume 92. Professional Engineering Publishing Ltd Suffolk, England, 2002\.
  * Gibson [1977] J. J. Gibson.  The theory of affordances.  _Hilldale, USA_ , 1(2):67–82, 1977.
  * Goodfellow et al. [2016] I. Goodfellow, Y. Bengio, and A. Courville.  _Deep learning_.  MIT press, 2016.
  * Goswami and Vadakkepat [2018] A. Goswami and P. Vadakkepat.  _Humanoid robotics: a reference_.  Springer Publishing Company, Incorporated, 2018.
  * Grandia et al. [2023] R. Grandia, F. Jenelten, S. Yang, F. Farshidian, and M. Hutter.  Perceptive locomotion through nonlinear model predictive control.  _IEEE Transactions on Robotics_ , 2023.
  * Green et al. [2021] K. Green, Y. Godse, J. Dao, R. L. Hatton, A. Fern, and J. Hurst.  Learning spring mass locomotion: Guiding policies with a reduced-order model.  _IEEE Robotics and Automation Letters_ , 6(2):3926–3932, 2021.
  * Grimminger et al. [2020] F. Grimminger, A. Meduri, M. Khadiv, J. Viereck, M. Wüthrich, M. Naveau, V. Berenz, S. Heim, F. Widmaier, T. Flayols, et al.  An open torque-controlled modular robot architecture for legged locomotion research.  _IEEE Robotics and Automation Letters_ , 5(2):3650–3657, 2020.
  * Ha and Schmidhuber [2018] D. Ha and J. Schmidhuber.  World models.  _arXiv preprint arXiv:1803.10122_ , 2018.
  * Ha et al. [2018] S. Ha, J. Kim, and K. Yamane.  Automated deep reinforcement learning environment for hardware of a modular legged robot.  In _2018 15th international conference on ubiquitous robots (UR)_ , pages 348–354. IEEE, 2018.
  * Ha et al. [2020] S. Ha, P. Xu, Z. Tan, S. Levine, and J. Tan.  Learning to walk in the real world with minimal human effort.  _arXiv preprint arXiv:2002.08550_ , 2020.
  * Haarnoja et al. [2018a] T. Haarnoja, K. Hartikainen, P. Abbeel, and S. Levine.  Latent space policies for hierarchical reinforcement learning.  In _International Conference on Machine Learning_ , pages 1851–1860. PMLR, 2018a.
  * Haarnoja et al. [2018b] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, et al.  Soft actor-critic algorithms and applications.  _arXiv preprint arXiv:1812.05905_ , 2018b.
  * Haarnoja et al. [2019] T. Haarnoja, S. Ha, A. Zhou, J. Tan, G. Tucker, and S. Levine.  Learning to walk via deep reinforcement learning.  _Robotics: Science and Systems_ , 2019.
  * Haarnoja et al. [2024] T. Haarnoja, B. Moran, G. Lever, S. H. Huang, D. Tirumala, J. Humplik, M. Wulfmeier, S. Tunyasuvunakool, N. Y. Siegel, R. Hafner, et al.  Learning agile soccer skills for a bipedal robot with deep reinforcement learning.  _Science Robotics_ , 9(89):eadi8022, 2024.
  * Hafner et al. [2019a] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi.  Dream to control: Learning behaviors by latent imagination.  _arXiv preprint arXiv:1912.01603_ , 2019a.
  * Hafner et al. [2019b] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson.  Learning latent dynamics for planning from pixels.  In _International conference on machine learning_ , pages 2555–2565. PMLR, 2019b.
  * Hammoud et al. [2021] B. Hammoud, M. Khadiv, and L. Righetti.  Impedance optimization for uncertain contact interactions through risk sensitive optimal control.  _IEEE Robotics and Automation Letters_ , 6(3):4766–4773, 2021.
  * Han et al. [2023] L. Han, Q. Zhu, J. Sheng, C. Zhang, T. Li, Y. Zhang, H. Zhang, Y. Liu, C. Zhou, R. Zhao, et al.  Lifelike agility and play on quadrupedal robots using reinforcement learning and generative pre-trained models.  _arXiv preprint arXiv:2308.15143_ , 2023.
  * Hansen et al. [2003] N. Hansen, S. D. Müller, and P. Koumoutsakos.  Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (cma-es).  _Evolutionary computation_ , 11(1):1–18, 2003\.
  * He et al. [2024] T. He, C. Zhang, W. Xiao, G. He, C. Liu, and G. Shi.  Agile but safe: Learning collision-free high-speed legged locomotion.  _arXiv preprint arXiv:2401.17583_ , 2024.
  * Heess et al. [2017] N. Heess, D. Tb, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, S. Eslami, et al.  Emergence of locomotion behaviours in rich environments.  _arXiv preprint arXiv:1707.02286_ , 2017.
  * Heijmink et al. [2017] E. Heijmink, A. Radulescu, B. Ponton, V. Barasuol, D. G. Caldwell, and C. Semini.  Learning optimal gait parameters and impedance profiles for legged locomotion.  In _2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids)_ , pages 339–346. IEEE, 2017.
  * Hessel et al. [2018] M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver.  Rainbow: Combining improvements in deep reinforcement learning.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 32, 2018.
  * Hester et al. [2010] T. Hester, M. Quinlan, and P. Stone.  Generalized model learning for reinforcement learning on a humanoid robot.  In _2010 IEEE International Conference on Robotics and Automation_ , pages 2369–2374. IEEE, 2010.
  * Hiraoka et al. [2021] T. Hiraoka, T. Imagawa, T. Hashimoto, T. Onishi, and Y. Tsuruoka.  Dropout q-functions for doubly efficient reinforcement learning.  _arXiv preprint arXiv:2110.02034_ , 2021.
  * Ho and Ermon [2016] J. Ho and S. Ermon.  Generative adversarial imitation learning.  _Advances in neural information processing systems_ , 29, 2016.
  * Hoeller et al. [2021] D. Hoeller, L. Wellhausen, F. Farshidian, and M. Hutter.  Learning a state representation and navigation in cluttered and dynamic environments.  _IEEE Robotics and Automation Letters_ , 6(3):5081–5088, 2021.
  * Hoeller et al. [2023] D. Hoeller, N. Rudin, D. Sako, and M. Hutter.  Anymal parkour: Learning agile navigation for quadrupedal robots.  _arXiv preprint arXiv:2306.14874_ , 2023.
  * Hornby et al. [2000] G. S. Hornby, S. Takamura, J. Yokono, O. Hanagata, T. Yamamoto, and M. Fujita.  Evolving robust gaits with aibo.  In _Proceedings 2000 iCRA. millennium conference. iEEE international conference on robotics and automation. symposia proceedings (cat. no. 00CH37065)_ , volume 3, pages 3040–3045. IEEE, 2000.
  * Hornik [1991] K. Hornik.  Approximation capabilities of multilayer feedforward networks.  _Neural networks_ , 4(2):251–257, 1991.
  * Howard [1960] R. A. Howard.  Dynamic programming and markov processes.  _Technology Press of Massachusetts Institute of Technology_ , 1960\.
  * Howell et al. [2022] T. A. Howell, S. L. Cleac’h, J. Z. Kolter, M. Schwager, and Z. Manchester.  Dojo: A differentiable simulator for robotics.  _arXiv preprint arXiv:2203.00806_ , 2022.
  * Hsu et al. [2023] K.-C. Hsu, H. Hu, and J. F. Fisac.  The safety filter: A unified view of safety-critical control in autonomous systems.  _Annual Review of Control, Robotics, and Autonomous Systems_ , 7, 2023\.
  * Hu et al. [2019] Y. Hu, L. Anderson, T.-M. Li, Q. Sun, N. Carr, J. Ragan-Kelley, and F. Durand.  Difftaichi: Differentiable programming for physical simulation.  _arXiv preprint arXiv:1910.00935_ , 2019.
  * Hu et al. [2023] Y. Hu, Q. Xie, V. Jain, J. Francis, J. Patrikar, N. Keetha, S. Kim, Y. Xie, T. Zhang, Z. Zhao, et al.  Toward general-purpose robots via foundation models: A survey and meta-analysis.  _arXiv preprint arXiv:2312.08782_ , 2023.
  * Huang et al. [2022] C. Huang, G. Wang, Z. Zhou, R. Zhang, and L. Lin.  Reward-adaptive reinforcement learning: Dynamic policy gradient optimization for bipedal locomotion.  _IEEE transactions on pattern analysis and machine intelligence_ , 2022.
  * Huang et al. [2023] X. Huang, Z. Li, Y. Xiang, Y. Ni, Y. Chi, Y. Li, L. Yang, X. B. Peng, and K. Sreenath.  Creating a dynamic quadrupedal robotic goalkeeper with reinforcement learning.  In _2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 2715–2722. IEEE, 2023.
  * Hubicki et al. [2016] C. Hubicki, J. Grimes, M. Jones, D. Renjewski, A. Spröwitz, A. Abate, and J. Hurst.  Atrias: Design and validation of a tether-free 3d-capable spring-mass bipedal robot.  _The International Journal of Robotics Research_ , 35(12):1497–1521, 2016.