# Reinforcement Learning and Control of a Lower Extremity Exoskeleton for
Squat Assistance

Shuzhen Luo
Department of Biomedical Engineering
New Jersey Institute of Technology
Newark, NJ 07102, USA
<EMAIL_ADDRESS>
&Ghaith Androwis
Department of Biomedical Engineering
New Jersey Institute of Technology
Newark, NJ 07102, USA
Kessler Foundation
West Orange, NJ 07052, USA
<EMAIL_ADDRESS>
&Sergei Adamovich
Department of Biomedical Engineering
New Jersey Institute of Technology
Newark, NJ 07102, USA
<EMAIL_ADDRESS>
&Hao Su
Department of Mechanical Engineering
The City University of New York
City College, NY, 10023, USA
<EMAIL_ADDRESS>
&Xianlian Zhou
Department of Biomedical Engineering
New Jersey Institute of Technology
Newark, NJ 07102, USA
<EMAIL_ADDRESS>
Corresponding author.

###### Abstract

A significant challenge for the control of a robotic lower extremity
rehabilitation exoskeleton is to ensure stability and robustness during
programmed tasks or motions, which is crucial for the safety of the mobility-
impaired user. Due to various levels of the user’s disability, the human-
exoskeleton interaction forces and external perturbations are unpredictable
and could vary substantially and cause conventional motion controllers to
behave unreliably or the robot to fall down. In this work, we propose a new,
reinforcement learning-based, motion controller for a lower extremity
rehabilitation exoskeleton, aiming to perform collaborative squatting
exercises with efficiency, stability, and strong robustness. Unlike most
existing rehabilitation exoskeletons, our exoskeleton has ankle actuation on
both sagittal and front planes and is equipped with multiple foot force
sensors to estimate center of pressure (CoP), an important indicator of system
balance. This proposed motion controller takes advantage of the CoP
information by incorporating it in the state input of the control policy
network and adding it to the reward during the learning to maintain a well
balanced system state during motions. In addition, we use dynamics
randomization and adversary force perturbations including large human
interaction forces during the training to further improve control robustness.
To evaluate the effectiveness of the learning controller, we conduct numerical
experiments with different settings to demonstrate its remarkable ability on
controlling the exoskeleton to repetitively perform well balanced and robust
squatting motions under strong perturbations and realistic human interaction
forces.

_K_ eywords lower extremity rehabilitation exoskeleton $\cdot$ reinforcement
learning $\cdot$ center of pressure $\cdot$ balanced squatting control $\cdot$
human-exoskeleton interaction

##  1 Introduction

Due to the aging population and other factors, an increasing number of people
are suffering from neurological disorders, such as stroke, central nervous
system disorder, and spinal cord injury (SCI) that affect the patient’s
mobility. Emerging from the field of robotics, robotic exoskeletons have
become a promising solution to enable mobility impaired people to perform the
activities of daily living (ADLs) [1, 2, 3]. Lower-limb rehabilitation
exoskeletons are wearable bionic devices that are equipped with powerful
actuators to assist people to regain their lower leg function and mobility.
With a built-in multi-sensor system, an exoskeleton can recognise the wearer’s
motion intentions and assist the wearer’s motion accordingly [4]. Compared to
traditional physical therapy, rehabilitation exoskeleton robots have the
advantages of providing more intensive patient repetitive training, better
quantitative feedback, and improved life quality for patients [4].

A degradation or loss of balance as a result of neuromusculoskeletal disorders
or impairments is a common symptom, for instance, in patients with SCI or
stroke. Balance training in the presence of external perturbations [5] is
considered as one of the more important factors in evaluating patients’
rehabilitation performance. A rehabilitation exoskeleton can be employed for
balance training to achieve static stability (quiet standing) or dynamic
stability (squatting, sit-to-stand, and walking) [6, 7, 8]. Squatting
exercises are very common for resistance-training programs because their
multiple-joint movements are a characteristic of most sports and daily living
activities. In rehabilitation, squatting is commonly performed as an important
exercise for patients during the recovery of various lower extremity injuries
[9, 10, 11, 12]. Squatting, which is symmetric by nature, can help coordinate
bilateral muscle activities and strengthen weaker muscles on one side (e.g.
among hemiplegia patients) or both sides. Compared to walking, squatting is
often perceived to be safer for patients who are unable to perform these
activities independently. In addition, the range of motion and the joint
torques required for squatting are often greater than walking [11]. With a
reliable lower extremity rehabilitation exoskeleton, performing squatting
exercises without external help (e.g. from a clinician) will be a confidence
boost for patients to use the exoskeleton independently. However, in order for
the exoskeletons to cooperate with the human without causing risks of harm,
advanced balance controllers to robustly perform squatting motion that can
deal with a broad range of environment conditions and external perturbations
need to be developed.

Most existing lower extremity rehabilitation exoskeletons require the human
operator to use canes for additional support or having a clinician or helper
to provide balance assistance to avoid falling down. They often offer
assistance via pre-planned trajectories of gait and provide limited control to
perform diverse human motions. Some well known exoskeletons include the ReWalk
(ReWalk Robotics), Ekso (Ekso bionics), Indego (Parker Hannifin), TWIICE [2]
and VariLeg [13]. When holding the crutches, the patient’s interactions with
the environment is also limited [14]. One example is that the patient is
unlikely to perform squatting with long canes or crutches. Very few
exoskeletons are able to assist human motions such as walking without the need
of crutches or helpers with the exception of a few well known ones: the Rex
(Rex Bionics) [15] and the Atalante (Wandercraft) [16]. These exoskeletons
free the user’s hands, but come at the cost of an very low walking speed and
an increased overall weight (38kg for the Rex, 60kg for the Atalante) and are
very expensive [2]. In this paper, we introduce a relatively light weight
lower extremity exoskeleton that includes a sufficient number of degrees of
freedom (DoF) with strong actuation. On each side, this skeleton system has a
1 DoF hip flexion/extension joint, a 1 DoF knee flexion/extension joint, and a
2-DoF ankle joint, which can perform ankle dorsi/plantar flexion as well as
inversion/eversion that can swing the center of mass laterally in the frontal
plane. Moreover, four force sensors are equipped on each foot for accurate
measurement of ground reaction forces (GRFs) to estimate the center of
pressure (CoP) so as to build automatic balance control without external
crutches assistance.

Designing a robust balance control policy for a lower extremity exoskeleton is
particularly important and represents a crucial challenge due to the balance
requirement and safety concerns for the user [4, 17]. First, the control
policy needs to run in real-time with limited sensing and capabilities
dictated by the exoskeletons. Second, due to various levels of patients’
disability, the human-exoskeleton interaction forces are unpredictable and
could vary substantially and cause conventional motion controllers to behave
unreliably or the robot to fall down. Virtual testing of a controller with
realistic human interactions in simulations is very challenging and the risk
of testing on real humans is even greater. To the best of our knowledge,
investigations presenting robust controllers against large and uncertain
perturbation forces (e.g. due to human interactions) have rarely been carried
out as biped balance control without perturbation itself is a challenging
task. Most existing balance controller designs for such lower extremity
rehabilitation exoskeletons focused mostly on the trajectory tracking method,
conventional control like Proportional–Integral–Derivative (PID) [18], model-
based predictive control [19], fuzzy control [20], impedance control [21, 22],
and momentum-based control for standing [6]. Although the trajectory tracking
approaches can be easily applied to regular motions, its robustness against
unexpected large perturbations is not great. On the other hand, model-based
predictive control could be ineffective or even unstable due to inaccurate
dynamics modeling, and it typically requires a laborious task-specific
parameters tuning. The momentum-based control strategies have also been
applied to impose standing balancing on the exoskeleton [6, 23], which was
first applied in humanoid robotics to impose standing and walking balance [24,
25]. This method aimed to simultaneously regulate both the linear and angular
component of the whole body momentum for balance maintenance with desired GRF
and CoP at each support foot. The movement of system CoP is an important
indicator of system balance [24]. When the CoP leaves or is about to leave the
support base, a possible toppling of a foot or loss of balance is imminent and
the control goal is to bring the CoP back inside the support base to keep
balance and stability. Although the CoP information can sometimes be estimated
from robot dynamics [24], the reliability of such estimation highly depends on
the accuracy of the robot model and sensing of joint states. When a human user
is involved, it is almost impossible to estimate the CoP accurately due to the
difficulty to estimate the user’s dynamic properties or real-time joint
motions. Therefore, it is highly desired to obtain the foot CoP information
directly and accurately. In most existing lower extremity rehabilitation
exoskeletons, the mechanical structures of the foot are either relatively
simple with no force or pressure sensors or with sensors but no capability to
process the GRF and CoP information for real-time fall detection or balance
control. In this work, a lower extremity rehabilitation exoskeleton with force
sensors equipped on each foot for accurate estimation of CoP is presented.
Inspired by the CoP-associated balance and stability [24], this paper aims to
explore a robust motion controller to encourage the system CoP to stay inside
a stable region when subjected to the uncertainty of human interaction and
perturbations.

Recently, model-free control methods, like reinforcement learning (RL),
promise to overcome the limitations of prior model-based approaches that
require an accurate dynamic model. It has gained considerable attention in
multi-legged robots control for their capability to produce controllers that
can perform a wide range of complicated tasks [26, 27, 28, 29, 30]. The RL-
based balance control approach for lower extremity rehabilitation exoskeletons
to perform squatting motion have not been investigated before, especially when
balancing with a human strapped inside is considered. Since the coupling
between the human and exoskeleton could leads to unexpected perturbation
forces, it is highly desired to develop a robust controller to learn
collaborative human-robot squatting skills. In this paper, we propose a novel
robust control framework based on RL to train a robust control policy that
operates on the exoskeleton in real-time so as to overcome the external
perturbations and unpredictable varying human-exoskeleton force.

The central contributions of this work are summarized in the following:

  * •

We build a novel RL-based motion control framework for a lower extremity
rehabilitation exoskeleton to imitate realistic human squatting motion under
random adversary perturbations or large uncertain human-exoskeleton
interaction forces.

  * •

We take advantage of the foot CoP information by incorporating it into the
state input of the control policy as well as the reward function to produce a
balance controller that is robust against various perturbations.

  * •

We demonstrate that the lightweight exoskeleton can carry a human to perform
robust and well-balanced squatting motions in a virtual environment with an
integrated exoskeleton and full-body human musculoskeletal model.

To demonstrate the effectiveness and robustness of the proposed control
framework, a set of numerical experiments under external random perturbations
and varying human-exoskeleton interactions are conducted. Dynamics
randomization is incorporated into the training to minimize the effects of
model inaccuracy and prepare for sim-to-real transfer.

##  2 Exoskeleton and Interaction Modeling

###  2.1 Mechanical Design of a Lower Extremity Robotic Exoskeleton

A lower extremity robotic exoskeleton device [31] is currently under
development by the authors to assist patients with ADL, such as balance,
ambulation and gait rehabilitation. In Figure 1A, the physical prototype of
this exoskeleton is shown. The total mass of the exoskeleton is $20.4kg$ and
the frame of the exoskeleton has been manufactured with Onyx (Markforged’s
nylon with chopped fiber) reinforced by continuous carbon fiber between
layers, using Markforged’s Mark Two printer (Markforged, INC, MA). The
exoskeleton has 14 independent DoFs (including 6 global DoFs for the pelvis
root joint): 1 DoF for the hip flexion/extension joint, 1 DoF for the knee
flexion/extension joint, and 2 DoFs for the ankle dorsiflexion/plantarflexion
and inversion/eversion joint on each side of the body. The 6-DoF pelvis (root)
joint is a free joint and unactuated, and the rest 8 DoFs on both sides are
actuated. Unlike most commercial rehabilitation exoskeletons that have either
passive or fixed ankles, the ankle of our system includes powered 2-DoF to
assist with dorsiflexion/plantarflexion and inversion/eversion [32]. All
joints of the robotic exoskeleton are powered by smart actuators (Dynamixel
Pro Motor H54-200-S500-R).

Both hip and knee joints are driven by bevel gears for compact design. The
ankle is actuated by two parallel motors that are attached to the posterior
side the shank support and operate simultaneously in a closed-loop to flex or
abduct the ankle. Figure 1C shows the foot model of our exoskeleton system. At
the bottom of each foot plate, four 2000N 3-axis force transducers (OptoForce
Kft, Hungary) are installed to measure GRFs. These measured forces can be used
to determine the CoP in real-time. The CoP is the point on the ground where
the tipping moment acting on the foot equals zero [33], with the tipping
moment defined as the moment component that is tangential to the foot support
surface. Let $\bm{O}_{i}$ denote the location of the $i-th$ force sensor and
$\bm{F}_{i}$ be its measured force, $\bm{n}$ denote the unit normal vector of
the foot support surface, and $\bm{C}$ denote the CoP point where the tipping
moment vanishes:

$\left[\sum_{i}^{4}(\bm{O}_{i}\bm{C}\times\bm{F}_{i})\right]\times\bm{n}=\bm{0}$
(1)

from which $\bm{C}$ can be computed.

Figure 1: The lower extremity exoskeleton and integrated human-exoskeleton
model. (A) the physical prototype of the exoskeleton. (B) its multibody
dynamics model. (C) the foot model with force sensors locations and the two
independent ankle DoFs indicated. The joint axes of the model are displayed as
well. (D) Integrated human and lower extremity exoskeleton model.

###  2.2 Exoskeleton Modeling

A multibody dynamics model of the exoskeleton, shown in Figure 1, is created
with mass and inertia properties estimated from each part’s 3D geometry and
density or measured mass. Simple revolute joints are used for the hips and
knees. Each ankle has 2 independent rotational DoFs but their rotation axes
are located at different positions (Figure 1C). These 2 DoFs are physically
driven by the closed-loop of two ankle motors together with linkage of
universal joints and screw joints. When extending or flexing the ankle, both
motors work in sync to move the two long screws up or down simultaneously.
When adducting or abducting the ankle, the motors move the screws in opposite
directions. The motors can generate up to 160$Nm$ dorsi/plantar flexion torque
[32]. In this study, we do not directly control these ankle motors. Instead,
we assume the two ankle DoFs can be independently controlled with sufficient
torques from these motors.

###  2.3 Modeling of Human Exoskeleton Interactions

To simulate realistic human exoskeleton interactions, the exoskeleton is
integrated with a full body human musculoskeletal model [34] that has a total
mass of $61kg$ and includes 284 musculotendon units. The assembled exoskeleton
has straps around the hip, femur and tibia to constraint the human motion. In
our model, the root of the human musculoskeletal model (pelvis) is attached to
the exoskeleton hip through a prismatic joint that allows relative movement
only along the vertical (up & down) direction (all rotations and the
translations along the lateral and fore-and-aft directions are fixed).
Meanwhile, an elastic spring with stiffness $k_{h}=10000N/m$ is utilized to
generate a vertical force and simulate the interaction between the human
pelvis and the robot waist structure. Similarly, we use four springs (arranged
in 90 degrees apart) with stiffness $k_{f}=2000N/m$ to simulate the
connections between the human femur and the strap on the robot femur. For the
human tibia, four springs with stiffness $k_{t}=2000N/m$ are also used to
connect with the robot tibia. During motion, spring forces are generated and
applied on both human and exoskeleton. The joints at the upper limb of the
human are assumed as weld joints. We also assume there is no relative motion
between the human foot and the exoskeleton foot due to tight coupling and
model that as a welding constraint. In Figure 1D, the integrated human
musculoskeletal and exoskeleton model is shown.

##  3 Learning Controller for Balanced Motion

In this section, we propose a robust motion control training and testing
framework based on RL that enables the exoskeleton to learn squatting skill
with strong balance and robustness. Figure 2 shows the overall learning
process of the motion controller. The details of the motion controller
learning process will be introduced in the following sections.

Figure 2: The overall learning process of the integrated robust motion
controller. We construct the learning controller as a Multi-Layer Perception
(MLP) neural network that consists of three hidden layers. The neural network
parameters are updated using the policy gradient method. The network produces
joint target positions, which are translated into torque-level commands by PD
(Proportional Derivative) control.

###  3.1 Reinforcement Learning with Motion Imitation

The controller (or control policy) is learned through a continuous RL process.
We design the control policy through a neural network with parameters
$\theta$, denoting the weights and bias in the neural network. The control
policy can be expressed as $\pi_{\theta}(a|s)$ and the agent (neural network
model) learns to update the parameters $\theta$ to achieve the maximum reward.
The learning controller (i.e. control policy network) is implemented as a
Multi-Layer Perception (MLP) network that consists of three fully connected
layers and ReLU as the activation function, as illustrated in Figure 2, the
sizes of three layers are set to 256, 256 and 128, respectively. At every time
step $t$, the agent observes the state $s_{t}$ from the environment, and
selects an action $a_{t}$ according to its control policy $\pi(a_{t}|s_{t})$.
$\pi(a_{t}|s_{t})$ is in the form of the probability distribution of actions
in a given state. The action $a_{t}$ is then converted to control commands
that drive the exoskeleton in the environment, which results a new state
$s_{t+1}$ and a scalar reward $r_{t}$ immediately. The objective is to learn a
control policy that maximizes the discounted sum of reward:

$\pi^{*}=\arg\\!\max_{\pi}\mathbb{E}_{\tau\sim p(\tau|\pi)}[\
\sum_{t=0}^{T-1}\gamma^{t}r_{t}]\ $ (2)

where $\gamma\in(0,1)$ is the discount factor, $\tau$ is the trajectory
$\\{(s_{0},a_{0},r_{0}),(s_{1},a_{1},r_{1}),\cdots\\}$ and $p(\tau|\pi)$
denotes the likelihood of a trajectory $\tau$ under a given control policy
$\pi$, and $T$ is the horizon of an episode.

The control policy is task- or motion-specific and starts with motion
imitation. Within the learning process (Figure 2), the input of the control
policy network is defined by
$s=\\{p_{t-2:t},v_{t-2:t},c_{t-2:t},a_{t-3:t-1},\hat{p}_{t+1:t+6}\\}$, in
which $p$ and $v$ are joint positions and velocities of the exoskeleton, and
$a_{t-3:t-1}$ represents the action history of three previous steps. To learn
a particular skill, we utilize the corresponding target joint pose from the
task motion at six future time-steps $\hat{p}_{t+1:t+6}$ as the motion prior
for feasible control strategies. Considering the importance of CoP as an
indicator of system balance and its ready availability from our exoskeleton
design, we incorporate the CoP position history $c_{t-2:t}$ into the state as
a feedback to the controller. As summarized in Figure 2, the learning
controller is given as input the combination of the state history, the action
history and the future target motions and outputs the joint target positions
as the actions. The use of task motion data alleviates the need to design
task-specific reward functions and thereby facilitates a general framework to
learn a diverse array of behaviors.

###  3.2 Learning with Proximal Policy Optimization (PPO)

To train the control policy network, we use a model-free RL algorithm known as
Proximal Policy Optimization (PPO). An effective solution to many RL problems
is the family of policy gradient algorithms, in which the gradient of the
expected return with respect to the policy parameters is computed and used to
update the parameters $\theta$ through gradient ascent. PPO is a model-free
policy gradient algorithm that samples data through interaction with the
environment and optimizes a “surrogate” objective function [35]. It utilizes a
trust region constraint to force the control policy update to ensure that the
new policy is not too far away from the old policy. The probability ratio
$r_{t}(\theta)$ is defined by:

$r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{old}}(a_{t}|s_{t})}.$
(3)

This probability ratio is a measure of how different the current policy is
from the old policy $\pi_{\theta_{old}}$ (the policy before the last update).
A large value of this ratio means that there is a large change in the updated
policy compared to the old one. PPO also introduces a modified objective
function that adopts clipped probability ratios which forms a pessimistic
estimate of the policy’s performance and avoids a reduction in performance
during the training process. The "surrogate" objective function is described
by considering the clipped objective:

$L(\theta)=\mathbb{E}_{t}\left[min\left(r_{t}(\theta)\hat{A}_{t},clip(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t}\right)\right]$
(4)

where $\epsilon$ is a small positive constant which constrain the probability
ratio $r_{t}(\theta)$. $\hat{A}_{t}$ denotes the advantage value at time step
$t$. $clip(\cdot)$ is the clipping function. Clipping the probability ratio
discourages the policy from changing too much and taking the minimum results
in using the lower, pessimistic bound of the unclipped objective. Thus any
change in the probability ratio is included when it makes the objective worse,
and otherwise is ignored. This can prevent the policy from changing too
quickly and leads to more stable learning. The control policy can be updated
by maximizing the clipped discounted total reward in Equation (4) with a
gradient ascent.

###  3.3 Proportional-Derivative(PD)-based torque

It has been reported that the performance of RL for continuous control depends
on the choice of action types [27]. It works better when the control policy
outputs PD position targets rather than joint torque directly. In this study,
we also let the learning controller output the joint target positions as
actions (shown in Figure 2). To obtain smooth motions, actions from the
control policy network are first processed by a second low-pass filter before
being applied to the robot. Our learning process allows the control policy
network and the environment to operate at different frequencies since the
environment often requires a small time step for integration. During each time
integration step, we apply preprocessed actions that are linear interpolated
from two consecutive filtered actions. Then the preprocessed actions are
specified as PD targets and the final PD-based torques applied to each joint
are calculated as

$\tau=clip\left(k_{p}(a_{t}-p_{t})-k_{v}\dot{p}_{t},-\hat{\tau},\hat{\tau}\right)$
(5)

where $k_{p}$ and $k_{v}$ are proportional gain and differential gain
respectively. The function $clip(\cdot)$ returns the upper bound $\hat{\tau}$
or the lower bound$-\hat{\tau}$ if the torque $\tau$ exceeds the limit.

###  3.4 Reward Function

We design the reward function to encourage the control policy to imitate a
target joint motion $\hat{p}_{t}$ of the exoskeleton while maintaining balance
with robustness. The reward function consists of pose reward $r_{t}^{p}$,
velocity reward $r_{t}^{v}$, end-effector reward $r_{t}^{e}$, root reward
$r_{t}^{root}$, center of mass reward $r_{t}^{com}$, CoP reward $r_{t}^{cop}$,
and torque reward $r_{t}^{torque}$, which is defined by:

$r_{t}=w^{p}r_{t}^{p}+w^{v}r_{t}^{v}+w^{e}r_{t}^{e}+w^{root}r_{t}^{root}+w^{com}r_{t}^{com}+w^{cop}r_{t}^{cop}+w^{torque}r_{t}^{torque}$
(6)

where $w^{p},w^{v},w^{e},w^{root},w^{com},w^{cop},w^{torque}$ are their
respective weights. The pose reward $r_{t}^{p}$ and velocity reward
$r_{t}^{v}$ match the current and task (target) motions in terms of the joint
positions $p_{t}$ and velocities $\dot{p}_{t}$:

$r_{t}^{p}=\exp[-\sigma_{p}\sum\limits_{j}||\hat{p}_{t}^{j}-p_{t}^{j}||^{2}]$
(7)
$r_{t}^{v}=\exp[-\sigma_{v}\sum\limits_{j}||\hat{\dot{p}}_{t}^{j}-\dot{p}_{t}^{j}||^{2}]$
(8)

where $j$ is the index of joints, $\hat{p}_{t}^{j}$ and
$\hat{\dot{p}}_{t}^{j}$ are target position and velocity respectively. The
end-effector reward is defined to encourage the robot to track the target
positions of selected end-effectors:

$r_{t}^{e}=\exp[-\sigma_{e}\sum\limits_{i}||\hat{x}_{t}^{i}-x_{t}^{i}||^{2}]$
(9)

where $i$ is the index of the end-effector. Let $x_{t}^{i}$ be the position of
end-effectors, that include left foot and right foot, relative to the moving
coordinate frame attached to the root (waist structure). The end-effectors
motions are supposed to match well if the joint angles match well, and vice
versa.

We also design the root reward function $r_{t}^{root}$ to track the task root
motion including the root’s position $\hat{x}_{t}^{root}$ and rotation
$\hat{q}_{t}^{root}$.

$r_{t}^{root}=\exp[-\sigma_{r1}||\hat{x}_{t}^{root}-x_{t}^{root}||^{2}-\sigma_{r2}||\hat{q}_{t}^{root}-q_{t}^{root}||^{2}]$
(10)

The overall center of mass reward $r_{t}^{com}$ is also considered in the
learning process, enabling the control policy to track the target height
during the complete squatting cycle.

$r_{t}^{com}=\exp[-\sigma_{com}||\hat{x}_{t}^{com}-x_{t}^{com}||^{2}]$ (11)

Besides the use of CoP positions in the state input as a feedback from the
controller, we also introduce a CoP reward function to encourage the current
CoP position $c_{t}^{cop}$ to stay inside a stable region around the center of
the foot support. By considering the geometric of the foot in the lower
extremity exoskeleton design, the stable region for foot CoP is defined as a
rectangle area $S$ around the foot geometric center whose width and length are
set to $7cm$ and $11cm$ respectively (narrower in the lateral direction than
forward direction). And the CoP reward function is expressed as

$r_{t}^{cop}=\left\\{\begin{aligned}
&\exp[-\sigma_{cop}||D(c_{t}^{cop},S)||^{2}],\text{if~{}}c_{t}^{cop}\in S\\\
&0,\ \text{if~{}}c_{t}^{cop}\notin S\\\ \end{aligned}\right.$ (12)

where $D(\cdot,\cdot)$ is the Euclidean distance between CoP and the center of
$S$. The goal of this CoP reward is to encourage the controller to predict an
action that will improve the balance and robustness of the exoskeleton’s
motion.

At last, we design the torque reward to reduce energy consumption or improve
efficiency and to prevent damaging joint actuators during the deployment.

$r_{t}^{torque}=\exp[-\sigma_{torque}\sum_{i}||\tau_{i}||^{2}]$ (13)

where $i$ is the index of joints.

###  3.5 Dynamics Randomization

Due to the model discrepancy between the physics simulation and the real-world
environment, well-known as reality or sim-to-real gap [36], the trained
control policy usually performs poorly in the real environment. In order to
improve the robustness of the controller against model inaccuracy and bridge
the sim-to-real gap, we need to develop a robust control policy capable of
handling various environments with different dynamics characteristics. To this
end, we adopt dynamics randomization [37, 38] in our training strategy, in
which dynamics parameters of the simulation environment are randomly sampled
from an uniform distribution for each episode. The objective in Equation (2)
is then modified to maximize the expected reward across a distribution of
dynamics characteristics $\rho(\mu)$:

$\pi^{*}=\arg\\!\max_{\pi}\mathbb{E}_{\mu\sim\rho(\mu)}\mathbb{E}_{\tau\sim
p(\tau|\pi,\mu)}[\sum\limits_{t=0}^{T-1}\gamma^{t}r_{t}],$ (14)

where $\mu$ represents the values of the dynamics parameters that are
randomized during training. By training policies to adapt to variability in
environment dynamics, the resulting policy will be more robust when
transferred to the real world.

##  4 Numerical Experiment Results and Discussion

We design a set of numerical experiments aiming to answer the following
questions: 1) Can the learning process generate feasible control policies to
control the exoskeleton to perform well-balanced squatting motions? 2) Will
the learned control policies be robust enough under large random external
perturbation? 3) Will the learned control policies be robust enough to sustain
stable motions when subjected to uncertain human-exoskeleton interaction
forces from a disabled human operator?

###  4.1 Simulation and Reinforcement Learning Settings

To demonstrate the effectiveness of our RL-based robust controller, we train
the lower extremity exoskeleton to imitate a 4s reference squatting motion
that is manually created based on human squatting motion. All 8 DoFs including
on each side 1 DoF on the hip, 1 DoF on the knee, and 2 DoFs on the ankle are
actuated to generate squatting motion. The open source library DART [39] is
utilized to simulate the exoskeleton dynamics. The GRFs are computed by a
Dantzig LCP (linear complementary problem) solver [40]. We utilize PyTorch
[41] to implement the neural network and the PPO method for the learning
process. The networks are initialized by the Xavier uniform method [42]. We
use a desktop computer with an Intel® Xeon(R) CPU E5-1660 v3 @ 3.00GHz × 16 to
generate samples in parallel during training. Totally about 20 million samples
are collected in each simulation. The policy and value networks are updated at
a learning rate of $10^{-4}$, which is linearly decreased to 0 when 20 million
samples are collected. PPO algorithm is robust in that hyperparameter
initialization is a bit more forgiving and it can handle a wide variety of RL
tasks. We do not deliberately tune hyperparameters and just use the common
setting as in the literature. Hyper-parameters settings for training using PPO
are shown in Table 1.

To verify the robustness of the trained controller, we test the control
policies in out-of-distribution simulated environments, where the dynamic
parameters of the exoskeleton are sampled randomly from a larger range of
values than those during training. Table 2 shows the dynamics parameters
details of the exoskeleton and their range during training and testing. Note
that the observation latency denotes the observation time delay in the real
physical system due to sensor noise and time delay during information
transfer. Considering the observation latency improves the reality of the
simulations and further increases the difficulty for policy training. The
simulation frequency (time step for the environment simulation) and control
policy output frequency are set to $900$Hz and $30$Hz, respectively. According
to the PD torque Equation (5), the parameters about the proportional gain
$k_{p}$ and differential gain $k_{v}$ are set to 900 and 40 respectively. The
differential gain $k_{v}$ is chosen to be sufficiently high to prevent
unwanted oscillation on the exoskeleton robot. From our experience, the
control performance is robust against the variance of gains to a certain
extent. For instance, increasing or decreasing the position gain $k_{p}$ to
1200 or 800 does not noticeably change the control performance. The reward
weight settings are:
$w^{p}=0.8,w^{cop}=0.8,w^{v}=0.1,w^{ee}=0.7,w^{com}=0.4,w^{root}=0.7,w^{torque}=0.1$.
The torque limit for each joint is set to 100$Nm$ in the simulation.

Table 1: Hyper-parameters settings for training Parameters | Value | Parameters | Value
---|---|---|---
Discount factor | 0.99 | epochs | 10
Policy Adam learning rate | $10^{-4}$ | clip threshold | 0.2
batch size | 128 | memory buffer | 2048
Table 2: Dynamic parameters and their respective range of values used during training and testing. A larger range of values are used during testing to evaluate the generalization ability of control policies in dynamics uncertainties. Dynamic Parameters | Training Range | Testing Range
---|---|---
Friction coefficient | [0.9,1.6]*default value | [0.7,2.0]*default value
Mass | [0.8,1.2]*default value | [0.7,1.5]*default value
Motor strength | [0.8,1.2]*default value | [0.7,1.3]*default value
Observation latency | [0,0.04]s | [0,0.06]s
Inertial | [0.5,1.5]*default value | [0.4,1.6]*default value
Center of Mass | [0.9,1.2]*default value | [0.8,1.3]*default value

###  4.2 Learned Squatting Skill

####  4.2.1 Case 1–feasibility demonstration

In the first case, a squatting motion controller is learned from the 4s
reference squatting motion without considering external perturbation. It is
worth noting that, compared with the training, we use the larger-range
dynamics randomization of the exoskeleton model to demonstrate the
generalization ability of our learned controller (as shown in Table 2). A
series of snapshots of the squatting behavior of the lower extremity
exoskeleton under the learned control policy are shown in Figure 3A. The lower
extremity exoskeleton can perform the squatting and stand-up cycle with a
nearly symmetric motion. Figure 4A displays the hip flexion/extension, knee
flexion/extension, ankle dorsiflexion/plantarflexion and ankle
inversion/eversion joint angles of the left leg in the first squatting cycle.
Joint torques are depicted in Figure 4B and the peak torque at the knee joint
for the squatting is around $13.5Nm$. Due to the weights of the waist support
structure and the battery (around $2.2kg$) mounted on the back of it, the hip
joint consistently produces a positive torque to prevent the waist structure
from rotating downwards due to gravity.

Figure 3: Snapshots of the squatting motion of the exoskeleton. This RL-based
controller enables the lower extremity exoskeleton to perform the squatting
skill under a broad range of external perturbations. (A) performing the
squatting skill without perturbation forces. (B) performing the squatting
skill with large random perturbation forces. (Red, cyan and blue arrows show
the random perturbation force applied on the hip, femur and tibia
respectively.) (C) performing the squatting skill with human interaction.
Figure 4: Case1: joint behaviors in the first squatting cycle. (A) hip, knee,
ankle dorsiflexion/plantarflexion, ankle inversion/eversion joint angles with
respect to time during the first squatting cycle. (B) corresponding joint
torques during the first squatting cycle. Figure 5: Case1: foot CoP
trajectories during the first squatting cycle. Green dotted line depicts the
stable region border of CoP. (A) left foot CoP and right foot CoP trajectories
(lateral direction). (B) left foot CoP and right foot CoP trajectories
(forward direction).

Figure 5 show the foot CoP trajectories of both left and right feet in the
lateral and forward directions, which are calculated in real-time using the
ground contact force information. As it can be seen, the exoskeleton
controller can keep the foot CoP well inside the stable region for both
lateral and forward directions in a complete squatting cycle. Noted at the
beginning the CoP is close to the back edge (due to its initial state) but
gradually it is bought to near the center. This indicates that the exoskeleton
controller is able to recognize the current state of CoP and capable of bring
it to a more stable state. And the balance in the lateral direction is better
than that in the forward direction due to the symmetric nature of the
squatting motion. The right foot CoP trajectories have very similar patterns
with those of the left foot CoP.

Figure 6: Case1: performance of the RL based controller without external
perturbation during multiple squatting cycles. The first figure demonstrates
the foot CoP reward (green and red lines) , end-effector tracking reward
(magenta line) and joint position tracking reward (black line) calculated
according to Equation (7)-(12). The bottom figure depicts the actions for the
hip, knee ankle joint predicted from the neural network.

Figure 6 presents the performance of the learned controller while performing
multiple squatting cycles. We can clearly observe the high CoP rewards,
indicating good system balance when the exoskeleton performs the squatting
motion. The relatively high joint position tracking and end-effector tracking
rewards illustrate strong tracking performance of the control system. The
second figure shows the torques for the hip, knee, and ankle joints. The last
figure demonstrates the predicted actions (PD target positions) for these
joints. It is clear from these plots that the actions predicted from the
policy network are smooth and exhibit clear cyclic patterns.

####  4.2.2 Case 2–robustness against random perturbation forces

In the second case, we aim to verify the robustness of the controller under
random external perturbation forces. From our tests, the learned control
policy from case 1, trained without any perturbation forces, could perform
well with random perturbation forces up to $100N$. To further improve the
robustness of our controller to handle greater perturbation forces, we now
introduce random perturbation forces during the training process of the
learning controller. The perturbation forces are applied to three parts of the
exoskeleton: hip, femur, and tibia. For femur and tibia, the magnitude of
forces are randomly sampled in the range $(0,100)N$ and no restriction is set
for the direction. For the hip, we randomly sample the magnitude of force in
$(0,200)N$ but restricted the direction to $0\sim 20$ degrees from the
vertical direction assuming no large lateral pushing perturbation.

Figure 7: Case2: joint behaviors under random perturbation forces in the first
squatting cycle. (A) hip, knee, ankle dorsiflexion/plantarflexion, ankle
inversion/eversion joint angles with respect to time in the first squatting
cycle. (B) corresponding joint torques in the first squatting cycle. (C)
magnitudes of the random perturbation forces applied on the three parts of the
exoskeleton: hip, femur and tibia during testing. Figure 8: Case2: foot CoP
trajectories under random external perturbations in the first squatting cycle
and robustness test on the foot CoP stability under 75% greater perturbation
forces compared with the training setting. (A) left and right foot CoP
trajectories (lateral direction). (B) left and right foot CoP trajectories
(forward direction). (C) left foot CoP trajectory (lateral direction) under
75% greater perturbation forces compared with the training setting. (D) left
foot CoP trajectory (forward direction) under greater perturbation forces
compared with the training setting.

Figure 3B shows a series of snapshots of the lower extremity exoskeleton
behavior with the newly trained controller when tested with random, large
perturbation forces during squatting motion. Joint angles and torques at the
hip, knee and ankle joint of the left leg in the first squatting cycle are
shown in Figure 7. We can clearly observe that the motion is still relative
smooth and the torques calculated from Equation (5) have more ripples in
response to the random perturbation forces. Figure 7C shows randomly varying
perturbation forces applied on the hip, femur and tibia. Compared to the joint
angles and torques without external perturbation (Case 1, Figure 4), the joint
torques under random perturbation forces are almost doubled but the joint
angles are relatively close. Moreover, under the random perturbations that
could influence the balance in the frontal plane, the learned controller can
still keep balance and stability with the actuation of the ankle
inversion/eversion joint.

Figure 8 shows the CoP trajectories of the left and right feet in both lateral
and forward directions in the first squatting cycle. As shown in this figure,
the foot CoP trajectories have oscillation under the random perturbation
forces compared with Case1. But the robot can still keep the CoP inside the
safe region of the support foot to guarantee stability and balance. It further
validates that this controller enables the robot to perform squatting motion
with strong stability and robustness. To further demonstrate the robustness of
the learning controller, we increase the random perturbation forces up to
$75\%$ greater than the training perturbation (e.g. up to $350N$ for the hip
force) and found that the exoskeleton can still perform the squatting motion
without losing its balance as illustrated in Figure 8C-D about the left foot
CoP stability. Real-time tracking results of the exoskeleton with multiple
squatting cycles are shown in Figure 9. From Figure 9, both the foot CoP
reward and the end-effector reward remain high under the random perturbation
forces. The torques are greater than that without external perturbation
forces, the peak torque for the knee joint is close to $40Nm$. The actions
predicted from the policy network are also smooth and exhibit cyclic patterns.

Figure 9: Case2: real-time performance of the reinforcement-learning based
controller with large, random external perturbation during multiple squatting
cycles. The first figure demonstrates the real-time foot CoP reward (green and
red lines), end-effector tracking reward (magenta line) and joint position
tracking reward (black line) calculated according to Equation (7)-(12). The
bottom figure depicts the actions for the hip, knee ankle joint predicted from
the neural network.

####  4.2.3 Case3–robustness under human-exoskeleton interaction

In the third case, the human musculoskeletal model is integrated with the
exoskeleton to simulate more realistic perturbation forces. As described in
the section 2.3, the springs at the strap locations can generate the varying
interaction forces between the human and exoskeleton robot during the motion,
which are applied on both human and exoskeleton. We first train the network
with the integrated human-exoskeleton model to account for the interaction
forces. Here we do not consider the active muscle contraction of the human
operator or actuation torques on the human joints, considering the operator
could be a patient suffering from spinal cord injury or stroke, with very
limited or no control of his or her own body. Nonetheless, the passive muscle
forces as described in [34] during movement are incorporated. The squatting
skill learned by the exoskeleton and performance of the motion controller are
shown in Figure 3C and Figure 10-12.

Figure 10: Case3: joint behaviors under human-exoskeleton interactions during
the first squatting cycle. (A) hip, knee, ankle dorsiflexion/plantarflexion,
ankle inversion/eversion joint angles in the first squatting cycle. (B)
corresponding joint torques in the first squatting cycle. (C) human-
exoskeleton interaction (strap) forces during multiple squatting cycles.
hip$\\_$force, femur$\\_$force and tibia$\\_$force in the figure represent the
interaction forces between the human and the strap on the exoskeleton
respectively.

As shown in Figure 3C, the rehabilitation exoskeleton is able to assist the
human to perform the squatting motion without external assistance. Angles and
torques at the hip, knee and ankle joint of the left leg in the first cycle
are shown in Figure 10A-B. The torques at the hip, knee and ankle joint are
greater than those without the human interaction while still below the maximum
torques. Figure 11 shows the CoP trajectories of the left foot and right foot
in the lateral and forward directions under the predicted human-skeleton
interaction forces (as shown in Figure 12A). From Figure 11A, there exists a
symmetric pattern between the left and right foot CoP trajectories in the
lateral direction, which indicates that the squatting motion is well balanced
in the frontal plane. Real-time tracking results of the controlled exoskeleton
under the human-exoskeleton interactions with more squatting cycles are shown
in Figure 12. As shown in Figure 12B, the high CoP reward in the first figure
indicates that the proposed control system has strong stability and robustness
to the human interaction forces. The peak torque for all joints are less than
$70Nm$. The largest peak torque happens at the ankle dorsi/plantar flexion
joint ($68Nm$), which is much smaller than its 160$Nm$ capacity [32]. The
actions predicted from the policy network, shown in the last figure of Figure
12, are still smooth and cyclic in general.

To further validate the learned controller’s ability to cope with unfamiliar
dynamics of the exoskeleton, we test the learned controller in 200 out-of-
distribution simulated environments, where the dynamics parameters are sampled
from a larger range of values than those used during training (as shown in
Table 2). Figure 13 visualizes the performance of the learned controller in
200 simulated environments with different dynamics. Figure 13A depicts the
rewards statistics (mean and standard deviation) with respect to time
calculated from Equation (7)-(12) under 200 simulated environments. The end-
effector reward indicating the foot tracking performance consistently
maintains a high value, revealing the exoskeleton robot has no falling
condition in 200 simulated environments with unfamiliar dynamics and it can
stand on the ground with stationary feet when performing the complete
squatting motion. The joint position tracking and foot CoP also achieve a high
reward with less variance under more diverse dynamics of the exoskeleton
robot. Figure 13B shows the average reward of a complete squatting cycle for
each simulated environment. These results suggest that the learned controller
is able to effortlessly generalize to environments that differ from those
encountered during training and achieve good control performance under very
diverse dynamics. This also suggests that the learned controller will likely
be robust when transferred to the real exoskeleton.

Figure 11: Case3: foot CoP trajectories under human-exoskeleton interactions
in the first squatting cycle. The stable region border is marked with green
dotted lines. (A) left foot CoP and right foot CoP trajectories (lateral
direction). (B) left foot CoP and right foot CoP trajectories (forward
direction). Figure 12: Case3: performance of the RL based controller under
human-exoskeleton interactions during more squatting cycles. (A) human-
exoskeleton interaction (strap) forces during multiple squatting cycles.
hip$\\_$force, femur$\\_$force and tibia$\\_$force in the figure represent the
interaction forces between the human and the strap on the exoskeleton
respectively. (B) performance of the RL based controller. The first figure
demonstrates the foot CoP reward (green and red lines) , end-effector reward
(magenta line) and joint position tracking reward (black line) calculated
according to Equation (7)-(12). The bottom figure depicts the actions for the
hip, knee ankle joint predicted from the neural network. Figure 13:
Performance of learned controller under human-exoskeleton interactions in 200
simulated environments with different dynamics. The dynamic model of the
exoskeleton is randomly initialized. (A) Rewards statistics (curve: mean;
shade: standard deviation) with respect to time for 200 simulated
environments. (B) Average reward of a complete squatting cycle for each
simulated environment.

##  5 Discussion

Through these designed numerical tests, we verified that our learning
framework can produce effective neural network-based control policies for the
lower extremity exoskeleton to perform well-balanced squatting motions. By
incorporate adverse perturbations in training, the learned control policies
are robust against large random external perturbations during testing. And it
can sustain stable motions when subjected to uncertain human-exoskeleton
interaction forces from a disabled human user. From all numerical tests
performed, the effectiveness and robustness of the learned balance controller
are demonstrated by its capability to maintain CoP inside the foot stable
region along both lateral and forward directions.

In this study, we evaluated the controllers in a specific case for which the
human musculoskeletal model has only passive muscle response (i.e. without
active muscle contraction). In reality, the human-exoskeleton interaction
forces might vary substantially for different users with different weights and
levels of disability. One may assume that a user with good body or muscle
control tends to minimize the interaction forces on the straps. On the other
hand, a passive human musculoskeletal model tends to generate larger
interaction forces. Therefore, using a passive musculoskeletal model can be
considered a more difficult task for the exoskeleton. Further investigations
with an active human musculoskeletal model are possible but will likely to
need additional information on the health condition of the user.

Through dynamics randomization, our learned controller is robust against
modeling inaccuracy and differences between the training and testing
environments [43]. Nonetheless, it is still beneficial to validate and further
improve the dynamic exoskeleton model. This can be done through a model
identification process, for which we can conveniently rely on the real-time
measurement of GRFs and derived CoP information from the foot force sensors
during programmed motions. Experiments can be conducted to correlate the CoP
position with the exoskeleton posture and the CoP movement with motion. Stable
squatting motions of different velocities can be used to record the dynamic
responses of the exoskeleton and the collected data then can be used to
further identify or optimize the model’s parameters (such as inertia
properties and joint friction coefficients).

By incorporating motion imitation into the learning process, the proposed
robust control framework has the potential to learn a diverse array of human
behaviors without the need to design skill-specific reward functions. Common
rehabilitation motions such as sit-to-stand, walking on flat or inclined
ground can be learning by feeding proper target motions. Due to the nature of
imitation learning and CoP based balance control, we foresee minimal changes
to the learning framework with the exception of crafting different target
motions for imitation. The learning process will automatically create specific
controllers that can produce physically feasible and stable target motions
(even when the target motion is coarsely generated and may not be physically
feasible).

Transferring or adapting RL control policies trained in the simulations to the
real hardware remains a challenge in spite of some demonstrated successes. To
bridge the so-called "sim-to-real" gap, we have adopted dynamics randomization
during training that is often used to prepare for sim-to-real transfer [43].
In a recent work by Exarchos et al. [44], it is shown kinematic domain
randomization can also be effective for policy transfer. Additionally,
efficient adaptation techniques such as latent space [45, 46] or meta-learning
[47] can also be applied to further improve the performance of pre-trained
policies in the real environment. We plan to construct an adaptation strategy
to realize the sim-to-real control policy transfer for the lower extremity
exoskeleton robot in the near future.

##  6 Conclusion

In this work, we have presented a robust, RL-based controller for exoskeleton
squatting assistance with human interaction. A relatively lightweight lower
extremity exoskeleton is presented and used to build a human-exoskeleton
interaction model in the simulation environment for learning the robust
controller. The exoskeleton foot CoP information collected from the force
sensors is leveraged as a feedback for balance control and adversary
perturbations and uncertain human interaction forces are used to train the
controller. We have successfully demonstrated the lower extremity
exoskeleton’s capability to carry a human to perform squatting motions with a
moderate torque requirement and provided evidence of its effectiveness and
robustness. With the actuation of the ankle inversion/eversion joint, the
learned controllers are also capable of maintaining the balance within the
frontal plane under large perturbation or interaction forces. The success
demonstrated in this study has implications for those seeking to apply
reinforcement learning to control robots to imitate diverse human behaviors
with strong balance and increased robustness even when subjected to the larger
perturbations. In the near future, We plan to extend this framework to learn a
family of controllers that can perform a variety of human skills like sit-to-
stand, flat and inclined ground walking. Lastly, these learned controllers
will be deployed to the hardware through a sim-to-real transfer method and
experimental tests will be performed to validate the controllers’ performance.

## References

  * [1] Thomas Mergner and Vittorio Lippi.  Posture control—human-inspired approaches for humanoid robot benchmarking: Conceptualizing tests, protocols and analyses.  Frontiers in Neurorobotics, 12:21, 2018.
  * [2] Tristan Vouga, Romain Baud, Jemina Fasola, Mohamed Bouri, and Hannes Bleuler.  Twiice—a lightweight lower-limb exoskeleton for complete paraplegics.  In 2017 International Conference on Rehabilitation Robotics (ICORR), pages 1639–1645. IEEE, 2017.
  * [3] T. Zhang, M. Tran, and H. Huang.  Design and experimental verification of hip exoskeleton with balance capacities for walking assistance.  IEEE/ASME Transactions on Mechatronics, 23(1):274–285, 2018.
  * [4] Bing Chen, Hao Ma, Lai-Yin Qin, Fei Gao, Kai-Ming Chan, Sheung-Wai Law, Ling Qin, and Wei-Hsin Liao.  Recent developments and challenges of lower extremity exoskeletons.  Journal of Orthopaedic Translation, 5:26–37, 2016.
  * [5] Fay B Horak, Sharon M Henry, and Anne Shumway-Cook.  Postural perturbations: new insights for treatment of balance disorders.  Physical therapy, 77(5):517–533, 1997.
  * [6] C Bayon, AR Emmens, Maarten Afschrift, T Van Wouwe, AQL Keemink, H Van Der Kooij, and EHF Van Asseldonk.  Can momentum-based control predict human balance recovery strategies?  IEEE transactions on neural systems and rehabilitation engineering, 28(9):2015–2024, 2020.
  * [7] M Eva Mungai and Jessy Grizzle.  Feedback control design for robust comfortable sit-to-stand motions of 3d lower-limb exoskeletons.  IEEE Access, 2020.
  * [8] Vijaykumar Rajasekaran, Joan Aranda, Alicia Casals, and Jose L Pons.  An adaptive control strategy for postural stability using a wearable robot.  Robotics and Autonomous Systems, 73:16–23, 2015.
  * [9] George J. Salem and Christopher M. Powers.  Patellofemoral joint kinetics during squatting in collegiate women athletes.  Clinical Biomechanics, 16(5):424–430, 2001.
  * [10] Kay M Crossley, Wan-Jing Zhang, Anthony G Schache, Adam Bryant, and Sallie M Cowan.  Performance on the single-leg squat task indicates hip abductor muscle function.  The American journal of sports medicine, 39(4):866–873, 2011.
  * [11] Shuangyue Yu, Tzu-Hao Huang, Dianpeng Wang, Brian Lynn, Dina Sayd, Viktor Silivanov, Young Soo Park, Yingli Tian, and Hao Su.  Design and control of a high-torque and highly backdrivable hybrid soft exoskeleton for knee injury prevention during squatting.  IEEE Robotics and Automation Letters, 4(4):4579–4586, 2019.
  * [12] Gerald McGinty, James J Irrgang, and Dave Pezzullo.  Biomechanical considerations for rehabilitation of the knee.  Clinical biomechanics, 15(3):160–166, 2000.
  * [13] Stefan O Schrade, Katrin Dätwyler, Marius Stücheli, Kathrin Studer, Daniel-Alexander Türk, Mirko Meboldt, Roger Gassert, and Olivier Lambercy.  Development of varileg, an exoskeleton with variable stiffness actuation: first results and user evaluation from the cybathlon 2016.  Journal of neuroengineering and rehabilitation, 15(1):1–18, 2018\.
  * [14] Romain Baud, Jemina Fasola, Tristan Vouga, Auke Ijspeert, and Mohamed Bouri.  Bio-inspired standing balance controller for a full-mobilization exoskeleton.  In 2019 IEEE 16th International Conference on Rehabilitation Robotics (ICORR), pages 849–854. IEEE, 2019.
  * [15] R. Bionics.  Rex technology, 2020.  https://www.rexbionics.com/us/product-information/ Accessed April, 2020.
  * [16] Wandercraft.  Atalante, 2020.  https://www.wandercraft.eu/en/exo/ Accessed April, 2020.
  * [17] Visak CV Kumar, Sehoon Ha, Gregory Sawicki, and C Karen Liu.  Learning a control policy for fall prevention on an assistive walking device.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4833–4840. IEEE, 2020.
  * [18] Ming Xiong.  Research on the control system of the lower limb rehabilitation robot under the single degree of freedom.  In Applied Mechanics and Materials, volume 643, pages 15–20. Trans Tech Publ, 2014.
  * [19] Di Shi, Wuxiang Zhang, Wei Zhang, and Xilun Ding.  A review on lower limb rehabilitation exoskeleton robots.  Chinese Journal of Mechanical Engineering, 32(1):74, 2019.
  * [20] Mustafa Sinasi Ayas and Ismail Hakki Altas.  Fuzzy logic based adaptive admittance control of a redundantly actuated ankle rehabilitation robot.  Control Engineering Practice, 59:44–54, 2017.
  * [21] Jin Hu, Zengguang Hou, Feng Zhang, Yixiong Chen, and Pengfeng Li.  Training strategies for a lower limb rehabilitation robot based on impedance control.  In 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pages 6032–6035. IEEE, 2012.
  * [22] Kiran Karunakaran, Kevin Abbruzzese, Ghaith Androwis, and Richard Foulds.  A novel user control for lower extremity rehabilitation exoskeletons.  Frontiers in Robotics and AI, 7:108, 2020.
  * [23] A. Emmens, E. van Asseldonk, M. Masciullo, M. Arquilla, I. Pisotta, N. L. Tagliamonte, F. Tamburella, M. Molinari, and H. van der Kooij.  Improving the standing balance of paraplegics through the use of a wearable exoskeleton.  In 2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob), pages 707–712, 2018.
  * [24] Sung-Hee Lee and Ambarish Goswami.  A momentum-based balance controller for humanoid robots on non-level and non-stationary ground.  Autonomous Robots, 33(4):399–414, 2012.
  * [25] Twan Koolen, Sylvain Bertrand, Gray Thomas, Tomas De Boer, Tingfan Wu, Jesper Smith, Johannes Englsberger, and Jerry Pratt.  Design of a momentum-based control framework and application to the humanoid robot atlas.  International Journal of Humanoid Robotics, 13(01):1650007, 2016\.
  * [26] Xue Bin Peng, Glen Berseth, and Michiel Van de Panne.  Terrain-adaptive locomotion skills using deep reinforcement learning.  ACM Transactions on Graphics (TOG), 35(4):1–12, 2016.
  * [27] Xue Bin Peng and Michiel van de Panne.  Learning locomotion skills using deeprl: Does the choice of action space matter?  In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pages 1–13, 2017.
  * [28] Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne.  Deepmimic: Example-guided deep reinforcement learning of physics-based character skills.  ACM Trans. Graph., 37(4):143:1–143:14, July 2018.
  * [29] Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter.  Learning agile and dynamic motor skills for legged robots.  Science Robotics, 4(26), 2019.
  * [30] Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang-Wei Lee, Jie Tan, and Sergey Levine.  Learning agile robotic locomotion skills by imitating animals.  arXiv preprint arXiv:2004.00784, 2020.
  * [31] Ghaith J Androwis, Kiran Karunakaran, Erick Nunez, Peter Michael, Guang Yue, and Richard A Foulds.  Research and development of new generation robotic exoskeleton for over ground walking in individuals with mobility disorders (novel design and control).  In 2017 International Symposium on Wearable Robotics and Rehabilitation (WeRob), pages 1–2. IEEE, 2017.
  * [32] Erick H Nunez, Peter A Michael, and Richard Foulds.  2-dof ankle-foot system: Implementation of balance for lower extremity exoskeletons.  In 2017 International Symposium on Wearable Robotics and Rehabilitation (WeRob), pages 1–2. IEEE, 2017.
  * [33] Philippe Sardain and Guy Bessonnet.  Forces acting on a biped robot. center of pressure-zero moment point.  IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 34(5):630–637, 2004.
  * [34] Seunghwan Lee, Moonseok Park, Kyoungmin Lee, and Jehee Lee.  Scalable muscle-actuated human simulation and control.  ACM Transactions on Graphics (TOG), 38(4):1–13, 2019.
  * [35] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [36] Wenhao Yu, C Karen Liu, and Greg Turk.  Policy transfer with strategy optimization.  arXiv preprint arXiv:1810.05751, 2018.
  * [37] Fereshteh Sadeghi and Sergey Levine.  Cad2rl: Real single-image flight without a single real image.  arXiv preprint arXiv:1611.04201, 2016.
  * [38] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel.  Domain randomization for transferring deep neural networks from simulation to the real world.  In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 23–30. IEEE, 2017.
  * [39] Jeongseok Lee, Michael Grey, Sehoon Ha, Tobias Kunz, Sumit Jain, Yuting Ye, Siddhartha Srinivasa, Mike Stilman, and C. Karen Liu.  Dart: Dynamic animation and robotics toolkit.  In Journal of Open Source Software, volume 3(22), 500, 2018.
  * [40] David Baraff.  Fast contact force computation for nonpenetrating rigid bodies.  In Proceedings of the 21st annual conference on Computer graphics and interactive techniques, pages 23–34, 1994.
  * [41] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [42] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256, 2010.
  * [43] Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke.  Sim-to-real: Learning agile locomotion for quadruped robots.  arXiv preprint arXiv:1804.10332, 2018.
  * [44] Ioannis Exarchos, Yifeng Jiang, Wenhao Yu, and C Karen Liu.  Policy transfer via kinematic domain randomization and adaptation.  arXiv preprint arXiv:2011.01891, 2020.
  * [45] W. Yu, V. C. Kumar, G. Turk, and C. K. Liu.  Sim-to-real transfer for biped locomotion.  In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3503–3510, 2019.
  * [46] Ignasi Clavera, Anusha Nagabandi, Simin Liu, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn.  Learning to adapt in dynamic, real-world environments through meta-reinforcement learning.  In International Conference on Learning Representations, 2019.
  * [47] Wenhao Yu, Jie Tan, Yunfei Bai, Erwin Coumans, and Sehoon Ha.  Learning fast adaptation with meta strategy optimization.  IEEE Robotics and Automation Letters, 5(2):2950–2957, 2020.