the sagittal plane while the $3$D device also allows actuation in the frontal
plane. We also consider three different configurations of sensors: an inertial
measurement unit (IMU) that provides the COM velocity and acceleration, a
motor encoder that measures hip joint angles, and the combination of IMU and
motor encoder. In total, we train six different recovery policies with three
sensory inputs and two different actuation capabilities. For each sensor
configuration, we train a fall predictor using only sensors available to that
configuration.

SensorEffect shows the stability region for each of the six design
configurations. The results indicate that 3D actuation expands the stability
region in all directions significantly comparing to 2D actuation, even when
the external force lies on the sagittal plane. We also found that the IMU
sensor plays a more important role than the motor encoder, which suggests that
COM information is more critical than the hip joint angle in informing the
action for recovery. The recovery policy performs the best when combining the
IMU and the joint encoder, as expected.

####  4.2.8 Conclusion

We presented an approach to automate the process of augmenting an assistive
walking device with ability to prevent falls. Our method has three key
components : A human walking policy, fall predictor and a recovery policy. In
a simulated environment we showed that an assistive device can indeed help
recover balance from a wider range of external perturbations. We introduced
_stability region_ as a quantitative metric to show the benefit of using a
recovery policy. In addition to this, _stability region_ can also be used to
analyze different design choices for an assistive device. We evaluated six
different sensor and actuator configurations.

In this work, we only evaluated the effectiveness of using a recovery policy
for an external push. It would be interesting to extend our work to other
kinds of disturbances such as tripping and slipping. Another future direction
we would like to take is deploying our recovery policy on the real-world
assistive device. This would need additional efforts to make sure that our
recovery policy also can adjust for the differences in body structure of
users.

##  Chapter 5 Error-aware policy learning

###  5.1 Motivation

Humans exhibit remarkably large variations in gait characteristics during
steady-state walking as well as push recovery. Due to this, assistive device
controllers tuned for one individual often do not work well when tested on
another person. Typically, fine tuning a controller for each person has been
the go-to approach thus far, but this process can be a very time consuming
,tedious and unsafe for the user.

In this work, we introduce a novel approach to tackle sim-to-real problems in
which the environment dynamics has high variance and is highly unobservable.
While our approach is motivated by physical assistive robotic applications,
the method can be applied to other tasks in which many dynamic parameters are
challenging to model. We propose to train a policy explicitly aware of the
effect of unobservable factors during training, called an Error-Aware policy
(EAP). Akin to the high-level idea of meta learning, we divide the dynamical
environments into training and validation sets and ”emulate” a reality gap in
simulation. Instead of estimating the model parameters that give rise to the
emulated reality gap, we train a function that predicts the deviation (i.e.
error) of future states due to the emulated reality gaps. Conditioned on the
error predictions, the error-aware policies (EAPs) can learn to overcome the
reality gap, in addition to mastering the task.

The main application in this work is to learn an error-aware policy for
assistive device control, such as a hip-exoskeleton that helps the user to
recover balance during locomotion. From biomechanical data of human gait, we
model multiple virtual human walking agents, each varying in physical
characteristics as well as parameters that affect the dynamics such as joint
damping, torque limits, ground friction, and sensing and actuation delay. We
then train a single policy on this group of human agents and show that that
the learned EAP works effectively when tested on a different human agent
without needing additional data. We extend the prior work,
[kumar2019learning], that trained a control policy for push-recovery assistive
device for just one simulated human agent, and develop an algorithm that
enables the learned policy to transfer to other human agents with unseen
biomechanical characteristics.

We evaluate our approach on assistive wearable device by quantifying the
stability and gait characteristics generated by an unseen human agent wearing
the device with the trained EAP. We present a comprehensive study of the
benefits of our approach over prior zero-shot methods such as universal policy
(UP) and domain randomization (DR). We also provide results on some standard
RL environments, such as Cartpole, Hopper, 2D walker and a quadrupedal robot.

###  5.2 Method

We present a method to achieve the zero-shot transfer of control policies in
partially observable dynamical environments. We consider robotic systems and
environments with unobservable or unmeasurable model parameters, which make
building accurate simulation models difficult.

We present a novel policy architecture, an Error-Aware Policy (EAP), that is
explicitly aware of errors induced by unobservable dynamics parameters and
self-corrects its actions according to the errors. An EAP takes the current
state, observable dynamic parameters, and predicted errors as inputs and
generates corrected actions. We learn an additional error-prediction function
that outputs the expected error. Both the error-aware policy and the error-
prediction function, are iteratively learned using model-free reinforcement
learning and supervised learning.

####  5.2.1 Problem Formulation

We formulate the problem as Partially Observable Markov Decision Processes
(PoMDPs), $(S,O,A,P,R,\rho_{0},\gamma)$, where $S$ is the state space, $O$ is
the observation space, $A$ is the action space, $P$ is the transition
function, $R$ is the reward function, $\rho_{0}$ is the initial state
distribution and $\gamma$ is a discount factor. In our formulation, we make a
clear distinction between observable model parameters $\bm{\mu}$ and
unobservable parameters $\bm{\nu}$ of the agent and environment. Observable
quantities are parameters that can be easily measured such as masses or link
lengths, whereas unobserved quantities are challenging to estimate, such as
circuit dynamics or backlash. Therefore, both $\bm{\mu}$ and $\bm{\nu}$ affect
the transition function
$P(\mathbf{s^{\prime}}|\mathbf{a},\mathbf{s},\bm{\mu},\bm{\nu})$. Since we can
configure our simulator with both $\bm{\mu}$ and $\bm{\nu}$, we can randomly
sample $\bm{\mu}$ and $\bm{\nu}$ and create a list of $K$ different
environments
$\vc{D}=\\{(\bm{\mu}_{0},\bm{\nu}_{0}),(\bm{\mu}_{1},\bm{\nu}_{1}),\cdots,(\bm{\mu}_{K},\bm{\nu}_{K})\\}$,
but it is hard to obtain $\bm{\nu}$ at testing time. In this case, the
transition function will be abbreviated as
$P(\mathbf{s^{\prime}}|\mathbf{s},\mathbf{a},\bm{\mu})$.

Instead of estimating the values of unobserved quantities, we capture the
effect of these parameters by defining a metric called a _state-error_. When
transferring from one environment to another, the action $\mathbf{a}$ applied
at a given state $\mathbf{s}$ will produce different next states due to the
differences in both $\bm{\mu}$ and $\bm{\nu}$, in other words, a state-error.

Figure 5.1: Overview of An Error-aware Policy (EAP). An EAP takes the
“expected” future state error as an additional input. The expected error is
predicted based on the current state $\vc{s}$, observable parameters
$\bm{\mu}$, and an uncorrected action $\vc{a}$ that assumes zero error.

We hypothesize that a policy which is explicitly aware of the state-error
would be able to make better decisions by self-correcting its action. We call
this an error-aware policy $\pi(\mathbf{a}|\mathbf{s},\bm{\mu},\mathbf{e})$
(EAP), which takes in observable parameters $\bm{\mu}$ as well as the
“expected” future state error in a new environment $\mathbf{e}$ as input
(Figure 5.1).

We present a novel training methodology using model-free reinforcement
learning that involves learning two functions: an error-aware policy and an
error prediction function. First, we learn an error-aware policy that takes
the output of error prediction function $E$ as an input and has the ability to
generalize to novel environments in a zero-shot manner. Simultaneously, we
learn an error-prediction function, which takes as inputs the state
$\mathbf{s}$, an uncorrected action $\mathbf{a}$ and observable parameters
$\bm{\mu}$, and outputs the expected state error $\mathbf{e}$ when a policy
trained in one environment is deployed to a different one
$E:(\mathbf{s},\mathbf{a},\bm{\mu})\mapsto\mathbb{R}^{n}$. We will discuss
more details of training in the following sections.

11: Input: Environments
$\vc{D}=\\{(\bm{\mu}_{0},\bm{\nu}_{0}),\cdots,(\bm{\mu}_{K},\bm{\nu}_{K})\\}$

2: Pre-train $\pi(\mathbf{a}|\mathbf{s},\bm{\mu}_{0},\mathbf{e}=0)$ for
$P(\vc{s^{\prime}}|\vc{s},\vc{a},\bm{{\mu}}_{0})$ reference environment with
$\vc{e}=0$

3: while not done do

4: Sample an environment with $(\bm{\mu},\bm{\nu})$ from $\vc{D}$

5: for each policy update iteration do

6: Initialize buffer $\vc{B}=\\{\\}$

7: Update an error function $E$ using Algorithm 7

8: $\vc{B}$ = Generate rollouts using Algorithm 6

29: Update policy $\pi$ using $\vc{B}$ with PPO.

10: end for

11: end while

12: return $\pi(\mathbf{a},|\mathbf{s},\bm{\mu},\mathbf{e})$

Algorithm 5 Train an Error Aware Policy.

####  5.2.2 Training an Error-aware Policy

Training Procedure. The training process of an error-aware policy is
summarized in Algorithm 5. Assume that we have an oracle error function
$E(\mathbf{s},\mathbf{a},\bm{\mu})$ that outputs the expected state error in a
novel environment, which will be explained in the following section. First,
the policy is pre-trained to achieve the desired behavior only in the
reference environment $(\bm{\mu}_{0},\bm{\nu}_{0})$ assuming there is no state
error, $\pi(\mathbf{a}|\mathbf{s},\bm{\mu}_{0},\mathbf{e}=0)$. Once the policy
is trained in the reference environment, we sample dynamics parameters
$\bm{\mu}_{i}$ and $\bm{\nu}_{i}$ ($i>0$) uniformly from the data set $\vc{D}$
and evaluate the EAP in this new environment. The policy parameters are
updated using a model-free reinforcement learning algorithm, Proximal Policy
Optimization [schulman2017proximal]. Sampling new testing environments and
updating policy parameters are repeated until the convergence.

2

1: Input: Observable dynamics parameters $\bm{\mu}$, Transition function $P$,
Current policy $\pi$ and error function $E$, Replay buffer $\vc{B}$

2: Sample state $\vc{s}$ from initial state distribution $\rho_{0}$

3: while not done do

4: $\vc{a}\sim\pi(\vc{a}|\vc{s},\bm{{\mu}},\vc{e}=0)$ // original action

5: $\vc{e}=E(\vc{s},\vc{a},\bm{{\mu}})$ // predicted error

6: $\hat{\vc{a}}\sim\pi(\vc{a}|\vc{s},\bm{{\mu}},\vc{e})$ // error-aware
action

7: $\vc{s}^{\prime}\sim P(\vc{s}^{\prime}|\vc{s},\hat{\vc{a}},\bm{\mu})$

8: $r=R(\vc{s},\hat{\vc{a}})$

9: $B=B\cup\\{(\vc{s},\hat{\vc{a}},r,\vc{s}^{\prime},\bm{\mu})\\}$

110: $\vc{s}=\vc{s}^{\prime}$

11: end while

12: return $B$

Algorithm 6 Generate Rollouts

Rollout Generation. A roll-out generation procedure is described in Algorithm
6. Given a state $\mathbf{s}$ in this environment $(\bm{\mu},\bm{\nu})$, we
query an action from policy $\pi$ as if the policy is being deployed in the
reference environment with $\mathbf{e}=0$. This action $\vc{a}$ is fed into
the error function $E$ which predicts the expected state error in this
environment, then the state error is passed into the error-aware policy to
query a corrected action $\hat{\vc{a}}$ which will be applied to the actual
system. The task reward $R(\vc{s},\vc{a})$ guides the policy optimization to
find the best “corrected” action that maximizes the reward.

6

1: Input: Reference environment with $\bm{\mu_{0}}$

2: Input: Target environment with $\bm{\mu}$

3: Input: Replay Buffer $\vc{B}$

4: Input: Dataset $\vc{Z}$

15: Input: Error Horizon $T$

6: while not done do

7: Sample the initial state $\vc{s}_{0}^{0}$ from $\vc{B}$

28: $\vc{s}^{0}=\vc{s}_{0}^{0}$

9: for $t=0:T-1$ do

10: // Simulation in Reference Env

11: $\vc{a}^{t}_{0}\sim\pi(\vc{a}|\vc{s}_{0}^{t},\bm{\mu}_{0},\vc{e}=0)$

312: $\vc{s}^{t+1}_{0}\sim P(\vc{s}^{t}_{0},\vc{a}^{t}_{0},\bm{\mu}_{0})$

13: // Simulation in Validation Env

14: $\vc{a}^{t}\sim\pi(\vc{a}|\vc{s}^{t},\bm{\mu},\vc{e}=0)$

415: $\vc{s}^{t+1}\sim P(\vc{s}^{t},\vc{a}^{t},\bm{\mu})$

16: end for

17:
$\vc{Z}=\vc{Z}\cup\\{(\vc{s^{0}},\vc{a^{0}},\vc{s}^{T},\vc{s}_{0}^{T},\bm{\mu})\\}$

18: end while

519: minimize the $L(\phi)$ in Eq. 5.1 using $\vc{Z}$.

20: return $\phi$

Algorithm 7 Train an Error Prediction Function.

####  5.2.3 Training an Error Function

In reality, we do not have an oracle error function that can predict the next
state due to the lack of unobservable parameters $\bm{\nu}$. To this end, we
will learn this function simultaneously with EAP, by splitting the dataset
$\vc{D}$ into the training and validation sets. Similar to training
methodology followed in meta-learning algorithms, we repeatedly apply the
trained policy into sampled environments from the validation set. Because our
nominal behavior is pre-trained in the reference environment
$(\bm{\mu}_{0},\bm{\nu}_{0})$, we compute the errors by measuring the
differences in the reference environment $(\bm{\mu}_{0},\bm{\nu}_{0})$ and the
validation environment $(\bm{\mu},\bm{\nu})$:
$\bm{e}=(\vc{\bar{s}}^{\prime}-\vc{s}^{\prime})\in\mathbb{R}^{n}$, generated
by two dynamic models ${P}(\vc{s}^{\prime}|\vc{s},\vc{a},\bm{{\mu}}_{0})$ and
$\bar{P}(\vc{s}^{\prime}|\vc{s},\vc{a},\bm{\mu})$.

Horizon of Error Prediction. In practice, we found that the error accumulated
during one step is often not sufficient to provide useful information to the
EAP. To overcome this challenge, we take the state in the collected trajectory
and further simulate it for $T$ steps in both the reference environment
${P}(\vc{s}^{\prime}|\vc{s},\vc{a},\bm{{\mu}}_{0})$ and the validation
environment $\bar{P}(\vc{s}^{\prime}|\vc{s},\vc{a},\bm{\mu})$. We provide
analysis on the effect of horizon length from $T=1$ to $T=8$ in the Section
LABEL:sec:experiment.

Loss Function. Since the differences between the two dynamical environments
reflects the reality gap caused by unobservable parameters, the error
prediction function $E$ enables us to learn the effect of the unobserved
parameters captured through the state error. We train our error prediction
function $E$ to learn this “emulated” sim-to-real gap by minimizing the
following loss:

$L(\bm{\phi})=\sum_{(\vc{s^{0}},\vc{a^{0}},\vc{s}^{T},\vc{s}_{0}^{T},\bm{\mu})\in\vc{Z}}||E(\vc{s}^{0},\vc{a}^{0},\bm{\mu})-(\vc{s}_{0}^{T}-\vc{s}^{T})||^{2},$
(5.1)

where $\vc{Z}$ is the collected dataset and $\bm{\phi}$ is the parameters or
the neural net representing $E$. Algorithm 7 summarizes the training
procedure.

Figure 5.2: Left : A full state error representation input into the policy vs
Right : Projected error representation as an input to the policy

Reduced Representations. We experiment with two different representations of
the error input to the policy. First, we input the full state error
$\vc{e}=\vc{s}_{0}^{T}-\vc{s}^{T}$(with the same dimension as the state)
approximated by a MLP neural network, into the policy. Second, we use a
network architecture with an information bottle neck, as illustrated in Figure
5.2, and input the latent representation $\vc{e}_{p}$ into the policy. The
same loss function $L$ is used to train both the functions.

###  5.3 Results

We design experiments to validate the performance of error-aware policies. We
aim to answer the following research questions.

  1. 1.

Does an EAP show better zero-shot transfer on unseen environments compared to
the baseline algorithms?

  2. 2.

How does the choice of hyperparameters affect the performance of an EAP?

####  5.3.1 Baseline Algorithms

We compare our method with two baselines commonly used for sim-to-real policy
transfer, Domain Randomization (DR)[openai2018learning, Pengdr] and Universal
Policy (UP) [yu2017preparing]. DR aims to learn a more robust policy for zero-
shot transfer, by training with randomly sampled dynamics parameters (in our
case, both $\bm{\mu}$ and $\bm{\nu}$). UP extends DR by taking dynamics
parameters as additional input. UP often transfer to target environments
better than DR, but it explicitly requires to know dynamics parameters, where
$\bm{\nu}$ is assumed to be unobservable in our scenario. We did not compare
EAPs against meta-learning algorithms [belkhale2020modelbased,
finn2017modelagnostic, ignasi], which require additional samples from the
validation environment.

####  5.3.2 Tasks

Table 5.1: Tasks and Network Architectures

Task | Observable Params. $\bm{\mu}$ | Unobservable Params. $\bm{\nu}$ | Net. Arch. | Err. Dim. $|\vc{e}_{p}|$
---|---|---|---|---
Assitive Walking | mass, height, leg length, and foot length | joint damping, max torques, PD gains and delay | (64, 32) | 6
Aliengo | PD gains, link masses | sensor delay, joint damping, ground friction | (64, 32) | 6
Cartpole | pole length, pole mass, cart mass | joint damping, joint friction | (32, 16) | 2
Hopper | thight mass, foot mass, shin length | joint damping, ground friction | (32, 16) | 4
Walker 2D | link masses, shin length | sensing delay, joint damping, ground friction | (64, 32) | 5

Table 5.2: Ranges of variation for observable parameters during training and
testing in the assistive walking task. Observable Params. $\bm{\mu}$
---
Parameter | Training range | Testing range
Mass | [45,76] kg | [55,95] kg
Height | [143,182] cm | [155,197] cm
Leg-length | [70,88] cm | [80,95] cm
foot length | [21,24] cm | [24,26] cm
Table 5.3: Ranges of variation for unobservable parameters during training and
testing in the assistive walking task. Unobservable Params. $\bm{\nu}$
---
Parameter | Training range | Testing range
Joint damping | [0.3,0.6] | [0.5,0.8]
Max torques | [120,180] | [155,200]
PD gains (P,D) | [(500,25),(750,50)] | [(650,30),(800,50)]
Delay | [30,60] ms | [45,70] ms

We evaluate the performance of error-aware policies on five different tasks.
The first task is about push-recovery of an assistive walking device for
simulated humans, inspired by the work of Kumar et al [kumar2019learning]. The
second task is locomotion of a quadrupedal robot, Aliengo
Explorer[UnitreeAliengo]. The rest three tasks are CartPole, Hopper, and
Walker2D, which are from the OpenAI benchmark suite [GymAI].

##### Assistive walking device for push recovery

Figure 5.3: Five different test subjects for the assistive walking experiment
with varying height, mass, leg length and foot length from the biomechanical
gait dataset [dataset2018].

In this task, the goal is to learn a policy for an assistive wearable device
(i.e. exoskeleton) to help a human recover balance after an external push is
applied. (inset figure). We use a hip exoskeleton that applies torques in
2-degrees of freedom at each hip joint. Our algorithm begins by training $15$
human agents using public biomechanical gait data [dataset2018] to walk in a
steady-state gait cycle, similar to the approach presented in
[peng2018deepmimic]. The $15$ agents vary in mass, height, leg length, and
foot length according to the biomechanical data used to train their
corresponding policies, which formulate the four-dimensional observable
parameters $\bm{\mu}$ (Figure 5.3). We also vary each human agent’s joint
damping, maximum joint torques, PD gains, and sensory delay as the four
dimensional unobservable parameters $\bm{\nu}$. We split the $15$ human agents
into $10$ for the training set and $5$ as the testing set.

Human Behavior Modeling. First, we capture the human behavior by training a
human-only walking policy $\pi_{h}$ that mimics the reference motion which
frames are denoted as $\bar{\vc{q}}$. Each human model has $23$ actuated
joints along with a floating base. The state space has $53$ dimensions,
$\vc{s}_{h}=[\vc{q},\vc{\dot{q}},\vc{v}_{com},\bm{\omega}_{com},\psi]$, which
represent joint positions, joint velocities, linear and angular velocities of
the center of mass, and a phase variable $\psi$ that indicates the target
frame in the reference biomechanical gait cycle. The action $\vc{a}$ is
defined as the offset to the reference biomechanical joint trajectory
$\bar{\vc{q}}(\psi)$, which results in the target angles:
$\vc{q}^{target}=\bar{\vc{q}}+\vc{a}$. The reward function encourages
mimicking the reference motions from public biomechanical data:

$R_{human}(\vc{s}_{h},\vc{a}_{h})=w_{q}(\vc{q}-\bar{\vc{q}})+w_{v}(\vc{\dot{q}}-\vc{\bar{\dot{q}}})+w_{c}(\vc{c}-\bar{\vc{c}})+w_{p}(\vc{p}-\bar{\vc{p}})-w_{\tau}||\bm{\tau}||^{2},$
(5.2)

where the terms include the reference joint positions $\vc{\bar{q}}$, joint
velocities $\vc{\bar{\dot{q}}}$, end-effector locations $\vc{\bar{p}}$,
contact flags $\vc{c}$, and the joint torques $\bm{\tau}$. During training, we
exert random forces to the agent during policy training. Each random force has
a magnitude uniformly sampled from $[0,800]\ N$ and a direction uniformly
sampled from [-$\pi/2$,$\pi/2$], applied for $50$ milliseconds on the agent’s
pelvis in parallel to the ground. The maximum force magnitude induces a
velocity change of roughly $0.6-0.8$ m/sec. This magnitude of change in
velocity is comparable to experiments found in biomechanics literature such as
[Wang2014],[Agarwal] and [hof2010balance]. We also randomize the time when the
force is applied within a gait cycle. The forces are applied once at a
randomly chosen time in each trajectory rollout. Similar to
[kumar2019learning], we enforce joint torque constraints and introduce sensing
delays during training to prevent the human agent to adapt to external
disturbance really well.

MDP Formulation. Once the human agents are trained, we begin learning the
push-recovery EAP for the assistive device. The objective is to stabilize the
human gait from external perturbations. The $17$ dimensional state of robot is
defined as
$\vc{s}_{e}=[{\bm{\omega}},\bm{\alpha},\vc{\ddot{x}},\vc{q}_{hip},\dot{\vc{q}}_{hip}]$,
which comprises angular velocity, orientation, linear acceleration, hip joint
positions, and hip joint velocity. The four dimensional action $\vc{a}_{e}$
consists of torques at two hip joints. The reward function maximizes the
quality of the gait while minimizing the impact of an external push.

$R_{exo}(\vc{s}_{h},\vc{s}_{e},\vc{a}_{e})=R_{human}(\vc{s}_{h})-w_{1}\|\vc{v}_{com}\|-w_{2}\|\bm{\omega}_{com}\|-w_{3}\|\vc{a}_{e}\|,$
(5.3)

where $R_{human}$ is defined in equation 5.2 , and $\vc{v}_{com}$ and
$\bm{\omega}_{com}$ are the global linear and angular velocities of the
pelvis. The last term penalizes the torque usage. We use the same weight
$w_{1}=2.0$, $w_{2}=1.2$ and $w_{3}=0.001$ for all our experiments.

##### Quadrupedal Locomotion

In our second task, we learn a control policy that generates a walking motion
for a quadrupedal robot, Aliengo Explorer [UnitreeAliengo]. For this task, the
$17$ observable parameters ($\bm{\mu}$) are PD gains of the joints, link and
root masses and the $10$ unobservable parameters $\bm{\nu}$ include sensing
delay, joint damping of thigh and knee joints and ground friction. The
39-dimensional state space consists of torso position and orientation and
corresponding velocities, joint position and velocities, foot contact variable
that indicates when each foot should be in contact with the ground, while the
$12$-dimensional action space consists of joint velocity targets which is fed
into a PD controller that outputs torques to each joint.

The reward function is designed to track the target motion that walks at $0.8$
m/s:

$r(\vc{s},\vc{a})=w_{1}e^{-k_{1}*(\vc{q}-\vc{\bar{q}})}+w_{2}e^{-k_{2}*(\vc{\dot{q}}-\vc{\bar{\dot{q}}})}+w_{3}\min(\dot{x},0.8)+\sum_{i=1}^{4}||c_{i}-\bar{c}_{i}||^{2}.$
(5.4)

In this equation, the first term encourages to track the desired joint
positions, the second term is to track the desired joint velocities, the third
term is for matching the forward velocity $\dot{x}$ to a target velocity of
$0.8$ m/s. and the four term tracks the predefined contact flags. We use the
same weight $k_{1}=35$,$w_{1}=0.75$, $k_{2}=2$,$w_{2}=0.20$, and $w_{3}=1.0$
for all experiments.

##### OpenAI Environments

We test our method on three OpenAI environments: CartPole, Hopper and
Walker2D. While using the same state spaces, action spaces, and the reward
functions described in the benchmark [GymAI], we additionally define
observable and unobservable dynamics parameters as follows:

  1. 1.

Cartpole. Observable parameters $\bm{\mu}\in R^{3}$ includes the length of the
pole, the mass of the pole, and the mass of cart. Unobservable parameters
$\bm{\nu}\in R^{3}$ include the damping at the rotational joint, the friction
at the rotational joint, and the friction at the translational joint.

  2. 2.

Hopper. Observable parameters $\bm{\mu}\in R^{3}$ include the mass of the
thigh and foot and the length of the shin bodynode. Unobservable parameters
$\bm{\nu}\in R^{3}$ include joint damping of shin and foot joints and ground
friction.

  3. 3.

Walker 2D. Observable parameters $\bm{\mu}\in R^{6}$ include the masses of
thigh and foot for both legs, the mass of pelvis, and the length of shin.
Unobservable parameters $\bm{\nu}\in R^{4}$ include joint damping of foot
joints, the delay in observation, and ground friction.

####  5.3.3 Zero-shot Transfer with EAPs

(a) Hopper

(b) Walker 2D

(c) Quadrupedal Locomotion

(d) Assistive Walking

Figure 5.4:  Learning curves for four tasks. The number of samples for EAP
include the ones generated for training an error function.

In this section, we compare the zero-shot transfer of error-aware policies
against two other baseline algorithms, Domain Randomization (DR) and Universal
Policies (UP).

Learning Curves. First, we compare the learning curves of the EAP, DR, and UP
approaches on four selected tasks in Figure 5.4. We set the same ranges of the
observable and unobservable parameters for all three algorithms. In our
experience, EAPs learn faster than DR and UP for three tasks, the Hopper,
Walker2D, and assistive walking tasks, while showing comparable performance
for the quadrupedal locomotion task. Note that, to make the comparison fair to
baselines, we also include the samples for training error functions (Algorithm
7) when we evaluate the performance of EAPs. We do not include the experiment
on the CartPole environment for brevity but the EAP outperforms the baselines
as well

Zero-shot Transfer.

Figure 5.5: Comparison of EAP and baselines DR and UP. The error bars
represent the variation in the average return of the policy in the target
environment when trained with 4 different seeds.

Then we evaluate the learned policies on unseen validation environments, where
their dynamics parameters $\bm{\mu}$ and $\bm{\nu}$ are sampled from the
outside of the training range. We conduct the experiments for the CartPole,
Hopper, Walker2D and quadrupedal locomotion tasks and compare the _normalized_
average returns (the average return divided by the maximum return). The
results are plotted in Figure 5.5, which indicate that EAP outperforms DR by
$60$% to $116$% and UP by $12$% to $77$%. Note that UP may perform well for
the real-world transfer due to the lack of the unobservable parameters. We
also observe that UP is consistently better than DR by being aware of the
dynamics parameters, $\bm{\mu}$ and $\bm{\nu}$, which meets our expectation.

Figure 5.6: Average stability region in five test subjects. The results
indicate the better zero-shot transfer of EAP over DR and UP.

For evaluating the zero-shot transfer for the assistive walking task, we
define an additional metric “stability region”, which depicts the ranges of
maximum perturbations in all directions that can be handled by the human with
the EAP-controlled exoskeleton. We train policies for 10 training human
subjects and test the learned policies for 5 new human subjects. Figure 5.6
compares the average performance of EAP with DR and UP. The larger area of
stability region indicates that EAP significantly outperforms two baselines.
The ranges of variation for observable and unobservable parameters during
training and testing phases are included in tables 5.2 and 5.3.

####  5.3.4 Ablation study

We further analyze the performance of EAPs by conducting a set of ablation
studies. We studied four categories of parameters: choices of observable
parameters, reference dynamics, error prediction horizons, and error
representations.

Figure 5.7: Ablation study with choosing different observable parameters as
$\bm{\mu}$. The result indicates that our approach (EAP) shows more reliable
zero-shot transfers for all different scenarios.

Choice of Observable and Unobservable Parameters. We check the robustness of
EAPs by testing with different choices of observable and unobservable
parameters. We randomly split the parameters into $\bm{\mu}$ and $\bm{\nu}$
and test three different splits. Figure 5.7 shows the stability regions for
all three algorithms for three different scenarios. In all cases, EAPs are
more robust than the baseline algorithms.

Figure 5.8: Ablation study with different reference dynamics. The results
indicate that our algorithm is robust against the choice of different
references.

Choice of Reference Dynamics. In this study, we analyze the effect of choosing
three different reference dynamics
$P(\vc{s}^{\prime}|\vc{a},\vc{s},\bm{\mu}_{0},\bm{\nu}_{0})$ on the
performance of EAP. We randomly choose three different human agents as the
reference dynamics and follow the learning procedure of EAPs to train three
different policies. These policies are then deployed on the same test subjects
along with UP and DR policies. Figure 5.8 shows that all the EAPs
outperforming the baselines by having larger stability regions, although EAPs
have slightly larger variances.

Figure 5.9: Ablation study with different parameter setting for EAP training.

Horizon of Error Prediction. As we motivated in Section 5.2.3, one step error
might be too subtle to inform the learning of EAPs and we may need $T$ step
expansion to enlarge them. We studied the effect of the error prediction
horizon $T$ in Algorithm 7 by varying its value from $T=1$ to $T=8$ for the
assistive walking task. Figure 5.9 shows the normalized average return over T
gradually changes over the different values of $T$ and peaks at $T=5$.
Therefore, we set $T=5$ for all the experiments.

The error representation. We also compare the effect of the error
representation. Figure 5.9 also plots the normalized average returns of the
unprojected errors (blue) and projected errors (orange), where projected
errors show slightly better performance for all the different $T$ values.

###  5.4 Hardware experiments

Figure 5.10: Unitree’s A1 quardupedal robot.

Although the primary inspiration for developing this algorithm was to enable
transfer of control policies for assistive devices, the proposed zero-shot
transfer approach is applicable to any robot whose dynamics are partial
observable due to challenges in measuring certain parameters. To validate the
effectiveness of EAP algorithm on a real robotic system, we use the popular
quadrupedal robot A1 from Unitree [UnitreeA1] robotics, shown in 5.10, as our
testbed. A common, well established, approach while working with real robotic
systems is to first perform a thorough system identification of the robot’s
kinematic and dynamic properties by collecting real world data. However,
standard system identification methods fail when a robot is tested in an
environment from which no prior data was available. For example, when data is
collected on regular ground but the robot is tested on a ground with different
contact parameters such as a soft foam mat, the resulting variation in
dynamics is not captured by the identified parameters. As a result, control
policies trained in simulation fail when tested on a novel environment in the
real world. Using EAP, we demonstrate that a policy trained using our approach
is able to overcome differences in the environments it is being tested in. We
take the following approach to illustrate this :

  1. 1.

We first perform a system identification of A1 robot parameters using data
collected over carpeted ground.

  2. 2.

Then we define two tasks for the quadrupedal robot - In-place walking and
walking forward. For each of these tasks, we define a set of observable and
unobservable parameters and train EAP along with policies using baseline
algorithms - UP and DR.

  3. 3.

We compare the performance of each of these policies on three novel
environments from which no data for system identification was collected. First
environment uses a soft foam mat as ground. Second, we add an additional
unknown mass to the robot’s torso. Third, we test the robot with additional
mass using soft foam mat as ground.

####  5.4.1 Data collection, simulation environment and system identification

##### Data collection

The A1 robot is controlled using a position command which is converted into
torques using a simple PD-controller. It is equipped with joint angle sensors,
Inertial-Measurement Unit (IMU) and a foot contact sensor. Using the joint
angle sensors we compute both joint angle and joint velocities. We can also
compute the robots torso’s orientation and rate of change of orientation using
the IMU sensor.

The data collection process on the real A1 robot involves generating two kinds
of data:

  1. 1.

We first suspend the robot in air, thus removing ground contact forces being
applied to the robot, and apply a variety of sinusoidal and step commands to
the robot. During this process, we make sure that the robots torso remains
fixed as much as possible. We collect joint position and velocity responses
from the robot.

  2. 2.

Next, we script simple sinusoidal commands with different frequencies and
amplitudes that the robot legs can follow while being in contact with the
ground (Note, we stick to simple commands because it is challenging to script
commands for the robot in contact with the ground while ensuring safety). We
collect joint position, velocities , the torso’s orientation , angular
velocity (computed using simple finite-difference method).

We collect 60 trajectories on the robot each of length 10 seconds. The sensors
are sampled at 30 Hz, which gives us a total of $60,000$ data points. These
form dataset $\bm{D}$ , each trajectory $\bm{\tau}$ is consists of the command
sent to the robot $\bm{a}$ and the state of the robot $\bm{s}$ (which includes
the joint positions,velocities, torso position and orientation) for length of
time $\bm{T}$.

$\displaystyle\bm{D=\\{(\tau_{1},),(\tau_{2})..(\tau_{n})\\}}$
$\displaystyle\bm{\tau_{i}=\\{(s_{1},a_{1}),(s_{2},a_{2})...(s_{T},a_{T})\\}}$

##### Simulation environment and system identification

We use DART physics engines [DART] to simulate the robot. In total, the robot
has $18$ degrees of freedom, $12$ controllable joints of the legs and $6$ free
degrees of freedom of the torso. The simulation runs at 500 Hz, however we use
a control frequency of 20 Hz. In simulation, we enforce a torque limit of 30
$N/m$ based on the torque capabilities of the real robot. In our experiments
with the real robot, we measured a sensor delay of 20-30 $ms$, we incorporate
this sensor delay into our simulation framework as well. The feet of A1 robot
are made from deformable rubber material which acts like a spring-damper when
there is contact, to model this interaction we use hunt-crossley contact model
[hunt1975coefficient, seth2018opensim]. Hunt-crossley model comprises of
parameters such as stiffness and friction which allows us to control the
softness of the contact. To identify a simulation model from real world data,
we choose $8$ dynamical parameters : The proportional and derivative gains of
the low-level controller $\bm{K_{p},K_{d}}$ on the robot. DART also allows us
to write custom low-level controllers (such as PD controllers), which enables
users to design these controllers to compute the torques. Since, the PD
controller gains affect the dynamic response of the robot joint legs, we
include these parameters as optimization variables. We also include joint
damping $\bm{\beta}$, contact stiffness $\bm{\kappa}$, ground friction
$\bm{\gamma}$, mass of torso $\bm{m_{torso}}$, mass of thigh link
$\bm{m_{thigh}}$ and mass of the calf link $\bm{m_{calf}}$. This makes the
vector containing these variables $\bm{\eta\in R^{8}}$.

$\bm{\eta=\\{K_{p},K_{d},\beta,\kappa,\gamma,m_{torso},m_{thigh},m_{calf}\\}}$
(5.5)

For optimization, we use Covariance Matrix Adaptation (CMA-ES) [Hansen16a]
method to find the best set of parameter values in simulation that matches
real world trajectories by minimizing the cost function described in 5.6.

$\bm{J=\frac{1}{N}\min_{\eta}\sum_{n=1}^{N}\sum_{t=0}^{T}||s^{\prime}_{t}-s_{t}||^{2}}$
(5.6)

Where $s_{t}$ is the states collected in real world and $s^{\prime}_{t}$ are
the states observed in simulation. The cost function minimizes the average
error generated by $N$ state trajectories each of $T$ time steps in simulation
when compared to real world data when the same actions are applied to both.
Once the optimized parameters are obtained, we use this as the base
environment in simulation.

Table 5.4: Tasks and Network Architectures on the real robot

Task | Observable Params. $\bm{\mu}$ | Unobservable Params. $\bm{\nu}$ | Net. Arch. | Err. Dim. $|\vc{e}_{p}|$
---|---|---|---|---
In-Place Walking | PD-gains, link masses | joint damping, contact params , delay | (64,64,64) | 6
Walking forward | PD gains, link masses | sensor delay, joint damping, contact params | (64,64,64) | 6
Walking forward w unknown mass | PD-gains | joint damping, delay, torso mass | (64,64,64) | 6

###  5.5 In-place walking

We define an in-place walking task for a quadrupedal robot in which the center
of mass of the robot remains in the same place while the legs follow a walking
motion. Inspired by the lower level control framework presented in
[Leeeabc5986], we first define a end-effector reference trajectory for each
leg and our action space is defined as delta end-effector cartesian positions
from the pre-defined reference trajectory. We then use analytical inverse
kinematics to compute the corresponding joint angles for each leg. For in-
place walking, since there is no movement in the x-y plane of the robot, the
reference trajectory only consists of motion in the z-axis which is
perpendicular to the ground. The reference trajectory is defined in the
equation below, where $A$ is the amplitude, $f$ is the frequency and $t$ is
the time. When the cartesian $z$ is below zero or the ground plane the value
just remains zero.

$z=\begin{cases}Asin(2\pi ft),&\text{if z $\geq$ 0}\\\
0,&\text{otherwise}\end{cases}$

The action space is 12-dimensional ($R^{12}$) which comprises of residual
cartesian position for each leg end-effector ($R^{3}$). So, the output of the
neural network policy is added to the reference trajectory to compute the
target leg positions. The state space consists of the angular velocity of
torso $R^{3}$ computed using the gyroscope. The angular position $R^{3}$ of
torso, joint angles $R^{12}$ and binary foot contact information for each leg
which indicates if the foot is in contact or not $R^{4}$. An additional phase
variable $\phi(t)$ is included as part of the state to indicate the phase of
the gait cycle the robot is in. In total, the state vector is 23 dimensional
$R^{23}$. The reward function , described in 5.7 encourages the policy to
generate actions that penalizes center of mass velocity of the robot while
following the reference trajectory $\bar{q}$ of the end-effector of each leg.

$r(\vc{s},\vc{a})=w_{1}e^{-k_{1}*(\vc{q}-\vc{\bar{q}})}+w_{2}e^{-k_{2}*(\vc{\dot{c}})}-w_{3}||a||^{2}$
(5.7)

Where $w_{1}=0.5$, $w_{2}=0.5$ , $k_{1}=20$, $k_{2}=10$ and $w_{3}=1e^{-3}$
During training, we consider the PD-gains , the mass of the robot links as
observable parameters $\vc{\mu}$. In our preliminary experiments, we found
that for simple tasks, these quantities can be identified by standard system
identification procedures. However, quantities like joint damping, sensor
delays and ground contact parameters are considered unobservable parameters
$\vc{\nu}$. We train EAP, UP and DR with these parameters as the set of
observable and unobservable quantities.

The neural network policy is represented as a multi-layered perceptron (MLP)
with three hidden layers of 64 neurons each. The error function is trained
with a horizon length of five $T=5$ and the projected error dimension is six
$R^{6}$.

###  5.6 Walking forward

In this task, we train a policy to walk forward at a certain velocity while
maintaining the yaw orientation. The framework for training the policy is
similar to in-place walking task. A pre-defined reference trajectory is
defined for each leg end-effector and a neural network policy outputs delta
cartesian positions for each leg. The reference trajectories are defined as
follows :

$z=\begin{cases}Asin(2\pi ft-\frac{\pi}{2}),&\text{if z $\geq$ 0}\\\
0,&\text{otherwise}\end{cases}$

$x=Asin(2\pi ft),y=0$

The reward function encourage forward walking at a velocity of $v=0.6m/s$
while ensuring that the end-effectors follow the pre-defined reference
trajectory. The reward function also encourages the policy to walk in a
straight path by penalizing the yaw orientation $\psi$.

$r(\vc{s},\vc{a})=w_{1}e^{-k_{1}*(\vc{q}-\vc{\bar{q}})}+w_{2}e^{-k_{2}*\psi}+\min(\dot{x},0.6)-w_{3}||a||^{2}$
(5.8)

Where $w_{1}=0.5$,$k_{1}=20$, $w_{2}=0.6$, $k_{2}=10$ and $w_{3}=1e^{-3}$. The
state space, action space, observed, unobserved parameters and test
environment remains the same as in-place walking task. The parameters of the
policy and the error function remains the same as the previous task as well.

###  5.7 Evaluation on real robot

We test policies on five environments : A training environment and four test
environments.

  1. 1.

Base training environment : The same environment from which data for system
identification was collected (robot moving on carpeted ground).

  2. 2.

Foam mat environment (T1): The robot is deployed on a soft foam mat with
different contact parameters than over ground.

  3. 3.

Additional mass environment (T2): Additional unknown mass is added to the
robot’s torso.

  4. 4.

Foam mat with additional mass environment (T3): Robot is tested with
additional mass and deployed over a foam mat.

Owing to the difficulty in computing the rewards in the real world. We use a
surrogate metric to evaluate the policies in the real world. For in-place
walking task, we collect the joint angles when the policy is being deployed
and compare how well the joint angles track the reference trajectory. This
quantity is computed using the term $w_{1}e^{-k_{1}*(\vc{q}-\vc{\bar{q}})}$ in
equation 5.7. For walking forward task, we just use the straight line distance
travelled by the robot as the surrogate metric, this is measured using a
simple measuring tape.

##### Performance in base training environment

In the base training environment we notice that for both in-place walking as
well as walking forward task all methods (EAP,DR and UP) show similar
performance. The results are illustrated in 5.12(a) and 5.12(b) where the task
performance for the base environment is similar. This corroborates with
findings of prior work [xie2020dynamics] which suggested that for simple
tasks, carefully randomizing parameters is sufficient for sim-2-real transfer
for quadrupedal robots. Since, the policies are being tested in the
environment from which data for system identification was collected, the
policies are able to transfer well.

##### Performance in test environments

Figure 5.11: Comparison of contact profile generated by ground and soft foam
mat

In our first test environment (T1), we change the surface on which the robot
performs the task, with the aim of changing the contact parameter values. The
differences in the contact profile generated by the two surfaces is
illustrated in figure 5.11. We notice that for in-place walking task, all
methods perform similarly 5.12(a). However, in the walking forward task
5.12(b), EAP beats both the baselines DR and UP by walking forward for a
longer straight-line distance on the mat with an average of $2.3$m whereas
both DR and UP move less than $2$m. We notice that both DR and UP tend to
change the yaw-orientation of the robot, which is not a desired behaviour.

In test environment 2 (T2), we attach a mass of 7.5 $lbs$ (unknown to the
policy parameters) to the robot’s torso. In this environment, the ability for
EAP to adapt is best illustrated, beating both baselines DR and UP in straight
line distance travelled on carpeted ground. With EAP the robot moves an
average distance of $1.7$m , UP and DR manage close $1$m. Results illustrated
in 5.12(b).

The performance of the policies in test environment 3 (T3) are similar to that
of T2 with EAP managing to travel the furthest distance compared to baselines
DR and UP.

(a) In-place walking task

(b) Walking forward task

Figure 5.12: Evaluation of real world results.

###  5.8 Conclusion

We presented a novel approach to train an error-aware policy (EAP) that
transfers effectively to unseen target environments in a zero-shot manner. Our
method learns an EAP for an assistive wearable device to help a human recover
balance after an external push is applied. We show that a single trained EAP
is able to assist different human agents with unseen biomechanical
characteristics. We also validate our approach by comparing EAP to common
baselines like Universal Policy and Domain randomization to show our
hypothesis that a policy which explicitly takes future state error as input
can enable better decision making. Our approach outperforms the baselines in
all the tasks. We also evaluated the performance of our algorithm through a
series of ablation studies that sheds some light on the importance of
parameters such as error horizon length, error representation, choice of
observable parameters and choice of reference dynamics. We find that EAP is
not sensitive to either the choice of observable parameters or the reference
dynamics, and outperforms the baselines with variations in these quantities as
well. Further, we validated the ability of EAP to transfer to real world
scenario using A1 quadrupedal robot. The preliminary results presented for
simple tasks such as in-place walking and walking forward shows promise of
applying the proposed approach to real world scenarios.

Our work has a few limitations. At the core, our algorithm relies on the error
function to make predictions of the expected state errors. The accuracy of
this prediction can be improved by better function approximators such as
recurrent neural networks (RNN) that takes a history of states as input, we
leave this for future work.

##  Chapter 6 Conclusion and future work

In conclusion, we presented of a set of learning based algorithms that address
the important challenge of safety in bipedal locomotion. Our contributions are
along two closely related directions 1) Fall prevention and safe falling for
bipedal robots and 2) Fall prevention for humans during walking using
assistive devices. With bipedal robots, we showed the effectiveness of
reinforcement learning (RL) based approaches to learn control policies that
demonstrate a wide range of falling and balancing strategies. Our algorithmic
contributions also included improving efficiency of learning by incorporating
abstract dynamical models as priors, curriculum learning, imitation learning
and a novel method of building a graph of policies into the standard RL
framework. In the chapters 4 and 5 of the thesis, we shifted focus towards an
important healthcare related topic of fall prevention for humans using
assistive devices. To this end, our contributions include using imitation
learning approach to create virtual human walking agents whose bio-mechanical
gait characteristics are similar to real world humans. Using these simulated
agents we showed that, assistive devices could indeed help prevent falls when
pushed whereas human agents without assistants fail. Finally, since humans
exhibit a wide range of variations in gait characteristics, we developed a
novel algorithm to enable zero-shot generalization of fall prevention policies
when tested on new human subjects. Although inspired by assistive devices, the
algorithm is applicable for any robotic system and we validated the proposed
approach on a real world quadrupedal A1 robot.

###  6.1 Safe locomotion for bipedal robots

In chapter 3, we developed an algorithm to generate safe falling motion for a
bipedal robot. Unlike steady state walking, which is periodic in nature,
optimizing for a falling motion is challenging because it involves reasoning
about discrete contact sequence planning (which body part should hit the
ground and in what sequence) as well a continuous joint motion to minimize
impact upon ground contact. We showed that we could solve this problem using
an offline policy learning algorithm. Our approach significantly improved on
the computation time required by prior dynamic programming methods while also
ensuring that a wide variety of falling strategies naturally emerge from the
algorithm. Towards the same goal of safety for bipedal locomotion, inspired by
strategies humans use to recover balance as a response to external pushes, in
our second work we presented a training methodology for a control policy which
learns residual control signals to those generated by traditional model-based
controllers to maintain balance. By learning residual control signals, we not
just simplify learning the task because the model-based controllers act as
strong priors, but also improve upon methods that use either just model-based
or naive RL approaches. Our algorithm also included a novel sampling method
that adaptively adjusts the difficulty in training samples to encourage
efficient learning. By doing so, we achieved better performance at a lower
sample cost compared to baseline naive RL approach.

We also addressed the common issue of sample inefficiency while solving
complex task such as locomotion with model free reinforcement learning
algorithms. RL has shown promising results in learning complex motor skills in
high dimensional systems, however the control policies typically perform well
only from a small set of initial states and take millions of samples to
complete the task. In the final section of chapter 3, we take a divide and
conquer approach to solve this by breaking down a complex task into multiple
sub-tasks and learn a directed-graph of control policies to achieve the same
task but by consuming less samples. The edges of our directed graph contains
policies and the nodes contain state distributions, so a policy can take the
robotic system from the one set of states to another set of goal states via
rolling out policy/policies. Starting from the first policy that attempts to
achieve the task from a small set of initial states, the algorithm
automatically discovers the next subtask with increasingly more difficult
initial states until the last subtask matches the initial state distribution
of the original task. We showed that our approach takes lesser samples than
common baselines such as naive RL, curriculum learning, manually generated
sub-tasks while ensuring that the task can be completed from a wider range of
initial states.

##### Future work

The limitation with our work on push recovery, be it either fall prevention or
safe falling, is that the control policies work best when the robot’s initial
center of mass velocity is close to zero when pushed. However, in real world
scenarios the ability to recover when the robot’s motion is more dynamic, such
as walking or running, would be crucial to ensure safety under all
circumstances. Developing algorithms that can ensure safety for a wider range
of scenarios would be an interesting extension to our work.

###  6.2 Fall prevention using assistive devices

In chapter 4, we shifted our focus towards learning fall prevention control
policies for assistive devices. We developed an approach to automate the
process of augmenting an assistive walking device with the ability to prevent
falls. Our method has three key components : A human walking policy, fall
predictor and a recovery policy. In a simulated environment we showed that an
assistive device can indeed help recover balance from a wider range of
external perturbations. We introduced stability region as a quantitative
metric to show the benefit of using a recovery policy, larger area of
stability region would imply better recovery motion. In addition to this,
stability region can also be used to analyze different sensor and actuator
design choices for an assistive device. We evaluated the stability region of
six different policies using various sensor and actuator configurations to
shed some light on appropriate design choices to help develop effective
assistive devices.

A significant bottleneck in making assistive devices more ubiquitous in
society is due to the fact that humans exhibit remarkable variations in gait
characteristics, making design of control policies that generalize for a large
population extremely challenging. A common approach has been to tune
controllers for each individual user, however this can be tedious and a time
consuming process. We addressed this issue by viewing it through the lens of
transfer learning problem. We developed a novel algorithm that enables a
policy to transfer in a zero-shot manner for partially observed dynamical
systems like humans. Our work introduces an Error-aware policy (EAP) which
explicitly takes as input a predicted state error generated in the target
environment to produce a corrected action that successfully completes the
task. We show that a single trained EAP is able to assist different human
agents with unseen biomechanical characteristics. We also validate our
approach by comparing EAP to common baselines like Universal Policy (UP) and
Domain randomization (DR) to show our hypothesis that a policy which
explicitly takes future state error as input can enable better decision
making. Our approach outperforms the baselines in all the tasks. We also
evaluated the performance of our algorithm on a real world quadrupedal robot,
which strengthens the case for applicability of EAP on a real system. We show
that for tasks such as walking forward, EAP is able to outperform baselines
such as DR and UP when tested on novel environments with unique ground contact
characteristics.

##### Future work

A key component of our work with assistive devices is virtual human walking
agent. Simulated agents that generate bio-mechanically accurate motion
[falisse2020physics, santos2021predictive, weng2021natural] can play a crucial
role in not just improving our understanding of human motion but also help in
developing better assistive and rehabilitation strategies for people with
disability. Algorithms that can generate such human-like motion for not just
periodic steady state walking gaits, but for more dynamic motion such as
running, push-recovery, slip/ trip recovery can be a very exciting research
direction. Incorporating musculo-tendon human models could also be a
contributing factor in closing the gap between real world and simulated human
agents. The major bottleneck for such applications is the shortage of real-
world data for validation purposes, recent advancements in machine learning
methods to learn from limited data as well as a collaborative efforts between
researchers working in bio-mechanics and machine learning can help overcome
some of these challenges.

In this work, our primary contributions have been on developing control
policies for robots, however in the spirit of the famous phrase ”form follows
function”, co-optimization of robot design along with policy learning can be a
powerful approach to create functional robots in the future. In chapter 4, we
provided some preliminary results by shedding light on the best sensor and
actuator design choices for assistive devices based on the policy’s
performance. Extending this to optimize for robot structural design can be an
interesting direction.

Another closely related topic to our work with assistive devices is modelling
human-robot interaction. In our current, work we assume that the human
perfectly adapts to the forces applied by the exoskeleton. However, in reality
this does not necessarily hold true. Better predictive modelling of the
interaction between the robot and the human is a promising research direction.
Modelling human intention while collaborating with a robot could also be key
towards wide-spread use of these robots. Contributions in this area can have a
lasting impact not just with assistive devices but also numerous other
applications such as human-robot collaboration in a factory setting, self-
driving cars and robots that are built for home environments.

As highlighted in Chapter 5, sim-2-real transfer of policies has gained a lot
of interest in the research community. However, the primary focus from
researchers has been through the lens of algorithmic contributions, an
exciting future direction will be to develop methods that improves simulation
environments by bringing them closer to reality. This could be done through
leveraging machine learning methods to improve contact models, focusing
efforts on differentiable simulators to enable end-to-end learning or by
improving computational efficiency of more traditional methods such as finite-
element methods (FEM) to improve accuracy of simulation, especially for
deformable bodies.

Visak Kumar was born in Bangalore, India in 1991. He graduated from National
Public School in 2009 and attended M.S.Ramaiah Institute of Technology from
2009 to 2013 where he majored in Mechanical Engineering. After completing
undergraduate degree, Visak worked at Indian Institute of Science, Bangalore,
under supervision of Prof. Ananthasuresh from 2013 - 2014, as a research
assistant where he worked on developing a micro-scale mechanical pump device
for drug delivery.

Visak obtained his Masters degree from University of Washington, Seattle in
2016. After obtaining his masters degree, he started PhD program in robotics
at Georgia Tech, under supervision of Dr. Karen Liu working on developing
learning algorithms for robot control. In 2017, Visak worked as a research
intern at Disney Research, Pittsburgh under supervision of Dr. Katsu Yamane
and Dr. Sehoon Ha. In 2019, Visak interned at Nvidia Research, Seattle under
supervision of Dr. Stan Birchfield and Dr. Jonathan Tremblay working on
dextrous manipulation. He returned as an intern to Nvidia in summer of 2020.