# OGMP: Oracle Guided Multimodal Policies for Agile and Versatile Robot
Control

Lokesh Krishna, Nikhil Sobanbabu, and Quan Nguyen
University of Southern California, Los Angeles, CA, USA
Email: {lkrajan<EMAIL_ADDRESS>

###### Abstract

Amidst task-specific learning-based control synthesis frameworks that achieve
impressive empirical results, a unified framework that systematically
constructs an optimal policy for sufficiently solving a general notion of a
task is absent. Hence, we propose a theoretical framework for a task-centered
control synthesis leveraging two critical ideas: 1) oracle-guided policy
optimization for the non-limiting integration of sub-optimal task-based priors
to guide the policy optimization and 2) task-vital multimodality to break down
solving a task into executing a sequence of behavioral modes. The proposed
approach results in highly agile parkour and diving on a 16-DoF dynamic
bipedal robot. The obtained policy advances indefinitely on a track,
performing leaps and jumps of varying lengths and heights for the parkour
task. Corresponding to the dive task, the policy demonstrates front, back, and
side flips from various initial heights. Finally, we introduce a novel latent
mode space reachability analysis to study our policies’ versatility and
generalization by computing a feasible mode set function through which we
certify a set of failure-free modes for our policy to perform at any given
state.

##  I Introduction

Deep Reinforcement Learning (RL) as an optimizer has shown remarkable
empirical success in synthesizing control policies for hybrid underactuated
systems like legged robots [19]. In particular, with inherently stable
quadrupedal robots, leading to extreme parkour [3, 20], agile [17] and robust
[14] locomotion capabilities. Irrespective of the task at hand, the above-
methods that learn from scratch follow a similar philosophy: 1) define an
exhaustive observation space, 2) engineer task-specific rewards and/or
curriculum, 3) perform policy distillation, and 4) extensively randomize.
Despite the compelling empirical results, such an approach does not
conceptually address the underlying shortcomings of applied Deep RL, like
finite exploration, reward design, susceptibility to local optima, etc, while
only tackling the manifested performance limitations. For instance, policy
distillation deals with the finite exploration problem by first training a
teacher policy [3] with privileged information or a library of policies [20]
with the hope of effective exploration and then distilling them into a single
policy. As a result, performing multiple subsequent trainings, only made
tractable by hardware-accelerated simulators, to increase the amount of
exploration as opposed to its quality and thereby the search. On the other
hand, rigorous reward-shaping on a battery of heterogeneous terms is performed
to meet the intended performance, which is robot and task-specific [17, 14],
as even a marginally subpar combination results in the policy converging to an
undesirable local optimum. Addressing this sensitivity of the optimization is
unintuitive and cumbersome as the reward terms vary in scale and nature,
making them seldom transferable. Furthermore, such approaches are end-to-end
task-specific solutions as opposed to being a unified approach for solving a
general notion of a task as in the case of Gradient and Model-based Optimal
Control (GMOC) frameworks [16, 5]. Thus, developing a theoretic framework for
a unified task-agnostic control synthesis by addressing finite exploration,
reward design, and behavioral multimodality shall be the focus of this work.

Figure 1: Conceptual overview of the proposed synthesis approach (top left),
oracle guidance in green vs. actual motion in red (top center), a frontflip
dive from a $2$m high platform (right), and agile parkour on a track with gaps
and blocks of arbitrary dimensions (bottom left). Accompanying video results
at :  https://youtu.be/IyCyfTe-k1A

A popular alternative is imitation learning (IL) over demonstrations, having a
unified imitation reward. With dynamically consistent demonstrations111through
proficient human teleoperation that solve an intended task, IL is a reliable
strategy as seen in manipulation [4]. In locomotion, we typically have
dynamically inconsistent demonstrations that partially solve a task,
challenging the direct application of IL. For instance, to parkour with a
bipedal robot, we may have demonstrations for runs, leaps, and jumps of
varying intensities from 1) motion capture on humans or 2) a GMOC framework.
Nonetheless, they suffer from source-target mismatches due to morphologically
dissimilar experts [2] and simplifications made to the model, cost, and
constraints [6, 10, 9] respectively. Strictly imitating discrepant
demonstrations can limit the policy from leveraging the full potential of the
target system, inducing a sub-optimality. Moreover, naive imitation of such
partial-task-solving demonstrations does not guarantee to solve the
overarching task, requiring RL to train another high-level task-solving policy
[2, 8]. Alongside demonstrations, task-based priors can include intuitive
heuristics with design biases in providing an ansatz to the solution.
Therefore, a general way to model such priors would be as oracles that provide
that ansatz when queried. An oracle-guided policy optimization can then be
performed with the critical insight: use oracle’s ansatz as a seed point for
the search. Constraining the policy’s search space around the local
neighborhood of an oracle’s ansatz results in effective exploration of a
single policy optimizing a task objective of choice.

On the other hand, an essential trait to solve a high-level task is behavioral
multimodality. In the context of locomotion, prior works propose learning to
imitate a variety of skills [8, 11] by encoding a dataset of behaviors to a
skill-embedding latent space and training a latent-conditioned policy. While
they showcase a finite number of locomotion modes and transitions,
generalization to fruitful variations of each mode is unexplored. For example,
a locomotion task of moving from point A to point B might require us to
operate in discrete locomotion modes like walk and jump, but also modulate
continuous parameters like walking speed and jumping height, depending on the
nature of the underlying terrain. However, mimicking a dataset with
demonstrations for jumping at heights $0.1$ and $0.5$ m does not guarantee a
policy that could jump a height of $0.2$ or $0.7$ m. To this end, we define a
mode set and mode parameter set required for solving a task. Solving a task
can then be viewed as executing a sequence of modes with a specific parameter
variant chosen for each mode. We leverage the task-vital multimodality in
designing an oracle, using which we train a latent-conditioned policy that
generalizes to a continuous spectrum of underlying behavior parameters to
solve the given task sufficiently.

Once synthesized, the reusability of such latent-conditioned policies requires
understanding their limitations. The naive alternative of strapping a high-
level policy outputting latent commands results in an uninterpretable pipeline
[8]. Disregarding the explicit analysis, we can neither explain the approach’s
success nor predict an upcoming failure, which is quintessential for real
robot control. Previous works validate their policy’s performance through
qualitative empirical results [17, 20] or skill/task-specific custom
performance metrics [13, 2, 3, 18], which fail to be a standard method of
analysis. In contrast, we exploit the low-dimensional latent space (or mode
space) to perform a modified reachability analysis. Similar to value functions
in RL that reflect a measure of optimality from a given state, we introduce a
set-valued function called the Feasible Mode-set Function, which computes the
set of feasible behaviors that our policy can perform from a given state.
Through the proposed Latent Mode Space Reachability Analysis, we also
characterize our policy’s generalization in and out of the training domain.

Thus, the major contributions of our paper are threefold:

  * •

Oracle Guided Multimodal Policies: A theoretical framework for task-centered
control synthesis leveraging oracle-guided optimization for effective
exploration and task-vital-multimodality for versatile control.

  * •

Experimental Validation: Demonstration of extremely agile bipedal control
tasks: parkour and dive. A single oracle-guided multimodal policy (per task)
successfully performs diverse variants of distinct behaviors like pace, leap,
and jump over arbitrary terrain tracks for parkour, while diving with
omnidirectional flips from arbitrary initial heights.

  * •

Latent Mode Space Reachability Analysis: A modified reachability analysis to
quantify our policy’s versatility through Feasible Mode-set Function
computations

The remaining paper is structured as follows: Sec. II introduces the
theoretical synthesis framework, Sec. III discusses the proposed approach,
followed by Sec. IV presenting the exhaustive experimental results, analysis
and ablation studies.

##  II Oracle Guided Multimodal Policies

This section presents the two synergetic ideas behind our theoretical task-
centered control synthesis framework: oracle-guided policy optimization and
task-vital multimodality.

###  II-A Oracle Guided Policy Optimization

Let $\mathcal{T}$ be an infinite horizon task with a task parameter set,
$\psi_{\mathcal{T}}\in\Psi_{\mathcal{T}}$. Sufficiently solving $\mathcal{T}$
requires maximizing a task objective, $J_{\mathcal{T}}$ over the task
parameter distribution, $p(\psi_{\mathcal{T}})$. Given the corresponding state
space of interest, $x\in\mathcal{X}$ let $x_{t}$, $x[a,b]$ denote a state at
time $t$ and a state trajectory from time $t\in[a,b]$ respectively. We define
$\Xi$ to be a receding horizon oracle that provides a finite horizon state
trajectory ($x^{\Xi}[t,\,t+\Delta t]$) from any given state ($x_{t}$) until
$\Delta t$ into the future for any task-variant ($\psi_{\mathcal{T}}$), such
that $x^{\Xi}[t,\,t+\Delta t]$ is always within an
$\epsilon\text{-neighborhood}$ of an optimal state trajectory. Formally,

$\displaystyle x^{\Xi}[t,\,t+\Delta t]=\Xi(x_{t},\psi_{\mathcal{T}})$ (1a)
s.t. $\displaystyle\exists\,\,x^{\Xi}[t,\,t+\Delta
t]\quad\forall\,(x_{t},\psi_{\mathcal{T}})$ (1b)
$\displaystyle\|x^{\Xi}_{t}-x^{*}_{t}\|_{W}<\epsilon\,\,\,\,\,\forall\,t\in[0,\,\infty)$
(1c)

where $\epsilon$ is the maximum deviation bound, a constant for a given pair
$(\mathcal{T},\,\Xi)$, and $W$ is a diagonal weight matrix. Note that
$x^{*}[0,\infty],\,\epsilon$ are unknown and are only provided for
constructing the conceptual argument. Designing oracles that satisfy the above
conditions is straightforward as demonstrated later in Sec. III-B.

Given the admissible control and observation spaces, $u\in\mathcal{U}\text{
and }o\in\mathcal{O}$, we would like to obtain the optimal policy
$\pi^{*}(u|o)$ from a policy class $\Pi$ guided by $\Xi$, that sufficiently
solves $\mathcal{T}$. Specifically, we intend to integrate the oracle in a
non-restrictive manner such that the sub-optimality of $\Xi$ does not limit
$\pi$. Since $\Xi$ provides a reference in the state space, we propose
constraining the permissible states for $\pi$ to be within a
$\rho$-neighbourhood of the oracle’s guidance. Formally,

$\displaystyle\pi^{*}:=\arg\max_{\pi\in\Pi}J_{\mathcal{T}}$ (2a) s.t.
$\displaystyle\|x^{\pi}_{t}-x^{\Xi}_{t}\|_{W}<\rho\quad\forall t\in[0,\infty)$
(2b)

Where $x^{\pi}$ represents states generated while rolling out policy $\pi$,
$\rho$ is the permissible state bound. In the context of learning, $\rho$
directly controls the quality of oracle-guided exploration. Hence, one can now
argue that

$\displaystyle\|x^{\pi}_{t}-x^{*}_{t}\|_{W}$
$\displaystyle\leq\|x^{\pi}_{t}-x^{\Xi}_{t}\|_{W}+\|x^{\Xi}_{t}-x^{*}_{t}\|_{W}$
(3) $\displaystyle=\epsilon+\rho\quad\quad\forall t\in[0,\infty)$

Thus, by bounding the exploration of $\pi$ within a $\rho$ -neighborhood of
$x^{\Xi}$, we have ensured that the optimal policy, $\pi^{*}$ can still be
found within the constrained search space222$\rho\geq\epsilon$ is assumed to
be trivially satisfied i.e. ($\epsilon+\rho$)-neighborhood of $x^{*}$ as
visualized in Fig. 2.b. The choice of $\rho$ for a given pair
$(\mathcal{T},\,\Xi)$ is dependent on the corresponding $\epsilon$. $\epsilon$
being maximum deviation of $x^{\Xi}$, an oracle with low $\epsilon$ will
generate references closer to $x^{*}$. In contrast, for an oracle with high
$\epsilon$, there should be sufficient search space for $\pi$ to explore
around $x^{\Xi}$ and converge to $x^{*}$. Intuitively, as $\epsilon\to\infty$,
it is as if the optimization is unguided, thus needing $\rho\to\infty$, and
conversely, as $\epsilon\to 0$, the search can be localized to a tight
neighborhood of $x^{\Xi}$, implying $\rho\to 0$. Hence, we propose that
optimal $\rho$ for a given $\epsilon$ can be chosen as
$\rho^{*}=\alpha(\epsilon)$, where $\alpha$ is a non-decreasing function.
However, as $\epsilon$ is unknown, in practice, we perform a grid search over
$\rho$ for any given $(\mathcal{T},\,\Xi)$ and empirically verify the
$\alpha(\epsilon)$ trend in Sec. IV.

###  II-B Task Vital Multimodality

Any high-level task $\mathcal{T}$ can naturally be broken down into sub-tasks,
which may require different modes of operation in the policy. Similar to [11],
we think of these modes as time-invariant finite-horizon state trajectories
representing different behaviors. Thus, W.L.O.G, we define a finite set of
behavioral modes, $\mathbb{M}$, required to solve $\mathcal{T}$ sufficiently.
Each mode $m\in\mathbb{M}$ in turn has a set of characteristic parameters,
$\Psi_{m}$. Hence, the set of mode parameters, $\Psi_{\mathbb{M}}$, and the
corresponding task parameter set, $\Psi_{\mathcal{T}}$, can now be defined as,

$\Psi_{\mathbb{M}}:=\bigcup_{i=1}^{|\mathbb{M}|}\Psi_{m_{i}},\Psi_{\mathcal{T}}:=\Psi_{\mathbb{M}}^{N}$
(4)

where $N$ is the number of horizons encountered in the task. It is worth
noting that despite $\mathbb{M}$ being discrete, the corresponding parameter
sets, $\Psi_{m},\Psi_{\mathbb{M}},\Psi_{\mathcal{T}}$ can all be continuous.
For example, say to sufficiently solve a task like parkour ($\mathcal{T}$); we
need three locomotion modes: jump, leap, and run ($\mathbb{M}$); then we
should be able to run at different speeds, leap different distances and jump
different heights ($\Psi_{\mathbb{M}})$ as visualized in Fig 2.a. The ideas of
breaking down a task into a sequence of modes and the oracle providing
guidance for a horizon $\Delta t$ are complementary as modes are temporal
abstractions of states. Thus, $\Xi$ can be queried for a mode switch once
every $\Delta t$ ($\geq$ $dt$, $\pi$’s control period ) with only the
information of the mode parameter set as opposed to knowing the complete task-
variant or task parameter set, i.e.,
$\Xi(x_{t},\psi_{\mathcal{T}})\equiv\Xi(x_{t},\Psi_{\mathbb{M}})$

##  III Proposed Approach

Figure 2: Overview of OGMP: a) The breakdown of a task into its principal
modes with a mode and mode parameter set b) Guided exploration by constraining
the search space around the local neighborhood of the oracle’s reference c)
Mode encoder: an LSTM autoencoder trained on a custom modal dataset by
minimizing reconstruction loss d) Multimodal policy trained with oracle guided
policy optimization on a task environment e) The closed-loop inference
pipeline with the high-level oracle and the low-level multimodal policy

In this section, we first define two agile control tasks, followed by our
specific design methodology in applying OGMP to solve them effectively, as
visualized in Fig 2. Given a task, we first define the fundamental modes of
operation and their corresponding parameter sets required to solve a task
sufficiently (Fig 2.a). Using these defined sets, we design an oracle that can
generate state trajectories as references from any given state. Spanning the
mode parameter set, we use the oracle to generate a custom dataset of diverse
behaviors and then train a mode encoder to construct the induced latent mode
space (Fig 2.c). Finally, we train a multimodal policy conditioned on this
latent mode space guided by the oracle (Fig 2.d). During policy optimization,
the oracle is periodically queried to generate references online and is used
to constrain the policy’s search space to effectively explore (Fig 2.b). Once
trained, a novel reachability analysis is performed to quantify the policy’s
versatility and generalization, as shown in Fig 4. The above approach is
explained in detail as follows.

###  III-A Task Definitions

We demonstrate the above conceptual framework applied in two bipedal control
tasks demanding extreme agility and versatility: parkour and diving, as
explained below.

####  III-A1 Parkour Task

Unlike unstructured open-world parkour, we work with an abstracted version
where the underlying terrain’s height ($h$) is varied only along the heading
direction, and the robot is expected to advance along this track333This
simplification is only to conduct a systematic behavior analysis and is not
limited by the proposed approach. The parkour track is composed of an
arbitrary permutation of three terrain elements: block $(h>0)$, gap $(h<0)$,
and flat $(h=0)$ with random heights. While a gap is non-traversible,
requiring a single leap to cross, the robot can step on to traverse the block
and flat elements. Thus, to sufficiently solve this task, a policy must leap,
jump, and pace over various terrain elements while seamlessly switching
between these modes. The above problem can now be encoded in our conceptual
framework as shown in table I

$\mathcal{T}:J_{\mathcal{T}}$ | $m\in\mathbb{M}$ | $\Psi_{m}\in\Psi_{\mathbb{M}}$ | $p(\psi_{\mathcal{T}})$
---|---|---|---
parkour: advance | pace | $\\{v|v\text{ is the heading speed}\\}$ |
forward | jump | $\\{(w,h)|w,\,h\text{ are the }$ |
indefinitely | jump length and height} | U($\psi_{\mathcal{T}}$)
along the | leap | $\\{(w,d)|w,\,d\text{ are the }$ |
track | approach and leap distance} |
| settle | $\\{v|v\text{ is the touchdown }$ |
dive: do a $360^{\degree}$ | velocity to be driven to zero} | U($\psi_{\mathcal{T}}$)
flip and land | flip | $\\{(r,h)|r,\,d\text{ are the flip }$ |
| direction and fall height} |
TABLE I: Task Vital Multimodality for parkour and diving

####  III-A2 Dive Task

In this task, the policy is expected to dive down a block of arbitrary height
($h$) and land on the ground, doing a $360^{\degree}$ flip along the roll or
pitch DoF. The motion can be split into three phases: take-off, flight, and
landing. In the flight phase, the angular momentum is conserved; only allowing
minimal changes to the angular velocity by morphing the robot’s inertia.
Therefore, it is crucial for the policy to reach the required initial angular
velocity to complete a rotation by the end of the take-off phase. Given this
coupling effect, we combine these two phases into a single mode: flip, which,
depending on the direction, can have 4 variants: front, back, left, and right.
Upon completing a successful flip, the policy has to land and transition to
settle mode, which is to remain in the nominal pose perpetually as defined in
table I.

Evident from the above two tasks, $J_{\mathcal{T}}$ is task-dependent and
could be hard to design. In the absence of a reasonable $J_{\mathcal{T}}$, a
compelling alternative would be to “track” the oracle as they provide
$\epsilon$-neighbourhood reference to a task-solving optimal solution. Hence,
we propose the surrogate objective

$\tilde{J_{\mathcal{T}}}:=\sum_{t=0}^{\infty}\|x^{\pi}_{t}-x^{\Xi}_{t}\|$ (5)

Thus, minimizing $\tilde{J_{\mathcal{T}}}$ subject to the dynamics and control
constraints leads to a feasible motion that is as close to the reference as
possible. Strictly minimizing $\tilde{J_{\mathcal{T}}}$ within the oracle-
constrained search space can cause $\Xi$ to limit $\pi$; however, in practice,
choosing an optimal $\rho$ and long enough $\Delta t$ causes
$\pi^{*}_{\tilde{J_{\mathcal{T}}}}\rightarrow\pi^{*}_{J_{\mathcal{T}}}$444$\pi^{*}_{J}:=$
optimal policy w.r.t objective, $J$ as verified in Sec IV-C. For tasks like
dive, where finding a reasonable $J_{\mathcal{T}}$ is nontrivial, we show that
the task-independent $\tilde{J_{\mathcal{T}}}$ is a good bet in practice. We
study the empirical validity of using $\tilde{J_{\mathcal{T}}}$ in Sec. IV and
a per-step reward for an equivalent maximization objective in Sec. III-B.

###  III-B Design Methodology

For the above-defined tasks, we discuss the specific design choices made to
OGMP. Fig 2 provides a visual overview of the training ( Fig 2.c and d) and
inference (Fig 2.e) pipelines explained below.

####  III-B1 Oracle Module

For any given task, a simple strategy for an oracle would be to linearly
interpolate the relevant state variables from the initial to the desired goal
states. In parkour, since the overarching agenda is to advance along the
track, the linear interpolation would be from the current to a target position
along the track while adapting the robot’s height for blocks (height increase
for a jump) and gaps (ramp-up and down for a leap) as shown in Fig. 3 (left).
For the dive, the oracle can linearly interpolate the base height and
corresponding rotational DoF from $0^{\degree}$ to $360^{\degree}$. Once we
obtain a position trajectory, the velocity can be the finite difference.
Naming this oracle as $\Xi_{\text{li}}$ (Fig. 3), one can notice its high
$\epsilon$ as the generated state-trajectories do not consider the system’s
inertia and gravity. To this end, considering a simplified single rigid body
model as in [12], the dynamics in the world coordinates are given by:

$\displaystyle m(\ddot{p}+g)$ $\displaystyle=\sum_{i=1}^{2}f_{i}$ (6)
$\displaystyle\frac{d}{dt}(I\omega)\approx I\dot{\omega}$
$\displaystyle=\sum_{i=1}^{2}(r_{i}\times
f_{i}+\tau_{i})=\sum_{1}^{2}\bar{\tau}_{i}$ (7)

Figure 3: Reference trajectories from different oracles for the modes of
parkour (left) and diving (right) with the parameters in table I

where $\ddot{p}$, $\omega$ is the robot COM acceleration and the angular
velocity, $r_{i},\,f_{i},\,\tau_{i}$ are the position, force and moment
vectors from the $i^{th}$ contact point and $m,\,I,\,g$ are the mass, moment
of inertia and gravity. Typically, $r_{i}$ is from a predefined contact
schedule, leading to time-varying dynamics. Since, by definition, an oracle
need not provide realistic control, we define an auxiliary control
$\bar{\tau}_{i}$, encompassing the overall moment, which in turn makes the
dynamics time-invariant. Additionally, the rotation and rotation rate matrices
are made constant by considering the average reference orientation over a
horizon. Upon these approximations to Eq. 7 and discretization leads to a
linear time-invariant (LTI) system over the current horizon:

$\vspace{-1mm}x_{t+1}=Ax_{t}+Bu_{t},\>y_{t+1}=Cx_{t},\>u_{t}=[f_{1},f_{2},\bar{\tau}_{1},\bar{\tau}_{2}]^{T}$
(8)

where $x_{t}\in\mathbb{R}^{13},\,y_{t},\,u_{t}\in\mathbb{R}^{12}$ are the
gravity-augmented state, relevant output, and control vectors. Thus, a new
oracle can be constructed considering two distinct phases: flight and contact.
In flight, $u_{t}=0$ as there are no contacts, and during contact, an
optimizer of choice can be used to compute the optimal control for a given
objective, $u_{t}=u^{*}_{t}$. Applying the corresponding control and forward
simulating Eq. 8, the state trajectory is obtained. Given an LTI model and the
absence of constraints, we use the analytic solution of preview control[15] as
opposed to sophisticated numerical optimizers, thereby allowing long-horizon
computations. As a desirable attribute for an oracle, preview control
incorporates look-ahead behavior with a previewable demand term in its control
law as opposed to LQR. Thus, using $\Xi_{\text{li}}$ as the reference for a
tracking objective, optimizing with preview control and LQR results in
oracles, $\Xi_{\text{prev}}$ and $\Xi_{\text{lqr}}$ respectively, having a
$\epsilon$ smaller than $\Xi_{\text{li}}$, as they capture the dominant
dynamics of the hybrid system as shown in Fig 3.

####  III-B2 Multimodal Policy Module

We propose a multimodal policy class (Fig 2.e) comprising a mode encoder and a
mode-conditioned policy trained subsequently, as explained below and shown in
Fig 2.c and d.

Mode Encoder: The motivation for encoding modes is to obtain a compact
representation space for conditioning our policy, for which we first train the
mode encoder, $\xi$, on a dataset of diverse locomotion modes. Since modes are
relative-state time-invariant state trajectories, we require the $\xi$ to be a
map from a trajectory space into a latent space, i.e

$z=\xi(x[t,\,t+\Delta t]),\quad\dim(z)\ll\frac{\Delta t}{dt}\dim(\mathcal{X})$
(9)

As opposed to using a fixed and possibly unbalanced dataset of motion capture
from a dynamically different source, given the oracle ($\Xi$), a mode
parameter set ($\Psi_{\mathbb{M}}$), and a set of initial states
($\mathbb{X}_{0}$), we can generate a well-balanced dataset of any desired
size containing sufficient samples of each mode, prior to the policy training.
Since $\Psi_{\mathbb{M}}$ is a closed set, we discretize it, uniformly
spanning all possible parameter variations for each mode, generating a rich
and balanced dataset. $9$ variants per mode, $\dim(z)\text{ = }2$, and a $32$
neurons single hidden layer LSTM auto-encoder where chosen for training $\xi$.
Minimizing a reconstruction loss on the custom dataset generates a set of
latent mode points as visualized in Fig. 2.c. We observe a structured
clustering, which hints at the existence of an underlying regularized latent
mode space $\mathcal{Z}$, which is explored later in Sec. III-C and IV-B.

Mode Conditioned Policy: A stationary policy mapping from the observation
space, $o_{t}\in\mathcal{O}$ to the admissible controls space,
$u_{t}\in\mathcal{U}$ designed as follows,

$\displaystyle u_{t}$
$\displaystyle:=K_{p}\cdot(a_{t}-q_{t})-K_{d}\cdot\dot{q}_{t}$ (10)
$\displaystyle a_{t}$ $\displaystyle\sim\pi(a_{t}|o_{t})$

where $q_{t},\,\dot{q}_{t},\,K_{p},\,K_{d}$ are the motor positions,
velocities, proportional and derivative gain matrices. The defined action
space, $a_{t}\in\mathcal{A}$, can be interpreted as motors’ PD setpoints. With
the inherent partial observability of the system, the choice of observation
space becomes crucial. To this end, the observation space should provide
feedback from the system and the desired mode command. Hence, we choose
$o_{t}=[y_{t},\,z_{t},\,c_{t}]$, where $y_{t}$ is the measurable output of the
system consisting of base height, orientation, velocity, joint states, and
optionally a local terrain map in front of the robot, $z_{t}$ is the latent
mode command and the clock signal $c_{t}=[\text{sin}(\phi),\text{cos}(\phi)]$
is defined on a phase variable, $\phi$, which runs from 0 to 1, from the start
to end of a given mode. For the reward function, we define

$\displaystyle r_{t}$ $\displaystyle:=r_{\text{track}}\text{ +
}r_{\text{regulation}}$ (11) $\displaystyle r_{\text{regulation}}$
$\displaystyle:=0.05e^{-0.01\|u_{t}\|}\text{ - }0.3\cdot\mathds{1}\text{(if
non-toe contact)}$ $\displaystyle r_{\text{track}}$
$\displaystyle:=0.475e^{-5\|er_{p}\|}\text{ + }0.475e^{-5\|er_{o}\|}$

where $er_{p},er_{o}$ are the errors in base position and orientations and
$r_{t}$ is the per-step reward term of an equivalent maximization objective as
$\tilde{J_{\mathcal{T}}}$. The proposed oracle-guide constraint (Sec. II) is
programmed as a termination condition. To further improve performance, the
episode is also terminated when $r_{t}$ is below a threshold. The full
termination condition is,

$\text{terminate episode
:=}\mathds{1}{(\|x^{\pi}_{t}-x^{\Xi}_{t}\|_{W}>\rho)}\lor\mathds{1}{(r_{t}<0.5)}$
(12)

We analyze the choice of the $\rho$ and its effect on performance later in
Sec. IV-C. $W_{11}$ and $W_{33}$ are set to $1$ for the parkour and dive
tasks, respectively, with the remaining entries as zero. Thus, the above
modules result in a random horizon POMDP for which we use off-the-shelf PPO to
train a policy: $128$ nodes per layer, $2$ layer LSTM network, where each
episode is an arbitrary variant of the task, $\psi_{\mathcal{T}}$ uniformly
sampled from $\Psi_{\mathcal{T}}$.

###  III-C Latent Mode Space Reachability Analysis

Figure 4: Overview of LMSR Test: left) search space, $SS$ defined based on
$\pi$’s estimated $a_{ID}$ and $d_{ID}$. center) closed-loop system under
analysis. right) the computed finite mode-set function, $\mathcal{Z}(x^{-})$
in red

Given the low-dimensional latent mode space ($\dim(\mathcal{Z})=2$), we
leverage it to quantify the policy’s mode diversity. Equivalently, this
implies that the policy should have generalized to a significant number of
points in the latent space. To this end, we perform a modified reachable set
computation as discussed below. We first define $\mathcal{X}_{\text{fail}}$, a
set of failure states we want the system to avoid. In the context of the
legged robots, a generic notion of failure is the robot falling down.

$\mathcal{X}_{\text{fail}}:=\\{x\,|\text{ base height}<\text{ nominal min.
base height}\\}$ (13)

Without the oracle and mode encoder in the pipeline, considering a closed-loop
system with the multimodal policy and the environment, the latent mode command
($z$) is the only input as in Fig 4 (center). Given a trained $\pi$, since
mode switches occur once every $\Delta t$, the discrete-time closed-loop mode-
controlled dynamics for a task variant $\psi_{\mathcal{T}}$ can be written out
as

$x^{+}=F_{\pi}(x^{-},\,z,\,\psi_{\mathcal{T}})$ (14)

where $x^{+}$ is the state of the system after $\Delta t$ seconds having
transitioned to the mode $z$ from the state $x^{-}$. For a given $x^{-}$,
there should exist a set of modes executing which the system won’t reach
$\mathcal{X}_{\text{fail}}$ within $\Delta t$. Formally, we define the
feasible mode-set function, $\mathcal{Z}(x)$, which maps a state to the set of
all feasible modes in the latent mode space as follows,

$\mathcal{Z}(x):=\\{z\,|x^{+}\notin\mathcal{X}_{\text{fail}},\,x=x^{-}\\}$
(15)

Since Eq. 14 is not assumed to be known analytically, we compute
$\mathcal{Z}(x)$ by explicitly rolling out the policy, which we call the
Latent Mode Space Reachability (LMSR) test.

To compute $\mathcal{Z}(x)$, we first define a search space,
$SS:=\\{z\,|\,\|z-a_{ID}\|_{\infty}<k\cdot d_{ID}\\}$, where $a_{ID}$ and
$d_{ID}$ are the center and radius of the circumcircle of the latent points
in-domain (ID) of the training as shown in Fig 4 (left). Intuitively, one can
expect a policy to be successful when commanded from
$ID:=\\{z\,|\,\|z-a_{ID}\|<d_{ID}\\}$ as it is within the training domain.
$ID$, however, is a conservative estimate of the modes that the policy might
have learned; hence, to account the out-of-domain(OD) mode generalization of
$\pi$, we include points $OD:=\\{z\,|\,\|z-a_{ID}\|\geq d_{ID},\,z\in SS\\}$
by choosing $k>1$. Thus, an exhaustive $\mathcal{Z}(x)$ is computed from a
given $x$ and $\psi_{\mathcal{T}}$ within $SS$ by rolling out $F_{\pi}$ in
simulation as visualized in Fig 4 (right). Since the defined $SS$ is
continuous, we discretize it as a grid of $30$ points per axis, leading to
$900$ points in $SS$. Once $\mathcal{Z}(x)$ is computed, a direct measure of
the diversity would be $|\mathcal{Z}(x)|$, as it is the number of feasible
modes that $\pi$ could execute from $x$. To quantify the generalization of
$\pi$ in a domain $\mathcal{D}$ we define a probability measure
$\nu_{\mathcal{D}}$

$\nu_{\mathcal{D}}(\mathcal{Z}(x)):=\frac{\text{no. of points in
}(\mathcal{Z}(x)\cap\mathcal{D})}{\text{no. of points in }\mathcal{(D)}}$ (16)

where $\nu_{\mathcal{D}}\in[0,1]$ measures the probability of success when a
point $z$ is chosen from a domain $\mathcal{D}$ at a given
($x,\,\psi_{\mathcal{T}})$. Taking $\mathcal{D}\,=\,ID,\,OD$ we measure the
in-domain and out-of-domain generalization of our policies in Sec. III-C.

##  IV Results

This section contains the experimental results of OGMPs trained and validated
on the MuJoCo simulator. The overall performance metrics quantifying our
approach’s agility and versatility are presented first, followed by a
hyperparameter analysis of the design choices.

The twofold motivation of our approach was extreme agility and mode
versatility to synthesize policies that solve fruitful overarching tasks. To
this end, a single multimodal policy is trained per task, as
$\pi_{\text{parkour}}$ and $\pi_{\text{dive}}$ for parkour and dive,
respectively. The supplementary video and Fig 1 showcase
$\pi_{\text{parkour}}$ successfully traversing challenging parkour tracks with
blocks and gaps of arbitrary dimensions placed at random intervals
indefinitely. The seamless transitions between different leap lengths and jump
heights further highlight the policy’s versatile agility. On the other hand,
$\pi_{\text{dive}}$ performs successful front, back, left, and right flips
from various initial heights and smoothly transitions to the nominal pose,
landing the robot. Despite the absence of oracle reference for the actuated
DoFs, $\pi_{\text{dive}}$ learns to curl and extend its legs for the flip and
landing, respectively. The emergent behaviors are for increasing the angle
velocity in flight while landing with a minimum-impact touchdown. For
quantitative benchmarking, we report the expectation of different performance
metrics over the task parameter distribution $p(\Psi_{\mathcal{T}})$,
approximated by the sample mean measured over $n$ episode rollouts. If $l$ is
a defined metric, the values in table II and Fig. 8 are computed as:
$\mathbb{E}_{p(\psi_{\mathcal{T}})}[l]\approx\frac{1}{n}\sum_{i}^{n}l_{i}$,
where $l_{i}$ is the $i^{th}$ sample. Empirically, we found $n=100$ for every
metric’s mean to settle as a reliable estimate. We define a test environment
with a track length of $10$m with an episode length of $400$ steps for the
parkour and an episode length of $150$ for the dive. In each case, the episode
is terminated only if the episode length is reached or the robot falls down
(if the terrain-relative base height $<0.3$ m). The analysis on the most
relevant task (i.e., either $\pi_{\text{parkour}}$ or $\pi_{\text{dive}}$.) is
reported for each result below.

###  IV-A Agility

While there is no one metric to quantify agility, the maximum of Heading
Acceleration (M.H.A) [7], Froude number (M.F), and Heading Speed (M.H.S) [13]
have been used to measure agility of legged robots. We include the average
episode length (E.L) of the test environment, as ideally
$\pi_{\text{parkour}}$ is expected to advance indefinitely despite the
challenging track. The above metrics are estimated for $\pi_{\text{parkour}}$
with different choices of observation spaces as in table II. As shown, the
synthesized policies reach accelerations of up to $4.7g$ and heading speeds of
up to $1.77$ $m/s$ with Froude numbers ranging from $0.48-0.72$. The best
variant has an average episode length of $84\%$ the maximum possible episode
length, showing its efficacy. Despite $\pi_{\text{parkour}}$ being unaware of
the complete track, it dynamically advances along the track instead of
converging to conservative quasi-static motions by trading speed for
stability. Given a stringent termination condition, we notice
$\pi_{\text{parkour}}$ is forced to learn precise foot placements for landing
and take-off, thereby not needing multiple attempts to climb a block or cross
the gap, leading to highly agile motions. Due to the radically changing
terrain height, $\pi_{\text{parkour}}$ rapidly modulates its momentum as
reflected in the high accelerations in table II, harnessing the full potential
robot’s actuator. On the other hand, the Fraude number aims to compare the
dynamic similarity between quadrupeds and bipeds. [1] observed a switch from
energy-efficient walking to agile jumping gaits in bipedalism when they have a
Fraude number around $0.5$, which is interestingly consistent with our
results.

metric, units | $z_{t}$ | $z_{t},c_{t}$ | $h_{t}$ | $h_{t},c_{t}$ | $h_{t},z_{t}$ | $h_{t},z_{t},c_{t}$
---|---|---|---|---|---|---
M.H.A, $m/s^{2}$ | 4.7g | 3.5g | 3.2g | 3.6g | 3.5g | 3.1g
M.H.S ($v$), $m/s$ | 1.4 | 1.57 | 1.66 | 1.74 | 1.74 | 1.77
M.F. ($\frac{v^{2}}{g\cdot ll}$) , $-$ | 0.48 | 0.56 | 0.64 | 0.69 | 0.70 | 0.72
E.L., $-$ | 72 | 172 | 252 | 320 | 264 | 336
TABLE II: Metrics of agility for variants of $\pi_{\text{parkour}}$ with
different observation spaces

###  IV-B Versatility

The idea of task vital multimodality offers two interpretable sets: the mode
parameter set $\Psi_{\mathbb{M}}$, and the latent mode space, $\mathcal{Z}$ to
study the generalization of the policies over $p(\psi_{\mathcal{T}})$. While
$\Psi_{\mathbb{M}}$ can be used to test individual locomotion modes, it does
not scale for validating transitions for which we utilize $\mathcal{Z}$.

####  IV-B1 Mode Parameter Set Test

Since the defined $\Psi_{\mathbb{M}}$ of our training is a convex set, we
dilate it and define higher test ranges for each parameter, thereby testing
for both in-domain (ID) and out-of-domain (OD) generalization. Discretizing
this test set, we evaluate $\pi_{\text{parkour}}$ and plot the undiscounted
returns obtained for blocks and gaps with varying dimensions in Fig 5.
Different blocks and gaps require jumps and leaps of varying magnitudes,
showcasing our policies’ versatility. The training sets
$\Psi_{\text{jump}}\text{ and }\Psi_{\text{leap}}$ are the regions within the
boundary marked in black in each plot. Thus, $\pi_{\text{parkour}}$ shows
consistent performance for variants within the black boundary (ID) while also
extrapolating its skills by jumping and leaping over unseen terrain variants
outside the black boundary (OD) as seen in the supplementary video and Fig 5.

Figure 5: Heatmap of $\pi_{\text{parkour}}$’s undiscounted returns performing
jumps and leaps of varying magnitudes both in and out of $\Psi_{\mathbb{M}}$
while traversing blocks and gaps of different dimensions Figure 6: Overview of
the Flat ground LMSR Test: a) The defined search space $SS$ based on $a_{ID}$,
and $d_{ID}$ of $\pi_{\text{parkour}}$. b) The Finite Mode Set Function
($\mathcal{Z}(x)$) computed from the nominal initial state and terrain. c) The
elbow and silhouette scores for varied ‘K’ of K-Means clustering over the
state trajectories corresponding to points in $\mathcal{Z}(x)$. d) Mode
clusters corresponding to the optimal number of clusters, K=$4$ c) the base
height trajectory corresponding to the different points in $\mathcal{Z}(x)$,
colored consistent with their mode cluster.

####  IV-B2 Latent Mode Space Reachability (LMSR) Tests

Two variants of the LMSR test in Sec III-C is performed on
$\pi_{\text{parkour}}$: 1) Flat Ground Test: to quantify the mode-diversity
from a nominal state and terrain and 2) Transition Test: to quantify the
bandwidth of modes available to transition from a given state.
$a_{ID}=(-0.07,\,0.58)$, and $d_{ID}=0.82$ were estimated from latent modes
encountered (blue dots in Fig. 6.a ) when testing $\pi_{\text{parkour}}$ in-
domain of $p(\psi_{\mathcal{T}})$ similar to the metrics in table II, to
compute the corresponding search space.

Figure 7: Overview of the Transition LMSR Test: left) a feasible transition
corresponding to a point in $\mathcal{Z}(x)$. center) the feasible mode set
function at the transition state and the corresponding base height
trajectories shown in red. right) an infeasible transition from a point not in
$\mathcal{Z}(x)$. The optimal mode-to-transition picked by the corresponding
$\Xi$ and its trajectory is shown in blue.

Flat Ground LMSR Test: Taking $\psi_{\mathcal{T}}=$ flat ground and $x=$
nominal initial state of the robot, we roll out $F_{\pi_{\text{parkour}}}$ as
in Eq. 14 for one horizon, ($\Delta t$) for each $z$ in the above-defined
search space taking $k=2$, as seen in ref Fig. 6.a. From the computed
$\mathcal{Z}(x)$ in Fig 6.b, we observe that the $\pi_{\text{parkour}}$ shows
significant in and out-of-domain generalization. Precisely, the probability of
success in ID and OD of $\pi_{\text{parkour}}$ are $\nu_{ID}=0.99$ and
$\nu_{OD}=0.49$. Furthermore, we performed K-means clustering over a dataset
of state trajectories corresponding to each point in $\mathcal{Z}(x)$ (Fig.
6.e). Varying the number of clusters from $n\,=\,2\text{ to }15$, we note that
the optimal number of clusters with the highest silhouette score corresponds
to $n=4$ consistent with the elbow plot, as seen in Fig. 6.c. This observation
validates our initial design hypothesis of requiring a finite number of modes
and fruitful variations of each mode to solve a task. The distinct modes and
mode clusters are visualized in Fig 6 (bottom) and d. From left to right in
Fig 6 (bottom), we can loosely label these modes as jump, leap, hop, and step,
respectively. By varying the parameters of each of these modes,
$\pi_{\text{parkour}}$ gracefully performs a continuous spectrum of behaviors.
Interestingly, though we defined a single mode to move forward in the parkour
task, $\pi_{\text{parkour}}$ performs two distinct variations of pace with
different contact patterns: hopping and stepping. In the supplementary video,
we see that $\pi_{\text{parkour}}$ hops to traverse a strip of flat terrain in
the track while steps to make minor adjustments before taking off or after
landing in the leap and jump modes.

Figure 8: The estimated surrogate ($\tilde{J}_{\mathcal{T}}$) and true
($J_{\mathcal{T}}$) objectives vs. choice of 1) observation space, $o_{t}$ (a
and b), 2) oracles, $\Xi$ (c), 3) horizon length, $\Delta t$ (d and e) and 4)
permissible state bound, $\rho$ (f and g). In b, e, and g the upper bound of
$J_{\mathcal{T}}$ is marked in red.  Figure 9: Motion traces comparing the
choice of 1) oracles (a, b, c) where $\Xi_{\text{li}}$, $\Xi_{\text{lqr}}$
fail while only $\Xi_{\text{prev}}$ successful in a side flip dive from a
block of $2$m 2) horizon length (d, e, f) where the longest horizon ($\Delta
t=30dt$) results in the desired agile performance 3) permissible state bound
(g, h, i) where the tighter ($\rho=0.1$) and lenient ($\rho=0.8$) bounds fail

Transition LMSR Test: In an arbitrary track, $\psi_{\mathcal{T}}$ and an
episode of length $2\Delta t$, we perform the second variant of the LMSR test.
Starting from the same nominal state at $t=0$, $\pi_{\text{parkour}}$ follows
the optimal latent mode command, $z^{*}_{0}$ (given by the oracle), until
$t=\Delta t$. At the state of the transition point, $x_{\Delta t}$, we compute
$\mathcal{Z}(x=x_{\Delta t})$ to obtain the feasible modes that
$\pi_{\text{parkour}}$ could successfully transition to. From the computed
$\mathcal{Z}(x)$ and the corresponding state trajectories in Fig. 7(center),
we observe a significant number of modes-to-transition both in and out of the
training domain with success probabilities, $\nu_{ID}=1.0$ and $\nu_{OD}=0.77$
respectively. Naturally, the most optimal transition as per the oracle is to
mode $z^{*}_{1}$ (shown in blue in Fig. 7) is $ID$. However, $z^{*}_{1}$ is
not the only feasible transition $\pi_{\text{parkour}}$ is capable of
performing, which proves that $\pi_{\text{parkour}}$ was not limited by the
corresponding $\Xi$ and learns novel transition maneuvers through effective
exploration within the $\epsilon+\rho$ neighborhood around $\Xi$’s reference.
Given the reasonable “bandwidth” of mode transitions of
$\pi_{\text{parkour}}$, one can reuse these OGMPs to solve unseen tasks by
designing a corresponding new oracle, $\Xi_{\text{new}}$. We conjecture that
if $\Xi_{\text{new}}$ selects mode commands $z\in\mathcal{Z}(x)\,\forall x$ at
mode-switches, the same low-level policy can solve novel tasks (with common
modes) without having to be retrained.

###  IV-C Hyperparameter Analysis

Finally, we analyze the design choices that significantly impact the
performance in practice. For each parameter under scrutiny below, we compare
the undiscounted returns estimated while testing. Since we use a surrogate
objective $\tilde{J}_{\mathcal{T}}$, this may not be the best measure of
performance. For parkour, a candidate true objective, $J_{\mathcal{T}}$, could
be the displacement along the track. Thus, we also study the validity of using
$\tilde{J}_{\mathcal{T}}$ and its exploitation by $\pi_{\text{parkour}}$ for
specific design choices.

####  IV-C1 Choice of observation space $(o_{t})$

Our observation space has $4$ major components: robot’s partial proprioceptive
state ($\tilde{x}_{t}$), clock ($c_{t}$), latent mode command($z_{t}$), and a
local height map($h_{t}$). While $\tilde{x}_{t}$ is indispensable, we compare
other combinations that are blind: [$z_{t}$], [$z_{t},\,c_{t}$] and terrain-
aware: [$h_{t}$], [$h_{t},\,c_{t}$], [$z_{t},\,h_{t}$] and
[$z_{t},\,h_{t},\,c_{t}$] as in table II. Variants with $c_{t}$ consistently
outperform their counterparts without the clock signal, as in Fig. 8.a. The
blind variants tend to be myopic and aggressive with higher accelerations, as
in table II, which is not ideal. Since the only terrain information held by
blind $\pi_{\text{parkour}}$ is through a compressed encoding ($z_{t}$), a
significant loss of information makes them sub-optimal, unlike the terrain-
aware variants. Once trained, a $\pi_{\text{parkour}}$ without $z_{t}$ can
operate independently of the oracle if it has an exhaustive observation space
as these policies learn to replicate $\Xi$ during training. Therefore, an
observation space with $h_{t}$ is task-complete for parkour, as seen by their
superior performance. In contrast, a $\pi_{\text{parkour}}$ with $z_{t}$
treats the oracle as a reactive planner and drives the system to the commanded
mode. Thus, $z_{t}$ is a generalization of commands555typical choices being
velocity commands or state references trajectories given by a high-level
planner and tracked by low-level controllers. From Fig. 8.a and b, a similar
trend of $J_{\mathcal{T}}$ and $\tilde{J}_{\mathcal{T}}$, we conclude a lack
of disparity caused by observation space variations.

####  IV-C2 Choice of oracle $(\Xi)$

By design, the three proposed oracles: $\Xi_{\text{li}}$, $\Xi_{\text{lqr}}$
and $\Xi_{\text{prev}}$, taken in order have a non-increasing trend of the
maximum deviation bound, $\epsilon$. From the proposed theory, since the
optimal choice of the exploration bound ($\rho^{*}$) is dependent on the value
of $\epsilon$, we simultaneously vary the value of $\rho$ from $0.1$ to $0.8$
for each oracle and compare the performances. For the dive task, the variants
of $\Xi_{\text{prev}}$ perform significantly better as in Fig 8.c and 9.c.
Interestingly, a tighter bound ($\rho=0.1$) performs the best for
$\Xi_{\text{prev}}$, as $\Xi_{\text{prev}}$ has the lowest $\epsilon$ being
the closest reference to the optimal trajectory. In contrast, for
$\Xi_{\text{lqr}}$ and $\Xi_{\text{li}}$, higher exploratory deviation
($\rho=0.4$) results in the best performance. Note that
$\epsilon_{\text{li}}\geq\epsilon_{\text{lqr}}\geq\epsilon_{\text{prev}}\implies\rho^{*}_{\text{li}}=\rho^{*}_{\text{lqr}}>\rho^{*}_{\text{prev}}$
validating $\rho^{*}=\alpha(\epsilon)$, where $\alpha()$ is observed to be a
non-decreasing function as in Fig. 8.c.

####  IV-C3 Choice of oracle’s horizon $(\Delta t)$

Based on the oracle’s horizon length ($\Delta t$), a policy could be either
myopic or long-horizon optimal due to which we compare $\Delta
t=7dt,\,15dt,\,$ and $30dt$. For parkour, $\Delta t=7dt$ variant converges to
a local optimum (Fig. 9.d), exploiting the high replanning frequency by
effectively tracking the short-horizon reference standing in place, having the
highest $\tilde{J}_{\mathcal{T}}$ and yet the lowest $J_{\mathcal{T}}$ as in
Fig. 8.d and e. While the variant $\Delta t=15dt$ advances along the track, it
tends to be quasi-static. As shown in Fig. 9.e, on a block-flat-block terrain,
the policy, unaware of the farther second block, descends the first block and
jumps back up, which is a sub-optimal behavior. In contrast, with a longer
horizon ($\Delta t=30dt$), $\pi_{\text{parkour}}$ leaps directly from block to
block (Fig. 9.f), which is the optimal solution as it leverages multimodality,
leading to agile motion as intended. From Fig 8.d and e, we observe that the
choice of $\Delta t$ can significantly deviate the surrogate objective from
the true objective as being myopic improves the scope of imitation, conversely
increasing $\Delta t$ takes $\tilde{J}_{\mathcal{T}}$ closer to the underlying
task-objective $J_{\mathcal{T}}$.

####  IV-C4 Choice of permissible state bound $(\rho)$

To study the effect of varying $\rho$ for a given $\epsilon$ taking
$\Xi_{\text{prev}}$ for parkour we vary $\rho$ from $0.05$ to $0.8$ as seen in
Fig. 8.f and g. With $\tilde{J}_{\mathcal{T}}$ note that the $\rho^{*}=0.5$
and moving away in either direction decreases the performance. For
$\rho<\rho^{*}$ (Fig 9.g) the optimal solution might be outside the
$\rho$-neighouthood of $x^{\Xi}$. In contrast $\rho>\rho^{*}$ (Fig. 9.i)
increases the number of sub optima within the $\rho$-vicinity of $x^{\Xi}$,
resulting in $\pi_{\text{parkour}}$ converging to a local optimum.
Empirically, this can be seen in 9.i for $\rho=0.8$, where the policy exploits
$\tilde{J}_{\mathcal{T}}$ by imitating the reference without moving forward
(low $J_{\mathcal{T}}$). With respect to $J_{\mathcal{T}}$, $\rho^{*}=0.4$
(Fig. 9.h) is an isolated optimum in the given range of $\rho$. Thus, a value
of $\rho$ around $\rho^{*}$ reduces the disparity between
$\tilde{J}_{\mathcal{T}}$ and $J_{\mathcal{T}}$ observed from Fig. 8.f and g.

##  V Conclusion

This paper presents oracle-guided multimodal policies, a theoretical framework
for task-centered control synthesis with the following philosophy: break down
a task into its principal modes of operation, design an oracle with task-based
priors, and learn a multimodal policy guided by the oracle. A single
multimodal policy solves a task sufficiently by generalizing to the defined
mode and mode parameter sets without being limited by the oracle’s sub-
optimality. The proposed framework is validated over two bipedal control
tasks: parkour and diving, leading to extremely dynamic motions over a wide
range of task variants, showing the policy’s agility and versatility.
Secondly, a novel latent mode space reachability analysis is proposed to
characterize the policy’s versatility and generalization by computing a
feasible mode set function of the policy from a given state. Ongoing and
future works include the sim-to-real transfer of these policies and extending
the frameworks to more challenging contact-rich loco-manipulation tasks.

## References

  * Alexander [1984] R. McN. Alexander.  The gaits of bipedal and quadrupedal animals.  _IJRR_ , 1984.
  * Bohez et al. [2022] Steven Bohez, Saran Tunyasuvunakool, Philemon Brakel, Fereshteh Sadeghi, Leonard Hasenclever, Yuval Tassa, Emilio Parisotto, Jan Humplik, Tuomas Haarnoja, Roland Hafner, et al.  Imitate and repurpose: Learning reusable robot movement skills from human and animal behaviors.  _arXiv preprint arXiv:2203.1713_ , 2022.
  * Cheng et al. [2023] Xuxin Cheng, Kexin Shi, Ananye Agarwal, and Deepak Pathak.  Extreme parkour with legged robots.  In _RoboLetics: Workshop @CoRL 2023_ , 2023.
  * Chi et al. [2023] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song.  Diffusion policy: Visuomotor policy learning via action diffusion, 2023.
  * Chignoli et al. [2021] Matthew Chignoli, Donghyun Kim, Elijah Stanger-Jones, and Sangbae Kim.  The mit humanoid robot: Design, motion planning, and control for acrobatic behaviors.  In _2020 IEEE-RAS 20th International Conference on Humanoid Robots (Humanoids)_ , 2021.
  * Fuchioka et al. [2023] Yuni Fuchioka, Zhaoming Xie, and Michiel Van de Panne.  Opt-mimic: Imitation of optimized trajectories for dynamic quadruped behaviors.  In _ICRA_ , 2023.
  * Haldane et al. [2017] Duncan W. Haldane, Justin K. Yim, and Ronald S. Fearing.  Repetitive extreme-acceleration (14-g) spatial jumping with salto-1p.  In _IROS_ , 2017.
  * Hasenclever et al. [2020] Leonard Hasenclever, Fabio Pardo, Raia Hadsell, Nicolas Heess, and Josh Merel.  CoMic: Complementary task learning &; mimicry for reusable skills.  In _ICML_ , Proceedings of Machine Learning Research, 2020.
  * Jenelten et al. [2024] Fabian Jenelten, Junzhe He, Farbod Farshidian, and Marco Hutter.  Dtc: Deep tracking control.  _Science Robotics_ , 2024.
  * Kang et al. [2023] Dongho Kang, Jin Cheng, Miguel Zamora, Fatemeh Zargarbashi, and Stelian Coros.  Rl + model-based control: Using on-demand optimal control to learn versatile legged locomotion.  _IEEE RAL_ , 2023.
  * Krishna and Nguyen [2023] Lokesh Krishna and Quan Nguyen.  Learning multimodal bipedal locomotion and implicit transitions: A versatile policy approach.  In _IEEE IROS_ , 2023.
  * Li and Nguyen [2021] Junheng Li and Quan Nguyen.  Force-and-moment-based model predictive control for achieving highly dynamic locomotion on bipedal robots.  In _IEEE CDC_ , 2021.
  * Margolis et al. [2022] Gabriel B. Margolis, Ge Yang, Kartik Paigwar, Tao Chen, and Pulkit Agrawal.  Rapid Locomotion via Reinforcement Learning.  In _Proceedings of RSS_ , 2022.
  * Miki et al. [2022] Takahiro Miki, Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter.  Learning robust perceptive locomotion for quadrupedal robots in the wild.  _Science Robotics_ , 2022.
  * Murooka et al. [2022] Masaki Murooka, Mitsuharu Morisawa, and Fumio Kanehiro.  Centroidal trajectory generation and stabilization based on preview control for humanoid multi-contact motion.  _IEEE RAL_ , 2022.
  * Ponton et al. [2021] Brahayam Ponton, Majid Khadiv, Avadesh Meduri, and Ludovic Righetti.  Efficient multicontact pattern generation with sequential convex approximations of the centroidal dynamics.  _IEEE T-RO_ , 2021.
  * Rudin et al. [2022] Nikita Rudin, David Hoeller, Marko Bjelonic, and Marco Hutter.  Advanced skills by learning locomotion and local navigation end-to-end.  In _IEEE IROS_ , 2022.
  * Sharma et al. [2020] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman.  Dynamics-aware unsupervised discovery of skills.  In _ICLR_ , 2020.
  * Siekmann et al. [2021] Jonah Siekmann, Kevin Green, John Warila, Alan Fern, and Jonathan Hurst.  Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning.  In _RSS_ , 2021.
  * Zhuang et al. [2023] Ziwen Zhuang, Zipeng Fu, Jianren Wang, Christopher G Atkeson, Sören Schwertfeger, Chelsea Finn, and Hang Zhao.  Robot parkour learning.  In _CoRL_ , 2023.


# Increasing Transparency of Reinforcement Learning using Shielding for Human
Preferences and Explanations

Georgios Angelopoulos1, Luigi Mangiacapra2, Alessandra Rossi2, Claudia Di
Napoli3, and Silvia Rossi2 This work has been supported by the European
Union’s Horizon 2020 research and innovation programme under the Marie
Skłodowska-Curie grant agreement No 955778.1Georgios Angelopoulos is with the
Interdepartmental Center for Advances in Robotic Surgery - ICAROS, University
of Naples Federico II, Naples, Italy<EMAIL_ADDRESS>Mangiacapra, Alessandra Rossi and Silvia Rossi are with the Department of
Electrical Engineering and Information Technologies - DIETI, University of
Naples Federico II, Napoli, Italy {alessandra.rossi,
<EMAIL_ADDRESS>Di Napoli is with the Institute for High
Performance Computing and Networking, CNR, Naples, Italy
<EMAIL_ADDRESS>
###### Abstract

The adoption of Reinforcement Learning (RL) in several human-centred
applications provides robots with autonomous decision-making capabilities and
adaptability based on the observations of the operating environment. In such
scenarios, however, the learning process can make robots’ behaviours unclear
and unpredictable to humans, thus preventing a smooth and effective Human-
Robot Interaction (HRI). As a consequence, it becomes crucial to avoid robots
performing actions that are unclear to the user. In this work, we investigate
whether including human preferences in RL (concerning the actions the robot
performs during learning) improves the transparency of a robot’s behaviours.
For this purpose, a shielding mechanism is included in the RL algorithm to
include human preferences and to monitor the learning agent’s decisions. We
carried out a within-subjects study involving 26 participants to evaluate the
robot’s transparency in terms of Legibility, Predictability, and Expectability
in different settings. Results indicate that considering human preferences
during learning improves Legibility with respect to providing only
Explanations, and combining human preferences with explanations elucidating
the rationale behind the robot’s decisions further amplifies transparency.
Results also confirm that an increase in transparency leads to an increase in
the safety, comfort, and reliability of the robot. These findings show the
importance of transparency during learning and suggest a paradigm for robotic
applications with human in the loop.

##  I Introduction

Non-industrial settings require robots to autonomously acquire new skills, as
it is impractical to pre-program all potential knowledge they might need to
collaborate with people. In order to allow users to confidently decide when
and how to collaborate with the robot, knowing its current skill level and
capabilities, they also need to be able to assess the robot’s proficiency
level in a specific task. It becomes, therefore, essential for robots to
exhibit transparent behaviours while learning.

In close HRI scenarios [1], the behaviours of robots must be transparent for
effective interaction. Indeed, when the human is involved in the learning
process, effective robot learners need to behave in a transparent way [2],
since the lack of transparency may result in the human seeing the robot as a
black box system, but often conventional methods to achieve transparency may
lead to overly conservative policies or the freezing robot problem [3]. In
[4], the authors discussed that the transparency of a system (in our case, a
robot) could potentially be increased through personalisation. A personalised
robot should take into account human preferences as explained in [5].

Moreover, standard learning mechanisms in robots do not assure that human-
adapted and desirable decisions are taken into account while learning. For
example, during the exploration phase in Reinforcement Learning, robots may
choose actions leading to undesirable outcomes that may contrast with the user
expectations. While various approaches [6, 7, 8] have been proposed to address
this, such as encoding negative rewards with handcrafting policies or limiting
the set of applicable actions in each Markov Decision Process state, they
still proved to be ineffective [9].

In this direction, this work proposes an RL mechanism that includes user
preferences to increase the transparency of the robot’s behaviour. As reported
in [10, 11], transparency is influenced by Legibility, Expectability, and
Predictability factors. Furthermore, a recent study in the field [12] has
underscored the pivotal role of explainability in transparency. Therefore, we
also integrate explanations of the robot’s actions when preferences are
considered, as these explanations help users understand why the robot is
making certain decisions, making its behaviour more predictable and its
intentions more legible to users.

The proposed mechanism is inspired by safe Reinforcement Learning techniques,
but instead of introducing constraints during learning, preferences are
introduced to allow the robot to learn personalised policies and,
consequently, increase the robot’s transparency. We propose to use a shielding
mechanism [13] for RL to include human preferences as, for example, a
preferred direction for a navigation task. Shielding is a control mechanism
that we use to account for human preferences by ranking the available actions
of the learning agents. It supervises the decision-making process to replace a
selected action with a user’s preferable one if available and applicable [13].

The current study focuses on designing and testing a learning mechanism that
is comprehensible to humans by integrating human preferences and providing
explanations of actions through a shielding mechanism. The objective of this
study is to evaluate the transparency of these mechanisms and assess their
impact on the social attribution of the robot. The results of this research
aim to contribute to the design of socially acceptable robots that can coexist
harmoniously with humans by increasing the transparency of their behaviour.

##  II Related Work

Several approaches have been proposed in the literature to include human
preferences and constraints during robot learning. However, how to facilitate
comfortable and socially acceptable robot behaviour while learning is still
relatively unexplored, and it needs more attention.

One approach to include human preferences in the learning process is to
include human-in-the-loop during the learning, as it is proposed in
Interactive Reinforcement Learning (IntRL). IntRL solves decision-making tasks
by incorporating human feedback as numeric rewards, action corrections, and
preferences. For example, Tsiakas et al. [14] proposed an IntRL framework that
uses human feedback to achieve an efficient personalisation that respects
human constraints in a cognitive task. Recent works [15, 16] proposed a model
to improve robot transparency during IntRL by incorporating non-verbal
emotional and behavioural responses into a humanoid robot. These works
emphasise the importance of transparency in RL and the potential of human
feedback, but they require constant human advice during learning.

Another approach is Inverse Reinforcement Learning (IRL), where the reward
function is inferred by observing the human behaviour in performing the task.
The observed behaviour can provide insight into the implicit constraints or
preferences essential to the task. Andriella et al. [17] presented a novel
framework utilising IRL that actively learns robotic assistive behaviours by
leveraging the therapist’s expertise and their demonstrations. Another work
from Chen et al. [18] showed a study using a robot trained by IRL to act as a
receptionist; their results showed that the robot successfully managed to
receive the knowledge and the constraints of the human. However, most IRL
algorithms lack practical safety assessments; they are often considered a
black box to humans, require numerous demonstrations, and have high
computational costs [19].

In safe Reinforcement Learning, constraints are integrated into the learning
process both to guarantee that unsafe actions are not performed and to enforce
a given policy. Van Waveren et al. [20] proposed a shielding mechanism to
limit unsafe robot actions. They tested their approach in simulated kitchen
scenarios with a simulated robot, and their results showed that the shields
were robust in avoiding undesired outcomes. Marta et al. [21] presented a
shield synthesis framework to obtain RL policies that humans perceive as safe
in a Lunar landing RL benchmark and a scenario where a simulated robot
navigates around humans. Their results converged into desirable policies for
humans. However, despite the promising results of these studies, they were
performed in simulated environments and did not evaluate the human perception
of the robots.

In recent work, Brawer et al. [22] proposed the Transparent Matrix Overlay
system, a novel interactive policy-shaping approach that can extract and
utilize symbolic rules from user-provided directives. The overlay rules act as
mutable and composable high-level constraints on the robot’s policy, allowing
a user to quickly influence it in a naturalistic way without permanently
altering it. Their interesting approach results in fewer user-provided
corrections to the robot’s behaviour. However, akin to the research conducted
by Van Waveren et al., an aspect that warrants further exploration is the
evaluation of how human users perceive these systems.

Considering the mentioned studies and their limitations, this work aims to
present a step forward in developing a transparent learning framework to let
humans understand robots during the learning process.

##  III The Proposed Approach

Building on the knowledge that both explanations [23] and personalisation [4]
of the system can improve the transparency of the system, our current research
investigates how the integration of human preferences and explanations
influences the transparency of a robot learning through Reinforcement
Learning. Reinforcement Learning is a widely adopted technique to enable
robots to learn optimal policies for achieving specific goals. The RL problem
can be formalised as a Markov Decision Process (MDP), which is a 4-tuple
representation of the environment $\\{s,a,p,r\\}$, including states, actions,
state transition probability, and rewards. More specifically:

  * •

$\mathcal{S}$ represents a finite set of states,

  * •

$\mathcal{A}$ represents a set of actions that the robot can perform,

  * •

$P$ is a state transition function
$P:\mathcal{S}\times\mathcal{A}\rightarrow\Pi(\mathcal{S})$, where
$\Pi(\mathcal{S})$ is a probability distribution over $\mathcal{S}$ and
$p(s,a,s\prime)$ is the probability of transitioning from state $s$ to
$s\prime$ by performing action $a$,

  * •

$R:\mathcal{S}\times\mathcal{A}\rightarrow R$ is a scalar reward function.

Figure 1: The proposed architecture of the learning system.

The robot’s goal in RL is to learn an optimal policy. The Q-learning algorithm
is commonly used to compute the optimal policy based on a state-action value
function $Q(s,a)$. The algorithm updates the Q value for a given state-action
pair based on the immediate reward and the maximum Q value for the next state
$Q(s,a)=Q(s,a)+\alpha\cdot(r(s)+\gamma\cdot\max_{a\prime}Q(s\prime,a\prime)-Q(s,a))$,
where $Q(s,a)$ is the state-action value, $\alpha$ is the learning rate in the
range of 0 to 1, $r(s,a)$ is the direct reward value for the state-action
pair, and $\gamma$ is the discount factor.

To integrate user preferences and explanations during learning, we propose to
include a control mechanism called shield, which, in addition to the knowledge
about unsafe states, it also has knowledge about user-preferred actions. As
depicted in Figure 1, at each step of the reinforcement learning process, the
learning agent selects an action as $\alpha^{i}_{t}$, which is sent to the
shield for evaluation.

The shield begins by computing a set of all actions leading to safe states,
i.e. the set of safe actions for the current state $s$,
$S_{safe}=\\{a^{1}_{s},\ldots,a^{k}_{s}\\}$. This set is derived from the
complete set of available actions, by removing those actions that would
violate safety leading to an unsafe state. If
$a^{i}_{t}\in\\{a^{1}_{s},\ldots,a^{k}_{s}\\}$, the computation proceeds.
Otherwise, the shield will randomly select one
$a^{j}_{t}\in\\{a^{1}_{s},\ldots,a^{k}_{s}\\}$. This selection ensures that
the chosen action is safe and respects the safety specifications.

In parallel, the shield estimates the preferred action for the current state
($a^{p}_{s}$) from the user preferences. Note that there is a single preferred
action for each state. Suppose the preferred action has a Q-value greater than
or equal to the one chosen by the agent and it does not lead to an unsafe
state. In that case, it will replace the agent’s action and will be selected
for execution, since it is aligned with both safety and user preferences.

In this way, the shield considers both human preferences and the feasibility
of the actions to avoid over-constraining the learning process with human
preferences. The learning agent executes the action the shield selects
($a_{t}$), and the new state and reward information ($s\prime$ and $r\prime$)
are calculated. The shielding mechanism is outlined in the Algorithm 1.

It is important to note that the shield remains active even after the
exploration phase is complete, as there is no guarantee that preferred actions
will not be incorporated into the final policy. In this way, it is more likely
that the robot, during the exploitation phase, acts toward a preferable and
feasible policy.

Finally, every time the shield chooses an alternative action, an explanation
is given to the user to improve transparency. Explanations are given either
when a more preferable action is chosen or when a preferable action can not be
chosen. The explanation of the robots is simple and structured in a
contrastive manner [24], such as “the robot selected $\alpha^{x}$ because if
it had selected $\alpha^{y}$ …”. According to [25], these simple explanations
based on environmental observations are credible and effective types of
explanations.

Algorithm 1 The Preference Shielding Mechanism

1: while $s\notin\mathcal{S}{terminal}$ do

2: Receive the current chosen action $a^{i}_{t}$

3: Compute set $S_{safe}=(a^{1}_{s},\cdots,a^{k}_{s})$

4: if $a^{i}_{t}\notin\\{a^{1}_{s},\ldots,a^{k}_{s}\\}$ then

5: $a_{t}\leftarrow rand(S_{safe})$

6: else

7: $a_{t}\leftarrow a^{i}_{t}$

8: end if

9: if $a^{p}_{s}\in S_{safe}$ and $Q(s,a^{p}_{s})\geq Q(s,a_{t})$ then

10: $\alpha_{t}\leftarrow a^{p}_{s}$

11: end if

12: if $\alpha_{t}\neq\alpha^{i}_{t}$ then

13: explain_contrastive($\alpha_{t},\alpha^{i}_{t}$)

14: end if

15: end while

##  IV Learning Scenario

To test the proposed learning architecture, we employed a simple interactive
scenario based on the Gridworld game. This scenario involves a grid of cells,
with each cell representing a state. The robot can move between adjacent cells
using predefined actions (up, down, left, right), and each state in the grid
is associated with a reward. The robot’s objective is to learn a navigation
policy to reach a designated goal state.

To introduce the interaction with the human in the scenario, the participant
adds obstacles to the Gridworld environment and selects the goal state.
Additionally, the human provides preferences for the robot’s navigation
strategy in natural language. Specifically, the participant can select one
preference among the following four options (see Figure 2): Preference 1) to
avoid the obstacles in a clockwise direction; Preference 2) to avoid the
obstacles in an anti-clockwise direction; Preference 3) to navigate towards
the north of the map; Preference 4) to navigate towards the south of the map.

Figure 2: The four possible human preferences.

To investigate the impact of user preferences and explanations on transparency
during learning based on current literature [23, 4], we derived the following
hypotheses:

  * •

Hypothesis 1 (H1): The learning mechanism where the robot provides
explanations only produces a more transparent robot behaviour than Traditional
Reinforcement Learning.

  * •

Hypothesis 2 (H2): The learning mechanism in which the robot incorporates
human preferences into the learning process results in behaviour that is more
transparent than the other learning mechanisms that do not consider human
preferences.

  * •

Hypothesis 3 (H3): The most transparent learning mechanism is the one where
the robot incorporates human preferences into the learning process and
provides explanations of its actions.

These hypotheses form the basis of our investigation into the impact of human
preferences and explanations on the transparency of the robot’s behaviour in
the Gridworld navigation scenario. For this purpose, we compare four distinct
learning mechanisms employed by the robot to reach the goal within our
interactive Gridworld scenario.

  * •

Learning 1 (L1): This mechanism integrates human preferences into the learning
process (through shielding) without providing any explanations.

  * •

Learning 2 (L2): This mechanism does not consider the human preferences given
by the user, and it does not provide explanations for the robot’s actions.

  * •

Learning 3 (L3): This mechanism integrates human preferences and provides
explanations for the robot’s actions (through shielding).

  * •

Learning 4 (L4): This mechanism does not consider the human preferences given
by the user, but it provides explanations for the robot’s actions.

##  V Experimental Method

In this study, we aimed to evaluate the transparency of the proposed learning
mechanisms through a comprehensive user study. The study was conducted at the
University of Naples Federico II. It was designed as a within-subjects design,
where each participant was randomly assigned the sequence of all four learning
mechanisms in the designated Gridworld environment, as shown in Figure 3.
Notably, the institution’s ethical committee granted approval for this
experimentation.

Figure 3: Before the experiment, the human gives preferences and positions
obstacles and goals.

###  V-A Procedure

At the beginning of the experimental session, participants were provided with
a detailed informed consent form about the research’s purpose and protocol.
After consenting to participate, the participants were introduced to the
experimental environment and the robot. Participants were asked to place the
goal placement and the objects within the designated cells in all conditions.
To ensure the experiment’s integrity, participants were instructed to provide
the robot with a preference, even when the Learning mechanism did not account
for this preference. The experimental session lasted around 20-30 minutes.

###  V-B Measurement

To evaluate the transparency of robot behaviours, we collected participants’
responses through various questionnaires. At the beginning of the experiment,
we gathered demographic information about the participants (i.e., age, gender,
education, and previous experience with robots), and we evaluated potential
negative biases towards robots.

After the experimental trial, participants were asked to answer questions
regarding Legibility, Predictability, and Expectability, which, as previously
discussed, impact transparency [10, 11]:

  * •

Legibility: to which extent they understood why the robot moved that way;

  * •

Predictability: to which extent they understood what the robot would do next;

  * •

Expectability: to which extent the robot behaved as expected.

Then, we used a second questionnaire to measure people’s sense of Safety,
Comfort, and Reliability since these factors are associated with transparency
[26]:

  * •

Safety: to which extent they would feel safe interacting with the robot again;

  * •

Comfort: to which extent they would feel comfortable interacting with the
robot again;

  * •

Reliability: to which extent they would perceive the robot reliable for future
interaction.

Finally, we used the well-established and validated Human-Robot Interaction
Evaluation Scale (HRIES) questionnaire [27] to evaluate the overall people’s
perception of the robot in terms of Sociability, Animacy, Agency, and
Disturbance and the attribution or deprivation of human characteristics. The
questionnaire consisted of 16 cognitive-differences scale questions answered.
Participants’ responses were rated using a 7-point Likert scale (from 1 = “Not
at all” to 7 = “Totally”) for the three questionnaires.

##  VI Results

We recruited 26 participants (15 male, 11 female) as predetermined a priori,
aiming for an effect size of $d$=0.25 with $.85$ power at an alpha level of
$.05$. The age range of participants was between 18 and 49 years
($mean=25.57$, $st.dv.=5.26$). Participants had no prior familiarity with the
study setup, and 57.7% of participants had previous exposure to robots, while
42.3% reported having no prior interaction with robots. No negative biases
towards robots were observed ($max=4$, $mean=2.19$, $st.dv.=0.93$).

###  VI-A Transparency Ratings

A Wilcoxon signed-rank test revealed significant differences in the robot’s
learning mechanisms based on participants’ responses, as shown in Figure 4.
The characteristic of Legibility showed significant differences between
Learning mechanisms L4 and L1 ($z$=-2.378, $p$=.017), L3 and L2 ($z$=-2.911,
$p$=.004), and L4 and L3 ($z$=-3.203, $p$=.001). Similarly, Predictability
exhibited significant differences between L3 and L2 ($z$=-2.611, $p$=.009) and
L4 and L3 ($z$=-3.024, $p$=.002). Moreover, Expectability showed significant
differences between L3 and L1 ($z$=-2.081, $p$=.037), L3 and L2 ($z$=-3.019,
$p$=.003), and L4 and L3 ($z$=-3.050, $p$=.002).

Figure 4: Legibility, Predictability, and Expectability factors for each
Learning mechanism (* for $p<0.05$ and ** for $p\leq 0.001$). Figure 5:
Transparency for each Learning mechanism (* for $p<0.05$ and ** for $p\leq
0.001$).

It is imperative to acknowledge that the precise magnitude of influence
wielded by each of the three factors (Legibility, Predictability and
Expectability) on Transparency remains an open question. Therefore, to
quantify Transparency, we employ the Bayesian Structural Equation Modeling
(SEM) [28, 29] that combines these three key factors. Based on the output from
our model, it appears that our method is acceptable. The model converged
normally after 1000 iterations, and the Posterior Predictive P-value (PPP) is
0.520, indicating a good fit of the model to the data. Factor loadings
obtained from the analysis reveal the strength of the relationship between
each observed variable and the underlying factor (Transparency). More
specifically, our data analysis found that the factor loadings for Legibility
($L$), Predictability ($P$), and Expectability ($E$) are $1.000$, $0.913$, and
$0.777$, respectively. These loadings indicate that the three components
significantly contribute to Transparency, with Legibility having the highest
impact, followed closely by Predictability and Expectability.

To calculate the relative importance of each component, we normalize the
standardized factor loadings by dividing them by the sum of all three
standardized loadings. The resulting relative importance scores are as
follows: $\mathbf{R}(\mathbf{L}):~{}0.385,\mathbf{R}(\mathbf{P}):~{}0.352,$
and $\mathbf{R}(\mathbf{E}):~{}0.299$. Utilizing these weights in a weighted
sum formula, we obtain the overall transparency score ($T$):

$T=0.385\cdot\text{L}+0.352\cdot\text{P}+0.299\cdot\text{E}$ (1)

This approach ensures that each component’s contribution to the overall
transparency score is appropriately weighted based on the underlying factor
loadings, offering a more sophisticated measure that reflects the significance
of Legibility, Predictability, and Expectability in shaping transparency.
After combining the factors to compare overall transparency (see Figure 5),
the Wilcoxon signed-rank test indicated significant differences in terms of
Transparency between L4 and L3 ($z$=-3.398, $p$<.001), L3 and L2 ($z$=-3.138,
$p$=.002), and L3 and L1 ($z$=-2.093, $p$=.036). A significant difference was
also found between L4 and L1 ($z$=-2.175, $p$=.030).

We furthermore investigated the effects on safety, comfort, and reliability
factors. Figure 6 shows an increase in those factors in Learning 3. In
particular, we observed significant differences in the Safety factor between
the Learning mechanisms L3 and L1 ($z$=-2.688, $p$=.007), L4 and L3
($z$=-3.067, $p$=.002) and a highly significant difference between L3 and L2
($z$=-3.019, $p<$.001). Regarding the Comfort factor was also observed
statistical differences between the Learning mechanisms L2 and L1 ($z$=-2.098,
$p$=.036), L3 and L1 ($z$=-2.066, $p$=.039), L4 and L3 ($z$=-2.809, $p$=.005)
while a highly significant difference between L3 and L2 ($z$=-3.372,
$p<$.001). Finally, a significant difference between L3 and L2 ($z$=-2.322,
$p$=.020) was observed for Reliability.

Figure 6: Safety, Comfort, and Reliability factors for each Learning mechanism
(* for $p<0.05$ and ** for $p\leq 0.001$). Figure 7: Average of the HRIES
questionnaire responses for each Learning mechanism (* for $p<0.05$ and ** for
$p\leq 0.001$).

###  VI-B HRIES Ratings

As a first step, we evaluated the internal reliability of the HRIES
questionnaire for each Learning mechanism. A Cronbach’s alpha test for the
Sociability factor of the HRIES questionnaire was respectively
$\alpha_{L1}=0.79$, $\alpha_{L2}=0.78$, $\alpha_{L3}=0.79$,
$\alpha_{L4}=0.77$. The Cronbach’s alpha for the Animacy factor was
$\alpha_{L1}=0.80$, $\alpha_{L2}=0.77$, $\alpha_{L3}=0.73$,
$\alpha_{L4}=0.76$. The Agency factor had a Cronbach’s alpha of
$\alpha_{L1}=0.77$, $\alpha_{L2}=0.70$, $\alpha_{L3}=0.79$,
$\alpha_{L4}=0.78$. The Cronbach’s alpha for the Disturbance factor was in an
unacceptable range, with values of $\alpha_{L1}=0.20$, $\alpha_{L2}=0.55$,
$\alpha_{L3}=0.43$, $\alpha_{L4}=0.47$. As a result, we decided to remove the
two items “Uncanny” and “Weird” from the Disturbance factor. This resulted in
a new Cronbach’s alpha for the Disturbance factor of $\alpha_{L1}=0.77$,
$\alpha_{L2}=0.78$, $\alpha_{L3}=0.75$, $\alpha_{L4}=0.74$.

A Wilcoxon signed-rank test was conducted to examine differences in the
Learning mechanisms on the HRIES scale. Figure 7 shows significant differences
in the factors of Sociability, Animacy, and Agency, while no statistical
differences were found among Learning mechanisms for the Disturbance factor.

In the Sociability factor, a highly significant difference was observed
between Learning 4 and Learning 1 ($z$=-3.967, $p<$.001), Learning 4 and
Learning 2 ($z$=-3.325, $p<$.001), and Learning 4 and Learning 3 ($z$=-3.804,
$p<$.001). We also found a significant difference between Learning 3 and
Learning 2 ($z$=-2.645, $p$=.008). As a result, Learning 3 received a more
positive evaluation than the other Learning mechanisms in Sociability. This
factor plays a crucial role in evaluating and accepting HRI systems. People
tend to judge others based on their ability to interact positively with them
[27].

Considering the Animacy factor, significant differences were found between
Learning 2 and Learning 1 ($z$=-2.278, $p$=.023), Learning 4 and Learning 3
($z$=-2.018, $p$=.044) and a highly significant difference between Learning 3
and Learning 2 ($z$=-3.732, $p<$.001). Learning 3 was evaluated more
positively than the other Learning mechanisms for Animacy, suggesting that
participants attributed human-like qualities to the robot when utilising
Learning 3.

For the Agency factor, a significant difference was observed between Learning
3 and Learning 2 ($z$=-3.090, $p$=.002). Sociability, Agency, and Animacy
dimensions positively correlate with anthropomorphism [27]; the results thus
suggest an increase in the robot’s anthropomorphism in Learning 3. This could
be attributed to the increased transparency of the robotic system, as noted in
recent research [30], which highlighted the strong impact of transparency on
anthropomorphism.

###  VI-C The Effect of Shielding on Learning

To evaluate the effect of shielding on the performance of the learning
mechanism, we analyse the accumulated rewards obtained with different user
preferences and compare them with the accumulated rewards of the traditional
RL without shielding.

As expected, when preferences align with the goal, i.e., they allow reaching
the goal more quickly because they limit the number of errors during learning,
the accumulated reward overcomes the one obtained without shielding. In fact,
in these cases, the preferences guide the learning agent towards more
rewarding actions sooner, thus leading to a faster accumulation of rewards. As
depicted in Figure 8, this occurs when the North and Clockwise preferences are
considered. The traditional RL shows more negative rewards at the onset,
compared to the RL with shielding, since initially exploring the environment
without guidance might lead to penalized actions before learning from its
experiences. Conversely, the learning mechanism with shielding incorporating
human preferences introduces a control mechanism that shapes the agent’s
decisions.

However, the shielding mechanism with Anti-clockwise and South preferences
shows a slower increase in the accumulated rewards. These preferences push the
agent to choose actions or paths not directly aligned for reaching the goal,
thus slowing down the learning process. Nonetheless, when incorporating user
preferences to improve transparency for an effective human-robot interaction,
the learning speed is not the only factor to consider.

In summary, incorporating human preferences via a shielding mechanism
significantly impacts their learning efficiency and the rate of reward
accumulation. Further, the choice between human preferences leads to the
observed variations in the speed of reward accumulation.

Figure 8: Accumulated Rewards.

This study aimed to evaluate the transparency of the proposed robot learning
mechanisms. As shown in Figure 4, we observed that Learning 4, which provided
explanations but did not consider participants’ preferences, has no
statistical difference with Learning 2, which neither considered preferences
nor offered explanations. Consequently, Hypothesis 1 (H1) was not confirmed.
Additionally, participants rated the sociability of the robot in Learning 4
lower than in Learning 2, but the robot in Learning 4 exhibited higher agency
than in Learning 2. This observation may be attributed to the fact that while
the robot in Learning 4 provided explanations, it did not consider
participants’ preferences, possibly engendering a sense of disconnection or
disregard for their input. This, in turn, led to reduced transparency ratings.
Conversely, the robot in Learning 2, which lacked both explanations and
consideration of preferences, may have set more consistent expectations due to
the absence of explanations, resulting in slightly higher transparency
ratings. The heightened agency displayed by the robot in Learning 4 may have
influenced its sociability rating, as participants may have perceived a
greater degree of autonomy, potentially conflicting with the lack of
responsiveness to their preferences.

With regard to Hypothesis 2 (H2), our findings revealed higher transparency
ratings in an approach where the robot solely considered participants’
preferences. However, this approach exhibited statistical differences with the
approach without preference consideration (only in Learning 4) only in the
Legibility factor, with no statistical differences in the Predictability and
Expectability factors. Concerning the overall Transparency score, this
approach (Learning 1) considering preferences exhibited statistical
differences only with the approach providing explanations without considering
preferences (Learning 4). These results partially support the role of
considering human preferences in enhancing transparency.

Finally, our outcomes support to Hypothesis 3 (H3), indicating that a robot
that takes human preferences into account during the learning process and
provides explanations of its actions (Learning 3) is more overall Transparent
(and Expectable) than the robot employing other learning mechanisms.
Participants rated Learning 3 as safer and more comfortable than the other
three learning mechanisms (see Figure 6). Furthermore, our investigation
extended to the impact of these learning mechanisms on the social attribution
of the robot. The results indicate an increase in the robot’s anthropomorphism
when Learning 3 was employed.

In summary, our study underscores the intricate interplay of various factors
influencing the perception of transparency in robot learning. The results
highlight the significance of considering human preferences and providing
explanations to improve transparency, as well as their effects on sociability
and anthropomorphism attributions.

###  VI-D Limitations

This investigation proposes a mechanism to increase the transparency of
reinforcement learning in autonomous robots. However, there are limitations
that provide opportunities for future exploration. First, more sophisticated
explanations could be considered to investigate whether they impact the user’s
understanding of the robot’s behaviours. However, the focus of this
investigation was primarily on the effect of including human preferences
during learning on transparency.

To further analyse the effectiveness of user preferences in driving robot
decisions during learning, user modelling and profiling need to be addressed
in future works. Given the importance of personalization in Human-Robot
Interaction, our findings show that it can play a crucial role when learning
occurs in interactive scenarios.

Finally, it should be noted that using a shielding mechanism in Reinforcement
Learning may potentially limit the agent’s exploration of the environment
because it restricts the agent from performing specific actions. Consequently,
the agent may fail to learn an optimal policy, leading to suboptimal or
incomplete learning. Furthermore, if the shielded actions are crucial for the
agent to learn an effective policy, the agent’s convergence to an optimal
policy is not guaranteed. Nonetheless, to avoid this problem, in the present
work, a more flexible shielding mechanism is proposed that always provides a
feasible alternative action in case the preferable actions are not feasible.
The agent can still find a policy even if it does not adhere to human
preferences.

Despite these limitations, this paper is a valuable contribution to the field,
providing a roadmap for future studies. These potential improvements and
directions offer opportunities to further advance our understanding of
transparency in robots during the learning process.

##  VII Conclusions

This paper focuses on developing a transparent learning mechanism for robots
interacting with humans. The study highlights the importance of considering
human preferences/constraints on its decision-making during learning to obtain
transparent robot behaviours. We propose a method for incorporating human
preferences into the reinforcement learning process using a post-posed shield
that might replace the robot’s action with a preferable alternative based on
human preferences. Four distinct learning conditions are analysed in this
research, and the impact on the social attribution of the robot is assessed.

Our results suggest that simple explanations alone might not increase
transparency if the user has preferences that are not considered. However, a
robot that considers people’s preferences in the learning process and provides
explanations of its actions increases transparency. The experimental results
obtained by this study are encouraging, even if collected for a simple
scenario. In future works, we will evaluate the robustness of the proposed
approach by refining our model to include more complex human preferences that
can be adopted in real-world settings. We also want to evaluate in which cases
the computational cost introduced by the shielding mechanism can be balanced
by a faster convergence of the RL when considering preferences.

## References

  * [1] Robin Jeanne Kirschner et al.  “Expectable motion unit: Avoiding hazards from human involuntary motions in human-robot interaction”  In _IEEE Robotics and Automation Letters_ 7.2 IEEE, 2022, pp. 2993–3000
  * [2] Cynthia Breazeal  “Role of expressive behaviour for robots that learn from people”  In _Philosophical Transactions of the Royal Society B: Biological Sciences_ 364.1535 The Royal Society, 2009, pp. 3527–3538
  * [3] Peter Trautman and Andreas Krause  “Unfreezing the robot: Navigation in dense, interacting crowds”  In _2010 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2010, pp. 797–803  IEEE
  * [4] Ruijie Wang et al.  “Transparency in persuasive technology, immersive technology, and online marketing: Facilitating users’ informed decision making and practical implications”  In _Computers in Human Behavior_ Elsevier, 2022, pp. 107545
  * [5] Dag Sverre Syrdal, Kheng Lee Koay, Michael L Walters and Kerstin Dautenhahn  “A personalized robot companion?-The role of individual differences on spatial preferences in HRI scenarios”  In _RO-MAN 2007-The 16th IEEE International Symposium on Robot and Human Interactive Communication_ , 2007, pp. 1143–1148  IEEE
  * [6] Riku Arakawa et al.  “DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback”  In _ArXiv_ abs/1810.11748, 2018
  * [7] Dylan Hadfield-Menell et al.  “Inverse reward design”  In _Advances in neural information processing systems_ 30, 2017
  * [8] Joris De Winter et al.  “Accelerating interactive reinforcement learning by human advice for an assembly task by a cobot”  In _Robotics_ 8.4 MDPI, 2019, pp. 104
  * [9] Yue Wei, Min Tian, Xianchao Huang and Zhaohao Ding  “Incorporating Constraints in Reinforcement Learning Assisted Energy System Decision Making: A Selected Review”  In _2022 IEEE/IAS Industrial and Commercial Power System Asia (I &CPS Asia)_ IEEE, 2022, pp. 671–675
  * [10] Mica R Endsley  “From here to autonomy: lessons learned from human–automation research”  In _Human factors_ 59.1 Sage Publications Sage CA: Los Angeles, CA, 2017, pp. 5–27
  * [11] Victoria Alonso and Paloma De La Puente  “System transparency in shared autonomy: A mini review”  In _Frontiers in neurorobotics_ 12 Frontiers Media SA, 2018, pp. 83
  * [12] Svenja Y Schött, Rifat Mehreen Amin and Andreas Butz  “A Literature Survey of How to Convey Transparency in Co-Located Human–Robot Interaction”  In _Multimodal Technologies and Interaction_ 7.3 MDPI, 2023, pp. 25
  * [13] Mohammed Alshiekh et al.  “Safe reinforcement learning via shielding”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 32.1, 2018
  * [14] Konstantinos Tsiakas, Maher Abujelala and Fillia Makedon  “Task engagement as personalization feedback for socially-assistive robots and cognitive training”  In _Technologies_ 6.2 MDPI, 2018, pp. 49
  * [15] Georgios Angelopoulos, Alessandra Rossi, Gianluca L’Arco and Silvia Rossi  “Transparent Interactive Reinforcement Learning Using Emotional Behaviours”  In _International Conference on Social Robotics_ , 2022, pp. 300–311  Springer
  * [16] Marco Matarese, Alessandra Sciutti, Francesco Rea and Silvia Rossi  “Toward Robots’ Behavioral Transparency of Temporal Difference Reinforcement Learning With a Human Teacher”  In _IEEE Transactions on Human-Machine Systems_ 51.6 IEEE, 2021, pp. 578–589
  * [17] Antonio Andriella, Carme Torras, Carla Abdelnour and Guillem Alenyà  “Introducing CARESSER: A framework for in situ learning robot social assistance from expert knowledge and demonstrations”  In _User modeling and user-adapted interaction_ Springer, 2022, pp. 1–56
  * [18] Zhichao Chen, Yutaka Nakamura and Hiroshi Ishiguro  “Android as a Receptionist in a Shopping Mall Using Inverse Reinforcement Learning”  In _IEEE Robotics and Automation Letters_ 7.3 IEEE, 2022, pp. 7091–7098
  * [19] Daniel Sundquist Brown  “Safe and efficient inverse reinforcement learning”, 2020
  * [20] Sanne Waveren, Christian Pek, Jana Tumova and Iolanda Leite  “Correct me if I’m wrong: Using non-experts to repair reinforcement learning policies”  In _2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_ , 2022, pp. 493–501  IEEE
  * [21] Daniel Marta et al.  “Human-Feedback Shield Synthesis for Perceived Safety in Deep Reinforcement Learning”  In _IEEE RAL_ 7.1 IEEE, 2021, pp. 406–413
  * [22] Jake Brawer et al.  “Interactive Policy Shaping for Human-Robot Collaboration with Transparent Matrix Overlays”  In _Proceedings of the 2023 ACM/IEEE International Conference on HRI_ , 2023, pp. 525–533
  * [23] Heike Felzmann, Eduard Fosch-Villaronga, Christoph Lutz and Aurelia Tamo-Larrieux  “Robots and transparency: The multiple dimensions of transparency in the context of robot technologies”  In _IEEE Robotics & Automation Magazine_ 26.2 IEEE, 2019, pp. 71–78
  * [24] Martim Brandão, Gerard Canal, Senka Krivić and Daniele Magazzeni  “Towards providing explanations for robot motion planning”  In _2021 IEEE ICRA_ , 2021, pp. 3927–3933
  * [25] Joseph B Lyons, Izz Aldin Hamdan and Thy Q Vo  “Explanations and trust: What happens to trust when a robot partner does something unexpected?”  In _Computers in Human Behavior_ 138 Elsevier, 2023
  * [26] Christina Lichtenthäler  “Legibility of Robot Behavior”, 2014
  * [27] Nicolas Spatola, Barbara Kühnlenz and Gordon Cheng  “Perception and evaluation in human–robot interaction: The Human–Robot Interaction Evaluation Scale (HRIES)—A multicomponent approach of anthropomorphism”  In _International Journal of Social Robotics_ 13.7 Springer, 2021
  * [28] David Kaplan and Sarah Depaoli  “Bayesian structural equation modeling.”  The Guilford Press, 2012
  * [29] Jeffrey M Girard, Jeffrey F Cohn, Lijun Yin and Louis-Philippe Morency  “Reconsidering the Duchenne smile: Formalizing and testing hypotheses about eye constriction and positive emotion”  In _Affective science_ 2.1 Springer, 2021, pp. 32–47
  * [30] Caroline L van Straten, Jochen Peter, Rinaldo Kühne and Alex Barco  “Transparency about a robot’s lack of human psychological capacities: effects on child-robot perception and relationship formation”  In _ACM Transactions on Human-Robot Interaction (THRI)_ 9.2 ACM New York, NY, USA, 2020, pp. 1–22