# Priors, Hierarchy, and Information Asymmetry for Skill Transfer in
Reinforcement Learning

Sasha Salter , Kristian Hartikainen∗, Walter Goodwin, Ingmar Posner
Applied AI Lab, University of Oxford,
{sasha, kristian, walter<EMAIL_ADDRESS>
Co-first authors.

###### Abstract

The ability to discover behaviours from past experience and transfer them to
new tasks is a hallmark of intelligent agents acting sample-efficiently in the
real world. Equipping embodied reinforcement learners with the same ability
may be crucial for their successful deployment in robotics. While hierarchical
and KL-regularized RL individually hold promise here, arguably a hybrid
approach could combine their respective benefits. Key to these fields is the
use of information asymmetry to bias which skills are learnt. While asymmetric
choice has a large influence on transferability, prior works have explored a
narrow range of asymmetries, primarily motivated by intuition. In this paper,
we theoretically and empirically show the crucial trade-off, controlled by
information asymmetry, between the _expressivity_ and _transferability_ of
skills across sequential tasks. Given this insight, we provide a principled
approach towards choosing asymmetry and apply our approach to a complex,
robotic block stacking domain, unsolvable by baselines, demonstrating the
effectiveness of hierarchical KL-regularized RL, coupled with correct
asymmetric choice, for sample-efficient transfer learning.

##  1 Introduction

While Reinforcement Learning (RL) algorithms recently achieved impressive
feats across a range of domains [1, 2, 3], they remain sample inefficient [4,
5] and are therefore of limited use for real-world robotics applications.
Intelligent agents during their lifetime discover and reuse skills at multiple
levels of behavioural and temporal abstraction to efficiently tackle new
situations. For example, in manipulation domains, beneficial abstractions
could include low-level instantaneous motor primitives as well as higher-level
object manipulation strategies. Endowing lifelong learning RL agents [6] with
a similar ability could be vital towards attaining comparable sample
efficiency.

To this end, two paradigms have recently been introduced. _KL-regularized_ RL
[7, 8] presents an intuitive approach for automating skill reuse in multi-task
learning. By regularizing policy behaviour against a learnt task-agnostic
prior, common behaviours across tasks are distilled into the prior, which
encourages their reuse. Concurrently, _hierarchical_ RL also enables skill
discovery [9, 10, 11, 12, 13] by considering a two-level hierarchy in which
the high-level policy is task-conditioned, whilst the low-level remains task-
agnostic. The lower level of the hierarchy therefore also discovers skills
that are transferable across tasks. Both hierarchy and priors offer their own
skill abstraction. However, when combined, one can in theory discover and
leverage multiple skill abstractions. Whilst prior works [14] have attempted
this, they were unable to yield transfer benefits from a learnt prior.

In fact, successful transfer for both approaches critically depends on the
correct choice of information asymmetry. IA more generally refers to an
asymmetric masking of information across architectural modules. This masking
forces independence to, and ideally generalisation across, the masked
dimensions [8]. Therefore, IA crucially biases learnt behaviour and how it
transfers across environments. Previous works have motivated their chosen IAs
primarily on intuition [14, 8, 9], which, if sub-optimal, limits transfer
benefits. We demonstrate that this indeed is the case for the hierarchical KL-
regularized method in [14]. A more systematic approach to identifying IA
schemes, underpinned by rigorous theory, is therefore required in order to
benefit from skills in lifelong learning.

In this paper, we employ hierarchical KL-regularized RL [14] to effectively
transfer skills across multiple abstraction levels. We begin by theoretically
showing the crucial trade-off, controlled by choice of IA, that exists between
the _expressivity_ and _transferability_ of skills across sequential tasks.
With this insight, we consider previously unexplored, distinct asymmetries
across hierarchical levels, policy and prior, to successfully benefit from
both priors and hierarchy. Our analysis provides concrete insights into which
design choices are crucial for effective transfer. Further, we demonstrate
that, while priors are significantly more effective than hierarchy, hierarchy
is still important for enabling expressive priors. We apply our approach to a
robot block-stacking, a complex, sparse-reward domain unsolvable by the state-
of-the-art baselines, and show that with the right choice of asymmetries,
sample-efficient learning can be achieved.

##  2 Reinforcement Learning for Skill Transfer

Our work considers multi-task reinforcement learning in Partially Observable
Markov Decision Processes (POMDPs), defined by
$M_{k}=(\mathcal{S},\mathcal{X},\mathcal{A},r_{k},p,p^{0}_{k},\gamma)$, where
tasks $k$ are sampled from $p(\mathcal{K})$. $\mathcal{S}$, $\mathcal{A}$, and
$\mathcal{X}$ denote observation, action, and observation-action history
spaces.
$p(\mathbf{x}^{\prime}|\mathbf{x},\mathbf{a}):\mathcal{X}\times\mathcal{X}\times\mathcal{A}\rightarrow\mathbb{R}_{\geq
0}$ represents the dynamics model. We denote the history of observations
$\mathbf{s}\in\mathcal{S}$ and actions $\mathbf{a}\in\mathcal{A}$ up to
timestep $t$ as
$\mathbf{x}_{t}=(\mathbf{s}_{0},\mathbf{a}_{0},\mathbf{s}_{1},\mathbf{a}_{1},\dots,{\mathbf{s}_{t}})$.
The reward functions
$r_{k}:\mathcal{X}\times\mathcal{A}\times\mathcal{K}\rightarrow\mathbb{R}$ are
history-, action- and task-dependent.

###  2.1 KL-Regularized Reinforcement Learning

The typical multi-task KL-regularized RL objective [15, 16, 17, 18] takes the
form:

$\displaystyle\mathcal{O}(\pi,\pi_{0})=\operatorname{\mathbb{E}}_{\begin{subarray}{c}\tau\sim\rho_{\pi},\\\
k\sim
p(\mathcal{K})\end{subarray}}\left[\sum_{t=0}^{\infty}\gamma^{t}\bigg{(}r_{k}({\mathbf{x}_{t}},{\mathbf{a}_{t}})-\alpha_{0}\mathrm{D_{KL}}\left(\pi(\mathbf{a}|{\mathbf{x}_{t}},k)\;\middle\|\;\pi_{0}(\mathbf{a}|{\mathbf{x}_{t}})\right)\bigg{)}\right]$
(1)

where $\gamma$ is the discount factor and $\alpha_{0}$ weighs the individual
objective terms. $\pi$ and $\pi_{0}$ denote the task-conditioned policy and
task-agnostic prior respectively. The expectation is taken over tasks and
trajectories from policy $\pi$ and initial state distribution
$p^{0}_{k}(\mathbf{s}_{0})$. When optimised with respect to $\pi$, this
objective can be viewed as a trade-off between maximising rewards whilst
remaining close to trajectories produced by $\pi_{0}$. When $\pi_{0}$ is
learnt, it can be viewed as a method for learning shared behaviours present
across tasks, and can thus bias multi-task exploration [7]. We consider the
sequential learning paradigm, where skills are learnt from past tasks,
$p_{past}(\mathcal{K})$, and leveraged while attempting the current set of
tasks, $p_{current}(\mathcal{K})$.

###  2.2 Hierarchical KL-Regularized RL

While KL-regularized RL has achieved success across various settings [4, 7,
19, 12], recently Tirumala et al. [14] proposed a hierarchical extension where
policy $\pi$ and prior $\pi_{0}$ are augmented with latent variables,
${\pi(\mathbf{a},\mathbf{z}|\mathbf{x},k)=\pi^{H}(\mathbf{z}|\mathbf{x},k)\pi^{L}(\mathbf{a}|\mathbf{z},\mathbf{x})}$
and
${\pi_{0}(\mathbf{a},\mathbf{z}|\mathbf{x})=\pi_{0}^{H}(\mathbf{z}|\mathbf{x})\pi_{0}^{L}(\mathbf{a}|\mathbf{z},\mathbf{x})}$,
where subscripts $H$ and $L$ denote the higher and lower hierarchical levels.
This structure encourages the shared low-level policy ($\pi^{L}=\pi^{L}_{0}$)
to discover task-agnostic behavioural primitives, whilst the high-level
discovers higher-level skills relevant to each task. By not conditioning the
high-level prior on task-id, Tirumala et al. [14] encourage the reuse of
common high-level abstractions across tasks. Tirumala et al. [14] propose the
following upper bound for approximating the KL-divergence between hierarchical
policy and prior:

$\displaystyle\mathrm{D_{KL}}\left(\pi(\mathbf{a}|\mathbf{x})\;\middle\|\;\pi_{0}(\mathbf{a}|\mathbf{x})\right)\leq\mathrm{D_{KL}}\left(\pi^{H}(\mathbf{z}|\mathbf{x})\;\middle\|\;\pi_{0}^{H}(\mathbf{z}|\mathbf{x})\right)+\operatorname{\mathbb{E}}_{\pi^{H}}\left[\mathrm{D_{KL}}\left(\pi^{L}(\mathbf{a}|\mathbf{x},\mathbf{z})\;\middle\|\;\pi_{0}^{L}(\mathbf{a}|\mathbf{x},\mathbf{z})\right)\right]$
(2)

where we omit task conditioning and explicitly declared shared modules to
emphasise that this bound is agnostic to these choices. Whilst in principle
this approach can be used to learn a high-level behavioural prior,
$\pi_{0}^{H}$, in practice Tirumala et al. [14] do not observe benefits from
learning it.

###  2.3 Information Asymmetry

Information Asymmetry (IA) is a key component in both of the aforementioned
approaches, promoting the discovery of behaviours that generalise. IA can be
understood as the masking of information accessible by certain modules. Not
conditioning on specific environment aspects forces independence and
generalisation across them [8]. In the context of hierarchical KL-regularized
RL, the explored asymmetries between the high-level policy, $\pi^{H}$, and
prior, $\pi_{0}^{H}$, have been narrow [14, 19]. Tirumala et al. [14], Pertsch
et al. [19] explore auto-regressive priors of the form:
$\pi_{0}^{H}(\mathbf{z}_{t}|\mathbf{x}_{t})=N(\mathbf{z}_{t}|\mu(\mathbf{z}_{t-1}),\sigma^{2}(\mathbf{z}_{t-1}))$
and
$\pi_{0}^{H}(\mathbf{z}_{t}|\mathbf{x}_{t})=N(\mathbf{z}_{t}|\mu(\mathbf{s}_{t}),\sigma^{2}(\mathbf{s}_{t}))$
respectively. Both choices condition on minimal information, limiting their
ability to distil and transfer knowledge across tasks. However, as discussed
next, with a more principled approach to choice of IA, richer behaviours can
be discovered and transferred.

##  3 Model Architecture and the
Expressivity-Transferability Trade-Off

Figure 1:  Hierarchical KL-regularized architecture. The hierarchical policy
modules $\pi^{H}$ and $\pi^{L}$ are regularized against their corresponding
prior modules $\pi_{i}^{H}$ and $\pi_{i}^{L}$. The inputs to each module are
filtered by an information gating function, depicted with colored rectangle.

To rigorously investigate the contribution of priors, hierarchy, and
information asymmetry on skill transfer, it is important to isolate each
individual mechanism while enabling the recovery of previous models of
interest. To this end, we present the unified architecture in Fig. 1, which
introduces _information gating functions_ (IGFs) as a means of decoupling IA
from architecture. Each component has its own IGF, depicted with colored
rectangle. Every module is fed all environment information
$\mathbf{x}_{k}=(\mathbf{x},k)$ and distinctly chosen IGFs mask which part of
the input each network has access to, thereby influencing which skills they
learn. By presenting multiple priors, we enable a comparison with existing
literature [14, 19, 20, 21]. With the right masking, one can recover
previously investigated asymmetries [14, 19], explore additional ones, and
also express purely hierarchical [9] and KL-regularized equivalents [8].

While prior works focused on the role of IA on policy ($\pi$) regularization
for multi-task learning [8], we focus on its influence over the prior’s
($\pi_{0}$) ability to handle covariate shift across sequential tasks. In the
sequential setting, there exists abrupt non-stationarities for task and agent
trajectory distributions. As such, in this setting, it is particularly
important that priors handle the non-stationarity and associated covariate
shift. IA plays a crucial role here. Specifically, the choice of asymmetry
between policy and prior affects the level of covariate shift encountered by
the prior:

######  Theorem 1.

The more random variables a network depends on, the larger the covariate shift
(input distributional shift, here represented by KL-divergence) encountered
across sequential tasks. That is, for distributions $p$, $q$ and inputs
$\mathbf{b}$, $\mathbf{c}$ such that
$\mathbf{b}=(b_{0},b_{1},...,b_{n})\textit{ and }\mathbf{c}\subset\mathbf{b}$:

$\displaystyle\mathrm{D_{KL}}\left(p(\mathbf{b})\;\middle\|\;q(\mathbf{b})\right)\geq\mathrm{D_{KL}}\left(p(\mathbf{c})\;\middle\|\;q(\mathbf{c})\right).$

###### Proof.

See Section B.1. ∎

In our case, $p$ and $q$ can be interpreted as training and transfer
distributions over network inputs (such as $\pi_{0}^{H}$). Intuitively,
Theorem 1 states that the more variables you condition your network on, the
less likely it will transfer due to increased covariate shift encountered
between training and transfer domains, thus promoting minimal information
conditioning. However, the less information a prior is conditioned on, the
less knowledge can be distilled and transferred. Therefore, IA leads to an
impactful trade-off between the transferability and expressivity of skills.
Interestingly, covariate shift is upper-bounded by policy shift:
$\mathrm{D_{KL}}\left(p_{\pi_{I}}(\mathbf{\tau})\;\middle\|\;p_{\pi_{T}}(\mathbf{\tau})\right)\geq\mathrm{D_{KL}}\left(p_{\pi_{I}}(\tau_{f})\;\middle\|\;p_{\pi_{T}}(\tau_{f})\right)$
(using Theorem 1), with $\tau_{f}=IGF(\tau)$ denoting IGF-filtered
trajectories (i.e. network inputs), and $\pi_{I}$ and $\pi_{T}$ the training
and transfer policies, respectively. It is therefore crucial, if possible, to
minimise _both_ behavioural and covariate shift across domains, if we wish to
benefit from previously attained skills.

While IAs influence what skills are discovered by hierarchy and priors, an
important distinction must be made between how each approach _transfers_
skills. Hierarchy transfers skills as _hard constraints_ on transfer
behaviour, by enforcing the reuse of low-level skills ($\pi^{L}$). In
contrast, priors transfer skills through _soft constraints_ (via KL-
regularization), allowing the policy to deviate from these behaviours when
necessary. As such, it is more crucial for IA between hierarchical levels to
handle covariate shift (and hence generalise), than between policy and prior.
This promotes reduced information conditioning for $\pi^{L}$ than $\pi_{0}$.
Thus, in-line with previous works [14, 9], we share the low-level policy
$\pi^{L}$ between policy and prior, and condition only on state
$\mathbf{s}_{t}$, ensuring minimal covariate shift and the discovery of
instantaneous behaviours that generalise favourably. Unlike prior works, we
consider conditioning the high-level prior, $\pi_{0}^{H}$, on additional
information enabling richer skill transfer. Specifically, we explore temporal
conditioning on varying levels of histories $\mathbf{x}_{t-i:t}$, where $i$
denoting depth, thus enabling priors to capture reusable, sequential, high-
level behaviours across tasks.

We additionally explore the importance of hierarchy for increased prior
expressivity. Here, hierarchy enables a richer prior distribution, capturing
multi-modal behaviours arguably present in many real-world examples.
Importantly, while hierarchy increases expressivity, it does not influence
covariate shift and is thus not expected to harm transferability. See Appendix
D for further architectural details.

##  4 Method

As described in Section 3, sequential learning poses distinct challenges:
primarily abrupt covariate shifts. As such, we designed experiments to isolate
each component’s importance under this setting.

###  4.1 Training Regime

In Tirumala et al. [14, 22], for each choice of IA, they first jointly train
separate hierarchical policies and priors in a multi-task setting, and then
freeze the shared low-level policy and high-level prior when training on the
sequential task. Accordingly, they cannot isolate the effects of distinct IAs
on sequential transfer, as distinct IAs lead to distinct low-level policies
during multi-task learning. To decouple the effects of IA and hierarchy for
sequential transfer, we propose the following regime. Stage 1 trains a single
hierarchical policy $\pi$ in the multi-task setup whilst regularizing against
_multiple_ high-level priors with distinct IGFs. Importantly, unlike Tirumala
et al. [14], we stop gradient flow to policy $\pi$ from the KL terms with
learnt priors, enabling the multiple learnt priors to imitate the policy
whilst not influencing it. In stage 2, we freeze the shared low-level policy
and high-level prior and train on the sequential task. See Algorithm 1. Unlike
Tirumala et al. [14, 22], during stage 1 we train using variational behavioral
cloning (BC) to bypass initial exploration issues. Given we focus on
sequential transfer, this choice is unimportant. Nevertheless, our skill
learning method can be considered of interest to the sequential learning
community. During BC, we apply DAGGER [23], as per Algorithm 2, improving
learning rates [24]. For further details refer to Appendix D.

###  4.2 Variational Behavioral Cloning as KL-Regularized Reinforcement
Learning

Variational BC can be considered as KL-regularized RL in the absence of
rewards, with the expert playing the role of the prior:

$\displaystyle\mathcal{O}_{bc}(\pi,\pi_{e})=-\operatorname{\mathbb{E}}_{\begin{subarray}{c}\tau\sim\rho_{\pi},\\\
k\sim
p(\mathcal{K})\end{subarray}}\left[\sum_{t=0}^{\infty}\gamma^{t}\alpha_{e}\mathrm{D_{KL}}\left(\pi(\mathbf{a}|{\mathbf{x}_{t}},k)\;\middle\|\;\pi_{e}(\mathbf{a}|{\mathbf{x}_{t}})\right)\right]$
(3)

More generally, Eqs. 1 and 3 can be extended to the setting of multiple
priors:

$\mathcal{O}_{bc}(\pi,\\{\pi_{i}\\}^{i\in I})=\sum_{i\in
I}{\mathcal{O}_{bc}(\pi,\pi_{i})}$ (4)

$\vspace{1mm}\mathcal{O}_{rl}(\pi,\\{\pi_{i}\\}^{i\subset
I})=E_{\pi}[R(\tau)]+O_{bc}(\pi,\\{\pi_{i}\\}^{i\subset I})$ (5)

For BC, $i\in\\{0,1,...,N,u,e\\}$, such that
$\\{\pi_{i}\\}_{i\in\\{0,...,N\\}}$ denote the learnt priors, $\pi_{u}$ the
uniform prior, and $\pi_{e}$ the expert prior. For RL, $i\in\\{0,u\\}$, as on
the transfer tasks we do not have access to the expert policy.
$E_{\pi}[R(\tau)]$ corresponds to the expected discounted return for policy
$\pi$. Combining these objectives with Eq. 2, we enable stage 1 and 2 training
and transferring of hierarchical policies and priors. During transfer, task-
conditioned policies, $\pi^{H}$, are not shared as they are task-specific.
Uniform priors enable highly-entropic policies, encouraging exploration and
stabilizing hierarchical learning [20]. We use Retrace [25] and double
Q-learning [26] to train our critic. Refer to Appendices A and D for full
training details and a deeper discussion on variational BC and RL.

##  5 Experiments

Our experiments are designed to answer the following questions related to
sequential task learning: (1) Can we benefit from hierarchy and priors to
effectively transfer knowledge over multiple levels of action and temporal
abstractions? (2) How important is the choice of IA between policy and prior,
and does it lead to an expressivity-transferability trade-off of behaviours?
(3) How important is hierarchy in enabling an expressive framework able to
capture complex behaviours across tasks? (4) What are the relative
contributions of priors, hierarchy, and IA for effective knowledge transfer?

###  5.1 Environments

We explore these questions in two domains: a toy domain designed for
controlled investigation of core agent capabilities and another, more
challenging, robotics domain demonstrating the applicability of our approach
in a practical setting (see Fig. 2). Both of these domains exhibit modular
behaviours at multiple-levels of abstraction whose discovery could yield
transfer benefits for downstream tasks. Refer to Appendix C for full
environmental setup details.

  * •

CorridorMaze. The agent must traverse five corridors in a given ordering. We
collect $4k$ trajectories from a scripted policy traversing any random
ordering of two corridors. During transfer, an inter- or extrapolated ordering
must be traversed (number of sequential corridors $=\\{2,4\\}$) allowing us to
inspect the _generalisation ability_ of distinct priors and how well they
handle increasing levels of covariate shift. To see how _priors affect
exploration_ , we consider two sparsity levels: _semi-sparse_ , rewarding per
correct half-corridor traversal, and _sparse_ , rewarding on task completion.
Our two transfer tasks are _sparse 2 corridor_ and _semi-sparse 4 corridor_.

  * •

Stack. The agent must stack a subset of four blocks over a target pad in a
given ordering. The blocks have distinct masses and only lighter blocks should
be placed on heavier ones. Therefore, for this domain, the ability to discover
long-term, sequential behaviours, namely skill priors corresponding to
sequentially stacking block with respect to their masses, is beneficial. We
collect $17.5k$ trajectories from a scripted policy, stacking any two blocks
given this requirement. To demonstrate the ability to achieve _generalisable
knowledge transfer_ and discover sequential coordination behavioural priors,
the transfer task requires all blocks be stacked according to mass. Rewards
are given per correct individual block stacked.

Figure 2: _Environments_. _a) CorridorMaze:_ A hard-exploration toy domain.
The agent starts in the hallway and must traverse a given number of corridors
in a given sequence. The agent completes a given corridor by traversing to its
depth and back. _b) Stack:_ A hard exploration, robotic, domain. The agent
must stack the cubes in a given ordering over the light blue target pad.

###  5.2 Hierarchy and Priors For Knowledge Transfer

We begin by exploring the relative importance of priors and hierarchy for
skill transfer. The full setup, which leverages learnt high-level priors and
hierarchy, is called APES. APES-no_prior represents a hierarchical model
without a learnt prior, as in Tirumala et al. [14]. RecSAC represents a
history-dependent SAC [27], trained directly on the transfer task and without
prior experience. Hier-RecSAC, represents a hierarchical-equivalent of RecSAC
using same hierarchical decomposition as APES, and is also trained directly on
the task at hand, and is akin to the method proposed in Wulfmeier et al. [9].
Table 3(a) compares performance of all approaches on the transfer tasks. As
expected, methods that leverage prior experience (APES, APES-no_prior)
outperform those that do not (Hier-RecSAC, RecSAC), highlighting the
importance of skill transfer. APES-no_prior, which only leverages hierarchy
for knowledge transfer yields marginal benefits on these domains. APES
strongly outperforms the rest of the methods, suggesting that the combination
of hierarchy and priors is important for learning. Unsurprisingly, for semi-
sparse domain, the smaller the observed benefits, given that rewards are
enough to guide the learning. We provide further qualitative analysis of the
performance of these methods in Section 5.6.

###  5.3 Information Asymmetry For Knowledge Transfer

To investigate the importance of IA for transfer, we compare high-level priors
with access to increasing levels of asymmetry: APES-{H20, H10, H1, S} where S
denotes a state-dependent prior without access to historical content and Hi
denotes a history-dependent prior with history dependency
$\mathbf{x}_{t-i:t}$. None of the priors are given access to task-dependent
information, namely task id or exteroceptive information (non-egocentric data,
e.g. cube position), ensuring skills transfer across these instances. The
results in Table 3(a) demonstrate the role that IA has in achieving effective
skill transfer across sequential tasks, with distinct choices leading to
drastically different transfer performance. There exists a general trend where
conditioning on too little (APES-S) or too much (APES-H20) information limits
effective knowledge transfer. We also observe that the influence of IA is
dependent on the reward type: for domains with semi-sparse rewards, transfer
performance varies less as the rewards guide exploration to such an extent
that effective knowledge transfer is unnecessary. We note, however, that in
many practical tasks of interest, the rewards are sparse. Regardless of
whether the transfer domain is interpolated or extrapolated , IA plays a
important role in effective transfer, suggesting that _IA is important over a
wide array of transfer tasks_.

###  5.4 Transferability-Expressivity Trade-Off

Figure 3: _CorridorMaze 2 corridor_. 4 rollouts shown for each method, with each episode rolled out horizontally. Corridors are colour coded and depth within them is denoted by shade; the darker the deeper. Hallway is white. See text for detailed analysis.  Table 1: Distillation Losses Environment | CorridorMaze | Stack
---|---|---
APES-H1 | $0.22\pm 0.00$ | $0.65\pm 0.00$
APES-H10 | $0.12\pm 0.00$ | $0.49\pm 0.00$
APES-H20 | $\mathbf{0.11\pm 0.00}$ | $\mathbf{0.47\pm 0.00}$
APES-S | $0.81\pm 0.00$ | $0.75\pm 0.00$
Max (min) |  $0.84$ $(0.00)$ |  $1.71$ $(0.00)$

To understand the effects of IA on transfer, we refer back to Section 3,
stating that reduced IA increases covariate shift and is upper-bounded by
_behavioural shift_ (shift in agent trajectories across tasks). Therefore, the
smaller the behavioural shift, the smaller the covariate shift. In sequential
learning, shift can occur due to three primary reasons: 1) the solution to the
task necessitates a shift, 2) any network components (e.g. the task-dependent
high-level policy, $\pi^{H}$) are reinitialised, 3) during online training, an
approximate critic incorrectly encourages sub-optimal, out-of-distribution,
behaviour. Our setting, like many, is influenced by a combination of these.

We investigate whether the increased covariate shift negatively impacts
transfer in practice with additional experiments focused on reducing
behavioural shift, and hence the covariate shift, across sequential tasks.
Table 3(b) presents the improvement in transfer performance attained for
experiments with an additionally pre-trained task-agnostic high-level policy
$\pi^{H}(\mathbf{x})$ shared across the tasks (or flat policy for non-
hierarchical equivalent). As such, no networks are reinitialised and initial
behavioural shift across domains is minimised. These new experiments will
reduce the covariate shift more for priors with less IA, as they inherently
experience more shift (Theorem 1). We thus expect larger improvement in
transfer performance for these priors. Table 3(b) confirms this trend
demonstrating the _importance of prior covariate shift in transferability of
behaviours_. Again, the trend is less apparent for the semi-sparse domain.
Additionally, for the interpolated transfer task (2 corridor), the solution is
entirely in the support of the training set of tasks. Naïvely, one would
expect pre-training to fully recover lost performance and match the most
performant method. However, as aforementioned, this is not the case as the
critic, trained solely on the transfer task, quickly encourages sub-optimal
out-of-distribution behaviours.

The previous experiments analysed the benefits that reduced IA has on positive
transfer. However, as discussed in Section 3, conditioning on too little
information limits the expressivity of the skills captured. To observe whether
this is the case, we compare distillation losses between prior and policy for
various IAs. The results are shown in Table 1 (showing mean/standard deviation
for $4$ seeds). As expected, _reduced IA reduces distillation losses_ showing
improved ability to distil behaviours present during training. Tables 3(a),
3(b) and 1 demonstrate the _transferability-expressivity trade-off_ , which is
independent of transfer task and more important for sparse domains, present
for distinct IAs.

###  5.5 Hierarchy for Expressivity

Table 2: Hierarchy Ablation Environment | CorridorMaze
---|---
Transfer task | 2 corridor | 4 corridor
APES-H1 | $0.80\pm 0.03$ | $6.37\pm 0.17$
APES-H1-KL-a | $0.76\pm 0.09$ | $5.59\pm 0.17$
APES-H1-flat | $0.05\pm 0.035$ | $4.52\pm 0.53$
Expert | $1$ | $8$

To investigate the role of hierarchy for effective task transfer, we compare
the performance of the most performant hierarchical method, APES-H1, with its
non-hierarchical equivalent, APES-H1-flat, which is otherwise equivalent to
the full APES setup except with a flat rather than hierarchical prior. As
discussed in Section 3, hierarchy increases the prior distribution
expressivity, enabling the representation of multi-modal behaviours, whilst
not increasing prior covariate shift and suffering from the associated
transfer problems. When regularizing against a flat prior, KL-regularization
must occur over the raw rather than latent action space. Therefore, to
adequately compare hierarchical and non-hierarchical equivalents, we
additionally compare against a hierarchical setup where regularization occurs
only over the action-space, APES-H1-KL-a.

Figure 4: _Stack 4 blocks_. Rollouts with end-effector and cube movement
depicted by dotted and dashed lines, colour-coded per rollout. Each rollout’s
end position represented by cross or cube respectively. Cube numbers
correspond to their inherent ordering. See text for detailed analysis.

Comparing transfer results for ablations APES-H1-KL-a and APES-H1-KL-flat, in
Table 2, we see the _benefits that a hierarchical prior brings_ to
transferability in CorridorMaze domain. The flat prior is unable to solve the
task, which on further inspection (see Section 5.6) can be seen to be caused
by its inability to capture multi-modal behaviours present in the training
tasks. For bottleneck states [28], where multi-modality is needed the most,
this leads to regularization against a highly suboptimal unimodal action
distribution. Interestingly, by comparing APES-H1 and APES-H1-KL-a, we observe
minimal benefits regularizing against latent rather than raw actions,
suggesting that with alternate methods for achieving multi-modal priors,
hierarchy may not be necessary. Finally, comparing APES-no_prior with RecSAC ,
we observe that hierarchy alone yields significantly less transfer benefits
for sparse-domain tasks than priors.

Table 3: Comparing average return (3(a)) and additional return (3(b)), across
100 episodes. Reporting mean and standard deviation over 4 random seeds.
Experiments that leverage prior experience were ran for $150k$ environment
steps. The remainder, (Hier-)RecSAC, were ran for $1M$ steps.

(a) _Transfer Results_. There exists a transferability-expressivity trade-off for chosen IAs, with APES-H1 performing the best. Sparse reward domains are more influenced by IA.  Environment | CorridorMaze | Stack
---|---|---
Transfer type | interpolate | extrapolate | extrapolate
Reward type | sparse | semi-sparse | sparse
Transfer task | 2 corridor | 4 corridor | 4 blocks
APES-H20 | $0.16\pm 0.09$ | $3.84\pm 0.19$ | $1.03\pm 0.45$
APES-H10 | $0.22\pm 0.09$ | $4.03\pm 0.55$ | $1.22\pm 0.26$
APES-H1 | $\mathbf{0.80\pm 0.03}$ | $\mathbf{6.37\pm 0.17}$ | $\mathbf{3.11\pm 0.12}$
APES-S | $0.00\pm 0.00$ | $3.03\pm 0.38$ | $1.98\pm 0.16$
APES-no_prior | $0.00\pm 0.00$ | $2.34\pm 0.19$ | $0.00\pm 0.00$
Hier-RecSAC | $0.00\pm 0.00$ | $0.07\pm 0.03$ | $0.01\pm 0.01$
RecSAC | $0.00\pm 0.00$ | $0.08\pm 0.02$ | $0.01\pm 0.01$
Expert | $1$ | $8$ | $4$

(b) _Covariate Shift Analysis_. In general, reduced IA benefits more from reduced covariate shift. Sparse domains suffer more from shift, seen by clearer IA covariate trends.  CorridorMaze | Stack
---|---
interpolate | extrapolate | extrapolate
sparse | semi-sparse | sparse
2 corridor | 4 corridor | 4 blocks
$\mathbf{0.28\pm 0.03}$ | $0.12\pm 0.22$ | $0.84\pm 0.06$
$0.24\pm 0.07$ | $\mathbf{0.62\pm 0.30}$ | $\mathbf{1.20\pm 0.31}$
$0.13\pm 0.02$ | $0.34\pm 0.32$ | $0.22\pm 0.24$
$0.00\pm 0.00$ | $0.48\pm 0.21$ | $0.00\pm 0.17$
$0.00\pm 0.00$ | $0.09\pm 0.08$ | $0.49\pm 0.26$
$0.00\pm 0.00$ | $0.02\pm 0.03$ | $0.00\pm 0.01$
$0.00\pm 0.00$ | $0.27\pm 0.13$ | $0.19\pm 0.05$
$0$ | $0$ | $0$

###  5.6 Exploration Analysis

To gain further understanding on the effects of hierarchy and priors, we
visualise the policy rollouts early on ($5k$ steps) in the transfer for APES,
its ablations, and baselines. For _CorridorMaze_ (Fig. 3), the full setup
using hierarchy and priors explores at the corridor level, traversing
corridors in a random order across episodes. Hierarchy alone, unable to
express preference over high-level skills, leads to temporally uncorrelated
behaviours, and thus unable to explore at the corridor level. The flat prior,
unable to represent multi-modal behaviours, leads to suboptimal exploration at
the bottleneck state, the intersection of corridors, often leading the agent’s
position to remain static. Without priors nor hierarchy, exploration is
further hindered, rarely traversing corridor depths. For _Stack_ task (Fig.
4), the full setup again explores at the individual block stacking level,
alternating the ordering of blocks between episodes, but the prior leads to
primarily stacking lighter upon heavier blocks. Hierarchy alone, explores
undirectedly without the knowledge of what blocks to operate on, leading to
temporally uncorrelated, inconsistent, high-frequency switching of low-level
manipulation skills.

##  6 Related Work

Hierarchical frameworks have a long history [28]. The option, conditioned, RL
literature tackle the semi-MDP setup and explore the benefits that hierarchy
and temporal abstraction bring [29, 13, 20]. Wulfmeier et al. [9, 13] use
hierarchy to enforce knowledge transfer through shared hierarchical modules.
However, for lifelong learning, where number of skills increase over time, it
is unclear how well these approaches will fare, without priors to narrow skill
exploration.

Priors have been used in various fields. In the context of offline-RL, Siegel
et al. [30], Wu et al. [31] primarily use priors to tackle value
overestimation [32]. In the variational literature, priors have been used to
guide latent-space learning [11, 20, 19, 33]. Hausman et al. [11] learn
episodic skills, limiting their ability to transfer. Igl et al. [20] learn
options and priors, but is on-policy and therefore suffers from sample
inefficiency, making the application to robotic domains challenging. In the
multi-task literature, priors have been used to guide exploration [19, 8, 30,
34, 7], yet without hierarchy expressivity in learnt behaviours is limited.
Recently, concurrent with our research, Tirumala et al. [22] were able to
exploit hierarchical priors to transfer entire history-dependent high-level
skills. However, their setup was in a high-data regime ($1e9$ data samples for
pre-training skills), where covariate shift is less predominant. Unlike the
aforementioned works, we consider the POMDP setting, arguably more suited for
real-world robotics.

Whilst most previous works rely on information asymmetry, choice is often
motivated by intuition. For example, Igl et al. [20], Wulfmeier et al. [9, 13]
only employ task or goal asymmetry and Tirumala et al. [14], Merel et al.
[10], Galashov et al. [8] use exteroceptive asymmetry. Salter et al. [35],
Galashov et al. [8] use asymmetry as a form of achieving robust and
generalizable behaviours. Salter et al. [35] also investigate a way of
learning asymmetry. We provide principled investigation on the role of
asymmetry and point out its crucial role in sequential transfer setting.

##  7 Conclusion

In this paper, we employ hierarchical KL-regularized RL to efficiently
transfer skills across multiple abstraction levels, showing the effectiveness
of combining hierarchy and priors. We propose temporal information asymmetry,
as an effective method for sequential skill discovery and transfer, and
leverage variational BC as a sample efficient method for discovering these. We
theoretically and empirically show the crucial trade-off, controlled by choice
of IA, between the _expressivity_ and _transferability_ of skills across
sequential tasks. Our experiments validate the importance of this trade-off
for not just extrapolated domains, but also for interpolated ones. We
demonstrate that, while priors are significantly more effective than
hierarchy, hierarchy is still important for enabling expressive priors.
Finally, we apply our approach to a complex, sparse-reward robot block-
stacking domain, unsolvable by the state-of-the-art baselines, showing that
with the right choice of asymmetries, sample-efficient transfer can be
achieved. Future work will include how to learn optimal IAs, given weak
assumptions about the distribution of transfer tasks.

## References

  * Silver et al. [2017] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al.  Mastering the game of go without human knowledge.  _nature_ , 550(7676):354–359, 2017.
  * Mnih et al. [2015] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al.  Human-level control through deep reinforcement learning.  _Nature_ , 518(7540):529–533, 2015.
  * Lillicrap et al. [2015] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra.  Continuous control with deep reinforcement learning.  _arXiv preprint arXiv:1509.02971_ , 2015.
  * Abdolmaleki et al. [2018] A. Abdolmaleki, J. T. Springenberg, Y. Tassa, R. Munos, N. Heess, and M. Riedmiller.  Maximum a posteriori policy optimisation.  _arXiv preprint arXiv:1806.06920_ , 2018.
  * Haarnoja et al. [2018] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _International Conference on Machine Learning (ICML)_ , 2018.
  * Parisi et al. [2019] G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter.  Continual lifelong learning with neural networks: A review.  _Neural Networks_ , 113:54–71, 2019.
  * Teh et al. [2017] Y. W. Teh, V. Bapst, W. M. Czarnecki, J. Quan, J. Kirkpatrick, R. Hadsell, N. Heess, and R. Pascanu.  Distral: Robust multitask reinforcement learning.  _arXiv preprint arXiv:1707.04175_ , 2017.
  * Galashov et al. [2019] A. Galashov, S. M. Jayakumar, L. Hasenclever, D. Tirumala, J. Schwarz, G. Desjardins, W. M. Czarnecki, Y. W. Teh, R. Pascanu, and N. Heess.  Information asymmetry in kl-regularized rl.  _arXiv preprint arXiv:1905.01240_ , 2019.
  * Wulfmeier et al. [2019] M. Wulfmeier, A. Abdolmaleki, R. Hafner, J. T. Springenberg, M. Neunert, T. Hertweck, T. Lampe, N. Siegel, N. Heess, and M. Riedmiller.  Compositional Transfer in Hierarchical Reinforcement Learning.  (1), 2019.  URL http://arxiv.org/abs/1906.11228.
  * Merel et al. [2020] J. Merel, S. Tunyasuvunakool, A. Ahuja, Y. Tassa, L. Hasenclever, V. Pham, T. Erez, G. Wayne, and N. Heess.  Catch & carry: reusable neural controllers for vision-guided whole-body tasks.  _ACM Transactions on Graphics (TOG)_ , 39(4):39–1, 2020.
  * Hausman et al. [2018] K. Hausman, J. T. Springenberg, Z. Wang, N. Heess, and M. Riedmiller.  Learning an embedding space for transferable robot skills.  In _International Conference on Learning Representations_ , 2018.
  * Haarnoja et al. [2018] T. Haarnoja, K. Hartikainen, P. Abbeel, and S. Levine.  Latent space policies for hierarchical reinforcement learning.  In _International Conference on Machine Learning (ICML)_ , 2018.
  * Wulfmeier et al. [2020] M. Wulfmeier, D. Rao, R. Hafner, T. Lampe, A. Abdolmaleki, T. Hertweck, M. Neunert, D. Tirumala, N. Siegel, N. Heess, et al.  Data-efficient hindsight off-policy option learning.  _arXiv preprint arXiv:2007.15588_ , 2020.
  * Tirumala et al. [2019] D. Tirumala, H. Noh, A. Galashov, L. Hasenclever, A. Ahuja, G. Wayne, R. Pascanu, Y. W. Teh, and N. Heess.  Exploiting hierarchy for learning and transfer in kl-regularized rl.  _arXiv preprint arXiv:1903.07438_ , 2019.
  * Todorov [2007] E. Todorov.  Linearly-solvable Markov decision problems.  In _Advances in Neural Information Processing Systems_ , pages 1369–1376. MIT Press, 2007.
  * Kappen et al. [2012] H. J. Kappen, V. Gómez, and M. Opper.  Optimal control as a graphical model inference problem.  _Machine learning_ , 87(2):159–182, 2012.
  * Rawlik et al. [2012] K. Rawlik, M. Toussaint, and S. Vijayakumar.  On stochastic optimal control and reinforcement learning by approximate inference.  _Robotics: Science and Systems (RSS)_ , 2012.
  * Schulman et al. [2017] J. Schulman, P. Abbeel, and X. Chen.  Equivalence between policy gradients and soft Q-learning.  _arXiv preprint arXiv:1704.06440_ , 2017.
  * Pertsch et al. [2020] K. Pertsch, Y. Lee, and J. J. Lim.  Accelerating reinforcement learning with learned skill priors.  _arXiv preprint arXiv:2010.11944_ , 2020.
  * Igl et al. [2019] M. Igl, A. Gambardella, J. He, N. Nardelli, N. Siddharth, W. Böhmer, and S. Whiteson.  Multitask soft option learning.  _arXiv preprint arXiv:1904.01033_ , 2019.
  * Haarnoja et al. [2018] T. Haarnoja, A. Zhou, S. Ha, J. Tan, G. Tucker, and S. Levine.  Learning to walk via deep reinforcement learning.  _arXiv preprint arXiv:1812.11103_ , 2018.
  * Tirumala et al. [2020] D. Tirumala, A. Galashov, H. Noh, L. Hasenclever, R. Pascanu, J. Schwarz, G. Desjardins, W. M. Czarnecki, A. Ahuja, Y. W. Teh, et al.  Behavior priors for efficient reinforcement learning.  _arXiv preprint arXiv:2010.14274_ , 2020.
  * Ross et al. [2011] S. Ross, G. Gordon, and D. Bagnell.  A reduction of imitation learning and structured prediction to no-regret online learning.  In _Proceedings of the fourteenth international conference on artificial intelligence and statistics_ , pages 627–635. JMLR Workshop and Conference Proceedings, 2011.
  * Le et al. [2018] H. Le, N. Jiang, A. Agarwal, M. Dudík, Y. Yue, and H. Daumé.  Hierarchical imitation and reinforcement learning.  In _International Conference on Machine Learning_ , pages 2917–2926. PMLR, 2018.
  * Munos et al. [2016] R. Munos, T. Stepleton, A. Harutyunyan, and M. G. Bellemare.  Safe and efficient off-policy reinforcement learning.  _arXiv preprint arXiv:1606.02647_ , 2016.
  * Hasselt [2010] H. V. Hasselt.  Double Q-learning.  In _Advances in Neural Information Processing Systems (NIPS)_ , pages 2613–2621, 2010.
  * Haarnoja et al. [2018] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, et al.  Soft actor-critic algorithms and applications.  _arXiv preprint arXiv:1812.05905_ , 2018.
  * Sutton et al. [1999] R. S. Sutton, D. Precup, and S. Singh.  Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning.  _Artificial intelligence_ , 112(1-2):181–211, 1999.
  * Nachum et al. [2018] O. Nachum, H. Lee, S. Gu, and S. Levine.  Data-efficient hierarchical reinforcement learning.  In _Advances in Neural Information Processing Systems_ , volume 2018-Decem, pages 3303–3313, 2018.  URL https://sites.google.com/view/efficient-hrl.
  * Siegel et al. [2020] N. Y. Siegel, J. T. Springenberg, F. Berkenkamp, A. Abdolmaleki, M. Neunert, T. Lampe, R. Hafner, N. Heess, and M. Riedmiller.  Keep doing what worked: Behavioral modelling priors for offline reinforcement learning.  _arXiv preprint arXiv:2002.08396_ , 2020.
  * Wu et al. [2019] Y. Wu, G. Tucker, and O. Nachum.  Behavior regularized offline reinforcement learning.  _arXiv preprint arXiv:1911.11361_ , 2019.
  * Levine et al. [2020] S. Levine, A. Kumar, G. Tucker, and J. Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  _arXiv preprint arXiv:2005.01643_ , 2020.
  * Merel et al. [2018] J. Merel, L. Hasenclever, A. Galashov, A. Ahuja, V. Pham, G. Wayne, Y. W. Teh, and N. Heess.  Neural probabilistic motor primitives for humanoid control.  _arXiv preprint arXiv:1811.11711_ , 2018.
  * [34] K. Pertsch, Y. Lee, Y. Wu, and J. Lim.  Guided reinforcement learning with learned skills.
  * Salter et al. [2020] S. Salter, D. Rao, M. Wulfmeier, R. Hadsell, and I. Posner.  Attention-privileged reinforcement learning.  In _Conference on Robot Learning_ , 2020.
  * Jang et al. [2016] E. Jang, S. Gu, and B. Poole.  Categorical reparameterization with gumbel-softmax.  _arXiv preprint arXiv:1611.01144_ , 2016.
  * Sutton [1988] R. S. Sutton.  Learning to predict by the methods of temporal differences.  _Machine learning_ , 3(1):9–44, 1988.
  * Plappert et al. [2018] M. Plappert, M. Andrychowicz, A. Ray, B. McGrew, B. Baker, G. Powell, J. Schneider, J. Tobin, M. Chociej, P. Welinder, et al.  Multi-goal reinforcement learning: Challenging robotics environments and request for research.  _arXiv preprint arXiv:1802.09464_ , 2018.

##  Appendix A Method

###  A.1 Training Regime

In this section, we algorithmically describe our training setup. We relate
each training phase to the principle equations in the main paper, but note
that Sections A.2 and A.3 outline a more detailed version of these equations
that were actually used.

Algorithm 1 APES training regime

1:# Full training and transfer regime. For BC, gradients are prevented from
flowing from $\pi_{0}$ to $\pi$. In practice
$\pi_{0}=\\{\pi_{i}\\}_{i\in\\{0,...,N\\}}$, multiple trained priors. During
transfer, $\pi_{H}$, is reinitialized.

2:

3:# Behavioral Cloning

4:Initialize: policy $\pi$, prior $\pi_{0}$, replay $R_{bc}$, DAGGER rate $r$,
environment $env$

5:for Number of BC training steps do

6: $R_{bc}$, $env$ $\leftarrow$ collect($\pi$, $R_{bc}$, $env$, True, $r$)

7: $\pi$, $\pi_{0}$ $\leftarrow$ BC_update($\pi$, $\pi_{0}$, $R_{bc}$) # Eq. 4

8:end for

9:# Reinforcement Learning

10:Initialize: high level policy $\pi^{H}$, critics $Q_{k\in\\{1,2\\}}$,
replay $R_{rl}$, transfer environment $env_{t}$

11:for Number of RL training steps do

12: $R_{rl}$, $env_{t}$ $\leftarrow$ collect($\pi$, $R_{rl}$, $env_{t}$)

13: $\pi^{H}$ $\leftarrow$ RL_policy_update($\pi$, $\pi_{0}$, $R_{rl}$) # Eq.
5

14: $Q_{k}$ $\leftarrow$ RL_critic_update($Q_{k}$, $\pi$, $R_{rl}$) # Eq. 10

15:end for

Algorithm 2 collect

1:# Collects experience from either $\pi_{i}$ or $\pi_{e}$, applying DAGGER at
a given rate if instructed, and updates $R_{j}$, $env$ accordingly.

2:function collect($\pi_{i}$, $R_{j}$, $env$, $dag$=False, $r$=$1$)

3: $\mathbf{x}$ $\leftarrow$ $env.\text{observation}()$

4: $\pi_{e}$ $\leftarrow$ $env.\text{expert}()$

5: $\mathbf{a}_{i}$ $\leftarrow$ $\pi_{i}(\mathbf{x})$

6: $\mathbf{a}_{e}$ $\leftarrow$ $\pi_{e}(\mathbf{x})$

7: $\mathbf{a}$ $\leftarrow$ Bernoulli([$\mathbf{a}_{i}$, $\mathbf{a}_{e}$],
[$r$, $1$ \- $r$])

8: $\mathbf{x}^{\prime},r_{k},env\leftarrow env.\text{step}(\mathbf{a})$

9: if $dag$ then

10: $\mathbf{a}_{f}$ $\leftarrow$ $\mathbf{a}_{e}$

11: else

12: $\mathbf{a}_{f}$ $\leftarrow$ $\mathbf{a}_{i}$

13: end if

14: $R_{j}$ $\leftarrow$
$R_{j}.\text{update}(\mathbf{x},\mathbf{a}_{f},$r${}_{k},\mathbf{x}^{\prime})$

15: return $R_{j}$, $env$

16:end function

###  A.2 Variational Behavioral Cloning and Reinforcement Learning

Behavioral Cloning (BC) and KL-Regularized RL, when considered from the
variational-inference perspective, share many similarities. These similarities
become even more apparent when dealing with hierarchical models. A
particularly unifying choice of objective functions for BC and RL that fit
with off-policy, generative, hierarchical RL: desirable for sample efficiency,
are:

$O_{bc}(\pi,\\{\pi_{i}\\}^{i\in I})=-\sum_{i\in
I}{\mathrm{D_{KL}}\left(\pi(\tau)\;\middle\|\;\pi_{i}(\tau)\right)},\text{
}\text{ }O_{rl}(\pi,\\{\pi_{i}\\}^{i\subset
I})=E_{\pi}(\tau)[R(\tau)]+O_{bc}(\pi,\\{\pi_{i}\\}^{i\subset I})$ (6)

$O_{bc}$, corresponds to the KL-divergence between trajectories from the
policy, $\pi$, and various priors, $\pi_{i}$. For BC, $i\in\\{0,u,e\\}$,
denote the learnt, uniform, and expert priors. For BC, in practice, we train
multiple priors in parallel: $\pi_{0}=\\{\pi_{i}\\}_{i\in\\{0,...,N\\}}$. We
leave this notation out for the remainder of this section for simplicity. When
considering only the expert prior, this is the reverse KL-divergence, opposite
to what is usually evaluated in the literature, [19]. $O_{rl}$, refers to a
lower bound on the expected optimality of each prior $\log p_{\pi_{i}}(O=1)$;
$O$ denoting the event of achieving maximum return (return referred to as
$R(.)$); refer to [4], appendix B.3.3 for proof, further explanation, and
necessary conditions. During transfer using RL, we do not have access to the
expert or its demonstrations ($i\subset I:=i\in\\{0,u\\}$).

For hierarchical policies, the KL terms are not easily evaluable.
$\mathrm{D_{KL}}\left(\pi(\tau)\;\middle\|\;\pi_{i}(\tau)\right)\leq\sum_{t}\operatorname{\mathbb{E}}_{\pi(\tau)}\left[\mathrm{D_{KL}}\left(\pi^{H}(\mathbf{z}_{t}|\mathbf{x}_{k})\;\middle\|\;\pi^{H}_{i}(\mathbf{z}_{t}|\mathbf{x}_{k})\right)+\operatorname{\mathbb{E}}_{\pi^{H}(\mathbf{z}_{t}|\mathbf{x}_{k})}\left[\mathrm{D_{KL}}\left(\pi^{L}(\mathbf{a}_{t}|\mathbf{x}_{k},\mathbf{z}_{t})\;\middle\|\;\pi^{L}_{i}(\mathbf{a}_{t}|\mathbf{x}_{k},\mathbf{z}_{t})\right)\right]\right]$
111For proof refer to Appendix B.2 [14], is a commonly chosen upper bound. If
sharing modules, e.g. $\pi^{L}_{i}=\pi^{L}$, or using non-hierarchical
networks, this bound can be simplified (removing the second or first terms
respectively). To make both Eq. 6 amendable to off-policy training (experience
from $\\{\pi_{e},\pi_{b}\\}$, for BC/RL respectively; $\pi_{b}$ representing
behavioral policy), we introduce importance weighting (IW), removing off-
policy bias at the expense of higher variance. Combining all the above with
additional individual term weighting hyperparameters,
$\\{\beta^{z}_{i},\beta^{a}_{i}\\}$, we attain:

$\displaystyle\mathrm{\tilde{D}}^{q(\tau)}_{\mathrm{{KL}}}(\pi(\tau)||\pi_{i}(\tau)):=\operatorname{\mathbb{E}}_{q(\tau)}\left[\sum_{t}{\nu^{q}[t]\cdot\big{(}\beta_{i}^{z}\cdot
C_{i,h}(\mathbf{z}_{t}|\mathbf{x}_{k})+\beta_{i}^{a}\cdot\operatorname{\mathbb{E}}_{\pi^{H}(\mathbf{z}_{t}|\mathbf{x}_{k})}\left[C_{i,l}(\mathbf{a}_{t}|\mathbf{x}_{k},\mathbf{z}_{t})\right]\big{)}}\right]$
$\displaystyle\zeta_{i}^{n}=\frac{\operatorname{\mathbb{E}}_{\pi^{H}(z_{i}|\mathbf{x}_{i},k)}\left[\pi^{L}\left(\mathbf{a}_{i}|\mathbf{x}_{i},\mathbf{z}_{i},k\right)\right]}{n(\mathbf{a}_{i}|\mathbf{x}_{i},k)},\text{
}\nu^{n}=\left[\zeta_{1}^{n},\zeta_{1}^{n}\zeta_{2}^{n},\ldots,\prod_{i=1}^{\tau_{t}}\zeta_{i}^{n}\right],\text{
}C_{\mu,\epsilon}(y)=\log\bigg{(}\frac{\pi^{r}_{\epsilon}(y)}{\pi^{r}_{\mu,\epsilon}(y)}\bigg{)}$
$\displaystyle-\mathrm{D}_{\mathrm{{KL}}}(\pi(\tau)||\pi_{e}(\tau))\geq-\sum_{i\in\\{0,u,e\\}}\mathrm{\tilde{D}}^{\pi_{e}(\tau)}_{\mathrm{{KL}}}(\pi(\tau)||\pi_{i}(\tau))$
(7)
$\displaystyle\operatorname{\mathbb{E}}_{\begin{subarray}{c}p(\mathcal{K}),\\\
\pi_{0}(\tau)\end{subarray}}\left[\log(O=1|\tau,k)\right]\geq\operatorname{\mathbb{E}}_{\pi_{b}(\tau)}\left[\sum_{t}\nu^{\pi_{b}}[t]\cdot
r_{k}(\mathbf{x}_{t},\mathbf{a}_{t})\right]-\sum_{i\in\\{0,u\\}}\mathrm{\tilde{D}}^{\pi_{b}(\tau)}_{\mathrm{{KL}}}(\pi(\tau)||\pi_{i}(\tau))$
(8)

Where $\mathrm{\tilde{D}}^{q(\tau)}_{\mathrm{{KL}}}(\pi(\tau)||\pi_{i}(\tau))$
(for $\\{\beta^{z}_{i},\beta^{a}_{i}\\}=1$) is an unbiased estimate for the
aforementioned upper bound, using $q$’s experience. $\zeta^{n}_{i}$ is the IW
for timestep $i$, between $\pi$ and arbitrary policy $n$. $\nu^{n}[t]$ is the
$t^{th}$ element of $\nu^{n}$; the cumulative IW product at timestep $t$.
Equations 7, 8 are the BC/RL lower bounds used for policy gradients. See
Appendix B.3 for a derivation, and necessary conditions, of these bounds. For
BC, this bounds the KL-divergence between hierarchical and expert policies,
$\pi,\pi_{e}$. For RL, this bounds the expected optimality, for the learnt
prior policy, $\pi_{0}$. Intuitively, maximising this particular bound,
maximizes return for both policy and prior, whilst minimizing the disparity
between them. Regularising against an uninformative prior, $\pi_{u}$,
encourages highly-entropic policies, further aiding at exploration and
stabilising learning [20].

In RL, IWs are commonly ignored [3, 4, 5], thereby considering each sample
equally important. This is also convenient for BC, as IWs require the expert
probability distribution: not usually provided. We did not observe benefits of
using them and therefore ignore them too. We employ module sharing
($\pi^{L}_{i}=\pi^{L}$; unless stated otherwize), and freeze certain modules
during distinct phases, and thus never employ more than 2 hyperparameters,
$\beta$, at any given time, simplifying the hyperparameter optimisation. These
weights balance an exploration/exploitation trade-off. We use a categorical
latent space, explicitly marginalising over, rather than using sampling
approximations [36]. For BC, we train for 1 epoch (referring to training in
the expectation once over each sample in the replay buffer).

###  A.3 Critic Learning

The lower bound presented in Eq. 8 is non-differentiable due to rewards being
sampled from the environment. Therefore, as is common in the RL literature [2,
3], we approximate the return of policy $\pi$ with a critic, $Q$. To be sample
efficient, we train in an off-policy manner with TD-learning [37] using the
Retrace algorithm [25] to provide a low-variance, low-bias, policy evaluation
operator:

$\displaystyle
Q_{t}^{\text{ret}}:=Q^{{}^{\prime}}(\mathbf{x}_{t},\mathbf{a}_{t},k)+\sum_{j=t}^{\infty}{\epsilon_{j}^{t}\bigg{[}r_{k}(\mathbf{x}_{j},\mathbf{a}_{j})+\operatorname{\mathbb{E}}_{\begin{subarray}{c}\pi^{H}(\mathbf{z}|\mathbf{x}_{j+1},k),\\\
\pi^{L}(\mathbf{a}^{\prime}|\mathbf{x}_{j+1},\mathbf{z},k)\end{subarray}}\left[Q^{{}^{\prime}}(\mathbf{x}_{j+1},\mathbf{a}^{\prime},k)\right]-Q^{{}^{\prime}}(\mathbf{x}_{j},\mathbf{a}_{j},k)\bigg{]}}$
(9)

$\displaystyle
L(Q)=\operatorname{\mathbb{E}}_{\begin{subarray}{c}p(\mathcal{K}),\\\
\pi_{b}(\tau)\end{subarray}}\left[(Q(\mathbf{x}_{t},\mathbf{a}_{t},k)-\operatorname*{arg\,min}_{Q_{t}^{\text{ret}}}(Q_{t}^{\text{ret}}))^{2}\right]\qquad\epsilon_{j}^{t}=\gamma^{j-t}\prod_{i=t+1}^{j}\zeta^{b}_{i}$
(10)

Where $Q_{t}^{\text{ret}}$ represents the policy return evaluated via Retrace.
$Q^{{}^{\prime}}$ is the target Q-network, commonly used to stabilize critic
learning [2], and is updated periodically with the current Q values. IWs are
not ignored here, and are clipped between $[0,1]$ to prevent exploding
gradients, [25]. To further reduce bias and overestimates of our target,
$Q_{t}^{\text{ret}}$, we apply the double Q-learning trick, [26], and
concurrently learn two target Q-networks, $Q^{{}^{\prime}}$. Our critic is
trained to minimize the loss in Eq. 10, which regularizes the critic against
the minimum of the two targets produced by both target networks.

##  Appendix B Theory and Derivations

In this section we provide proofs for the theory introduced in the main paper
and in Appendix A.

###  B.1 Theorem 1.

##### Theorem 1.

The more random variables a network depends on, the larger the covariate shift
(input distributional shift, here represented by KL-divergence) encountered
across sequential tasks. That is, for distributions $p$, $q$ and inputs

$\displaystyle\mathrm{D_{KL}}\left(p(\mathbf{b})\;\middle\|\;q(\mathbf{b})\right)\geq\mathrm{D_{KL}}\left(p(\mathbf{c})\;\middle\|\;q(\mathbf{c})\right)$
(11) $\displaystyle\textit{with }\mathbf{b}=(b_{0},b_{1},...,b_{n})\textit{
and }\mathbf{c}\subset\mathbf{b}\textit{.}$

##### Proof

$\displaystyle\mathrm{D_{KL}}\left(p(\mathbf{b})\;\middle\|\;q(\mathbf{b})\right)$
$\displaystyle=\operatorname{\mathbb{E}}_{p(\textbf{b})}\left[\log\bigg{(}\frac{p(\mathbf{b})}{q(\mathbf{b})}\bigg{)}\right]$
(12) $\displaystyle=\operatorname{\mathbb{E}}_{p(\textbf{d}|\textbf{c})\cdot
p(\textbf{c})}\left[\log\bigg{(}\frac{p(\mathbf{d}|\mathbf{c})\cdot
p(\mathbf{c})}{q(\mathbf{d}|\mathbf{c})\cdot
q(\mathbf{c})}\bigg{)}\right]\quad\textit{with}\quad\textbf{d}\in\textbf{b}\oplus\textbf{c}$
$\displaystyle=\operatorname{\mathbb{E}}_{p(\textbf{c})}\left[\operatorname{\mathbb{E}}_{p(\textbf{d}|\textbf{c})}\left[1\right]\cdot\log\bigg{(}\frac{p(\mathbf{c})}{q(\mathbf{c})}\bigg{)}\right]+\operatorname{\mathbb{E}}_{p(\textbf{c})}\left[\operatorname{\mathbb{E}}_{p(\textbf{d}|\textbf{c})}\left[\log\bigg{(}\frac{p(\mathbf{d}|\mathbf{c})}{q(\mathbf{d}|\mathbf{c})}\bigg{)}\right]\right]$
$\displaystyle=\mathrm{D_{KL}}\left(p(\mathbf{c})\;\middle\|\;q(\mathbf{c})\right)+\operatorname{\mathbb{E}}_{p(\mathbf{c})}\left[\mathrm{D_{KL}}\left(p(\mathbf{d}|\mathbf{c})\;\middle\|\;q(\mathbf{d}|\mathbf{c})\right)\right]$
$\displaystyle\geq\mathrm{D_{KL}}\left(p(\mathbf{c})\;\middle\|\;q(\mathbf{c})\right)\quad\textit{given}\quad\operatorname{\mathbb{E}}_{p(\mathbf{c})}\left[\mathrm{D_{KL}}\left(p(\mathbf{d}|\mathbf{c})\;\middle\|\;q(\mathbf{d}|\mathbf{c})\right)\right]\geq
0$

###  B.2 Hierarchical KL-Divergence Upper Bound

All proofs in this section ignore multi-task setup for simplicity. Extending
to this scenario is trivial.

##### Upper Bound

$\displaystyle\mathrm{D_{KL}}\left(\pi(\tau)\;\middle\|\;\pi_{i}(\tau)\right)\leq\sum_{t}\mathbb{E}_{\pi(\tau)}[$
$\displaystyle\mathrm{D_{KL}}\left(\pi^{H}(\mathbf{z}_{t}|\mathbf{x}_{t})\;\middle\|\;\pi^{H}_{i}(\mathbf{z}_{t}|\mathbf{x}_{t})\right)$
(13)
$\displaystyle+\operatorname{\mathbb{E}}_{\pi^{H}(\mathbf{z}_{t}|\mathbf{x}_{t})}\left[\mathrm{D_{KL}}\left(\pi^{L}(\mathbf{a}_{t}|\mathbf{x}_{t},\mathbf{z}_{t})\;\middle\|\;\pi^{L}_{i}(\mathbf{a}_{t}|\mathbf{x}_{t},\mathbf{z}_{t})\right)\right]]$

##### Proof

$\displaystyle\mathrm{D_{KL}}\left(\pi(\tau)\;\middle\|\;\pi_{i}(\tau)\right)$
$\displaystyle=\operatorname{\mathbb{E}}_{\pi(\tau)}\left[\log\bigg{(}\frac{\pi(\tau)}{\pi_{i}(\tau))}\bigg{)}\right]$
(14)
$\displaystyle=\operatorname{\mathbb{E}}_{\pi(\tau)}\left[\log\bigg{(}\frac{p(\mathbf{s}_{0})\cdot\prod_{t}p(\mathbf{s}_{t+1}|\mathbf{x}_{t},\mathbf{a}_{t})\cdot\pi(\mathbf{a}_{t}|\mathbf{x}_{t})}{p(\mathbf{s}_{0})\cdot\prod_{t}p(\mathbf{s}_{t+1}|\mathbf{x}_{t},\mathbf{a}_{t})\cdot\pi_{i}(\mathbf{a}_{t}|\mathbf{x}_{t})}\bigg{)}\right]$
$\displaystyle=\operatorname{\mathbb{E}}_{\pi(\tau)}\left[\log\bigg{(}\prod_{t}\frac{\pi(\mathbf{a}_{t}|\mathbf{x}_{t})}{\pi_{i}(\mathbf{a}_{t}|\mathbf{x}_{t})}\bigg{)}\right]$
$\displaystyle=\sum_{t}\operatorname{\mathbb{E}}_{\pi(\tau)}\left[\mathrm{D_{KL}}\left(\pi(\mathbf{a}_{t}|\mathbf{x}_{t})\;\middle\|\;\pi_{i}(\mathbf{a}_{t}|\mathbf{x}_{t})\right)\right]$
$\displaystyle\leq\sum_{t}\mathbb{E}_{\pi(\tau)}[\mathrm{D_{KL}}\left(\pi(\mathbf{a}_{t}|\mathbf{x}_{t})\;\middle\|\;\pi_{i}(\mathbf{a}_{t}|\mathbf{x}_{t})\right)+$
$\displaystyle\operatorname{\mathbb{E}}_{\pi(\mathbf{a}_{t}|\mathbf{x}_{t})}\left[\mathrm{D_{KL}}\left(\pi(\mathbf{z}_{t}|\mathbf{x}_{t},\mathbf{a}_{t})\;\middle\|\;\pi(\mathbf{z}_{t}|\mathbf{x}_{t},\mathbf{a}_{t})\right)\right]]$
$\displaystyle=\sum_{t}\operatorname{\mathbb{E}}_{\pi(\tau)}\left[\operatorname{\mathbb{E}}_{\pi(\mathbf{a}_{t},\mathbf{z}_{t}|\mathbf{x}_{t})}\left[\log\bigg{(}\frac{\pi(\mathbf{a}_{t}|\mathbf{x}_{t})}{\pi_{i}(\mathbf{a}_{t}|\mathbf{x}_{t})}\bigg{)}+\log\bigg{(}\frac{\pi(\mathbf{z}_{t}|\mathbf{x}_{t},\mathbf{a}_{t})}{\pi_{i}(\mathbf{z}_{t}|\mathbf{x}_{t},\mathbf{a}_{t})}\bigg{)}\right]\right]$
$\displaystyle=\operatorname{\mathbb{E}}_{\pi(\tau)}\left[\mathrm{D_{KL}}\left(\pi(\mathbf{a}_{t},\mathbf{z}_{t}|\mathbf{x}_{t})\;\middle\|\;\pi_{i}(\mathbf{a}_{t},\mathbf{z}_{t}|\mathbf{x}_{t})\right)\right]$
$\displaystyle=\sum_{t}\mathbb{E}_{\pi(\tau)}[\mathrm{D_{KL}}\left(\pi^{H}(\mathbf{z}_{t}|\mathbf{x}_{t})\;\middle\|\;\pi^{H}_{i}(\mathbf{z}_{t}|\mathbf{x}_{t})\right)$
$\displaystyle+\operatorname{\mathbb{E}}_{\pi^{H}(\mathbf{z}_{t}|\mathbf{x}_{t})}\left[\mathrm{D_{KL}}\left(\pi^{L}(\mathbf{a}_{t}|\mathbf{x}_{t},\mathbf{z}_{t})\;\middle\|\;\pi^{L}_{i}(\mathbf{a}_{t}|\mathbf{x}_{t},\mathbf{z}_{t})\right)\right]]$

###  B.3 Policy Gradient Lower Bounds

####  B.3.1 Importance Weights Derivation

$\displaystyle\mathrm{\tilde{D}}^{q(\tau)}_{\mathrm{{KL}}}(\pi(\tau)||\pi_{i}(\tau))=ub(\mathrm{D_{KL}}\left(\pi(\tau)\;\middle\|\;\pi_{i}(\tau)\right))$
(15)

For $\beta^{z}_{i},\beta^{a}_{i}=1$, where
$ub(\mathrm{D_{KL}}\left(\pi(\tau)\;\middle\|\;\pi_{i}(\tau)\right))$
corresponds to the hierarchical upper bound introduced in Section A.2.

##### Proof

$\displaystyle
ub(\mathrm{D_{KL}}\left(\pi(\tau)\;\middle\|\;\pi_{i}(\tau)\right))$
$\displaystyle=\sum_{t}\mathbb{E}_{\pi(\tau)}[\mathrm{D_{KL}}\left(\pi^{H}(\mathbf{z}_{t}|\mathbf{x}_{t})\;\middle\|\;\pi^{H}_{i}(\mathbf{z}_{t}|\mathbf{x}_{t})\right)$
(16)
$\displaystyle+\operatorname{\mathbb{E}}_{\pi^{H}(\mathbf{z}_{t}|\mathbf{x}_{t})}\left[\mathrm{D_{KL}}\left(\pi^{L}(\mathbf{a}_{t}|\mathbf{x}_{t},\mathbf{z}_{t})\;\middle\|\;\pi^{L}_{i}(\mathbf{a}_{t}|\mathbf{x}_{t},\mathbf{z}_{t})\right)\right]]$
$\displaystyle=\sum_{t}\mathbb{E}_{q(\tau)\cdot\frac{\pi(\tau)}{q(\tau)}}[\mathrm{D_{KL}}\left(\pi^{H}(\mathbf{z}_{t}|\mathbf{x}_{t})\;\middle\|\;\pi^{H}_{i}(\mathbf{z}_{t}|\mathbf{x}_{t})\right)$
$\displaystyle+\operatorname{\mathbb{E}}_{\pi^{H}(\mathbf{z}_{t}|\mathbf{x}_{t})}\left[\mathrm{D_{KL}}\left(\pi^{L}(\mathbf{a}_{t}|\mathbf{x}_{t},\mathbf{z}_{t})\;\middle\|\;\pi^{L}_{i}(\mathbf{a}_{t}|\mathbf{x}_{t},\mathbf{z}_{t})\right)\right]]$
$\displaystyle=\sum_{t}\mathbb{E}_{q(\tau)\cdot\prod_{i=0}^{t}\frac{\pi(\mathbf{a}_{i}|\mathbf{x}_{i})}{q(\mathbf{a}_{i}|\mathbf{x}_{i})}}[\mathrm{D_{KL}}\left(\pi^{H}(\mathbf{z}_{t}|\mathbf{x}_{t})\;\middle\|\;\pi^{H}_{i}(\mathbf{z}_{t}|\mathbf{x}_{t})\right)$
$\displaystyle+\operatorname{\mathbb{E}}_{\pi^{H}(\mathbf{z}_{t}|\mathbf{x}_{t})}\left[\mathrm{D_{KL}}\left(\pi^{L}(\mathbf{a}_{t}|\mathbf{x}_{t},\mathbf{z}_{t})\;\middle\|\;\pi^{L}_{i}(\mathbf{a}_{t}|\mathbf{x}_{t},\mathbf{z}_{t})\right)\right]]$
$\displaystyle=\mathrm{\tilde{D}}^{q(\tau)}_{\mathrm{{KL}}}(\pi(\tau)||\pi_{i}(\tau))$

####  B.3.2 Behavioral Cloning Upper Bound

$\displaystyle-\mathrm{D}_{\mathrm{{KL}}}(\pi(\tau)||\pi_{e}(\tau))$
$\displaystyle\geq-\sum_{i\in\\{0,u,e\\}}\mathrm{\tilde{D}}^{\pi_{e}(\tau)}_{\mathrm{{KL}}}(\pi(\tau)||\pi_{i}(\tau))$
(17) $\displaystyle\textit{for}\quad\beta_{i}^{z},\beta_{i}^{a}\geq 1$

##### Proof

$\displaystyle\mathrm{D_{KL}}\left(\pi(\tau)\;\middle\|\;\pi_{e}(\tau)\right)$
$\displaystyle\leq\sum_{i\in\\{0,u,e\\}}\mathrm{D_{KL}}\left(\pi(\tau)\;\middle\|\;\pi_{i}(\tau)\right)$
(18)
$\displaystyle\leq\sum_{i\in\\{0,u,e\\}}ub(\mathrm{D_{KL}}\left(\pi(\tau)\;\middle\|\;\pi_{i}(\tau)\right))$
$\displaystyle=\sum_{i\in\\{0,u,e\\}}\mathrm{\tilde{D}}^{q(\tau)}_{\mathrm{{KL}}}(\pi(\tau)||\pi_{i}(\tau))\quad\textit{for}\quad\beta_{i}^{z},\beta_{i}^{a}=1$
$\displaystyle\leq\sum_{i\in\\{0,u,e\\}}\mathrm{\tilde{D}}^{q(\tau)}_{\mathrm{{KL}}}(\pi(\tau)||\pi_{i}(\tau))\quad\textit{for}\quad\beta_{i}^{z},\beta_{i}^{a}\geq
1$

The last line holds true as each weighted term in
$\mathrm{\tilde{D}}^{q(\tau)}_{\mathrm{{KL}}}(\pi(\tau)||\pi_{i}(\tau))$
corresponds to KL-divergences which are positive.

####  B.3.3 Reinforcement Learning Upper Bound

$\displaystyle\operatorname{\mathbb{E}}_{\begin{subarray}{c}p(\mathcal{K}),\\\
\pi_{0}(\tau)\end{subarray}}\left[\log(O=1|\tau,k)\right]\geq\operatorname{\mathbb{E}}_{\pi_{b}(\tau)}\left[\sum_{t}\nu^{\pi_{b}}[t]\cdot
r_{k}(\mathbf{x}_{t},\mathbf{a}_{t})\right]-\sum_{i\in\\{0,u\\}}\mathrm{\tilde{D}}^{\pi_{b}(\tau)}_{\mathrm{{KL}}}(\pi(\tau)||\pi_{i}(\tau))$
(19) $\displaystyle\textit{for}\quad\beta_{i}^{z},\beta_{i}^{a}\geq
1\quad\textit{and}\quad r_{k}<0$

##### Proof

$\displaystyle\operatorname{\mathbb{E}}_{\begin{subarray}{c}p(\mathcal{K}),\\\
\pi_{0}(\tau)\end{subarray}}\left[\log(O=1|\tau,k)\right]$
$\displaystyle\geq\operatorname{\mathbb{E}}_{\pi_{b}(\tau)}\left[\sum_{t}\nu^{\pi_{b}}[t]\cdot
r_{k}(\mathbf{x}_{t},\mathbf{a}_{t})\right]-\mathrm{D_{KL}}\left(\pi(\tau)\;\middle\|\;\pi_{i}(\tau)\right)$
(20)
$\displaystyle\geq\operatorname{\mathbb{E}}_{\pi_{b}(\tau)}\left[\sum_{t}\nu^{\pi_{b}}[t]\cdot
r_{k}(\mathbf{x}_{t},\mathbf{a}_{t})\right]-\mathrm{\tilde{D}}^{\pi_{b}(\tau)}_{\mathrm{{KL}}}(\pi(\tau)||\pi_{i}(\tau))\textit{,
}\textit{ }\textit{for }\beta_{i}^{z},\beta_{i}^{a}=1$
$\displaystyle\geq\operatorname{\mathbb{E}}_{\pi_{b}(\tau)}\left[\sum_{t}\nu^{\pi_{b}}[t]\cdot
r_{k}(\mathbf{x}_{t},\mathbf{a}_{t})\right]-\mathrm{\tilde{D}}^{\pi_{b}(\tau)}_{\mathrm{{KL}}}(\pi(\tau)||\pi_{i}(\tau))\textit{,
}\textit{ }\textit{for }\beta_{i}^{z},\beta_{i}^{a}\geq 1$
$\displaystyle\geq\operatorname{\mathbb{E}}_{\pi_{b}(\tau)}\left[\sum_{t}\nu^{\pi_{b}}[t]\cdot
r_{k}(\mathbf{x}_{t},\mathbf{a}_{t})\right]-\sum_{i\in\\{0,u\\}}\mathrm{\tilde{D}}^{\pi_{b}(\tau)}_{\mathrm{{KL}}}(\pi(\tau)||\pi_{i}(\tau))\textit{,
}\textit{ }$ $\displaystyle\textit{for}\quad\beta_{i}^{z},\beta_{i}^{a}\geq 1$

For proof for line $1$, please refer to [4]. The final $2$ lines hold due to
positive KL-divergences.

##  Appendix C Environments

We continue by covering each environment setup in detail.

###  C.1 CorridorMaze

Intuitively, the agent starts at the intersection of corridors, at the origin,
and must traverse corridors, aligned with each dimension of the state space,
in a given ordering. This requires the agent to reach the end of the corridor
(which we call half-corridor cycle), and return back to the origin, before the
corridor is considered complete.

$s\in\\{0,l\\}^{c}$, $p(s_{0})=\mathbf{0}^{c}$, $k=$ one-hot task encoding,
$a\in[0,1]$, $r_{k}^{\text{semi-
sparse}}(\mathbf{x}_{t},\mathbf{a}_{t})=1\quad\textit{if}$ agent has correctly
completed the entire or half-corridor cycle $\quad\textit{else}\quad 0$,
$r_{k}^{\text{sparse}}(\mathbf{x}_{t},\mathbf{a}_{t})=1\quad\textit{if}\quad\text{task
complete}\quad\textit{else}\quad 0$. Task is considered complete when a
desired ordering of corridors have been traversed. $c=5$ represents the number
of corridors in our experiments. $l=6$, the lengths of each corridor. States
transition according to deterministic transition function
$s^{j_{t}}_{t+1}=f(\mathbf{s}^{j_{t}}_{t},\mathbf{a}_{t})$. $j_{t}$
corresponds to the index of the current corridor that the agent is in (i.e.
$j_{t}=0$ if the agent is in corridor $0$ at timestep $t$). $\mathbf{s}^{i}$
corresponds to the $i^{th}$ dimension of the state. States transition
incrementally or decrementally down a corridor, and given state dimension
$s^{i}$, if actions fall into corresponding transition action bins
$\psi_{inc},\psi_{dec}$. Specifically we define the transition function as
follows:

$f(\mathbf{s}^{j_{t}}_{t},\mathbf{a}_{t})=\left\\{\begin{array}[]{ll}\mathbf{s}_{t}^{j_{t}}+1,&\mbox{if
$\psi_{inc}^{w}(\mathbf{a}_{t},j_{t})$}.\\\
\mathbf{s}_{t}^{j_{t}}-1,&\mbox{elif
$\psi_{dec}^{w}(\mathbf{a}_{t},j_{t})$}.\\\
0,&\mbox{otherwise}.\par\end{array}\right.$ (21)

$\psi_{inc}^{w}(\mathbf{a}_{t},j)=\text{bool}(\mathbf{a}_{t}$ in
$[j/c,(j+0.5\cdot w)/c])$,
$\psi_{dec}^{w}(\mathbf{a}_{t},j)=\text{bool}(\mathbf{a}_{t}$ in $[j/c,(2\cdot
j-0.5\cdot w)/c])$. The smaller the $w$ parameter, the narrower the
distribution of actions that lead to transitions. As such, $w$, together with
$r_{k}$ controls the exploration difficulty of task $k$. We set $w=0.9$. We
constrain the state transitions to not transition outside of the corridor
boundaries. Furthermore, if the agent is at the origin, $s=\mathbf{0}^{c}$ (at
the intersection of corridors), then the transition function is ran for all
values of $j_{t}$, thereby allowing the agent to transition into any corridor.

###  C.2 Stack

This domain is adapted from the well known gym robotics FetchPickAndPlace-v0
environment [38]. The following modifications were made: 1) 3 additional
blocks were introduced, with different colours, and a goal pad, 2) object
spawn locations were not randomized and were instantiated equidistantly around
the goal pad, see Fig. 2, 3) the number of substeps was increased from $20$ to
$60$, as this reduced episodic lengths, 4) a transparent hollow rectangular
tube was placed around the goal pad, to simplify the stacking task and prevent
stacked objects from collapsing due to structural instabilities, 5) the arm
was always spawned over the goal pad, see figure Fig. 2, 6) the state space
corresponded to gripper position and grasp state, as well as the object
positions and relative positions with respect to the arm: velocities were
omitted as access to such information may not be realistic for real robotic
systems. $k=$ one-hot task encoding,
$r_{k}^{\text{sparse}}(\mathbf{x}_{t},\mathbf{a}_{t})=1\quad\textit{if}\quad\text{correct
object has been placed on stack in correct ordering}\quad\textit{else}\quad
0$.

##  Appendix D Experimental Setup

We provide the reader with the experimental setup for all training regimes and
environments below. We build off the softlearning code base [5]. Algorithmic
details not mentioned in the following sections are omitted as are kept
constant with the original code base. For all experiments, we sample batch
size number of entire episodes of experience during training.

###  D.1 Model Architectures

We continue by outlining the shared model architectures across domains and
experiments. Each policy network (e.g.
$\pi^{H},\pi^{L},\pi^{H}_{0},\pi^{L}_{0}$) is comprized of a feedforward
module outlined in Table 4. The softlearning repository that we build off [5],
applies tanh activation over network outputs, where appropriate, to match the
predefined outpute ranges of any given module. The critic is also comprized of
the same feedforward module, but is not hierarchical. To handle historical
inputs, we tile the inputs and flatten, to become one large input
1-dimensional array. We ensure the input always remains of fixed size by
appropriately left padding zeros. For $\pi^{H},\pi^{H}_{0}$ we use a
categorical latent space of size $10$. We found this dimensionality sufficed
for expressing the diverse behaviours exhibited in our domains. Table 5
describes the setup for all the experiments in the main paper, including
inputs to each module, level over which KL-regularization occurs ($z$ or $a$),
which modules are shared (e.g. $\pi^{L}$ and $\pi^{L}_{0}$), and which modules
are reused across sequential tasks. For the covariate-shift designed
experiments in Table 3(b), we additionally reuse $\pi^{H}$ (or $\pi^{L}$ for
RecSAC) across domains, and whose input is $\mathbf{x}_{t}$. For all the above
experiments, any reused modules are not given access to task-dependent
information, namely task-id ($k$) and exteroceptive information (cube
locations for Stack domain). This design choice ensures reused modules
generalize across task instances.

Table 4: Feedforward Module, $\pi^{\\{H,L\\}}_{(0)}$ hidden layers | $(512,512)$
---|---
hidden layer activation | relu
output activation | linear
Table 5: Full experimental setup. Describes inputs to each module, level over which KL-regularization occurs, which modules are shared, and which are reused across training and transfer tasks. Name | $\pi^{H}_{0}$ | $\pi^{L}_{0}$ | $\pi^{L}$ | $\pi^{H}$ | KL level | $\pi^{L}=\pi^{L}_{0}$ | reused modules
---|---|---|---|---|---|---|---
APES-H20 | $\mathbf{x}_{t-20:t}$ | $\mathbf{s}_{t}$ | $\mathbf{s}_{t}$ | $\mathbf{x}_{k}$ | $z$ | ✓ | $\pi^{H}_{0},\pi^{L}_{0},\pi^{L}$
APES-H10 | $\mathbf{x}_{t-10:t}$ | $\mathbf{s}_{t}$ | $\mathbf{s}_{t}$ | $\mathbf{x}_{k}$ | $z$ | ✓ | $\pi^{H}_{0},\pi^{L}_{0},\pi^{L}$
APES-H1 | $\mathbf{x}_{t-1:t}$ | $\mathbf{s}_{t}$ | $\mathbf{s}_{t}$ | $\mathbf{x}_{k}$ | $z$ | ✓ | $\pi^{H}_{0},\pi^{L}_{0},\pi^{L}$
APES-S | $\mathbf{s}_{t}$ | $\mathbf{s}_{t}$ | $\mathbf{s}_{t}$ | $\mathbf{x}_{k}$ | $z$ | ✓ | $\pi^{H}_{0},\pi^{L}_{0},\pi^{L}$
APES-no_prior | $\\_$ | $\\_$ | $\mathbf{s}_{t}$ | $\mathbf{x}_{k}$ | $\\_$ | $\\_$ | $\pi^{L}$
Hier-RecSAC | $\\_$ | $\\_$ | $\mathbf{s}_{t}$ | $\mathbf{x}_{k}$ | $\\_$ | $\\_$ | $\\_$
RecSAC | $\\_$ | $\\_$ | $\mathbf{x}_{k}$ | $\\_$ | $\\_$ | $\\_$ | $\\_$
APES-H1-KL-a | $\mathbf{x}_{t-1:t}$ | $\mathbf{s}_{t}$ | $\mathbf{s}_{t}$ | $\mathbf{x}_{k}$ | $a$ | ✗ | $\pi^{H}_{0},\pi^{L}_{0}$
APES-H1-flat | $\\_$ | $\mathbf{x}_{t-1:t}$ | $\mathbf{s}_{t}$ | $\mathbf{x}_{k}$ | $a$ | ✗ | $\pi^{L}_{0}$

###  D.2 Behavioural Cloning

For the BC setup, we use a deterministic, noisy, expert controller to create
experience to learn off. We apply DAGGER [23] during data collection and
training of policy $\pi$ as we found this aided at achieving a high success
rate at the BC tasks. Our DAGGER setup intermittently during data collection,
with a predefined rate, samples an action from $\pi$ instead of $\pi_{e}$, but
still saves BC target action $a_{t}$ as the one that would have been taken by
the expert for $\mathbf{x}_{k}$. This setup helps mitigate covariate shift
during training, between policy and expert. Noize levels were chosen to be
small enough so that the expert still succeeded at the task. We trained our
policies for one epoch (once over each collected data sample in the
expectation). It may be possible to be more sample efficient, by increasing
the ratio of gradient steps to data collection, but we did not explore this
direction. The interplay we use between data collection and training over the
collected experience, is akin to the RL paradigm. We build off the
softlearning code base [5], so please refer to it for details regarding this
interplay.

Table 6: Full Training Setup

(a) Behavioural Cloning Setup  Environment | CorridorMaze | Stack
---|---|---
$\pi_{(i)}$ learning rate | $3e^{-4}$ | $3e^{-4}$
$z$ categorical size | $10$ | $10$
$\pi^{H}$ history-depth | $24$ | $5$
$\beta^{z}_{u}$ | $1e^{-3}$ | $1e^{-3}$
$\beta^{a}_{u}$ | $1e^{-2}$ | $1e^{-2}$
$\beta^{a}_{e}$ | $1$ | $1$
$\beta^{z}_{0}$ | $1$ | $1$
$\beta^{a}_{0}$ | $1$ | $1$
DAGGER rate | $0.1$ | $0.1$
batch size | $128$ | $128$
episodic length | $24$ | $35$

(b) Reinforcement Learning Setup  Environment | CorridorMaze | Stack
---|---|---
Reward Type | sparse | semi-sparse | sparse
Transfer task | 2 corridor | 4 corridor | 4 blocks
$Q$ learning rate | $3e^{-6}$ | $3e^{-5}$ | $3e^{-5}$
$\pi$ learning rate | $3e^{-4}$ | $3e^{-4}$ | $3e^{-4}$
$\beta^{z/a}_{0}$ | $1e^{-2}$ | $1e^{-1}$ | $5e^{-2}$
$\beta^{z/a}_{u}$ | $1e^{-2}$ | $0$ | $5e^{-4}$
$Q$ update rate | $6e^{-4}$ | $6e^{-4}$ | $6e^{-4}$
Retrace $\lambda$ | $0.99$ | $0.99$ | $0.99$
batch size | $128$ | $128$ | $128$
episodic length | $30$ | $60$ | $65$

Refer to Table 6(a) for BC algorithmic details. It is important to note here
that, although we report five $\beta$ hyper-parameter values, there are only
two degrees of freedom. As we stop gradients flowing from $\pi_{0}$ to $\pi$,
choice of $\beta_{0}$ is unimportant (as long as it is not $0$) as it does not
influence the interplay between gradients from individual loss terms. We set
these values to $1$. $\beta^{a}_{e}$’s absolute value is also unimportant, and
only its relative value compared to $\beta^{z}_{u}$ and $\beta^{a}_{u}$
matters. We also set $\beta^{a}_{e}$ to $1$. For the remaining two hyper-
parameters, $\beta^{z}_{u},\beta^{a}_{u}$, we performed a hyper-parameter
sweep over three orders of magnitude, three values across each dimension, to
obtain the reported optimal values. In practice
$\pi_{0}=\\{\pi_{i}\\}_{i\in\\{0,...,N\\}}$, multiple trained priors each
sharing the same $\beta_{0}$ hyper-parameters. Four seeds were ran, as for all
experiments reported in this paper. We observed very small variation in
learning across the seeds, and used the best performing seed to bootstrap
learning off for all latter experiments. We separately also performed a hyper-
parameter sweep over $\pi$ learning rate, in the same way as before. We did
not perform a sweep for batch size. We found for both BC and RL setups, that
conditioning on entire history for $\pi^{H}$ was not always necessary, and
sometimes hurt performance. We state the history lengths used for $\pi^{H}$
for BC in Table 6(a). This value was also used for both $\pi^{H}$ and $Q$ for
the RL setup.

We prevent gradient flow from $\pi_{0}$ to $\pi$, to ensure as fair a
comparison between ablations as possible: each prior distils knowledge from
the same, high performing, policy $\pi$ and dataset. If we simultaneously
trained multiple $\pi$ and $\pi_{0}$ pairs (for each distinct prior), it is
possible that different learnt priors would influence the quality of each
policy $\pi$ which knowledge is distilled off. In this paper, we are not
interested in investigating how priors affect $\pi$ during BC, but instead how
priors influence what knowledge can be distilled and transferred. We observed
prior KL-distillation loss convergence across tasks and seeds, ensuring a fair
comparison.

###  D.3 Reinforcement Learning

During this stage of training we freeze the prior and low-level policy (if
applicable, depending on the ablation). In general, any reused modules across
sequential tasks are frozen (apart from $\pi^{H}$ for the covariate shift
experiments in Table 3(b)). Any modules that are not shared (such as $\pi^{H}$
for most experiments), are initialized randomly across tasks. The RL setup is
akin to the softlearning repository [5] that we build off. We note any changes
in Table 6(b). We regularize against the latent or action level for, depending
on the ablation, whether or not our models are hierarchical, share low level
policies, or use pre-trained modules (low-level policy and prior). Therefore,
we only ever regularize against, at most, two $\beta$ hyper-parameters. Hyper-
parameter sweeps are performed in the same way as in the previous experiments.
We did not sweep over Retrace $\lambda$, batch size, or episodic length. For
Retrace, we clip the importance weights between $[0,1]$, like in the original
paper [25], and perform $\lambda$ returns rather than n-step. We found the
Retrace operator important for sample-efficient learning.

##  Appendix E Final Policy Rollouts and Analysis

(a) _CorridorMaze 2 corridor_. 10 final policy rollouts (episodes vertically,
rollouts horizontally) for most performant method. Task: Blue -> Orange
corridors. Policy distribution over categorical latent space for $\pi^{H}$ and
$\pi^{H}_{0}$ plotted (between policy rollouts, denoted by $x_{t}$) as
vertical histograms, with colour and width denoting category and probability.
Colour here does not correlate to corridor.

(b) _Stack 4 blocks_. _Top)_ Policy rollouts for most performant method. Task:
stack blocks in order 0 -> 1 -> 2 -> 3\. _Bottom)_ policy distributions akin
to Fig. 5(a). Horizontal dashed lines in bottom plots refer to current sub-
task (block stack), vertically transitioning upon each completion.

Figure 5: Final transfer performance for most performant method. Tasks are
solved. Displaying latent distribution for both policy and prior. For both
domains, policy deviates most from prior at the bottleneck state
(hallway/stacking zone, for CorridorMaze/Stack), where prior multimodality
exists (e.g. which corridor/block to stack next), but where determinism is
required for the task at hand. Figure 6: _CorridorMaze 4 corridor_. 10 final
policy rollouts (episodes vertically, rollouts horizontally) for most
performant method. Task: Blue -> Orange -> Green -> Red corridors. Policy
distribution over categorical latent space for $\pi^{H}$ and $\pi^{H}_{0}$
plotted (between policy rollouts, denoted by $x_{t}$) as vertical histograms,
with colour and width denoting category and probability.

In this section we show final policy performance (in terms of episodic
rollouts) for the most performant method, _APES-H1_ , across each transfer
domain. We additionally display the categorical probability distributions for
$\pi^{H}(\mathbf{x}_{k})$ and $\pi^{H}_{0}(F(\mathbf{x}_{k}))$ across each
rollout to analyse the behaviour of each. $F(.)$ denotes the chosen
_information gating function_ for the prior. For _CorridorMaze 2 and 4
corridor_ , seen in Figs. 5(a) and 6, we see that the full method successfully
solves each respective task, correctly traversing the correct ordering of
corridors. The categorical distributions for these domains remain relatively
entropic. In general, latent categories cluster into those that lead the agent
deeper down a corridor, and those that return the agent to the hallway. Policy
and prior align their categorical distributions in general, as expected.
Interestingly, however, the two categorical distributions deviate the most
from each-other at the hallway, the _bottleneck state_ [28], where prior
multimodality (for hierarchical $\pi_{0}$) exists most (e.g. which corridor to
traverse next). In this setting, the policy needs to deviate from the
multimodal prior, and traverse only the optimal next corridor. We also
observe, for the hallway, that the prior allocates one category to each of the
five corridors. Such behaviour would not be possible with a flat prior.

Fig. 5(b) plots the same information for _Stack 4 blocks_. APES-H1
successfully solves the transfer task, stacking all blocks according to their
masses. Similar categorical latent-space trends exist for this domain as the
previous. Most noteworthy is the behaviour of both policy and prior at the
_bottleneck state_ , the location above the block stack, where blocks are
placed. This location is visited five times within the episode: at the start
$\mathbf{s}_{0}$, and four more times upon each stacked block. Interestingly,
for this state, the prior becomes increasingly less entropic upon each
successive visit. This suggests that the prior has learnt that the number of
feasible _high-level_ actions (corresponding to which block to stack next),
reduces upon each visit, as there remains fewer lighter blocks to stack. It is
also interesting that for $\mathbf{s}_{0}$, the red categorical value is more
favoured than the rest. Here, the red categorical value corresponds to moving
towards cube $0$, the heaviest cube. This behaviour is as expected, as during
BC, this cube was stacked first more often than the others, given its mass.
For this domain, akin to _CorridorMaze_ , the policy deviates most from the
prior at the bottleneck state, as here it needs to behave deterministically
(regrading which block to stack next).