# Adaptive Policy Transfer in Reinforcement Learning

Girish Joshi and Girish Chowdhary *This work was supported in part by ONR MURI
Navy N00014-19-1-2373, and AFOSR YIP FA9550-15-1-0146Coordinated Sciences
Laboratory, University of Illinois, Urbana-Champaign<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Efficient and robust policy transfer remains a key challenge for reinforcement
learning to become viable for real-wold robotics. Policy transfer through warm
initialization, imitation, or interacting over a large set of agents with
randomized instances, have been commonly applied to solve a variety of
Reinforcement Learning tasks. However, this seems far from how skill transfer
happens in the biological world: Humans and animals are able to quickly adapt
the learned behaviors between similar tasks and learn new skills when
presented with new situations. Here we seek to answer the question: Will
learning to combine adaptation and exploration lead to a more efficient
transfer of policies between domains? We introduce a principled mechanism that
can “Adapt-to-Learn", that is adapt the source policy to learn to solve a
target task with significant transition differences and uncertainties. We show
that the presented method learns to seamlessly combine learning from
adaptation and exploration and leads to a robust policy transfer algorithm
with significantly reduced sample complexity in transferring skills between
related tasks.

##  I Introduction

Lack of principled mechanisms to quickly and efficiently transfer learned
policies when the robot dynamics or tasks significantly change, has become the
major bottleneck in Reinforcement Learning (RL). This inability to transfer
policies is one major reason why RL has still not proliferated physical
application like robotics [1, 2, 3, 4]. This lack of methods to efficiently
transfer robust policies forces the robot to learn to control in isolation and
from scratch, even when only a few parameters of the dynamics change across
tasks (e.g. walking with different weight or surface friction, etc.). This is
both computation and sample expensive, and also makes simulation-to-real
transfer difficult.

Adapt-to-Learn (ATL) is inspired by the fact that adaptation of behaviors
combined with learning through experience is a primary mechanism of learning
in biological creatures [5, 6]. Imitation Learning (IL) [7, 8] also seems to
play a very crucial part in skill transfer, and as such has been widely
studied in RL. In control theory, adaptation has been typically studied for
tracking problems on deterministic dynamical systems with well-defined
reference trajectories [9, 10]. Inspired by learning in biological world, we
seek to answer the question: Will combining adaptation of transferred policies
and learning from exploration lead to more efficient policy trasnfer? In this
paper, we show that the ability to adapt and incorporate further learning can
be achieved through optimizing over combined environmental and intrinsic
adaptation rewards. Learning over the combined reward feedback, we ensure that
the agent quickly adapts and also learns to acquire skills beyond what the
source policy can teach. Our empirical results show that the presented policy
transfer algorithm is able to adapt to significant differences in the
transition models, which otherwise, using Imitation Learning or Guided Policy
Search (GPS) [11] would fail to produce stable solution. We posit that the
presented method can be the foundation of a broader class of RL algorithms
that can choose seamlessly between learning through exploration and supervised
adaption using behavioral imitation.

##  II Related work

Deep Reinforcement Learning (D-RL) has recently enabled agents to learn
policies for complex robotic tasks in simulation [1, 2, 3, 4]. However, D-RL
applications to robotics have been plagued by the curse of sample complexity.
Transfer Learning (TL) seeks to mitigate this learning inefficiency of D-RL
[12]. A significant body of literature on transfer in RL is focused on
initialized RL in the target domain using learned source policy; known as
jump-start/warm-start methods [13, 14, 15]. Some examples of these transfer
architectures include transfer between similar tasks [16], transfer from human
demonstrations [17] and transfer from simulation to real [18, 19, 20].
However, warm-start based approaches do not always work in transfer for RL,
even when similar warm-starts are quite effective in supervised learning.
Efforts have also been made in exploring accelerated learning directly on real
robots, through Guided Policy Search (GPS) [21] and parallelizing the training
across multiple agents using meta-learning [22, 23, 8], but these approaches
are prohibitively sample expensive for real-world robotics. Sim-to-Real
transfers have been widely adopted in the recent works and can be viewed as a
subset of same domain transfer problems. Daftry et al. [24] demonstrated the
policy transfer for control of aerial vehicles across different vehicle models
and environments. Policy transfer from simulation to real using an inverse
dynamics model estimated interacting with the real robot is presented by [25].
The agents trained to achieve robust policies across various environments
through learning over an adversarial loss is presented in [26]. Here we
present a general architecture capable of transferring policies across MDPs
with significantly different transition dynamics.

##  III Preliminaries and Problem Specification

Consider an infinite horizon MDP defined as a tuple
$\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\rho_{0},\gamma)$,
where $\mathcal{S}$ denote set of continuous states; $\mathcal{A}$ is a set of
continuous bounded actions,
$\mathcal{P}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\to\mathbb{R}_{+}$
is the state transition probability distribution of reaching $s^{\prime}$ upon
taking action $a$ in $s$, $\rho_{0}:\mathcal{S}\to\mathbb{R}_{+}$ is the
distribution over initial states $s_{0}$ and
$\mathcal{R}:\mathcal{S}\times\mathcal{A}\to\mathbb{R}_{+}$ is deterministic
reward function, s.t. $\mathcal{R}\in[0,1]$ and $\gamma$ is the discount
factor s.t $0<\gamma\leq 1$.

Let $\pi(a|s):\mathcal{S}\times\mathcal{A}\to[0,1]$ be stochastic policy over
continuous state and action space. The action from a policy is a draw from the
distribution $a_{i}\sim\pi(.|s_{i})$. The agent’s goal is to find a policy
$\pi^{*}$ which maximize the total return. The total return under a policy
$\pi$ is given as,

$\eta_{\pi}(s_{0})=\mathbb{E}_{s_{0},a_{0},\dots}\left(\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})\right).$
(1)

where, $s_{0}\sim\rho_{0}$, $a_{t}\sim\pi(.|s_{t})$ and $s_{t+1}\sim
p(.|s_{t},a_{t})$.

We will use the following definition of the state value function $V^{\pi}$ and
state-action value function $Q^{\pi}$ defined under any policy $\pi$

$V^{\pi}(s_{t})=\mathbb{E}_{a_{t},s_{t+1},a_{t+1},\dots}\left(\sum_{i=0}^{\infty}\gamma^{i}r(s_{i+t},a_{i+t})\right),$
$Q^{\pi}(s_{t},a_{t})=\mathbb{E}_{s_{t+1},a_{t+1},\dots}\left(\sum_{i=0}^{\infty}\gamma^{i}r(s_{i+t},a_{i+t})\right).$

With the quick overview of the preliminaries, we now specify the problem of
policy transfer studied here: Consider a source MDP
$\mathcal{M}^{S}=(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\rho_{0},\gamma)^{S}$,
and a target MDP
$\mathcal{M}^{T}=(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\rho_{0},\gamma)^{T}$,
each with its own state-action space and transition model respectively. We
will mainly focus on the problem of same domain transfer in this paper, where
the state and action space are analogous
$\mathcal{S}^{(S)}=\mathcal{S}^{(T)}=\mathcal{S}\subseteq\mathbb{R}^{m}$ and
$\mathcal{A}^{(T)}=\mathcal{A}^{(S)}=\mathcal{A}\subseteq\mathbb{R}^{k}$, but
the source and target state transition models differ significantly due to
unmodeled dynamics or external environment interactions. We are concerned with
this problem because such policy adaptation in the presence of significant
transition model change happens quite often in robotics, e.g. change in
surface friction, payload carried by robot, modeling differences, or component
deterioration. Note that extension to cross-domain transfer could be achieved
with domain alignment techniques such as manifold alignment (MA) [27], see
[28] for a model-based policy transfer method for RL with MA.

Let $\pi^{*}$ be a parameterized optimal stochastic policy for source MDP
$\mathcal{M}^{S}$. The source policy $\pi^{*}$ can be obtained using any
available RL methods [29, 30, 31]. We use Proximal Policy Optimization (PPO)
[31] algorithm to learn a optimal source policy on a unperturbed source task.
A warm-started PPO-TL [15] policy, trained over a perturbed target task and
ideal imitation learning is used as the TL solutions against which our
proposed ATL policy is compared.

##  IV Adaptive Policy Transfer

In this paper, we approach the problem of transfer learning for RL through
adaptation of previously learned policies. Our goal in the rest of this paper
is to show that an algorithm that can judiciously combine adaptation and
learning is capable of avoiding brute force random exploration to a large
extent and therefore can be significantly less sample expensive.

Towards this goal, our approach is to enable the agent to learn the optimal
mixture of adaptation (which we term as behavioral imitation) and learning
from exploration. Our method differs from existing transfer methods that rely
on warm start for TL or policy imitation [8, 7] in a key way: Unlike imitation
learning, we do not aim to emulate the source optimal policy. Instead, we try
to adapt and encourage directional search for optimal reward by mimicking the
source transitions under source optimal policy as projected onto the target
state space. The details of Adaptive policy Transfer is presented further in
this section.

###  IV-A Adapt-to-Learn: Policy Transfer in RL

ATL has two components to policy transfer, Adaptation and Learning. We begin
by mathematically describing our approach of adaptation for policy transfer in
RL and state all the necessary assumptions in Section IV-A1. We then develop
the Adapt-to-Learn algorithm in Section IV-A2 by adding the learning component
by random exploration through reward mixing.

####  IV-A1 Adaptation for policy learning

Adaptation of policies aims to generate the policy $\pi_{\theta}(.|s)$ such
that at every given $s\in\mathcal{S}^{T}$ the Target transition approximately
mimics the Source transitions under Source optimal policy as projected onto
the Target state space. We can loosely state the adaptation objective as
$\mathcal{P}^{T}(.|s,\pi_{\theta}(a|s))\approx\mathcal{P}^{S}(.|\hat{s},\pi^{*}(a^{\prime}|\hat{s})),\hskip
5.69054pt\forall s\in\mathcal{S}^{T},$. Where $\hat{s}=\chi_{T}^{S}(s)$, i.e.
target state projected onto source manifold. The mapping $\chi_{T}^{S}$ is a
bijective Manifold Alignment (MA) between source and target, state-action
spaces $\\{\mathcal{S},\mathcal{A}\\}^{S}$,
$\\{\mathcal{S},\mathcal{A}\\}^{T}$. The MA mapping is required for cross-
domain transfers [15, 28]. For ease of exposition, in this paper, we focus on
same domain transfer and assume MA mapping to be identity, such that
$\chi_{T}^{S}=\chi_{S}^{T}=I$.

Note that our goal is not to directly imitate the policy itself, but rather
use the source behavior as a baseline for finding optimal transitions in the
target domain. This behavioral imitation objective can be achieved by
minimizing the average KL divergence between point-wise local target
trajectory deviation likelihoods
$p^{T}(s^{\prime}_{t+1}|s_{t},\pi_{\theta}(.|s_{t}))$, and the source
transition under source optimal policy
$p^{S}(s^{\prime}_{t+1}|s_{t},\pi^{*}(.|s_{t}))$.

We define the target transition deviation likelihood as conditional
probability of landing in the state $s^{\prime}_{t+1}$ starting from state
$s_{t}$ under some action $a_{t}\sim\pi_{\theta}(.|s_{t})$. Unlike state
transition probability, the state $s^{\prime}_{t+1}$ is not the next
transitioned state but a random state at which this probability is evaluated
(Hence $s^{\prime}_{t+1}$ need not have the time stamp).

The adaptation objective can be formalized as minimizing the average KL-
divergence [30] between source and target transition trajectories as follows,

$\displaystyle\eta_{{}_{KL}}(\pi_{\theta},\pi^{*})$ $\displaystyle=$
$\displaystyle D_{KL}(p_{\pi_{\theta}}(\tau)\|q_{\pi^{*}}(\tau)),$
$\displaystyle\eta_{{}_{KL}}(\pi_{\theta},\pi^{*})$ $\displaystyle=$
$\displaystyle\int_{\mathcal{S},\mathcal{A}}p_{\pi_{\theta}}(\tau)\log\left(\frac{p_{\pi_{\theta}}(\tau)}{q_{\pi^{*}}(\tau)}\right)d\tau$
(2)

where $\tau=(s_{0},s_{1},s_{2},\dots)$ is the trajectory in the target domain
under the policy $\pi_{\theta}(.|s)$ defined as collection states visited
starting from state $s_{0}\sim\rho_{0}$ and making transitions under target
transition model $p^{T}(.|s_{t},a_{t})$. We now define the probability of the
trajectories $p_{\pi_{\theta}}(\tau)$ and $q_{\pi^{*}}(\tau)$:

The term $p_{\pi_{\theta}}(\tau)$ is defined as total likelihood of one-step
deviations to reference states $\\{s^{\prime}_{t+1}\\}_{t=1}^{\infty}$ over
the trajectory $\tau$. The reference states are obtained by re-initializing
the source simulator at every time “$t$” to the states along the trajectory
$\tau$ and making optimal transitions to
$\\{s^{\prime}_{t+1}\\}_{t=1}^{\infty}$ using source optimal policy. The total
trajectory deviation likelihood can be expressed as follows,

$p_{\pi_{\theta}}(\tau)=\rho(s_{0})\prod_{t=0}^{\infty}\pi_{\theta}(a_{t}|s_{t})p^{T}(s^{\prime}_{t+1}|s_{t},\pi_{\theta}(s_{t})).$
(3)

###### Assumption 1

Access to source simulator with restart capability is assumed. That is, source
transition model can be initialized to any desired states $s_{t}$ and under
optimal actions $a^{\prime}_{t}\sim\pi^{*}(.|s_{t})$ the source simulator
provides next transitioned state $s^{\prime}_{t+1}$.

Using Assumption-1, $q_{\pi^{*}}(\tau)$ can be defined as total probability of
piece-wise transitions at every state $s_{t}\in\tau$ under the source optimal
policy $a^{\prime}_{t}\sim\pi^{*}(.|s_{t})$, and the source transitions
$p^{S}(.|s_{t},a^{\prime}_{t})$.

$q_{\pi^{*}}(\tau)=\rho(s_{0})\prod_{t=0}^{\infty}\pi^{*}(a^{\prime}_{t}|s_{t})p^{S}(s^{\prime}_{t+1}|s_{t},\pi^{*}(s_{t})).$
(4)

Unlike the conventional treatment of KL-term in the RL literature, the
transition probabilities in the KL divergence in (18) are treated as
transition likelihoods and the transitioned state $s_{t+1}$ as random
variables. The policy $\pi_{\theta}$ learns to mimic the optimal transitions
by minimizing the KL distance between the transition likelihoods of the target
agent reaching the reference states $\\{s^{\prime}_{i}\\}_{i=1}^{\infty}$ and
the source transitions evalauted at $\\{s_{i}\\}_{i=0}^{\infty}$.

Using the definition of the probabilities of the trajectories under
$\pi_{\theta}$ and $\pi^{*}$ ( Equation-(3) & (4) ) the KL divergence (18)
over the trajectory $\tau$ can be expressed as follows

$\displaystyle\int_{\tau}p_{\pi_{\theta}}(\tau)\log\left(\frac{p_{\pi_{\theta}}(\tau)}{q_{\pi^{*}}(\tau)}\right)d\tau=$
(5)
$\displaystyle\displaystyle\mathop{\mathbb{E}}_{s_{t}\sim\tau}\left(\log\left(\frac{\rho(s_{0})\pi(a_{0}|s_{0})p^{T}(s^{\prime}_{1}|s_{0},\pi_{\theta}(s_{0}))\ldots}{\rho(s_{0})\pi^{*}(a^{\prime}_{0}|s_{0})p^{S}(s^{\prime}_{1}|s_{0},\pi^{*}(s_{0}))\dots}\right)\right)$

###### Assumption 2

We assume that optimal source policy is available, and the probability of
taking optimal actions $\pi^{*}(a^{\prime}_{t}|s_{t})\approx 1$, $\forall
s_{t}\in\mathcal{S}^{T}$.

We make this simplifying Assumption-2 i.e.
$\pi^{*}(a^{\prime}_{t}|s_{t})=1\forall s_{t}$ to derive a surrogate objective
$\bar{\eta}_{{}_{KL}}(\pi_{\theta},\pi^{*})$ which lower bounds the true KL
loss (IV-A1), such that,
$\bar{\eta}_{{}_{KL}}(\pi_{\theta},\pi^{*})\leq\eta_{{}_{KL}}(\pi_{\theta},\pi^{*})$.

The expression for surrogate objective is defined follows,

$\displaystyle\bar{\eta}_{{}_{KL}}(\pi_{\theta},\pi^{*})$ $\displaystyle=$
$\displaystyle\displaystyle\mathop{\mathbb{E}}_{s_{t}\sim\tau}\left(\sum_{t=0}^{\infty}\zeta_{t}\right),$
(6)

where
$\zeta_{t}=\log\left(\frac{\pi_{\theta}(a_{t}|s_{t})p^{T}(s^{\prime}_{t+1}|s_{t},\pi_{\theta}(s_{t}))}{p^{S}(s^{\prime}_{t+1}|s_{t},\pi^{*}(s_{t}))}\right)$
defined as the intrinsic/adaptation reward term.

We need this assumption only for deriving the expression for intrinsic reward.
Assumption-2 allows us to extend the proposed transfer architecture to
deterministic source policy or source policy derived from a human expert;
since a human teacher cannot specify the probability of choosing an action.
However, in the empirical evaluation of the algorithm, the source policy used
as a stochastic optimal policy. The stochastic nature of such a policy adds
natural exploration, and also demonstrates that the presented method is not
restricted in its application by the above assumption.

####  IV-A2 Combined Adaptation and Learning

To achieve “Adaption” and “Learning” simultaneously, we propose to augment the
environment reward $r_{t}\in\mathcal{R}^{T}$ with intrinsic reward
$\zeta_{t}$. By optimizing over mixture of intrinsic and environmental return
simultaneously, we achieve transferred policies that try to both follow source
advice and learn to acquire skills beyond what source can teach. This trade-
off between learning by exploration and learning by adaptation can be realized
as follows:

$\bar{\eta}_{{}_{KL,\beta}}=\displaystyle\mathop{\mathbb{E}}_{s_{t},a_{t}\sim\tau}\left(\sum_{t=0}^{\infty}\gamma^{t}((1-\beta)r_{t}-\beta\zeta_{t})\right).$
(7)

where the term $\beta$ is the mixing coefficient. The inclusion of intrinsic
reward can be seen as adding a directional search to RL exploration. The
source optimal policy and source transitions provide a possible direction of
search for one step optimal reward at every state along the trajectory $\tau$.
For consistency of the reward mixing, the rewards $r_{t},\zeta_{t}$ are
normalized to form the total reward.

The optimal mixing coefficient $\beta$ is learned over episodic data collected
interacting with environment [32]. The details of hierarchical learning of
mixing coefficient is provided in Section-V-A, and its behavior in experiments
is analyzed in Section-VII.

###### Assumption 3

The true transition distribution for source $p^{S}(.|s,a)$ and the target
$p^{T}(.|s,a)$ are unknown. However, we assume both source and target
transition models follow a Gaussian distribution centered at the next
propagated state with fixed variance “$\sigma$”, which is chosen heuristically
s.t

$p^{S}(.|s,a)\sim\mathcal{N}(s^{\prime}_{t+1},\sigma),\hskip
11.38109ptp^{T}(.|s,a)\sim\mathcal{N}(s_{t+1},\sigma)$

Although Assumption-3 might look unrealistic, in reality it is not very
restrictive. We empirically show that for all the experimented MuJoCo
environments, a bootstrapped Gaussian transition assumption is good enough for
ATL agent to transfer the policy efficiently.

Using the Assumptions-2 & 3, we can simplify the individual KL term (intrinsic
reward) as follows

$\zeta_{t}=\log\left(\pi_{\theta}(a_{t}|s_{t})e^{-(s_{t+1}-s^{\prime}_{t+1})^{2}/2\sigma^{2}}\right).$
(8)

The individual terms in the expectation (22), $\zeta_{t}$ represent the
distance between two transition likelihoods of landing in the next state
$s^{\prime}_{t+1}$ starting in $s_{t}$ and under actions
$a_{t},a^{\prime}_{t}$. The target agent is encouraged to take actions that
lead to states which are close in expectation to a reference state provided by
an optimal baseline policy operating on the source model. The intuition is
that by doing so, a possible direction of search for the higher environmental
rewards “$r_{t}$” is provided.

###  IV-B Actor-Critic

Using the definition of value function, the objective in (7) can be written as
$\bar{\eta}_{{}_{KL,\beta}}=V^{\pi_{\theta}}(s)$. The expectation can be
rewritten as sum over states and actions as follows:

$\bar{\eta}_{{}_{KL,\beta}}=\sum_{s\in\mathcal{S}}d^{\pi_{\theta}}(s)\sum_{a\in\mathcal{A}}\pi_{\theta}(a|s)Q^{\pi_{\theta}}(s,a),$
(9)

where $d^{\pi_{\theta}}(s)$ is the state visitation distributions [29].

Considering the case of an off-policy RL, we use a behavioral policy
$\psi(a|s)$ for generating trajectories. This behavioral policy is different
from the policy $\pi_{\theta}(a|s)$ to be optimized. The objective function in
an off-policy RL measures the total return over the behavioral state
visitation distribution, while the mismatch between the training data
distribution and the true policy state distribution is compensated by
importance sampling estimator as follows,

$\bar{\eta}_{{}_{KL,\beta}}=\sum_{s\in\mathcal{S}}d^{\psi}(s)\sum_{a\in\mathcal{A}}\left(\psi(a|s)\frac{\pi_{\theta}(a|s)}{\psi(a|s)}Q^{\psi}(s,a)\right),$
(10)

Using the previously updated policy as the behavioral policy i.e
$\psi(a|s)=\pi_{\theta^{-}}(a|s)$, the objective expression can be rewritten
as,

$\bar{\eta}_{{}_{KL,\beta}}=\displaystyle\mathop{\mathbb{E}}_{s_{t}\sim
d^{\pi_{\theta^{-}}},a_{t}\sim\pi_{\theta^{-}}}\left(\frac{\pi_{\theta}(a|s)}{\pi_{\theta^{-}}(a|s)}\hat{Q}^{\pi_{\theta^{-}}}(s,a)\right).$
(11)

where $\theta^{-}$ is the parameter before update and is known to us. An
estimated state-action value function $\hat{Q}$ replaces the true value
function $Q$ as the true value function is usually unknown. The mixed reward
$r^{\prime}_{t}=(1-\beta)r_{t}-\beta\zeta_{t}$ is used to compute the critic
state-action value function $\hat{Q}$. Further, any policy update algorithm
[29, 31, 30, 33] can be used to update the policy in the direction of
optimizing this objective.

Learning over a mixture of intrinsic and environmental rewards helps in the
directional search for maximum total return. The source transitions provide a
possible direction in which maximum environmental reward can be achieved. This
trade-off between directional search and random exploration is achieved using
the mixed reward.

##  V Optimization of Target Policy

In the previous section, we formulated an Adapt-to-Learn policy transfer
algorithm; we now describe how to derive a practical algorithm from these
theoretical foundations under finite sample counts and arbitrary policy
parameterizations.

Optimizing the following objective (11) we can generate adaptive policy
updates for the target task.

$\displaystyle\pi^{*T}_{\theta}$ $\displaystyle=$
$\displaystyle\arg\max_{\pi_{\theta}\in\Pi,\beta}\left(\bar{\eta}_{{}_{KL,\beta}}\right),$
(12)

If calculating the above expectation is feasible, it is possible to maximize
the objective in (11) and move the policy parameters in the direction of
achieving a higher total discounted return. However, this is not generally the
case since the true expectation is intractable, therefore a common practice is
to use an empirical estimate of the expectation.

The previous section proposed an optimization method to find the adaptive
policy using KL-divergence as an intrinsic reward, enforcing the target
transition model to mimic the source transitions. This section describes how
this objective can be approximated using a Monte Carlo simulation. The
approximate policy update method works by computing an estimate of the
gradient of the return and plugging it into a stochastic gradient ascent
algorithm The gradient estimate over i.i.d data is computed as follows:

$\displaystyle\hat{g}$ $\displaystyle=$ $\displaystyle
P_{Z^{n}}(\nabla_{\theta}\bar{\eta}_{{}_{KL,\beta}})$
$\displaystyle\triangleq$
$\displaystyle\frac{1}{n}\sum_{i=1}^{n}\left(\frac{\pi_{\theta}(a_{i}|s_{i})}{\pi_{\theta^{-}}(a_{i}|s_{i})}\hat{Q}^{\pi_{\theta^{-}}}(s_{i},a_{i})\nabla_{\theta}\log\pi_{\theta}(a_{i}|s_{i})\right).$

where $P_{Z^{n}}$ is empirical distribution over the data
$(Z^{n}:\\{s_{i},a_{i},a^{\prime}_{i}\\}_{i}^{n})$.

Algorithm 1 Adapt-to-Learn Policy Transfer in RL

1:$\pi^{*}(.|s),p^{S}$ $\triangleright$ Source Policy, source simulator

2:Initialize $s_{0}^{T}\in\rho_{0}$. $\triangleright$ Draw initial state from
the given distribution in target task

3:for $i=1\leq K$ do

4: while $s_{i}\neq terminal$ do

5: $a^{\prime}_{i}\sim\pi^{*}(.|{s}_{i})$ $\triangleright$ Generate the
optimal action using Source policy

6: $a_{i}\sim{\pi}_{\theta}(.|{s}_{i})$ $\triangleright$ Generate the action
using the ${\pi}_{\theta}$

7: $s_{i+1}\sim p^{T}(.|{s}_{i},a_{i})$ $\triangleright$ Apply the action
$a_{i}$ at state ${s}_{i}$ in the target task

8: $s^{\prime}_{i+1}\sim p^{S}(.|{s}_{i},a^{\prime}_{i})$ $\triangleright$
Apply the action $a^{\prime}_{i}$ at state ${s}_{i}$ in the source simulator

9:
$\zeta_{t}={\pi}_{\theta}(a_{i}|s_{i})e^{-(s_{i+1}-s^{\prime}_{i+1})^{2}/2\sigma^{2}}$
$\triangleright$ Compute the point-wise KL intrinsic reward

10: $Z_{i}=(\\{s_{i},r_{i},\zeta_{i},a_{i},a^{\prime}_{i}\\})$
$\triangleright$ Incrementally store the trajectory for policy update

11: $P_{Z_{train}^{n}}(\bar{\eta}_{{}_{KL,\beta}})$ $\triangleright$ Form the
Empirical loss

12: $\theta^{\prime}\leftarrow\theta+\alpha
P_{Z_{train}^{n}}\left(\nabla_{\theta}\bar{\eta}_{{}_{KL,\beta}}\right)$
$\triangleright$ Maximize the total return to update the policy

13: Collect test trajectories $Z_{test}^{n}$ using $\pi_{\theta^{\prime}}$

14:
$\beta\leftarrow\beta+\bar{\alpha}P_{Z_{test}^{n}}\left(\nabla_{\beta}\bar{\eta}_{{}_{KL,\beta}}\right)$
$\triangleright$ Maximize the total return for optimal mixing coefficient

###  V-A Sample-Based Estimation of the Gradient

The previous section proposed an optimization method to find the adaptive
policy using KL-divergence as an intrinsic reward, enforcing the target
transition model to mimic the source transitions. This section describes how
this objective can be approximated using a Monte Carlo simulation. The
approximate policy update method works by computing an estimate of the
gradient of the return and plugging it into a stochastic gradient ascent
algorithm The gradient estimate over i.i.d data from the collected
trajectories is computed as follows:

$\displaystyle\hat{g}$ $\displaystyle=$ $\displaystyle
P_{Z^{n}}(\nabla_{\theta}\bar{\eta}_{{}_{KL,\beta}})$
$\displaystyle\triangleq$
$\displaystyle\frac{1}{n}\sum_{i=1}^{n}\left(\frac{\pi_{\theta}(a_{i}|s_{i})}{\pi_{\theta^{-}}(a_{i}|s_{i})}\hat{Q}^{\pi_{\theta^{-}}}(s_{i},a_{i})\nabla_{\theta}\log\pi_{\theta}(a_{i}|s_{i})\right).$

where $P_{Z^{n}}$ is empirical distribution over the data
$(Z^{n}:\\{s_{i},a_{i},a^{\prime}_{i}\\}_{i}^{n})$. The details of empirical
estimate of the gradient $\hat{g}$ of the total return
$\bar{\eta}_{{}_{KL,\beta}}$ is provided in Appendix.

Note that the importance sampling based off-policy update objective renders
our discounted total return and its gradient independent of policy parameter
$\theta$. Hence the gradient of state-action value estimates
$\nabla_{\theta}\hat{Q}^{\pi_{\theta^{-}}}(s,a)$ with respect to $\theta$ is
zero in the above expression for total gradient.

###  V-B Learning Mixing Coefficient from Data

A hierarchical update of the mixing coefficient $\beta$ is carried out over
n-test trajectories, collected using the updated policy network
$\pi_{\theta^{\prime}}(a|s)$. Where $\theta^{\prime}$ is parameter after the
policy update step. We use stochastic gradient ascent to update the mixing
coefficient $\beta$ as follows

$\beta\leftarrow\beta+\bar{\alpha}\hat{g}_{\beta},\hskip 5.69054pts.t\hskip
5.69054pt0\leq\beta\leq 1.$

where $\bar{\alpha}$ is the learning rate and
$\hat{g}_{\beta}=P_{Z^{n}_{test}}(\nabla_{\beta}\bar{\eta}_{{}_{KL,\beta}})$
is the empirical estimate of the gradient of the total return
$\bar{\eta}_{{}_{KL,\beta}}(\pi_{\theta^{\prime}},\pi^{*})$.

A hierarchical update of the mixing coefficient $\beta$ is carried out over
n-test trajectories, collected using the updated policy network
$\pi_{\theta^{\prime}}(a|s)$. The mixing coefficient $\beta$ is learnt by
optimizing the return over trajectory as follows,

$\beta=arg\max_{\beta}(\bar{\eta}_{KL,\beta}(\pi_{\theta^{\prime}},\pi^{*}))$

where $\theta^{\prime}$ is parameter after the policy update step.

$\beta=arg\max_{\beta}\mathop{\mathbb{E}}_{s_{t},a_{t}\sim\tau}\left(p_{\pi_{\theta}}(\tau)\sum_{t=1}^{\infty}\gamma^{t}r^{\prime}_{t}\right)$

We can use gradient ascent to update parameter $\beta$ in direction of
optimizing the reward mixing as follows,

$\beta\leftarrow\beta+\bar{\alpha}\nabla_{\beta}(\bar{\eta}_{KL,\beta}(\pi_{\theta^{\prime}},\pi^{*})).$

Using the definition of mixed reward as
$r^{\prime}_{t}=(1-\beta)r_{t}-\beta\zeta_{t}$, we can simplify the above
gradient as,

$\beta\leftarrow\beta+\bar{\alpha}\mathop{\mathbb{E}}_{s_{t},a_{t}\sim\tau}\left(p_{\pi_{\theta}}(\tau)\sum_{t=1}^{\infty}\gamma^{t}\nabla_{\beta}(r^{\prime}_{t})\right)$
$\beta\leftarrow\beta+\bar{\alpha}\mathop{\mathbb{E}}_{s_{t},a_{t}\sim\tau}\left(\sum_{t=1}^{\infty}\gamma^{t}(r_{t}-\zeta_{t})\right).$

We use stochastic gradient ascent to update the mixing coefficient $\beta$ as
follows

$\beta\leftarrow\beta+\bar{\alpha}\hat{g}_{\beta},\hskip 5.69054pts.t\hskip
5.69054pt0\leq\beta\leq 1.$

where $\bar{\alpha}$ is the learning rate and
$\hat{g}_{\beta}=P_{Z^{n}_{test}}(\nabla_{\beta}\bar{\eta}_{{}_{KL,\beta}})$
is the empirical estimate of the gradient of the total return
$\bar{\eta}_{{}_{KL,\beta}}(\pi_{\theta^{\prime}},\pi^{*})$. The gradient
estimate $\hat{g}_{\beta}$ over data
$(Z^{n}_{test}:\\{s_{i},a_{i},a^{\prime}_{i}\\}_{i}^{T})$ is computed as
follows,

$\hat{g}_{\beta}=\frac{1}{N}\sum_{i=1}^{N}\left(\sum_{t=1}^{\mathcal{H}}\gamma^{t}(r_{t}-\zeta_{t})\right)$

where $\mathcal{H}$ truncated trajectory length from experiments.

As we can see the gradient of objective with respect to mixing coefficient
$\beta$ is an average over difference between environmental and intrinsic
rewards. If $r_{t}-\zeta_{t}\geq 0$ the update will move parameter $\beta$
towards favoring learning through exploration more than learning through
adaptation and visa versa.

As $\beta$ update is a constrained optimization with constraint
$0\leq\beta\leq 1$. We handle this constrained optimization by modelling as
output of sigmoidal network $\beta=\sigma(\phi)$ parameterized by parameters
$\phi$. The constrained optimization can be equivalently written as optimizing
w.r.to $\phi$ as follows

$\phi\leftarrow\phi+\bar{\alpha}\hat{g}_{\beta}\nabla_{\phi}(\beta),\hskip
5.69054ptwhere\hskip 5.69054pt\beta=\sigma(\phi)$

The details of policy and $\beta$ update is shown in Figure-1.

Figure 1: Total Policy Update scheme to learn Target task. The Hyper-parameter $\beta$ are updated over test trajectories generated using the updated policy parameters. Env | Property | Source | Target | %Change
---|---|---|---|---
Hopper | Floor Friction | 1.0 | 2.0 | +100%
HalfCheetah | gravity | -9.81 | -15 | +52%
| Total Mass | 14 | 35 | +150%
| Back-Foot Damping | 3.0 | 1.5 | -100%
| Floor Friction | 0.4 | 0.1 | -75%
Walker2d | Density | 1000 | 1500 | +50%
| Right-Foot Friction | 0.9 | 0.45 | -50%
| Left-Foot Friction | 1.9 | 1.0 | -47.37%
TABLE I: Transition Model and environment properties for Source and Target task and % change | Hopper | Walker2d | HalfCheetah
---|---|---|---
State Space | 12 | 18 | 17
Action Space | 3 | 6 | 6
Number of layers | 3 | 3 | 3
Layer Activation | tanh | tanh | tanh
Network Parameters | 10530 | 28320 | 26250
Discount | 0.995 | 0.995 | 0.995
Learning rate ($\alpha$) | $1.5\times 10^{-5}$ | $8.7\times 10^{-6}$ | $9\times 10^{-6}$
$\beta$ Initial Value | $0.5$ | $0.5$ | $0.5$
$\beta$-Learning rate ($\bar{\alpha}$) | $0.1$ | $0.1$ | $0.1$
Batch size | 20 | 20 | 5
Policy Iter | 3000 | 5000 | 1500
TABLE II: Policy Network details and Network learning parameter details

##  VI Sample Complexity and $\epsilon$-Optimality

In this section we will provide without proofs the sample complexity and
$\epsilon$-optimality results of the proposed policy transfer method. The
proofs of the following theorem can be found in.

###  VI-A Lower bounds on Sample Complexity

Although there is some empirical evidence that transfer can improve
performance in subsequent reinforcement-learning tasks, there are not many
theoretical guarantees. Since many of the existing transfer algorithms
approach the problem of transfer as a method of providing good initialization
to target task RL, we can expect the sample complexity of those algorithms to
still be a function of the cardinality of state-action pairs
$|N|=|\mathcal{S}|\times|\mathcal{A}|$. On the other hand, in a supervised
learning setting, the theoretical guarantees of the most algorithms have no
dependency on size (or dimensionality) of the input domain (which is analogous
to $|N|$ in RL). Having formulated a policy transfer algorithm using labeled
reference trajectories derived from optimal source policy, we construct
supervised learning like PAC property of the proposed method. For deriving,
the lower bound on the sample complexity of the proposed transfer problem, we
consider only the adaptation part of the learning i.e., the case when
$\beta=1$ in Eq-(7). This is because, in ATL, adaptive learning is akin to
supervised learning, since the source reference trajectories provide the
target states

Suppose we are given the learning problem specified with training set
$Z^{n}=(Z1,\ldots Z_{n})$ where each $Z_{i}=(\\{s_{i},a_{i}\\})_{i=0}^{n}$ are
independently drawn trajectories according to some distribution $P$. Given the
data $Z^{n}$ we can compute the empirical return
$P_{Z^{n}}(\bar{\eta}_{{}_{KL,\beta}})$ for every $\pi_{\theta}\in\Pi$, we
will show that the following holds:

$\|P_{Z^{n}}(\bar{\eta}_{{}_{KL,\beta}})-P(\bar{\eta}_{{}_{KL,\beta}})\|\leq\epsilon.$
(15)

with probability at least $1-\delta$, for some very small $\delta$ s.t
$0\leq\delta\leq 1$. We can claim that the empirical return for all
$\pi_{\theta}$ is a sufficiently accurate estimate of the true return
function. Thus a reasonable learning strategy is to find a
$\pi_{\theta}\in\Pi$ that would minimize empirical estimate of the objective
Eq-(7).

###### Theorem VI.1

If the induced class of the policy $\pi_{\theta}$:$\mathcal{L}_{\Pi}$ has
uniform convergence property in empirical mean; then the empirical risk
minimization is PAC. s.t

$P^{n}(P(\bar{\eta}_{{}_{KL,\hat{\pi}^{*}}})-P(\bar{\eta}_{{}_{KL,{\pi^{*}}}})\geq\epsilon)\leq\delta.$
(16)

and number of trajectory samples required can be lower bounded as

$n(\epsilon,\delta)\geq\frac{2}{\epsilon^{2}(1-\gamma)^{2}}\log\left(\frac{2|\Pi|}{\delta}\right).$
(17)

Please refer Appendix for the proof of the above theorem.

###  VI-B $\epsilon$-Optimality result under Adaptive Transfer-Learning

Consider MDP $M^{*}$ and $\hat{M}$ which differ in their transition models.
For the sake of analysis, let $M^{*}$ be the MDP with ideal transition model,
such that target follows source transition $p^{*}$ precisely. Let $\hat{p}$ be
the transition model achieved using the estimated policy learned over data
interacting with the target model and the associated MDP be denoted as
$\hat{M}$. We analyze the $\epsilon$-optimality of return under adapted source
optimal policy through ATL.

###### Definition VI.2

Given the value function $V^{*}=V^{\pi^{*}}$ and model $M_{1}$ and $M_{2}$,
which only differ in the corresponding transition models $p_{1}$ and $p_{2}$.
Define $\forall s,a\in\mathcal{S}\times\mathcal{A}$

$d_{M_{1},M_{2}}^{V^{*}}=\sup_{s,a\in\mathcal{S}\times\mathcal{A}}\left|\displaystyle\mathop{\mathbb{E}}_{s^{\prime}\sim
P_{1}(s,a)}[V^{*}(s^{\prime})]-\displaystyle\mathop{\mathbb{E}}_{s^{\prime}\sim{P}_{2}(s,a)}[V^{*}(s^{\prime})]\right|.$

###### Lemma VI.3

Given $M^{*}$, $\hat{M}$ and value function $V^{\pi^{*}}_{M^{*}}$,
$V^{\pi^{*}}_{\hat{M}}$ the following bound holds
$\left\|V^{\pi^{*}}_{M^{*}}-V^{\pi^{*}}_{\hat{M}}\right\|_{\infty}\leq\frac{\gamma\epsilon}{(1-\gamma)^{2}}$

where $\max_{s,a}\|\hat{p}(.|s,a)-p^{*}(.|s,a)\|\leq\epsilon$ and $\hat{p}$
and $p^{*}$ are transition of MDP $\hat{M},M^{*}$ respectively.

The proof of this lemma is based on the simulation lemma [34]. For the proof
of above lemma refer Appendix. Similar results for RL with imperfect models
were reported by [35].

##  VII Policy transfer in simulated robotic locomotion tasks

(a)

(b)

(c)

Figure 2: Learning curves for locomotion tasks, averaged across five runs of
each algorithm: Adapt-to-Learn(Ours), Randomly Initialized RL(PPO), Warm-
Started PPO using source policy parameters and Best case imitation learning
using Source policy directly on Target Task without any adaptation.

(a)

(b)

(c)

Figure 3: Trajectory KL divergence Total Intrinsic Return $\left(-\sum
e^{\zeta_{t}}\right)$ averaged across five runs. Figure 4: Mixing coefficient
$\beta$ over one run for HalfCheetah, Hopper and Walker2d Environment

(a)

(b)

(c)

Figure 5: (a) Learning curves for Crippled-Fat HalfCheetah Env, averaged
across five runs: Adapt-to-Learn (ATL (Ours)), Randomly Initialized RL (PPO),
Warm-Started PPO using source policy parameters. (b) Avg Intrinsic reward for
ATL. (c) Mixing coefficient $\beta$.

To evaluate Adapt-to-Learn Policy Transfer in reinforcement learning, we
design our experiments using sets of tasks based on the continuous control
environments in MuJoCo simulator [36]. Our experimental results demonstrate
that ATL can adapt to significant changes in transition dynamics. We perturb
the parameters of the simulated target models for the policy transfer
experiments (see Table-III for original and perturbed parameters of the target
model). To create a challenging training environment, we changed the
parameters of the transition model such that the optimal source policy alone
without further learning cannot produce any stable results (see source policy
performance in Figure-2 & 5). We compare our results against two baselines:
(a) Initialized RL (Initialized PPO) (Warm-Start-RL [15]) (b) Stand-alone
reinforcement policy learning (PPO) [31].

We experiment with the ATL algorithm on Hopper, Walker2d, and HalfCheetah
Environments. The states of the robots are their generalized positions and
velocities, and the actions are joint torques. High dimensionality, non-smooth
dynamics due to contact discontinuity, and under-actuated dynamics make these
tasks very challenging. We use deep neural networks to represent the source
and target policy, the details of which are in the Table-IV.

Learning curves showing the total reward averaged across five runs of each
algorithm are provided in Figure-2 & 5. Adapt-to Learn policy transfer solved
all the tasks, yielding quicker learning compared to other baseline methods.
Figure-4 provide the beta vs episode plot for HalfCheetah, Hopper and Walker2d
environment. Recall that the extent of mixing of adaptation and learning from
exploration is driven by the mixing coefficient $\beta$. Figure-4 shows that
for all these environments with parameter perturbation, the ATL agent found it
valuable to follow source baseline and therefore favored adaptation over
exploration in search for optimal policy. In cases where source policy is not
good enough to learn the target task the proposed algorithm seamlessly
switches to initialized RL by favouring the exploration, as demonstrated in
Crippled Half Cheetah experiment:

To further test the algorithm’s capability to adapt, we evaluate its
performance in the crippled-Halfcheetah environment (Figure-5). With a loss of
actuation in the front leg, adaptation using source policy might not aid
learning, and therefore learning from exploration is necessary. We observe the
mixing coefficient $\beta$ evolves closer to zero, indicating that ATL is
learning to rely more on exploration rather than following the source policy.
This results in a slight performance drop initially, but overall, ATL
outperforms the baselines. This shows that in the scenarios where adaptation
using source policy does not solve the task efficiently, ATL learns to
seamlessly switch to exploring using environmental rewards.

##  VIII Conclusion

We introduced a new transfer learning technique for RL: Adapt-to-Learn (ATL),
that utilizes combined adaptation and learning for policy transfer from source
to target tasks. We demonstrated on nonlinear and continuous robotic
locomotion tasks that our method leads to a significant reduction in sample
complexity over the prevalent warm-start based approaches. We demonstrated
that ATL seamlessly learns to rely on environment rewards when learning from
Source does not provide a direct benefit, such as in the Crippled Cheetah
environment. There are many exciting directions for future work. A network of
policies that can generalize across multiple tasks could be learned based on
each new adapted policies. How to train this end-to-end is an important
question. The ability of Adapt-to-Learn to handle significant perturbations to
the transition model indicates that it should naturally extend to sim-to-real
transfer [18, 19, 20], and cross-domain transfer [27]. Another exciting
direction is to extend the work to other combinatorial domains (e.g.,
multiplayer games). We expect, therefore, follow on work will find other
exciting ways of exploiting such adaptation in RL, especially for robotics and
real-world applications.

## References

  * [1] Xue Bin Peng, Glen Berseth, and Michiel Van de Panne.  Terrain-adaptive locomotion skills using deep reinforcement learning.  ACM Transactions on Graphics (TOG), 35(4):81, 2016.
  * [2] Xue Bin Peng, Glen Berseth, KangKang Yin, and Michiel Van De Panne.  Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning.  ACM Transactions on Graphics (TOG), 36(4):41, 2017.
  * [3] Libin Liu and Jessica Hodgins.  Learning to schedule control fragments for physics-based characters using deep q-learning.  ACM Transactions on Graphics (TOG), 36(3):29, 2017.
  * [4] Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, Ali Eslami, Martin Riedmiller, et al.  Emergence of locomotion behaviours in rich environments.  arXiv preprint arXiv:1707.02286, 2017.
  * [5] John W Krakauer and Pietro Mazzoni.  Human sensorimotor learning: adaptation, skill, and beyond.  Current opinion in neurobiology, 21(4):636–644, 2011.
  * [6] Mitch J Fryling, Cristin Johnston, and Linda J Hayes.  Understanding observational learning: An interbehavioral approach.  The Analysis of Verbal Behavior, 27(1):191–203, 2011.
  * [7] Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAI Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba.  One-shot imitation learning.  In Advances in neural information processing systems, pages 1087–1098, 2017.
  * [8] Henry Zhu, Abhishek Gupta, Aravind Rajeswaran, Sergey Levine, and Vikash Kumar.  Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost.  arXiv preprint arXiv:1810.06045, 2018.
  * [9] Karl J Åström and Björn Wittenmark.  Adaptive control.  Courier Corporation, 2013.
  * [10] Girish Chowdhary, Tongbin Wu, Mark Cutler, and Jonathan P How.  Rapid transfer of controllers between uavs using learning-based adaptive control.  In Robotics and Automation (ICRA), 2013 IEEE International Conference on, pages 5409–5416. IEEE, 2013.
  * [11] Sergey Levine and Vladlen Koltun.  Guided policy search.  In International Conference on Machine Learning, pages 1–9, 2013\.
  * [12] Matthew E Taylor and Peter Stone.  Transfer learning for reinforcement learning domains: A survey.  Journal of Machine Learning Research, 10(Jul):1633–1685, 2009.
  * [13] Matthew E Taylor, Peter Stone, and Yaxin Liu.  Value functions for rl-based behavior transfer: A comparative study.  In Proceedings of the National Conference on Artificial Intelligence, volume 20, page 880. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 2005.
  * [14] Haitham B Ammar, Karl Tuyls, Matthew E Taylor, Kurt Driessens, and Gerhard Weiss.  Reinforcement learning transfer via sparse coding.  In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems-Volume 1, pages 383–390. International Foundation for Autonomous Agents and Multiagent Systems, 2012.
  * [15] Haitham Bou Ammar, Eric Eaton, Paul Ruvolo, and Matthew E Taylor.  Unsupervised cross-domain transfer in policy gradient reinforcement learning via manifold alignment.  In Proc. of AAAI, 2015.
  * [16] Bikramjit Banerjee and Peter Stone.  General game learning using knowledge transfer.  In IJCAI, pages 672–677, 2007.
  * [17] Jan Peters and Stefan Schaal.  Policy gradient methods for robotics.  In Intelligent Robots and Systems, 2006 IEEE/RSJ International Conference on, pages 2219–2225. IEEE, 2006.
  * [18] Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel.  Sim-to-real transfer of robotic control with dynamics randomization.  arXiv preprint arXiv:1710.06537, 2017.
  * [19] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell.  A reduction of imitation learning and structured prediction to no-regret online learning.  In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627–635, 2011.
  * [20] Mengyuan Yan, Iuri Frosio, Stephen Tyree, and Jan Kautz.  Sim-to-real transfer of accurate grasping with eye-in-hand observations and continuous control.  arXiv preprint arXiv:1712.03303, 2017.
  * [21] Sergey Levine, Nolan Wagener, and Pieter Abbeel.  Learning contact-rich manipulation skills with guided policy search.  In Robotics and Automation (ICRA), 2015 IEEE International Conference on, pages 156–163. IEEE, 2015.
  * [22] Sergey Levine, Peter Pastor, Alex Krizhevsky, and Deirdre Quillen.  Learning hand-eye coordination for robotic grasping with large-scale data collection.  In International Symposium on Experimental Robotics, pages 173–184. Springer, 2016.
  * [23] Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine.  Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning.  In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7559–7566. IEEE, 2018.
  * [24] Shreyansh Daftry, J Andrew Bagnell, and Martial Hebert.  Learning transferable policies for monocular reactive mav control.  In International Symposium on Experimental Robotics, pages 3–11. Springer, 2016.
  * [25] Paul Christiano, Zain Shah, Igor Mordatch, Jonas Schneider, Trevor Blackwell, Joshua Tobin, Pieter Abbeel, and Wojciech Zaremba.  Transfer from simulation to real world through learning deep inverse dynamics model.  arXiv preprint arXiv:1610.03518, 2016.
  * [26] Markus Wulfmeier, Ingmar Posner, and Pieter Abbeel.  Mutual alignment transfer learning.  arXiv preprint arXiv:1707.07907, 2017.
  * [27] Chang Wang and Sridhar Mahadevan.  Manifold alignment without correspondence.  In IJCAI, volume 2, page 3, 2009.
  * [28] Girish Joshi and Girish Chowdhary.  Cross-domain transfer in reinforcement learning using target apprentice.  In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7525–7532. IEEE, 2018.
  * [29] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour.  Policy gradient methods for reinforcement learning with function approximation.  In Advances in neural information processing systems, pages 1057–1063, 2000.
  * [30] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz.  Trust region policy optimization.  In International Conference on Machine Learning, pages 1889–1897, 2015.
  * [31] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [32] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li.  Meta-sgd: Learning to learn quickly for few-shot learning, 2017.
  * [33] Jan Peters and Stefan Schaal.  Natural actor-critic.  Neurocomputing, 71(7):1180–1190, 2008.
  * [34] Michael Kearns and Satinder Singh.  Near-optimal reinforcement learning in polynomial time.  Machine Learning, 49(2-3):209–232, 2002.
  * [35] Nan Jiang.  Pac reinforcement learning with an imperfect model.  In Proc. of AAAI, 2018.
  * [36] Emanuel Todorov, Tom Erez, and Yuval Tassa.  Mujoco: A physics engine for model-based control.  In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033. IEEE, 2012.

###  -A Related work

Deep Reinforcement Learning (D-RL) has recently enabled agents to learn
policies for complex robotic tasks in simulation [1, 2, 3, 4]. However, D-RL
has been plagued by the curse of sample complexity. Therefore, the
capabilities demonstrated in the simulated environment are hard to replicate
in the real world. This learning inefficiency of D-RL has led to significant
work in the field of Transfer Learning (TL) [12]. A significant body of
literature on transfer in RL is focused on initialized RL in the target domain
using learned source policy; known as jump-start/warm-start methods [13, 14,
15]. Some examples of these transfer architectures include transfer between
similar tasks [16], transfer from human demonstrations [17] and transfer from
simulation to real [18, 19, 20]. Efforts have also been made in exploring
accelerated learning directly on real robots, through Guided Policy Search
(GPS) [21] and parallelizing the training across multiple agents using meta-
learning [22, 23, 8]. Sim-to-Real transfers have been widely adopted in the
recent works and can be viewed as a subset of same domain transfer problems.
Daftry et al. [24] demonstrated the policy transfer for control of aerial
vehicles across different vehicle models and environments. Policy transfer
from simulation to real using an inverse dynamics model estimated interacting
with the real robot is presented by [25]. The agents trained to achieve robust
policies across various environments through learning over an adversarial loss
is presented in [26].

###  -B Derivation of Behavioral Adaptation KL Divergence Intrinsic Reward

(a)

(b)

Figure 6: (a) Target Trajectory under policy $\pi_{\theta}$ and local
trajectory deviation under source optimal policy $\pi^{*}$ and source
transition $p^{S}$ (b) One step Target and Source transition(simulated)
starting from state $s_{t}$. Transition likelihood
$p^{T}(s^{\prime}_{t+1}|s_{t},a_{t})$ is the probability of landing in state
$s^{\prime}_{t+1}$ starting from state $s_{t}$ and using action target $a_{t}$
under target transition model.

The adaptation objective can be formalized as minimizing the average KL-
divergence [30] between source and target transition trajectories as follows,

$\displaystyle\eta_{{}_{KL}}(\pi_{\theta},\pi^{*})$ $\displaystyle=$
$\displaystyle D_{KL}(p_{\pi_{\theta}}(\tau)\|q_{\pi^{*}}(\tau)),$
$\displaystyle\eta_{{}_{KL}}(\pi_{\theta},\pi^{*})$ $\displaystyle=$
$\displaystyle\int_{\mathcal{S},\mathcal{A}}p_{\pi_{\theta}}(\tau)\log\left(\frac{p_{\pi_{\theta}}(\tau)}{q_{\pi^{*}}(\tau)}\right)d\tau$
(18)

where $\tau=(s_{0},s_{1},s_{2},\dots)$ is the trajectory in the target domain
under the policy $\pi_{\theta}(.|s)$ defined as collection states visited
starting from state $s_{0}\sim\rho_{0}$ and making transitions under target
transition model $p^{T}(.|s_{t},a_{t})$.

In the above defined KL divergence term the random variable is the trajectory
$\tau=(s_{0},s_{1},s_{2},\dots)$. We explain the flow of the algorithm in
Figure 6. The algorithm starts with some random state $s_{0}\sim\rho_{0}$ and
using source optimal policy $\pi_{\theta}(.|s_{0})$ and target transition
model $p^{T}(.|s_{0},\pi_{\theta}(s_{0}))$, make a transition to state $s_{1}$
(Red arrow, Figure-6a). The source simulator is now initialized to state
$s_{0}$ and using source optimal policy $\pi^{*}(.|s_{0})$ we make optimal
transition under source transition model $p^{S}(.|s_{0},\pi^{*}(s_{0}))$ to
state $s^{\prime}_{1}$ (Blue arrow, this is a simulated step using source
simulator Figure-6a).

The likelihood of landing in the reference state $s^{\prime}_{1}$( obtained
from optimal source transition) is now evaluated, under target transition
model and target policy. We call this likelihood as trajectory deviation
likelihood. The trajectory deviation likelihood can be expressed as
$p^{T}(s^{\prime}_{1}|s_{0},\pi_{\theta}(s_{0}))$. Note that we obtain this
likelihood by evaluation the target transition probability at state
$s^{\prime}_{1}$. For the next step of learning the source model is
reinitialized to the target transitioned state $s_{1}$ (dotted arrow in
Figure-6b and the process is repeated as above.

One step KL divergence between transition probabilities or one-step Intrinsic
reward can be written as

$\zeta_{t}=\pi_{\theta}(a_{0}|s_{0})p^{T}(s^{\prime}_{1}|s_{0},\pi_{\theta}(s_{0}))\left(\frac{\pi_{\theta}(a_{0}|s_{0})p^{T}(s^{\prime}_{1}|s_{0},\pi_{\theta}(s_{0}))}{\pi^{*}a(a^{\prime}_{0}|s_{0})p^{S}(s^{\prime}_{1}|s_{0},\pi^{*}(s_{0}))}\right).$
(19)

Note that above expression is a proper definition of KL divergence where the
random variable for two probabilities $p_{\pi_{\theta}}(.)$ and
$q_{\pi^{*}}(.)$ is the trajectory $\tau=(s_{0},s_{1},s_{2},\dots)$. The KL
divergence expression also satisfies the absolute continuity condition owing
to fact that the state space for source and target are same.

Computing the KL divergence over the entire trajectory, we can derive the
expression for total behavioral adaptation intrinsic return as follows

$\displaystyle\int_{\tau}p_{\pi_{\theta}}(\tau)\log\left(\frac{p_{\pi_{\theta}}(\tau)}{q_{\pi^{*}}(\tau)}\right)d\tau$
$\displaystyle=$
$\displaystyle\displaystyle\mathop{\mathbb{E}}_{s_{t}\sim\tau}\left(\log\left(\frac{\rho(s_{0})\pi(a_{0}|s_{0})p^{T}(s^{\prime}_{1}|s_{0},\pi_{\theta}(s_{0}))\ldots}{\rho(s_{0})\pi^{*}(a^{\prime}_{0}|s_{0})p^{S}(s^{\prime}_{1}|s_{0},\pi^{*}(s_{0}))\dots}\right)\right).$

##  Appendix A Total return gradient with respect to policy parameters

The total return which we aim to maximize in adapting the source policy to
target is the mixture of environmental rewards and Intrinsic KL divergence
reward as follows,

$\bar{\eta}_{{}_{KL,\beta}}(\pi_{\theta},\pi^{*})=\displaystyle\mathop{\mathbb{E}}_{s_{t},a_{t}\sim\tau}\left(p_{\pi_{\theta}}(\tau)\sum_{t=0}^{\mathcal{H}}r^{\prime}_{t}\right),$
(21)

Taking the expectation over policy and transition distribution we can write
the above expression

$\bar{\eta}_{{}_{KL,\beta}}(\pi_{\theta},\pi^{*})=\displaystyle\mathop{\mathbb{E}}_{s_{t}\sim
p^{T},a_{t}\sim\pi_{\theta}}\left(\sum_{t=0}^{\infty}\gamma^{t}r^{\prime}_{t}\right)=V^{\pi_{\theta}}(s).$
(22)

Using the definition of the state-value function, the above objective function
can be re-written as

$\bar{\eta}_{{}_{KL,\beta}}(\pi_{\theta},\pi^{*})=\sum_{a}\left(\pi_{\theta}(a|s)Q^{\pi_{\theta}}(s,a)\right).$
(23)

The adaptive policy update methods work by computing an estimator of the
gradient of the return and plugging it into a stochastic gradient ascent
algorithm

$\pi^{*T}_{\theta}=arg\max_{\pi_{\theta}\in\Pi}P_{Z^{n}}(\bar{\eta}_{{}_{KL,\beta}}).$
(24) $\theta=\theta+\alpha\hat{g},$

where $\alpha$ is the learning rate and $\hat{g}$ is the empirical estimate of
the gradient of the total discounted return $\eta_{{}_{KL}}$.

Taking the derivative of the total return term

$\displaystyle\nabla_{\theta}(\bar{\eta}_{{}_{KL,\beta}})$ $\displaystyle=$
$\displaystyle\nabla_{\theta}V^{\pi_{\theta}}(s)=\nabla_{\theta}\left(\sum_{a}\left(\pi_{\theta}(a|s)Q^{\pi_{\theta}}(s,a)\right)\right),$
$\displaystyle\nabla_{\theta}V^{\pi_{\theta}}(s)$ $\displaystyle=$
$\displaystyle\sum_{a}\nabla_{\theta}\pi_{\theta}(a|s)Q^{\pi_{\theta}}(s,a)+\sum_{a}\pi_{\theta}(a|s)\nabla_{\theta}Q^{\pi_{\theta}}(s,a).$

Using the following definition in above expression,

$Q^{\pi_{\theta}}(s_{i},a)=p^{T}(s_{i+1},|s_{i},a)(r+\gamma
V^{\pi}_{\theta}(s_{i+1})).$

We can rewrite the gradient to total return over policy $\pi_{\theta}$ as,

$\displaystyle\nabla_{\theta}V^{\pi_{\theta}}(s_{0})$ $\displaystyle=$
$\displaystyle\sum_{a}\nabla_{\theta}\pi_{\theta}(a|s_{0})Q^{\pi_{\theta}}(s_{0},a)$
$\displaystyle+$
$\displaystyle\sum_{s_{1}}p^{T}(s_{1},|s_{0},a)\sum_{a}\pi_{\theta}(a|s_{0})\nabla_{\theta}(r_{0}+\gamma
V^{\pi}_{\theta}(s_{1})).$

As the reward $r_{t}$ is independent of $\theta$, we can simplify the above
expression and can be re-written as

$\displaystyle\nabla_{\theta}V^{\pi_{\theta}}(s_{0})$ $\displaystyle=$
$\displaystyle\sum_{a}\nabla_{\theta}\pi_{\theta}(a|s_{0})Q^{\pi_{\theta}}(s_{0},a)$
$\displaystyle+$ $\displaystyle\sum_{s_{1}}\gamma
p^{T}(s_{1},|s_{0},a)\sum_{a}\pi_{\theta}(a|s_{0})\nabla_{\theta}V^{\pi}_{\theta}(s_{1}).$

As we can see the above expression has a recursive property involving term
$\nabla_{\theta}V^{\pi_{\theta}}(s)$. Using the following definition of a
discounted state visitation distribution $d^{\pi_{\theta}}$

$\displaystyle d^{\pi_{\theta}}(s_{0})$ $\displaystyle=$
$\displaystyle\rho(s_{0})+\gamma\sum_{a}\pi(a|s_{0})\sum_{s_{1}}p^{T}(s_{1}|s_{0},a)$
(28)
$\displaystyle+\gamma^{2}\sum_{a}\pi(a|s_{1})\sum_{s_{2}}p^{T}(s_{2}|s_{1},a)\dots$

we can write the gradient of transfer objective as follows,

$\nabla_{\theta}(\eta_{{}_{KL,\beta}})=\sum_{s\in\mathcal{S}}d^{\pi_{\theta}}(s)\sum_{a\in\mathcal{A}}\nabla_{\theta}\pi_{\theta}(a|s)Q^{\pi_{\theta}}(s,a)$
(29)

Considering an off-policy RL update, where $\pi_{\theta^{-}}$ is used for
collecting trajectories over which the state-value function is estimated, we
can rewrite the above gradient for offline update as follows,

Multiplying and dividing Eq-29 by $\pi_{\theta^{-}}(a|s)$ and
$\pi_{\theta}(a|s)$ we form a gradient estimate for offline update,

$=\sum_{s\in\mathcal{S}}d^{\pi_{\theta^{-}}}(s)\sum_{a\in\mathcal{A}}\pi_{\theta^{-}}(a|s)\frac{\pi_{\theta}(a|s)}{\pi_{\theta^{-}}(a|s)}\frac{\nabla_{\theta}\pi_{\theta}(a|s)}{\pi_{\theta}(a|s)}Q^{\pi_{\theta^{-}}}(s,a)$
(30)

where the ratio $\left(\frac{\pi_{\theta}(a|s)}{\pi_{\theta^{-}}(a|s)}\right)$
is importance sampling term, and using the following identity the above
expression can be rewritten as

$\displaystyle\frac{\nabla_{\theta}\pi_{\theta}(a|s)}{\pi_{\theta}(a|s)}=\nabla_{\theta}\log\pi_{\theta}(a|s)$
$\displaystyle\Rightarrow$
$\displaystyle\displaystyle\mathop{\mathbb{E}}_{s_{t}\sim
d^{\pi_{\theta^{-}}},a_{t}\sim\pi_{\theta^{-}}}\left(\frac{\pi_{\theta}(a|s)}{\pi_{\theta^{-}}(a|s)}Q^{\pi_{\theta^{-}}}(s,a)\nabla_{\theta}\log\pi_{\theta}(a|s)\right)$

###  A-A Theoretical bounds on sample complexity

Although there is some empirical evidence that transfer can improve
performance in subsequent reinforcement-learning tasks, there are not many
theoretical guarantees in the literature. Since many of the existing transfer
algorithms approach the problem of transfer as a method of providing good
initialization to target task RL, we can expect the sample complexity of those
algorithms to still be a function of the cardinality of state-action pairs
$|N|=|\mathcal{S}|\times|\mathcal{A}|$. On the other hand, in a supervised
learning setting, the theoretical guarantees of the most algorithms have no
dependency on size (or dimensionality) of the input domain (which is analogous
to $|N|$ in RL). Having formulated a policy transfer algorithm using labeled
reference trajectories derived from optimal source policy, we construct
supervised learning like PAC property of the proposed method. For deriving,
the lower bound on the sample complexity of the proposed transfer problem, we
consider only the adaptation part of the learning i.e., the case when
$\beta=1$. This is because, in ATL, adaptive learning is akin to supervised
learning, since the source reference trajectories provide the target states
given every $(s_{t},a_{t})$ pair.

Suppose we are given the learning problem specified with training set
$Z^{n}=(Z1,\ldots Z_{n})$ where each $Z_{i}=(\\{s_{i},a_{i}\\})_{i=0}^{n}$ are
independently drawn trajectories according to some distribution $P$. Given the
data $Z^{n}$ we can compute the empirical return
$P_{Z^{n}}(\bar{\eta}_{{}_{KL,\beta}})$ for every $\pi_{\theta}\in\Pi$, we
will show that the following holds:

$\|P_{Z^{n}}(\bar{\eta}_{{}_{KL,\beta}})-P(\bar{\eta}_{{}_{KL,\beta}})\|\leq\epsilon.$
(32)

with probability at least $1-\delta$, for some very small $\delta$ s.t
$0\leq\delta\leq 1$. We can claim that the empirical return for all
$\pi_{\theta}$ is a sufficiently accurate estimate of the true return
function. Thus a reasonable learning strategy is to find a
$\pi_{\theta}\in\Pi$ that would minimize empirical estimate of the objective

$\displaystyle\pi^{*T}_{\theta}$ $\displaystyle=$
$\displaystyle\arg\max_{\pi_{\theta}\in\Pi,\beta}\left(\bar{\eta}_{{}_{KL,\beta}}\right),$
(33)

###### Theorem A.1

If the induced class $\mathcal{L}_{\Pi}$ has uniform convergence in empirical
mean property then empirical risk minimization is PAC.

For notation simplicity we drop the superscript $T$ (for Target domain) and
subscript $\theta$ (policy parameters) in further analysis. Unless stated we
will using following simplifications $\hat{\pi}^{*}=\hat{\pi}^{*T}_{\theta}$
and ${\pi}^{*}={\pi}^{*T}_{\theta}$

###### Proof:

Fix $\epsilon,\delta>0$ we will show that for sufficiently large $n\geq
n(\epsilon,\delta)$

$P^{n}(P(\bar{\eta}_{{}_{KL,\hat{\pi}^{*}}})-P(\bar{\eta}_{{}_{KL,{\pi^{*}}}})\geq\epsilon)\leq\delta$
(34)

Let $\pi^{*}\in\Pi$ be the minimizer of true return $P(\bar{\eta}_{{}_{KL}})$,
further adding and subtracting the terms
$P_{Z^{n}}(\bar{\eta}_{{}_{KL,\hat{\pi}^{*}}})$ and
$P_{Z^{n}}(\bar{\eta}_{{}_{KL,{\pi^{*}}}})$ we can write

$\displaystyle
P(\bar{\eta}_{{}_{KL,\hat{\pi}^{*}}})-P(\bar{\eta}_{{}_{KL,{\pi^{*}}}})=$
$\displaystyle
P(\bar{\eta}_{{}_{KL,\hat{\pi}^{*}}})-P_{Z^{n}}(\bar{\eta}_{{}_{KL,\hat{\pi}^{*}}})$
$\displaystyle+P_{Z^{n}}(\bar{\eta}_{{}_{KL,\hat{\pi}^{*}}})-P_{Z^{n}}(\bar{\eta}_{{}_{KL,{\pi^{*}}}})$
$\displaystyle+P_{Z^{n}}(\bar{\eta}_{{}_{KL,{\pi^{*}}}})-P(\bar{\eta}_{{}_{KL,{\pi^{*}}}})$

To simplify, the three terms in the above expression can be handled
individually as follows,

  1. 1.

$P(\bar{\eta}_{{}_{KL,\hat{\pi}^{*}}})-P_{Z^{n}}(\bar{\eta}_{{}_{KL,\hat{\pi}^{*}}})$

  2. 2.

$P_{Z^{n}}(\bar{\eta}_{{}_{KL,\hat{\pi}^{*}}})-P_{Z^{n}}(\bar{\eta}_{{}_{KL,{\pi^{*}}}})$

  3. 3.

$P_{Z^{n}}(\bar{\eta}_{{}_{KL,{\pi^{*}}}})-P(\bar{\eta}_{{}_{KL,{\pi^{*}}}})$

Lets consider the term
$P_{Z^{n}}(\bar{\eta}_{{}_{KL,\hat{\pi}^{*}}})-P_{Z^{n}}(\bar{\eta}_{{}_{KL,{\pi^{*}}}})$
in the above expression is always negative semi-definite, since
$\hat{\pi}^{*}$ is a maximizer wrto $P_{Z^{n}}(\bar{\eta}_{{}_{KL}})$, hence
$P_{Z^{n}}(\bar{\eta}_{{}_{KL,\hat{\pi}^{*}}})\leq
P_{Z^{n}}(\bar{\eta}_{{}_{KL,{\pi^{*}}}})$ always, i.e

$P_{Z^{n}}(\bar{\eta}_{{}_{KL,\hat{\pi}^{*}}})-P_{Z^{n}}(\bar{\eta}_{{}_{KL,{\pi^{*}}}})\leq
0$

Next the 1st term can be bounded as

$P(\bar{\eta}_{{}_{KL,\hat{\pi}^{*}}})-P_{Z^{n}}(\bar{\eta}_{{}_{KL,\hat{\pi}^{*}}})\leq\sup_{\pi\in\Pi}[P_{Z^{n}}(\eta_{{}_{KL}})-P(\bar{\eta}_{{}_{KL}})]$
$\leq\sup_{\pi\in\Pi}\|P_{Z^{n}}(\bar{\eta}_{{}_{KL}})-P(\bar{\eta}_{{}_{KL}})\|$

Similarly upper bound can be written for the 3rd term Therefore we can upper
bound the above expression as

$\displaystyle
P(\bar{\eta}_{{}_{KL,\hat{\pi}^{*}}})-P(\bar{\eta}_{{}_{KL,{\pi^{*}}}})\leq
2\sup_{\pi\in\Pi}\|P_{Z^{n}}(\bar{\eta}_{{}_{KL}})-P(\bar{\eta}_{{}_{KL}})\|$

From Equation-(34) we have

$\sup_{\pi\in\Pi}\|P_{Z^{n}}(\bar{\eta}_{{}_{KL}})-P(\bar{\eta}_{{}_{KL}})\|\geq\epsilon/2$
(36)

Using McDiarmids inequality and union bound, we can state the probability of
this event as

$P^{n}(\|P_{Z^{n}}(\bar{\eta}_{{}_{KL}})-P(\bar{\eta}_{{}_{KL}})\|\geq\epsilon/2)\leq
2|\Pi|e^{-\frac{n\epsilon^{2}}{2C^{2}}}$ (37)

The finite difference bound

$C=\frac{1}{1-\gamma}$

Equating the RHS of the expression to $\delta$ and solving for $n$ we get

$n(\epsilon,\delta)\geq\frac{2}{\epsilon^{2}(1-\gamma)^{2}}\log\left(\frac{2|\Pi|}{\delta}\right)$
(38)

for $n\geq n(\epsilon,\delta)$ the probability of receiving a bad sample is
less than $\delta$. ∎

###  A-B $\epsilon$-Optimality result under Adaptive Transfer-Learning

Consider MDP $M^{*}$ and $\hat{M}$ which differ in their transition models.
For the sake of analysis, let $M^{*}$ be the MDP with ideal transition model,
such that target follows source transition $p^{*}$ precisely. Let $\hat{p}$ be
the transition model achieved using the estimated policy learned over data
interacting with the target model and the associated MDP be denoted as
$\hat{M}$. We analyze the $\epsilon$-optimality of return under adapted source
optimal policy through ATL.

###### Definition A.2

Given the value function $V^{*}=V^{\pi^{*}}$ and model $M_{1}$ and $M_{2}$,
which only differ in the corresponding transition models $p_{1}$ and $p_{2}$.
Define $\forall s,a\in\mathcal{S}\times\mathcal{A}$

$d_{M_{1},M_{2}}^{V^{*}}=\sup_{s,a\in\mathcal{S}\times\mathcal{A}}\left|\displaystyle\mathop{\mathbb{E}}_{s^{\prime}\sim
P_{1}(s,a)}[V^{*}(s^{\prime})]-\displaystyle\mathop{\mathbb{E}}_{s^{\prime}\sim{P}_{2}(s,a)}[V^{*}(s^{\prime})]\right|.$

###### Lemma A.3

Given $M^{*}$, $\hat{M}$ and value function $V^{\pi^{*}}_{M^{*}}$,
$V^{\pi^{*}}_{\hat{M}}$ the following bound holds
$\left\|V^{\pi^{*}}_{M^{*}}-V^{\pi^{*}}_{\hat{M}}\right\|_{\infty}\leq\frac{\gamma\epsilon}{(1-\gamma)^{2}}$

where $\max_{s,a}\|\hat{p}(.|s,a)-p^{*}(.|s,a)\|\leq\epsilon$ and $\hat{p}$
and $p^{*}$ are transition of MDP $\hat{M},M^{*}$ respectively.

The proof of this lemma is based on the simulation lemma [34] (see
Supplementary document). Similar results for RL with imperfect models were
reported by [35].

###### Lemma A.4

Given $M^{*}$, $\hat{M}$ and value function $V^{\pi^{*}}_{M^{*}}$,
$V^{\pi^{*}}_{\hat{M}}$ the following bound holds
$\left\|V^{\pi^{*}}_{M^{*}}-V^{\pi^{*}}_{\hat{M}}\right\|_{\infty}\leq\frac{\gamma\epsilon}{(1-\gamma)^{2}}$

where $\max_{s,a}\|\hat{p}(.|s,a)-p^{*}(.|s,a)\|\leq\epsilon$ and $\hat{p}$
and $p^{*}$ are transition of MDP $\hat{M},M^{*}$ respectively.

###### Proof:

For any $s\in\mathcal{S}$

$\displaystyle|V^{\pi^{*}}_{\hat{M}}(s)-V^{\pi^{*}}_{M^{*}}(s)|_{\infty}$
$\displaystyle=|r(s,a)+\gamma\left<\hat{p}(s^{\prime}|s,a),V^{\pi^{*}}_{\hat{M}}(s^{\prime})\right>$
$\displaystyle-r(s,a)-\gamma\left<p^{*}(s^{\prime}|s,a),V^{\pi^{*}}_{M^{*}}(s^{\prime})\right>|_{\infty}$

Add and subtract the term
$\gamma\left<p^{*}(s^{\prime}|s,a),V^{\pi^{*}}_{\hat{M}}(s^{\prime})\right>$

$\displaystyle=$
$\displaystyle|\gamma\left<\hat{p}(s^{\prime}|s,a),V^{\pi^{*}}_{\hat{M}}(s^{\prime})\right>-\gamma\left<p^{*}(s^{\prime}|s,a),V^{\pi^{*}}_{\hat{M}}(s^{\prime})\right>$
$\displaystyle+$
$\displaystyle\gamma\left<p^{*}(s^{\prime}|s,a),V^{\pi^{*}}_{\hat{M}}(s^{\prime})\right>-\gamma\left<p^{*}(s^{\prime}|s,a),V^{\pi^{*}}_{M^{*}}(s^{\prime})\right>|_{\infty}$
$\displaystyle\leq$
$\displaystyle\gamma|\left<\hat{p}(s^{\prime}|s,a),V^{\pi^{*}}_{\hat{M}}(s^{\prime})\right>-\left<p^{*}(s^{\prime}|s,a),V^{\pi^{*}}_{\hat{M}}(s^{\prime})\right>|$
$\displaystyle+$
$\displaystyle\gamma|\left<p^{*}(s^{\prime}|s,a),V^{\pi^{*}}_{\hat{M}}(s^{\prime})\right>-\gamma\left<p^{*}(s^{\prime}|s,a),V^{\pi^{*}}_{M^{*}}(s^{\prime})\right>|_{\infty}$
$\displaystyle\leq$
$\displaystyle\gamma|\hat{p}(s^{\prime}|s,a)-p^{*}(s^{\prime}|s,a)|_{\infty}|V^{\pi^{*}}_{\hat{M}}(s^{\prime})|_{\infty}$
$\displaystyle+\gamma|V^{\pi^{*}}_{\hat{M}}(s)-V^{\pi^{*}}_{M^{*}}(s)|_{\infty}$

Using the definition of $\epsilon$ in above expression, we can write

$|V^{\pi^{*}}_{\hat{M}}(s)-V^{\pi^{*}}_{M^{*}}(s)|_{\infty}\leq\gamma\epsilon|V^{\pi^{*}}_{\hat{M}}(s^{\prime})|_{\infty}+\gamma|V^{\pi^{*}}_{\hat{M}}(s)-V^{\pi^{*}}_{M^{*}}(s)|_{\infty}$

Therefore

$|V^{\pi^{*}}_{\hat{M}}(s)-V^{\pi^{*}}_{M^{*}}(s)|_{\infty}\leq\frac{\gamma\epsilon|V^{\pi^{*}}_{\hat{M}}(s^{\prime})|_{\infty}}{1-\gamma}$

Now we solve for expression $|V^{\pi^{*}}_{\hat{M}}(s^{\prime})|_{\infty}$. We
know that this term is bounded as

$|V^{\pi^{*}}_{\hat{M}}(s^{\prime})|_{\infty}\leq\frac{R_{max}}{1-\gamma}$

where $R_{max}=1$, therefore we can write the complete expression as

$|V^{\pi^{*}}_{\hat{M}}(s)-V^{\pi^{*}}_{M^{*}}(s)|_{\infty}\leq\frac{\gamma\epsilon}{(1-\gamma)^{2}}$

∎

Env | Property | source | Target | %Change
---|---|---|---|---
Hopper | Floor Friction | 1.0 | 2.0 | +100%
HalfCheetah | gravity | -9.81 | -15 | +52%
| Total Mass | 14 | 35 | +150%
| Back-Foot Damping | 3.0 | 1.5 | -100%
| Floor Friction | 0.4 | 0.1 | -75%
Walker2d | Density | 1000 | 1500 | +50%
| Right-Foot Friction | 0.9 | 0.45 | -50%
| Left-Foot Friction | 1.9 | 1.0 | -47.37%
TABLE III: Transition Model and environment properties for Source and Target task and % change | Hopper | Walker2d | HalfCheetah
---|---|---|---
State Space | 12 | 18 | 17
Control Space | 3 | 6 | 6
Number of layers | 3 | 3 | 3
Layer Activations | tanh | tanh | tanh
Total num. of network params | 10530 | 28320 | 26250
Discount | 0.995 | 0.995 | 0.995
Learning rate ($\alpha$) | $1.5\times 10^{-5}$ | $8.7\times 10^{-6}$ | $9\times 10^{-6}$
$\beta$ initial Value | 0.5 | 0.5 | 0.5
$\beta$-Learning rate ($\bar{\alpha}$) | 0.1 | 0.1 | 0.1
Batch size | 20 | 20 | 5
Policy Iter | 3000 | 5000 | 1500
TABLE IV: Policy Network details and Network learning parameter details

##  Appendix B Learning the Mixing Coefficient $\beta$

A hierarchical update of the mixing coefficient $\beta$ is carried out over
n-test trajectories, collected using the updated policy network
$\pi_{\theta^{\prime}}(a|s)$. The mixing coefficient $\beta$ is learnt by
optimizing the return over trajectory as follows,

$\beta=arg\max_{\beta}(\bar{\eta}_{KL,\beta}(\pi_{\theta^{\prime}},\pi^{*}))$

where $\theta^{\prime}$ is parameter after the policy update step.

$\beta=arg\max_{\beta}\mathop{\mathbb{E}}_{s_{t},a_{t}\sim\tau}\left(p_{\pi_{\theta}}(\tau)\sum_{t=1}^{\infty}\gamma^{t}r^{\prime}_{t}\right)$

We can use gradient ascent to update parameter $\beta$ in direction of
optimizing the reward mixing as follows,

$\beta\leftarrow\beta+\bar{\alpha}\nabla_{\beta}(\bar{\eta}_{KL,\beta}(\pi_{\theta^{\prime}},\pi^{*})).$

Using the definition of mixed reward as
$r^{\prime}_{t}=(1-\beta)r_{t}-\beta\zeta_{t}$, we can simplify the above
gradient as,

$\beta\leftarrow\beta+\bar{\alpha}\mathop{\mathbb{E}}_{s_{t},a_{t}\sim\tau}\left(p_{\pi_{\theta}}(\tau)\sum_{t=1}^{\infty}\gamma^{t}\nabla_{\beta}(r^{\prime}_{t})\right)$
$\beta\leftarrow\beta+\bar{\alpha}\mathop{\mathbb{E}}_{s_{t},a_{t}\sim\tau}\left(\sum_{t=1}^{\infty}\gamma^{t}(r_{t}-\zeta_{t})\right).$

We use stochastic gradient ascent to update the mixing coefficient $\beta$ as
follows

$\beta\leftarrow\beta+\bar{\alpha}\hat{g}_{\beta},\hskip 5.69054pts.t\hskip
5.69054pt0\leq\beta\leq 1.$

where $\bar{\alpha}$ is the learning rate and
$\hat{g}_{\beta}=P_{Z^{n}_{test}}(\nabla_{\beta}\bar{\eta}_{{}_{KL,\beta}})$
is the empirical estimate of the gradient of the total return
$\bar{\eta}_{{}_{KL,\beta}}(\pi_{\theta^{\prime}},\pi^{*})$. The gradient
estimate $\hat{g}_{\beta}$ over data
$(Z^{n}_{test}:\\{s_{i},a_{i},a^{\prime}_{i}\\}_{i}^{T})$ is computed as
follows,

$\hat{g}_{\beta}=\frac{1}{N}\sum_{i=1}^{N}\left(\sum_{t=1}^{\mathcal{H}}\gamma^{t}(r_{t}-\zeta_{t})\right)$

where $\mathcal{H}$ truncated trajectory length from experiments.

As we can see the gradient of objective with respect to mixing coefficient
$\beta$ is an average over difference between environmental and intrinsic
rewards. If $r_{t}-\zeta_{t}\geq 0$ the update will move parameter $\beta$
towards favoring learning through exploration more than learning through
adaptation and visa versa.

As $\beta$ update is a constrained optimization with constraint
$0\leq\beta\leq 1$. We handle this constrained optimization by modelling
$\beta$ as output of Sigmoidal network parameterized by parameters $\phi$.

$\beta=\sigma(\phi)$

And the constrained optimization can be equivalently written as optimizing
w.r.to $\phi$ as follows

$\phi\leftarrow\phi+\bar{\alpha}\hat{g}_{\beta}\nabla_{\phi}(\beta),\hskip
5.69054ptwhere\hskip 5.69054pt\beta=\sigma(\phi)$

The reward mixing co-efficient $\beta$ learned for HalfCheetah, Hopper and
Walker2d envs is provided in Figure-4. For all the experiments we start with
$\beta=0.5$ that is placing equal probability of learning through adaptation
and learning through exploration. As we can observe the reward mixing leans
towards learning through adaptation for HalfCheetah and Hopper envs. Whereas,
as for Walker2d the beta initially believes learning from exploration more,
but quickly leans toward learning from source policy and adaptation.