# oIRL: Robust Adversarial Inverse Reinforcement Learning with Temporally
Extended Actions

David Venuto  Jhelum Chakravorty  Leonard Boussioux  Junhao Wang  Gavin
McCracken  Doina Precup

###### Abstract

Explicit engineering of reward functions for given environments has been a
major hindrance to reinforcement learning methods. While Inverse Reinforcement
Learning (IRL) is a solution to recover reward functions from demonstrations
only, these learned rewards are generally heavily entangled with the dynamics
of the environment and therefore not portable or _robust_ to changing
environments. Modern adversarial methods have yielded some success in reducing
reward entanglement in the IRL setting. In this work, we leverage one such
method, Adversarial Inverse Reinforcement Learning (AIRL), to propose an
algorithm that learns hierarchical disentangled rewards with a policy over
options. We show that this method has the ability to learn _generalizable_
policies and reward functions in complex transfer learning tasks, while
yielding results in continuous control benchmarks that are comparable to those
of the state-of-the-art methods.

Machine Learning, ICML

##  1 Introduction

Reinforcement learning (RL) has been able to learn policies in complex
environments but it usually requires designing suitable reward functions for
successful learning. This can be difficult and may lead to learning sub-
optimal policies with unsafe behavior (Amodei et al., 2016) in the case of
poor engineering. Inverse Reinforcement Learning (IRL) (Ng & Russell, 2000;
Abbeel & Ng, 2004) can facilitate such reward engineering through learning an
expert’s reward function from expert demonstrations.

IRL, however, comes with many difficulties and the problem is not well-defined
because, for a given set of demonstrations, the number of optimal policies and
corresponding rewards can be very large, especially for high dimensional
complex tasks. Also, many IRL algorithms learn reward functions that are
heavily shaped by environmental dynamics. Policies learned on such reward
functions may not remain optimal with slight changes in the environment.
Adversarial Inverse Reinforcement Learning (AIRL) (Fu et al., 2018) generates
more generalizable policies with disentangled reward functions that are
invariant to the environmental dynamics. The reward and value function are
learned simultaneously to compute the reward function in a state-only manner.
This is an instance of a transfer learning problem with changing dynamics,
where the agent learns an optimal policy in one environment and then transfers
it to an environment with different environmental dynamics. A practical
example of this transfer learning problem would be teaching a robot to walk
with some mechanical structure, and then generalize this knowledge to perform
the task with different sized structural components.

Other methods have been developed to learn demonstrations in environments with
differing dynamics and then exploit this knowledge while performing tasks in a
novel environment. As complex tasks in different environments often come from
several reward functions, methods such as Maximum Entropy IRL and GAN-Guided
Cost Learning (GAN-GCL) tend to overgeneralize (Finn et al., 2016b). One way
to help solve the problem of over-fitting is to break down a policy into small
_option_ (temporally extended action)-policies that solve various aspects of
an overall task. This method has been shown to create a policy that is more
generalizable (Sutton et al., 1999; Taylor & Stone, 2009). Methods such as
Option-Critic have implemented modern RL architectures with a policy over
options and have shown improvements for generalization of policies (Bacon et
al., 2017). OptionGAN (Henderson et al., 2018) also proposed an IRL framework
for a policy over options and is shown to have some improvement in one-shot
transfer learning tasks, but does not return disentangled rewards.

In this paper, we introduce Option-Inverse Reinforcement Learning (oIRL), to
investigate transfer learning with options. Following AIRL, we propose an
algorithm that computes disentangled rewards to learn joint reward-policy
options, with each option-policy having rewards that are disentangled from
environmental dynamics. These policies are shown to be heavily portable in
transfer learning tasks with differences in environments. We evaluate this
method in a variety of continuous control tasks in the Open AI Gym environment
using the MuJoCo simulator (Brockman et al., 2016; Todorov et al., ) and
GridWorlds with MiniGrid (Chevalier-Boisvert et al., 2018). Our method shows
improvements in terms of reward on a variety of transfer learning tasks while
still performing better than benchmarks for standard continuous control tasks.

##  2 Preliminaries

Markov Decision Processes (MDP) are defined by a tuple
$\langle\mathcal{S},\mathcal{A},\mathbb{P},R,\gamma\rangle$ where
$\mathcal{S}$ is a set of states, $\mathcal{A}$ is the set of actions
available to the agent, $\mathbb{P}$ is the transition kernel giving a
probability over next states given the current state and action,
$R:\mathcal{S}\times\mathcal{A}\rightarrow[0,R_{\text{max}}]$ is a reward
function and $\gamma\in[0,1]$ is a discount factor. $s_{t}$ and $a_{t}$ are
respectively the state and action of the expert at time instant $t$. We define
a policy $\pi$ as the probability distribution over actions conditioned on the
current state; $\pi:S\times\mathcal{A}\rightarrow[0,1]$. A policy is modeled
by a Gaussian distribution $\pi_{\theta}\sim\mathcal{N}(\mu,\,\sigma^{2})$
where $\theta$ is the policy parameters. The value of a policy is defined as
$V_{\pi}(s)=\operatorname{\mathbb{E}}_{\pi}[\sum_{t=0}^{\infty}\gamma^{t}r_{t+1}|s]$,
where $\mathbb{E}$ denotes the expectation. An agent follows a policy $\pi$
and receives reward from the environment. A state-action value function is
$Q_{\pi}(s,a)=\operatorname{\mathbb{E}}_{\pi}[\sum_{t=0}^{\infty}\gamma^{t}r_{t+1}|s,a]$.
The advantage is $A_{\pi}(s,a)=Q_{\pi}(s,a)-V_{\pi}(s)$. $r(s,a)$ represents a
one-step reward.

Options ($\omega\in\Omega$) are defined as a triplet
($I_{\omega},\pi_{\omega},\beta_{\omega}$), where $\pi_{\omega}$ is a policy
over options, $I_{\omega}\in S$ is the initiation set of states and
$\beta_{\omega}:S\rightarrow[0,1]$ is the termination function. The policy
over options is defined by $\pi_{\Omega}$. An option has a reward $r_{\omega}$
and an option policy $\pi_{\omega}$.

The policy over options is parameterized by $\zeta$, the intra-option policies
by $\alpha$ for each option, the reward approximator by $\theta$, and the
option termination probabilities by $\delta$.

In the one-step case, selecting an option using the policy-over-options can be
viewed as a mixture of completely specialized experts. This overall policy can
be defined as
$\pi_{\Theta}(a|s)=\sum_{\omega\in\Omega}\pi_{\Omega}(\omega|s)\pi_{\omega}(a|s)$.

Disentangled Rewards are formally defined as a reward function
$r_{\theta}^{*}(s,a,s^{\prime})$ that is disentangled with respect to (w.r.t.)
a ground-truth reward and a set of environmental dynamics $\mathcal{T}$ such
that under all possible dynamics $T\in\mathcal{T}$, the optimal policy
computed w.r.t. the reward function is the same.

##  3 Related Work

Generative Adversarial Networks (GANs) learn the generator distribution
$p_{g}$ and discriminator $D_{\theta_{D}}(\mathbf{x})$. They use a prior
distribution over input noise variables $p(\mathbf{z})$. Given these input
noise variables the mapping $G_{\theta_{g}}(\mathbf{z})$ is learned, which
maps these input noise variables to the data set space. $G$ is a neural
network. Another neural network, $D_{\theta_{D}}(\mathbf{x})$, learns to
estimate the probability that $\mathbf{x}$ came from the data set and not the
generator $p_{g}$.

In our two-player adversarial training procedure, $D$ is trained to maximize
the probability of assigning the correct labels to the dataset and the
generated samples. $G$ is trained to minimize the objective
$\text{log}(1-D_{\theta_{D}}(G_{\theta_{G}}(\mathbf{z})))$, which causes it to
generate samples that are more likely to fool the discriminator.

Policy Gradient methods optimize a parameterized policy $\pi_{\theta}$ using a
gradient ascent. Given a discounting term, the objective to be optimized is
$p(\theta,s_{0})=\operatorname{\mathbb{E}}[\sum_{t=0}^{\infty}\gamma^{t}r_{\theta}(s_{t})|s_{0}]$.
Proximal policy optimization (PPO) (Schulman et al., 2017) is a policy
gradient method that uses policy gradient theorem, which states
$\frac{\partial
p(\theta,s_{0})}{\partial\theta}=\sum_{s}\sum_{t=0}^{\infty}\gamma^{t}P(s_{t}=s|s_{0})\sum_{a}\frac{\partial\pi(a|s)}{\partial\theta}Q_{\pi_{\theta}}(s,a)$.
PPO has been adapted for the option-critic architecture (PPOC) (Klissarov et
al., 2017).

Inverse Reinforcement Learning (IRL) is a form of _imitation learning_ 111Here
the agent learns the expert’s policy by observing expert demonstrations (Ng &
Russell, 2000)., where the expert’s reward is estimated from demonstrations
and then forward RL is applied to that estimated reward to find the optimal
policy. Generative Adversarial Imitation Learning (GAIL) directly extracts
optimal policies from expert’s demonstrations (Ho & Ermon, 2016). IRL infers a
reward function from expert demonstrations, which is then used to optimize a
generator policy.

In IRL, an agent observes a set of state-action trajectories from an expert
demonstrator $\mathcal{D}$. We let
$\mathcal{T}_{\mathcal{D}}=\\{\tau^{E}_{1},\tau^{E}_{2},\dots,\tau^{E}_{n}\\}$
be the state-action trajectories of the expert,
$\tau^{E}_{i}\sim\tau_{\mathcal{D}}$ where
$\tau^{E}_{i}=\\{s_{0},a_{0},s_{1},a_{1}\dots,s_{k},a_{k}\\}$. We wish to find
the reward function $r(s,a)$ given the set of demonstrations
$\mathcal{T}_{\mathcal{D}}$. It is assumed that the demonstrations are drawn
from the optimal policy $\pi^{*}(a|s)$. The Maximum Likelihood Estimation
(MLE) objective in the IRL problems is therefore:

$\text{}\max_{\theta}J(\theta)=\text{}\max_{\theta}\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}[\log(p_{\theta}(\tau))],$
(1)

with $p_{\theta}(\tau)\propto
p(s_{0})\prod_{t=0}^{T}p(s_{t+1}|s_{t},a_{t})\exp{\left({\gamma^{t}r_{\theta}(s_{t},a_{t})}\right)}$.

Adversarial Inverse Reinforcement Learning (AIRL) is based on GAN-Guided Cost
Learning (Finn et al., 2016a), which casts the MLE objective as a Generative
Adversarial Network (GAN) (Goodfellow et al., 2014) optimization problem over
trajectories. In AIRL (Fu et al., 2018), the discriminator probability
$D_{\theta}$ is evaluated using the state-action pairs from the generator
(agent), as given by

$D_{\theta}(s,a)=\frac{\exp(f_{\theta}(s,a))}{\exp(f_{\theta}(s,a))+\pi(a|s)}.$
(2)

The agent tries to maximize
$R(s,a)=\log(1-D_{\theta}(s,a))-\log(D_{\theta}(s,a))$ where $f_{\theta}(s,a)$
is a learned function and $\pi$ is pre-computed. This formula is similar to
GAIL but with a _recoverable reward function_ since GAIL outputs 0.5 for the
reward of all states and actions at optimality. The discriminator function is
then formulated as $f_{\theta,\Phi}(s,a,s^{\prime})=g_{\theta}(s,a)+\gamma
h_{\Phi}(s^{\prime})-h_{\Phi}(s)$ given shaping function $h_{\Phi}$ and reward
approximator $g_{\theta}$. Under deterministic dynamics, it is shown in AIRL
that there is a state-only reward approximator (
$f^{*}(s,a,s^{\prime})=r^{*}(s)+\gamma V^{*}(s^{\prime})-V^{*}(s)=A^{*}(s,a)$
where the reward is invariant to transition dynamics and is disentangled.

Hierarchical Inverse Reinforcement Learning learns policies with high level
temporally extended actions using IRL. OptionGAN (Henderson et al., 2018)
provides an adversarial IRL objective function for the discriminator with a
policy over options. It is formulated such that $L_{\text{reg}}$ defines the
regularization terms on the mixture of experts so that they converge to
options. The discriminator objective in OptionGAN takes state-only input and
is formulated as:

$\displaystyle L_{\Omega}$
$\displaystyle=\operatorname{\mathbb{E}}_{\omega}[\pi_{\Omega,\zeta}(\omega|s)(L_{\alpha,\omega})]+L_{reg},$
where $\displaystyle L_{\alpha,\omega}$
$\displaystyle=\operatorname{\mathbb{E}}_{\tau^{N}}[\log(r_{\theta,\omega}(s))]+\operatorname{\mathbb{E}}_{\tau^{E}}[\log(1-r_{\theta,\omega}(s))].$
(3)

In Directed-Info GAIL (Sharma et al., 2019) implements GAIL in a policy over
options framework.

Work such as (Krishnan et al., 2016) solves this hierarchical problem of
segmenting expert demonstration transitions by analyzing the changes in _local
linearity w.r.t a kernel function_. It has been suggested that decomposing the
reward function is not enough (Henderson et al., 2018). Other works have
learned the latent dimension along with the policy for this task (Hausman et
al., 2017; Wang et al., 2017). In this formulation, the latent structure is
encoded in an unsupervised manner so that the desired latent variable does not
need to be provided. This work parallels many hierarchical IRL methods but
with recoverable robust rewards.

##  4 MLE Objective for IRL Over Options

Let $(s_{0},a_{0},\dots s_{T},a_{T})\in\tau^{E}_{i}$ be an expert trajectory
of state-action pairs. Denote by $(s_{0},a_{0},\omega_{0}\dots
s_{T},a_{T},\omega_{T})\in\tau_{\pi_{\Theta,t}}$ a novice trajectory generated
by policy over options $\pi_{\Theta,t}$ of the generator at iteration $t$.

Given a trajectory of state-action pairs, we first define an option transition
probability given a state and an option. Similar transition probabilities
given state, action or option information are defined in (Appendix A).

$\displaystyle P(s_{t+1},\omega_{t+1}\,|\,s_{t},\omega_{t})$
$\displaystyle=\sum_{a\in
A}\pi_{\omega,\alpha}(a|s_{t})P(s_{t+1}|s_{t},a)((1-\beta_{\omega_{t},\delta}(s_{t+1}))\textbf{1}_{\omega_{t}=\omega_{t+1}}$
$\displaystyle\hskip
40.00006pt+\beta_{\omega_{t},\delta}(s_{t+1})\pi_{\Omega,\zeta}(\omega_{t+1}|s_{t+1})).$
(4)

We can similarly define a discounted return recursively. Consider the policy
over options based on the probabilities of terminating or continuing the
option policies given a reward approximator $\hat{r}_{\theta}(s,a)$ for the
state-action reward.

$\displaystyle
R_{\theta,\delta}(s,\omega,a)\coloneqq\operatorname{\mathbb{E}}\Big{[}\hat{r}_{\theta,\omega}(s,a)+\gamma\sum_{s^{{}^{\prime}}\in
S}P(s^{{}^{\prime}}|s,a)$
$\displaystyle\Big{(}\beta_{\omega,\delta}(s^{{}^{\prime}})R^{\Omega}_{\zeta,\theta,\alpha,\delta}(s^{{}^{\prime}})+\big{(}1-\beta_{\omega,\delta}(s^{{}^{\prime}})\big{)}R_{\theta,\alpha,\delta}(s^{{}^{\prime}},\omega)\Big{)}\Big{]}.$
(5)

$\omega_{0}$ is selected according to $\pi_{\zeta,\Omega}(\omega|s)$. The
expressions for all relevant discounted returns appearing in the analysis are
given in Appendix B. A suitable parameterization of the discounted return $R$
can be found by maximizing the causal entropy
$\operatorname{\mathbb{E}}_{\tau\sim\mathcal{D}}[\log(p_{\theta}(\tau))]$
w.r.t parameter $\theta$. We then have for a trajectory $\tau$ with $T$ time-
steps:

$\displaystyle p_{\theta}(\tau)$ (6) $\displaystyle\approx
p(s_{0},\omega_{0})\prod_{t=0}^{T-1}P(s_{t+1},\omega_{t+1}|s_{t},\omega_{t},a_{t})e^{R_{\theta,\delta}(s_{t},\omega_{t},a_{t})}.$

###  4.1 MLE Derivative

Similar to (Fu et al., 2018) and (Finn et al., 2016a), we define the MLE
objective for the generator $p_{\theta}$ as

$\displaystyle
J(\theta)=\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}[\sum_{t=0}^{T}R^{\Omega}_{\zeta,\theta,\delta}(s_{t},a_{t})]$
$\displaystyle\hskip
20.00003pt-\operatorname{\mathbb{E}}_{p_{\theta}}[\sum_{t=0}^{T}\sum_{\omega\in\Omega}\pi_{\zeta,\Omega}(\omega|s_{t})R_{\theta,\delta}(s_{t},\omega,a_{t})].$
(7)

Note that we may or may not know the option trajectories in our expert
demonstrations, instead they are estimated according to the policy over
options. The gradient of (7) w.r.t $\theta$ (See Appendix B for detailed
derivations) is given by:

$\displaystyle\frac{\partial}{\partial\theta}J(\theta)=\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\Big{[}\frac{\partial}{\partial\theta}\log(p_{\theta}(\tau))\Big{]}$
$\displaystyle\approx\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\Big{[}\sum_{t=0}^{T}\frac{\partial}{\partial\theta}R^{\Omega}_{\zeta,\theta,\delta}(s_{t},a_{t})\Big{]}-\operatorname{\mathbb{E}}_{p_{\theta}}\Big{[}\sum_{t=0}^{T}\frac{\partial}{\partial\theta}R^{\Omega}_{\zeta,\theta,\delta}(s_{t},a_{t})\Big{]}.$

We define $p_{\theta,t}(s_{t},a_{t})=\int_{s_{t^{{}^{\prime}}\neq
t},a_{t^{{}^{\prime}}\neq t}}p_{\theta}(\tau)ds_{t^{\prime}}da_{t^{\prime}}$
as the state action marginal at time $t$. This allows us to examine the
trajectory from step $t$ as defined similarly in (Fu et al., 2018).
Consequently, we have

$\displaystyle\frac{\partial}{\partial\theta}J(\theta)$
$\displaystyle=\sum_{t=0}^{T}\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\biggl{[}\frac{\partial}{\partial\theta}R^{\Omega}_{\zeta,\theta,\delta}(s_{t},a_{t})\biggr{]}$
(xyz) $\displaystyle\hskip
30.00005pt-\operatorname{\mathbb{E}}_{p_{\theta,t}}\biggl{[}\frac{\partial}{\partial\theta}R^{\Omega}_{\zeta,\theta,\delta}(s_{t},a_{t})\biggr{]}.$
(8)

Since $p_{\theta}$ is difficult to draw samples from, we estimate it using
importance sampling distribution over the generator density. Then, we compute
an importance sampling estimate of a _mixture policy_ $\mu_{t,w}(\tau)$ for
each option $w$ as follows.

We sample a mixture policy $\mu_{\omega}(a|s)$ defined as
$\frac{1}{2}\pi_{\omega}(a|s)+\frac{1}{2}\hat{p}_{\omega}(a|s)$ and
$\hat{p}_{\omega}(a|s)$ is a density estimate trained on the demonstrations.
We wish to minimize $D_{KL}(\pi_{w}(\tau)|p_{\omega}(\tau))$ to reduce the
importance sampling distribution variance, where $D_{KL}$ is the
_Kullback–Leibler divergence_ metric (Kullback & Leibler, 1951) between two
probability distributions. Applying the aforementioned density estimates in
(8), we can express the gradient of the MLE objective $J$ follows:

$\displaystyle\frac{\partial}{\partial\theta}J(\theta)=\sum_{t=0}^{T}\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}[\frac{\partial}{\partial\theta}R^{\Omega}_{\zeta,\theta,\delta}(s_{t},a_{t})]-$
$\displaystyle\hskip-20.00003pt\operatorname{\mathbb{E}}_{\mu_{t}}\biggl{[}\sum_{\omega\in\Omega}\pi_{\zeta,\Omega}(\omega|s_{t})\frac{p_{\theta,t,\omega}(s_{t},a_{t})}{\mu_{t,w}(s_{t},a_{t})}\frac{\partial}{\partial\theta}R_{\theta,\delta}(s_{t},\omega,a_{t})\biggr{]},$
(9)

where

$\displaystyle\frac{\partial}{\partial\theta}R^{\Omega}_{\zeta,\theta,\alpha,\delta}(s)=\operatorname{\mathbb{E}}\Bigg{[}\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s)\Big{[}\sum_{a\in
A}\pi_{\omega,\alpha}(a|s)$
$\displaystyle\Big{(}\frac{\partial}{\partial\theta}\hat{r}_{\theta}(s,a)+\gamma\sum_{s^{{}^{\prime}}\in
S}P(s^{{}^{\prime}}|s,a)\Big{(}\beta_{w,\delta}(s^{{}^{\prime}})\frac{\partial}{\partial\theta}R^{\Omega}_{\zeta,\theta,\alpha,\delta}(s^{{}^{\prime}})$
$\displaystyle\hskip
30.00005pt+(1-\beta_{\omega,\delta}(s^{{}^{\prime}}))\frac{\partial}{\partial\theta}R_{\theta,\alpha,\delta}(s^{{}^{\prime}},\omega)\Big{)}\Big{)}\Big{]}\Bigg{]}.$
(10)

##  5 Discriminator Objective

In this section we formulate the discriminator, parameterized by $\theta$, as
the odds ratio between the policy and the exponentiated reward distribution
for option $\omega$. We have a discriminator $D_{\theta,\omega}$ for each
option $\omega$ and a sample generator option policy $\pi_{w}$, defined as
follows:

$D_{\theta,\omega}(s,a)=\frac{\exp(f_{\theta,\omega}(s,a))}{\exp(f_{\theta,\omega}(s,a))+\pi_{w}(a|s)}.$
(11)

###  5.1 Recursive Loss Formulation

The discriminator $D_{\theta,\omega}$ is trained by minimizing the cross-
entropy loss between expert demonstrations and generated examples assuming we
have the same number of options in the generated and expert trajectories. We
define the loss function $L_{\theta}$ as follows:

$\displaystyle l_{\theta}(s,a,\omega)$ (12)
$\displaystyle=-\operatorname{\mathbb{E}}_{\mathcal{D}}[\log(D_{\theta,\omega}(s,a))]-\operatorname{\mathbb{E}}_{\pi_{\Theta,t}}[\log(1-D_{\theta,\omega}(s,a))].$

The parameterized total loss for the entire trajectory,
${}_{L\theta,\alpha,\delta}(s,a,\omega)$, can be expressed recursively as
follows by taking expectations over the next options and states:

$\displaystyle L_{\theta,\delta}(s,a,\omega)$
$\displaystyle=l_{\theta}(s,a,\omega)+\gamma\sum_{s^{{}^{\prime}}\in
S}P(s^{{}^{\prime}}|s,a)\Big{(}\beta_{w,\delta}(s^{{}^{\prime}})L^{\Omega}_{\zeta,\theta,\alpha,\delta}(s^{{}^{\prime}})$
$\displaystyle\hskip
70.0001pt+\big{(}1-\beta_{w,\delta}(s^{{}^{\prime}})\big{)}L_{\theta,\alpha,\delta}(s^{{}^{\prime}},w)\Big{)}$
(13) $\displaystyle
L_{\theta,\alpha,\delta}(s,w)\coloneqq\operatorname{\mathbb{E}}_{a\in
A}[L_{\theta,\delta}(s,w,a)]$ (14) $\displaystyle
L^{\Omega}_{\zeta,\theta,\delta}(s,a)\coloneqq\operatorname{\mathbb{E}}_{w\in\Omega}[L_{\theta,\delta}(s,w,a)]$
(15) $\displaystyle
L^{\Omega}_{\zeta,\theta,\alpha,\delta}(s)\coloneqq\operatorname{\mathbb{E}}_{\omega\in\Omega}[L_{\theta,\alpha,\delta}(s,\omega)].$
(16)

The agent wishes to minimize $L_{\theta,\alpha,\delta}$ to find its optimal
policy.

###  5.2 Optimization Criteria

For a given option $\omega$, define the reward function
$\hat{R}_{\theta,\delta}(s,\omega,a)$, which is to be maximised. We then write
a negative discriminator loss ($-L^{D}$) to turn our loss minimization problem
into a maximization problem, as follows:

$\displaystyle\hskip-20.00003pt-L^{D}=\hat{R}_{\theta,\delta}(s,\omega,a)=$
$\displaystyle\log(D_{\theta,\omega}(s,a))-\log(1-D_{\theta,\omega}(s,a)).$
(17)

We set a mixture of experts and novice as $\bar{\mu}$ observations in our
gradient. We then wish to take the derivative of the inverse discriminator
loss as,

$\displaystyle\frac{\partial}{\partial\theta}(-L^{D})=\sum_{t=0}^{T}\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\left[\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s_{t})\frac{\partial}{\partial\theta}\Big{(}-L_{\theta,\delta}(s_{t},\omega,a_{t})\Big{)}\right]-$
(18)
$\displaystyle\operatorname{\mathbb{E}}_{\bar{\mu}_{t}}\Big{[}\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s_{t})\left(\frac{\exp(-L_{\theta,\delta}(s_{t},\omega,a_{t}))}{\frac{1}{2}\exp(-L_{\theta,\delta}(s_{t},\omega,a_{t}))+\frac{1}{2}\pi_{\omega}(a_{t}|s_{t})}\right)$
$\displaystyle\frac{\partial}{\partial\theta}\Big{(}-L_{\theta,\delta}(s_{t},\omega,a_{t})\Big{)}\Big{]}.$

We can multiply the top and bottom of the fraction in the mixture expectation
by the state marginal $\pi_{\omega}(s_{t})=\int_{a\in
A}\pi_{\omega}(s_{t},a_{t})$. This allows us to write
$\hat{p}_{\theta,t,\omega}(s_{t},a_{t})=\exp(L_{\theta,\delta}(s_{t},\omega,a_{t}))\pi_{\omega,t}(s_{t})$.
Using this, we can derive an importance sampling distribution in our loss,

$\displaystyle\frac{\partial}{\partial\theta}(-L^{D})=\sum_{t=0}^{T}\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\left[\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s_{t})\frac{\partial}{\partial\theta}\Big{(}-L_{\theta,\delta}(s_{t},\omega,a_{t})\Big{)}\right]$
(19)
$\displaystyle-\operatorname{\mathbb{E}}_{\bar{\mu}_{t}}\left[\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s_{t})\left(\frac{\hat{p}_{\theta,t,\omega}(s_{t},a_{t})}{\hat{\mu}_{t,\omega}(s_{t},a_{t})}\right)\frac{\partial}{\partial\theta}\Big{(}-L_{\theta,\delta}(s_{t},\omega,a_{t})\Big{)}\right].$

The gradient of this parametrized reward function corresponds to the inverse
of the discriminator’s objective:

$\displaystyle\frac{\partial}{\partial\theta}\hat{R}_{\theta,\delta}(s,\omega,a)\approx\frac{\partial}{\partial\theta}\Big{(}-L_{\theta,\delta}(s,\omega,a)\Big{)}$
$\displaystyle\hskip-10.00002pt=\operatorname{\mathbb{E}}\Big{[}\frac{\partial}{\partial\theta}r_{\theta,\omega}(s,a)+\gamma\sum_{s^{{}^{\prime}}\in
S}P(s^{{}^{\prime}}|s,a)\Big{(}\beta_{\omega,\delta}(s^{{}^{\prime}})$
$\displaystyle\frac{\partial}{\partial\theta}\big{(}-L^{\Omega}_{\zeta,\theta,\alpha,\delta}(s^{{}^{\prime}})\big{)}+\big{(}1-\beta_{\omega,\delta}(s^{{}^{\prime}})\big{)}\frac{\partial}{\partial\theta}\big{(}-L_{\theta,\delta}(s^{{}^{\prime}},\omega)\big{)}\Big{)}\Big{]}.$
(20)

See Appendix C for the detailed derivations of the terms appearing in (5.2).
Substituting (5.2) into (19) one can see that (9) (derivative of MLE
objective) and (10) are of the same form as of (19) (derivative of the
discriminator objective and (5.2)).

##  6 Learning Disentangled State-only Rewards with Options

In this section, we provide our main algorithm for learning robust rewards
with options. Similar to AIRL, we implement our algorithm with a discriminator
update that considers the rollouts of a policy over options. We perform this
update with $(s,a,s^{\prime})$ triplets and a discriminator function in the
form of $f_{\theta,\omega}(s,a,s^{\prime})$ as given in (21). This allows us
to formulate the discriminator with state-only rewards in terms of option-
value function estimates. We can then compute an option-advantage estimate.
Since the reward function only requires state, we learn a reward function and
corresponding policy that is disentangled from the environmental transition
dynamics.

$f_{\omega,\theta}(s,a,s^{\prime})=\hat{r}_{\omega,\theta}(s)+\gamma\hat{V}_{\Omega}(s^{\prime})-\hat{V}_{\Omega}(s)=\hat{A}(s,a,\omega)$
(21)

Where
$Q(s,\omega)=\sum_{a\in\mathcal{A}}\pi_{\omega,\alpha}(a|s)[r_{\omega,\theta}(s,a)+\gamma\sum_{s^{\prime}\in\mathcal{S}}P(s^{\prime}|s,a)((1-\beta_{\delta,\omega}(s^{\prime}))Q(s^{\prime},\omega)+\beta_{\delta,\omega}(s^{\prime})V_{\Omega}(s^{\prime}))]$
and
$V_{\Omega}(s)=\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s)Q(s,\omega)$.

Our discriminator model must learn a parameterization of the reward function
and the value function for each option, given the total loss function in (37).
These parameterized models are learned with a multi-layer perceptron. For each
option, the termination functions $\beta_{\omega,\delta}$ and option-policies
$\pi_{\omega,\alpha}$ are learned using PPOC.

###  6.1 The main algorithm: Option-Adversarial Inverse Reinforcement Learning
(oIRL)

Our main algorithm, oIRL, is given by Algorithm 1. Here, we iteratively train
a discriminator from expert and novice sampled trajectories using the derived
discriminator objective. This allows us to obtain reward function estimates
for each option. We then use any policy optimization method for a policy over
options given these estimated rewards.

We can also have discriminator input of state-only format as described in
(21). It is important to note that in our recursive loss, we recursively
simulate a trajectory to compute the loss a finite number of times (and return
if the state is terminal). We show the adversarial architecture of this
algorithm in Appendix D.

Algorithm 1 IRL Over Options with Robust Rewards (oIRL)

0: $\text{Expert Trajectories:
}\\{\tau_{1}^{E},\dots,\tau_{n}^{E}\\}\in\mathcal{T}_{\mathcal{D}},\text{Initial
Parameters: }(\theta_{0},\zeta_{0},\delta_{0},\alpha_{0}),\gamma$

1: $\text{Initialize policies
}\pi_{\omega,\alpha_{0}},\pi_{\Omega,\zeta_{0}}\text{ and discriminators
}D_{\theta_{0},\omega},\text{ and
}\beta_{\omega,\delta_{0}}\forall\omega\in\Omega$

2: for $\text{step }t=0,1,2,\dots,T$ do

3: $\text{Collect trajectories }\tau_{i}=(s_{0},a_{0},\omega_{0},\dots)\text{
from
}\pi_{\omega,\alpha_{t}},\pi_{\Omega,\zeta_{t}},\beta_{\omega,\delta_{t}}$

4: $\textbf{Train discriminator }D_{\theta_{t},\omega}$

5: for $\text{step }k=0,1,2,\dots$ do

6: $\text{Sample }(s_{k},a_{k},{s^{\prime}}_{k},\omega_{k})\sim\tau_{i,t}$

7: if $s^{\prime}$ not terminal state then

8: $\text{Sample
}\omega_{k}^{{}^{\prime}}\sim\pi_{\Omega,\zeta_{t}}(\omega|s_{k}^{{}^{\prime}}),a_{k,1}^{{}^{\prime}}\sim\pi_{\omega_{k}^{{}^{\prime}},\alpha_{t}}(a|s_{k}^{{}^{\prime}}),a_{k,2}^{{}^{\prime}}\sim\pi_{\omega_{k},\alpha_{t}}(a|s_{k}^{{}^{\prime}})$

9: $\text{ Observe
}s^{{}^{\prime\prime}}_{k,1},s^{{}^{\prime\prime}}_{k,2}\text{ from
environment}$

10:
$L_{k}(s_{k},a_{k},s_{k}^{{}^{\prime}},\omega_{k})=-\operatorname{\mathbb{E}}_{\mathcal{D}}[\log(D_{\theta_{t},\omega_{k}}(s_{k},a_{k},s_{k}^{{}^{\prime}}))]-\operatorname{\mathbb{E}}_{\pi_{\Theta,t}}[\log(1-D_{\theta_{t},\omega_{k}}(s_{k},a_{k},s_{k}^{{}^{\prime}}))]$

11: $\text{Optimize model parameters w.r.t.:
}-L^{D}=L_{k}+\gamma(\beta_{\delta_{t},\omega_{k}}(s^{\prime})L(s_{k}^{{}^{\prime}},a_{k,1}^{{}^{\prime}},s_{k,1}^{{}^{\prime\prime}},\omega_{k}^{{}^{\prime}})$

12:
$+(1-\beta_{\delta_{t},\omega_{k}}(s_{k}^{{}^{\prime}}))L(s_{k}^{{}^{\prime}},a_{k,2}^{{}^{\prime}},s_{k,2}^{{}^{\prime\prime}},\omega_{k}))$

13: end if

14: end for

15: $\text{Obtain reward
}r_{\theta_{t},\omega}(s,a,s^{\prime})\leftarrow\log(D_{\theta_{t},\omega}(s,a,s^{\prime}))-\log(1-D_{\theta_{t},\omega}(s,a,s^{\prime}))$

16: $\text{Update
}\pi_{\omega,\alpha_{t}},\beta_{\omega,\delta_{t}}\forall\omega\in\Omega\text{
and }\pi_{\Omega,\zeta_{t}}\text{ with any policy optimization method (e.g.
PPOC)}$

17: end for

##  7 Convergence Analysis

In this section we explain the gist of the analysis of convergence of oIRL.
The detailed proofs can be found in Appendix E and F.

We first show that the actual reward function is recovered (up to a constant)
by the reward estimators. We show that for each option’s reward estimator
$g_{\theta,\omega}(s)$, we have $g^{*}_{\omega}(s)=r^{*}(s)+c_{\omega}$, where
$c_{\omega}$ is a finite constant. Using the fact that
$g_{\theta,\omega}(s)\rightarrow g^{*}_{\omega}(s)=r^{*}(s)+c_{\omega}$, and
by using Cauchy-Schwarz inequality of sup-norm, we prove that the update of
the TD-error is a contraction, i.e.,

$\max_{s^{\prime\prime},\omega^{\prime\prime}}|Q_{\pi_{\Omega,t}}(s,\omega)-Q^{*}(s,\omega)|\leq\epsilon+\max_{\omega\in\Omega}c_{\omega}.$
(22)

In order to prove asymptotic convergence to the optimal option-value $Q^{*}$,
we show using the contraction argument that $g_{\theta,\omega}(s)+\gamma
Q(s^{\prime},\omega)$ converges to $Q^{*}$ by establishing the following
inequality:

$\displaystyle|\mathbb{E}[g_{\theta,\omega}(s)]+\gamma\mathbb{E}[Q(s^{\prime},\omega)|s]-Q^{*}(s^{\prime},\omega)|$
$\displaystyle\leq(\max_{\omega\in\Omega}c_{\omega})(\epsilon+\max_{\omega\in\Omega}c_{\omega})\gamma.$
(23)

##  8 Experiments

oIRL learns disentangled reward functions for each option policy, which
facilitates policy generalizability and is instrumental in _transfer
learning_.

Transfer learning can be described as using information learned by solving one
problem and then applying it to a different but related problem. In the RL
sense, it means taking a policy trained on one environment and then using the
policy to solve a similar task in a different previously unseen environment.

We run experiments in different environments to address the following
questions:

  * •

Does learning a policy over options with the AIRL framework improve policy
generalization and reward robustness in transfer learning tasks where the
environmental dynamics are manipulated?

  * •

Can the policy over options framework match or exceed benchmarks for imitation
learning on complex continuous control tasks?

To answer these questions, we compare our model against AIRL (the current
state of the art for transfer learning) in a transfer task by learning in an
_ant_ environment and modifying the physical structure of the ant and compare
our method on various benchmark IRL continuous control tasks. We wish to see
if learning disentangled rewards for sub-tasks through the options framework
is more portable.

We train a policy using each of the baseline methods and our method on these
expert demonstrations for 500 time steps on the _gait_ environments and 500
time steps on the hierarchical ones. Then we take the trained policy (the
parameterized distribution) and use this policy on the transfer environments
and observe the reward obtained. Such a method of transferring the policy is
called a _direct policy transfer_.

###  8.1 Gait Transfer Learning Environments

For the transfer learning tasks we use _Transfer Environments for MuJoCo_ (Chu
& Arnold, 2018), a set of gym environments for studying potential improvements
in transfer learning tasks. The task involves an _Ant_ as an agent which
optimizes a gait to crawl sideways across the landscape. The expert
demonstrations are obtained from the optimal policy in the basic Ant
environment. We disable the agent ant in two ways for two transfer learning
tasks. In _BigAnt_ tasks, the length of all legs is doubled, no extra joints
are added though. The _Amputated Ant_ task modifies the agent by shortening a
single leg to disable it. These transfer tasks require the learning of a true
disentangled reward of walking sideways instead of directly imitating and
learning the reward specific to the gait movements. These manipulations are
shown in Figure 1.

(a) Ant environment

(b) Big Ant environment

(c) Amputated Ant environment

Figure 1: MuJoCo Ant Gait transfer learning task environments. When the ant is
disabled, it must position itself correctly to crawl forward. This requires a
different initial policy than the original environment where the ant must only
crawl sideways.

Table 1 shows the results in terms of reward achieved for the ant gait
transfer tasks. As we can see, in both experiments our algorithm performs
better than AIRL. Remark that the ground truth is obtained with PPO after 2
million iterations (therefore much less sample efficient than IRL).

Table 1: The mean reward obtained (higher is better) over 100 runs for the
Gait transfer learning tasks. We also show the results of PPO optimizing the
ground truth reward.
| Big Ant | Amputated Ant
---|---|---
AIRL (Primitive) | -11.6 | 134.3
2 Options oIRL | 4.7 | 122.6
4 Options oIRL | -1.7 | 167.1
Ground Truth | 142.9 | 335.4

###  8.2 Maze Transfer Learning Tasks

We also create transfer learning environments in a 2D Maze environment with
lava blockades. The goal of the agent is to go through the opening in a row of
lava cells and reach a goal on the other end. For the transfer learning task,
we train the agent on an environment where the "crossing" path requires the
agent to go through the middle for (_LavaCrossing-M_) and then the policy is
directly transferred and used on a GridWorld of the same size where the
crossing is on the right end of the room (_LavaCrossing-R_). An additional
task would be changing a blockade in a _Maze_ (_FlowerMaze-(R,T)_). The two
environments are shown in Figure 2. We can think of two sub-tasks in this
environment, _going to the lava crossing_ and then _going to the goal_.

In all of these environments, the rewards are sparse. The agent receives a
non-zero reward only after completing the mission, and the magnitude of the
reward is $1-0.9\cdot n/n_{\text{max}}$, where $n$ is the length of the
successful episode and $n_{\text{max}}$ is the maximum number of steps that we
allowed for completing the episode, different for each mission.

(a) width=

(b) LavaCrossing-R MiniGrid Env

(c) FlowerMaze-R MiniGrid Env

(d) FlowerMaze-T MiniGrid Env

Figure 2: The MiniGrid transfer learning task set 1. Here the policy is
trained on (a or c) using our method and the baseline methods and then
transferred to be used on environment (b or d). The green cell is the goal.

We show the mean reward after 10 runs using the direct policy transfers on the
environments in Table 2. The 4 option oIRL achieved the highest reward on the
LavaCrossing tasks. The FlowerMaze task was quite difficult with most
algorithms obtaining very low reward. Options still result in a large
improvement.

Table 2: The mean reward obtained (higher is better) over 10 runs for the Maze
transfer learning tasks. We also show the results of PPO optimizing the ground
truth reward.
| LavaCrossing | FlowerMaze
---|---|---
AIRL (Primitive) | 0.64 | 0.11
2 Options oIRL | 0.67 | 0.20
4 Options oIRL | 0.81 | 0.23
Ground Truth | 1.00 | 1.00

###  8.3 Hierarchical Transfer Learning Tasks

In addition, we adopt more complex hierarchical environments that require both
locomotion and object interaction. In the first environment, the ant must
interact with a large movable block. This is called the _Ant-Push_ environment
(Duan et al., 2016). To reach the goal, the ant must complete two successive
processes: first, it must move to the left of the block and then push the
block right, which clears the path towards the target location. There is a
maximum of 500 timesteps. These can be thought of as hierarchical tasks with
_pushing to the left_ , _pushing to the right_ and _going to the goal_ as sub-
goals.

We also utilize an Ant-Maze environment (Florensa et al., 2017) where we have
a simple maze with a goal at the end. The agent receives a reward of $+1$ if
it reaches the goal and $0$ elsewhere. The ant must learn to make two turns in
the maze, the first is down the hallway for one step and then a turn towards
the goal. Again, we see hierarchical behavior in this task: we can think of
sub-goals consisting of _learning to exit the first hall of the maze_ , then
_making the turn_ and finally _going down the final hall towards the goal_.
The two complex environments are shown in Figure 3.

(a) Ant-Maze environment

(b) Ant-Push environment

Figure 3: MuJoCo Ant Complex Gait transfer learning task environments. We
perform these transfer learning tasks with the Big Ant and the Amputated Ant.

Table 3 shows that oIRL performs better than AIRL in all of the complex
hierarchical transfer tasks. In some tasks such as the Maze environment, AIRL
fails to have any or very few successful runs while our method achieves
reasonably high reward. In the BigAnt push task, AIRL achieves only very
minimal reward where oIRL succeeds to perform the task in some cases.

Table 3: The mean reward obtained (higher is better) over 100 runs for the
MuJoCo Ant Complex Gait transfer learning tasks. We also show the results of
PPO optimizing the ground truth reward.
| Big Ant Maze | Amputated Ant Maze | Big Ant Push | Amputated Ant Push
---|---|---|---|---
AIRL (Primitive) | 0.28 | 0.14 | 0.02 | 0.17
2 Options oIRL | 0.62 | 0.29 | 0.46 | 0.34
4 Options oIRL | 0.55 | 0.31 | 0.55 | 0.41
Ground Truth | 0.96 | 0.98 | 0.90 | 0.86

(a) Ant

(b) Half Cheetah

(c) Walker

Figure 4: MuJoCo Continuous control locomotion tasks showing the mean reward
(higher is better) achieved over 500 iterations of the benchmark algorithms
for 10 random seeds. The shaded area represents the standard deviation.

###  8.4 MuJoCo Continuous Control Benchmarks

We also test our algorithm on a number of robotic continuous control benchmark
tasks. These tasks do not involve transfer.

We show the plots of the average reward for each iteration during training in
Figure 4. Achieving a higher reward in fewer iterations is better for these
experiments. We examine the Ant, the Half Cheetah, the and Walker MuJoCo
gait/locomotion tasks. We run these experiments with 10 random seeds. The
results are quite similar between the benchmarks. Using a policy over options
shows reasonable improvements in each task.

##  9 Discussion

This work presents Option-Inverse Reinforcement Learning (oIRL), the first
hierarchical IRL algorithm with disentangled rewards. We validate oIRL on a
wide variety of tasks, including transfer learning tasks, locomotion tasks,
complex hierarchical transfer RL environments and GridWorld transfer
navigation tasks and compare our results with the state-of-the-art algorithm.
Combining options with a disentangled IRL framework results in highly portable
policies. Our empirical studies show clear and significant improvements for
transfer learning. The algorithm is also shown to perform well in continuous
control benchmark tasks.

For future work, we wish to test other sampling methods (e.g., Markov-chain
Monte Carlo) to estimate the implicit discriminator-generator pair’s
distribution in our GAN, such as Metropolis-Hastings GAN (Turner et al.,
2019). We also wish to investigate methods to reduce the computational
complexity for the step of computing the recursive loss function, which
requires simulating some short trajectories, lowering the variance. Analyzing
our algorithm using physical robotic tests for tasks that require multiple
sub-tasks would be an interesting future course of research.

## References

  * Abbeel & Ng (2004) Abbeel, P. and Ng, A. Y.  Apprenticeship learning via inverse reinforcement learning.  _in ICML_ , 2004.
  * Amodei et al. (2016) Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D.  Concrete problems in ai safety.  _arXiv preprint arXiv:1606.06565_ , 2016.
  * Bacon et al. (2017) Bacon, P.-L., Harb, J., and Precup, D.  The option-critic architecture.  _in AAAI_ , 2017.
  * Brockman et al. (2016) Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W.  Openai gym, 2016.
  * Chevalier-Boisvert et al. (2018) Chevalier-Boisvert, M., Willems, L., and Pal, S.  Minimalistic gridworld environment for openai gym.  https://github.com/maximecb/gym-minigrid, 2018.
  * Chu & Arnold (2018) Chu, E. and Arnold, S.  Transfer environments for mujoco.  GitHub, 2018.  URL https://github.com/seba-1511/shapechanger.
  * Duan et al. (2016) Duan, Y., Chen, X., Houthooft, R., Schulman, J., and Abbeel, P.  Benchmarking deep reinforcement learning for continuous control, 2016\.
  * Finn et al. (2016a) Finn, C., Christiano, P., Abbeel, P., and Levine, S.  A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models.  _in NeurIPS_ , 2016a.
  * Finn et al. (2016b) Finn, C., Levine, S., and Abbeel, P.  Guided cost learning: Deep inverse optimal control via policy optimization.  _in ICML_ , 2016b.
  * Florensa et al. (2017) Florensa, C., Held, D., Wulfmeier, M., Zhang, M., and Abbeel, P.  Reverse curriculum generation for reinforcement learning.  _in CoRL_ , 2017.
  * Fu et al. (2018) Fu, J., Luo, K., and Levine, S.  Learning robust rewards with adversarial inverse reinforcement learning.  _in ICLR_ , 2018.
  * Goodfellow et al. (2014) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y.  Generative adversarial nets.  _In NeurIPS_ , 2014.
  * Hausman et al. (2017) Hausman, K., Chebotar, Y., Schaal, S., Sukhatme, G., and Lim, J. J.  Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets.  _in NeurIPS_ , 2017.
  * Henderson et al. (2018) Henderson, P., Chang, W.-D., Bacon, P.-L., Meger, D., Pineau, J., and Precup, D.  Optiongan: Learning joint reward-policy options using generative adversarial inverse reinforcement learning.  _in AAAI_ , 2018.
  * Ho & Ermon (2016) Ho, J. and Ermon, S.  Generative adversarial imitation learning.  _in NeurIPS_ , 2016.
  * Klissarov et al. (2017) Klissarov, M., Bacon, P., Harb, J., and Precup, D.  Learnings options end-to-end for continuous action tasks.  _CoRR_ , abs/1712.00004, 2017.
  * Krishnan et al. (2016) Krishnan, S., Garg, A., Liaw, R., Miller, L., Pokorny, F. T., and Goldberg, K. Y.  Hirl: Hierarchical inverse reinforcement learning for long-horizon tasks with delayed rewards.  _ArXiv_ , abs/1604.06508, 2016.
  * Kullback & Leibler (1951) Kullback, S. and Leibler, R. A.  On information and sufficiency.  _The Annals of Mathematical Statistics_ , 22(1):79–86, 1951.  ISSN 00034851.
  * Ng & Russell (2000) Ng, A. Y. and Russell, S.  Algorithms for inverse reinforcement learning.  _in ICML_ , 2000.
  * Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Sharma et al. (2019) Sharma, M., Sharma, A., Rhinehart, N., and Kitani, K. M.  Directed-info GAIL: Learning hierarchical policies from unsegmented demonstrations using directed information.  _in ICLR_ , 2019.
  * Sutton et al. (1999) Sutton, R., Precup, D., and Singh, S.  Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning.  _Artificial Intelligence_ , 1999.
  * Taylor & Stone (2009) Taylor, M. E. and Stone, P.  Transfer learning for reinforcement learning domains: A survey.  _J. Mach. Learn. Res._ , 2009.
  * (24) Todorov, E., Erez, T., and Tassa, Y.  Mujoco: A physics engine for model-based control.  In _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_.
  * Turner et al. (2019) Turner, R., Hung, J., Frank, E., Saatchi, Y., and Yosinski, J.  Metropolis-hastings generative adversarial networks.  _ICML_ , 2019.
  * Wang et al. (2017) Wang, Z., Merel, J., Reed, S., Wayne, G., de Freitas, N., and Heess, N.  Robust imitation of diverse behaviors.  _in NeurIPS_ , 2017.

##  Appendix A Option Transition Probabilities

It is useful to redefine transition probabilities in terms of options. Since
at each step we have a additional consideration, we can continue following the
policy of the current option we are in or terminate the option with some
probability, sample a new option and follow that option’s policy from a
stochastic policy dependent on states. We have

$\begin{split}P(s_{t+1},\omega_{t+1}|s_{t},\omega_{t})&=\sum_{a\in\mathcal{A}}\pi_{\omega,\alpha}(a|s_{t})P(s_{t+1}|s_{t},a)((1-\beta_{\omega_{t},\delta}(s_{t+1}))\textbf{1}_{\omega_{t}=\omega_{t+1}}+\\\
&\beta_{\omega_{t},\delta}(s_{t+1})\pi_{\Omega,\zeta}(\omega_{t+1}|s_{t+1}))\end{split}$
(24)
$\begin{split}P(s_{t+1},\omega_{t+1}|s_{t})&=\sum_{\omega\in\Omega}\pi_{\zeta,\Omega}(\omega|s_{t})\sum_{a\in\mathcal{A}}\pi_{\omega,\alpha}(a|s_{t})P(s_{t+1}|s_{t},a)((1-\beta_{\omega_{t},\delta}(s_{t+1}))\textbf{1}_{\omega_{t}=\omega_{t+1}}+\\\
&\beta_{\omega_{t},\delta}(s_{t+1})\pi_{\Omega,\zeta}(\omega_{t+1}|s_{t+1}))\end{split}$
(25)
$\begin{split}P(s_{t+1},w_{t+1}|s_{t},\omega_{t},a_{t})&=P(s_{t+1}|s_{t},a_{t})((1-\beta_{\omega_{t},\delta}(s_{t+1}))\textbf{1}_{\omega_{t}=\omega_{t+1}}+\\\
&\beta_{\omega_{t},\delta}(s_{t+1})\pi_{\Omega,\zeta}(\omega_{t+1}|s_{t+1}))\end{split}$
(26)

##  Appendix B MLE Objective for IRL Over Options

We can define a discounted return recursively for a policy over options in a
similar manner to the transition probabilities. Consider the policy over
options based on the probabilities of terminating or continuing the option
policies given a reward approximator $\hat{r}_{\theta}(s,a)$ for the state-
action reward.

$\displaystyle
R^{\Omega}_{\zeta,\theta,\alpha,\delta}(s)=\operatorname{\mathbb{E}}_{\omega\in\Omega}[R_{\theta,\alpha,\delta}(s,\omega)]$
(27) $\displaystyle
R_{\theta,\alpha,\delta}(s,\omega)=\operatorname{\mathbb{E}}_{a\in
A}[R_{\theta,\delta}(s,\omega,a)]$ $\displaystyle
R^{\Omega}_{\zeta,\theta,\delta}(s,a)=\operatorname{\mathbb{E}}_{\omega\in\Omega}[R_{\theta,\delta}(s,\omega,a)]$
$\displaystyle
R_{\theta,\delta}(s,\omega,a)=\operatorname{\mathbb{E}}\biggl{[}\hat{r}_{\omega,\theta}(s,a)+$
$\displaystyle\gamma\sum_{s^{{}^{\prime}}\in
S}P(s^{{}^{\prime}}|s,a,\omega)(\beta_{\omega,\delta}(s^{{}^{\prime}})R^{\Omega}_{\zeta,\theta,\alpha,\delta}(s^{{}^{\prime}})+(1-\beta_{\omega,\delta}(s^{{}^{\prime}}))R_{\theta,\alpha,\delta}(s^{{}^{\prime}},\omega))\biggr{]},$

These formulations of the reward function account for option transition
probabilities, including the probability of terminating the current option and
therefore selecting a new one according to the policy over options.

With $\omega_{0}$ selected according to $\pi_{\zeta,\Omega}(\omega|s)$, we can
define a parameterization of the discounted return $R$ in the style of a
maximum causal entropy RL problem with objective
$\max_{\theta}\operatorname{\mathbb{E}}_{\tau\sim\mathcal{D}}[\log(p_{\theta}(\tau))]$,
where

$p_{\theta}(\tau)\sim
p(s_{0},\omega_{0})\prod_{t=0}^{T-1}P(s_{t+1},\omega_{t+1}|s_{t},\omega_{t},a_{t})e^{R_{\theta,\delta}(s_{t},\omega_{t},a_{t})}.$
(28)

MLE Derivative

We can write out our MLE objective for our generator. We may or may not know
the option trajectories in our expert demonstrations, but they are estimated
below according to the policy over options. This is defined similarly in (Fu
et al., 2018) and (Finn et al., 2016a) as
$J(\theta)=\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}[\sum_{t=0}^{T}R^{\Omega}_{\zeta,\theta,\delta}(s_{t},a_{t})]-\operatorname{\mathbb{E}}_{p_{\theta}}[\sum_{t=0}^{T}\sum_{\omega\in\Omega}\pi_{\zeta,\Omega}(\omega|s_{t})R_{\theta,\delta}(s_{t},\omega,a_{t})]$.
The full derivation is shown as (with generator $p_{\theta}$):

$\displaystyle
J(\theta)=\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\left[\log(p_{\theta}(\tau))\right]$
(29)
$\displaystyle=\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\left[R_{\theta,\delta}(s_{t},\omega,a_{t})\right]-\log(Z_{\theta})$
$\displaystyle=\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\left[\sum_{t=0}^{T}\sum_{\omega\in\Omega}\pi_{\zeta,\Omega}(\omega|s_{t})R_{\theta}(s_{t},\omega,a_{t})\right]-\log(Z_{\theta})$
$\displaystyle\approx\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\left[\sum_{t=0}^{T}\sum_{\omega\in\Omega}\pi_{\zeta,\Omega}(\omega|s_{t})R_{\theta,\delta}(s_{t},\omega,a_{t})\right]-\operatorname{\mathbb{E}}_{p_{\theta}}\left[\sum_{t=0}^{T}\sum_{\omega\in\Omega}\pi_{\zeta,\Omega}(\omega|s_{t})R_{\theta,\delta}(s_{t},\omega,a_{t})\right]$
$\displaystyle=\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\left[\sum_{t=0}^{T}R^{\Omega}_{\zeta,\theta,\delta}(s_{t},a_{t})\right]-\operatorname{\mathbb{E}}_{p_{\theta}}\left[\sum_{t=0}^{T}R^{\Omega}_{\zeta,\theta,\delta}(s_{t},a_{t})\right]$

We go from Line 4 to 5 seeing
$R^{\Omega}_{\zeta,\theta,\delta}(s_{t},a_{t})=\sum_{\omega\in\Omega}\pi_{\zeta,\Omega}(\omega|s_{t})R_{\theta,\delta}(s_{t},\omega,a_{t})$.

Now, we take the gradient of the MLE objective w.r.t $\theta$ yields,

$\displaystyle\frac{\partial}{\partial\theta}J(\theta)=\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\biggl{[}\frac{\partial}{\partial\theta}\log(p_{\theta}(\tau))\biggr{]}$
(30)
$\displaystyle\frac{\partial}{\partial\theta}J(\theta)=\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\biggl{[}\sum_{t=0}^{T}\sum_{\omega\in\Omega}\pi_{\zeta,\Omega}(\omega|s_{t})\frac{\partial}{\partial\theta}R_{\theta,\delta}(s_{t},\omega,a_{t})\biggr{]}-\frac{\partial}{\partial\theta}\log(Z_{\theta})$
$\displaystyle\approx\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\biggl{[}\sum_{t=0}^{T}\frac{\partial}{\partial\theta}R^{\Omega}_{\zeta,\theta,\delta}(s_{t},a_{t})\biggr{]}-\operatorname{\mathbb{E}}_{p_{\theta}}\biggl{[}\sum_{t=0}^{T}\frac{\partial}{\partial\theta}R^{\Omega}_{\zeta,\theta,\delta}(s_{t},a_{t})\biggr{]}$

Remark we define $p_{\theta,t}(s_{t},a_{t})=\int_{s_{t^{{}^{\prime}}\neq
t},a_{t^{{}^{\prime}}\neq t}}p_{\theta}(\tau)ds_{t^{\prime}}da_{t^{\prime}}$
as the state action marginal at time $t$.

$\frac{\partial}{\partial\theta}J(\theta)=\sum_{t=0}^{T}\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\biggl{[}\frac{\partial}{\partial\theta}R^{\Omega}_{\zeta,\theta,\delta}(s_{t},a_{t})\biggr{]}-\operatorname{\mathbb{E}}_{p_{\theta,t}}\biggl{[}\frac{\partial}{\partial\theta}R^{\Omega}_{\zeta,\theta,\delta}(s_{t},a_{t})\biggr{]}$
(31)

We perform importance sampling over the hard to estimate generator density. We
make an importance sampling distribution $\mu_{t,w}(\tau)$ for option $w$.

We sample a mixture policy $\mu_{\omega}(a|s)$ defined as
$\frac{1}{2}\pi_{\omega}(a|s)+\frac{1}{2}\hat{p}_{\omega}(a|s)$ and
$\hat{p}_{\omega}(a|s)$ is a rough density estimate trained on the
demonstrations. We wish to minimize the
$D_{KL}(\pi_{w}(\tau)|p_{\omega}(\tau))$. KL refers to the Kullback–Leibler
divergence metric between two probability distributions. Our new gradient is:

$\frac{\partial}{\partial\theta}J(\theta)=\sum_{t=0}^{T}\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}[\frac{\partial}{\partial\theta}R^{\Omega}_{\zeta,\theta,\delta}(s_{t},a_{t})]-\operatorname{\mathbb{E}}_{\mu_{t}}\biggl{[}\sum_{\omega\in\Omega}\pi_{\zeta,\Omega}(\omega|s_{t})\frac{p_{\theta,t,\omega}(s_{t},a_{t})}{\mu_{t,w}(s_{t},a_{t})}\frac{\partial}{\partial\theta}R_{\theta,\delta}(s_{t},\omega,a_{t})\biggr{]}.$
(32)

Taking the derivative of the discounted option return results in

$\displaystyle\frac{\partial}{\partial\theta}R^{\Omega}_{\zeta,\theta,\alpha,\delta}(s)=\operatorname{\mathbb{E}}\biggl{[}\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s)[\sum_{a\in
A}[\pi_{w,\alpha}(a|s)\biggl{(}\frac{\partial}{\partial\theta}\hat{r}_{\omega,\theta}(s,a)$
(33) $\displaystyle+\gamma$ $\displaystyle\sum_{s^{{}^{\prime}}\in
S}P(s^{{}^{\prime}}|s,a)(\beta_{\omega,\delta}(s^{{}^{\prime}})\frac{\partial}{\partial\theta}R^{\Omega}_{\zeta,\theta,\alpha,\delta}(s^{{}^{\prime}})+(1-\beta_{\omega,\delta}(s^{{}^{\prime}}))\frac{\partial}{\partial\theta}R_{\theta,\alpha,\delta}(s^{{}^{\prime}},\omega))\biggr{)}]\biggr{]}.$

$\displaystyle\frac{\partial}{\partial\theta}R^{\Omega}_{\zeta,\theta,\delta}(s,a)=\operatorname{\mathbb{E}}\biggl{[}\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s)\biggl{(}\frac{\partial}{\partial\theta}\hat{r}_{\omega,\theta}(s,a)$
(34) $\displaystyle+\gamma$ $\displaystyle\sum_{s^{{}^{\prime}}\in
S}P(s^{{}^{\prime}}|s,a)(\beta_{\omega,\delta}(s^{{}^{\prime}})\frac{\partial}{\partial\theta}R^{\Omega}_{\zeta,\theta,\alpha,\delta}(s^{{}^{\prime}})+(1-\beta_{\omega,\delta}(s^{{}^{\prime}}))\frac{\partial}{\partial\theta}R_{\theta,\alpha,\delta}(s^{{}^{\prime}},\omega))\biggr{)}\biggr{]}$

##  Appendix C Discriminator Objective

We formulate the discriminator as the odds ratio between the policy and the
exponentiated reward distribution for option $\omega$ as in AIRL parameterized
by $\theta$. We have a discriminator for each option $\omega$ and generator
option policy $\pi_{w}$,

$D_{\theta,\omega}(s,a)=\frac{\exp(f_{\theta,\omega}(s,a))}{\exp(f_{\theta,\omega}(s,a))+\pi_{w}(a|s)}.$
(35)

###  C.1 Recursive Loss Formulation

We minimize the cross-entropy loss between expert demonstrations and generated
examples assuming we have the same number of options in the generated and
expert trajectories. We define the loss function $L_{\theta}$ as follows:

$\displaystyle
L_{\theta}(s,a,\omega)=-\operatorname{\mathbb{E}}_{\mathcal{D}}[\log(D_{\theta,\omega}(s,a))]-\operatorname{\mathbb{E}}_{\pi_{\Theta,t}}[\log(1-D_{\theta,\omega}(s,a))].$
(36)

The total loss for the entire trajectory can be expressed recursively as
follows by taking expectations over the next options or states:

$\displaystyle
L_{\theta,\delta}(s,a,\omega)=l_{\theta}(s,a,\omega)+\gamma\sum_{s^{{}^{\prime}}\in
S}P(s^{{}^{\prime}}|s,a)(\beta_{w,\delta}(s^{{}^{\prime}})L^{\Omega}_{\zeta,\theta,\alpha,\delta}(s^{{}^{\prime}})+(1-\beta_{w,\delta}(s^{{}^{\prime}}))L_{\theta,\alpha,\delta}(s^{{}^{\prime}},w))$
(37) $\displaystyle
L_{\theta,\alpha,\delta}(s,w)=\operatorname{\mathbb{E}}_{a\in
A}[[L_{\theta,\delta}(s,w,a)]$ $\displaystyle
L^{\Omega}_{\zeta,\theta,\delta}(s,a)=\operatorname{\mathbb{E}}_{w\in\Omega}[L_{\theta,\delta}(s,w,a)]$
$\displaystyle
L^{\Omega}_{\zeta,\theta,\alpha,\delta}(s)=\operatorname{\mathbb{E}}_{\omega\in\Omega}[L_{\theta,\alpha,\delta}(s,\omega)]$

The agent wishes to minimize $L_{\theta,\delta}$ to find its optimal policy.

We can let cost function $f_{\theta,w}(s,a)=L_{\theta,\delta}(s,\omega,a)$ as
shown in AIRL and we have:

$D_{\theta,\omega}=\frac{\exp(L_{\theta,\delta}(s,\omega,a))}{\exp(L_{\theta,\delta}(s,\omega,a))+\pi_{\omega}(a|s)}$
(38)

###  C.2 Optimization Criteria

For a given option $\omega$, we can write the reward function
$\hat{R}_{\theta,\delta}(s,\omega,a)$ to be maximised, as follows. Note that
$\theta$ parameterizes the state-action reward function estimate for option
$\omega$. $-L^{D}$ is the negative discriminator loss. We therefore turn our
minimization problem into a maximization problem. We define our objective
similar to the GAN objective from AIRL:

$\displaystyle-L^{D}=\hat{R}_{\theta,\delta}(s,\omega,a)=\log\left(D_{\theta,\omega}(s,a)\right)-\log\left(1-D_{\theta,\omega}(s,a)\right)$
(39)

Now we can write out our reward function in terms of the optimal discriminator

$\displaystyle\hat{R}_{\theta,\delta}(s,\omega,a)=\log\left(\frac{\exp\left(-L_{\theta,\delta}(s,\omega,a)\right)}{\exp\left(-L_{\theta,\delta}(s,\omega,a)\right)+\pi_{\omega}(a|s)}\right)-\log\left(\frac{\pi_{\omega}(a|s)}{\exp\left(-L_{\theta,\delta}(s,\omega,a)\right)+\pi_{\omega}(a|s)}\right)$
(40) $\displaystyle=-L_{\theta,\delta}(s,\omega,a)-\log(\pi_{\omega}(a|s))$

The derivative of this reward function can now be computed as follows:

$\displaystyle\frac{\partial}{\partial\theta}\hat{R}_{\theta,\delta}(s,\omega,a)\approx\frac{\partial}{\partial\theta}-L_{\theta,\delta}(s,\omega,a)$
(41)
$\displaystyle=\operatorname{\mathbb{E}}\left[\frac{\partial}{\partial\theta}r_{\omega,\theta}(s,a)+\gamma\sum_{s^{{}^{\prime}}\in
S}P(s^{{}^{\prime}}|s,a)\left(\beta_{\omega,\delta}(s^{{}^{\prime}})\frac{\partial}{\partial\theta}-L^{\Omega}_{\zeta,\theta,\alpha,\delta}(s^{{}^{\prime}})+(1-\beta_{\omega,\delta}(s^{{}^{\prime}}))\frac{\partial}{\partial\theta}-L_{\theta,\alpha,\delta}(s^{{}^{\prime}},\omega)\right)\right]$
$\displaystyle=\operatorname{\mathbb{E}}\left[\frac{\partial}{\partial\theta}r_{\omega,\theta}(s,a)\right]+\operatorname{\mathbb{E}}\biggr{[}\gamma\sum_{s^{{}^{\prime}}\in
S}P(s^{{}^{\prime}}|s,a)\biggl{(}\beta_{\omega,\delta}(s^{{}^{\prime}})\frac{\partial}{\partial\theta}-L^{\Omega}_{\zeta,\theta,\alpha,\delta}(s^{{}^{\prime}})+$
$\displaystyle(1-\beta_{\omega,\delta}(s^{{}^{\prime}}))\frac{\partial}{\partial\theta}-L_{\theta,\alpha,\delta}(s^{{}^{\prime}},\omega)\biggr{)}\biggr{]}$

Writing out our discriminator objective yields:

$\displaystyle-L^{D}=\sum_{t=0}^{T}\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\bigg{(}\Big{[}\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s_{t})\log(D_{\theta,\omega}(s_{t},a_{t}))\Big{]}+$
(42)
$\displaystyle\operatorname{\mathbb{E}}_{\pi_{t}}\Big{[}\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s_{t})\log(1-D_{\theta,\omega}(s_{t},a_{t}))\Big{]}\bigg{)}$
$\displaystyle=\sum_{t=0}^{T}\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\left[\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s_{t})\log\left(\frac{\exp(-L_{\theta,\delta}(s_{t},\omega,a_{t}))}{\exp(-L_{\theta,\delta}(s_{t},\omega,a_{t}))+\pi_{\omega}(a_{t}|s_{t})}\right)\right]$
$\displaystyle+\operatorname{\mathbb{E}}_{\pi_{t}}\left[\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s_{t})\log\left(\frac{\pi_{\omega}(a_{t}|s_{t})}{\exp(-L_{\theta,\delta}(s_{t},\omega,a_{t}))+\pi_{\omega}(a_{t}|s_{t})}\right)\right]$
$\displaystyle=\sum_{t=0}^{T}\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\left[\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s_{t})-L_{\theta,\delta}(s_{t},\omega,a_{t})\right]-$
$\displaystyle\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\left[\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(w|s_{t})\log(\exp(-L_{\theta,\delta}(s_{t},w,a_{t}))+\pi_{\omega}(a|s_{t}))\right]$
$\displaystyle+\operatorname{\mathbb{E}}_{\pi_{t}}\left[\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s_{t})\log(\pi_{\omega}(a_{t}|s_{t}))\right]-$
$\displaystyle\operatorname{\mathbb{E}}_{\pi_{t}}\left[\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s_{t})\log(\exp(-L_{\theta,\delta}(s_{t},\omega,a_{t}))+\pi_{\omega}(a_{t}|s_{t}))\right]$

We set a mixture of experts and novice as $\bar{\mu}$ observations.

$\displaystyle=\sum_{t=0}^{T}\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\left[\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s_{t})-L_{\theta,\delta}(s_{t},\omega,a_{t})\right]+\operatorname{\mathbb{E}}_{\pi_{t}}\left[\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s_{t})\log(\pi_{\omega}(a_{t}|s_{t}))\right]$
(43)
$\displaystyle-2\operatorname{\mathbb{E}}_{\bar{\mu}_{t}}\left[\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s_{t})\log\left(\exp(-L_{\theta,\delta}(s_{t},\omega,a_{t}))+\pi_{\omega}(a_{t}|s_{t})\right)\right]$

We can take the derivative w.r.t $\theta$ (state-action reward function
estimate parameter):

$\displaystyle\frac{\partial}{\partial\theta}(-L^{D})=\sum_{t=0}^{T}\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\left[\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s_{t})\frac{\partial}{\partial\theta}-L_{\theta,\delta}(s_{t},\omega,a_{t})\right]$
(44)
$\displaystyle-\operatorname{\mathbb{E}}_{\bar{\mu}_{t}}\left[\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s_{t})\left(\frac{\exp(-L_{\theta,\delta}(s_{t},\omega,a_{t}))}{\frac{1}{2}\exp(-L_{\theta,\delta}(s_{t},\omega,a_{t}))+\frac{1}{2}\pi_{\omega}(a_{t}|s_{t})}\right)\frac{\partial}{\partial\theta}-L_{\theta,\delta}(s_{t},\omega,a_{t})\right]$

We can multiply the top and bottom of the fraction in the mixture expectation
by the state marginal $\pi_{\omega}(s_{t})=\int_{a\in
A}\pi_{\omega}(s_{t},a_{t})$. This allows us to write
$\hat{p}_{\theta,t,\omega}(s_{t},a_{t})=\exp(L_{\theta,\delta}(s_{t},\omega,a_{t}))\pi_{\omega,t}(s_{t})$.
Now we have an importance sampling.

$\displaystyle\frac{\partial}{\partial\theta}(-L^{D})=\sum_{t=0}^{T}\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\left[\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s_{t})\frac{\partial}{\partial\theta}-L_{\theta,\delta}(s_{t},\omega,a_{t})\right]$
(45)
$\displaystyle-\operatorname{\mathbb{E}}_{\bar{\mu}_{t}}\left[\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s_{t})\left(\frac{\hat{p}_{\theta,t,\omega}(s_{t},a_{t})}{\hat{\mu}_{t,\omega}(s_{t},a_{t})}\right)\frac{\partial}{\partial\theta}-L_{\theta,\delta}(s_{t},\omega,a_{t})\right]$

It is now easy to see we have the same form as our MLE objective loss
function, our loss (the function we approximate with the GAN) is the
discounted reward for a state action pair with the expectation over options.
We change the loss functions to reward functions to show this, as they are
defined equivalently.

$\displaystyle\frac{\partial}{\partial\theta}(-L^{D})=\sum_{t=0}^{T}\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\left[\frac{\partial}{\partial\theta}R_{\zeta,\theta,\delta}(s_{t},a_{t})\right]$
(46)
$\displaystyle-\operatorname{\mathbb{E}}_{\bar{\mu}_{t}}\left[\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s)\left(\frac{\hat{p}_{\theta,t,\omega}(s_{t},a_{t})}{\hat{\mu}_{t,\omega}(s_{t},a_{t})}\right)\frac{\partial}{\partial\theta}R_{\theta,\delta}(s_{t},\omega,a_{t})\right]$

In addition, we can decompose the reward into a state-action reward and a
future discounted sum of rewards considering the policy over options as
follows:

$\displaystyle\frac{\partial}{\partial\theta}(-L^{D})=\sum_{t=0}^{T}\underbrace{\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\left[\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s_{t})\frac{\partial}{\partial\theta}r_{\omega,\theta}(s_{t},a_{t})\right]}_{\text{State-
Action Reward}}$ (47)
$\displaystyle+\operatorname{\mathbb{E}}_{\tau\sim\tau^{E}}\biggl{[}\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s_{t})\gamma\sum_{s_{t+1}\in
S}P(s_{t+1}|s_{t},a_{t})(\beta_{\omega,\delta}(s_{t+1})\frac{\partial}{\partial\theta}R^{\Omega}_{\zeta,\theta,\alpha,\delta}(s_{t+1})+$
$\displaystyle(1-\beta_{w,\delta}(s_{t+1}))\frac{\partial}{\partial\theta}R_{\theta,\alpha,\delta}(s_{t+1},\omega))\biggr{]}$
$\displaystyle-\operatorname{\mathbb{E}}_{\bar{\mu}_{t}}\left[\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s_{t})\left(\frac{\hat{p}_{\theta,t,\omega}(s_{t},a_{t})}{\hat{\mu}_{t,\omega}(s_{t},a_{t})}\right)\frac{\partial}{\partial\theta}R_{\theta,\delta}(s_{t},\omega,a_{t})\right]$

##  Appendix D GAN Architecture

The architecture for our GAN-IRL framework is described in Figure 5.

Figure 5: Architecture of GAN-IRL framework.

##  Appendix E Proof of Recoverable Rewards

A substantial amount of this proof is derived from (Fu et al., 2018).

Lemma 1: $f_{\theta,\omega}(s,a)$ recovers the advantage.

Proof: It is known that when $\pi_{\omega}=\pi_{\omega}^{E}$, we have achieved
the global min of the discriminator objective. The discriminator must then
output 0.5 for all state action pairs. This results in
$\text{exp}(f_{\theta,\omega}(s,a))=\pi^{E}_{\omega}(a|s)$. Equivalently we
have $f^{*}_{\omega}(s,a)=\text{log }\pi_{\omega}^{E}(a|s)=A^{*}(s,a,\omega)$.

Definition 1: Decomposability condition. We first define 2 states
$s_{1},s_{2}$ as 1-step linked under dynamics $T(s^{\prime}|s,a)$ if there
exists a state $s$ that can reach $s_{1}$ and $s_{2}$ with non-zero
probability in one timestep. The transitivity property holds for the linked
relationship. We can say that if $s_{1}$ and $s_{2}$ and linked, $s_{2}$ and
$s_{3}$ are linked then $s_{1}$ and $s_{3}$ must also be linked.

The Decomposability condition for transition dynamics $T$ holds if all states
in the MDP are linked with all other states.

Lemma 2: For an MDP, where the decomposability condition holds for all
dynamics. For arbitrary functions $a(s),b(s),c(s),d(s)$, if for all $s$ and
$s^{\prime}$

$a(s)+b(s^{\prime})=c(s)+d(s^{\prime})$ (48)

and for all $s$

$a(s)=c(s)+\text{const}_{s}$ (49) $b(s)=d(s)+\text{const}_{s},$ (50)

where $\text{const}_{s}$ is a constant dependent with respect to state $s$.

Proof: If we rearrange Equation 48, we can obtain the quality
$a(s)-c(s)=b(s^{\prime})-d(s^{\prime})$.

Now we define $f(s)=a(s)-c(s)$. Given our equality, we have
$f(s)=a(s)-c(s)=b(s^{\prime})-d(s^{\prime})$. This holds for some function
dependent on $s$.

To represent this, $b(s^{\prime})-d(s^{\prime})$ must be equal to a constant
(with the constant’s value dependent on the state $s$) for all one-step
successor states $s^{\prime}$ from $s$.

Now, under decomposability, all one step successor states ($s^{\prime}$) from
$s$ must be equal through the transitivity property so
$b(s^{\prime})-d(s^{\prime})$ must be a constant with respect to state $s$.
Therefore, we can write $a(s)=c(s)+\text{const}_{+}s$ for an arbitrary state
$s$ and functions $b$ and $d$.

Substituting this into the Equation 48, we can obtain
$b(s)=d(s)+\text{const}_{s}$. This completes our proof.

Inductive proof for any successor state
Let us consider for any MDP and any arbitrary functions $a(\cdot)$,
$b(\cdot)$, $c(\cdot)$ and $d(\cdot)$,

$a(s)+b(S^{(k)})=c(s)+d(S^{(k)}),$ (51)

where $S^{(k)}$ is the $k$-th successor state reached in $k$ time-steps from
the current state. Let us denote by $T^{\pi,(k)}(s,S^{(k)})$ the probability
of transitioning from state $s$ to $S^{(k)}$ in $k$ steps using policy $\pi$.
Then, we can express $T^{\pi,(k)}(s,S^{(k)})$ recursively as follows:

$T^{\pi,(k)}(s,S^{(k)})=\sum_{s^{\prime}\in\mathcal{S}}T^{\pi,(k-1)}(s,s^{\prime})T^{\pi}(s^{\prime},S^{(k)}),$
(52)

where $T^{\pi}(s^{\prime},S^{(k)})$ is the one-step transition probability
from state $s^{\prime}$ to state $S^{(k)}$ (by definition of the Bellman
operator).

Denote by $P(S^{(k)})$ the probability of landing in state $S^{(k)}$ in $k$
steps from any current state. We can write $P(S^{(k)})$ using (52) as follows:

$P(S^{(k)})\coloneqq\sum_{s\in\mathcal{S}}T^{\pi,(k)}(s,S^{(k)})\mu(s),$ (53)

where $\mu$ is the state-distribution.

The unbiased estimator $\hat{s}^{(k)}$ of an unknown successor state $S^{(k)}$
is given by:

$\hat{s}^{(k)}\coloneqq\mathbb{E}(S^{(k)})=\sum_{s^{(k)}\in\mathcal{S}}s^{(k)}P(S^{(k)}),$
(54)

where $P(S^{(k)})$ is given in (53).

Now, replacing $S^{(k)}$ in (51) with its unbiased estimator $\hat{s}^{(k)}$
as given by (54), we have

$a(s)-c(s)=b(\hat{s}^{(k)})-d(\hat{s}^{(k)})\stackrel{{\scriptstyle(a)}}{{=}}f(k),$
(55)

for some function $f$, where $(a)$ holds since $\hat{s}^{(k)}$ depends only on
$k$. Thus, we get $a(s)=c(s)+\text{const.}$ and $b(s)=d(s)+\text{const.}$
where the constant is with respect to the state $s$.

Theorem 1: Suppose we have, for a MDP where the decomposability condition
holds,

$f_{\theta,\omega}(s,a,s^{\prime})=g_{\omega}(s,a)+\gamma
h_{\Phi}(s^{\prime})-h_{\Phi}(s)$ (56)

where $h_{\Phi}$ is a shaping term. If we obtain the optimal
$f^{*}_{\theta,\omega}(s,a,s^{\prime})$, with a reward approximator
$g^{*}_{\omega}(s,a)$. Under deterministic dynamics the following holds

$g^{*}_{\omega}(s,a)+\gamma
h^{*}_{\Phi}(s^{\prime})-h^{*}_{\Phi}(s)=r^{*}_{\omega}(s)+\gamma
V^{*}_{\Omega}(s^{\prime})-V^{*}_{\Omega}(s)$ (57)

and

$g^{*}_{\omega}(s)=r^{*}_{\omega}(s)+c_{\omega}.$ (58)

Proof: We know
$f^{*}_{\omega}(s,a,s^{\prime})=A^{*}(s,a,\omega)=Q^{*}(s,a,\omega)-V_{\Omega}^{*}(s)=r^{*}_{\omega}(s)+\gamma
V^{*}_{\Omega}(s^{\prime})-V^{*}_{\Omega}(s)$. We can substitute the
definition of $f^{*}_{\omega}(s,a,s^{\prime})$ to obtain our Theorem.

Where
$Q(s,\omega)=\sum_{a\in\mathcal{A}}\pi_{\omega,\alpha}(a|s)[r_{\omega,\theta}(s,a)+\gamma\sum_{s^{\prime}\in\mathcal{S}}P(s^{\prime}|s,a)\left((1-\beta_{\delta,\omega}(s^{\prime}))Q(s^{\prime},\omega)+\beta_{\delta,\omega}(s^{\prime})V_{\Omega}(s^{\prime})\right)]$
and
$V_{\Omega}(s)=\sum_{\omega\in\Omega}\pi_{\Omega,\zeta}(\omega|s)Q(s,\omega)$

$Q(s,a,\omega)=\pi_{\omega,\alpha}(a|s)[r_{\omega,\theta}(s,a)+\gamma\sum_{s^{\prime}\in\mathcal{S}}P(s^{\prime}|s,a)\left((1-\beta_{\delta,\omega}(s^{\prime}))Q(s^{\prime},\omega)+\beta_{\delta,\omega}(s^{\prime})V_{\Omega}(s^{\prime})\right)]$

which holds for all $s$ and $s^{\prime}$. Now we apply Lemma 2. We say that
$a(s)=g^{*}_{\omega}(s)-h^{*}_{\Phi}(s),b(s^{\prime})=\gamma
h^{*}_{\Phi}(s^{\prime}),c(s)=r(s)-V^{*}_{\Omega}(s)$ and
$d(s^{\prime})=\gamma V^{*}_{\Omega}(s^{\prime})$ and rearrange according to
Lemma 2. We therefore have our results that
$g^{*}_{\omega}(s)=r_{\omega}(s)+c_{\omega}$. Where $c_{\omega}$ is a
constant.

##  Appendix F Proof of Convergence

Definition 2: Reward Approximator Error. From Theorem 1, we can see that our
reward approximator $g^{*}_{\omega}(s)=r_{\omega}(s)+c_{\omega}$. We define a
reward approximator error over all options as
$\delta_{r}=\sum_{\omega\in\Omega}\pi_{\Omega}(\omega)|{g}^{*}_{\omega}(s)-r^{*}(s)|$.
This error is bounded by

$\delta_{r}=\sum_{\omega\in\Omega}\pi_{\Omega}(\omega)|g^{*}_{\omega}(s)-r^{*}(s)|\leq\max_{\omega\in\Omega}c_{\omega}$
(59)

By definition of $g^{*}_{\omega}(s)$.

Lemma 3: The Bellman operator for options in the IRL problem is a contraction.

Proof: We prove this by Cauchy-Schwarz and the definition of the sup-norm. We
must define this inequality in terms of the IRL problem where we have a reward
estimator $\hat{g}_{{\theta}_{\omega}}(s)$ under our learned parameter
$\theta$ and an optimal reward estimator $r^{*}(s)$.

$\begin{split}&||Q_{\pi_{\Omega,t}}(s,\omega)-Q^{*}(s,\omega)||_{\infty}\\\
&=||\hat{g}_{\theta}(s)+\gamma\sum_{s^{\prime}\in\mathcal{S}}P(s^{\prime}|s,a)((1-\beta(s^{\prime})Q_{\pi_{\Omega,t}}(s^{\prime},\omega)+\beta(s^{\prime})\max_{\omega\in\Omega}Q_{\pi_{\Omega,t}}(s^{\prime},\omega))-\\\
&r^{*}(s)+\gamma\sum_{s^{\prime}\in\mathcal{S}}P(s^{\prime}|s,a)((1-\beta(s^{\prime})Q^{*}(s^{\prime},\omega)+\beta(s^{\prime})\max_{\omega\in\Omega}Q^{*}(s^{\prime},\omega))||_{\infty}\\\
&=||\hat{g}_{\theta}(s)-r^{*}(s)+\sum_{s^{\prime}\in\mathcal{S}}P(s^{\prime}|s,a)[(1-\beta(s^{\prime}))(Q_{\pi_{\Omega,t}}(s^{\prime},\omega)-Q^{*}(s^{\prime},\omega))]+\\\
&[\beta(s^{\prime})(\max_{\omega\in\Omega}Q_{\pi_{\Omega,t}}(s^{\prime},\omega)-\max_{\omega\in\Omega}Q^{*}(s^{\prime},\omega))]||_{\infty}\\\
&=||\sum_{s^{\prime}\in\mathcal{S}}P(s^{\prime}|s,a)[(1-\beta(s^{\prime}))(Q_{\pi_{\Omega,t}}(s^{\prime},\omega)-Q^{*}(s^{\prime},\omega))]+\\\
&[\beta(s^{\prime})(\max_{\omega\in\Omega}Q_{\pi_{\Omega,t}}(s^{\prime},\omega)-\max_{\omega\in\Omega}Q^{*}(s^{\prime},\omega))]||_{\infty}+\max_{\omega\in\Omega}c_{\omega}\\\
&\leq\sum_{s^{\prime}\in\mathcal{S}}P(s^{\prime}|s,a)\max_{s^{\prime\prime},\omega^{\prime\prime}}||Q_{\pi_{\Omega,t}}(s^{\prime\prime},\omega^{\prime\prime})-Q^{*}(s^{\prime\prime},\omega^{\prime\prime})||_{\infty}+\max_{\omega\in\Omega}c_{\omega}\\\
&\leq\gamma\max_{s^{\prime\prime},\omega^{\prime\prime}}||Q_{\pi_{\Omega,t}}(s^{\prime\prime},\omega^{\prime\prime})-Q^{*}(s^{\prime\prime},\omega^{\prime\prime})||_{\infty}+\max_{\omega\in\Omega}c_{\omega}\\\
\end{split}$ (60)

This is given by Lemma 3 and (Sutton et al., 1999) [Theorem 3].

Giving our results
$\max_{s^{\prime\prime},\omega^{\prime\prime}}|Q_{\pi_{\Omega,t}}(s,\omega)-Q^{*}(s,\omega)|\leq\epsilon+\max_{\omega\in\Omega}c_{\omega}$.
For $\epsilon\in\mathbb{R}_{>0}$

Theorem 2: $g_{\theta}(s)+\gamma Q(s^{\prime},\omega)$ converges to $Q^{*}$.

Proof: We know $g_{\theta}(s)\rightarrow
g^{*}_{\theta}(s)=r^{*}(s)+\text{const}$. Given this we can show by Cauchy-
Schwarz:

$\displaystyle|\mathbb{E}[g_{\theta}(s)]+\gamma\mathbb{E}[Q(s^{\prime},\omega)|s]-Q^{*}(s^{\prime},\omega)|$
$\displaystyle=|\mathbb{E}[g_{\theta}(s)]+\gamma\sum_{s^{\prime}\in\mathcal{S}}P(s^{\prime}|s,a)((1-\beta_{\omega}(s^{\prime}))Q(s^{\prime}\omega)+\beta_{\omega}(s^{\prime})V_{\Omega}(s^{\prime}))$
$\displaystyle-r^{*}(s)-\sum_{s^{\prime}\in\mathcal{S}}P(s^{\prime}|s,a)((1-\beta_{\omega}(s^{\prime})Q^{*}(s^{\prime},\omega))+\beta_{\omega}(s^{\prime})\max_{\omega\in\Omega}Q^{*}(s^{\prime},\omega)|$
$\displaystyle=|\mathbb{E}[g_{\theta}(s)]-r^{*}(s)+\gamma\sum_{s^{\prime}\in\mathcal{S}}P(s^{\prime}|s,a)[\beta_{\omega}(s^{\prime})[\max_{\omega\in\Omega}Q(s^{\prime}\omega)-\max_{\omega\in\Omega}Q^{*}(s^{\prime},\omega)]$
$\displaystyle+(1-\beta_{\omega}(s^{\prime}))[Q(s^{\prime}\omega)-Q^{*}(s^{\prime},\omega)]]|$
$\displaystyle\stackrel{{\scriptstyle(a)}}{{\leq}}(\max_{\omega\in\Omega}c_{\omega})|\gamma\sum_{s^{\prime}\in\mathcal{S}}P(s^{\prime}|s,a)[\max_{s^{\prime\prime},\omega^{\prime\prime}}||Q(s^{\prime\prime},\omega^{\prime\prime})-Q^{*}(s^{\prime\prime},\omega^{\prime\prime})|]|$
$\displaystyle\stackrel{{\scriptstyle(b)}}{{\leq}}(\max_{\omega\in\Omega}c_{\omega})(\epsilon+\max_{\omega\in\Omega}c_{\omega})\gamma\sum_{s^{\prime}\in\mathcal{S}}P(s^{\prime}|s,a)$
$\displaystyle\leq(\max_{\omega\in\Omega}c_{\omega})(\epsilon+\max_{\omega\in\Omega}c_{\omega})\gamma,$
(61)

where $(a)$ follows from Lemma 3 and $(b)$ holds since
$\sum_{s^{\prime}\in\mathcal{S}}P(s^{\prime}|s,a)\leq 1$.

##  Appendix G Parameters for Experiments

###  G.1 MuJoCo Tasks

For these experiments, we use PPO to obtain an optimal policy given our ground
truth rewards for 2 million iterations and 20 million on the complex tasks.
This is used to obtain the expert demonstrations. We sample 50 expert
trajectories. PPOC is used for the policy optimization step for the policy
over options. We tune the deliberation cost hyper-parameter via cross-
validation. The optimal deliberation cost found was $0.1$ for PPOC. We also
use state-only rewards for the policy transfer tasks. The hyperparameters for
our policy optimization are given in Table 4.

Our discriminator is a neural network with the optimal architecture of 2
linear layers of 50 hidden states, each with ReLU activation followed by a
single node linear layer for output. We also tried a variety of hidden states
including 100 and 25 and tanh activation during our hyperparameter
optimization step using cross-validation.

The policy network has 2 layers of 64 hidden states. A batch size of 64 or 32
is used for 1 and any number of options greater than 1 respectively. No mini-
batches are used in the discriminator since the recursive loss must be
computed. There are 2048 timesteps per batch. Generalized Advantage Estimation
is used to compute advantage estimates. We list additional network parameters
in the next section. The output of the policy network gives the Gaussian mean
and the standard deviation. This is the same procedure as in (Schulman et al.,
2017).

Table 4: Policy Optimization parameters for MuJoCo Parameter | Value
---|---
Discr. Adam optimizer learning rate | $1\cdot 10^{-3}$
Adam $\epsilon$ | $1\cdot 10^{-5}$
PPOC Adam optimizer learning rate | $3\cdot 10^{-4}$
GAE $\lambda$ | $0.95$
Entropy coefficient | $10^{-2}$
value loss coefficient | 0.5
discount | 0.99
batch size for PPO | 64 or 32
PPO epochs | 10
entropy coefficient | $10^{-2}$
clip parameter | 0.2

###  G.2 MuJoCo Continuous Control Tasks

In this section, we describe the structure of the objects that gait in the
continuous control benchmarks and the reward functions. For the transfer
learning tasks, we use the same reward function described here for the Ant.

Walker: The walker is a planar biped. There are 7 rigid links comprised of
legs, a torso. This includes 6 actuated joints. This task is particularly
prone to falling. The state space is of 21 dimensions. The observations in the
states include joint angles, joint velocities, the center of mass’s
coordinates. The reward function is $r(s,a)=v_{x}-0.005||a||^{2}_{2}$. The
termination condition occurs when $z_{\text{body}}<0.8,z_{\text{body}}>2.0$ or
$||\theta_{y}||>1.0$.

Half-Cheetah: The half-cheetah is a planar biped also like the Walker. There
are 9 rigid links comprised of 9 actuated joints, a leg and a torso. The state
space is of 20 dimensions. The observations include joint angles, the center
of mass’s coordinates, and joint velocities. The reward function is
$r(s,a)=v_{x}-0.005||a||_{2}^{2}$. There is no termination condition.

Ant: The ant has four legs with 13 rigid links in its structure. The legs have
8 actuated joints. The state space is of 125 dimensions. This includes joint
angles, joint velocities, coordinates of the center of mass, the rotation
matrix for the body, and a vector of contact forces. The function is
$r(s,a)=v_{x}-0.005||a||^{2}_{2}-C_{\text{contact}}+0.05$, where
$C_{\text{contact}}$ is a penalty for contacts to the ground. This is $5\times
10^{-4}||F_{\text{contact}}||_{2}^{2}$. $F_{\text{contact}}$ is the contact
force. It’s values are clipped to be between 0 and 1. The termination
condition occurs $z_{\text{body}}<0.2$ or $z_{\text{body}}>1.0$.

###  G.3 MiniGrid Tasks

For experiments, we used the PPOC algorithm with parallelized data collection
and GAE. 0.1 is the optimal deliberation cost. Each environment is run with 10
random network initialization. As before, in Table 5, we show some of the
policy optimization parameters for MiniGrid Tasks. We rely on an actor-critic
network architecture for these tasks. Since the state space is relatively
large and spatial features are relevant, we use 3 convolutional layers in the
network. The network architecture is detailed in Figure 6. $n$ and $m$ are
defined by the grid dimensions.

The discriminator network is again an neural network with the optimal
architecture of 3 linear layers of 150 hidden states, each with ReLU
activation followed by a single node linear layer for output.

Table 5: Policy optimization parameters for benchmark tasks in MiniGrid Parameter | Value
---|---
Adam optimizer learning rate | $7\cdot 10^{-4}$
Adam $\epsilon$ | $10^{-5}$
entropy coefficient | $10^{-2}$
value loss coefficient | 0.5
discount | 0.99
maximum norm of gradient in PPO | 0.5
number of PPO epochs | 4
batch size for PPO | 256
entropy coefficient | $10^{-2}$
clip parameter | 0.2
Figure 6: Architecture of the actor-critic policies on MiniGrid. Conv is
Convolutional Layer and filter sized is described below. FC is a fully
connected layer.