§ INTRODUCTION

Deep Reinforcement Learning (RL) <cit.> methods have demonstrated impressive performance in continuous control <cit.>, and robotics <cit.>. However, a broader application of these methods in real-world domains is impeded by the challenges in designing a proper reward function <cit.>. Imitation Learning (IL) algorithms <cit.> address this issue by replacing reward functions with expert demonstrations, which are easier to collect in most scenarios. However, despite the success of IL algorithms, they typically impose the somewhat unrealistic requirement that the state-action demonstrations must be collected from the same environment as the one in which the imitator is trained. In this work, we focus on a more realistic setting for imitation learning, where:

* the expert demonstrations collected from the real (deployment) environment by executing an expert policy only contain states,

* the learner is trained in a simulation (training) environment, and does not have access to the real environment during the training phase beyond the batch of demonstrations given, and

* the simulation environment does not model the real environment exactly, i.e., there exists a transition dynamics mismatch between these environments.

The learned policy under the above setting is transferred to the real environment on which its final performance is evaluated. Existing IL methods either do not apply under the above setting or result in poor transfer performance.

-1A large body of work in IL, such as Generative Adversarial Imitation Learning (GAIL <cit.>) and its variants, has focused on the setting with demonstrations that contain both states and actions, which are difficult to obtain for real-world settings such as learning from videos <cit.>. Further, closely following the state-action demonstrations limits the the ability to generalize across environments <cit.>. Training agents in simulation environments not only provides data at low-cost, but also alleviates safety concerns related to the trial-and-error process with real robots. However, building a high-fidelity simulator that perfectly models the real environment would require a large computational budget. Low-fidelity simulations are feasible, due to their speed, but the gap between the simulated and real environments degrades the performance of the policies when transferred to real robots <cit.>. To this end, we consider the following research question: how to train an imitator policy in an offline manner with state-only expert demonstrations and a misspecified simulator such that the policy performs well in the real environment?

Comparison of our method with the existing imitation learning methods that also consider dynamics mismatch. However, the existing methods do not fit under the specific setting that we study. The expert, training, and deployment are denoted by $M^\mathrm{exp}$, $M^\mathrm{tr}$, and $M^\mathrm{dep}$ respectively. The corresponding transition dynamics are denoted by $T^\mathrm{exp}$, $T^\mathrm{tr}$, and $T^\mathrm{dep}$ respectively. Note that the expert demonstrations are collected from $M^\mathrm{exp}$, the imitation learning agent is trained on $M^\mathrm{tr}$, and the trained policy is finally evaluated on $M^\mathrm{dep}$. Our Robust-GAILfO method has minimal access to $M^\mathrm{dep}$ to select an appropriate $\alpha$. Note that our robust GAILfO method is applicable in both: (i) $T^{\mathrm{dep}} = T^{\mathrm{exp}} \neq T^{\mathrm{tr}}$ setting, and (ii) $T^{\mathrm{dep}} \neq T^{\mathrm{exp}} \neq T^{\mathrm{tr}}$ setting. In setting (i), our primary motivation is that accessing the deployment environment is costly, e.g., interacting with a remote deployment environment is costly due to communication constraints. In setting (ii), after the deployment, the agent has to be robust against potential environmental changes during the test time.

IL Methods     Type of Demonstrations     Access to $M^{\mathrm{dep}}$ during training     Dynamics mismatch
GAIL <cit.>     state-action     yes     $T^{\mathrm{exp}} = T^{\mathrm{tr}} = T^{\mathrm{dep}}$

GAILfO <cit.>     state-only     yes     $T^{\mathrm{exp}} = T^{\mathrm{tr}} = T^{\mathrm{dep}}$

AIRL <cit.>     state-action     yes     $T^{\mathrm{exp}} = T^{\mathrm{tr}} \neq T^{\mathrm{dep}}$

I2L <cit.>     state-only     yes     $T^{\mathrm{exp}} \neq T^{\mathrm{tr}} = T^{\mathrm{dep}}$

SAIL <cit.>     state-only     yes     $T^{\mathrm{exp}} \neq T^{\mathrm{tr}} = T^{\mathrm{dep}}$

GARAT <cit.>     state-only     yes     $T^{\mathrm{dep}} = T^{\mathrm{exp}} \neq T^{\mathrm{tr}}$

HIDIL <cit.>     state-action     no     $T^{\mathrm{dep}} = T^{\mathrm{exp}} \neq T^{\mathrm{tr}}$

IDDM <cit.>     state-only     yes     $T^{\mathrm{exp}} = T^{\mathrm{tr}} = T^{\mathrm{dep}}$

ILPO <cit.>     state-only     yes     $T^{\mathrm{exp}} = T^{\mathrm{tr}} =  T^{\mathrm{dep}}$

Robust-GAILfO (ours)     state-only     no     $T^{\mathrm{dep}} = T^{\mathrm{exp}} \neq T^{\mathrm{tr}}$ and $T^{\mathrm{dep}} \neq T^{\mathrm{exp}} \neq T^{\mathrm{tr}}$

-1The Adversarial Inverse Reinforcement Learning (AIRL) method from <cit.> recovers reward functions that can be used to transfer behaviors across changes in dynamics. However, one needs to retrain a policy in the deployment environment with the recovered reward function, whereas we consider a zero-shot transfer setting. In addition, AIRL depends on state-action demonstrations. Recently, <cit.> have studied the imitation learning problem under the transition dynamics mismatch between the expert and the learner environments. However, they do not aim to learn policies that are transferable to the expert (real) environment; instead, they optimize the performance in the learner (simulation) environment. In <cit.>, the authors attempt to match the simulation environment closer to the real environment by interacting with the real environment during the training phase. A setting very close to ours is considered in <cit.>; their method involves learning an inverse dynamics model of the real environment based on the state-action expert demonstrations. None of these methods are directly applicable under our setting (see Table <ref>).

-1We propose a robust IL method for learning robust policies under the above-discussed setting that can be effectively transferred to the real environment without further fine-tuning during deployment. Our method is built upon the robust RL literature <cit.> and the IL literature inspired by GAN-based adversarial learning <cit.>. In particular, our algorithm is a robust variant of the Generative Adversarial Imitation Learning from Observation (GAILfO <cit.>) algorithm, a state-only IL method based on GAIL. We discuss how our method addresses the dynamics mismatch issue by exploiting the equivalence between the robust MDP formulation and the two-player Markov game <cit.>. In the finite MDP setting, <cit.> have proposed a robust inverse reinforcement learning method to address the transition dynamics mismatch between the expert and the learner. Our Markov game formulation in Section <ref> closely follows that of <cit.>, and in Section <ref>, we scale it high-dimensional continuous control setting using the techniques from GAIL literature. On the empirical side, we are interested in the sim-to-real transfer performance, whereas <cit.> have considered the performance in the learner environment itself.

-1We evaluate the efficacy of our method on the continuous control MuJoCo environments. In our experiments, we consider different sources of dynamics mismatch such as joint-friction, and agent-mass. An expert policy is trained under the default dynamics (acting as the real environment). The imitator policy is learned under a modified dynamics (acting as the simulation environment), where one of the mass and friction configurations is changed. The experimental results show that, with appropriate choice of the level of adversarial perturbation, the robustly trained IL policies in the simulator transfer successfully to the real environment compared to the standard GAILfO. We also empirically show that the policies learned by our method are robust to environmental shift during testing.

§ RELATED WORK

Imitation Learning
-1Ho and Ermon <cit.> propose a framework, called Generative Adversarial Imitation Learning (GAIL), for directly extracting a policy from trajectories without recovering a reward function as an intermediate step. GAIL utilizes a discriminator to distinguish between the state-action pairs induced by the expert and the learner policy. GAIL was further extended by Fu et al. <cit.> to produce a scalable inverse reinforcement learning algorithm based on adversarial reward learning. This approach gives a policy as well as a reward function. Our work is closely related to the state-only IL methods that do not require actions in the expert demonstrations <cit.>. Inspired by GAIL, <cit.> have proposed the Generative Adversarial Imitation Learning from Observation (GAILfO) algorithm for state-only IL. GAILfO tries to minimize the divergence between the state transition occupancy measures of the learner and the expert.

Robust Reinforcement Learning
-1In the robust MDP formulation <cit.>, the policy is evaluated by the worst-case performance in a class of MDPs centered around a reference environment. In the context of forward RL, some works build on the robust MDP framework, such as <cit.>. However, our work is closer to the line of work that leverages the equivalence between action-robust and robust MDPs. In <cit.>, the authors have introduced the notion of worst-case disturbance in the $H_\infty$-control literature to the reinforcement learning paradigm. They consider an adversarial game where an adversary tries to make the worst possible disturbance while an agent tries to make the best control input. Recent literature in RL has proposed a range of robust algorithms based on this game-theoretic perspective <cit.>.

§ PROBLEM SETUP AND BACKGROUND

This section formalizes the learning from observation (LfO) problem with model misspecification.

Environment and Policy
The environment is formally represented by a Markov decision process (MDP) $M_c := \br{\mathcal{S},\mathcal{A},T,\gamma,P_0,c}$. The state and action spaces are denoted by $\mathcal{S}$ and $\mathcal{A}$, respectively. $T: \mathcal{S} \times \mathcal{S} \times \mathcal{A} \rightarrow \bs{0,1}$ captures the state transition dynamics, i.e., $T\br{s' \mid s,a}$ denotes the probability of landing in state $s'$ by taking action $a$ from state $s$. Here, $c: \mathcal{S} \times \mathcal{S} \to \mathbb{R}$ is the cost function, $\gamma \in \br{0,1}$ is the discounting factor, and $P_0: \mathcal{S} \rightarrow \bs{0,1}$ is an initial distribution over the state space $\mathcal{S}$. We denote an MDP without a cost function by $M = M_c \backslash c = \bc{\mathcal{S}, \mathcal{A}, T, \gamma, P_0}$. We denote a policy $\pi: \mathcal{S} \times \mathcal{A} \rightarrow \bs{0,1}$ as a mapping from a state to a probability distribution over the action space. The set of all stationary stochastic policies is denoted by $\Pi$. For any policy $\pi$ in the MDP $M$, we define the state transition occupancy measure as follows: $\rho^\pi_M \br{s,s'} := \sum_a T\br{s' \mid s,a} \cdot \pi\br{a \mid s} \cdot \sum_{t=0}^\infty \gamma^t \P{S_t = s \mid \pi, M}$. Here, $\P{S_t = s \mid \pi, M}$ denotes the probability of visiting the state $s$ after $t$ steps by following the policy $\pi$ in $M$. The total expected cost of any policy $\pi$ in the MDP $M_c$ is defined as follows: $\Eee{\rho^\pi_{M}}{c\br{s,s'}} := \E{\sum_{t = 0}^\infty \gamma^t c\br{s_t,s_{t+1}}}$, where $s_0 \sim P_0$, $a_t \sim \pi\br{\cdot | s_t}$, $s_{t+1} \sim T \br{\cdot | s_t, a_t}$. A policy $\pi$ is optimal for the MDP $M_c$ if $\pi \in \argmin_{\pi'} \Eee{\rho^{\pi'}_{M}}{c\br{s,s'}}$, and we denote an optimal policy by $\pi^*_{M_c}$.

Learner and Expert
-1We have two entities: an imitation learner, and an expert. We consider two MDPs, $M^{\mathrm{sim}} = \bc{\mathcal{S}, \mathcal{A}, T^{\mathrm{sim}}, \gamma, P_0}$ and $M^{\mathrm{real}} = \bc{\mathcal{S}, \mathcal{A}, T^{\mathrm{real}}, \gamma, P_0}$, that differ only in the transition dynamics. The true cost function $c^*: \mathcal{S} \times \mathcal{S} \to \mathbb{R}$ is known only to the expert. The learner is trained in the MDP $M^{\mathrm{sim}}$ and is not aware of the true cost function, i.e., it only has access to $M_{c^*}^{\mathrm{sim}} \backslash c^*$. The expert provides demonstrations to the learner  by following the optimal policy $\pi^{*}_{M_{c^*}^{\mathrm{real}}}$ in the expert MDP $M^{\mathrm{real}}$. Typically, in the imitation learning literature, it is assumed that $T^{\mathrm{sim}} = T^{\mathrm{real}}$.  In this work, we consider the setting where there is a transition dynamics mismatch between the learner and the expert, i.e., $T^{\mathrm{sim}} \neq T^{\mathrm{real}}$. The learner tries to recover a policy that closely matches the intention of the expert, based on the occupancy measure $\rho_E \br{s,s'} := \rho^{\pi^*_{M^{\mathrm{real}}_{c^*}}}_{M^{\mathrm{real}}}\br{s,s'}$ (or the demonstrations drawn according to it) received from the expert. The learned policy is evaluated in the expert environment w.r.t. the true cost function, i.e., $M_{c^*}^{\mathrm{real}}$.

Imitation Learning
-1We consider the imitation learner model that matches the expert's state transition occupancy measure $\rho_E$ <cit.>. In particular, the learner policy is obtained via solving the following primal problem:

\begin{align}
\min_{\pi \in \Pi} \quad& - H_{\rho^\pi_{M^{\mathrm{sim}}}}\br{\pi} \label{opt_start}\\
\text{subject to} \quad& \rho^{\pi}_{M^{\mathrm{sim}}}\br{s,s'} ~=~ \rho_E \br{s,s'} , \quad \forall s,s' \in \mathcal{S} ,
\label{opt_end}
\end{align}

where $H_{\rho^\pi_{M^{\mathrm{sim}}}}\br{\pi} :=  \E{\sum_{t = 0}^\infty - \gamma^t \log \pi(a_t | s_t)}$ is the $\gamma$-discounted causal entropy of $\pi$. The corresponding dual problem is given by:

\[
\max_{c \in \mathbb{R}^{\mathcal{S} \times \mathcal{S}}} \br{\min_{\pi \in \Pi} ~ - H_{\rho^\pi_{M^{\mathrm{sim}}}}\br{\pi} + \Eee{\rho^{\pi}_{M^{\mathrm{sim}}}}{c\br{s,s'}}} - \Eee{\rho_E}{c\br{s,s'}} ,
\]

where the costs $c\br{s,s'}$ serve as dual variables for the equality constraints.

Maximum Causal Entropy (MCE) Inverse Reinforcement Learning (IRL)
MCE-IRL algorithm <cit.> involves a two-step procedure. First, it looks for a cost function $c \in \mathcal{C}$ that assigns low cost to the expert policy and high cost to other policies. Then, it learns a policy by solving a certain reinforcement learning problem with the found cost function. Formally, given a convex cost function regularizer[$\overline{\mathbb{R}}$ denotes the extended real numbers $\mathbb{R} \cup \bc{+ \infty}$] $\psi: \mathbb{R}^{\mathcal{S} \times \mathcal{S}} \to \overline{\mathbb{R}}$, first, we recover a cost function $\tilde{c}$ by solving the following $\psi$-regularized problem:

\begin{align*}
\textsc{IRL}_\psi \br{\rho_E} ~=~& \argmax_{c \in \mathbb{R}^{\mathcal{S} \times \mathcal{S}}} ~ - \psi \br{c} - \Eee{\rho_E}{c\br{s,s'}} \\
&\quad \quad \quad \quad + \br{\min_{\pi \in \Pi} ~ - \lambda H_{\rho^\pi_{M^{\mathrm{sim}}}}\br{\pi} + \Eee{\rho^{\pi}_{M^{\mathrm{sim}}}}{c\br{s,s'}}}
\end{align*}

Then, we input the learned cost function $\tilde{c} \in \textsc{IRL}_\psi \br{\rho_E}$ into an entropy-regularized reinforcement learning problem:

\[
\textsc{RL} \br{c} ~=~ \argmin_{\pi \in \Pi} ~ - \lambda H_{\rho^\pi_{M^{\mathrm{sim}}}}\br{\pi} + \Eee{\rho^{\pi}_{M^{\mathrm{sim}}}}{c\br{s,s'}} ,
\]

which aims to find a policy that minimizes the cost function and maximizes the entropy.

Generative Adversarial Imitation Learning from Observation (GAILfO)
Recently, <cit.> have shown that, for a specific choice of the regularizer $\psi$, the two-step procedure $\textsc{RL} \circ \textsc{IRL}_\psi \br{\rho_E}$ of the MCE-IRL algorithm can be reduced to the following optimization problem using GAN discriminator:

\begin{align*}
\min_{\pi \in \Pi} ~ \max_{D \in \br{0,1}^{\mathcal{S} \times \mathcal{S}}} & - \lambda H_{\rho^\pi_{M^{\mathrm{sim}}}}\br{\pi} + \Eee{\rho^{\pi}_{M^{\mathrm{sim}}}}{\log{D\br{s,s'}}} \\
& \quad \quad \quad \quad \quad \quad \quad + \Eee{\rho_E}{\log\br{1-D\br{s,s'}}} ,
\end{align*}

where $D:\mathcal{S} \times \mathcal{S} \to (0,1)$ is a classifier trained to discriminate between the state-next state pairs that arise from the
expert and the imitator. Excluding the entropy term,
the above loss function is similar to the loss of generative
adversarial networks <cit.>. Even though the occupancy measure matching methods were shown to scale well to high-dimensional problems, they are not robust against dynamics mismatch <cit.>.

§ ROBUST LEARNING FROM OBSERVATION VIA MARKOV GAME

§.§ Markov Game

In this section, we focus on recovering a learner policy via imitation learning framework in a robust manner, under the setting described in Section <ref>. To this end, we consider a class of transition matrices such that it contains both $T^\mathrm{sim}$ and $T^\mathrm{real}$. In particular, for a given $\alpha > 0$, we define the class $\mathcal{T}^{\alpha}$ as follows:

\begin{equation}
\mathcal{T}^{\alpha} := \bc{\alpha T^{\mathrm{sim}} (s' | s,a) + \bar{\alpha} \sum_b \pi (b | s) \cdot T^{\mathrm{sim}} (s' | s,b), \forall \pi \in \Pi} ,
\label{learner_unc_set}
\end{equation}

where $\bar{\alpha} = (1 - \alpha)$. We define the corresponding class of MDPs as follows: $\mathcal{M}^{\alpha} := \bc{\bc{\mathcal{S}, \mathcal{A}, T^{\alpha}, \gamma, P_0}, \, \forall T^{\alpha} \in \mathcal{T}^{\alpha}}$. We need to choose $\alpha$ such that $M^\mathrm{real} \in \mathcal{M}^\alpha$.

Our aim is to find a learner policy that performs well in the MDP $M_{c^*}^{\mathrm{real}}$ by using the state-only demonstrations from $\rho_E$, without knowing or interacting with $M^\mathrm{real}$ during training. Thus, we try to learn a robust policy over the class $\mathcal{M}^\alpha$, while aligning with the expert's state transition occupancy measure $\rho_E$, and acting only in $M^\mathrm{sim}$. By doing this, we ensure that the resulting policy performs reasonably well on any MDP $M \in \mathcal{M}^\alpha$ including $M^\mathrm{real}$ w.r.t. the true cost function $c^*$. Based on this idea, we propose the following robust learning from observation (LfO) problem:

\begin{align}
\min_{\pi^{\mathrm{pl}} \in \Pi} \max_{M \in \mathcal{M}^{\alpha}} \quad& - H_{\rho^{\pi^{\mathrm{pl}}}_{M}}\br{\pi^{\mathrm{pl}}} \label{mdp_opt_start}\\
\text{subject to} \quad& \rho^{\pi^{\mathrm{pl}}}_{M}\br{s,s'} ~=~ \rho_E \br{s,s'} , \, \forall s,s' \in \mathcal{S} ,
\label{mdp_opt_end}
\end{align}

where our learner policy matches the expert's state transition occupancy measure $\rho_E$ under the most adversarial MDP belonging to the set $\mathcal{M}^\alpha$. The corresponding dual problem is given by:

\begin{align}
&\max_{c \in \mathbb{R}^{\mathcal{S} \times \mathcal{S}}} \br{\min_{\pi^{\mathrm{pl}} \in \Pi} \max_{M \in \mathcal{M}^{\alpha}} ~ - H_{\rho^{\pi^{\mathrm{pl}}}_{M}}\br{\pi^{\mathrm{pl}}} + \Eee{\rho^{\pi^{\mathrm{pl}}}_{M}}{c\br{s,s'}}} \nonumber \\
& \quad \quad \quad \quad  \quad \quad \quad \quad - \Eee{\rho_E}{c\br{s,s'}} . \label{eq:robust-mdp-il-dual-form}
\end{align}

-1In the dual problem, for any $c$, we attempt to learn a robust policy over the class $\mathcal{M}^{\alpha}$ with respect to the entropy regularized reward function. The parameter $c$ plays the role of aligning the learner's policy with the expert's occupancy measure via constraint satisfaction.

-1For any given $c$, we need to solve the inner min-max problem of (<ref>). However, during training, we only have access to the MDP $M^\mathrm{sim}$. To this end, we utilize the equivalence between the robust MDP <cit.> formulation and the action-robust MDP <cit.> formulation shown in <cit.>. We can interpret the minimization over the environment class as the minimization over a set of opponent policies that with probability $1 - \alpha$ take control of the agent and perform the worst possible move from the current agent state. We can write:

\begin{align}
& \min_{\pi^{\mathrm{pl}} \in \Pi} \max_{M \in \mathcal{M}^{\alpha}} - H_{\rho^{\pi^{\mathrm{pl}}}_{M}}\br{\pi^{\mathrm{pl}}} + \Eee{\rho^{\pi^{\mathrm{pl}}}_{M}}{c\br{s,s'}} \nonumber \\
~=~& \min_{\pi^{\mathrm{pl}} \in \Pi} \max_{T^\alpha \in \mathcal{T}^{\alpha}}  \E{G_c \bigm| \pi^{\mathrm{pl}}, P_0, T^\alpha} \nonumber \\
~=~& \min_{\pi^{\mathrm{pl}} \in \Pi} \max_{\pi^{\mathrm{op}} \in \Pi} \E{G_c \bigm| \alpha \pi^{\mathrm{pl}} + (1 - \alpha) \pi^{\mathrm{op}}, M^{\mathrm{sim}}} , \label{equivalence_new}
\end{align}

where $G_c := \sum_{t=0}^{\infty} \gamma^t \bc{c\br{s_t, s_{t+1}} - H^{\pi^{pl}}(A|S=s_t)}$. The above equality holds due to the derivation in section 3.1 of <cit.>. We can formulate the problem (<ref>) as a two-player zero-sum Markov game <cit.> with transition dynamics given by

\begin{align*}
T^{\mathrm{two},\alpha}(s' | s, a^{\mathrm{pl}}, a^{\mathrm{op}}) ~=~& \alpha T^{\mathrm{sim}}(s' | s, a^{\mathrm{pl}}) + (1 - \alpha) T^{\mathrm{sim}}(s' | s, a^{\mathrm{op}}) ,
\end{align*}

where $a^{\mathrm{pl}}$ is an action chosen according to the player policy and $a^{\mathrm{op}}$ according to the opponent policy. As a result, we reach a two-player Markov game with a regularization term for the player as follows:

\begin{equation}
\argmin_{\pi^{\mathrm{pl}} \in \Pi} \max_{\pi^{\mathrm{op}} \in \Pi} \E{G_{c} \bigm| \pi^{\mathrm{pl}}, \pi^{\mathrm{op}}, M^{\mathrm{two},\alpha}} ,
\label{objective}
\end{equation}

where $M^{\mathrm{two},\alpha} = \bc{\mathcal{S}, \mathcal{A}, \mathcal{A}, T^{\mathrm{two},\alpha}, \gamma, P_0}$ is the two-player MDP associated with the above game.

Robust GAILfO

Input: state-only expert demonstrations $\mathcal{D}^E$, opponent strength parameter $\alpha$.
Initialize: discriminator $D_w$, actor policy $\pi_\theta$, and adversary policy $\pi_\phi$.
$n \in \bc{1,2,\dots, N}$
-1collect trajectories $\tau_i$ by executing the policies $\pi^{\mathrm{pl}}_\theta$ and $ \pi^{\mathrm{op}}_\phi$ (see Algorithm <ref>), and store them in the demonstrations buffer $\mathcal{D}$.
update the discriminator $D_w$ to classify the expert demonstrations $\tau_E \in \mathcal{D}^E$ from the samples $\tau_i \in \mathcal{D}$, i.e., update $w$ via gradient ascent with the following gradient:

\begin{align*}
\widehat{\mathbb{E}}_{\tau_i\in \mathcal{D}}[\nabla_w \log D_w (s, s^\prime)] + \widehat{\mathbb{E}}_{\tau_E \in \mathcal{D}^E}[\nabla_w \log (1 - D_w (s, s^\prime))] .
\end{align*}

update the reward function $R_w(s, s^\prime) \gets - \log D_w(s, s^\prime)$.
compute the following gradient estimates:

\begin{align*}
\widehat{\nabla}_{\theta} J(\theta, \phi) =& \frac{1}{\abs{\mathcal{D}}}  \sum_{\tau_i \in \mathcal{D}} \sum_{t}\gamma^t \nabla_{\theta} \log \pi_{\theta, \phi}^\mathrm{mix}(a^i_t|s^i_t) \bs{G^i_t + \lambda G^{\mathrm{log}, i}_t} \\
\widehat{\nabla}_{\phi} J(\theta, \phi) =& \frac{1}{\abs{\mathcal{D}}} \sum_{\tau_i \in \mathcal{D}} \sum_{t}\gamma^t \nabla_{\phi} \log \pi_{\theta, \phi}^\mathrm{mix}(a^i_t|s^i_t) \bs{G^i_t + \lambda G^{\mathrm{log}, i}_t} ,
\end{align*}

where $G^i_t = \sum^{T}_{k=t+1} \gamma^{k - t - 1}R_w(s^i_k, s^i_{k+1})$ and $G^{\mathrm{log},i}_t = \sum^{T}_{k=t+1} - \gamma^{k - t - 1} H^{\pi^{\mathrm{pl}}_{\theta}}(A|S=s^i_k)$
update the policies $\pi^{\mathrm{pl}}_\theta$ and $\pi^{\mathrm{op}}_\phi$ using PPO with the gradient estimates above.

Collecting Trajectories

Input: total number of trajectories $N_{\mathrm{traj}}$, reward function $R_w$.
$n \in \bc{1,2, \dots, N_{\mathrm{traj}}}$
$t \gets 0$
initialize an empty trajectory $\tau$.
not $\mathrm{done}$
observe state $s_t$.
sample actions $a^{\mathrm{pl}}_t \sim \pi^{\mathrm{pl}}_\theta(\cdot | s_t)$, $a^{\mathrm{op}}_t = \pi^{\mathrm{op}}_\phi(\cdot |s_t)$.
execute $a^{\mathrm{op}}_t$ with probability $\bar{\alpha}$, or $a^{\mathrm{pl}}_t$ with probability $\alpha$.
observe $r_{t+1} = R_w(s_t, s_{t+1})$, next state $s_{t+1}$, and $\mathrm{done}$.
store the tuple $(s_t, a^{\mathrm{pl}}_t, a^{\mathrm{op}}_t, s_{t+1}, r_{t+1})$ in the trajectory $\tau$.
$\mathcal{D} \leftarrow \mathcal{D} \cup \bc{\tau}$.
Output: $\mathcal{D}$

§.§ Robust GAILfO

In this section, we present our robust Generative Adversarial Imitation Learning from Observation (robust GAILfO) algorithm based on the discussions in Section <ref>. We begin with the robust variant of the two-step procedure $\textsc{RL} \circ \textsc{IRL}_\psi \br{\rho_E}$ of the MCE-IRL algorithm:

\begin{align*}
\textsc{IRL}_\psi \br{\rho_E} =& \argmax_{c \in \mathbb{R}^{\mathcal{S} \times \mathcal{S}}} - \psi \br{c} - \Eee{\rho_E}{c\br{s,s'}} \\
& \quad \quad + \min_{\pi^{\mathrm{pl}} \in \Pi} \max_{\pi^{\mathrm{op}} \in \Pi} - \lambda H_{\rho^{\pi^\mathrm{mix}}_{M^{\mathrm{sim}}}}\br{\pi^\mathrm{pl}} + \Eee{\rho^{\pi^\mathrm{mix}}_{M^{\mathrm{sim}}}}{c\br{s,s'}} \\
\textsc{RL} \br{c} =& \argmin_{\pi^{\mathrm{pl}} \in \Pi}  \max_{\pi^{\mathrm{op}} \in \Pi} - \lambda H_{\rho^{\pi^\mathrm{mix}}_{M^{\mathrm{sim}}}}\br{\pi^\mathrm{pl}} + \Eee{\rho^{\pi^\mathrm{mix}}_{M^{\mathrm{sim}}}}{c\br{s,s'}} ,
\end{align*}

where $\pi^\mathrm{mix} = \alpha \pi^{\mathrm{pl}} + (1-\alpha) \pi^{\mathrm{op}}$. Then, similar to <cit.>, the above two step procedure  can be reduced to the following optimization problem using the discriminator $D:\mathcal{S} \times \mathcal{S} \to (0,1)$:

\begin{align*}
\min_{\pi^{\mathrm{pl}} \in \Pi} \max_{\pi^{\mathrm{op}} \in \Pi} \max_{D \in \br{0,1}^{\mathcal{S} \times \mathcal{S}}} & - \lambda H_{\rho^{\pi^\mathrm{mix}}_{M^{\mathrm{sim}}}}\br{\pi^\mathrm{pl}} + \Eee{\rho^{\pi^\mathrm{mix}}_{M^{\mathrm{sim}}}}{\log{D\br{s,s'}}} \\
& \quad \quad + \Eee{\rho_E}{\log\br{1-D\br{s,s'}}} .
\end{align*}

We parameterize the policies and the discriminator as $\pi^\mathrm{pl}_\theta$, $\pi^\mathrm{op}_\phi$, and $D_w$ (with parameters $\theta$, $\phi$, and $w$), and rewrite the above problem as follows:

\begin{align*}
\min_{\theta} \max_{\phi} \max_{w} & - \lambda H_{\rho^{\pi^\mathrm{mix}_{\theta,\phi}}_{M^{\mathrm{sim}}}}\br{\pi^\mathrm{pl}_\theta} + \Eee{\rho^{\pi^\mathrm{mix}_{\theta,\phi}}_{M^{\mathrm{sim}}}}{\log{D_w\br{s,s'}}} \\
& \quad \quad + \Eee{\rho_E}{\log\br{1-D_w\br{s,s'}}} ,
\end{align*}

where $\pi^\mathrm{mix}_{\theta,\phi} = \alpha \pi^{\mathrm{pl}}_\theta + (1-\alpha) \pi^{\mathrm{op}}_\phi$. We solve the above problem by taking gradient steps alternatively w.r.t. $\theta$, $\phi$, and $w$. The calculation for the gradient estimates are given in Appendix <ref>. Following <cit.>, we use the proximal policy optimization (PPO <cit.>) to update the policies parameters. Our complete algorithm is given in Algorithm <ref>.

We also note that one could use any robust RL approach (including domain randomization) to solve the inner min-max problem of (<ref>). In our work, we used the action-robustness approach since: (i) in the robust RL literature, the equivalence between the domain randomization approach and the action-robustness approach is already established <cit.>, and (ii) compared to the domain randomization approach, the action-robustness approach only requires access to a single simulation environment and creates a range of environments via action perturbations.

§ EXPERIMENTS

The average (over $3$ seeds) transfer performance of Algorithm <ref> with different values of $\alpha$ for each MuJoCo task as reported in the legend of each plot. The x-axis denotes the relative friction of the learner environment $M^\mathrm{sim}$. The policies are evaluated in $M^\mathrm{real}_{c^*}$ over $1e5$ steps truncating the last episode if it does not terminate.

The average (over $3$ seeds) transfer performance of Algorithm <ref> with different values of $\alpha$ for each MuJoCo task as reported in the legend of each plot. The x-axis denotes the relative mass of the learner environment $M^\mathrm{sim}$. The policies are evaluated in $M^\mathrm{real}_{c^*}$ over $1e5$ steps truncating the last episode if it does not terminate.

The average (over $3$ seeds) robust performance of Algorithm <ref> with different values of $\alpha$ for each MuJoCo task as reported in the legend of each plot. The expert environment $M^\mathrm{real}$, in which the demonstrations are collected, has relative friction $1.0$. In each plot, the black vertical line corresponds to the relative friction of the learner environment $M^\mathrm{sim}$ where we trained the policy with Algorithm <ref>. The x-axis denotes the relative friction of the test environment $M^\mathrm{test}$ in which the policies are evaluated. The policies are evaluated over $1e5$ steps truncating the last episode if it does not terminate.

The average (over $3$ seeds) robust performance of Algorithm <ref> with different values of $\alpha$ for each MuJoCo task as reported in the legend of each plot. The expert environment $M^\mathrm{real}$, in which the demonstrations are collected, has relative mass $1.0$. In each plot, the black vertical line corresponds to the relative mass of the learner environment $M^\mathrm{sim}$ where we trained the policy with Algorithm <ref>. The x-axis denotes the relative mass of the test environment $M^\mathrm{test}$ in which the policies are evaluated. The policies are evaluated over $1e5$ steps truncating the last episode if it does not terminate.

We compare the performance of our robust GAILfO algorithm with different values of $\alpha \in \bc{1.0, 0.999, 0.99, 0.98,0.97,0.96,0.95,0.90}$ against the standard GAILfO algorithm proposed in <cit.>. To the best of our knowledge, GAILfO is the only large-scale imitation learning method that is applicable under the setting described in Section <ref> (see Table <ref>).

§.§ Continuous Control Tasks on MuJoCo

In this section, we evaluate the performance of our method on standard continuous control benchmarks available on OpenAI Gym <cit.> utilizing the MuJoCo environment <cit.>. Specifically, we benchmark on five tasks: Half-Cheetah, Walker, Hopper, Swimmer, and Inverted-Double-Pendulum. Details of these environments can be found in <cit.> and on the GitHub website.

The default configurations of the MuJoCo environment (provided in OpenAI Gym) is regarded as the real or deployment environment ($M^\mathrm{real}$), and the expert demonstrations are collected there. We do not assume any access to the expert MDP beyond this during the training phase. We construct the simulation or training environments ($M^\mathrm{sim}$) for the imitator by modifying some parameters independently: (i) the mass of the bot in $M^\mathrm{sim}$ is $\bc{0.5, 0.75, 1.0, 1.5, 2.0} \times$ the mass in $M^\mathrm{real}$, and (ii) the friction coefficient on all the joints of the bot in $M^\mathrm{sim}$ is $\bc{0.5, 0.75, 1.0, 1.5, 2.0} \times$ the coefficient in $M^\mathrm{real}$.

We train an agent on each task by proximal policy optimization (PPO) algorithm <cit.> using the rewards defined in the OpenAI Gym <cit.>. We use the resulting stochastic policy as the expert policy $\pi^E$. In all our experiments, 10 state-only expert demonstrations collected by the expert policy $\pi^E$ in the real environment $M^\mathrm{real}$ is given to the learner.

Our Algorithm <ref> implementation is based on the codebase from <https://github.com/Khrylx/PyTorch-RL>. We use a two-layer feedforward neural network structure of (128, 128, tanh) for both actors (agent and adversary) and discriminator. The actor or policy networks are trained by the proximal policy optimization (PPO) method. For training the discriminator $D$, we use Adam <cit.> with a learning rate of $1e-4$. For each environment-mismatch pair, we identified the best performing $\alpha$ parameter based on the ablation study reported in Appendix <ref>. The learner is trained in the simulator $M^\mathrm{sim}$ for $\approx$3M time steps. We run our experiments, for each environment, with 3 different seeds. We report the mean and standard error of the performance (cumulative true rewards) over 3 trials. The cumulative reward is normalized with ones earned by $\pi^E$ and a random policy so that 1.0 and 0.0 indicate the performance of $\pi^E$ and the random policy, respectively.

Figures <ref>, and <ref> plot the performance of the policy evaluated on the deployment environment ($M^\mathrm{real}$). The x-axis corresponds to the simulation environment ($M^\mathrm{sim}$) on which the policy is trained on. We observe that our robust GAILfO produces policies that can be successfully transferred to the $M^\mathrm{real}$ environment from $M^\mathrm{sim}$ compared to the standard GAILfO.

Finally, we evaluate the robustness of the policies trained by our algorithm (with different dynamics mismatch) under different testing conditions. At test time, we evaluate the learned policies by changing the mass and friction values and estimating the cumulative rewards. As shown in Figures <ref> and <ref>, our Algorithm <ref> outperforms the baseline in terms of robustness as well.

§.§ Continuous Gridworld Tasks under Additive Transition Dynamics Mismatch

The contour curves for the reward function of the 2D gridworld environment.

-1In this section, we evaluate the effectiveness of our method on a continuous gridworld environment under a transition dynamics mismatch induced by additive noise. Specifically, we consider a 2D environment, where we denote the horizontal coordinate as $x \in [0,1]$ and vertical one as $y \in [0,1]$. The agent starts in the upper left corner, i.e., the coordinate $(0,1)$, and the episode ends when the agent reaches the lower right region defined by the indicator function $\mathbf{1}\{x\in [0.95, 1], y \in [-1, -0.95]\} $. The reward function is given by: $R(x,y) = -(x-1)^2 -(y+1)^2 -80 e^{-8(x^2 + y^2)} + 10 \cdot \mathbf{1}\{x\in [0.95, 1], y \in [-1, -0.95]\}$. Figure <ref> provides a graphical representation of the reward function. Note that the central region of the 2D environment represents a low reward area that should be avoided. The action space for the agent is given by $\mathcal{A} = [-0.5, 0.5]^2$, and the transition dynamics are given by: $s_{t+1} = s_t + \frac{a_t}{10}$ with probability (w.p.) $1 - \epsilon$, and $s_{t+1} = s_t - \frac{s_t}{10 \norm{s_t}_2}$ w.p. $\epsilon$. Thus, with probability $\epsilon$, the environment does not respond to the action taken by the agent, but it takes a step towards the low reward area centered at the origin, i.e., $- \frac{s_t}{10 \norm{s_t}_2}$. The agent should therefore pass far enough from the origin. The parameter $\epsilon$ can be varied to create a dynamic mismatch, e.g., higher $\epsilon$ corresponds to a more difficult environment.

We use three experts trained with $\epsilon = 0.0, \epsilon=0.05, \text{ and } \epsilon = 0.1$. The learners act in a different environment with the following values for $\epsilon$: $0.0, 0.05, 0.1, 0.15, 0.2$. Figure <ref> plots the performance of the trained learner policy evaluated on the expert environment. The x-axis corresponds to the learner environment on which the learner policy is trained. In general, we observe a behavior comparable to the MuJoCo experiments. We can often find an appropriate value for $\alpha$ such that Robust GAILfO learns to imitate under mismatch largely better than standard GAILfO.

[Expert $\epsilon = 0.0$]

[Expert $\epsilon = 0.05$]

[Expert $\epsilon = 0.1$]

Average performance (over $3$ seeds) of Algorithm <ref> with different values of $\alpha$ for each mismatch (i.e., each point on the x-axis) in the environment shown in Figure <ref>. The $\alpha$ values are chosen based on the ablation study in Figure <ref> (see Appendix <ref>). The x-axis denotes the $\epsilon$ value of the learner environment. The policies are evaluated over $1e5$ steps truncating the last episode if it does not terminate. In Appendix <ref>, we verify that our strategy of choosing appropriate $\alpha$ value does not introduce maximization bias.

§.§ Choice of $\alpha$

We note that one has to carefully choose the value of $\alpha$ to avoid too conservative behavior (see Figure <ref> in Appendix <ref>). In principle, given a rough estimate $\widehat T^E$ of the expert dynamics $T^E$, one could choose this value based on Eq. (<ref>). However, the choice of suitable $\alpha$ value is also affected by the other design choices of the algorithm, e.g., how many iterations the player and adversary are updated in the inner loop, and function approximators used.

In order to estimate the accuracy of the simulator, we can execute a safe baseline policy in both the simulator and the real environment, collect trajectories or datasets, and compute an estimate of the transition-dynamics distance between them. We can also utilize the performance difference lemma from <cit.> to obtain a lower bound on the transition dynamics mismatch based on the value function difference in the two environments.

Apart from the final evaluation, we also minimally access (in our experiments) the deployment environment for choosing the appropriate value for $\alpha$. Compared to training a policy in the deployment environment from scratch, accessing the deployment environment to choose $\alpha$ is sample-efficient. We only need to evaluate the final policies (trained in the simulation environment) once for each value of $\alpha$. When we already have a reasonable estimate of $\alpha$, we can also reduce these evaluations.

§ CONCLUSIONS

In this work, we propose a robust LfO method to solve an offline imitation-learning problem, in which a few state-only expert demonstrations and a simulator with misspecified dynamics are given to the learner. Even though our Algorithm <ref> is not essentially different from the standard robust RL methods, the robust optimization problem formulation to derive our algorithm is important and novel in the IL context. Experiment results in continuous control tasks on MuJoCo show that our method clearly outperforms the standard GAILfO in terms of the transfer performance (with model misspecification) in the real environment, as well as the robust performance under varying testing conditions.

Our algorithm falls under the category of zero-shot sim-to-real transfer <cit.> with expert demonstrations, making our method well suited for robotics applications. In principle, one can easily incorporate the two-player Markov game idea into any imitation learning algorithm and derive its robust version. This work can be considered a direction towards improving the sample efficiency of IL algorithms in terms of the number of environment interactions through robust training on a misspecified simulator.

Luca Viano has received financial support from the Enterprise for Society Center (E4S). Parameswaran Kamalaruban acknowledges support from The Alan Turing Institute. Craig Innes and Subramanian Ramamoorthy are supported by a grant from the UKRI Strategic Priorities Fund to the UKRI Research Node on Trustworthy Autonomous Systems Governance and Regulation (EP/V026607/1, 2020-2024). Adrian Weller acknowledges support from a Turing AI Fellowship under grant EP/V025379/1, EPSRC grant EP/V056522/1, The Alan Turing Institute, and the Leverhulme Trust via CFI.

#1     #1#1#1     #1     #1     #1     #1#1          #1#1

[Amodei, Olah, Steinhardt, Christiano,
Schulman, and ManéAmodei et al2016]
authorpersonDario Amodei, personChris Olah,
personJacob Steinhardt, personPaul Christiano,
personJohn Schulman, and personDan Mané.
Concrete problems in AI safety.
journalarXiv preprint arXiv:1606.06565

[Brockman, Cheung, Pettersson, Schneider,
Schulman, Tang, and ZarembaBrockman et al2016]
authorpersonGreg Brockman, personVicki
Cheung, personLudwig Pettersson, personJonas
Schneider, personJohn Schulman, personJie Tang,
and personWojciech Zaremba. year2016.
Openai gym.
journalarXiv preprint arXiv:1606.01540

[Desai, Durugkar, Karnan, Warnell, Hanna, and
StoneDesai et al2020]
authorpersonSiddharth Desai, personIshan
Durugkar, personHaresh Karnan, personGarrett
Warnell, personJosiah Hanna, and personPeter
Stone. year2020.
An Imitation from Observation Approach to Transfer
Learning with Dynamics Mismatch. In booktitleAdvances in
Neural Information Processing Systems.

[Doyle, Francis, and TannenbaumDoyle
et al2013]
authorpersonJohn C Doyle, personBruce A
Francis, and personAllen R Tannenbaum.
booktitleFeedback control theory.
publisherCourier Corporation.

[Edwards, Sahni, Schroecker, and
IsbellEdwards et al2019]
authorpersonAshley Edwards, personHimanshu
Sahni, personYannick Schroecker, and personCharles
Isbell. year2019.
Imitating latent policies from observation. In
booktitleInternational Conference on Machine Learning.

[Even-Dar and MansourEven-Dar and
authorpersonEyal Even-Dar and personYishay
Mansour. year2003.
Approximate equivalence of Markov decision
In booktitleLearning Theory and Kernel Machines.
publisherSpringer, pages581–594.

[Everitt and HutterEveritt and
authorpersonTom Everitt and personMarcus
Hutter. year2016.
Avoiding wireheading with value reinforcement
learning. In booktitleInternational Conference on
Artificial General Intelligence.

[Fu, Luo, and LevineFu et al2018]
authorpersonJustin Fu, personKatie Luo,
and personSergey Levine. year2018.
Learning Robust Rewards with Adversarial Inverse
Reinforcement Learning. In booktitleInternational
Conference on Learning Representations.

[Gangwani and PengGangwani and Peng2020]
authorpersonTanmay Gangwani and personJian
Peng. year2020.
State-only Imitation with Transition Dynamics
Mismatch. In booktitleInternational Conference on Learning

[Goodfellow, Pouget-Abadie, Mirza, Xu,
Warde-Farley, Ozair, Courville, and BengioGoodfellow et al2014]
authorpersonIan J. Goodfellow, personJean
Pouget-Abadie, personMehdi Mirza, personBing Xu,
personDavid Warde-Farley, personSherjil Ozair,
personAaron Courville, and personYoshua Bengio.
Generative Adversarial Networks. In
booktitleAdvances in Neural Information Processing

[Handa, Van Wyk, Yang, Liang, Chao, Wan,
Birchfield, Ratliff, and FoxHanda et al2020]
authorpersonAnkur Handa, personKarl
Van Wyk, personWei Yang, personJacky Liang,
personYu-Wei Chao, personQian Wan,
personStan Birchfield, personNathan Ratliff, and
personDieter Fox. year2020.
DexPilot: Vision-Based Teleoperation of Dexterous
Robotic Hand-Arm System. In booktitleIEEE International
Conference on Robotics and Automation.

[Ho and ErmonHo and Ermon2016]
authorpersonJonathan Ho and personStefano
Ermon. year2016.
Generative adversarial imitation learning. In
booktitleAdvances in Neural Information Processing

authorpersonGarud N Iyengar.
Robust dynamic programming.
journalMathematics of Operations Research

[Jiang, Pang, and YuJiang
et al2020]
authorpersonShengyi Jiang, personJingcheng
Pang, and personYang Yu. year2020.
Offline Imitation Learning with a Misspecified
Simulator. In booktitleAdvances in Neural Information
Processing Systems.

[Kamalaruban, Huang, Hsieh, Rolland, Shi, and
CevherKamalaruban et al2020]
authorpersonParameswaran Kamalaruban,
personYu-Ting Huang, personYa-Ping Hsieh,
personPaul Rolland, personCheng Shi, and
personVolkan Cevher. year2020.
Robust reinforcement learning via adversarial
training with langevin dynamics. In booktitleAdvances in
Neural Information Processing Systems.

[Kingma and BaKingma and Ba2015]
authorpersonDiederik P Kingma and
personJimmy Ba. year2015.
Adam: A Method for Stochastic Optimization. In
booktitleInternational Conference on Learning

[Levine, Finn, Darrell, and AbbeelLevine
et al2016]
authorpersonSergey Levine, personChelsea
Finn, personTrevor Darrell, and personPieter
Abbeel. year2016.
End-to-end training of deep visuomotor policies.
journalThe Journal of Machine Learning Research

[Lillicrap, Hunt, Pritzel, Heess, Erez, Tassa,
Silver, and WierstraLillicrap et al2016]
authorpersonTimothy P Lillicrap,
personJonathan J Hunt, personAlexander Pritzel,
personNicolas Heess, personTom Erez,
personYuval Tassa, personDavid Silver, and
personDaan Wierstra. year2016.
Continuous control with deep reinforcement
learning. In booktitleInternational Conference on Learning

authorpersonMichael L Littman.
Markov Games as a Framework for Multi-Agent
Reinforcement Learning. In booktitleInternational
Conference on Machine Learning.

[Liu, Ling, Mu, and SuLiu
et al2020]
authorpersonFangchen Liu, personZhan Ling,
personTongzhou Mu, and personHao Su.
State Alignment-based Imitation Learning. In
booktitleInternational Conference on Learning

[Mankowitz, Levine, Jeong, Abdolmaleki,
Springenberg, Shi, Kay, Hester, Mann, and RiedmillerMankowitz
et al2020]
authorpersonDaniel J Mankowitz, personNir
Levine, personRae Jeong, personAbbas Abdolmaleki,
personJost Tobias Springenberg, personYuanyuan Shi,
personJackie Kay, personTodd Hester,
personTimothy Mann, and personMartin Riedmiller.
Robust Reinforcement Learning for Continuous
Control with Model Misspecification. In
booktitleInternational Conference on Learning

[Morimoto and DoyaMorimoto and Doya2005]
authorpersonJun Morimoto and personKenji
Doya. year2005.
Robust reinforcement learning.
journalNeural Computation (year2005).

[Ng, Russell, et alNg
et al2000]
authorpersonAndrew Y Ng, personStuart J
Russell, et al year2000.
Algorithms for inverse reinforcement learning. In
booktitleInternational Conference on Machine Learning.

[Nilim and El GhaouiNilim and
El Ghaoui2005]
authorpersonArnab Nilim and personLaurent
El Ghaoui. year2005.
Robust control of Markov decision processes with
uncertain transition matrices.
journalOperations Research (year2005).

[Peng, Andrychowicz, Zaremba, and AbbeelPeng
et al2018]
authorpersonXue Bin Peng, personMarcin
Andrychowicz, personWojciech Zaremba, and
personPieter Abbeel. year2018.
Sim-to-real transfer of robotic control with
dynamics randomization. In booktitleIEEE International
Conference on Robotics and Automation.

[Pinto, Davidson, Sukthankar, and GuptaPinto
et al2017]
authorpersonLerrel Pinto, personJames
Davidson, personRahul Sukthankar, and personAbhinav
Gupta. year2017.
Robust Adversarial Reinforcement Learning. In
booktitleInternational Conference on Machine Learning.

[Radosavovic, Wang, Pinto, and
MalikRadosavovic et al2020]
authorpersonIlija Radosavovic,
personXiaolong Wang, personLerrel Pinto, and
personJitendra Malik. year2020.
State-only imitation learning for dexterous
journalarXiv preprint arXiv:2004.04650

[Rajeswaran, Ghotra, Ravindran, and
LevineRajeswaran et al2017]
authorpersonAravind Rajeswaran,
personSarvjeet Ghotra, personBalaraman Ravindran,
and personSergey Levine. year2017.
EPOpt: Learning Robust Neural Network Policies
Using Model Ensembles. In booktitleInternational Conference
on Learning Representations.

authorpersonStefan Schaal.
Is imitation learning the route to humanoid
journalTrends in Cognitive Sciences

[Schulman, Levine, Abbeel, Jordan, and
MoritzSchulman et al2015]
authorpersonJohn Schulman, personSergey
Levine, personPieter Abbeel, personMichael Jordan,
and personPhilipp Moritz. year2015.
Trust region policy optimization. In
booktitleInternational Conference on Machine Learning.

[Schulman, Wolski, Dhariwal, Radford, and
KlimovSchulman et al2017]
authorpersonJohn Schulman, personFilip
Wolski, personPrafulla Dhariwal, personAlec Radford,
and personOleg Klimov. year2017.
Proximal policy optimization algorithms.
journalarXiv preprint arXiv:1707.06347

[Silver, Lever, Heess, Degris, Wierstra, and
RiedmillerSilver et al2014]
authorpersonDavid Silver, personGuy Lever,
personNicolas Heess, personThomas Degris,
personDaan Wierstra, and personMartin Riedmiller.
Deterministic policy gradient algorithms. In
booktitleInternational Conference on Machine Learning.

[Sutton, McAllester, Singh, and MansourSutton
et al1999]
authorpersonRichard S Sutton, personDavid A
McAllester, personSatinder P Singh, and
personYishay Mansour. year1999.
Policy gradient methods for reinforcement learning
with function approximation.. In booktitleAdvances in
Neural Information Processing Systems.

[Tessler, Efroni, and MannorTessler
et al2019]
authorpersonChen Tessler, personYonathan
Efroni, and personShie Mannor.
Action Robust Reinforcement Learning and
Applications in Continuous Control. In
booktitleInternational Conference on Machine Learning.

[Todorov, Erez, and TassaTodorov
et al2012]
authorpersonEmanuel Todorov, personTom
Erez, and personYuval Tassa. year2012.
Mujoco: A physics engine for model-based control.
In booktitleIEEE/RSJ International Conference on Intelligent
Robots and Systems.

[Torabi, Warnell, and StoneTorabi
et al2018]
authorpersonFaraz Torabi, personGarrett
Warnell, and personPeter Stone.
Generative adversarial imitation from observation.
journalarXiv preprint arXiv:1807.06158

[Viano, Huang, Kamalaruban, Weller, and
CevherViano et al2021]
authorpersonLuca Viano, personYu-Ting
Huang, personParameswaran Kamalaruban, personAdrian
Weller, and personVolkan Cevher.
Robust Inverse Reinforcement Learning under
Transition Dynamics Mismatch. In booktitleAdvances in
Neural Information Processing Systems.

[Yang, Ma, Huang, Sun, Liu, Huang, and
GanYang et al2019]
authorpersonChao Yang, personXiaojian Ma,
personWenbing Huang, personFuchun Sun,
personHuaping Liu, personJunzhou Huang, and
personChuang Gan. year2019.
Imitation learning from observations by minimizing
inverse dynamics disagreement. In booktitleAdvances in
Neural Information Processing Systems.

[Zhao, Queralta, and WesterlundZhao
et al2020]
authorpersonWenshuai Zhao,
personJorge Peña Queralta, and personTomi
Westerlund. year2020.
Sim-to-Real Transfer in Deep Reinforcement Learning
for Robotics: a Survey. In booktitleIEEE Symposium Series
on Computational Intelligence.

authorpersonBrian D Ziebart.
Modeling purposeful adaptive behavior with the
principle of maximum causal entropy.

[Ziebart, Maas, Bagnell, and DeyZiebart
et al2008]
authorpersonBrian D Ziebart, personAndrew L
Maas, personJ Andrew Bagnell, and personAnind K
Dey. year2008.
Maximum entropy inverse reinforcement learning. In
booktitleAAAI Conference on Artificial Intelligence.

§ CODE REPOSITORY

§ DETAILS ON THE EQUIVALENCE BETWEEN ACTION ROBUST MDP AND ROBUST MDP

In the following we prove the last equality of Eq. (<ref>).

Given the set

\begin{equation*}
\mathcal{T}^{\alpha} ~:=~ \bc{ T : T(s^\prime|s,a) =  \alpha T^{\mathrm{sim}}(s^\prime|s,a) + (1 - \alpha)\bar{T}(s^\prime|s) , \, \bar{T}(s^\prime|s) = \sum_a \pi(a|s) T^{\mathrm{sim}}(s^\prime|s,a), \quad \forall \pi \in \Pi}
\end{equation*}

and a cost function depending only on states, i.e. $r: \mathcal{S}\times\mathcal{S}\rightarrow \mathbb{R}$, define $G_c = \sum^{\infty}_{t=0} \gamma^t r(s_t, s_{t+1})$.
Then, the following holds:

\begin{equation*}
\min_{\pi^{\mathrm{pl}} \in \Pi} \max_{T^\alpha \in \mathcal{T}^{\alpha}}  \E{G_c \bigm| \pi^{\mathrm{pl}}, P_0, T^\alpha} = \min_{\pi^{\mathrm{pl}} \in \Pi} \max_{\pi^{\mathrm{op}} \in \Pi} \E{G_c \bigm| \alpha \pi^{\mathrm{pl}} + (1 - \alpha) \pi^{\mathrm{op}}, M^{\mathrm{sim}}} .
\end{equation*}

In particular, the result in Eq. (<ref>) follows from the choice: $ r(s_t, s_{t+1}) = c(s_t,s_{t+1}) + H^{\pi^{\mathrm{pl}}}(A|S=s_t)$.

Let us define $P^{\pi, T}(s_0, \dots, s_N) := P_0(s_0) \prod^{N-1}_{t=0}\sum_{a}\pi(a|s_t) T(s_{t+1}|s_t, a)$.
We need to show equality between the distributions $P^{\pi^{\mathrm{pl}}, \alpha T^{\mathrm{sim}} + (1 - \alpha) \bar{T}}$ and $P^{\alpha \pi^{\mathrm{pl}} + (1 - \alpha)\pi^{\mathrm{op}}, T^{\mathrm{sim}}}$.
Due to the Markov property, this is equivalent to show:

\begin{equation}
\sum_{a}\pi^{\mathrm{pl}}(a|s_t) \bs{\alpha T^{\mathrm{sim}}(s_{t+1}|s_t, a) + (1 - \alpha) \bar{T}(s_{t+1}| s_t)} = \sum_{a} \bs{\alpha \pi^{\mathrm{pl}}(a|s_t) + (1 - \alpha)\pi^{\mathrm{op}}(a|s_t)}T^{\mathrm{sim}}(s_{t+1}|s_t, a) ,
\end{equation}

that implies:

\begin{equation*}
\underbrace{\sum_{a}\pi^{\mathrm{pl}}(a|s_t)}_{=1} \bar{T}(s_{t+1}| s_t) = \sum_{a}\pi^{\mathrm{op}}(a|s_t)T^{\mathrm{sim}}(s_{t+1}|s_t, a) .
\end{equation*}

Hence, it follows that equality between $P^{\pi^{\mathrm{pl}}, \alpha T^{\mathrm{sim}} + (1 - \alpha) \bar{T}}$ and $P^{\alpha \pi^{\mathrm{pl}} + (1 - \alpha)\pi^{\mathrm{op}}, T^{\mathrm{sim}}}$ holds for:

\begin{equation*} \bar{T}(s_{t+1}| s_t) = \sum_{a}\pi^{\mathrm{op}}(a|s_t)T^{\mathrm{sim}}(s_{t+1}|s_t, a) ,
\end{equation*}

as we used in the definition of the set $\mathcal{T}^{\alpha}$.

§ ADDITIONAL DETAILS ON ALGORITHM <REF>

By interpreting $R_w\br{s,s'} = - \log{D_w\br{s,s'}}$ as the reward function, we have (for a fixed $w$):

\begin{align*}
J \br{\theta,\phi} ~:=~& \Eee{\rho^{\pi^\mathrm{mix}_{\theta,\phi}}_{M^{\mathrm{sim}}}}{{R_w\br{s,s'}}} + \lambda H_{\rho^{\pi^\mathrm{mix}_{\theta,\phi}}_{M^{\mathrm{sim}}}}\br{\pi^\mathrm{pl}_\theta} ~=~ J_1 \br{\theta,\phi} + J_2 \br{\theta,\phi} ,
\end{align*}

\begin{align*}
J_1 \br{\theta,\phi} ~:=~& \E{\sum_{t} \gamma^t R_w(s_t, s_{t+1})\big\vert \pi_{\theta, \phi}^\mathrm{mix}, M^{\mathrm{sim}}} \\
J_2 \br{\theta,\phi} ~:=~& \lambda \E{\sum_{t} \gamma^t H^{\pi_{\theta}^{pl}}(A|S=s_t) \big\vert \pi_{\theta, \phi}^\mathrm{mix}, M^{\mathrm{sim}}} .
\end{align*}

By the policy gradient theorem, the derivatives of the first term w.r.t the player and the opponent policy parameters are given by:

\begin{align*}
\nabla_{\theta} J_1(\theta, \phi) ~=~& \sum_{s \in \mathcal{S}}\sum_{t}\gamma^t \P{S_t = s \mid \pi_{\theta, \phi}^\mathrm{mix}, M^{\mathrm{sim}}} \sum_{a} \nabla_{\theta} \pi_{\theta, \phi}^\mathrm{mix}(a|s) Q_{\pi_{\theta, \phi}^\mathrm{mix}}(s,a) \\
\nabla_{\phi} J_1(\theta, \phi) ~=~& \sum_{s \in \mathcal{S}}\sum_{t}\gamma^t \P{S_t = s \mid \pi_{\theta, \phi}^\mathrm{mix}, M^{\mathrm{sim}}} \sum_{a} \nabla_{\phi} \pi_{\theta, \phi}^\mathrm{mix}(a|s) Q_{\pi_{\theta, \phi}^\mathrm{mix}}(s,a) ,
\end{align*}

\begin{align*}
Q_{\pi_{\theta, \phi}^\mathrm{mix}}(s,a) ~=~& \sum_{s^\prime} T^{\mathrm{sim}}(s^\prime \mid a, s) \br{R_w(s,s^\prime) + \gamma V_{\pi_{\theta, \phi}^\mathrm{mix}}(s^\prime)} \\
V_{\pi_{\theta, \phi}^\mathrm{mix}}(s) ~=~& \E{\sum_{t} \gamma^t R_w(s_t, s_{t+1}) \bigg\vert \pi_{\theta, \phi}^\mathrm{mix}, M^{\mathrm{sim}}, s_0 = s} .
\end{align*}

For the second term, we introduce the following quantities:

\begin{align*}
Q_{\pi_{\theta, \phi}^\mathrm{mix}}^{\mathrm{log}}(s,a) ~=~& \sum_{s^\prime} T^{\mathrm{sim}}(s^\prime \mid s,a) \br{\lambda H^{\pi_{\theta}^{pl}}(A|S=s_t) + \gamma V_{\pi_{\theta, \phi}^\mathrm{mix}}^{\mathrm{log}}(s')} \\
V_{\pi_{\theta, \phi}^\mathrm{mix}}^{\mathrm{log}}(s) ~=~& \E{\sum_{t} \lambda \gamma^t H^{\pi_{\theta}^{pl}}(A|S=s_t)\bigg \vert \pi_{\theta, \phi}^\mathrm{mix}, M^{\mathrm{sim}}, s_0 = s}
\end{align*}

Then, we obtain the following derivatives of the second term:

\begin{align*}
\nabla_{\theta} J_2(\theta, \phi) ~=~& \sum_{s \in \mathcal{S}}\sum_{t}\gamma^t \P{S_t = s \mid \pi_{\theta, \phi}^\mathrm{mix}, M^{\mathrm{sim}}} \sum_{a} \nabla_{\theta} \pi_{\theta, \phi}^\mathrm{mix}(a|s) Q^{\mathrm{log}}_{\pi_{\theta, \phi}^\mathrm{mix}}(s,a) \\
\nabla_{\phi} J_2(\theta, \phi) ~=~& \sum_{s \in \mathcal{S}}\sum_{t}\gamma^t \P{S_t = s \mid \pi_{\theta, \phi}^\mathrm{mix}, M^{\mathrm{sim}}} \sum_{a} \nabla_{\phi} \pi_{\theta, \phi}^\mathrm{mix}(a|s) Q^{\mathrm{log}}_{\pi_{\theta, \phi}^\mathrm{mix}}(s,a) .
\end{align*}

For a practical algorithm, we need to compute gradient estimates from a data-set of sampled trajectories $\mathcal{D} = \bc{\tau_i}_i$ with $\tau_i = (s^i_0, a^i_0, \dots, s^i_T, a^i_T)$. The gradient estimates are given by:

\begin{align*}
\widehat{\nabla}_{\theta} J_1(\theta, \phi) ~=~& \sum_{\tau_i \in \mathcal{D}} \sum_{t}\gamma^t \nabla_{\theta} \log \pi_{\theta, \phi}^\mathrm{mix}(a^i_t|s^i_t) \widehat{Q}_{\pi_{\theta, \phi}^\mathrm{mix}}(s^i_t,a^i_t) \\
\widehat{\nabla}_{\phi} J_1(\theta, \phi) ~=~& \sum_{\tau_i \in \mathcal{D}} \sum_{t}\gamma^t \nabla_{\phi} \log \pi_{\theta, \phi}^\mathrm{mix}(a^i_t|s^i_t) \widehat{Q}_{\pi_{\theta, \phi}^\mathrm{mix}}(s^i_t,a^i_t) \\
\widehat{\nabla}_{\theta} J_2(\theta, \phi) ~=~& \sum_{\tau_i \in \mathcal{D}} \sum_{t}\gamma^t \nabla_{\theta} \log \pi_{\theta, \phi}^\mathrm{mix}(a^i_t|s^i_t) \widehat{Q}^{\mathrm{log}}_{\pi_{\theta, \phi}^\mathrm{mix}}(s^i_t,a^i_t) \\
\widehat{\nabla}_{\phi} J_2(\theta, \phi) ~=~& \sum_{\tau_i \in \mathcal{D}} \sum_{t}\gamma^t \nabla_{\phi} \log \pi_{\theta, \phi}^\mathrm{mix}(a^i_t|s^i_t) \widehat{Q}^{\mathrm{log}}_{\pi_{\theta, \phi}^\mathrm{mix}}(s^i_t,a^i_t) ,
\end{align*}

where the estimator $\widehat{Q}_{\pi_{\theta, \phi}^\mathrm{mix}}(s^i_t, a^i_t)$ is the future return observed for the trajectory $i$ after time $t$, i.e., $\widehat{Q}_{\pi_{\theta, \phi}^\mathrm{mix}}(s^i_t, a^i_t) = \sum^{T}_{k=t+1} \gamma^{k - t - 1}R_w(s^i_k, s^i_{k+1})=G^i_t$. Similarly, for the entropy term we have $\widehat{Q}^{\mathrm{log}}_{\pi_{\theta, \phi}^\mathrm{mix}}(s^i_t, a^i_t) = \sum^{T}_{k=t+1} - \gamma^{k - t - 1} H^{\pi^{\mathrm{pl}}_{\theta}}(A|S=s^i_k) =G^{\mathrm{log},i}_t$. The trajectory sampling process is given in Algorithm <ref>.

§ TRANSFER PERFORMANCE: MUJOCO

We present the following results:

* The ablation study on the transfer performance of Algorithm <ref> with different values of $\alpha$ under the relative friction mismatches (see Figure <ref>).

* The ablation study on the transfer performance of Algorithm <ref> with different values of $\alpha$ under the relative mass mismatches (see Figure <ref>).

* The transfer performance of Algorithm <ref> with different (best) values of $\alpha$ for each relative friction mismatch of a task (see Figure <ref>).
* The transfer performance of Algorithm <ref> with different (best) values of $\alpha$ for each relative mass mismatch of a task (see Figure <ref>).

The average (over $3$ seeds) transfer performance of Algorithm <ref> with different values of $\alpha$. The ablation shown here is used to choose $\alpha$ in Figure <ref>. The x-axis denotes the relative friction of the learner environment $M^\mathrm{sim}$. The policies are evaluated in $M^\mathrm{real}_{c^*}$ over $1e5$ steps truncating the last episode if it does not terminate. Note that robust-GAILfO with $\alpha = 1$ corresponds to GAILfO.

The average (over $3$ seeds) transfer performance of Algorithm <ref> with different values of $\alpha$. The ablation shown here is used to choose $\alpha$ in Figure <ref>. The x-axis denotes the relative mass of the learner environment $M^\mathrm{sim}$. The policies are evaluated in $M^\mathrm{real}_{c^*}$ over $1e5$ steps truncating the last episode if it does not terminate. Note that robust-GAILfO with $\alpha = 1$ corresponds to GAILfO.

Best value for $\alpha < 1$ chosen independently for each mismatch based on the ablation in Figure <ref>. The performance of this configuration is reported by the red line in Figure <ref>. We add a $1$ in brackets when standard GAILfO outperforms the robust version. The value outside brackets denotes the best value found for the robust version.

5cRelative Friction

0.5         0.75         1.0       1.5       2.0
HalfCheetah     0.999       0.999        0.999     0.999     0.999

Walker        0.98        0.999        0.97      0.97      0.97

Hopper         0.9 (1)     0.99 (1)     0.97      0.95      0.95

Best value for $\alpha < 1$ chosen independently for each mismatch based on the ablation in Figure <ref>. The performance of this configuration is reported by the red line in Figure <ref>. We add a $1$ in brackets when standard GAILfO outperforms the robust version. The value outside brackets denotes the best value found for the robust version.

5cRelative Mass

0.5         0.75         1.0       1.5       2.0
HalfCheetah     0.96       0.97        0.98     0.96     0.97

Walker        0.98        0.95        0.97      0.999      0.98

Hopper          0.9     0.97     0.97      0.98      0.999

InvDoublePendulum     0.98     0.99     0.97     0.96     0.97

Swimmer     0.96 (1)     0.999 (1)     0.95     0.95     0.98

Average performance (over $3$ seeds ) of Algorithm <ref> with the value of $\alpha$ that is chosen indipendently for each mismatch (i.e. each point on the x-axis). The choice is made picking the best performing $\alpha$ for each mismatch in Figure <ref>. The x-axis reports the relative friction of the learner environment. The policies are evaluated over $1e5$ steps truncating the last episode if it does not terminate.The values chosen for $\alpha$ are given in Table <ref>.

Average performance (over $3$ seeds ) of Algorithm <ref> with the value of $\alpha$ that is chosen indipendently for each mismatch (i.e. each point on the x-axis). The choice is made picking the best performing $\alpha$ for each mismatch in Figure <ref>. The x-axis reports the relative mass of the learner environment. The policies are evaluated over $1e5$ steps truncating the last episode if it does not terminate. The values chosen for $\alpha$ are given in Table <ref>.

§ ROBUST PERFORMANCE: MUJOCO

We present the following results:

* The ablation study on the robust performance of Algorithm <ref> with different values of $\alpha$ under the relative friction variations (see Figure <ref>).

* The ablation study on the robust performance of Algorithm <ref> with different values of $\alpha$ under the relative mass variations (see Figure <ref>).

The average (over $3$ seeds) robust performance of Algorithm <ref> with different values of $\alpha$. The ablation shown here is used to choose $\alpha$ in Figure <ref>. The expert environment $M^\mathrm{real}$, in which the demonstrations are collected, has relative friction $1.0$. In each plot, the black vertical line corresponds to the relative friction of the learner environment $M^\mathrm{sim}$ where we trained the policy with Algorithm <ref>. The x-axis denotes the relative friction of the test environment $M^\mathrm{test}$ in which the policies are evaluated. The policies are evaluated over $1e5$ steps truncating the last episode if it does not terminate. Note that robust-GAILfO with $\alpha = 1$ corresponds to GAILfO.

The average (over $3$ seeds) robust performance of Algorithm <ref> with different values of $\alpha$. The ablation shown here is used to choose $\alpha$ in Figure <ref>. The expert environment $M^\mathrm{real}$, in which the demonstrations are collected, has relative mass $1.0$. In each plot, the black vertical line corresponds to the relative mass of the learner environment $M^\mathrm{sim}$ where we trained the policy with Algorithm <ref>. The x-axis denotes the relative mass of the test environment $M^\mathrm{test}$ in which the policies are evaluated. The policies are evaluated over $1e5$ steps truncating the last episode if it does not terminate. Note that robust-GAILfO with $\alpha = 1$ corresponds to GAILfO.

§ TRANSFER PERFORMANCE: CONTINUOUS GRIDWORLD

[Expert $\epsilon = 0.0$]

[Expert $\epsilon = 0.05$]

[Expert $\epsilon = 0.1$]

The average (over $3$ seeds) transfer performance of Algorithm <ref> with fixed value of $\alpha$ for each mismatch (i.e., each point on the x-axis) in the environment shown in Figure <ref>. The x-axis denotes the $\epsilon$ value of the learner environment. The policies are evaluated over $1e5$ steps truncating the last episode if it does not terminate.

[Expert $\epsilon = 0.0$]

[Expert $\epsilon = 0.05$]

[Expert $\epsilon = 0.1$]

The average (over $3$ seeds) transfer performance of Algorithm <ref> with different values of $\alpha$ for each mismatch (i.e., each point on the x-axis) in the environment shown in Figure <ref>. The x-axis denotes the $\epsilon$ value of the learner environment. The policies are evaluated over $1e5$ steps truncating the last episode if it does not terminate. The ablation shown here is used to choose $\alpha$ in Figure <ref>. Note that robust-GAILfO with $\alpha = 1$ corresponds to GAILfO.

§ ADDITIONAL EXPERIMENTS ON CHOICE OF $\ALPHA$

In this section, we aim to understand whether our strategy of choosing suitable $\alpha$ value introduces maximization bias. For example, in Figure <ref>, the best performing $\alpha$ is chosen, and its performance curve (w.r.t. the original seeds used for training) is presented in Figure <ref>. To avoid this bias, for the chosen best performing $\alpha$ in Figure <ref>, we conduct a new set of runs with a new set of seeds. The new results presented in Figure <ref> suggest that our $\alpha$ selection process does not introduce maximization bias.

[Expert $\epsilon = 0.0$; Original seeds]

[Expert $\epsilon = 0.0$; New seeds]

[Expert $\epsilon = 0.05$; Original seeds]

[Expert $\epsilon = 0.05$; New seeds]

[Expert $\epsilon = 0.1$; Original seeds]

[Expert $\epsilon = 0.1$; New seeds]

Experiments for understanding whether our strategy of choosing suitable $\alpha$ value introduces maximization bias.