# Cross-Domain Imitation Learning
via Optimal Transport

Arnaud Fickinger13 Samuel Cohen23 Stuart Russell1 Brandon Amos3
1Berkeley AI Research 2University College London 3Facebook AI
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Cross-domain imitation learning studies how to leverage expert demonstrations
of one agent to train an imitation agent with a different embodiment or
morphology. Comparing trajectories and stationary distributions between the
expert and imitation agents is challenging because they live on different
systems that may not even have the same dimensionality. We propose _Gromov-
Wasserstein Imitation Learning (GWIL)_ , a method for cross-domain imitation
that uses the Gromov-Wasserstein distance to align and compare states between
the different spaces of the agents. Our theory formally characterizes the
scenarios where GWIL preserves optimality, revealing its possibilities and
limitations. We demonstrate the effectiveness of GWIL in non-trivial
continuous control domains ranging from simple rigid transformation of the
expert domain to arbitrary transformation of the state-action space.
111Project site with videos and code: https://arnaudfickinger.github.io/gwil/

##  1 Introduction

Reinforcement learning (RL) methods have attained impressive results across a
number of domains, e.g., Berner et al. (2019); Kober et al. (2013); Levine et
al. (2016); Vinyals et al. (2019). However, the effectiveness of current RL
method is heavily correlated to the quality of the training reward. Yet for
many real-world tasks, designing dense and informative rewards require
significant engineering effort. To alleviate this effort, imitation learning
(IL) proposes to learn directly from expert demonstrations. Most current IL
approaches can be applied solely to the simplest setting where the expert and
the agent share the same embodiment and transition dynamics that live in the
same state and action spaces. In particular, these approaches require expert
demonstrations from the agent domain. Therefore, we might reconsider the
utility of IL as it seems to only move the problem, from designing informative
rewards to providing expert demonstrations, rather than solving it. However,
if we relax the constraining setting of current IL methods, then natural
imitation scenarios that genuinely alleviate engineering effort appear.
Indeed, not requiring the same dynamics would enable agents to imitate humans
and robots with different morphologies, hence widely enlarging the
applicability of IL and alleviating the need for in-domain expert
demonstrations.

This relaxed setting where the expert demonstrations comes from another domain
has emerged as a budding area with more realistic assumptions (Gupta et al.,
2017; Liu et al., 2019; Sermanet et al., 2018; Kim et al., 2020; Raychaudhuri
et al., 2021) that we will refer to as _Cross-Domain Imitation Learning._ A
common strategy of these works is to learn a mapping between the expert and
agent domains. To do so, they require access to proxy tasks where both the
expert and the agent act optimally in there respective domains. Under some
structural assumptions, the learned map enables to transform a trajectory in
the expert domain into the agent domain while preserving the optimality.
Although these methods indeed relax the typical setting of IL, requiring proxy
tasks heavily restrict the applicability of Cross-Domain IL. For example, it
rules out imitating an expert never seen before as well as transferring to a
new robot.

In this paper, we relax the assumptions of Cross-Domain IL and propose a
benchmark and method that do not need access to proxy tasks. To do so, we
depart from the point of view taken by previous work and formalize Cross-
Domain IL as an optimal transport problem. We propose a method, that we call
_Gromov Wasserstein Imitation Learning (GWIL)_ , that uses the Gromov-
Wasserstein distance to solve the benchmark. We formally characterize the
scenario where GWIL preserves optimality (theorem 1), revealing the
possibilities and limitations. The construction of our proxy rewards to
optimize optimal transport quantities using RL generalizes previous work that
assumes uniform occupancy measures (Dadashi et al., 2020; Papagiannis & Li,
2020) and is of independent interest. Our experiments show that GWIL learns
optimal behaviors with a single demonstration from another domain without any
proxy tasks in non-trivial continuous control settings.

##  2 Related Work

Figure 1:  The Gromov-Wasserstein distance enables us to compare the
stationary state-action distributions of two agents with different dynamics
and state-action spaces. We use it as a pseudo-reward for cross-domain
imitation learning.  Figure 2:  Isometric policies (definition 2) have the
same pairwise distances within the state-action space of the stationary
distributions. In Euclidean spaces, isometric transformations preserve these
pairwise distances and include rotations, translations, and reflections.

Imitation learning. An early approach to IL is Behavioral Cloning (Pomerleau,
1988; 1991) which amounts to training a classifier or regressor via supervised
learning to replicate the expert’s demonstration. Another key approach is
Inverse Reinforcement Learning (Ng & Russell, 2000; Abbeel & Ng, 2004; Abbeel
et al., 2010), which aims at learning a reward function under which the
observed demonstration is optimal and can then be used to train a agent via
RL. To bypass the need to learn the expert’s reward function, Ho & Ermon
(2016) show that IRL is a dual of an occupancy measure matching problem and
propose an adversarial objective whose optimization approximately recover the
expert’s state-action occupancy measure, and a practical algorithm that uses a
generative adversarial network (Goodfellow et al., 2014). While a number of
recent work aims at improving this algorithm relative to the training
instability caused by the minimax optimization, Primal Wasserstein Imitation
Learning (PWIL) (Dadashi et al., 2020) and Sinkhorn Imitation Learning (SIL)
(Papagiannis & Li, 2020) view IL as an optimal transport problem between
occupancy measures to completely eliminate the minimax objective and
outperforms adversarial methods in terms of sample efficiency. Heess et al.
(2017); Peng et al. (2018); Zhu et al. (2018); Aytar et al. (2018) scale
imitation learning to complex human-like locomotion and game behavior in non-
trivial settings. Our work is an extension of Dadashi et al. (2020);
Papagiannis & Li (2020) from the Wasserstein to the Gromov-Wasserstein
setting. This takes us beyond limitation that the expert and imitator are in
the same domain and into the cross-domain setting between agents that live in
different spaces.

Transfer learning across domains and morphologies. Work transferring knowledge
between different domains in RL typically learns a mapping between the state
and action spaces. Ammar et al. (2015) use unsupervised manifold alignment to
find a linear map between states that have similar local geometry but assume
access to hand-crafted features. More recent work in transfer learning across
viewpoint and embodiment mismatch learn a state mapping without handcrafted
features but assume access to paired and time-aligned demonstration from both
domains (Gupta et al., 2017; Liu et al., 2018; Sermanet et al., 2018).
Furthermore, Kim et al. (2020); Raychaudhuri et al. (2021) propose methods to
learn a state mapping from unpaired and unaligned tasks. All these methods
require proxy tasks, _i.e_. a set of pairs of expert demonstrations from both
domains, which limit the applicability of these methods to real-world
settings. Stadie et al. (2017) have proposed to combine adversarial learning
and domain confusion to learn a policy in the agent’s domain without proxy
tasks but their method only works in the case of small viewpoint mismatch.
Zakka et al. (2021) take a goal-driven perspective that seeks to imitate task
progress rather than match fine-grained structural details to transfer between
physical robots. In contrast, our method does not rely on learning an explicit
cross-domain latent space between the agents, nor does it rely on proxy tasks.
The Gromov-Wasserstein distance enables us to directly compare the different
spaces without a shared space. The existing benchmark tasks we are aware of
assume access to a set of demonstrations from _both_ agents whereas the
experiments in our paper _only_ assume access to expert demonstrations.
Finally, other domain adaptation and transfer learning settings use Gromov-
Wasserstein variants, _e.g_. for transfer between word embedding spaces
(Alvarez-Melis & Jaakkola, 2018) and image spaces (Vayer et al., 2020b).

##  3 Preliminaries

Metric Markov Decision Process. An infinite-horizon discounted _Markov
decision Process (MDP)_ is a tuple $(S,A,R,P,p_{0},\gamma)$ where $S$ and $A$
are state and action spaces, $P:S\times A\rightarrow\Delta(S)$ is the
transition function, $R:S\times A\rightarrow\mathbb{R}$ is the reward
function, $p_{0}\in\Delta(S)$ is the initial state distribution and $\gamma$
is the discount factor. We equip MDPs with a distance $d:S\times
A\rightarrow\mathbb{R^{+}}$ and call the tuple $(S,A,R,P,p_{0},\gamma,d)$ a
_metric MDP_.

Gromov-Wasserstein distance. Let
$(\mathcal{X},d_{\mathcal{X}},\mu_{\mathcal{X}})$ and
$(\mathcal{Y},d_{\mathcal{Y}},\mu_{\mathcal{Y}})$ be two metric measure
spaces, where $d_{\mathcal{X}},d_{\mathcal{Y}}$ are distances, and
$\mu_{\mathcal{X}},\mu_{\mathcal{Y}}$ are measures on their respective
spaces222We use discrete spaces for readability but show empirical results in
continuous spaces.. Optimal transport (Villani, 2009; Peyré et al., 2019)
studies how to compare measures. We will use the _Gromov-Wasserstein_ distance
(Mémoli, 2011) between metric measure spaces, which has been theoretically
generalized and further studied in Sturm (2012); Peyré et al. (2016); Vayer
(2020) and is defined by

$\displaystyle\mathcal{GW}((\mathcal{X},d_{\mathcal{X}},\mu_{\mathcal{X}}),(\mathcal{Y},d_{\mathcal{Y}},\mu_{\mathcal{Y}}))^{2}=\min_{u\in\mathcal{U}(\mu_{\mathcal{X}},\mu_{\mathcal{Y}})}\sum_{\mathcal{X}^{2}\times\mathcal{Y}^{2}}|d_{\mathcal{X}}(x,x^{\prime})-d_{\mathcal{Y}}(y,y^{\prime})|^{2}u_{x,y}u_{x^{\prime},y^{\prime}},$
(1)

where $\mathcal{U}(\mu_{\mathcal{X}},\mu_{\mathcal{Y}})$ is the set of
couplings between the atoms of the measures defined by

$\mathcal{U}(\mu_{\mathcal{X}},\mu_{\mathcal{Y}})=\left\\{u\in\mathbb{R}^{\mathcal{X}\times\mathcal{Y}}\
\middle|\ \forall
x\in\mathcal{X},\sum_{y\in\mathcal{Y}}u_{x,y}=\mu_{\mathcal{X}}(x),\forall
y\in\mathcal{Y},\sum_{x\in\mathcal{X}}u_{x,y}=\mu_{\mathcal{Y}}(y)\right\\}.$

$\mathcal{GW}$ compares the structure of two metric measure spaces by
comparing the pairwise distances within each space to find the best isometry
between the spaces. Figure 1 illustrates this distance in the case of the
metric measure spaces $(S_{E}\times A_{E},d_{E},\rho_{\pi_{E}})$ and
$(S_{A}\times A_{A},d_{A},\rho_{\pi_{A}})$.

##  4 Cross-Domain Imitation Learning via Optimal Transport

###  4.1 Comparing policies from arbitrarily different MDPs

For a stationary policy $\pi$ acting on a metric MDP $(S,A,R,P,\gamma,d)$, the
_occupancy measure_ is:

$\rho_{\pi}:S\times
A\rightarrow\mathbb{R}\qquad\rho(s,a)=\pi(a|s)\sum_{t=0}^{\infty}\gamma^{t}P(s_{t}=s|\pi).$

We compare policies from arbitrarily different MDPs in terms of their
occupancy measures.

######  Definition 1 (Gromov-Wasserstein distance between policies).

Given an expert policy $\pi_{E}$ and an agent policy $\pi_{A}$ acting,
respectively, on

$M_{E}=(S_{E},A_{E},R_{E},P_{E},T_{E},d_{E})\quad{\rm and}\quad
M_{A}=(S_{A},A_{A},R_{A},P_{A},T_{A},d_{A}).$

We define the Gromov-Wasserstein distance between $\pi_{E}$ and $\pi_{A}$ as
the Gromov-Wasserstein distance between the metric measure spaces
$(S_{E}\times A_{E},d_{E},\rho_{\pi_{E}})$ and $(S_{A}\times
A_{A},d_{A},\rho_{\pi_{A}})$:

$\displaystyle\mathcal{GW}(\pi,\pi^{\prime})$
$\displaystyle=\mathcal{GW}((S_{E}\times
A_{E},d_{E},\rho_{\pi_{E}}),(S_{A}\times A_{A},d_{A},\rho_{\pi_{A}})).$ (2)

We now define an isometry between policies by comparing the distances between
the state-action spaces and show that $\mathcal{GW}$ defines a distance up to
an isometry between the policies. Figure 2 illustrates examples of simple
isometric policies.

######  Definition 2 (Isometric policies).

Two policies $\pi_{E}$ and $\pi_{A}$ are isometric if there exists a bijection
$\phi:\operatorname{supp}[\rho_{\pi_{E}}]\rightarrow\operatorname{supp}[\rho_{\pi_{A}}]$
that satisfies for all
$(s_{E},a_{E}),({s_{E}}^{\prime},{a_{E}}^{\prime})\in\operatorname{supp}[\rho_{\pi_{E}}]^{2}$:

$d_{E}\left((s_{E},a_{E}),({s_{E}}^{\prime},{a_{E}}^{\prime})\right)=d_{A}\left(\phi(s_{E},a_{E}),\phi({s_{E}}^{\prime},{a_{E}}^{\prime})\right)$

In other words, $\phi$ is an isometry between
$(\operatorname{supp}[\rho_{\pi_{E}}],d_{E})$ and
$(\operatorname{supp}[\rho_{\pi_{A}}],d_{A})$.

######  Proposition 1.

$\mathcal{GW}$ defines a metric on the collection of all isometry classes of
policies.

###### Proof.

By definition 1, $\mathcal{GW}(\pi_{E},\pi_{A})=0$ if and only if
$\mathcal{GW}((S_{E},d_{E},\rho_{\pi_{E}}),(S_{A},d_{A},\rho_{\pi_{A}}))=0$.
By Mémoli (2011, Theorem 5.1), this is true if and only if there is an
isometry that maps $\operatorname{supp}[\rho_{\phi_{E}}]$ to
$\operatorname{supp}[\rho_{\phi_{A}}]$. By definition 2, this is true if and
only if $\pi_{A}$ and $\pi_{E}$ are isometric. The symmetry and triangle
inequality follow from Mémoli (2011, Theorem 5.1). ∎

The next theorem333Our proof is in finite state-action spaces for readability
and can be directly extended to infinite spaces. gives a sufficient condition
to recover, by minimizing $\mathcal{GW}$, an optimal policy444A policy is
optimal in the MDP $(S,A,R,P,\gamma,d)$ if it maximizes the expected return
$\mathbb{E}\sum_{t=0}^{\infty}R(s_{t},a_{t})$. in the agent’s domain up to an
isometry.

######  Theorem 1.

Consider two MDPs

$M_{E}=(S_{E},A_{E},R_{E},P_{E},p_{E},\gamma)\quad{\rm and}\quad
M_{A}=(S_{A},A_{A},R_{A},P_{A},p_{A},\gamma).$

Suppose that there exists four distances
$d_{E}^{S},d_{E}^{A},d_{A}^{S},d_{A}^{A}$ defined on $S_{E}$, $A_{E}$, $S_{A}$
and $A_{E}$ respectively, and two isometries
$\phi:(S_{E},d_{E}^{S})\rightarrow(S_{A},d_{A}^{S})$ and
$\psi:(A_{E},d_{E}^{S})\rightarrow(A_{S},d_{A}^{S})$ such that for all
$(s_{E},a_{E},s_{E}^{\prime})\in S_{E}\times A_{E}\times S_{E}$ the three
following conditions hold:

$\displaystyle R(s_{E},a_{E})=R_{A}(\phi(s_{E}),\psi(a_{E}))$ (3)
$\displaystyle{P_{E}}_{s_{E},a_{E}}(s_{E}^{\prime})={P_{A}}_{\phi(s_{E})\psi(a_{E})}(\phi(s_{E}^{\prime}))$
(4) $\displaystyle p_{E}(s_{E})=p_{A}(\phi(s_{E})).$ (5)

Consider an optimal policy $\pi^{*}_{E}$ in $M_{E}$. Suppose that $\pi_{GW}$
minimizes $\mathcal{GW}(\pi^{*}_{E},\pi_{GW})$ with

$d_{E}:(s_{E},a_{E})\mapsto d_{E}^{S}(s_{E})+d_{E}^{A}(a_{E})\quad{\rm
and}\quad d_{A}:(s_{A},a_{A})\mapsto d_{A}^{S}(s_{A})+d^{A}_{A}(a_{A}).$

Then $\pi_{GW}$ is isometric to an optimal policy in $M_{A}$.

###### Proof.

Consider the occupancy measure $\rho^{*}_{A}:S_{A}\times
A_{A}\rightarrow\mathbb{R}$ given by

$(s_{A},a_{A})\mapsto\rho_{\pi^{*}_{E}}(\phi^{-1}(s_{A}),\psi^{-1}(a_{A})).$

We first show that $\rho^{*}_{A}$ is feasible in $M_{A}$, _i.e_. there exists
a policy $\pi^{*}_{A}$ acting in $M_{A}$ with occupancy measure $\rho^{*}_{A}$
(a). Then we show that $\pi^{*}_{A}$ is optimal in $M_{A}$ (b) and is
isometric to $\pi^{*}_{E}$ (c). Finally we show that $\pi_{GW}$ is isometric
to $\pi^{*}_{A}$, which concludes the proof (d).

(a) Consider $s_{A}\in S_{A}$. By definition of $\rho^{*}_{A}$,

$\sum_{a_{A}\in A_{A}}\rho^{*}_{A}(s_{A})=\sum_{a_{A}\in
A_{A}}\rho_{\pi^{*}_{E}}(\phi^{-1}(s_{A}),\psi^{-1}(a_{A}))=\sum_{a_{E}\in
A_{E}}\rho_{\pi^{*}_{E}}(\phi^{-1}(s_{A}),a_{E}).$

Since $\rho_{\pi^{*}_{E}}$ is feasible in $M$, it follows from Puterman (2014,
Theorem 6.9.1) that

$\sum_{a_{E}\in
A_{E}}\rho_{\pi^{*}_{E}}(\phi^{-1}(s_{A}),a_{E})=p_{E}(\phi^{-1}(s_{A}))+\gamma\sum_{s_{E}\in
S_{E},a_{E}\in
A_{E}}{P_{E}}_{s_{E},a_{E}}(\phi^{-1}(s_{A}))+\rho_{\pi^{*}_{E}}(s_{E},a_{E}).$

By conditions 4 and 5 and by definition of $\rho^{*}_{A}$,

$\displaystyle p_{E}(\phi^{-1}(s_{A}))+\gamma\sum_{s_{E}\in S_{E},a_{E}\in
A_{E}}{P_{E}}_{s_{E},a_{E}}(\phi^{-1}(s_{A}))+\rho_{\pi^{*}_{E}}(s_{E},a_{E})$
$\displaystyle\hskip 8.53581pt=p_{A}(s_{A})+\gamma\sum_{s_{E}\in
S_{E},a_{E}\in
A_{E}}{P_{A}}_{\phi(s_{E}),\psi(a_{E})}(s_{A})+\rho^{*}_{A}(\phi(s_{E}),\psi(a_{E}))$
$\displaystyle\hskip 8.53581pt=p_{A}(s_{A})+\gamma\sum_{s_{A}^{\prime}\in
S_{A},a_{A}\in
A_{A}}{P_{A}}_{s_{A}^{\prime},a_{A}}(s_{A})+\rho^{*}_{A}(s_{A}^{\prime},a_{A}).$

It follows that

$\sum_{a_{A}\in
A_{A}}\rho^{*}_{A}(s_{A})=p_{A}(s_{A})+\gamma\sum_{s_{A}^{\prime}\in
S_{A},a_{A}\in
A_{A}}{P_{A}}_{s_{A}^{\prime},a_{A}}(s_{A})+\rho^{*}_{A}(s_{A}^{\prime},a_{A}).$

Therefore, by Puterman (2014, Theorem 6.9.1), $\rho^{*}_{A}$ is feasible in
$M_{A}$, _i.e_. there exists a policy $\pi^{*}_{A}$ acting in $M_{A}$ with
occupancy measure $\rho^{*}_{A}$.

(b) By condition 5 and definition of $\rho^{*}_{A}$, the expected return of
$\pi^{*}_{A}$ in $M_{A}$ is then

$\displaystyle\sum_{s_{A}\in S_{A},a_{A}\in
A_{A}}\rho^{*}_{A}(s_{A},a_{A})R_{A}(s_{A},a_{A})$ $\displaystyle\hskip
28.45274pt=\sum_{s_{A}\in S_{A},a_{A}\in
A_{A}}\rho^{*}_{E}(\phi^{-1}(s_{A}),\psi^{-1}(a_{A}))R_{E}(\phi^{-1}(s_{A}),\psi^{-1}(a_{A}))$
$\displaystyle\hskip 28.45274pt=\sum_{s_{E}\in S_{E},a_{E}\in
A_{E}}\rho^{*}_{E}(s_{E},a_{E})R_{E}(s_{E},a_{E})$

Consider any policy $\pi_{A}$ in $M^{\prime}$. By condition 5, the expected
return of $\pi_{A}$ is

$\sum_{s_{A}\in S_{A},a_{A}\in
A_{A}}\rho_{\pi_{A}}(s_{A},a_{A})R_{A}(s_{A},a_{A})=\sum_{s_{E}\in
S_{E},a_{E}\in
A_{E}}\rho_{\pi_{A}}(\phi(s_{E}),\psi(a_{E}))R_{E}(s_{E},a_{E}).$

Using the same arguments that we used to show that $\rho^{*}_{A}$ is feasible
in $M^{\prime}$, we can show that

$(s_{E},a_{E})\mapsto\rho_{\pi_{A}}(\phi(s_{E}),\psi(a_{E}))$

is feasible in $M$. It follows by optimality of $\pi^{*}_{E}$ in $M$ that

$\displaystyle\sum_{s_{E}\in S_{E},a_{E}\in
A_{E}}\rho_{\pi_{A}}(\phi(s_{E}),\psi(a_{E}))R_{E}(s_{E},a_{E})$
$\displaystyle\leq\sum_{s_{E}\in S_{E},a_{E}\in
A_{E}}\rho_{\pi^{*}_{E}}(\phi(s_{E}),\psi(a_{E}))R_{E}(s_{E},a_{E})$
$\displaystyle=\sum_{s_{A}\in S_{A},a_{A}\in
A_{A}}\rho^{*}_{A}(s_{A},a_{A})R_{A}(s_{A},a_{A}).$

It follows that $\pi^{*}_{A}$ is optimal in $M^{\prime}$.

(c) Notice that

$\xi:(s_{E},a_{E})\mapsto(\phi(s_{E}),\psi(a_{E}))$

is an isometry between $(S_{E}\times A_{E},d_{E})$ and $(S_{A}\times
A_{A},d_{A})$, where $d_{E}$ and $d_{A}$ and given, resp., by

$(s_{E},a_{E})\mapsto d^{S}_{E}(s_{E})+d^{A}_{E}(a_{E})\quad{\rm
and}\quad(s_{A},a_{A})\mapsto d^{S}_{A}(s_{A})+d^{A}_{A}(a_{A}).$

Therefore by definition of $\rho^{*}_{A}$, $\pi^{*}_{A}$ is isometric to
$\pi^{*}_{E}$.

(d) Recall from the statement of the theorem that $\pi_{GW}$ is a minimizer of
$\mathcal{GW}(\pi^{*}_{E},\pi_{GW})$. Since $\pi^{*}_{A}$ is isometric to
$\pi^{*}_{E}$, it follows from prop. 1 that
$\mathcal{GW}(\pi^{*}_{E},\pi^{*}_{A})=0$. Therefore
$\mathcal{GW}(\pi^{*}_{E},\pi_{GW})$ must be 0. By prop. 1, it follows that
there exists an isometry

$\chi:(\operatorname{supp}[\rho^{*}_{E}],d_{E})\rightarrow(\operatorname{supp}[\rho_{\pi_{GW}}],d_{A}).$

Notice that $\chi\circ\xi^{-1}|_{\operatorname{supp}[\rho^{*}_{A}]}$ is an
isometry from $(\operatorname{supp}[\rho^{*}_{A}],d_{A})$ to
$(\operatorname{supp}[\rho_{\pi_{GW}}],d_{A})$. It follows that $\pi_{GW}$ is
isometric to $\pi^{*}_{A}$, an optimal policy in $M_{A}$, which concludes the
proof. ∎

######  Remark 1.

Theorem 1 shows the possibilities and limitations of our method. It shows that
our method can recover optimal policies even though arbitrary isometries are
applied to the state and action spaces of the expert’s domain. Importantly, we
don’t need to know the isometries, hence our method is applicable to a wide
range of settings. We will show empirically that our method produces strong
results in other settings where the environment are not isometric and don’t
even have the same dimension. However, a limitation of our method is that it
recovers optimal policy only up to isometries. We will see that in practice,
running our method on different seeds enables to find an optimal policy in the
agent’s domain.

Inputs: expert demonstration $\tau$, metrics on the expert ($d_{E}$) and agent
($d_{A}$) space

Initialize the imitation agent’s policy $\pi_{\theta}$ and value estimates
$V_{\theta}$

while Unconverged do

Collect an episode $\tau^{\prime}$

Compute $\mathcal{GW}(\tau,\tau^{\prime})$

Set pseudo-rewards $r$ with eq. 7

Update $\pi_{\theta}$ and $V_{\theta}$ to optimize the pseudo-rewards

end while

Algorithm 1 Gromov-Wasserstein imitation learning from a single expert
demonstration.

###  4.2 Gromov-Wasserstein Imitation Learning

Minimizing $\mathcal{GW}$ between an expert and agent requires derivatives
through the transition dynamics, which we typically don’t have access to. We
introduce a reward proxy suitable for training an agent’s policy that
minimizes $\mathcal{GW}$ via RL. Figure 1 illustrates the method. For
readability, we combine expert state and action variables $(s_{E},a_{E})$ into
single variables $z_{E}$, and similarly for agent state-action pairs. Also, we
define $Z_{E}=S_{E}\times A_{E}$ and $Z_{A}=S_{A}\times A_{A}$.

######  Definition 3.

Given an expert policy $\pi_{E}$ and an agent policy $\pi_{A}$, the Gromov-
Wasserstein reward of the agent is defined as
$r_{\mathcal{GW}}:\operatorname{supp}[\rho_{\pi_{A}}]\rightarrow\mathbb{R}$
given by

$r_{\mathcal{GW}}(z_{A})=-\frac{1}{\rho_{\pi}(z_{A})}\sum_{\begin{subarray}{c}z_{E}\in
Z_{E}\\\ z^{\prime}_{E}\in Z_{E}\\\ z_{A}^{\prime}\in
Z_{A}\end{subarray}}|d_{E}(z_{E},z_{E}^{\prime}))-d_{A}(z_{A},z_{A}^{\prime})|^{2}u^{\star}_{z_{E},z_{A}}u^{\star}_{z_{E}^{\prime},z_{A}^{\prime}}$

where $u^{\star}$ is the coupling minimizing objective 1.

######  Proposition 2.

The agent’s policy $\pi_{A}$ trained with $r_{\mathcal{GW}}$ minimizes
$\mathcal{GW}(\pi_{E},\pi_{A})$.

###### Proof.

Suppose that $\pi_{A}$ maximizes
$\mathbb{E}(\sum_{t=0}^{\infty}\gamma^{t}r_{\mathcal{GW}}(s^{A}_{t},a^{A}_{t}))$
and denote by $\rho_{\pi_{A}}$ its occupancy measure. By Puterman (2014,
Theorem 6.9.4), $\pi_{A}$ maximizes the following objective:

$\displaystyle\mathop{\mathbb{E}}_{z_{A}\sim\rho_{\pi_{A}}}r_{\mathcal{GW}}(z_{A})=$
$\displaystyle-\sum_{z_{A}\in\operatorname{supp}[\rho_{\pi_{A}}]}\frac{\rho_{\pi_{A}}(z_{A})}{\rho_{\pi_{A}}(z_{A})}\sum_{\begin{subarray}{c}z_{E}\in
Z_{E}\\\ z^{\prime}_{E}\in Z_{E}\\\ z^{\prime}_{A}\in
Z_{A}\end{subarray}}|d_{E}(z_{E},z^{\prime}_{E})-d_{A}(z_{A},z^{\prime}_{A})|^{2}u^{\star}_{z_{A},z_{E}}u^{\star}_{z^{\prime}_{A},z^{\prime}_{E}}$
$\displaystyle=$ $\displaystyle-\sum_{\begin{subarray}{c}z_{E}\in Z_{E}\\\
z^{\prime}_{E}\in Z_{E}\\\ z_{A}\in Z_{A}\\\ z^{\prime}_{A}\in
Z_{A}\end{subarray}}|d_{E}(z_{E},z^{\prime}_{E})-d_{A}(z_{A},z^{\prime}_{A})|^{2}u^{\star}_{z_{A},z_{E}}u^{\star}_{z^{\prime}_{A},z^{\prime}_{E}}$
$\displaystyle=$ $\displaystyle-\mathcal{GW}^{2}(\pi_{E},\pi_{A})$

∎

In practice we approximate the occupancy measures of $\pi$ by
$\hat{\rho}_{\pi}(s,a)=\frac{1}{T}\sum_{t=1}^{T}\mathbb{1}(s=s_{t}\wedge
a=a_{t})$ where $\tau=(s_{1},a_{1},..,s_{T},a_{T})$ is a finite trajectory
collected with $\pi$. Assuming that all state-action pairs in the trajectory
are different555We can add the time step to the state to distinguish between
two identical state-action pairs in the trajectory., $\hat{\rho}$ is a uniform
distribution. Given an expert trajectory $\tau_{E}$ and an agent trajectory
$\tau_{A}$ 666Note that the Gromov-Wasserstein distance defined in equ. (6)
does not depend on the temporal ordering of the trajectories., the (squared)
Gromov-Wasserstein distance between the empirical occupancy measures is

$\displaystyle\mathcal{GW}^{2}(\tau_{E},\tau_{A})=\min_{\theta\in\Theta^{T_{E}\times
T_{A}}}\sum_{\begin{subarray}{c}1\leq i,i^{\prime}\leq T_{E}\\\ 1\leq
j,j^{\prime}\leq
T_{A}\end{subarray}}|d_{E}((s^{E}_{i},a^{E}_{i}),(s^{E}_{i^{\prime}},a^{E}_{i^{\prime}}))-d_{A}((s^{A}_{j},s^{A}_{j}),(s^{A}_{j^{\prime}},a^{A}_{j^{\prime}}))|^{2}\theta_{i,j}\theta_{i^{\prime},j^{\prime}}$
(6)

where $\Theta$ is the set of is the set of couplings between the atoms of the
uniform measures defined by

$\Theta^{T\times T^{\prime}}=\left\\{\theta\in\mathbb{R}^{T\times T^{\prime}}\
\middle|\ \forall i\in[T],\sum_{j\in[T^{\prime}]}\theta_{i,j}=1/T,\forall
j\in[T^{\prime}],\sum_{i\in[T]}\theta_{i,j}=1/T^{\prime}\right\\}.$

In this case the reward is given for every state-action pairs in the
trajectory by:

$\displaystyle r(s^{A}_{j},s^{A}_{j})=-T_{A}\sum_{\begin{subarray}{c}1\leq
i,i^{\prime}\leq T_{E}\\\ 1\leq j^{\prime}\leq
T_{A}\end{subarray}}|d_{E}((s^{E}_{i},a^{E}_{i}),(s^{E}_{i^{\prime}},a^{E}_{i^{\prime}}))-d_{A}((s^{A}_{j},s^{A}_{j}),(s^{A}_{j^{\prime}},a^{A}_{j^{\prime}}))|^{2}\theta^{\star}_{i,j}\theta^{\star}_{i^{\prime},j^{\prime}}$
(7)

where $\theta^{\star}$ is the coupling minimizing objective 6.

In practice we drop the factor $T_{A}$ because it is the same for every state-
action pairs in the trajectory.

######  Remark 2.

The construction of our reward proxy is defined for any occupancy measure and
extends to previous work optimizing optimal transport quantities via RL that
assumes uniform occupancy measure in the form of a trajectory to bypass the
need for derivatives through the transition dynamics (Dadashi et al., 2020;
Papagiannis & Li, 2020).

Computing the pseudo-rewards. We compute the Gromov-Wasserstein distance using
Peyré et al. (2016, Proposition 1) and its gradient using Peyré et al. (2016,
Proposition 2). To compute the coupling minimizing 6, we use the conditional
gradient method as discussed in Ferradans et al. (2013).

Optimizing the pseudo-rewards. The pseudo-rewards we obtain from
$\mathcal{GW}$ for the imitation agent enable us to turn the imitation
learning problem into a reinforcement learning problem (Sutton & Barto, 2018)
to find the optimal policy for the Markov decision process induced by the
pseudo-rewards. We consider agents with continuous state-action spaces and
thus do policy optimization with the soft actor-critic algorithm (Haarnoja et
al., 2018). Algorithm 1 sums up GWIL in the case where a single expert
trajectory is given to approximate the expert occupancy measure.

##  5 Experiments

We propose a benchmark set for cross-domain IL methods consisting of 3 tasks
and aiming at answering the following questions:

  1. 1.

_Does GWIL recover optimal behaviors when the agent domain is a rigid
transformation of the expert domain?_ Yes, we demonstrate this with the maze
in sect. 5.1.

  2. 2.

_Can GWIL recover optimal behaviors when the agent has different state and
action spaces than the expert?_ Yes, we show in sect. 5.2 for _slightly_
different state-action spaces between the cartpole and pendulum, and in sect.
5.3 for _significantly_ different spaces between a walker and cheetah.

To answer these three questions, we use simulated continuous control tasks
implemented in Mujoco (Todorov et al., 2012) and the DeepMind control suite
(Tassa et al., 2018). We include videos of learned policies on our project
site777https://arnaudfickinger.github.io/gwil/. In all settings we use the
Euclidean metric within the expert and agent spaces for $d_{E}$ and $d_{A}$.

###  5.1 Agent domain is a rigid transformation of the expert domain

We evaluate the capacity of IL methods to transfer to rigid transformation of
the expert domain by using the PointMass Maze environment from Hejna et al.
(2020). The agent’s domain is obtained by applying a reflection to the
expert’s maze. This task satisfies the condition of theorem 1 with $\phi$
being the reflection through the central horizontal plan and $\psi$ being the
reflection through the $x$-axis in the action space. Therefore by theorem 1,
the agent’s optimal policy should be isometric to the policy trained using
GWIL. By looking at the geometry of the maze, it is clear that every policy in
the isometry class of an optimal policy is optimal. Therefore we expect GWIL
to recover an optimal policy in the agent’s domain. Figure 3 shows that GWIL
indeed recovers an optimal policy.

(a) Expert

(b) Agent

Figure 3: Given a single expert trajectory in the expert’s domain (a), GWIL
recovers an optimal policy in the agent’s domain (b) without any external
reward, as predicted by theorem 1. The green dot represents the initial state
position and the episode ends when the agent reaches the goal represented by
the red square. Figure 4:  Given a single expert trajectory in the pendulum’s
domain (above), GWIL recovers the optimal behavior in the agent’s domain
(cartpole, below) without any external reward.

###  5.2 Agent and the expert have slightly different state and action spaces

We evaluate here the capacity of IL methods to transfer to transformation that
does not have to be rigid but description map should still be apparent by
looking at the domains. A good example of such transformation is the one
between the pendulum and cartpole. The pendulum is our expert’s domain while
cartpole constitutes our agent’s domain. The expert is trained on the swingup
task. Even though the transformation is not rigid, GWIL is able to recover the
optimal behavior in the agent’s domain as shown in fig. 4. Notice that
pendulum and cartpole do not have the same state-action space dimension: The
pendulum has 3 dimensions while the cartpole has 5 dimensions. Therefore GWIL
can indeed be applied to transfer between problems with different dimension.

###  5.3 Agent and the expert have significantly different state and action
spaces

We evaluate here the capacity of IL methods to transfer to non-trivial
transformation between domains. A good example of such transformation is two
arbitrarily different morphologies from the DeepMind Control Suite such as the
cheetah and walker. The cheetah constitutes our expert’s domain while the
walker constitutes our agent’s domain. The expert is trained on the run task.

Although the mapping between these two domains is not trivial, minimizing the
Gromov-Wasserstein solely enables the walker to interestingly learn to move
backward and forward by imitating a cheetah. Since the isometry class of the
optimal policy – moving forward– of the cheetah and walker contains a
suboptimal element –moving backward–, we expect GWIL to recover one of these
two trajectories. Indeed, depending on the seed used, GWIL produces a cheetah-
imitating walker moving forward or a cheetah-imitating walker moving backward,
as shown in fig. 5.

Figure 5:  Given a single expert trajectory in the cheetah’s domain (above),
GWIL recovers the two elements of the optimal policy’s isometry class in the
agent’s domain (walker), moving forward which is optimal (middle) and moving
backward which is suboptimal (below). Interestingly, the resulting walker
behaves like a cheetah.

##  6 Conclusion

Our work demonstrates that optimal transport distances are a useful
foundational tool for cross-domain imitation across incomparable spaces.
Future directions include exploring:

  1. 1.

Scaling to more complex environments and agents towards the goal of
transferring the structure of many high-dimensional demonstrations of complex
tasks into an agent.

  2. 2.

The use of $\mathcal{GW}$ to help agents explore in extremely sparse-reward
environments when we have expert demonstrations available from other agents.

  3. 3.

How $\mathcal{GW}$ compares to other optimal transport distances that work
apply between two metric MDPs, such as Alvarez-Melis et al. (2019), that have
more flexibility over how the spaces are connected and what invariances the
coupling has.

  4. 4.

Metrics aware of the MDP’s temporal structure such as Zhou & Torre (2009);
Vayer et al. (2020a); Cohen et al. (2021) that build on dynamic time warping
(Müller, 2007). The Gromov-Wasserstein ignores the temporal information and
ordering present within the trajectories.

## References

  * Abbeel & Ng (2004) Pieter Abbeel and Andrew Y Ng.  Apprenticeship learning via inverse reinforcement learning.  In _Proceedings of the twenty-first international conference on Machine learning_ , pp. 1, 2004.
  * Abbeel et al. (2010) Pieter Abbeel, Adam Coates, and Andrew Y Ng.  Autonomous helicopter aerobatics through apprenticeship learning.  _The International Journal of Robotics Research_ , 29(13):1608–1639, 2010.
  * Alvarez-Melis & Jaakkola (2018) David Alvarez-Melis and Tommi S. Jaakkola.  Gromov-wasserstein alignment of word embedding spaces.  In _Empirical Methods in Natural Language Processing_ , 2018.
  * Alvarez-Melis et al. (2019) David Alvarez-Melis, Stefanie Jegelka, and Tommi S. Jaakkola.  Towards optimal transport with global invariances.  In Kamalika Chaudhuri and Masashi Sugiyama (eds.), _Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics_ , volume 89 of _Proceedings of Machine Learning Research_ , pp. 1870–1879. PMLR, 16–18 Apr 2019.  URL https://proceedings.mlr.press/v89/alvarez-melis19a.html.
  * Ammar et al. (2015) Haitham Bou Ammar, Eric Eaton, Paul Ruvolo, and Matthew E Taylor.  Unsupervised cross-domain transfer in policy gradient reinforcement learning via manifold alignment.  In _Twenty-Ninth AAAI Conference on Artificial Intelligence_ , 2015\.
  * Aytar et al. (2018) Yusuf Aytar, Tobias Pfaff, David Budden, Tom Le Paine, Ziyu Wang, and Nando de Freitas.  Playing hard exploration games by watching youtube.  _arXiv preprint arXiv:1805.11592_ , 2018.
  * Berner et al. (2019) Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al.  Dota 2 with large scale deep reinforcement learning.  _arXiv preprint arXiv:1912.06680_ , 2019.
  * Cohen et al. (2021) Samuel Cohen, Giulia Luise, Alexander Terenin, Brandon Amos, and Marc Deisenroth.  Aligning time series on incomparable spaces.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 1036–1044. PMLR, 2021.
  * Dadashi et al. (2020) Robert Dadashi, Léonard Hussenot, Matthieu Geist, and Olivier Pietquin.  Primal wasserstein imitation learning.  _arXiv preprint arXiv:2006.04678_ , 2020.
  * Ferradans et al. (2013) Sira Ferradans, Nicolas Papadakis, Julien Rabin, Gabriel Peyré, and Jean-François Aujol.  Regularized discrete optimal transport.  In _International Conference on Scale Space and Variational Methods in Computer Vision_ , pp. 428–439. Springer, 2013.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27, 2014.
  * Gupta et al. (2017) Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine.  Learning invariant feature spaces to transfer skills with reinforcement learning.  _arXiv preprint arXiv:1703.02949_ , 2017.
  * Haarnoja et al. (2018) Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al.  Soft actor-critic algorithms and applications.  _arXiv preprint arXiv:1812.05905_ , 2018.
  * Heess et al. (2017) Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, et al.  Emergence of locomotion behaviours in rich environments.  _arXiv preprint arXiv:1707.02286_ , 2017.
  * Hejna et al. (2020) Donald Hejna, Lerrel Pinto, and Pieter Abbeel.  Hierarchically decoupled imitation for morphological transfer.  In _International Conference on Machine Learning_ , pp. 4159–4171. PMLR, 2020.
  * Ho & Ermon (2016) Jonathan Ho and S. Ermon.  Generative adversarial imitation learning.  In _NIPS_ , 2016.
  * Kim et al. (2020) Kuno Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon.  Domain adaptive imitation learning.  In _International Conference on Machine Learning_ , pp. 5286–5295. PMLR, 2020.
  * Kober et al. (2013) Jens Kober, J Andrew Bagnell, and Jan Peters.  Reinforcement learning in robotics: A survey.  _The International Journal of Robotics Research_ , 32(11):1238–1274, 2013.
  * Levine et al. (2016) Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel.  End-to-end training of deep visuomotor policies.  _The Journal of Machine Learning Research_ , 17(1):1334–1373, 2016.
  * Liu et al. (2019) Fangchen Liu, Zhan Ling, Tongzhou Mu, and Hao Su.  State alignment-based imitation learning.  _arXiv preprint arXiv:1911.10947_ , 2019.
  * Liu et al. (2018) YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine.  Imitation from observation: Learning to imitate behaviors from raw video via context translation.  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 1118–1125. IEEE, 2018.
  * Mémoli (2011) Facundo Mémoli.  Gromov–wasserstein distances and the metric approach to object matching.  _Foundations of computational mathematics_ , 11(4):417–487, 2011.
  * Müller (2007) Meinard Müller.  Dynamic time warping.  _Information retrieval for music and motion_ , pp. 69–84, 2007\.
  * Ng & Russell (2000) Andrew Y. Ng and Stuart J. Russell.  Algorithms for inverse reinforcement learning.  In _Proceedings of the Seventeenth International Conference on Machine Learning_ , ICML ’00, pp. 663–670, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc.  ISBN 1558607072.
  * Papagiannis & Li (2020) Georgios Papagiannis and Yunpeng Li.  Imitation learning with sinkhorn distances.  _arXiv preprint arXiv:2008.09167_ , 2020.
  * Peng et al. (2018) Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne.  Deepmimic: Example-guided deep reinforcement learning of physics-based character skills.  _ACM Transactions on Graphics (TOG)_ , 37(4):1–14, 2018.
  * Peyré et al. (2016) Gabriel Peyré, Marco Cuturi, and Justin Solomon.  Gromov-wasserstein averaging of kernel and distance matrices.  In _International Conference on Machine Learning_ , pp. 2664–2672. PMLR, 2016.
  * Peyré et al. (2019) Gabriel Peyré, Marco Cuturi, et al.  Computational optimal transport: With applications to data science.  _Foundations and Trends® in Machine Learning_ , 11(5-6):355–607, 2019.
  * Pomerleau (1988) D. Pomerleau.  Alvinn: An autonomous land vehicle in a neural network.  In _NIPS_ , 1988.
  * Pomerleau (1991) D. Pomerleau.  Efficient training of artificial neural networks for autonomous navigation.  _Neural Computation_ , 3:88–97, 1991.
  * Puterman (2014) Martin L Puterman.  _Markov decision processes: discrete stochastic dynamic programming_.  John Wiley & Sons, 2014.
  * Raychaudhuri et al. (2021) Dripta S Raychaudhuri, Sujoy Paul, Jeroen van Baar, and Amit K Roy-Chowdhury.  Cross-domain imitation from observations.  _arXiv preprint arXiv:2105.10037_ , 2021.
  * Sermanet et al. (2018) Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain.  Time-contrastive networks: Self-supervised learning from video.  In _2018 IEEE international conference on robotics and automation (ICRA)_ , pp. 1134–1141. IEEE, 2018.
  * Stadie et al. (2017) Bradly C Stadie, Pieter Abbeel, and Ilya Sutskever.  Third-person imitation learning.  _arXiv preprint arXiv:1703.01703_ , 2017.
  * Sturm (2012) Karl-Theodor Sturm.  The space of spaces: curvature bounds and gradient flows on the space of metric measure spaces.  _arXiv preprint arXiv:1208.0434_ , 2012.
  * Sutton & Barto (2018) Richard S Sutton and Andrew G Barto.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Tassa et al. (2018) Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al.  Deepmind control suite.  _arXiv preprint arXiv:1801.00690_ , 2018.
  * Todorov et al. (2012) Emanuel Todorov, Tom Erez, and Yuval Tassa.  Mujoco: A physics engine for model-based control.  In _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pp. 5026–5033. IEEE, 2012.
  * Vayer (2020) Titouan Vayer.  A contribution to optimal transport on incomparable spaces.  _arXiv preprint arXiv:2011.04447_ , 2020.
  * Vayer et al. (2020a) Titouan Vayer, L. Chapel, N. Courty, Rémi Flamary, Yann Soullard, and R. Tavenard.  Time series alignment with global invariances.  _ArXiv_ , abs/2002.03848, 2020a.
  * Vayer et al. (2020b) Titouan Vayer, Ievgen Redko, Rémi Flamary, and Nicolas Courty.  Co-optimal transport.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), _Advances in Neural Information Processing Systems_ , volume 33, pp. 17559–17570. Curran Associates, Inc., 2020b.  URL https://proceedings.neurips.cc/paper/2020/file/cc384c68ad503482fb24e6d1e3b512ae-Paper.pdf.
  * Villani (2009) Cédric Villani.  _Optimal transport: old and new_ , volume 338.  Springer, 2009.
  * Vinyals et al. (2019) Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al.  Grandmaster level in starcraft ii using multi-agent reinforcement learning.  _Nature_ , 575(7782):350–354, 2019.
  * Zakka et al. (2021) Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg, and Debidatta Dwibedi.  Xirl: Cross-embodiment inverse reinforcement learning.  _arXiv preprint arXiv:2106.03911_ , 2021.
  * Zhou & Torre (2009) Feng Zhou and Fernando Torre.  Canonical time warping for alignment of human behavior.  _Advances in neural information processing systems_ , 22:2286–2294, 2009.
  * Zhu et al. (2018) Yuke Zhu, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool, János Kramár, Raia Hadsell, Nando de Freitas, et al.  Reinforcement and imitation learning for diverse visuomotor skills.  _arXiv preprint arXiv:1802.09564_ , 2018.

##  Appendix A Optimization of the proxy reward

In this section we show that the proxy reward introduced in equation 7
constitutes a learning signal that is easy to optimize using standards RL
algorithms. Figure 6 shows proxy reward curves across 5 different seeds for
the 3 environments. We observe that in each environment the SAC learner
converges quickly and consistently to the asymptotic episodic return. Thus
there is reason to think that the proxy reward introduced in equation 7 will
be similarly easy to optimize in other cross-domain imitation settings.

Figure 6:  The proxy reward introduced in equation 7 gives a learning signal
that is easily optimized using a standard RL algorithm.

##  Appendix B Transfer to sparse-reward environments

In this section we show that GWIL can be used to facilitate learning in
sparse-reward environments when the learner has only access to one expert
demonstration from another domain. We compare GWIL to a baseline learner
having access to a single demonstration from the same domain and minimizing
the Wasserstein distance, as done in Dadashi et al. (2020). In these
experiments, both agents are given a sparse reward signal in addition to their
respective optimal transport proxy reward. We perform experiments in two
sparse-reward environment. In the first environment, the agent controls a
point mass in a maze and obtain a non-zero reward only if it reaches the end
of the maze. In the second environment, which is a sparse version of cartpole,
the agent controls a cartpole and obtains a non-zero reward only if he can
maintain the cartpole up for 10 consecutive time steps. Note that a SAC agent
fails to learn any meaningful behavior in both environments. Figure 7 shows
that GWIL is competitive with the baseline learner in the sparse maze
environment even though GWIL has only access to a demonstration from another
domain, while the baseline learner has access to a demonstration from the same
domain. Thus there is reason to think that GWIL efficiently and reliably
extracts useful information from the expert domain and hence should work well
in other cross-domain imitation settings.

Figure 7:  In sparse-reward environments, GWIL obtains similar performance
than a baseline learner minimizing the Wasserstein distance to an expert in
the same domain.

##  Appendix C Scalability of GWIL

In this section we show that our implementation of GWIL offers good
performance in terms of wall-clock time. Note that the bottleneck of our
method is in the computation of the optimal coupling which only depends on the
number of time steps in the trajectories, and not on the dimension of the
expert and the agent. Hence our method naturally scales with the dimension of
the problems. Furthermore, while we have not used any entropy regularizer in
our experiments, entropy regularized methods have been introduced to enable
Gromov-Wasserstein to scale to demanding machine learning tasks and can be
easily incorporated into our code to further improve the scalability. Figure 8
compares the time taken by GWIL in the maze with the time taken by the
baseline learner introduced in the previous section. It shows that imitating
with Gromov-Wasserstein requires the same order of time than imitating with
Wasserstein. Figure 9 compares the wall-clock time taken by a walker imitating
a cheetah using GWIL to reach a walking speed (i.e., a horizontal velocity of
1) and the wall-clock time taken by a SAC walker trained to run. It shows that
a GWIL walker imitating a cheetah reaches a walking speed faster than a SAC
agent trained to run. Even though the SAC agent is optimizing for standing in
addition to running, it was not obvious that GWIL could compete with SAC in
terms of wall-clock time. These results gives hope that GWIL has the potential
to scale to more complex problems (possibly with an additional entropy
regularizer) and be a useful way to learn by analogy.

Figure 8:  In the sparse maze environment, GWIL requieres the same order of
wall-clock time than a baseline learner minimizing the Wasserstein distance to
an expert in the same domain.  Figure 9:  A GWIL walker imitating a cheetah
reaches a walking speed faster than a SAC walker trained to run in terms of
wall-clock time.