# Smooth Imitation Learning via Smooth Costs and Smooth Policies

Sapana Chaudhary  Texas A&M UniversityCollege StationTexasUSA<EMAIL_ADDRESS>and  Balaraman Ravindran  Robert Bosch Centre for Data Science and AIIndian
Institute of Technology MadrasChennaiIndia<EMAIL_ADDRESS>
###### Abstract.

Imitation learning (IL) is a popular approach in the continuous control
setting as among other reasons it circumvents the problems of reward mis-
specification and exploration in reinforcement learning (RL). In IL from
demonstrations, an important challenge is to obtain agent policies that are
smooth with respect to the inputs. Learning through imitation a policy that is
smooth as a function of a large state-action ($s$-$a$) space (typical of high
dimensional continuous control environments) can be challenging. We take a
first step towards tackling this issue by using smoothness inducing
regularizers on both the policy and the cost models of adversarial imitation
learning. Our regularizers work by ensuring that the cost function changes in
a controlled manner as a function of $s$-$a$ space; and the agent policy is
well behaved with respect to the state space. We call our new smooth IL
algorithm Smooth Policy and Cost Imitation Learning (SPaCIL, pronounced
“Special”). We introduce a novel metric to quantify the smoothness of the
learned policies. We demonstrate SPaCIL’s superior performance on continuous
control tasks from MuJoCo. The algorithm not just outperforms the state-of-
the-art IL algorithm on our proposed smoothness metric, but, enjoys added
benefits of faster learning and substantially higher average return.

imitation learning, continuous control, smooth policy, regularization, deep
reinforcement learning

##  1\. Introduction

A vast majority of problems of interest, including but not limited to
autonomous control and robotics, are characterized by high dimensional
continuous (real-valued) state and action spaces (Recht, 2019). Recently, a
multitude of reinforcement learning (RL) approaches (like DDPG (Lillicrap et
al., 2015), TRPO (Schulman et al., 2015a), PPO (Schulman et al., 2017), SAC
(Haarnoja et al., 2018), etc) have been proposed to solve these challenging
continuous control tasks. However, these algorithms require specification of a
proper cost (or reward) function for any learning to be possible (Abbeel and
Ng, 2004). To circumvent this requirement, and to guide exploration in high
dimensional continuous state-action spaces, Imitation Learning (IL) using
demonstrations has been studied extensively (Schaal, 1999; Calinon, 2009;
Argall et al., 2009; Ho and Ermon, 2016; Fu et al., 2017).

In high dimensional continuous control, an important challenge is to obtain an
input-robust agent policy, i.e., a policy that varies in a controlled manner
with respect to changes in the input state-action pair (see Fig. 1). This view
of policy smoothness is equivalent to Lipschitz-continuity of the learned
agent policy. Such a smooth policy can ensure agent safety, reduce agent
energy consumption, and make agent behavior predictable in desired situations
(Kormushev et al., 2013). From a control-theoretic perspective, policy
smoothness certifies input-output stability (i.e, finite $\mathcal{L}_{2}$
gain) during both exploration and deployment (Jin and Lavaei, 2018). In
certain environments, smoothness is desired from a visual standpoint, e.g.,
smooth autonomous camera control while recording a live basketball match (Chen
et al., 2016).

Figure 1. Understanding smoothness and non-smoothness of a policy. Consider
the grey region to be a portion of the environment. We use blue circles to
denote the environment states, and arrows denote the actions. Let state
$s_{1}$ be perturbed to another state $s_{2}$ within a radius $\varrho$ of
$s_{1}$. Action $a_{2}$ comes from a smooth policy as its direction and
magnitude are similar to $a_{1}$. On the other hand, $a_{3}$ comes from a non-
smooth policy because of significant deviation from $a_{1}$ in both magnitude
and direction.

While policy smoothness has recently gained attention in the RL community
(Shen et al., 2020), it has been largely untouched in IL settings. To the best
of our knowledge, policy smoothness with respect to the inputs has not been
previously characterized and studied in IL. It is important to note here that
by virtue of learning from demonstrations (and by employing a cost (or reward)
recovery scheme), IL provides us with an additional degree of control over the
final agent policy. This additional control is not available in RL. To address
the challenge of obtaining smooth agent policies in high dimensional
continuous environments via IL using demonstrations, we propose a smooth IL
algorithm: Smooth Policy and Cost Imitation Learning (SPaCIL). SPaCIL learns
smooth agent policies by regularizing both the cost and the policy
optimization steps of adversarial imitation learning setup. Adversarial IL is
a state-of-the-art IL framework that uses Inverse Reinforcement Learning (IRL)
approach (Russell, 1998; Ng et al., 2000), and recovers both a cost function
and an agent policy. We demonstrate that our dual regularization of
parameterized function approximators (for the cost and the policy) can assure
the desired smoothness. At a higher level, the regularizers work by penalizing
drastic changes in the cost and the policy at each step of the agent learning
process. The policy regularization step is essential to control the smoothness
of the learned policy model directly. The need for cost regularization stems
from the observation that a cost function is a succinct definition of a task,
and by imposing proper structure on costs, we can not only recover better
costs but guide our policy optimization step towards desirable policies. It is
important to note that our regularizers are not tied to a specific IL
objective but are general entities that can be used with wide array of
algorithms. Additionally, as a byproduct of our goal, we show that we achieve
considerable gains in the training and performance of the agent policy.

##### Related Work.

A few recent works (Kim et al., 2004; Ko et al., 2007; Shen et al., 2020;
Miyato et al., 2018; Zhang et al., 2019; Hendrycks et al., 2019; Xie et al.,
2019; Jiang et al., 2019; Blondé et al., 2020; Le et al., 2016) address some
or the other notion of smoothness in adjacent/orthogonal problem settings.
However, none of these works deal with smoothness in IL. The work by Kim et
al. (Kim et al., 2004) on using reinforcement learning (RL) for the autonomous
helicopter flight uses quadratic penalties for actions to modify the overall
cost to encourage small actions and smooth control of the helicopter. That
work deals with a low dimensional finite action setting and explicitly weights
the cost associated with each action by a fixed factor. Such a weighing is not
possible in high dimensional $s$-$a$ spaces. For learning controllers for
autonomous blimps, Ko et al. (Ko et al., 2007) parameterize their controller
using the slope of a policy-smoothing function that determines the
controller’s smoothness near a certain switching curve. However, a definite
expression for policy-smoothing function exists, and the smoothness is desired
over a limited region of the state space. This setting is in stark contrast to
our setting, where we desire smoothness over the entire state space. Le et al.
(Le et al., 2016) study the problem of smoothly imitating an expert behaviour
(using structured prediction) by ensuring that actions for adjacent states
along a trajectory are similar, irrespective of proximity of adjacent states
in the state space. In contrast, our notion of smoothness is not conditioned
on the trajectory information and we work with the idea of smoothness of the
policy space with respect to the state space. The works by Blonde et al.
(Blondé et al., 2020) and Shen et al. (Shen et al., 2020) are closest to our
goal. Blonde et al. (Blondé et al., 2020) show that enforcing Lipschitz-
continuity of the learned reward function is essential for off-policy
imitation learning to work well. Shen et al. (Shen et al., 2020) discuss a
policy regularizer to obtain smooth policies in Reinforcement Learning (RL).
We, on the other hand, focus on obtaining smooth policies in IL. Additionally,
Shen et al.’s (Shen et al., 2020) work does not provide a proper evaluation of
smoothness. Smoothness-inducing regularizers, similar to Shen et al.’s (Shen
et al., 2020), have been previously discussed in the context of semi-
supervised learning, unsupervised domain adaptation, and harnessing
adversarial examples (Miyato et al., 2018; Zhang et al., 2019; Hendrycks et
al., 2019; Xie et al., 2019; Jiang et al., 2019). State-of-the-art IL
algorithm, GAIL (Ho and Ermon, 2016) introduces various cost function
regularizers to obtain various instances of IL algorithms; however, none of
those regularizes enforce any special structure on the costs.

In what follows, we describe our smooth IL approach in greater detail. The
main contributions of the paper are as follows:

  1. (1)

Formalization of the notion of smooth policies using Lipschitz continuity.

  2. (2)

Theoretical study of how Lipschitz continuous rewards facilitate in obtaining
Lipschitz continuous agent policy in on-policy continuous control.

  3. (3)

Introduction of smoothness inducing cost function and policy function
regularizers to realize a smooth IL algorithm.

  4. (4)

Introduction of a novel metric (that captures Lipschitz continuity of a
learned policy) to assess the smoothness of a learned policy.

  5. (5)

Empirical testing of smoothness of the learned policies, and validation of
other claims on high dimensional continuous control tasks.

##  2\. Background

##### Markov Decision Process.

We consider gamma discounted infinite horizon continuous Markov Decision
Processes (MDPs) (Sutton et al., 1998) as the core framework. An MDP is
specified by the tuple $<\mathcal{S},\mathcal{A},P,c,$ $\gamma,\rho_{0}>$,
where $\mathcal{S}\subset\mathbb{R}^{D_{s}}$ and
$\mathcal{A}\subset\mathbb{R}^{D_{a}}$ are compact sets with non-zero Lebesgue
measure. $\mathcal{S}$ is the set of possible states, $\mathcal{A}$ is the set
of possible actions and $c:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$
is the cost function. $D_{s}$ and $D_{a}$ are the dimensions of the state and
the action spaces, respectively. $P$ for any
$(s,a,s^{\prime})\in\mathcal{S}\times\mathcal{A}\times\mathcal{S}$ triplet
gives the probability of moving from state $s$ to state $s^{\prime}$ on taking
action $a$ at $s$. $\gamma$ is the discount factor and $\rho_{0}$ is the
initial state distribution.

##### Policy, occupancy measure, and expected cost.

A stationary stochastic policy, $\pi(a|s)$ gives agent’s behaviour in the
environment. Return, $G$, is a measure of goodness of a policy and is defined
as $G(\pi)=-\mathbb{E}_{\pi}[c(s,a)]=-\mathbb{E}[\sum_{t\geq
0}\gamma^{t}c(s_{t},a_{t}))]$. The return is estimated from trajectories as:
$\hat{G}(\pi)=-\mathbb{E}_{\tau}[\sum_{t=0}^{T}\gamma^{t}c(s_{t},a_{t})]$,
where $\tau\sim\pi$ is a trajectory of the form
$\\{(s_{i},a_{i})\\}_{i=1}^{T}$. Here $s_{0}\sim\rho_{0}$ is the starting
state, $a_{t}\sim\pi(\cdot|s_{t})$ and $s_{t+1}\sim P(\cdot|s_{t},a_{t})$. $T$
denotes the time step until which we sample a trajectory. There is a one-to-
one mapping between a policy, $\pi$ and its occupancy measure,
$\rho_{\pi}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ defined as
$\rho_{\pi}(s,a)=\pi(a|s)\sum_{t=0}^{\infty}\gamma^{t}P(s_{t}=s|\pi)$.
Expected cost in terms of $\rho_{\pi}$ is given by
$\mathbb{E}_{\pi}[c(s,a)]=\int_{\mathcal{S}\times\mathcal{A}}\rho_{\pi}(s,a)c(s,a)\leavevmode\nobreak\
dsda$. The stationary distribution of the policy $\pi$ is denoted by
$\rho_{\pi}(s)$. The expert policy is denoted by $\pi_{E}$. For a general
function of states, $f(s)$, $\mathbb{E}_{s\sim\rho}[f(s)]$ means a
$\gamma$-discounted expectation (as is for $G(\pi)$), unless stated otherwise.

##### Value functions and advantage.

The value function $V^{\pi}$ and action value function $Q^{\pi}$ can be
written as $V^{\pi}(s)=-\mathbb{E}_{\pi}\left[c(\cdot,\cdot)\mid
s_{0}=s\right]$ and $Q^{\pi}(s,a)=-\mathbb{E}_{\pi}\left[c(\cdot,\cdot)\mid
s_{0}=s,a_{0}=a\right]$. Advantage function
$A^{\pi}(s,a)=Q^{\pi}(s,a)-V^{\pi}(s)$ reflects the expected additional cost
that the agent bears after taking action $a$ in state $s$.

##### Parameterized representations.

We use parameterized function approximators (neural networks) for realization
of the cost and the policy. When needed, we denote the cost function as
$c_{\omega}(s,a)$ and the policy as $\pi_{\theta}(a|s)$, where
$\omega\in\Omega\subset\mathbb{R}^{D_{\omega}}$ and
$\theta\in\Theta\subset\mathbb{R}^{D_{\theta}}$. $D_{\omega}$ and $D_{\theta}$
are the dimensions of the parameters. We assume that $\Omega$ and $\Theta$ are
compact sets. $\mathcal{C}$ is the space of all cost functions. $\Pi$ is the
space of all policies. $\lVert\cdot\rVert$ denotes the $L^{2}$ norm unless
specified otherwise. $|\cdot|$ represents the usual norm.

##### Imitation Learning.

IL solution approaches can be broadly divided into: Behaviour Cloning (BC)
(Bain and Sammut, 1995; Saksena et al., 2019) and imitation learning using
inverse reinforcement learning (IRL, (Abbeel and Ng, 2004; Ziebart et al.,
2008; Finn et al., 2016; Levine et al., 2011)). For high dimensional
continuous control tasks, IL using IRL is the method of choice as BC suffers
from covariate shift errors (Ross and Bagnell, 2010; Ross et al., 2011). IL
using IRL can be cast as the following bi-level optimization problem (GAIL,
(Ho and Ermon, 2016)):

$\displaystyle\text{IL}(\pi_{E})=\operatorname*{arg\,max}_{c\in\mathcal{C}}$
$\displaystyle-\psi(c)+$ (1)
$\displaystyle\left(\min_{\pi\in\Pi}\mathbb{E}_{\pi}[c(s,a)]\right)-\mathbb{E}_{\pi_{E}}[c(s,a)],$

where $\psi:\mathcal{C}\rightarrow\overline{\mathbb{R}}$ is a closed proper
convex cost function loss specifier, i.e., it helps specify a trainable loss
for the cost function model. In the first level of optimization, we fix the
cost function and update the policy using
$\min_{\pi\in\Pi}\mathbb{E}_{\pi}[c(s,a)]$. Next, we fix the policy from the
previous update, and update the cost function using
$\operatorname*{arg\,max}_{c\in\mathcal{C}}-\psi(c)+\mathbb{E}_{\pi}[c(s,a)]-\mathbb{E}_{\pi_{E}}[c(s,a)]$.

##### Policy optimization.

The policy update step (performed using the trust region based policy
optimization algorithm (TRPO, (Schulman et al., 2015a))) is given by:

$\displaystyle\pi_{k+1}=\underset{\pi}{\arg\min}-\mathbb{E}_{s\sim\rho^{\pi_{{k}}},a\sim\pi_{{k}}}\left[\frac{\pi(a\mid
s)}{\pi_{{k}}(a\mid s)}A^{\pi_{{k}}}(s,a)\right]$ (2)
$\displaystyle\operatorname{\textit{subject to}}\leavevmode\nobreak\
\mathbb{E}_{s\sim\rho^{\pi_{{k}}}}\left[\mathcal{D}_{\mathrm{KL}}\left(\pi_{{k}}(\cdot\mid
s)\|\pi(\cdot\mid s)\right]\leq\delta.\right.$

Here, $\mathcal{D}_{\mathrm{KL}}\left(\pi_{{k}}(\cdot\mid s)\|\pi(\cdot\mid
s)\right)=\mathbb{E}_{a\sim\pi_{{k}}}\left[\frac{\pi_{{k}}(\cdot\mid
s)}{\pi(\cdot\mid s)}\right]$ is the Kullback–
Leibler (KL) divergence and the constraint (in Eqn. 2) bounds the KL-
divergence between two consecutive policies by $\delta$. Here, the expectation
in constraint is not $\gamma$-discounted.

##### Cost optimization.

Using the cost function loss specifier $\psi_{\text{GA}}(c)$ of GAIL (Ho and
Ermon, 2016) the cost parameters are updated as:

$\displaystyle c_{k+1}=$
$\displaystyle\underset{c}{\operatorname{\arg\max}}\leavevmode\nobreak\
\leavevmode\nobreak\
\mathbb{E}_{s\sim\rho^{\pi_{k}}}[\log(D(s,\pi_{k}(\cdot\mid s))]+$ (3)
$\displaystyle\mathbb{E}_{s\sim\rho^{\pi_{E}}}[\log(1-D(s,\pi_{E}(\cdot\mid
s))]$

where $\log(D(s,\pi_{k}(\cdot\mid s))=c(s,\pi_{k}(\cdot\mid s))$ is the cost
function and $D:\mathcal{S}\times\mathcal{A}\rightarrow(0,1)$ is the
classifier.

##### Gaussian policy representation.

Additionally, to include stochasticity (Papini et al., [n.d.]) in the on-
policy (Sutton et al., 1998) policy gradient approach (TRPO) we consider our
stationary stochastic policies to be Guassian distributed with
$\mu_{\theta}(s)$ as the mean and standard deviation (std) given by a fixed
quantity $\sigma$. Here, $\mu_{\theta}(s)$ is a deterministic function of the
states parameterized by $\theta$. Thus, an action $a$ sampled from our policy
can be written as,

(4) $a\sim\mathcal{N}(\mu_{\theta}(s),\sigma)\implies a=\mu_{\theta}(s)+\sigma
z,\quad z\sim\mathcal{N}(0,1),$

where $\mathcal{N}(\cdot)$ is the standard Normal distribution.

##  3\. Problem Definition

###  3.1. Defining smooth policy, smooth cost, and smooth transition model

This section formally defines a smooth policy, a smooth cost, and a smooth
transition model.

###### Definition 1 (Smooth Policy).

Let $\xi$ be a metric on the space of policies, $\Pi$. A stationary stochastic
policy $\pi(a|s):\mathcal{S}\times\mathcal{A}\rightarrow[0,1]$ is smooth with
respect to the inputs, $\mathcal{S}$, if for all $s_{1},s_{2}\in\mathcal{S}$
it is Lipschitz continuous and hence, there exist an $M_{\pi}\geq 0$ such that

(5) $\displaystyle\forall s_{1},s_{2}\in\mathcal{S},\leavevmode\nobreak\
\leavevmode\nobreak\ \xi(\pi(\cdot\mid s_{1}),\pi(\cdot\mid s_{2}))\leq
M_{\pi}\lVert s_{1}-s_{2}\rVert.$

###### Definition 2 (Smooth Cost).

A cost function $c:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is
smooth with respect to the inputs, $\mathcal{S}\times\mathcal{A}$, if for all
$(s_{1},a_{1}),(s_{2},a_{2})\in\mathcal{S}\times\mathcal{A}$ it is Lipschitz
continuous and hence, there exist an $M_{c}\geq 0$ such that

$\displaystyle\forall(s_{1},a_{1}),(s_{2},a_{2})\in\mathcal{S},$ (6)
$\displaystyle\lVert(c(s_{1},a_{1})-c(s_{2},a_{2}))\rVert\leq M_{c}(\lVert
s_{2}-s_{1}\rVert+\lVert a_{2}-a_{1}\rVert).$

###### Definition 3 (Smooth Transition Model).

If the transition model, $P$ is $L_{p}$-Lipschitz continuous it satisfies the
following constraint for every two state action pairs $(s_{1},a_{1})$ ,
$(s_{2},a_{2})\in\mathcal{S}\times\mathcal{A}$ and all $1$-Lipschitz value
functions $V$:

(7)
$\displaystyle\left|\int_{s^{\prime}\in\mathcal{S}}{(P(s^{\prime}|s_{1},a_{1})-P(s^{\prime}|s_{2},a_{2}))V(s^{\prime})ds^{\prime}}\right|$
(8) $\displaystyle\leq L_{p}(\|s_{1}-s_{2}\|+\|a_{1}-a_{2}\|).$

If $V$ is $L_{v}$ Lipschitz, the right hand side of the inequality in Eqn. 8
will be scaled by $L_{v}$.

###  3.2. IL with smooth policy

Smooth policies are crucial in high dimensional continuous control for diverse
reasons ranging from critical (robot safety) to aesthetic (visual appeal).
When a cost function can be appropriately constructed and specified by a
problem designer, reinforcement learning is the go-to solution approach.
However, cost function design is a tedious task, and cost functions tend to be
grossly mis-specified (Abbeel and Ng, 2004; Dayan and Balleine, 2002).
Imitation learning helps overcome the challenge of cost function design by
defining strategies that learn agent policies from (expert) demonstrations.
However, existing IL approaches do not guarantee that the learned policies are
smooth, i.e., there is no existing approach that solves the following
(general) problem:

$\displaystyle\pi^{\star}_{\text{smooth}}$
$\displaystyle=\arg\min_{\pi\in\Pi}\leavevmode\nobreak\ \leavevmode\nobreak\
\mathbb{E}_{\pi}\left[c(s,a)|\\{(s_{E},a_{E})\\}\sim\pi_{E}\right]$ (9)
$\displaystyle\text{subject to}\leavevmode\nobreak\ \leavevmode\nobreak\
\xi\left(\pi,\pi_{\text{perturbed}}\right)\leq\epsilon^{\prime},$

where $\mathbb{E}_{\pi}\left[c(s,a)|\\{(s_{E},a_{E})\\}\sim\pi_{E}\right]$
represents a general IL objective, $\pi_{\text{perturbed}}$ represents a
policy obtained by perturbing the original policy ($\pi$) by a small amount,
and $\epsilon^{\prime}$ captures desired policy smoothness. The policy
perturbation can be achieved in numerous ways. For our purpose of policy
smoothness, $\pi_{\text{perturbed}}$ is obtained by inducing controlled amount
of noise in the states along trajectories sampled by a policy $\pi$. Hence,
the problem is to obtain optimal smooth agent policy,
$\pi^{\star}_{\text{smooth}}$ from the class of policies, $\Pi$ using expert
demonstration data ($N$ trajectories of the form:
$\tau^{E}_{i}=\\{(s_{j},a_{j})\\}_{j=1}^{T},i=1,2,...N$). The policy
optimization is subject to the constraint:
$\xi\left(\pi,\pi_{\text{perturbed}}\right)\leq\epsilon^{\prime}$,
$\textit{i.e.},$ choose that policy which when perturbed in a controlled
manner behaves similar to the original unperturbed policy. The similarity in
behaviour is characterized using the distance metric $\xi$ defined over $\Pi$.

##  4\. Method for Obtaining smooth policies in IL

We take the approach of adversarial imitation learning (Ho and Ermon, 2016; Fu
et al., 2017) to solve the problem discussed above (Eqn. 9). Adversarial IL is
a high dimensional counterpart of IL using IRL, and is mathematically
formulated as:

(10)
$\displaystyle\text{IL}(\pi_{E})=\operatorname*{arg\,max}_{c\in\mathcal{C}^{\prime}}\left(\min_{\pi\in\Pi}\mathbb{E}_{\pi}[c(s,a)]\right)-\mathbb{E}_{\pi_{E}}[c(s,a)],$

where $\mathcal{C}^{\prime}$ is the cost function class modified by the loss
function choosen to train the cost model. In this approach, the agent policy
($\pi$) and the cost function ($c$) are simultaneously learned using bi-level
optimization of parameterized models. The cost function is learned using the
expert demonstration data and the samples from the current agent policy model.
The cost model update rule is defined in Eqn. 3. The agent policy is learned
using trust region-based policy optimization algorithm (Eqn. 2). To achieve
the goal of policy smoothness, we propose to include smoothness inducing
regularizers in both the policy and the cost optimization steps of adversarial
imitation learning:

$\displaystyle\text{IL}(\pi_{E})=\operatorname*{arg\,max}_{c\in\mathcal{C}^{\prime}}\left(\min_{\pi\in\Pi}\mathbb{E}_{\pi}[c(s,a)]+\mathcal{R}^{\pi}_{(s)}\right)$
(11) $\displaystyle-\mathbb{E}_{\pi_{E}}[c(s,a)]-\mathcal{R}^{c}_{(s,a)},$

where $\mathcal{R}^{\pi}_{(s)}$ is the smoothness inducing regularizer on
policy, and $\mathcal{R}^{c}_{(s,a)}$ is the smoothness inducing regularizer
on cost function. The exact forms of these regularizers is discussed shortly.
While policy regularization using a regularizer that captures the constraint
in Eqn. 9 seems like an obvious first approach, it is not immediately clear as
to why we need to regularize the cost function. In the following section we
describe the need for cost smoothness.

###  4.1. Smooth costs assist in learning a smooth policy

A cost function is a central tool to perform learning through interaction.
Using the following theorems, we show that a smooth cost function helps in
obtaining smooth RL policies (, and hence in IL using IRL). We show that a
Lipschitz continuous (smooth) cost function ensures resulting optimal value
functions $V^{\star}(s)$ and $Q^{\star}(s,a)$ are Lipschitz continuous
(smooth). We then show that if a Lipschitz continuous mapping is used to
obtain $\pi^{\star}$ from $Q^{\star}$, then $\pi^{\star}$ is Lipschitz
continuous. These results are of independent interest apart from providing a
clear motivation to regularize cost function learning in our method.

###### Theorem 1 (Generalization of Theorem 5.9 in (Pazis and Parr, 2011) for
the case of continuous state and action space).

For a given MDP, if the cost function, $c(s,a)$ and the transition model,
$P(s^{\prime}|s,a)$ are $L_{c}$ and $L_{p}$-Lipschitz continuous,
respectively, with respect to the state and the action pairs and $\gamma
L_{p}<1$ then
a) the optimal state value function $V^{*}(s)$ is $\frac{L_{c}}{1-\gamma
L_{p}}$ Lipschitz with respect to states, and
b) the optimal state action value function $Q^{*}(s,a)$ is
$\frac{L_{c}}{1-\gamma L_{p}}$ Lipschitz continuous with respect to states and
actions.

This theorem is a generalization of Theorem 5.9 in (Pazis and Parr, 2011) for
the case of continuous state and action space. The proof follows along the
lines of one in (Pazis and Parr, 2011) by replacing discrete Bellman
optimality operator for $\mathcal{V}$ and $\mathcal{Q}$ with continuous
counterparts. The proof is included in Appendix A.1.

###### Theorem 2.

Let $\kappa$ be a pseudo-metric on the space of conditional state-action value
functions $Q(s,\cdot)$. Let $H:Q(s,\cdot)\rightarrow\mu(s)$ be a smooth
mapping that outputs (near) greedy stationary mean policies with respect to
the input conditional state action value function $Q(s,\cdot)$ such that
$\|H(Q(s_{1}))-H(Q(s_{2}))\|\leq
L_{\mu}\kappa\left(Q(s_{1},\cdot),Q(s_{2},\cdot)\right),\forall
s_{1},s_{2}\in\mathcal{S}$.
For a given $\frac{L_{c}}{1-\gamma L_{p}}$-Lipschitz continuous state action
value function $Q^{*}(s,a)$, the policy obtained as
$\mu^{\star}(s)=H(Q^{*}(s,\cdot))$ is Lipschitz continuous with respect to
states. Then, the stationary stochastic policy $\pi^{\star}$ obtained from
$\mu^{\star}$ as $\mathcal{N}(\mu^{\star},\sigma)$ (for a fixed $\sigma$) is
Lipschitz continuous with respect to states.

The proof is included in Appendix A.2.

###  4.2. Exact forms of regularizers

With the said motivations and goal in mind, we now discuss our regularizers’
exact forms to achieve smooth policies in IL.

##### Policy Smoothing

The aim of the policy regularizer is to encourage the agent policy to be
smooth within an $\epsilon$ neighbourhood of all the states sampled according
to the current policy model. At each iteration of the policy update step, we
sample N trajectories from the current policy where each trajectory is of the
form $\tau_{i}=\\{(s_{i}^{t},a_{i}^{t})\\}_{t=1}^{T}$. We then nudge every
state, $s$, in the sampled batch to obtain, $\tilde{s}$, such that $\tilde{s}$
lies in an $\epsilon$-radius ball around $s$ (i.e.,
$\tilde{s}\in\mathbb{B}_{d}(s,\epsilon)$) and the policy, $\pi_{\theta}$
changes the most at this $s^{\prime}$. The maximum policy change is defined
using a suitable divergence
$\mathcal{D}\left(\pi_{\theta}(s),\pi_{\theta}(\widetilde{s})\right)$. In this
work, we consider $\mathcal{D}$ to be Jeffrey’s Divergence:
$\mathcal{D}_{\mathrm{J}}(P\|Q)=\frac{1}{2}\mathcal{D}_{\mathrm{KL}}(P\|Q)+\frac{1}{2}\mathcal{D}_{\mathrm{KL}}(Q\|P)$.
This policy regularizer is similar in spirit to the one considered in (Shen et
al., 2020). At a fundamental level, the regularizer measures the local
Lipschitz continuity of policy $\pi_{\theta}$ under the divergence
$\mathcal{D}$ and thus limits the policy output decision change if a small
perturbation is added to a certain state $s$.

(12)
$\displaystyle\mathcal{R}_{s}^{\pi}(\theta_{k})=\mathbb{E}_{s\sim\rho^{\pi_{\theta_{k}}}}\max_{\tilde{s}\in\mathbb{B}_{d}(s,\epsilon)}\mathcal{D}_{\mathrm{J}}\left(\pi_{\theta_{k}}(\cdot\mid
s),\pi_{\theta_{k}}(\cdot\mid\tilde{s})\right).$

Note that the term inside expectation is $\gamma$ discounted. This regularizer
is now added to the policy optimization (TRPO) objective function to obtain
smooth policy update rule:

$\displaystyle\theta_{k+1}=$
$\displaystyle\underset{\theta}{\arg\min}-\mathbb{E}_{s\sim\rho^{\pi}_{\theta_{k}},a\sim\pi_{\theta_{k}}}\left[\frac{\pi_{\theta}(a\mid
s)}{\pi_{\theta_{k}}(a\mid s)}A^{\pi_{\theta_{k}}}(s,a)\right]+$ (13)
$\displaystyle\lambda_{1}\mathbb{E}_{s\sim\rho^{\pi}_{\theta_{k}}}\max_{\widetilde{s}\in\mathbb{B}_{d}(s,\epsilon)}\mathcal{D}_{\mathrm{J}}\left(\pi_{\theta}(\cdot\mid
s)\|\pi_{\theta}(\cdot\mid\tilde{s})\right),$
$\displaystyle\operatorname{\textit{subject
to}}\mathbb{E}_{s\sim\rho^{\pi_{\theta_{k}}}}\left[\mathcal{D}_{\mathrm{KL}}\left(\pi_{\theta_{k}}(\cdot\mid
s)\|\pi_{\theta}(\cdot\mid s)\right]\leq\delta.\right.$

##### Cost Smoothing

To obtain smooth costs, we propose to regularize the cost optimization step.
The goal of this regularizer is to regulate the worst change in cost function
within an $\epsilon$-ball around states obtained from the mixture
trajectories, $\hat{\tau}\leftarrow\zeta\tau_{E}+(1-\zeta)\tau_{i}$, where
$\zeta$ is a mixing parameter. The smoothness is induced for $(s,a)$ pairs
sampled from the mixture trajectory to generalize smoothness across the entire
state-action space.

A few more words about trajectory mixing are warranted here. The mixing of
trajectories considered in our work is reminiscent of policy mixing introduced
in RL in Kakade et al. (Kakade and Langford, 2002) and in IL introduced in
Ross et al. (Ross and Bagnell, 2014). Policy mixing in these works aims to
ensure conservative policy update at each policy iteration step. A greedy
policy update based purely on approximate state-action values has shown to
affect the policy learning process adversely. A similar mixing of expert and
non-expert data is used in the regularizer sampling distribution of WGAN-GP
(Gulrajani et al., 2017). The goal of data mixing in WGAN-GP is to ensure
Lipschitzness property is satisfied by the gradients (of a model) for both the
expert and the non-expert data. Our regularizer draws inspiration from the
observations mentioned above. We mix agent and expert trajectories to ensure
costs are smoothened both as a function of agent and expert $s$-$a$ pairs and
not just imperfect agent data. Such mixing guarantees conservative enforcement
of smoothness regularizer over cost function, rather than changing cost
function drastically over iterations, making the learning algorithm unstable.
Additionally, our regularizer provides a consistent learning signal to the
cost model even when the agent and the expert policy supports do not overlap
(Arjovsky and Bottou, 2017; Gulrajani et al., 2017). The cost regularizer
takes the following exact form:

(14)
$\displaystyle\mathcal{R}_{(s,a)}^{c}(\theta)=\mathbb{E}_{s\sim\rho^{\hat{\pi}_{\theta}}}\max_{\tilde{s}\in\mathbb{B}_{d}(s,\epsilon)}\|c(s,\pi_{\theta}(.|s))-c(s,\pi_{\theta}(.|\tilde{s}))\|_{2}.$

Here, $\hat{\pi_{\theta}}$ is a mixture policy whose exact form is not
required for our purposes. We only need samples, $\tau$ from this policy
(which are obtained as $\hat{\tau}\leftarrow\zeta\tau_{E}+(1-\zeta)\tau_{i}$,
where $\zeta$ is the mixing parameter). Using the regularizer of Eqn. 14, the
cost optimization problem becomes:

$\displaystyle w_{k+1}=$
$\displaystyle\underset{w}{\operatorname{\arg\max}}\leavevmode\nobreak\
\mathbb{E}_{s\sim\rho^{\pi_{\theta_{k}}}}[\log(D_{w}(s,\pi_{\theta_{k}}(\cdot\mid
s))]+$ (15)
$\displaystyle\mathbb{E}_{s\sim\rho^{\pi_{E}}}[\log(1-D_{w}(s,\pi_{\theta_{E}}(\cdot\mid
s))]-$
$\displaystyle\lambda_{2}\mathbb{E}_{s\sim\rho^{\hat{\pi}_{\theta}}}\max_{\tilde{s}\in\mathbb{B}_{d}(s,\epsilon)}\|c(s,\pi_{\theta}(.|s))-c(s,\pi_{\theta}(.|\tilde{s}))\|_{2}.$

##  5\. Practical algorithm

##### Maximization over $\epsilon$-Ball

Both the regularized optimization problems in Eqn. 13 and Eqn. 15 require us
to solve a maximization problem over the $\epsilon$-ball around a certain
state $s$. The goal of this maximization problem is to find a state
$s^{\prime}$ within an $\epsilon$-ball of a state $s$ at which a certain
function, $f(s,s^{\prime})$ takes the maximum value. In Algorithm 1, we
discuss a general projected gradient based approach to solve this maximization
problem that is applicable to both Eqn. 13 and Eqn. 15. For the policy
smoothing regularizer of Eqn. 12 the $f$ in Algorithm 1 is given by
$f(s,s^{\prime})=D_{J}(\pi_{\theta}(s),\pi_{\theta}(s^{\prime}))$ (the
Jeffrey’s divergence between policies $\pi_{\theta}(s)$ and
$\pi_{\theta}(s^{\prime})$). For the cost smoothing regularizer of Eqn. 14,
$f(s,s^{\prime})=\|c(s,\pi_{\theta}(.|s))-c(s,\pi_{\theta}(.|s^{\prime}))\|_{2}$
(the $L_{2}$ distance between costs ). Now that we have a procedure to obtain
the regularizers, our proposed Smooth IL algorithm is provided in Algorithm 2.

Algorithm 1 Maximization of $f$ over $\epsilon$-ball of a state $s$

1:Input: $s$, $\epsilon$, $\eta_{\delta}$

2:Initialize: $\delta_{0}$

3:($N$ steps of projected gradient descent)

4:for $\ell=1,2,\ldots,N$-$1$

5: (Update $\delta$ in the direction of increase in $f$)

6:
$\delta_{\ell+1}=\delta_{\ell}+\eta_{\delta}\nabla_{\delta}f(s,s+\delta_{\ell})$

7: (Project $\delta_{\ell+1}$ onto the $\epsilon$-ball)

8:
$\delta_{\ell+1}=\Pi_{\mathbb{B}_{d}(0,\epsilon)}\left(\delta_{\ell+1}\right)$

9:end for

Algorithm 2 SPaCIL: Smooth Reward and Policy Imitation Learning

1:Input: Expert trajectories $\tau_{E}\sim\pi_{E}$, initial policy and
discriminator parameters $\theta_{0},w_{0}$

2:for $k=1,2,\ldots,K$

3: Smooth policy update:

4: Sample $N$ trajectories $\tau_{i}^{k}\sim\pi_{\theta_{k}}$,
$i=1,2,\cdots,N$

5: Estimate regularizer in Eqn. 12 using Algorithm 1

6: Update policy using regularized TRPO of Eqn. 13

7: Smooth cost update:

8: Estimate regularizer in Eqn. 14 using Algorithm 1

9: Update cost using Eqn. 15

10:end for

###  5.1. Evaluating smoothness of the learned policy

To quantify the smoothness of the learned agent policy, we introduce the
following (general) novel metric that captures local Lipschitz continuity of
the policy:

(16) $\displaystyle
J(\pi)=\mathbb{E}_{s\sim\rho^{\pi}}\left[\max_{\tilde{s}\in\mathbb{B}_{d}(s,\epsilon)}\mathcal{D}_{\mathrm{J}}\left(\pi(\cdot\mid
s),\pi(\cdot\mid\tilde{s})\right)\right],$

where the term inside expectation is not $\gamma$-discounted. We do not
include discounting here because we desire policy at any state sampled by the
policy to be smooth. For the particular case of Gaussian policies, to assess
the Lipschitz continuity of a stochastic policy $\pi$, we can look at the
Lipschitz smoothness of its deterministic mean function $\mu$ (see Eqn. 4).
For this case, practical smoothness metric takes the following form:

(17) $\displaystyle
J(\\{\tau_{i}\\}_{i=1}^{N})=\frac{1}{NT}\sum_{i=1}^{N}\sum_{j=1}^{T}\max_{\tilde{s_{j}}\in\mathbb{B}_{d}(s_{j},\epsilon)}\frac{\|\mu(s_{j})-\mu(\tilde{s_{j}})\|}{\|s-\tilde{s_{j}}\|},s_{j}\sim\tau_{i}.$

We do not (on purpose) consider a global Lipschitz constant $L$ corresponding
to the deterministic mean function, $\mu(s)$ of the following form:

(18) $\displaystyle L=\max_{s_{1}\neq
s_{2}}\frac{\|\mu(s_{1})-\mu(s_{2})\|}{\|s_{1}-s_{2}\|},\quad
s_{1},s_{2}\in\mathcal{S}.$

The choice to quantify smoothness using local Lipschitz constant stems from
the fact that finding the maximum over $\mathcal{S}$ in Eqn. 18 is impractical
for high dimensional environments.

##  6\. Experiments

Table 1.  Average Return (and one standard deviation) from 100 trajectories sampled from 5 different best agent models (i.e., five different random seeds). We observe that SPaCIL outperforms GAIL by a substantial margin, and enjoys much lesser variance in average return across different runs. | Reacher | Hopper | Walker2d | HalfCheetah | Ant
---|---|---|---|---|---
$\mathcal{S}$ / $\mathcal{A}$ |  $\mathbb{R}^{11}$ / $\mathbb{R}^{2}$ |  $\mathbb{R}^{11}$ / $\mathbb{R}^{3}$ |  $\mathbb{R}^{17}$ / $\mathbb{R}^{6}$ |  $\mathbb{R}^{17}$ / $\mathbb{R}^{6}$ |  $\mathbb{R}^{111}$ / $\mathbb{R}^{8}$
Expert (non-smooth) |  -4.18$\pm$1.79 |  3562.04$\pm$26.61 |  4224.34$\pm$18.79 |  4022.09$\pm$79.72 |  4872.84$\pm$568.69
GAIL | -5.27$\pm$2.72 | 3326.43$\pm$872.42 | 4275.00$\pm$14.69 | 4064.12$\pm$120.30 | 4587.81$\pm$117.84
SPaCIL | -4.39$\pm$1.61 | 3662.71$\pm$39.27 | 4397.58$\pm$6.92 | 4141.10$\pm$93.84 | 4788.73$\pm$75.18
Table 2.  Smoothness metric ($J$) (and one standard deviation) of the best trained models estimated over 150 randomly sampled trajectories from five different policy models. SPaCIL learns a substantially smooth policy irrespective of how smoothness encoded in expert demonstrations. | Reacher | Hopper | Walker2d | HalfCheetah | Ant
---|---|---|---|---|---
Expert (non-smooth) |  1.77e-05$\pm$3.34e-06 |  12.34$\pm$0.089 |  56.61$\pm$0.031 |  23.88$\pm$11.70 |  6.4$\pm$3.81e-3
GAIL | 5.86e-5$\pm$9.79e-5 | 12.91$\pm$0.85 | 117.31$\pm$6.65 | 24.25$\pm$0.77 | 1.543$\pm$0.62
SPaCIL | 2.69e-5$\pm$2.02e-5 | 7.77$\pm$0.067 | 35.61$\pm$0.47 | 17.73$\pm$0.12 | 0.93$\pm$0.09
Figure 2. Learning curves for GAIL and SPaCIL on challenging continuous
control tasks. The x axis is the number of algorithmic iterations. The dotted
green line represents the demonstration dataset’s average return. The dual
regularization of SPaCIL results in faster learning.

##### Overview

In this section, we aim at investigating the following research questions
(RQs):

  1. (1)

Using average return ($G$) as a metric, how well does the learned agent policy
for SPaCIL and GAIL behave in the environment? (Sec. 6.1, Table 1, Fig. 2)

  2. (2)

Does regularization of the policy and the cost space result in faster learning
for SPaCIL? (Sec. 6.1, Fig. 2)

  3. (3)

Using our smoothness metric ($J$), which algorithm gives the best resulting
smooth policies? (Sec. 6.1, Table 2)

  4. (4)

How good is our smoothness evaluation metric ($J$)? If we sufficiently perturb
our smooth agent policy model, does $J$ worsen? (Sec. 6.2, Fig. 3)

  5. (5)

How do the two regularizations affects SPaCIL’s performance and in what ways?
(Sec. 6.3, Fig. 4)

We answer these questions by performing multiple experiments on continuous
control tasks from MuJoCo (Todorov et al., 2012). We specifically work with
these environments: Reacher-v2, Hopper-v2, Walker2d-v2, HalfCheetah-v2, and
Ant-v2. We would drop v2 from these environments in all future references to
them.

##### Implementation details

For all the environments, the expert is trained using TRPO (Schulman et al.,
2015a). We then sample trajectories from the best expert model and form our
demonstration dataset. The demonstration dataset does not necessarily come
from a smooth expert policy because we do not explicitly incorporate
smoothness during TRPO training. This setting is practical as in reality we
might have non-smooth demonstrations but would still desire a smooth
controller. The imitation learning algorithms are trained on these datasets.
The algorithms do not have access to the actual environment rewards and
dynamics. To make the comparison fairer, for each environment, all the
hyperparameters of GAIL and SPaCIL are the same except for SPaCIL’s
regularization-specific hyperparameters. More implementation details can be
found in Appendix B.

###  6.1. Policy average return and smoothness

As the first set of experiments, we train GAIL and SPaCIL on demonstration
datasets from each environment. The algorithms are trained for 400 to 500
iterations depending on when the average return stabilizes. The learning
curves are included in Fig. 2. The average return from SPaCIL is better than
GAIL across all the environments. Higher returns mean that our smoothness
regularizations result in overall better-behaved agents (RQ 1). SPaCIL also
enjoys lesser variance in average return, implying the learned policies will
give the said average returns with greater confidence. SPaCIL additionally
performs better than the average return of the demonstration data for Hopper,
Walker, and HalfCheetah.

The smoothness metric $J$ is estimated from 150 trajectories (sampled using
different random seeds) from five best agent models. Here as well, SPaCIL
outperforms GAIL and recovers much smoother learned agent policies (RQ 3). The
training curves and $J$ for Walker are particularly interesting. GAIL’s
inability to tap onto the smoothness (seen from very high $J$ for this task)
results in a much higher variance of the training curve. From the nature of
the learning curve for Reacher, we see that the advantage of SPaCIL over GAIL
is more pronounced in high-dimensional tasks. From Fig. 2 it is evident that
SPaCIL results in faster stabilized returns (RQ 2).

###  6.2. Validating smoothness metric

Figure 3. Variation of scaled average policy smoothness metric (J) and scaled
average return for perturbed SPaCIL models for HalfCheetah and Hopper. Scaling
is performed w.r.t. SPaCIL J and G, respectively. The x-axis quantifies
perturbation as the standard deviation of noise added to model parameters. For
sufficiently perturbed models, $J$ is very high, validating $J$ as a
smoothness metric.

This section aims to answer if the proposed smoothness metric ($J$) is truly
meaningful. To answer this, we take the best SPaCIL policy model for
HalfCheetah and Hopper, and perturb this model by adding a zero mean, fixed
standard deviation Gaussian noise to the model parameters. Standard deviations
considered are: $0.001,0.01,0.005,0.009$ and $0.1$. We then obtain the average
return, and the average smoothness metric ($J$) from all these perturbed
models. Fig. 3 depicts scaled average returns and average $J$ for HalfCheetah
and Hopper. The scaling is done with respect to SPaCIL model’s average return
and average $J$ (in green colour), i.e., in Fig. 3 SPaCIL model’s both average
return and average $J$ are equal to one. We observe that the average return
decreases by a small amount with small perturbation with severe decrease when
perturbation standard deviation is 0.1 (high). This means that our learned
policy model is quite robust to model parameter perturbation. The smoothness
metric initially decreases showing existence of a policy that is smoother than
the baseline at the cost of decrease in return. This highlights an important
facet of our method - we want high performing policies to be smooth, and not
desire excessive smoothness at the cost of lesser return. Also, an extremely
smooth policy might mean the agent can hardly move and hence gets a low
return. When the model is sufficiently perturbed (with a perturbation standard
deviation of 0.1), the resulting $J$ is very high, showing that the policy
model is non-smooth (RQ 4).

###  6.3. Policy regularization vs. cost regularization: Which is more
important?

We run Hopper with varying amounts of cost and policy regularizations. Fig. 4
depicts the results from this experiment. We get the best smooth policy for
$\lambda_{1}=0.001$ and $\lambda_{2}=1$ with $G=3642.76$ and $J=9.556$. The
agent obtains the maximum return of $3715.08$ at $\lambda_{1}=0$ and
$\lambda_{2}=1$, i.e., merely smoothing the costs results in policies that
enjoy higher return. However, this policy is highly non-smooth with a
$J=68.67$. Thus, the agent has to pay some additional cost to obtain visual
smoothness. We can see from Fig. 4 that a delicate balance of both policy and
cost regularization is needed to obtain high performing smooth policies.

Figure 4. Average return ($G$) and average smoothness metric ($J$) variation
with varying policy and cost regularization parameters ($\lambda_{1}$ and
$\lambda_{2}$) for Hopper. Both policy and cost regularizations are needed to
obtain high performing smooth policies.

##  7\. Conclusion

In this work, we develop a novel model-free on-policy IL algorithm: Smooth
Policy and Cost Imitation Learning (SPaCIL) that learns a smooth agent policy.
Smoothness in policies is achieved by regularizing both the cost and the
policy models of adversarial imitation learning framework. Through smoothness-
inducing regularization, our algorithm can encode the domain knowledge about
the smoothness of costs and policies. Our algorithm not only obtains smooth IL
policies (measured by the policy smoothness metric that we introduce), it
results in policies with a higher return than state-of-the-art adversarial IL
algorithm GAIL. Our algorithm enjoys added benefits of faster learning and a
much lower variance in average returns and smoothness metric.

## References

  * (1)
  * Abbeel and Ng (2004) Pieter Abbeel and Andrew Y Ng. 2004.  Apprenticeship learning via inverse reinforcement learning. In _Proceedings of the twenty-first international conference on Machine learning_. ACM, 1.
  * Argall et al. (2009) Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. 2009.  A survey of robot learning from demonstration.  _Robotics and autonomous systems_ 57, 5 (2009), 469–483.
  * Arjovsky and Bottou (2017) Martin Arjovsky and Léon Bottou. 2017.  Towards principled methods for training generative adversarial networks. In _NIPS 2016 Workshop on Adversarial Training. In review for ICLR_ , Vol. 2016.
  * Bain and Sammut (1995) Michael Bain and Claude Sammut. 1995.  A Framework for Behavioural Cloning.. In _Machine Intelligence 15_. 103–129.
  * Blondé et al. (2020) Lionel Blondé, Pablo Strasser, and Alexandros Kalousis. 2020\.  Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning.  _arXiv preprint arXiv:2006.16785_ (2020).
  * Brockman et al. (2016) Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016.  Openai gym.  _arXiv preprint arXiv:1606.01540_ (2016).
  * Calinon (2009) Sylvain Calinon. 2009\.  _Robot programming by demonstration_.  EPFL Press.
  * Chen et al. (2016) Jianhui Chen, Hoang M Le, Peter Carr, Yisong Yue, and James J Little. 2016.  Learning online smooth predictors for realtime camera planning using recurrent decision trees. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 4688–4696.
  * Dayan and Balleine (2002) Peter Dayan and Bernard W Balleine. 2002.  Reward, motivation, and reinforcement learning.  _Neuron_ 36, 2 (2002), 285–298.
  * Eriksson et al. (2003) Kenneth Eriksson, Donald Estep, and Claes Johnson. 2003\.  _Applied mathematics: Body and soul: Calculus in several dimensions_. Vol. 3.  Springer Science & Business Media.
  * Finn et al. (2016) Chelsea Finn, Sergey Levine, and Pieter Abbeel. 2016\.  Guided cost learning: Deep inverse optimal control via policy optimization. In _Proceedings of the 33rd International Conference on Machine Learning_ , Vol. 48.
  * Fu et al. (2017) Justin Fu, Katie Luo, and Sergey Levine. 2017.  Learning Robust Rewards with Adversarial Inverse Reinforcement Learning.  _arXiv preprint arXiv:1710.11248_ (2017).
  * Gulrajani et al. (2017) Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. 2017\.  Improved Training of Wasserstein GANs.  _arXiv preprint arXiv:1704.00028_ (2017).
  * Haarnoja et al. (2018) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In _International conference on machine learning_. PMLR, 1861–1870.
  * Hecht-Nielsen (1992) Robert Hecht-Nielsen. 1992\.  Theory of the backpropagation neural network.  In _Neural networks for perception_. Elsevier, 65–93.
  * Hendrycks et al. (2019) Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. 2019.  Using self-supervised learning can improve model robustness and uncertainty. In _Advances in Neural Information Processing Systems_. 15663–15674.
  * Ho and Ermon (2016) Jonathan Ho and Stefano Ermon. 2016.  Generative adversarial imitation learning. In _Advances in Neural Information Processing Systems_. 4565–4573.
  * (https://math.stackexchange.com/users/8297/martin sleziak) ([n.d.]) Martin Sleziak (https://math.stackexchange.com/users/8297/martin sleziak). [n.d.].  Is the maximum function Lipschitz continuous?  Mathematics Stack Exchange.  arXiv:https://math.stackexchange.com/q/1742410 https://math.stackexchange.com/q/1742410 URL:https://math.stackexchange.com/q/1742410 (version: 2016-04-14).
  * Jiang et al. (2019) Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2019\.  Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization.  _arXiv preprint arXiv:1911.03437_ (2019).
  * Jin and Lavaei (2018) Ming Jin and Javad Lavaei. 2018.  Control-theoretic analysis of smoothness for stability-certified reinforcement learning. In _2018 IEEE Conference on Decision and Control (CDC)_. IEEE, 6840–6847.
  * Kakade and Langford (2002) Sham Kakade and John Langford. 2002.  Approximately optimal approximate reinforcement learning. In _In Proc. 19th International Conference on Machine Learning_. Citeseer.
  * Kim et al. (2004) H Jin Kim, Michael I Jordan, Shankar Sastry, and Andrew Y Ng. 2004.  Autonomous helicopter flight via reinforcement learning. In _Advances in neural information processing systems_. 799–806.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Ko et al. (2007) Jonathan Ko, Daniel J Klein, Dieter Fox, and Dirk Haehnel. 2007\.  Gaussian processes and reinforcement learning for identification and control of an autonomous blimp. In _Proceedings 2007 ieee international conference on robotics and automation_. IEEE, 742–747.
  * Kormushev et al. (2013) Petar Kormushev, Sylvain Calinon, and Darwin G Caldwell. 2013\.  Reinforcement learning in robotics: Applications and real-world challenges.  _Robotics_ 2, 3 (2013), 122–148.
  * Le et al. (2016) Hoang Le, Andrew Kang, Yisong Yue, and Peter Carr. 2016\.  Smooth imitation learning for online sequence prediction. In _International Conference on Machine Learning_. PMLR, 680–688.
  * Levine et al. (2011) Sergey Levine, Zoran Popovic, and Vladlen Koltun. 2011\.  Nonlinear inverse reinforcement learning with gaussian processes. In _Advances in neural information processing systems_. 19–27.
  * Lillicrap et al. (2015) Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015.  Continuous control with deep reinforcement learning.  _arXiv preprint arXiv:1509.02971_ (2015).
  * Miyato et al. (2018) Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. 2018.  Virtual adversarial training: a regularization method for supervised and semi-supervised learning.  _IEEE transactions on pattern analysis and machine intelligence_ 41, 8 (2018), 1979–1993.
  * Ng et al. (2000) Andrew Y Ng, Stuart J Russell, et al. 2000\.  Algorithms for inverse reinforcement learning.. In _Icml_. 663–670.
  * Papini et al. ([n.d.]) Matteo Papini, Andrea Battistello, and Marcello Restelli. [n.d.].  Safe Exploration in Gaussian Policy Gradient.  ([n. d.]).
  * Pazis and Parr (2011) Jason Pazis and Ronald Parr. 2011.  Non-Parametric Approximate Linear Programming for MDPs.. In _AAAI_.
  * Puterman (2014) Martin L Puterman. 2014\.  Markov decision processes: discrete stochastic dynamic programming.
  * Recht (2019) Benjamin Recht. 2019\.  A tour of reinforcement learning: The view from continuous control.  _Annual Review of Control, Robotics, and Autonomous Systems_ 2 (2019), 253–279.
  * Ross and Bagnell (2010) Stéphane Ross and Drew Bagnell. 2010.  Efficient reductions for imitation learning. In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_. JMLR Workshop and Conference Proceedings, 661–668.
  * Ross and Bagnell (2014) Stephane Ross and J Andrew Bagnell. 2014.  Reinforcement and imitation learning via interactive no-regret learning.  _arXiv preprint arXiv:1406.5979_ (2014).
  * Ross et al. (2011) Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. 2011\.  A reduction of imitation learning and structured prediction to no-regret online learning. In _Proceedings of the fourteenth international conference on artificial intelligence and statistics_. 627–635.
  * Russell (1998) Stuart Russell. 1998\.  Learning agents for uncertain environments. In _Proceedings of the eleventh annual conference on Computational learning theory_. ACM, 101–103.
  * Saksena et al. (2019) Saumya Kumaar Saksena, B Navaneethkrishnan, Sinchana Hegde, Pragadeesh Raja, and Ravi M Vishwanath. 2019.  Towards Behavioural Cloning for Autonomous Driving. In _2019 Third IEEE International Conference on Robotic Computing (IRC)_. IEEE, 560–567.
  * Schaal (1999) Stefan Schaal. 1999\.  Is imitation learning the route to humanoid robots?  _Trends in cognitive sciences_ 3, 6 (1999), 233–242.
  * Schulman et al. (2015a) John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. 2015a.  Trust region policy optimization. In _Proceedings of the 32nd International Conference on Machine Learning (ICML-15)_. 1889–1897.
  * Schulman et al. (2015b) John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2015b.  High-dimensional continuous control using generalized advantage estimation.  _arXiv preprint arXiv:1506.02438_ (2015).
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ (2017).
  * Shen et al. (2020) Qianli Shen, Yan Li, Haoming Jiang, Zhaoran Wang, and Tuo Zhao. 2020.  Deep Reinforcement Learning with Smooth Policy.  _arXiv preprint arXiv:2003.09534_ (2020).
  * Sutton et al. (1998) Richard S Sutton, Andrew G Barto, et al. 1998\.  _Introduction to reinforcement learning_. Vol. 135.  MIT press Cambridge.
  * Teel (1995) Andrew R Teel. 1995\.  Examples of stabilization using saturation: an input-output approach.  _IFAC Proceedings Volumes_ 28, 14 (1995), 209–214.
  * Todorov et al. (2012) Emanuel Todorov, Tom Erez, and Yuval Tassa. 2012.  MuJoCo: A physics engine for model-based control. In _Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on_. IEEE, 5026–5033.
  * Xie et al. (2019) Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. 2019.  Unsupervised data augmentation for consistency training.  _arXiv preprint arXiv:1904.12848_ (2019).
  * Zhang et al. (2019) Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing, Laurent El Ghaoui, and Michael I Jordan. 2019.  Theoretically principled trade-off between robustness and accuracy.  _arXiv preprint arXiv:1901.08573_ (2019).
  * Ziebart et al. (2008) Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. 2008.  Maximum Entropy Inverse Reinforcement Learning.. In _AAAI_ , Vol. 8. Chicago, IL, USA, 1433–1438.

##  Appendix A Proofs

###  A.1. Proof of Theorem 1

###### Proof.

This theorem is a generalization of Theorem 5.9 in (Pazis and Parr, 2011) for
the case of continuous state and action space. The proof follows along the
lines of one in (Pazis and Parr, 2011) by replacing discrete Bellman
optimality operator for $\mathcal{V}$ and $\mathcal{Q}$ with continuous
counterparts. Let $\mathcal{T}\colon\mathcal{V}\rightarrow\mathcal{V}$ be the
Bellman optimality operator, where $\mathcal{V}$ is the space of value
functions. Then,

(19)
$\displaystyle(\mathcal{T}V)(s)=\min_{a}\left(c(s,a)+\gamma\int_{s^{\prime}\in\mathcal{S}}{p(s^{\prime}|s,a)V(s^{\prime})ds^{\prime}}\right)$

If $V(s)$ is $L_{v}$ Lipschitz in $s$, then, so is $(\mathcal{T}V)(s)$. This
can be seen as follows:
$\forall s_{1},s_{2}\in\mathcal{S}$ we have,

$\displaystyle\left|(\mathcal{T}V)(s_{1})-(\mathcal{T}V)(s_{2})\right|=$
$\displaystyle|\min_{a}\left(c(s_{1},a)+\gamma\int_{s^{\prime}\in\mathcal{S}}{p(s^{\prime}|s_{1},a)V(s^{\prime})ds^{\prime}}\right)$
(20)
$\displaystyle-\min_{a}\left(c(s_{2},a)+\gamma\int_{s^{\prime}\in\mathcal{S}}{p(s^{\prime}|s_{2},a)V(s^{\prime})ds^{\prime}}\right)|.$

Define $\hat{a}$ as,

(21)
$\displaystyle\hat{a}\triangleq\begin{dcases*}\operatorname*{arg\,min}_{a\in\mathcal{A}}{(c(s_{1},a)+\gamma\int_{s^{\prime}\in\mathcal{S}}{p(s^{\prime}|s_{1},a)V(s^{\prime})ds^{\prime}})},\\\
\quad\quad\quad\text{if}\leavevmode\nobreak\ V(s_{1})\leq V(s_{2})\\\
\operatorname*{arg\,min}_{a\in\mathcal{A}}{(c(s_{2},a)+\gamma\int_{s^{\prime}\in\mathcal{S}}{p(s^{\prime}|s_{2},a)V(s^{\prime})ds^{\prime}})},\\\
\quad\quad\quad\text{if}\leavevmode\nobreak\ V(s_{1})>V(s_{2}).\\\
\end{dcases*}$

Then,

$\displaystyle\left|(\mathcal{T}V)(s_{1})-(\mathcal{T}V)(s_{2})\right|\leq$
$\displaystyle|c(s_{1},\hat{a})+\gamma\int_{s^{\prime}\in\mathcal{S}}{p(s^{\prime}|s_{1},\hat{a})V(s^{\prime})ds^{\prime}}$
(22)
$\displaystyle-c(s_{2},\hat{a})-\gamma\int_{s^{\prime}\in\mathcal{S}}{p(s^{\prime}|s_{2},\hat{a})V(s^{\prime})ds^{\prime}}|,$

where we have used the fact that $\forall
f_{1},f_{2}\colon\mathcal{S}\rightarrow\mathbb{R}$ bounded functions such that
$\min_{s\in\mathcal{S}}{f_{1}(s)}\leq\min_{s\in\mathcal{S}}f_{2}(s)$ and
$\hat{s}=\operatorname*{arg\,min}_{s\in\mathcal{S}}f_{1}(s)$, we have
$|\min_{s\in\mathcal{S}}f_{1}(s)-\min_{s\in\mathcal{S}}f_{2}(s)|\leq|f_{1}(\hat{s})-f_{2}(\hat{s})|$.
Then using triangle inequality, we have $\forall s_{1},s_{2}\in\mathcal{S}$,

$\displaystyle\left|(\mathcal{T}V)(s_{1})-(\mathcal{T}V)(s_{2})\right|\leq\|c(s_{1},\hat{a})-c(s_{2},\hat{a})\|+$
(23)
$\displaystyle\gamma\left|\int_{s^{\prime}\in\mathcal{S}}{(p(s^{\prime}|s_{1},\hat{a})-p(s^{\prime}|s_{2},\hat{a}))V(s^{\prime})ds^{\prime}}\right|.$

$(1/L_{v})V$ is 1-Lipschitz, hence using the Definition 3 and Lipschitzness of
cost we get,

(24)
$\displaystyle|(\mathcal{T}V)(s_{1})-(\mathcal{T}V)(s_{2})|\leq(L_{c}+L_{v}L_{p})\|s_{1}-s_{2}\|.$

Hence, $(\mathcal{T}V)(s)$ is $L_{c}+\gamma L_{v}L_{p}$-Lipschitz in $s$ and
$a$, if $V$ is $L_{v}$-Lipschitz in $s$ and $a$. Choose $V_{0}=0$. $V^{*}$ is
the stationary point of $\mathcal{T}$ (Puterman, 2014):
$V^{*}=\lim_{n\to\infty}\mathcal{T}^{n}V_{0}$. Hence, $V^{*}$ is $L_{c}+\gamma
L_{p}L_{c}+\gamma^{2}L_{p}^{2}L_{c}+\dots=L_{c}/(1-\gamma L_{p})$-Lipschitz
continous.

Proof. b)we know that

(25) $\displaystyle Q^{*}(s,a)$
$\displaystyle=c(s,a)+\gamma\int_{s^{\prime}\in\mathcal{S}}{p(s^{\prime}|s,a)V^{*}(s^{\prime})ds^{\prime}}$

For this optimal state action value function $Q^{*}$ and
$s_{1},s_{2}\in\mathcal{S}$, we then have

$\displaystyle|Q^{*}(s_{1},a_{1})-Q^{*}(s_{2},a_{2})|\leq|c(s_{1},a_{1})-c(s_{2},a_{2})|+$
(26)
$\displaystyle\gamma\left|\int_{s^{\prime}\in\mathcal{S}}{(p(s^{\prime}|s_{1},a_{1})-p(s^{\prime}|s_{2},a_{2}))V(s^{\prime})ds^{\prime}}\right|$
(27) $\displaystyle\implies|Q^{*}(s_{1},a_{1})-Q^{*}(s_{2},a_{2})|$
$\displaystyle\leq(L_{c}+\gamma L_{v*}L_{p})(\|s_{1}-s_{2}\|+\|a_{1}-a_{2}\|)$

Using the value of $L_{v*}$ from the part a), we get

(28)
$\displaystyle|Q^{*}(s_{1},a_{1})-Q^{*}(s_{2},a_{2})|\leq(\frac{L_{c}}{1-\gamma
L_{p}})(\|s_{1}-s_{2}\|+\|a_{1}-a_{2}\|)$

Hence, Proved. ∎

###  A.2. Proof of Theorem 2

###### Proof.

From Lipschitz continuity of optimal $Q^{\star}$ function, we have $\forall
s_{1},s_{2}\in\mathcal{S}$,

(29)
$\displaystyle\left|Q^{*}(s_{1},a_{1})-Q^{*}(s_{2},a_{2})\right|\leq\left(\frac{L_{c}}{1-\gamma
L_{p}}\right)(\|s_{1}-s_{2}\|+\|a_{1}-a_{2}\|).$

This can be re-written as

(30)
$\displaystyle\left|\max_{a_{1}}Q^{*}(s_{1},a_{1})-\max_{a_{2}}Q^{*}(s_{2},a_{2})\right|$
(31) $\displaystyle\leq\left(\frac{L_{c}}{1-\gamma
L_{p}}\right)(\|s_{1}-s_{2}\|+\|a_{1}-a_{2}\|).$

Now, it is easy to see that
$|\max_{a_{1}}Q^{*}(s_{1},a_{1})-\max_{a_{2}}Q^{*}(s_{2},a_{2})|\leq\max_{a}|Q^{*}(s_{1},a)-Q^{*}(s_{1},a)|$
((https://math.stackexchange.com/users/8297/martin sleziak), [n.d.]). From
equivalence of norms, there exists a $\frac{1}{K_{1}}\geq 0$ such that

$\max_{a}|Q^{*}(s_{1},a)-Q^{*}(s_{1},a)|\leq
K_{1}|\max_{a_{1}}Q^{*}(s_{1},a_{1})-\max_{a_{2}}Q^{*}(s_{2},a_{2})|.$

Note that $\max_{a}|Q^{*}(s_{1},a)-Q^{*}(s_{1},a)|$ and $\kappa$ are both
pseudo-metrics on the space of $Q(s,\cdot)$ functions. Thus, from topological
equivalence of pseudo-metrics, there exists a $K_{2}\geq 0$ such that
$\kappa\left(Q(s_{1},\cdot),Q(s_{2},\cdot)\right)$ $\leq
K_{2}\max_{a}|Q(s_{1},a)-Q(s_{2},a)|$. Thus, we have
$\kappa\left(Q(s_{1},\cdot),Q(s_{2},\cdot)\right)\leq
K_{1}K_{2}|\max_{a_{1}}Q^{*}(s_{1},a_{1})-\max_{a_{2}}Q^{*}(s_{2},a_{2})|$.
Thus, for all $s_{1},s_{2}\in\mathcal{S}$ and for all $a\in\mathcal{A}$ we
have,

(32) $\displaystyle\kappa\left(Q(s_{1},\cdot),Q(s_{2},\cdot)\right)\leq
K_{1}K_{2}\left(\frac{L_{c}}{1-\gamma L_{p}}\right)(\|s_{1}-s_{2}\|).$

Now, using definition and Lipschitz continuity of $H$, we get

(33) $\displaystyle\|\mu(s_{1})-\mu(s_{2})\|$
$\displaystyle=\|H(Q^{\star}(s_{1},\cdot))-H(Q^{\star}(s_{2},\cdot))\|$ (34)
$\displaystyle\leq
L_{\mu}\kappa\left(Q^{\star}(s_{1},\cdot),Q^{\star}(s_{2},\cdot)\right)$ (35)
$\displaystyle\leq K_{1}K_{2}L_{\mu}\frac{L_{c}}{1-\gamma
L_{p}}\|s_{1}-s_{2}\|.$

Thus, the optimal mean policy $\mu^{\star}$ is
$\frac{K_{1}K_{2}L_{\mu}L_{c}}{1-\gamma L_{p}}$-Lipschitz continuous with
respect to the states. Then, the stationary stochastic policy $\pi^{\star}$
obtained as $\mathcal{N}(\mu^{\star}(s),\sigma)$ (for a fixed $\sigma$) is
$\frac{K_{1}K_{2}L_{\mu}L_{c}}{1-\gamma L_{p}}$-Lipschitz continuous with
respect to the states. ∎

##  Appendix B Experimental Setup and Hyperparameter Details

###  B.1. Policy, Cost and Value Models

##### Policy

In our work, the policy, $\pi_{\theta}(a\mid s),$ is parameterized with a
neural network. The neural network takes in a state vector $s$ and
deterministically maps $s$ to a vector $\mu$. The neural network is
additionally used to learn a vector $r$ of log standard deviation with the
same dimension as $a.$ During training, the action is sampled stochastically
from $\mathcal{N}(\mu,exp(r))$. While evaluating a policy, the mean action
$\mu$ is chosen.

##### Value

The value neural network simply takes in a state $s$ and outputs scalar
$V(s)$.

##### Discriminator

The discriminator network takes in a state-action pair ($s,a$) and outputs a
scalar $x$. The cost of this state-action pair is then defined as
$c(s,a)=log(S(x)),$ where $S(x)=\frac{e^{x}}{e^{x}+1}$.

####  B.1.1. Neural network details

All the models (policy, value, and cost functions) in this work are multi-
layer perceptrons (MLPs). The policy models for all the algorithms (TRPO,
GAIL, and SPaCIL) have two layers of 400 and 300 neurons each. The value
functions across all the algorithms have two layers of 100 neurons each.
Similarly, the discriminator network (a representative for cost model) has two
layers of 100 neurons each. The non-linearity used in all the models is tanh.
All the weights and biases are initialized using $U(-k,k)$ where
$k=\frac{1}{\sqrt{\text{size of weights}}}$. The final layer across models has
zero bias and weights multiplied by $0.1$. The step size and learning rate for
the policy model is determined by the TRPO algorithm. All the other models are
trained using Adam optimizer (Kingma and Ba, 2014) with a learning rate of
$1e$-$3$ for the value network, and a learning rate of $0.01$ for the
discriminator network.

###  B.2. Expert learning using TRPO

The MuJoCo environments used in our experiments are from the OpenAI Gym
(Brockman et al., 2016). We specifically work with v2 of Reacher, Hopper,
Walker2d, HalfCheetah, and Ant. The expert is trained using TRPO (Schulman et
al., 2015a). The TRPO hyperparameters max KL and damping are both fixed to
$0.01$ for all the environments. The batch size is $50000$ for all the
environments except Reacher’s $5000$. We train all the environments for $500$
iterations except Reacher’s $200$. We fix $\gamma=0.995$ for all the
environments except Reacher’s $0.99$. We fix $\tau=0.95$ for Reacher,
$\tau=0.99$ for Hopper and Walker, and $\tau=0.97$ for the rest.

###  B.3. GAIL and SPaCIL parameters

The best set of hyperparameters for SPaCIL are listed in Table 3. The policy
regularization strength is denoted by $\lambda_{1}$. The cost regularization
strength is denoted by $\lambda_{2}$. The step size parameter in the projected
gradient descent part of regularizer estimation is denoted as $\nu$ and takes
a value of $0.02$ across environments. The perturbation strength around a
particular state (to project onto $\epsilon$-Ball) is denoted by $\epsilon$.
It takes a value of $0.01$ across environments. For regularization purposes,
we keep the policy $\sigma$ fixed to 1. The mixing parameter $\mu$ is randomly
sampled from a uniform distribution over $[0,1)$. sampled Generalized
advantage estimation (GAE, (Schulman et al., 2015b)) parameters, $\gamma$ and
$\tau$ are same as TRPO for all the environments. GAIL has all the
hyperparameters, except training data size, same as that of TRPO. We are
provided with a fixed number of expert trajectories before training begins.
This number is the same for both GAIL and SPaCIL, and is listed in Table 3.
Each trajectory consists of $50$ ($s,a$) pairs for Reacher-v2, and $1000$
($s,a$) pairs for all the other tasks.

Table 3. Environments details and performance of expert policies.

Environment | $\lambda_{1}$ | $\lambda_{2}$ | Expert traj No. | Agent traj No.
---|---|---|---|---
Reacher-v2 | $0.01$ | $0.001$ | $50$ | $50$
Hopper-v2 | $0.001$ | $0.001$ | $6$ | $6$
Walker2d-v2 | $0.001$ | $0.001$ | $10$ | $10$
HalfCheetah-v2 | $0.01$ | $0.001$ | $6$ | $6$
Ant-v2 | $0.001$ | $0.001$ | $15$ | $15$

###  B.4. The MuJoCo Tasks

##### Reacher-v2.

In this environment, the agent is a grasping arm with a hinge, a body with a
joint, and a tip that grasps (see Fig. 5). With the hinge fixed at the center
of a square grid environment, the agent’s tip is placed at a random starting
state. The agent’s goal is to be able to grasp a target that is randomly
spawned in the square grid.

##### Hopper-v2.

In this environment, the agent is a robot with a torso and a leg (see Fig. 5).
The agent is tasked with learning to hop through the environment. The learned
walking behaviour is desired to have a stable gait.

##### HalfCheetah-v2.

In this environment, the agent is a robot with only one forelimb and one hind-
limb (see Fig. 5). The agent is tasked with learning to walk and hop through
the environment. The learned walking behaviour is desired to have a stable
gait.

##### Walker2d-v2.

In this environment, the agent is a robot with a torso and two legs (see Fig.
5). The agent is tasked with learning to jump and walk through the
environment. The learned walking behaviour is desired to have a stable gait.

##### Ant-v2.

In this environment, the agent is a robot with four legs (see Fig. 5). The
agent is tasked with learning to hop through the environment. The learned
walking behaviour is desired to have a stable gait.

Figure 5. Robotic continuous control tasks from MuJoCo.

###  B.5. Deep RL tricks used in the implementation

We use the following standard tricks from deep reinforcement learning (DRL) in
our implementation:

  1. (1)

We keep and use a running average of the states to deal with covariate shift
in the input data.

  2. (2)

We evaluate the performance of our algorithms on a separate test environment.
We keep track of the best evaluation return to save our best agent model. We
sample $20,000$ $s$-$a$ pairs at each test iteration for testing. The action
is sampled as the mean action ($\mu(s)$) at a certain state ($s$) rather than
from the stochastic policy. The evaluation performance curves for GAIL and
SPaCIL are included in Fig. 6).

  3. (3)

Different components of our code are run on different devices (i.e., CPU and
GPU) to optimize for algorithm’s run time. While sampling trajectories from
the agent policy, we use multi-threading and run the code on the CPU. All the
gradient calculations vis back-propagation are performed on GPU cores.

Figure 6. Evaluation curves for GAIL and SPaCIL on challenging continuous control tasks. The dotted green line represents the average return of the demonstration dataset. The dual regularization of policies and costs results in faster learning for SPaCIL. Table 4.  Validating our smoothness metric is a good measure of smoothness | 0 | 0.1 | 0.09 | 0.05 | 0.01 | 0.001
---|---|---|---|---|---|---
Reacher G |  -4.18$\pm$1.79 |  -163.44$\pm$43.95 |  -47.07$\pm$11.08 |  -30.12$\pm$10.25 |  -11.67$\pm$2.18 |  -12.35$\pm$2.14
Reacher J |  2.69e-5$\pm$2.02e-5 |  14.42$\pm$2.06 |  1.02$\pm$0.12 |  0.29$\pm$0.05 |  8.67e-4$\pm$1.79e-4 |  3.72e-4$\pm$7.85e-5
Hopper G |  3662.71$\pm$39.27 |  79.73$\pm$73.48 |  4.69$\pm$0.033 |  383.59$\pm$313.75 |  2644.44$\pm$1207.97 |  3667.35$\pm$9.72
Hopper J |  7.77$\pm$0.067 |  18.24$\pm$3.01 |  18.34$\pm$0.0162 |  5.43$\pm$0.13 |  6.732$\pm$0.27 |  7.86$\pm$0.19
Walker2d G |  4397.58$\pm$6.92 |  3.21$\pm$2.81 |  61.92$\pm$53.04 |  245.21$\pm$119.8 |  4199.56$\pm$17.19 |  4329.44$\pm$10.69
Walker2d J |  35.61$\pm$0.067 |  85.91$\pm$1.06 |  125.97$\pm$6.06 |  37.56$\pm$1.60 |  39.37$\pm$1.1 |  38.53$\pm$0.12
HalfCheetah G |  4141.10$\pm$93.84 |  -743.56$\pm$60.59 |  -729.94$\pm$242.07 |  128.97$\pm$141.52 |  3642.23$\pm$75.71 |  4061.67$\pm$75.59
HalfCheetah J |  17.73$\pm$0.11 |  61.63$\pm$6.72 |  88.67$\pm$33.76 |  4.47$\pm$2.03 |  17.72$\pm$0.19 |  18.51$\pm$0.21
Ant G |  4788.73$\pm$75.18 |  -4880.08$\pm$1719.81 |  -1454.50$\pm$961.96 |  1685.58$\pm$482.09 |  4561.18$\pm$617.04 |  4628.54$\pm$893.13
Ant J |  0.93$\pm$0.09 |  186.91$\pm$74.77 |  43.58$\pm$27.72 |  1.64$\pm$2.27 |  0.91$\pm$0.07 |  0.97$\pm$0.04

##  Appendix C Implementation of Projection

At the crux of our algorithm are the following policy and cost regularizers:

(36)
$\displaystyle\mathcal{R}_{s}^{\pi}(\theta_{k})=\mathbb{E}_{s\sim\rho^{\pi_{\theta_{k}}}}\max_{\tilde{s}\in\mathbb{B}_{d}(s,\epsilon)}\mathcal{D}_{\mathrm{J}}\left(\pi_{\theta_{k}}(s),\pi_{\theta_{k}}(\widetilde{s})\right),$

and

(37)
$\displaystyle\mathcal{R}_{(s,a)}^{c}(\theta)=\mathbb{E}_{s\sim\rho^{\hat{\pi}_{\theta}}}\max_{\tilde{s}\in\mathbb{B}_{d}(s,\epsilon)}\|c(s,\pi_{\theta}(.|s))-c(s,\pi_{\theta}(.|\tilde{s}))\|_{2}.$

Both the regularizers in Eqn. 36 and Eqn. 37 require us to solve a
maximization problem over the $\epsilon$-ball around a certain state $s$. The
goal of this maximization problem is to find a state $s^{\prime}$ within an
$\epsilon$-ball of a state $s$ at which a certain function, $f(s,s^{\prime})$
takes the maximum value. In Algorithm 3, we discuss a general projected
gradient based approach to solve this maximization problem that is applicable
to both Eqn. 36 and Eqn. 37. For the policy smoothing regularizer of Eqn. 36
the $f$ in Algorithm 3 is given by
$f(s,s^{\prime})=D_{\mathrm{J}}(\pi_{\theta}(s),\pi_{\theta}(s^{\prime}))$
(the Jeffrey’s divergence between policies $\pi_{\theta}(s)$ and
$\pi_{\theta}(s^{\prime})$). For the cost smoothing regularizer of Eqn. 14,
$f(s,s^{\prime})=\|c(s,\pi_{\theta}(.|s))-c(s,\pi_{\theta}(.|s^{\prime}))\|_{2}$
(the $L_{2}$ distance between costs ). Here we provide exact details of how to
obtain $\nabla_{\delta}f(s,s+\delta_{\ell})$ (in step 6) and how to project
onto the ball $\Pi_{\mathbb{B}_{d}}$ (step 8).

For $f(s,s^{\prime})=D_{J}(\pi_{\theta}(s),\pi_{\theta}(s^{\prime}))$ and
Guassian distributed policies (i.e., $\pi_{\theta}(\cdot\mid
s)\stackrel{{\scriptstyle d}}{{=}}\mathcal{N}(\mu_{\theta}(s),\sigma^{2})$),
$f$ is reduced to
$\frac{\|\mu_{\theta}(s)-\mu_{\theta}(s^{\prime})\|_{2}^{2}}{\sigma^{2}}$.
Gradient of this quantity is estimated using automatic back-propagation
(Hecht-Nielsen, 1992) through the policy neural network. Gradient of
$f(s,s^{\prime})=\|c(s,\pi_{\theta}(.|s))-c(s,\pi_{\theta}(.|s^{\prime}))\|_{2}$
can be equivalently estimated using back-propagation.

Once we have $\delta_{\ell+1}$ from Step 6 of Algorithm 3, the projection onto
the ball ${\mathbb{B}_{d}(0,\epsilon)}$ is evaluated using the following
formula:

(38)
$\displaystyle\delta_{\text{new}}=\delta_{\text{old}}\text{min}\Big{\\{}1,\frac{\epsilon}{\|\delta_{\text{old}}\|_{2}}\Big{\\}}.$

Algorithm 3 Maximization of $f$ over $\epsilon$-ball of a state $s$

1:Input: $s$, $\epsilon$, $\eta_{\delta}$

2:Initialize: $\delta_{0}$

3:($N$ steps of projected gradient descent)

4:for $\ell=1,2,\ldots,N$-$1$

5: (Update $\delta$ in the direction of increase in $f$)

6:
$\delta_{\ell+1}=\delta_{\ell}+\eta_{\delta}\nabla_{\delta}f(s,s+\delta_{\ell})$

7: (Project $\delta_{\ell+1}$ onto the $\epsilon$-ball)

8:
$\delta_{\ell+1}=\Pi_{\mathbb{B}_{d}(0,\epsilon)}\left(\delta_{\ell+1}\right)$

9:end for

##  Appendix D Discussion on smoothness metric

##### An alternative view of smoothness metric

To quantify the smoothness of the learned policy, we had introduced a novel
metric that took the following form:

(39) $\displaystyle
J(\\{\tau_{i}\\}_{i=1}^{N})=\frac{1}{NT}\sum_{i=1}^{N}\sum_{j=1}^{T}\max_{\tilde{s_{j}}\in\mathbb{B}_{d}(s_{j},\epsilon)}\frac{\|\mu(s_{j})-\mu(\tilde{s_{j}})\|}{\|s-\tilde{s_{j}}\|},s_{j}\sim\tau_{i}.$

where $\mu(s)$ is the deterministic mean function for $\pi(\cdot\mid s)$. We
had gotten the above form for the metric by starting out by defining Lipschitz
constant $L$ corresponding to the deterministic mean function, $\mu(s)$ for a
general norm is given by

(40) $\displaystyle L=\max_{s_{1}\neq
s_{2}}\frac{\|\mu(s_{1})-\mu(s_{2})\|}{\|s_{1}-s_{2}\|},\quad
s_{1},s_{2}\in\mathcal{S}.$

Taking $s_{1}=s+\delta s$ and $s_{2}=s$

(41) $\displaystyle L=\max_{\delta s\neq 0}\frac{\|\mu(s+\delta
s)-\mu(s)\|}{\|s+\delta s-s\|},$

Using the first order Taylor series approximation for $\mu$ at $s$ :
$\mu(s+\delta s)=\mu(s)+J_{\mu(s)}\delta s$ we get

(42) $\displaystyle L=\max_{\delta s_{2}\neq 0}\frac{\|J_{\mu(s)}\delta
s\|}{\|\delta s\|},$

For $L^{2}$ norm, the quantity on the right in Eqn. 42 is the spectral norm of
$J_{\mu(s)}$. Hence, the local Lipschitz constant, $L$ at a particular state
$s$ is given by the spectral norm of the Jacobian, $J$ at that state:
$\|J_{\mu}(s)\|_{2}=\sigma_{max}(J_{\mu}(s))$ , i.e., the maximum singular
value (Teel, 1995; Eriksson et al., 2003). Therefore, another approach to
evaluate the smoothness of $\pi$, is to estimate the expected Jacobian norm
using sampled trajectories as,

(43)
$\displaystyle\hat{E}[\|J_{\mu(s)}\|_{2}]=\frac{1}{NT}\sum_{i=1}^{N}\sum_{j=1}^{T}\|J_{\mu(s_{j})}\|_{2},\quad
s_{j}\sim\tau_{i}$

where $T$ and $N$ are the number of sampled trajectories and time steps,
respectively. A sampled trajectory, $\tau\sim\pi$ is a trajectory of the form
$\\{s_{0},a_{0},s_{1},a_{1},.....,s_{T}\\}$, where $s_{0}\sim\rho_{0}$ is the
starting state, $a_{t}\sim\pi(\cdot|s_{t})$, and $T$ denotes the time step at
which we terminate an episode. The quantity in Eqn. 43 is the average spectral
norm instead of the maximum of the norms. Hence, in our work, we use Eqn. 39
as the smoothness metric. $J(\\{\tau_{i}\\}_{i=1}^{N})$ in Eqn. 39 can be
estimated using the batch $\\{\tau_{i}\\}_{i=1}^{N}$ of data sampled from any
policy.

##  Appendix E More results

We include results from some more experiments here. Fig. 6 shows the average
return (and one standard deviation) over an evaluation environment. Table 4
reports average return and average metric for perturbed models various tasks
where perturbation is the variance of Gaussian noise added to the original
model parameters.