# A Novel Variational Lower Bound For
Inverse Reinforcement Learning

Yikang Gui, Prashant Doshi
School of Computing
University of Georgia
{yikang.gui<EMAIL_ADDRESS>

###### Abstract

Inverse reinforcement learning (IRL) seeks to learn the reward function from
expert trajectories, to understand the task for imitation or collaboration
thereby removing the need for manual reward engineering. However, IRL in the
context of large, high-dimensional problems with unknown dynamics has been
particularly challenging. In this paper, we present a new Variational Lower
Bound for IRL (VLB-IRL), which is derived under the framework of a
probabilistic graphical model with an optimality node. Our method
simultaneously learns the reward function and policy under the learned reward
function by maximizing the lower bound, which is equivalent to minimizing the
reverse Kullback-Leibler divergence between an approximated distribution of
optimality given the reward function and the true distribution of optimality
given trajectories. This leads to a new IRL method that learns a valid reward
function such that the policy under the learned reward achieves expert-level
performance on several known domains. Importantly, the method outperforms the
existing state-of-the-art IRL algorithms on these domains by demonstrating
better reward from the learned policy.

##  1 Introduction

Reinforcement learning (RL) is a popular method for automating decision making
and control. However, to achieve practical effectiveness, significant
engineering of reward features and reward functions has traditionally been
necessary. Recently, the advent of deep reinforcement learning has eased the
need for feature engineering for policies and value functions, and
demonstrated encouraging outcomes for various complex tasks such as vision-
based robotic control and video games like Atari and Minecraft. Despite these
advancements, reward engineering continues to be a significant hurdle to the
application of reinforcement learning in practical contexts.

The problem at hand is to learn a reward function that explains the task
performed by the expert, where the learner only has access to a limited number
of expert trajectories and cannot ask for more data. While imitation learning
is a popular technique for training agents to mimic expert behavior,
conventional methods such as behavior cloning and generative adversarial
imitation learning Ho & Ermon (2016) often do not explicitly learn the
underlying reward function, which is essential for a deeper understanding of
the task. To overcome this limitation, researchers have developed inverse
reinforcement learning (IRL) to infer the reward function from expert
trajectories. IRL offers several advantages over direct policy imitation,
including the ability to analyze imitation learning (IL) algorithms, deduce
agents’ intentions, and optimize rewards in new environments.

In this paper, we present a new probabilistic graphical model involving both
the reward function and optimality as key random variables, which can serve as
a framework for representing IRL as a probabilistic inference problem. Within
the context provided by this model, our main contributions are:

  1. 1.

A lower bound on the likelihood of the reward model given the expert
trajectories as data, which is derived from first principles, and involves
minimizing the reverse Kullback–Leibler divergence between an approximated
distribution of optimality given the reward function and the true distribution
of optimality given trajectories.

  2. 2.

A novel IRL algorithm which learns the reward function as variational
inference that optimizes the lower bound in domains where the state and action
spaces can be continuous.

  3. 3.

Improved learning performance of the algorithm compared to state of the art
techniques as seen through the policy given the learned reward, which is
demonstrated on multiple well-known continuous domains of various complexity.

Our novel lower bound can not only serve as a point of departure for the
formulation of other principled bounds but also stimulate discussions and
design of new IRL methods that utilize such bounds.

##  2 A Novel Variational Lower Bound

Inspired by the significant impact of variational lower bounds such as ELBO in
RL, we present a variational lower bound on the likelihood of the reward
function. To derive this lower bound from first principles, we first introduce
a graphical model to represent the problem in Section 2.1 followed by the
derivation of the lower bound in Section 2.2. We discuss the appropriateness
of an inherent approximation in Section 2.3. All proof can be found in
Appendix A.

###  2.1 A Probabilistic Graphical Model of IRL

Inspired by the graphical model for forward RL as shown in Levine (2018) and a
general lack of such modeling of IRL, we aim to fill this gap in this section.
Recall that we are now interested in learning the reward function given the
state and action trajectories from the expert. Therefore, it is necessary to
embed the reward function explicitly in the graphical model, as shown in
Figure 1. The reward value $r_{t}$ is conditioned on state $\mathbf{s_{t}}$
and action $\mathbf{a_{t}}$ respectively denoting the state and action in the
trajectory. We introduce an optimality random variable
$\mathbf{\mathcal{O}_{t}}$, whose value is conditioned on the reward value
$r_{t}$. The optimality follows the definition of Levine (2018) in that the
optimality variable is a binary random variable, where $\mathcal{O}_{t}=1$
denoting that the observed action at time step $t$ is optimal given the state,
and $\mathcal{O}_{t}=0$ denotes that it is not optimal. We will discuss the
reward function $\mathbf{\mathcal{R}}$ in more detail in Section 2.3.

$\mathbf{a_{t}}$$\mathbf{s_{t}}$$r_{t}$$\mathbf{\mathcal{R}}$$\mathbf{\mathcal{O}_{t}}$$\mathbf{a_{t+1}}$$\mathbf{s_{t+1}}$$r_{t+1}$$\mathbf{\mathcal{R}}$$\mathbf{\mathcal{O}_{t+1}}$
(a) Graphical model with reward function included

$\mathbf{a_{t}}$$\mathbf{s_{t}}$$r_{t}$$V_{t}$$\mathbf{\mathcal{R}}$$\mathbf{\mathcal{O}_{t}}$$\mathbf{a_{t+1}}$$\mathbf{s_{t+1}}$$r_{t+1}$$V_{t+1}$$\mathbf{\mathcal{R}}$$\mathbf{\mathcal{O}_{t+1}}$
(b) Graphical model with value function included

Figure 1: A probabilistic graphical model for the IRL problem. Shaded nodes
represent the observed variables, unshaded nodes represent the latent
variables. The dashed line represents the reparameterization trick described
later in Section 2.3. Reward value $r_{t}$ is sampled from distribution
$\mathcal{R}(s_{t},a_{t})$ using reparameterization trick.

The choice of a probabilistic graphical model to represent the relationships
between the state, action, reward, and optimality variables in IRL is
motivated by the desire to capture the complex dependencies between these
variables in a lucid manner. Indeed, the graphical model has a natural
property when it comes to analyzing optimality. Specifically, the graphical
model represents the conditional independence of the optimality from the state
and action variables, given the reward variable. In other words, when given
the state-action pair $(s_{t},a_{t})$, we sum over the reward value $r_{t}$ to
get the marginal conditional distribution of the optimality $\mathcal{O}_{t}$,
that is $p(\mathcal{O}_{t}|s_{t},a_{t})$. However, if the reward value $r_{t}$
is given, then the state-action pair is of course unnecessary in order to get
the marginal distribution of the optimality $\mathcal{O}_{t}$, that is
$p(\mathcal{O}_{t}|r_{t})$.

Given the conditional independence in a probabilistic graphical model, which
is similar to a Bayesian network, we may write:

$\displaystyle p(\mathcal{O}_{t}|r_{t},s_{t},a_{t})=p(\mathcal{O}_{t}|r_{t})$
(1)

which leads to the following marginal:

$\displaystyle\begin{split}p(\mathcal{O}_{t}|s_{t},a_{t})=\int_{r_{t}}p(\mathcal{O}_{t}|r_{t},s_{t},a_{t})\leavevmode\nobreak\
p(r_{t}|s_{t},a_{t})=\int_{r_{t}}p(\mathcal{O}_{t}|r_{t})\leavevmode\nobreak\
p(r_{t}|s_{t},a_{t})\\\ \end{split}$ (2)

###  2.2 Variational Lower Bound for IRL

We use the graphical models of Fig. 1 to formulate the log-likelihood of the
observed trajectories. 111In practice, using full trajectories to estimate the
reward function in IRL may lead to high variance especially when the
trajectories are long. On the other hand, using single state-action pairs to
infer the reward function can lead to more stable and efficient estimates,
because each observed pair provides a separate estimate of the reward function
that can be combined to reduce variance. Here, we are aided by Eqs. 1 and 2 in
our derivation, which we detail below.

$\begin{split}\log
p(\tau)&=\log\int_{\mathcal{O}_{t}}\int_{r_{t}}p(\tau,\mathcal{O}_{t},r_{t})\\\
&=\log
p(s_{1})+\log\prod_{t}\int_{\mathcal{O}_{t}}\int_{r_{t}}p(r_{t}|s_{t},a_{t})\leavevmode\nobreak\
p(\mathcal{O}_{t}|r_{t})\leavevmode\nobreak\
p(s_{t+1}|s_{t},a_{t})\leavevmode\nobreak\ p(a_{t})\\\ &=\log
p(s_{1})+\sum_{t}\log\int_{\mathcal{O}_{t}}\leavevmode\nobreak\
p(\mathcal{O}_{t}|s_{t},a_{t})\leavevmode\nobreak\
p(s_{t+1}|s_{t},a_{t})\leavevmode\nobreak\ p(a_{t})\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \text{(using
Eq.\leavevmode\nobreak\ \ref{eqn:marginal-distribution})}\\\ &=\log
p(s_{1})+\sum_{t}\log\int_{\mathcal{O}_{t}}\frac{p(\mathcal{O}_{t}|s_{t},a_{t})}{q(\mathcal{O}_{t}|r_{t})}q(\mathcal{O}_{t}|r_{t})\leavevmode\nobreak\
p(s_{t+1}|s_{t},a_{t})\leavevmode\nobreak\ p(a_{t}).\\\ \end{split}$

Here, we use variational distribution $q(\mathcal{O}_{t}|r_{t})$ to
approximate the true distribution $p(\mathcal{O}_{t}|s_{t},a_{t})$. We may
think of $q(\mathcal{O}_{t}|r_{t})$ as an approximation of the true
distribution of optimality given the trajectories,
$p(\mathcal{O}_{t}|s_{t},a_{t})$ and we will discuss in more detail in Section
2.3. We may rewrite the equation above by introducing the expectation
$\mathbb{E}_{q(\mathcal{O}_{t}|r_{t})}$ and applying Jensen’s inequality:

$\begin{split}\log p(\tau)&=\log
p(s_{1})+\sum_{t}\log\mathbb{E}_{q(\mathcal{O}_{t}|r_{t})}\left[\frac{p(\mathcal{O}_{t}|s_{t},a_{t})}{q(\mathcal{O}_{t}|r_{t})}p(s_{t+1}|s_{t},a_{t})p(a_{t})\right]\\\
&\geq\log
p(s_{1})+\sum_{t}\mathbb{E}_{q(\mathcal{O}_{t}|r_{t})}\left[\log\frac{p(\mathcal{O}_{t}|s_{t},a_{t})}{q(\mathcal{O}_{t}|r_{t})}+\log
p(s_{t+1}|s_{t},a_{t})+\log p(a_{t})\right]\\\ &=\log
p(s_{1})+\sum_{t}[-D_{KL}(q(\mathcal{O}_{t}|r_{t})\parallel
p(\mathcal{O}_{t}|s_{t},a_{t}))+\log p(s_{t+1}|s_{t},a_{t})+\log
p(a_{t})].\end{split}$ (3)

We end up with the following Evidence Lower BOund (ELBO):

$ELBO=\log p(s_{1})+\sum_{t}[-D_{KL}(q(\mathcal{O}_{t}|r_{t})\parallel
p(\mathcal{O}_{t}|s_{t},a_{t}))+\log p(s_{t+1}|s_{t},a_{t})+\log p(a_{t})].$
(4)

The motivation for using $q(\mathcal{O}_{t}|r_{t})$ to approximate
$p(\mathcal{O}_{t}|s_{t},a_{t})$ is quite intuitive that both state-action
pair and reward value can explain the optimality. An interpretation of
$p(\mathcal{O}_{t}|s_{t},a_{t})$ is that the probability can be viewed as the
confidence with which the state-action pair comes from the expert.
Analogously, $q(\mathcal{O}_{t}|r_{t})$ is the confidence with which the
reward value $r_{t}$ belongs to the expert. Additionally, to optimize the
reward function, we have to incorporate the reward function into the log-
likelihood of trajectories.

Note that if we take the partial derivative of Eq. 4 with respect to $r_{t}$,
we obtain,

$\frac{\partial ELBO}{\partial r_{t}}=-\sum_{t}\frac{\partial
D_{KL}(q(\mathcal{O}_{t}|r_{t})\parallel
p(\mathcal{O}_{t}|s_{t},a_{t}))}{\partial r_{t}}$ (5)

because the remaining terms in equation 3 are constant w.r.t. $r_{t}$. The
derivation from first principles above leads to the following theorem.

######  Theorem 1.

Optimizing the evidence lower bound of the log-likelihood of trajectories
w.r.t $r_{t}$ is equivalent to optimizing the reverse KL divergence between
$q(\mathcal{O}_{t}|r_{t})$ and $p(\mathcal{O}_{t}|s_{t},a_{t})$ w.r.t $r_{t}$.

###  2.3 Approximated Distribution of Optimality

Next, we discuss the approximation of the distribution over optimality under
the framework of the probabilistic graphical model and offer insights.

Given a state-action pair $(s_{t},a_{t})$ obtained from some trajectory
$\tau$, $p(\mathcal{O}_{t}|s_{t},a_{t})$ informs whether the action $a_{t}$ is
optimal at state $s_{t}$, or in other words, whether the state-action pair
comes from the expert’s trajectory. This term represents the true distribution
in the reverse KL divergence contained in the lower bound. To make this
classification, we may simply use binary logistic regression,
$C_{\bm{\theta}}$. In algorithms such as GAIL and AIRL, the input to the
classifier consists of trajectories from both the expert and the learner. The
classifier utilizes these trajectories as input to make predictions about the
optimality of the state-action pair.

$p(\mathcal{O}_{t}|s_{t},a_{t})\triangleq C_{\bm{\theta}}(s_{t},a_{t})$ (6)

Similarly, given reward value $r_{t}$, the approximation
$q(\mathcal{O}_{t}|r_{t})$ informs whether the reward value leads to optimal
behavior, i.e., whether it is induced by the expert or not.

Recall that the reward value $r_{t}$ is the feedback from the environment.
Therefore, we propose this reward value as our first approach to estimate the
optimality as defined in Eq. 7.

$\displaystyle q(\mathcal{O}_{t}=1|r_{t})\propto
e^{r_{t}},\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \text{where $r_{t}\sim\mathcal{R}(s_{t},a_{t})$.}$ (7)

Note that the left hand side of Eq. 7 is conditioned on reward value $r_{t}$,
which is distributed according to the distribution $\mathcal{R}(s_{t},a_{t})$.
Thus, we may apply a parameterization trick to sample a specific reward value
$r_{t}$ without losing track of the gradient propagation under the
probabilistic graphical model. The dashed line in Fig. 1 denotes the
reparameterization trick and $r_{t}$ represents the sampled specific reward
value from the distribution of $\mathcal{R}(s_{t},a_{t})$. To illustrate this,
consider the simplistic case where the reward value distribution is a
univariate Gaussian: $\mathcal{R}(s_{t},a_{t})=\mathcal{N}(\mu,\sigma^{2})$
and let $r_{t}\sim\mathcal{R}(s_{t},a_{t})$. In this case, a valid
reparameterization is $r_{t}=\mu+\sigma\epsilon$ (reward values are
distributed around the mean), where $\epsilon$ is an auxiliary noise variable,
$\epsilon\sim\mathcal{N}(0,1)$.

However, it is not sufficient to use the reward values to represent the
optimality because, in an optimal trajectory, not every state-action pair has
the highest reward value at each time step; often we may perform an action
that obtains longer term gain while forgoing greedy rewards. Hence, a more
robust way of computing the optimality given the distribution over reward
value is needed. Here, the advantage function is a candidate for the solution:

$\begin{split}A(s_{t},a_{t})=Q(s_{t},a_{t})-V(s_{t})=r_{\bm{\phi}}(s_{t},a_{t})+\gamma
V(s_{t+1})-V(s_{t}).\end{split}$ (8)

In practice, we can use actor-critic-based policy to retrieve the value
function estimation, such as PPO, SAC, or TD3. . As the reward value is a
component of the advantage function, we can continue to keep track of the
reward function. Therefore, the optimality of reward value can be expressed
as:

$\begin{split}q(\mathcal{O}_{t}=1|r_{t})\propto
A(s_{t},a_{t})=\sigma\left(r_{t}+\gamma
V(s_{t+1})-V(s_{t})\right).\end{split}$ (9)

where $r_{t}\sim\mathcal{R}(s_{t},a_{t})$ and $\sigma$ is the Sigmoid function
to normalize the advantage function.

###  2.4 Understanding the Relationship between Two Distributions over
Optimality

Notice that the two distributions over optimality
$p(\mathcal{O}_{t}|s_{t},a_{t})$ and $q(\mathcal{O}_{t}|r_{t})$ have different
definitions according to Eqs. 1 and 2. In this subsection, we demonstrate the
validity of using $q(\mathcal{O}_{t}|r_{t})$ to approximate
$p(\mathcal{O}_{t}|s_{t},a_{t})$. We begin with Lemma 1 which establishes an
upper bound that is utilized later, and whose proof is in the Appendix.

######  Lemma 1.

$|\mathbb{E}[f(X)]-f(\mathbb{E}[X])|\leq M\text{Var}(X)$, where
$|f^{\prime\prime}(x)|\leq M$ and $\text{Var}(X)$ denotes the variance of the
distribution of $X$.

We directly use Lemma 1 to arrive at the theorem below and present its proof
in the appendix.

######  Theorem 2.

If $|p^{\prime\prime}(\mathcal{O}_{t}=1|r_{t})|<M$, then the approximation
distribution $q(\mathcal{O}_{t}\mid\mathbb{E}[r_{t}])$, where
$r_{t}\sim\mathcal{R}(s_{t},a_{t})$ and
$\mathcal{R}(s_{t},a_{t})=p(r_{t}|s_{t},a_{t})$, approximates the true
distribution $p(\mathcal{O}_{t}|s_{t},a_{t})$ with an approximation error that
is bounded by $M$Var$[r_{t}]$.

Note that the approximation error is bounded by the variance of $r_{t}$
instead of $\mathcal{O}_{t}$ and the motivation of using reward $r_{t}$ to
approximate state-action pair $(s_{t},a_{t})$ comes from the Figure.1, in
which holds the connectivity that node $r_{t}$ is connected to node
$s_{t},a_{t}$ and node $\mathcal{O}_{t}$. Without the connectivity, even if
the variance of $r_{t}$ is zero, the approximation error cannot be bounded.
More detail can be found in Appendix A.2. It is also the connectivity that
holds so that we can derive Eq. 1 and Eq. 2.

From Theorem 2, we know that we can use
$q(\mathcal{O}_{t}\mid\mathbb{E}[r_{t}])$ to approximate
$p(\mathcal{O}_{t}|s_{t},a_{t})$ with a bounded approximation error w.r.t
$\mathrm{Var}(r_{t})$ . Additionally, if the variance of $r_{t}$ is small
enough, then we can use $r_{t}$ to estimate $\mathbb{E}[r_{t}]$. Subsequently,
the objective function of VLB-IRL is defined in the following:

$\begin{split}\mathcal{L}(r_{t})&=\sum_{t}[-D_{KL}(q(\mathcal{O}_{t}|r_{t})\parallel
p(\mathcal{O}_{t}|s_{t},a_{t})))+\lambda\mathrm{Var}(r_{t})]\\\
&=\sum_{t}\left\\{-D_{KL}\left[\sigma(e^{r_{t}+\gamma
V(s_{t+1})-V(s_{t})})\parallel
C_{\bm{\theta}}(s_{t},a_{t})\right]+\lambda\mathrm{Var}(r_{t})\right\\}\end{split}$
(10)

where $\sigma$ is the Sigmoid function to normalize the advantage function and
$\lambda$ is the hyperparameter to control the contribution of variance in the
objective function.

###  2.5 The Reward Ambiguity Problem

In this section, we discuss the reward ambiguity problem. As deduced in Ho &
Ermon (2016), IRL is a dual of an occupancy measure matching problem and the
induced optimal policy is the primal optimum. In the following, we further
draw the conclusion that IRL is a dual of the optimality matching problem and
the reward function induced by the optimality best explains the expert
$\pi^{E}$.

######  Lemma 2.

$p(\mathcal{O}_{t}|s_{t},a_{t}^{*})$ is maximal if and only if for any given
$s_{t}$, the action $a_{t}^{*}$ from the expert among all actions has the
highest probability, $p(\mathcal{O}_{t}=1|s_{t},a_{t})$.

The lemma follows from the definition of the classifier as discussed in
Section 2.3, which classifies whether the state-action pairs are from the
expert or not.

######  Theorem 3.

The reward function $\mathcal{R}$ best explains the expert policy $\pi^{E}$ if
$p(\mathcal{O}_{t}|s_{t},a_{t})$ is maximal and $q(\mathcal{O}_{t}|r_{t})$ is
identical to $p(\mathcal{O}_{t}|s_{t},a_{t})$, where
$r_{t}\sim\mathcal{R}(s_{t},a_{t})$.

From Theorem 3, we are guaranteed to have a reward function $\mathcal{R}$ that
best explains the expert policy. Theorem 3 also offers the benefit that we are
guaranteed to avoid a degenerate reward function as the solution by optimizing
Eq. 5 if $p(\mathcal{O}_{t}|s_{t},a_{t})$ is maximal.

###  2.6 The Noisy Expert Trajectories Problem

Recall that the objective function of VLB-IRL consists of a reverse KL
divergence between $q(\mathcal{O}_{t}|r_{t})$ and
$p(\mathcal{O}_{t}|s_{t},a_{t})$. Compared to the mean-seeking behavior
induced by minimizing forward KL divergence, minimizing reverse KL divergence
has the behavior that it is mode-seeking, which tends to be more robust to
overfitting and can provide better estimates of the true posterior
distribution if the posterior distribution is noisy. This is beneficial for
IRL because trajectories could be noisy and may contain suboptimal actions,
which challenges the learning. By emphasizing the modes of the data
distribution, the reverse KL divergence helps identify the most likely
explanations for the observed behavior, even in the presence of noise or
uncertainty.

Compared to existing state-of-the-art algorithms, most of them use a single
neural network to estimate the reward function. However, in VLB-IRL, we use
two separate neural networks to update the reward function. The first neural
network is the classifier, defined in Eq. 7. The second neural network is the
approximation optimality, defined in Eq. 7 and Eq. 9. The architecture of two
separate neural networks has a natural property that is robust to overfitting
and has better generalization. It is essential for the IRL algorithm with the
presence of noise in expert trajectories.

Another major difference is that most prior algorithms focus on learning
identical state-action marginal distributions and consequently end up learning
the expert’s noisy state-action representation as well. However, in VLB-IRL,
since the true distribution $p(\mathcal{O}_{t}|s_{t},a_{t})$ represents the
optimality conditioned on the state-action pair, it has the ability to
distinguish and generalize the noisy trajectories.

##  3 VLB-IRL Algorithm

We present an algorithm, which we call variational lower bound for IRL, VLB-
IRL (Algorithm 1), for optimizing Eq. 10. To do so, we introduce function
approximations for $\pi$ and $\mathcal{R}$: we fit a parameterized policy
$\pi_{\bm{\psi}}$ with weights $\bm{\psi}$, a parameterized classifier
$C_{\bm{\theta}}$ with weights $\bm{\theta}$ and a parameterized reward
function $R_{\bm{\phi}}$ with weights $\bm{\phi}$. From line 4 to Line 5, we
collect learner policy trajectories and save them into the buffer. From line 8
to line 9, we update the classifier and reward function defined in Section
2.3. At line 10, we update the learner policy based on the learned reward
function.

Algorithm 1 VLB-IRL: Variational Lower Bound for Inverse Reinforcement
Learning

1:Expert trajectories $\tau_{i}^{E}$, $N$: # of iterations

2:Initialize learner policy $\pi^{L}_{\psi}$, trajectory buffer $B$,
classifier $C_{\bm{\theta}}$ and reward function $\mathcal{R}_{\bm{\phi}}$

3:for step $t$ in ${1,...,N}$ do

4: Collect learner policy trajectories
$\tau_{i}^{L}=(s_{0},a_{0},...,s_{T},a_{T})$ by executing $\pi^{L}_{\psi}$

5: Add trajectories $\tau_{i}^{L}$ into trajectory buffer $B$

6: Sample random minibatch state-action pairs $(s_{t}^{L},a_{t}^{L})$ from
Buffer $B$

7: Sample random minibatch state-action pairs $(s_{t}^{E},a_{t}^{E})$ from
expert trajectories $\tau_{i}^{E}$

8: Train classifier $C_{\bm{\theta}}$ via binary logistic regression to
classify $(s_{t}^{L},a_{t}^{L})$ from $(s_{t}^{E},a_{t}^{E})$

9: Update reward function $\mathcal{R}_{\bm{\phi}}$ via Equation 10

10: Update learner policy $\pi^{L}_{\psi}$ with reward function
$\mathcal{R}_{\bm{\phi}}$

11:end for

##  4 Experiments

To ascertain the efficiency of our method VLB-IRL, we use multiple Mujoco
benchmark domains (Todorov et al., 2012), such as LunarLander, Hopper,
Walker2d, HalfCheetah, and Ant. We also use another more realistic, robot-
centered, and goal-based environment, Assistive Gym (Erickson et al., 2020),
to further analyze the performance of VLB-IRL. In our experiments, we compare
the average true reward accrued over 50 episodes by our method versus the
current state-of-the-art methods such as AIRL (Fu et al., 2018a), $f$-IRL (Ni
et al., 2020), EBIL (Liu et al., 2021), IQ-Learn (Garg et al., 2021) and GAIL
(Ho & Ermon, 2016). Through these experiments, we examine two main scenarios:
$(a)$ VLB-IRL’s ability to learn an optimal reward function as compared to the
baselines when noise-free optimal expert trajectories are available. $(b)$
VLB-IRL’s capacity to generalize and learn a robust reward function as
compared to the baselines, when noisy expert trajectories are provided.

###  4.1 Mujoco domains

As part of our generalizability analysis, we consider both continuous and
discrete control environments from the Mujoco library and use twin-delayed
DDPG (TD3) (Fujimoto et al., 2018) and proximal policy optimization (PPO)
(Schulman et al., 2017) for continuous and discrete domains respectively
222The same RL algorithms are used for expert trajectory generation.. We
readily obtain the expertly trained RL models from the popular RL library RL-
Baselines3 Zoo (Raffin, 2020) and generate several expert trajectories
employing them. Using the generated expert trajectories, each method was
trained with 5 random seeds and for each seed, the model with the highest
return was saved. At the end of the training, we evaluate these models on 50
test episodes. We use MLP for the reward function approximation. The
hyperparameters and other relevant details needed for reproducibility are
provided in the Appendix B. and our code is openly available on
GitHub333Double blind review note: we will share our code once our paper gets
accepted.

Performance on optimal trajectories For our first set of experiments, we
directly use the optimal expert trajectories obtained from the aforementioned
pretrained models. Table 1 shows the summary of results obtained and the
statistically best average reward accrued per domain is highlighted in bold.
Our method VLB-IRL improves upon the baselines indicating that VLB-IRL’s
learned reward function successfully explains the expert’s policy.

Table 1: Performance when optimal trajectories are provided. The average
return and standard deviation of the return are reported. The bold method
statistically (t-test using a significance level of 0.01) outperforms other
methods.

Algorithm | LunarLander | Hopper | Walker2d | HalfCheetah | Ant
---|---|---|---|---|---
Random | $-195.92\pm 97.17$ | $21.94\pm 24.36$ | $1.16\pm 6.26$ | $-282.06\pm 83.09$ | $-59.50\pm 103.75$
GAIL | $255.58\pm 8.51$ | $3518.80\pm 54.12$ | $4529.11\pm 102.53$ | $9551.28\pm 96.01$ | $5321.00\pm 26.81$
AIRL | $239.11\pm 55.03$ | $2611.74\pm 890.14$ | $2524.16\pm 824.00$ | $5342.70\pm 1291.19$ | $4360.28\pm 139.10$
$f$-IRL | - | $3458.61\pm 90.05$ | $4595.97\pm 144.92$ | $9618.82\pm 90.21$ | $4037.52\pm 721.58$
IQ-Learn | - | $3523.88\pm 14.83$ | $4719.93\pm 35.33$ | $9592.53\pm 64.74$ | $5072.59\pm 79.30$
VLB-IRL(ours) | $\mathbf{267.99\pm 10.85}$ | $\mathbf{3588.41\pm 6.22}$ | $\mathbf{4779.76\pm 32.25}$ | $\mathbf{9677.64\pm 76.64}$ | $\mathbf{5422.10\pm 83.02}$
Expert | $266.67\pm 12.73$ | $3606.22\pm 4.06$ | $4835.76\pm 52.30$ | $9714.77\pm 154.35$ | $5547.41\pm 1376.67$

Learning performance We find that VLB-IRL continues to perform well in a
limited data regime. The empirical result demonstrates that VLB-IRL exhibits
fairly good sample efficiency. With increasing amount of data, the standard
deviation of the return from the learned policy becomes lower as shown in
Figure 2, indicating better stability in the learned policy, which is a direct
result of a better reward function. To further analyze the training progress
of VLB-IRL, we use Inverse Learning Error (ILE) from Arora & Doshi (2021a), as
$||V^{\pi_{E}}-V^{\pi_{L}}||_{p}$ where $\pi_{E}$ is the expert policy and
$\pi_{L}$ is the learned policy. Here we use the $L2$ norm and the result is
presented in Figure 2. The decreasing ILE shows that both the learned reward
function is getting closer to the true reward function and the learned policy
is more similar to the expert policy.

Figure 2: Left: Performance on different number of expert trajectories. The
return has been normalized to $[0,1]$. Right: ILE during the training. The
training progress has been normalized to $[0,1]$ due to different training
steps for different environments. 0 represents the beginning of the training
and 1 represents the end of the training.

Performance on noisy trajectories In order to further test VLB-IRL’s
generalizability, in our second set of experiments, we provide noisy
trajectories and examine if the learned reward function is robust enough to
learn the noisy expert’s preferences and perhaps outperform them. The noisy
trajectories are generated by using A2C for discrete environments and PPO for
continuous environments in the library RL-Baselines3 Zoo. Compared to PPO for
discrete environments and TD3 for continuous environments, they are less
optimal and contain noisy actions in the trajectories. Compare the expert
performance in Table 1 and Table 2 for detail. The results presented in Table
2 show that VLB-IRL’s accrues comparable or higher reward on average as
compared to the noisy expert.

Table 2: Performance when noisy trajectories are provided. The average return
and standard deviation of the return are reported. The bold method
statistically (t-test using a significance level of 0.01) outperforms other
methods.

Algorithm | LunarLander | Hopper | Walker2d | HalfCheetah | Ant
---|---|---|---|---|---
Random | $-195.92\pm 97.17$ | $21.94\pm 24.36$ | $1.16\pm 6.26$ | $-282.06\pm 83.09$ | $-59.50\pm 103.75$
GAIL | $238.11\pm 18.10$ | $2644.29\pm 412.57$ | $3489.23\pm 211.55$ | $2682.65\pm 366.92$ | $3884.23\pm 938.26$
$f$-IRL | - | $2371.61\pm 236.14$ | $3603.85\pm 164.74$ | $2980.33\pm 124.99$ | $4140.15\pm 508.96$
EBIL | $\mathbf{239.47\pm 54.81}$ | $2073.39\pm 89.69$ | $3295.84\pm 52.73$ | $2499.83\pm 496.12$ | $1520.84\pm 575.88$
IQ-Learn | - | $2617.43\pm 13.82$ | $3612.85\pm 132.51$ | $3201.72\pm 132.54$ | $4738.80\pm 151.51$
VLB-IRL(ours) | $237.61\pm 7.66$ | $\mathbf{2786.22\pm 13.57}$ | $\mathbf{3694.82\pm 117.15}$ | $\mathbf{3375.81\pm 109.29}$ | $\mathbf{5125.00\pm 121.72}$
Noisy Expert | $240.60\pm 45.68$ | $2409.77\pm 9.80$ | $3873.86\pm 165.73$ | $2948.79\pm 383.29$ | $4588.19\pm 1258.34$

For Hopper, HalfCheetah, and Ant environments, the VLB-IRL’s learned policy
outperforms the noisy expert by $15.6\%$, $14.4\%$, and $11.7\%$ respectively.
However, previous IRL techniques fail to generalize well when noisy
trajectories are provided and we suspect that they fall short due to their
optimization objective and divergence metric listed in Table 4.

###  4.2 Realistic Scenario

To further examine the performance of VLB-IRL, we use Assistive Gym (Erickson
et al., 2020), a more realistic, robot-centered, and goal-based environment.
For our experiments, we use environments with an active robot and a static
human, such as FeedingSaywer, BedBathingSawyer, and ScratchItchSawyer. A
description of the environments can be found in Figure.3. We use SAC to train
an expert policy and use the trained expert policy to generate 50 expert
episodes. The detailed hyperparameter can be found in Appendix B.

(a) FeedingSawyer

(b) BedBathingSawyer

(c) ScratchItchSawyer

Figure 3: Assistive gym environment. (a) A Sawyer robot holds a spoon with
small spheres representing food on the spoon and must bring this food to a
person’s mouth without spilling it. (b) A person lies on a bed in a random
resting position while a Sawyer robot must use a washcloth tool to clean off a
person’s right arm. (c) A Sawyer robot holds a small scratching tool and must
reach towards a random target scratching location along a person’s right arm.

The challenge in Assistive Gym compared to Mujoco benchmark is that it is a
goal-based environment and the agent only receives a positive reward after the
completion of the goal, unlike Mujoco where the agent can receive positive
rewards at every timestep. This goal-based environment will generate much more
noisy expert trajectories. Table 3 shows that VLB-IRL outperforms all other
baselines and VLB-IRL is the only method that can successfully achieve the
goal.

Table 3: Performance on Assistive Gym environments. The bold method
statistically (t-test using a significance level of 0.01) outperforms other
methods.

Algorithm | FeedingSawyer | BedBathingSawyer | ScratchItchSawyer |
---|---|---|---|---
Random | $-106.39\pm 9.21$ | $-21.42\pm 3.26$ | $-31.74\pm-5.50$ |
GAIL | $-50.68\pm 76.33$ | $-6.63\pm 14.38$ | $-23.75\pm 6.48$ |
AIRL | $-76.13\pm 15.58$ | $-15.22\pm 6.95$ | $-27.85\pm 8.40$ |
$f$-IRL | $-65.38\pm 19.64$ | $-9.69\pm 19.82$ | $-25.91\pm 5.08$ |
IQ-Learn | $-30.33\pm 49.39$ | $-2.34\pm 11.93$ | $-15.32\pm 9.55$ |
VLB-IRL(ours) | $\mathbf{88.11\pm 52.95}$ | $\mathbf{10.86\pm 13.94}$ | $\mathbf{11.94\pm 24.44}$ |
Expert | $117.74\pm 30.42$ | $67.60\pm 37.22$ | $61.64\pm 29.16$ |

##  5 Related Work

IRL was introduced more than two decades ago and Abbeel & Ng (2004)’s
Apprenticeship Learning, which sought to learn the reward function with the
maximum margin, provided significant early impetus to IRL’s development.
Subsequently, Bayesian methods for IRL, which viewed the trajectories as
observations, were introduced Ramachandran & Amir (2007). Subsequently, Choi &
Kim (2011) searches for the maximum-a-posteriori reward function instead of
settling for the posterior mean. As it is hard to integrate over the entire
reward space, the posterior mean may not be an ideal proposal for the reward
inference. However, Bayesian methods are now known to be severely impacted by
the issue of being unable to scale to large domains. Toward this challenge,
recently Chan & van der Schaar (2021) scales Bayesian IRL to learn in the
context of complex state spaces and mitigates reward uncertainty by using a
variational approximation of the posterior distribution over reward and can be
executed entirely offline.

Fu et al. (2018a) presents adversarial IRL (AIRL), a practical and scalable
IRL algorithm based on an adversarial reward learning formulation, which has
received significant attention and is one of our baseline techniques in this
paper. More recently, Ghasemipour et al. (2019) demonstrated that the
objective in AIRL is equivalent to minimizing the reverse KL divergence
between the joint state-action distribution induced by the learned policy and
that of the expert. However, this has been disputed by Ni et al. (2020), which
claims that AIRL is not optimizing reverse KL divergence in the state-action
marginal. Ni et al. (2020) also introduces $f$-IRL that optimizes the
$f$-divergence measure, and is one of our baseline methods as well.

Although VLB-IRL and AIRL can be identified as adversarial method, VLB-IRL and
AIRL come from different frameworks. AIRL comes from generative adversarial
network guided cost learning (GAN-GCL) and the discriminator is interpreted as
$D_{\theta,\phi}=\frac{\exp(f_{\theta})}{\exp(f_{\theta})+\pi_{\phi}}$. By
training $D_{\theta,\phi}$ via binary logistic regression and placing a
special structure, i.e. advantage function, on the discriminator, the reward
can be recovered. VLB-IRL comes from the graphical model and the graphical
model naturally hold the conditional independence, leading to two proposed
interpretation of optimality, i.e. $q(\mathcal{O}_{t}|r_{t})$ and
$p(\mathcal{O}_{t}|s_{t},a_{t})$. By optimizing the reverse KL divergence in
ELBO of log-likelihood of trajectory $p(\tau)$, i.e.
$KL(q(\mathcal{O}_{t}|r_{t})\parallel p(\mathcal{O}_{t}|s_{t},a_{t}))$, the
reward can be recovered. In practice, VLB-IRL is more flexible. AIRL only
accepts stochastic policy, while VLB-IRL is compatible with both stochastic
and deterministic policy.

More recently, IQ-Learn has been proposed by Garg et al. (2021). IQ-Learn is a
method for dynamics-aware IL that avoids adversarial training by learning a
single Q-function, implicitly representing both reward and policy. However, it
is not an IRL method. It sacrifices reward estimation accuracy by indirectly
recovering rewards through a soft Q-function approximator, which relies
heavily on dynamic environmental factors and doesn’t strictly adhere to the
soft-Bellman equation.

Table 4: Relevant IRL and imitation learning methods.

Algorithm | Optimization Space | Optimization Objective | Divergence measure | Type
---|---|---|---|---
GAIL | $s,a$ | $\rho(s,a)$ | Jensen-Shannon | IL
AIRL | $\tau$ | $\rho(s,a)$ | Forward KL | IRL
$f$-IRL | $s$ | $\rho(s)$ |  $f$-divergence | IRL
EBIL | $\tau$ | $\rho(s,a)$ | Reverse KL | IRL
IQ-Learn | $s,a$ | $\rho(s,a)$ |  $f$-divergence | IL
VLB-IRL(ours) | $s,a$ | $\mathcal{O}(s,a)$ | Reverse KL | IRL

##  6 Conclusion

In this paper, we derive a novel variational lower bound on the log-likelihood
of the trajectories for IRL and present an algorithm that maximizes this lower
bound. By posting IRL through the framework of a probabilistic graphical model
with an optimality node, the optimization dual problem becomes minimizing the
reverse KL divergence between an approximate distribution of optimality given
the reward function and the true distribution of optimality given the
trajectories. In particular, it models IRL as an inference problem within the
framework of PGMs. Our experiments demonstrate that VLB-IRL has better
generalization in the presence of noisy expert trajectories compared to
popular baselines. In addition, when (optimal) expert trajectories are
available, VLB-IRL maintains a performance advantage compared to existing
algorithms. The formulation of the graphical model opens up the possibility
for a wide variety of new IRL techniques that can interpret the framework in
various ways. Extending VLB-IRL to multi-agent environments and extending the
interpretation of the optimality node for multi-agent learning is an
intriguing direction.

A limitation of the VLB-IRL is that the tightness of its lower bound to the
true log-likelihood is yet to be established, and VLB-IRL is not guaranteed to
converge to the optimum, which is a limitation shared by many other
adversarial IRL algorithms as well. Another common limitation of the
adversarial algorithm is the unstable training progress.

## References

  * Abbeel & Ng (2004) Pieter Abbeel and Andrew Y Ng.  Apprenticeship learning via inverse reinforcement learning.  In _Proceedings of the twenty-first international conference on Machine learning_ , pp. 1, 2004.
  * Arora & Doshi (2021a) Saurabh Arora and Prashant Doshi.  A survey of inverse reinforcement learning: Challenges, methods and progress.  _Artificial Intelligence_ , 297:103500, 2021a.  ISSN 0004-3702.  doi: https://doi.org/10.1016/j.artint.2021.103500.
  * Arora & Doshi (2021b) Saurabh Arora and Prashant Doshi.  A survey of inverse reinforcement learning: Challenges, methods and progress.  _Artificial Intelligence_ , 297:103500, 2021b.
  * Bengio & LeCun (2007) Yoshua Bengio and Yann LeCun.  Scaling learning algorithms towards AI.  In _Large Scale Kernel Machines_. MIT Press, 2007.
  * Chan & van der Schaar (2021) Alex James Chan and Mihaela van der Schaar.  Scalable bayesian inverse reinforcement learning.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.
  * Choi & Kim (2011) Jaedeug Choi and Kee-Eung Kim.  Map inference for bayesian inverse reinforcement learning.  _Advances in neural information processing systems_ , 24, 2011.
  * Choi & Kim (2012) Jaedeug Choi and Kee-Eung Kim.  Nonparametric bayesian inverse reinforcement learning for multiple reward functions.  _Advances in neural information processing systems_ , 25, 2012.
  * Erickson et al. (2020) Zackory Erickson, Vamsee Gangaram, Ariel Kapusta, C. Karen Liu, and Charles C. Kemp.  Assistive gym: A physics simulation framework for assistive robotics.  _IEEE International Conference on Robotics and Automation (ICRA)_ , 2020.
  * Finn et al. (2016) Chelsea Finn, Sergey Levine, and Pieter Abbeel.  Guided cost learning: Deep inverse optimal control via policy optimization.  In _International conference on machine learning_ , pp. 49–58. PMLR, 2016.
  * Fu et al. (2018a) Justin Fu, Katie Luo, and Sergey Levine.  Learning robust rewards with adverserial inverse reinforcement learning.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net, 2018a.
  * Fu et al. (2018b) Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine.  Variational inverse control with events: A general framework for data-driven reward definition.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc., 2018b.
  * Fujimoto et al. (2018) Scott Fujimoto, Herke Hoof, and David Meger.  Addressing function approximation error in actor-critic methods.  In _International conference on machine learning_ , pp. 1587–1596. PMLR, 2018.
  * Garg et al. (2021) Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon.  IQ-learn: Inverse soft-q learning for imitation.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), _Advances in Neural Information Processing Systems_ , 2021.
  * Ghasemipour et al. (2019) Seyed Kamyar Seyed Ghasemipour, Shane Gu, and Richard Zemel.  Understanding the relation between maximum-entropy inverse reinforcement learning and behaviour cloning, 2019.
  * Ghasemipour et al. (2020) Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu.  A divergence minimization perspective on imitation learning methods.  In _Conference on Robot Learning_ , pp. 1259–1277. PMLR, 2020.
  * Goodfellow et al. (2016) Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio.  _Deep learning_ , volume 1.  MIT Press, 2016.
  * Guan et al. (2021) Ziwei Guan, Tengyu Xu, and Yingbin Liang.  When will generative adversarial imitation learning algorithms attain global convergence.  In Arindam Banerjee and Kenji Fukumizu (eds.), _The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event_ , volume 130 of _Proceedings of Machine Learning Research_ , pp. 1117–1125. PMLR, 2021.
  * Haarnoja et al. (2018) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In Jennifer G. Dy and Andreas Krause (eds.), _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ , volume 80 of _Proceedings of Machine Learning Research_ , pp. 1856–1865. PMLR, 2018.
  * Hinton et al. (2006) Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh.  A fast learning algorithm for deep belief nets.  _Neural Computation_ , 18:1527–1554, 2006.
  * Ho & Ermon (2016) Jonathan Ho and Stefano Ermon.  Generative adversarial imitation learning.  _Advances in neural information processing systems_ , 29, 2016.
  * Jain et al. (2019) Vinamra Jain, Prashant Doshi, and Bikramjit Banerjee.  Model-free IRL using maximum likelihood estimation.  In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019_ , pp. 3951–3958. AAAI Press, 2019.  doi: 10.1609/aaai.v33i01.33013951.
  * Kim et al. (2021) Kuno Kim, Shivam Garg, Kirankumar Shiragur, and Stefano Ermon.  Reward identification in inverse reinforcement learning.  In Marina Meila and Tong Zhang (eds.), _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , pp. 5496–5505. PMLR, 2021.
  * Levine (2018) Sergey Levine.  Reinforcement learning and control as probabilistic inference: Tutorial and review.  _arXiv preprint arXiv:1805.00909_ , 2018.
  * Liu et al. (2021) Minghuan Liu, Tairan He, Minkai Xu, and Weinan Zhang.  Energy-based imitation learning.  In _Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems_ , AAMAS ’21, pp. 809–817. International Foundation for Autonomous Agents and Multiagent Systems, 2021.
  * Mnih et al. (2013) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing atari with deep reinforcement learning, 2013.
  * Ni et al. (2020) Tianwei Ni, Harshit Sikchi, Yufei Wang, Tejus Gupta, Lisa Lee, and Ben Eysenbach.  f-irl: Inverse reinforcement learning via state marginal matching.  In _Conference on Robot Learning_ , 2020.
  * Qureshi et al. (2019) Ahmed Hussain Qureshi, Byron Boots, and Michael C. Yip.  Adversarial imitation via variational inverse reinforcement learning.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net, 2019.
  * Raffin (2020) Antonin Raffin.  Rl baselines3 zoo.  https://github.com/DLR-RM/rl-baselines3-zoo, 2020.
  * Ramachandran & Amir (2007) Deepak Ramachandran and Eyal Amir.  Bayesian inverse reinforcement learning.  In _Proceedings of the 20th International Joint Conference on Artifical Intelligence_ , IJCAI’07, pp. 2586–2591. Morgan Kaufmann Publishers Inc., 2007.
  * Schlaginhaufen & Kamgarpour (2023) Andreas Schlaginhaufen and Maryam Kamgarpour.  Identifiability and generalizability in constrained inverse reinforcement learning.  In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), _International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA_ , volume 202 of _Proceedings of Machine Learning Research_ , pp. 30224–30251. PMLR, 2023.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Todorov et al. (2012) Emanuel Todorov, Tom Erez, and Yuval Tassa.  Mujoco: A physics engine for model-based control.  In _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pp. 5026–5033. IEEE, 2012.  doi: 10.1109/IROS.2012.6386109.
  * Viano et al. (2021) Luca Viano, Yu-Ting Huang, Parameswaran Kamalaruban, Adrian Weller, and Volkan Cevher.  Robust inverse reinforcement learning under transition dynamics mismatch.  In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , pp. 25917–25931, 2021.
  * Ziebart et al. (2008) Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey.  Maximum entropy inverse reinforcement learning.  In Dieter Fox and Carla P. Gomes (eds.), _Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence, AAAI 2008, Chicago, Illinois, USA, July 13-17, 2008_ , pp. 1433–1438. AAAI Press, 2008.

##  Appendix A Proof

###  A.1 Proof for Lemma 1

###### Proof.

Let $x_{0}:=\mathbb{E}[X]$ and we have the following Taylor expansion:

$\displaystyle
f(x)=f(x_{0})+f^{\prime}(x_{0})(x-x_{0})+\frac{f^{\prime\prime}(\xi(x))}{2}(x-x_{0})^{2}$
(11)

where $\xi(x)\in(x_{0},x)$. Replacing $x$ with the random variable $X$ and
taking the expectation results in,

$\displaystyle\mathbb{E}[f(x)]$
$\displaystyle=f(x_{0})+f^{\prime}(x_{0})\mathbb{E}[x-x_{0}]+\mathbb{E}\left[\frac{f^{\prime\prime}(\xi(x))}{2}(x-x_{0})^{2}\right]=f(x_{0})+\mathbb{E}\left[\frac{f^{\prime\prime}(\xi(x))}{2}(x-x_{0})^{2}\right].$

If function $f$ has a bounded second derivative, $|f^{\prime\prime}(x)|<M$ for
all $x$, then we have

$\displaystyle|\mathbb{E}[f(x)]-f(x_{0})|=|\mathbb{E}[f(x)]-f(\mathbb{E}[x])|$
$\displaystyle\leq M\mathbb{E}[(X-x_{0})^{2}]=M\text{Var}(X).$

∎

###  A.2 Proof for Theorem 2

###### Proof.

As $\mathcal{O}_{t}$ is a binary variable, $p(\mathcal{O}_{t}|s_{t},a_{t})$
and $q(\mathcal{O}_{t}\mid\mathbb{E}[r_{t}])$ are Bernoulli distributions,

$\displaystyle q(\mathcal{O}_{t}=0\mid\mathbb{E}[r_{t}])$
$\displaystyle=1-q(\mathcal{O}_{t}=1\mid\mathbb{E}[r_{t}])$ $\displaystyle
p(\mathcal{O}_{t}=0|s_{t},a_{t})$
$\displaystyle=1-p(\mathcal{O}_{t}=1|s_{t},a_{t})$

We prove that $q(\mathcal{O}_{t}=1\mid\mathbb{E}[r_{t}])$ is a valid candidate
for approximating $p(\mathcal{O}_{t}=1|s_{t},a_{t})$.

$p(\mathcal{O}_{t}=1|s_{t},a_{t})=\int_{r_{t}}p(\mathcal{O}_{t}=1|s_{t},a_{t},r_{t})\leavevmode\nobreak\
p(r_{t}|s_{t},a_{t})=\mathbb{E}_{r_{t}}[p(\mathcal{O}_{t}=1|r_{t})]\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \text{(using
Eq.\leavevmode\nobreak\ \ref{eqn:marginal-distribution})}$ (12)

Let $f(r_{t}):=p(\mathcal{O}_{t}=1|r_{t})$, Eq. 12 now becomes

$\displaystyle p(\mathcal{O}_{t}=1|s_{t},a_{t})=\mathbb{E}_{r_{t}}[f(r_{t})]$
(13)

And

$\displaystyle f(\mathbb{E}[r_{t}]):$
$\displaystyle=p(\mathcal{O}_{t}=1\mid\mathbb{E}[r_{t}])$ (14)
$\displaystyle=q(\mathcal{O}_{t}=1\mid\mathbb{E}[r_{t}])$ (15)

Given the condition $|p^{\prime\prime}(\mathcal{O}_{t}=1|r_{t})|<M$, Eq. 13
and Eq. 15, applying Lemma 1, we get

$\displaystyle|\mathbb{E}[f(r_{t})]-f(\mathbb{E}[r_{t}])|$
$\displaystyle=|p(\mathcal{O}_{t}=1|s_{t},a_{t})-q(\mathcal{O}_{t}=1\mid\mathbb{E}[r_{t}])|\leq
M\text{Var}(r_{t})$ (16)

∎

###  A.3 Proof for Theorem 3

###### Proof.

If $p(\mathcal{O}_{t}|s_{t},a_{t})$ is maximal, from Lemma 2, we obtain that

$a^{*}_{t}=\arg\max_{a_{t}}p(\mathcal{O}_{t}=1|s_{t},a_{t})$ (17)

In addition, if $q_{\phi}(\mathcal{O}_{t}|r_{t})$ is identical to
$p_{\theta}(\mathcal{O}_{t}|s_{t},a_{t})$,

$\begin{split}a^{*}_{t}&=\arg\max_{a_{t}}p(\mathcal{O}_{t}=1|s_{t},a_{t})\\\
&=\arg\max_{a_{t}}q_{\phi}(\mathcal{O}_{t}=1|r_{t})\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \text{(given the assumption
above)}\\\ &=\arg\max_{a_{t}}\sigma(A_{\phi}(s_{t},a_{t})).\end{split}$ (18)

Here, $A_{\phi}(s_{t},a_{t}^{*})$ parameterized by $r_{\phi}(s_{t},a_{t})$ has
the highest value given $s_{t}$ for any $a_{t}$, which best explains the
expert policy $\pi^{E}$. ∎

##  Appendix B Implementation and Hyperparameter

###  B.1 Implementation

In this section, we discuss several implementation details for making training
stable and getting a better reward function and learner policy.

Iterative updates. Many algorithms for IRL exhibit a nested structure,
involving an inner loop and an outer loop. The inner loop focuses on finding
the optimal policy given parameterized rewards. However, this step is costly
and we can avoid this by applying the iterative updates. We borrow the idea
from Finn et al. (2016), updating the reward function and learning policy in
parallel without finding the optimal policy given parameterized rewards.

Learner Policy Rollouts Buffer. There are two reasons to use the learner
policy rollouts buffer. First, recall that in Section 2.3, we get the
approximation of the distribution of optimality given trajectory,
$p(\mathcal{O}_{t}|s_{t},a_{t})$ by using a classifier $C_{\theta}$
classifying whether a trajectory comes from an expert or learner. In
supervised learning with deep neural networks, it is common practice to
compute gradient estimates based on a minibatch of samples rather than
individual samples. It is essential to ensure that the samples are
independently and identically distributed (i.i.d) from a fixed distribution.
However, if we sample minibatches from the most recent learner policy, the
i.i.d assumption cannot be held anymore because they are correlated. This is
similar to the scenario described in Mnih et al. (2013). By applying the
learner policy rollouts buffer, we can alleviate the problems of correlated
data and non-stationary distributions. Second, to avoid catastrophic
forgetting, which impedes the training process heavily, the simplest solution
is to store past experiences. By including previous experiences in the
training set, the neural network benefits from a diverse range of samples,
leading to a reduced risk of catastrophic forgetting.

###  B.2 Hyperparameter and Other Implementation Details

Environment: We use the LunarLander-v2, Hopper-v3, Ant-v3, HalfCheetah-v3,
Walker2d-v3 from OpenAI Gym. We use FeedingSawyer-v1, BedBathingSawyer-v1,
ScratchItchSawyer-v1 from Assistive Gym.

Expert Policy: For OpenAI Gym environments, we readily obtain the expertly
trained RL models from the popular RL library RL-Baselines3 Zoo (Raffin,
2020). We use TD3 and PPO for continuous and discrete action environments for
optimal trajectories collection respectively. We use PPO and A2C for
continuous and discrete action environments for sub-optimal trajectories
collection respectively. For Assistive Gym environments, we train the expert
policy using SAC from Stable-Baselines3. We use policy network as $[512,512]$
and action noise with $\mathcal{N}(0,0.2)$. The remaining hyperparameter
remains the default value.

Training Details: We train 6 algorithms namely VLB-IRL, IQ-Learn, GAIL, AIRL,
$f$-IRL, and EBIL to recover the expert reward function and imitate the expert
behavior using the given expert trajectories.

For OpenAI Gym environments, we train VLB-IRL using Algorithm 1. We use TD3
and PPO for continuous and discrete action environments from the popular RL
library Stable-Baselines3 respectively. The hyperparameter for learner policy
is listed in Table 5 and Table 7 and the unmentioned hyperparameters are the
default value in the Stable-Baselines3 package. The hyperparameters for
learner policy are borrowed from the RL-Baselines3 Zoo, which includes a
collection of hyperparameters for various kinds of algorithms and
environments. The hyperparameter for the reward function network and the
classifier are listed in Table 8 and Table 10 respectively.

For Assistive Gym environments, we train VLB-IRL using Algorithm 1. We use SAC
from the popular RL library Stable-Baselines3. The hyperparameter for learner
policy is listed in Table 6

For GAIL and AIRL, we use the Python package Imitation, which provides clean
implementations of imitation and reward learning algorithms. For IQ-Learn,
EBIL, and $f$-IRL, we refer to their authors’ official implementation. We use
the same hyperparameter setting for learner policy and reward function
network. as VLB-IRL except for AIRL. AIRL only accepts stochastic policy,
which is not true for TD3. Therefore, we use SAC as the underlying RL
algorithm. The hyperparameter for SAC is borrowed from Ni et al. (2020).

Table 5: Hyperparameter setting for learner policy for continuous action OpenAI Gym environments. TD3 | Hopper | Ant | HalfCheetah | Walker2d
---|---|---|---|---
Learning rate | $1e^{-3}$ | $1e^{-3}$ | $1e^{-3}$ | $1e^{-3}$
Gamma | 0.99 | 0.98 | 0.98 | 0.98
Batch size | 256 | 256 | 256 | 256
Gradient steps | 1000 | 1000 | 1000 | 1000
Net arch | [400, 300] | [400, 300] | [400, 300] | [400, 300]
Buffer size | 2000000 | 200000 | 200000 | 200000
Action noise | $\mathcal{N}(0,0.2)$ | $\mathcal{N}(0,0.2)$ | $\mathcal{N}(0,0.2)$ | $\mathcal{N}(0,0.2)$
Table 6: Hyperparameter setting for learner policy for Assistive Gym environments. SAC | FeedingSawyer | BedBathingSawyer | ScratchItchSawyer
---|---|---|---
Learning rate | $3e^{-4}$ | $3e^{-4}$ | $3e^{-4}$
Gamma | 0.97 | 0.97 | 0.97
Batch size | 256 | 256 | 256
Net arch | [512, 512] | [512, 512] | [512, 512]
Action noise | $\mathcal{N}(0,0.2)$ | $\mathcal{N}(0,0.2)$ | $\mathcal{N}(0,0.2)$
Table 7: Hyperparameter setting for learner policy for discrete action OpenAI Gym environments. PPO | LunarLander
---|---
Learning rate | $1e^{-3}$
Gamma | 0.99
Batch size | 100
Entropy coefficient | $1e^{-2}$
Table 8: Hyperparameter setting for reward function network in OpenAI Gym environments. MLP | LunarLander | Hopper | Ant | HalfCheetah | Walker2d
---|---|---|---|---|---
Learning rate | $1e^{-3}$ | $1e^{-3}$ | $1e^{-3}$ | $1e^{-3}$ | $1e^{-3}$
Net arch | [16, 16] | [16, 16] | [32, 32] | [32, 32] | [32, 32]
Optimizer | Adam | Adam | Adam | Adam | Adam
Batch size | 64 | 256 | 256 | 256 | 256
Activation | LeakyReLU | LeakyReLU | LeakyReLU | LeakyReLU | LeakyReLU
Table 9: Hyperparameter setting for reward function network in Assistive Gym environments. MLP | FeedingSawyer | BedBathingSawyer | ScratchItchSawyer
---|---|---|---
Learning rate | $1e^{-3}$ | $1e^{-3}$ | $1e^{-3}$
Net arch | [64, 64] | [64, 64] | [64, 64]
Optimizer | Adam | Adam | Adam
Batch size | 256 | 256 | 256
Activation | LeakyReLU | LeakyReLU | LeakyReLU
Table 10: Hyperparameter setting for the classifier in OpenAI Gym environments. MLP | LunarLander | Hopper | Ant | HalfCheetah | Walker2d
---|---|---|---|---|---
Learning rate | $1e^{-3}$ | $1e^{-3}$ | $1e^{-3}$ | $1e^{-3}$ | $1e^{-3}$
Net arch | [16, 16] | [16, 16] | [32, 32] | [32, 32] | [32, 32]
Optimizer | Adam | Adam | Adam | Adam | Adam
Batch size | 64 | 256 | 256 | 256 | 256
Activation | LeakyReLU | LeakyReLU | LeakyReLU | LeakyReLU | LeakyReLU
Table 11: Hyperparameter setting for the classifier in Assistive Gym environments. MLP | FeedingSawyer | BedBathingSawyer | ScratchItchSawyer
---|---|---|---
Learning rate | $1e^{-3}$ | $1e^{-3}$ | $1e^{-3}$
Net arch | [64, 64] | [64, 64] | [64, 64]
Optimizer | Adam | Adam | Adam
Batch size | 256 | 256 | 256
Activation | LeakyReLU | LeakyReLU | LeakyReLU


# Hydrodynamic study of edge spin-vortex excitations of fractional quantum
Hall fluid

M. Rabiu<EMAIL_ADDRESS>University for Development Studies, Faculty of
Applied Sciences, Department of Applied Physics, Navrongo Campus, Ghana.  S.
Y. Mensah  University of Cape Coast, Department of Physics, Laser and Fiber
Optics Center, Cape Coast, Ghana.  I. Y. Seini  University for Development
Studies, School of Engineering, Department of Mechanical Engineering,
Nyankpala Campus, Ghana.  S. S. Abukari  University of Cape Coast, Department
of Physics, College of Physical Sciences, Cape Coast, Ghana.

###### Abstract

We undertake a theoretical study of edge spin-vortex excitations in fractional
quantum Hall fluid. This is done in view of quantised Euler hydrodynamics
theory. The dispersions of true excitations for fractions within $0\leq\nu\leq
1$ are simulated which exhibit universal similarities and differences in
behaviour. The differences arise from different edge smoothness and spin
(pseudo-spin) polarisations, in addition to spin-charge competition. In
particular, tuning the spin-charge factor causes coherent spin flipping
associated with partial and total polarisations of edge spin-vortices. This
observation is tipped as an ideal mechanism for realisation of functional
spintronic devices.

Euler hydrodynamics, Quantum hall vortex fluid, Quantised vortex dynamics,
Fractional quantum hall effect, Edge spin excitations.

††preprint: AIP/123-QED

##  I Introduction

Two dimensional electron gas system submitted to strong magnetic field usually
form strongly correlated quantum Hall fluids. The fluids are often
incompressible Laughlin1983  and characterised by dissipationless superfluid
flow PAlexandr2012  and formation of quantised vortices LagoudakisK2008 .
Integer and fractional quantum Hall states are examples of quantum Hall fluids
(QHFs). The distinction arises from an integer or fractional factor connecting
the number of formed quantised vortices to a magnetic flux number associated
with the applied field. The fractional factors present richer physics content
than its integer cousin. These include the braiding statistics Laughlin1983
and the recently conjectured double boundary layer WiegmannP2013 . In the same
vein, the edges of fractional QHFs, under small Zeeman splitting, supports
spin textures HZi-Xiang2011 . The case of a QHF with the accompanying
properties in graphene will be particularly interesting. Chiefly due to
graphene unique magnetotransport properties MRabiu2012 ; LKayoung2011 ;
OostingaJ2010 , unusual THz generation and amplification MRabiu2012b  and
observation of anomalous quantum Hall states at room temperature DXu2009 ;
BolotinI2009 ; DeanC2011 . Another important feature missed by conventional
two dimensional semiconductor heterostructures, caused by edge reconstruction,
is the unusual edge charge and spin propagation HZi-Xiang2011 . In graphene
fluid, there is a parameter window in which the edge reconstruction can be
avoided HZi-Xiang2011 . In practice, this seem to have paved the way for the
experimental observations of charge and spin excitations at the edges of
fractional quantum Hall liquids HZi-Xiang2011 ; ZhangY2013 ; STakei2014 .

On the other hand, quantum interpretations of experimental measurements based
on quasiparticle theory revealed that edge spin (spin-flip) excitations are
more energetically favourable compare to edge charge (conserved-spin)
excitations and even spin waves in the bulk RRoldan2010 . Ability to tune such
spin excitations is very crucial in spintronics, which is witnessing an
increasing interests through coherent spin dynamical properties. For instance,
dissipationless spin superfluid transport realised on the platform of
spintronics device could be very attractive, in terms of large data storage
and information processing. Bulk spin superfluid transport has been reported
STakei2014 . The case of spin-polarised fractional quantum Hall states have
also been studied for spin reversed excitations MajunderD2014 ;
WurstbauerU2011 . However, the main hindering factor towards realising a
practical system will be the energy cost. Edge spin transport should be able
to overcome the obstacles.

In this work, we study collective excitations concerning spin-vortex dynamics
localised at Hall fluid edges. The fluid boundaries are modelled as smooth
edges to capture the true double boundary layer property of fractional quantum
Hall liquids. This is done in view of quantised Euler hydrodynamics theory for
vorticity. The hydrodynamic approach is motivated by the fact that QHF is a
manifestation of microscopic properties on the macroscopic scale. Hydrodynamic
modes at finite frequencies emerge at long wavelength limit due to slow motion
of quasiparticles. It is also based on its simplicity and ability to properly
capture quasiparticles interactions. The new term arising from such
interactions, called anomalous term WiegmannP2013b ; AbanovA2013  has non-
trivial consequence on transport properties, which is missed by microscopic
theories (HZi-Xiang2011, ; ZhangY2013, ). The hydrodynamic theory have been
used to correctly yield exact Laughlin states WiegmannP2013b  and bulk charge
excitations AbanovA2013 .

Microscopically, graphene has fourfold spin ($\uparrow,\downarrow$) and
pseudo-spin ($K,K^{\prime}$) degeneracy. Within the macroscopic picture, the
system possesses four fluid components though only one momentum flux is
measured. Each component belongs to the space
$\\{K,K^{\prime}\\}\otimes\\{\uparrow,\downarrow\\}$.

The remaining of the paper is organized as follows. In Section II, a solution
for vortex flow will be derived from quantized Euler hydrodynamic equation. We
will calculate the edge density and edge spin excitations for filling factors
within $0\leq\nu\leq 1$ together. The universal similarities and differences
in behaviour will be discussed in Section III. We will conclude in Section IV
highlighting possible applications of our results.

##  II Theoretical model

###  II.1 Euler hydrodynamics of charge vortices

The dynamics of incompressible vortex fluid is governed by Euler and
continuity equations,

$D_{t}\rho^{\alpha}=0\quad\mbox{ and }\quad D_{t}{\bf u}^{\alpha}+\nabla{\rm
p}^{\alpha}=0.$ (1)

Where the material derivative $D_{t}\equiv\partial_{t}+{\bf
u}\cdot{\bf\nabla}$ and ${\bf u}$ is macroscopic fluid velocity connected with
the microscopic electron velocity, ${\bf v}=v_{F}{\bf k}/k$. ${\rm p}$ is the
partial pressure per density and $\alpha$ is the fluid component index
(K$\uparrow$, K$\downarrow$, K’$\uparrow$, K’$\downarrow$). Taking the curl of
Eq. (1), we get

$D_{t}\omega^{\alpha}=0.$ (2)

Where the vorticity $\omega={\bf\nabla}\times{\bf u}$. The continuity equation
for the vortex, $D_{t}\rho^{\alpha}_{\rm v}=0$ also holds. The solution of Eq.
2 have been obtained by Helmholtz and reported in Flutcher1999 . In a disk
geometry having image vortices, the solution consists of point-like vortices
of the form,

${\rm u}^{\alpha}={\rm
i}\sum\frac{\Gamma^{\alpha}}{z^{\alpha}-z_{i}^{\alpha}(t)}+c.c+b.t,$ (3)

where $c.c$ is the complex conjugate, $z=x+{\rm i}y$ and $u=u_{x}-{\rm
i}u_{y}$. $b.t$ is the boundary term which includes drift velocity arising
from a confining potential at fluid boundary. We have assumed a flow in which
the strength of circulation, $\Gamma^{\alpha}_{i}$ ($=\Gamma^{\alpha}$) is
both minimal and chiral such that in the thermodynamic limit, rotation can be
compensated by the large number of vortices. This means, one can have Bohr-
Sommerfeld phase-space quantization of the circulations,
$m_{\text{v}}\Gamma^{\alpha}=2\pi\beta^{\alpha}\hbar$. Where $m_{\text{v}}$ is
the vortex inertia and $\beta^{\alpha}$ is an integer or fraction. The
equation of motion for vortices is such that

$\text{ v}_{i}^{\alpha}(t)={\rm i}\sum_{i\neq
j}\frac{\Gamma^{\alpha}}{z_{i}^{\alpha}(t)-z_{j}^{\alpha}(t)}+c.c+b.t.$ (4)

Writing v interms of u using the identities
$\pi\delta(r)\equiv\bar{\partial}(1/z)$ and $\sum_{i\neq
j}[2/(z-z_{i})(z_{i}-z_{j})]\equiv[\sum(1/(z-z_{i})]^{2}-\sum[1/(z-z_{i})]^{2}$
, it is straight forward to show that WiegmannP2013b  ${\rm v}={\rm u}+({\rm
i}\Gamma/2)\partial_{z}{\rm log}\rho_{\rm v}$. We can recast the expression as

$\rho_{\rm v}{\rm v}=\rho_{\rm v}{\rm u}+\frac{\Lambda_{\nu}}{4}\Delta{\rm
u}+\rho_{\rm v}{\rm v}_{D}.$ (5)

using the commutation relation $[{\rm u},\rho]={\rm i}(\hbar/m_{\rm
v})\nabla\rho$. Where ${\rm v}_{D}$ ($=e\ell_{B}^{2}\nabla V/\hbar$) is the
drift velocity of vortices due to the boundary terms and
$\Lambda_{\nu}=1-1/\beta+2m_{max}$. With $m_{max}$ the angular momentum of
vortices localised near the boundary.

###  II.2 Gilbert-Landau-Lifshitz equation of spin vortices

The equation of motion for spin-vortices having a unit magnitisation, m is
best decribed by the Gilbert-Landau-Lifshitz equation (GLLE) ThieleAA1973 ;
HuberDL1982

$\bar{D}_{t}{\bf m}_{i}+(\gamma/m_{0})\varepsilon_{ijk}{\bf
m}_{j}\partial_{{\bf m}_{k}}\mathcal{H}_{s}+\varepsilon_{ijk}{\bf
m}_{j}\tilde{D}_{t}{\bf m}_{k}=0.$ (6)

Where the material derivatives are $\bar{D}_{t}=\partial_{t}+{\bf
j}\cdot\nabla$ and $\tilde{D}_{t}=\alpha\partial_{t}+\beta{\bf j}\cdot\nabla$.
j is the spin-vortex current and $\mathcal{H}_{s}$ is the spin Hamiltonian
ThieleAA1973 . $m_{0}$ is the magnitude of the magnetic moment per area.
$\alpha$ is the Gilbert damping constant, which quantifies the macroscopic
response of spin angular momentum polarised along z-axis. It takes small
values for temperatures far below room temperature (HuberDL1982, ). $\gamma$
is the gyromagnetic ratio and $\beta$ is the non-adiabatic effect. The
magnetisation is projected out-of-plane, for this reason one can map the in-
plane coordinate ($(r,\phi)$) onto a 3D unit sphere ($\Theta(r),\Phi(\phi)$).
That is, ${\bf
m}=(sin\Theta(r)sin\Phi(\phi),sin\Theta(r)cos\Phi(\phi),cos\Theta(r))$. We are
concerned with the dynamics in the vicinity of of an equilibrium fluid that
may result in the deformation of the excitation spectrum. Equation (6) can be
transformed into non-Newtonian force of interactions between vortices
(HuberDL1982, )

$F_{\mu}=\varepsilon_{\mu\nu}N_{0}(U_{\nu}-J_{\nu})+\delta_{\mu\nu}D_{0}(\alpha
U_{\nu}-J_{\nu}).$ (7)

governing the drift of the spin-vortices. $U$ and $J$ are the spin velocity
and current, respectively. Assuming a vanishing dissipation dyadic, $D_{0}=0$,
the number $N_{0}$ is expressed as

$N_{0}=m_{0}\int{\rm
d}^{2}r\left[\nabla\Theta\times\nabla\Phi\right]sin\Theta.$ (8)

Equation (8) is solved using the constraint $\Phi=\Gamma\phi+\pi/2$ together
with the quantisation rule $[\Phi,\nabla\Theta]=\pm i\beta\hbar$. These yield,
$N_{0}=\pm 2\hbar\beta(m_{0}+\nabla\rho_{\rm v})$. Assuming for the moment
that $J<<U$, one obtains the the spin velocity in a plane $U=F/N_{0}$ or in
terms of energy of vortex-vortex interactions (HuberDL1982, ),
$W=-2\pi(m_{0}\beta^{2}\hbar^{2}J_{s}/\gamma)\sum_{i}\log|z-z_{i}|$.

$U=\pm\frac{m_{0}\beta J_{s}}{\gamma(m_{0}\pm\nabla\rho_{\rm v})}{\rm u}.$ (9)

For the intrinsic spin degree of freedom, one can redefine the charge and spin
densities as $2\rho_{c}=\rho_{\uparrow}+\rho_{\downarrow}$,
$2\rho_{s}=\rho_{\uparrow}-\rho_{\downarrow}$ and velocities as ${\rm u}={\rm
u}_{\uparrow}+{\rm u}_{\downarrow}$, ${\rm U}={\rm u}_{\uparrow}-{\rm
u}_{\downarrow}$. Equation (5) is transformed into the velocity field

${\rm v}={\rm
u}+\frac{\Lambda_{\nu}}{4(\rho_{c}^{2}-\rho_{s}^{2})}\left(\rho_{c}\Delta{\rm
u}-\rho_{s}\Delta{\rm U}\right)+{\rm v}_{D,c},$ (10)

which can be simplified further in terms of u using Eq. (9).

###  II.3 Edge velocity and model of boundary term

In real space and at the boundary, Eq. (3) assumes the form

${\rm
u}(r)\Big{|}_{\partial\Omega}=-\sum_{i}\frac{g\Gamma(r-r_{i})}{|r-r_{i}|^{2}}+\sum_{i}\frac{g\Gamma(r-r^{\prime}_{i})}{|r-r^{\prime}_{i}|^{2}}$

or in momentum space

${\rm
u}(q)\Big{|}_{\partial\Omega}=-\sum_{i}g\Gamma\left(\frac{r^{2}-r_{i}^{2}}{2\pi
r}\right)\int_{r=R}{\rm d}^{2}r\frac{e^{{\rm i}qrcos\theta}}{|r-r_{i}|^{2}}.$

Following standard elementary calculations, this simplifies to give

${\rm
u}(q)\Big{|}_{\partial\Omega}=-\sum_{i}g\Gamma\frac{\varepsilon_{\ell}\Gamma}{R}\left(\frac{r_{i}}{R}\right)^{|\ell|}{\rm
i}^{\ell}J_{\ell}(qR)e^{i\ell\theta_{q}}.$ (11)

$\theta_{q}$ is the angle between q and R introduced after performing the
$\theta$ integration.

The spin edge dispersion is calculated using the continuity equations defined
in the bulk and on the edge of the fluid, as

${\rm i}\omega\rho_{c,B}=\nabla{\rm P}\Theta(R-r),\,{\rm
i}\omega\rho_{c,E}={\rm P}\delta(R-r).$ (12)

Where the momentum flux is ${\rm P}=\rho_{c}{\rm v}+\rho_{s}{\rm V}$. ${\rm
V}$ is the spin-vortex velocity which differs from the charge velocity-vortex
${\rm v}$ by a proportionality constant. $\delta(x)$ and $\Theta(x)$ are delta
and step functions of $x$. Combining Eqs. (12) and using
$\rho_{c}=\rho_{c,B}+\rho_{c,E}$, we get

$\omega=[{\rm v}+\mathcal{S}_{z}{\rm V}](\delta(r-R)-{\rm
i}\Theta(r-R)\nabla[\log({\rm v}+\mathcal{S}_{z}{\rm V})+\log\rho_{c}]$ (13)

integrating over ${\rm d}^{2}r$ on both sides utilising
$\int\,\Theta(r-R)\nabla_{r}^{(n)}f(r)dr=-\nabla_{r}^{(n-1)}f(R)$ yields

$\omega R=\left[(1+\mu\mathcal{S}_{z})(2+\log\rho_{c})\right]{\rm
v}\Big{|}_{\partial\Omega}.$ (14)

Where the spin-charge competition effect $\mu=U/u=V/v=\xi/\bar{\xi}$ with
$\bar{\xi}=(\hbar/2\pi m_{\rm v})R_{0}^{2}$ and
$\mathcal{S}_{z}=\rho_{s}/\rho_{c}$. The velocity for vortices localised at
the boundary $\partial\Omega$ is computed from Eq. (10). In momentum space,
this can be express as

$\displaystyle{\rm v}(q)\Big{|}_{\partial\Omega}$ $\displaystyle=$
$\displaystyle\sum_{i,\ell}\bar{\xi}\beta
R\Big{[}\left(\frac{r_{i}}{R}\right)^{|\ell|}-\frac{q\Lambda_{\nu}}{2\rho_{c}R}\frac{(1-\mu\mathcal{S}_{z})}{(1-\mathcal{S}_{z}^{2})}\times$
(15)
$\displaystyle\left(\frac{r_{i}}{R}\right)^{|\ell-1|}\Big{]}K_{\ell}(qR)+{\rm
v}_{D}\delta_{\ell,1}J_{\ell}(qR).$

We now introduce the smooth edge model of FQHE fluid, which gives us
qualitative understanding of spin-vortex dynamics at the edges of the system.
To model the edges, we consider a sufficiently large but finite droplet
consisting of $N_{\rm v}$ vortices occupying a disk with edges at equilibrium
radius $R_{0}=\ell_{B}\sqrt{N/\pi\rho_{0}}$. The smoothness is caused by
clustering of vortices at the edges and subsequently performing small
cyclotron oscillations. It is possible to have dipole moment within the
boundary layer. Modulations of charges in the layer can be related to the
change in local radius, $\delta R$. For sufficiently smooth edges, this can be
expanded in Fourier series $\delta R(\varphi)=R_{0}\sum_{m}b_{m}Exp({\rm
i}m\varphi)+c.c$ for all possible angular momenta excitations. $m$ is an
integer. Where the dimensionless coefficient $b_{\ell}$ is a geometric
(deformations) parameter. It measures the smoothness (strength) of the edge
and varied within $0<b_{m}<1$. After some algebra and simplifications, the
velocity Eq. (15) together with Eq. (14) now takes the form

$\displaystyle\omega$ $\displaystyle=$
$\displaystyle\sum_{i,\ell}\bar{\xi}\beta\Big{[}\left(\frac{r_{i}}{R_{0}}\right)^{|\ell|}F_{1,n}-\frac{q\Lambda_{\nu}}{2\rho_{c}R}\frac{(1-\mu\mathcal{S}_{z})}{(1-\mathcal{S}_{z}^{2})}$
(16) $\displaystyle\times
F_{2,n}\left(\frac{r_{i}}{R_{0}}\right)^{|\ell-1|}\Big{]}K_{\ell}(qR_{0})$
$\displaystyle+{\rm v}_{D}(-1)^{n}F_{1,n}\delta_{\ell,1}J_{\ell}(qR_{0}).$

Where we have used the modified Bessel function relation $K_{\ell}(\Lambda
x)=\Lambda^{-\ell}\sum_{n}[(\Lambda^{2}-1)x/2]^{n}K_{\ell+n}(x)$ and the
Bessel function $J_{\ell}(\Lambda
x)=\Lambda^{-\ell}\sum_{n}[(1-\Lambda^{2})x/2]^{n}J_{\ell+n}(x)$ with
$\Lambda=\sum b_{m}Exp({\rm i}m\varphi)+c.c$. In Eq. (16), we have integrated
over $\varphi$ to get $F_{m,n}(b_{m})=\int{\rm
d}\varphi\,\Lambda^{m-1}(\Lambda^{2}+1)^{n}$.

So far, we have not provided physical interpretations to $\beta$. In the
context of fractional quantum Hall effect fluids, it was realised as the
Laughlin filling fractions for a single component fluid WiegmannP2013b . For
the case of graphene, which is a multicomponent fluid, we interpret
$\beta(=1/\nu_{G})$ as the Laughlin-Halperin filling factors GoerbigMO2012 .
Where $\nu_{G}$ is the graphene filling fraction calculated from
$\nu_{G}=\sum_{i=1}^{4}\nu_{i}$. $i$ is the component index. The component
fractions are computed from GoerbigMO2012  $N_{\phi}=m_{\alpha}\rho_{\rm
v,\alpha}+\sum_{\beta\neq\alpha}n_{\alpha\beta}\rho_{\rm v,\beta}$. The intra
and inter component factors are well chosen to yield a particular $\nu_{G}$
value. By defining the intrinsic spin polarisation as
$\mathcal{S}_{z}=\nu_{\uparrow K}+\nu_{\uparrow K^{\prime}}-\nu_{\downarrow
K}-\nu_{\downarrow K}$, each $\nu_{G}$ will yield a particular value for the
$\mathcal{S}_{z}$. Previous equations equally hold for the pseudo-spin
polarisation also defined as $\tilde{\mathcal{S}}_{z}=\rho_{ps}/\rho_{c}$ in
terms of charge and $\tilde{\mathcal{S}}_{z}=\nu_{\uparrow K}+\nu_{\downarrow
K}-\nu_{\uparrow K^{\prime}}-\nu_{\downarrow K^{\prime}}$ in terms of
component index factors.

We are interested in simulating Eq. (16) for various values of $\nu_{G}$,
$\mathcal{S}_{z}$ ($\tilde{\mathcal{S}}_{z}$), $b_{m}$ and $\mu$.

##  III Results and Discussions

In Fig. 1 and Fig. 2, we observed the spin-vortex excitations for $\mu>1$,
$\mu<1$ and deformation parameters; $b_{1}=0.097$, $b_{4}=0.79$.

In Fig. 1 (Top) for $\mathcal{S}_{z}=(1/3,0,1/5,2/3)/2$ and
$\tilde{\mathcal{S}}_{z}=0$. When $\mu<1$, quasiparticle charge dynamics
completely masked the spin dynamics. What is observed is the one resembling
edge magnetoplasmons excitations. However, the excitation curves depicts
resonance peaks originating from the boundary layer edges. Except the strong
dependence on deformations, all filling factors have similar behavior. In Fig.
1 (Buttom) for the $\mu>1$, the spin degree of freedom is very responsive
making excitations appear unusual compared to reported observations ZhangY2013
. This can be seen in all the dispersions curves. Remarkably, an unusual
behavior of coherent spin flipping are observed. On one hand, both $\beta=5/3$
and $\beta=3$ indicate partial spin polarisations (spin flip up and down),
with respect to boundary edges. The $\beta=5/2$ is unexpectedly, remain
robust. However, the $\beta=3/2$ have complete reorientation of spins. On the
other hand, what it means is that, there is a fraction of the spins pointing
one direction at the outer edge and the other fraction pointing the opposite
direction. Also, all spins point up at one edge and down at the second edge.
The behavior allows the system to have pure spin Hall liquid behavior which is
very useful in spintronic devices for data storage and information processing.

In the pseudo-spin polarisation sector; $\mathcal{S}_{z}=0$ and
$\tilde{\mathcal{S}}_{z}=(-1/3,2/5,1/5,2/9)/2$. When $\mu>1$, excitations
similar to those of the Fig. 1 (Top) are obtained. This is shown in Fig. 2
(Top). However, an interesting and unique spin-vortex dynamics resembling the
case of Fig. 1 (Buttom) occurs. In Fig. 2 (Buttom), the spin-vortex flipping
mechanisms for the pseudo-spin polarisations are quite different from the case
of intrinsic spin polarisations. The robust state turn out to be the usual
Laughlin state, $\beta=3$ as oppose to $\beta=5/2$. In general, boundary
deformation causes more vortices to have their spins polarised. Less (more)
energy is required to create $\beta=3$ ($\beta=5/3$) excitations as indicated
by the large difference in the observations for $b_{1}=0.097$ and $b_{4}=0.79$
graphs.

Figure 1: Spin excitation dispertions of (Top) $\mu>1$ and (Top) $\mu<1$ for
spin, $\mathcal{S}_{z}$ polarisations. $\beta=3$ (blue), $\beta=5/3$ (green),
$\beta=5/2$ (brown), $\beta=3/2$ (red).

Figure 2: Spin excitation dispertions of (Top) $\mu>1$ and (Top) $\mu<1$ for
spin, $\tilde{\mathcal{S}}_{z}$ polarisations. $\beta=3$ (blue), $\beta=5/3$
(green), $\beta=5/2$ (brown), $\beta=3/2$ (red).

A three dimensional plot showing the effects of varying deformations on the
low-energy spin -vortex excitations is shown in Fig. 3. The pseudo-spin
polarisation corresponding to $\beta=3/2$ is used. It is immediately clear how
the deformation parameter influences the vortex excitations when the spin-
charge competition parameter is tune from right to left. Oscillations
gradually appears at relatively high deformations. This means that plasmonic
signals with constant amplitude are generated and the phenomenon is ideal for
device applications. Similarly, for a constant $\mu$ but varying polarisation
from pseudo-spin to spin polarisations, there is a total spin flip from up to
down at both boundaries. This is depicted in Fig. 4. Again, the
controllability of the spin degree of freedom can be utilised in spintronic
devices.

Figure 3: Behaviour of sEMP excitations at varying edge deformation
$b_{\ell}$, $\beta=3/2$ and $\mathcal{S}_{z}=2/3$. (Left) $\mu>1$ and (Right)
$\mu<1$.

Figure 4: Behaviour of sEMP excitations at varying edge deformation
$b_{\ell}$, $\beta=3/2$ and $\mu>1$. (Left) $\mathcal{S}_{z}=0$,
$\tilde{\mathcal{S}}_{z}=2/3$ and (Right) $\mathcal{S}_{z}=2/9$,
$\tilde{\mathcal{S}}_{z}=0$.

##  IV Conclusions

In conclusion, we have demonstrated detailed theoretical studies of spin-
vortex dynamics, using quantised Euler and Gilbert-Landau-Lifshitz equations.
Two excitation peaks show up depicting the double boundary character of FQHE.
In a physically relevant parameter space, some filling factors exhibit full
and partial spin flipping at fluid boundaries. An inconsequential robustness
is demonstrated by the $\beta=1/3$ and $\beta=5/2$ for pseudo-spin and spin
polarisations, respectively. On the other hand, the other filling fractions
show an unexpected evolution to partial or full polarisations. This is
achieved by tuning both deformation and spin-charge parameters.

A natural direction to extend this studies is the edge excitations of spin-
textured (Skyrmions) of the fractional quantum Hall fluid. A motivation was
already presented for the integer case ZhangY2013 .

###### Acknowledgements.

Rabiu will like to thank International Center for Theoretical Physics
(ICTP)-Italy for hospitality and provision of travel grants to conduct part of
the work at the center in Trieste.

## References

  * (1) Laughlin, R. B., ”Anomalous quantum Hall effect: an incompressible quantum fluid with fractionally charged excitations”, Physical Review Letters 50 (18) 1395, (1983).
  * (2) Pikalov, Alexandr A. and Dmitrii V. Fil, ”Graphene bilayer structures with superfluid magnetoexcitons”, Nanoscale Research Letters 7(1), pp.1-9 (2012).
  * (3) Lagoudakis, K. G., Wouters, M., Richard, M., Baas A., Carusotto, I., André, R., Dang, L.S. and Deveaud-Plédran, B., ”Quantized vortices in an exciton–polariton condensate”, Nature Physics 4(9), pp.706-710 (2008).
  * (4) Wiegmann, P., ”Nonlinear hydrodynamics and fractionally quantized solitons at the fractional quantum Hall edge”, Physical Review Letters 108(20), 206810 (2012).
  * (5) Zi-Xiang H., Bhatt,R. N., Xin W., and Yang, K., ”Realizing universal edge properties in graphene fractional quantum Hall liquids”, Physical review Letters 107(23), 236806 (2011).
  * (6) Rabiu, M., Mensah, S.Y. and Abukari, S.S, ”General Scattering Mechanism and Transport in Graphene”, Graphene 2(01), 49 (2013).
  * (7) Kayoung L., Seyoung K., Points, M. S., Beechem, T. E., Taisuke O. and Tutuc, E., ”Magnetotransport properties of quasi-free-standing epitaxial graphene bilayer on SiC: Evidence for Bernal stacking”, Nano Letters 11(9), 3624-3628 (2011).
  * (8) Oostinga, J.B., Benjamin S., Monica F. C. and Alberto F. M., ”Magnetotransport through graphene nanoribbons.” Physical Review B 81(19) 193408, (2010).
  * (9) Rabiu, M., Mensah, S.Y., and Abukari, S.S, ”Terahertz generation and amplification in graphene nanoribbons in multi-frequency electric fields.” Physica E: Low-dimensional Systems and Nanostructures 61 pp.90-94 (2014).
  * (10) Xu D., Ivan S., Fabian D., Adina L. and Andre, E.Y., ”Fractional quantum Hall effect and insulating phase of Dirac electrons in graphene”, Nature 462(7270), pp.192-195 (2009).
  * (11) Bolotin, K.I., Ghahari, F., Shulman, F.D., Stormer, H.L. and Kim, P., ”Observation of the fractional quantum Hall effect in graphene”, Nature 462(7270), pp.196-199 (2009).
  * (12) Dean, C. R., Young, A. F., Cadden-Zimansky, P., Wang , L., Ren, H., Watanabe, K., Taniguchi, T., Kim, P., Hone, J. and Shepard, K.L., ”Multicomponent fractional quantum Hall effect in graphene”, Nature Physics 7(9), pp.693-696 (2011).
  * (13) Zhang, Y. and Kun Y., ”Edge spin excitations and reconstructions of integer quantum Hall liquids”, Physical Review B 87(12), 125140 (2013).
  * (14) Takei, S., Halperin, B.I., Yacoby, A. and Tserkovnyak, Y., ”Superfluid spin transport through antiferromagnetic insulators.”Physical Review B 90(9), 094408 (2014).
  * (15) Roldan, R., J-N. Fuchs, and M. O. Goerbig. ”Spin-flip excitations, spin waves, and magnetoexcitons in graphene Landau levels at integer filling factors.” Physical Review B 82(20), 205418 (2010).
  * (16) Majumder, D., and Sudhansu S. M., ”Neutral collective modes in spin-polarized fractional quantum Hall states at filling factors 1/3, 2/5, 3/7, and 4/9”. Physical Review B 90(15), (2014) 155310.
  * (17) Wurstbauer, U, Majumder, D., Mandal, S. S., Dujovne, I., Rhone, T. D., Dennis, B. S., Rigosi, A. F., Jain, J. K., Pinczuk, A., West, K. W., and Pfeiffer, L. N., ”Observation of Nonconventional Spin Waves in Composite-Fermion Ferromagnets.”Physical Review Letters 107(6), 066804 (2011).
  * (18) Wiegmann, P. B. ”Hydrodynamics of Euler incompressible fluid and the fractional quantum Hall effect.” Physical Review B 88(24), 241305 (2013).
  * (19) Abanov, Alexander G., ”On the effective hydrodynamics of the fractional quantum Hall effect”, Journal of Physics A: Mathematical and Theoretical 46(29), 292001 (2013).
  * (20) Flucher, Martin. Vortex motion in two dimensional hydrodynamics, Birkhäuser Basel (1999).
  * (21) Thiele, A. A., ”Steady-state motion of magnetic domains”, Physical Review Letters 30(6), 230 (1973).
  * (22) Huber, D. L., ”Dynamics of spin vortices in two-dimensional planar magnets.” Physical Review B 26(7), 3758 (1982).
  * (23) Goerbig, M. O. and Regnault, N., ”Theoretical aspects of the fractional quantum Hall effect in graphene.” Physica Scripta 146, 014017 (2012).