# Learning from Imperfect Demonstrations
from Agents with Varying Dynamics

Zhangjie Cao∗1 and Dorsa Sadigh2 Manuscript received October 15, 2020; revised
January 26, 2021; accepted February 23, 2021.This paper was recommended for
publication by Editor Dana Kulic upon evaluation of the Associate Editor and
Reviewers’<EMAIL_ADDRESS>and<EMAIL_ADDRESS>The
authors are with the Department of Computer Science, Stanford University,
Stanford, CA 94305, USADigital Object Identifier (DOI): see top of this page.

###### Abstract

Imitation learning enables robots to learn from demonstrations. Previous
imitation learning algorithms usually assume access to optimal expert
demonstrations. However, in many real-world applications, this assumption is
limiting. Most collected demonstrations are not optimal or are produced by an
agent with slightly different dynamics. We therefore address the problem of
imitation learning when the demonstrations can be sub-optimal or be drawn from
agents with varying dynamics. We develop a metric composed of a feasibility
score and an optimality score to measure how useful a demonstration is for
imitation learning. The proposed score enables learning from more informative
demonstrations, and disregarding the less relevant demonstrations. Our
experiments on four environments in simulation and on a real robot show
improved learned policies with higher expected return.

###### Index Terms:

Imitation Learning, Learning from Demonstrations, Robot Learning

##  I Introduction

Today’s imitation learning algorithms often assume access to expert
demonstrations [1, 2, 3, 4]. However, this assumption is limiting in many
real-world environments. In practice, it is expensive to obtain large number
of expert demonstrations, but we usually have access to a plethora of
imperfect demonstrations — demonstrations that can range from random noise or
failures to expert or even optimal demonstrations, and demonstrations that are
collected from agents with different dynamics, such as different embodiments,
body schema, or joint or rigid body structures. This is even evident when
collecting human demonstrations on a robot as we often have access to more
suboptimal data from humans due to factors such as their bounded rationality
[5] or difficulty of controlling robots with different or high degrees of
freedom [6].

Prior work has studied learning from suboptimal demonstrations by assigning an
optimality score for each demonstration, and only relying on learning from the
optimal data [7, 8, 9, 10]. In this paper, we go beyond this assumption and
consider imperfect demonstrations that can also be collected from agents with
different dynamics, e.g., demonstrations on a robot with different number of
joints than our target agent. In this setting, simply considering optimality
can fail because some demonstrations assessed as optimal might not even be
feasible on the target robot. Our key insight is that we need to measure
_optimality_ along with _feasibility_ of demonstrated trajectories to
effectively learn policies for a target domain. This further requires us to go
beyond existing optimality measures, as an optimal demonstration in another
dynamics may not be optimal for our target agent.

Figure 1: Learning from Imperfect Demonstrations from Agents with Varying
Dynamics. The target agent is a 3-DoF robot arm, and the demonstrations can
be: optimal demonstrations in the same dynamics (_3-DoF optimal_), optimal
demonstrations from another dynamics (_7-DoF optimal_), or suboptimal
demonstrations in the same dynamics (_3-DoF suboptimal_). We develop a score
$w$ derived from a feasibility $w_{f}$ and an optimality $w_{o}$ to define a
distribution $p_{w}$ over state transitions in demonstrations. $p_{w}$ assigns
higher probability to desirable demonstrations to enable learning an optimal
policy.

We introduce an algorithm for learning from agents with varying dynamics. Our
algorithm leverages a new metric that combines a feasibility score and an
optimality score to assess the informativeness of the demonstrations. The
feasibility score measures the likelihood of a trajectory to be achieved by a
target agent. The optimality score measures optimality under different
dynamics as opposed to prior work that only assess optimality based on the
demonstrator’s dynamics. Leveraging our new metric, we can learn from
demonstrations that are both more likely to be feasible and have higher
expected return. The proposed score only changes how we sample from the
demonstrations, and our algorithm is agnostic to the choice of imitation
learning algorithm used after scoring the demonstrations.

The main contribution of the paper can be summarized as:

  * •

We propose a more realistic problem setting of learning from imperfect
demonstrations, where the demonstrations may not be optimal and can be drawn
from agents with different dynamics.

  * •

We propose a scoring function composed of a feasibility score and an
optimality score to measure how informative each imperfect demonstration is.
The feasibility score measures how likely it is for a state transition to be
achieved by the target agent, while the optimality score measures how optimal
a state transition is with respect to the expert behavior according to its
expected return.

  * •

We conduct experiments in four simulation environments and an obstacle
avoidance task on a robot. We show that our algorithm outperforms other
methods when learning from imperfect demonstrations with varying dynamics.

##  II Related work

Imitation Learning. The goal of imitation learning is to find a policy that
best imitates expert demonstrations. Behavior cloning (BC) directly learns the
policy from a sequence of state-action pairs [1]. However, BC can result in
compounding errors that is usually addressed by dataset aggregation [11] or
policy aggregation [12, 13]. On the other hand, Inverse reinforcement learning
(IRL) aims to first recover the reward function from the demonstrations and
then find a policy through reinforcement learning [14, 15, 2, 16]. Generative
Adversarial Imitation Learning (GAIL) was proposed for learning the expert
behavior by matching the occupancy measure between the policy and the
demonstrations [3]. GAIL has been extended to improve the divergence
measurement [17, 18], but these extensions rely on access to a sequence of
states and actions. When only state observations are available, imitation
learning first recovers the actions [19], or directly matches the state
distribution [20, 21, 22].

Learning from Demonstrations with Different dynamics. The above imitation
learning works assume the demonstrator shares the same MDP as the target
agent. However, in reality, the demonstrator may have a different dynamics
than the target agent, which introduces the correspondence problem between the
demonstrator and the target agent [23, 24]. Englert _et al._ address the
correspondence problem by aligning the state trajectory distributions of the
expert demonstrations and of the policy [25]. Calinon _et al._ model the
demonstrations as a Gaussian mixture model within a projected lower-
dimensional subspace, and reproduce the optimal behavior by optimizing a time-
dependent similarity measurement for different robot contexts [26]. Eppner _et
al._ learn a task description as the relation of objects and robot body parts
using a dynamic Bayesian network and reproduce the joint action by maximizing
the likelihood [27]. Liu _et al._ aim to address the different dynamics for
imitation learning by matching the state sequence distribution to learn a
state policy and recover the actions using inverse dynamics [28]. These works
need a correspondence between the demonstrations and the target agent, but
under different dynamics, there could exist situations that the demonstrations
are not even achievable by the target agent, and the correspondence may not
exist. Our feasibility metric is designed to address exactly this problem.
Also, our work unlike prior works does not assume access to optimal
demonstrations.

Learning from Imperfect Demonstrations. Previous learning from demonstration
works usually assume that the demonstrations are provided by an optimal agent
who is successful at achieving the task [29, 3, 11, 14, 16]. In practice, most
demonstrations may be sub-optimal or even failures due to reasons, such as the
difficulty of providing expert data on robots with high degrees of freedom, or
noisy demonstrations due to bounded rationality of humans [30, 31, 6, 32].
Recently, several works have addressed the problem of learning from imperfect
demonstrations including learning a confidence weight to model the optimality
of state-action pairs [7], or modeling the probability of optimal
demonstrations [10]. Zhu _et al._ utilize a sparse reward function to measure
the optimality of each demonstration [9]. Grollman _et al._ assume access to
only low-quality demonstrations, and treat the data as negative examples [8].
All of these works assume the same MDP between the provided demonstrations and
the target agent. This results in simplified optimality metrics based on the
demonstrator’s dynamics. However, when the dynamics of a demonstrator and the
target agent are different, the previous optimality measurements are not
applicable anymore. Here, we introduce a new optimality measure combined with
a feasibility metric to tackle the problem of learning from imperfect
demonstrations from agents with varying dynamics.

##  III Problem Statement

We consider the standard Markov decision process (MDP):
$\mathcal{M}=\langle\mathcal{S},\mathcal{A},p,\mathcal{R},\rho_{0},\gamma\rangle$,
where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space,
$p:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1]$ is the
transition probability, $\rho_{0}$ is the initial state distribution,
$\mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is the reward
function, and $\gamma$ is the discount factor.

A policy $\pi:\mathcal{S}\times\mathcal{A}\rightarrow[0,1]$ defines a
probability distribution over the space of actions in a given state. An
optimal policy $\pi^{*}$ maximizes the expected return
$\eta_{\pi}=\mathbb{E}_{\xi\sim\pi}\large[G_{t}(\xi)\large]=\mathbb{E}_{s_{0}\sim\rho_{0},\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}\mathcal{R}(s_{t},a_{t},s_{t+1})\right]$,
where $t$ indicates the time step.

We formalize the problem of learning the optimal policy $\pi^{*}$ for the MDP
$\mathcal{M}$ as an imitation learning problem: Given a set of demonstration
trajectories $\Xi=\\{\xi_{1},\dots,\xi_{D}\\}$, where each trajectory is a
sequence of state-action pairs
$\xi=\\{s_{0},a_{1},s_{1},a_{2},\dots,a_{N},s_{N}\\}$, we aim to learn a
policy $\pi^{*}$ that best matches the demonstration trajectories. Note that
we focus on the offline imitation setting as employed in [3, 7], where a set
of demonstrations are provided ahead of time instead of gradually incrementing
our dataset as in online imitation learning [33].

One of the core assumptions in prior works in imitation learning is that the
demonstrations are drawn from the expert policy of the MDP $\mathcal{M}$,
which is referred as the _target_ MDP [3, 16, 34]. Here, we are interested in
a variant of the imitation learning problem, where we do not make the strong
assumption of having access to _expert_ demonstrations. Instead, we consider a
setting, where the demonstrations can be drawn from either a _sub-optimal
policy_ or from a _different MDP, $\mathcal{M}^{d}$,_ that has a different
dynamics from the target MDP $\mathcal{M}$. We note that the transition
probability of an MDP is affected by changes both in the environment and in
the agent’s dynamics. The _demonstrator_ MDP, $\mathcal{M}^{d}$, differs from
the _target_ MDP, $\mathcal{M}$, only in the agent’s dynamics. For example, in
a driving environment, we might have demonstrations from a vehicle with
different vehicle dynamics that drive in the same environment. Unlike prior
imitation learning work, we can only rely on state-transitions as opposed to
state-action sequences since the two MDPs might not share the same transition
probabilities, and some actions in the demonstrator’s MDP might not even be
feasible in the target MDP. We thus assume the reward is only a function of
state transitions, i.e.,
$\mathcal{R}:\mathcal{S}\times\mathcal{S}\rightarrow\mathbb{R}$, and instead
of imitating the state-action trajectories, _we discard actions in the
demonstration trajectories and let the agent imitate the state transitions_ ,
i.e., $(s_{t},s_{t+1})$ pairs.

Challenges. The core challenge in learning from imperfect demonstrations from
agents with varying dynamics is assessing how beneficial a given trajectory is
for learning a policy for the target MDP. This requires measuring two main
characteristics of a given trajectory: 1) whether the trajectory is achievable
in the target MDP (_feasibility_), and 2) the expected return of the
trajectory in the target MDP (_optimality_).

We argue that learning a policy can only benefit from a trajectory if and only
if neither the feasibility nor the optimality are low for that given
trajectory. If a trajectory has low feasiblity, it is not achievable by the
target agent and thus the target agent cannot imitate the behavior. Similarly,
if a trajectory has low optimality, imitating it can induce a sub-optimal
policy leading to low expected return.

##  IV Learning from Imperfect Demonstrations from Agents with Varying
Dynamics

In this section, we first introduce our method that incorporates a scoring
mechanism for assessing feasibility and optimality, and then discuss our
algorithm.

###  IV-A Feasibility

Feasibility measures how likely it is for each state transition in the
demonstration to be generated by the target MDP. Previous learning from
imperfect demonstration works do not consider a measure of feasibility since
they often assume the demonstrator follows the same dynamics as the target MDP
[7, 10]. Formally, a state transition $(s_{t},s_{t+1})$ is _feasible_ if and
only if there exists an action $a_{t}\in\mathcal{A}$ in the target MDP
$\mathcal{M}$ satisfying $p(s_{t},a_{t},s_{t+1})>0$.

However, we cannot simply apply this definition to our demonstrations to
verify feasibility. First, the transition probability function is usually
unknown or difficult to learn in most realistic robotics tasks, and thus we
cannot directly use the function to check for the feasible actions. If we do
not use the transition probability function, but assume access to a simulator,
where we can sample the next state given a state and action – as is assumed by
prior works [3, 7, 16, 34] – we would still need to search over a large space
of possible actions to verify existence of an action $a_{t}$ for the state
transition $(s_{t},s_{t+1})$ to be successful in the target MDP.

We instead propose computing the most probable action using an inverse
dynamics model of the target MDP
$f_{\text{id}}:\mathcal{S}\times\mathcal{S}\rightarrow\mathcal{A}$, which
takes a state transition pair $(s_{t},s_{t+1})$ that could be generated by
either the target MDP or the demonstrator MDP as an input and outputs all the
actions $a_{t}$ that satisfy $p(s_{t},a_{t},s_{t+1})>0$. For every $\xi$ from
the set of demonstrations provided by the demonstrator MDP, we initialize the
agent at the state $s_{0}$, which is the initial state of $\xi$. We then make
the agent take an action $a^{\prime}_{t}$ generated by $f_{\text{id}}$ at each
time step to generate a new state trajectory
$\xi^{\prime}=\\{s^{\prime}_{0},s^{\prime}_{1},s^{\prime}_{2}\dots,s^{\prime}_{N}\\}$
that is achievable by the target MDP:

$\displaystyle s^{\prime}_{0}=s_{0},\quad
a^{\prime}_{t}=f_{\text{id}}(s^{\prime}_{t-1},s_{t}),t\geq 1$ (1)
$\displaystyle s^{\prime}_{t}\sim
p(s^{\prime}_{t-1},a^{\prime}_{t},s^{\prime}_{t}),t\geq 1$

Learning Inverse Dynamics. If we assume access to an accurate inverse dynamics
model $f_{\text{id}}$ – meaning that $f_{\text{id}}$ gives the correct action
for a feasible state transition and outputs “None” for an infeasible state
transition, we can detect the infeasible trajectories based on the “None”
output. However, such a binary feasibility metric cannot differentiate between
nearly feasible and totally infeasible trajectories, where the former is
useful for learning and the latter does not have much learning value. So
instead we aim to learn a $f_{\text{id}}$ that outputs the correct action if
the state transition is feasible while still outputting the closest action,
when the state transition is infeasible. We model $f_{\text{id}}$ as a neural
network, and train it on a randomly sampled set of trajectories
$\Xi_{f}=\\{\xi_{f}\\}$ from the target MDP. We emphasize that similar to
prior imitation learning works [3, 7, 16, 34], we assume access to a simulator
for the target MDP, where we can sample feasible but random trajectories. We
learn $f_{\text{id}}$ using a regression loss:

$\min_{f_{\text{id}}}\mathbb{E}_{(s_{t},a,s_{t+1})\sim\xi_{f},\xi_{f}\sim\Xi_{f}}L_{1;\text{smooth}}(f_{\text{id}}(s_{t},s_{t+1}),a).$
(2)

Computing the Feasibility Score. Here, the trained $f_{\text{id}}$ will output
similar actions if the input state transition is similar to a state transition
seen in the training data. With the learned $f_{\text{id}}$, for every
trajectory $\xi$ – even if it is infeasible – we can compute a corresponding
trajectory $\xi^{\prime}$. In addition, we can compute a distance metric
between the two trajectories $\xi$ and $\xi^{\prime}$, $F(\xi,\xi^{\prime})$,
where lower distances correspond to higher feasibility scores. We compute the
feasibility score by normalizing this distance within a range
$[d_{\text{min}},d_{\text{max}}]$:

$w_{f}(\xi)=\begin{cases}1&F(\xi,\xi^{\prime})<d_{\text{min}}\\\
1-\frac{F(\xi,\xi^{\prime})-d_{\text{min}}}{d_{\text{max}}-d_{\text{min}}}&d_{\text{min}}\leq
F(\xi,\xi^{\prime})\leq d_{\text{max}}\\\
0&F(\xi,\xi^{\prime})>d_{\text{max}}\\\ \end{cases}$ (3)

Here, $\xi^{\prime}$ is a corresponding trajectory to $\xi$ generated by the
inverse dynamics $f_{\text{id}}$. Further, we can choose $F(\xi,\xi^{\prime})$
to be any sequence metric between the two trajectories. In our experiments, we
computed the mean of the $l_{2}$ distance between each pair of states in $\xi$
and $\xi^{\prime}$. The proposed continuous feasibility score can measure
different degrees of feasibility and preserve nearly feasible trajectories
that can be useful for learning a policy for the target agent.

Discussion of Feasibility Score. Using our feasibility measurement, the more
infeasible transitions contained in $\xi$ are, the deviation between
$\xi^{\prime}$ and $\xi$ becomes larger, and hence it would be less likely
that the trajectory is drawn from the target MDP. A possible drawback of our
feasibility score is compounding errors from our inverse dynamics model
$f_{\text{id}}$. To address this problem, we normalize the distance between
the trajectories $F(\xi,\xi^{\prime})$ using $d_{\text{min}}$ and
$d_{\text{max}}$. Here, $d_{\text{min}}$ models the lowest compounding error
that could be achieved by a feasible trajectory, while $d_{\text{max}}$ is the
highest compounding error. Thus, our feasibility measure can assign non-zero
feasibility to any feasible trajectories even with compounding errors from the
inverse dynamics. We include the method to set the thresholds in Section II of
the supplementary materials.

###  IV-B Optimality

We would like to define an optimality score that not only measures the
instance reward received by the state transition but also the influence of a
given state transition on the future state transitions, e.g., whether the
given state transition leads to higher expected return trajectories. Previous
works evaluate optimality based on pre-defined measurements provided by the
demonstrator, such as confidence score [7] and probability of taking an
optimal action [10]. However, these scores are evaluated based on the
demonstrator’s dynamics.

Here, the demonstrator follows a different dynamics as the target agent, and
thus the set of state transitions achievable by the target agent may be
different from those achievable by the demonstrator. For example, consider a
navigation setting in Fig. 2(a), where we allow three possible dynamics: (1)
unlimited turns, (2) no turns are allowed, and (3) can only turn in
$\frac{\pi}{2}$ increments. We set the reward as driving to the orange goal as
fast as possible without colliding with the brown obstacle. Here, the red,
green and blue trajectories are the optimal trajectories achieved by the
dynamics of _unlimited turns_ , _no turns allowed_ , and _only turning in
$\frac{\pi}{2}$ increments_ respectively. Note that the trajectory with
_unlimited turns_ receives higher expected return than the other trajectories.
So a state transition that is optimal under one dynamics may be suboptimal or
even useless for another agent. Thus, we need an optimality measurement that
can handle the more general case of learning from demonstrations with varying
dynamics.

We compute the expected return received by each state trajectory
$\xi=\\{s_{1},s_{2},\dots,s_{N}\\}$:
$\eta_{\xi}=\sum_{t}\gamma^{t}\mathcal{R}(s_{t},s_{t+1})$ as an estimate for
how optimal a trajectory is. Since the reward function only depends on state
transitions, the expected return of a trajectory can be naturally generalized
across different dynamics. We emphasize that we do not assume access to the
reward function, and only assume access to the expected return of a trajectory
similar to prior work that leverage a given confidence measure for each
demonstration [7].

Effects of Initial States on Optimality. The expected return is general but it
still suffers from some problems. Is a trajectory with higher expected return
really more useful for the policy learning of the target agent? If the
trajectories are starting from the same initial state, this claim is correct,
because higher expected return means we have a more optimal path from this
initial state. However, when the trajectories start from different initial
states, the claim may not be correct. Imagine the case shown in Fig. 2(b),
where the car navigates to the goal from different initial states. If the car
is close to the goal, e.g., one step from the goal, the navigation seems
easier while if the car is far away from the goal, the navigation is more
difficult. The optimal trajectory for the close initial location receives much
higher reward than the optimal trajectory for the far one and thus is assigned
with a much higher optimality if we only use the expected return as our
measure of optimality. However, both trajectories (starting close or far) are
important for the target agent since they give the agent guidance on how to
navigate to the goal from different initial locations. In fact, we should
assign similar optimalilty to both trajectories. Therefore, naïve optimality
scores based on the expected return may negatively influence the diversity of
the demonstrations.

(a) Different Dynamics

(b) Different Initial States

Figure 2: Fig. (a) shows optimal trajectories of navigation to the orange goal
without colliding with the brown obstacle under different dynamics. The green
trajectory is for a car that can never turn and only go straight. The blue
trajectory is for a car that can only turn in $\frac{\pi}{2}$ increments. The
red trajectory is for a car that can turn in any direction. We can see the
optimal trajectories for different dynamics under the same reward can be
different. Fig. (b) shows the optimal trajectories from different initial
states with the same dynamics under the navigation task to reach the orange
goal as fast as possible. So the car receives positive reward for reaching the
goal but is punished with negative reward for every time step. The expected
returns received by the optimal trajectories are different for different
initial states.

The Optimality Score. To address this problem, we rectify the expected return
to make it conditioned on the initial states. Our key idea is that if
demonstration trajectories are from the same initial states, we should learn
from the one with higher expected return. Based on this idea, we define a
function $f_{\text{rec}}:\mathcal{S}\rightarrow\mathbb{R}$. The input is an
initial state $s_{0}$ and the output is the highest expected return of a
demonstration starting from $s_{0}$. We then define the optimality of a
trajectory $\xi$ as:

$w_{o}(\xi)=\exp\left(-\frac{\left(\eta_{\xi}-f_{\text{rec}}(s_{0})\right)^{2}}{2\sigma^{2}}\right).$
(4)

We compute the distance between the expected return of each trajectory
$\xi=\\{s_{0},s_{1},\dots,s_{N}\\}$ and the best expected return the
demonstration can achieve by starting from $s_{0}$, and this can determine the
optimality score assigned to a trajectory. However, the distance
$\eta_{\xi}-f_{\text{rec}}(s_{0})$ is smaller than 0 but the lower bound
depends on the reward function, which may have highly different scales from
the feasibility. So we normalize this score in the range of $[0,1]$. We use a
Gaussian function for normalization instead of the widely-used min-max
normalization [35], which can more effectively filter out extremely low return
trajectories.

Learning the Rectifying Function. The key to the designed optimality is
estimating $f_{\text{rec}}$ – the highest expected return for an initial
state. Given that we only have access to a small number of trajectories, where
the initial states and the expected returns are known, we need a set of
assumptions for generating training data to learn $f_{\text{rec}}$. For
example, if the initial states of two trajectories $\xi_{1}$ and $\xi_{2}$ are
very close, the policy learned from one trajectory can also perform on the
other and achieve similar expected return. So we may assume that
$f_{\text{rec}}$ of the two close initial states should be the same or at
least similar. A realistic assumption for $f_{\text{rec}}$ is the neighborhood
property: $\exists\delta>0,\forall\xi=\\{s_{0},s_{1},\dots,s_{N}\\}$, we set
the highest expected return for $s_{0}$ as:

$f_{\text{rec}}(s_{0})=\max_{\xi^{\prime}\in\Xi,\lvert
s^{\prime}_{0}-s_{0}\rvert<\delta,w_{f}(\xi)>0}\eta_{\xi^{\prime}},$ (5)

where $s^{\prime}_{0}$ is the initial state of $\xi^{\prime}$ and $\Xi$ is the
set of all demonstrations. We set the highest expected return for a initial
state as the highest expected return of neighboring initial states of feasible
trajectories ($w_{f}(\xi)>0$), because the rectify function will be influenced
by infeasible trajectories with high expected returns otherwise. Using this
rectifying function $f_{\text{rec}}$, we can compute the rectified optimality,
which induces more diverse optimal demonstrations.

(a) Reacher Clockwise

(b) Reacher Counter Clockwise

(c) Driving Slow

(d) Driving Fast

(e) Reacher Clockwise (f) Reacher Counter Clockwise

(g) Driving Slow (h) Driving Fast

Figure 3: (a-b) are the illustrations of Reacher environments with clockwise
and counter-clockwise dynamics. (c-d) are the illustrations of Driving
environments with low speed and high speed dynamics. (e,f,g,h) are the
expected return results on environments (a,b,c,d) respectively. One x-axis
unit is $10^{7}$ steps.

###  IV-C Algorithm

Using the defined notions of feasibility and optimality, we now can compute
the final score of how useful each state transition is for learning a policy
for the target agent. We compute the final measurement as the product of the
feasibility and optimality scores:

$w(\xi)=w_{f}(\xi)\times w_{o}(\xi).$ (6)

We then re-weigh each state transition in the demonstration by the score
$w(\xi)$. The weighting-based work for learning from imperfect demonstrations
often directly incorporate the weight in the imitation loss [7]. However, such
re-weighing can suffer from numerical issues, when the weight of some state
transitions are too large. Instead, for a more stable training, we define a
distribution $p_{w}$ over state transitions based on the score $w(\xi)$. For
every state transition in a demonstration from the dataset, we assign a weight
$w_{i}$ (for the $i$-th state transition) equal to the weight assigned to the
trajectory $w(\xi)$, i.e., $\forall i\quad w_{i}=w(\xi)$. We compute a
probability distribution over the state transitions as
$p_{w_{i}}=\frac{w_{i}}{\sum_{i}w_{i}}$. Using the sampling distribution
$p_{w}$, we can embed our method into any imitation learning algorithm to
enable it to learn from imperfect demonstrations from various dynamics.

Note that given a trajectory, we assign equal scores to each state-transition
within the trajectory, which fails to emphasize the most important
transitions. However, this is not a problem as the key transitions repeatedly
appear in high score trajectories, and everytime they appear, their sampling
probability $p_{w}$ accumulates. This ensures the key transitions are
considered. The detailed description of the algorithm is shown in Section I of
the supplementary materials.

##  V Experiment

We conduct experiments in five environments including three MuJoCo
environments: Reacher, Swimmer, and Ant, one driving, and a Robot Arm
environment, which are all kinematic without force interactions. However, our
approach is not specific to kinematics, and can also be applied to settings
with force interactions as long as we can interact with the environment to
generate trajectories. We include the implementation details, parameter
selection, and results on varied demonstration ratios in the supplementary
materials.

Source of Demonstrations. We collect imperfect demonstrations using a policy
trained with only $10$ epochs of reinforcement learning, i.e., the policy is
not converged. These generated trajectories range from random trajectories to
near-optimal trajectories. We also collect optimal demonstrations with an
optimal policy. For Reacher, Swimmer, and Ant environments, we train the
optimal policy by TRPO [36]. For Driving and Panda Robot Arm, it is difficult
for TRPO to learn an optimal policy, so we hand-craft an optimal policy.

Composition of Demonstrations. We follow the experiment setting of the state-
of-the-art weighting-based learning from imperfect demonstration work [7]. We
do not make extra assumptions on the quality of demonstrations, and consider
the setting, where demonstrations can range from random to fully optimal.
Therefore, we employ a mixture of perfect and imperfect demonstrations, where
some demonstrations are useful and informative, while others are not as useful
or informative. Specifically, we construct a mixture of three types of data:
demonstrations of the optimal policy of the target environment, demonstrations
of the sub-optimal policy of the target environment, and the demonstrations
from an agent with different dynamics.

In each experiment, we use a set of $1000$ demonstrations for each
environment. To achieve a non-trivial imitation learning problem, we select
ratios of each type of demonstration appropriately to ensure that the problem
cannot easily be solved by vanilla imitation learning. The specific
composition of demonstrations in each environment is discussed below.

Other Methods. We compare our proposed approach with the most relevant
learning from demonstration works: imitation learning (specifically we compare
with Generative Adversarial Imitation Learning (GAIL)) [3], imitation learning
from imperfect demonstration: 2IWIL and IC-GAIL [7], and imitation learning
from different dynamics: SAIL [28].

###  V-A Environments with Return Influenced by the Initial State

Reacher. We experiment with openAI gym’s reacher environment [37, 38], where
the end effector of the agent is supposed to reach a final location. We modify
the original reacher environment, where the agent only has one joint and one
link as shown in Fig. 3(a) and 3(b). In addition, we limit the final goal
location (the red ball) within a particular area, which is shown as the shaded
yellow area composing of two triangles, where the vertex angle of the triangle
at the center is $53.1^{\circ}(2\arctan(0.5))$. Here, we consider two possible
dynamics: (1) the agent can only rotate clockwise with its joint limit
reaching at most the red line shown in Fig. 3(a) and (2) the agent can only
rotate counter-clockwise with its joint limit reaching at most the red line
shown in Fig. 3(b). We use the shaded yellow area and the limit on rotation to
make the optimal trajectories for one dynamics to be quite different from the
other. For instance, for a goal location in the up triangle, the optimal
trajectories are quite different: for the first dynamics the optimal
trajectory is to stay still, while the optimal trajectory for the second
dynamics is to rotate counter-clockwise toward the goal. So it is difficult
for agents with different dynamics to learn from each other.

Driving. We consider a simple 2D driving environment. As shown in Fig. 3(c)
and 3(d), the blue car aims to drive to the goal while avoiding the white
obstacle. As introduced in Section IV-B, when different initial states induce
different highest expected returns, the optimality based on the raw expected
return may fail. To test the difficulty, we set the initial states to be in
the yellow areas and define the reward function (just for evaluation) as
$\mathcal{R}(s_{t},s_{t+1})=\alpha_{1}\mathbf{1}_{\text{goal}}[s_{t+1}]+\alpha_{2}\mathbf{1}_{\text{obstacle}}[s_{t+1}]+\alpha_{3}\mathbf{1}_{\text{out}}[s_{t+1}]+\alpha_{4}$,
where $\mathbf{1}_{\text{goal}}$ is an indicator of when the next state
$s_{t+1}$ reaches the goal, $\mathbf{1}_{\text{obstacle}}$ is an indicator of
when $s_{t+1}$ collides with the obstacle, and $\mathbf{1}_{\text{out}}$ is an
indicator of when $s_{t+1}$ moves out, and $\alpha_{i}$ are appropriate
weights for each indicator. Then the initial location closer to the goal has
higher optimal expected return. Here, we fix the car speed and only control
the steering of the car. As shown in Fig. 3(d) and Fig. 3(c), we create
different dynamics by setting a high and a low speed for the car.

Compostion of Demonstrations. The ratio of demonstrations from the optimal
policy of the target environment, the sub-optimal policy of the target
environment, and the optimal policy in a different environment is
$5\%,90\%,5\%$ respectively for the Reacher environment and $30\%,60\%,10\%$
respectively for the Driving environment.

(a) Swimmer BackDisabled

(b) Swimmer FrontDisabled

(c) Ant X-axis

(d) Ant Y-axis

(e) Swimmer BackDisabled (f) Swimmer FrontDisabled

(g) Ant X-axis (h) Ant Y-axis

Figure 4: (a-b) are the illustrations of Swimmer with disabled back joint and
disabled front joint. (c-d) are the illustrations of Ant environment with
dynamics x-axis aligned and y-axis aligned dynamics. (e,f,g,h) are the
expected return results on environments (a,b,c,d) respectively. One x-axis
unit is $10^{7}$ steps.

Results. Fig. 3 shows that our proposed approach outperforms 2IWIL, IC-GAIL,
and SAIL. 2IWIL and IC-GAIL focus on suboptimal demonstrations while SAIL
focuses on demonstrations with varying dynamics. When both kinds of
demonstrations are present, 2IWIL, IC-GAIL, and SAIL cannot learn a high
return policy from the demonstrations. Our method employs feasibility and
optimality to filter harmful demonstrations and enable successful imitation
learning.

We also perform ablation studies on our method. In the driving and Reacher
environments, since the expected return of the trajectory is influenced by the
initial state, the proposed $f_{\text{rec}}$ is needed. So the optimality
depends on the feasibility as shown in Eqn. (5), and we cannot remove
feasibility while preserving optimality. We demonstrate the necessity of
$f_{\text{rec}}$ by removing $f_{\text{rec}}$ and using a naïve optimality
purely based on the reward (Naïve). We observe that our method outperforms
Naïve with a large margin in both environments. We then remove the optimality
and only use the feasibility score. Our method outperforms Feasibility in both
Reacher and Driving environments, indicating the importance of optimality to
filter sub-optimal demonstrations. Note that Naïve performs similar to
Feasibility. This is because in the environments where the expected return is
influenced by the initial state, directly using expected return as the
optimality tends to select initial states inducing high expected returns but
ignore other useful initial states. This can influence the diversity of the
demonstrations and make the policy fail on low expected return initial states.

###  V-B Environments with Similar Return across Initial States

Swimmer. In the Swimmer environment, we have an agent with three links and two
joints. The goal for the swimmer is to move forward by rotating the joints. As
shown in Fig. 4(a) and Fig. 4(b), we create two different dynamics: (1) the
blue joint is disabled (BackDisabled) and (2) the red joint is disabled
(FrontDisabled), where disabled means the joint angle is fixed to $0$ (the two
links are in a line).

Ant. In the Ant environment, we have an ant with four legs where each leg
consists of two links and two joints. The goal for the ant is to move forward
in the x-axis direction as fast as possible. As shown in Fig. 4(c) and Fig.
4(d), we create different dynamics by setting the legs at different angles
relative to each other: (1) the four legs can only rotate within
$[0^{\circ},53^{\circ}]$ from the x-axis. and (2) the four legs can only
rotate within $[0^{\circ},53^{\circ}]$ from the y-axis.

Composition of Demonstrations. The Swimmer and the Ant environments are two
other interesting environments that we study. The ratio of demonstrations from
optimal policy of the target environment, the sub-optimal policy of the target
environment, and the optimal policy of a different environment is
$1\%,49.5\%,49.5\%$ respectively for the Swimmer environment and for the Ant
environment.

Results. In Fig. 4, we observe that the proposed method outperforms all the
other methods with large margin in the expected return while having similar
converging speed even on these complex tasks. 2IWIL significantly outperforms
other baseline methods. This is because the influence of the sub-optimal
demonstrations are larger than demonstrations from different dynamics. 2IWIL
tends to filter more sub-optimal demonstrations, and thus performs better.
However, 2IWIL still performs worse than our proposed method since our method
further filters infeasible demonstrations, and learns from more feasible
demonstrations from different dynamics.

The expected return of the trajectory is not influenced by the initial state
for the Swimmer and the Ant environments, so the optimality does not depend on
the feasibility. Therefore, our proposed optimality equals to using the raw
expected return as the optimality score. Here, we run an ablation study, where
we remove optimality and feasibility individually, denoted as Optimality and
Feasibility. We observe that our approach outperforms both Feasibility and
Optimality, which demonstrates that both feasibility and optimality are
important elements in selecting useful demonstrations.

###  V-C Robot Experiments

The Panda Robot Arm environment uses a $7$ Degrees of Freedom (DoF) Franka
Panda Robot arm [39]. We create two dynamics: (1) the original 7-DoF arm and
(2) a 3-DoF arm by fixing all the joints that rotate around the z-axis. As
shown in Fig. 5(a) and Fig. 5(b), we conduct experiments on a task, where we
want the robot arm to take an object from a starting configuration above the
table and move to the table without colliding with the obstacle. The 7-DoF arm
can have various paths shown in blue trajectories while the 3-DoF robot can
only move along the x-axis resulting in either one of the two red trajectories
to avoid the obstacle. So different dynamics can introduce different optimal
trajectories.

Composition of Demonstrations. The ratio of demonstrations from optimal policy
of the target environment, the suboptimal policy of the target environment,
and the optimal policy of a different environment is $1\%,49.5\%,49.5\%$.

Results. In the robot environment the initial state does not influence the
highest expected return much, so we compare the proposed method with
Feasibility and Optimality similar to the Swimmer and Ant environments. We
conduct experiments both on a simulated panda robot arm with pybullet [40] and
a real robot arm. In Fig. 5, we observe that the proposed method outperforms
all the baselines with a large margin. Our method outperforms the variants
with only optimality or only feasibility considered individually, which
demonstrates the necessity of both feasibility and optimality in realistic
environments.

(a) Simulated Robot Arm

(b) Real Robot Arm

(c) Simulated Robot Arm

(d) Real Robot Arm

Figure 5: (a-b) are the illustrations of a simulated robot arm and a real
robot arm, where the blue curves are demonstrated by the 7-DoF arm, while the
3-DoF arm can only demonstrate the red curves. (c-d) are the corresponding
expected returns. One x-axis unit is $10^{5}$ steps.

##  VI Conclusion

Summary. In this paper, we propose an algorithm to learn from imperfect
demonstrations from agents with different dynamics, where the demonstrations
can be drawn from non-expert policies or from agents with different dynamics.
We design a feasibility score and an optimality score to select demonstrations
that are useful for the target agent. We show that the policy learned from the
selected demonstrations outperforms other imitation learning methods on
various environments and different composition of demonstrations.

Limitations and Future Work. Although our work relaxes stringent assumptions
placed on imitation learning algorithms, it is also limited in a few ways: To
compute feasibility, our method uses the inverse dynamics function and the
environment for trajectory sampling, which might not be accessible in some
settings. To compute the rectify function $f_{\text{rec}}$, we need the
neighborhood property, which may also be violated in particular environments.
For example, in navigation, if two nearby locations are separated by an
obstacle, their navigation route can differ significantly and the highest
expected return might also be quite different. In the future, we plan to
address these challenges by incorporating structures about the problem that
can help us better learn the inverse dynamics or rectify functions that do not
rely on the neighborhood property.

##  VII Acknowledgements

We would like to thank FLI grant RFP2-000 and NSF Award Number 1849952 for
their support.

## References

  * [1] M. Bain and C. Sammut, “A framework for behavioural cloning.” in _Machine Intelligence 15_ , 1995.
  * [2] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey, “Maximum entropy inverse reinforcement learning.” in _AAAI_ , 2008.
  * [3] J. Ho and S. Ermon, “Generative adversarial imitation learning,” in _NeurIPS_ , 2016.
  * [4] Z. Cao, E. Bıyık, W. Z. Wang, A. Raventos, A. Gaidon, G. Rosman, and D. Sadigh, “Reinforcement learning based control of imitative policies for near-accident driving,” _arXiv:2007.00178_ , 2020.
  * [5] H. A. Simon, _Models of bounded rationality: Empirically grounded economic reason_. MIT press, 1997.
  * [6] D. P. Losey, K. Srinivasan, A. Mandlekar, A. Garg, and D. Sadigh, “Controlling assistive robots with learned latent actions,” in _ICRA_ , 2020.
  * [7] Y.-H. Wu, N. Charoenphakdee, H. Bao, V. Tangkaratt, and M. Sugiyama, “Imitation learning from imperfect demonstration,” in _ICML_ , 2019.
  * [8] D. H. Grollman and A. Billard, “Donut as i do: Learning from failed demonstrations,” in _ICRA_ , 2011.
  * [9] Z. Zhu, K. Lin, B. Dai, and J. Zhou, “Learning sparse rewarded tasks from sub-optimal demonstrations,” _arXiv:2004.00530_ , 2020.
  * [10] V. Tangkaratt, B. Han, M. E. Khan, and M. Sugiyama, “Variational imitation learning with diverse-quality demonstrations,” in _ICML_ , 2020.
  * [11] S. Ross, G. Gordon, and D. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” in _AISTATS_ , 2011\.
  * [12] H. Daumé, J. Langford, and D. Marcu, “Search-based structured prediction,” _Machine learning_ , 2009.
  * [13] S. Ross and D. Bagnell, “Efficient reductions for imitation learning,” in _AISTATS_ , 2010.
  * [14] P. Abbeel and A. Y. Ng, “Apprenticeship learning via inverse reinforcement learning,” in _ICML_ , 2004.
  * [15] A. Y. Ng, S. J. Russell, _et al._ , “Algorithms for inverse reinforcement learning.” in _ICML_ , 2000.
  * [16] J. Fu, K. Luo, and S. Levine, “Learning robust rewards with adverserial inverse reinforcement learning,” in _ICLR_ , 2018.
  * [17] H. Xiao, M. Herman, J. Wagner, S. Ziesche, J. Etesami, and T. H. Linh, “Wasserstein adversarial imitation learning,” _arXiv preprint arXiv:1906.08113_ , 2019.
  * [18] L. Ke, M. Barnes, W. Sun, G. Lee, S. Choudhury, and S. Srinivasa, “Imitation learning as $f$-divergence minimization,” in _Workshop on the Algorithmic Foundations of Robotics_ , 2020.
  * [19] F. Torabi, G. Warnell, and P. Stone, “Behavioral cloning from observation,” in _IJCAI_ , 2018.
  * [20] Y. Schroecker and C. L. Isbell, “State aware imitation learning,” in _NeurIPS_ , 2017.
  * [21] F. Torabi, G. Warnell, and P. Stone, “Generative adversarial imitation from observation,” _Imitation, Intent, and Interaction (I3) Workshop at ICML_ , 2019.
  * [22] W. Sun, A. Vemula, B. Boots, and D. Bagnell, “Provably efficient imitation learning from observation alone,” in _ICML_ , 2019.
  * [23] C. L. Nehaniv, K. Dautenhahn, _et al._ , “The correspondence problem,” _Imitation in animals and artifacts_ , 2002.
  * [24] B. D. Argall, S. Chernova, M. Veloso, and B. Browning, “A survey of robot learning from demonstration,” _Robotics and autonomous systems_ , 2009.
  * [25] P. Englert, A. Paraschos, J. Peters, and M. P. Deisenroth, “Addressing the correspondence problem by model-based imitation learning,” in _ICRA Workshop on Autonomous Learning_ , 2013.
  * [26] S. Calinon, F. Guenter, and A. Billard, “On learning, representing, and generalizing a task in a humanoid robot,” _TSMC_ , 2007.
  * [27] C. Eppner, J. Sturm, M. Bennewitz, C. Stachniss, and W. Burgard, “Imitation learning with generalized task descriptions,” in _ICRA_ , 2009.
  * [28] F. Liu, Z. Ling, T. Mu, and H. Su, “State alignment-based imitation learning,” in _ICLR_ , 2019.
  * [29] F. Codevilla, M. Miiller, A. López, V. Koltun, and A. Dosovitskiy, “End-to-end driving via conditional imitation learning,” in _ICRA_ , 2018\.
  * [30] C. Basu, Q. Yang, D. Hungerman, M. Singhal, and A. D. Dragan, “Do you want your autonomous car to drive like you?” in _HRI_ , 2017.
  * [31] B. Akgun, M. Cakmak, K. Jiang, and A. L. Thomaz, “Keyframe-based learning from demonstration,” _IJSR_ , 2012.
  * [32] M. Kwon, E. Biyik, A. Talati, K. Bhasin, D. P. Losey, and D. Sadigh, “When humans aren’t optimal: Robots that collaborate with risk-aware humans,” in _HRI_ , 2020.
  * [33] J. Lee, C.-A. Cheng, K. Goldberg, and B. Boots, “Continuous online learning and new insights to online imitation learning,” _arXiv preprint arXiv:1912.01261_ , 2019.
  * [34] A. H. Qureshi, B. Boots, and M. C. Yip, “Adversarial imitation via variational inverse reinforcement learning,” in _ICLR_ , 2019.
  * [35] J. Zhang, Z. Ding, W. Li, and P. Ogunbona, “Importance weighted adversarial nets for partial domain adaptation,” in _CVPR_ , 2018.
  * [36] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in _ICML_ , 2015.
  * [37] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” _arXiv:1606.01540_ , 2016.
  * [38] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in _IROS_ , 2012.
  * [39] F. E. GmbH. Introduction. [Online]. Available: https://www.franka.de/
  * [40] E. Coumans, Y. Bai, and J. Hsu, “Pybullet physics engine,” 2018.
  * [41] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.

##  Appendix A Algorithm

In Algorithm 1, we go through the steps of the algorithm for learning from
imperfect demonstrations from agents with varying dynamics.

Input: Demonstrations $\Xi$, random trajectories from the target agent
$\Xi_{f}$

Compute the feasibility $w_{f}$ $\forall\xi$ in $\Xi$ using Eqn. (3)

Compute the optimality $\forall\xi$ in $\Xi$ using Eqn. (5)

Compute the state transition distribution $p_{w}$ with the probability
proportion to the score $w$ in Eqn. (7)

while  _not converging_ do

Train $\pi$ with an state-based imitation learning algorithm with state
transitions sampled from $p_{w}$;

end while

Output: Learned optimal policy $\pi^{*}$ for $\mathcal{M}$.

Algorithm 1 Algorithm

##  Appendix B Setting Threshold

We set two threshold values for the distance between trajectories. First,
$d_{\text{min}}$ sets the minimum distance between $\xi$ and $\xi^{\prime}$,
and any trajectory with distance smaller than $d_{\text{min}}$ will be
assigned feasibility of $1$. We set $d_{\text{min}}$ by taking the minimum in
the sampled feasible trajectories $\Xi_{f}$:
$d_{\text{min}}=\min_{\xi_{f}\in\Xi_{f}}F(\xi_{f},\xi^{\prime}_{f})$.

Second, $d_{\text{max}}$ sets the maximum distance between $\xi$ and
$\xi^{\prime}$ that can be tolerated. There are two clear approaches for
computing $d_{\text{max}}$: the maximum distance in the demonstration
trajectories $\max_{\xi\in\Xi}F(\xi,\xi^{\prime})$ and the maximum distance in
the sampled feasible trajectories
$\max_{\xi_{f}\in\Xi_{f}}F(\xi_{f},\xi^{\prime}_{f})$. Here, $\xi^{\prime}$
and $\xi^{\prime}_{f}$ are computed using the inverse dynamics as in Eqn. (1)
in the main text. The former can be too large and can assign positive
feasibility to harmful and completely infeasible trajectories at times. On the
other hand, the latter may filter nearly feasible trajectories, which are
useful for imitation learning. Instead of these two extreme measures, we
propose estimating $d_{\text{max}}$ from the environment. For every trajectory
$\xi_{f}$ in $\Xi_{f}$, we derive a trajectory
$\xi^{\prime\prime}_{f}=\\{s^{\prime\prime}_{0},s^{\prime\prime}_{1},\cdots,s^{\prime\prime}_{N}\\}$:

$\displaystyle s^{\prime\prime}_{0}=s_{0},\quad
a^{\prime\prime}_{t}=f_{\text{id}}(s^{\prime\prime}_{t-1},s_{t}),t\geq 1$ (7)
$\displaystyle s^{\text{sample}}_{t}\sim
p(s^{\prime\prime}_{t-1},a^{\prime\prime}_{t},s^{\prime\prime}_{t}),t\geq 1$
$\displaystyle s^{\prime\prime}_{t}=s^{\text{sample}}_{t}+\Delta s.$

We perturb the state at each step with a $\Delta s$, where $\lvert\Delta
s\rvert\leq\delta$, and then we set $d_{\text{max}}$ as the max distance
$\max_{\xi_{f}\in\Xi_{f}}F(\xi_{f},\xi^{\prime\prime}_{f})$. We note that
$\xi^{\prime}_{f}$ is the corresponding trajectory for $\xi$ while
$\xi^{\prime\prime}_{f}$ is a trajectory with perturbation $\delta s$ over the
states of $\xi$. This modeling allows us to know that a distance smaller than
$d_{\text{max}}$ means the trajectory is within $\delta$ state averaged
distance from a feasible trajectory. We can control $\delta$ in different
environments to decide if a trajectory should be assigned positive or zero
feasibility weights.

TABLE I: Expected return after convergence for all the environments.

Methods | Environments
---|---
Reacher | Driving | Swimmer | Ant | Panda
Clockwise | Counter Clockwise | Slow | Fast | BackDisabled | FrontDisabled | X-axis | Y-axis | Simulation | Real
None | -122.51$\pm$5.98 | -127.68$\pm$9.46 | -6218$\pm$179 | -6232$\pm$69 | 19.57$\pm$6.98 | 11.89$\pm$2.64 | -48$\pm$8 | 1152$\pm$85 | -1415$\pm$9 | -1177$\pm$1156
SAIL | -129.52$\pm$6.83 | -129.43$\pm$8.12 | -5743$\pm$1043 | -8976$\pm$1772 | 12.51$\pm$0.43 | -2.63$\pm$1.11 | -255$\pm$5 | 1239$\pm$92 | -1645$\pm$208 | -1498$\pm$78
IC-GAIL | -121.48$\pm$14.56 | -126.95$\pm$10.26 | -8189$\pm$1696 | -7202$\pm$403 | 25.69$\pm$8.39 | 14.56$\pm$5.79 | -46$\pm$6 | 1532$\pm$133 | -2241$\pm$273 | -876$\pm$503
2IWIL | -134.68$\pm$2.78 | -118.92$\pm$5.54 | -5189$\pm$1754 | -5632$\pm$84 | 74.12$\pm$9.14 | 123.89$\pm$5.12 | -247$\pm$241 | 1449$\pm$297 | -1148$\pm$22 | -915$\pm$96
Feasibility | -122.03$\pm$5.46 | -121.32$\pm$7.53 | -7364$\pm$455 | -7188$\pm$845 | 23.89$\pm$4.77 | 0.32$\pm$6.32 | -42$\pm$10 | 1621$\pm$94 | -1412$\pm$6 | -1203$\pm$412
Naïve | -111.53$\pm$3.85 | -118.96$\pm$10.89 | -4812$\pm$765 | -5913$\pm$1924 | - | - | - | - | - | -
Optimality | - | - | - | - | 63.12$\pm$5.01 | 150.32$\pm$3.45 | 689$\pm$648 | 1389$\pm$356 | -1124$\pm$85 | -902$\pm$23
Ours |  -94.32$\pm$9.67 |  -90.89$\pm$4.02 |  -654$\pm$103 |  -1429$\pm$672 |  122.44$\pm$13.42 |  167.43$\pm$6.23 |  1979$\pm$83 |  1709$\pm$412 |  -979$\pm$89 |  -500$\pm$201

##  Appendix C Time Complexity

Compared to vanilla imitation learning, our approach introduces additional
computation costs in computing $w$ and $p_{w}$. The computational cost of
vanilla imitation learning is bounded as shown in [3, 19, 21]. So here we only
need to demonstrate that computing $w$ and $p_{w}$ is tractable.

We let the size of the demonstration set $\Xi$ be $D$, the size of the
feasible trajectory set $\Xi_{f}$ be $D_{f}$, and the largest number of steps
of a trajectory in $\Xi$ and $\Xi_{f}$ be $N$. We let $\text{dim}(s)$ be the
dimension of the state. The average time to compute the inverse dynamics
function is $T_{\text{id}}$.

For the feasibility, we first derive the feasible trajectory set $\Xi_{f}$,
and the time complexity is $O(D_{f}N)$. Then we compute the corresponding
trajectory $\xi^{\prime}$ for each trajectory $\xi$ in $\Xi$ and the time
complexity is $O(DNT_{\text{id}})$. With the corresponding trajectory, we then
compute $F(\xi,\xi^{\prime})$ as shown in Equation (3) in the main text for
all trajectories in $\Xi$, and the time complexity is $O(DN\text{dim}(s))$ for
iterating over all the states in the trajectory. Then we assess the time to
compute the two thresholds $d_{\text{max}}$ and $d_{\text{min}}$.
$d_{\text{max}}$ needs simulating the trajectories in $\Xi_{f}$ with
perturbation $\delta s$ and computing $F(\xi_{f},\xi^{\prime\prime}_{f})$, so
the time to compute Equation (7) is $O(D_{f}N(T_{\text{id}}+\text{dim}(s)))$.
$d_{\text{min}}$ needs simulating $\xi^{\prime}_{f}$ with each trajectory
$\xi_{f}\in\Xi_{f}$ and computing $F(\xi_{f},\xi^{\prime}_{f})$ as shown in
Equation (1) of the main text, so the time is
$O(D_{f}N(T_{\text{id}}+\text{dim}(s)))$. Therefore, the total complexity time
is the sum of the above computations:
$O((D_{f}+D)N(T_{\text{id}}+\text{dim}(s)))$.

For the optimality, to estimate the function $f_{\text{rec}}$, we need to
compute the distance between the initial states of every pair of trajectories.
So the time complexity is $O(D^{2}\text{dim}(s))$. The time to compute $w_{o}$
is $O(D\text{dim}(s))$. As a result the total time complexity is
$O(D^{2}\text{dim}(s))$.

Therefore, the total time to compute the proposed $w$ is
$O(D^{2}\text{dim}(s)+(D_{f}+D)N(T_{\text{id}}+\text{dim}(s)))$. The time
complexity increases linearly with respect to the dimension of the state,
which depends on the degrees of freedom of the agent.

In practice, the time to compute the inverse dynamics is negligible. We would
like to emphasize that most imitation learning algorithms traverse over the
demonstrations multiple times, and the time complexity of traversing over
trajectories once is $O(DN\text{dim}(s))$. Therefore, compared to the time for
running imitation learning algorithms, the time cost
$O(DN(T_{\text{id}}+\text{dim}(s)))$ is negligible.

So the main time burden is on $O(D^{2}\text{dim}(s))$ and
$O(D_{f}N(T_{\text{id}}+\text{dim}(s)))$. In practice, if the size of the
demonstration set $D$ is too large, when estimating $f_{\text{rec}}$, we can
sample a subset of demonstrations to reduce the time complexity while still
obtaining an relatively accurate estimation of $f_{\text{rec}}$. $D_{f}N$
means how many feasible trajectories we collect in $\Xi_{f}$, which is decided
by the environment. More complex environments need more feasible trajectories
to cover the trajectory space. To reduce $D_{f}N$, we can first learn an
initial policy by direct imitation learning from all the demonstrations and
use that policy to collect feasible trajectories. These trajectories are
likely to be more close to the optimal trajectories and we can avoid wasting
time to explore the trajectories that are far from optimal trajectories.

##  Appendix D Implementation Details

For the network architecture, we use a three-layer fully-connected network
with hidden state size $100$ and tanh activation function for the policy
network, the value network and the discriminator (the value network and the
discriminator are only needed for GAIL-based methods). For the inverse
dynamics model, we use an eight-layer fully-connected network with ReLU
activation. For the behavior-cloning-based methods, we minimize the smooth l1
loss between the predicted action and the ground truth action. For the GAIL-
based methods, we employ TRPO-based GAIL. We use an Adam optimizer [41], fix
the learning rate for the policy network and the discriminator as $10^{-3}$,
and fix the value network as $3\times 10^{-4}$ for all the environments. For
the two hyper-parameters $\Delta s$ and $\sigma$, we tune the parameters for
one task in each environment while fixed the parameters for the other tasks in
the environment. We set $\Delta s=0.001$ for $\sigma=50$ for Reacher, $\Delta
s=0.001$ for $\sigma=100$ for Driving, $\Delta s=0.0005$ for $\sigma=40$ for
Swimmer, and $\Delta s=0.001$ for $\sigma=500$ for Ant and We set $\Delta
s=0.1$ for $\sigma=100$ for Robot Arm.

##  Appendix E Tabularised results

To more clearly compare the expected return of different methods, we show the
expected return for each methods and environments after convergence in Table
I. We can observe that the proposed approach achieves the highest expected
return after convergence.

Figure 6: (left) Expected return with respect to different $\log(\Delta s)$
test on the BackDisabled Swimmer environment; (right) Expected return with
respect to different $\log(\sigma)$ test on the Slow Driving environment.

##  Appendix F Parameter Sensitivity

We have two key hyper-parameters to set in our method: the $\Delta s$ in Eqn.
(4) and the $\sigma$ in Eqn. (5). We note that the two hyper-parameters are
necessary to achieve the optimal performance of the method but the algorithm
can work stably in a reasonable range for these hyper-parameters. We run the
experiments on the BackDisabled Swimmer environment with respect to different
$\log(\Delta s)$ and on the Slow Driving environment with respect to different
$\log(\sigma)$. We use these two environments because Slow Driving uses our
full version of optimality with the rectify function while Swimmer relies more
on the feasibility. The solid red line is the performance of the proposed
method while the dashed black line is the expected return of the optimal
demonstrations. Fig. 6 shows that there exist a wide range of values for both
$\Delta s$ and $\sigma$ to lead to stable results and high expected return.
However, when $\Delta s$ is too large, $d_{\text{max}}$ can become larger and
lead to positive feasibility be assigned to infeasible demonstrations. On a
similar note, if $\sigma$ is too large or too small, the performance drops
severely. Small values of $\sigma$ can cause a large difference in optimality
for similar rectified rewards. This can overweigh some optimal demonstrations
over others, while too large values of $\sigma$ can lead to the optimality
failing to discriminate between optimal demonstrations and sub-optimal ones.

(a) Vary Perfect Demonstrations

(b) Vary Imperfect Demonstrations

Figure 7: Ratio of Perfect, Infeasible, and Suboptimal Demonstrations. In (a),
we change the ratio of perfect demonstrations, while keeping the ratio of
infeasible demonstrations equal to the ratio of suboptimal demonstrations. In
(b), we fix the ratio of perfect demonstrations, while changing the ratio of
infeasible demonstrations and suboptimal demonstrations.

##  Appendix G Varying Composition of Demonstrations

Effects of the Ratio of Perfect Demonstrations. Here we conduct an experiment
specifically on the Swimmer environment, to demonstrate the effects of
gradually increasing the portion of perfect demonstrations – demonstrations
provided by an optimal policy on the target MDP – while keeping the relative
size of other types of demonstrations constant. We observe that our method
consistently performs well with different ratios of perfect demonstrations as
opposed to all the other methods where performance significantly drops with
small ratio of perfect demonstrations (see Fig. 7(a)). When the ratio of
perfect demonstrations becomes high enough, the suboptimal and infeasible
demonstrations have no influence on learning and even vanilla GAIL learns the
optimal policy.

Effects of the Relative Ratio of Demonstrations with Varying Dynamics and
Suboptimal Demonstrations. We conduct an experiment in the Swimmer
environment, where we vary the ratio of demonstrations with varying dynamics
and suboptimal demonstrations. We fix the portion of perfect demonstrations at
$1\%$ of all trajectories. In Fig. 7(b), we observe that our method
consistently outperforms other methods for all ratios of these suboptimal
demonstrations. We also observe that the performance of the other methods
including Optimality increases with increasing the ratio of imperfect
demonstrations. However, the performance of Feasibility decreases, which
demonstrates that our proposed optimality can handle under learning with sub-
optimal demonstrations and our proposed feasibility score can learn from
demonstrations from agents with varying dynamics.


# Translating Solitons to Flows by Powers of The Gaussian Curvature in
Riemannian Products

Ronaldo Freire de Lima  Departamento de Matemática - Universidade Federal do
Rio Grande do Norte<EMAIL_ADDRESS>
###### Abstract.

We consider translating solitons to flows by positive powers $\alpha$ of the
Gaussian curvature — called $K^{\alpha}$-flows — in Riemannian products
$M\times\mathbb{R}.$ We prove that, when $M$ is the Euclidean space
$\mathbb{R}^{n},$ the sphere $\mathbb{S}^{n},$ or one of the hyperbolic spaces
$\mathbb{H}_{\mathbb{F}}^{m},$ there exist complete rotational translating
solitons to $K^{\alpha}$-flow in $M\times\mathbb{R}$ for certain values of
$\alpha.$

###### Key words and phrases:

translating soliton – Gaussian curvature – Riemannian product.

###### 2010 Mathematics Subject Classification:

53B25 (primary), 53C42 (secondary).

##  1\. Introduction

Over the last decades, the subject of extrinsic curvature flows in Riemannian
manifolds has experienced a significant development. Along this time, special
attention has been given to mean curvature and Gaussian curvature flows in
Euclidean space, resulting in achievements such as the proof of short time
existence of solutions, and of their convergence (after rescaling) to round
spheres. (For a thorough account of these and other extrinsic flows, we refer
to the recent book [3].)

Likewise, flows by powers of the Gaussian curvature, which we call
$K^{\alpha}$-_flows_ , have been considered by many authors, most notably B.
Andrews (see, e.g., [2, 4, 5, 7, 9, 16, 20]). Among other reasons, the
interest in $K^{\alpha}$-flows comes from the fact that they naturally relate
to a wide range of research fields, varying from image processing to affine
geometry and geometric analysis (cf. [1]).

In the context of extrinsic curvature flows in Euclidean space, there are
special hypersurfaces, called _translating solitons_ , which have the
distinguished property of moving by translation under such a flow. In fact,
the construction and classification of complete translating solitons
constitutes a major problem in this theory (see, e.g., [10, 18, 19, 20]). On
this regard, we point out that J. Urbas [20] established, for all
$\alpha\in(0,1/2],$ the existence of complete rotational translating solitons
to the $K^{\alpha}$-flow in Euclidean space.

More generally, translating solitons can be defined in Riemannian products
$M\times\mathbb{R},$ where $M$ is an arbitrary Riemannian manifold. In [17],
assuming $M$ complete with non-positive sectional curvature and rotationally
invariant metric, Lira and Martín obtained a one-parameter family of
rotational translating solitons to the mean curvature flow in
$M\times\mathbb{R}.$ Also, in [14, 15], results of the same nature were
obtained in the context of Lorentzian products. On the other hand, to our
knowledge, there are no works on translating solitons to $K^{\alpha}$-flows in
products $M\times\mathbb{R}.$

In this paper, we prove existence of complete rotational translating solitons
to the $K^{\alpha}$-flow in $M\times\mathbb{R}$ (for certain positive values
of $\alpha$) when $M$ is the Euclidean space $\mathbb{R}^{n},$ the sphere
$\mathbb{S}^{n},$ or one of the hyperbolic spaces
$\mathbb{H}_{\mathbb{F}}^{m}$ (rank-one symmetric spaces of noncompact type).
In particular, we recover the aforementioned result of Urbas, giving it a new
proof.

We remark that the translating solitons we construct here are all bowl-type
graphs of radial functions. In the Euclidean and hyperbolic cases, these
graphs are entire, whereas in the spherical case they project onto an open
hemisphere $B$ of $\mathbb{S}^{n},$ being asymptotic to the half-cylinder
$\partial B\times[0,+\infty).$

The paper is organized as follows. In Section 2, we fix notation regarding
hypersurfaces of Riemannian products $M\times\mathbb{R}.$ In Section 3, we
discuss graphs defined on families of parallel hypersurfaces, setting some
formulae. In Section 4, we introduce the $K^{\alpha}$-flow in
$M\times\mathbb{R}$ and establish a key lemma. Finally, in Section 5, we state
and prove our main result.

##  2\. Preliminaries

Given a Riemannian manifold $M^{n},$ $n\geq 2,$ consider the Riemannian
product $M\times\mathbb{R}$ endowed with its standard metric

$\langle\,,\,\rangle=\langle\,,\,\rangle_{M}+dt^{2},$

and denote its Levi-Civita connection by $\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern 1.5mu.$

Let $\varSigma^{n}$ be an oriented hypersurface of $M\times\mathbb{R}.$ Set
$N$ for its unit normal field and $A$ for its shape operator with respect to
$N,$ so that

$AX=-\mkern 1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu_{X}N,\,\,X\in T\varSigma,$

where $T\varSigma$ stands for the tangent bundle of $\varSigma$. The principal
curvatures of $\varSigma,$ that is, the eigenvalues of the shape operator $A,$
will be denoted by $k_{1}\,,\dots,k_{n}$. In this setting, the _Gaussian
curvature_ of $\varSigma$ is the function $K:\varSigma\rightarrow\mathbb{R}$
defined as

$K:=\det A=k_{1}\dots k_{n}\,.$

We denote by $\partial_{t}$ the gradient of the projection
$\pi_{\scriptscriptstyle\mathbb{R}}$ of $M\times\mathbb{R}$ on the factor
$\mathbb{R}.$ Given a hypersurface $\varSigma\subset M\times\mathbb{R},$ its
_height function_ $\xi$ and its _angle function_ $\varTheta$ are defined by
the following identities:

$\xi(x):=\pi_{\scriptscriptstyle\mathbb{R}}|_{\varSigma}\quad\text{and}\quad\varTheta(x):=\langle
N(x),\partial_{t}\rangle,\,\,x\in\varSigma.$

We shall denote the gradient of $\xi$ on $\varSigma$ by $T,$ so that

(1) $T=\partial_{t}-\varTheta N.$

Given $t\in\mathbb{R},$ the set $P_{t}:=M\times\\{t\\}$ is called a
_horizontal hyperplane_ of $M\times\mathbb{R}.$ Horizontal hyperplanes are all
isometric to $M$ and totally geodesic in $M\times\mathbb{R}.$ In this context,
we call a transversal intersection
$\varSigma_{t}:=\varSigma\mathrel{\text{\vbox{
\halign{#\cr\smash{$-$}\crcr$\pitchfork$\crcr} }}}P_{t}$ a _horizontal
section_ of $\varSigma.$ Any horizontal section $\varSigma_{t}$ is a
hypersurface of $P_{t}$ . So, at any point
$x\in\varSigma_{t}\subset\varSigma,$ the tangent space $T_{x}\varSigma$ of
$\varSigma$ at $x$ splits as the orthogonal sum

(2) $T_{x}\varSigma=T_{x}\varSigma_{t}\oplus{\rm Span}\\{T\\}.$

######  Definition 1.

A hypersurface $\varSigma\subset M\times\mathbb{R}$ is called _rotational_ ,
if there exists a fixed point $o\in M$ such that any connected component of
any horizontal section $\varSigma_{t}$ of $\varSigma$ is contained in a
geodesic sphere of $M\times\\{t\\}$ with center at $o\times\\{t\\}.$ If so,
the set $\\{o\\}\times\mathbb{R}$ is called the _axis_ of $\varSigma.$

Recall that the rank-one symmetric spaces of non-compact type are the
hyperbolic spaces
$\mathbb{H}_{\mathbb{R}}^{m},\,\,\mathbb{H}_{\mathbb{C}}^{m},\,\,\mathbb{H}_{\mathbb{K}}^{m}$
and $\mathbb{H}_{\mathbb{O}}^{2},$ $m\geq 1,$ called _real hyperbolic space,
complex hyperbolic space, quaternionic hyperbolic space_ and _Cayley
hyperbolic plane,_ respectively. We will adopt the unified notation
$\mathbb{H}_{\mathbb{F}}^{m}$ for these spaces with $m=2$ for
$\mathbb{F}=\mathbb{O}.$ The real dimension of $\mathbb{H}_{\mathbb{F}}^{m}$
is $n=m\dim\mathbb{F}.$

We remark that the hyperbolic spaces $\mathbb{H}_{\mathbb{F}}^{m}$ are all
Einstein–Hadamard manifolds whose sectional curvatures are negative and
pinched. In particular, their geodesic spheres are all strictly convex (see,
e.g., [6]).

##  3\. Graphs on Parallel Hypersurfaces

Consider an isometric immersion

$f:M_{0}^{n-1}\rightarrow M^{n}$

between two Riemannian manifolds $M_{0}^{n-1}$ and $M^{n},$ and suppose that
there is a neighborhood $\mathscr{U}$ of $M_{0}$ in $TM_{0}^{\perp}$ without
focal points of $f,$ that is, the restriction of the normal exponential map
$\exp^{\perp}_{M_{0}}:TM_{0}^{\perp}\rightarrow M$ to $\mathscr{U}$ is a
diffeomorphism onto its image. In this case, denoting by $\eta$ the unit
normal field of $f,$ there is an open interval $I\owns 0$ such that, for all
$p\in M_{0},$ the curve

(3) $\gamma_{p}(s)=\exp_{\scriptscriptstyle M}(f(p),s\eta(p)),\,s\in I,$

is a well defined geodesic of $M$ without conjugate points. Thus, for all
$s\in I,$

$\begin{array}[]{cccc}f_{s}:&M_{0}&\rightarrow&M\\\
&p&\mapsto&\gamma_{p}(s)\end{array}$

is an immersion of $M_{0}$ into $M,$ which is said to be _parallel_ to $f.$
Observe that, given $p\in M_{0}$, the tangent space $f_{s_{*}}(T_{p}M_{0})$ of
$f_{s}$ at $p$ is the parallel transport of $f_{*}(T_{p}M_{0})$ along
$\gamma_{p}$ from $0$ to $s.$ We also remark that, with the induced metric,
the unit normal $\eta_{s}$ of $f_{s}$ at $p$ is given by

$\eta_{s}(p)=\gamma_{p}^{\prime}(s).$

######  Definition 2.

Let $\phi:I\rightarrow\phi(I)\subset\mathbb{R}$ be an increasing
diffeomorphism, i.e., $\phi^{\prime}>0.$ With the above notation, we call the
set

(4) $\varSigma:=\\{(f_{s}(p),\phi(s))\in M\times\mathbb{R}\,;\,p\in
M_{0},\,s\in I\\},$

the _graph_ determined by $\\{f_{s}\,;\,s\in I\\}$ and $\phi,$ or
$(f_{s},\phi)$-_graph_ , for short.

For an arbitrary point $x=(f_{s}(p),\phi(s))$ of an $(f_{s},\phi)$-graph
$\varSigma,$ one has

$T_{x}\varSigma=f_{s_{*}}(T_{p}M_{0})\oplus{\rm
Span}\,\\{\partial_{s}\\},\,\,\,\partial_{s}=\eta_{s}+\phi^{\prime}(s)\partial_{t}.$

So, a unit normal to $\varSigma$ is

(5)
$N=\frac{-\phi^{\prime}}{\sqrt{1+(\phi^{\prime})^{2}}}\eta_{s}+\frac{1}{\sqrt{1+(\phi^{\prime})^{2}}}\partial_{t}\,.$

In particular, its angle function is

(6) $\varTheta=\frac{1}{\sqrt{1+(\phi^{\prime})^{2}}}\,\cdot$

As shown in [11, Theorem 6]), any $(f_{s},\phi)$-graph $\varSigma$ has the
$T$-_property_ , meaning that $T$ is a principal direction at any point of
$\varSigma$. More precisely,

(7) $AT=\frac{\phi^{\prime\prime}}{(\sqrt{1+(\phi^{\prime})^{2}})^{3}}T.$

Conversely, any hypersurface of $M\times\mathbb{R}$ with non vanishing angle
function having $T$ as a principal direction is given locally as an
$(f_{s},\phi)$-graph.

Given an $(f_{s},\phi)$-graph $\varSigma,$ let $\\{X_{1}\,,\dots,X_{n}\\}$ be
the orthonormal frame of principal directions of $\varSigma$ in which
$X_{n}=T/\|T\|.$ In this case, for $1\leq i\leq n-1,$ the fields $X_{i}$ are
all horizontal, that is, tangent to $M$ (cf. (2)). Therefore, setting

(8) $\varrho:=\frac{\phi^{\prime}}{\sqrt{1+(\phi^{\prime})^{2}}}$

and considering (5), we have, for all $i=1,\dots,n-1,$ that

$k_{i}=\langle AX_{i},X_{i}\rangle=-\langle\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu_{X_{i}}N,X_{i}\rangle=\varrho\langle\mkern
1.5mu\overline{\mkern-1.5mu\nabla\mkern-1.5mu}\mkern
1.5mu_{X_{i}}\eta_{s},X_{i}\rangle=-\varrho k_{i}^{s},$

where $k_{i}^{s}$ is the $i$-th principal curvature of $f_{s}\,.$ Also, it
follows from (7) that $k_{n}=\varrho^{\prime}.$ Thus, the principal curvatures
of the $(f_{s},\phi)$-graph $\varSigma$ at $(f_{s}(p),\phi(s))\in\varSigma$
are

(9) $k_{i}=-\varrho(s)k_{i}^{s}(p)\,\,(1\leq i\leq n-1)\quad\text{and}\quad
k_{n}=\varrho^{\prime}(s).$

In particular, the Gaussian curvature $K$ of $\varSigma$ at
$(f_{s}(p),\phi(s))$ is given by

(10) $K=(-\varrho(s))^{n-1}K_{s}(p)\varrho^{\prime}(s)\,,$

where $K_{s}$ denotes the Gaussian curvature of the hypersurface $f_{s}$ .

We remark that the function $\varrho$ defined in (8) determines the function
$\phi.$ Indeed, it follows from equality (8) that

(11)
$\phi(s)=\int_{s_{0}}^{s}\frac{\varrho(u)}{\sqrt{1-\varrho^{2}(u)}}du+\phi(s_{0}),\,\,\,s_{0},s\in
I.$

In addition, from (6) and (8), the unit normal $N$ defined in (5) can be
written as $N=-\varrho\eta_{s}+\varTheta\partial_{t}$ . Hence, the relation

(12) $\varTheta=\sqrt{1-\varrho^{2}}$

holds everywhere on any $(f_{s},\phi)$-graph $\varSigma.$ In particular, one
has $\varrho=\|T\|.$

Now, let us assume that the family of parallel hypersurfaces

$\mathscr{F}:=\\{f_{s}:M_{0}\rightarrow M\,;\,s\in I\\}$

is _isoparametric_ , that is, for each $s\in I,$ any principal curvature
$k_{i}^{s}$ of $f_{s}\in\mathscr{F}$ is constant (possibly depending on $i$
and $s$). We shall assume further that each $f_{s}$ is strictly convex with
$k_{i}^{s}<0.$

In this setting, the Gaussian curvature $K_{s}$ of each hypersurface $f_{s}$
is a (non vanishing) function of $s$ alone. Hence, from (10), the same is true
for any $(f_{s},\phi)$-graph $\varSigma$ built on $\mathscr{F},$ that is,

(13)
$K(f_{s}(p),\phi(s))=(-\varrho(s))^{n-1}\varrho^{\prime}(s)K_{s}\,,\,\,\,(f_{s}(p),\phi(s))\in\varSigma.$

It should also be noticed that, by equalities (9), such a graph $\varSigma$ is
strictly convex.

##  4\. Translating $K^{\alpha}$-Solitons in $M\times\mathbb{R}$

Given a real number $\alpha\neq 0,$ we say that an oriented strictly convex
hypersurface $\varSigma$ of $M\times\mathbb{R}$ _moves under_
$K^{\alpha}$-flow if there exists a one-parameter family of immersions
$F\colon M_{0}\times[0,u_{0})\rightarrow M\times\mathbb{R},$
$u_{0}\leq+\infty,$ such that

(14) $\left\\{\begin{array}[]{l}\frac{\partial F}{\partial
u}^{\perp}(p,u)=K^{\alpha}(p,u)N(p,u).\\\\[4.30554pt]
F(M_{0}\,,0)=\varSigma,\end{array}\right.$

where $N(p,u)$ is the inward unit normal to the hypersurface
$F_{u}:=F(.\,,u)$, $K(p,u)$ is the Gaussian curvature of $F_{u}$ with respect
to $N_{u}:=N(.\,,u),$ and $\frac{\partial F}{\partial u}^{\perp}$ denotes the
normal component of $\frac{\partial F}{\partial u},$ i.e.,

$\frac{\partial F}{\partial u}^{\perp}=\left\langle\frac{\partial F}{\partial
u},N_{u}\right\rangle N_{u}\,.$

In particular, the first equality in (14) is equivalent to

(15) $\left\langle\frac{\partial F}{\partial
u}(p,u),N(p,u)\right\rangle=K^{\alpha}(p,u).$

We call such a map $F$ a $K^{\alpha}$-_flow_ in $M\times\mathbb{R}.$

Denote by $\exp$ the exponential map of $M\times\mathbb{R}$ and consider an
isometric immersion $F_{0}\colon M_{0}\rightarrow M\times\mathbb{R}$. Define
then the map

$F(p,u):=\exp_{F_{0}(p)}(u\partial_{t}),\,\,(p,u)\in M_{0}\times[0,+\infty),$

and notice that, for each $u\in(0,+\infty),$ the hypersurface $F(M_{0}\,,u)$
is nothing but a vertical translation of $\varSigma:=F(M_{0},0).$ Since
vertical translations are isometries of $M\times\mathbb{R}$ keeping the second
factor invariant, we have that $\varSigma$ and $F(M_{0}\,,u)$ are congruent
with coinciding angle functions and Gaussian curvature, that is,

(16) $\varTheta(p,u)=\varTheta(p,0)\quad\text{and}\quad
K(p,u)=K(p,0)\,\,\,\forall(p,u)\in M_{0}\times[0,u_{0}).$

Now, differentiating $F$ with respect to $u,$ we have

(17) $\frac{\partial F}{\partial
u}(p,u)=(d\exp_{F_{0}(p)})(u\partial_{t})\partial_{t}=\partial_{t}\,.$

From (16) and (17), we have that $F$ satisfies (15) if and only if the
equality

$\varTheta(p,0)=K^{\alpha}(p,0)$

holds on $M_{0}$ . This fact motivates the following concept.

######  Definition 3.

Given $\alpha>0,$ we say that an oriented strictly convex hypersurface
$\varSigma$ of the Riemannian product $M\times\mathbb{R}$ is a _translating
soliton to the $K^{\alpha}$-curvature flow_ (or simply a _translating
$K^{\alpha}$-soliton_) if the equality

$K^{\alpha}=\varTheta$

holds everywhere on $\varSigma.$ (From the above discussion, any such
hypersurface $\varSigma$ is the initial data of a $K^{\alpha}$-flow by
vertical translations.)

Let us consider now an $(f_{s},\phi)$-graph $\varSigma$ in $M\times\mathbb{R}$
such that the family

$\mathscr{F}:=\\{f_{s}:M_{0}\rightarrow M\,;\,s\in I\\}$

is isoparametric with $k_{i}^{s}<0.$ In this case, it follows from (12) and
(13) that $\varSigma$ is a translating $K^{\alpha}$-soliton if and only if its
associated $\varrho$ function satisfies

(18)
$(-\varrho(s))^{n-1}\varrho^{\prime}(s)K_{s}=(1-\varrho^{2}(s))^{\frac{1}{2\alpha}},\,\,\,s\in
I.$

Since $K_{s}$ never vanishes on $I,$ we have that (18) holds for $\varrho$ if
and only if

(19)
$\uptau^{\prime}(s)=\frac{(-1)^{n-1}n(1-\uptau^{\frac{2}{n}})^{\frac{1}{2\alpha}}}{K_{s}}\,,\quad\uptau:=\varrho^{n}.$

Summarizing, we have the following result.

######  Lemma 1.

Let $\varSigma$ be an $(f_{s},\phi)$-graph in $M\times\mathbb{R}$ whose
associated family

$\mathscr{F}:=\\{f_{s}:M_{0}\rightarrow M\,;\,s\in I\\}$

of oriented parallel hypersurfaces is isoparametric with each of them having
negative principal curvatures. Then, $\varSigma$ is a translating
$K^{\alpha}$-soliton if and only if (19) holds for $\uptau:=\varrho^{n},$
where $\varrho:I\rightarrow\mathbb{R}$ is as in (8).

##  5\. The Main Result

For $n\geq 2$ and $m\geq 1,$ let $M$ be one of the following manifolds, each
of them endowed with its canonical Riemannian metric: Euclidean space
$\mathbb{R}^{n},$ the sphere $\mathbb{S}^{n},$ or one of the hyperbolic spaces
$\mathbb{H}_{\mathbb{F}}^{m}.$ Then, define

(20)
$\mathcal{R}_{M}:=\left\\{\begin{array}[]{ccc}+\infty&\text{if}&M\neq\mathbb{S}^{n}\\\\[4.30554pt]
\pi/2&\text{if}&M=\mathbb{S}^{n}\end{array}\right.$

and consider a family

(21) $\mathscr{F}:=\\{f_{s}:\mathbb{S}^{n-1}\rightarrow
M\,;\,\,\,s\in(0,\mathcal{R}_{M})\\}$

of concentric geodesic spheres of $M$ indexed by their radiuses, that is, for
a fixed point $o\in M$, and for each $s\in(0,\mathcal{R}_{M}),$
$f_{s}(\mathbb{S}^{n-1})$ is the (strictly convex) geodesic sphere $S_{s}(o)$
of $M$ with center at $o$ and radius $s.$ In accordance to the notation of
Section 3, for each $s\in(0,\mathcal{R}_{M}),$ we choose the outward
orientation of $f_{s}$ , so that _any principal curvature $k_{i}^{s}$ of
$f_{s}$ is negative_.

For $M=\mathbb{H}_{\mathbb{F}}^{m},$ the principal curvatures $k_{i}^{s}$ of
the geodesic spheres $f_{s}\in\mathscr{F}$ are:

(22) $\displaystyle k_{1}^{s}$
$\displaystyle=-\frac{1}{2}\coth(s/2)\,\,\,\text{with
multiplicity}\,\,\,n-p-1.$ $\displaystyle k_{2}^{s}$
$\displaystyle=-\coth(s)\,\,\,\text{with multiplicity}\,\,\,p,$

where $n=\dim\mathbb{H}_{\mathbb{F}}^{m},$ $p=n-1$ for $\mathbb{H}^{n}$, $p=1$
for $\mathbb{H}_{\mathbb{C}}^{m}$, $p=3$ for $\mathbb{H}_{\mathbb{K}}^{m}$,
and $p=7$ for $\mathbb{H}_{\mathbb{O}}^{2}$ (see, e.g., [8, pgs. 353, 543] and
[13]). Thus, the Gaussian curvature $K_{s}$ of the geodesic sphere $S_{s}(o)$
of $\mathbb{H}_{\mathbb{F}}^{m}$ is given by

$K_{s}=(-1)^{n-1}\left(\frac{1}{2}\coth(s/2)\right)^{n-p-1}(\coth s)^{p}\,.$

In particular, the function

$\frac{(-1)^{n-1}}{K_{s}}=(2\tanh(s/2))^{n-p-1}(\tanh s)^{p}$

is well defined and nonnegative on $[0,+\infty).$

As is well known, in the cases $M=\mathbb{R}^{n}$ and $M=\mathbb{S}^{n}$ one
has

$\frac{(-1)^{n-1}}{K_{s}}=s^{n-1}\quad\text{and}\quad\frac{(-1)^{n-1}}{K_{s}}=(\tan
s)^{n-1},$

respectively.

It is easily seen that, in any of these three cases, the equality

(23)
$\lim_{s\rightarrow\mathcal{R}_{M}}\int_{0}^{s}\frac{(-1)^{n-1}}{K_{v}}dv=+\infty.$

holds. Finally, set

  * •

$\delta_{n}:=\,\,\left\\{\begin{array}[]{ccc}1/2&\text{if}&n=2.\\\\[4.30554pt]
\max\\{\frac{1}{4},\frac{1}{n-1}\\}&\text{if}&n>2.\\\\[4.30554pt]
\end{array}\right.$

  * •

$\mathcal{I}_{M}:=\left\\{\begin{array}[]{lcc}(0,1/2]&\text{if}&M\neq\mathbb{S}^{n}.\\\\[4.30554pt]
[\delta_{n},1/2]&\text{if}&M=\mathbb{S}^{n}.\end{array}\right.$

With this notation, we now state and prove our main result.

######  Theorem 1.

Let $M$ and $\mathcal{I}_{M}\subset(0,1/2]$ be as above. Then, for all
$\alpha\in\mathcal{I}_{M},$ there exists a complete rotational strictly convex
translating $K^{\alpha}$-soliton in the closed half-space
$M\times[0,+\infty),$ whose height function is unbounded. In addition, the
following assertions hold:

  * •

For $M\neq\mathbb{S}^{n},$ $\varSigma$ is an entire graph over $M.$

  * •

For $M=\mathbb{S}^{n},$ $\varSigma$ is a graph over an open hemisphere
$B\subset\mathbb{S}^{n},$ being asymptotic to the half-cylinder $\partial
B\times[0,+\infty).$

###### Proof.

Let $\mathscr{F}$ be a family of parallel geodesic spheres of $M$ as in (21).
We intend to determine a function $\phi\in C^{\infty}[0,\mathcal{R}_{M})$ such
that the corresponding $(f_{s},\phi)$-graph $\varSigma$ becomes the desired
$K^{\alpha}$-soliton. With this purpose, let us consider the equality (19) as
an ODE with variable $\uptau,$ and rewrite it as

(24)
$\frac{d\uptau}{(1-\uptau^{\frac{2}{n}})^{\frac{1}{2\alpha}}}=\frac{(-1)^{n-1}n}{K_{s}}ds.$

In order to solve (24), consider the functions

$\Phi\colon[0,1)\rightarrow[0,+\infty)\quad\text{and}\quad\Psi\colon[0,\mathcal{R}_{M})\rightarrow[0,+\infty)$

given by

$\Phi(\uptau):=\int_{0}^{\uptau}\frac{du}{(1-u^{\frac{2}{n}})^{\frac{1}{2\alpha}}}\qquad\text{and}\qquad\Psi(s):=\int_{0}^{s}\frac{(-1)^{n-1}n}{K_{v}}dv.$

It follows from (23) that $\Psi$ is a diffeomorphism. Let us show that the
same is true for $\Phi.$ Indeed, $\Phi$ is a $C^{\infty}$ function with
positive derivative, which implies that it is a diffeomorphism over its image.
So, it remains to prove that $\Phi([0,1))=[0,+\infty).$ To this end, notice
first that $1-u^{2/n}\leq 1-u\,\forall u\in[0,1).$ Therefore, setting
$p:=1/(2\alpha)\geq 1,$ one has

$\Phi(\uptau)\geq\int_{0}^{\uptau}\frac{du}{(1-u)^{p}}=\left\\{\begin{array}[]{ccc}\log\left(\frac{1}{1-\uptau}\right)&\text{if}&p=1.\\\\[8.61108pt]
\frac{(1-\uptau)^{1-p}-1}{p-1}&\text{if}&p>1,\end{array}\right.$

which implies that $\Phi(\uptau)\rightarrow+\infty$ as $\uptau\rightarrow 1.$
Hence, $\Phi([0,1))=[0,+\infty),$ as asserted.

Now, we can define $\uptau:[0,\mathcal{R}_{M})\rightarrow\mathbb{R}$ by

(25) $\uptau(s)=\Phi^{-1}(\Psi(s)),$

which is clearly a solution of (24) (and so of (19)) satisfying

(26) $0=\uptau(0)\leq\uptau(s)<1\,\,\,\,\forall
s\in[0,\mathcal{R}_{M})\quad\text{and}\quad\lim_{s\rightarrow\mathcal{R}_{M}}\uptau(s)=1.$

Therefore, by Lemma 1, the corresponding $(f_{s},\phi)$-graph $\varSigma$ is a
translating $K^{\alpha}$-soliton in $M\times\mathbb{R}.$ Moreover, since
$\varrho={\uptau}^{1/n}$, we have from (11) (with $s_{0}=0$) and (26) that the
function $\phi$ is defined in $[0,\mathcal{R}_{M})$ and satisfies

$\phi(0)=\phi^{\prime}(0)=0\quad\text{and}\quad\lim_{s\rightarrow\mathcal{R}_{M}}\phi^{\prime}(s)=+\infty.$

We conclude from the above discussion that, if $M\neq\mathbb{S}^{n}$, then
$\varSigma$ is an unbounded entire graph over $M$ which is contained in the
closed half-space $M\times[0,+\infty).$ In particular, $\varSigma$ is complete
(Fig. 1).

Figure 1. A translating $K^{\alpha}$-soliton in $M\times\mathbb{R}.$

If $M=\mathbb{S}^{n}$, then $\varSigma$ is a graph over the open hemisphere
$B$ centered at $o\in\mathbb{S}^{n}$ (Fig. 2). Thus, to conclude that
$\varSigma$ is complete and asymptotic to $\partial B\times[0,+\infty),$ we
must prove that $\phi$ is unbounded.

For $n=2,$ we have from the hypothesis that $\alpha=1/2.$ In this case, it is
easily checked that $\varrho(s)=\sin s$ is the solution of (18) satisfying
$\varrho(0)=0.$ So, from (11),

$\phi(s)=\int_{0}^{s}\frac{\varrho(u)}{\sqrt{1-\varrho^{2}(u)}}du=\int_{0}^{s}\tan(u)du=\log\left(\frac{1}{\cos
s}\right),\,\,\,\,s\in(0,\pi/2),$

which implies that $\phi(s)\rightarrow+\infty$ as $s\rightarrow\pi/2.$

For $n>2,$ let us show first that $\uptau^{\prime}$ is bounded in $(0,\pi/2).$
Indeed, from (19),

$\uptau^{\prime}(s)=n(\tan s)^{n-1}(1-(\uptau(s))^{2/n})^{p},\quad
p=\frac{1}{2\alpha}\cdot$

So, setting

$\mu_{1}(s):=n(\tan s)^{n-2}(\sec
s)^{2}(1-(\uptau(s))^{2/n})^{p}\quad\text{and}\quad\mu_{2}(s):=(1-(\uptau(s))^{\frac{2}{n}})\tan
s,$

a straightforward calculation yields

(27) $\displaystyle\uptau^{\prime\prime}(s)$ $\displaystyle=$
$\displaystyle\mu_{1}(s)\left[(n-1)-\frac{2p(\uptau(s))^{\frac{2-n}{n}}(\cos
s)^{2}(\tan s)\uptau^{\prime}(s)}{n(1-(\uptau(s))^{\frac{2}{n}})}\right]$
$\displaystyle=$
$\displaystyle\mu_{1}(s)\left[(n-1)-\frac{2p(\uptau(s))^{\frac{2-n}{n}}(\sin
s)^{2}\uptau^{\prime}(s)}{n\mu_{2}(s)}\right],\,\,\,s\in(0,\pi/2).$

Let us suppose, by contradiction, that there exists a sequence $(s_{k})$ in
$(0,\pi/2)$ such that $\uptau^{\prime}(s_{k})\rightarrow+\infty.$ We can
assume, without loss of generality, that $\uptau^{\prime\prime}(s_{k})>0$ for
all $k\in\mathbb{N}.$ Under this assumption, we have from (27) that the
sequence $\mu_{2}(s_{k})$ is unbounded above. Otherwise, for a sufficiently
large $k,$ the expression in the brackets would be negative for $s=s_{k},$
which would give $\uptau^{\prime\prime}(s_{k})<0$ — a contradiction.

Figure 2. A translating $K^{\alpha}$-soliton in
$\mathbb{S}^{n}\times\mathbb{R}.$

So, we can assume $\mu_{2}(s_{k})\rightarrow+\infty.$ However, from our choice
of $\alpha,$ we have that $0\leq p-1\leq 1,$ which yields
$(1-(\uptau(s_{k}))^{2/n})^{p-1}\geq(1-(\uptau(s_{k}))^{2/n}).$ Thus,

$\frac{\uptau^{\prime}(s_{k})}{\mu_{2}(s_{k})}=n(\tan
s_{k})^{n-2}(1-(\uptau(s_{k}))^{2/n})^{p-1}\geq n(\tan
s_{k})^{n-3}\mu_{2}(s_{k})\rightarrow+\infty,$

which, together with equality (27), gives $\uptau^{\prime\prime}(s_{k})<0$ for
a sufficiently large $k$ — again a contradiction. Therefore, $\uptau^{\prime}$
(and so $\varrho^{\prime}$) is bounded in $(0,\pi/2).$

Now, let $C>0$ be such that $\varrho^{\prime}(s)<C$ for all $s\in(0,\pi/2).$
Since we are assuming $\alpha(n-1)\geq 1,$ equalities (11) and (18) yield

$\phi(s)=\int_{0}^{s}\frac{\varrho(u)}{\sqrt{1-\varrho^{2}(u)}}du=\int_{0}^{s}\frac{(\tan
u)^{\alpha(n-1)}}{(\varrho(u))^{\alpha(n-1)-1}(\varrho^{\prime}(u))^{\alpha}}du\geq\frac{1}{C^{\alpha}}\int_{0}^{s}\tan
u\,du,$

which implies that $\phi(s)\rightarrow+\infty$ as $s\rightarrow\pi/2.$ This
concludes the proof. ∎

We finish by noting that, for a given Riemannian manifold $M,$ we have from
Lemma 1 that there exist local translating $K^{\alpha}$-solitons in
$M\times\mathbb{R}$ as long as $M$ admits families of isoparametric, strictly
convex hypersurfaces. This applies, for instance, to the hyperbolic spaces
$\mathbb{H}_{\mathbb{F}}^{m}$ and their families of parallel horospheres (see
Proposition-(vi), pg 88, in [6]), to the real hyperbolic space
$\mathbb{H}_{\mathbb{R}}^{n}$ and its families of equidistant hypersurfaces to
a given totally geodesic hyperplane, as well as to $\mathbb{S}^{n}$ and any of
its many families of strictly convex isoparametric hypersurfaces (cf. [12] and
the references therein).

_Acknowledgements._ The author is indebt to Isabel L. Rios and Roberto T. de
Oliveira for helpful conversations during the preparation of this paper.

## References

  * [1] B. Andrews: Motion of hypersurfaces by Gauss curvature, Pacific J. Math. 195 (2000), no. 1, 1–34.
  * [2] B. Andrews, X. Chen: Surfaces moving by powers of Gauss curvature, Pure Appl. Math. 8 (2012), no. 4, 825–834.
  * [3] B. Andrews, B. Chow, C. Guenther, M. Langford: Extrinsic Geometric Flows. Graduate studies in mathematics 206, American Mathematical Society (2020).
  * [4] B. Andrews, P. Guan, L. Ni: Flow by powers of the Gauss curvature, Adv. Math. 299 (2016), 174–201.
  * [5] B. Andrews, J. McCoy: Convex hypersurfaces with pinched principal curvatures and flow of convex hypersurfaces by high powers of curvature, Trans. Amer. Math. Soc. 364 (2012), no. 7, 3427–3447.
  * [6] J. Berndt, F. Tricerri, L. Vanhecke: Generalized Heisenberg groups and Damek-Ricci harmonic spaces. Lecture Notes in Mathematics 1598, Springer Verlag (1995).
  * [7] S. Brendle, K. Choi, P. Daskalopoulos: Asymptotic behavior of flows by powers of the Gaussian curvature, Acta Math. 219 (2017), no. 1, 1–16.
  * [8] T. Cecil, P. Ryan: Geometry of hypersurfaces. Springer Verlag (2015).
  * [9] K. Choi, P. Daskalopoulos, L. Kim, K.-A. Lee: The evolution of complete non-compact graphs by powers of Gauss curvature, J. Reine Angew. Math. 757 (2019), 131–158.
  * [10] K. Choi, P. Daskalopoulos, K.-A. Lee: Translating solutions to the Gauss curvature flow with flat sides. Anal. PDE 14 (2) (2021), 595–616.
  * [11] R. F. de Lima, P. Roitman: Helicoids and catenoids in $M\times\mathbb{R}.$ Ann. Mat. Pura Appl. 200, 2385–2421 (2021).
  * [12] M. Domínguez-Vázquez: An introduction to isoparametric foliations. Preprint (2018) (available at: http://xtsunxet.usc.es/miguel/teaching/jae2018.html).
  * [13] S. Kim, Y. Nikolayevsky, J. Park: Einstein hypersurfaces of the Cayley projective plane. Differential Geom. Appl. 69, (2020), 1–6.
  * [14] E. Kocakusakli, M. Ortega: Extending translating solitons in semi-Riemannian manifolds, in: Lorentzian Geometry and Related Topics, in: Springer Proc. Math. Stat., vol. 211, (2016).
  * [15] M.A. Lawn, M. Ortega, Translating solitons from semi-Riemannian submersions, Preprint arXiv :1607.04571.
  * [16] X. Li, K. Wang: Nonparametric hypersurfaces moving by powers of Gauss curvature, Michigan Math. J. 66 (2017), no. 4, 675–682.
  * [17] J. H. Lira, F. Martín: Translating solitons in Riemannian products. J. Differential Equations 266, (2019), 7780–7812.
  * [18] F. Martín, A. Savas-Halilaj, K. Smoczyk: On the topology of translating solitons of the mean curvature flow, Calc. Var. Partial Differential Equations 54 (2015), no. 3, 2853–2882.
  * [19] J. Spruck, L. Xiao: Complete translating solitons to the mean curvature flow in $\mathbb{R}^{3}$ with nonnegative mean curvature, arXiv:1703.01003v3, (2017).
  * [20] J. Urbas: Complete noncompact self-similar solutions of Gauss curvature flows. I. Positive powers, Math. Ann. 311 (1998), no. 2, 251–274.