# Latent Skill Planning
for Exploration and Transfer

Kevin Xie1,∗, Homanga Bharadhwaj1, Danijar Hafner1,2
Animesh Garg1,3, Florian Shkurti1
1University of Toronto and Vector Institute, 2Google Brain, 3Nvidia
<EMAIL_ADDRESS><EMAIL_ADDRESS>Kevin and Homanga contributed
equally to this work.

###### Abstract

To quickly solve new tasks in complex environments, intelligent agents need to
build up reusable knowledge. For example, a learned world model captures
knowledge about the environment that applies to new tasks. Similarly, skills
capture general behaviors that can apply to new tasks. In this paper, we
investigate how these two approaches can be integrated into a single
reinforcement learning agent. Specifically, we leverage the idea of partial
amortization for fast adaptation at test time. For this, actions are produced
by a policy that is learned over time while the skills it conditions on are
chosen using online planning. We demonstrate the benefits of our design
decisions across a suite of challenging locomotion tasks and demonstrate
improved sample efficiency in single tasks as well as in transfer from one
task to another, as compared to competitive baselines. Videos are available
at: https://sites.google.com/view/latent-skill-planning/

##  1 Introduction

Figure 1: Visual illustration of the 2D root position of the quadruped trained
with LSP on an environment with random obstacles and transferred to this
environment with obstacles aligned in a line. The objective is to reach the
goal location in red.

Humans can effortlessly compose skills, where skills are a sequence of
temporally correlated actions, and quickly adapt skills learned from one task
to another. In order to build re-usable knowledge about the environment,
Model-based Reinforcement Learning (MBRL) (Wang et al., 2019) provides an
intuitive framework which holds the promise of training agents that generalize
to different situations, and are sample efficient with respect to number of
environment interactions required for training. For temporally composing
behaviors, hierarchical reinforcement learning (HRL) (Barto & Mahadevan, 2003)
seeks to learn behaviors at different levels of abstraction explicitly.

A simple approach for learning the environment dynamics is to learn a world
model either directly in the observation space (Chua et al., 2018; Sharma et
al., 2019; Wang & Ba, 2019) or in a latent space (Hafner et al., 2019; 2018).
World models summarize an agent’s experience in the form of learned transition
dynamics, and reward models, which are used to learn either parametric
policies by amortizing over the entire training experience (Hafner et al.,
2019; Janner et al., 2019), or perform online planning as done in Planet
(Hafner et al., 2018), and PETS (Chua et al., 2018). Amortization here refers
to learning a parameterized policy, whose parameters are updated using samples
during the training phase, and which can then be directly queried at each
state to output an action, during evaluation.

Fully online planning methods such as PETS (Chua et al., 2018) only learn the
dynamics (and reward) model and rely on an online search procedure such as
Cross-Entropy Method (CEM; Rubinstein, 1997) on the learned models to
determine which action to execute next. Since rollouts from the learned
dynamics and reward models are not executed in the actual environment during
training, these learned models are sometimes also referred to as imagination
models Hafner et al. (2018; 2019). Fully amortized methods such as Dreamer
Hafner et al. (2019), train a reactive policy with many rollouts from the
imagination model. They then execute the resulting policy in the environment.

The benefit of the amortized method is that it becomes better with experience.
Amortized policies are also faster. An action is computed in one forward pass
of the reactive policy as opposed to the potentially expensive search
procedure used in CEM. Additionally, the performance of the amortized method
is more consistent as CEM relies on drawing good samples from a random action
distribution. On the other hand, the shortcoming of the amortized policy is
generalization. When attempting novel tasks unseen during training, CEM will
plan action sequences for the new task, as per the new reward function while a
fully amortized method would be stuck with a behaviour optimized for the
training tasks. Since it is intractable to perform fully online random
shooting based planning in high-dimensional action spaces (Bharadhwaj et al.,
2020; Amos & Yarats, 2019), it motivates the question: can we combine online
search with amortized policy learning in a meaningful way to learn useful and
transferable skills for MBRL?

To this end, we propose a partially amortized planning algorithm that
temporally composes high-level skills through the Cross-Entropy Method (CEM)
(Rubinstein, 1997), and uses these skills to condition a low-level policy that
is amortized over the agent’s experience. Our world model consists of a
learned latent dynamics model, and a learned latent reward model. We have a
mutual information (MI) based intrinsic reward objective, in addition to the
predicted task rewards that are used to train the low level-policy, while the
high level skills are planned through CEM using the learned task rewards. We
term our approach Learning Skills for Planning (LSP).

The key idea of LSP is that the high-level skills are able to abstract out
essential information necessary for solving a task, while being agnostic to
irrelevant aspects of the environment, such that given a new task in a similar
environment, the agent will be able to meaningfully compose the learned skills
with very little fine-tuning. In addition, since the skill-space is low
dimensional, we can leverage the benefits of online planning in skill space
through CEM, without encountering intractability of using CEM for planning
directly in the higher dimensional action space and especially for longer time
horizons (Figure 1).

In summary, our main contributions are developing a partially amortized
planning approach for MBRL, demonstrating that high-level skills can be
temporally composed using this scheme to condition low level policies, and
experimentally demonstrating the benefit of LSP over challenging locomotion
tasks that require composing different behaviors to solve the task, and
benefit in terms of transfer from one quadruped locomotion task to another,
with very little adaptation in the target task.

##  2 Background

We discuss learning latent dynamics for MBRL, and mutual information skill
discovery, that serve as the basic theoretical tools for our approach.

###  2.1 Learning Latent Dynamics and Behaviors in Imagination

Latent dynamics models are special cases of world models used in MBRL, that
project observations into a latent representation, amenable for planning
(Hafner et al., 2019; 2018). This framework is general as it can model both
partially observed environments where sensory inputs can be pixel
observations, and fully observable environments, where sensory inputs can be
proprioceptive state features. The latent dynamics models we consider in this
work, consist of four key components, a representation module
$p_{\theta}(s_{t}|s_{t-1},a_{t-1},o_{t})$ and an observation module
$q_{\theta}(o_{t}|s_{T})$ that encode observations and actions to continuous
vector-valued latent states $s_{t}$, a latent forward dynamics module
$q_{\theta}(s_{t}|s_{t-1},a_{t-1})$ that predicts future latent states given
only the past states and actions, and a task reward module
$q_{\theta}(r_{t}|s_{t})$, that predicts the reward from the environment given
the current latent state. To learn this model, the agent interacts with the
environment and maximizes the following expectation under the dataset of
environment interactions $\mathcal{D}=\\{(o_{t},a_{t},r_{t})\\}$

$\displaystyle\mathcal{J}\doteq\mathbb{\operatorname{E}}_{\mathcal{D}}\bigg{(}\sum_{t}\Big{(}\mathcal{J}_{\mathrm{O}}^{t}+\mathcal{J}_{\mathrm{R}}^{t}+\mathcal{J}_{\mathrm{D}}^{t}\Big{)}\bigg{)}+\text{const}\qquad\mathcal{J}_{\mathrm{O}}^{t}\doteq\
\ln q(o_{t}\mids_{t})$ (1) $\displaystyle\mathcal{J}_{\mathrm{R}}^{t}\doteq\ln
q(r_{t}\mids_{t})\qquad\mathcal{J}_{\mathrm{D}}^{t}\doteq\
-\beta\operatorname{KL}\big{(}p(s_{t}\mids_{t-1},a_{t-1},o_{t})\;\big{\|}\;q(s_{t}\mids_{t-1},a_{t-1})\big{)}.$

For optimizing behavior under this latent dynamics model, the agent rolls out
trajectories in imagination and estimates the value $V(\cdot)$ of the imagined
trajectories $\\{s_{\tau},a_{\tau},r_{\tau}\\}_{\tau=t}^{t+H}$ through
$\text{TD}(\lambda)$ estimates as described by Sutton & Barto (2018); Hafner
et al. (2019). The agent can either learn a fully amortized policy
$q_{\phi}(a|s)$ as done in Dreamer, by backpropagating through the learned
value network $v_{\psi}(\cdot)$ or plan online through CEM, for example as in
Planet.

###  2.2 Mutual Information Skill Discovery

Some methods for skill discovery have adopted a probabilistic approach that
uses the mutual information between skills and future states as an objective
Sharma et al. (2019). In this approach, skills are represented through a
latent variable $z$ upon which a low level policy $\pi(a|s,z)$ is conditioned.
Given the current state $s_{0}$, skills are sampled from some selection
distribution $p(z|s_{0})$. The skill conditioned policy is executed under the
environment dynamics $p_{d}(s_{t+1}|s_{t},a)$ resulting in a series of future
states abbreviated $s^{\prime}:=\\{s\\}$.

Mutual information is defined as:

$\mathcal{MI}(z,\\{s\\}|s_{0})=\mathcal{H}(z|s_{0})-\mathcal{H}(z|\\{s\\},s_{0})=\mathcal{H}(\\{s\\}|s_{0})-\mathcal{H}(\\{s\\}|s_{0},z)$

It quantifies the reduction in uncertainty about the future states given the
skill and vice versa. By maximizing the mutual information with respect to the
low level policy, the skills are encouraged to produce discernible future
states.

(a) Fully Online

(b) Fully Amortized

Figure 2: (a) Online planning where actions are sampled from a distribution
with parameters $\theta_{i}$ (b) Fully amortized policy with parameters $\phi$
(c) [LSP ] Skills are sampled from a distribution with parameters
$\theta_{i}$, and actions are sampled from a skill-conditioned policy with
parameters $\phi$. Here $a$ are actions, $s$ states and $z$ latent plan
variables. $\theta_{i}$ represents parameters of the planning distribution and
$\phi$ are the parameters of the policy.

##  3 Partial Amortization through Hierarchy

Our aim is to learn behaviors suitable for solving complex control tasks, and
amenable to transfer to different tasks, with minimal fine-tuning. To achieve
this, we consider the setting of MBRL, where the agent builds up re-usable
knowledge of the environment dynamics. For planning, we adopt a partial
amortization strategy, such that some aspects of the behavior are re-used over
the entire training experience, while other aspects are learned online. We
achieve partial amortization by forming high level latent plans and learning a
low level policy conditioned on the latent plan. The three different forms of
amortization in planning are described visually through probabilistic
graphical models in Figure 2 and Figure 3.

We first describe the different components of our model, motivate the mutual
information based auxiliary objective, and finally discuss the complete
algorithm.

Figure 3:  LSP samples kills $z$ from a distribution optimized with CEM
online, and samples actions $a$ for every latent state $s$ from skill-
conditioned policy. The skill distribution is updated by maximizing mutual
information between the skills and the observed latent state distribution. The
policy is updated by backpropagating value estimates based on the environment
rewards $r$ through the learned latent synamics model.

#### World model.

Our world model is a latent dynamics model consisting of the components
described in section 2.

#### Low level policy.

The low-level policy $q_{\phi}(a_{t}|s_{t},z)$ is used to decide which action
to execute given the current latent state $s_{t}$ and the currently active
skill $z$. Similar to Dreamer (Hafner et al., 2019), we also train a value
model $v_{\psi}(s_{t})$ to estimate the expected rewards the action model
achieves from each state $s_{t}$. We estimate value the same way as in
equation 6 of Dreamer, balancing bias and variance. The action model is
trained to maximize the estimate of the value, while the value model is
trained to fit the estimate of the value that alters as the action model is
updated, as done in a typical actor-critic setup (Konda & Tsitsiklis, 2000).

#### High level skills.

In our framework high level skills are continuous random variables that are
held for a fixed number $K$ steps. The high-level skills $z$ are sampled from
a skill selection distribution $p(z_{1:\lceil
H/K\rceil}|\zeta)=\mathcal{N}(\mu,\Sigma)$ which is optimized for task
performance through CEM. Here, $H$ denotes the planning horizon. For the sake
of notational convenience we denote $z_{1:\lceil H/K\rceil}$ as $z$. Let (j)
denote the $j^{\text{th}}$ CEM iteration. We first sample $G$ skills
$\\{z^{(g)}\\}_{g=1}^{G}\sim p(z|\zeta^{(j)})$, execute $G$ parallel imaginary
rollouts of horizon $H$ in the learned model with the skill-conditioned policy
$q_{\phi}(a_{t}|s_{t},z^{(g)})$. Instead of evaluating rollouts based only on
the sum of rewards, we utilize the value network and compute value estimates
$\\{V_{g}\\}_{g=1}^{G}$. We sort $\\{V_{g}\\}_{g=1}^{G}$, choose the top $M$
values, and use the corresponding skills to update the sampling distribution
parameters as $\zeta^{(j+1)}=(\mu^{(j+1)},\Sigma^{(j+1)})$

$\mu^{(j+1)}=\operatorname{Mean}(\\{z^{(m)}\\}_{m=1}^{M})\;\;\;\;\;\Sigma^{(j+1)}=\operatorname{Variance}(\\{z^{(m)}\\}_{m=1}^{M})$

###  3.1 Overall Algorithm

Our overall algorithm consists of the three phases typical in a MBRL pipeline,
that are performed iteratively. The complete algorithm is shown in Algorithm
1. The sub-routine for CEM planning that gets called in Algorithm 1 is
described in Algorithm 2.

#### Model Learning.

We sample a batch of tuples from the dataset of environment interactions
$\\{(a_{t},o_{t},r_{t})\\}_{t=k}^{k+L}\sim\mathcal{D}$, compute the latent
states $s_{t}\sim p_{\theta}(s_{t}\mids_{t-1},a_{t-1},o_{t})$, and use the
resulting data to update the models
$p_{\theta}(s_{t}\mids_{t\text{-}1},a_{t\text{-}1},o_{t})$,
$q_{\theta}(s_{t}\mids_{t\text{-}1},a_{t\text{-}1})$, and
$q_{\theta}(r_{t}\mids_{t})$ through the variational information bottleneck
(VIB) (Tishby et al., 2000; Alemi et al., 2016) objective as in equation 13 of
Hafner et al. (2019) and as described in section 2.

#### Behavior Learning.

Here, we roll out the low-level policy $q_{\phi}(a_{t}|s_{t},z)$ in the world
model and use the state transitions and predicted rewards to optimize the
parameters of the policy $\phi$, the skill distribution $\zeta$, the value
model $\psi$, and the backward skill predictor $\chi$. The backward skill
predictor predicts the skill $z$ given latent rollouts $\\{s\\}$.

#### Environment Interaction.

This step is to collect data in the actual environment for updating the model
parameters. Using Model-Predictive Control (MPC), we re-sample high-level
skills from the optimized $p_{\zeta}(z)$ every $K$ steps, and execute the low-
level policy $q_{\phi}(a_{t}|s_{t},z)$, conditioned on the currently active
skill $z$. Hence, the latent plan has a lower temporal resolution as compared
to the low level policy. This helps us perform temporally abstracted
exploration easily in the skill space. We store the (observation, action,
reward) tuples in the dataset $\mathcal{D}$.

###  3.2 Mutual Information Skill Objective

Merely conditioning the low level policy $q_{\phi}(a_{t}|s_{t},z)$ on the
skill $z$ is not sufficient as it is prone to ignoring it. Hence, we
incorporate maximization of the mutual information (MI) between the latent
skills $z$ and the sequence of states $\\{s\\}$ as an auxiliary objective.

In this paper, we make use of imagination rollouts to estimate the mutual
information under the agent’s learned dynamics model. We decompose the mutual
information in terms of skill uncertainty reduction
$\mathcal{MI}(z,\\{s\\}|s_{0})=\mathcal{H}(z|s_{0})-\mathcal{H}(z|\\{s\\},s_{0})$.

#### Estimating $\mathcal{MI}(z,\\{s\\}|s_{0})$.

Explicitly writing out the entropy terms, we have

$\mathcal{MI}(z,\\{s\\}|s_{0})=\mathcal{H}(z|s_{0})-\mathcal{H}(z|\\{s\\},s_{0})=\int
p(z,\\{s\\},s_{0})\log\frac{p(z|\\{s\\},s_{0})}{p(z|s_{0})}$

In this case we need a tractable approximation to the skill posterior
$p(z|s_{0},s^{\prime})$.

$\mathcal{MI}(z,\\{s\\}|s_{0})=\int
p(z,\\{s\\},s_{0})\left(\log\frac{q(z|\\{s\\},s_{0})}{p(z|s_{0})}+\log\frac{p(z|\\{s\\},s_{0})}{q(z|\\{s\\},s_{0})}\right)$

Here the latter term is a KL divergence and must hence be positive, providing
a lower bound for $\mathcal{MI}$.

$\mathcal{MI}(z,\\{s\\}|s_{0})\geq\int
p(z,\\{s\\},s_{0})\log\frac{q(z|\\{s\\},s_{0})}{p(z|s_{0})}$ $=\int
p(z,\\{s\\},s_{0})\log q(z|\\{s\\},s_{0})-\int p(z|s_{0})p(s_{0})\log
p(z|s_{0})$ $={\mathbb{E}}_{p(z,\\{s\\},s_{0})}[\log
q(z|\\{s\\},s_{0})]+{\mathbb{E}}_{s_{0}}[\mathcal{H}[p(z|s_{0})]]$

We parameterize $q(z|\\{s\\},s_{0})$ with $\chi$, i.e.
$q_{\chi}(z|\\{s\\},s_{0})$, and call it the backward skill predictor, as it
predicts the skill $z$ given latent rollouts $\\{s\\}$. It is trained through
standard supervised learning to maximize the likelihood of imagined rollouts
$\mathbb{E}_{p(z,\\{s\\},s_{0})}[\log q(z|\\{s\\},s_{0})]$. This mutual
information objective is only a function of the policy through the first term
and hence we use it as the intrinsic reward for the agent $r_{i}=\log
q(z|\\{s\\},s_{0})$.

The second term ${\mathbb{E}}_{s_{0}}[\mathcal{H}[p(z|s_{0})]]$ is the entropy
of the skill selection distribution. When skills begin to specialize, the CEM
distribution will naturally decrease in entropy and so we add Gaussian noise
$\epsilon$ to the mean of the CEM-based skill distribution,
$\mu\leftarrow\mu+\epsilon\;\;\;\text{where}\;\;\;\epsilon\sim\mathcal{N}(0,\mathbb{I\sigma})$.
By doing this we lower bound the entropy of the skill selection distribution.

##  4 Experiments

We perform experimental evaluation over locomotion tasks based on the DeepMind
Control Suite framework (Tassa et al., 2018) to understand the following
questions:

  * •

Does LSP learn useful skills and compose them appropriately to succeed in
individual tasks?

  * •

Does LSP adapt to a target task with different environment reward functions
quickly, after being pre-trained on another task?

To answer these, we perform experiments on locomotion tasks, using agents with
different dynamics - Quadruped, Walker, Cheetah, and Hopper, and environments
where either pixel observations or proprioceptive features are available to
the agent. Our experiments consist of evaluation in single tasks, in transfer
from one task to another, ablation studies, and visualization of the learned
skills.

###  4.1 Setup

#### Baselines.

We consider Dreamer Hafner et al. (2019), which is a state of the art model-
based RL algorithm with fully amortized policy learning, as the primary
baseline, based on its open-source tensorflow2 implementation. We consider a
Hierarchical RL (HRL) baseline, HIRO (Nachum et al., 2018) that trains a high
level amortized policy (as opposed to high level planning). For consistency,
we use the same intrinsic reward for HIRO as our method. We consider two other
baselines, named Random Skills that has a hierarchical structure exactly as
LSP but the skills are sampled randomly and there is no planning at the level
of skills, and RandSkillsInit that is similar to Random Skills but does not
include the intrinsic rewards (this is essentially equivalent to Dreamer with
an additional random skill input)  These two baselines are to help understand
the utility of the learned skills. For the transfer experiments, we consider
an additional baseline, a variant of our method that keeps the low-level
policy fixed in the transfer environment. All results are over three random
seeds.

#### Environments.

We consider challenging locomotion environments from DeepMind Control Suite
(Tassa et al., 2018) for evaluation, that require learning walking, running,
and hopping gaits which can be achieved by temporally composing skills. In the
Quadruped GetUp Walk task, a quadruped must learn to stand up from a randomly
initialized position that is sometimes upside down, and walk on a plane
without toppling, while in Quadruped Reach, the quadruped agent must walk in
order to reach a particular goal location. In Quadruped Run, the same
quadruped agent must run as fast as possible, with higher rewards for faster
speed. In the Quadruped Obstacle environments (Fig. 6), the quadruped agent
must reach a goal while circumventing multiple cylindrical obstacles. In
Cheetah Run, and Walker Run, the cheetah and walker agents must run as fast as
possible. In Hopper Hop, a one legged hopper must hop in the environment
without toppling. It is extremely challenging to maintain stability of this
agent.

Figure 4: Single task performance. x-axis represents number of environment
interactions in terms of fraction of 1 Million. Comparison between LSP, and
the baselines Dreamer, HIRO, RandSkillsInp, and Random Skills on a suite of
challenging individual locomotion tasks. Cheetah Pixels, Quadruped Run Pixels,
and WalkerRun Pixels are environments with image-based observations while the
rest have proprioceptive features (states) as observations to the agent.
Higher is better.

###  4.2 Solving single locomotion tasks.

In Figure 4 we evaluate our approach LSP in comparison to the fully amortized
baseline Dreamer, and the Random Skills baseline on a suite of challenging
locomotion tasks. Although the environments have a single task objective, in
order to achieve high rewards, the agents need to learn different walking
gaits (Quadruped Walk, Walker Walk), running gaits (Quadruped Run, Walker
Run), and hopping gaits (Hopper Hop) and compose learned skills appropriately
for locomotion.

From the results, it is evident that LSP either outperforms Dreamer or is
competitive to it on all the environments. This demonstrates the benefit of
the hierarchical skill-based policy learning approach of LSP. In addition, we
observe that LSP significantly outperforms the Random Skills and RandSkillsInp
baselines, indicating that learning skills and planning over them is important
to succeed in these locomotion tasks. In order to tease out the benefits of
hierarchy and partial amortization separately, we consider another
hierarchical RL baseline, HIRO (Nachum et al., 2018), which is a state-of-the-
art HRL algorithm that learns a high level amortized policy. LSP outperforms
HIRO in all the tasks suggesting the utility of temporally composing the
learned skills through planning as opposed to amortizing over them with a
policy. HIRO has not been shown to work from images in the original paper
(Nachum et al., 2018), and so we do not have learning curves for HIRO in the
image-based environments, as it cannot scale directly to work with image based
observations.

###  4.3 Transfer from one task to another.

In Figure 5 we show results for a quadruped agent that is pre-trained on one
task and must transfer to a different task with similar environment dynamics
but different task description.

#### Quadruped GetUp Walk $\rightarrow$ Reach Goal.

The quadruped agent is pre-trained on the task of standing up from a randomly
initialized position that is sometimes upside down, and walking on a plane
without toppling. The transfer task consists of walking to reach a goal, and
environment rewards are specified in terms of distance to goal. The agent is
randomly initialized and is sometimes initialized upside down, such that it
must learn to get upright and then start walking towards the goal. We see that
LSP can adapt much quickly to the transfer task, achieving a reward of
$\leavevmode\nobreak\ 500$ only after $70,000$ steps, while Dreamer requires
$130,000$ steps to achieve the same reward, indicating sample efficient
transfer of learned skills.

We observe that the variant of our method LSP with a fixed policy in the
transfer environment performs as well as or slightly better than LSP. This
suggests that while transferring from the GetUp Walk to the Reach task, low
level control is useful to be directly transferred while planning over high
level skills which have changed is essential. As the target task is different,
so it requires composition of different skills.

Figure 5: Transfer. x-axis represents no. of environment steps in terms of
fraction of 1 Million. Comparison between our method (LSP), and the baselines
Dreamer, HIRO, and a variant of our method that keeps the policy fixed in the
transfer environment. The agents need to transfer to a different locomotion
task, after being pretrained on another task. (Higher is better).

#### Quadruped Walk $\rightarrow$ Reach Goal.

The quadruped agent is randomly initialized, but it is ensured that it is
upright at initialization. In this setting, after pre-training, we re-label
the value of rewards in the replay buffer of both the baseline Dreamer, and
LSP with the reward function of the target Quadruped Reach Goal environment.
To do this, we consider each tuple in the replay buffer of imagined
trajectories during pre-training, and change the reward labels to the reward
value obtained by querying the reward function of the target task at the
corresponding state and action of the tuple. From the plot in Figure 5, we see
that LSP is able to quickly bootstrap learning from the re-labeled Replay
Buffer and achieve better target adaptation than the baseline.

From the figure it is evident that for HIRO, the transfer task rewards
converge at a much lower value than our method LSP and Dreamer, suggesting
that the learned skills by an amortized high level policy overfits to the
source task, and cannot be efficiently adapted to the target task. Similar to
the previous transfer task, we also observe that the variant of our method LSP
with a fixed policy in the transfer environment performs as well as or
slightly better than LSP. This provides further evidence that since the
underlying dynamics of the Quadruped agent is similar across both the tasks,
low level control is useful to be directly transferred while the high level
skills need to be adapted through planning.

(a) Quadruped obstacle environments

(b) Ablation study

Figure 6: (a) Quadruped obstacle environments. Visual illustration of the
environment variants we consider. For the environment on the right, the agent
must reach the goal while navigating across randomly located cylindrical
obstacles while on the left, the agent must walk around a pillar of obstacles
to reach the goal. We provide details and results in the appendix. (b)
Ablation study. x-axis represents no. of environment steps in terms of
fraction of 1 Million. Comparison of dreamer, LSP and the no_MI ablation. In
no_MI, we consider LSP but do not use the mutual information maximizing
intrinsic reward to train the low level policy, keeping other details exactly
the same. Results are on the Quadruped GetUp Walk task. Higher is better.

###  4.4 Mutual Information Ablation study

In order to better understand the benefit of the mutual information skill
objective, we compare performance against a baseline that is equivalent to LSP
but does not use the intrinsic reward to train the low level policy. We call
this ablation baseline that does not have mutual information maximization
between skills and states, as no_MI. We show the respective reward curves for
the Quadruped GetUp Walk task in Figure 6. Without the mutual information
objective, LSP learns less quickly but still faster than the baseline Dreamer.
This emphasizes the necessity of the MI skill objective in section 3.2 and
suggests that merely conditioning the low-level policy on the learned skills
is still effective to some extent but potentially suffers from the low-level
policy learning to ignore them.

###  4.5 Visualization of the learned skills

In Figure 7, we visualize learned skills of LSP, while transferring from the
Quadruped Walk to the Quadruped Reach task. Each sub-figure (with composited
images) corresponds to a different trajectory rolled out from the same initial
state. It is evident that the learned skills are reasonably diverse and useful
in the transfer task.

Figure 7: Qualitative results visualizing some learned behaviors of LSP for
the quadruped agent while transferring from the walking to the goal reaching
task. Each image depicts the agent performing a different skill for 40 time
steps in the environment starting from the same initial state at the origin
depicted by the red spot. We see that the learned skills are diverse,
correspond to reasonable gaits, and help in performing the task. Video
visualizations are in the website https://sites.google.com/view/partial-
amortization-hierarchy/home

##  5 Related Work

#### Skill discovery.

Some RL algorithms explicitly try to learn task decompositions in the form of
re-usable skills, which are generally formulated as temporally abstracted
actions (Sutton et al., 1999). Most recent skill discovery algorithms seek to
maximize the mutual information between skills and input observations (Gregor
et al., 2016; Florensa et al., 2017), sometimes resulting in an unsupervised
diversity maximization objective (Eysenbach et al., 2018; Sharma et al.,
2019). DADS (Sharma et al., 2019) is an unsupervised skill discovery algorithm
for learning diverse skills with a skill-transition dynamics model, but does
not learn a world model for low-level actions and observations, and hence
cannot learn through imagined rollouts and instead requiring many environment
rollouts with different sampled skills.

#### Hierarchical RL.

Hierarchical RL (HRL) (Barto & Mahadevan, 2003) methods decompose a complex
task to sub-tasks and solve each task by optimizing a certain objective
function. HIRO (Nachum et al., 2018) learns a high-level policy and a low-
level policy and computes intrinsic rewards for training the low-level policy
through sub-goals specified as part of the state-representation the agent
observes. Some other algorithms follow the options framework (Sutton et al.,
1999; Bacon et al., 2017), where options correspond to temporal abstractions
that need specifying some termination conditions. In practice, it is difficult
to learn meaningful termination conditions without additional regularization
(Harb et al., 2017). These HRL approaches are inherently specific to the tasks
being trained on, and do not necessarily transfer to new domains, even with
similar dynamics.

#### Transfer in RL.

Multiple previous works have investigated the problem of transferring policies
to different environments. Progressive Networks (Rusu et al., 2016) bootstrap
knowledge from previously learned tasks by avoiding catastrophic forgetting of
the learned models, (Byravan et al., 2020) perform model-based value
estimation for learning an amortized policy and transfer to tasks with
different specified reward functions keeping the dynamics the same, while
Plan2Explore (Sekar et al., 2020) first learns a global world model without
task rewards through a self-supervised objective, and given a user-specified
reward function at test time, quickly adapts to it. In contrast to these,
several meta RL approaches learn policy parameters that generalize well with
little fine-tuning (often in the form of gradient updates) to target
environments (Finn et al., 2017; Xu et al., 2018; Wang et al., 2016; Rakelly
et al., 2019; Yu et al., 2019).

#### Amortization for planning.

Most current MBRL approaches use some version of the ‘Cross-Entropy Method’
(CEM) or Model-Predictive Path Integral (MPPI) for doing a random population
based search of plans given the current model Wang & Ba (2019); Hafner et al.
(2018); Williams et al. (2016); Sharma et al. (2019). These online non-
amortized planning approaches are typically very expensive in high-dimensional
action spaces. Although Wang & Ba (2019) introduces the idea of performing the
CEM search in the parameter space of a distilled policy, it still is very
costly and requires a lot of samples for convergence. To mitigate these
issues, some recent approaches have combined gradient-descent based planning
with CEM Bharadhwaj et al. (2020); Amos & Yarats (2019). In contrast, (Janner
et al., 2019; Hafner et al., 2019) fully amortize learned policies over the
entire training experience, which is fast even for high-dimensional action
spaces, but cannot directly transfer to new environments with different
dynamics and reward functions. We combined the best of both approaches by
using CEM to plan online for high-level skills (of low dimensionality) and
amortize the skill conditioned policy for low-level actions (of higher
dimensionality).

##  6 Discussion

In this paper, we analyzed the implications of partial amortization with
respect to sample efficiency and overall performance on a suite of locomotion
and transfer tasks. We specifically focused on the setting where partial
amortization is enforced through a hierarchical planning model consisting of a
fully amortized low-level policy and a fully online high level skill planner.
Through experiments in both state-based and image-based environments we
demonstrated the efficacy of our approach in terms of planning for useful
skills and executing high-reward achieving policies conditioned on those
skills, as evaluated by sample efficiency (measured by number of environment
interactions) and asymptotic performance (measured by cumulative rewards and
success rate).

One key limitation of our algorithm is that CEM planning is prohibitive in
high-dimensional action spaces, and so we cannot have a very high dimensional
skill-space for planning with CEM, that might be necessary for learning more
expressive/complex skills in real-world robot control tasks. One potential
direction of future work is to incorporate amortized learning of skill
policies during training, and use CEM for online planning of skills only
during inference. Another direction could be to incorporate gradient-descent
based planning instead of a random search procedure as CEM, but avoiding local
optima in skill planning would be a potential challenge for gradient descent
planning.

## Acknowledgement

We thank Vector Institute Toronto for compute support. We thank Mayank Mittal,
Irene Zhang, Alexandra Volokhova, Dylan Turpin, Arthur Allshire, Dhruv Sharma
and other members of the UofT CS Robotics group for helpful discussions and
feedback on the draft.

## Contributions

All the authors were involved in designing the algorithm, shaping the design
of experiments, and in writing the paper. Everyone participated in the weekly
meetings and brainstorming sessions.

Kevin and Homanga led the project by deciding the problem to work on, setting
up the coding infrastructure, figuring out details of the algorithm, running
experiments, and logging results.

Danijar guided the setup of experiments, and helped provide detailed insights
when we ran into bottlenecks and helped figure out how to navigate challenges,
both with respect to implementation, and algorithm design.

Animesh and Florian provided valuable insights on what contributions to focus
on, and helped us in understanding the limitations of the algorithm at
different stages of its development. They also motivated the students to keep
going during times of uncertainty and stress induced by the pandemic.

## References

  * Alemi et al. (2016) Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy.  Deep variational information bottleneck.  _arXiv preprint arXiv:1612.00410_ , 2016.
  * Amos & Yarats (2019) Brandon Amos and Denis Yarats.  The differentiable cross-entropy method.  _arXiv preprint arXiv:1909.12830_ , 2019.
  * Bacon et al. (2017) Pierre-Luc Bacon, Jean Harb, and Doina Precup.  The option-critic architecture.  In _Thirty-First AAAI Conference on Artificial Intelligence_ , 2017\.
  * Barto & Mahadevan (2003) Andrew G Barto and Sridhar Mahadevan.  Recent advances in hierarchical reinforcement learning.  _Discrete event dynamic systems_ , 13(1-2):41–77, 2003.
  * Bharadhwaj et al. (2020) Homanga Bharadhwaj, Kevin Xie, and Florian Shkurti.  Model-predictive planning via cross-entropy and gradient-based optimization.  In _Learning for Dynamics and Control_ , pp. 277–286, 2020.
  * Byravan et al. (2020) Arunkumar Byravan, Jost Tobias Springenberg, Abbas Abdolmaleki, Roland Hafner, Michael Neunert, Thomas Lampe, Noah Siegel, Nicolas Heess, and Martin Riedmiller.  Imagined value gradients: Model-based policy optimization with tranferable latent dynamics models.  In _Conference on Robot Learning_ , pp. 566–589, 2020.
  * Chua et al. (2018) Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine.  Deep reinforcement learning in a handful of trials using probabilistic dynamics models.  In _Advances in Neural Information Processing Systems_ , pp. 4754–4765, 2018.
  * Eysenbach et al. (2018) Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine.  Diversity is all you need: Learning skills without a reward function.  In _International Conference on Learning Representations_ , 2018.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pp. 1126–1135. JMLR. org, 2017.
  * Florensa et al. (2017) Carlos Florensa, Yan Duan, and Pieter Abbeel.  Stochastic neural networks for hierarchical reinforcement learning.  _arXiv preprint arXiv:1704.03012_ , 2017.
  * Grant et al. (2018) Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths.  Recasting gradient-based meta-learning as hierarchical bayes.  _arXiv preprint arXiv:1801.08930_ , 2018.
  * Gregor et al. (2016) Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra.  Variational intrinsic control.  _arXiv preprint arXiv:1611.07507_ , 2016.
  * Haarnoja et al. (2018) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  _arXiv preprint arXiv:1801.01290_ , 2018.
  * Hafner et al. (2018) Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson.  Learning latent dynamics for planning from pixels.  _arXiv preprint arXiv:1811.04551_ , 2018.
  * Hafner et al. (2019) Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi.  Dream to control: Learning behaviors by latent imagination.  _arXiv preprint arXiv:1912.01603_ , 2019.
  * Harb et al. (2017) Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup.  When waiting is not an option: Learning options with a deliberation cost.  _arXiv preprint arXiv:1709.04571_ , 2017.
  * Janner et al. (2019) Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine.  When to trust your model: Model-based policy optimization.  In _Advances in Neural Information Processing Systems_ , pp. 12498–12509, 2019.
  * Konda & Tsitsiklis (2000) Vijay R Konda and John N Tsitsiklis.  Actor-critic algorithms.  In _Advances in neural information processing systems_ , pp. 1008–1014, 2000.
  * Levine (2018) Sergey Levine.  Reinforcement learning and control as probabilistic inference: Tutorial and review.  _arXiv preprint arXiv:1805.00909_ , 2018.
  * Nachum et al. (2018) Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine.  Data-efficient hierarchical reinforcement learning.  In _Advances in Neural Information Processing Systems_ , pp. 3303–3313, 2018.
  * Okada & Taniguchi (2019) Masashi Okada and Tadahiro Taniguchi.  Variational inference mpc for bayesian model-based reinforcement learning.  _arXiv preprint arXiv:1907.04202_ , 2019.
  * Rakelly et al. (2019) Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen.  Efficient off-policy meta-reinforcement learning via probabilistic context variables.  In _International conference on machine learning_ , pp. 5331–5340, 2019.
  * Rubinstein (1997) Reuven Y Rubinstein.  Optimization of computer simulation models with rare events.  _European Journal of Operational Research_ , 99(1):89–112, 1997.
  * Rusu et al. (2016) Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  _arXiv preprint arXiv:1606.04671_ , 2016.
  * Sekar et al. (2020) Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak.  Planning to explore via self-supervised world models.  _arXiv preprint arXiv:2005.05960_ , 2020.
  * Sharma et al. (2019) Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman.  Dynamics-aware unsupervised discovery of skills.  _arXiv preprint arXiv:1907.01657_ , 2019.
  * Sutton & Barto (2018) Richard S Sutton and Andrew G Barto.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Sutton et al. (1999) Richard S Sutton, Doina Precup, and Satinder Singh.  Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning.  _Artificial intelligence_ , 112(1-2):181–211, 1999.
  * Tassa et al. (2018) Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al.  Deepmind control suite.  _arXiv preprint arXiv:1801.00690_ , 2018.
  * Tishby et al. (2000) Naftali Tishby, Fernando C Pereira, and William Bialek.  The information bottleneck method.  _arXiv preprint physics/0004057_ , 2000.
  * Todorov (2008) Emanuel Todorov.  General duality between optimal control and estimation.  In _2008 47th IEEE Conference on Decision and Control_ , pp. 4286–4292. IEEE, 2008.
  * Wang et al. (2016) Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick.  Learning to reinforcement learn.  _arXiv preprint arXiv:1611.05763_ , 2016.
  * Wang & Ba (2019) Tingwu Wang and Jimmy Ba.  Exploring model-based planning with policy networks.  _arXiv preprint arXiv:1906.08649_ , 2019.
  * Wang et al. (2019) Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba.  Benchmarking model-based reinforcement learning.  _arXiv preprint arXiv:1907.02057_ , 2019.
  * Williams et al. (2016) Grady Williams, Paul Drews, Brian Goldfain, James M Rehg, and Evangelos A Theodorou.  Aggressive driving with model predictive path integral control.  In _2016 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 1433–1440. IEEE, 2016.
  * Xu et al. (2018) Zhongwen Xu, Hado P van Hasselt, and David Silver.  Meta-gradient reinforcement learning.  In _Advances in neural information processing systems_ , pp. 2396–2407, 2018.
  * Yu et al. (2019) Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine.  Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.  _arXiv preprint arXiv:1910.10897_ , 2019.

##  Appendix A Appendix

###  A.1 Algorithm

Initialize dataset $\mathcal{D}$ with $S$ random seed episodes. Initialize
neural network parameters $\theta,\phi,\psi$ ;

Define skill duration $K$, plan horizon $H$, CEM population size $G$,
MaxCEMiter, skill noise $\epsilon$;

while  _not converged_ do

for  _update step $c=1..C$_ do

// Model learning

Draw $B$ data sequences
$\\{(a_{t},o_{t},r_{t})\\}_{t=k}^{k+L}\sim\mathcal{D}$. ;

Compute model states $s_{t}\sim p_{\theta}(s_{t}\mids_{t-1},a_{t-1},o_{t})$.
$\mathcal{S}\leftarrow\mathcal{S}\cup\\{s_{t}\\}$;

Update $\theta$ using representation learning. ;

// Behavior learning

$\zeta=(\mu,\Sigma)$ $\leftarrow$CEM($\mathcal{S}$, MaxCEMiter, $G$, $H$, $K$,
$\zeta^{(0)}$); Add noise $\mu=\mu+\epsilon$;

Compute rewards
$R=\mathbb{\operatorname{E}}\big{(}q_{\theta}(r_{\tau}\mids_{\tau})\big{)}$
with the optimized CEM distribution $\zeta$;

Compute the corresponding intrinsic rewards $r^{i}$;

Store the corresponding states into $K-$sized sequences
$\\{\\{s\\}_{k}\\}_{k=1}^{\lceil H/K\rceil}$ (for $\chi$);

Use total rewards $R+r^{i}$ to form value estimates, and update
$\phi,\psi,\chi$

// Environment interaction

$o_{1}\leftarrow\texttt{env.reset()}$ ;

for  _time step $t=0..T-1$_ do

// MPC in z space. Resample skill every K timesteps

if  _$t\%K==0$_ then

Sample skill $z_{1:\lceil H/K\rceil}\sim p(z_{1:\lceil H/K\rceil}|\zeta)$.
Choose the first skill, $z=z_{1}$;

Compute $s_{t}\sim p_{\theta}(s_{t}\mids_{t-1},a_{t-1},o_{t})$ from history,
choose $a_{t}\sim q_{\phi}(a_{t}\mids_{t},z)$;

$r_{t},o_{t+1}\leftarrow\texttt{env.step(}a_{t}\texttt{)}.$ ;

Add experience to dataset
$\mathcal{D}\leftarrow\mathcal{D}\cup\\{(o_{t},a_{t},r_{t})_{t=1}^{T}\\}.$ ;

Algorithm 1 Learning Skills for Planning

###  A.2 Quadruped Obstacle Transfer Experiment

We evaluate the ability of our method to transfer to more complex tasks. Here
the source task is to walk forward at a constant speed in a random obstacle
environment. The policy is trained in this source task for 500k steps before
transferring to the target task which is a pure sparse reward task. The
obstacles are arranged in a cove like formation where the straight line to the
sparse target leads into a local minima, being stuck against the obstacles. To
be able to solve this task, the agent needs to be able to perform long term
temporally correlated exploration. We keep all other settings the same but
increase the skill length $K$ to 30 time steps and skill horizon $H$ to 120
time steps after transfer in the target task to make the skills be held for
longer and let the agent plan further ahead. We see in Figure 10 that the
trajectories explored by Dreamer are restricted to be near the initialization
and it does not explore beyond the obstacles. In contrast, LSP is able to
fully explore the environment and reach the sparse goal multiple times. By
transferring skills, LSP is able to explore at the level of skills and hence
produces much more temporally coherent trajectories.

Figure 8: Quadruped obstacle environment trajectories. Visual illustration of
the 2D root position of the quadruped for the 1st 40k steps after transfer to
the cove environment. On the left is Dreamer and on the right is LSP with
fixed policy. Nodes in the trajectory correspond to beginning state of new
skills in the case of LSP.

Figure 9: Quadruped obstacle environment transfer learning curves. We plot
total rewards against number of transfer training time steps on the sparse
cove environment. Note that after 70k steps (70 episodes), the LSP agent is
able reliably rech the target every time, whereas dreamer does not even come
close to the target, stunted by the sparse reward.

###  A.3 Settings and Hyperparameters

Our method is based on the tensorflow2 implementation of Dreamer (Hafner et
al., 2019) and retains most of the original hyperparameters for policy and
model learning. Training is performed more frequently, in particular 1
training update of all modules is done every 5 environment steps. This
marginally improves the training performance and is used in the dreamer
baseline as well as our method. For feature-based experiments we replace the
convolutional encoder and decoder with 2-layer MLP networks with 256 units
each and ELU activation. Additionally, the dense decoder network outputs both
the mean and log standard deviation of the Gaussian distribution over
observations. The standard deviation is softly clamped between 0.1 and 1.5 as
described in Chua et al. (2018).

For LSP, skill vectors are 3-dimensional and are held for $K=10$ steps before
being updated. The CEM method has a planning horizon of $H=10$, goes through
$MaxCEMiter=4$ iterations, proposes $G=16$ skills and uses the top $M=4$
proposals to recompute statistics in each iteration. The additional noise
$\epsilon$ added to the CEM optimized distribution is Normal(0, 0.1).

The backwards skill predictor shares the same architecture and settings as the
feature-based decoder module described above. It is trained with Adam with
learning rate of $8e-5$.

###  A.4 Control as Inference

The control as inference framework Todorov (2008); Levine (2018) provides a
heuristic to encourage exploration. It performs inference in a surrogate
probabilistic graphical model, where the likelihood of a trajectory being
optimal (an indicator random variable $\mathcal{O}$) is a hand-designed
(monotonically increasing) function of the reward, $\log
p(\mathcal{O}|\tau)=f(r)$. The induced surrogate posterior
$p(\tau|\mathcal{O})$ places higher probability on higher reward trajectories.

Sampling from $p(\tau|\mathcal{O})$ is in general intractable and so a common
solution is to employ variational inference for obtaining an approximation
$q_{\theta}(\tau)$. The objective is to minimize the $\mathbb{KL}$-divergence
with respect to the true posterior:

$\displaystyle\min_{\theta}\mathbb{KL}(q_{\theta}(\tau)||p(\tau|\mathcal{O}))$
$\displaystyle=\max_{\theta}-\mathbb{KL}(q_{\theta}(\tau)||p(\tau|\mathcal{O}))$
(2) $\displaystyle=\max_{\theta}\mathbb{E}_{q_{\theta}(\tau)}[\log
p(\mathcal{O}|\tau)-\log q_{\theta}(a)]$ (3)
$\displaystyle=\max_{\theta}\mathbb{E}_{q_{\theta}(\tau)}[f(r)]+\mathcal{H}[q_{\theta}(a)]$
(4)

Note that this objective boils down to maximizing a function of the reward and
an entropy term $\mathcal{H}(\cdot)$.

State of the art model-free reinforcement learning approaches can be
interpreted as instances of this framework Haarnoja et al. (2018). The
connection has also been made in model-based RL Okada & Taniguchi (2019).
Specifically they show how the ubiquitous MPPI/CEM planning methods can be
derived from this framework when applying variational inference to simple
Gaussian action distributions. Though they show that more sophisticated
distributions can be used in the framework as well (in principle), the only
other model they investigate in the paper is a mixture Gaussians distribution.

###  A.5 Hierarchical Inference

We consider hierarchical action distributions that combine amortizing low
level behaviours and online planning by temporally composing these low level
behaviours. Specifically we use the following variational distribution:

$q_{\phi,\theta}(\tau)=\prod_{t=1}^{T}p(s_{t+1}|s_{t},a_{t})q_{\phi}(a_{t}|s_{t},z_{k(t)})\prod_{k=1}^{K}q_{\theta_{i}}(z_{k})$

Here $z_{1:K}$ are latent variables defining a high level plan that modulates
the behaviour of the low level policy $q_{\phi}(a_{t}|s_{t},z_{k(t)})$. $k(t)$
is an assignment defining which of the $K$ $z$’s will act at time $t$. For all
our experiments, we chose a fixed size window assignment such that
$k(t)=\lfloor t(K/T)\rfloor$.

Here $p(\mathcal{T})$ is the task distribution over which we wish to amortize.
For example using $p(s_{0})$ would amortize over plans starting from different
initial states. Note that the minimization over $\phi$ is outside the
expectation which means that it is amortized, whereas the minimization over
$\theta$ is inside such that it is inferred online.

$\displaystyle\min_{\phi}\mathbb{E}_{p(\mathcal{T})}[\min_{\theta}KL(q_{\theta}(\tau)||p(\tau|\mathcal{O}))]$
(5)
$\displaystyle=\max_{\phi}\mathbb{E}_{p(\mathcal{T})}[\max_{\theta}\mathbb{E}_{q_{\phi,\theta}(\tau)}[\log
p(\mathcal{O}|\tau)p_{0}(\mathbf{z})-\log
q_{\phi}(\mathbf{a}|\mathbf{s},\mathbf{z})q_{\theta_{i}}(\mathbf{z})]]$ (6)

This formulation also directly relates to some meta-learning approaches that
can be interpreted as hierarchical Bayes Grant et al. (2018). There are
different options for the inner and outer optimization in this type of meta-
optimization.

###  A.6 Note on Mutual Information

In the paper we used the reverse skill predictor approach to estimate the
mutual information objective. The alternate decomposition in terms of
reduction in future state uncertainty is given by:

$\mathcal{H}(\\{s\\}|s_{0})-\mathcal{H}(\\{s\\}|z,s_{0})=\int
p(z,\\{s\\},s_{0})\log\frac{p(\\{s\\}|z,s_{0})}{p(\\{s\\}|s_{0})}$

In this case tractable approximations of the the skill dynamics
$p(s^{\prime}|z,s_{0})$ and marginal dynamics $p(s^{\prime}|s_{0})$ are
required.

In theory, these can be formed exactly as compositions of the policy, skill
selection distribution and dynamics model:

$\log p(\\{s\\}|z,s_{0})=\log\prod_{t}p(s_{t+1}|s_{t},a)\pi(a|s_{t},z)$ $\log
p(\\{s\\}|s_{0})=\log\mathbb{E}_{z^{\prime}|s_{0}}\left[p(\\{s\\}|z^{\prime},s_{0})\right]$

The difficulty in using this formulation in our case is due to the marginal
future state distribution $p(\\{s\\}|s_{0})$. To form the marginal we rely on
a finite number $K$ monte carlo samples which is biased and our lower bound on
mutual information will be reduced by $\log K$. This means that for every
given initial state we wish to train on, we must also sample a sufficiently
large number of skills from the skill selection distribution to form an
acceptable marginal approximation, greatly increasing the computation cost
compared to using the reverse skill predictor.

###  A.7 Mutual Information Skll Predictor Noise

Although mutual information is a popular objective for skill discovery, it is
important to understand its limitations. When the underlying dynamics of the
system are deterministic, a powerful world model may be able to make future
predictions with high certainty. Consider if the agent is then able to
communicate the skill identity to the skill predictor through a smaller
subspace of the state space. For example, imagine that in a quadruped walking
task the height of a specific end effector at a certain future frame can be
very confidently predicted by the dynamics model. Whether to lift the end
effector up 9cm or 10cm may be largely negligible in terms of the dynamics of
the motion, but the policy may elect to map the skill variable to the heights
of a specific end effector at a certain future frame. When this occurs,
diverse skills can become quite indistinguishable visually yet be very
distinguishable by the skill predictor. In this failure case, the mutual
information may be minimized without truly increasing the diversity of the
motion. We use a simple idea to combat this by artificially adding noise to
the input of the skill predictor. The higher the skill predictor input noise
is, the harder it is to distinguish similar trajectories which forces the
skills to produce more distinct outcomes.

Figure 10: Example of collapsed skill policy. Here we show skill sequences
sampled from a policy where input noise is not added to the skill predictor
and subsequently the skill policy seems to perform the same motion for
different skills (turning clockwise), despite inclusion of the mutual
information objective. Each row is a different latent variable.

Function  _CEM(_$\mathcal{S}$ , MaxCEMiter, $G$, $H$, $K$, $\zeta^{(0)}$_)_:

Sample $G$ initial skills $\\{z_{1:\lceil H/K\rceil}^{(g)}\\}_{g=1}^{G}$ from
prior $p(z|\zeta^{(0)})$

for  _j in range(MaxCEMiter)_ do

for  _g in range(G)_ do

for  _each $s_{t}$_ do

$\tau=t$ // Imagine trajectories $\\{(s_{\tau},a_{\tau})\\}_{\tau=t}^{t+H}$
from each $s_{t}$.

while  _$\tau <H$, $\tau++$_ do

if  _$(\tau-t)\%K==0$_ then

Sample skill $z_{1:\lceil H/K\rceil}\sim p(z_{1:\lceil
H/K\rceil}|\zeta^{(j)})$

Sample action $a_{\tau}\sim q_{\phi}(a_{\tau}\mids_{\tau},z_{1})$ // Choose
$z_{1}$ via MPC

Observe new state $s_{\tau+1}$ and compute new intrinsic reward $r_{i}$

Compute value estimates $V_{g}$ from rewards of each imagined rollout

Sort $\\{V_{g}\\}_{g=1}^{G}$ and use the top M sequences to update
$\zeta^{(j)}=(\mu^{(j)},\Sigma^{(j)})$

Sample $G$ skills $\\{z^{(g)}\\}_{g=1}^{G}$ from $p(z|\zeta^{(j)})$

return $\zeta^{(\text{MaxCEMiter})}$

Algorithm 2 CEM subroutine for Algorithm 1


# Reversibility in the Seifert-fibered spaces

Anushree Das and Debattam Das  Indian Institute of Science Education and
Research (IISER) Mohali, Knowledge City, Sector 81, S.A.S. Nagar 140306,
Punjab, India<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract.

An element $a$ in a group $\Gamma$ is called _reversible_ if there exists
$g\in\Gamma$ such that $gag^{-1}=a^{-1}$. The reversible elements are also
known as ‘real elements’ or ‘reciprocal elements’ in literature. In this
paper, we classify the reversible elements in Fuchsian groups, and use this
classification to find all reversible elements in a Seifert-fibered group. In
the last section we apply this classification to the braid groups,
particularly to the braid group on $3$ strands.

###### Key words and phrases:

Reversible elements , Fuchsian groups, surface groups, Seifert fibered space,
Braid groups

###### 2020 Mathematics Subject Classification:

Primary 57K; Secondary 20H10, 20H05

##  1\. Introduction

An element $a$ in a group $\Gamma$ is said to be ‘reversible’ if it is
conjugate to its own inverse, and the conjugating elements are called
‘reverser’ or ‘reversing symmetry’. If the conjugating element is also an
involution, then it is called ‘strongly reversible’ element. It has also been
called by other terms like ‘reciprocals’ or ‘real elements’ in other
literature, e.g., [BR], [La] etc. In this paper our discussion is limited to
some properties of reversible elements, and a discussion of reversible
elements in some discrete geometric groups. Reversibility has its own
geometric motivations and has been discussed widely in literature. For
instance, in a Fuchsian group, the collection of conjugacy classes of
reversible elements have a 1-1 correspondence to the conjugacy classes of
infinite dihedral groups contained in that Fuchsian group. But in general,
supposing $G$ is a 3-manifold group (the fundamental group of some
3-manifold), if $G$ is torsion free then a reversible element $g$ in $G$ is
represented as a loop which is homotopic to its own inverse (i.e., it will
form an embedded Klein bottle with the reversing symmetry). So, it can be
considered that finding a reversible element with a reverser in a 3-manifold
is equivalent to finding an embedded Klein bottle upto homotopy inside the
manifold. In this paper we are specially interested in Seifert fibered
manifolds since the fundamental group of a Seifert fibered manifold is a
universal extension of Fuchsian groups or non-orientable 2-orbifold groups.

A Seifert fibered manifold is a compact $3$-manifold which can be seen as a
$S^{1}$ bundle over a $2$-dimensional orbifold as the base space, such that
every fiber has a solid fibered torus neighbourhood. These spaces were first
studied by Herbert Seifert in his doctoral thesis [ST], and they form an
important example of $3$-manifolds. In fact, all the the compact orientable
$3$-manifolds in $6$ of the $8$ possible $3$-manifold geometric structures can
be seen as Seifert fibered spaces. If we consider the space of fibers of $M$
with a quotient topology, we get a resulting surface called the orbit surface
of $M$. We denote the orbit surface by $G$. The orbit surface can also be seen
as a quotient of the base space of $M$. There is a map
$\phi:\pi_{1}(G)\to\\{-1,1\\}$ called the classifying homomorphism. A loop
$\alpha$ on $G$ has image $\phi(\alpha)=1$ if the fiber above $\alpha$ is
mapped to itself in an orientation preserving way as we follow the loop
$\alpha$, and has image $\phi(\alpha)=-1$ otherwise. The classifying
homomorphism plays a crucial role in determining the fiber type of a Seifert
fibered space.

Following Seifert’s work and Brin’s note in [Br], the closed Seifert fibered
spaces can be completely classified based on certain symbols. If $M$ is a
closed and oriented Seifert fibered space, then $M$ is uniquely determined by
a symbol

$(O,o,g|b,\mu_{1},\beta_{1},\dots,\mu_{n},\beta_{n})$

or

$(O,n,k|b,\mu_{1},\beta_{1},\dots,\mu_{n},\beta_{n}).$

Here $O$ refers to the orientability of $M$, $o$ and $n$ refer to the
orientability or non-orientability of the base space respectively, $g$ is the
genus of the orientable base space, and $k$ is the number of crosscaps of the
non-orientable base space. $b$ is an integer depending on the fibration of the
space, and the $\mu_{i}$ and $\beta_{i}$ depend on the fiber type of the
$i^{th}$ essential fiber. If $M$ is non-orientable, then there are $4$
possible symbols associated:

${(N,o,g|b,\mu_{1},\beta_{1},\dots,\mu_{n},\beta_{n})}$

if the base space is orientable, and otherwise, one of the follwing $3$ types
based on the classifying homomorphism of the base space:

$\displaystyle(N,n,\textrm{I},k|b,\mu_{1},\beta_{1},\dots,\mu_{n},\beta_{n});$
$\displaystyle(O,n,\textrm{II},k|b,\mu_{1},\beta_{1},\dots,\mu_{n},\beta_{n});$
$\displaystyle(O,n,\textrm{III},k|b,\mu_{1},\beta_{1},\dots,\mu_{n},\beta_{n}).$

Given any such Seifert fibered space, this paper is concerned with checking
the reversibility of the elements of $\pi_{1}(M)$, the fundamental group of
$M$. The fundamental group of $M$ depends on the orientability of the orbit
surface $G$. If $G$ is orientable, it can be shown that the fundamental group
of $M$ is of the form

(1.1) $\displaystyle\pi_{1}(M)=$ $\displaystyle\langle
a_{i},b_{i},c_{i},d_{i},h|a_{i}ha_{i}^{-1}=h^{\phi(a_{i})},$ (1.2)
$\displaystyle
b_{i}hb_{i}^{-1}=h^{\phi(b_{i})},c_{i}hc_{i}^{-1}=h,d_{i}hd_{i}^{-1}=h^{\phi(d_{i})},$
(1.3) $\displaystyle c_{i}^{\mu_{i}}=h^{\beta}_{i},\Pi[a_{i},b_{i}]\Pi
c_{i}\Pi d_{i}h^{b}=1\rangle.$

If $G$ is not orientable, $\pi_{1}(M)$ has the presentation

(1.4) $\displaystyle\pi_{1}(M)=$ $\displaystyle\langle
x_{i},c_{i},d_{i},h|x_{i}hx_{i}^{-1}=h^{\phi(x_{i})},c_{i}hc_{i}^{-1}=h,d_{i}hd_{i}^{-1}=h^{\phi(d_{i})},$
(1.5) $\displaystyle c_{i}^{\mu_{i}}=h^{\beta_{i}},\Pi x_{i}^{2}\Pi c_{i}\Pi
d_{i}h^{b}=1\rangle.$

In both cases the $\mu_{i}$, $\beta_{i}$ and $b$ are the same as used in the
representation of $M$, and $h$ represents an ordinary fiber. The $a_{i}$ and
$b_{i}$ are the generators of the handlebodies of the orbit surface, the
$d_{i}$ represent the boundaries of the orbit surface, the $x_{i}$ generate
the crosscaps in the orbit surface, and the $c_{i}$ are the generators of the
essential fibers. $\phi$ is the classifying homomorphism. $\langle h\rangle$
is a normal cyclic subgroup of $\pi_{1}(M)$, and so we can consider the
quotient group $\pi_{1}(M)/\langle h\rangle$. If the orbit surface $G$ is
orientable, this quotient is a Fuchsian group which can be written as

$\displaystyle\pi_{1}(M)/\langle h\rangle=$ $\displaystyle\langle
a_{i},b_{i},c_{i},d_{i}|c_{i}^{\mu_{i}}=1,\Pi[a_{i},b_{i}]\Pi c_{i}\Pi
d_{i}=1\rangle.$

If $G$ is non-orientable, then

$\displaystyle\pi_{1}(M)/\langle h\rangle=$ $\displaystyle\langle
x_{i},c_{i},d_{i}|c_{i}^{\mu_{i}}=1,\Pi x_{i}^{2}\Pi c_{i}\Pi d_{i}=1\rangle.$

From the representation of $\pi_{1}(M)$, we can see that the reversibility of
the element $h$ is dependent on the classifying homomorphism $\phi$. If there
is any generator $z$ of the form $a_{i},b_{i},x_{i},$ or $d_{i}$ such that
$\phi(z)=-1$, then the relations in the definition of $\pi_{1}(M)$ give that
$h$ is reversible, i.e., $zhz^{-1}=h^{\phi(z)}=h^{-1}$. Our aim is to classify
all the reversible elements of $\pi_{1}(M)$. The main result of our paper is:

######  Theorem 1.1.

If $M$ has an orientable orbit surface and $\phi$ is nontrivial, the
reversible elements of $\pi_{1}(M)$ are the elements conjugate to $h$ and
$c_{i}^{\mu_{i}}$ for all $i$ and the conjugates of the elements of either of
the form $c_{i}^{\frac{\mu_{i}}{2}}kc_{j}^{\frac{\mu_{j}}{2}}k^{-1}$ with
$\phi(k)=-1$ or of the form
$c_{i}^{\frac{\mu_{i}}{2}}kc_{j}^{\frac{-\mu_{j}}{2}}k^{-1}$ with $\phi(k)=1$,
where $\mu_{i}$ and $\mu_{j}$ are even, and $k$ is any element in
$\pi_{1}(M)$, $\beta_{i}=\beta_{j}$. $\pi_{1}(M)$ has reversible elements as
the elements conjugate to
$c_{i}^{\frac{\mu_{i}}{2}}kc_{j}^{\frac{-\mu_{j}}{2}}k^{-1}$ if $\phi$ is
trivial.

We also similarly classify the reversible elements in case of Seifert fibered
spaces with non-orientable orbit surfaces.

In Section 2, we discuss about the reversible elements of fundamental groups
of two-dimensional hyperbolic orbifolds, and we classify the reversible
elements of such groups. This classification is then applied to the
fundamental groups of any Seifert-fibered space in Section 3. Section 4 then
discusses an application of the results of Section 3 on the fundamental group
of $B_{3}$, the braid group on $3$ strands. The possibility of using the same
methods for any $B_{n}/\langle h\rangle$ is also discussed there.

After completion of the first draft, the authors were informed by Prof.
Masakazu Teragaito regarding the article [HMT] written by Himeno, Motegi, and
Teragaito, where they classify 3-manifolds whose fundamental groups admit
generalised torsion elements of order two, and further classify such elements.
Their proof is topological and completely different from the proof in this
paper. The two articles were written completely independently and at around
the same time. The proof in [HMT] crucially uses important results by Jaco-
Shalen and Hass, while the techniques in this article are simpler by
comparison.

### Acknowledgment

The authors would like to acknowledge and thank Prof. Masakazu Teragaito for
his helpful comments and for pointing out an error in the first draft. The
authors further thank Prof. Krishnendu Gongopadhyay and Dr. Soma Maity for
helpful feedback. A. Das acknowledges full support from an NBHM research
fellowship during this work. D. Das acknowledges full support from a UGC-NFSC
research fellowship during this work.

##  2\. Reversiblity in Fundamental groups of the surfaces

###  2.1. Reversible classes of Fuchsian group

In [DG2, Remark 1], we have seen the classifications of the $z$-classes for
hyperbolic elements. In the same way, we can see the classifications of
reciprocal elements for any finite type Fuchsian groups. We have discussed the
reciprocal elements of Hecke groups in [DG1], which is a particular class of
Fuchsian groups.

From now on, the hyperbolic axis for an hyperbolic element $h$ is written as
$I_{h}$.

######  Theorem 2.1.

Let $\Gamma$ be an arbitrary finitely generated Fuchsian group. Let
$g\in\Gamma$. Then the following are equivalent.

  1. (1)

$g$ is reciprocal.

  2. (2)

$g$ is either an involution or conjugate to a hyperbolic element $h$ such that
the hyperbolic axis of $h$ passes through some of the even order vertices of
the fundamental domain of the group $\Gamma$ .

To prove this theorem, we first need to prove the following lemmas.

######  Lemma 2.2.

If a hyperbolic element $h$ in the Fuchsian group $\Gamma$ is reciprocal, then
the fixed point of the reciprocator of the reciprocal elements lies on the
hyperbolic axis of $h$, i.e., on $I_{h}$.

Proof. $\Gamma$ is in $\rm PSL(2,\mathbb{R})$, and so we will conjugate
$\Gamma$ by a Möbius map $f$ such that in the new group after conjugation, $h$
will be a hyperbolic element in the Poincaré disk where the hyperbolic axis of
$h$ passes through $(0,0)$. Now, the reciprocator for $h$ is $g$, which is an
involution having fixed point $z$. So, let us suppose $z$ does not lie on
$I_{h}$ and $I_{h}$ is a straight line passing through the origin. Then, $z$
lies one side of the axis. In our earlier work [DG1, Lemma 2.2] we have shown
that in the Fuchsian group, the reciprocators of a reciprocal element fix the
hyperbolic axis of that particular hyperbolic element and interchange the
fixed points of hyperbolic axis. Now, we apply the map $g$ to the disk. This
map is a $\pi$-rotation along $I_{h}$ of the disk, and therefore the point $z$
must change its position. This is contradiction. So, the fixed point of $g$
must lie on $I_{h}$. ∎

######  Lemma 2.3.

A hyperbolic element $g$ in the Fuchsian group $\Gamma$ is reciprocal if and
only if it passes through some of the even-order vertices of the tessellation
which is done by the fundamental domain of $\Gamma$ .

Proof. We first show that if $g$ is reciprocal, then $I_{g}$ passes through at
least one even order vertex in the fundamental domain. First, the fixed points
of the reciprocator of $g$ will always be a vertex of the domain. From [DG2,
Theorem 3], we obtain that $I_{g}$ passing through a fixed domain represents a
$z$-class of that Fuchsian group. And if it is reciprocal then there is a
fixed point of one of the reciprocator of $g$ that lies on $I_{g}$ by 2.2, and
it is an even ordered-vertex of the fundamental domain (from the definition of
fundamental domain). This proves one side of the statement.

Conversely, if the axis of the primitive hyperbolic element $g$ passes through
an even-order vertex $z$ of the tiling which is done by the fundamental
domain, then let $k$ be the elliptic finite order element in the Fuchsian
group such that, $k(z)=z$ and $k^{2m}=Id$. So, $k^{m}$ is an involution and it
is a is $\pi$-rotation. Then, it will interchange the fixed points of $g$, and
so $k^{m}gk^{m}=g^{-1}$ since $g$ is primitive. This shows that any non-
trivial power of $g$ is reciprocal. This proves the lemma.

∎

###  2.2. Proof of Theorem 2.1

Combining Lemma 2.2 and Lemma 2.3, we get the proof of the theorem. ∎

###  2.3. Reversible classes of non-orientable surface fundamental groups

Let $S$ be a compact connected non-orientable surface. Due to the
classification of surfaces, the fundamental group of $S$ can be presented in
the form

(2.1) $\displaystyle\langle
x_{1},x_{2},\dots,x_{k},d_{1},d_{2},\dots,d_{m}|\Pi x^{2}_{i}\Pi
d_{i}=1\rangle$

where $k$ represents number of cross-caps and the $d_{i}$ represent the
boundary components. In such cases, we have the following lemma.

######  Lemma 2.4.

If $S$ is a compact, connected non-orientable surface having the fundamental
group 2.1, then there are no reversible elements in $\pi_{1}(S)$ except when
$S$ is the Klein-bottle or $\mathbb{RP}^{2}$.

Proof. By the classification of non-orientable surfaces, we know that $S$ is
homeomorphic to
$\underbrace{\mathbb{RP}^{2}\\#\mathbb{RP}^{2}\\#\dots\\#\mathbb{RP}^{2}}_{k\
times}$. We also know that a loop $l$ is reversible iff it will freely
homotope to its own inverse, that is, they will form a Klein bottle. So, if
$S$ is itself a Klein bottle, then any power of the generating loop is
reversible. If $S$ is $\mathbb{RP}^{2}$, then the fundamental group will be
$\mathbb{Z}_{2}$, and the generator is reversible.

If $k>2$, there is no finite order element in the fundamental group. Then our
claim is that there are no more reversible elements in $S$. If on the contrary
there exists any loop $l$ that homotopes to its own inverse, then that element
along with the reverser will form a Klein bottle which is a closed surface (if
the reverser is infinite order). That implies, there is an embedding of the
Klein bottle in $S$. Since both $S$ and the Klein bottle have same dimension
and are both complete surfaces, therefore by the Hopf–Rinow theorem, they must
be homeomorphic. This contradicts our hypothesis and hence proves the lemma.

∎
Now we classify the reversible elements in the fundamental group of a non-
orientable two-dimensional orbifold $\Gamma^{\prime}$ which is neither
$\mathbb{RP}^{2}$ nor Klein bottle.

######  Lemma 2.5.

Infinite order reversible elements in $\Gamma^{\prime}$ are strongly
reversible elements.

The proof is similar to the proof of [DG1, Lemma 3.4 ].
Proof. The group $\Gamma^{\prime}$ is a discrete subgroup of $\rm
PGL(2,\mathbb{R})$. Let $r$ in ${\bf H}^{2}/\Gamma^{\prime}$ be a reversible
element in $\Gamma^{\prime}$ . Then there is an element $b$ in
$\Gamma^{\prime}$ such that $b^{-1}rb=r^{-1}$. At first we consider that $r$
is orientation preserving isometry in ${\bf H}^{2}$ (which implies that will
not be a parabolic element). Then, $b$ either fixes the fixed points of $r$ or
reflects the fixed points to each other. If $b$ is fixes the fixed points of
$r$ then it will either commute with $r$ or reverse $r$. In the case where it
commutes with $r$ it leads us to a contradiction that $b$ is not a reverser,
and in the case where it is reversing, $b$ will be an involution. If $b$
interchanges the fixed points, then $b$ must be an involution.

In $r$ is a orientation reversing element, then $r^{2}$ will be orientation
preserving element with the same reverser $b$. From the previous case,
$b^{2}=1$. Hence, in any case, $r$ will be strongly reversible. ∎

######  Theorem 2.6.

The infinite ordered reversible elements in $\Gamma^{\prime}$ are conjugate to
$c_{i}^{\frac{\mu_{i}}{2}}kc_{j}^{\frac{\mu_{j}}{2}}k^{-1}$ where $\mu_{i}$
and $\mu_{j}$ are even and $k\in\Gamma^{\prime}$.

Proof. Since there is no finite order element in $\Gamma^{\prime}$ other than
the $c_{i}$, this theorem follows from the previous lemma. ∎

##  3\. Seifert fibered space

To find the reversible elements of the fundamental group of a Seifert fibered
space, we first focus on the quotient group $\pi_{1}(M)/\langle h\rangle$ and
discuss its reversibility. We first deal with those spaces which have
orientable orbit surfaces.

######  Corollary 3.1.

Any element conjugate to either $c_{i}^{\frac{\mu_{i}}{2}}$ or
$c_{i}^{\frac{\mu_{i}}{2}}kc_{j}^{\frac{\mu_{j}}{2}}k^{-1}$ where $k\
\in\pi_{1}(M)/\langle h\rangle$ is reversible in $\pi_{1}(M)/\langle h\rangle$
iff $\mu_{i}$ and $\mu_{j}$ are even. No other elements in $\pi_{1}(M)/\langle
h\rangle$ are reversible.

Proof. Since $\pi_{1}(M)/\langle h\rangle$ is a Fuchsian group $\Gamma$, all
reversible elements in $\Gamma$ are strongly reversible. By [DG1, Theorem
1.1], it is known that an element in a Fuchsian group is reversible if and
only if it is an involution or a product of $2$ involutions. By the
representation of the group, the involutions are the elements conjugate to any
$c_{i}^{\frac{\mu_{i}}{2}}$ where $\mu_{i}$ is even. Hence, the claim follows.
∎

Now we can discuss the reversibility of the fundamental group of $M$. Since
there is no finite order element in $\pi_{1}(M)$, we cannot talk about
strongly reversible elements here. But there may be reversible elements, as
stated earlier. Not that if an element is reversible, then so are its
conjugates.

######  Lemma 3.2.

Elements conjugate to the power of the fiber generator $h$ is a reversible
element iff the classifying homomorphism on $\pi_{1}(G)$ is non-trivial.

Proof. If an element in $\pi_{1}(M)$ is reversible then it must be free
homotopic to its inverse. This free homotopy forms an embedded Klein bottle in
$M$. Let $p:\pi_{1}(M)\to\pi_{1}(G)$ be the projection map from $M$ to the
orbit space. If there is a homotopy class of a loop $z$ that is generated by
the generating elements of $\pi_{1}(M)$ and has a non-trivial part which cuts
the fiber $h$ transitively, then $p(z)$ is a loop in the orbit space. So $h$
is reversible iff the image of $p(z)$ under $\phi$ is non-trivial. Since the
image of $\phi$ of $z$ depends on image of $\phi$ of generating set elements,
this implies the lemma.

Alternatively, since $\phi$ is a homomorphism, it is determined by its actions
on the generators of $\pi_{1}(G)$. If $\phi$ is non-trivial, there is some
generator $x$ of the form $a_{i}$, $b_{i}$, $x_{i}$ or $d_{i}$ such that
$\phi(x)=-1$. Then by the presentation of $\pi_{1}(M)$,
$xhx^{-1}=h^{\phi(x)}=h^{-1}$, and so $h$ is reversible.

∎

######  Corollary 3.3.

If $h$ is reversible in $\pi_{1}(M)$, the $c^{\mu_{i}}_{i}$ are reversible
elements although each $c_{i}$ is an infinite order element in $\pi_{1}(M)$.

Proof. Let $h$ be reversible. Then so is $h^{\beta_{i}}$. The relation
$c_{i}^{\mu_{i}}=h^{\beta_{i}}$ now implies that $c_{i}^{\mu_{i}}$ is also
reversible. ∎

The two following lemmas investigate the reversibility of the other elements
of $\pi_{1}(M)$.

######  Lemma 3.4.

The powers of $c^{\mu_{i}}_{i}$ are reversible, but any other multiple of
$c_{i}$ is not reversible for any $\pi_{1}(M)$.

Proof. On the contrary, let a multiple of $c_{i}$ be a reversible element in
$\pi_{1}(M)$ with reverser $\rho$. We can consider that $\rho$ is not a power
of $h$, since $h$ is contained in the centralizer of $c_{i}$. Then, the
reverser $\rho$ is transverse to $h$, and this implies any power of $\rho$
will always be transverse to $h$. Hence $\rho$ has a non-trivial image in the
group $\pi_{1}(M)/\langle h\rangle$, denoted by $[\rho]$. Since, from [DG1],
every reversible element in a Fuchsian group is strongly reversible, $[\rho]$
must be of order $2$. This implies, $\rho^{2}=h^{k}$ for some
$k\in\mathbb{Z}$, which contradicts our assumption. So, there is no such
reverser that reverses any other power of $c_{i}$ except for the powers of the
$c^{\mu_{i}}_{i}$. ∎

######  Lemma 3.5.

If $M$ has an orientable orbit surface, the only reversible elements in
$\pi_{1}(M)$ can be the conjugates of the powers of $h$ and the conjugates of
the elements either of the form
$c_{i}^{\frac{\mu_{i}}{2}}kc_{j}^{\frac{\mu_{j}}{2}}k^{-1}$ with $\phi(k)=-1$
or of the form $c_{i}^{\frac{\mu_{i}}{2}}kc_{j}^{\frac{-\mu_{j}}{2}}k^{-1}$
with $\phi(k)=1$, where $\mu_{i}$ and $\mu_{j}$ are even, and $k$ is any
element in $\pi_{1}(M)$, $\beta_{i}=\beta_{j}$.

Proof. Let $x$ be an element in $\pi_{1}(M)$, other than any power of $h$. Let
$x$ be reversible. Then $x$ is freely homotopic to $x^{-1}$. Consider the
projection of this free homotopy to the Fuchsian group $\pi_{1}(M)/\langle
h\rangle$. This would be a free homotopy of a loop in this Fuchsian complex to
its inverse. But this would imply that this is a reversible element in
$\pi_{1}(G)/\langle h\rangle$. But as we have shown earlier, the reversible
elements in this group are of the form of conjugates of
$c_{i}^{\frac{\mu_{i}}{2}}kc_{j}^{\frac{\mu_{j}}{2}}k^{-1}$ or
$c_{i}^{\frac{\mu_{i}}{2}}$ for even $\mu_{i}$ and $\mu_{j}$. But, for
elements that are conjugates of $c_{i}^{\frac{\mu_{i}}{2}}$, i.e., $\alpha
c^{\frac{\mu_{i}}{2}}\alpha^{-1}$, the reverser of $x$ must be $x$ itself.
Then the element in $x\in\pi_{1}(M)$ will satisfy,

$\alpha c^{\frac{\mu_{i}}{2}}\alpha^{-1}\alpha
c^{\frac{\mu_{i}}{2}}\alpha^{-1}\alpha
c^{\frac{-\mu_{i}}{2}}\alpha^{-1}=\alpha c^{\frac{-\mu_{i}}{2}}\alpha^{-1}$
$\Rightarrow h^{\beta_{i}}=1.$

Since $h$ is an infinite order element, this is a contradiction. Hence, the
lifts of the elements conjugate to $c^{\frac{\mu_{i}}{2}}$ are not reversible
in $\pi_{1}(M)$. So, the only other candidates from $\pi_{1}(M)/\langle
h\rangle$ that may be reversible in $\pi_{1}(M).$ are the products of the
involutions.
Let the element $x=c_{i}^{\frac{\mu_{i}}{2}}kc_{j}^{\frac{\mu_{j}}{2}}k^{-1}$
be reversible in $\pi_{1}(M)/\langle h\rangle$ with reverser
$c^{\frac{\mu_{i}}{2}}.$ Then in $\pi_{1}(M),$

$c_{i}^{\frac{\mu_{i}}{2}}c_{i}^{\frac{\mu_{i}}{2}}kc_{j}^{\frac{\mu_{j}}{2}}k^{-1}c_{i}^{-\frac{\mu_{i}}{2}}c_{i}^{\frac{\mu_{i}}{2}}kc_{j}^{\frac{\mu_{j}}{2}}k^{-1}=c_{i}^{\mu_{i}}kc_{j}^{\frac{\mu_{j}}{2}}k^{-1}kc_{j}^{\frac{\mu_{j}}{2}}k^{-1}=h^{\beta_{i}}kh^{\beta_{j}}k^{-1}.$

Then, $h^{\beta_{i}}kh^{\beta_{j}}k^{-1}$ will be identity only if
$\beta_{i}=\beta_{j}$ and $\phi(k)=-1.$ It will be same if we choose any other
reverser for that particular reversible element since all the reversers are in
the form of $x^{l}c^{\frac{\mu_{i}}{2}}.$ In the similar way, we can show that
$c_{i}^{\frac{\mu_{i}}{2}}kc_{j}^{-\frac{\mu_{j}}{2}}k^{-1}$ is reversible if
$\phi(k)=1$.

If $x$ is not reversible in $\pi_{1}(M)/\langle h\rangle$, a reverser of $x$
in $\pi_{1}(M)$, say $\rho$, must be a power of $h$, since the projection to
$\pi_{1}(M)/\langle h\rangle$ must be constant. Then in $\pi_{1}(M)/\langle
h\rangle$, $[x]=[x^{-1}]$. This implies that $[x^{2}]=1$. Then, $x^{2}$ is a
power of $h$, and $x^{2}$ is also reversible with same reverser $\rho$ that
$x$ is. Then,

$h^{k}x^{2}h^{-k}=x^{-2}$ $\Rightarrow x^{2}=x^{-2}$ $\Rightarrow x^{4}=Id.$

This is contradiction since $\pi_{1}(M)$ is torsion free. This proves our
lemma. ∎

###  3.1. Proof of Theorem 1.1

The previous lemma shows that there are no reversible elements in $\pi_{1}(M)$
other than those conjugate to $h$. As shown earlier, the reversibility of $h$
and its conjugates depends on the classifying homomorphism. This is summarised
in the statement of Theorem 1.1. ∎

In the case of non-orientable surfaces, we have certain exceptions to this
situation, as shown in the previous section.

The reversible elements of $\pi_{1}(M)$ in case of a non-orientable orbit
surface can be summarised as follows, by the same proofs as above.

######  Corollary 3.6.

If $M$ has an non-orientable orbit surface other than the Klein bottle or
$\mathbb{RP}^{2}$, the reversible elements of $\pi_{1}(M)$ are either
conjugate to any power of $h$ or either of the form
$c_{i}^{\frac{\mu_{i}}{2}}kc_{j}^{\frac{\mu_{j}}{2}}k^{-1}$ with $\phi(k)=-1$
or of the form $c_{i}^{\frac{\mu_{i}}{2}}kc_{j}^{\frac{-\mu_{j}}{2}}k^{-1}$
with $\phi(k)=1$, where $\mu_{i}$ and $\mu_{j}$ are even, and $k$ is any
element in $\pi_{1}(M)$, $\beta_{i}=\beta_{j}$.

When the orbit surface is the Klein bottle or $\mathbb{RP}^{2}$, we get
additional reversible elements which are the generators of the orbit surface.
Thus, in the case of a given orbit surface, the reversibility of the elements
of the fundamental group of a Seifert fibered space over that orbit surface is
solely determined by $\phi$.

######  Remark 3.7.

The classification of the reversible elements in a given Seifert-fibered space
depends only on the classifying homomorphism.

##  4\. Application to universal extension of Fuchsian groups

In the previous section, we have classified the reversible elements of the
fundamental group of a Seifert-fibered space. We can see the fundamental group
in the form of an exact sequence:

(4.1) $1\longrightarrow\mathbb{Z}\longrightarrow\pi_{1}(M)\longrightarrow
F\longrightarrow 1$

Here, $\pi_{1}(M)$ is the fundamental group of $M$, and $F$ is a Fuchsian
group or discrete isometry group of $\rm PGL(2,\mathbb{R})$. So, $\pi_{1}(M)$
is then a universal extension of $F$, which can now be seen as
$\pi_{1}(M)/\langle h\rangle$. This gives us a new view-point to consider
certain groups as fundamental groups of a Seifert-fibered space. Let us
consider $F=\rm PSL(2,\mathbb{Z})$ which is presented by $\langle
a,b|a^{2},b^{3}\rangle.$ We have already mentioned the presentation of
$\pi_{1}(M)$ for a given $M$ in Equation (1) and Equation (1).

We can choose an orientable connected $3$-manifold $M$ in such a way that the
$\pi_{1}(M)\cong B_{3}$. $B_{3}$ has a presentation as
$\langle\sigma_{1}\sigma_{2}|\sigma_{1}\sigma_{2}\sigma_{1}=\sigma_{2}\sigma_{1}\sigma_{2}\rangle$.
The normal subgroup of $B_{3}$ is $\mathbb{Z}$ and it is generated by
$(\sigma_{1}\sigma_{2})^{3}$. Thus,
$B_{3}/\langle\sigma_{1}\sigma_{2}\rangle=\langle\sigma_{1}\sigma_{2}\sigma_{1},\sigma_{1}\sigma_{2}|(\sigma_{1}\sigma_{2})^{3},(\sigma_{1}\sigma_{2}\sigma_{1})^{2}\rangle=\rm
PSL(2,\mathbb{Z})\cong\Delta(2,3,\infty)$. Choosing the $c_{i}$ and
$\beta_{i}$ suitably, we can consider the Seifert-fibered manifold $M$
determined by $(O,o,0|1,(2,1),(3,1))$. The presentation of the fundamental
group of $M$ is then

(4.2) $\langle
c_{1},c_{2},d,h|c_{i}hc_{i}^{-1}=h,dhd^{-1}=h^{\phi(d)},c_{i}^{\mu_{i}}=h^{\beta}_{i},c_{1}c_{2}dh=1\rangle.$

Now, consider the generating elements
$c_{1}=\sigma_{1}\sigma_{2}\sigma_{1},c_{2}=\sigma_{1}\sigma_{2},$ which
implies that $h=(\sigma_{1}\sigma_{2})^{3}$ and
$d^{-1}=\sigma_{1}\sigma_{2}\sigma_{1}^{2}\sigma_{2}.$
This determines the Fuchsian complex $\pi_{1}(M)/\langle h\rangle$, and we
obtain the resultant orbit surface $G$ as the open disk $D$ by further
quotienting it by the $c_{i}$. Then,
$\phi:\pi_{1}(D)\longrightarrow\mathbb{Z}_{2}$ is the trivial homomorpshim.
Then, the group 4.2 can be presented as

$\displaystyle\langle\sigma_{1}\sigma_{2}\sigma_{1},\sigma_{1}\sigma_{2},\sigma_{1}\sigma_{2}\sigma_{1}^{2}\sigma_{2},(\sigma_{1}\sigma_{2})^{3}|(\sigma_{1}\sigma_{2})^{3}\sigma_{1}\sigma_{2}\sigma_{1}=\sigma_{1}\sigma_{2}\sigma_{1}(\sigma_{1}\sigma_{2})^{3},(\sigma_{1}\sigma_{2}\sigma_{1})^{2}=(\sigma_{1}\sigma_{2})^{3}\rangle$
$\displaystyle=\langle\sigma_{1},\sigma_{2}|\sigma_{1}\sigma_{2}\sigma_{1}=\sigma_{2}\sigma_{1}\sigma_{2}\rangle=B_{3}.$

So, we can deduce the exact sequence

(4.3) $\displaystyle 1\longrightarrow\mathbb{Z}\longrightarrow
B_{3}\longrightarrow\rm PSL(2,\mathbb{Z})\longrightarrow 1$

######  Corollary 4.1.

The only reversible elements of $B_{3}$ are conjugates to
$[\sigma_{1}\sigma_{2}\sigma_{1},k]$ for any element $k$ in $B_{3}$.

Proof. From the above discussion, $\phi$ is trivial, and applying Theorem 3.5
we know that reversible elements for the orientable case depend only on
$\phi$. The claim thus follows from this theorem. ∎

Note that we cannot use the same technique to find the reversible elements of
$B_{n}$ since $B_{n}$ is not a Seifert-fibered group for any $n>3$. This
follows from the theorem below.

######  Theorem 4.2.

For any $n>3$ the braid group $B_{n}$ is not isomorphic to any Seifert-fibered
group.

Proof. On the contrary let us assume that braid groups are Seifert-fibered
groups. So, let there be a Seifert-fibered manifold $M$ such that
$B_{n}\cong\pi_{1}(M)$, with maximal normal subgroup $\langle h\rangle$ where
$h$ represents an ordinary fiber. It is well known that the braid group
$B_{n}$ where $n>2$ is not virtually abelian. Then, from [Ma, Corollary
10.4.6], the center of $\pi_{1}(M)$ is either $\langle h\rangle$ or trivial.
Since the center of $B_{n}$ is not trivial, therefore $Z(B_{n})$ is $\langle
h\rangle$. Let us consider the braid group $B_{n}$

$B_{n}=\langle\sigma_{1}\sigma_{2}\dots\sigma_{n-1}|\sigma_{i}\sigma_{{i+1}}\sigma_{i}=\sigma_{{i+1}}\sigma_{i}\sigma_{{i+1}},\sigma_{i}\sigma_{j}=\sigma_{j}\sigma_{i}\hbox{
for }|i-j|>2\rangle.$

In another presentation we can rewrite the group as mentioned in [MK, Chapter
2, Section 2, Exercise 2.4]:

(4.4) $B_{n}=\langle x,y|x^{n}=(xy)^{n-1},x^{i}yx^{-i}y=yx^{i}yx^{-i}\hbox{
where }i=2,3,\dots,m\rangle$

where, $m=\lfloor\frac{n}{2}\rfloor.$ Then, the fiber
$h=(\sigma_{1}\sigma_{2}\dots\sigma_{n-1})^{n}=x^{n}$ is the generator of
$Z(\pi_{1}(M))$. Then,

$B_{n}/\langle h\rangle=\langle
x,y|x^{n}=(xy)^{n-1}=1,x^{i}yx^{-i}y=yx^{i}yx^{-i}\hbox{ where
}i=2,3,\dots,m\rangle$

is a Fuchsian group or a fundamental group of non-orientable orbifold surface.
So, it acts as an isometry group in ${\bf H}^{2}$. If $B_{n}/\langle h\rangle$
acts in ${\bf H}^{2}$, then $x^{i}yx^{-i}$ and $y$ are powers of the same
element $\psi\in B_{n}/\langle h\rangle$, that is, $x^{i}yx^{-i}=\psi^{u}$ and
$y=\psi^{v}$ for some integers $u,v$. But in [DG1], we noticed that $x^{i}$
sends fixed points of $y$ to the fixed points of $x^{i}yx^{-i}$. That means
either $x^{i}$ fixes the fixed points of $y$, or interchanges them for every
$i$. If it fixes the fixed points then that implies $xy=yx.$ Then $x$ is an
element in the center of $B_{n}/\langle h\rangle$, so it must be trivial. This
is contradiction in our assumption.

So, $x^{i}$ must interchange the fixed points of $y$ for every $i$. Then
$x^{2}$ must be an involution (orientation preserving or orientation
reversing). This implies $n=4$, and thus $x^{2}yx^{-2}=y^{-1}.$ $y^{-1}$ is
$\sigma_{3}$ in $B_{4},$ and then $\sigma_{1}\sigma_{3}$ is an element in the
center which contradicts our assumption. So, $B_{n}/\langle h\rangle$ will not
be the fundamental group of a Seifert-fibered space for any $n>3$. ∎
However, there always exits some reversible elements in $B_{n}$ by the
following remark. Here, we are assuming the presentation of $B_{n}$ as given
in (4.4).

######  Remark 4.3.

From Corollary 4.1, there are some reversible elements in $B_{n}$ which are
conjugate to the elements of the form $[a,b]$, where the image of $a$ is an
involution in the quotient space $B_{n}/\langle x^{n}\rangle$ and $b$ is any
other element in $B_{n}$. This is a sufficient condition to be a reversible
element in $B_{n}$.

######  Remark 4.4.

From the above theorem, we obtain that the same idea that we have used for
Seifert-fibered spaces will not work for $B_{n}$. Furthermore, by [HMT,
Theorem 1.3] and Theorem 4.2, $B_{n}$ cannot be a fundamental group of any
$3$-manifold. This has been proven earlier using different techniques, for
example in [Go] where they use the theory of coherent groups, however this
gives a different perspective using reversible elements.

## References

  * [Br] M. G. Brin.  _Seifert Fibered Spaces: Notes for a course given in the Spring of 1993_.  arXiv preprint arXiv:0711.1346
  * [BR] M. Baake and J. A. G. Roberts,  The structure of reversing symmetry groups.  Bull. Austral. Math. Soc. 73 (2006), no. 3, 445–459.
  * [DG1] D. Das, K. Gongopadhyay.  _Reciprocity in the Hecke Groups_.  arXiv preprint arXiv:2307.00496.(2023)
  * [DG2] D. Das, K. Gongopadhyay.  _On The z-classes in Fuchsian group_.  (In preparation)
  * [Go] C. M. Gordon.  _Artin groups, 3-manifolds and coherence._ Boletín de la Sociedad Matemática Mexicana, Vol. 10, (2004), 193-198.  https://web.ma.utexas.edu/users/gordon/artin-dedicated.pdf
  * [HMT] K. Himeno, K. Motegi, and M. Teragaito. _Classification of generalized torsion elements of order two in 3–manifold groups._ https://sites.google.com/izt.uam.mx/citaandppicta2023/home
  * [La] J. S. W. Lamb,  Reversing symmetries in dynamical systems.  J. Phys. A 25 (1992), no. 4, 925–937.
  * [Ma] B. Martelli.  _Introduction to Geometric Topology_ arXiv preprint arXiv:1610.02592
  * [MK] K. Murasugi and B. I. Kurpita,  _A study of braids_ ,  Mathematics and its Applications 484, Kluwer Academic Publishers, Dordrecht, 1999.
  * [ST] H. Seifert, W. Threlfall.  _Seifert and Threlfall: a textbook of topology_ Pure Appl. Math., 89 Academic Press, Inc. [Harcourt Brace Jovanovich, Publishers], New York-London, 1980. xvi+437 pp.