# Unsupervised Discovery of Continuous Skills on a Sphere

Takahisa Imagawa  Takuya Hiraoka  Yoshimasa Tsuruoka

###### Abstract

Recently, methods for learning diverse skills to generate various behaviors
without external rewards have been actively studied as a form of unsupervised
reinforcement learning. However, most of the existing methods learn a finite
number of discrete skills, and thus the variety of behaviors that can be
exhibited with the learned skills is limited. In this paper, we propose a
novel method for learning potentially an infinite number of different skills,
which is named discovery of continuous skills on a sphere (DISCS). In DISCS,
skills are learned by maximizing mutual information between skills and states,
and each skill corresponds to a continuous value on a sphere. Because the
representations of skills in DISCS are continuous, infinitely diverse skills
could be learned. We examine existing methods and DISCS in the MuJoCo Ant
robot control environments and show that DISCS can learn much more diverse
skills than the other methods.

Reinforcement Learning, Unsupervised Learning, Self-Supervised Learning,
Mutual Infomation

##  1 Introduction

Figure 1: Overview of our proposed method. Our agent learns policies by a
multi-objective reinforcement learning method introduced in Section 3.1 and
its reward vectors are generated by a method explained in Section 3.2.

Deep reinforcement learning (RL) has shown promising results in various
domains such as robotic control (OpenAI et al., 2019; Chen et al., 2021) and
games (Silver et al., 2017; Berner et al., 2019; Vinyals et al., 2019).
However, a typical RL agent learns each task from scratch by using external
rewards in terms of how well the task is solved. This is in stark contrast to
the way humans explore the environment and learn various skills and strategies
without such external evaluation.

To fill the gap, methods for learning various skills (i.e., potentially useful
sequences of actions) without external rewards have been studied as a form of
unsupervised or self-supervised RL. These methods are important in practice
since it is generally difficult and costly to design appropriate rewards for
individual tasks and hence external rewards are not always available (Roijers
et al., 2013; Dulac-Arnold et al., 2019). In such cases, task-agnostic skills
learned by unsupervised RL help the agent to quickly solve the task once the
the external rewards are provided. Such skills are also useful in helping the
agent to efficiently explore the environment.

A common approach to learning skills by unsupervised RL is to maximize mutual
information between skills and states. The existing mutual information-based
methods differ in how they treat skills. Specifically, a skill is treated as a
discrete variable (Gregor et al., 2016; Eysenbach et al., 2018; Baumli et al.,
2020; Sharma et al., 2019), a variable in the goal space (Warde-Farley et al.,
2019), or a variable in a space in which state sequences are embedded by a
variational autoencoder (Campos et al., 2020; Kim et al., 2021). These methods
have been shown to help the agent learn useful skills.

Among the mutual information-based methods, a recently proposed one,
Variational Intrinsic Successor FeatuRes (VISR) (Hansen et al., 2020) has two
advantages. First, skills in VISR are continuous skills. Since its skills are
continuous, the agent can pontentially learn a myriad of skills. Second,
skills are learned in association with rewards (more specifically, as weights
of the reward vectors). By learning skills in a such manner, when external
rewards are given, estimating the weights (i.e., appropriate skills for the
given rewards) is relatively easy (Barreto et al., 2018; Yang et al., 2019).

While VISR has these advantages, it also has drawbacks. VISR has been tested
only in discrete action domains in the original and subsequent research (Liu &
Abbeel, 2021), and according to the experimental results of Kim et al. (2021)
in continuous action control environments, the diversity of skills learned by
VISR was limited. Futhermore, the analysis of unsupervised learning process
itself (e.g., sample efficiency) has been rarely performed. Due to the
computational cost of unsupervised learning, its sample efficiency is
important, and so is the analysis from that perspective.

In this paper, we propose a new unsupervised RL method, discovery of
continuous skills on a sphere (DISCS), which learns continuous skills as
weights of reward vectors like VISR. We show an overview of our method in
Figure 1. We investigate the process of unsupervised learning in existing
methods and DISCS in the MuJoCo Ant robot control environment with continuous
actions, and show that DISCS can sample-efficiently learn various skills
compared to existing methods. We also show that learning skills in VISR is
more difficult than DISCS because of its generation of rewards. Furthermore,
we show that an existing discrete skill learning method with many skills
cannot be a substitute for DISCS. In addition, we propose hindsight preference
posterior sampling (HIPPS) as one of the techniques of DISCS and show that it
helps learning in DISCS.

The paper is organized as follows. We introduce the background of DISCS,
multi-objective RL in Section 2 and details of DISCS in Section 3. In Section
4, related work including VISR and differences between VISR and DISCS are
introduced. In Section 5, experimental analysis and comparisons between
existing methods and DISCS are shown. In Section 6, concluding remarks are
given.

##  2 Background

This section briefly introduces multi-objective RL (MORL), upon which our
method is based.

The tasks in MORL are modeled as multi-objective Markov decision processes
(MOMDPs) (Roijers et al., 2013). An MOMDP is an extension of well-known Markov
decision processes (MDP) (Sutton & Barto, 2018). An MOMDP can be represented
by a tuple
$({\mathcal{S},\mathcal{A},R,T,s_{0},\mathcal{W},f_{\mathcal{W}}})$, where
$\mathcal{S}$ and $\mathcal{A}$ are the spaces of states and actions,
respectively, $s_{0}\in\mathcal{S}$ is the initial state,
$r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}^{m}$ is a reward vector
function whose output dimension is $m$,
$T:\mathcal{S}\times\mathcal{A}\times\mathcal{S}$ is a function that
determines the probability of transition to a state when an action is taken at
a state, $\mathcal{W}\subset\mathbb{R}^{m}$ is the space of preferences, and
$f_{\mathcal{W}}:\mathcal{W}\times\mathbb{R}^{m}\rightarrow\mathbb{R}$ is a
scalarization function that transforms the total reward to a scalarized total
reward according to a preference. We consider the class of MOMDPs with a
linear scalarization function. That is,
$f_{\mathcal{W}}(w,{V}^{\pi}(s,w))=w^{T}{V}^{\pi}(s,w)$, where
${V}^{\pi}(s,w)$ is
$\mathbb{E}_{\pi_{w}}\left[\sum_{t=0}\gamma^{t}r(s_{t},a_{t})|s_{0}=s,w\right]$,
$w\in\mathcal{W}$ and $\pi_{w}$ is a policy (a distribution over actions) for
$w$.

The goal of MORL is to learn a policy that maximizes the total scalarized
reward for each preference. An MOMDP with only one preference corresponds to
one MDP. MORL is a framework for improving learning efficiency by learning the
optimal policy set for the given preference set, rather than learning policies
from scratch in individual MDPs.

In this paper, as in previous work (Abels et al., 2019; Yang et al., 2019;
Chen et al., 2020), we focus on learning a preference conditional policy and
Q-function, which returns the expected cumulative reward vectors for the
policy. Also, we put constraints on $\mathcal{W}$ to remove redundancy of $w$
(e.g., a multiplication of rewards leads to the same optimal policy). For
example, Yang et al. (2019) regularize the L1 norm of preferences. In our
method, we regularize the L2 norm of preferences instead because of the
tractability of distributions on $\mathcal{W}$ and
$\mathcal{W}=\\{w\>|\>||w||_{2}=1,w\in\mathbb{R}^{m}\\}$.

Note that an MORL agent learns preference conditional policies, which means
that preference $w$ controls sequential actions (often referred to as a
skill). Thus, we refer to $w$ as not only a preference but also a skill.

##  3 Discovery of Continuous Skills on a Sphere

In this section, we introduce three main components in DISCS: 1) multi-
objective soft actor-critic (MOSAC), 2) reward vector generation, and 3) their
effective training, HIPPS. DISCS learns a policy by MOSAC, which is a simple
extension of soft actor-critic (SAC) (Haarnoja et al., 2018) to MORL, which is
one of the most sample-efficient off-policy RL methods. A DISCS agent learns
how to generate reward vectors on the basis of mutual information between
states and skills on a unit sphere, and the generated reward vectors are used
for the learning in MOSAC. DISCS uses HIPPS, which aims to improve the sample
efficiency of DISCS by adding data sampled from the distribution (posterior)
learned in the reward generation. Pseudo code of DISCS is shown in Section B.

###  3.1 Multi-Objective Soft Actor-Critic

An MOSAC agent collects data from the environment (rollouts), preserves them
in a replay buffer. By using data in the replay buffer, the agent iteratively
learns preference conditional policies and Q-functions as MORL. The agent
maximizes the sum of the policy entropy and the total reward, as in SAC.

For simplicity, we introduce $m+1$-dimensional extended reward vector and
preference, whose the $0$-th dimension is reserved for the entropy of policy.
Let $\tilde{r}$ denote $(c,r_{1},\dots,r_{m})^{\top}$, where $c$ is typically
$0$ and $r_{i}$ ($1\leq i\leq m$) is the $i$-th element of the original reward
vector, and $\tilde{w}$ denote $(1,w_{1},\dots,w_{m})^{\top}$, where $w_{i}$
($1\leq i\leq m$) is the $i$-th element of the original preference. Let
$h^{\pi}(s^{\prime},a^{\prime},w)$ denote a vector
$(-\alpha\log\pi(a^{\prime}|s^{\prime},w),0,\dots,0)^{\top}$ for entropy of
its policy whose dimension is $m+1$, where $\alpha$ is the coefficient of the
entropy. The Q-function is updated based on a Bellman operation with reward
and entropy vectors,

$\displaystyle{\mathcal{T}}Q^{\pi}(s,a,w)=\tilde{r}(s,a)+\gamma\mathbb{E}_{s^{\prime}}[V^{\pi}(s^{\prime},w)]$
(1) $\displaystyle
V^{\pi}(s^{\prime},w)=\mathbb{E}_{\pi(a^{\prime}|s^{\prime},w)}[Q^{\pi}(s^{\prime},a^{\prime},w)+h^{\pi}(s^{\prime},a^{\prime},w)].$
(2)

Applying Bellman operation $\mathcal{T}$ defined above repetitively leads to a
fixed point because ${\mathcal{T}}$ is a contraction mapping (see e.g.,
(Bertsekas, 2012)). In MOSAC, its policy is updated as follows:

$\displaystyle\arg\min_{\pi^{\prime}\in\Pi}\mathrm{D}_{\mathrm{KL}}\left(\pi^{\prime}(a|s,w)\middle\|\frac{\exp(\frac{\tilde{w}^{\top}}{\alpha}{Q}^{\pi}(s,a,w))}{Z^{\pi}(s,w)}\right)$
(3)

For these updates, extensions of two theorems in SAC (Haarnoja et al., 2018)
can be derived in the same way as the proofs in SAC from the fact that one $w$
corresponds to one MDP.

######  Theorem 1.

For any $s\in\mathcal{S},a\in\mathcal{A},w\in W$ and $\pi$, $\pi^{\prime}$
which is updated by (3), then
$\tilde{w}^{\top}(Q^{\pi^{\prime}}(s,a,w)-Q^{\pi}(s,a,w))\geq 0$, assuming
$|\mathcal{A}|<\infty$.

This means that $\pi$ can be improved by (3).

######  Theorem 2.

Repeated application of the updates of Q-functions and policies converges to a
policy $\pi^{\ast}$ such that
$\tilde{w}^{\top}({Q}^{\pi^{\ast}}(s,a,w)-{Q}^{\pi}(s,a,w))\geq 0$ for all
$\pi$ and $s,a,w$, assuming $|\mathcal{A}|<\infty$.

In this paper, the above policy and Q-function are approximated by neural
networks. Let $Q_{\theta_{Q}}(s,a,w)$ denote a Q-function and
$\pi_{\theta_{\pi}}(a|s,w)$ a policy, whose parameter vectors are $\theta_{Q}$
and $\theta_{\pi}$, respectively.

In the same way as SAC, as the target for a Q-function update, we use a
Q-function with parameter $\bar{\theta}$. $\bar{\theta}$ is an exponential
moving average of $\theta_{Q}$ and updated as
$\bar{\theta}\leftarrow\tau\theta_{Q}+(1-\tau)\bar{\theta}$.

The policy and Q-fuction are updated by minimizing losses,
$\mathcal{L}_{\mbox{actor}}$ and $\mathcal{L}_{\mbox{critic}}$, which are
respectively,

$\displaystyle\mathbb{E}\left[\alpha\log\pi_{\theta_{\pi}}(a_{t}|s_{t},w)-\tilde{w}^{\top}Q_{\theta_{Q}}(s_{t},a_{t},w)\right],\mbox{and
}$ (4)
$\displaystyle\mathbb{E}\left[-(\tilde{w}^{\top}(Q_{\theta_{Q}}(s_{t},a_{t},w)-\hat{\mathcal{T}}Q_{\bar{\theta}}(s_{t},a_{t},w)))^{2}\right],$
(5)

where $\mathbb{E}$ in the above equations mean the expectations over tuples,
${(w,s_{t},a_{t},s_{t+1})}$, which are sampled from the replay buffer, and
$\hat{\mathcal{T}}Q_{\bar{\theta}}(s_{t},a_{t},w)$ is

$\displaystyle\tilde{r}(s_{t},a_{t})+\gamma\mathbb{E}_{a\sim\pi_{w}}[Q_{\bar{\theta}}(s_{t+1},a,w)+h^{\pi}(s_{t+1},a,w)].$
(6)

###  3.2 Reward Vector Generation by Mutual Information

In DISCS, the agent learns diverse skills by maximizing $I(S_{t},W)$ the
mutual information between states and preferences. Due to the use of MOSAC,
the agent also aims to maximize the policy entropy,
$\mathcal{H}(A_{t}|S_{t},W)$. Intuitively, maximizing $I(S_{t},W)$ means to go
to the preference’s own state as much as possible. $I(S_{t};W)$ can be
expressed as $\mathbb{E}[\log p(w|s_{t})-\log p(w)]$. We fix $p(w)$ as the
uniform distribution, so $\log p(w)$ is constant and can be ignored. Hence,
our objective is maximizing the expected sum of $\log
p(w|s_{t})-\log\pi(a|s_{t},w)$ and this value corresponds to a scalarized
reward including the policy entropy.

The expected sum of the scalarized rewards, including the expectation over the
entire preferences, which is denoted as $\eta(\pi)$, has the following lower
bound:

$\displaystyle{\eta}(\pi)\geq\eta_{\phi}(\pi),$ (7)

where ${\eta}(\pi)$ and $\eta_{\phi}(\pi)$ are

$\displaystyle\mathbb{E}_{w,\pi_{w}}\left[\sum_{t=0}\gamma^{t}\log
p(w|s_{t})-\alpha\log\pi(a_{t}|s_{t},w)\right],\mbox{and}$ (8)
$\displaystyle\mathbb{E}_{w,\pi_{w}}\left[\sum_{t=0}\gamma^{t}\log
q_{\phi}(w|s_{t})-\alpha\log\pi(a_{t}|s_{t},w)\right],$ (9)

respectively, and $\phi$ is a parameter vector. This inequality can be derived
by an inequation of KL-divergence,
$\mathrm{D}_{\mathrm{KL}}(p(w|s_{t})||q_{\phi}(w|s_{t}))\geq 0$. Hereafter,
$q_{\phi}(w|s)$ is referred to as a discriminator.

We aim to improve $\eta_{\phi}(\pi)$ instead of $\eta(\pi)$ by updating the
policy and Q-function and the discriminator. Let us assume $\phi^{\prime}$ is
a parameter vector updated from $\phi$ and the following inequality holds:

$\displaystyle\eta_{\phi^{\prime}}(\pi)-\eta_{\phi}(\pi)=\mathbb{E}\left[\sum_{t=0}\gamma^{t}\Delta(\log
q(w|s_{t}))\right]\geq 0,$ (10)

where $\Delta(\log q(w|s_{t}))=\log q_{\phi^{\prime}}(w|s_{t})-\log
q_{\phi}(w|s_{t})$. Under fixed $\phi^{\prime}$, i.e., a fixed reward
function, Theorem 1 (and Theorem 2) hold. Thus, updating the policy $\pi$ to
$\pi^{\prime}$ by (3), Q-values of any $(s,a,w)$ improve and the following
inequalities are derived:

$\displaystyle\eta_{\phi^{\prime}}(\pi^{\prime})\geq\eta_{\phi^{\prime}}(\pi)\geq\eta_{\phi}(\pi).$
(11)

These mean that the $\eta$ value can be improved monotonically under condition
(10). Therefore, we update $\phi$ to improve $\eta_{\phi}(\pi)$.

As for our discriminator, we use the von Mises-Fisher distribution (vMF),
because our preferences are on a unit sphere (recall Section 2) and vMF is a
common probability distribution defined there. vMF has two parameters, $\mu$
and $\kappa$, which are the mean direction and concentration parameters,
respectively and let vMF$(\mu,\kappa)$ denote vMF with the parameters. More
concretely, our discriminator is as follows:

$\displaystyle
q_{\phi}(w|s)=C_{m}(\kappa_{\phi_{2}}(s))\exp(\kappa_{\phi_{2}}(s)w^{\top}\mu_{\phi_{1}}(s)),$
(12)

where $\kappa_{\phi_{2}}(s)$ is a scalar value, $\mu_{\phi_{1}}(s)$ is a
$m$-dimensional vector,
$C_{m}(\kappa)=\frac{\kappa^{m/2-1}}{(2\boldsymbol{\pi})^{m/2}I_{m/2-1}(\kappa)}$
is a normalization constant, $\boldsymbol{\pi}$ is the ratio of a circle’s
circumference to its diameter, and $I_{m/2-1}(\kappa)$ is the modified Bessel
function of the first kind at order $m/2-1$.

Note that $\log C_{m}(\kappa)$ is partially differentiable with respect to
$\kappa$ as follows:

$\displaystyle\frac{\partial}{\partial\kappa}\log
C_{m}(\kappa)=-\frac{I_{m/2}(\kappa)}{I_{m/2-1}(\kappa)}$ (13)

The equation above can be derived by using
$\frac{\partial}{\partial\kappa}I_{m/2-1}(\kappa)=\frac{m/2-1}{\kappa}I_{m/2-1}(\kappa)+I_{m/2}(\kappa)$.
This means that the gradients with respect to $\phi_{2}$ can be backpropagated
via Equation (13).

Now that we have defined $q_{\phi}$ as above, $\log
q_{\phi}(w|s)=\tilde{w}^{\top}\tilde{r}_{\phi}$, where $\tilde{r}_{\phi}$ is

$\displaystyle\kappa_{\phi_{2}}(s)\left(\frac{\log
C_{m}(\kappa_{\phi_{2}}(s))}{\kappa_{\phi_{2}}(s)},\mu_{1,\phi_{1}}(s),\dots,\mu_{m,\phi_{1}}(s)\right)^{\top},$
(14)

where $\mu_{i,\phi_{1}}$ is $i$-th element of $\mu_{\phi_{1}}$ and we use
$\tilde{r}_{\phi}$ as a reward vector for MOSAC.

We use the following value as the loss of discriminator, and update the
parameter vector $\phi$ to minimize the loss:

$\displaystyle\mathcal{L}_{\mbox{disc}}:=-\mathbb{E}_{(w,s_{t})\sim
D}\left[\log q_{\phi}(w|s_{t})\right],$ (15)

where $\mathbb{E}_{(w,s_{t})\sim D}$ means the expectation for $(w,s_{t})$
sampled from the replay buffer, $D$. The loss defined above is different from
that in theoretical analysis around inequation (10) in some respects. As for
the update of the discriminator, in the theoretical analysis, we considered a
discounted objective with $\gamma$, but in the implementation, it was not
included. Also, in the theoretical analysis, we considered updating the
discriminator for the state distribution defined by the most recent policy,
but using only the recent data would reduce the amount of them. In the
implementation, the discriminator is updated by sampling from the entire
replay buffer. Data in the replay buffer are collected by the previous
policies which are generally different from the latest policy. More details of
these differences are examined in Section 5.4.

2

Figure 2: Two types of environments, NoWall and U-Wall in our experiments.

4

(a) the number of occupied cells

(b) discriminator loss

(c) average reward

(d) ciritic loss

Figure 3: Comparisons of learning curves in NoWall. Thin lines are actual data
and thick lines are the averages of them.

6

(a) VISR

(b) NoHIPPS

(c) HIPPS8

(d) DIAYN

(e) DIAYN40

(f) SAC

Figure 4: Heatmaps in NoWall at 3 million timesteps in VISR, SAC and at 5
million timesteps in the other methods.

4

(a) the number of occupied cells

(b) discriminator loss

(c) average reward

(d) ciritic loss

Figure 5: Comparisons of learning curves in NoWall. Thin lines are actual data
and thick lines are the averages of them.

###  3.3 Hindsight Preference Posterior Sampling

The discriminator can be optimized as described in the previous section.
However, learning (infinitely) many diverse policies may need much more data
than learning a single policy. We also propose a method to generate artificial
data to learn policies effectively, named hindsight preference posterior
sampling (HIPPS).

Since our method is off-policy RL, it can learn from data that are not
actually collected by the policy. Therefore, it is possible to make learning
more efficient by adding data. HIPPS modifies the data in a hindsight manner,
as in HER (Andrychowicz et al., 2017). In HIPPS, in addition to actual data
stored in the replay buffer $(w,s,a,s^{\prime})$, additional data
$(w^{\prime},s,a,s^{\prime})$, where $w^{\prime}$ is a generated preference,
are used for learning.

However, it may not be a good idea to train with arbitrary generated
preferences. DISCS learns the policy, Q-function for each preference, as
described in Section 3.1. However, learning to correctly approximate the
Q-value for any $(w,s,a,s^{\prime})$ is impractical and difficult, e.g., in
terms of computational cost. Therefore, if we choose $w^{\prime}$ poorly, the
critic loss, for example, may be high for $(w^{\prime},s,a,s^{\prime})$. As a
result, the parameter update is affected by the loss, and the prediction of
the Q-value for the actual data $(w,s,a,s^{\prime})$ may become poor. Thus, if
we can choose a more plausible $w^{\prime}$, our learning would be more
efficient. Motivated by this, we propose to sample additional preferences
$w^{\prime}$ from the discriminator, i.e., posterior, $q_{\phi}(w|s)$, and to
use it as a tuple $(w^{\prime},s,a,s^{\prime})$ for the training of policy and
Q-network.

We sample additional preferences from the projected normal distribution (PN)
(Mardia, 1975; Wang & Gelfand, 2013), instead of sampling from vMF. In
general, sampling from vMF is difficult. To address this difficulty, several
sampling methods, including rejection sampling have been proposed (Ulrich,
1984; Kurz & Hanebeck, 2015). We apply PN to HIPPS for its tractability. PN is
a probability distribution of $Y=\frac{X}{||X||_{2}}$, where $X$ is a random
vector which follows the multivariate normal distribution
$\mathcal{N}(\mu,\Sigma)$, which is denoted as PN($\mu,\Sigma$). If $X$ is
sampled from the multivariate normal distribution
$\mathcal{N}(\mu,\frac{1}{\kappa}I)$, where $I$ is the identity matrix, the
distribution of $Y$ is vMF$(\mu,a\kappa)$ under condition, $||X||_{2}=a$
(Mardia, 1975). Moreover, PN and vMF converge to the uniform distribution and
delta function as $\kappa$ approaches $0$ and $\infty$, respectively. In
addition to the above properties, the similarity of the two distributions is
shown throughout experiments (Campbell et al., 2019). Thus we sample from
PN($\mu,\frac{1}{\kappa}I$), instead of vMF. In Section 5, we confirm that the
approximation by PN is reasonable enough in terms of experimental results.

##  4 Related Work

Unsupervised RL methods based on mutual information are already introduced in
Section 1. Among them, we will discuss the differences between the most
related methods, VISR and DIAYN, and DISCS. Other related methods will also be
briefly reviewed.

VISR and DIAYN. VISR and DIAYN have the same objective, i.e., maximizing the
mutual information between states and skills, as DISCS. While VISR is applied
to Q-learning in the original paper (Sutton & Barto, 2018), VISR is applied
for MOSAC in this paper. Apart from this difference, VISR can be seen as a
special case of DISCS, where $\kappa$ is $1$ in the discriminator and HIPPS is
not applied. In this case, $\log C_{m}(\kappa)$ is constant, so VISR ignores
it. By ignoring $\kappa$ and $\log C_{m}(\kappa)$, for the output of
discriminator in VISR, $\log q_{\mathrm{VI}}(w|s)$, the following inequalities
hold because of the L2 norm constraint: $-1\leq\log
q_{\mathrm{VI}}(w|s)=w^{\top}\mu(s)\leq 1$. To learn more fine-grained skills,
it is necessary to change the reward more finely according to the differences
in the distribution of states induced by the skill, but this is difficult if
$\kappa$ is constant, i.e., $\kappa$ in the distributions generated by the
discriminator are the same value in any states. DIAYN can also be seen as a
special case of DISCS, where its skill $z$ is a discrete variable and it does
not deal with reward vectors. Its discriminator’s outcomes, $\log
q_{\mathrm{DI}}(z|s_{t})$, are used for its rewards. Also, HIPPS is not
applied for DIAYN.

Reward vectors. The existing methods for MORL (Roijers et al., 2014; Mossalam
et al., 2016; Xu et al., 2020; Cao & Zhan, 2021) and succussor features (SF)
(Barreto et al., 2017; Borsa et al., 2018; Hunt et al., 2019; Barreto et al.,
2019; Zahavy et al., 2021) are related in terms of using reward vectors. In
conventional SF settings, the agent optimizes its policy under condition that
scalar rewards are given. The SF agent approximates the reward by
$w^{\top}\phi$, where $w$ and $\phi$ are a weight vector and a reward vector,
respectively and learns the policy that maximizes total rewards.

Other viewpoints. Our method is related to hierarchical RL (Barto & Mahadevan,
2003), although our skill is only chosen at the initial state. In addition,
our method gradually changes rewards, which corresponds to gradually changing
tasks. This is relevant to curriculum learning (Narvekar et al., 2020). Our
method is also relevant to intrinsic motivation and curiosity (Schmidhuber,
2006; Bellemare et al., 2016), as the agent itself generates the reward. As
for vMF, Kumar & Tsvetkov (2018) applied it for natural language generation
tasks. As for the preference conditional Q-function in our method, the
Q-functions in Schaul et al. (2015) and Borsa et al. (2018) are similar to
ours, although their studies are not about unsupervised RL.

##  5 Experiments

In this section, we mainly examine the following questions: 1) Why is VISR
difficult to learn diverse skills? 2) Can diverse skills be learned
efficiently by the discrete skill learning method, DIAYN? 3) How much can
DISCS outperform these methods? and 4) How can HIPPS help learning by DISCS?
We also examine different update methods for discriminator in DISCS.

We conducted experiments in the MuJoCo Ant robot control environments shown in
Figure 2. In these environments, agents cannot get any rewards from them. We
ran five trials with different random seeds. In the experiments, to evaluate
how diverse the learned skills is, we discretize the x-y positions of agents
in rollouts and show heatmaps about the positions. The episode length was set
to 500 timesteps, and heatmaps were drawn for every 100 episodes, i.e., 0.05
million timesteps of data. In addition, to analyze the progress of the diverse
skill learning, we measure the number of the discretized x-y positions whose
visitation counts are positive (we refer to it as the number of occupied
cells). Also, we analyze the discrimination loss, critic loss and the average
of scalarized rewards in the batch data excluding the policy entropy bonus. In
our experiments, all discriminators are trained with “x-y prior”, which means
that the inputs of the discriminators are x-y positions instead of states. In
general, the state space is large, so it is difficult to learn skills without
x-y prior that are diverse in terms of x-y positions. In fact, in the
experiments in DIAYN and DADS (Eysenbach et al., 2018; Sharma et al., 2019),
the agents could not learn diverse skills in terms of x-y position without it.

4

(a) the number of occupied cells

(b) discriminator loss

(c) average reward

(d) ciritic loss

Figure 6: Comparisons of learning curves in U-Wall. Thin lines are actual data
and thick lines are the averages of them.

5

(a) VISR at 1M

(b) VISR at 3M

(c) NoHIPPS

(d) HIPPS8

(e) DIAYN

Figure 7: Trajectories in U-Wall at 5 million timesteps in DIAYN and 3 million
timesteps in HIPPS8, NoHIPPS, and VISR. Trajectories of VISR at 1 million
timesteps are also shown.

4

(a) the number of occupied cells

(b) discriminator loss

(c) average reward

(d) ciritic loss

Figure 8: Comparisons of learning curves in NoWall. Thin lines are actual data
and thick lines are the averages of them.

###  5.1 Difficulties of VISR and DIAYN

We analyze why VISR does not work in the NoWall environment and examine
whether a method for learning a large number of discrete skills, e.g., DIAYN,
can be a substitute for that for learning continuous skills, e.g., DISCS. We
compare VISR and DIAYN, in which the numbers of discrete skills are $10$ and
$40$, with DISCS without HIPPS (NoHIPPS) because of the similarities mentioned
in Section 4. In addition, we show the performance of SAC, where there is no
reward other than entropy of policy. The results are shown in Figure 3.

The number of occupied cells of VISR was slightly larger than that of SAC.
Although one of the trials of NoHIPPS failed to learn, the other trails show
that much more diverse skills were learned than VISR. Also, the heatmaps
(Figure 4) showed that the area covered by VISR was much smaller than NoHIPPS.
VISR was stable in terms of critic loss, except for the last 1 million
timesteps. From this, it appears that the critic learning is fine. We can see
that the discrimination loss of VISR has decreased to around $-1$, which means
the outputs of discriminator are nearly the minimum value (recall Section 4).
What this means is that in order to learn more different skills, it is
necessary to identify small differences in the rewards (i.e., the output of
the discriminator) and learn a policy that reflects those differences. In the
same way, the discriminator also needs to reflect the differences in the state
distribution defined by the policy for each skill. For those reasons, it is
quite difficult to learn diverse skills by VISR. On the other hand, the
discrimination loss in NoHIPPS was decreasing and much larger than its minimum
value which was $-\infty$ in theory ($-6\log 10$ in our implementation).

The number of occupied cells in DIAYN10 was much larger than in VISR but its
sample efficiency was worse than NoHIPPS. The number of occupied cells of
DIAYN40 was almost the same throughout the trials. The discrimination loss of
DIAYN40 finally began to decrease after around 5 million timesteps, which
indicates that DIAYN40 was starting to learn diverse skills. These results
show that the learning in DIAYN needs more samples when the number of skills
is increased, while that is not the case in DISCS.

###  5.2 HIPPS

We examine whether DISCS learns efficiently with more data by HIPPS in the
NoWall environment. We also examine why the posterior is important in HIPPS by
comparing it to the case where we sample from the prior rather than the
posterior. In addition, we compare DISCS with HIPPS to NoHIPPS with a large
batch and show that simply increasing the batch size is not helpful. Note that
DISCS with HIPPS uses a larger batch than NoHIPPS owing to its additional
preferences. The results are shown in Figure 5. Batchx4 in the figure means
simply quadrupling the batch size without using HIPPS. HIPPS4/HIPPS8 means
that HIPPS with 3/7 preferences are sampled for each tuple (and the total
batch size is increased by 4/8 times). Prior4 is a variant of HIPPS4 where the
prior is used for the preference sampling instead of the posterior.

The number of occupied cells of HIPPS4,8 and NoHIPPS were larger than the
other methods. In particular, HIPPS4,8 showed that the critic losses were low
and the number of occupied cells were high in all trials. The critic loss of
Batchx4 was huge, which may be due to overtraining on the same data and one of
the reasons for the failure in learning by Batchx4.

In Prior4, the number of occupied cells started to decrease from about 1
million timesteps, and the critic loss started to increase from about 0.5
million timesteps. The discriminator loss of Prior4 decreased, which suggests
that the state distribution changed with each preference and that the
discriminator was able to correctly discriminate against the state
distribution and that the distribution of discriminator became peaky. On the
other hand, the average reward of Prior4 decreased, which indicates that
preferences with lower probability in terms of the distribution of
discriminator were sampled. These results support the claim made in Section
5.2 that sampling less plausible (in terms of the distribution of posterior)
preferences from the prior increases the loss of critics and that it has
negative effects on the learning.

###  5.3 Comparisons in Environment with Obstacle

When dealing with complex problems such as controlling ant robots in
unsupervised RL, comparisons have been made mainly in tasks without obstacles.
In this work, we investigate how much the robot can bypass the obstacles by
using U-Wall in Figure 2.

We compare the results of the existing methods, DIAYN and VISR with DISCS. As
confirmed in the results in Section 5.1, DIAYN learns more slowly when the
number of skills is increased. In this comparison, the number of skills in
DIAYN was set to $10$. The learning curves for NoHIPPS and DIAYN were measured
up to 5 million timesteps, while those for other methods were 3 millon
timesteps. The results are shown in Figure 6.

The number of occupied cells of DISCS increased quickly. In particular, DISCS
with HIPPS shows better results than the other methods while the number of
occupied cells of NoHIPPS decreased from around 3 million timesteps and its
critic loss increased. For DIAYN and VISR, the critic loss also increased
during the skill learning process. As for VISR, the number of occupied cells
decreased after 1 million timesteps. These results show that skill learning in
U-wall tends to be unstable and difficult. On the other hand, the critic loss
was stable in DISCS with HIPPS.

For more detailed analysis, instead of heatmaps, we show trajectories when
$100$ different generated skills were executed in Figure 7 (heatmaps are shown
in Section A). The execution of skills was deterministic, i.e., the action was
executed whose probability in the skill conditional policy was the highest. In
the DIAYN case, $10$ different skills were executed $10$ times for each skill.
As for VISR, trajectories at 1 million timesteps, the timesteps before the
number of occupied cells of VISR started to decrease, are also shown. The
trajectories of VISR at 1 million timesteps showed diverse behaviors although
their covered area was limited. As for the trajectories of DIAYN, the same
skills showed almost the same trajectories and their diversity was limited.
Compared with them, the results of DISCS showed that it learned a variety of
skills.

###  5.4 Detailed Analysis of Discriminator Updates

We updated $\phi$ in a way that minimizes (15). This deviates from theoretical
analysis in Section 3.2 in the following aspects. 1) All data in the replay
buffer are sampled for the update. 2) It does not consider the discount of the
reward by $\gamma$. We examine these deviations.

With respect to the first point, we examine the performance when the data used
to update the discriminator are limited to the latest data (Recent). From the
theoretical analysis in Section 3.2, it is ideal to update the discriminator
with the latest policy data to increase its value, but on the other hand, the
more we limit the data used to the latest one, the less data we can use. As
the latest data, we sampled from the recent 0.1 million steps data.

In addition, with respect to the second point, we examine a variant of the
discriminator update where the rewards in the discrimination loss are
discounted by $\gamma$ (Gamma). Even if the deviation of the first point is
ignored and assumed that the data are the latest, because the reward is not
discounted by $\gamma$, an estimate of $\eta_{\phi}(\pi)$ is biased as
discussed by Thomas (2014). To consider the discount of the reward, we also
keep timesteps $t$ in the replay buffer, and use $-\gamma^{t}\log
q_{\phi}(w|s_{t})$ as the loss for the sampled $(w,s_{t},t)$. The results are
shown in Figure 8.

The critic loss was more likely to increase when using only recent data than
when using the entire replay buffer. One possible explanation for these
results is catastrophic forgetting, where the learned relationships between
inputs and outputs of the neural networks are forgotten and cannot be reused,
so the output of discriminator is not stable. A method in Abels et al. (2019)
may alleviate the catastrophic forgetting, where mainly the latest data are
used for the training, but also older data are used. For simplicity, however,
we sampled from the entire replay buffer to train the discriminator.

The performance about the number of occupied cells in Gamma was almost the
same as that of NoHIPPS. Although the estimation is biased, in our
discriminator updates, we ignored the discount by $\gamma$ in (15), because it
was also ignored in the discriminator loss in VISR and DIAYN and the
performances of Gamma and NoHIPPS were almost the same.

##  6 Conclusion

In this paper, we proposed DISCS, an unsupervised RL method for learning
skills, and HIPPS, a method for effective training in DISCS. DISCS is
different from most of the existing methods in that it has a clear
correspondence with reward, it is a continuous skill learning method, and it
uses HIPPS. We conducted experiments in the MuJoCo Ant robot control
environment with continuous actions and analyzed the process of unsupervised
learning. Through the analysis of the experiments, we showed that the existing
method, VISR, has difficulty learning diverse skills due to the low expressive
power of the discriminator, and that increasing the expressive power of the
discriminator like DISCS is important. In addition, through the analysis of
DIAYN, we showed that the learning became slower when the number of skills in
DIAYN was increased. This indicates that learning many discrete skills does
not substitute for learning continuous skills. Moreover, we examined DISCS
with and without HIPPS and showed that HIPPS contributed efficient and stable
learning of skills in DISCS.

## References

  * Abels et al. (2019) Abels, A., Roijers, D., Lenaerts, T., Nowé, A., and Steckelmacher, D.  Dynamic weights in multi-objective deep reinforcement learning.  In _Proceedings of International Conference on Machine Learning_ , pp. 11–20. PMLR, 2019.
  * Andrychowicz et al. (2017) Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, O. P., and Zaremba, W.  Hindsight experience replay.  In _Proceedings of Advances in Neural Information Processing Systems_ , pp. 5048–5058, 2017.
  * Barreto et al. (2017) Barreto, A., Dabney, W., Munos, R., Hunt, J. J., Schaul, T., van Hasselt, H. P., and Silver, D.  Successor features for transfer in reinforcement learning.  In _Proceedings of Advances in neural information processing systems_ , pp. 4055–4065, 2017.
  * Barreto et al. (2018) Barreto, A., Borsa, D., Quan, J., Schaul, T., Silver, D., Hessel, M., Mankowitz, D., Zidek, A., and Munos, R.  Transfer in deep reinforcement learning using successor features and generalised policy improvement.  In _International Conference on Machine Learning_ , pp. 501–510. PMLR, 2018.
  * Barreto et al. (2019) Barreto, A., Borsa, D., Hou, S., Comanici, G., Aygün, E., Hamel, P., Toyama, D., Mourad, S., Silver, D., Precup, D., et al.  The option keyboard: Combining skills in reinforcement learning.  In _Proceedings of Advances in Neural Information Processing Systems_ , pp. 13052–13062, 2019.
  * Barto & Mahadevan (2003) Barto, A. G. and Mahadevan, S.  Recent advances in hierarchical reinforcement learning.  _Discrete event dynamic systems_ , 13(1):41–77, 2003.
  * Baumli et al. (2020) Baumli, K., Warde-Farley, D., Hansen, S., and Mnih, V.  Relative variational intrinsic control.  _arXiv preprint arXiv:2012.07827_ , 2020.
  * Bellemare et al. (2016) Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R.  Unifying count-based exploration and intrinsic motivation.  In _Advances in Neural Information Processing Systems_ , pp. 1471–1479, 2016.
  * Berner et al. (2019) Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al.  Dota 2 with large scale deep reinforcement learning.  _arXiv preprint arXiv:1912.06680_ , 2019.
  * Bertsekas (2012) Bertsekas, D.  _Dynamic programming and optimal control: Volume II_ , volume 2.  Athena scientific, 2012.
  * Borsa et al. (2018) Borsa, D., Barreto, A., Quan, J., Mankowitz, D. J., van Hasselt, H., Munos, R., Silver, D., and Schaul, T.  Universal successor features approximators.  In _Proceedings of International Conference on Learning Representations_ , 2018.
  * Campbell et al. (2019) Campbell, D., Petersson, L., Kneip, L., Li, H., and Gould, S.  The alignment of the spheres: Globally-optimal spherical mixture alignment for camera pose estimation.  2019\.
  * Campos et al. (2020) Campos, V., Trott, A., Xiong, C., Socher, R., Giró-i Nieto, X., and Torres, J.  Explore, discover and learn: Unsupervised discovery of state-covering skills.  In _International Conference on Machine Learning_ , pp. 1317–1327. PMLR, 2020.
  * Cao & Zhan (2021) Cao, Y. and Zhan, H.  Efficient multi-objective reinforcement learning via multiple-gradient descent with iteratively discovered weight-vector sets.  _Journal of Artificial Intelligence Research_ , 70:319–349, 2021.
  * Chen et al. (2020) Chen, D., Wang, Y., and Gao, W.  A two-stage multi-objective deep reinforcement learning framework.  In _Proceedings of European conference on artificial intelligence_ , 2020.
  * Chen et al. (2021) Chen, T., Xu, J., and Agrawal, P.  A system for general in-hand object re-orientation.  In _5th Annual Conference on Robot Learning_ , 2021.  URL https://openreview.net/forum?id=7uSBJDoP7tY.
  * Dulac-Arnold et al. (2019) Dulac-Arnold, G., Mankowitz, D., and Hester, T.  Challenges of real-world reinforcement learning.  _arXiv preprint arXiv:1904.12901_ , 2019.
  * Eysenbach et al. (2018) Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S.  Diversity is all you need: Learning skills without a reward function.  _arXiv preprint arXiv:1802.06070_ , 2018.
  * Gregor et al. (2016) Gregor, K., Rezende, D. J., and Wierstra, D.  Variational intrinsic control.  _arXiv preprint arXiv:1611.07507_ , 2016.
  * Haarnoja et al. (2018) Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al.  Soft actor-critic algorithms and applications.  _arXiv preprint arXiv:1812.05905_ , 2018.
  * Hansen et al. (2020) Hansen, S., Dabney, W., Barreto, A., Warde-Farley, D., de Wiele, T. V., and Mnih, V.  Fast task inference with variational intrinsic successor features.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=BJeAHkrYDS.
  * Hunt et al. (2019) Hunt, J., Barreto, A., Lillicrap, T., and Heess, N.  Composing entropic policies using divergence correction.  In _International Conference on Machine Learning_ , pp. 2911–2920. PMLR, 2019.
  * Kim et al. (2021) Kim, J., Park, S., and Kim, G.  Unsupervised skill discovery with bottleneck option learning.  In _ICML_ , pp. 5572–5582, 2021.  URL http://proceedings.mlr.press/v139/kim21j.html.
  * Kumar & Tsvetkov (2018) Kumar, S. and Tsvetkov, Y.  Von mises-fisher loss for training sequence to sequence models with continuous outputs.  _arXiv preprint arXiv:1812.04616_ , 2018.
  * Kurz & Hanebeck (2015) Kurz, G. and Hanebeck, U. D.  Stochastic sampling of the hyperspherical von mises–fisher distribution without rejection methods.  In _2015 Sensor Data Fusion: Trends, Solutions, Applications (SDF)_ , pp. 1–6, 2015.  doi: 10.1109/SDF.2015.7347705.
  * Liu & Abbeel (2021) Liu, H. and Abbeel, P.  Aps: Active pretraining with successor features.  In _International Conference on Machine Learning_ , pp. 6736–6747. PMLR, 2021.
  * Mardia (1975) Mardia, K. V.  Statistics of directional data.  _Journal of the Royal Statistical Society: Series B (Methodological)_ , 37(3):349–371, 1975.
  * Mossalam et al. (2016) Mossalam, H., Assael, Y. M., Roijers, D. M., and Whiteson, S.  Multi-objective deep reinforcement learning.  _arXiv preprint arXiv:1610.02707_ , 2016.
  * Narvekar et al. (2020) Narvekar, S., Peng, B., Leonetti, M., Sinapov, J., Taylor, M. E., and Stone, P.  Curriculum learning for reinforcement learning domains: A framework and survey.  _Journal of Machine Learning Research_ , 21(181):1–50, 2020.
  * OpenAI et al. (2019) OpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., Schneider, J., Tezak, N., Tworek, J., Welinder, P., Weng, L., Yuan, Q., Zaremba, W., and Zhang, L.  Solving rubik’s cube with a robot hand.  2019\.
  * Raffin et al. (2019) Raffin, A., Hill, A., Ernestus, M., Gleave, A., Kanervisto, A., and Dormann, N.  Stable baselines3.  https://github.com/DLR-RM/stable-baselines3, 2019.
  * Roijers et al. (2014) Roijers, D., Scharpff, J., Spaan, M., Oliehoek, F., De Weerdt, M., and Whiteson, S.  Bounded approximations for linear multi-objective planning under uncertainty.  In _Proceedings of the International Conference on Automated Planning and Scheduling_ , volume 24, 2014.
  * Roijers et al. (2013) Roijers, D. M., Vamplew, P., Whiteson, S., and Dazeley, R.  A survey of multi-objective sequential decision-making.  _Journal of Artificial Intelligence Research_ , 48:67–113, 2013.
  * Schaul et al. (2015) Schaul, T., Horgan, D., Gregor, K., and Silver, D.  Universal value function approximators.  In Bach, F. and Blei, D. (eds.), _Proceedings of the 32nd International Conference on Machine Learning_ , volume 37 of _Proceedings of Machine Learning Research_ , pp. 1312–1320, Lille, France, 07–09 Jul 2015\. PMLR.  URL https://proceedings.mlr.press/v37/schaul15.html.
  * Schmidhuber (2006) Schmidhuber, J.  Developmental robotics, optimal artificial curiosity, creativity, music, and the fine arts.  _Connection Science_ , 18(2):173–187, 2006.
  * Sharma et al. (2019) Sharma, A., Gu, S., Levine, S., Kumar, V., and Hausman, K.  Dynamics-aware unsupervised discovery of skills.  _arXiv preprint arXiv:1907.01657_ , 2019.
  * Silver et al. (2017) Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al.  Mastering the game of go without human knowledge.  _Nature_ , 550(7676):354, 2017.
  * Sutton & Barto (2018) Sutton, R. S. and Barto, A. G.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Thomas (2014) Thomas, P.  Bias in natural actor-critic algorithms.  In _International conference on machine learning_ , pp. 441–448. PMLR, 2014.
  * Ulrich (1984) Ulrich, G.  Computer generation of distributions on the m-sphere.  _Journal of the Royal Statistical Society: Series C (Applied Statistics)_ , 33(2):158–163, 1984.
  * Vinyals et al. (2019) Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., Oh, J., Horgan, D., Kroiss, M., Danihelka, I., Huang, A., Sifre, L., Cai, T., Agapiou, J. P., Jaderberg, M., Vezhnevets, A. S., Leblond, R., Pohlen, T., Dalibard, V., Budden, D., Sulsky, Y., Molloy, J., Paine, T. L., Gulcehre, C., Wang, Z., Pfaff, T., Wu, Y., Ring, R., Yogatama, D., Wünsch, D., McKinney, K., Smith, O., Schaul, T., Lillicrap, T., Kavukcuoglu, K., Hassabis, D., Apps, C., and Silver, D.  Grandmaster level in starcraft ii using multi-agent reinforcement learning.  _Nature_ , 575(7782):350–354, 2019.  doi: 10.1038/s41586-019-1724-z.  URL https://doi.org/10.1038/s41586-019-1724-z.
  * Wang & Gelfand (2013) Wang, F. and Gelfand, A. E.  Directional data analysis under the general projected normal distribution.  _Statistical methodology_ , 10(1):113–127, 2013\.
  * Warde-Farley et al. (2019) Warde-Farley, D., de Wiele, T. V., Kulkarni, T., Ionescu, C., Hansen, S., and Mnih, V.  Unsupervised control through non-parametric discriminative rewards.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=r1eVMnA9K7.
  * Xu et al. (2020) Xu, J., Tian, Y., Ma, P., Rus, D., Sueda, S., and Matusik, W.  Prediction-guided multi-objective reinforcement learning for continuous robot control.  In _Proceedings of the 37th International Conference on Machine Learning_ , 2020.
  * Yang et al. (2019) Yang, R., Sun, X., and Narasimhan, K.  A generalized algorithm for multi-objective reinforcement learning and policy adaptation.  In _Proceedings of Advances in Neural Information Processing Systems_ , pp. 14636–14647, 2019.
  * Zahavy et al. (2021) Zahavy, T., O’Donoghue, B., Barreto, A., Mnih, V., Flennerhag, S., and Singh, S.  Discovering diverse nearly optimal policies with successor features.  2021\.

##  Appendix A Additional Experiments

In this section, we show additional experimental results which are omitted in
the main article.

###  A.1 Trajectories and Heatmaps

First, we show data about trajectories of learned skills in NoWall environment
in Figure 9. The trajectories are drawn in the same way as those in U-Wall
environment (Figure 7). The results are almost same as those of heatmaps
(Figure 4) except for DIAYN. In particular, the result of DIAYN40 show that
the agent cannot move at all in any direction. Note that actions are chosen
deterministically in evaluations for drawing trajectories as explained in
Section 5.3. The trajectories VISR cover a limited area, however they are more
diverse than those of DIAYN40.

5

(a) VISR

(b) NoHIPPS

(c) HIPPS8

(d) DIAYN

(e) DIAYN40

Figure 9: Trajectories in NoWall at 3 million timesteps in VISR and at 5
millon timesteps in the other methods.

Second, we show heatmaps in U-Wall environment in Figure 10. The results show
that DISCS learned more diverse skills than the other methods.

5

(a) VISR at 1M

(b) VISR at 3M

(c) NoHIPPS

(d) HIPPS8

(e) DIAYN

Figure 10: Heatmaps in U-Wall at 5 million timesteps in DIAYN and at 3 million
timesteps in HIPPS8, NoHIPPS, and VISR. Trajectories of VISR at 1 million
timesteps are also shown.

###  A.2 The Number of Discrete Skills in DIAYN

In Section 5.1, we argued that many discrete skills cannot be substitutes for
continuous skills. For more detailed analysis about the argument, we show data
of DIAYN whose number of discrete skills is $20$, in addition to the data of
$10$ and $40$ in Section 5.1. The results are shown in Figure 11. The results
of DIAYN20 show intermediate properties and indicate that the sample
efficiency decreses as number of discrete skills increases.

4

(a) the number of occupied cells

(b) discriminator loss

(c) average reward

(d) ciritic loss

Figure 11: Comparisons of learning curves of DIAYN in NoWall. Data of DIAYN10
and DIAYN40 are the same as those in Figure 3

###  A.3 The Number of Dimensions for Reward Vectors

In this section, we examine how much performance of DISCS without HIPPS
differs by changing the number of dimensions for the reward vectors. In
Section 5, the number of dimensions for the reward vectors are two. In
addition to the setting, we examine performance of DISCS without HIPPS when
the dimensions are three and four. The results (Figure 12) shows that the
average performance worsened as the number of the dimension increased.

4

(a) the number of occupied cells

(b) discriminator loss

(c) average reward

(d) ciritic loss

Figure 12: Comparisons of learning curves of NoHIPPS whose number of
dimensions of reward vector are 2, 3, and 4 in NoWall. Data of Dim2 are the
same as those in Figure 3

##  Appendix B Our Implementation Details

Algorithm 1 DISCS

replay buffer $D$, distribution of preference $\mathbb{P}(w)$

1: while not end do

2: $s\leftarrow s_{0}$ and sample preference $w$ from $\mathbb{P}(w)$

3: for step in data collection steps do

4: Sample action $a$ from policy $\pi(\cdot|s,w)$

5: $s^{\prime}\leftarrow$ env.step($a$)

6: Add $({w,s,a,s^{\prime}})$ to $D$

7: $s\leftarrow s^{\prime}$

8: if episode end then

9: initilize as line 2

10: end if

11: end for

12: if update discriminator then

13: Sample $({w,s,a,s^{\prime}})$ from $D$

14: Update $q_{\phi}$

15: end if

16: for step in update steps do

17: Sample $({w,s,a,s^{\prime}})$ from $D$ and sample hindsight preferences
$w^{\prime}$ from $q_{\phi}(w|s)$

18: Generate reward vector $r$ and $r^{\prime}$ for $({w,s,a,s^{\prime}})$ and
$({w^{\prime},s,a,s^{\prime}})$

19: Update $Q$ and $\pi$ by using $({w,s,a,r,s^{\prime}})$ and
$({w^{\prime},s,a,r^{\prime},s^{\prime}})$

20: Update Q-target

21: end for

22: end while

We implemented our method by modifying SAC in stable-baseline3 (Raffin et al.,
2019). A summary of the modification from the SAC implementation is provided
here.

  * •

Change SAC to MOSAC
More concretely, we implemented the preference conditional policy and
Q-network. Also, we modified the replay buffer to preserve preferences in
rollouts.

  * •

Implement discriminator $q_{\phi}(s_{t}|w)$ and its training procedure

  * •

Implement the partial derivative of $\log C_{m}(\kappa)$ with regard to
$\kappa$
We used the modified Bessel function in SciPy. To backpropagate the gradients,
their calculation has to be implemented manually.

  * •

Implement HIPPS

We will release the code when the paper is accepted.

A summary of hyperparameters in our experiments is provided as Table 1.
Although this is ommitted for simplicity of explanation in algorithm 1, $\pi$
and $\bar{\theta_{Q}}$ is not updated every loop. For the sake of clarity, we
show the number of updates of each network per timesteps. Incidentally, “data
collection steps” and “update steps” in the pseudo code are both 8.

the number of dimensions in reward vectors | 2
---|---
the number of Q-networks | 2
size of hidden layers in Q-networks | 256, 256, 64
size of hidden layers in policy network | 256, 256
size of hidden layers in discriminator | 256, 256
the number of Q updates per timesteps | 1
the number of Q-target updates per timesteps | 1/8
the number of policy updates per timesteps | 1/8
discriminator update per timesteps | 1/50000
batch size | 1024
batch size for discriminator updates | 16384
replay buffer size | 2e+6
$\gamma$ | 0.99
$\alpha$, i.e., entropy coeffient | 0.1
$\tau$, i.e., learning rate of Q-target | 0.005
optimizer | Adam
learning rate | 3e-4
Table 1: The hyperparameters in our experiments.

Although there is one critic in explanation in Section 3.1 for simplicity, in
our actual implementation, two Q-functions are used in the same way as SAC. In
this case, the parameter vectors of Q-targets are updated as follows

$\displaystyle\bar{\theta}_{i}\leftarrow\tau\theta_{i}+(1-\tau)\bar{\theta}_{i},\;(i=1,2),$
(16)

and Q-targets are calculated as follows

$\displaystyle
Q_{\bar{\theta}_{Q}}=\arg\min_{Q\in\\{Q_{\bar{\theta}_{1}},Q_{\bar{\theta}_{2}}\\}}\tilde{w}^{\top}Q.$
(17)

We use the following critic loss to train each Q-network:

$\displaystyle\mathbb{E}\left[-(\tilde{w}^{\top}(Q_{{\theta_{Q}}_{i}}(s_{t},a_{t},w)-\hat{\mathcal{T}}Q_{\bar{\theta}}(s_{t},a_{t},w)))^{2}\right]\;(i=1,2),\;\mbox{where}$
(18)
$\displaystyle\hat{\mathcal{T}}Q_{\bar{\theta}}(s_{t},a_{t},w)=\tilde{r}_{\phi}(s_{t})+\gamma\mathbb{E}_{a_{t+1}\sim\pi(\cdot|s_{t+1},w)}[Q_{\bar{\theta}}(s_{t+1},a_{t+1},w)+h^{\pi}(s_{t+1},a_{t+1},w)].$
(19)


# OdomBeyondVision: An Indoor Multi-modal Multi-platform Odometry Dataset
Beyond the Visible Spectrum

Peize Li1, Kaiwen Cai2, Muhamad Risqi U. Saputra3, Zhuangzhuang Dai4, Chris
Xiaoxuan Lu11,
Andrew Markham4 and Niki Trigoni4 1Corresponding authorThis work was partially
supported by Amazon Web Services via the Oxford-Singapore Human-Machine
Collaboration Programme and EPSRC ACE-OPS (EP/S030832/1)1Peize Li and Chris
Xiaoxuan Lu are with the School of Informatics, University of Edinburgh,
United Kingdom<EMAIL_ADDRESS><EMAIL_ADDRESS>Cai is with
University of Liverpool, United Kingdom<EMAIL_ADDRESS>Risqi
U. Saputra is with Monash University, Indonesia.
<EMAIL_ADDRESS>Dai, Andrew Markham and Niki Trigoni are
with University of Oxford, United Kingdom. {zhudai, andrew.markham,
<EMAIL_ADDRESS>
###### Abstract

This paper presents a multimodal indoor odometry dataset, OdomBeyondVision,
featuring multiple sensors across the different spectrum and collected with
different mobile platforms. Not only does OdomBeyondVision contain the
traditional navigation sensors, sensors such as IMUs, mechanical LiDAR, RGBD
camera, it also includes several emerging sensors such as the single-chip
mmWave radar, LWIR thermal camera and solid-state LiDAR. With the above
sensors on UAV, UGV and handheld platforms, we respectively recorded the
multimodal odometry data and their movement trajectories in various indoor
scenes and different illumination conditions. We release the exemplar radar,
radar-inertial and thermal-inertial odometry implementations to demonstrate
their results for future works to compare against and improve upon. The full
dataset including toolkit and documentation is publicly available at:
https://github.com/MAPS-Lab/OdomBeyondVision.

##  I Introduction

Ego-motion estimation, aka. odometry estimation, finds a plethora of
applications ranging from the localization and mapping for mobile robots to
the pose tracking of wearable devices. By leveraging different navigation
sensors (e.g., cameras, LiDAR, and inertial measurement unit (IMU)), last
decades witnessed an upsurge of egomotion estimation systems proposed in
various use cases [1, 2, 3]. These odometry systems, however, all have
intrinsic limitations under certain conditions attributed to their underlying
sensing mechanism. To name a few, LiDAR data suffers from self-similar
structures and environments full of airborne particles (e.g., smoke), while
RGB cameras fall short in low illumination and high dynamic range scenarios.
IMUs are susceptible to the drifts in large distances or a large magnitude of
movement. Realizing the intrinsic limitation of individual sensors, multimodal
odometry that jointly uses heterogeneous navigation sensors has received
recent attention to enhance the overall robustness of an egomotion estimation.
The success of such an odometry system starts from the public availability of
multi-sensor datasets which enable systematic testing and community benchmark
of different algorithms.

Figure 1: A data example captured in poor illumination from our multimodal odometry dataset. In addition to the above five modalities, our dataset also includes solid-state LiDAR and IMU sensors. TABLE I: Comparison of multimodal odometry datasets Dataset | Year | Environment | Platform | Radar | LiDAR | RGB | Stereo | Thermal | IMU
---|---|---|---|---|---|---|---|---|---
KAIST Day/Night[4] | 2018 | Urban | Car | ✗ | ✓ | ✓ | ✓ | ✓ | ✓
Oxford Radar RobotCar[5] | 2020 | Urban | Car | ✓ | ✓ | ✓ | ✓ | ✗ | ✓
USVInland[6] | 2021 | Inland Water | USV | ✓ | ✓ | ✓ | ✓ | ✗ | ✓
ViViD[7] | 2019 | In-/outdoor | Handheld | ✗ | ✓ | ✓ | ✓ | ✓ | ✓
ColoRadar[8] | 2020 | In-/outdoor | Handheld/Longboard | ✓ | ✓ | ✗ | ✗ | ✗ | ✓
Weichen Dai et al.[9] | 2020 | In-/outdoor | Handheld | ✗ | ✗ | ✓ | ✓ | ✓ | ✓
Ours | 2022 | Indoor | Handheld/UGV/UAV | ✓ | ✓ | ✓ | ✓ | ✓ | ✓

While there exists quite some multimodal odometry datasets [9, 7, 10, 4] in
the community, these datasets are limited to the _diversity_ in terms of both
the navigation sensors onboard and the mobile platforms carrying them.
Moreover, multimodal datasets to date are found heavily skewed towards outdoor
or urban scenarios, with the indoor datasets mainly dictated by the visual or
visual-inertial odometry [11, 12]. Unfortunately, such a scenario skewness
does not coincide with the recent breakthroughs of miniaturised sensors such
as single-chip mmWave radars, solid-state sensors and portable thermal
cameras. These miniaturized sensors not only become increasingly cost-
effective and portable than ever, but also complement the electromagnetic (EM)
spectrum covered by conventional navigation sensors. For example, compared
with the cameras that only occupy the visible spectrum, mmWave radars and Long
Wave Infrared (LWIR) thermal cameras operate at 77 GHz ($4\,mm$ wavelength)
and 30 THz ($10\,\mu m$ wavelength) respectively. The extra EM spectrum
covered by these emerging sensors can in turn, yield more sensing robustness
against poor illumination, airborne particles and visual aliasing etc.

Towards a more _comprehensive_ multimodal dataset for odometry evaluation,
this work presents a new odometry dataset constituted of the sensory data
covering a wide range of electromagnetic frequencies (c.f., Fig.1). To our
best knowledge, this presented dataset features the widest range of mobile
platforms and includes robust navigation sensors that emerged recently.
Specifically, we recorded 6-Degree-of-Freedom (6DoF) motion with accurate
ground truth on (i) Unmanned Aerial Vehicle (UAV), (ii) Unmanned Ground
Vehicle (UGV) and (iii) handheld platforms. We equip these platforms with the
emerging mmWave radars, portable thermal cameras and solid-state LiDARs, co-
located with the traditional navigation modalities such as spinning LiDAR,
RGB-D camera and IMUs. We hope that by making available our data from both
emerging and conventional sensors, unprecedented sensor fusion approaches can
be yielded on various platforms and different use cases in the wild. The main
contributions of our datasets are summarised as follows:

  * •

We present an indoor odometry dataset containing the data from both emerging
and traditional navigation sensors on diverse platforms. This is the first
indoor dataset that simultaneously covers the mmWave and LWIR thermal
spectrum.

  * •

We record the odometry data in 7 different categories of indoor scenes which
feature dynamic illumination and various motion patterns. The whole dataset
exceeds 10 km in trajectory length.

  * •

We provide extrinsic calibration parameters for all platforms and intrinsic
parameters for all optic sensors, accompanied by several plug-and-play codes
for developing multimodal odometry systems with our dataset.

Figure 2: Our sensor setup for UAV, UGV and handheld platform.

##  II Related Work

A multitude of odometry datasets have been proposed over the last decades, but
most of them focus on common visual or LiDAR modalities, while ignoring the
emerging radar and thermal sensors. We refer the readers interested in
traditional multimodal odometry datasets to [13, 14], and discuss the work
related to:

LWIR Thermal Cameras. Recent years have seen multiple indoor datasets
utilizing the LWIR thermal spectrum. The SubT-Tunnel dataset [15] features
trajectories in an underground mine from the DARPA Subterranean challenge.
Since it is based on the UGV platform only, its movement is restricted to 2D.
Indoor trajectories in the ViVid dataset [7] are limited to their Vicon room.
This guarantees they have accurate ground truth pose and trajectory, but it
also means they do not contain enough scene variety. The multi-spectral
dataset [9] is another RGB/thermal camera dataset featuring hardware time
synchronization.

mmWave Radars. In the meantime, datasets that include radars are mostly
recorded for developing advanced driver assistance systems (ADAS). Notable
examples include Oxford Radar RobotCar [5] and nuScenes [16] datasets.
However, the radar used in [5] is a mechanical spinning radar while the
single-chip radar used in [16] can only give point clouds in a 2D plane. The
only radar dataset comparable to ours in terms of sensors and scenes is the
ColoRadar dataset [8], which features a single-chip mmWave radar, and a
cascaded radar. However, the lack of RGB, thermal and Stereo modalities means
it cannot support the comparison between radar algorithms and other more
popular modalities in the same situation. Furthermore, due to the sparse point
cloud generated by radars, our wide-fov 3-radar setup on the UGV platform
provides a lot more information than the forward-facing radar setup in
ColoRadar.

Unlike all the above works, to our best knowledge, our dataset is the first
one that _simultaneously_ contain the LWIR camera and mmWave radar sensors,
while featuring long trajectories in diversified indoor environments, wide-
angle radar arrays, and three different platforms.

Mobile Platforms. Platform-wise, different mobile platforms have different
movement characteristics and impose different challenges for odomerty
estimation. Compared to other indoor multi-modal datasets that mostly only use
handheld platform[7, 9], our dataset is the only one that includes all three
viable mobile agents in an indoor environment: UAV, UGV and handheld.

##  III Dataset

###  III-A Sensor Setup

(a)

(b)

(c)

(d)

(e)

Figure 3: Examples of Scene Variety - top row from RGB cameras and bottom row
from co-located LWIR thermal cameras. We took our sensor rigs to 4 different
buildings to collect a variety of scenes. (a) to (c) show different
illumination levels, from bright with glares to dimmed to complete darkness.
(d) shows large quantities of reflective and transparent surfaces, while (e)
shows complex 3D topology in the canteen area. TABLE II: Sensor specs. on
three platforms. Handheld
---
1x RGBD Camera |  | Intel RealSense D435i stereo camera
---
20Hz 8bit 640x480 RGB image
30Hz 16bit 640x480 depth image
1x Thermal Camera |  | FLIR Boson 640 thermal camera
---
30Hz 16bit 640x512 thermal image
1x LiDAR |  | Velodyne VLP-16 LiDAR
---
10Hz 16-channel point cloud
1x Radar |  | TI AWR1843BOOST mmWave radar
---
20Hz point cloud
1x IMU |  | Xsens MTi
---
100Hz inertial data and orientation
UGV
1x RGBD Camera |  | Intel RealSense D435i stereo camera
---
30Hz 8bit 640x480 RGB image
30Hz 16bit 640x480 depth image
1x Thermal Camera |  | FLIR Boson 640 thermal camera
---
30Hz 16bit 640x512 thermal image
1x LiDAR |  | Velodyne HDL-32E LiDAR
---
10Hz 32-channel point cloud
3x Radar |  | TI AWR1843BOOST radars at 90deg angles
---
20Hz point cloud
1x IMU |  | Xsens MTi
---
100Hz inertial data and orientation
UAV
1x Solid-state LiDAR |  | Intel RealSense L515 Solid-state LiDAR
---
15Hz 8bit 960x540 RGB image
30Hz 16bit 640x480 depth image
1x Thermal Camera |  | FLIR Boson 640 thermal camera
---
8.7Hz 8bit 640x512 thermal images
1x Radar |  | TI AWR1843BOOST mmWave radar
---
10Hz point cloud
1x IMU |  | Xsens MTi
---
100Hz inertial data and orientation

We collected the data from three platforms, handheld, UAV and UGV, shown in
Fig. 2. An overview of sensors and their platforms and characteristics are
presented in Table. II. The sensor kit includes Intel Realsense D435 stereo
camera, RealSense L515 Solid-state LiDAR, FLIR Boson thermal camera, Velodyne
VLP-16 and VLP-32, TI AWR1843 mmWave radar and Xsens MTi IMU. UGV has the most
room for sensor installment, thus, we used Velodyne HDL-32E and two more
radars than on the Handheld platform. On the UAV platform, we switched D435i
stereo camera and Velodyne LiDAR for an Intel RealSense L515 Solid-state LiDAR
due to the limited payloads of drones.

###  III-B Dataset Collection

Indoor scenes can vary in many aspects, and such data variety impacts many
models, especially learning-based ones, in terms of generalizability and
transferability. In this regard, we collect the dataset under diverse
environments and motion patterns. Detailed environment and motion description
in each sequence is provided along-side the dataset. We give a brief summary
in this section.

####  III-B1 Sequences

Our dataset collects 44 handheld sequences and 75 UGV sequences through
multiple floors in 4 different buildings, providing a large variety of
different visual traits, spatial characteristics and illumination conditions.
Representative samples are shown in Fig. 3. 3D complexity in our dataset like
crowded rooms with densely packed chairs could provide difficulty for range
sensors, while darkness, glare, high dynamic range and texture-less walls
could be challenging to visual sensors. Several of the sequences include
moving targets like humans. This would test a model’s robustness to dynamic
environments. Due to the need for ground truth from the Vicon tracking system,
UAV sequences are recorded in the Vicon room only. All sequences collected
from the above platforms are recorded with their 6-DoF movements.

Sequences are named after their characteristics as
_[Platform]_[Scene]_[TrajectoryShape]_[LightingCondition]_ , examples of which
can be seen in Table III. For platforms, _H_ , _G_ and _A_ denote handheld,
UGV and UAV platforms respectively. For scenes, we divide them into 7
categories: (1)_vicon_ denotes a single room that the platform’s motion is
confined to, (2)_corridor_ means the platform go through narrow long
corridors, (3)_glassy_ denotes environments with large amounts of reflective
surfaces, (4)_atrium_ denotes lightly furnished larger spaces, (5)_hallway_
denotes large unfurnished spaces, while (6)_canteen_ and (7)_office_ are
heavily furnished spaces. For trajectories, _linear_ means long forward-moving
trajectories, while _circular_ and _eight_ denote looped O-shape or 8-shape
trajectories with plenty of loop-closure opportunities. For lighting
conditions, _bright_ denotes normal daylight, _dim_ denotes the sequence
includes under-exposed RGB frames, and _dark_ means the sequence includes
pitch-black RGB frames.

####  III-B2 Motion Characteristics

For fully evaluating odometry algorithms, we included a variety of motion
patterns when collecting data:

_drastic turns_ : The trajectory length ranges from 10 meters to 300 meters.
Drastic turns as large as 180 degrees are included in many of the trajectories
on all three platforms. Many of the turns make the platform face featureless
walls.

_human-controlled operation_ : UAV motion is controlled by a human operator
with stabilization assistance from the Vicon tracking system. Compared to
flight fully controlled by Vicon, control inputs from the human operator are
more complex and mimic a genuine trajectory performed by an exploring
autonomous agent.

###  III-C Calibration

To make the dataset applicable to various processing techniques, we provide
extrinsic parameters of all sensors and intrinsic parameters of all cameras.
Most extrinsic calibrations are done with RGB camera on Realsense D435 as
center. A detailed record of coordinate frames and extrinsic parameters for
all sensors are provided alongside the dataset.

####  III-C1 Calibrate Thermal Camera vs. RGB Camera

For extrinsic parameters between thermal camera and RGB camera, we need a
calibration target that presents good contrast in RGB as well as thermal
spectrum. To achieve this, we implemented the masking method of [17], covering
a hot-running laptop with an acrylic mask with hollowed windows.

(a) Original thermal image

(b) Undistorted thermal image

Figure 4: Thermal masks used to calibrate thermal cameras and their
calibration results.

####  III-C2 Calibrate LiDAR vs. RGB Camera

We implemented the LiDAR vs. RGB camera calibration by Geiger et al.[18],
using a chessboard patterned corner.

####  III-C3 Calibrate mmWave Radar vs. LiDAR

Due to strong reflection of mmWave signal by metal materials, We use a
metallic corner reflector as target to calibrate extrinsic parameters between
mmWave radar and LiDAR. Starting from hand measurements, we tune the extrinsic
parameter between mmWave radar and LiDAR by finding the best alignment between
the corner reflector’s signal from the two sensors.

####  III-C4 Calibrate IMU vs. RGB Camera

Extrinsic parameters between IMU and RGB camera are obtained using kalibr
tools[19] with AprilTag target.

###  III-D Ground Truth

For sequences that are collected in large-scale spaces with long trajectories,
it’s unrealistic to use motion tracking systems (e.g., Vicon) to collect
ground truth motions. We follow the practices of [12, 20] to use more accurate
odometry modalities as pseudo ground truth for modalities with much less
accuracy.

For the UGV platform, we used wheel odometry as the pseudo ground truth. For
handheld platform, we use trajectories generated by the SOTA LiDAR odometry,
ALOAM [1], as pseudo ground truth. For the UAV platform, whose trajectories
are confined to the Vicon lab, the ground truth comes from the Vicon tracking
system, which provides mm-level accuracy 6-DoF poses in 100 Hz.

Figure 5: Sample Trajectories of TIO and RIO results on UAV, UGV and Handheld
platform. Dot denotes the starting point.

##  IV Experiments

Since the LWIR thermal and mmWave radar spectrum are our particular
contribution, we will focus on evaluating our novel thermal and radar data for
odometry purposes. Since there are few open-source models involving the fusion
of more than one spectrum, we will be implementing radar odometry (RO), radar-
inertial odometry (RIO) and thermal-inertial odometry (TIO) methods to
estimate 6-DoF trajectories. To demonstrate the dataset in different use-
cases, odometry using radar and thermal will each be implemented using a
learning-based method and a none-learning based method.

We use two common metrics, Absolute Trajectory Error (ATE)[21] and Relative
Pose Error (RPE) [22], to evaluate 6-DoF odometry on our dataset. We perform a
train/test split on the dataset that ensures the scenes are evenly split.
Along with our dataset, we provide data pre-processing, model training, and
model testing codes.

###  IV-A Odometry with mmWave Radar

MilliEgo [23] is a first-of-its-kind deep learning framework for radar-
inertial odometry. It uses Convolutional Neural Network (CNN) and Recurrent
Neural Network (RNN) to encode radar and IMU streams respectively, after which
the 2 encoded streams are fused together using mixed-attention mechanism. The
fused sensor features are temporally processed by long short-term memory
(LSTM) and regressed by fully-connected (FC) layers.

We trained the network using its open-sourced configuration on our handheld,
UGV and UAV platforms, generating pose predictions for each platform at 4 Hz
(UGV), 5 Hz (UAV) and 6.7 Hz (handheld). The results shown in Fig. 5 and Table
III demonstrate that milliEgo is able to generate good trajectories, with
acceptable drift even in long trajectories. As expected, milliEgo performs
better on the UGV platform than handheld and UAV platforms with the wide-fov
3-radar array on the UGV compared to only 1 radar on the other platforms.

For the non-learning based method, we implemented a Radar-ICP model, which
uses iterative closest point (ICP)[24] to register radar frames at fixed
intervals. Adjacent point cloud frames are overlaid to reduce noise. As can be
seen in Table III, the trajectories proved difficult for Radar-ICP to
estimate. The high noise level in the mmWave radar’s data can explain the
large error in Radar-ICP, which fails to match point cloud frames with many
points flickering on and off. Like milliEgo, Radar-ICP also shows better
results with 3-radar setup than the 1-radar handheld and UAV platform.

TABLE III: Performance of Different Methods on Test Sequences: Mean Absolute Trajectory Errors (ATE) [m] and Mean Relative Pose Error (RPE). RPE shown below is in the format of Translation[m]/Rotation[deg]. |  | milliEgo[23] | DeepTIO[25] | ICP[24]
---|---|---|---|---
Test Sequence | Length[m]/Duration[s] | ATE | RPE | ATE | RPE | ATE | RPE
Handheld |  |  |  |  |  |
H_corridor_circle_bright_5 | 141.5/162.7 | 8.72 | 0.071/2.44 | 8.29 | 0.087/4.18 | 15.82 | 0.338/13.56
H_vicon_circle_dark_3 | 58.3/87.0 | 2.49 | 0.056/3.20 | 2.35 | 0.067/5.26 | 5.61 | 0.296/13.03
H_vicon_circle_dark_4 | 43.1/61.3 | 1.24 | 0.064/2.88 | 5.55 | 0.072/5.13 | 3.14 | 0.278/12.69
H_viconcorridor_circle_bright_3 | 104.2/114.3 | 2.70 | 0.062/2.68 | 3.36 | 0.076/4.51 | 12.45 | 0.271/12.73
H_office_circle_bright_2 | 76.0/94.8 | 2.29 | 0.061/3.77 | 5.35 | 0.086/5.83 | 7.33 | 0.345/17.06
H_atrium_linear_bright_3 | 146.6/122.6 | 4.37 | 0.078/2.95 | 5.54 | 0.101/4.58 | 13.09 | 0.388/11.11
H_multifloor_linear_1 | 318.0/261.8 | 7.91 | 0.096/3.46 | 22.54 | 0.114/5.04 | 18.98 | 0.368/11.24
H_hallway_circle_bright_2 | 202.2/282.9 | 16.00 | 0.071/4.03 | 13.39 | 0.083/6.26 | 24.81 | 0.317/15.49
H_glassy_circle_bright_2 | 73.6/86.1 | 7.18 | 0.077/2.33 | 3.58 | 0.082/3.52 | 8.27 | 0.314/9.96
H_canteen_circle_bright_1 | 153.4/159.3 | 4.98 | 0.080/3.88 | 5.34 | 0.090/5.55 | 10.32 | 0.395/15.77
H_atrium_circle_bright_1 | 146.8/147.8 | 7.47 | 0.074/3.05 | 7.24 | 0.084/4.68 | 7.88 | 0.400/15.03
UGV |  |  |  |  |  |
G_vicon_eight_bright_1 | 24.6/63.2 | 0.54 | 0.022/2.99 | 0.46 | 0.014/1.77 | 1.40 | 0.119/4.83
G_stairwell_linear_bright_1 | 62.5/139.6 | 1.90 | 0.017/1.50 | 1.24 | 0.014/0.87 | 7.71 | 0.091/3.18
G_atrium_linear_bright_2 | 91.6/209.1 | 2.62 | 0.018/1.82 | 3.23 | 0.015/0.97 | 29.68 | 0.100/3.73
G_canteen_circle_bright_1 | 71.0/171.7 | 1.95 | 0.019/2.19 | 2.86 | 0.017/1.27 | 5.58 | 0.102/4.16
G_hallway_linear_bright_1 | 78.7/183.3 | 2.19 | 0.019/1.58 | 4.20 | 0.015/0.95 | 10.01 | 0.096/3.75
G_hallway_linear_bright_3 | 66.0/148.87 | 2.07 | 0.019/1.67 | 3.98 | 0.014/0.95 | 6.93 | 0.101/3.72
G_corridor_linear_bright_11 | 40.2/93.0 | 2.38 | 0.017/1.83 | 1.50 | 0.016/1.06 | 9.39 | 0.111/3.88
G_corridor_linear_bright_13 | 25.1/57.2 | 1.32 | 0.018/1.32 | 1.35 | 0.013/0.78 | 3.44 | 0.092/3.55
G_vicon_eight_bright_5 | 61.3/150.0 | 0.80 | 0.019/2.44 | 0.56 | 0.015/1.35 | 2.26 | 0.112/6.02
G_corridor_linear_bright_18 | 39.3/88.3 | 0.77 | 0.018/1.00 | 2.28 | 0.015/0.56 | 3.11 | 0.098/3.33
UAV |  |  |  |  |  |
A_vicon2_circle_bright_6 | 5.55/40.0 | 1.27 | 0.023/0.75 | 0.62 |  0.020/0.96 | 2.51 | 0.451/13.30
A_vicon2_circle_bright_7 | 11.27/40.0 | 0.45 | 0.024/1.41 | 0.34 |  0.022/1.78 | 3.68 | 0.508/18.54
A_vicon2_circle_bright_8 | 10.75/40.0 | 1.33 | 0.032/1.47 | 0.45 |  0.028/1.89 | 6.49 | 0.494/16.75
A_vicon2_circle_bright_9 | 10.98/40.0 | 1.13 | 0.027/1.11 | 0.55 |  0.025/1.52 | 3.32 | 0.510/15.85
A_vicon2_circle_bright_10 | 11.14/40.0 | 0.76 | 0.025/1.20 | 0.44 |  0.023/1.62 | 5.72 | 0.737/22.45

###  IV-B Odometry with Thermal Camera

Existing Visual-Inertial Odometry (VIO) models can be converted to a TIO since
they share the same input format. Thus, for our non-learning TIO model, we
adapted the popular visual-inertial state estimation framework VINS-Mono [3]
to TINS-Mono by taking the thermal image as input. Different from the random
noise pattern of RGB sensors, thermal sensors accumulate noise on fixed pixel
locations (fixed-pattern noise) for a short duration, in which those noises
will be typically suppressed through the camera’s Non-Uniformity Correction
(NUC), but could still be severe enough to affect models. [26]. Feature
extraction techniques used by TINS-Mono could mistake those fixed noise
artifacts as visual features. To mitigate this delusion, TINS-Mono requires
the thermal images to go through a series of pre-processing including contrast
adjustments, denoising and sharpening. As shown in Fig. 6, the TINS-Mono is
able to stably track features in most scenarios after pre-processing, but the
shortage and inconsistency in visual features makes the model underestimate
translation motion, and it would still understandably lose track in the
presence of thermally-flat images.

(a)

(b)

(c)

Figure 6: TINS-Mono tracking and trajectory. (a) and (b) shows TINS-Mono
feature tracking before and after our pre-processing. Red dots are features
successfully matched with previous frames, while blue dots are failed matches.
Noise artifacts are falsely extracted as features in (a). The trajectory shows
TINS-Mono severely underestimates translation motion.

DeepTIO [25] is a deep learning thermal-inertial odometry method aimed at
alleviating the lack of visual features in thermal images. It benefits from a
hallucination network that aims to hallucinate upon the thermal images such
that it can produce similar visual features as the ones extracted from RGB
cameras. In a word, it is equivalent to fusing together 3 modalities:
(hallucinated) visual, thermal, and IMU.

We trained the network on handheld, UAV and UGV platform with exactly the same
configuration, giving out pose predictions at a rate of 6 Hz (on handheld and
UGV) and 4.3 Hz (on UAV). Before training, a pre-trained VINet [27] was used
to first generate the visual features from the RGB stream. In the first stage
of training, the hallucination network was trained to generate similar
features from thermal stream. In the second stage, the rest of the network was
trained while the hallucination layers were frozen. The estimated trajectories
for DeepTIO are shown in Fig.5 and the quantitative results are presented in
Table. III. Notably, DeepTIO doesn’t show noticeable drop in performance in
pitch-dark sequences where Visual-Inertial models would completely fail. The
better performance of DeepTIO on the UGV than on the handheld may be a result
of the unstable motion and faster speed of the handheld devices, making
feature tracking slightly more challenging. A customized and tailored network
configuration might be required to generate a better trajectory in this
challenging scenario.

To summarize our experiments, the evaluated methods show that sensible
odometry results can be generated from our dataset using LWIR and mmWave
spectrum, but their performance still leaves much to be desired on our
challenging dataset. With the two non-learning methods, Radar-ICP and TINS-
Mono, we demonstrated the common challenges of the LWIR thermal and mmWave
modalities: the high level of noise and the sparsity of features. Comparing
results of non-learning and learning based methods, it appears learning based
methods consistently outperform non-learning based one across three platforms,
while also avoiding manual tuning of model parameters. Platform-wise, handheld
and UAV platforms prove to be more challenging than UGV due to their complex
6-DoF movements. We hope that our diverse and challenging dataset would
facilitate multi-modal and multi-platform odometry research in the future.

##  V Conclusion

This paper presents the OdomBeyondVision dataset. By including the traditional
navigation sensors and the emerging single-chip mmWave radar and LWIR cameras,
our odometry dataset features a wide sensor coverage across both visible and
invisible spectrums. We collected the odometry data with multiple different
mobile platforms in a variety of indoor scenes and illumination, aiming to
bridge existing gaps and further diversify odometry datasets in the community.
We demonstrated the use of the dataset in exemplar odometry systems and
released their implementations for plug-and-play usage. Owing to the
increasing interest in localization and navigation research, our plan in the
near future is to provide better dataset services with more benchmarking
results.

## References

  * [1] J. Zhang and S. Singh, “Loam: Lidar odometry and mapping in real-time.” in _Robotics: Science and Systems_ , vol. 2, no. 9, 2014.
  * [2] C. Campos, R. Elvira, J. J. G. Rodríguez, J. M. Montiel, and J. D. Tardós, “Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam,” _IEEE Transactions on Robotics_ , 2021\.
  * [3] T. Qin, P. Li, and S. Shen, “Vins-mono: A robust and versatile monocular visual-inertial state estimator,” _IEEE Transactions on Robotics_ , vol. 34, no. 4, pp. 1004–1020, 2018.
  * [4] Y. Choi, N. Kim, S. Hwang, K. Park, J. S. Yoon, K. An, and I. S. Kweon, “Kaist multi-spectral day/night data set for autonomous and assisted driving,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 19, no. 3, pp. 934–948, 2018.
  * [5] D. Barnes, M. Gadd, P. Murcutt, P. Newman, and I. Posner, “The oxford radar robotcar dataset: A radar extension to the oxford robotcar dataset,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 6433–6438.
  * [6] Y. Cheng, M. Jiang, J. Zhu, and Y. Liu, “Are we ready for unmanned surface vehicles in inland waterways? the usvinland multisensor dataset and benchmark,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 3964–3970, 2021.
  * [7] A. J. Lee, Y. Cho, S. Yoon, Y.-S. Shin, and A. Kim, “Vivid : Vision for visibility dataset,” 2019.
  * [8] A. Kramer, K. Harlow, C. Williams, and C. Heckman, “Coloradar: The direct 3d millimeter wave radar dataset,” _arXiv preprint arXiv:2103.04510_ , 2021\.
  * [9] W. Dai, Y. Zhang, S. Chen, D. Sun, and D. Kong, “A dataset for evaluating multi-spectral motion estimation methods,” _arXiv preprint arXiv:2007.00622_ , 2020.
  * [10] G. Kim, Y. S. Park, Y. Cho, J. Jeong, and A. Kim, “Mulran: Multimodal range dataset for urban place recognition,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 6246–6253.
  * [11] D. Schubert, T. Goll, N. Demmel, V. Usenko, J. Stueckler, and D. Cremers, “The tum vi benchmark for evaluating visual-inertial odometry,” in _International Conference on Intelligent Robots and Systems (IROS)_ , October 2018.
  * [12] S. Cortés, A. Solin, E. Rahtu, and J. Kannala, “Advio: An authentic dataset for visual-inertial odometry,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 419–434.
  * [13] H. Zhan, C. S. Weerasekera, J.-W. Bian, and I. Reid, “Visual odometry revisited: What should be learnt?” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 4203–4210.
  * [14] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The kitti dataset,” _The International Journal of Robotics Research_ , vol. 32, no. 11, pp. 1231–1237, 2013.
  * [15] J. G. Rogers, J. M. Gregory, J. Fink, and E. Stump, “Test your slam! the subt-tunnel dataset and metric for mapping,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 955–961.
  * [16] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 11 621–11 631.
  * [17] S. Vidas, R. Lakemond, S. Denman, C. Fookes, S. Sridharan, and T. Wark, “A mask-based approach for the geometric calibration of thermal-infrared cameras,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 61, no. 6, pp. 1625–1635, 2012.
  * [18] A. Geiger, F. Moosmann, Ö. Car, and B. Schuster, “Automatic camera and range sensor calibration using a single shot,” in _2012 IEEE international conference on robotics and automation_. IEEE, 2012, pp. 3936–3943.
  * [19] J. Rehder, J. Nikolic, T. Schneider, T. Hinzmann, and R. Siegwart, “Extending kalibr: Calibrating the extrinsics of multiple imus and of individual axes,” in _2016 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2016, pp. 4304–4311.
  * [20] C. Doer and G. F. Trommer, “Yaw aided radar inertial odometry using manhattan world assumptions,” in _2021 28th Saint Petersburg International Conference on Integrated Navigation Systems (ICINS)_. IEEE, 2021, pp. 1–9.
  * [21] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in _2012 IEEE conference on computer vision and pattern recognition_. IEEE, 2012, pp. 3354–3361.
  * [22] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “A benchmark for the evaluation of rgb-d slam systems,” in _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2012, pp. 573–580.
  * [23] C. X. Lu, M. R. U. Saputra, P. Zhao, Y. Almalioglu, P. P. de Gusmao, C. Chen, K. Sun, N. Trigoni, and A. Markham, “milliego: single-chip mmwave radar aided egomotion estimation via deep sensor fusion,” in _Proceedings of the 18th Conference on Embedded Networked Sensor Systems_ , 2020, pp. 109–122.
  * [24] S. Rusinkiewicz and M. Levoy, “Efficient variants of the icp algorithm,” in _Proceedings third international conference on 3-D digital imaging and modeling_. IEEE, 2001, pp. 145–152.
  * [25] M. R. U. Saputra, P. P. de Gusmao, C. X. Lu, Y. Almalioglu, S. Rosa, C. Chen, J. Wahlström, W. Wang, A. Markham, and N. Trigoni, “Deeptio: A deep thermal-inertial odometry with visual hallucination,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 1672–1679, 2020.
  * [26] M. R. U. Saputra, C. X. Lu, P. P. de Gusmao, B. Wang, A. Markham, and N. Trigoni, “Graph-based thermal-inertial slam with probabilistic neural networks,” _arXiv preprint arXiv:2104.07196_ , 2021.
  * [27] R. Clark, S. Wang, H. Wen, A. Markham, and N. Trigoni, “Vinet: Visual-inertial odometry as a sequence-to-sequence learning problem,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 31, no. 1, 2017.