# CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery

Michael Laskin  Hao Liu  Xue Bin Peng  Denis Yarats  Aravind Rajeswaran
Pieter Abbeel

###### Abstract

We introduce Contrastive Intrinsic Control (CIC), an algorithm for
unsupervised skill discovery that maximizes the mutual information between
state-transitions and latent skill vectors. CIC utilizes contrastive learning
between state-transitions and skills to learn behavior embeddings and
maximizes the entropy of these embeddings as an intrinsic reward to encourage
behavioral diversity. We evaluate our algorithm on the Unsupervised
Reinforcement Learning Benchmark, which consists of a long reward-free pre-
training phase followed by a short adaptation phase to downstream tasks with
extrinsic rewards. CIC substantially improves over prior methods in terms of
adaptation efficiency, outperforming prior unsupervised skill discovery
methods by $1.79\times$ and the next leading overall exploration algorithm by
$1.18\times$.111Project website and code: https://sites.google.com/view/cicrl/

Machine Learning, ICML

##  1 Introduction

Figure 1: This work deals with unsupervised skill discovery through mutual
information maximization. We introduce Contrastive Intrinsic Control (CIC) – a
new unsupervised RL algorithm that explores and adapts more efficiently than
prior methods.

Deep Reinforcement Learning (RL) is a powerful approach toward solving complex
control tasks in the presence of extrinsic rewards. Successful applications
include playing video games from pixels (Mnih et al., 2015), mastering the
game of Go (Silver et al., 2017, 2018), robotic locomotion (Schulman et al.,
2016, 2017; Peng et al., 2018) and dexterous manipulation (Rajeswaran et al.,
2018; OpenAI, 2018, 2019) policies. While effective, the above advances
produced agents that are unable to generalize to new downstream tasks beyond
the one they were trained to solve. Humans and animals on the other hand are
able to acquire skills with minimal supervision and apply them to solve a
variety of downstream tasks. In this work, we seek to train agents that
acquire skills without supervision with generalization capabilities by
efficiently adapting these skills to downstream tasks.

Over the last few years, unsupervised RL has emerged as a promising framework
for developing RL agents that can generalize to new tasks. In the unsupervised
RL setting, agents are first pre-trained with self-supervised intrinsic
rewards and then finetuned to downstream tasks with extrinsic rewards.
Unsupervised RL algorithms broadly fall into three categories - knowledge-
based, data-based, and competence-based methods222These categories for
exploration algorithms were introduced by Srinivas & Abbeel (2021) and
inspired by Oudeyer et al. (2007).. Knowledge-based methods maximize the error
or uncertainty of a predictive model (Pathak et al., 2017, 2019; Burda et al.,
2019). Data-based methods maximize the entropy of the agent’s visitation (Liu
& Abbeel, 2021a; Yarats et al., 2021b). Competence-based methods learn skills
that generate diverse behaviors (Eysenbach et al., 2019; Gregor et al., 2017).
This work falls into the latter category of competence-based exploration
methods.

Figure 2: Qualitative visualizations of unsupervised skills discovered in
Walker, Quadruped, and Jaco arm environments. The Walker learns to balance and
move, the Quadruped learns to flip upright and walk, and the 6 DOF robotic arm
learns how to move without locking. Unlike prior competence-based methods for
continuous control which evaluate on OpenAI Gym (e.g. Eysenbach et al.
(2019)), which reset the environment when the agent loses balance, CIC is able
to learn skills in fixed episode length environments which are much harder to
explore (see Appendix J).

Unlike knowledge-based and data-based algorithms, competence-based algorithms
simultaneously address both the exploration challenge as well as distilling
the generated experience in the form of reusable skills. This makes them
particularly appealing, since the resulting skill-based policies (or skills
themselves) can be finetuned to efficiently solve downstream tasks. While
there are many self-supervised objectives that can be utilized, our work falls
into a family of methods that learns skills by maximizing the mutual
information between visited states and latent skill vectors. Many earlier
works have investigated optimizing such objectives (Eysenbach et al., 2019;
Gregor et al., 2017; Kwon, 2021; Sharma et al., 2020). However, competence-
based methods have been empirically challenging to train and have under-
performed when compared to knowledge and data-based methods (Laskin et al.,
2021).

In this work, we take a closer look at the challenges of pre-training agents
with competence-based algorithms. We introduce Contrastive Intrinsic Control
(CIC) – an exploration algorithm that uses a new estimator for the mutual
information objective. CIC combines particle estimation for state entropy
(Singh et al., 2003; Liu & Abbeel, 2021a) and noise contrastive estimation
(Gutmann & Hyvärinen, 2010) for the conditional entropy which enables it to
both generate diverse behaviors (exploration) and discriminate high-
dimensional continuous skills (exploitation). To the best of our knowledge,
CIC is the first exploration algorithm to utilize noise contrastive estimation
to discriminate between state transitions and latent skill vectors.
Empirically, we show that CIC adapts to downstream tasks more efficiently than
prior exploration approaches on the Unsupervised Reinforcement Learning
Benchmark (URLB). CIC achieves $79\%$ higher returns on downstream tasks than
prior competence-based algorithms and $18\%$ higher returns than the next-best
exploration algorithm overall.

##  2 Background and Notation

Markov Decision Process: We operate under the assumption that our system is
described by a Markov Decision Process (MDP) (Sutton & Barto, 2018). An MDP
consiss of the tuple $(\mathcal{S},\mathcal{A},\mathcal{P},r,\gamma)$ which
has states $s\in\mathcal{S}$, actions $a\in\mathcal{A}$, transition dynamics
$p(s^{\prime}|s,a)\sim\mathcal{P}$, a reward function $r$, and a discount
factor $\gamma$. In an MDP, at each timestep $t$, an agent observes the
current state $s$, selects an action from a policy $a\sim\pi(\cdot|s)$, and
then observes the reward and next state once it acts in the environment:
$r,s^{\prime}\sim\text{env.step}(a)$. Note that usually $r$ refers to an
extrinsic reward. However, in this work we will first be pre-training an agent
with intrinsic rewards $r^{\text{int}}$ and finetuning on extrinsic rewards
$r^{\text{ext}}$.

Figure 3: Architecture illustrating the practical implementation of CIC .
During a gradient update step, random $\tau=(s,s^{\prime})$ tuples are sampled
from the replay buffer, then a particle estimator is used to compute the
entropy and a noise contrastive loss to compute the conditional entropy. The
contrastive loss is backpropagated through the entire architecture. The
entropy and contrastive terms are then scaled and added to form the intrinsic
reward. The RL agent is optimized with a DDPG (Lillicrap et al., 2016).

For convenience we also introduce the variable $\tau(s)$ which refers to any
function of the states $s$. For instance $\tau$ can be a single state, a pair
of states, or a sequence depending on the algorithm. Our method uses
$\tau=(s,s^{\prime})$ to encourage diverse state transitions while other
methods have different specifications for $\tau$. Importantly, $\tau$ does not
denote a state-action trajectory, but is rather shorthand for any function of
the states encountered by the agent. In addition to the standard MDP notation,
we will also be learning skills $z\in\mathcal{Z}$ and our policy will be
skill-conditioned $a\sim\pi(\cdot|s,z)$.

Unsupervised Skill Discovery through Mutual Information Maximization:  Most
competence-based approaches to exploration maximize the mutual information
between states and skills. Our work and a large body of prior research
(Eysenbach et al., 2019; Sharma et al., 2020; Gregor et al., 2017; Achiam et
al., 2018; Lee et al., 2019; Liu & Abbeel, 2021b) aims to maximize a mutual
information objective with the following general form:

$\displaystyle
I(\tau;z)=\mathcal{H}(z)-\mathcal{H}(z|\tau)=\mathcal{H}(\tau)-\mathcal{H}(\tau|z)$
(1)

Competence-based algorithms use different choices for $\tau$ and can condition
on additional information such as actions or starting states. For a full
summary of competence-based algorithms and their objectives see Table 3 in
Appendix D.

Lower Bound Estimates of Mutual Information:  The mutual information $I(s;z)$
is intractable to compute directly. Since we wish to maximize $I(s;z)$, we can
approximate this objective by instead maximizing a lower bound estimate. Most
known mutual information maximization algorithms use the variational lower
bound introduced in Barber & Agakov (2003):

$\displaystyle
I(\tau;z)=\mathcal{H}(z)-\mathcal{H}(z|\tau)\geq\mathcal{H}(z)+\operatorname*{\mathbb{E}}[\log
q(z|\tau)]$ (2)

The variational lower bound can be applied to both decompositions of the
mutual information. The design decisions of a competence-based algorithm
therefore come down to (i) which decomposition of $I(\tau;z)$ to use, (ii)
whether to use discrete or continuous skills, (iii) how to estimate $H(z)$ or
$H(\tau)$, and finally (iv) how to estimate $H(z|\tau)$ or $H(\tau|z)$.

##  3 Motivation

Results from the recent Unsupervised Reinforcement Learning Benchmark (URLB)
(Laskin et al., 2021) show that competence-based approaches underperform
relative to knowledge-based and data-based baselines on DeepMind Control
(DMC). We argue that the underlying issue with current competence-based
algorithms when deployed on harder exploration environments like DMC has to do
with the currently used estimators for $I(\tau;z)$ rather than the objective
itself. To produce structured skills that lead to diverse behaviors,
$I(\tau;z)$ estimators must (i) explicitly encourage diverse behaviors and
(ii) have the capacity to discriminate between high-dimensional continuous
skills. Current approaches do not satisfy both criteria.

Competence-base algorithms do not ensure diverse behaviors: Most of the best
known competence-based approaches (Eysenbach et al., 2019; Gregor et al.,
2017; Achiam et al., 2018; Lee et al., 2019), optimize the first decomposition
of the mutual information $\mathcal{H}(z)-\mathcal{H}(z|\tau)$. The issue with
this decomposition is that while it ensures diversity of skill vectors it does
not ensure diverse behavior from the policy, meaning $\max\mathcal{H}(z)$ does
not imply $\max\mathcal{H}(\tau)$. Of course, if $H(z)-\mathcal{H}(z|\tau)$ is
maximized and the skill dimension is sufficiently large, then
$\mathcal{H}(\tau)$ will also be maximized implicitly. Yet in practice, to
learn an accurate discriminator $q(z|\tau)$, the above methods assume skill
spaces that are much smaller than the state space (see Table 3), and thus
behavioral diversity may not be guaranteed. In contrast, the decomposition
$I(\tau;z)=\mathcal{H}(\tau)-\mathcal{H}(\tau|z)$ ensures diverse behaviors
through the entropy term $\mathcal{H}(\tau)$. Methods that utilize this
decomposition include Liu & Abbeel (2021b); Sharma et al. (2020).

Figure 4: To empirically demonstrate issues inherent to competence-based
exploration methods, we run DIAYN (Eysenbach et al., 2019) and compare it to
ICM (Pathak et al., 2017) and a Fixed baseline where the agent receives an
intrinsic reward of 1.0 for each timestep and no extrinsic reward on both
OpenAI Gym (episode resets when agent loses balance) and DeepMind Control
(DMC) (episode is fixed for 1k steps) Hopper environments. Since Gym and DMC
rewards are on different scales, we normalize rewards based on the maximum
reward achieved by any algorithm ( 1k for Gym, 3 for DMC). While DIAYN is able
to achieve higher extrinsic rewards than ICM on Gym, the Fixed intrinsic
reward baseline performs best. However, on DMC the Fixed and DIAYN agents
achieve near-zero reward while ICM does not. This is consistent with findings
of prior work that DIAYN is able to learn diverse behaviors in Gym (Eysenbach
et al., 2019) as well as the observation that DIAYN performs poorly on DMC
environments (Laskin et al., 2021)

Why it is important to utilize high-dimensional skills: Once a policy is
capable of generating diverse behaviors, it is important that the
discriminator can distill these behaviors into distinct skills. If the set of
behaviors outnumbers the set of skills, this will result in degenerate skills
– when one skill maps to multiple different behaviors. It is therefore
important that the discriminator can accommodate continuous skills of
sufficiently high dimension. Empirically, the discriminators used in prior
work utilize only low-dimensional continuous skill vectors. DIAYN (Eysenbach
et al., 2019) utilized 16 dimensional skills, DADS (Sharma et al., 2020)
utilizes continuous skills of dimension $2-5$, while APS (Liu & Abbeel,
2021b), an algorithm that utilizes successor features (Barreto et al., 2016;
Hansen et al., 2020) for the discriminator, is only capable of learning
continuous skills with dimension $10$. We show how small skill spaces can lead
to ineffective exploration in a simple gridworld setting in Appendix H and
evidence that skill dimension affects performance in Fig. 6.

On the importance of benchmarks for evaluation: While prior competence-based
approaches such as DIAYN (Eysenbach et al., 2019) were evaluated on OpenAI Gym
(Brockman et al., 2016), Gym environment episodes terminate when the agent
loses balance thereby leaking some aspects of extrinsic signal to the
exploration agent. On the other hand, DMC episodes have fixed length. We show
in Fig 4 that this small difference in environments results in large
performance differences. Specifically, we find that DIAYN is able to learn
diverse skills in Gym but not in DMC, which is consistent with both
observations from DIAYN and URLB papers. Due to fixed episode lengths, DMC
tasks are harder for reward-free exploration since agents must learn to
balance without supervision.

##  4 Method

Figure 5:  We report the aggregate statistics using stratified bootstrap
intervals (Agarwal et al., 2021) for 12 downstream tasks on URLB with 10
seeds, so each statistic for each algorithm has 120 seeds in total. We find
that overall, CIC achieves leading performance on URLB in terms of the IQM,
mean, and OG statistics. As recommended by Agarwal et al. (2021), we use the
IQM as our primary performance measure. In terms of IQM, CIC improves upon the
next best skill discovery algorithm (APS) by $79\%$ and the next best
algorithm overall (ProtoRL) by $18\%$.

###  4.1 Contrastive Intrinsic Control

From Section 3 we are motivated to find a lower bound for $I(\tau;z)$ with a
discriminator that is capable of supporting high-dimensional continuous
skills333In high-dimensional state-action spaces the number of distinct
behaviors can be quite large.. Additionally, we wish to increase the diversity
of behaviors so that the discriminator can continue learning new skills
throughout training. To improve the discriminator, we propose to utilize noise
contrastive estimation (NCE) (Gutmann & Hyvärinen, 2010) between state-
transitions and latent skills as a lower bound for $I(\tau;z)$.444Note that
$\tau$ is not a trajectory but some function of states. It has been shown
previously that such estimators provide a valid lower bound for mutual
information (Oord et al., 2018). However, to the best of our knowledge, this
is the first work to investigate contrastive representation learning for
intrinsic control.

Representation Learning: Specifically, we propose to learn embeddings with the
following representation learning objective, which is effectively CPC between
state-transitions and latent skills:

$\displaystyle
I(\tau;z)\geq\operatorname*{\mathbb{E}}[f(\tau,z)-\log\frac{1}{N}\sum_{j=1}^{N}\exp(f(\tau_{j},z))].$
(3)

where $f(\tau,z)$ is any real valued function. For convenience, we define the
discriminator $\log q(\tau|z)$ as

$\displaystyle\log q(\tau|z)\coloneqq f(\tau,z)$
$\displaystyle-\log\frac{1}{N}\sum_{j=1}^{N}\exp(f(\tau_{j},z)).$ (4)

For our practical algorithm, we parameterize this function as
$f(\tau,z)=g_{\psi_{1}}(\tau)^{\top}g_{\psi_{2}}(z)/{\|g_{\psi_{1}}(\tau)\|\|g_{\psi_{2}}(z)\|T}$
where $\tau=(s,s^{\prime})$ is a transition tuple, $g_{\psi_{k}}$ are neural
encoders, and $T$ is a temperature parameter. This inner product is similar to
the one used in SimCLR (Chen et al., 2020).

The representation learning loss backpropagates gradients from the NCE loss
which maximizes similarity between state-transitions and corresponding skills.

$\displaystyle F_{NCE}(\tau)$
$\displaystyle=\frac{g_{\psi_{1}}(\tau_{i})^{\top}g_{\psi_{2}}(z_{i})}{\|g_{\psi_{1}}(\tau_{i})\|\|g_{\psi_{2}}(z_{i})\|T}$
$\displaystyle-$
$\displaystyle\log\frac{1}{N}\sum_{j=1}^{N}\exp\left(\frac{g_{\psi_{1}}(\tau_{j})^{\top}g_{\psi_{2}}(z_{i})}{\|g_{\psi_{1}}(\tau_{j})\|\|g_{\psi_{2}}(z_{i})\|T}\right)$
(5)

We provide pseudocode for the CIC representation learning loss below:

⬇

1"""

2PyTorch-like pseudocode for the CIC loss

3"""

4

5def cic_loss(s, s_next, z, temp):

6 """

7 - states: s, s_next (B, D)

8 - skills: z (B, D)

9 """

10

11 tau = concat(s, s_next, dim=1)

12

13 query = query_net(z)

14 key = key_net(tau)

15

16 query = normalize(query, dim=1)

17 key = normalize(key, dim=1)

18

19 """

20 positives are on diagonal

21 negatives are off diagonal

22 """

23

24 logits = matmul(query, key.T) / temp

25 labels = arange(logits.shape[0])

26

27 loss = cross_entropy(logits, labels)

28

29 return loss

Listing 1: Pseudocode for the CIC loss

Intrinsic reward: Although we have a representation learning objective, we
still need to specify the intrinsic reward for the algorithm for which there
can be multiple choices. Prior works consider specifying an intrinsic reward
that is proportional to state-transition entropy (Liu & Abbeel, 2021a), the
discriminator (Eysenbach et al., 2019), a similarity score between states and
skills (Warde-Farley et al., 2018), and the uncertainty of the discriminator
(Strouse et al., 2021). We investigate each of these choices and find that an
intrinsic reward that maximizes state-transition entropy coupled with
representation learning via the CPC loss defined in Sec. 4.1 is the simplest
variant that also performs well (see Table 1).

\begin{overpic}[width=104.07117pt]{figures/projection.pdf}
\put(10.0,84.0){{\small(a)}\hskip 5.0pt\small{Skill projection}} \end{overpic}
\begin{overpic}[width=104.07117pt]{figures/skill_size.pdf}
\put(10.0,84.0){{\small(b)}\hskip 5.0pt\small{Skill dimension}} \end{overpic}
\begin{overpic}[width=104.07117pt]{figures/skill_adapt.pdf}
\put(10.0,84.0){{\small(c)}\hskip 5.0pt\small{Skill adaptation}} \end{overpic}
\begin{overpic}[width=104.07117pt]{figures/skill_tuning.pdf}
\put(10.0,84.0){{\small(d)}\hskip 5.0pt\small{Skill grid sweep}} \end{overpic}

Figure 6: Design choices for pre-training and adapting with skills have
significant impact on performance. In (a) and (b) the agent’s zero-shot
performance is evaluated while sampling skills randomly while in (c) and (d)
the agent’s performance is evaluated after finetuning the skills vector. (a)
we show empirically that the projecting skill vectors after sampling them from
noise significantly improves the agent’s performance. (b) The skill dimension
is a crucial hyperparameter and, unlike prior methods, CIC scales to large
skill vectors achieving optimal performance at 64 dimensional skills. (c) We
test several adapation strategies and find that a simple grid search performs
best given the small 4k step adaptation budget, (d) Choosing the right skill
vector has substantial impact on performance and grid sweeping allows the
agent to select the appropriate skill.

For the intrinsic reward, we use a particle estimate (Singh et al., 2003;
Beirlant, 1997) as in Liu & Abbeel (2021a) of the state-transition entropy.
Similar to Liu & Abbeel (2021a); Yarats et al. (2021b) we estimate the entropy
up to a proportionality constant, because we want the agent to maximize
entropy rather than estimate its exact value.

The APT particle entropy estimate is proportional to the distance between the
current visited state transition and previously seen neighboring points.

$\displaystyle\mathcal{H}_{particle}(\tau)\propto\frac{1}{N_{k}}\sum_{h_{i}^{\star}\in
N_{k}}^{N_{k}}\log\|h_{i}-h_{i}^{\star}\|$ (6)

where $h_{i}$ is an embedding of $\tau_{i}$ shown in Fig. 3, $h_{i}^{*}$ is a
kNN embedding, $N_{k}$ is the number of kNNs, and $N-1$ is the number of
negatives. The total number of elements in the summation is $N$ because it
includes one positive.

Explore and Exploit: With these design choices the two components of the CIC
algorithm can be interpreted as exploration with intrinsic rewards and
exploitation using representation learning to distill behaviors into skills.
The marginal entropy maximizes the diversity of state-transition embeddings
while the contrastive discriminator $\log q(\tau|z)$ encourages exploitation
by ensuring that skills $z$ lead to predictable states $\tau$. Together the
two terms incentivize the discovery of diverse yet predictable behaviors from
the RL agent. While CIC shares a similar intrinsic reward structure to APT
(Liu & Abbeel, 2021a), we show that the new representation learning loss from
the CIC estimator results in substantial performance gains in Sec 6.

##  5 Practical Implementation

Our practical implementation consists of two main components: the RL
optimization algorithm and the CIC architecture. For fairness and clarity of
comparison, we use the same RL optimization algorithm for our method and all
baselines in this work. Since the baselines implemented in URLB (Laskin et
al., 2021) use a DDPG555It was recently was shown that a DDPG achieves state-
of-the-art performance (Yarats et al., 2021a) on DeepMind Control (Tassa et
al., 2018) and is more stable than SAC (Haarnoja et al., 2018) on this
benchmark. (Lillicrap et al., 2016) as their backbone, we opt for the same
DDPG architecture to optimize our method as well (see Appendix B).

CIC Architecture: We use a particle estimator as in Liu & Abbeel (2021a) to
estimate $\mathcal{H}(\tau)$. To compute the variational density $q(\tau|z)$,
we first sample skills from uniform noise $z\sim p(z)$ where $p(z)$ is the
uniform distribution over the $[0,1]$ interval. We then use two MLP encoders
to embed $g_{\psi_{1}}(\tau)$ and $g_{\psi_{2}}(z)$, and optimize the
parameters $\psi_{1},\psi_{2}$ with the CPC loss similar to SimCLR (Chen et
al., 2020) since $f(\tau,z)=g_{\psi_{1}}(\tau)^{T}g_{\psi_{2}}(z)$. We fix the
hyperparameters across all domains and downstream tasks. We refer the reader
to the Appendices E and F for the full algorithm and a full list of
hyperparameters.

Adapting to downstream tasks:  To adapt to downstream tasks we follow the same
procedure for competence-based method adaptation as in URLB (Laskin et al.,
2021). During the first 4k environment interactions we populate the DDPG
replay buffer with samples and use the extrinsic rewards collected during this
period to finetune the skill vector $z$. While it’s common to finetune skills
with Cross Entropy Adaptation (CMA), given our limited budget of 4k samples
(only 4 episodes) we find that a simple grid sweep of skills over the interval
$[0,1]$ produces the best results (see Fig. 6). After this, we fix the skill
$z$ and finetune the DDPG actor-critic parameters against the extrinsic reward
for the remaining 96k steps. Note that competence-based methods in URLB also
finetune their skills during the first 4k finetuning steps ensuring a fair
comparison between the methods. The full adaptation procedure is detailed in
Appendix E.

##  6 Experimental Setup

Environments We evaluate our approach on tasks from URLB, which consists of
twelve downstream tasks across three challenging continuous control domains
for exploration algorithms – walker, quadruped, and Jaco arm. Walker requires
a biped constrained to a 2D vertical plane to perform locomotion tasks while
balancing. Quadruped is more challenging due to a higher-dimensional state-
action space and requires a quadruped to in a 3D environment to learn
locomotion skills. Jaco arm is a 6-DOF robotic arm with a three-finger gripper
to move and manipulate objects without locking. All three environments are
challenging in the absence of an extrinsic reward.

Baselines: We compare CIC to baselines across all three exploration
categories. Knowledge-based basedlines include ICM (Pathak et al., 2017),
Disagreement (Pathak et al., 2019), and RND (Burda et al., 2019). Data-based
baselines incude APT (Liu & Abbeel, 2021a) and ProtoRL (Yarats et al., 2021b).
Competence-based baselines include DIAYN (Eysenbach et al., 2019), SMM (Lee et
al., 2019), and APS (Liu & Abbeel, 2021b). The closest baselines to CIC are
APT, which is similar to CIC but without state-skill CPC representation
learning (no discriminator), and APS which uses the same decomposition of
mutual information as CIC and also uses a particle entropy estimate for
$\mathcal{H}(\tau)$. The main difference between APS and CIC is that APS uses
successor features while CIC uses a contrastive estimator for the
discriminator. For further details regarding baselines we refer the reader to
Appendix C.

Evaluation: We follow an identical evaluation to the 2M pre-training setup in
URLB. First, we pre-train each RL agent with the intrinsic rewards for 2M
steps. Then, we finetune each agent to the downstream task with extrinsic
rewards for 100k steps. All baselines were run for 10 seeds per downstream
task for each algorithm using the code and hyperparameters provided by URLB
(Laskin et al., 2021). Built on top of URLB, CIC is also run for 10 seeds per
task. A total of $1080=9\text{ algorithms}\times 12\text{ tasks}\times
10\text{ seeds}$ experiments were run for the main results. Importantly, all
baselines and CIC use a DDPG agent as their backbone.

To ensure that our evaluation statistics are unbiased we use stratified
bootstrap confidence intervals to report aggregate statistics across $M$ runs
with $N$ seeds as described in Rliable (Agarwal et al., 2021) to report
statistics for our main results in Fig. 5. Our primary success metric is the
interquartile mean (IQM) and the Optimality Gap (OG). IQM discards the top and
bottom 25$\%$ of runs and then computes the mean. It is less susceptible to
outliers than the mean and was shown to be the most reliable statistic for
reporting results for RL experiments in Agarwal et al. (2021). OG measures how
far a policy is from optimal (expert) performance. To define expert
performance we use the convention in URLB, which is the score achieved by a
randomly initialized DDPG after 2M steps of finetuning (20x more steps than
our finetuning budget).

##  7 Results

We investigate empirical answers to the following research questions: (Q1) How
does CIC adaptation efficiency compare to prior competence-based algorithms
and exploration algorithms more broadly? (Q2) Which intrinsic reward
instantiation of CIC performs best? (Q3) How do the two terms in the CIC
objective affect algorithm performance? (Q4) How does skill selection affect
the quality of the pre-trained policy? (Q5) Which architecture details matter
most?

Adaptation efficiency of CIC and exploration baslines: Expert normalized
scores of CIC and exploration algorithms from URLB are shown in Fig. 3. We
find that CIC substantially outperforms prior competence-based algorithms
(DIAYN, SMM, APS) achieving a $79\%$ higher IQM than the next best competence-
based method (APS) and, more broadly, achieving a $18\%$ higher IQM than the
next best overall baseline (ProtoRL). In further ablations, we find that the
contributing factors to CIC’s performance are its ability to accommodate
substantially larger continuous skill spaces than prior competence-based
methods.

Intrinsic reward specification: The intrinsic reward for competence-based
algorithms can be instantiated in many different ways. Here, we analyze
intrinsic reward for CIC with the form $r_{int}=H(\tau)+D(\tau,z)$, where $D$
is some function of $(\tau,z)$. Prior works, select $D$ to be (i) the
discriminator (Liu & Abbeel, 2021b), (ii) a cosine similarity between
embeddings (Warde-Farley et al., 2018), (iii) uncertainty of the discriminator
(Strouse et al., 2021), and (iv) just the entropy $D(\tau,z)=0$ (Liu & Abbeel,
2021a). We run CIC with each of these variants on the walker and quadruped
tasks and measure the final mean performance across the downstream tasks (see
Tab. 1). The results show that the entropy-only intrinsic reward performs
best. For this reason the intrinsic reward and representation learning aspects
of CIC are decoupled as shown in Eq. LABEL:eq:rewint. We hypothesize that the
reason why a simple entropy-only intrinsic reward works well is that state-
skill CPC representation learning clusters similar behaviors together. Since
redundant behaviors are clustered, maximizing the entropy of state-transition
embeddings produces increasingly diverse behaviors.

| disc. | similarity | uncertainty | entropy
---|---|---|---|---
walker | 0.80 | 0.79 | 0.78 | 0.82
quad. | 0.44 | 0.63 | 0.75 | 0.74
mean | 0.62 | 0.71 | 0.77 | 0.78
Table 1: Analyzing four different intrinsic reward specifications for CIC, we
find that entropy-based intrinsic reward performs best, suggesting that the
CIC discriminator is primarily useful for representation learning. These are
normalized scores averaged over 3 seeds across 8 downstream tasks (24 runs per
data point).

The importance of representation learning: To what extent does representation
learning with state-skill CPC (see Eq. 3) affect the agent’s exploration
capability? To answer this question we train the CIC agent with the entropy
intrinsic reward with and without the representation learning auxiliary loss
for 2M steps. The zero-shot reward plotted in Fig. 7 indicates that without
representation learning the policy collapses. With representation learning,
the agent is able to discover diverse skills evidenced by the non-zero reward.
This result suggests that state-skill CPC representation learning is a
critical part of CIC.

Qualitative analysis of CIC behaviors: Qualitatively, we find that CIC is able
to learn locomotion behaviors in DMC without extrinsic information such as
early termination as in OpenAI Gym. While most skills are higher entropy and
thus more chaotic, we show in Fig 2 that structured behaviors can be isolated
by fixing a particular skill vector. For example, in the walker and quadruped
domains - balancing, walking, and flipping skills can be isolated. For more
qualitative investigations we refer the reader to Appendix I.

Figure 7: Mean zero-shot extrinsic rewards for Quadruped stand over 3 seeds
with and without state-skill representation learning. Without representation
learning, the algorithm collapses. Similarly, with CIC representation learning
but no entropy term (in which case we use the discriminator as the intrinsic
reward) the policy also collapses. Note that there is no finetuning happening
here. We’re showing the task-specific extrinsic reward during reward-free pre-
training as a way to sense-check exploration policy.

Skill architecture and adaptation ablations: We find that projecting the skill
to a latent space before inputting it as the key for the contrastive loss is
an important design decision (see Fig. 6a), most likely because this reduces
the diversity of the skill vector making the discriminator task simpler.

We also find empirically that the skill dimension is an important
hyperparameter and that larger skills results in better zero-shot performance
(see Fig. 6b), which empirically supports the hypothesis posed in Section 3
and Appendix H that larger skill spaces are important for internalizing
diverse behaviors. Interestingly, CIC zero-shot performance is poor in lower
skill dimensions (e.g. $\text{dim}(z)<10$), suggesting that when
$\text{dim}(z)$ is small CIC performs no better than prior competence-based
methods such as DIAYN, and that scaling to larger skills enables CIC to pre-
train effectively.

To measure the effect of skill finetuning described in Section 5, we sweep
mean skill values along the interval of the uniform prior $[0,1]$ with a
budget of 4k total environment interactions and read out the performance on
the downstream task. By sweeping, we mean simply iterating over the interval
$[0,1]$ with fixed step size (e.g. $v=0,0.1,\dots,0.9,1)$ and setting
$z_{i}=v$ for all $i$. This is not an optimal skill sampling strategy but
works well due to the extremely limited number of samples for skill selection.
We evaluate this ablation on the Quadruped Stand and Run downstream tasks. The
results shown in Fig. 6 indicate that skill selection can substantially affect
zero-shot downstream task performance.

##  8 Conclusion

We have introduced a new competence-based algorithm – Contrastive Intrinsic
Control (CIC) – which enables more effective exploration than prior
unsupervised skill discovery algorithms by explicitly encouraging diverse
behavior while distilling predictable behaviors into skills with a contrastive
discriminator. We showed that CIC is the first competence-based approach to
achieve leading performance on URLB. We hope that this encourages further
research in developing RL agents capable of generalization.

##  9 Acknowledgements

We would like to thank Ademi Adeniji, Xinyang Geng, Fangchen Liu for helpful
discussions. We would also like to thank Phil Bachman for useful feedback.
This work was partially supported by Berkeley DeepDrive, NSF AI4OPT AI
Institute for Advances in Optimization under NSF 2112533, and the Office of
Naval Research grant N00014-21-1-2769.

## References

  * Achiam et al. (2018) Achiam, J., Edwards, H., Amodei, D., and Abbeel, P.  Variational option discovery algorithms.  _arXiv preprint arXiv:1807.10299_ , 2018.
  * Agarwal et al. (2021) Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., and Bellemare, M. G.  Deep reinforcement learning at the edge of the statistical precipice, 2021\.
  * Barber & Agakov (2003) Barber, D. and Agakov, F. V.  The im algorithm: A variational approach to information maximization.  In _Advances in neural information processing systems_ , 2003.
  * Barreto et al. (2016) Barreto, A., Dabney, W., Munos, R., Hunt, J. J., Schaul, T., Van Hasselt, H., and Silver, D.  Successor features for transfer in reinforcement learning.  _arXiv preprint arXiv:1606.05312_ , 2016.
  * Beirlant (1997) Beirlant, J.  Nonparametric entropy estimation: An overview.  _International Journal of the Mathematical Statistics Sciences_ , 6:17–39, 1997.
  * Brockman et al. (2016) Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W.  Openai gym.  _arXiv preprint arXiv:1606.01540_ , 2016.
  * Burda et al. (2019) Burda, Y., Edwards, H., Storkey, A., and Klimov, O.  Exploration by random network distillation.  In _International Conference on Learning Representations_ , 2019.
  * Caron et al. (2020) Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A.  Unsupervised learning of visual features by contrasting cluster assignments.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Chen et al. (2020) Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. E.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , 2020.
  * Eysenbach et al. (2019) Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S.  Diversity is all you need: Learning skills without a reward function.  In _International Conference on Learning Representations_ , 2019.
  * Florensa et al. (2018) Florensa, C., Duan, Y., and Abbeel, P.  Stochastic neural networks for hierarchical reinforcement learning.  In _International Conference on Learning Representations_ , 2018.
  * Gregor et al. (2017) Gregor, K., Rezende, D. J., and Wierstra, D.  Variational intrinsic control.  In _International Conference on Learning Representations_ , 2017.
  * Gutmann & Hyvärinen (2010) Gutmann, M. and Hyvärinen, A.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In Teh, Y. W. and Titterington, M. (eds.), _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_ , volume 9 of _Proceedings of Machine Learning Research_ , pp. 297–304, Chia Laguna Resort, Sardinia, Italy, 13–15 May 2010. PMLR.  URL https://proceedings.mlr.press/v9/gutmann10a.html.
  * Haarnoja et al. (2018) Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _International Conference on Machine Learning_ , 2018.
  * Hansen et al. (2020) Hansen, S., Dabney, W., Barreto, A., Warde-Farley, D., de Wiele, T. V., and Mnih, V.  Fast task inference with variational intrinsic successor features.  In _International Conference on Learning Representations_ , 2020.
  * Kwon (2021) Kwon, T.  Variational intrinsic control revisited.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=P0p33rgyoE.
  * Laskin et al. (2020) Laskin, M., Srinivas, A., and Abbeel, P.  Curl: Contrastive unsupervised representations for reinforcement learning.  In _International Conference on Machine Learning_ , 2020.
  * Laskin et al. (2021) Laskin, M., Yarats, D., Liu, H., Lee, K., Zhan, A., Lu, K., Cang, C., Pinto, L., and Abbeel, P.  Urlb: Unsupervised reinforcement learning benchmark, 2021.  URL https://openreview.net/forum?id=lwrPkQP_is.
  * Lee et al. (2019) Lee, L., Eysenbach, B., Parisotto, E., Xing, E. P., Levine, S., and Salakhutdinov, R.  Efficient exploration via state marginal matching.  _CoRR_ , abs/1906.05274, 2019.
  * Lillicrap et al. (2016) Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D.  Continuous control with deep reinforcement learning.  In _International Conference on Learning Representations_ , 2016.
  * Liu & Abbeel (2021a) Liu, H. and Abbeel, P.  Behavior from the void: Unsupervised active pre-training.  _arXiv preprint arXiv:2103.04551_ , 2021a.
  * Liu & Abbeel (2021b) Liu, H. and Abbeel, P.  APS: active pretraining with successor features.  In _International Conference on Machine Learning_ , 2021b.
  * Mnih et al. (2015) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al.  Human-level control through deep reinforcement learning.  _Nature_ , 518(7540):529, 2015.
  * Oord et al. (2018) Oord, A. v. d., Li, Y., and Vinyals, O.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * OpenAI (2018) OpenAI.  Learning dexterous in-hand manipulation.  _CoRR_ , abs/1808.00177, 2018.
  * OpenAI (2019) OpenAI.  Solving rubik’s cube with a robot hand.  _ArXiv_ , abs/1910.07113, 2019.
  * Oudeyer et al. (2007) Oudeyer, P.-Y., Kaplan, F., and Hafner, V. V.  Intrinsic motivation systems for autonomous mental development.  _IEEE transactions on evolutionary computation_ , 11(2):265–286, 2007.
  * Pathak et al. (2017) Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T.  Curiosity-driven exploration by self-supervised prediction.  In _International Conference on Machine Learning_ , 2017.
  * Pathak et al. (2019) Pathak, D., Gandhi, D., and Gupta, A.  Self-supervised exploration via disagreement.  In _International Conference on Machine Learning_ , 2019.
  * Peng et al. (2018) Peng, X. B., Abbeel, P., Levine, S., and van de Panne, M.  Deepmimic: Example-guided deep reinforcement learning of physics-based character skills.  _ACM Trans. Graph._ , 37:143:1–143:14, 2018.
  * Poole et al. (2019) Poole, B., Ozair, S., van den Oord, A., Alemi, A., and Tucker, G.  On variational bounds of mutual information.  In Chaudhuri, K. and Salakhutdinov, R. (eds.), _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , pp. 5171–5180. PMLR, 2019.  URL http://proceedings.mlr.press/v97/poole19a.html.
  * Rajeswaran et al. (2018) Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., and Levine, S.  Learning complex dexterous manipulation with deep reinforcement learning and demonstrations.  In _Proceedings of Robotics: Science and Systems (RSS)_ , 2018.
  * Schulman et al. (2016) Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P.  High-dimensional continuous control using generalized advantage estimation.  In _International Conference on Learning Representations_ , 2016.
  * Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Sharma et al. (2020) Sharma, A., Gu, S., Levine, S., Kumar, V., and Hausman, K.  Dynamics-aware unsupervised discovery of skills.  In _International Conference on Learning Representations_ , 2020.
  * Silver et al. (2017) Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al.  Mastering the game of go without human knowledge.  _Nature_ , 550(7676):354, 2017.
  * Silver et al. (2018) Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al.  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  _Science_ , 362(6419):1140–1144, 2018.
  * Singh et al. (2003) Singh, H., Misra, N., Hnizdo, V., Fedorowicz, A., and Demchuk, E.  Nearest neighbor estimates of entropy.  _American Journal of Mathematical and Management Sciences_ , 23(3-4):301–321, 2003.
  * Srinivas & Abbeel (2021) Srinivas, A. and Abbeel, P.  Unsupervised learning for reinforcement learning, 2021.  URL https://icml.cc/media/icml-2021/Slides/10843_QHaHBNU.pdf.
  * Strouse et al. (2021) Strouse, D., Baumli, K., Warde-Farley, D., Mnih, V., and Hansen, S.  Learning more skills through optimistic exploration.  _CoRR_ , abs/2107.14226, 2021.  URL https://arxiv.org/abs/2107.14226.
  * Sutton & Barto (2018) Sutton, R. S. and Barto, A. G.  _Reinforcement learning: An introduction_.  MIT Press, 2018.
  * Tassa et al. (2018) Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al.  Deepmind control suite.  _arXiv preprint arXiv:1801.00690_ , 2018.
  * Tschannen et al. (2020) Tschannen, M., Djolonga, J., Rubenstein, P. K., Gelly, S., and Lucic, M.  On mutual information maximization for representation learning.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020.  URL https://openreview.net/forum?id=rkxoh24FPH.
  * Warde-Farley et al. (2018) Warde-Farley, D., de Wiele, T. V., Kulkarni, T., Ionescu, C., Hansen, S., and Mnih, V.  Unsupervised control through non-parametric discriminative rewards, 2018\.
  * Yarats et al. (2021a) Yarats, D., Fergus, R., Lazaric, A., and Pinto, L.  Mastering visual continuous control: Improved data-augmented reinforcement learning, 2021a.
  * Yarats et al. (2021b) Yarats, D., Fergus, R., Lazaric, A., and Pinto, L.  Reinforcement learning with prototypical representations.  In _International Conference on Machine Learning_ , 2021b.
  * Zahavy et al. (2021) Zahavy, T., Barreto, A., Mankowitz, D. J., Hou, S., O’Donoghue, B., Kemaev, I., and Singh, S. B.  Discovering a set of policies for the worst case reward, 2021.

##  Appendix A Competence-based Exploration Algorithms

The competence-based algorithms considered in this work aim to maximize
$I(\tau;s)$. The algorithms differ by ho they decompose mutual information,
whether they explicitly maximize behavioral entropy, their skill space
(discrete or continuous) and their intrinsic reward structure. We provide a
list of common competence-based algorithms in Table 3.

Table 2: Prior Competence-based Unsupervised Skill Discovery Algorithms

Algorithm | Intrinsic Reward | Decomposition | Explicit $\max\mathcal{H}(\tau)$ | Skill Dim. | Skill Space
---|---|---|---|---|---
SSN4HRL (Florensa et al., 2018) | $\log q_{\psi}(z|s_{t})$ | $H(z)-H(z|\tau)$ | No | 6 | discrete
VIC (Gregor et al., 2017) | $\log q_{\psi}(z|s_{H}))$ | $H(z)-H(z|\tau)$ | No | 60 | discrete
VALOR (Achiam et al., 2018) | $\log q_{\psi}(z|s_{1:H})$ | $H(z)-H(z|\tau)$ | No | 64 | discrete
DIAYN (Eysenbach et al., 2019) | $\log q_{\psi}(z|s_{t})$ | $H(z)-H(z|\tau)$ | No | 128 | discrete
DADS (Sharma et al., 2020) | $q_{\psi}(s^{\prime}|z,s)-\sum_{i}\log q(s^{\prime}|z_{i},s)$ | $H(\tau)-H(\tau|z)$ | Yes | 5 | continuous
VISR (Hansen et al., 2020) | $\log q_{\psi}(z|s_{t})$ | $H(z)-H(z|\tau)$ | No | 10 | continuous
APS (Liu & Abbeel, 2021b) | $F_{\text{Successor}}(s|z)+\mathcal{H}_{\text{particle}}(s)$ | $\mathcal{H}(\tau)-\mathcal{H}(\tau|z)$ | Yes | 10 | continuous

Table 3: A list of competence-based algorithms. We describe the intrinsic
reward optimized by each method and the decomposition of the mutual
information utilized by the method. We also note whether the method explicitly
maximizes state transition entropy. Finally, we note the maximal dimension
used in each work and whether the skills are discrete or continuous. All
methods prior to CIC only support small skill spaces, either because they are
discrete or continuous but low-dimensional.

##  Appendix B Deep Deterministic Policy Gradient (DDPG)

A DDPG is an actor-critic RL algorithm that performs off-policy gradient
updates and learns a Q function $Q_{\phi}(s,a)$ and an actor
$\pi_{\theta}(a|s)$. The critic is trained by satisfying the Bellman equation.

$\mathcal{L}_{Q}(\phi,\mathcal{D})=\mathbb{E}_{(s_{t},a_{t},r_{t},s_{t+1})\sim\mathcal{D}}\left[\Big{(}Q_{\phi}(s_{t},a_{t})-r_{t}-\gamma
Q_{\bar{\phi}}(s_{t+1},\pi_{\theta}(s_{t+1})\Big{)}^{2}\right].$ (7)

Here, $\bar{\phi}$ is the Polyak average of the parameters $\phi$. As the
critic minimizes the Bellman error, the actor maximizes the action-value
function.

$\mathcal{L}_{\pi}(\theta,\mathcal{D})=\mathbb{E}_{s_{t}\sim\mathcal{D}}\left[Q_{\phi}(s_{t},\pi_{\theta}(s_{t}))\right].$
(8)

##  Appendix C Baselines

For baselines, we choose the existing set of benchmarked unsupervised RL
algorithms on URLB. We provide a quick summary of each method. For more
detailed descriptions of each baseline we refer the reader to URLB (Laskin et
al., 2021)

Competence-based Baselines: CIC is a competence-based exploration algorithm.
For baselines, we compare it to DIAYN (Eysenbach et al., 2019), SMM (Lee et
al., 2019), and APS (Liu & Abbeel, 2021b). Each of these algorithms is
described in Table 3. Notably, APS is a recent state-of-the-art competence-
based method that is the most closely related algorithm to the CIC algorithm.

Knowledge-based Baselines: For knowledge-based baselines, we compare to ICM
(Pathak et al., 2017), Disagreement (Pathak et al., 2019), and RND (Burda et
al., 2019). ICM and RND train a dynamics model and random network prediction
model and define the intrinsic reward to be proportional to the prediction
error. Disagreement trains an ensemble of dynamics models and defines the
intrinsic reward to be proportional to the uncertainty of an ensemble.

Data-based Baselines: For data-based baselines we compare to APT (Liu &
Abbeel, 2021a) and ProtoRL (Yarats et al., 2021b). Both methods use a particle
estimator to estimate the state visitation entropy. ProtoRL also performs
discrete contrastive clustering as in Caron et al. (2020) as an auxiliary task
and uses the resulting clusters to compute the particle entropy. While ProtoRL
is more effective than APT when learning from pixels, on state-based URLB APT
is competitive with ProtoRL. Our method CIC is effectively a skill-conditioned
APT agent with a contrastive discriminator.

##  Appendix D Relation to Prior Skill Discovery Methods

The most closely relatd prior algorithms to CIC are APT (Liu & Abbeel, 2021a)
and APS (Liu & Abbeel, 2021b). Both CIC and APS use the
$\mathcal{H}(\tau)-\mathcal{H}(\tau|z)$ decomposition of the mutual
information and both used a particle estimator (Singh et al., 2003) to compute
the state entropy as in Liu & Abbeel (2021a). The main difference between CIC
and APS is the discriminator. APS uses successor features as in Hansen et al.
(2020) for its discriminator while CIC uses a noise contrastive estimator.
Unlike successor features, which empirically only accommodate low-dimensional
continuous skill spaces (see Table 3), the noise contrastive discriminator is
able to leverage higher continuous dimensional skill vectors. Like APT, CIC
has an intrinsic reward that maximizes $\mathcal{H}(\tau)$. However, CIC also
does contrastive skill learning to shape the embedding space and outputs a
skill-conditioned policy.

The CIC discriminator is similar to the one used in DISCERN (Warde-Farley et
al., 2018), a goal-conditioned unsupervised RL algorithm. Both methods use a
contrastive discriminator by sampling negatives and computing an inner product
between queries and keys. The main differences are (i) that DISCERN maximizes
$I(\tau;g)$ where $g$ are image goal embeddings while CIC maximizes
$I(\tau;z)$ where $z$ are abstract skill vectors; (ii) DISCERN uses the DIAYN-
style decomposition $I(\tau;g)=H(g)-H(g|\tau)$ while CIC decomposes through
$H(\tau)-H(\tau|z)$, and (iii) DISCERN discards the $H(g)$ term by sampling
goals uniformly while CIC explicitly maximizes $\mathcal{H}(\tau)$. While
DISCERN and CIC share similarities, DISCERN operates over image goals while
CIC operates over abstract skill vectors so the two methods are not directly
comparable.

Finally, another similar algorithm to CIC is DADS (Sharma et al., 2020) which
also decomposes through $H(\tau)-H(\tau|z)$. While CIC uses a contrastive
density estimate for the discriminator, DADS uses a maximum likelihood
estimator similar to DIAYN. DADS maximizes $I(s^{\prime}|s,z)$ and estimates
entropy $\mathcal{H}(s^{\prime}|s)$ by marginalizing over $z$ such that
$\mathcal{H}(s^{\prime}|s)=-\log\sum_{i}q(s^{\prime}|s,z_{i})$ while CIC uses
a particle estimator.

##  Appendix E Full CIC Algorithm

The full CIC algorithm with both pre-training and fine-tuning phases is shown
in Algorithm 1. We pre-train CIC for 2M steps, and finetune it on each task
for 100k steps.

Algorithm 1 Contrastive Intrinsic Control

1:Initialize all networks: encoders $g_{\psi_{1}}$ and $g_{\psi_{2}}$, actor
$\pi_{\theta}$, critic $Q_{\phi}$, replay buffer $\mathcal{D}$.

2:Environment (env), $M$ downstream tasks $T_{k},k\in[1,\dots,M]$.

3:pre-train $N_{\mathrm{PT}}=2M$ and fine-tune $N_{\mathrm{FT}}=100K$ steps.

4:for $t=1..N_{\mathrm{PT}}$ do $\triangleright$ Part 1: Unsupervised Pre-
training

5: Sample and encode skill $z\sim p(z)$ and $z\leftarrow g_{\psi_{2}}(z)$

6: Encode state $s_{t}\leftarrow g_{\psi_{1}}(s_{t})$ and sample action
$a_{t}\leftarrow\pi_{\theta}(s_{t},z)+\epsilon$ where
$\epsilon\sim\mathcal{N}(0,\sigma^{2})$

7: Observe next state $s_{t+1}\sim P(\cdot|s_{t},a_{t})$

8: Add transition to replay buffer
$\mathcal{D}\leftarrow\mathcal{D}\cup(s_{t},a_{t},s_{t+1})$

9: Sample a minibatch from $\mathcal{D}$, compute contrastive loss in Eq.3 and
update encoders $g_{\psi_{1}},g_{\psi_{2}}$, compute CIC intrinsic reward with
Eq. LABEL:eq:rewint and update actor $\pi_{\theta}$ and critic $Q_{\phi}$

10:end for

11:for $T_{k}\in[T_{1},\dots,T_{M}$] do $\triangleright$ Part 2: Supervised
Fine-tuning

12: Initialize all networks with weights from pre-training phase and an empty
replay buffer $\mathcal{D}$.

13: for $t=1\dots 4,000$ do

14: Take random action $a_{t}\sim\mathcal{N}(0,1)$

15: Select skill with grid sweep over unit interval $[0,1]$ every 100 steps

16: Sample minibatch from $\mathcal{D}$ and update actor $\pi_{\theta}$ and
critic $Q_{\phi}$

17: end for

18: Fix skill $z$ that achieved highest extrinsic reward during grid sweep.

19: for $t=4,000\dots N_{\mathrm{FT}}$ do

20: Encode state $s_{t}\leftarrow g_{\psi_{1}}(s_{t})$ and sample action
$a_{t}\leftarrow\pi_{\theta}(s_{t},z)+\epsilon$ where
$\epsilon\sim\mathcal{N}(0,\sigma^{2})$

21: Observe next state and reward $s_{t+1},r^{\mathrm{ext}}_{t}\sim
P(\cdot|s_{t},a_{t})$

22: Add transition to replay buffer
$\mathcal{D}\leftarrow\mathcal{D}\cup(s_{t},a_{t},r^{\mathrm{ext}}_{t},s_{t+1})$

23: Sample minibatch from $\mathcal{D}$ and update actor $\pi_{\theta}$ and
critic $Q_{\phi}$.

24: end for

25: Evaluate performance of RL agent on task $T_{k}$

26:end for

##  Appendix F Hyper-parameters

Baseline hyperparameters are taken from URLB (Laskin et al., 2021), which were
selected by performing a grid sweep over tasks and picking the best performing
set of hyperparameters. Except for the skill dimension, hyperparameters for
CIC are borrowed from URLB.

Table 4:  Hyper-parameters used for CIC .  DDPG hyper-parameter | Value
---|---
Replay buffer capacity | $10^{6}$
Action repeat | $1$
Seed frames | $4000$
$n$-step returns | $3$
Mini-batch size | $1024$
Seed frames | $4000$
Discount ($\gamma$) | $0.99$
Optimizer | Adam
Learning rate | $10^{-4}$
Agent update frequency | $2$
Critic target EMA rate ($\tau_{Q}$) | $0.01$
Features dim. | $1024$
Hidden dim. | $1024$
Exploration stddev clip | $0.3$
Exploration stddev value | $0.2$
Number pre-training frames | $2\times 10^{6}$
Number fine-turning frames | $1\times 10^{5}$
CIC hyper-parameter | Value
Skill dim | 64 continuous
Prior | Uniform [0,1]
Skill sampling frequency (steps) | 50
State net arch. $g_{\psi_{1}}(s)$ |  $\dim(\mathcal{O})\to 1024\to 1024\to 64$ ReLU MLP
Skill net arch. $g_{\psi_{2}}(z)$ |  $64\to 1024\to 1024\to 64$ ReLU MLP
Prediction net arch. |  $64\to 1024\to 1024\to 64$ ReLU MLP

##  Appendix G Raw Numerical Results

Statistic | ICM | Dis. | RND | APT | Proto | DIAYN | APS | SMM | CIC |  $\%$ CIC $>$ APS |  $\%$ CIC $>$ Proto
---|---|---|---|---|---|---|---|---|---|---|---
Median $\uparrow$ | 0.45 | 0.56 | 0.58 | 0.62 | 0.66 | 0.44 | 0.47 | 0.22 | 0.76 | +61$\%$ | +15$\%$
IQM $\uparrow$ | 0.41 | 0.51 | 0.61 | 0.65 | 0.65 | 0.40 | 0.43 | 0.25 | 0.77 | +79$\%$ | +18$\%$
Mean $\uparrow$ | 0.43 | 0.51 | 0.63 | 0.66 | 0.65 | 0.44 | 0.46 | 0.35 | 0.76 | +65$\%$ | +17$\%$
OG $\downarrow$ | 0.57 | 0.49 | 0.37 | 0.35 | 0.35 | 0.56 | 0.54 | 0.65 | 0.24 | -44$\%$ | -68$\%$
Table 5: Statics for downstream task normalized scores for CIC and baselines
from URLB (Laskin et al., 2021). CIC improves over both the prior leading
competence-based method APS (Liu & Abbeel, 2021b) and overall next-best
exploration algorithm ProtoRL (Yarats et al., 2021b) across all readout
statistics. Each data point is a statistic computed using 10 seeds and 12
downstream tasks (120 experiments per data point). The statistics are computed
using RLiable (Agarwal et al., 2021).

We provide a list of raw numerical results for finetuning CIC and baselines in
Tables 5 and 6. All baselines were run using the code provided by URLB (Laskin
et al., 2021) for 10 seeds per downstream task.

Pre-trainining for $2\times 10^{6}$ environment steps |  |
---|---|---
Domain | Task | Expert | DDPG | CIC | ICM | Disagreement | RND | APT | ProtoRL | SMM | DIAYN | APS
Walker | Flip | 799 | 538$\pm$27 | 631 $\pm$ 34 | 417$\pm$16 | 346$\pm$13 | 474$\pm$39 | 544$\pm$14 | 456$\pm$12 | 450$\pm$24 | 319$\pm$17 | 465$\pm$20
Run | 796 | 325$\pm$25 | 486 $\pm$ 25 | 247$\pm$21 | 208$\pm$15 | 406$\pm$30 | 392$\pm$26 | 306$\pm$13 | 426$\pm$26 | 158$\pm$8 | 134$\pm$16
Stand | 984 | 899$\pm$23 | 959 $\pm$ 2 | 859$\pm$23 | 746$\pm$34 | 911$\pm$5 | 942$\pm$6 | 917$\pm$27 | 924$\pm$12 | 695$\pm$46 | 721$\pm$44
Walk | 971 | 748$\pm$47 | 885 $\pm$ 28 | 627$\pm$42 | 549$\pm$37 | 704$\pm$30 | 773$\pm$70 | 792$\pm$41 | 770$\pm$44 | 498$\pm$27 | 527$\pm$79
Quadruped | Jump | 888 | 236$\pm$48 | 595 $\pm$ 42 | 178$\pm$35 | 389$\pm$62 | 637$\pm$12 | 648$\pm$18 | 617$\pm$44 | 96$\pm$7 | 660$\pm$43 | 463$\pm$51
Run | 888 | 157$\pm$31 | 505 $\pm$ 47 | 110$\pm$18 | 337$\pm$30 | 459$\pm$6 | 492$\pm$14 | 373$\pm$33 | 96$\pm$6 | 433$\pm$29 | 281$\pm$17
Stand | 920 | 392$\pm$73 | 761 $\pm$ 54 | 312$\pm$68 | 512$\pm$89 | 766$\pm$43 | 872$\pm$23 | 716$\pm$56 | 123$\pm$11 | 851$\pm$43 | 542$\pm$53
Walk | 866 | 229$\pm$57 | 723 $\pm$ 43 | 126$\pm$27 | 293$\pm$37 | 536$\pm$39 | 770$\pm$47 | 412$\pm$54 | 80$\pm$6 | 576$\pm$81 | 436$\pm$79
Jaco | Reach bottom left | 193 | 72$\pm$22 | 138 $\pm$ 9 | 111$\pm$11 | 124$\pm$7 | 110$\pm$5 | 103$\pm$8 | 129$\pm$8 | 45$\pm$7 | 39$\pm$6 | 76$\pm$8
Reach bottom right | 203 | 117$\pm$18 | 145 $\pm$ 7 | 97$\pm$9 | 115$\pm$10 | 117$\pm$7 | 100$\pm$6 | 132$\pm$8 | 46$\pm$11 | 38$\pm$5 | 88$\pm$11
Reach top left | 191 | 116$\pm$22 | 153 $\pm$ 7 | 82$\pm$14 | 106$\pm$12 | 99$\pm$6 | 73$\pm$12 | 123$\pm$9 | 36$\pm$3 | 19$\pm$4 | 68$\pm$6
Reach top right | 223 | 94$\pm$18 | 163 $\pm$ 4 | 103$\pm$11 | 139$\pm$7 | 100$\pm$6 | 90$\pm$10 | 159$\pm$7 | 47$\pm$6 | 28$\pm$6 | 76$\pm$10

Table 6: Performance of CIC and baselines on state-based URLB after first pre-
training for $2\times 10^{6}$ steps and then finetuning with extrinsic rewards
for $1\times 10^{5}$. All baselines were run for 10 seeds per downstream task
for each algorithm using the code provided by URLB (Laskin et al., 2021). A
total of $1080=9\text{ algorithms}\times 12\text{ tasks}\times 10\text{
seeds}$ experiments were run.

##  Appendix H Toy Example to Illustrate the Need for Larger Skill Spaces

Figure 8: A gridworld example motivating the need for large skill spaces. In
this environment, we place an agent in a $10\times 10$ gridworld and provide
the agent access to four discrete skills. We show that the mutual information
objective can be maximized by mapping these four skills to the nearest
neighboring states resulting in low behavioral diversity and exploring only
four of the hundred available states.

We illustrate the need for larger skill spaces with a gridworld example.
Suppose we have an agent in a $10\times 10$ sized gridworld and that we have
four discrete skills at our disposal. Now let $\tau=s$ and consider how we may
achieve maximal $I(\tau;z)$ in this setting. If we decompose
$I(\tau;z)=\mathcal{H}(z)-\mathcal{H}(z|\tau)$ then we can achieve maximal
$\mathcal{H}(z)$ by sampling the four skills uniformly $z\sim p(z)$. We can
achieve $\mathcal{H}(z|\tau)=0$ by mapping each skill to a distinct
neighboring state of the agent. Thus, our mutual information is maximized but
as a result the agent only explores four out of the hundrend available states
in the gridworld.

Now suppose we consider the second decomposition
$I(\tau;z)=\mathcal{H}(\tau)-\mathcal{H}(\tau|z)$. Since the agent is
maximizing $\mathcal{H}(\tau)$ it is likely to visit a diverse set of states
at first. However, as soon as it learns an accurate discriminator we will have
$\mathcal{H}(\tau|z)$ and again the skills can be mapped to neighboring states
to achieve minimal conditional entropy. As a result, the skill conditioned
policy will only be able to reach four out of the hundrend possible states in
this gridworld. This argument is shown visually in Fig. 8.

Skill spaces that are too large can also be an issue. Consider if we had $100$
skills at our disposal in the same gridworld. Then the agent could minimize
the conditional entropy by mapping each skill to a unique state which would
result in the agent memorizing the environment by finding a one-to-one mapping
between states and skills. While this is a potential issue it has not been
encountered in practice yet since current competence-based methods support
small skill spaces relative to the observation space of the environment.

##  Appendix I Qualitative Analysis of Skills

We provide two additional qualitative analyses of behaviors learned with the
CIC algorithm. First, we take a simple pointmass setting and set the skill
dimension to 1 in order to ablate the skills learned by the CIC agent in a
simple setting. We sweep over different values of $z$ and plot the behavioral
flow vector field (direction in which point mass moves) in Fig.9. We find that
the pointmass learns skills that produce continuous motion and that the
direction of the motion changes as a function of the skill value. Near the
origin the pointmass learns skills that span all directions, while near the
edges the point mass learns to avoid wall collisions. Qualitatively, many
behaviors are periodic.

Figure 9: Learning curves for finetuning pre-trained agents for 100k steps.
Task performance is aggregated for each domain, such that each curve
represents the mean normalized scores over $4\times 10=40$ seeds. The shaded
regions represent the standard error. CIC surpasses the performance of the
prior state-of-the-art on Walker and Jaco tasks while tying on Quadruped. CIC
is the only algorithm that performs consistently well across all three
domains.

Qualitatively, we find that methods like DIAYN that only support low
dimensional skill vectors and do not explicitly incentivize diverse behaviors
in their objective produce policies that map skills to a small set of static
behaviors. These behaviors shown in Fig. 10 are non-trivial but also have low
behavioral diversity and are not particularly useful for solving the
downstream task. This observation is consistent with Zahavy et al. (2021)
where the authors found that DIAYN maps to static “yoga” poses in DeepMind
Control. In contrast, behaviors produce by CIC are dynamic resulting flipping,
jumping, and locomotive behaviors that can then be adapted to efficiently
solve downstream tasks.

Figure 10: Qualitative visualization of DIAYN and CIC pre-training on the
Walker and Quadruped domains from URLB. Confirming findings in prior work
(Zahavy et al., 2021), we also find that DIAYN policies produce static but
non-trivial behaviors mapping to “yoga” poses while CIC produces diverse and
dynamic behaviors such as walking, flipping, and standing. Though it’s hard to
see from these images, all the DIAYN skills get stuck in frozen poses while
the CIC skills are producing dynamic behavior with constant motion.

##  Appendix J OpenAI Gym vs. DeepMind control: How Early Termination Leaks
Extrinsic Signal

Prior work on unsupervised skill discovery for continuous control (Eysenbach
et al., 2019; Sharma et al., 2020) was evaluated on OpenAI Gym (Brockman et
al., 2016) and showed diverse exploration on Gym environments. However, Gym
environment episodes terminate early when the agent loses balance, thereby
leaking information about the extrinsic task (e.g. balancing or moving).
However, DeepMind Control (DMC) episodes have a fixed length of 1k steps. In
DMC, exploration is therefore harder since the agent needs to learn to balance
without any extrinsic signal.

To evaluate whether the difference in the two environments has impact on
competence-based exploration, we run DIAYN on the hopper environments from
both Gym and DMC. We compare to ICM, a popular exploration baseline, and a
Fixed baseline where the agent receives an intrinsic reward of 1 for each
timestep and no algorithms receive extrinsic rewards. We then measure the
extrinsic reward, which loosely corresponds to the diversity of behaviors
learned. Our results in Fig. 4 show that indeed DIAYN is able to learn diverse
behaviors in Gym but not in DMC while ICM is able to learn diverse behaviors
in both environments. Interestingly, the Fixed baseline achieves the highest
reward on the Gym environment by learning to stand and balance. These results
further motivate us to evaluate on URLB which is built on top of DMC.

##  Appendix K CIC vs Other Types of Contrastive Learning for RL

Contrastive learning in CIC is different than prior vision-based contrastive
learning in RL such as CURL (Laskin et al., 2020), since we are not performing
contrastive learning over augmented images but rather over state transitions
and skills. The contrastive objective in CIC is used for unsupervised learning
of behaviors while in CURL it is used for unsupervised learning of visual
features.

We provide pseudocode for the CIC loss below:

⬇

1def discriminator_loss(states, next_states, skills, temp):

2 """

3 - states and skills are sampled from replay buffer

4 - skills were sampled from uniform dist [0,1] during agent rollout

5 - states / next_states: dim (B, D_state)

6 - skills: dim (B, D_skill)

7 """

8

9 transitions = concat(states, next_states, dim=1)

10

11 query = skill_net(skills) # (B, D_hidden) -> (B, D_hidden)

12 key = transition_net(transitions) # (B, 2*D_state) -> (B, D_hidden)

13

14 query = normalize(query, dim=1)

15 key = normalize(key, dim=1)

16

17 logits = matmul(query, key.T) / temp # (B, B)

18 labels = arange(logits.shape[0])

19

20 # positives are on diagonal, negatives are off diagonal

21 # for each skill, negatives are sampled from transitions

22 # while skills are fixed

23 loss = cross_entropy(logits, labels)

24

25 return loss

Listing 2: CIC discriminator loss

This is substantially different from prior contrastive learning works in RL
such as CURL (Laskin et al., 2020), which perform contrastive learning over
images.

⬇

1def curl_loss(obs, W, temp):

2 """

3 - observation images are sampled from replay buffer

4 - obs: dim (B, C, H, W)

5 - W: projection matrix (D_hidden, D_hidden)

6 """

7

8 query = aug(obs)

9 key = aug(obs)

10

11 query = cnn_net(query) # (B, D_hidden)

12 key = cnn_net(key) # (B, D_hidden)

13

14 logits = matmul(matmul(query, W), key.T) / temp # (B, B)

15 labels = arange(logits.shape[0])

16

17 # positives are on diagonal

18 # negatives are off diagonal

19 loss = cross_entropy(logits, labels)

20

21 return loss

Listing 3: CURL contrastive loss

##  Appendix L On estimates of Mutual Information

In this work we have presented CIC - a new competence-based algorithm that
achieves leading performance on URLB compared to prior unsupervised RL
methods.

One might wonder whether estimating the exact mutual information (MI) or
maximizing the tightest lower bound thereof is really the goal for
unsupervised RL. In unsupervised representation learning, state-of-the-art
methods like CPC and SimCLR maximize the lower bound of MI based on Noise
Contrastive Estimation (NCE). However, as proven in CPC (Oord et al., 2018)
and illustrated in Poole et al. (2019) NCE is upper bounded by $\log N$,
meaning that the bound is loose when the MI is larger than $\log N$.
Nevertheless, these methods have been repeatedly shown to excel in practice.
In Tschannen et al. (2020) the authors show that the effectiveness of NCE
results from the inductive bias in both the choice of feature extractor
architectures and the parameterization of the employed MI estimators.

We have a similar belief for unsupervised RL - that with the right
parameterization and inductive bias, the MI objective will facilitate behavior
learning in unsupervised RL. This is why CIC lower bounds MI with (i) the
particle based entropy estimator to ensure explicit exploration and (ii) a
contrastive conditional entropy estimator to leverage the power of contrastive
learning to discriminate skills. As demonstrated in our experiments, CIC
outperforms prior methods, showing the effectiveness of optimizing an
intrinsic reward with the CIC MI estimator.

##  Appendix M Limitations

While CIC achieves leading results on URLB, we would also like to address its
limitations. First, in this paper we only consider MDPs (and not partially
observed MDPs) where the full state is observable. We focus on MDPs because
generating diverse behaviors in environments with large state spaces has been
the primary bottleneck for competence-based exploration. Combining CIC with
visual representation learning to scale this method to pixel-based inputs is a
promising future direction for research not considered in this work. Another
limitation is that our adaptation strategy to downstream tasks requires
finetuning. Since we learn skills, it would be interesting to investigate
alternate ways of adapting that would enable zero-shot generalization such as
learning generalized reward functions during pre-training.