# TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware
Representations

Junik Bae Kwanyoung Park Youngwoon Lee
Yonsei University
https://heatz123.github.io/tldr

###### Abstract

Unsupervised goal-conditioned reinforcement learning (GCRL) is a promising
paradigm for developing diverse robotic skills without external supervision.
However, existing unsupervised GCRL methods often struggle to cover a wide
range of states in complex environments due to their limited exploration and
sparse or noisy rewards for GCRL. To overcome these challenges, we propose a
novel unsupervised GCRL method that leverages TemporaL Distance-aware
Representations (TLDR). TLDR selects faraway goals to initiate exploration and
computes intrinsic exploration rewards and goal-reaching rewards, based on
temporal distance. Specifically, our exploration policy seeks states with
large temporal distances (i.e. covering a large state space), while the goal-
conditioned policy learns to minimize the temporal distance to the goal (i.e.
reaching the goal). Our experimental results in six simulated robotic
locomotion environments demonstrate that our method significantly outperforms
previous unsupervised GCRL methods in achieving a wide variety of states.

> Keywords: Unsupervised Goal-Conditioned Reinforcement Learning, Temporal
> Distance-Aware Representations

##  1 Introduction

Human babies can autonomously learn goal-reaching skills, starting from
controlling their own bodies and gradually improving their capabilities to
achieve more challenging goals, involving longer-horizon behaviors. Similarly,
for intelligent agents like robots, the ability to reach a large set of
states–including both the environment states and agent states–is crucial. This
capability not only serves as a foundational skill set by itself but also
enables achieving more complex tasks.

(a) TLDR (ours)

(b) METRA

(c) PEG

Figure 1: Trajectories (red) of an ant robot in a complex maze trained by TLDR
(ours), METRA [1], and PEG [2]. While prior methods yield limited exploration,
TLDR explores the entire maze.

Can robots autonomously learn such long-horizon goal-reaching skills like
humans? This is particularly compelling as learning goal-reaching behaviors in
robots is task-agnostic and does not require any external supervision,
offering a scalable approach for unsupervised pre-training of robots [3, 4, 5,
6, 7, 8, 9]. However, prior unsupervised goal-conditioned reinforcement
learning (GCRL) [10, 2] and unsupervised skill discovery [1] methods exhibit
limited coverage of reachable states in complex environments, as shown in
Figure 1.

The major challenges in unsupervised GCRL are twofold: (1) exploring diverse
states to ensure the agent can learn to achieve a wide variety of goals, and
(2) effectively learning a goal-reaching policy. Previous methods focus on
exploring novel states [11] or states with high uncertainty in next state
prediction [10, 2]. However, these methods aim to discover unseen states or
state transitions, which may not be meaningful. Additionally, training a goal-
reaching policy to maximize sparse goal-reaching rewards [8] or minimize
heuristically-defined distances to a goal [10, 12] is often insufficient for
long-horizon goal-reaching behaviors in complex environments.

In this paper, we propose a novel unsupervised GCRL method that leverages
TemporaL Distance-aware Representations (TLDR) to improve both goal-directed
exploration and goal-conditioned policy learning. TLDR uses temporal distance
(i.e. the minimum number of environment steps between two states) induced by
temporal distance-aware representations [1, 13, 14] for (1) selecting faraway
goals to initiate exploration, (2) learning an exploration policy that
maximizes temporal distance, and (3) learning a goal-conditioned policy that
minimizes temporal distance to a goal.

TLDR demonstrates superior state coverage compared to prior unsupervised GCRL
and skill discovery methods in complex AntMaze environments (see Figure 1).
Our ablation studies confirm that our temporal distance-aware approach
enhances both goal-directed exploration and goal-conditioned policy learning.
Furthermore, our method outperforms prior work across diverse locomotion
environments, underscoring its general applicability.

##  2 Related Work

Unsupervised goal-conditioned reinforcement learning (GCRL) aims to learn a
goal-conditioned policy that can reach diverse goal states without external
supervision [15, 16, 10, 2]. The major challenges of unsupervised GCRL can be
summarized in two aspects: (1) optimizing goal-conditioned policies and (2)
collecting trajectories with novel goals that effectively enlarge its state
coverage.

Recent techniques such as hindsight experience reply (HER) [8] and model-based
policy optimization [10, 12] have improved the efficiency of GCRL. However,
learning complex, long-horizon goal-reaching behaviors remains difficult due
to sparse (e.g. whether it reaches the goal) [8] or heuristic rewards (e.g.
cosine similarity between the state and goal) [10, 12]. Instead, temporal
distance, defined as the number of environment steps between states estimated
from data, can provide more dense and grounded rewards [17, 10, 18].
Nonetheless, this often leads to sub-optimal goal-reaching behaviors since it
does not reflect the “shortest temporal distance” between states. In this
paper, we propose to use the estimated shortest temporal distance as reward
signals for GCRL, inspired by QRL [14] and HILP [13]. We apply the learned
representations to compute goal-reaching rewards rather than directly learning
the value function in QRL or using it for skill-learning rewards in HILP.

Exploration in unsupervised GCRL relies heavily on selecting exploratory goals
that lead to novel states and expand state coverage. Various strategies for
exploratory goal selection have been introduced, including selecting less
visited states [19], states with low-density in state distributions [20, 11],
and states with high uncertainty in dynamics [10, 2]. Instead of sampling
uncertain or less visited states as goals, we select goals that are temporally
distant from the visited state distribution, encouraging coverage of broader
state spaces which require more environment steps to reach.

Unsupervised skill discovery [21, 22, 23, 24, 25, 26, 27, 1] is another
approach to learning diverse behaviors without supervision, yet often lacks
robust exploration capabilities [27], requiring manual feature engineering or
limiting to low-dimensional state spaces. METRA [1] addresses these
limitations by computing skill-learning rewards with temporal distance-aware
representations, though it exhibits limited coverage in complex environments,
as depicted in Figure 1.

Temporal distance-aware representations have been extensively used in
imitation learning [28], representation learning [29, 30], unsupervised skill
discovery [1], offline skill learning [13], and GCRL [14]. Methods like QRL
[14], HILP [13], and METRA [1] are closely related to our work as they learn
temporal distance-preserving representations or goal-reaching value functions.
HILP and METRA use temporal distance-aware representations for skill
representations and skill rewards. On the other hand, QRL learns a quasimetric
model as a (negated) goal-reaching value function. In contrast, our method
uses temporal distance-aware representations across the entire unsupervised
GCRL pipeline.

Figure 2: Overview of TLDR algorithm. TLDR leverages temporal distance-aware
representations for unsupervised GCRL (Go-Explore [19] in this paper). (a) We
start by learning a state encoder $\phi$ that maps states to temporal
distance-aware representations. With the temporal distance-aware
representations, TLDR (b) selects the temporally farthest state from the
visited states as an exploratory goal, (c) reaches the chosen goal using a
goal-conditioned policy, which learns to minimize temporal distance to the
goal, and (d) collects exploratory trajectories using an exploration policy
that visits states with large temporal distance from the visited states.

##  3 Approach

In this paper, we introduce TemporaL Distance-aware Representations (TLDR), an
unsupervised goal-conditioned reinforcement learning (GCRL) method,
integrating temporal distance within unsupervised GCRL. As illustrated in
Figure 2, TLDR integrates temporal distance-aware representations (Section
3.2) into every facet of the Go-Explore [19] strategy (Section 3.3), which
chooses a goal from experience (Section 3.4), reaches the selected goal via
the goal-conditioned policy, and executes the exploration policy to gather
diverse experiences. We then refine both the exploration policy (Section 3.5)
and goal-conditioned policy (Section 3.6) based on the collected data and
rewards computed using the temporal distance-aware representations. We
describe the full algorithm in Algorithm 1. Please refer to Appendix A for
further implementation details.

###  3.1 Problem Formulation

We formulate the unsupervised GCRL problem with a goal-conditioned Markov
decision process, defined as the tuple
$\mathcal{M}=(\mathcal{S},\mathcal{A},p,\mathcal{G})$. $\mathcal{S}$ and
$\mathcal{A}$ denote the state and action spaces, respectively.
$p:\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathcal{S})$ denotes the
transition dynamics, where $\Delta(\mathcal{X})$ denotes the set of
probability distributions over $\mathcal{X}$. The goal of the agent is to
learn an optimal goal-conditioned policy
$\pi^{G}:\mathcal{S}\times\mathcal{G}\rightarrow\mathcal{A}$, where
$\pi^{G}(\mathbf{a}\mid\mathbf{s},\mathbf{g})$ outputs an action
$\mathbf{a}\in\mathcal{A}$ that can navigate to the goal
$\mathbf{g}\in\mathcal{G}$ as fast as possible from the current state
$\mathbf{s}$. In this paper, we set $\mathcal{G}=\mathcal{S}$, allowing any
state as a potential goal for the agent.

###  3.2 Learning Temporal Distance-Aware Representations

Temporal distance, defined as the minimum number of environment steps between
states, can provide more dense and grounded rewards for goal-conditioned
policy learning as well as exploration. For GCRL, instead of relying on sparse
and binary goal-reaching rewards, the change in temporal distance before and
after taking an action can be an informative learning signal. Moreover,
exploration in unsupervised GCRL can be incentivized by discovering temporally
faraway states. Therefore, in this paper, we propose to use temporal distance
for unsupervised GCRL.

We first estimate the temporal distance by learning temporal distance-aware
representations, inspired by Park et al. [13], Wang et al. [14]. We learn the
representation $\phi:\mathcal{S}\rightarrow\mathcal{Z}$, that encodes the
temporal distance between two states into the latent space $\mathcal{Z}$,
where $\lVert\phi(\mathbf{s}_{1})-\phi(\mathbf{s}_{2})\rVert$ represents the
shortest temporal distance between $\mathbf{s}_{1}$ and $\mathbf{s}_{2}$. This
representation is then used across the entire unsupervised GCRL algorithm:
exploratory goal selection, intrinsic reward for exploration, and reward for
goal-conditioned policy.

To train temporal distance-aware representations, we adopt a constrained
optimization approach similar to the objective of QRL [14]:

$\ \max_{\phi}\mathbb{E}_{\mathbf{s}\sim p_{\mathbf{s}},\mathbf{g}\sim
p_{\mathbf{g}}}\left[\lVert
f(\phi(\mathbf{s})-\phi(\mathbf{g}))\rVert\right]\quad\text{ s.t.
}\quad\mathbb{E}_{(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime})\sim
p_{\text{transition}}}\left[\lVert\phi(\mathbf{s})-\phi(\mathbf{s}^{\prime})\rVert\right]\leq
1,$ (1)

where $f$ represents an affine-transformed softplus function that assigns
lower weights to larger distances
$\lVert\phi(\mathbf{s})-\phi(\mathbf{g})\rVert$. We optimize this constrained
objective using dual gradient descent with a Lagrange multiplier $\lambda$,
and we randomly sample $\mathbf{s}$ and $\mathbf{g}$ from a minibatch during
training.

###  3.3 Unsupervised GCRL with Temporal Distance-Aware Representations

With temporal distance-aware representations, we can integrate the concept of
temporal distance into unsupervised GCRL. Our approach is built upon the Go-
Explore procedure [19], a widely-used unsupervised GCRL algorithm comprising
two phases: (1) the “Go-phase,” where the goal-conditioned policy $\pi^{G}$
navigates toward a goal $\mathbf{g}$, and (2) the “Explore-phase,” where the
exploration policy $\pi^{E}$ gathers new state trajectories to refine the
goal-conditioned policy.

While Go-Explore relies on task-specific information for goal selection and
exploration policy training, our method uses task-agnostic temporal distance
metrics induced by temporal distance-aware representations. The subsequent
sections detail how our method leverages the representation for selecting
goals in the Go-phase (Section 3.4), enhancing the exploration policy (Section
3.5), and facilitating the GCRL policy training (Section 3.6).

Algorithm 1 TLDR: unsupervised goal-conditioned reinforcement learning
algorithm

1: Initialize goal-conditioned policy $\pi^{G}_{\theta}$, exploration policy
$\pi^{E}_{\theta}$, and temporal distance-aware representation $\phi$

2: $\mathcal{D}\leftarrow\emptyset$

3: while not converged do

4: $\mathbf{s}_{0}\sim p(\mathbf{s}_{0})$

5: Sample a minibatch $\mathcal{B}\sim\mathcal{D}$

6:
$\mathbf{g}\leftarrow\arg\max_{\mathbf{s}\in\mathcal{B}}(r_{\text{TLDR}}(\mathbf{s}))$
$\triangleright$ Select state with the highest TLDR reward (Eq. 2)

7: for $t=0,\ldots,T-1$ do

8: if $t<T_{G}$ then

9: $\mathbf{a}_{t}\sim\pi^{G}_{\theta}(\cdot\mid\mathbf{s}_{t},\mathbf{g})$
$\triangleright$ Follow goal-conditioned policy $\pi^{G}_{\theta}$ for $T_{G}$
steps

10: else

11: $\mathbf{a}_{t}\sim\pi^{E}_{\theta}(\cdot\mid\mathbf{s}_{t})$
$\triangleright$ Explore using exploration policy $\pi^{E}_{\theta}$

12: $\mathbf{s}_{t+1}\sim p(\cdot\mid\mathbf{s}_{t},\mathbf{a}_{t})$

13:
$\mathcal{D}\leftarrow\mathcal{D}\cup\\{\mathbf{s}_{t},\mathbf{a}_{t},\mathbf{s}_{t+1}\\}$

14: Train exploration policy $\pi^{E}_{\theta}$ to maximize Eq. 3

15: Train goal-conditioned policy $\pi^{G}_{\theta}$ using HER with dense
reward in Eq. 4

16: Train representations $\phi$ to minimize $\mathcal{L}_{\phi}$ in Eq. 1

###  3.4 Exploratory Goal Selection

For unsupervised GCRL, selecting low-density (less visited) states as
exploratory goals can enhance goal-directed exploration [15, 16]. However, the
concept of “density” of a state does not necessarily indicate how rare or hard
to reach the state. For example, while a robotic arm might actively seek out
unseen (low-density) joint positions, interacting with objects could offer
more significant learning opportunities [27]. Thus, we propose selecting goals
that are temporally distant from states that are already visited (i.e. in the
replay buffer) to explore not only diverse but also hard-to-reach states.

To sample a faraway goal at the start of each episode, we employ the non-
parametric particle-based entropy estimator [25] on top of our temporal
distance-aware representations. We choose $N$ goals with the top-$N$ highest
entropy, which we refer to as TLDR reward, and collect $N$ corresponding
trajectories using the goal-reaching policy. The TLDR reward for each state is
computed as follows:

$r_{\text{TLDR}}(\mathbf{s})=\log\left(1+\frac{1}{k}\sum_{{\mathbf{z}^{(j)}}\in
N_{k}(\phi(\mathbf{s}))}\lVert\phi(\mathbf{s})-\mathbf{z}^{(j)}\rVert\right),$
(2)

where $N_{k}(\cdot)$ denotes the $k$-nearest neighbors around
$\phi(\mathbf{s})$ within a single minibatch.

###  3.5 Learning Exploration Policy

After the goal-conditioned policy navigates towards the chosen goal
$\mathbf{g}$ for $T_{G}$ steps, the exploration policy $\pi_{\theta}^{E}$ is
executed to discover states even more distant from the visited states. This
objective of the exploration policy can be simply defined as:

$r^{E}(\mathbf{s},\mathbf{s}^{\prime})=r_{\text{TLDR}}(\mathbf{s}^{\prime})-r_{\text{TLDR}}(\mathbf{s}).$
(3)

Similar to LEXA [10], we alternate between goal-reaching episodes and
exploration episodes. For goal-reaching episodes, we execute the goal-
conditioned policy until the end of the episodes. For exploration episodes, we
sample the timestep $T_{G}\sim\text{Unif}(0,T-1)$ at the beginning of each
episode and execute the exploration policy if timestep $t\geq T_{G}$.

###  3.6 Learning Goal-Conditioned Policy

The goal-conditioned policy aims to minimize the distance to the goal.
However, defining “distance” to the goal often requires domain knowledge.
Instead, we propose leveraging a task-agnostic metric, temporal distance, as
the learning signal for the goal-conditioned policy:

$r^{G}(\mathbf{s},\mathbf{s}^{\prime},\mathbf{g})=\lVert\phi(\mathbf{s})-\phi(\mathbf{g})\rVert-\lVert\phi(\mathbf{s}^{\prime})-\phi(\mathbf{g})\rVert.$
(4)

If our representations accurately capture temporal distances between states,
optimizing this reward in a greedy manner becomes sufficient for learning an
optimal goal-reaching policy.

(a) Ant

(b) HalfCheetah

(c) Humanoid-Run

(d) Quadruped-Escape

(e) AntMaze-Large

(f) AntMaze-Ultra

(g) Quadruped (Pixel)

(h) Kitchen (Pixel)

Figure 3: Benchmark environments. We evaluate our method on $8$ robotic
locomotion and manipulation environments.

##  4 Experiments

In this paper, we propose TLDR, a novel unsupervised GCRL method that utilizes
temporal distance-aware representations for both exploration and optimizing a
goal-conditioned policy. Through our experiments, we aim to answer the
following $3$ questions: (1) Does TLDR explore better compared to other
exploration methods? (2) Is our goal-conditioned policy better than prior
unsupervised GCRL methods? (3) How crucial is TLDR for goal-conditioned policy
learning and exploration?

###  4.1 Experimental Setup

(a) Ant

(b) HalfCheetah

(c) Humanoid-Run

(d) Quadruped-Escape

(e) AntMaze-Large

(f) AntMaze-Ultra

Figure 4: State coverage on state-based environments. We measure the state
coverage of unsupervised exploration methods. Our method consistently shows
superior state coverage compared to other methods, except in HalfCheetah
compared against METRA.

##### Tasks.

We evaluate our method in $6$ state-based environments and $2$ pixel-based
environments, as illustrated in Figure 3. For state-based environments, we use
Ant and HalfCheetah from OpenAI Gym [31], Humanoid-Run and Quadruped-Escape
from DeepMind Control Suite (DMC) [32], AntMaze-Large from D4RL [33], and
AntMaze-Ultra [34]. For Humanoid-Run and Quadruped-Escape, we include the 3D
coordinates of the agents in their observations. For pixel-based environments,
we use Quadruped (Pixel) from METRA [1] and Kitchen (Pixel) from D4RL [33],
with the image size of $64\times 64\times 3$ as the observation.

##### Comparisons.

We compare our method with $6$ prior unsupervised GCRL, skill discovery, and
exploration methods. For state-based environments, we compare with METRA, PEG,
APT, RND, and Disagreement. For pixel-based environments, we compare with
METRA and LEXA.

  * •

METRA [1]: the state-of-the-art unsupervised skill discovery method which
leverages temporal distance-aware representations.

  * •

PEG [2]: the state-of-the-art unsupervised GCRL method which plans to obtain
goals with maximum exploration rewards.

  * •

LEXA [10]: uses world model to train an Achiever and Explorer policy.

  * •

APT [25]: maximizes the entropy reward estimated from the $k$-nearest
neighbors in a minibatch.

  * •

RND [35]: uses the distillation loss of a network to a random target network
as rewards.

  * •

Disagreement [36]: utilizes the disagreement among an ensemble of world models
as rewards.

##### Evaluation setups.

Following METRA [1] and PEG [2], we evaluate unsupervised exploration using
state coverage or queue state coverage, and evaluate goal-reaching performance
using goal distance or the number of reached goals (achieved tasks). State
coverage is calculated as the number of $1\times 1$ sized $(x,y)$-bins
($x$-bins for HalfCheetah) occupied by any of the training trajectories. Queue
state coverage for Kitchen (Pixel) is the number of tasks achieved at least
once during the last $100,000$ environment steps. For Ant, HalfCheetah,
Humanoid-Run, and Quadruped (Pixel), we compute the goal distance by randomly
selecting a target goal, executing the goal-reaching policy, and measuring the
distance between the final state of the policy and the target goal. For
AntMaze and Kitchen (Pixel), we measure the number of reached goals and
achieved tasks, respectively. More experimental details are described in
Appendix A.

###  4.2 Quantitative Results

(a) Ant ($\downarrow$)

(b) HalfCheetah ($\downarrow$)

(c) Humanoid-Run ($\downarrow$)

(d) AntMaze-Large ($\uparrow$)

(e) AntMaze-Ultra ($\uparrow$)

Figure 5: Goal-reaching metrics of a goal-conditioned policy. We first report
the average distance between goals and the last states of trajectories (lower
is better $\downarrow$) in (a) Ant, (b) HalfCheetah, and (c) Humanoid-Run.
TLDR achieves a comparable average goal distance to METRA. For AntMaze
environments, we report the number of pre-defined goals reached by a goal-
reaching policy ($7$ for (d) AntMaze-Large and $21$ for (e) AntMaze-Ultra),
and TLDR significantly outperforms prior works.

In Figure 4, we compare the state coverage during training. TLDR outperforms
all prior works, except in HalfCheetah compared to METRA. METRA learns low-
dimensional skills and extends the temporal distance along a few directions
specified by the skills, providing a strong inductive bias for simple
locomotion tasks like HalfCheetah. On the other hand, TLDR achieves much
larger state coverage in complex environments than METRA, including AntMaze-
Large, AntMaze-Ultra, and Quadruped-Escape, where all other methods struggle
and only explore limited regions. This shows the strength of our method in the
exploration of complex environments.

We then compare the goal-reaching performance of our method with PEG and METRA
in Figure 5. We first report the average distance between goals and the last
states of trajectories. The results in Figures 5(a), 5(b) and 5(c) show that
TLDR can navigate towards the given goals closer than, or at least on par with
METRA. For the AntMaze environments, we report the number of pre-defined goals
reached by the goal-conditioned policy. Figures 5(d) and 5(e) show that TLDR
is the only method that can navigate towards a various set of goals in both
mazes, demonstrating its superior exploration and goal-conditioned policy
learning with temporal distance.

(a) Quadruped (Pixel)

(b) Kitchen (Pixel)

Figure 6: Results in pixel-based environments. We compare TLDR with prior
works in pixel-based Quadruped and Kitchen environments. In Quadruped (Pixel),
TLDR demonstrates a slow learning speed compared to METRA and LEXA. For
Kitchen (Pixel), TLDR could interact with all six objects during training, but
shows low success rates for evaluation.

Figure 6 shows the results in pixel-based environments. In Quadruped (Pixel),
TLDR explores diverse regions but learns slower than LEXA and METRA. For
Kitchen (Pixel), TLDR interacts with all six objects during training, but
struggles at learning the goal-conditioned policy. We hypothesize that
learning a temporal abstraction is more challenging with pixel observations,
which may lead $\phi$ to encode erroneous temporal information. We leave more
detailed analyses for future works.

(a) TLDR (ours)

(b) METRA

(c) PEG

Figure 7: Goal-reaching ability in AntMaze-Ultra. TLDR can cover more goals
compared to METRA and PEG.

###  4.3 Qualitative Results

We visualize the learned goal-reaching behaviors on the AntMaze-Ultra
environment in Figure 7. TLDR can successfully reach both near and faraway
goals in diverse regions. On the other hand, METRA and PEG fail to navigate to
diverse goals. METRA could reach some goals distant from the initial position,
whereas PEG fails to reach temporally faraway goals. This clearly shows the
benefit of using temporal distance in unsupervised GCRL.

###  4.4 Ablation Studies

To investigate the importance of temporal distance-aware representations in
our algorithm, we conduct ablation studies on exploration strategies and GCRL
reward designs.

##### Exploration strategy.

For goal selection and exploration rewards, we replace temporal distance,
$\lVert\phi(\mathbf{s})-\mathbf{z}^{(j)}\rVert$ in Equation 2, with other
exploration bonuses: RND, APT (with ICM [37] representations), and
Disagreement. Note that goal-conditioned policies are still trained with the
same temporal distance-based rewards as TLDR, thereby comparing only
exploration strategies. As shown in Figure 8(a), using TLDR reward for goal
selection and exploration rewards achieves significantly higher performance
than other exploration bonuses. This result implies that our temporal
distance-based rewards are effective for unsupervised exploration.

##### GCRL reward design.

We compare with two goal-conditioned policy learning methods: (1) QRL [14],
which uses a quasimetric value function, and (2) sparse HER [8], which uses
the sparse goal-reaching reward $-\mathbbm{1}(\mathbf{s}\neq\mathbf{g})$.
Figure 8(b) shows the superior performance of our temporal distance-based GCRL
reward. This highlights the importance of incorporating temporal distance-
aware representations in training goal-conditioned policies.

(a) TLDR with different exploration methods

(b) TLDR with different GCRL rewards

Figure 8: Impact of temporal distance-aware representations in exploration and
GCRL reward design. We evaluate our method with different design choices for
(a) exploration methods and (b) GCRL rewards on Ant and AntMaze-Large. TLDR
shows better state coverages than its ablated versions in both ablation
studies, indicating the importance of using temporal distance for both
exploration and GCRL.

##  5 Conclusion

In this paper, we introduce TLDR, an unsupervised GCRL algorithm that
incorporates temporal distance-aware representations. TLDR leverages temporal
distance for exploration and learning the goal-reaching policy. By pursuing
states with larger temporal distances, TLDR can continuously explore
challenging regions, achieving better state coverage. The experimental results
demonstrate that our method can cover significantly larger state spaces across
diverse environments than existing unsupervised reinforcement learning
algorithms.

###  5.1 Limitations

While TLDR achieves remarkable state coverage, it still has several
limitations. Firstly, TLDR shows a slower learning speed compared to METRA in
pixel-based environments. Secondly, our temporal distance-aware
representations do not capture the asymmetric temporal distance between the
states, which can make policy learning challenging for asymmetric
environments. Finally, TLDR achieves high efficiency in terms of wall clock
time, but with a relatively low update-to-data ratio (number of gradient steps
divided by number of environment steps) of $1/32$ in state-based experiments,
as used in METRA. However, we believe that increasing the update-to-data ratio
or using model-based approaches could potentially enhance sample efficiency.

#### Acknowledgments

This work was supported in part by the Institute of Information &
Communications Technology Planning & Evaluation (IITP) grant
(RS-2020-II201361, Artificial Intelligence Graduate School Program (Yonsei
University)) and the National Research Foundation of Korea (NRF) grant
(RS-2024-00333634), and the Electronics and Telecommunications Research
Institute (ETRI) grant (24ZR1100) funded by the Korean Government (MSIT).

## References

  * Park et al. [2024] S. Park, O. Rybkin, and S. Levine.  Metra: Scalable unsupervised rl with metric-aware abstraction.  In _International Conference on Learning Representations_ , 2024.
  * Hu et al. [2022] E. S. Hu, R. Chang, O. Rybkin, and D. Jayaraman.  Planning goals for exploration.  In _International Conference on Learning Representations_ , 2022.
  * Kaelbling [1993] L. P. Kaelbling.  Learning to achieve goals.  In _International Joint Conference on Artificial Intelligence_ , volume 2, pages 1094–8, 1993.
  * Deguchi and Takahashi [1999] K. Deguchi and I. Takahashi.  Image-based simultaneous control of robot and target object motions by direct-image-interpretation method.  In _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 375–380, 1999.
  * Schaul et al. [2015] T. Schaul, D. Horgan, K. Gregor, and D. Silver.  Universal value function approximators.  In _International Conference on Machine Learning_ , pages 1312–1320, 2015.
  * Watter et al. [2015] M. Watter, J. Springenberg, J. Boedecker, and M. Riedmiller.  Embed to control: A locally linear latent dynamics model for control from raw images.  In _Advances in Neural Information Processing Systems_ , pages 2746–2754, 2015.
  * Finn et al. [2016] C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and P. Abbeel.  Deep spatial autoencoders for visuomotor learning.  In _IEEE International Conference on Robotics and Automation_ , pages 512–519. IEEE, 2016.
  * Andrychowicz et al. [2017] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba.  Hindsight experience replay.  In _Advances in Neural Information Processing Systems_ , volume 30, 2017.
  * Zhu et al. [2017] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi.  Target-driven visual navigation in indoor scenes using deep reinforcement learning.  In _IEEE International Conference on Robotics and Automation_ , pages 3357–3364, 2017.
  * Mendonca et al. [2021] R. Mendonca, O. Rybkin, K. Daniilidis, D. Hafner, and D. Pathak.  Discovering and achieving goals via world models.  In _Neural Information Processing Systems_ , 2021.
  * Pitis et al. [2020] S. Pitis, H. Chan, S. Zhao, B. Stadie, and J. Ba.  Maximum entropy gain exploration for long horizon multi-goal reinforcement learning.  In _International Conference on Machine Learning_ , pages 7750–7761. PMLR, 2020.
  * Hafner et al. [2022] D. Hafner, K.-H. Lee, I. Fischer, and P. Abbeel.  Deep hierarchical planning from pixels.  In _Neural Information Processing Systems_ , volume 35, pages 26091–26104, 2022.
  * Park et al. [2024] S. Park, T. Kreiman, and S. Levine.  Foundation policies with hilbert representations.  In _International Conference on Machine Learning_ , 2024.
  * Wang et al. [2023] T. Wang, A. Torralba, P. Isola, and A. Zhang.  Optimal goal-reaching reinforcement learning via quasimetric learning.  In _International Conference on Machine Learning_ , pages 36411–36430. PMLR, 2023.
  * Pong et al. [2020] V. H. Pong, M. Dalal, S. Lin, A. Nair, S. Bahl, and S. Levine.  Skew-Fit: State-covering self-supervised reinforcement learning.  In _International Conference on Machine Learning_ , 2020.
  * Pitis et al. [2020] S. Pitis, H. Chan, S. Zhao, B. C. Stadie, and J. Ba.  Maximum entropy gain exploration for long horizon multi-goal reinforcement learning.  In _International Conference on Machine Learning_ , 2020.
  * Lee et al. [2019] Y. Lee, S.-H. Sun, S. Somasundaram, E. S. Hu, and J. J. Lim.  Composing complex skills by learning transition policies.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=rygrBhC5tQ.
  * Lee et al. [2021] Y. Lee, A. Szot, S.-H. Sun, and J. J. Lim.  Generalizable imitation learning from observation via inferring goal proximity.  In _Neural Information Processing Systems_ , 2021.
  * Ecoffet et al. [2021] A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune.  First return, then explore.  _Nature_ , 590(7847):580–586, 2021.
  * Pong et al. [2020] V. H. Pong, M. Dalal, S. Lin, A. Nair, S. Bahl, and S. Levine.  Skew-fit: State-covering self-supervised reinforcement learning.  In _International Conference on Machine Learning_ , 2020.
  * Gregor et al. [2016] K. Gregor, D. J. Rezende, and D. Wierstra.  Variational intrinsic control.  _ArXiv_ , abs/1611.07507, 2016.
  * Achiam et al. [2018] J. Achiam, H. Edwards, D. Amodei, and P. Abbeel.  Variational option discovery algorithms.  _ArXiv_ , abs/1807.10299, 2018.
  * Eysenbach et al. [2019] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine.  Diversity is all you need: Learning skills without a reward function.  In _International Conference on Learning Representations (ICLR)_ , 2019.
  * Sharma et al. [2020] A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman.  Dynamics-aware unsupervised discovery of skills.  In _International Conference on Learning Representations (ICLR)_ , 2020.
  * Liu and Abbeel [2021] H. Liu and P. Abbeel.  Behavior from the void: Unsupervised active pre-training.  In _Neural Information Processing Systems_ , 2021.
  * Park et al. [2022] S. Park, J. Choi, J. Kim, H. Lee, and G. Kim.  Lipschitz-constrained unsupervised skill discovery.  In _International Conference on Learning Representations_ , 2022.
  * Park et al. [2023] S. Park, K. Lee, Y. Lee, and P. Abbeel.  Controllability-aware unsupervised skill discovery.  In _International Conference on Machine Learning_ , pages 27225–27245. PMLR, 2023.
  * Sermanet et al. [2018] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain.  Time-contrastive networks: Self-supervised learning from video.  In _IEEE International Conference on Robotics and Automation_ , pages 1134–1141. IEEE, 2018.
  * Nair et al. [2022] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta.  R3m: A universal visual representation for robot manipulation.  In _Conference on Robot Learning_ , 2022.
  * Ma et al. [2023] Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang.  Vip: Towards universal visual reward and representation via value-implicit pre-training.  In _International Conference on Learning Representations_ , 2023.
  * Brockman et al. [2016] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba.  OpenAI Gym.  _ArXiv_ , abs/1606.01540, 2016.
  * Tassa et al. [2018] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, T. P. Lillicrap, and M. A. Riedmiller.  Deepmind control suite.  _arXiv preprint arXiv:1801.00690_ , 2018.
  * Fu et al. [2020] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine.  D4rl: Datasets for deep data-driven reinforcement learning.  _arXiv preprint arXiv:2004.07219_ , 2020.
  * Jiang et al. [2023] Z. Jiang, T. Zhang, M. Janner, Y. Li, T. Rocktäschel, E. Grefenstette, and Y. Tian.  Efficient planning in a compact latent action space.  In _International Conference on Learning Representations_ , 2023.
  * Burda et al. [2019] Y. Burda, H. Edwards, A. J. Storkey, and O. Klimov.  Exploration by random network distillation.  In _International Conference on Learning Representations_ , 2019.
  * Pathak et al. [2019] D. Pathak, D. Gandhi, and A. K. Gupta.  Self-supervised exploration via disagreement.  In _International Conference on Machine Learning_ , 2019.
  * Pathak et al. [2017] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell.  Curiosity-driven exploration by self-supervised prediction.  In _International conference on machine learning_ , pages 2778–2787. PMLR, 2017.
  * Haarnoja et al. [2018] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _International Conference on Machine Learning_ , 2018.
  * Laskin et al. [2021] M. Laskin, D. Yarats, H. Liu, K. Lee, A. Zhan, K. Lu, C. Cang, L. Pinto, and P. Abbeel.  Urlb: Unsupervised reinforcement learning benchmark.  In _Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track_ , 2021.
  * Kingma and Ba [2015] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations_ , 2015.
  * Ba et al. [2016] J. L. Ba, J. R. Kiros, and G. E. Hinton.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * Gupta et al. [2019] A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman.  Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning.  In _Conference on Robot Learning_ , 2019.

##  Appendix A Training Details

###  A.1 Computing Resources and Experiments

All experiments are done on a single RTX 4090 GPU and $4$ CPU cores. Each
state-based experiment takes $12$ hours for all methods, following METRA [1],
which trains each method for $9$-$10$ hours. It corresponds to the different
environment steps used for different experiments, as described in Table 1. We
use $5$ random seeds for all experiments, and report the mean and standard
deviation of the results.

Table 1: The number of environment steps for experiments. Environment | TLDR | METRA | PEG | LEXA | APT | RND | Disagreement
---|---|---|---|---|---|---|---
Ant |  $56.5$M |  $83.2$M |  $0.7$M | - |  $2.4$M |  $4.1$M |  $4.8$M
HalfCheetah |  $51.4$M |  $103.5$M |  $0.7$M | - |  $2.5$M |  $4.2$M |  $5.0$M
AntMaze-Large |  $42.6$M |  $62.5$M |  $0.7$M | - |  $2.4$M |  $6.4$M |  $5.0$M
AntMaze-Ultra |  $31.2$M |  $44.5$M |  $0.6$M | - |  $2.4$M |  $4.5$M |  $3.4$M
Quadruped-Escape |  $28.0$M |  $34.8$M |  $0.6$M | - |  $2.2$M |  $4.5$M |  $4.4$M
Humanoid-Run |  $40.8$M |  $59.9$M |  $0.6$M | - |  $3.5$M |  $4.7$M |  $4.7$M
Quadruped (Pixel) |  $3.9$M |  $4.1$M | - |  $2.1$M | - | - | -
Kitchen (Pixel) |  $1.1$M |  $1.7$M | - |  $1.0$M | - | - | -

###  A.2 Implementation Details

Our method, TLDR, is implemented on top of the official implementation of
METRA. Similar to METRA, we use SAC [38] for learning the goal-reaching policy
and exploration policy. We train our temporal distance-aware representation
$\phi(\mathbf{s})$ by maximizing the following objective:

$\mathbb{E}_{\mathbf{s}\sim p_{\mathbf{s}},\mathbf{g}\sim
p_{\mathbf{g}}}\left[f(\lVert\phi(\mathbf{s})-\phi(\mathbf{g})\rVert)+\lambda\cdot\min\left(\epsilon,1-\lVert\phi(\mathbf{s})-\phi(\mathbf{s}^{\prime})\rVert\right)\right],$
(5)

where we apply affine-transformed softplus $f$ to Equation 1:

$f(x)=-\text{softplus}(500-x,\beta=0.01),$ (6)

which alleviates the effect of too long distances
$\lVert\phi(\mathbf{s})-\phi(\mathbf{g})\rVert$, following QRL [14].

For training the exploration policy, we normalize the TLDR reward used in
Equation 3 to keep the rewards on a consistent scale. We simply divide the
TLDR reward by a running estimate of its mean value, following APT [25].

For METRA, PEG, and LEXA, we use their official implementation. For random
exploration approaches (APT, RND, Disagreement), we use the implementation
from URLB [39].

###  A.3 Hyperparameters

The hyperparameters used in our experiments are summarized in Table 2.

For METRA, we use $2$-D continuous skills for Ant, $16$-D discrete skills for
HalfCheetah, $24$-D discrete skills for Kitchen (Pixel), and $4$-D continuous
skills for other environments. We use the batch size of $1024$ for state-based
environments and $256$ for pixel-based environments. We set the number of
gradient steps for each experiment to be the same as ours. We use the default
values for the remaining hyperparameters. To perform goal-reaching tasks with
METRA, we set the skill $\mathbf{z}$ as
$\frac{\phi(\mathbf{g})-\phi(\mathbf{s})}{\lVert\phi(\mathbf{g})-\phi(\mathbf{s})\rVert}$
for continuous skills or
$\arg\max_{\text{dim}}\left(\phi(\mathbf{g})-\phi(\mathbf{s})\right)$ for
discrete skills.

In PEG, we use the same hyperparameters used in their AntMaze experiments.
Since PEG uses the normalized goal space, we measure the range of the
observations and normalize the goal states according to the minimum and
maximum range.

In LEXA, we follow their hyperparameters and opt for the temporal distance
reward for training the Achiever policy.

For APT with ICM encoder, RND, and Disagreement, we use the same
hyperparemeters as in URLB [39].

For the ablation with QRL, we use the learning rate of $0.0003$ for critics.
We use an (input dim)-$1024$-$1024$-$128$ network for the encoder,
$256$-$1024$-$2048$ for the projector, IQE-maxmean head of $64$ components of
size $32$, and $128$-$1024$-$128$ for the latent dynamics model. The
transition loss is weighted by $1$. For HER, we use the discount factor
$\gamma=0.99$.

Table 2: List of hyperparameters. Hyperparameter | Value
---|---
Learning rate | $0.0001$
Learning rate for $\phi$ | $0.0005$
Batch size |  $1024$ (State), $256$ (Pixel)
Replay buffer size |  $10^{6}$ (State), $3\times 10^{5}$ (Quadruped (Pixel)), $10^{5}$ (Kitchen)
Frame stack (Pixel) | $3$
Optimizer | Adam [40]
Relaxation constant $\epsilon$ in Eq. 5 | $10^{-3}$
$\dim\phi(\mathbf{s})$ |  $8$ (Kitchen), $4$ (Others)
$k$ in Eq. 2 | $12$
Initial $\lambda$ | $3\times 10^{3}$
SAC entropy coefficient |  $0.01$ (Kitchen), target entropy as $(-\dim\mathcal{A})$ (others)
Discount factor $\gamma$ |  $0.97$ (Goal-reaching policy), $0.99$ (Exploration policy)
Normalization | LayerNorm [41] for the critics, None for $\phi$ and actors
Encoder for image observations | CNN
MLP dimensions | $1024$
MLP depths | $2$
Goal relabelling |  $0.8$ (sampled from future observations), $0.2$ (no relabelling)
# of gradient steps per epoch |  $50$ (Ant, HalfCheetah, Humanoid-Run, Quadruped-Escape), $75$ (AntMaze-Large), $100$ (Kitchen), $150$ (AntMaze-Ultra), $200$ (Quadruped (Pixel))
# of episode rollouts per epoch | $8$
$\tau$ for updating the target network | $0.995$

###  A.4 Environment Details

##### Ant.

We use the MuJoCo Ant environment in OpenAI gym [31]. The observation space is
$29$-D and the action space is $8$-D. Following METRA, we normalize the
observations for Ant with a fixed mean and standard deviation of observations
computed from randomly generated trajectories. The episode length is $200$.

##### HalfCheetah

We use the MuJoCo HalfCheetah environment in OpenAI gym [31]. The observation
space is $18$-D and the action space is $6$-D. Following METRA, we normalize
the observations for HalfCheetah with a fixed mean and standard deviation of
observations from randomly generated trajectories. The episode length is
$200$.

##### Humanoid-Run.

We use the Humanoid-Run task from DeepMind Control Suite [32]. The global
$x,y,z$ coordinates of the agent are added to the observation. Humanoid has
$55$-D observation space with $21$-D action space. The episode length is
$200$.

##### Quadruped-Escape.

Quadruped-Escape is included in DeepMind Control Suite [32]. The quadruped
robot is initialized in a basin surrounded by complex terrains, as described
in Figure 3(d). Due to the complex terrains, moving further away from the
initial position is challenging. Similar to the AntMaze environments, we fix
the terrain shape. Also, we add the global $x,y,z$ coordinates of the agent to
the observation. Quadruped-Escape has $104$-D observation space with $12$-D
action space. The episode length is $200$.

##### AntMaze-Large.

We use antmaze-large-play-v2 in D4RL [33]. The observation and action spaces
are the same as the Ant environment. The episode length is $300$. To make
exploration more challenging, we fix the initial location of the agent to be
the bottom right corner of the maze, as shown in Figure 3(e).

##### AntMaze-Ultra.

We use antmaze-ultra-play-v0 proposed by Jiang et al. [34]. The observation
and action spaces are the same as the Ant environment. The episode length is
$600$. Similar to AntMaze-Large, we fix the initial location of the agent to
be the bottom right corner of the maze, as shown in Figure 3(f).

##### Quadruped (Pixel).

We use the pixel-based version of the Quadruped environment [32] used in METRA
[1]. Specifically, we use the image size of $64\times 64\times 3$ with $200$
episode length.

##### Kitchen (Pixel).

We use the pixel-based version of the Kitchen environment [42] used in METRA
[1] and LEXA [10]. Specifically, we use the image size of $64\times 64\times
3$ with $50$ episode length. The action space has $9$ dimensions.

###  A.5 Evaluation Protocol

For Ant, Humanoid, and Quadruped (Pixel), we sample goals with
$(x,y)$-coordinates from $[-50,50]^{2}$, $[-40,40]^{2}$, and $[-15,15]^{2}$,
respectively. For the rest of the goal state (e.g. joint poses), we use the
initial robot configuration following Park et al. [1].

For HalfCheetah, we sample goals with $x$-coordinates from $[-100,100]$.

For AntMaze-Large and AntMaze-Ultra, we use the pre-defined goals as shown in
Figure 7. A goal is deemed to be reached when an ant gets closer than $0.5$ to
the goal.

For Kitchen, we use the same $6$ single-task goal images used in LEXA [10],
which consist of interactions with Kettle, Microwave, Light switch, Hinge
cabinet, Slide cabinet, and Bottom burner. We report the total number of
achieved tasks during evaluation.

For all environments, we use a full state as a goal. Specifically, for state-
based observations, we use the observation upon reset as the base observation
and switch the $x,y$ coordinates (or $x$ for HalfCheetah) to the corresponding
dimensions. For Quadruped (Pixel), we render the image of the state where the
agent is at the goal position and use it as the goal.

##  Appendix B More Ablation Studies

We conduct the ablation studies on the number of nearest neighbors $k$ (Figure
9) and $\dim\phi(\mathbf{s})$ (Figure 10) used in Equation 2. Figure 9 shows
that different $k$ affects exploration in Ant, but for the other environments,
the performance is not affected by the values of $k$. For
$\dim\phi(\mathbf{s})$, the performance is nearly the same across different
settings.

(a) Ant

(b) AntMaze-Large

(c) Humanoid-run

Figure 9: State coverage on state-based environments with different $k$. We
measure the state coverage of our method with $k\in\\{5,12,20\\}$ used for
calculating the TLDR reward in Equation 2. For Ant, $k=12$ works the best. For
other environments, $k$ does not affect the state coverage.

(a) Ant

(b) AntMaze-Large

(c) Humanoid-run

Figure 10: State coverage on state-based environments with different
$\mathbf{\dim\phi(\mathbf{s})}$. We measure the state coverage of our method
with $\dim\phi(\mathbf{s})\in\\{2,4,8,16\\}$, where $\dim\phi(\mathbf{s})$ is
the dimension of the temporal distance-aware representations. The results show
that $\dim\phi(\mathbf{s})$ does not have a critical impact on the performance
in these environments.

##  Appendix C More Qualitative Results

We include more qualitative results in Figures 14, 13, 11 and 12. For the
qualitative results in Quadruped-Escape (Figure 12), we evenly select $48$
states satisfying $x^{2}+y^{2}=10^{2}$, where $x$, $y$ represents the agent
position. The $z$ coordinate is selected as the minimum possible height that
the agent does not collide with the terrain. For all environments, TLDR
achieves the best goal-reaching behaviors compared to the other unsupervised
GCRL methods, covering the goals in more diverse regions.

(a) TLDR

(b) METRA

(c) PEG

Figure 11: Goal-reaching ability in Humanoid-Run. We evaluate each method with
the goals sampled according to Section A.5. TLDR moves further towards the
goal in diverse directions compared to other methods.

(a) TLDR

(b) METRA

(c) PEG

Figure 12: Goal-reaching ability in Quadruped-Escape. We evaluate each method
with the goals that are evenly selected at the same distance from the origin.
TLDR can not only cover more regions but also have a better goal-reaching
capability, compared to other methods.

(a) TLDR

(b) METRA

(c) PEG

Figure 13: Goal-reaching ability in AntMaze-Large. TLDR can reach most of the
goals in AntMaze-Large, while other GCRL methods struggle to reach distant
goals.

(a) TLDR

(b) METRA

(c) PEG

Figure 14: Goal-reaching ability in AntMaze-Ultra. TLDR can cover the most
number of goals in AntMaze-Ultra compared to other methods.