# Rapid Exploration for Open-World Navigation
with Latent Goal Models

Dhruv Shah1, Benjamin Eysenbach2, Nicholas Rhinehart1, Sergey Levine1
1UC Berkeley, 2Carnegie Mellon University

###### Abstract

We describe a robotic learning system for autonomous exploration and
navigation in diverse, open-world environments. At the core of our method is a
learned latent variable model of distances and actions, along with a non-
parametric topological memory of images. We use an information bottleneck to
regularize the learned policy, giving us (i) a compact visual representation
of goals, (ii) improved generalization capabilities, and (iii) a mechanism for
sampling feasible goals for exploration. Trained on a large offline dataset of
prior experience, the model acquires a representation of visual goals that is
robust to task-irrelevant distractors. We demonstrate our method on a mobile
ground robot in open-world exploration scenarios. Given an image of a goal
that is up to 80 meters away, our method leverages its representation to
explore and discover the goal in under 20 minutes, even amidst previously-
unseen obstacles and weather conditions. Please check out the project website
for videos of our experiments and information about the real-world dataset
used111Project website: sites.google.com/view/recon-robot..

##  1 Introduction

Robustness is a key challenge in learning to navigate diverse, real-world
environments. A robotic learning system must be robust to the difference
between an offline training dataset and the real world (i.e., it must
generalize), be robust to non-stationary changes in the real world (i.e., it
must ignore visual distractors), and be equipped with mechanisms to actively
explore to gather information about traversability. Different environments may
exhibit similar physical structures, and these similarities can be used to
accelerate exploration of new environments. Learning-based methods provide an
appealing approach for learning a representation of this shared structure
using prior experience.

Figure 1: We demonstrate RECON  on a Clearpath Jackal.

In this work, we consider the problem of navigating to a user-specified goal
in a previously unseen environment. The robot has access to a large and
diverse dataset of experience from _other_ environments, which it can use to
learn general navigational affordances. Our approach to this problem uses an
information bottleneck architecture to learn a compact representation of
goals. Learned from prior data, this latent goal model encodes prior knowledge
about perception, navigational affordances, and short-horizon control. We use
a non-parametric memory to incorporate experience from the new environment.
Combined, these components enable our system to learn to navigate to goals in
a new environment after only a few minutes of exploration.

The primary contribution of this work is a method for exploring novel
environments to discover user-specified goals. Our method operates directly on
a stream of image observations, without relying on structured sensors or
geometric maps. An important part of our method is a compressed representation
of goal images that simultaneously affords robustness while providing a simple
mechanism for exploration. Such a representation allows us, for example, to
specify a goal image at one time of day, and then navigate to that same place
at a different time of day: despite variation in appearance, the latent goal
representations must be sufficiently close that the model can produce the
correct actions. Robustness of this kind is critical in real-world settings,
where the appearance of landmarks can change significantly with times of day
and seasons of the year.

We demonstrate our method, Rapid Exploration Controllers for Outcome-driven
Navigation (RECON), on a mobile ground robot (Fig. 1) and evaluate against 6
competitive baselines spanning over 100 hours of real-world experiments in 8
distinct open-world environments (Fig. 2). Our method can discover user-
specified goals up to 80m away after just 20 minutes of interaction in a new
environment. We also demonstrate robustness in the presence of visual
distractors and novel obstacles. We make this dataset publicly available as a
source of real-world interaction data for future resesarch.

Figure 2: System overview: Given a goal image _(a)_ , RECON explores the
environment _(b)_ by sampling prospective _latent_ goals and constructing a
topological map of images (white dots), operating only on visual observations.
After finding the goal _(c)_ , RECON can reuse the map to reach arbitrary
goals in the environment (red path in _(b)_). RECON uses data collected from
diverse training environments _(d)_ to learn navigational priors that enable
it to quickly explore and learn to reach visual goals a variety of unseen
environments _(e)_.

##  2 Related Work

Exploring a new environment is often framed as the problem of efficient
mapping, posed in terms of information maximization to guide the robot to
uncertain regions of the environment. Some prior exploration methods use local
strategies for generating control actions for the robots [1, 2, 3, 4], while
others use use global strategies based on the frontier method [5, 6, 7].
However, building high-fidelity geometric maps can be hard without reliable
depth information. Further, such maps do not encode semantic aspects of
traversability, e.g., tall grass is traversable but a wire fence is not.

Inspired by prior work [8, 9, 10, 11, 12], we construct a topological map by
learning a distance function and a low-level policy. We estimate distances via
supervised regression and learn a local control policy via goal-conditioned
behavior cloning [13, 14]. However, these prior methods do not describe how to
learn to navigate in _new_ , unseen environments. We equip RECON with an
explicit mechanism for exploring new environments and transferring knowledge
across environments.

Well-studied methods for exploration in reinforcement learning (RL) utilize a
novelty-based bonus, computed from a predictive model [15, 16, 17, 18, 19, 20,
21], information gain [22, 23], or methods based on counts, densities, or
distance from previously-visited states [24, 25, 26]. However, these methods
learn to reason about the novelty of a state only after visiting it. Recent
works [27, 28] improve upon this by predicting explorable areas for
interesting parts of the environment to accelerate visual exploration. While
these methods can yield state-of-the-art results in simulated domains [29,
30], they come at the cost of high sample complexity (over 1M samples) and are
infeasible to train in open-world environments without a simulated
counterpart. Instead, our method enables the robot to explore an environment
from scratch in just 20 minutes, using prior experience from other
environments.

The problem of reusing experience across tasks is studied in the context of
meta-learning [31, 32, 33] and transfer learning [34, 35, 36, 37, 38]. Our
method uses an information bottleneck [39], which serves a dual purpose:
first, it provides a representation that can aid the generalization
capabilities of RL algorithms [40, 41], and second, it serves as a measure of
task-relevant uncertainty [42], allowing us to incorporate prior information
for proposing goals that are functionally-relevant for learning control
policies in the new environment.

The problem of learning goal-directed behavior has been studied extensively
using RL [43, 44, 45, 46] and imitation learning (IL) [47, 13, 14, 48, 49,
50]. Our method builds upon prior goal-conditioned IL methods to solve a
slightly different problem: how to reach goals in a _new_ environment. Once
placed in a new environment, our method explores by carefully choosing which
goals to visit, inspired by prior work [51, 52, 53, 54, 55]. Unlike these
prior methods, however, our method makes use of previous experience in
_different_ environments to accelerate learning in the current environment.

##  3 Problem Statement and System Overview

We consider the problem of goal-directed exploration for visual navigation in
novel environments: a robot is tasked with navigating to a goal location $G$,
given an image observation $o_{g}$ taken at $G$. Broadly, this consists of
three separate stages: (1) learning from offline data, (2) building a map in a
new environment, and (3) navigating to goals in the new environment. We model
the task of navigation as a Markov decision process with time-indexed states
$s_{t}\in{\mathcal{S}}$ and actions $a_{t}\in{\mathcal{A}}$. We _do not
assume_ the robot has access to spatial localization or a map of the
environment, or access to the system dynamics. We use videos of robot
trajectories in a variety of environments to learn general navigational skills
and build a compressed representation of the perceptual inputs, which can be
used to guide the exploration of novel environments. We make no assumption on
the nature of the trajectories: they may be obtained by human teleoperation,
self-exploration, or as a result of a preset policy. These trajectories need
not exhibit intelligent behavior. Since the robot only observes the world from
a single on-board camera and does not run any state estimation, our system
operates in a partially observed setting. Our system commands continuous
linear and angular velocities.

###  3.1 Mobile Robot Platform

We implement RECON on a Clearpath Jackal UGV platform (see Fig. 1). The
default sensor suite consists of a 6-DoF IMU, a GPS unit for approximate
global position estimates, and wheel encoders to estimate local odometry. In
addition, we added a forward-facing $170^{\circ}$ field-of-view RGB camera and
an RPLIDAR 2D laser scanner. Inside the Jackal is an NVIDIA Jetson TX2
computer. The GPS and laser scanner can become unreliable in some environments
[56], so we use them solely as safety controllers during data collection. Our
method operates only using images taken from the onboard RGB camera, without
other sensors or ground-truth localization.

###  3.2 Self-Supervised Data Collection & Labeling

Our aim is to leverage data collected in a wide range of different
environments to enable the robot to discover and learn to navigate to novel
goals in novel environments. We curate a dataset of self-supervised
trajectories collected by a time-correlated random walk in diverse real-world
environments (see Fig. 2 (d,e)). This data was collected over a span of $18$
months and exhibits significant variation in appearance due to seasonal and
lighting changes. We make this dataset publicly available222Available for
download at sites.google.com/view/recon-robot/dataset. and provide further
details in Appendix A.

##  4 RECON : A Method for Goal-Directed Exploration

Our objective is to design a robotic system that uses visual observations to
efficiently discover and reliably reach a target image in a previously unseen
environment. RECON consists of two components that enable it to explore new
environments. The first is an uncertainty-aware, context-conditioned
representation of goals that can quickly adapt to novel scenes. The second
component is a topological map, where nodes represent egocentric observations
and edges are the predicted distance between them, constructed incrementally
from frontier-based exploration, maintaining a compact memory of the target
environment.

###  4.1 Learning to Represent Goals

Our method learns a compact representation of goal images that is robust to
task-irrelevant factors of variation. We learn this representation using a
variant of the information bottleneck architecture [42, 57]. We use a context-
conditioned representation of goals to learn a control policy in the target
environment (Fig. 3 describes the graphical model). Letting $I(\cdot;\cdot)$
denote mutual information, the objective in Eq. 1 encourages the model to
compress the incoming goal image $o_{g}$ into a representation $z_{t}^{g}$
conditioned on the current observation $o_{t}$ that is predictive of the best
action $a_{t}^{g}$ and the temporal distance $d_{t}^{g}$ to the goal (upper-
case denotes random variables):

$\displaystyle I\big{(}(A_{t}^{g},D_{t}^{g});Z_{t}^{g}\mid o_{t}\big{)}-\beta
I(Z_{t}^{g};O_{g}\mid o_{t})$ (1)

Following [42], we approximate the intractable objective in Eq. 1 with a
variational posterior and decoder (an upper bound), resulting in the
maximization objective:

$\displaystyle L=\frac{1}{|\mathcal{D}|}\sum_{\begin{subarray}{c}\\\
(o_{t},o_{g},a_{t}^{g},d_{t}^{g})\in\mathcal{D}\end{subarray}}$
$\displaystyle\mathbb{E}_{p_{\phi}(z_{t}^{g}\mid o_{g},o_{t})}\left[\log
q_{\theta}\big{(}a_{t}^{g},d_{t}^{g}\mid
z_{t}^{g},o_{t}\big{)}\right]-\beta\mathrm{KL}\left(p_{\phi}(\cdot\mid
o_{g},o_{t})||r(\cdot)\right)$ (2)

Encoder $p_{\phi}$$(z_{t}^{g}|o_{g},o_{t})$Decoder
$q_{\theta}$$(a_{t}^{g},d_{t}^{g}|z_{g}^{t},o_{t})$$O_{g}$$Z_{t}^{g}$$A_{t}^{g}$$O_{t}$$D_{t}^{g}$

Figure 3: Graphical model of actions and distances

where we define the prior $r(z_{t}^{g})\triangleq\mathcal{N}(0,I)$ and
$\mathcal{D}$ is a dataset of trajectories characterized by
$(o_{t},o_{g},a_{t}^{g},d_{t}^{g})$ quadruples. The first term measures the
model’s ability to predict actions and distances from the encoded
representation, and the second term measures the model’s compression of
incoming goal images.

As the encoder $p_{\phi}$ and decoder $q_{\theta}$ are conditioned on $o_{t}$,
the representation $z_{t}^{g}$ only encodes information about _relative_
location of the goal from the context – this allows the model to represent
_feasible_ goals. If, instead, we had a typical VAE (in which the input images
are autoencoded), the samples from the prior over these representations would
not necessarily represent goals that are reachable from the current state.
This distinction is crucial when exploring _new_ environments, where most
states from the training environments are not valid goals.

###  4.2 Goal-Directed Exploration with Topological Memory

Algorithm 1 RECON for Exploration: RECON takes as input an encoder $p_{\phi}$,
a decoder $q_{\theta}$, prior $r$, the current observation $o_{t}$ and goal
observation $o_{g}$.
$\delta_{1},\delta_{2},\epsilon,\beta\in\mathbb{R}_{+};H,\gamma\in\mathbb{N}$
are hyperparameters.

1:function RECON
($q_{\theta},p_{\phi},r,o_{t},o_{g};\delta_{1},\delta_{2},\epsilon,\beta,\gamma,H$)

2: ${\mathcal{G}}\leftarrow\emptyset,{\mathcal{D}}\leftarrow\emptyset$
$\triangleright$ _Initialize graph and data_

3: while not reached goal $[\bar{d}_{t}^{g}<\delta_{1}]$ do $\triangleright$
_Continue while not at goal_

4:
$o_{n}\leftarrow\mathrm{LeastExploredNeighbor}({\mathcal{G}},o_{t};\delta_{2})$

5: $z_{t}^{g}\sim p_{\phi}(z\mid o_{t},o_{g})$ $\triangleright$ _Encode
relative goal_

6: if goal is feasible $[r(z_{t}^{g})>\epsilon]$ then

7: $z_{t}^{w}\leftarrow z_{t}^{g}$ $\triangleright$ Will go to the goal

8: else if robot at frontier $[\bar{d}_{t}^{n}<\delta_{1}]$ then

9: $z_{t}^{w}\sim r(z)$ $\triangleright$ _Will explore from frontier_

10: else

11: $z_{t}^{w}\sim p_{\phi}(z\mid o_{t},o_{n})$ $\triangleright$ Will go to
frontier

12: $\mathcal{D}_{w},o_{t}\leftarrow\mathrm{SubgoalNavigate}(z_{t}^{w};H)$

13: ${\mathcal{D}}\leftarrow{\mathcal{D}}\cup\mathcal{D}_{w}$

14: $\mathrm{ExpandGraph}({\mathcal{G}},o_{t})$

15: Step $L(\phi,\theta;\mathcal{D},\beta)$ for $\gamma$ epochs
$\triangleright$ Eq. 2

16: return networks $p_{\phi},q_{\theta}$ and graph ${\mathcal{G}}$

The second component of our system is a topological memory constructed
incrementally as the robot explores a new environment. It provides an estimate
of the exploration frontier as well as a map that the robot can use to later
navigate to arbitrary goals. To build this memory, the robot uses the model
from the previous section to propose _subgoals_ for data collection. Note that
this is done in the exploration phase and have a latent goal model pre-trained
on the offline dataset. Given a subgoal, our algorithm (Alg. 1) proceeds by
executing actions towards the subgoal for a fixed number of timesteps (Alg. 1
L12). The data collected during subgoal navigation expands the topological
memory (Alg. 1 L14) and is used to fine-tune the model (Alg. 1 L15). Thus, the
task of efficient exploration is reduced to the task of choosing subgoals.

Algorithm 2 _RECON for Goal-Reaching:_ After exploration, RECON uses the
topological graph ${\mathcal{G}}$ to quickly navigate towards the goal
$o_{g}$.

1:procedure $\mathrm{GoalNavigate}$(${\mathcal{G}},o_{t},o_{g};H$)

2: $v_{t}\leftarrow\mathrm{AssociateToVertex}({\mathcal{G}},o_{t})$

3: $v_{g}\leftarrow\mathrm{AssociateToVertex}({\mathcal{G}},o_{g})$

4:
$(v_{t},\dots,v_{g})\leftarrow\mathrm{ShortestPath}({\mathcal{G}},v_{t},v_{g})$

5: for $v\in(v_{t},\dots,v_{g})$ do

6: $z\leftarrow p_{\phi}(z\mid o_{t},o_{g}=v\mathrm{.o})$

7: $\mathcal{D}_{w},o_{t}\leftarrow\mathrm{SubgoalNavigate}(z;H)$

Subgoals are represented by latent variables in our model, which may either
come from the posterior $p_{\phi}(z|o_{t},o_{g})$, or from the prior $r(z)$.
Given a subgoal $z$ and observation $o_{t}$, the model decodes it into an
action and distance pair $q(a_{t}^{g},d_{t}^{g}|z,o_{t})$; the action is used
to control the robot towards the goal, and the distance is used to construct
edges in the topological graph. The choice of intermediate subgoal to navigate
toward at any step is based on the robot’s estimate of the goal reachability
and its proximity to the frontier. To determine the frontier of the graph, we
track the number of times each node in the graph was selected as the
navigation goal; nodes with low counts are considered to be on the frontier.
In the following, we use $\bar{z}_{t}^{g}$ to denote the mean of the encoder
$p_{\phi}(z\mid o_{t},o_{g})$, and $\bar{d}_{t}^{g}$ to denote the distance
component of the mean of the decoder
$q_{\theta}(a_{t},d_{t}\mid\bar{z}_{t}^{g},o_{t})$ (i.e., the predicted number
of time steps from $o_{t}$ to $\bar{z}_{t}^{g}$). The choice of subgoal at
each step is made as follows:

(i) Feasible Goal: The robot believes it can reach the goal and adopts the
representation of the goal image as the subgoal (Alg. 1 L7). The robot’s
confidence in reaching the goal is based on the probability of the current
goal embedding $z_{t}^{g}$ under the prior $r(z)$. Large $r(z_{t}^{g})$
implies the relationship between the observation $o_{t}$ and the goal $o_{g}$
is _in-distribution_ , suggesting that the model’s estimates of the distances
is reliable – intuitively, this means that the model is confident about the
distance to $o_{g}$ and can reach it.

(ii) Explore at Frontier: The robot is at the “least-explored node” (frontier)
$o_{n}$ and explores by sampling a random conditional subgoal latent
$z_{t}^{w}$ from the prior (Alg. 1 L9). The robot determines whether it is at
the frontier based on the distance (estimated by querying the model) to its
“least explored neighbor” $\bar{d}_{t}^{n}$ – the node in the graph within a
distance threshold ($\delta_{2}$) of the current observation that has the
lowest visitation count. If the distance to this node $\bar{d}_{t}^{n}$ is low
(threshold $\delta_{1}$), then the robot is at the frontier.

(iii) Go to Frontier: The robot adopts its “least-explored neighbor” $o_{n}$
as a subgoal (Alg. 1 L11).

The $\mathrm{SubgoalNavigate}$ function rolls out the learned policy for a
fixed time horizon $H$ to navigate to the desired subgoal latent $z^{w}_{t}$,
by querying the decoder $q_{\theta}(a_{t},d_{t}|z_{t}^{w},o_{\tau})$ with a
fixed subgoal latent. The endpoint of such a rollout is used to update the
visitation counts in the graph ${\mathcal{G}}$. At the end of each trajectory,
the $\mathrm{ExpandGraph}$ subroutine is used to update the edge and node sets
$\\{{\mathcal{E}},{\mathcal{V}}\\}$ of the graph ${\mathcal{G}}$ to update the
representation of the environment. We provide the pseudocode for these
subroutines in Appendix B.1. We also share broader implementation details
including choice of hyperparameters, model architectures and training details
in Appendix B.2.

###  4.3 System Summary

RECON uses the latent goal model and topological graph to quickly explore new
environments and discovers user-specified goals. Our complete system consists
of three stages:

  1. A)

_Prior Experience:_ The goal-conditioned distance and action model (Sec. 4.1)
is trained using experience from previously visited environments. Supervision
for training our model is obtained by using time steps as a proxy for
distances and a relabeling scheme (Appendix A).

  2. B)

_Exploring a Novel Environment:_ When placed in a new environment, RECON uses
a combination of frontier-based exploration and latent goal-sampling with the
learned model. The learned model is also fine-tuned to this environment. These
steps are summarized in Alg. 1 and Sec. 4.2.

  3. C)

_Navigating an Explored Environment:_ Given an explored environment
(represented by a topological graph ${\mathcal{G}}$) and the model, RECON uses
${\mathcal{G}}$ to navigate to a goal image by planning a path of subgoals
through the graph. This process is summarized in Alg. 2.

Method | Expl. Time (mm:ss) $\downarrow$ | Nav. Time (mm:ss) $\downarrow$ | SCT [58] $\uparrow$
---|---|---|---
PPO + RND [18] | 21:18 | 00:47 | 0.22
InfoBot [41] | 23:36 | 00:48 | 0.21
Active Neural SLAM (ANS) [21] | 21:00 | 00:45 | 0.33
ViNG [11] | 19:48 | 00:34 | 0.60
Ours + Episodic Curiosity (ECR) [20] | 14:54 | 00:31 | 0.73
RECON (Ours) | 09:54 | 00:26 | 0.92
Table 1: Exploration and goal reaching performance: Exploring 8 real-world
environments, RECON reaches the goal 50% faster than the best baseline (ECR).
ANS takes up to 2x longer to find the goal and NTS [27] fails to find the goal
in every environment. On subsequent traversals, RECON navigates to the goal
20–85% faster than other baselines, and exhibits $>\\!30\%$ higher weighted
success.

##  5 Experimental Evaluation

We designed our experiments to answer four questions:

  1. Q1.

How does RECON compare to prior work for visual goal discovery in novel
environments?

  2. Q2.

After exploration, can RECON leverage its experience to navigate to the goal
efficiently?

  3. Q3.

What is the range of perturbations and non-stationary elements to which RECON
is robust?

  4. Q4.

How important are the various components of RECON, such as sampling from an
information bottleneck and non-parametric memory, to its performance?

###  5.1 Goal-Directed Exploration in Novel Environments

Figure 4: Visualizing goal-reaching behavior of the system: _(left)_ Example
trajectories to goals discovered by RECON in previously unseen environments.
_(right)_ Policies learned by the different methods in one such environment.
Only RECON and ECR reach the goal successfully, and RECON takes the shorter
route.

We perform our evaluation in a diverse variety of outdoor environments
(examples in Fig. 2), including parking lots, suburban housing, sidewalks, and
cafeterias. We train our self-supervised navigation model using an offline
navigation dataset (Sec.3.2) collected in a distinct set of training
environments, and evaluate our system’s ability to discover user-specified
goals in previously unseen environments. We compare RECON to five baselines,
each trained on the same 20 hours of offline data as our method, and finetuned
in the target environment with online interaction.

  1. 1.

PPO + RND: Random Network Distillation (RND) is a widely used prediction
bonus-based exploration strategy in RL [18], which we use with PPO [59, 60].
This comparison is representative of a frequently used approach for
exploration in RL using a novelty-based bonus.

  2. 2.

InfoBot: An offline variant of InfoBot [41], which uses goal-conditioned
information bottleneck, analogous to our method, but does not use the non-
parametric memory.

  3. 3.

Active Neural SLAM (ANS): A popular indoor navigation approach based on metric
spatial maps proposed for coverage-maximizing exploration [21]. We adapt it to
the goal-directed task by using the distance function from RECON to detect
when the goal is nearby.

  4. 4.

Visual Navigation with Goals (ViNG): A method that uses random action
sequences to explore and incrementally build a topological graph without
reasoning about visitation counts [11].

  5. 5.

Episodic Curiosity (ECR): A method that executes random action sequences at
the frontier of a topological graph for exploration [20]. We implement this as
an ablation of our method that samples random action sequences, rather than
rollouts to sampled goals (Alg. 1 Line 7).

We evaluate the ability of RECON to discover visually-indicated goals in $8$
_unseen_ environments and navigate to them repeatedly. For each trial, we
provide an RGB image of the desired target (one per environment) to the robot
and report the time taken by each method to _(i)_ discover the desired goal
(Q1), and _(ii)_ reliably navigate to the discovered goal a second time using
prior exploration (Q2). Additionally, we quantify navigation performance using
Success weighed by Completion Time (SCT), a success metric that takes into
account the agent’s dynamics [58]. We show quantitative results in Table 1,
and visualize sample trajectories of RECON and the baselines in Fig. 4.

Figure 5: Exploring non-stationary environments: The learned representation
and topological graph is robust to visual distractors, enabling reliable
navigation to the goal under novel obstacles _(c–e)_ and appearance changes
_(f–h)_.

RECON outperforms all the baselines, discovering goals that are up to 80m away
in under 20 minutes, including instances where no other baseline can reach the
goal successfully. RECON+ECR and ViNG discover the goal in only the easier
environments, and take up to 80% more time to discover the goal in those
environments. RND, InfoBot and ANS are able to discover goals that are up to
$25$m away but fails to discover more distant goals, likely because using
reinforcement learning for fine-tuning is data-inefficient. We exclude
reporting metrics on NTS, which fails to successfully explore any environment,
likely due to overfitting to the offline trajectories. Indeed, the simulation
experiments reported in each of these online algorithms require upwards of
$1$M timesteps to adapt to new environments [41, 21, 27]. We attribute RECON’s
success to the context-conditioned sampling strategy (described in Sec. 4.1),
which proposes goals that can accelerate the exploration of new environments.

We then study RECON’s ability to quickly reach goals after initial discovery.
Table 1 shows that RECON variants are able to quickly recall a feasible path
to the goal. These methods create a compact topological map from experience in
the target environment, allowing them to quickly reach previously-seen states.
The other baselines are unsuccessful at recalling previously seen goals for
all but the simplest environments. Fig. 4 shows an aerial view of the paths
recalled by various methods in one of the environments. Only the RECON
variants are successfully able to navigate to the checkerboard goal; all other
baselines result in collisions in the environment. Further, RECON discovers a
shorter path to the goal and takes $30\%$ less time to navigate to it than ECR
ablation.

###  5.2 Exploring Non-Stationary Environments

Outdoor environments exhibit non-stationarity due to dynamic obstacles, such
as automobiles and people, as well as changes in appearance due to seasons and
time of day. Successful exploration and navigation in such environments
requires learning a representation that is invariant to such distractors. This
capability is of central interest when using a non-parametric memory: for the
topological map to remain valid when such distractors are presented, we must
ensure the invariance of the learned representation to such factors (Q3).

To test the robustness of RECONto unseen obstacles and appearance changes, we
first had RECON explore in a new “junkyard” to learn to reach a goal image
containing a blue dumpster (Fig. 5-a). Then, without any more exploration, we
evaluated the learned goal-reaching policy when presented with _previously
unseen_ obstacles (trash cans, traffic cones, and a car) and and weather
conditions (sunny, overcast, and twilight). Fig. 5 shows trajectories taken by
the robot as it successfully navigates to the goal in scenarios with varying
obstacles and lighting conditions. These results suggest that the learned
representations are invariant to visual distractors that do not affect robot’s
decisions to reach a goal (e.g., changes in lighting conditions do not affect
the trajectory to goal, and hence, are discarded by the bottleneck).

###  5.3 Dissecting RECON

RECON explores by sampling goals from the prior distribution over state-goal
representations. To quantify the importance of this exploration strategy (Q4),
we deploy RECON to perform undirected exploration in a novel target
environment _without building a graph_ of the environment. We compare the
coverage of trajectories of the robot over $5$ minutes of exploration when:
_(a)_ it executes random action sequences [20], and _(b)_ it performs rollouts
towards sampled goals. We see that performing rollouts to sampled goals
results in $5$x faster exploration in novel environments (see Fig. 6).

We also evaluate several variants of RECON that ablate its goal sampling and
non-parametric memory on the end-to-end task of visual goal discovery in novel
environments:

  1. -

_Reactive:_ our method deployed _without_ the topological graph for memory.

  2. -

_Random Actions:_ a variant of our method that executes random action
sequences at the frontier rather than rollouts to sampled goals. This is
identical to the ECR baseline described in Sec. 5.1.

  3. -

_Vanilla Sampling:_ a variant of our method which learns a goal-conditioned
policy and distances _without_ an information bottleneck to obtain compressed
representations.

Figure 6: Exploration via sampling from our context-conditioned prior
(_right_) allows the robot to explore 5 times faster than using random
actions, e.g. in ECR [20] (_left_).

Method |  Expl. Time (mm:ss) $\downarrow$ |  Nav. Time (mm:ss) $\downarrow$ |  SCT [58] $\uparrow$
---|---|---|---
Reactive |  11:54 |  00:37.4 |  0.63
Random Actions |  14:54 |  00:31.4 |  0.73
Vanilla Sampling |  14:06 |  00:28.7 |  0.83
Ours |  09:56 |  00:25.8 |  0.92

Table 2: Ablation experiments confirm the importance of using an information
bottleneck and a non-parametric memory.

We deploy these variants in a subset of the unseen test environments and
summarize their performance in Table 2. These results corroborate the
observations in Fig. 6: learning a compressed goal representation is key to
the performance of RECON. “Vanilla Sampling”, despite sampling from a joint
prior, performs poorly and is unable to discover distant goals. We hypothesize
that our method is more robust because the information bottleneck helps learn
a representation that ignores task-irrelevant information. We also observe
that “Reactive” experiences a smaller degradation in exploration performance,
suggesting that goal-sampling can help with the exploration problem even
without the graph. However, we find a massive degradation in its ability to
recall previously discovered goals, suggesting that the memory is key to the
navigation performance of RECON.

##  6 Discussion

We proposed a system for efficiently learning goal-directed policies in new
open-world environments. The key idea behind our method is to use a learned
goal-conditioned distance model with a latent variable model representing
visual goals for rapid goal-directed exploration. The problem setup studied in
this paper, using past experience to accelerate learning in a new environment,
is reflective of real-world robotics scenarios: collecting new experience at
deployment time is costly, but experience from prior environments can provide
useful guidance to solve new tasks.

In future work, we aim to provide theoretical guarantees for when and where we
can expect stochastic policies and the information bottleneck to provide
efficient exploration. One limitation of the current method is that it does
not explicitly account for the value of information. Accounting for such
states can generate a better goal-reaching policy.

## Acknowledgments

This research was supported by ARL DCIST CRA W911NF-17-2-0181, DARPA Assured
Autonomy, and the Office of Naval Research. The authors would like to thank
Suraj Nair and Brian Ichter for useful discussions, and Gregory Kahn for
setting up the infrastructure used for autonomous collection of real-world
data.

## References

  * [1] B. Kuipers and Y.-T. Byun, “A robot exploration and mapping strategy based on a semantic hierarchy of spatial representations,” _Robotics and Autonomous Systems_ , 1991, special Issue Toward Learning Robots.
  * [2] F. Bourgault, A. A. Makarenko, S. B. Williams, B. Grocholsky, and H. F. Durrant-Whyte, “Information based adaptive robotic exploration,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , vol. 1, 2002, pp. 540–545 vol.1.
  * [3] T. Kollar and N. Roy, “Efficient optimization of information-theoretic exploration in slam,” in _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , 2008.
  * [4] W. Tabib, K. Goel, J. Yao, M. Dabhi, C. Boirum, and N. Michael, “Real-time information-theoretic exploration with gaussian mixture model maps.” in _Robotics: Science and Systems_ , 2019.
  * [5] B. Yamauchi, “A frontier-based approach for autonomous exploration,” in _IEEE International Symposium on Computational Intelligence in Robotics and Automation (CIRA)_ , 1997.
  * [6] B. Charrow, S. Liu, V. Kumar, and N. Michael, “Information-theoretic mapping using cauchy-schwarz quadratic mutual information,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2015.
  * [7] D. Holz, N. Basilico, F. Amigoni, and S. Behnke, “A comparative evaluation of exploration strategies and heuristics to improve them,” 2011.
  * [8] N. Savinov, A. Dosovitskiy, and V. Koltun, “Semi-Parametric Topological Memory for Navigation,” in _International Conference on Learning Representations_ , 2018.
  * [9] A. Faust, K. Oslund, O. Ramirez, A. Francis, L. Tapia, M. Fiser, and J. Davidson, “Prm-rl: Long-range robotic navigation tasks by combining reinforcement learning and sampling-based planning,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 5113–5120.
  * [10] B. Eysenbach, R. R. Salakhutdinov, and S. Levine, “Search on the Replay Buffer: Bridging Planning and RL,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.
  * [11] D. Shah, B. Eysenbach, G. Kahn, N. Rhinehart, and S. Levine, “ViNG: Learning Open-World Navigation with Visual Goals,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2021.
  * [12] X. Meng, N. Ratliff, Y. Xiang, and D. Fox, “Scaling Local Control to Large-Scale Topological Navigation,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2020.
  * [13] D. Ghosh, A. Gupta, J. Fu, A. Reddy, C. Devin, B. Eysenbach, and S. Levine, “Learning to reach goals without reinforcement learning,” _arXiv preprint arXiv:1912.06088_ , 2019.
  * [14] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet, “Learning latent plans from play,” in _Conference on Robot Learning_. PMLR, 2020, pp. 1113–1132.
  * [15] B. C. Stadie, S. Levine, and P. Abbeel, “Incentivizing exploration in reinforcement learning with deep predictive models,” _arXiv preprint arXiv:1507.00814_ , 2015.
  * [16] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, “Curiosity-driven exploration by self-supervised prediction,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 2778–2787.
  * [17] R. Bajcsy, Y. Aloimonos, and J. K. Tsotsos, “Revisiting active perception,” _Autonomous Robots_ , vol. 42, no. 2, pp. 177–196, 2018.
  * [18] Y. Burda, H. Edwards, A. Storkey, and O. Klimov, “Exploration by random network distillation,” _arXiv preprint arXiv:1810.12894_ , 2018.
  * [19] Y. Satsangi, S. Lim, S. Whiteson, F. Oliehoek, and M. White, “Maximizing information gain in partially observable environments via prediction reward,” _arXiv preprint arXiv:2005.04912_ , 2020.
  * [20] N. Savinov, A. Raichuk, R. Marinier, D. Vincent, M. Pollefeys, T. Lillicrap, and S. Gelly, “Episodic curiosity through reachability,” _International Conference on Learning Representations (ICLR_ , 2019.
  * [21] D. S. Chaplot, D. Gandhi, S. Gupta, A. Gupta, and R. Salakhutdinov, “Learning to Explore using Active Neural SLAM,” in _International Conference on Learning Representations (ICLR)_ , 2020.
  * [22] R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel, “Vime: Variational information maximizing exploration,” _arXiv preprint arXiv:1605.09674_ , 2016.
  * [23] A. Mirchev, B. Kayalibay, M. Soelch, P. van der Smagt, and J. Bayer, “Approximate bayesian inference in spatial environments,” _arXiv preprint arXiv:1805.07206_ , 2018.
  * [24] M. G. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos, “Unifying count-based exploration and intrinsic motivation,” _arXiv preprint arXiv:1606.01868_ , 2016.
  * [25] E. Hazan, S. Kakade, K. Singh, and A. Van Soest, “Provably efficient maximum entropy exploration,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 2681–2691.
  * [26] L. Lee, B. Eysenbach, E. Parisotto, E. Xing, S. Levine, and R. Salakhutdinov, “Efficient exploration via state marginal matching,” _arXiv preprint arXiv:1906.05274_ , 2019.
  * [27] D. Singh Chaplot, R. Salakhutdinov, A. Gupta, and S. Gupta, “Neural topological slam for visual navigation,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [28] D. S. Chaplot, H. Jiang, S. Gupta, and A. Gupta, “Semantic curiosity for active visual learning,” in _ECCV_ , 2020.
  * [29] Manolis Savva*, Abhishek Kadian*, Oleksandr Maksymets*, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra, “Habitat: A Platform for Embodied AI Research,” in _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019.
  * [30] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi, “AI2-THOR: An Interactive 3D Environment for Visual AI,” _ArXiv_ , vol. abs/1712.05474, 2019.
  * [31] Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel, “Rl2: Fast reinforcement learning via slow reinforcement learning,” _arXiv preprint arXiv:1611.02779_ , 2016.
  * [32] S. Sæmundsson, K. Hofmann, and M. P. Deisenroth, “Meta reinforcement learning with latent variable gaussian processes,” _arXiv preprint arXiv:1803.07551_ , 2018.
  * [33] A. Nagabandi, I. Clavera, S. Liu, R. S. Fearing, P. Abbeel, S. Levine, and C. Finn, “Learning to adapt in dynamic, real-world environments through meta-reinforcement learning,” _arXiv preprint arXiv:1803.11347_ , 2018.
  * [34] M. E. Taylor and P. Stone, “Cross-domain transfer for reinforcement learning,” in _Proceedings of the 24th international conference on Machine learning_ , 2007, pp. 879–886.
  * [35] A. Lazaric, “Transfer in reinforcement learning: a framework and a survey,” in _Reinforcement Learning_. Springer, 2012, pp. 143–173.
  * [36] E. Parisotto, J. L. Ba, and R. Salakhutdinov, “Actor-mimic: Deep multitask and transfer reinforcement learning,” _arXiv preprint arXiv:1511.06342_ , 2015\.
  * [37] A. Gupta, C. Devin, Y. Liu, P. Abbeel, and S. Levine, “Learning invariant feature spaces to transfer skills with reinforcement learning,” _arXiv preprint arXiv:1703.02949_ , 2017.
  * [38] S. Gamrian and Y. Goldberg, “Transfer learning for related reinforcement learning tasks via image-to-image translation,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 2063–2072.
  * [39] N. Tishby, F. C. Pereira, and W. Bialek, “The information bottleneck method,” _arXiv preprint physics/0004057_ , 2000.
  * [40] M. Igl, K. Ciosek, Y. Li, S. Tschiatschek, C. Zhang, S. Devlin, and K. Hofmann, “Generalization in reinforcement learning with selective noise injection and information bottleneck,” _arXiv preprint arXiv:1910.12911_ , 2019.
  * [41] A. Goyal, R. Islam, D. Strouse, Z. Ahmed, M. Botvinick, H. Larochelle, Y. Bengio, and S. Levine, “Infobot: Transfer and exploration via the information bottleneck,” _arXiv preprint arXiv:1901.10902_ , 2019.
  * [42] A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy, “Deep variational information bottleneck,” _arXiv preprint arXiv:1612.00410_ , 2016.
  * [43] L. P. Kaelbling, “Learning to achieve goals,” in _IJCAI_. Citeseer, 1993, pp. 1094–1099.
  * [44] T. Schaul, D. Horgan, K. Gregor, and D. Silver, “Universal Value Function Approximators,” in _International Conference on Machine Learning (ICML)_ , 2015.
  * [45] V. Pong, S. Gu, M. Dalal, and S. Levine, “Temporal difference models: Model-free deep rl for model-based control,” _arXiv preprint arXiv:1802.09081_ , 2018.
  * [46] B. Eysenbach, R. Salakhutdinov, and S. Levine, “C-learning: Learning to achieve goals via recursive classification,” _arXiv preprint arXiv:2011.08909_ , 2020.
  * [47] Y. Ding, C. Florensa, M. Phielipp, and P. Abbeel, “Goal-conditioned imitation learning,” _arXiv preprint arXiv:1906.05838_ , 2019.
  * [48] H. Sun, Z. Li, X. Liu, D. Lin, and B. Zhou, “Policy continuation with hindsight inverse dynamics,” _arXiv preprint arXiv:1910.14055_ , 2019.
  * [49] R. K. Srivastava, P. Shyam, F. Mutz, W. Jaśkowski, and J. Schmidhuber, “Training agents using upside-down reinforcement learning,” _arXiv preprint arXiv:1912.02877_ , 2019.
  * [50] N. Rhinehart, R. McAllister, and S. Levine, “Deep imitative models for flexible inference, planning, and control,” in _International Conference on Learning Representations_ , 2020.
  * [51] V. H. Pong, M. Dalal, S. Lin, A. Nair, S. Bahl, and S. Levine, “Skew-fit: State-covering self-supervised reinforcement learning,” _arXiv preprint arXiv:1903.03698_ , 2019.
  * [52] C. Colas, P. Fournier, M. Chetouani, O. Sigaud, and P.-Y. Oudeyer, “Curious: intrinsically motivated modular multi-goal reinforcement learning,” in _International conference on machine learning_. PMLR, 2019, pp. 1331–1340.
  * [53] L. Zhang, G. Yang, and B. C. Stadie, “World model as a graph: Learning latent landmarks for planning,” _arXiv preprint arXiv:2011.12491_ , 2020.
  * [54] S. Pitis, H. Chan, S. Zhao, B. Stadie, and J. Ba, “Maximum entropy gain exploration for long horizon multi-goal reinforcement learning,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 7750–7761.
  * [55] K. Liu, T. Kurutach, C. Tung, P. Abbeel, and A. Tamar, “Hallucinative topological memory for zero-shot visual planning,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 6259–6270.
  * [56] G. Kahn, P. Abbeel, and S. Levine, “BADGR: An Autonomous Self-Supervised Learning-Based Navigation System,” 2020.
  * [57] A. Achille and S. Soatto, “Emergence of invariance and disentanglement in deep representations,” _The Journal of Machine Learning Research_ , vol. 19, no. 1, pp. 1947–1980, 2018.
  * [58] N. Yokoyama, S. Ha, and D. Batra, “Success weighted by completion time: A dynamics-aware evaluation criteria for embodied navigation,” 2021.
  * [59] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms,” 2017.
  * [60] E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra, “DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames,” in _International Conference on Learning Representations (ICLR)_ , 2020.
  * [61] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications,” 2017.
  * [62] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” _arXiv preprint arXiv:1312.6114_ , 2013.

##  Appendix A Dataset

In this work, we emphasize that data collected from prior experience in
unrelated environments can be a rich source of supervision, even if the
interactions in the dataset are suboptimal. To demonstrate this, we curate a
dataset of over $5000$ self-supervised trajectories collected over $9$
distinct real-world environments. These trajectories capture the interaction
of the robot in diverse environments, including phenomena like collisions with
obstacles and walls, getting stuck in the mud or pits, or flipping due to
bumpy terrain. The dataset contains measurements from a wide range of sensors
including a pair of stereo RGB cameras, thermal camera, 2D LiDAR, GPS and IMU
to support offline evaluation using an alternative suite of sensors. While a
lot of these sensor measurements can be noisy and unreliable, we believe that
learning-based techniques coupled with multimodal sensor fusion can provide a
lot of benefits in the real-world. This dataset was collected over a span of
$18$ months, including parts collected by Kahn et al. [56] and Shah et al.
[11] for earlier research projects, and exhibits significant variation in
appearance due to seasonal and lighting changes.

This dataset is available for download at sites.google.com/view/recon-
robot/dataset, along with helper scripts to load and visualize the
trajectories.

(a) Junkyard

(b) Fire Station

(c) Warehouse

(d) Cafeteria

(e) Parking Lot 1

(f) Forest Cabin

(g) Farmlands

(h) Parking Lot 2

(i) Residential

Figure 7: We collect data in $9$ diverse environments. Example trajectories
are shown in cyan.

###  A.1 Self-Supervised Data Collection and Labeling

We design the data collection methodology to enable gathering large amounts of
diverse data with minimal human intervention. Due to the high cost of
gathering data with real-world robotic systems, we choose to use an off-policy
learning algorithm in order to be able to gather data using any control policy
and train on all of the gathered data. To ensure that the control policy
achieves sufficient coverage of the environment while also ensuring that the
action sequences executed by the robot are realistic, we use a time-correlated
random walk to gather data. A naïve uniform random control policy is
inadequate because the robot will primarily drive straight due to the linear
and angular velocity action interface of the robot, which will result in both
insufficient exploration and unrealistic test time action sequences.

During data collection using the random control policy, the robot requires a
mechanism to detect if it is in collision or stuck, and an automated
controller to reset itself in order to continue gathering data. We detect
collisions in one of two ways, either using the LIDAR to detect when an
obstacle is near or the IMU to detect when the robot is stuck due to an
obstacle or uneven terrain. We program an automated backup maneuver that
drives the robot out of collision (whenever possible) so it can initiate a new
trajectory.

We also use these collision detectors as a weak source of supervision by
generating _event labels_ for the collected trajectories, giving us a self-
supervised relabeling pipeline as proposed in BADGR [56]. We consider three
different events: collision, bumpiness, and position. A collision event is
detected the LIDAR measures an obstacle to be close or, in off-road
environments, when the IMU detects a sudden drop in linear acceleration (jerk)
and angular velocity magnitudes. A bumpiness event is calculated as occurring
when the angular velocity magnitudes measured by the IMU are above a certain
threshold. The position is determined by an onboard state estimator that fuses
wheel odometry and the IMU to form a local position estimate. Note that all
experiments reported in this paper only use the collision labels; these labels
are used to dissect the random walks into smooth trajectories that end in
collision.

Figure 8: Exploring and learning to reach goals: _(left)_ Amount of time
needed for each method to search for the goals in a new environment
($\downarrow$ is better; hashed out bars represent failure). _(right)_ Amount
of time needed to reach the goal a second time, after reaching the goal once
and constructing the map, in seconds ($\downarrow$ is better).

###  A.2 Environments

To learn general navigational affordances across a wide range of environments,
we curate over 40 hours of trajectories in 9 diverse open-world environments
of varying complexity (see Figure 7).

Figure 8 shows the exploration and navigation performance of RECON and the
baselines (see Sec. 5 for details) on the individual environments. As the
environment complexity increases, most methods are not able to explore the
environment efficiently to discover the goal. For videos of our system
exploring these environments, please check out the supplemental video
submission.

##  Appendix B Reproducibility

###  B.1 Algorithmic Components

The $\mathrm{SubgoalNavigate}$ function rolls out the learned policy for a
fixed time horizon $H$ to navigate to the desired subgoal latent $z^{w}_{t}$,
by querying the decoder $q_{\theta}(a_{t},d_{t}|z_{t}^{w},o_{\tau})$ in an
open loop manner. The endpoint of such a rollout is used to update the
visitation counts $v$ in the graph ${\mathcal{G}}$ using the
$\mathrm{AssociateToVertex}$ subroutine. To nudge the robot to the frontier,
we use a heuristic $\mathrm{LeastExploredNeighbor}$ routine that uses the
visitation counts of the neighbors to identify unexplored areas in the local
neighborhood. At the end of each trajectory, the $\mathrm{ExpandGraph}$
subroutine is used to update the edge and node sets
$\\{{\mathcal{E}},{\mathcal{V}}\\}$ of the graph ${\mathcal{G}}$ to update the
non-parametric representation of the environment. Pseudocode for these
subroutines are given in Alg. 3.

Algorithm 3 Pseudocode for subroutines referenced in the exploration algorithm
shown in Alg. 1

1:function $\mathrm{SubgoalNavigate}$($z_{t}^{w};H$)

2: $\mathrm{trajectory}~{}\leftarrow()$

3: for $t\in[1,\dots,H]$ do

4: $\mathrm{trajectory.append}((o_{t},a_{t},t))$

5: $a_{t},d_{t}^{g}\sim q_{\theta}(a_{t},d_{t}\mid z_{t}^{w},o_{\tau})$
$\triangleright$ _Sample action_

6: $o_{t}\leftarrow\mathrm{Env.step}(a_{t})$ $\triangleright$ _Execute action_

7: $v_{H}\leftarrow\mathrm{AssociateToVertex}({\mathcal{G}},o_{H})$

8: $v_{H}\mathrm{.count}\leftarrow v_{H}\mathrm{.count}+1$

9:
$\mathcal{D}_{w}\leftarrow((o_{t},o_{H},a_{t},H-t)~{}\mathrm{for}~{}(o_{t},a_{t},t)\in\mathrm{trajectory})$

10: return $\mathcal{D}_{w},o_{H}$

1:function
$\mathrm{AssociateToVertex}$(${\mathcal{G}}=({\mathcal{V}},{\mathcal{E}}),o_{t}$)

2:
$\mathbf{d}\leftarrow\mathrm{sort}((\bar{d}_{t}^{v},v)~{}\mathrm{for}~{}v\in{\mathcal{V}})$
$\triangleright$ _Predict distances_

3: $v,d\leftarrow\mathbf{d}$[0] $\triangleright$ _Associate $o_{t}$ with
nearest vertex_

4: return $v$

1:function
$\mathrm{LeastExploredNeighbor}$(${\mathcal{G}}=({\mathcal{V}},{\mathcal{E}}),o_{t},\delta_{2}$)

2: $v\leftarrow\mathrm{AssociateToVertex}({\mathcal{G}},o_{t})$

3:
${\mathcal{V}}_{n}\leftarrow\\{v^{\prime}:{\mathcal{E}}(v,v^{\prime})<\delta_{2},v^{\prime}\in{\mathcal{V}}\\}$
$\triangleright$ _Retrieve neighbors_

4:
$\mathbf{c}\leftarrow\mathrm{sort}((v^{\prime}.\mathrm{count},v^{\prime}.\mathrm{o})~{}\mathrm{for}~{}v^{\prime}\in{\mathcal{V}}_{n})$

5: $v_{c},o_{c}\leftarrow\mathbf{c}[0]$ $\triangleright$ _Retrieve neighbor
with smallest count_

6: return $o_{c}$

1:procedure
$\mathrm{ExpandGraph}$(${\mathcal{G}}=({\mathcal{V}},{\mathcal{E}}),o_{t}$)

2: $v_{t}\leftarrow\mathrm{Node}(\mathrm{count}=1,o=o_{t})$ $\triangleright$
_Create node for $o_{t}$_

3:
${\mathcal{E}}\leftarrow{\mathcal{E}}\cup\\{(v_{t},v_{g}):\bar{d}_{t}^{g},g\in{\mathcal{V}}\\}$
$\triangleright$ _Add edges_

4: ${\mathcal{V}}\leftarrow{\mathcal{V}}\cup\\{v_{t}\\}$ $\triangleright$ _Add
vertex_

###  B.2 Implementation Details

Hyperparam. | Value | Meaning
---|---|---
$\delta_{1}$ | $4$ | Threshold of identification
$\delta_{2}$ | $15$ | Threshold of neighbors
$\epsilon$ | $10^{-2}$ | Exploration threshold on prior
$\beta$ | $1.0$ | Model complexity
$\gamma$ | 10 | Epochs to finetune model
H |  $5$ seconds | Horizon to navigate to subgoal
Table 3: Hyperparameters used in our experiments.

Inputs to the encoder $p_{\phi}$ are pairs of observations of the environment
– current and goal – represented by a stack of two RGB images obtained from
the onboard camera at a resolution of $160\times 120$ pixels. $p_{\phi}$ is
implemented by a MobileNet encoder [61] followed by a fully-connected layer
projecting the $1024$-dimensional latents to a stochastic, context-conditioned
representation $z_{t}^{g}$ of the goal that uses $64$-dimensions each to
represent the mean and diagonal covariance of a Gaussian distribution. Inputs
to the decoder $q_{\theta}$ are the context (current observation) – processed
with another MobileNet – and $z_{t}^{g}$. We use the reparametrization trick
[62] to sample from the latent and use the concatenated encodings to learn the
optimal actions $a_{t}^{g}$ and distances $d_{t}^{g}$. Details of our network
architecture are provided in Table 4. During pretraining, we maximize Eq. 2
with a batch size of 128 and perform gradient updates using the Adam optimizer
with learning rate $\lambda=10^{-4}$ until convergence. We provide the
hyperparameters associated with our algorithms in Table 3.

Layer | Input [Dimensions] | Output [Dimensions] | Layer Details
---|---|---|---
_Encoder_ $p_{\phi}(z\mid o_{t},o_{g})=\mathcal{N}(\cdot;\mu_{p},\Sigma_{p})$
1 |  $o_{t}$, $o_{g}$ [3, 160, 120] |  $I_{t}^{g}$ [6, 160, 120] | Concatenate along channel dimensio.
2 |  $I_{t}^{g}$ [6, 160, 120] |  $E_{t}^{g}$ [1024] | MobileNet Encoder [61]
3 |  $E_{t}^{g}$ [1024] |  $\mu_{p}$ [64], $\sigma_{p}$ [64] | Fully-Connected Layer, $\mathrm{exp}$ activation of $\sigma_{p}$
4 |  $\sigma_{p}$ [64] |  $\Sigma_{p}$ [64, 64] | $\mathrm{torch.diag}(\sigma_{p})$
_Decoder_ $q_{\theta}(a,d\mid
o_{t},z_{t}^{g})=\mathcal{N}(\cdot;\mu_{q},\Sigma_{q})$
1 |  $o_{t}$ [3, 160, 120] |  $E_{t}$ [1024] | MobileNet Encoder [61]
2 |  $E_{t}$ [1024], $z_{t}^{g}$ [64] |  $F=E_{t}\oplus z_{t}^{g}$ [1088] | Concatenate image and goal representation
3 |  $F$ [1088] |  $\mu_{q}$ [3], $\sigma_{q}$ [3] | Fully-Connected Layer, $\mathrm{exp}$ activation of $\sigma_{q}$
4 |  $\sigma_{q}$ [3] |  $\Sigma_{q}$ [3, 3] | $\mathrm{torch.diag}(\sigma_{q})$
5 |  $\mu_{q}$ [3] | $\bar{a}_{t}^{g}[2],\bar{d}_{t}^{g}[1]$ | Split into actions and distances.
Table 4: Architectural Details of RECON: The inputs to the model are RGB
images $o_{t}\in[0,1]^{3\times 160\times 120}$ and $o_{g}\in[0,1]^{3\times
160\times 120}$, representing the current and goal image.

##  Appendix C Supplemental Video

For more results and videos of our system deployed in unstructured, outdoor
environments in the real-world, please check out the supplemental video
submission.