uses 21 object categories for the goal object. As these object categories are
not overlapping with the COCO categories, we use DeepLabv3 [40] model for
semantic segmentation. We predict the semantic segmentation and the semantic
map with all 40 categories in the MP3D dataset including the 21 goal
categories. We fine-tune the DeepLabv3 segmentation model by retraining the
final layer to predict semantic segmentation for all 40 categories. This
segmentation model is also trained with the map-based loss in addition to the
first-person segmentation loss. The performance of the top 5 entries to the
challenge are shown in Table 3. The proposed approach outperforms all the
other entries with a success rate of 25.3% as compared to 18.8% for the second
place entry.

Team Name |  | SPL | Success | Dist To Goal (m)
---|---|---|---|---
SemExp |  | 0.102 | 0.253 | 6.328
SRCB-robot-sudoer |  | 0.099 | 0.188 | 6.908
Active Exploration (Pre-explore) |  | 0.046 | 0.126 | 7.336
Black Sheep |  | 0.028 | 0.101 | 7.033
Blue Ox |  | 0.021 | 0.069 | 7.233
Table 3: Results on CVPR 2020 Habitat ObjectNav Challenge. Table showing the
performance of top 5 entries on the Test-Challenge dataset. Our submission
based on the SemExp model won the challenge. Figure 6: Real-world Transfer.
Figure showing an example trajectory of the SemExp model transferred to the
real-world for the object goal ‘potted plant’. Sample images seen by the robot
are shown on the top and the predicted semantic map is shown below. The long-
term goal selected by the Goal-driven Policy is shown in blue.

####  3 Real World Transfer

We used the Locobot hardware platform and PyRobot API [146] to deploy the
trained policy in the real world. In Figure 6 we show an episode of the robot
when it was provided ‘potted plant’ as the object goal. The long-term goals
sampled by the Goal-driven policy (shown by blue circles on the map) are often
towards spaces where there are high chances of finding a potted plant. This
indicates that it is learning to exploit the structure in the semantic map to
some extent. Out of 20 trials in the real-world, our method succeeded in 13
episodes leading to a success rate of 65%. End-to-end learning-based policies
failed consistently in the real-world due to the visual domain gap between
simulation environments and the real-world. Our model performs well in the
real-world as (1) it is able to leverage Mask RCNN which is trained on real-
world data and (2) the other learned modules (map denoising and the goal
policy) work on top-down maps which are domain-agnostic. Our trials in the
real-world also indicate that perfect pose and depth are not critical to the
success of our model as it can be successfully transferred to the real-world
where pose and depth are noisy.

###  5 Discussion

In this chapter, we presented a semantically-aware exploration model to tackle
the object goal navigation task in large realistic environments. The proposed
model makes two major improvements over prior methods, incorporating semantics
in explicit episodic memory and learning goal-oriented semantic exploration
policies. Our method achieves state-of-the-art performance on the object goal
navigation task and won the CVPR2020 Habitat ObjectNav challenge. Ablation
studies show that the proposed model learns semantic priors which lead to more
efficient goal-driven navigation. Domain-agnostic module design led to
successful transfer of our model to the real-world. We also analyze the error
modes for our model and quantify the scope for improvement along two important
dimensions (semantic mapping and goal-oriented exploration) in the future
work. The proposed model can also be extended to tackle a sequence of object
goals by utilizing the episodic map for more efficient navigation for
subsequent goals.

The semantic mapping learnt by the SemExp model can also be used for active
visual learning, where an embodied agent actively explores the environment to
collect observations to learn and improve perception models. In the next
chapter, we explore a self-supervised approach for training an exploration
policy for improving an object detection model.

##  Chapter 7 Semantic Curiosity

00footnotetext: Webpage:
https://devendrachaplot.github.io/projects/SemanticCuriosity

Imagine an agent whose goal is to learn how to detect and categorize objects.
How should the agent learn this task? In the case of humans (especially
babies), learning is quite interactive in nature. We have the knowledge of
what we know and what we don’t, and we use that knowledge to guide our future
experiences/supervision. Compare this to how current algorithms learn – we
create datasets of random images from the internet and label them, followed by
model learning. The model has no control over what data and what supervision
it gets – it is resigned to the static biased dataset of internet images. Why
does current learning look quite different from how humans learn? During the
early 2000s, as data-driven approaches started to gain acceptance, the
computer vision community struggled with comparisons and knowing which
approaches work and which don’t. As a consequence, the community introduced
several benchmarks from BSDS [131] to VOC [60]. However, a negative side
effect of these benchmarks was the use of static training and test datasets.
While the pioneering works in computer vision focused on active vision and
interactive learning, most of the work in the last two decades focuses on
static internet vision. But as things start to work on the model side, we
believe it is critical to look at the big picture again and return our focus
to an embodied and interactive learning setup.

Figure 1: Semantic Curiosity: We propose semantic curiosity to learn
exploration for training object detectors. Our semantically curious policy
attempts to take actions such that the object detector will produce
inconsistent outputs.

In an embodied interactive learning setup, an agent has to perform actions
such that observations generated from these actions can be useful for learning
to perform the semantic task. Several core research questions need to be
answered: (a) what is the policy of exploration that generates these
observations? (b) what should be labeled in these observations - one object,
one frame, or the whole trajectory? (c) and finally, how do we get these
labels? In this chapter, we focus on the first task – what should the
exploration policy be to generate observations which can be useful in training
an object detector? Instead of using labels, we focus on learning these
trajectories in an unsupervised/self-supervised manner. Once the policy has
been learned, we use the policy in novel (previously unseen) scenes to perform
actions. As observations are generated, we assume that an oracle will densely
label all the objects of interest in the trajectories.

So what are the characteristics of a good exploration policy for visual
learning, and how do we learn it? A good semantic exploration policy is one
which generates observations of objects and not free-space or the
wall/ceiling. But not only should the observations be objects, but a good
exploration policy should also observe many unique objects. Finally, a good
exploration policy will move to parts of the observation space where the
current object detection model fails or does not work. Given these
characteristics, how should we define a reward function that could be used to
learn this exploration policy? Note, as one of the primary requirements, we
assume the policy is learned in a self-supervised manner – that is, we do not
have ground-truth objects labeled which can help us figure out where the
detections work or fail.

Inspired by recent work in intrinsic motivation and curiosity for training
policies without external rewards [155, 157], we propose a new intrinsic
reward called semantic curiosity that can be used for the exploration and
training of semantic object detectors. In the standard curiosity reward, a
policy is rewarded if the predicted future observation does not match the true
future observation. The loss is generally formulated in the pixel-based
feature space. A corresponding reward function for semantic exploration would
be to compare semantic predictions with the current model and then confirm
with ground-truth labels – however, this requires external labels (and hence
is not self-supervised anymore). Instead, we formulate semantic curiosity
based on the meta-supervisory signal of consistency in semantic prediction –
that is, if our model truly understands the object, it should predict the same
label for the object even as we move around and change viewpoints. Therefore,
we exploit consistency in label prediction to reward our policies. Our
semantic curiosity rewards trajectories which lead to inconsistent labeling
behavior of the same object by the semantic classifier. Our experiments
indicate that training an exploration policy via semantic curiosity
generalizes to novel scenes and helps train an object detector which
outperforms baselines trained with other possible alternatives such as random
exploration, pixel curiosity, and free space/map curiosity. We also perform a
large set of experiments to understand the behavior of a policy trained with
semantic curiosity.

We study the problem of how to sample training data in embodied contexts. This
is related to active learning (picking what sample to label), active
perception (how to move around to gain more information), intrinsic motivation
(picking what parts of the environment to explore). Learning in embodied
contexts can also leverage spatio-temporal consistency. We survey these
research areas below.

Active Perception. Active perception [12] refers to the problem of actively
moving the sensors around at test time to improve performance on the task by
gaining more information about the environment. This has been instantiated in
the context of object detection [3], amodal object detection [215], scene
completion [99], and localization [33, 69]. We consider the problem in a
different setting and study how to efficiently move around to best learn a
model. Furthermore, our approach to learn this movement policy is self-
supervised and does not rely on end-task rewards, which were used in [99, 33,
215].

Active Learning. Our problem is more related to that of active learning [175],
where an agent actively acquires labels for unlabeled data to improve its
model at the fastest rate [173, 73, 217]. This has been used in a number of
applications such as medical image analysis [119], training object detectors
[198, 216], video segmentation [66], and visual question answering [140]. Most
works tackle the setting in which the unlabeled data has already been
collected. In contrast, we study learning a policy for efficiently acquiring
effective unlabeled data, which is complementary to such active learning
efforts.

Intrinsic Rewards. Our work is also related to work on exploration in
reinforcement learning [170, 184, 8, 98]. The goal of these works is to
effectively explore a Markov Decision Process to find high reward paths. A
number of works formulate this as a problem of maximizing an intrinsic reward
function which is designed to incentivize the agent to seek previously unseen
[61] or poorly understood [155] parts of the environment. This is similar to
our work, as we also seek poorly understood parts of the environment. However,
we measure this understanding via multi-view consistency in semantics. This is
in a departure from existing works that measure it in 2D image space [155], or
consistency among multiple models [157]. Furthermore, our focus is not
effective exhaustive exploration, but exploration for the purpose of improving
semantic models.

Spatio-Temporal smoothing at test time. A number of papers use spatio-temporal
consistency at test time for better and more consistent predictions [28, 72].
Much like the distinction from active perception, our focus is using it to
generate better data at train time.

Spatio-temporal consistency as training signal. Labels have been propagated in
videos to simplify annotations [199], improve prediction performance given
limited data [10, 18], as well as collect images [42]. This line of work
leverages spatio-temporal consistency to propagate labels for more efficient
labeling. Researchers have also used multi-view consistency to learn about 3D
shape from weak supervision [196]. We instead leverage spatio-temporal
consistency as a cue to identify what the model does not know. [177] is more
directly related, but we tackle the problem in an embodied context and study
how to navigate to gather the data, rather than analyzing passive datasets for
what labels to acquire.

Visual Navigation and Exploration. Prior work on visual navigation can broadly
be categorized into two classes based on whether the location of the goal is
known or unknown. Navigation scenarios, where the location of the goal is
known, include the most common pointgoal task where the coordinate to the goal
is given [81, 167]. Another example of a task in this category is vision and
language navigation [5] where the path to the goal is described in natural
language. Tasks in this category do not require exhaustive exploration of the
environment as the location of the goal is known explicitly (coordinates) or
implicitly (path).

Navigation scenarios, where the location of the goal is not known, include a
wide variety of tasks. These include navigating to a fixed set of objects
[120, 58, 209, 31, 137, 81], navigating to an object specified by language
[89, 34] or by an image [222, 37], and navigating to a set of objects in order
to answer a question [52, 78]. Tasks in this second category essentially
involve efficiently and exhaustively exploring the environment to search the
desired object. However, most of the above approaches overlook the exploration
problem by spawning the target a few steps away from the goal and instead
focus on other challenges. For example, models for playing FPS games [120, 58,
209, 31] show that end-to-end RL policies are effective at reactive navigation
and short-term planning such as avoiding obstacles and picking positive reward
objects as they randomly appear in the environment. Other works show that
learned policies are effective at tackling challenges such as perception (in
recognizing the visual goal) [222, 37], grounding (in understanding the goal
described by language) [89, 34] or reasoning (about visual properties of the
target object) [52, 78]. While end-to-end reinforcement learning is shown to
be effective in addressing these challenges, they are ineffective at
exhaustive exploration and long-term planning in a large environment as the
exploration search space increases exponentially as the distance to the goal
increases.

Some very recent works explicitly tackle the problem of exploration by
training end-to-end RL policies maximizing the explored area [35, 41, 63]. The
difference between these approaches and our method is twofold: first, we train
semantically-aware exploration policies as compared spatial coverage
maximization in some prior works [35, 41], and second, we train our policy in
an unsupervised fashion, without requiring any ground truth labels from the
simulator as compared to prior works trained using rewards based on ground-
truth labels [63].

###  1 Overview

Our goal is to learn an exploration policy such that if we use this policy to
generate trajectories in a novel scene (and hence observations) and train the
object detector from the trajectory data, it would provide a robust, high-
performance detector. In literature, most approaches use on-policy
exploration; that is, they use the external reward to train the policy itself.
However, training an action policy to sample training data for object
recognition requires labeling objects. Specifically, these approaches would
use the current detector to predict objects and compare them to the ground-
truth; they reward the policy if the predictions do not match the ground-truth
(the policy is being rewarded to explore regions where the current object
detection model fails). However, training such a policy via semantic
supervision and external rewards would have a huge bottleneck of supervision.
Given that our RL policies require millions of samples (in our case, we train
using 10M samples), using ground-truth supervision is clearly not the way.
What we need is an intrinsic motivation reward that can help train a policy
which can help sample training data for object detectors.

We propose a semantic curiosity formulation. Our work is inspired by a
plethora of efforts in active learning [175] and recent work on intrinsic
reward using disagreement [157]. The core idea is simple – a good object
detector has not only high mAP performance but is also consistent in
predictions. That is, the detector should predict the same label for different
views of the same object. We use this meta-signal of consistency to train our
action policy by rewarding trajectories that expose inconsistencies in an
object detector. We measure inconsistencies by measuring temporal entropy of
prediction – that is, if an object is labeled with different classes as the
viewpoint changes, it will have high temporal entropy. The trajectories with
high temporal entropy are then labeled via an oracle and used as the data to
retrain the object detector (See Figure 2).

Figure 2: Embodied Active Visual Learning: We use semantic curiosity to learn
an exploration policy on $\mathcal{E}_{U}$ scenes. The exploration policy is
learned by projecting segmentation masks on the top-down view to create
semantic maps. The entropy of semantic map defines the inconsistency of the
object detection module. The learned exploration policy is then used to
generate training data for the object detection/segmentation module. The
labeled data is then used to finetune and evaluate the object
detection/segmentation.

###  2 Methodology

Consider an agent $\mathcal{A}$ which can perform actions in environments
$\mathcal{E}$. The agent has an exploration policy $a_{t}=\pi(x_{t},\theta)$
that predicts the action that the agent must take for current observation
$x_{t}$. $\theta$ represents the parameters of the policy that have to be
learned. The agent also has an object detection model $\mathcal{O}$ which
takes as input an image (the current observation) and generates a set of
bounding boxes along with their categories and confidence scores.

The goal is to learn an exploration policy $\pi$, which is used to sample $N$
trajectories $\tau_{1}...\tau_{N}$ in a set of novel environments (and get
them semantically labeled). When used to train an object detector, this
labeled data would yield a high-performance object detector. In our setup, we
divide the environments into three non-overlapping sets
$(\mathcal{E}_{U},\mathcal{E}_{tr},\mathcal{E}_{t})$ – the first set is the
set of environments where the agent will learn the exploration policy $\pi$,
the second set is the object detection training environments where we use
$\pi$ to sample trajectories and label them, and the third set is the test
environment where we sample random images and test the performance of the
object detector on those images.

Figure 3: Semantic Mapping. The Semantic Mapping module takes in a sequence of
RGB ($I_{t}$) and Depth ($D_{t}$) images and produces a top-down Semantic Map.

####  1 Semantic Curiosity Policy

We define semantic curiosity as the temporal inconsistency in object detection
and segmentation predictions from the current model. We use a Mask RCNN to
obtain the object predictions. In order to associate the predictions across
frames in a trajectory, we use a semantic mapping module as described below.

Semantic Mapping. The Semantic Mapping module takes in a sequence of RGB
($I_{t}$) and Depth ($D_{t}$) images and produces a top-down semantic map
($M^{Sem}_{t}$) represented by a 3-dimensional tensor $C\times M\times M$,
where $M$ is the length of the square top-down map, and $C$ is the number of
semantic categories. Each element $(c,i,j)$ in this semantic map is $1$ if the
Mask RCNN predicted the object category $c$ at coordinates $(i,j)$ on the map
in any frame during the whole trajectory and $0$ otherwise. Figure 3 shows how
the semantic map is generated for a single frame. The input RGB frame
($I_{t}$) is passed through a current Mask RCNN to obtain object segmentation
predictions, while the Depth frame is used to calculate the point cloud. Each
point in the point cloud is associated with its semantic labels based on Mask
RCNN predictions. Note that these are not ground-truth labels, as each pixel
is assigned the category of the highest-confidence Mask RCNN segmentation
prediction on the corresponding pixel. The point cloud with the associated
semantic labels is projected to a 3-dimensional voxel map using geometric
computations. The voxel representation is converted to a top-down map by max-
pooling the values across the height. The resulting 2D map is converted to a
3-dimensional Semantic Map, such that each channel represents an object
category.

The above gives a first-person egocentric projection of the semantic map at
each time-step. The egocentric projections at each time step are used to
compute a geocentric map over time using a spatial transformation technique
similar to Chaplot et al. [35]. The egocentric projections are converted to
the geocentric projections by doing a spatial transformation based on the
agent pose. The semantic map at each time step is computed by pooling the
semantic map at the previous timestep with the current geocentric prediction.
Please refer to [35] for more details on these transformations.

Semantic Curiosity Reward. The semantic map allows us to associate the object
predictions across different frames as the agent is moving. We define the
semantic curiosity reward based on the temporal inconsistency of the object
predictions. If an object is predicted to have different labels across
different frames, multiple channels in the semantic map at the coordinates
corresponding to the object will have $1$s. Such inconsistencies are
beneficial for visual learning in downstream tasks, and hence, favorable for
the semantic curiosity policy. Thus, we define the cumulative semantic
curiosity reward to be proportional to the sum of all the elements in the
semantic map. Consequently, the semantic curiosity reward per step is just the
increase in the sum of all elements in the semantic map as compared to the
previous time step:

$r_{SC}=\lambda_{SC}\Sigma_{(c,i,j)\in(C,M,M)}(M^{Sem}_{t}[c,i,j]-M^{Sem}_{t-1}[c,i,j])$

where $\lambda_{SC}$ is the semantic curiosity reward coefficient. Summation
over the channels encourages exploring frames with temporally inconsistent
predictions. Summation across the coordinates encourages exploring as many
objects with temporally inconsistent predictions as possible.

The proposed Semantic Curiosity Policy is trained using reinforcement learning
to maximize the cumulative semantic curiosity reward. Note that although the
depth image and agent pose are used to compute the semantic reward, we train
the policy only on RGB images.

###  3 Experimental Setup

We use the Habitat simulator [168] with three different datasets for our
experiments: Gibson [211], Matterport [29] and Replica [183]. While the RGB
images used in our experiments are visually realistic as they are based on
real-world reconstructions, we note that the agent pose and depth images in
the simulator are noise-free unlike the real-world. Prior work has shown that
both depth and agent pose can be estimated effectively from RGB images under
noisy odometry [35]. In this chapter, we assume access to perfect agent pose
and depth images, as these challenges are orthogonal to the focus of this
chapter. Furthermore, these assumptions are only required in the unsupervised
pre-training phase for calculating the semantic curiosity reward and not at
inference time when our trained semantic-curiosity policy (based only on RGB
images) is used to gather exploration trajectories for training the object
detector.

In a perfectly interactive learning setup, the current model’s uncertainty
will be used to sample a trajectory in a new scene, followed by labeling and
updating the learned visual model (Mask-RCNN). However, due to the complexity
of this online training mechanism, we show results on batch training. We use a
pre-trained COCO Mask-RCNN as an initial model and train the exploration
policy on that model. Once the exploration policy is trained, we collect
trajectories in the training environments and then obtain the labels on these
trajectories. The labeled examples are then used to fine-tune the Mask-RCNN
detector.

####  1 Implementation details

Exploration Policy: We train our semantic curiosity policy on the Gibson
dataset and test it on the Matterport and Replica datasets. We train the
policy on the set of 72 training scenes in the Gibson dataset specified by
Savva et al. [168]. Our policy is trained with reinforcement learning using
Proximal Policy Optimization [172]. The policy architecture consists of
convolutional layers of a pre-trained ResNet18 visual encoder, followed by two
fully connected layers and a GRU layer leading to action distribution as well
as value prediction. We use 72 parallel threads (one for each scene) with a
time horizon on 100 steps and 36 mini batches per PPO epoch. The curiosity
reward coefficient is set to $\lambda_{SC}=2.5\times 10^{-3}$. We use an
entropy coefficient of 0.001, the value loss coefficient of 0.5. We use Adam
optimizer with a learning rate of $1\times 10^{-5}$. The maximum episode
length during training is 500 steps.

Fine-tuned Object Detector: We consider 5 classes of objects, chosen because
they overlap with the COCO dataset [128] and correspond to objects commonly
seen in an indoor scene: chair, bed, toilet, couch, and potted plant. To
start, we pre-train a Faster-RCNN model [160] with FPN [129] using ResNet-50
as the backbone on the COCO dataset labeled with these 5 overlapping
categories. We then fine-tuned our models on the trajectories collected by the
exploration policies for 90000 iterations using a batch size of 12 and a
learning rate of 0.001, with annealing by a factor of 0.1 at iterations 60000
and 80000. We use the Detectron2 codebase [210] and set all other
hyperparameters to their defaults in this codebase. We compute the AP50 score
(i.e., average precision using an IoU threshold of 50) on the validation set
every 5000 iterations.

Figure 4: Temporal Inconsistency Examples. Figure showing example trajectories
sampled from the semantic curiosity exploration policy. We highlight the
segmentation/detection inconsistencies of Mask RCNN. By obtaining labels for
these images, the Mask RCNN pipeline improves the detection performance
significantly.

####  2 Baselines

We use a range of baselines to gather exploration trajectories and compare
them to the proposed Semantic Curiosity policy:

  * •

Random. A baseline sampling actions randomly.

  * •

Prediction Error Curiosity. This baseline is based on Pathak et al. [155],
which trains an RL policy to maximize error in a forward prediction model.

  * •

Object Exploration. Object Exploration is a naive baseline where an RL policy
is trained to maximize the number of pre-trained Mask R-CNN detections. The
limitation of simply maximizing the number of detections is that the policy
can learn to look at frames with more objects but might not learn to look at
different objects across frames or objects with low confidence.

  * •

Coverage Exploration. This baseline is based on Chen et al. [41], where an RL
policy is trained to maximize the total explored area.

  * •

Active Neural SLAM. This baseline is based on Chaplot et al. [35] and uses a
modular and hierarchical system to maximize the total explored area.

After training the proposed policy and the baselines in the Gibson domain, we
use them directly (without fine-tuning) in the Matterport and Replica domains.
We sample trajectories using each exploration policy, using the images and
ground-truth labels to train an object detection model.

Method Name |  |  | Semantic Curiosity
---
Reward
| Explored
---
Area
| Number of Object
---
Detections
Random |  | 1.631 | 4.794 | 82.83
Curiosity |  | 2.891 | 6.781 | 112.24
Object exploration reward |  | 2.168 | 6.082 | 382.27
Coverage Exploration |  | 3.287 | 10.025 | 203.73
Active Neural SLAM |  | 3.589 | 11.527 | 231.86
Semantic Curiosity |  | 4.378 | 9.726 | 291.78
Table 1: Analysis. Comparing the proposed Semantic Curiosity policy with the baselines along different exploration metrics. Method Name | Chair | Bed | Toilet | Couch | Potted Plant | Average
---|---|---|---|---|---|---
Random | 46.7 | 28.2 | 46.9 | 60.3 | 39.1 | 44.24
Curiosity | 49.4 | 18.3 | 1.8 | 67.7 | 49.0 | 37.42
Object Exploration | 54.3 | 24.8 | 5.7 | 76.6 | 49.6 | 42.2
Coverage Exploration | 48.5 | 23.1 | 69.2 | 66.3 | 48.0 | 51.02
Active Neural SLAM | 51.3 | 20.5 | 49.4 | 59.7 | 45.6 | 45.3
Semantic Curiosity | 51.6 | 14.6 | 14.2 | 65.2 | 50.4 | 39.2
Table 2: Quality of object detection on training trajectories. We also analyze
the training trajectories in terms of how well the pre-trained object
detection model works on the trajectories. We want the exploration policy to
sample hard data where the pre-trained object detector fails. Data on which
the pre-trained model already works well would not be useful for fine-tuning.
Thus, lower performance is better.

###  4 Analyzing Learned Exploration Behavior

Before we measure the quality of the learned exploration policy for the task
of detection/segmentation, we first want to analyze the behavior of the
learned policy. This will help characterize the quality of data that is
gathered by the exploration policy. We will compare the learned exploration
policy against the baselines described above. For all the experiments below,
we trained our policy on Gibson scenes and collected statistics in 11 Replica
scenes.

Figure 4 shows some examples of temporal inconsistencies in trajectories
sampled using the semantic curiosity exploration policy. The pre-trained Mask-
RCNN detections are also shown in the observation images. Semantic curiosity
prefers trajectories with inconsistent detections. In the top row, the chair
and couch detector fire on the same object. In the middle row, the chair is
misclassified as a toilet and there is inconsistent labeling in the last
trajectory. The bed is misclassified as a couch. By selecting these
trajectories and obtaining their labels from an oracle, our approach learns to
improve the object detection module.

Table 1 shows the behavior of all of the policies on three different metrics.
The first metric is the semantic curiosity reward itself which measures
uncertain detections in the trajectory data. Since our policy is trained for
this reward, it gets the highest score on the sampled trajectories. The second
metric is the amount of explored area. Both [41] and [35] optimize this metric
and hence perform the best (they cover a lot of area but most of these areas
will either not have objects or not enough contradictory overlapping
detections). The third metric is the number of objects in the trajectories.
The object exploration baseline optimizes for this reward and hence performs
the best but it does so without exploring diverse areas or uncertain
detections/segmentations. If we look at the three metrics together it is clear
that our policy has the right tradeoff – it explores a lot of area but still
focuses on areas where objects can be detected. Not only does it find a large
number of object detections, but our policy also prefers inconsistent object
detections and segmentations. In Figure 5, we show some examples of
trajectories seen by the semantic curiosity exploration along with the
semantic map. It shows examples of the same object having different object
predictions from different viewpoints and also the representation in the
semantic map. In Figure 6, we show a qualitative comparison of maps and
objects explored by the proposed model and all the baselines. Example
trajectories in this figure indicate that the semantic curiosity policy
explores more unique objects with higher temporal inconsistencies.

Figure 5: Example trajectories. Figure showing example trajectories sampled
from the semantic curiosity exploration policy. In each episode the top row
shows the first-person images seen by the agent and the pre-trained Mask R-CNN
predictions. The bottom rows show a visualization of the semantic map where
colors denote different object categories. Different colors for the same
object indicate that the same object is predicted to have different categories
from different view points.

Next, we analyze the trajectories created by different exploration policies
during the object detection training stage. Specifically, we want to analyze
the kind of data that is sampled by these trajectories. How is the performance
of a pre-trained detector on this data? If the pre-trained detector already
works well on the sampled trajectories, we would not see much improvement in
performance by fine-tuning with this data. In Table 2, we show the results of
this analysis for these trajectories. As the results indicate, the mAP50 score
is low for the data obtained by the semantic curiosity policy.111Note that
curiosity-based policy has the lowest mAP because of outlier toilet category.
As the pre-trained object detector fails more on the data sampled by semantic
curiosity, labeling this data would intuitively improve the detection
performance.

Figure 6: Qualitative Comparison. Figure showing map and objects explored by the proposed Semantic Curiosity policy and the baselines in 3 example episodes. Semantic Curiosity Policy explores more unique objects with higher temporal inconsistency (denoted by different colors for the same object). Method Name | Chair | Bed | Toilet | Couch | Potted Plant | Average
---|---|---|---|---|---|---
PreTrained | 41.8 | 17.3 | 34.9 | 41.6 | 23.0 | 31.72
Random | 51.7 | 17.2 | 43.0 | 45.1 | 30.0 | 37.4
Curiosity | 48.4 | 18.5 | 42.3 | 44.3 | 32.8 | 37.26
Object Exploration | 50.3 | 16.4 | 40.0 | 39.7 | 29.9 | 35.26
Coverage Exploration | 50.0 | 19.1 | 38.1 | 42.1 | 33.5 | 36.56
Active Neural SLAM | 53.1 | 19.5 | 42.0 | 44.5 | 33.4 | 38.5
Semantic Curiosity | 52.3 | 22.6 | 42.9 | 45.7 | 36.3 | 39.96
Table 3: Object Detection Results. Object detection results in the Matterport
domain using the proposed Semantic Curiosity policy and the baselines. We
report AP50 scores on randomly sampled images in the test scenes. Training on
data gathered from the semantic curiosity trajectories results in improved
object detection scores.

###  5 Actively Learned Object Detection

Finally, we evaluate the performance of our semantic curiosity policy for the
task of object detection. The semantic curiosity exploration policy is trained
on 72 Gibson scenes. The exploration policy is then used to sample data on 50
Matterport scenes. Finally, the learned object detector is tested on 11
Matterport scenes. For each training scene, we sample 5 trajectories of 300
timesteps leading to 75,000 total training images with ground-truth labels.
For test scenes, we randomly sample images from test scenes.

In Table 3, we report the top AP50 scores for each method. Our results
demonstrate that the proposed semantic curiosity policy obtains higher quality
data for performing object detection tasks over alternative methods of
exploration. First, we outperform the policy that tries to see maximum
coverage area (and hence the most novel images). Second, our approach also
outperforms the policy that detects a lot of objects. Finally, apart from
outperforming the random policy, visual curiosity [155], and coverage; we also
outperform the highly-tuned approach of [35]. The underlying algorithm is
tuned on this data and was the winner of the RGB and RGBD challenge in
Habitat.

###  6 Discussion

We argue that we should go from detection/segmentation driven by static
datasets to a more embodied active learning setting. In this setting, an agent
can move in the scene and create its own datapoints. An oracle labels these
datapoints and helps the agent learn a better semantic object detector. This
setting is closer to how humans learn to detect and recognize objects. In this
chapter, we focus on the exploration policy for sampling images to be labeled.
We ask a basic question – how should an agent explore to learn how to detect
objects? Should the agent try to cover as many scenes as possible in the hopes
of seeing more diverse examples, or should the agent focus on observing as
many objects as possible?

We propose semantic curiosity as a reward to train the exploration policy.
Semantic curiosity encourages trajectories which will lead to inconsistent
detection behavior from an object detector. Our experiments indicate that
exploration driven by semantic curiosity shows all of the good characteristics
of an exploration policy: uncertain/high entropy detections, attention to
objects rather than the entire scene and also high coverage for diverse
training data. We also show that an object detector trained on trajectories
from a semantic curiosity policy leads to the best performance compared to a
plethora of baselines. For future work, this chapter is just the first step in
embodied active visual learning. It assumes perfect odometry, localization and
zero trajectory labeling costs. It also assumes that the trajectories will be
labeled – a topic of interest would be to sample trajectories with which
minimal labels can learn the best detector. Finally, the current approach is
demonstrated in simulators - it will be interesting to see whether the
performance can transfer to real-world robots.

##  Chapter 8 Conclusion

In this thesis, we presented several methods which make progress towards
building intelligent autonomous navigation agents. In chapters 1 and 2, we
presented end-to-end reinforcement learning methods to tackle short-term
navigation tasks involving both spatial and semantic understanding. We trained
navigation policies in game-like environments which learn fine-grained motor
from only raw-pixels and are able to tackle challenges such as obstacle
avoidance, semantic perception, language grounding, and reasoning. They are
able to learn only from rewards and generalize to unseen environments and
unseen language instructions.

While the above method and many other embodied navigational agents trained
with end-to-end reinforcement learning are shown to perform well on reactive
tasks, where the optimal behavior depends only on the last few observations,
they struggle in larger environments requiring long-term memory and planning.
In the second part of this thesis, we designed a new class of navigation
methods based on modular learning and structured explicit map representations,
which leverage the strengths of both classical and end-to-end learning
methods. We presented different modular learning models in chapters 3 to 7,
tackling different challenges involved in long-term spatial and semantic
navigation tasks. In this chapter, we summarize different methods presented in
this thesis along with their strengths and shortcomings and discuss future
directions to address these shortcomings.

Figure 1: Modular Learning-based Navigation Methods. Figure showing the
similarities in the design of Active Neural SLAM, Semantic Exploration and
Neural Topological SLAM models presented in this thesis.

###  1 Summary

The modular navigation methods presented in this thesis include Active Neural
SLAM (Ch. 4) for the task of exploration and Point-Goal Navigation, Neural
Topological SLAM (Ch. 5) for Image-Goal Navigation and Semantic Exploration
(Ch. 6) for Object-Goal Navigation. The design of all these models is very
similar as shown in Figure 1, which consists of three modules: the first
module is responsible for perception, localization, mapping, and building an
episodic memory, the second module is responsible for long-term waypoint
selection based on the task and the third module is responsible for low-level
control and obstacle avoidance.

We compared each of the above modular learning methods to both end-to-end
learning as well classical navigation methods and observed consistent trends.
The end-to-end learning-based models performed well at short-term spatial as
well as semantic understanding tasks, but as we increased the size of the
environment, and increased the distance to the goal location, end-to-end
learning methods consistently led to poor performance for all different
navigation tasks and goal specifications. This is because implicitly learning
about mapping and planning from just rewards is difficult and exploration
sample complexity increases exponentially with increase in the distance to the
goal.

The classical navigation approaches outperformed end-to-end learning-based
approaches at some long-term spatial navigation tasks given access to a depth
sensor. The use of explicit maps allows classical methods to be very effective
at spatial or geometric understanding of the scene. Explicit maps are also an
effective form of long-term memory and can be used for planning to distant
goals easily. However, they lack semantic understanding and can not tackle
semantic navigation tasks as they build only geometric maps and use a rule-
based policy. Also, their scalability is limited as the long-term policy or
goal locations need to be manually coded, they typically require pre-computed
maps, and they often rely on specialized sensors.

Figure 2: Advantages of Modular Learning. Figure summarizing the proerties,
advantages and disadvantages of navigation systems based on classical, end-to-
end learning and modular learning approaches. Modular Learniing combines the
benefits of the other two classes of methods.

Our modular learning methods leverage the strengths of both classical and end-
to-end learning methods, using two keys properties: first, they modularize the
navigation system but use learning to train each module, and second, they use
explicit structured map representations. Explicit maps are an effective form
of episodic memory which allow keeping a track of explored and unexplored
areas. They also allow us to use analytical planning algorithms like classical
approaches which is much more effective than learning to plan implicitly just
from rewards. At the same time, the use of learning within each module allows
us to learn statistical regularities and semantic priors present in the data
and generalize to unseen environments and goals. We saw that modular learning
methods consistently outperformed both classical as well as end-to-end
learning methods for various long-term navigation tasks such as exploration,
image-goal navigation, and object-goal navigation. Figure 2 summarizes the
advantages of modular learning approaches over both classical methods as well
as end-to-end learning.

Figure 3: Real-World Navigation. Figure showing an example of the Goal-
Oriented Semantic Exploration model (Chapter 6) deployed on a mobile robot and
used for Object Goal navigation.

Another benefit of our modular learning methods is effective real-world
transfer. While end-to-end learning methods typically suffer from the physical
and visual domain gap between simulation and the real-world, we were able to
successfully transfer the Active Neural SLAM and Semantic Exploration models
to a mobile robot platform in the real-world and achieve similar performance
at both exploration and object-goal navigation (Figure 3 shows an example).
The effective real-world transfer was a result of two key approaches:

  * •

Modularity: Due to the modularity, the long-term policy in both Active Neural
SLAM and Semantic Exploration models just takes the top-down map as input and
does not require first-person observations. The top-down map space is domain-
invariant which allows for easy transfer of long-term policies from simulation
to the real-world. Furthermore, modularity also allows us to leverage the
MaskRCNN semantic segmentation model which is pretrained on real-world data.
This allows us to bridge the visual domain gap between simulation and the
real-world.

  * •

Realistic noise models: In chapter 4, we built actuation and motion noise
models using data collected in the real-world. We implemented these noise
models in simulation and used them for training Pose Estimator modules in our
methods. Realistic motion and actuation noise models allow us to bridge the
physical domain gap and lead to much better real-world transfer.

###  2 Future Directions

Modular learning methods achieved reasonably good performance at spatial or
geometric navigation tasks achieving $92\%$ coverage in the Exploration task
and $95\%$ success rate in Point-Goal Navigation in our experiments. The
majority of the errors were attributed to errors in mapping. The semantic
navigation tasks still have ample room for improvement. Our methods achieve
around $55-65\%$ success rate at Object-Goal and Image-Goal Navigation tasks.
We found the errors were approximately equally attributed to inefficient
exploration and ineffective perception. This means that in around $20\%$ of
the episodes, the goal location was not found and in another $20\%$ of the
episodes, the goal location was found but the agent did not decide to stop at
the goal location. Both exploration efficiency and perception performance can
be improved in future methods.

A key limitation of all modular learning methods presented is inefficient low-
level control. Our methods are efficient in terms of the number of actions but
not in terms of wall-clock time. We assumed a discrete action space which
leads to slow low-level control when deployed on a physical robot. We also
assumed sequential processing of observations, taking physical actions, and
sampling next observations. This means the robot needs to stop after taking
each action, take the next observation from the sensors, process the
observations to compute the action, and then take the next action.
Asynchronous observation processing and motion will lead to much more
efficient navigation in terms of wall-clock time. This will introduce
challenges such as motion blur, lowering model inference time, and tackling
the delay between perception and control, which need to be tackled in the
future.

Another big avenue of research is the use of modular models for Embodied
Active Visual Learning. In biological beings, perception and action is
essentially a loop where perception is used to perform actions in the world,
and actions are used to better perceive the world. Navigation is just one part
of the loop where we are using perception to move in the world, but inversely
we can use actions to learn and improve perception. In an embodied active
learning setup, an agent has to perform actions such that observations
generated from these actions can be useful for learning better perception. We
barely scratched the surface of the potential of modular models for active
visual learning in Chapter 7. Several core research questions need to be
answered for large-scale applications: (a) what is the policy of exploration
that generates these observations? (b) what should be labeled in these
observations - one object, one frame, or the whole trajectory? (c) and
finally, how do we get these labels? Active Visual Learning can potentially
enable machine learning models to overcome the requirement of large-scale
annotated datasets and learn more efficiently with fewer labels by being able
to look at the world from different viewpoints, just like humans.

## Bibliography

  * hab [a] Habitat Challenge 2019.  https://aihabitat.org/challenge/2019/, a.
  * hab [b] Habitat Challenge 2020.  https://aihabitat.org/challenge/2020/, b.
  * Ammirato et al. [2017] Phil Ammirato, Patrick Poirson, Eunbyung Park, Jana Košecká, and Alexander C Berg.  A dataset for developing and benchmarking active vision.  In _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 1378–1385. IEEE, 2017.
  * Anderson et al. [2018a] Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, and Amir Zamir.  On evaluation of embodied navigation agents.  _arXiv preprint arXiv:1807.06757_ , 2018a.
  * Anderson et al. [2018b] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel.  Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3674–3683, 2018b.
  * Armeni et al. [2019] Iro Armeni, Zhi-Yang He, JunYoung Gwak, Amir R Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese.  3d scene graph: A structure for unified semantics, 3d space, and camera.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 5664–5673, 2019.
  * Artzi and Zettlemoyer [2013] Yoav Artzi and Luke Zettlemoyer.  Weakly supervised learning of semantic parsers for mapping instructions to actions.  _Transactions of the Association for Computational Linguistics_ , 1:49–62, 2013.
  * Auer [2002] Peter Auer.  Using confidence bounds for exploitation-exploration trade-offs.  _Journal of Machine Learning Research_ , 3(Nov):397–422, 2002.
  * Ba et al. [2014] Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu.  Multiple object recognition with visual attention.  _arXiv preprint arXiv:1412.7755_ , 2014.
  * Badrinarayanan et al. [2010] Vijay Badrinarayanan, Fabio Galasso, and Roberto Cipolla.  Label propagation in video sequences.  In _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , pages 3265–3272. IEEE, 2010.
  * Bagnell [2015] J Andrew Bagnell.  An invitation to imitation.  Technical report, DTIC Document, 2015.
  * Bajcsy [1988] Ruzena Bajcsy.  Active perception.  _Proceedings of the IEEE_ , 76(8):966–1005, 1988\.
  * Bansal et al. [2019] Somil Bansal, Varun Tolani, Saurabh Gupta, Jitendra Malik, and Claire Tomlin.  Combining optimal control and learning for visual navigation in novel environments.  _arXiv preprint arXiv:1903.02531_ , 2019.
  * Barto and Mahadevan [2003] Andrew G Barto and Sridhar Mahadevan.  Recent advances in hierarchical reinforcement learning.  _Discrete event dynamic systems_ , 13(1-2):41–77, 2003.
  * Batra et al. [2020] Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, and Erik Wijmans.  Objectnav revisited: On evaluation of embodied agents navigating to objects.  _arXiv preprint arXiv:2006.13171_ , 2020.
  * Beetz et al. [2011] Michael Beetz, Ulrich Klank, Ingo Kresse, Alexis Maldonado, Lorenz Mösenlechner, Dejan Pangercic, Thomas Rühr, and Moritz Tenorth.  Robotic roommates making pancakes.  In _Humanoid Robots (Humanoids), 2011 11th IEEE-RAS International Conference on_ , pages 529–536. IEEE, 2011.
  * Bellemare et al. [2012] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling.  The arcade learning environment: An evaluation platform for general agents.  _Journal of Artificial Intelligence Research_ , 2012.
  * Bengio et al. [2006] Yoshua Bengio, Olivier Delalleau, and Nicolas Le Roux.  Label propagation and quadratic criterion.  In _Semi-Supervised Learning_ , 2006.
  * Bollini et al. [2013] Mario Bollini, Stefanie Tellex, Tyler Thompson, Nicholas Roy, and Daniela Rus.  Interpreting and executing recipes with a cooking robot.  In _Experimental Robotics_ , pages 481–495. Springer, 2013.
  * Bonin-Font et al. [2008] Francisco Bonin-Font, Alberto Ortiz, and Gabriel Oliver.  Visual navigation for mobile robots: A survey.  _Journal of intelligent and robotic systems_ , 53(3):263–296, 2008.
  * Borenstein et al. [1996] Johann Borenstein, H. R. Everett, and Liqiang Feng.  _Navigating Mobile Robots: Systems and Techniques_.  A. K. Peters, Ltd., Natick, MA, USA, 1996.  ISBN 1568810660.
  * Bowman et al. [2017] Sean L Bowman, Nikolay Atanasov, Kostas Daniilidis, and George J Pappas.  Probabilistic data association for semantic slam.  In _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 1722–1729. IEEE, 2017.
  * Bruce et al. [2018] Jake Bruce, Niko Sunderhauf, Piotr Mirowski, Raia Hadsell, and Michael Milford.  Learning deployable navigation policies at kilometer scale from a single traversal.  In Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto, editors, _Proceedings of The 2nd Conference on Robot Learning_ , volume 87 of _Proceedings of Machine Learning Research_ , pages 346–361. PMLR, 29–31 Oct 2018.  URL http://proceedings.mlr.press/v87/bruce18a.html.
  * Burgard et al. [1996] Wolfram Burgard, Dieter Fox, Daniel Hennig, and Timo Schmidt.  Estimating the absolute position of a mobile robot using position probability grids.  In _Proceedings of the national conference on artificial intelligence_ , pages 896–901, 1996.
  * Burgard et al. [1998] Wolfram Burgard, Andrcas Derr, Dieter Fox, and Armin B Cremers.  Integrating global position estimation and position tracking for mobile robots: the dynamic markov localization approach.  In _Intelligent Robots and Systems, 1998. Proceedings., 1998 IEEE/RSJ International Conference on_ , volume 2, pages 730–735. IEEE, 1998.
  * Canny [1988] John Canny.  _The complexity of robot motion planning_.  MIT press, 1988.
  * Carlone et al. [2014] Luca Carlone, Jingjing Du, Miguel Kaouk Ng, Basilio Bona, and Marina Indri.  Active slam and exploration with particle filters using kullback-leibler divergence.  _Journal of Intelligent & Robotic Systems_, 75(2):291–311, 2014.
  * Chandra et al. [2018] Siddhartha Chandra, Camille Couprie, and Iasonas Kokkinos.  Deep spatio-temporal random fields for efficient video segmentation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 8915–8924, 2018.
  * Chang et al. [2017] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang.  Matterport3d: Learning from rgb-d data in indoor environments.  _International Conference on 3D Vision (3DV)_ , 2017.
  * Chao et al. [2011] Crystal Chao, Maya Cakmak, and Andrea L Thomaz.  Towards grounding concepts for transfer in goal learning from demonstration.  In _Development and Learning (ICDL), 2011 IEEE International Conference on_ , volume 2, pages 1–6. IEEE, 2011.
  * Chaplot and Lample [2017] Devendra Singh Chaplot and Guillaume Lample.  Arnold: An autonomous agent to play fps games.  In _Thirty-First AAAI Conference on Artificial Intelligence_ , 2017\.
  * Chaplot et al. [2016] Devendra Singh Chaplot, Guillaume Lample, Kanthashree Mysore Sathyendra, and Ruslan Salakhutdinov.  Transfer deep reinforcement learning in 3d environments: An empirical study.  In _NIPS Deep Reinforcemente Leaning Workshop_ , 2016.
  * Chaplot et al. [2018a] Devendra Singh Chaplot, Emilio Parisotto, and Ruslan Salakhutdinov.  Active neural localization.  _ICLR_ , 2018a.
  * Chaplot et al. [2018b] Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov.  Gated-attention architectures for task-oriented language grounding.  In _AAAI_ , 2018b.
  * Chaplot et al. [2020a] Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov.  Learning to explore using active neural slam.  In _ICLR_ , 2020a.
  * Chaplot et al. [2020b] Devendra Singh Chaplot, Helen Jiang, Saurabh Gupta, and Abhinav Gupta.  Semantic curiosity for active visual learning.  In _ECCV_ , 2020b.
  * Chaplot et al. [2020c] Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, and Saurabh Gupta.  Neural topological slam for visual navigation.  In _CVPR_ , 2020c.
  * Chen and Mooney [2011] David L Chen and Raymond J Mooney.  Learning to interpret natural language navigation instructions from observations.  In _AAAI_ , volume 2, pages 1–2, 2011.
  * Chen et al. [2019a] Kevin Chen, Juan Pablo de Vicente, Gabriel Sepulveda, Fei Xia, Alvaro Soto, Marynel Vázquez, and Silvio Savarese.  A behavioral approach to visual navigation with graph localization networks.  In _Robotics: Science and Systems_ , 2019a.
  * Chen et al. [2017] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  _arXiv preprint arXiv:1706.05587_ , 2017.
  * Chen et al. [2019b] Tao Chen, Saurabh Gupta, and Abhinav Gupta.  Learning exploration policies for navigation.  In _International Conference on Learning Representations_ , 2019b.  URL https://openreview.net/forum?id=SyMWn05F7.
  * Chen et al. [2013] Xinlei Chen, Abhinav Shrivastava, and Abhinav Gupta.  Neil: Extracting visual knowledge from web data.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 1409–1416, 2013.
  * Cho et al. [2014] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio.  On the properties of neural machine translation: Encoder-decoder approaches.  _Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation_ , 2014.
  * Choset and Nagatani [2001] Howie Choset and Keiji Nagatani.  Topological simultaneous localization and mapping (slam): toward exact localization without explicit localization.  _IEEE Transactions on robotics and automation_ , 17(2):125–137, 2001.
  * Chu et al. [2013] Vivian Chu, Ian McMahon, Lorenzo Riano, Craig G McDonald, Qin He, Jorge Martinez Perez-Tejada, Michael Arrigo, Naomi Fitter, John C Nappo, and Trevor Darrell.  Using robotic exploratory procedures to learn the meaning of haptic adjectives.  In _Robotics and Automation (ICRA)_ , pages 3048–3055, 2013.
  * Chung et al. [2014] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _arXiv preprint arXiv:1412.3555_ , 2014.
  * Chunjie et al. [2017] Luo Chunjie, Yang Qiang, et al.  Cosine normalization: Using cosine similarity instead of dot product in neural networks.  _arXiv preprint arXiv:1702.05870_ , 2017.
  * Clark et al. [2017a] Ronald Clark, Sen Wang, Andrew Markham, Niki Trigoni, and Hongkai Wen.  Vidloc: 6-dof video-clip relocalization.  _arXiv preprint arXiv:1702.06521_ , 2017a.
  * Clark et al. [2017b] Ronald Clark, Sen Wang, Hongkai Wen, Andrew Markham, and Niki Trigoni.  Vinet: Visual-inertial odometry as a sequence-to-sequence learning problem.  In _AAAI_ , pages 3995–4001, 2017b.
  * Cox and Leonard [1994] Ingemar J Cox and John J Leonard.  Modeling a dynamic environment using a bayesian multiple hypothesis approach.  _Artificial Intelligence_ , 66(2):311–344, 1994\.
  * Cox and Wilfong [1990] Ingemar J. Cox and Gordon T. Wilfong, editors.  _Autonomous Robot Vehicles_.  Springer-Verlag New York, Inc., New York, NY, USA, 1990.  ISBN 0-387-97240-4.
  * Das et al. [2018a] Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra.  Embodied question answering.  In _CVPR_ , 2018a.
  * Das et al. [2018b] Abhishek Das, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra.  Neural modular control for embodied question answering.  In _Conference on Robot Learning_ , pages 53–62, 2018b.
  * Dayan and Hinton [1993] Peter Dayan and Geoffrey E Hinton.  Feudal reinforcement learning.  In _Advances in neural information processing systems_ , pages 271–278, 1993.
  * Dellaert et al. [1999] Frank Dellaert, Dieter Fox, Wolfram Burgard, and Sebastian Thrun.  Monte carlo localization for mobile robots.  In _ICRA_ , volume 2, pages 1322–1328, 1999.
  * Dhingra et al. [2017] Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov.  Gated-attention readers for text comprehension.  _ACL_ , 2017.
  * Dornhege and Kleiner [2013] Christian Dornhege and Alexander Kleiner.  A frontier-void-based approach for autonomous exploration in 3d.  _Advanced Robotics_ , 27(6):459–468, 2013.
  * Dosovitskiy and Koltun [2017] Alexey Dosovitskiy and Vladlen Koltun.  Learning to act by predicting the future.  In _ICLR_ , 2017.
  * Elfes [1989] Alberto Elfes.  Using occupancy grids for mobile robot perception and navigation.  _Computer_ , 22(6):46–57, 1989.
  * [60] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.  The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results.  http://www.pascal-network.org/challenges/VOC/voc2007/workshop/index.html.
  * Eysenbach et al. [2019a] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine.  Diversity is all you need: Learning skills without a reward function.  In _International Conference on Learning Representations_ , 2019a.  URL https://openreview.net/forum?id=SJx63jRqFm.
  * Eysenbach et al. [2019b] Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine.  Search on the replay buffer: Bridging planning and reinforcement learning.  _arXiv preprint arXiv:1906.05253_ , 2019b.
  * Fang et al. [2019a] Kuan Fang, Alexander Toshev, Li Fei-Fei, and Silvio Savarese.  Scene memory transformer for embodied agents in long-horizon tasks.  In _CVPR_ , 2019a.
  * Fang et al. [2019b] Kuan Fang, Alexander Toshev, Li Fei-Fei, and Silvio Savarese.  Scene memory transformer for embodied agents in long-horizon tasks.  In _CVPR_ , 2019b.
  * Fasola and Mataric [2013] Juan Fasola and Maja J Mataric.  Using semantic fields to model dynamic spatial relations in a robot architecture for natural language instruction of service robots.  In _Intelligent Robots and Systems (IROS)_ , pages 143–150. IEEE, 2013.
  * Fathi et al. [2011] Alireza Fathi, Maria Florina Balcan, Xiaofeng Ren, and James M Rehg.  Combining self training and active learning for video segmentation.  In _Proceedings of the British Machine Vision Conference_. Georgia Institute of Technology, 2011.
  * Foerster et al. [2016] Jakob N Foerster, Yannis M Assael, Nando de Freitas, and Shimon Whiteson.  Learning to communicate to solve riddles with deep distributed recurrent q-networks.  _arXiv preprint arXiv:1602.02672_ , 2016.
  * Fox [1998] Dieter Fox.  _Markov localization-a probabilistic framework for mobile robot localization and navigation._ PhD thesis, Universität Bonn, 1998.
  * Fox et al. [1998] Dieter Fox, Wolfram Burgard, and Sebastian Thrun.  Active markov localization for mobile robots.  _Robotics and Autonomous Systems_ , 25(3-4):195–207, 1998.
  * Fox et al. [2003] V Fox, Jeffrey Hightower, Lin Liao, Dirk Schulz, and Gaetano Borriello.  Bayesian filtering for location estimation.  _IEEE pervasive computing_ , 2(3):24–33, 2003\.
  * Fuentes-Pacheco et al. [2015] J. Fuentes-Pacheco, J. Ruiz-Ascencio, and J. M. Rendón-Mancha.  Visual simultaneous localization and mapping: a survey.  _Artificial Intelligence Review_ , 2015.
  * Gadde et al. [2017] Raghudeep Gadde, Varun Jampani, and Peter V Gehler.  Semantic video cnns through representation warping.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 4453–4462, 2017.
  * Gal et al. [2017] Yarin Gal, Riashat Islam, and Zoubin Ghahramani.  Deep bayesian active learning with image data.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pages 1183–1192. JMLR. org, 2017.
  * Gandhi et al. [2017] Dhiraj Gandhi, Lerrel Pinto, and Abhinav Gupta.  Learning to fly by crashing.  In _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 3948–3955. IEEE, 2017.
  * Gaskett et al. [1999] Chris Gaskett, David Wettergreen, and Alexander Zelinsky.  Q-learning in continuous state and action spaces.  In _Australasian Joint Conference on Artificial Intelligence_ , pages 417–428. Springer, 1999.
  * Gillner and Mallot [1998] Sabine Gillner and Hanspeter A Mallot.  Navigation and acquisition of spatial knowledge in a virtual maze.  _Journal of cognitive neuroscience_ , 10(4):445–463, 1998.
  * Godard et al. [2017] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow.  Unsupervised monocular depth estimation with left-right consistency.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 270–279, 2017.
  * Gordon et al. [2018] Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi.  Iqa: Visual question answering in interactive environments.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 4089–4098, 2018.
  * Guadarrama et al. [2013] Sergio Guadarrama, Lorenzo Riano, Dave Golland, Daniel Go, Yangqing Jia, Dan Klein, Pieter Abbeel, Trevor Darrell, et al.  Grounding spatial relations for human-robot interaction.  In _IROS_ , pages 1640–1647. IEEE, 2013.
  * Guadarrama et al. [2014] Sergio Guadarrama, Erik Rodner, Kate Saenko, Ning Zhang, Ryan Farrell, Jeff Donahue, and Trevor Darrell.  Open-vocabulary object retrieval.  In _Robotics: science and systems_ , volume 2, page 6. Citeseer, 2014\.
  * Gupta et al. [2017] Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik.  Cognitive mapping and planning for visual navigation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 2616–2625, 2017.
  * Hartley and Zisserman [2003] Richard Hartley and Andrew Zisserman.  _Multiple view geometry in computer vision_.  Cambridge university press, 2003.
  * Hausknecht and Stone [2015] Matthew Hausknecht and Peter Stone.  Deep recurrent q-learning for partially observable mdps.  _arXiv preprint arXiv:1507.06527_ , 2015.
  * He et al. [2017] K. He, G. Gkioxari, P. Dollár, and R. Girshick.  Mask r-cnn.  In _2017 IEEE International Conference on Computer Vision (ICCV)_ , pages 2980–2988, 2017.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Heess et al. [2015] Nicolas Heess, Gregory Wayne, David Silver, Tim Lillicrap, Tom Erez, and Yuval Tassa.  Learning continuous control policies by stochastic value gradients.  In _Advances in Neural Information Processing Systems_ , pages 2944–2952, 2015.
  * Henriques and Vedaldi [2018] Joao F Henriques and Andrea Vedaldi.  Mapnet: An allocentric spatial memory for mapping environments.  In _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 8476–8484, 2018.
  * Henry et al. [2014] Peter Henry, Michael Krainin, Evan Herbst, Xiaofeng Ren, and Dieter Fox.  Rgb-d mapping: Using depth cameras for dense 3d modeling of indoor environments.  In _Experimental robotics_ , pages 477–491. Springer, 2014.
  * Hermann et al. [2017] Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojtek Czarnecki, Max Jaderberg, Denis Teplyashin, et al.  Grounded language learning in a simulated 3d world.  _arXiv preprint arXiv:1706.06551_ , 2017.
  * Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780, 1997.
  * Holz et al. [2010] Dirk Holz, Nicola Basilico, Francesco Amigoni, and Sven Behnke.  Evaluating the efficiency of frontier-based exploration strategies.  In _ISR 2010 (41st International Symposium on Robotics) and ROBOTIK 2010 (6th German Conference on Robotics)_ , pages 1–8. VDE, 2010.
  * Horn [1990] Roger A Horn.  The hadamard product.  In _Proc. Symp. Appl. Math_ , volume 40, pages 87–169, 1990.
  * Hornung et al. [2013] Armin Hornung, Kai M. Wurm, Maren Bennewitz, Cyrill Stachniss, and Wolfram Burgard.  OctoMap: An efficient probabilistic 3D mapping framework based on octrees.  _Autonomous Robots_ , 2013.  10.1007/s10514-012-9321-0.  URL http://octomap.github.com.
  * Huang et al. [2013] Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck.  Learning deep structured semantic models for web search using clickthrough data.  In _Proceedings of the 22nd ACM international conference on Conference on information & knowledge management_, pages 2333–2338. ACM, 2013\.
  * Huber [1964] Peter J Huber.  Robust estimation of a location parameter.  _The Annals of Mathematical Statistics_ , 35(1):73–101, 1964.
  * Izadi et al. [2011] Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, and Andrew Fitzgibbon.  KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera.  _UIST_ , 2011.
  * Jaderberg et al. [2015] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al.  Spatial transformer networks.  In _Advances in neural information processing systems_ , pages 2017–2025, 2015.
  * Jaksch et al. [2010] Thomas Jaksch, Ronald Ortner, and Peter Auer.  Near-optimal regret bounds for reinforcement learning.  _Journal of Machine Learning Research_ , 11(Apr):1563–1600, 2010.
  * Jayaraman and Grauman [2018] Dinesh Jayaraman and Kristen Grauman.  Learning to look around: Intelligently exploring unseen environments for unknown tasks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1238–1247, 2018.
  * Jensfelt and Kristensen [2001] Patric Jensfelt and Steen Kristensen.  Active global localization for a mobile robot using multiple hypothesis tracking.  _IEEE Transactions on Robotics and Automation_ , 17(5):748–760, 2001.
  * Kalman et al. [1960] Rudolph Emil Kalman et al.  A new approach to linear filtering and prediction problems.  _Journal of basic Engineering_ , 82(1):35–45, 1960.
  * Karkus et al. [2017] Peter Karkus, David Hsu, and Wee Sun Lee.  Qmdp-net: Deep learning for planning under partial observability.  _CoRR_ , abs/1703.06692, 2017.  URL https://arxiv.org/abs/1703.06692.
  * Kaufmann et al. [2019] Elia Kaufmann, Mathias Gehrig, Philipp Foehn, René Ranftl, Alexey Dosovitskiy, Vladlen Koltun, and Davide Scaramuzza.  Beauty and the beast: Optimal methods meet learning for drone racing.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pages 690–696. IEEE, 2019.
  * Kavraki et al. [1996] Lydia E Kavraki, Petr Svestka, J-C Latombe, and Mark H Overmars.  Probabilistic roadmaps for path planning in high-dimensional configuration spaces.  _RA_ , 1996.
  * Kearns and Singh [2002] Michael Kearns and Satinder Singh.  Near-optimal reinforcement learning in polynomial time.  _Machine learning_ , 49(2-3):209–232, 2002.
  * Kempka et al. [2016] Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaśkowski.  Vizdoom: A doom-based ai research platform for visual reinforcement learning.  _arXiv preprint arXiv:1605.02097_ , 2016.
  * Kendall et al. [2015] Alex Kendall, Matthew Grimes, and Roberto Cipolla.  Posenet: A convolutional network for real-time 6-dof camera relocalization.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2938–2946, 2015.
  * Khan et al. [2017] Arbaaz Khan, Clark Zhang, Nikolay Atanasov, Konstantinos Karydis, Daniel D Lee, and Vijay Kumar.  End-to-end navigation in unknown environments using neural networks.  _arXiv preprint arXiv:1707.07385_ , 2017.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Koch et al. [2015] Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov.  Siamese neural networks for one-shot image recognition.  In _ICML Deep Learning Workshop_ , volume 2, 2015.
  * Kollar and Roy [2008] Thomas Kollar and Nicholas Roy.  Trajectory optimization using reinforcement learning for map exploration.  _The International Journal of Robotics Research_ , 27(2):175–196, 2008.
  * Kostrikov [2018] Ilya Kostrikov.  Pytorch implementations of reinforcement learning algorithms.  https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail, 2018.
  * Koutník et al. [2013] Jan Koutník, Giuseppe Cuccu, Jürgen Schmidhuber, and Faustino Gomez.  Evolving large-scale neural networks for vision-based reinforcement learning.  In _Proceedings of the 15th annual conference on Genetic and evolutionary computation_ , pages 1061–1068. ACM, 2013.
  * Kruskal [1956] Joseph B Kruskal.  On the shortest spanning subtree of a graph and the traveling salesman problem.  _Proceedings of the American Mathematical society_ , 7(1):48–50, 1956.
  * Kuipers and Byun [1991] Benjamin Kuipers and Yung-Tai Byun.  A robot exploration and mapping strategy based on a semantic hierarchy of spatial representations.  _Robotics and autonomous systems_ , 8(1-2):47–63, 1991.
  * Kulick et al. [2013] Johannes Kulick, Marc Toussaint, Tobias Lang, and Manuel Lopes.  Active learning for teaching a robot grounded relational symbols.  In _IJCAI_ , 2013.
  * Kulkarni et al. [2016] Tejas D Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel J Gershman.  Deep successor reinforcement learning.  _arXiv preprint arXiv:1606.02396_ , 2016.
  * Kumar et al. [2018] Ashish Kumar, Saurabh Gupta, David Fouhey, Sergey Levine, and Jitendra Malik.  Visual memory for robust path following.  In _Advances in Neural Information Processing Systems_ , 2018.
  * Kuo et al. [2018] Weicheng Kuo, Christian Häne, Esther Yuh, Pratik Mukherjee, and Jitendra Malik.  Cost-sensitive active learning for intracranial hemorrhage detection.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 715–723. Springer, 2018.
  * Lample and Chaplot [2017] Guillaume Lample and Devendra Singh Chaplot.  Playing FPS games with deep reinforcement learning.  In _Thirty-First AAAI Conference on Artificial Intelligence_ , 2017\.
  * Lavalle and Kuffner Jr [2000] Steven M Lavalle and James J Kuffner Jr.  Rapidly-exploring random trees: Progress and prospects.  In _Algorithmic and Computational Robotics: New Directions_ , 2000\.
  * LeCun et al. [1995] Yann LeCun, Yoshua Bengio, et al.  Convolutional networks for images, speech, and time series.  _The handbook of brain theory and neural networks_ , 3361(10):1995, 1995.
  * Lee et al. [2017] Alex X Lee, Sergey Levine, and Pieter Abbeel.  Learning visual servoing with deep features and fitted q-iteration.  _arXiv preprint arXiv:1703.11000_ , 2017.
  * Lee et al. [2018] Lisa Lee, Emilio Parisotto, Devendra Singh Chaplot, Eric Xing, and Ruslan Salakhutdinov.  Gated path planning networks.  _arXiv preprint arXiv:1806.06408_ , 2018.
  * Lemaignan et al. [2012] Séverin Lemaignan, Raquel Ros, E Akin Sisbot, Rachid Alami, and Michael Beetz.  Grounding the interaction: Anchoring situated discourse in everyday human-robot interaction.  _International Journal of Social Robotics_ , 4(2):181–199, 2012.
  * Levine et al. [2016] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel.  End-to-end training of deep visuomotor policies.  _Journal of Machine Learning Research_ , 17(39):1–40, 2016.
  * Lin [1993] Long-Ji Lin.  Reinforcement learning for robots using neural networks.  Technical report, DTIC Document, 1993.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _ECCV_ , 2014.
  * Lin et al. [2017] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2117–2125, 2017.
  * Ma et al. [2017] Lingni Ma, Jörg Stückler, Christian Kerl, and Daniel Cremers.  Multi-view deep learning for consistent semantic mapping with rgb-d cameras.  In _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 598–605. IEEE, 2017.
  * Martin et al. [2001] D. Martin, C. Fowlkes, D. Tal, and J. Malik.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics.  In _Proc. 8th Int’l Conf. Computer Vision_ , volume 2, pages 416–423, July 2001.
  * Martinez-Cantin et al. [2009] Ruben Martinez-Cantin, Nando de Freitas, Eric Brochu, José Castellanos, and Arnaud Doucet.  A bayesian exploration-exploitation approach for optimal online sensing and planning with a visually guided mobile robot.  _Autonomous Robots_ , 27(2):93–103, 2009.
  * McPartland and Gallagher [2008] Michelle McPartland and Marcus Gallagher.  Learning to be a bot: Reinforcement learning in shooter games.  In _AIIDE_ , 2008.
  * Mei et al. [2015] Hongyuan Mei, Mohit Bansal, and Matthew R Walter.  Listen, attend, and walk: Neural mapping of navigational instructions to action sequences.  _arXiv preprint arXiv:1506.04089_ , 2015.
  * Mei et al. [2016] Hongyuan Mei, Mohit Bansal, and Matthew R Walter.  Listen, attend, and walk: Neural mapping of navigational instructions to action sequences.  In _AAAI_ , volume 1, page 2, 2016.
  * Meng and Kak [1993] Min Meng and Avinash C Kak.  Mobile robot navigation using neural networks and nonmetrical environmental models.  _IEEE Control Systems Magazine_ , 13(5):30–39, 1993.
  * Mirowski et al. [2016] Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al.  Learning to navigate in complex environments.  _arXiv preprint arXiv:1611.03673_ , 2016.
  * Mirowski et al. [2018] Piotr Mirowski, Matthew Koichi Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Koray Kavukcuoglu, Andrew Zisserman, and Raia Hadsell.  Learning to navigate in cities without a map.  In _Neural Information Processing Systems (NeurIPS)_ , 2018.
  * Misra et al. [2017] Dipendra K Misra, John Langford, and Yoav Artzi.  Mapping instructions and visual observations to actions with reinforcement learning.  _arXiv preprint arXiv:1704.08795_ , 2017.
  * Misra et al. [2018] Ishan Misra, Ross Girshick, Rob Fergus, Martial Hebert, Abhinav Gupta, and Laurens van der Maaten.  Learning by Asking Questions.  In _CVPR_ , 2018.
  * Mnih et al. [2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ , 2013.
  * Mnih et al. [2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.  Asynchronous methods for deep reinforcement learning.  In _ICML_ , 2016.
  * Mousavian et al. [2019] Arsalan Mousavian, Alexander Toshev, Marek Fišer, Jana Košecká, Ayzaan Wahid, and James Davidson.  Visual representations for semantic target driven navigation.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pages 8846–8852. IEEE, 2019.
  * Mur-Artal and Tardós [2017] Raul Mur-Artal and Juan D Tardós.  Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras.  _IEEE Transactions on Robotics_ , 33(5):1255–1262, 2017.
  * Mur-Artal et al. [2015] Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos.  Orb-slam: a versatile and accurate monocular slam system.  _IEEE transactions on robotics_ , 31(5):1147–1163, 2015.
  * Murali et al. [2019] Adithyavairavan Murali, Tao Chen, Kalyan Vasudev Alwala, Dhiraj Gandhi, Lerrel Pinto, Saurabh Gupta, and Abhinav Gupta.  Pyrobot: An open-source robotics framework for research and benchmarking.  _arXiv preprint arXiv:1906.08236_ , 2019.
  * Nair and Hinton [2010] Vinod Nair and Geoffrey E Hinton.  Rectified linear units improve restricted boltzmann machines.  In _Proceedings of the 27th international conference on machine learning (ICML-10)_ , pages 807–814, 2010.
  * Newcombe et al. [2011a] Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohli, Jamie Shotton, Steve Hodges, and Andrew W Fitzgibbon.  Kinectfusion: Real-time dense surface mapping and tracking.  In _ISMAR_ , volume 11, pages 127–136, 2011a.
  * Newcombe et al. [2011b] Richard A Newcombe, Steven J Lovegrove, and Andrew J Davison.  Dtam: Dense tracking and mapping in real-time.  In _2011 international conference on computer vision_ , pages 2320–2327. IEEE, 2011b.
  * Ng [2003] Andrew Y Ng.  _Shaping and policy search in reinforcement learning_.  PhD thesis, University of California, Berkeley, 2003.
  * Nistér et al. [2006] David Nistér, Oleg Naroditsky, and James Bergen.  Visual odometry for ground vehicle applications.  _Journal of Field Robotics_ , 23(1):3–20, 2006\.
  * Oh et al. [2016] Junhyuk Oh, Valliappa Chockalingam, Satinder Singh, and Honglak Lee.  Control of memory, active perception, and action in minecraft.  In _ICML_ , 2016.
  * Parisotto and Salakhutdinov [2018] Emilio Parisotto and Ruslan Salakhutdinov.  Neural map: Structured memory for deep reinforcement learning.  In _ICLR_ , 2018.
  * Paszke et al. [2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  2017\.
  * Pathak et al. [2017a] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell.  Curiosity-driven exploration by self-supervised prediction.  In _International Conference on Machine Learning (ICML)_ , 2017a.
  * Pathak et al. [2017b] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell.  Curiosity-driven exploration by self-supervised prediction.  In _ICML_ , 2017b.
  * Pathak et al. [2019] Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta.  Self-supervised exploration via disagreement.  In _ICML_ , 2019.
  * Pronobis and Jensfelt [2012] Andrzej Pronobis and Patric Jensfelt.  Large-scale semantic mapping and reasoning with heterogeneous modalities.  In _2012 IEEE International Conference on Robotics and Automation_ , pages 3515–3522. IEEE, 2012.
  * Quigley et al. [2009] Morgan Quigley, Brian Gerkey, Ken Conley, Josh Faust, Tully Foote, Jeremy Leibs, Eric Berger, Rob Wheeler, and Andrew Ng.  Ros: an open-source robot operating system.  In _Proc. of the IEEE Intl. Conf. on Robotics and Automation (ICRA) Workshop on Open Source Robotics_ , Kobe, Japan, May 2009.
  * Ren et al. [2015] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In _Advances in neural information processing systems_ , pages 91–99, 2015.
  * Ross et al. [2011] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell.  A reduction of imitation learning and structured prediction to no-regret online learning.  In _Proceedings of the fourteenth international conference on artificial intelligence and statistics_ , pages 627–635, 2011.
  * Roumeliotis and Bekey [2000] Stergios I. Roumeliotis and George A. Bekey.  Bayesian estimation and kalman filtering: A unified framework for mobile robot localization, 2000.
  * Russell and Norvig [1995] Stuart Russell and Peter Norvig.  A modern approach.  _Artificial Intelligence. Prentice-Hall, Egnlewood Cliffs_ , 25:27, 1995.
  * Sadeghi and Levine [2016] Fereshteh Sadeghi and Sergey Levine.  Cad2rl: Real single-image flight without a single real image.  _arXiv preprint arXiv:1611.04201_ , 2016.
  * Savinov et al. [2018a] Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun.  Semi-parametric topological memory for navigation.  In _International Conference on Learning Representations (ICLR)_ , 2018a.
  * Savinov et al. [2018b] Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Marc Pollefeys, Timothy Lillicrap, and Sylvain Gelly.  Episodic curiosity through reachability.  _arXiv preprint arXiv:1810.02274_ , 2018b.
  * Savva et al. [2017] Manolis Savva, Angel X. Chang, Alexey Dosovitskiy, Thomas Funkhouser, and Vladlen Koltun.  MINOS: Multimodal indoor simulator for navigation in complex environments.  _arXiv:1712.03931_ , 2017.
  * Savva et al. [2019] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al.  Habitat: A platform for embodied ai research.  In _ICCV_ , 2019.
  * Schaul et al. [2015] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver.  Prioritized experience replay.  _arXiv preprint arXiv:1511.05952_ , 2015.
  * Schmidhuber [1991] Jürgen Schmidhuber.  A possibility for implementing curiosity and boredom in model-building neural controllers.  In _Proc. of the international conference on simulation of adaptive behavior: From animals to animats_ , pages 222–227, 1991.
  * Schulman et al. [2015] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel.  High-dimensional continuous control using generalized advantage estimation.  _arXiv preprint arXiv:1506.02438_ , 2015.
  * Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Sener and Savarese [2018] Ozan Sener and Silvio Savarese.  Active learning for convolutional neural networks: A core-set approach.  In _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=H1aIuk-RW.
  * Sethian [1996] James A Sethian.  A fast marching level set method for monotonically advancing fronts.  _Proceedings of the National Academy of Sciences_ , 93(4):1591–1595, 1996.
  * Settles [2009] Burr Settles.  Active learning literature survey.  Technical report, University of Wisconsin-Madison Department of Computer Sciences, 2009.
  * Shah et al. [2017] Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor.  Airsim: High-fidelity visual and physical simulation for autonomous vehicles.  In _Field and Service Robotics_ , 2017.  URL https://arxiv.org/abs/1705.05065.
  * Siddiqui et al. [2020] Yawar Siddiqui, Julien Valentin, and Matthias Nießner.  Viewal: Active learning with viewpoint entropy for semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9433–9443, 2020.
  * Silver et al. [2016] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  _Nature_ , 529(7587):484–489, 2016.
  * Smith et al. [1990] Randall Smith, Matthew Self, and Peter Cheeseman.  Estimating uncertain spatial relationships in robotics.  In _Autonomous robot vehicles_ , pages 167–193. Springer, 1990.
  * Snavely et al. [2008] Noah Snavely, Steven M Seitz, and Richard Szeliski.  Modeling the world from internet photo collections.  _International journal of computer vision_ , 80(2):189–210, 2008.
  * Song et al. [2017] Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser.  Semantic scene completion from a single depth image.  In _CVPR_ , 2017.
  * Stachniss et al. [2005] Cyrill Stachniss, Giorgio Grisetti, and Wolfram Burgard.  Information gain-based exploration using rao-blackwellized particle filters.  In _Robotics: Science and Systems_ , volume 2, pages 65–72, 2005\.
  * Straub et al. [2019] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe.  The Replica dataset: A digital replica of indoor spaces.  _arXiv preprint arXiv:1906.05797_ , 2019.
  * Sutton and Barto [1998] Richard S. Sutton and Andrew G. Barto.  _Reinforcement Learning: An Introduction_.  MIT Press, 1998.  URL http://www.cs.ualberta.ca/~sutton/book/the-book.html.
  * Sutton and Barto [2018] Richard S Sutton and Andrew G Barto.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Sutton et al. [1999] Richard S Sutton, Doina Precup, and Satinder Singh.  Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning.  _Artificial intelligence_ , 112(1-2):181–211, 1999.
  * Tamar et al. [2016] Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, and Pieter Abbeel.  Value iteration networks.  In _Advances in Neural Information Processing Systems_ , pages 2154–2162, 2016.
  * Tastan and Sukthankar [2011] Bulent Tastan and Gita Reese Sukthankar.  Learning policies for first person shooter games using inverse reinforcement learning.  In _AIIDE_. Citeseer, 2011.
  * Tellex et al. [2011] Stefanie A Tellex, Thomas Fleming Kollar, Steven R Dickerson, Matthew R Walter, Ashis Banerjee, Seth Teller, and Nicholas Roy.  Understanding natural language commands for robotic navigation and mobile manipulation.  2011\.
  * Thrun et al. [1998] Sebastian Thrun, Jens-Steffen Gutmann, Dieter Fox, Wolfram Burgard, Benjamin Kuipers, et al.  Integrating topological and metric maps for mobile robot navigation: A statistical approach.  In _AAAI/IAAI_ , pages 989–995, 1998.
  * Thrun et al. [2001] Sebastian Thrun, Dieter Fox, Wolfram Burgard, and Frank Dellaert.  Robust monte carlo localization for mobile robots.  _Artificial intelligence_ , 128(1-2):99–141, 2001\.
  * Thrun et al. [2005] Sebastian Thrun, Wolfram Burgard, and Dieter Fox.  _Probabilistic robotics_.  MIT press, 2005.
  * Tieleman and Hinton [2012] Tijmen Tieleman and Geoffrey Hinton.  Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude.  _COURSERA: Neural networks for machine learning_ , 4(2), 2012.
  * Tobin et al. [2017] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel.  Domain randomization for transferring deep neural networks from simulation to the real world.  In _Intelligent Robots and Systems (IROS), 2017 IEEE/RSJ International Conference on_ , pages 23–30. IEEE, 2017.
  * Tomatis et al. [2001] Nicola Tomatis, Illah Nourbakhsh, and Roland Siegwart.  Combining topological and metric: A natural integration for simultaneous localization and map building.  In _Proceedings of the Fourth European Workshop on Advanced Mobile Robots (Eurobot)_. ETH-Zürich, 2001.
  * Tulsiani et al. [2019] Shubham Tulsiani, Tinghui Zhou, Alexei A. Efros, and Jitendra Malik.  Multi-view supervision for single-view reconstruction via differentiable ray consistency.  _TPAMI_ , 2019.
  * Van Hasselt et al. [2015] Hado Van Hasselt, Arthur Guez, and David Silver.  Deep reinforcement learning with double q-learning.  _CoRR, abs/1509.06461_ , 2015.
  * Vijayanarasimhan and Grauman [2014] Sudheendra Vijayanarasimhan and Kristen Grauman.  Large-scale live active learning: Training object detectors with crawled data and crowds.  _International Journal of Computer Vision_ , 108(1-2):97–114, 2014.
  * Vondrick et al. [2013] Carl Vondrick, Donald Patterson, and Deva Ramanan.  Efficiently scaling up crowdsourced video annotation.  _International Journal of Computer Vision_ , pages 1–21, 2013.  ISSN 0920-5691.  URL http://dx.doi.org/10.1007/s11263-012-0564-1.  10.1007/s11263-012-0564-1.
  * Walter et al. [2013] Matthew R Walter, Sachithra Hemachandra, Bianca Homberg, Stefanie Tellex, and Seth Teller.  Learning semantic maps from natural language descriptions.  Robotics: Science and Systems, 2013.
  * Wang and Spelke [2002] Ranxiao Frances Wang and Elizabeth S Spelke.  Human spatial representation: Insights from animals.  _Trends in cognitive sciences_ , 6(9):376–382, 2002.
  * Wang et al. [2017] Sen Wang, Ronald Clark, Hongkai Wen, and Niki Trigoni.  Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks.  In _Robotics and Automation (ICRA), 2017 IEEE International Conference on_ , pages 2043–2050. IEEE, 2017.
  * Wang et al. [2015] Ziyu Wang, Nando de Freitas, and Marc Lanctot.  Dueling network architectures for deep reinforcement learning.  _arXiv preprint arXiv:1511.06581_ , 2015.
  * Wijmans et al. [2020] Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra.  Decentralized distributed ppo: Solving pointgoal navigation.  In _ICLR_ , 2020.
  * Wortsman et al. [2019] Mitchell Wortsman, Kiana Ehsani, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.  Learning to learn how to learn: Self-adaptive visual navigation using meta-learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6750–6759, 2019.
  * Wu et al. [2018] Yi Wu, Yuxin Wu, Aviv Tamar, Stuart Russell, Georgia Gkioxari, and Yuandong Tian.  Learning and planning with a semantic model.  _arXiv preprint arXiv:1809.10842_ , 2018.
  * Wu et al. [2019a] Yi Wu, Yuxin Wu, Aviv Tamar, Stuart Russell, Georgia Gkioxari, and Yuandong Tian.  Bayesian relational memory for semantic visual navigation.  In _CVPR_ , pages 2769–2779, 2019a.
  * Wu et al. [2016] Yuhuai Wu, Saizheng Zhang, Ying Zhang, Yoshua Bengio, and Ruslan Salakhutdinov.  On multiplicative integration with recurrent neural networks.  In _NIPS_ , 2016.
  * Wu and Tian [2017] Yuxin Wu and Yuandong Tian.  Training agent for first-person shooter game with actor-critic curriculum learning.  In _ICLR_ , 2017.
  * Wu et al. [2019b] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019b.
  * Xia et al. [2018a] Fei Xia, Amir R. Zamir, Zhi-Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese.  Gibson Env: real-world perception for embodied agents.  In _Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on_. IEEE, 2018a.
  * Xia et al. [2018b] Fei Xia, Amir R. Zamir, Zhi-Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese.  Gibson Env: real-world perception for embodied agents.  In _Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on_. IEEE, 2018b.
  * Xu et al. [2017] Kai Xu, Lintao Zheng, Zihao Yan, Guohang Yan, Eugene Zhang, Matthias Niessner, Oliver Deussen, Daniel Cohen-Or, and Hui Huang.  Autonomous reconstruction of unknown indoor scenes guided by time-varying tensor fields.  _ACM Transactions on Graphics (TOG)_ , 36(6):202, 2017.
  * Yamauchi [1997] Brian Yamauchi.  A frontier-based approach for autonomous exploration.  In _cira_ , volume 97, page 146, 1997.
  * Yang et al. [2019] Jianwei Yang, Zhile Ren, Mingze Xu, Xinlei Chen, David J Crandall, Devi Parikh, and Dhruv Batra.  Embodied amodal recognition: Learning to move to perceive objects.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 2040–2050, 2019.
  * Yang et al. [2018] Wei Yang, Xiaolong Wang, Ali Farhadi, Abhinav Gupta, and Roozbeh Mottaghi.  Visual semantic navigation using scene priors.  _arXiv preprint arXiv:1810.06543_ , 2018.
  * Yoo and Kweon [2019] Donggeun Yoo and In So Kweon.  Learning loss for active learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 93–102, 2019.
  * Yu et al. [2017] Haonan Yu, Haichao Zhang, and Wei Xu.  A deep compositional framework for human-like language acquisition in virtual environment.  _arXiv preprint arXiv:1703.09831_ , 2017.
  * Zhang et al. [2018a] Amy Zhang, Adam Lerer, Sainbayar Sukhbaatar, Rob Fergus, and Arthur Szlam.  Composable planning with attributes.  _arXiv preprint arXiv:1803.00512_ , 2018a.
  * Zhang et al. [2017] Jingwei Zhang, Lei Tai, Joschka Boedecker, Wolfram Burgard, and Ming Liu.  Neural slam: Learning to explore with external memory.  _arXiv preprint arXiv:1706.09520_ , 2017.
  * Zhang et al. [2018b] Liang Zhang, Leqi Wei, Peiyi Shen, Wei Wei, Guangming Zhu, and Juan Song.  Semantic slam based on object detection and improved octomap.  _IEEE Access_ , 6:75545–75559, 2018b.
  * Zhu et al. [2017] Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi.  Target-driven visual navigation in indoor scenes using deep reinforcement learning.  In _2017 IEEE international conference on robotics and automation (ICRA)_ , pages 3357–3364. IEEE, 2017.