11institutetext: Istituto Italiano di Tecnologia, Genova, Italy
<EMAIL_ADDRESS>
# Look around and learn: self-improving object detection by exploration

Gianluca Scarpellini  Stefano Rosa  Pietro Morerio  Lorenzo Natale  Alessio
Del Bue

###### Abstract

Object detectors often experience a drop in performance when new environmental
conditions are insufficiently represented in the training data. This paper
studies how to automatically fine-tune a pre-existing object detector while
exploring and acquiring images in a new environment without relying on human
intervention, i.e., in an utterly self-supervised fashion. In our setting, an
agent initially learns to explore the environment using a pre-trained off-the-
shelf detector to locate objects and associate pseudo-labels. By assuming that
pseudo-labels for the same object must be consistent across different views,
we learn an exploration policy mining hard samples and we devise a novel
mechanism for producing refined predictions from the consensus among
observations. Our approach outperforms the current state-of-the-art, and it
closes the performance gap against a fully supervised setting without relying
on ground-truth annotations. We also compare various exploration policies for
the agent to gather more informative observations. Code and dataset will be
made available upon paper acceptance

###### Keywords:

Action-Perception; active exploration; self-training

##  1 Introduction

When newborns interact with novel objects, they are naturally inclined to
actively explore their hidden facets by manipulation or by moving around to
have views from a new angle, thus unveiling blindsides. Yet, a child is aware
that, no matter the viewpoint, the object is always the same. Findings in
social science suggest that humans and animals develop perception and learn by
actively exploring an environment [14] in a so called action-perception loop
[25], with very little or no external supervision. Inspired by human
perception, in this work we posit that the process of unsupervised fine-tuning
of object detectors can indeed benefit from an active exploration of the
environment (Figure 1). Such benefit can be further amplified if the agent can
actively learn how to choose the views it experiences, in order to make
learning more efficient.

Figure 1: We equip an agent with an off-the-shelf MaskRCNN [19] detector. The
agent explores a new environment and collects a set of noisy detections
$D_{0:N}$ following a trajectory $S_{0:N}$ resulting from a learned
exploration policy $\pi$. Such detections are then used for finetuning the
detector.

We suggest that mining training samples in the environment by seeing the same
object instances from different views may improve the agent’s object detection
skills. For this reason, we introduce a learnable policy that is indeed
oriented to the ultimate goal of improving object detection, without external
supervision, by directly rewarding the acquisition of uncertain samples. Our
proposal extends previous literature on environment exploration, which mostly
leverages reinforcement learning as a way of maximizing the total area
explored in the process [7], the novelty of the observation [29], or success
in reaching a predefined goal [45, 10].

A perception module (object detector) driven by active exploration is somehow
in contrast with the most classic paradigm of computer vision, where models
are usually trained with static datasets and do not fully benefit from the
realistic continuity of the 3D world. Our perception module adopts its
predictions as pseudo-labels to improve the performances with no external
supervision. Although self-supervised learning with multiple “views” of the
same image has been recently gaining popularity, such views are usually
obtained with a limited set of augmentations ([18, 17, 11]), which cannot
fully capture the complexity of a 3D object seen by different viewpoints.

The proposed pipeline includes learning both action and perception stages in a
completely self-supervised fashion (see Figure 2) and it assumes an agent
exploring an unknown environment using an off-the-shelf object detector
(MaskRCNN [19] in our experiments) to detect objects in the scene. Action: we
design and train a reinforcement learning policy with the objective of
maximizing the utility of the observations collected by the agent. We
hypothesize that the utility of a pseudo-label for an object is higher if that
object is predicted as belonging to different classes when seen from different
points of views. Building upon this intuition, we propose a score to measure
predictions disagreement as the average distance of the normalized logits, and
we reward the policy if it maximizes the disagreement score, which is
progressively gathered in a disagreement map. During the exploration phase, a
3D semantic voxel map is progressively built, which contains all the
predictions gathered during exploration. Such predictions are extremely noisy,
since the detector is far from perfect and images from the new scenario are
likely to be visually different from the training ones. Also, the same object
instance are predicted as belonging to different classes when seen from
different points of view, or even not detected at all in some images.
Perception: The predictions gathered during exploration are then used as
pseudo-labels to finetune the detector, in a three-fold manner. First, we
enforce the set of predictions to be consistent across multiple views via a
consensus mechanism. The refined pseudo-labels are assigned to all
corresponding images. This consensus is used as a feedback to fine-tune the
detector. Second, no matter the assigned pseudo-label, we force the model
backbone to map different views of the same object close in the feature space
while views of different instances are pushed afar, in a contrastive learning
fashion. Third, we exploit the soft targets obtained by averaging the multiple
prediction acquired for each instance as additional feedback, to fully exploit
the multiple acquired views.

Figure 2: Our approach consists of an action and a perception phase. Action:
(a) our policy predicts a long-term goal for the agent. (b) During the
exploration, the agent builds a semantically consistent voxel map of the
environment by projecting the predictions of its object-detection into a tri-
dimensional space. (c) We project the map onto top-down view and compute a
disagreement map by assigning disagreement score value to each cell. (d) The
disagreement map is the input of our policy network. Perception: (e) we
explore an environment via the learned policy and (f) project the semantic
voxel map onto each observation to build the pseudo-labels for the self-
training scheme. (g) Finally, we fine-tune the object detector relying only on
the pseudo-labels.

In summary, the contributions of this work are the following:

  * •

We revisit the Action-Perception paradigm to tightly couple action and
perception stages. Action and perception work synergically with the ultimate
goal of improving the object detector;

  * •

A novel disagreement map and reward for guiding the action phase. The action
policy learns how to gather observations which maximize disagreement, as a way
of mining hard training sample for the perception step;

  * •

A self-training perception module specifically designed to leverage the
inconsistent, noisy and contrasting pseudo-labels gathered during exploration;

  * •

A comparison between different action policies and perception modules.

Our experiments show that finding hard examples to feed the detector leads to
more informative pseudo-labels resulting in more accurate object detectors.

##  2 Related work

This paper aims at improving computer-vision models, such as object detectors,
by leveraging an embodied agent that actively explores an area. Object
detectors are indeed a key component for goal-oriented exploration [8, 5, 7]
and recent literature showed that object detection improvements lead to
reliable semantic mapping of the environment [8], with useful implications for
robotics exploration. The discussion proceeds as follows. This section first
highlights recent literature in pseudo-labeling and self-training. Next, we
focus on object detection for active perception and visual exploration.
Pseudo-labeling in object detection. Pseudo-labeling is a fundamental
technique in self-supervised learning. The approach is commonly defined as
adopting predictions from a neural-network as labels from unlabeled data [22].
Despite its effectiveness, simply providing neural-network outputs commonly
suffer from confirmation bias—network confidence improves even wrong
predictions—and high class imbalance [1]. Similar behaviors are commonly
observed in object detectors, where pseudo-labels are hard to exploit due to
high foreground-background imbalance [26]. Inspired by previous works [40, 39]
on consistency training, we propose to solve for the confirmation bias by
correcting the pseudo-labels through a consensus mechanism. In particular, our
approach leverages the 3D information of the environment to produce pseudo-
labels for every observations, even for observations where the model do not
provide predictions. This technique can mitigate the imbalance issue by
limiting the number of false negatives examples in training data.
Self-training. Our method shares similarities with the self-training paradigm.
During self-training, a model is trained jointly on labeled and unlabeled data
[22]. A pre-trained model is adopted to provide pseudo-labels for unlabeled
data. Recent literature suggests to generate pseudo-labels with noise-free
models and data while training with strong data augmentation and noise
injection (e.g., dropout) [1, 41, 12]. Li et al. [23] adapted self-training
via data augmentation to object detection by introducing a selective model to
provide good positive-negative pairs for object detection. Nevertheless, their
method adds overhead to the training, as the selective model must be trained
jointly with the object detector. Chaplot et al. [6] proposed an approach for
collecting consistent pseudo-labels in an environment. They build a semantic
map of the environment and re-project it onto each frame for getting
consistent pseudo-labels across views. Differently from them, we leverage the
inconsistency for helping the policy during the exploration phase and provide
this information to the object-detector.

We introduce instead a novel mechanism to leverage different points of view of
the same object as a data-augmentation strategy. We adopt a contrastive loss
[32] to impose that features for an object should be alike across views, while
different objects should be distinguishable in the feature space. In this
sense, we interpret our approach as a generalization of the cropping strategy,
which is quite popular in self-supervised learning [11].
Object detection for active visual exploration. Our proposal shares its
underlining principle with active perception, especially in the context of
visual exploration of the environment. Active perception can represent a broad
spectrum of concepts [3], but in the context of robotics it describes the
problem of actively moving in an environment to collect useful examples and
subsequently improve the performance on a given task.

Active perception has been extensively studied in the context of question and
answering [16, 36] and goal-oriented exploration [8, 5, 7, 15]. More recently,
there has been an interest in embodied perception for improving object
detection. Yang et al. [44] propose to merge FasterRCNN with an actor-critic
agent to control the viewing angle of a partially occluded object to improve
its recognition, using a custom dataset extracted from a simulator. Chaplot et
al. [9] proposed to learn exploration policies for online data collection.
They introduced an intrinsic reward based on discrepancy between different
detections projected to the ground plane. This does not fully address the
problem of objects overlapping when projected from their 3D pose to the ground
(e.g., a cup on a table). Chaplot et al. [6] train an exploration policy based
on the number of confident objects predictions. Nevertheless, their approach
does not consider the inconsistency of the detections as a viable training
signal.

Recent works also proposed alternative solutions to the problem which rely
only partially on semantic annotations [4, 27, 13]. These works rely on
ground-truth annotations and do not consider self-supervised settings where
labels are provided by a pre-trained model.

##  3 Methodology

We equip a simulated agent with an object-detector, a RGB-D sensor, and a
position sensor. The agent looks around and collects samples from a set of
simulated environments [38] in the Habitat simulator [35]. A policy sets the
long-term goals for the agent, which moves toward the goal by choosing actions
among moving forward, turn left, and turn right. We aim to fine-tune the
agent’s object-detector by relying only on the samples it generates
autonomously, i.e. without ground-truth annotations. As in previous literature
[9, 6], we adopt the object-detector MaskRCNN [19] with a Resnet50 backbone
[20]; the model is provided by [37] and is pre-trained on COCO dataset [24].
We now formalise and provide further details on the two stages of our
methodology: the action phase and the perception phase (Figure 2).
Action. During the action phase, we train a policy $\pi$ parametrised as a
neural network to predict long-term goals. We propose the use of a
disagreement reward to train the agent’s movement policy.

We first project the objects detected by the object detector into a semantic
voxel-map by leveraging the agent’s RGB-D and position sensors. For each
unique object instance, we compute a disagreement score as the cosine distance
among the logits predictions for that object (Equation 1). We aggregate the
disagreement scores into a 2D disagreement map, which we input into the
policy, and we reward the agent for maximizing their sum. Our intuition is
that, by maximizing this reward, the agent is encouraged to explore different
points of view of each object and find inconsistencies among the resulting
predictions. Next, a planner produces the set of optimal sub-goals to reach
the final goal. The agent takes actions toward each intermediate goal for
$N_{\text{replanning}}$ steps before predicting a new long-term goal. Section
3.1 discusses our policy and the disagreement map.
Perception. During the perception phase, we adopt the policy $\pi$ to collect
a sequence of states $S_{0:N}$ for each environment. Each state $S_{i}$
contains the input RGB-D image $x_{i}$ and MaskRCNN’s predictions $D_{i}$—a
set of bounding-boxes $b^{i}_{j}\dots b^{i}_{n}$, instance segmentation masks
$m^{i}_{1}\dots m^{i}_{n}$, and normalized logits
$\lambda_{1}\dots\lambda_{n}$. Then, we build a semantic 3D representation of
the environment. We propose a novel mechanism for producing consistent pseudo-
labels across different views of the same environment, while injecting the
original inconsistencies as a soft-target to further help training. Finally,
we leverage the rich information contained in our semantic point-cloud to
fine-tune the MaskRCNN model. In Section 3.2 we discuss the details for the
pseudo-labels cleaning and the self-training method.

###  3.1 Action: Policy learning

Figure 3 gives an overview of the policy training stage. All the steps are
detailed in the following.
Disagreement map. At each time step $t$, we build a _disagreement map_ $H_{t}$
as following. For each unique 3D instance object $u$ we compute a disagreement
score $s_{\text{disag}}$ as the average of the cosine distance between all the
logits assigned to that object:

$s_{\text{disag}}=\frac{1}{|Q(u)|^{2}}\sum_{\lambda_{a},\lambda_{b}\in
Q(u)}d(\lambda_{a},\lambda_{b}),$ (1)

where $Q(u)$ is the set of logits $\\{\lambda_{i}\\}_{i}$ assigned to the
unique instance object $u$ and $d(\lambda_{a},\lambda_{b})$ is the cosine
distance between two vectors. Next, we build $H_{t}$ as top-down map of the
environment. Each cell $(i,j)$ of the map contains the sum of the disagreement
scores of the objects mapped in top-down position $(i,j)$. Policy network. In
order to exploit the embodied nature of the agent, a suitable movement policy
must be devised for collecting informative new samples. The problem can be
formalized as a Markovian Decision Process and the objective becomes learning
a policy $\pi$ that, given the current observation $S_{t}$, predicts the next
action to take $a_{t}$. We parametrise the policy $\pi$ with a neural network.
The policy network takes as input the state of the agent:

$S_{t}=\\{S^{map}_{t}\in[0,1]^{2\times K\times K},e_{t}\\},$ (2)

where the first channel of $S^{map}_{t}$ contains the current disagreement map
$H_{t}$ with canonical dimension $K\times K$; the second channel contains the
map of the explored environment $M_{t}$ with the agent’s position superimposed
(also resized to dimensions $K\times K$); finally $e_{t}$ contains the global
orientation of the agent with respect to the map. The policy network encoder
is implemented in our experiments as a convolutional network, composed by 5 2D
convolutional layers. The agent’s orientation $e_{t}$ is fed through a linear
embedding layer and concatenated to the convolutional feature. This is then
followed by 2 linear layers. Based on the agent’s state, the policy predicts
an action $a_{t}=(x,y)$ that represents a new long-term goal position in the
map for the agent to explore. The goal is then fed to a graph-based path
planner ($A^{*}$) in order to compute a feasible path to the goal. In
particular, we first abstract the map to a visibility graph, where each node
of the graph is a reachable position in the map and each edge connects two
nodes if they are reachable in a straight line without collision with
obstacles. We then adopt the closest reachable node to the policy prediction
as the goal position. Graph-search is finally used to extract the shortest
node sequence that connects the agent to the goal. These nodes are then fed to
the agent that implements a simple local planner (Figure 3). After
$N_{\text{replanning}}$ steps, the policy network predicts a new long-term
goal.

Policy training.

We train the policy by implementing an actor-critic agent with PPO [33].

To address the issue of sparse rewards (such as the ones that could be
computed from opportunistic detection events), we take advantage of the
disagreement map and define a dense reward as the accumulated disagreement
over the whole environment as:

$r_{t}=\sum_{i,j}H_{t}[i,j],$ (3)

where $H_{t}[i,j]$ is the value of the $(i,j)$ cell of the disagreement map at
time $t$.

Figure 3: Overview of policy training. Given all previous detections, a
disagreement map is fed as input to the policy network together with the
agent’s pose in it. The policy predicts a new long-term goal in the map; a
traditional planner extracts a series of sub-goals to the predicted goal and
the agent moves to each sub-goal.

###  3.2 Perception: pseudo-labels and self-training

Pseudo-labels collected by an agent while navigating in a new environment
might be very noisy. Moreover, the agent may miss a detection if the object is
hard to recognize (e.g., if it sees the object from the back). Both scenarios
introduce inconsistencies in the training data that can compromise the model’s
performance if directly adopted for self-training.

Figure 4: Semantic voxel map creation and projection of detections onto 2D
frames. First, we aggregate the detection $D_{0},\dots,D_{N}$ into semantic
voxel-map. We solve the inconsistency of the voxel-map by assigning to each
voxel the class with maximum score among the predictions of the voxel. Next,
we project the semantic voxel-map back onto each observations, therefore
obtaining $\overline{D}_{0},\dots,\overline{D}_{N}$. $\overline{D}_{N}$ is the
consistent pseudo-label for observation $N$ and is obtained by reprojecting
the voxel-map onto RGB-D frame $x_{N}$. Each pseudo-label $\overline{D}_{i}$
is associated to an object instance via the identifier $u_{i}$ and contains
the consistent logits vector $\overline{\lambda}_{u_{i}}$.

We propose to leverage tri-dimensional projection to build a consistent voxel
map from noisy predictions as summarised in Figure 4. Building upon the
standard voxelization process, we introduce a consensus mechanism to improve
the quality of the voxel map. By projecting the voxel map back onto each
observation, we obtain consistent pseudo-labels that we employ our self-
training scheme.

During the exploration phase, we exploit depth and position sensors of the
agent to build a semantic point cloud by projecting the agent’s detector
predictions $D_{i}=(b_{i},m_{i},y_{i},\lambda_{i})$ (respectively, instance
segmentation masks, classes, and logits from the 2D detector) to 3D
coordinates. First, we voxelize the resulting point-cloud with
$\mathtt{voxel\\_size}=0.05m$. In the resulting voxel map, each non-empty
voxel can be associated with one or more predictions $D_{i}$. This results in
an inconsistent map, where a voxel is assigned to possibly multiple classes.
For this reason, we process the voxel map by keeping, for each voxel, a
consistent hard-label $\overline{y}$ and a consistent logits vector
$\overline{\lambda}$. We then compute the hard label $\overline{y}$ of a voxel
as the class with maximum score among all the predictions associated to that
particular voxel. We aggregate connected voxels with equal classes
$\overline{y}$ via a 26 connect-components algorithm [34]; it follows that
each 3D object instance is uniquely identified in the voxel map by an object
identifier $u$. The resulting voxel map is consistent, as every object
instance has a unique class $\overline{y}$. We highlight however that this
voxelization process is lossy, as we compute $\overline{y}$ via a max
operator, therefore losing the important information among inconsistent
logits. To overcome this issue, we compute for each object instance $u_{i}$ of
detection $i$ a consistent object logits $\overline{\lambda}_{u_{i}}$ as:

$\overline{\lambda}_{u_{i}}=\frac{1}{|\mathcal{Q}(u_{i})|}\sum_{\lambda_{j}\in
Q(u_{i})}\text{Softmax}(\lambda_{j}),$ (4)

where $\mathcal{Q}(u_{i})$ is the set of predicted vector logits
$\\{\lambda_{i}\\}_{i}$ for the object instance $u_{i}$ associated to
detection $i$. We further discuss the consistent vector logits in the
following Sections, where we underline the importance of this soft-target for
the fine-tuning of the detector.

By projecting back the semantic voxel map onto each observation $x_{i}$, we
produce a set of pseudo-labels $\\{\overline{D}_{i}\\}_{i}$. The projection
phase solves both issues of the agent’s predictions since: (i) as the voxel
map is self-consistent, it follows that pseudo-labels for an object instance
are consistent across different views; (ii) the consistency also applies for
frames where no detections were obtained at first. Our training set
$\mathcal{D}$ consists of the resulting set of observations and pseudo-
annotations
$\\{x_{i},(u_{i},\overline{\lambda}_{u_{i}},\overline{m}_{i},\overline{b}_{i},\overline{y}_{i})\\}_{i}$.
The first for-cycle in Algorithm 1 summarizes the collection phase.
Self-training phase. We propose a fine-tuning strategy for MaskRCNN’s head
which leverages the consistent pseudo-labels described above. Given the
training set $\mathcal{D}$, we first compute mask, bounding-boxes, and classes
losses $\mathcal{L}_{\text{head}}$ on pseudo-labels
$\overline{D}_{0},\dots\overline{D}_{N}$ as in [19]. To fully exploit the
information contained in our semantic voxel map, we propose two additional
step in the self-training strategy: instance-matching and soft distillation.
The second for-cycle in Algorithm 1 summarizes the fine-tuning method.

Initialize observation queue $Q$, detector’s box-head parameters $w$, policy
$\pi$; for  _each training environment_ do

Collect a trajectory $S_{0},\dots S_{N-1}$ following policy $\pi$ ;

Build semantic point-cloud from predictions $D_{0:N-1}$, depth information,
and position sensor readings;

Obtain a consistent point-cloud via voxelization and 3d connected components;

Produce hard labels, soft labels, and unique object identifiers
$\overline{D}_{0:N-1}$ by raycasting the consistent point-cloud onto each
observation $x_{0:N-1}$;

for  _K epochs_ do

Sample a mini-batch from Q ;

Predict the region proposals and compute $\mathcal{L}_{\text{rpn}}$;

Predict classes, bounding-boxes, and semantic masks, and compute
$\mathcal{L}_{\text{head}}$;

For each class prediction, compute $\mathcal{L}_{\text{distill}}$ by comparing
the predicted logits $\lambda_{i}$ with the consistent logits
$\overline{\lambda}_{u_{i}}$;

For each bounding-box prediction, extract the associated feature-vector and
compute $\mathcal{L}_{\text{im}}$, as in Figure 5;

Compute $\mathcal{L}_{\text{detection}}$ in Equation 5;

Aggregate $\mathcal{L}_{\text{rpn}}$ and $\mathcal{L}_{\text{detection}}$ and
update object detector parameters $w$;

Algorithm 1 Perception: our proposed self-training scheme. The perception
phase relies only on agent sensors and on a off-the-shelf object-detector. No
ground-truth labels are required. Figure 5: Our policy explores the
environment by maximising the disagreements between predictions for the same
object. The instance-matching loss exploits this behavior. In fact, it
enforces the feature vectors of the same object (e.g., $\text{u}_{k}$ in the
Figure) to be close in the feature space, while moving away feature vectors of
different objects (e.g., $\text{u}_{j}$ in the Figure).

Instance-matching. The first component of our strategy is a contrastive
triplet loss between detections. Intuitively, a contrastive loss helps
$f_{\text{box-head}}$ constructing an embedded feature-space where feature
vectors of the same objects are closer than feature vectors of different
objects, as shown in Figure 5. If an object is seen from extremely diverse and
challenging points of views, nearing the respective feature vectors is
beneficial for the model’s feature projector. Indeed, experiments in Section 4
highlight the benefits of combining our action phase with the instance
matching loss, as the proposed exploration phase acts as a strong data
augmentation for the object observations. Given a batch of observations
$x_{i}$, we predict the regions of interests $\text{ROIs}_{i}$ through
$f_{\text{ROI}}$ of MaskRCNN. Next, we provide the ROIs to the model’s head
$f_{\text{box-head}}$ to predict bounding-boxes $B\in\mathbb{R}^{k\times 4}$,
per-class logits $\lambda\in\mathbb{R}^{k\times|C|}$, and a set of feature
vectors $F\in\mathbb{R}^{k\times 1024}$. For each observation, we associate
the predictions with the corresponding pseudo-labels and assign their object
instance identifier to each feature vector. Next, for a given set of feature
vectors $F$, we compute positive and negative relationships by leveraging the
object identifiers and finally compute $\mathcal{L}_{\text{im}}$ by taking the
distance between positive and negative examples.
Consistent logits as soft pseudo-labels. As we generate class pseudo-labels by
taking the maximum class for each voxel, we lose the additional information
provided by the disagreement across views. Thus, we expect that the model can
benefit from an additional loss which guides the prediction toward a smooth
average of the logits provided by the off-the-shelf detector from different
views. The knowledge distillation loss–our second component–introduces the
consistent logits as a form of soft distillation. We compute
$\mathcal{L}_{\text{distill}}$ as the cross-entropy between the predicted
logits $\lambda_{i}$ and the consistent logits $\overline{\lambda}_{u_{i}}$.

Training. To summarize, we propose the combined loss
$\mathcal{L}_{\text{detection}}$ as:

$\mathcal{L}_{\text{detection}}=\mathcal{L}_{\text{im}}+\alpha\mathcal{L}_{\text{distill}}+\mathcal{L}_{\text{head}},$
(5)

by merging three different losses: (i) a triplet loss based on different views
of the same object $\mathcal{L}_{\text{im}}$, (ii) a soft distillation loss
$\mathcal{L}_{\text{distill}}$ for leveraging consistent logits
$\overline{\lambda}$, and (iii) mask, bounding-boxes, and classes losses
$\mathcal{L}_{\text{head}}$ on pseudo-labels. We proceed to fine-tune the
model by jointly minimizing $\mathcal{L}_{\text{detection}}$, and the standard
region proposal loss $\mathcal{L}_{\text{rpn}}$ introduced in [30].

##  4 Experiments

We devise a set of experiments to test policies and fine-tuning techniques
separately. For each policy described in Section 4.1, we let an agent move in
the environment and collect 7,500 frames from the Gibson dataset [38]. We opt
for MaskRCNN [19] as the object detector for being comparable with previous
approaches, although our proposed approach is model-agnostic.
Training details For the policy, we adopt Adam optimizer [21] with learning
rate $2.5e-5$, a discount factor $\gamma=0.99$, an entropy coefficient
$0.001$, value loss coefficient $0.5$, and replanning steps
$N_{\text{replanning}}=40$. For the self-training stage, we adopt SGD
optimizer with learning rate $\text{lr}=1e-4$, $\text{epochs}=10$,
$\text{weight-decay}=0.00001$, $\text{momentum}=0.9$ and $\text{batch
size}=16$. Intuitively, the soft-distillation loss
$\mathcal{L}_{\text{distill}}$ should not dominate over cross-entropy, thus
leading to incompatibilities between the two. We choose to set the soft-
distillation weight $\alpha=0.7$ by empirically comparing the gradients of the
two losses. We conduct additional ablation experiments in Section 1 to study
the impact of soft-distillation on the training. For the triplet-loss, we
adopt the default $\text{margin}=0.3$.
Evaluation To evaluate our approach, we sample 1,000 observations from each
test scene, which results in 4,000 testing samples. We adopt the mean Average
Precision with IOU threshold on bounding-boxes at 0.5 as evaluation score
[24]. We stress that no semantic annotations are required during training. We
introduce the baseline exploration and perception policies in Section 4.1.
Section 4.2 compares both our policy and our self-training approach with the
baselines as well as with supervised labels. For the latter, we use the
annotations provided by Habitat and originally proposed in Armani et al. [2].

###  4.1 Action and perception methods

Action. We compare our action module with classic and learned policies.
Random goals – implements a random policy based on long-term goals. The agent
chooses a random goal among the possible goals. Next, the agent moves to the
selected point via a global path planner that produces smooth trajectories.
Frontier exploration [43] – implements classical frontier-based active
exploration [43]. The agent keeps an internal representation of the explored
map, computes goal points of interest at the edge of the explored map, selects
the next best point according to its information gain (the amount of open
space around the goal point), and moves toward the goal via a global path
planner.
NeuralSLAM [7] – reinforcement learning policy based on long-term and short-
term goal prediction for maximising map coverage. A global policy predicts the
next long-term goal, while a local policy predicts a sequence of short-term
steps to move toward the final goal. A local path planner guides the agent
toward each short-term step. The reward is the percentage of exploration of
the environment.
Semantic Curiosity [9] – RL exploration policy that predicts local steps to
maximize inconsistency between detections projected onto the ground plane.
SEAL [6] – reinforcement learning policy for maximising the number of
confident predictions of the detector. The agent builds a voxel map of the
environment by leveraging the sensor information. A global policy processes
the current voxel map and position information and predicts the next long-term
goal, while a local path-planner guides the agent toward the goal. The reward
is the number of voxel assigned to any possible class with score above $0.9$.
Ours – our reinforcement learning policy, which maximizes the disagreement
across different views of the objects of the scenes, as discussed in Section
3.1.

Perception. We compare our perception module with two baselines:
Self-training [42] – the perception phase consists in adopting the prediction
of the off-the-shelf object detector as pseudo-labels for training, without
any further processing. We note that self-training often requires several
millions of images to provide strong benefits.
SEAL perception [6]– SEAL aggregates the predictions of the off-the-shelf
object-detector into a semantic voxel map by assigning to each voxel the class
with maximum score among the predicted classes. Then, the voxel-map back is
projected onto each observation. Finally, the detector is fine-tuned by
minimizing the classical MaskRCNN losses [19].
Ours – our perception module applies multi-view consistencies among
predictions while retaining the inconsistency information as soft-target and
imposing feature vectors of the same object to be close when the object is
seen from different views, as discussed in Section 3.2.

###  4.2 Results

To validate our method, we conduct experiments across different policies and
different perception methods. Table 1 reports our results. As a reference, we
assess that the off-the-shelf MaskRCNN [19] reaches mAP@50 $40.33$ on our test
set. Our perception method outperforms self-training [42] and SEAL’s
perception [6] in combination with random, frontier, and NeuralSLAM baselines.
The improvement is evident when we combine our self-training mechanism and our
policy: in this scenario, our action + our perception reaches the overall best
result of mAP@50 $45.65$, which is $5.2\%$ over off-the-shelf performance.
Even when using the random goals baseline, therefore without relying on any
predetermined behavior, our perception module improves over self-training by
$2.21\%$. When compared with exploration policies, both learned [7] and
classical [43] ones, our perception module outperforms SEAL perception. In
particular, we notice that a classical frontier exploration baseline
outperforms NeuralSLAM policy in our self-learning task. We believe NeuralSLAM
exploration collects fewer observations for each object instances. In a
similar scenario, where the variance of the observations is low, the semantic
voxel map reflects the labels predicted by the off-the-shelf detector. In
fact, we report similar results for all the perception methods when NeuralSLAM
is adopted. Semantic Curiosity predicts only local actions, which limits the
capacity of the policy to explore the environment and, therefore, the
collection of training data. On the other hand, a frontier exploration
baseline combined with our perception module achieves an AP of $43.09$. Our
intuition is that, due to the greedy exploration policy, the agent tends to
move through the same areas multiple times, collecting more views of the same
object, that are then cleaned by the consensus approach. SEAL perception
strategy slightly outperforms our perception when both are applied in
conjunction with SEAL policy. Intuitively, SEAL agent finds views of an object
with the highest prediction score, which could lead to less variance,
therefore vanishing the benefits of our consensus mechanism. Indeed we
underline that our action-perception loop outperforms SEAL’s one by $2.64\%$.
We provide more in-depth analysis of the policies behaviors in the
Supplementary Material.

Table 1: We compare mAP of our action and perception phases with the baselines. For the action, we compare our policy against classical (Random goals, Frontier [43]) and learned (NeuralSLAM [7], SEAL [6]) ones. Our action-perception loop outperforms the baselines and reaches the state-of-the-art result of mAP@50 45.65. As reference, off-the-shelf MaskRCNN reaches mAP@50 40.33. Policy | Self-training [42] | SEAL perc. [6] | Our perc. | Ground-truth
---|---|---|---|---
Random goals | 39.67 | 41.19 | 41.88 | 47.20
Frontier [43] | 40.18 | 41.98 | 43.09 | 45.06
NeuralSLAM [7] | 39.98 | 39.56 | 40.32 | 44.86
Sem. Curiosity [9] | 40.23 | 41.06 | 41.37 | 44.67
SEAL policy [6] | 39.33 | 43.01 | 42.38 | 44.57
Our policy | 38.50 | 44.50 | 45.65 | 46.59

When our policy is combined with self-training, our approach underperforms
with respect to the other policies. We underline that this result is expected.
Our policy, unlike others, explores the environment to purposely find
conflicting observations. Therefore, off-the-shelf predictions contain
disagreeing pseudo-labels, which leads to poor results. Both SEAL and our
perception are able to solve the inconsistencies. Nonetheless, we show that
our perception strategy fully exploits the inconsistencies and outperforms
SEAL perception by $1.15\%$.

We conduct an additional study by fine-tuning with ground truth labels, which
can be interpreted as an upper-bound of our approach. Not surprisingly, the
ground-truth labels outperform pseudo-labels for every policy. Nevertheless,
our action+perception scores the highest performance with respect to
supervised training, as it is only slightly worse than ground-truth.
Ablations We conduct ablations studies to further evaluate the components of
our perception module. In particular, we considered the impact of our
instance-matching loss on the overall performance and the importance of the
consistent logits for reaching the best performance.

Table 2: Ablation study on the impact of $\mathcal{L}_{im}$ and $\mathcal{L}_{distill}$. For $\mathcal{L}_{im}$, the performance improvement is significant if applied in conjunction with our policy, confirming the validity of our exploration objective. For $\mathcal{L}_{distill}$, improvements are more significant if the agent collects multiple and diverse predictions, as for our policy. Policy | $\mathcal{L}_{im}$ | $\mathcal{L}_{distill}$
---|---|---
w/o $\mathcal{L}_{im}$ | w/ $\mathcal{L}_{im}$ | w/o $\mathcal{L}_{distill}$ | $\alpha=0.1$ | $\alpha=0.7$ | $\alpha=1.0$
Random | 41.58 | 41.88 | 41.01 | 41.74 | 41.88 | 41.32
Frontier [43] | 42.32 | 43.09 | 42.23 | 42.29 | 43.09 | 43.08
NeuralSLAM [7] | 40.05 | 40.32 | 38.70 | 39.65 | 40.32 | 40.26
Sem. Curiosity [9] | 42.01 | 41.37 | 43.16 | 42.60 | 41.37 | 41.25
SEAL policy [6] | 42.35 | 42.38 | 42.72 | 42.28 | 42.38 | 41.16
Our policy | 44.19 | 45.65 | 41.09 | 44.55 | 45.65 | 41.45

Instance matching. As previous works showed in the classical contrastive
scenario [11, 12], contrastive approaches $\mathcal{L}_{\text{im}}$ improve
the results when hard positive examples are provided, e.g. when strong
augmentation is applied. In the active exploration scenario, we expect our
instance-matching method to benefit from an exploration that increases the
number of views for an object. Table 2 reports the results of applying
instance-matching in conjunction with different policies. We notice that the
object-detector benefits from the contrastive approach in conjuction with all
the policies. Moreover, when backed by our policy it shows the highest
absolute improvement, with an increase in mAP@50 by $1.4\%$.
Distillation loss. We ablate the contribution of the aggregated soft
consistent logits by ablating on different possible values of $\alpha$, which
regulates the weight of the distillation loss $\mathcal{L}_{\text{im}}$ and
thus the training signal derived by injecting the consistent logits
$\overline{\lambda}$ with a soft-distillation loss. We underline that all the
policies, with the exception of SEAL, improves their performance if we
introduce $\mathcal{L}_{\text{distill}}$, even when its weight is only
$\alpha=0.1$. As discussed in Section 3.2, we expect the soft-distillation to
improve the training when it does not exceed the cross-entropy loss for hard
pseudo-labels. We find that weighting $\mathcal{L}_{\text{distill}}$ with
$\alpha=0.7$ is indeed the best option: our policy + our perception
outperform the baselines and reach the state-of-the-art results. On the other
hand, $\mathcal{L}_{\text{distill}}$ damages the results when backed by SEAL
policy. This result is still not surprising. As discussed in Section 4, SEAL
policy reduces the variance of the detections and therefore cannot benefit
from injecting the consistent logits through $\mathcal{L}_{\text{distill}}$.

##  5 Conclusion

In this work we proposed a novel self-training mechanism for object detection
that relies on the active exploration of the environment. The object-detector
benefits from hard positive examples, e.g. where the off-the-shelf detector
fails. Namely, the perception phase can exploit disagreements among pseudo-
labels. Building upon this intuition, our contribution covers the whole
action-perception loop. From the action side, we design a reinforcement
learning policy to maximize the number of disagreeing examples, i.e. when the
object detector assigns different classes to the same object instance.

We propose to clean the inconsistencies by using a semantic voxel map by
accumulating each prediction into the 3D space, and in particular we further
inject the conflicting information among predictions into the training as
consistent logits vector for each object. Finally, we force the feature vector
of each object to be consistent across different views. Experiments show the
benefits of combining the proposed action and perception stages which
outperform the baselines and reaches state-of-the-art performance, improving
the off-the-shelf object-detector mAP by $5.3\%$.

## References

  * [1] Arazo, E., Ortego, D., Albert, P., O’Connor, N.E., McGuinness, K.: Pseudo-labeling and confirmation bias in deep semi-supervised learning. In: 2020 International Joint Conference on Neural Networks (IJCNN). pp. 1–8 (2020). https://doi.org/10.1109/IJCNN48605.2020.9207304
  * [2] Armeni, I., He, Z.Y., Gwak, J., Zamir, A.R., Fischer, M., Malik, J., Savarese, S.: 3d scene graph: A structure for unified semantics, 3d space, and camera. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 5664–5673 (2019)
  * [3] Bajcsy, R., Aloimonos, Y., Tsotsos, J.K.: Revisiting active perception. Autonomous Robots 42(2), 177–196 (2018)
  * [4] Cai, H., Mostofi, Y.: Exploiting object similarity for robotic visual recognition. IEEE Transactions on Robotics 37(1), 16–33 (2020)
  * [5] Cartillier, V., Ren, Z., Jain, N., Lee, S., Essa, I., Batra, D.: Semantic mapnet: Building allocentric semanticmaps and representations from egocentric views (2020)
  * [6] Chaplot, D.S., Dalal, M., Gupta, S., Malik, J., Salakhutdinov, R.: SEAL: Self-supervised embodied active learning using exploration and 3d consistency. In: Thirty-Fifth Conference on Neural Information Processing Systems (2021), https://openreview.net/forum?id=guHXB1dcD3l
  * [7] Chaplot, D.S., Gandhi, D., Gupta, S., Gupta, A., Salakhutdinov, R.: Learning to explore using active neural slam. In: International Conference on Learning Representations (2020), https://openreview.net/forum?id=HklXn1BKDH
  * [8] Chaplot, D.S., Gandhi, D.P., Gupta, A., Salakhutdinov, R.R.: Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems 33 (2020)
  * [9] Chaplot, D.S., Jiang, H., Gupta, S., Gupta, A.: Semantic curiosity for active visual learning. In: European Conference on Computer Vision. pp. 309–326. Springer (2020)
  * [10] Chen, A.S., Nam, H., Nair, S., Finn, C.: Batch exploration with examples for scalable robotic reinforcement learning. IEEE Robotics and Automation Letters 6(3), 4401–4408 (2021)
  * [11] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International conference on machine learning. pp. 1597–1607. PMLR (2020)
  * [12] Chen, T., Kornblith, S., Swersky, K., Norouzi, M., Hinton, G.E.: Big self-supervised models are strong semi-supervised learners. In: Advances in Neural Information Processing Systems. vol. 33, pp. 22243–22255 (2020)
  * [13] Fang, Z., Jain, A., Sarch, G., Harley, A.W., Fragkiadaki, K.: Move to see better: Self-improving embodied object detection. ArXiv (2021)
  * [14] Gibson, J.J.: The perception of the visual world. (1950)
  * [15] Giuliari, F., Castellini, A., Berra, R., Del Bue, A., Farinelli, A., Cristani, M., Setti, F., Wang, Y.: Pomp++: Pomcp-based active visual search in unknown indoor environments. arXiv preprint arXiv:2107.00914 (2021)
  * [16] Gordon, D., Kembhavi, A., Rastegari, M., Redmon, J., Fox, D., Farhadi, A.: Iqa: Visual question answering in interactive environments. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4089–4098 (2018)
  * [17] Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., Piot, B., kavukcuoglu, k., Munos, R., Valko, M.: Bootstrap your own latent - a new approach to self-supervised learning. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 21271–21284. Curran Associates, Inc. (2020)
  * [18] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9729–9738 (2020)
  * [19] He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn (2018)
  * [20] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015), http://arxiv.org/abs/1512.03385
  * [21] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (Poster) (2015), http://arxiv.org/abs/1412.6980
  * [22] Lee, D.H., et al.: Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In: Workshop on challenges in representation learning, ICML. vol. 3, p. 896 (2013)
  * [23] Li, Y., Huang, D., Qin, D., Wang, L., Gong, B.: Improving object detection with selective self-supervised self-training. In: European Conference on Computer Vision. pp. 589–607. Springer (2020)
  * [24] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014)
  * [25] Little, D., Sommer, F.: Learning and exploration in action-perception loops. Frontiers in Neural Circuits 7 (2013). https://doi.org/10.3389/fncir.2013.00037, https://www.frontiersin.org/article/10.3389/fncir.2013.00037
  * [26] Liu, Y.C., Ma, C.Y., He, Z., Kuo, C.W., Chen, K., Zhang, P., Wu, B., Kira, Z., Vajda, P.: Unbiased teacher for semi-supervised object detection. In: International Conference on Learning Representations (2021)
  * [27] Nilsson, D., Pirinen, A., Gärtner, E., Sminchisescu, C.: Embodied visual active learning for semantic segmentation. In: AAAI (2021)
  * [28] Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in pytorch (2017)
  * [29] Ramakrishnan, S.K., Jayaraman, D., Grauman, K.: An exploration of embodied visual exploration (2020)
  * [30] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems 28, 91–99 (2015)
  * [31] Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., Parikh, D., Batra, D.: Habitat: A platform for embodied ai research. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). p. nil (10 2019). https://doi.org/10.1109/iccv.2019.00943, https://doi.org/10.1109/iccv.2019.00943
  * [32] Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: A unified embedding for face recognition and clustering. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (Jun 2015). https://doi.org/10.1109/cvpr.2015.7298682, http://dx.doi.org/10.1109/CVPR.2015.7298682
  * [33] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms (2017)
  * [34] Smith, W.S.: connected-components-3d. GitHub. Note: https://pypi.org/project/connected-components-3d/ (2022)
  * [35] Szot, A., Clegg, A., Undersander, E., Wijmans, E., Zhao, Y., Turner, J., Maestre, N., Mukadam, M., Chaplot, D., Maksymets, O., Gokaslan, A., Vondrus, V., Dharur, S., Meier, F., Galuba, W., Chang, A., Kira, Z., Koltun, V., Malik, J., Savva, M., Batra, D.: Habitat 2.0: Training home assistants to rearrange their habitat (2021)
  * [36] Wijmans, E., Datta, S., Maksymets, O., Das, A., Gkioxari, G., Lee, S., Essa, I., Parikh, D., Batra, D.: Embodied question answering in photorealistic environments with point cloud perception. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6659–6668 (2019)
  * [37] Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R.: Detectron2. https://github.com/facebookresearch/detectron2 (2019)
  * [38] Xia, F., R. Zamir, A., He, Z.Y., Sax, A., Malik, J., Savarese, S.: Gibson env: real-world perception for embodied agents. In: Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on. IEEE (2018)
  * [39] Xie, Q., Dai, Z., Hovy, E., Luong, M.T., Le, Q.V.: Unsupervised data augmentation for consistency training. arXiv preprint arXiv:1904.12848 (2019)
  * [40] Xie, Q., Luong, M.T., Hovy, E., Le, Q.V.: Self-training with noisy student improves imagenet classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10687–10698 (2020)
  * [41] Xie, Q., Luong, M.T., Hovy, E., Le, Q.V.: Self-training with noisy student improves imagenet classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10687–10698 (2020)
  * [42] Yalniz, I.Z., Jégou, H., Chen, K., Paluri, M., Mahajan, D.: Billion-scale semi-supervised learning for image classification. arXiv preprint arXiv:1905.00546 (2019)
  * [43] Yamauchi, B.: Frontier-based exploration using multiple robots. In: Proceedings of the second international conference on Autonomous agents. pp. 47–53 (1998)
  * [44] Yang, J., Ren, Z., Xu, M., Chen, X., Crandall, D.J., Parikh, D., Batra, D.: Embodied amodal recognition: Learning to move to perceive objects. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2040–2050 (2019)
  * [45] Zhu, Y., Mottaghi, R., Kolve, E., Lim, J.J., Gupta, A., Fei-Fei, L., Farhadi, A.: Target-driven visual navigation in indoor scenes using deep reinforcement learning. In: 2017 IEEE international conference on robotics and automation (ICRA). pp. 3357–3364. IEEE (2017)

##  Appendix 0.A Introduction

In this supplementary material, we provide additional qualitative studies that
could not fit in the main paper. Our studies support further the intuition
provided in the experiments section, namely that the perception phase benefits
from actively find disagreeing views. Section 0.B shows qualitatively the
behaviors of the baselines, while Section 0.C discusses examples and failure
cases for our policy. Section 0.D concludes with additional implementation and
reproducibility details. We additionally provide a video demonstrating the
building process of the semantic voxel map and the disagreement map that are
discussed in the methodology section of the main paper.

##  Appendix 0.B Qualitative results on baselines policy

In this section, we discuss qualitatively how our policy compares to the
baselines. Figure 6 reports a trajectory sampled with each policy on an unseen
scene (Pablo) of the Gibson dataset [38]. Frontier and NeuralSLAM maximize
exploration directly, while SEAL’s reward is proportional to the number of
detected objects. On the other hand, our policy is rewarded for (i) maximising
the number of views of an object which could provide disagreeing predictions,
and (ii) finding new objects in the scene.

Although, al policies cover most of the explorable area of the scene, we
proved in the experiment sections that exploration is not sufficient for
improving object-detector performance. We speculate that training the policy
to maximize the number of objects (as in SEAL) or the disagreement between
views of objects (as in our policy) lead to well-behaved exploration while
benefiting the object-detector. Indeed, our experiments prove that our policy
outperforms other policies for the ultimate task of self-training the object
detector.

(a) Random goals

(b) Frontier exploration [43]

(c) NeuralSLAM [7]

(d) SEAL [6]

(e) Ours

Figure 6: We sample a trajectory of 300 steps for a scene that is not part of
the training scenes (Pablo [38]). The agent trajectory is shown as a red line.
Area covered by the agent is brighter than the background. All policies are
capable of exploring a good portion of the environment.

##  Appendix 0.C Exploration examples

In this Section we present a success and a failure example of our policy.
Figure 7 proposes a successful exploration of the environment. In particular,
our policy explores the scene while building an extensive disagreement map. On
the other hand, Figure 8 proposes a failure scenario for our policy. In the
latter, only a single object is detected. Our policy is still able to somewhat
explore the environment but remains close to the detected object, instead of
expanding the disagreement map. In fact, the disagreement map in Figure 8 is
empty apart from a single spot.

(a)

(b)

(c)

(d)

Figure 7: (a) Environment map with agent trajectory superimposed; (b) Explored
map; (c) Disagreement map; (d) RGB observation with predictions. Our policy
successfully explores the environment and builds an extensive disagreement
map. The agent visits each object multiple times, building its disagreement
map.

(a)

(b)

(c)

(d)

Figure 8: (a) Environment map. The object’s location is marked with a red
rectangle; (b) Explored map; (c) Disagreement map; (d) RGB observation. When
few to no objects are present in the scene, our policy is still able to
explore, but fails to build an extensive disagreement map.

##  Appendix 0.D Implementation details

This Section provides additional details regarding the implementation and the
reproducibility of our work. We adopted PyTorch [28] for the deep learning
components of our proposal, and Habitat [31] for simulating the agent’s
exploration. We provide the list of training and testing environments from the
Gibson dataset [38] in Table 3.

Random goals and frontier exploration [43] baselines were implemented from
scratch, while we adopted NeuralSLAM [7] pretrained policy model and code
base. Finally, we implemented and trained SEAL [6] from scratch, to the best
of our effort, following the details provided in the original paper. Table 4
shows additional implementation details for the policy network.

Both our policy and SEAL were trained for 100,000 steps. The training took
approximately 2 days on our internal cluster (2 NVIDIA GPUs V100 with 32 GB of
RAM). No training is required for random goals and frontier exploration [43]
policies.

Table 3: Composition of the Gibson [38] training and testing splits in our experiments. Dataset |  | Train split |  | Test split
---|---|---|---|---
Gibson |  |  | Allensville
---
Beechwood
Benevolence
Coffeen
Cosmos
| Forkland
---
Hanson
Hiteman
Klickitat
Lakeville
| Leonardo
---
Lindenwood
Marstons
Merom
Mifflinburg
| Newfields
---
Onaga
Pinesdale
Pomaria
Ranchester
| Shelbyville
---
Stockman
Tolstoy
Wainscott
Woodbine
|  | Collierville
---
Corozal
Darden
Markleeville

We adopted the MaskRCNN model [19] provided by Detectron2 [37] and pre-trained
on the COCO dataset [24]. MaskRCNN provides bounding-boxes and masks for each
prediction. We set the score threshold for object-detection predictions at
$0.7$. Similarly to previous works [9, 8], we focus on a subset of objects
which are common in houses and living spaces: toilet, couch, bed, dining
table, potting plant, and tv.

Map encoder
---
$[$ input $]$ Two stacked images: $B\times 2\times 256\times 256$
$[$ layer 1 $]$ Max pool 2.
$[$ layer 2 $]$ Conv. $3^{2}$, Stride $1^{2}$, Padding 1, ReLU activ.
$[$ layer 3 $]$ Max pool 2.
$[$ layer 4 $]$ Conv. $3^{2}$, Stride $1^{2}$, Padding 1, ReLU activ.
$[$ layer 5 $]$ Max pool 2.
$[$ layer 6 $]$ Conv. $3^{2}$, Stride $1^{2}$, Padding 1, ReLU activ.
$[$ layer 7 $]$Max pool 2.
$[$ layer 8 $]$ Conv. $3^{2}$, Stride $1^{2}$, Padding 1, ReLU activ.
$[$ layer 9 $]$ Conv. $3^{2}$, Stride $1^{2}$, Padding 1, ReLU activ.
Orientation encoder
$[$ input $]$ $B\times 1$
$[$ layer 1 $]$ FC 256.
Policy network
$[$ input $]$ Map feature $\oplus$ Orient. feature
$[$ layer 1 $]$ FC 512, ReLU activ.
$[$ layer 2 $]$ FC 256, ReLU activ.
$[$ layer 3 $]$ FC 1.

Table 4: Implementation details for the proposed policy. $B$ is the number of
concurrent episodes; $\oplus$ denotes a concatenation operation; Conv denotes
2D convolutional layer; FC denotes fully connected layer.

Licenses for referenced datasets:
Gibson: http://svl.stanford.edu/gibson2/assets/GDS_agreement.pdf

Reproducibility Checklist:

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [Yes]

    3. (c)

Did you discuss any potential negative societal impacts of your work? [N/A]

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A]

    2. (b)

Did you include complete proofs of all theoretical results? [N/A]

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
We provide detailed information regarding our implementation. Code will be
released upon acceptance.

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes]

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experi- ments multiple times)? [No] We run all the experiments with a single
randomly chosen seed. Each experiments includes action and perception phases
and is computationally demanding. We provide extensive results in support of
our method.

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes]

    2. (b)

Did you mention the license of the assets? [Yes]

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [Yes] Test set and collected observations will be provided upon
acceptance

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [N/A]

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [N/A]

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]

    4. (d)

Did you discuss any potential negative societal impacts of your work? [Yes]

    5. (e)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]