# Self-Supervised Visual Place Recognition
by Mining Temporal and Feature Neighborhoods

Chao Chen1, Xinhao Liu1, Xuchu Xu1, Yiming Li1, Li Ding2, Ruoyu Wang1, and
Chen Feng✉,1
✉ Corresponding author.1Chao Chen, Xinhao Liu, Xuchu Xu, Yiming Li, Ruoyu
Wang, and Chen Feng are with New York University, Brooklyn, NY 11201, USA
<EMAIL_ADDRESS>Ding is with University of Rochester, Rochester, NY 14627,
USA<EMAIL_ADDRESS>
###### Abstract

Visual place recognition (VPR) using deep networks has achieved state-of-the-
art performance. However, most of them require a training set with ground
truth sensor poses to obtain positive and negative samples of each
observation’s spatial neighborhood for supervised learning. When such
information is unavailable, temporal neighborhoods from a sequentially
collected data stream could be exploited for self-supervised training,
although we find its performance suboptimal. Inspired by noisy label learning,
we propose a novel self-supervised framework named TF-VPR that uses temporal
neighborhoods and learnable feature neighborhoods to discover unknown spatial
neighborhoods. Our method follows an iterative training paradigm which
alternates between: (1) representation learning with data augmentation, (2)
positive set expansion to include the current feature space neighbors, and (3)
positive set contraction via geometric verification. We conduct comprehensive
experiments on both simulated and real datasets, with either RGB images or
point clouds as inputs. The results show that our method outperforms our
baselines in recall rate, robustness, and heading diversity, a novel metric we
propose for VPR. Our code and datasets can be found at
https://ai4ce.github.io/TF-VPR/.

##  I Introduction

Visual place recognition (VPR), which aims to identify previously visited
places based on current visual observation, is a well-known problem in
computer vision and plays a crucial role in autonomous robots [1]. Meanwhile,
VPR is closely related to re-localization [2], loop closure detection [3], and
image retrieval [4]. Despite all the efforts, VPR remains a difficult task due
to various challenges such as perceptual aliasing and view direction
differences [5, 6]. Classic VPR methods based on hand-crafted feature matching
do not require supervised learning, but are less robust to the challenges
mentioned above [7, 8, 9]. Therefore, learning-based methods have been
proposed to learn local or global feature descriptors [10] by place
classification [11] or contrastive-like similarity learning [12]. Some works
also use a short image sequence instead of a single image to mitigate the
issue of perceptual aliasing [13, 14, 15].

So far, most learning-based VPR methods are supervised, focusing on either
learning better feature representations or designing robust matching
strategies. They assume that accurate positions (and sometimes orientations)
are available in their training set, for obtaining the positive and negative
neighbors of each visual observation [12, 10], or defining place categories
[11]. However, this information could be onerous to obtain due to GPS errors
and its indoor unavailability or SLAM robustness challenges, especially at a
large scale. Considering human’s extraordinary VPR ability that does not seem
to need ground truth pose information for its training, we ask the following
question: is it possible to relax such an assumption and design a learning-
based VPR approach without pose-dependent supervision?

(a) A partial sensory stream & 3 types of neighbors (red/blue/green circles).

(b) A spatial neighborhood’s iterative update.

Figure 1: Our idea is based on the interconnections between the temporal,
spatial, and feature neighborhoods in sensory data: a query’s spatial
neighborhood expands from its temporal to feature neighbors, then contracts to
exclude wrong neighbors, iterated in training until such neighborhoods’
convergence.

To achieve this goal, our main idea is to leverage fixed temporal
neighborhoods and learnable feature neighborhoods to discover the unknown
spatial neighborhoods (which require ground truth pose to compute), leading to
a self-supervised VPR method shown in Fig. 1. We are inspired by research work
utilizing sensory streams (RGB videos or point cloud sequences) to obtain the
positive and negative neighbors in the temporal domain such as [16]. However,
we find that VPR learned from temporal cues alone will miss spatial
neighboring places with large viewpoint differences, because temporal
neighbors tend to share similar viewpoints.

This is suboptimal in applications such as visual navigation or loop closure
for SLAM. To automatically discover the true spatial neighbors with diverse
viewpoints, we propose a novel iterative learning strategy inspired by noisy
label training such as bootstrapping [17]. More specifically, we exploit the
temporal information for label initialization as shown in Fig. 1(a). We
further augment the labels by simulating observations perceived from different
view directions. Afterward, a feature representation will be learned based on
the current labels. Finally, as shown in Fig. 1(b), we re-label the dataset
using the learned feature space by: (1) adding feature-space neighbors as
tentative positive labels and (2) rejecting false positives via geometric
verification, in order to further refine the feature representation. Note that
the above steps are iteratively conducted until convergence.

To evaluate our self-supervised VPR with Temporal and Feature neighborhood
interactions (TF-VPR), we simulate datasets with different input modalities,
and develop a real-world dataset NYU-VPR-360 including two scenes with 38,426
GPS-georeferenced images in total. All the datasets are sequentially-collected
sensory streams. Meanwhile, we develop a novel metric to measure the heading
diversity of the retrieval results. In summary, our contributions are:

  1. 1.

We propose a novel self-supervised VPR solution termed TF-VPR that eliminates
pose-dependent supervision by mining temporal and feature neighborhoods.

  2. 2.

We propose a new evaluation metric to assess the heading diversity of VPR
retrieval results.

  3. 3.

We conduct comprehensive experiments in both simulation and real-world to
demonstrate the advantages of our solution compared with other baseline
methods. Our codes and data will be released with this paper.

##  II Related Work

Visual place recognition. Visual place recognition (VPR) is the problem of
identifying a previously visited place based on visual information [5].
Existing VPR methods mainly lie in two categories: (1) traditional VPR
techniques using hand-crafted features [18, 19, 20, 21, 22, 23], and (2)
state-of-the-art VPR techniques using deep learning [12, 24, 25, 26, 27].
Among those, NetVLAD [12] is a seminal deep-learning-based VPR framework,
followed by various research extensions such as (1) learning powerful feature
representation [28, 29], (2) designing robust matching strategies [13, 14,
15], and (3) investigating different input modalities in VPR [30, 31, 32, 33].
However, most methods are supervised by pose-dependent data [24, 25, 12, 26].
To relax such a constraint, several attempts have been made [34, 35, 36, 37],
yet failed to handle diverse re-visiting viewpoints. The most relevant work to
our method is the semi-parametric topological memory (SPTM) [16], which
utilizes temporal positives and negatives to train a binary classification
network for adding edges for topological mapping, similar to VPR. However,
since temporal neighbors tend to have very similar viewpoints, SPTM still
struggles to recognize revisits of the same place from different viewpoints.
To the best of our knowledge, no research has addressed self-supervised VPR
that can recognize places observed from various viewpoints as either 2D images
or 3D point clouds.

Noisy label learning. Noisy labels become a problem as training data size
increases, resulting in degraded performance [38]. To mitigate the issues of
noisy labels, several learning attempts [39] have been made from directions
including latent variable optimization [40], loss function design [41], and
pseudo-label-based self-training [42, 17, 43, 44]. Among all pseudo-label
methods, label refurbishment was firstly introduced by Bootstrapping [17].
Later on, another method addressed this problem using a self-training-based
approach with an iterative workflow [43] which was used as a baseline
architecture in their more recent framework confidence regularized [44]. Our
method’s iterative training paradigm takes inspiration from [43, 44].

Contrastive learning. It is a self-supervised learning technique to find
feature representations that differentiate similar data pairs from dissimilar
ones without labels. Data augmentation is often used, and the learning
objective is to decrease feature distances between the original and augmented
images (positive samples), while increasing those distances between different
images (negative samples) [45, 46, 47, 48]. In VPR, NetVLAD [30] uses the
triplet loss which is similar to contrastive learning, yet relies on ground
truth pose to define positive/negative samples. Building on NetVLAD, we
further adopt data augmentation to synthesize observations captured by
different viewpoints in order to learn more robust features for VPR.

VPR evaluation. There are several evaluation metrics for visual place
recognition, e.g., the popular AUC-PR [1] provides a good overview of
precision and recall performance but is less indicative in the cases when
ground truth match could take multiple values. Recall Rate@N, as used in [49,
50, 12], is designed to address such cases that the correct retrieval may be
in the top-N results, and multiple correct query matches are neither penalized
nor rewarded. However, existing VPR metrics rarely evaluate the viewpoint
diversity of the retrieved results [5], which is important in downstream
applications such as SLAM. In this work, we develop such a metric to fill this
gap. It assesses a VPR model’s capacity to recognize places revisited from
different directions.

##  III Methodology

Figure 2: Overview of TF-VPR. Labeling, training, expansion, and contraction
are four major steps in our approach. Labels can be refurbished by iteratively
learning feature representations (training), adding feature neighborhoods
(expansion), and removing false positives by geometric verification
(contraction). Initial labels are generated by temporal adjacency.

We focus on a robot that is collecting a sensory stream of surround-view
visual observations while navigating in a certain area. Our goal is to enable
the robot to achieve visual place recognition (VPR) in the same spatial area
where the data stream is collected, without relying on any frame-wise pose
information. To this end, we utilize a learnable neural network $f_{\theta}$
parameterized by $\theta$ to map each visual observation to discriminative
feature space for VPR. However, it is non-trivial to train such a neural
network without ground-truth labels. In this work, we propose a novel
iterative learning paradigm based on the following intuition: for the $i$-th
query observation ${\bf{\mathbf{q}}}_{i}$, its spatial positive set
$\mathcal{P}_{{\bf{\mathbf{q}}}_{i}}$ could be inferred by its temporal
positive set $\mathcal{P}^{t}_{{\bf{\mathbf{q}}}_{i}}$ together with its
tentative feature-space positive set
$\mathcal{P}^{f}_{{\bf{\mathbf{q}}}_{i}}$.

Our objective: auto-labeling with self-supervised learning. Different from
existing supervised VPR methods that need to address the generalizability of
learned models, we aim to solve VPR only for a certain spatial area where the
data has been collected. Given an observation sequence, we want to
automatically label each data frame’s spatial topology without ground truth
poses for supervision. Similar to DeepMapping [51], we do not expect the
learned $f_{\theta}$ to generalize its VPR ability either to other areas or to
the same area but under different times/seasons/weather conditions than what
has been covered in the training dataset. Note that this is non-trivial and
useful in real-world, because (1) ground truth pose information is not always
easy to obtain, and (2) achieving our objective would enable auto-labeling of
large-scale datasets without relying on ground truth pose information, which
can be further used to provide training dataset for supervised VPR methods.

###  III-A Overall framework

Given a sensory stream $\\{{\bf{\mathbf{o}}}_{i}\\}_{i=1}^{I}$ which can cover
the entire spatial area (each observation denoted by ${\bf{\mathbf{o}}}_{i}$
is either an RGB image or a LiDAR point cloud), we aim to discover the spatial
connectivity for these observations without any spatial information.
Specifically, we propose a self-supervised learning paradigm that utilizes the
interconnection of spatial, temporal, and feature neighborhoods to iteratively
refine the noisy pseudo-labels as well as the feature representations, as
shown in Fig. 2. The final output is the discovered spatial connectivity for
the given observation sequence, as well as the learned neural network
$f_{\theta}$ which can map each observation ${\bf{\mathbf{o}}}_{i}$ to a
discriminative feature space that supports VPR. Note that the optimized
$f_{\theta}$ can also discover the spatial connectivity for a new observation
sequence collected in the same spatial area: for each query observation
${\bf{\mathbf{q}}}_{i}$, a set of nearest feature neighbors
$\mathcal{P}^{f}_{{\bf{\mathbf{q}}}_{i}}=\\{{\bf{p}}_{j}\\}_{j\neq i}$ which
can approximate the spatial neighbors is retrieved based on the Euclidean
distance in the learned feature space:
$d\left({\bf{\mathbf{q}}}_{i},{\bf{p}}_{j}\right)=||f_{\theta}\left({\bf{\mathbf{q}}}_{i}\right)-f_{\theta}\left({\bf{p}}_{j}\right)||$.
The four sub-modules are detailed next.

###  III-B Label initialization

Temporal adjacency between any two data frames generally implies their spatial
adjacency, but not vice versa [16]. Therefore, we could utilize temporal
adjacency to generate noisy labels for each observation. Note that the term
noisy means that such labels may include false negatives, e.g., it would
overlook positive labels for observations that are spatially adjacent yet
temporally distant. In this work, the labels are initialized solely with the
knowledge of temporal adjacency: given a query ${\bf{q}}_{i}$, the temporal
positive set is
$\mathcal{P}^{t}_{{\bf{\mathbf{q}}}_{i}}=\\{{\bf{p}}_{j}\\}_{|i-j|<n}$, and
the temporal negative set is
$\mathcal{N}^{t}_{{\bf{\mathbf{q}}}_{i}}=\\{{\bf{n}}_{v}\\}_{|i-v|>u\times
n}$, where $i,j,v$ denote the frame indices of the observations, $n$ controls
the size of temporal positives, $u$ controls the range of temporal negatives.
After initialization, the feature will be refined iteratively by inferring
spatial neighborhoods according to the known temporal neighborhoods and
current feature neighborhoods.

###  III-C Training

In order to learn a more robust feature representation for recognizing places
viewed from diverse directions, we augment the training set by simulating
various sensor headings, i.e., an observed panoramic image/point cloud is
randomly rolled/rotated around the sensor’s vertical axis. Meanwhile, the
pseudo ground-truth for the $e$-th training epoch includes spatial positives
denoted by $\mathcal{P}_{{\bf{\mathbf{q}}}_{i}}^{(e)}$ and spatial negatives
denoted by $\mathcal{N}_{{\bf{\mathbf{q}}}_{i}}^{(e)}$, both for the $i$-th
query ${\bf{q}}_{i}$. Note that they are initialized by
$\mathcal{P}^{t}_{{\bf{\mathbf{q}}}_{i}}$ and
$\mathcal{N}^{t}_{{\bf{\mathbf{q}}}_{i}}$ for $e=1$ and will be iteratively
updated. For the training objective, we employ weakly supervised ranking loss
denoted by $L_{\theta}$ for the $i$-th training tuple
$({\bf{q}}_{i},\\{{\bf{p}}_{j}\\},\\{{\bf{n}}_{v}\\})$, following [12]:

$L_{\theta}=\sum_{i}\sum_{v}l\left(\min_{j}d\left({\bf{q}}_{i},{\bf{p}}_{j}\right)+m-d\left({\bf{q}}_{i},{\bf{n}}_{v}\right)\right),$
(1)

where ${\bf{p}}_{j}\in\mathcal{P}_{{\bf{\mathbf{q}}}_{i}}^{(e)}$ and
${\bf{n}}_{v}\ \in\mathcal{N}_{{\bf{\mathbf{q}}}_{i}}^{(e)}$, $d$ is the
Euclidean distance in feature space, $l$ is the hinge loss: $l(x)=\max(x,0)$,
and $m$ defines the margin (set to $0.2$ in this paper).

###  III-D Expansion

At the initial stage (epoch $e=1$), the current positive label set
$\mathcal{P}_{{\bf{\mathbf{q}}}_{i}}^{(e)}$ has a limited size since we only
incorporate the temporal positives. Actually, there could be a number of
spatial positives in $\mathcal{N}_{{\bf{\mathbf{q}}}_{i}}^{(e)}$ which are
temporally distant yet spatially adjacent to the query. To improve the labels,
we need to expand our positive set $\mathcal{P}_{{\bf{\mathbf{q}}}_{i}}^{(e)}$
by including current feature space neighbors. Specifically, once the
$f_{\theta}$ is trained for one epoch with current labels, the $K$ nearest
neighbors (KNN) could be retrieved as the feature neighbors.

Dynamic KNN. However, it is hard to use the same $K$ for different query
observations, because places like intersections naturally have larger spatial
neighborhoods than hallways. Thus, we need a frame-specific threshold. To
design such a mechanism, we observe the following hypothesis: if an
observation is closer to the query than at least one of the query’s temporal
neighbors in the feature space, then such an observation is more likely to be
a ground truth spatial neighbor of the query. Therefore, we firstly select $K$
nearest feature neighbors as the feature neighborhood candidates
$\\{{\bf{\mathbf{o}}}_{k}\\}_{k=1}^{K}$. Then we utilize the maximum feature
distance between the query and its temporal neighbors as a frame-specific
threshold $\tau_{i}$ to determine which candidates should be included in the
expanded positive set:

$\tau_{i}=\max\limits_{{\bf{p}}_{j}\in\mathcal{P}^{t}_{{\bf{\mathbf{q}}}_{i}}}d\left({\bf{q}}_{i},{\bf{p}}_{j}\right),$
(2)

where $\mathcal{P}^{t}_{{\bf{\mathbf{q}}}_{i}}$ is the temporal neighborhood
set for query ${\bf{q}}_{i}$, $i$ and $j$ are the frame indices. Finally, any
candidate ${\bf{o}}_{k}$ (selected from KNN) with a smaller feature distance
to the query than the threshold $\tau_{i}$ will form the feature neighborhood
set:
$\mathcal{P}^{f}_{{\bf{\mathbf{q}}}_{i}}=\\{{\bf{\mathbf{o}}}_{k}\\}_{d\left({\bf{q}}_{i},{\bf{\mathbf{o}}}_{k}\right)<\tau_{i}}$.

###  III-E Contraction

To avoid potential false positives caused by noisy feature space during
learning, we employ geometric verification to check the validity of feature
neighborhoods $\mathcal{P}^{f}_{{\bf{\mathbf{q}}}_{i}}$ before using it to
update $\mathcal{P}_{{\bf{\mathbf{q}}}_{i}}^{(e)}$. We adopt different
distance measures (independent of the neural network) based on input modality:
the number of matching points for images [52] and the Chamfer distance after
ICP for point clouds [53]. The verification strategy is similar to the one for
KNN feature neighborhoods verification: we consider the maximum distance
between the query and its temporal neighborhoods as the threshold. Then any
candidate with a distance value within the threshold will pass the
verification. The verified feature neighborhoods denoted by
$\hat{\mathcal{P}}^{f}_{{\bf{\mathbf{q}}}_{i}}$ are permanently added into the
positive set:

$\mathcal{P}^{(e+1)}_{{\bf{q}}_{i}}=\mathcal{P}^{(e)}_{{\bf{q}}_{i}}\cup\hat{\mathcal{P}}^{f}_{{\bf{q}}_{i}},$
(3)

where $e$ is the number of learning epochs. Note that
$\mathcal{N}_{{\bf{\mathbf{q}}}_{i}}^{(e)}$ will also be updated similarly.
Afterward, the model will be trained with the updated labels until
convergence.

##  IV Experiments

We test TF-VPR in three different kinds of environments: simulated 2D point
clouds [51], simulated RGB images [54], and real-world RGB images. Our
codebase uses PyTorch [55] with network parameters optimized using Adam [56].
The learning rate is tuned to 0.001. We compare TF-VPR with both supervised
and self-supervised baseline methods.

###  IV-A Evaluation metrics

Recall rate (Recall@N) is the ratio of successful retrievals to a total number
of queries, where a successful retrieval means at least one of the top-N
retrieved results is also the ground-truth spatial neighbor of the query.
Ground truth can be obtained by K-D tree search on geographical location
$(x,y,z)$ within a certain radius $R$. Note that when computing this metric,
we need to exclude the temporal neighbors of a query from its top-$N$
retrievals. This is because in our auto-labeling setup, the temporal-based
methods can easily overfit the temporal neighborhood, leading to uninformative
evaluation with recall rates that are always close to $100\%$ if the temporal
neighbors are kept in ground truth.

Heading diversity (HD) measures the diversity of sensor headings of the true
positives (w.r.t. the query’s heading) among the top-$|GT|$ retrievals. $|GT|$
is the size of ground-truth set that can be obtained based on a specific
radius $R$ as described in Recall@N. First, we believe that those positives
with headings different than the query are more valuable in downstream
applications. Thus, we evenly divide the 360∘ range of headings into 8 angular
bins in our setup, ignoring the first and last bins because they contain
positives with similar headings w.r.t. the query. In this case, the $m$-th bin
covers the heading difference range ${\mathcal{Q}_{m}}$ as:

$\mathcal{Q}_{m}=[m\times 45^{\circ},(m+1)\times
45^{\circ}],\,m\in[1,2,3,4,5,6].$ (4)

Then, we define HD for a query $\bf{q}$ as the bin coverage ratio between the
true positives and the ground truth:

$\text{HD(\bf{q})}=\frac{\sum_{m\in{[1...6]}}\mathds{1}(\exists\bf{x}\in\tilde{\mathcal{P}}^{|GT|}_{{\bf{\mathbf{q}}}}\land(\theta_{\bf{q}}-\theta_{\bf{x}})\in\mathcal{Q}_{m})}{\varepsilon+\sum_{m\in{[1...6]}}\mathds{1}(\exists\bf{y}\in\mathcal{P}^{|GT|}_{{\bf{\mathbf{q}}}}\land(\theta_{\bf{q}}-\theta_{\bf{y}})\in\mathcal{Q}_{m})},$
(5)

where $\theta_{\bf{q}}$ and $\theta_{\bf{x}}$ are respectively the heading of
the query and a frame ${\bf{x}}$,
$\tilde{\mathcal{P}}^{|GT|}_{{\bf{\mathbf{q}}}}$ is the set of true positives
in the top-$|GT|$ retrievals, $\mathcal{P}^{|GT|}_{{\bf{\mathbf{q}}}}$ is the
ground truth positive set, $\varepsilon$ is an arbitrarily small positive
quantity to avoid zero division error, and $\mathds{1}(\cdot)$ is the
indicator function. See Fig. 3 for an example. Finally, we report the averaged
HD for all queries.

Figure 3: Heading diversity illustration. The angle represents the heading
difference between the query and the evaluated positives. HD represents how
many angular bins are covered by true positives vs. that by the ground truth.
The figure gives an example of how to calculate HD. Excluding the first and
last bins, $\mathcal{P}^{f}_{{\bf{\mathbf{q}}}_{i}}$ contains 5 retrieved non-
temporal positives, 4 of which are true positives, and they fall into 3
different bins, while ground truth covers 5 bins, so HD is $3/5$ (not $4/5$).

###  IV-B Experiments on simulated point cloud dataset

Dataset. We generate the 2D point cloud dataset using the tool provided in
[51]. Specifically, we create a large environment as a $1024\times 1024$
binary image in which black and white pixels respectively represent the
occupied and free-space locations in the 2D environment. We then manually
generate a set of trajectories in this environment, each of which contains a
sequence of $2048$ poses. At each pose, a point cloud scan is simulated by
finding the intersection points between 2D LiDAR rays and occupied space in
the environment. The simulated 2D point cloud dataset contains $3$ different
environments and $18$ trajectories in each environment. Each scan contains
$256$ points.

Baseline methods. We compare TF-VPR with the following baselines: (1)
PointNetVLAD [30] trained with pose-based supervision, (2) PointNetVLAD
trained with temporal pseudo labels as in [16] (SPTM), (3) SPTM with data
augmentation (SPTM+A), (4) SPTM with feature-neighbor KNN expansion
(SPTM+F(K)) , and (5) SPTM+A+F(K). Note that the contraction step is not used
for this simulated toy dataset, because we found the expansions are all
correct in this case.

Implementation details. The network architecture for the SPTM baseline is the
same as PointNetVLAD. We select $n=5$ and $u=2$ in Section III-B. The value of
$n$ depends on the sampling rate of the sensor. Additionally, similar to the
training in [30], we randomly select only $2$ positives and $18$ negatives to
speed up the loss computation. The output feature dimension is $512$. Based on
the sampling rate of the sensor, for each query, we exclude its closest $10$
temporal neighbors from the top-N retrievals as explained in IV-A.

Figure 4: Qualitative VPR results on 2D point cloud dataset. The first row
shows the results of SPTM [16] and the second row shows the results of TF-VPR.
The first column is the query point cloud and column 2-5 are the top 4
retrievals. Green and red respectively indicate true and false positives.
Figure 5: Recall@10 and HD with respect to training epoch on 2D point cloud
dataset. F denotes method with feature neighborhoods, A denotes method with
data augmentation, (K) denotes method with KNN during expansion, (D) denotes
method with dynamic KNN in expansion. Figure 6: Per-frame performance
visualization over ground truth trajectories on point cloud (top), simulated
RGB (middle), and NYU-VPR-360 (bottom) datasets. The left block shows the
Recall@10 and the right block shows the HD metrics. Each query’s metrics at
epoch 30 are color-coded.

Data augmentation. Fig. 5 shows a significant improvement in both recall@10
and HD after augmentation is added to SPTM. SPTM fails to retrieve the true
neighbors from different headings, leading to insufficient retrievals.
Differently, SPTM+A uses augmented positives in the triplet loss, leading to a
network that is insensitive to the orientation of the point cloud inputs.
Thus, in Fig. 4, with augmentation, TF-VPR demonstrates its capability to
retrieve positives from diverse directions.

Feature-neighbor expansion. SPTM+F(K) has slightly poorer performance than the
original SPTM as shown in Fig. 5, because only adding positives from the same
direction does not provide extra valuable training data. In comparison, SPTM
with data augmentation and feature space neighborhood (SPTM+A+F(K)) helps
feature-neighbor expansion more effectively discover true neighbors from
different headings, thereby improving performance.

Dynamic KNN. We further study the effectiveness of dynamic KNN compared to
fixed KNN in TF-VPR, i.e., SPTM+A+F(D) vs. SPTM+A+F(K). The difference is
negligible on the simulated point cloud dataset. However, the difference would
be larger on NYU-VPR-360 dataset.

Performance difference between TF-VPR and other baselines. In Table I, among
all of these methods, TF-VPR shows a comparable result of recall rate and
heading diversity to PointNetVLAD, and outperforms SPTM by a large margin.
Moreover, Fig. 6 visualizes the per-frame retrieval quality over the datasets’
ground truth trajectories, and SPTM clearly performs the worst with more
failures.

TABLE I: Quantitative results on 2D simulated point cloud data. We employ best
recall rate (R) and heading diversity (HD).

Method / Metric |  | R@10 | R@5 | R@1 | HD
---|---|---|---|---|---
SPTM [16] |  | 93.36 | 91.06 | 79.10 | 4.25
TF-VPR (Ours) |  | 99.71 | 99.66 | 98.73 | 93.62
PointNetVLAD [30] |  | 100.00 | 100.00 | 100.00 | 89.38

###  IV-C Experiments on photorealistic RGB dataset

TABLE II: Quantitative results on photorealistic RGB data. We employ best
recall rate (R) and heading diversity (HD). We report results on three rooms
in habitat-sim (Goffs, Micanopy, and Spotswood).

Scene | Goffs | Micanopy | Spotswood
---|---|---|---
Metric | R@10 | R@5 | R@1 | HD | R@10 | R@5 | R@1 | HD | R@10 | R@5 | R@1 | HD
SPTM [16] | 95.52 | 94.69 | 90.15 | 55.28 | 95.13 | 94.32 | 90.06 | 56.37 | 96.33 | 95.65 | 92.16 | 54.41
SPTM (epoch 30) [16] | 94.50 | 93.28 | 88.03 | 49.61 | 92.89 | 90.20 | 78.27 | 50.47 | 93.53 | 91.23 | 81.52 | 49.69
PCL [57] | 49.27 | 42.86 | 27.00 | 0.35 | 60.36 | 54.18 | 37.73 | 0.19 | 57.36 | 51.66 | 35.33 | 0.03
VLAD [9, 8] | 42.48 | 30.69 | 12.54 | 0.07 | 49.23 | 37.82 | 17.36 | 0.00 | 53.52 | 41.23 | 16.53 | 0.12
TF-VPR (Ours) | 96.31 | 96.00 | 93.62 | 67.67 | 95.59 | 95.21 | 93.42 | 65.97 | 96.91 | 96.57 | 95.35 | 69.39
TF-VPR (Ours) (epoch 30) | 96.15 | 95.82 | 93.31 | 66.54 | 95.46 | 94.93 | 92.82 | 63.94 | 96.66 | 96.32 | 94.66 | 65.59
NetVLAD [12] | 99.63 | 99.16 | 96.36 | 60.48 | 99.48 | 99.26 | 97.38 | 63.38 | 99.56 | 99.17 | 96.60 | 64.64

Figure 7: Recall@10 and HD vs. training epoch on the photorealistic dataset
(Goffs). V denotes method with geometric verification. Other abbreviations
follow Fig. 5. Figure 8: Qualitative VPR results on the photorealistic RGB
dataset. The upper part shows an example where TF-VPR outperforms SPTM [16],
and the lower part depicts a challenging example in which none of the methods
retrieve the data correctly. Green and red indicates true and false non-
temporal spatial positives, respectively. Although the frames in the first row
appear to be positives, their positions are still far from the query. Due to
the high frame density in the habitat-sim dataset, frames can only be
considered positives if they are no more than $20$cm apart from the query.

Dataset. TF-VPR has also been tested via habitat-sim [54] simulator on the
Gibson photorealistic RGB dataset [58], which provides panoramic RGB images
for a variety of indoor scans. We capture RGB images by a panoramic camera
mounted on a robot moving randomly in the virtual environment. We captured a
total of $33,679$ RGB images in three Gibson rooms. Each image is downsampled
to $256\times 64$ pixels. In contrast to other datasets, this simulated RGB
dataset contains a large number of revisits of places from both similar and
different directions, which is useful for testing recall rate and heading
diversity for VPR.

Baseline methods. The following baselines are evaluated: (1) NetVLAD [12]
trained with pose-based supervision, (2) NetVLAD trained with temporal pseudo
labels as in [16] (SPTM), (3) prototypical contrastive learning (PCL) [57] as
another self-supervised VPR method used in visual navigation [37], (4) VLAD
[9, 8] as a classic non-deep-learning VPR, and the previous ablation study
baselines.

Implementation details. To speed up computation, the output dimension is set
to $512$. Our implementation is based on the code from NetVLAD. The method for
converting NetVLAD to SPTM is similar to Section IV-B. Similarly, we also
choose $n=5$ and $u=2$ in Section III-B, and exclude for each query’s closest
$30$ temporal neighbors from the top-N retrievals as explained in IV-A.

Considering new baselines involved, PCL is implemented as the default setting.
We need to tune the total number of clusters. In this experiment, we tried to
set the total number of clutter to $200$, $500$, and $1000$ and select the
best result. For the VLAD baseline, we use the classic 128-dimensional SIFT
features, and a cluster size of $32$. The raw VLAD descriptor dimension of
$32\times 128$ is further reduced to $512$ by PCA.

Importance of contraction. Geometric verification becomes more important for
RGB data. The generated positive candidate is not as trustworthy as in the toy
dataset in Section IV-B. As a result, including geometric verification helps
maintain or even increase accuracy. From Fig. 7, we can observe that both
metrics in all methods other than TF-VPR decline with respect to training
epochs, because they overfit temporal neighbors as depicted in the first stage
in Fig. 1(b). And they tend to miss true spatial neighbors. To prevent this,
the feature space neighborhood together with the verification discovers more
reliable positives in $\hat{\mathcal{P}}^{f}_{{\bf{\mathbf{q}}}_{i}}$ to be
added into $\mathcal{P}_{{\bf{\mathbf{q}}}_{i}}$, as depicted in the third
stage of Fig. 1(b). The stability of the model performance over training
epochs is critical for self-supervised VPR in reality because we would not
know when to stop training without ground truth poses.

Ablation study on habitat-sim. As shown in Fig. 7 and Table II, TF-VPR
outperforms all unsupervised baselines, and approaches the supervised NetVLAD
performance in both recall rate and heading diversity (HD). Particularly, HD
is improved by a large margin, particularly for scenes with a high number of
revisits from different directions. On average, TF-VPR improves recall rate by
$1\%$ and heading diversity by $10\%$. Fig. 6 visualizes the retrieval quality
on the simulated RGB dataset. SPTM can make reasonably accurate estimates in
recall@10 and HD. TF-VPR outperforms SPTM, but the improvement is not
significant. Especially, the improvement of HD in the RGB dataset is not
comparable to that in the point cloud dataset, because RGB images appear to be
less resistant to environmental changes such as lighting and seasons. The
qualitative results are in Fig. 8.

PCL, VLAD, and conventional visual SLAM system. Table II shows the poor
performance of PCL and VLAD. We believe that PCL might not be a good VPR
solution, because most contrastive learning methods do form several disjoint
clusters for each category, which is not suitable to represent a continuous
feature in vision-based SLAM problems. Similarly, VLAD performs poorly when
the input resolution is low and the output dimension is small. Moreover, we
test the conventional visual SLAM system, like OpenVSLAM [59]. However,
OpenVSLAM easily loses track of odometry. A total of $17.75\%$ of the frames
are lost during the tracking process. OpenVSLAM builds a disjoint topology
graph while tracking odometry, and the recall rate is $54.98\%$ versus
$96.23\%$ for TF-VPR. As a result of the poor performance, we do not use
OpenVSLAM as a benchmark.

###  IV-D Experiments on NYU-VPR-360 dataset

Figure 9: Recall@10 and HD vs. training epoch on NYU-VPR-360 (scene 1).
Abbreviations follow Fig. 7. TABLE III: Quantitative results on NYU-VPR-360
dataset. We employ best recall rate (R) and heading diversity (HD). Two scenes
have been reported (Scene1, Scene2).

Scene | Scene 1 | Scene 2
---|---|---
Metric | R@10 | R@5 | R@1 | HD | R@10 | R@5 | R@1 | HD
SPTM [16] | 69.02 | 60.08 | 48.27 | 79.49 | 65.97 | 62.52 | 53.25 | 50.79
SPTM (epoch 30) [16] | 60.16 | 50.27 | 39.34 | 75.46 | 58.12 | 54.06 | 43.92 | 46.39
PCL [57] | 64.75 | 57.30 | 40.99 | 0.04 | 62.80 | 58.42 | 46.96 | 13.46
VLAD [9, 8] | 37.31 | 27.89 | 10.53 | 0.01 | 21.23 | 16.92 | 8.78 | 0.01
TF-VPR (Ours) | 71.94 | 63.89 | 52.56 | 83.34 | 66.16 | 62.05 | 52.49 | 51.19
TF-VPR (Ours) (epoch 30) | 70.97 | 62.61 | 50.99 | 80.79 | 64.18 | 60.09 | 51.44 | 49.33
NetVLAD [12] | 100.00 | 100.00 | 100.00 | 86.38 | 100.00 | 100.00 | 99.99 | 54.45

NYU-VPR-360 dataset. There are several VPR datasets with panoramic images [60,
50], yet few of them have repeated visits to the same place from a variety of
angles. To show the ability of our method in retrieving images of different
headings, we proposed the NYU-VPR-360 dataset captured by Gopro MAX (a dual-
lens 360∘ camera with GPS recording), which is composed of sequentially
collected panoramic RGB images of street views in New York City. The GoPro
camera was mounted on the top of the driving vehicle. We utilize the GPS
readings of the camera to provide the ground truth of spatial neighborhoods.
Note that we select the panoramic images from the whole video to make them
synchronized with GPS. The dataset is composed of two driving trajectories,
covering an area of approximately $80,000m^{2}$. There are over $15,000$
images of $3840\times 1920$ pixels in the dataset with their corresponding
locations for each scene. Most junctions have at least two types of actions
with different driving directions, with the exception of a few intersections
that are for traffic reasons.

Baseline methods. Following IV-C, we use SPTM [16], NetVLAD [12], VLAD [9, 8],
and PCL [57] as baselines.

Implementation details. The images are resized to $128\times 64$ pixels. We
set $n=10$ and $u=5$ as described in Section III-B. For scene 1 and scene 2,
based on different sampling rates of the sensors, for each query, we
respectively exclude its closest $30$ and $100$ temporal neighbors from the
top-N retrievals as explained in IV-A.

Comparison with baselines. The edge of our method is more distinguishable on
NYU-VPR-360 dataset as proven by the qualitative results in Fig. 10. As shown
in Fig. 6, there is an improvement in the recall rate using TF-VPR .
Furthermore, as shown in Table III, TF-VPR surpasses other baselines in recall
rate and heading diversity in scene 1 by about $2\%$ and $4\%$ respectively.
Furthermore, performance in scene 2 does not improve because there are no
spatial positives from different headings in scene 2. More importantly, the
performance gap between TF-VPR and other baselines becomes larger over epochs.
As shown in Fig. 9, the recall@10 of TF-VPR outperforms the baselines by
approximately $8\%$-$10\%$ at epoch $30$ as discussed in IV-C.

Dynamic KNN mechanism. The distinction between the K-nearest-neighbor
specification and the dynamic KNN described in Section III-D is sharper in our
real-world experiment. This proves that the dynamic KNN is a better mechanism
for selecting feature neighborhoods because its flexibility allows the model
to find an adequate number of neighbors for each location in the dataset.

Figure 10: Qualitative VPR results on NYU-VPR-360 dataset. The upper part
shows an example where TF-VPR outperforms SPTM, and the lower part shows a
challenging example in the dataset where only the top-1 retrieval of TF-VPR
and VLAD is correct. Green and red respectively indicate true and false
positives .

##  V Conclusion

We propose TF-VPR as a self-supervised auto-labeling method for determining
the unknown spatial neighbors from the fixed temporal neighbors and learnable
feature neighbors. Extensive experiments show that TF-VPR not only improves
the recall rate over the existing method but also can retrieve spatial
positives with more diverse viewpoints on various datasets. TF-VPR enables
easier use of VPR in real-world robotics and computer vision applications, and
can be applied to most existing deep-learning-based VPR methods.

Acknowledgment. This work is supported by NSF grants under CMMI-1932187,
CNS-2121391, and EEC-2036870.

## References

  * [1] S. Lowry, _et al._ , “Visual place recognition: A survey,” _T-RO_ , 2015\.
  * [2] E. Brachmann and C. Rother, “Visual camera re-localization from RGB and RGB-D images using DSAC,” _TPAMI_ , 2021.
  * [3] A. Angeli, S. Doncieux, J.-A. Meyer, and D. Filliat, “Real-time visual loop-closure detection,” in _2008 ICRA_ , 2008.
  * [4] F. Radenović, A. Iscen, G. Tolias, Y. Avrithis, and O. Chum, “Revisiting oxford and paris: Large-scale image retrieval benchmarking,” in _CVPR_ , 2018\.
  * [5] M. Zaffar, _et al._ , “VPR-Bench: An open-source visual place recognition evaluation framework with quantifiable viewpoint and appearance change,” _IJCV_ , 2021.
  * [6] D. Sheng, _et al._ , “NYU-VPR: Long-term visual place recognition benchmark with view direction and data anonymization influences,” in _IROS_ , 2021.
  * [7] D. G. Lowe, “Object recognition from local scale-invariant features,” in _ICCV_ , 1999.
  * [8] H. Jégou, M. Douze, C. Schmid, and P. Pérez, “Aggregating local descriptors into a compact image representation,” in _CVPR_ , 2010.
  * [9] R. Arandjelovic and A. Zisserman, “All about vlad,” in _CVPR_ , June 2013\.
  * [10] P.-E. Sarlin, C. Cadena, R. Siegwart, and M. Dymczyk, “From coarse to fine: Robust hierarchical localization at large scale,” in _CVPR_ , 2019.
  * [11] Z. Chen, _et al._ , “Deep learning features at scale for visual place recognition,” in _ICRA_ , 2017.
  * [12] R. Arandjelovic, P. Gronat, A. Torii, T. Pajdla, and J. Sivic, “Netvlad: Cnn architecture for weakly supervised place recognition,” in _CVPR_ , June 2016\.
  * [13] S. Garg, M. Vankadari, and M. Milford, “Seqmatchnet: Contrastive learning with sequence matching for place recognition & relocalization,” in _CoRL_. PMLR, 2022.
  * [14] A. Forechi, A. F. De Souza, C. Badue, and T. Oliveira-Santos, “Sequential appearance-based global localization using an ensemble of knn-dtw classifiers,” in _IJCNN_ , 2016.
  * [15] O. Vysotska and C. Stachniss, “Lazy data association for image sequences matching under substantial appearance changes,” _RA-L_ , 2015.
  * [16] N. Savinov, A. Dosovitskiy, and V. Koltun, “Semi-parametric topological memory for navigation,” in _ICLR_ , 2018.
  * [17] S. E. Reed, H. Lee, D. Anguelov, C. Szegedy, D. Erhan, and A. Rabinovich, “Training deep neural networks on noisy labels with bootstrapping,” in _ICLR (Workshop)_ , 2015.
  * [18] M. Cummins and P. Newman, “FAB-MAP: Probabilistic localization and mapping in the space of appearance,” _IJRR_ , 2008.
  * [19] N. Cummins, Mark and Paul, “Appearance-only SLAM at large scale with FAB-MAP 2.0,” _IJRR_ , 2011.
  * [20] M. J. Milford, G. F. Wyeth, and D. Prasser, “Ratslam: a hippocampal model for simultaneous localization and mapping,” in _ICRA_ , 2004.
  * [21] D. Galvez-Lopez and J. D. Tardos, “Real-time loop detection with bags of binary words,” in _IROS_ , 2011.
  * [22] M. J. Milford and G. F. Wyeth, “Seqslam: Visual route-based navigation for sunny summer days and stormy winter nights,” in _ICRA_ , 2012.
  * [23] G. Costante, T. A. Ciarfuglia, P. Valigi, and E. Ricci, “A transfer learning approach for multi-cue semantic place recognition,” in _IROS_ , 2013.
  * [24] N. Sünderhauf, _et al._ , “Place recognition with convnet landmarks: Viewpoint-robust, condition-robust, training-free,” in _Robotics: Science and Systems XI_. Robotics: Science and Systems Conference, 2015.
  * [25] M. Lopez-Antequera, R. Gomez-Ojeda, N. Petkov, and J. Gonzalez-Jimenez, “Appearance-invariant place recognition by discriminatively training a convolutional neural network,” _Pattern Recognition Letters_ , 2017.
  * [26] T. Naseer, G. L. Oliveira, T. Brox, and W. Burgard, “Semantics-aware visual localization under challenging perceptual conditions,” in _ICRA_ , 2017.
  * [27] S. Garg, N. Suenderhauf, and M. Milford, “Lost? appearance-invariant place recognition for opposite viewpoints using visual semantics,” _arXiv preprint arXiv:1804.05526_ , June 2018.
  * [28] C. Choy, J. Park, and V. Koltun, “Fully convolutional geometric features,” in _ICCV_ , 2019.
  * [29] H. Deng, T. Birdal, and S. Ilic, “Ppfnet: Global context aware local features for robust 3D point matching,” in _CVPR_ , 2018.
  * [30] M. A. Uy and G. H. Lee, “PointNetVLAD: Deep point cloud based retrieval for large-scale place recognition,” in _CVPR_ , 2018.
  * [31] Z. Liu, _et al._ , “Lpd-net: 3d point cloud learning for large-scale place recognition and environment analysis,” in _ICCV_ , 2019.
  * [32] X. Chen, _et al._ , “Overlapnet: Loop closing for lidar-based slam,” in _Robotics: Science and Systems_ , 2020.
  * [33] K. Cai, B. Wang, and C. X. Lu, “Autoplace: Robust place recognition with low-cost single-chip automotive radar,” in _ICRA_ , 2022.
  * [34] S. Lowry and M. J. Milford, “Supervised and unsupervised linear learning techniques for visual place recognition in changing environments,” _T-RO_ , 2016.
  * [35] N. Merrill and G. Huang, “Lightweight unsupervised deep loop closure,” in _Robotics: Science and Systems_ , 2018.
  * [36] X. Gao and T. Zhang, “Unsupervised learning to detect loops using deep neural networks for visual SLAM system,” _Autonomous robots_ , 2017.
  * [37] O. Kwon, N. Kim, Y. Choi, H. Yoo, J. Park, and S. Oh, “Visual graph memory with unsupervised representation for visual navigation,” in _ICCV_ , 2021\.
  * [38] N. Natarajan, I. S. Dhillon, P. K. Ravikumar, and A. Tewari, “Learning with noisy labels,” _NeurIPS_ , 2013.
  * [39] H. Song, M. Kim, D. Park, Y. Shin, and J.-G. Lee, “Learning from noisy labels with deep neural networks: A survey,” _TNNLS_ , 2022.
  * [40] Z. Yu, _et al._ , “Simultaneous edge alignment and learning,” in _ECCV_ , 2018.
  * [41] G. Patrini, A. Rozza, A. Krishna Menon, R. Nock, and L. Qu, “Making deep neural networks robust to label noise: A loss correction approach,” in _CVPR_ , 2017.
  * [42] D.-H. Lee _et al._ , “Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks,” in _ICML_ , 2013\.
  * [43] Y. Zou, Z. Yu, B. Kumar, and J. Wang, “Unsupervised domain adaptation for semantic segmentation via class-balanced self-training,” in _ECCV_ , 2018\.
  * [44] Y. Zou, Z. Yu, X. Liu, B. Kumar, and J. Wang, “Confidence regularized self-training,” in _ICCV_ , 2019.
  * [45] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _CVPR_ , 2020.
  * [46] A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” _arXiv preprint arXiv:1807.03748_ , 2018\.
  * [47] Y. Tian, D. Krishnan, and P. Isola, “Contrastive multiview coding,” in _ECCV_. Springer, 2020.
  * [48] Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola, “What makes for good views for contrastive learning?” _NeurIPS_ , 2020.
  * [49] R. Arandjelović and A. Zisserman, “Visual vocabulary with a semantic twist,” in _ACCV_. Springer, 2014\.
  * [50] A. Torii, R. Arandjelovic, J. Sivic, M. Okutomi, and T. Pajdla, “24/7 place recognition by view synthesis,” in _CVPR_ , 2015.
  * [51] L. Ding and C. Feng, “Deepmapping: Unsupervised map estimation from multiple point clouds,” in _CVPR_ , 2019.
  * [52] P. C. Ng and S. Henikoff, “Sift: Predicting amino acid changes that affect protein function,” _Nucleic acids research_ , 2003.
  * [53] P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas, “Learning representations and generative models for 3d point clouds,” in _ICML_. PMLR, 2018.
  * [54] M. Savva, _et al._ , “Habitat: A platform for embodied AI research,” in _ICCV_ , 2019.
  * [55] A. Paszke, _et al._ , “Automatic differentiation in pytorch,” _arXiv preprint arXiv:2010.07922_ , 2017.
  * [56] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _ICLR_ , 2015.
  * [57] J. Li, P. Zhou, C. Xiong, and S. C. Hoi, “Prototypical contrastive learning of unsupervised representations,” in _ICLR_ , 2021.
  * [58] F. Xia, A. R. Zamir, Z.-Y. He, A. Sax, J. Malik, and S. Savarese, “Gibson env: real-world perception for embodied agents,” in _CVPR_ , 2018.
  * [59] S. Sumikura, M. Shibuya, and K. Sakurada, “Openvslam: A versatile visual slam framework,” in _ACM Multimedia_ , 2019.
  * [60] A. Torii, J. Sivic, T. Pajdla, and M. Okutomi, “Visual place recognition with repetitive structures,” in _CVPR_ , 2013.