11institutetext: Polytechnic University of Turin, Turin, Italy
22institutetext: CINI - Consorzio Interuniversitario Nazionale per
l’Informatica, Rome, Italy 22email:
<EMAIL_ADDRESS>
# Learning Semantics for Visual Place Recognition through Multi-Scale
Attention

Valerio Paolicelli 11 Antonio Tavera 11 Carlo Masone 22 Gabriele Berton 11
Barbara Caputo 1111

###### Abstract

In this paper we address the task of visual place recognition (VPR), where the
goal is to retrieve the correct GPS coordinates of a given query image against
a huge geotagged gallery. While recent works have shown that building
descriptors incorporating semantic and appearance information is beneficial,
current state-of-the-art methods opt for a top down definition of the
significant semantic content. Here we present the first VPR algorithm that
learns robust global embeddings from both visual appearance and semantic
content of the data, with the segmentation process being dynamically guided by
the recognition of places through a multi-scale attention module. Experiments
on various scenarios validate this new approach and demonstrate its
performance against state-of-the-art methods. Finally, we propose the first
synthetic-world dataset suited for both place recognition and segmentation
tasks.

###### Keywords:

Visual Place Recognition Semantic Segmentation Attention

##  1 Introduction

Visual place recognition (VPR) [17], i.e. , the task of recognizing the
location where a photo was taken, is usually cast as an image retrieval
problem: the location of a photo (_query_) is estimated by comparing it to a
huge database of geo-tagged images (_gallery_). Much of the recent research on
this subject has focused on finding better image representations to perform
the retrieval. Despite the advances in this direction made possible by the use
of deep convolutional neural networks (DCNN) [2, 21, 25], current VPR
solutions still fail to achieve the degree of generality and flexibility
required to work across different environments and conditions [31]. Recent
studies have found that these problems can be mitigated by building image
descriptors based not only on visual appearance, but also on the semantic
content in the scene [19, 15, 11]. Intuitively, dynamic objects or elements
that are both common across all places and that lack distinctive features
(e.g. , roads and sky) are not very informative for VPR. On the other hand,
content that is stable across different conditions and that has a wide range
of intra-class variations, such as buildings [19], can more robustly describe
places.

Figure 1: Prior works have shown that the VPR task can be improved by imbuing
the features with semantic information. However, when the semantic task is not
controlled by the place recognition, it draws information from all the scene
indistinctly (middle). By conditioning the semantic segmentation on the place
recognition task (right), the model learns to draw information only from the
semantic categories that are most discriminative for a place, e.g. , buildings
and traffic lights.

While this intuition holds promise, it currently faces two intertwined open
challenges, i.e. , the need to define a priori the semantic content to use and
the lack of an appropriate annotated database from where to learn it. Indeed,
previous works on combining appearance and semantic information to generate
global descriptors for VPR have either empirically defined what semantic
classes should be used to describe places [18, 19] or used all the semantic
content available [11]. While this can be seen as a legitimate choice, it
remains that it does not exist a public VPR dataset containing images
annotated with pixel-wise semantic maps. What is currently available to the
community are large-scale, multi-scenario VPR datasets without semantic
annotations [16, 27, 3, 28], or autonomous driving datasets that provide
pixel-wise labels but either lack GPS annotation [6, 24, 23] or are too small
in scale to effectively build query and gallery sets [30, 8, 12, 9].

Here we argue that it is best to let the model figure out what semantic
information is more relevant to describe and recognize a place. Hence, we
present a large scale synthetic database, annotated with both GPS and pixel-
wise semantic maps, jointly with a new architecture that builds global
descriptors for VPR in a data driven manner, informed by both visual
appearance and semantic content of the training data. To do so, we introduce
an attention-based mechanism to dynamically condition, during training, the
segmentation process on the recognition of places. The attention mechanism
operates by leveraging features located at multiple spatial scales to capture
the discriminative urban objects with different sizes in the scene.

In summary, the contributions of this paper are:

  * •

a new data-driven method to generate highly informative global descriptors for
VPR, leveraging both visual appearance and semantic content at different
scales;

  * •

a new synthetic dataset for large-scale visual place recognition that also
contains pixel-wise semantic labels;

  * •

extensive validation on various real-world scenarios, demonstrating both the
effectiveness of each component in our architecture and a consistent
improvement over previous state-of-the-art methods.

##  2 Related Work

Semantically informed Visual Place Recognition. Most modern approaches for
visual place recognition rely on an image retrieval formulation [17], using
DCNNs to extract appearance features which then generate global descriptors by
means of aggregation [2] or pooling [25, 21]. Recent works have proposed to
enhance this paradigm by introducing attention mechanisms [29, 3, 34, 14] and
domain adaptation techniques to align features of different scenarios [3, 13].
Few studies suggest building the global descriptors not only using the visual
information in the images, but also their semantic content. Along these lines,
the method presented in [19] requires segmenting an images also at inference
time while [15] requires a 3D point cloud of the scene. Closely related to our
work is DASGIL [11], an architecture that uses a single encoder shared by
three tasks (VPR, depth mask reconstruction and semantic mask reconstruction)
to create embeddings that fuse visual, geometric and semantic information.
Similar to our solution, DASGIL is trained on a synthetic dataset and it uses
domain adaptation to align the features extracted from the synthetic and real-
world domains. Besides that, in [11] the segmentation task focuses
indiscriminately on all the semantic classes. On the contrary, our solution is
built on the intuition that not all the semantic content is useful for VPR and
we let the place recognition task guide the segmentation one via an attention
mechanism. Moreover, DASGIL builds a global descriptor by flattening and
concatenating the features extracted at multi-scale, without an embedding
step. This produces extremely large descriptors which are not well suited for
large scale problems. Instead, we use a novel multi-scale aggregation layer
which produces more compact descriptors.

##  3 Method

We consider having at training a collection $X=\\{(x,y,z)\\}$ of $N$ triplets,
where $x$ is an RGB image composed by $\mathcal{I}$ pixels, $y$ is the
semantic map that associates to each pixel $i$ a class from a set of semantic
classes $C$, and $z$ is the GPS coordinate where the image was taken. We
propose a novel framework for visual place recognition that leverages both the
appearance and pixel-wise semantic information available during training to
learn image representations that are more effective for the place recognition
problem. Our architecture, depicted in Fig. 2, consists of a single encoder
shared by two tasks:

  * •

a visual place recognition task (_VPR_) (Sec. 3.1), that implements a novel
multi-scale pooling layer to generate the global embeddings used for the
retrieval process;

  * •

a semantic segmentation task (_SemSeg_) (Sec. 3.2), that implements a decoder
for parsing the scene according to the set of classes $C$.

Figure 2: Illustration of the proposed architecture. The Multi-Scale attention
module (middle) is trained by the place recognition task (top) and guides the
segmentation module (bottom) towards relevant semantic classes. Blue lines
refer to the visual place recognition task (VPR) that implements the novel
multi-scale pooling layer (top). Purple lines refer to the segmentation task
(bottom).

By sharing the same encoder, the two tasks induce it to learn features that
combine both the visual information used for VPR and the semantic information
present in the scene. However, without a proper mechanism to control this
fusion, the model would equally focus on all semantic classes, regardless of
their actual relevance for the VPR task. We introduce such a mechanism in the
form of a multi-scale attention module that is trained only on the VPR task
but modulates also the features extracted by the SemSeg decoder. In this way,
the VPR guides the semantic segmentation, informing it where to focus on the
scene. Note that at inference time the SemSeg decoder is not used, thus the
deployed model is quite lightweight.

###  3.1 Multi-scale and attentive VPR task

The shared encoder in our architecture is a ResNet [10] truncated after the
last convolutional block. We indicate as $f_{4}$ and $f_{5}$ the outputs of
the last two convolutional blocks, _conv4_ and _conv5_ , with shapes
$C_{4}\times H_{4}\times W_{4}$ and $C_{5}\times H_{5}\times W_{5}$,
respectively. These features are used as input to both the multi-scale
attention module (Fig. 2, middle) and to the novel multi-scale pooling layer
(Fig. 2, top).

Multi-scale attention. Recent works have demonstrated the use of multi-scale
attention mechanisms in place recognition as a way to focus on the salient
regions in the image [29, 34, 14]. In our architecture, the attention module
becomes instrumental to make place recognition guide the semantic segmentation
during training. The module, depicted in Fig. 4, takes the output $f_{4}$ from
the encoder and passes it through a bank of $O\times k_{s}\times k_{s}$
filters with $O$ number of output channels and different $k_{s}$ kernel sizes
($64\times 3\times 3$, $64\times 5\times 5$ and $64\times 7\times 7$). The
outputs of these filters are upsampled and concatenated channel-wise, before
passing through a $1\times 1\times 1$ filter and a softplus function that
produces a $1\times H\times W$ attention map $M$. The scores in the attention
map $M$ indicate where the retrieval is focusing.

Figure 3: The attention module leverages features at multiple spatial scales
to capture objects with different sizes, and produces a map $M$ that marks the
retrieval salient regions. $\oplus$ indicates up-sampling and concatenation.

Figure 4: Illustration of the multi-scale pooling module. Features extracted
at different levels of the shared encoder are exploited to compute the global
descriptors. $\otimes$ indicates up-sampling and dot product. $\oplus$
indicates concatenation.

Multi-scale pooling. In VPR it is widely common to use pooling layers after
the convolutional backbone to extract compact global descriptors for the
retrieval, the state-of-the-art being GeM [21]. In order to exploit semantic
and appearance information at different abstraction levels, we introduce a
multi-scale GeM layer (_ms-GeM_) that uses both $f_{4}$ and $f_{5}$, as
illustrated in Fig. 4. These features are first weighted by the attention
scores M via dot-product and L2-normalized. Then, they are pooled using
vanilla GeM layers to produce the global descriptors
$F_{w4}\in\mathbb{R}^{C_{4}}$ and $F_{w5}\in\mathbb{R}^{C_{5}}$. Finally,
these descriptors are concatenated to form a representation
$F\in\mathbb{R}^{C_{4}+C_{5}}$.

VPR loss. We use the weakly supervised triplet margin loss and training
protocol from [2] to train the model to extract descriptors for the VPR task.
For each training query we consider a positive and a negative example drawn
from the gallery. The positive example is an image of the same place as the
one depicted in the query, whereas the negative example is an image of a
different location. Both the query and its corresponding positive/negative
examples are from the source domain, therefore we use the available GPS
information to select the examples. In particular, we consider as negative
example the most similar image in the features space far from the query GPS
coordinates. Finally, for each query descriptor $F_{q}$ the loss is

$\mathcal{L}_{VPR}=h(d(F_{q},F_{p})+m-d(F_{q},F_{n}))$ (1)

where $h$ is the hinge loss, $d(\cdot,\cdot)$ is the Euclidean distance, $m$
is a fixed margin, $F_{p}$ and $F_{n}$ are the descriptors of the positive and
negative examples, respectively. The goal pursued by $\mathcal{L}_{VPR}$ is to
learn descriptors so that the distance between a training query and its
positive example is smaller than the distance between the query and its
negative example by at least the margin $m$. Additionally, it is also used to
optimize the parameters of the multi-scale attention module and thus focus the
segmentation task on the salient regions for place recognition.

###  3.2 Guided semantic segmentation task

The SemSeg task informs the features extraction process with semantic
information. For this purpose, we use a semantic segmentation decoder
$D_{seg}$ (see Fig. 2). However, to force the model to focus on the semantic
information that is most discriminative for places, the output of the decoder
$D_{seg}$ is weighted by the attention map $M$, which is trained by the VPR
task alone. To train the shared encoder and the decoder, the SemSeg branch
uses a cross-entropy loss computed for each class $y_{i}$ at pixel
$\mathcal{I}$, i.e.,

$\displaystyle\mathcal{L}_{SemSeg}=-\frac{1}{|\mathcal{I}|}\sum_{i\in\mathcal{I}}y_{i}\cdot\log
p_{i}^{y_{i}}((M^{i}\cdot f_{d}^{i}))$ (2)

where $M^{i}$ is the attention map related to the feature $f_{d}^{i}$
extracted from the decoder $D_{seg}$, while $p_{i}^{y_{i}}$ denotes the
probability for class $y_{i}$ at pixel $\mathcal{I}$. Fig. 5 shows some
examples of heatmaps resulting from the attention module on the source domain
images. From these heatmaps it emerges that the network learns to focus on
distinctive man-made structures such as buildings, shop signs and
streetlights.

###  3.3 Training loss

Summarizing, the VPR-SemSeg loss function is:

$\mathcal{L}_{VPR-SemSeg}=\mathcal{L}_{VPR}+\alpha\cdot\mathcal{L}_{SemSeg}$
(3)

where $\alpha$ is a scalar weight. Both $\mathcal{L}_{VPR}$ and
$\mathcal{L}_{SemSeg}$ affect the encoder weights to produce features that are
informative for place recognition and combine visual and semantic information.
However, only $\mathcal{L}_{VPR}$ impacts the attention module weights, thus
making the segmentation dependent on the place recognition.

Figure 5: Attention heatmaps extracted from our framework. We see that
attention is focused only on relevant semantic classes according to the VPR
task, such as buildings.

##  4 A VPR and semantic segmentation dataset

The proposition of using semantic information in VPR in a data-driven approach
is limited by the lack of a dataset that is both built for place recognition,
i.e. , containing multiple views of the same places tagged with GPS
coordinates, and that provides fine-grained semantic labels. There is only a
couple of synthetic datasets that come close to this requirement, but
eventually fall short. One is Virtual KITTI 2 [4], which was used by [11], but
it only guarantees 447 images per scenario. Moreover, images are not
associated to GPS coordinates, so place matching is only done by the name of
the images. The other is SYNTHIA [24], which contains very dense sequences of
images that are suitable for the visual localization task [20] ($\leq 5$m from
one gps coordinate to the other) but not for the coarser place recognition
task usually considered in literature ($\leq 25$m). To develop our data-driven
approach for VPR that combines both visual and semantic knowledge, and also to
enable further research in this direction, we created a new synthetic dataset.

This new dataset was inspired by IDDA [1], which was built from the CARLA
virtual simulator [7] specifically for semantic segmentation but without GPS
annotations. Following the methodology from [1] we used CARLA 0.9.10 to build
a new dataset that includes both GPS/IMU information and pixel-wise semantic
annotations with 25 semantic classes (with 17 of them in common to the
Cityscapes [6] standard). This new synthetic dataset contains more than 40000
images (10091 per scenario) captured across two different urban maps (Town03
and Town10 from CARLA notation) and in two weather conditions, Clear Noon and
Hard Rain Sunset. To collect the data we equipped the ego-vehicle with four
cameras (front view, rear view, left view and right view). We split the front
and rear view frames captured in the Town10 in a gallery and a query set; the
first including the Clear Noon images and the latter the Hard Rain Sunset
ones, see Fig. 6 a-b). The left and right view images captured in the Town03
are used as a validation set, following the same criterion used to split the
training set, see Fig. 6 c-d).

(a)

(b)

(c)

(d)

Figure 6: a-b) show examples of the Town10 gallery and query sets from the
front view, while c-d) are from the Town03 left view. Gallery and query sets
are collected under the Clear Noon and Hard Rain Sunset weather conditions,
respectively.

###  4.1 Synthetic-to-real domain adaptation

The new synthetic dataset provides the training data for the method presented
in Sec. 3. However, there is a significant gap between the images from this
simulator (_source domain_ $X_{s}$) and from the real-world (_target domain_
$X_{t}$). To reduce this gap we use adversarial training that aims at aligning
the features extracted from the synthetic and target domains. We assume having
available at training time a set of unlabeled target domain images, besides
the labeled synthetic data. While the VPR and SemSeg tasks illustrated in
Secs. 3.1 and 3.2 are trained using only the synthetic data $X_{s}$, we
introduce a domain discriminator $D$ similar to [22] to distinguish the source
features produced by $D_{seg}$ from the target ones. The discriminator is
trained with the binary cross-entropy loss:

$\displaystyle\mathcal{L}_{discr}=-\sum_{i\in\mathcal{I}}(1-z)logD(p_{i}(x_{s}))+zlogD(p_{i}(x_{t}))$
(4)

with $x_{s}\in X_{s}$ and $x_{t}\in X_{t}$, $p_{i}(x)$ the features from
$D_{seg}$ and z a parameter which is 0 if the features are from the source and
1 if the features are from the target domain. Overall, the adversarial
training tries to fool the encoder and the SemSeg decoder, computing a binary
cross-entropy loss on $\mathcal{X}_{t}$ labeled like source:

$\mathcal{L}_{adv}=-\frac{1}{|\mathcal{I}|}\sum_{i\in\mathcal{I}}logD(p_{i}(x_{t}))$
(5)

In this way, the model learns to align the features for every domain and the
encoder becomes able to extract discriminative local embeddings for the real
world VPR task.

Summarizing, the overall training loss function becomes:

$\mathcal{L}_{tot}=\mathcal{L}_{VPR-
SemSeg}+\beta\cdot\mathcal{L}_{adv}+\gamma\cdot\mathcal{L}_{discr}$ (6)

where $\beta$ and $\gamma$ are scalar weights and $\mathcal{L}_{adv}$ affects
the encoder weights.

##  5 Experiments

Comparisons with other methods. We assess the effectiveness of our approach
comparing it to GeM [21] and RMAC [25], two state-of-the-art VPR methods that
use global descriptors. We also compare to DASGIL [11] which, similarly to our
method, uses semantic information albeit selecting it in a top-down manner.
Besides semantics, DASGIL can also leverage depth information to further
enhance the descriptors. We compare to three different versions of DASGIL: the
pre-trained models released by the authors using their proposed Flatten
Discriminator (DASGIL_FD $\bullet$) and Cascade Discriminator (DASGIL_CD
$\bullet$), as well as the model with Flatten Discriminator trained by us on
our dataset but without depth information (DASGIL_nodepth). DASGIL_FD
$\bullet$ and DASGIL_CD $\bullet$ are pretrained on KITTI and VIRTUAL KITTI 2,
while just DASGIL_nodepth is pretrained on our novel dataset. In the
comparison with DASGIL we must also note that it extracts and concatenates the
local features from the i-th and j-th _conv_ layer of the shared encoder,
producing final embeddings of variable dimension $1\times 64\times(H_{i}\times
W_{i}+H_{j}\times W_{j})$. On RobotCar [16]), this results in $15360$-D
(dimensional) descriptors w.r.t the more compact $3072$-D descriptors produced
by our method.

Concerning the implementation of our method, the details are provided in the
supplementary material.

Datasets and protocol. For the evaluation we used Oxford RobotCar [16] as the
inference dataset. Oxford RobotCar is a collection of images from the city of
Oxford taken in different environmental conditions from a car-mounted camera.
We use the Overcast scenario as the gallery, while the queries are divided
into four scenarios: Rain, Snow, Sun, and Night, with one image sampled every
5 meters in order to decrease data redundancy. In order to have a fair
comparison and truly assess the effectiveness of our solution, which cannot be
trained directly on the target domain because of the lack of semantic labels,
we trained all models on our new synthetic dataset, adding to GeM and RMAC the
same unsupervised domain adaptation (DA) module used in our network. On other
hand, DASGIL, which also uses semantic labels, already includes a DA module.
Despite the usage of DA techniques we can expect the result to be considerably
lower than if the models would be trained directly on the target domain, due
to the strong domain shift. For completeness, in the supplementary material we
show some additional comparisons and also results of the models trained on the
real-world target domain, using imperfect semantic labels generated by an
expert network. Finally, we also compare the generalization capability of
these methods, trained on the synthetic dataset with Oxford RobotCar as target
domain, and testing them on three other datasets: Pitts30k [27], the revisited
version of Tokyo24/7 [26] (RTokyo) proposed in the supplementary material of
[28], and the recent MapillarySLS [28] validation set (since the test set
labels have not yet been released). In all experiments we used the standard
VPR metric Recall@N [2], considering a retrieved gallery image as positive if
it is within 25 meters from the query.

Method Overcast/Rain 1 / 5 / 10 Overcast/Snow 1 / 5 / 10 Overcast/Sun 1 / 5 /
10 Overcast/Night 1 / 5 / 10 Avg 1 / 5 / 10 Avg GeM 71.4/ 83.8 / 87.9 37.6 /
55.1 / 63.6 30.9 / 47.9 / 55.7 4.4 / 12.2 / 18.3 36.1 / 49.7 / 56.4 47.4 GeM +
DA 73.6 / 87.4 / 91.2 41.2 / 59.9 / 68.6 32.6 / 52.0 / 60.4 7.3 / 20.5 / 29.4
38.7 / 54.9 / 62.4 52.0 GeM $\star$ 69.6 / 83.8 / 88.3 40.3 / 58.4 / 66.3 32.7
/ 51.2 / 61.1 6.0 / 16.5 / 23.6 37.2 / 52.5 / 59.8 49.8 GeM + DA $\star$ 78.7
/ 90.8 / 93.7 43.8 / 64.7 / 72.8 33.8 / 54.3 / 64.1 10.2 / 25.2 / 34.2 41.6 /
58.7 / 66.2 55.5 RMAC 73.9 / 86.9 / 90.9 35.1 / 53.5 / 61.6 22.7 / 39.9 / 48.0
2.5 / 8.2 / 13.0 33.6 / 47.1 / 53.4 44.7 RMAC + DA 69.8 / 83.2 / 88.0 39.8 /
58.4 / 66.5 24.9 / 42.6 / 51.7 3.9 / 10.3 / 16.3 34.6 / 48.6 / 55.6 46.3 RMAC
$\star$ 68.6 / 82.5 / 87.0 35.6 / 52.8 / 60.3 27.0 / 43.4 / 52.6 2.9 / 10.0 /
15.8 33.5 / 47.2 / 53.9 44.9 RMAC + DA $\star$ 65.3 / 80.0 / 84.6 32.0 / 52.1
/ 61.6 25.6 / 44.0 / 54.2 3.3 / 9.5 / 14.9 31.5 / 46.4 / 53.8 43.9
DASGIL_nodepth 87.2 / 92.9 / 94.5 39.3 / 55.3 / 63.1 20.8 / 33.3 / 40.6 6.5 /
14.2 / 19.5 38.4 / 48.9 / 54.4 47.3 DASGIL_FD $\bullet$ 76.8 / 83.7 / 86.4
53.4 / 65.0 / 70.2 46.5 / 59.3 / 65.0 2.0 / 6.3 / 11.4 44.7 / 53.6 / 58.3 52.2
DASGIL_CD $\bullet$ 75.3 / 84.7 / 87.9 34.0 / 45.7 / 51.9 12.0 / 21.7 / 28.0
1.0 / 4.1 / 6.4 30.6 / 39.1 / 43.6 37.7 Ours w DL 92.2 / 96.8 / 97.8 68.0 /
83.4 / 89.0 62.6 / 79.6 / 85.0 11.3 / 24.5 / 34.1 58.5 / 71.1 / 76.5 68.7 Ours
w DL $\star$ 90.5 / 96.9 / 97.7 62.5 / 79.6 / 85.7 55.1 / 73.7 / 81.0 10.8 /
25.3 / 34.7 54.7 / 68.9 / 74.8 66.1 Ours w PSP 90.7 / 96.5 / 97.3 59.5 / 75.5
/ 82.4 47.0 / 65.2 / 71.9 12.0 / 26.4 / 34.7 52.3 / 65.9 / 71.6 63.3 Ours w
PSP $\star$ 91.2 / 96.4 / 97.7 61.4 / 83.1 / 88.4 55.4 / 74.0 / 81.0 14.2 /
32.6 / 42.3 55.6 / 71.5 / 77.4 68.2

Table 1: Results on the Oxford RobotCar [16], using Overcast as gallery.
$\star$ means ResNet101 as encoder, all the others are with ResNet50. DA stays
for Domain Adaptation, DL is for DeepLabv2 [5] segmentation decoder, while PSP
is for PSPNet [32]. $\bullet$ indicates the experiments of [11] with the
trained models provided by the authors.

###  5.1 Results

The results of the experiments are reported in Tab. 1. We observe that all
methods perform worse when tested on queries from the Night and Sun scenarios,
due to a stronger visual dissimilarity with respect to the gallery images that
are taken from the Overcast scenario.

Overall, we see that our architecture largely outperforms all other methods,
with and without DA, from a minimum of 8% to a maximum of 24%. Curiously, with
RMAC [25] results seem to get worse when adding DA. We do not have a
definitive explanation for this negative effect, but we confirmed it with the
generalization experiments presented in Tab. 2. Even DASGIL, which is the most
similar method to our work, Tab. 1 struggles to generalize to all scenarios.
The model trained on our datasets and without the depth information
(DASGIL_nodepth) performs very well on the Rain scenario, but it still results
on average worse than the GeM. We suspect that this method suffers
considerably from the large domain gap between our synthetic dataset and the
RobotCar [16] target images. For this reason we extend the experiments by
using also the pretrained models provided by the authors (DASGIL_FD $\bullet$
and DASGIL_CD $\bullet$), that are trained on synthetic and real datasets
Virtual KITTI 2 [4] and KITTI [8]. Nevertheless, these implementations still
remain lower than our method by at least 11%. We further confirm our
assumptions in Tab. 2 testing the generalization capability on very large city
datasets. Finally, the generalization results shown in Tab. 2 demonstrate that
our solution generalizes to unseen domains better than all the other methods
by usually a large margin. Qualitative results are provided in the
supplementary material.

Method Pitts30k - Val 1 / 5 / 10 Pitts30k - Test 1 / 5 / 10 RTokyo - Val 1 / 5
/ 10 RTokyo - Test 1 / 5 / 10 MapillarySLS 1 / 5 / 10 GeM 37.6 / 56.5 / 64.9
39.5 / 59.1 / 67.3 28.7 / 43.5 / 51.2 8.1 / 17.4 / 24.8 26.5 / 38.6 / 44.5 GeM
+ DA 41.1 / 61.0 / 69.0 41.7 / 61.6 / 69.7 33.7 / 49.4 / 57.0 13.4 / 25.7 /
31.6 27.2 / 40.0 / 46.1 GeM $\star$ 38.0 / 57.9 / 67.4 38.8 / 58.5 / 67.3 26.4
/ 41.5 / 49.2 10.1 / 20.4 / 27.4 25.7 / 39.5 / 45.0 GeM + DA $\star$ 47.1 /
68.9 / 77.3 47.7 / 68.4 / 75.9 36.0 / 52.6 / 60.3 17.6 / 34.6 / 43.5 28.9 /
41.5 / 48.4 RMAC 39.4 / 58.5 / 67.3 43.3 / 62.9 / 71.1 40.1 / 55.7 / 62.6 13.8
/ 28.3 / 36.9 33.6 / 44.3 / 50.6 RMAC + DA 37.8 / 57.2 / 66.1 41.9 / 62.1 /
71.3 36.3 / 51.7 / 59.0 10.4 / 23.8 / 31.5 28.8 / 40.4 / 46.7 RMAC $\star$
33.8 / 53.5 / 63.0 37.7 / 57.4 / 66.9 31.7 / 46.9 / 54.6 10.0 / 20.9 / 27.7
31.5 / 44.0 / 48.5 RMAC + DA $\star$ 32.1 / 51.3 / 60.6 36.4 / 56.3 / 65.3
30.5 / 45.8 / 53.4 14.8 / 25.7 / 33.0 30.2 / 41.4 / 46.2 DASGIL_nodepth 11.1 /
17.6 / 21.6 12.8 / 20.6 / 26.0 17.7 / 28.7 / 35.0 2.0 / 3.8 / 6.4 11.1 / 16.6
/ 19.7 DASGIL_FD $\bullet$ 8.3 / 12.6 / 15.3 8.7 / 13.8 / 16.6 5.9 / 9.1 /
11.9 0.0 / 0.7 / 1.0 6.4 / 8.6 / 10.1 DASGIL_CD $\bullet$ 6.8 / 10.7 / 13.9
8.5 / 12.3 / 15.8 7.2 / 11.9 / 15.3 0.3 / 1.0 / 1.0 7.7 / 11.4 / 14.1 Ours w
DL 56.3 / 73.8 / 80.1 58.9 / 75.2 / 80.4 49.6 / 64.1 / 70.3 20.7 / 37.1 / 45.5
34.6 / 45.8 / 52.3 Ours w DL $\star$ 57.9 / 76.3 / 82.8 59.4 / 76.2 / 81.5
49.0 / 64.0 / 70.3 21.4 / 38.0 / 45.0 34.5 / 46.8 / 53.0 Ours w PSP 52.9 /
70.9 / 77.9 56.3 / 73.0 / 79.0 47.3 / 62.1 / 68.6 17.4 / 30.9 / 38.6 33.0 /
47.2 / 53.0 Ours w PSP $\star$ 57.7 / 76.4 / 83.3 59.1 / 76.6 / 82.2 51.3 /
66.2 / 72.5 26.9 / 44.6 / 51.9 32.9 / 46.1 / 51.9

Table 2: Results using Pitts30k [27] and RTokyo [28] validation and test sets,
and MapillarySLS [28] validation cities. * means ResNet101 as encoder while
all the others are with ResNet50. DA stays for Domain Adaptation, DL means
DeepLabv2 [5] segmentation decoder and PSP is for PSPNet [32]. $\bullet$
indicates the experiments of [11] with the trained models provided by the
authors.

###  5.2 Ablation studies

We perform extensive experiments to assess the impact of the various modules
in our architecture. Due to lack of space, these experiments and the related
discussion of results are reported in the supplementary material.

##  6 Conclusions

We have presented a new method for generating global descriptors for VPR,
exploiting both visual appearance and semantic features at different scales.
Our solution is founded on the intuition that not all the semantic content is
useful for VPR. Unlike previous works that select the semantic information in
a top-down manner, we let the model determine what semantic information to
use, in a data driven way. The key for this, is an attention mechanism that
lets the VPR guide the semantic segmentation. Experiments on well-known VPR
benchmarks, where we surpass the current state-of-the-art methods, validate
our intuition and architecture. We also show that our model generalizes well
to unseen target domains. Finally, we contribute a new dataset, rich of RGB
images under different conditions, pixel-wise semantic masks and GPS
coordinate, that is instrumental to explore the connection between semantics
and appearance in VPR task, and that we believe will be useful to the research
community. Following acceptance, we will make the dataset and code publicly
available.

##  7 Supplementary Material

###  7.1 Implementation details.

The experiments with our architecture reported in the main paper are conducted
using the ResNet50 and the ResNet101 (pretrained on ImageNet) as the shared
encoder, which is truncated before the last average pooling. We trained our
method with DeepLabv2 [5] and PSPNet [32] as SemSeg decoder. Training is
performed using the 17 classes in common with the Cityscapes standard in
semantic segmentation. The domain discriminator consists of 5 convolutional
layers with kernel $4\times 4$, stride 2, padding 1 and channel numbers {64,
128, 256, 512, 1}. Each layer, except the last one is followed by a Leaky ReLU
activation function with a negative slope of 0.2. The SemSeg decoder is
initialized with the normal distribution, while the multi-scale attention
module with the Xavier initialization. The encoder, the SemSeg decoder and the
multi-scale attention layer are trained with SGD with an initial learning rate
of 1e-4. The domain discriminator is trained with Adam with an initial
learning rate of 4e-4. The ”poly” learning rate decay with a power of 0.9,
momentum 0.9 and weight decay to 0.0005 is used for all the modules. $\alpha$
and $\gamma$ are fixed to 0.5 while $\beta$ is set to 0.0005. We used the
caching mechanisms as in [2] to refresh the gallery embeddings during the
training. We pre-process the triplets with random-crops and horizontal-flips.
The size of training images is 768x432. For fairness of comparison, all the
experiments (baselines and ours) are validated on the source domain using the
left/right views of Town3, not considering the accuracy on the real world
images as a metric to stop the training. The overall network is trained end-
to-end, while no image pre-processing and no whitening-PCA is performed for
the evaluation phase.

###  7.2 Ablation studies

Impact of different modules. We performed an extensive ablation study on the
RobotCar [16] scenarios to evaluate the impact of each component of our
method. For this study we used ResNet50 as encoder and DeepLabv2 [5] as SemSeg
decoder. As baseline, we consider the shared encoder followed by the GeM [21]
pooling layer. Then, we added each component, i.e. , the multi-scale attention
block (Att), the segmentation decoder (SemSeg), the attention to guide the
segmentation (G-SemSeg) and the unsupervised domain adaptation mechanism (DA),
trying all combinations. As shown in LABEL:{tab:ablation_extra} each module
improves the results, in particular across the single-scale embedding
experiments even without the domain adaptation (DA) task, the semantic
segmentation (SemSeg) and the guided configuration (G-SemSeg) provide strong
boosts respectively of +10% and +14% on the average. Finally, we build the
final architecture that adds up to an overall improvement of approximately 17%
over the baseline. Moreover, adding our novel multi-scale embeddings (ms-GeM)
gives a further gain of 4%.

ms-GeM Att SemSeg G-SemSeg DA Overcast/Rain 1 / 5 / 10 Overcast/Snow 1 / 5 /
10 Overcast/Sun 1 / 5 / 10 Overcast/Night 1 / 5 / 10 Avg 1 / 5 / 10 Avg 71.4 /
83.8 / 87.9 37.6 / 55.1 / 63.6 30.9 / 47.9 / 55.7 4.4 / 12.2 / 18.3 36.1 /
49.7 / 56.4 47.4 $\checkmark$ 71.4 / 83.7 / 88.0 40.6 / 59.3 / 67.2 31.1 /
50.3 / 59.1 4.9 / 15.1 / 21.9 37.0 / 52.1 / 59.1 49.4 $\checkmark$
$\checkmark$ 82.7 / 93.3 / 95.5 50.0 / 70.1 / 79.2 45.5 / 66.2 / 74.4 7.5 /
17.9 / 25.9 46.4 / 61.9 / 68.8 59.0 $\checkmark$ $\checkmark$ 86.8 / 93.8 /
96.2 56.1 / 73.6 / 80.6 43.8 / 64.0 / 72.6 6.5 / 16.9 / 25.4 48.3 / 62.1 /
68.7 59.7 $\checkmark$ $\checkmark$ $\checkmark$ 81.4 / 92.3 / 94.8 53.6 /
73.3 / 81.3 44.1 / 64.6 / 72.9 10.2 / 24.6 / 34.7 47.3 / 63.7 / 70.9 60.6
$\checkmark$ $\checkmark$ $\checkmark$ 87.6 / 95.1 / 96.7 56.3 / 75.7 / 83.0
50.3 / 69.6 / 84.1 8.8 / 20.2 / 29.2 50.8 / 65.1 / 73.3 63.1 $\checkmark$
$\checkmark$ $\checkmark$ $\checkmark$ 85.2 / 94.1 / 96.3 59.5 / 78.7 / 85.3
50.5 / 69.1 / 76.9 11.7 / 27.2 / 37.3 51.7 / 67.3 / 73.9 64.3 $\checkmark$
85.8 / 93.4 / 95.6 56.5 / 72.6 / 79.1 51.0 / 69.5 / 76.6 7.4 / 20.0 / 29.2
50.2 / 63.9 / 70.1 61.4 $\checkmark$ $\checkmark$ 90.5 / 96.3 / 97.5 59.2 /
75.9 / 82.6 50.1 / 67.8 / 75.2 5.8 / 16.4 / 25.7 51.4 / 64.1 / 70.3 61.9
$\checkmark$ $\checkmark$ $\checkmark$ 91.1 / 96.0 / 97.0 55.6 / 72.7 / 79.5
54.8 / 72.2 / 79.2 7.1 / 17.4 / 25.5 52.2 / 64.6 / 70.3 62.4 $\checkmark$
$\checkmark$ $\checkmark$ 87.6 / 94.8 / 96.6 60.3 / 78.0 / 84.1 48.9 / 68.7 /
75.7 7.9 / 20.2 / 29.0 51.2 / 65.4 / 71.3 62.6 $\checkmark$ $\checkmark$
$\checkmark$ $\checkmark$ 89.9 / 96.0 / 97.3 63.1 / 80.4 / 86.4 57.9 / 75.6 /
81.6 7.7 / 18.6 / 27.7 54.6 / 67.6 / 73.3 65.2 $\checkmark$ $\checkmark$
$\checkmark$ $\checkmark$ 91.5 / 96.4 / 97.3 58.4 / 74.6 / 80.6 53.9 / 72.2 /
78.9 7.9 / 20.8 / 29.8 53.0 / 66.0 / 71.7 63.5 $\checkmark$ $\checkmark$
$\checkmark$ $\checkmark$ $\checkmark$ 92.2 / 96.8 / 97.8 68.0 / 83.4 / 89.0
62.6 / 79.6 / 85.0 11.3 / 24.5 / 34.1 58.5 / 71.1 / 76.5 68.7 $\checkmark$
73.6 / 87.4 / 91.2 41.2 / 59.9 / 68.6 32.6 / 52.0 / 60.4 7.3 / 20.5 / 29.4
38.7 / 54.9 / 62.4 52.0 $\checkmark$ 83.3 / 92.7 / 95.0 46.3 / 66.1 / 75.4
40.7/ 60.9 / 69.0 8.1 / 18.8 / 26.7 44.6 / 59.6 / 66.5 56.9 $\checkmark$
$\checkmark$ 81.6 / 92.2 / 94.7 52.7 / 72.6 / 80.3 45.5 / 65.8 / 74.2 8.4 /
20.7 / 30.7 47.1 / 62.8 / 70.0 60.0 $\checkmark$ $\checkmark$ 85.7 / 94.1 /
95.9 59.6 / 76.9 / 83.6 45.8 / 67.3 / 74.8 5.2 / 16.4/ 25.4 49.1 / 63.7 / 69.9
60.9 $\checkmark$ $\checkmark$ 88.9 / 95.5 / 96.8 58.4 / 75.3 / 82.1 54.0 /
72.8 / 80.1 7.7 / 18.2 / 25.8 52.3 / 65.5 / 71.2 63.0 $\checkmark$
$\checkmark$ $\checkmark$ 91.4 / 96.8 / 97.6 61.7 / 78.9 / 85.7 57.4 / 75.2 /
81.5 7.7 / 20.2 / 29.4 54.6 / 67.8 / 73.6 65.3

Table 3: Ablation study to assess the contribution of each component in our
architecture. _ms-GeM_ stays for Multi-Scale Embeddings obtained through our
new multi-scale pooling layer, _Att_ indicates that the attention module is
used in the VPR task, _SemSeg_ and _G-SemSeg_ denote that Semantic
Segmentation and Guided Semantic Segmentation are active, while _DA_ stands
for the Domain Adaptation task.

Multi-scale embeddings. In Fig. 7 we present an ablation analysis of the
encoder local features that should be aggregated to produce robust embeddings.
We get the best results when we use the local features from the last two
_conv_ blocks (ms-GeM 4+5), which have a higher level of abstraction and are
useful for distinguishing different classes. Local features from previous
layers, on the other hand, create global embeddings that are inadequate to
accurately represent the places.

Figure 7: Average recalls across Rain, Snow, Sun and Night queries of RobotCar
[16], getting local features at different scale.

###  7.3 Qualitative results

Fig. 8, Fig. 9(a), Fig. 10(a), Fig. 11 provide a qualitative comparison
between our framework and the best competitors tested in the companion paper:
the best baseline (GeM [21] \+ DA $\star$) and DASGIL_nodepth [11]. We use
_Ours w PSP $\star$_ configuration, testing on Oxford RobotCar [16] scenarios,
Pitts30k [25] and RTokyo [28] (as in the Sec. 5, 6 of the main paper). All of
the provided results confirm the assumption that by conditioning the semantic
segmentation on the place recognition task our model learns to draw
information only from the semantic categories that are most discriminative for
a place, thereby outperforming the current state of the art on all the tested
scenarios.

###  7.4 Extra experiments

In this section, we propose additional analysis comparing our method to
NetVLAD [2], a commonly used baseline in Visual Place Recognition (VPR), as
well as other experiments involving our architecture and baselines trained on
real-world datasets.

#### Comparisons with NetVLAD.

Method Oxford RobotCar Avg 1 / 5 / 10 Avg RTokyo - Test 1 / 5 / 10 Avg
MapillarySLS 1 / 5 / 10 Avg NetVLAD [2] 43.6 / 58.5 / 65.7 55.9 19.2 / 38.7 /
46.8 34.9 14.9 / 24.9 / 29.2 23.0 NetVLAD [2] \+ DA 50.5 / 63.9 / 73.5 62.6
20.5 / 40.1 / 47.8 36.1 11.8 / 20.8 / 25.6 19.4 NetVLAD [2] $\star$ 45.7 /
61.3 / 68.9 58.6 16.8 / 34.7 / 38.7 30.1 23.1 / 35.3 / 41.5 33.3 NetVLAD [2]
\+ DA $\star$ 53.3 / 67.3 / 75.5 65.3 23.6 / 41.4 / 52.9 39.3 32.6 / 46.4 /
51.8 43.6 Ours w DL [5] 58.5 / 71.1 / 76.5 68.7 20.7 / 37.1 / 45.5 34.4 34.6 /
45.8 / 52.3 44.2 Ours w DL [5] $\star$ 54.7 / 68.9 / 74.8 66.1 21.4 / 38.0 /
45.0 34.8 34.5 / 46.8 / 53.0 44.8 Ours w PSP [32] 52.3 / 65.9 / 71.6 63.3 17.4
/ 30.9 / 38.6 29.0 33.0 / 47.2 / 53.0 44.4 Ours w PSP [32] $\star$ 55.6 / 71.5
/ 77.4 68.2 26.9 / 44.6 / 51.9 41.1 32.9 / 46.1 / 51.9 43.6

Table 4: Additional experiments involving our novel synthetic dataset and the
Oxford RobotCar scenarios [16] (used for the DA branch). All of the
experiments employ a ResNet50 as the backbone, with the exception of $\star$,
which uses a ResNet101. DA stands for the domain adaptation branch. DeepLab V2
[5] is abbreviated as DL and PSPNet is abbreviated as PSP [32]. The results to
the left of the double line indicate the average recall over the RobotCar
scenarios (Rain, Snow, Sun, and Night as the query set, whereas Overcast as
the gallery), while the results to the right of the double line show the
models’ ability to generalize to two previously unseen datasets, RTokyo [28]
and MapillarySLS [28], respectively.

We extended to NetVLAD [2] the experimental setup described in paragraphs 5.1
and 5.2 of the companion paper. As encoders, we used ResNet50 and ResNet101,
and 64 cluster centroids inside the NetVLAD pooling layer, then trained the
model using our newly synthetic dataset. Similarly to what we did for GeM [21]
and RMAC [25], we replicated the configurations of [2], adding unsupervised
domain adaptation to reduce the shift between real and synthetic data. For
this purpose, we leverage all the scenarios of the Oxford RobotCar dataset
[16]: Rain, Snow, Sun and Night.

Tab. 4 presents two experimental groups and shows the 1/5/10 recall as well as
the overall average accuracy. The first set of experiments (to the left of the
double line) refers to the validation on RobotCar [16], which adheres to the
same protocol as section 5 of the main paper. The Overcast scenario serves as
the gallery, and the Rain, Snow, Sun, and Night scenarios serve as the query
set. The results demonstrate how our strategy outperforms the baseline by
$+3.4\%$. The second set of experiments (to the right of the double line) is
designed to demonstrate the generalization capability of our model over images
never seen during training. We evaluate the effectiveness of our approach on
the two RTokyo (a revisited version of Tokyo24/7 [26] proposed by [28]) and
MapillarySLS [28] datasets. Also in this case, the results demonstrate the
usefulness of our strategy in generalizing to a whole new target, yielding a
boost of $+1.8\%$ and $+1.2\%$ to the two evaluated datasets, respectively.
Note that the architecture proposed by [2] produces $131072$-D (dimensional)
descriptors w.r.t the more compact (around 43 times lower) $3072$-D
descriptors produced by our method. Descriptors with these proportions
representing large datasets, such as RTokyo or MapillarySLS, not only require
a large amount of RAM to be saved concurrently, but also increase the time
required to complete the retrieval (k-NN) between query and gallery. Tab. 5
confirms this by reporting the expected inference time (descriptor extraction
and k-NN) using the same resources (NVIDIA TITAN X GPU and Intel Core i7-5930K
CPU @ 3.50GHz CPU) and full resolution dataset.

Method Descriptor dimension RTokyo 64608 Gallery Imgs 247 Query Imgs
MapillarySLS 18920 Gallery Imgs 11120 Query Imgs NetVLAD [2] 131072-D 1:25:19
0:38:53 Ours 3072-D 0:35:13 0:16:43

Table 5: Comparison of the inference time [h:mm:ss] between NetVLAD [2] and
our technique over the RTokyo [28] and MapillarySLS [28] test sets.

Method Pitts30k - Val 1 / 5 / 10 Pitts30k - Test 1 / 5 / 10 GeM [21] 75.2 /
90.5 / 93.9 73.3 / 86.9 / 90.7 RMAC [25] 73.5 / 90.5 / 94.6 74.1 / 88.5 / 92.7
NetVLAD [2] 79.5 / 93.0 / 96.1 77.3 / 88.8 / 92.1 Ours w DL [5] 81.9 / 92.7 /
95.8 79.8 / 90.7 / 93.5

Table 6: Experiments using Pitts30k [27] training set previously segmented
with PSPNet [32] pretrained on ADE20k [33]. Evaluation results using Pitts30k
[27] validation and test sets.

#### Real-world dataset with imperfect semantic labels.

Despite the usage of a domain adaptation module, the results with all the
models trained on the synthetic dataset suffer from the considerable gap with
the real-word target domain. To demonstrate that our solution provides a
performance uplift against the comparison even when all the methods are
trained on real-world data, we conducted a series of experiments on a real-
world VPR dataset coarsely annotated using a pretrained semantic segmentation
model. More specifically, we annotated the Pitts30k [27] train set using a
PSPNet [32] model, with a ResNet50 as a backbone, pretrained on ADE20k [33].
Note that this is a disadvantageous scenario for our method because of the
imperfect semantic labels, which limit how well the model can learn to exploit
the semantic content of the scene. Tab. 6 presents the results of GeM [21],
RMAC [25], NetVLAD [2], and our novel approach. Nonetheless, our work remains
cutting-edge, demonstrating that its usage of the semantic information guided
by the place recognition task holds promise to achieve even better results
when a real-world dataset with precise pixel-wise semantic labels becomes
available.

Figure 8: Qualitative predictions testing on Oxford RobotCar [16] with queries
from Rain scenario and gallery from Overcast scenario. (a) (b) Figure 9:
Qualitative predictions testing on Oxford RobotCar [16] with queries from Snow
(a) and Sun (b) scenarios, while the gallery is always composed by the
Overcast scenario.

(a) (b) Figure 10: Qualitative predictions testing on: a) Oxford RobotCar [16]
with queries from Night scenario and gallery from Overcast; b) Pitts30k [27].

Figure 11: Qualitative predictions testing on RTokyo [28]

## References

  * [1] Alberti, E., Tavera, A., Masone, C., Caputo, B.: Idda: A large-scale multi-domain dataset for autonomous driving. IEEE Robot. and Autom. Lett. 5(4), 5526–5533 (2020). https://doi.org/10.1109/LRA.2020.3009075
  * [2] Arandjelović, R., Gronat, P., Torii, A., Pajdla, T., Sivic, J.: Netvlad: Cnn architecture for weakly supervised place recognition. IEEE Trans. Pattern Anal. Mach. Intell. 40(6), 1437–1451 (2018). https://doi.org/10.1109/TPAMI.2017.2711011
  * [3] Berton, G.M., Paolicelli, V., Masone, C., Caputo, B.: Adaptive-attentive geolocalization from few queries: A hybrid approach. In: IEEE Winter Conf. Appl. Comput. Vis. pp. 2918–2927 (January 2021)
  * [4] Cabon, Y., Murray, N., Humenberger, M.: Virtual KITTI 2 (2020)
  * [5] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell. 40, 834–848 (2018)
  * [6] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: IEEE Conf. Comput. Vis. Pattern Recog. (2016)
  * [7] Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: An open urban driving simulator. In: Levine, S., Vanhoucke, V., Goldberg, K. (eds.) Proceedings of the 1st Annual Conference on Robot Learning. Proceedings of Machine Learning Research, vol. 78, pp. 1–16. PMLR (13–15 Nov 2017), http://proceedings.mlr.press/v78/dosovitskiy17a.html
  * [8] Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? the kitti vision benchmark suite. In: IEEE Conf. Comput. Vis. Pattern Recog. (2012)
  * [9] Geyer, J., Kassahun, Y., Mahmudi, M., Ricou, X., Durgesh, R., Chung, A.S., Hauswald, L., Pham, V.H., Mühlegg, M., Dorn, S., Fernandez, T., Jänicke, M., Mirashi, S., Savani, C., Sturm, M., Vorobiov, O., Oelker, M., Garreis, S., Schuberth, P.: A2D2: Audi Autonomous Driving Dataset (2020), https://www.a2d2.audi
  * [10] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90
  * [11] Hu, H., Qiao, Z., Cheng, M., Liu, Z., Wang, H.: DASGIL: domain adaptation for semantic and geometric-aware image-based localization. IEEE Trans. Image Process. 30, 1342–1353 (2021). https://doi.org/10.1109/TIP.2020.3043875, https://doi.org/10.1109/TIP.2020.3043875
  * [12] Huang, X., Cheng, X., Geng, Q., Cao, B., Zhou, D., Wang, P., Lin, Y., Yang, R.: The apolloscape dataset for autonomous driving. In: IEEE Conf. Comput. Vis. Pattern Recog. (June 2018)
  * [13] Jenicek, T., Chum, O.: No fear of the dark: Image retrieval under varying illumination conditions. In: Int. Conf. Comput. Vis. (2019)
  * [14] Kim, H.J., Dunn, E., Frahm, J.: Learned contextual feature reweighting for image geo-localization. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 3251–3260 (2017). https://doi.org/10.1109/CVPR.2017.346
  * [15] Larsson, M., Stenborg, E., Toft, C., Hammarstrand, L., Sattler, T., Kahl, F.: Fine-grained segmentation networks: Self-supervised segmentation for improved long-term visual localization. In: Int. Conf. Comput. Vis. (October 2019)
  * [16] Maddern, W., Pascoe, G., Linegar, C., Newman, P.: 1 Year, 1000km: The Oxford RobotCar Dataset. The International Journal of Robotics Research 36(1), 3–15 (2017). https://doi.org/10.1177/0278364916679498
  * [17] Masone, C., Caputo, B.: A survey on deep visual place recognition. IEEE Access 9, 19516–19547 (2021). https://doi.org/10.1109/ACCESS.2021.3054937
  * [18] Mousavian, A., Košecká, J., Jyh-Ming Lien: Semantically guided location recognition for outdoors scenes. In: IEEE Int. Conf. on Robotics and Automation. pp. 4882–4889 (2015). https://doi.org/10.1109/ICRA.2015.7139877
  * [19] Naseer, T., Oliveira, G., Brox, T., Burgard, W.: Semantics-aware visual localization under challenging perceptual conditions. In: IEEE Int. Conf. on Robotics and Automation. IEEE (2017), http://lmb.informatik.uni-freiburg.de/Publications/2017/OB17
  * [20] Pion, N., Humenberger, M., Csurka, G., Cabon, Y., Sattler, T.: Benchmarking image retrieval for visual localization. In: 2020 International Conference on 3D Vision (3DV). pp. 483–494 (2020)
  * [21] Radenović, F., Tolias, G., Chum, O.: Fine-tuning cnn image retrieval with no human annotation. IEEE Trans. Pattern Anal. Mach. Intell. 41(7), 1655–1668 (2019). https://doi.org/10.1109/TPAMI.2018.2846566
  * [22] Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. In: Bengio, Y., LeCun, Y. (eds.) Int. Conf. Learn. Represent. (2016), http://arxiv.org/abs/1511.06434
  * [23] Richter, S.R., Vineet, V., Roth, S., Koltun, V.: Playing for data: Ground truth from computer games. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Eur. Conf. Comput. Vis. LNCS, vol. 9906, pp. 102–118. Springer International Publishing (2016)
  * [24] Ros, G., Sellart, L., Materzynska, J., Vazquez, D., Lopez, A.M.: The SYNTHIA dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 3234–3243 (June 2016)
  * [25] Tolias, G., Sicre, R., Jégou, H.: Particular Object Retrieval With Integral Max-Pooling of CNN Activations. In: Int. Conf. Learn. Represent. pp. 1–12. Int. Conf. Learn. Represent., San Juan, Puerto Rico (May 2016), https://hal.inria.fr/hal-01842218
  * [26] Torii, A., Arandjelović, R., Sivic, J., Okutomi, M., Pajdla, T.: 24/7 place recognition by view synthesis. In: IEEE Conf. Comput. Vis. Pattern Recog. (2015)
  * [27] Torii, A., Sivic, J., Okutomi, M., Pajdla, T.: Visual place recognition with repetitive structures. IEEE Trans. Pattern Anal. Mach. Intell. 37(11), 2346–2359 (2015)
  * [28] Warburg, F., Hauberg, S., Lopez-Antequera, M., Gargallo, P., Kuang, Y., Civera, J.: Mapillary street-level sequences: A dataset for lifelong place recognition. In: IEEE Conf. Comput. Vis. Pattern Recog. (June 2020)
  * [29] Xin, Z., Cai, Y., Lu, T., Xing, X., Cai, S., Zhang, J., Yang, Y., Wang, Y.: Localizing discriminative visual landmarks for place recognition. In: IEEE Int. Conf. on Robotics and Automation. pp. 5979–5985 (2019). https://doi.org/10.1109/ICRA.2019.8794383
  * [30] Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan, V., Darrell, T.: Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In: IEEE Conf. Comput. Vis. Pattern Recog. (June 2020)
  * [31] Zaffar, M., Khaliq, A., Ehsan, S., Milford, M., McDonald-Maier, K.: Levelling the playing field: A comprehensive comparison of visual place recognition approaches under changing condition. In: IEEE Int. Conf. on Robotics and Automation Worksh. pp. 1–8 (2019)
  * [32] Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 6230–6239 (2017). https://doi.org/10.1109/CVPR.2017.660
  * [33] Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ade20k dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)
  * [34] Zhu, Y., Wang, J., Xie, L., Zheng, L.: Attention-based pyramid aggregation network for visual place recognition. In: ACM Int. Conf. Multimedia. p. 99–107. MM ’18, Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3240508.3240525, https://doi.org/10.1145/3240508.3240525