11institutetext: Visual Recognition Group, FEE, Czech Technical University in
Prague
11email: {weitong<EMAIL_ADDRESS>22institutetext: Computer Vision and
Geometry Group, ETH Zurich
22email: {danielbela.barath<EMAIL_ADDRESS>
# Breaking the Frame:
Image Retrieval by Visual Overlap Prediction

Tong Wei 11 Philipp Lindenberger 22 Jiří Matas 11 Daniel Barath 22

###### Abstract

We propose a novel visual place recognition approach, VOP, that efficiently
addresses occlusions and complex scenes by shifting from traditional reliance
on global image similarities and local features to image overlap prediction.
The proposed method enables the identification of visible image sections
without requiring expensive feature detection and matching. By focusing on
obtaining patch-level embeddings by a Vision Transformer backbone and
establishing patch-to-patch correspondences, our approach uses a voting
mechanism to assess overlap scores for potential database images, thereby
providing a nuanced image retrieval metric in challenging scenarios. VOP leads
to more accurate relative pose estimation and localization results on the
retrieved image pairs than state-of-the-art baselines on a number of large-
scale, real-world datasets. The code is available at
https://github.com/weitong8591/vop.

###### Keywords:

image retrieval image overlap relative camera pose vision transformer

##  1 Introduction

Image retrieval algorithms, particularly those concerning visual place
recognition (VPR), play an essential role in computer vision and robotics [17,
15] problems. Place recognition is defined as the task of identifying the
approximate location where a query image was taken, given a certain tolerance
level [43, 77, 19, 2, 28, 85, 78, 24, 27, 36, 6, 31, 83]. This capability is
crucial for estimating the state of robots and is widely utilized in
autonomous, unmanned aerial, terrestrial, and underwater vehicles, as well as
recent AR/VR devices.

Typically, VPR is approached as an image retrieval problem, where the image to
be localized is compared against a large database of posed images and,
optionally, a 3D reconstruction of the scene. The most similar images
retrieved from the database are used to estimate the precise location of the
input query image, _e.g_., via local feature detection and matching. The
complexity of VPR stems from a variety of factors such as perceptual aliasing,
seasonal variations, changes in viewpoint, the presence of dynamic objects,
illumination differences, occlusions, weather conditions, and intricate
environments, as highlighted in the studies by Hong _et al_.[30], Doan _et
al_.[20], and Subramaniam _et al_. [72].

Recent approaches conceptualize VPR as a task of estimating image similarity,
conducting retrieval through learned image embeddings generated by a feature
extraction architecture augmented with an aggregation or pooling mechanism,
the most notable method being NetVLAD [2]. In this framework, images from the
database are selected for retrieval based on their similarity to the query
image, for example, by optimizing the cosine similarity between their
embeddings. One notable drawback of this method is its failure to address
partial overlaps resulting from occlusions. In practice, occlusions can
significantly reduce image similarity scores, despite the unobstructed
portions of the scene potentially offering valuable cues for localization.

To be robust to occlusions and to provide a more nuanced understanding of the
query image in relation to the database, we propose a novel approach for VPR.
Instead of relying on global similarities or local features, we frame the
problem as an overlap prediction, enabling us to methodically understand which
parts of a particular image are visible without having to perform feature
detection and matching. We approach the problem by generating embeddings for
individual image patches rather than the entire image, thereby facilitating
the efficient establishment of patch-to-patch correspondences. These matches
are subsequently utilized within a voting mechanism to calculate an overlap
score for potential database images, providing a nuanced metric for image
retrieval amidst complex visual environments.

Illustrative examples are provided in Fig. 1. In these examples, AnyLoc [33],
a state-of-the-art retrieval solution, processes a query image alongside two
database images: one sharing the scene with the query albeit with occlusion
and another from a different environment. AnyLoc erroneously attributes higher
similarity to the unrelated image. In contrast, our proposed approach
successfully identifies the correct image and the areas likely to overlap.

Figure 1: An outdoor and indoor image retrieval example of cases when image
similarity prediction by the state-of-the-art AnyLoc [33] methods fails, while
the proposed overlap prediction selects the correct database image, as
indicated by the retrieval scores visualized above/below the database images.
The middle column shows the same scene as the query (left) with heavy
occlusion. The right column depicts a different location.

The contributions of this paper are summarized as follows:

  1. 1.

A novel approach, transitioning from global image similarity to overlap
assessment which exhibits enhanced robustness to challenges such as
occlusions. The proposed approach is built on patch-level analysis by learning
patch embeddings suitable for visual localization. It implements a single
radius search across embeddings of image patches in the database.

  2. 2.

The introduction of a robust voting mechanism for image retrieval which
demonstrates improvement over conventional similarity-based methods.

Moreover, we identify a limitation inherent in the standard metric, namely
recall, for evaluating image retrieval accuracy as done in recent studies [5,
33]. Recall quantifies the success of a method in retrieving correct database
images without considering any application-specific evaluation metrics.
Consequently, we opt for the Area Under the Curve (AUC) metric, specifically
focusing on relative pose accuracy, to provide a more nuanced and relevant
performance assessment.

##  2 Related Work

Image retrieval methods for VPR can be categorized into two groups, one
globally representing the whole image and the other locally for keypoints.

Localization the process of determining the position and orientation of an
agent within a pre-built map, is pivotal across various domains such as
autonomous driving [29], drones [40], and augmented reality [13, 45]. The
differentiation in localization techniques arises from their scene
representation methods – be it through explicit 3D models [25, 26, 32, 38, 42,
46, 60, 63, 64, 66, 73, 86, 62], sets of posed images [87, 88, 56, 7], or
implicitly via neural network weights [35, 79, 81, 34, 8, 9, 4, 14, 11, 47] –
and their approach to camera pose estimation, whether by 2D-3D [25, 26, 60,
62, 64, 73, 86] or 2D-2D [88, 7] matches, or through a composite of base poses
[35, 81, 34, 65, 56, 47]. Enhanced robustness is often sought through
sequential image analysis rather than single-frame methods, with [50] adapting
sequence matching to graph search and [80] developing an online sequence
matching technique for real-time localization. In practice, localization
comprises two main steps: place recognition and precise localization stages.
Here, we focus on the first step, finding potential locations of a query
image.

Place recognition is often cast as an image retrieval problem [23, 2, 6, 28,
31, 37, 53, 54, 83, 5, 33] that consists of two phases. In the offline
indexing phase, a reference map represented by an image database is created.
In the online retrieval phase, a query image, taken during a later traverse of
the environment, is coarsely localized by identifying its closest matches
within the reference map.

For measuring closeness, traditional methods like Bag of Visual Words [16]
(BoW) identify key local image patches and store them as a codebook. BoW is
often used in conjunction with 128-dimensional SIFT [44] descriptors both for
map creation and image retrieval. These high-dimensional features often
require approximate nearest neighbor (NN) search or GPUs to accelerate the
search process. Showcasing the unbroken popularity of such approaches, they
are still widely employed in applications, _e.g_., in ORB-SLAM [48, 49, 12].

As demonstrated by Zheng _et al_.[89], using CNNs for outperforming hand-
crafted features in various applications, namely through compact (fixed-
length) representations, has gradually become the norm. Most recent methods
utilize learned embeddings generated by a feature extraction backbone with an
aggregation or pooling head, the most notable method being NetVLAD [2]. In
recent works, transformer-based retrieval methods [21] have been introduced,
incorporating metric learning and attention mechanisms. These methods utilize
local and global features, or a fusion of the two, to identify similar images.

Global retrieval methods integrate or combine handcrafted or learning-based
local descriptors, such as SIFT [44] through the bag-of-visual-words model
[55, 70], or employ alternative algorithms [76]. There is a growing trend
towards learning-based techniques that devise improved methods for feature
aggregation. NetVLAD [2] introduces an end-to-end trainable image
representation extractor, employing weak supervision through soft assignment
to cluster centers. Generalized Mean Pooling (GeM) [57] improves by
introducing a learnable pooling layer that generalizes both max and average
pooling, with a CNN serving as the backbone. Additionally, AP-GeM [59]
enhances ranking accuracy by minimizing a relaxed global mean Average
Precision (mAP) over a set of images. A CNN-based solution, CosPlace [5],
utilizes the same CNN backbone and GeM pooling but approaches the place
retrieval issue as a classification problem. MixVPR [1] transitions feature
maps into a compact representation space through cascade MLP blocks. More
recently, AnyLoc [33] emerges as a versatile method suitable for various image
retrieval contexts, exploring CNN-based or Vision Transformer (ViT)-based
foundational models and aggregation techniques to construct a vocabulary for
searching in a large database.

Image reranking. Visual place recognition techniques have increasingly
embraced a two-phase approach that pairs global retrieval with subsequent
reranking based on local features. MultiVLAD [3] is proposed to use the
maximum similarity between the query VLAD descriptors and the query as the
matching score. Similarly, Razavian.  _et al_. [69] rely on the average L2
distance of each query sub-patch to the reference image. Patch-NetVLAD [28],
for instance, innovates by integrating NetVLAD with region-level descriptors
and multi-scale patch tokens for enhanced geometric verification. It
effectively merges the strengths of local and global descriptors, offering
high resilience to changes in condition and viewpoint. RRT [75] and TransVPR
[82] further this development by incorporating image-level supervision and
attention-based mechanisms, respectively, focusing on spatial relationships
and feature relevance within images. CVNet and $R^{2}$Former [90] introduce
methodologies for replacing traditional geometric verification with dense
feature correlation and transformer-based reranking, focusing on the precise
alignment of features and relevance of image pairs. SuperGlobal [68]
simplifies the reranking process by relying on global features and K-nearest-
neighbors aggregation, showcasing a move towards more scalable and
computationally efficient VPR solutions. These advancements highlight a trend
toward utilizing machine learning techniques, including deep learning and
transformers, to refine the accuracy and efficiency of image retrieval.

##  3 Visual Overlap Prediction (VOP)

In this section, we describe the method proposed for efficiently predicting
visual overlap in large image collections.

###  3.1 Problem Statement

In place recognition problems, the aim is to find images
$I_{j}\in\mathbb{R}^{H\times W\times C}$ containing the same landmark or scene
as the query $I_{i}\in\mathbb{R}^{H\times W\times C}$ and distinguishable from
others. Usually, it is done by transforming the images to a high-dimensional
embedding space and measuring the distances to select the top-$k$ most similar
images. In this embedding space, an image $I_{i}$ is represented by a high-
dimensional global $\mathbf{g}_{i}\in\mathbb{R}^{d_{g}}$ or local descriptors
$\mathbf{d}_{i},\in\mathbb{R}^{n\times d_{l}}$, using encoder $f(\cdot)$,
where $n\in\mathbb{N}$ denotes the number of local features, and
$d_{g}\in\mathbb{N}$ is set to one for the whole image, $d_{l}\in\mathbb{N}$
to 256, 128, etc. The matching score, then, can be computed as the
similarities of the descriptors,
$s(I_{i},I_{j})=sim(\mathbf{d}_{i},\mathbf{d}_{j})$ or
$sim(\mathbf{g}_{i},\mathbf{g}_{j})$, for example, as their cosine similarity.

Instead of relying on a single score that quantifies the similarity of two
images, we focus on patch-level representations and formulate the matching
score as the overlap between the images. Let us split image $I$ into a set
$\mathcal{P}$ of rectangular patches by a uniform grid consisting of $n\times
n$ rows and columns. Given sets of such rectangular patches $\mathcal{P}_{i}$
and $\mathcal{P}_{j}$ in the two images, we can define a patch-level overlap
score as follows:

$o(p,q)=\text{volume}(\text{cone}(\mathbf{R}_{i},\mathbf{t}_{i},\mathbf{K}_{i},p),\text{cone}(\mathbf{R}_{j},\mathbf{t}_{j},\mathbf{K}_{j},q)),$
(1)

for each pair $(p,q)$, where $p\in\mathcal{P}_{i}$ and $q\in\mathcal{P}_{j}$
are the image patches, $\mathbf{R}\in\text{SO}(3)$ are the absolute rotations
of the cameras, $\mathbf{t}\in\mathbb{R}^{3}$ are their absolute positions,
$\mathbf{K}\in\mathbb{R}^{3\times 3}$ represents the intrinsic camera
parameters, function $\text{cone}(\cdot)$ represents the 3D cone defined by
the camera pose and the current patch, function $\text{volume}(\cdot,\cdot)$
measures the overlap of two posed cones in three dimensions. While this
measure is easy to calculate given the camera poses and intrinsics, we
observed that it does not reflect real-world applications, _e.g_., 3D
reconstruction. Thus, we reformulate this measure as follows:

$o^{\prime}(p,q)=\sum_{X\in\mathcal{X}}\llbracket\text{inside}(\mathbf{K}_{i}(\mathbf{R}_{i}\mathbf{X}+\mathbf{t}_{i}),p)\rrbracket*\llbracket\text{inside}(\mathbf{K}_{j}(\mathbf{R}_{j}\mathbf{X}+\mathbf{t}_{j}),q)\rrbracket,$
(2)

where $\mathcal{X}\subset\mathbb{R}^{3}$ comprises 3D points from a 3D
reconstruction of the environment, formula
$\mathbf{K}(\mathbf{R}\mathbf{X}+\mathbf{t})$ projects a 3D point to the
image, function $\text{inside}(\cdot)$ checks whether a 2D point falls into a
2D patch, $\llbracket\cdot\rrbracket$ is the Iverson bracket which equals one
if the condition inside holds and zero otherwise. Basically, Eq. 2 defines the
patch overlap as the number of 3D points commonly visible throughout both
patches. This measure will help us learn whether two image patches overlap.

The overlap of two entire images $I_{i}$ and $I_{j}$ is defined as the sum of
overlap scores across match pairs as follows:

$O(I_{i},I_{j})=\sum_{p\in\mathcal{P}_{i}}\sum_{q\in\mathcal{P}_{j}}o^{\prime}(p,q)\approx\sum_{(p,q)\in\mathcal{C}}o^{\prime}(p,q),$
(3)

where set $\mathcal{C}=\\{(p,q)\;|\;p\in\mathcal{P}_{i},q=\arg_{w}\max
o^{\prime}(p,w)\in\mathcal{P}_{j},w\in\mathcal{P}_{j}\\}$ is a set of patch-
to-patch correspondences defined by assigning the most overlapping patch from
the database image to each patch in the query image. Please note that while we
could easily define image overlap directly by Eq. 2, such a formulation does
not allow for the next steps that allow learning an overlap score.

However, this formulation still does not allow for retrieval, as in practice,
the camera pose corresponding to the query image is unknown, and our aim is to
estimate it. To do so, we further approximate the image overlap as follows:

$O(I_{i},I_{j})\approx\sum_{(p,q)\in\mathcal{C}}\rho(sim(p,q)),$ (4)

where $\rho:\mathbb{R}\to\mathbb{R}$ is a robust function. For example, $\rho$
can be defined as a voting mechanism as $\rho(x)=\llbracket
x>\epsilon\rrbracket$, where $\epsilon$ is a manually set threshold and $x$ is
the patch similarity $sim(p,q)$.

Given such a scheme to quantify image overlap via the similarity of a set of
patch correspondences, the most similar image $I^{*}$ that we retrieve for
query $I_{i}$ from a set of database images $\\{I_{j}\\},j\in 1,\dots,N$ is
the one with the highest overlap, calculated as follows:

$I^{*}=\arg_{j\in 1,\dots,N}\max O(I_{i},I_{j}).$ (5)

In the scenario where multiple images need to be retrieved for a single query,
we straightforwardly define a ranking by sorting images with high visual
overlap. Thus, image retrieval from image collections becomes a patch-matching
problem. Example matches are visualized in Fig. 2.

Figure 2: Patch matching examples in an image pair (column) with varying patch
sizes (16, 64, and 256). When using 256 patches, the embeddings are directly
provided by Dino V2 [51]. We used average pooling (16 and 64 patches) to
calculate other resolutions. The numbers inside a particular patch denote the
match index. The color overlay is calculated by applying PCA to the patch
embeddings.

###  3.2 Learning Patch Matching via Patch Similarities

In order to define the similarity of image patches $p$ and $q$ in images
$I_{i}$ and $I_{j}$, respectively, we aim to learn an embedding that allows
for casting the problem as a descriptor matching. Thus, we aim to distill
embeddings $e_{p}\in\mathbb{R}^{d}$ and $e_{q}\in\mathbb{R}^{d}$ such that
$sim(p,q)=\delta(e_{p},e_{q})$ is high if and only if $p$ and $q$ are
overlapping and zero otherwise. The function $\delta(\cdot)$ measures the
similarity in the embedding space, _e.g_., as the cosine similarity.

Figure 3:  The learning pipeline of the proposed Visual Overlap Prediction
(VOP) method is shown, with a frozen ViT backbone [51], trainable encoder
head, attention layers, and contrastive loss on the similarities. The input to
the training is image pairs that are broken down into rectangular patches to
learn patch-to-patch overlap scores. Figure 4:  The proposed Visual Overal
Prediction (VOP) method at inference. Given an input query image and an image
collection representing the reference map, the images are broken down into
rectangular patches, the frozen backbone [51] extracts patch-level features,
which are then fed into an encoder network, obtaining the final embeddings.
The image retrieval is performed by running a radius search in the embedding
space for each query patch to find potential database matches. The final
overlap scores are determined by robust voting on the formed “query
patch”-to-“database patch” correspondences. In practice, the map embeddings
are obtained in an offline.

Pipeline for learning embeddings is visualized in Fig. 3. Given a pair of
images $I_{i}$ and $I_{j}$, the problem is to learn embeddings $e_{p}$ and
$e_{q}$ as defined previously. First, we obtain patch features by inputting
the images into a Visual Transformer architecture. We use a frozen DINO V2
[51] backbone. These features are then fed into an encoder head, comprising
one linear projection layer plus a few fully connected layers with GELU
activation and dropout layer to encode the centers of the patches. Given the
corresponding patch pair $(p,q)$ as a positive sample, we train this encoder
to predict similar embeddings for $p$ and $q$. To do so, we employ self and
cross-attention layers and contrastive learning on the similarities.

Contrastive Learning. We use contrastive learning to learn a joint embedding
space for the image patches. To do so, we form patch pairs $(p,q)$. Real-world
scenes are rarely static, _e.g_., objects move or undergo non-rigid
deformations and illumination changes. To ensure that the learned embedding is
robust to such temporal changes, we use image augmentation techniques as
implemented in GlueFactory [41], such as adding random brightness, blur, flip,
and noise.

For a query patch $p$, a set of candidate patches
$\\{q,q_{1},q_{2},...,q_{n}\\}$ are provided for training, where
$\\{q_{1},q_{2},...,q_{n}\\}$ act as $n$ negative samples, depicting patches
that do not have a common field-of-view with the query patch. In our training
pipeline, half of the negative samples come from images that overlap with the
query image in general, but the selected patches do not. There are 30-90%
negative patches in those generally overlapping image pairs. The remaining set
of samples contains images capturing other scenes or other parts of the same
scene.

We train our model by optimizing both a contrastive loss and an attention-
based one as follows:

$\mathcal{L}=\alpha_{1}*\mathcal{L}_{\text{contrastive}}+\alpha_{2}*\mathcal{L}_{\text{attention}}.$
(6)

The contrastive loss is defined as follows:

$\mathcal{L}_{\text{contrastive}}=\frac{\sum_{i}^{n}\sum_{j}^{n}l_{\text{GT}}(\sigma-
sim(e_{p},e_{q}))^{2}-\sum_{i}^{n}\sum_{j}^{n}(1-l_{\text{GT}})sim(e_{p},e_{q})^{2}}{n^{2}},$
(7)

where $l_{\text{GT}}$ is the ground-truth labels for each pair of patches,
_e.g_., as the similarities between two patches from a image pair, $\llbracket
sim(\mathbf{p}_{n_{i}},\mathbf{p}_{n_{j}})>\epsilon\rrbracket$, where
$sim(\cdot)$ is the similarity measurement of the $d$-dimensional descriptors,
_i.e_., cosine similarity with $\sigma=1$ as the margin. It penalizes the
cases where there are high similarities on the patches but non-overlapping in
the ground truth, _i.e_., no visible 3D points. We resize the input images to
a fixed size. Therefore, the overall loss is averaged over $n^{2}$ tentative
matching patches on one image pair.

Additionally, the self and cross attention layers $A(\cdot)$ are applied to
select the relevant information and measure the similarity of two patch
descriptors. Then we replace $sim(\cdot)$ by $A(\cdot)$ in Eq. 7, and the
final loss, is computed by Eq. 6. The attentions are only used for training
and do not change the descriptors in inference. Note that the same losses can
be employed to project global descriptors, and the zero-overlapping image
pairs can be used as the negative examples.

In order to provide means for quick prefiltering, we add a classification
(CLS) token to the employed Transformer network that is trained to distill
global image embeddings based on the image overlap scores, shown in Table 5.
These tokens will be used to select a subset of potentially overlapping images
given the query.

Constructing Training Data. Now, we introduce the visual supervision used to
build GT labels for training. The ground-truth overlapping patches of the
training image pairs are achieved following LightGlue [41] employing pixel-
wise depth and relative pose. The ground-truth matching local patches are
found in two steps: First, a set of dense pixel coordinates $c_{i}$ are
detected in the first image $I_{i}$, and transformed to 3D points $X_{i}$. The
obtained 3D point cloud is filtered by non-zero depths and transformed to the
reference camera, _i.e_., $kp_{i2j}$, by the given transformation matrix.
Given camera intrinsics, we project the 3D points $X_{i}$ to the camera planes
and check for visibility. We make sure that only circle-consistent points are
kept by using the projected point, calculating its 3D coordinates, and back-
projecting to the original image to perform a visibility check again. The
ground-truth matches are supposed to contain the points with a consistent
depth [41]. This procedure provides the initial set of positive and negative
pairs of matching image patches.

Next, given the visible pixel matches, we first check which patches they
belong to and define the matching patches by having at least one visible pixel
correspondence. We assume one-to-one matching relations, _i.e_., the patch
with the most patches matches in the image pair will be chosen if there are
multiple. Note that the visual supervision is built mutually from the image to
the reference and vice versa.

###  3.3 Image Retrieval by Overlap Scores

Given a query image $I_{i}$ and a set of database images
${I_{j}}_{j=1,...,N}$, our goal is to efficiently determine the top-$k$ images
for $I_{i}$ maximizing overlap. To do so, in the offline mapping phase of the
method, we pre-generate the patch embeddings of all images in the database.

The retrieval pipeline is visualized in Fig. 4. In the online phase, we obtain
the patch and, optionally, the global embeddings of the query image. Next, for
each query image patch, we perform a radius search in the database of patch
embeddings to determine all potentially overlapping candidates. Setting a
radius threshold $\epsilon$ is easy as the employed cosine similarity falls
between 0 and 1. We tune this threshold during training and use a fixed one in
all our experiments. Note that k-nearest-neighbors algorithm is insufficient
in our case as we would like to find all potentially overlapping patches to
determine image overlaps. Such a radius search can be efficiently performed by
standard bounding structures, _e.g_., kd-tree, and other implementation
tricks. Optionally, we also use the global descriptor to prefilter the images
to a shortlist.

To obtain the top-$k$ images, we perform voting such that the overlap
calculation becomes

$O(I_{i},I_{j})\approx\sum_{(p,q)\in\mathcal{C}}\llbracket
sim(p,q)>\epsilon\rrbracket,$ (8)

quantifying the image overlap as the count of overlapping patches inside the
images, or the sum of the similarities of the descriptors as

$O(I_{i},I_{j})\approx\sum_{(p,q)\in\mathcal{C}}\max(sim(p,q)-\epsilon,0).$
(9)

Weighted Voting. To highlight the distinguishable query patches, we propose to
weight the query patches from each image using a bag of visual words approach
[16]. Specifically, we apply TF-IDF [58] weights to all the query patches,
assigning lower weights to patches that frequently appear in queries but are
rare in the matched database patches. As illustrated below, it is preferable
to have more neighbor patches if fewer images share the same neighborhood with
the query patch. For each patch, the weight $t_{i}$ is calculated as follows:

$t_{i}=\frac{n_{id}}{n_{d}}\log\frac{N}{n_{i}},$ (10)

where $N$ represents the total number of database images, $n_{d}$ is the total
number of patches in each image, 256 in our case, with a patch size of (14,
14) on the resized image size of 224. Parameter $n_{id}$ is the number of
neighbor patches in database images, defined as the cosine similarities of the
patch descriptors within a specified radius, and $n_{i}$ denotes the number of
database images that contain at least one neighbor of the query patch. Patches
that frequently appear are deemed less important and thus are down-weighted.
Weight $t_{i}$ for each query patch is incorporated into the overlap score
computation as follows: $t_{i}*O(I_{i},I_{j})$. Note that the map weights are
precomputed during the mapping phase.

##  4 Experimental Results

Our experiments concentrate on estimating two-view epipolar geometry for pairs
of retrieved images. The model is trained on 153 scenes from the MegaDepth
dataset [39], which includes Internet-sourced images and depths reconstructed
via Structure from Motion (SfM) and multi-view stereo techniques. Beyond the
MegaDepth testing scenes, we evaluate its generalization on 11 scenes from
PhotoTourism [71], the 13 scenes from ETH3D [67], and on the InLoc dataset
[74]. We benchmark the proposed method against several competitors, including
the foundational DINO V2, CNN-based retrieval methods like NetVLAD [2] and
CosPlace [5], as well as the state-of-the-art (SOTA) universal retrieval
method AnyLoc [33] utilizing the same DINO V2 backbone as we do. Besides the
global-only retrieval methods, we compare VOP with the SOTA reranking-based
methods: ViT-based $R^{2}$Former [90] and CNN-based Patch-NetVLAD [28]. Also,
we show the results of CosPlace when fine-tuned on the same dataset we train
on.

Training Details. The models underwent training on the 153 MegaDepth scenes,
employing contrastive loss to differentiate between negative and positive
patch pairs. Each scene contributes 150 positive and 150 negative image pairs
to the training set, standardized to an image resolution of $224\times 224$
pixels. We use the DINO V2 pre-trained model with a patch size of $14$ and
extract $256$-dimensional descriptors. Negative images, characterized by zero
overlaps (_e.g_., sourced from distinct scenes), contrast with positive pairs,
which are stratified into three groups based on overlap scores ranging from
0.1 to 0.7 for random selection. Notably, the contrastive loss framework is
predicated on patch-level negative/positive distinctions rather than entire
images. Following the methodology of LightGlue [41], patch-level labels are
generated from depth information and visual features, utilizing the
GlueFactory library [52] for implementation.

The training was conducted over 30 epochs with a batch size of 64, employing
matching recalls as the primary criterion for best checkpoint selection and
utilizing a learning rate of $1e-4$. The voting scheme and the radius for
neighbor searching were optimized on the validation set, which included a
single scene for this purpose. During inference, scenes “0015” (comprising
approximately 0.4K images) and “0022” (encompassing approximately 0.5K images)
from MegaDepth were designated for testing, enabling a focused evaluation of
the model performance on these specific subsets.

Evaluation Metrics. In our evaluation, we align with standard practices in
image retrieval research, employing recall$@k$ for $k\in{1,5,10}$ as our
primary metric, following methodologies outlined in recent studies [33, 1, 82,
2]. Moreover, we assess the relative poses between the query and the top-$k$
retrieved images using robust estimation techniques. This involves deploying
SuperPoint [18] and the state-of-the-art LightGlue [41, 61] for feature
detection and matching, followed by the application of a 10K-iterated OpenCV-
RANSAC [22] to compute the relative poses. The performance is further measured
against ground truth (GT) poses by calculating the Area Under the Recall curve
(AUC) thresholded at $10^{\circ}$ [84, 10], along with the median pose error.
The RANSAC threshold is optimized from a set of pixel values
$\\{0.5,1.,2.,3.\\}$ on the training set.

Moreover, we report the average number of inliers across image pairs as
detected by the retrieval methods. It is important to note that in scenarios
where $k>1$, for each query, we follow the practical approach and select only
the image pair that has the highest number of inliers. This comprehensive
evaluation allows us to thoroughly assess the performance not just in
retrieving relevant images but also in accurately estimating their relative
poses.

###  4.1 Pose Estimation on Megadepth

Table 1 presents the results on the MegaDepth dataset, highlighting the
performance of two variants of our method. For VOP, we select the model that
maximizes the pose AUC on the validation set. For VOP-recall, we choose the
model that maximizes retrieval recall. This experiment highlights that these
are two different objectives, and “recall” should not be used for retrieval if
the final goal is localization or reconstruction. Retrieval recall provides
the most similar images, but they may not be suitable for pose estimation due
to short baselines. The proposed method secures the highest scores for the
@10∘ criterion and exhibits the lowest median pose errors, thereby suggesting
that recall alone may not be a comprehensive metric for evaluating the
efficacy of VPR methodologies. Notably, the VOP-recall variant demonstrates
superior accuracy compared to other methods when tailored to enhance recall,
which underscores the limitations of recall as a sole metric.

The underlined results mark the optimal configuration for relative pose
estimation accuracy, evidencing the superior performance of the proposed VOP
approach when utilizing the top-5 images. This configuration outperforms the
global-only retrieval baselines (the first four rows) on this dataset,
reinforcing the significance of selecting appropriate metrics and model
configurations tailored to the specific demands of real-world downstream
applications. Besides, rows 5 and 6 in Table 1 show our method outperforms the
recent reranking approaches $R^{2}$Former [90] and Patch-NetVLAD [28] on
Megadepth. Both lead to comparable results in top-10 retrieval but worse in
the others.

| Recall (%) $\uparrow$ | AUC@10∘ $\uparrow$ | Med. pose error (∘) $\downarrow$ | # inliers $\uparrow$
---|---|---|---|---
Top $\rightarrow$ |  —1— |  —5— |  —10— |  —1— |  —5— |  —10— |  —1— |  —5— |  —10— |  —1— |  —5— |  —10—
AnyLoc [33] | 16.7 | 24.4 | 29.5 | 60.8 | 65.3 | 64.1 | 2.38 | 2.26 | 2.35 | 220.5 | 273 | 287.5
CosPlace [5] | 19.3 | 31.2 | 37.5 | 54.5 | 53.5 | 64.3 | 3.14 | 2.34 | 2.27 | 239.5 | 285.0 | 294.5
NetVLAD [2] | 19.5 | 27.4 | 32.0 | 56.5 | 64.2 | 63.6 | 2.85 | 2.36 | 2.38 | 234.0 | 286.5 | 294.0
DINO V2 [51] | 18.2 | 26.5 | 32.1 | 61.1 | 65.6 | 64.8 | 2.32 | 2.29 | 2.31 | 219.0 | 270.0 | 282.5
$R^{2}$Former [90] |  15.2 | 16.9 | 26.5 | 57.9 | 65.4 | 65.7 | 3.03 | 2.24 | 2.21 | 108.0 | 185.5 | 214.0
P-NetVLAD [28] |  18.9 | 22.5 | 29.7 | 59.7 | 64.9 | 64.5 | 2.67 | 2.22 | 2.27 | 212.5 | 264.0 | 283.5
VOP |  17.3 | 22.7 | 29.2 | 61.8 | 67.6 | 66.0 | 2.36 | 2.02 | 2.28 | 177.5 | 246.5 | 251.0
VOP-recall | 22.4 | 31.1 | 37.6 | 59.0 | 64.0 | 64.1 | 2.56 | 2.42 | 2.49 | 222.9 | 286.4 | 296.2

Table 1:  Relative pose estimation on the MegaDepth dataset on the top-$k$
retrieved images using different retrieval methods, with the best results in
bold at each $k$ and the overall best underlined. Standard VOP is trained to
maximize the AUC score of the relative pose errors. Version VOP-recall
maximizes the recall during training.

| AUC@10∘ $\uparrow$ | med. pose error (∘) $\downarrow$ | #inliers $\uparrow$
---|---|---|---
Top $\rightarrow$ |  —1— |  —5— |  —10— |  —1— |  —5— |  —10— |  —1— |  —5— |  —10—
AnyLoc [33] | 88.9 | 90.2 | 90.2 | 0.76 | 0.78 | 0.77 | 373.2 | 425.1 | 438.0
CosPlace [5] | 88.3 | 89.7 | 89.7 | 0.77 | 0.77 | 0.78 | 349.7 | 426.8 | 437.2
NetVLAD [2] | 89.5 | 90.3 | 89.9 | 0.74 | 0.74 | 0.79 | 365.8 | 147.3 | 440.1
DINO V2 [51] | 87.1 | 89.2 | 88.9 | 0.69 | 0.67 | 0.67 | 338.9 | 378.2 | 399.8
$R^{2}$Former[90] | 49.1 | 74.5 | 85.2 | 35.2 | 6.94 | 0.83 | 116.5 | 299.1 | 379.6
P-NetVLAD [28] | 86.7 | 88.4 | 90.2 | 0.81 | 0.83 | 0.81 | 333.3 | 412.4 | 466.8
VOP | 83.6 | 91.1 | 90.4 | 0.69 | 0.61 | 0.72 | 293.9 | 414.7 | 460.0

Table 2: Relative pose estimation on the ETH3D dataset [67] on the top-$k$
retrieved images using different retrieval methods, with the best results in
bold at each $k$ and the overall best underlined. The proposed VOP is trained
on the MegaDepth dataset.

###  4.2 Generalization Experiments

Moreover, we test the proposed method on multiple real-world scenes from the
ETH3D [67] and PhotoTourism [71] datasets for relative pose estimation, and
InLoc [74] for localization. We use the AUC-validated model from MegaDepth,
_i.e_., no fine-tuning or retraining were done on the following datasets.

ETH3D. The ETH3D [67] benchmark consists of images with various viewpoints
from indoor and outdoor scenes, with ground-truth poses reconstructed by laser
point clouds. 13 scenes are used for testing, tens of images from each. All
the images in the dataset are used as queries to select database images from
the rest. Table 2 shows that the proposed VOP method is always among the top-
performing methods, being either the best or second best across all
configurations. The overall best results in each metric are always obtained by
the proposed VOP. This highlights that VOP performs accurately even without
retraining and, thus, is applicable for unseen scenes. Unfortunately, R2Former
does not generalize well on ETH3D.

PhotoTourism. Table 3 reports the pose estimation results on 11 testing scenes
from PhotoTurism [71], with 100 images in each. We outperform the SOTA
competitors in terms of pose estimation accuracy, which shows that the
overlap-retrieved images are beneficial in such a geometric pipeline
estimating relative poses. Moreover, all Tables 1, 2, 3 show that the number
of inliers is not an indicator of accuracy, as we result in fewer inliers
while being more accurate than baselines. We have an exceptionally large
performance improvement on the top-1 retrieval scores and on the top-5 score
on the PhotoTourism dataset.

| AUC@10∘ $\uparrow$ | med. pose error (∘) $\downarrow$ | #inliers $\uparrow$
---|---|---|---
Top $\rightarrow$ |  —1— |  —5— |  —10— |  —1— |  —5— |  —10— |  —1— |  —5— |  —10—
AnyLoc [33] | 56.3 | 60.3 | 60.6 | 3.57 | 2.90 | 2.81 | 265.1 | 334.6 | 348.6
CosPlace [5] | 53.4 | 60.8 | 61.2 | 3.78 | 2.91 | 2.87 | 291.5 | 347.0 | 354.1
NetVLAD [2] | 55.0 | 59.3 | 60.3 | 3.75 | 3.10 | 2.94 | 283.9 | 340.0 | 348.1
DINO V2 [51] | 61.1 | 63.1 | 63.0 | 3.17 | 2.82 | 2.75 | 266.0 | 338.6 | 356.3
$R^{2}$Former [90] | 60.6 | 67.8 | 67.9 | 4.49 | 2.36 | 2.35 | 180.0 | 241.9 | 277.6
P-NetVLAD [28] | 61.9 | 62.4 | 61.8 | 2.76 | 2.65 | 2.76 | 261.9 | 315.2 | 339.4
VOP | 62.5 | 67.5 | 65.4 | 2.63 | 2.00 | 2.32 | 224.5 | 286.1 | 281.3

Table 3: Relative pose estimation on the PhotoTourism dataset [71] on the
top-$k$ retrieved images using different retrieval methods, with the best
results in bold at each $k$ and the overall best underlined. VOP is trained on
the MegaDepth dataset.

InLoc. Additionally, we ran all methods on the InLoc dataset [74] (without
fine-tuning) using hloc [61] for localization to demonstrate their
generalization capabilities in the case of a large domain gap
(indoor/outdoor). To retrieve top-40 database images, we prefilter the top-100
by CLS tokens and rerank them by VOP. Columns ’InLoc-DUC1’ and ’InLoc-DUC2’ in
Table 4 show the proposed VOP achieves high accuracy, showcasing its
generalization abilities.

| MegaDepth | ETH3D | PhotoTourism | InLoc-DUC1 | InLoc-DUC2 | AVG
---|---|---|---|---|---|---
Metrics | AUC@10∘ on top-5 $\uparrow$ | recall@5∘ on top-40 $\uparrow$
AnyLoc [33] | 65.3 | 90.2 | 60.3 | 74.7 | 75.6 | 73.2
Cosplace [5] | 53.5 | 89.7 | 60.8 | 69.2 | 74.8 | 69.6
Cosplacee* [5] | 65.8 | 79.7 | 65.9 | 41.4 | 29.0 | 56.4
NetVLAD [2] | 64.2 | 90.3 | 59.3 | 65.7 | 71.0 | 70.1
DINO V2 [51] | 65.6 | 89.2 | 63.1 | 63.6 | 71.0 | 70.5
VOP | 67.6 | 91.1 | 67.5 | 72.2 | 77.1 | 75.1

Table 4: Relative pose estimation (AUC@10∘) on three benchmarks and indoor
localization (recall@5∘) on InLoc. Cosplace* is fine-tuned on Megadepth
dataset.

Fairness of Comparison. To demonstrate that the improvements from the proposed
VOP do not stem from using different training data, we fine-tuned Cosplace [5]
on MegaDepth. As shown in the fourth row (Cosplace*) of Table 4, fine-tuning
improves the performance of Cosplace on relative pose estimation on Megadepth
and PhotoTourism. However, it largely reduces its accuracy on the ETH3D and
InLoc datasets.

###  4.3 Discussion on Patch-level Descriptors

DINO V2 Features. We investigated the number of patches to be used to split
the images. The experiments were conducted on the testing scenes of MegaDepth,
with all the images resized to $224$. From the DINO V2 (ViT-14) pretrained
model with a patch size of $14^{2}$, we can extract 256 patch descriptors.
Then, average pooling is applied to aggregate the patch descriptors to
different numbers of patches, such as 4, 16, 64. When the patch number is set
to 1, DINO V2 is applied directly to obtain a global image descriptor. Fig. 5
shows the global descriptors (#patches = 1) do not always perform well on the
image retrieval problem, as indicated by having a lower recall than when 64
patches are used. This clearly demonstrates that patch-level features have the
potential to improve visual localization. Similar to the experiments in the
previous sections, the patch-level descriptors learned for our method enhance
the original DINO V2 features.

Figure 5: Image retrieval evaluation by DINO V2 backbone with different
numbers of patches, _e.g_., 256. The recalls, AUC@10∘ scores, and median pose
errors in degrees are shown at top-1, 5, and 10 retrievals. When only one
patch is used (orange bar), directly DINO V2 features are employed.

| Validation | Prefilter | Augment | Dropout | AUC@10∘ $\uparrow$ | Med. pose error (∘) $\downarrow$ | # inliers $\uparrow$
---|---|---|---|---|---|---|---
Top $\rightarrow$ |  —1— |  —5— |  —10— |  —1— |  —5— |  —10— |  —1— |  —5— |  —10—
VOP | Recall | $\times$ | $\checkmark$ | $\times$ | 57.4 | 66.0 | 63.7 | 2.87 | 2.17 | 2.34 | 232.5 | 262 | 285.5
$\checkmark$ | $\checkmark$ | $\times$ | 57.5 | 65.2 | 64.5 | 2.96 | 2.19 | 2.31 | 235.5 | 272.5 | 288.5
AUC | $\checkmark$ | $\times$ | $\times$ | 61.9 | 65.1 | 64.7 | 2.37 | 2.19 | 2.30 | 209.0 | 272.5 | 282.0
$\times$ | $\checkmark$ | $\times$ | 63.7 | 66.3 | 65.3 | 2.23 | 2.18 | 2.31 | 179.5 | 222.0 | 233.0
$\checkmark$ | $\checkmark$ | $\times$ | 64.0 | 66.7 | 66.9 | 2.26 | 2.18 | 2.22 | 213.0 | 263.0 | 256.5
$\checkmark$ | $\checkmark$ | $\checkmark$ | 61.8 | 67.6 | 66.0 | 2.36 | 2.02 | 2.28 | 177.5 | 246.5 | 251.0

Table 5:  Relative pose estimation on the MegaDepth dataset [71] on the
top-$k$ retrieved images using different configurations, with the best results
in bold at each $k$ and the overall best underlined. The validation column
refers to the VOP model used. Prefilter indicates if the CLS token was
employed to shortlist the potential candidates before overlap prediction.
Augment refers to whether data augmentation was used.

Ablations. Table 5 shows the relative pose estimation performance on the
top-$k$ images retrieved by the model chosen from the best recall/AUC, with or
w/o global prefilter (CLS tokens), and dropout layer. Testing sets of
MegaDepth are used, as in Table 1. The CLS prefiltering improves the AUC
scores. However, it increases the median pose errors marginally at the same
time. As shown in the fifth row, data augmentation is essential in robustly
learning the embeddings. Also, the last two rows in Table 4 and Table 5 show
that the dropout layer improves the performance on Megadepth and helps with
generalization for pose estimation on other data and indoor localization.

Training. To better understand the training process, Fig. 6 illustrates the
training and validation losses on patch-level contrastive loss and the average
similarity changing among different epochs on MegaDepth. It shows the
contrastive loss helps to learn the embeddings of negative patches less
similar and closer to positive ones, and it converges fast. The similarities
shown are averaged over all positive/negative patches of the validation set
indicated by the GT labels built from 3D reconstructions.

Figure 6: Contrastive losses on the training set (left) and validation
(middle) over different epochs shown for negative or positive patch pairs. The
right plot shows the average cosine similarities over different patch samples.

Run-time & Query Speed. As we reduce the dimensionality of the DINO v2
features to 256, the embeddings of all patches of each image need a total of
512 kB, while, for AnyLoc, the storage per image is 384 kB. While we require
slightly more storage than AnyLoc [33], the difference is small. In addition,
we compare the time of querying an image from the database of different sizes
using AnyLoc or VOP in seconds. Prefiltering top 20 images by DINO V2 CLS
tokens and running VOP for reranking to get top-1 out of 500 images cost 0.009
seconds, while 0.003 for AnyLoc. Querying top-1 from 5k images (prefiltered to
100), our method costs 0.03 seconds, while AnyLoc runs in 0.02s. Note that VOP
needs much more time for radius search without prefiltering. We recommend
using VOP as a reranking method in retrieval.

##  5 Conclusion

Our paper introduces a novel Visual Place Recognition (VPR) method focusing on
patch-level overlap prediction to address challenging view changes and
occlusions. Through extensive testing on the MegaDepth, PhotoTourism and ETH3D
datasets, we demonstrate the effectiveness of the proposed VOP method. It
outperforms state-of-the-art retrieval approaches in terms of relative pose
accuracy. Critically, we demonstrate the inadequacy of recall as the sole
metric for VPR performance, advocating for metrics aligned with real-world
applications. By comparing configurations optimized for AUC score and recall,
our results underline the necessity of metric selection tailored to specific
VPR tasks. We believe this research marks a significant step forward in VPR,
emphasizing the importance of patch-level analysis and metric consideration
for improved accuracy and application relevance.

## References

  * [1] Ali-bey, A., Chaib-draa, B., Giguère, P.: Mixvpr: Feature mixing for visual place recognition. In: WACV (2023)
  * [2] Arandjelovic, R., Gronat, P., Torii, A., Pajdla, T., Sivic, J.: NetVLAD: CNN architecture for weakly supervised place recognition. In: CVPR (2016)
  * [3] Arandjelovic, R., Zisserman, A.: All about vlad. In: CVPR (2013)
  * [4] Balntas, V., Li, S., Prisacariu, V.: Relocnet: Continuous metric learning relocalisation using neural nets. In: The European Conference on Computer Vision (ECCV) (September 2018)
  * [5] Berton, G., Masone, C., Caputo, B.: Rethinking visual geo-localization for large-scale applications. In: CVPR (2022)
  * [6] Berton, G., Paolicelli, V., Masone, C., Caputo, B.: Adaptive-attentive geolocalization from few queries: A hybrid approach. In: WACV (2021)
  * [7] Bhayani, S., Sattler, T., Barath, D., Beliansky, P., Heikkilä, J., Kukelova, Z.: Calibrated and partially calibrated semi-generalized homographies. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5936–5945 (2021)
  * [8] Brachmann, E., Krull, A., Nowozin, S., Shotton, J., Michel, F., Gumhold, S., Rother, C.: Dsac - differentiable ransac for camera localization. In: CVPR (2017)
  * [9] Brachmann, E., Rother, C.: Learning less is more - 6d camera localization via 3d surface regression. In: CVPR (2018)
  * [10] Brachmann, E., Rother, C.: Neural-guided ransac: Learning where to sample model hypotheses. In: ICCV (2019)
  * [11] Brachmann, E., Rother, C.: Visual camera re-localization from rgb and rgb-d images using dsac. IEEE transactions on pattern analysis and machine intelligence 44(9), 5847–5865 (2021)
  * [12] Campos, C., Elvira, R., Rodríguez, J.J.G., Montiel, J.M., Tardós, J.D.: Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam. IEEE Transactions on Robotics (2021)
  * [13] Castle, R., Klein, G., Murray, D.W.: Video-rate localization in multiple maps for wearable augmented reality. In: 2008 12th IEEE International Symposium on Wearable Computers. IEEE (2008)
  * [14] Cavallari, T., Bertinetto, L., Mukhoti, J., Torr, P., Golodetz, S.: Let’s take this online: Adapting scene coordinate regression network predictions for online rgb-d camera relocalisation. In: 3DV (2019)
  * [15] Chen, Z., Maffra, F., Sa, I., Chli, M.: Only look once, mining distinctive landmarks from convnet for visual place recognition. In: IROS (2017)
  * [16] Csurka, G., Dance, C., Fan, L., Willamowski, J., Bray, C.: Visual categorization with bags of keypoints. In: Workshop on statistical learning in computer vision, ECCV (2004)
  * [17] Delhumeau, J., Gosselin, P.H., Jégou, H., Pérez, P.: Revisiting the vlad image representation. In: Proceedings of the 21st ACM international conference on Multimedia (2013)
  * [18] DeTone, D., Malisiewicz, T., Rabinovich, A.: Superpoint: Self-supervised interest point detection and description. In: CVPR Workshops (2018)
  * [19] Doan, A.D., Latif, Y., Chin, T.J., Liu, Y., Do, T.T., Reid, I.: Scalable place recognition under appearance change for autonomous driving. In: ICCV (2019)
  * [20] Doan, A.D., Latif, Y., Chin, T.J., Liu, Y., Do, T.T., Reid, I.: Scalable place recognition under appearance change for autonomous driving. In: ICCV (2019)
  * [21] El-Nouby, A., Neverova, N., Laptev, I., Jégou, H.: Training vision transformers for image retrieval. arXiv preprint arXiv:2102.05644 (2021)
  * [22] Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM (1981)
  * [23] Garg, S., Fischer, T., Milford, M.: Where is your place, visual place recognition? (2021)
  * [24] Garg, S., Suenderhauf, N., Milford, M.: Semantic–geometric visual place recognition: a new perspective for reconciling opposing views. The International Journal of Robotics Research (2019)
  * [25] Germain, H., Bourmaud, G., Lepetit, V.: Sparse-to-dense hypercolumn matching for long-term visual localization. In: International Conference on 3D Vision (3DV) (2019)
  * [26] Germain, H., Bourmaud, G., Lepetit, V.: S2dnet: Learning image features for accurate sparse-to-dense matching. In: European Conference on Computer Vision (ECCV) (2020)
  * [27] Hausler, S., Jacobson, A., Milford, M.: Multi-process fusion: Visual place recognition using multiple image processing methods. IEEE Robotics and Automation Letters (2019)
  * [28] Hausler, S., Garg, S., Xu, M., Milford, M., Fischer, T.: Patch-netvlad: Multi-scale fusion of locally-global descriptors for place recognition. In: ICPR (2021)
  * [29] Heng, L., Choi, B., Cui, Z., Geppert, M., Hu, S., Kuan, B., Liu, P., Nguyen, R., Yeo, Y.C., Geiger, A., et al.: Project autovision: Localization and 3d scene perception for an autonomous vehicle with a multi-camera system. In: ICRA. IEEE (2019)
  * [30] Hong, Z., Petillot, Y., Lane, D., Miao, Y., Wang, S.: Textplace: Visual place recognition and topological localization through reading scene texts. In: ICCV (2019)
  * [31] Ibrahimi, S., van Noord, N., Alpherts, T., Worring, M.: Inside out visual place recognition. In: British Machine Vision Conference (2021)
  * [32] Irschara, A., Zach, C., Frahm, J.M., Bischof, H.: From structure-from-motion point clouds to fast location recognition. In: CVPR (2009)
  * [33] Keetha, N., Mishra, A., Karhade, J., Jatavallabhula, K.M., Scherer, S., Krishna, M., Garg, S.: Anyloc: Towards universal visual place recognition. IEEE Robotics and Automation Letters (2023)
  * [34] Kendall, A., Cipolla, R.: Geometric loss functions for camera pose regression with deep learning. In: CVPR (2017)
  * [35] Kendall, A., Grimes, M., Cipolla, R.: Posenet: A convolutional network for real-time 6-dof camera relocalization. In: ICCV (2015)
  * [36] Khaliq, A., Ehsan, S., Chen, Z., Milford, M., McDonald-Maier, K.: A holistic visual place recognition approach using lightweight cnns for significant viewpoint and appearance changes. IEEE Transactions on Robotics (2020)
  * [37] Kim, H.J., Dunn, E., Frahm, J.M.: Learned contextual feature reweighting for image geolocalization. In: ICPR (2017)
  * [38] Li, Y., Snavely, N., Huttenlocher, D.P., Fua, P.: Worldwide pose estimation using 3d point clouds. In: ECCV (2012)
  * [39] Li, Z., Snavely, N.: Megadepth: Learning single-view depth prediction from internet photos. In: Computer Vision and Pattern Recognition (CVPR) (2018)
  * [40] Lim, H., Sinha, S.N., Cohen, M.F., Uyttendaele, M.: Real-time image-based 6-dof localization in large-scale environments. In: CVPR. IEEE (2012)
  * [41] Lindenberger, P., Sarlin, P.E., Pollefeys, M.: LightGlue: Local Feature Matching at Light Speed. In: ICCV (2023)
  * [42] Liu, L., Li, H., Dai, Y.: Efficient global 2d-3d matching for camera localization in a large-scale 3d map. In: ICCV (2017)
  * [43] Liu, L., Li, H., Dai, Y.: Stochastic attraction-repulsion embedding for large scale image localization. In: ICCV (2019)
  * [44] Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision (IJCV) (2004)
  * [45] Lynen, S., Zeisl, B., Aiger, D., Bosse, M., Hesch, J., Pollefeys, M., Siegwart, R., Sattler, T.: Large-scale, real-time visual–inertial localization revisited. The International Journal of Robotics Research 39(9), 1061–1084 (2020)
  * [46] Lynen, S., Zeisl, B., Aiger, D., Bosse, M., Hesch, J., Pollefeys, M., Siegwart, R., Sattler, T.: Large-scale, real-time visual–inertial localization revisited. The International Journal of Robotics Research (IJRR) 39(9), 1061–1084 (2020)
  * [47] Moreau, A., Piasco, N., Tsishkou, D., Stanciulescu, B., de La Fortelle, A.: Lens: Localization enhanced by nerf synthesis. In: CoRL (2021)
  * [48] Mur-Artal, R., Montiel, J.M.M., Tardos, J.D.: Orb-slam: a versatile and accurate monocular slam system. IEEE transactions on robotics (2015)
  * [49] Mur-Artal, R., Tardós, J.D.: Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE transactions on robotics (2017)
  * [50] Naseer, T., Spinello, L., Burgard, W., Stachniss, C.: Robust visual robot localization across seasons using network flows. In: Proceedings of the AAAI conference on artificial intelligence. vol. 28 (2014)
  * [51] Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)
  * [52] Pautrat*, R., Suárez*, I., Yu, Y., Pollefeys, M., Larsson, V.: GlueStick: Robust Image Matching by Sticking Points and Lines Together. In: ICCV (2023)
  * [53] Peng, G., Yue, Y., Zhang, J., Wu, Z., Tang, X., Wang, D.: Semantic reinforced attention learning for visual place recognition. In: IEEE International Conference on Robotics and Automation (2021)
  * [54] Peng, G., Zhang, J., Li, H., Wang, D.: Attentional pyramid pooling of salient visual residuals for place recognition. In: ICCV (2021)
  * [55] Philbin, J., Chum, O., Isard, M., Sivic, J., Zisserman, A.: Object retrieval with large vocabularies and fast spatial matching. In: CVPR (2007)
  * [56] Pion, N., Humenberger, M., Csurka, G., Cabon, Y., Sattler, T.: Benchmarking image retrieval for visual localization. In: 2020 International Conference on 3D Vision (3DV). pp. 483–494. IEEE (2020)
  * [57] Radenović, F., Tolias, G., Chum, O.: Fine-tuning cnn image retrieval with no human annotation. IPAMI (2018)
  * [58] Ramos, J., et al.: Using tf-idf to determine word relevance in document queries. In: Proceedings of the first instructional conference on machine learning (2003)
  * [59] Revaud, J., Almazán, J., Rezende, R.S., Souza, C.R.d.: Learning with average precision: Training image retrieval with a listwise loss. In: ICCV (2019)
  * [60] Sarlin, P.E., Cadena, C., Siegwart, R., Dymczyk, M.: From coarse to fine: Robust hierarchical localization at large scale. In: CVPR (2019)
  * [61] Sarlin, P.E., Cadena, C., Siegwart, R., Dymczyk, M.: From coarse to fine: Robust hierarchical localization at large scale. In: CVPR (2019)
  * [62] Sarlin, P.E., DeTone, D., Malisiewicz, T., Rabinovich, A.: Superglue: Learning feature matching with graph neural networks. In: CVPR (2020)
  * [63] Sarlin, P.E., Unagar, A., Larsson, M., Germain, H., Toft, C., Larsson, V., Pollefeys, M., Lepetit, V., Hammarstrand, L., Kahl, F., et al.: Back to the feature: Learning robust camera localization from pixels to pose. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3247–3257 (2021)
  * [64] Sattler, T., Leibe, B., Kobbelt, L.: Efficient & effective prioritized matching for large-scale image-based localization. PAMI (2017)
  * [65] Sattler, T., Zhou, Q., Pollefeys, M., Leal-Taixe, L.: Understanding the limitations of cnn-based absolute camera pose regression. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3302–3312 (2019)
  * [66] Schönberger, J.L., Pollefeys, M., Geiger, A., Sattler, T.: Semantic visual localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6896–6906 (2018)
  * [67] Schops, T., Schonberger, J.L., Galliani, S., Sattler, T., Schindler, K., Pollefeys, M., Geiger, A.: A multi-view stereo benchmark with high-resolution images and multi-camera videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3260–3269 (2017)
  * [68] Shao, S., Chen, K., Karpur, A., Cui, Q., Araujo, A., Cao, B.: Global features are all you need for image retrieval and reranking. In: ICCV (2023)
  * [69] Sharif Razavian, A., Azizpour, H., Sullivan, J., Carlsson, S.: Cnn features off-the-shelf: an astounding baseline for recognition. In: CVPR workshops (2014)
  * [70] Sivic, Zisserman: Video google: A text retrieval approach to object matching in videos. In: ICCV. IEEE (2003)
  * [71] Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: exploring photo collections in 3D. In: SIGGRAPH (2006)
  * [72] Subramaniam, A., Balasubramanian, P., Mittal, A.: Ncc-net: Normalized cross correlation based deep matcher with robustness to illumination variations. In: WACV. IEEE (2018)
  * [73] Svarm, L., Enqvist, O., Kahl, F., Oskarsson, M.: City-scale localization for cameras with known vertical direction. PAMI 39(7), 1455–1461 (2017)
  * [74] Taira, H., Okutomi, M., Sattler, T., Cimpoi, M., Pollefeys, M., Sivic, J., Pajdla, T., Torii, A.: Inloc: Indoor visual localization with dense matching and view synthesis. In: CVPR (2018)
  * [75] Tan, F., Yuan, J., Ordonez, V.: Instance-level image retrieval using reranking transformers. In: ICCV (2021)
  * [76] Tolias, G., Avrithis, Y., Jégou, H.: Image search with selective match kernels: aggregation across single and multiple images. IJCV (2016)
  * [77] Torii, A., Arandjelovic, R., Sivic, J., Okutomi, M., Pajdla, T.: 24/7 place recognition by view synthesis. IEEE Transactions on Pattern Analysis and Machine Intelligence (2018)
  * [78] Torii, A., Taira, H., Sivic, J., Pollefeys, M., Okutomi, M., Pajdla, T., Sattler, T.: Are large-scale 3d models really necessary for accurate visual localization? IEEE Transactions on Pattern Analysis and Machine Intelligence (2021)
  * [79] Valentin, J., Nießner, M., Shotton, J., Fitzgibbon, A., Izadi, S., Torr, P.: Exploiting uncertainty in regression forests for accurate camera relocalization. In: CVPR (2015)
  * [80] Vysotska, O., Stachniss, C.: Lazy data association for image sequences matching under substantial appearance changes. IEEE Robotics and Automation Letters 1(1), 213–220 (2015)
  * [81] Walch, F., Hazirbas, C., Leal-Taixe, L., Sattler, T., Hilsenbeck, S., Cremers, D.: Image-based localization using lstms for structured feature correlation. In: ICCV (2017)
  * [82] Wang, R., Shen, Y., Zuo, W., Zhou, S., Zheng, N.: Transvpr: Transformer-based place recognition with multi-level attention aggregation. In: CVPR (2022)
  * [83] Warburg, F., Hauberg, S., Lopez-Antequera, M., Gargallo, P., Kuang, Y., Civera, J.: Mapillary street-level sequences: A dataset for lifelong place recognition. In: CVPR (2020)
  * [84] Yi*, K.M., Trulls*, E., Ono, Y., Lepetit, V., Salzmann, M., Fua, P.: Learning to find good correspondences. In: CVPR (2018)
  * [85] Zaffar, M., Garg, S., Milford, M., et al.: Vpr-bench: An open-source visual place recognition evaluation framework with quantifiable viewpoint and appearance change. IJCV (2021)
  * [86] Zeisl, B., Sattler, T., Pollefeys, M.: Camera pose voting for large-scale image-based localization. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2704–2712 (2015)
  * [87] Zhang, W., Kosecka, J.: Image based localization in urban environments. In: International Symposium on 3D Data Processing, Visualization, and Transmission (2006)
  * [88] Zheng, E., Wu, C.: Structure from motion using structure-less resection. In: The IEEE International Conference on Computer Vision (ICCV) (2015)
  * [89] Zheng, L., Yang, Y., Tian, Q.: Sift meets cnn: A decade survey of instance retrieval. TPAMI (2017)
  * [90] Zhu, S., Yang, L., Chen, C., Shah, M., Shen, X., Wang, H.: R2former: Unified retrieval and reranking transformer for place recognition. In: CVPR (2023)