# D2S: Representing local descriptors and global scene coordinates for camera
relocalization

Bach-Thuan Bui, Dinh-Tuan Tran, and Joo-Ho Lee B. Bui is with the Graduate
School of Information Science and Engineering, Ritsumeikan University, Japan.
D. Tran and J. Lee are with the College of Information Science and
Engineering, Ritsumeikan University, Japan.

###### Abstract

State-of-the-art visual localization methods mostly rely on complex procedures
to match local descriptors and 3D point clouds. However, these procedures can
incur significant cost in terms of inference, storage, and updates over time.
In this study, we propose a direct learning-based approach that utilizes a
simple network named D2S to represent local descriptors and their scene
coordinates. Our method is characterized by its simplicity and cost-
effectiveness. It solely leverages a single RGB image for localization during
the testing phase and only requires a lightweight model to encode a complex
sparse scene. The proposed D2S employs a combination of a simple loss function
and graph attention to selectively focus on robust descriptors while
disregarding areas such as clouds, trees, and several dynamic objects. This
selective attention enables D2S to effectively perform a binary-semantic
classification for sparse descriptors. Additionally, we propose a new outdoor
dataset to evaluate the capabilities of visual localization methods in terms
of scene generalization and self-updating from unlabeled observations. Our
approach outperforms the state-of-the-art CNN-based methods in scene
coordinate regression in indoor and outdoor environments. It demonstrates the
ability to generalize beyond training data, including scenarios involving
transitions from day to night and adapting to domain shifts, even in the
absence of the labeled data sources. The source code, trained models, dataset,
and demo videos are available at the following link:
https://thpjp.github.io/d2s.

###### Index Terms:

Descriptors Regression, Pose Estimation, Reliablility Descriptors, 2D–3D
Regression.

##  1 Introduction

The capacity for re-localization has become a crucial aspect of our daily
lives, offering numerous potential applications. As humans, we possess the
ability to partially visualize and compare our current perspective with
previous experiences and re-orient ourselves in a relatively cost-effective
manner, even in a highly dynamic environment. However, accurately determining
our position and orientation to fractions of degrees and centimeter-level
precision is beyond our capabilities. This challenge is also encountered by
GPS-based systems when satellite signals are obstructed or delayed.

Camera-based re-localization enables highly accurate estimations in both
indoor and outdoor environments. State-of-the-art methods perform re-
localization using a single RGB image captured using a handheld device,
offering a reliable alternative to GPS-based systems.

Figure 1: D2S learning and inference pipeline. Our D2S learns from labeled
descriptors and is able to self-update with unlabeled ones. At the test time,
D2S receives a set of local descriptors and generates robust 3D coordinates
for highly accurate re-localization.

The most common methods rely on sparse reconstruction models and local feature
matching (FM) [1, 2, 3, 4, 5, 6, 7] owing to their robustness in handling
appearance changes introduced by different devices and viewpoints. However,
these algorithms tend to be complex, as the image matching process requires a
detailed 3D cloud map along with local and global visual descriptors [2, 8, 9,
10, 6, 11]. In addition, storing 3D maps can be resource-intensive, as they
often contain up to a million points. This hinders their execution on consumer
devices with limited computing capacity. Conversely, end-to-end learning of
the implied 2D–3D correspondences from dense pixel images offers a better
trade-off between accuracy and computational efficiency [12, 13, 14, 15].
These methods, commonly referred to as scene coordinate regression (SCR),
directly regress 3D scene coordinates to corresponding 2D pixels in the input
image, thereby naturally establishing the 2D–3D matching directly from the RGB
image without the need for 3D models. However, such methods are prone to
overfitting if the training data does not adequately represent the environment
[16]. Although previous SCR-based approaches have demonstrated high accuracy
for re-localization in small-scale and stationary environments [12, 17, 15],
they still face challenges in achieving the same performance as FM-based
methods in long-term environments [18, 19] or in situations involving a high
domain shift. Furthermore, from the design perspective, careful selection of
receptive field sizes is crucial for optimal performance [15].

In this study, we introduce a new approach that combines the robustness of FM-
based methods in handling appearance changes with the low-cost inference of
SCR-based methods. In contrast to learning directly from RGB images, we
initiate the regression network from salient keypoints as extracted
descriptors. This approach enhances the network’s robustness to high domain
shift situations. However, sparse keypoints are more suitable for matching
pipelines, as they are unordered and locally separated. We first propose a
sparse regression network named D2S as a representation of Descriptors and
their Scene coordinates. These correspondences serve as the basic input for
RANSAC-based pose optimization. Previous re-localization methods often lack
the ability to adapt to non-stationary environments, where new observations
can be captured over time using various devices. By leveraging existing robust
sparse matching pipelines, we additionally propose a self-supervised mechanism
for updating the re-localizer with new observations, enabling efficient
adaptation to non-stationary environments. In Fig. 1, we present a short
summary of the proposed pipeline. In detail, our contributions in this work
can be summarized as follows:

  * •

We present a novel approach for the representation of descriptors and 3D
coordinates through a lightweight regression network, termed D2S (Descriptors
to Scenes). Our approach achieves re-localization with remarkable precision,
with errors as low as a few centimeters and a fraction of a degree in real-
time, in both indoor and outdoor environments.

  * •

Our D2S leverages a combination of graph attention and a simple loss function
for learning to identify reliable descriptors. Consequently, D2S can strongly
focus on robust descriptors, leading to significant enhancements in re-
localization performance. This also renders D2S a natural filter for
appearances of dynamic objects.

  * •

We propose a self-supervised learning procedure for updating the D2S with new
observations without requiring camera poses, camera parameters, and ground
truth scene coordinates.

  * •

We introduce a new dataset to evaluate the performance of visual re-localizers
under conditions of high domain shift, such as varying weather conditions
(day, night, and rain) and locations with limited training data. We also
provide additional unlabeled data to facilitate the evaluation of methods with
self-supervised updating.

  * •

We report a comprehensive evaluation of the proposed D2S on four different
indoor and outdoor datasets. The results demonstrate that our method
outperforms the state of the art.

The remainder of this paper is organized as follows: Section 2 reviews the
related work on visual re-localization methods. Section 3 presents our
proposed D2S approach and its components. Section 4 reports the experimental
results and analysis on four different datasets. Section 5 discusses the
ablation study and limitations of our method. Finally, Section 6 concludes the
paper.

##  2 Related Work

In this section, we discuss the main research approaches for solving visual
re-localization. We also review related work on self-supervised updating with
unlabeled observations.

###  2.1 Image Retrieval and Absolute Pose Regression

Visual re-localization methods based on image retrieval were among the early
efficient solutions [20]. These methods represent environments by storing
known images in a database for later retrieval. Given a query image, the most
similar image within the database is identified by matching global image
descriptors. Global extractors such as learning-based NetVLAD[9] or DenseVLAD
[21] are commonly used. The final matched image is representative of a nearby
place or location associated with the query image [21, 9, 22]. Although these
approaches demonstrate robustness in scaling to extremely large environments,
they are limited in accurately estimating camera pose, as the accuracy is
confined to the discretization of the database. Some other approaches leverage
image retrieval as an efficient initialization step and subsequently use local
refinement optimization [23, 24] or relative pose estimation [25, 26],
resulting in improved accuracy estimation. However, these methods still face
challenges in terms of high memory demand and real-time estimation.

To address these challenges, 6DoF absolute camera pose regression approaches
(APR) have been developed such as PoseNet[27], along with subsequent
developments [28, 29, 30, 31, 32, 33]. APR learns to directly encode the 6DoF
camera pose from the image into a deep network, resulting in a lightweight
system where memory demand and estimation time are independent of the
reference data size. However, theoretical conclusions [34] suggest that APR
behaves similarly to image retrieval, and achieving high accuracy remains a
challenge.

###  2.2 Sparse Feature Matching

Features matching (FM) approaches represent the localization map as a 3D point
cloud SfM model [35]. Each 3D point is associated with one or several 2D
pixel-level key points in the images, which are represented by local
descriptors. Early FM-based re-localization solutions performed direct
matching between local descriptors in 2D images and 3D points in the SfM model
[36, 37, 38, 39]. These methods achieve accurate pose estimations but are
computationally intensive owing to exhaustive matching. Additionally, as SfM
models grow in size, the direct matching becomes ambiguous and difficult to
perform under high appearance changes, such as day and night. Taking advantage
of the robustness of image retrieval in scalability and handling appearance
changes, [6] proposed a hierarchical re-localization approach that uses image
retrieval as an initialization step before performing 2D–3D direct matching.
This approach was further improved by integrating with the SuperGlue matcher
[40], leading to a state-of-the-art performance on various benchmarks. Instead
of performing a complex 2D–3D matching, [41] and [42] proposed a deep feature
alignment that reduces the complexity of these systems while preserving highly
flexible and accurate estimation. However, the need to store 3D models,
local–global descriptors, and reference images of these methods, renders
deploying these methods on low-cost devices a challenge. Certain strategies,
such as descriptor compression and point cloud removal, are available to
reduce these burdens [10, 43, 44, 45, 46, 47]. However, these approaches often
involve a trade-off, sacrificing accuracy for other benefits.

###  2.3 Scene Coordinate Regression

In contrast to the FM-based pipeline, scene coordinate regression (SCR)
approaches are simpler, where 2D–3D correspondences can be directly extracted
using a single feedforward pass of a regression network. These methods either
used a random forest [48, 49, 50] or a convolutional neural network (CNN) [12,
17, 51, 13, 14, 15, 52, 53] to predict 3D scene coordinates from an RGB image,
thereby establishing the 2D–3D correspondences, which serve as the input for
RANSAC-based pose optimization. These methods do not require storing the 3D
models and descriptors, as this information is densely encoded in a learnable
neural network. SCR methods have demonstrated highly accurate estimation of
camera pose at small-scale and stationary environments while serving as
extremely compact re-localization systems. However, these methods still face
challenges in larger scale, non-stationary environments [19, 18] and in terms
of generalization capacity for novel viewpoints. Moreover, SCR-based methods
that utilize CNN architecture to estimate the scene also face challenges in
selecting the appropriate receptive field size [16, 14, 15]. A smaller
receptive field can improve generalization but may introduce ambiguity. In
contrast, a full-sized receptive field can resolve ambiguity but may result in
overfitting. In contrast, this work proposes a new approach for learning
directly from local descriptors combined with graph attention. Consequently,
it effectively mitigates the ambiguity of local features and achieves superior
generalization capabilities compared to previous CNN-based SCRs[12, 17, 15,
13, 14].

###  2.4 Self-Supervised Updating with Unlabeled Data

In contrast to visual SLAM systems [54, 55, 56, 57, 58], where the map
representation can be continuously updated over time, most visual re-
localization approaches, including FMs [45, 6, 46, 47] and SCR-based methods
[49, 50, 17, 51, 13, 14, 15, 53], were not initially designed to update the
model with new observations. In addition, updating with new data is often
difficult with these re-localization methods, as they require an additional
reconstruction to label the images and update 3D models. In contrast, APR-
based approaches have employed several techniques for self-supervised updating
with unlabeled images, leveraging either sequential frames [59] or direct
feature matching [33, 31]. As data can be captured at different times and
locations and by various devices, the re-localizer has to adapt to these
situations. However, to the best of our knowledge, no scene regression
approaches have attempted to address this issue, or they may be challenging to
train on unlabeled data. In this study, we introduce a straightforward
approach for self-supervised updating of the proposed SCR-based method with
unlabeled observations. This enables the method to learn with minimal initial
data and subsequently update itself during operation, significantly reducing
the labor-intensive task of data collection and labeling.

Figure 2: D2S Architecture. Proposed D2S network architecture for direct
learning sparse scene coordinates from descriptors. The network first receives
a set of descriptors and learns to communicate with each other using graph
attention. It then produces robust correspondent world coordinates with
reliability scores. The estimation of camera poses finally can be simply
computed by applying a PnP [60, 61] solver inside a RANSAC loop [62, 63].

##  3 Proposed Approach

Motivation: Sparse keypoints associated with descriptors are the most widely-
used fundamental elements in visual localization problems. These descriptors
are used to generate a sparse map through structure from motion (SfM) or
visual SLAM and are subsequently used for localization by finding the nearest
descriptor positions. Although various methods are available for solving this
search problem [38, 39, 6, 41], direct end-to-end learning from the descriptor
to its 3D position is yet to be explored in this research field. This is
because individual descriptors contain limited contextual awareness of the
entire map, making learning directly from them impractical. Additionally,
although an image frame often contains thousands of keypoint descriptors, only
a subset of them are projections of salient 3D points or robust features.
Therefore, the formulation of a direct learning method for 2D–3D
correspondences must adhere to certain physical constraints: i) several
descriptors may be unreliable for localization because they belong to dynamic
objects or are the result of detector failures, ii) individual descriptors may
have minimal or no significance to the entire image and global environment,
and iii) latent descriptors may have a distribution variance associated with
the changes in the environment over time since the map was created. We
formulated D2S (see Figure. 2) as a learnable method for 2D–3D correspondences
using single image descriptors. Our proposed approach offers a robust and
lightweight alternative to sparse SfM or visual SLAM-based 3D models.

Formulation: Given a set of images
$\\{\mathbf{I}^{i}_{\mathcal{T}}\\}_{i=1}^{n}$ and its reconstructed SfM model
$\mathcal{E}$, we aim to develop a sparse regression module, which can encode
the entire environment $\mathcal{E}$ using a compact function
$\mathfrak{F}(.)$, where $\mathfrak{F}$ is a deep neural network. The proposed
function $\mathfrak{F}(.)$ inputs local descriptors $\mathbf{D}^{i}$ extracted
from $\mathbf{I}^{i}$ and outputs its corresponding 3D global cloud
coordinates $\mathbf{W}^{i}$. The ultimate goal of the proposed module is to
perform visual localization, a task of estimating camera pose
$\mathbf{T}\in\mathbb{R}^{4\times 4}$ of the query image $\mathbf{I}_{q}$.

Overview: To unfold the constraints of unordered sets and different reliable
descriptors, constraint i, we initially propose a straightforward regression
module capable of accurately generating 3D positions for individual
descriptors, as well as 1D channels representing reliability probabilities
(Section 3.1). As the relationship between all image descriptors is crucial in
understanding the implied global visual and geometric context, as stated in
constraint ii, we propose a middle-optimization of D2S using a graph-attention
module to learn both global and local context (Section 3.2). Although this
module is similar to the works of feature matching [40, 64, 65], it is simpler
and more lightweight, and thereby more suitable for the scene regression task.
Given a successful representative method of scene coordinates and visual
descriptors, we design an extension algorithm for learning with any unlabeled
observations (Section 3.3). This can make D2S adopt the variance of
descriptors from a free data source, constraint iii. As the nature of the
proposed self-supervised learning in Section 3.3 can be further applied for
data augmentation, we present this procedure in Section 3.4. The loss
functions used to train the proposed system are described in Section 3.5.
Finally, we use PnP+RANSAC [62, 63], as in previous works [14, 15], to
estimate the camera pose (Section 3.6).

###  3.1 Representing Descriptors and Scene Coordinates as D2S

Inspired by the universal continuous set function approximator [66, 67], we
first propose a simple element set learning function $\mathfrak{F}(.)$. It
receives a set of descriptors $\\{\mathbf{d}^{i}\\}_{i=0}^{k}$ as input and
outputs corresponding scene coordinates $\\{\mathbf{w}^{i}\\}_{i=0}^{k}$. The
descriptors extractor can be a hand-crafted or learning-based method such as
SIFT [8] or SuperPoint [2], respectively. Overall, the proposed function can
be described as follows:

$\displaystyle\mathfrak{F}\left(\\{\mathbf{d}^{i}\\}_{i=1}^{k}\right)$
$\displaystyle=\left\\{f(\mathbf{d}^{i})\right\\}_{i=1}^{k}$ (1)
$\displaystyle=\\{\mathbf{w}^{i}\\}_{i=1}^{k},$

where $\mathfrak{F}:\mathbb{R}^{K\times D}\to\mathbb{R}^{K\times 4}$ and
$f:\mathbb{R}^{D}\to\mathbb{R}^{4}$. $K$ is the number descriptors, and $D$ is
its description dimension. The function $f(.)$ is a shared nonlinear function.
It receives a descriptor vector $\mathbf{d}\in\mathbb{R}^{D}$ and outputs a
scene coordinate vector $\mathbf{w}=(x,y,z,p)^{T}$. We introduce an additional
dimension $p$ for the scene coordinate, which represents the reliability
probability for localization of the input descriptor $\mathbf{d}$. To ensure
that the range of the reliability probability output lies within $[0,1]$ while
enabling function $f(.)$ to produce $p\in\mathbb{R}$, we compute the final
prediction for $\hat{p}$ using the following equation:

$\displaystyle\hat{z}=\frac{1}{1+|\beta p|}\in(0,1],$ (2)

where $\beta$ is a scale factor chosen to make the expected value of
reliability prediction $\hat{z}$ easy to reach a small value when the input
descriptors belong to high uncertainty regions.

The proposed module is theoretically simple. As illustrated in Eq. 1, it
requires only a non-linear function $f$ to compute the scene coordinates of
given descriptors. In practice, we approximate $f$ using a multi-layer
perceptron (MLP) network. This approach has been shown to work well in
experiments, particularly for small-scale indoor environments.

###  3.2 Optimizing D2S with Graph Attention

In the previous section, we described the core component for regressing scene
coordinates from sparse descriptors. Although this simple component is
advantageous, it is insufficient for achieving state-of-the-art accuracy, as
demonstrated in Section 5.2. This is due to the nature of local descriptors,
which are designed to describe local regions or patches within a single frame.
Therefore, relying solely on individual descriptors to regress scene
coordinates can result in a lack of understanding of the global context and
the distinctiveness of several feature clusters. To address this issue, we
draw inspiration from recent works on feature matching [40, 64, 65], which
leverage attention aggregation [68] to learn both local and global attributes
of sparse features. We employ a simplified version for the scenes regression
task, which can be described as follows:

$\displaystyle\mathfrak{F}\left(\\{\mathbf{d}^{i}\\}_{i=1}^{k}\right)$
$\displaystyle=f\circ\mathcal{A}^{l}\circ...\mathcal{A}^{1}\left(\\{\mathbf{d}^{i}\\}_{i=1}^{k}\right)$
(3) $\displaystyle=\\{\mathbf{w}^{i}\\}_{i=1}^{k},$

where $\mathcal{A}:\mathbb{R}^{K\times D}\to\mathbb{R}^{K\times D}$, and
$\mathcal{A}^{l}\circ...\mathcal{A}^{1}(.)$ represents a nested residual
attention update for each descriptor $\mathbf{d}^{i}$ across $l$ graph layers.
Given a coarse local descriptor $\mathbf{d}^{i}$, $\mathcal{A}^{l}(.)$ is used
to compute its fine regression descriptor $\prescript{(l)}{}{\mathbf{d}^{i}}$
by letting it communicate with each other across $l$ attention layers. This
implicitly incorporates visual relationships and contextual cues with other
descriptors in the same image, enabling it to extract robust features
automatically through an end-to-end learning pipeline from scene coordinates.
In addition, the proposed reliability-aware detection, as described in Eq. 2,
can also encourage our attention module to favor features from reliable areas
(e.g. buildings, stable objects) and suppress those from unreliable regions
(e.g. trees, sky, human).

As $\prescript{(l)}{}{\mathbf{d}^{i}}$ is the representation for the output
regression descriptor of element $i$ at layer $l$, Eq. 3 can be simplified as
follows:

$\displaystyle\mathfrak{F}(\\{\mathbf{d}^{i}\\}_{i=1}^{k})$
$\displaystyle=\left\\{f(\prescript{(l)}{}{\mathbf{d}^{i}})\right\\}_{i=1}^{k},$
(4)

which now has a similar form to the original Eq. 1, where the final regression
layer $f$ is simply a shared MLP.

Similar to a previous study [40], we also consider the attentional module as a
multi-layer graph neural network. However, we only leverage the self-edge,
which is based on self-attention [68], to connect the descriptor $i$ to all
others in the same image. The message passing formulation [69] for element $i$
at the layer $(l+1)$ is described as

$\prescript{(l+1)}{}{\mathbf{d}^{i}}=\prescript{(l)}{}{\mathbf{d}^{i}}+MLP\left(\left[\prescript{(l)}{}{\mathbf{d}^{i}}||\mathbf{m}_{\mathfrak{I}\rightarrow
i}\right]\right),$ (5)

where $\mathbf{m}_{\mathfrak{I}\rightarrow i}$ denotes the aggregated message
result from all descriptors
$\\{\prescript{(l)}{}{\mathbf{d}^{i}}\\}_{i=1}^{k}$, with
$\mathfrak{I}=\\{1,...,k\\}$ indicating the set of descriptor indices, and
$[.||.]$ denoting concatenation. This module consists of $L$ chained layers
with different parameters. Therefore, the message
$\mathbf{m}_{\mathfrak{I}\rightarrow i}$ is different in each layer.

The self-attention mechanism performs an interaction to map the query vector
$\mathbf{q}_{i}$ against a set of key vectors
$\\{\mathbf{k}_{j}\\}_{j\in\mathfrak{I}}$, associated with candidate
descriptors, to compute the attention score $\alpha_{ij}=$
Softmax${}_{j}(\mathbf{q}_{i}^{T}\mathbf{k}_{j})$. It then presents the best-
matched descriptors with their value vectors
$\\{\mathbf{v}_{j}\\}_{j\in\mathfrak{I}}$ represented by a scored average of
values, which in practice is the computation of message
$\mathbf{m}_{\mathfrak{I}\rightarrow i}$:

$\displaystyle\mathbf{m}_{\mathfrak{I}\rightarrow
i}=\sum_{j:(i,j)\in\mathfrak{I}}{\alpha_{ij}\mathbf{v}_{j}}$ (6)

The query, key, and value are derived from linear projections between the
descriptor $\prescript{(l)}{}{\mathbf{d}}$ with three different weight
matrices $\mathbf{W}_{q}$, $\mathbf{W}_{k}$, and $\mathbf{W}_{v}$. This linear
projection for all descriptors is expressed as

$\displaystyle\begin{bmatrix}\mathbf{q}\\\ \mathbf{k}\\\
\mathbf{v}\end{bmatrix}=\begin{bmatrix}\mathbf{W}_{q}\\\ \mathbf{W}_{k}\\\
\mathbf{W}_{v}\end{bmatrix}\prescript{(l)}{}{\mathbf{d}}+\begin{bmatrix}\mathbf{b}_{q}\\\
\mathbf{b}_{k}\\\ \mathbf{b}_{v}\end{bmatrix}.$ (7)

Note that the weight parameters are different in each graph layer. In
practice, we also applied multi-head attention [68] in each layer to improve
the expressivity from different representation subspaces of different weight
parameters. Fig. 3 presents an example result of self-attention on predicted
robust descriptors.

Figure 3: Self-Attention Results. We visualize the attention scores
$\alpha_{ij}$ (blue rays) at different keypoints $i$ for attentional graph
layer 5. The red points denote the predicted features with low reliability for
localization, whereas green points are robust features. This shows that the
proposed method has successfully learned to focus on features that have robust
properties for localization.

###  3.3 Updating D2S with Unlabeled Observations

1

Input: $\mathcal{T}$, $\mathcal{D}_{\mathcal{T}}$,
$\mathcal{W}_{\mathcal{T}}$, $\mathcal{U}$

  * -

$\mathcal{T}=\\{\mathbf{I}^{i}_{\mathcal{T}}\\}_{i=1}^{n}$ training images,

  * -

$\mathcal{D}_{\mathcal{T}}=\\{\mathbf{D}^{i}_{\mathcal{T}}\\}_{i=1}^{n}$
training descriptors,

  * -

$\mathcal{W}_{\mathcal{T}}=\\{\mathbf{W}^{i}_{\mathcal{T}}\\}_{i=1}^{n}$
training scene coordinates,

  * -

$\mathcal{U}=\\{\mathbf{I}^{i}_{\mathcal{U}}\\}_{i=1}^{m}$ unlabeled images,

Output:
$\mathcal{A}=\\{\mathbf{D}^{i}_{\mathcal{U}},\mathbf{W}^{i}_{\mathcal{U}}\\}_{i=1}^{v}$

  * -

$\mathcal{D}_{\mathcal{U}}=\\{\mathbf{D}^{i}_{\mathcal{U}}\\}_{i=1}^{m}$
unlabeled descriptors,

  * -

$\mathcal{W}_{\mathcal{U}}=\\{\mathbf{W}^{i}_{\mathcal{U}}\\}_{i=1}^{m}$
pseudo scene coordinates,

  * -

$v\leq m$.

2 $\mathcal{D}_{\mathcal{U}}\leftarrow$ ExtractDescs$(\mathcal{U});$

3 $i:=1;\mathcal{A}\leftarrow\\{\\};$

4 do

5 $\mathbf{ids}_{\mathcal{T}}\leftarrow$
NearestVLAD$(\mathbf{I}_{\mathcal{U}}^{i},\mathcal{T});$ # nearest indices.

6 $\mathcal{M}\leftarrow$
Matching$(\mathbf{D}_{\mathcal{U}}^{i},\mathcal{D}_{\mathcal{T}}^{\mathbf{ids}_{\mathcal{T}}});$

7 $s,\mathbf{W}^{i}_{\mathcal{U}}\leftarrow$
Copy$(\mathcal{M},\mathcal{W}_{\mathcal{T}}^{\mathbf{ids}_{\mathcal{T}}});$

8 if  _$s\geq 50$_ then

9
$\mathcal{A}\leftarrow\mathcal{A}\cup\\{\mathbf{D}^{i}_{\mathcal{U}},\mathbf{W}^{i}_{\mathcal{U}}\\};$

10

11while  _$i\leq m,i++$_ ;

Algorithm 1 Pseudo-labelling Figure 4: Data flow of the proposed algorithm for
updating D2S with available unlabeled data. D2S learns purely from training
data sources. D2S+ then can be updated using the pre-trained D2S with new
observations of the environments by self-supervised learning.

Due to the significant changes that can occur in environments over time since
map creation, the ability to update from unlabeled data is crucial for most
localization systems. Previous SCR approaches [17, 14, 13, 15] are offline
methods, where the learned network weights are fixed after training. In
addition, these methods require absolute camera poses, camera parameters, and
pixel-level scene coordinates to supervise the network. However, obtaining
this information often requires reconstructing the 3D map. Therefore, updating
the system with new observations is challenging. In this section, we introduce
a straightforward algorithm to update the proposed D2S with additional
unlabeled data in a self-supervised learning manner without the need to
reconstruct the 3D map. Fig. 4 illustrates the proposed data flow for updating
D2S with available unlabeled data.

Given a set of training images $\mathcal{T}$ and its labeled database of
correspondent 2D–3D descriptors $\mathcal{D}_{\mathcal{T}}$ and
$\mathcal{W}_{\mathcal{T}}$, we aim to find a set of pseudo-labels for new
observation images $\mathcal{U}$. As D2S receives descriptors and produces
world coordinates, the pseudo-labels accordingly denote the newly generated
2D–3D correspondences of the new data $\mathcal{U}$. In detail, we first
extract the local descriptors of all images in $\mathcal{U}$, which can be
denoted as $\mathcal{D}_{\mathcal{U}}$. For each query image
$\mathbf{I}_{\mathcal{U}}^{i}$, the top 10 nearest training images are listed
in $\mathcal{T}$ before matching the local descriptors. In this searching
step, image retrieval NetVLAD[9] is used to determine the most similar images,
which is described in line 4 of the algorithm 1. Subsequently, for each
matching pair, we simply match two descriptor sources using a specific
matching algorithm such as the nearest neighbor search. In particular, recent
advancements in keypoints matching methods, such as SuperGlue [1] and LoFTR
[2], can serve as a reliable tool for generating matching keypoints between
two images. Thus, we used SuperGlue and the nearest neighbor method to
evaluate our algorithm.

Finally, we merged the top 10 matching pairs by copying their ground truth
world coordinates from $\mathcal{T}$ to $\mathcal{U}$. This step is shown as
line 6 of the algorithm 1, where the $\mathbf{W}^{i}_{\mathcal{U}}$ is the
obtained pseudo scene coordinates of descriptor matrix
$\mathbf{D}^{i}_{\mathcal{U}}$ and $s$ is the number of valid world
coordinates. Only the obtained pseudo-labels are added to the final training
source with $s\geq 50$. The whole process of generating pseudo-labels for
unlabeled data is summarized in algorithm 1.

###  3.4 Data Augmentation

General approaches for data augmentation include color changing, rotation,
distortions, and cropping. Applying these methods to the training data can
enrich the DNN model with unseen data. However, in our study, these
augmentation techniques could affect the camera pose and latent descriptors.
Using algorithm 1, we can derive a simple procedure for data augmentation of
D2S with any transformation technique. To achieve this outcome, we transform a
given image $\mathbf{I}_{\mathcal{T}}$ by changing brightness, cropping, or
distorting, which result in a new image
$\mathbf{I}^{{}^{\prime}}_{\mathcal{T}}$. Thereafter, we simply assume that
these two images are nearby and apply algorithm 1 for the matching and copying
operations (lines 5 and 6). This simple augmentation step successively
improves performance, as explained in Section 4.

Figure 5: Reliability prediction results. We visualize the prediction of
robust localization features as follows: The first image row describes the
original keypoints predicted by SuperPoint[2]. The second and third rows are
the results of our D2S on predicting reliable keypoints for localization. The
red points denote the predicted keypoints with low reliability for
localization, whereas green points are robust keypoints.

###  3.5 Loss Function

According to Eq. 1, the output of sparse scene coordinates $\mathbf{w}$,
predicted by D2S, includes four variables $(x,y,z,p)^{T}$. This output is
rewritten as $(\hat{\mathbf{y}},p)^{T}$, where $\hat{\mathbf{y}}$ is the
abbreviation of the estimated world coordinates vector and $p$ is the
reliability probability. The loss function to minimize the estimated scene
coordinates can be defined as

$\mathcal{L}_{m}=\frac{1}{N}\sum^{N}\sum_{i=1}^{K}z_{i}\lVert\mathbf{y}_{i}-\hat{\mathbf{y}}_{i}\lVert^{2}_{2},$
(8)

where $N$ is the number of mini-batch sizes, $K$ is the number of descriptors
in the current frame, and $\mathbf{y}_{i}$ is the ground truth scene
coordinate. We multiply the difference between estimated and ground truth
world coordinates by $z_{i}\in\\{0,1\\}$, which is the ground truth
reliability of descriptors. The goal is to ensure that the optimization
focuses on robust descriptors while learning to ignore unreliable descriptors.

Because the loss Eq. 8 is used to optimize only the robust features, we
propose an additional loss function to simultaneously learn to be aware of
this assumption, as follows:

$\mathcal{L}_{u}=\frac{1}{N}\sum^{N}\sum_{i=1}^{K}\lVert
z_{i}-\frac{1}{1+|\beta p|}\lVert^{2}_{2}.$ (9)

As described in Section 3.1, $\beta$ is a scale factor chosen to enable the
optimization to focus on the descriptors belonging to high uncertainty regions
or failures of the detector. The combination of $\mathcal{L}_{m}$ and
$\mathcal{L}_{u}$ can result in a stable optimization on two different
criteria of robust descriptors. In practice, we chose the factor value as
$\beta=100$ for successively learning to classify unreliable descriptors and
focus on robust features (refer to Fig. 3 for an example result).

In addition, as the camera poses are known for the training data, we also
apply the re-projection loss for each 2D descriptor’s position. The aim of
this loss function is to align the general scene coordinates following the
correct camera rays. This loss function can be described as follows:

$\mathcal{L}_{r}=\frac{1}{N}\sum_{j=1}^{N}\sum_{i=1}^{K}z_{i}\lVert\pi(\mathbf{R}_{j}\mathbf{y}_{i}+\mathbf{t}_{j})-\mathbf{u}_{i}\lVert^{2}_{2},$
(10)

where $\mathbf{R}_{j}$ is the rotation matrix and $\mathbf{t}_{j}$ is the
translation vector of camera pose $j$. $\pi(.)$ is a function to convert the
estimated 3D camera coordinates to 2D pixel position, with $u_{i}$ as its
ground truth pixel coordinate.

Finally, these loss functions are combined with the scale factor as follows:

$\mathcal{L}=\alpha_{m}\mathcal{L}_{m}+\alpha_{u}\mathcal{L}_{u}+\alpha_{r}\mathcal{L}_{r}$
(11)

###  3.6 Camera Pose Estimation

The proposed method directly learns to compress 2D–3D correspondences into a
D2S network. A sparse set of scene coordinates is estimated based on D2S with
its corresponding reliability values. To estimate the camera pose, we filtered
out the most robust predicted scene coordinates based on its reliability. The
visualization results are illustrated in Fig. 5. Moreover, a robust minimal
solver (PnP[62]+RANSAC[63]) is used, followed by a Levenberg–Marquardt-based
nonlinear refinement, to compute the camera pose.

In contrast to DSAC∗ [15] and several SCR-based works [12, 17, 14], which only
focus on estimating dense 3D scene coordinates of the input image, our D2S can
simultaneously focus on the detection of salient keypoints of the scene and
output its coordinates. We believe that the SCR network should learn to pay
more attention to salient features (the most crucial solution to localization
problems); this can lead to a better localization performance [19, 70, 41].
Furthermore, directly learning scene coordinates from an RGB image may result
in decreased performance when the test images differ significantly from
training images. This finding is supported by empirical evidence presented in
Section 4.5. Notably, the proposed D2S demomnstrates robustness against these
challenges.

##  4 Experiments

Figure 6: Proposed Ritsumeikan BKC dataset. We visualize the proposed dataset
for evaluation of the proposed method. The dataset consists of different image
sequences used for training, testing, and self-supervised learning with
unlabeled sequences. The red sequences are the training data whereas the
others including blue, green, and cyan are used for evaluation. In each
sequence, we display an example image indicating its difference in time and
distance on the map. We do not visualize the unlabeled data here, as it has
not been used to generate the 3D cloud map, as well as in labeling step.

We evaluate the proposed re-localization method on four different datasets:
two indoor and two outdoor datasets. First, we present the experimental
settings used in our experiments, including network architecture, datasets,
and competitors (Section 4.1). Second, we report the results of the two
different categories: indoor and outdoor environments, in Sections 4.2 and
4.5, respectively.

###  4.1 Experimental Settings

####  4.1.1 Settings

First, we describe the network architecture used in our experiments. As
described in Section 3.1 and Section 3.2, the proposed network includes two
different parts: a graph attention module and a shared multi-layer perception
(MLP). For the graph module, we used five different layers $L=5$, starting
from coarse descriptors extracted by any specific local feature extractors
(e.g. SIFT [8] and SuperPoint [2]). At each graph layer, we used the same
number of four attention heads. The second module is an MLP with the following
setting: $MLP(D,512,1024,1024,512,4)$, where $D$ is the number of dimensions
of the input descriptors. Our architecture for regressing sparse scene
coordinates incurs a parameter count that depends on the local feature
dimension. For example, the SuperPoint [2] descriptor, which has 256
dimensions, uses 5.5 million parameters, whereas only 3.0 million parameters
are required for SIFT descriptors, which have 128 dimensions. However, in this
study, we primarily experiment with the proposed method using SuperPoint
features, due to its posterior performance compared with other local features
[2, 40, 6]. We also verified this result by conducting experiments with other
local feature extractors (see Section 5.1).

Second, we present the hyperparameters and implementation details of our
experiments. We used Pytorch [71] to implement the proposed method and trained
it with a mini-batch size of eight images. The training of D2S was separated
into two steps. We first trained the network with 300K iterations, where the
hyperparameters of the loss Eq. 11 are established as follows: $\alpha_{m}=1$,
$\alpha_{u}=1$, and $\alpha_{r}=0$. In this step, we used a learning rate of
$10^{-4}$ with the Adam optimizer [72]. The learning rate reduces with a decay
of 0.5 for every one-fourth of the total iterations. In the second training
step, we trained the network with another 100K iterations, with a smaller
learning rate of $10^{-5}$, where the $\alpha_{r}$ is increased to
$\alpha_{r}=10$. The aim of the second training step is to enhance the network
to focus on the re-projection loss Eq. 10, which depends on the ground truth
camera poses. For data augmentation, we applied the algorithm 1 and the method
described in Section 3.4 to obtain the pseudo-labels of augmented descriptors.
In particular, we used Pytorch [71] transformation to randomly adjust the
brightness and contrast of the input images within the range $[0.5,1.5]$ and
$[0,0.5]$ respectively. We also applied a random perspective transformation to
the training images with a probability of 0.3 and a distortion scale of 0.4.
Finally, for camera pose estimation using the predicted sparse scene
coordinates, we used PnP+RANSAC implemented in pyColmap [73] with an inlier
threshold of 12 px.

For the self-supervised updating of the model with unlabeled data, we applied
Algorithm 1 to generate pseudo-labels for the unlabeled descriptors. We then
updated the pre-trained models with an additional 50K iterations. In this
step, we used $\alpha_{m}=1$, $\alpha_{u}=1$, and $\alpha_{r}=1$. In each
mini-batch, we sampled half of the training data and half of the unlabeled
data. This step ensures that the updating is not biased toward only the
unlabeled data.

####  4.1.2 Datasets

We evaluated the proposed pipeline on two indoor and two outdoor camera re-
localization datasets. For the indoor environment, we used a standard small-
scale 7-Scenes dataset [48], and the recently released Indoor6 [19]. For the
outdoor environment, we used the Cambridge Landmarks [27] dataset. However, we
observed that the performance of various camera localization methods is
already saturated on two benchmarks of 7-Scenes and Cambridge[27, 12, 17, 32,
33, 13]. In addition, the properties of these datasets are not completely
suitable to evaluate scene-specific generality and availability of unlabeled
observations. Therefore, we proposed a new outdoor dataset named Ritsumeikan
BKC for the evaluation of our hypothesis on generality and the proposed self-
supervised pipeline. Details of the datasets used in our experiments are as
follows:

7-Scenes [48]: An RGB-D dataset of seven small-scale indoor environments
ranging in size from $1m^{3}$ to $18m^{3}$. This dataset features challenges
such as repeating structures, motion blur, and texture-less areas. Each scene
consists of several thousand RGB images recorded by KinectFusion [74], which
also provides the ground truth for camera poses. The depth channels and camera
poses are available for generating the ground truth scene coordinates for our
training of D2S. The training and validation sets are split by the authors. In
addition, 7-Scenes is a stationary dataset of images captured under fixed
illumination. Consequently, validating localization methods against the
challenges of changing environments is difficult.

Figure 7: KingsCollge [27] training and testing sequences. We visualize the
KingsCollge scene as an example for showing the similarity between the
training (red) and testing data (blue) compared to the BKC dataset (see Fig.
6). TABLE I: 7-scenes results. We report the median errors in cm for position
and degree (∘) for the orientation of the 7-scenes dataset. We also display
the obtained recall for each method, which is the percentage of frames that
have the error within the threshold of 5 cm and 5∘. The best results are in
bold.

7scenes
---
Method | Chess | Fire | Heads | Office | Pumpkin | Kitchen | Stairs
Err. | Acc. | Err. | Acc. | Err. | Acc. | Err. | Acc. | Err. | Acc. | Err. | Acc. | Err. | Acc.
(cm/deg.) | (%) | (cm/deg.) | (%) | (cm/deg.) | (%) | (cm/deg.) | (%) | (cm/deg.) | (%) | (cm/deg.) | (%) | (cm/deg.) | (%)
APR | MapNet[59] | 9/3.24 | - | 20/9.29 | - | 12/8.45 | - | 19/5.42 | - | 19/3.96 | - | 20/4.94 | - | 27/10.6 | -
FeatLoc[32] | 7/3.66 | - | 17/5.95 | - | 10/7.57 | - | 16/5.2 | - | 11/3.86 | - | 20/6.43 | - | 16/8.57 | -
DFNet[33] | 5/1.88 | - | 17/6.45 | - | 6/3.63 | - | 8/2.48 | - | 10/2.78 | - | 22/5.45 | - | 16/3.29 | -
FM | AS[38] | 2.6/0.9 | 86.2 | 2.3/1.02 | 86.6 | 1.1/0.8 | 95.7 | 4.0/1.2 | 65.9 | 6.5/1.7 | 32.5 | 5.3/1.7 | 46.2 | 3.8/1.1 | 68.1
Inloc[24] | 3/1.05 | - | 3/1.07 | - | 2/1.16 | - | 3/1.05 | - | 5/1.55 | - | 4/1.31 | - | 9/2.47 | -
Hloc[6, 40] | 2.4/0.77 | 94.2 | 1.8/0.75 | 93.7 | 0.9/0.59 | 99.7 | 2.6/0.77 | 83.2 | 4.4/1.15 | 55.1 | 4.0/1.38 | 61.2 | 5.1/1.46 | 49.4
SCR | SCoCR[14] | 2/0.7 | 97.5 | 2/0.9 | 96.7 | 1/0.9 | 100 | 3/0.8 | 86.5 | 4/1.0 | 59.9 | 4/1.2 | 65.5 | 3/0.8 | 87.5
KFNet[13] | 1.8/0.65 | - | 2.3/0.90 | - | 1.4/0.82 | - | 2.5/0.69 | - | 3.7/1.02 | - | 3.8/1.16 | - | 3.3/0.94 | -
NBE+SLD[19] | 2.2/0.75 | 93.7 | 1.8/0.73 | 94.1 |  0.9/0.68 | 96.6 | 3.2/0.91 | 74.8 | 5.6/1.55 | 44.6 | 5.3/1.52 | 45.7 | 5.5/1.41 | 44.6
DSAC*[15] | 1.8/0.59 | 97.8 |  1.7/0.77 | 94.5 | 1.0/0.66 | 98.8 | 2.7/0.79 | 83.9 | 3.9/1.05 | 62.0 | 3.9/1.24 | 65.5 | 3.5/0.93 | 78.0
D2S (ours) | 1.7/0.57 | 98.6 | 1.8/0.74 | 92.9 | 1.2/0.75 | 98.7 | 2.1/0.62 | 91.3 | 2.9/0.83 | 72.8 | 3.0/1.04 | 77.9 | 13/2.02 | 24.2

Figure 8: Cumulative distributions of pose error (maximum of position and
rotation errors) for 7-Scenes dataset. †This excludes Stairs scene, since D2S
failed to obtain a stable result. Figure 9: Estimated results for indoor
7scenes[48]. First Column: Camera frames of training in gray and test frames
in cyan for all scenes of the 7-Scenes dataset. From Second Column: We
visualize the estimated camera positions with their color errors of Hloc [6,
40], DSAC*[15] and the proposed D2S. We also attach the percentage of
successive frames with pose errors below 5 cm and 5∘, where the best results
are in bold. For a more informative illustration, we also visualize the 3D
cloud map as a faint backdrop. We only connect the two consecutive frames
within a 50 cm tolerance of position error.

Indoor6 [19]: In contrast to the 7-Scenes dataset, Indoor6 is a non-stationary
dataset. The images are captured in much larger environments at different
times and days and exhibit strong illumination variations. It contains six
different indoor environments, each with multiple rooms. The ground truth of
the camera poses and 3D point clouds are labeled using COLMAP [35]. We used
the split of training and testing data, as well as the 3D SfM models provided
by the authors.

Cambridge [27]: An RGB outdoor dataset scaling from $875m^{2}$ to $5600m^{2}$.
Each scene contains 200–1500 training and validation samples, which are split
by the authors. The ground truth of camera poses is obtained through SfM
reconstruction. We used this rendered 3D model to generate the ground truth
scene coordinates for the training of D2S.

Ritsumeikan BKC: We introduce a novel outdoor RGB dataset for evaluating the
generalization capabilities of visual localization methods. In contrast to the
Cambridge dataset, our dataset possesses training trajectories that are
distinct from those used for testing. As illustrated in Fig. 6, the data were
collected at different times of day and night. In particular, it includes two
different scenes named WestWing and CentralArc. In the WestWing scene (Fig.
6a), a total of seven sequences are present: one for training (daytime), three
for evaluation (nighttime and far-distance daytime), and the remaining three
sequences are unlabeled. Note that we do not visualize the unlabeled
trajectories here as we do not know their camera poses. The CentralArc (Fig.
6c) is similar but has an additional training sequence. In Fig. 6b and d, we
visualize the 3D cloud map generated when using only training data. Note that
the training data was recorded to cover most of the important areas for
localization. Our dataset also involves certain challenges such as changing
conditions, domain shifts, and scanty training data. For comparison, Fig. 7
visualizes the training and testing data of Cambridge KingsCollege.

In this work, we used Hloc [6, 40] to pre-process the data for the evaluation
of D2S. We used the SfM pseudo ground truth (pGT) generated from COLMAP [35]
to triangulate 3D points from SuperPoint features. Subsequently, we split the
extracted features and their labels of 3D coordinates for training and
testing. The evaluation of D2S with other local feature extractors (e.g., SIFT
[8], R2D2 [4]) also used the aforementioned pre-processing procedure.

TABLE II: Indoor6 results. We report the median errors in cm for position and
degree (∘) for the orientation of the Indoor6 dataset. We also show the
percentage of frame errors within the threshold of 5 cm and 5∘.

Indoor6
---
Method | Size | scene1 | scene2a | scene3 | scene4a | scene5 | scene6
(GB) | (cm/deg.) | (%) | (cm/deg.) | (%) | (cm/deg.) | (%) | (cm/deg.) | (%) | (cm/deg.) | (%.) | (cm.) | (%)
PoseNet[28] | 0.050 | 159.0/7.46 | 0.0 | -/- | - | 141.0/9.26 | 0.0 | -/- | - | 179.3/9.37 | 0.0 | 118.2/9.26 | 0.0
DSAC*[15] | 0.028 | 12.3/2.06 | 18.7 | 7.9/0.9 | 28.0 | 13.1/2.34 | 19.7 |  3.7/0.95 | 60.8 | 40.7/6.72 | 10.6 | 6.0/1.40 | 44.3
NBE+SLD(E)[19] | 0.029 | 7.5/1.15 | 28.4 | 7.3/0.7 | 30.4 | 6.2/1.28 | 43.5 | 4.6/1.01 | 54.4 | 6.3/0.96 | 37.5 | 5.8/1.3 | 44.6
NBE+SLD[19] | 0.132 | 6.5/0.90 | 38.4 | 7.2/0.68 | 32.7 | 4.4/0.91 | 53.0 | 3.8/0.94 | 66.5 | 6.0/0.91 | 40.0 | 5.0/0.99 | 50.5
D2S(ours) | 0.022 | 5.6/0.90 | 44.6 | 8.7/0.8 | 22.0 | 4.8/0.92 | 52.4 |  3.7/0.88 | 69.0 | 8.6/1.26 | 28.5 | 4.9/1.06 | 53.3
D2S+(ours) | 0.022 | 5.0/0.82 | 50.6 | 6.9/0.62 | 27.2 | 4.3/0.74 | 59.1 | 3.7/0.86 | 70.3 | 8.5/1.16 | 29.3 | 3.8/0.79 | 62.9

TABLE III: Median position and rotation errors of different relocalization
methods on Cambridge dataset. Cambridge Landmarks
---
Method | King’s College | Old Hospital | Shop Facade | St Mary’s Church | Great Court
Size | Error | Size | Error | Size | Error | Size | Error | Size | Error
(MB) | (m/deg.) | (MB) | (m/deg.) | (MB) | (m/deg.) | (MB) | (m/deg.) | (MB) | (m/deg.)
APR | PoseNet[28] | 50 | 0.88/1.04 | 50 | 3.2/3.29 | 50 | 0.88/3.78 | 50 | 1.6/3.32 | 50 | 6.83/3.47
FeatLoc[32] | 34 | 1.3/3.84 | 34 | 2.1/6.1 | 34 | 0.91/7.5 | 34 | 3.0/10.4 | - | -/-
DFNet[33] | 60 | 0.73/2.37 | 60 | 2.0/2.98 | 60 | 0.67/2.21 | 60 | 1.37/4.03 | - | -/-
FM | Hloc[6, 40] | 1877 | 0.11/0.20 | 1335 | 0.15/0.30 | 316 | 0.04/0.19 | 2009 | 0.07/0.22 | 1746 | 0.17/0.11
AS[38] | 275 | 0.57/0.70 | 140 | 0.52/1.12 | 37.7 | 0.12/0.41 | 359 | 0.22/0.62 | - | 0.24/0.13
SS[47] | 0.3 | 0.27/0.38 | 0.53 | 0.37/0.53 | 0.13 | 0.11/0.38 | 0.95 | 0.15/0.37 | - | -/-
SCR | DSAC++[17] | 104 | 0.18/0.3 | 104 | 0.20/0.3 | 104 | 0.06/0.3 | 104 | 0.13/0.4 | 104 | 0.40/0.2
SCoCR[14] | 165 | 0.18/0.3 | 165 | 0.19/0.3 | 165 | 0.06/0.3 | 165 | 0.09/0.3 | 165 | 0.28/0.2
DSAC*[15] | 28 |  0.15/0.3 | 28 | 0.21/0.4 | 28 |  0.05/0.3 | 28 | 0.13/0.4 | 28 | 0.49/0.25
D2S (ours) | 22 | 0.15/0.24 | 22 | 0.21/0.40 | 22 | 0.06/0.32 | 22 | 0.16/0.50 | 22 | 0.38/0.18
D2S+ (ours) | 22 |  0.15/0.25 | 22 |  0.19/0.34 | 22 | 0.05/0.26 | 22 | 0.13/0.40 | 22 | 0.25/0.15

####  4.1.3 Competitors

We compared our methods with other re-localization methods. For absolute pose
regression methods, we performed comparisons with PoseNet [28] (the updated
version), MapNet [59], FeatLoc [32], and DFNet [33]. For feature-matching-
based competitors, we reported the results of Inloc [24], Active Search [38],
and the state-of-the-art pipeline Hloc [40, 6]. For scene coordinates
regression approaches, we performed comparisons with SCoCR [14], KFNet [13],
the state-of-the-art pipeline DSAC* [15], DSAC++ [12] (the former version of
DSAC*), and a recent landmarks-based regression method NBE+SLD [19].

###  4.2 Results for Indoor Localization (7-Scenes)

We trained D2S on 7-Scenes using the experimental settings described in
Section 4.1. However, we did not apply any augmentation step on this dataset
considering its stationary nature, where the illumination remains consistent
between the training and testing data. The complete results are presented in
Table I. Notably, our D2S outperforms the previous baselines of SCR methods
including SCoCR[14], DSAC*[15], NBE+SLD[19], and KFNet [13] on Chess, Office,
Pumpkin, and Kitchen scenes. Note that all these baselines are CNN-based
regression methods: KFNet[13] is a temporal relocalization approach that
requires a sequence of input frames for high-accuracy localization, whereas
the proposed D2S is a one-shot frame method. Particularly, the proposed D2S
borrows the SuperPoint [2] extractor, which is similar to Hloc [6, 40]. Our
D2S still offers better accuracy on this dataset compared to Hloc. In
addition, Hloc requires several GB of memory for localization for these
scenes, whereas D2S requires only 22 MB for encoding the scene. For a clearer
difference in obtained results, we visualize the cumulative distributions of
pose error in Fig. 8. On average across six scenes, we obtained the best
accuracy among these baselines. However, in the Stairs scene, D2S and Hloc
exhibited the worst performance due to the textureless nature of this scene,
which challenges feature-based methods. Fig. 9 visualizes the estimated
trajectories of the proposed methods, including Hloc and DSAC*.

###  4.3 Results for Indoor Localization (Indoor6)

Indoor6 [19] is a challenging indoor dataset consisting of images collected at
different times and days. Therefore, we trained our D2S with each scene of
Indoor6 using the entire experimental settings demonstrated in Section 4.1.
Note that we performed an offline data augmentation step before training,
which required a runtime of approximately 8 minutes for pre-processing of 1000
unlabeled images. Subsequently, we proceeded with simultaneous training using
both the total augmented and original training data.

The results for this dataset are listed in Table II. Our D2S significantly
outperforms DSAC* [15], PoseNet[28], and NBE+SLD(E) [19] (lightweight
version). However, in certain cases, its performance is considerably lower
than that of NBE+SLD [19]. NBE+SLD trains two different models of landmarks
detector (SLD) and scene coordinate regression network (NBE) for each scene.
However, the proposed method can implement both criteria in the same D2S
network. Consequently, our D2S requires only a small memory for final
localization, whereas NBE+SLD demands six times more memory for both networks
(132 MB).

The Indoor6 [19] dataset provides a small amount of validation data that is
distinct from the training and test data. We used this data to evaluate our
proposed self-supervised learning method with unlabeled data. As shown in
Table II, the results for D2S+ (the version that updates with unlabeled data)
demonstrate that our proposed self-supervised learning procedure successfully
improved the localization performance of D2S by up to 9.6%.

TABLE IV: Median errors for BKC dataset, where the accuracy is calculated using the threshold of 0.5m and 10∘. Method | WestWing
---|---
Seq-2 (daytime - one view) | Seq-3 (daytime - two views) | Seq-4 (nighttime) | Average
Trans. Err | Rot. Err | Acc. | Trans. Err | Rot. Err | Acc. | Trans. Err | Rot. Err | Acc. | Trans. Err | Rot. Err | Acc.
(m) | (∘) | (%) | (m) | (∘) | (%) | (m) | (∘) | (%) | (m) | (∘) | (%)
DSAC*[15] | 29.6 | 67.1 | 0.0 | 8.92 | 28.5 | 0.0 | 8.85 | 57.0 | 0.0 | 15.8 | 50.8 | 0.0
D2S(ours) | 0.43 | 3.62 | 51.5 | 0.36 | 2.24 | 56.0 | 0.18 | 2.12 | 63.0 | 0.32 | 2.6 | 56.8
D2S+(ours) | 0.16 | 1.34 | 75.7 | 0.09 | 0.48 | 93.6 | 0.08 | 1.29 | 70.4 | 0.11 | 1.04 | 79.9
Hloc[6, 40] | 0.25 | 2.12 | 57.3 | 0.04 | 0.19 | 83.5 | 0.03 | 0.30 | 74.1 | 0.11 | 0.87 | 71.6
|  |  |  |  |  |  |  |  |  |  |  |
Method | CentralArc
Seq-3 (daytime - rain) | Seq-4 (daytime - sunny) | Seq-5 (nighttime) | Average
Trans. Err | Rot. Err | Acc. | Trans. Err | Rot. Err | Acc. | Trans. Err | Rot. Err | Acc. | Trans. Err | Rot. Err | Acc.
(m) | (∘) | (%) | (m) | (∘) | (%) | (m) | (∘) | (%) | (m) | (∘) | (%)
DSAC*[15] | 226.7 | 15.5 | 0.0 | 97.2 | 6.7 | 0.0 | 104.8 | 11.4 | 0.0 | 142.7 | 11.2 | 0.0
D2S(ours) | 1.26 | 5.97 | 29.6 | 0.66 | 3.19 | 46.7 | 10.5 | 82.3 | 4.65 | 4.14 | 30.5 | 27.0
D2S+(ours) | 0.73 | 3.43 | 33.7 | 0.23 | 1.15 | 72.8 | 0.93 | 7.25 | 30.2 | 0.63 | 3.91 | 45.6
Hloc[6, 40] | 3.86 | 14.3 | 25.5 | 0.04 | 0.14 | 77.2 | 1.16 | 4.69 | 41.9 | 1.69 | 6.38 | 48.2

Figure 10: Cumulative distributions of pose error (maximum of position and
rotation error) on different test sequences of BKC dataset. The deg† is a note
for calculation of the real threshold for rotation by $\times 10$ the
threshold. For example, the threshold of ”max. of m/deg† err. of 1.0” can be
seen as 1 m and 10∘.

###  4.4 Results for Outdoor Localization (Cambridge)

To evaluate the quality of re-localization performance on the Cambridge
dataset, we computed the median pose error for each scene, and the results are
reported in Table III. Among the evaluated approaches, APR-based approaches,
including PoseNet [28], FeatLoc [32], and DFNet [33], achieved the lowest
performance with low memory demand for all scenes. In contrast, the FM-based
Hloc [40, 6] pipeline achieved the highest re-localization accuracy. However,
Hloc requires saving global and local descriptors for training images, leading
to a high memory demand for each scene, as indicated in Table III. We also
report the results of the compressed version SceneSqueezer (SS) [47] on this
dataset, which has a significantly low memory footprint. However, its accuracy
is much lower than that of Hloc.

Moreover, we report the results of SRC-based approaches as well as the storage
demand of each scene. Overall, all the SRC-based methods reported in Table III
achieved a similar accuracy. However, considering the trade-off between memory
footprint and re-localization accuracy, the proposed method of D2S achieves
the best trade-off compared to other methods. Note that DSAC* also has a
compact version measuring 4 MB in size, but it yields significantly higher
localization errors. Therefore, we only reported the most accurate version of
DSAC* herein.

Finally, we evaluated the proposed self-supervised learning of D2S+ on the
Cambridge dataset. As this dataset does not provide unlabeled data, we used
half of the validation data as unlabeled data, which is similar to the
approach used in previous works [59, 33, 31]. Subsequently, we updated the
model using algorithm 1. The results are presented in Table III; notably, D2S+
also successfully improves the localization performance of D2S.

###  4.5 Results for Outdoor Localization (BKC)

This section reports the performance of the proposed method and other
baselines on the BKC dataset. This dataset has not been evaluated by previous
localization methods; therefore, we selected two state-of-the-art baselines,
FM-based Hloc [6, 40] and SCR-based DSAC* [15], to draw a comparison with the
proposed D2S.

For the evaluation of Hloc on this dataset, we used the following procedure.
In the offline phase, we used the SfM pseudo ground truths (pGT) generated
from COLMAP [35] to triangulate 3D points from SuperPoint features. In the
online phase, we used NetVLAD [9] to retrieve the top-10 nearest images from
the training database, followed by feature matching between the query and
retrieved images to determine 2D-–3D correspondences. Finally, we computed the
camera pose using pyColmap [73] with an inlier threshold of 12 px.

Figure 11: Test and unlabeled data. We show examples of similar images
collected for different purposes of testing and unlabeled images for the BKC
WestWing scene, which were recorded at different times.

For DSAC*+3D model (SfM) mode, we used the same configurations stated by the
authors. We trained the model with 1M iterations for an initial training step
and maintained an end-to-end training of differentiable RANSAC and pose
estimation with an additional 100K iterations.

Table IV reports the median localization errors of DSAC*, Hloc, and the
proposed D2S on different test sequences of the BKC dataset. For a more
detailed understanding, we also visualized the cumulative distribution of
errors on this dataset in Fig. 10. The BKC dataset features several
challenging properties for evaluating the visual localization methods on their
capacity to generalize beyond the training data. Thus, it is particularly
challenging for CNN-based SCR approaches as significant domain shifts can
result in substantial differences between training and test images. This is
evident in the DSAC* results on all testing sequences (Table IV and in Fig.
10). In the WestWing scene, as an example, DSAC* obtained the worst median
localization errors on Seq-2, which were 29.6m and 67.1∘, whereas the proposed
D2S achieved localization errors of 0.43m and 3.62∘. On average for three test
sequences, Hloc achieves the highest accuracy within 50cm and 10∘, which is
71.6%, compared to 56.8% for D2S and 0% for DSAC*. The same result is obtained
when evaluating BKC CentralArc, where Hloc still demonstrates its state-of-
the-art FM-based localization methods. In particular, Hloc achieves an
accuracy of 48.2%, D2S achieves an accuracy of 27%, whereas DSAC* still fails
in these testing situations. These results demonstrate that the proposed
sparse SCR-based D2S outperforms the dense CNN-based DSAC* in generalizing
beyond training data, including transitioning from day to night and adapting
to domain shifts even in the absence of training data sources.

In addition, the BKC dataset is also used for the evaluation of visualization
methods on updating with unlabeled observations. Each test sequence in
WestWing and CentralArc has a corresponding unlabeled sequence. We recorded
these data at different times and near the test sequence. Fig. 11 shows a
comparison of test and unlabeled images for the BKC WestWing scene.

Figure 12: Estimated trajectory. We visualize the estimated camera positions
on WestWing Seq-2. The small middle blue line is the ground truth, and the red
line is the estimated trajectory, which is connected by consecutive frames.

We report the results of D2S+ when updated with these unlabeled data in Table
IV and Fig. 10. Notably, the improvement in accuracy obtained through D2S,
from 56.8% to 79.9%, within the threshold of 50cm and 10∘ on the WestWing
scene surpasses Hloc’s accuracy of 71.6%. A large margin of improvement with
18.6% is also observed on the CentralArc scene. Particularly, when observing
the results with a higher threshold of up to 2m and 20∘, D2S+ outperforms
Hloc. Fig. 12 shows an example of estimated trajectories of Hloc, D2S, and
D2S+ on WestWing Seq-2, where D2S+ successfully predicts the camera position,
even at the positions located far from the scene.

##  5 Ablation Study

This section presents various ablation studies to further understand the
underlying mechanism of D2S.

###  5.1 Different Local Extractors

We measure the performance of D2S when working on different local extractors.
We selected three popular feature extractors: SIFT [8], R2D2 [4], and
SuperPoint [2]. SIFT and R2D2 possess the same descriptor dimension of 128,
which results in a minimal memory footprint of 12MB for D2S. In contrast,
SuperPoint has 254 dimensions for its descriptors, requiring 22MB of memory
consumption. Additionally, we also report the results of Hloc when using these
different feature extractors.

TABLE V: Median localization errors of D2S when learning on different local
feature extractors. We also report the results of Hloc for using different
extractors as a comparison to D2S.

Localization Method | Extractor | Size | 7scenes | Size | Cambridge
---|---|---|---|---|---
Chess | KingsCollege
(MB) | (m/deg./%) | (MB) | (m/deg.)
Hloc[6, 40] | SIFT[8] | 8.8K | 0.025/0.86/90.0 | 1.2K | 0.12/0.22
R2D2[4] | 2.1K | 0.028/0.95/84.8 | 0.7K | 0.13/0.22
SuperPoint[2] | 4.5K | 0.024/0.77/94.2 | 1.9K | 0.11/0.20
D2S (ours) | SIFT[8] | 12 | 0.039/1.46/64.7 | 12 | 0.52/0.72
R2D2[4] | 12 | 0.023/0.77/89.1 | 12 | 0.33/0.44
SuperPoint[2] | 22 | 0.017/0.57/98.6 | 22 | 0.15/0.24

We conducted experiments on two scenes, one indoor and one outdoor, using the
7-scenes Chess and the Cambridge KingsCollege. Table V reports the results of
D2S and Hloc when using different local features. For the indoor chess scene,
both D2S and Hloc achieved the highest re-localization accuracy when using
SuperPoint as the local feature extractor. However, considering the learning-
based features of R2D2 and SuperPoint, D2S obtained a better accuracy while
requiring only 12 and 22MB, compared to 2.1K and 4.5K MB used by Hloc,
respectively. For the outdoor scene, Hloc obtained the best localization
accuracy but also suffered from a huge memory requirement. The proposed D2S
performed better when using learning-based descriptors compared to hand-
crafted features.

Figure 13: Predicted uncertainty results on different descriptors including
SIFT[8], R2D2[4], and SuperPoint[2].

Furthermore, we also examined the behavior of D2S when learning with different
descriptors for reliability detection. Fig. 13 shows the predicted uncertainty
results on different descriptors of SIFT[8], R2D2[4], and SuperPoint[2] by the
proposed D2S with an uncertain threshold of 0.5. Notably, D2S demonstrates its
state-of-the-art classification performance when using SuperPoint features. In
Section 5.4, Table VII, we further report the efficient effects of this factor
on re-localization performance in both accuracy and inference time.

###  5.2 Graph Attention

This section presents a further examination of the manner in which graph
attention operates in D2S architecture. Fig. 14 displays the attention results
of both indoor and outdoor data. D2S shows a concentration on all existing
descriptors in the first layers, but finally (in layer 5), it shows special
interest only in robust keypoints. This trend illustrates that the proposed
D2S has succeeded at regressing the scene’s coordinates from noisy
descriptors.

To further understand the influence of graph attention on the improvement in
final re-localization accuracy, we conducted an ablation study by changing the
number of graph layers. The results are reported in Table VI, where the
proposed D2S using all five graph layers achieves a large margin of
enhancement compared to D2S (without using graph attention), under both indoor
and outdoor environments.

Figure 14: Attention results on different graph layers. We visualize the attention results (blue rays) for three attention layers of 1, 3, and 5 with the attention threshold of $\alpha_{ij}>5\times 10^{-4}$ for both indoor and outdoor data, where the green keypoints are the predicted robust features. TABLE VI: Median position and rotation errors of different graph attention settings on both indoor and outdoor scenes. The best results are in bold. | +Config. | 7scenes | Cambridge
---|---|---|---
| Chess | KingsCollege
D2S | No graph network | 0.022m/0.79∘/93.2% | 0.82m/1.10∘
Smaller (2 layers) | 0.019m/0.62∘/97.0% | 0.22m/0.36∘
Full (5 layers) | 0.017m/0.57∘/98.6% | 0.15m/0.24∘
Figure 15: Top-4 Query Results. We show the top-4 results of similar images to the query of test image using NetVLAD[9]. The dashed square of red color is a correct similar region for localization. The others are the failure cases. TABLE VII: Uncertainty Effects. We report the effects of uncertainty prediction on re-localization performance using the Cambridge [27] dataset. The error is reported as a median error of position and rotation (m and degree). The PnP RANSAC [73] estimation times (ms) are calculated by averaging over all the test images for each scene. D2S Uncer. | Kings College | Old Hospital | Shop Facade | St Mary’s Church | Great Court
---|---|---|---|---|---
Error | PnP time | Error | PnP time | Error | PnP time | Error | PnP time | Error | PnP time
(m/deg.) | (ms) | (m/deg.) | (ms) | (m/deg.) | (ms) | (m/deg.) | (ms) | (m/deg.) | (ms)
No | 0.16/0.25 | 76 | 0.27/0.44 | 302 | 0.08/0.35 | 195 | 0.18/0.54 | 548 | 0.42/0.21 | 257
Yes | 0.15/0.24 | 30 | 0.21/0.40 | 22 | 0.06/0.32 | 27 | 0.16/0.50 | 106 | 0.38/0.18 | 61
TABLE VIII: Inference time. We used an image size of 1024 for outdoor evaluation and 640 for indoor evaluation. In D2S, we report the feedforward times with two different numbers of input descriptors, 2048 and 1024. The inference time of PnP RANSAC [73] is dependent on the quality of predicted 3D coordinates (see Table VII). Image size | SuperPoint | D2S
---|---|---
| (2048/1024)
GPU | CPU | GPU | CPU
1024 | 19ms | 735ms | 4ms/4ms | 244ms/76ms
640 | 10ms | 278ms

###  5.3 Hloc and D2S+ Comparison

This section presents a comparison between Hloc and D2S+. As shown in Section
4.5, D2S+ performed better in locations lacking training data. Notably, both
D2S+ and Hloc used the same amount of reference/training data. We found
several failed cases in the image retrieval step, as illustrated in Fig. 15.
This resulted in an inferior performance of Hloc at locations lacking
reference data. However, D2S+ overcomes this drawback by learning the general
hints from unlabeled data even when facing this difficulty. Note that the
unlabeled data contains no information about the reconstructed SfM models.

###  5.4 Training and Inference Time

This section reports the training time and inference performance of the
proposed D2S. In our experiments, we used an Nvidia GeForce GTX 1080ti GPU and
an Intel Core i9-9900 CPU for both training and inference. For each
environment, D2S required 1.5 days to train using the aforementioned hardware.
In comparison, DSAC* [15] required 2.5 days to train, and DSAC++ [17] even
required up to 6 days to train on Tesla K80. However, we verified the training
time of DSAC* on our hardware, which requires the same duration of 1.5 days
for each scene, similar to the proposed D2S. SCR-based approaches encounter
protracted training times, but recent work using gradient decorrelation has
efficiently solved this issue [53]. Thus, we plan to optimize this in future
work.

Additionally, we report the inference performance of D2S in Table VII and
Table VIII. Herein, Table VII reports the average estimation times of the
RANSAC PnP [73] loop on two different settings (without and with D2S
uncertainty). Table VIII reports the feedforward times of SuperPoint [2] and
D2S when using the GPU or CPU. Overall, the proposed D2S infers with an
average of 53 ms from an RGB image to a 6DoF camera pose with the KingsCollege
scene while requiring only 45 ms with Old Hospital. Note that the total
estimation time depends on the quality of predicted scene coordinates.

##  6 Conclusion

In this study, we introduced D2S, a novel direct sparse regression approach
for establishing 2D–3D correspondences in visual localization. Our proposed
pipeline is both simple and cost-effective, enabling the efficient
representation of complex descriptors in 3D visual cloud maps. By utilizing an
attention graph and straightforward loss functions, D2S was able to accurately
classify the reliability of descriptors, leading to significant improvements
in re-localization performance. We conducted a comprehensive evaluation of our
proposed D2S method on four distinct indoor and outdoor datasets,
demonstrating its superior re-localization accuracy compared to state-of-the-
art baselines. Furthermore, we introduced a new dataset for assessing the
generalization capabilities of visual localization approaches. Our results
demonstrate that D2S is able to generalize beyond its training data, even in
challenging domain shifts such as day-to-night transitions and in locations
lacking training data. Additionally, D2S is capable of self-supervised
updating with new observations, eliminating the need for camera poses, camera
parameters, and ground truth scene coordinates. Overall, our approach
represents a promising step forward in the field of visual localization,
particularly in the context of sparse descriptors. Future work could focus on
further optimizing the training time and exploring other self-supervised
learning strategies to enhance localization performance in challenging
scenarios.

## References

  * [1] Herbert Bay, Tinne Tuytelaars, and Luc Van Gool.  Surf: Speeded up robust features.  Lecture notes in computer science, 3951:404–417, 2006.
  * [2] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superpoint: Self-supervised interest point detection and description.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 224–236, 2018.
  * [3] Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler.  D2-net: A trainable cnn for joint detection and description of local features.  arXiv preprint arXiv:1905.03561, 2019.
  * [4] Jerome Revaud, Philippe Weinzaepfel, César De Souza, Noe Pion, Gabriela Csurka, Yohann Cabon, and Martin Humenberger.  R2d2: repeatable and reliable detector and descriptor.  arXiv preprint arXiv:1906.06195, 2019.
  * [5] Mihai Dusmanu, Johannes L Schönberger, and Marc Pollefeys.  Multi-view optimization of local feature geometry.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pages 670–686. Springer, 2020.
  * [6] Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymczyk.  From coarse to fine: Robust hierarchical localization at large scale.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12716–12725, 2019.
  * [7] Philipp Lindenberger, Paul-Edouard Sarlin, Viktor Larsson, and Marc Pollefeys.  Pixel-perfect structure-from-motion with featuremetric refinement.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5987–5997, 2021.
  * [8] David G Lowe.  Distinctive image features from scale-invariant keypoints.  International journal of computer vision, 60:91–110, 2004.
  * [9] Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic.  Netvlad: Cnn architecture for weakly supervised place recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5297–5307, 2016.
  * [10] Arnold Irschara, Christopher Zach, Jan-Michael Frahm, and Horst Bischof.  From structure-from-motion point clouds to fast location recognition.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 2599–2606. IEEE, 2009.
  * [11] Simon Lynen, Bernhard Zeisl, Dror Aiger, Michael Bosse, Joel Hesch, Marc Pollefeys, Roland Siegwart, and Torsten Sattler.  Large-scale, real-time visual–inertial localization revisited.  The International Journal of Robotics Research, 39(9):1061–1084, 2020.
  * [12] Eric Brachmann, Alexander Krull, Sebastian Nowozin, Jamie Shotton, Frank Michel, Stefan Gumhold, and Carsten Rother.  Dsac-differentiable ransac for camera localization.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6684–6692, 2017.
  * [13] Lei Zhou, Zixin Luo, Tianwei Shen, Jiahui Zhang, Mingmin Zhen, Yao Yao, Tian Fang, and Long Quan.  Kfnet: Learning temporal camera relocalization using kalman filtering.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4919–4928, 2020.
  * [14] Xiaotian Li, Shuzhe Wang, Yi Zhao, Jakob Verbeek, and Juho Kannala.  Hierarchical scene coordinate classification and regression for visual localization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11983–11992, 2020.
  * [15] Eric Brachmann and Carsten Rother.  Visual camera re-localization from rgb and rgb-d images using dsac.  IEEE transactions on pattern analysis and machine intelligence, 44(9):5847–5865, 2021.
  * [16] Xiaotian Li, Juha Ylioinas, and Juho Kannala.  Full-frame scene coordinate regression for image-based localization.  arXiv preprint arXiv:1802.03237, 2018.
  * [17] Eric Brachmann and Carsten Rother.  Learning less is more-6d camera localization via 3d surface regression.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4654–4662, 2018.
  * [18] Clémentin Boittiaux, Claire Dune, Aurélien Arnaubec, Ricard Marxer, Maxime Ferrera, and Vincent Hugel.  Long-term visual localization in deep-sea underwater environment.  In ORASIS, 2023.
  * [19] Tien Do, Ondrej Miksik, Joseph DeGol, Hyun Soo Park, and Sudipta N Sinha.  Learning to detect scene landmarks for camera localization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11132–11142, 2022.
  * [20] Grant Schindler, Matthew Brown, and Richard Szeliski.  City-scale location recognition.  In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–7. IEEE, 2007.
  * [21] Akihiko Torii, Relja Arandjelovic, Josef Sivic, Masatoshi Okutomi, and Tomas Pajdla.  24/7 place recognition by view synthesis.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1808–1817, 2015.
  * [22] Tobias Weyand, Ilya Kostrikov, and James Philbin.  Planet-photo geolocation with convolutional neural networks.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14, pages 37–55. Springer, 2016.
  * [23] Torsten Sattler, Akihiko Torii, Josef Sivic, Marc Pollefeys, Hajime Taira, Masatoshi Okutomi, and Tomas Pajdla.  Are large-scale 3d models really necessary for accurate visual localization?  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1637–1646, 2017.
  * [24] Hajime Taira, Masatoshi Okutomi, Torsten Sattler, Mircea Cimpoi, Marc Pollefeys, Josef Sivic, Tomas Pajdla, and Akihiko Torii.  Inloc: Indoor visual localization with dense matching and view synthesis.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7199–7209, 2018.
  * [25] Vassileios Balntas, Shuda Li, and Victor Prisacariu.  Relocnet: Continuous metric learning relocalisation using neural nets.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 751–767, 2018.
  * [26] Soham Saha, Girish Varma, and CV Jawahar.  Improved visual relocalization by discovering anchor points.  arXiv preprint arXiv:1811.04370, 2018.
  * [27] Alex Kendall, Matthew Grimes, and Roberto Cipolla.  Posenet: A convolutional network for real-time 6-dof camera relocalization.  In Proceedings of the IEEE international conference on computer vision, pages 2938–2946, 2015.
  * [28] Alex Kendall and Roberto Cipolla.  Geometric loss functions for camera pose regression with deep learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5974–5983, 2017.
  * [29] Pulak Purkait, Cheng Zhao, and Christopher Zach.  Spp-net: Deep absolute pose regression with synthetic views.  arXiv preprint arXiv:1712.03452, 2017.
  * [30] Bing Wang, Changhao Chen, Chris Xiaoxuan Lu, Peijun Zhao, Niki Trigoni, and Andrew Markham.  Atloc: Attention guided camera localization.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10393–10401, 2020.
  * [31] Shuai Chen, Zirui Wang, and Victor Prisacariu.  Direct-posenet: absolute pose regression with photometric consistency.  In 2021 International Conference on 3D Vision (3DV), pages 1175–1185. IEEE, 2021.
  * [32] Thuan Bui Bach, Tuan Tran Dinh, and Joo-Ho Lee.  Featloc: Absolute pose regressor for indoor 2d sparse features with simplistic view synthesizing.  ISPRS Journal of Photogrammetry and Remote Sensing, 189:50–62, 2022\.
  * [33] Shuai Chen, Xinghui Li, Zirui Wang, and Victor A Prisacariu.  Dfnet: Enhance absolute pose regression with direct feature matching.  In European Conference on Computer Vision, pages 1–17. Springer, 2022.
  * [34] Torsten Sattler, Qunjie Zhou, Marc Pollefeys, and Laura Leal-Taixe.  Understanding the limitations of cnn-based absolute camera pose regression.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3302–3312, 2019.
  * [35] Johannes L Schonberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4104–4113, 2016.
  * [36] Linus Svärm, Olof Enqvist, Fredrik Kahl, and Magnus Oskarsson.  City-scale localization for cameras with known vertical direction.  IEEE transactions on pattern analysis and machine intelligence, 39(7):1455–1461, 2016.
  * [37] Carl Toft, Erik Stenborg, Lars Hammarstrand, Lucas Brynte, Marc Pollefeys, Torsten Sattler, and Fredrik Kahl.  Semantic match consistency for long-term visual localization.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 383–399, 2018.
  * [38] Torsten Sattler, Bastian Leibe, and Leif Kobbelt.  Efficient & effective prioritized matching for large-scale image-based localization.  IEEE transactions on pattern analysis and machine intelligence, 39(9):1744–1756, 2016.
  * [39] Liu Liu, Hongdong Li, and Yuchao Dai.  Efficient global 2d-3d matching for camera localization in a large-scale 3d map.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2372–2381, 2017.
  * [40] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superglue: Learning feature matching with graph neural networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4938–4947, 2020.
  * [41] Paul-Edouard Sarlin, Ajaykumar Unagar, Mans Larsson, Hugo Germain, Carl Toft, Viktor Larsson, Marc Pollefeys, Vincent Lepetit, Lars Hammarstrand, Fredrik Kahl, et al.  Back to the feature: Learning robust camera localization from pixels to pose.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3247–3257, 2021.
  * [42] Maxime Pietrantoni, Martin Humenberger, Torsten Sattler, and Gabriela Csurka.  Segloc: Learning segmentation-based representations for privacy-preserving visual localization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15380–15391, 2023.
  * [43] Torsten Sattler, Bastian Leibe, and Leif Kobbelt.  Fast image-based localization using direct 2d-to-3d matching.  In 2011 International Conference on Computer Vision, pages 667–674. IEEE, 2011.
  * [44] Yunpeng Li, Noah Snavely, and Daniel P Huttenlocher.  Location recognition using prioritized feature matching.  In Computer Vision–ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part II 11, pages 791–804. Springer, 2010.
  * [45] Federico Camposeco, Andrea Cohen, Marc Pollefeys, and Torsten Sattler.  Hybrid scene compression for visual localization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7653–7662, 2019.
  * [46] Marcela Mera-Trujillo, Benjamin Smith, and Victor Fragoso.  Efficient scene compression for visual-based localization.  In 2020 International Conference on 3D Vision (3DV), pages 1–10. IEEE, 2020.
  * [47] Luwei Yang, Rakesh Shrestha, Wenbo Li, Shuaicheng Liu, Guofeng Zhang, Zhaopeng Cui, and Ping Tan.  Scenesqueezer: Learning to compress scene for camera relocalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8259–8268, 2022.
  * [48] Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon.  Scene coordinate regression forests for camera relocalization in rgb-d images.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2930–2937, 2013.
  * [49] Julien Valentin, Matthias Nießner, Jamie Shotton, Andrew Fitzgibbon, Shahram Izadi, and Philip HS Torr.  Exploiting uncertainty in regression forests for accurate camera relocalization.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4400–4408, 2015.
  * [50] Abner Guzman-Rivera, Pushmeet Kohli, Ben Glocker, Jamie Shotton, Toby Sharp, Andrew Fitzgibbon, and Shahram Izadi.  Multi-output learning for camera relocalization.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1114–1121, 2014.
  * [51] Eric Brachmann and Carsten Rother.  Expert sample consensus applied to camera re-localization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7525–7534, 2019.
  * [52] Siyan Dong, Shuzhe Wang, Yixin Zhuang, Juho Kannala, Marc Pollefeys, and Baoquan Chen.  Visual localization via few-shot scene region classification.  arXiv preprint arXiv:2208.06933, 2022.
  * [53] Eric Brachmann, Tommaso Cavallari, and Victor Adrian Prisacariu.  Accelerated coordinate encoding: Learning to relocalize in minutes using rgb and poses.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5044–5053, 2023.
  * [54] Georg Klein and David Murray.  Improving the agility of keyframe-based slam.  In Computer Vision–ECCV 2008: 10th European Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part II 10, pages 802–815. Springer, 2008.
  * [55] Jakob Engel, Thomas Schöps, and Daniel Cremers.  Lsd-slam: Large-scale direct monocular slam.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part II 13, pages 834–849. Springer, 2014.
  * [56] Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos.  Orb-slam: a versatile and accurate monocular slam system.  IEEE transactions on robotics, 31(5):1147–1163, 2015.
  * [57] Martin Rünz and Lourdes Agapito.  Co-fusion: Real-time segmentation, tracking and fusion of multiple objects.  In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 4471–4478. IEEE, 2017.
  * [58] Shinya Sumikura, Mikiya Shibuya, and Ken Sakurada.  Openvslam: A versatile visual slam framework.  In Proceedings of the 27th ACM International Conference on Multimedia, pages 2292–2295, 2019.
  * [59] Samarth Brahmbhatt, Jinwei Gu, Kihwan Kim, James Hays, and Jan Kautz.  Geometry-aware learning of maps for camera localization.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2616–2625, 2018.
  * [60] Xiao-Shan Gao, Xiao-Rong Hou, Jianliang Tang, and Hang-Fei Cheng.  Complete solution classification for the perspective-three-point problem.  IEEE transactions on pattern analysis and machine intelligence, 25(8):930–943, 2003.
  * [61] Tong Ke and Stergios I Roumeliotis.  An efficient algebraic solution to the perspective-three-point problem.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7225–7233, 2017.
  * [62] Zuzana Kukelova, Martin Bujnak, and Tomas Pajdla.  Real-time solution to the absolute pose problem with unknown radial distortion and focal length.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2816–2823, 2013.
  * [63] Ondřej Chum, Jiří Matas, and Josef Kittler.  Locally optimized ransac.  In Pattern Recognition: 25th DAGM Symposium, Magdeburg, Germany, September 10-12, 2003. Proceedings 25, pages 236–243. Springer, 2003.
  * [64] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou.  Loftr: Detector-free local feature matching with transformers.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8922–8931, 2021.
  * [65] Wei Jiang, Eduard Trulls, Jan Hosang, Andrea Tagliasacchi, and Kwang Moo Yi.  Cotr: Correspondence transformer for matching across images.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6207–6217, 2021.
  * [66] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.
  * [67] Thuan B Bui, Dinh-Tuan Tran, and Joo-Ho Lee.  Fast and lightweight scene regressor for camera relocalization.  arXiv preprint arXiv:2212.01830, 2022.
  * [68] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [69] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl.  Neural message passing for quantum chemistry.  In International conference on machine learning, pages 1263–1272. PMLR, 2017.
  * [70] Mohammad Altillawi.  Pixselect: Less but reliable pixels for accurate and efficient localization.  In 2022 International Conference on Robotics and Automation (ICRA), pages 4156–4162. IEEE, 2022.
  * [71] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems, 32, 2019.
  * [72] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [73] Python bindings for colmap.  In https://github.com/colmap/pycolmap.
  * [74] Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon.  Kinectfusion: Real-time dense surface mapping and tracking.  In 2011 10th IEEE international symposium on mixed and augmented reality, pages 127–136. Ieee, 2011.