# Fast and Lightweight Scene Regressor for Camera Relocalization

Thuan B. Bui1, Dinh-Tuan Tran2, and Joo-Ho Lee2 1Graduate School of
Information Science and Engineering, Ritsumeikan University, Japan
<EMAIL_ADDRESS>of Information Science and Engineering,
Ritsumeikan University, Japan.

###### Abstract

Camera relocalization involving a prior 3D reconstruction plays a crucial role
in many mixed reality and robotics applications. Estimating the camera pose
directly with respect to pre-built 3D models can be prohibitively expensive
for several applications with limited storage and/or communication bandwidth.
Although recent scene and absolute pose regression methods have become popular
for efficient camera localization, most of them are computation-resource
intensive and difficult to obtain a real-time inference with high accuracy
constraints. This study proposes a simple scene regression method that
requires only a multi-layer perceptron network for mapping scene coordinates
to achieve accurate camera pose estimations. The proposed approach uses sparse
descriptors to regress the scene coordinates, instead of a dense RGB image.
The use of sparse features provides several advantages. First, the proposed
regressor network is substantially smaller than those reported in previous
studies. This makes our system highly efficient and scalable. Second, the pre-
built 3D models provide the most reliable and robust 2D-3D matches. Therefore,
learning from them can lead to an awareness of equivalent features and
substantially improve the generalization performance. A detailed analysis of
our approach and extensive evaluations using existing datasets are provided to
support the proposed method. The implementation detail is available at
https://github.com/ais-lab/feat2map.

##  I INTRODUCTION

Precise camera relocalization using visual sensors is a crucial step for many
applications including robotics [1], autonomous navigation [2], and augmented
reality [3]. Classical visual localization approaches build a three-
dimensional (3D) map, which is constructed based on dense or sparse image
pixels, of given environments beforehand. Each cloud point in a 3D sparse
cloud map is associated with one or many image descriptors, which are
subsequently used to match against two-dimensional (2D) keypoints detected
from query images [4]. Once the 2D-3D matches are established, a robust
perspective-n-points (PnP) solver within the random sample consensus (RANSAC)
algorithm [4] is implemented to estimate the camera pose. Although 3D
classical-based solutions have demonstrated a favorable recall performance,
these solutions cause several issues. For instance, extensive memory and
computational power are required when a large 3D map is provided.
Additionally, accurate 3D structures from motion (SfM) models are not widely
available for consumer-grade devices owing to limited computational resources.

To obtain solutions with increased efficiency, several approaches have been
introduced to learn to surrogate the entire map using only a neural network or
to compress the map. In general, these solutions can be categorized into three
classes: (1) The first class is absolute pose regression (APR). The goal of
this class method is to learn the entire pipeline of the relocalization system
in an end-to-end manner. Although this class demonstrates a potential solution
in terms of computing resources, its localization accuracy is poor [5]. (2)
The second class is scene compression, which aims to reduce memory consumption
by removing non-robust and unnecessary cloud points [6] or compressing the
descriptors of the 3D points [7] using quantization. (3) The third common
approach is scene coordinate regression, which was first introduced by Shotton
et al. [8]. This approach encodes entire 3D scene models using network weights
as regressor modules without explicitly storing the 3D models [9, 10].
Overall, 3D scene coordinate regression-based methods can achieve state-of-
the-art accuracy. However, most of them are computational-resource intensive
and reaching real-time inferences is difficult.

In this study, we propose a simple scene regression method that can solve the
aforementioned problems while achieving state-of-the-art localization
performance. Rather than learning the PnP or RANSAC in a differentiable manner
[11, 12], we keep geometry pose estimation to classic algorithms.
Additionally, we do not exploit deep learning to encode entire RGB data into a
dense 3D scene model. Instead, we begin our regressor from sparse descriptors.
This approach has several advantages:

  * •

Lightweight regressor architecture. Sparse data requires fewer computational
resources than that of dense data. The scene regressor network for this work
is, therefore, simple and lightweight.

  * •

Reasoning underlying 2D-3D matches. Mapping from sparse descriptors to world
coordinates can make the regressor aware of the robust equivalent image
features. Notably, these matches are extensively corrected beforehand by SfM
triangulation and feature matching. Therefore, training a network to reason
underlying matched descriptors can lead to a significant enhancement in
generalization performance.

  * •

Learning reliable descriptors. A 3D model created by an SfM method will
release reliable descriptors for learning its coordinates. Since these
descriptors have passed through a verification step and are matched from
different views through SfM, they are most likely reliable and belong to the
discriminatory areas in the image.

  * •

Handling unlabeled data. Interestingly, choosing to learn the scene
coordinates from descriptors presents a new potential to handle any unlabeled
data. We can match the key points detected in unlabeled images to the
available training descriptors and subsequently train the model in a semi-
supervised manner. This provides a straightforward technique to tackle new
unlabeled data. However, this study does not exploit this advantage in the
experiments. Instead, we demonstrate a simple learning procedure, which is
sufficient to achieve state-of-the-art performance.

The objective of this study is to address camera relocalization problems by
satisfying the constraints of low consumption of computational resources and
high accuracy. Fig. 1 illustrates the motivation for applying sparse scene
regression in this study. Our contributions can be summarized as follows:

  * •

We present a new camera relocalization pipeline and demonstrate that only a
simple multi-layer perceptron is sufficient to regress the scene coordinates.

  * •

We demonstrate that the proposed regression model is far lightweight and
faster than those of recent state-of-the-art dense regression methods and can
maintain the same localization errors.

  * •

We conduct an extensive evaluation on two datasets namely, 7scenes and
12scenes, and achieve state-of-the-art performance.

  * •

Interestingly, our proposed pipeline can work well by using a tiny regressor
network of only 0.7 million parameters and requires only approximately 10% of
the training data to generalize beyond the environments.

##  II RELATED WORK

Herein, we discuss the studies related to efficient camera relocalization that
leverages deep learning.

###  II-A Absolute Pose Regression

The absolute pose regression approach typically requires a large classifier
convolutional neural network that has been pre-trained on a large
classification image dataset. This neural network is then fine-tuned to
regress the 6DoF pose from either a single [13, 14] or sequence of images [15,
16]. The common practice of this area is first introduced by PoseNet [13].
Different learning strategies have different constraints either in their
network architectures [17, 9, 18] or loss functions [14, 9]. To improve
network architectures, long short-term memory (LSTM) PoseNet [17] includes a
combination of LSTM and a convolutional neural network to structure latent
features. A further improvement in the architecture is achieved by applying
the attention mechanism [18] for auto-learning the localization regions of
interest. By contrast, the constraints of improving the loss functions are
based on the properties of sequence images [9, 16] or underlying 3D geometry,
such as leveraging reprojection errors [14]. Owing to the directness of the
regression procedure, the execution time is short. However, these efforts do
not guarantee that these APR methods can be generalized beyond their training
data [5]. Recent studies additionally leveraged either 3D SfM models [19, 20]
or Nerf [21, 22, 23] to augment more training data. Although these approaches
seem promising, they are highly dependent on the quality of an expensive view
synthesizer and demonstrate no understanding of the underlying geometric
concepts.

###  II-B Scene Compression

To reduce the number of 3D scene points while maintaining the image-based
localization accuracy, Li et al. [6] proposed a K-cover-inspired algorithm to
identify a subset of points that sees at least K most quality points. Several
subsequent studies have extended this idea either to maximize the visual
distinctiveness [24] or visibility [25] of the selected K-points. Camposeco et
al. [26] proposed a K-cover hybrid-based algorithm to compress two sets of
points at different descriptor resolutions. Another common scene compression
algorithm uses quadratic programming to depict the problem as a convex
optimization formulation [27]. This algorithm, therefore, dramatically
improves the optimization speed of scene compression. Owing to heavy handcraft
and manual hyperparameter tuning of the above approaches, an efficient
learning-based method [7] has been proposed to select the compression points.
These approaches are excellent choices for efficient camera relocalization
tasks that are especially useful for mobile agents that have limited storage.
However, this study focuses on a special class of scene compression, in which
an efficient scene coordinate regression is favorable.

Figure 1: Motivation for this study, in which the sparse map is represented by
a simple scene regressor function $\mathfrak{F}$.

###  II-C Scene Coordinate Regression

The scene coordinate regression approach was proposed by Shotton et al. [8],
in which 2D-3D correspondences are formulated as continuous regression
problems. In this approach, estimating accurate camera poses using predicted
3D scene coordinates is straightforward because this replaces expensive
matching components with a random forest regressor. Several follow-up studies
have improved this approach in terms of accuracy [28] and learning camera
localization on-the-fly [29]. Recently, dense coordinates regression that
either learns from a single RGB image combined with differentiable PnP RANSAC
[9, 12] or leverages sequential images as a Kalman Filter [10], have become
popular. In contrast to these methods, we consider only discriminatory and
reliable image regions to regress the scene coordinates. This allows our study
to achieve a fast and lightweight localization pipeline while maintaining high
localization accuracy.

Figure 2: Simple regressor architecture.

##  III PROPOSED METHOD

In this section, we present the proposed approach for descriptor coordinates
regression. Our study is inspired by recent studies on camera relocalization,
including absolute pose and scene coordinate regression. The ultimate
objectives of these methods are low memory requirement, as well as fast and
accurate camera pose estimation within an inference time. To achieve these
objectives, we propose a straightforward method that requires lower
computational power than those required by the above methods. In the following
sections, we present the details of the proposed method for scene regression,
including the loss function used to train the network.

TABLE I: The median position and rotation errors of different relocalization methods on 7scenes dataset. Best results are in bold. Scene | Volume | E-PoseNet | FeatLoc | CamNet | Active | DSAC++ | SCoordNet | Proposed
---|---|---|---|---|---|---|---|---
[30] | [20] | [31] | Search [4] | [9] | [10]
Chess | 6m3 | 0.08m, 2.57∘ | 0.07m, 3.66∘ | 0.04m, 1.73∘ | 0.04m, 1.96∘ | 0.02m, 0.5∘ | 0.02m, 0.63∘ | 0.02m, 0.80∘
Fire | 2.5m3 | 0.21m, 11.0∘ | 0.17m, 5.95∘ | 0.03m, 1.74∘ | 0.03m, 1.53∘ | 0.02m, 0.9∘ | 0.02m, 0.91∘ | 0.03m, 1.13∘
Heads | 1m3 | 0.16m, 10.3∘ | 0.10m, 7.57∘ | 0.05m, 1.98∘ | 0.02m, 1.45∘ | 0.01m, 0.8∘ | 0.02m, 1.26∘ | 0.01m, 0.90∘
Office | 7.5m3 | 0.15m, 6.80∘ | 0.16m, 5.20∘ | 0.04m, 1.62∘ | 0.09m, 3.61∘ | 0.03m, 0.7∘ | 0.03m, 0.73∘ | 0.03m, 0.87∘
Pumpkin | 5m3 | 0.16m, 3.82∘ | 0.11m, 3.86∘ | 0.04m, 1.64∘ | 0.08m, 3.10∘ | 0.04m, 1.1∘ | 0.04m, 1.09∘ | 0.05m, 1.38∘
RedKitchen | 18m3 | 0.20m, 6.81∘ | 0.20m, 6.43∘ | 0.04m, 1.63∘ | 0.07m, 3.37∘ | 0.04m, 1.1∘ | 0.04m, 1.18∘ | 0.05m, 1.54∘
Stairs | 7.5m3 | 0.24m, 9.92∘ | 0.16m, 8.57∘ | 0.04m, 1.51∘ | 0.03m, 2.22∘ | 0.09m, 2.6∘ | 0.04m, 1.06∘ | 0.11m, 3.37∘
Average |  | 0.17m, 7.32∘ | 0.14m, 5.89∘ | 0.04m, 1.69 ∘ | 0.05m, 2.46 ∘ | 0.04m, 1.10∘ | 0.03m, 0.98∘ | 0.04m, 1.43∘
Figure 3: Localization accuracy obtained using the 7scenes and 12scenes datasets. The percentages of accurate test frames with pose error below 5 cm, 5∘ obtained using the 7scenes (a) and 12scenes (b) datasets are presented. The results obtained using the proposed method are marked in blue.  TABLE II: The median position and rotation errors of the proposed method against FeatLoc [20] on 12scenes dataset. Scene | Volume | FeatLoc | Proposed
---|---|---|---
[20]
A1 Kitchen | 33m3 | 0.32m, 5.19∘ | 0.02m, 0.83∘
A1 Living | 30m3 | 0.26m, 3.89∘ | 0.03m, 0.72∘
A2 Bed | 14m3 | 0.37m, 5.39∘ | 0.02m, 0.99∘
A2 Kitchen | 21m3 | 0.73m, 6.37∘ | 0.01m, 0.40∘
A2 Living | 42m3 | 0.40m, 5.71∘ | 0.04m, 1.14∘
A2 Luke | 53m3 | 0.33m, 4.85∘ | 0.02m, 1.03∘
O1 Gates362 | 29m3 | 0.52m, 5.22∘ | 0.01m, 0.30∘
O1 Gates381 | 44m3 | 0.42m, 6.23∘ | 0.02m, 1.01∘
O1 Lounge | 38m3 | 0.39m, 4.50∘ | 0.03m, 0.84∘
O1 Manolis | 50m3 | 0.30m, 4.67∘ | 0.02m, 0.82∘
O2 5a | 38m3 | 0.31m, 4.32∘ | 0.03m, 1.35∘
O2 5b | 79m3 | 0.23m, 4.14∘ | 0.03m, 0.94∘
Average |  | 0.38m, 5.04∘ | 0.02m, 0.86∘

###  III-A Overview of the Method

Given a sparse feature set
$\mathcal{P}_{t}=\\{\mathbf{d}_{i}\in\mathbb{R}^{M}|i=1,...,k\\}$ extracted
from a RGB image $\mathcal{I}_{t}$, where $\mathbf{d}$ is a descriptor vector
with a dimension of $M$ . In this study, we identify an accurate estimation of
the world coordinate
$\mathcal{W}_{t}=\\{\mathbf{w}_{i}\in\mathbb{R}^{3}|i=1,...,k\\}$ by proposing
a simple elements set regressor
$\mathfrak{F}(\mathcal{P}_{t}|\theta)=\mathcal{W}_{t}$. Here, $\mathfrak{F}$
is typically a deep neural network and $\theta$ is its network weight. As the
regressor receives the sparse features as the input, the proposed network is
remarkably lightweight and fast for relocalization. The details of the
proposed regressor are presented in the next section.

###  III-B Regressor Architecture

In this section, we propose a simple network architecture for learning to
regress the world coordinates of sparse features. The model accepts a
descriptors set of $\\{\mathbf{d}_{1},...,\mathbf{d}_{k}\\}$ as the input and
outputs a corresponding world coordinates of
$\\{\mathbf{w}_{1},...,\mathbf{w}_{k}\\}$. Notably, the extracted sparse
descriptors set has two properties, i.e., it is an unordered set and has a
variable number of descriptors. Therefore, the proposed architecture must be a
function of the elements set as follows:

$\displaystyle\mathfrak{F}(\\{\mathbf{d}_{1},...,\mathbf{d}_{k}\\})$
$\displaystyle=\\{h(\mathbf{d}_{1}),...,h(\mathbf{d}_{k})\\}$ (1)
$\displaystyle=\\{\mathbf{w}_{1},...,\mathbf{w}_{k}\\},$

where $\mathfrak{F}:\mathbb{R}^{K\times M}\to\mathbb{R}^{K\times 3}$ and
$h:\mathbb{R}^{M}\to\mathbb{R}^{3}$.

Empirically, the proposed module is simple. We only use a shared non-linear
function $h$ for every input descriptor, which is approximated as a multi-
layer perceptron (MLP) network. The final network architecture, therefore, is
simply a shared-parallel MLP function.

In the proposed module, we assume that the extracted descriptors have their
own distinctiveness and pair-wise matching proximity. With this assumption,
the proposed regressor architecture is well-suited to environments that have a
rich texture and less similar regions. Our proposed module is displayed in
Fig. 2, in which the $mlp$ function shares its weights with every input
descriptor.

###  III-C Loss Function

Once the scene coordinates are computed, we minimize the following loss
function to train the network:

$Loss(\theta)=\frac{1}{\lvert
N\lvert}\sum\limits_{i}^{N}\lVert\mathcal{\hat{W}}_{i}-\mathcal{W}_{i}\lVert_{2}$
(2)

where
$\mathcal{\hat{W}}=\\{\mathcal{\hat{W}}_{i},...,\mathcal{\hat{W}}_{N}\\}$ are
the predicted coordinates,
$\mathcal{W}=\\{\mathcal{W}_{i},...,\mathcal{W}_{N}\\}$ are the ground truth
coordinates, and $N$ is the mini-batch size.

This loss function can be improved using an additional method, which uses a
common reprojection loss combined with above function [9]. However, during the
experiments, we identified that this combination is prone to get stuck at the
local minimums and is unstable during training. Therefore, we simply used this
function to train all subsequent experiments.

##  IV EXPERIMENTS

In this section, we evaluated the proposed relocalization pipeline and
compared it with standard benchmarks to test the robustness of the pipeline.
We subsequently conducted an extensive ablation study to confirm our
hypotheses of choosing to regress scene coordinates from sparse descriptors.

###  IV-A Datasets

In accordance with previous studies [20, 9, 31, 10], we used two datasets
namely, 7scenes [8] and 12scenes [32] to evaluate the proposed method. 7scenes
is a collection of RGB-D sequence images that were captured by a handheld
Kinect RGB-D camera and consists of seven different indoor environments, in
which the spatial extent is less than 4 m. This dataset poses certain
difficulties for most localization pipelines using sparse features, such as
texture-less regions in the stairs scene or poor-quality blurred images. By
contrast, 12scenes contains images with better quality and larger environments
than those of 7scenes. The training sets of 12scenes are smaller and contain
only several hundred frames, whereas 7scenes has several thousand images in
its training and testing sequences.

###  IV-B Architecture and Setup

The architecture comprised several conventional MLP layers, and we used
SuperPoint [33] descriptors as the input features. Therefore, the input
dimension was $\mathbf{d}_{i}\in\mathbb{R}^{256}$, and the final regressor
architecture setup was MLPs $(512,1024,1024,512,3)$. Additionally, ReLU was
used as the nonlinearity activator for the entire network, excluding the last
layer, for producing scene coordinates.

To generate the ground truth scene coordinates, we reconstructed the SfM
models of the entire 7scenes and 12scenes datasets with a default image size
of $640\times 480$. We subsequently used the depth information to refine the
constructed 3D cloud maps to make scene coordinate labels reliable for
subsequent evaluations. Notably, the final inference pipeline required only
RGB images to estimate the scene coordinates. For the camera pose estimation,
we used PnP-RANSAC, which was implemented in pycolmap python binding using the
maximum error pixels of 12.

We implemented our algorithm using Pytorch [34] with the ADAM [35] optimizer
by considering $\beta_{1}=0.9,\beta_{2}=0.999$, and a weight decay of $5\times
10^{-4}$. We set the initial learning rate to $1e^{-3}$ with a decay of $0.5$
applied at each one-fifth of the total epochs. All experiments were conducted
on an NVIDIA GTX 1080ti GPU, in which the regressor network requires
approximately 1 ms of run-time. In accordance with previous studies [20, 18,
9, 10, 31], we computed the median errors for position and orientation. All
the experiments were trained using 1000 epochs, and the mini-batch size was
$8$.

###  IV-C Baselines

We compared our results with those obtained from recent APR methods that learn
from sparse features [20] or dense RGB images [22, 30]. These studies improved
localization performance using a large amount of additional synthetic data;
however, the proposed method can use a significant small amount of training
data for learning. More importantly, the proposed method exploits the 3D scene
geometry for learning and subsequently uses classical algorithms such as PnP
and RANSAC to estimate the camera poses. Therefore, we compared the proposed
method with Active Search [4], which is a long-standing benchmark for
geometry-based methods with all the components being classical. We
additionally compared our method with that reported by Valentin et al. [32],
DSAC [11], DSAC++[9], and SCoordNet [10] which also apply scene regression. We
further compared our method with the recent state-of-the-art relative pose
regression method i.e., CamNet [31]. For the listing of the results of the
baselines, we report the results mentioned by their original publications,
with the exception of the results obtained by Active Search, which were
reported by [10, 9].

TABLE III: Accuracy within different thresholds on 7scenes and 12scenes datasets. Threshold | 7scenes | 12scenes
---|---|---
3cm, 3∘ | 25% | 64%
5cm, 5∘ | 66% | 87%
10cm, 5∘ | 85% | 97%
Figure 4: Camera relocalization results on Chess-seq-03 and Redkitchen-seq-12
from the 7scenes dataset. The green points represent the ground truths and red
points represent the predictions. The result marked with a star indicates the
pipeline using only a tiny regressor network of 0.7 million parameters.

###  IV-D Relocalization Performance

Localization Errors. Table I lists the median localization errors such as
translation and rotation errors corresponding to the 7scenes dataset. The
proposed approach achieved the same position errors for the Chess, Heads, and
Office scenes as those of DSAC++ [9], and ScoordNet [10]. On the average of
the total scenes, the proposed approach approximately obtained the same
translation error as that of DSAC++, and, CamNet [31]. However, the rotational
error was higher than about $0.33^{\circ}$ and $0.45^{\circ}$ as that of
DSAC++ and SCoordNet respectively. In contrast, the proposed method
outperformed the conventional baseline of Active Search [4] and that of
absolute pose regression FeatLoc [20], and E-PoseNet [30]. Fig. 3a
demonstrates a comparison in terms of classification accuracy of the proposed
method with those of ORB + PnP [36], DSAC [11], and DSAC++. The results were
calculated based on the threshold of $5cm,5^{\circ}$, and our method achieved
approximately 66% accuracy. DSAC++ obtained a better one of 76.1%.

Similarly, we used the same comparison criteria for the 12scenes dataset, as
seen in the Table. II. However, previous studies have not reported the median
localization errors pertaining to this dataset. Therefore, we compared our
proposed method with a recent APR method namely, FeatLoc [20]. FeatLoc is a
lightweight APR approach that learns from sparse features, similar to our
approach. In details, the proposed method exhibited substantially lower
translational and rotational errors than those of FeatLoc. In Fig. 3b, we
compared the accuracy of our approach with those of the previous 3D scene
learning-based approaches. As shown in the figure, DSAC++ [9] achieves the
highest accuracy of 96.4%, whereas this study reached approximately 86.6%
accuracy. The proposed method outperformed the methods reported by Valentin et
al. [32], and Svarm et al. [37] with a difference of approximately 18.4% and
24.5% respectively. For some applications that do not require this level of
precision, our method scored 97% within $10cm,5^{\circ}$, as seen in Table
III. Fig. 4 displays some of the camera relocalization results.

TABLE IV: Comparison on the number parameters and running time of different scene regressor networks. Method | DSAC++ [9] | SCoordNet [10] | Proposed
---|---|---|---
#params | 210M | 24M | 3.6M
Time | 486.07ms | 156.6ms | 14ms
TABLE V: Localization accuracy of a significant small regressor network, which has only 0.7M parameters. Threshold | 7scenes |  | 12scenes |
---|---|---|---|---
| Accuracy | Median Errors | Accuracy | Median Errors |
5cm, 5∘ | 47.0% | 5.3cm, 1.64∘ | 69.1% | 3.7cm, 1.46∘ |
10cm, 5∘ | 81.5% | 90.6% |
TABLE VI: Median localization errors of the proposed method when learning with different numbers of training data, against LENS[22]. The results show that the proposed method only requires about 10% amount of training data to reach the same performance as that of LENS[22], where LENS requires 1100% amount of training data. The in bold results are the nearest ones which have an equal or lower error compared with that of LENS. Scene | The proposed |  | LENS [22]
---|---|---|---
100% | 80% | 60% | 40% | 20% | 10% | $real.+\frac{syn.}{real.}=1,100\%$
Chess | 0.02m, 0.8∘ | 0.02m, 0.8∘ | 0.02m, 0.8∘ | 0.03m, 0.9∘ |  0.03m, 1.0∘ | 0.04m, 1.3∘ | 0.03m, 1.3∘
Fire | 0.03m, 1.1∘ | 0.03m, 1.3∘ | 0.04m, 1.4∘ | 0.04m, 1.5∘ | 0.06m, 2.2∘ | 0.09m, 3.0∘ | 0.10m, 3.7∘
Heads | 0.01m, 0.9∘ | 0.02m, 1.2∘ | 0.02m, 1.2∘ | 0.03m, 1.7∘ | 0.04m, 2.3∘ | 0.06m, 3.7∘ | 0.07m, 5.8∘
Office | 0.03m, 0.9∘ | 0.03m, 1.0∘ | 0.04m, 1.0∘ | 0.04m, 1.1∘ | 0.05m, 1.3∘ | 0.08m, 2.0∘ | 0.07m, 1.9∘
Pumpkin | 0.05m, 1.4∘ | 0.05m, 1.4∘ | 0.05m, 1.5∘ | 0.05m, 1.5∘ | 0.06m, 1.5∘ | 0.08m, 1.8∘ | 0.08m, 2.2∘
Redkitchen | 0.05m, 1.5∘ | 0.06m, 1.6∘ | 0.06m, 1.6∘ | 0.06m, 1.7∘ | 0.08m, 1.9∘ | 0.10m, 2.3∘ | 0.09m, 2.2∘
Stairs | 0.11m, 3.4∘ | 0.18m, 5.1∘ | 0.20m, 5.6∘ | 0.23m, 6.1∘ | 0.31m, 8.1∘ | 0.57m,13.8∘ | 0.14m, 3.6∘

Systems Efficiency. As reported in the previous subsections, our proposed
method can outperform those reported in previous studies in terms of median
localization errors, while obtaining approximately the same results as those
of DSAC++ [9] and SCoorNet [10]. However, the proposed pipeline is cost-
effective, unlike the other methods. The scene regressor in our method
requires only 2.3 million parameters, whereas DSAC++ requires 210 million
parameters. Notably, this study uses SuperPoint [33] as the feature extractor.
The total number of parameters is, therefore, 3.6 million. Owing to the large
network architecture, DSAC++ has a runtime of 486 ms for producing the scene
coordinates. A single feedforward from the descriptors of the proposed method
requires only approximately 1 ms with 2048 input descriptors. In total, from
an RGB image with the size of $640\times 480$, the proposed pipeline runs
within 14 ms, which is $\times 34$ faster than that of DSAC++ and $\times 11$
faster than that of SCoordNet [10]. These comparisons are summarized in Table
IV.

Simple Regressor Achitecture. Here, we examine the proposed pipeline using a
small regressor network to determine the efficiency of the method when
learning from sparse descriptors. The network setting for this experiment is
as follows: MLPs$(512,512,512,128,3)$, where ReLU is the activation function
of the network. The total number of learnable parameters was only 0.7 million.
Table V lists the average median localization errors pertaining to the 7scenes
and 12scenes datasets. Additionally, we report the classification accuracies
obtained using these datasets within two different thresholds: $5cm,5^{\circ}$
and $10cm,5^{\circ}$. With a small network of 0.7 million parameters, our
approach achieved classification accuracies of 81.5% and 90.6% within
$10cm,5^{\circ}$ corresponding to the 7scenes and 12scenes datasets,
respectively. In comparison with previous studies, this result is even
superior to that of Active Search [4] in terms of rotation errors, while
presenting its outstanding lightweight system.

###  IV-E Generalization Performance

This section reports the generalization performance of the proposed method
when learning using less training data. We trained the proposed network with a
decreasing amount of training data from 100% to 10% in a random sample manner.
Table. VI reports the median localization errors in comparison with those of
LENS [22] at different percentages. With only a random selection of 60% data,
our pipeline can approximately maintain the same localization errors as those
obtained using 100% data. Interestingly, when learning with only 10-20% of the
data, the proposed method can outperform a recent state-of-the-art APR
approach of LENS. Notably, this study [22] has been improved by an additional
1000% synthetic images about the environments to reach this baseline. This
point of view has shown an important confirmation of the proposed approach to
generalization performance.

Furthermore, we conducted an ablation study on the behavior of the proposed
pipeline by reducing the number of input descriptors. Fig. 5 illustrates the
changes in the median localization errors, including translational and
rotational errors, on the Chess scene. All experiments were trained on 2048
descriptors per image. The model was found to be stable over 180 input
descriptors. For instance, the pipeline obtained median localization errors of
$0.022m,0.794^{\circ}$ at 2048 descriptors, $0.024m,0.864^{\circ}$ at 640
descriptors, and $0.030m,1.04^{\circ}$ at 180 descriptors. Additionally, the
pipeline obtained a high error of $0.053m,1.80^{\circ}$ at only 40 input
descriptors. However, the model can be considered to be generalizable with the
change of number input descriptors.

##  V CONCLUSIONS

In this study, we proposed a simple approach for scene coordinates regression
to achieve efficient camera relocalization. The proposed method is simple and
achieves state-of-the-art performance. Our method achieved 85% and 97%
accuracy within $10cm,5^{\circ}$ corresponding to the 7scenes and 12scenes
datasets, respectively, and operated $\times 11$ faster than recent state-of-
the-art scene regression methods [9, 10]. Interestingly, the proposed approach
requires only approximately 20% of the training data to outperform a recent
APR method [22], which has been improved by an expensive data augmentation
procedure.

In future work, we aim to pursue further uses of unlabeled data as a training
data source and explore the probabilistic filtering applied to dynamic
environments of this algorithm.

Figure 5: Median localization errors pertaining to the Chess scene against the
number of input descriptors per image.

## References

  * [1] Andrew J Davison, Ian D Reid, Nicholas D Molton, and Olivier Stasse.  Monoslam: Real-time single camera slam.  IEEE transactions on pattern analysis and machine intelligence, 29(6):1052–1067, 2007.
  * [2] Eric Royer, Maxime Lhuillier, Michel Dhome, and Jean-Marc Lavest.  Monocular vision for mobile robot localization and autonomous navigation.  International Journal of Computer Vision, 74(3):237–260, 2007.
  * [3] Robert Castle, Georg Klein, and David W Murray.  Video-rate localization in multiple maps for wearable augmented reality.  In 2008 12th IEEE International Symposium on Wearable Computers, pages 15–22. IEEE, 2008.
  * [4] Torsten Sattler, Bastian Leibe, and Leif Kobbelt.  Efficient & effective prioritized matching for large-scale image-based localization.  IEEE transactions on pattern analysis and machine intelligence, 39(9):1744–1756, 2016.
  * [5] Torsten Sattler, Qunjie Zhou, Marc Pollefeys, and Laura Leal-Taixe.  Understanding the limitations of cnn-based absolute camera pose regression.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3302–3312, 2019.
  * [6] Yunpeng Li, Noah Snavely, and Daniel P Huttenlocher.  Location recognition using prioritized feature matching.  In European conference on computer vision, pages 791–804. Springer, 2010.
  * [7] Luwei Yang, Rakesh Shrestha, Wenbo Li, Shuaicheng Liu, Guofeng Zhang, Zhaopeng Cui, and Ping Tan.  Scenesqueezer: Learning to compress scene for camera relocalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8259–8268, 2022.
  * [8] Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon.  Scene coordinate regression forests for camera relocalization in rgb-d images.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2930–2937, 2013.
  * [9] Eric Brachmann and Carsten Rother.  Learning less is more-6d camera localization via 3d surface regression.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4654–4662, 2018.
  * [10] Lei Zhou, Zixin Luo, Tianwei Shen, Jiahui Zhang, Mingmin Zhen, Yao Yao, Tian Fang, and Long Quan.  Kfnet: Learning temporal camera relocalization using kalman filtering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4919–4928, 2020.
  * [11] Eric Brachmann, Alexander Krull, Sebastian Nowozin, Jamie Shotton, Frank Michel, Stefan Gumhold, and Carsten Rother.  Dsac-differentiable ransac for camera localization.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6684–6692, 2017.
  * [12] Eric Brachmann and Carsten Rother.  Expert sample consensus applied to camera re-localization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7525–7534, 2019.
  * [13] Alex Kendall, Matthew Grimes, and Roberto Cipolla.  Posenet: A convolutional network for real-time 6-dof camera relocalization.  In Proceedings of the IEEE international conference on computer vision, pages 2938–2946, 2015.
  * [14] Alex Kendall and Roberto Cipolla.  Geometric loss functions for camera pose regression with deep learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5974–5983, 2017.
  * [15] Ronald Clark, Sen Wang, Andrew Markham, Niki Trigoni, and Hongkai Wen.  Vidloc: A deep spatio-temporal model for 6-dof video-clip relocalization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6856–6864, 2017.
  * [16] Abhinav Valada, Noha Radwan, and Wolfram Burgard.  Deep auxiliary learning for visual localization and odometry.  In 2018 IEEE international conference on robotics and automation (ICRA), pages 6939–6946. IEEE, 2018.
  * [17] Florian Walch, Caner Hazirbas, Laura Leal-Taixe, Torsten Sattler, Sebastian Hilsenbeck, and Daniel Cremers.  Image-based localization using lstms for structured feature correlation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 627–637, 2017.
  * [18] Bing Wang, Changhao Chen, Chris Xiaoxuan Lu, Peijun Zhao, Niki Trigoni, and Andrew Markham.  Atloc: Attention guided camera localization.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10393–10401, 2020.
  * [19] Pulak Purkait, Cheng Zhao, and Christopher Zach.  Spp-net: Deep absolute pose regression with synthetic views.  arXiv preprint arXiv:1712.03452, 2017.
  * [20] Thuan Bui Bach, Tuan Tran Dinh, and Joo-Ho Lee.  Featloc: Absolute pose regressor for indoor 2d sparse features with simplistic view synthesizing.  ISPRS Journal of Photogrammetry and Remote Sensing, 189:50–62, 2022\.
  * [21] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In European conference on computer vision, pages 405–421. Springer, 2020.
  * [22] Arthur Moreau, Nathan Piasco, Dzmitry Tsishkou, Bogdan Stanciulescu, and Arnaud de La Fortelle.  Lens: Localization enhanced by nerf synthesis.  In Conference on Robot Learning, pages 1347–1356. PMLR, 2022.
  * [23] Shuai Chen, Xinghui Li, Zirui Wang, and Victor Adrian Prisacariu.  Dfnet: Enhance absolute pose regression with direct feature matching.  arXiv preprint arXiv:2204.00559, 2022.
  * [24] Song Cao and Noah Snavely.  Minimal scene descriptions from structure from motion models.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 461–468, 2014.
  * [25] Wentao Cheng, Weisi Lin, Xinfeng Zhang, Michael Goesele, and Ming-Ting Sun.  A data-driven point cloud simplification framework for city-scale image-based localization.  IEEE Transactions on Image Processing, 26(1):262–275, 2016.
  * [26] Federico Camposeco, Andrea Cohen, Marc Pollefeys, and Torsten Sattler.  Hybrid scene compression for visual localization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7653–7662, 2019.
  * [27] Marcela Mera-Trujillo, Benjamin Smith, and Victor Fragoso.  Efficient scene compression for visual-based localization.  In 2020 International Conference on 3D Vision (3DV), pages 1–10. IEEE, 2020.
  * [28] Julien Valentin, Matthias Nießner, Jamie Shotton, Andrew Fitzgibbon, Shahram Izadi, and Philip HS Torr.  Exploiting uncertainty in regression forests for accurate camera relocalization.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4400–4408, 2015.
  * [29] Tommaso Cavallari, Stuart Golodetz, Nicholas A Lord, Julien Valentin, Luigi Di Stefano, and Philip HS Torr.  On-the-fly adaptation of regression forests for online camera relocalisation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4457–4466, 2017.
  * [30] Mohamed Adel Musallam, Vincent Gaudillière, Miguel Ortiz del Castillo, Kassem Al Ismaeil, and Djamila Aouada.  Leveraging equivariant features for absolute pose regression.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6876–6886, 2022.
  * [31] Mingyu Ding, Zhe Wang, Jiankai Sun, Jianping Shi, and Ping Luo.  Camnet: Coarse-to-fine retrieval for camera re-localization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2871–2880, 2019.
  * [32] Julien Valentin, Angela Dai, Matthias Nießner, Pushmeet Kohli, Philip Torr, Shahram Izadi, and Cem Keskin.  Learning to navigate the energy landscape.  In 2016 Fourth International Conference on 3D Vision (3DV), pages 323–332. IEEE, 2016.
  * [33] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superpoint: Self-supervised interest point detection and description.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 224–236, 2018.
  * [34] Nikhil Ketkar.  Introduction to pytorch.  In Deep learning with python, pages 195–208. Springer, 2017.
  * [35] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [36] Grant Schindler, Matthew Brown, and Richard Szeliski.  City-scale location recognition.  In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–7. IEEE, 2007.
  * [37] Linus Svarm, Olof Enqvist, Magnus Oskarsson, and Fredrik Kahl.  Accurate localization and pose estimation for large 3d models.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 532–539, 2014.