# Fusing Convolutional Neural Network and Geometric Constraint for Image-based
Indoor Localization

Jingwei Song, Mitesh Patel, and Maani Ghaffari J. Song and M. Ghaffari are
with the University of Michigan, Ann Arbor, MI 48109, USA.
<EMAIL_ADDRESS>Patel is with Nvidia Corporation, Santa Clara,
CA - 95051, USA<EMAIL_ADDRESS>
###### Abstract

This paper proposes a new image-based localization framework that explicitly
localizes the camera/robot by fusing Convolutional Neural Network (CNN) and
sequential images’ geometric constraints. The camera is localized using a
single or few observed images and training images with 6-degree-of-freedom
pose labels. A Siamese network structure is adopted to train an image
descriptor network, and the visually similar candidate image in the training
set is retrieved to localize the testing image geometrically. Meanwhile, a
probabilistic motion model predicts the pose based on a constant velocity
assumption. The two estimated poses are finally fused using their
uncertainties to yield an accurate pose prediction. This method leverages the
geometric uncertainty and is applicable in indoor scenarios predominated by
diffuse illumination. Experiments on simulation and real data sets demonstrate
the efficiency of our proposed method. The results further show that combining
the CNN-based framework with geometric constraint achieves better accuracy
when compared with CNN-only methods, especially when the training data size is
small.

###### Index Terms:

Vision-Based Navigation, Localization, Probability and Statistical Methods.

##  I Introduction

Image-based localization refers to retrieving 6 Degree-of-Freedom (DoF) pose
of the camera (monocular in most cases) using the obtained single or few
images (testing) and previous pose labeled images (training) [1]. Typical
applications include pedestrian localization with one image taken by mobile
phone or localization of service robot in indoor with low-cost camera.

Contrary to prior-free monocular Simultaneous Localization And Mapping (SLAM),
image-based localization can align the observed single/few images to the
training images’ coordinate and recover the pose globally. In monocular SLAM,
the system can only recover the camera pose locally (relative to the starting
pose) using a sequence of images without the correct scale. The scale and pose
in monocular SLAM suffer from drift [2]. Table I summarizes the differences.
Since image-based localization works with consumer-level monocular cameras, it
provides a fast and low-cost solution for a wide variety of applications.

Figure 1: The proposed method contains five major parts colored in blue: The similarity-based image locator, keyframe selection module, the geometric locator, the motion model, and the fusion module. The geometric locator and motion model are two parallel processes, and the fusion module combines the two predictions. The images are processed in a non sequential manner. TABLE I: The table shows the difference between monocular image-based localization and monocular SLAM. |  Monocular SLAM |  Image-based localization
---|---|---
Prior data |  N/A |  Images and pose labels
Scale |  No |  Yes
Pose coordinate |  Local |  Global
Number of images |  Sequential |  Single or few

Previous researches aligned the obtained image with the viewed scenery in the
training set based on the visual similarity (observed from a similar viewing
angle). The similarity is encoded in feature extractors like Speeded Up Robust
Features (SURF) or Oriented FAST and Rotated BRIEF (ORB). Later, PoseNet [3],
and SCORE forest [4] showed the machine learning method, especially the deep
Convolutional Neural Network (CNN), can map the images to its 6 DoF poses. The
trained deep CNN model was used to infer the pose of the testing image
directly. The follow up work developed different variants of PoseNet to
enhance the performance of image-to-pose deep CNN models [5, 6, 7, 1, 8, 9].

To estimate the uncertainties for the image-to-pose deep CNN model, Kendall
and Cipolla [10] adopted a Bayesian deep CNN framework to quantify the
estimated pose’s uncertainty. It addressed uncertainty based on the training
data set coverage, as it maintained the assumption that the “model is more
uncertain about images which are dissimilar to the training examples” [10].
The epistemic uncertainty measures the output in a data-fitting manner.
Similar studies can be found in [11, 12, 13]. In the Bayesian deep CNN
network, the pose uncertainty refers to the ambiguity caused by the unequaled
distributions between the training and testing samples, distinguishing itself
from the traditional geometric uncertainty. Unlike the Bayesian deep CNN
model, Liu et al. [14] inferred the covariance of the measurement from raw
sensory data directly. In contrast to the data-driven methods, the model-
driven aleatoric uncertainty [15, 16, 17, 18, 19] addresses the geometric
uncertainty. These geometric works followed the classical pinhole camera model
and obtained the 6 DoF pose uncertainty propagated from the uncertainty in 2D
tracking, measured as the 2D re-projection residuals sum.

In general, deep CNN-based image-to-pose approaches are only efficient if
images in the training and testing set are visually similar. Deep CNN-based
methods cannot retrieve good results if the images are collected from very
different perspectives. Moreover, the uncertainties in the deep CNN framework
measure the similarity between the training and testing images. The obtained
pose uncertainties from the Bayesian network have only valid statistical
meaning instead of geometric interpretations. Our work exploits the pinhole
camera projection model and is less dependent on the size of the training
data. Moreover, benefiting from the geometric uncertainty quantification, we
use a constant velocity camera motion model to incorporate the information of
consecutive images to gain robustness. By introducing the motion model, we
overcome the issues caused by blurry images, which randomly yields inaccurate
far-out predictions. The pose predictions provided by the camera projection
model and motion model are further fused based on the uncertainties.

The contributions of this work are as follows.

  1. 1.

A novel image-based localization framework that explicitly localizes the robot
by fusing deep CNN and sequential images’ geometric constraints.

  2. 2.

The predictions from the geometric locator and the motion model are fused by
incorporating their respective uncertainties during pose fusion.

  3. 3.

Our proposed framework requires relatively small training data set and can be
easily deployed for indoor localization tasks.

##  II Related Work

Image-based localization can be categorized as single monocular and sequential
monocular image-based localization.

Early researches used the monocular image captured by the consumer-level
camera for image-based localization. The prior-free Bag of Words (BoW)
algorithm [20] obtains the image descriptors to characterize the image
appearance. Due to its efficiency, BoW has been widely applied in SLAM systems
for loop closure detection [20]. Following BoW, other methods used SIFT
probability [21], deep CNN framework [22] and contextual re-weight network
[23] to generate image descriptors. All these methods only provide rough
camera localization and not the precise 6 DoF pose.

PoseNet [3] (monocular) and SCORE forest [4] (RGB-D) are learning-based
approaches to map from image-to-6 DoF pose. The learning-based approaches
bridge the gap between the learning and the geometric domain. Wang et al. [5]
introduced an attention model for better feature extraction. Weinzaepfel et
al. [6] claimed the benefits of an intermediate objects-of-interest extraction
and matching. Tian et al. [9] trained a deep CNN model with better efficiency
by addressing the 3D scene geometry-aware constraints in consecutive images.
The idea proposed by Tian et al. [9] is similar to our approach where
geometry-aware constraints are used via an implicit encoding in the training
process. However, we explicitly convert the image-to-pose CNN into a hybrid of
image-to-image matching and geometric localization. We also exploit the inter-
image relationship by explicitly using a motion model constraint.

Considering that short-term consecutive images may benefit the robustness and
accuracy of the image-to-pose fitting, further studies improved the deep CNN-
based framework with sequential images. Patel et al. [1] proposed a Long Short
Term Memory (LSTM) network termed as ContextualNet. The information within
sequential images is exploited to enhance the performance of the deep CNN
network. VLocNet [7] and VLocNet$++$ [8] integrated multiple deep CNNs to
predict 6 DoF pose, visual odometry, and semantic map segmentation
simultaneously. This multi-task deep CNN strategy significantly enhanced the
performance of the pose prediction. Brahmbhatt et al. [24], Yang et al. [13]
integrated conventional SLAM technique by feeding the redundant pose
predictions of multiple images to the pose graph or Bundle Adjustment (BA)
optimizations.

##  III Notation

Denote $(\cdot)^{\vee}:\mathfrak{se}(3)\to\mathbb{R}^{6}$ as an isomorphism
that converts an element of Lie algebra $\mathfrak{se}(3)$ to a 6-vector
(i.e., twist). Adversely,
$(\cdot)^{\wedge}:\mathbb{R}^{6}\to\mathfrak{se}(3)$. For the testing image
$i$, the goal of this work is to estimate the camera’s pose
$\mathbf{T}^{F}_{i}\in\mathrm{SE}(3)$. Additionally, the associated covariance
matrix is denoted ${\mathbf{C}}^{F}_{i}\in\mathbb{R}^{6\times 6}$. We define
$\operatorname{d}_{\mathfrak{se}(3)}(\mathbf{T}_{1},\mathbf{T}_{2})$ as a
distance between two poses $\mathbf{T}_{1},\mathbf{T}_{2}\in\mathrm{SE}(3)$ as

$\operatorname{d}_{\mathfrak{se}(3)}(\mathbf{T}_{1},\mathbf{T}_{2}):=\lVert\operatorname{log}(\mathbf{T}^{-1}_{1}\mathbf{T}_{2})^{\vee}\rVert_{2},$

where $\operatorname{log}(\cdot)$ is the matrix (Lie) logarithm map. The
misalignment angle between two orientations
$\mathbf{R}_{1},\mathbf{R}_{2}\in\mathrm{SO}(3)$ is

$\Theta(\mathbf{R}_{1}^{\mathsf{T}}\mathbf{R}_{2})=\operatorname{acos}((\operatorname{Tr}(\mathbf{R}_{1}^{\mathsf{T}}\mathbf{R}_{2})-1)/2),$
(1)

where $\operatorname{Tr}(\cdot)$ yields the trace of the input matrix. Denote
$\mathbf{T}|_{r}\in\mathrm{SO}(3)$ and $\mathbf{T}|_{p}\in\mathbb{R}^{3}$ as
the orientation and position from the pose $\mathbf{T}\in\mathrm{SE}(3)$.

##  IV Proposed Methodology

Fig. 1 illustrates the proposed framework, which consists of five modules: (a)
Aligning the single testing image to the most visually similar image in the
training set. (b) Selecting co-visible keyframes to the visually similar image
in the training set. (c) The testing image is localized using the selected co-
visible images using geometric features. (d) The camera pose of the testing
image is retrieved using a motion model based on the constant velocity
assumption to account for the sequential predictions. (e) The poses predicted
from the geometric locator and the motion model are fused, addressing both
uncertainties. If the image has no sequential prior predictions, only the
single image locator is used in the first few starting frames.

###  IV-A Image descriptor network

The image descriptor network aims at estimating the camera pose by aligning
the most visually similar image in the training set with the testing image,
using the trained deep CNN model. We modify the end-to-end image-to-pose
regression network PoseNet [3] as an image-to-image similarity searching for
following reasons. First, Kendall and Cipolla [10] pointed out that the
uncertainty of the PoseNet is closely related to the similarity of the
training and testing images. The Siamese network structure can be explicitly
used to compare the image similarities to exploit the information better.
Second, Zagoruyko and Komodakis [25] demonstrates that the Siamese network
with deep CNN model is efficient in measuring image similarity with limited
training data. Lastly, BA addresses the inherent geometric constraint and
provides an error propagation formulation mathematically derived from the
pinhole camera model.

Fig. 2 shows the structure of the Siamese network structure. Among the off-
the-shelf methods like [26, 27], or deep CNN-based method [28, 25], GoogLeNet
[29] is adopted as the base network since it works well in PoseNet [3] in
extracting image descriptors. The twin GoogLeNet models share the same
weights. All input images are regularized to grayscale images with $224\times
224$. Each twin network takes one of the two gray images. The branch’s outputs
are concatenated and fed into a fully connected layer to generate a one-
dimensional image descriptor. In the training process, arbitrary thresholds of
the pose are chosen to label the similarity of the image. We used contrastive
loss function [30] to measure the similarity between the encoded descriptors
(2). $\mathbf{s}_{1}$ and $\mathbf{s}_{2}$ are vectors yielded from the image
descriptor networks. Before training, we build a K-D tree on the descriptors
of the training images, which is further used to find the most similar
training image using the image descriptors. In (2), $\mathrm{m}$ is the
margin.

$\displaystyle
L\left(\mathbf{s}_{1},\mathbf{s}_{2}\right)=(1-\mathbf{s}_{1})\frac{1}{2}\left(\mathbf{s}_{2}\right)^{2}+(\mathbf{s}_{1})\frac{1}{2}\left\\{\max\left(0,\mathrm{m}-\mathbf{s}_{2}\right)\right\\}^{2}.$
(2)

###  IV-B Co-visible keyframes selection

After the testing image is aligned with the optimal training image, the
optimal training image’s co-visible keyframes are selected following two
principles: (I) Every two keyframes should have a suitable baseline to allow
enough parallax and overlap for triangulation. (II) The orientational
difference between the two images is trivial so that there is enough overlap
for triangulation. (I) ensures large enough parallax while (II) is to avoid
too small overlap within the selected images. We follow [31] and modify the
keyframe selection strategy by calculating the score. For two arbitrary
keyframes with camera poses $\mathbf{T}_{1}$ and $\mathbf{T}_{2}$, the score
is defined as

$\displaystyle\operatorname{s}$
$\displaystyle(\mathbf{T}_{1}\\!,\\!\mathbf{T}_{2}\\!)\\!=\\!\exp\\!\left(\\!-\frac{\left(\lVert\mathbf{p}_{1}^{2}-\mathbf{p}_{2}^{2}\lVert_{2}\\!-\\!d_{m}\right)^{2}}{\ell^{2}}\\!\right)\\!\cdot\\!\max\\!\left(\frac{\alpha_{m}}{\Theta\left(\\!\mathbf{R}_{1}^{\top}\\!\mathbf{R}_{2}\\!\right)},\\!1\\!\right),$

where $\mathbf{T}_{1}$ is composed of orientation $\mathbf{R}_{1}$ and
position $\mathbf{p}_{1}$ and $\mathbf{T}_{2}$ is composed of orientation
$\mathbf{R}_{2}$ and position $\mathbf{p}_{2}$. $d_{m}=0.1\mathop{\mathrm{m}}$
is the optimal baseline. $\alpha_{m}$ is set as $5^{\circ}$.

Figure 2: Illustrated is the structure of the Siamese network used for
training the image descriptor network, which is a modified version of
GoogLeNet.

The neighboring keyframes selection is done by searching the optimal training
image’s neighbors. Once the frame $k$ is aligned by deep CNN, the backward (or
forward) keyframe seeds $k_{b}$ (or $k_{f}$) are set as $k$. The backward (or
forward) direction searches the optimal (highest score) keyframe in the range
$[k_{b}-\triangle k,k_{b}]$/$[k_{f},k_{f}+\triangle k]$ ($\triangle k>0$).
Then the optimal keyframe is assigned as $k_{b}$ (or $k_{f}$). This process is
iteratively carried out on both directions until the joint keyframe group
reaches the maximum number.

###  IV-C The single image locator

The geometric locator consists of training image sparse mapping (forward-
intersection) and testing image localization (backward-intersection). It
localizes a single image geometrically, and both modules are simplified from
BA [15].

Forward-intersection triangulates sparse 3D map points from the 2D tracking in
the training set. The predicted most similar image and its neighbors in
Section IV-B are adopted. As Fig. 3 shows, the predicted image, as well as its
co-visible images (blue camera in Fig. 3), go through the SURF to be tracked
with 2D key corners. All selected images are paired and matched for reliable
tracking. The poses of the selected images are
$\mathbf{T}_{k}\in\mathrm{SE}(3)$. Define the 2D point $j$ tracked on image
$k$ as $\mathbf{x}^{j}_{k}$. Note that some points cannot be tracked on all
images. Projecting the 3D point $\mathbf{X}\in\mathbb{R}^{3}$ in global
coordinate to the image with pose $\mathbf{T}\in\mathrm{SE}(3)$ is

Figure 3: The figure shows the forward-intersection and the backward-
intersection. The red camera (I) is the testing pose $\mathbf{T}_{i}$ while
the three blue cameras are the training set (II, III and IV)
$\mathbf{T}_{t-2}$, $\mathbf{T}_{t-1}$ and $\mathbf{T}_{t}$. Three 2D feature
points are tracked on the training and testing images. The forward-
intersection triangulates the global coordinates of the 3D feature points
tracked in the training images using the labeled poses. After the forward-
intersection, the backward-intersection estimates the pose of the testing
images by minimizing the re-projection errors of the 3D triangulated feature
points.

$(\Pi(\mathbf{T},\mathbf{X})^{\mathsf{T}}\mid
1)^{\mathsf{T}}:=\mathbf{K}\mathbf{T}^{-1}(\mathbf{X}^{\mathsf{T}}\mid
1)^{\mathsf{T}},$

where $\mathbf{x}:=\Pi(\mathbf{T},\mathbf{X})^{\mathsf{T}}\in\mathbb{R}^{2}$
is the 2D projection. It obtains its homogeneous version.
$\mathbf{K}\in\mathbb{R}^{3\times 4}$ is the camera intrinsic matrix. Forward-
intersection retrieves the 3D map points $\mathbf{X}^{1},\dots,\mathbf{X}^{t}$
by minimizing the sum re-projection residual as

$\begin{split}\min\limits_{\mathbf{X}^{1},\dots,\mathbf{X}^{t}}&\sum_{k\in\mathit{T}}\sum_{j\in\Omega_{k}}\lVert\Pi(\mathbf{T}_{k},\mathbf{X}^{j})-{\mathbf{x}}_{k}^{j}\lVert^{2},\end{split}$
(3)

where $\mathit{T}$ is the set of all selected training images and $\Omega_{k}$
is the group of key points ${\mathbf{x}}_{k}^{j}$ tracked on image $k$. Huber-
loss [32, 33] is applied as the robustifier to handle outliers. All trackings
are treated with equal weights. Moreover, the triangulated points with an
average residual greater than threshold $\epsilon_{r}$ are deleted.

Backward-intersection localizes the camera of the testing image
$\mathbf{T}^{G}_{i}\in\mathrm{SE}(3)$ with the triangulated 3D points. The 3D
feature points are matched with the testing image (red camera in Fig. 3).
Define $\Omega_{i}$ as the group of the tracked 2D points on image $i$.
$\mathbf{T}^{G}_{i}$ is estimated by minimizing the sum of re-projection
residual (4) as

$\begin{split}\min\limits_{\mathbf{T}^{G}_{i}\in\mathrm{SE}(3)}&\sum_{j\in\Omega_{i}}\lVert\Pi(\mathbf{T}^{G}_{i},\mathbf{X}^{j})-\tilde{\mathbf{x}}_{i}^{j}\lVert^{2},\end{split}$
(4)

where $\tilde{\mathbf{x}}_{i}^{j}$ is the $j$th tracked points on the testing
image $i$. Denote the sum of the residuals in (4) as $\mathbf{r}_{i}$. Similar
to forward-itersection, Huber-loss [32, 33] is applied and all tracking are
treated with equal weights (variance is set to $1$ pixel). Combining the
identity variance matrix of the pixel-level and the Huber loss factors, the
information matrix of each map point $\mathbf{I}(\tilde{\mathbf{x}}_{i}^{j})$
can be obtained. Denote $\mathcal{J}^{-1}_{j}(\mathbf{T}^{G}_{i})$ as the
right Jacobian of the per-pixel residuals in (4) following the definition in
[34], the information matrix of the pose $\mathbf{T}^{G}_{i}$ is given by

$\mathbf{I}(\mathbf{T}^{G}_{i})=\frac{1}{\sigma(\overline{\mathbf{r}}_{i})^{2}}\sum_{j\in\Omega_{i}}\mathcal{J}^{-1}_{j}(\mathbf{T}^{G}_{i})^{\mathsf{T}}\mathbf{I}(\tilde{\mathbf{x}}_{i}^{j})\mathcal{J}^{-1}_{j}(\mathbf{T}^{G}_{i}).$

We follow [18] to approximate the average residual variance
$\sigma(\overline{\mathbf{r}}_{i})^{2}$ as

$\sigma(\overline{\mathbf{r}}_{i})^{2}\approx\lVert\mathbf{r}_{i}\lVert^{2}/(2\mathrm{p}-1),$

where $\mathrm{p}$ is the number of tracking in $\Omega_{i}$. With the
information matrix $\mathbf{I}(\mathbf{T}^{G}_{i})$, the covariance matrix
${\mathbf{C}}^{G}_{i}\in\mathbb{R}^{6\times 6}$ can be obtained as

${\mathbf{C}}^{G}_{i}=\mathbf{I}(\mathbf{T}^{G}_{i})^{-1}.$ (5)

###  IV-D Probabilistic motion model

As Patel et al. [1] pointed out, single image-based localization suffers from
predictions that deviate significantly from consecutive predictions.
Sequential predictions can alleviate this issue as it uses information from
multiple images. Researchers developed different techniques that exploits
sequential information using both deep CNN based methods [1, 7, 8, 9] and
geometry perspective [24, 13]. We applied short-term robot motion model [35,
34] to constrain the consecutive predictions. The short-term robot motion
model is often categorized as Brownian, constant velocity, constant
acceleration, and constant turn [35, 36]. We employ a constant velocity-based
probabilistic motion model to fit the short-term robot motion. The motion
model is first fitted with the short-term historical prediction and then
applied to predict the testing image’s pose and the uncertainty. Following the
constant motion interpolation (in $\mathrm{SE}(3)$) in [34], we define the
short-term constant motion $\delta_{\mathbf{T}}\in\mathrm{SE}(3)$ and
optimized the objective function as

$\begin{split}\min\limits_{\delta_{\mathbf{T}},\mathbf{T}^{M}_{i}}&\sum_{j=1}^{t}\lVert\operatorname{d}_{\mathfrak{se}(3)}({\mathbf{T}^{F}_{i-j}},\mathbf{T}^{M}_{i}\delta_{\mathbf{T}}^{j-1})\lVert^{2},\end{split}$
(6)

where $t$ is the number of historical images in the window of the motion
model, $\mathbf{T}^{M}_{i}\in\mathrm{SE(3)}$ is the $i$th estimated pose. The
basic idea is to search for the optimal $\mathbf{T}^{M}_{i}$ and constant
motion $\delta_{\mathbf{T}}$ to fit the consecutive predictions. After the
estimation of the probabilistic motion model (6), the one-step prediction of
robot pose can be achieved.

The covariance matrix (diagonal) of $\mathbf{T}^{M}_{i}$ from the
probabilistic motion model, which is ${\mathbf{C}}^{M}_{i}$, is calculated
statistically. It describes the fitting accuracy of the probabilistic motion
model. The uncertainty only quantifies the short-term epistemic uncertainty
and suffers from uncertainty drift in the long run. Precisely, if the
uncertainties of the motion model are propagated, they tend to be over-
confident.

###  IV-E Fusion of the geometric locator and the motion model

The predictions of the geometric locator and the probabilistic motion model
are fused with the quantified uncertainties following [37] as

$\begin{split}\min\limits_{\mathbf{T}^{F}_{i}}&\operatorname{d}_{\mathfrak{se}(3)}(\mathbf{T}_{i}^{M},\mathbf{T}^{F}_{i})^{\mathsf{T}}\left({\mathbf{C}}^{M}_{i}\right)^{-1}\operatorname{d}_{\mathfrak{se}(3)}(\mathbf{T}_{i}^{M},\mathbf{T}^{F}_{i})\\\
&+\operatorname{d}_{\mathfrak{se}(3)}(\mathbf{T}_{i}^{G},\mathbf{T}^{F}_{i})^{\mathsf{T}}\left({\mathbf{C}}^{G}_{i}\right)^{-1}\operatorname{d}_{\mathfrak{se}(3)}(\mathbf{T}_{i}^{G},\mathbf{T}^{F}_{i}),\end{split}$
(7)

$\displaystyle{\mathbf{C}}^{F}_{i}\\!=\\!\left({\mathcal{J}_{r}^{M}}^{-\mathsf{T}}\left({\mathbf{C}}^{M}_{i}\right)^{-1}{{\mathcal{J}_{r}}^{M}}^{-1}+{\mathcal{J}_{r}^{G}}^{-\mathsf{T}}\\!\left({\mathbf{C}}^{G}_{i}\right)^{-1}{\mathcal{J}_{r}^{G}}^{{}^{-1}}\\!\right)^{-1}\\!,$

where ${\mathcal{J}^{M}_{r}}^{-1}$ and ${\mathcal{J}^{G}_{r}}^{-1}$ are the
right Jacobians of objective function (7) regarding $\mathbf{T}^{M}_{i}$ and
$\mathbf{T}^{G}_{i}$ following the definition in [34]. To filter outliers from
the single image locator, the isometric standard deviation
$3\sigma({\mathbf{T}_{i}^{G}})|_{p}$ and $3\sigma({\mathbf{T}_{i}^{G}})|_{r}$
are enforced for filtering $\mathbf{T}_{i}^{G}$. If the positional or
orientational predictions of $\mathbf{T}_{i}^{G}$ exceeds the bounds,
$\mathbf{T}_{i}^{G}$ is identified as failure and the motion model pose
prediction $\mathbf{T}^{M}_{i}$ is assigned to $\mathbf{T}^{F}_{i}$. The
isometric standard deviation ${\sigma}(\mathbf{T}^{G}_{i})|_{p}$ and
${\sigma}(\mathbf{T}^{G}_{i})|_{r}$ are

${\sigma}(\mathbf{T}^{G}_{i})|_{p}=\frac{1}{3}\sum_{i=1}^{3}\sqrt{{\mathbf{C}}^{G}_{i}|_{p}},\
{\sigma}(\mathbf{T}^{G}_{i})|_{r}=\Theta(\exp(\sqrt{{\mathbf{C}}^{G}_{i}|_{r}})^{\wedge}),$

where scalar ${\sigma}(\mathbf{T}^{G}_{i})|_{p}$ represents position’s
ellipsoid uncertainty. Similarly, scalar ${\sigma}(\mathbf{T}^{G}_{i})|_{r}$
is the orientational uncertainty.

The motion model provides an independent and cheap way to rectify the
outliers. Unlike the global positioning of the geometric locator, the
probabilistic motion model retrieves the local pose with the estimated short-
term historical poses. Its uncertainty is only valid in the short term.
Assuming the error of the pose from the geometric locator follows the Gaussian
distribution if the motion model prediction deviates the geometric locator for
more than 3-$\sigma$ bound, the prediction of the geometric locator is
discarded. The large difference indicates unreasonable abrupt pose prediction.
Other filtering methods, such as the extended Kalman filter, are unsuitable
because the geometric locator yields consecutive outliers.

##  V Experiments

We evaluated our method on three data sets (one synthetic and two real-world).
The synthetic data set was generated in a virtual warehouse using Isaac Sim
[38] engine. The real-world public data set used for our experiments were
7Scenes [4] and TUM RGB-D data sets [39]. We also used a Monte-Carlo
experiment to validate the uncertainty quantification proposed in (5).

###  V-A Monte-Carlo synthetic experiment

To validate the uncertainty quantification in (5), a Monte-Carlo experiment
was conducted to test the geometric locator. It follows the scenario in Fig.
3. $119$ 3D points were partially projected on the images of the 4 camera
poses and the target camera. Gaussian noises (i.i.d.) with variance
$\sigma_{p}^{2}$ were exerted on all 2D tracking. The coverage rate is defined
as the difference between the estimated pose and the ground truth pose
$\lVert\mathbf{T}^{G}_{k}|_{p}-\overline{\mathbf{T}}^{G}|_{p}\rVert_{2}$ and
$\Theta(\mathbf{T}^{G}_{k}|_{r}^{\mathsf{T}}\overline{\mathbf{T}}^{G}|_{r})$
within
$[-1.96{\sigma}(\mathbf{T}^{G}_{i})|_{p},1.96{\sigma}(\mathbf{T}^{G}_{i})|_{p}]$
and
$[-1.96{\sigma}(\mathbf{T}^{G}_{i})|_{r},1.96{\sigma}(\mathbf{T}^{G}_{i})|_{r}]$,
where $k$ is the $k$th Monte-Carlo experiment and $\overline{\mathbf{T}}^{G}$
is the ground truth. To further evaluate the efficiency with the presence of
noise in pose labeling, we corrupt pose labels using simulated noise to
replicate a more realistic scenario. Table II shows the coverage rate using
different $\sigma_{p}$ and different pose label noises. The ideal coverage
rate is $95\%$.

TABLE II: The table shows the Monte-Carlo experiments with 1000 trials. We
show the coverage rate of different levels of tracking noises. We also tested
adding isometric pose noises to pose labels (cells are standard deviation of
the noises).

| Pose noise
---
$0\mathop{\mathrm{m}}$, $0^{\circ}$ |  $0.01\mathop{\mathrm{m}}$, $0.5^{\circ}$ |  $0.03\mathop{\mathrm{m}}$, $1^{\circ}$
|  $\sigma_{p}$ (pixel)
---
1 |  2 |  1 |  2 |  1 |  2
Angular |  $95.2\%$ |  $94.6\%$ |  $84.6\%$ |  $79.8\%$ |  $49.8\%$ |  $62.4\%$
Positional |  $96.0\%$ |  $94.2\%$ |  $92.8\%$ |  $91.0\%$ |  $74.4\%$ |  $80.6\%$

###  V-B Data set

Our method was tested on the synthetic data set generated using Isaac Sim
engine [38], that is photorealistic and physically accurate. The monocular
images captured by the camera mounted on the robot were logged along with the
corresponding poses while the robot maneuvered in the warehouse. Two data sets
were generated in two different virtual environments with different levels of
complexity. The synthesized images were in $1280\times 720$. Fig. 4 shows
sample images.

Figure 4: Shown are the examples of the synthetic data set generated using
Isaac Sim simulator. The upper two and lower two figures are taken in diffuse
lighting and indoor lighting respectively.

The proposed approach was also tested on the small-range 7Scenes [4] indoor
data set, which was collected with a handheld Kinect RGB-D camera at the
resolution of $480\times 640$. The pose labels were generated by implementing
the KinectFusion [40] SLAM on the RGB-D images. The monocular images contain
motion blur, flat surfaces, and various lighting conditions. The TUM RGB-D
data set [39], which was designed to test the performance of RGB-D SLAM
algorithms, was also adopted. In TUM RGB-D data set, a motion capture system
was used to track the reflectance markers on the Kinect camera accurately, and
the intrinsic parameter was calibrated. Therefore, the accuracy of the pose
label and intrinsic parameters are substantially improved compared to the
7Scenes. The image resolution is $480\times 640$ and contains apparent
challenges including motion blur, flat surfaces, and various lighting
conditions. We chose the last 100 images as the testing data set and the
remaining as the training data set. Data sets “freiburg1 desk2”, “freiburg1
floor”, “freiburg1 room”, “freiburg2 360 hemisphere”, “freiburg2 360 kidnap”,
“freiburg2 large with loop” were chosen since most of the last 100 frames were
revisited in the training set. We only used RGB images and the pose labels.

Figure 5: (a,c,e,g) are the sample testing images in the TUM data sets “desk2”, “floor”, “room” and “large_with_loop”. (b,d,f,h) are the corresponding most visually similar images in the training data set predicted by the image descriptor network. We achieve the accuracy $(0.02\mathop{\mathrm{m}},1.51^{\circ})$, $(0.01\mathop{\mathrm{m}},2.45^{\circ})$, $(0.03\mathop{\mathrm{m}},1.37^{\circ})$ and $(0.55\mathop{\mathrm{m}},6.54^{\circ})$ in contrast to the predictions $(0.33\mathop{\mathrm{m}},5.53^{\circ})$, $(0.05\mathop{\mathrm{m}},1.66^{\circ})$, $(0.35\mathop{\mathrm{m}},2.27^{\circ})$ and $(0.97\mathop{\mathrm{m}},7.99^{\circ})$ from PoseNet. TABLE III: The table shows the median localization errors on the synthetic data set generated using Isaac Sim. “Warehouse1” is the normal illumination, and “warehouse2” is in dark illumination. “BoW+Locator” is the integration of BoW with our geometric Locator. |  BoW + Locator |  PoseNet |  ContexualNet |  Our method (Single image) |  Our method (Multiple image)
---|---|---|---|---|---
| Warehouse1
---
(simple)
$0.55\mathop{\mathrm{m}}$, $3.09^{\circ}$ |  $0.24\mathop{\mathrm{m}}$, $2.26^{\circ}$ |  $0.29\mathop{\mathrm{m}}$, $3.65^{\circ}$ |  $0.21\mathop{\mathrm{m}}$, $1.95^{\circ}$ |  $0.19\mathop{\mathrm{m}}$, $2.01^{\circ}$
| Warehouse2
---
(complex)
$3.12\mathop{\mathrm{m}}$, $13.26^{\circ}$ |  $3.58\mathop{\mathrm{m}}$, $21.77^{\circ}$ |  $1.89\mathop{\mathrm{m}}$, $14.32^{\circ}$ |  $0.79\mathop{\mathrm{m}}$, $10.42^{\circ}$ |  $0.51\mathop{\mathrm{m}}$, $8.24^{\circ}$
TABLE IV: The table shows the median localization errors of SCORE forest [4], directional PoseNet [3], Bayesian PoseNet [10], the ContexturalNet [1] and the 3D geometry-aware network [9] in the 7Scenes data set. |  Score Forest (RGB-D) |  PoseNet |  Directional PoseNet |  Bayesian PoseNet |  ContexualNet |  Geometry- aware DNN [9] |  Our method (Single image) |  Our method (Multiple model)
---|---|---|---|---|---|---|---|---
Chess |  $0.03\mathop{\mathrm{m}}$, $0.66^{\circ}$ |  $0.32\mathop{\mathrm{m}}$, $8.12^{\circ}$ |  $0.24\mathop{\mathrm{m}}$, $5.77^{\circ}$ |  $0.37\mathop{\mathrm{m}}$, $7.24^{\circ}$ |  $0.15\mathop{\mathrm{m}}$, $6.12^{\circ}$ |  $\bm{0.09\mathop{\mathrm{m}}}$, $4.39^{\circ}$ |  $0.12\mathop{\mathrm{m}}$, $4.06^{\circ}$ |  $0.13\mathop{\mathrm{m}}$, $\bm{3.87^{\circ}}$
Fire |  $0.05\mathop{\mathrm{m}}$, $1.50^{\circ}$ |  $0.47\mathop{\mathrm{m}}$, $14.40^{\circ}$ |  $0.34\mathop{\mathrm{m}}$, $11.90^{\circ}$ |  $0.43\mathop{\mathrm{m}}$, $13.70^{\circ}$ |  $0.16\mathop{\mathrm{m}}$, $10.93^{\circ}$ |  $0.25\mathop{\mathrm{m}}$, $4.79^{\circ}$ |  $\bm{0.08\mathop{\mathrm{m}}}$, $4.11^{\circ}$ |  $\bm{0.08\mathop{\mathrm{m}}}$, $\bm{4.04^{\circ}}$
Heads |  $0.06\mathop{\mathrm{m}}$, $5.50^{\circ}$ |  $0.29\mathop{\mathrm{m}}$, $12.00^{\circ}$ |  $0.21\mathop{\mathrm{m}}$, $13.70^{\circ}$ |  $0.29\mathop{\mathrm{m}}$, $12.00^{\circ}$ |  $0.25\mathop{\mathrm{m}}$, $13.20^{\circ}$ |  $0.14\mathop{\mathrm{m}}$, $12.56^{\circ}$ |  $0.08\mathop{\mathrm{m}}$, $\bm{4.42^{\circ}}$ |  $\bm{0.05\mathop{\mathrm{m}}}$, $4.50^{\circ}$
Office |  $0.04\mathop{\mathrm{m}}$, $0.78^{\circ}$ |  $0.48\mathop{\mathrm{m}}$, $7,68^{\circ}$ |  $0.30\mathop{\mathrm{m}}$, $8.08^{\circ}$ |  $0.48\mathop{\mathrm{m}}$, $7.68^{\circ}$ |  $0.25\mathop{\mathrm{m}}$, $7.45^{\circ}$ |  $0.17\mathop{\mathrm{m}}$, $\bm{6.46^{\circ}}$ |  $0.23\mathop{\mathrm{m}}$, $9.64^{\circ}$ |  $\bm{0.14\mathop{\mathrm{m}}}$, $7.31^{\circ}$
Pumpkin |  $0.04\mathop{\mathrm{m}}$, $0.68^{\circ}$ |  $0.47\mathop{\mathrm{m}}$, $8.42^{\circ}$ |  $0.33\mathop{\mathrm{m}}$, $7.00^{\circ}$ |  $0.47\mathop{\mathrm{m}}$, $8.42^{\circ}$ |  $0.26\mathop{\mathrm{m}}$, $6.62^{\circ}$ |  $0.19\mathop{\mathrm{m}}$, $\bm{5.91^{\circ}}$ |  $0.25\mathop{\mathrm{m}}$, $8.79^{\circ}$ |  $\bm{0.16\mathop{\mathrm{m}}}$, $6.30^{\circ}$
| Red Kitchen
---
$0.04\mathop{\mathrm{m}}$, $0.76^{\circ}$ |  $0.59\mathop{\mathrm{m}}$, $8.43^{\circ}$ |  $0.37\mathop{\mathrm{m}}$, $8.83^{\circ}$ |  $0.59\mathop{\mathrm{m}}$, $8.64^{\circ}$ |  $0.20\mathop{\mathrm{m}}$, $6.97^{\circ}$ |  $0.21\mathop{\mathrm{m}}$, $6.47^{\circ}$ |  $\bm{0.02\mathop{\mathrm{m}}}$, $\bm{1.10^{\circ}}$ |  $\bm{0.02\mathop{\mathrm{m}}}$, $1.40^{\circ}$
Stairs |  $0.03\mathop{\mathrm{m}}$, $1.32^{\circ}$ |  $0.47\mathop{\mathrm{m}}$, $13.80^{\circ}$ |  $0.40\mathop{\mathrm{m}}$, $13.70^{\circ}$ |  $0.47\mathop{\mathrm{m}}$, $13.80^{\circ}$ |  $0.17\mathop{\mathrm{m}}$, $10.83^{\circ}$ |  $0.26\mathop{\mathrm{m}}$, $11.51^{\circ}$ |  $0.11\mathop{\mathrm{m}}$, $\bm{5.22^{\circ}}$ |  $\bm{0.08\mathop{\mathrm{m}}}$, $5.40^{\circ}$
TABLE V: The table shows the median localization errors on the TUM data set. |  PoseNet |  ContexualNet |  Our method (Single image) |  Our method (Multiple image)
---|---|---|---|---
desk2 |  $0.21\mathop{\mathrm{m}}$, $15.81^{\circ}$ |  $0.05\mathop{\mathrm{m}}$, $10.75^{\circ}$ |  $0.07\mathop{\mathrm{m}}$, $\bm{2.91^{\circ}}$ |  $\bm{0.03\mathop{\mathrm{m}}}$, $2.74^{\circ}$
floor |  $0.16\mathop{\mathrm{m}}$, $1.87^{\circ}$ |  $0.05\mathop{\mathrm{m}}$, $2.86^{\circ}$ |  $\bm{0.02\mathop{\mathrm{m}}}$, $0.96^{\circ}$ |  $\bm{0.02\mathop{\mathrm{m}}}$, $\bm{0.84^{\circ}}$
room |  $0.19\mathop{\mathrm{m}}$, $7.56^{\circ}$ |  $0.06\mathop{\mathrm{m}}$, $4.99^{\circ}$ |  $\bm{0.02\mathop{\mathrm{m}}}$, $1.44^{\circ}$ |  $0.03\mathop{\mathrm{m}}$, $\bm{1.21^{\circ}}$
360_hemisphere |  $1.00\mathop{\mathrm{m}}$, $8.53^{\circ}$ |  $\bm{0.62\mathop{\mathrm{m}}}$, $\bm{7.69^{\circ}}$ |  $1.35\mathop{\mathrm{m}}$, $8.48^{\circ}$ |  $1.14\mathop{\mathrm{m}}$, $8.41^{\circ}$
large_with_loop |  $0.58\mathop{\mathrm{m}}$, $9.11^{\circ}$ |  $0.41\mathop{\mathrm{m}}$, $5.07^{\circ}$ |  $0.38\mathop{\mathrm{m}}$, $\bm{3.55^{\circ}}$ |  $\bm{0.27\mathop{\mathrm{m}}}$, $3.62^{\circ}$
TABLE VI: The table compares our method (single image) and PoseNet on the downsampled synthetic warehouse (different ratio of downsampling) data set. Values are in median errors. |  100% |  75% |  50% |  33% |  25% |  12%
---|---|---|---|---|---|---
Ours (Simple) |  0.21m |  0.21m |  0.22m |  0.24m |  0.23m |  0.25m
Ours (Complex) |  0.79m |  0.79m |  0.83m |  0.88m |  0.97m |  0.94m
PoseNet (Simple) |  0.24m |  0.25m |  0.30m |  0.31m |  0.34m |  0.39m
PoseNet (Complex) |  3.58m |  3.54m |  4.66m |  4.32m |  4.73m |  4.91m

###  V-C Experiment setting

Following settings were applied in the training process. In each epoch, 128
positive image pairs and the same amount of negative pairs were randomly
selected. The model was trained for $500$ epochs with the learning rate of
$0.001$ and the batch size of $128$. The training was achieved by minimizing
the contrastive loss; the weights were optimized using Adaptive movement
estimation (Adam) [41]. Margin $\mathrm{m}$ in the contrastive loss function
is set as $1$. Tensorflow libraries [42] were employed for training, and the
model was trained on NVIDIA Titan X GPUs. The model was initialized using the
weights trained on the Places [43] data set.

In the keyframe selection, the searching range $\triangle k$ is set to 100.
Standard deviation $\ell=0.2$. $\alpha_{m}$ is set to 3 degrees. Seven
keyframes are selected (3 backward and 3 forward).

In the geometric locator, the threshold of the re-projection error $\eta_{r}$
was set to $5$. $t=4$ historical images were selected for the motion model.
The proposed method was compared with the PoseNet [3], ContextualNet [1],
Bayesian PoseNet [10], the geometry-aware DNN [9] on the 7Scenes data. It
should be noted that the results for PoseNet, ContextualNet, Bayesian PoseNet,
and the geometry-aware DNN [9] on the 7Scenes data set were directly cited
from the published papers. For TUM and Isaac Sim data sets, we compared the
results with the PoseNet [3] and ContextualNet [1]. The accuracy comparisons
are presented in Tables III, IV and V. Each image is processed individually
(with several neighboring predictions) and sequential processing is not
required.

We note that monocular SLAM cannot be implemented on the 3 data sets as the
initialization fails due to the predominant rotational motion. To compare with
the prior-free BoW (the backbone of the relocalization module in ORB-SLAM
series), we implemented BoW for image alignment and adopted our proposed
geometric locator for 6 DoF pose estimation, results of which are presented in
Table III. Moreover, in the first few starting frames, only single image
locator is used.

##  VI Discussion and Limitations

###  VI-A Performance of the proposed method

The results of Monte-Carlo experiment listed in Table II shows (5) is accurate
when pose labels are noise-free. The accuracy of the estimated uncertainty
deteriorates as the pose label noise increases. We could not evaluate the
probabilistic motion model using the Monte-Carlo tests as the accuracy is
dependent on the validity of the constant displacement and turning assumption.

Table III, IV and V demonstrate our proposed method significantly outperforms
the baseline methods on the synthetic, 7Scenes and TUM data sets. It can
partially be attributed to the abundant texture in all three data sets.
Adequate accurate and robust corner points contribute to the good performance
of the method on all data sets. More importantly, the inferior performance of
the image-to-pose methods is due to the different viewing angles between the
training and testing image sets. Fig. 5 reveals several sample testing images
and their visually similar correspondences in the training set. The figures
indicate that the scenarios were observed from different viewing angles. Our
proposed method substantially outperforms image-to-pose fitting methods in
these cases due to the efficiency of the BA and the dissimilarity between the
training and testing images. Learning-based methods cannot achieve similar
performance in this situation. The BA localization, however, obtains the
average residual less than 1 pixel after the optimization.

The motion model further addresses the misclassification issues of the Siamese
network, which are caused by blurry images and textureless regions. The
uncertainty estimated by the motion model, along with the geometric locator,
assists in filtering incorrect results and improving the overall results.

Further, our proposed method is more suitable to be applied in the scenario
where there are fewer training data and abundant textures. Conventional
learning-based methods heavily rely on the size of the training data. We
tested PoseNet and our method with downsampled training data set as listed in
Table VI, which shows the feasibility of our approach with smaller data set.
Moreover, as Table V indicates, the deep CNN-based model is not well trained
with limited data (540 and 1261 training samples for “desk2” and “room” data
set respectively). Take the training samples 540, e.g., the Siamese training
procedure can provide $6480$ samples at most (positive and negative pairs).
Furthermore, our method’s image similarity network task is easier to achieve
than the image-to-pose fitting.

###  VI-B Accuracy of the image descriptor network

“Single image” results in Table III, IV and V validate the efficiency of the
image descriptor network. The accuracy of the geometric locator heavily relies
on testing and training image similarity alignment. The geometric locator
fails if the overlap between the two images is sufficiently small that there
are not enough valid features to be tracked. We measure the median distance
between the testing and aligned training image in TUM data set, which are:
($0.90\mathop{\mathrm{m}}$, $39.19^{\circ}$), ($0.72\mathop{\mathrm{m}}$,
$20.41^{\circ}$), ($2.06\mathop{\mathrm{m}}$, $37.76^{\circ}$),
($1.17\mathop{\mathrm{m}}$, $58.76^{\circ}$), ($0.93\mathop{\mathrm{m}}$,
$76.01^{\circ}$) and ($2.15\mathop{\mathrm{m}}$, $58.71^{\circ}$)
respectively.

###  VI-C The benefit of introducing the geometric uncertainty

One critical drawback of the deep image-to-pose fitting is that it relies on
the domain coverage between the training and testing data. The Bayesian
network [10] estimates the uncertainty with its dropout approximated framework
and passively “provides a measure of similarity between a testing image and
the data set’s training image”. Our method actively searches for key features
to align the images when the testing image and training image are observed
from different viewing directions. Moreover, the aleatoric uncertainty from
the re-projection errors geometrically depicts the confidence of the estimated
pose well. The uncertainty addresses the quality of the testing image and
shows geometric confidence. In the experiments, images with blurriness are
assigned with high uncertainty, while images with different viewing directions
maintain high confidence. The wrongly classified images can also be identified
if the prediction deviates from the motion model prediction significantly.

With the geometric uncertainty, the fusion module enhances the robustness of
the localization. The motion model provides complimentary short-term inertial
information when the geometric method does not perform well (indicated by the
uncertainty). As demonstrated in the three experiments, the motion model
cannot improve the performance when the geometric localization works well.
However, when the geometric locator performs poorly (“Office” and “Pumpkin”
(7Scenes), “Warehouse2” (simulation), e.g.), the motion model substantially
improves the accuracy.

###  VI-D Limitations

The proposed method has two drawbacks. First, the geometric locator suffers
from illumination changes. The matched corner points are unreliable if the
training image and the aligned testing image have very different illumination.
We observed satisfactory performance in the indoor scenarios with relatively
uniform illumination and poor performance in outdoor environments because of
heavy influence by sunlight. Moreover, this approach requires the known camera
intrinsic parameter. The two drawbacks indicate our proposed method and the
image-to-pose learning-based methods [3, 1] are complementary. Although both
our proposed method and the work of Tian et al. [9] address the geometric
constraint, experiments show that our method is more efficient given the
correct camera parameter. It remains unclear which method is more suitable
when the camera intrinsic parameter is unknown, and this problem is an
interesting future work direction.

##  VII Conclusions

We developed a geometry-aware image-based localization system for indoor
environments using an image descriptor network, a geometric locator, a motion
model, and an uncertainty-based pose fusion. The proposed framework has higher
accuracy and is even well suited for scenarios where the data sets are small.
We extensively tested our method using synthetic and publicly available data
sets and provided comparisons with other learning and feature-based methods.

In the future, we plan to extend the algorithm to localize any image with
unknown intrinsic parameters. We will also explore developing a fully
automatic localization failure detection mechanism.

## Acknowledgment

Toyota Research Institute provided funds to support this work. Funding for M.
Ghaffari was in part provided by NSF Award No. 2118818. We would also like to
acknowledge FX Palo Alto Laboratory Inc. where this work initially started.

## References

  * Patel et al. [2018] M. Patel, B. Emery, and Y.-Y. Chen, “ContextualNet: Exploiting contextual information using LSTMs to improve image-based localization,” in _Proc. IEEE Int. Conf. Robot. and Automation_. IEEE, 2018, pp. 1–7.
  * Mur-Artal and Tardós [2017] R. Mur-Artal and J. D. Tardós, “ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras,” _IEEE Trans. Robot._ , vol. 33, no. 5, pp. 1255–1262, 2017.
  * Kendall et al. [2015] A. Kendall, M. Grimes, and R. Cipolla, “PoseNet: A convolutional network for real-time 6-DOF camera relocalization,” in _Proc. IEEE Int. Conf. Comput. Vis._ , 2015, pp. 2938–2946.
  * Shotton et al. [2013] J. Shotton, B. Glocker, C. Zach, S. Izadi, A. Criminisi, and A. Fitzgibbon, “Scene coordinate regression forests for camera relocalization in RGB-D images,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recog._ , 2013, pp. 2930–2937.
  * Wang et al. [2019] B. Wang, C. Chen, C. X. Lu, P. Zhao, N. Trigoni, and A. Markham, “AtLoc: Attention guided camera localization,” _arXiv preprint arXiv:1909.03557_ , 2019.
  * Weinzaepfel et al. [2019] P. Weinzaepfel, G. Csurka, Y. Cabon, and M. Humenberger, “Visual localization by learning objects-of-interest dense match regression,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recog._ , 2019, pp. 5634–5643.
  * Valada et al. [2018] A. Valada, N. Radwan, and W. Burgard, “Deep auxiliary learning for visual localization and odometry,” in _Proc. IEEE Int. Conf. Robot. and Automation_. IEEE, 2018, pp. 6939–6946.
  * Radwan et al. [2018] N. Radwan, A. Valada, and W. Burgard, “VlocNet++: Deep multitask learning for semantic visual localization and odometry,” _IEEE Robot. and Autom. Lett._ , vol. 3, no. 4, pp. 4407–4414, 2018.
  * Tian et al. [2020] M. Tian, Q. Nie, and H. Shen, “3D scene geometry-aware constraint for camera localization with deep learning,” in _Proc. IEEE Int. Conf. Robot. and Automation_. IEEE, 2020, pp. 4211–4217.
  * Kendall and Cipolla [2016] A. Kendall and R. Cipolla, “Modelling uncertainty in deep learning for camera relocalization,” in _Proc. IEEE Int. Conf. Robot. and Automation_. IEEE, 2016, pp. 4762–4769.
  * Zhao et al. [2019] C. Zhao, M. Shen, L. Sun, and G.-Z. Yang, “Generative localization with uncertainty estimation through Video-CT data for bronchoscopic biopsy,” _IEEE Robot. and Autom. Lett._ , vol. 5, no. 1, pp. 258–265, 2019.
  * Costante and Mancini [2020] G. Costante and M. Mancini, “Uncertainty estimation for data-driven visual odometry,” _IEEE Trans. Robot._ , vol. 36, no. 6, pp. 1738–1757, 2020\.
  * Yang et al. [2020] N. Yang, L. v. Stumberg, R. Wang, and D. Cremers, “D3VO: Deep depth, deep pose and deep uncertainty for monocular visual odometry,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recog._ , 2020, pp. 1281–1292.
  * Liu et al. [2018] K. Liu, K. Ok, W. Vega-Brown, and N. Roy, “Deep inference for covariance estimation: Learning Gussian noise models for state estimation,” in _Proc. IEEE Int. Conf. Robot. and Automation_. IEEE, 2018, pp. 1436–1443.
  * Hartley and Zisserman [2003] R. Hartley and A. Zisserman, _Multiple view geometry in computer vision_. Cambridge university press, 2003\.
  * Vega-Brown and Roy [2013] W. Vega-Brown and N. Roy, “CELLO-EM: Adaptive sensor models without ground truth,” in _Proc. IEEE/RSJ Int. Conf. Intell. Robots and Syst._ IEEE, 2013, pp. 1907–1914.
  * Ozog and Eustice [2013] P. Ozog and R. M. Eustice, “On the importance of modeling camera calibration uncertainty in visual SLAM,” in _Proc. IEEE Int. Conf. Robot. and Automation_. IEEE, 2013, pp. 3777–3784.
  * Polic and Pajdla [2017] M. Polic and T. Pajdla, “Camera uncertainty computation in large 3D reconstruction,” in _Int. Conf. on 3D Vis._ IEEE, 2017, pp. 282–290.
  * Briskin et al. [2017] G. Briskin, A. Geva, E. Rivlin, and H. Rotstein, “Estimating pose and motion using bundle adjustment and digital elevation model constraints,” _IEEE Trans. Aerosp. Electron. Syst._ , vol. 53, no. 4, pp. 1614–1624, 2017\.
  * Gálvez-López and Tardos [2012] D. Gálvez-López and J. D. Tardos, “Bags of binary words for fast place recognition in image sequences,” _IEEE Trans. Robot._ , vol. 28, no. 5, pp. 1188–1197, 2012.
  * Cummins and Newman [2008] M. Cummins and P. Newman, “FAB-MAP: Probabilistic localization and mapping in the space of appearance,” _Int. J. Robot. Res._ , vol. 27, no. 6, pp. 647–665, 2008.
  * Sünderhauf et al. [2015] N. Sünderhauf, S. Shirazi, F. Dayoub, B. Upcroft, and M. Milford, “On the performance of convnet features for place recognition,” in _Proc. IEEE/RSJ Int. Conf. Intell. Robots and Syst._ IEEE, 2015, pp. 4297–4304.
  * Kim et al. [2017] H. J. Kim, E. Dunn, and J.-M. Frahm, “Learned contextual feature reweighting for image geo-localization,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recog._ IEEE, 2017, pp. 3251–3260.
  * Brahmbhatt et al. [2018] S. Brahmbhatt, J. Gu, K. Kim, J. Hays, and J. Kautz, “Geometry-aware learning of maps for camera localization,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recog._ , 2018, pp. 2616–2625.
  * Zagoruyko and Komodakis [2015] S. Zagoruyko and N. Komodakis, “Learning to compare image patches via convolutional neural networks,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recog._ , 2015, pp. 4353–4361.
  * Gálvez-López and Tardós [2012] D. Gálvez-López and J. D. Tardós, “Bags of binary words for fast place recognition in image sequences,” _IEEE Trans. Robot._ , vol. 28, no. 5, pp. 1188–1197, October 2012.
  * Schmitt and Schuller [2017] M. Schmitt and B. Schuller, “OpenXBOW: Introducing the passau open-source crossmodal bag-of-words toolkit,” _J. Mach. Learning Res._ , vol. 18, no. 1, pp. 3370–3374, 2017.
  * Wang et al. [2014] J. Wang, Y. Song, T. Leung, C. Rosenberg, J. Wang, J. Philbin, B. Chen, and Y. Wu, “Learning fine-grained image similarity with deep ranking,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recog._ , 2014, pp. 1386–1393.
  * Szegedy et al. [2015] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recog._ , 2015, pp. 1–9.
  * Hadsell et al. [2006] R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by learning an invariant mapping,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recog._ , vol. 2. IEEE, 2006, pp. 1735–1742.
  * Ondrúška et al. [2015] P. Ondrúška, P. Kohli, and S. Izadi, “Mobilefusion: Real-time volumetric surface reconstruction and dense tracking on mobile phones,” _IEEE Trans. Vis. Comput. Graphics_ , vol. 21, no. 11, pp. 1251–1258, 2015\.
  * Huber [1992] P. J. Huber, “Robust estimation of a location parameter,” in _Breakthroughs in statistics_. Springer, 1992, pp. 492–518.
  * Kümmerle et al. [2011] R. Kümmerle, G. Grisetti, H. Strasdat, K. Konolige, and W. Burgard, “G 2 o: A general framework for graph optimization,” in _Proc. IEEE Int. Conf. Robot. and Automation_. IEEE, 2011, pp. 3607–3613.
  * Barfoot [2017] T. D. Barfoot, _State estimation for robotics_. Cambridge University Press, 2017.
  * Thrun [2002] S. Thrun, “Probabilistic robotics,” _Communications of the ACM_ , vol. 45, no. 3, pp. 52–57, 2002.
  * Shi et al. [2002] K. Shi, T. Chan, Y. Wong, and S. Ho, “Speed estimation of an induction motor drive using an optimized extended Kalman filter,” _IEEE Trans. on Indust. Electron._ , vol. 49, no. 1, pp. 124–133, 2002.
  * Barfoot and Furgale [2014] T. D. Barfoot and P. T. Furgale, “Associating uncertainty with three-dimensional poses for use in estimation problems,” _IEEE Trans. Robot._ , vol. 30, no. 3, pp. 679–693, 2014.
  * Andrews [2021] G. Andrews, “NVIDIA isaac sim on omniverse now available in open beta,” Jun 2021\. [Online]. Available: https://developer.nvidia.com/blog/nvidia-isaac-sim-on-omniverse-now-available-in-open-beta/
  * Sturm et al. [2012] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “A benchmark for the evaluation of RGB-D SLAM systems,” in _Proc. IEEE/RSJ Int. Conf. Intell. Robots and Syst._ , Oct. 2012.
  * Newcombe et al. [2011] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohi, J. Shotton, S. Hodges, and A. Fitzgibbon, “KinectFusion: Real-time dense surface mapping and tracking,” in _IEEE Internat. Sympo. on Mixed and Augment. Reality_. IEEE, 2011, pp. 127–136.
  * Kingma and Ba [2014] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _Intern. Conf. for Learning Represent._ , 2014.
  * Abadi et al. [2016] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin _et al._ , “Tensorflow: Large-scale machine learning on heterogeneous distributed systems,” _arXiv preprint arXiv:1603.04467_ , 2016.
  * Zhou et al. [2014] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva, “Learning deep features for scene recognition using places database,” in _Proc. Advances Neural Inform. Process. Syst. Conf._ , 2014, pp. 487–495.