# A Survey on Deep Learning for Localization and Mapping:
Towards the Age of Spatial Machine Intelligence

Changhao Chen, Bing Wang, Chris Xiaoxuan Lu, Niki Trigoni and Andrew Markham
Department of Computer Science, University of Oxford
Corresponding author: Changhao Chen<EMAIL_ADDRESS>A project
website that updates additional material and extended lists of references, can
be found at https://github.com/changhao-chen/deep-learning-localization-
mapping.

###### Abstract

Deep learning based localization and mapping has recently attracted
significant attention. Instead of creating hand-designed algorithms through
exploitation of physical models or geometric theories, deep learning based
solutions provide an alternative to solve the problem in a data-driven way.
Benefiting from ever-increasing volumes of data and computational power, these
methods are fast evolving into a new area that offers accurate and robust
systems to track motion and estimate scenes and their structure for real-world
applications. In this work, we provide a comprehensive survey, and propose a
new taxonomy for localization and mapping using deep learning. We also discuss
the limitations of current models, and indicate possible future directions. A
wide range of topics are covered, from learning odometry estimation, mapping,
to global localization and simultaneous localization and mapping (SLAM). We
revisit the problem of perceiving self-motion and scene understanding with on-
board sensors, and show how to solve it by integrating these modules into a
prospective spatial machine intelligence system (SMIS). It is our hope that
this work can connect emerging works from robotics, computer vision and
machine learning communities, and serve as a guide for future researchers to
apply deep learning to tackle localization and mapping problems.

###### Index Terms:

Deep Learning, Localization, Mapping, SLAM, Perception, Correspondence
Matching, Uncertainty Estimation

##  1 Introduction

Localization and mapping is a fundamental need for human and mobile agents. As
a motivating example, humans are able to perceive their self-motion and
environment via multimodal sensory perception, and rely on this awareness to
locate and navigate themselves in a complex three-dimensional space [1]. This
ability is part of human spatial ability. Furthermore, the ability to perceive
self-motion and their surroundings plays a vital role in developing cognition,
and motor control [2]. In a similar vein, artificial agents or robots should
also be able to perceive the environment and estimate their system states
using on-board sensors. These agents could be any form of robot, e.g. self-
driving vehicles, delivery drones or home service robots, sensing their
surroundings and autonomously making decisions [3]. Equivalently, as emerging
Augmented Reality (AR) and Virtual Reality (VR) technologies interweave cyber
space and physical environments, the ability of machines to be perceptually
aware underpins seamless human-machine interaction. Further applications also
include mobile and wearable devices, such as smartphones, wristbands or
Internet-of-Things (IoT) devices, providing users with a wide range of
location-based-services, ranging from pedestrian navigation [4], to
sports/activity monitoring [5], to animal tracking [6], or emergency response
[7] for first-responders.

Figure 1: A spatial machine intelligence system exploits on-board sensors to
perceive self-motion, global pose, scene geometry and semantics. (a)
Conventional model based solutions build hand-designed algorithms to convert
input sensor data to target values. (c) Data-driven solutions exploit learning
models to construct this mapping function. (b) Hybrid approaches combine both
hand-crafted algorithms and learning models. This survey discusses (b) and
(c).  Figure 2: A taxonomy of existing works on deep learning for localization
and mapping.

Enabling a high level of autonomy for these and other digital agents requires
precise and robust localization, and incrementally building and maintaining a
world model, with the capability to continuously process new information and
adapt to various scenarios. Such a quest is termed as ‘Spatial Machine
Intelligence System (SMIS)’ in our work or recently as Spatial AI in [8]. In
this work, broadly, localization refers to the ability to obtain the internal
system states of robot motion, including locations, orientations and
velocities, whilst mapping indicates the capacity to perceive external
environmental states and capture the surroundings, including the geometry,
appearance and semantics of a 2D or 3D scene. These components can act
individually to sense the internal or external states respectively, or jointly
as in simultaneous localization and mapping (SLAM) to track pose and build a
consistent environmental model in a global frame.

###  1.1 Why to Study Deep Learning for Localization and Mapping

The problems of localization and mapping have been studied for decades, with a
variety of intricate hand-designed models and algorithms being developed, for
example, odometry estimation (including visual odometry [9, 10, 11], visual-
inertial odometry [12, 13, 14, 15] and LIDAR odometry [16]), image-based
localization[17, 18], place recognition[19], SLAM[20, 10, 21], and structure
from motion (SfM)[22, 23]. Under ideal conditions, these sensors and models
are capable of accurately estimating system states without time bound and
across different environments. However, in reality, imperfect sensor
measurements, inaccurate system modelling, complex environmental dynamics and
unrealistic constraints impact both the accuracy and reliability of hand-
crated systems.

The limitations of model based solutions, together with recent advances in
machine learning, especially deep learning, have motivated researchers to
consider data-driven (learning) methods as an alternative to solve problem.
Figure 1 summarizes the relation between input sensor data (e.g. visual,
inertial, LIDAR data or other sensors) and output target values (e.g.
location, orientation, scene geometry or semantics) as a mapping function.
Conventional model-based solutions are achieved by hand-designing algorithms
and calibrating to a particular application domain, while the learning based
approaches construct this mapping function by learned knowledge. The
advantages of learning based methods are three-fold:

First of all, learning methods can leverage highly expressive deep neural
network as an universal approximator, and automatically discover features
relevant to task. This property enables learned models to be resilience to
circumstances, such as featureless areas, dynamic lightning conditions, motion
blur, accurate camera calibration, which are challenging to model by hand [3].
As a representative example, visual odometry has achieved notable improvements
in terms of robustness by incorporating data-driven methods in its design [24,
25], outperforming the state-of-the-art conventional algorithms. Moreover,
learning approaches are able to connect abstract elements with human
understandable terms[26, 27], such as semantics labelling in SLAM, which is
hard to describe in a formal mathematical way.

Figure 3: High-level conceptual illustration of a spatial machine intelligence
system (i.e. deep learning based localization and mapping). Rounded rectangles
represent a function module, while arrow lines connect these modules for data
input and output. It is not necessary to include all modules to perform this
system.

Secondly, learning methods allow spatial machine intelligence systems to learn
from past experience, and actively exploit new information. By building a
generic data-driven model, it avoids human effort on specifying the full
knowledge about mathematical and physical rules[28], to solve domain specific
problem, before being deployed. This ability potentially enables learning
machines to automatically discover new computational solutions, further
develop themselves and improve their models, within new scenarios or
confronting new circumstances. A good example is that by using novel view
synthesis as a self-supervision signal, self-motion and depth can be recovered
from unlabelled videos[29, 30]. In addition, the learned representations can
further support high-level tasks, such as path planning[31], and decision
making[32], by constructing task-driven maps.

The third benefit is its capability of fully exploiting the increasing amount
of sensor data and computational power. Deep learning or deep neural network
has the capacity to scale to large-scale problems. The huge amount of
parameters inside a DNN framework are automatically optimized by minimizing a
loss function, by training on large datasets through backpropagation and
gradient-descent algorithms. For example, the recent released GPT-3[33], the
largest pretrained language model, with incredibly over 175 Billion
parameters, achieves the state-of-the-art results on a variety of natural
language processing (NLP) tasks, even without fine-tuning. In addition, a
variety of large-scale datasets relevant to localization and mapping have been
released, for example, in the autonomous vehicles scenarios, [34, 35, 36] are
with a collection of rich combinations of sensor data, and motion and semantic
labels. This gives us an imagination that it would be possible to exploit the
power of data and computation in solving localization and mapping.

However, it must also be pointed out that these learning techniques are
reliant on massive datasets to extract statistically meaningful patterns and
can struggle to generalize to out-of-set environments. There is lack of model
interpretability. Additionally, although highly parallelizable, they are also
typically more computationally costly than simpler models. Details of
limitations are discussed in Section 7.

###  1.2 Comparison with Other Surveys

There are several survey papers that have extensively discussed model-based
localization and mapping approaches. The development of SLAM problem in early
decades has been well summarized in [37, 38]. The seminal survey [39] provides
a thorough discussion on existing SLAM work, reviews the history of
development and charts several future directions. Although this paper contains
a section which briefly discusses deep learning models, it does not overview
this field comprehensively, especially due to the explosion of research in
this area of the past five years. Other SLAM survey papers only focus on
individual flavours of SLAM systems, including the probabilistic formulation
of SLAM [40], visual odometry [41], pose-graph SLAM [42], and SLAM in dynamic
environments [43]. We refer readers to these surveys for a better
understanding of the conventional model based solutions. On the other hand,
[3] has a discussion on the applications of deep learning to robotics
research; however, its main focus is not on localization and mapping
specifically, but a more general perspective towards the potentials and limits
of deep learning in a broad context of robotics, including policy learning,
reasoning and planning.

Notably, although the problem of localization and mapping falls into the key
notion of robotics, the incorporation of learning methods progresses in tandem
with other research areas such as machine learning, computer vision and even
natural language processing. This cross-disciplinary area thus imposes non-
trivial difficulty when comprehensively summarizing related works into a
survey paper. To the best of our knowledge, this is the first survey article
that thoroughly and extensively covers existing work on deep learning for
localization and mapping.

###  1.3 Survey Organization

The remainder of the paper is organized as follows: Section 2 offers an
overview and presents a taxonomy of existing deep learning based localization
and mapping; Sections 3, 4, 5, 6 discuss the existing deep learning works on
relative motion (odometry) estimation, mapping methods for geometric, semantic
and general, global localization, and simultaneous localization and mapping
with a focus on SLAM back-ends respectively; Open questions are summarized in
Section 7 to discuss the limitations and future prospects of existing work;
and finally Section 8 concludes the paper.

##  2 Taxonomy of Existing Approaches

We provide a new taxonomy of existing deep learning approaches, relevant to
localization and mapping, to connect the fields of robotics, computer vision
and machine learning. Broadly, they can be categorized into odometry
estimation, mapping, global localization and SLAM, as illustrated by the
taxonomy shown in Figure 2:

1) Odometry estimation concerns the calculation of the relative change in
pose, in terms of translation and rotation, between two or more frames of
sensor data. It continuously tracks self-motion, and is followed by a process
to integrate these pose changes with respect to an initial state to derive
global pose, in terms of position and orientation. This is widely known as the
so-called dead reckoning solution. Odometry estimation can be used in
providing pose information and as odometry motion model to assist the feedback
loop of robot control. The key problem is to accurately estimate motion
transformations from various sensor measurements. To this end, deep learning
is applied to model the motion dynamics in an end-to-end fashion or extract
useful features to support a pre-built system in a hybrid way.

2) Mapping builds and reconstructs a consistent model to describe the
surrounding environment. Mapping can be used to provide environment
information for human operators and high-level robot tasks, constrain the
error drifts of odometry estimation, and retrieve the inquiry observation for
global localization [39]. Deep learning is leveraged as an useful tool to
discover scene geometry and semantics from high-dimensional raw data for
mapping. Deep learning based mapping methods are sub-divided into geometric,
semantic, and general mapping, depending on whether the neural network learns
the explicit geometry or semantics of a scene, or encodes the scene into an
implicit neural representation respectively.

3) Global localization retrieves the global pose of mobile agents in a known
scene with prior knowledge. This is achieved by matching the inquiry input
data with a pre-built 2D or 3D map, other spatial references, or a scene that
has been visited before. It can be leveraged to reduce the pose drift of a
dead reckoning system or solve the ’kidnapped robot’ problem[40]. Deep
learning is used to tackle the tricky data association problem that is
complicated by the changes in views, illumination, weather and scene dynamics,
between the inquiry data and map.

4) Simultaneous Localisation and Mapping (SLAM) integrates the aforementioned
odometry estimation, global localization and mapping processes as front-ends,
and jointly optimizes these modules to boost performance in both localization
and mapping. Except these abovementioned modules, several other SLAM modules
perform to ensure the consistency of the entire system as follows: _local
optimization_ ensures the local consistency of camera motion and scene
geometry; _global optimization_ aims to constrain the drift of global
trajectories, and in a global scale; _keyframe detection_ is used in keyframe-
based SLAM to enable more efficient inference, while system error drifts can
be mitigated by global optimization, once a loop closure is detected by _loop-
closure detection_ ; _uncertainty estimation_ provides a metric of belief in
the learned poses and mapping, critical to probabilistic sensor fusion and
back-end optimization in SLAM systems.

Figure 4: The typical structure of supervised learning of visual odometry,
i.e. DeepVO [24] and unsupervised learning of visual odometry, i.e. SfmLearner
[29].

Despite the different design goals of individual components, the above
components can be integrated into a spatial machine intelligence system (SMIS)
to solve real-world challenges, allowing for robust operation, and long-term
autonomy in the wild. A conceptual figure of such an integrated deep-learning
based localization and mapping system is indicated in Figure 3, showing the
relationship of these components. In the following sections, we will discuss
these components in details.

##  3 Odometry Estimation

We begin with odometry estimation, which continuously tracks camera egomotion
and produces relative poses. Global trajectories are reconstructed by
integrating these relative poses, given an initial state, and thus it is
critical to keep motion transformation estimates accurate enough to ensure
high-prevision localization in a global scale. This section discusses deep
learning approaches to achieve odometry estimation from various sensor data,
that are fundamentally different in their data properties and application
scenarios. The discussion mainly focuses on odometry estimation from visual,
inertial and point-cloud data, as they are the common choices of sensing
modalities on mobile agents.

###  3.1 Visual Odometry

Visual odometry (VO) estimates the ego-motion of a camera, and integrates the
relative motion between images into global poses. Deep learning methods are
capable of extracting high-level feature representations from images, and
thereby provide an alternative to solve the VO problem, without requiring
hand-crafted feature extractors. Existing deep learning based VO models can be
categorized into _end-to-end VO_ and _hybrid VO_ , depending on whether they
are purely neural-network based or whether they are a combination of classical
VO algorithms and deep neural networks. Depending on the availability of
ground-truth labels in the training phase, end-to-end VO systems can be
further classified into _supervised_ VO and _unsupervised_ VO.

####  3.1.1 Supervised Learning of VO

We start with the introduction of supervised VO, one of the most predominant
approaches to learning-based odometry, by training a deep neural network model
on labelled datasets to construct a mapping function from consecutive images
to motion transformations directly, instead of exploiting the geometric
structures of images as in conventional VO systems[41]. At its most basic, the
input of deep neural network is a pair of consecutive images, and the output
is the estimated translation and rotation between two frames of images.

One of the first works in this area was Konda et al. [44]. This approach
formulates visual odometry as a classification problem, and predicts the
discrete changes of direction and velocity from input images using a
convolutional neural network (ConvNet). Costante et al. [45] used a ConvNet to
extract visual features from dense optical flow, and based on these visual
features to output frame-to-frame motion estimation. Nonetheless, these two
works have not achieved end-to-end learning from images to motion estimates,
and their performance is still limited.

DeepVO [24] utilizes a combination of convolutional neural network (ConvNet)
and recurrent neural network (RNN) to enable end-to-end learning of visual
odometry. The framework of DeepVO becomes a typical choice in realizing
supervised learning of VO, due to its specialization in end-to-end learning.
Figure 4 (a) shows the architecture of this RNN+ConvNet based VO system, which
extracts visual features from pairs of images via a ConvNet, and passes
features through RNNs to model the temporal correlation of features. Its
ConvNet encoder is based on a FlowNet structure to extract visual features
suitable for optical flow and self-motion estimation. Using a FlowNet based
encoder can be regarded as introducing the prior knowledge of optical flow
into the learning process, and potentially prevents DeepVO from being
overfitted to the training datasets. The reccurent model summarizes the
history information into its hidden states, so that the output is inferred
from both past experience and current ConvNet features from sensor
observation. It is trained on large-scale datasets with groundtruthed camera
poses as labels. To recover the optimal parameters $\bm{\theta}^{*}$ of
framework, the optimization target is to minimize the Mean Square Error (MSE)
of the estimated translations $\mathbf{\hat{p}}\in\mathbb{R}^{3}$ and euler
angle based rotations $\hat{\bm{\varphi}}\in\mathbb{R}^{3}$:

$\bm{\theta}^{*}=\operatorname*{arg\,min}_{\bm{\theta}}\frac{1}{N}\displaystyle\sum_{i=1}^{N}\displaystyle\sum_{t=1}^{T}\|\hat{\mathbf{p}}_{t}-\mathbf{p}_{t}\|_{2}^{2}+\|\hat{\bm{\varphi}}_{t}-\bm{\varphi}_{t}\|_{2}^{2},$
(1)

where $(\hat{\mathbf{p}}_{t},\hat{\bm{\varphi}}_{t})$ are the estimates of
relative pose at the timestep $t$, $(\mathbf{p},\bm{\varphi})$ are the
corresponding groundtruth values, $\bm{\theta}$ are the parameters of the DNN
framework, $N$ is the number of samples.

DeepVO reports impressive results on estimating the pose of driving vehicles,
even in previously unseen scenarios. In the experiment on the KITTI odometry
dataset[46], this data-driven solution outperforms conventional representative
monocular VO, e.g. VISO2[47] and ORB-SLAM (without loop closure) [21]. Another
advantage is that supervised VO naturally produces trajectory with the
absolute scale from monocular camera, while classical VO algorithm is scale-
ambiguous using only monocular information. This is because deep neural
network can implicitly learn and maintain the global scale from large
collection of images, which can be viewed as learning from past experience to
predict current scale metric.

Based on this typical model of supervised VO, a number of works further
extended this approach to improve the model performance. To improve the
generalization ability of supervised VO, [48] incorporates curriculum learning
(i.e. the model is trained by increasing the data complexity) and geometric
loss constraints. Knowledge distillation (i.e. a large model is compressed by
teaching a smaller one) is applied into the supervised VO framework to greatly
reduce the number of network parameters, making it more amenable for real-time
operation on mobile devices [49]. Furthermore, Xue et al. [50] introduced a
memory module that stores global information, and a refining module that
improves pose estimates with the preserved contextual information.

In summary, these end-to-end learning methods benefit from recent advances in
machine learning techniques and computational power, to automatically learn
pose transformations directly from raw images that can tackle challenging
real-world odometry estimation.

####  3.1.2 Unsupervised Learning of VO

There is growing interest in exploring unsupervised learning of VO.
Unsupervised solutions are capable of exploiting unlabelled sensor data, and
thus it saves human effort on labelling data, and has better adaptation and
generalization ability in new scenarios, where no labelled data are available.
This has been achieved in a self-supervised framework that jointly learns
depth and camera ego-motion from video sequences, by utilizing view synthesis
as a supervisory signal[29].

As shown in Figure 4 (b), a typical unsupervised VO solution consists of a
depth network to predict depth maps, and a pose network to produce motion
transformations between images. The entire framework takes consecutive images
as input, and the supervision signal is based on novel view synthesis - given
a source image $\mathbf{I}_{s}$, the view synthesis task is to generate a
synthetic target image $\mathbf{I}_{t}$. A pixel of source image
$\mathbf{I}_{s}(p_{s})$ is projected onto a target view
$\mathbf{I}_{t}(p_{t})$ via:

$p_{s}\sim\mathbf{K}\mathbf{T}_{t\to
s}\mathbf{D}_{t}(p_{t})\mathbf{K}^{-1}p_{t}$ (2)

where $\mathbf{K}$ is the camera’s intrinsic matrix, $\mathbf{T}_{t\to s}$
denotes the camera motion matrix from target frame to source frame, and
$\mathbf{D}_{t}(p_{t})$ denotes the per-pixel depth maps in the target frame.
The training objective is to ensure the consistency of the scene geometry by
optimizing the photometric reconstruction loss between the real target image
and the synthetic one:

$\mathcal{L}_{\text{photo}}=\sum_{<\mathbf{I}_{1},...,\mathbf{I}_{N}>\in
S}\sum_{p}|\mathbf{I}_{t}(p)-\hat{\mathbf{I}}_{s}(p)|,$ (3)

where p denotes pixel coordinates, $\mathbf{I}_{t}$ is the target image, and
$\hat{\mathbf{I}}_{s}$ is the synthetic target image generated from the source
image $\mathbf{I}_{s}$.

TABLE I: A summary of existing methods on deep learning for odometry
estimation.

Model | Sensor | Supervision | Scale | Performance | Contributions
---|---|---|---|---|---
Seq09 | Seq10
VO |  Konda et al.[44] | MC | Supervised | Yes | - | - | formulate VO as a classification problem
Costante et al.[45] | MC | Supervised | Yes | 6.75 | 21.23 | extract features from optical flow for VO estimates
Backprop KF[51] | MC | Hybrid | Yes | - | - | a differentiable Kalman filter based VO
DeepVO[24] | MC | Supervised | Yes | - | 8.11 | combine RNN and ConvNet for end-to-end learning
SfmLearner[29] | MC | Unsupervised | No | 17.84 | 37.91 | novel view synthesis for self-supervised learning
Yin et al.[52] | MC | Hybrid | Yes | 4.14 | 1.70 | introduce learned depth to recover scale metric
UnDeepVO[53] | SC | Unsupervised | Yes | 7.01 | 10.63 | use fixed stereo line to recover scale metric
Barnes et al.[54] | MC | Hybrid | Yes | - | - | integrate learned depth and ephemeral masks
GeoNet[55] | MC | Unsupervised | No | 43.76 | 35.6 | geometric consistency loss and 2D flow generator
Zhan et al.[56] | SC | Unsupervised | No | 11.92 | 12.45 | use fixed stereo line for scale recovery
DPF[57] | MC | Hybrid | Yes | - | - | a differentiable particle filter based VO
Yang et al.[58] | MC | Hybrid | Yes | 0.83 | 0.74 | use learned depth into classical VO
Zhao et al.[59] | MC | Supervised | Yes | - | 4.38 | generate dense 3D flow for VO and mapping
Struct2Depth[60] | MC | Unsupervised | No | 10.2 | 28.9 | introduce 3D geometry structure during learning
Saputra et al.[48] | MC | Supervised | Yes | - | 8.29 | curriculum learning and geometric loss constraints
GANVO[61] | MC | Unsupervised | No | - | - | adversarial learning to generate depth
CNN-SVO[62] | MC | Hybrid | Yes | 10.69 | 4.84 | use learned depth to initialize SVO
Xue et al.[50] | MC | Supervised | Yes | - | 3.47 | memory and refinement module
Wang et al.[63] | MC | Unsupervised | Yes | 9.30 | 7.21 | integrate RNN and flow consistency constraint
Li et al.[64] | MC | Unsupervised | No | - | - | global optimization for pose graph
Saputra et al.[49] | MC | Supervised | Yes | - | - | knowledge distilling to compress deep VO model
Gordon[65] | MC | Unsupervised | No | 2.7 | 6.8 | camera matrix learning
Koumis et al.[66] | MC | Supervised | Yes | - | - | 3D convolutional networks
Bian et al.[30] | MC | Unsupervised | Yes | 11.2 | 10.1 | scale recovery from only monocular images
Zhan et al.[67] | MC | Hybrid | Yes | 2.61 | 2.29 | integrate learned optical flow and depth
D3VO[25] | MC | Hybrid | Yes | 0.78 | 0.62 | integrate learned depth, uncertainty and pose
VIO |  VINet[68] | MC+I | Supervised | Yes | - | - | formulate VIO as a sequential learning problem
VIOLearner[69] | MC+I | Unsupervised | Yes | 1.51 | 2.04 | online correction module
Chen et al.[70] | MC+I | Supervised | Yes | - | - | feature selection for deep sensor fusion
DeepVIO[71] | SC+I | Unsupervised | Yes | 0.85 | 1.03 | learn VIO from stereo images and IMU
LO |  Velas et al.[72] | L | Supervised | Yes | 4.94 | 3.27 | ConvNet to estimate odometry from point clouds
LO-Net[73] | L | Supervised | Yes | 1.37 | 1.80 | geometric constraint loss
DeepPCO[74] | L | Supervised | Yes | - | - | parallel neural network
Valente et al.[75] | MC+L | Supervised | Yes | - | 7.60 | sensor fusion for LIDAR and camara

  * •

Model: VO, VIO and LO represent visual odometry, visual-inertial odometry and
LIDAR odometry respectively.

  * •

Sensor: MC, SC, I and L represent monocular camera, stereo camera, inertial
measurement unit, and LIDAR respectively.

  * •

Supervision represents whether this work is a purely neural network based
model trained with groundtruth labels (Supervised) or without labels
(Unsupervised), or it is a combination of classical and deep neural network
(Hybrid)

  * •

Scale indicates whether a trajectory with a global scale can be produced.

  * •

Performance reports the localization error (a small number is better), i.e.
the averaged translational RMSE drift (%) on lengths of 100m-800m on the KITTI
odometry dataset[46]. Most works were evaluated on the Sequence 09 and 10, and
thus we took the results on these two sequences from their original papers for
a performance comparison. Note that the training sets may be different in each
work.

  * •

Contributions summarize the main contributions of each work compared with
previous research.

However, there are basically two main problems that remained unsolved in the
original work[29]: 1) this monocular image based approach is not able to
provide pose estimates in a consistent global scale. Due to the scale
ambiguity, no physically meaningful global trajectory can be reconstructed,
limiting its real use. 2) The photometric loss assumes that the scene is
static and without camera occlusions. Although the authors proposed the use of
an explainability mask to remove scene dynamics, the influence of these
environmental factors is still not addressed completely, which violates the
assumption. To address these concerns, an increasing number of works [53, 55,
56, 58, 59, 61, 64, 76, 77] extended this unsupervised framework to achieve
better performance.

To solve the global scale problem, [53, 56] proposed to utilize stereo image
pairs to recover the absolute scale of pose estimation. They introduced an
additional spatial photometric loss between the left and right pairs of
images, as the stereo baseline (i.e. motion transformation between the left
and right images) is fixed and known throughout the dataset. Once the training
is complete, the network produces pose predictions using only monocular
images. Thus, although it is unsupervised in the context of not having access
to ground-truth, the training dataset (stereo) is different to the test set
(mono). [30] tackles the scale issue by introducing a geometric consistency
loss, that enforces the consistency between predicted depth maps and
reconstructed depth maps. The framework transforms the predicted depth maps
into a 3D space, and projects them back to produce reconstructed depth maps.
In doing so, the depth predictions are able to remain scale-consistent over
consecutive frames, enabling pose estimates to be scale-consistent meanwhile.

The photometric consistency constraint assumes that the entire scenario
consists only of rigid static structures, e.g. buildings and lanes. However,
in real-world applications, environmental dynamics (e.g. pedestrians and
vehicles), will distort the photometric projection and degrade the accuracy of
pose estimation. To address this concern, GeoNet [55] divides its learning
process into two sub-tasks by estimating static scene structures and motion
dynamics separately through a rigid structure reconstructor and a non-rigid
motion localizer. In addition, GeoNet enforces a geometric consistency loss to
mitigate the issues caused by camera occlusions and non-Lambertian surfaces.
[59] adds a 2D flow generator along with a depth network to generate 3D flow.
Benefiting from better 3D understanding of environment, their framework is
able to produce more accurate camera pose, along with a point cloud map. GANVO
[61] employs a generative adversarial learning paradigm for depth generation,
and introduces a temporal recurrent module for pose regression. Li et al. [76]
also utilized a generative adversarial network (GAN) to generate more
realistic depth maps and poses, and further encourage more accurate synthetic
images in the target frame. Instead of a hand-crafted metric, a discriminator
is adopted to evaluate the quality of synthetic images generation. In doing
so, the generative adversarial setup facilitates the generated depth maps to
be more texture-rich and crisper. In this way, high-level scene perception and
representation are accurately captured and environmental dynamics are
implicitly tolerated.

Although unsupervised VO still cannot compete with supervised VO in
performance, as illustrated in Figure 5, its concerns of scale metric and
scene dynamics problem have been largely resolved. With the benefits of self-
supervised learning, and ever-increasing improvement on performance,
unsupervised VO would be a promising solution in providing pose information,
and tightly coupled with other modules in spatial machine intelligence system.

####  3.1.3 Hybrid VO

Unlike end-to-end VO that only relies on a deep neural network to interpret
pose from data, hybrid VO integrates classical geometric models with deep
learning framework. Based on mature geometric theory, they use a deep neural
network to expressively replace parts of a geometry model.

A straightforward way is to incorporate the learned depth estimates into a
conventional visual odometry algorithm to recover the absolute scale metric of
poses [52]. Learning depth estimation is a well-researched area in the
computer vision community. For example, [78, 79, 80, 81] provide per-pixel
depths in a global scale by employing a trained deep neural model. Thus the
so-called scale problem of conventional VO is mitigated. Barnes et al. [54]
utilize both the predicted depth maps and ephemeral masks (i.e. the area of
moving objects) into a VO system to improve its robustness to moving objects.
Zhan et al. [67] integrate the learned depth and optical flow predictions into
a conventional visual odometry model, achieving competitive performance over
other baselines. Other works combine physical motion models with deep neural
network e.g. via a differentiable Kalman filter [82], and a particle filter
[83]. The physical model serves as an algorithmic prior in the learning
process. Furthermore, D3VO [25] incorporates the deep predictions of depth,
pose, and uncertainty into a direct visual odometry.

Combining the benefits from both geometric theory and deep learning, hybrid
models are normally more accurate than end-to-end VO at this stage, as shown
in Table 1. It is notable that hybrid models even outperform the state-of-the-
art conventional monocular VO or visual-inertial odometry (VIO) systems on
common benchmarks, for example, D3VO[25] defeats several popular conventional
VO/VIO systems, such as DSO[84], ORB-SLAM[21], VINS-Mono[15]. This
demonstrates the rapid rate of progress in this area.

###  3.2 Visual-Inertial Odometry

Integrating visual and inertial data as visual-inertial odometry (VIO) is a
well-defined problem in mobile robotics. Both cameras and inertial sensors are
relatively low-cost, power-efficient and widely deployed. These two sensors
are complementary: monocular cameras capture the appearance and structure of a
3D scene, while they are scale-ambiguous, and not robust to challenging
scenarios, e.g. strong lighting changes, lack of texture and high-speed
motion; In contrast, IMUs are completely ego-centric, scene-independent, and
can also provide absolute metric scale. Nevertheless, the downside is that
inertial measurements, especially from low-cost devices, are plagued by
process noise and biases. An effective fusion of the measurements from these
two complementary sensors is of key importance to accurate pose estimation.
Thus, according to their information fusion methods, conventional model based
visual-inertial approaches are roughly segmented into three different classes
: filtering approaches [12], fixed-lag smoothers [13] and full smoothing
methods [14].

Data-driven approaches have emerged to consider learning 6-DoF poses directly
from visual and inertial measurements without human intervention or
calibration. VINet [68] is the first work that formulated visual-inertial
odometry as a sequential learning problem, and proposed a deep neural network
framework to achieve VIO in an end-to-end manner. VINet uses a ConvNet based
visual encoder to extract visual features from two consecutive RGB images, and
an inertial encoder to extract inertial features from a sequence of IMU data
with a long short-term memory (LSTM) network. Here, the LSTM aims to model the
temporal state evolution of inertial data. The visual and inertial features
are concatenated together, and taken as the input into a further LSTM module
to predict relative poses, conditioned on the history of system states. This
learning approach has the advantage of being more robust to calibration and
relative timing offset errors. However, VINet has not fully addressed the
problem of learning a meaningful sensor fusion strategy.

To tackle the deep sensor fusion problem, Chen et al. [70] proposed selective
sensor fusion, a framework that selectively learns context-dependent
representations for visual inertial pose estimation. Their intuition is that
the importance of features from different modalities should be considered
according to the exterior (i.e., environmental) and interior (i.e.,
device/sensor) dynamics, by fully exploiting the complementary behaviors of
two sensors. Their approach outperforms those without a fusion strategy, e.g.
VINet, avoiding catastrophic failures.

Similar to unsupervised VO, Visual-inertial odometry can also be solved in a
self-supervised fashion using novel view synthesis. VIOLearner [69] constructs
motion transformations from raw inertial data, and converts source images into
target images with the camera matrix and depth maps via the Equation 2
mentioned in Section 3.1.2. In addition, an online error correction module
corrects the intermediate errors of the framework. The network parameters are
recovered by optimizing a photometric loss. Similarly, DeepVIO [71]
incorporates inertial data and stereo images into this unsupervised learning
framework, and is trained with a dedicated loss to reconstruct trajectories in
a global scale.

Learning-based VIO cannot defeat the state-of-the-art classical model based
VIOs, but they are generally more robust to real issues[68, 70, 71] such as
measurement noises, bad time synchronization, thanks to the impressive ability
of DNNs in feature extraction and motion modelling.

###  3.3 Inertial Odometry

Beyond visual odometry and visual-inertial odometry, an inertial-only
solution, i.e. inertial odometry provides an ubiquitous alternative to solve
the odometry estimation problem. Compared with visual methods, an inertial
sensor is relatively low-cost, small, energy efficient and privacy preserving.
It is relatively immune to environmental factors, such as lighting conditions
or moving objects. However, low-cost MEMS inertial measurement units (IMU)
widely found on robots and mobile devices are corrupted with high sensor bias
and noise, leading to unbounded error drifts in the strapdown inertial
navigation system (SINS), if inertial data are doubly integrated.

Chen et al. [85] formulated inertial odometry as a sequential learning problem
with a key observation that 2D motion displacements in the polar coordinate
(i.e. polar vector) can be learned from independent windows of segmented
inertial data. The key observation is that when tracking human and wheeled
configurations, the frequency of their vibrations is relevant to the moving
speed, which is reflected by inertial measurements. Based on this, they
proposed IONet, a LSTM based framework for end-to-end learning of relative
poses from sequences of inertial measurements. Trajectories are generated by
integrating motion displacements. [86] leveraged deep generative models and
domain adaptation technique to improve the generalization ability of deep
inertial odometry in new domains. [87] extends this framework by an improved
triple-channel LSTM network to predict polar vectors for drone localization
from inertial data and sampling time. RIDI [88] trains a deep neural network
to regress linear velocities from inertial data, calibrates the collected
accelerations to satisfy the constraints of the learned velocities, and doubly
integrates the accelerations into locations with a conventional physical
model. Similarly, [89] compensates the error drifts of the classical SINS
model with the aid of learned velocities. Other works have also explored the
usage of deep learning to detect zero-velocity phase for navigating
pedestrians [90] and vehicles [91]. This zero-velocity phase provides context
information to correct system error drifts via Kalman filtering.

Inertial only solution can be a backup plan to offer pose information in
extreme environments, where visual information is not available or is highly
distorted. Deep learning has proven its capability to learn useful features
from noisy IMU data, and compensate the error drifts of inertial dead
reckoning, which is difficult to solve by classical algorithms.

Figure 5: A comparison of the performance of deep learning based visual
odometry with an evaluation on the Trajectory 10 of the KITTI dataset.

###  3.4 LIDAR Odometry

LIDAR sensors provide high-frequency range measurements, with the benefits of
working consistently in complex lighting conditions and optically featureless
scenarios. Mobile robots and self-driving vehicles are normally equipped with
LIDAR sensors to obtain relative self-motion (i.e. LIDAR odometry) and global
pose with respect to a 3D map (LIDAR relocalization). The performance of LIDAR
odometry is sensitive to point cloud registration errors due to non-smooth
motion. In addition, the data quality of LIDAR measurements is also affected
by extreme weather conditions, for example, heavy rain or fog/mist.

Traditionally, LIDAR odometry relies on point cloud registration to detect
feature points, e.g. line and surface segments, and uses a matching algorithm
to obtain the pose transformation by minimizing the distance between two
consecutive point-cloud scans. Data-driven methods consider solving LIDAR
odometry in an end-to-end fashion, by leveraging deep neural networks to
construct a mapping function from point cloud scan sequences to pose estimates
[72, 73, 74]. As point cloud data are challenging to be directly ingested by
neural networks due to their sparse and irregularly sampled format, these
methods typically convert point clouds into a regular matrix through
cylindrical projection, and adopt ConvNets to extract features from
consecutive point cloud scans. These networks regress relative poses and are
trained via ground-truth labels. LO-Net [73] reports competitive performance
over the conventional state-of-the-art algorithm, i.e. the LIDAR Odometry and
Mapping (LOAM) algorithm [16].

###  3.5 Comparison of Odometry Estimation

Table I compares existing work on odometry estimation, in terms of their
sensor type, model, whether a trajectory with an absolute scale is produced,
and their performance evaluation on the KITTI dataset, where available. As
deep inertial odometry has not been evaluated on the KITTI dataset, we do not
include inertial odometry in this table. The KITTI dataset [46] is a common
benchmark for odometry estimation, consisting of a collection of sensor data
from car-driving scenarios. As most data-driven approaches adopt the
trajectory 09 and 10 of the KITTI dataset to evaluate model performance, we
compared them according to the averaged Root Mean Square Error (RMSE) of the
translation for all the subsequences of lengths (100, 200, .., 800) meters,
which is provided by the official KITTI VO/SLAM evaluation metrics.

We take visual odometry as an example. Figure 5 reports the translational
drifts of deep visual odometry models over time on the 10th trajectory of the
KITTI dataset. Clearly, hybrid VO shows the best performance over supervised
VO and unsupervised VO, as the hybrid model benefits from both the mature
geometry models of traditional VO algorithms and the strong capacity for
feature extraction of deep learning. Although supervised VO still outperforms
unsupervised VO, the performance gap between them is diminishing as the
limitations of unsupervised VO are gradually addressed. For example, it has
been found that unsupervised VO now can recover global scale from monocular
images [30]. Overall, data-driven visual odometry shows a remarkable increase
in model performance, indicating the potentials of deep learning approaches in
achieving more accurate odometry estimation in the future.

##  4 Mapping

Mapping refers to the ability of a mobile agent to build a consistent
environmental model to describe the surrounding scene. Deep learning has
fostered a set of tools for scene perception and understanding, with
applications ranging from depth prediction, to semantic labelling, to 3D
geometry reconstruction. This section provides an overview of existing works
relevant to deep learning based mapping methods. We categorize them into
geometric mapping, semantic mapping, and general mapping. Table II summarizes
the existing methods on deep learning based mapping.

TABLE II: A summary of existing methods on deep learning for mapping.

| Output Representation | Employed by
---|---|---
Geometric Map | Depth Representation |  [78, 92, 79], [80, 81], [29], [53, 55, 56, 58, 59, 93, 61, 64], [76, 77]
Voxel Representation |  [94, 95], [96](Object), [97], [98](Object), [99](Object), [100](Object)
Point Representation |  [101](Object)
Mesh Representation |  [102](Object), [103](Object), [104](Object), [105](Object), [106, 107]
Semantic Map | Semantic Segmentation | [26, 27, 108]
Instance Segmentation | [109, 110, 111]
Panoptic Segmentation | [112]
General Map | Neural Representation |  [113],[114, 115, 116], [117, 118, 31, 32]

  * •

Object indicates that this method is only validated on reconstructing single
objects rather than a scene.

###  4.1 Geometric Mapping

Broadly, geometric mapping captures the shape and structural description of a
scene. Typical choices of the scene representations used in geometric mapping
include depth, voxel, point and mesh. We follow this representational taxonomy
and categorize deep learning for geometric mapping into the above four
classes. Figure 6 demonstrates these geometric representations on the Stanford
Bunny benchmark.

Figure 6: An illustrations of scene representations on the Stanford Bunny
benchmark: (a) original model, (b) depth representation, (c) voxel
representation (d) point representation (e) mesh representation.

####  4.1.1 Depth Representation

Depth maps play a pivotal role in understanding the scene geometry and
structure. Dense scene reconstruction has been achieved by fusing depth and
RGB images [119, 120]. Traditional SLAM systems represent scene geometry with
dense depth maps (i.e. 2.5D), such as DTAM [121]. In addition, accurate depth
estimation can contribute to the absolute scale recovery for visual SLAM.

Learning depth from raw images is a fast evolving area in computer vision
community. The earliest work formulates depth estimation as a mapping function
of input single images, constructed by a multi-scale deep neural network [78]
to output the per-pixel depth maps from single images. More accurate depth
prediction is achieved by jointly optimizing the depth and self-motion
estimation [79]. These supervised learning methods [78, 92, 79] can predict
per-pixel depth by training deep neural networks on large data collections of
images with corresponding depth labels. Although they are found outperforming
the traditional structure based methods, such as [122], their effectiveness
are largely reliant on model training and can be difficult to generalize to
new scenarios in absence of labeled data.

On the other side, recent advances in this field focus on unsupervised
solutions, by reformulating depth prediction as a novel view synthesis
problem. [80, 81] utilized photometric consistency loss as a self-supervision
signal for training neural models. With stereo images and a known camera
baseline, [80, 81] synthesize the left view from the right image, and the
predicted depth maps of the left view. By minimizing the distance between
synthesized images and real images, i.e. the spatial consistency, the
parameters of the networks can be recovered via this self-supervision in an
end-to-end manner. Besides the spatial consistency, [29] proposed to apply
temporal consistency as a self-supervised signal, by synthesizing the image in
the target time frame from the source time frame. At the same time, egomotion
is recovered along with the depth estimation. This framework only requires
monocular images to learn both the depth maps and egomotion. A number of
following works [53, 55, 56, 58, 59, 93, 61, 64, 76, 77] extended this
framework and achieved better performance in depth and egomotion estimation.
We refer the readers to Section 3.1.2, in which a variety of additional
constraints haven been discussed.

With the depth maps predicted by ConvNets, learning based SLAM systems can
integrate depth information to address some limitations of classical monocular
solution. For example, CNN-SLAM [123] utilizes the learned depths from single
images into a monocular SLAM framework (i.e. LSD-SLAM [124]). Their experiment
shows how the learned depth maps contribute to mitigate the absolute scale
recovery problem in pose estimates and scene reconstruction. CNN-SLAM achieves
dense scene predictions even in texture-less areas, which is normally hard for
a conventional SLAM system.

####  4.1.2 Voxel Representation

Voxel-based formulation is a natural way to represent 3D geometry. Similar to
the usage of pixel (i.e. 2D element) in images, voxel is a volume element in a
three-dimensional space. Previous works have explored to use multiple input
views, to reconstruct the volumetric representation of scene [94, 95] and
objects [96]. For example, SurfaceNet [94] learns to predict the confidence of
a voxel to determine whether it is on surface or not, and reconstruct the 2D
surface of a scene. RayNet [95] reconstructs the scene geometry by extracting
view-invariant features while imposing geometric constraints. Recent works
focus on generating high-resolution 3D volumetric models [98, 97]. For
example, Tatarchenko et al. [97] designed a convolutional decoder based on
octree-based formulation to enable scene reconstruction in much higher
resolution. Other work can be found on scene completion from RGB-D data [99,
100]. One limitation of voxel representation is the high computational
requirement, especially when attempting to reconstruct a scene in high
resolution.

####  4.1.3 Point Representation

Point-based formulation consists of the 3-dimensional coordinates (x, y, z) of
points in 3D space. Point representation is easy to understand and manipulate,
but suffers from the ambiguity problem, which means that different forms of
point clouds can represent a same geometry. The pioneer work, PointNet [125],
processes unordered point data with a single symmetric function - max pooling,
to aggregate point features for classification and segmentation. Fan et al.
[101] developed a deep generative model that generates 3D geometry in point-
based formulation from single images. In their work, a loss function based on
Earth Mover’s distance is introduced to tackle the problem of data ambiguity.
However, their method is only validated on the reconstruction task of single
objects. No work on point generation for scene reconstruction has been found
yet.

####  4.1.4 Mesh Representation

Mesh-based formulation encodes the underlying structure of 3D models, such as
edges, vertices and faces. It is a powerful representation that naturally
captures the surface of 3D shape. Several works considered the problem of
learning mesh generation from images [102, 103] or point clouds data [104,
105]. However, these approaches are only able to reconstruct single objects,
and limited to generating models with simple structures or from familiar
classes. To tackle the problem of scene reconstruction in mesh representation,
[106] integrates the sparse features from monocular SLAM with dense depth maps
from ConvNet for the update of 3D mesh representation. The depth predictions
are fused into the monocular SLAM system to recover the absolute scale of pose
and scene features estimation. To allow efficient computation and flexible
information fusion, [107] utilizes 2.5D mesh to represent scene geometry. In
their approach, the image plane coordinates of mesh vertices are learned by
deep neural networks, while the depth maps are optimized as free variables.

###  4.2 Semantic Map

Figure 7: (b) semantic segmentation, (c) instance segmentation and (d)
panoptic segmentation for semantic mapping [126].

Semantic mapping connects semantic concepts (i.e. object classification,
material composition etc) with the geometry of environments. This is treated
as a data association problem. The advances in deep learning greatly fosters
the developments of object recognition and semantic segmentation. Maps with
semantic meanings enable mobile agents to have high-level understandings of
their environments beyond pure geometry, and allow for a greater range of
functionality and autonomy.

SemanticFusion [26] is one of the early works that combined the semantic
segmentation labels from deep ConvNet with the dense scene geometry from a
SLAM system. It incrementally integrates per-frame semantic segmentation
predictions into a dense 3D map by probabilistically associating the 2D frames
with the 3D map. This combination not only generates a map with useful
semantic information, but also shows the integration with a SLAM system helps
to enhance the single frame segmentation. The two modules are loosely coupled
in SemanticFusion. [27] proposed a self-supervised network that predicts
consistent semantic labels for a map, by imposing constraints on the
consistency of semantic predictions in multiple views. DA-RNN [108] introduces
recurrent models into semantic segmentation framework to learn the temporal
connections over multiple view frames, producing more accurate and consistent
semantic labelling for volumetric maps from KinectFusion [127]. Yet these
methods provide no information on object instances, which means that they are
not able to distinguish among different ojects from the same category.

With the advances in instance segmentation, semantic mapping evolves into the
instance level. A good example is [109] that offers object-level semantic
mapping by identifying individual objects via a bounding box detection module
and an unsupervised geometric segmentation module. Unlike other dense semantic
mapping approaches, Fusion++ [110] builds a semantic graph-based map, which
predicts only object instances and maintains a consistent map via loop closure
detection, pose-graph optimization and further refinement. [111] presented a
framework that achieves instance-aware semantic mapping, and enables novel
object discovery. Recently, panoptic segmentation [126] attracts a lot of
attentions. PanopticFusion [112] advanced semantic mapping to the level of
stuff and things level that classifies static objects, e.g. walls, doors,
lanes as stuff classes, and other accountable objects as things classes, e.g.
moving vehicles, human and tables. Figure 7 compares semantic segmentation,
instance segmentation and panoptic segmentation.

###  4.3 General Map

Beyond the explicit geometric and semantic map representation, deep learning
models are able to encode the whole scene into an implicit representation,
i.e. a general map representation to capture the underlying scene geometry and
appearance.

Utilizing deep autoencoders can automatically discover the high-level compact
representation of high-dimensional data. A notable example is CodeSLAM [113]
that encodes observed images into a compact and optimizable representation to
contain the essential information of a dense scene. This general
representation is further used into a keyframe-based SLAM system to infer both
pose estimates and keyframe depth maps. Due to the reduced size of learned
representations, CodeSLAM allows efficient optimization of tracking camera
motion and scene geometry for a global consistency.

Neural rendering models are another family of works that learn to model 3D
scene structure implicitly by exploiting view synthesis as a self-supervision
signal. The target of neural rendering task is to reconstruct a new scene from
an unknown viewpoint. The seminar work, Generative Query Network (GQN) [128]
learns to capture representation and render a new scene. GQN consists of a
representation network and a generation network: the representation network
encodes the observations from reference views into a scene representation; the
generation network which is based on recurrent model, reconstructs the scene
from a new view conditioned on the scene representation and a stochastic
latent variable. Taking inputs as the observed images from several viewpoints,
and the camera pose of a new view, GQN predicts the physical scene of this new
view. Intuitively, through end-to-end training, the representation network can
capture the necessary and important factors of 3D environment for the scene
reconstruction task via the generation network. GQN is extended by
incorporating a geometric-aware attention mechanism to allow more complex
environment modelling [114], as well as including multimodal data for scene
inference [115]. Scene representation network (SRN) [116] tackles the scene
rendering problem via a learned continuous scene representation that connects
a camera pose and its corresponding observation. A differentiable Ray Marching
algorithm is integrated into SRN to enforce the network to model 3D structure
consistently. However, these frameworks can only be applied to synthetic
datasets due to the complexity of real-world environments.

Last but not least, in the quest of ‘map-less’ navigation, task-driven maps
emerge as a novel map representation. This representation is jointly modelled
by deep neural networks with respect to the task at hand. Generally those
tasks leverage location information, such as navigation or path planning,
requiring mobile agents to understand the geometry and semantics of
environment. Navigation in unstructured environments (even in a city scale) is
formulated as an policy learning problem in these works [117, 118, 31, 32],
and solved by deep reinforcement learning. Different from traditional
solutions that follow a procedure of building an explicit map, planning path
and making decisions, these learning based techniques predict control signals
directly from sensor observations in an end-to-end manner, without explicitly
modelling the environment. The model parameters are optimized via sparse
reward signals, for example, whenever agents reach a destination, a positive
reward will be given to tune the neural network. Once a model is trained, the
actions of agents can be determined conditioned on the current observations of
environment, i.e. images. In this case, all of environmental factors, such as
the geometry, appearance and semantics of a scene, are embedded inside the
neurons of a deep neural network and suitable for solving the task at hand.
Interestingly, the visualization of the neurons inside a neural model that is
trained on the navigation task via reinforcement learning, has similar
patterns as the grid and place cells inside human brain. This provides
cognitive cues to support the effectiveness of neural map representation.

##  5 Global Localization

Global localization concerns the retrieval of absolute pose of a mobile agent
within a known scene. Different from odometry estimation that relies on
estimating the internal dynamical model and can perform in an unseen scenario,
in global localization, prior knowledge about the scene is provided and
exploited, through a 2D or 3D scene model. Broadly, it describes the relation
between the sensor observations and map, by matching a query image or view
against a pre-built model, and returning an estimate of global pose.

We categorize deep learning based global localization into three categories,
according to the types of inquiry data and map: _2D-to-2D localization_
queries 2D images against an explicit database of geo-referenced images or
implicit neural map; _2D-to-3D localization_ establishes correspondences
between 2D pixels of images and 3D points of a scene model; and _3D-to-3D
localization_ matches 3D scans to a pre-built 3D map. Table III, IV and V
summarize the existing approaches on deep learning based 2D-to-2D
localization, 2D-to-3D localization and 3D-to-3D localization respectively.

###  5.1 2D-to-2D Localization

2D-to-2D localization regresses the camera pose of an image against a 2D map.
Such 2D map is explicitly built by a geo-referenced database or implicitly
encoded in a neural network.

Figure 8: The typical architectures of 2D-to-2D based localization through (a)
explict map, i.e. RelocNet [129] and (b) implicit map, i.e. e.g. PoseNet [130]
TABLE III: A summary on existing methods on deep learning for 2D-to-2D global
localization

Model | Agnostic | Performance (m/degree) | Contributions
---|---|---|---
7Scenes | Cambridge
2D-to-2D Localization |  Explicit Map |  NN-Net [131] | Yes | 0.21/9.30 | - | combine retrieval and relative pose estimation
DeLS-3D [132] | No | - | - | jointly learn with semantics
AnchorNet [133] | Yes | 0.09/6.74 | 0.84/2.10 | anchor point allocation
RelocNet [129] | Yes | 0.21/6.73 | - | camera frustum overlap loss
CamNet [134] | Yes | 0.04/1.69 | - | multi-stage image retrieval
Implicit Map |  PoseNet [130] | No | 0.44/10.44 | 2.09/6.84 | first neural network in global pose regression
Bayesian PoseNet [135] | No | 0.47/9.81 | 1.92/6.28 | estimate Bayesian uncertainty for global pose
BranchNet [136] | No | 0.29/8.30 | - | multi-task learning for orientation and translation
VidLoc [137] | No | 0.25/- | - | efficient localization from image sequences
Geometric PoseNet [138] | No | 0.23/8.12 | 1.63/2.86 | geometry-aware loss
SVS-Pose [139] | No | - | 1.33/5.17 | data augmentation in 3D space
LSTM PoseNet [140] | No | 0.31/9.85 | 1.30/5.52 | spatial correlation
Hourglass PoseNet [141] | No | 0.23/9.53 | - | hourglass-shaped architecture
VLocNet [142] | No | 0.05/3.80 | 0.78/2.82 | jointly learn global localization and odometry
MapNet [143] | No | 0.21/7.77 | 1.63/3.64 | impose spatial and temporal constraints
SPP-Net [144] | No | 0.18/6.20 | 1.24/2.68 | synthetic data augmentation
GPoseNet [145] | No | 0.30/9.90 | 2.00/4.60 | hybrid model with Gaussian Process Regressor
VLocNet++ [146] | No | 0.02/1.39 | - | jointly learn with odometry and semantics
LSG [147] | No | 0.19/7.47 | - | odometry-aided localization
PVL [148] | No | - | 1.60/4.21 | prior-guided dropout mask to improve robustness
AdPR [149] | No | 0.22/8.8 | - | adversarial architecture
AtLoc [150] | No | 0.20/7.56 | - | attention-guided spatial correlation

  * •

Agnostic indicates whether it can generalize to new scenarios.

  * •

Performance reports the position (m) and orientation (degree) error (a small
number is better) on the 7-Scenes (Indoor)[151] and Cambridge (Outdoor)
dataset[130]. Both datasets are split into training and testing set. We report
the averaged error on the testing set.

  * •

Contributions summarize the main contributions of each work compared with
previous research.

####  5.1.1 Explicit Map Based Localization

Explicit map based 2D-to-2D localization typically represents the scene by a
database of geo-tagged images (references) [152, 153, 154]. Figure 8 (a)
illustrates the two stages of this localization with 2D references: image
retrieval determines the most relevant part of a scene represented by
reference images to the visual queries; pose regression obtains the relative
pose of query image with respect to the reference images.

One problem here is how to find suitable image descriptors for image
retrieval. Deep learning based approaches [155, 156] are based on a pre-
trained ConvNet model to extract image-level features, and then use these
features to evaluate the similarities against other images. In challenging
situations, local descriptors are first extracted, followed by being
aggregated to obtain robust global descriptors. A good example is NetVLAD
[157] that designs a trainable generalized VLAD (the Vector of Locally
Aggregated Descriptors) layer. This VLAD layer can be plugged into the off-
the-shelf ConvNet architecture to encourage better descriptors learning for
image retrieval.

In order to obtain more precise poses of the queries, additional relative pose
estimation with respect to the retrieved images is required. Traditionally,
relative pose estimation is tackled by epipolar geometry, relying on the 2D-2D
correspondences determined by local descriptors [158, 159]. In contrast, deep
learning approaches regress the relative poses straightforwardly from pairwise
images. For example, NN-Net [131] utilized neural network to estimate the
pairwise relative poses between the query and the top N ranked references. A
triangulation-based fusion algorithm coalesces the predicted N relative poses
and the ground truth of 3D geometry poses, and the absolute query pose can be
naturally calculated. Furthermore, Relocnet [129] introduces a frustum overlap
loss to assist global descriptors learning that are suitable for camera
localization. Motivated by these, CamNet [134] applies two stages retrieval,
image-based coarse retrieval and pose-based fine retrieval, to select the most
similar reference frames for finally precise pose estimation. Without the need
of training on specific scenarios, reference-based approaches are naturally
scalable and flexible to be utilized in new scenarios. Since reference-based
methods need to maintain a database of geo-tagged images, they are more
trivial to scale to large-scale scenarios, compared with the structure-based
counterparts. Overall, these image retrieval based methods achieve a trade-off
between accuracy and scalability.

####  5.1.2 Implicit Map Based Localization

Implicit map based localization directly regresses camera pose from single
images, by implicitly representing the structure of entire scene inside a deep
neural network. The common pipeline is illustrated in Figure 8 (b) - the input
to a neural network is single images, while the output is the global position
and orientation of query images.

PoseNet [130] is the first work to tackle camera relocalization problem by
training a ConvNet to predict camera pose from single RGB images in an end-to-
end manner. PoseNet is based on the main structure of GoogleNet [160] to
extract visual features,, but removes the last softmax layers. Instead, a
fully connected layer was introduced to output a 7 dimensional global pose,
consisting of position and orientation vector in 3 and 4 dimensions
respectively. However, PoseNet was designed with a naive regression loss
function without any consideration for geometry, in which the hyper-parameters
inside requires expensive hand-engineering to be tuned. Furthermore, it also
suffers from the overfitting problem due to the high dimensionality of the
feature embedding and limited training data. Thus various extensions enhance
the original pipeline by exploiting LSTM units to reduce the dimensionality
[140], applying synthetic generation to augment training data [139, 136, 144],
replacing backbone with ResNet34 [141], modelling pose uncertainty [135, 145]
and introducing geometry-aware loss function [138]. Alternatively, Atloc [150]
associates the features in spatial domain with attention mechanism, which
encourages the network to focus on parts of the image that are temporally
consistent and robust. Similarly, a prior guided dropout mask is additionally
adopted in RVL [148] to further eliminate the uncertainty caused by dynamic
objects. Different such methods only considering spatial connections, VidLoc
[137] incorporates temporal constraints of image sequences to model the
temporal connections of input images for visual localization. Moreover,
additional motion constraints, including spatial constraints and other sensor
constraints from GPS or SLAM systems are exploited in MapNet [143], to enforce
the motion consistency between predicted poses. Similar motion constraints are
also added by jointly optimizing a relocalization network and visual odometry
network [142, 147]. However, being application-specific, scene representations
learned from localization tasks may ignore some useful features they are not
designed for. Out of this, VLocNet++ [146] and FGSN [161] additionally
exploits the inter-task relationship among learning semantics and regressing
poses, achieving impressive results.

Implicit map based localization approaches take the advantages of deep
learning in automatically extracting features, that play a vital role in
global localization in featureless environments, where conventional methods
are prone to fail. However, the requirement of scene-specific training
prohibits it from generalizing to unseen scenes without being retrained. Also,
current implicit map based approaches have not shown comparable performance
over other explicit map based methods [162].

###  5.2 2D-to-3D Localization

2D-to-3D localization refers to methods that recover the camera pose of a 2D
image with respect to a 3D scene model. This 3D map is pre-built before
performing global localization, via approaches such as structure from motion
(SfM)[43]. As shown in Figure 9, 2D-to-3D approaches establish 2D-3D
correspondences between the 2D pixels of query image and the 3D points of
scene model through local descriptor matching [163, 164, 165] or by regressing
3D coordinates from pixel patches [166, 167, 151, 168]. Such 2D-3D matches are
then used to calculate camera pose by applying a Perspective-n-Point (PnP)
solver [169, 170] inside a RANSAC loop [171].

Figure 9: The typical architectures of 2D-to-3D based localization through (a)
descriptor matching, i.e. HF-Net [172] and (b) scene coordinate regression,
i.e. Confidence SCR [173]. TABLE IV: A summary on existing methods on deep
learning for 2D-to-3D global localization

Model | Agnostic | Performance (m/degree) | Contributions
---|---|---|---
7Scenes | Cambridge
2D-3D Localization |  Descriptor Based |  NetVLAD [157] | Yes | - | - | differentiable VLAD layer
DELF [174] | Yes | - | - | attentive local feature descriptor
InLoc [175] | Yes | 0.04/1.38 | 0.31/0.73 | dense data association
SVL [176] | No | - | - | leverage a generative model for descriptor learning
SuperPoint [177] | Yes | - | - | jointly extract interest points and descriptors
Sarlin et al. [178] | Yes | - | - | hierarchical localization
NC-Net [179] | Yes | - | - | neighbourhood consensus constraints
2D3D-MatchNet [180] | Yes | - | - | jointly learn the descriptors for 2D and 3D keypoints
Unsuperpoint [181] | Yes | - | - | unsupervised detector and descriptor learning
HF-Net [172] | Yes | - | - | coarse-to-fine localization
D2-Net [182] | Yes | - | - | jointly learn keypoints and descriptors
Speciale et al [183] | No | - | - | privacy preserving localization
OOI-Net [184] | No | - | - | objects-of-interest annotations
Camposeco et al. [185] | Yes | - | 0.56/0.66 | hybrid scene compression for localization
Cheng et al. [186] | Yes | - | - | cascaded parallel filtering
Taira et al. [187] | Yes | - | - | comprehensive analysis of pose verification
R2D2 [188] | Yes | - | - | learn a predictor of the descriptor discriminativeness
ASLFeat [189] | Yes | - | - | leverage deformable convolutional networks
Scene Coordinate Regression |  DSAC [190] | No | 0.20/6.3 | 0.32/0.78 | differentiable RANSAC
DSAC++ [191] | No | 0.08/2.40 | 0.19/0.50 | without using a 3D model of the scene
Angle DSAC++ [192] | No | 0.06/1.47 | 0.17/0.50 | angle-based reprojection loss
Dense SCR [193] | No | 0.04/1.4 | - | full frame scene coordinate regression
Confidence SCR [173] | No | 0.06/3.1 | - | model uncertainty of correspondences
ESAC [194] | No | 0.034/1.50 | - | integrates DSAC in a Mixture of Experts
NG-RANSAC [195] | No | - | 0.24/0.30 | prior-guided model hypothesis search
SANet [196] | Yes | 0.05/1.68 | 0.23/0.53 | scene agnostic architecture for camera localization
MV-SCR [197] | No | 0.05/1.63 | 0.17/0.40 | multi-view constraints
HSC-Net [198] | No | 0.03/0.90 | 0.13/0.30 | hierarchical scene coordinate network
KFNet [199] | No | 0.03/0.88 | 0.13/0.30 | extends the problem to the time domain

  * •

Agnostic indicates whether it can generalize to new scenarios.

  * •

Performance reports the position (m) and orientation (degree) error (a small
number is better) on the 7-Scenes (Indoor)[151] and Cambridge (Outdoor)
dataset[130]. Both datasets are split into training and testing set. We report
the averaged error on the testing set.

  * •

Contributions summarize the main contributions of each work compared with
previous research.

####  5.2.1 Descriptor Matching Based Localization

Descriptor matching methods mainly rely on feature detector and descriptor,
and establish the correspondences between the features from 2D input and 3D
model. They can be further divided into three types: detect-then-describe,
detect-and-describe, and describe-to-detect, according to the role of detector
and descriptor in the learning process.

Detect-then-describe approach first performs feature detection and then
extracts a feature descriptor from a patch centered around each keypoint [200,
201]. The keypoint detector is typically responsible for providing robustness
or invariance against possible real issues such as scale transformation,
rotation, or viewpoint changes by normalizing the patch accordingly. However,
some of these responsibilities might also be delegated to the descriptor. The
common pipeline varies from using hand-crafted detectors [202, 203] and
descriptors [204, 205], replacing either the descriptor [206, 207, 208, 179,
209, 210] or detector [211, 212, 213] with a learned alternative, or learning
both the detector and descriptor [214, 215]. For efficiency, the feature
detector often considers only small image regions and typically focuses on
low-level structures such as corners or blobs [216]. The descriptor then
captures higher level information in a larger patch around the keypoint.

In contrast, detect-and-describe approaches advance description stage. By
sharing a representation from deep neural network, SuperPoint [177],
UnSuperPoint [181] and R2D2 [188] attempt to learn a dense feature descriptor
and a feature detector. However, they rely on different decoder branches which
are trained independently with specific losses. On the contrary, D2-net [182]
and ASLFeat [189] shares all parameters between detection and description and
uses a joint formulation that simultaneously optimizes for both tasks.

Similarly, the describe-to-detect approach, e.g. D2D [217], also postpones the
detection to a later stage but applies such detector on pre-learned dense
descriptors to extract a sparse set of keypoints and corresponding
descriptors. Dense feature extraction foregoes the detection stage and
performs the description stage densely across the whole image [218, 219, 220,
176]. In practice, this approach has shown to lead to better matching results
than sparse feature matching, particularly under strong variations in
illumination [221, 222]. Different from these works, which purely rely on
image features, 2D3D-MatchNet [180] proposed to learn local descriptors that
allow direct matching of key points across a 2D image and 3D point cloud.
Similarly, LCD [223] introduced a dual auto-encoder architecture to extract
cross-domain local descriptors. However, they still require pre-defined 2D and
3D keypoints separately, which will result in poor matching results caused by
inconsistent keypoint selection rules.

Figure 10: The typical architecture of 3D-to-3D localization, e.g. L3-Net
[224]. TABLE V: A summary of existing approaches on deep learning for 3D-to-3D
localization

Models | Agnostic | Contributions
---|---|---
LocNet[225] | No | convert 3D points into 2D matrix, search in global prior map
PointNetVLAD[226] | Yes | learn global descriptor from point clouds
Barsan et al.[227] | No | learn from LIDAR intensity maps and online point clouds
L3-Net[224] | No | extract feature by PointNet
PCAN[228] | Yes | predict the significance of each local point based on context
DeepICP[229] | Yes | generate matching correspondence from learned matching probabilities
DCP[230] | Yes | a learning based iterative closest point
D3Feat[231] | Yes | jointly learn detector and descriptors for 3D points

  * •

Agnostic indicates whether it can generalize to new scenarios.

  * •

Contributions summarize the main contributions of each work compared with
previous research.

####  5.2.2 Scene Coordinate Regression Based Localization

Different from match-based methods that establish 2D-3D correspondences before
calculating pose, scene coordinate regression approaches estimate the 3D
coordinates of each pixel from the query image within the world coordinate
system, i.e. the scene coordinates. It can be viewed as learning a
transformation from the query image to the global coordinates of the scene.
DSAC [190] utilizes a ConvNet model to regress scene coordinates, followed by
a novel differentiable RANSAC to allow end-to-end training of the whole
pipeline. Such common pipeline was then improved by introducing the
reprojection loss [191, 232, 192] or multi-view geometric constraints [197] to
enable unsupervised learning, jointly learning the observation confidences
[173, 195] to enhance the sampling efficiency and accuracy, exploiting Mixture
of Experts (MoE) strategy [194] or hierarchical coarse-to-fine [198] to
eliminate environment ambiguities. Different from these, KFNet [199] extends
the scene coordinate regression problem to the time domain and thus bridges
the existing performance gap between temporal and one-shot relocalization
approaches. However, they still trained for a specific scene and cannot be
generalized to unseen scenes without retraining. To build a scene agnostic
method, SANet [196] regress the scene coordinate map of the query by
interpolating the 3D points associated with retrieved scene images. Unlike
aforementioned methods trained in a patch-based manner, Dense SCR [193]
propose to perform the scene coordinate regression in a full-frame manner to
make the computation efficient at test time and, more importantly, to add more
global context to the regression process to improve the robustness.

Scene coordinate regression methods often perform better robustness and higher
accuracy under small indoor scenarios, outperforming traditional algorithms
such as [18]. But they have not yet proven their capacity in large-scale
scenes.

###  5.3 3D-to-3D Localization

3D-to-3D localization (or LIDAR localization) refers to methods that recover
the global pose of 3D points (i.e. LIDAR point cloud scans) against a pre-
built 3D map by establishing a 3D-to-3D correspondence matching. Figure 10
shows the pipeline of 3D-to-3D localization: online scans or predicted coarse
poses are applied to query the most similar 3D map data, which are further
used for precise localization by calculating the offset between predicted
poses and ground truths or estimating the relative poses between online scans
and queried scene.

By formulating LIDAR localization as a recursive Bayesian inference problem,
[227] embeds both LIDAR intensity maps and online point cloud sweeps in a
sharing space for fully differentiable pose estimation. Instead of operating
on 3D data directly, LocNet [225] converts point cloud scans to 2D rotational
invariant representation for searching similar frames in the global prior map,
and then performs the iterative closest point (ICP) methods to calculate
global pose. Towards proposing a learning based LIDAR localization framework
that directly processes point clouds, L3-Net [224] processes point cloud data
with PointNet [125] to extract feature descriptors that encode certain useful
properties, and models the temporal connections of motion dynamics via a
recurrent neural network. It optimizes the loss between the predicted poses
and ground truth values by minimizing the matching distance between the point
cloud input and the 3D map. Some techniques, such as PointNetVLAD [226], PCAN
[228] and D3Feat [231] explored to retrieve the reference scene at the
beginning, while other techniques such as DeepICP [229] and DCP [230] allow to
estimate relative motion transformations from 3D scans. Compared with image-
based relocalization including 2D-to-3D and 2D-to-2D localization, 3D-to-3D
localization is relatively underexplored.

##  6 SLAM

Simultaneously tracking self-motion and estimate the structure of surroundings
constructs a simultaneous localization and mapping (SLAM) system. The
individual modules of localization and mapping discussed in the above sections
can be viewed as modules of a complete SLAM systems. This section overviews
the SLAM systems using deep learning, with the main focus on the modules that
contribute to the integration of a SLAM system, including local/global
optimization, keyframe/loop closure detection and uncertainty estimation.
Table VI summarizes the existing approaches that employ the deep learning
based SLAM modules discussed in this section.

###  6.1 Local Optimization

When jointly optimizing estimated camera motion and scene geometry, SLAM
systems enforce them to satisfy a certain constraint. This is done by
minimizing a geometric or photometric loss to ensure their consistency in the
local area - the surroundings of camera poses, which can be viewed as a bundle
adjustment (BA) problem[233]. Learning based approaches predict depth maps and
ego-motion through two individual networks [29] trained above large datasets.
During the testing procedure when deployed online, there is a requirement that
enforces the predictions to satisfy the local constraints. To enable local
optimization, traditionally, the second-order solvers, e.g. Gauss-Newton (GN)
method or Levenberg-Marquadt (LM) algorithm [234], are applied to optimize
motion transformations and per-pixel depth maps

To this end, LS-Net [235] tackled this problem via a learning based optimizer
by integrating the analytical solvers into its learning process. It learns a
data-driven prior, followed by refining the DNN predictions with an analytical
optimizer to ensure photometric consistency. BA-Net [236] integrates a
differentiable second-order optimizer (LM algorithm) into a deep neural
network to achieve end-to-end learning. Instead of minimizing geometric or
photometric error, BA-Net is performed on feature space to optimize the
consistency loss of features from multiview images extracted by ConvNets. This
feature-level optimizer can mitigate the fundamental problems of geometric or
photometric solution, i.e. some information may be lost in the geometric
optimization, while environmental dynamics and lighting changes may impact the
photometric optimization). These learning based optimizers provide an
alternative to solve bundle adjustment problem.

TABLE VI: A summary of existing approaches on deep learning for SLAM Modules | Employed by
---|---
Local optimization | [235, 236]
Global optimization | [123, 237, 64, 238]
Keyframe detection | [77]
Loop-closure detection | [239, 240, 241, 242]
Uncertainty Estimation | [243, 135, 137]

###  6.2 Global Optimization

Odometry estimation suffers from the accumulative error drifts during long-
term operations, due to the fundamental problems of path integration, i.e. the
system error accumulate without effective restrictions. To address this,
graph-SLAM [42] constructs a topological graph to represent camera poses or
scene features as graph nodes, which are connected by edges (measured by
sensors) to constrain the poses. This graph-based formulation can be optimized
to ensure the global consistency of graph nodes and edges, mitigating the
possible errors on pose estimates and the inherent sensor measurement noise. A
popular solver for global optimization is through Levenberg-Marquardt (LM)
algorithm.

In the era of deep learning, deep neural networks excel at extracting
features, and constructing functions from observations to poses and scene
representations. A global optimization upon the DNN predictions is necessary
to reducing the drifts of global trajectories and support large-scale mapping.
Compared with a variety of well-researched solutions in classical SLAM,
optimizing deep predictions globally is underexplored.

Existing works explored to combine learning modules into a classical SLAM
system at different levels - in the front-end, DNNs produce predictions as
priors, followed by incorporating these deep predictions into the back-end for
next step optimization and refinement. One good example is CNN-SLAM [123],
which utilizes the learned per-pixel depths into LSD-SLAM [124], a full SLAM
system to support loop closing and graph optimization. Camera poses and scene
representations are jointly optimized with depth maps to produce consistent
scale metrics. In DeepTAM [237], both the depth and pose predictions from deep
neural networks are introduced into a classical DTAM system [121], that is
optimized globally by the back-end to achieve more accurate scene
reconstruction and camera motion tracking. A similar work can be found on
integrating unsupervised VO with a graph optimization back-end [64].
DeepFactors [238] vice versa integrates the learned optimizable scene
representation (their so-called code representation) into a different style of
back-end - probabilistic factor graph for global optimization. The advantage
of the factor-graph based formulation is its flexibility to include sensor
measurements, state estimates, and constraints. In a factor graph bach-end, it
is quite easy and convenient to add new sensor modalities, pairwise
constraints and system states into the graph for optimization. However, these
back-end optimizers are not yet differentiable.

###  6.3 Keyframe and Loop-closure Detection

Detecting keyframe and loop-closing is of key importance to the back-end
optimization of SLAM systems.

Keyframe selection facilitates SLAM systems to be more efficient. In the key-
frame based SLAM systems, pose and scene estimates are only refined when a
keyframe is detected. [77] provides a learning solution to detect key-frames
together with unsupervised learning of ego-motion tracking and depths
estimation [29]. Whether an image is the keyframe is determined by comparing
its feature similarity with existing keyframes (i.e. if the similarity is
below a threshold, this image will be treated as a new keyframe).

Loop-closure detection or place recognition is also an important module in
SLAM back-end to reduce open-loop errors. Conventional works are based on bag-
of-words (BoW) to store and use the visual features from the hand-crafted
detectors. However, this problem is complicated by the changes of
illumination, weather, viewpoints and moving objects in real-world scenarios.
To solve this, previous researchers such as [239] proposed to use the ConvNet
features instead, that are from a pre-trained model on a generic large-scale
image processing dataset. These methods are more robust against the variance
of viewpoints and conditions due to the high-level representations extracted
by deep neural networks. Other representative works [240, 241, 242] are built
on deep auto-encoder structure to extract a compact representation, that
compresses scene in an unsupervised manner. Deep learning based loop closing
contributes more robust and effective visual features, and achieves state-of-
the-art performance in place recognition, which is suitable to be integrated
in SLAM systems.

###  6.4 Uncertainty Estimation

Safety and interpretability are an critical step towards the practical
deployment of mobile agents in everyday life: the former enables agents to
live and act with human reliably, while the latter allows users to have better
understanding over the model behaviours. Although deep learning models achieve
state-of-the-art performance in a wide range of regression and classification
tasks, some corner cases should be given enough attention as well. In these
failure cases, errors from one component will propagate to the other
downstream modules, causing catastrophic consequences. To this end, there is
an emerging need to estimate uncertainty for deep neural networks to ensure
safety and provide interpretability.

Deep learning models usually only produce the mean values of predictions, for
example, the output of a DNN-based visual odometry model is a 6-dimensional
relative pose vector, i.e. the translation and rotation. In order to capture
the uncertainty of deep models, learning models can be augmented into a
Bayesian model [244, 245]. The uncertainty from Bayesian models is broadly
categorized into Aleatoric uncertainty and epistemic uncertainty: Aleatoric
uncertainty reflects observation noises, e.g. sensor measurement or motion
noises; epistemic uncertainty captures the model uncertainty [245]. In the
context of this survey, we focus on the work of estimating uncertainty on the
specific task of localization and mapping, with regard to their usages, i.e.
whether they capture the uncertainty with the purpose of motion tracking or
scene understanding.

The uncertainty of DNN-based odometry estimation has been explored by [243,
246]. They adopted a common strategy to convert the target predictions into a
Gaussian distribution, conditioned on the mean value of pose estimates and its
covariance. The parameters inside the framework are optimized via the loss
function with a combination of mean and covariance. By minimizing the error
function to find the best combination, the uncertainty is automatically
learned in an unsupervised fashion. In this way, the uncertainty of motion
transformation is recovered. The motion uncertainty plays a vital role in
probabilistic sensor fusion or the back-end optimization of SLAM systems. To
validate the effectiveness of uncertainty estimation in SLAM systems, [243]
integrated the learned uncertainty into a graph-SLAM as the covariances of
odometry edges. Based on these covariances a global optimization is then
performed to reduce system drifts. It also confirms that uncertainty
estimation improves the performance of SLAM systems over the baseline with a
fixed predefined value of covariance. Similar Bayesian models are applied to
the global relocalization problem. As illustrated in [135, 137], the
uncertainty from deep models are able to reflect the global location errors,
in which the unreliable pose estimates are avoided with this belief metric.

In addition to the uncertainty for motion/relocalization, estimating the
uncertainty for scene understanding also contributes to SLAM systems. This
uncertainty offers a belief metric in to what extent the environmental
perception and scene structure should be trusted. For example, in the semantic
segmentation and depth estimation tasks, uncertainty estimation provides per-
pixel uncertainties for the DNN predictions [247, 245, 248, 249]. Further
more, scene uncertainty is applicable to building hybrid SLAM systems. For
example, photometric uncertainty can be learned to capture the variance of
intensity on each image pixel, and hence enhances the robustness of SLAM
system to observation noise [25].

##  7 Open Questions

Although deep learning has brought great success to the research in
localization and mapping, as aforementioned, existing models are not
sophisticated enough to completely solve the problem at hand. The current form
of deep solutions is still in its infancy state. Towards great autonomy in the
wild, there are numerous challenges for future researchers to investigate.
Practical applications of these techniques should be considered a systematic
research problem. We discuss several open questions that likely lead the
further development in this area.

1) End-to-end model vs. hybrid model. End-to-end learning models are able to
predict self-motion and scene directly from raw data, without any hand-
engineering. Benefited from the advances of deep learning, end-to-end models
are evolving fast to achieve increasing performance in accuracy, efficiency
and robustness. Meanwhile, these models have been shown easier to be
integrated with other high-level learning tasks, e.g. path planning and
navigation[31]. Fundamentally, there exist underlying physical or geometric
models to govern localization and mapping systems. Whether we should develop
end-to-end models relying only on the power of data-driven approaches or
integrate deep learning modules into the pre-built physical/geometric models
as a hybrid model is a critical question for future research. As we can see,
hybrid models already achieved the state-of-the art results in many tasks,
e.g. visual odometry[25], and global localization[191]. Thus, it is reasonable
to investigate in the way to take better advantage of the prior empirical
knowledge from deep learning for hybrid models. On the other side, pure end-
to-end models are data hunger. The performance of current models can be
limited by the size of training dataset, and it is essential to create large
and diverse dataset to enlarge the capacity of data-driven models.

2) Unifying evaluation benchmark and metric. Finding suitable evaluating
benchmark and metric is always a concern for SLAM systems. This is especially
the case for DNN based systems. The predictions from DNNs are affected by the
characteristics of both training and test data, including the dataset size,
hyperparameters (batch size and learning rate etc.), and the difference in
testing scenarios. Therefore, it is hard to fairly compare them when
considering dataset differences, training/testing configuration, or evaluation
metric adopted in each work. For example, the KITTI dataset is a common choice
to evaluate visual odometry, but previous works split training and testing
data in different ways (e.g. [24, 48, 50] used Sequence 00, 02, 08, 09 as
training set, and Sequence 03, 04, 05, 06, 07, 10 as testing set, while [30,
25] used Sequence 00 - 08 as training, and left 09, and 10 as testing set).
Some of them are even based on different evaluation metrics (e.g. [24, 48, 50]
applied the KITTI official evaluation metric, while [29, 56] applied absolute
trajectory error (ATE) as their evaluation metric). All these factors bring
difficulties to a direct and fair comparison across them. Moreover, the KITTI
dataset is relatively simple (the vehicle only moves in 2D translation) and in
small size. It is not convincing if only results on KITTI benchmark are
provided without a comprehensive evaluation in the long-term real-world
experiment. In fact, there is a growing need in creating a benchmark for a
though system evaluation covering various environments, self-motions and
dynamics.

3) Real-world deployment. Deploying deep learning models in real-world
environments is a systematic research problem. In the existing research, the
prediction accuracy is always their ‘golden rule’ to follow, while other
crucial issues are overlooked, such as whether the model structure and the
parameter number of framework is optimal. The computational and energy
consumption have to be considered on resource-constrained systems, such as
low-cost robots or VR wearable devices. The prallerization opportunities, such
as convolutional filters or other parallel neural network modules should be
exploited in order to take better use of GPUs. Examples for consideration
include in which situations the feed-back should be returned to fine-tune the
systems, how to incorporate the self-supervised models into the systems and
whether the systems allow the real-time online learning.

4) Lifelong learning. Most previous works we discussed so far have only been
validated on simple closed-form dataset, such as visual odometry and depth
predictions are performed on the KITTI dataset. However, in an open world, the
mobile agents will confront everchanging environmental factors, and moving
dynamics. This will require the DNN models to continuously and coherently
learn and adapt to the changes of the world. Moreover, new concepts and
objects will appear unexpectedly, requiring an object discovery and new
knowledge extension phase for robots.

5) New sensors: Beyond the common choice of on-board sensors, such as cameras,
IMU and LIDAR, the emerging new sensors provide an alternative to construct a
more accurate and robust multimodal system. New sensors including event
camera[250], thermo camera[251], mm-wave device[252], radio signals[253],
magnetic sensor[254], have distinct properties and data format compared to
predominant SLAM sensors such as cameras, IMU and LIDAR. Nevertheless, the
effective learning approaches to processing these unusual sensors are still
underexplored.

6) Scalability. Both the learning based localization and mapping models have
now achieved promising results on the evaluation benchmark. However, they are
restricted to some scenarios. For example, odometry estimation is always
evaluated in the city area or on the roads. Whether these techniques could be
applied to other environments, e.g. rural area or forest area is still an open
question. Moreover, existing works on scene reconstruction are restricted on
single-objects, synthetic data or room level. It is worthy exploring whether
these learning methods are capable of scaling to much more complex and large-
scale reconstruction problems.

7) Safety, reliability and interpretability. Safety and reliability are
critical to practical applications, e.g. self-driving vehicles. In these
scenarios, even a small error of pose or scene estimates will cause disasters
to the entire system. Deep neural networks have been long-critisized as
’black-box’, exacerbating the safety concerns for critical tasks. Some initial
efforts explored the interpretability on deep models [255]. For example,
uncertainty estimation[244, 245] can offer a belief metric, representing to
what extent we trust our models. In this way, the unreliable predictions (with
low uncertainty) are avoided in order to ensure the systems to stay safe and
reliable.

##  8 Conclusions

This work comprehensively overviews the area of deep learning for localization
and mapping, and provides a new taxonomy to cover the relevant existing
approaches from robotics, computer vision and machine learning communities.
Learning models are incorporated into localization and mapping systems to
connect input sensor data and target values, by automatically extracting
useful features from raw data without any human effort. Deep learning based
techniques have so far achieved the state-of-the-art performance in a variety
of tasks, from visual odometry, global localization to dense scene
reconstruction. Due to the highly expressive capacity of deep neural networks,
these models are capable of implicitly modelling the factors such as
environmental dynamics or sensor noises, that are hard to be modelled by hand,
and thus are relatively more robust in real-world applications. In addition,
high-level understanding and interaction are easy to perform for mobile agents
with the learning based framework. The fast development of deep learning
provides an alternative to solve classical localization and mapping problem in
a data-driven way, and meanwhile paves the road towards a next-generation AI
based spatial perception solution.

## Acknowledgments

This work is supported by the EPSRC Project “ACE-OPS: From Autonomy to
Cognitive assistance in Emergency OPerationS” (Grant Number: EP/S030832/1).

## References

  * [1] C. R. Fetsch, A. H. Turner, G. C. DeAngelis, and D. E. Angelaki, “Dynamic Reweighting of Visual and Vestibular Cues during Self-Motion Perception,” Journal of Neuroscience, vol. 29, no. 49, pp. 15601–15612, 2009.
  * [2] K. E. Cullen, “The Vestibular System: Multimodal Integration and Encoding of Self-motion for Motor Control,” Trends in Neurosciences, vol. 35, no. 3, pp. 185–196, 2012.
  * [3] N. Sünderhauf, O. Brock, W. Scheirer, R. Hadsell, D. Fox, J. Leitner, B. Upcroft, P. Abbeel, W. Burgard, M. Milford, and P. Corke, “The Limits and Potentials of Deep Learning for Robotics,” International Journal of Robotics Research, vol. 37, no. 4-5, pp. 405–420, 2018.
  * [4] R. Harle, “A Survey of Indoor Inertial Positioning Systems for Pedestrians,” IEEE Communications Surveys and Tutorials, vol. 15, no. 3, pp. 1281–1293, 2013.
  * [5] M. Gowda, A. Dhekne, S. Shen, R. R. Choudhury, X. Yang, L. Yang, S. Golwalkar, and A. Essanian, “Bringing IoT to Sports Analytics,” in USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2017.
  * [6] M. Wijers, A. Loveridge, D. W. Macdonald, and A. Markham, “Caracal: a versatile passive acoustic monitoring tool for wildlife research and conservation,” Bioacoustics, pp. 1–17, 2019.
  * [7] A. Dhekne, A. Chakraborty, K. Sundaresan, and S. Rangarajan, “TrackIO : Tracking First Responders Inside-Out,” in USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2019.
  * [8] A. J. Davison, “FutureMapping: The Computational Structure of Spatial AI Systems,” arXiv, 2018.
  * [9] D. Nister, O. Naroditsky, and J. Bergen, “Visual odometry,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. I–652–I–659 Vol.1, 2004.
  * [10] J. Engel, J. Sturm, and D. Cremers, “Semi-Dense Visual Odometry for a Monocular Camera,” in IEEE International Conference on Computer Vision (ICCV), pp. 1449–1456, 2013.
  * [11] C. Forster, M. Pizzoli, and D. Scaramuzza, “SVO: Fast Semi-Direct Monocular Visual Odometry,” in IEEE International Conference on Robotics and Automation (ICRA), pp. 15–22, 2014.
  * [12] M. Li and A. I. Mourikis, “High-precision, Consistent EKF-based Visual-Inertial Odometry,” The International Journal of Robotics Research, vol. 32, no. 6, pp. 690–711, 2013.
  * [13] S. Leutenegger, S. Lynen, M. Bosse, R. Siegwart, and P. Furgale, “Keyframe-Based Visual–Inertial Odometry Using Nonlinear Optimization,” The International Journal of Robotics Research, vol. 34, no. 3, pp. 314–334, 2015.
  * [14] C. Forster, L. Carlone, F. Dellaert, and D. Scaramuzza, “On-Manifold Preintegration for Real-Time Visual-Inertial Odometry,” IEEE Transactions on Robotics, vol. 33, no. 1, pp. 1–21, 2017.
  * [15] T. Qin, P. Li, and S. Shen, “VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator,” IEEE Transactions on Robotics, vol. 34, no. 4, pp. 1004–1020, 2018.
  * [16] J. Zhang and S. Singh, “LOAM: Lidar Odometry and Mapping in Real-time,” in Robotics: Science and Systems, 2010.
  * [17] W. Zhang and J. Kosecka, “Image based localization in urban environments.,” in International Symposium on 3D Data Processing Visualization and Transmission (3DPVT), vol. 6, pp. 33–40, Citeseer, 2006.
  * [18] T. Sattler, B. Leibe, and L. Kobbelt, “Fast image-based localization using direct 2d-to-3d matching,” in International Conference on Computer Vision (ICCV), pp. 667–674, IEEE, 2011.
  * [19] S. Lowry, N. Sünderhauf, P. Newman, J. J. Leonard, D. Cox, P. Corke, and M. J. Milford, “Visual place recognition: A survey,” IEEE Transactions on Robotics, vol. 32, no. 1, pp. 1–19, 2015.
  * [20] A. J. Davison, I. D. Reid, N. D. Molton, and O. Stasse, “MonoSLAM: Real-Time Single Camera SLAM,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29, no. 6, pp. 1052–1067, 2007.
  * [21] R. Mur-Artal, J. Montiel, and J. D. Tardos, “ORB-SLAM : A Versatile and Accurate Monocular SLAM System,” IEEE Transactions on Robotics, vol. 31, no. 5, pp. 1147–1163, 2015.
  * [22] H. C. Longuet-Higgins, “A computer algorithm for reconstructing a scene from two projections,” Nature, vol. 293, no. 5828, pp. 133–135, 1981.
  * [23] C. Wu, “Towards linear-time incremental structure from motion,” in 2013 International Conference on 3D Vision-3DV 2013, pp. 127–134, IEEE, 2013.
  * [24] S. Wang, R. Clark, H. Wen, and N. Trigoni, “DeepVO : Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks,” in International Conference on Robotics and Automation (ICRA), 2017.
  * [25] N. Yang, L. von Stumberg, R. Wang, and D. Cremers, “D3vo: Deep depth, deep pose and deep uncertainty for monocular visual odometry,” CVPR, 2020.
  * [26] J. McCormac, A. Handa, A. Davison, and S. Leutenegger, “Semanticfusion: Dense 3d semantic mapping with convolutional neural networks,” in 2017 IEEE International Conference on Robotics and automation (ICRA), pp. 4628–4635, IEEE, 2017.
  * [27] L. Ma, J. Stückler, C. Kerl, and D. Cremers, “Multi-view deep learning for consistent semantic mapping with rgb-d cameras,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 598–605, IEEE, 2017.
  * [28] I. Goodfellow, Y. Bengio, and A. Courville, Deep learning.  MIT press, 2016.
  * [29] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe, “Unsupervised Learning of Depth and Ego-Motion from Video,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [30] J. Bian, Z. Li, N. Wang, H. Zhan, C. Shen, M.-M. Cheng, and I. Reid, “Unsupervised scale-consistent depth and ego-motion learning from monocular video,” in Advances in Neural Information Processing Systems, pp. 35–45, 2019.
  * [31] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi, “Target-driven visual navigation in indoor scenes using deep reinforcement learning,” in 2017 IEEE international conference on robotics and automation (ICRA), pp. 3357–3364, IEEE, 2017.
  * [32] P. Mirowski, M. Grimes, M. Malinowski, K. M. Hermann, K. Anderson, D. Teplyashin, K. Simonyan, A. Zisserman, R. Hadsell, et al., “Learning to navigate in cities without a map,” in Advances in Neural Information Processing Systems, pp. 2419–2430, 2018.
  * [33] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165, 2020.
  * [34] W. Maddern, G. Pascoe, C. Linegar, and P. Newman, “1 Year, 1000km: The Oxford RobotCar Dataset,” The International Journal of Robotics Research, vol. 36, no. 1, pp. 3–15, 2016.
  * [35] P. Wang, X. Huang, X. Cheng, D. Zhou, Q. Geng, and R. Yang, “The apolloscape open dataset for autonomous driving and its application,” IEEE transactions on pattern analysis and machine intelligence, 2019.
  * [36] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, V. Vasudevan, W. Han, J. Ngiam, H. Zhao, A. Timofeev, S. Ettinger, M. Krivokon, A. Gao, A. Joshi, Y. Zhang, J. Shlens, Z. Chen, and D. Anguelov, “Scalability in perception for autonomous driving: Waymo open dataset,” 2019.
  * [37] H. Durrant-Whyte and T. Bailey, “Simultaneous localization and mapping: part i,” IEEE robotics & automation magazine, vol. 13, no. 2, pp. 99–110, 2006\.
  * [38] T. Bailey and H. Durrant-Whyte, “Simultaneous localization and mapping (slam): Part ii,” IEEE robotics & automation magazine, vol. 13, no. 3, pp. 108–117, 2006.
  * [39] C. Cadena, L. Carlone, H. Carrillo, Y. Latif, D. Scaramuzza, J. Neira, I. Reid, and J. J. Leonard, “Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age,” IEEE Transactions on robotics, vol. 32, no. 6, pp. 1309–1332, 2016.
  * [40] S. Thrun, W. Burgard, and D. Fox, Probabilistic robotics.  MIT press, 2005.
  * [41] D. Scaramuzza and F. Fraundorfer, “Visual odometry [tutorial],” IEEE robotics & automation magazine, vol. 18, no. 4, pp. 80–92, 2011.
  * [42] G. Grisetti, R. Kummerle, C. Stachniss, and W. Burgard, “A tutorial on graph-based slam,” IEEE Intelligent Transportation Systems Magazine, vol. 2, no. 4, pp. 31–43, 2010.
  * [43] M. R. U. Saputra, A. Markham, and N. Trigoni, “Visual slam and structure from motion in dynamic environments: A survey,” ACM Computing Surveys (CSUR), vol. 51, no. 2, pp. 1–36, 2018.
  * [44] K. Konda and R. Memisevic, “Learning Visual Odometry with a Convolutional Network,” in International Conference on Computer Vision Theory and Applications, pp. 486–490, 2015.
  * [45] G. Costante, M. Mancini, P. Valigi, and T. A. Ciarfuglia, “Exploring representation learning with cnns for frame-to-frame ego-motion estimation,” IEEE robotics and automation letters, vol. 1, no. 1, pp. 18–25, 2015.
  * [46] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The KITTI dataset,” The International Journal of Robotics Research, vol. 32, no. 11, pp. 1231–1237, 2013.
  * [47] A. Geiger, J. Ziegler, and C. Stiller, “Stereoscan: Dense 3d reconstruction in real-time,” in 2011 IEEE Intelligent Vehicles Symposium (IV), pp. 963–968, Ieee, 2011.
  * [48] M. R. U. Saputra, P. P. de Gusmao, S. Wang, A. Markham, and N. Trigoni, “Learning monocular visual odometry through geometry-aware curriculum learning,” in 2019 International Conference on Robotics and Automation (ICRA), pp. 3549–3555, IEEE, 2019.
  * [49] M. R. U. Saputra, P. P. de Gusmao, Y. Almalioglu, A. Markham, and N. Trigoni, “Distilling knowledge from a deep pose regressor network,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 263–272, 2019.
  * [50] F. Xue, X. Wang, S. Li, Q. Wang, J. Wang, and H. Zha, “Beyond tracking: Selecting memory and refining poses for deep visual odometry,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8575–8583, 2019.
  * [51] T. Haarnoja, A. Ajay, S. Levine, and P. Abbeel, “Backprop kf: Learning discriminative deterministic state estimators,” in Advances in Neural Information Processing Systems, pp. 4376–4384, 2016.
  * [52] X. Yin, X. Wang, X. Du, and Q. Chen, “Scale recovery for monocular visual odometry using depth estimated with deep convolutional neural fields,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 5870–5878, 2017.
  * [53] R. Li, S. Wang, Z. Long, and D. Gu, “Undeepvo: Monocular visual odometry through unsupervised deep learning,” in 2018 IEEE international conference on robotics and automation (ICRA), pp. 7286–7291, IEEE, 2018.
  * [54] D. Barnes, W. Maddern, G. Pascoe, and I. Posner, “Driven to distraction: Self-supervised distractor learning for robust monocular visual odometry in urban environments,” in 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1894–1900, IEEE, 2018.
  * [55] Z. Yin and J. Shi, “GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [56] H. Zhan, R. Garg, C. S. Weerasekera, K. Li, H. Agarwal, and I. Reid, “Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 340–349, 2018.
  * [57] R. Jonschkowski, D. Rastogi, and O. Brock, “Differentiable particle filters: End-to-end learning with algorithmic priors,” Robotics: Science and Systems, 2018.
  * [58] N. Yang, R. Wang, J. Stuckler, and D. Cremers, “Deep virtual stereo odometry: Leveraging deep depth prediction for monocular direct sparse odometry,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 817–833, 2018.
  * [59] C. Zhao, L. Sun, P. Purkait, T. Duckett, and R. Stolkin, “Learning monocular visual odometry with dense 3d mapping from dense 3d flow,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6864–6871, IEEE, 2018.
  * [60] V. Casser, S. Pirk, R. Mahjourian, and A. Angelova, “Depth prediction without the sensors: Leveraging structure for unsupervised learning from monocular videos,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 8001–8008, 2019.
  * [61] Y. Almalioglu, M. R. U. Saputra, P. P. de Gusmao, A. Markham, and N. Trigoni, “Ganvo: Unsupervised deep monocular visual odometry and depth estimation with generative adversarial networks,” in 2019 International Conference on Robotics and Automation (ICRA), pp. 5474–5480, IEEE, 2019.
  * [62] S. Y. Loo, A. J. Amiri, S. Mashohor, S. H. Tang, and H. Zhang, “Cnn-svo: Improving the mapping in semi-direct visual odometry using single-image depth prediction,” in 2019 International Conference on Robotics and Automation (ICRA), pp. 5218–5223, IEEE, 2019.
  * [63] R. Wang, S. M. Pizer, and J.-M. Frahm, “Recurrent neural network for (un-) supervised learning of monocular video visual odometry and depth,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5555–5564, 2019.
  * [64] Y. Li, Y. Ushiku, and T. Harada, “Pose graph optimization for unsupervised monocular visual odometry,” in 2019 International Conference on Robotics and Automation (ICRA), pp. 5439–5445, IEEE, 2019.
  * [65] A. Gordon, H. Li, R. Jonschkowski, and A. Angelova, “Depth from videos in the wild: Unsupervised monocular depth learning from unknown cameras,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 8977–8986, 2019.
  * [66] A. S. Koumis, J. A. Preiss, and G. S. Sukhatme, “Estimating metric scale visual odometry from videos using 3d convolutional networks,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 265–272, IEEE, 2019.
  * [67] H. Zhan, C. S. Weerasekera, J. Bian, and I. Reid, “Visual odometry revisited: What should be learnt?,” The International Conference on Robotics and Automation (ICRA), 2020.
  * [68] R. Clark, S. Wang, H. Wen, A. Markham, and N. Trigoni, “VINet : Visual-Inertial Odometry as a Sequence-to-Sequence Learning Problem,” in The AAAI Conference on Artificial Intelligence (AAAI), pp. 3995–4001, 2017\.
  * [69] E. J. Shamwell, K. Lindgren, S. Leung, and W. D. Nothwang, “Unsupervised deep visual-inertial odometry with online error correction for rgb-d imagery,” IEEE transactions on pattern analysis and machine intelligence, 2019.
  * [70] C. Chen, S. Rosa, Y. Miao, C. X. Lu, W. Wu, A. Markham, and N. Trigoni, “Selective sensor fusion for neural visual-inertial odometry,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10542–10551, 2019.
  * [71] L. Han, Y. Lin, G. Du, and S. Lian, “Deepvio: Self-supervised deep learning of monocular visual inertial odometry using 3d geometric constraints,” arXiv preprint arXiv:1906.11435, 2019.
  * [72] M. Velas, M. Spanel, M. Hradis, and A. Herout, “Cnn for imu assisted odometry estimation using velodyne lidar,” in 2018 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), pp. 71–77, IEEE, 2018\.
  * [73] Q. Li, S. Chen, C. Wang, X. Li, C. Wen, M. Cheng, and J. Li, “Lo-net: Deep real-time lidar odometry,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8473–8482, 2019.
  * [74] W. Wang, M. R. U. Saputra, P. Zhao, P. Gusmao, B. Yang, C. Chen, A. Markham, and N. Trigoni, “Deeppco: End-to-end point cloud odometry through deep parallel neural network,” The 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2019), 2019.
  * [75] M. Valente, C. Joly, and A. de La Fortelle, “Deep sensor fusion for real-time odometry estimation,” 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019.
  * [76] S. Li, F. Xue, X. Wang, Z. Yan, and H. Zha, “Sequential adversarial learning for self-supervised deep visual odometry,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2851–2860, 2019.
  * [77] L. Sheng, D. Xu, W. Ouyang, and X. Wang, “Unsupervised collaborative learning of keyframe detection and visual odometry towards monocular deep slam,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 4302–4311, 2019.
  * [78] D. Eigen, C. Puhrsch, and R. Fergus, “Depth map prediction from a single image using a multi-scale deep network,” in Advances in neural information processing systems, pp. 2366–2374, 2014.
  * [79] B. Ummenhofer, H. Zhou, J. Uhrig, N. Mayer, E. Ilg, A. Dosovitskiy, and T. Brox, “Demon: Depth and motion network for learning monocular stereo,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5038–5047, 2017.
  * [80] R. Garg, V. K. BG, G. Carneiro, and I. Reid, “Unsupervised cnn for single view depth estimation: Geometry to the rescue,” in European Conference on Computer Vision, pp. 740–756, Springer, 2016.
  * [81] C. Godard, O. Mac Aodha, and G. J. Brostow, “Unsupervised monocular depth estimation with left-right consistency,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 270–279, 2017.
  * [82] T. Haarnoja, A. Ajay, S. Levine, and P. Abbeel, “Backprop KF: Learning Discriminative Deterministic State Estimators,” in Advances In Neural Information Processing Systems (NeurIPS), 2016.
  * [83] R. Jonschkowski, D. Rastogi, and O. Brock, “Differentiable Particle Filters: End-to-End Learning with Algorithmic Priors,” in Robotics: Science and Systems, 2018.
  * [84] L. Von Stumberg, V. Usenko, and D. Cremers, “Direct sparse visual-inertial odometry using dynamic marginalization,” in 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 2510–2517, IEEE, 2018.
  * [85] C. Chen, X. Lu, A. Markham, and N. Trigoni, “Ionet: Learning to cure the curse of drift in inertial odometry,” in Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
  * [86] C. Chen, Y. Miao, C. X. Lu, L. Xie, P. Blunsom, A. Markham, and N. Trigoni, “Motiontransformer: Transferring neural inertial tracking between domains,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 8009–8016, 2019.
  * [87] M. A. Esfahani, H. Wang, K. Wu, and S. Yuan, “Aboldeepio: A novel deep inertial odometry network for autonomous vehicles,” IEEE Transactions on Intelligent Transportation Systems, 2019.
  * [88] H. Yan, Q. Shan, and Y. Furukawa, “Ridi: Robust imu double integration,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 621–636, 2018.
  * [89] S. Cortés, A. Solin, and J. Kannala, “Deep learning based speed estimation for constraining strapdown inertial navigation on smartphones,” in 2018 IEEE 28th International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1–6, IEEE, 2018.
  * [90] B. Wagstaff and J. Kelly, “Lstm-based zero-velocity detection for robust inertial navigation,” in 2018 International Conference on Indoor Positioning and Indoor Navigation (IPIN), pp. 1–8, IEEE, 2018.
  * [91] M. Brossard, A. Barrau, and S. Bonnabel, “Rins-w: Robust inertial navigation system on wheels,” 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019.
  * [92] F. Liu, C. Shen, G. Lin, and I. Reid, “Learning depth from single monocular images using deep convolutional neural fields,” IEEE transactions on pattern analysis and machine intelligence, vol. 38, no. 10, pp. 2024–2039, 2015\.
  * [93] C. Wang, J. Miguel Buenaposada, R. Zhu, and S. Lucey, “Learning depth from monocular videos using direct methods,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2022–2030, 2018.
  * [94] M. Ji, J. Gall, H. Zheng, Y. Liu, and L. Fang, “Surfacenet: An end-to-end 3d neural network for multiview stereopsis,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 2307–2315, 2017.
  * [95] D. Paschalidou, O. Ulusoy, C. Schmitt, L. Van Gool, and A. Geiger, “Raynet: Learning volumetric 3d reconstruction with ray potentials,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3897–3906, 2018.
  * [96] A. Kar, C. Häne, and J. Malik, “Learning a multi-view stereo machine,” in Advances in neural information processing systems, pp. 365–376, 2017.
  * [97] M. Tatarchenko, A. Dosovitskiy, and T. Brox, “Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 2088–2096, 2017.
  * [98] C. Häne, S. Tulsiani, and J. Malik, “Hierarchical surface prediction for 3d object reconstruction,” in 2017 International Conference on 3D Vision (3DV), pp. 412–420, IEEE, 2017.
  * [99] A. Dai, C. Ruizhongtai Qi, and M. Nießner, “Shape completion using 3d-encoder-predictor cnns and shape synthesis,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5868–5877, 2017\.
  * [100] G. Riegler, A. O. Ulusoy, H. Bischof, and A. Geiger, “Octnetfusion: Learning depth fusion from data,” in 2017 International Conference on 3D Vision (3DV), pp. 57–66, IEEE, 2017.
  * [101] H. Fan, H. Su, and L. J. Guibas, “A point set generation network for 3d object reconstruction from a single image,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 605–613, 2017.
  * [102] T. Groueix, M. Fisher, V. G. Kim, B. C. Russell, and M. Aubry, “A papier-mâché approach to learning 3d surface generation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 216–224, 2018.
  * [103] N. Wang, Y. Zhang, Z. Li, Y. Fu, W. Liu, and Y.-G. Jiang, “Pixel2mesh: Generating 3d mesh models from single rgb images,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 52–67, 2018.
  * [104] L. Ladicky, O. Saurer, S. Jeong, F. Maninchedda, and M. Pollefeys, “From point clouds to mesh using regression,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 3893–3902, 2017.
  * [105] A. Dai and M. Nießner, “Scan2mesh: From unstructured range scans to 3d meshes,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5574–5583, 2019.
  * [106] T. Mukasa, J. Xu, and B. Stenger, “3d scene mesh from cnn depth predictions and sparse monocular slam,” in Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 921–928, 2017.
  * [107] M. Bloesch, T. Laidlow, R. Clark, S. Leutenegger, and A. J. Davison, “Learning meshes for dense visual slam,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 5855–5864, 2019.
  * [108] Y. Xiang and D. Fox, “Da-rnn: Semantic mapping with data associated recurrent neural networks,” Robotics: Science and Systems, 2017.
  * [109] N. Sünderhauf, T. T. Pham, Y. Latif, M. Milford, and I. Reid, “Meaningful maps with object-oriented semantic mapping,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5079–5085, IEEE, 2017.
  * [110] J. McCormac, R. Clark, M. Bloesch, A. Davison, and S. Leutenegger, “Fusion++: Volumetric object-level slam,” in 2018 international conference on 3D vision (3DV), pp. 32–41, IEEE, 2018.
  * [111] M. Grinvald, F. Furrer, T. Novkovic, J. J. Chung, C. Cadena, R. Siegwart, and J. Nieto, “Volumetric instance-aware semantic mapping and 3d object discovery,” IEEE Robotics and Automation Letters, vol. 4, no. 3, pp. 3037–3044, 2019.
  * [112] G. Narita, T. Seno, T. Ishikawa, and Y. Kaji, “Panopticfusion: Online volumetric semantic mapping at the level of stuff and things,” IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019.
  * [113] M. Bloesch, J. Czarnowski, R. Clark, S. Leutenegger, and A. J. Davison, “CodeSLAM — Learning a Compact, Optimisable Representation for Dense Visual SLAM,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [114] J. Tobin, W. Zaremba, and P. Abbeel, “Geometry-aware neural rendering,” in Advances in Neural Information Processing Systems, pp. 11555–11565, 2019\.
  * [115] J. H. Lim, P. O. Pinheiro, N. Rostamzadeh, C. Pal, and S. Ahn, “Neural multisensory scene inference,” in Advances in Neural Information Processing Systems, pp. 8994–9004, 2019.
  * [116] V. Sitzmann, M. Zollhöfer, and G. Wetzstein, “Scene representation networks: Continuous 3d-structure-aware neural scene representations,” in Advances in Neural Information Processing Systems, pp. 1119–1130, 2019\.
  * [117] M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu, “Reinforcement learning with unsupervised auxiliary tasks,” ICLR, 2017.
  * [118] P. Mirowski, R. Pascanu, F. Viola, H. Soyer, A. J. Ballard, A. Banino, M. Denil, R. Goroshin, L. Sifre, K. Kavukcuoglu, et al., “Learning to navigate in complex environments,” ICLR, 2017.
  * [119] C. Kerl, J. Sturm, and D. Cremers, “Dense visual slam for rgb-d cameras,” in 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2100–2106, IEEE, 2013.
  * [120] T. Whelan, M. Kaess, H. Johannsson, M. Fallon, J. J. Leonard, and J. McDonald, “Real-time large-scale dense rgb-d slam with volumetric fusion,” The International Journal of Robotics Research, vol. 34, no. 4-5, pp. 598–626, 2015\.
  * [121] R. A. Newcombe, S. J. Lovegrove, and A. J. Davison, “DTAM : Dense Tracking and Mapping in Real-Time,” in IEEE International Conference on Computer Vision (ICCV), pp. 2320–2327, 2011.
  * [122] K. Karsch, C. Liu, and S. B. Kang, “Depth transfer: Depth extraction from video using non-parametric sampling,” IEEE transactions on pattern analysis and machine intelligence, vol. 36, no. 11, pp. 2144–2158, 2014.
  * [123] K. Tateno, F. Tombari, I. Laina, and N. Navab, “Cnn-slam: Real-time dense monocular slam with learned depth prediction,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6243–6252, 2017\.
  * [124] J. Engel, T. Schöps, and D. Cremers, “Lsd-slam: Large-scale direct monocular slam,” in European conference on computer vision, pp. 834–849, Springer, 2014.
  * [125] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652–660, 2017\.
  * [126] A. Kirillov, K. He, R. Girshick, C. Rother, and P. Dollár, “Panoptic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9404–9413, 2019.
  * [127] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohi, J. Shotton, S. Hodges, and A. Fitzgibbon, “Kinectfusion: Real-time dense surface mapping and tracking,” in 2011 10th IEEE International Symposium on Mixed and Augmented Reality, pp. 127–136, IEEE, 2011.
  * [128] S. A. Eslami, D. J. Rezende, F. Besse, F. Viola, A. S. Morcos, M. Garnelo, A. Ruderman, A. A. Rusu, I. Danihelka, K. Gregor, et al., “Neural scene representation and rendering,” Science, vol. 360, no. 6394, pp. 1204–1210, 2018.
  * [129] V. Balntas, S. Li, and V. Prisacariu, “Relocnet: Continuous metric learning relocalisation using neural nets,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 751–767, 2018.