# VIRAL SLAM: Tightly Coupled Camera-IMU-UWB-Lidar SLAM

Thien-Minh Nguyen, , Shenghai Yuan, Muqing Cao,
Thien Hoang Nguyen, and Lihua Xie This work was supported by the Wallenberg
AI, Autonomous Systems and Software Program (WASP) funded by the Knut and
Alice Wallenberg Foundation, under the Grant Call 10013 - Wallenberg-NTU
Presidential Postdoctoral Fellowship 2020. (Corresponding Author: Thien-Minh
Nguyen)1The authors are with School of Electrical and Electronic Engineering,
Nanyang Technological University, Singapore 639798, 50 Nanyang Avenue.
(e-mail: {thienminh.nguyen@, shyuan@, mqcao<EMAIL_ADDRESS>elhxie@}ntu.edu.sg).

###### Abstract

In this paper, we propose a tightly-coupled, multi-modal simultaneous
localization and mapping (SLAM) framework, integrating an extensive set of
sensors: IMU, cameras, multiple lidars, and Ultra-wideband (UWB) range
measurements, hence referred to as VIRAL (visual-inertial-ranging-lidar) SLAM.
To achieve such a comprehensive sensor fusion system, one has to tackle
several challenges such as data synchronization, multi-threading programming,
bundle adjustment (BA), and conflicting coordinate frames between UWB and the
onboard sensors, so as to ensure real-time localization and smooth updates in
the state estimates. To this end, we propose a two stage approach. In the
first stage, lidar, camera, and IMU data on a local sliding window are
processed in a core odometry thread. From this local graph, new key frames are
evaluated for admission to a global map. Visual feature-based loop closure is
also performed to supplement the global factor graph with loop constraints.
When the global factor graph satisfies a condition on spatial diversity, the
BA process will be triggered to update the coordinate transform between UWB
and onboard SLAM systems. The system then seamlessly transitions to the second
stage where all sensors are tightly integrated in the odometry thread. The
capability of our system is demonstrated via several experiments on high-
fidelity graphical-physical simulation and public datasets.

##  I Introduction

Localization is arguably one of the most important capabilities for mobile
robots, especially for Unmanned Aerial Vehicles (UAVs). Obviously, a common
approach to ensure reliable and accurate localization is to combine multiple
sensors for their complementary advantages as well as redundancy. For example,
since lidar is not affected by lighting condition or lack of visual features,
which can easily destabilize most visual-inertial-odometry (VIO) systems, the
robot can still rely on this type of sensor for localization in low light or
low-texture conditions. In addition, camera can also enable loop closure
capability, while lidar pointcloud map can help augment the visual feature’s
depth estimation process [1, 2]. This is one of the main advantages that
motivates us to develop a tightly coupled camera-lidar-IMU-based Simultaneous
Localization and Mapping (SLAM) system in this paper.

\begin{overpic}[width=433.62pt]{hardware.png} \end{overpic} Figure 1: Hardware
setup of the VIRAL SLAM system on a UAV: a hardware-synchronized stereo camera
rig, a 400 Hz IMU, two 16-channel lidars, four body-offset UWB ranging nodes
and a crystal prism that is tracked by a Leica total station for millimeter-
accuracy groundtruth.

Besides the aforementioned benefits of a visual-lidar localization system,
integration of Ultra-wideband (UWB) into the SLAM system can also provide
another layer of backup in case both lidar and camera lose track, and also
allows user to obtain global localization information relative to the
inspected object [3, 4]. However, to successfully integrate UWB with SLAM,
especially in the real-time localization process, one must first estimate the
coordinates of the anchors in the SLAM coordinate frame L, which is the
methodology used in previous works [5, 6]. In this paper, we propose a new
approach. Specifically, using the distance measurements between the anchors,
we can set up a nominal coordinate of the anchors, which effectively defines a
preferred frame W that aligns with the mission to be conducted in the
environment (more details in Sec. III-A), and then further refine the
transform between L and W in the BA process. Subsequently, the anchors’
coordinates in W can be converted to L and used for constructing the range-
based factors in the optimization process over the local sliding window. The
separation of estimating the anchor coordinates and estimating the robot
states is a deliberate choice to ensure convergence, especially in the case
when the movement in the sliding window is too short, which lacks excitation
for convergence of the anchor position estimates. It is also noted that we
focus on a simple yet effective UWB network of two or three anchors, with
multiple body-offset ranging nodes in the UAV. This simple network allows
relatively accurate initialization of the robot and anchor position in W,
which facilitates accurate and seamless integration of UWB into the SLAM
system.

The contribution of our work can be stated as follows:

  * •

We propose a comprehensive SLAM framework that tightly integrates multiple
sensors of different sensing modalities, i.e. lidars, cameras, IMU and UWB
ranging sensors, in a seamless manner.

  * •

We propose a map-matching marginalization (MMM) scheme for visual features
using the local map constructed from lidar pointclouds.

  * •

We devise a loop closure scheme that is triggered by visual place recognition
and further refined via a two-stage pointcloud alignment.

  * •

We propose a novel scheme to fuse UWB, where estimation of the anchor position
and ranging bias is delegated to the bundle adjustment (BA) thread, and their
values are fixed in the local sliding window optimization for the fusion of
UWB range.

  * •

We conduct extensive experiments to validate VIRAL SLAM and compare it with
other state-of-the-art methods in a variety of real-world scenarios and high-
fidelity simulations.

##  II Related Works

To the best of our knowledge, our work features a tightly coupled SLAM system
that integrates one of the most comprehensive sensor suites. While
localization methods based on mainly camera or lidar (with or without IMU) are
abundant, only a handful of works have investigated tightly coupled visual and
lidar information in the literature. In [1], Zhang et al proposed a method
where VIO and lidar data were employed in a cascaded manner. In this
framework, high rate VIO data is used to help initialize the scan-matching
process, while pointcloud map can be used to help retrieve the visual
feature’s depth. On the other hand, in [7, 8] Zuo et al proposed an MSCKF
framework to asynchronously update the robot states when lidar and visual
features are obtained, and IMU is used for propagation in between of these
states. In [9], stereo camera, IMU, lidar were fused together on a unified
framework, synchronized with camera data. We note that loop closure and BA are
not considered in the aforementioned works (as opposed to our proposed VIRAL
SLAM), therefore drift is still an intrinsic problem in these approaches.

To address the drift issue, in [10], Graeter et al studied the problem of
estimating the scale of a pose graph obtained from monocular visual SLAM with
lidar information. In [11], Shao et al considered an approach similar to [1],
but also used camera and ICP for loop closure. However they require hardware-
synchronized camera-lidar messages, which can only produce very low rate data.
In [2], Shan et al proposed a loose integration of VIO output from VINS-Mono
[12] to LIO-SAM [13], which itself loosely integrates LeGO-LOAM output [14]
with IMU preintegration, in a gtsam pose-graph optimization framework. This
approach can be unreliable as the whole chain depends on whether the core
lidar-based process runs well. If there is a low-texture case when lidar
localization is unstable, its error can reverberate up the chain, which
appears to be the case in some of our experiments. We also note that the
aforementioned works [1, 10, 11, 2, 13, 14] do not consider the integration of
multiple lidars like VIRAL SLAM. On the other hand, while the MLOAM and BLS
methods [15, 16] did address this issue, they focused purely on lidar and no
camera and IMU is involved. In [17], we proposed a multi-input lidar-inertia
odometry and mapping scheme called MILIOM, which clearly demonstrates the
robustness, accuracy, and real-time performance. VIRAL SLAM system is
developed based on this lidar-based system.

Another trend in the literature is the use of UWB to aid VIO or SLAM process.
For example, UWB has been integrated with monocular VIO for drift correction
[5, 6], or can be used as a variable baseline for cameras on different UAVs
[18]. In recent years, VIO, UWB and lidar have also been used in a loosely
coupled manner for relative localization [19, 20, 21, 22, 23, 24]. In our
previous work [4], a tightly-coupled fusion of body-offset range measurement
with IMU preintegration and pose displacement derived from LOAM/VINS
subsystems was proposed. We showed that the system can achieve better accuracy
compared to traditional onboard self-localization methods. Based on this work,
tight coupling of UWB with lidar and IMU preintegration factors was
investigated in [3]. However, this preliminary work is still restrictive in
that the lidar processing pipeline inherited from LIO-Mapping [25] was quite
inefficient, and no visual information was considered.

The remainder of the paper is organized as follows: in Sec. III, we lay out
some basic definitions and our general approach towards synchronization. Sec.
IV then presents the main function blocks that process the sensor data for the
local sliding window optimization, while Sec. V goes into detail of the global
map management, which includes loop closure and BA processes. We demonstrate
the capability of our method via several experiments on public and simulated
datasets in Sec. VI. Finally, Sec. VII concludes our work.

##  III Preliminaries

###  III-A Coordinate frames

In this paper, we define a so-called local frame L whose origin coincides with
the position of the body frame at the initial time, and the $z$ axis points to
the opposite direction of gravity, and the robot’s initial yaw angle is zero.
In addition to L, we fix the coordinates of the anchors, which defines another
so-called world frame W. Since three anchors reside on a plane, the 3D
coordinates of these anchors in W can be determined via the distances between
the anchors, plus a nominal height $z^{*}$. Fig. 2 describes our coordinate
systems in more details.

\begin{overpic}[width=433.62pt]{ranging_and_coordinates.png}
\par\put(0.0,50.0){\footnotesize$a_{2}$} \put(90.0,50.0){\footnotesize$a_{0}$}
\put(51.0,20.0){\footnotesize$a_{1}$}
\par\put(-2.0,56.0){\footnotesize$(x_{2},y_{2},z^{*})\triangleq(\frac{r_{01}^{2}-r_{12}^{2}+r_{02}^{2}}{2r_{01}},\pm\sqrt{r_{02}^{2}-x_{2}^{2}},z^{*})$}
\put(82.0,56.0){\footnotesize$(0,0,z^{*})$}
\put(45.0,0.0){\footnotesize$(r_{01},0,z^{*})$}
\par\put(75.0,54.5){\footnotesize$r_{02}$}
\put(20.0,42.0){\footnotesize$r_{12}$} \put(74.0,44.0){\footnotesize$r_{01}$}
\par\put(52.0,48.0){\footnotesize$d_{k}^{0}$}
\put(20.0,48.0){\footnotesize$d_{k}^{2}$}
\par\put(8.0,36.0){\footnotesize$d_{k+1}^{2}$}
\put(34.0,17.5){\footnotesize$d_{k+1}^{1}$}
\par\put(58.0,33.0){\footnotesize$\texttt{L}$}
\put(90.0,44.0){\footnotesize$\texttt{W}$}
\par\put(58.0,14.0){\footnotesize$\vv{g}$}
\par\put(37.0,34.5){\footnotesize${}_{\texttt{B}_{k}}^{\texttt{L}}\mathbf{T}$}
\put(40.0,29.0){\footnotesize${}_{\texttt{B}_{k+1}}^{\texttt{L}}\mathbf{T}$}
\put(75.0,33.0){\footnotesize${}_{\texttt{W}}^{\texttt{L}}\mathbf{T}=?$}
\end{overpic} Figure 2: A so-called world frame W can be defined when fixing
the coordinates of the anchor nodes using the anchor-to-anchor distances. On
the other hand, the SLAM system takes reference to a local coordinate frame L
that coincides with the initial key frame’s pose. To successfully combine UWB
with SLAM, the transform ${}^{\texttt{L}}_{\texttt{W}}\mathbf{T}$ needs to be
resolved.

###  III-B State estimates

At each time step $t_{k}$, we define a sliding window $\hat{\mathcal{T}}_{k}$
consisting of the robot’s state estimates over the last $M$ time steps as
follows:

$\displaystyle\hat{\mathcal{T}}_{k}$
$\displaystyle=\left(\hat{\mathcal{X}}_{w},\hat{\mathcal{X}}_{w+1},\dots,\hat{\mathcal{X}}_{k}\right),w\triangleq
k-M+1.$ (1) $\displaystyle\hat{\mathcal{X}}_{k}$
$\displaystyle=\Big{(}\hat{\mathbf{q}}_{k},\hat{\mathbf{p}}_{k},\hat{\mathbf{v}}_{k},\hat{\mathbf{b}}^{\omega}_{k},\hat{\mathbf{b}}^{a}_{k}\Big{)}\in\mathrm{SO(3)}\times\mathbb{R}^{12},$
(2)

where $\hat{\mathbf{q}}_{k}$, $\hat{\mathbf{p}}_{k}$, $\hat{\mathbf{v}}_{k}$
are respectively the orientation quaternion, position and velocity state
estimates w.r.t. the local frame L at time $t_{k}$;
$\hat{\mathbf{b}}^{a}_{k},\hat{\mathbf{b}}^{\omega}_{k}$ are respectively the
IMU accelerometer and gyroscope biases.

Besides, we also define the states for the inverse depth of
$N_{\mathcal{V}}^{k}$ visual features being tracked on the sliding window as:

$\hat{\lambda}^{1},\hat{\lambda}^{2},\dots\hat{\lambda}^{N_{\mathcal{V}}^{k}},\
\hat{\Lambda}_{k}\triangleq(\hat{\lambda}^{1},\hat{\lambda}^{2},\dots\hat{\lambda}^{N_{\mathcal{V}}^{k}})\in\mathbb{R}^{N_{\mathcal{V}}^{k}}$
(3)

####  III-B1 Global pose graph and UWB parameters

A global pose graph is developed with marginalized key frames from the sliding
window estimation process. For each key frame $i$ stored in the memory, we
define its pose estimate as ${}^{\texttt{L}}\hat{\mathbf{T}}_{i}$. The pose
estimates will be updated in the BA process whenever a certain number of new
key frames are admitted, or a loop factor is obtained. Besides the key frame
pose, we also seek to estimate the following UWB-related parameters:

${}^{\texttt{L}}_{\texttt{W}}\mathbf{T}=({}^{\texttt{L}}_{\texttt{W}}\mathbf{R},{}^{\texttt{L}}_{\texttt{W}}\mathbf{p}),\
{}^{\texttt{L}}_{\texttt{W}}\mathbf{R}\in\mathrm{SO(3)},\
{}^{\texttt{L}}_{\texttt{W}}\mathbf{p}\in\mathbb{R}^{3};\
\mathbf{b}^{r}\in\mathbb{R},$ (4)

where ${}^{\texttt{L}}_{\texttt{W}}\mathbf{T}$ is the coordinate transform
between the local and world frames that were introduced in Sec. III-A, and
$\mathbf{b}^{r}$ is the ranging bias that is present in our problem due to the
use of extension cables to place the UWB ranging nodes at different points on
the robot [26]. Taking VIO as an analogy,
${}^{\texttt{L}}_{\texttt{W}}\mathbf{T}$ and $\mathbf{b}^{r}$ are similar to
the extrinsic and intrinsic parameters of the UWB ranging and communication
network.

###  III-C Synchronization

In this work, our synchronization scheme is an combination of previous schemes
for multiple lidars with IMU [17] and lidar with IMU and UWB data [3], with
stereo-images being the new addition. Fig. 3 is an illustration of our
synchronization scheme.

\begin{overpic}[width=433.62pt]{sync.png} \end{overpic} Figure 3:
Synchronization among the sensors. The light blue circles represent the
interpolated IMU samples.

Briefly speaking, one lidar is arbitrarily chosen as the primary whose
timestamps are used to determine the sliding window’s time steps, and other
lidars’ inputs are merged into the primary lidar’s, yielding a combined
feature cloud (CFC) as a single sensor input. IMU data are associated with the
time steps for propagation and preintegration. For UWB samples, they are
grouped into ”bundles” based on the intervals between the time steps. Here we
denote the timestamp of a UWB sample as $\tau^{i}_{k}$, which implies that
$\tau^{i}_{k}\in(t_{k-1},\ t_{k}]$. Knowing this will allow us to associate
the sample with the correct state in the construction of the cost factor in
the later part.

For the cameras, they are triggered by an external hardware apparatus, thus
their images can be synchronized into pairs before further synchronized with
the lidar CFCs. For each time step, we admit the image pair that is closest in
time to it, and measure the time delay $t^{d}_{k}$. This time delay will be
used to compensate for the visual feature’s pixel coordinate when they are
tracked in the image plane.

Fig. 3 is a snapshot of the sliding window at time $t$, where all of the
sensor data needed for constructing the cost function in the local sliding
window optimization block are available. After the optimization process
elapses, we can obtain optimized states
$\hat{\mathcal{X}}_{w},\hat{\mathcal{X}}_{w+1},\dots\hat{\mathcal{X}}_{k}$ and
nominate one of them as a key frame candidate $\mathcal{K}$ to the global map.
This is the snapshot of the system as shown in Fig. 4.

###  III-D System overview

Fig. 4 presents the main function blocks of our VIRAL SLAM system. Most
expansive of all is the real-time localization thread, where all sensor data
are synchronized and processed to eventually create factors in a cost function
that is optimized using the ceres solver[27]. Besides this time-critical
thread, another thread runs in the background to manage the key frames, detect
loop closure, and BA optimization. The details of these blocks will be
described in the next sections.

\begin{overpic}[width=433.62pt]{overview.png}
\put(95.0,63.5){\footnotesize$\hat{\mathcal{X}}_{t}$}
\put(95.0,34.5){\footnotesize$\hat{\mathcal{T}}_{k}$}
\put(50.0,23.0){\footnotesize${}^{\mathcal{M}}$}
\put(40.0,34.0){\footnotesize$\mathcal{K}$}
\put(38.0,39.5){\footnotesize$\\{{}^{\texttt{W}}\mathcal{F}_{w},\dots,{}^{\texttt{W}}\mathcal{F}_{k}\\}$}
\put(64.0,39.0){\footnotesize$\\{\mathcal{I}_{m}\\}$}
\put(80.0,39.0){\footnotesize$\\{\mathcal{U}^{i}_{m}\\}$ }
\put(71.0,24.0){\footnotesize$\\{\mathcal{L}^{i}_{m}\\}$}
\put(71.0,14.0){\footnotesize${}^{\texttt{L}}_{\texttt{W}}\hat{\mathbf{T}}$,
$\mathbf{b}^{r}$} \put(71.0,7.0){\footnotesize$\\{\mathcal{V}^{i}_{m}\\},$}
\put(81.0,7.0){\footnotesize$\\{\bar{\lambda}^{i}_{m}\\}$} \end{overpic}
Figure 4: Overview of the VIRAL SLAM system. The colors are used to
distinguish those lines that intersect each other but do not connect.

##  IV Real-Time Localization Function Blocks

###  IV-A Local sliding window optimization

To estimate the states on the local sliding window, we seek to construct and
optimize the following cost function:

$\displaystyle
f(\hat{\mathcal{T}}_{k},\hat{\Lambda}_{k})\triangleq\Bigg{\\{}\sum_{m=w+1}^{k}\left\lVert\bm{r}_{\mathcal{I}}(\hat{\mathcal{X}}_{m-1},\hat{\mathcal{X}}_{m},\mathcal{I}_{m})\right\rVert^{2}_{{\bm{\mathbf{P}}}_{\mathcal{I}_{m}}^{-1}}$
$\displaystyle+\sum_{m=w}^{k}\sum_{i=1}^{N_{\mathcal{L}}^{m}}\rho_{H}\left(\left\lVert\bm{r}_{\mathcal{L}}(\hat{\mathcal{X}}_{m},\mathcal{L}_{m}^{i})\right\rVert^{2}_{{\bm{\mathbf{P}}}_{\mathcal{L}_{m}^{i}}^{-1}}\right)$
$\displaystyle+\sum_{m=w}^{k}\sum_{i=1}^{N_{\mathcal{U}}^{m}}\left\lVert\bm{r}_{\mathcal{U}}(\hat{\mathcal{X}}_{m-1},\hat{\mathcal{X}}_{m},{}^{\texttt{L}}_{\texttt{W}}\hat{\mathbf{T}},\hat{\mathbf{b}}^{r},\mathcal{U}^{i}_{m})\right\rVert_{{\bm{\mathbf{P}}}_{\mathcal{U}_{m}^{i}}^{-1}}$
$\displaystyle+\sum_{i=1}^{N_{\mathcal{V}}^{k}}\sum_{b\in\mathcal{C}^{i}}\rho_{A}\left(\left\lVert\bm{r}_{\mathcal{V}}(\hat{\mathcal{X}}_{m_{a}},\hat{\mathcal{X}}_{m_{b}},\tilde{\lambda}^{i},\mathcal{V}^{i}_{ab})\right\rVert^{2}_{{\bm{\mathbf{P}}}_{\mathcal{V}^{i}_{ab}}^{-1}}\right)\Bigg{\\}},$
(5)

where $\rho_{H}(\cdot)$ and $\rho_{A}(\cdot)$ are the Huber and arctan loss
functions used to reduce the effects of outliers; $\mathcal{I}_{m}$,
$\mathcal{L}^{i}_{m}$, $\mathcal{U}^{i}_{m}$, $\mathcal{V}^{i}_{ab}$ are the
elementry observations from IMU, Lidar, UWB and visual feature, respectively;
$N_{\mathcal{L}}^{m}\in\mathbb{N}$ is the number of feature-map matching FMM
coefficients extracted from the CFC $\mathcal{F}_{m}$,
$N_{\mathcal{U}}^{m}\in\mathbb{N}$ is the number of UWB samples obtained in
the interval $(t_{m-1},t_{m}]$, $N_{\mathcal{V}}^{k}\in\mathbb{N}$ is the
number of visual features that are tracked on the sliding window from $t_{w}$
to $t_{k}$, and $\mathcal{C}^{i}$ refers to the set of cameras that observe
the visual feature ${\bm{\mathbf{f}}}^{i}$, excluding $\texttt{C}_{a}$,
$\tilde{\lambda}^{i}$ can be either the state estimate $\hat{\lambda}^{i}$ or
the marginalized inverse depth $\bar{\lambda}^{i}$ of the MMM features. The
cost function (5) summarizes the coupling of each sensor’s factor with the
state estimate. We will elaborate on how to construct these factors in the
next sections.

###  IV-B Sensor data processing

####  IV-B1 Lidar & IMU

We refer to our previous work [17] for the details on how to construct the
lidar and IMU factors from the sensor data.

####  IV-B2 UWB

Similar to [3], we define each UWB sample as
$\mathcal{U}^{i}_{m}=\left(\breve{d}^{i},{}^{\texttt{W}}{\bm{\mathbf{x}}}^{i},{\bm{\mathbf{y}}}^{i},\tau^{i}_{m},t_{m-1},t_{m}\right)$,
where $\breve{d}^{i}$ is the range measurement,
${}^{\texttt{W}}{\bm{\mathbf{x}}}^{i}$ is the coordinate of the anchor w.r.t.
W, ${\bm{\mathbf{y}}}^{i}$ is the UAV ranging node in the body frame
$\texttt{B}_{\tau^{i}_{k}}$, $\tau^{i}_{m}$ is the message’s timestamp,
$t_{m-1}$ and $t_{m}$ are the preceding and succeeding time steps of
$\tau^{i}_{m}$. However, what is different now is that the distance
measurement $\breve{d}^{i}$ at time $t_{k}+{\delta t}^{i}$ is defined by the
norm of the vector ${}^{\texttt{L}}{\bm{\mathbf{d}}}^{i}$, corrupted by
Gaussian noise and bias as follows:

$\displaystyle\breve{d}^{i}$
$\displaystyle=\left\lVert{}^{\texttt{L}}{\bm{\mathbf{d}}}^{i}\right\rVert+{\bm{\mathbf{\eta}}}_{\mathcal{U}^{i}}+\mathbf{b}^{r},\
{\bm{\mathbf{\eta}}}_{\mathcal{U}^{i}}\sim\
\mathcal{N}(0,\sigma_{\mathcal{U}}^{2});$ (6)
$\displaystyle{}^{\texttt{L}}{\bm{\mathbf{d}}}^{i}$
$\displaystyle\triangleq\mathbf{p}_{m}+\mathbf{R}_{m-1}\mathrm{Exp}\left(s^{i}\mathrm{Log}(\mathbf{R}_{m-1}^{-1}\mathbf{R}_{m})\right){\bm{\mathbf{y}}}^{i}$
$\displaystyle\qquad\qquad-a^{i}\mathbf{v}_{m-1}-b^{i}{\mathbf{v}}_{m}-{}^{\texttt{L}}_{\texttt{W}}\mathbf{R}{\bm{\mathbf{x}}}^{i}-{}^{\texttt{L}}_{\texttt{W}}\mathbf{t},$
(7)

where $s^{i}\triangleq\frac{{\delta t}_{i}}{{\Delta t}_{m}}$,
$a^{i}\triangleq\frac{{\Delta t}_{m}^{2}-{\delta t}_{i}^{2}}{2\Delta t_{m}}$,
$b^{i}\triangleq\frac{({\Delta t}_{m}-{\delta t}_{i})^{2}}{2{\Delta t}_{m}}$,
${\delta t}^{i}\triangleq\tau^{i}_{k}-t_{m-1}$, ${\Delta t}_{m}\triangleq
t_{m}-t_{m-1}$.

Thus, the UWB range residual is defined as:

$\displaystyle\bm{r}_{\mathcal{U}}\triangleq\|{}^{\texttt{L}}{\bm{\mathbf{d}}}(\hat{\mathcal{X}}_{m-1},\
\hat{\mathcal{X}}_{m},\ {}^{\texttt{L}}_{\texttt{W}}\hat{\mathbf{T}},\
\mathcal{U}^{i}_{m})\|+\hat{\mathbf{b}}^{r}-\breve{d}^{i}.$ (8)

Note that ${}^{\texttt{L}}_{\texttt{W}}\hat{\mathbf{T}}$ and
$\hat{\mathbf{b}}^{r}$ are kept fixed during the sliding window optimization
process. Moreover, UWB is not fused until the BA process has updated
${}^{\texttt{L}}_{\texttt{W}}\hat{\mathbf{T}}$ and $\hat{\mathbf{b}}^{r}$,
which is discussed in Sec. V-B.

####  IV-B3 Camera

Over the sliding window, BRIEF features are tracked and associated with the
time steps. For a visual feature ${\bm{\mathbf{f}}}^{i}$ and its pair of
projected coordinate
$\mathcal{V}^{i}_{ab}\triangleq\left({}^{\texttt{C}_{a}}\mathcal{Z}^{i},\
{}^{\texttt{C}_{b}}\mathcal{Z}^{i}\right)$ in two cameras $\texttt{C}_{a}$ and
$\texttt{C}_{b}$, the residual of this observation is defined as:

$\displaystyle\bm{r}_{\mathcal{V}}(\hat{\mathcal{X}}_{a},\hat{\mathcal{X}}_{b},\mathcal{V}^{i}_{ab})=\pi\left({}^{\texttt{C}_{b}}\hat{{\bm{\mathbf{f}}}}^{i}\right)-{}^{\texttt{C}_{b}}\mathcal{Z}^{i},$
$\displaystyle{}^{\texttt{C}_{b}}\hat{{\bm{\mathbf{f}}}}^{i}\triangleq{}^{\texttt{B}}_{\texttt{C}_{b}}\mathbf{R}^{-1}\left(\hat{\mathbf{R}}_{m_{b}}^{-1}\left({}^{\texttt{L}}\hat{{\bm{\mathbf{f}}}}^{i}-\hat{\mathbf{p}}_{m_{b}}\right)-{}^{\texttt{B}}_{\texttt{C}_{b}}\mathbf{t}\right),$
$\displaystyle{}^{\texttt{L}}\hat{{\bm{\mathbf{f}}}}^{i}\triangleq\hat{\mathbf{R}}_{m_{a}}\left({}^{\texttt{B}}_{\texttt{C}_{a}}\mathbf{R}\left[(\hat{\lambda}^{i})^{-1}{}^{\texttt{C}_{a}}\mathcal{Z}^{i}\right]+{}^{\texttt{B}}_{\texttt{C}_{a}}\mathbf{t}\right)+\hat{\mathbf{p}}_{m_{a}},$
(9)

Note that in this formulation, ${}^{\texttt{B}}_{\texttt{C}_{a}}\mathbf{R}$,
${}^{\texttt{B}}_{\texttt{C}_{b}}\mathbf{R}$,
${}^{\texttt{B}}_{\texttt{C}_{a}}\mathbf{t}$,
${}^{\texttt{B}}_{\texttt{C}_{b}}\mathbf{t}$ are constant extrinsic parameters
of the cameras, and the frames $\texttt{C}_{a}$ and $\texttt{C}_{b}$ could be
coupled with the same state but different cameras, or different states of the
same camera, or both states and cameras are different.

By checking if a visual feature’s estimated 3D coordinates fit well on the 3D
local map $\mathcal{M}$, we can marginalize this feature to be a fixed prior
in the sliding window. Algorithm 1 presents the details of this so-called MMM
scheme. Fig. 5 illustrates the result of the MMM process for some features.

\begin{overpic}[width=433.62pt]{margmark.png} \end{overpic} Figure 5: Matching
of some visual features from the image to the 3D local map thanks to the MMM
process.

When a feature is marginalized, its cost factor is similar to (IV-B3), only
that the state estimate of the inverse depth $\lambda^{i}$ is calculated
directly from the marginalized 3D coordinates
${}^{\texttt{L}}\bar{{\bm{\mathbf{f}}}}^{i}$ and kept fixed during the
optimization process.

Input: ${}^{\texttt{C}_{a}}\mathcal{Z}^{i}$, $\hat{\lambda}^{i}$,
$\hat{\mathbf{T}}_{m_{a}}$, $\mathcal{M}$.

Output: ${}^{\texttt{L}}\bar{{\bm{\mathbf{f}}}}^{i}$.

1 Compute:
${}^{\texttt{C}_{a}}\hat{{\bm{\mathbf{f}}}}^{i}={}^{\texttt{B}}_{\texttt{C}_{a}}\mathbf{R}\left[(\hat{\lambda}^{i})^{-1}{}^{\texttt{C}_{a}}\mathcal{Z}^{i}\right]+{}^{\texttt{B}}_{\texttt{C}_{a}}\mathbf{t}$;

2 Compute:
${}^{\texttt{L}}\hat{{\bm{\mathbf{f}}}}^{i}=\hat{\mathbf{R}}_{m_{a}}{}^{\texttt{C}_{a}}\hat{{\bm{\mathbf{f}}}}^{i}+\hat{\mathbf{p}}_{m_{a}}$;

3 Find $\mathcal{N}=\text{KNN}({}^{\texttt{L}}\hat{{\bm{\mathbf{f}}}}^{i},\
\mathcal{M})$;

4 Find
${\bm{\mathbf{n}}}^{*}=\operatorname*{argmin}_{{\bm{\mathbf{n}}}\in\mathbb{R}^{3}}\sum_{{\bm{\mathbf{x}}}\in\mathcal{N}}||{\bm{\mathbf{n}}}^{\top}{\bm{\mathbf{x}}}+1||^{2}$;

5 Compute:
$\bar{{\bm{\mathbf{n}}}}=\frac{{\bm{\mathbf{n}}}^{*}}{\left\lVert{\bm{\mathbf{n}}}^{*}\right\rVert}$
and
${}^{\texttt{L}}_{\texttt{C}_{a}}\hat{\mathbf{p}}=\hat{\mathbf{R}}_{m_{a}}{}^{\texttt{B}}_{\texttt{C}_{a}}\mathbf{t}+\hat{\mathbf{p}}_{m_{a}}$;

6 Compute:
${\bm{\mathbf{u}}}=\hat{\mathbf{R}}_{m_{a}}{}^{\texttt{B}}_{\texttt{C}_{a}}\mathbf{R}{}^{\texttt{C}_{a}}\mathcal{Z}^{i}$;

7 Compute:
${\bm{\mathbf{c}}}={}^{\texttt{L}}_{\texttt{C}_{a}}\hat{\mathbf{p}}-{\bm{\mathbf{u}}}(1+\bar{{\bm{\mathbf{n}}}}^{\top}{}^{\texttt{L}}_{\texttt{C}_{a}}\hat{\mathbf{p}})/(\bar{{\bm{\mathbf{n}}}}^{\top}{\bm{\mathbf{u}}})$;

8 if
$\min_{{\bm{\mathbf{x}}}\in\mathcal{N}}||{\bm{\mathbf{x}}}-{}^{\texttt{L}}\hat{{\bm{\mathbf{f}}}}^{i}||\leq
0.25$

9 and $\left\lVert{\bm{\mathbf{c}}}-{\bm{\mathbf{x}}}\right\rVert<1.0,\
\forall{\bm{\mathbf{x}}}\in\\{{}^{\texttt{L}}\hat{{\bm{\mathbf{f}}}}^{i}\\}\cup\mathcal{N}$

10 and
$\left\lvert\bar{{\bm{\mathbf{n}}}}^{\top}{\bm{\mathbf{x}}}+1\right\rvert<0.1,\
\forall{\bm{\mathbf{x}}}\in\mathcal{N}$

11

12

13 then

14 Set: ${}^{\texttt{L}}\bar{{\bm{\mathbf{f}}}}^{i}={\bm{\mathbf{c}}}$;

15

16 end if

Algorithm 1 MMM process on a visual feature
${}^{\texttt{C}_{a}}{\bm{\mathbf{f}}}^{i}$

##  V Global Optimization Blocks

###  V-A Key frame management

####  V-A1 Key frame admission

The key frame admission procedure is similar to our previous work [17].
Briefly speaking, after each optimization on the sliding window, we find a
number of nearest neighbors of the state at time $t_{k-M/2}$ and if the
relative distance or relative rotation to all of these neighbours exceed a
certain threshold, the information associated with this time step will be
marginalized as prior.

####  V-A2 Key frame selection

The selection of the key frames is needed for construction of a local
pointcloud map for FMM process. This selection process takes place before the
optimization process and is based on the IMU-propagated pose
$\breve{\mathbf{T}}_{k}$. Hence, the set of these key frames is a union of
$\\{\mathcal{K}_{a}\\}\cup\\{\mathcal{K}_{b}\\}\cup\\{\mathcal{K}_{c}\\}$,
where $\\{\mathcal{K}_{a}\\}$ is the set of the last $M$ key frames,
$\\{\mathcal{K}_{b}\\}$ is the set of $M$ nearest neighbors of
$\breve{\mathbf{T}}_{k}$, and $\\{\mathcal{K}_{c}\\}$ is the set of key frames
representing their $2m\times 2m\times 2m$ voxel cells that are within a radius
from $\breve{\mathbf{T}}_{k}$.

###  V-B Bundle Adjustment

For the BA process, our task is to construct and optimize the following cost
function:

$\displaystyle
f(\hat{\mathcal{Y}},{}^{\texttt{L}}_{\texttt{W}}\hat{\mathbf{T}},\hat{\mathbf{b}}^{r})\triangleq\Bigg{\\{}\sum_{n=1}^{N}\left\lVert\bm{r}_{1}(\hat{\mathbf{T}}_{n-1},\hat{\mathbf{T}}_{n},{}^{n-1}_{n}\bar{\mathbf{T}})\right\rVert^{2}_{{\bm{\mathbf{P}}}_{1}^{-1}}$
$\displaystyle\qquad+\sum_{(p,c)\in\mathcal{\mathcal{H}}}\left\lVert\bm{r}_{2}(\hat{\mathbf{T}}_{p},\hat{\mathbf{T}}_{c},{}^{p}_{c}\bar{\mathbf{T}})\right\rVert^{2}_{{\bm{\mathbf{P}}}_{2}^{-1}}$
$\displaystyle\qquad+\sum_{n=1}^{N}\sum_{j=1}^{N_{\mathcal{U}}^{n}}\left\lVert\bm{r}_{3}(\hat{\mathbf{T}}_{n},{}^{\texttt{L}}_{\texttt{W}}\hat{\mathbf{T}},\hat{\mathbf{b}}^{r},\bar{\mathcal{U}}^{i}_{n})\right\rVert_{{\bm{\mathbf{P}}}_{3}^{-1}}\Bigg{\\}},$
(10)

where
$\hat{\mathcal{Y}}\triangleq(\hat{\mathbf{T}}_{0},\hat{\mathbf{T}}_{1},\dots\hat{\mathbf{T}}_{N})$
is the key frames’ poses, ${}^{n-1}_{n}\bar{\mathbf{T}}$ and
${}^{p}_{c}\bar{\mathbf{T}}$ are respectively the relative pose and loop
closure priors, $\mathcal{H}$ is the set of loop closure pairs, and
$\bar{\mathcal{U}}^{i}_{n}$ is a marginalized UWB measurement whose timestamp
is within 0.2s of the key frame at time $t_{n}$, i.e.:

$\bar{\mathcal{U}}^{i}_{n}=\left(\breve{d}^{i},{}^{\texttt{W}}{\bm{\mathbf{x}}}^{i},{\bm{\mathbf{y}}}^{i},{}^{t_{n}}_{\tau^{i}_{n}}\bar{\mathbf{R}},{}^{t_{n}}_{\tau^{i}_{n}}\bar{\mathbf{t}}\right),$
(11)

where $\breve{d}^{i}$, ${}^{\texttt{W}}{\bm{\mathbf{x}}}^{i}$
${\bm{\mathbf{y}}}^{i}$ are defined similarly to (7), and
$({}^{t_{n}}_{\tau^{i}_{n}}\bar{\mathbf{R}},{}^{t_{n}}_{\tau^{i}_{n}}\bar{\mathbf{t}})$
is the relative transform between $\texttt{B}_{t_{n}}$ and
$\texttt{B}_{\tau^{i}_{n}}$, which can be obtained from IMU propagation.

The residuals $\bm{r}_{1}(\cdot)$, $\bm{r}_{2}(\cdot)$ over the relative poses
are straightforward, while the residual $\bm{r}_{3}$ can be stated as:

$\displaystyle\bm{r}_{3}(\hat{\mathbf{T}}_{n},{}^{\texttt{L}}_{\texttt{W}}\hat{\mathbf{T}},\hat{\mathbf{b}}^{r},\bar{\mathcal{U}}^{i}_{n})=\left\lVert\hat{{\bm{\mathbf{d}}}}^{i}\right\rVert+\hat{\mathbf{b}}^{r}-\breve{d}^{i},$
$\displaystyle\hat{{\bm{\mathbf{d}}}}^{i}\triangleq\hat{\mathbf{p}}_{n}+\hat{\mathbf{R}}_{n}{}^{t_{n}}_{\tau^{i}_{n}}\bar{\mathbf{R}}({\bm{\mathbf{y}}}^{i}+{}^{t_{n}}_{\tau^{i}_{n}}\bar{\mathbf{t}})-{}^{\texttt{L}}_{\texttt{W}}\hat{\mathbf{R}}{\bm{\mathbf{x}}}^{i}-{}^{\texttt{L}}_{\texttt{W}}\hat{\mathbf{t}}.$
(12)

To ensure the pose graph has enough excitation to help the the anchor-related
states converge, we do not add the factors of $\bm{r}_{3}$ into the BA cost
function right from the beginning. Rather, they are only added in after the
spatial distribution of the key frame poses has satisfied a certain condition.
Specifically, we calculate the geometric dilution of the key frame positions
via the quantity

$\Gamma=\left(\sum_{n=1}^{N}(\bar{\mathbf{p}}_{n}-\mu)(\bar{\mathbf{p}}_{n}-\mu)^{\top}\right)^{-1},\mu=\frac{1}{N}\sum_{n=1}^{N}\bar{\mathbf{p}}_{n}.\
$ (13)

Hence we perform singular value decomposition on $\Gamma$ to obtain the
singular values $\sigma_{1}\geq\sigma_{2}\geq\sigma_{3}>0$. If
$\sigma_{1}<c_{1}$ and $\sigma_{1}/\sigma_{3}<c_{2}$, where $c_{1}$ and
$c_{2}>1$ are some user-defined parameters, then we can start adding the
factors $\bm{r}_{3}(\cdot)$ to (10). Afterwards, we can obtain the estimates
of ${}^{\texttt{L}}_{\texttt{W}}\mathbf{T}$ and $\mathbf{b}^{r}$ which can be
used for the fusion of UWB factors in (5).

###  V-C Loop Closure

To construct the loop priors ${}^{p}_{c}\bar{\mathbf{T}}$ in Sec. V-B, a
three-stage process is conducted as follows:

TABLE I: ATE of the SLAM methods over the datasets. The best odometry result is highlighted in bold, and the second best is underlined. The first nine datasets belong to the NTU VIRAL dataset, the next three belong to the building inspection trials, and the last five are generated from AirSim. The symbol ’-’ indicates that the method diverges during the experiment. Dataset |  | VINS-Mono
---
(right camera,
odom/BA)
| VINS-Fusion
---
(both cameras,
odom/BA)
| A-LOAM
---
(horz. / vert. / latr.)
| LIO-
---
SAM
(horz.)
| MLOAM
---
(all lidars)
| VIRAL-SLAM
---
(horz. lidar,
odom/BA)
| VIRAL-SLAM
---
(all lidars,
odom/BA)
eee_01 | 1.650 / 0.568 | 0.608 / 0.306 | 0.212 / 6.827 | 0.075 | 0.249 |  0.064 / 0.084 |  0.060 / 0.086
eee_02 | 0.722 / 0.443 | 0.506 / 0.266 | 0.199 / 1.845 | 0.069 | 0.166 |  0.051 / 0.056 |  0.058 / 0.050
eee_03 | 1.037 / 0.886 | 0.494 / 0.383 | 0.148 / 3.852 | 0.101 | 0.232 |  0.060 / 0.073 |  0.037 / 0.049
nya_01 | 1.475 / 0.830 | 0.397 / 0.237 | 0.077 / 3.206 | 0.076 | 0.123 |  0.063 / 0.061 |  0.051 / 0.058
nya_02 | 0.580 / 0.422 | 0.424 / 0.297 | 0.091 / 0.377 | 0.090 | 0.191 |  0.042 / 0.051 |  0.043 / 0.055
nya_03 | 1.333 / 0.501 | 0.787 / 0.368 | 0.080 / 0.715 | 0.137 | 0.226 |  0.039 / 0.063 |  0.032 / 0.062
sbs_01 | 4.142 / 3.739 | 0.508 / 0.372 | 0.203 / 6.762 | 0.089 | 0.173 |  0.051 / 0.055 |  0.048 / 0.059
sbs_02 | 1.605 / 0.890 | 0.564 / 0.369 | 0.091 / 2.496 | 0.083 | 0.147 |  0.056 / 0.062 |  0.062 / 0.052
sbs_03 | 1.306 / 0.802 | 0.878 / 0.276 | 0.363 / 3.996 | 0.140 | 0.153 |  0.060 / 0.075 |  0.054 / 0.072
bid_01 | 3.749 / 3.632 | 2.416 / 2.045 | 0.936 / 14.670 | - | 4.264 |  0.178 / 0.159 |  0.161 / 0.158
bid_02 | 1.257 / 1.238 | 0.837 / 0.603 | 4.359 / 5.043 | - | 0.257 |  0.752 / 1.320 |  0.343 / 0.603
bid_03 | 0.670 / 0.659 | 0.914 / 0.814 | 1.961 / 4.789 | - | 3.330 |  2.181 / 1.813 |  0.128 / 0.177
nbh_01 | 4.709 / 4.474 | 1.413 / 1.388 | 86.399 / 53.680 / 53.757 | - | 0.321 |  0.149 / 0.200 |  0.146 / 0.194
nbh_02 | 3.526 / 2.960 | 2.268 / 1.436 | 47.326 / 40.881 / 40.638 | - | 0.369 |  0.084 / 0.142 |  0.096 / 0.162
nbh_03 | 3.560 / 2.759 | 1.837 / 0.643 | 6.764 / 50.710 / 50.578 | - | 0.282 |  0.091 / 0.114 |  0.098 / 0.141
nbh_04 | 2.707 / 1.981 | 1.974 / 1.513 | 24.448 / 35.747 / 35.970 | - | 0.375 |  0.113 / 0.239 |  0.099 / 0.196
nbh_05 | 118.644 / 120.532 | 1.255 / 0.825 | 0.834 / 24.624 / 24.852 | - | 0.377 |  0.116 / 0.291 |  0.110 / 0.276

First, when a new key frame is admitted, we compare its visual features with
the database using the DBoW library. If a match is flagged, we can extract the
transforms $\mathbf{T}_{c}$ and $\mathbf{T}_{p}$, referred to as the current
and previous key poses, respectively. Then, we search for a number of key
frames that were admitted before and after $\mathbf{T}_{p}$ to build a local
map ${}^{\texttt{B}_{p}}\mathcal{M}_{p}$ using their corresponding
marginalized CFCs, and proceed to the second stage.

At the second stage, we will use ICP to align the CFC
${}^{\texttt{B}_{c}}\mathcal{F}_{c}$ with ${}^{\texttt{B}_{p}}\mathcal{M}_{p}$
to obtain a fitness score, as well as an initial guess of
${}^{\texttt{B}_{p}}_{\texttt{B}_{c}}\hat{\mathbf{T}}$. If the fitness score
is below a threshold, we proceed to the third stage.

At the third stage, we perform FMM between
${}^{\texttt{B}_{c}}\mathcal{F}_{c}$ and ${}^{\texttt{B}_{p}}\mathcal{M}_{p}$
to calculate the FMM coefficients, then construct the following cost function
and optimize it:

$f\left({}^{\texttt{B}_{p}}_{\texttt{B}_{c}}\hat{\mathbf{T}}\right)=\sum_{i=1}^{N_{\mathcal{L}}^{c}}\rho\left(\left\lVert\bm{r}_{\mathcal{L}}({}^{\texttt{B}_{p}}_{\texttt{B}_{c}}\hat{\mathbf{T}},\mathcal{L}_{c}^{i})\right\rVert^{2}_{{\bm{\mathbf{P}}}_{\mathcal{L}_{c}^{i}}^{-1}}\right).$
(14)

After optimizing (14) and obtaining the optimal relative pose
${}^{\texttt{B}_{p}}_{\texttt{B}_{c}}\hat{\mathbf{T}}^{*}$, if the ratio
$f\left({}^{\texttt{B}_{p}}_{\texttt{B}_{c}}\hat{\mathbf{T}}^{*}\right)/N_{\mathcal{L}}^{c}$
is below a threshold,
${}^{\texttt{B}_{p}}_{\texttt{B}_{c}}\hat{\mathbf{T}}^{*}$ will be registered
as a loop closure prior
${}^{\texttt{B}_{p}}_{\texttt{B}_{c}}\bar{\mathbf{T}}$.

##  VI Experiment

###  VI-A Datasets

We first employ our recently published NTU VIRAL dataset111https://ntu-
aris.github.io/ntu_viral_dataset/ [28], which features all sensor types
covered by VIRAL SLAM. To further demonstrate the robustness of VIRAL SLAM in
low-texture condition, we conduct further experiments on some building
inspections datasets with significant challenges collected near a building
facade. Finally, since no ground truth on the anchor position and the ranging
bias are available, to clearly verify this capability of VIRAL-SLAM, we employ
AirSim simulator to construct a dataset with absolute ground truth for more
accurate evaluate.

###  VI-B Comparison

Figure 6: VIRAL SLAM result on eee_02 dataset. The estimated trajectory is in
blue, and ground truth is in red. The key frame poses are marked by the yellow
circles. The activated key frames for local map building are highlighted by
the green squares. The UWB anchors are also marked with red dots. The loop
edges are marked with light cyan lines. Some visual features can be are marked
with small green circles. Due to page constraint we refer the readers to the
online video recording at https://youtu.be/LerAfvZMb7M for more detailed
illustration of our experiments.

For comparison VIRAL SLAM, we run other state-of-the-art localization
techniques with all of the aforementioned datasets. All algorithms are run on
an NUC 10 computer with core i7 processor. Each method is slightly modified
and configured for their best performance with the dataset. The details of
these modified packages can be found on the NTU VIRAL dataset website1. Since
several lidar-based methods are not designed to for multiple lidars, we also
include experiments of VIRAL SLAM using only the horizontal lidar for a fairer
comparison. Tab. I summarizes the Absolute Trajectory Error (ATE) of these
methods.

TABLE II: ATE of VIRAL SLAM’s key frame positions with different sensor combinations over the employed datasets (IMU is always used). All values are in m. The average ATE is calculated for each class of datasets. Dataset |  | Lidars
---
| Lidars
---
+Cameras
| Lidars
---
+UWB
| Lidars
---
+UWB
+Cameras
eee_01 | 0.0380 | 0.0390 | 0.0822 | 0.0861
eee_02 | 0.0451 | 0.0347 | 0.0647 | 0.0505
eee_03 | 0.0385 | 0.0438 | 0.0608 | 0.0494
nya_01 | 0.0429 | 0.0436 | 0.0545 | 0.0584
nya_02 | 0.0463 | 0.0416 | 0.0635 | 0.0551
nya_03 | 0.0383 | 0.0392 | 0.0696 | 0.0621
sbs_01 | 0.0441 | 0.0483 | 0.0585 | 0.0587
sbs_02 | 0.0512 | 0.0476 | 0.0547 | 0.0518
sbs_03 | 0.0514 | 0.0524 | 0.0696 | 0.0716
Average | 0.0440 | 0.0434 | 0.0642 | 0.0604
bid_01 | 0.2104 | 0.2039 | 0.1585 | 0.1583
bid_02 | 0.6046 | 0.6033 | 0.6111 | 0.6029
bid_03 | 0.1904 | 0.1865 | 0.1822 | 0.1765
Average | 0.3351 | 0.3312 | 0.3173 | 0.3126
nbh_01 | 0.0393 | 0.0508 | 0.1910 | 0.1938
nbh_02 | 0.0374 | 0.0364 | 0.1489 | 0.1618
nbh_03 | 0.0453 | 0.0342 | 0.1653 | 0.1413
nbh_04 | 0.0178 | 0.0201 | 0.1811 | 0.1963
nbh_05 | 0.0225 | 0.0257 | 0.3224 | 0.2763
Average | 0.0325 | 0.0335 | 0.2017 | 0.1939
TABLE III: Anchor coordinates and the final values estimated by the BA process using the AirSim-generated dataset. All values are in m. Note that the coordinates of anchor 0 is fixed at $(0,0,1)$. | Anchor 1 | Anchor 2 | $\mathbf{b}^{r}$
---|---|---|---
True values | 15.000, 0.000, 1.250 | 7.500, -5.000, 1.500 | 0.050
Initial values | 15.020, 0.000, 1.000 | 7.450, -5.070, 1.000 | 0.000
nbh_01 est. | 15.018, 0.019, 1.245 | 7.449, -5.046, 1.510 | 0.026
nbh_02 est. | 15.018, 0.038, 1.243 | 7.456, -5.038, 1.484 | 0.021
nbh_03 est. | 15.011, 0.109, 1.506 | 7.467, -4.995, 1.706 | 0.008
nbh_04 est. | 15.018, 0.025, 1.253 | 7.451, -5.043, 1.502 | 0.031
nbh_05 est. | 15.018, 0.055, 1.253 | 7.462, -5.032, 1.449 | 0.020
Figure 7: Error of the estimates on anchor position and ranging bias by the BA
process over time.

From Tab. I we can clearly see that VIRAL SLAM consistently achieves better
performance compared to existing methods in all datasets, even when only one
lidar is used. For VINS-Mono, VINS-Fusion and VIRAL SLAM, we also report the
BA results, i.e. the positions of the key frames refined by loop closure and
BA process. We note that the pure odometry and BA results of VIRAL SLAM do not
differ much, while there is a large difference between odometry and BA results
of the VIO methods. This can be explained as VIRAL SLAM has much less drift
than VINS methods, thus there is not a lot of correction made in the BA
process. Fig. 6 presents the result of VIRAL SLAM in one experiment. Also, we
can see from Tab. III and Fig. 7 that the BA process does bring down the error
in the extrinsic and intrinsic parameters of the UWB network.

###  VI-C Ablation study

We further conduct extra experiments with different subsets of sensor suites
to study the contribution of each sensor in the overall localization scheme.
Tab. II reports the ATE of the key frame positions of VIRAL SLAM over the
experimented datasets. From this table we can see that the lidar-only and
lidar-camera setups has roughly similar average ATE over the datasets. From
lidar-only to lidar-range, there is a significant increase in error. This can
be explained as that with UWB we are not just estimating the key frame poses,
but also states relating to the UWB anchor position and bias. Thus the
increase in ATE is a trade-off of knowledge on the UWB states. Interestingly,
when comparing the ATE of lidar-UWB with lidar-camera-UWB setup, we can
observe a decrease in ATE. Thus, we can conclude that the loop closure
constraints introduced by using camera do help bring down the error in the BA
process.

##  VII Conclusion and Future Works

In this paper we have developed a multi-sensor SLAM method leveraging an
extensive set of sensors: stereo camera, lidar, IMU, UWB, so-called VIRAL
SLAM. The system features synchronization and integration of multiple lidars
with complementary field of view (FOV), depth-matching of stereo camera visual
features with pointcloud map, vision-triggered lidar-refined loop closure, UWB
extrinsic and intrinsic parameter estimation. Via extensive experiments
results, we have demonstrated that VIRAL SLAM can achieve highly accurate
localization results as well as robustness in challenging conditions.

## References

  * [1] J. Zhang and S. Singh, “Laser–visual–inertial odometry and mapping with high robustness and low drift,” _Journal of Field Robotics_ , vol. 35, no. 8, pp. 1242–1264, 2018.
  * [2] T. Shan, B. Englot, C. Ratti, and R. Daniela, “Lvi-sam: Tightly-coupled lidar-visual-inertial odometry via smoothing and mapping,” in _IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. to–be–added.
  * [3] T.-M. Nguyen, M. Cao, S. Yuan, Y. Lyu, T. H. Nguyen, and L. Xie, “Liro: Tightly coupled lidar-inertia-ranging odometry,” _2021 IEEE International Conference on Robotics and Automation (ICRA), Accepted_ , 2020.
  * [4] ——, “Viral-fusion: A visual-inertial-ranging-lidar sensor fusion approach,” _IEEE Transactions on Robotics_ , 2021.
  * [5] T. H. Nguyen, T.-M. Nguyen, and L. Xie, “Tightly-coupled ultra-wideband-aided monocular visual slam with degenerate anchor configurations,” _Autonomous Robots_ , vol. 44, no. 8, pp. 1519–1534, 2020.
  * [6] ——, “Range-focused fusion of camera-imu-uwb for accurate and drift-reduced localization,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 1678 – 1685, 2021.
  * [7] X. Zuo, P. Geneva, W. Lee, Y. Liu, and G. Huang, “Lic-fusion: Lidar-inertial-camera odometry,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2019, pp. 5848–5854.
  * [8] X. Zuo, Y. Yang, P. Geneva, L. Jiajun, Y. Liu, G. Huang, and M. Pollefey, “Lic-fusion 2.0: Lidar-inertial-camera odometry with sliding-window plane-feature tracking,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2020, pp. 5112–5119.
  * [9] D. Wisth, M. Camurri, S. Das, and M. Fallon, “Unified multi-modal landmark tracking for tightly coupled lidar-visual-inertial odometry,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 1004–1011, 2021.
  * [10] J. Graeter, A. Wilczynski, and M. Lauer, “Limo: Lidar-monocular visual odometry,” in _2018 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2018, pp. 7872–7879.
  * [11] W. Shao, S. Vijayarangan, C. Li, and G. Kantor, “Stereo visual inertial lidar simultaneous localization and mapping,” in _2019 IEEE/RSJ international conference on intelligent robots and systems (IROS)_ , 2019.
  * [12] T. Qin, P. Li, and S. Shen, “Vins-mono: A robust and versatile monocular visual-inertial state estimator,” _IEEE Transactions on Robotics_ , vol. 34, no. 4, pp. 1004–1020, 2018.
  * [13] T. Shan, B. Englot, D. Meyers, W. Wang, C. Ratti, and R. Daniela, “Lio-sam: Tightly-coupled lidar inertial odometry via smoothing and mapping,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 5135–5142.
  * [14] T. Shan and B. Englot, “Lego-loam: Lightweight and ground-optimized lidar odometry and mapping on variable terrain,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 4758–4765.
  * [15] J. Jiao, H. Ye, Y. Zhu, and M. Liu, “Robust odometry and mapping for multi-lidar systems with online extrinsic calibration,” _arXiv preprint arXiv:2010.14294_ , 2020.
  * [16] P. Chen, W. Shi, S. Bao, M. Wang, W. Fan, and H. Xiang, “Low-drift odometry, mapping and ground segmentation using a backpack lidar system,” _IEEE Robotics and Automation Letters_ , 2021.
  * [17] T.-M. Nguyen, S. Yuan, M. Cao, Y. Lyu, T. H. Nguyen, and L. Xie, “Miliom: Tightly coupled multi-input lidar-inertia odometry and mapping,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 3, pp. 5573–5580, May 2021.
  * [18] M. Karrer and M. Chli, “Distributed variable-baseline stereo slam from two uavs.” [Online]. Available: https://arxiv.org/pdf/2009.04801.pdf
  * [19] T.-M. Nguyen, T. H. Nguyen, M. Cao, Z. Qiu, and L. Xie, “Integrated uwb-vision approach for autonomous docking of uavs in gps-denied environments,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 9603–9609.
  * [20] T.-M. Nguyen, Z. Qiu, T. H. Nguyen, M. Cao, and L. Xie, “Distance-based cooperative relative localization for leader-following control of mavs,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 4, pp. 3641–3648, 2019\.
  * [21] ——, “Persistently excited adaptive relative localization and time-varying formation of robot swarms,” _IEEE Transactions on Robotics_ , vol. 36, no. 2, pp. 553–560, 2019.
  * [22] J. P. Queralta, L. Qingqing, F. Schiano, and T. Westerlund, “Vio-uwb-based collaborative localization and dense scene reconstruction within heterogeneous multi-robot systems.” [Online]. Available: https://arxiv.org/pdf/2006.00420.pdf
  * [23] J. Xu, J. Hu, L. Xie, and K.-Y. Lum, “Distributed coverage control under generalized locational optimization framework,” in _Proceedings of the 31st Chinese Control Conference_. IEEE, 2012, pp. 6015–6020.
  * [24] H. Xu, L. Wang, Y. Zhang, K. Qiu, and S. Shen, “Decentralized visual-inertial-uwb fusion for relative state estimation of aerial swarm,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 8776–8782.
  * [25] H. Ye, Y. Chen, and M. Liu, “Tightly coupled 3d lidar inertial odometry and mapping,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 3144–3150.
  * [26] T.-M. Nguyen, A. H. Zaini, C. Wang, K. Guo, and L. Xie, “Robust target-relative localization with ultra-wideband ranging and communication,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 2312–2319.
  * [27] S. Agarwal and K. Mierle, “Ceres solver: Tutorial & reference.” [Online]. Available: http://ceres-solver.org/
  * [28] T.-M. Nguyen, S. Yuan, M. Cao, Y. Lyu, T. H. Nguyen, and L. Xie, “Ntu viral: A visual-inertial-ranging-lidar dataset, from an aerial vehicle viewpoint,” The International Journal of Robotics Research. [Online]. Available: https://ntu-aris.github.io/ntu_viral_dataset/


# Learning-Based Dimensionality Reduction
for Computing Compact and Effective Local Feature Descriptors

Hao Dong  Xieyuanli Chen  Mihai Dusmanu  Viktor Larsson  Marc Pollefeys
Cyrill Stachniss H. Dong, M. Dusmanu, and M. Pollefeys are with ETH Zürich,
Switzerland. X. Chen and C. Stachniss are with the University of Bonn,
Germany. V. Larsson is with Lund University, Sweden. M. Pollefeys is
additionally with Microsoft. C. Stachniss is additionally with the Department
of Engineering Science at the University of Oxford, UK, and with the Lamarr
Institute for Machine Learning and Artificial Intelligence, Germany.

###### Abstract

A distinctive representation of image patches in form of features is a key
component of many computer vision and robotics tasks, such as image matching,
image retrieval, and visual localization. State-of-the-art descriptors, from
hand-crafted descriptors such as SIFT to learned ones such as HardNet, are
usually high dimensional; 128 dimensions or even more. The higher the
dimensionality, the larger the memory consumption and computational time for
approaches using such descriptors. In this paper, we investigate multi-layer
perceptrons (MLPs) to extract low-dimensional but high-quality descriptors. We
thoroughly analyze our method in unsupervised, self-supervised, and supervised
settings, and evaluate the dimensionality reduction results on four
representative descriptors. We consider different applications, including
visual localization, patch verification, image matching and retrieval. The
experiments show that our lightweight MLPs achieve better dimensionality
reduction than PCA. The lower-dimensional descriptors generated by our
approach outperform the original higher-dimensional descriptors in downstream
tasks, especially for the hand-crafted ones. The code will be available at
https://github.com/PRBonn/descriptor-dr.

##  I Introduction

Local feature descriptors [24, 6, 33] are used to represent characteristics of
image patches and are designed to be robust to partial occlusions, viewpoint
changes, and variations in illumination. They play an essential role in many
robotics applications such as robot localization [35], object recognition
[14], and image retrieval [28].

The traditional pipeline for local feature extraction often starts by
detecting the position, scale, and orientation of keypoints in the image.
Then, a normalized image patch is extracted with respect to the estimated
keypoint, which usually provides the basis for the descriptor computation.
Distinctive and invariant keypoints and descriptors are key to achieving good
performance in the subsequent matching, retrieval, and localization tasks.
This paper focuses on the descriptor part of this pipeline, specifically the
dimensionality reduction of local feature descriptors to generate compact and
at the same time effective features.

Figure 1: Overview of our approach. We first compute descriptors of given
image patches. Then an MLP-based network is used for dimensionality reduction.
We aim to learn an MLP-based projection better than PCA to generate lower-
dimensional descriptors.

Multiple visual feature descriptors have been introduced in the literature.
Among all the hand-crafted descriptors, SIFT [24] is one of the most famous
because of its robustness under blurring, translation, rotation, and scale
changes. With the advent of neural networks, more and more learning-based
descriptors have been proposed [40, 45, 38, 29] and pushed the state-of-the-
art forward in benchmarks for image matching, patch verification, and image
retrieval. While the results are promising, a common issue for both hand-
crafted and learned methods is that the dimensionality of the generated
descriptors is usually high. When the image database increases, substantial
time and space might be needed for computing and storing such high-dimensional
descriptors. That may hinder the application to mobile and resource-constraint
robots. Several principal component analysis (PCA) [19] based dimensionality
reduction methods have been proposed to alleviate this problem. For example,
Valenzuela et al. [16] apply PCA to reduce the dimensionality of SIFT and SURF
descriptors. Instead of the original SIFT’s smoothed weighted histograms, Ke
et al. [20] apply PCA to patches for generating lighter descriptors. PCA
performs a linear projection from high- to low-dimensional descriptor space,
which has limited capabilities of generating high-quality dimensionally
reduced descriptors. Moreover, the components with low eigenvalues by PCA are
not necessarily less important but down-weighted or even eliminated, which
will cause information loss and performance degradation.

Unlike previous PCA-based methods, we aim to learn an MLP-based dimensionality
reduction to better transform large descriptors into lighter ones. To this
end, we thoroughly analyze the MLP-based network for dimensionality reduction
and design three learning schemes, including unsupervised, self-supervised,
and supervised methods. For the unsupervised scheme, we use an auto-encoder
with reconstruction and distance losses to improve the projection. While for
the self-supervised one, we iteratively cluster descriptors using $k$-means
and use cluster assignments as pseudo-labels to train the MLPs. We also
propose a supervised method that uses ground truth patch labels with triplet
loss to supervise the MLPs generating more distinctive lower-dimensional
descriptors. We evaluate our MLP-based method with all proposed learning
schemes on four common descriptors: SIFT [24], MKD [30], TFeat [5], and
HardNet [29]. We train our network only on one dataset and apply it directly
to other datasets with different downstream tasks, including visual
localization, patch verification, image matching, and image retrieval. The
experimental results show that our method consistently outperforms the PCA-
based method with a strong generalization ability. Furthermore, using the
lower-dimensional descriptors generated by our supervised MLP, we achieve even
better performance in downstream tasks than the original higher-dimensional
descriptors, especially for hand-crafted ones with faster speed and less
memory consumption. Overall, our contributions are as follows:

  * •

We propose and evaluate an MLP-based network for descriptor dimensionality
reduction and show its superiority over PCA on multiple descriptors in various
tasks;

  * •

We demonstrate that the lighter descriptors by our supervised MLP projection
achieve even better performance in downstream tasks than the original
descriptors;

  * •

We thoroughly analyze the improvement of using our method for different
descriptors on multiple datasets and show a good generalization of our method.

##  II Related Work

Various local image descriptors have been proposed over the past decades
ranging from hand-crafted ones like SIFT [24], BRIEF [9], and ORB [33] to
learned ones like TFeat [5], HardNet [29], and SOSNet [41]. Hand-crafted
descriptors are typically based on human insights into which qualities are
invariant under certain transformations, such as differential or moment
invariants, correlations, and gradients histograms. For example, SIFT by Lowe
[24] generates descriptors based on the gradient distribution in the detected
patches. BRIEF by Calonder et al. [9] uses simple binary intensity comparisons
between pixels in an image patch. More details of classical hand-crafted
descriptors can be found in surveys [26, 13]. Benefiting from large-scale
datasets [8], learning-based descriptors recently achieved state-of-the-art
performances [29, 5, 25, 12, 17, 37]. For example, TFeat by Balntas et al. [5]
uses a CNN with hard-negative mining by anchor swap in the triplet loss to
compute descriptors. In contrast, Tian et al. [40] propose L2-Net, adding
different error terms in the loss function to improve the distinctiveness of
the descriptors. HardNet by Mishchuk et al. [29] also uses a simple triplet
margin loss for hard negative mining which outperforms other descriptors with
an advanced sampling procedure. Both, state-of-the-art hand-crafted and
learning-based descriptors are often high dimensional.

Dimensionality reduction techniques can be used to shorten the
dimensionalities of feature descriptors. Traditional dimensionality reduction
usually refers to reducing the dimension of data while keeping as much
information as possible. Classical examples are backward elimination [27],
forward selection [27], and random forests [7]. Another type is to find a
combination of new features to describe the data. For example. linear
dimensionality reduction methods include PCA [19], factor analysis [44], and
linear discriminant analysis [2], and non-linear methods including Kernel PCA
[36], t-distributed stochastic neighbor embedding (t-SNE) [43], and isometric
mapping [32]. Among them, PCA [19] has been widely used for dimensionality
reduction of image patch-based descriptors. For example, Gil et al. [15] and
Valenzuela et al. [16] use PCA to reduce the dimensionality of SIFT and SURF
descriptors. Ke et al. [20] apply PCA to the patches instead of using smoothed
weighted histograms in SIFT. There are few works using neural networks for
dimensionality reduction. The work most related to ours is the one by
Loquercio et al. [23], which uses a supervised method to train a linear
projection. However, they work on hand-crafted descriptors like FREAK [1] and
focus only on the visual localization task. Different from them, we thoroughly
analyze MLP-based non-linear projections using unsupervised, self-supervised,
and supervised training schemes on multiple downstream tasks. Moreover, our
method works on both hand-crafted descriptors such as SIFT [24] and MKD [30],
and learned ones such as TFeat [5] and HardNet [29], and shows a strong
generalization ability.

##  III Methodology

We aim to reduce the dimensionality of local feature descriptors using an MLP
network. To understand the ability of the MLP-based method in this task, we
investigate three learning schemes, including unsupervised, self-supervised,
and supervised methods. This section will introduce the principle of each
approach in detail.

Figure 2: The pipeline of using an auto-encoder for unsupervised
dimensionality reduction. It consists of an encoder and a decoder. The encoder
maps the original descriptors to lower-dimensional descriptors. The decoder
tries to reconstruct the original descriptors from the projected lower-
dimensional descriptors.

###  III-A Unsupervised Reduction

Auto-encoders [3] are unsupervised learning techniques for dimensionality
reduction. Here, we use it to see whether an MLP-based network can learn a
good projection from high to low dimensionality in an unsupervised way. To
achieve fast and lightweight dimensionality reduction, we build our auto-
encoder using MLPs with no more than two hidden layers as shown in Fig. 2. Our
auto-encoder consists of a symmetric encoder and decoder. The encoder projects
the input feature into lower-dimensional embeddings, while the decoder
reconstructs the original input from the lower-dimensional embeddings. By
checking the consistency between inputs and outputs, the auto-encoder learns
how to extract lower-dimensional descriptors from the original ones without
labels. We apply the consistency constraints by minimizing the reconstruction
loss between inputs and outputs. The reconstruction loss $\mathcal{L}^{R}$
measures the differences between the input descriptors
$\\{\boldsymbol{x}_{i}\\}_{i=1}^{N}$ in a training mini-batch and the
reconstructed output descriptors $\\{\boldsymbol{x}_{i}^{\prime}\\}_{i=1}^{N}$
as

$\mathcal{L}^{R}=\frac{1}{N}\sum_{i=1}^{N}\left\|\boldsymbol{x}_{i}-\boldsymbol{x}_{i}^{\prime}\right\|_{2}.$
(1)

We propose an additional distance loss $\mathcal{L}^{D}$ for HardNet. The
distance loss calculates the difference between the distance of the original
high-dimensional descriptors and the distance of the lower-dimensional
descriptors in the embedding space. Given two descriptors $\boldsymbol{x}_{i}$
and $\boldsymbol{x}_{j}$, their corresponding lower-dimensional descriptors in
the embedding space are $\boldsymbol{\hat{x}}_{i}$ and
$\boldsymbol{\hat{x}}_{j}$. The loss $\mathcal{L}^{D}$ is to make the
$\ell_{2}$ distance between the lower-dimensional descriptors
$d\left(\boldsymbol{\hat{x}}_{i},\boldsymbol{\hat{x}}_{j}\right)$ as similar
as possible compared to that between the original descriptors
$d\left(\boldsymbol{x}_{i},\boldsymbol{x}_{j}\right)$

$\mathcal{L}^{D}=\frac{1}{N(N-1)}\sqrt{\sum_{i=1}^{N}\sum_{j\neq
i}\left(d\left(\boldsymbol{x}_{i},\boldsymbol{x}_{j}\right)-d\left(\boldsymbol{\hat{x}}_{i},\boldsymbol{\hat{x}}_{j}\right)\right)^{2}},$
(2)

where
$d\left(\boldsymbol{x}_{i},\boldsymbol{x}_{j}\right)=\|\boldsymbol{x}_{i}-\boldsymbol{x}_{j}\|_{2}$
is the $\ell_{2}$ distance and the loss is calculated on all different pairs
in a mini-batch. The final loss for HardNet is a weighted sum of the
reconstruction and distance losses

$\mathcal{L}=\mathcal{L}^{R}+\alpha\mathcal{L}^{D},$ (3)

more explanation about this design is given in Sec. V.

Note that we need no other information but image patches to train our auto-
encoder. Moreover, the non-linearity of the auto-encoder allows it to learn
better projection than PCA, thus generating better lower-dimensional
descriptors. This will be shown in the experimental evaluation.

###  III-B Self-Supervised Reduction

Figure 3: Pipeline of our self-supervised method. We iteratively cluster
descriptors and use the clustering assignments as pseudo-labels to train the
network.

Unlike unsupervised methods, self-supervised methods usually use traditional
heuristic-based methods to generate pseudo-labels and guide networks to learn
certain tasks. It combines human priors with learning-based methods to achieve
good performance. Inspired by deep feature clustering [10, 22], we propose a
self-supervised method for MLP-based dimensionality reduction. The main idea
is to apply $k$-means clustering with descriptors and use the clustering
assignments as pseudo-labels, i.e., descriptors in the same cluster considered
to have the same label. A classification layer [11] is added during training
to guide our MLPs learning to generate similar lower-dimensional descriptors
if their high-dimensional ones have the same pseudo-label. In the first
training epoch, we cluster the original descriptors into different groups and
use them as labels to supervise our MLPs. From that on, we use the clusters of
the lower-dimensional descriptors from the previous epoch as supervision.
Given a set of descriptors extracted from image patches, $k$-means clustering
generates a centroid matrix $\boldsymbol{C}$ and the clustering assignments
$\boldsymbol{y}_{i}$ for each descriptor $\boldsymbol{x}_{i}$ by solving

$\small{\min_{C\in\mathbb{R}^{d\times
k}}\frac{1}{N}\sum_{i=1}^{N}\min_{\boldsymbol{y}_{i}\in\\{0,1\\}^{k}}\left\|\boldsymbol{x}_{i}-\boldsymbol{C}\boldsymbol{y}_{i}\right\|_{2}^{2}~{},~{}\textrm{s.t.}~{}\boldsymbol{y}_{i}^{\top}\boldsymbol{1}_{k}=1,}$
(4)

where $\boldsymbol{1}_{k}$ is a vector whose all elements are $1$ and
$\boldsymbol{y}_{i}$ is a one-hot vector.

The training loss used for our self-supervised method is the standard cross-
entropy loss with the pseudo-labels as targets.

###  III-C Supervised Reduction

Figure 4: Pipeline of the supervised method. A training sample includes an
anchor with positive and negative patches. The extracted descriptors are fed
into MLPs to get lower-dimensional descriptors and a triplet loss is applied
to supervise the MLPs.

We next introduce our supervised method for dimensionality reduction. As shown
in Fig. 4, we use ground truth patch labels together with the triplet loss to
train the MLP. A batch of matching patches is denoted as
${\\{{{{a}_{i}},{p}_{i}}\\}}_{{i=1\dots N}}$, where $a$ stands for the anchor
patch and $p$ for the positive patch. The non-matching negative patches
${\\{{{{n}_{i}}}\\}}_{{i=1\dots N}}$ are sampled by the hardest-within-batch
strategy as introduced by Mishchuk et al. [29]. For all training patches, we
first use existing methods to generate higher-dimensional descriptors and feed
them to our MLPs to generate low-dimensional descriptors where the triplet
margin loss is applied

$\small{\mathcal{L}^{T}=\frac{1}{N}\sum_{i=1}^{N}{\max{(0,m+d(f(a_{i}),f(p_{i}))-d(f(a_{i}),f(n_{i})))}},}$
(5)

where $f$ is the MLP-based projection. The main idea is to learn an MLP-based
projection $f$ such that the distance between the anchor and positive
descriptors $d(f(a_{i}),f(p_{i}))$ is smaller than that between the anchor and
negative descriptors $d(f(a_{i}),f(n_{i}))$ with a margin $m$ in the lower-
dimensional embedding space. We also add a distance loss term for HardNet as
used in the auto-encoder to target for the similarity of the distance between
input descriptors and embedding descriptors in a batch. The final loss for
HardNet is a weighted sum of the triplet margin and distance losses

$\mathcal{L}=\mathcal{L}^{T}+\beta\mathcal{L}^{D}.$ (6)

###  III-D Training and Parameters

We use the UBC Phototour Liberty dataset [8] for training. After training our
networks on this dataset, we apply the trained model to other tasks and
datasets without fine-tuning.

For the auto-encoder, we train the network for $5$ epochs using Adam [21] with
a learning rate of $0.001$ and a batch size of $1024$. We choose the distance
loss weighting factor $\alpha=0.1$ for HardNet. For the self-supervised
method, we train the network for $200$ epochs using Adam with a learning rate
of $0.001$ and a batch size of $256$. The number of cluster for $k$-means is
$100\,000$. The clustering is repeated and the classification layer is re-
initialized every $10$ epochs. For the supervised method, we choose the margin
$m=1$ and train the network for $10$ epochs using Adam with a learning rate of
$0.001$ and a batch size of $1024$. The learning rate is linearly decayed to
zero within $10$ epochs. We choose the weighting of the distance loss
$\beta=3$ for HardNet. We use ReLU followed by batch normalization [18] after
each linear layer except the last one. The embeddings are
$\ell_{2}$-normalized. For hand-crafted descriptors, we use two hidden layers
for our MLPs, while for learning-based ones we use only one hidden layer. More
detailed parameters and network architectures for each method can be found in
our open-source implementation. For PCA as the baseline, we use the
implementation from scikit-learn [31].

##  IV Experiments

We present our experiments to show the capabilities of our MLP-based methods
for different tasks. We choose SIFT [24], MKD [30], TFeat [5], and HardNet
[29] as the base descriptors. Their original dimensions are all $128$. We
convert these descriptors to lower dimensions of $64$, $32$, $24$, and $16$
and apply them on three publicly available datasets, HPatches [4], Aachen Day-
Night v1.1 [42], and InLoc [39] for different downstream tasks, including
visual localization, patch verification, image matching, and patch retrieval.
We name our unsupervised method ‘Ours-US’, the self-supervised method ‘Ours-
SS’, and the supervised method ‘Ours-SV’ in all experiments.

| Aachen Day-Night v1.1 | InLoc
---|---|---
| day | night | duc1 | duc2
| distance [m]
---
orient. [deg]
| 0.25 / 0.5 / 5.0
---
2 / 5 / 10
| 0.5 / 1.0 / 5.0
---
2 / 5 / 10
| 0.25 / 0.5 / 1.0
---
N / A
| 0.25 / 0.5 / 1.0
---
N / A
SIFT | 88.1 / 94.7 / 98.4 | 64.9 / 77.5 / 92.1 |  31.3 / 46.0 / 56.1 | 21.4 / 33.6 / 43.5
PCA-16 | 84.5 / 90.7 / 95.9 |  38.2 / 47.1 / 58.6 | 19.7 / 32.3 / 37.9 | 11.5 / 20.6 / 25.2
Ours-US-16 |  84.5 / 92.4 / 96.0 |  35.6 / 45.0 / 60.7 | 22.2 / 33.8 / 39.4 | 13.0 / 20.6 / 24.4
Ours-SS-16 | 85.1 / 90.9 / 95.6 | 34.0 / 41.9 / 57.6 | 23.7 / 35.4 / 41.4 |  16.0 / 22.1 / 26.0
Ours-SV-16 |  85.9 / 91.5 / 96.2 | 34.6 / 46.1 / 58.6 |  24.2 / 36.9 / 43.9 |  15.3 / 26.7 / 30.5
PCA-24 |  86.2 / 93.2 / 97.5 | 50.3 / 60.7 / 77.5 | 24.2 / 34.8 / 46.5 | 15.3 / 26.0 / 29.8
Ours-US-24 | 86.8 / 93.3 / 97.2 |  50.3 / 64.4 / 78.5 | 24.2 / 39.9 / 48.5 |  21.4 / 29.8 / 35.9
Ours-SS-24 | 87.7 / 93.4 / 97.3 | 48.7 / 59.7 / 74.9 | 25.8 / 42.4 / 48.5 | 17.6 / 26.7 / 32.8
Ours-SV-24 |  87.9 / 93.8 / 97.3 |  51.3 / 62.8 / 78.5 |  30.8 / 43.4 / 52.0 |  21.4 / 34.4 / 41.2
PCA-32 | 87.1 / 93.0 / 97.6 | 56.5 / 69.1 / 80.6 | 28.3 / 40.9 / 53.5 | 21.4 / 29.0 / 35.9
Ours-US-32 | 87.0 / 94.1 / 97.8 |  58.1 / 71.2 / 83.2 | 25.8 / 40.9 / 52.5 | 19.8 / 32.1 / 37.4
Ours-SS-32 | 87.9 / 93.8 / 98.1 | 57.1 / 66.0 / 82.7 |  30.8 / 43.9 / 55.6 | 24.4 / 32.8 / 38.2
Ours-SV-32 |  88.7 / 94.4 / 98.3 |  58.1 / 71.7 / 82.7 | 29.8 / 42.4 / 54.5 |  27.5 / 38.9 / 44.3
PCA-64 | 87.1 / 94.3 / 98.2 | 60.2 / 74.3 / 88.5 | 28.8 / 41.4 / 53.0 | 19.8 / 31.3 / 40.5
Ours-US-64 |  87.6 / 95.3 / 98.5 |  66.0 / 79.6 / 90.6 | 31.3 / 43.9 / 57.1 |  26.7 / 37.4 / 45.0
Ours-SS-64 | 87.9 / 94.9 / 98.2 | 60.2 / 75.9 / 89.0 | 30.8 / 43.4 / 54.5 |  23.7 / 39.7 / 45.8
Ours-SV-64 |  89.1 / 94.8 / 98.8 |  63.4 / 79.6 / 92.7 |  33.3 / 46.0 / 57.6 |  26.0 / 39.7 / 45.8
TABLE I: Evaluation of the localization on the Aachen Day-Night v1.1 and InLoc datasets with SIFT [24] features. We report the recall [%] at different distances and orientation thresholds. The overall best results are in red and the best results with the same low dimension are in blue.  | Aachen Day-Night v1.1 | InLoc
---|---|---
| day | night | duc1 | duc2
| distance [m]
---
orient. [deg]
| 0.25 / 0.5 / 5.0
---
2 / 5 / 10
| 0.5 / 1.0 / 5.0
---
2 / 5 / 10
| 0.25 / 0.5 / 1.0
---
N / A
| 0.25 / 0.5 / 1.0
---
N / A
MKD | 88.7 / 95.1 / 98.8 |  67.0 / 79.6 / 92.7 |  33.3 / 50.5 / 65.2 | 26.0 / 40.5 / 49.6
PCA-16 |  85.7 / 92.5 / 96.1 |  37.2 / 45.0 / 56.0 | 23.7 / 35.4 / 42.9 | 17.6 / 25.2 / 28.2
Ours-US-16 | 84.6 / 91.3 / 95.5 | 34.0 / 41.4 / 52.4 | 22.2 / 32.3 / 39.4 | 14.5 / 19.8 / 25.2
Ours-SS-16 | 85.0 / 92.2 / 96.5 | 36.6 / 44.0 / 55.5 |  24.2 / 34.8 / 43.9 | 14.5 / 26.0 / 31.3
Ours-SV-16 |  85.7 / 93.0 / 96.8 |  37.7 / 42.9 / 53.9 |  23.2 / 35.9 / 45.5 |  21.4 / 28.2 / 32.8
PCA-24 | 87.4 / 93.8 / 97.5 | 49.2 / 59.2 / 70.7 | 26.8 / 37.9 / 48.0 | 20.6 / 31.3 / 37.4
Ours-US-24 | 87.6 / 93.8 / 97.2 | 47.6 / 60.2 / 73.8 | 24.7 / 38.4 / 49.0 | 18.3 / 27.5 / 35.1
Ours-SS-24 |  88.0 / 94.1 / 97.9 |  51.3 / 64.4 / 76.4 |  27.8 / 40.4 / 51.5 | 18.3 / 30.5 / 38.2
Ours-SV-24 | 87.5 / 93.7 / 97.3 |  53.9 / 63.9 / 75.4 |  29.3 / 40.4 / 53.5 |  22.9 / 35.9 / 39.7
PCA-32 | 87.6 / 93.9 / 98.2 | 52.4 / 67.0 / 81.7 | 30.8 / 42.4 / 51.5 | 26.0 / 36.6 / 43.5
Ours-US-32 | 87.6 / 94.2 / 98.4 |  59.2 / 71.2 / 85.9 | 27.8 / 40.9 / 55.6 | 27.5 / 37.4 / 42.0
Ours-SS-32 |  88.6 / 94.7 / 98.2 | 56.0 / 70.7 / 83.2 |  30.3 / 44.4 / 57.6 |  28.2 / 42.7 / 47.3
Ours-SV-32 |  88.3 / 94.5 / 98.7 |  60.7 / 73.8 / 85.9 |  31.3 / 43.4 / 56.6 | 26.7 / 38.9 / 46.6
PCA-64 | 88.1 / 94.9 / 98.5 |  62.3 / 80.6 / 92.1 |  31.3 / 47.5 / 61.1 | 26.0 / 41.2 / 48.1
Ours-US-64 | 88.6 / 94.5 / 98.5 |  65.4 / 80.6 / 92.7 | 30.8 / 47.5 / 59.6 | 28.2 / 42.0 / 48.1
Ours-SS-64 |  88.8 / 95.6 / 98.9 | 62.3 / 79.6 / 92.1 |  31.3 / 44.9 / 58.1 |  30.5 / 44.3 / 50.4
Ours-SV-64 |  89.0 / 95.1 / 98.5 | 63.4 / 79.6 / 90.1 |  31.3 / 49.5 / 61.6 |  28.2 / 45.0 / 51.9
TABLE II: Evaluation of the localization on the Aachen Day-Night v1.1 and InLoc datasets with MKD [30] features.  | Aachen Day-Night v1.1 | InLoc
---|---|---
| day | night | duc1 | duc2
| distance [m]
---
orient. [deg]
| 0.25 / 0.5 / 5.0
---
2 / 5 / 10
| 0.5 / 1.0 / 5.0
---
2 / 5 / 10
| 0.25 / 0.5 / 1.0
---
N / A
| 0.25 / 0.5 / 1.0
---
N / A
TFeat | 87.4 / 93.9 / 98.2 | 53.4 / 72.8 / 83.8 |  32.8 / 44.9 / 55.6 | 24.4 / 41.2 / 45.8
PCA-16 | 82.6 / 89.4 / 94.7 |  30.4 / 37.7 / 49.7 | 20.7 / 29.8 / 39.9 |  16.0 / 24.4 / 29.8
Ours-US-16 |  84.5 / 91.1 / 95.5 |  30.4 / 39.8 / 49.7 | 21.2 / 31.3 / 39.9 | 13.7 / 19.1 / 24.4
Ours-SS-16 | 84.0 / 89.9 / 95.4 |  31.4 / 36.6 / 47.1 | 21.2 / 30.3 / 37.9 | 10.7 / 19.8 / 22.1
Ours-SV-16 |  83.9 / 90.8 / 95.6 | 28.8 / 34.0 / 47.1 |  21.7 / 32.3 / 41.9 | 14.5 / 20.6 / 24.4
PCA-24 | 86.3 / 93.2 / 96.6 |  42.9 / 55.0 / 67.0 | 23.7 / 38.9 / 49.0 | 20.6 / 30.5 / 35.1
Ours-US-24 |  86.3 / 93.8 / 97.1 | 42.4 / 55.0 / 63.4 | 26.3 / 37.9 / 48.0 | 19.1 / 31.3 / 35.9
Ours-SS-24 |  86.8 / 92.7 / 96.4 |  36.6 / 46.6 / 67.5 | 27.3 / 38.9 / 48.0 |  22.9 / 32.1 / 36.6
Ours-SV-24 | 86.0 / 92.5 / 96.8 |  42.9 / 56.0 / 67.5 |  27.8 / 39.4 / 50.5 | 19.1 / 28.2 / 34.4
PCA-32 | 87.3 / 93.7 / 97.2 | 47.1 / 61.8 / 71.7 | 25.8 / 38.9 / 52.0 |  22.9 / 33.6 / 42.7
Ours-US-32 |  87.6 / 94.3 / 98.1 |  48.2 / 61.3 / 75.4 |  30.3 / 40.9 / 53.0 |  22.1 / 35.9 / 40.5
Ours-SS-32 | 87.3 / 93.1 / 97.7 | 45.5 / 59.2 / 73.3 |  30.3 / 40.4 / 50.5 | 22.9 / 35.1 / 41.2
Ours-SV-32 |  88.1 / 94.2 / 98.1 |  48.2 / 62.3 / 72.8 |  30.3 / 41.4 / 51.5 |  23.7 / 35.1 / 39.7
PCA-64 |  88.2 / 94.4 / 98.2 | 52.9 / 70.2 / 79.6 | 29.3 / 43.4 / 54.0 | 28.2 / 38.9 / 46.6
Ours-US-64 | 87.6 / 93.9 / 98.3 |  55.0 / 73.3 / 84.8 | 29.3 / 41.9 / 58.6 |  26.0 / 39.7 / 50.4
Ours-SS-64 |  89.0 / 93.9 / 98.7 |  56.5 / 69.1 / 82.7 |  33.3 / 44.4 / 60.1 | 23.7 / 39.7 / 46.6
Ours-SV-64 |  87.0 / 94.4 / 98.4 |  55.0 / 71.2 / 84.8 |  32.3 / 44.4 / 57.1 |  32.1 / 44.3 / 50.4
TABLE III: Evaluation of the localization on the Aachen Day-Night v1.1 and InLoc datasets with TFeat [5] features.  | Aachen Day-Night v1.1 | InLoc
---|---|---
| day | night | duc1 | duc2
| distance [m]
---
orient. [deg]
| 0.25 / 0.5 / 5.0
---
2 / 5 / 10
| 0.5 / 1.0 / 5.0
---
2 / 5 / 10
| 0.25 / 0.5 / 1.0
---
N / A
| 0.25 / 0.5 / 1.0
---
N / A
HardNet |  88.8 / 95.6 / 99.3 |  63.9 / 81.2 / 92.7 | 37.9 / 56.6 / 70.7 | 31.3 / 44.3 / 53.4
PCA-16 |  84.7 / 91.6 / 96.2 | 27.2 / 34.6 / 46.1 |  21.7 / 36.4 / 43.9 |  15.3 / 24.4 / 29.0
Ours-US-16 | 84.1 / 91.1 / 95.0 | 29.8 / 34.0 / 45.0 | 20.7 / 33.3 / 40.4 | 12.2 / 19.8 / 24.4
Ours-SS-16 | 84.0 / 90.7 / 96.0 | 28.3 / 35.1 / 42.4 | 22.7 / 32.8 / 41.9 | 12.2 / 19.8 / 22.9
Ours-SV-16 | 84.0 / 91.3 / 95.8 |  30.4 / 36.1 / 47.6 |  24.7 / 34.8 / 42.4 | 13.0 / 22.9 / 26.7
PCA-24 |  86.9 / 93.4 / 98.3 | 44.5 / 55.0 / 69.6 |  29.3 / 42.9 / 53.0 |  22.1 / 35.9 / 38.9
Ours-US-24 |  87.4 / 94.1 / 97.2 |  40.8 / 58.6 / 69.6 | 28.3 / 38.9 / 52.5 | 18.3 / 29.0 / 35.9
Ours-SS-24 | 86.4 / 93.6 / 97.6 | 46.1 / 57.1 / 66.5 |  29.8 / 41.4 / 50.0 |  22.9 / 32.8 / 35.9
Ours-SV-24 |  87.4 / 94.1 / 97.8 |  48.7 / 58.6 / 71.2 |  25.8 / 42.9 / 55.6 | 21.4 / 31.3 / 36.6
PCA-32 |  88.0 / 94.8 / 98.9 |  55.0 / 69.6 / 82.7 | 32.3 / 49.0 / 60.1 |  28.2 / 38.9 / 44.3
Ours-US-32 | 87.9 / 94.9 / 98.7 | 53.9 / 65.4 / 81.2 |  33.8 / 48.5 / 59.1 | 21.4 / 37.4 / 46.6
Ours-SS-32 |  88.6 / 94.8 / 98.3 | 53.9 / 68.1 / 79.6 | 33.3 / 48.5 / 59.6 | 25.2 / 38.2 / 45.0
Ours-SV-32 |  88.3 / 95.1 / 98.9 |  55.0 / 69.1 / 84.8 |  33.3 / 51.0 / 61.1 |  28.2 / 42.0 / 47.3
PCA-64 |  88.7 / 95.6 / 99.3 |  60.7 / 78.5 / 93.2 | 35.4 / 56.1 / 68.2 | 30.5 / 45.0 / 53.4
Ours-US-64 | 88.8 / 95.5 / 99.2 |  57.6 / 80.1 / 92.1 |  40.9 / 56.6 / 71.2 |  32.1 / 45.8 / 51.9
Ours-SS-64 | 89.1 / 95.3 / 99.2 | 61.3 / 79.6 / 91.1 | 35.9 / 53.0 / 66.7 |  31.3 / 45.8 / 55.7
Ours-SV-64 |  89.2 / 95.9 / 99.2 |  62.8 / 79.1 / 92.1 |  38.9 / 58.6 / 70.2 |  30.5 / 48.1 / 54.2
TABLE IV: Evaluation of the localization on the Aachen and InLoc datasets with
HardNet [29] features.

###  IV-A Visual Localization

In the first experiment, we evaluate how well our reduced features perform in
robot visual localization tasks. We analyze different methods using two
challenging localization datasets, with severe illumination changes and
complex indoor scenes. We use the hierarchical localization toolbox [34] to
achieve visual localization, but replace the feature extractors with generated
lower-dimensional descriptors.

For the first day-night localization challenge, we evaluate different
dimensionality reduction methods on the Aachen Day-Night v1.1 dataset [42]. It
contains $6\,697$ day-time database images from an old European town and
$1\,015$ queries ($824$ taken in day and $191$ in night conditions). We use
the code and evaluation protocol from [42] and report the percentage of day-
night queries localized within a given error bound on the estimated camera
position and orientation. For the complex indoor localization challenge, we
exploit the InLoc dataset [39]. It is a challenging indoor localization
dataset with large differences in viewpoint and illumination between the query
and database images. We also use the code and evaluation protocol from [42]
and report the percentage of queries localized within a given error bound on
the estimated camera position.

SIFT-PCA-64

SIFT-Ours-SV-64

Figure 5: Localization with SIFT-PCA-64 and SIFT-Ours-SV-64 on Aachen Day-
Night v1.1 and InLoc. For each image pair, the left image is the query and the
right image is the retrieved database image with the most inlier matches, as
returned by PnP+RANSAC. Red lines represent wrong feature matches and red
boxes represent wrong localization results. Green lines and boxes are correct.
Best view in color and zoom in.

Tables I to IV report the quantitative localization results and Fig. 5 shows
the qualitative results. For all four descriptors, our MLP-based method with
auto-encoder, self-supervised, and supervised learning schemes perform better
in terms of visual localization than PCA and even the original descriptors for
most queries and lower dimensions. This indicates that the learned lower-
dimensional descriptors are more distinctive and invariant under challenging
environments, which improves the visual localization performance. Besides, the
degree of improvement for hand-crafted descriptors is larger than the learned
ones. We will analyze and discuss this phenomenon in Sec. V.

Figure 6: Verification, matching, and retrieval results of SIFT on test set
‘a’ of HPatches dataset. None of the MLPs is trained on HPatches. Different
colors represent different difficulties and the numbers are the average mAP
values.

###  IV-B Applications on Patch Verification, Image Matching, and Patch
Retrieval

This experiment shows more robotics applications using our dimensionality-
reduced features, including the patch pair verification, image matching, and
patch retrieval tasks on the HPatches dataset [4]. There are $116$ sequences
with over $1.5$ million patches in this dataset. $59$ sequences of them show
significant viewpoint changes and $57$ sequences have significant illumination
changes. The patches are divided into three groups: easy, hard, and tough,
based on the levels of geometric noise. Evaluation results with SIFT is shown
in Fig. 6. Note that, all models of our MLP are pre-trained _only_ on Liberty
sequence of UBC Phototour and we test them directly on HPatches dataset in a
zero-shot fashion to show the good generalization of our proposed MLP-based
method.

As can be seen from Fig. 6, our methods with supervised and self-supervised
learning schemes perform better than PCA and auto-encoder for all three tasks
and all lower dimensions with a large margin. Besides, we also observe that
the $64$-dimensional descriptors generated by our method even outperform the
original $128$-dimensional descriptors. Even the learned $24$-dimensional
descriptors have the on par performances as the original $128$-dimensional
SIFT descriptors, which shows the superiority of our proposed learned MLP-
based dimensionality reduction. More explanation will be given in Sec. V.

###  IV-C Ablation Study

In this section, we perform several experiments to provide a more in-depth
analysis of how each component in MLP contributes to the final performance. We
choose different numbers of hidden layers ($0$, $1$, and $2$) with different
sizes of $96$, $128$, $256$, $512$, and $1\,024$. We make evaluations on the
matching task of the HPatches benchmark. We select the lower dimension of $64$
for all experiments, and Fig. 7 shows the ablation study results with SIFT and
HardNet. We can see that the more number and size of hidden layers, the better
the results for the hand-crafted descriptor. However, for the learned
descriptor, more hidden layers do not help.

Figure 7: Impact of the number and the size of hidden layers for image
matching mAP on Hatches dataset. For SIFT, the more number and size of hidden
layers, the better the results are for most cases. For HardNet, the trend is
opposite.

###  IV-D Runtime and Memory Evaluation

For each image patch, our MLP-based method adds less than $0.002\,\mathrm{ms}$
extra computational time compared to calculating the original descriptors,
which can be basically ignored. Meanwhile, the lower-dimensional descriptors
can save the memory $2$, $4$, $5$, and $8$ times respectively for the $64$,
$32$, $24$, and $16$-dimensional descriptors, which shows the advantages of
the proposed MLP-based method.

##  V Discussion

From the above experiments, we find that our learned MLP projections using
unsupervised, supervised, and self-supervised methods achieve better
performances than reduction using PCA. Furthermore, our features are partially
even better than the original 128-dimensional ones. The degree of improvement
for different descriptors is different. A key result is that the improvement
for hand-crafted descriptors, like SIFT and MKD, is larger than those of
learned ones, like TFeat and HardNet. Moreover, the performance with hand-
crafted descriptors benefits from complex network architectures, while for
learned ones, simple network architectures achieve better results.

We interpret the results as follows. For SIFT and MKD, the descriptor space is
not optimized for the $\ell_{2}$ metric because of its hand-crafted design,
and there are overlaps between non-matching patches. When applying PCA
directly on them, matching and non-matching patches will still overlap.
However, after learning a more discriminative representation using triplet
loss, the projected descriptor space will be more distinctive for the lower-
dimensional descriptors. This might also be why our method’s performance on
hand-crafted descriptors benefits from the complex network architecture - more
parameters are needed to rearrange the descriptor space. This is also the
reason that we do not need additional distance loss for hand-crafted ones.

For learning-based descriptors, such as HardNet and TFeat, a relatively
discriminative descriptor space has already been learned using triplet loss.
Similar features are close in the descriptor space, otherwise further apart.
Therefore, this kind of distinctive distance information may also be preserved
after the PCA projection. This might be the reason why different
dimensionality reduction methods perform similarly on learning-based
descriptors. However, since the descriptor space is already quite regular, it
is very easy for an MLP to overfit. Thus simple architectures obtain better
results for dimensionality reduction. We added an additional distance loss for
HardNet to restrict the learned descriptor space and avoid overfitting. Since
HardNet is trained with more advanced hard-negative mining techniques than
TFeat, the performance of TFeat can still be improved significantly using our
supervised MLP-based method. Due to the page limitation, we put more results
and discussion in our code repository: https://github.com/PRBonn/descriptor-
dr.

##  VI Conclusion

In this paper, we investigated an MLP-based network with unsupervised, self-
supervised, and supervised learning schemes for dimensionality reduction of
local feature descriptors. We thoroughly evaluate our method on four
descriptors including hand-crafted and learning-based on multiple datasets
with various downstream tasks. The experimental results show that our MLP-
based projections work better than PCA in challenging tasks, including visual
localization, patch verification, image matching, and patch retrieval for most
cases. Besides, the $64$-dimensional descriptors generated from learning-based
projections even outperform the original $128$-dimensional descriptors. We
also provided ablation studies and analyzed the degree of improvement for
different descriptors in terms of the distribution of the descriptor space.
Additional memory and runtime experiments show that learned lower-dimensional
descriptors can be used for saving memory consumption without adding extra
runtime and thus are useful for real-world robotics applications.

## References

  * [1] A. Alahi, R. Ortiz, and P. Vandergheynst.  Freak: Fast retina keypoint.  In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2012.
  * [2] S. Balakrishnama and A. Ganapathiraju.  Linear discriminant analysis-a brief tutorial.  Institute for Signal and information Processing, 18(1998):1–8, 1998\.
  * [3] P. Baldi.  Autoencoders, unsupervised learning, and deep architectures.  In Proc. of the Int. Conf. on Machine Learning (ICML), 2012.
  * [4] V. Balntas, K. Lenc, A. Vedaldi, and K. Mikolajczyk.  Hpatches: A benchmark and evaluation of handcrafted and learned local descriptors.  In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [5] V. Balntas, E. Riba, D. Ponsa, and K. Mikolajczyk.  Learning local feature descriptors with triplets and shallow convolutional neural networks.  In Proc. of British Machine Vision Conference (BMVC), 2016.
  * [6] H. Bay, T. Tuytelaars, and L. Van Gool.  Surf: Speeded up robust features.  In Proc. of the Europ. Conf. on Computer Vision (ECCV), 2006.
  * [7] L. Breiman.  Random forests.  Machine learning, 45(1):5–32, 2001.
  * [8] M. Brown, G. Hua, and S. Winder.  Discriminative learning of local image descriptors.  IEEE Trans. on Pattern Analalysis and Machine Intelligence (TPAMI), 33(1):43–57, 2011.
  * [9] M. Calonder, V. Lepetit, M. Ozuysal, T. Trzcinski, C. Strecha, and P. Fua.  Brief: Computing a local binary descriptor very fast.  IEEE Trans. on Pattern Analalysis and Machine Intelligence (TPAMI), 34(7):1281–1298, 2012.
  * [10] M. Caron, P. Bojanowski, A. Joulin, and M. Douze.  Deep clustering for unsupervised learning of visual features.  In Proc. of the Europ. Conf. on Computer Vision (ECCV), 2018.
  * [11] J. Deng, J. Guo, N. Xue, and S. Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [12] P. Ebel, A. Mishchuk, K.M. Yi, P. Fua, and E. Trulls.  Beyond cartesian representations for local descriptors.  In Proc. of the IEEE Intl. Conf. on Computer Vision (ICCV), 2019\.
  * [13] B. Fan, Z. Wang, and F. Wu.  Local Image Descriptor: Modern Approaches, volume 108.  Springer, 2016.
  * [14] R. Fergus, P. Perona, and A. Zisserman.  Object class recognition by unsupervised scale-invariant learning.  In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2003.
  * [15] A. Gil, O. Reinoso, O. Martínez-Mozos, C. Stachniss, and W. Burgard.  Improving Data Association in Vision-based SLAM.  In Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), 2006.
  * [16] R.E. González Valenzuela, W.R. Schwartz, and H. Pedrini.  Dimensionality reduction through pca over sift and surf descriptors.  In Proc. of the IEEE Intl. Conf. on Cybernetic Intelligent Systems (CIS), 2012.
  * [17] X. Han, T. Leung, Y. Jia, R. Sukthankar, and A.C. Berg.  Matchnet: Unifying feature and metric learning for patch-based matching.  In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [18] S. Ioffe and C. Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In Proc. of the Int. Conf. on Machine Learning (ICML), pages 448–456. PMLR, 2015.
  * [19] I.T. Jolliffe.  Principal Component Analysis.  Springer, 2002.
  * [20] Y. Ke and R. Sukthankar.  Pca-sift: a more distinctive representation for local image descriptors.  In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2004.
  * [21] D.P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [22] M. Larsson, E. Stenborg, C. Toft, L. Hammarstrand, T. Sattler, and F. Kahl.  Fine-grained segmentation networks: Self-supervised segmentation for improved long-term visual localization.  In Proc. of the IEEE Intl. Conf. on Computer Vision (ICCV), 2019\.
  * [23] A. Loquercio, M. Dymczyk, B. Zeisl, S. Lynen, I. Gilitschenski, and R. Siegwart.  Efficient descriptor learning for large scale localization.  In Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA), 2017.
  * [24] D.G. Lowe.  Distinctive image features from scale-invariant keypoints.  In Proc. of the IEEE Intl. Conf. on Computer Vision (ICCV), 2004\.
  * [25] Z. Luo, T. Shen, L. Zhou, S. Zhu, R. Zhang, Y. Yao, T. Fang, and L. Quan.  Geodesc: Learning local descriptors by integrating geometry constraints.  In Proc. of the Europ. Conf. on Computer Vision (ECCV), 2018.
  * [26] J. Ma, X. Jiang, A. Fan, J. Jiang, and J. Yan.  Image matching from handcrafted to deep features: A survey.  Intl. Journal of Computer Vision (IJCV), 129(1):23–79, 2021.
  * [27] K.Z. Mao.  Orthogonal forward selection and backward elimination algorithms for feature subset selection.  IEEE Trans. on Systems, Man, and Cybernetics, Part B (Cybernetics), 34(1):629–634, 2004.
  * [28] K. Mikolajczyk and C. Schmid.  Indexing based on scale invariant interest points.  In Proc. of the IEEE Intl. Conf. on Computer Vision (ICCV), 2001\.
  * [29] A. Mishchuk, D. Mishkin, F. Radenovic, and J. Matas.  Working hard to know your neighbor’s margins: Local descriptor learning loss.  In Proc. of the Advances in Neural Information Processing Systems (NIPS), 2017.
  * [30] A. Mukundan, G. Tolias, A. Bursuc, H. Jégou, and O. Chum.  Understanding and improving kernel local descriptors.  Intl. Journal of Computer Vision (IJCV), 127:1723–1737, 2019.
  * [31] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12:2825–2830, 2011.
  * [32] T.M. Rassias.  Properties of isometric mappings.  Journal of Mathematical Analysis and Applications, 235(1):108–121, 1999.
  * [33] E. Rublee, V. Rabaud, K. Konolige, and G. Bradski.  Orb: An efficient alternative to sift or surf.  In Proc. of the IEEE Intl. Conf. on Computer Vision (ICCV), 2011\.
  * [34] P.E. Sarlin, C. Cadena, R. Siegwart, and M. Dymczyk.  From coarse to fine: Robust hierarchical localization at large scale.  In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [35] T. Sattler, B. Leibe, and L. Kobbelt.  Efficient amp; effective prioritized matching for large-scale image-based localization.  IEEE Trans. on Pattern Analalysis and Machine Intelligence (TPAMI), 39(9):1744–1756, 2017.
  * [36] B. Schölkopf, A. Smola, and K.R. Müller.  Kernel principal component analysis.  In Proc. of the Intl. Conf. on artificial neural networks (ICANN), 1997.
  * [37] E. Simo-Serra, E. Trulls, L. Ferraz, I. Kokkinos, P. Fua, and F. Moreno-Noguer.  Discriminative learning of deep convolutional feature point descriptors.  In Proc. of the IEEE Intl. Conf. on Computer Vision (ICCV), 2015\.
  * [38] K. Simonyan, A. Vedaldi, and A. Zisserman.  Learning local feature descriptors using convex optimisation.  IEEE Trans. on Pattern Analalysis and Machine Intelligence (TPAMI), 36(8):1573–1585, 2014.
  * [39] H. Taira, M. Okutomi, T. Sattler, M. Cimpoi, M. Pollefeys, J. Sivic, T. Pajdla, and A. Torii.  InLoc: Indoor Visual Localization with Dense Matching and View Synthesis.  In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [40] Y. Tian, B. Fan, F. Wu, et al.  L2-net: Deep learning of discriminative patch descriptor in euclidean space.  In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [41] Y. Tian, X. Yu, B. Fan, F. Wu, H. Heijnen, and V. Balntas.  Sosnet: Second order similarity regularization for local descriptor learning.  In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [42] C. Toft, W. Maddern, A. Torii, L. Hammarstrand, E. Stenborg, D. Safari, M. Okutomi, M. Pollefeys, J. Sivic, T. Pajdla, F. Kahl, and T. Sattler.  Long-term visual localization revisited.  IEEE Trans. on Pattern Analalysis and Machine Intelligence (TPAMI), 44(4):2074–2088, 2022.
  * [43] L. van der Maaten and G. Hinton.  Visualizing data using t-SNE.  Journal of Machine Learning Research, 9:2579–2605, 2008.
  * [44] M.W. Watkins.  Exploratory factor analysis: A guide to best practice.  Journal of Black Psychology, 44(3):219–246, 2018.
  * [45] S. Zagoruyko and N. Komodakis.  Learning to compare image patches via convolutional neural networks.  In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2015.