# OverlapTransformer: An Efficient and Yaw-Angle-Invariant
Transformer Network for LiDAR-Based Place Recognition

Junyi Ma, Jun Zhang, Jintao Xu, Rui Ai, Weihao Gu, Xieyuanli Chen∗ Manuscript
received: February 21, 2022; Revised: April 26, 2022; Accepted: May 19, 2022.
This paper was recommended for publication by Editor Javier Civera upon
evaluation of the Associate Editor and Reviewers’ comments. J. Ma is with the
Beijing Institute of Technology. J. Ma, J. Zhang, J. Xu, R. Ai and W. Gu are
with HAOMO.AI Technology Co., Ltd.∗corresponding author email:
<EMAIL_ADDRESS>work has partially been funded by the HAOMO.AI
Technology Co. Ltd., and by the Chinese Scholarship Committee. Digital Object
Identifier (DOI): see top of this page.

###### Abstract

Place recognition is an important capability for autonomously navigating
vehicles operating in complex environments and under changing conditions. It
is a key component for tasks such as loop closing in SLAM or global
localization. In this paper, we address the problem of place recognition based
on 3D LiDAR scans recorded by an autonomous vehicle. We propose a novel
lightweight neural network exploiting the range image representation of LiDAR
sensors to achieve fast execution with less than 2 ms per frame. We design a
yaw-angle-invariant architecture exploiting a transformer network, which
boosts the place recognition performance of our method. We evaluate our
approach on the KITTI and Ford Campus datasets. The experimental results show
that our method can effectively detect loop closures compared to the state-of-
the-art methods and generalizes well across different environments. To
evaluate long-term place recognition performance, we provide a novel dataset
containing LiDAR sequences recorded by a mobile robot in repetitive places at
different times. The implementation of our method and dataset are released
here: https://github.com/haomo-ai/OverlapTransformer

###### Index Terms:

SLAM; Deep Learning Methods; Data Sets for Robot Learning

##  I Introduction

Place recognition plays an essential role for autonomously navigating systems.
In contrast to camera-based place recognition [1, 2, 3, 4, 5], LiDAR-based
place recognition [6, 7, 8, 9] is comparably robust to day-and-night light
changes and different weather conditions. Therefore, LiDARs are an attractive
sensing modality that can be used for autonomous driving in outdoor large-
scale environments. LiDAR-based place recognition is the task to determine if
the LiDAR sensor is currently in a place that has been visited before by
comparing the current LiDAR observations with the map or a database of
formerly taken observations. It supports tasks such as simultaneous
localization and mapping (SLAM) to find loop closure candidates and global
localization to obtain an initial guess of the robot’s position.

In this paper, we propose a novel place recognition method utilizing range
images produced by 3D LiDARs installed on an autonomous vehicle. The range
image is a natural representation of a single 3D scan from a rotating LiDAR
sensor such as Velodyne or Ouster sensors. It is a compact representation and
is especially suitable for online tasks such as online SLAM [10], loop closing
[11, 6], or localization [12, 13] due to its image-like structure. Instead of
using handcrafted descriptors [14, 15, 16], we propose a new transformer
neural network to extract yaw-angle-invariant global descriptors from LiDAR
scans. We apply a similar overlap concept as described in OverlapNet [11, 6]
to supervise the network learning and to estimate the similarity between pairs
of scans. Different to OverlapNet, which exploits multiple cues such as
normal, intensity, and semantic information as the network input, in this work
we only use the depth information of the range image to achieve faster online
performance and make the approach easier to generalize. Due to the yaw-angle-
invariant design of the network, our approach can recognize places even when
the vehicle drives in different directions, such as the reverse loop in our
new Haomo dataset illustrated in Fig. 1.

Figure 1: Query scan (blue) and reference scan (orange) with adjacent
locations but opposite viewpoints in our novel Haomo dataset. Our
OverlapTransformer is able to generate yaw-angle-invariant global descriptors
with only range images, which is robust to viewpoint changing for place
recognition.

The main contribution of this paper is a lightweight transformer neural
network that exploits only depth information of range images to achieve place
recognition. Our approach is very fast to execute and at the same time yields
very good recognition results. Based on the attention mechanism of the
Transformer [17] and the NetVLAD head [1], our proposed OverlapTransformer
compresses LiDAR range images into global descriptors. We build the
architecture of our OverlapTransformer to ensure that each descriptor is yaw-
angle-invariant, which makes our method robust to viewpoint changes. We train
the proposed OverlapTransformer only on a part of the KITTI dataset and
evaluate it on both, KITTI and Ford Campus datasets with the loop closure
metric in line with OverlapNet [11]. Besides, we recorded and released a new
dataset, which contains three different challenges including place recognition
for long time spans, reverse driving, and different appearance scenes to
evaluate different methods.

In sum, we make the claims that our approach is able to (i) detect loop
closure candidates for SLAM using only LiDAR data without any other
information, and generalize well into the different environments without fine-
tuning, (ii) achieve long-term place recognition on our Haomo dataset in
outdoor large-scale environments with a different LiDAR sensor,  (iii)
recognize places with changing viewpoints exploiting the proposed yaw-angle-
invariant descriptors up to potentially discretization errors, (iv) run faster
than most state-of-the-art place recognition methods.

##  II Related Work

Place recognition is a common topic in computer vision and robotics with a
large number of scientific work proposed using RGB images [18, 1, 2, 4, 5]. We
refer more image-based methods to survey by Lowry [3] and focus our discussion
here more on 3D LiDAR-based approaches.

Due to the high accuracy of the range information and illumination invariance,
LiDAR-based place recognition has attracted attention in the field of
autonomous vehicles. For example, He et al. [14] propose M2DP, which projects
point cloud to multiple planes and combines descriptors from different planes
to generate the global signature. Röhling et al. [15] count the height of
point cloud to generate the histogram-based 1-D global descriptor for fast
retrieval. Scan Context (SC) proposed by Kim et al. [7] encodes the maximum
height of point cloud in different bins to generate the 2D global descriptor
for high discrimination, but leads to an increased computational matching
time. In contrast to SC using only geometric information, Cop et al. [19]
propose Delight to encode intensity-reading of LiDAR into a group of
histograms for comprehensive utilization of intensity and geometric
information. Inspired by Cop et al., Wang et al. [20] extend SC by also
exploiting both geometry and intensity, which outperforms geometric-only
descriptors with the same space division method as used in SC. Recently, Wang
et al. [9] propose LiDAR Iris exploiting the Fourier transform to generate a
binary signature image. It firstly generates LiDAR iris images by expanding
the bird-eye view of the LiDAR scan into an image strip. Then, it applies
Fourier transform on LiDAR iris images and solves the spatial place
recognition in the frequency domain.

With the development of deep learning, more learning-based approaches are
utilized for place recognition divided into two groups, local feature-based
methods, and global descriptor-based methods. Local feature-based approaches
often have two steps, first extracting local features from the LiDAR scans and
then recognizing places based on the extracted local features. For example,
Dube et al. [21] propose SegMatch to firstly segment the filtered point cloud
into sets of point clusters and then use features encoded by a CNN on such
clusters to find place matches. Based on SegMatch, Vidanapathirana et al. [22]
propose Locus, which uses higher-order pooling along with a non-linear
transformation to aggregate multi-level features and generate a fixed-length
global descriptor for place recognition. LPD-Net by Liu et al. [23] uses ten
types of local features from raw point cloud clusters as input of a graph
neural network to generate global descriptors. There are other learning-based
approaches directly generating global descriptors on LiDAR scans for place
recognition. For example, PointNetVLAD by Uy et al. [8] firstly uses PointNet
[24] to map laser points into a higher dimension and then uses the NetVLAD
architecture [1] previously used for visual place recognition to generate
global descriptors for LiDAR-based place recognition. Komorowski et al. [25]
utilize sparse 3D convolutions based on MinkowskiEngine with pooling layers to
generate descriptors. In contrast, LoGG3D-Net by Vidanapathirana et al. [26]
uses a local consistency loss to improve the performance of the global
descriptor. SOE-Net by Xia et al. [27] combines the orientation-encoding
module with PointNet to generate point-wise features, which are fed to a self-
attention network to generate discriminative and compact global descriptors.
Different to SOE-Net, Zhou et al. [28] propose NDT-Transformer, which
transforms raw point cloud into NDT cells and uses the attention mechanism
from Transformer [17] to improve the representation ability. PPT-Net by Hui et
al. [29] exploits the pyramid point transformer module to enhance the
discrimination of local features and generates a global descriptor.

Our method also directly generates global descriptors on LiDAR scans.
Different from methods that use local point cloud maps [23, 8, 27, 28, 29],
our method only uses range images generated from single 3D LiDAR scans. This
yields fast computations suitable for online operation and natural yaw-angle-
invariance for better place recognition performance. Our method also uses the
Transformer similar to the work by Zhou et al. [28] to boost place recognition
performance, but our method operates on range images instead of NDT cells.

Most recently, there are also works exploiting semantic information for place
recognition. For example, Chen et al. [6, 11] propose OverlapNet to exploit
multiple cues generated from LiDAR scan, including depth, normal, intensity,
and semantics for LiDAR-based loop closure detection and localization. SGPR by
Kong et al. [30] exploits the semantics and topological information of the raw
point cloud and extracts the semantic graph representation with graph neural
networks to find loop closures. Li et al. [31] use semantics to enhance SC and
propose semantic scan context. Similarly, Cramariuc et al. [32] utilize
semantic information to improve the performance of SegMatch. In contrast to
these semantic-enhanced methods, our approach only uses the raw depth
information to achieve online performance, which enables our method easier to
generalize to different environments and datasets collected by different LiDAR
sensors.

Figure 2: Pipeline overview of our proposed approach. We take a 64-beam LiDAR
as an example to introduce the dimensions of feature maps. The Range Image
Encoder (RIE) compresses range images from the Lidar sensor to yaw-angle-
equivariant feature maps. The Transformer Module (TM) concatenates feature
maps from Range Image Encoder and discriminativeness-enhanced feature maps
from Transformer Attention. Global Descriptor Generator (GDG) generates yaw-
angle-invariant descriptors exploiting the combination of MLP and NetVLAD. The
generated 1-D global descriptors are used for fast retrieval of place
recognition.

##  III Our Approach

The overview of our OverlapTransformer is depicted in Fig. 2. The range image
from raw point cloud is fed to an encoder, which is a modified OverlapNetLeg
to extract features from range images (see Sec. III-A). Then, the encoded
feature volume is fed to the transformer module, where we utilize a
transformer to embed the relative location of features and the global
information across the whole range image (see Sec. III-B). In the end, we use
a global descriptor generator with NetVLAD and multi-layer perceptrons (MLPs)
to compress the features and generate a yaw-angle-invariant 1-D global
descriptor (see Sec. III-C). During training, we use the triplet loss with
calculated overlap labels to better distinguish the positive and negative
training examples (see Sec. III-D).

###  III-A Range Image Encoder

We use range images from LiDAR scans as input data. The range image is an
intermediate representation of LiDAR data obtained from a typical spinning
mechanical LiDAR scanner with enough vertical resolution. Given the LiDAR
sensor parameters, there is a projection transformation between a point cloud
and a range image. A point cloud $\mathcal{P}$ can be projected to a range
image $\mathcal{R},\Pi:~{}\mathbb{R}^{3}\mapsto\mathbb{R}^{2}$, where each
pixel contains one 3D point. Each point ${\mbox{\boldmath$p$}}_{i}=(x,y,z)$ is
converted to image coordinates $(u,v)$ by

$\displaystyle\left(\begin{array}[]{c}u\vspace{0.0em}\\\ v\end{array}\right)$
$\displaystyle=\left(\begin{array}[]{cc}\frac{1}{2}\left[1-\arctan(y,x)\pi^{-1}\right]w\vspace{0.5em}\\\
\left[1-\left(\arcsin(zr^{-1})+\mathrm{f}_{\mathrm{up}}\right)\mathrm{f}^{-1}\right]h\end{array}\right),$
(5)

where $r=||{\mbox{\boldmath$p$}}||_{2}$ is the range,
$\mathrm{f}=\mathrm{f}_{\mathrm{up}}+\mathrm{f}_{\mathrm{down}}$ is the
vertical field-of-view of the sensor, and $w,h$ are the width and height of
the resulting range image $\mathcal{R}$.

The yaw rotation $\theta$ of a point cloud is yaw-angle-equivariant to the
horizontal shift $s$ of the corresponding range image, and the following
equations hold:

$\displaystyle\left(\begin{array}[]{c}u^{\prime}\vspace{0.0em}\\\
v^{\prime}\end{array}\right)$
$\displaystyle=\left(\begin{array}[]{cc}u+s\vspace{0.5em}\\\
v\end{array}\right),\hskip 5.0pts=\frac{1}{2}\left[1-\theta\pi^{-1}\right]w,$
(10) $\displaystyle\mathcal{R}C_{s}=\Pi(R_{\theta}\mathcal{P}).$ (11)

The term $C_{s}$ represents the column shift of the range image $\mathcal{R}$
by matrix right multiplication, and $R_{\theta}$ represents the yaw rotation
matrix of the point cloud $\mathcal{P}$. Note that, there might be
discretization errors in pixel coordinates when generating the range images
from point clouds.

The range image with size of $h\times w\times 1$ is fed to the modified
OverlapNetLeg, where $1$ refers to the one range channel, while $h,w$ are the
height and width of the range image. Compared to OverlapNet [11], the
convolution filters in our encoder only compress the range image in the
vertical dimension but not the width dimension to avoid the discretization
error to the yaw equivariance. Besides, there is no padding and dropout in our
proposed architecture to keep the yaw equivariance for every intermediate
feature generated by each network layer. We denote the output feature volume
of the range image encoder as
$\mathcal{F}={\color[rgb]{0,0,0}\text{RIE}}(\mathcal{R})$, with the size of
$1\times w\times c$, and
$C_{s}\mathcal{F}={\color[rgb]{0,0,0}\text{RIE}}(\Pi(R_{\theta}\mathcal{P}))$
holds. The variable $c$ refers to the channel number of the encoded feature.

###  III-B Transformer Module

Inspired by NDT-transformer [28], we also exploit a transformer to extract
more distinctive features for LiDAR place recognition. As shown in the middle
of Fig. 2, our attentional feature transformer is composed of three modules,
multi-head self-attention (MHSA), feed-forward network (FFN), and layer
normalization (LN). Different from NDT-transformer, we use one transformer
block in our transformer module to achieve high accuracy as well as
efficiency.

The MHSA can learn the relationship between features captured by a self-
attention mechanism. We denote the feature volume extracted by the MHSA as
$\mathcal{A}$, and the self-attention mechanism of the transformer can then be
fomulated as:

$\displaystyle\mathcal{A}$
$\displaystyle=\text{Attention}(Q,K,V)=\text{softmax}\Big{(}\frac{Q{K}^{T}}{\sqrt{d_{k}}}\Big{)}V\vspace{0.5em},$
(12)

where $\\{Q,K,V\\}$ are the query, key and value splits from the feature
volumes generated by our range image encoder, and $d_{k}$ represents the
dimension of splits.

$\mathcal{A}$ is then fed into the FFN and LN to generate the final
attentional feature volume $\mathcal{S}$, which can be calculated as:

$\displaystyle\mathcal{S}$
$\displaystyle=\text{LN}(\text{FFN}(\text{LN}(\text{Conc}(\mathcal{F},\mathcal{A})))+\text{LN}(\text{Conc}(\mathcal{F},\mathcal{A}))),$
(13)

where $\text{Conc}(\cdot)$ represents the concatenation operation across
channels. In this way, a coarse feature $\mathcal{F}$ extracted by our range
image encoder is upgraded to an attentional feature $\mathcal{S}$.

Here we further show that our transformer module is also yaw-angle-
equivariant. It is obvious that concatenation across channels, linear
transformations, ReLU function, and LN are yaw-angle-equivariant. Since the
query, key, and value feature volumes in attention mechanism are split along
the channel dimension, given a shifted feature generated by our range image
encorder $C_{s}\mathcal{F}$, we have:

Attention $\displaystyle(C_{s}Q,C_{s}K,C_{s}V)\vspace{0.5em}$ (14)
$\displaystyle=\text{softmax}\Big{(}\frac{C_{s}Q(C_{s}K)^{T}}{\sqrt{d_{k}}}\Big{)}C_{s}V\vspace{0.5em}$
(15)
$\displaystyle=\text{softmax}\Big{(}\frac{C_{s}Q{K}^{T}{C_{s}}^{T}}{\sqrt{d_{k}}}\Big{)}C_{s}V\vspace{0.5em}$
(16)
$\displaystyle=C_{s}\text{softmax}\Big{(}\frac{Q{K}^{T}}{\sqrt{d_{k}}}\Big{)}{C_{s}}^{T}C_{s}V\vspace{0.5em}$
(17) $\displaystyle=C_{s}\mathcal{A}.$ (18)

Since FFN is applied to each element in the feature volume separately and
identically [17], it is therefore also yaw-angle-equivariant towards features.
Then, for the rest part of our attentional feature transformer, we also have:

$\displaystyle\text{LN}(\text{FFN}(\text{LN}(\text{Conc}(C_{s}\mathcal{F},C_{s}\mathcal{A})))+\text{LN}(\text{Conc}(C_{s}\mathcal{F},C_{s}\mathcal{A})))$
$\displaystyle=C_{s}\text{LN}(\text{FFN}(\text{LN}(\text{Conc}(\mathcal{F},\mathcal{A})))+\text{LN}(\text{Conc}(\mathcal{F},\mathcal{A})))$
$\displaystyle=C_{s}\mathcal{S}.$ (19)

The transformer module finds the inner connections between different parts of
the input feature volume, which exploits the spatial relations of different
features in the scene. Similar to humans using distinctive landmarks and their
relationship to determine the places, our transformer module also focuses on
spatial features and their relationship using the attention mechanism, which
boosts the place recognition performance of our method.

###  III-C Global Descriptor Generator

NetVLAD was first proposed by Arandjelovic et al. [1] to tackle image-based
place recognition in an end-to-end manner and outperform its non-learning-
based counterparts. Uy et al. [8] transfer it for LiDAR-based place
recognition and proof that NetVLAD is permutation invariant, thus suitable for
unordered point clouds. However, the original PointNetVLAD is not able to
directly generate yaw-angle-invariant global descriptors for the point cloud
because the yaw rotation of the point cloud leads to a different input of
point coordinates and features. This makes the PointNetVLAD less robust to the
rotation or noise of the sensor pose.

In this work, we leverage the permutation invariance of NetVLAD together with
our yaw-angle-equivariant features to achieve yaw-rotation invariance and
generate yaw-angle-invariant descriptors. We represent the input feature
volume with size of $1\times w\times c$ as a set of 1-D vectors over channel
dimension
$\mathcal{S}=\mathcal{Z}=\\{{\mbox{\boldmath$z$}}_{1},{\mbox{\boldmath$z$}}_{2},...,{\mbox{\boldmath$z$}}_{i},...,{\mbox{\boldmath$z$}}_{w}\\}$,
where ${\mbox{\boldmath$z$}}_{i}$ is the $i$-th vector with size of $c$
channels. Given that NetVLAD is permutation invariant [8], we have:

$\displaystyle\text{GDG}(\mathcal{Z}C_{s})$
$\displaystyle=\text{GDG}(\\{{\mbox{\boldmath$z$}}_{w-s},...,{\mbox{\boldmath$z$}}_{w},{\mbox{\boldmath$z$}}_{1},...,{\mbox{\boldmath$z$}}_{w-s-1}\\})$
$\displaystyle=\text{GDG}(\\{{\mbox{\boldmath$z$}}_{1},{\mbox{\boldmath$z$}}_{2},...,{\mbox{\boldmath$z$}}_{i},...,{\mbox{\boldmath$z$}}_{w}\\})$
(20) $\displaystyle=\text{GDG}(\mathcal{Z}).$ (21)

We denote the output descriptor of our global descriptor generator as
$\mathcal{V}$ and we have:

$\displaystyle\mathcal{V}=\text{GDG}(\mathcal{S})=\text{GDG}(C_{s}\mathcal{S}),$
(22)

which means our proposed global descriptor generator is yaw-angle-invariant.

Figure 3: An illustration of our yaw-angle-invariant descriptors exploiting a
toy example. As can be seen that the range image presentation, our range image
encoder, and attentional feature transformer are yaw-angle-equivariant, and
our global descriptor generator generates yaw-angle-invariant global features.

In Fig. 3, we depict a toy example to illustrate that our model can generate
yaw-angle-invariant descriptors with yaw-angle-equivariant range image
representations. We show at each row a different yaw rotation and the
visualizations of the intermediate results of each module of our method. The
yaw rotation of point clouds corresponds to the horizontal shift of range
image pixels along the width dimension. Supposing the first row is the results
of the raw LiDAR data, rotated 0 degree, the second and the third are rotated
by 90 and 180 degrees along the yaw angle, which corresponds to the range
images shifted by $\frac{1}{4}w$ and $\frac{1}{2}w$ respectively. As can be
seen, the outputs of the yaw-angle-equivariant OverlapNetLeg and transformer
module are also shifted accordingly (shown by the second and the third
column). Since the global descriptor generator is yaw-angle-invariant, it thus
generates the same global descriptor in all three cases as shown in the fourth
column.

During online operation, we use the yaw-angle-invariant global descriptors to
represent LiDAR scans and use the Euclidean distance between pairs of
descriptors to find the nearest reference places.

###  III-D Network Training

We follow OverlapNet [11] using overlap to supervise the network rather than
directly using ground truth distances. Overlap is a more natural way to
describe the similarity between two LiDAR scans than distance. Moreover, the
overlap between two LiDAR scans also corresponds to the quality of the
following registration, which can be a good criterion for the final metric
localization [11]. For a query scan $\mathcal{R}_{q}$ and a reference scan
$\mathcal{R}_{r}$, we use the ground truth poses to reproject
$\mathcal{R}_{r}$ into the coordinate frame of $\mathcal{R}_{q}$ and get
$\mathcal{R}^{\prime}_{r}$. The overlap between them is calculated as:

$\displaystyle O_{\mathcal{R}_{q}\mathcal{R}_{r}}$
$\displaystyle=\frac{\sum_{(u,v)}\mathbb{I}\Big{\\{}\left|\left|\mathcal{R}_{q}(u,v)-\mathcal{R}^{\prime}_{r}(u,v)\right|\right|\leq\delta\Big{\\}}}{\min\left(\textrm{valid}(\mathcal{R}_{q}),\textrm{valid}(\mathcal{R}^{\prime}_{r})\right)},$
(23)

where $\mathbb{I}(a)=1$ if $a$ is true and $\mathbb{I}(a)=0$ otherwise,
$\text{valid}(\mathcal{R})$ refers to the counts of valid pixels of range
image $\mathcal{R}$, and $\delta$ is the threshold to decide the overlapped
pixel.

For each training tuple, we utilize one query descriptor $\mathcal{V}_{q}$,
$k_{p}$ positive descriptors $\\{\mathcal{V}_{p}\\}$, and $k_{n}$ negative
descriptors $\\{\mathcal{V}_{n}\\}$ to compute lazy triplet loss:

$\displaystyle\mathcal{L}_{T}(\mathcal{V}_{q},\\{\mathcal{V}_{p}\\},\\{\mathcal{V}_{n}\\})=$
$\displaystyle
k_{p}(\alpha+\max_{p}(d(\mathcal{V}_{q},\mathcal{V}_{p})))-\sum_{k_{n}}(d(\mathcal{V}_{q},\mathcal{V}_{n})),$
(24)

where $\alpha$ is the margin to avoid negative loss and $d(\cdot)$ is the
squared Euclidean distance. We take a pair of scans whose overlap is larger
than 0.3 as a positive sample, otherwise a negative sample. We use the triplet
loss to minimize the distance between the query and the hardest positive
global descriptors, and maximize the distance between the query and all
sampled negative global descriptors.

TABLE I: Statistics of Haomo dataset Sequence | 1-1 | 1-2 | 1-3 | 2-1 | 2-2
---|---|---|---|---|---
Date | 2021.12.8 | 2021.12.8 | 2021.12.8 | 2021.12.28 | 2022.1.13
N${}_{\text{scans}}$ | 12500 | 22345 | 13500 | 100887 | 88154
N${}_{\text{pos.}}$ | 40000 | 40000 | – | 48000 | –
N${}_{\text{neg.}}$ | 60000 | 60000 | – | 72000 | –
N${}_{\text{queries}}$ | – | – | 1350 | – | 8815
Length | 2.3 km | 2.3 km | 2.3 km | 11.5 km | 11.1 km
Direction | Same | Reverse | – | Same | –
Role | Database | Database | Query | Database | Query
N${}_{\text{scans}}$, N${}_{\text{pos.}}$, N${}_{\text{neg.}}$ and
N${}_{\text{queries}}$ are the numbers of scans, positive samples, negative
samples and query scans respectively.

##  IV Haomo Dataset

There are several LiDAR-based datasets for autonomous driving [33, 34, 35],
which can be used for evaluating loop closure detection methods. However, not
many of the publicly available datasets show significant repetitive reverse
routes for long-term large-scale place recognition of autonomous driving. In
this work, we therefore provide such a new challenging dataset called _Haomo_
dataset and released it together with our code to support future research. The
dataset was collected in urban environments of Beijing by a mobile robot built
by HAOMO.AI Technology company equipped with a HESAI PandarXT 32-beam LiDAR
sensor, a SENSING-SG2 wide-angle camera, and an ASENSING-INS570D RTK GNSS.
There are currently five sequences in Haomo dataset as listed in Tab. I.
Sequences 1-1 and 1-2 are collected from the same route in 8th December 2021
with opposite driving direction. An additional sequence 1-3 from the same
route is utilized as the online query with respect to both 1-1 and 1-2
respectively to evaluate place recognition performance of forward and reverse
driving. Sequences 2-1 and 2-2 are collected along a much longer route from
the same direction, but on different dates, 2-1 on 28th December 2021 and 2-2
on 13th January 2022, where the old one is used as a database while the newer
one is used as query. The two sequences are for evaluating the performance for
large-scale long-term place recognition.

##  V Experimental Evaluation

The experimental evaluation is designed to showcase the performance of our
approach and to evaluate the claims that our approach is able to: (i) detect
loop closure candidates for SLAM using only LiDAR data without any other
information, and generalize well into the different environments without fine-
tuning, (ii) achieve long-term place recognition on our Haomo dataset in
outdoor large-scale environments, (iii) recognize places with changing
viewpoints exploiting the proposed yaw-angle-invariant descriptors, (iv) run
faster than most state-of-the-art place recognition methods.

###  V-A Implementation and Experimental Setup

We use three different datasets to evaluate our method, including KITTI
dataset [33] collected in Germany with a 64-beam LiDAR sensor, Ford Campus
dataset [34] collected in U.S. with a 64-beam LiDAR sensor, and our Haomo
dataset collected in China with a 32-beam LiDAR sensor. Following OverlapNet
[11], we use range images of size $1\times 64\times 900$ for 64-beam LiDAR
data of KITTI and Ford Campus datasets, and range images with size of $1\times
32\times 900$ for our Haomo dataset with 32-beam LiDAR data. For our
transformer module, we set the embedding dimension $d_{model}=256$, the number
of heads $n_{head}=4$, and the intermediate dimension of the feed-forward
layer $d_{\mathit{ffn}}=1024$ . We do not utilize dropout to achieve yaw-
angle-equivariant. For NetVLAD, we set the intermediate feature dimension
$d_{inter}=1024$, the output feature dimension $d_{output}=256$, and the
number of clusters $d_{K}=64$ . The output of our global descriptor generator
is a vector of size $256$. For calculating overlap ground truth by Eq. (23),
we set $\delta=1$ for KITTI odometry benchmark with provided 64-beam LiDAR
scans, $\delta=1.2$ for our Haomo dataset depending on the density of the
points. We set $k_{p}=6$, $k_{n}=6$, and $\alpha=0.5$ for the triplet loss.

###  V-B Evaluation for Loop Closure Detection

TABLE II: Comparison of loop closure detection performance Dataset | Approach |  AUC |  F1max |  Recall @1 |  Recall @1%
---|---|---|---|---|---
KITTI |  Histogram [15] |  0.826 |  0.825 |  0.738 |  0.871
Scan Context [7] |  0.836 |  0.835 |  0.820 |  0.869
LiDAR Iris [9] |  0.843 |  0.848 |  0.835 |  0.877
PointNetVLAD [8] |  0.856 |  0.846 |  0.776 |  0.845
OverlapNet [11] |  0.867 |  0.865 |  0.816 |  0.908
| NDT-Transformer-P [28] |  0.855 |  0.853 |  0.802 |  0.869
| MinkLoc3D [25] |  0.894 |  0.869 |  0.876 |  0.920
| Ours |  0.907 |  0.877 |  0.906 |  0.964
Ford Campus |  Histogram [15] |  0.841 |  0.800 |  0.812 |  0.897
Scan Context [7] |  0.903 |  0.842 |  0.878 |  0.958
LiDAR Iris [9] |  0.907 |  0.842 |  0.849 |  0.937
PointNetVLAD [8] |  0.872 |  0.830 |  0.862 |  0.938
OverlapNet [11] |  0.854 |  0.843 |  0.857 |  0.932
| NDT-Transformer-P [28] |  0.835 |  0.850 |  0.900 |  0.927
| MinkLoc3D [25] |  0.871 |  0.851 |  0.878 |  0.942
| Ours |  0.923 |  0.856 |  0.914 |  0.954

The first experiment supports our claim that our approach detects loop closure
candidates for SLAM using only LiDAR data without any other information, and
generalizes well into the different environments without fine-tuning.
Following the experimental setup of OverlapNet [11], we train and evaluate our
approach and other learning-based baseline methods on the KITTI Odometry
Benchmark [33]. We use sequences $03$–$10$ for training, sequence $02$ for
validation, and sequence $00$ for evaluation. We regard two scans as a loop
closure if their overlap value is larger than $0.3$. To evaluate the
generalization ability of our method, we also test it on sequence $00$ of the
Ford Campus dataset [34]. Note that we did not train our approach on the Ford
Campus dataset, which shows the generalization ability of our method.

Since the semantics are not always available for the different datasets, we
compare our method with existing methods which do not utilize semantic
information, including Histogram [15], Scan Context [7] with augmentation,
LiDAR Iris [9], PointNetVLAD [8], OverlapNet (Delta-Geo-Only) [11], NDT-
Transformer-P (4096 cells) [28], and MinkLoc3D [25]. Loop closure detection
during SLAM takes previous scans as the database, excluding the nearby 100
scans to avoid detecting the most recent scans. We utilize AUC, F1 max scores,
recall@1 and recall@1% for evaluation and the results are shown in Tab. II. As
can be seen, our method outperforms all the baseline methods on the KITTI
dataset. Using our network pre-trained on KITTI directly on an unseen
environment, Ford Campus dataset, without any fine-tuning, our proposed method
still outperforms other methods, and only for recall@1%, our method is on par
with the state-of-the-art non-learning-based method Scan Context [7], which
shows the good generalization ability of our method.

###  V-C Evaluation for Place Recognition

In the second experiment, we investigate the long-term place recognition of
our method on our Haomo dataset in outdoor large-scale environments with a
different type of LiDAR sensor. The difference between loop closure detection
and place recognition is that for place recognition the database/map is
usually given and we need to find the location of the vehicle within the
database/map. For loop closing in contrast, the database grows during
operation. Therefore, we train our approach and other learning-based baseline
methods on the database sequences and evaluate all the methods using the same
new query sequences that were not parts of the database. We have three
challenges in Haomo dataset. From easy to difficult, the first is a short-term
same-direction challenge, sequence 1-1 as database and 1-3 for query. The
second is the short-term inverse-direction challenge, sequence 1-2 as database
and 1-3 for query. The last is a long-term large-scale challenge, sequence 2-1
as database and 2-2 for query. We use database sequences for training too. As
shown in Tab. I, we firstly downsample the database and for each sampled laser
scan, we calculate the overlaps between this scan to all other scans. For
testing, we take one scan sampled from every ten scans in query sequences, and
acquire its ground truth reference scans with overlap values larger than
$0.3$.

We utilize recall@N to evaluate the performance of algorithms on Haomo dataset
as the most large-scale place recognition approaches do. We consider one query
as a successful loop closure once we find one of the true references. The
results of the first two challenges are shown in Fig. 4 and Fig. 5. As can be
seen, our method outperforms other methods on these two challenges, especially
for the reverse driving one due to the yaw-angle-invariant architecture design
of our method. Although Scan Context and Histogram are also robust to pure
rotation, our method outperforms them, since in real application there is not
only pure yaw rotation but also changes in locations and environment
appearance due to the dynamic objects. Our method is more robust to all
challenging conditions compared to baseline methods. The evaluation on long-
term large-scale sequences 2-2 and 2-1 are illustrated in Fig. 6, and our
method also keeps its superiority on large-scale long-term datasets, which
shows the ability of our method to be used for real autonomous driving
applications.

Figure 4: Place recognition results using Haomo dataset sequence 1-3 as query
and 1-1 as database (driving forward). Figure 5: Place recognition results
using Haomo dataset sequence 1-3 as query and 1-2 as database (driving in
reverse).

###  V-D Study on Yaw-Angle-Invariance

The third experiment investigates the yaw-angle-invariance of our approach.
The results support the claim that our method generates yaw-angle-invariant
descriptors and can recognize places with changing viewpoints. For that, we
rotate each query scan of the KITTI dataset along the yaw-axis in steps of 30
degrees and search the places with respect to the same original database. In
this experiment, we test all baselines and use Recall@1 as the evaluation
metric. As illustrated in Fig. 7, our proposed OverlapTransformer is not
influenced by pure rotation along the yaw-axis, and maintains the best
performance compared to the baseline methods. On the contrary, PointNetVLAD,
Scan Context, OverlapNet, NDT-Transformer-P and MinkLoc3D are all affected by
rotation. PointNetVLAD, OverlapNet, and NDT-Transformer-P lose efficacy
quickly for increasing yaw angle discrepancies. Here, we only use OverlapNet
with the head of the similarity estimation for a fair comparison. Scan Context
loses efficacy to some extent but is better than PointNetVLAD since the ring
key representing occupancy ratio is yaw-angle-invariant. MinkLoc3D has the
similar performance as Scan Context. The LiDAR Iris and histogram-based method
are also yaw-angle-invariant as our method but have lower recall@1 than ours.
LiDAR Iris achieves yaw-angle-invariant by rotating its features multiple
times and choosing the best scored one, which makes it very slow as shown in
Sec. V-F. Our OverlapTransformer outperforms other baseline methods
significantly due to our devised yaw-angle-invariant model exploiting yaw-
angle-equivariant range images.

Figure 6: Place recognition results using Haomo dataset sequence 2-2 as query
and 2-1 as database (driving in large-scale environments). Figure 7: yaw-
angle-invariant test.

###  V-E Ablation Study on Transformer Module

This ablation study validates the effectiveness of the transformer module (TM)
we used between our range image encoder (RIE) and global descriptor generator
(GDG). We use Haomo dataset sequence 1-2 as database and 1-3 for query to
compare 5 different setups, including RIE+GDG, RIE+Conv+GDG, RIE+1TM+GDG,
RIE+3TM+GDG, and RIE+6TM+GDG. RIE+GDG uses no transformer module. RIE+Conv+GDG
replaces the transformer module with two 1$\times$1 convolution layers without
changing the channel numbers. RIE+1/3/6TM+GDG uses 1, 3, 6 TMs respectively.
As shown in Tab. III, RIE+3TM+GDG and RIE+1TM+GDG outperform the other setups
for place recognition. The results show that one transformer block already
increases the performance significantly, while more transformer blocks lead to
lower efficiency. When using more than 3 transformer blocks, the performance
even decreases slightly. More transformer blocks might need more training data
and time to obtain good performance. Thus, we only use one transformer block.

TABLE III: Ablation study on the transformer module Network |  Runtime [ms] |  Recall@1 |  Recall@5 |  Recall@20
---|---|---|---|---
RIE+GDG |  0.86 |  0.642 |  0.750 |  0.841
RIE+Conv+GDG |  0.92 |  0.698 |  0.841 |  0.923
RIE+1TM+GDG |  1.37 |  0.776 |  0.878 |  0.931
RIE+3TM+GDG |  2.39 |  0.788 |  0.875 |  0.940
RIE+6TM+GDG |  3.95 |  0.741 |  0.852 |  0.913

###  V-F Runtime

TABLE IV: Comparison of runtime with state-of-the-art methods Approach |  Descriptor Extraction [ms] |  Searching [ms]
---|---|---
Hand crafted |  Histogram [15] |  1.07 |  0.46
Scan Context [7] |  57.95 |  492.63
LiDAR Iris [9] |  7.13 |  9315.16
Learning based |  PointNetVLAD [8] |  13.87 |  1.43
OverlapNet [11] |  4.85 |  3233.30
NDT-Transformer-P [28] |  15.73 |  0.49
MinkLoc3D [25] |  15.94 |  8.10
Ours |  1.37 |  0.44

The experiment evaluates the runtime requirements of our method. It supports
our last claim that our method runs faster than most of the state-of-the-art
place recognition methods and that it can run at 730 Hz, i.e., much faster
than the scanning rate. We compare the runtime of our OverlapTransformer with
all baseline methods. We conduct all experiments on a system with an Intel
i7-11700K CPU and an Nvidia RTX 3070 GPU. We run all the methods to find top-1
candidates for one query scan with respect to a database consisting of 2000
reference scans. We report the averaged results over ten experiments. As shown
in Tab. IV, we count the runtime separately for descriptor generation and
searching. For the time cost by generating the descriptor, we take also the
LiDAR data preprocessing into consideration. As can be seen, for descriptor
generation, our method is the fastest method among the state-of-the-art
learning-based methods with 1.37 ms for each scan, and slightly slower than
the pure geometric histogram-based method. PointNetVLAD, NDT-Transformer-P,
and MinkLoc3D need an extra downsampling process for raw point cloud, which is
accelerated by GPU. OverlapNet needs to estimate the normals. For the time
cost by searching, our method outperforms all baselines including the non-
learning-based methods. Since Histogram, PointNetVLAD, NDT-Transformer-P,
MinkLoc3D and our method do not need an ad-hoc function to compute the
similarity of descriptors, FAISS library [36] is used to accelerate the
searching. Note that, PointNetVLAD, NDT-Transformer-P and MinkLoc3D generate
descriptors with the same size ($1\times 256$) as ours, but consume more time.
The reason could be that the descriptors generated by our method are more
descriptive than those by PointNetVLAD, NDT-Transformer-P, and MinkLoc3D, and
thus need less time for searching.

##  VI Conclusion

In this paper, we presented a novel approach for LiDAR-based place recognition
with a runtime less than 2 ms and outperforms the state-of-the-art methods in
terms of place recognition performance. Our approach utilizes a lightweight
network with a Transformer attention mechanism to generate yaw-angle-invariant
descriptors, which allows us to handle challenging place recognition
effectively and efficiently. We trained our method on the KITTI dataset and
evaluated it on both, the KITTI and the Ford Campus dataset for loop closure
detection, which shows a solid generalization capability. For further
evaluating the place recognition ability on long time spans, reverses driving,
and large-scale scenes, we developed a new challenging Haomo dataset and
conducted extensive evaluations with multiple baseline approaches on it. The
experimental results suggest that our method outperforms the other state-of-
the-art methods in different challenging environments in terms of recognition
performance and speed.

## References

  * [1] R. Arandjelovic, P. Gronat, A. Torii, T. Pajdla, and J. Sivic, “Netvlad: Cnn architecture for weakly supervised place recognition,” in _Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [2] S. Hausler, S. Garg, M. Xu, M. Milford, and T. Fischer, “Patch-netvlad: Multi-scale fusion of locally-global descriptors for place recognition,” in _Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * [3] S. Lowry, N. Sunderhauf, P. Newman, J. Leonard, D. Cox, P. Corke, and M. Milford, “Visual place recognition: A survey,” _IEEE Trans. on Robotics (TRO)_ , vol. 32, no. 1, pp. 1–19, 2016.
  * [4] O. Vysotska and C. Stachniss, “Lazy Data Association For Image Sequences Matching Under Substantial Appearance Changes,” _IEEE Robotics and Automation Letters (RA-L)_ , vol. 1, no. 1, pp. 213–220, 2016.
  * [5] ——, “Effective Visual Place Recognition Using Multi-Sequence Maps,” _IEEE Robotics and Automation Letters (RA-L)_ , vol. 4, pp. 1730–1736, 2019\.
  * [6] X. Chen, T. Läbe, A. Milioto, T. Röhling, O. Vysotska, A. Haag, J. Behley, and C. Stachniss, “OverlapNet: Loop Closing for LiDAR-based SLAM,” in _Proc. of Robotics: Science and Systems (RSS)_ , 2020.
  * [7] G. Kim and A. Kim, “Scan context: Egocentric spatial descriptor for place recognition within 3d point cloud map,” in _Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_ , 2018.
  * [8] M. A. Uy and G. H. Lee, “Pointnetvlad: Deep point cloud based retrieval for large-scale place recognition,” in _Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * [9] Y. Wang, Z. Sun, C.-Z. Xu, S. E. Sarma, J. Yang, and H. Kong, “Lidar iris for loop-closure detection,” in _Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_ , 2020.
  * [10] X. Chen, A. Milioto, E. Palazzolo, P. Giguère, J. Behley, and C. Stachniss, “SuMa++: Efficient LiDAR-based Semantic SLAM,” in _Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_ , 2019.
  * [11] X. Chen, T. Läbe, A. Milioto, T. Röhling, J. Behley, and C. Stachniss, “OverlapNet: A Siamese Network for Computing LiDAR Scan Similarity with Applications to Loop Closing and Localization,” _Autonomous Robots_ , vol. 46, pp. 61–81, 2021.
  * [12] X. Chen, T. Läbe, L. Nardi, J. Behley, and C. Stachniss, “Learning an Overlap-based Observation Model for 3D LiDAR Localization,” in _Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_ , 2020.
  * [13] X. Chen, I. Vizzo, T. Läbe, J. Behley, and C. Stachniss, “Range Image-based LiDAR Localization for Autonomous Vehicles,” in _Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA)_, 2021.
  * [14] L. He, X. Wang, and H. Zhang, “M2DP: A Novel 3D Point Cloud Descriptor and Its Application in Loop Closure Detection,” in _Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_ , 2016.
  * [15] T. Röhling, J. Mack, and D. Schulz, “A Fast Histogram-Based Similarity Measure for Detecting Loop Closures in 3-D LIDAR Data,” in _Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_ , 2015, pp. 736–741.
  * [16] B. Steder, R. Rusu, K. Konolige, and W. Burgard, “NARF: 3D range image features for object recognition,” in _Proc. of the IROS Workshop on Defining and Solving Realistic Perception Problems in Personal Robotics_ , 2010\.
  * [17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Proc. of the Advances in Neural Information Processing Systems (NIPS)_ , 2017\.
  * [18] H. Jégou, M. Douze, C. Schmid, and P. Pérez, “Aggregating local descriptors into a compact image representation,” in _Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2010.
  * [19] K. P. Cop, P. V. Borges, and R. Dubé, “Delight: An efficient descriptor for global localisation using lidar intensities,” in _Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA)_, 2018.
  * [20] H. Wang, C. Wang, and L. Xie, “Intensity scan context: Coding intensity and geometry relations for loop closure detection,” in _Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA)_, 2020.
  * [21] R. Dubé, D. Dugas, E. Stumm, J. Nieto, R. Siegwart, and C. Cadena, “Segmatch: Segment based place recognition in 3d point clouds,” in _Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA)_, 2017.
  * [22] K. Vidanapathirana, P. Moghadam, B. Harwood, M. Zhao, S. Sridharan, and C. Fookes, “Locus: Lidar-based place recognition using spatiotemporal higher-order pooling,” in _Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA)_, 2021.
  * [23] Z. Liu, S. Zhou, C. Suo, P. Yin, W. Chen, H. Wang, H. Li, and Y.-H. Liu, “Lpd-net: 3d point cloud learning for large-scale place recognition and environment analysis,” in _Proc. of the IEEE/CVF Intl. Conf. on Computer Vision (ICCV)_ , 2019.
  * [24] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in _Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * [25] J. Komorowski, “MinkLoc3D: Point Cloud Based Large-Scale Place Recognition,” in _Proc. of the IEEE Winter Conf. on Applications of Computer Vision (WACV)_ , 2021.
  * [26] K. Vidanapathirana, M. Ramezani, P. Moghadam, S. Sridharan, and C. Fookes, “LoGG3D-Net: Locally Guided Global Descriptor Learning for 3D Place Recognition,” in _Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA)_, 2022.
  * [27] Y. Xia, Y. Xu, S. Li, R. Wang, J. Du, D. Cremers, and U. Stilla, “Soe-net: A self-attention and orientation encoding network for point cloud based place recognition,” in _Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * [28] Z. Zhou, C. Zhao, D. Adolfsson, S. Su, Y. Gao, T. Duckett, and L. Sun, “Ndt-transformer: Large-scale 3d point cloud localisation using the normal distribution transform representation,” in _Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA)_, 2021.
  * [29] L. Hui, H. Yang, M. Cheng, J. Xie, and J. Yang, “Pyramid Point Cloud Transformer for Large-Scale Place Recognition,” in _Proc. of the IEEE/CVF Intl. Conf. on Computer Vision (ICCV)_ , 2021.
  * [30] X. Kong, X. Yang, G. Zhai, X. Zhao, X. Zeng, M. Wang, Y. Liu, W. Li, and F. Wen, “Semantic graph based place recognition for 3d point clouds,” in _Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_ , 2020.
  * [31] L. Li, X. Kong, X. Zhao, T. Huang, W. Li, F. Wen, H. Zhang, and Y. Liu, “Ssc: Semantic scan context for large-scale place recognition,” in _Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_ , 2021.
  * [32] A. Cramariuc, F. Tschopp, N. Alatur, S. Benz, T. Falck, M. Brühlmeier, B. Hahn, J. Nieto, and R. Siegwart, “Semsegmap–3d segment-based semantic localization,” in _Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_ , 2021.
  * [33] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite,” in _Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2012.
  * [34] G. Pandey, J. McBride, and R. Eustice, “Ford campus vision and lidar data set,” _Intl. Journal of Robotics Research (IJRR)_ , vol. 30, no. 13, pp. 1543–1552, 2011.
  * [35] P. Wang, X. Huang, X. Cheng, D. Zhou, Q. Geng, and R. Yang, “The apolloscape open dataset for autonomous driving and its application,” _IEEE Trans. on Pattern Analalysis and Machine Intelligence (TPAMI)_ , 2019.
  * [36] J. Johnson, M. Douze, and H. Jégou, “Billion-scale similarity search with gpus,” _IEEE Trans. on Big Data_ , vol. 7, no. 3, pp. 535–547, 2019.