# OverlapNet: Loop Closing for LiDAR-based SLAM

Xieyuanli Chen∗ Thomas Läbe∗ Andres Milioto∗ Timo Röhling∗,‡
Olga Vysotska†,∗ Alexandre Haag† Jens Behley∗ Cyrill Stachniss∗
∗Photogrammetry & Robotics Lab, University of Bonn, Germany
‡Fraunhofer FKIE, Wachtberg, Germany
†Autonomous Intelligent Driving GmbH, Munich, Germany

###### Abstract

Simultaneous localization and mapping (SLAM) is a fundamental capability
required by most autonomous systems. In this paper, we address the problem of
loop closing for SLAM based on 3D laser scans recorded by autonomous cars. Our
approach utilizes a deep neural network exploiting different cues generated
from LiDAR data for finding loop closures. It estimates an image overlap
generalized to range images and provides a relative yaw angle estimate between
pairs of scans. Based on such predictions, we tackle loop closure detection
and integrate our approach into an existing SLAM system to improve its mapping
results. We evaluate our approach on sequences of the KITTI odometry benchmark
and the Ford campus dataset. We show that our method can effectively detect
loop closures surpassing the detection performance of state-of-the-art
methods. To highlight the generalization capabilities of our approach, we
evaluate our model on the Ford campus dataset while using only KITTI for
training. The experiments show that the learned representation is able to
provide reliable loop closure candidates, also in unseen environments.

##  I Introduction

Simultaneous localization and mapping or SLAM [1, 29] is an integral part of
most robots and autonomous cars. Graph-based SLAM often relies on (i) pose
estimation relative to a recent history, which is called odometry or
incremental scan matching, and (ii) loop closure detection, which is needed
for data association on a global scale. Loop closures enable SLAM approaches
to correct accumulated drift resulting in a globally consistent map.

In this paper, we propose a new method to loop closing for laser range scans
produced by a rotating 3D LiDAR sensor installed on a wheeled robot or similar
vehicle. Instead of using handcrafted features [14, 31], we propose a deep
neural network designed to find loop closure candidates. Our network predicts
both, a so-called overlap defined on range images and a relative yaw angle
between two 3D LiDAR scans recorded with a typical sensor setup often used on
automated cars. The concept of overlap has been used in photogrammetry to
estimate image overlaps, see also Sec. III-A and III-B, and we use it on LiDAR
range images. It is a useful tool for loop closure detection as illustrated in
Fig. 1 and can quantify the quality of matches. The yaw estimate serves as an
initial guess for a subsequent application of iterative closest point (ICP)
[3] to determine the relative pose between scans to derive loop closures
constraints for the pose graph optimization. Instead of ICP, one could also
use global scan matching [6, 37, 34] to estimate the relative pose between
scans.

The main contribution of this paper is a deep neural network that exploits
different types of information generated from LiDAR scans to provide overlap
and relative yaw angle estimates between pairs of 3D scans. This information
includes depth, normals, and intensity or remission values. We additionally
exploit a probability distribution over semantic classes that can be computed
for each laser beam. Our approach relies on a spherical projection of LiDAR
scans, rather than the raw point clouds, which makes the proposed OverlapNet
comparably lightweight. We furthermore integrate it into a state-of-the-art
SLAM system [2] for loop closure detection and evaluate its performance also
with respect to generalization to different environments.

Figure 1: Overlap of two scans (blue and orange points) at a loop closure
location but computed with different relative transformations. The overlap
depends on the relative transformation and larger overlap values often
correspond to better alignment between the point clouds. Our approach can
predict the overlap without knowing the relative transformation between the
scans.

We train the proposed OverlapNet on parts of the KITTI odometry dataset and
evaluate it on unseen data.We thoroughly evaluate our approach, provide
ablation studies using different modalities, and test the integrated SLAM
system in an online manner. Furthermore, we provide results for the Ford
campus dataset, which was recorded using a different sensor setup in a
different country and a differently structured environment. The experimental
results suggest that our method outperforms other state-of-the-art baseline
methods and is also able to generalize well to unseen environments.

In sum, our approach is able to (i) predict the overlap and relative yaw angle
between pairs of LiDAR scans by exploiting multiple cues without using
relative poses, (ii) combine odometry information with overlap predictions to
detect correct loop closure candidates, (iii) improve the overall pose
estimation results in a state-of-the-art SLAM system yielding more globally
consistent maps, (iv) solve loop closure detection without prior pose
information, (v) initialize ICP using the OverlapNet predictions yielding
correct scan matching results. The implementation of our approach is available
at: https://github.com/PRBonn/OverlapNet

##  II Related Work

Loop closure detection using various sensor modalities [15, 28] is a classical
topic in robot mapping. We refer to the article by Lowry _et al_. [21] for an
overview of approaches using cameras. Here, we mainly concentrate on related
work addressing 3D LiDAR-based approaches.

Steder _et al_. [31] propose a place recognition system operating on range
images generated from 3D LiDAR data that uses a combination of bag-of-words
and a NARF-feature-based [30] relative poses estimation exploiting ideas of
FABMAP [10]. Röhling _et al_. [26] present an efficient method for detecting
loop closures through the use of similarity measures on histograms extracted
from 3D LiDAR scans. The work by He _et al_. [14] presents M2DP, which
projects a LiDAR scan into multiple reference planes to generate a descriptor
using a density signature of points in each plane. Besides using pure
geometric information, there is also work [8, 13] exploiting the remission
information, i.e., how well LiDAR beams are reflected by a surface, to create
descriptors for localization and loop closure detection with 3D LiDAR data.

Motivated by the success of deep learning in computer vision [20], deep
learning-based methods have been proposed recently. Barsan _et al_. [18]
propose a deep network-based localization method, which embeds LiDAR sweeps
and intensity maps into a joint embedding space and achieves localization by
matching between these embeddings. Dubé _et al_. [11] advocate the usage of
segments for loop closure detection. Cramariuc _et al_. [9] train a CNN to
extract descriptors from segments and use it to retrieve near-by place
candidates. Schaupp _et al_. [27] propose a system called OREOS for place
recognition, that also estimates the yaw discrepancy between scans.
Furthermore, Yin _et al_. [35] develop LocNet, which uses semi-handcrafted
feature learning based on a siamese network to solve place recognition. Lu _et
al_. [22] proposed $L^{3}$-net, which uses 3D convolutions and a recurrent
neural network to learn local descriptors for global localization. Uy and Lee
[33] proposed PointNetVLAD to generate a global descriptor for 3D point
clouds. Kim _et al_. [19] proposed a learning-based descriptor called SCI to
solve long-term global localization. Most recently, Sun _et al_. [32] also
proposed a learning-based method combined with Monte Carlo localization to
achieve a fast global localization.

Contrary to the above-mentioned methods, our method exploits multiple types of
information extracted from 3D LiDAR scans, including depth, normal
information, intensity/remission and probabilities of semantic classes
generated by a semantic segmentation system [23].

Similar to LocNet [35], we also use a siamese network, but we learn features
and yield predictions end-to-end. Our network can directly provide estimates
for overlap and the relative yaw angle between pairs of LiDAR scans. Different
from OREOS [27], our method not only provides loop closures candidates but
also an estimate of the matching quality in terms of the overlap.

Recently, Zaganidis _et al_. [36] proposed a Normal Distributions Transform
(NDT) histogram-based loop closure detection method, which is also assisted by
semantic information. In contrast to ours, their method needs a dense global
map and cannot estimate the relative yaw angle.

##  III Our Approach

###  III-A The Concept of Overlap

The idea of overlap that we are using here has its origin in the
photogrammetry and computer vision community [17]. To successfully match two
images and calculate their relative pose, the images must overlap. This can be
quantified by defining the overlap percentage as the percentage of pixels in
the first image, which can successfully be projected back into the second
image without occlusion. Note that this measure is not symmetric: If there is
a large scale difference of the image pair, e.g., one image shows a wall and
the other shows many buildings around that wall, the overlap percentage for
the first to the second image can be large and from the second to the first
image low. In this paper, we use the idea of overlap for range images,
exploiting the range information explicitly.

For loop closing, a threshold on the overlap percentage can be used to decide
whether two LiDAR scans are at the same place and/or a loop closing can be
done. For loop closing, this measure maybe even better than the commonly used
distance between the recorded positions of a pair of scans, since the
positions might be affected by drift and therefore unreliable. The overlap
predictions are independent of the relative poses and can be therefore used to
find loop closures without knowing the correct relative pose between scans.
Fig. 1 shows the overlap of two scans as an example.

###  III-B Definition of the Overlap between Pairs of LiDAR Scans

We use spherical projections of LiDAR scans as input data, which is often used
to speed up computations [2, 4, 7]. We project the point cloud $\mathcal{P}$
to a so-called vertex map $\mathcal{V}:\mathbb{R}^{2}\mapsto\mathbb{R}^{3}$,
where each pixel is mapped to the nearest 3D point. Each point
$\mathbf{p}_{i}=(x,y,z)$ is converted via the function
$\Pi:\mathbb{R}^{3}\mapsto\mathbb{R}^{2}$ to spherical coordinates and finally
to image coordinates $(u,v)$ by

$\displaystyle\left(\begin{array}[]{c}u\vspace{0.0em}\\\ v\end{array}\right)$
$\displaystyle=\left(\begin{array}[]{cc}\frac{1}{2}\left[1-\arctan(y,x)\pi^{-1}\right]w\vspace{0.5em}\\\
\left[1-\left(\arcsin(zr^{-1})+\mathrm{f}_{\mathrm{up}}\right)\mathrm{f}^{-1}\right]h\end{array}\right),$
(5)

where $r=||\mathbf{p}||_{2}$ is the range,
$\mathrm{f}=\mathrm{f}_{\mathrm{up}}+\mathrm{f}_{\mathrm{down}}$ is the
vertical field-of-view of the sensor, and $w,h$ are the width and height of
the resulting vertex map $\mathcal{V}$.

For a pair of LiDAR scans $\mathcal{P}_{1}$ and $\mathcal{P}_{2}$, we generate
the corresponding vertex maps $\mathcal{V}_{1}$, $\mathcal{V}_{2}$. We denote
the sensor-centered coordinate frame at time step $t$ as $C_{t}$. Each pixel
in coordinate frame $C_{t}$ is associated with the world frame $W$ by a pose
$\mathbf{T}_{WC_{t}}\in\mathbb{R}^{4\times 4}$. Given the poses
$\mathbf{T}_{WC_{1}}$ and $\mathbf{T}_{WC_{2}}$, we can reproject scan
$\mathcal{P}_{1}$ into the coordinate frame of the other’s vertex map
$\mathcal{V}_{2}$ and generate a reprojected vertex map
$\mathcal{V}^{\prime}_{1}$:

$\displaystyle\mathcal{V}^{\prime}_{1}$
$\displaystyle=\Pi\left(\mathbf{T}_{WC_{1}}^{-1}\mathbf{T}_{WC_{2}}\mathcal{P}_{1}\right).$
(6)

We then calculate the absolute difference of all corresponding pixels in
$\mathcal{V}^{\prime}_{1}$ and $\mathcal{V}_{2}$, considering only those
pixels that correspond to valid range readings in both range images. The
overlap is then calculated as the percentage of all differences in a certain
distance $\epsilon$ relative to all valid entries, i.e., the overlap of two
LiDAR scans $O_{C_{1}C_{2}}$ is defined as follows:

$\displaystyle O_{C_{1}C_{2}}$
$\displaystyle=\frac{\sum_{(u,v)}\mathbb{I}\Big{\\{}\left|\left|\mathcal{V}^{\prime}_{1}(u,v)-\mathcal{V}_{2}(u,v)\right|\right|\leq\epsilon\Big{\\}}}{\min\left(\textrm{valid}(\mathcal{V}^{\prime}_{1}),\textrm{valid}(\mathcal{V}_{2})\right)},$
(7)

where $\mathbb{I}\\{a\\}=1$ if $a$ is true and $0$ otherwise.
$\textrm{valid}(\mathcal{V})$ is the the number of valid pixels in
$\mathcal{V}$, since not all pixel might have a valid LiDAR measurement
associated after the projection.

We use Eq. (7) only for creating training data, i.e., only positive examples
of correct loop closures get a non-zero overlap assigned using the relative
poses between scans, as shown in Fig. 2(c). However, when performing loop
closure detection for online SLAM, the approximate relative poses from SLAM
before loop closure are not accurate enough to calculate usable overlaps by
using Eq. (7) because of accumulated drift. We tried directly estimating
overlaps using Eq. (7) assuming the relative pose as identity and applying
different orientations, e.g., every $30$ degrees rotation around the vertical
axis, and using the maximum over all these overlaps as an estimate. Fig. 2
shows the estimated overlaps for all scans using a query scan produced by this
method and the result of the estimated overlap for all scans using OverlapNet.
We leave out the 100 most recent scans because they will not be loop closure
candidates. In the case of the exhaustive approach, many wrong loop closure
candidates get high overlap values, while our approach performs better since
it produces a highly distinctive peak around the correct location.
Furthermore, it takes on average $1.2$ s to calculate the overlap for one pair
of scans using the exhaustive approach, which makes it unusable in real-world
scenarios. In contrast, the complete OverlapNet needs on average $17$ ms for
one pair overlap estimation when using depth and normal information only.

###  III-C Overlap Network Architecture

(a) Exhaustive
evaluation of Eq. (7) (b) OverlapNet
estimates

(c) Ground truth

Figure 2: Overlap estimations of one frame to all others. The red arrow points
out the position of the query scan. If we directly use Eq. (7) to estimate the
overlap between two LiDAR scans without knowing the accurate relative poses,
it is hard to decide which pairs of scans are true loop closures, since most
evaluations of Eq. (7) show high values. In contrast, our OverlapNet can
predict the overlaps between two LiDAR scans well. Figure 3: Pipeline overview
of our proposed approach. The left-hand side shows the preprocessing of the
input data which exploits multiple cues generated from a single LiDAR scan,
including range $\mathcal{R}$, normal $\mathcal{N}$, intensity $\mathcal{I}$,
and semantic class probability $\mathcal{S}$ information. The right-hand side
shows the proposed OverlapNet which consists of two legs sharing weights and
the two heads use the same pair of feature volumes generated by the two legs.
The outputs are the overlap and relative yaw angle between two LiDAR scans.

The overview of the proposed OverlapNet is depicted in Fig. 3. We exploit
multiple cues, which can be generated from a single LiDAR scan, including
depth, normal, intensity, and semantic class probability information. The
depth information is stored in the range map $\mathcal{R}$, which consists of
one channel. We use neighborhood information of the vertex map to generate a
normal map $\mathcal{N}$, which has three channels encoding the normal
coordinates. We directly obtain the intensity information, also called
remission, from the sensor and represent the intensity information as a one-
channel intensity map $\mathcal{I}$. The point-wise semantic class
probabilities are computed using RangeNet++ [23] and we represent them as a
semantic map $\mathcal{S}$. RangeNet++ delivers probabilities for $20$
different classes. For efficiency sake, we reduce the $20$-dimensional
RangeNet++ output to a compressed $3$-dimensional vector using principal
component analysis. The information is combined into an input tensor of size
$64\times 900\times D$, where $64,900$ are the height and width of the inputs,
and $D$ depends on the types of data used.

Our proposed OverlapNet is a siamese network architecture [5], which consists
of two legs sharing weights and two heads that use the same pair of feature
volumes generated by the two legs. The trainable layers are listed in Tab. I.

####  III-C1 Legs

The proposed OverlapNet has two legs, which have the same architecture and
share the same weights. Each leg is a fully convolutional network (FCN)
consisting of $11$ convolutional layers. This architecture is quite
lightweight and generates feature volumes of size $1\times 360\times 128$.
Note that our range images are cyclic projections and that a change in the yaw
angle of the vehicle results in a cyclic column shift of the range image.
Thus, the single row in the feature volume can represent a relative yaw angle
estimate (because a yaw angle rotation results in a pure horizontal shift of
the input maps). As the FCN is translation-equivariant, the feature volume
will be shifted horizontally. The number of columns of the feature volume
defines the resolution of the yaw estimation, which is $1$ degree in the case
of our leg architecture.

TABLE I: Layers of our network architecture | Operator | Stride | Filters | Size | Output Shape
---|---|---|---|---|---
Legs | Conv2D | (2, 2) | 16 | (5, 15) |  $30\times 443\times 16$
Conv2D | (2, 1) | 32 | (3, 15) |  $14\times 429\times 32$
Conv2D | (2, 1) | 64 | (3, 15) |  $6\times 415\times 64$
Conv2D | (2, 1) | 64 | (3, 12) |  $2\times 404\times 64$
Conv2D | (2, 1) | 128 | (2, 9) |  $1\times 396\times 128$
Conv2D | (1, 1) | 128 | (1, 9) |  $1\times 388\times 128$
Conv2D | (1, 1) | 128 | (1, 9) |  $1\times 380\times 128$
Conv2D | (1, 1) | 128 | (1, 9) |  $1\times 372\times 128$
Conv2D | (1, 1) | 128 | (1, 7) |  $1\times 366\times 128$
Conv2D | (1, 1) | 128 | (1, 5) |  $1\times 362\times 128$
Conv2D | (1, 1) | 128 | (1, 3) |  $1\times 360\times 128$
Delta Head | Conv2D | (1, 15) | 64 | (1, 15) |  $360\times 24\times 64$
Conv2D | (15, 1) | 128 | (15, 1) |  $24\times 24\times 128$
Conv2D | (1, 1) | 256 | (3, 3) |  $22\times 22\times 256$
| Dense | - | - | - |  $1$

####  III-C2 Delta Head

The delta head is designed to estimate the overlap between two scans. It
consists of a delta layer, three convolutional layers, and one fully connected
layer.

The delta layer, shown in Fig. 4, computes all possible absolute differences
of all pixels. It takes the output feature volumes
$\mathbf{L}^{l}\in\mathbb{R}^{H\times W\times C}$ from the two legs $l$ as
input. These are stacked in a tiled tensor
$\mathbf{T}^{l}\in\mathbb{R}^{HW\times HW\times C}$ as follows:

$\displaystyle\mathbf{T}^{0}(iW+j,k,c)$ $\displaystyle=\mathbf{L}^{0}(i,j,c)$
(8) $\displaystyle\mathbf{T}^{1}(k,iW+j,c)$
$\displaystyle=\mathbf{L}^{1}(i,j,c),$ (9)

with $k=\\{0,\dots,HW-1\\}$, $i=\\{0,\dots,H-1\\}$ and $j=\\{0,\dots,W-1\\}$.

Note that $\mathbf{T}^{1}$ is transposed in respect to $\mathbf{T}^{0}$, as
depicted in the middle of Fig. 4. After that, all differences are calculated
by element-wise absolute differences between $\mathbf{T}^{0}$ and
$\mathbf{T}^{1}$.

By using the delta layer, we can obtain a representation of the latent
difference information, which can be later exploited by the convolutional and
fully-connected layers to estimate the overlap. Different overlaps induce
different patterns in the output of the delta layer.

####  III-C3 Correlation Head

The correlation head [24] is designed to estimate the yaw angle between two
scans using the feature volumes of the two legs. To perform the cross-
correlation, we first pad horizontally one feature volume by copying the same
values (as the range images are cyclic projections around the yaw angle). This
doubles the size of the feature volume. We then use the other feature volume
as a kernel that is shifted over the first feature volume generating a 1D
output of size $360$. The $\mathop{\rm argmax}$ of this feature serves as the
estimate of the relative yaw angle of the two input scans with a $1$ degree
resolution.

###  III-D Loss Functions

We train our OverlapNet end-to-end to estimate the overlap and the relative
yaw angle between two LiDAR scans at the same time. Typically, to train a
neural network one needs a large amount of manually labeled ground truth data.
In our case, this is ($I_{1}$, $I_{2}$, $Y_{O}$, $Y_{Y}$), where $I_{1},I_{2}$
are two inputs and $Y_{O},Y_{Y}$ are the ground truth overlaps and the ground
truth yaw angles respectively. We are however able to generate the input and
the ground truth without any manual effort in a fully automated fashion given
a dataset with pose information. From given poses, we can calculate the ground
truth overlap and relative yaw angles directly. We denote the legs part
network with trainable weights as $f_{L}(\cdot)$, the delta head as
$f_{D}(\cdot)$ and the correlation head as $f_{C}(\cdot)$.

For training, we combine the loss $L_{O}(\cdot)$ for the overlap and the loss
$L_{Y}(\cdot)$ for the yaw angle using a weight $\alpha$:

$\displaystyle L\left(I_{1},I_{2},Y_{O},Y_{Y}\right)$
$\displaystyle=L_{O}\left(I_{1},I_{2},Y_{O}\right)+\alpha
L_{Y}\left(I_{1},I_{2},Y_{Y}\right).$ (10)

Figure 4: Delta layer. Computation of pairwise differences is efficiently
performed by concatenating the feature volumes and transposition of one
concatenated feature volume.

We treat the overlap estimation as a regression problem and use a weighted
absolute difference of ground truth $Y_{O}$ and network output
$\hat{Y}_{O}=f_{D}\left(f_{L}\left(I_{1}\right),f_{L}\left(I_{2}\right)\right)$
as the loss function. For weighting, we use a scaled sigmoid function:

$\displaystyle L_{O}\left(I_{1},I_{2},Y_{O}\right)$
$\displaystyle=\textrm{sigmoid}\left(s\left(\left|\hat{Y}_{O}-Y_{O}\right|+a\right)-b\right),$
(11)

with $\textrm{sigmoid}(v)=(1+\exp(-v))^{-1}$, $a,b$ are offsets and $s$ being
a scaling factor.

For the yaw angle estimation, we use a lightweight representation of the
correlation head output, which leads to a one-dimensional vector of size
$360$. We take the index of the maximum, the $\mathop{\rm argmax}$, as the
estimate of the relative angle in degrees. As the $\mathop{\rm argmax}$ is not
differentiable, we cannot treat this as a simple regression problem. The yaw
angle estimation, however, can be regarded as a binary classification problem
that decides for every entry of the head output whether it is the correct
angle or not. Therefore, we use the binary cross-entropy loss given by

$\displaystyle
L_{Y}\left(I_{1},I_{2},Y_{Y}\right)=\sum\nolimits_{i=\\{1,\dots,N\\}}\textrm{H}\left(Y_{Y}^{i},\hat{Y}_{Y}^{i}\right),$
(12)

where $\textrm{H}(p,q)=p\log(q)-(1-p)\log(1-q)$ is the binary cross entropy
and $N$ is the size of the output 1D vector.
$\hat{Y}_{Y}=f_{C}\left(f_{L}\left(I_{1}\right),f_{L}\left(I_{2}\right)\right)$
is the relative yaw angle estimate. Note that we only train the network to
estimate the relative yaw angle of a pair of scans with overlap larger than
$30\%$, since this minimum overlap was needed to result in correct pose
estimates of the ICP as explained in Sec. IV-A, but also experimentally
validated in Sec. IV-F.

###  III-E SLAM Pipeline

We use the surfel-based mapping system called SuMa [2] as our SLAM pipeline
and integrate OverlapNet in SuMa replacing its original heuristic loop closure
detection method. We only summarize here the steps of SuMa relevant to our
approach and refer for more details to the original paper [2].

SuMa uses the same vertex map $\mathcal{V}_{D}$ and normal map
$\mathcal{N}_{D}$ as discussed in Sec. III-B. Furthermore, SuMa uses
projective ICP with respect to a rendered map view $\mathcal{V}_{M}$ and
$\mathcal{N}_{M}$ at timestep $t-1$, the pose update
$\mathbf{T}_{C_{t-1}C_{t}}$ and consequently $\mathbf{T}_{WC_{t}}$ by chaining
all pose increments. Therefore, each vertex $\mathbf{u}\in\mathcal{V}_{D}$ is
projectively associated to a reference vertex
$\mathbf{v}_{\mathbf{u}}\in\mathcal{V}_{M}$. Given this association
information, SuMa estimates the transformation between scans by incrementally
minimizing the point-to-plane error given by

$\displaystyle E(\mathcal{V}_{D},\mathcal{V}_{M},\mathcal{N}_{M})$
$\displaystyle=\sum_{\mathbf{u}\in\mathcal{V}_{D}}\bigg{(}\mathbf{n}_{u}^{\top}\left(\mathbf{T}^{(k)}_{C_{t-1}C_{t}}\mathbf{u}-\mathbf{v}_{u}\right)\bigg{)}^{2}.$
(13)

Each vertex $\mathbf{u}\in\mathcal{V}_{D}$ is projectively associated to a
reference vertex $\mathbf{v}_{u}\in\mathcal{V}_{M}$ and its normal
$\mathbf{n}_{u}\in\mathcal{N}_{M}$ via

$\displaystyle\mathbf{v}_{u}$
$\displaystyle=\mathcal{V}_{M}\left(\Pi\left(\mathbf{T}^{(k)}_{C_{t-1}C_{t}}\mathbf{u}\right)\right)$
(14) $\displaystyle\mathbf{n}_{u}$
$\displaystyle=\mathcal{N}_{M}\left(\Pi\left(\mathbf{T}^{(k)}_{C_{t-1}C_{t}}\mathbf{u}\right)\right).$
(15)

SuMa then minimizes the objective of Eq. (13) using Gauss-Newton and
determines increments $\delta$ by iteratively solving

$\displaystyle\delta$
$\displaystyle=\left(\mathbf{J}_{\delta}^{\top}\mathbf{W}\mathbf{J}_{\delta}\right)^{-1}\mathbf{J}_{\delta}^{\top}\mathbf{W}\mathbf{r},$
(16)

where $\mathbf{W}\in\mathbb{R}^{n\times n}$ is a diagonal matrix containing
weights $w_{u}$, $\mathbf{r}\in\mathbb{R}^{n}$ is the stacked residual vector,
and $\mathbf{J}_{\delta}\in\mathbb{R}^{n\times 6}$ the Jacobian of
$\mathbf{r}$ with respect to the increment $\delta$.

SuMa employs a loop closure detection module, which considers the nearest
frame in the built map as the candidate for loop closure given the current
pose estimate. Loop closure detection works well for small loops, but the
heuristic fails in areas with only a few large loops. Furthermore, drift in
the odometry estimate can lead to large displacements, where the heuristic of
just taking the nearest frame in the already mapped areas does not yield
correct candidates, which will be also shown in our experiments.

###  III-F Covariance Propagation for Geometric Verification

SuMa’s loop closure detection uses a fixed search radius. In contrast, we use
the covariance of the pose estimate and error propagation to automatically
adjust the search radius.

We assume a noisy pose
$\mathbf{T}_{C_{t-1}C_{t}}=\\{\bar{\mathbf{T}}_{C_{t-1}C_{t}},\Sigma_{C_{t-1}C_{t}}\\}$
with mean $\bar{\mathbf{T}}_{C_{t-1}C_{t}}$ and covariance
$\Sigma_{C_{t-1}C_{t}}$. We can estimate the covariance matrix by

$\displaystyle\Sigma_{C_{t-1}C_{t}}$
$\displaystyle=\frac{1}{K}\frac{E}{N-M}\left(\mathbf{J}_{\delta}^{\top}\mathbf{W}\mathbf{J}_{\delta}\right)^{-1},$
(17)

where $K$ is the correction factors of the Huber robustized covariance
estimation [16], $E$ is the sum of the squared point-to-plane errors (sum of
squared residuals) given the pose $\mathbf{T}_{C_{t-1}C_{t}}$, see Eq. (13),
$N$ is the number of correspondences, $M=6$ is the dimension of the
transformation between two 3D poses.

To estimate the propagated uncertainty during the incrementally pose
estimation, we can update the mean and covariance as follows:

$\displaystyle\bar{\mathbf{T}}_{C_{t-1}C_{t+1}}$
$\displaystyle=\bar{\mathbf{T}}_{C_{t-1}C_{t}}\bar{\mathbf{T}}_{C_{t}C_{t+1}}$
(18) $\displaystyle\Sigma_{C_{t-1}C_{t+1}}$
$\displaystyle\approx\Sigma_{C_{t-1}C_{t}}+\mathbf{J}_{C_{t}C_{t+1}}^{\top}\Sigma_{C_{t}C_{t+1}}\mathbf{J}_{C_{t}C_{t+1}},$
(19)

where $\mathbf{J}_{C_{t}C_{t+1}}$ is the Jacobian of
${\mathbf{T}}_{C_{t}C_{t+1}}$.

Since we need the Mahalanobis distance $D_{M}$ as a probabilistic distance
measure between two poses, we make use of Lie algebra to express $\mathbf{T}$
as a 6D vector $\mathbf{\xi}\in\mathfrak{se}(3)$ using
$\mathbf{\xi}=\log{\mathbf{T}}$, yielding

$\displaystyle D_{M}\left(\mathbf{T}_{C1},\mathbf{T}_{C2}\right)$
$\displaystyle=\sqrt{\Delta\mathbf{\xi}_{C1C2}^{\top}\Sigma_{C1C2}^{-1}\Delta\mathbf{\xi}_{C1C2}}.$
(20)

Using the scaled distance, we can now restrict the search space depending on
the pose uncertainty to save computation time. However, we can use our
framework also without any prior information, i.e., perform place recognition.

##  IV Experimental Evaluation

The experimental evaluation is designed to support the key claims that our
approach is able to: (i) predict the overlap and relative yaw angle between
pairs of LiDAR scans by exploiting multiple cues without given poses, (ii)
combine odometry information with overlap predictions to detect correct loop
closure candidates, (iii) improve the overall pose estimation results in
graph-based SLAM yielding more globally consistent maps, (iv) solve loop
closure detection without prior pose information, (v) initialize ICP using
OverlapNet predictions yielding correct scan matching results.

We train and evaluate our approach on the KITTI odometry benchmark [12], which
provides LiDAR scans recorded with a Velodyne HDL-64E of urban areas around
Karlsruhe in Germany. We follow the experimental setup of Schaupp _et al_.
[27] and use sequence $00$ for evaluation. Sequences $03-10$ are used for
training and sequence $02$ is used for validation.

To evaluate the generalization ability of our method, we also test it on the
Ford campus dataset [25], which is recorded on the Ford research campus and
downtown Dearborn in Michigan using a different version of the Velodyne
HDL-64E. In the case of the Ford campus dataset, we test our method on
sequence $00$ which has several large loops. Note that we never trained our
approach on the Ford campus dataset.

For generating overlap ground truth, we only use points within a distance of
$75$ m to the sensor. For overlap computation, see Eq. (7), we use
$\epsilon=1\,$m. We use a learning rate of $10^{-3}$ with a decay of $0.99$
every epoch and train at most $100$ epochs. For the combined loss, Eq. (10),
we set $\alpha=5$. For the overlap loss, Eq. (11), we use $a=0.25,b=12,$ and
scale factor $s=24$.

###  IV-A Loop Closure Detection

(a) KITTI sequence $00$ (b) Ford campus sequence $00$

Figure 5: Precision-Recall curves of different approaches.

In our first experiments, we investigate the loop closure performance of our
approach and compare it to existing methods. Loop closure detection typically
assumes that robots revisit places during the mapping while moving with
uncertain odometry. Therefore, the prior information about robot poses
extracted from the pose graph is available for the loop closure detection. The
following criteria are used in these experiments:

  * •

To avoid detecting a loop closure in the most recent scans, we do not search
candidates in the latest $100$ scans.

  * •

For each query scan, only the best candidate is considered throughout this
evaluation.

  * •

Most SLAM systems search for potential closures only within the $3\sigma$ area
around the current pose estimate. We do the same, either using the Euclidean
or the Mahalanobis distance, depending on the approach.

  * •

We use a relatively low threshold of $30$ % for the overlap to decide if a
candidate is a true positive. We aim to find more loops even in some
challenging situations with low overlaps, e.g., when the car drives back to an
intersection from the opposite direction (as highlighted in the supplementary
video111https://youtu.be/YTfliBco6aw). Furthermore, ICP can find correct poses
if the overlap between pairs of scans is around $30$ %, as illustrated in the
experimental evaluation.

We evaluate OverlapNet on both KITTI sequence $00$ and Ford campus sequence
$00$ using the precision-recall curves shown in Fig. 5. We compare our method,
trained with two heads and all cues (labeled as Ours (AllChannel, TwoHeads))
with three state-of-the-art approaches, M2DP [14], Histogram [26], and the
original SuMa [2]. Since SuMa always uses the nearest frame as the candidate
for loop closure detection, we can only get one pair of precision and recall
value resulting in a single point. We also show the result of our method using
prior information, named Ours_CovNearestOfTop10, which uses covariance
propagation (Sec. III-F) to define the search space with the Mahalanobis
distance and use the nearest in Mahalanobis distance of the top $10$
predictions of OverlapNet as the loop closure candidates.

Tab. II shows the comparison between our approach and the state of the art
using the F1 score and the area under the curve (AUC) on both KITTI and Ford
campus dataset. For the KITTI dataset, our approach uses the model trained
with all cues, including depth, normals, intensity, and a probability
distribution over semantic classes. For the Ford campus dataset, our approach
uses the model trained with geometric information only, namely Ours (GeoOnly),
since other cues are not available in this dataset. We can see that our method
outperforms the other methods on the KITTI dataset and attains a similar
performance on the Ford campus dataset. There are two reasons to explain the
worse performance on the Ford campus dataset. First, we never trained our
network on the Ford campus dataset or even US roads, and secondly, there is
only geometric information available on the Ford campus dataset. However, our
method outperforms all baseline methods in both, KITTI and Ford campus
dataset, if we integrate prior information.

We also show the performance in comparison to variants of our method in Tab.
III. We compare our best model AllChannel using two heads and all available
cues to a variant which only uses a basic multilayer perceptron as the head
named MLPOnly which consists of two hidden fully connected layers and a final
fully connected layer with two neurons (one for overlap, one for yaw angle).
The substantial difference of the AUC and F1 scores shows that such a simple
network structure is not sufficient to get a good result. Training the network
with only one head (only the delta head for overlap estimation, named
DeltaOnly), has not a significant influence on the performance. A huge gain
can be observed when regarding the nearest frame in Mahalanobis distance of
the top $10$ candidates in overlap percentage (CovNearestOfTop10).

TABLE II: Comparison with state of the art. Dataset |  Approach |  AUC |  F1 score
---|---|---|---
KITTI |  Histogram [26] |  0.83 |  0.83
M2DP [14] |  0.83 |  0.87
SuMa [2] |  - |  0.85
Ours (AllChannel, TwoHeads) |  $\mathbf{0.87}$ |  $\mathbf{0.88}$
Ford Campus |  Histogram [26] |  0.84 |  0.83
M2DP [14] |  0.84 |  $\mathbf{0.85}$
SuMa [2] |  - |  0.33
Ours (GeoOnly) |  $\mathbf{0.85}$ |  0.84
TABLE III: Comparison with our variants. Dataset |  Variant |  AUC |  F1 score
---|---|---|---
KITTI |  MLPOnly |  0.58 |  0.65
DeltaOnly |  0.85 |  0.88
CovNearestOfTop10 |  $\mathbf{0.96}$ |  $\mathbf{0.96}$
Ours (AllChannel, TwoHeads) |  0.87 |  0.88
Ford Campus |  Ours (GeoOnly) |  0.85 |  0.84
GeoCovNearestOfTop10 |  $\mathbf{0.85}$ |  $\mathbf{0.88}$

###  IV-B Qualitative Results

The second experiment is designed to support the claim that our method is able
to improve the overall mapping result. Fig. 6 shows the odometry results on
KITTI sequence $02$. The color in Fig. 6 shows the 3D translation error
(including height). The left figure shows the SuMa and the right figure shows
Ours_CovNearestOfTop10 using the proposed OverlapNet to detect loop closures.
We can see that after integrating our method, the overall odometry is much
more accurate since we can provide more loop closure candidates with higher
accuracy in terms of overlap. The colors represent the translation error of
the estimated poses with respect to the ground truth. Furthermore, after
integrating the proposed OverlapNet, the SLAM system can find more loops even
in some challenging situations, e.g., when the car drives back to an
intersection from the opposite direction, which is highlighted in the
supplementary video.

(a) SuMa (b) Ours_CovNearestOfTop10

Figure 6: Qualitative result on KITTI sequence $02$.

###  IV-C Loop Closure Detection without Odometry Information

Figure 7: Loop closure detection performance on KITTI sequence $00$.

The third experiment is designed to support the claim that our approach is
well-suited for solving the more general loop closure detection task without
using odometry information.

In this case, we assume that we have no prior information about the robot
pose. To compare with the state-of-the-art method OREOS [27], we follow their
experimental setup and refer to the original paper for more details. The OREOS
results are those produced by the authors of OREOS.

The respective loop closure candidates recall results are shown in Fig. 7. Our
method outperforms all the baseline methods with a small number of candidates
and attains similar performance as baseline methods for higher values of
numbers of candidates. However, OREOS and LocNet++ attain a slightly higher
recall if more candidates are considered.

###  IV-D Yaw Estimation

TABLE IV: Yaw estimation errors without ICP Approach |  Mean[deg] |  std[deg] |  Recall[%]
---|---|---|---
FPFH+RANSAC* |  $13.28$ |  $32.19$ |  $97$
OREOS* |  $12.67$ |  $15.23$ |  $100$
Ours (AllChannel, TwoHeads) |  $\mathbf{1.13}$ |  $\mathbf{3.34}$ |  $\mathbf{100}$
*: The results are those produced by the authors of OREOS [27].

(a) KITTI sequence $00$ (b) Ford campus sequence $00$

Figure 8: Overlap and yaw estimation relationship.

We aim at supporting our claim that our network provides good relative yaw
angle estimates. We use the same experimental setup as described in Sec. IV-C.
Tab. IV summarizes the yaw angle errors on KITTI sequence $00$.

We can see that our method outperforms the other methods in terms of mean
error and standard deviations. In terms of recall, OverlapNet and OREOS always
provide a yaw angle estimate, since both approaches are designed to estimate
the relative yaw angle for any pairs of scans in contrast to the RANSAC-based
method that sometimes fails.

The superior performance can be mainly attributed to the correlation head
exploiting the fact that the orientation in LiDAR scans can be well
represented by the shift in the range projection. Therefore, it is easier to
train the correlation head to accurately predict the relative yaw angles
rather than a multilayer perceptron used in OREOS [27]. Furthermore, there is
also a strong relationship between overlap and yaw angle, which also improves
the results when trained together.

Fig. 8 shows the relationship between real overlap and yaw angle estimation
error. As expected, the yaw angle estimate gets better with increasing
overlap. Based on these plots, our method not only finds candidates but also
measures the quality, i.e., when the overlap of two scans is larger than
$90\%$, our method can accurately estimate the relative yaw angle with an
average error of about only $1$ degree.

###  IV-E Ablation Study on Input Modalities

TABLE V: Ablation study on usage of input modalities. Depth |  Normals |  Intensity |  Semantics | overlap | yaw angle[deg]
---|---|---|---|---|---
|  |  |  |  AUC |  F1 |  Mean |  Std
✓ |  |  |  |  0.86 |  0.87 |  11.67 |  25.32
✓ |  ✓ |  |  |  0.86 |  0.85 |  2.97 |  14.28
✓ |  ✓ |  ✓ |  |  0.87 |  0.87 |  2.53 |  14.56
✓ |  ✓ |  ✓ |  ✓ |  0.87 |  0.88 |  1.13 |  3.34

An ablation study on the usage of different inputs is shown in Tab. V. As can
be seen, when employing more input modalities, the proposed method is more
robust. We notice that exploiting only depth information with OverlapNet can
already perform reasonable in terms of overlap prediction, while it does not
perform well in yaw angle estimation. When combining with normal information,
the OverlapNet can perform well in both tasks. Another interesting finding is
the drastic reduction of yaw angle mean error and standard deviation when
using semantic information. One reason could be that adding semantic
information will make the input data more distinguishable when the car drives
in symmetrical environments. We also notice that semantic information will
increase the computation time, see Sec. IV-G. However, from the ablation
study, one could also notice that the proposed method can also achieve good
performance by only employing geometric information (depth and normals).

###  IV-F Using OverlapNet Predictions as Initial Guesses for ICP

We aim at supporting the claim that our network provides good initializations
for ICP with 3D laser scans collected on autonomous cars. Fig. 9 shows the
relation between the overlap and ICP registration error with and without using
OverlapNet predictions as initial guesses. The error of ICP registration is
here depicted by the Euclidean distance between the estimated relative
translation and the ground-truth translation. As can be seen, the yaw angle
prediction of the OverlapNet increases the chance to get a good result from
the ICP even if two frames are relatively far away from each other (with low
overlap). Therefore in some challenging cases, e.g. the car drives back into
an intersection from a different street, our approach can still find loop
closures (see in the supplementary video). The results also show that the
overlap estimates measure the quality of the found loop closure: larger
overlap values result in better registration results of the involved ICP.

###  IV-G Runtime

We tested our method on a system equipped with an Intel i7-8700 with $3.2$ GHz
and an Nvidia GeForce GTX1080 Ti with $11$ GB memory.

For the KITTI sequence $00$, we could exploit all input cues including the
semantic classes provided by RangeNet++ [23]. We need on average $75\,$ms per
frame for the input data preprocessing, $6\,$ms per frame for the legs feature
extraction, $27\,$ms per frame for the head matching. The worst case for the
head matching takes $630\,$ms for all candidates in the search space.

For the Ford campus dataset, we used only geometric information, which could
be generated in $10\,$ms on average per frame, $2\,$ms for feature extraction
and $24\,$ms for matching with the worst case of $550\,$ms. In real SLAM
operation, we only search loop closure candidates inside a certain search
space given by pose uncertainty using the Mahalanobis distance (see Sec.
III-F). Therefore, our method can achieve online operation in long-term tasks,
since we usually only have to evaluate a small number of candidate poses.

(a) With identity as initial guess (b) With yaw angle as initial guess

Figure 9: ICP using OverlapNet predictions as initial guess. The error of ICP
registration here is the Euclidean distance between the estimated translation
and the ground-truth translation.

##  V Conclusion

In this paper, we presented a novel approach for LiDAR-based loop closure
detection. It is based on the overlap between LiDAR scan range images and
provides a measure for the quality of the loop closure. Our approach utilizes
a siamese network structure to leverage multiple cues and allows us to
estimate the overlap and relative yaw angle between scans. The experiments on
two different datasets suggest that when combined with odometry information
our method outperforms other state-of-the-art methods and that it generalizes
well to different environments never seen during training.

Despite these encouraging results, there are several avenues for future
research. First, we want to investigate the integration of other input
modalities, such as vision and radar information. We furthermore plan to test
our approach with other datasets collected in different seasons.

## Acknowledgments

This work has been supported in part by the German Research Foundation (DFG)
under Germany’s Excellence Strategy, EXC-2070 - 390732324 (PhenoRob) and under
grant number BE 5996/1-1 as well as by the Chinese Scholarship Committee.

## References

  * Bailey and Durrant-Whyte [2006] T. Bailey and H.F. Durrant-Whyte.  Simultaneous localisation and mapping (SLAM): Part I.  _IEEE Robotics and Automation Magazine (RAM)_ , 13(2):99–110, 2006.  ISSN 1070-9932.  doi: 10.1109/MRA.2006.1638022.
  * Behley and Stachniss [2018] J. Behley and C. Stachniss.  Efficient Surfel-Based SLAM using 3D Laser Range Data in Urban Environments.  In _Proc. of Robotics: Science and Systems (RSS)_ , 2018.
  * Besl and McKay [1992] P.J. Besl and N.D. McKay.  A Method for Registration of 3D Shapes.  _IEEE Trans. on Pattern Analalysis and Machine Intelligence (TPAMI)_ , 14(2):239–256, 1992.
  * Bogoslavskyi and Stachniss [2016] I. Bogoslavskyi and C. Stachniss.  Fast range image-based segmentation of sparse 3d laser scans for online operation.  In _Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_ , 2016.
  * Bromley et al. [1993] J. Bromley, I. Guyon, Y. LeCun, E. Säckinger, and R. Shah.  Signature Verification using a “Siamese” Time Delayed Neural Network.  _IEEE Trans. on Pattern Analalysis and Machine Intelligence (TPAMI)_ , 07(04):669–688, 1993.  doi: 10.1142/S0218001493000339.
  * Censi and Carpin [2009] A. Censi and S. Carpin.  HSM3D: feature-less global 6DOF scan-matching in the Hough/Radon domain.  In _Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA)_, pages 3899–3906, 2009.
  * Chen et al. [2019] X. Chen, A. Milioto, E. Palazzolo, P. Giguère, J. Behley, and C. Stachniss.  SuMa++: Efficient LiDAR-based Semantic SLAM.  In _Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_ , 2019.
  * Cop et al. [2018] K.P. Cop, P.V.K. Borges, and R. Dub.  Delight: An efficient descriptor for global localisation using lidar intensities.  In _Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA)_, 2018.
  * Cramariuc et al. [2018] A. Cramariuc, R. Dube, H. Sommer, R. Siegwart, and I. Gilitschenski.  Learning 3D Segment Descriptors for Place Recognition.  _arXiv preprint_ , 2018.
  * Cummins and Newman [2009] M. Cummins and P. Newman.  Highly scalable appearance-only SLAM - FAB-MAP 2.0.  In _Proc. of Robotics: Science and Systems (RSS)_ , 2009.
  * Dubé et al. [2017] R. Dubé, D. Dugas, E. Stumm, J. Nieto, R. Siegwart, and C.C. Lerma.  SegMatch: Segment Based Place Recognition in 3D Point Clouds.  In _Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA)_, 2017.
  * Geiger et al. [2012] A. Geiger, P. Lenz, and R. Urtasun.  Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite.  In _Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 3354–3361, 2012.
  * Guo et al. [2019] J. Guo, P.V.K. Borges, C. Park, and A. Gawel.  Local descriptor for robust place recognition using LiDAR intensity.  _IEEE Robotics and Automation Letters (RA-L)_ , 4(2):1470–1477, 2019.
  * He et al. [2016] L. He, X. Wang, and H. Zhang.  M2DP: A Novel 3D Point Cloud Descriptor and Its Application in Loop Closure Detection.  In _Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_ , 2016.
  * Hess et al. [2016] W. Hess, D. Kohler, H. Rapp, and D. Andor.  Real-Time Loop Closure in 2D LIDAR SLAM.  In _Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA)_, 2016.
  * Huber [1981] P. J. Huber.  _Robust Statistics_.  Wiley, 1981.
  * Hussain and Bethel [2004] Mushtaq Hussain and James Bethel.  Project and mission planing.  In Chris McGlone, Edward Mikhail, James Bethel, and Roy Mullen, editors, _Manual of Photogrammetry_ , chapter 15.1.2.6, pages 1109–1111. American Society for Photogrammetry and Remote Sensing, 2004.
  * I.A. Barsan et al. [2018] I. Andrei I.A. Barsan, S. Wang, A. Pokrovsky, and R. Urtasun.  Learning to Localize Using a LiDAR Intensity Map.  In _Proc. of the Second Conference on Robot Learning (CoRL)_ , pages 605–616, 2018.
  * Kim et al. [2019] G. Kim, B. Park, and A. Kim.  1-day learning, 1-year localization: Long-term LiDAR localization using scan context image.  _IEEE Robotics and Automation Letters (RA-L)_ , 4(2):1948–1955, 2019.
  * Krizhevsky et al. [2017] A. Krizhevsky, I. Sutskever, and G.E. Hinton.  ImageNet Classification with Deep Convolutional Neural Networks.  _Communications of the ACM_ , 60(6):84–90, 2017\.
  * Lowry et al. [2016] S. Lowry, N. Sunderhauf, P. Newman, J.J Leonard, D. Cox, P. Corke, and M.J. Milford.  Visual place recognition: A survey.  _IEEE Trans. on Robotics (TRO)_ , 32(1):1–19, 2016.  ISSN 1552-3098.  doi: 10.1109/TRO.2015.2496823.
  * Lu et al. [2019] W. Lu, Y. Zhou, G. Wan, S. Hou, and S. Song.  L3-Net: Towards Learning Based LiDAR Localization for Autonomous Driving.  In _Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Milioto et al. [2019] A. Milioto, I. Vizzo, J. Behley, and C. Stachniss.  RangeNet++: Fast and Accurate LiDAR Semantic Segmentation.  In _Proceedings of the IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS)_ , 2019.
  * Nagashima et al. [2007] S. Nagashima, K. Ito, T. Aoki, H. Ishii, and K. Kobayashi.  A high-accuracy rotation estimation algorithm based on 1D phase-only correlation.  In _Proc. of the Intl. Conf. on Image Analysis and Recognition_ , pages 210–221, 2007.
  * Pandey et al. [2011] G. Pandey, J.R. McBride, and R.M. Eustice.  Ford campus vision and lidar data set.  _Intl. Journal of Robotics Research (IJRR)_ , 30(13):1543–1552, 2011.
  * Röhling et al. [2015] T. Röhling, J. Mack, and D. Schulz.  A Fast Histogram-Based Similarity Measure for Detecting Loop Closures in 3-D LIDAR Data.  In _Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_ , pages 736–741, 2015.
  * Schaupp et al. [2019] L. Schaupp, M. Bürki, R. Dubé, R. Siegwart, and C. Cadena.  OREOS: Oriented Recognition of 3D Point Clouds in Outdoor Scenarios.  _Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_ , 2019.
  * Stachniss et al. [2005] C. Stachniss, D. Hähnel, W. Burgard, and G. Grisetti.  On Actively Closing Loops in Grid-based FastSLAM.  _Advanced Robotics_ , 19(10):1059–1080, 2005\.
  * Stachniss et al. [2016] C. Stachniss, J. Leonard, and S. Thrun.  _Springer Handbook of Robotics, 2nd edition_ , chapter Chapt. 46: Simultaneous Localization and Mapping.  Springer Verlag, 2016.
  * Steder et al. [2010] B. Steder, R.B. Rusu, K. Konolige, and W. Burgard.  NARF: 3D range image features for object recognition.  In _Workshop on Defining and Solving Realistic Perception Problems in Personal Robotics at the IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS)_ , 2010.
  * Steder et al. [2011] B. Steder, M. Ruhnke, S. Grzonka, and W. Burgard.  Place Recognition in 3D Scans Using a Combination of Bag of Words and Point Feature Based Relative Pose Estimation.  In _Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_ , 2011.
  * Sun et al. [2020] L. Sun, D. Adolfsson, M. Magnusson, H. Andreasson, I. Posner, and T. Duckett.  Localising Faster: Efficient and precise lidar-based robot localisation in large-scale environments.  In _Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA)_, 2020.
  * Uy and Lee [2018] A.M. Uy and G.H. Lee.  PointNetVLAD: Deep point cloud based retrieval for large-scale place recognition.  In _Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 4470–4479, 2018.
  * Yang et al. [2020] H. Yang, J. Shi, and L. Carlone.  TEASER: Fast and Certifiable Point Cloud Registration.  _arXiv preprint_ , 2020.
  * Yin et al. [2019] H. Yin, Y. Wang, X. Ding, L. Tang, S. Huang, and R. Xiong.  3D LiDAR-Based Global Localization Using Siamese Neural Network.  _IEEE Trans. on Intelligent Transportation Systems (ITS)_ , 2019.
  * Zaganidis et al. [2019] A. Zaganidis, A. Zerntev, T. Duckett, and G. Cielniak.  Semantically Assisted Loop Closure in SLAM Using NDT Histograms.  In _Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_ , 2019.
  * Zhou et al. [2016] Q. Zhou, J. Park, and V. Koltun.  Fast Global Registration.  In _Proc. of the Europ. Conf. on Computer Vision (ECCV)_ , pages 766–782, 2016.


# Detailed balance in micro- and macrokinetics
and micro-distinguishability of macro-processes

A. N. Gorban<EMAIL_ADDRESS>Department of Mathematics, University of
Leicester, Leicester, LE1 7RH, UK

###### Abstract

We develop a general framework for the discussion of detailed balance and
analyse its microscopic background. We find that there should be two additions
to the well-known $T$\- or $PT$-invariance of the microscopic laws of motion:

1\. Equilibrium should not spontaneously break the relevant $T$\- or
$PT$-symmetry.

2\. The macroscopic processes should be microscopically distinguishable to
guarantee persistence of detailed balance in the model reduction from micro-
to macrokinetics.

We briefly discuss examples of the violation of these rules and the
corresponding violation of detailed balance.

###### keywords:

kinetic equation , random process , microreversibility , detailed balance ,
irreversibility

##  1 The history of detailed balance in brief

> VERY deep is the well of the past. … For the deeper we sound, the further
> down into the lower world of the past we probe and press, the more do we
> find that the earliest foundation of humanity, its history and culture,
> reveal themselves unfathomable.
>
> T. Mann [1]

Detailed balance as a consequence of the reversibility of collisions (at
equilibrium, each collision is equilibrated by the reverse collision, Fig. 1)
was introduced by Boltzmann for the Boltzmann equation and used in the proof
of the $H$-theorem [2] (Boltzmann’s arguments were analyzed by Tolman [3]).
Five years earlier, Maxwell used the principle of detailed balance for gas
kinetics with the reference to the principle of sufficient reason [4]. He
analyzed equilibration in cycles of collisions and in the pairs of mutually
reverse collisions and mentioned “Now it is impossible to assign a reason why
the successive velocities of a molecule should be arranged in this cycle,
rather than in the reverse order.”

Figure 1: Schematic representation of detailed balance for collisions: at
equilibrium, each collision is equilibrated by the reverse collision.

In 1901, Wegscheider introduced detailed balance for chemical kinetics on the
basis of classical thermodynamics [5]. He used the assumption that each
elementary reaction is reversible and should respect thermodynamics (i.e.
entropy production in this reaction should be always non-negative). Onsager
used this work of Wegsheider in his famous paper [6]. Instead of direct
citation he wrote: “Here, however, the chemists are accustomed to impose a
very interesting additional restriction, namely: when the equilibrium is
reached each individual reaction must balance itself.” Einstein used detailed
balance as a basic assumption in his theory of radiation [7]. In 1925, Lewis
recognized the principle of detailed balance as a new general principle of
equilibrium [8]. The limit of the detailed balance for systems which include
some irreversible elementary processes (without reverse processes) was
recently studied in detail [9, 10].

In this paper, we develop a general formal framework for discussion of
detailed balance, analyse its microscopic background and persistence in the
model reduction from micro- to macrokinetics.

##  2 Sampling of events, $T$-invariance and detailed balance

###  2.1 How detailed balance follows from microreversibility

In the sequel, we omit some technical details assuming that all the operations
are possible, all the distributions are regular and finite Borel (Radon)
measures, and all the integrals (sums) exist.

The basic notations and notions:

  * 1.

$\Omega$ – a space of states of a system (a locally compact metric space);

  * 2.

Ensemble $\nu$ – a non-negative distribution on $\Omega$;

  * 3.

Elementary process has a form $\alpha\to\beta$ (Fig. 2), where $\alpha$,
$\beta$ are non-negative distributions;

  * 4.

Complex – an input or output distribution of an elementary process.

  * 5.

$\Upsilon$ – the set of all complexes participating in elementary processes.
It is equipped with the weak topology and is a closed and locally compact set
of distributions.

  * 6.

The reaction rate $r$ is a measure defined on
$\Upsilon^{2}=\\{(\alpha,\beta)\\}$. It describes the rates of all elementary
processes $\alpha\to\beta$.

  * 7.

The support of $r$, supp$r\subset\Upsilon^{2}$, is the mechanism of the
process, i.e. it is the set of pairs $(\alpha,\beta)$, each pair represents an
elementary process $\alpha\to\beta$. (Usually,
supp$r\varsubsetneq\Upsilon^{2}$.)

  * 8.

The rate of the whole kinetic process is a distribution $W$ on $\Omega$ (the
following integral should exist):

$W=\frac{1}{2}\int_{(\alpha,\beta)\in\Upsilon^{2}}(\beta-\alpha){\mathrm{d}}[r(\alpha,\beta)-r(\beta,\alpha)].$

Figure 2: Schematic representation of an elementary process. Input ($\alpha$)
and output ($\beta$) distributions are represented by column histograms.

The distribution $\nu$ depends on time $t$. For systems with continuous time,
$\dot{\nu}=W$. For systems with discrete time, $\nu(t+\tau)-\nu(t)=W$, where
$\tau$ is the time step. To create the closed kinetic equation (the associated
nonlinear Markov process [11]) we have to define the map $\nu\mapsto r$ that
puts the reaction rate $r$ (a Radon measure on $\Upsilon^{2}$) in
correspondence with a non-negative distribution $\nu$ on $\Omega$ (the closure
problem). In this definition, some additional restrictions on $\nu$ may be
needed. For example, one can expect that $\nu$ is absolutely continuous with
respect to a special (equilibrium) measure. There are many standard examples
of kinetic systems: mass action law for chemical kinetics [12, 13], stochastic
models of chemical kinetics [18], the Boltzmann equation [14] in quasichemical
representation [15] for space-uniform distributions, the lattice Boltzmann
models [16], which represent the space motion as elementary discrete jumps
(discrete time), and the quasichemical models of diffusion [17].

We consider interrelations between two important properties of the measure
$r(\alpha,\beta)$:

(EQ) $W=0$ (equilibrium condition);

(DB) $r(\alpha,\beta)\equiv r(\beta,\alpha)$ (detailed balance condition).

It is possible to avoid the difficult closure question about the map
$\nu\mapsto r$ in discussion of $T$-invariance and relations between EQ and DB
conditions.

Obviously, DB$\Rightarrow$EQ. There exists a trivial case when
EQ$\Rightarrow$DB (a sort of linear independence of the vectors
$\gamma=\beta-\alpha$ for elementary processes joined in pairs with their
reverse processes): if ($\mu(\alpha,\beta)=-\mu(\beta,\alpha)$)

$\int_{(\alpha,\beta)\in{\rm
supp}r}(\beta-\alpha){\mathrm{d}}\mu(\alpha,\beta)=0\Rightarrow\mu=0$

for every antisymmetric measure $\mu$ on $\Upsilon^{2}$
($\mu(\alpha,\beta)=-\mu(\beta,\alpha)$), then EQ$\Rightarrow$DB.

There is a much more general reason for detailed balance, $T$-invariance.
Assume that the kinetics give a coarse-grained description of an ensemble of
interacting microsystems and this interaction of microsystems obeys a
reversible in time equation: if we look on the dynamics backward in time
(operation $T$) we will observe the solution of the same dynamic equations.
For $T$-invariant microscopic dynamics, $T$ maps an equilibrium ensemble into
an equilibrium ensemble. Assuming uniqueness of the equilibrium under given
values of the conservation laws, one can just postulate the invariance of
equilibria with respect to the time reversal transformation or $T$-invariance
of equilibria: if we observe an equilibrium ensemble backward in time, nothing
will change.

Let the complexes remain unchanged under the action of $T$. In this case, the
time reversal transformation for collisions (Fig. 1) leads to the reversal of
arrow: the direct collision is transformed into the reverse collision. The
same observation is valid for inelastic collisions. Following this hint, we
can accept that the reversal of time $T$ transforms every elementary process
$\alpha\to\beta$ into its reverse process $\beta\to\alpha$. This can be
considered as a restriction on the definition of direct and reverse processes
in the modelling (a “model engineering” restriction): the direct process is an
ensemble of microscopic events and the reverse process is the ensemble of the
time reversed events.

Under this assumption, $T$ transforms $r(\alpha,\beta)$ into
$r(\beta,\alpha)$. If the rates of elementary processes may be observed (for
example, by the counting of microscopic events in the ensemble) then
$T$-invariance of equilibrium gives DB: at equilibrium,
$r(\alpha,\beta)=r(\beta,\alpha)$, i.e. EQ$\Rightarrow$DB under the hypothesis
of $T$-invariance.

The assumption that the complexes are invariant under the action of $T$ may be
violated: for example, in Boltzmann’s collisions (Fig. 2) the input measure is
$\alpha=\delta_{v}+\delta_{w}$ and the output measure is
$\beta=\delta_{v^{\prime}}+\delta_{w^{\prime}}$. Under time reversal,
$\delta_{v}{\overset{T}{\mapsto}}\delta_{-v}$. Therefore
$\alpha{\overset{T}{\mapsto}}\delta_{-v}+\delta_{-w}$ and
$\beta{\overset{T}{\mapsto}}\delta_{-v^{\prime}}+\delta_{-w^{\prime}}$. We
need an additional invariance, the space inversion invariance (transformation
$P$) to prove the detailed balance (Fig. 1). Therefore, the detailed balance
condition for the Boltzmann equation (Fig. 1) follows not from $T$-invariance
alone but from $PT$-invariance because for Boltzmann’s kinetics

$\\{\alpha\to\beta\\}{\overset{PT}{\mapsto}}\\{\beta\to\alpha\\}.$

In any case, the microscopic reasons for the detailed balance condition
include existence of a symmetry transformation $\mathfrak{T}$ such that

$\\{\alpha\to\beta\\}{\overset{\mathfrak{T}}{\mapsto}}\\{\beta\to\alpha\\}$
(1)

and the microscopic dynamics is invariant with respect to $\mathfrak{T}$. In
this case, one can conclude that (i) the equilibrium is transformed by
$\mathfrak{T}$ into the same equilibrium (it is, presumably, unique) and (ii)
the reaction rate $r(\alpha,\beta)$ is transformed into $r(\beta,\alpha)$ and
does not change because nothing observable can change (equilibrium is the
same). Finally, at equilibrium $r(\alpha,\beta)\equiv r(\beta,\alpha)$ and
EQ$\Rightarrow$DB.

There remain two question:

  1. 1.

We are sure that $\mathfrak{T}$ transforms the equilibrium state into an
equilibrium state but is it necessarily the same equilibrium? Is it forbidden
that the equilibrium is degenerate and $\mathfrak{T}$ acts non-trivially on
the set of equilibria?

  2. 2.

We assume that the rates of different elementary processes are physical
observables and the ensemble with different values of these rates may be
distinguished experimentally. Is it always true?

The answer to both questions is “no”. The principle of detailed balance can be
violated even if the physical laws are $T$, $P$ and $PT$ symmetric. Let us
discuss the possible reasons for these negative answers and the possible
violations of detailed balance.

###  2.2 Spontaneous breaking of $\mathfrak{T}$-symmetry

Spontaneous symmetry breaking is a well known effect in phase transitions and
particle physics. It appears when the physical laws are invariant under a
transformation, but the equilibrium of the system transforms into another
state, which should be also equilibrium. Hence, the equilibrium is
degenerated. The best known examples are magnets. They are not rotationally
symmetric (there is a continuum of equilibria that differ by the direction of
magnetic field). Crystals are not symmetric with respect to translation (there
is continuum of equilibria that differ by a shift in space). In these two
examples, the multiplicity of equilibria is masked by the fact that all these
equilibrium states can be transformed into each other by a proper rigid motion
transformation (translation and rotation).

The nonreciprocal media violate $T$ and $PT$ invariance [19, 20, 21]. These
media are transformed by $T$ and $PT$ into different (dual) equilibrium media
and cannot be transformed back by a proper rigid motion. The implication
EQ$\Rightarrow$DB for the nonreciprocal media may be wrong and for its
validity some strong additional assumptions are needed, like the linear
independence of elementary processes.

Spontaneous breaking of $\mathfrak{T}$-symmetry provides us a counterexample
to the proof of detailed balance. In this proof, we used the assumption that
under transformation $\mathfrak{T}$ elementary processes transform into their
reverse processes (1) and, at the same time, the equilibrium ensemble does not
change.

If the equilibrium is transformed by $\mathfrak{T}$ into another (but
obviously also equilibrium) state then our reasoning cannot be applied to
reality and the proof is not valid. Nevertheless, the refutation of the proof
does not mean that the conclusion (detailed balance) is necessarily wrong.
Following the Lakatos terminology [23] we should call the spontaneous breaking
of $\mathfrak{T}$-symmetry the local counterexample to the principle detailed
balance. It is an intriguing question whether such a local counterexample may
be transformed into a global one: does the violation of the Onsager reciprocal
relation mean the violation of detailed balance (and not only the refutation
of its proof)?

###  2.3 Reciprocal relation and detailed balance

It is known that for many practically important kinetic laws the Onsager
reciprocal relations follow from detailed balance. In these cases, violation
of the reciprocal relations implies violation of the principle of detailed
balance. For example, for the systems in magnetic fields the reciprocal
relations may be violated [24], and we can expect that detailed balance for
these systems will be also violated.

For master equation (first order kinetics or continuous time Markov chains)
the principle of detailed balance is equivalent to the reciprocal relations
([24] Ch. 10, $\S$ 4). For the nonlinear mass action law the implication
“detailed balance $\Rightarrow$ reciprocal relations” is also well known (see,
for example, [12]) but the equivalence is not correct because the number of
nonlinear reactions for a given number of components may be arbitrarily large
and it is possible to select such values of reaction rate constants that the
reciprocal relations are satisfied but the principle of detailed balance does
not hold. For transport processes, the quasichemical models [17] also
demonstrate how the reciprocal relations follow from detailed balance for the
mass action law kinetics or the generalized mass action law. We confine the
discussion of kinetic laws to systems with finite sets of components.

Consider a finite-dimensional system with the set of components (species or
states) $A_{1},\ldots,A_{n}$ given. For each $A_{i}$ the extensive variable
$N_{i}$ (“amount” of $A_{i}$) is defined. The Massieu -Planck function
$\Phi(N,\ldots)$ (free entropy [25]) depends on the vector $N$ with
coordinates $N_{i}$ and on the variables that are constant under given
conditions. For isolated systems instead of $(\ldots)$ in $\Phi$ we should use
internal energy $U$ and volume $V$ (and this $\Phi$ is the entropy), for
isothermal isochoric systems these variables are $1/T$ and $V$, where $T$ is
temperature, and for isothermal isobaric systems we should use $1/T$ and
$P/T$, where $P$ is pressure. For all such conditions,

$\frac{\partial\Phi}{\partial N_{i}}=-\frac{\mu_{i}}{T},$

where $\mu_{i}$ is the chemical potential of $A_{i}$ or the generalized
chemical potential for the quasichemical models where interpretation of
$A_{i}$ is wider than just various atomic particles.

Elementary processes in the finite-dimensional systems are represented by
their stoichiometric equations

$\sum_{i}\alpha_{{\rho}i}A_{i}\to\sum_{i}\beta_{{\rho}i}A_{i}.$

This is a particular case of the general picture presented in Fig. 2. The
stoichiometric vector is $\gamma_{\rho}$:
$\gamma_{{\rho}i}=\beta_{{\rho}i}-\alpha_{{\rho}i}$ (gain minus loss). The
generalized mass action law represents the reaction rate in the following
form:

$r_{\rho}=\phi_{\rho}\exp\left(\sum_{i}\alpha_{{\rho}i}\frac{\mu_{i}}{RT}\right),$
(2)

where $\exp(\sum_{i}\alpha_{{\rho}i}{\mu_{i}}/{RT})$ is the Boltzmann factor
($R$ is the gas constant) and $\phi_{r}>0$ is the kinetic factor (this
representation is closely related to the transition state theory [26] and its
generalizations [27]).

The equilibria and conditional equilibria are described as the maximizers of
the free entropy under given conditions. For a system with detailed balance
every elementary process has a reverse process and the couple of processes
$\sum_{i}\alpha_{{\rho}i}A_{i}\rightleftharpoons\sum_{i}\beta_{{\rho}i}A_{i}$
should move the system from the initial state to the partial equilibrium, that
is the maximizer of the function $\Phi$ in the direction $\gamma_{\rho}$.
Assume that the equilibrium is not a boundary point of the state space. For a
smooth function $\Phi$, the conditional maximizer in the direction
$\gamma_{\rho}$ should satisfy the necessary condition
$\sum_{i}\gamma_{{\rho}i}\mu_{i}=0$. In the generalized mass action form (2)
the detailed balance condition has a very simple form:

$\phi_{\rho}^{+}=\phi_{\rho}^{-},$ (3)

where $\phi_{\rho}^{+}$ is the kinetic factor for the direct reaction and
$\phi_{\rho}^{-}$ is the kinetic factor for the reverse reaction.

Assume that the detailed balance condition (3) holds. Let us join the
elementary processes in pairs, direct with reverse ones, with the
corresponding change in their numeration. The kinetic equation is
$\dot{N}=V\sum_{\rho}\gamma_{\rho}(r_{\rho}^{+}-r_{\rho}^{-})$. The Jacobian
matrix at equilibrium is

$\left.\frac{\partial\dot{N}_{i}}{\partial N_{j}}\right|_{\rm
eq}=-\frac{V}{R}\sum_{k}\left(\sum_{\rho}r_{\rho}^{\rm
eq}\gamma_{{\rho}i}\gamma_{{\rho}k}\right)\left.\frac{\partial(\mu_{k}/T)}{\partial
N_{j}}\right|_{\rm eq},$

where $r_{\rho}^{\rm eq}=r_{\rho}^{+{\rm eq}}=r_{\rho}^{-{\rm eq}}$ is the
rate at equilibrium of the direct and reverse reactions (they coincide due to
detailed balance) and the subscript ‘eq’ corresponds to the derivatives at the
equilibrium. The linear approximation to the kinetic equations near the
equilibrium is

$\frac{{\mathrm{d}}{\Delta
N_{i}}}{{\mathrm{d}}t}=-\frac{V}{R}\sum_{k}\left(\sum_{\rho}r_{\rho}^{\rm
eq}\gamma_{{\rho}i}\gamma_{{\rho}k}\right)\Delta\left(\frac{\mu_{k}}{T}\right),$

where $\Delta N_{i}$ and $\Delta(\mu_{k}/T)$ are deviations from the
equilibrium values. The variables $\Delta N_{i}$ are extensive thermodynamic
coordinates and $\Delta(\mu_{k}/T)$ are intensive conjugated variables –
thermodynamic forces. Time derivatives ${\mathrm{d}}{\Delta
N_{i}}/{\mathrm{d}}t$ are thermodynamic fluxes. Symmetry of the matrix of
coefficients and, therefore, validity of the reciprocal relations is obvious.

Thus, for a wide class of kinetic laws the reciprocal relations in a vicinity
of a regular (non-boundary) equilibrium point follow from detailed balance in
the linear approximation. In these cases, the non-reciprocal media give global
counterexamples to the detailed balance. Without reference to a kinetic law
they remain local counterexamples to the proof of detailed balance.

###  2.4 Sampling of different macro-events from the same micro-events

In kinetics, only the total rate $W$ is observable (as $W=\dot{\nu}$ or
$W=\Delta\nu=\nu(t+\tau)-\nu(t)$). In the macroscopic world the observability
of the rates of the elementary processes is just a hypothesis.

Imagine a microscopic demon that counts collisions or other microscopic events
of various types. If different elementary processes correspond to different
types of microscopic events then the rates of elementary processes can be
observed. If the equilibrium ensemble is invariant with respect to
$\mathfrak{T}$ then the demon cannot detect the difference between the
equilibrium and the transformed equilibrium and the rates of elementary
processes should satisfy DB. But it is possible to sample the elementary
processes of macroscopic kinetics from the events of microscopic kinetics in
different manner.

For example, in chemical mass action law kinetics we can consider the reaction
mechanism $A\rightleftharpoons B$ (rate constants $k_{\pm 1}$),
$A+B\rightleftharpoons 2B$ (rate constants $k_{\pm 1}$) [22]. We can also
create a stochastic model for this system with the states $(xA,yB)$ ($x$, $y$
are nonnegative integers) and the elementary transitions
$(xA,yB)\rightleftharpoons((x-1)A,(y+1)B)$ (rate constants
$\kappa_{+}=k_{+1}x+k_{+2}x^{2}$, $\kappa_{-}=k_{-1}(y+1)+k_{-2}(x-1)(y+1)$).
The elementary transitions in this stochastic model are linearly independent
and EQ$\Leftrightarrow$DB. In the corresponding mass action law chemical
kinetics detailed balance requires additional relation between constants:
$k_{+1}/k_{-1}=k_{+2}/k_{-2}$.

Thus, macroscopic detailed balance may be violated in this example when
microscopic detailed balance holds. (For more examples and theoretic
consideration of the relations between detailed balance in mass action law
chemical kinetics and stochastic models of these systems see [22].) Indeed,
both of the macroscopic elementary processes $A\rightleftharpoons B$ and
$A+B\rightleftharpoons 2B$ correspond to the same set of microscopic
elementary processes $(xA,yB)\rightleftharpoons((x-1)A,(y+1)B)$. Each of these
elementary event is “shared” between two different macroscopic elementary
processes. Therefore, the macroscopic elementary processes in this example are
microscopically indistinguishable.

The microscopic indistinguishability in this example follows from the
coincidence of the stoichiometric vectors for two macroscopic processes
$A\rightleftharpoons B$ and $A+B\rightleftharpoons 2B$. If the stoichiometric
vectors are just linear dependent then it does not imply microscopic
indistinguishability.

For example, let us take two reactions $A\rightleftharpoons B$ and
$2A\rightleftharpoons 2B$. For the first reaction the corresponding
microscopic processes have the form $(xA,yB)\rightleftharpoons((x-1)A,(y+1)B)$
(if all the coefficients are nonnegative). For the reaction
$2A\rightleftharpoons 2B$ the microscopic processes have the form
$(xA,yB)\rightleftharpoons((x-2)A,(y+2)B)$ (if all the coefficients are
nonnegative). These sets do not intersect, the elementary processes are
microscopically distinguishable and macroscopic detailed balance follows from
microscopic detailed balance.

Nontrivial Wegscheider identities appear in this example at the microscopic
level (in the first example all the microscopic transitions are linearly
independent and there exist no additional relations). Let the microscopic
reaction rate constants for the reaction
$(xA,yB)\rightleftharpoons((x-1)A,(y+1)B)$ be $\kappa_{1}^{\pm}(x,y)$ and
$\kappa_{2}^{\pm}(x,y)$ for the reaction
$(xA,yB)\rightleftharpoons((x-2)A,(y+2)B)$. Due to detailed balance, in each
cycle of a linear reaction network the product of reaction rate constants in
the clockwise direction coincides with the product in the anticlockwise
direction. It is sufficient to consider the basis cycles (and their
reversals):

$\begin{split}(xA,yB)&\to((x-1)A,(y+1)B)\to\\\
&\to((x-2)A,(y+2)B)\to(xA,yB).\end{split}$

Therefore,

$\begin{split}\kappa_{1}^{+}(x,y)&\kappa_{1}^{+}(x-1,y+1)\kappa_{2}^{-}(x,y)\\\
&=\kappa_{2}^{+}(x,y)\kappa_{1}^{-}(x-1,y+1)\kappa_{1}^{-}(x,y).\end{split}$

In the macroscopic limit these conditions transform into the macroscopic
detailed balance conditions.

##  3 Relations between elementary processes beyond microreversibility and
detailed balance

If microreversibility does not exist, is everything permitted? What are the
the relations between the reaction rates beyond the microreversibility
conditions if such universal relations exist? The radical point of view is:
beyond the microreversibility we face just the world of kinetic equations with
preservation of positivity, various specific restrictions on the coefficients
appear in some specific cases and the variety of these cases in unobservable.
Development of this point of view leads to the general theory of nonlinear
Markov processes [11], i.e. the general theory of kinetic equations with
preservation of positivity.

The problem of the relations between elementary processes beyond
microreversibility and detailed balance was stated by Lorentz in 1887 [28].
Boltzmann immediately proposed the solution [29] and used it for extension of
his $H$-theorem beyond microreversibility. These conditions have the form of
partially summed conditions of detailed balance (Fig. 3, compare to Fig. 1).
This solution was analyzed, generalized and proved by several generations of
researchers (Heitler, Coester, Watanabe, Stueckelberg [30] and others, see the
review in [27]). It was rediscovered in 1972 [31] in the context of chemical
kinetics and popularized as the complex balance condition.

Figure 3: Boltzmann’s cyclic balance (1887) (or semi-detailed balance or
complex balance) is a summarised detailed balance condition: at equilibrium
the sum of intensities of collisions with a given input $v+w\to\ldots$
coincides with the sum of intensities of collisions with the same output
$\ldots\to v+w$ (for general systems see (4)).

For finite-dimensional systems which obey the generalized mass action law (2)
the complex balance condition is also the summarized detailed balance
condition (3). Consider the set $\Upsilon$ of all input and output vectors
$\alpha_{\rho}$ and $\beta_{\rho}$. The complex balance condition reads: for
every $y\in\Upsilon$

$\sum_{\rho,\,\alpha_{\rho}=y}\phi_{\rho}=\sum_{\rho,\,\beta_{\rho}=y}\phi_{\rho}.$
(4)

Now, the complex balance conditions in combination with generalized mass
action law are proven for the finite-dimensional systems in the asymptotic
limit proposed first by Michaelis and Menten [32] for fermentative reactions
and Stueckelberg [30] for the Boltzmann equation. This limit is constituted by
three assumptions (Fig. 4): (i) the elementary processes go through the
intermediate compounds, (ii) the compounds are in fast equilibria with the
components (therefore, these equilibria can be described by thermodynamics)
and (iii) the concentrations of compounds are small with respect to
concentrations of components (hence, (iiiA) the quasi steady state assumption
is valid for the compound kinetics and (iiiB) the transitions between
compounds follow the first order kinetics) [27]. (It is worth mentioning that
Michaelis and Menten in 1913 [32] found the asymptotic limit where the
fermentative reaction can be described by the mass action law. The so-called
Michaelis–Menten kinetics is different and was invented 12 years later by
Haldane and Briggs [33]).

Figure 4: Schematic representation of the Michaelis–Menten–Stueckelberg
asymptotic assumptions: an elementary process $\sum\alpha_{\rho
i}A_{i}\to\sum\alpha_{\rho i}A_{i}$ goes through intermediate compounds
$B_{\rho}^{\pm}$. The fast equilibria $\sum\alpha_{\rho
i}A_{i}\rightleftharpoons B_{\rho}^{+}$ and $\sum\beta_{\rho
i}A_{i}\rightleftharpoons B_{\rho}^{-}$ can be described by conditional
maximum of the free entropy. Concentrations of $B_{\rho}^{\pm}$ are small and
reaction between them obeys linear kinetic equation.

Thus, beyond microreversibility, Boltzmann’s cyclic balance (or semi-detailed
balance, or complex balance) holds and it is as universal as the idea of
intermediate compounds (activated complexes or transition states) which exist
in small concentrations and are in fast equilibria with the basic reagents.

##  4 Conclusion

Thus, EQ$\Leftrightarrow$DB if:

  1. 1.

There exists a transformation $\mathfrak{T}$ that transforms the elementary
processes into reverse processes and the microscopic laws of motion are
$\mathfrak{T}$-invariant;

  2. 2.

The equilibrium is symmetric with respect to $\mathfrak{T}$, that is, there is
no spontaneous breaking of $\mathfrak{T}$-symmetry;

  3. 3.

The macroscopic elementary processes are microscopically distinguishable. That
is, they represent disjoint sets of microscopic events.

In applications, $\mathfrak{T}$ is usually either time reversal $T$ or the
combined transform $PT$.

For level jumping (reduction of kinetic models [15]), the equivalence
EQ$\Leftrightarrow$DB persists in the reduced (“macroscopic”) model if:

  1. 1.

EQ$\Leftrightarrow$DB in the original (“microscopic”) model;

  2. 2.

Equilibria of the macroscopic model correspond to equilibria of the
microscopic model. That is, the reduced kinetic model has no equilibria, which
correspond to non-stationary dynamical regimes of the original kinetic model;

  3. 3.

The macroscopic elementary processes are microscopically distinguishable. That
is, they represent disjoint sets of microscopic processes.

In this note, we avoid the discussion of an important part of Boltzmann’s
legacy which is very relevant to the topic under consideration. Boltzmann
represented kinetic process as an ensemble of indivisible elementary events —
collisions. In the microscopic world, a collision is a continuous in time and
infinitely divisible process (and it requires infinite time in most of the
models of pair interaction). In the macroscopic world it is instant and
indivisible. The transition from continuous motion of particles to an ensemble
of indivisible instant collisions is not digested by modern mathematics up to
now, more than 130 years after its invention. The known results [34, 35] state
that the Boltzmann equation for an ensemble of classical particles with pair
interaction and short–range potentials is asymptotically valid starting from a
non-correlated state during a fraction of the mean free flight time. That is
very far from the area of application. Nevertheless, if we just accept that it
is possible to count microscopic events then the reasons of validity and
violations of detailed balance in kinetics are clear.

## References

  * [1] T. Mann, Joseph and his brothers. Prelude, Translated by H. T. Lowe-Porter, A.A. Knopf, Inc., NY, 1945.
  * [2] L. Boltzmann, Weitere Studien über das Wärmegleichgewicht unter Gasmolekülen, Sitzungsber. Kais. Akad. Wiss. 66 (1872), 275–370.
  * [3] R.C. Tolman, The Principles of Statistical Mechanics. Oxford University Press, London, UK, 1938.
  * [4] J.C. Maxwell, On the dynamical theory of gases. Philosophical Transactions of the Royal Society of London, 157 (1867), 49–88.
  * [5] R. Wegscheider, Über simultane Gleichgewichte und die Beziehungen zwischen Thermodynamik und Reactionskinetik homogener Systeme, Monatshefte für Chemie / Chemical Monthly 32(8) (1901), 849–906.
  * [6] L. Onsager, Reciprocal relations in irreversible processes. I, Phys. Rev. 37 (1931), 405–426.
  * [7] A. Einstein, Strahlungs–Emission und –Absorption nach der Quantentheorie, Verhandlungen der Deutschen Physikalischen Gesellschaft 18 (13/14) (1916). Braunschweig: Vieweg, 318–323.
  * [8] G.N. Lewis, A new principle of equilibrium, Proceedings of the National Academy of Sciences of the United States 11 (1925), 179–183.
  * [9] A.N. Gorban, G.S.Yablonsky, Extended detailed balance for systems with irreversible reactions, Chemical Engineering Science 66 (2011) 5388–5399; arXiv:1012.2908 [cond-mat.mtrl-sci].
  * [10] A.N. Gorban, E.M. Mirkes, G.S. Yablonsky, Thermodynamics in the limit of irreversible reactions, Physica A 392 (2013) 1318–1335; arXiv:1207.2507 [cond-mat.stat-mech].
  * [11] V.N. Kolokoltsov, Nonlinear Markov processes and kinetic equations, Cambridge University Press, London, 2010.
  * [12] G.S. Yablonskii, V.I. Bykov, A.N. Gorban, V.I. Elokhin, Kinetic Models of Catalytic Reactions, Elsevier, Amsterdam, The Netherlands, 1991.
  * [13] G. Marin, G.S. Yablonsky, Kinetics of chemical reactions. John Wiley & Sons, Weinheim, Germany, 2011.
  * [14] C. Cercignani, The Boltzmann equation and its applications, Springer, New York, 1988.
  * [15] A.N. Gorban, I.V. Karlin, Invariant Manifolds for Physical and Chemical Kinetics, Lect. Notes Phys. 660, Springer, Berlin–Heidelberg, 2005.
  * [16] S. Succi, The lattice Boltzmann equation for fluid dynamics and beyond, Clarendon Press, Oxford, 2001.
  * [17] A.N. Gorban, H.P. Sargsyan, H.A. Wahab, Quasichemical Models of Multicomponent Nonlinear Diffusion, Mathematical Modelling of Natural Phenomena 6 (05) (2011), 184–262; arXiv:1012.2908 [cond-mat.mtrl-sci].
  * [18] D.T. Gillespie, Stochastic simulation of chemical kinetics, Annu. Rev. Phys. Chem. 58 (2007), 35–55.
  * [19] C.M. Krowne, Nonreciprocal electromagnetic properties of composite chiral-ferrite media, In IEE Proceedings H (Microwaves, Antennas and Propagation) 140 (3) (1993), 242–248.
  * [20] E.O. Kamenetskii, Onsager–Casimir principle and reciprocity relations for bianisotropic media. Microwave and Optical Technology Letters 19 (6) (1998), 412–416.
  * [21] A. Guo, G.J. Salamo, D. Duchesne, R. Morandotti, M. Volatier-Ravat, V. Aimez, G.A. Siviloglou, D.N. Christodoulides, Observation of PT-symmetry breaking in complex optical potentials, Phys. Rev. Lett. 103 (2009), 093902.
  * [22] B. Joshi, Deterministic detailed balance in chemical reaction networks is sufficient but not necessary for stochastic detailed balance (2013), arXiv:1312.4196 [math.PR].
  * [23] I. Lakatos, Proofs and Refutations, Cambridge University Press, Cambridge, 1976.
  * [24] S.R. de Groot, P. Mazur, Non-equilibrium thermodynamics, Dover Publ. Inc., NY, 1984.
  * [25] H.B. Callen, Thermodynamics and an Introduction to Themostatistics (2nd ed.), John Wiley & Sons, NY, 1985.
  * [26] H. Eyring, The activated complex in chemical reactions, J. Chem. Phys. 3 (1935), 107–115.
  * [27] A.N. Gorban, M. Shahzad, The Michaelis–Menten–Stueckelberg theorem, Entropy 13 (2011), 966–1019; arXiv:1008.3296 [physics.chem-ph].
  * [28] H.-A. Lorentz, Über das Gleichgewicht der lebendigen Kraft unter Gasmolekülen, Sitzungsber. Kais. Akad. Wiss. 95 (2) (1887), 115–152.
  * [29] L. Boltzmann, Neuer Beweis zweier Sätze über das Wärmegleichgewicht unter mehratomigen Gasmolekülen. Sitzungsber. Kais. Akad. Wiss. 95 (2) (1887), 153–164.
  * [30] E.C.G. Stueckelberg, Theoreme $H$ et unitarite de $S$, Helv. Phys. Acta 25 (1952), 577–580.
  * [31] F. Horn, R. Jackson, General mass action kinetics, Arch. Ration. Mech. Anal. 47 (1972), 81–116.
  * [32] L. Michaelis, M. Menten, Die kinetik der Intervintwirkung, Biochem. Z. 49 ( 1913), 333–369.
  * [33] G.E. Briggs, J.B.S. Haldane, A note on the kinetics of enzyme action, Biochem. J. 19 (1925), 338–339.
  * [34] O. E. Lanford III, Time evolution of large classical systems, In Dynamical systems, theory and applications, J. Moser (ed.), Lect. Notes Phys., 38, Springer, Berlin–Heidelberg, 1–111, 1975.
  * [35] I. Gallagher, L. Saint-Raymond, and B. Texier, From Newton to Boltzmann: hard spheres and short-range potentials, Zürich Lectures in Advanced Mathematics, European Mathematical Society Publishing House, Zürich, 2014; arXiv:1208.5753 [math.AP].