# Analysis of Scalable Anomalous Reflectors through Ray Tracing and
Measurements

Le Hao, Sravan K. R. Vuyyuru, Sergei A. Tretyakov,
Markus Rupp, and Risto Valkonen This work was supported in part by the
European Union’s Horizon 2020 MSCA-ITN-METAWIRELESS project, under the Marie
Skłodowska-Curie grant agreement No 956256 and the Research Council of
Finland, grant 345178. (Corresponding author: Le Hao)L. Hao and M. Rupp are
with TU Wien, Gusshausstrasse 25, 1040 Vienna, Austria. (e-mail: {le.hao,
markus.rupp}@tuwien.ac.at).S. K. R. Vuyyuru is with Nokia Bell Labs, Karakaari
7, 02610 Espoo, Finland and the Department of Electronics and Nanoengineering,
School of Electrical Engineering, Aalto University, 02150 Espoo, Finland
(e-mail<EMAIL_ADDRESS>sravan.vuyyuru@aalto.fi).S. A. Tretyakov is
with the Department of Electronics and Nanoengineering, School of Electrical
Engineering, Aalto University, 02150 Espoo, Finland (e-mail:
sergei.tretyakov@aalto.fi).R. Valkonen is with Nokia Bell Labs, Karakaari 7,
02610 Espoo, Finland (e-mail: risto.valkonen@nokia-bell-labs.com).

###### Abstract

In this study, we elaborate on the concept of scalable anomalous reflector
(AR) to analyze the angular response, frequency response, and spatial
scalability of a designed AR across a broad range of angles and frequencies.
We utilize theoretical models and ray tracing simulations to investigate the
communication performance of two different-sized scalable finite ARs, one
smaller configuration with $48\times 48$ array of unit cells and the other
constructed by combining four smaller ARs to form a larger array with
$96\times 96$ unit cells. To validate the developed theoretical approach, we
conducted measurements in an auditorium to evaluate the received power through
an AR link at different angles and frequencies. In addition, models of
scalable deflectors are implemented in the MATLAB ray tracer to simulate the
measurement scenario. The results from theoretical calculations and ray
tracing simulations achieve good agreement with measurement results.

###### Index Terms:

Anomalous reflector, measurement, ray tracing, 6G.

BS

    base station
ECDF

    empirical cumulative distribution function
EM

    electromagnetic
LoS

    line-of-sight
SISO

    single-input single-output
SLS

    system-level simulator
AR

    anomalous reflector
OFDM

    orthogonal frequency-division multiplexing
EVM

    error vector magnitude

##  I Introduction

Anomalous reflectors with dense two-dimensional grids of subwavelength unit
cells have gained attention in realizing deflection of incident waves towards
desired directions [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]. The
electromagnetic (EM) properties of each unit cell are controlled to redirect
the impinging waves in the arbitrary desired direction. The initial research
on ARs using conventional methods [14, 15] suffers from lower efficiencies for
anomalous reflection angles beyond $45\degree$, e.g. [1]. Extensive research
on ARs established advanced design methods using periodic reflectors, such as
highly efficient anomalous reflectors at a steep angle, e.g. [2, 3] or
multichannel reflection [11, 12].

Even though extensive research on ARs has been carried out in the past years,
current researches usually focus on the electromagnetic parameters of ARs,
such as the anomalous reflection efficiency and scattering pattern, but
significantly lacking in comprehensive analysis of communication aspects in
realistic environments [16, 17, 18, 19]. More studies are needed to find out
how to incorporate these models into network simulators. Contrarily, the
majority of research communication aspects of the use of AR are based on
simplified or even idealized models that have not been validated for realistic
ARs that are designed using electromagnetic-theory methods [20, 21]. There is
also a deficiency in research addressing the frequency selectivity and spatial
scalability of ARs. In this paper, we address these gaps by integrating the
electromagnetic parameters of an EM-designed and manufactured AR into
different communication models and software that are often used in
communication system modeling. We theoretically evaluate large-scale fading of
AR-assisted communication links and utilize a ray tracer to simulate realistic
indoor scenarios involving an anomalous reflector. The theoretical predictions
and ray tracing model simulations have been compared and verified by empirical
measurements.

To the best knowledge of the authors, this is the first work that analyzes the
angular response and frequency response of a manufactured 3-bit AR in the
communication models through ray tracing and measurements in a dense indoor
scenario. In addition, we analyze the size scalability of the manufactured ARs
by tiling multiple small AR panels into a larger-size AR. Since the
implementation of the AR in a ray tracer has been verified by simulation and
measurements, this method can be used for any type of AR. As long as the
scattering pattern of the AR is available from analytical estimations,
simulations, or test measurements, we can use a ray tracer to obtain very
accurate results for communication links in complex environments without the
need to perform measurements in every particular setting.

The remainder of this article is organized as follows: Section II briefly
introduces the designed and manufactured AR, as well as its validated
scattering pattern. Section III is devoted to the measurement system,
including the measurement scenario and setups. In Section IV, we present the
comparison of the results between the theoretical model, the ray tracing
simulations, and measurements for two ARs of different sizes. Finally,
conclusions are formulated in Section V.

##  II Scalable Anomalous Reflector: Design and Manufacturing Strategy

Following the methodology proposed in [12, 13] for modeling AR from EM and
communication-related aspects, we employed the same design synthesis and
manufacturing strategy to develop scalable ARs. In [12], we introduce a
reconfigurable anomalous reflector enabled by an algebraic array antenna
scattering synthesis technique [3] to optimize the scattering characteristics
of passively loaded periodic arrays of patch elements. Using the load
impedances as the optimization variables for the scattering synthesis problem
replaces heavy computations with supercell-level optimization. As proposed in
[12], the flat multimode electrically large 16-element supercell engineers the
response of the excited fixed normal incident wave into nine anomalous
reflection directions, i.e., allowing reconfigurable reflected Floquet
harmonics up to fourth order with an infinite periodic array. The Floquet
expansion of currents induced on a periodic metasurface with no parasitic
scattering and the supercell dimension determined from Floquet-Bloch theory is
$d=4\lambda/|\sin\theta^{r,\text{max}}|$, with the metasurface element
periodicity being $d/16$, where the maximum desired deflection direction is
$\theta^{r,\text{max}}=65\degree$. Following [12], we utilize the same
synthesized load parameters to reflect the normal incidence wave to the
$+65\degree$ anomalous direction at the center frequency of $26$ GHz.

Recalling the manufacturing process outlined in [22, 12], we built a passive
and fixed-response anomalous reflector prototype deflecting a normal incident
wave to a 65-degree anomalous angle. Each square structured unit cell with
dimensions of $\Delta x\times\Delta y=d/16\times d/16=0.2758\lambda\times
0.2758\lambda$ are linearly arranged in the $x$-dimension, resulting in the
16-element supercell size being $4.4135\lambda\times 0.2758\lambda$ in
$(x\times y)$ dimensions. Each unit cell is constructed by a copper square
patch applied on top of a printed circuit board (PCB) which uses Rogers
RO4350B (LoPro) laminate with a $0.55$ mm thickness ($\epsilon_{r}=3.66$,
$\tan\delta=0.0037$). Each unit cell within a supercell is loaded with a
reactive impedance implemented by shorted or open coplanar lines according to
the simulation-based optimization described in [12]. A 3-bit quantization
resolution for the reactive loads is assumed sufficient, based on the analysis
in [12, 13].

The manufactured 3-bit $48\times 48$ square-shape array prototype is shown in
Fig. 1, and its scattering property was verified through a measurement in an
anechoic chamber in [12]. Now, we want to use this realistic reflector in our
theoretical models and ray tracing tool to verify our implementations and
communication models, as well as to evaluate the performance of this reflector
in a real environment. Therefore, more measurement in a realistic scenario is
needed. For convenience of our analysis, we reformulate the measured and
simulated scattering pattern results for the $48\times 48$-sized AR in Fig. 2,
as also seen in Fig. 7 in [12]. The measurement results are in good agreement
with the CST simulation results in the main beam directions. It should be
mentioned that the AR was originally only designed for $26$ GHz, but the
results indicate it also works well at $25$ and $27$ GHz, with the main
reflection direction shifted for $5\degree$. For example, the main reflection
direction occurs at $70\degree,65\degree,$ and $60\degree$ from $25$ GHz to
$27$ GHz, respectively.

(a) (b)

Figure 1: A manufactured $48\times 48$-sized anomalous reflector (a) front,
(b) back.  Figure 2: Normalized scattering pattern of the $48\times 48$-sized
AR at different frequencies and angles from CST simulation and measurement.

##  III Measurement System

In this section, we perform over-the-air measurements in an auditorium with
the manufactured $48\times 48$-sized AR ($152.6$ mm$\times 152.6$ mm) to
evaluate its beamforming performance in the communication aspect, i.e., the
large scale fading through the Tx-AR-Rx link. In addition, we combine four
identical pieces of the $48\times 48$-sized AR to form a large AR with the
size of $96\times 96$ ($305.3$ mm$\times 305.3$ mm).

We use copper tape to glue the edges to ensure continuous alignment of
elements with neighboring small ARs in the same plane, minimizing
misalignment. The combined $96\times 96$-sized AR is displayed in Fig. 3(c).
We then repeat the measurement with the big AR in the auditorium to
investigate its performance.

(a) (b) (c)

Figure 3: Measurement scenario and equipment. (a) The measurement scenario in
an auditorium (b) $48\times 48$-sized AR ($152.6$ mm$\times 152.6$ mm), (c)
$96\times 96$-sized AR ($305.3$ mm$\times 305.3$ mm).

###  III-A Measurement scenario and setups

The measurement was conducted in a $14\times 8\times 3$ (m3) sized auditorium
at the Nokia Bell Labs office in Espoo, Finland. As shown in Fig. 3, the Tx
and Rx antennas are the same horn antennas. We fix the positions of the AR and
the Tx antenna and move the Rx antenna to
$55\degree,60\degree,65\degree,70\degree,75\degree,80\degree,$ and $85\degree$
from the AR, respectively. The heights of the Tx, Rx antennas, and the AR
center are $1.5$ m. The distance between the Tx and the AR is $5.5$ m, and the
distance between the AR and the Rx antenna is always $7$ m. The estimated
Fraunhofer far-field distance of the $48\times 48$ and $96\times 96$-sized ARs
are $4.04$ m and $16.17$ m, respectively. Therefore, the $48\times 48$-sized
AR is in the far-field region of the Tx and the Rx antennas and the $96\times
96$-sized AR is in the radiating near-field region. There are line-of-sight
(LoS) paths between the Tx and the AR and between the AR and the Rx antenna,
while the direct link between the Tx and the Rx antennas is blocked by a wave
absorber. Throughout the measurement campaign, we avoided moving objects in
the room.

Figure 4: Measurement setup.

The measurement setup is shown in Fig. 4. A Tx horn antenna is connected to an
R&S vector signal generator (VSG) SMW200A and then connected to a control
computer. We generated a 5G NR PUSCH signal characterized by full resource
blocks and $50\%$ duty cycle within a channel bandwidth of $400$ MHz or $100$
MHz and a subcarrier spacing of $120$ kHz for $26$ GHz mmWave band by the VSG
and the Tx output power calibrated to $+6$ dBm. The Rx horn antenna is
connected to a low-noise-amplifier (LNA) Miteq JS4 with a noise figure of
$2.7$ dB; the LNA is then connected to an R&S FSW vector signal analyzer (VSA)
with a capture buffer length of $20.1$ ms. Vertically polarized Tx and Rx horn
antennas were configured with a gain of $18$ dB with a beamwidth of
$22\degree$. The VSA is connected to the VSG via an Ethernet cable and
reference clock for synchronizing. The parameters of the devices are listed in
Table I. In our measurement, we transmit 16 QAM orthogonal frequency-division
multiplexing (OFDM) modulated signals.

Next, the error vector magnitude (EVM) and received power levels were measured
through the Tx-AR-Rx link. EVM is a communications systems metric to indicate
the received signal quality throughout the measured channel bandwidth for
evaluating the conditions for demodulation of the received signal. The
received power defines the amount of power acquired by the user receiver in
assessing the performance. Since this kind of measurement takes a long time
with the used instrumentation, we measure only at three distinct frequencies,
$25,26,$ and $27$ GHz with the modulated signals. For reference, we use
continuous wave signal to sweep from $24.5$ GHz to $27.5$ GHz with a step of
$0.25$ GHz to have proper frequency responses and a quicker measurement setup.

TABLE I: MEASUREMENT PARAMETERS | Item | value
---|---|---
Transmit power | $P_{t}$ |  $6$ dBm
Cable loss at Tx side | $(-)L_{t}$ |  $-2.5$ dB
Tx antenna gain | $G_{t}$ |  $18$ dB
Rx antenna gain | $G_{r}$ |  $18$ dB
LNA gain + cable loss at Rx side | $G_{a}$ |  $19.9$ dB
Distance between the Tx and AR | $R_{1}$ |  $5.5$ m
Distance between the Rx and AR | $R_{2}$ |  $7$ m

###  III-B Path loss models

Different path loss models for ARs have been discussed in [13, 23]. For
periodic metasurfaces, the distribution of reflected fields over all angles
can be estimated based on the results of Section II in [24], also for
imperfect ARs that produce some parasitic scattering into undesired Floquet
modes. Propagation of signals due to reflections from some objects is
conventionally modeled in terms of the bi-static scattering cross-section that
can be determined based on the analytical model in [24]. For reflections into
the desired direction that model gives a very simple analytical formula, which
we use as the first method in this work for a theoretical analysis of our
manufactured AR. We denote the received power estimated by Eq. (38) in [24] as
$P_{1}$. Adding the measurement setup parameters, we obtain the received power
$P_{r}=P_{1}-L_{t}+G_{a}$ in dBm from method 1.

While using the theory of [24] it is possible to estimate the bi-static cross
section for any reflection direction, for integration of the RIS models into
ray-tracing simulators it is more convenient to use the model for the
scattering performance of AR in terms of the AR panel gains in the directions
of illumination and observation [23, 13], which we call method 2. This
approach is based on the Friis formula and gives the received power estimation

$P_{2}=\frac{P_{t}G_{t}G_{\text{rx}}G_{\text{tx}}G_{r}\lambda^{4}}{(4\pi)^{4}(R_{1}R_{2})^{2}}.$
(1)

Here, $G_{\text{rx}}$ is the gain of the AR in the direction from AR to Tx,
and $G_{\text{tx}}$ is the gain in the direction from AR to Rx. They are
obtained from the CST simulation of the AR, where we first calculate the
directivity of the AR (denoted as $D_{i}$) from the far-field pattern in CST
simulation, then we use the directivity value to approximate the gain of the
AR according to $G=e_{\text{cd}}D_{i}$ [25] and assume the panel efficiency
$e_{\text{cd}}=1$. This theoretical model combines analytical estimations of
the propagation loss with the radiation pattern results from EM simulations.
By using the simulated AR gain at different angles, we can obtain received
power results at the corresponding angles. Similarly, the received power from
method 2 is calculated as $P_{r}=P_{2}-L_{t}+G_{a}$ in dBm. Note that Eq. (1)
is the same as Eq. (6) in [26], but here we use it for the whole RIS panel,
and not for a single array element, as in [26]. Finally, we note that the
product of the two gains of the AR panel is related to its bi-static
scattering cross-section $\sigma$ as

$G_{\text{rx}}G_{\text{tx}}=\frac{4\pi\sigma}{\lambda^{2}},$ (2)

which is a function of both angles of incidence and observation. This equation
establishes the relation between the results given by the two used methods.

Figure 5: A front view of the 3D auditorium model, which is replicated with a
1:1 ratio from the actual sizes.

###  III-C Ray tracing model

A ray tracing simulation tool accurately evaluates the wireless communication
link performance in realistic scenarios. To reproduce the measurement scenario
for ray tracing simulations, we first create a 3D model of the auditorium and
objects inside with their actual dimensions in the software SketchUp, then
import the 3D model to the MATLAB ray tracer for simulations. The front view
of the 3D model is shown in Fig. 5. We model a horn antenna with the maximum
gain of $18$ dB as Tx and Rx antennas and import the radiation patterns of the
AR from CST simulations to the ray tracer.

Initially, CST simulations are conducted on a periodic (infinite) array of
supercells. Subsequently, finite-size array scattering patterns are derived
using the embedded element approximation in CST by applying an array factor
calculation on the “element pattern” of the simulated single supercell, as
stated in [12]. The $+65\degree$ anomalous reflection scattering pattern with
normal incidence is utilized when the AR functions as a Tx during the AR-Rx
link. Conversely, the $0\degree$ scattering pattern is employed with
$-65\degree$ incidence when the AR serves as an Rx in the Tx-AR link. The ray
tracer cannot perform a direct link Tx-AR-Rx due to the definition of the AR
as an antenna. Specifically, if the scattering pattern only reflects at a
$+65\degree$ angle with normal incidence, the AR will not receive any signal
from the Tx because the scattering pattern has a low value at a $0\degree$
angle. It is necessary to integrate two scattering patterns from CST into the
ray tracer to address the problem.

We place the Rx antenna at different positions from $55\degree$ to $85\degree$
and orient the main beam of the Rx antenna pattern to always face the AR. In
Fig. 6, we plot the LoS paths between the Tx and the AR, and between the AR
and the Rx in the MATLAB ray tracer. The locations and heights of the Tx, Rx,
and AR are the same as in the measurement. For ray tracing simulations with
the two ARs, we set the material type of the auditorium as “concrete” and set
the reflection paths as zero, or three for comparison. When there is no
reflection path, it will only simulate the LoS paths as shown in Fig. 6, and
the path loss calculation is based on the free space path loss model which is
the same as from method 2. The zero-reflection case is used to compare the
simulation results with the theoretical model, and to allow for a more
practical propagation environment, we set three reflection paths to compare
the results with measurement results.

Figure 6: The Tx, Rx, and AR positions and radiation patterns in the MATLAB
ray tracer, as well as the LoS paths between them.

##  IV Measurement and Simulation Results

In this section, we compare received power results through the AR-assisted
communication links between the theoretical models, ray tracing simulations,
and measurements. We analyze the AR’s angular response across various Rx
locations, evaluate frequency selectivity by varying frequencies, and assess
spatial scalability performance through AR prototypes.

###  IV-A Reference measurement of LoS links

To obtain accurate measurement results and have a fair comparison with
simulation results, we first conduct a reference measurement with only LoS
links between the Tx and the Rx antenna. Next, we compare the measured results
with theory to find out how much power difference we obtain between them.
During the measurement, we fix the Tx antenna position and move the Rx antenna
to $7$ different locations from $55\degree$ to $85\degree$ with a $5\degree$
step because the simulated half power beam width (HPBW) of the $48\times
48$-sized AR is $9\degree$, a $5\degree$ step is enough to catch the peak
power. However, the HPBW of the $96\times 96$-sized AR is $5\degree$.
Therefore, we add an additional angle of $62.5\degree$ for the $96\times
96$-sized AR measurement. At each Rx location, we orient the Tx and Rx
antennas to let their main beam directions toward each other, and we measure
the received power at the Rx antenna through the LoS path from the Tx antenna.
We conduct the measurement first with 16 QAM modulated waves at $25,26,$ and
$27$ GHz, then we use continuous waves to sweep from $24.5$ GHz to $27.5$ GHz
with $0.25$ GHz step.

We denote the measured received power value as $P_{m}$. Then we calculate the
free space path loss between the Tx and the Rx using the Friis formula
$P_{\text{FSPL}}=\left(\frac{\lambda}{4\pi R_{3}}\right)^{2}$ in $W$, where
$R_{3}$ denotes the distances between the Tx and the Rx antenna. The final
calculated received power from theory is
$P_{\text{theory}}=P_{t}+G_{t}+G_{r}+P_{\text{FSPL}}-L_{t}+G_{a}$ in dBm. The
power difference between the theoretical and measurement results is
$P_{\text{diff}}=P_{\text{theory}}-P_{m}$ in dB. For modulated-wave
measurement, we only use the results at $25,26,$ and $27$ GHz, while for
continuous-wave measurement, we use values at all frequencies to calculate the
power difference.

TABLE II: POWER DIFFERENCES BETWEEN THEORY AND MEASUREMENTS $P_{\text{diff}}$ (dB) | $55\degree$ | $60\degree$ | $65\degree$ | $70\degree$ | $75\degree$ | $80\degree$ | $85\degree$
---|---|---|---|---|---|---|---
$25$ GHz | $0.87$ | $0.78$ | $0.77$ | $0.25$ | $0.14$ | $0.28$ | $1.52$
$26$ GHz | $1.46$ | $1.55$ | $1.08$ | $0.68$ | $0.26$ | $1.05$ | $0.28$
$27$ GHz | $1.04$ | $2.13$ | $1.56$ | $1.33$ | $1.79$ | $-0.04$ | $1.85$

In Table II, we list an example of the $P_{\text{diff}}$ with modulated
measurement results at the three frequencies. These small differences may come
from the losses in the measurement system that we could not count in our
model. The losses due to measurement devices may have a little deviation
between the measurement and simulation. For instance, the antenna gain
variability at different frequencies is about $\pm 0.5$ dB. The LNA gain and
the Rx cable loss together have about $1.6$ dB variation, and the Tx cable
loss has around $0.065$ dB variability in the frequency range. When we take
the $P_{\text{diff}}$ into account and use it to correct the calculated and
simulated results with the AR, we can eliminate the influence of these factors
and obtain more accurate results between the simulation model and the
measurement.

(a) (b)

Figure 7: Measured results of the $48\times 48$-sized AR with 16 QAM modulated
waves. (a) received power vs. angle, (b) EVM vs. angle.

###  IV-B Results of the $48\times 48$-sized AR

For the $48\times 48$-sized AR, we measured the received power at $7$ Rx
locations from $55\degree$ to $85\degree$ with a $5\degree$ step. Same as the
previous LoS measurement, we perform measurements with 16 QAM modulated OFDM
waves with $400$ MHz bandwidth at the frequencies $25,26,$ and $27$ GHz, and
obtain the results of received power and EVM that are displayed in Fig. 7(a)
and Fig. 7(b), respectively. Fig. 7(b) only shows the EVM values that passed
the frame-averaged EVM test with the 5G NR signal (EVM less than 12.5% for
16QAM). We observe that from $25,26,$ to $27$ GHz, the maximum received power
increases while the minimum EVM decreases. The maximum received power and
minimum EVM for $25,26,$ and $27$ GHz occur at $70\degree,65\degree$, and
$60\degree$, respectively. The angles of the received power peaks at the three
frequencies are consistent with the scattering pattern shown in Fig. 2.

(a) (b) (c)

Figure 8: Results with the $48\times 48$-sized AR from corrected calculation
and simulation, and from measurements using the modulated waves, (a) at $25$
GHz, (b) at $26$ GHz, (c) at $27$ GHz.

(a) (b) (c)

Figure 9: Results with the $48\times 48$-sized AR from corrected calculation
and simulation, and from measurements using the continuous waves, (a) at
$60\degree$, (b) at $65\degree$, (c) at $70\degree$.

The corrected calculation and simulation results for the AR-assisted links are
obtained by $P_{\text{correct}}=P_{r}-P_{\text{diff}}$ with $P_{r}$ being the
original power calculated from method 1 and method 2. The comparison of the
corrected results and the measurement results are plotted in Fig. 8. From
these three figures, we observe that the results from method 2 and from ray
tracing with zero reflection are consistent, which validates our AR
implementation method in the ray tracer. Ray tracing with zero reflection and
three reflections also does not show much difference in the received power. At
$25$ GHz, the measurement and simulation results have very little difference
at most angles, except at $60\degree$. This is highly related to the simulated
and measured scattering pattern of the AR, as we see similar circumstances in
Fig. 2. At $26$ GHz, the measurement results at $60\degree,65\degree,$ and
$70\degree$ are almost the same as the ray tracing simulation results with
three reflections. In addition, at $65\degree$ (the designed reflection
angle), the theoretical results for perfect AR given by method 1 are very
close to those given by method 2 and to the ray tracing results with zero
reflection, which indicates that the 3-bit quantized lossy AR does not have
much power loss at the designed reflection angle compared to a perfectly
working lossless AR. At $27$ GHz, the simulation results and measurement
results are almost the same at $55\degree,60\degree$, and $65\degree$.

To evaluate the results at more frequencies, we do the same correction
procedure for the continuous-wave measurement results. Here we use the
continuous-wave measured results $P_{m}$ and the theoretical results both from
$24.5$ GHz to $27.5$ GHz and calculate the $P_{\text{diff}}$ between them, the
corrected results $P_{\text{correct}}$ are from the updated $P_{\text{diff}}$.
Since the nominal main beam direction of the AR is between $60\degree$ and
$70\degree$ due to the frequency steering phenomenon, we observe the received
power results at all $13$ frequencies and at $60\degree$, $65\degree$, and
$70\degree$ which are shown in Figs. 9(a)-9(c). Generally, we achieve good
agreement between the simulations and measurement results. There are big
differences only at a few frequency points, which is related to the AR
scattering pattern difference between simulations and measurements. Since
$65\degree$ is the designed reflection angle, we see that the agreement at
this angle is better than for $60\degree$ and $70\degree$. In addition, we
find that the “frequency steering” phenomenon limits the use of this AR for
signal bandwidths greater than approximately $1$ GHz. But for any individual
center frequency within the given almost $3$ GHz range, we can operate.

These observations confirm that our ray tracing model with the realistic AR
agrees well with measurements in such an indoor scenario, even though the 3D
model in the ray tracer does not fully replicate the auditorium since we can
only set one material type for all objects in the MATLAB version up to 2024a,
whereas the materials of different objects in the rooms are different in
reality. Next, we can use this method to predict the communication performance
of other ARs, especially those that are difficult to measure. For example, we
could not measure the far-field scattering pattern of a $96\times 96$-sized AR
due to the limitation of the measurement setup, but we can use our models to
predict the far-field performance of this AR.

(a) (b)

Figure 10: Measured results of the $96\times 96$-sized AR with 16 QAM
modulated waves. (a) received power vs. angle, (b) EVM vs. angle.

###  IV-C Results for the $96\times 96$-sized AR

In this section, we analyze the results for the $96\times 96$-sized AR which
is a tiled combination of four pieces of the $48\times 48$-sized AR. Since the
size of the AR becomes four times larger than that of the small AR, the far-
field distance of the big AR is not fulfilled anymore in our measurement. From
the general properties of reflecting panels and [12] we know that the main
scattering beam of a larger AR is narrower than a smaller AR, and the
scattering pattern of a larger AR is more sensitive in terms of the angles.
Therefore, we repeat all the previous measurements for the big AR at the
previous $7$ angles and additionally at $62.5\degree$, i.e., at $8$ angles in
total for this AR measurement and simulations. In addition to 16 QAM modulated
waves with $400$ MHz bandwidth, we also did measurements using 64 QAM
modulated waves with $100$ MHz bandwidth. The measurement procedures are the
same as those for the $48\times 48$-sized AR, i.e., with modulated waves and
continuous waves, correcting the calculation and simulation results with the
power difference $P_{\text{diff}}$.

(a) (b) (c)

Figure 11: Results with the $96\times 96$-sized AR from corrected calculation
and simulation, and from measurements using the modulated waves, (a) at $25$
GHz, (b) at $26$ GHz, (c) at $27$ GHz.

(a) (b) (c) (d)

Figure 12: Results with the $96\times 96$-sized AR from corrected calculation
and simulation, and from measurements using continuous waves, (a) at
$60\degree$, (b) at $62.5\degree$, (c) at $65\degree$, (d) at $70\degree$.

The received power and EVM results from the big AR are shown in Fig. 10(a) and
Fig. 10(b), respectively. We only plot the EVM results that passed the EVM
tests. These results indicate that the maximum power for $25,26,$ and $27$ GHz
appear at $65\degree$, $62.5\degree$, and $60\degree$, respectively. The EVM
results are consistent, i.e., the minimum EVM results for the big AR also
appear at the same angles for the three frequencies. The main beam direction
moves to a smaller angle when increasing the frequency, which is the same as
for the small AR. The only difference is that the peak power from the small AR
appears at $70\degree$, $65\degree$, and $60\degree$ for $25,26,$ and $27$
GHz, respectively. It also shows that the big AR works well with both 16 QAM
and 64 QAM modulation schemes. When changing from 16 QAM to 64 QAM, it can
achieve higher data rates and throughput. From the comparison of the results
between the two modulation schemes, the 64 QAM modulation results in lower EVM
and received power than using 16 QAM, but the differences are not very
significant. Therefore, we only compare the simulation and calculation results
with 16 QAM measurements in the following process.

The received power results from the corrected calculation and simulations, as
well as from measurements using modulated waves are shown in Fig. 11. Fig.
11(a)-11(c) display the results at $25,26,$ and $27$ GHz, respectively. These
three figures show different beam shapes compared to the $48\times 48$-sized
AR. The measured power peak appears on $62.5\degree$, while the simulated peak
is still on $65\degree$ at $26$ GHz. However, an interesting finding is that
at some angles, the simulation and measurement results match very well, such
as the power in $62.5\degree$ at $25$ GHz and $26$ GHz. Since the beam width
of the large AR is much narrower than that of the small AR, the simulation and
measurement results have generally larger differences in different angles.
These big differences may be due to the following reasons: 1) the accuracy of
measurement is reduced due to the more sensitive angular response of the large
AR, and the angle samples for measurement are too sparse for this AR; 2) the
simulated AR gain used in (1) and the radiation pattern used in ray tracing
simulations are from a $96\times 96$ complete panel without discontinuity,
while the measured AR is a manually combined one. Any misalignment of the unit
cells from the four small ARs will reduce the total scattering efficiency of
the big AR, and the copper glue at the back of the ARs may also influence
reflections and the frequency selectivity of the AR; 3) the analytical
expression (1) and the ray tracing simulations give far-field results for the
big AR while the measured AR is in the near-field, thus the comparison between
them does not match very well.

Similar to the $48\times 48$-sized AR, we compare the results corrected by
continuous wave measurement. The results at the $13$ frequencies are presented
in Fig. 12, which again show significant differences between the simulation
and measurements. These results reveal that the big AR still works at $26$
GHz, but the main beam has a slight shift, and the working bandwidth for this
AR is approximately $1$ GHz. It would be interesting to also compare the
performance of this combined AR with a manufactured $96\times 96$-sized
complete AR and measure the scattering pattern of the big AR when the setup
conditions allow. This will help us find out whether the differences come from
the manual combining of the four ARs or from the radiation pattern differences
between simulations and measurements. This work is postponed to the future as
a next step.

##  V Conclusion

In this work, we introduce and study a manufactured AR that is designed to
reflect normally incident plane waves toward $65\degree$ at $26$ GHz. The
scattering pattern of this AR from CST simulation and measurement is
consistent in the main reflection directions. We then conduct over-the-air
measurements in an auditorium with the $48\times 48$-sized AR, to measure the
received power at an Rx antenna through the Tx-AR-Rx link. In addition, we
calculate the received power from the AR-assisted link theoretically. These
results are compared with the ray tracing simulations, where we implement an
AR model into the MATLAB ray tracer and simulate the actual complex
propagation scenario as in the measurement. The calculated and simulated
results show good agreement with the measurement results over different
frequencies and different angles.

In addition, we combine four pieces of the $48\times 48$-sized AR to form a
$96\times 96$-sized AR and measure the received power at the Rx through
anomalous reflection in this combined AR. Results show that the peak received
power at observed angles shifts to $62.5\degree$ at $26$ GHz with the combined
AR from measurement. The frequency response of this AR is not as good as that
of the small AR, but the simulation and measurement results show good
agreement at several angles and frequencies. It is proved that our method of
implementing an AR in the MATLAB ray tracer is correct and that it can be
extended to any kind of AR as long as the scattering pattern of the AR is
available. Our designed AR has a wideband property and can work well within a
$2.75$ GHz band. The designed AR also has good spatial scalability. When
combining multiple smaller pieces of ARs to form a bigger AR, we observe that
the big AR can achieve good performance with little deviation from the design
goals. This property offers a convenient alternative to completely redesigning
a new AR with a bigger size.

## References

  * [1] V. S. Asadchy, M. Albooyeh, S. N. Tcvetkova, A. Díaz-Rubio, Y. Ra’di, and S. Tretyakov, “Perfect control of reflection and refraction using spatially dispersive metasurfaces,” _Phys. Rev. B_ , vol. 94, no. 7, p. 075142, 2016.
  * [2] A. Díaz-Rubio, V. S. Asadchy, A. Elsakka, and S. A. Tretyakov, “From the generalized reflection law to the realization of perfect anomalous reflectors,” _Science Advances_ , vol. 3, no. 8, p. e1602714, 2017. [Online]. Available: https://www.science.org/doi/abs/10.1126/sciadv.1602714
  * [3] S. K. R. Vuyyuru, R. Valkonen, D.-H. Kwon, and S. A. Tretyakov, “Efficient anomalous reflector design using array antenna scattering synthesis,” _IEEE Antennas Wireless Propag. Lett._ , vol. 22, no. 7, pp. 1711–1715, 2023.
  * [4] M. Movahediqomi, G. Ptitcyn, and S. Tretyakov, “Comparison between different designs and realizations of anomalous reflectors for extreme deflections,” _IEEE Transactions on Antennas and Propagation_ , pp. 1–1, 2023.
  * [5] A. Díaz-Rubio, S. Tcvetkova, and S. Tretyakov, “Analytical models of reflection and scattering by finite-size anomalously reflecting metasurfaces,” in _2021 15th European Conference on Antennas and Propagation (EuCAP)_ , 2021, pp. 1–3.
  * [6] S. I. Raptis and T. V. Yioultsis, “Synthesis of polarization-independent perfect anomalous reflectors via modulated metasurfaces and an analytical design model,” _IEEE Transactions on Microwave Theory and Techniques_ , vol. 71, no. 11, pp. 4703–4712, 2023.
  * [7] A. M. H. Wong and G. V. Eleftheriades, “Perfect anomalous reflection with a bipartite Huygens’ metasurface,” _Phys. Rev. X_ , vol. 8, 2018, Art. No. 011036.
  * [8] D.-H. Kwon, “Lossless scalar metasurfaces for anomalous reflection based on efficient surface field optimization,” _IEEE Antennas Wireless Propag. Lett._ , vol. 17, no. 7, pp. 1149–1152, Jul. 2018.
  * [9] A. Díaz-Rubio and S. A. Tretyakov, “Macroscopic modeling of anomalously reflecting metasurfaces: Angular response and far-field scattering,” _IEEE Transactions on Antennas and Propagation_ , vol. 69, no. 10, pp. 6560–6571, 2021.
  * [10] S. K. R. Vuyyuru, R. Valkonen, D.-H. Kwon, and S. A. Tretyakov, “Array scattering synthesis for anomalous deflection using passive aperiodic loadings,” in _2024 18th European Conference on Antennas and Propagation (EuCAP)_ , 2024, pp. 1–5.
  * [11] V. S. Asadchy, A. Díaz-Rubio, S. N. Tcvetkova, D.-H. Kwon, A. Elsakka, M. Albooyeh, and S. A. Tretyakov, “Flat engineered multichannel reflectors,” _Phys. Rev. X_ , vol. 7, 2017, Art. No. 031046.
  * [12] S. K. R. Vuyyuru, L. Hao, M. Rupp, S. A. Tretyakov, and R. Valkonen, “Modeling RIS from Electromagnetic Principles to Communication Systems–Part I: Synthesis and Characterization of a Scalable Anomalous Reflector,” _arXiv preprint arXiv:2403.12790_ , 2024.
  * [13] L. Hao, S. K. R. Vuyyuru, S. A. Tretyakov, A. Salihu, M. Rupp, and R. Valkonen, “Modeling RIS from Electromagnetic Principles to Communication Systems–Part II: System-Level Simulation, Ray Tracing, and Measurement,” _arXiv preprint arXiv:2403.13210_ , 2024.
  * [14] J. Huang and J. A. Encinar, _Reflectarray Antennas_. Hoboken, NJ, USA: Wiley, 2008.
  * [15] N. Yu, P. Genevet, M. A. Kats, F. Aieta, J.-P. Tetienne, F. Capasso, and Z. Gaburro, “Light propagation with phase discontinuities: generalized laws of reflection and refraction,” _Science_ , vol. 334, pp. 333–337, Oct. 2011.
  * [16] Y. Jiang, F. Gao, M. Jian, S. Zhang, and W. Zhang, “Reconfigurable intelligent surface for near field communications: Beamforming and sensing,” _IEEE Transactions on Wireless Communications_ , vol. 22, no. 5, pp. 3447–3459, 2023.
  * [17] M. Jian, G. C. Alexandropoulos, E. Basar, C. Huang, R. Liu, Y. Liu, and C. Yuen, “Reconfigurable intelligent surfaces for wireless communications: Overview of hardware designs, channel models, and estimation techniques,” _Intelligent and Converged Networks_ , vol. 3, no. 1, pp. 1–32, 2022.
  * [18] J. Huang, C.-X. Wang, Y. Sun, R. Feng, J. Huang, B. Guo, Z. Zhong, and T. J. Cui, “Reconfigurable intelligent surfaces: Channel characterization and modeling,” _Proceedings of the IEEE_ , vol. 110, no. 9, pp. 1290–1311, 2022.
  * [19] W. Tang, X. Chen, M. Z. Chen, J. Y. Dai, Y. Han, M. D. Renzo, S. Jin, Q. Cheng, and T. J. Cui, “Path loss modeling and measurements for reconfigurable intelligent surfaces in the millimeter-wave frequency band,” _IEEE Transactions on Communications_ , vol. 70, no. 9, pp. 6259–6276, 2022.
  * [20] M. Najafi, V. Jamali, R. Schober, and H. V. Poor, “Physics-based modeling and scalable optimization of large intelligent reflecting surfaces,” _IEEE Trans. Commun._ , vol. 69, no. 4, pp. 2673–2691, 2021.
  * [21] O. Özdogan, E. Björnson, and E. G. Larsson, “Intelligent reflecting surfaces: Physics, propagation, and pathloss modeling,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 5, pp. 581–585, Dec. 2020.
  * [22] S. K. R. Vuyyuru, R. Valkonen, S. A. Tretyakov, and D.-H. Kwon, “Efficient synthesis of passively loaded finite arrays for tunable anomalous reflection,” _arXiv preprint arXiv:2312.04441_ , 2023.
  * [23] L. Hao, F. S. Cuesta, and S. A. Tretyakov, “Comparison of simplistic system-level ris models and diffraction-theory solutions,” in _2024 18th European Conference on Antennas and Propagation (EuCAP)_ , 2024, pp. 1–5.
  * [24] S. Kosulnikov, F. S. Cuesta, X. Wang, and S. A. Tretyakov, “Simple link-budget estimation formulas for channels including anomalous reflectors,” _IEEE Transactions on Antennas and Propagation_ , vol. 71, no. 6, pp. 5276–5288, 2023. Corrections: vol. 72, no. 6, p. 5432, 2024.
  * [25] C. A. Balanis, _Antenna Theory: Analysis and Design_. John wiley & sons, 2015.
  * [26] S. W. Ellingson, “Path loss in reconfigurable intelligent surface-enabled channels,” in _2021 IEEE 32nd Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC)_ , 2021, pp. 829–835.

  *[AR]: anomalous reflector
  *[EM]: electromagnetic
  *[LoS]: line-of-sight
  *[OFDM]: orthogonal frequency-division multiplexing
  *[EVM]: error vector magnitude


# DK-SLAM: Monocular Visual SLAM with Deep Keypoints Adaptive Learning,
Tracking and Loop-Closing

Hao Qu, Lilian Zhang, Jun Mao, Junbo Tie, Xiaofeng He, Xiaoping Hu, Yifei Shi,
Changhao Chen The authors are with the College of Intelligence Science and
Technology, National University of Defense Technology, Changsha, 410073,
China*Corresponding author: Changhao Chen (changhao.chen66@outlook.com).Hao Qu
and Lilian Zhang contribute equally to this work.This work was supported by
National Natural Science Foundation of China (NFSC) under the Grant Number of
62103427, 62073331, 62103430, and Major Project of Natural Science Foundation
of Hunan Province (No.2021JC0004). Changhao Chen is funded by the Young Elite
Scientist Sponsorship Program by CAST (No. YESS20220181)

###### Abstract

Unreliable feature extraction and matching in handcrafted features undermine
the performance of visual SLAM in complex real-world scenarios. While learned
local features, leveraging CNNs, demonstrate proficiency in capturing high-
level information and excel in matching benchmarks, they encounter challenges
in continuous motion scenes, resulting in poor generalization and impacting
loop detection accuracy. To address these issues, we present DK-SLAM, a
monocular visual SLAM system with adaptive deep local features. MAML optimizes
the training of these features, and we introduce a coarse-to-fine feature
tracking approach. Initially, a direct method approximates the relative pose
between consecutive frames, followed by a feature matching method for refined
pose estimation. To counter cumulative positioning errors, a novel online
learning binary feature-based online loop closure module identifies loop nodes
within a sequence. Experimental results underscore DK-SLAM’s efficacy,
outperforms representative SLAM solutions, such as ORB-SLAM3 on publicly
available datasets.

###### Index Terms:

Monocular SLAM, Deep Learning, Feature Extraction and Matching, Loop Closing.

##  I Introduction

Visual localization and mapping constitute crucial components within
intelligent mobile agents for motion tracking and environment understanding,
with diverse applications ranging from autonomous vehicles, unmanned aerial
vehicles (UAVs), and robots to immersive technologies such as Virtual Reality
(VR) and Augmented Reality (AR). The imperative nature of achieving robust
localization for unmanned platforms across various environmental conditions
has garnered escalating attention within the research community.

A Visual Simultaneous Localization and Mapping (SLAM) system comprises front-
end perception and back-end optimization. The front-end uses sensors for
environmental perception, employing keypoint features in stages like
extraction, matching, outlier removal, and pose estimation. The back-end
includes loop detection and global map optimization, refining the system state
with geometric models. Visual SLAM methods are categorized as keypoint-based
and direct methods. Direct methods optimize pixel-level photometric loss but
can be unstable due to lighting variations [1, 2, 3]. Keypoint-based methods,
like ORB-SLAM2 and ORB-SLAM3 [4, 5], enhance stability by using detectors and
descriptors, although handcrafted descriptors are susceptible to lighting and
texture variations. Addressing these challenges is crucial for improving the
reliability of visual localization and mapping systems in diverse
applications.

In recent years, deep learning has shown impressive performance in image
processing, particularly in image classification and object detection. The
application of deep learning to VO/SLAM methods has gained attention, with two
predominant approaches: end-to-end and learned features based methods. End-to-
end methods utilize Deep Neural Networks (DNNs) to directly convert images
into pose estimates [6]. However, they exhibit low interpretability and
suboptimal generalization. In contrast, learned-feature-based SLAM replaces
local features, allowing deep learning to focus on feature perception [7, 8,
9, 10]. This approach combines the strengths of deep learning and geometric
models but may still face challenges in generalization. Additionally, it may
lack crucial low-level keypoint information, impacting feature matching
performance. The training complexity of learned-feature-based Bag-of-Words
(BoW) adds further challenges, often requiring offline training and
exacerbating generalization issues.

To address the aforementioned challenges, we introduce DK-SLAM, a novel Deep
Keypoints based SLAM system incorporating adaptive learning, coarse-to-fine
feature tracking, and online loop-closing capabilities. This monocular visual
SLAM framework utilizes a modified MAML-based training strategy for robust
deep feature extraction. It enhances feature matching precision by
incorporating patch pixels and constructs a photometric loss using adjacent
frame patches to optimize coarse relative pose estimates. An online BoW model
with binary deep features adapts dynamically to new environments, mitigating
generalization challenges in deep learning, and improves adaptability without
requiring additional training. Our proposed DK-SLAM outperforms the
representative baselines, e.g. ORB-SLAM3, LDSO and LIFT-SLAM, on the public
KITTI and EuRoC datasets.

Our contributions can be summarized as follows:

  * •

We propose DK-SLAM, a monocular SLAM system with adaptive deep keypoints
learning. Our deep feature extractor, trained using a MAML strategy, enhances
adaptability to diverse scenes.

  * •

Our method employs a coarse-to-fine feature matching strategy, estimating
relative poses through patch photometric loss optimization and refining them
based on the 2D-3D relationship for improved accuracy.

  * •

We introduce an deep keypoints online learning-based Bag-of-Words (BoW) models
that overcomes poor generalization, ensuring correct loop detection in the
sequence.

##  II Related Works

###  II-A Deep Feature Extractor for Visual SLAM

Mainstream handcrafted extractors, including ORB [11], SIFT [12], and Shi-
Thomas [13], are employed in visual SLAMs like ORB-SLAM2 [4] and VINS-Mono
[14]. However, these methods relying on gradient information face challenges
in dynamic lighting and low-texture environments. To address these issues,
researchers explored deep feature extractors. Superpoint [15] employs a shared
encoder for high-level information and distinct decoders for keypoints and
descriptors, using synthesized labels for self-supervised training. D2-Net
[16] eliminates the need for a separate keypoint decoder, ensuring robustness.
To enhance repeatability detection, [17] combines multi-scale gradients with
depth feature maps. Additionally, L2-Net [18] introduces a local matching loss
based on the L2 distance, improving feature matching performance. Experimental
results demonstrate the effective feature matching performance of the learned
local features on public datasets. Despite their success, the mentioned deep
extractors exhibit inherent flaws in the face of diverse scenes, leading to
potential performance degradation.

###  II-B Hybrid methods for Visual SLAM

Recent research explores hybrid SLAM methods to combine deep learning and
multi-view geometry advantages. GCNv2-SLAM integrates GCN [19] into SLAM,
constructing a complete system with offline BoW training. In [20], knowledge
distillation from HF-Net [21] enhances feature detection in a compact model.
[22] uses a self-supervised thermal optical flow tracking network, while [10]
builds on Superpoint [15] with edge and multi-scale information. Our proposed
SLAM methodology shares similarities with [8]. In [8], a LIFT [23] extractor
serves as the SLAM front-end, implementing an adaptive feature matching
strategy tailored to different datasets. In contrast, our SLAM adopts a meta-
learning based training strategy, eliminating the need for separate
hyperparameter designs for distinct datasets.

###  II-C Loop Closure Module

Loop closure is a critical module of visual SLAM, responsible for detecting
loop nodes and refining initial pose estimates. The BoW model, such as DBow2,
DBow3, and FBow [24], relies on the binary descriptor BRIEF [25]. DBow
utilizes a k-d tree structure based on Hamming distance for descriptor
storage. However, conventional BoW methods require extensive offline training
data, prompting the introduction of iBow [26]. iBow achieves online word bag
training using testing datasets, demonstrating robust transfer performance
across various scenarios and addressing adaptability concerns. Another closed-
loop detection method, akin to NetVLAD [27] in [20, 9], is highly reliant on
the training set. Substantial differences between test and training sets may
result in performance degradation.

##  III Deep Keypoint based Monocular SLAM

An overview of our proposed DK-SLAM is depicted in Figure 2. We employ an
adaptive local feature extractor, based on Superpoint, for extracting
keypoints across multi-scale image pyramids. To balance the number of
keypoints on varying scale images, we implement an averaging distribution
strategy. For enhanced generalization, the local feature extractor undergoes
MAML during training. Subsequently, we optimize memory consumption by
converting floating descriptors into binary hash codes. Lastly, an online BoW
module is introduced to establish a loop closure mechanism, facilitating
closed-loop detection and global map optimization.

Figure 1: An overview of our proposed DK-SLAM framework with adaptive
keypoints, two-stage tracking and loop closure.

###  III-A Adaptive Local Feature Learning

####  III-A1 Feature Extractor Network

Inspired by Superpoint, our deep local feature extractor also utilizes VGG16
[28] as the backbone. Different from Superpoint, we introduce BatchNorm layers
after each Convolutional Neural Network (CNN) to enhance training convergence.
Post the backbone, two heads emerge, each comprising multi-layer CNNs. The
first, the detector head, generates a probability mask matching the input
image size. We posit that positions with a probability surpassing a set
threshold on the mask denote keypoint positions. The second head, the
descriptor head, predicts feature descriptors for keypoints, normalized to a
dimension of 256 using the L2 norm.

####  III-A2 Self-supervised Adaptive Feature Learning

Our network undergoes self-supervised training, employing a multi-step
process. Initially, we obtain the homography-warped image from current
training image. To train keypoint detector, we leverage the pre-trained
MagicPoint network [15] to acquire keypoint pseudo labels for both the
original and warped images. In both images, keypoint predictions and pseudo
labels contribute to the formation of detector loss, denoted as ${L_{p}}$ and
${L_{wp}}$, with CrossEntropy serving as the metric. Moving on to descriptor
training, we utilize the sparse descriptor loss ${L_{d}}$ proposed in [29].
This involves obtaining both the warped image and pixel-wise correspondences.
Descriptor loss functions are then constructed solely for pseudo-labeled
keypoints, employing triplet loss. The sparse descriptor loss is composed of
two components: homography-matched descriptor loss ($L_{dm}$) and non-matched
descriptor loss ($L_{dn}$). The homography-matched descriptor loss,
represented by Equation 1, involves a subscript $i$ ranging between
$(0,1,...,N)$, where $(\mathbf{p}_{i},\hat{\mathbf{p}_{i}})$ denotes the $N$
matched keypoint position pairs.

$L_{dm}(\mathbf{p}_{i})={\left\|\mathbf{d}_{I}(\mathbf{p}_{i})-\mathbf{d}_{wI}(\hat{\mathbf{p_{i}}})\right\|}^{2}$
(1)

As depicted in Equation 2, we use $M$ pairs of descriptors from the vicinity
of matched descriptors to formulate a non-matched descriptor loss. The
subscript $j$ ranges from 0 to $M$, where
$(\mathbf{p}_{i},\mathbf{p^{\prime}}_{j})$ represents unmatched keypoint
position pairs.

$L_{dn}(\mathbf{p}_{i})=\frac{1}{M}\sum_{j=1}^{M}(\left\|\mathbf{d}_{I}\mathbf{(p_{i})}-\mathbf{d}_{wI}(\mathbf{p^{\prime}}_{j})\right\|^{2})$
(2)

The descriptor loss ${L_{d}}$, illustrated in Equation 3 with $z$ representing
the boundary margin, is expressed as:

$L_{d}=\frac{1}{N}\sum_{i=1}^{N}\text{max}(0,z+L_{dm}-L_{dn})$ (3)

This descriptor loss ${L_{d}}$ aims to minimize the feature distance between
matching descriptors while maximizing the feature distance between non-
matching descriptors. The overall training loss is the sum of the
aforementioned detector loss and descriptor loss. Here, $\lambda$ acts as a
parameter to balance these different losses.

$L_{\text{all}}=(L_{p}+L_{wp})+\lambda{L_{d}}$ (4)

####  III-A3 MAML for Feature Extractor Training

Learning-based local feature extractors face poor generalization due to
variances in visual features across scenes. Significant feature disparity
between training and unseen datasets can lead to catastrophic forgetting. To
address this, we leverage insights from Model-Agnostic Meta-Learning (MAML),
adapting network parameters through meta-training for improved generalization
and rapid adaptation to new data while preventing catastrophic forgetting. The
MAML-based meta-training consists of inner loop training and outer loop
training. The training set is partitioned into a support set $\mathbf{D}_{s}$
and a query set $\mathbf{D}_{q}$, where $D_{s}$ is involved in inner loop
training, and $\mathbf{D}_{q}$ participates in outer loop training.

In the MAML training strategy, the original network parameter
$\mathbf{\theta}_{a}$ is initially duplicated to $\mathbf{\theta}_{b}$.
Subsequently, both parameters undergo training in both the inner and outer
loops. Within the inner loop, the support dataset $\mathbf{D}_{s}$ is utilized
for iterative updates to $\mathbf{\theta}_{b}$. This involves dividing a batch
of support dataset into $n$ distinct tasks $\tau$, performing $m$ parameter
updates on each task. Consequently, within one inner loop, the parameters are
updated $n*m$ times, resulting in the updated parameter
$\mathbf{\theta}^{n*m}_{b}$. After completing one iteration of inner loop
training, a complete batch of query set data is used to train the network.
Although the gradient $\mathbf{\bigtriangledown_{\theta^{n*m}_{b}}}$ is
calculated for $\mathbf{\theta}^{n*m}_{b}$ in the query set data, it is not
used to update $\mathbf{\theta}^{n*m}_{b}$; instead, the original
$\mathbf{\theta}_{a}$ is updated.

####  III-A4 Keypoint Distribution Strategy

Furthermore, we incorporate a keypoint distribution strategy during the
feature extraction stage. We uniformly distribute the entire set of keypoints
across a multi-scale image pyramid. Within each layer of the image pyramid, a
predefined number of keypoints is distributed evenly within the grid of
different regions on the image. This distribution strategy ensures a balanced
dispersion of keypoints across all scales and corners within an image,
preventing concentration in specific areas.

Figure 2: A diagram of our proposed Coarse-to-Fine Tracking.

###  III-B Coarse-to-Fine Keypoint Matching and Tracking

Keypoint matching accuracy is crucial for visual SLAM performance. Traditional
methods like ORB-SLAM use a motion tracking module based on adjacent frames,
assuming a uniform motion model. However, this assumption often fails, leading
to difficulties in finding correct matching correspondences and complicating
the optimization process.

####  III-B1 Semi-direct Coarse Keypoint Tracking

Taking inspiration from the semi-direct method [3], we introduce a coarse
tracking method based on photometric constraints. We assume that the light
intensity and texture around matching keypoints are similar in adjacent
frames. Map points from the last frame are projected onto the current frame
using the relative pose. If the relative pose is accurate, the photometric
loss of the projected keypoint’s patch will be minimal. As depicted in
equations 5 and 6, the pixels of the keypoint’s patch, the map points in the
last frame coordinate $\mathbf{p}^{l}_{i}$, and the relative pose of adjacent
frames $\mathbf{\xi}^{\text{coarse}}_{lc}$ collectively form the photometric
loss $L_{p}$. Given the known map points and patch pixel values of the last
frame, we adjust the relative pose $\mathbf{\xi}^{\text{coarse}}_{lc}$ to
minimize the photometric loss $L_{p}$.

$\mathbf{\xi}^{\text{coarse}}_{lc}=\arg\min_{\mathbf{\xi}^{\text{coarse}}_{lc}}\frac{1}{2}\sum_{i\in\overline{\mathcal{\chi}}}\left\|{L_{p}}\left({\mathbf{\xi}}^{\text{coarse}}_{lc},\mathbf{p}^{l}_{i}\right)\right\|^{2}$
(5)

${L_{p}}\left(\mathbf{\xi}^{\text{coarse}}_{lc},\mathbf{p}^{l}_{i}\right)=\mathbf{I}_{c}\left(\pi\left({\mathbf{\xi}}^{\text{coarse}}_{lc}\cdot\mathbf{p}^{l}_{i}\right)\right)-\mathbf{I}_{l}\left(\pi\left(\mathbf{p}^{l}_{i}\right)\right)$
(6)

Here, $\pi$ denotes the projection of the 3D map point from camera coordinates
to pixel coordinates. $\mathbf{\overline{\mathcal{\chi}}}$ represents the set
of keypoint indices. The photometric loss is constructed using the patches
around the projected points.

####  III-B2 Coarse-to-Fine Keypoint Tracking

Having obtained the coarse relative pose in the initial stage, we proceed to
match map points between adjacent frames and refine the pose of the current
frame $\mathbf{\xi}^{c}_{w}$ through the 3D-2D projection relationship.
Subsequently, leveraging the coarse relative pose
$\mathbf{\xi}^{\text{coarse}}_{lc}$, we project the 3D map points from the
last frame’s coordinate system to the pixel coordinate system of the current
frame. Subsequently, we search for matching keypoints within a fixed radius
range centered on the projection location, utilizing the Hamming distance of
the descriptors as the search criterion.

Upon establishing the 3D-2D matching relationships, we construct a pose graph.
As expressed in Equation 7, we employ the coarse relative pose in conjunction
with the last frame’s pose to derive the initial pose of the current frame.

$\hat{\mathbf{\xi}}^{c}_{w}=\mathbf{\xi}^{\text{coarse}}_{lc}\mathbf{\xi}^{l}_{w}$
(7)

The current frame’s pose serves as the vertex in the graph, with the number of
vertices being subject to optimization. Map points from the current frame and
the reprojection loss of these map points constitute the edges. We exclusively
optimize the pose within the tracking module. As detailed in Equation 8, we
iteratively refine the current frame pose $\hat{\mathbf{\xi}}^{c}_{w}$ to
minimize the reprojection loss. The g2o [30] framework is employed for
optimizing the pose graph.

${\mathbf{\xi}}^{c}_{w}=\arg\min_{\hat{\mathbf{\xi}}^{c}_{w}}\frac{1}{2}\sum_{i\in\mathbf{\overline{\mathcal{\chi}}}}\left\|{u_{i}}-\pi(\mathbf{\hat{\mathcal{\xi}}}^{c}_{w}.\mathbf{p}^{w}_{i})\right\|^{2}$
(8)

###  III-C Deep Keypoints based Loop Closing

Our DK-SLAM’s loop closure module includes detection and correction. To
address poor generalization, we employ online learning-based Bag of Words
(BoW) for the loop detector. Closed-loop nodes are identified, and the global
map is optimized using the loop relative pose.

####  III-C1 Online learning based Binary Bow

Unlike handcrafted descriptors, deep descriptors occupy a larger feature
space. Offline-trained BoW models may struggle to capture this space, making
it difficult to distinguish deep feature indices in BoW leaf nodes. To tackle
this, we introduce an online learning-based BoW that uses features exclusively
from testing scene data. Constructing a tree-like structure with floating
descriptors is challenging, so, inspired by [31], we employ binary hash
transformation for processing deep features. As depicted in Equation 9, for
descriptor vector values $\mathbf{d}_{i}$ less than 0, the processed
$\mathbf{\hat{d}}_{i}$ is set to 0. For descriptor values greater than 0, the
processed $\mathbf{\hat{d}}_{i}$ is set to 1, where $i$ ranges from 0 to 256.

Figure 3: Illustration of our proposed Online Learning based Binary BoW. The
BoW is constructed incrementally, with matched descriptors in the keyframes
database stored within the same leaf node. In the presence of unmatched
descriptors in the current keyframe, a new leaf node is created.

$\mathbf{\hat{d}_{i}}=\begin{cases}1&\text{ if }\mathbf{d}_{i}\geq 0\\\
0&\text{ if }\mathbf{d}_{i}<0\end{cases}$ (9)

####  III-C2 Loop Node Detection

As shown in the Figure 3, the loop closure thread receives keyframes filtered
from the local mapping thread, storing them in the inverted database to
construct a Bag-of-Words (BoW). The iBoW-LCD utilizes the feature descriptors
of the current frame $K_{c}$ to match descriptors stored in the database. In
iBoW-LCD, each keyframe is converted into a unique BoW vector for
representation. We compute the similarity score between the current keyframe
$K_{c}$ and the BoW vectors of inverted keyframes to identify the most similar
candidate keyframe $K_{h}$. However, the candidate keyframe $K_{h}$ only
considers differences in the numerical values of descriptors and neglects
differences in descriptor positions, which may lead to mismatches.

(a) Sequence 00 (b) Sequence 02 (c) Sequence 05 (d) Sequence 07 (e) Sequence
09 (f) Sequence 10

Figure 4: The generated trajectories of our proposed DK-SLAM on the Sequence
00, 02, 05, 07, 09 and 10 of the KITTI dataset, comparing with LDSO and ORB-
SLAM3.

Initially, we employ the Grid-Based Motion Statistics (GMS) [32] based brute
force matching method to match keypoints between candidate closed-loop nodes.
If the number of matches is below 20, the candidate loop keyframe is
classified as a mismatch. GMS operates on the principle that the number of
keypoints near correctly matched feature points should surpass the number near
incorrectly matched keypoints. Following a procedure similar to ORB-SLAM2, we
compute the similarity transformation matrix $\mathbf{T}_{\text{sim}}$ between
candidate closed-loop nodes through matched keypoints. Subsequently,
$\mathbf{T}_{\text{sim}}$ is applied to match map points between the current
keyframe $K_{c}$ and the candidate keyframe $K_{h}$. Further optimization of
the similarity transformation matrix $\mathbf{T}_{sim}$ is conducted using the
matched map points. If the number of inliers in the optimization function
exceeds 20, the candidate keyframe $K_{h}$ is deemed a matched keyframe
$K_{m}$.

####  III-C3 Global Map Correction by Loop Closure

Optimizing the similarity transformation matrix $\mathbf{T}_{\text{sim}}$
refines the pose of the current keyframe and its co-view related keyframes.
Concurrently, we update the positions of co-observed map points across these
keyframes. Next, we align the map points of the matched keyframe $K_{m}$ and
its connected keyframes with those of the current keyframe $K_{c}$ and its
connected keyframes. Updating the visibility graph between keyframes based on
matched map points follows. Utilizing $K_{c}$, $K_{m}$, global map points, and
the visibility graph, we construct and optimize the essential graph. Finally,
a global bundle adjustment refines all keyframes and map point positions
within the global map.

##  IV Experiments

TABLE I: The pose evaluation on the KITTI odometry dataset. Method | Sensor | Metric | 00 | 02 | 05 | 07 | 09 | 10 | Avg |
---|---|---|---|---|---|---|---|---|---|---
VISO-M | Mono | $t_{rel}$ | 36.95 | 21.98 | 17.20 | 20.00 | 29.01 | 28.52 | 25.61 |
$r_{rel}$ | 2.42 | 1.22 | 3.52 | 5.30 | 1.32 | 3.23 | 2.84 |
LDSO | Mono | $t_{rel}$ | 2.84 | 4.91 | 2.07 | 12.76 | 23.14 | 9.20 | 9.15 |
$r_{rel}$ | 0.36 | 0.74 | 0.24 | 5.13 | 0.21 | 0.21 | 1.15 |
ORB-SLAM3 | Mono | $t_{rel}$ | 2.80 | 6.04 | 3.41 | 2.14 | 3.13 | 10.78 | 4.72 |
$r_{rel}$ | 0.3 | 0.38 | 0.39 | 0.42 | 0.54 | 0.36 | 0.40 |
LIFT-SLAM | Mono | $t_{rel}$ | 3.18 | 8.73 | 6.09 | 2.42 | 19.91 | 9.72 | 8.34 |
$r_{rel}$ | 2.99 | 2.49 | 3.11 | 4.02 | 2.14 | 2.24 | 2.83 |
Ours | Mono | $t_{rel}$ | 2.57 | 4.38 | 1.98 | 1.11 | 2.97 | 7.03 | 3.34 |
$r_{rel}$ | 0.31 | 0.27 | 0.26 | 0.28 | 0.29 | 0.23 | 0.27 |

###  IV-A Training Details and Datasets

####  IV-A1 Training Details

In MAML-based training, we set the batch size to 8, with 4 batches dedicated
to support sets and 4 batches to query sets. Each minibatch is treated as a
task, and 4 tasks are trained in the inner loop. The trained model calculates
gradients on the query set, updating raw model parameters. The training spans
200k iterations, using the MS-COCO dataset [33].

####  IV-A2 KITTI Odometry Dataset

The KITTI odometry dataset [34] serves as a benchmark for self-driving
scenarios, offering stereo RGB and grayscale images, along with Lidar and pose
ground truth data. Grayscale images from sequences 00, 02, 05, 07, 09, and 10
are utilized for pose evaluation.

####  IV-A3 EuRoc MAV Dataset

The EuRoc dataset [35] represents an indoor MAV flight dataset, encompassing
stereo grayscale images, IMU data, and Leica Vicon’s pose ground truth.
Grayscale images are collected at a frequency of 20Hz. Due to space
limitations, we focus on sequences MH01-05 and V101 for SLAM performance
evaluation.

###  IV-B Pose Evaluation on the KITTI Dataset

To assess our SLAM’s effectiveness, we experimented with the KITTI dataset
using the official evaluation, computing root-mean-square error (RMSE) for
translation and rotation vectors. The evaluation spanned sequences from 100m
to 800m, providing an overall pose accuracy metric.

We thoroughly evaluated our SLAM’s tracking and loop closure in complex
environments, focusing on KITTI’s 00, 02, 05, 07, 09, and 10 sequences. Pose
estimation comparisons were made with LDSO [2], ORB-SLAM3 [12], VISO-M [36],
and LIFT-SLAM [8]. VISO-M exhibited the lowest accuracy, facing failure
scenarios, particularly in low-light conditions where corner detection proved
unstable. LDSO, featuring a loop closure module, addressed accumulated errors
in challenging scenes. ORB-SLAM3, an advanced keypoint-based SLAM with FAST
corner detection and BRIFE descriptors, demonstrated exceptional performance
across most scenes. Table I highlights ORB-SLAM3’s excellent performance in
most sequences. However, a notable performance drop is observed in sequence
10, attributed to a lack of reliable corner points leading to unstable
tracking and impacting overall pose estimation accuracy. Contrastingly, LIFT-
SLAM, relying on learning-based local features, exhibits subpar performance
across most sequences. This underscores that learning-based features may not
universally surpass traditional features. ORB’s consideration of scale,
direction, and a traditional SLAM technique for even keypoint distribution
contribute to its competitive edge.

TABLE II: Comparison of absolute translation errors (in meters) between the proposed DK-SLAM and other baselines on the EuRoC dataset. Scaling with the ground truth is necessary for evaluation due to the absence of absolute scale in monocular visual SLAM methods, as noted in the table. ”-” indicates performance failure. Method | MH01 | MH02 | MH03 | MH04 | MH05 | Avg
---|---|---|---|---|---|---
DSO | 0.046 | 0.046 | 0.172 | 3.810 | 0.110 | 0.8368
SVO | 0.100 | 0.120 | 0.410 | 0.430 | 0.300 | 0.2720
DSM | 0.039 | 0.036 | 0.055 | 0.057 | 0.067 | 0.0508
ORB-SLAM3 | 0.016 | 0.027 | 0.028 | 0.138 | 0.072 | 0.0562
LIFT-SLAM | 0.044 | 0.053 | 0.049 | - | - | -
Ours | 0.013 | 0.013 | 0.027 | 0.077 | 0.055 | 0.0370

Learning-based local features perform well in matching benchmarks, but face
challenges in SLAM due to correlated keypoints in dynamic scenes. Ignoring
these correlations limits the exploitation of learning-based features’
advantages. Additionally, the feature space of learning-based features differs
from handcrafted ones. Offline Bag-of-Words (BoWs) classify input features
quickly, but traditional descriptors like BRIEF have a limited description
space. Learning-based feature training is a ”black box,” complicating
understanding, and testing performance depends on the training set’s
distribution, hindering the acquisition of a generalized BoW.

Our proposed SLAM excels in most KITTI sequences, employing a coarse-to-fine
matching strategy using keypoint-surrounding patches. Pose estimation
optimization leverages the 3D-2D matching relationship. Online training
constructs a learning-based feature BoW using prior experienced data, limiting
the BoW’s feature description space. Results confirm the online BoW’s
effectiveness in loop scene detection, with the loop closure module accurately
rectifying cumulative errors.

Figure 5: The samples of keypoints detection and matching. Up: matching without two-stage strategy (Ours1). Bottom: matching with two-stage strategy (Ours3). TABLE III: The ablation study into our DK-SLAM modules. Method | MAML | Two-Stage | Metric | 00 | 02 | 05 | 07 | 09 | 10 | Avg |
---|---|---|---|---|---|---|---|---|---|---|---
Ours1 | $\checkmark$ |  | $t_{rel}$ | 2.88 | - | 2.80 | 1.76 | 2.70 | - | - |
$r_{rel}$ | 0.41 | - | 0.35 | 0.64 | 0.24 | - | - |
Ours2 |  | $\checkmark$ | $t_{rel}$ | 2.92 | 4.42 | 2.48 | 2.11 | 4.03 | 8.48 | 4.07 |
$r_{rel}$ | 0.38 | 0.34 | 0.34 | 0.46 | 0.20 | 0.23 | 0.325 |
Ours3 | $\checkmark$ | $\checkmark$ | $t_{rel}$ | 2.57 | 4.38 | 1.98 | 1.11 | 2.97 | 7.03 | 3.34 |
$r_{rel}$ | 0.31 | 0.27 | 0.26 | 0.28 | 0.29 | 0.23 | 0.27 |

###  IV-C Pose Evaluation on the EuRoC Dataset

We validate our proposed DK-SLAM on the EuRoC dataset, addressing scale
ambiguity with umeyama alignment [37] for monocular vision methods.
Comparisons include ORB-SLAM3, DSO, DSM [38], SVO, and LIFE-SLAM [8]. Results,
derived from [6] for ORB-SLAM3, DSO, DSM, and SVO, and [8] for LIFE-SLAM,
utilize absolute translation error as the experiment metric . The proposed
SLAM exhibits exceptional positioning performance, leveraging a two-stage
tracking strategy for precise feature point location and reduced incorrect
matching. Learnt features demonstrate robust detection in low-light
environments, fostering stable feature tracking. The online learning-based
deep BoW excels in loop detection, evident in MH02 where the proposed SLAM
outperforms ORB-SLAM3. This advantage is attributed to learned features
constructing BoW, providing detailed scene distinction, and online training
for swift adaptation to varying environments.

###  IV-D Ablation Study

Table III summarizes the results of our ablation study, assessing the impact
of our MAML-based deep feature training and coarse-to-fine feature tracking.
In ”Ours1,” lacking a coarse-to-fine tracking strategy, reliance on a constant
velocity motion model for keypoint matching leads to errors, resulting in
inaccurate correspondence and tracking failure. Both ”Ours1” and ”Ours3” use a
keypoint search radius of 7.

Figure 5 shows a failure case for ”Ours1” in accurate feature matching due to
uniform motion model challenges. In contrast, ”Ours3” uses a two-stage
strategy for stable tracking, combining a semi-direct method for coarse pose
estimation and refined feature matching. ”Ours2” with the original Superpoint
strategy performs slightly worse than ”Ours3” trained with MAML. In ”Ours3,”
MAML is applied to train the local feature extractor by iteratively enhancing
generalization on support and query sets, resulting in a robust and adaptable
detector. The enhanced SLAM performance attributed to MAML is primarily
observed in feature matching, as depicted in Figure 6. Using Superpoints for
matching increases the number of matches, addressing ORB-SLAM3’s limitations
in texture-less areas. ORB-SLAM3’s descriptors, describing lighting changes
around corners, are susceptible to environmental variations. In contrast,
learning-based local feature methods like Superpoint capture robust deep
features unaffected by lighting changes, ensuring stable tracking across
consecutive frames.

Table IV presents the average number of matching points for ORB-SLAM3, Ours2,
and Ours3 on Sequence 00, 05, 09, and 10 of the KITTI dataset. Setting the
total number of keypoints to 3000 and pyramid level to 3, MAML-based training
significantly improves the feature detection performance of superpoints,
allowing them to acquire more robust features and enhancing SLAM accuracy.

TABLE IV: The average number of matching points for ORB-SLAM3, and DK-SLAM without (Ours2) or with MAML feature training (Ours3). Seq | ORB-SLAM3 | Ours2 | Ours3
---|---|---|---
K00 | 228.3 | 352.6 | 376.7
K05 | 219.5 | 371.2 | 383.1
K09 | 150.4 | 258.0 | 265.3
K10 | 171.4 | 306.0 | 323.1
Figure 6: The samples of keypoints detection and matching. From top to bottom: ORB-SLAM3 matching (ORB3), DK-SLAM without (Ours2) or with MAML feature training (Ours3). Figure 7: Precision-Recall curves depicting the performance of the Bag-of-Words (BoW) approach in the proposed DK-SLAM on Sequences 00, 05, and 06 of the KITTI dataset. TABLE V:  Performance comparison of loop-closing with the maximum recalls across various methods, all achieving 100% precision. The numerical data presented in this table is sourced from [26, 39], and [40]. Seq | FAB-MAP2 | Emilio | Milford | seqSLAM | iBoW | Ours
---|---|---|---|---|---|---
K00 | 0.49 | 0.90 | 0.67 | 0.67 | 0.77 | 0.98
K05 | 0.32 | 0.76 | 0.41 | 0.36 | 0.26 | 0.87
K06 | 0.55 | 0.95 | 0.65 | 0.65 | 0.96 | 1

We perform ablation studies on the proposed loop closure method to assess its
performance. Precision-Recall metrics on KITTI 00, 05, and 06 sequences are
illustrated in Figure 7, with additional evaluation at 100% precision. The
quantitative results, presented in Table V, reveal our deep BoW’s superior
recall rate compared to traditional BoWs. Notably, in the KITTI 06 sequence,
our BoW achieves a remarkable 100% recall rate. In contrast, the recall rate
of the iBoW, belonging to the traditional BoW category, is significantly
lower, suggesting that handcrafted descriptors might struggle to accurately
identify loop nodes in a sequence. The learned local descriptor captures high-
level information, enhancing robustness across diverse environments. This
feature stability, unaffected by lighting changes, results in superior loop
detection performance.

##  V Conclusion

This work presents DK-SLAM, a monocular visual SLAM with adaptive learning
features, featuring a learned local feature detector, a coarse-to-fine
matching strategy, and an online binary learned feature BoW. To enhance
generalization, we employ MAML for adaptive training, yielding an extractor
with robust capabilities. Coarse relative poses between frames are estimated
via a semi-direct method for accurate feature point matching. An online binary
learned feature BoW corrects SLAM accumulation errors. Our proposed DK-SLAM
outperforms representative baselines, e.g. ORB-SLAM3, on the KITTI and EuRoC
datasets. However, the GPU-based online front-end faces efficiency challenges
during information transmission to the CPU-based back-end. Future work will
explore knowledge distillation for parameter compression and SLAM framework
modifications to enhance efficiency.

## References

  * [1] J. J. Engel, V. Koltun, and D. Cremers, “Direct sparse odometry,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 40, pp. 611–625, 2016.
  * [2] X. Gao, R. Wang, N. Demmel, and D. Cremers, “Ldso: Direct sparse odometry with loop closure,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2018, pp. 2198–2204.
  * [3] C. Forster, M. Pizzoli, and D. Scaramuzza, “Svo: Fast semi-direct monocular visual odometry,” in _2014 IEEE International Conference on Robotics and Automation (ICRA)_ , 2014, pp. 15–22.
  * [4] R. Mur-Artal and J. D. Tardós, “Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras,” _IEEE Transactions on Robotics_ , vol. 33, no. 5, pp. 1255–1262, 2017.
  * [5] C. Campos, R. Elvira, J. J. G. Rodríguez, J. M. M. Montiel, and J. D. Tardós, “Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam,” _IEEE Transactions on Robotics_ , vol. 37, no. 6, pp. 1874–1890, 2021.
  * [6] Z. Teed and J. Deng, “Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras,” in _Neural Information Processing Systems_ , 2021.
  * [7] J. Tang, L. Ericson, J. Folkesson, and P. Jensfelt, “Gcnv2: Efficient correspondence prediction for real-time slam,” _IEEE Robotics and Automation Letters_ , vol. 4, pp. 3505–3512, 2019.
  * [8] H. M. S. Bruno and E. Colombini, “Lift-slam: a deep-learning feature-based monocular visual slam method,” _Neurocomputing_ , vol. 455, pp. 97–110, 2020.
  * [9] Y. S. Huaiyang Huang, Haoyang Ye and M. Liu, “Monocular visual odometry using learned repeatability and description,” in _IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020.
  * [10] Y. Lu and G. Lu, “Superthermal: Matching thermal as visible through thermal feature exploration,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 2690–2697, 2021.
  * [11] E. Rublee, V. Rabaud, K. Konolige, and G. Bradski, “Orb: An efficient alternative to sift or surf,” in _2011 International Conference on Computer Vision_ , 2011, pp. 2564–2571.
  * [12] T. Lindeberg, _Scale Invariant Feature Transform_ , 05 2012, vol. 7.
  * [13] J. Shi and C. Tomasi, “Good features to track,” _1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 593–600, 1994.
  * [14] T. Qin, P. Li, and S. Shen, “Vins-mono: A robust and versatile monocular visual-inertial state estimator,” _IEEE Transactions on Robotics_ , vol. 34, no. 4, pp. 1004–1020, 2018.
  * [15] D. DeTone, T. Malisiewicz, and A. Rabinovich, “Superpoint: Self-supervised interest point detection and description,” _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pp. 337–33 712, 2017.
  * [16] M. Dusmanu, I. Rocco, T. Pajdla, M. Pollefeys, J. Sivic, A. Torii, and T. Sattler, “D2-net: A trainable cnn for joint description and detection of local features,” in _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 8084–8093.
  * [17] A. Barroso-Laguna and K. Mikolajczyk, “Key.net: Keypoint detection by handcrafted and learned cnn filters revisited,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 45, no. 1, 2023.
  * [18] Y. Tian, B. Fan, and F. Wu, “L2-net: Deep learning of discriminative patch descriptor in euclidean space,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * [19] J. Tang, J. Folkesson, and P. Jensfelt, “Geometric correspondence network for camera motion estimation,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 2, pp. 1010–1017, 2018.
  * [20] D. Li, X. Shi, Q. Long, S. Liu, W. Yang, F. Wang, Q. Wei, and F. Qiao, “Dxslam: A robust and efficient visual slam system with deep features,” _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pp. 4958–4965, 2020.
  * [21] P.-E. Sarlin, C. Cadena, R. Siegwart, and M. Dymczyk, “From coarse to fine: Robust hierarchical localization at large scale,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [22] J. Jiang, X. Chen, W. Dai, Z. Gao, and Y. Zhang, “Thermal-inertial slam for the environments with challenging illumination,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 4, pp. 8767–8774, 2022.
  * [23] K. M. Yi, E. Trulls, V. Lepetit, and P. V. Fua, “Lift: Learned invariant feature transform,” in _European Conference on Computer Vision_ , 2016.
  * [24] D. Galvez-López and J. D. Tardos, “Bags of binary words for fast place recognition in image sequences,” _IEEE Transactions on Robotics_ , vol. 28, no. 5, pp. 1188–1197, 2012.
  * [25] M. Calonder, V. Lepetit, C. Strecha, and P. V. Fua, “Brief: Binary robust independent elementary features,” in _European Conference on Computer Vision_ , 2010.
  * [26] E. Garcia-Fidalgo and A. Ortiz, “ibow-lcd: An appearance-based loop-closure detection approach using incremental bags of binary words,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 4, 2018.
  * [27] R. Arandjelović, P. Gronat, A. Torii, T. Pajdla, and J. Sivic, “Netvlad: Cnn architecture for weakly supervised place recognition,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2018.
  * [28] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _CoRR_ , vol. abs/1409.1556, 2014.
  * [29] Y.-Y. Jau, R. Zhu, H. Su, and M. Chandraker, “Deep keypoint-based camera pose estimation with geometric constraints,” _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pp. 4950–4957, 2020.
  * [30] R. Kümmerle, G. Grisetti, H. Strasdat, K. Konolige, and W. Burgard, “G2o: A general framework for graph optimization,” in _IEEE International Conference on Robotics and Automation_ , 2011, pp. 3607–3613.
  * [31] Y. Wang, B. Xu, W. Fan, and C. Xiang, “A robust and efficient loop closure detection approach for hybrid ground/aerial vehicles,” _Drones_ , vol. 7, no. 2, 2023.
  * [32] J. Bian, W.-Y. Lin, Y. Matsushita, S.-K. Yeung, T.-D. Nguyen, and M.-M. Cheng, “Gms: Grid-based motion statistics for fast, ultra-robust feature correspondence,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 2828–2837.
  * [33] T.-Y. Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European Conference on Computer Vision_ , 2014.
  * [34] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The kitti dataset,” _The International Journal of Robotics Research_ , vol. 32, pp. 1231 – 1237, 2013.
  * [35] M. Burri, J. Nikolic, P. Gohl, T. Schneider, J. Rehder, S. Omari, M. Achtelik, and R. Y. Siegwart, “The euroc micro aerial vehicle datasets,” _The International Journal of Robotics Research_ , 2016.
  * [36] A. Geiger, J. Ziegler, and C. Stiller, “Stereoscan: Dense 3d reconstruction in real-time,” in _IEEE Intelligent Vehicles Symposium_ , 2011.
  * [37] S. Umeyama, “Least-squares estimation of transformation parameters between two point patterns,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 13, pp. 376–380, 1991.
  * [38] J. Zubizarreta, I. Aguinaga, and J. M. M. Montiel, “Direct sparse mapping,” _IEEE Transactions on Robotics_ , vol. 36, no. 4, 2020.
  * [39] H. Yue, J. Miao, Y. Yu, W. Chen, and C. Wen, “Robust loop closure detection based on bag of superpoints and graph verification,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2019, pp. 3787–3793.
  * [40] G. Singh, M. Wu, S. Lam, and D. Van Minh, “Hierarchical loop closure detection for long-term visual slam with semantic-geometric descriptors,” in _2021 IEEE International Intelligent Transportation Systems Conference (ITSC)_ , 2021, pp. 2909–2916.