#  Online Learning of Neural Surface Light Fields
alongside Real-time Incremental 3D Reconstruction

Yijun Yuan and Andreas Nüchter Manuscript received: January, 24, 2023; Revised
March, 29, 2023; Accepted April, 28, 2023.This paper was recommended for
publication by Editor Civera Javier upon evaluation of the Associate Editor
and Reviewers’ comments.All authors are with Informatics XVII : Robotics,
Julius-Maximilians-Universität Würzburg, Germany. {yijun.yuan|andreas.
<EMAIL_ADDRESS>
###### Abstract

Immersive novel view generation is an important technology in the field of
graphics and has recently also received attention for operator-based human-
robot interaction. However, the involved training is time-consuming, and thus
the current test scope is majorly on object capturing. This limits the usage
of related models in the robotics community for 3D reconstruction since robots
(1) usually only capture a very small range of view directions to surfaces
that cause arbitrary predictions on unseen, novel direction, (2) requires
real-time algorithms, and (3) work with growing scenes, e.g., in robotic
exploration. The paper proposes a novel Neural Surface Light Fields model that
copes with the small range of view directions while producing a good result in
unseen directions. Exploiting recent encoding techniques, the training of our
model is highly efficient.

In addition, we design Multiple Asynchronous Neural Agents (MANA), a universal
framework to learn each small region in parallel for large-scale growing
scenes. Our model learns online the Neural Surface Light Fields (NSLF) aside
from real-time 3D reconstruction with a sequential data stream as the shared
input. In addition to online training, our model also provides real-time
rendering after completing the data stream for visualization. We implement
experiments using well-known RGBD indoor datasets, showing the high
flexibility to embed our model into real-time 3D reconstruction and
demonstrating high-fidelity view synthesis for these scenes. The code is
available on github111https://jarrome.github.io/NSLF-OL.

###### Index Terms:

Mapping; SLAM

##  I Introduction

In robotics, mapping, and 3D reconstruction have long been of great interest
and have been studied for decades. Initially, researchers focused on 3D point
clouds and voxel grids, and this later shifted towards using Signed Distance
Functions (SDF). An SDF, is the starting point of many of the following state-
of-the-art papers. It has been well-developed from Point-to-SDF [1] to SDF-to-
SDF [2, 3], from explicit voxel field [4] to neural implicit representation
[5, 6]. These methods have achieved high-quality 3D reconstructions in real-
time while maintaining high regression functionality of the neural models.

Aside from the reconstruction of geometries, neural rendering for colors is
also a hot topic, however more in the field of computer graphics [7]. Neural
rendering focuses on the synthesis of novel views from 3D models. The goal of
this topic is to enhance the immersive experience for users. In the context of
robotics, for instance, it is crucial for human-robot interaction or situation
awareness of an operator of a telerobot. In 2020, the Neural Radiance Field
(NeRF) [8] was introduced as an innovative approach to high-resolution
rendering for view synthesis, activating the trend of neural rendering. It
learns a neural radiance field that produces both an occupancy field and a
light field via differentiable rendering for custom immersive view synthesis,
and it has extended the testing scope of 3D reconstruction to large scene-
level [9, 10]. The drawback is the extremely high training time, and it is
therefore unsuitable for real-time 3D reconstruction. Nevertheless, it still
affects the field of 3D reconstruction. Recent works on 3D reconstructions,
i.e., iMAP [11] and NICE-SLAM [12], build on the differentiable rendering from
NeRF to approach an online reconstruction with color. However, their
visualization of color appears blurred.

[width=1]im/eps/limited_light

Figure 1: Light difference between object capturing for graphics applications
and scene capturing in robotic SLAM & reconstruction. At 3D point
$\mathbf{p}$, lights are cast from direction $\mathbf{d}$. The sphere shows
the $\mathbf{S}^{2}$ space for the direction vectors $\mathbf{d}$ that are
partially covered by light rays.

It is worth mentioning that shape reconstruction has already been extensively
studied and has already achieved high-quality 3D reconstruction performance.
Aside from the high quality, the robotics community values online capable
reconstructions.

However, NeRFs in the graphics community concentrate _more_ on (1) object
capturing instead of capturing large scene surfaces (captures contain dense
view-directions to surface), (2) rendering high-quality images while caring
less on depth and surface accuracies (shape-radiance ambiguity) [13], and (3)
more on rendering (testing) speed instead of online training. Therefore, we
explore the high potential to use surface reconstruction research as the basis
of this topic to meet the interest of the robotics community to have a
universal way to simulate real environments.

We utilize a real-time reconstruction model to provide surfaces and simplify
the problem to online learn the light field on surfaces aside from real-time
reconstruction as given in Fig. 3. Where our model relies on data from
reconstruction without affecting it. During the online learning phase, we
employ a colored point cloud. Subsequently, in the inference phase, we apply
ray-rasterizing on the reconstruction mesh to obtain the point cloud for color
prediction.

The Surface Light Fields (SLF) has been proposed by Wood et al. [14] to model
the surface reflectance. For a surface point $\mathbf{p}$, the radiance of a
reflection ray (with direction $\mathbf{d}_{o}$) is computed from an
accumulation of incident rays (with direction $\mathbf{d}_{i}$ for ray $i$).
However, due to the inefficient dense sampling requirement of SLF, more work
has focused on modeling compression [15]. Recently, Chen et al. [16] and Yu et
al. [17] have introduced Deep Surface Light Fields (DSLF), which utilize a
neural network to replace the previous handcrafted formulation. Still, these
approaches have an extremely slow training speed. Similarly, these techniques
are mainly concentrated on capturing a 360-degree view of the object.Large-
scale scenes have not been considered.

Unlike the graphics communities, in robotics, SLAM, and 3D reconstruction _(1)
capture scene frames that contain only a limited range of surface view-
direction_ as depicted Fig. 1, apply for _(2) growing large-scale scenes
without prior knowledge of their size_ and require _(3) real-time processing
speed_. These three points are the primary concerns for using Neural Surface
Light Fields (NSLF) in robotics. Thus, this paper aims to address the limited
view-direction challenge during testing, where most unseen directions will
cause arbitrary results. To cope with this problem, we introduce Spherical
Harmonics for _decoding_ , as depicted in Fig. 2. Furthermore, relying on
recent advances in graphics, i.e., the Multi-resolution Hash-encoding, we
train grid-latent as an encoder for (3) real-time online learning. We further
introduce Multiple Asynchronous Neural Agents (MANA) to handle (2), i.e., we
handle large scenes without pre-knowing the size.

We take inspiration from [18] which claims that multiple local MLPs converge
faster. However, [18] has to train all local MLPs with one optimizer because
NeRF’s differentiable rendering accumulates data from different MLPs. While
Surface Lighting Field training does not depend on ray integration in NeRF.
Therefore, each region has its model and optimizer to learn individually.

To deal with scalable data when data is distributed, we dynamically allocate
neural models and optimizers for new regions and run them in a new thread
independently as depicted in Fig. 3. Every region has a neural model and an
optimizer running independently without synchronizing with others.

The contributions of this paper are as follows:

  * •

Proposing a novel Neural Surface Light Fields model to address the issue of
arbitrary prediction on unseen direction causing from the small range of view-
direction to surface in SLAM & 3D reconstruction captures,

  * •

Proposing the first framework (MANA) for online-learning neural surface light
fields on growing large-scale scenes,

  * •

Implementing MANA aside from real-time reconstruction for experiments. Both
online learning and real-time testing are supported. Agents can be distributed
to multiple GPUs.

##  II Related Works

Our work is mainly related to NSLF. Here, we also review the hotter sister
topic, NeRF because its development process highly inspires and directs us to
use NSLF online for the large scenes.

###  II-A Neural Radiance Fields (NeRF)

Mildenhall et al. [8] propose to represent a scene as a continuous mapping
from five-dimensional coordinates $(x,y,z,\theta,\phi)$ to volume density and
view-dependent RGB color $(\sigma RGB)$, which are so-called NeRF
representations. The training of NeRF relies on the use of differentiable
rendering along rays. NeRF represents a significant advancement in graphics
due to its immersive view-synthesis performance. However, NeRF does have some
limitations. Firstly, the speed for training and rendering is extremely slow.
Secondly, the NeRF model suffers from shape-radiance ambiguity.

####  II-A1 Speed-aware NeRF

Although NeRFs show a very high quality of view synthesis, their computational
cost for training and testing is extremely high. Therefore, lots of recent
work shows interest in the speed issue. Tewari et al. [7] summarize and show
many works exploring speedups using pruning [19], sampling [20], fast
integration [21], data structure [22, 23, 24], and so on [18, 25, 26]. Many
different data structures are also used to speed up the rendering and the
training process benefits. A recently highlighted work from NVIDIA, Instant
Neural Graphics Primitives, uses a multi-resolution hash encoding to realize
fast NeRF training in a few seconds [24]. It provides an encoder with very
fast convergence. We will also employ this encoder in our proposed model. In
the most related scope, KiloNeRF, which uses thousands of tiny MLPs to imitate
standard NeRF, shows a factor of 2000 speedup in rendering [18]. Real-time
rendering already meets the needs of the field of computer graphics (CG).

However, from a robotics perspective, online algorithms such as 3D
reconstruction and SLAM are always preferred. In such an environment, online
training makes the task of light fields much more challenging.

####  II-A2 Depth Aided NeRF

Another problem with NeRF is shape-radiance ambiguity. NeRF++ [13] points out
that given an incorrect shape, there exists a family of radiance fields that
perfectly explains the training images, but generalizes poorly to novel views.
Although a well-trained NeRF does not show this phenomenon, imperfect depth
prediction is common but is neglected since the application targets color
prediction. However, this also leads to another research direction, a group of
works uses depth as an input to reduce the complexity of geometry modeling.
This group of works embed Structure-from-motion (SfM) [27, 28], Multi-view
Stereo (MVS) [29, 30] or directly input RGBD data [31].

The above algorithms show a trend that the facilitation of geometry learning
well-reduces the complexity of NeRF while improving the high quality.
Moreover, in the robotics community, recent mature real-time 3D
reconstructions already provide high-quality online shape reconstruction [5].

Therefore, we consider solving the geometric and color regression separately.
Where the geometry is estimated using a mature online 3D reconstruction model.
Meanwhile, an NSLF is independently trained for surface colors. Without ray
sampling, our model uses only surface points during training and inference.
This breaks the constraint of NeRF’s rendering loss and allows asynchronous
training and inference for each intelligent agent.

###  II-B Neural Surface Light Fields (NSLF)

NSLF learns a mapping function
$f_{NSLF}:\mathbf{X}\times\mathbf{S}^{3}\rightarrow\mathbf{C}$ where
$\mathbf{X}\subseteq\mathbb{R}^{3}$ denotes the point set on the surface,
$\mathbf{S}^{3}$ is the unit 3-sphere, $\mathbf{C}$ denotes the RGB color
space.

With the high regression capability of deep learning, Chen et al. [16] propose
Deep Surface Light Fields (DSLF) which uses MLPs to replace the accumulation
function. This parametric modeling greatly reduces the computational and
spatial burden. And thus provides more efficient rendering. Yu et al. [17]
have introduced Anisotropic Fourier Features Mapping (AFFM) to encode points.
This increases convergence speed and rendering quality.

Similar to the NeRF design, the NSLFs also follow the encoding-decoding
pattern. For a given point, its position vector is encoded as
$\mathbf{F}_{\mathbf{p}}=\phi_{p}(\mathbf{p})$. In the other branch, the
direction vector is also encoded with a non-parameterized encoder, spherical
harmonics, or frequency encoding:
$\mathbf{F}_{\mathbf{d}}=\varphi_{sh/freq}(\mathbf{d})$. Shallow MLPs are then
applied to concatenated features (position encoding and direction) to predict
color:
$\mathbf{c}=\phi_{dec}(\mathbf{F}_{\mathbf{p}},\mathbf{F}_{\mathbf{d}})$.

However, similar to NeRF the NSLF has not been used in large-scale incremental
3D reconstruction. Thus, in the following, we fill this gap and provide an
online learned neural surface light fields function for large scales.

##  III Methodology

In the following, we introduce our proposed NSLF model, that addresses the
problem of limited view directions toward the surfaces. Then we further design
a Multiple Asynchronous Neural Agents (MANA) framework that learns the NSLF
alongside the reconstruction. The geometry regression that provides the
surface relies on the existing incremental 3D reconstruction models.

###  III-A NSLF Design

In the use of capturing of 360 degrees of an object, for a certain surface
point, the trained view directions are densely sampled over a large range and
thus, the novel view inference is mostly on the touched view directions.
However, in large-scale 3D SLAM and reconstruction, this feature of data
acquisition is not guaranteed. Therefore, training on a small range of view
directions will lead to arbitrary prediction on unseen views.

[width=1]im/eps/models

Figure 2: Patterns of models designed for Neural Light Field. (a) is most
widely used. Our (b) learn one sphere for each $\mathbf{p}$ and use a
$\phi_{\mathbf{w}_{p}}$ to map the 1D value on direction $\mathbf{d}$ to 3D
RGB.

Unlike previous encoding schemes that concatenate position and direction
encoding, in Fig. 2, we propose to learn Spherical Harmonics (SH) parameters
from the position encodings. On the decoding side, we run deterministic
formulas with a known SH basis.

To achieve the speedup from current techniques, we use the novel
Multiresolution Hash Encoding [24] to bear the main burden of position
encoding: $\mathbf{F}_{\mathbf{p}}=\phi_{p,\theta_{HG}}(\mathbf{p})$. Such an
encoding produces an encoding by interpolation of voxel features, which
mitigates the global effect problem of MLPs in DSLF. In addition, the surfaces
occupy a small space in a cubic region, and thus the allocation during use is
more efficient in space. Considering that scene sequences only cover very
limited direction space on surface surface, training on specific positions and
directions easily leads to an arbitrary result in unseen view. To prevent
arbitrary guessing in the unseen direction, we introduce learning of SH
parameters that affect the full direction. We add an MLP-layer to generate the
SH parameters:
$\mathbf{F}_{sh}=\phi_{mlp}(\phi_{\theta_{HG}}(\mathbf{p}))=(v_{\ell}^{m})_{\ell:0\leq\ell\leq\ell_{max}}^{m:\ell\leq
m\leq\ell}$. So here we are creating one sphere of latent to continuously
represent a small space of $1$ dimensional color range for corresponding
lights on a point. Then, with the known spherical harmonics function
$Y_{\ell}^{m}:\mathbf{S}^{2}\rightarrow\mathbb{C}$, we extract the latent in a
given direction of the sphere utilizing the SH formula
$\sum_{\ell=0}^{\ell_{max}}\sum_{m=-\ell}^{\ell}v^{m}_{\ell}Y_{\ell}^{m}(\mathbf{d})$
to deterministically decode on direction $\mathbf{d}$:
$s=\mathbf{F}_{sh}^{T}\varphi_{sh}(\mathbf{d})$. Meanwhile, the other branch
of the encoder computes features $\mathbf{w}_{p}$ to parameterize MLPs
$\phi_{\mathbf{w}_{p}}$. It maps (or extracts) the color range value $s$ to
the color $\mathbf{c}$.

Hash encoding requires normalizing the data into a unit cube [24]. This means
that we need to know the scale of the data. Thus, using this model in each
region avoids this problem as the regional scale is predefined. In the
following, we present a framework that operates on the growing scenes.

###  III-B NSLF with Region-wise Neural Agents

DSLF [16] uses MLPs to learn an entire scene. But such a model is not robust
for the incrementally creation of large scenes, because newly fed data still
changes the scene globally even the non-effected parts. Some ideas in NeRF
provide possible solutions: (1) using voxels of multiple MLPs (KiloNeRF [18])
or (2) using a voxel multi-resolution representation (Instant-NGP [24]).

####  III-B1 Multiple Asynchronous Neural Agents

[width=1.]im/eps/workflow

Figure 3: MANA learns online a NSLF by serving as an external function to 3D
reconstruction.

We use RGBD and rasterization to directly approach points on the surface
during training and testing respectively.

To speed up training while making it scalable, we divide the space equally
into regions and dynamically assign models for each newly touched region. Each
region is assigned with an Intelligent Agent (IA) that maintains its own
thread, neural model and optimizer and is trained autonomously.In training
mode, when data is fed into an agent, it will train continuously until it
reaches max-iterations. In evaluation mode, the IA will predict input data and
output colors. Since the implementation is done with neural network, we will
refer to such an IA as a Neural Agent (NA) in the rest of this paper.

In addition, we assign an optimizer to each NA, making it an independent model
that does the training itself. Each region maintains its own neural model and
optimizer to train independently. With this feature, our model distinguishes
itself from KiloNeRF which synchronizes voxels . And thus, our model is
capable of asynchronous training. We call it Multiple Asynchronous Neural
Agents (MANA) with each NA $\mathbf{U}=(\phi_{\theta},Optim)$.

Then following KiloNeRF, we use an axis-aligned bounding box (AABB) to enclose
the scene. We preset $\mathbf{b}_{min}$ and $\mathbf{b}_{max}$ as the minimum
and maximum bound of AABB and discretize the space uniformly with the
resolution $\mathbf{r}=(r_{x},r_{y},r_{z})$. We assign to each grid cell the
index $\mathbf{i}=(i_{x},i_{y},i_{z})$ for the NA $\mathbf{U}_{\mathbf{i}}$.
For growing scenes, we define extremely large box that cost almost nothing.

Given a RGBD frame, we unproject it to 3D space as a point cloud
$\mathbf{P}\in\mathbb{R}^{N_{\mathbf{P}_{j}}\times 3}$ and their corresponding
color $\mathbf{C}\in\mathbb{R}^{N_{\mathbf{P}}\times 3}$. The corresponding
direction $\mathbf{D}\in\mathbb{R}^{N_{\mathbf{P}}\times 3}$ is also obtained
for light fields. Each point $\mathbf{p}\in\mathbf{P}$ is assigned to the
corresponding region with

$v(\mathbf{p})=\lfloor(\mathbf{p}-\mathbf{b}_{min})/(\mathbf{b}_{max}-\mathbf{b}_{min})/\mathbf{r}\rfloor.$
(1)

Then the color is predicted with the point $\mathbf{p}_{j}\in\mathbf{P}$ and
its direction vector $\mathbf{d}_{j}\in\mathbf{D}$ as input:

$\mathbf{c}_{j,pred}=\phi_{\theta(v(\mathbf{p}_{j}))}(\mathbf{p}_{j},\mathbf{d}_{j})$
(2)

Thus, for each agent $\mathbf{i}$, it optimizes

$\min_{\theta(\mathbf{i})}\sum_{j,\
v(\mathbf{p}_{j})=\mathbf{i}}||\mathbf{c}_{j}-\mathbf{c}_{j,pred}||^{2}_{2}$
(3)

independently with $\text{Optim}_{\mathbf{i}}$.

###  III-C Online Learning of Surface Light Fields

As indicated in Fig. 3, our MANA works aside from the real-time 3D
reconstruction of the model. Note that, our model uses colored point clouds
and poses sequentially from the reconstruction. But the training of MANA does
not depend on the reconstruction result. The reconstruction mesh is only used
during the testing, i.e., view rendering, where the surface is needed to
determine the intersection points of rays. In our implementation, the main
thread is used to feed data into MANA and to perform the reconstruction.

We plot the diagram of MANA as a frame is fed into the pink window of Fig. 4.

[width=1]im/eps/pipeline2

Figure 4: Online Learning of NSLF. (a) shows MANA distributing data into
different Neural Agents by region. Each Neural agent maintains its thread and
optimizes individually as (b).

####  III-C1 Distribution Module

When a frame arrives, we generate a colored point cloud with direction
$(\mathbf{P},\mathbf{C},\mathbf{D})$. The distribution module feeds the points
to their corresponding regions with their position. Together with the feed
data, to ensure equal training of each region, our distribution module assigns
iterations to less trained models, while setting zero iterations to more
trained models.

Since the distribution module and agents run in different threads, the data
distribution is sufficient for real-time data streaming while the training of
the color models is optimized independently in the background.

####  III-C2 Asynchronous Neural Agents

When training data of region $k$ is passed to agent $k$, it is appended to the
data memory stack. Each agent maintains its own thread to independently train
the color field model $\theta$ with memory data. As described in Fig. 4 (b), a
thread of NA maintains the optimization of the neural model. The newly
distributed data for an agent is appended into its own memory stack.
Meanwhile, left iterations $a$ is modified for more iterations (in the
experiments $a$ is set large enough such that it does not block the whole
algorithm). Then a signal is given to continue training.

###  III-D View Rendering

After a complete online pass of the data stream, a learned NSLF and a surface
are obtained.

By rasterizing with a surface mesh and camera pose as input, we obtain from
the unprojected surface point $\mathbf{P}$. Rendering is then implemented by
assigning points to different agents and synchronously predicting using Eq. 2.

We show in the experiments, that in addition to the real-time training, the
rendering is also done in real-time.

##  IV Results and Experiments

Our experiments are on the scenario of Real-time Incremental sequences, which
run the online learning modules (MANA) aside from reconstructing.

###  IV-A Dataset

##### ICL-NUIM

ICL-NUIM [32] is one of the most widely used RGBD datasets for SLAM and
reconstruction purposes. ICL-NUIM contains two synthetic scenes, i.e., room
and office.

##### Replica

The Replica dataset [33] provides synthetic indoor space reconstructions that
contain clean dense geometry, high resolution, and dynamic textures. It is
suitable for our surface light field purpose.

###  IV-B Baselines

Our comparisons are mainly on _online_ learning of Neural Surface Light Fields
given incremental reconstruction frames in real-time. The reconstruction model
that we work aside is a current SOTA, a large-scale real-time incremental
reconstruction model, DiFusion [5], which is based on Neural Implicit Maps.

First, we compare _NSLF models_ under our online learning framework (MANA) to
demonstrate the advances of our proposed model. The baselines are the recent
Deep Surface Light Field models DSLF [16], AFFM [17], and HashGrid (HG), an
SLF model migrated from NeRF based on Instant-NGP [24].

Then we evaluate the model and framework _as a whole_ to compare the
photometric performance with the latest incremental neural implicit
reconstruction SOTA, NICE-SLAM [12].

DSLF [16] | AFFM [17] | HG [24] | Ours | Ground Truth
---|---|---|---|---
|  |  |  |
|  |  |  |

Figure 5: Demonstration on ICL-NUIM dataset lrkt0n sequence.

TABLE I: Comparison on ICL-NUIM sequence lrkt0n. | DSLF [16] | AFFM [17] | HG [24] | Ours
---|---|---|---|---
PSNR $\uparrow$ | 21.74 | 21.29 | 28.46 | 28.73
SSIM $\uparrow$ | 0.836 | 0.846 | 0.908 | 0.915
LPIPSalex $\downarrow$ | 0.523 | 0.578 | 0.285 | 0.277
LPIPSvgg$\downarrow$ | 0.616 | 0.652 | 0.409 | 0.403

###  IV-C Implementation Details

TABLE II: PSNR comparison on Replica sequences. | Ofc0 | Ofc1 | Ofc2 | Ofc3 | Ofc4 | Rm0 | Rm1 | Rm2
---|---|---|---|---|---|---|---|---
HG | 38.12 | 37.63 | 28.38 | 26.64 | 35.02 | 30.06 | 33.51 | 34.37
Ours | 38.13 | 37.85 | 28.57 | 26.89 | 35.17 | 30.20 | 33.47 | 34.35

HG | Ours
---|---
|
|
|
|

Figure 6: Demonstration on unseen direction of observed surface.

The implementations of DSLF and AFFM are taken from [17]. For the HG baseline,
we use a multi-resolution hash grid with resolution=512 to encode position,
and spherical harmonics to encode direction [24]. The decoder operates on the
concatenated feature using 4 layers of MLPs with net-width=32. The sigmoid at
the end is used to normalize value in $[0,1]$ as normalized color. Our model
is also implemented with the same multi-resolution hash grid settings. Instead
of encoding directions, we learn spherical harmonics parameters and extract a
value directly in that direction. Then, the extracted latent is operated on
another MLPs with learned parameters at that position for color extraction.
For all models, learning rates are set to $lr=1e-3$ to train with the Adam
optimizer.

For online learning of growing scenes, we set the region scale to $4m$. Our
MAMA works alongside the recent 3D reconstruction models (DiFusion [5]) for
reconstruction support. The main thread processes the data and passes it to
reconstruction and MANA. In the MANA, the data_feeding function in the main
thread takes 0.01 s for each frame. The agents run asynchronously in their own
threading. On the sequence, we skip and take every 20th frame into DiFusion
and MANA. During the evaluation, we infer all frames.

For comparison, we use peak signal-to-noise ratio (PSNR), structural
similarity index measure (SSIM), and Learned Perceptual Image Patch Similarity
(LPIPS) to evaluate the produced image quality.

Experiments run on a computer with an AMD Ryzen 9 5950X 16-Core Processor CPU
and a GTX3090Ti GPU.

TABLE III: Full model comparison on Replica sequences. Header indicates scene names. |  | Office0 | Office1 | Office2 | Office3 | Office4 | Room0 | Room1 | Room2
---|---|---|---|---|---|---|---|---|---
NICE-SLAM [12] |  PSNR $\uparrow$ | 28.38 | 30.68 | 23.90 | 24.88 | 25.18 | 23.46 | 23.97 | 25.94
SSIM $\uparrow$ | 0.908 | 0.935 | 0.893 | 0.888 | 0.902 | 0.798 | 0.838 | 0.882
LPIPSalex $\downarrow$ | 0.386 | 0.278 | 0.330 | 0.287 | 0.326 | 0.443 | 0.401 | 0.315
LPIPSvgg$\downarrow$ | 0.455 | 0.403 | 0.433 | 0.405 | 0.430 | 0.496 | 0.486 | 0.451
DiFusion[5]+MANA |  PSNR $\uparrow$ | 28.59 | 26.70 | 21.10 | 21.89 | 25.74 | 23.24 | 25.68 | 24.88
SSIM $\uparrow$ | 0.913 | 0.879 | 0.863 | 0.847 | 0.893 | 0.816 | 0.883 | 0.888
LPIPSalex $\downarrow$ | 0.371 | 0.497 | 0.362 | 0.368 | 0.401 | 0.371 | 0.308 | 0.330
LPIPSvgg$\downarrow$ | 0.395 | 0.446 | 0.421 | 0.416 | 0.425 | 0.417 | 0.415 | 0.422
|  PSNR $\uparrow$ | 30.76 | 30.52 | 22.48 | 22.57 | 25.94 | 24.08 | 25.43 | 26.43
NICE-SLAM mesh |  SSIM $\uparrow$ | 0.942 | 0.927 | 0.891 | 0.872 | 0.911 | 0.819 | 0.879 | 0.904
+MANA inference |  LPIPSalex $\downarrow$ | 0.214 | 0.216 | 0.265 | 0.263 | 0.304 | 0.345 | 0.310 | 0.266
|  LPIPSvgg$\downarrow$ | 0.346 | 0.371 | 0.386 | 0.377 | 0.406 | 0.416 | 0.420 | 0.403

NICE-SLAM [12] | DiFusion[5]+MANA | NICE-SLAM mesh+MANA inf. | Ground Truth
---|---|---|---
|  |  |
|  |  |
|  |  |
|  |  |

Figure 7: Demonstration of full model comparison. The green box emphasized our
better texture. With the red box we shows the mesh difference between Di-
Fusion and NICE-SLAM.

###  IV-D Online NSLFs alongside Real-time 3D Reconstruction

We implement MANA alongside Real-time 3D reconstruction for Online NSLF. We
first run MANA and the reconstruction on the real-time sequence ICL-NUIM
lrkt0n. For a fair comparison, we should use the same mesh for all four
models. Since ICL-NUIM does not provide a ground-truth mesh, we pre-run
DiFusion to obtain the same mesh for all four NSLF models. Four NSLF models
are embedded respectively into MANA. Then, we sequentially feed those frames
into MANA. The real-time learning skips every 20 frames and the main thread
sleeps 1 s when every 20th frame arrives. The evaluation is done on all
frames.

Table I shows the photometric quantitative evaluation. DSLF and AFFM perform
much worse than HG and ours. Our model shows the best performance on all
metrics. We also find this in Fig. 5 that both DSLF and AFFM show blur or
incorrect image prediction. While HG and ours show a more realistic quality on
the painting and sofa.

We find that HG has results, very similar to ours. Therefore, we select HG and
our model for further testing and investigation on the Replica dataset. For
the experiment on Replica which provides a ground truth mesh, to mitigate
other effects, we use the ground truth mesh for prediction. Similarly, MANA
learns in real-time the sequences that skip every 20 frames and sleep 1 s
then. In Table II, HG and ours also provide very similar quality.

However, Table I and Table II are compared on the whole sequence, which is
very close to the selected trained frames. To better demonstrate the advantage
of our model, we show in Fig. 6 the unseen direction of the observed surface.
For HG, these surfaces show the correct color when we capture them in the
trained direction. But we find that HG gives arbitrary results for the unseen
direction of the observed surface. For example, in the first row, HG lost the
color of the painting and the sofa cushion. In the second and third rows, HG
shows arbitrary colors on the sofa and desk. In the fourth row, HG incorrectly
predicts the color of the carpet. This problem happens to HG because the novel
directions on those points are not learned. While our model works fine. Our
model learns a sphere on each point instead of just for the observed
directions.

To better support this, we further use object Chicken data from DSLF [16],
which gives a dense view of the object. We perform the experiment on the
Chicken test set (200 frames). We train both HG and ours using a single frame
(135th) for $1000$ iterations and render on all. The rendered video is
demonstrated in our project page222https://jarrome.github.io/NSLF-OL. Where HG
shows high color bias when viewing angle changes while our model gives a
better prediction.

From the video, most of the surface in the data is not trained. This means
that quantitative evaluation of the whole sequence can hardly find the result
differences. Therefore, we compute the angle between the view directions of
the inferences frame and the trained frame ($135$th). Then we threshold the
angle to count only the frame result in a certain range. In Table IV, we use
three options $\leq 15^{\circ}$, $\leq 30^{\circ}$, $\leq 60^{\circ}$. Where
we find ours are always better. In addition, the large angle causes inference
on the unlearned surfaces and will give a more similar score. Which is
revealed by the smaller gap in the larger angle threshold.

TABLE IV: PSNR comparison on object sequences. | $\leq 15^{\circ}$ | $\leq 30^{\circ}$ | $\leq 60^{\circ}$
---|---|---|---
HG | 26.97 | 22.87 | 21.45
Ours | 27.63 | 23.13 | 21.55

###  IV-E Comparing with Incremental Surface & Color Reconstruction

In the task of _real-time incremental reconstruction_ , surface and color are
not necessarily decoupled. For example, a recent Neural Implicit
Reconstruction SOTA, NICE-SLAM provides high-quality surface and color
reconstruction at the same time as online reconstruction. Therefore, we
compare it to the _SOTA of real-time incremental reconstruction_ to
demonstrate the advantages of our setting. Table III shows the quantitative
evaluation; we find that NICE-SLAM shows close performance to DiFusion
combined with our MANA. However, we find in Fig. 7 that this is not the case.
Our photometric result is more realistic compared to NICE-SLAM. The reason for
this phenomenon is that DiFusion’s reconstruction quality is worse than NICE-
SLAM, MANA score is weakened. For example, in the first two rows,
DiFusion+MANA shows a clear plot on the wall painting while NICE-SLAM shows a
very blurry result. Also in the third and fourth rows, MANA shows fine texture
on the quilt and carpet, while NICE-SLAM fails to find the detail. However,
DiFusion produces a worse reconstruction (shown in the edge of the tables and
chairs), which causes defects to the NSLF rendering.

To better demonstrate the advantage of our work along the reconstruction, we
use the same learned NSLFs to infer on NICE-SLAM’s mesh. This shows a better
result in most of the scenes of Table III. The specific detail is also shown
in the colored region of Fig. 7.

###  IV-F Time Efficiency

Real-time reconstruction while online learning NSLF on ICL-NUIM lrkt0n (1508
frames) sequence in Section IV-D takes about 28 s. This means that our online
learning framework incrementally trains this fixed amount of time. Rendering a
480$\times$640 image in ICL-NUIM data takes on average 0.07 s. Therefore, our
model easily achieves both real-time learning (training) and rendering
(inference) in a similar size.

##  V Conclusion

In this paper, we have proposed an online learning method for neural surface
light fields during real-time incremental 3D reconstruction on large scenes.

We have proposed a novel Neural Surface Light Fields model to address the
challenge that in a SLAM and reconstruction scenario the captured surface
directions are very limited, the learned model easily produces arbitrary
predictions from unseen directions.

For online learning in growing scenes where we do not pre-know the boundaries
in advance, we have designed Multiple Asynchronous Neural Agents to work
alongside real-time incremental 3D reconstruction.

The performance of the proposed method has been demonstrated in our
experiments. Our implementation achieves real-time learning of Neural Surface
Light Fields alongside real-time incremental reconstruction.

## References

  * [1] E. Bylow, J. Sturm, C. Kerl, F. Kahl, and D. Cremers, “Real-time camera tracking and 3d reconstruction using signed distance functions.” in _Robotics: Science and Systems_ , vol. 2, 2013, p. 2.
  * [2] M. Slavcheva, W. Kehl, N. Navab, and S. Ilic, “Sdf-2-sdf: Highly accurate 3d object reconstruction,” in _European Conference on Computer Vision_. Springer, 2016, pp. 680–696.
  * [3] Y. Yuan and A. Nüchter, “Indirect point cloud registration: Aligning distance fields using a pseudo third point set,” _IEEE Robotics and Automation Letters_ , 2022.
  * [4] M. Nießner, M. Zollhöfer, S. Izadi, and M. Stamminger, “Real-time 3d reconstruction at scale using voxel hashing,” _ACM Transactions on Graphics (ToG)_ , vol. 32, no. 6, pp. 1–11, 2013.
  * [5] J. Huang, S.-S. Huang, H. Song, and S.-M. Hu, “Di-fusion: Online implicit 3d reconstruction with deep priors,” in _Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 8932–8941.
  * [6] Y. Yuan and A. Nüchter, “An algorithm for the se (3)-transformation on neural implicit maps for remapping functions,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 3, pp. 7763–7770, 2022.
  * [7] A. Tewari, J. Thies, B. Mildenhall, P. Srinivasan, E. Tretschk, W. Yifan, C. Lassner, V. Sitzmann, R. Martin-Brualla, S. Lombardi _et al._ , “Advances in neural rendering,” in _Computer Graphics Forum_ , vol. 41, no. 2. Wiley Online Library, 2022, pp. 703–735.
  * [8] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” in _European Conf. on computer vision_. Springer, 2020, pp. 405–421.
  * [9] M. Tancik, V. Casser, X. Yan, S. Pradhan, B. Mildenhall, P. P. Srinivasan, J. T. Barron, and H. Kretzschmar, “Block-nerf: Scalable large scene neural view synthesis,” in _Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 8248–8258.
  * [10] X. Zhang, S. Bi, K. Sunkavalli, H. Su, and Z. Xu, “Nerfusion: Fusing radiance fields for large-scale scene reconstruction,” in _Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5449–5458.
  * [11] E. Sucar, S. Liu, J. Ortiz, and A. J. Davison, “imap: Implicit mapping and positioning in real-time,” in _Proc. of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 6229–6238.
  * [12] Z. Zhu, S. Peng, V. Larsson, W. Xu, H. Bao, Z. Cui, M. R. Oswald, and M. Pollefeys, “Nice-slam: Neural implicit scalable encoding for slam,” in _Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 12 786–12 796.
  * [13] K. Zhang, G. Riegler, N. Snavely, and V. Koltun, “Nerf++: Analyzing and improving neural radiance fields,” _arXiv preprint arXiv:2010.07492_ , 2020\.
  * [14] D. N. Wood, D. I. Azuma, K. Aldinger, B. Curless, T. Duchamp, D. H. Salesin, and W. Stuetzle, “Surface light fields for 3d photography,” in _Proc. of the 27th annual Conf. on Computer graphics and interactive techniques_ , 2000, pp. 287–296.
  * [15] E. Miandji, J. Kronander, and J. Unger, “Learning based compression of surface light fields for real-time rendering of global illumination scenes,” in _SIGGRAPH Asia 2013 Technical Briefs_ , 2013, pp. 1–4.
  * [16] A. Chen, M. Wu, Y. Zhang, N. Li, J. Lu, S. Gao, and J. Yu, “Deep surface light fields,” _Proc. of the ACM on Computer Graphics and Interactive Techniques_ , vol. 1, no. 1, pp. 1–17, 2018.
  * [17] H. Yu, A. Chen, X. Chen, L. Xu, Z. Shao, and J. Yu, “Anisotropic fourier features for neural image-based rendering and relighting,” 2022.
  * [18] C. Reiser, S. Peng, Y. Liao, and A. Geiger, “Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps,” in _Proc. of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 14 335–14 345.
  * [19] L. Liu, J. Gu, K. Zaw Lin, T.-S. Chua, and C. Theobalt, “Neural sparse voxel fields,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 15 651–15 663, 2020.
  * [20] T. Neff, P. Stadlbauer, M. Parger, A. Kurz, J. H. Mueller, C. R. A. Chaitanya, A. Kaplanyan, and M. Steinberger, “Donerf: Towards real-time rendering of compact neural radiance fields using depth oracle networks,” in _Computer Graphics Forum_ , vol. 40, no. 4. Wiley Online Library, 2021, pp. 45–59.
  * [21] D. B. Lindell, J. N. Martel, and G. Wetzstein, “Autoint: Automatic integration for fast neural volume rendering,” in _Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 14 556–14 565.
  * [22] A. Yu, R. Li, M. Tancik, H. Li, R. Ng, and A. Kanazawa, “Plenoctrees for real-time rendering of neural radiance fields,” in _Proc. of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 5752–5761.
  * [23] P. Hedman, P. P. Srinivasan, B. Mildenhall, J. T. Barron, and P. Debevec, “Baking neural radiance fields for real-time view synthesis,” in _Proc. of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 5875–5884.
  * [24] T. Müller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” _arXiv preprint arXiv:2201.05989_ , 2022.
  * [25] A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su, “Tensorf: Tensorial radiance fields,” _arXiv preprint arXiv:2203.09517_ , 2022.
  * [26] V. Sitzmann, S. Rezchikov, B. Freeman, J. Tenenbaum, and F. Durand, “Light field networks: Neural scene representations with single-evaluation rendering,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 19 313–19 325, 2021.
  * [27] K. Deng, A. Liu, J.-Y. Zhu, and D. Ramanan, “Depth-supervised nerf: Fewer views and faster training for free,” _arXiv preprint arXiv:2107.02791_ , 2021\.
  * [28] Q. Xu, Z. Xu, J. Philip, S. Bi, Z. Shu, K. Sunkavalli, and U. Neumann, “Point-nerf: Point-based neural radiance fields,” _arXiv preprint arXiv:2201.08845_ , 2022.
  * [29] A. Chen, Z. Xu, F. Zhao, X. Zhang, F. Xiang, J. Yu, and H. Su, “Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo,” in _Proc. of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 14 124–14 133.
  * [30] J. Zhang, Y. Yao, and L. Quan, “Learning signed distance field for multi-view surface reconstruction,” in _Proc. of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 6525–6534.
  * [31] D. Azinović, R. Martin-Brualla, D. B. Goldman, M. Nießner, and J. Thies, “Neural rgb-d surface reconstruction,” _arXiv preprint arXiv:2104.04532_ , 2021.
  * [32] A. Handa, T. Whelan, J. McDonald, and A. J. Davison, “A benchmark for rgb-d visual odometry, 3d reconstruction and slam,” in _2014 ICRA_.
  * [33] J. Straub, T. Whelan, L. Ma, Y. Chen, E. Wijmans, S. Green, J. J. Engel, R. Mur-Artal, C. Ren, S. Verma _et al._ , “The replica dataset: A digital replica of indoor spaces,” _arXiv preprint arXiv:1906.05797_ , 2019\.


# Occlusion-Resistant LiDAR Fiducial Marker Detection

Yibo Liu, Jinjun Shan, Hunter Schofield This work was supported in part by
NSERC Alliance Program under Grant ALLRP 555847-20, and in part by Mitacs
Accelerate Program under Grant IT26108.The authors are with Department of
Earth and Space Science and Engineering, York University, Toronto, Ontario M3J
1P3, Canada<EMAIL_ADDRESS>open-source implementation
of this work is available at: https://github.com/York-SDCNLab/Marker-
Detection-General.

###### Abstract

The LiDAR fiducial marker, akin to the well-known AprilTag used in camera
applications, serves as a convenient resource to impart artificial features to
the LiDAR sensor, facilitating robotics applications. Unfortunately, current
LiDAR fiducial marker detection methods are limited to occlusion-free point
clouds. In this work, we present a novel approach for occlusion-resistant
LiDAR fiducial marker detection. We first extract 3D points potentially
corresponding to the markers, leveraging the 3D intensity gradients.
Afterward, we analyze the 3D spatial distribution of the extracted points
through clustering. Subsequently, we determine the potential marker locations
by examining the geometric characteristics of these clusters. We then
successively transfer the 3D points that fall within the candidate locations
from the raw point cloud onto a designed intermediate plane. Finally, using
the intermediate plane, we validate each location for the presence of a
fiducial marker and compute the marker’s pose if found. We conduct both
qualitative and quantitative experiments to demonstrate that our approach is
the first LiDAR fiducial marker detection method applicable to point clouds
with occlusion while achieving better accuracy.

##  I INTRODUCTION

The visual fiducial marker, such as AprilTag [1], ArUco [2], and CCTag [3],
have been widely utilized in many robotic applications, including SLAM [4, 5],
multi-sensor calibration [6], autonomous navigation [7, 8, 9, 10], object
tracking [11, 12], and Augmented Reality (AR) [13] as they can provide
artificial features (a.k.a. fiducials) for the camera.

The development of the LiDAR fiducial marker is also crucial, as it fulfills
fundamental needs for feature extraction and matching in diverse LiDAR
applications [14, 15, 16]. However, detecting LiDAR fiducials is a challenging
task since the 3D point cloud is unordered and sparse compared to a camera
image [17]. While there have been some pioneering LiDAR fiducial marker
detection methods, they require an occlusion-free point cloud as input, making
them incapable of handling the fiducial marker detection problem in a scenario
depicted in Fig. 1. Resolving this problem is favorable to downstream tasks,
such as relocalization [18] and navigation [7, 19].

To this end, we develop an occlusion-resistant LiDAR fiducial marker detection
method in this work. In particular, our method first conducts downsampling on
the raw point cloud using the 3D intensity gradients to locate objects with
high-intensity contrast. Then, we cluster the intensity features and analyze
whether a cluster is a potential fiducial marker from a geometric perspective,
according to the characteristics of a planar marker. Finally, we check whether
each preserved location of the raw point cloud contains a fiducial marker
through a designed intermediate plane and calculate the pose if found. We
carry out experiments on both synthetic and real-world scenes against the
state-of-the-art methods to demonstrate that our method is the first LiDAR
fiducial marker detection method capable of dealing with point clouds with
occlusion while achieving better accuracy.

The main contributions of this work are as follows:

  * •

We propose a novel LiDAR fiducial marker detection method specifically for
point clouds with occlusion, which is a problem that cannot be resolved by the
existing approaches [20, 17, 21].

  * •

We design a new feature extraction pipeline that jointly leverages both 3D
intensity features and 3D geometric features, considering that fiducial
markers are planar objects with high-intensity contrast and are non-
distinguishable from the plane to which they are attached. This differs from
conventional 3D object detection methods [22, 23, 24, 25, 26, 27] that rely
solely on 3D geometric features and can only detect spatially distinguishable
objects.

  * •

We conduct thorough experiments qualitatively and quantitatively to
demonstrate the superiority of our approach over the existing methods [20, 21,
17]. Moreover, we release the open-source implementation based on C++ to the
community.

Figure 1:  An example of a point cloud with occlusion. The points are rendered
by intensity values through the color map [28]. The point cloud is generated
by the modified livox-mapping SLAM framework [29, 30]. As shown in the zoomed
view 1, there is an ArUco marker [2] (family: original; ID: 0) placed in front
of the shutter door. As seen in the zoomed view 2, occlusion exists seeing
from the origin of the point cloud towards the fiducial marker.

##  II Related Work

3D Object Detection. 3D object detection methods can be categorized into two
groups: conventional approaches [22, 23, 24, 25, 26] and deep-learning-based
approaches [27, 31, 32, 33, 34]. The conventional methods utilize mathematical
and geometric methods to segment 3D objects from the scene, including edge-
based [22], region-based [23], attributes-based [24], model-based [25], and
graph-based [26] methods. The deep-learning-based approaches [27, 31, 32, 33,
34] adopt elaborately designed neural networks to detect specific 3D objects,
such as vehicles, pedestrians, desks, chairs, etc. These methods rely on the
assumption that objects are spatially distinguishable from each other, which
means they cannot effectively detect objects that lack such spatial
distinction. For instance, detecting a thin sheet of paper attached to a wall
poses a challenge for these approaches.

LiDAR Fiducial Marker Detection. LiDAR fiducial marker is inherently a 3D
object. Therefore, LiDARTag [17] and A4LiDARtag [21] follow conventional 3D
object detection methods [22, 26] to develop fiducial markers for LiDAR
sensors. However, to guarantee that fiducial markers have spatial distinction,
their fiducial markers have to be additional 3D objects added to the
environment. In contrast, the Intensity Image-based LiDAR Fiducial Marker
(IILFM) [20] proposes to utilize the intensity image and an embedded 2D marker
detector to localize LiDAR fiducial markers. Thus, adding a marker does not
affect the 3D environment, as the marker is simply a thin sheet of paper
attached to a plane. In this work, LiDAR fiducial marker refers to this type
of marker. Furthermore, a few works [35, 36, 37] adopt high-reflectivity
targets as fiducials for LiDAR-camera calibration purposes. These approaches
cannot assign each fiducial a unique identification number, as is done with
the AprilTag [1], LiDARTag [17], and IILFM [20]. Despite the proposal of the
aforementioned methods [17, 20, 21], LiDAR fiducial marker detection in a
point cloud with occlusion remains an unresolved problem as they all require
an occlusion-free single-view point cloud as the input.

##  III Methodology

###  III-A Preliminaries

In this work, we aim for the LiDAR fiducial markers to be sheets of thin paper
and not to affect the 3D environment, akin to the IILFM [20]. Thus, we choose
IILFM as the baseline model. In this section, we introduce preliminaries
regarding IILFM to further explain why occlusion is a fatal hindrance in this
method.

IILFM first projects the 3D points of the raw point cloud into a 2D image
plane to construct the intensity image and then adopts the 2D marker detector
to find 2D fiducials on it. Finally, the detected 2D fiducials are projected
back into 3D space. In particular, the intensity image is generated by
transforming each 3D point in the raw point cloud into a 2D pixel on an image
plane:

$u=\lceil\frac{\theta}{\Theta_{a}}\rfloor+u_{o},\;v=\lceil\frac{\phi}{\Theta_{i}}\rfloor+v_{o}$
(1)

where $[u,v]^{T}$ are the image coordinates of the projected pixel.
$[\theta,\phi,r]^{T}$ are the spherical coordinates of the 3D point.
$\Theta_{a}$ and $\Theta_{i}$ are the angular resolutions in $u$ (azimuth) and
$v$ (inclination) directions, respectively. $u_{o}$ and $v_{o}$ are the
offsets. Apparently, for any two points sharing the same $[\theta,\phi]^{T}$,
they will overlap for the same pixel. This indicates that occlusion ruins the
intensity image due to pixels overlapping.

###  III-B Downsampling Based on 3D Intensity Gradients

As declared above, the fiducial markers in this work refer to sheets of thin
paper that are not spatially distinguishable from the attached planes. Thus,
it is infeasible to adopt previous geometric features-based 3D object
detection methods [22, 23, 24, 25, 26, 27, 31, 32, 33, 34] to find the
fiducial markers.

Despite this, a fiducial marker is composed of a black-and-white pattern and,
as a result, presents as a high-intensity contrast object in the view of a
LiDAR (see the zoomed view 1 of Fig. 1). This indicates that we can first
analyze the point cloud from the intensity perspective. In particular, we
conduct downsampling on the raw point cloud based on the 3D intensity
gradients. We take the intensity as a function $I(\mathbf{x})$ of the 3D
coordinates $\mathbf{x}=[x,y,z]^{T}$. Suppose that the given 3D point/location
is $\textbf{p}_{0}$, we define the point set composed of the neighbouring $n$
points around $\textbf{p}_{0}$ as
$\mathcal{P}_{I}=\\{\textbf{p}_{1},\textbf{p}_{2},\cdots,\textbf{p}_{n}\\}$.
In practice, we use a linear model, $\hat{I}(\mathbf{x})$, to approximate
$I(\mathbf{x})$:

$\hat{I}(\mathbf{x})=\mathbf{A}^{T}\mathbf{x}+b$ (2)

where $\mathbf{A}\in\mathbb{R}^{3\times 1}$ and $b=\bar{I}$.
$\bar{I}\in\mathbb{R}$ is the mean of the intensity values of the points in
$\mathcal{P}_{I}$. Since the intensity values of points in $\mathcal{P}_{I}$
are known, we can estimate $\mathbf{A}$ by resolving the following model fit
(least square) problem:

$\underset{\mathbf{A}}{\arg\min}\sum_{i=1}^{n}\left\|\hat{I}(\mathbf{x}_{i})-I(\mathbf{x}_{i})\right\|^{2}$
(3)

The detailed calculation is given in our open-source code implementation. Once
$\hat{I}(\mathbf{x})$ is obtained, we can compute the 3D intensity gradients
$\nabla I\in\mathbb{R}^{3\times 1}$ at $\textbf{p}_{0}$. The direction of
$\nabla I$ of a given point indicates the direction where the intensity has
the fastest decline and the norm, $|\nabla I|$, implies the rate of descent.
Inspired by LOAM [15, 38], which selects a point as a feature if its geometric
curvature is larger than a threshold, we preserve the 3D point if its $|\nabla
I|$ is larger than a threshold in the downsampling procedure.

###  III-C Spatial Distribution Analysis of Downsampling Result

The downsampling preserves the points belonging to the outlines of objects
with high intensity-contrast, such as stop signs, notices on the wall, and
fiducial markers. In this section, we analyze the point cloud from the
geometric perspective. The foundation of doing so is the fact that the points
belonging to the fiducial markers will be isolated from those of the other
objects after the downsampling (see the zoomed view of Fig. 7(a)). To explain
the isolation of the points, we need to review the design of the marker’s
pattern as shown in Fig. 2.

Figure 2:  A diagram to illustrate the design of a typical square fiducial
marker [1]. A square fiducial marker is a combination of the prototype marker
(a black frame inside a white frame) and the encoding area.

In the visual fiducial marker system, such as AprilTag [1], the aim of
adopting the prototype marker (the black-white protector frame) is to isolate
the marker from the environment and to construct a square shape that can be
easily recognized by the visual quad detector. Despite the fact that the
design of the prototype marker does not consider any factors related to LiDAR,
we found that the prototype marker also brings an important benefit to this
LiDAR application: the white regions of the marker naturally have higher
intensity values than the black regions, and thus, the prototype marker after
the downsampling is rendered as a square double-ring that isolates the points
inside the coding area from the environment. We utilize this benefit to
develop the followed-up pipeline. The isolation makes the points belonging to
the markers spatially distinguishable from those of the other objects. Thus,
we employ the method introduced in [39] to further segment the downsampling
result into clusters. Each cluster is represented by an oriented bounding box
(OBB) as shown in Fig. 7(b).

###  III-D Geometric Characteristic Analysis of Clustering Result

In this section, we analyze the geometric characteristic of each cluster to
verify if it has the potential to be a LiDAR fiducial marker. Specifically,
the bounding box of a cluster needs to satisfy two criteria to be recognized
as a candidate that could contain a LiDAR fiducial marker. The first criterion
is subject to the marker size:

$\sqrt{2a^{2}+\delta^{2}}\leq L_{OBB}\leq\sqrt{4a^{2}+\delta^{2}}$ (4)

where $L_{OBB}=\sqrt{l^{2}+w^{2}+h^{2}}$ is the cuboid diagonal of the OBB
with $l$, $w$, and $h$ ($h\leq\delta$) being the length, width, and height,
respectively. $a$ denotes the side length of the marker. $\delta$ is the
trifling thickness of the marker.

The second criterion, as shown in Eq. (5), is based on the fact that the shape
of the marker is square:

$\frac{1}{1.5}\leq\frac{l}{w}\leq 1.5$ (5)

We explain these two criteria as follows. The OBB is the minimum 3D box that
encloses the cluster where its sides that correspond to the length, width, and
height are parallel to the first, second, and third primary components (PCs)
of the feature point cluster. Moreover, the height is trivial compared to the
length and width since the marker is a planar object. Thus, we neglect the
height for now and consider the possible size range of the OBB in 2D space.
The possible size of the OBB for a given marker is shown in Fig. 3. In light
of the definition of the OBB [40], we know that AC $\perp$ AE and AE $\perp$
AF. In this work, we assume that the side length of the marker, $a$, is given.
It is illustrated in Fig. 3 that BC = $a\mathrm{cos}\theta$ and DC =
$a\mathrm{sin}\theta$. Thus, AB = DC, such that AC =
$a(\mathrm{cos}\theta+\mathrm{sin}\theta)$. Similarly, AE = EF = CF =
$a(\mathrm{cos}\theta+\mathrm{sin}\theta)$. At this point, we conclude that:
1) the OBB in this case is a square; 2) the side length of the OBB is
$a(\mathrm{cos}\theta+\mathrm{sin}\theta)$. Given that $\theta\in[0,2\pi]$, we
have $a(\mathrm{cos}\theta+\mathrm{sin}\theta)\in[a,\sqrt{2}a]$. In practice,
the orientations of the first and second PCs are subject to the feature points
distribution, which is determined by the unique black-and-white pattern of the
marker (the encoding area as shown in Fig. 2). Although the pattern cannot be
known in advance, the cases where the OBB reaches the smallest and largest
sizes (see Fig. 4) are already found according to the proof above.

Figure 3:  A diagram of the possible OBB size for a given marker.
$O_{M}-X_{M}-Y_{M}$ is the marker coordinate system. $\theta$ is the angle
between the first PC and the x-axis. On account of the fact that the first PC
and second PC are perpendicular to each other, the angle between the second PC
and the y-axis is $\theta$ and $\angle DBC=\theta$ as well. Figure 4: A rough
schematic of the smallest (a) and the largest (b) OBBs. (a): 1st PC
$\parallel$ $s_{1}$ and 2nd PC $\parallel$ $s_{2}$ where $s_{1}$ and $s_{2}$
are the horizontal side and vertical sides of the marker, respectively. (b):
1st PC $\parallel$ $d_{1}$ and 2nd PC $\parallel$ $d_{2}$ where $d_{1}$ and
$d_{2}$ being the two diagonals of the marker.

In summary, if the side length of the marker is $a$, the area of the 2D OBB,
$S_{OBB}$, is in the following range:

$a^{2}\leq S_{OBB}\leq 2a^{2}$ (6)

Taking the marker’s thickness, $\delta$, into consideration as well, we can
obtain the first criterion as shown in Eq. (4). In addition, the OBB
projection on the plane of length and width is square. Thus, Eq. (5) is
adopted as the second criterion. Ideally, $l\approx w$ since the marker is a
square. Whereas in the real world, the marker cannot be perfectly scanned by
LiDAR, and LiDAR also has ranging noise. As a result, the shape of the OBB on
the length and width plane could be distorted. Hence, we make the requirement
on $l\approx w$ less strict as shown in Eq. (5).

###  III-E Marker Detection via Intermediate Plane

After analyzing the point cloud from the 3D intensity gradients and 3D
geometric perspectives, we obtain the locations of objects with high
intensity-contrast and shapes/sizes similar to the fiducial marker. We then
inspect these locations (OBBs) in the raw point cloud. When extracting points
falling into the OBBs, a buffer is adopted to extend the OBBs, preserving more
regions around an OBB in case it does not completely cover the fiducial
marker. This indicates that, for each OBB, the pose (position and
orientation), $\mathbf{T}_{OBB}$ 111$\mathbf{T}_{OBB}$ denotes the
transmission from the global coordinate system $\\{\mathbf{G}\\}$ to the OBB
frame. The OBB frame refers to a coordinate system whose X, Y, and Z axes are
parallel to the length, width, and height of the bounding box. The origin of
the OBB frame is located at the center of the bounding box. , is kept while
the size is enlarged by multiplying the length, width, and height with an
amplification factor, $t_{b}$. The recommended value of $t_{b}$ is twice the
marker’s side length based on our experiment.

Figure 5: An illustration of the intermediate plane.

Given that the pose of each OBB ($\mathbf{T}_{OBB}$) is acquired from the
analysis in Sections III-B, III-C, and III-D, we propose to eliminate the
effect of occlusion by transferring the points extracted from each OBB to an
intermediate plane (denoted by $\mathbf{P}$) as shown in Fig. 5. The detailed
transmission process is as follows. Define the point set of the $j$th OBB as
$\mathcal{P}_{j}$ and a point of $\mathcal{P}_{j}$ as
$\textbf{p}_{j}\in\mathbb{R}^{3}$. $\textbf{p}_{j}$ is first transmitted to
the origin of the global coordinate system $\\{\mathbf{G}\\}$ through the
inverse of $\mathbf{T}_{OBB}$:

$\textbf{p}_{G}=\mathbf{T}_{OBB}^{-1}\cdot\textbf{p}_{j}=\mathbf{R}^{-1}\cdot\textbf{p}_{j}-\mathbf{R}^{-1}\mathbf{t}$
(7)

where $\textbf{p}_{G}$ is the point transmitted to the origin of
$\\{\mathbf{G}\\}$. $\mathbf{T}_{OBB}=[\mathbf{R}|\mathbf{t}]$ where
$\mathbf{R}$ is the $3\times 3$ orthogonal rotation matrix and $\mathbf{t}$ is
the $3\times 1$ translation vector. After transmitting all the points in
$\mathcal{P}_{j}$ using Eq. (7), we obtain a new point set
$\mathcal{P}^{G}_{j}$. Then, all points belonging to $\mathcal{P}^{G}_{j}$ are
transferred to the intermediate plane $\mathbf{P}$:

$\textbf{p}^{\prime}_{j}=\mathbf{T}_{in}\cdot\textbf{p}_{G}=\mathbf{R}_{in}\cdot\textbf{p}_{G}+\mathbf{t}_{in}\\\
$ (8)

where $\textbf{p}^{\prime}_{j}$ is the point transmitted to the intermediate
plane. $\mathbf{T}_{in}=[\mathbf{R}_{in}|\mathbf{t}_{in}]$ where
$\mathbf{R}_{in}=\mathbf{I}_{3\times 3}$ is an identity matrix and
$\mathbf{t}_{in}=[1\mathrm{m},0,0]^{T}$ since the plane equation is
$x=1\mathrm{m}$. Define the point set on the intermediate plane as
$\mathcal{P}^{\prime}_{j}$. Given that $\mathcal{P}^{\prime}_{j}$ is a point
cloud with no occlusions, marker detection in it is straightforward following
the preliminary method introduced in Section III-A. Define the point set
composed of the detected 3D fiducials as $\mathcal{F}_{j}$. If the set
$\mathcal{F}{j}$ exists, we can transmit the points in $\mathcal{F}{j}$ from
the intermediate plane back to their original locations using the reverse
processes of Eqs. (7-8).

##  IV Experimental Results

In this section, we present experimental results comparing the proposed
approach to existing LiDAR fiducial marker systems, including IILFM [20],
LiDARTag [17], and A4LiDARTag [21]. In Section IV-A, we test the approaches on
both synthesis and real-world point clouds to demonstrate that only our method
is applicable to a point cloud with occlusion. In Section IV-B, we demonstrate
quantitatively that our method has better pose estimation accuracy.

###  IV-A Qualitative Evaluation

We first compare the existing methods [20, 17, 21] with our approach on a
synthesis point cloud (see Fig. 6), which is a stitch of two subpoint clouds.
We want to use this simple scene to explicitly present the visual results of
the main steps of the proposed method. Then, we compare the approaches on the
real-world 3D map shown in Fig. 1. The visual results after completing the
major steps of the proposed approach are shown in Fig. 7 and Fig. 8. Please
refer to the caption for details.

Figure 6:  A synthesis point cloud with occlusion. The two presenters are
holding two different AprilTag markers [1]. As illustrated in the top view,
observing along the X-axis of the global coordinate system $\\{\mathbf{G}\\}$,
the back subpoint cloud is totally blocked by the front one. Thus, it is
infeasible to project the 3D point cloud to a 2D image plane due to the
occlusion. Figure 7:  Visual results of the main steps. The raw point cloud is
shown in Fig. 6. (a): The point cloud after downsampling using 3D intensity
gradients. (b): Clustering of the downsampling result. Each cluster is labeled
with an OBB. (c): Preserved OBBs according to the proposed criterion. (d):
Points extracted from the raw point cloud and the points transmitted to the
intermediate plane. (e): LiDAR fiducial markers detection result. The detected
vertices are rendered in red. Figure 8:  An illustration of the marker
detection process on the 3D map built from SLAM. (a): The result after
conducting downsampling and clustering. (b): The result after adopting the
proposed geometric criterion. As seen, all the clusters but the one belonging
to the marker are filtered out. (c): The zoomed view of the preserved OBB.
(d): A zoomed view of the detected 3D fiducials, which are rendered in red.
TABLE I: Comparison with the previous methods on general point clouds. GT
refers to ground truth. Num denotes the number of LiDAR fiducial markers.

Scene | GT Num | Method | Detected Num
---|---|---|---
Fig. 6 | 2 | LiDARTag [17] | 0
A4LiDARTag [21] | 0
IILFM [20] | 1
Ours | 2
Fig. 1 | 1 | LiDARTag [17] | 0
A4LiDARTag [21] | 0
IILFM [20] | 0
Ours | 1

TABLE II: Comparison of our approach, IILFM [20], and AprilTag3 [1] with respect to pose estimation accuracy. The distance refers to the distance from the fiducial marker to the LiDAR. Distance | Method | X error (m) | Y error (m) | Z error (m) | Roll error (deg) | Pitch error (deg) | Yaw error (deg)
---|---|---|---|---|---|---|---
2m | AprilTag3 [1] | 0.016 | 0.034 | 0.016 | 0.807 | 3.166 | 1.244
IILFM [20] | 0.003 | 0.006 | 0.017 | 0.423 | 0.457 | 0.399
Ours | 0.002 | 0.005 | 0.011 | 0.315 | 0.305 | 0.391
3m | AprilTag3 [1] | 0.058 | 0.124 | 0.044 | 1.369 | 7.963 | 2.904
IILFM [20] | 0.026 | 0.021 | 0.093 | 0.930 | 1.182 | 0.859
Ours | 0.006 | 0.009 | 0.015 | 0.343 | 0.322 | 0.455
4m | AprilTag3 [1] | 0.072 | 0.407 | 0.233 | 1.433 | 9.292 | 13.343
IILFM [20] | 0.024 | 0.024 | 0.107 | 0.342 | 2.020 | 1.111
Ours | 0.008 | 0.014 | 0.016 | 0.302 | 0.389 | 0.478

The summary of the qualitative evaluation is presented in Table LABEL:tab1. As
seen in the Table, only the proposed approach successfully detects all the
fiducials in the two scenes. We take the point cloud shown in Fig. 6 as an
example to analyze the reasons for the detection failures of the existing
methods. LiDARTag [17] supports patterns from AprilTag3 [1], however, it
requires adequate clearance around the markers. In our scenes, markers are
papers attached to other planes with no clearance. A4LiDARTag [21] proposes to
utilize the range image to detect fiducials. Whereas, as shown in the range
image of Fig. 9, there is no useful information in it. IILFM [20] projects the
raw point cloud onto the image plane directly. Thus, only the marker in the
front point cloud is preserved in the intensity image due to occlusion (see
the intensity image in Fig. 9). Therefore, only one marker is detected. By
contrast, our approach generates an intensity image for each location that
could contain a fiducial marker. Thus, as shown in Fig. 9, each marker is
preserved in its corresponding intensity image.

Figure 9:  (a): The 2D intensity/range images after projecting the 3D point
cloud shown in Fig. 6 onto the 2D image plane. (b): The intensity images
obtained through our approach from the point cloud shown in Fig. 6.

###  IV-B Quantitative Evaluation

In this section, we conduct experiments introduced in Fig. 10 to demonstrate
that the proposed method has better pose estimation accuracy. Since we want to
compare our method with other approaches in terms of pose estimation accuracy,
it is necessary to use this occlusion-free scene because other approaches are
not applicable to point clouds with occlusion.

Figure 10:  The experimental setup for quantitative evaluation. The LiDAR
model is Livox MID-40. A Letter size AprilTag is put on the wall. The
OptiTrack system provides the ground truth of the 6-DOF LiDAR pose.

We test the methods under different distances from the LiDAR to the fiducial
marker. Unfortunately, LiDARTag [17] and A4LiDARTag [21] fail to detect the
marker, and thus, we compare the proposed approach with IILFM [20] and the
widely-used visual fiducial marker system, AprilTag3 [1]. Note that when
testing AprilTag3, the adopted sensor is a camera. As shown in Table II, both
the accuracy of IILFM and AprilTag3 degrades as the distance from the sensor
to the marker increases while the accuracy degradation of IILFM is less serve
compared to AprilTag3. The accuracy of the proposed approach is slightly
better than IILFM at a distance of 2m. However, unlike IILFM, our approach
maintains a decent accuracy when the distance increases.

Figure 11:  An illustration of the negative correlation between the projection
size and the distance. It is straightforward to prove the negative correlation
using the similarity of triangles. Figure 12:  The visual comparison of the
intensity images obtained by our method and IILFM [20] from the same point
cloud (shown on the right side) when the distance from the marker to the LiDAR
is 3m.

The analysis of the result is given as follows. Suppose that both IILFM [20]
and our approach are employed to process the same point cloud where the
distance from the fiducial marker to the LiDAR is 3m. IILFM [20] projects the
entire raw point cloud to the image plane directly without any analysis. By
contrast, our approach first transforms the points potentially corresponding
to the marker to the intermediate plane and then projects them to the image
plane. Based on the negative correlation between the projection size and
distance, as illustrated in Fig. 11, our method results in a larger marker
projection on the image plane than IILFM [20]. The visual comparison is given
in Fig. 12. Apparently, using our approach, the intensity image has fewer
points overlapping and thus retains more information. This is favorable for
pose estimation as it leads to more accurate observations.

##  V Conclusions

In this work, we propose a novel occlusion-resistant LiDAR fiducial marker
detection method. Despite the existence of previous LiDAR fiducial marker
detection methods [20, 17, 21], our method is the first that can detect the
LiDAR fiducial markers in a point cloud with occlusion and return their poses
with respect to the global frame. Qualitative and quantitative experiments are
presented to demonstrate the superiority of the proposed approach over the
existing methods.

## ACKNOWLEDGMENT

The authors would like to thank Yicong Fu, Guile Wu, Han Wang, Brian Lynch,
Yuan Ren, Shuo Zhang, Marc Savoie, and Shiyuan Jia for insightful discussions.

## References

  * [1] M. Krogius, A. Haggenmiller, and E. Olson, “Flexible layouts for fiducial tags,” in _Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2019, pp. 1898–1903.
  * [2] F. J. Romero-Ramirez, R. Muñoz-Salinas, and R. Medina-Carnicer, “Speeded up detection of squared fiducial markers,” _Image and vision Computing_ , vol. 76, pp. 38–47, 2018.
  * [3] L. Calvet, P. Gurdjos, C. Griwodz, and S. Gasparini, “Detection and Accurate Localization of Circular Fiducials under Highly Challenging Conditions,” in _Proc. 2016 IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 562 – 570.
  * [4] R. Muñoz-Salinas, M. J. Marin-Jimenez, and R. Medina-Carnicer, “Spm-slam: Simultaneous localization and mapping with squared planar markers,” _Pattern Recognition_ , vol. 86, pp. 156–171, 2019.
  * [5] S. Zhang, J. Shan, and Y. Liu, “Variational bayesian estimator for mobile robot localization with unknown noise covariance,” _IEEE/ASME Transactions on Mechatronics_ , pp. 1–9, 2022.
  * [6] X. Liu, H. Madhusudanan, W. Chen, D. Li, J. Ge, C. Ru, and Y. Sun, “Fast eye-in-hand 3-d scanner-robot calibration for low stitching errors,” _IEEE Transactions on Industrial Electronics_ , vol. 68, no. 9, pp. 8422–8432, 2021.
  * [7] Y. Liu, H. Schofield, and J. Shan, “Navigation of a self-driving vehicle using one fiducial marker,” in _Proc. IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems_ , 2021, pp. 1–6.
  * [8] L. Zhang, F. Deng, J. Chen, Y. Bi, S. K. Phang, X. Chen, and B. M. Chen, “Vision-based target three-dimensional geolocation using unmanned aerial vehicles,” _IEEE Transactions on Industrial Electronics_ , vol. 65, no. 10, pp. 8052–8061, 2018.
  * [9] J. Lin, Z. Miao, H. Zhong, W. Peng, Y. Wang, and R. Fierro, “Adaptive image-based leader–follower formation control of mobile robots with visibility constraints,” _IEEE Transactions on Industrial Electronics_ , vol. 68, no. 7, pp. 6010–6019, 2021.
  * [10] Y. Wang, M. Shan, Y. Yue, and D. Wang, “Autonomous target docking of nonholonomic mobile robots using relative pose measurements,” _IEEE Transactions on Industrial Electronics_ , vol. 68, no. 8, pp. 7233–7243, 2021\.
  * [11] Y. Zhu, C. Yang, Z. Tu, Y. Ling, and Y. Chen, “A haptic shared control architecture for tracking of a moving object,” _IEEE Transactions on Industrial Electronics_ , vol. 70, no. 5, pp. 5034–5043, 2023.
  * [12] L. Zheng, H. Wu, L. Yang, Y. Lao, Q. Lin, and R. Yang, “A novel respiratory follow-up robotic system for thoracic-abdominal puncture,” _IEEE Transactions on Industrial Electronics_ , vol. 68, no. 3, pp. 2368–2378, 2021\.
  * [13] D. Avola, L. Cinque, G. L. Foresti, C. Mercuri, and D. Pannone, “A practical framework for the development of augmented reality applications by using aruco markers,” in _Proc. International Conference on Pattern Recognition Applications and Methods_ , vol. 2, 2016, pp. 645–654.
  * [14] Livox, “Livox Mapping: A mapping package for livox lidars,” [online] https://github.com/Livox-SDK/livox_mapping.
  * [15] J. Lin and F. Zhang, “Loam livox: A fast, robust, high-precision lidar odometry and mapping package for lidars of small fov,” in _Proc. IEEE International Conference on Robotics and Automation_ , 2020, pp. 3126–3131.
  * [16] J.-K. Huang and J. W. Grizzle, “Improvements to target-based 3d lidar to camera calibration,” _IEEE Access_ , vol. 8, pp. 134 101–134 110, 2020\.
  * [17] J.-K. Huang, S. Wang, M. Ghaffari, and J. W. Grizzle, “Lidartag: A real-time fiducial tag system for point clouds,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 3, pp. 4875–4882, 2021.
  * [18] K. Koide, S. Oishi, M. Yokozuka, and A. Banno, “Scalable fiducial tag localization on a 3d prior map via graph-theoretic global tag-map registration,” in _Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2022, pp. 5347–5353.
  * [19] Y. Huang, J. Zhao, X. He, S. Zhang, and T. Feng, “Vision-based semantic mapping and localization for autonomous indoor parking,” in _Proc. of IEEE Intelligent Vehicles Symposium_ , 2018, pp. 636–641.
  * [20] Y. Liu, H. Schofield, and J. Shan, “Intensity image-based lidar fiducial marker system,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 3, pp. 6542–6549, 2022.
  * [21] Y. Xie, L. Deng, T. Sun, Y. Fu, J. Li, X. Cui, H. Yin, S. Deng, J. Xiao, and B. Chen, “A4lidartag: Depth-based fiducial marker for extrinsic calibration of solid-state lidar and camera,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 3, pp. 6487–6494, 2022.
  * [22] A. D. Sappa and M. Devy, “Fast range image segmentation by an edge detection strategy,” in _Proc. International Conference on 3-D Digital Imaging and Modeling_ , 2001, pp. 292–299.
  * [23] K. Koster and M. Spann, “Mir: An approach to robust clustering-application to range image segmentation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 22, no. 5, pp. 430–444, 2000.
  * [24] J. M. Biosca and J. L. Lerma, “Unsupervised robust planar segmentation of terrestrial laser scanner point clouds based on fuzzy clustering methods,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 63, no. 1, pp. 84–98, 2008.
  * [25] M. A. Fischler and R. C. Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,” _Communications of the ACM_ , vol. 24, no. 6, pp. 381–395, 1981.
  * [26] S. C. Johnson, “Hierarchical clustering schemes,” _Psychometrika_ , vol. 32, no. 3, pp. 241–254, 1967.
  * [27] L. Fan, X. Xiong, F. Wang, N. Wang, and Z. Zhang, “Rangedet: In defense of range view for lidar-based 3d object detection,” in _Proc. IEEE/CVF International Conference on Computer Vision_ , 2021.
  * [28] P. Kovesi, “Good colour maps: How to design them,” _arXiv preprint arXiv:1509.03700_ , 2015.
  * [29] Y. Liu. Modified Livox Mapping. (2022). [Online]. Available: https://github.com/York-SDCNLab/m_livox_mapping
  * [30] Livox-SDK. Livox Mapping. (2020). [Online]. Available: https://github.com/Livox-SDK/livox_mapping
  * [31] G. P. Meyer, A. Laddha, E. Kee, C. Vallespi-Gonzalez, and C. K. Wellington, “Lasernet: An efficient probabilistic 3d object detector for autonomous driving,” in _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 12 677–12 686.
  * [32] J. Mao, Y. Xue, M. Niu, H. Bai, J. Feng, X. Liang, H. Xu, and C. Xu, “Voxel transformer for 3d object detection,” in _Proc. the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 3164–3173.
  * [33] Y. Chen, Y. Li, X. Zhang, J. Sun, and J. Jia, “Focal sparse convolutional networks for 3d object detection,” in _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5428–5437.
  * [34] Y. Chen, J. Liu, X. Zhang, X. Qi, and J. Jia, “Voxelnext: Fully sparse voxelnet for 3d object detection and tracking,” in _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 21 674–21 683.
  * [35] J. Beltrán, C. Guindel, A. de la Escalera, and F. García, “Automatic extrinsic calibration method for lidar and camera sensor setups,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 10, pp. 17 677–17 689, 2022.
  * [36] L. Tao, L. Pei, T. Li, D. Zou, Q. Wu, and S. Xia, “Cpi: Lidar-camera extrinsic calibration based on feature points with reflection intensity,” in _Spatial Data and Intelligence: First International Conference, SpatialDI 2020, Virtual Event, May 8–9, 2020, Proceedings 1_. Springer, 2021, pp. 281–290.
  * [37] K. Koide, S. Oishi, M. Yokozuka, and A. Banno, “General, single-shot, target-less, and automatic lidar-camera extrinsic calibration toolbox,” _arXiv preprint arXiv:2302.05094_ , 2023.
  * [38] J. Zhang and S. Singh, “Loam: Lidar odometry and mapping in real-time.” in _Robotics: Science and Systems_ , vol. 2, no. 9, 2014, pp. 1–9.
  * [39] R. B. Rusu, “Semantic 3d object maps for everyday manipulation in human living environments,” _KI-Künstliche Intelligenz_ , vol. 24, no. 4, pp. 345–348, 2010.
  * [40] P. Schneider and D. H. Eberly, _Geometric tools for computer graphics_. Elsevier, 2002.