1174 Research algorithm/technique Hongjia Zhai, Gan Huang, Qirui Hu, Guanglin
Li, Hujun Bao, and Guofeng Zhang are with the State Key Lab of CAD&CG,
Zhejiang University, Hangzhou 310058, China. E-mails: {zhj1999, huanggan,
3200105912, liguanglin, baohujun<EMAIL_ADDRESS>Corresponding
Author: Guofeng Zhang.

# NIS-SLAM: Neural Implicit Semantic RGB-D SLAM for 3D Consistent Scene
Understanding

Hongjia Zhai0000-0002-7729-8787  Gan Huang0000-0001-8515-2721  Qirui
Hu0009-0009-1854-689X  Guanglin Li0009-0000-8996-3775  Hujun
Bao0000-0002-2662-0334  and Guofeng Zhang0000-0001-5661-8430

###### Abstract

In recent years, the paradigm of neural implicit representations has gained
substantial attention in the field of Simultaneous Localization and Mapping
(SLAM). However, a notable gap exists in the existing approaches when it comes
to scene understanding. In this paper, we introduce NIS-SLAM, an efficient
neural implicit semantic RGB-D SLAM system, that leverages a pre-trained 2D
segmentation network to learn consistent semantic representations.
Specifically, for high-fidelity surface reconstruction and spatial consistent
scene understanding, we combine high-frequency multi-resolution tetrahedron-
based features and low-frequency positional encoding as the implicit scene
representations. Besides, to address the inconsistency of 2D segmentation
results from multiple views, we propose a fusion strategy that integrates the
semantic probabilities from previous non-keyframes into keyframes to achieve
consistent semantic learning. Furthermore, we implement a confidence-based
pixel sampling and progressive optimization weight function for robust camera
tracking. Extensive experimental results on various datasets show the better
or more competitive performance of our system when compared to other existing
neural dense implicit RGB-D SLAM approaches. Finally, we also show that our
approach can be used in augmented reality applications. Project page:
https://zju3dv.github.io/nis_slam.

###### keywords:

Implicit representation, Neural dense SLAM, Semantic segmentation, Scene
understanding

We present NIS-SLAM, a neural implicit semantic RGB-D SLAM system that
incrementally reconstructs the environment with 3D consistent scene
understanding. As shown in the figure, taking continuous RGB-D frames and 2D
noise segmentation results as input, our system can reconstruct high-fidelity
surface and geometry, learn 3D consistent semantic field, and recover the
objects in the scene.

Introduction

Dense Simultaneous Localization and Mapping (SLAM) is a fundamental research
topic in 3D computer vision, which aims to localize the 6 degrees of freedom
(6DoF) pose of the camera and reconstruct a dense map in an unknown
environment. It is an essential part of Virtual/Augmented Reality (VR/AR),
robot localization/navigation, and visual perception. For example, in VR/AR,
the SLAM system can provide pose information for mobile devices and head-
mounted devices to help users interact with virtual content, and the geometric
results of dense reconstruction can better simulate physical space collisions,
thereby creating a better immersive and engaging experience.

Traditional RGB-D SLAM [13, 42, 68] usually performs frame-to-model
optimization to track camera based on the pixel-level depth or color
observations from RGB-D camera. Benefit from the depth sensor, traditional
approaches use iterative closest points (ICP) [1] and truncated signed
distance function (TSDF) [11] to update a global map represented by geometric
primitives, such as voxels, cost volumes, and surfels. With the development of
deep learning, recent works have turned to exploring data-driven priors and
leveraging the smoothness properties of neural networks [20, 3, 62, 66, 29].
Although traditional and learning-based dense SLAM systems have shown good
localization and reconstruction performance, they cannot perform novel view
rendering and produce watertight surfaces. For VR/AR applications, realistic
image rendering and novel view synthesis also play an important role.

As one of the research fields, Neural Radiance Field (NeRF) based SLAM methods
[58, 64, 23, 50] have shown significant advantages in novel view rendering,
high-fidelity map reconstruction, and hole filling. Different from the
traditional SLAM approaches, neural implicit SLAM uses a neural radiance field
to represent scene property and uses multilayer-perceptron (MLP) to decode the
attributes of the scene (color, density, signed distance function, semantic
information, etc.). Benefiting from volume rendering [24] and positional
encoding [37], implicit representation enables high-fidelity image rendering
and new perspective synthesis. iMAP [58] is the first neural implicit dense
SLAM, which directly uses a single MLP to model the geometry and appearance
information. However, it is hard to reconstruct large indoor scenes due to the
forgetting problem of MLP in continual learning. To overcome this challenge,
many works propose to use additional parametric encoding, like dense feature
grid/plane/point [79, 23, 50, 71, 78], hash table [64, 22, 17], to increase
the representation ability of the MLP. However, additional dense features will
lead to more memory usage, which is not efficient. Besides, semantic
information also plays an important role in SLAM, which allows robots to
perceive and understand the world. Although there exist works [74, 54, 14, 28]
that combine neural implicit representation and semantics modeling, little
attention has been paid to modeling semantic information in a neural implicit
dense SLAM. This is because using neural implicit representation to
simultaneously perform camera tracking and semantic learning with inconsistent
2D semantic input is rather challenging.

vMAP [26] and Haghighi _et al._ [17] are the most related works that use
implicit representation to model 3D instance/semantic information. vMAP [26]
uses ground truth pose and instance mask for commonly used datasets [55, 12]
to reconstruct the object in the scene. Haghighi [17] uses ORB-SLAM3 [5] to
provide pose for the mapping process and they ignore the inconsistency problem
of 2D segmentation. Using the traditional SLAM approach as a decoupled front
end makes them turn into a mapping method instead of an SLAM method. Besides,
they don’t handle the problem of camera pose estimation and multi-view
inconsistency of 2D segmentation. As mentioned above, currently, there is no
implicit-based approach that simultaneously performs camera tracking, dense
surface reconstruction, and 3D consistent scene semantic understanding from
noisy 2D input.

To this end, in this paper, we introduce NIS-SLAM, an efficient neural
implicit semantic RGB-D SLAM system, which enables simultaneous scene
reconstruction and learning 3D consistent semantics from the inconsistent
segmentation results of 2D CNN. Specifically, for high-fidelity surface
reconstruction and spatial consistent scene understanding, we use the hybrid
representation of high-frequency multi-resolution tetrahedron-based features
and low-frequency positional encoding as the input of our system. Besides, to
learn a 3D consistent semantic field, an effective multi-view semantic fusion
strategy is introduced to handle the inconsistency of 2D segmentation results.
Finally, semantic-guided pixel sampling and progressive optimization weight
are also used for robust camera tracking. In summary, our contributions
include:

  * •

We propose an efficient neural implicit semantic RGB-D SLAM system with hybrid
implicit representation. Our system can simultaneously reconstruct the
environment and model 3D consistent semantic information from 2D segmentation
results.

  * •

We propose an effective multi-view semantic fusion that enables learning 3D
consistent semantic information. Besides, semantic guided sampling and
progressive optimization weight are used for robust camera tracking.

  * •

We perform extensive experiments on commonly used datasets to demonstrate the
state-of-the-art and comparable performance of camera tracking,
reconstruction, and semantic segmentation.

This paper is structured as follows: In Section 1, we provide a review of
related works to contextualize our research. Next, in Section 2, we explain
each key component in our reconstruction pipeline. Subsequently, in Section 3,
we evaluate the performance of our system through various synthetic and real-
world datasets. Finally, conclusion is given in Section 4.

##  1 Related Work

###  1.1 Dense Visual SLAM

DTAM [43] was the first dense SLAM system that leveraged pixel-level
photometric information to perform camera pose tracking. Taking advantage of
RGB-D cameras, some traditional approaches use iterative closest points (ICP)
[1] and truncated signed distance function (TSDF) [11] to achieve good 3D
reconstruction results, such as KinectFusion [42], BundleFusion [13], and
ElasticFusion [69]. Benefiting from deep learning technology, some learning-
based approaches [61, 66, 59, 62, 20, 3] are proposed to improve the
robustness and accuracy. For example, DI-Fusion [20] and CodeSLAM [3] learn
the scene representation implicitly which encodes the geometry information in
a low dimensional latent feature. BAD-SLAM [51] uses a direct bundle
adjustment (BA) to jointly optimize the poses of keyframe and 3D scene
geometry. There are still some approaches that pay attention to scene
understanding, like SemanticFusion [36] and PanopticFusion [41]. Besides,
semantic-based object alignment [18] and reconstruction [16] also play an
important role in robot navigation and perception. Although these methods use
learning-based components, the scene and map representation of the environment
and the system framework are still the same as traditional SLAM methods. They
cannot achieve realistic image rendering and novel view synthesis, which is
more important for the applications of VR/AR.

###  1.2 Neural Implicit Scene Representation

Implicit representation has facilitated novel applications in 3D computer
vision, such as novel view synthesis [37, 63, 60], surface reconstruction [2,
44, 65, 30, 7], and dynamic scene reconstruction [46, 15, 47, 45]. Depending
on the scene representation approaches, three main categories emerge: network-
based, grid-based, and point-based. Network-based methods [37, 35, 46, 65, 2]
offer a continuous representation of the scene through implicit modeling via
coordinate MLPs, enabling rendering and surface reconstruction. However, due
to the limited representation capacity of MLPs, they often exhibit reduced
efficacy in representing and reconstructing large-scale scenes. Consequently,
grid-based approaches have evolved, employing explicit representations like
dense grids to encode scene information, partially mitigating the limitations
of MLP-based decoding. Additionally, some methods utilize sparse voxels [32,
30, 31], hashtables [38, 49], tetrahedron [27, 39, 53] or other efficient
representations [6, 23] to reduce the parameters of the grid. Point-based
methods [70, 50] allow for a more flexible scene representation, where points
can be concentrated near complex surfaces to avoid wasting memory on modeling
unused free space. 3D Gaussian Splatting [25] also can be viewed as the point-
based method, which uses Gaussian to encode the local information of 3D points
in the scene. The splatting strategy avoids dense point sampling along the
camera ray, which accelerates the rendering speed of the neural implicit
model. However, the overall parameter count for the entire representation is
directly proportional to the number of points which usually needs to be at
least semi-dense to ensure a relatively good reconstruction performance.

Figure 1: Our system takes RGB-D frames as input to perform camera tracking
and mapping via volume rendering and models 3D semantics with the noise 2D
segmentation results from Mask2Former [9]. Based on the hybrid implicit
representation of multi-resolution tetrahedron feature $\theta$ and positional
encoding $\texttt{PE}(p)$, we decode the SDF $\sigma$, latent feature $h$,
color $c$, and semantic probability $s$ with three MLPs {$\mathcal{M}_{geo}$,
$\mathcal{M}_{color}$, and $\mathcal{M}_{sem}$}. To model consistent semantic
property, we fuse multi-view semantics of nearby non-keyframes for learning 3D
consistent representation.

###  1.3 Neural Implicit Dense SLAM

Recently, there have been successful approaches that combine dense visual SLAM
with neural implicit radiance fields [58, 71, 64, 23]. Compared to the
traditional SLAM approaches, neural implicit SLAM can reconstruct the
watertight surface of the scene and perform realistic novel view rendering.
iMAP [58] was the first neural implicit SLAM approach that uses coordinate
MLPs to represent the scene and continuously update it during incremental
mapping. To address the challenge of reconstructing large indoor scenes, some
approaches utilize multi-level dense feature grids [79, 78, 23], hash table
[64], and voxel/point embeddings [71, 22, 50, 72] to mitigate the
representation ability of coordinate MLPs. Besides, some methods [22, 10, 48,
73, 26] decouple the front and back ends, employing traditional or learning-
based VO/VIO [40, 52, 62] for camera pose tracking and utilizing implicit
representation in the back end for dense reconstruction. To perform scene
understanding, Haghighi _et al._ [17] and vMAP [26] incorporate
semantic/instance information into neural implicit reconstruction, which both
use ORB-SLAM3 [5] to provide camera pose. Zhu  _et al._ [76] propose the first
neural semantic mapping system, which learns consistent semantic fields based
on the pose estimated by neural SLAM. These kinds of approaches using
decoupled modules can accelerate tracking speed and accuracy in complex scenes
but depart from a couple of pure implicit SLAMs.

##  2 Method

The pipeline of our method is shown in Fig. 1. Given an input sequence of
RGB-D images {$I_{i}\in\mathbb{R}^{3},D_{i}\in\mathbb{R}$}, we first generate
the 2D semantic segmentation results {$S_{i}$} and confidence {$Conf_{i}$} via
a pre-trained CNN model (Mask2Former [9]). Based on those inputs, our system
is capable of recovering camera poses, and implicit SDF fields, and
additionally reconstructing a 3D consistent semantic representation. These
individual components will be explained in detail in the following
subsections. We first introduce our tetrahedron-based neural implicit
representation (Section 2.1) and how color/depth/semantic information is
rendered via SDF-based volume rendering (Section 2.2). Then, the multi-view
semantic fusion strategy is proposed to handle the noise semantic information
(Section 2.3). Besides, the overall optimization objective functions are
introduced in Section 2.4. Finally, the tracking and mapping processes of our
system are shown in Section 2.5 and Section 2.6, respectively.

Figure 2: Barycentric interpolation of multi-resolution tetrahedron feature.
For tetrahedrons of different resolutions, we obtain the features of the query
point according to the vertex features and barycentric.

###  2.1 Tetrahedron-based Representation

Only using MLPs [58] to decode the scene property can lead to worse
performance due to the forgetting problem during incremental reconstruction of
large scenes. Although grid-based or point-based methods [64, 71, 50] can
enhance the encoding capacity of MLPs for large indoor scenes, they can not
fit the scene surface very well due to the inherent flaws of their
representation. Using tetrahedron [27, 53] to encode the scene information is
more appropriate than using cube/grid/point due to the fundamental geometric
primitive is the triangular face.

Same to [38, 49], we employ a multi-resolution feature to encode high-
frequency information (e.g. color) and use a hash table to reduce the
additional parameters. Besides, benefiting from the joint encoding [64], we
additionally apply a low-frequency positional encoding [37] for reasoning
spatial semantics. Additionally, as shown in [4], the semantic information is
the low-frequency information compared to the color. So, for semantic property
decoding, we concatenate the positional encoding with the latent feature as
the input. For color property decoding, we concatenate the tetrahedron feature
with the latent feature as the input. Those processes are shown in the top
part of Fig. 1.

Specifically, for a 3D point $p\in\mathbb{R}^{3}$, we first compute its low-
frequency positional encoding value $\texttt{PE}(p)$ with following equation:

$\texttt{PE}(p)=[\gamma(x),\gamma(y),\gamma(z)],$ (1)
$\gamma(x)=(\sin(2^{0}\pi x),\cos((2^{0}\pi x)),\cdots,\sin(2^{n-1}\pi
x),\cos((2^{n-1}\pi x))),$ (2)

where $n$ represents the frequency, and we set $n$=8 in our experiments.

Then, we interpolate the high-frequency feature of $p$ from multi-resolution
tetrahedron features $\Theta$ = {$\theta_{l}$}${}_{l=1}^{L}$. Here, we
retrieve multi-resolution tetrahedrons that contain the point $p$. We obtain
its feature via barycentric interpolation according to the feature in the
vertexes of the tetrahedrons with the following equation
$\mathcal{F}(p;\Theta)$.

$\mathcal{F}(p;\Theta)=[\mathcal{V}(p,\theta_{1}),\cdots,\mathcal{V}(p,\theta_{l})],$
(3)

where $\mathcal{V}(\cdot)$ refers to the barycentric interpolation function
for each resolution tetrahedron. The barycentric interpolation of the
tetrahedron is shown in Fig. 2. After obtaining the multi-resolution features,
we decode the geometry property with geometry MLP $\mathcal{M}_{geo}$:

$(h,\sigma)=\mathcal{M}_{geo}(\texttt{PE}(p),\mathcal{F}(p;\Theta)),$ (4)

where $h$ and $\sigma$ are the latent feature vector and signed distance of
location $p$. Finally, we decode the color and semantic probability based on
the latent feature $h$ and input encodings via separated MLPs:

$c=\mathcal{M}_{color}(\mathcal{F}(p;\Theta),h),\quad
s=\mathcal{M}_{sem}(\texttt{PE}(p),h).$ (5)

So, the optimizable parameters for the whole implicit representation are three
MLP decoders {$\mathcal{M}_{geo}$, $\mathcal{M}_{color}$, $\mathcal{M}_{sem}$}
and multi-resolution tetrahedron features $\Theta$.

###  2.2 SDF-based Volume Rendering

Compared to occupancy or density-based rendering approaches [79, 50], using
SDF-based volume rendering can lead to more accurate surface reconstruction.
To render color, depth, and semantics for each pixel, we adopt the SDF-based
volume rendering strategy [2, 65] to aggregate the property of points along
the ray.

Specifically, for ray $r$ emitted from the center of camera with origin
$o_{r}\in\mathbb{R}^{3}$ and view direction $d_{r}\in\mathbb{R}^{3}$, we can
sample points along $r$:

$p_{r}(i)=o_{r}+z_{r}(i)\cdot d_{r},\quad i\in\\{1,\cdots,M\\},$ (6)

where $M$ is the number of total sampled points, $z_{r}(i)\in\mathbb{R}$ is
the depth of $i$-th sampled point. Similar to other approaches [64, 23, 79],
we replace the importance sampling with a depth-guided sampling strategy.
Specifically, we first uniformly sample $M_{c}$ points between the near and
far bound. We then uniformly sample $M_{f}$ points around the surface
according to the depth values of the selected pixel.

For all $M=M_{c}+M_{f}$ points along the ray, we query their signed distance,
color, and semantic probability values using MLP decoders (Section 2.1). As
shown in [2], a simple bell-shaped weight function can be employed to
transform the signed distance function, $\sigma$, into rendering weights.
Here, we adopt the Gaussian distribution with mean 0 as the rendering weight
distribution, which is shown as follows:

$\widetilde{w}_{i}=\frac{w_{i}}{\sum_{j}w_{j}}\quad\text{with}\quad
w_{i}=\frac{1}{\delta\sqrt{2\pi}}\exp(-\sigma_{i}^{2}/\delta^{2}),$ (7)

where $\delta$ is the parameter, which controls the shape of the weight
distribution for points with different signed distance values.
$\widetilde{w}_{i}$ is the normalized rendering weight for $i$-th point.

So, for ray $r$, we can obtain its scene property via:

$\hat{c}=\sum_{i=1}^{M}\widetilde{w}_{i}c_{i},\quad\hat{d}=\sum_{i=1}^{M}\widetilde{w}_{i}d_{i},\quad\hat{s}=\sum_{i=1}^{M}\widetilde{w}_{i}s_{i}.$
(8)

###  2.3 Multi-view Noisy Semantic Fusion

Due to the inherent noise and multi-frame inconsistencies of results generated
by Mask2Former [9], the SLAM system may learn wrong 3D semantic
representations when the segmentation results of keyframes are inaccurate.
Inspired by the warp strategy [75, 67], we propose an efficient approach to
fuse semantic information from multi-view frames to address the inconsistency
issue. At the bottom of Fig. 1, we first back project the pixel, $P_{t}$, of
keyframe $I_{t}$ into $k$ previous non-keyframes $I_{t-i}$ ($i=\\{1,...,k\\}$)
according to the relative pose, $T_{t\rightarrow t-i}$, estimated by our
neural implicit SLAM system. Then, we synthesize the projected semantics via
the differentiable bilinear sampling proposed in [21]. The formula is shown in
the following:

$S_{t-i\rightarrow t}=\texttt{interp}(KT_{t\rightarrow
t-i}D_{t}K^{-1}P_{t},S_{t-i}),$ (9)

where interp denotes the differentiable bilinear sampling operation, $K$ is
the camera intrinsics matrix, and $S_{t-i\rightarrow t}$ is the semantic
probility projected from $I_{t-i}$.

Thus, we can obtain the fused semantic information for keyframe $I_{t}$ from
adjacent $k$ non-keyframes. Since 2D segmentation is often accompanied with
confidence, we fuse multi-view semantic information based on the normalized
confidence weight with Softmax function:

$S^{m}_{t}=\sum_{i}^{k}\texttt{softmax}(Conf_{t-i})\cdot S_{t-i\rightarrow
t},$ (10)

where $Conf_{t-i}$ is the confidence of $I_{t-i}$, and $S^{m}_{t}$ is the
multi-view fused semantic probability, which can be used as the supervision of
semantic MLP.

###  2.4 Objective Functions

We perform camera tracking and bundle adjustment via objective function
minimization. Like [2, 71], to optimize the whole system, we apply five
different objective functions: reconstruction loss, SDF loss, free space loss,
semantic loss, and regularization loss.

Reconstruction loss. For each ray with ground truth depth and color, we apply
the reconstruction losses between the rendered value and ground truth value
measured by the camera:

$\mathcal{L}_{c}=\frac{1}{|R|}\sum_{r\in
R}(\hat{c}(r)-I(r))^{2},\quad\mathcal{L}_{d}=\frac{1}{|R|}\sum_{r\in
R}(\hat{d}(r)-D(r))^{2},$ (11)

where $R$ is the set of sampled pixels for optimization, {$I(r)$,$D(r)$} are
the color and depth value of ray $r$, and {$\hat{c}(r)$, $\hat{d}(r)$} are the
rendered value of sampled ray $r$, which are computed by Eq. 8.

SDF and free space loss. Apart from the reconstruction loss, we also consider
the geometrical information to supervise the reconstruction process. To learn
a better geometry representation, we apply constraints on the signed distance
value, $\sigma$, with the following SDF loss and free space loss. For the
points near the surface and within the truncation region, e.g.
$|\hat{d}(p)-D(r)|<tr$, we use depth measurement to approximate the signed
distance value:

$\mathcal{L}_{sdf}=\frac{1}{|R|}\sum_{r\in R}\frac{1}{|P_{r}^{tr}|}\sum_{p\in
P_{r}^{tr}}(\hat{d}(p)+\sigma\cdot tr-D(r))^{2},$ (12)

where $tr$ is the truncation distance, $P^{tr}_{r}$ is the set of sampled
points within the truncation region. The SDF loss $\mathcal{L}_{sdf}$ pushes
the signed distance function to fit the surface of the scene.

Besides, for sampled points that are far away from the surface, e.g.
$|\hat{d}(p)-D(r)|>tr$, we apply a regularization on the free space that
pushes its signed distance to 1:

$\mathcal{L}_{fs}=\frac{1}{|R|}\sum_{r\in R}\frac{1}{|P_{r}^{fs}|}\sum_{p\in
P_{r}^{fs}}(\sigma-1)^{2},$ (13)

where $P_{r}^{fs}$ is the set of sampled points within the free space. The
free space loss $\mathcal{L}_{fs}$ pushes the signed distance value of empty
space to 1, which reduces the noise or redundant points during the marching
cubes.

Semantic loss. To learn a consistent semantic field, we simultaneously use
single-view predicted probability and the fused multi-view semantic
probability as the supervision for the semantic decoder. Here, we stop the
gradient propagation from the semantic MLP to geometry MLP for stable geometry
learning. The semantic loss is defined as:

$\mathcal{L}_{sem}=-\frac{1}{|R|}\sum_{r\in
R}\sum_{i=0}^{N_{c}}[(S_{r}(i)+\alpha\cdot S_{r}^{m}(i))\log\hat{s}_{r}(i)],$
(14)

where $N_{c}$ is the number of classes for semantic segmentation,
$S_{r},S_{r}^{m}\in\mathbb{R}^{N_{c}}$ are the single-view and fused multi-
view semantic probability for ray $r$, respectively. $\alpha$ is the weight
parameter that balances two semantic components.

Regularization loss. Unlike [58] the built-in smoothness priors, which uses a
single MLP to represent the scene. To ensure the smoothness of the tetrahedron
feature space and to avoid hash collisions when sampling vertex features, we
apply regularization on the interpolated multi-resolution features:

$\mathcal{L}_{reg}=-\frac{1}{|R|}\sum_{r\in
R}(\mathcal{F}(p;\Theta)-\mathcal{F}(p+\epsilon;\Theta))^{2},$ (15)

where $\epsilon$ denotes the small perturbation of point location in 3D space,
and $\mathcal{F}(\cdot;\cdot)$ is the feature query functions in Section 2.1.
We only perform the regularization term of the multi-resolution tetrahedron
feature during the mapping process on small random regions.

So, the final loss function is presented as follows:

$\mathcal{L}=\lambda_{1}\mathcal{L}_{c}+\lambda_{2}\mathcal{L}_{depth}+\lambda_{3}\mathcal{L}_{fs}+\lambda_{4}\mathcal{L}_{sdf}+\lambda_{5}\mathcal{L}_{sem}+\lambda_{6}\mathcal{L}_{reg},$
(16)

where {$\lambda_{i}$} are the weight for each optimization component.

###  2.5 Tracking

For camera tracking, we fix the pose of the first frame as the identity
matrix. And for later coming frame $I_{t}$, we first initialize its camera-to-
world pose, $T_{wc}^{t}=\exp(\xi)\in\mathbb{SE}(3)$, under the simple constant
speed assumption. Then we sample $M_{t}$ pixels across the frame for pose
optimization. During the tracking process, we fix the parameters of decoders
and multi-resolution tetrahedron features and only update $T_{wc}^{t}$.

|
---|---
Confidence Map | Tracking Error Map
Figure 3: Segmentation confidence map and tracking error map. Complex geometry
and ambiguous regions usually lead to lower confidence and slow convergence
speed for color and geometry fields. As shown in the right figure, those
regions will lead to large tracking errors.

Semantic-guided Pixel Sampling. For pixel sampling, we use confidence values
from the semantic segmentation as the sampling probabilities for pixels.
Pixels with higher confidence values are deemed more reliable for camera
tracking compared to those with lower confidence. As shown in Fig. 3, we show
the confidence map (left part) and tracking error map of all pixels (right
part). Edge and challenge regions usually have low segmentation confidence and
high tracking errors due to sensor noise and slow convergence speed of SDF
filed in the complex geometry regions. So, based on the confidence, we can
avoid sampling these regions and use high-reliability and robust pixels to
optimize the camera pose during tracking.

Progressive Optimization Weight Function. As shown in [8, 77], NeRF-based pose
optimization is sensitive to the initialization. The high-frequency
information may lead to suboptimal results when the initial pose is far away
from the ground truth. Hence, we propose a progressive weight function that
progressively updates the rendering weights during the iteration of the
tracking process. Specifically, according to the render function in Section
2.2, the distribution of rendering weight is controlled by the $\delta$. Large
$\delta$ means coarse constraints while small values represent fine-level
constraints. So, we apply a coarse-to-fine optimization loss for camera
tracking and update $\delta$ in a decreasing manner via:

$\delta=\delta_{max}+(\delta_{min}-\delta_{max})\sin(\frac{i}{N}\cdot\frac{\pi}{2}),$
(17)

where $i$ and $N$ are the current and total iteration steps for camera
tracking, $\delta_{min}$, $\delta_{min}$ are the minimum and maximum values
during optimization, and we use the $\sin(\cdot)$ function to update $\delta$
non-linearly.

Besides, for tracking, we set the weight of semantic loss as 0, because the
predicted semantic results are noisy and inaccurate, and ground RGB and depth
images are enough for camera tracking.

###  2.6 Mapping

We first initialize our permutohedral lattice features and MLP decoders with
the first frame {$I_{0}$,$D_{0}$} for several iterations. Then, we perform
global bundle adjustments to update our model for every $k$ frame. Following
the practice in Co-SLAM [64], we use subset rays to construct the keyframe
database instead of storing the whole keyframe [79]. We randomly choose
$M_{m}$ rays from the global keyframe database to perform bundle adjustments
using all optimization functions in Section 2.4.

##  3 Experiments

In this section, we introduce three commonly used datasets and our
experimental setting. When comparing with multiple baselines, we highlight the
best three results as first, second, and third for better view.

Dataset. We evaluate our approach on a variety of scenes from three different
datasets. For Replica dataset [55], we use the 8 synthetic scenes collected by
iMAP [58] for camera tracking and reconstruction evaluation. For the
performance on real-world scenes, we use 5 scenes from ScanNet [12] and 3
scenes from TUM RGB-D dataset [56] for pose estimation.

Implementation Details. Our geometry, color, and semantic decoders are
implemented as MLP which consists of 2 fully connected (FC) layers with 32
latent dimensional channels. The geometry decoder outputs a 15-dimensional
latent feature vector and SDF scalar. We use the ReLU activation function
except for the final output. Besides, we apply the Sigmoid function to the
output of the color decoder, which limits the color value to $(0,1)$. We use
Adam optimizer and set hyper-parameters $beta=$(0.9, 0.999), $eps=1e-15$. The
learning rate for the parameters is set to 0.01. We set the truncation
distance $tr$ to 5cm in our method. For the Replica dataset, we use 40 and 100
iterations for tracking and mapping. For the ScanNet dataset, we use 40 and 50
iterations for tracking and mapping. The loss weights for different
optimization components are $\lambda_{1}$=5.0, $\lambda_{2}$=0.1,
$\lambda_{3}$=1e3, $\lambda_{4}$=10, $\lambda_{5}$=1.0, $\lambda_{6}$=1e-6. We
use [49] to encode the tetrahedron feature, and the multi-resolution
tetrahedron features contain 16 level details, each containing a feature
vector of 2 dimensions. For positional encoding, we set the frequency to 8,
which leads to a 48-dimensional feature vector. So, the final input feature of
our system is the concatenated 80 channels feature. The finest tetrahedron
resolution is set to 2cm.

Method | Rm0 | Rm1 | Rm2 | Off0 | Off1 | Off2 | Off3 | Off4 | Avg.
---|---|---|---|---|---|---|---|---|---
NICE-SLAM [79] | 0.97 | 1.31 | 1.07 | 0.88 | 1.00 | 1.06 | 1.10 | 1.13 | 1.06
Vox-Fusion [71] | 0.40 | 0.54 | 0.54 | 0.50 | 0.46 | 0.75 | 0.50 | 0.60 | 0.54
Co-SLAM [64] | 0.77 | 1.04 | 1.09 | 0.58 | 0.53 | 2.05 | 1.49 | 0.84 | 0.99
ESLAM [23] | 0.71 | 0.70 | 0.52 | 0.57 | 0.55 | 0.58 | 0.72 | 0.63 | 0.63
Point-SLAM [50] | 0.61 | 0.41 | 0.37 | 0.38 | 0.48 | 0.54 | 0.69 | 0.72 | 0.52
Ours | 0.30 | 0.40 | 0.36 | 0.29 | 0.31 | 0.92 | 0.67 | 0.44 | 0.46

Table 1: Tracking Performance on Replica [55] (ATE RMSE $\downarrow$ [cm]).
The numbers of compared methods are taken from the original papers or [50,
64].

Method | 0000 | 0059 | 0106 | 0169 | 0207 | Avg.
---|---|---|---|---|---|---
DI-Fusion [20] | 62.99 | 128.00 | 18.50 | 75.80 | 100.19 | 77.10
iMAP [58] | 55.95 | 32.06 | 17.50 | 70.51 | 11.91 | 37.59
NICE-SLAM [79] | 12.00 | 14.00 | 7.90 | 10.90 | 6.20 | 10.20
Vox-Fusion [71] | 8.39 | 9.13 | 7.44 | 6.53 | 5.57 | 7.41
Co-SLAM [64] | 7.18 | 12.29 | 9.57 | 6.62 | 7.13 | 8.56
Point-SLAM [50] | 10.24 | 7.81 | 8.65 | 22.16 | 9.54 | 11.68
Ours | 8.70 | 9.62 | 8.35 | 5.64 | 7.10 | 7.88

Table 2: Tracking Performance on ScanNet [12] (ATE RMSE $\downarrow$ [cm]).
The numbers of all compared methods are taken from the original papers or[50,
64]. The value of Vox-Fusion [71] on 0059 is obtained with their released
code. All scenes are evaluated on the 00 trajectory.

Method | fr1/desk | fr2/xyz | fr3/office | Avg.
---|---|---|---|---
DI-Fusion [20] | 4.4 | 2.0 | 5.8 | 4.06
NICE-SLAM [79] | 4.26 | 6.19 | 3.87 | 4.77
Vox-Fusion [71] | 3.52 | 1.49 | 26.01 | 10.34
Co-SLAM [64] | 2.70 | 1.90 | 2.60 | 2.50
Point-SLAM [50] | 4.34 | 1.31 | 3.48 | 2.40
Ours | 2.27 | 1.63 | 2.36 | 2.08

Table 3: Tracking Performance on TUM RGB-D [56] (ATE RMSE $\downarrow$ [cm]). The numbers of compared methods are taken from [50, 64]. |  NICE-SLAM [79] |  Point-SLAM [50] | Ours | GT
---|---|---|---|---
scene0000 |  |  |  |
scene0106 |  |  |  |
scene0169 |  |  |  |
Figure 4: Reconstruction Results on ScanNet [12]. Compared to the baselines, our method can reconstruct more accurate detailed geometry and generate more complete, smoother mesh. vMAP [26] |
---|---
Ours |
Figure 5: Object reconstruction of Replica [55]. We show some selected objects
for comparison with vMAP [26].

###  3.1 Evaluation of Camera Tracking

In this part, we evaluate the performance of camera tracking on Replica [55],
ScanNet [12], and TUM RGB-D [56] datasets. The compared baseline methods,
evaluation metrics, and results of this part are shown in the following.

Baselines and Metrics. Following the setting in [50, 64], we take iMAP [58],
NICE-SLAM [79], Vox-Fusion [71], Co-SLAM [64], ESLAM [23], and Point-SLAM [50]
as the compared baselines. In order to measure the camera tracking accuracy of
the proposed method, we use the commonly absolute trajectory error ATE RMSE
[57] as the evaluation metric. We align the estimated trajectory and ground
truth trajectory using Horn‘s closed form solution [19] for a fair comparison.

Qualitative and Quantitative Results. We show the camera tracking performance
on the Replica dataset in Table 1. As shown by the results, we achieve the
best tracking performance across most scenarios, except for the Office 2
scene. Overall, our method outperforms the compared baselines in terms of
average results. Besides, to validate the performance on real-world scenes, we
conduct experiments on ScanNet [12] and TUM-RGBD [56] datasets. The results
are shown in Table 2 and Table 3, respectively. For TUM-RGBD dataset, only the
sequence fr3/xyz achieved third performance. On the rest sequences, our
approach achieves the best results, including average data. For ScanNet
dataset, our camera tracking performance is slightly worse than Vox-Fusion.
The reason why Vox-Fusion can achieve good results is that it performs dense
BA for each frame. The running speed of Vox-Fusion is rather slow ($\sim$3s
per frame). Real-world scenarios often involve complex environments, including
depth noise, motion blur, and non-Lambertian materials, which makes the neural
radiance field hard to model. Compared to the point or voxel based approaches,
our performance of reconstructing high-frequency color information will be
slightly worse due to the lack of explicit representation.

Method | Metric | Room0 | Room1 | Room2 | Office0 | Office1 | Office2 | Office3 | Office4 | Avg.
---|---|---|---|---|---|---|---|---|---|---
iMAP [58] |  Acc.[cm]$\downarrow$ | 3.58 | 3.69 | 4.68 | 5.87 | 3.71 | 4.81 | 4.27 | 4.83 | 4.43
Comp.[cm]$\downarrow$ | 5.06 | 4.87 | 5.51 | 6.11 | 5.26 | 5.65 | 5.45 | 6.59 | 5.56
Comp. Ratio[$<5$cm %]$\uparrow$ | 83.91 | 83.45 | 75.53 | 77.71 | 79.64 | 77.22 | 77.34 | 77.63 | 79.06
NICE-SLAM [79] |  Acc.[cm]$\downarrow$ | 2.97 | 3.23 | 3.46 | 5.47 | 3.33 | 4.40 | 3.55 | 2.87 | 3.66
Comp.[cm]$\downarrow$ | 3.30 | 3.07 | 3.75 | 4.54 | 3.83 | 3.90 | 4.49 | 3.91 | 3.85
Comp. Ratio[$<5$cm %]$\uparrow$ | 89.51 | 86.01 | 81.14 | 85.27 | 88.01 | 82.61 | 79.49 | 85.33 | 84.67
Co-SLAM [64] |  Acc.[cm]$\downarrow$ | 1.61 | 1.31 | 1.55 | 1.33 | 1.11 | 1.83 | 1.97 | 1.73 | 1.56
Comp.[cm]$\downarrow$ | 2.96 | 2.46 | 2.36 | 1.43 | 1.82 | 3.26 | 3.26 | 3.36 | 2.61
Comp. Ratio[$<5$cm %]$\uparrow$ | 91.12 | 92.18 | 91.44 | 95.65 | 93.56 | 88.53 | 87.67 | 87.76 | 90.99
Point-SLAM [50] |  Acc.[cm]$\downarrow$ | 1.45 | 1.14 | 1.19 | 1.05 | 0.86 | 1.31 | 1.57 | 1.51 | 1.26
Comp.[cm]$\downarrow$ | 3.46 | 3.02 | 2.65 | 1.65 | 2.21 | 3.62 | 3.47 | 3.90 | 3.00
Comp. Ratio[$<5$cm %]$\uparrow$ | 88.48 | 89.44 | 90.13 | 93.39 | 90.51 | 86.17 | 86.00 | 85.74 | 88.73
Ours |  Acc.[cm]$\downarrow$ | 1.54 | 1.25 | 1.41 | 1.20 | 1.09 | 1.58 | 2.16 | 1.60 | 1.48
Comp.[cm]$\downarrow$ | 2.81 | 2.37 | 2.23 | 1.37 | 1.67 | 2.86 | 2.98 | 3.24 | 2.44
Comp. Ratio[$<5$cm %]$\uparrow$ | 92.14 | 92.88 | 92.98 | 97.13 | 95.08 | 90.87 | 90.17 | 88.67 | 92.49
Table 4: Reconstruction Performance on Replica [55]. For NICE-SLAM and iMAP, we take the number from [79]. For Point-SLAM [50] and Co-SLAM [64], we generate the mesh with their open-source codes. Method | Metric | Room0 | Room1 | Room2 | Office0 | Office1 | Office2 | Office3 | Office4 | Avg.
---|---|---|---|---|---|---|---|---|---|---
vMAP [26] |  Object Acc.[cm]$\downarrow$ | 3.50 | 3.06 | 3.37 | 2.45 | 3.04 | 3.07 | 3.18 | 3.07 | 3.09
Object Comp.[cm]$\downarrow$ | 3.10 | 3.84 | 7.11 | 3.43 | 1.47 | 3.93 | 4.94 | 9.33 | 4.64
Object Comp. Ratio[$<5$cm %]$\uparrow$ | 84.02 | 88.32 | 61.84 | 69.66 | 94.58 | 82.52 | 77.97 | 62.45 | 77.67
Ours |  Object Acc.[cm]$\downarrow$ | 2.05 | 1.47 | 2.87 | 1.96 | 1.27 | 1.84 | 2.15 | 2.76 | 2.04
Object Comp.[cm]$\downarrow$ | 3.50 | 3.37 | 4.87 | 2.62 | 1.39 | 3.46 | 4.41 | 7.87 | 3.94
Object Comp. Ratio[$<5$cm %]$\uparrow$ | 80.75 | 87.75 | 73.21 | 88.05 | 95.29 | 85.27 | 76.12 | 65.92 | 81.52
Table 5: Object Reconstruction Performance on Replica [55]. We reconstruct the object with the pose estimate by ourselves. The numbers of vMAP [26] are evaluated on the meshes obtained from their code with the same pose of ours. Mask2Former [9] |  |  |  |  |
---|---|---|---|---|---
No Fusion |  |  |  |  |
Ours |  |  |  |  |
Figure 6: Semantic Segmentation Results on Replica [55]. We show the multi-view segmentation results of different approaches. The top, middle, and bottom parts show the segmentation results of Mask2Former [9], our approach without semantic fusion, and our approach with semantic fusion respectively. Comparing the segmentation results from different views, we can see that our method can learn more consistent semantic representation. | Room 0 | Room 1 | Room 2
---|---|---|---
NICE-SLAM [79] |  |  |
Point-SLAM [50] |  |  |
Ours |  |  |
GT |  |  |
Figure 7: Reconstruction Performance on Replica [55]. We show the novel view
rendering results of the reconstructed mesh.

###  3.2 Evaluation of Scene and Object Reconstruction

In this part, we evaluate the performance of scene-level and object-level
surface reconstruction on Replica dataset [55] and ScanNet [12]. The compared
baseline methods, evaluation metrics, and results of this part are shown in
the following.

Baseline and Metrics. For scene-level reconstruction evaluation, we follow the
setting in [79, 58], we take iMAP [58], NICE-SLAM [79], Co-SLAM [64], and
Point-SLAM [50] as the compared baselines. The meshes of [50, 64] are
generated via their open-source codes. For object-level reconstruction
evaluation, we take vMAP [26] as the baseline and follow the setting in vMAP.
Due to vMAP needing camera pose as input, we use the camera pose generated via
our system as the input of vMAP for a fair comparison.

For the evaluation metrics, we evaluate the generated meshes on Accuracy (cm),
Completion (cm), and Completion ratio (%) with a threshold of 5cm.
Specifically, we sample points $P$ and $Q$ from the reconstructed and ground
truth meshes and compute the average distance between a point in one set and
its nearest point in the other set. For scene-level and object-level
reconstruction evaluation, we sample 200000 and 10000 points, respectively.

Qualitative and Quantitative Results. Table 4 shows the quantitative scene-
level reconstruction comparison results on the Replica dataset [55]. The
results of iMAP and NICE-SLAM are taken from the reported value in [79]. The
mesh results of Co-SLAM and Point-SLAM are obtained via their open-source
codes. Please note that Co-SLAM [64] manually selected additional virtual
camera views to assist mesh culling, and we can’t obtain their virtual camera
pose. So, we use the same strategy [79] for the meshes generated by Co-SLAM,
which leads to better Accuracy (cm) and worse Completion (cm), and Completion
ratio (%) than it reported. Besides, Point-SLAM [50] generates mesh using
TSDF-Fusion [13], which leads to a more sparse surface. So, their Accuracy is
relatively better than other approaches, but with worse Completion ratio
metrics.

As can be seen from the results in Table 4, our method achieves better or
comparable reconstruction performance, thanks to the accurate camera pose
estimation and the inherent smoothness provided by our tetrahedron-based
parametric encoding. We achieved the best results on the Completion (cm), and
Completion ratio metrics. Compared with the second-best approach, we achieved
6.9%, and 1.6% performance improvement on the average results, respectively.
The performance of our approach on Accuracy (cm) metric is only slightly worse
than Point-SLAM [50], because [50] uses TSDF-Fusion [13] as the post-process
to obtain the mesh. Among those methods whose meshes are generated via
marching cubes [33], we achieve the best performance on Accuracy (cm) metric.
Additionally, we show some qualitative comparison results of Replica and
ScanNet datasets in the Fig. 7 and Fig. 4, respectively. The geometry results
in Fig. 7 are rendered from novel camera views of three selected Replica
scenes. Our method can restore surface details well and complete the
observation area. The results in Fig. 4 are rendered from the top view to show
the overall reconstruction results of the whole scene. As can be seen,
compared with NICE-SLAM [79] and Point-SLAM [50], we can reconstruct a more
watertight, smoother surface for the back side and unobserved areas.

For object-level reconstruction, we compared with vMAP [26] on the Replica
dataset. Because vMAP [26] needs the camera poses as the input, we use the
camera pose recovered by our system as the input of vMAP for a fair
comparison. The quantitative comparison results are shown in Table 5. As can
be seen from the table, we achieve the best results on object-level
reconstruction. The decoder parameters of vMAP are approximately $N$ times
greater than ours, where $N$ is the number of objects in the scene, as it
allocates separate network parameters for each object. Despite using fewer
parameters, we can still estimate the camera pose and achieve comparable
reconstruction results.

Method |  Acc.(%) |  Avg.Acc.(%) |  mIOU(%) |  fwIOU(%)
---|---|---|---|---
Haghighi ∗ [17] | 98.47 | 94.14 | 82.61 | 97.22
Ours ∗ | 99.09 | 97.95 | 98.55 | 98.20
Mask2Former [9] | 78.84 | 75.66 | 68.20 | 72.75
Ours w/o Fusion | 82.47 | 80.11 | 69.53 | 75.48
Ours | 84.88 | 82.76 | 70.68 | 76.72

Table 6: Semantic Segmentation Performance on Replica [55]. The symbol $*$
denotes the results obtained with ground truth semantic labels. The numbers of
[17] are taken from their paper.

###  3.3 Evaluation of Semantic Segmentation

In this part, we evaluate the performance of semantic segmentation on the
Replica dataset [55]. The compared baseline methods, evaluation metrics, and
results of this part are shown in the following.

Baseline and Metrics. For the evaluation of semantic segmentation, we take
Haghighi _et al._ [17] as the comparison baseline. The ground truth semantic
labels for Replica [55] are rendered with Habitat [34] with the original
trajectory. Besides, we map the label set following the setting in [9, 54] for
semantic segmentation evaluation. And following the setting of [17], we use
four commonly used metrics: Total Accuracy (%), Avg.Accuracy (%), mIOU (%),
and fwIOU (%). Besides, we also compare the learned semantic results of our 3D
semantic field with the 2D semantic segmentation model Mask2Former [9].

Qualitative and Quantitative Results. Due to the lack of open-source code for
[17], we conducted semantic experiments and comparisons on four scenes (Room
0, Room 1, Room 2, Office 0) based on their settings [17]. The results of
average metrics are shown in the top part of Section 3.2. The symbol * denotes
the segmentation results obtained with the supervision of semantic annotation
of [55], which is the same to [17]. As can be seen, we outperform [17] on all
averaged metrics of 4 scenes. Especially, for the metric mIOU, we achieve a
significant improvement over [17] (82.61$\%$ v.s. 98.55$\%$).

Besides, we also test the performance with 2D noise inputs which are generated
by Mask2Former [9] for all scenes. As shown in the bottom of Section 3.2,
compared to the noise 2D segmentation, our approach can improve the multi-view
semantic consistency, which leads to 3D consistent semantic representation.
Those quantitative experimental results verify that our approach can learn 3D
consistent semantic information from 2D noisy and multi-view inconsistent
segmentation results. Additionally, we also show some qualitative semantic
segmentation results in Fig. 6. We show the segmentation results of different
views to validate the consistency. The first row shows different view
segmentation results from Mask2Former [9], the second row shows the results
generated by our approaches without semantic fusion (Section 2.3), and the
third row shows the results generated by our approaches. As shown in the
figure, the results generated by Mask2Former usually contain multi-view
inconsistency, e.g. complex edges of furniture, or ambiguous areas. Under
different viewpoints, the same object or region would be recognized as
different categories (different colors represent different categories). Those
noisy and inconsistent semantic observations can lead to unstable or
conflicting 3D semantic information learning without any process. For example,
the segmentation results in the middle row usually contain a lot of noise and
semantic inconsistent areas. For the performance of our approach (the third
row), even with noisy input data, our method can still produce 3D spatially
consistent semantic segmentation results across different views.

###  3.4 Ablation Study

In this part, we perform the ablation study to investigate the effect of each
part in our system.

Experiment | Room 1 | Office 1
---|---|---
Acc.[cm]$\downarrow$ |  Comp.[cm]$\downarrow$ |  Acc.[cm]$\downarrow$ |  Comp.[cm]$\downarrow$
Only PE | 3.42 | 3.78 | 3.16 | 4.61
\+ Cube | 1.30 | 2.48 | 1.14 | 1.88
\+ Tetrahedron | 1.25 | 2.37 | 1.09 | 1.67

Table 7: Reconstruction performance of using different encoding features.

Effect of Multi-view Semantic Fusion. The bottom part of Section 3.2 shows the
performance without multi-view semantic fusion, which is lower than our
approach. From the table, we know that without using our strategy, our
baseline system can also achieve better semantic segmentation results than
Mask2Former [9]. This is because our neural implicit representation can learn
3D consistent representation from noise 2D input, and progressively converge
to a global optimal from 2D multi-view observations.

Effect of Using Different Neural Representation. Here, we show the
reconstruction performance of using different neural representations for our
system. We select two scenes from Replica [55] for evaluation and the results
are shown in Table 7. As can be seen from the table, without any additional
feature encoding, only using PE can lead to a worse reconstruction of the
scene. Using additional features can enhance the encoding capacity of MLPs and
lead to performance improvement. Besides, we also show the performance of
using additional multi-resolution cube-based feature encoding [38]. Compared
with the cube representation, our tetrahedron representation can achieve a
better surface reconstruction.

Figure 8: Tracking Performance on 4 Scenes from ScanNet [12] (ATE RMSE
$\downarrow$ [cm]). The left and right figures respectively show the effects
of using confidence-based pixel sampling and progressive optimization weight
during the camera tracking process.

Effect of Confidence-guided Pixel Sampling. Here, we show the results to
validate the effectiveness of confidence-based pixel sampling for camera
tracking. We select four scenes from ScanNet [12] dataset for evaluation and
the results are shown in the left part of Fig. 8. As can be seen from the
figure, with the guidance of semantic confidence we can reduce the camera
tracking error (ATE RMSE) on the selected scenes. With the help of confidence,
we can avoid sampling pixels in the object edges areas and complex geometric
regions during the tracking process. The pixels in these areas usually have a
relatively slow convergence speed in the mapping process, which will lead to
relatively large tracking errors.

Effect of Progressive Optimization Weight. We show the performance of using
progressive optimization weight for camera tracking. The comparison results of
4 selected scenes are shown in the right part of Fig. 8. As can be seen from
the figure, with progressive optimization weight we can also reduce the camera
tracking error (ATE RMSE) on the selected scenes. The progressive weight can
lead to a coarse-to-fine optimization manner and avoid the model falling into
local minima.

###  3.5 Memory and Running Time

Figure 9: Tracking speed (FPS) and error (ATE RMSE) of different approaches.
We use different tracking iterations to obtain the ATE RMSE erorr curve at
different tracking FPS.

In this part, we show the memory usage and runtime performance of our approach
on the Replica dataset.

For the memory usage of our system, the whole training parameters of our
system are three MLP decoders {$\mathcal{M}_{geo}$,$\mathcal{M}_{rgb}$,
$\mathcal{M}_{sem}$} and multi-resolution tetrahedron features $\Theta$. For
example, on scene Office 0, the whole parameters of MLP decoders and embedding
features are 8.02MB (0.02MB + 8.00MB), which is much less than the parameters
of Point-SLAM, 27.74MB (0.51MB + 27.23MB). For the running speed of our
system, we show the camera tracking FPS with respect to the ATE RMSE (cm) on
the Replica dataset. We perform global bundle adjustments for 10 iterations
when a new keyframe comes, and use 5-50 iterations for camera tracking. The
tracking FPS results with different tracking iterations are shown in Fig. 9.
The runtimes are profiled on a single NVidia RTX 4090 card. We can achieve
good tracking performance at the ~30 tracking FPS with about 0.5 cm tracking
error. Additionally, we also show the performance of ESLAM [23], Co-SLAM [64],
and Point-SLAM [50] in the figure. Compared to ESLAM and Co-SLAM, we can
achieve better camera tracking accuracy at the same speed. Compared to Point-
SLAM, we can achieve faster tracking speed with comparable tracking
performance.

Figure 10: The AR demo on scene Office 4 from Replica [55]. The top-left is
the original scene image, and on the right are three AR objects and the logo
of ISMAR 2024. We render those objects in the original scene with the camera
pose estimated by our approach. The rendered results are shown in the lower
left. For more AR demo videos, please refer to our supplementary material.

###  3.6 AR Application

In this part, we show the application of our system in AR. Our system can not
only localize the pose of the camera but also reconstruct watertight scene
geometry/depth and perform novel view synthesis via our implicit
representation. Therefore, our method can be very suitable for VR/AR
applications. We demonstrate an AR example, as shown in Fig. 10. The scene
image is selected from Office 4 of Replica [55] dataset. We place three
virtual objects and one ISMAR 2024 logo (shown on the right of the figure)
into reconstructed scenes according to the pose and depth recovered by our
system. The rendered AR image is shown at the bottom of Fig. 10, and we
highlighted the virtual objects and logo with red boxes. We can accurately
represent the occlusion relationship between real and virtual contents via our
accurate camera pose and rendered dense depth map. For the multi-view AR demo,
please refer to our supplementary video, which shows we can handle occlusion
between different objects very well.

##  4 Conclusion

In this paper, we present NIS-SLAM, a neural implicit dense semantic RGB-D
SLAM that can model consistent semantic information from inconsistent
segmentation results generated by a pre-trained 2D CNN. For high-fidelity
surface reconstruction and spatial consistent scene understanding, we use a
hybrid representation of high-frequency multi-resolution tetrahedron features
and low-frequency positional encoding for our system. And the multi-view
semantic fusion is proposed to handle the inconsistency of 2D segmentation
results. Besides, the semantic guided pixel sampling and progressive
optimization weight are used for robust camera tracking. Extensive experiments
on a variety of datasets have shown the effectiveness and application of our
proposed system.

Currently, the proposed NIS-SLAM approach relies on the segmentation results
of a close-set model, which limits its applications for open-set world tasks.
Combined with the large language/open-set models may be more practical for
many applications. Compared to methods with explicit representation (point, 3D
Gaussian), our approach cannot recover high-frequency information well.

##  5 ACKNOWLEDGMENTS

This work was partially supported by NSF of China (No.61932003).

## References

  * [1] K. S. Arun, T. S. Huang, and S. D. Blostein.  Least-squares fitting of two 3-d point sets.  IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-9(5):698–700, 1987.
  * [2] D. Azinović, R. Martin-Brualla, D. B. Goldman, M. Nießner, and J. Thies.  Neural RGB-D surface reconstruction.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6290–6301, 2022.
  * [3] M. Bloesch, J. Czarnowski, R. Clark, S. Leutenegger, and A. J. Davison.  CodeSLAM - learning a compact, optimisable representation for dense visual SLAM.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2560–2568, 2018.
  * [4] K. Blomqvist, L. Ott, J. J. Chung, and R. Siegwart.  Baking in the feature: Accelerating volumetric segmentation by rendering feature maps.  CoRR, abs/2209.12744, 2022.
  * [5] C. Campos, R. Elvira, J. J. G. Rodríguez, J. M. M. Montiel, and J. D. Tardós.  ORB-SLAM3: an accurate open-source library for visual, visual-inertial, and multimap SLAM.  IEEE Trans. Robotics, 37(6):1874–1890, 2021.
  * [6] A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su.  Tensorf: Tensorial radiance fields.  In European Conference on Computer Vision (ECCV), 2022.
  * [7] D. Chen, H. Li, W. Ye, Y. Wang, W. Xie, S. Zhai, N. Wang, H. Liu, H. Bao, and G. Zhang.  Pgsr: Planar-based gaussian splatting for efficient and high-fidelity surface reconstruction.  2024\.
  * [8] Y. Chen, X. Chen, X. Wang, Q. Zhang, Y. Guo, Y. Shan, and F. Wang.  Local-to-global registration for bundle-adjusting neural radiance fields.  In EEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8264–8273, 2023.
  * [9] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar.  Masked-attention mask transformer for universal image segmentation.  In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
  * [10] C. Chung, Y. Tseng, Y. Hsu, X. Q. Shi, Y. Hua, J. Yeh, W. Chen, Y. Chen, and W. H. Hsu.  Orbeez-SLAM: A real-time monocular visual SLAM with ORB features and nerf-realized mapping.  CoRR, abs/2209.13274, 2022.
  * [11] B. Curless and M. Levoy.  A volumetric method for building complex models from range images.  In Proceedings of annual conference on Computer graphics and interactive techniques, pp. 303–312, 1996.
  * [12] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner.  ScanNet: Richly-annotated 3D reconstructions of indoor scenes.  In IEEE Conference on Computer Vision and Pattern Recognition, pp. 2432–2443, 2017.
  * [13] A. Dai, M. Nießner, M. Zollhöfer, S. Izadi, and C. Theobalt.  BundleFusion: Real-time globally consistent 3D reconstruction using on-the-fly surface reintegration.  ACM Trans. Graph., 36(3):24:1–24:18, 2017.
  * [14] X. Fu, S. Zhang, T. Chen, Y. Lu, L. Zhu, X. Zhou, A. Geiger, and Y. Liao.  Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene segmentation.  In International Conference on 3D Vision, pp. 1–11, 2022.
  * [15] C. Gao, A. Saraf, J. Kopf, and J.-B. Huang.  Dynamic view synthesis from dynamic monocular video.  In IEEE/CVF International Conference on Computer Vision, pp. 5712–5721, 2021.
  * [16] M. Grinvald, F. Furrer, T. Novkovic, J. J. Chung, C. Cadena, R. Siegwart, and J. Nieto.  Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery.  IEEE Robotics and Automation Letters, 4(3):3037–3044, 2019.
  * [17] Y. Haghighi, S. Kumar, J. P. Thiran, and L. Van Gool.  Neural implicit dense semantic slam.  arXiv preprint arXiv:2304.14560, 2023.
  * [18] M. Han, Z. Zhang, Z. Jiao, X. Xie, Y. Zhu, S.-C. Zhu, and H. Liu.  Reconstructing interactive 3d scenes by panoptic mapping and cad model alignments.  In IEEE International Conference on Robotics and Automation, pp. 12199–12206, 2021.
  * [19] B. K. Horn, H. M. Hilden, and S. Negahdaripour.  Closed-form solution of absolute orientation using orthonormal matrices.  JOSA A, 5(7):1127–1135, 1988.
  * [20] J. Huang, S.-S. Huang, H. Song, and S.-M. Hu.  Di-fusion: Online implicit 3d reconstruction with deep priors.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8932–8941, 2021.
  * [21] M. Jaderberg, K. Simonyan, A. Zisserman, et al.  Spatial transformer networks.  Advances in neural information processing systems, 28, 2015.
  * [22] C. Jiang, H. Zhang, P. Liu, Z. Yu, H. Cheng, B. Zhou, and S. Shen.  H2-Mapping: Real-time dense mapping using hierarchical hybrid representation.  arXiv preprint arXiv:2306.03207, 2023.
  * [23] M. M. Johari, C. Carta, and F. Fleuret.  ESLAM: Efficient dense slam system based on hybrid representation of signed distance fields.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  * [24] J. T. Kajiya and B. V. Herzen.  Ray tracing volume densities.  In SIGGRAPH, pp. 165–174, 1984.
  * [25] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis.  3d gaussian splatting for real-time radiance field rendering.  ACM Transactions on Graphics, 42(4), 2023.
  * [26] X. Kong, S. Liu, M. Taher, and A. J. Davison.  vmap: Vectorised object mapping for neural field slam.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 952–961, 2023.
  * [27] J. Kulhánek and T. Sattler.  Tetra-nerf: Representing neural radiance fields using tetrahedra.  In IEEE/CVF International Conference on Computer Vision, pp. 18412–18423, 2023.
  * [28] A. Kundu, K. Genova, X. Yin, A. Fathi, C. Pantofaru, L. J. Guibas, A. Tagliasacchi, F. Dellaert, and T. Funkhouser.  Panoptic neural fields: A semantic object-aware neural scene representation.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12871–12881, 2022.
  * [29] H. Li, T. Fan, H. Zhai, Z. Cui, H. Bao, and G. Zhang.  Bdloc: Global localization from 2.5d building map.  In IEEE International Symposium on Mixed and Augmented Reality, pp. 80–89, 2021.
  * [30] H. Li, X. Yang, H. Zhai, Y. Liu, H. Bao, and G. Zhang.  Vox-surf: Voxel-based implicit surface representation.  IEEE Trans. Vis. Comput. Graph., 30(3):1743–1755, 2024.
  * [31] H. Li, H. Zhai, X. Yang, Z. Wu, Y. Zheng, H. Wang, J. Wu, H. Bao, and G. Zhang.  Imtooth: Neural implicit tooth for dental augmented reality.  IEEE Trans. Vis. Comput. Graph., 29(5):2837–2846, 2023.
  * [32] L. Liu, J. Gu, K. Z. Lin, T. Chua, and C. Theobalt.  Neural sparse voxel fields.  In Annual Conference on Neural Information Processing Systems, 2020.
  * [33] W. E. Lorensen and H. E. Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  In Seminal graphics: pioneering efforts that shaped the field, pp. 347–353. 1998.
  * [34] Manolis Savva*, Abhishek Kadian*, Oleksandr Maksymets*, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra.  Habitat: A Platform for Embodied AI Research.  In IEEE/CVF International Conference on Computer Vision, 2019.
  * [35] R. Martin-Brualla, N. Radwan, M. S. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duckworth.  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7210–7219, 2021.
  * [36] J. McCormac, A. Handa, A. Davison, and S. Leutenegger.  Semanticfusion: Dense 3d semantic mapping with convolutional neural networks.  In IEEE International Conference on Robotics and Automation, pp. 4628–4635. IEEE, 2017.
  * [37] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In European Conference on Computer Vision, 2020.
  * [38] T. Müller, A. Evans, C. Schied, and A. Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  ACM Trans. Graph., 41(4):102:1–102:15, 2022.
  * [39] J. Munkberg, J. Hasselgren, T. Shen, J. Gao, W. Chen, A. Evans, T. Müller, and S. Fidler.  Extracting triangular 3d models, materials, and lighting from images.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8280–8290, 2022.
  * [40] R. Mur-Artal and J. D. Tardós.  ORB-SLAM2: an open-source SLAM system for monocular, stereo, and RGB-D cameras.  IEEE Trans. Robotics, 33(5):1255–1262, 2017.
  * [41] G. Narita, T. Seno, T. Ishikawa, and Y. Kaji.  Panopticfusion: Online volumetric semantic mapping at the level of stuff and things.  In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4205–4212, 2019.
  * [42] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohli, J. Shotton, S. Hodges, and A. Fitzgibbon.  KinectFusion: real-time dense surface mapping and tracking.  In IEEE International Symposium on Mixed and Augmented Reality, pp. 127–136, 2011.
  * [43] R. A. Newcombe, S. J. Lovegrove, and A. J. Davison.  Dtam: Dense tracking and mapping in real-time.  In IEEE/CVF International Conference on Computer Vision, pp. 2320–2327, 2011.
  * [44] M. Oechsle, S. Peng, and A. Geiger.  UNISURF: unifying neural implicit surfaces and radiance fields for multi-view reconstruction.  In IEEE/CVF International Conference on Computer Vision, pp. 5569–5579, 2021.
  * [45] K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R. Martin-Brualla.  Nerfies: Deformable neural radiance fields.  In IEEE/CVF International Conference on Computer Vision, pp. 5865–5874, 2021.
  * [46] K. Park, U. Sinha, P. Hedman, J. T. Barron, S. Bouaziz, D. B. Goldman, R. Martin-Brualla, and S. M. Seitz.  Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields.  arXiv preprint arXiv:2106.13228, 2021.
  * [47] A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10318–10327, 2021.
  * [48] A. Rosinol, J. J. Leonard, and L. Carlone.  NeRF-SLAM: Real-time dense monocular SLAM with neural radiance fields.  CoRR, abs/2210.13641, 2022.
  * [49] R. A. Rosu and S. Behnke.  Permutosdf: Fast multi-view reconstruction with implicit surfaces using permutohedral lattices.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  * [50] E. Sandström, Y. Li, L. Van Gool, and M. R. Oswald.  Point-slam: Dense neural point cloud-based slam.  In IEEE/CVF International Conference on Computer Vision, 2023.
  * [51] T. Schöps, T. Sattler, and M. Pollefeys.  BAD SLAM: bundle adjusted direct RGB-D SLAM.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 134–144, 2019.
  * [52] Z. Shan, R. Li, and S. Schwertfeger.  Rgbd-inertial trajectory estimation and mapping for ground robots.  Sensors, 19(10):2251, 2019.
  * [53] T. Shen, J. Gao, K. Yin, M. Liu, and S. Fidler.  Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis.  In Advances in Neural Information Processing Systems, pp. 6087–6101, 2021.
  * [54] Y. Siddiqui, L. Porzi, S. R. Bulò, N. Müller, M. Nießner, A. Dai, and P. Kontschieder.  Panoptic lifting for 3d scene understanding with neural fields.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9043–9052, 2023.
  * [55] J. Straub, T. Whelan, L. Ma, Y. Chen, E. Wijmans, S. Green, J. J. Engel, R. Mur-Artal, C. Ren, S. Verma, A. Clarkson, M. Yan, B. Budge, Y. Yan, X. Pan, J. Yon, Y. Zou, K. Leon, N. Carter, J. Briales, T. Gillingham, E. Mueggler, L. Pesqueira, M. Savva, D. Batra, H. M. Strasdat, R. D. Nardi, M. Goesele, S. Lovegrove, and R. Newcombe.  The Replica dataset: A digital replica of indoor spaces.  arXiv preprint arXiv:1906.05797, 2019.
  * [56] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers.  A benchmark for the evaluation of RGB-D SLAM systems.  In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 573–580, 2012.
  * [57] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers.  A benchmark for the evaluation of rgb-d slam systems.  In IEEE/RSJ international conference on intelligent robots and systems, pp. 573–580, 2012.
  * [58] E. Sucar, S. Liu, J. Ortiz, and A. J. Davison.  iMAP: Implicit mapping and positioning in real-time.  In IEEE/CVF International Conference on Computer Vision, pp. 6209–6218. IEEE, 2021.
  * [59] J. Sun, Y. Xie, L. Chen, X. Zhou, and H. Bao.  NeuralRecon: Real-time coherent 3D reconstruction from monocular video.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
  * [60] M. Tancik, V. Casser, X. Yan, S. Pradhan, B. P. Mildenhall, P. P. Srinivasan, J. T. Barron, and H. Kretzschmar.  Block-nerf: Scalable large scene neural view synthesis.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8238–8248, 2022.
  * [61] K. Tateno, F. Tombari, I. Laina, and N. Navab.  Cnn-slam: Real-time dense monocular slam with learned depth prediction.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6243–6252, 2017.
  * [62] Z. Teed and J. Deng.  DROID-SLAM: deep visual SLAM for monocular, stereo, and RGB-D cameras.  In Annual Conference on Neural Information Processing Systems, pp. 16558–16569, 2021.
  * [63] H. Turki, D. Ramanan, and M. Satyanarayanan.  Mega-nerf: Scalable construction of large-scale nerfs for virtual fly- throughs.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12912–12921, 2022.
  * [64] H. Wang, J. Wang, and L. Agapito.  Co-SLAM: joint coordinate and sparse parametric encodings for neural real-time slam.  In IEEE/CVF international conference on Computer Vision and Pattern Recognition, 2023.
  * [65] P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang.  NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  In Annual Conference on Neural Information Processing Systems 2021, pp. 27171–27183, 2021.
  * [66] S. Weder, J. L. Schonberger, M. Pollefeys, and M. R. Oswald.  Neuralfusion: Online depth fusion in latent space.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3162–3172, 2021.
  * [67] Y. Wei, M. Liu, H. Wang, R. Zhu, G. Hu, and W. Zuo.  Learning flow-based feature warping for face frontalization with illumination inconsistent supervision.  In Proceedings of the European Conference on Computer Vision, 2020.
  * [68] T. Whelan, M. Kaess, M. Fallon, H. Johannsson, J. Leonard, and J. McDonald.  Kintinuous: Spatially extended kinectfusion.  In Proceedings of RSS ’12 Workshop on RGB-D: Advanced Reasoning with Depth Cameras, 2012.
  * [69] T. Whelan, R. F. Salas-Moreno, B. Glocker, A. J. Davison, and S. Leutenegger.  ElasticFusion: Real-time dense SLAM and light source estimation.  In International Journal of Robotics Research, vol. 35, pp. 1697–1716, 2016.
  * [70] Q. Xu, Z. Xu, J. Philip, S. Bi, Z. Shu, K. Sunkavalli, and U. Neumann.  Point-nerf: Point-based neural radiance fields.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5438–5448, 2022.
  * [71] X. Yang, H. Li, H. Zhai, Y. Ming, Y. Liu, and G. Zhang.  Vox-Fusion: Dense tracking and mapping with voxel-based neural implicit representation.  In IEEE International Symposium on Mixed and Augmented Reality, pp. 499–507, 2022.
  * [72] H. Zhai, H. Li, X. Yang, G. Huang, Y. Ming, H. Bao, and G. Zhang.  Vox-fusion++: Voxel-based neural implicit dense tracking and mapping with multi-maps.  arXiv preprint arXiv:2403.12536, 2024.
  * [73] Y. Zhang, F. Tosi, S. Mattoccia, and M. Poggi.  Go-slam: Global optimization for consistent 3d instant reconstruction.  In IEEE/CVF International Conference on Computer Vision, pp. 3727–3737, 2023.
  * [74] S. Zhi, T. Laidlow, S. Leutenegger, and A. J. Davison.  In-place scene labelling and understanding with implicit scene representation.  In IEEE/CVF International Conference on Computer Vision, pp. 15838–15847, 2021.
  * [75] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe.  Unsupervised learning of depth and ego-motion from video.  In IEEE/CVF conference on computer vision and pattern recognition, pp. 1851–1858, 2017.
  * [76] S. Zhu, G. Wang, H. Blum, J. Liu, L. Song, M. Pollefeys, and H. Wang.  Sni-slam: Semantic neural implicit slam.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21167–21177, 2024.
  * [77] Z. Zhu, Y. Chen, Z. Wu, C. Hou, Y. Shi, C. Li, P. Li, H. Zhao, and G. Zhou.  Latitude: Robotic global localization with truncated dynamic low-pass filter in city-scale nerf.  In IEEE International Conference on Robotics and Automation, pp. 8326–8332, 2023.
  * [78] Z. Zhu, S. Peng, V. Larsson, Z. Cui, M. R. Oswald, A. Geiger, and M. Pollefeys.  NICER-SLAM: neural implicit scene encoding for RGB SLAM.  CoRR, abs/2302.03594, 2023.
  * [79] Z. Zhu, S. Peng, V. Larsson, W. Xu, H. Bao, Z. Cui, M. R. Oswald, and M. Pollefeys.  NICE-SLAM: neural implicit scalable encoding for SLAM.  In IEEE Conference on Computer Vision and Pattern Recognition, pp. 12786–12796, 2022.