# DNS SLAM: Dense Neural Semantic-Informed SLAM

Kunyi Li1 Michael Niemeyer2 Nassir Navab1,3 Federico Tombari1,2
1Technical University of Munich 2Google 3Johns Hopkins University

###### Abstract

In recent years, coordinate-based neural implicit representations have shown
promising results for the task of Simultaneous Localization and Mapping
(SLAM). While achieving impressive performance on small synthetic scenes,
these methods often suffer from oversmoothed reconstructions, especially for
complex real-world scenes. In this work, we introduce DNS SLAM, a novel neural
RGB-D semantic SLAM approach featuring a hybrid representation. Relying only
on 2D semantic priors, we propose the first semantic neural SLAM method that
trains class-wise scene representations while providing stable camera tracking
at the same time. Our method integrates multi-view geometry constraints with
image-based feature extraction to improve appearance details and to output
color, density, and semantic class information, enabling many downstream
applications. To further enable real-time tracking, we introduce a lightweight
coarse scene representation which is trained in a self-supervised manner in
latent space. Our experimental results achieve state-of-the-art performance on
both synthetic data and real-world data tracking while maintaining a
commendable operational speed on off-the-shelf hardware. Further, our method
outputs class-wise decomposed reconstructions with better texture capturing
appearance and geometric details.

##  1 Introduction

Figure 1: Class-Wise Reconstruction using DNS SLAM. In contrast to previous
neural SLAM systems, our method leads to semantically-decomposed 3D
reconstructions of the scene (top) required for many downstream applications.
Further, by using image-based features, we improve camera tracking and our
reconstructions contain more geometric and appearance details (bottom).

Dense Visual Simultaneous Localization and Mapping (SLAM) is a fundamental
problem in the field of computer vision and plays a crucial role in various
applications such as autonomous driving, indoor robotics, mixed reality, and
more. Its primary goal is to create a 3D map of an unknown environment while
simultaneously estimating camera poses. Traditional SLAM systems [7, 25, 16,
13] rely on multi-view geometry and focus more on camera pose accuracy but are
prone to trajectory drifting. Recent learning-based dense visual SLAM methods
[7, 43, 44, 12, 40, 38, 54, 14, 42, 33, 11] have aimed instead to generate
meaningful global 3D maps, albeit with limited reconstruction accuracy and
demanding extensive training for accurate pose estimation.

Inspired by the advancements in neural field research, neural implicit SLAM
methods like iMAP [38] and NICE-SLAM [54] have recently emerged. These methods
estimate poses from a scene representation, prioritizing world-space geometry
over image-space geometry, showing promising results on synthetic indoor
datasets. They leverage the inherent smoothness and coherence encoded in
Multilayer Perceptron (MLP) weights, making them suitable for sequential
tracking and mapping tasks. However, these methods tend to oversmooth details
in the reconstruction, which causes additional tracking errors. To address
this challenge, Co-SLAM [42] uses parametric embeddings, while Point-SLAM [33]
combines point cloud data with neural implicit representations to enhance
reconstruction details. However, we find that their remarkable performance
heavily relies on accurate pixel-perfect depth supervision. But in the real-
world the depth is usually noisy, and we observe that these methods degrate
drastically in these scenarios.

For robotics and interactive real-world vision applications, a semantic model
that represents scene classes separately is required. vMAP [17] excels at
accurate and complete scene reconstruction by modeling objects separately, but
focuses only on reconstruction and does not perform localization.

To bridge the gap and fully leverage multi-view geometry and semantic
information, we introduce Dense Neural Semantic-Informed (DNS) SLAM. Our
method utilizes class-wise scene representations which can build constraints
between each class and the current camera pose. Further, 2D features from
reference images provide multi-view geometry constraints for better camera
pose estimation. In summary, our main contributions include:

  * •

Leveraging 2D semantic priors, we investigate a semantic-informed multi-class
scene representation, yielding an efficient, comprehensive, and semantically
decomposed geometry representation.

  * •

Utilizing multi-view geometry, we extract image features by back-projecting
points into reference frames, establishing constraints on relative camera
poses and enhancing the appearance details.

  * •

To speed up tracking, we introduce a lightweight coarse scene representation
which is trained with a novel self-supervision strategy, utilizing the multi-
class representation as pseudo ground-truth.

  * •

To achieve accurate and smooth reconstructions, we approximate occupancy
probabilities with Gaussian distributions as the ground-truth for additional
geometry supervision.

Compared with state-of-the-art (SOTA) methods, our method achieves better
performance on camera pose estimation on both synthetic and real-world
datasets, improving ATE RMSE over 10% on average.

##  2 Related Work

Scene Representations. Since the introduction of neural fields in the context
of 3D reconstruction [20, 4, 30], they have been applied to many 3D computer
vision tasks, including novel view synthesis [22], semantic scene
reconstruction [51], and camera pose estimation [2]. Approaches such as Neural
Radiance Fields (NeRF) [21] have achieved remarkable scene reconstruction
results through differentiable rendering. However, it’s worth noting that
coordinate-based encoding methods [54, 13, 6, 43, 44, 45, 48, 8, 1] come with
the drawback of extended training times. To mitigate this, recent approaches
[24, 28, 15] employ parametric encoding techniques expediting the training
process. This enables neural implicit methods for real-time SLAM applications.
While some works focus on improving the efficiency, Pixel-NeRF [47] preserves
the spatial alignment between images and 3D representation, which is also an
important concept in SLAM, by extracting 2D features from reference images and
operating in view-space. Semantic-NeRF [51] provides precise geometry and
semantic outcomes even with minimal semantic guidance. This demonstrates that
with solely 2D semantic supervision, neural implicit methods can generate
compelling 3D semantic scene reconstruction. For real-world SLAM applications,
a semantic model is undoubtedly more meaningful. Our work partly builds upon
the foundations of both Pixel-NeRF and Semantic-NeRF, but takes a significant
step forward by simultaneously optimizing the camera poses while they assuming
pose data is given.

Dense Visual SLAM. Modern SLAM methods follow the overall architecture
introduced by MonoSLAM [7], PTAM [16] and ORB SLAM [25], decomposing the task
into mapping and tracking. Based on the working space of map reconstruction,
SLAM methods can be generally divided into two categories: image view-centric
space and world view-centric space. The former one aims at maintaining multi-
view geometry among keyframes in the dense setting, and the later one anchors
the 3D geometry representation in a uniform world coordinate. DTAM [26] is an
early example of image view-centric methods and has been adapted in many
recent learning-based SLAM system [39, 50, 3, 37]. DeepTAM [52] combines the
cost volume and keyframe image to update the depth prediction. D3VO [46]
models the photometric uncertainties of pixels on the input images, which
improves the depth estimation accuracy. And Droid SLAM [40] uses optical flow
to define geometrical residuals. World view-centric methods store global map
in surfels [34, 45], octrees [41] or grids like voxel grid [13, 6, 43, 44, 45]
and hash grid [15, 28]. KinectFusion [13] performs a frame-to-model camera
tracking strategy and updates the scene geometry via Truncated Signed Distance
Function (TSDF) fusion. RoutedFusion [43] outputs the TSDF update of the
volumetric grid. NeuralFusion [44] extends this concept by learning the scene
representation implicitly. In our method, the benefits from both image view
and world are adopted, using a hybrid scene representation.

Neural Implicit SLAM. More recently, there has been a popularity in neural
implicit representations for dense visual SLAM. iMAP [38], as the first neural
implicit SLAM method, employs an MLP-based representation to conduct joint
tracking and mapping in quasi-real time, but the reconstruction and camera
pose result are far worse than traditional methods. To address computational
overhead and scalability, NICE-SLAM [54] introduces a multi-level feature grid
as scene representation. Nevertheless, the feature grid’s local update
approach limits its hole-filling capabilities. On the other hand, Co-SLAM [42]
attains real-time performance by combining coordinate and sparse parametric
encodings for scene representation, employing dense global bundle adjustment
using rays sampled from all keyframes. E-SLAM [14] suggests the use of multi-
scale axis-aligned feature planes to prevent the model size from growing
cubically concerning the scene’s size. Point-SLAM [33] introduces a neural
point cloud that iteratively grows in a data-driven manner during scene
exploration for both mapping and tracking. ADFP [11] proposes an attentive
depth fusion prior and uses either currently learned geometry or the one from
depth fusion in volume rendering.

##  3 Method

Figure 2: Overview. 1) Scene Representation: For each point along the ray, we
first query a 3D feature from our hash-based feature grid and a 2D feature
from reference image(s). Next, we use multiple geometry together with color
and semantic representations to render occupancy, color, semantic logits to
2D. 2) Tracking: We optimize per-frame camera poses by utilizing a coarse
geometry representation which in turn is trained in a self-supervised manner
using our fine decomposed representation. 3) Mapping: In global and local
bundle adjustment, we jointly optimize the scene representation and camera
poses.

###  3.1 Scene Representation

Notation: Let $\xi_{t}^{c2w}=[R_{t}^{c2w}|\mathbf{t}_{t}^{c2w}]\in SE(3)$ be
the camera-to-world transform of frame $t$, where $R_{t}^{c2w}\in SO(3)$ and
$\mathbf{t}_{i}^{c2w}\in\mathbb{R}^{3}$ are the rotation and translation,
respectively. Denote camera intrinsic matrix as $K\in\mathbb{R}^{3\times 3}$.
For the rest of the manuscript we set $\xi=\xi^{c2w}$ to avoid cluttered
notation. Camera projection and back-projection operator is denoted as
$\pi^{-1}$ and $\pi$ , respectively, where $\pi$ maps a 3D point
$\mathbf{x}^{c}$ in the camera coordinate system into the image coordinate
system and $\pi^{-1}$ maps a pixel $\mathbf{u}$ in image coordinate system
with depth $d(\mathbf{u})$ into the camera coordinate system:

$\pi(\mathbf{x}^{c})\cong
K\mathbf{x}^{c},\pi^{-1}(\mathbf{u},d(\mathbf{u}))\cong
d(\mathbf{u})K^{-1}{\mathbf{u}}$ (1)

Note that we drop explicit homogeneous notation for clarity.

Neural Rendering: The input of our SLAM system is a RGB-D and semantic label
stream $\\{I_{t}\\}^{T}_{t=1},\\{D_{t}\\}^{T}_{t=1},\\{S_{t}\\}^{T}_{t=1}$
with known intrinsics $K$. Our goal is to estimate camera poses
$\\{\hat{\xi}_{t}\\}^{T}_{t=1}$ and train a scene representation $\theta$.
Each pixel in image space $\mathbf{u}\in\mathbb{R}^{2}$ determines a ray in
world space with origin $\mathbf{o}=\mathbf{\hat{t}}_{t}$ and ray direction
$\mathbf{d}={\hat{R}_{t}}K^{-1}\mathbf{u}$. For rendering, we randomly sample
$M$ points $\mathbf{x}_{i}=\mathbf{o}+d_{i}\mathbf{d},i\in\\{1,...,M\\}$ along
the ray where $d_{i}$ denotes depth value of sampled point $\mathbf{x}_{i}$.
We use One-blob [23] encoding $\gamma(\mathbf{x}_{i})$ similar to [42] and a
multi-resolution hash-based feature grid
$\mathcal{V}_{\alpha}=\\{\mathcal{V}_{\alpha}^{l}\\}^{L}_{l=1}$ [24] as
geometry representation. Our method first maps $\mathbf{x}_{i}$ into occupancy
value $\hat{o}_{i}$ and latent vector $\mathbf{h}_{i}$ with geometry
representation:

$f_{\psi}(\gamma(\mathbf{x}_{i}),\mathcal{V}_{\alpha}(\mathbf{x}_{i}))\mapsto(\mathbf{h}_{i},\hat{o}_{i}).$
(2)

We use multiple geometry representations $f_{\psi}^{k}$ for each class in the
scene instead of single one, which will be explained in Sec. 3.2. Then, the
color representation predicts color value $\hat{\mathbf{c}}_{i}$ as:

$f_{\phi}(\gamma(\mathbf{x}_{i}),\mathbf{h}_{i})\mapsto\hat{\mathbf{c}}_{i}.$
(3)

Besides geometry and appearance, we further use a semantic representation to
predict semantic logit value $\hat{\mathbf{s}}_{i}$ as:

$f_{\tau}(\gamma(\mathbf{x}_{i}),\mathbf{h}_{i})\mapsto\hat{\mathbf{s}}_{i}.$
(4)

We set all learnable parameters of our scene representation as
$\theta=\\{\alpha,\psi,\phi,\tau\\}$. We render depth $\hat{d}$, color
$\hat{\mathbf{c}}$ and semantic logit value $\hat{\mathbf{s}}$ by integrating
the predicted values along the sample rays [21, 29]:

$\mathbf{\hat{g}}=\mathbf{\hat{g}}(\mathbf{u};\hat{\xi}_{t},\theta)=\sum_{i=1}^{M}w_{i}\hat{\mathbf{g}}_{i},$
(5)

where $\mathbf{\hat{g}}\in\\{\mathbf{\hat{c}},\hat{d},\mathbf{\hat{s}}\\}$.
The weight $w_{i}$ represents the discretized probability that the ray
terminates at point $\mathbf{x}_{i}$:

$w_{i}=\hat{o}_{i}\prod_{j=1}^{i-1}(1-\hat{o}_{i}).$ (6)

Like traditional SLAM methods [25], our method can be split into two
processes: tracking and mapping. Tracking process first estimates camera pose
of each frame while keeping the scene representations fixed. Mapping process
jointly optimizes both the scene representations and camera poses. More
details will be explained in the following sections. Fig. 2 shows an overview
of our method.

###  3.2 Multi-Class Consistency

Instead of using a single shared MLP similar to previous neural SLAM
approaches, decomposing the prediction into smaller class-specific MLPs can
lead to to a significant performance boost and speedup [31, 18, 17]. While
obtaining 3D semantic priors is difficult in most cases, 2D semantic
segmentation [19, 32, 53] is a well-studied task. As a result, in our method,
we choose to utilize 2D semantic maps as they are often directly provided in
datasets or alternatively can also be obtained from off-the-shelf methods.
While we focus on class-level semantic maps in this work, other decompositions
such as instance-level maps could be further explored in the future.

Multi-Class Rendering: For each pixel $\mathbf{u}$ in the image, first the 2D
semantic map is queried to obtain its corresponding class id
$k=\mathbf{s}(\mathbf{u})$, where $\mathbf{s}(\mathbf{u})$ denotes ground-
truth semantic label at pixel $\mathbf{u}$. In this case, scene representation
Eq. 2 can be rewritten as:

$f_{\psi}^{k}(\gamma(\mathbf{x}_{i}),\mathcal{V}_{\alpha}(\mathbf{x}_{i}))\mapsto(\mathbf{h}_{i},\hat{o}_{i}),$
(7)

and density can be rendered as:

$\hat{d}=\hat{d}(\mathbf{u};\hat{\xi},\psi^{k})=\sum_{i=1}^{M}w_{i}\hat{o}_{i},$
(8)

where $k=1,...,K$, denotes different class. Therefore, the geometry loss
function can be expressed as:

$\begin{matrix}\min&\mathcal{L}_{geo}=\sum_{\mathbf{u}\in|I|}\left\|d(\mathbf{u})-\hat{d}(\mathbf{u};\hat{\xi},\theta)\right\|_{1}\end{matrix}$
(9)

where $d(\mathbf{u})$ denotes ground-truth depth value at pixel $\mathbf{u}$,
and $|I|$ represents a set of pixels sampled from image $I$. We use the
$L_{1}$ norm as our geometry loss $\mathcal{L}_{geo}$.

###  3.3 Multi-View Consistency

Previous neural implicit SLAM methods directly predict RGB colors from latent
representations and they tend to lead to oversmoothed reconstructions. As
shown in image-based rendering approaches such as Pixel-NeRF [47], spatial
image features aligned to each pixel as an input allows the model to learn not
only more texture information, but also scene priors. Therefore, we use
features from 2D images as a conditional input in our model.

Image-Based Feature Pooling: Specifically, for every target frame $I$, we
choose a reference frame $I_{r}$ and extract 2D features $\mathcal{E}(I_{r})$
using off-the-shelf method (ResNet-18 [10] in our case). Next, we cast rays
for the target frame $I$ to get sets of 3D points $\\{\mathbf{x}_{i}\\}$, and
each 3D point
$\mathbf{x}_{i}={\hat{\xi}_{t}}\pi^{-1}(\mathbf{u},d_{i}(\mathbf{u}))$ is
back-projected to the reference frame coordinate to retrieve the corresponding
image feature vectors $\mathbf{w}^{r}_{i}$:

$\mathbf{w}^{r}_{i}=\mathcal{E}(I_{r}(\pi(({\hat{\xi}_{r}})^{-1}{\hat{\xi}_{t}}\pi^{-1}(\mathbf{u},d_{i}(\mathbf{u}))))).$
(10)

Instead of directly passing these features to our scene representation, we
encode each feature with the corresponding reference view
$(\mathbf{o}_{r},\mathbf{d}_{r})$ to obtain intermediate feature vectors:

$f_{\sigma}(\gamma(\mathbf{o}_{r},\mathbf{d}_{r}),\mathbf{w}^{r}_{i})\mapsto\bar{\mathbf{w}}^{r}_{i},$
(11)

where $\sigma$ is the learnable parameters. In multi-reference frames case,
feature vectors from each frame are then aggregated with an average pooling
operator as the final image feature vector
$\mathbf{w}_{i}=(\sum_{r=1}^{N}\bar{\mathbf{w}}^{r}_{i})/N$. With image
feature as conditional input, the color and semantic representation Eq. 3 and
Eq. 4 can be rewirtten as:

$f_{\phi}(\gamma(\mathbf{x}_{i}),\mathbf{h}_{i},\mathbf{w}_{i})\mapsto\hat{\mathbf{c}}_{i},$
(12)
$f_{\tau}(\gamma(\mathbf{x}_{i}),\mathbf{h}_{i},\mathbf{w}_{i})\mapsto\hat{\mathbf{s}}_{i}.$
(13)

Color $\hat{\mathbf{c}}(\mathbf{\mathbf{u}};\hat{\xi}_{t},\theta)$ and
semantic logits $\hat{\mathbf{s}}(\mathbf{\mathbf{u}};\hat{\xi}_{t},\theta)$
are rendered as in Eq. 5. The photometric and semantic loss are defined as:

$\mathcal{L}_{p}=\sum_{\mathbf{u}\in|I|}\left\|\mathbf{c}(\mathbf{u})-\mathbf{\hat{c}}(\mathbf{\mathbf{u}};\hat{\xi}_{t},\theta)\right\|_{2},$
(14)
$\mathcal{L}_{s}=-\sum_{\mathbf{u}\in|I|}\sum_{k=1}(\mathbf{s}^{k}(\mathbf{u})\log(\mathbf{\hat{s}}^{k}(\mathbf{\mathbf{u}};\hat{\xi}_{t},\theta))),$
(15)

where $\mathbf{c}(\mathbf{u}),\mathbf{u}(\mathbf{x})$ are ground-truth color
and semantic logits. For the photometric loss $\mathcal{L}_{p}$, we use the
$L_{2}$ norm and for the semantic loss $\mathcal{L}_{s}$, we use multi-class
cross-entropy loss.

###  3.4 Self-Supervised Coarse Scene Representation

SLAM requires real-time performance, but our multi-class scene representations
can become hardware-intense on smaller-compute devices. To tackle this, we
exploit the fact that while a single MLP is not ideal to represent an entire
scene due to catastrophic forgetting, it can however be used to represent a
small neighborhood of the scene in a lightweight manner and is hence an ideal
candidate for tracking. Thus, we introduce a lightweight coarse scene
representation:

$f_{\varphi}^{c}(\gamma(\mathbf{x}_{i}),\mathcal{V}_{\alpha}(\mathbf{x}_{i}))\mapsto(\mathbf{h}_{i}^{c},\hat{o}_{i}^{c}).$
(16)

Instead of training this coarse scene representation separately, which is time
consuming, latent multi-class scene representations are used to supervise it
during mapping:

$\mathcal{L}_{l}=\sum_{i=1}\left\|\mathbf{h}_{i}-\mathbf{h}_{i}^{c}\right\|_{2},$
(17)

where $\mathbf{h}_{i}$ is latent vectors from Eq. 7. $L_{2}$ norm is applied
on this self-supervised latent loss $\mathcal{L}_{l}$.

###  3.5 Occupancy Probability Approximation

To achieve accurate and smooth reconstructions with detailed geometry, we
propose an occupancy probability approximation loss inspired by the
approximate SDF and feature smoothness losses in [42]. Specifically, for
samples within the truncation region, i.e., points where
$|d_{i}-d(\mathbf{u})|\leq tr$, we assume the occupancy probability follows a
Gaussian distribution with mean value at the observed distance:

${o}_{i}=f(|d_{i}-d(\mathbf{u})|;\mu,\sigma)=\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}\left(\frac{|d_{i}-d(\mathbf{u})|-\mu}{\sigma}\right)^{2}}.$
(18)

We use this approximation as the ground-truth occupancy value for supervision:

$\mathcal{L}_{o}=\sum_{\mathbf{u}\in|I|}\sum_{|d_{i}-d(\mathbf{u})|\leq
tr}\left\|{o}_{i}-\hat{o}_{i}\right\|_{2}.$ (19)

For points that are far from the surface $|d_{i}-d(\mathbf{u})|>tr$, we apply
a free-space loss [27] which forces the occupancy to be zero:

$\mathcal{L}_{fs}=\sum_{\mathbf{u}\in|I|}\sum_{|d_{i}-d(\mathbf{u})|>tr}\left\|\hat{o}_{i}\right\|_{2}.$
(20)

###  3.6 Mapping

Our model consisting of a feature grid $\mathcal{V}_{\alpha}$ and scene
representations $\theta=\\{\alpha,\psi^{k},\phi,\tau,\varphi\\}$ are randomly
initialized at the beginning. During the whole process, 2D feature encoder
$\mathcal{E}$ is fixed. For the first input frame, camera pose is fixed and
only the hash grid and scene representations are optimized. For subsequent
frames, scene representations and camera poses are optimized jointly and
iteratively every $k$ frames.

Keyframe Selection and BA: We follow [54] to choose the keyframes and frames
for bundle adjustment (BA). For each mapping step, bundle adjustment is
applied with current frame, the latest keyframe, and $W-2$ selected keyframes.
Instead of only carrying local BA, which means selecting $W-2$ keyframes which
has overlaps with current frame, we also randomly choose $W-2$ keyframes from
the whole keyframes for global BA.

Reference Frame Selection. We denote these selected BA frames as target
frames. For each target frame, two reference frames are chosen by following
strategy:

  * •

for current frame, the two latest keyframes are chosen,

  * •

for latest keyframe, two previous keyframes are chosen,

  * •

for other keyframe, one previous keyframe and one later keyframe are chosen.

Ray Sampling: Fisrt, $R$ pixels are randomly sampled from $W$ target frames.
$60\%$ of them are randomly sampled among the image plane, $40\%$ of them are
randomly sampled among each class. Then, as describeed in Sec. 3.1, points are
sample along the rays. We use depth guided sampling such that $M_{s}$ points
are sampled near the surface and $M_{a}$ points are sampled in free space.

New Class Initialization: If we encounter a previously unseen class, a new
scene representation will be created and initialized. Then, it will be trained
separately for 100 iterations before it joins the multi-class scene
representations. This allows the system dynamically adding new class, which is
more meaningful in practice.

Mapping Loss: Mapping process is performed via minimizing the loss functions
with respect to the learnable parameters $\theta$ and camera poses $\xi$. The
final loss $\mathcal{L}_{M}$ is

$\displaystyle\mathcal{L}_{M}(\theta,\xi)=$ (21)
$\displaystyle\mathcal{L}_{geo}+\lambda_{p}\mathcal{L}_{p}+\lambda_{s}\mathcal{L}_{s}+\lambda_{l}\mathcal{L}_{l}+\lambda_{o}\mathcal{L}_{o}+\lambda_{fs}\mathcal{L}_{fs}$

where $\lambda_{p},\lambda_{s},\lambda_{l},\lambda_{o},\lambda_{fs}$ are
respective loss weighting factors.

###  3.7 Tracking

For tracking, the hash-based feature grid and the scene representations are
fixed, and only the camera pose are updated. The latest optimized frame is
chosen as the reference frame for extracting features and $R_{t}$ pixels are
randomly sampled among the whole image. We use the same ray sampling strategy
as in mapping. Occupancy values $\hat{o}_{i}^{c}$ and latent vectors
$\mathbf{h}_{i}^{c}$ are obtained by Eq. 16 and latent vectors are passed to
Eq. 12 and Eq. 13 to obtain the color $\hat{\mathbf{c}}_{i}$ and semantic
values $\hat{\mathbf{s}}_{i}$.

Tracking Loss: A modified version of our geometry loss function is used in
tracking, where the geometry term is weighted by the standard deviation of the
depth prediction $\hat{d}_{var}$:

$\mathcal{L}_{geo}^{track}=\sum_{\mathbf{u}\in|I|}\frac{\left\|d(\mathbf{u})-\hat{d}(\mathbf{u};\hat{\xi}_{t},\theta)\right\|_{1}}{{\sqrt{\hat{d}_{var}(\mathbf{u};\hat{\xi}_{t},\theta)}}},$
(22)

where
$\hat{d}_{var}(\mathbf{u};\hat{\xi}_{t},\theta)=\sum_{i=1}^{M}{w_{i}(\hat{d}(\mathbf{u};\hat{\xi}_{t},\theta)-\hat{d}_{i})^{2}}$.

Given an initial guess of current camera pose and under the constant speed
assumption [54], camera poses are iteratively updated by minimizing the
following loss $\mathcal{L}_{T}$:

$\mathcal{L}_{T}({\xi})=\mathcal{L}_{geo}^{track}+\lambda_{p}\mathcal{L}_{p}+\lambda_{s}\mathcal{L}_{s},$
(23)

where $\xi$ is the camera poses and $\lambda_{p},\lambda_{s}$ are respective
loss weighting factors.

##  4 Experiments

###  4.1 Experimental Setup

Datasets We evaluate our method on both synthetic and real-world datasets with
semantic maps. Following other neural implicit SLAM methods, for the
reconstruction quality, we evaluate quantitatively on 8 synthetic scenes from
Replica [35] and qualitatively on 6 scenes from ScanNet [5]. As for camera
pose accuracy, we evaluate quantitatively on both Replica and ScanNet. The
ground-truth poses of Replica are from simulation, while ground-truth poses of
ScanNet are obtained with BundleFusion [6].

Metrics We evaluate the reconstruction quality using
$DepthL1(cm),Accuracy(cm),Completion(cm)$ and $CompletionRatio(\%)$ with a
threshold of 5cm. For evaluation of camera tracking, we adopt $ATE\ RMSE(cm)$
[36]. To evaluate our semantic reconstruction, we report $mIoU$.

Baselines We compare against the state-of-the-art methods NICE-SLAM [54], Co-
SLAM [42], ESLAM [14], Point-SLAM [33] and ADFP [11] as our main baselines for
reconstruction quality and camera tracking. For semantic evaluation, we
compare against the very recent NIDS-SLAM [9] as all other methods do not
predict semantic information.

Implementation Details We perform single GPU training (NVIDIA 2080ti), and for
experiments with default settings, we use $R_{t}=500$ pixels with 30
iterations for tracking and $R=2000$ pixels for mapping with 100 iterations on
Replica, and 200 iterations on ScanNet. For our multi-resolution hash grid
settings, we follow Co-SLAM [42]. We use two-layer MLPs with 32 hidden units
for each scene representation. We use a learning rate of 0.005 and 0.001 for
all learnable parameters on Replica and ScanNet, respectively. For camera
poses, we use a learning rate of 0.001 in tracking and 0.0005 in mapping. We
use the first layer of ResNet [10] output as our 2D features. Please see the
sup. mat. for additional details.

| ScanNet [5] | Replica [35]
---|---|---
| 0000 | 0059 | 0106 | 0169 | 0181 | 0207 | Avg. | room_0 | room_1 | room_2 | office_0 | office_1 | office_2 | office_3 | office_4 | Avg.
NICE-SLAM [54] | 8.64 | 12.25 | 8.09 | 10.28 | 12.93 | 5.59 | 9.63 | 1.69 | 2.04 | 1.55 | 0.99 | 1.39 | 3.97 | 3.08 | 1.95 | 2.08
Co-SLAM [42] | 7.18 | 12.29 | 9.57 | 6.62 | 13.43 | 7.13 | 9.37 | 0.65 | 1.13 | 1.43 | 0.55 | 0.50 | 0.46 | 1.40 | 0.77 | 0.86
ESLAM [14] | 7.3 | 8.5 | 7.5 | 6.5 | 9.0 | 5.7 | 7.42 | 0.76 | 0.71 | 0.56 | 0.53 | 0.49 | 0.58 | 0.74 | 0.64 | 0.62
Point-SLAM [33] | 10.24 | 7.81 | 8.65 | 22.16 | 14.77 | 9.54 | 13.97 | 0.61 | 0.41 | 0.37 | 0.38 | 0.48 | 0.54 | 0.72 | 0.63 | 0.52
ADFP [11] | - | 10.50 | 7.48 | 9.31 | - | 5.67 | 8.24 | 1.39 | 1.55 | 2.60 | 1.09 | 1.23 | 1.61 | 3.61 | 1.42 | 1.81
Ours | 5.42 | 5.20 | 9.11 | 7.70 | 10.12 | 4.91 | 7.07 | 0.49 | 0.46 | 0.38 | 0.34 | 0.35 | 0.39 | 0.62 | 0.60 | 0.45
Table 1:  Quantitative Tracking Comparison. We show ATE RMSE$\downarrow$ (cm)
results of an average of 3 runs on ScanNet [5] and Replica [35]. We find that
our method leads to state-of-the-art performance, improving tracking results
on both synthetic and real-world data.

###  4.2 Tracking Evaluation

We show quantitative results on both Replica [35] and ScanNet [5] in Table 1.
Our method demonstrates state-of-the-art tracking performance on the Replica
dataset, outperforming other methods over 10% on average in terms of ATE RMSE.
It is important to highlight that while both our method and Point-SLAM [33]
achieve impressive tracking results on Replica, our method proves to be more
robust and effective, particularly in complex real-world scenarios like those
encountered in the ScanNet dataset. We attribute this is to the fact that
Point-SLAM strongly relies on accurate depth supervision, while our
approximated occupancy probability formulation provides reliable supervision
and our multi-view consistency constraints can effectively address depth map
ambiguities even if the depth input is not accurate. Fig. 3 shows a
qualitative comparison on ScanNet. Compared with NICE-SLAM [54] and ESLAM
[14], our method suffers from less trajectory drifting and, especially for
those scenes with more texture, our method can improve ATE RMSE significantly.
This highlights the adaptability and robustness of our approach, especially in
challenging scenarios, making it a promising solution for practical
applications.

Figure 3: Qualitative Comparison on ScanNet. The ground-truth camera
trajectory is shown in green and the estimated trajectory is shown in red. Our
method predicts more accurate camera trajectories and does not suffer from
pose drifting.

###  4.3 Reconstruction Evaluation

Qualitative and quantitative reconstruction results on Replica [35] are shown
in Fig. 4 and Table 2, respectively. We find that SDF-based Co-SLAM [42] leads
to good but overly-smoothed reconstructions and both Co-SLAM and NICE-SLAM
[54] do not lead to satisfactory texture reconstruction. In contrast, our
method is able to reconstruct fine geometric structures as well as appearance
details (see e.g. the flowers on the table of ”room_1” or the side table of
”office_4” in Fig. 4). While we improve also quantitatively over NICE-SLAM, we
find that the reconstruction metrics favor the overly-smoothed reconstructions
from the SDF-based methods such as Co-SLAM, mainly due to the fact that errors
in unobserved regions (such as under a table) dominate the evaluation metrics.
We plan to investigate appropriate regularization techniques for unobserved
regions, e.g. based on diffusion priors, for our image-based approach in the
future.

Figure 4: Qualitative Reconstruction on Replica. Our method achieves reconstructions with more geometric and appearance details. Figure 5: Decomposed Reconstruction on Replica. We show the 3D semantic mesh (bottom) and its decomposition with RGB colors (top) of two scenes from Replica. | Method | Depth L1$\downarrow$ | Acc.$\downarrow$ | Comp.$\downarrow$ | Comp. Ratio$\uparrow$
---|---|---|---|---|---
SDF-Based | Co-SLAM | 1.58 | 2.10 | 2.08 | 92.99
ESLAM | 1.18 | 0.97 | 1.05 | 98.60
Occupancy-Based | NICE-SLAM | 3.53 | 2.85 | 3.00 | 89.33
ADFP | 1.81 | 2.59 | 2.28 | 93.38
Ours | 3.16 | 2.76 | 2.74 | 91.73
Table 2: Quantitative Reconstruction Comparison on Replica.

###  4.4 Semantics Evaluation

Table 3 shows the quantitative evaluation of our method in comparison to
another neural semantic SLAM method NIDS-SLAM [9]. We follow NIDS-SLAM and
report the mIoU on four scenes. Our method achieves better results despite
NIDS-SLAM using ORB-SLAM [25] as its tracking processor, and we are the first
neural SLAM method to achieve simultaneous localization, reconstruction and
segmentation all at once. Fig. 5 shows class-wise reconstruction results on
Replica dataset. Our method can represent and reconstruct each class
separately leading to decomposed representations, which can be used as a 3D
prior for further downstream tasks.

Scene | room_0 | room_1 | room_2 | office_0 | Avg.
---|---|---|---|---|---
NIDS-SLAM | 82.45 | 84.08 | 76.99 | 85.94 | 82.37
Ours | 88.32 | 84.90 | 81.20 | 84.66 | 84.77
Table 3:  Semantic Segmentation Comparison on Replica.

###  4.5 Runtime Analysis

We report runtimes on Replica’s room_0 scene in Table 4, using a NVIDIA 2080ti
GPU. The tracking and mapping time are reported per frame. We observe that our
approach effectively balances performance and runtime considerations, enabling
high-fidelity as well as fast SLAM.

Per frame | NICE-SLAM | Co-SLAM | Point-SLAM | Ours
---|---|---|---|---
Tracking | 1.32 s | 0.12 s | 0.85 s | 0.36 s
Mapping | 10.92 s | 0.33 s | 9.85 s | 7.58 s
Table 4:  Runtime Comparison on Replica. Method | w/o multi-class | w/o 2D feature | w/o init. | w/o occ. approx. | full model
---|---|---|---|---|---
Replica | 1.16 | 0.71 | 0.53 | 0.86 | 0.49
ScanNet | 7.52 | 6.22 | 174.4 | 5.77 | 5.20
Table 5:  Quantitative Ablation Study (ATE RMSE).

###  4.6 Ablation Study

In Table 5 we report quantitative results for our method where we ablate
various components which we discuss in greater detail in the following. We
test on scene room_0 from Replica and scene 0059 from ScanNet.

New Class Initialization. We find that if a new class representation is not
initialized appropriately before it is added to the system, it leads to
inadequate RGB predictions not only for the new class but also for existing
classes due to a bleeding effect. It is important to note that this in turn
also leads to an increased camera pose error and sometimes causing further
tracking failures. This is in particular visible on the real-world dataset
ScanNet.

Multi-Class Scene Representations. We ablate our multi-class scene
representations by using only a single representation during the whole
process. Table 5 shows that our full mode leads to higher accuracy considering
ATE RMSE than only using a single geometry representation. The difference is
larger on ScanNet where the input data is noisy and the size of the scene is
larger compared to Replica.

Conditional Image Feature Input. Image features serve as a conditional input
for our system, enforcing a multi-view geometry constraint and enhancing the
texture prediction. Table 5 shows a decline in ATE RMSE on both synthetic and
real-world datasets when removing the conditional input. Fig. 6 shows a visual
comparison of the reconstructed meshes. We find that without 2D image
features, the reconstruction become smoother and contain less details.

Occupancy Probability Approximation. Finally, We investigate the effectiveness
of our occupancy probability approximation and show a qualitative comparison
in Fig. 6. The output mesh tends to be noisier and artifacts appear. In
contrast, the meshes of our full model are smoother while containing fine
details. Table 5 further shows that our full model also leads to better
tracking performance.

Figure 6: Ablation Study. We show that 2D image feature can provide
conditional information for reconstruction (top). And occupancy probability
approximation can help to smooth the reconstruction (bottom).

##  5 Conclusion

We present DNS SLAM, the first method that uses a neural representation for
simultaneously localization, reconstruction and segmentation. We propose to
use a semantically decomposed scene representation in combination with
conditional 2D image features, enforcing stronger geometric constraints and
hence enabling better tracking and more detailed appearance prediction while
being more robust to noisy depth inputs. Our occupancy probability
approximation serves as a strong supervision signal, enabling accurate and
smooth mesh reconstruction. Extensive experiments show that our method
significantly improves the tracking accuracy compared to state-of-the-art
baselines while also providing class-wise decompositions.

Limitations. Our method relies on RGB-D and semantic maps as input. While
being more robust to noisy depth inputs than previous works, we plan to
investigate RGB only input in the future.

## References

  * Azinović et al. [2022] Dejan Azinović, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, and Justus Thies.  Neural rgb-d surface reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6290–6301, 2022.
  * Bian et al. [2023] Wenjing Bian, Zirui Wang, Kejie Li, Jia-Wang Bian, and Victor Adrian Prisacariu.  Nope-nerf: Optimising neural radiance field with no pose prior.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4160–4169, 2023.
  * Bloesch et al. [2018] Michael Bloesch, Jan Czarnowski, Ronald Clark, Stefan Leutenegger, and Andrew J Davison.  Codeslam—learning a compact, optimisable representation for dense visual slam.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2560–2568, 2018.
  * Chen and Zhang [2019] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  In _CVPR_ , 2019.
  * Dai et al. [2017a] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In _CVPR_ , 2017a.
  * Dai et al. [2017b] Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt.  Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration.  _ACM Transactions on Graphics (ToG)_ , 36(4):1, 2017b.
  * Davison et al. [2007] Andrew J Davison, Ian D Reid, Nicholas D Molton, and Olivier Stasse.  Monoslam: Real-time single camera slam.  _IEEE TPAMI_ , 29(6):1052–1067, 2007.
  * Guo et al. [2022] Haoyu Guo, Sida Peng, Haotong Lin, Qianqian Wang, Guofeng Zhang, Hujun Bao, and Xiaowei Zhou.  Neural 3d scene reconstruction with the manhattan-world assumption.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5511–5520, 2022.
  * Haghighi et al. [2023] Yasaman Haghighi, Suryansh Kumar, Jean Philippe Thiran, and Luc Van Gool.  Neural implicit dense semantic slam.  _arXiv preprint arXiv:2304.14560_ , 2023.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , pages 770–778, 2016.
  * Hu and Han [2023] Pengchong Hu and Zhizhong Han.  Learning neural implicit through volume rendering with attentive depth fusion priors.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2023.
  * Huang et al. [2021] Jiahui Huang, Shi-Sheng Huang, Haoxuan Song, and Shi-Min Hu.  Di-fusion: Online implicit 3d reconstruction with deep priors.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8932–8941, 2021.
  * Izadi et al. [2011] Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, et al.  Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera.  In _Proceedings of the 24th annual ACM symposium on User interface software and technology_ , pages 559–568, 2011.
  * Johari et al. [2023] Mohammad Mahdi Johari, Camilla Carta, and François Fleuret.  Eslam: Efficient dense slam system based on hybrid representation of signed distance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 17408–17419, 2023.
  * Kähler et al. [2015] Olaf Kähler, Victor Prisacariu, Julien Valentin, and David Murray.  Hierarchical voxel block hashing for efficient integration of depth images.  _IEEE Robotics and Automation Letters_ , 1(1):192–197, 2015.
  * Klein and Murray [2007] Georg Klein and David Murray.  Parallel tracking and mapping for small ar workspaces.  In _2007 6th IEEE and ACM international symposium on mixed and augmented reality_ , pages 225–234. IEEE, 2007.
  * Kong et al. [2023] Xin Kong, Shikun Liu, Marwan Taher, and Andrew J Davison.  vmap: Vectorised object mapping for neural field slam.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 952–961, 2023.
  * Kundu et al. [2022] Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas J Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser.  Panoptic neural fields: A semantic object-aware neural scene representation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12871–12881, 2022.
  * Long et al. [2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3431–3440, 2015.
  * Mescheder et al. [2019] Lars M. Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In _CVPR_ , 2019.
  * Mildenhall et al. [2021] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Communications of the ACM_ , 65(1):99–106, 2021.
  * Mildenhall et al. [2022] Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul P Srinivasan, and Jonathan T Barron.  Nerf in the dark: High dynamic range view synthesis from noisy raw images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16190–16199, 2022.
  * Müller et al. [2019] Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Novák.  Neural importance sampling.  _ACM Transactions on Graphics (ToG)_ , 38(5):1–19, 2019.
  * Müller et al. [2022] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Transactions on Graphics (ToG)_ , 41(4):1–15, 2022.
  * Mur-Artal et al. [2015] Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos.  Orb-slam: a versatile and accurate monocular slam system.  _IEEE transactions on robotics_ , 31(5):1147–1163, 2015.
  * Newcombe et al. [2011] Richard A Newcombe, Steven J Lovegrove, and Andrew J Davison.  Dtam: Dense tracking and mapping in real-time.  In _2011 international conference on computer vision_ , pages 2320–2327. IEEE, 2011.
  * Niemeyer et al. [2020] Michael Niemeyer, Lars M. Mescheder, Michael Oechsle, and Andreas Geiger.  Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision.  In _CVPR_ , 2020.
  * Nießner et al. [2013] Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Marc Stamminger.  Real-time 3d reconstruction at scale using voxel hashing.  _ACM Transactions on Graphics (ToG)_ , 32(6):1–11, 2013.
  * Oechsle et al. [2021] Michael Oechsle, Songyou Peng, and Andreas Geiger.  Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5589–5599, 2021.
  * Park et al. [2019] Jeong Joon Park, Peter R. Florence, Julian Straub, Richard A. Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In _CVPR_ , 2019.
  * Reiser et al. [2021] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger.  Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps.  In _ICCV_ , 2021.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Sandström et al. [2023] Erik Sandström, Yue Li, Luc Van Gool, and Martin R Oswald.  Point-slam: Dense neural point cloud-based slam.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 18433–18444, 2023.
  * Schops et al. [2019] Thomas Schops, Torsten Sattler, and Marc Pollefeys.  Bad slam: Bundle adjusted direct rgb-d slam.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 134–144, 2019.
  * Straub et al. [2019] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al.  The replica dataset: A digital replica of indoor spaces.  _arXiv preprint arXiv:1906.05797_ , 2019.
  * Sturm et al. [2012] Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers.  A benchmark for the evaluation of rgb-d slam systems.  In _2012 IEEE/RSJ international conference on intelligent robots and systems_ , pages 573–580. IEEE, 2012.
  * Sucar et al. [2020] Edgar Sucar, Kentaro Wada, and Andrew Davison.  Nodeslam: Neural object descriptors for multi-view shape reconstruction.  In _2020 International Conference on 3D Vision (3DV)_ , pages 949–958. IEEE, 2020.
  * Sucar et al. [2021] Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew J Davison.  imap: Implicit mapping and positioning in real-time.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 6229–6238, 2021.
  * Teed and Deng [2018] Zachary Teed and Jia Deng.  Deepv2d: Video to depth with differentiable structure from motion.  _arXiv preprint arXiv:1812.04605_ , 2018.
  * Teed and Deng [2021] Zachary Teed and Jia Deng.  Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras.  _NeurIPS_ , 34:16558–16569, 2021.
  * Vespa et al. [2018] Emanuele Vespa, Nikolay Nikolov, Marius Grimm, Luigi Nardi, Paul HJ Kelly, and Stefan Leutenegger.  Efficient octree-based volumetric slam supporting signed-distance and occupancy mapping.  _IEEE Robotics and Automation Letters_ , 3(2):1144–1151, 2018.
  * Wang et al. [2023] Hengyi Wang, Jingwen Wang, and Lourdes Agapito.  Co-slam: Joint coordinate and sparse parametric encodings for neural real-time slam.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13293–13302, 2023.
  * Weder et al. [2020] Silvan Weder, Johannes Schonberger, Marc Pollefeys, and Martin R Oswald.  Routedfusion: Learning real-time depth map fusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4887–4897, 2020.
  * Weder et al. [2021] Silvan Weder, Johannes L Schonberger, Marc Pollefeys, and Martin R Oswald.  Neuralfusion: Online depth fusion in latent space.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3162–3172, 2021.
  * Whelan et al. [2015] Thomas Whelan, Stefan Leutenegger, Renato Salas-Moreno, Ben Glocker, and Andrew Davison.  Elasticfusion: Dense slam without a pose graph.  Robotics: Science and Systems, 2015.
  * Yang et al. [2020] Nan Yang, Lukas von Stumberg, Rui Wang, and Daniel Cremers.  D3vo: Deep depth, deep pose and deep uncertainty for monocular visual odometry.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 1281–1292, 2020.
  * Yu et al. [2021] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4578–4587, 2021.
  * Yu et al. [2022] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger.  Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction.  _Advances in neural information processing systems_ , 35:25018–25032, 2022.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 586–595, 2018.
  * Zhi et al. [2019] Shuaifeng Zhi, Michael Bloesch, Stefan Leutenegger, and Andrew J Davison.  Scenecode: Monocular dense semantic reconstruction using learned encoded scene representations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11776–11785, 2019.
  * Zhi et al. [2021] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J Davison.  In-place scene labelling and understanding with implicit scene representation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 15838–15847, 2021.
  * Zhou et al. [2018] Huizhong Zhou, Benjamin Ummenhofer, and Thomas Brox.  Deeptam: Deep tracking and mapping.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 822–838, 2018.
  * Zhou et al. [2022] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra.  Detecting twenty-thousand classes using image-level supervision.  In _European Conference on Computer Vision_ , pages 350–368. Springer, 2022.
  * Zhu et al. [2022] Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Oswald, and Marc Pollefeys.  Nice-slam: Neural implicit scalable encoding for slam.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12786–12796, 2022.

##  Appendix A Implementation details

###  A.1 Hyperparameters

In the following, we report implementation details and hyperparameters used
for our method to achieve high-accuracy tracking and mapping.

Default Settings. We use $L=16$ level hash grids with $R_{min}=16$ to
$R_{max}$, where we set the maximal voxel size to 2cm for determining
$R_{max}$. We use 16 bins for the OneBlob encoding of each dimension. For the
network heads, we use two-layer MLPs with 32 hidden units for each scene
presentation. The loss weights we set to $\lambda_{p}=3,\ \lambda_{s}=0.1,\
\lambda_{l}=10,\ \lambda_{o}=10,\ \lambda_{fs}=5$ and the truncation distance
$tr$ is set to 10cm. For ray sampling, we use $M_{s}=15$ and $M_{a}=32$
points. In a forward pass, We sample $R_{t}=500$ pixels during tracking and
$R=2000$ pixels during mapping. In tracking, we estimate camera pose for each
frame. In mapping, bundle adjustment will be applied every 5 frame, which
updates the scene representations and camera poses jointly. The window size
$W$ for bundle adjustment is set to 5 keyframes. For initialization, we start
with 500 iterations for the first frame.

Replica. For the Replica dataset [35], we perform 30 iterations for tracking
and 100 iterations for mapping. We choose a keyframe every 30 frames. We set
the learning rate for all learnable parameters to 0.005 except for the camera
parameters for which we use a learning rate of 0.001 in tracking and 0.0005 in
mapping.

ScanNet. For the ScanNet dataset [5], we use $L=20$ level hash grids with
$R_{min}=16$ to $R_{max}$, where we use a maximal voxel size of 4cm to
determine $R_{max}$. We perform 30 iterations of tracking and 200 iterations
of mapping iteratively. We choose a keyframe every 15 frames. We set the
learning rate for all learnable parameters to 0.001 except for the camera
parameters for which we use a learning rate of 0.001 during tracking and
0.0005 during mapping.

##  Appendix B Additional Experimental Results

###  B.1 More Reconstruction Results

In this section, we report additional reconstruction results. Fig. 8 shows a
top-down view comparison and Fig. 9 shows zoom-in views on Replica. Even
though SDF-based methods such as Co-SLAM [42] and ESLAM [14] lead to better
results in terms of quantitative metrics, we observe that our method leads to
meshes with more geometric details. For example, our method reconstructs the
individual objects in the scene (see e.g. the tiny object on the table of
”room_0” in Fig. 9) while the other methods struggle to do so. We also show
reconstruction results on ScanNet. Fig. 10 shows that our method leadst to
better reconstructions while NICE-SLAM [54] and ESLAM [14] show limited
reconstruction capabilities on these challenging real-world scenes (see e.g.
the water dispenser of ”0059” and the toilet of ”0207” in Fig. 10).

###  B.2 More Semantic Results on Synthetic Scenes

We first show qualitative results for semantic segmentation compared with the
ground-truth in Fig. 11. We are able to accurately segment the 3D space with
errors only occur near the boundaries of objects. Fig. 12 shows decomposition
results of the remaining scenes. Our method enables class-wise decomposed
reconstructions with fine geometric details.

Figure 7: Rendering Results on Replica.

###  B.3 Novel-View Synthesis Evaluation

Table 6 shows a comparison for novel view synthesis performance across 8 senes
from Replica. We follow the evaluation protocol in [33]. Note that our method
can reach on par results with Point-SLAM on SSIM and LPIPs [49] without
generating intermediate point cloud representations which requires significant
computational resource and time. Further, Fig. 7 shows that our method better
maintains the image structure of the ground truth next to the color, which is
especially important for real-world scenes where the input color images can
suffer from bad light conditions or motion blur. We hypothesize that this is
largely coming from incorporating a pre-trained ResNet [10] model, which was
trained for image recognition, and in turn tends to preserve texture details
of images instead of only the RGB color. We plan to implement an additional
exposure compensation mechanism to further improve color prediction in future
research.

###  B.4 Limitation Discussion

Fig. 13 shows our reconstruction of an unobserved region. We observe that our
occupancy-based approach tends to predict a surface continuation in the
unobserved region, while being freespace in the ground truth. We plan to
investigate more complex regularization strategies and partial infilling
approaches to make our approach more robust for sparser captures.

Figure 8: Normal Map of Reconstructions on Replica. Figure 9: Zoom-In View of Reconstructions on Replica. Figure 10: Zoom-In View of Reconstructions on ScanNet. Figure 11: Semantic Segmentation on Replica. From left to right: rendered semantic labels, ground-truth semantic labels and information entropy, highlighting regions with niosy predictions. Figure 12: Decomposed Reconstruction on Replica Figure 13: Failure Case on Replica. . Method | PSNR(dB)$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---|---
NICE-SLAM | 24.42 | 0.809 | 0.233
Point-SLAM | 35.17 | 0.975 | 0.124
Our | 22.96 | 0.963 | 0.119
Table 6: Rendering Results of an average of 3 runs on Replica.