# Uni-Fusion: Universal Continuous Mapping

Yijun Yuan and Andreas Nüchter The authors are with Informatics XVII –
Robotics at Julius-Maximilians-University of Würzburg, Germany.
{yijun.yuan|andreas<EMAIL_ADDRESS>
###### Abstract

We introduce Uni-Fusion, an universal continuous mapping framework for
surfaces, surface properties (color, infrared, etc.) and more (latent features
in CLIP embedding space, etc.). We propose the first Universal Implicit
Encoding model that supports encoding of both geometry and various types of
properties (RGB, infrared, feature and etc.) without the need for any
training. Based on that, our framework divides the point cloud into regular
grid voxels and produces a latent feature in each voxel to form a Latent
Implicit Map (LIM) for geometries and arbitrary properties. Then, by fusing a
Local LIM of new frame to Global LIM, an incremental reconstruction is
approached. Encoded with corresponding types of data, our Latent Implicit Map
is capable to generate continuous surfaces, surface properties fields, surface
feature fields and any other possible options. To demonstrate the capabilities
of our model, we implement three applications: (1) incremental reconstruction
for surfaces and color (2) 2D-to-3D fabricated properties transfers (3) open-
vocabulary scene understanding by producing a text CLIP feature field on
surfaces. We evaluate Uni-Fusion by comparing in corresponding applications,
from which, Uni-Fusion shows high flexibility to various of application while
performing best or competitive. The project page of Uni-Fusion is available at
https://jarrome.github.io/Uni-Fusion/

###### Index Terms:

Mapping, RGB-D perception, Semantic scene understanding, Universal mapping

##  I Introduction

For robotics, 3D perception is crucial for robots to perform interaction with
the surroundings. Operating different types of sensors and perform
reconstructions and scene understandings are essential to 3D perception. In
those tasks, various properties information need to be considered, such as
geometry and surface properties. However, algorithms have to be accordingly
design.

So in this beginning, we post one question: Is it possible to address all
those information with one universal mapping model?

Reconstruction, as one of the most blossom topic in field, has been developped
for decades. During which, batches of works pushed the limits [1, 2, 3, 4, 5,
6, 7, 8, 9, 10, 11, 12, 13, 14]. Given a set of points, reconstruction models
aim to extract the zero-level surface. Those approaches are usually based on
occupancy fields and signed distance functions (SDFs). Occupancy field are
more used in 2D and objective-level shape reconstruction, where Gaussian
Process Occupancy Maps (GPOM), Gaussian Process Implicit Surface (GPIS) and
Hilbert Maps are invented to produce continuous probabilistic occupancy maps.

Figure 1: One Universal Continues Mapping for all reconstructions. Such as
surface, properties including RGB, saliency, style and $...$, even high
dimensional features (CLIP embeddings, and etc).

For scene-level reconstruction, most recent works rely on Signed Distance
Fields. TSDF-Fusion [8], as the most widely used reconstruction model, paved
the way for real-time 3D reconstruction. With the fast developemt of RGB-D
sensors, such as the Kinect or RealSense series, standard models have been
invented [9, 10]. These models utilize discretized Signed Distance Fields
(SDFs) and the Marching Cubes algorithm [15].

To cope the high demand of memory consumption of such representation, recent
techniques relies on Deep Neural Networks for encoding the geometry [11, 12,
16, 14].

Using a sparse grid of local clusters, high dimensional vectors are extracted
for each cluster. With such a representation, researchers have proposed the
fusion on the map of latent vectors instead of the explicit field. The
explicit field is then extracted with arbitrary resolution. Thus, such form of
representation produces a continuous mapping. Those methods are called Neural
Implicit Maps (NIMs).

Encoding functions for recent NIM techniques using a pre-trained model from
objective datasets were shown in [11, 12, 16]. It becomes impossible to pre-
train for context of color or other point properties, since the context
pattern, like color, is much more complex than shape. The only recent solution
for continuous color reconstruction are based on back-propagation of local
latent features [14] or using Neural Radiance Fields (NeRF) [17]. However,
those methods requires adequate epochs to train and are not capable for real-
time. Thus, in this paper aims to fill the gap: We propose a universal model
that encodes arbitrary properties directly without any time-consuming learning
(and training).

Our model is based on Gaussian Process Regression (GPR). Trained from nothing,
we propose to approximate kernels and thus are able to decouple the GPR to
reduce the point cluster to one feature vector. Then relies on the sparse
voxel structure, we build such feature vector locally in each voxel to form a
map of latents. Because our encoding-decoding function is with GPR, the whole
model does not pre-touch any format of data properties. Thus, our Uni-Fusion
is applicable to arbitrary reconstruction applications.

We consider that there does not exist such a model to operate all types of
data for all perception needs of robot. Therefore, based on that encoder, we
introduce Uni-Fusion, a universal model for all types of data, that produces
continuous maps. A selected set of examples of “what Uni-Fusion can do” is
shown in Fig. 1. Given various types of input data, our Uni-Fusion is capable
to encode and produce continuous maps for surfaces, color, styles, and more.
To better explore the potential of such a model, we even construct an field
for high dimensional CLIP-embedding [18].

The contribution of this paper are as follows:

  * •

we propose an universal encoding-decoding model for local information that
does not require any training,

  * •

we introduce the first universal continuous mapping model for construction of
continuous surfaces and various surface property fields, even for high
dimensional features, such as CLIP-embeddings.

  * •

we implement applications to demonstrate the capabilities: (1) Incremental
surface & color reconstruction, (2) 2D-to-3D transfer on 3D canvas, (3) open-
vocabulary scene understanding.

##### Content Overview

In the following we review related works (Section II). We first introduce the
continues mapping developed from 2D scenes to 3D scenes. Then we discuss about
the importance of recent state-of-the-art (SOTA) of neural implicit-based
reconstructions. Then, we introduce the development of Kernel methods where
our method is belonging to. Afterwards, in Section III and Section IV, we
respectively introduce the universal encoder and the universal continuous
mapping model Uni-Fusion that based on such a encoder. Finally, in Section V,
we implement a series of applications to demonstrate the wide applicability of
Uni-Fusion in various scenarios. The evaluation of Uni-Fusion is given in
Section VI to demonstrate its high capabilities. In the end, we describe the
future work of the Uni-Fusion project and conclude the paper.

##  II Related Works

###  II-A Continues Mapping

Continuous Mapping started with 2D scenarios. Gaussian Process Occupancy Maps
(GPOM) [1] first uses Gaussian Process Regression (GPR) to predict a
continuous representation to enable the map construction at arbitrary
resolution. Then, by borrowing the Bayesian Committee Machine (BCM) technique,
incremental GPOM is approached [2, 3].

In the meanwhile, the Hilbert Maps approach [19, 20] were presented. Which is
a mapping approach without an explicit formulation. Hilbert Maps perform
continuous mapping by keeping optimizing parameters in the background.

With increasing popularity of 3D sensors, the focus shifted towards 3D cases.
Gaussian Process Implicit Surface (GPIS), has entered the field [4, 5, 6, 7].
Given zero-level surface points, those methods either sample points along
normal direction and assign distance values or directly utilize derivative
models with normals as labels (we will give more detail in our mapping Section
IV-B). However, those methods majorly work on shapes but not scenes. It is due
to the fact, that GPR based methods naturely suffer under the high
computational costs when large amount of data is involved. Especially when the
test scope moves from 2D to 3D. Thus, we borrow the idea of Neural Implicit
Maps (Section II-B) to address this issue and work on scene reconstruction.

###  II-B Neural implicit based Reconstruction

Neural implicit reconstruction have orginally be introduced to SDF and
Occupancy based reconstruction. The seminal work of [21], i.e., DeepSDF,
utilizes deep model to encode geometry prior with Multi-layer perceptrons
(MLPs) and extract discretized SDF by querying from decoding. Then, an
algorithm similiar to Marching Cubes [15] is applied to extract meshes. For
the latter, Occupancy Networks [22] learn to estimate the occupancy
probability at positions from an implicit function. Meshes are generated with
the Multiresolution IsoSurface Extraction (MISE) technique.

To improve the efficiency of the reconstruction, DeepLS [23] resorts to
multiple local deep priors and reconstructs on a set of local SDFs. Jiang et
al. [24] further proposes to utilize a local implicit grid that further eases
the complexity of learning the encoding. In contrast, Convolutional Occupancy
Networks [25] explore the local latent by replacing the encoder-decoder with
optimization on a grid of local features to ease the burden of simple MLPs.

Figure 2: Uniform, sparse voxels. Each voxel is encoded to one feature vector
$\mathbf{F}_{m}$ [11].

More recently, Neural Implicit Maps (NIM) achieve real-time real-scene 3D
reconstruction. Huang et al. propose DI-Fusion [11], a neural implicit mapping
method that incrementally reconstructs a scene. By fusing on maps of latent
features, DI-Fusion achieves high efficiency of memory while yielding better
reconstructions. Similarly, NeuralBlox [26] also fuse a grid of latent
features given a known external pose estimation. To further improve the
feature quality, BNV-Fusion [16] additionally add post-optimization after
fusion. To handle large scene reconstructions, that may contain loops, NIM-REM
[12] proposes a SE(3)-transformation algorithm on NIM and implements a
mapping&remapping model supporting working alongside Bundle Adjustment or SLAM
with Loop-closing. Thinking outside of the box, Sucar et al. propose iMAP [13]
a novel SLAM pipeline with neural implicit based incremental reconstruction.
With a differentiable rendering model, iMAP is able to online optimize the
reconstruction by minimizing the image distances. Zhu et al. presents NICE-
SLAM [14], that replaces the MLPs with convolutional occupancy grid to further
improve the efficiency and quality of the reconstruction.

In this work, our model also relies on the regular spaced voxels as in Fig. 2
for local encoding-decoding. Since we propose to use one model for all
properties, it is not possible to pre-train such a model. Therefore, we rely
on a kernel method to achieve this goal.

###  II-C Kernel Function Approximation

Kernel methods are not scalable as the mapping from $n$ points to a kernel
requires $O(n^{2})$ time complexity. One possible solution is with kernel
matrix approximation but is not in the scope of this paper. The other
solution, kernel function approximation, proposes to improve the scalibility
of kernel method using explicit vector maps. For example,
$k(\mathbf{x}_{1},\mathbf{x}_{2})\approx\boldsymbol{v}(\mathbf{x}_{1})^{T}\boldsymbol{v}(\mathbf{x}_{2})$
with mapping function $\boldsymbol{v}:\mathbb{X}\rightarrow\mathbb{R}^{l}$
[27]. There are two approaches to address the approximation. One is Random
Fourier Features (RFFs) [28, 29, 30, 31, 32] and the second is Nyström method
[32, 33, 27].

RFFs explicitly handle the shift-invariant kernels by mapping the data using
the Fourier transform technique [32]. However, RFFs are majorly used to handle
shift-invariant kernels and maintain a large $l$. In addition, as RFFs are
data independent, this leads to a significantly worse generalization
performance than Nyström methods [34]. Differently, Nyström methods can be
applied to approximate any positive-definite kernel [27]. It resorts to
finding the eigenfunctions to form the approximation:

$\displaystyle k(\mathbf{x}_{1},\mathbf{x}_{2})=\sum_{i\geq
1}\mu_{i}\varphi_{i}(\mathbf{x}_{1})\varphi_{i}(\mathbf{x}_{2}),$

where $\varphi_{i}$ and $\mu_{i}\geq 0$ are eigenfunctions and eigenvalues of
kernel $k(\mathbf{x}_{1},\mathbf{x}_{2})$ with respect to the probability
measure $q$. With the top-$l$ eigenpairs, the Nyström method approximate a
kernel with $k(\mathbf{x}_{1},\mathbf{x}_{2})=\sum^{l}_{i\geq
1}\mu_{i}\varphi_{i}(x)\varphi_{i}(y)$.

However, Nyström methods are still costly on medium-sized training data and
require $N$-times evaluation of each sample which is not efficient to direct
regression of a continuous map.

We instead encode the local geometry and properties. This means a full space
approximation is not necessary. By restricting the approximation to the
limited space $\mathbf{x}\in[-0.5,0.5]^{3}$, and applying the function in each
local region, we overcome the high-computational burden.

##  III Decoupled Regression as an Universal Encoding

In this section, we propose an universal encoding model for point clouds.
Based on this encoder, in Section IV, we introduce Universal Continuous
Mapping.

###  III-A Decoupled Gaussian Process Regression as Encoder-Decoder

Inspired by DI-Fusion [11] that introduces maps of latent features for
continuous surface prediction, we aim to produce a latent vector for each
local patch. Firstly, an universal encoder design comes from the Gaussian
Process Regression (GPR) which is a widely used techniques for low dimensional
regression. It has been utilized in various mapping models [3, 4]. Given a
dataset with $N$ observation points
$\\{(\mathbf{x}_{n},\mathbf{y}_{n})\\}_{n=1}^{N}$ where point positions are
$\mathbf{x}_{n}\in\mathbb{R}^{d}$ ($d=3$ in this paper) and point properties
are $\mathbf{y}_{n}\in\mathbb{R}^{c}$, the regression model estimation is
$f(\mathbf{x})=E[\mathbf{y}|\mathbf{X}=\mathbf{x}]$.

Gaussian process assumes data are sampled from multivariate Gaussian
distribution, i.e.,

$\begin{bmatrix}\mathbf{y}\\\
\mathbf{y}_{*}\end{bmatrix}\sim\mathcal{N}(\mathbf{0},\begin{bmatrix}\mathbf{K}(\mathbf{X},\mathbf{X})\
\mathbf{K}(\mathbf{X},\mathbf{X}_{*})\\\
\mathbf{K}(\mathbf{X}_{*},\mathbf{X}),\mathbf{K}(\mathbf{X}_{*},\mathbf{X}_{*})\end{bmatrix}).$
(1)

With the derivative in [35], we yield

$\mathbf{y}_{*}|\mathbf{X},\mathbf{y},\mathbf{X}_{*}\sim\mathcal{N}(\mathbf{K}_{*,}\mathbf{K}^{-1}\mathbf{y},\mathbf{K}_{*,*}-\mathbf{K}_{*,}\mathbf{K}^{-1}\mathbf{K}_{,*}).$
(2)

When an additional Gaussian error is appended,
$\mathbf{y}=f(\mathbf{x})+\epsilon$, the covariance of $\mathbf{X}$ is
rewriten as $\mathbf{K}+\delta^{2}_{n}\mathbf{I}$. The regression result is
usually considering the mean

$\mathbf{y}_{*}=\mathbf{K}_{*,}(\mathbf{K}+\delta^{2}_{n}\mathbf{I})^{-1}\mathbf{y}.$
(3)

Which well-reveals the challenge to use Gaussian process regression directly
in large-scale reconstructions, as $\mathcal{O}(n^{3})$ time complexity is
unaffordable. In addition, such a formula (Eq. 3) is not usable as the input
large point cloud data is required to be maintained for the $\mathbf{K}_{*,}$
computation.

To cope with the high-complexity and to avoid the maintenance of the point
cloud, we propose to decouple the GPR to approach the low-dimensional latent
vector for local regions. The decoupling requires an approximation of the
kernel function

$\mathbf{K}(\mathbf{X},\mathbf{X}_{*})\approx
f_{posi}(\mathbf{X})^{T}f_{\text{posi}}(\mathbf{X}_{*})$ (4)

where $f_{\text{posi}}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{l}$. We name
$f_{\text{posi}}$ positional encoding function to be coherent to Neural
Implicit Maps model Di-Fusion [11]. Thus, Eq. 3 is rewriten as

$\begin{split}\mathbf{y}_{*}=f_{\text{posi}}(\mathbf{X}_{*})^{T}f_{\text{enc}}(\mathbf{X},\mathbf{y})\end{split}$
(5)

where the content encoder function is

$f_{\text{enc}}(\mathbf{X},\mathbf{y})=f_{\text{posi}}(\mathbf{X})(f_{\text{posi}}(\mathbf{X})^{T}f_{\text{posi}}(\mathbf{X})+\delta^{2}_{n}\mathbf{I})^{-1}\mathbf{y}\in\mathbb{R}^{l\times
c}.$ (6)

We denote the encoded feature as
$\mathbf{F}_{(\mathbf{X},\mathbf{y})}=f_{enc}(\mathbf{X},\mathbf{y})\in\mathbb{R}^{l\times
c}$. This is the basis for latent map construction in following universal
continuous mapping model (Section IV). To be more specific, for geometry
encoding we have $l=20$ and $c=1$, thus, the feature is a $20$-dimensional
vector. For color encoding we yield $l=20$ and $c=3$.

Correspondingly, the decoding value for inferencing point $\mathbf{x}_{*}$ is

$f_{dec}(\mathbf{x}_{*},F_{(\mathbf{X},\mathbf{y})})=f_{posi}(\mathbf{x}_{*})^{T}F_{(\mathbf{X},\mathbf{y})}.$
(7)

Thus, a signed distance field or surface property field is approached.

In the following, we derive the approximation function.

[width=1]im/eps/encoder1

Figure 3: Interpreting formula with a graph that is coherent to the Encoder-
decoder structure in Neural Implicit Maps [11].

###  III-B Position Encoding with Approximated Kernel Function

Considering our mapping requires encoding of local geometry & property, the
encoding function merely touch points in a limited region ($[-.5,.5]^{3}$ in
our case).

As we introduced in related works, the Nytröm methods, different from the
RFFs, are more accurate because it is dependent to the given points. This
property well match to our application.

Nyström method for kernel approximation starts with eigenfunctions according
to Mercer’s theorem:

$k(\mathbf{x}_{1},\mathbf{x}_{2})=\sum_{i\geq
1}q_{i}\psi_{i}(\mathbf{x}_{1})^{T}\psi_{i}(\mathbf{x}_{2})$ (8)

where $\psi_{i}$ and $q_{i}\geq 0$ are eigenfunctions and eigenvalues of
kernel $k(\mathbf{x}_{1},\mathbf{x}_{2})$ with respect to probability measure
$q$.

Given third set of anchor samples
$\hat{\mathbf{X}}=\\{\hat{\mathbf{x}}_{1},\cdots,\hat{\mathbf{x}}_{N}\\}$, we
eigen-decompose matrix $\mathbf{K}(\hat{\mathbf{X}},\hat{\mathbf{X}})$ for its
eigenpairs $\\{(\lambda_{i},\mathbf{u}_{i})\\}_{i\in\\{1,\cdots,l\\}}$ with
rank $l$.

Then Nyström method produces

$\psi_{i}(\mathbf{x})=\sum_{n}k(\mathbf{x},\hat{\mathbf{x}}_{n})\mathbf{u}_{i,n},\forall
i>1.$ (9)

For simplicity, we rewrite eigenfunction as

$\psi_{i}(\mathbf{x})=\mathbf{K}(\mathbf{x},\hat{\mathbf{X}})\mathbf{u}_{i}.$
(10)

And correspondingly eigenvalue $q_{i}=\frac{1}{\lambda_{i}}$.

To clarify the presentation, we notate $\mathbf{q}=[{q_{1}},\cdots,{q_{l}}]$
and ${\Psi}=[\psi_{1},\cdots,{\psi_{l}}]^{T}$ from Eq. 8 where
${\Psi}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{l}$.

Make the formula further coherent to Eq. 4, we set

$f_{posi}(\mathbf{x})=diag(\sqrt{\mathbf{q}})\Psi(\mathbf{x})$ (11)

where $diag$ produce diagnal matrix. $f_{posi}$ above is the position encoder
in the Eq. 6.

In this paper, we utilize Matérn
kernel111https://en.wikipedia.org/wiki/Matérn_covariance_function:

$k(\mathbf{x}_{i},\mathbf{x}_{j})=\sigma^{2}\frac{2^{1-\nu}}{\Gamma(\nu)}(\sqrt{2\nu}\frac{dist(\mathbf{x}_{i},\mathbf{x}_{j})}{\rho})K_{\nu}(\sqrt{2\nu}\frac{dist(\mathbf{x}_{i},\mathbf{x}_{j})}{\rho})$
(12)

where $\Gamma$ is gamma function, $K_{\nu}$ is the modified Bessel function of
the second kind, $dist$ denotes the euclidean distance, $\sigma$ and $\rho$
are kernel’s hyper-parameters. We utilize half integer $\nu=3+\frac{1}{2}$ and
thus the specific function is

$k(d)=\sigma^{2}(1+\frac{\sqrt{7}d}{\rho}+\frac{2}{5}(\frac{\sqrt{7}d}{\rho})^{2}+\frac{1}{15}(\frac{\sqrt{7}d}{\rho})^{3})exp(-\frac{\sqrt{7}d}{\rho})$
(13)

where $d=dist(\mathbf{x}_{i},\mathbf{x}_{j})$ for short. To approximate above
kernel, we first uniformly sample $256$ points $\hat{\mathbf{X}}$ in
$[-5,5]^{3}$ cube to compute kernel
$\mathbf{K}_{a}=\mathbf{K}(\hat{\mathbf{X}},\hat{\mathbf{X}})$. Then
eigendecomposition is applied to this kernel,
$\mathbf{K}_{a}=\mathbf{U}\Lambda\mathbf{U}^{T}$.

Note that, the encoded feature dimension $l$ used in Section III-A is depend
on the $\mathbf{U}$. Which further determines the map size in the next section
( Section IV). There is a trade-off between the approximation and map size.

Figure 4: Sorted eigenvalues for $\mathbf{K}_{a,a}$’s eigendecomposition.

We plot the eigenvalue from $\Lambda$ in Fig. 4 and find the matrix is mainly
affect by the few pairs with large eigenvalues. Most of the eigenvalues are
lower than 1. We thus choose $l=20$ which is around $0.8$ in that plot to
approximate the kernel while maintain a small feature dimension.

Note that, we only sample and decomposite once. Then encoding-decoding in Fig.
3 merely need to load and reuse the parameters for the $f_{posi}$, $f_{enc}$
and $f_{dec}$. Related works requires pretrain on large dataset of objects
[11, 16] or indoor scenes [14]. But for our model no training is needed.

##  IV Universal Continuous Mapping

The previous Section III proposes universal encoding model for various data.
Taking that function as a basis, in this section, our Universal Continuous
Mapping produces a map of latents to implicitly represent the scene. We name
such scene representation Latent Implicit Maps (LIM). Which supports surface,
surface properties, and high-dimensional surface features.

[width=1]im/eps/graph

Figure 5: Inheritance graph for the class of Latent Implicit Maps (LIM).

The inheritance graph is plotted in Fig. 5. We design a BaseMap for operating
the voxel structure (Section IV-A), dynamically allocating space and fusing
maps (Section IV-E). The derived SurfaceMap (Section IV-B) and PropertyMap
(Section IV-C) process specific data and operate for the specific application.
The LatentMap (Section IV-D) is derived from PropertyMap. The main difference
is that the PropertyMap mainly operates low dimensional properties, such as
color ($c=3$) or infrared ($c=1$) data, etc. While the LatentMap operates much
higher dimensional feature, such as CLIP embeddings [18] ($c=768$), for
different application purposes.

###  IV-A Map Representation

We follow Neural Implicit Maps ( [11, 12, 16]) to use uniform-spaced voxels to
sparsely represent the scene. The scene is
$\mathbf{V}=\\{\mathbf{v}_{m}=(\mathbf{c}_{m},\mathbf{F}_{m},w_{m})\\}$. $m$
is the voxel index. $\mathbf{v}_{m}$ denotes the contents of that voxel which
are $\mathbf{c}_{m}\in\mathbb{R}^{3}$, $\mathbf{F}_{m}\in\mathbb{R}^{l\times
c}$ and $w_{m}\in\mathbb{N}$ respectively representing the voxel center, voxel
latent feature and observed points count.

With a sequence of incremental frames as input, our model constructs local
LIMs (Section IV-B, Section IV-C, Section IV-D) and fuses (Section IV-E) into
a global LIM. Then the resulting explicit map is inferenced from the global
LIM.

###  IV-B Surface Mapping

Because the input point cloud $\mathbf{X}$ is on zero-level of the surface, it
is not adequate to recover a 3D field of scene,
$f_{SDF}:\mathbb{R}^{3}\rightarrow\mathbb{R}$. Thus, we explopit the idea of
Gaussian Process Implicit Surfaces (GPIS) [4, 5, 6, 7] to feed derivatives to
kernel or to additionally sample non zero-level points. Both, derivative-based
and sample-based GPISs use normal information. Thus, we first pre-process
$\mathbf{X}$ to obtain normals $\mathbf{S}$.

####  IV-B1 Using Derivatives based GPIS

From [4], derivatives of a GP are also Gaussian. So the covariance between
data and derivatives is the differentiation of the covariance function [36]:

$\begin{split}cov(\frac{f_{SDF}(\mathbf{x})}{\partial
x_{i}},f_{SDF}((x^{{}^{\prime}})))&=\frac{\partial
k(\mathbf{x},\mathbf{x}^{{}^{\prime}})}{\partial x_{i}}\\\
&=\frac{\partial}{\partial
x_{i}}[f_{posi}(\mathbf{x})]^{T}f_{posi}(\mathbf{x}^{{}^{\prime}}).\end{split}$
(14)

Additionally,

$\begin{split}cov(\frac{f_{SDF}(\mathbf{x})}{\partial
x_{i}},\frac{f_{SDF}(\mathbf{x}^{{}^{\prime}})}{\partial
x_{j}})&=\frac{\partial^{2}k(\mathbf{x},\mathbf{x}^{{}^{\prime}})}{\partial
x_{i}\partial x^{{}^{\prime}}_{j}}\\\ &=\frac{\partial}{\partial
x_{i}}[f_{posi}(\mathbf{x})]^{T}\frac{\partial}{\partial
x^{{}^{\prime}}_{j}}[f_{posi}(\mathbf{x}^{{}^{\prime}})].\end{split}$ (15)

So given points $\\{\mathbf{x}_{n}\\}^{N}_{n=1}$ with normals
$\\{\mathbf{s}_{n}\\}^{N}_{n=1}$ and field values
$\\{\mathbf{y}_{n}=0\\}^{N}_{n=1}$, the positional encoding function for
derivatives is $f_{posi,deri}(\mathbf{x},i)=\frac{\partial}{\partial
x_{i}}[f_{posi}(\mathbf{x})]$. Its corresponding field value is the normal
value $s_{i}$ on the axis $i$. Then,

$f_{posi,gpis}(\mathbf{X})=\\\
[f_{posi}(\mathbf{X}),f_{posi,deri}(\mathbf{x},1),f_{posi,deri}(\mathbf{x},2),f_{posi,deri}(\mathbf{x},3)]$
(16)

with regression values
$\mathbf{y}_{gpis}=[\mathbf{0},\mathbf{s}_{\cdot,1},\mathbf{s}_{\cdot,2},\mathbf{s}_{\cdot,3}]^{T}$,
where $\mathbf{0}=zeros(1,N)$,
$\mathbf{s}_{\cdot,i}=[\mathbf{s}_{1,i},\cdots,s_{N,i}]$.

Therefore the local geometric encoding function is

$f_{enc,GPIS}(\mathbf{X},\mathbf{y},\mathbf{S})=\\\
f_{posi,gpis}(\mathbf{X})(f_{posi,gpis}(\mathbf{X})^{T}f_{posi,gpis}(\mathbf{X})+\sigma_{n}^{2}\mathbf{I})^{-1}\mathbf{y}_{gpis}.$
(17)

By introducing derivation into kernels, the matrix size is enlarged 15-times,
while the encoded feature is still of low dimension $l$:
$F_{\mathbf{X},\mathbf{y},\mathbf{S}}=f_{enc,GPIS}(\mathbf{X},\mathbf{y},\mathbf{S})\in\mathbb{R}^{l}$.

For inferenceing with the points $\mathbf{x}_{*}$, predictions are consistant
to Eq. 7,
$y_{*}=f_{posi}(\mathbf{x}_{*})^{T}F_{\mathbf{X},\mathbf{y},\mathbf{S}}$.

####  IV-B2 Using Sample based GPIS

Sample-based GPIS are mostly used in GPIS researches. This is because such
method does not compute Jacobians and the kernel size can be small.This
highly-reduces the processing cost both on time and memory. Given points
$\\{\mathbf{x}_{n}\\}^{N}_{n=1}$ with normals $\\{\mathbf{s}_{n}\\}^{N}_{n=1}$
and field value $\\{\mathbf{y}_{n}=0\\}^{N}_{n=1}$, sample-based GPIS extends
the dataset by sample points along the normal direction. Corresponding field
values are the signed distance as the sampled points walk along the normal.
Afterwards, with the extended points and distances $\mathbf{X}_{ext}$,
$\mathbf{y}_{ext}$, Eq. 6 is applied. The inference of such model is the same
as derivative-based GPIS Section IV-B1.

For each frame, points are distributed to its corresponding voxel. Then we
encode the local geometry in voxel using Section IV-B1 or Section IV-B2 to
obtain $\mathbf{v}=(\mathbf{c},\mathbf{F},w)$ with $F\in\mathbb{R}^{l}$ the
geometric latent vector. Then local LIMs are fused to a global LIM following
Section IV-E.

To achieve a surface result for visualization, we construct the Signed
Distance Field from global LIM by inferring on sample points. The sample
points are from a grid in each voxel with a certain resolution. By applying
the MarchingCube algorithm on SDF, a surface mesh is obtained.

###  IV-C Surface Property Fields

The previous surface mapping (Section IV-B) can be considered as a special
case of this Surface Property Mapping. But the two mappings actually do not
handling in the same space. More precisely for implementation, we do not
derive the SurfaceMap class from PropertyMap. Instead, as shown in Fig. 5, we
further introduce a BaseMap to perform the common operations and let them
operate specificly on local map construction and visualization, e.g., meshing
and coloring.

We introduce the more general mapping of surface properties. Since the points
are all on zero-level in signed distance fields, the PropertyMap naturally
handles in a subspace of $\mathbb{R}^{3}$, the surface $\mathcal{S}$. A
surface property in this paper could be any properties for each point, such as
color, infrared, and etc. They only differ on $\mathbf{y}$ in Fig. 3 with
dimension $c$.

We set the most widely used data color as an example. Given an observed
colored point cloud $\\{(\mathbf{x}_{n},\mathbf{c}_{n})|n=0\cdots N\\}$ as
input, where $\mathbf{c}_{n}$ denote the RGB. Its surface field properties
values are $\\{\mathbf{y}_{n}=c_{n}\\}^{N}_{n=1}$. We aggregate the values in
two $N\times 3$ matrices $\mathbf{X}$ and $\mathbf{C}$. So encoded feature for
this point cloud is

$\mathbf{F}_{color}=f_{posi}(\mathbf{X})(f_{posi}(\mathbf{X})^{T}f_{posi}(\mathbf{X})+\delta^{2}_{n}\mathbf{I})^{-1}\mathbf{C},$
(18)

where $\mathbf{F}_{color}\in\mathbb{R}^{l\times 3}$.

Since we use $l=20$ in the approximation function for experiments, the color
feature are small. The color map only stores $20\times 3$ float in each voxel
for a continuous color field. Note that, our model does not need any training.
It can be directly applied to different types of data. For the inference,
because the field is on the surface space, we sample points $\mathbf{x}_{*}$
in arbitrary resolution, from the given know mesh or surface construction of a
previous surface mapping (Section IV-B). Following Eq. 7, the inference points
$\mathbf{x}_{*}$ are positional-encoded and directly multiplied with color
feature $\mathbf{F}_{color,m}$ in the corresponding voxel $m$ to obtain its
value $\mathbf{c}_{*}=f_{dec}(\mathbf{x}_{*},\mathbf{F}_{color,m})$.

###  IV-D Surface Feature Fields

Surface Feature Fields are considered as an extensive use of previous Surface
Property Fields (Section IV-C). Here we extend the surface properties scope to
include features. This extension further demonstrates the capability of our
mapping model as it is directly used without any training.

Given an embedding function $\phi:\mathbb{R}^{N\times
3}\rightarrow\mathbb{R}^{N\times c}$ to process the data $\mathbf{X}$, we
approach the feature of each point as surface properties
$\\{\mathbf{y}_{n}=\phi(\mathbf{X})_{\mathbf{x}_{n}}\in\mathbb{R}^{l\times
c}\\}^{N}_{n=1}$.

Following the encoding and fusion as Section IV-C and Section IV-E, we
construct latent implicit maps for the surface feature fields. Thus, arbitrary
resolutional maps of feature are extracted with
$f_{dec}(\cdot,\mathbf{F}_{\cdot})$.

We provide one application in Section V-C: Open Vocabulary Scene
Understanding. Our model constructs a CLIP space feature field on the surface.
And is therefore able to answer a text input. The differences compared to
Surface Property Fields are demonstrated in Fig. 6. In (c), a CLIP text
encoder $f_{text}$ is appended to encode the text command $\boldsymbol{u}$ to
the CLIP feature $\boldsymbol{U}$. Considering the left branch produces a
surface field for CLIP embeddings, our model is capable to find the wanted
region by computing the similarity between features.

[width=1]im/eps/latent

Figure 6: Encoding-decoding diagram in different applications. (a) fold
applications directly obtain point properties ($\mathbf{y}_{\text{m}}$) from
sensor. (b) fold applications achieves point properties with (style, saliency
and etc.) function $\phi_{im}$. (c) fold applications use feature as
$\mathbf{y}_{\text{m}}$ to build a LIM for a (CLIP) feature field. Then a text
command is used for extracting the semantic information.

###  IV-E Map Fusion

We follow Neural Implicit Maps [11] to update the LIM in a voxel-to-voxel
manner:

$\mathbf{F}_{m}\leftarrow\frac{\mathbf{F}_{m}w_{m}+\mathbf{F}^{{}^{\prime}}_{m}w^{{}^{\prime}}_{m}}{w_{m}+w^{{}^{\prime}}_{m}},w_{m}\leftarrow
w_{m}+w^{{}^{\prime}}_{m},$ (19)

where $\mathbf{v}_{m}=(\mathbf{c}_{m},\mathbf{F}_{m},w_{m})$ and
$\mathbf{v}^{{}^{\prime}}_{m}=(\mathbf{c}^{{}^{\prime}}_{m},\mathbf{F}^{{}^{\prime}}_{m},w^{{}^{\prime}}_{m})$
are the voxel $m$ from global and local LIMs respectively.

##  V Applications

To demonstrate the wide range of usagages of our model, we implement a series
of applications:

  1. 1.

Incremental surface & color reconstruction

  2. 2.

3D saliency detection

  3. 3.

Open vocabulary scene understanding

  4. 4.

Surface infrared field

  5. 5.

3D style transfer

Originating from our motivation in inspection and service robotics, we
implement 1) Incremental surface & color reconstruction for visualization of
robot surroundings. For robot exploration, we implement 2) 3D saliency
detection to indicate the salient regions in maps. For recovering object-level
semantic information in environments, we implement 3) open vocabulary scene
understanding to yield the regions containing the objects.. Furthermore, to
demonstrate the flexibility, we implement 4) surface infrared fields and 5) 3D
style transfer for artistic purposes.

In Fig. 6, we classify those 3 applications into 3 categories: (a) directly
obtaining the properties from sensor observation, such as application 1) and
4). (b) processing on sensor data and predict properties, such as application
2), 5). (c) extending (b) to operating beyond latent features, such as
application 3).

Application 1) and 4) are in the first one category. Thus, we mainly describe
1) incremental surface & color reconstruction (Section V-A), while for 4) we
can easily exchange color with infrared. For the second with 2) and 5) in
Section V-B, we mainly describe the usage of fabricated properties. As the
mapping part is redundant to previous category, it will not be detailed. The
third category is the application 3) that maps a LIM for high dimensional
latent fields. We demonstrate that this application provides a flexible
inference in Section V-C.

###  V-A Application: Incremental Reconstruction

Figure 7: Reconstruction and scene understanding applications pipeline. On the
left incremental reconstruction application, external tracking runs in
parallel to reconstruction to provide coarse poses. While reconstructing,
internal tracking refines the pose estimation to have better surface fit.
“$\cdots$” means that we can add more other properties from Section V-A2 and
Section V-B into this pipeline. On the right scene understanding application,
we assume the pose of the frame is pre-known. The upper part of the white line
is the fusing of LIM for the feature field. The lower part infers specific
semantic information by input text command.

In this section, we build an incremental 3D reconstruction application with
RGBD sequences. Since RGBD sequences provide point positions and color values,
the application constructs two kinds of LIMs: one for surface ( Section IV-B),
and one for color ( Section IV-C).

The pipeline is shown in Fig. 7. When a RGBD frame $i$ is fed into the
framework, it is firstly converted to a colored point cloud
($\mathbf{X}\in\mathbb{R}^{N\times 3},\mathbf{Q}\in\mathbb{R}^{N\times 3}$).
Then, tracking module takes $(\mathbf{X},\mathbf{Q})$ to compute the current
pose $\mathbf{T}$. Afterwards we feed transformed point cloud
$\mathbf{X}\mathbf{T}^{T}$ into surface mapping (Section IV-B), colored point
cloud $(\mathbf{X}\mathbf{T}^{T},\mathbf{Q})$ into surface color mapping
(Section IV-C) to obtain the local LIMs. With Eq. 19, we fuse local LIM to
global LIM voxel-to-voxel.

For visualization, we first sample the grid in each voxel and inference with
the global surface LIM to achieve the SDF. The MarchingCube algorithm is
applied to extract the mesh.

With the surface reconstructed, we are able to sample points in an arbitrary
resolution from it and inference with global color LIM to reconstruct the
surface color.

####  V-A1 Tracking

From [14], we find that the current implicit scene representation based
tracking models, i.e., iMAP, DI-Fusion and NICE-SLAM, still have a performance
gap to state-of-the-art tracking approaches like BAD-SLAM and ORB-SLAM2. Thus,
instead of following the neural implicit models to track with frame-to-model
or ray-tracing based optimization, we run ORB-SLAM2 in the other thread to
provide the pose prior.

To note that, the SLAM model is designed for localization which cares less on
reconstruction. This means directly utilizing ORB-SLAM2 provides coarse
surface construction. We further use colored point cloud registration (CPCR)
[37] as the tracking refiner for better surface and color fitness. This is
tested much more robust than more recent tracking algorithms in fusion models.
ORB-SLAM2 runs independently over all frames. Every few frames, CPCR track one
initially posed colored point cloud to compute the odometry in local window.
Mapping is applied afterwards.

####  V-A2 Other types of datas

Application 4) uses a point cloud with infrared, which is simply replacing the
color with infrared. LIM feature dimension is also correspondingly reduced.
For all other kind of point cloud properties, this application is easily
adapted.

###  V-B Application: 2D-to-3D Transfer

Applications like 2) and 4) work easily alongside application 1): incremental
reconstruction (Section V-A) by feeding the fabricated result together with
the point cloud. For example, given RGB-D frames, we detect saliency or
transfer image styles for a fabricated $X$ image. Here $X$ denotes saliency,
style, and etc. Then by unprojecting it together with depth, the fabricated
value is assigned to points for points pairs ($\mathbf{X}$, $\mathbf{C}_{X}$).

Similar to Fig. 7 the reconstruction pipeline, with encoding (Section III-A)
and fusion (Eq. 19), a global LIM for the fabricated properties $X$ is
constructed. Which is also a surface $X$ fields for later inference. It is
also possible to similarly transfer a 2D semantic image to 3D. But it is not
affordable as different types of semantic information may requires a remapping
on the same data. Therefore in the following, we demonstrate construction of a
surface feature field for the scene understanding application that uses one
pass of mapping to meet various needs.

###  V-C Application: Open Vocabulary Scene Understanding

This application follows OpenScene [38], a learning approach that predicts
dense features for 3D scene points. Where the features are co-embedded with
text and image pixels in CLIP feature space. Inspired by this, we designed the
mapping for surface feature fields ( Section IV-D). The difference comparing
to surface property fields is shown in Fig. 6. $\phi$ is taken from pretrained
OpenSeg [18]. The voxel latent $\mathbf{F}_{m}$ is encoded from points and
points $\mathbf{X}_{m}$ features $\mathbf{y}_{m}$. During inferencing, given
open vocabulary text input $\mathbf{u}$ and position input $\mathbf{x}_{*}$,
we obtain the CLIP space features. By computing similarity, the point property
is determined. Pipeline of this application is shown in Fig. 7.

In this application, image encoding function $f_{im}$ and text encoding
function $f_{text}$ are taken from pre-trained model. While $f_{enc}$ and
$f_{dec}$ in our Uni-Fusion are actually deterministic functions. With those
functions, our model construct a continues field for the CLIP feature on
surface.

An highly-interesting and similar work to Uni-Fusion scene understanding
application is VLMaps [39]. This work produces a 2D map while our model
produces a surface CLIP feature field in 3D space.

##  VI Experiments

In this section, we demonstrate the wide-range of usages and high capabilities
of Uni-Fusion on various applications. First, we evaluate Uni-Fusion on
application 1) Incremental Surface and Color Reconstruction with respect to
the SOTAs. Considering that the applications 2) and 5) are a novel topic, we
cannot find related benchmarks for comparison. Thus, we demonstrate the
performance on exist results. Afterwards, application 3) is implemented and
compared to the SOTA zero-shot semantic segmentation models. Finally, for
application 4) as infrared data is not widely used, we capture our own dataset
containing infrared and demonstrate all applications on it.

###  VI-A Implementation Details

In the experiments, we use our sample-based GPIS for local geometry encoding.
For each point, 2 points along normal direction are sampled (one positive, one
negative) with distance $d=0.1$ in the local voxel’s normalized space.
Comparing to derivative-based GPIS, our sample based GPIS is more efficient in
both space and time. For the encoder, we use $256$ anchor points that are
randomly sampled from $[-0.5,0.5]^{3}$. The first $20$ eigen-pairs are
utilized, which yield the feature dimension $20$. The model choosing will be
discussed in ablation study.

Different maps of latents use different granularity. For surface LIM, we use
voxel-size $5$\mathrm{cm}$$. For color which requires later comparison to
NeRF, we use voxel-size $2$\mathrm{cm}$$. For other property LIM and feature
LIM, we use voxel-size $10$\mathrm{cm}$$.

For smoothness of reconstruction, the encoded voxel is overlapped. This means
it uses the double-size to ensure a half space overlap with each neighbor.
Then during meshing, SDF is fetched from the overlapped voxel. While for the
remaining properties, we only sample from its own voxel part.

The implementation runs on a computer with AMD Ryzen 9 5950X 16-core CPU and a
Nvidia Geforce RTX 3090 GPU (24 GB).

###  VI-B Datasets

We evaluate incremental reconstruction on the ScanNet dataset [40], TUM RGB-D
dataset [41], and Replica dataset [13]. Using MSG-Net [42]’s material set, we
transfer styles to the 3D canvas. For Open Vocabulary Scene Understanding, we
evaluate on ScanNet segmentation data [43] and S3DIS dataset [44].

####  VI-B1 ScanNet [40]

ScanNet is a densely annotated RGB-D video dataset. It is captured with the
structure sensor [45] and contains 1513 scenes for training and validation.
For each scene, both images and a 3D mesh is provided, together with their 2D,
3D semantic annotations.

ScanNet provide 312 scenes for validation, which contains a wide range of
various room structures. It has now been widely used in the thorough
evaluation of the performance of reconstruction and semantic segmentation.

####  VI-B2 TUM RGB-D [41]

TUM RGB-D is a benchmark to mainly evaluate the performance of tracking. It is
captured with Microsoft Kinect sensor together with ground-truth trajectory
from the sensor.

####  VI-B3 Replica [13]

Replica dataset is referred to iMAP’s pre-processed dataset [13]. It is a
synthetic rendered RGB-D dataset from given 3D models. The advantage to
include this dataset is that Replica does not have motion blur. Which is
better to evaluate the capability of algorithms on reconstructing surface
color.

####  VI-B4 MSG-Net Style [42]

MSG-Net provide material images for transfering the styles. We select 21style
fold for demonstration. Those images are given in Fig. 13 together with our
result.

####  VI-B5 ScanNet Point Cloud Segmentation Data [43]

For point cloud semantic segmentation benchmarking, PointNet++ [43] pre-
processes the original ScanNet [40] and generates subsampled point clouds and
corresponding annotations for each scene.

####  VI-B6 S3DIS [44] and 2D-3D-S [46]

S3DIS is a semantic segmentation dataset for 3D point clouds. Which is also a
subset of the 2D-3D-S dataset. The 2D-3D-S dataset is a multi-modalities
dataset containing 2D, 2.5D and 3D domains. This dataset is densely annotated
with semantic labels.

Note that 2D-3D-S’s 2D captures is not a RGB-D video as ScanNet. 2D-3D-S’s
images only have small overlap. So it is merely for semantic segmentation and
not capable for incremental reconstruction application.

###  VI-C Baselines

For online surface mapping evaluation, we choose TSDF Fusion [8], iMAP [13],
SOTA DI-Fusion [11] and BNV-Fusion [16] as four baselines methods.

For the color field, we choose TSDF Fusion [8], $\sigma$-Fusion [47], iMAP
[13], NICE-SLAM [14] and even recent hot Neural Radiance Fields model NeRF-
SLAM [17] as five baselines. Even though it is not fair to include NeRF in
comparison, we show in following experiment that Uni-Fusion significantly
reduce the gap.

For Scene Understanding application, we evaluate generalized zero-shot point
cloud semantic segmentation with ZSLPC [48], DeViSe [49] and SOTA 3DGenZ [50]
for comparison.

###  VI-D Metrics

For incremental reconstruction, we evaluate the geometric reconstruction using
Accuracy, Completeness, and F1 score. We follow BNV-Fusion [16] to uniformly
sample $100,000$ points from reconstruction and ground-truth mesh
respectively. Then Accuracy (Completeness) measures the percentage of
reconstruction-to-ground-truth (ground-truth-to-reconstruction) distances that
are lower than $2.5$\mathrm{cm}$$ threshold. F1 score is the harmonic mean of
accuracy and completeness. For tracking performance, we use ATE RMSE.

Then for evaluation of color reconstruction, we follow SOTA on this topic, the
NeRF to render both depth and RGB images to compare the image level Depth L1
and RGB PSNR.

Afterwards, for comparing scene understanding, we follow zero-shot point cloud
semantic segmentation SOTA 3DGenZ to evaluate the Intersection-of-Union (IoU)
and Accuracy.

###  VI-E Reconstruction Results

For evaluation, we first use the ScanNet validation set with 312 sequences to
thoroughly test the geometric reconstruction on a large variant of scenes.
Then, we use TUM RGB-D to compare our modified tracking model with related
works. Because this part is not the main contribution, we give a rough
overview of the tracking results. To fairly evaluate the color reconstruction,
we use the high quality rendered Replica dataset to compare with related
works, including NeRF.

TABLE I: Comparison to ScanNet [40]. Our method generalizes better to various scenes. ∗ indicates the result from our runs of the official BNV-Fusion code. Method |  | Pre-Train
---
with extra dataset
| Train
---
with sequences
Real-time |  Accuracy (%) $\uparrow$ |  Completeness (%) $\uparrow$ |  F1 score $\uparrow$
TSDF Fusion [51] | None | None | $\checkmark$ | 73.83 | 85.85 | 78.84
iMAP [13] | None | Online train |  | 68.96 | 82.12 | 74.96
DI-Fusion [11] | Object Pretrain | None | $\checkmark$ | 66.34 | 79.65 | 72.97
BNV-Fusion [16] | Object Pretrain | Post Optimization |  | 74.90 | 88.12 | 80.56
BNV-Fusion∗ [16] | Object Pretrain | Post Optimization |  | 73.42 | 81.75 | 77.18
Uni-Fusion (Ours) | None | None | $\checkmark$ | 80.43 | 84.91 | 82.44

(a) width=.33

(b) width=.33

(c) width=.33

Figure 8: Quantitative comparison on 312 scenes of the ScanNet validation set.
We demonstrate the performance of SOTA BNV-Fusion and our Uni-Fusion. We sort
our evaluation value and reordered all of the scores. The zigzag pink is the
BNV-Fusion result; we also plot a deep-pink smoothed curve for better
visualization.

####  VI-E1 Evaluation on ScanNet Dataset [40]

We use the 312 diversified scenes from the ScanNet validation set to evaluate
the performance of surface reconstruction. We follow the pure mapping SOTA
BNV-Fusion to take every 10th posed frame as input. Without using any learning
(in contrast iMAP, DI-Fusion, and BNV-Fusion do) or any post optimization (as
BNV-Fusion does), our Uni-Fusion is capable to achieve precise continuous
mapping performance.

As shown in Table I, our Uni-Fusion achieves $+6$ higher accuracy than the
incremental surface reconstruction SOTA BNV-Fusion. Our model does not exceed
on completeness comparing to BNV-Fusion that support completion in post-
optimization. Though, Uni-Fusion’s completion is still much higher than one
other optimization based model iMAP. Overall, our Uni-Fusion model achieves
higher F1-scores that quantifies the overall quality.

Please note that, SOTA BNV-Fusion is not real-time capable, since it requires
post optimization of all fed frames. Without the post-optimization, the real-
time model Di-Fusion shows much worse results. However, our real-time model
Uni-Fusion is able to achieves much better reconstruction quality than these
approaches even without post-optimization.

BNV-Fusion | Uni-Fusion | Ground Truth | BNV-Fusion | Uni-Fusion | Ground Truth
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
Figure 9: Surface reconstruction on ScanNet dataset.

We additionally run BNV-Fusion’s official implementation (emphasized with ∗)
on the 312 videos of ScanNet and do a post element-wise comparison in Fig. 8c.
Our result is the light blue curve, BNV-Fusion’s result is colored with pink.
Scene index is sorted corresponding to the score value of Uni-Fusion. For
better visualization, we smooth BNV-Fusion’s curve and plot it with dark pink.
It is obvious that the score of Uni-Fusion is overall higher than BNV-
Fusion’s. Moreover, we use box-plot to conclude the statistics besides the
curve plot. Uni-Fusion’s scores are distributed in a higher region. For
completeness which is less obvious better, Uni-Fusion’s box is smaller while
in a relative higher position. This means that Uni-Fusion has more stable
completeness result while BNV-Fusion is more likely to get low completeness in
some cases.

To summarize, our model is almost better on all 312 scenes on all accuracy,
completeness and F1-score. Which is also revealed in Table I with BNV-Fusion∗,
that the BNV-Fusion’s official implementation does not exceed Uni-Fusion on
all metrics.

We plot reconstruction on selected scenes from ScanNet in Fig. 9. Both BNV-
Fusion and our Uni-Fusion are able to produce high quality reconstruction. We
see that BNV-Fusion gives lots of small meshes on walls, which are shown as
small particles in the reconstruction. We consider it is because BNV-Fusion
use very small voxel size ($0.02$\mathrm{m}$$) to get a high score. This is
also revealed by their $247\text{\,}\mathrm{M}\mathrm{B}$ mesh in average,
while ours is only $54\text{\,}\mathrm{M}\mathrm{b}$ in average. Furthermore,
our Uni-Fusion’s mesh is more smooth and also provides high-precise color to
the mesh which is not available for the Surface SOTA.

####  VI-E2 Tracking Evaluation on TUM RGB-D Dataset [41]

In the above test, we compare the performance of pure mapping. While tracking
is not the contribution focus in our paper, it is part of the reconstruction
model. We follow the novel reconstruction model NICE-SLAM [14] to evaluate the
camera tracking on the small-scale TUM RGB-D dataset. Our Uni-Fusion uses a
coarse-to-fine strategy for 3D reconstruction tracking. From Table II, it
demonstrates overall better ATE RMSE than other implicit representation
models.

TABLE II: Tracking on TUM RGB-D [41]. ATE RMSE [$\mathrm{cm}$] ($\downarrow$)
is used as the evaluation metric.

| fr1/desk | fr2/xyz | fr3/office
---|---|---|---
iMAP [13] | 4.9 | 2.0 | 5.8
iMAP∗ [13] | 7.2 | 2.1 | 9.0
DI-Fusion [11] | 4.4 | 2.3 | 15.6
NICE-SLAM [14] | 2.7 | 1.8 | 3.0
Ours | 1.8 | 0.5 | 2.1
BAD-SLAM[52] | 1.7 | 1.1 | 1.7
Kintinuous[53] | 3.7 | 2.9 | 3.0
ORB-SLAM2[54] | 1.6 | 0.4 | 1.0

On the other hand, there also exist high accuracy algorithms from SLAM. By
additional using Bundle Adjustment and Loop-closing techniques, their tracking
quality is much better than all of the implicit based models.

Even though, our coarse-to-fine strategy firstly ensure not easy to lose
track. Secondly, it is more suitable for surface fitting.

Which further support our test in Replica dataset.

TABLE III: Geometric (L1) and Photometric (PSNR) evaluation on the Replica dataset [13]. |  |  office-0 |  office-1 |  office-2 |  office-3 |  office-4 |  room-0 |  room-1 |  room-2 | Avg.
---|---|---|---|---|---|---|---|---|---|---
Non-continuous mapping method |  |  |  |  |  |
TSDF-Fusion [8] |  Depth L1 [$\mathrm{cm}$] $\downarrow$ | 14.11 | 10.50 | 30.89 | 28.92 | 22.83 | 23.51 | 20.94 | 23.34 | 21.88
PSNR  [$\mathrm{dB}$] $\uparrow$ | 11.16 | 15.92 | 4.86 | 5.68 | 5.46 | 3.43 | 4.51 | 5.57 | 7.07
$\sigma$-Fusion[47] |  Depth L1 [$\mathrm{cm}$] $\downarrow$ | 13.80 | 10.21 | 22.27 | 28.70 | 22.21 | 21.92 | 19.28 | 22.40 | 20.10
PSNR  [$\mathrm{dB}$] $\uparrow$ | 11.16 | 15.92 | 4.86 | 5.69 | 5.46 | 3.45 | 4.51 | 5.57 | 7.08
Continuous mapping method |  |  |  |  |  |
iMAP∗ [13] |  Depth L1 [$\mathrm{cm}$] $\downarrow$ | 6.43 | 7.41 | 14.23 | 8.68 | 6.80 | 5.70 | 4.93 | 6.94 | 7.64
PSNR  [$\mathrm{dB}$] $\uparrow$ | 7.39 | 11.89 | 8.12 | 5.62 | 5.98 | 5.66 | 5.31 | 5.64 | 6.95
Nice-SLAM [14] |  Depth L1 [$\mathrm{cm}$] $\downarrow$ | 1.51 | 0.93 | 8.41 | 10.48 | 2.43 | 2.53 | 3.45 | 2.93 | 4.08
PSNR  [$\mathrm{dB}$] $\uparrow$ | 22.44 | 25.22 | 22.79 | 22.94 | 24.72 | 29.90 | 29.12 | 19.80 | 24.61
Uni-Fusion (Ours) |  Depth L1 [$\mathrm{cm}$] $\downarrow$ | 0.79 | 0.56 | 1.59 | 2.71 | 1.66 | 1.94 | 0.69 | 1.80 | 1.47
PSNR  [$\mathrm{dB}$] $\uparrow$ | 33.88 | 33.31 | 25.84 | 26.01 | 28.14 | 24.02 | 26.20 | 27.17 | 28.07
Neural radiance field method |  |  |  |  |  |
NeRF-SLAM [17] |  Depth L1 [$\mathrm{cm}$] $\downarrow$ | 2.49 | 1.98 | 9.13 | 10.58 | 3.59 | 2.97 | 2.63 | 2.58 | 4.49
PSNR  [$\mathrm{dB}$] $\uparrow$ | 48.07 | 53.44 | 39.30 | 38.63 | 39.21 | 34.90 | 36.95 | 40.75 | 41.40
TABLE IV: Differences among different Surface & Color reconstruction models. Method |  | Pre-Train
---
with extra dataset
| Train
---
with sequences
Real-time | Direct Output |  | Light
---
direction
Render
TSDF-Fusion | None | None | $\checkmark$ | Discrete TSDF |  | Ray Rasterization
$\sigma$-Fusion | None | None | $\checkmark$ | Discrete TSDF |  | Ray Rasterization
iMAP | None | Online Train |  | MLPs |  | Volumetric Rendering
NICE-SLAM |  | Pretrain
---
with indoor dataset
Online Train |  | Neural Implicit Grid |  | Volumetric Rendering
NeRF-SLAM | None | Train hundred epochs | - | NeRF | $\checkmark$ | Volumetric Rendering
Uni-Fusion | None | None | $\checkmark$ | Latent Implicit Map |  | Ray Rasterization

####  VI-E3 Evaluation on Replica RGB-D Dataset [13]

In this evaluation, we compare with implicit reconstruction (TSDF-Fusion,
$\sigma$-Fusion) and latent implicit reconstruction models (iMAP, NICE-SLAM)
that support colors. We also add a large-scale NeRF model, NeRF-SLAM in to the
table. NeRF is SOTA in view-synthesis task, which is unfair to direct compare
with the rest. As the rest model does not even model light directions. We add
NeRF in this part to demonstrate that Uni-Fusion strongly reduce the gap. Note
that, NeRF-SLAM embeds external tracking model [55] to provide poses while
using SOTA NeRF implementation Instance-ngp [56] for NeRF construction.

Uni-Fusion track and follow our previous setting in ScanNet test to take every
10 frames for mapping. NICE-SLAM and NeRF-SLAM produce depth and color by
rendering, To obtain result from Uni-Fusion, we cast rays from virtual camera
to our result surface mesh for depth image. Then Uni-Fusion infer the cast
points in Uni-Fusion’s color LIM for color result.

From Table III, Uni-Fusion demonstrate best Depth L1 on all scenes with an
average of $\boldsymbol{1.47}$$\mathrm{cm}$ depth L1. Which is
$\boldsymbol{177\%}$ boost comparing to the second best.

Moreover, excluding NeRF, our Uni-Fusion also shows the best performance to
model the colors with an average of $28.07$$\mathrm{dB}$ PSNR.

However, it is strange that NICE-SLAM lost details while in two cases, it
shows better PSNR than Uni-Fusion. To highlight the true result, we plot the
rendered image in Fig. 10. It is obvious that our Uni-Fusion models the
details of painting, carpet and quilt well. While NICE-SLAM just roughly
models the average color.

Moreover, from the Fig. 10, our Uni-Fusion’s rendering quality is as precise
as NeRF. Please also find the painting, carpet and quilt, Uni-Fusion recovered
the original appearance well. Please find the  green window for the emphasized
region. Uni-Fusion reproduce the high-quality appearance which is very close
to NeRF on qualitative evaluation. But, Uni-Fusion still has a quantitative
score gap to the color rending of NeRF ($41.4$$\mathrm{dB}$). Though the Uni-
Fusion’s rendering result is highly close to NeRF and ground truth. We
consider the main reasons are that 1. Uni-Fusion does not model the light
directions to points, which is essential to NeRF. 2. NeRF optimizes on the
rendering image quality by focussing mainly on color while less on depth. It
can be revealed by the higher color rendering score with much worse depth
rendering than our Uni-Fusion. 3. our Uni-Fusion does not support hole
filling. This directly leads to black holes in our rendered images.

We summarize the differences in Table IV. Similar to TSDF-Fusion and
$\sigma$-Fusion, our Uni-Fusion is a forward method which, does not need any
training, i.e., pre- or online training. Uni-Fusion also produces similar to
NICE-SLAM and NeRF-SLAM an implicit map with set of latent that outputs
results at arbitrary resolution. However, we differ on the extracting of the
signed distance field. Uni-Fusion’s latent feature rule its own region
independently. Each query value is directly inferred with the corresponding
ruling latent. While NICE-SLAM and NeRF-SLAM use a much denser grid to
interpolate query features. This requires volumetric rendering for inference.

Similar to TSDF-Fusion, $\sigma$-Fusion, our Uni-Fusion is also a real-time
algorithm. iMAP, NICE-SLAM and NeRF-SLAM run hardly in real-time. NeRF-SLAM is
claiming to be real-time, which is questionable as they still need hundreds of
epochs training after feeding the data.

Nevertheless, optimization with backpropagation learns pixel-to-pixel well. It
is theoretically advanced for a regression-and-fusion strategy. Though Uni-
Fusion demonstrates its high capability to model the color, NeRF-like post-
optimization would still be a good direction for further improvements of Uni-
Fusion.

NICE-SLAM | NeRF-SLAM | Uni-Fusion | Ground Truth
---|---|---|---
|  |  |
|  |  |
|  |  |
|  |  |
Figure 10: Demonstration of color rendering on the Replica dataset. Fine
appearances are highlighted in green window. Small flaws are in a red box.

###  VI-F Ablation study

Figure 11: Ablation study on surface construction basis. (a) Sample based. (b)
Derivative based. TABLE V: Ablation study on tracking.

| fr1/desk | fr2/xyz | fr3/office
---|---|---|---
External | 2.1 | 0.5 | 2.5
External+Internal | 1.8 | 0.5 | 2.1

Figure 12: Ablation study on voxel size.

|  |  |  |
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |

Figure 13: Style transfer on 3D canvas.

####  VI-F1  Sample-based or Derivative-based

We select the surface model with our own captured sequences. All settings are
detailed in Section VI-A. As shown in Fig. 11, reconstruction of Yijun’s
office is demonstrated. Both models are able to construct, but the derivative-
based model produces a lot of noise on the surface. This is because for
smoothness purpose, we build voxels that are overlapped to its neighbor, which
causes redundant voxels near the surface. For those redundant voxels, no
center sample is provided and thus the derivative based surface construction
builds bad SDFs on unknow region of the voxels.

Instead, sample-based surface construction does not have this problem as it
adds more points in voxels and is able to construct highly-smooth surfaces.
From which, we find well constructed and colored white board, chair, school
bag and even the oranges.

####  VI-F2 Tracking

Our Uni-Fusion use a coarse-to-fine strategy for tracking. An external
tracking model is running in one thread aside from the mapping thread. In the
mapping thread, it takes pose result from the external tracking and applies
the internal tracking for colored point cloud.

The result is demonstrated in Table V. The coarse-to-fine is relatively better
on trajectory estimation.

####  VI-F3 Voxel size

Testing the office scene, we vary the voxel size from low to high. From Fig.
12, when low voxel size $0.02$m is used, the surface is rough. Then when voxel
size goes larger, the smoothness is improved. However, when we use $0.1$m
voxel size, the surface color is blur. Considering Uni-Fusion produces a
surface color field, the quality of surface directly affect the coloring.
Thus, continuing enlarging the voxel size also results in worse surface
results.

Therefore, in the above experiments, $0.05$m voxel size is utilized for
surface construction. In addition, each voxel for encoding are actually with
size $0.1$m, since we use overlapped voxel.

| scene0568_00 | scene0249_00 | scene0435_00 | office3 | room0
---|---|---|---|---|---
Color |  |  |  |  |
Saliency |  |  |  |  |
Figure 14: Saliency transfer on the 3D canvas. The upper row is the result of
the colored mesh. The lower row is saliency mesh.

###  VI-G Results for 2D-to-3D transfer

Except for surface and color, Uni-Fusion also supports continuous mapping of
other fabricated data. Therefore, we implement Uni-Fusion on style-transfer
data and saliency data to approach a style- and saliency-transfer on the 3D
canvas.

In Fig. 13, we demonstrate artistic painting in 3D canvas. We use MSG-Net [42]
to transfer style for each fed frame and use Uni-Fusion to construct the style
LIM for the surface style field. The test scene is office0 of the Replica
dataset from one view angle. $20$ images are used to provide the style and are
attached at bottom-left corner correspondingly. We see that our Uni-Fusion
well-constructs style meshes in the taste of the provided style images. For
example, with pure style or abstract painting, the 3D “canvas” demonstrates a
very close style. Our favorite style is in the middle of the fourth row, it
even produces a high quality 3D sketch painting.

In Fig. 14, we demonstrate saliency detection in 3D. In this figure we select
three scenes from ScanNet and two scenes from Replica. Similarly, we detect
saliency on each frame using InSPyReNet[57] and construct saliency LIM for
surface saliency field.

In the second row, high salient regions are colored in yellow. Here we see the
object that is interest. This further directs a robot in 3D scene. For
example, in first column, sofa, chair, curtain and television in the room
surely gets more attention in daily life. In the second, a meeting room,
obviously the long desk and chairs are the major components. Also for the bed
in the third column of the hotel room, the sofa, desk, lamp, television in the
fourth and the sofas and chairs in the last.

TABLE VI: GZSL semantic segmentation results. Scores are in %. † indicate 3DGenZ’s adaption of the method. Note that, Uni-Fusion does not even train with the seen classes. | Training set | Inference input | ScanNet | S3DIS
---|---|---|---|---
| Backbone | Classifier |  | $Seen$ | $Unseen$ | $All$ | $Seen$ | $Unseen$ | $All$
Supervised methods with different levels of supervision |  |  |  |  |
Full supervision | $seen\cup unseen$ | $seen\cup unseen$ | Point Cloud | 43.3 | 51.9 | 45.1 | 74.0 | 50.0 | 66.6
ZSL backbone | $seen$ | $seen\cup unseen$ | Point Cloud | 41.5 | 39.2 | 40.3 | 60.9 | 21.5 | 48.7
ZSL-trivial | $seen$ | $seen$ | Point Cloud | 39.2 | 0.0 | 31.3 | 70.2 | 0.0 | 48.6
Generalized zero-shot-learning methods |  |  |  |  |
ZSLPC-Seg [48]† | $seen$ | $unseen$ | Point Cloud | 28.2 | 0.0 | 22.6 | 65.6 | 0.0 | 45.3
DeViSe-3DSeg [49]† | $seen$ | $unseen$ | Point Cloud | 20.0 | 0.0 | 16.0 | 70.2 | 0.0 | 48.6
3DGenZ [50] | $seen$ | $seen\cup\hat{unseen}$ | Point Cloud | 32.8 | 7.7 | 27.8 | 53.1 | 7.3 | 39.0
Zero-shot learning + map fusion |  |  |  |  |
Uni-Fusion-SU (Ours) | None | None | Sparse Frames | 31.0 | 41.9 | 32.9 | 31.3 | 24.0 | 29.0

3DGenZ | Uni-Fusion | Ground Truth | 3DGenZ | Uni-Fusion-SU | Ground Truth
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |

Figure 15: Semantic segmentation demo on the ScanNet dataset. Selected scenes
are consistent to Fig. 9

###  VI-H Scene Understanding Results

The saliency detection highlights the interesting object well. This is
considered part of 3D semantic understanding. However, the semantics may vary
and we find it is inefficient to fuse different semantic information in
multiple LIMs. Thus in this section, we use Uni-Fusion to fuse and build a
surface field for high-dimensional CLIP-embeddings. In this way, with one LIM,
we construct different semantic results for corresponding commands. Since now
our Uni-Fusion works with OpenSeg for Scene Understanding purpose, we call it
Uni-Fusion-SU.

####  VI-H1 Semantic Segmentation

We first evaluate model on generalized zero-shot point cloud semantic
segmentation application. Generalized Zero-Shot Learning (GZSL) differs from
Zero-Shot Learning (ZSL) that ZSL only predicts classes unseen at training
time while GZSL predicts both unseen and seen [50]. Thus the comparison with
GZSL SOTAs better reveals the potential of Uni-Fusion-SU as it does not train
on both.

In this test, ScanNet and S3DIS datasets are involved for benchmarking. Note
that, the compared baselines are trained on the corresponding datasets. Our
Uni-Fusion-SU use OpenSeg to provide the 2D image level feature ebmedding.
Thus though Uni-Fusion-SU is also Zero-shot, it does not touch any ScanNet or
S3DIS annotations.

We demonstrate the mIoU scores in Table VI. Here, we see that our model
achieves best result in Zero-shot learning methods on the ScanNet dataset. In
addition, it is also competitive to fully-supervised methods.

We also list results on unseen classes in Table VII. Though not as good as the
fully supervised result, Uni-Fusion-SU works much better than 3DGenZ.
Especially, our Uni-Fusion-SU is highly precise on classes like sofa and
Toilet comparing to the even fully supervised model.

TABLE VII: Classwise GZSL semantic segmentation performance (%) on the ScanNet unseen split. |  | Bookshelf | Desk | Sofa | Toilet |  mean
---|---|---|---|---|---|---
FSL (Fully supervise) | IoU | 56.9 | 30.0 | 57.4 | 63.4 | 51.9
3DGenZ (Zero-shot) | IoU | 6.3 | 3.3 | 13.1 | 8.1 | 7.7
Uni-Fusion-SU (Ours) | IoU | 38.3 | 16.8 | 51.7 | 60.9 | 41.9
3DGenZ (Zero-shot) | Acc. | 13.4 | 5.9 | 49.6 | 26.3 | 23.8
Uni-Fusion-SU (Ours) | Acc. | 61.9 | 29.6 | 67.4 | 91.6 | 62.6

However, in S3DIS, our model doesnot exceed 3DGenZ and others in Table VI.

Even in unsceen data result in Table VIII, we find Uni-Fusion-SU hardly finds
some class, e.g. Beam and Column, which is not a commonly annotated object.
While for common object sofa and window, our result is still much higher.

We demonstrate the semantic segmentation result in Fig. 15. We find 3DGenZ’s
result is more noisy. This is shown in the spotted sofa, bed and etc. While
Uni-Fusion-SU’s result is overall smoother and precise.

TABLE VIII: Classwise GZSL semantic segmentation performance (%) on the S3DIS unseen split. |  | Beam | Column | Sofa | Window |  mean
---|---|---|---|---|---|---
FSL (Fully supervise) | IoU | 63.1 | 10.2 | 54.1 | 72.4 | 50.0
3DGenZ (Zero-shot) | IoU | 13.9 | 2.4 | 4.9 | 8.1 | 7.3
Uni-Fusion-SU (Ours) | IoU | 5.5 | 0.02 | 57.4 | 32.9 | 24.0
3DGenZ (Zero-shot) | Acc. | 20.0 | 9.1 | 62.4 | 23.7 | 28.8
Uni-Fusion-SU (Ours) | Acc. | 41.5 | 0.02 | 78.3 | 42.1 | 40.5

####  VI-H2 Scene Understanding with Different Properties

The main contribution of this application is that, Uni-Fusion is the first
model that builds a continuous mapping of high-dimensional embedding on
surface swithout any training. In above experiment (Section VI-H1), we
evaluate the performance of generalized zero-shot semantic segmentation. But
the potential is not limited to semantic segmentation. With a LIM constructed,
a surface CLIP feature field is obtained. Then without the need of multiple
LIMs or rerun for other properties, we query Object, Room Type, Material,
Affordance and Activity.

We demonstrate results in Fig. 16, where we query object (desk, sofa),
activity (work), affordance (sittable) and material (wood). Uni-Fusion-SU
precisely finds the object and material region and highlights it. However,
when the searching command is less accurate, such as work or sittable, it
gives a wider range while in some place less confident (with dull yellow). The
options are also roughly correct.

| scene0568_00 | scene0249_00 | scene0435_00 | office3 | room0
---|---|---|---|---|---
|  |  |  |  |
Desk |  |  |  |  |
Sofa |  |  |  |  |
Work |  |  |  |  |
Sittable |  |  |  |  |
Wood |  |  |  |  |

Figure 16: Demonstration of the original mesh, highlighted semantic mesh given
various queries.

###  VI-I Time

We run all applications in one pass using our captured office sequences and
evaluate the time cost for the construction and fusion of each LIM. The
average time cost over frames are shown in Table IX.

TABLE IX: Time required in seconds for each frame.

| Surface | Color | Infrared | Style | Saliency | Latent | Internal Track
---|---|---|---|---|---|---|---
Time ($\mathrm{s}$) | 0.100 | 0.038 | 0.045 | 0.048 | 0.045 | 0.011 | 0.225

From this table, with depth and properties images of size $720\times 1280$ as
input, our model works with a frequency of $\sim 10$\mathrm{Hz}$$ for surface
(sample mode) LIM construction and integration. While even $20+$\mathrm{Hz}$$
for color, infrared, style, and saliency. This shows our Uni-Fusion is
adequate for real-time applications.

However, our internal tracking takes around $0.225$\mathrm{s}$$ for a frame
which is comparatively slower than the mapping module. Though, Uni-Fusion
exploits an external tracking to protect the model from losing track, which
enables our internal tracking and mapping to have a lower frequency. And thus
the whole model is applicable to be real-time in various scenarios.

##  VII Extensive experiment on Our Own dataset

In previous experiments, we assessed the capabilities of our Uni-Fusion in
various applications. For robotic understanding of the surroundings, we
capture our own dataset to show all applications together.

We capture two scenes: The office and apartment of the first author using a
Microsoft Kinect Azure. RGB-D and infrared video are captured. After
calibration, RGB, depth, infrared inputs are with the resolution of $720\times
1280$. Uni-Fusion tracks and reconstructs all applications in one pass. Office
data has been involved in ablation study Section VI-F. We demonstrate all
applications on the apartment dataset that is shown in Fig. 17.

|  |
---|---|---
Color | Infrared | Saliency
|  |
Style | Object-sofa | Object-desk
|  |
Object-coat | Affordance-sit | Material-wood
Figure 17: Demonstration on the captured apartment data.

Again, the ceiling of reconstruction is removed for better visualization. In
the top row, we show the colored mesh with room details, infrared mesh which
reveales the light affect, saliency reconstruction that indicates the objects
that are important for navigation. Next, we choose the second style in Fig. 13
to transfer onto the apartment canvas. The wooden floor in the room is colored
with dark green. The overall apartment is in a warm style.

The rest are generated from the surface field of the CLIP embeddings. We
command object to find, e.g., where is the sofa, desk and coat. In addition,
it easily finds affordances like sit-able. For material, it successfully finds
the wooden floor in each room.

##  VIII Limitations and Future Works

####  VIII-1 Remapping

Right now, Uni-Fusion does not support deintegrating local LIM from global LIM
that is essential to include bundle adjustment (BA) or loop closing. More
importantly, the local LIM does not support transformation of LIMs as NIM-REM
[12] does it at the current stage. To foster a better qualities and higher
computational performances and to support large scale mapping, loop closing
and bundle adjustment are future targets.

####  VIII-2 Visual Language Navigation

Uni-Fusion provides a good reconstruction and scene understanding basis for
Visual-Language Robot Navigation (VLN). Related works produces a 2D embedding
map [39] while Uni-Fusion is capable to construct a 3D embedding map with the
scene. Thus Uni-Fusion can help the robot to have a richer understanding of
the scene. In future work, we will investigate applications like navigation.

##  IX Conclusion

In this work, we propose a novel universal model for all continuous mapping
applications, namely Uni-Fusion. Without any training, Uni-Fusion constructs
Latent Implicit Maps that support geometry and arbitrary properties. Moving
one step further to scene understanding, Uni-Fusion is also the first model
that is capable to construct continuous maps with high dimensional embeddings.
With such a basis, we have implemented a high-quality incremental
Surface&Color reconstruction application, a 2D-to-3D fabricated properties
transfer application and an open-vocabulary scene understanding application.

## Acknowledgements

This work was in parts supported by the Federal Ministry for Economic Affairs
and Climate Action (BMWK) on the basis of a decision by the German Bundestag
und the grant number KK5150104GM1. We also acknowledge the support by the
Elite Network Bavaria (ENB) through the “Satellite Technology” academic
program..

## References

  * [1] S. T. O’Callaghan and F. T. Ramos, “Gaussian process occupancy maps,” _The Intl. Journal of Robotics Research_ , vol. 31, no. 1, pp. 42–62, 2012\.
  * [2] M. Ghaffari Jadidi, J. Valls Miro, and G. Dissanayake, “Gaussian processes autonomous mapping and exploration for range-sensing mobile robots,” _Autonomous Robots_ , vol. 42, pp. 273–290, 2018.
  * [3] Y. Yuan, H. Kuang, and S. Schwertfeger, “Fast gaussian process occupancy maps,” in _2018 15th Intl. Conf. on Control, Automation, Robotics and Vision (ICARCV)_. IEEE, 2018, pp. 1502–1507.
  * [4] W. Martens, Y. Poffet, P. R. Soria, R. Fitch, and S. Sukkarieh, “Geometric priors for gaussian process implicit surfaces,” _IEEE Robotics and Automation Letters_ , vol. 2, no. 2, pp. 373–380, 2016.
  * [5] B. Lee, C. Zhang, Z. Huang, and D. D. Lee, “Online continuous mapping using gaussian process implicit surfaces,” in _2019 Intl. Conf. on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 6884–6890.
  * [6] L. Wu, K. M. B. Lee, L. Liu, and T. Vidal-Calleja, “Faithful euclidean distance field from log-gaussian process implicit surfaces,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 2461–2468, 2021.
  * [7] J.-P. A. Ivan, T. Stoyanov, and J. A. Stork, “Online distance field priors for gaussian process implicit surfaces,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 4, pp. 8996–9003, 2022.
  * [8] B. Curless and M. Levoy, “A volumetric method for building complex models from range images,” in _Proc. of the 23rd annual conf. on Computer graphics and interactive techniques_ , 1996, pp. 303–312.
  * [9] S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman, A. Davison _et al._ , “Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera,” in _Proc. of the 24th annual ACM symposium on User interface software and technology_ , 2011, pp. 559–568.
  * [10] A. Dai, M. Nießner, M. Zollhöfer, S. Izadi, and C. Theobalt, “Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration,” _ACM Transactions on Graphics (ToG)_ , vol. 36, no. 4, p. 1, 2017.
  * [11] J. Huang, S.-S. Huang, H. Song, and S.-M. Hu, “Di-fusion: Online implicit 3d reconstruction with deep priors,” in _Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition_ , 2021, pp. 8932–8941.
  * [12] Y. Yuan and A. Nüchter, “An algorithm for the se (3)-transformation on neural implicit maps for remapping functions,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 3, pp. 7763–7770, 2022.
  * [13] E. Sucar, S. Liu, J. Ortiz, and A. J. Davison, “imap: Implicit mapping and positioning in real-time,” in _Proc. of the IEEE/CVF Intl. Conf. on Computer Vision_ , 2021, pp. 6229–6238.
  * [14] Z. Zhu, S. Peng, V. Larsson, W. Xu, H. Bao, Z. Cui, M. R. Oswald, and M. Pollefeys, “Nice-slam: Neural implicit scalable encoding for slam,” in _Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition_ , 2022, pp. 12 786–12 796.
  * [15] W. E. Lorensen and H. E. Cline, “Marching cubes: A high resolution 3d surface construction algorithm,” _ACM siggraph computer graphics_ , vol. 21, no. 4, pp. 163–169, 1987.
  * [16] K. Li, Y. Tang, V. A. Prisacariu, and P. H. Torr, “Bnv-fusion: Dense 3d reconstruction using bi-level neural volume fusion,” in _Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition_ , 2022, pp. 6166–6175.
  * [17] A. Rosinol, J. J. Leonard, and L. Carlone, “Nerf-slam: Real-time dense monocular slam with neural radiance fields,” _arXiv preprint arXiv:2210.13641_ , 2022.
  * [18] G. Ghiasi, X. Gu, Y. Cui, and T.-Y. Lin, “Scaling open-vocabulary image segmentation with image-level labels,” in _Computer Vision–ECCV 2022: 17th European Conf., Tel Aviv, Israel, October 23–27, 2022, Proc., Part XXXVI_. Springer, 2022, pp. 540–557.
  * [19] R. Senanayake and F. Ramos, “Bayesian hilbert maps for dynamic continuous occupancy mapping,” in _Conf. on Robot Learning_. PMLR, 2017, pp. 458–471.
  * [20] W. Zhi, L. Ott, R. Senanayake, and F. Ramos, “Continuous occupancy map fusion with fast bayesian hilbert maps,” in _2019 Intl. Conf. on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 4111–4117.
  * [21] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove, “Deepsdf: Learning continuous signed distance functions for shape representation,” in _Proc. of the IEEE/CVF conf. on computer vision and pattern recognition_ , 2019, pp. 165–174.
  * [22] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger, “Occupancy networks: Learning 3d reconstruction in function space,” in _Proc. of the IEEE/CVF conf. on computer vision and pattern recognition_ , 2019, pp. 4460–4470.
  * [23] R. Chabra, J. E. Lenssen, E. Ilg, T. Schmidt, J. Straub, S. Lovegrove, and R. Newcombe, “Deep local shapes: Learning local sdf priors for detailed 3d reconstruction,” in _Computer Vision–ECCV 2020: 16th European Conf., Glasgow, UK, August 23–28, 2020, Proc., Part XXIX 16_. Springer, 2020, pp. 608–625.
  * [24] C. Jiang, A. Sud, A. Makadia, J. Huang, M. Nießner, T. Funkhouser _et al._ , “Local implicit grid representations for 3d scenes,” in _Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition_ , 2020, pp. 6001–6010.
  * [25] S. Peng, M. Niemeyer, L. Mescheder, M. Pollefeys, and A. Geiger, “Convolutional occupancy networks,” in _Computer Vision–ECCV 2020: 16th European Conf., Glasgow, UK, August 23–28, 2020, Proc., Part III 16_. Springer, 2020, pp. 523–540.
  * [26] S. Lionar, L. Schmid, C. Cadena, R. Siegwart, and A. Cramariuc, “Neuralblox: Real-time neural representation fusion for robust volumetric mapping,” in _2021 Intl. Conf. on 3D Vision (3DV)_. IEEE, 2021, pp. 1279–1289.
  * [27] Z. Deng, J. Shi, and J. Zhu, “Neuralef: Deconstructing kernels by deep neural networks,” _arXiv preprint arXiv:2205.00165_ , 2022.
  * [28] A. Rahimi and B. Recht, “Random features for large-scale kernel machines,” _Advances in neural information processing systems_ , vol. 20, 2007.
  * [29] ——, “Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning,” _Advances in neural information processing systems_ , vol. 21, 2008.
  * [30] F. X. X. Yu, A. T. Suresh, K. M. Choromanski, D. N. Holtmann-Rice, and S. Kumar, “Orthogonal random features,” _Advances in neural information processing systems_ , vol. 29, 2016.
  * [31] M. Munkhoeva, Y. Kapushev, E. Burnaev, and I. Oseledets, “Quadrature-based features for kernel approximation,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [32] D. P. Francis and K. Raimond, “Major advancements in kernel function approximation,” _Artificial Intelligence Review_ , vol. 54, no. 2, pp. 843–876, 2021.
  * [33] C. Williams and M. Seeger, “Using the nyström method to speed up kernel machines,” _Advances in neural information processing systems_ , vol. 13, 2000.
  * [34] T. Yang, Y.-F. Li, M. Mahdavi, R. Jin, and Z.-H. Zhou, “Nyström method vs random fourier features: A theoretical and empirical comparison,” _Advances in neural information processing systems_ , vol. 25, 2012.
  * [35] C. K. Williams and C. E. Rasmussen, _Gaussian processes for machine learning_. MIT press Cambridge, MA, 2006, vol. 2, no. 3.
  * [36] E. Solak, R. Murray-Smith, W. Leithead, D. Leith, and C. Rasmussen, “Derivative observations in gaussian process models of dynamic systems,” _Advances in neural information processing systems_ , vol. 15, 2002.
  * [37] J. Park, Q.-Y. Zhou, and V. Koltun, “Colored point cloud registration revisited,” in _Proc. of the IEEE intl. conf. on computer vision_ , 2017, pp. 143–152.
  * [38] S. Peng, K. Genova, C. Jiang, A. Tagliasacchi, M. Pollefeys, T. Funkhouser _et al._ , “Openscene: 3d scene understanding with open vocabularies,” _arXiv preprint arXiv:2211.15654_ , 2022.
  * [39] C. Huang, O. Mees, A. Zeng, and W. Burgard, “Visual language maps for robot navigation,” in _ICRA_ , 2023.
  * [40] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” in _Proc. of the IEEE conf. on computer vision and pattern recognition_ , 2017, pp. 5828–5839.
  * [41] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “A benchmark for the evaluation of rgb-d slam systems,” in _2012 IEEE/RSJ intl. conf. on intelligent robots and systems_. IEEE, 2012, pp. 573–580.
  * [42] H. Zhang and K. Dana, “Multi-style generative network for real-time transfer,” in _Proc. of the European Conf. on Computer Vision (ECCV) Workshops_ , 2018, pp. 0–0.
  * [43] C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [44] I. Armeni, O. Sener, A. R. Zamir, H. Jiang, I. Brilakis, M. Fischer, and S. Savarese, “3d semantic parsing of large-scale indoor spaces,” in _Proc. of the IEEE conf. on computer vision and pattern recognition_ , 2016, pp. 1534–1543.
  * [45] Occipital, “Occipital: The structure sensor,” 2016.
  * [46] I. Armeni, S. Sax, A. R. Zamir, and S. Savarese, “Joint 2d-3d-semantic data for indoor scene understanding,” _arXiv preprint arXiv:1702.01105_ , 2017\.
  * [47] A. Rosinol, J. J. Leonard, and L. Carlone, “Probabilistic volumetric fusion for dense monocular slam,” in _Proc. of the IEEE/CVF Winter Conf. on Applications of Computer Vision_ , 2023, pp. 3097–3105.
  * [48] A. Cheraghian, S. Rahman, and L. Petersson, “Zero-shot learning of 3d point cloud objects,” in _2019 16th Intl. Conf. on Machine Vision Applications (MVA)_. IEEE, 2019, pp. 1–6.
  * [49] A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, and T. Mikolov, “Devise: A deep visual-semantic embedding model,” _Advances in neural information processing systems_ , vol. 26, 2013.
  * [50] B. Michele, A. Boulch, G. Puy, M. Bucher, and R. Marlet, “Generative zero-shot learning for semantic segmentation of 3d point clouds,” in _2021 Intl. Conf. on 3D Vision (3DV)_. IEEE, 2021, pp. 992–1002.
  * [51] Q.-Y. Zhou, J. Park, and V. Koltun, “Open3d: A modern library for 3d data processing,” _arXiv preprint arXiv:1801.09847_ , 2018.
  * [52] T. Schops, T. Sattler, and M. Pollefeys, “Bad slam: Bundle adjusted direct rgb-d slam,” in _Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition_ , 2019, pp. 134–144.
  * [53] T. Whelan, M. Kaess, M. Fallon, H. Johannsson, J. Leonard, and J. McDonald, “Kintinuous: Spatially extended kinectfusion,” 2012.
  * [54] R. Mur-Artal and J. D. Tardós, “Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras,” _IEEE transactions on robotics_ , vol. 33, no. 5, pp. 1255–1262, 2017.
  * [55] Z. Teed and J. Deng, “Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras,” _Advances in neural information processing systems_ , vol. 34, pp. 16 558–16 569, 2021.
  * [56] T. Müller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” _ACM Transactions on Graphics (ToG)_ , vol. 41, no. 4, pp. 1–15, 2022.
  * [57] T. Kim, K. Kim, J. Lee, D. Cha, J. Lee, and D. Kim, “Revisiting image pyramid structure for high resolution salient object detection,” in _Proc. of the Asian Conf. on Computer Vision_ , 2022, pp. 108–124.