# Offsets in Electrostatically Determined Distances: Implications for Casimir
Force Measurements

S. K. Lamoreaux<EMAIL_ADDRESS>Yale University, Department of
Physics, P.O. Box 208120, New Haven, CT 06520-8120  A. O. Sushkov
<EMAIL_ADDRESS>Yale University, Department of Physics, P.O. Box 208120,
New Haven, CT 06520-8120

###### Abstract

The imperfect termination of static electric fields at semiconducting surfaces
has been long known in solid state and transistor physics. We show that the
imperfect shielding leads to an offset in the distance between two surfaces as
determined by electrostatic force measurements. The effect exists even in the
case of good conductors (metals) albeit much reduced.

It has been long known that a static electric field applied external to a
semiconductor will not be perfectly internally shielded. We previously
considered the case of Debye screening, which predicts a penetration depth of
0.7 $\mu$m for Ge, and 24 $\mu$m for Si (intrinsic undoped materials at 300 K)
debye , which leads to distance offset when the force due to a voltage applied
between the plates is used for distance determination. The distance correction
given by the twice the Debye length divided by the (bare) dielectric constant.
Our measurements with Ge kim  did not support such a large distance
correction, and indeed it has been long known that surface states lead to a
shielding of the internal field. The first attempts to build a field effect
transistor showed an internal field at least 100 times smaller than expected
based on bulk material considerations.shockley  Bardeen provided a complete
explanation of the importance of surface states, and the basic premises are
well established fundamental considerations in transistor physics.bardeen
These effects largely explain why no distance correction was observed in our
Ge experiment. The analysis presented here, which included band bending and
surface state effects, shows that the simple Debye screening treatment of this
problem is an oversimplification.

We are reconsidering the problem because in our recent attempts at measuring
the Casimir force between high resistivity ($>10{\rm k}\Omega$cm) Si plates,
we see a distance offset of between 60 nm and 600 nm, depending on how the
plates were cleaned. The sensitivity of the effect to surface condition
suggests that the surface states are likely the most important part of the
offset correction. In our study of the effect, it has become apparent that
distance offsets are probably affecting all Casimir experiments to date, and
is an effect that needs to be considered among the panoply of systematic
effects that are only recently being acknowledged. It is interesting to note
that all of these systematic effects (roughness, vibration, surface patch
potentials, and now the offset effect considered here) all lead to an apparent
increase in the Casimir force compared to its true value at a given distance
as determined electrostatically.

The basic physical principle is shown and explained in Fig. 1. It is usually
assumed that the charge density in the depletion region is constant ($n_{d}$)
over its width $d_{1}$.dekker  This approximation is very good in the case
where there Fermi level is well below the conduction band; raising the
potential of the conduction band a small amount essentially empties it.

Shown in Fig. 2 are the related processes that are important for an
electrostatic distance calibration. We consider a case where the system is in
equilibrium when a potential $V_{0}$ is applied between the plates, and then
ask what happens when $V_{0}$ is changed by $\delta V$.

First, when $\delta V=0$, we have (in units with $\epsilon_{0}=1$, and the
electron charge $e=1$)

$\sigma_{0}d_{0}+V_{1}=V_{0}$ (1)

by energy conservation and

$\sigma_{0}+\sigma_{1}-n_{d}d_{1}=0$ (2)

by charge neutrality, where $\sigma_{0}$ is the charge on the perfectly
conducting plate, $\sigma_{1}$ is the surface state charge on the
semiconducting plate, $d_{0}$ is the physical plate separation. Integrating
the electric field from the surface into the bulk, using Poisson’s equation,
indicates that

$V_{1}=\alpha d_{1}^{2}$ (3)

where $\alpha\approx n_{d}/\kappa$ with $\kappa$ the bare dielectric constant.
Furthermore, we have

$\sigma_{1}=(E_{F}-V_{1})n_{s}$ (4)

where $n_{s}$ is the surface density of states (e.g., electrons/cm2volt in
appropriate units). We thus find that

$\sigma_{0}=-(E_{F}-V_{1})n_{s}+n_{d}\sqrt{V_{1}/\alpha}$ (5)

or

$\sigma_{0}=-(E_{F}-(V_{0}-\sigma_{0}d_{0})n_{s})+n_{d}\sqrt{(V_{0}-\sigma_{0}d_{0})/\alpha}$
(6)

which gives the relationship between $\sigma_{0}$ and $V_{0}$ which can be
used to find the net effective differential capacitance. It is easiest to
change $V_{0}$ by $\delta V$ and determine $\delta V_{1}$, $\delta d_{1}$,
$\delta\sigma_{0}$ and $\delta\sigma_{1}$, with fixed $n_{d}$, $E_{F}$,
$n_{s}$, and $d_{0}$. From charge neutrality, we have

$\delta\sigma_{0}-n_{s}\delta V_{1}-n_{d}\delta d_{1}=0.$ (7)

Furthermore,

$\delta d_{1}={1\over 2}{\delta V_{1}\over\sqrt{\alpha V_{1}}}$ (8)

and

$d_{0}\delta\sigma_{0}+\delta V_{1}=\delta V.$ (9)

We thus arrive at

$\delta\sigma_{0}+n_{s}d_{0}\delta\sigma_{0}+d_{0}n_{d}(4\alpha
V_{1})^{-1/2}=n_{s}\delta V+n_{d}(4\alpha V_{1})^{-1/2}\delta V$ (10)

which gives a differential capacitance (per unit area) of

$C={\delta\sigma_{0}\over\delta V}={n_{s}+n_{d}(4\alpha V_{1})^{-1/2}\over
1/d_{0}+n_{s}+n_{d}(4\alpha V_{1})^{-1/2}}\times{1\over d_{0}}.$ (11)

Evidently, this equation can be written in a form

$C={C_{A}C_{B}\over C_{A}+C_{B}}$ (12)

where

$C_{A}={1\over d_{0}}=C_{gap}$ (13)

and

$C_{B}=n_{s}+n_{d}(4\alpha
V_{1})^{-1/2}=C_{surface}+C_{bulk}=C_{maximum}={1\over d_{offset}}$ (14)

which is the maximum capacitance that can be observed as $d_{0}\rightarrow 0$.
It should be noted that the offset distance does not depend on $d_{0}$; the
capacitance between a spherical and flat surface has the same offset as that
for flat plates, in the limit that the proximity force approximation applies.

Taking into account surface and bulk (space charge) effects shows that the net
(differential) capacitance is lowered due to the parallel combination of
$C_{surface}$ and $C_{bulk}$ in series with $C_{gap}$. Thus, the plates are
closer than the distance given by an electrostatic calibration. When both
plates are made of the same semiconducting material, the distance offset
calculated here is simply multiplied by 2.

It is almost impossible to know the parameters required to calculate the
surface and bulk differential capacitances. For our recent measurements with
Si, we have measured the distance offset directly in the experiment (in situ)
by performing an electrostatic distance determination, then measuring how far
the plates must be moved until they touch. A more accurate determination of
the offset was possible by firmly mounting the plates, with one on a
translation stage. The offset was determined by measuring the capacitance as a
function of distance until the plates touched, which could be readily
electrically determined. A maximum capacitance, at zero plate separation as
determined by the electrical contact, was directly measured, and found to be
62 $\pm$ 5 nm. After further cleaning of the plates, an in situ measurement
shows a 600 nm distance offset.

We further note that even in the case of good conductors, there is a space
charge (bulk) correction to the capacitance. In wallmark  (Sec. 3-2.4) it is
shown that in the case of a parabolic conduction band, the effective distance
offset is about 0.1 nm per plate. This means that for distances of
approximately 100 nm, in a force gradient type experiment between a flat and
smooth surface, noting that the electrostatic distance offset means the plates
are closer than expected, the correction is

${\delta F\over F}=-4{\delta d\over d}=-4\times(-0.2)/100\approx 1\%$ (15)

which is at the level of the claimed accuracy of several experiments.

Because the offset is very sensitive to the physical surface properties, it
would appear as prudent to either directly measure the offset, or include it
as an adjustable parameter in comparing theory to experiment. We have found
that it is simple to measure the maximum capacitance that occurs just before
the plates come into physical (electrical) contact. This distance needs to be
corrected by the surface roughness, and the measurements must be done at
sufficiently low frequency for equilibrium to be established, however, the
measurements are straightforward.

Figure 1: On the left is shown a non-equilibrium distribution of electrons;
taking $E=0$ as an “uncharged” surface, the surface state energy does not
equal the Fermi energy $E_{F}$ which lies between the valence and conduction
bands for a semiconductor. To establish equilibrium, electrons flow from the
bulk to surface states until the surface states are filled to $E_{F}$, a,
leaving a depletion, or “space charge” region of depth $d_{1}$. This
redistribution of charge causes the bands to bend, as shown. Figure 2:
Schematic model of electric field penetration through the surface of a
semiconductor.

## References

  * (1) S.K. Lamoreaux, arXiv:0801.1283 .
  * (2) W.-J. Kim, A.O. Sushkov, D.A.R. Dalvit, and S.K. Lamoreaux, Phys. Rev. Lett. 103, 060401 (2009).
  * (3) W. Shockley adn G.L. Pearson, Phys. Rev. 74, 232 (1948).
  * (4) J. Bardeen, Phys. Rev. 71, 717 (1947).
  * (5) A.J. Dekker, Solid State Physics (Prentice-Hall, Englewood Cliffs, N.J., 1957, ninth printing), Sec. 14-4.
  * (6) Dietrich Meyerhofer, “Conduction through Insulating Layers,” in Field Effect Transistors (J. Torkel Wallmark and Harwick Johnson, editors) (Prentice-Hall, Englewood Cliffs, N.J., 1966), Sec. 3-2.4.


# NeRF-SLAM: Real-Time Dense Monocular SLAM
with Neural Radiance Fields

Antoni Rosinol  John J. Leonard
Massachusetts Institute of Technology
{arosinol, jleonard<EMAIL_ADDRESS>Luca Carlone

###### Abstract

We propose a novel geometric and photometric 3D mapping pipeline for accurate
and real-time scene reconstruction from monocular images. To achieve this, we
leverage recent advances in dense monocular SLAM and real-time hierarchical
volumetric neural radiance fields. Our insight is that dense monocular SLAM
provides the right information to fit a neural radiance field of the scene in
real-time, by providing accurate pose estimates and depth-maps with associated
uncertainty. With our proposed uncertainty-based depth loss, we achieve not
only good photometric accuracy, but also great geometric accuracy. In fact,
our proposed pipeline achieves better geometric and photometric accuracy than
competing approaches (up to $179\%$ better PSNR and $86\%$ better L1 depth),
while working in real-time and using only monocular images.

##  1 Introduction

3D reconstruction from monocular images remains one of the most difficult
computer vision problems. Achieving 3D reconstructions in real-time from
images alone is a key enabler for applications in robotics, surveying, and
gaming, such as autonomous vehicles, crop monitoring, and augmented reality.

While many 3D reconstruction solutions are based on RGB-D or Lidar sensors,
scene reconstruction from monocular imagery provides a more convenient
solution. RGB-D cameras can fail under certain conditions, such as direct
sunlight, and Lidar remains heavier and more expensive than a monocular RGB
camera. Alternatively, stereo cameras simplify the depth estimation problem to
a 1D disparity search, but rely on accurate calibration of the cameras that is
prone to miscalibration during practical operations. Instead, monocular
cameras are inexpensive, lightweight, and represent the simplest sensor
configuration to calibrate.

Unfortunately, monocular 3D reconstruction is a challenging problem due to the
lack of explicit measurements of the depth of the scene. Nonetheless, great
progress has been recently made towards monocular 3D reconstructions by
leveraging deep-learning approaches. Given that deep-learning currently
achieves the best performance for optical flow [30], and depth [36]
estimation, a plethora of works have tried to use deep-learning modules for
SLAM. For example, using depth estimation networks from monocular images [29],
multiple images, as in multi-view stereo [14], or using end-to-end neural
networks [4].

Figure 1: From left to right, input RGB image, estimated depth uncertainty,
back-projected depth-maps into a pointcloud, after thresholding the depth by
its uncertainty ($\sigma_{d}\leq 1.0$) for visualization, and our resulting
neural radiance field, rendered from the same viewpoint as the input image.
Our pipeline is capable of reconstructing neural radiance fields in real-time
given only a stream of RGB images.

However, even with the improvements due to deep-learning, building both
geometrically and photometrically accurate 3D maps of the scene from a
casually taken monocular video in real-time is not currently possible.

Neural Radiance Fields (NeRFs) have recently enabled 3D representations of the
world that are photometrically accurate. Unfortunately, NeRFs are difficult to
infer, given the costly volumetric rendering necessary to build a NeRF,
leading to slow reconstructions. Further, NeRFs originally required ground-
truth pose estimates to converge.

Nevertheless, recent work shows that it is possible to fit a radiance field in
real-time given posed-images [17], while others show that the ground-truth
poses are not strictly necessary[15, 6], as long as a sufficiently good
initial estimate is given. Hence, real-time pose-free NeRF reconstructions
could lead to metrically accurate 3D maps.

Despite this, a fundamental problem of NeRF representations, given no depth
supervision, is that the parametrization of the surfaces by a density is prone
to ‘floaters’, ghost geometry that appears because of bad initializations or
convergence to bad local minima. It has been shown that adding depth
supervision significantly improves the removal of this ghost geometry, and the
depth signal leads to faster convergence of radiance fields [10].

Given these findings, our insight is that having a dense monocular SLAM
pipeline, that outputs close-to-perfect pose estimates, together with dense
depth maps and uncertainty estimates, provides the right information for
building neural radiance fields of the scene on the fly. Our experiments show
that this is indeed possible, and that compared to other approaches we achieve
more accurate reconstructions in less time.

Contributions We propose the first scene reconstruction pipeline combining the
benefits of dense monocular SLAM and hierarchical volumetric neural radiance
fields. Our approach builds accurate radiance fields from a stream of images,
without requiring poses or depths as input, and runs in real-time. We achieve
state-of-the-art performance on the Replica dataset for monocular approaches.

##  2 Related Work

We review the literature on two different lines of work, dense monocular SLAM
and neural radiance fields, and discuss the literature at the intersection of
both fields.

###  2.1 Dense SLAM

The main challenges to achieve dense SLAM are (i) the computational
complexity, due to the shear amount of depth variables to be estimated, and
(ii) dealing with ambiguous or missing information to estimate the depth of
the scene, such as textureless surfaces or aliased images.

Historically, the first problem has been bypassed by decoupling the pose and
depth estimation. For example, DTAM [19] achieves dense SLAM by using the same
paradigm as the sparse PTAM[13], which tracked the camera pose first and then
the depth, in a de-coupled fashion. The second problem is also typically
avoided by using RGB-D or Lidar sensors, that provide explicit depth
measurements, or stereo cameras that simplify depth estimation.

Nevertheless, recent research on dense SLAM has achieved impressive results in
these two fronts. To reduce the number of depth variables, CodeSLAM[4]
optimizes instead the latent variables of an auto-encoder that infers depth
maps from images. By optimizing these latent variables, the dimensionality of
the problem is significantly reduced, while the resulting depth maps remain
dense. Tandem[14] is able to reconstruct 3D scenes with only monocular images
by using a pre-trained MVSNet-style neural-network on monocular depth
estimation, and then decoupling the pose/depth problem by performing frame-to-
model photometric tracking. Droid-SLAM[31] shows that by adapting a state-of-
the-art dense optical flow estimation architecture [30] to the problem of
visual odometry, it is possible to achieve competitive results in a variety of
challenging datasets (such as the Euroc [5] and TartanAir [34] datasets),
Droid-SLAM avoids the dimensionality problem by using downsampled depth maps
that are subsequently upsampled using a learned upsampling operator. Rosinol
et al. [23] further show that dense monocular SLAM can reconstruct faithful 3D
meshes of the scene by weighting the depths estimated in dense SLAM by their
marginal covariance, and subsequently fusing them in a volumetric
representation. The resulting mesh is geometrically accurate, but due to the
limitations of TSDF representations, their reconstruction lacks photometric
detail and is not fully complete.

Our approach is inspired by the work from Rosinol et al. [23], where we
replace the volumetric TSDF for a hierarchical volumetric neural radiance
field as our map representation. By using radiance fields, our approach
achieves photometrically accurate maps and improves the completeness of the
reconstruction, while also allowing the optimization of poses and the map
simultaneously.

###  2.2 Neural Radiance Fields (NeRFs)

To enable photometrically accurate maps, NeRFs have proved to be a useful
representation that allows capturing view-dependent effects while maintaining
multi-view consistency. NeRF [16] is the seminal work that gave name to this
representation, and sparked a revolution in terms of new papers that tried to
improve several aspects, the most prominent ones being: reducing the training
time to build a NeRF and the improvement of the underlying 3D reconstruction.

While the vanilla NeRF approach, using one large MLP, requires hours of
training to converge, several authors show that a smaller MLP, combined with
3D spatial data structures to partition the scene, leads to substantial speed-
ups. In particular, NGLOD [27] proposes to use tiny MLPs in a volumetric grid,
leading to faster reconstructions, but not quite real-time. Plenoxels [40]
further improved the speed by parametrizing the directional encoding using
spherical harmonics, while bypassing the use of an MLP. Finally, Instant-
NGP[17] shows that with a hash-based hierarchical volumetric representation of
the scene, it is possible to train a neural radiance field in real-time.

Several works also show how to improve the 3D reconstruction when using
radiance fields. While a variety of works have proposed to replace the density
field by a signed distance functions or other representations[20, 41, 33, 37],
we are most interested on the insights related to the use of alternative
supervisory signals to increase the convergence speed of NeRFs, as well as the
quality of the underlying geometry [32, 41]. In particular, Mono-SDF [41]
shows that state-of-the-art deep learning models for depth and normal
estimation from monocular images provides useful information that can
significantly improve the convergence speed and quality of the radiance field
reconstruction.

Our work leverages these insights by using the information provided from dense
SLAM, which estimates poses and dense depth maps. We also use the fact that
dense SLAM outputs are probabilistic in nature, and use this information,
which is typically discarded in current approaches, to weight the supervisory
signals to fit a radiance field.

###  2.3 SLAM with NeRFs

Another important axis of research in neural radiance fields is to remove its
dependency on partially known camera poses. This is particularly enticing for
building NeRFs without having to process the data to get the camera poses of
the images, a task that is usually long and is typically done using
COLMAP[25].

iNeRF [38] was the first to show that it is possible to regress the camera
pose given an already built NeRF of the scene. It also made evident that the
basin of convergence for pose estimation is small, as it is typical from
direct image alignment. Barf [15] further shows how to simultaneously fit a
NeRF while estimating camera poses, given an inaccurate initial guess, by
formulating the optimization as an iterative image alignment problem. They
also propose to modulate NeRF’s positional encoding (similarly to
FourierFeatures[28]) to imitate coarse-to-fine approaches that improve the
basin of convergence for direct approaches. Unfortunately, these approaches
are too slow for online inference due to their choice of a large MLP as map
representation.

iMap[26] and Nice-SLAM[42] subsequently showed how to build accurate 3D
reconstructions, without the need for poses, by partially de-coupling pose and
depth estimation, similarly to our approach, and by using RGB-D images. While
iMAP used a single MLP for the whole scene, Nice-SLAM leveraged the learnings
on volumetrically partitioning the space, as detailed above, to make inference
faster and more accurate. Both approaches use depth from RGB-D cameras as
input, instead, our work only uses monocular images.

Finally, VolBA [8], Orbeez-SLAM [7] and Abou-Chakra et al. [1] also use a
hierarchical volumetric map for real-time SLAM, but this time from monocular
images. VolBA uses a direct RGB loss for pose estimation and mapping, while
Orbeez-SLAM and Abou-Chakra et al. use ORB-SLAM [18] to provide the initial
poses. By using dense monocular SLAM, our approach both leverages an indirect
loss for pose estimation, which is known to be more robust than direct image
alignment, and has the ability to depth-supervise the radiance field, which is
known to improve quality and convergence speed. Further, differently than
other depth-supervised approaches [10, 39, 26, 11, 3], our depth loss is
weighted by the depth’s marginal covariance, and the depth is estimated from
optical-flow rather than measured by an RGB-D camera.

Overall, our work leverages recent work on dense monocular SLAM (Droid-
SLAM[31]), probabilistic volumetric fusion (Rosinol et al. [23]), and hash-
based hierarchical volumetric radiance fields (Instant-NGP [17]), to estimate
geometric and photometric maps of the scene in real-time, without the need of
depth images or poses.

##  3 Methodology

The main idea of our approach is to supervise a neural radiance field using
the output from dense monocular SLAM. Dense monocular SLAM can estimate dense
depth maps and camera poses, while also providing uncertainty estimates for
both depths and poses. With this information, we can train a radiance field
with a dense depth loss weighted by the depths’ marginal covariances. By using
real-time implementations of both dense SLAM and radiance field training, and
by running these in parallel, we achieve real-time performance. Figure 2 shows
the flow of information in our pipeline. We now explain our architecture,
starting with our tracking frontend (Sec. 3.1) and following with our mapping
backend (Sec. 3.2).

Figure 2: The input to our pipeline consists of sequential monocular images
(here represented as Img 1 & Img 2). Starting from the top-right, our
architecture fits a NeRF using Instant-NGP [17], which we supervise using RGB
images $\mathbf{I}$, depths $\mathbf{D}$, where the depths are weighted by
their marginal covariance, $\mathbf{\Sigma_{D}}$. Inspired by Rosinol et al.
[23], we compute these covariances from dense monocular SLAM. In our case, we
use Droid-SLAM [31]. We provide more details about the flow of information in
Sec. 3.1. In blue, we show Droid-SLAM’s [31] contributions and flow of
information, similarly, in pink are Rosinol’s contribution [23], and in red,
our contribution.

###  3.1 Tracking: Dense SLAM with Covariances

We use as our tracking module Droid-SLAM [31], which provides dense depth maps
and poses for every keyframe. Starting from a sequence of images, Droid-SLAM
first computes the dense optical-flow $\mathbf{p}_{ij}$ between pairs of
frames $i$ and $j$, using a similar architecture to Raft [30]. At the core of
Raft is a Convolutional GRU (ConvGRU in Fig. 2) that, given the correlation
between pairs of frames and a guess of the current optical flow
$\mathbf{p}_{ij}$, computes a new flow $\mathbf{p}_{ij}$, as well as a weight
$\mathbf{\Sigma}_{\mathbf{p}_{ij}}$ for each optical flow measurement.

With these flows and weights as measurements, Droid-SLAM solves a dense bundle
adjustment (BA) problem where the 3D geometry is parametrized as a set of
inverse depth maps per keyframe. This parametrization of the structure leads
to an extremely efficient way of solving the dense BA problem, which can be
formulated as a linear least-squares problem by linearizing the system of
equations into the familiar cameras/depths arrow-like block-sparse Hessian
$H\in\mathbb{R}^{(c+p)\times(c+p)}$, where $c$ and $p$ are the dimensionality
of the cameras and the points.

As can be seen in Fig. 2, to solve the linear least-squares problem, we take
the Schur complement of the Hessian to compute the reduced camera matrix
$H_{T}$, which does not depend on the depths, and has a much smaller
dimensionality of $\mathbb{R}^{c\times c}.$ The resulting smaller problem over
the camera poses is solved by taking the Cholesky factorization of
$H_{T}=LL^{T}$, where $L$ is the lower-triangular Cholesky factor, and then
solving for the poses $T$ by front and back-substitution. As shown at the
bottom of Fig. 2, given these poses $T$, we can solve for the depths $d$.
Furthermore, given poses $T$ and depths $D$, Droid-SLAM proposes to compute
the induced optical-flow and feeds it again as an initial guess to the ConvGRU
network, as seen on the left side of Fig. 2, where $\Pi$ and $\Pi^{-1}$, are
the projection and back-projection functions. The blue arrows in Fig. 2 show
the tracking loop, and corresponds to Droid-SLAM.

Then, inspired by Rosinol et al. [23], we further compute the marginal
covariances of both the dense depth maps and the poses from Droid-SLAM (purple
arrows in Fig. 2). For this, we need to leverage the structure of the Hessian,
which we block-partition as follows:

$H\mathbf{x}=\mathbf{b},\quad\text{\emph{i.e}.\hbox{}
missing}\quad\left[\begin{array}[]{cc}C&E\\\
E^{T}&P\end{array}\right]\left[\begin{array}[]{l}\Delta\boldsymbol{\xi}\\\
\Delta\mathbf{d}\end{array}\right]=\left[\begin{array}[]{c}\mathbf{v}\\\
\mathbf{w}\end{array}\right],$ (1)

where $H$ is the Hessian matrix, $\mathbf{b}$ the residuals, $C$ is the block
camera matrix, and $P$ is the diagonal matrix corresponding to the inverse
depths per pixel per keyframe. We represent by $\Delta\boldsymbol{\xi}$ the
delta updates on the lie algebra of the camera poses in $SE(3)$, while
$\Delta\mathbf{d}$ is the delta update to the per-pixel inverse depths. $E$ is
the camera/depth off-diagonal Hessian’s block matrices, and $v$ and $w$
correspond to the pose and depths residuals.

From this block-partitioning of the Hessian, we can efficiently calculate the
marginal covariances for the dense depths $\mathbf{\Sigma_{d}}$ and poses
$\mathbf{\Sigma_{T}}$, as shown in [23]:

$\displaystyle\mathbf{\Sigma}_{\mathbf{d}}$
$\displaystyle=P^{-1}+P^{-T}E^{T}\mathbf{\Sigma}_{\mathbf{T}}EP^{-1}$ (2)
$\displaystyle\mathbf{\Sigma}_{\mathbf{T}}$ $\displaystyle=(LL^{T})^{-1}.$

We refer to [23] for details on how to compute these in real-time.

Finally, given all the information computed by the tracking module – the
poses, the depths, their respective marginal covariances, as well as the input
RGB images – we can optimize our radiance field’s parameters and refine the
camera poses simultaneously.

###  3.2 Mapping: Probabilistic Volumetric NeRF

Given the dense depth-maps for each keyframe, it is possible to depth-
supervise our neural volume. Unfortunately, the depth-maps are extremely noisy
due to their density, since even textureless regions are given a depth value.
Figure 3 shows that the resulting pointcloud from dense monocular SLAM is
particularly noisy and contains large outliers (top image in Fig. 3).
Supervising our radiance field given these depth-maps can lead to biased
reconstructions, as shown later in Sec. 4.4.

Rosinol et al.[23] shows that the uncertainty of the depth estimates is an
excellent signal to weight the depth values for classical TSDF volumetric
fusion. Inspired by these results, we use the depth uncertainty estimation to
weight the depth loss which we use to supervise our neural volume. Figure 1
shows the input RGB image, its corresponding depth-map uncertainty, the
resulting pointcloud (after thresholding its uncertainty by $\sigma_{d}\leq
1.0$ for visualization), and our results when using our uncertainty-weighted
depth loss.

Given the uncertainty-aware losses, we formulate our mapping loss as:

$\mathcal{L}_{M}\left(\mathbf{T},\Theta\right)=\mathcal{L}_{\text{rgb}}\left(\mathbf{T},\Theta\right)+\lambda_{D}\mathcal{L}_{\text{D}}\left(\mathbf{T},\Theta\right)$
(3)

which we minimize with respect to both poses $\mathbf{T}$ and neural
parameters $\Theta$, given hyper-parameter $\lambda_{D}$ balancing depth and
color supervision (we set $\lambda_{D}$ to $1.0$). In particular, our depth
loss is given by:

$\mathcal{L}_{\text{D}}\left(\mathbf{T},\Theta\right)=\|D-D^{\star}(\mathbf{T},\Theta)\|^{2}_{\Sigma_{D}},$
(4)

where $D^{\star}$ is the rendered depth, and $D,\Sigma_{D}$ are the dense
depth and uncertainty as estimated by the tracking module. We render the depth
$D^{\star}$ as the expected ray termination distance, similarly to [16, 10].
Each depth per pixel is computed by sampling 3D positions along the pixel’s
ray, evaluating the density $\sigma_{i}$ at sample $i$, and alpha-compositing
the resulting densities, similarly to standard volumetric rendering:

$\displaystyle d^{\star}$
$\displaystyle=\sum_{i}\mathcal{T}_{i}\Bigl{(}1-\exp\left(-\sigma_{i}\delta_{i}\right)\Bigl{)}d_{i},$
(5)

with $d_{i}$ the depth of a sample $i$ along the ray, and
$\delta_{i}=d_{i+1}-d_{i}$ the distance between consecutive samples.
$\sigma_{i}$ is the volume density, generated by evaluating an MLP at the 3D
world coordinate of sample $i$. We refer to [17] for more details on the
inputs given to the MLP. Lastly, $\mathcal{T}_{i}$ is the accumulated
transmittance along the ray up to sample $i$, defined as:

$\mathcal{T}_{i}=\exp\Bigl{(}-\sum_{j<i}\sigma_{j}\delta_{j}\Bigl{)}.$ (6)

Our color loss is defined as in the original NeRF [16]:

$\mathcal{L}_{\text{rgb}}\left(\mathbf{T},\Theta\right)=\|I-I^{\star}(\mathbf{T},\Theta)\|^{2},$
(7)

where $I^{\star}$ is the rendered color image, synthesized similarly to the
depth image, by using volumetric rendering. Each color per pixel is likewise
computed by sampling along the pixel’s ray and alpha-compositing the resulting
densities and colors:
$\sum_{i}\mathcal{T}_{i}\bigl{(}1-\exp\left(-\sigma_{i}\delta_{i}\right)\bigl{)}\mathbf{c}_{i},$
where $\mathcal{T}_{i}$ is again the transmittance as in Eq. 6, and
$\mathbf{c}_{i}$ is the color estimated by the MLP. Both the density
$\sigma_{i}$ and the color $\mathbf{c}_{i}$ are estimated simultaneously for a
given sample $i$.

The mapping thread continuously minimizes our mapping loss function
$\mathcal{L}_{M}(\mathbf{T},\Theta)$ (Eq. 3). In Sec. 4, we show that this
approach achieves accurate results while running in real-time.

###  3.3 Architecture

Our pipeline consists of a tracking and a mapping thread, both running in
real-time and in parallel. The tracking thread continuously minimizes the BA
re-projection error for an active window of keyframes. The mapping thread
always optimizes all of the keyframes received from the tracking thread, and
does not have a sliding window of active frames.

The only communication between these threads happens when the tracking
pipeline generates a new keyframe. On every new keyframe, the tracking thread
sends the current keyframes’ poses with their respective images and estimated
depth-maps, as well as the depths’ marginal covariances, to the mapping
thread. Only the information currently available in the sliding optimization
window of the tracking thread is sent to the mapping thread. The active
sliding window of the tracking thread consists of at most $8$ keyframes. The
tracking thread generates a new keyframe as soon as the mean optical-flow
between the previous keyframe and the current frame is higher than a threshold
(in our case $2.5$ pixels).

Finally, the mapping thread is also in charge of rendering for interactive
visualization of the reconstruction.

###  3.4 Implementation Details

We perform all computations in Pytorch and CUDA, and use an RTX 2080 Ti GPU
for all our experiments (11Gb of memory). We use Instant-NGP[17] as our
hierarchical volumetric neural radiance field map, which we modify to add our
proposed mapping loss $\mathcal{L}_{M}$ in Eq. 3. As our tracking frontend, we
use Droid-SLAM[31], and re-use their pre-trained weights. We compute depth and
pose uncertainties using Rosinol’s approach [23] which computes uncertainties
in real-time. We use the same GPU for both tracking and mapping, although our
approach allows the use of two separate GPUs for tracking and mapping.

##  4 Results

|  |  room-0 |  room-1 |  room-2 |  office-0 |  office-1 |  office-2 |  office-3 |  office-4 | Avg.
---|---|---|---|---|---|---|---|---|---|---
iMAP∗ [26] (GT depth) |  Depth L1 [cm] $\downarrow$ | 5.70 | 4.93 | 6.94 | 6.43 | 7.41 | 14.23 | 8.68 | 6.80 | 7.64
PSNR  [dB] $\uparrow$ | 5.66 | 5.31 | 5.64 | 7.39 | 11.89 | 8.12 | 5.62 | 5.98 | 6.95
Nice-SLAM [42] (GT depth) |  Depth L1 [cm] $\downarrow$ | 2.53 | 3.45 | 2.93 | 1.51 | 0.93 | 8.41 | 10.48 | 2.43 | 4.08
PSNR  [dB] $\uparrow$ | 29.90 | 29.12 | 19.80 | 22.44 | 25.22 | 22.79 | 22.94 | 24.72 | 24.61
TSDF-Fusion Res. = 256 (our depth) |  Depth L1 [cm] $\downarrow$ | 23.51 | 20.94 | 23.34 | 14.11 | 10.50 | 30.89 | 28.92 | 22.83 | 21.88
PSNR  [dB] $\uparrow$ | 3.43 | 4.51 | 5.57 | 11.16 | 15.92 | 4.86 | 5.68 | 5.46 | 7.07
$\sigma$-Fusion[23] Res. = 256 (our depth) |  Depth L1 [cm] $\downarrow$ | 21.92 | 19.28 | 22.40 | 13.80 | 10.21 | 22.27 | 28.70 | 22.21 | 20.10
PSNR  [dB] $\uparrow$ | 3.45 | 4.51 | 5.57 | 11.16 | 15.92 | 4.86 | 5.69 | 5.46 | 7.08
Nice-SLAM [42] (no depth) |  Depth L1 [cm] $\downarrow$ | 11.12 | 9.42 | 19.03 | 11.12 | 10.24 | 16.36 | 21.33 | 14.81 | 14.18
PSNR  [dB] $\uparrow$ | 18.15 | 18.22 | 17.82 | 20.23 | 19.14 | 15.22 | 16.12 | 17.24 | 17.76
Ours (our depth) |  Depth L1 [cm] $\downarrow$ | 2.97 | 2.63 | 2.58 | 2.49 | 1.98 | 9.13 | 10.58 | 3.59 | 4.49
PSNR  [dB] $\uparrow$ | 34.90 | 36.95 | 40.75 | 48.07 | 53.44 | 39.30 | 38.63 | 39.21 | 41.40
Table 1: Geometric (L1) and Photometric (PSNR) results for the Replica
dataset. iMAP and Nice-SLAM are first evaluated using the ground-truth depth
from Replica as supervision (top two rows). We also evaluate Nice-SLAM when
not using ground-truth depth as supervision for comparison. TSDF-Fusion,
$\sigma$-Fusion, and our approach are evaluated using the poses and depths
from dense monocular SLAM, as explained in Sec. 3.1. Best results in bold.

In the following, we evaluate our approach against competing techniques, and
provide ablation experiments to evaluate the improvements deriving from our
proposed architecture and suggested uncertainty-aware loss.

###  4.1 Datasets

We use two datasets for evaluation: the Cube-Diorama dataset [1] and the
Replica dataset [12].

Cube-Diorama is a synthetic dataset generated with Blender, which provides
ground-truth poses, depths, and images. Using this dataset, we are able to do
accurate ablation experiments to evaluate the benefits of our proposed
approach.

The Replica dataset provides real-world scenes which we use to evaluate and
compare our approach with related work. The dataset consists of high quality
3D reconstructions of 5 offices and 3 apartments. For evaluation, we use the
data generated by rendering a random trajectory of 2000 RGB and depth frames
per scene (as generated by the authors of iMAP[26]). The rendered depth-maps
can be considered ground-truth, since they have been rendered from the ground-
truth mesh.

###  4.2 Methods for Evaluation

The approaches we are comparing are classical TSDF-fusion [9], probabilistic
TSDF-Fusion from Rosinol et al. [23] (which we label as $\sigma$-Fusion), iMAP
[26], and Nice-SLAM [42]. This allows us to compare geometric and
probabilistic approaches (TSDF-Fusion, $\sigma$-Fusion), as well as learning-
based approaches (iMAP and Nice-SLAM).

TSDF-Fusion and $\sigma$-Fusion use a hash-based volumetric representation to
fuse posed depth-maps estimated from our tracking module. While TSDF-Fusion
uses a uniform weight for the depth-maps, $\sigma$-Fusion uses the depth-maps
uncertainties to weight down the depths, similarly to ours. iMAP uses one
large MLP to represent the 3D scene, while Nice-SLAM uses hierarchical dense
volumetric grids for mapping, similarly to our choice of hash-based
hierarchical volumes.

Finally, both iMAP and Nice-SLAM use the depth-maps rendered from the ground-
truth meshes as measurements, which leads to better results than what we would
reasonably expect from a casually moving RGB-D camera. Therefore, we also
evaluate Nice-SLAM without ground-truth depth-maps as supervisory signal.

Figure 3: Qualitative results on the Replica office-0 dataset using different
mapping approaches. From top to bottom, raw pointcloud from our tracking
module, TSDF reconstruction using $\sigma$-Fusion, Nice-SLAM’s results, and
ours.

###  4.3 Geometric and Photometric Accuracy

The Replica dataset allows us to evaluate the geometric and photometric
quality of the different approaches. In particular, we use the L1 depth error
between the estimated and the ground-truth depth-maps as a proxy for geometric
accuracy (Depth L1), as well as the peak signal-to-noise ratio (PSNR) between
the input RGB images and the rendered images for photometric accuracy.

Table 1 shows the results achieved by the different approaches we evaluate.
The first two rows correspond to iMAP and Nice-SLAM under their default setup,
with ground-truth depth used as supervisory signal. Nice-SLAM is superior to
iMAP in terms of geometric and photometric accuracy. The following rows
correspond to approaches that do not use the ground-truth depth-maps. We run
TSDF-Fusion and $\sigma$-Fusion with the poses and depth-maps estimated by our
tracking module, with $\sigma$-Fusion further using the depths’ uncertainties.
While $\sigma$-Fusion achieves better geometric reconstructions than TSDF-
Fusion, both achieve poor photometric accuracy (we render the resulting 3D
mesh extracted using marching cubes). We also run Nice-SLAM without ground-
truth depth. As expected, Nice-SLAM’s results deteriorate when not using
ground-truth depth, but it still achieves competitive results when compared to
TSDF fusion, particularly in terms of photometric accuracy (PSNR). Finally,
our approach achieves the best results in terms of photometric accuracy, and
geometric accuracy by a substantial margin in most scenes. Compared to Nice-
SLAM, we achieve up to $179\%$ PSNR improvements in office-1, and up to $86\%$
better L1 accuracy in room-2, with the best combined improvements seen in
office-1 ($179\%$ better PSNR, $80\%$ better L1).

Figure 3 provides a qualitative comparison between the different map
representations and approaches, where we render the different 3D
representations for comparison. The top image is the raw pointcloud estimated
by our tracking module, where each 3D point is generated by back-projecting
the pixel’s depth and colored using the pixel’s RGB values. The second image
corresponds to the rendered 3D mesh extracted using marching cubes from the
volumetric TSDF map built using $\sigma$-Fusion. The next two images are
rendered images from Nice-SLAM and our approach.

Overall, our approach achieves noticeably better results than the other
representations. Remarkably, while the TSDF representation (using
$\sigma$-Fusion) uses the same pose, depth, and uncertainty estimates than our
approach, it achieves significantly worst results compared to our approach.

Figure 4: Impact on the performance when using depth supervision with and
without ground-truth depth, and when initializing the poses with ground-truth
or noisy poses; compared with our approach which estimates dense depths and
poses. Results after $60s$ of convergence.

###  4.4 Depth Loss Ablation

Depth supervision of neural radiance fields from raw depth-maps, either
estimated from dense SLAM or coming from RGB-D, is prone to errors, because
depth-maps are most often noisy and with outliers. For dense monocular SLAM,
this is particularly problematic, since depth values are estimated even for
textureless or aliased regions.

Fig. 4 shows that the ideal scenario is to use ground-truth pose and ground-
truth depth for fast and accurate neural radiance field reconstruction (top-
right image). If ground-truth depths are not provided, but ground-truth poses
are available, the radiance field also converges, although at a slower pace
(mid-left); this is the classical input for NeRF (posed images). Instead, if
we provide noisy poses and no depth-maps, the radiance field does not converge
in less than $60s$ (bottom-left), while using the ground-truth depth and noisy
poses still leads to great results (mid-right). Our approach aims to reach
this last result. The bottom-right image in Fig. 4 shows that our approach can
achieve great results despite using noisy poses and depths, as long as these
are weighted by their uncertainty.

In particular, in Fig. 5, we show that if the noisy depths (top-left image)
are used as priors without weighting them by their covariance, the convergence
is slower and biased. PSNR and L1-depth are worst by $4$dB and $7$cm,
respectively, after $120$s (top-right image). Instead, our approach is
resilient to these depths, if we weight them by their marginal covariance
(bottom-right image).

Fig. 6’s convergence plot shows that if we do not use depths, and only
leverage the poses from our tracking module, the photometric convergence is
already good, showing that the pose estimates are sufficiently accurate for
convergence. Nonetheless, the radiance field depth estimates are inaccurate
($7.8$cm L1 depth error after $500$s). If we use our dense depths (‘raw’ in
Fig. 6), the depth error reduces by almost half, to $4.1$cm. Unfortunately,
using depth without uncertainty weighting leads to worst PSNR ($3$dB) than
when not using depths, as it may bias the geometry of the scene. Finally,
weighting the tracking’s depth estimates achieves the best PSNR and L1 depth
metrics: it achieves similar PSNR than when not using depths, while having the
same L1 error than when using depths.

Figure 5: (Top-Left) Raw pointcloud estimated by the tracking module, (Bottom-
Left) Pointcloud after thresholding the depth uncertainty ($\sigma_{d}\leq
1.0$) for visualization. (Right Column) Radiance field reconstructions after
$120$s of optimization, with and without depth weighting (top-right and
bottom-right respectively). Room scene in Cube-Diorama dataset [1].  Figure 6:
Convergence plot comparing the PSNR and L1 depth metrics over time
($\Delta$t[s]) and iteration (Iter[-]) when not using depths (‘no depth’),
when using the raw depths without weighting (‘raw’), and using our approach
with weighted depth (‘weighted’). Room scene in Cube-Diorama dataset [1].

###  4.5 Real-Time Performance

Our pipeline is capable of running at $12$ frames per second, with images of
$640\times 480$ resolution. The tracking thread runs at $15$ frames per second
on average, creating around $10$ keyframes per second, depending on the amount
of motion. We add a keyframe every time the mean optical flow is larger than
$2.5$ pixels. The mapping thread runs at $10$ frames per second on average.
Overall, our pipeline is able to reconstruct the scene in real-time at $10$
frames per second, by parallelizing camera tracking and radiance field
reconstruction, and by using custom CUDA kernels.

##  5 Limitations

Our approach requires $\sim$11Gb of GPU memory to operate, given that we use
dense correlation volumes between pairs of images for tracking and that we use
hierarchical volumetric grids for mapping. The resulting memory requirements
can be prohibitively large for robotics applications with low compute power,
such as for drones.

Nevertheless, these memory requirements can be alleviated in two ways. On the
one hand, correlation volumes can be computed on the fly, as noted in [30],
reducing the need to store dense correlation volumes. On the other hand, the
volumetric information can be streamed to the CPU for inactive regions, only
loading to GPU a sliding window around the region of interest, similarly to
[35].

##  6 Conclusion

We show that dense monocular SLAM provides the ideal information for building
a NeRF representation of a scene from a casually taken monocular video. The
estimated poses and depth-maps from dense SLAM, weighted by their marginal
covariance estimates, provide the ideal source of information to optimize a
hierarchical hash-based volumetric neural radiance field. With our approach,
users can generate a photometrically and geometrically accurate reconstruction
of the scene in real-time.

Future work can leverage our approach to extend the definition of metric-
semantic SLAM [24], which typically only considers geometric and semantic
properties, by building representations that are also photometrically
accurate. Beyond metric-semantic SLAM, our approach can be used as the mapping
engine for high-level scene understanding, such as for building 3D Dynamic
Scene Graphs [2, 22, 21].

## Acknowledgments

This work is partially funded by ‘la Caixa’ Foundation (ID 100010434),
LCF/BQ/AA18/11680088 (A. Rosinol), ‘Rafael Del Pino’ Foundation (A. Rosinol),
ARL DCIST CRA W911NF-17-2-0181, and ONR MURI grant N00014-19-1-2571. We thank
Bernardo Aceituno and Yen-Chen Lin for helpful discussions.

## References

  * [1] Jad Abou-Chakra, Feras Dayoub, and Niko Sünderhauf.  Implicit object mapping with noisy data.  arXiv preprint arXiv:2204.10516, 2022.
  * [2] I. Armeni, O. Sener, A. Zamir, H. Jiang, I. Brilakis, M. Fischer, and S. Savarese.  3D semantic parsing of large-scale indoor spaces.  In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 1534–1543, 2016.
  * [3] D. Azinović, R. Martin-Brualla, D. B Goldman, M. Nießner, and J. Thies.  Neural RGB-D surface reconstruction.  In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [4] M. Bloesch, J. Czarnowski, R. Clark, S. Leutenegger, and A. J. Davison.  CodeSLAM: learning a compact, optimisable representation for dense visual SLAM.  In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [5] M. Burri, J. Nikolic, P. Gohl, T. Schneider, J. Rehder, S. Omari, M.W. Achtelik, and R. Siegwart.  The EuRoC micro aerial vehicle datasets.  Intl. J. of Robotics Research, 2016.
  * [6] S.-F. Chng, S. Ramasinghe, J. Sherrah, and S. Lucey.  Garf: Gaussian activated radiance fields for high fidelity reconstruction and pose estimation.  arXiv e-prints, 2022.
  * [7] C.-M. et al. Chung.  Orbeez-SLAM: A Real-time Monocular Visual SLAM with ORB Features and NeRF-realized Mapping.  arXiv preprint arXiv:2209.13274, 2022.
  * [8] R. Clark.  Volumetric bundle adjustment for online photorealistic scene capture.  In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [9] B. Curless and M. Levoy.  A volumetric method for building complex models from range images.  In SIGGRAPH, 1996.
  * [10] K. Deng, A. Liu, J.-Y. Zhu, and D. Ramanan.  Depth-supervised NeRF: Fewer views and faster training for free.  In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [11] A. Dey, Y. Ahmine, and A. I Comport.  Mip-NeRF RGB-D: Depth Assisted Fast Neural Radiance Fields.  arXiv preprint arXiv:2205.09351, 2022.
  * [12] J. Straub et al.  The Replica dataset: A digital replica of indoor spaces.  2019\.
  * [13] G. Klein and D. Murray.  Parallel tracking and mapping for small AR workspaces.  In IEEE and ACM Intl. Sym. on Mixed and Augmented Reality (ISMAR), 2007.
  * [14] L. Koestler, N. Yang, N. Zeller, and D. Cremers.  Tandem: Tracking and dense mapping in real-time using deep multi-view stereo.  In Conference on Robot Learning (CoRL), 2022.
  * [15] C.-H. Lin, W.-C. Ma, A. Torralba, and S. Lucey.  Barf: Bundle-adjusting neural radiance fields.  In Intl. Conf. on Computer Vision (ICCV), 2021.
  * [16] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng.  NeRF: Representing scenes as neural radiance fields for view synthesis.  In European Conf. on Computer Vision (ECCV), 2020.
  * [17] T. Müller, A. Evans, C. Schied, and A. Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  ACM Transactions on Graphics (SIGGRAPH), 2022.
  * [18] Raúl Mur-Artal and Juan D. Tardós.  ORB-SLAM2: an open-source SLAM system for monocular, stereo and RGB-D cameras.  IEEE Trans. Robotics, 33(5):1255–1262, 2017.
  * [19] R.A. Newcombe, S.J. Lovegrove, and A.J. Davison.  DTAM: Dense tracking and mapping in real-time.  In Intl. Conf. on Computer Vision (ICCV), 2011.
  * [20] M. Oechsle, S. Peng, and A. Geiger.  Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction.  In Intl. Conf. on Computer Vision (ICCV), 2021.
  * [21] J. Ost, F. Mannan, N. Thuerey, J. Knodt, and F. Heide.  Neural scene graphs for dynamic scenes.  In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [22] A. Rosinol, A. Gupta, M. Abate, J. Shi, and L. Carlone.  3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans.  In Robotics: Science and Systems (RSS), 2020.  (pdf), (media), (video).
  * [23] A. Rosinol, J. Leonard, and L. Carlone.  Probabilistic Volumetric Fusion for Dense Monocular SLAM.  In IEEE Winter Conf. on Applications of Computer Vision (WACV), 2022\.
  * [24] A. Rosinol, A. Violette, M. Abate, N. Hughes, Y. Chang, J. Shi, A. Gupta, and L. Carlone.  Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs.  Intl. J. of Robotics Research, 2021.  (pdf).
  * [25] J. L Schonberger and J.-M. Frahm.  Structure-from-motion revisited.  In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [26] E. Sucar, S. Liu, J. Ortiz, and A. J Davison.  iMAP: Implicit mapping and positioning in real-time.  In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [27] T. Takikawa, J. Litalien, K. Yin, K. Kreis, C. Loop, D. Nowrouzezahrai, A. Jacobson, M. McGuire, and S. Fidler.  Neural geometric level of detail: Real-time rendering with implicit 3D shapes.  In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [28] M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. Barron, and R. Ng.  Fourier features let networks learn high frequency functions in low dimensional domains.  Advances in Neural Information Processing Systems (NIPS), 2020.
  * [29] K. Tateno, F. Tombari, I. Laina, and N. Navab.  CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction.  In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [30] Z. Teed and J. Deng.  Raft: Recurrent all-pairs field transforms for optical flow.  In European Conf. on Computer Vision (ECCV), 2020.
  * [31] Z. Teed and J. Deng.  Droid-SLAM: Deep visual SLAM for monocular, stereo, and RGB-D cameras.  Advances in Neural Information Processing Systems (NIPS), 2021.
  * [32] J. Wang, P. Wang, X. Long, C. Theobalt, T. Komura, L. Liu, and W. Wang.  Neuris: Neural reconstruction of indoor scenes using normal priors.  arXiv preprint arXiv:2206.13597, 2022.
  * [33] P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  Advances in Neural Information Processing Systems (NIPS), 2021.
  * [34] W. Wang, D. Zhu, X. Wang, Y. Hu, Y. Qiu, C. Wang, Y. Hu, A. Kapoor, and S. Scherer.  TartanAir: A dataset to push the limits of visual SLAM.  In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), 2020.
  * [35] T. Whelan, J. B. McDonald, M. Kaess, M. F. Fallon, H. Johannsson, and J. J. Leonard.  Kintinuous: Spatially extended Kinect-Fusion.  In RSS Workshop on RGB-D: Advanced Reasoning with Depth Cameras, Sydney, Australia, July 2012.
  * [36] Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan.  MVSNet: Depth inference for unstructured multi-view stereo.  In European Conf. on Computer Vision (ECCV), 2018.
  * [37] L. Yariv, J. Gu, Y. Kasten, and Y. Lipman.  Volume rendering of neural implicit surfaces.  Advances in Neural Information Processing Systems (NIPS), 2021.
  * [38] L. Yen-Chen, P. Florence, J.T. Barron, A. Rodriguez, P. Isola, and T.-Y. Lin.  iNeRF: Inverting neural radiance fields for pose estimation.  In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), 2021.
  * [39] L. Yen-Chen, P. Florence, J. T Barron, T.-Y. Lin, A. Rodriguez, and P. Isola.  NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields.  arXiv preprint arXiv:2203.01913, 2022.
  * [40] Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks.  arXiv preprint arXiv:2112.05131, 2021.
  * [41] Z. Yu, S. Peng, M. Niemeyer, T. Sattler, and A. Geiger.  MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction.  arXiv preprint arXiv:2206.00665, 2022.
  * [42] Z. Zhu, S. Peng, V. Larsson, W. Xu, H. Bao, Z. Cui, M. R Oswald, and M. Pollefeys.  Nice-SLAM: Neural implicit scalable encoding for SLAM.  In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022.