# LiveNVS: Neural View Synthesis on Live RGB-D Streams

Laura Fink<EMAIL_ADDRESS>0009-0007-8950-1790 Friedrich-Alexander-
Universität Erlangen-Nürnberg
Fraunhofer IISCauerstr. 11Germany ,  Darius Rückert<EMAIL_ADDRESS>0000-0001-8593-3974 Friedrich-Alexander-Universität Erlangen-Nürnberg
Voxray GmbHCauerstr. 11Germany ,  Linus Franke<EMAIL_ADDRESS>0000-0001-8180-0963 Friedrich-Alexander-Universität Erlangen-NürnbergCauerstr.
11Germany91058 ,  Joachim Keinert<EMAIL_ADDRESS>0000-0003-1857-3862 Fraunhofer IISAm Wolfsmantel 33ErlangenGermany91058 and
Marc Stamminger<EMAIL_ADDRESS>0000-0001-8699-3442 Friedrich-
Alexander-Universität Erlangen-NürnbergCauerstr. 11Germany91058

###### Abstract.

Existing real-time RGB-D reconstruction approaches, like Kinect Fusion, lack
real-time photo-realistic visualization. This is due to noisy, oversmoothed or
incomplete geometry and blurry textures which are fused from imperfect depth
maps and camera poses. Recent neural rendering methods can overcome many of
such artifacts but are mostly optimized for offline usage, hindering the
integration into a live reconstruction pipeline.

In this paper, we present LiveNVS, a system that allows for neural novel view
synthesis on a live RGB-D input stream with very low latency and real-time
rendering. Based on the RGB-D input stream, novel views are rendered by
projecting neural features into the target view via a densely fused depth map
and aggregating the features in image-space to a target feature map. A
generalizable neural network then translates the target feature map into a
high-quality RGB image. LiveNVS achieves state-of-the-art neural rendering
quality of unknown scenes during capturing, allowing users to virtually
explore the scene and assess reconstruction quality in real-time.

Novel view synthesis, Neural rendering, Live preview, RGB-D Stream

††ccs: Computing methodologies Rendering††ccs: Computing methodologies Image-
based rendering††ccs: Computing methodologies Reconstruction Figure 1.
LiveNVS allows real-time novel view synthesis on a live RGB-D stream. The
pipeline allows for live scene exploration during capturing. Additionally, we
can display the reconstructed surface and a confidence map of the estimated
render quality, guiding the operator to identify weakly reconstructed parts of
the scene.

This is the author’s version of the work. It is posted here for your personal
use. Not for redistribution. The definitive version of record was published at
SIGGRAPH Asia 2023, http://dx.doi.org/10.1145/3610548.3618213.

You can find the video that was part of the supplemental material here
https://youtu.be/aMbE5WAgD2k. The code is published on
https://github.com/Fraunhofer-IIS/livenvs.

##  1\. Introduction

Live 3D reconstruction approaches, such as Kinect Fusion (Newcombe et al.,
2011), take as input a stream of 3D sensor data plus an RGB stream, and
provide an instant reconstruction of the seen geometry. They also show colored
previews of the reconstruction during the capturing process. However, for
practical application in VR, AR, telepresence and others, the rendering
quality of the previews is not sufficient, mostly due to lacking texture
detail and resolutions, as well as missing view-dependent effects.

Table 1.  Applicability of comparable NVS-methods for real-time reconstruction and live preview regarding their ability to (i) provide interactive reconstruction and (ii) rendering. The method can handle (iii) fast growing datasets and can (iv) instantaneously incorporate updates on camera poses from an concurrent calibration process (SLAM). (v) The pipeline implements mechanisms to be less vulnerable to an imperfect capturing process (hand-held, no studio setup) and camera poses (reliance on very accurate MVS calibration). (vi) Plausible results even for sparsely captured scene regions (e.g. no cloudy artifacts). Mip-NeRF 360 (Barron et al., 2022), Instant Neural Graphics Primitives (Müller et al., 2022a), Volumetric Bundle Adjustment (VBA) (Clark, 2022), NeRF-SLAM (Rosinol et al., 2022), MVS-NeRF (Chen et al., 2021), Stable View Synthesis (SVS) (Riegler and Koltun, 2021), Real-Time Novel View Synthesis With Forward Warping (FWD) (Cao et al., 2022), LiveNVS (ours). |  | MIP-NeRF | Instant | VBA | NeRF- | MVS- | SVS | FWD | LiveNVS
---|---|---|---|---|---|---|---|---|---
| Capability | 360 | NGP |  | SLAM | NeRF |  |  | (ours)
(i) | Live reconstruction | - | + | ++ | ++ | - | - | ++ | ++
(ii) | Real-time rendering | - | + | + | + | - | + | + | ++
(iii) | Growing datasets | - | + | + | ++ | ++ | - | ++ | ++
(iv) | Fast response to pose corrections | - | + | + | + | ++ | - | ++ | ++
(v) | Robust to imperfect capturing or camera poses | - | - | - | + | - | + | - | +
(vi) | Sparse input views | - | - | - | - | ++ | - | + | +

On the other hand, recent improvements in deep neural network architectures
have led to a resurgence of interest in image-based rendering (IBR) and novel
view synthesis (NVS), as common artifacts can be rectified effectively by
these methods (see Figure 2). State-of-the-art IBR and NVS approaches (Riegler
and Koltun, 2021; Barron et al., 2022) are able to render novel views that are
hard to differentiate from real photographs. However, these methods are not
directly applicable to live 3D reconstructions for multiple reasons, in
particular:

  * •

high preprocessing times / scene-specific optimizations

  * •

non-interactive render times

  * •

reliance on dense and/or high-quality input data

  * •

no support of growing datasets and late pose corrections (e.g. loop closure)

This is not only detrimental for on-site 3D reconstruction, because a user
does not get instantaneous feedback whether they have captured sufficient
images, but also prevents latency critical applications such as interactive
telepresence (Jahromi et al., 2020; Tan and Sato, 2020). Tab. 1 gives an
overview of recent related neural novel view synthesis approaches and their
properties relevant for real-time reconstruction from a live RGB(-D) stream.

Figure 2.  Conventional IBR (left) generates sharp results but introduces
artifacts at incorrect or missing geometry. Neural IBR methods (like ours)
reduce these artifacts without losing image sharpness (right).

In this work, we close this gap and present an integrated 3D reconstruction
and neural rendering pipeline LiveNVS (see Fig. 1), that overcomes the above
mentioned limitations. LiveNVS is executed on live RGB-D streams, and delivers
instant renderings of novel views in a quality that is on par with comparable
state-of-the-art NVS approaches. Our major contribution is an image-space
fusion algorithm that aggregates a varying number of warped neural features in
a novel view. After that, a neural network translates the feature map of the
novel view to a photorealistic RGB image. Since only a few RGB-D images
instead of a global scene representation are required, our approach can
naturally support growing datasets and is very time and memory efficient.
Furthermore, we support continuous scene and pose updates during the capturing
process, which is relevant for e.g. loop closure.

In short, our contributions are:

  * •

A real-time novel view synthesis pipeline with neural rendering capabilities
that runs on a live RGB-D stream as input

  * •

A novel differentiable image-space forward-warping and fusion method for
neural image features which allows for camera pose updates like loop closures
during capturing

  * •

Multiple other features, such as robust view selections and motion blur
defeating strategies, which greatly improve our quality and can easily be
integrated in common pipelines

  * •

An open source implementation available on https://github.com/Fraunhofer-
IIS/livenvs.

##  2\. Related Work

Synthesis of novel views from a sequence of input RGB-D input requires to (i)
compute the extrinsic and intrinsic camera parameters, (ii) fuse several or
all input frames into a common representation, and (iii) synthesize an image
representing the scene or object from the desired target camera pose.

##### Camera parameters

Immediate visual feedback on the current digitization status of the scene (or
the object) requires all the previous steps to run in real-time with low
latency. This excludes many of the methods typically used for pose estimation.
For instance, the commonly used COLMAP (Schönberger et al., 2016) has no real-
time capabilities and requires all input views before any geometry
reconstruction can be started, thus preventing early user feedback.

In contrast, simultaneous localization and mapping (SLAM) allows for instant
feedback about the camera trajectory and recorded surfaces (Mur-Artal and
Tardos, 2017; Macario Barros et al., 2022). Since frames are added
sequentially, camera extrinsics can contain significant errors and even drift,
and can be updated in retrospective by global bundle adjustment or loop
closure.

##### Fusion into a common representation

After successful determination of the camera parameters, the input frames may
be combined into a common representation. Multi-view stereo algorithms
(Griwodz et al., 2021; Zhu et al., 2021; Furukawa and Hernández, 2015;
Schönberger et al., 2016; Wang et al., 2021a) combine the input views into
point-clouds or textured meshes that can then be used to render a novel view.
However, those methods are time consuming. (Truncated) signed distance fields
(Newcombe et al., 2011; Dai et al., 2017b; Koestler et al., 2022) and methods
based on surface elements (surfels) (Whelan et al., 2016; Rückert et al.,
2019; Schöps et al., 2020) can be implemented in real-time, but they lack
photorealistic rendering.

##### View synthesis based on scene dependent optimization

Use of neural rendering methods (Tewari et al., 2020) can boost rendering
quality. Thies et al. (2019) optimize an object-specific neural texture for a
surface mesh thus being able to reproduce view-dependent effects and fixing
erroneous silhouettes in image space. Aliev et al. (2020), Kopanas et al.
(2021) as well as Rückert et al. (2022) use point clouds as proxy and optimize
a descriptor per point.

Alternatively, discrete voxel grids can be used (Müller et al., 2022a; Sun et
al., 2021; Clark, 2022). Rendering can be fast (Li et al., 2022a; Lin et al.,
2022; Esposito et al., 2022; Reiser et al., 2023) to even work on mobile
devices (Cao et al., 2023). While real-time reconstruction is significantly
more difficult, careful optimization and camera parameter refinement permits
fast capture and view synthesis (Haitz et al., 2023; Müller et al., 2022b;
Jiang et al., 2023; Clark, 2022; Rosinol et al., 2022). Other approaches
demonstrate their application on video data with dynamic content (Song et al.,
2022; Li et al., 2022b, 2023). Nevertheless, many explicit approaches are
computation intensive and memory-hungry if they store voxel grids of
complexity $O(n^{3})$. While some lower memory demands using hash grids
$(O(n^{2})$ (Müller et al., 2022a), implicit representations like Neural
Radiance Fields (NeRFs) (Mildenhall et al., 2020; Barron et al., 2022) can
reduce the required memory even further, albeit at the expense of huge
training times. Xie et al. (2022) provide an extensive overview on this topic.

As all above methods come at the cost of scene specific optimization or heavy
pre-preprocessing (camera calibration, proxy geometry computation), many only
support limited resolution (Rosinol et al., 2022) or need massive hardware
requirements (Clark, 2022).

##### Generalizing neural networks for view synthesis

Scene specific optimization can be avoided by using generalizing neural
networks trained only once. Consequently, novel view synthesis promises to
take less time from capturing to the first available frame.

Hedman et al. (2019) acquire pixel-wise blending weights for image-based
rendering. Riegler and Koltun (2020) add an additional CNN that encodes source
views into neural feature maps prior to warping, then use a decoder to ouput
images and blending weights. Similarly, there exist several other approaches
that integrate encoding networks to the pipeline (Rakhimov et al., 2022;
Riegler and Koltun, 2021; Jain et al., 2023; Wang et al., 2021b), yielding an
encoding-warping-aggregation-decoding or encoding-warping-decoding-aggregation
procedure. These encoders predict feature maps for source images, thus
allowing to acquire neural descriptors in a generative fashion. When trained
on sufficiently big datasets, such pipelines can be scene-agnostic and
generalize well also for unseen objects. These principles are applied to
point-based rendering (Rakhimov et al., 2022), signed distance functions
(Bergman et al., 2021) and proxy meshes (Riegler and Koltun, 2021; Jain et
al., 2023). Chen et al. (2021) alternatively predict NeRFs. Wang et al.
(2021b) predict densities along rays of the target image using a ray
transformer that is given features from encoded source images.

However, despite avoiding scene specific training, none of those methods work
in real-time on a continuous stream of input frames due to heavy pre-
processing, complex network architectures or slow rendering. Following these
observations, we identify a significant gap regarding interactivity within the
state of the art.

##### Real-time neural scene reconstruction and view synthesis

To cope with those limitations, several works specifically address the
challenge of an end-to-end real-time pipeline. This can be achieved using
global neural fields (Sucar et al., 2021; Lionar et al., 2021; Ortiz et al.,
2022; Zhu et al., 2022) or separate neural representations for depth map
fusion (Weder et al., 2021; Choe et al., 2021).

However, all of these methods focus on geometry reconstruction and struggle in
creating photo-realistic high quality renderings due to memory intensive data
structures or long rendering times. Cao et al. (2022) tries to overcome these
difficulties by proposing an encoding-warping-aggregation-decoding scheme for
real-time novel view synthesis. The individually warped views are merged by a
transformer network. By these means, it is the closest method to our approach.
However, due to the transformer complexity, the supported image resolutions
are small. Moreover, the evaluation is restricted to DTU (Jensen et al., 2014)
and ShapeNet (Chang et al., 2015) datasets, excluding many real-world
challenges such as imperfect camera parameters.

##### Our contributions

We combine photo-realistic neural rendering with real-time 3D reconstruction.
To avoid a lack of memory for large-scale scenes, we resign to any global data
structure and rely on the principles of (depth) image-based rendering
((D)IBR). We augment the fusion algorithm by Rückert et al. (2019) with a
refined neural weighting scheme to be robust against noisy depth maps and
inaccurate lens undistortions. A light-weight implementation achieves
interactive results using a fast and screen-space neural descriptor fusion
algorithm. Thus, we allow for free view point synthesis already during
capturing, which is a feature underrepresented in the field of NVS.

Figure 3.  Overview of our Pipeline. The input is a stream of RGB-D images,
which are encoded by a neural network. After that, the features are warped to
a target frame and fused in screen-space. Finally, a lightweight decoder
network translates the features to a photo-realistic color image.

##  3\. Method

Fig. 3 shows an overview of our neural rendering pipeline. The input is a set
of RGB-D images and the corresponding camera poses, for example, from a real-
time SLAM system (Sec. 3.1). We carefully select suitable images as keyframes
(Sec. 3.2) considering overlap between keyframes and motion blur. To
synthesize a novel view, several nearby keyframes are selected (Sec. 3.3),
encoded to latent space by a neural network (Sec. 3.4), and warped to the
target image (Sec. 3.5). The warped feature vectors are then fused by our
novel screen-space feature fusion function (Sec. 3.6) using tailored feature
weighting in order to achieve stable results. We also present a deferred
warping mode that further increases stability (Sec. 3.8). Finally, a decoder
network reconstructs the final RGB image of the novel view (Sec. 3.9), where
temporal stability is improved by including information from prior frames.

###  3.1. SLAM

The first step in our pipeline is a simultaneous localization and mapping
(SLAM) system that computes a 6-DoF pose of each input image. We are using
Snake-SLAM (2021) due to its high efficiency and accurate pose estimates. In
the SLAM-system previous key frames are continuously optimized by local bundle
adjustment and global loop closure operations. Our surface fusion (see Sec.
3.6) can handle such late updates, because we have no global model and the 3D
information is stored in the local space of the key frames, and thus does not
change when the key frame’s pose is modified.

###  3.2. Key Frame Selection from Input Stream

To make the input video stream more manageable for rendering, we derive a set
of keyframes. Our keyframe selection mechanism is similar to that of other
SLAM-systems such as ORB-SLAM (Mur-Artal and Tardos, 2017) but we select
frames based on a motion score. The score is based on the observation that
effects like motion blur or rolling shutter distortions degrade input data. It
is calculated from the per-pixel motion vector using consecutive poses and
depthmaps. The average of the length of the motion vectors across the image
constitutes the score per frame. We evaluate this score over a 1 s window,
selecting the optimal frame. Despite neglecting shutter speed, our approach’s
efficacy is validated in Sec. 4.

###  3.3. View Selection for Rendering

To achieve real-time frame rates and counteract over-smoothing the final
result, we select the $N$ (in our examples $N=15$ if not noted otherwise) best
nearby key frames for the target image. A careful selection is essential to
avoid large holes and blurry edges. To that end, we follow the idea of Hedman
et al. (2019) by computing the coverage of a source frame in the target view
but extend it in two ways. First, our coverage is calculated on a per-tile
basis, which guarantees that each tile of the target is seen at least once.
Second, the candidate views of each tile are sorted by the projected weighting
scheme (see Sec. 3.7) to prefer candidates with good warping properties.
Hence, the algorithmic complexity depends on the number of available source
views.

###  3.4. Encoding

After view selection, the $N$ best RGB-D source images are transformed to
latent space by a deep neural network. The used architecture is based on an
adapted ResUNet with the encoder part being pre-trained on the Image-Net
dataset (Riegler and Koltun, 2021). It receives the linear depth as extra
auxiliary input. The main output is a 4D feature vector for each pixel, and
additionally a confidence value that self masks the prediction, e.g. if depth
and color information do not match. We concatenate the RGB-D input with the 4D
output to generate the 8D feature vector used for warping. Since the encoded
views are usually required multiple times, they get cached in a least-
recently-used buffer. To limit the variance in rendering time, we set a fixed
number of encoding passes per frame resulting in a temporal build-up of the
target view.

###  3.5. Warping

To warp the latent space features maps from the source to the target view, we
follow a forward warping approach that requires no global scene model. The
forward warping is implemented by triangulating the depth maps (with occlusion
edges removed) and rendering them to the target view using the standard
rasterization pipeline. The warped triangles are then immediately blended in
the target view, as described in the next section. This implementation is
computationally very efficient and does not require dynamic memory allocation
during rasterization.

###  3.6. Screen-Space Surface Fusion

From the warping stage, we obtain triangles textured with features from the
input images, which are rasterized into the target view. Our fusion algorithm,
which is inspired by the work of Rückert et al. (2019), incrementally combines
these fragments to obtain a final single neural color vector.

For each output pixel, we keep track of the current linear depth $d$, weight
$w$, and the neural feature $f$. We take into account that depth accuracy of
3D sensors decreases with distance, which we describe using a depth error
function $\Delta(d)$, which is modelled as inversely proportional to a
quadratic polynomial (Mallick et al., 2014):

(1) $\Delta(d)=\dfrac{1}{ad^{2}+bd+c},$

where weights $a,b,c$ have to be selected according to Mallick et al. (2014).

If a new fragment with $d_{f}$, $w_{f}$, and $f_{f}$ is rendered to the screen
the per pixel values are updated based on one of the following cases:

  1. (1)

If $d_{f}$ ¡ $d-\Delta(d)$

     * $\rightarrow$

New fragment belongs to a new surface in front of the current surface.

     * $\rightarrow$

Overwrite current pixel values

  2. (2)

If $d_{f}$ ¿ $d+\Delta(d)$

     * $\rightarrow$

New fragment is behind the current surface

     * $\rightarrow$

Discard fragment

  3. (3)

Else

     * $\rightarrow$

New fragment is near the current surface

     * $\rightarrow$

Fuse fragment values into pixel

In the third case, the fragment is fused into the pixel updating the current
pixel values using an incremental averaging scheme:

$\displaystyle\alpha$ $\displaystyle\leftarrow w/(w+w_{f})$ $\displaystyle w$
$\displaystyle\leftarrow w+w_{f}$ $\displaystyle d$
$\displaystyle\leftarrow\alpha d+(1-\alpha)d_{f}$ $\displaystyle f$
$\displaystyle\leftarrow\alpha f+(1-\alpha)f_{f}$

###  3.7. Feature Weighting

The final feature descriptor of each pixel is a weighted sum of the
contributing source features. The weight for each source feature should
closely match the image quality at that 3D point, less reliable and biased
input should be weighted lower. For example, if the source view is close to
the surface and its incident angle is similar to the target angle, a large
weight should be used. If the source view is far away or observes the surface
from a different angle, a small weight should reduce the influence because we
expect an erroneous and blurry warping procedure. To that end, we apply a
weighting consisting of three parts:

(2) $w_{f}=(w_{d}w_{v}w_{i})^{5},$

where $w_{d}$ is the depth accuracy, $w_{v}$ the view direction weight, and
$w_{i}$ a vignetting coefficient. To compute depth accuracy, we use the
previously defined depth error function:

(3) $w_{d}=\Delta(d_{f}).$

The viewing direction weight $w_{v}$ depends on the angle between the
fragment’s target $v_{t}$ and source view direction $v_{s}$. Using the dot
product, the weight can then be defined as:

(4) $w_{v}=\max(0,v_{s}\cdot v_{t}).$

Finally, the vignetting coefficient $w_{i}$ downweights pixels with their
distance $c$ to the image center:

(5) $w_{i}=1-\frac{c}{c_{\mathrm{max}}}.$

This weight has multiple purposes: accounting for imperfect lens undistortion
and degrading depth accuracy with distance to the center. Additionally, sharp
transitions at boundaries of the blended input images are smoothed out.

##### Reconstruction Guidance

The average of the per-pixel weight $w$ can in addition be beneficial as
guidance for the expected image quality. When overlayed on the rendering (see
Fig. 4), the operator can be guided to undersampled regions of the scan.

Figure 4.  Ground truth, our neural rendering, and our predicted
reconstruction confidence map. Quality ranging from green (good) over white
(normal) to red (bad).

Figure 5.  Live novel view synthesis results shown on an outdoor and indoor
scene. Ours and FWD output novel views using up to 6 source views. Values on
the lower left corner indicate LPIPS $\downarrow$ score and values on the
right the render times in ms measured on a Nvidia RTX 2080Ti at a resolution
of 1280$\times$720 (*FWD only renders at resolution 400$\times$300.) INGP was
optimized for the time span from capturing start up to the last source view
previous to the novel view using the full trajectory. Our system outperformed
FWD and INGP in regards of LPIPS and render times in this scenario.

###  3.8. Deferred Warping Mode

We have observed that the fused depth images are strongly influenced by the
selection of source views, resulting in temporal instability in the output. To
address this issue, we introduce a two-stage deferred warping mode in our
pipeline to separate the surface and feature fusion processes. In the first
stage, the depth map of the target view is computed as described earlier.
Following that, the features are aggregated by sampling from the source views
using the estimated reprojected depth value. Since the depth of the surface is
already known during the reprojection, we update $w_{d}$ to reflect the
distance between the estimated and projected depth values:

(6) $w_{d}=1-(|d-d_{f}|/\Delta(d))^{2},$

The deferred warping mode of our pipeline offers a notable advantage: it
allows us to leverage a larger number of contributing depth maps for surface
fusion, while at the same time reducing the number of input images for feature
aggregation. This approach thus enhances the temporal stability of renderings
while having minimal impact on performance, because the time required for
depth map fusion is negligible compared to feature encoding and caching.

###  3.9. Decoding with Temporal Feedback Loop

In the final stage of our pipeline, we convert the fused feature map into an
RGB image, as shown in Fig. 3. To do so, we employ a decoder that builds upon
the well-established UNet architecture, extended by a feedback loop to improve
temporal stability. For each resolution level, the previous frame’s
intermediate features are linearly blended with the intermediate features of
the current frame.

Since some of our training datasets do not include video data, we mimic
previous frames by applying minor spatial transformations to the current frame
during training. This approach enables us to train a temporally stable decoder
network even when we only have a limited number of images with significant
differences. Note that we do not reproject the previous frame’s features,
instead the decoder network is trained to account for this.

##  4\. Results

In this section, we show results produced by our method. Our experimental
setup and procedure is described in Sec. 4.1 and the supplemental material.
Our results are compared to other methods with respect to quality and view
synthesis time. We provide extensive ablation studies on key frame selection,
view extrapolation, late pose updates, source of depth maps and our view
selection.

Table 2.  Quantitative comparison given as LPIPS$\downarrow$ score. The best two methods are printed in bold. | Scannet |  | Motor-
---
cycle
Sofa
Ours | 0.274 | 0.303 | 0.234
FWD | 0.428 | 0.451 | 0.295
SVS | 0.316 | 0.247 | 0.237
IBRNet | 0.248 | 0.179 | 0.149
NPBG++ | 0.396 | 0.500 | 0.366
DIBR | 0.325 | 0.290 | 0.313

Table 3.  Time consumption for different methods. Preprocessing includes training time and the time to generate the proxy geometry. (*FWD only renders at resolution 400$\times$300.)  Method | Proxy Geometry | Preprocessing | Rendering Time
---|---|---|---
Ours | depth maps | ¡ 1 s |  $\sim$ 46 ms
FWD | depth maps | ¡ 50 s |  $\sim$ 84 ms*
SVS | mesh |  $\sim$ 4 h |  $\sim$ 3 s
IBRNet | — | ¡ 1 min |  $\sim$ 2 min
NPBG++ | point cloud |  $\sim$ 1 h |  $\sim$ 100 ms
DIBR | depth maps | ¡ 1 min |  $\sim$ 5 ms
ADOP | point cloud |  $\sim$ 8 h |  $\sim$ 15 ms
Instant-NGP | volumetric hash |  $\sim$ 2 min |  $\sim$ 2 s

Table 4.  Keyframe selection results on Scannet using frames that passed our selection (s) or using every 20th view (mod20) during training.  Training Views |  LPIPS $\downarrow$
---|---
Ourss | 0.2705
Oursmod20 | 0.3401
SVSs | 0.2673
SVSmod20 | 0.3162

###  4.1. Datasets & Training Procedure

We conducted extensive evaluation utilizing a combination of publicly
available datasets, namely the Scannet (Dai et al., 2017a), Redwood (Choi et
al., 2016) (Motorcycle and Sofa) and Tanks and Temples dataset (Knapitsch et
al., 2017), along with our own custom data captured using the Zed 2i camera.
The Zed 2i employed is a stereo camera equipped with a resolution up to 2K.
Notably, the depth estimation process relies on a state-of-the-art deep
learning algorithm (Stereolabs, 2023).

We have trained the involved networks end-to-end in deferred warping mode (see
Sec. 3.8) on a combination of Scannet and Tanks and Temples scenes.
Remarkably, all experiments in the following sections, as well as the
supplemental video, were generated using identical network weights. This shows
that our method can generalize well to unseen scenes and even to new
modalities like the Zed camera, which was not included in the training
dataset.

For more details on training and dataset preparation, we refer to the
supplemental material.

###  4.2. Live RGB-D Stream Novel View Synthesis

To evaluate the quality of novel view synthesis during capturing in relation
to previous work, we have extended the FWD (Cao et al., 2022) and Instant-NGP
(INGP) (Müller et al., 2022b) pipeline to run on growing datasets from our Zed
camera. For a fair comparison, we ran the INGP optimization for the amount of
time up to the captured frame. Our method and FWD do not require a scene
specific optimization. The results are shown in Fig. 5. The neural rendering
results show that our method is able to produce the sharpest results with the
least amount of artifacts. For example, the colorful wall painting (left) and
the world map (right) is clearly recognizable in our approach and heavily
blurred for FWD and INGP.

###  4.3. Novel View Synthesis on Full Datasets

We compare the rendering quality of our approach to state-of-the-art methods
that also do not require a scene-specific optimization procedure. These
methods are Stable View Synthesis (SVS) (Riegler and Koltun, 2021), IBRNet
(Wang et al., 2021b), NPBG++ (Rakhimov et al., 2022), FWD (Cao et al., 2022),
and DIBR (a depth image-based rendering method similar to FragmentFusion
(Rückert et al., 2019)). Fig. 11 shows some exemplary results. In Tab. 7, we
summarize the measured image quality of the produced unseen views evaluated
using the LPIPS (Zhang et al., 2018) perceptual loss. Our approach reaches a
similar LPIPS score compared to the state-of-the-art, while being
significantly more efficient without any preprocessing required.

###  4.4. Preprocessing and Render Time

In Tab. 4, we provide an overview of preprocessing and rendering time for ours
and related work for medium-sized scenes, e.g. rooms or buildings. SVS and
NPBG++ require an initial point cloud or mesh as a geometric proxy. Using
COLMAP, this takes 2 - 6 hours. Additionally, several preprocessing steps have
to be conducted before rendering can take place. For example, INGP needs a
training stage to optimize the hash grid. Our approach, requires only minimal
preprocessing and can be operated on live RGB-D streams. Rendering a
$1296\times 968$ image takes around 46 ms with our method. The other systems
require seconds to minutes. A detailed benchmark for differently sized scenes
is presented in Tab. 5. Note that the high inference time of the encoder is
hidden in the average total render time as our caching reduces the number of
encodings drastically. Even though render times seem competitive, note that
warping all available feature maps per frame relies on an excessively sized
cache that can store the feature maps of all available source view.

Table 5.  Benchmarks of our pipeline (1296$\times$968 px , Nvidia RTX 2080Ti, averaged over 256 frames for a moving camera). The average total frame time includes view selection (if used), encoding of uncached views, (a separate warping pass of all depth maps for the deferred mode), warping of (selected) feature maps, and decoding. Total Number of Source Views Available | 50 Views | 207 Views
---|---|---
Inference Encoder | 35 ms | 35 ms
Inference Decoder | 12 ms | 12 ms
| Avg Total Time: Forward
---
15 selected Views for Features & 15 for Depth
46 ms | 67 ms
| Avg Total Time: Deferred
---
15 selected Views for Features & all for Depth
82 ms | 160 ms
Including View Selection | 7 ms | 26 ms
| Avg Total Time: Forward
---
all Views for Features & all for Depth
66 ms | 131 ms
| Avg Total Time: Deferred
---
all Views for Features& all for Depth
83 ms | 168 ms
Without View Selection | - | -

###  4.5. Ablation Studies

####  4.5.1. Keyframe Selection

We incorporate an effective motion-blur detection and compensation scheme by
removing blurry frames from the input stream (see Sec. 3.2). In Tab. 4, we
evaluate our improved keyframe selection on the Scannet dataset for our
approach and SVS (Riegler and Koltun, 2021). Both methods profit from our
keyframe selection and show a significantly improved LPIPS score, thus,
indicating that also methods with extensive preprocessing and rendering (see
Tab. 4) can benefit from our selection. A visual comparison on two indoor
scenes is presented in Figure 6.

Figure 6.  Left: neural rendering with default keyframe selection. Right: our
improved keyframe selection.

####  4.5.2. View Extrapolation

Figure 7 shows a synthesized novel view for a virtual camera that is far away
from the input trajectory. Volumetric-based methods like INGP introduce cloud-
like artifacts in this case. Our approach is able to render a clean image with
only few noticeable inaccuracies.

Figure 7.  View extrapolation results for INGP (left) and our method (center).
Note that there are severe cloud-like artefacts in the INGP rendering, while
ours gets close to the reference (right).

####  4.5.3. Loop Closure

An advantage of LiveNVS is that the poses of previous keyframes can be updated
at any time. Thus, our renderings can immediately mirror detected loops of the
tracking system. Fig. 8 shows our rendering right before and after a loop
closure. The ghosting artifacts due to miss-aligned poses before the loop
closure are removed on the right.

Figure 8.  Tracking a loop can cause ghosting artifacts in the rendering due
to accumulated drift (see left image). The SLAM system automatically detects
and closes these loops. Our rendering immediately reflects these changes,
since no global model is optimized (center image).

####  4.5.4. Source of depth maps

Alternatively to depth maps from active sensors, we can also use depth maps
rendered from offline reconstructed meshes. This allows us to profit from
subsequent post-processing of the original depth maps or other offline methods
delivering high quality 3D reconstructions. Especially, in areas where depth
maps from active sensors are prone to be noisy or erroneous, visual fidelity
can be improved, as shown in Fig. 9. However, on the Scannet dataset a
quantitative evaluation shows only a minor improvement of the LPIPS score from
$0.271$ to $0.269$.

####  4.5.5. View Selection

As described in Sec. 3.3, we do a coverage-based view selection where we tile
the target image frame. Fig. 10 visualizes the benefit of such a selection
scheme as big holes in the background are avoided effectively in this view.

Figure 9.  Crops of renderings using depth maps from active sensor (left) vs.
mesh (center). Groundtruth(right). Figure 10.  Renderings using untiled (left)
and tiled (right) view selection.

##  5\. Limitations and Future Work

During our evaluation, we found a few limitations of our work. The major
limitation becomes visible if the pose estimates are extremely inaccurate or
the depth maps are too noisy. In this case, our rendering network gets
inconsistent feature descriptors and outputs a blurred image of the scene. Our
surface fusion algorithm might fail if the noise of the depth values exceeds
the threshold (see $\Delta(d)$ in Eq. 3). Other approaches, which rely on
inverse rendering optimization, can correct such input errors and output
sharper renderings. However, this optimization step is often time consuming,
while ours can be directly applied to live video stream.

The second limitation is a potential temporal instability when the view
selection swaps out an input image for another, resulting in flickering. This
flickering is especially visible if exposure time or white balance vary
throughout the stream. The temporal feedback loop incorporated in our decoder
alleviates this issue to a certain extent but a too high contribution of
previous frames to counter extreme flickering may lead to ghosting. In such
cases, we see the warping of the previous feature maps as a promising
extension.

Since both limitations are a result of imperfect input data, a combination of
our approach with an inverse rendering back-end might solve these issues. The
idea would be to run the inverse optimization in a secondary thread on the
previous keyframes to refine the pose, depth map, exposure time and white
balance. Our pipeline can then use the optimized keyframes to synthesize high-
quality real-time rendering of the scene.

##  6\. Conclusion

In this work, we have proposed a novel pipeline for real-time novel view
synthesis on RGB-D video streams. The core of our pipeline is the flexible
neural fusion function, which amalgamates the color and depth data of nearby
keyframes in image space of the target view on-the-fly. The major advantage of
our approach is that it requires zero preprocessing or training time on new
scenes and can be readily applied on consumer hardware. During an extensive
evaluation, we have shown that the rendering quality matches or outperforms
other generalizing neural rendering approaches even if they relied on a time-
consuming preprocessing on the dataset.

###### Acknowledgements.

We would like to thank all members of the Visual Computing Lab Erlangen for
their support and fruitful discussions. Specifically, we appreciate Mathias
Harrer’s contribution to the evaluation and Dominik Penk’s help regarding the
dataset preparation. We also thank Ashutosh Mishra for his insights about
prior arts. The authors gratefully acknowledge the scientific support and HPC
resources provided by the National High Performance Computing Center of the
Friedrich-Alexander-Universität Erlangen-Nürnberg (NHR@FAU) under the project
b162dc. NHR funding is provided by federal and Bavarian state authorities.
NHR@FAU hardware is partially funded by the German Research Foundation (DFG) –
440719683. Linus Franke was supported by the Bavarian Research Foundation
(Bay. Forschungsstiftung) AZ-1422-20. Joachim Keinert was supported by the
Free State of Bavaria in the DSAI project.

## References

  * (1)
  * Aliev et al. (2020) Kara-Ali Aliev, Artem Sevastopolsky, Maria Kolos, Dmitry Ulyanov, and Victor Lempitsky. 2020\.  Neural Point-Based Graphics (NPBG). In _European Conference on Computer Vision (ECCV)_ _(Lecture urls in Computer Science)_ , Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer, Cham, 696–712.  https://doi.org/10.1007/978-3-030-58542-6_42
  * Barron et al. (2022) Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. 2022\.  Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields.  _CVPR_ (2022).
  * Bergman et al. (2021) Alexander W. Bergman, Petr Kellnhofer, and Gordon Wetzstein. 2021\.  Fast Training of Neural Lumigraph Representations Using Meta Learning.  _arXiv_ (June 2021).  arXiv:2106.14942
  * Cao et al. (2022) Ang Cao, Chris Rockwell, and Justin Johnson. 2022.  FWD: Real-Time Novel View Synthesis With Forward Warping and Depth. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 15713–15724.
  * Cao et al. (2023) Junli Cao, Huan Wang, Pavlo Chemerys, Vladislav Shakhrai, Ju Hu, Yun Fu, Denys Makoviichuk, Sergey Tulyakov, and Jian Ren. 2023.  Real-Time Neural Light Field on Mobile Devices. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 8328–8337.
  * Chang et al. (2015) Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. 2015.  ShapeNet: An Information-Rich 3D Model Repository.  arXiv:1512.03012
  * Chen et al. (2021) Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. 2021.  Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.  (2021), 14124–14133.
  * Choe et al. (2021) Jaesung Choe, Sunghoon Im, Francois Rameau, Minjun Kang, and In So Kweon. 2021.  Volumefusion: Deep depth fusion for 3d scene reconstruction.  (2021), 16086–16095.
  * Choi et al. (2016) Sungjoon Choi, Qian-Yi Zhou, Stephen Miller, and Vladlen Koltun. 2016.  A Large Dataset of Object Scans.  _arXiv:1602.02481 [cs]_ (May 2016).  arXiv:1602.02481 [cs]
  * Clark (2022) Ronald Clark. 2022\.  Volumetric Bundle Adjustment for Online Photorealistic Scene Capture. In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, New Orleans, LA, USA, 6114–6122.  https://doi.org/10.1109/CVPR52688.2022.00603
  * Dai et al. (2017a) Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. 2017a.  Scannet: Richly-annotated 3d Reconstructions of Indoor Scenes. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5828–5839.
  * Dai et al. (2017b) Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt. 2017b.  Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration.  _ACM Transactions on Graphics (ToG)_ 36, 4 (2017), 1\.
  * Esposito et al. (2022) Stefano Esposito, Daniele Baieri, Stefan Zellmann, André Hinkenjann, and Emanuele Rodolà. 2022\.  KiloNeuS: A Versatile Neural Implicit Surface Representation for Real-Time Rendering.  arXiv:2206.10885 [cs]
  * Furukawa and Hernández (2015) Yasutaka Furukawa and Carlos Hernández. 2015.  _Multi-View Stereo: A Tutorial_.  Number 9,1/2 in Foundation and Trends in Computer Graphics and Vision. Now, Boston Delft.
  * Griwodz et al. (2021) Carsten Griwodz, Simone Gasparini, Lilian Calvet, Pierre Gurdjos, Fabien Castan, Benoit Maujean, Gregoire De Lillo, and Yann Lanthony. 2021\.  AliceVision Meshroom: An Open-Source 3D Reconstruction Pipeline. In _MMSys: ACM Multimedia Systems Conference_. Istanbul Turkey, 241–247.  https://doi.org/10.1145/3458305.3478443
  * Haitz et al. (2023) Dennis Haitz, Boris Jutzi, Markus Ulrich, Miriam Jaeger, and Patrick Huebner. 2023.  Combining HoloLens with Instant-NeRFs: Advanced Real-Time 3D Mobile Mapping.  arXiv:2304.14301 [cs]
  * Hedman et al. (2019) Peter Hedman, Julien Philip, True Price, Jan-Michael Frahm, George Drettakis, and Gabriel Brostow. 2019\.  Deep Blending for Free-Viewpoint Image-Based Rendering.  _ACM Transactions on Graphics_ 37, 6 (Jan. 2019), 1–15.  https://doi.org/10.1145/3272127.3275084
  * Jahromi et al. (2020) Hamed Z. Jahromi, Ivan Bartolec, Edwin Gamboa, Andrew Hines, and Raimund Schatz. 2020.  You Drive Me Crazy! Interactive QoE Assessment for Telepresence Robot Control. In _2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX)_. 1–6.  https://doi.org/10.1109/QoMEX48832.2020.9123117
  * Jain et al. (2023) Nishant Jain, Suryansh Kumar, and Luc Van Gool. 2023\.  Enhanced Stable View Synthesis.  arXiv:2303.17094 [cs]
  * Jensen et al. (2014) Rasmus Jensen, Anders Dahl, George Vogiatzis, Engil Tola, and Henrik Aanaes. 2014.  Large Scale Multi-view Stereopsis Evaluation. In _2014 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, Columbus, OH, USA, 406–413.  https://doi.org/10.1109/CVPR.2014.59
  * Jiang et al. (2023) Yuheng Jiang, Kaixin Yao, Zhuo Su, Zhehao Shen, Haimin Luo, and Lan Xu. 2023\.  Instant-NVR: Instant Neural Volumetric Rendering for Human-object Interactions from Monocular RGBD Stream.  arXiv:2304.03184 [cs]
  * Knapitsch et al. (2017) Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. 2017.  Tanks and temples: Benchmarking large-scale scene reconstruction.  _ACM Transactions on Graphics (ToG)_ 36, 4 (2017), 1–13.
  * Koestler et al. (2022) Lukas Koestler, Nan Yang, Niclas Zeller, and Daniel Cremers. 2022\.  TANDEM: Tracking and Dense Mapping in Real-time Using Deep Multi-view Stereo. In _Proceedings of the 5th Conference on Robot Learning_. PMLR, 34–45.
  * Kopanas et al. (2021) Georgios Kopanas, Julien Philip, Thomas Leimkühler, and George Drettakis. 2021.  Point-Based Neural Rendering with Per-View Optimization.  _Computer Graphics Forum_ (2021), 16.  https://doi.org/10/gk669d
  * Li et al. (2022a) Chaojian Li, Sixu Li, Yang Zhao, Wenbo Zhu, and Yingyan Lin. 2022a.  RT-NeRF: Real-Time On-Device Neural Radiance Fields Towards Immersive AR/VR Rendering.  arXiv:2212.01120 [cs]
  * Li et al. (2022b) Lingzhi Li, Zhen Shen, Zhongshu Wang, Li Shen, and Ping Tan. 2022b.  Streaming Radiance Fields for 3D Video Synthesis.  arXiv:2210.14831 [cs]
  * Li et al. (2023) Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. 2023.  DynIBaR: Neural Dynamic Image-Based Rendering.  arXiv:2211.11082 [cs]
  * Lin et al. (2022) Haotong Lin, Sida Peng, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowei Zhou. 2022.  Efficient Neural Radiance Fields for Interactive Free-viewpoint Video. In _SIGGRAPH Asia 2022 Conference Papers_. ACM, Daegu Republic of Korea, 1–9.  https://doi.org/10.1145/3550469.3555376
  * Lionar et al. (2021) Stefan Lionar, Lukas Schmid, Cesar Cadena, Roland Siegwart, and Andrei Cramariuc. 2021.  NeuralBlox: Real-Time Neural Representation Fusion for Robust Volumetric Mapping. In _2021 International Conference on 3D Vision (3DV)_. 1279–1289.  https://doi.org/10/gpcgwz
  * Macario Barros et al. (2022) Andréa Macario Barros, Maugan Michel, Yoann Moline, Gwenolé Corre, and Frédérick Carrel. 2022.  A Comprehensive Survey of Visual SLAM Algorithms.  _Robotics_ 11, 1 (Feb. 2022), 24\.  https://doi.org/10/gpjddp
  * Mallick et al. (2014) Tanwi Mallick, Partha Pratim Das, and Arun Kumar Majumdar. 2014.  Characterizations of Noise in Kinect Depth Images: A Review.  _IEEE Sensors Journal_ 14, 6 (June 2014), 1731–1740.  https://doi.org/10.1109/JSEN.2014.2309987
  * Mildenhall et al. (2020) Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020\.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In _European Conference on Computer Vision (ECCV)_ , Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer, Cham, 405–421.
  * Müller et al. (2022b) Thomas Müller, Alex Evans, Christoph Schied, Marco Foco, András Bódis-Szomorú, Isaac Deutsch, Michael Shelley, and Alexander Keller. 2022b.  Instant Neural Radiance Fields. In _Special Interest Group on Computer Graphics and Interactive Techniques Conference Real-Time Live!_ ACM, Vancouver BC Canada, 1–2.  https://doi.org/10/gqkshw
  * Müller et al. (2022a) Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022a.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Transactions on Graphics (ToG)_ 41, 4 (2022), 1–15.
  * Mur-Artal and Tardos (2017) Raul Mur-Artal and Juan D. Tardos. 2017.  ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras (arXiv).  _IEEE Transactions on Robotics_ 33, 5 (Oct. 2017), 1255–1262.  https://doi.org/10/gdz797 arXiv:1610.06475
  * Newcombe et al. (2011) Richard A. Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J. Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. 2011.  KinectFusion: Real-time Dense Surface Mapping and Tracking. In _2011 10th IEEE International Symposium on Mixed and Augmented Reality_. 127–136.  https://doi.org/10/dhvm3p
  * Ortiz et al. (2022) Joseph Ortiz, Alexander Clegg, Jing Dong, Edgar Sucar, David Novotny, Michael Zollhoefer, and Mustafa Mukadam. 2022.  iSDF: Real-Time Neural Signed Distance Fields for Robot Perception.  _arXiv:2204.02296 [cs]_ (April 2022).  arXiv:2204.02296 [cs]
  * Rakhimov et al. (2022) Ruslan Rakhimov, Andrei-Timotei Ardelean, Victor Lempitsky, and Evgeny Burnaev. 2022\.  NPBG++: Accelerating Neural Point-Based Graphics. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 15969–15979.
  * Reiser et al. (2023) Christian Reiser, Richard Szeliski, Dor Verbin, Pratul P. Srinivasan, Ben Mildenhall, Andreas Geiger, Jonathan T. Barron, and Peter Hedman. 2023.  MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes.  arXiv:2302.12249 [cs]
  * Riegler and Koltun (2020) Gernot Riegler and Vladlen Koltun. 2020.  Free View Synthesis.  _arXiv:2008.05511 [cs]_ (Aug. 2020).  arXiv:2008.05511 [cs]
  * Riegler and Koltun (2021) Gernot Riegler and Vladlen Koltun. 2021.  Stable View Synthesis. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 12216–12225.
  * Rosinol et al. (2022) Antoni Rosinol, John J. Leonard, and Luca Carlone. 2022\.  NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fields.  arXiv:2210.13641 [cs]
  * Rückert et al. (2022) Darius Rückert, Linus Franke, and Marc Stamminger. 2022\.  Adop: Approximate differentiable one-pixel point rendering.  _ACM Transactions on Graphics (ToG)_ 41, 4 (2022), 1–14.
  * Rückert et al. (2019) D. Rückert, M. Innmann, and M. Stamminger. 2019\.  FragmentFusion: A Light-Weight SLAM Pipeline for Dense Reconstruction. In _2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)_. 342–347.  https://doi.org/10/gjhfv4
  * Rückert and Stamminger (2021) Darius Rückert and Marc Stamminger. 2021.  Snake-SLAM: Efficient Global Visual Inertial SLAM Using Decoupled Nonlinear Optimization. In _2021 International Conference on Unmanned Aircraft Systems (ICUAS)_. 219–228.  https://doi.org/10.1109/ICUAS51884.2021.9476760
  * Schönberger et al. (2016) Johannes L. Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. 2016.  Pixelwise View Selection for Unstructured Multi-View Stereo (COLMAP). In _European Conference on Computer Vision (ECCV)_ , Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (Eds.), Vol. 9907. European Conference on Computer Vision (ECCV), Springer, Cham, 501–518.  https://doi.org/10.1007/978-3-319-46487-9_31
  * Schöps et al. (2020) Thomas Schöps, Torsten Sattler, and Marc Pollefeys. 2020\.  SurfelMeshing: Online Surfel-Based Mesh Reconstruction.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 42, 10 (Oct. 2020), 2494–2507.  https://doi.org/10/gh84zp
  * Song et al. (2022) Liangchen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger. 2022\.  NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fields.  arXiv:2210.15947 [cs]
  * Stereolabs (2023) Documentation Stereolabs. 2023\.  Depth Sensing: Depth Settings.  https://www.stereolabs.com/docs/depth-sensing/depth-settings/.
  * Sucar et al. (2021) Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew J Davison. 2021\.  iMAP: Implicit Mapping and Positioning in Real-Time. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 6229–6238.
  * Sun et al. (2021) Cheng Sun, Min Sun, and Hwann-Tzong Chen. 2021.  Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction (DVGO).  _arXiv:2111.11215 [cs]_ (Nov. 2021).  arXiv:2111.11215 [cs]
  * Tan and Sato (2020) Joo Kooi Tan and Akitoshi Sato. 2020.  Human-Robot Cooperation Based on Visual Communication.  https://doi.org/10.24507/ijicic.16.02.543
  * Tewari et al. (2020) Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, et al. 2020\.  State of the art on neural rendering. In _Computer Graphics Forum_ , Vol. 39. Wiley Online Library, 701–727.
  * Thies et al. (2019) Justus Thies, Michael Zollhöfer, and Matthias Nießner. 2019\.  Deferred Neural Rendering: Image Synthesis Using Neural Textures.  _ACM Transactions on Graphics_ 38, 4 (July 2019), 1–12.  https://doi.org/10.1145/3306346.3323035 arXiv:1904.12356
  * Wang et al. (2021a) Fangjinhua Wang, Silvano Galliani, Christoph Vogel, Pablo Speciale, and Marc Pollefeys. 2021a.  Patchmatchnet: Learned multi-view patchmatch stereo.  (2021), 14194–14203.
  * Wang et al. (2021b) Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P. Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. 2021b.  IBRNet: Learning Multi-View Image-Based Rendering. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. Virtual, 4690–4699.
  * Weder et al. (2021) Silvan Weder, Johannes L Schonberger, Marc Pollefeys, and Martin R Oswald. 2021.  Neuralfusion: Online depth fusion in latent space.  (2021), 3162–3172.
  * Whelan et al. (2016) Thomas Whelan, Renato F Salas-Moreno, Ben Glocker, Andrew J Davison, and Stefan Leutenegger. 2016\.  ElasticFusion: Real-time Dense SLAM and Light Source Estimation.  _The International Journal of Robotics Research_ 35, 14 (Dec. 2016), 1697–1716.  https://doi.org/10/f9k45t
  * Xie et al. (2022) Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. 2022\.  Neural fields in visual computing and beyond.  , 641–676 pages.
  * Zhang et al. (2018) Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_. IEEE, Salt Lake City, UT, 586–595.  https://doi.org/10/gfz33w
  * Zhu et al. (2021) Qingtian Zhu, Chen Min, Zizhuang Wei, Yisong Chen, and Guoping Wang. 2021.  Deep Learning for Multi-View Stereo via Plane Sweep: A Survey.  _arXiv:2106.15328 [cs]_ (July 2021).  arXiv:2106.15328 [cs]
  * Zhu et al. (2022) Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Oswald, and Marc Pollefeys. 2022\.  NICE-SLAM: Neural Implicit Scalable Encoding for SLAM.  (2022), 12786–12796.

Figure 11. NVS results on Motorcycle, Sofa and Scannet.

Supplemental Material

In this supplementary material, we provide extra ablation studies and details
on experiments and implementation.

##  Appendix A Implementation details

###  A.1. Camera Pose Estimation

For the purpose of tracking, we employed Snake-SLAM (Rückert and Stamminger,
2021), for both the Redwood and Zed datasets. The evaluation on Scannet was
done using the provided camera parameters and for Tanks and Temples we used
COLMAP (Schönberger et al., 2016) reconstructions.

###  A.2. Details on Training

The used dataset is a combination of 82 Scannet scenes and 14 Tanks and
Temples scenes. In each iteration, eight views are drawn from the list
provided by the view selection algorithm and sampled according to pre-computed
per-pixel lists. These views are then warped and fused into a target view. As
loss we solely used the VGG loss (Zhang et al., 2018), computed from predicted
target frame and reference. In total, we run this training for 500K steps with
a batch size of 1 and learning rate $10^{-4}$ using the Adam optimizer. This
took 24h on an RTX 3090 GPU.

###  A.3. Temporal Feedback of the Decoder

During inference, intermediate feature maps of the current and the previous
frame are blended according to a blending factor within [0, 1] that can be set
interactively. To allow for this feature, we randomly sampled its value during
training. In all of our evaluation experiments, we set this blending factor to
0.1.

##  Appendix B Further Evaluation

###  B.1. Weighting Scheme

Figure 12 illustrates the influence of the involved weights on the output
image.

Figure 12. Influence of weighting scheme. (a) Reference, (b)
$w=(w_{d}w_{v}w_{i})^{5}$, (c) $w=w_{d}w_{v}w_{i}$, (d) $w=1$, (e) $w=w_{v}$,
(f) $w=w_{d}$, (g) $w=w_{i}$.

###  B.2. Auxiliary Features

We evaluated different pipeline variations which mainly focused on providing
additional in- and outputs of the en- and decoding networks to improve
quality. We did not incorporate additional networks like MLPs for feature
translation (Riegler and Koltun, 2021) or transformers (Wang et al., 2021b) as
done in related work to keep the pipeline as light-weight as possible.

The simplest one is using RGB as input for the encoder only. Additionally, we
may add depth (+D) as additional feature, where +Do means the original depth
from the depth camera was provided as input for the encoder and the fused
depth of the target view was provided for the decoder. A fused depth map of
the source images was provided for the encoder for the variant +Df (which is
only possible when backwards warping). +V indicates that we additionally
compute per-pixel feature variance of the target feature map. The variant +$c$
lets the encoder output an additional channel which is used as weight during
the fusion phase and expresses confidence of predicted neural features.

The Table 6 shows mean results for the evaluation scenes of the Scannet
dataset. Most variations performed very similar and are within a 1 % margin.
SSIM was not sensitive to image differences at all, while L1 showed up to 2.5
% relative difference between the best and worst performing variant.

In general, we can observe slightly improved metric values, less blurring and
artifact reduction when adding input features to the networks. We identify
RGB+Df+$c$ as best compromise between image quality improvements and
investment of additional compute time and memory consumption.

Variant | PSNR $\uparrow$ | L1 $\downarrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
---|---|---|---|---
RGB | 22.44 | 0.1428 | 0.7967 | 0.4016
RGB+$c$ | 22.48 | 0.1422 | 0.7976 | 0.3986
RGB+V | 22.51 | 0.1428 | 0.7974 | 0.4032
RGB+Df | 22.60 | 0.1399 | 0.7981 | 0.4019
RGB+Df+$c$ | 22.57 | 0.1392 | 0.7972 | 0.4007
RGB+Df+V | 22.55 | 0.1405 | 0.7973 | 0.4010
RGB+Do+V | 22.47 | 0.1411 | 0.7953 | 0.3987
RGB+Df+$c$+V | 22.46 | 0.1427 | 0.7962 | 0.4011
Table 6. Performance of various network variants. Bold entries performed best
within 0.5% relative difference.

###  B.3. Additional Results

Table 7. Quantitative comparison of the rendering quality. Note, that our approach is the only method that directly processes the live RGB-D stream, while all other approach use the globally fused point cloud or mesh. | Scannet |  |  |  | Sofa |  |  |  | Motorcycle |  |  |
---|---|---|---|---|---|---|---|---|---|---|---|---
| PSNR $\uparrow$ | L1 $\downarrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | L1 $\downarrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | L1 $\downarrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
Ours | 20.40 | 0.061 | 0.747 | 0.274 | 17.15 | 0.093 | 0.594 | 0.303 | 19.48 | 0.082 | 0.714 | 0.234
FWD | 17.27 | 0.106 | 0.695 | 0.428 | 15.27 | 0.121 | 0.488 | 0.451 | 19.87 | 0.072 | 0.690 | 0.295
SVS | 23.76 | 0.042 | 0.796 | 0.316 | 20.01 | 0.061 | 0.694 | 0.247 | 22.64 | 0.051 | 0.744 | 0.237
IBRNet | 24.96 | 0.035 | 0.802 | 0.248 | 23.56 | 0.041 | 0.758 | 0.179 | 27.77 | 0.028 | 0.821 | 0.149
NPBG++ | 20.90 | 0.066 | 0.758 | 0.396 | 15.34 | 0.125 | 0.535 | 0.500 | 17.40 | 0.103 | 0.654 | 0.366
DIBR | 19.62 | 0.065 | 0.708 | 0.325 | 16.69 | 0.091 | 0.592 | 0.290 | 18.28 | 0.084 | 0.662 | 0.313
Table 8. Quantitative comparison of the rendering quality for finetuned (ft) experiments. | Scannet |  |  |  | Sofa |  |  |  | Motorcycle |  |  |
---|---|---|---|---|---|---|---|---|---|---|---|---
| PSNR $\uparrow$ | L1 $\downarrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | L1 $\downarrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | L1 $\downarrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
Oursft | 21.15 | 0.055 | 0.758 | 0.260 | 19.59 | 0.077 | 0.716 | 0.200 | 17.36 | 0.092 | 0.618 | 0.247
SVSft | 23.26 | 0.044 | 0.785 | 0.260 | 25.04 | 0.038 | 0.775 | 0.162 | 20.11 | 0.060 | 0.688 | 0.206
IBRNetft | 25.42 | 0.033 | 0.804 | 0.229 | 28.14 | 0.027 | 0.812 | 0.163 | 23.59 | 0.042 | 0.743 | 0.182
NPBG++ft | 21.66 | 0.058 | 0.763 | 0.363 | 17.02 | 0.111 | 0.637 | 0.392 | 15.26 | 0.122 | 0.521 | 0.496
ADOP | 24.78 | 0.039 | 0.694 | 0.208 | 24.26 | 0.044 | 0.598 | 0.172 | 20.34 | 0.066 | 0.585 | 0.236

Table 7 show additional metrics to the experiment conducted in the paper.
Additionally, we performed some tests regarding scene-specific fine-tuning.
Compared to methods which are tailored to scene-specific reconstruction (often
needing hours per scene or multiple minutes per view), metrics indicate only
mid-range results, see Table 8. However, Figure 13 illustrates that
synthesized views generally look plausible.

Figure 13. Finetuned NVS Results on Motorcycle, Sofa and Scannet.