figure. Given a sequence of depth images, with pose, corrupted by noise and missing data,
our framework models the scene geometry and  contextual information with
a fully-convolutional neural network, generating a high-quality, complete mesh for the underlying scene, which is represented using
local implicit grid. Benefiting from sparsity, our method is fast and accurate: the inference time for this scene is only 17s, 10$\times$ faster than the method in [32].

We present , a framework for large-scale scene completion and geometric refinement based on local implicit signed distance functions.
It is based on an end-to-end sparse convolutional network, , that jointly models local geometric details and global scene structural contexts, allowing it to preserve fine-grained object detail while recovering missing regions commonly arising in traditional 3D scene data.
A novel differentiable rendering module enables test-time refinement for better reconstruction quality.
Extensive experiments on both real-world and synthetic
datasets show that our concise framework is  efficient and effective, achieving  better reconstruction quality than the closest competitor while being 10–50 $\times$ faster.

§ INTRODUCTION

In recent years, 3D reconstruction from RGB-D camera data has been widely explored thanks to its ease of acquisition with many applications in robotic perception, virtual reality,  games, .
It is well-accepted that an ideal reconstruction algorithm should be capable of simultaneously (i) restoring fine-grained geometric detail in the target scene, (ii) handling
large scenes efficiently, and (iii) completing missing regions of the scene.
Additionally, the underlying 3D representation should be flexible enough to allow further optimization of geometric quality.

However, traditional algorithms along with their accompanying representations fail to effectively fulfil the above requirements.
For instance, methods using the
truncated signed distance function (TSDF) [8, 27] are hampered by limited voxel resolution and lack robustness to noisy data.
Surfels [43] offer more flexibility by treating the 3D scene
as unstructured points, but maintaining  correct topology is challenging.
Furthermore, such methods as these cannot fill in missing geometry in the scene, which is common in practice due to sensor limitations, incomplete coverage of the scanning trajectory, or unreachable areas.

The recent introduction of deep implicit representations [31, 25, 5] has enabled a plethora of research directions for 2D and 3D data processing.
Parameterized by a neural network, implicit functions are inherently continuous and differentiable.
Notably, in the field of 3D reconstruction, various works [6, 7, 13, 14, 29] have already demonstrate their ability to learn object-level geometric priors from shape repositories.
However, when applied to large-scale scenes, the above methods are typically impractical, fro three reasons. Firstly,
scene structures are substantially more complicated than a single object. A typical end-to-end, optimization-free, framework is weak at capturing the entangled geometric priors of cluttered regions.
Secondly, while other work exists [19, 35, 39, 1] that overfits the scene geometry, to avoid the necessity of prior learning,
it usually involves costly optimization procedures.
Thirdly, some efforts [32, 18, 37] have been made to reconstruct  scenes in real-time with deep implicit functions, but they do so at the cost of low reconstruction quality.

To tackle these issues, we introduce the  framework, as shown in <ref>. It employs a novel , short for fully-convolutional implicit network for reconstruction and completion of large-scale indoor 3D scenes from partial point clouds. It is capable of both preserving scene geometric details and completing missing regions of the scene in a semantically-meaningful way.
Specifically, we adopt local implicit grid to represent  local details of the overall scene, and learn global contextual information for scene completion via a sparse U-Net.

Our network is also efficient, in that it encodes and decodes the sparsity pattern of the scene geometry by learning, and only non-empty portions need to be evaluated.
Furthermore, we provide a fast and novel differentiable rendering approach tailored for refining our output representation, which can greatly improve the geometric quality during inferencing to provide resilience in the face of  errors in the raw input.
Extensive experiments using various datasets demonstrate the effectiveness of our framework, which sets a new state-of-the-art for scene reconstruction and completion. In benchmarks it is 10–50 $\times$ faster than previous methods.

§ RELATED WORK

Scene Reconstruction
Building a high-quality and coherent scene-level reconstruction is challenging due to  noise, occlusion and missing data inherent in 3D data acquisition sensors.
While traditional methods [27, 43, 10, 30] incrementally fuse input depth observations using a moving average [8], learning methods [41, 42] can further reduce  noise using data-driven geometric biases.
The recent trend of using implicit neural representations, such as DI-Fusion [18] and its successors [38, 3], either uses localized priors or the continuous nature of a globally-supported network function.
In comparison, our method can not only accurately recover detailed scene geometry, but also rebuild missing parts via global structural reasoning based on learning.

Scene Completion
The main challenge in scene completion is to fill missing regions with data that are semantically coherent with the existing content.
[36] casts the problem in terms of panoramic image completion but important geometric details are significantly missing.
[11] first brings the aid of semantic segmentation to the completion problem in the 3D domain.

Subsequent lines of work [9, 12] tackle the problems of geometric sparsity and color generation.
We note that many end-to-end frameworks [32, 1] using implicit representations also provide decent scene extrapolation due to the continuous nature of networks, even though they are not specifically designed for this task.

Differentiable Rendering
The technique of differentiating the rendering process bridges the gap between 3D geometry and 2D observations of it by allowing for end-to-end optimization directly from captured raw sensor data, which was first applied to triangular meshes [22, 20] and later to implicit fields [23, 28].
The prevalence of NeRF [26] motivates many studies to improve rendering efficiency and fitting speed, either through localized structures [21], level-of-detail rendering [39], caching [45], or multi-view stereo [44, 34].
In conjunction with our novel local implicit representation, we devise a new differentiable rendering approach can rapidly and effectively refine detail geometries of the reconstructed scene during inferencing.

Pipeline. We first voxelize the accumulated unprojected points from the input posed depth frames into sparse grid. The feature volume is then passed through , which comprises 3 neural networks: $\encnet$, $\unet$, and $\decnet$. The output reconstructed geometry is an implicit completed surface. Inference-time refinement is enabled by our differentiable rendering algorithm; it optimizes both  scene geometry and  camera pose, leading to better and more complete reconstruction.

§ : CONVOLUTIONAL IMPLICIT SCENE RECONSTRUCTION AND COMPLETION

Problem Formulation
The input to our method is a sequence of depth frames with pose $\{ \depth_t, \pose_t \}_{t=1}^T$, with $\depth_t \in \R^{W \times H}$ and $\pose_t \in \SE(3)$ being the depth image and the
6-DoF camera pose, respectively.
Our goal is to build a high-quality and complete 3D reconstruction of the scene, represented using $M$
local sparse implicit voxel grid $\V = \{(\c_m,\lvec_m)\}_{m=1}^M$
that contain
the surface of the scene geometry. Here, $\c_m \in \R^3$ is
the voxel coordinate and $\lvec_m \in \R^L$ is the latent vector describing the local voxel grid's geometry, from which we can decode the signed distance values of the full scene and finally extract the mesh. The size
of each voxel is $b\times b\times b$.

As <ref> shows, we first unproject all the depths $\depth_t$ under the given poses $\pose_t$ to obtain an accumulated point cloud $\P = \{(\p_i,\normal_i)\}_{i=1}^N$ where $\p_i \in \R^3$ and $\normal_i \in \R^3$ are  point positions and their estimated normals, using [27].
$\P$ is then voxelized into initial sparse 3D grid and processed by   (see <ref>).
Being aware of both global scene structure and local geometric details,  simultaneously refines the voxelized points and adds additional points with a point encoder $\encnet$ and U-Net $\unet$, and produces $\V$ defining the latent vector of local implicit geometry, which is then decoded to TSDF values
by a multi-layer perceptron (MLP) $\decnet$.
One can later extract the mesh using marching cubes [24] from these TSDF values.
Moreover, the reconstructed geometry can be further optimized during inferencing time via a novel differentiable rendering scheme described in <ref>, to refine both the scene geometry and the camera pose.
Detailed loss functions for the training procedure and inference-time refinement are discussed in <ref>.

§.§  Architecture

Given the unprojected point cloud $\P$ from the input views,   sequentially applies three trainable components: a point encoder network $\encnet$, a U-Net $\unet$, and an SDF decoder $\decnet$ to produce an implicit representation of the underlying scene. We now describe these in turn.

Point Encoder
We first split the input point cloud into multiple voxel grid.
For point $\p_i$, the index of its corresponding voxel $m_i$ is determined by $m$ satisfying $\p_i \in [\c_m , \c_m + b)$.
We define the local coordinates of $\p_i$ within its voxel as $\p_i^l = (\p_i - \c_m) / b \in [0,1]^3$.
Next, for each voxel $m$, we feed all  local coordinates of  points within the voxel, along with their normals: $\{ (\p_i^l, \normal_i) \in \R^6 \eqgap | \eqgap m_i = m \}$ into a point encoder $\encnet$.
This uses a basic PointNet [33] structure by first  mapping all the input features into $L$-dimensions with a shared MLP and then aggregating the features via mean pooling.
The resulting sparse feature voxel grid is denoted $\V_0$.

The goal of the U-Net $\unet$ in this step is to complete and refine the reconstruction from $\V_0$ into $\V$.

This is done by propagating contextual features in the hierarchical U-Net structure with a large receptive field.
A trivial implementation falls back to a dense convolution that generates a dense feature grid even if many voxels are actually empty.
Due to the sparse nature of the geometry, we instead use submanifold sparse convolution [15] for our convolution layer.

For the decoder branch, inspired by [40], we append a sparsity prediction module to each layer of the decoder.
This module is instantiated with a shared MLP applied to each voxel and predicts the confidence of the current voxel containing true surfaces; voxels with scores lower than 0.5 are pruned.
Accordingly, usual skip connections are replaced by sparsity-guided skip connections: connections are only added for  voxels predicted to be non-empty.
Apart from the efficiency gain, this design also eases network training by obviating the need to model the full geometry of empty regions.

SDF Decoder
To recover the final scene geometry, we traverse all  points $\p$ in the non-empty regions of $\V$ and learn signed distance values  using an implicit decoder instantiated with an MLP $\decnet: (\p^l, \hat{\lvec}) \in \R^{3+L} \mapsto [-1,1]$, where $\p^l$ is the local coordinate of $\p$  and $\hat{\lvec}$ is the interpolated feature taken from $\V$.
To achieve smooth geometric interpolation across  voxel boundaries, we apply an additional $2\times 2 \times 2$ convolution over $\V$ to propagate the features stored at voxel centers to voxel corners, obtaining $\{\lvec'_m \}$.
The input feature $\hat{\lvec}$ can  then be trilinearly interpolated $\psi(\cdot)$ from the features stored at its 8 nearest voxel corners: $\hat{\lvec} = \psi(\p^l, \{ \lvec_{(1)}', \dots, \lvec_{(8)}'\} )$.

§.§ Differentiable Local Implicit Rendering

Despite the good-quality, end-to-end reconstruction provided by , some desired geometric details can be lost.
The reasons are two-fold.
Firstly, real-world depth captures usually suffer from noisy pose and sensor limitations, resulting in erroneous reconstruction and severe missing regions.
Secondly, a simple feed-forward network trained on large-scale datasets can underfit geometric features or generate excessive contents [31, 25].
Noting these issues, we propose a novel differentiable renderer for our implicit representation, allowing for effective differentiation through both geometry and  camera pose.
Specifically, for each pixel to be rendered, we emit a ray with an origin $\rayc$ and a unit direction $\rayd$, and compute the depth of the intersection $t$ so that the intersection point is $\p = \rayc + t \rayd$, and forward and backward passes are defined as follows:

Forward Pass
The forward pass is composed of two steps as shown in <ref> (c–d):

* Voxel-level Intersection. As the sparsity prediction modules from the different layers of our U-Net decoder naturally form an octree structure thanks to the upsampling operator, we can use any existing ray-octree intersection algorithm for this step. In our implementation, we choose the fast algorithm in [39] that generates a list of intersection pairs $\{(t^v,m^v)\}$, where $t^v$ is the depth and $m^v$ is the voxel index of the intersection.

* Geometry-level Intersection. The sphere tracing algorithm [17] is applied for each intersecting voxel $m^v$, starting from $\rayc + t^v \rayd$ and ending at $\p^g = \rayc + t^g \rayd$ that hits the surface. Note that only the smallest $t^g$ among all the voxels is returned as the final depth $t$ due to occlusion.

Backward Pass
For clarity,
we abstract our full  as an implicit network $f(\p ; \theta)$ whose inputs are the position $\p$ and the intermediate features or network parameters $\theta$,
and the output is the signed distance value.
We wish to compute the first-order derivative of the depth $t$ $\theta$ as well as the camera ray $\rayc$ and $\rayd$ for optimization.
Inspired by [44], we employ the fact that $f(\rayc + t\rayd; \theta) \equiv 0$ and use  implicit differentiation to obtain:

\begin{align}
\label{ep:dr}
\frac{\partial t}{\partial \theta} = -\gamma \frac{\partial{f}}{\partial \theta}, \quad
\frac{\partial t}{\partial \rayc} = -\gamma \frac{\partial f}{\partial \p}, \quad
\frac{\partial t}{\partial \rayd} = -\gamma t  \frac{\partial f}{\partial \p},
\end{align}

where $\gamma = \langle \rayd, {\partial f}/{\partial \p} \rangle ^{-1}$ is a scalar, $\langle\cdot,\cdot\rangle$ denotes vector inner product, and other derivatives related to $f$ can be efficiently evaluated using reverse-mode back-propagation.
Empirically, we observe that full gradient-based optimization
over all network parameters fails to converge.
Hence we choose to  only optimize the latent vectors in $\V$: $\theta = \{ \lvec_m \}$, and fix all other parts of the networks.

Rendering strategies. Rendering a globally-supported implicit representation with (a) uniform query points [26] and (b) differentiable sphere tracing [23]. (c,d) Our approach with both sphere tracing and implicit differentiation, where $\p^i=\rayc + t^i \rayd, i \in \{v,g\}$.

A comparison between our method and previous approaches is shown in <ref>.
Methods similar to, , NeRF [26] exhaustively query all  points along the ray; most of the unnecessary computations far away from the surface can be saved with sphere tracing [17, 23].
Our use of localized grid further speed up the process thanks to the explicit ray-voxel intersection step that greatly reduces the number of steps in tracing.
Nevertheless, a naive implementation of the backward pass requires unrolling the tracing steps, leading to inaccurate gradients.
We for the first time marry the merits of implicit differentiation, originally designed for global representations [44], with our local feature grid, so that only the intersection points need to be stored in the computation graph, leading to a fast, stable, accurate and memory-efficient method for both  forward and backward passes.
Experiments verifying our design choices are shown in <ref>.

§.§ Loss Functions

Loss Function
The three networks $\encnet$, $\unet$ and $\decnet$ are jointly trained in an end-to-end manner, using the following loss function:

\begin{equation}
\label{total_loss}
\mathcal{L} = \mathcal{L}_\mathrm{sdf} + \alpha \mathcal{L}_\mathrm{norm} + \beta \mathcal{L}_\mathrm{struct} + \delta \sum_{m=1}^M \lVert \lvec_m \rVert,
\end{equation}

where $\lVert \cdot \rVert$ is the vector norm. $\mathcal{L}_\mathrm{sdf}$ is the data term defined as the L1 distance between the predicted signed distance from the decoder $\decnet(\p^l, \hat{\lvec})$ and the ground-truth values $s^{\gt}(\p)$:

\begin{equation}
\label{sdf_loss}
\mathcal{L}_\mathrm{sdf} = \int_{\Omega_u\cup\Omega_n} {|\decnet (\p^l,\hat{\lvec}) - s^\gt (\p)|} \eqgap \mathrm{d} \p.
\end{equation}

Here $\Omega_u$ denotes the occupied region of the voxels $\V$ while $\Omega_n$ is a narrow band region near the surface.

The normal of the predicted geometry, computed as $\nabla_{\p}\decnet$, is constrained by the normal loss:

\begin{align}
\label{eq:normal_loss}
\mathcal{L}_\mathrm{norm} = & \int_{\Omega_u\cup\Omega_n} \big| \| \nabla_{\p}\decnet\| - 1 \big| \eqgap \mathrm{d} \p \eqgap + \nonumber\\
& \int_{\Omega_n} \big( 1 - \langle \nabla_{\p} \decnet, \normal^{\gt} (\p) \rangle \big) \eqgap \mathrm{d} \p,
\end{align}

where the first term enforces the eikonal equation of the signed distance field while the second term minimizes the angle between predicted normal and  ground-truth normal $\normal^\gt$.

$\mathcal{L}_\mathrm{struct}$ uses cross-entropy loss to supervise the sparsity prediction module for each layer in the decoder branch of $\unet$.
Specifically, we obtain the ground-truth sparsity pattern of the target geometry at multiple resolutions in accordance with the output sparsity map from the U-Net, and directly supervise the predicted confidence score.

During training, we use the ground-truth sparsity map instead of the predicted one for the skip-connections and pruning of the next layer.

Inference-time Refinement
During inferencing, our differentiable rendering module is applied to refine the predicted geometry and the camera poses. For each depth image $\depth_t$ and its pose $\pose_t$, we can render a depth image as $\depth'_t(\pose_t,\theta) \in \R^{W\times H}$, whose pixels are the depths $\{t\}$ from <ref>.
By minimizing the error between the rendered depth and the observed depth, we can jointly optimize the quality of geometry and input poses:

\begin{equation}
\label{render_optimize}
\min_{ \theta, \{\delta\pose_t \}} \eqgap \eqgap \sum_{t=1}^T \big| \depth_t - \depth'_t(\delta\pose_t\pose_t,\theta)) \big|,
\end{equation}

we optimize an increment to pose $\delta\pose_t$ instead of $\pose_t$ itself, for better convergence.

§ EXPERIMENTS

§.§ Dataset and Settings

The main dataset used to evaluate our framework is N-Matterport3D.
Adapted from [4], this dataset contains 1788 + 394 (for training / validation and testing respectively) scans of rooms from 90 buildings captured by a Matterport Pro Camera.
We follow the self-supervised setting from [9] by randomly sampling 50% of the frames to generate an incomplete version of each room and supervise our method with a complete version reconstructed from all frames.
To further demonstrate the robustness of our method to noise, we follow [41] and add synthetic noise to each individual depth frame (denoted by the prefix `N-').

We additionally used the well-known ICL-NUIM [16] public benchmark containing 4 scan trajectories for testing only, to demonstrate the generalizability of our method.

Parameter Settings
Our  was trained and tested on a single Nvidia GeForce RTX 2080Ti GPU.

The weights of the loss terms are empirically set to $\alpha = 0.1$, $\beta = 1$ and $\delta = 0.001$.
We used the Adam optimizer with a learning rate of $0.001$.
For efficient training, we uniformly split the input point cloud $\P$ into
patches of size  $3.2\mathrm{m}\times 3.2\mathrm{m}\times 3.2\mathrm{m}$, although as a fully convolutional architecture, our pipeline could easily scale to the full scene during inferencing.
$\encnet$, $\unet$ and $\decnet$ have 4, 5, and  3 layers respectively.
With the scale of indoor scenes, the voxel size $b$ is set to 0.05m and the width of $\Omega_n$ is set to 2.5mm.
Further details of our network structure are given in the supplementary material.

Our method is compared to a full spectrum of methods, including those providing reconstruction from sequential depth frames, , RoutedFusion [41] (denoted R-Fusion) and DI-Fusion [18] using representations of either local implicit grid or a neural signed-distance volume.
We further consider methods operating on  fully-fused geometry, , the
convolutional occupancy network [32] (denoted ConvON) is the state-of-art local implicit network for surface reconstruction considering global information, while SPSG [12] is the up-to-date scene completion approach that takes TSDF volumes as input.
For methods that are cannot be trained on large-scale scenes, we used pre-trained weights obtained from synthetic datasets.

We use root mean square error (RMSE), chamfer distance (CD), surface precision, recall, and F-score during evaluation. RMSE, CD, and surface precision mainly measure the accuracy of the reconstruction, surface recall mainly assesses  the degree of completeness, and F-score reflects both  accuracy and completeness.
All reconstruction results from different methods are converted to point clouds for comparisons.
RMSE and CD are measured in meters, and for
precision and recall,
a predicted or ground truth point is accepted if its distance to the closest ground truth or predicted point is smaller than $0.02$ m.

§.§ Comparisons to Other Methods

<ref> shows that our proposed method works best according to all metrics, for the N-Matterport3D dataset.
Qualitative results are presented in <ref>.
The dense structure of ConvON makes it difficult for it to simultaneously capture local and global information from real-world datasets.
R-Fusion and DI-Fusion only learn local geometric priors from the synthetic datasets.
Specifically, although DI-Fusion fits local details with local implicit functions and achieves competitive performance, its lack of global information prevents it from completing missing regions.
SPSG shows a capability for scene completion; however, limited by the discrete TSDF representation, the precision of the reconstructed surface is unsatisfactory.
Our method learns global contextual information from local implicit grid by the convolutional neural network $\unet$, and thus can faithfully reconstruct local geometric details and recover many missing regions.

We further evaluate the generalizability of all  approaches using the ICL-NUIM dataset; quantitative results are given in <ref>.
Remarkably, although our method is trained using panoramic scans as in [4], thanks to our effective learning scheme in 3D space, it generalizes well to hand-held trajectories whose geometric distributions are drastically different.

Visual comparison using N-Matterport3D. Results show both  global views (part 1, top three rows) and  close-up views (part 2, bottom four rows). The last row in each part shows each method's per-point error, the distance between each reconstructed vertex and the corresponding closest ground truth point.

§.§ Ablation Study

Differentiable Rendering
To demonstrate the capability of our differentiable renderer, we introduce a challenging scenario by adding  zero-mean Gaussian noise to the poses of frames from the N-Matterport3D dataset with a standard deviation of 3 cm

and 2$^\circ$ for the translation and rotation, respectively.
Apart from direct comparisons with the version without differentiable rendering,

we verify the effectiveness of our implicit-differentiation-based gradient by replacing it by unrolled iterations obtained through automatic-differentiation [2], denoted by Ours-AD.

As <ref> shows, our renderer is able to denoise the input poses, reaching a higher reconstruction quality than its counterparts, the refinements of which are non-trivial due to the discrete TSDF representation used.

Moreover, compared to Ours-AD, our full gradient optimization is also more effective, thanks to the accuracy and stability provided by the closed-form derivative computation.

Our method also saves a considerable amount of optimization time and memory by avoiding propagating gradients through all  points along the ray. A detailed time and memory analysis of our differentiable rendering is given in the supplementary material.

Inference-time refinement with differentiable rendering. Our proposed method (Ours) effectively fixes the initial pose error (Ours w/o optim.) and produces a better reconstruction than the baseline (Ours-AD).

Weight of $\mathcal{L}_\mathrm{norm}$
After fixing the gauge freedom of the weights for $\mathcal{L}_\mathrm{sdf}$ and $\mathcal{L}_\mathrm{struct}$ to 1, we show the effect of changing $\mathcal{L}_\mathrm{norm}$ in <ref>  by varying its weight $\alpha \in [0,1]$.
The addition of normal loss can effectively improve the precision of the reconstruction.
However it only works when $\alpha$ is small, showing the importance of carefully choosing the weight parameter, especially in our setting with a small localized voxel size.

Effect of normal loss. Reconstructions obtained by training with varying weights for $\mathcal{L}_\mathrm{norm}$, , $\alpha$. Red boxes highlight differences.

Voxel Size
<ref> shows how the voxel size $b$ (using 5 cm, 7.5 cm and 10 cm)

affects  reconstruction quality.
A smaller voxel size captures more details from the input and models the surface more accurately.
It also improves recall by avoiding mis-predicting large regions.

Furthermore, empirically we find our method generalizes well across different voxel sizes:
The test error with $b=7.5$ cm is stable even if trained using a different voxel size (test CD $\approx 0.019$ for both 5 cm and 10 cm).
Nevertheless, we recommend a larger voxel size during training to learn more complicated geometries for better generalization.

§.§ Timing and Memory

Due to the differences in scene representation used by each approach, it is hard to fairly compare the timing and memory consumption of the whole pipeline of each method. So we only compare the time to provide the discrete TSDF volume for a fair comparison.
<ref> compares the inference time and memory footprint of the baselines for different scene sizes.
Thanks to the sparse feature volume, our method runs 10–50$\times$ faster than ConvON and SPSG, and is comparable in speed to DI-Fusion. However, as the scene gets larger, the time taken by DI-Fusion increases more rapidly than our method due to the difference in voxel interpolation strategy.

As for memory cost, ConvON stays constant due to its sliding window inference scheme.
SPSG maintains a dense discrete TSDF volume, so memory requirements grow drastically with  scene size.
Our method is memory-efficient due to its sparse representation and uses only marginally more memory than DI-Fusion while providing better reconstruction accuracy.

§.§ Limitations and Discussion

Our approach has three main limitations.
Firstly, our network makes no use of  object-level priors, resulting in partially reconstructed objects even after completion. Training with semantic supervision may improve  completion performance.
Secondly, reconstruction quality relies on a small voxel size that limits  further improvements in efficiency. This can be overcome with local implicit grid [19], which can learn local geometric priors from CAD models using large voxels with further optimization for real-world scenes.
Thirdly, textures are not recovered by our method. Inspired by NeRF [26], training a neural radiance field together with SDF using differentiable rendering may be able to help  incorporate texture information into our pipeline.

Performance for varying voxel sizes. Our method works best with a small voxel size $b$.

Run time and memory. Results for different methods and  different input scene sizes.

§ CONCLUSIONS

This paper has  introduced , a framework for large-scale scene reconstruction and completion using local implicit signed distance functions.
The key part of our method is a convolutional neural network that can learn global contextual information from local implicit grid, contributing to the completion of missing regions.
Together with our novel differentiable rendering strategy, we are able to generate an accurate and detailed reconstruction, while being fast and memory-efficient.
In the future, we hope to bridge the gap between large-scale geometric reconstruction and the use of object shape priors, as well as to incorporate color information into our pipeline, for better completion and reconstruction.

[1]
Dejan Azinović, Ricardo Martin-Brualla, Dan B Goldman, Matthias
Nießner, and Justus Thies.
Neural rgb-d surface reconstruction.
arXiv preprint arXiv:2104.04532, 2021.

[2]
Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and
Jeffrey Mark Siskind.
Automatic differentiation in machine learning: a survey.
Journal of machine learning research, 18, 2018.

[3]
Aljaž Božič, Pablo Palafox, Justus Thies, Angela Dai, and
Matthias Nießner.
Transformerfusion: Monocular rgb scene reconstruction using
arXiv preprint arXiv:2107.02191, 2021.

[4]
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner,
Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang.
Matterport3d: Learning from rgb-d data in indoor environments.
International Conference on 3D Vision (3DV), 2017.

[5]
Yinbo Chen, Sifei Liu, and Xiaolong Wang.
Learning continuous image representation with local implicit image
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 8628–8638, 2021.

[6]
Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang.
Bsp-net: Generating compact meshes via binary space partitioning.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 45–54, 2020.

[7]
Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll.
Implicit functions in feature space for 3d shape reconstruction and
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 6970–6981, 2020.

[8]
Brian Curless and Marc Levoy.
A volumetric method for building complex models from range images.
In Proceedings of the 23rd annual conference on Computer
graphics and interactive techniques, pages 303–312, 1996.

[9]
Angela Dai, Christian Diller, and Matthias Nießner.
Sg-nn: Sparse generative neural networks for self-supervised scene
completion of rgb-d scans.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 849–858, 2020.

[10]
Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and
Christian Theobalt.
Bundlefusion: Real-time globally consistent 3d reconstruction using
on-the-fly surface reintegration.
ACM Transactions on Graphics (ToG), 36(4):1, 2017.

[11]
Angela Dai, Daniel Ritchie, Martin Bokeloh, Scott Reed, Jürgen Sturm, and
Matthias Nießner.
Scancomplete: Large-scale scene completion and semantic segmentation
for 3d scans.
In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 4578–4587, 2018.

[12]
Angela Dai, Yawar Siddiqui, Justus Thies, Julien Valentin, and Matthias
Spsg: Self-supervised photometric scene generation from rgb-d scans.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 1747–1756, 2021.

[13]
Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser.
Local deep implicit functions for 3d shape.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), June 2020.

[14]
Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T. Freeman,
and Thomas Funkhouser.
Learning shape templates with structured implicit functions.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision (ICCV), October 2019.

[15]
Benjamin Graham and Laurens van der Maaten.
Submanifold sparse convolutional networks.
arXiv preprint arXiv:1706.01307, 2017.

[16]
A. Handa, T. Whelan, J.B. McDonald, and A.J. Davison.
A benchmark for RGB-D visual odometry, 3D reconstruction and
In IEEE Intl. Conf. on Robotics and Automation, ICRA, Hong
Kong, China, May 2014.

[17]
John C Hart.
Sphere tracing: A geometric method for the antialiased ray tracing of
implicit surfaces.
The Visual Computer, 12(10):527–545, 1996.

[18]
Jiahui Huang, Shi-Sheng Huang, Haoxuan Song, and Shi-Min Hu.
Di-fusion: Online implicit 3d reconstruction with deep priors.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 8932–8941, 2021.

[19]
Chiyu "Max" Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias
Niessner, and Thomas Funkhouser.
Local implicit grid representations for 3d scenes.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), June 2020.

[20]
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada.
Neural 3d mesh renderer.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 3907–3916, 2018.

[21]
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt.
Neural sparse voxel fields.
NeurIPS, 2020.

[22]
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li.
Soft rasterizer: A differentiable renderer for image-based 3d
In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pages 7708–7717, 2019.

[23]
Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, and Zhaopeng
Dist: Rendering deep implicit signed distance function with
differentiable sphere tracing.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 2019–2028, 2020.

[24]
William E Lorensen and Harvey E Cline.
Marching cubes: A high resolution 3d surface construction algorithm.
ACM siggraph computer graphics, 21(4):163–169, 1987.

[25]
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and
Andreas Geiger.
Occupancy networks: Learning 3d reconstruction in function space.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 4460–4470, 2019.

[26]
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi
Ramamoorthi, and Ren Ng.
Nerf: Representing scenes as neural radiance fields for view
In European conference on computer vision, pages 405–421.
Springer, 2020.

[27]
Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim,
Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew
Kinectfusion: Real-time dense surface mapping and tracking.
pages 127–136, 2011.

[28]
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger.
Differentiable volumetric rendering: Learning implicit 3d
representations without 3d supervision.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 3504–3515, 2020.

[29]
Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas
Texture fields: Learning texture representations in function space.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pages 4531–4540, 2019.

[30]
Helen Oleynikova, Zachary Taylor, Marius Fehr, Roland Siegwart, and Juan Nieto.
Voxblox: Incremental 3d euclidean signed distance fields for on-board
mav planning.
In 2017 IEEE/RSJ International Conference on Intelligent Robots
and Systems (IROS), pages 1366–1373. IEEE, 2017.

[31]
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven
Deepsdf: Learning continuous signed distance functions for shape
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 165–174, 2019.

[32]
Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas
Convolutional occupancy networks.
In Computer Vision–ECCV 2020: 16th European Conference,
Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16, pages 523–540.
Springer, 2020.

[33]
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.
Pointnet: Deep learning on point sets for 3d classification and
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 652–660, 2017.

[34]
Radu Alexandru Rosu and Sven Behnke.
Neuralmvs: Bridging multi-view stereo and novel view synthesis.
arXiv preprint arXiv:2108.03880, 2021.

[35]
Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell,
and Gordon Wetzstein.
Implicit neural representations with periodic activation functions.
In Proc. NeurIPS, 2020.

[36]
Shuran Song, Andy Zeng, Angel X Chang, Manolis Savva, Silvio Savarese, and
Thomas Funkhouser.
Im2pano3d: Extrapolating 360 structure and semantics beyond the field
of view.
In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 3847–3856, 2018.

[37]
Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew J. Davison.
imap: Implicit mapping and positioning in real-time.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision (ICCV), pages 6229–6238, October 2021.

[38]
Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew J Davison.
imap: Implicit mapping and positioning in real-time.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pages 6229–6238, 2021.

[39]
Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek
Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler.
Neural geometric level of detail: Real-time rendering with implicit
3d shapes.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 11358–11367, 2021.

[40]
Peng-Shuai Wang, Yang Liu, and Xin Tong.
Deep octree-based cnns with output-guided skip connections for 3d
shape and scene completion.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition Workshops, pages 266–267, 2020.

[41]
Silvan Weder, Johannes Schonberger, Marc Pollefeys, and Martin R Oswald.
Routedfusion: Learning real-time depth map fusion.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 4887–4897, 2020.

[42]
Silvan Weder, Johannes L Schonberger, Marc Pollefeys, and Martin R Oswald.
Neuralfusion: Online depth fusion in latent space.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 3162–3172, 2021.

[43]
Thomas Whelan, Stefan Leutenegger, R Salas-Moreno, Ben Glocker, and Andrew
Elasticfusion: Dense slam without a pose graph.

[44]
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen,
and Yaron Lipman.
Multiview neural surface reconstruction by disentangling geometry and
Advances in Neural Information Processing Systems, 33, 2020.

[45]
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa.
Plenoctrees for real-time rendering of neural radiance fields.
arXiv preprint arXiv:2103.14024, 2021.

[1]
Dejan Azinović, Ricardo Martin-Brualla, Dan B Goldman, Matthias
Nießner, and Justus Thies.
Neural rgb-d surface reconstruction.
arXiv preprint arXiv:2104.04532, 2021.

[2]
Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and
Jeffrey Mark Siskind.
Automatic differentiation in machine learning: a survey.
Journal of machine learning research, 18, 2018.

[3]
Aljaž Božič, Pablo Palafox, Justus Thies, Angela Dai, and
Matthias Nießner.
Transformerfusion: Monocular rgb scene reconstruction using
arXiv preprint arXiv:2107.02191, 2021.

[4]
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner,
Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang.
Matterport3d: Learning from rgb-d data in indoor environments.
International Conference on 3D Vision (3DV), 2017.

[5]
Yinbo Chen, Sifei Liu, and Xiaolong Wang.
Learning continuous image representation with local implicit image
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 8628–8638, 2021.

[6]
Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang.
Bsp-net: Generating compact meshes via binary space partitioning.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 45–54, 2020.

[7]
Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll.
Implicit functions in feature space for 3d shape reconstruction and
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 6970–6981, 2020.

[8]
Brian Curless and Marc Levoy.
A volumetric method for building complex models from range images.
In Proceedings of the 23rd annual conference on Computer
graphics and interactive techniques, pages 303–312, 1996.

[9]
Angela Dai, Christian Diller, and Matthias Nießner.
Sg-nn: Sparse generative neural networks for self-supervised scene
completion of rgb-d scans.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 849–858, 2020.

[10]
Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and
Christian Theobalt.
Bundlefusion: Real-time globally consistent 3d reconstruction using
on-the-fly surface reintegration.
ACM Transactions on Graphics (ToG), 36(4):1, 2017.

[11]
Angela Dai, Daniel Ritchie, Martin Bokeloh, Scott Reed, Jürgen Sturm, and
Matthias Nießner.
Scancomplete: Large-scale scene completion and semantic segmentation
for 3d scans.
In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 4578–4587, 2018.

[12]
Angela Dai, Yawar Siddiqui, Justus Thies, Julien Valentin, and Matthias
Spsg: Self-supervised photometric scene generation from rgb-d scans.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 1747–1756, 2021.

[13]
Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser.
Local deep implicit functions for 3d shape.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), June 2020.

[14]
Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T. Freeman,
and Thomas Funkhouser.
Learning shape templates with structured implicit functions.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision (ICCV), October 2019.

[15]
Benjamin Graham and Laurens van der Maaten.
Submanifold sparse convolutional networks.
arXiv preprint arXiv:1706.01307, 2017.

[16]
A. Handa, T. Whelan, J.B. McDonald, and A.J. Davison.
A benchmark for RGB-D visual odometry, 3D reconstruction and
In IEEE Intl. Conf. on Robotics and Automation, ICRA, Hong
Kong, China, May 2014.

[17]
John C Hart.
Sphere tracing: A geometric method for the antialiased ray tracing of
implicit surfaces.
The Visual Computer, 12(10):527–545, 1996.

[18]
Jiahui Huang, Shi-Sheng Huang, Haoxuan Song, and Shi-Min Hu.
Di-fusion: Online implicit 3d reconstruction with deep priors.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 8932–8941, 2021.

[19]
Chiyu "Max" Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias
Niessner, and Thomas Funkhouser.
Local implicit grid representations for 3d scenes.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), June 2020.

[20]
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada.
Neural 3d mesh renderer.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 3907–3916, 2018.

[21]
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt.
Neural sparse voxel fields.
NeurIPS, 2020.

[22]
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li.
Soft rasterizer: A differentiable renderer for image-based 3d
In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pages 7708–7717, 2019.

[23]
Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, and Zhaopeng
Dist: Rendering deep implicit signed distance function with
differentiable sphere tracing.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 2019–2028, 2020.

[24]
William E Lorensen and Harvey E Cline.
Marching cubes: A high resolution 3d surface construction algorithm.
ACM siggraph computer graphics, 21(4):163–169, 1987.

[25]
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and
Andreas Geiger.
Occupancy networks: Learning 3d reconstruction in function space.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 4460–4470, 2019.

[26]
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi
Ramamoorthi, and Ren Ng.
Nerf: Representing scenes as neural radiance fields for view
In European conference on computer vision, pages 405–421.
Springer, 2020.

[27]
Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim,
Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew
Kinectfusion: Real-time dense surface mapping and tracking.
pages 127–136, 2011.

[28]
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger.
Differentiable volumetric rendering: Learning implicit 3d
representations without 3d supervision.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 3504–3515, 2020.

[29]
Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas
Texture fields: Learning texture representations in function space.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pages 4531–4540, 2019.

[30]
Helen Oleynikova, Zachary Taylor, Marius Fehr, Roland Siegwart, and Juan Nieto.
Voxblox: Incremental 3d euclidean signed distance fields for on-board
mav planning.
In 2017 IEEE/RSJ International Conference on Intelligent Robots
and Systems (IROS), pages 1366–1373. IEEE, 2017.

[31]
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven
Deepsdf: Learning continuous signed distance functions for shape
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 165–174, 2019.

[32]
Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas
Convolutional occupancy networks.
In Computer Vision–ECCV 2020: 16th European Conference,
Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16, pages 523–540.
Springer, 2020.

[33]
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.
Pointnet: Deep learning on point sets for 3d classification and
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 652–660, 2017.

[34]
Radu Alexandru Rosu and Sven Behnke.
Neuralmvs: Bridging multi-view stereo and novel view synthesis.
arXiv preprint arXiv:2108.03880, 2021.

[35]
Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell,
and Gordon Wetzstein.
Implicit neural representations with periodic activation functions.
In Proc. NeurIPS, 2020.

[36]
Shuran Song, Andy Zeng, Angel X Chang, Manolis Savva, Silvio Savarese, and
Thomas Funkhouser.
Im2pano3d: Extrapolating 360 structure and semantics beyond the field
of view.
In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 3847–3856, 2018.

[37]
Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew J. Davison.
imap: Implicit mapping and positioning in real-time.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision (ICCV), pages 6229–6238, October 2021.

[38]
Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew J Davison.
imap: Implicit mapping and positioning in real-time.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pages 6229–6238, 2021.

[39]
Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek
Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler.
Neural geometric level of detail: Real-time rendering with implicit
3d shapes.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 11358–11367, 2021.

[40]
Peng-Shuai Wang, Yang Liu, and Xin Tong.
Deep octree-based cnns with output-guided skip connections for 3d
shape and scene completion.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition Workshops, pages 266–267, 2020.

[41]
Silvan Weder, Johannes Schonberger, Marc Pollefeys, and Martin R Oswald.
Routedfusion: Learning real-time depth map fusion.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 4887–4897, 2020.

[42]
Silvan Weder, Johannes L Schonberger, Marc Pollefeys, and Martin R Oswald.
Neuralfusion: Online depth fusion in latent space.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 3162–3172, 2021.

[43]
Thomas Whelan, Stefan Leutenegger, R Salas-Moreno, Ben Glocker, and Andrew
Elasticfusion: Dense slam without a pose graph.

[44]
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen,
and Yaron Lipman.
Multiview neural surface reconstruction by disentangling geometry and
Advances in Neural Information Processing Systems, 33, 2020.

[45]
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa.
Plenoctrees for real-time rendering of neural radiance fields.
arXiv preprint arXiv:2103.14024, 2021.

§ NETWORK ARCHITECTURE

For our point encoder $\phi_\mathrm{E}$, we use a shared MLP model, which contains 4 layers including the input and output layers. The output feature size is set to 32.

For our sparse U-Net $\phi_\mathrm{U}$, we illustrate it in <ref>. Convolution parameters are given in the format of (, , , , ), where the stride and padding are set to 1 and 0 respectively as the default values. All convolutional layers and fully-connected layers except for the output layer are followed by instance normalization and LeakyRelu layers.

For our SDF decoder $\phi_\mathrm{D}$, we use a small network which only contains 3 linear layers including the input and output layers and the channel sizes of the hidden layers are 64. Unlike other SDF decoders, the input latent vectors are not concatenated with the intermediate output of the network.

§ DIFFERENTIABLE RENDERER

§.§ Derivation

In this section, we detail the procedure of implicit differentiation to obtain <ref> of the main paper. We denote the ray origin as $\rayc$, ray direction as $\rayd$, and the rendered depth as $t$, the hit point can be expressed as $\p = \rayc + t\rayd$.
Compute the total derivative of $f(\p,\theta)=0$ and we get:

\begin{equation}
\label{eq:total}
\frac{\partial{f}}{\partial{\p}}\frac{\partial\p}{\partial\rayc}\mathrm{d}\rayc + \frac{\partial{f}}{\partial{\p}}\frac{\partial\p}{\partial\rayd}\mathrm{d}\rayd + \frac{\partial{f}}{\partial{\p}}\frac{\partial\p}{\partial{t}}\mathrm{d}t + \frac{\partial{f}}{\partial\theta}\mathrm{d}\theta = 0,
\end{equation}

and according to $\p = \rayc + t\rayd$, we replace  ${\partial\p}/{\partial\rayc}$, ${\partial\p}/{\partial\rayd}$ and  ${\partial\p}/{\partial{t}}$ with $1$, $t$ and $\rayd$ respectively:

\begin{equation}
\label{eq:total_2}
\frac{\partial{f}}{\partial{\p}}\mathrm{d}\rayc + \frac{\partial{f}}{\partial{\p}}t\mathrm{d}\rayd + \langle\frac{\partial{f}}{\partial{\p}},\rayd\rangle\mathrm{d}t + \frac{\partial{f}}{\partial\theta}\mathrm{d}\theta = 0.
\end{equation}

To compute ${\partial{t}}/{\partial\theta}$, we ignore $\mathrm{d}\rayd$ and $\mathrm{d}\rayc$:

\begin{equation}
\langle\frac{\partial{f}}{\partial{\p}},\rayd\rangle\mathrm{d}t + \frac{\partial{f}}{\partial\theta}\mathrm{d}\theta = 0 \Rightarrow   \frac{\partial{t}}{\partial\theta} = -\langle\frac{\partial{f}}{\partial{\p}},\rayd\rangle^{-1}\frac{\partial{f}}{\partial\theta}.
\end{equation}

Similarly, we can compute the partial derivatives for $\rayc$ and $\rayd$:

\begin{align}
\frac{\partial{t}}{\partial\rayc} = -\langle\frac{\partial{f}}{\partial{\p}},\rayd\rangle^{-1}\frac{\partial{f}}{\partial\p},\quad
\frac{\partial{t}}{\partial\rayd} = -t\langle\frac{\partial{f}}{\partial{\p}},\rayd\rangle^{-1}\frac{\partial{f}}{\partial\p}.
\end{align}

In our implementation, to satisfy the above partial derivatives, we construct the forward equation as:

\begin{equation}
t = t_0 + \frac{f(\p_0,\theta_0) - f(\p,\theta)}{\langle{\partial{f}}/{\partial{\p}}|_{\p = \p_0},\rayd_0\rangle},
\end{equation}

where $f(\p_0,\theta_0)$ means the SDF value provided by $\decnet$, and all the variables with subscript 0 are the constant values evaluated at the hit point.

§.§ More Results

Speed comparison of two renderers. We show that `Ours' is faster than `Ours-AD'. With the increasing number of rendered rays, rendering time grows slowly, implying that the performance bottleneck lies in the loop of the sphere tracing that is hard to be parallelized.

Detailed architecture of $\unet$. Notice that all Fully-Connected (FC) layers receive sparse features as input and no dense tensor is built throughout the graph, significantly reducing the memory consumption.

We further demonstrate an alternative renderer using auto  differentiation provided by the deep learning framework, Ours-AD, and show our differentiable renderer is faster and more accurate than Ours-AD in <ref> and <ref>.
For Ours-AD, rendered depth is given by:

\begin{equation}
t = c + \sum^N_{i = 0}{f(\p_i,\theta)},
\end{equation}

where $c$ is the depth of ray-voxel intersect point and $\p_i$ is the $i$th point in sphere tracking procedure.

When rendering 300,000 rays, `Ours' takes about 1.3G GPU memory while `Ours-AD' takes about 4.6G. It is because `Ours-AD' stores all of the points $\p_i$ in the compute graph while `Ours' only stores the hit point.

Qualitative results of differentiable renderer. Our approach optimize geometry and poses jointly and generate fine-detailed mesh. Using implicit differentiation, our renderer provides more accurate gradient for poses than Ours-AD.

Results on large scenes of Matterport3D. We show two building-scale reconstructions from our method, with a single feed-forward pass. The sizes of buildings and the inference time are given on the right side of the figure and the subfigures in the bordered boxes show each floor.

§ RECONSTRUCTION OF LARGE SCENES

As illustrated in <ref>, our method has the ability to reconstruct large scenes using a single feed-forward pass with a small run-time memory usage thanks to the sparse structure.