# StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street
Views

Jianfei Guo1 Nianchen Deng1 Xinyang Li1 Yeqi Bai1 Botian Shi1 Chiyu Wang2
Chenjing Ding2 Dongliang Wang1,2 Yikang Li1
1Shanghai Artificial Intelligence Laboratory 2Sensetime Research
https://ventusff.github.io/streetsurf_web Corresponding author

###### Abstract

We present a novel multi-view implicit surface reconstruction technique,
termed StreetSurf, that is readily applicable to street view images in widely-
used autonomous driving datasets, such as Waymo-perception sequences, without
necessarily requiring LiDAR data. As neural rendering research expands
rapidly, its integration into street views has started to draw interests.
Existing approaches on street views either mainly focus on novel view
synthesis with little exploration of the scene geometry, or rely heavily on
dense LiDAR data when investigating reconstruction. Neither of them
investigates multi-view implicit surface reconstruction, especially under
settings without LiDAR data. Our method extends prior object-centric neural
surface reconstruction techniques to address the unique challenges posed by
the unbounded street views that are captured with non-object-centric, long and
narrow camera trajectories. We delimit the unbounded space into three parts,
close-range, distant-view and sky, with aligned cuboid boundaries, and adapt
cuboid/hyper-cuboid hash-grids along with road-surface initialization scheme
for finer and disentangled representation. To further address the geometric
errors arising from textureless regions and insufficient viewing angles, we
adopt geometric priors that are estimated using general purpose monocular
models. Coupled with our implementation of efficient and fine-grained multi-
stage ray marching strategy, we achieve state of the art reconstruction
quality in both geometry and appearance within only one to two hours of
training time with a single RTX3090 GPU for each street view sequence.
Furthermore, we demonstrate that the reconstructed implicit surfaces have rich
potential for various downstream tasks, including ray tracing and LiDAR
simulation.

##  1 Introduction

Neural rendering has seen rapid growth in recent years since the emergence of
the NeRF (Neural Radiance Field) [30]. With a collection of posed images as
inputs or supervisions, NeRF is capable of providing high-fidelity novel view
synthesis results with simple yet effective assumptions and structures, which
also makes it a flexible technique that sparks a flourishing of following
research. Recent research [35, 49, 40, 51, 60, 63, 20] starts to apply NeRFs
to street views or large-scale urban scenes. Among them, there are some works
[35, 17, 12, 63] that focus on the relatively smaller-scale scenes readily
available in autonomous driving datasets, such as KITTI [13], Waymo Open
Dataset [47], or NuScenes [3]. Reconstruction and rendering in these datasets
are helpful for various downstream tasks including data augmentation,
3D/multi-modal perception and AR/VR. However, existing approaches on street
views either mainly focus on novel view synthesis [35, 55, 63, 20] with little
exploration of the scene geometry, or rely heavily on dense LiDAR data when
investigating reconstruction [40, 60]. Neither of them investigates multi-view
implicit surface reconstruction, especially under settings without LiDAR data.

3D reconstruction from street views are much more challenging compared to
previous works under bounded or object-centric multi-view settings [54, 65,
34], since scenes in street views are typically unbounded and captured with
non-object-centric, long and narrow camera trajectories.

(a) NeRF-360: Unbounded scene with object-centric camera trajectories. Using
spherical bounds and spherical/cubic model.

(b) Street views: Unbounded scene captured with non-object-centric, long and
narrow camera trajectories. We propose to use aligned cuboid bounds and cuboid
model.

Figure 1: The challenges of street views

To deal with unbounded scenes, previous research [54, 69, 1, 2] leverage
inverse-spherical warping for object-centric camera settings, as shown in Fig.
1(a). NeuS [54] and other concurrent works [34, 65] introduce this idea to
multi-view implicit surface reconstruction, in which they represent the close-
range scene with SDFs (Signed Distance Functions) and the distant-view with
NeRF++ [69]. However, when it comes to non-object-centric settings like street
views, inverse-spherical warping becomes wasteful of capacity and thus no
longer suitable. This is evidenced by the very recent works [55, 29], who
subsequently introduce perspective warping or progressive warping that warp
the full space according to different camera trajectories to better allocate
network capacity.

However, we have found that these full-space warping strategies are not well-
suited for geometry represented by SDFs as they break the definition of
distance fields and sphere tracing. In addition, although effective for the
novel view synthesis task, modeling the entire space with one single model
does not match the different needs of different viewing spaces when it comes
to the multi-view reconstruction task. Specifically, the close-range space
requires complete and accurate surface geometry, whereas the distant-view and
sky only need their appearance to be properly fit and should not interfere
with the geometry of the close-range model. For this purpose, we propose to
delimit the close-range and distant-view spaces with long and narrow cuboid
boundaries that are aligned with the camera trajectories, as shown in Fig.
1(b). Then we assign a cuboid NeuS model and a hyper-cuboid NeRF++ model to
the close-range and distant-view spaces, respectively. If an optional set of
sky masks are provided, we further distinguish the sky part out of the
distant-view by using a directional MLP.

Following the above discussions, we are led to the disentanglement of close-
range and distant-view models, which is an unsupervised and ill-posed problem
with almost no constraints. Its impact is negligible in setups with object-
centric camera trajectories [54, 69]. However, in our non-object-centric
multi-view setup, lack of sufficiently varying view angles often leads to
ambiguous solutions as two models are in competition with each other. To
address this issue, we first examine different initialization schemes of the
SDF field in our close-range model. We have found that enclosed initial shapes
like inside-out spheres [16, 67] or ellipsoids/capsules can hinder the
disentanglement since geometries in street-view scenes are usually open and
not enclosed. Therefore, we propose a novel road-surface initialization scheme
that pre-train the SDF to be roughly aligned with the road surface. We also
apply an entropy regularization to further encourage crisp disentanglement.

Another challenge posed by the non-object-centric views is the geometric
errors arising from the insufficient coverage of viewing angles. It is even
worse regarding street-view images in autonomous driving datasets, since each
data sequence in these datasets only cover one single street with no repeated
observations. Fig. 3(b) shows that while the state of the art novel view
synthesis method ($\text{F}^{2}$-NeRF [55]) already aims at non-object-centric
camera trajectories, they still struggle at the geometry of textureless
objects and surfaces, which are often mistakenly interpreted as large cavity
or holes. To tackle this challenge, we follow MonoSDF [67] and leverage
monocular estimations to guide the volume-rendered normals and depths of the
close-range NeuS model.

Apart from disentanglement and geometric priors, we also implement a novel ray
marching strategy to achieve efficient and fine-grained ray sampling in the
long and narrow street-view spaces with complex occlusions. We combine the
previous idea of occupancy-grid-based acceleration [32] with multi-stage
hierarchical sampling [30, 54], and implement the atomic pack-like operations
under CUDA, which ensures granularity without sacrificing efficiency.

In summary, we make the following contributions:

  * •

We present StreetSurf, a novel multi-view implicit surface reconstruction
framework that achieves state of the art geometry and appearance quality in
street views within an hour or two with a single RTX3090 GPU, without
necessarily requiring LiDAR data.

  * •

We propose a close-range-vs-distant-view disentanglement approach suitable for
unbounded scenes captured with non-object-centric, long and narrow camera
trajectories.

  * •

We conduct experiments in challenging real-world autonomous driving datasets
such as Waymo Open Dataset [47] and provide a benchmark for the reconstructed
geometry and appearance.

Figure 2: Qualitative results of reconstruction without LiDAR data. We apply
our method to Waymo Open Dataset [47] seg1534950…with a trajectory length of
112.1 (m).

Images |  |  |
---|---|---|---
Depths |  |  |
Reconstructed surfaces |  |  |
| Waymo Open Dataset [47], seg1006130…
Images |  |  |
Depths |  |  |
Reconstructed surfaces |  |  |
| Waymo Open Dataset [47], seg1347637…
Images |  |  |
Depths |  |  |
Reconstructed surfaces |  |  |
| Waymo Open Dataset [47], seg1527063…
|  |  |

(a) NGP [32], with LiDAR [40]

(b) $\text{F}^{2}$-NeRF [55], without LiDAR

(c) Ours, without LiDAR

Figure 3:  Qualitative comparison of our work with recent studies, including
one that requires LiDAR data input and another that does not. For NGP [32] and
our work, we apply Marching Cubes [26] to extract meshes. For
$\text{F}^{2}$-NeRF [55], we leverage vdbfusion [52] to the rendered camera
depth since we are temporarily unable to extract mesh using their recently
released code-base.

##  2 Related work

NeRF on street views. Research on integrating NeRF [30] to street views or
urban scenes has recently started to gain attention. Some addresses the
challenges of large-scale outdoor scenes by using local blocks [49, 50, 51] or
other scalable representations [20, 40]. Others [35, 63, 17, 12] instead,
propose to handle relatively smaller street-view scenes available in widely-
used autonomous driving datasets, such as KITTI [13], Waymo [47], or NuScenes
[3]. In this paper, we also focus on these autonomous driving scenarios since
we expect our reconstruction results to be directly useful for perception
methods trained on these datasets. Many research in this field studies
compositional scene representations, either under multi-object settings [35,
63] or static-dynamic settings [51, 28, 46]. Without explicit multi-object
decomposition, [17, 12] model semantic fields using panoptic segmentation.
Another line of research [49, 20, 55, 40, 60] focuses on static street
backgrounds and ignores dynamic foreground objects. Most of them aim only at
the task of novel view synthesis. While UrbanNeRF [40] and FEGR [60] are
capable of reconstructing decent surfaces, they rely on dense LiDAR inputs and
they do not handle the distant-view that can interfere with the close-range
reconstruction. In this paper, we focus on the task of multi-view implicit
surface reconstruction for static close-range street-view backgrounds without
necessarily requiring LiDAR data.

Spatial scene representations. Neural fields [62] or implicit scene
representations are experiencing a surge of interest. [9, 22, 30, 54, 38, 44]
are based on different modification of multi layer perceptrons (MLPs). Another
line of research establish their scene representation on spatial grids or
voxels, known as local implicits or hybrid representations. [43, 45] and
earlier works use dense feature grids. To improve spatial efficiency, voxel-
pruning [24, 48, 66, 10, 18], hash-indexing [32], tensor-decomposition [5, 4],
or multi-scale voxels [27] are introduced. Instead of cubic voxels, other
works utilize special forms of voxels like tetrahedra [33] or permutohedra
[41]. In this paper, we establish 3D/4D cuboid/hyper-cuboid hash-grids for the
close-range/distant-view spaces respectively. To due with unbounded scenes,
two spatial warping functions are commonly used, namely inverse-spherical
warping [1, 2, 69] and NDC warping [30]. $\text{F}^{2}$-NeRF [55] and [29]
that warps the full space according to certain rules to further reduce
capacity waste. In this paper, instead of full-space warping, we explicitly
delimit the close-range and distant-view spaces with a long and narrow cuboid
boundary that is aligned with the camera trajectory.

Multi-view implicit surface reconstruction. 3D Reconstruction from multi-view
images has been a long-time aim in the field of computer vision. Recently,
instead of modeling the scene with explicit shape representation as
traditional multi-view stereo pipelines like COLMAP [42] did, using neural
representations for implicit surfaces [64, 54, 65, 34] has begun to show
extraordinary results using object-centric views. NeuS [54] or VolSDF [65]
models the underlying 3D geometry with SDF fields and utilizes volume
rendering to produce differentiable colors. Follow-up works replace the MLP
network with local implicit grids [56, 41], sparse voxels [18], MLP blocks [8]
or displacement fields [58] for better efficiency or local details. Multi-view
3D reconstruction is typically an under-constrained problem, driving
researchers to introduce different priors or regularizations, such as multi-
view stereo prior [11], geometric priors [67, 16, 31, 57], shadow information
[23] and sparse regularization [68]. After reconstruction, further research
[37, 6] conduct study on decouple and dealing with ambient light or specular
effect. Besides single objects, [16, 53, 67, 57, 21] have been applied to
indoor datasets. Meanwhile, [46, 68] extends to outdoor datasets. However,
they are still using data captured with object-centric views. In this paper,
we extend this line of research to handle the unique challenges in street
views.

##  3 Method

Figure 4: StreetSurf overview. By using a collection of posed images, along
with an optional set of LiDAR data or monocular cues as inputs (or
supervisions), we reconstruct the street-view scene through an optimization
process. Each scene is divided into three parts according to the viewing
distance, namely close-range (cr), distant-view (dv) and sky.

###  3.1 Multi-shell networks

To tackle the challenge of unbounded spaces and significantly varying viewing
distance in street-views, let us first consider an observer standing on a
street, whose view can be divided into three parts according to the distance
of objects. Buildings and roads can be located within several to tens of
meters, while landscapes and objects that extend beyond the end of the
trajectory may be hundreds or thousands of meters away, and the sky seemingly
infinitely far off. In the following, we refer to these three parts as the
close-range (cr), distant-view (dv), and sky parts, respectively.

We employ a cuboid NeuS model (modified from [54]), a hyper-cuboid NeRF++
model (modified from [69]), and a directional MLP for the close-range scene,
distant-view, and sky, respectively.

For each ray $\mathbf{r}$, we query all three models and jointly render a
differentiable pixel color $\widehat{C}^{(\text{all})}(\mathbf{r})$.

Specifically, we first merge queried samples of close-range and distant-view
models on the ray $\mathbf{r}$ from near to far,
$\\{t^{(\text{cr,dv})}_{i}\\}_{i=1}^{k}=\\{t^{(\text{cr})}_{1},t^{(\text{cr})}_{2},\dots,t^{(\text{cr})}_{\textrm{n}_{\text{cr}}},t^{(\text{dv})}_{1},t^{(\text{dv})}_{2},\dots,t^{(\text{dv})}_{\mathrm{n}_{\text{dv}}}\\}$,
where $t^{\text{(cr)}}$ stands for depth samples on of the close-range model
and $t^{\text{(dv)}}$ for the distant-view model.

Then, we perform volume rendering on the combined samples of depth
$t^{(\text{cr,dv})}$ and color $c^{(\text{cr,dv})}$ (we will omit the
superscript of $(\text{cr,dv})$ on
${\\{T_{i},\alpha_{i},t_{i},c_{i}\\}}^{(\text{cr,dv})}$ for simplicity, unless
otherwise specified),

$\widehat{O}^{(\text{cr,dv})}(\mathbf{r})=\sum_{i=1}^{k}T_{i}\alpha_{i}\qquad\widehat{D}^{(\text{cr,dv})}(\mathbf{r})=\sum_{i=1}^{k}T_{i}\alpha_{i}t_{i}\qquad\widehat{C}^{(\text{cr,dv})}(\mathbf{r})=\sum_{i=1}^{k}T_{i}\alpha_{i}c_{i}$
(1)

where $T_{i}=\prod_{j=1}^{i-1}(1-\alpha_{j})$ is the accumulated transmittance
and $\alpha_{i}$ is calculated the same way as NeuS [54] for the close-range
model.
$\widehat{O}^{(\text{cr,dv})},\widehat{C}^{(\text{cr,dv})},\widehat{D}^{(\text{cr,dv})}$
are the predicted opacity, color and depth, respectively, volume-rendered by
the close-range and distant-view models jointly.

Finally, as shown in Equ. 2, we use $\widehat{c}^{(\text{sky})}(\mathbf{v})$,
the colors queried from the sky model on ray direction vectors $\mathbf{v}$,
as the radiance of the last and infinitely long ray marching interval, whose
$\alpha$-value equals to $(1-\widehat{O}^{(\text{cr,dv})}(\mathbf{r}))$. In
addition, the sky model is useful only if an optional set sky masks is given.
Otherwise, the distant-view model can inclusively represent the sky and in
this case $\widehat{C}^{(\mathrm{all})}=\widehat{C}^{(\mathrm{cr,dv})}$.

$\widehat{C}^{(\text{all})}(\mathbf{r})=\begin{cases}\widehat{C}^{(\text{cr,dv})}(\mathbf{r})+\left(1-\widehat{O}^{(\text{cr,dv})}(\mathbf{r})\right)\widehat{c}^{(\text{sky})}(\mathbf{v})&\text{w/
sky}\\\ \widehat{C}^{(\text{cr,dv})}(\mathbf{r})&\text{w/o sky}\end{cases}$
(2)

###  3.2 Cuboid space, cuboid hash-grids and hyper-cuboid distant-view models

Figure 5:  Ray sampling pattern for the close-range, distant-view and sky
models. We delimit the close-range space with long and narrow cuboid shapes
using the camera trajectories. As shown by the gray points
$\mathbf{x}^{(\text{dv})}$, the samples of our distant-view model lies on
cuboid shells (perturbed in between when training). The cuboid shells are
scaled proportionally from the AABB (Axis-Aligned Bounding Box) of the close-
range space with inverse-proportionally increasing scales $\\{r_{i}\mid
r_{i}=\frac{1}{(1-i/\textrm{n}_{\text{dv}})\cdot
1+(i/\textrm{n}_{\text{dv}})\cdot(1/r_{\max})},i\in[0,\textrm{n}_{\text{dv}}]\\}$,
where $r_{\max}=1000$ typically.

Since most streets occupy long and narrow space with relatively low heights,
defining cubic or spherical close-range boundaries along with cubic hash-grids
may lead to significant capacity waste, especially for the distant-view NeRF++
model [55]. Therefore, we propose to delimit the close-range scene with long
and narrow cuboid-shaped space that is aligned with the camera trajectories,
and introduce cuboid-shell sampling and inverse cuboid warping for the
distant-view model. We adapt multi-resolution 3D cuboid hash-grids for the
close-range model and multi-resolution 4D hyper-cuboid hash-grids for the
distant-view model.

Cuboid space. The cuboid close-range volume is delimited automatically
according to the motion trajectory of ego car, without the need of manual
specification. To fully exploits network capacity, we first define our
coordinate system that aligns its orientation about the vertical axis with
that of the close-range scene. This orientation is estimated by the average of
ego motion, which roughly indicates the orientation of the street. Then, we
extend all camera frustums in each frame by a certain length, typically tens
of meters, and define the AABB (Axis-Aligned Bounding Box) of the close-range
scene with the minimum and maximum points of the vertex set that contains all
the vertices of the extended frustums in the aligned coordinate system.

Cuboid close-range hash-grid. Within the cuboid boundary, we assign a NeuS
[54] model to represent the scene’s geometry and appearance. For its SDF field
$\widehat{\mathcal{S}}(\mathbf{x})$, we adapt a 3D hash-grid with cuboid
resolution that is proportional to the size of the AABB. This ensures that
grid voxels are equally stretched in all three dimensions, which is more
appropriate for the SDF representation and its gradients (i.e. normals), and
can achieve finer granularity with the same amount of parameters compared to
cubic ones. In addition, despite the presence of diagonal angles or curves in
some streets, the hash-grid representation remains adaptive [32] to this and
incurs minimal representational waste.

Hyper-cuboid distant-view model. Different from NeRF++ [69] that samples on
spherical shells for ray marching and apply inverse spherical warping to form
4D inputs, we sample on cuboid shells, as shown in Fig. 5, and apply inverse
cuboid warping. Then, on top of the inversely warped 4D input space, we
establish a 4D hyper-cuboid hash-grid with the first three elements of its
resolution to be proportional to the size of the AABB of the close-range
scene.

###  3.3 Optimization

####  3.3.1 Disentanglement of close-range, distant-view and sky

Road-surface initialization. The disentanglement of close-range and distant-
view models is an unsupervised and ill-posed problem with almost no
constraints. To address this issue, we first examine different initialization
schemes of the SDF field in our close-range NeuS model.

(a) Capsule initialization

(b) Road-surface initialization

Figure 6:  Different close-range SDF initialization schemes, taking Waymo-
perception sequence seg1434813… for example. We mark the zero-level set with
*. The vertical scale has been 5$\times$ exaggerated for better illustration.

Previous research [54, 64] has shown that in order to achieve a proper SDF
representation, it is essential to properly initialize the geometry network so
that its zero-level set forms a sphere shape. While this initialization is
designed for MLPs and not directly applicable to SDFs represented by local
implicits such as hash-tables, we can simply pre-train the SDF network with
manually constructed pseudo ground truths following the same intuition.
Furthermore, for long and narrow camera trajectories in street views, we can
construct pseudo SDF ground truths that are shaped as inside-out capsules or
bent ellipsoids, which encases the ego motion tracks inside, as shown in Fig.
6(a).

However, we have found that initialization schemes that form enclosed
isosurfaces are not well-suited for street views, since geometries in these
scenarios are usually expected to be open and not enclosed. Subsequently,
enclosed initial shapes can hinder the disentanglement of close-range and
distant-view models, or often leads to failures in the convergence of sky
mask.

Therefore, as shown in Fig. 6(b), we propose a novel road-surface
initialization scheme for street view reconstruction, in which the close-range
SDF model is pre-trained to have a zero-level set that is roughly aligned with
the road surface. This surface is automatically estimated by first
horizontally extending ego motion tracks and then lowering it by an amount
roughly equivalent to ego height.

Entropy regularization loss. To further eliminate ambiguous occupancy of the
close-range and distant-view models, we apply an entropy regularization on
$\widehat{O}^{(\text{cr})}$, the predicted occupancy of the close-range model,
that encourages opaque renderings and penalizes translucent ones:

$\mathcal{L}_{\text{entropy}}=f_{\text{entropy}}\left(\widehat{O}^{(\text{cr})}(\mathbf{r})\right),\text{where}\>f_{\text{entropy}}(x)=-\left(x\ln{x}+(1-x)\ln{(1-x)}\right)$
(3)

Optional sky model and sky mask loss. Given sky masks $M^{(\text{sky})}$, we
explicit distinguish the sky model out of distant-view. $M^{(\text{sky})}$ is
used to supervise the rendered occupancy $\widehat{O}^{(\text{cr,dv})}$ with
$\mathcal{L}_{\text{mask}}=\text{BCE}(\widehat{O}^{(\text{cr,dv})},1-M^{(\text{sky})})$,
where BCE stands for the binary cross entropy loss. We can efficiently use
SegFormer [61] to extract sky masks if sky annotations are not provided. In
addition, thanks to our unsupervised disentangled design, we also support not
requiring sky masks and sky model. See supplementary for more details.

####  3.3.2 All supervisions

Photometric loss. We use the $L_{1}$ photometric loss to minimize the
differences between the rendered pixel color $\widehat{C}^{(\text{all})}$ and
the ground truth color $C$. In our observation, $L_{1}$ loss results in less
noisy reconstruction and is more beneficial for pose refinement while
sacrificing a little performance in appearance, compared to the other two
commonly used losses, namely MSE and Huber losses.

$\mathcal{L}_{\text{photometric}}=\lVert\widehat{C}^{(\text{all})}-C\rVert_{1}$
(4)

Geometry loss. To address the geometric errors arising from textureless
regions and insufficient viewing angles, we develop two strategies, one for
when LiDAR is available and the other for not.

If LiDAR data is available, we use the range measurements as ground truth
sparse depths. On each sampled LiDAR beam $\mathbf{r}_{\text{lidar}}$, we
apply a logarithm $L_{1}$ loss on $\widehat{D}^{(\text{cr,dv})}$, the rendered
depth of the combined close-range and distant-view model, as shown in Equ 5.

If LiDAR data is not provided, we can take inspiration from MonoSDF [67] and
use Omnidata [7] to estimate the monocular normals $\bar{N}$ and depths
$\bar{D}$ for each camera image. $\bar{N}$ and $\bar{D}$ are then used as a
guide for the rendered normals $\widehat{N}^{(\text{cr})}$ and depths
$\widehat{D}^{(\text{cr,dv})}$, as shown in Equ. 5, where
$\mathcal{L}_{\text{mono\\_normal}}$ and $\mathcal{L}_{\text{mono\\_depth}}$
is defined the same way as MonoSDF [67].

$\mathcal{L}_{\text{geometry}}=\begin{cases}\ln(|\widehat{D}^{(\text{cr,dv})}(\mathbf{r}_{\text{lidar}})-D(\mathbf{r}_{\text{lidar}})|+1)&\text{if
LiDAR data is provided}\\\
\mathcal{L}_{\text{mono\\_normal}}+\lambda\mathcal{L}_{\text{mono\\_depth}}&\text{if
no LiDAR}\end{cases}$ (5)

Optional sky mask loss. As aforementioned in section 3.3.1, if an optional set
of sky mask annotations or segmentation is provided, we apply a BCE mask loss
$\mathcal{L}_{\text{mask}}$.

Entropy regularization loss. As aforementioned in section 3.3.1, we adopt an
entropy regularization $\mathcal{L}_{\text{entropy}}$ in Equ. 3 to encourage
crisp disentanglement of the close-range and distant-view models.

Eikonal regularization loss. Following [15, 54], we add an Eikonal term on the
sampled points to regularize the close-range SDF network by
$\mathcal{L}_{\text{eikonal}}=(\lVert\nabla\widehat{\mathcal{S}}(\mathbf{x})\rVert_{2}-1)^{2}$.

Sparsity regularization loss. Under the settings of non-object-centric camera
trajectories in street views, a large proportion of pre-allocated space are
always occluded in any of the input views. To penalize free geometry in these
unobserved regions, we take inspiration from [66, 68] and penalizes the
density values that are mapped from the uniformly sampled close-range SDF
values $\widehat{\mathcal{S}}(\mathbf{x})$ using the normalized logistic
density function $\text{NLD}(x)$, with a constant scaling parameter
$s_{\text{reg}}$.

$\mathcal{L}_{\text{sparisty}}=\sum_{\mathbf{x}}\text{NLD}(\widehat{\mathcal{S}}(\mathbf{x})),\text{where}\>\text{NLD}(x)=\frac{1}{\cosh^{2}(\frac{x}{2s_{\text{reg}}})}$
(6)

To summarize:

$\mathcal{L}_{\text{all}}=\mathcal{L}_{\text{photometric}}+\lambda_{\text{1}}\mathcal{L}_{\text{geometry}}+\lambda_{\text{2}}\mathcal{L}_{\text{mask}}+\lambda_{\text{3}}\mathcal{L}_{\text{eikonal}}+\lambda_{\text{4}}\mathcal{L}_{\text{sparisty}}+\lambda_{\text{5}}\mathcal{L}_{\text{entropy}}$
(7)

##  4 Experiments

###  4.1 Implementation details

(a) NGP [32] samples in occupied area.

(b) NeuS [54] samples points along the entire ray from the front to the back,
and then conduct multi-stage hierarchical sampling.

(c) We propose a sampling strategy that begins with sampling in the occupied
area, followed by multi-stage hierarchical sampling, and concludes with the
reduction of meaningless samples.

Figure 7: Schematic illustration of different ray-marching approach.

Ray marching with multi-stage hierarchical sampling on occupancy grids. Street
views occupy long and narrow spaces with complex occlusions, posing challenges
to ray marching, especially if we need to balance with sampling efficiency. As
shown in Fig. 7(a), NGP [32] and others propose to boost ray marching
efficiency with occupancy information, but its sampling granularity is not
enough for long and narrow spaces due to no up-sampling. Shown in Fig. 7(b),
NeRF [30] utilizes hierarchical sampling and NeuS [54] introduces multi-stage
hierarchical sampling in their implementation that progressively increases up-
sampling granularity from one stage to the next, while sacrificing efficiency.

Standing on their shoulders, as illustrated in Fig. 7(c), we propose a novel
ray marching strategy that apply multi-stage hierarchical sampling to the
marched ray samples of the occupancy grid for our close-range model. This
occupancy information is periodically updated from the close-range SDF network
in a bootstrap manner that does not require reconstructing the scene in
advance with COLMAP [42] as [46] or feeding LiDAR pointclouds to pre-compute
occupied voxels as VoxSurf [18]. To further ensure efficiency, we reduce
meaningless samples for the close-range NeuS [54] model, similar to NGP [32]
and nerf_acc [19]. We implement a Pytorch [36] extension library under CUDA
named nr3d_lib for inference and gradient calculations of cuboid hash-grids
and pack-like operations involved in the whole rendering process. See
supplementary for more details.

Reconstruction in the wild. Data captured from street views is often
inaccurate and subject to complex and fluctuating illumination or other
environmental conditions. We follow the approach of [49, 63, 59] and use
learnable pose refinement deltas to correct sensor pose errors in an
unsupervised manner, which only requires photometric supervisions. Inspired by
[28, 46], we adopt per-frame image appearance embeddings in all of our close-
range, distant-view and sky models to address the differences and fluctuation
in exposure and color domains across various frames and cameras.

Table 1: Quantitative comparison on selected Waymo Open Dataset [47]
sequences. We employ PSNR to assess the reconstructed appearance and RMSE (See
Sec. 4.3) for geometry.

| With 1 dense + 4 sparse LiDARs | With 4 sparse LiDARs | Without LiDAR data
---|---|---|---
Sequence |  PSNR$\uparrow$ |  RMSE$\downarrow$ |  PSNR$\uparrow$ |  RMSE$\downarrow$ |  PSNR$\uparrow$ |  RMSE$\downarrow$
NeuS | NGP+L | Ours | NeuS | NGP+L | Ours | NGP+L | Ours | NGP+L | Ours |  $\text{F}^{2}$-NeRF | Ours | COLMAP |  $\text{F}^{2}$-NeRF | Ours
seg1006130… | 14.57 | 25.15 | 27.38 | 3.14 | 1.67 | 1.22 | 23.86 | 27.26 | 3.22 | 2.58 | 24.41 | 27.39 | 7.10 | 8.87 | 2.99
seg1027514… | 9.10 | 23.92 | 27.19 | 3.30 | 1.94 | 1.40 | 23.48 | 27.11 | 3.25 | 2.82 | 22.14 | 27.58 | 7.47 | 16.52 | 2.91
seg1067626… | 12.90 | 25.94 | 27.68 | 4.40 | 2.61 | 1.94 | 24.87 | 28.68 | 7.61 | 4.12 | 26.43 | 29.01 | 9.06 | 35.59 | 4.34
seg1137922… | 13.75 | 25.93 | 27.64 | 5.77 | 2.56 | 2.14 | 25.15 | 27.80 | 9.62 | 5.91 | 25.42 | 28.33 | 12.39 | 20.10 | 5.70
seg1172406… | 21.28 | 26.16 | 27.86 | 2.18 | 1.51 | 1.03 | 24.89 | 28.21 | 2.80 | 1.83 | 27.16 | 28.50 | 13.62 | 9.00 | 2.57
seg1287964… | 10.64 | 26.31 | 29.92 | 3.34 | 1.64 | 1.32 | 25.98 | 29.45 | 6.79 | 3.50 | 25.40 | 29.89 | 10.34 | 6.73 | 3.19
seg1308545… | 16.57 | 24.22 | 25.59 | 4.09 | 2.03 | 1.59 | 22.99 | 25.61 | 7.85 | 4.40 | 23.55 | 25.95 | 8.64 | 15.50 | 4.12
seg1314219… | 10.78 | 22.85 | 25.77 | 3.52 | 2.14 | 1.53 | 22.18 | 25.66 | 6.13 | 3.94 | 21.76 | 26.07 | 6.75 | 19.30 | 3.48
seg1319679… | 10.71 | 24.69 | 24.79 | 4.97 | 2.15 | 1.68 | 23.78 | 25.30 | 7.94 | 5.21 | 24.54 | 25.66 | 7.63 | 23.50 | 4.76
seg1323841… | 8.59 | 24.13 | 28.16 | 3.64 | 1.75 | 1.37 | 23.37 | 28.16 | 3.70 | 2.66 | 24.03 | 28.49 | 7.32 | 20.19 | 3.13
seg1347637… | 14.19 | 26.18 | 28.20 | 2.08 | 1.03 | 0.84 | 25.30 | 28.18 | 1.84 | 1.41 | 26.12 | 28.26 | 5.93 | 21.72 | 1.84
seg1400454… | 11.43 | 24.73 | 24.79 | 3.22 | 1.81 | 1.38 | 23.96 | 25.40 | 4.35 | 2.91 | 24.58 | 25.49 | 8.08 | 39.85 | 3.29
seg1434813… | 10.71 | 26.31 | 26.98 | 4.59 | 2.12 | 1.81 | 25.43 | 27.75 | 8.71 | 4.30 | 26.15 | 28.23 | 8.48 | 35.96 | 4.74
seg1442480… | 13.33 | 25.16 | 27.22 | 3.30 | 1.70 | 1.42 | 24.28 | 27.28 | 3.69 | 2.77 | 25.14 | 27.56 | 7.85 | 36.35 | 2.97
seg1486973… | 16.23 | 23.59 | 24.21 | 2.90 | 1.52 | 1.22 | 21.99 | 24.20 | 2.89 | 2.17 | 22.87 | 24.42 | 5.52 | 3.53 | 2.82
seg1506235… | 12.01 | 25.07 | 27.54 | 2.67 | 1.43 | 1.04 | 24.12 | 27.08 | 3.22 | 1.89 | 25.07 | 27.41 | 7.84 | 27.61 | 2.40
seg1522170… | 12.67 | 25.26 | 25.66 | 4.79 | 2.14 | 1.68 | 24.77 | 25.67 | 6.87 | 4.87 | 25.71 | 26.29 | 11.28 | 16.66 | 4.87
seg1527063… | 17.46 | 27.04 | 28.95 | 1.92 | 1.04 | 0.76 | 26.03 | 28.76 | 2.46 | 1.55 | 26.81 | 29.06 | 2.62 | 7.82 | 1.98
seg1534950… | 12.35 | 25.42 | 27.26 | 2.30 | 1.34 | 1.06 | 24.58 | 27.20 | 2.77 | 1.94 | 25.30 | 27.47 | 4.31 | 7.80 | 2.56
seg1536582… | 17.03 | 26.35 | 27.56 | 2.42 | 1.52 | 1.07 | 24.96 | 27.43 | 2.72 | 1.96 | 26.21 | 27.69 | 6.57 | 10.41 | 2.47
seg1586862… | 13.35 | 24.26 | 26.71 | 3.28 | 1.63 | 1.21 | 23.20 | 26.68 | 3.37 | 2.44 | 24.29 | 26.83 | 5.94 | 18.78 | 2.60
seg1634531… | 17.38 | 25.68 | 26.57 | 2.38 | 1.14 | 0.99 | 24.32 | 26.50 | 2.21 | 1.85 | 26.27 | 26.82 | 5.31 | 11.85 | 2.23
seg1647019… | 11.35 | 22.67 | 21.26 | 5.32 | 2.42 | 3.18 | 21.98 | 23.88 | 6.08 | 4.38 | 21.90 | 24.18 | 10.36 | 12.25 | 4.31
seg1660852… | 15.75 | 23.14 | 24.58 | 3.68 | 2.13 | 1.82 | 22.23 | 24.47 | 3.51 | 3.49 | 21.37 | 24.82 | 5.11 | 4.72 | 3.91
seg1664636… | 16.34 | 24.87 | 26.86 | 2.65 | 1.36 | 1.03 | 23.99 | 26.71 | 2.36 | 1.79 | 25.23 | 26.83 | 6.54 | 13.86 | 2.26
seg1776195… | 8.55 | 24.70 | 25.35 | 4.34 | 2.14 | 1.89 | 24.30 | 26.37 | 5.94 | 3.87 | 24.95 | 26.71 | 14.52 | 25.24 | 3.90
seg3224923… | 17.36 | 25.97 | 27.20 | 3.27 | 1.89 | 1.43 | 24.91 | 27.20 | 3.23 | 2.86 | 26.31 | 27.26 | 5.42 | 7.16 | 3.53
seg3425716… | 8.48 | 27.03 | 29.42 | 3.05 | 2.01 | 1.38 | 26.18 | 29.60 | 6.85 | 2.93 | 26.33 | 29.79 | 18.81 | 30.68 | 3.00
seg3988957… | 11.70 | 22.62 | 24.19 | 3.43 | 1.87 | 1.52 | 21.75 | 24.25 | 4.18 | 3.00 | 22.02 | 24.65 | 6.07 | 5.66 | 3.30
seg4058410… | 12.14 | 26.40 | 28.08 | 2.93 | 1.31 | 1.04 | 25.03 | 28.09 | 3.13 | 2.18 | 25.83 | 28.33 | 5.46 | 7.02 | 2.62
seg8811210… | 11.18 | 24.87 | 26.60 | 3.07 | 1.44 | 1.31 | 23.69 | 26.49 | 3.92 | 2.64 | 24.28 | 26.47 | 7.16 | 27.30 | 3.83
seg9385013… | 13.80 | 24.16 | 26.07 | 4.75 | 2.39 | 1.67 | 23.23 | 25.92 | 7.47 | 6.28 | 22.85 | 26.55 | 9.10 | 49.34 | 4.52
Average | 13.24 | 25.02 | 26.66 | 3.46 | 1.79 | 1.44 | 24.09 | 26.85 | 4.71 | 3.04 | 24.70 | 27.12 | 8.08 | 18.65 | 3.35

###  4.2 Street-view reconstruction

As shown in Tab. 1, we manually select 32 sequences from the Waymo Open
Dataset [47] to serve as a benchmark for evaluating street-view
reconstruction. The criteria for choosing these sequences are to minimize the
presence of dynamic vehicles and ensure an unobstructed field of view, while
avoiding extremely adverse weather conditions. All of the results are
conducted using street-view images captured from three frontal cameras for
fair comparison. We also demonstrate example qualitative comparisons of
reconstruction without LiDAR in Fig. 8 and with LiDAR in Fig. 9. See more
details and results in the supplementary.

GT |
---|---
Ours |
$\text{F}^{2}$-NeRF [55] |
Ours |
Ours |
$\text{F}^{2}$-NeRF [55] |
Reconstructed surfaces |  |
| Ours |  $\text{F}^{2}$-NeRF [55]
Figure 8: Qualitative comparison of reconstruction without LiDAR, on Waymo Open Dataset [47] seg1586862…. GT |
---|---
Ours |
NGP [32]+L [40] |
Ours |
Ours + Normal cues |
NeuS [54] |
Ours |
NGP [32]+L [40] |
Figure 9: Qualitative comparison of reconstruction with LiDAR, using all five
LiDARs, on Waymo Open Dataset [47] seg1006130….

###  4.3 Geometric metrics

Ground truth 3D scene geometry is often unavailable or difficult to obtain for
street views. However, LiDAR sensors in autonomous driving datasets can
provide indirect 3D geometry information. Despite the potential for noise and
inaccuracy, having a metric is still preferable to having none at all.
Therefore, we propose to use the the Chamfer Distance (C-D)111For chamfer-
distance, we utilize the implementation of Pytorch3D [39]. between the
rendered and original LiDAR point clouds and the RMSE (Root mean square error)
[14] of the rendered depth. See supplementary for more details on these
metrics.

###  4.4 Ablation

Disentangled vs. using only close-range model. The distant-view model mainly
accounts for frontal or lateral distant landscapes, as well as objects that
extend beyond the end of the capture trajectory, both of which are not
distinguishable even using semantic segmentation. But with our approach, they
can be disentangled with no supervisions, as shown by the "Disentangled"
section in Fig. 10.

Disentangled |  |  |  |
---|---|---|---|---
Only close-range |  |  |  |
Waymo Open Dataset [47] seg1347637…
Disentangled |  |  |  |
Only close-range |  |  |  |
Waymo Open Dataset [47] seg1172406…
Disentangled |  |  |  |
Only close-range |  |  |  |
Waymo Open Dataset [47] seg1586862….
| Ground truth image | Predicted normals | Predicted close-range | Predicted distant-view
Figure 10: Ablation study of reconstruction using disentangled close-range and
distant-view models against that using the close-range model alone. In Waymo
seg1347637…and seg1172406…, distant-view space includes distant landscapes,
whereas in seg1586862…and seg1172406…the distant-view also represents scenes
placed at the end of capture journey.

Road-surface vs. enclosed initialization schemes. As shown in Fig. 11,
compared to our road-surface initialization scheme, enclosed initial shapes
may lead to failed disentanglement when training, in which the reconstructed
close-range geometry encases all ego trajectories and is completely enclosed.

seg1006130… |  |
---|---|---
seg1347637… |  |
|  |

(a) Reconstructed surfaces using capsule initialization.

(b) Reconstructed surfaces using our proposed road-surface initialization.

Figure 11:  Qualitative comparison of the reconstructed surfaces using capsule
initialization (left) and using our proposed road-surface initialization
(right). We cut and paint a transparent yellow region on the reconstructed
surface using capsule initialization for better illustration.

Figure 12: Scatter plot showcasing the roughly positive correlation of PSNR improvement with respect to trajectory length when we change from cubic to the proposed cuboid spaces and hash-grids. Table 2: Average results of the selected Waymo sequences: a comparison between utilizing cuboid space and models versus cubic ones. Hash-grid parameter size (cr+dv, number of cr levels) | Cubic | Cuboid
---|---|---
PSNR$\uparrow$ | PSNR$\uparrow$
16+8 Mi, 9-10 levels | 24.49 | 26.14
32+16 Mi, 17-18 levels | 26.73 | 26.86

Cuboid vs. cubic.

In Sec. 3.2, we delimit the close-range space using automatically computed
cuboid boundaries that align with the long and narrow camera trajectories. We
report the results of comparing this cuboid boundary and model with cubic ones
in Tab. 2. It can be observed that cuboid boundaries and hash-grids prove to
be more fine-grained and achieve comparable performance with that of cubic
ones while using only half of the parameter size and hash-grid levels.

As illustrated in Fig. 12, we also provide a detailed scatter plot
demonstrating the improvement of PSNR performance with respect to trajectory
length when switching from the original cubic boundaries and hash-grids to our
proposed cuboid ones. The results indicate a greater performance improvement
can be gained for longer trajectories or streets, thus validating the benefits
of our proposed cuboid space delimitation and models for street-view
scenarios.

##  5 Downstream applications of the reconstructed surface

###  5.1 LiDAR simulation

|
---|---
(a) Qualitative comparison of the ground truth LiDAR pointcloud (left) and the
volume-rendered pointcloud (right) of our close-range NeuS model. The colors
of the predicted pointcloud (right) depict L1 error of the rendered depth.

(b) Directly feeding our volume rendered pointcloud into a segmentation model
based on PCSeg [25]. Different colors indicate different inferred semantics
using this coordinate-feature-only pointcloud segmentation model.

Figure 13: Qualitative demonstration of the volume-rendered LiDAR pointclouds
using our reconstructed geometry of Waymo Open Dataset [47] seg4058410….

One of the major fallback of volume rendering is that its depth rendering is
often indefinite and inaccurate. Even when the training concludes with high
average opacity, there will still be a significant interval of non-negligible
width that contributes to the rendering. This is contrary to common real-world
geometry where each ray intersects the surfaces of objects at one definite
depth, which can be viewed as an infinitesimal interval.

Luckily under the NeuS [54] setting, the geometry of the scene is explicitly
modeled by implicit surfaces, which naturally provides guidance to the volume
rendering process. In addition, we can also directly apply implicit surface
tracing techniques, such as sphere tracing, which guarantees a definite
intersection depth.

In this paper, we manually set the mapping parameter $s$ in Supp. .1 to a
large value, e.g. 64000. Fig. 13(a) demonstrates a qualitative example of the
simulated pointcloud. As shown in Fig. 13(b), in order to further demonstrate
the accuracy of our LiDAR simulation, we directly feed the volume-rendered
pointclouds into a pointcloud segmentation model obtained using the PCSeg [25]
framework. This pointcloud segmentation model to evaluate our LiDAR simulation
is trained on original Waymo Open Dataset [47] and only uses coordinate
features as input since we can not produce intensity channel for now.

###  5.2 High-resolution mesh and occupancy grid extraction

Figure 14: Qualitative example of the extracted occupancy grid @0.1(m)
resolution on Waymo Open Dataset [47] seg1006130…. The geometry is
reconstructed without using LiDAR data.

With the implicit surface being reconstructed, we obtain an continuous
representation of scene geometry that has infinitesimal granularity.
Subsequently, we can extract high-resolution meshes or occupancy grids out of
the reconstructed implicit surface, as shown in Fig. 4, 14.

###  5.3 Implicit-surface-guided ray tracing

|  |  |
---|---|---|---
GT image | Volume-rendered normals | Volume-rendered image | Sphere-traced image
|  |  |
|  |  |
Puresky | Night | Courtyard | Sunrise

(a) Example comparison of volume-rendering vs. sphere tracing.

(b) Shading the implicit surface (bottom) under different envmaps (top) based
on sphere-tracing.

Figure 15: Preliminary examples of sphere-tracing and shading the implicit
surface based on sphere-tracing. Experiments are conducted on Waymo Open
Dataset [47] seg1347637….

Apart from indefinite ray intersection depth, another major fallback of volume
rendering is that it can not obtain definite surface geometry and surface
normals for shading, re-lighting, change of environment map, or other graphics
applications requiring further factorization of the radiance representation.
Using our approach, however, a definite implicit surface with high-fidelity
surface normals can be obtained for street views even without requiring 3D
sensors like LiDAR, as shown in Fig. 2, 8. Subsequently, we can conduct
shading or re-lighting on the reconstructed implicit surface. We present a
preliminary example of the rendered image using sphere-tracing in Fig. 15(a),
as well as applying different environment maps to shade the implicit surfaces
in Fig. 15(b).

##  6 Conclusion, limitations and future work

In this paper, we present StreetSurf, a novel multi-view implicit surface
reconstruction method designed for street views. We mainly tackle the problem
of close-range-vs-distant-view disentanglement and geometric errors caused by
non-object-centric views and unbounded scenes.

However, scenes in autonomous driving datasets are still much challenging and
StreetSurf only handles a small proportion of scenarios in these datasets.
Many other challenges are remained, including dark or dawn illuminations,
numerous dynamic objects, long-tail environmental conditions and so on. In
addition, StreetSurf ignores dynamic foregrounds in street views, but they are
also important to reconstruct and render for various downstream tasks. In our
future work, we will study the task of multi-object rendering and
reconstruction, with StreetSurf as the background model.

## Acknowledgments and Disclosure of Funding

The research was supported by the Science and Technology Commission of
Shanghai Municipality (grant No. 22DZ1100102).

## References

  * Barron et al. [2021] J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In _Proc. IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * Barron et al. [2022] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5470–5479, 2022.
  * Caesar et al. [2020] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 11621–11631, 2020.
  * Chan et al. [2022] E. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. De Mello, O. Gallo, L. J. Guibas, J. Tremblay, S. Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 16123–16133, 2022.
  * Chen et al. [2022a] A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su.  Tensorf: Tensorial radiance fields.  In _Proc. European Conference on Computer Vision (ECCV)_ , pages 333–350, 2022a.
  * Chen et al. [2022b] Z. Chen, C. Ding, J. Guo, D. Wang, Y. Li, X. Xiao, W. Wu, and L. Song.  L-tracing: Fast light visibility estimation on neural surfaces by sphere tracing.  In _Proc. European Conference on Computer Vision (ECCV)_ , pages 217–233, 2022b.
  * Eftekhar et al. [2021] A. Eftekhar, A. Sax, J. Malik, and A. Zamir.  Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans.  In _Proc. IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 10786–10796, 2021.
  * Esposito et al. [2022] S. Esposito, D. Baieri, S. Zellmann, A. Hinkenjann, and E. Rodolà.  Kiloneus: Implicit neural representations with real-time global illumination.  _arXiv preprint arXiv:2206.10885_ , 2022.
  * Fathony et al. [2021] R. Fathony, A. K. Sahu, D. Willmott, and J. Z. Kolter.  Multiplicative filter networks.  In _Proc. International Conference on Learning Representations (ICLR)_ , 2021.
  * Fridovich-Keil et al. [2022] S. Fridovich-Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa.  Plenoxels: Radiance fields without neural networks.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Fu et al. [2022a] Q. Fu, Q. Xu, Y. S. Ong, and W. Tao.  Geo-neus: Geometry-consistent neural implicit surfaces learning for multi-view reconstruction.  In _Proc. Advances in Neural Information Processing Systems (NeurIPS)_ , volume 35, pages 3403–3416, 2022a.
  * Fu et al. [2022b] X. Fu, S. Zhang, T. Chen, Y. Lu, L. Zhu, X. Zhou, A. Geiger, and Y. Liao.  Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene segmentation.  In _International Conference on 3D Vision (3DV)_ , 2022b.
  * Geiger et al. [2012] A. Geiger, P. Lenz, and R. Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3354–3361. IEEE, 2012.
  * Godard et al. [2019] C. Godard, O. Mac Aodha, M. Firman, and G. J. Brostow.  Digging into self-supervised monocular depth estimation.  In _Proc. IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 3828–3838, 2019.
  * Gropp et al. [2020] A. Gropp, L. Yariv, N. Haim, M. Atzmon, and Y. Lipman.  Implicit geometric regularization for learning shapes.  In _Proc. International Conference on Machine Learning (ICML)_ , pages 3789–3799, 2020.
  * Guo et al. [2022] H. Guo, S. Peng, H. Lin, Q. Wang, G. Zhang, H. Bao, and X. Zhou.  Neural 3d scene reconstruction with the manhattan-world assumption.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5511–5520, 2022.
  * Kundu et al. [2022] A. Kundu, K. Genova, X. Yin, A. Fathi, C. Pantofaru, L. Guibas, A. Tagliasacchi, F. Dellaert, and T. Funkhouser.  Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Li et al. [2022a] H. Li, X. Yang, H. Zhai, Y. Liu, H. Bao, and G. Zhang.  Vox-surf: Voxel-based implicit surface representation.  _IEEE Transactions on Visualization and Computer Graphics_ , 2022a.
  * Li et al. [2022b] R. Li, M. Tancik, and A. Kanazawa.  Nerfacc: A general nerf acceleration toolbox.  _arXiv preprint arXiv:2210.04847_ , 2022b.
  * Li et al. [2023] Z. Li, L. Li, and J. Zhu.  Read: Large-scale neural scene rendering for autonomous driving.  In _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , 2023.
  * Liang et al. [2023] Z. Liang, Z. Huang, C. Ding, and K. Jia.  Helixsurf: A robust and efficient neural implicit surface learning of indoor scenes with iterative intertwined regularization.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Lindell et al. [2022] D. B. Lindell, D. Van Veen, J. J. Park, and G. Wetzstein.  Bacon: Band-limited coordinate networks for multiscale scene representation.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 16252–16262, 2022.
  * Ling et al. [2022] J. Ling, Z. Wang, and F. Xu.  Shadowneus: Neural sdf reconstruction by shadow ray supervision.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Liu et al. [2020] L. Liu, J. Gu, K. Zaw Lin, T.-S. Chua, and C. Theobalt.  Neural sparse voxel fields.  In _Proc. Advances in Neural Information Processing Systems (NeurIPS)_ , volume 33, pages 15651–15663, 2020.
  * Liu et al. [2023] Y. Liu, Y. Bai, K. Lingdong, R. Chen, Y. Hou, B. Shi, and Y. Li.  Pcseg: An open source point cloud segmentation codebase.  https://github.com/PJLab-ADG/PCSeg, 2023.
  * Lorensen and Cline [1987] W. E. Lorensen and H. E. Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  In _Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques_ , SIGGRAPH ’87, page 163–169, New York, NY, USA, 1987. Association for Computing Machinery.  ISBN 0897912276.  doi: 10.1145/37401.37422.  URL https://doi.org/10.1145/37401.37422.
  * Martel et al. [2021] J. N. P. Martel, D. B. Lindell, C. Z. Lin, E. R. Chan, M. Monteiro, and G. Wetzstein.  Acorn: Adaptive coordinate networks for neural scene representation.  _ACM Trans. Graph. (SIGGRAPH)_ , 40(4), 2021.
  * Martin-Brualla et al. [2021] R. Martin-Brualla, N. Radwan, M. S. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duckworth.  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 7210–7219, 2021.
  * Meuleman et al. [2023] A. Meuleman, Y.-L. Liu, C. Gao, J.-B. Huang, C. Kim, M. H. Kim, and J. Kopf.  Progressively optimized local radiance fields for robust view synthesis.  _arXiv preprint arXiv:2303.13791_ , 2023.
  * Mildenhall et al. [2021] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Communications of the ACM_ , 65(1):99–106, 2021\.
  * Mu et al. [2023] T.-J. Mu, H.-X. Chen, J.-X. Cai, and N. Guo.  Neural 3d reconstruction from sparse views using geometric priors.  _Computational Visual Media_ , pages 1–11, 2023.
  * Müller et al. [2022] T. Müller, A. Evans, C. Schied, and A. Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Transactions on Graphics (ToG)_ , 41(4):1–15, 2022.
  * Munkberg et al. [2022] J. Munkberg, J. Hasselgren, T. Shen, J. Gao, W. Chen, A. Evans, T. Müller, and S. Fidler.  Extracting Triangular 3D Models, Materials, and Lighting From Images.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8280–8290, June 2022.
  * Oechsle et al. [2021] M. Oechsle, S. Peng, and A. Geiger.  Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction.  In _Proc. IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 5589–5599, 2021.
  * Ost et al. [2021] J. Ost, F. Mannan, N. Thuerey, J. Knodt, and F. Heide.  Neural scene graphs for dynamic scenes.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2856–2865, 2021.
  * Paszke et al. [2019] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In _Proc. Advances in Neural Information Processing Systems (NeurIPS)_ , pages 8024–8035, 2019.
  * Qiu et al. [2023] J. Qiu, P.-T. Jiang, Y. Zhu, Z.-X. Yin, M.-M. Cheng, and B. Ren.  Looking through the glass: Neural surface reconstruction against high specular reflections.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Ramasinghe and Lucey [2022] S. Ramasinghe and S. Lucey.  Beyond periodicity: towards a unifying framework for activations in coordinate-mlps.  In _Proc. European Conference on Computer Vision (ECCV)_ , pages 142–158, 2022.
  * Ravi et al. [2020] N. Ravi, J. Reizenstein, D. Novotny, T. Gordon, W.-Y. Lo, J. Johnson, and G. Gkioxari.  Accelerating 3d deep learning with pytorch3d.  _arXiv:2007.08501_ , 2020.
  * Rematas et al. [2022] K. Rematas, A. Liu, P. P. Srinivasan, J. T. Barron, A. Tagliasacchi, T. Funkhouser, and V. Ferrari.  Urban radiance fields.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 12932–12942, 2022.
  * Rosu and Behnke [2023] R. A. Rosu and S. Behnke.  Permutosdf: Fast multi-view reconstruction with implicit surfaces using permutohedral lattices.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Schönberger and Frahm [2016] J. L. Schönberger and J.-M. Frahm.  Structure-from-motion revisited.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * Sitzmann et al. [2019] V. Sitzmann, J. Thies, F. Heide, M. Nießner, G. Wetzstein, and M. Zollhofer.  Deepvoxels: Learning persistent 3d feature embeddings.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2437–2446, 2019.
  * Sitzmann et al. [2020] V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein.  Implicit neural representations with periodic activation functions.  In _Proc. Advances in Neural Information Processing Systems (NeurIPS)_ , volume 33, pages 7462–7473, 2020.
  * Sun et al. [2022a] C. Sun, M. Sun, and H.-T. Chen.  Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5459–5469, 2022a.
  * Sun et al. [2022b] J. Sun, X. Chen, Q. Wang, Z. Li, H. Averbuch-Elor, X. Zhou, and N. Snavely.  Neural 3d reconstruction in the wild.  In _ACM SIGGRAPH Conference Proceedings_ , pages 1–9, 2022b.
  * Sun et al. [2020] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, et al.  Scalability in perception for autonomous driving: Waymo open dataset.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2446–2454, 2020.
  * Takikawa et al. [2021] T. Takikawa, J. Litalien, K. Yin, K. Kreis, C. Loop, D. Nowrouzezahrai, A. Jacobson, M. McGuire, and S. Fidler.  Neural geometric level of detail: Real-time rendering with implicit 3d shapes.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 11358–11367, 2021.
  * Tancik et al. [2022] M. Tancik, V. Casser, X. Yan, S. Pradhan, B. Mildenhall, P. P. Srinivasan, J. T. Barron, and H. Kretzschmar.  Block-nerf: Scalable large scene neural view synthesis.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8248–8258, 2022.
  * Turki et al. [2022] H. Turki, D. Ramanan, and M. Satyanarayanan.  Mega-nerf: Scalable construction of large-scale nerfs for virtual fly-throughs.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 12922–12931, June 2022.
  * Turki et al. [2023] H. Turki, J. Y. Zhang, F. Ferroni, and D. Ramanan.  Suds: Scalable urban dynamic scenes.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Vizzo et al. [2022] I. Vizzo, T. Guadagnino, J. Behley, and C. Stachniss.  Vdbfusion: Flexible and efficient tsdf integration of range sensor data.  _Sensors_ , 22(3), 2022.  ISSN 1424-8220.  doi: 10.3390/s22031296.  URL https://www.mdpi.com/1424-8220/22/3/1296.
  * Wang et al. [2022a] J. Wang, P. Wang, X. Long, C. Theobalt, T. Komura, L. Liu, and W. Wang.  Neuris: Neural reconstruction of indoor scenes using normal priors.  In _Proc. European Conference on Computer Vision (ECCV)_ , pages 139–155, 2022a.
  * Wang et al. [2021a] P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  In _Proc. Advances in Neural Information Processing Systems (NeurIPS)_ , volume 34, pages 27171–27183, 2021a.
  * Wang et al. [2023a] P. Wang, Y. Liu, Z. Chen, L. Liu, Z. Liu, T. Komura, C. Theobalt, and W. Wang.  F2-nerf: Fast neural radiance field training with free camera trajectories.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023a.
  * Wang et al. [2022b] Y. Wang, Q. Han, M. Habermann, K. Daniilidis, C. Theobalt, and L. Liu.  Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction.  _arXiv preprint arXiv:2212.05231_ , 2022b.
  * Wang et al. [2022c] Y. Wang, Z. Li, Y. Jiang, K. Zhou, T. Cao, Y. Fu, and C. Xiao.  Neuralroom: Geometry-constrained neural implicit surfaces for indoor scene reconstruction.  _ACM Transactions on Graphics (TOG)_ , 41(6):1–15, 2022c.
  * Wang et al. [2022d] Y. Wang, I. Skorokhodov, and P. Wonka.  Hf-neus: Improved surface reconstruction using high-frequency details.  In _Proc. Advances in Neural Information Processing Systems (NeurIPS)_ , volume 35, pages 1966–1978, 2022d.
  * Wang et al. [2021b] Z. Wang, S. Wu, W. Xie, M. Chen, and V. A. Prisacariu.  NeRF$--$: Neural radiance fields without known camera parameters.  _arXiv preprint arXiv:2102.07064_ , 2021b.
  * Wang et al. [2023b] Z. Wang, T. Shen, J. Gao, S. Huang, J. Munkberg, J. Hasselgren, Z. Gojcic, W. Chen, and S. Fidler.  Neural fields meet explicit geometric representations for inverse rendering of urban scenes.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2023b.
  * Xie et al. [2021] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo.  Segformer: Simple and efficient design for semantic segmentation with transformers.  In _Proc. Advances in Neural Information Processing Systems (NeurIPS)_ , volume 34, pages 12077–12090, 2021.
  * Xie et al. [2022] Y. Xie, T. Takikawa, S. Saito, O. Litany, S. Yan, N. Khan, F. Tombari, J. Tompkin, V. Sitzmann, and S. Sridhar.  Neural fields in visual computing and beyond.  In _Computer Graphics Forum_ , volume 41, pages 641–676. Wiley Online Library, 2022.
  * Xie et al. [2023] Z. Xie, J. Zhang, W. Li, F. Zhang, and L. Zhang.  S-nerf: Neural radiance fields for street views.  _arXiv preprint arXiv:2303.00749_ , 2023.
  * Yariv et al. [2020] L. Yariv, Y. Kasten, D. Moran, M. Galun, M. Atzmon, B. Ronen, and Y. Lipman.  Multiview neural surface reconstruction by disentangling geometry and appearance.  In _Proc. Advances in Neural Information Processing Systems (NeurIPS)_ , volume 33, pages 2492–2502, 2020.
  * Yariv et al. [2021] L. Yariv, J. Gu, Y. Kasten, and Y. Lipman.  Volume rendering of neural implicit surfaces.  In _Proc. Advances in Neural Information Processing Systems (NeurIPS)_ , volume 34, pages 4805–4815, 2021.
  * Yu et al. [2021] A. Yu, R. Li, M. Tancik, H. Li, R. Ng, and A. Kanazawa.  PlenOctrees for real-time rendering of neural radiance fields.  In _Proc. IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * Yu et al. [2022] Z. Yu, S. Peng, M. Niemeyer, T. Sattler, and A. Geiger.  Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction.  In _Proc. Advances in Neural Information Processing Systems (NeurIPS)_ , 2022.
  * Zhang et al. [2022] J. Zhang, Y. Yao, S. Li, T. Fang, D. McKinnon, Y. Tsin, and L. Quan.  Critical regularizations for neural surface reconstruction in the wild.  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6270–6279, 2022.
  * Zhang et al. [2020] K. Zhang, G. Riegler, N. Snavely, and V. Koltun.  Nerf++: Analyzing and improving neural radiance fields.  _arXiv preprint arXiv:2010.07492_ , 2020.

Supplemental Material for StreetSurf: Extending Multi-view Implicit Surface
Reconstruction to Street Views

In this supplementary document, we first demonstrate more results on street-
view reconstruction in Sec. . Then, we go through more implementation details
over the whole optimization process in Sec. .

##  Additional results

For all Waymo Open Dataset [47] experiments, we use images captured from three
frontal cameras for fair comparison. Some of sequences are trimmed due to
limited ego-motion or a high density of dynamic objects. The details of
selection are shown in Tab. 3.

Table 3: Information on the selected and trimmed Waymo Open Dataset [47]
sequences. "–" indicates using the original start or end frames (typically 198
frames) without trimming.

Sequence | Start | End | Trajectory
---|---|---|---
frame | frame | length (m)
seg1006130… | – | 163 | 100.6
seg1027514… | – | – | 153.9
seg1067626… | – | – | 210.6
seg1137922… | – | – | 167.6
seg1172406… | – | – | 141.4
seg1287964… | – | – | 71.0
seg1308545… | – | – | 106.1
seg1314219… | 17 | – | 121.1
seg1319679… | – | – | 73.0
seg1323841… | – | – | 186.6
seg1347637… | – | 140 | 49.4
seg1400454… | 24 | – | 129.8
seg1434813… | – | – | 69.6
seg1442480… | – | – | 157.6
seg1486973… | – | – | 119.3
seg1506235… | – | – | 197.2

Sequence | Start | End | Trajectory
---|---|---|---
frame | frame | length (m)
seg1522170… | – | – | 72.8
seg1527063… | 30 | – | 113.1
seg1534950… | 80 | – | 70.5
seg1536582… | – | 170 | 111.9
seg1586862… | 70 | – | 83.0
seg1634531… | – | – | 160.1
seg1647019… | – | – | 125.0
seg1660852… | – | 120 | 66.2
seg1664636… | – | – | 107.8
seg1776195… | – | – | 153.5
seg3224923… | – | – | 151.4
seg3425716… | – | – | 69.7
seg3988957… | – | – | 137.2
seg4058410… | – | 90 | 55.4
seg8811210… | – | – | 137.3
seg9385013… | – | – | 263.9

###  .1 Street-view reconstruction without LiDAR

GT |
---|---
Ours |
$\text{F}^{2}$-NeRF [55] |
Ours |
Ours |
$\text{F}^{2}$-NeRF [55] |
Reconstructed surfaces |  |
| Ours |  $\text{F}^{2}$-NeRF [55]
Figure 16: Qualitative comparison of reconstruction without LiDAR, on Waymo Open Dataset [47] seg1027514…. GT |
---|---
Ours |
$\text{F}^{2}$-NeRF [55] |
Ours |
Ours |
$\text{F}^{2}$-NeRF [55] |
Reconstructed surfaces |  |
| Ours |  $\text{F}^{2}$-NeRF [55]
Figure 17: Qualitative comparison of reconstruction without LiDAR, on Waymo
Open Dataset [47] seg4058410….

For reconstruction without LiDAR data, we compare our approach with two
baselines. The first is the traditional multi-view reconstruction pipeline
COLMAP [42]. The second is the state of the art novel view synthesis framework
aiming at non-object-centric camera trajectories, $\text{F}^{2}$-NeRF [56].
Example qualitative comparisons are shown in Fig. 16, 17. Quantitative
comparison is reported in Tab. 4.

GT |
---|---
Ours |
NGP [32]+L [40] |
Ours |
Ours + Normal cues |
NeuS [54] |
Ours |
NGP [32]+L [40] |
Figure 18: Qualitative comparison of reconstruction with LiDAR, using all five LiDARs, on Waymo Open Dataset [47] seg1347637…. GT |
---|---
Ours |
NGP [32]+L [40] |
Ours |
Ours + Normal cues |
NeuS [54] |
Ours |
NGP [32]+L [40] |
Figure 19: Qualitative comparison of reconstruction with LiDAR, using all five LiDARs, on Waymo Open Dataset [47] seg1506235…. Table 4: Reconstruction without LiDAR on selected Waymo Open Dataset [47] sequences. Sequence | PSNR$\uparrow$ | C-D$\downarrow$ | RMSE$\downarrow$
---|---|---|---
$\text{F}^{2}$-NeRF | Ours | COLMAP |  $\text{F}^{2}$-NeRF | Ours | COLMAP |  $\text{F}^{2}$-NeRF | Ours
seg1006130… | 24.41 | 27.39 | 99.01 | 70.90 | 0.88 | 7.10 | 8.87 | 2.99
seg1027514… | 22.14 | 27.58 | 127.19 | 295.85 | 0.81 | 7.47 | 16.52 | 2.91
seg1067626… | 26.43 | 29.01 | 235.26 | 906.56 | 1.58 | 9.06 | 35.59 | 4.34
seg1137922… | 25.42 | 28.33 | 241.81 | 398.55 | 2.67 | 12.39 | 20.10 | 5.70
seg1172406… | 27.16 | 28.50 | 175.13 | 480.75 | 0.56 | 13.62 | 9.00 | 2.57
seg1287964… | 25.40 | 29.89 | 152.94 | 21.50 | 0.89 | 10.34 | 6.73 | 3.19
seg1308545… | 23.55 | 25.95 | 150.52 | 560.63 | 1.43 | 8.64 | 15.50 | 4.12
seg1314219… | 21.76 | 26.07 | 139.69 | 588.07 | 0.88 | 6.75 | 19.30 | 3.48
seg1319679… | 24.54 | 25.66 | 115.46 | 517.39 | 1.48 | 7.63 | 23.50 | 4.76
seg1323841… | 24.03 | 28.49 | 144.61 | 473.01 | 0.80 | 7.32 | 20.19 | 3.13
seg1347637… | 26.12 | 28.26 | 69.94 | 932.15 | 0.41 | 5.93 | 21.72 | 1.84
seg1400454… | 24.58 | 25.49 | 148.23 | 11196.87 | 1.09 | 8.08 | 39.85 | 3.29
seg1434813… | 26.15 | 28.23 | 111.42 | 1567.38 | 1.95 | 8.48 | 35.96 | 4.74
seg1442480… | 25.14 | 27.56 | 171.49 | 1111.72 | 0.84 | 7.85 | 36.35 | 2.97
seg1486973… | 22.87 | 24.42 | 89.58 | 9.85 | 0.57 | 5.52 | 3.53 | 2.82
seg1506235… | 25.07 | 27.41 | 132.43 | 638.32 | 0.67 | 7.84 | 27.61 | 2.40
seg1522170… | 25.71 | 26.29 | 145.69 | 388.61 | 2.11 | 11.28 | 16.66 | 4.87
seg1527063… | 26.81 | 29.06 | 93.03 | 142.34 | 0.44 | 2.62 | 7.82 | 1.98
seg1534950… | 25.30 | 27.47 | 99.29 | 53.91 | 0.58 | 4.31 | 7.80 | 2.56
seg1536582… | 26.21 | 27.69 | 71.87 | 56.03 | 0.56 | 6.57 | 10.41 | 2.47
seg1586862… | 24.29 | 26.83 | 112.04 | 653.87 | 0.57 | 5.94 | 18.78 | 2.60
seg1634531… | 26.27 | 26.82 | 90.64 | 187.37 | 0.44 | 5.31 | 11.85 | 2.23
seg1647019… | 21.90 | 24.18 | 170.05 | 96.24 | 1.08 | 10.36 | 12.25 | 4.31
seg1660852… | 21.37 | 24.82 | 102.14 | 4.21 | 0.98 | 5.11 | 4.72 | 3.91
seg1664636… | 25.23 | 26.83 | 97.62 | 265.26 | 0.52 | 6.54 | 13.86 | 2.26
seg1776195… | 24.95 | 26.71 | 234.98 | 540.94 | 1.32 | 14.52 | 25.24 | 3.90
seg3224923… | 26.31 | 27.26 | 126.61 | 26.14 | 1.14 | 5.42 | 7.16 | 3.53
seg3425716… | 26.33 | 29.79 | 196.65 | 2673.64 | 0.69 | 18.81 | 30.68 | 3.00
seg3988957… | 22.02 | 24.65 | 126.66 | 23.62 | 0.77 | 6.07 | 5.66 | 3.30
seg4058410… | 25.83 | 28.33 | 88.46 | 34.89 | 0.62 | 5.46 | 7.02 | 2.62
seg8811210… | 24.28 | 26.47 | 154.04 | 680.01 | 1.35 | 7.16 | 27.30 | 3.83
seg9385013… | 22.85 | 26.55 | 179.23 | 2780.03 | 1.87 | 9.10 | 49.34 | 4.52
Average | 24.70 | 27.12 | 137.30 | 886.77 | 1.02 | 8.08 | 18.65 | 3.35
Table 5: Reconstruction with LiDAR on selected Waymo Open Dataset [47]
sequences.

| Using 1 dense + 4 sparse LiDARs | Using 4 sparse LiDARs
---|---|---
Sequence |  PSNR$\uparrow$ |  C-D$\downarrow$ |  RMSE$\downarrow$ |  PSNR$\uparrow$ |  C-D$\downarrow$ |  RMSE$\downarrow$
NeuS | NGP+L | Ours | NeuS | NGP+L | Ours | NeuS | NGP+L | Ours | NGP+L | Ours | NGP+L | Ours | NGP+L | Ours
seg1006130… | 14.57 | 25.15 | 27.38 | 0.68 | 0.19 | 0.13 | 3.14 | 1.67 | 1.22 | 23.86 | 27.26 | 0.40 | 0.42 | 3.22 | 2.58
seg1027514… |  9.10 | 23.92 | 27.19 | 0.88 | 0.44 | 0.25 | 3.30 | 1.94 | 1.40 | 23.48 | 27.11 | 0.69 | 0.66 | 3.25 | 2.82
seg1067626… | 12.90 | 25.94 | 27.68 | 1.11 | 0.58 | 0.35 | 4.40 | 2.61 | 1.94 | 24.87 | 28.68 | 3.03 | 1.50 | 7.61 | 4.12
seg1137922… | 13.75 | 25.93 | 27.64 | 1.23 | 0.57 | 0.33 | 5.77 | 2.56 | 2.14 | 25.15 | 27.80 | 6.60 | 3.27 | 9.62 | 5.91
seg1172406… | 21.28 | 26.16 | 27.86 | 0.40 | 0.17 | 0.13 | 2.18 | 1.51 | 1.03 | 24.89 | 28.21 | 0.33 | 0.25 | 2.80 | 1.83
seg1287964… | 10.64 | 26.31 | 29.92 | 0.63 | 0.20 | 0.16 | 3.34 | 1.64 | 1.32 | 25.98 | 29.45 | 2.29 | 1.31 | 6.79 | 3.50
seg1308545… | 16.57 | 24.22 | 25.59 | 0.95 | 0.25 | 0.20 | 4.09 | 2.03 | 1.59 | 22.99 | 25.61 | 4.43 | 2.98 | 7.85 | 4.40
seg1314219… | 10.78 | 22.85 | 25.77 | 0.77 | 0.37 | 0.22 | 3.52 | 2.14 | 1.53 | 22.18 | 25.66 | 1.26 | 1.59 | 6.13 | 3.94
seg1319679… | 10.71 | 24.69 | 24.79 | 1.10 | 0.25 | 0.17 | 4.97 | 2.15 | 1.68 | 23.78 | 25.30 | 2.45 | 2.36 | 7.94 | 5.21
seg1323841… |  8.59 | 24.13 | 28.16 | 0.83 | 0.29 | 0.18 | 3.64 | 1.75 | 1.37 | 23.37 | 28.16 | 0.69 | 0.50 | 3.70 | 2.66
seg1347637… | 14.19 | 26.18 | 28.20 | 0.39 | 0.09 | 0.08 | 2.08 | 1.03 | 0.84 | 25.30 | 28.18 | 0.17 | 0.15 | 1.84 | 1.41
seg1400454… | 11.43 | 24.73 | 24.79 | 0.80 | 0.29 | 0.19 | 3.22 | 1.81 | 1.38 | 23.96 | 25.40 | 0.87 | 0.76 | 4.35 | 2.91
seg1434813… | 10.71 | 26.31 | 26.98 | 1.67 | 0.22 | 0.25 | 4.59 | 2.12 | 1.81 | 25.43 | 27.75 | 4.40 | 0.97 | 8.71 | 4.30
seg1442480… | 13.33 | 25.16 | 27.22 | 0.70 | 0.31 | 0.20 | 3.30 | 1.70 | 1.42 | 24.28 | 27.28 | 0.81 | 0.73 | 3.69 | 2.77
seg1486973… | 16.23 | 23.59 | 24.21 | 0.52 | 0.20 | 0.15 | 2.90 | 1.52 | 1.22 | 21.99 | 24.20 | 0.31 | 0.36 | 2.89 | 2.17
seg1506235… | 12.01 | 25.07 | 27.54 | 0.66 | 0.24 | 0.13 | 2.67 | 1.43 | 1.04 | 24.12 | 27.08 | 0.64 | 0.39 | 3.22 | 1.89
seg1522170… | 12.67 | 25.26 | 25.66 | 0.99 | 0.33 | 0.18 | 4.79 | 2.14 | 1.68 | 24.77 | 25.67 | 3.46 | 1.84 | 6.87 | 4.87
seg1527063… | 17.46 | 27.04 | 28.95 | 0.28 | 0.11 | 0.07 | 1.92 | 1.04 | 0.76 | 26.03 | 28.76 | 0.31 | 0.24 | 2.46 | 1.55
seg1534950… | 12.35 | 25.42 | 27.26 | 0.36 | 0.09 | 0.11 | 2.30 | 1.34 | 1.06 | 24.58 | 27.20 | 0.37 | 0.18 | 2.77 | 1.94
seg1536582… | 17.03 | 26.35 | 27.56 | 0.34 | 0.20 | 0.11 | 2.42 | 1.52 | 1.07 | 24.96 | 27.43 | 0.24 | 0.27 | 2.72 | 1.96
seg1586862… | 13.35 | 24.26 | 26.71 | 0.57 | 0.21 | 0.13 | 3.28 | 1.63 | 1.21 | 23.20 | 26.68 | 0.54 | 0.49 | 3.37 | 2.44
seg1634531… | 17.38 | 25.68 | 26.57 | 0.37 | 0.10 | 0.10 | 2.38 | 1.14 | 0.99 | 24.32 | 26.50 | 0.17 | 0.23 | 2.21 | 1.85
seg1647019… | 11.35 | 22.67 | 21.26 | 1.24 | 0.36 | 0.29 | 5.32 | 2.42 | 3.18 | 21.98 | 23.88 | 1.28 | 1.23 | 6.08 | 4.38
seg1660852… | 15.75 | 23.14 | 24.58 | 0.65 | 0.24 | 0.22 | 3.68 | 2.13 | 1.82 | 22.23 | 24.47 | 0.73 | 0.78 | 3.51 | 3.49
seg1664636… | 16.34 | 24.87 | 26.86 | 0.63 | 0.22 | 0.13 | 2.65 | 1.36 | 1.03 | 23.99 | 26.71 | 0.62 | 0.20 | 2.36 | 1.79
seg1776195… |  8.55 | 24.70 | 25.35 | 1.13 | 0.47 | 0.31 | 4.34 | 2.14 | 1.89 | 24.30 | 26.37 | 1.70 | 1.03 | 5.94 | 3.87
seg3224923… | 17.36 | 25.97 | 27.20 | 0.75 | 0.43 | 0.20 | 3.27 | 1.89 | 1.43 | 24.91 | 27.20 | 0.68 | 0.80 | 3.23 | 2.86
seg3425716… |  8.48 | 27.03 | 29.42 | 0.58 | 0.43 | 0.19 | 3.05 | 2.01 | 1.38 | 26.18 | 29.60 | 1.80 | 0.56 | 6.85 | 2.93
seg3988957… | 11.70 | 22.62 | 24.19 | 0.75 | 0.28 | 0.19 | 3.43 | 1.87 | 1.52 | 21.75 | 24.25 | 0.99 | 0.64 | 4.18 | 3.00
seg4058410… | 12.14 | 26.40 | 28.08 | 0.48 | 0.15 | 0.10 | 2.93 | 1.31 | 1.04 | 25.03 | 28.09 | 0.39 | 0.48 | 3.13 | 2.18
seg8811210… | 11.18 | 24.87 | 26.60 | 0.72 | 0.19 | 0.21 | 3.07 | 1.44 | 1.31 | 23.69 | 26.49 | 0.68 | 0.70 | 3.92 | 2.64
seg9385013… | 13.80 | 24.16 | 26.07 | 1.20 | 0.58 | 0.27 | 4.75 | 2.39 | 1.67 | 23.23 | 25.92 | 3.02 | 4.94 | 7.47 | 6.28
Average | 13.24 | 25.02 | 26.66 | 0.76 | 0.28 | 0.19 | 3.46 | 1.79 | 1.44 | 24.09 | 26.85 | 1.45 | 0.90 | 4.71 | 3.04

###  .2 Street-view reconstruction with LiDAR

For reconstruction with LiDAR of Waymo Open Dataset [47], we compare our
reconstruction method using LiDAR with two baselines. The first is our basis
implicit surface reconstruction method, NeuS [54]. We deepen and widen its MLP
(Multi Layer Perceptron) to 10 layers with 512 neurons, and increase its ray
batch size from 512 to 2048. The second one is the combination of NGP [32]
with the line-of-sight and depth losses from UrbanNeRF [40] for leveraging
LiDAR supervision, dubbed "NGP+L".

We conduct experiments in two different settings. In the first setting, we
only employ the four relatively sparser auxiliary LiDARs excluding the top
LiDAR, while using the top LiDAR data for evaluation. In the second setting,
we utilize all five LiDARs, including the top LiDAR data as well as four
additional auxiliary LiDARs, and evaluate using the top LiDAR. Example
qualitative comparisons are shown in Fig., 18, 19. Detailed quantitative
comparisons are shown in Tab. 5.

##  Implementation Details

###  .1 Volume rendering of close-range implicit surfaces

Following NeuS [54], we use SDF (Signed Distance Function) fields for the
geometry of close-range scene and map the SDF values
$\widehat{\mathcal{S}}_{i}$ of sample points
$\\{\mathbf{x}^{(\text{cr})}_{i}=\mathbf{o}+t_{i}^{(\text{cr})}\mathbf{v}\\}$
to opacity $\alpha_{i}^{(\text{cr})}$ in order to apply volume rendering:

$\alpha^{(\text{cr})}_{i}=\max\left(\frac{\Phi_{s}(\widehat{\mathcal{S}}_{i}))-\Phi_{s}(\widehat{\mathcal{S}}_{i+1})}{\Phi_{s}(\widehat{\mathcal{S}}_{i})},0\right)$
(8)

where $s$ is a learnable scaling parameter of the Sigmoid function
$\Phi_{s}(x)=(1+e^{-s\cdot x})^{-1}$. This mapping strategy ensures unbiased
calculation of color contribution (i.e. visibility weights) while respecting
occlusion. For more details, please refer to NeuS [54].

In practice, it is difficult for $s$ to converge to a sufficient extent when
training without LiDAR data. In this case, we manually adjust $s$ to increase
linearly after the initial training stage, which is typically around 5k
iterations.

###  .2 Optional sky masks

Previous research studying multi-view neural reconstruction [46, 18] or novel
view synthesis [40, 49] on outdoor scenes typically require mask annotation or
segmentation that marks sky pixels in order to penalize non-occupied areas.
Thanks to the disentangled design of our close-range and distant-view models,
our approach works effectively with or without sky mask inputs.

As shown in Fig. 20, getting rid of mask dependency is beneficial because the
sky masks produced by segmentation models are often noisy and may incorrectly
include foreground elements.

|  With mask and sky |  |
---|---|---|---
GT image |  | Rendered image | Rendered normals
|  Without mask or sky |  |
Estimated occupancy mask |  | Rendered image | Rendered normals
Figure 20: Example illustration of the influence of incorrectly inferred sky
masks on reconstruction (top row) and the benefit of reconstruction without
masks (bottom row) to avoid such segmentation errors. Experiments are
conducted on Waymo Open Dataset [47] seg1664636…. The bottom-left figure
demonstrates an incorrect occupancy mask inferred using Segformer [61], which
has mistakenly classified the white wall as sky.

If sky masks $M^{(\text{sky})}$ are provided, we explicit distinguish the sky
model out of distant-view. We use $M^{(\text{sky})}$ to supervise the rendered
occupancy $\widehat{O}^{(\text{cr,dv})}$ of the joint close-range and distant-
view models as shown in Equ. 9, where BCE stands for the binary cross entropy
loss. It is worth noting that the distant-view model is still necessary even
with the use of sky masks and the sky model. This is because the close-range
scene we defined only applies to objects within a few hundred meters, while a
distant-view model is required to represent landscapes as well as objects that
extend beyond the end of the capture trajectory.

$\mathcal{L}_{\text{mask}}=\begin{cases}\text{BCE}\left(\widehat{O}^{(\text{cr,dv})}(\mathbf{r}),1-M^{(\text{sky})}(\mathbf{r})\right)&\text{with
sky}\\\ 0&\text{without sky}\end{cases}$ (9)

If sky masks are not provided, disentanglement of distant view and the sky is
found to be difficult. Therefore, we let the distant-view model to inclusively
represent the sky by setting a large step at the end of marching interval for
each ray. In this case, the close-range and distant-view models can still be
disentangled without extra supervision.

In practice, distinguishing the sky model out of the distant-view model is
preferred due to its better performance. If sky annotations are not provided,
we use SegFormer [61] to efficiently and automatically extract sky masks.

###  .3 Ray marching with multi-stage hierarchical sampling based on occupancy
information

We propose a ray marching strategy that apply multi-stage hierarchical
sampling to the marched ray samples of the occupancy grid. Examples of the
updated occupancy information at the end of training stage are shown in Fig.
21.

Previous approach studying multi-view reconstruction of outdoor scenes require
either reconstructing the scene in advance [46] with COLMAP [42] or feeding
LiDAR pointclouds to pre-compute occupied voxels [18]. Following NGP [32], we
update the occupancy information in a bootstrap manner and do not rely on any
geometry information. Examples of the update process of occupancy information
is shown in Fig. 22.

Figure 21: Illustration of the updated occupancy information for ray marching at the end of the training step of each data sequence. The cuboid wire-frame indicates the AABB (Axis-Aligned-Bounding-Box) of the close-range scene. seg1137922… |  |  |  |  |
---|---|---|---|---|---
| 0 iter | 400 iter | 1.2k iter | 1.6k iter | Final
seg1006130… |  |  |  |  |
| 0 iter | 400 iter | 1.2k iter | 1.6k iter | Final
Figure 22: Bootstrap update procedure of occupancy information for ray
marching.

###  .4 Additional details

We use a batch size of 8192 rays with up to $48(\text{cr})+64(\text{dv})$
meaningful samples on each ray and conduct meaningful sample reduction as [32,
19] to ensure efficiency. The optimization takes 15k iterations for
reconstruction with LiDAR data and 12k for without LiDAR data, both of which
takes one hour or two on a single RTX3090 GPU.

To further stabilize the disentanglement of close-range and distant-view, we
follow [56] and introduce an annealing binary mask on different resolutions of
the close-range hash-grid that gradually unleashes higher resolutions at the
initial phase of training.

Image pixels are importance-sampled based on periodically accumulated and
decaying error maps following NGP [32]. All codes are implemented under
PyTorch [36] with certain atomic or pack-like operations implemented under
Pytorch-CUDA extension.

###  .5 Geometry evaluation metrics

Ground truth 3D scene geometry is often unavailable or difficult to obtain for
street views. Nonetheless, the LiDAR sensor in the autonomous driving datasets
that we focus on can provide indirect 3D geometry information. Despite the
potential for noise and inaccuracy in LiDAR sensors, having a metric is still
preferable to having none at all. In practice, we find that the LiDAR sensor
data in Waymo-perception dataset [47] is of relatively high quality and can
serve as a reliable evaluation basis. Therefore, we propose a set of metrics
to evaluate the reconstruction accuracy based on the comparison of the ground
truth LiDAR sensor data with the rendered results on the provided LiDAR beams.

LiDAR sensors can be viewed as sparse depth sensors if we only use the first
range return. Following common practice of depth estimation [14], we use the
usual metrics RMSE (Root mean square error).

Depth metrics are sensitive for LiDAR beams closing to the edge of object
boundaries. Hence, we also use the Chamfer Distance (C-D)222We utilize the
implementation of Pytorch3D [39]. between the rendered and original LiDAR
point clouds $\widehat{G},G$:

$\text{C-D}(\widehat{G},G)=\frac{1}{|\widehat{G}|}\sum_{\mathbf{x}\in\widehat{G}}\min_{\mathbf{y}\in
G}\lVert\mathbf{x}-\mathbf{y}\rVert_{2}^{2}+\frac{1}{|G|}\sum_{\mathbf{y}\in
G}\min_{\mathbf{x}\in\widehat{G}}\lVert\mathbf{y}-\mathbf{x}\rVert_{2}^{2}$
(10)

In practice, we have found that LiDAR sensors may produce some outlier
observation points that significantly affect the calculation of the average
chamfer distance. Therefore, we limit the calculation of chamfer distance to
the first closest 97% of ground truth points and discard the remaining 3%.