(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: IIIS, Tsinghua University
11email<EMAIL_ADDRESS>University of Science
and Technology Beijing33institutetext: University of Southern California

# PreSight: Enhancing Autonomous Vehicle Perception with City-Scale NeRF
Priors

Tianyuan Yuan 11 Yucheng Mao, 1122 Jiawei Yang 33 Yicheng Liu 11 Yue Wang 33
Hang Zhao Correspoding Author.11

###### Abstract

Autonomous vehicles rely extensively on perception systems to navigate and
interpret their surroundings. Despite significant advancements in these
systems recently, challenges persist under conditions like occlusion, extreme
lighting, or in unfamiliar urban areas. Unlike these systems, humans do not
solely depend on immediate observations to perceive the environment. In
navigating new cities, humans gradually develop a preliminary mental map to
supplement real-time perception during subsequent visits. Inspired by this
human approach, we introduce a novel framework, Pre-Sight, that leverages past
traversals to construct static prior memories, enhancing online perception in
later navigations. Our method involves optimizing a city-scale neural radiance
field with data from previous journeys to generate neural priors. These
priors, rich in semantic and geometric details, are derived without manual
annotations and can seamlessly augment various state-of-the-art perception
models, improving their efficacy with minimal additional computational cost.
Experimental results on the nuScenes dataset demonstrate the framework’s high
compatibility with diverse online perception models. Specifically, it shows
remarkable improvements in HD-map construction and occupancy prediction tasks,
highlighting its potential as a new perception framework for autonomous
driving systems. Our code will be released at
https://github.com/yuantianyuan01/PreSight.

###### Keywords:

Autonomous Driving Vision-Based Perception Neural Implicit Field

##  1 Introduction

Vision-based perception modules serve as the cornerstone of autonomous vehicle
navigation, ensuring the safety and reliability of these systems. These
modules support a variety of perception tasks, such as 3D object detection
[42, 43, 22, 16, 15, 28], online map construction [20, 29, 26, 49, 45], and
occupancy prediction [18, 40, 25, 21], each focusing on different aspects of
the vehicle’s surroundings. Perception tasks can be broadly classified into
dynamic object perception, which deals with moving entities like vehicles and
pedestrians, and static environment perception, focusing on invariant features
such as lane markers, vegetation, and obstacles. While dynamic objects require
real-time observation, static environments can be pre-mapped and annotated,
enhancing navigational safety. Conventionally, static environments are
captured in high-definition (HD) maps using Simultaneous Localization and
Mapping (SLAM) techniques [37, 51, 36, 7], supplemented with manual
annotations. Despite their precision, expanding these methods to cover new
geographic regions or incorporate new attributes presents considerable
challenges, primarily due to the extensive reliance on manual labor for
annotations and updates. In response, online perception models powered by deep
learning have been developed [20, 29, 26, 23], utilizing only onboard sensor
data to infer the surrounding environment. However, these models often
struggle with occlusions, adverse weather, or extreme lighting conditions, and
when deployed in unfamiliar urban settings not included in their training
datasets, their reliability is compromised [49, 27].

Figure 1: Leveraging historical traversal data, our framework enhances online
perception through Neural Radiance Fields (NeRF), compressing extensive
observations into implicit fields. A ray-marching algorithm extracts
structured city-level priors, facilitating improved perception in subsequent
visits.

Consider the innate human capability to navigate through cities. Individuals
inherently develop preliminary mental maps of urban landscapes, with varying
degrees of accuracy. These priors serve as valuable supplements to real-time
sensory perception during subsequent explorations. Inspired by this human
capability, we propose a novel framework, PreSight, illustrated in Figure 1,
that utilizes historical traversal data—gathered through crowdsourcing or
personal journeys—to automatically generate static priors without manual
annotation efforts. These unsupervised, semantically and geometrically rich
priors significantly enhance the efficacy of online perception models in
subsequent navigations. Our framework includes city-scale neural radiance
fields (NeRF) [33] priors and a straightforward integration module for various
online perception models. Initially, we employ city-scale NeRFs to reconstruct
urban environments using camera images and poses captured from past
traversals, without requiring LiDAR input. Given the extensive spatial expanse
of these cities, we divide them into tiles, each encompassing approximately
$1~{}km^{2}$. For each tile, several Instant-NGP [34] sub-fields are deployed
and optimized in parallel for detailed coverage. Semantic information is
distilled into the NeRFs by leveraging features from pre-trained 2D visual
foundation models with semantic supervision. Structured priors are then
extracted using a ray marching algorithm inspired by EmerNeRF [47], selecting
occupied voxels and gathering their semantic features from the constructed
NeRFs. These featurized prior voxels serve as robust supplements to online
perception. To integrate with prevalent online models [23, 15, 26, 49], we
encode these priors into BEV or 3D features, using straightforward CNNs to
merge features from both priors and real-time observations. The details of
each part of our framework are provided in Section 3.

We validate our framework on the widely used nuScenes [4] dataset, focusing on
two important online perception tasks: local HD-map construction and occupancy
prediction. To illustrate our method’s efficacy in enhancing the adaptability
of online perception models to novel environments—and to avoid data leakage—we
re-split the training-validation set to eliminate any geometric overlaps. The
experimental results demonstrate that our framework significantly boosts the
performance of several perception models across both tasks. For an in-depth
discussion on the dataset and the experimental setup, please see Section 4.

To summarize, our contributions are as follows:

  * •

We introduce a novel framework, PreSight, that leverages historical data to
automatically construct powerful priors using NeRF.

  * •

We demonstrate that the generated priors can be seamlessly integrated into
various perception models and tasks, significantly enhancing their performance
with minimal modification.

##  2 Related Work

###  2.1 Conventional 3D Reconstruction Techniques

Various 3D reconstruction techniques have been developed to create 3D models
from 2D images. Structure from Motion (SfM) uses the relative motion between
different views in a set of 2D images to reconstruct 3D structures [9, 32]. It
is commonly applied for estimating camera positions and generating 3D point
clouds of scenes. However, SfM typically produces sparse and less accurate
geometric reconstructions. It also faces challenges in complex scenarios with
occlusions, where feature correspondences are difficult to establish and track
using hand-crafted rules [1, 38, 6]. Visual Simultaneous Localization and
Mapping (vSLAM) is regarded as a specialized variant of the SfM problem,
tailored for robotics and autonomous driving applications. By incorporating
pose information from GPS or IMU, vSLAM achieves more precise camera pose
estimation and thus, better reconstruction quality. Nonetheless, it encounters
similar limitations in dynamic, complex, and large-scale environments [2, 10,
11, 52]. Additionally, infusing semantic information into the reconstructed
models is not inherently straightforward with conventional 3D reconstruction
techniques.

###  2.2 Large-Scale Neural Radiance Fields

The application of Neural Radiance Field (NeRF) in urban scenarios has
attracted significant interest. Block NeRF [39] attempts to reconstruct city-
scale scenes using traditional MLP-based NeRF from densely captured camera
images, but suffers from prolonged training durations. SUDS [41] employs hash
grids [34] for faster training and divides large areas into multiple sub-
fields for detailed coverage. StreetSurf [12], focusing on static scenarios in
autonomous driving, delivers high-quality novel view synthesis and geometry
but overlooks dynamic objects. EmerNeRF [47] distinguishes between dynamic and
static elements in a self-supervised manner and leverages vision foundation
models for knowledge distillation. However, these approaches often depend on
LiDAR depth data or dense observational views, limiting scalability in new
regions via crowdsourcing. OccNeRF [50] and SelfOcc [17] explore using NeRF
for autonomous driving scene occupancy estimation but neglect dynamic entities
and underperform compared to online perception models, questioning their
suitability as ground truth. Our methodology, in contrast, focuses on
generating priors to enhance various online perception models.

###  2.3 Priors by Multi-Traversals

Recent work has explored on the nuScenes dataset [4] to enhance online
perception in autonomous driving using priors from previous traversals.
HindSight [48] utilizes LiDAR point clouds from past journeys to aid online
detection tasks and proposes to re-split the nuScenes dataset to provide
multi-traversal samples without geometrical overlaps, mitigating data leakage
risks. Yet, these historical point clouds lack semantic depth, narrowing their
utility. Neural Map Prior [45] employs neural representations to build and
update a global map prior, boosting online map perception. Nonetheless, the
substantial memory requirement for its neural representation limits
scalability. Both approaches, while innovative, are confined to particular
perception tasks and underscore the need for versatile priors that can support
a broader range of tasks.

###  2.4 Visual Online Perception Model

Visual online perception models leverage camera image input to predict the
surrounding environment in real-time, including a range of tasks from 3D
object detection [22, 16, 15, 46, 30, 24] to online HD map construction [49,
29, 20, 26, 8] and occupancy prediction [15, 23, 14]. Particularly, we focus
on online HD map construction and occupancy prediction. Online HD map
construction is crucial for autonomous navigation, utilizing onboard sensors
to create local HD maps. These maps can be vectorized [29, 26, 49] or
rasterized [35, 45], providing detailed semantic guidance to ensure vehicular
safety. This approach offers a cost-effective, scalable alternative to
traditional, manually annotated HD maps by minimizing the need for human
labor. The occupancy prediction task, pioneered by Occ3D [40], involves
predicting the occupancy and semantic label of each voxel in a scene from
camera images. This task delivers comprehensive information about the
vehicle’s surrounding environment but presents significant challenges in
achieving accurate geometric and semantic predictions.

##  3 Methodologies

Our framework leverages self-supervised Neural Radiance Fields (NeRFs) [33] to
generate static city priors from historical data and integrates these priors
into online perception models, as demonstrated in Figure 1. This section is
structured as follows: We begin by defining the problem in Section 3.1,
proceed to describe the construction of city-scale NeRFs fused with semantic
features in Section 3.2, introduce the extraction of structured priors in
Section 3.3, and conclude with our approach to integrating these priors with
state-of-the-art online perception models in Section 3.4.

###  3.1 Problem Formulation

Consider a set of training samples $X_{train}$ and their corresponding
annotations $Y_{train}$. Typically, an online perception model $M$ is trained
on the dataset
$\left(X_{train},Y_{train}\right)$. At deployment, this model is expected to
effectively generalize to unseen samples $X_{test}$, including those from
entirely new locations—a scenario where many advanced perception models
encounter difficulties. Suppose we also have access to a set of unlabeled
samples $X_{prior}$, which geographically overlap with $X_{test}$ and can be
readily acquired through crowdsourcing. Our objective is to effectively
utilize the unlabeled $X_{prior}$ to autonomously generate priors $P_{prior}$,
thereby enhancing the performance of model $M$ on $X_{test}$.

More specifically, during the training phase, we generate priors $P_{train}$
based on $X_{train}$. An enhanced perception model $M^{\prime}$, equipped with
an integration module, is then trained on the combined dataset
$\left(X_{train}+P_{train},Y_{train}\right)$. Before evaluation, we generate
priors $P_{prior}$ using the unlabeled $X_{prior}$. The model $M^{\prime}$ is
subsequently evaluated on the dataset
$\left(X_{test}+P_{prior},Y_{test}\right)$. It is crucial to note that
$X_{train}$ is geographically distinct from $X_{test}\cup X_{prior}$ to
prevent data leakage. For more details about the experimental setup, please
refer to Section 4.1.

###  3.2 Building City-Scale NeRF

Inputs. Our method begins by taking sequences of RGB images from videos and
their associated camera poses. To enhance these inputs, we utilize pre-trained
vision foundation models for generating additional descriptors on pixels.
Specifically, we employ DINO [5] for camera-view semantic feature generation
and SegFormer [44] for producing camera-view semantic segmentation masks.

Benefits of Using NeRF in Our Framework. Neural Radiance Fields (NeRF) [34]
excel at generating continuous, consistent, and high-fidelity 3D
representations from sparse and irregularly spaced images, setting them apart
from traditional 3D reconstruction techniques. Unlike methods that depend on
specialized inputs such as LiDAR data or stereo camera imagery, NeRF operates
effectively with standard 2D images. Moreover, NeRF’s volumetric rendering
approach allows for the straightforward exclusion of dynamic objects from
scenes and the integration of pixel-wise semantic features [47]. These
features can be extracted from powerful vision foundation models and
seamlessly incorporated into the 3D field. In our framework, we use NeRF not
merely as a tool for synthesizing novel views but as an efficient, self-
supervised learner of environmental priors. Our focus is on the quality of
geometric and semantic reconstructions rather than on the fidelity of novel
view generation, leveraging NeRF’s strengths to enhance the robustness and
accuracy of online perception models in static environment tasks.

(a) Our approach constructs distinct fields for nearby scenes and the sky,
utilizing video embeddings to accommodate variations in lighting conditions
across scenes. Semantic features are merged similar RGB color blending,
enabling the distillation of semantic knowledge from DINO [5].

(b) Multiple sub-fields are used to guarantee comprehensive coverage.
Volumetric rendering directly integrates sample points across these sub-
fields.

Figure 2: An overview of our NeRF’s scene representation.

Scene Representation. Given the vast spatial extent of input videos throughout
the city, we partition the city into tiles, each approximately $1~{}km^{2}$.
Each tile is independently optimized using video data overlapping with its
designated area. We apply a straightforward application of K-Means clustering
on all camera poses to determine tile boundaries.

For each tile, we construct a large static scene representation consisting of
multiple sub-fields ${\mathbf{F}^{i}}$ to ensure comprehensive coverage. The
query outputs are aggregated from these sub-fields along the ray during the
rendering process. Similar to the initial clustering, we employ K-Means
clustering on camera poses to establish centroids for these sub-fields. Each
sub-field mimics the structure found in Instant-NGP [34], utilizing multi-
resolution hash grids $\mathcal{H}$ coupled with a small MLP $g$ to encode the
implicit spatial features. Separate heads
$\phi_{\mathrm{color}},\phi_{\mathrm{feat}},\phi_{\mathrm{sky}}$ are utilized
for predictions of color, semantic information, and sky appearance,
respectively. The queried inputs include $(\mathbf{x},\mathbf{d},vid)$, where
$\mathbf{x}$ represents the 3D location, $\mathbf{d}$ the view direction, and
$vid$ the video identifier:

$\mathbf{g},\sigma=g\left(\mathcal{H}(\mathbf{x})\right)$ (1)

$\mathbf{g},\sigma$ represent the per-point feature and density, respectively.
For color predictions:

$\mathbf{c}=\phi_{\mathrm{color}}\left(\mathbf{g},\gamma(\mathbf{d}),\mathbf{V}(vid)\right)$
(2)

$\gamma(\cdot)$ denotes spherical harmonic encoding for direction, and
$\mathbf{V}(\cdot)$ represents per-video embeddings to account for video-
specific lighting conditions. For semantic feature predictions:

$\mathbf{f}=\phi_{\mathrm{feat}}\left(\mathbf{g}\right)$ (3)

This process models direction-invariant semantic features. Overall, the output
for each query is given by:

$\sigma,\mathbf{c},\mathbf{f}=\mathbf{F}(\mathbf{x},\mathbf{d},vid)$ (4)

For sky features, a position-independent MLP $\phi_{sky}$ predicts both color
and semantic features:

$\mathbf{c}_{\mathrm{sky}}(\mathbf{d}),\mathbf{f}_{\mathrm{sky}}(\mathbf{d})=\phi_{\mathrm{sky}}\left(\gamma(\mathbf{d}),\mathbf{V}(vid)\right)$
(5)

Rendering. To render a camera ray $\mathbf{r}=(\mathbf{o},\mathbf{d})$ within
a video $vid$, we sample $N$ points $\left\\{\mathbf{x}^{i}\right\\}$ along
the direction $\mathbf{d}$. Each sample point is assigned to the nearest sub-
field. The assignment process is defined as:

$\delta(\mathbf{x}^{i})=\mathop{\arg\min}\limits_{j}\left\|\mathbf{x}^{i}-\mathbf{centroid}^{j}\right\|_{2}$
(6)

where $\mathbf{centroid}^{j}$ denotes the centroid of $j$-th sub-field. After
assignment, outputs for each sample point are queried exclusively from the
assigned sub-field:

$\sigma^{i},\mathbf{c}^{i},\mathbf{f}^{i}=\mathbf{F}^{\delta(\mathbf{x}^{i})}(\mathbf{x}^{i},\mathbf{d},vid)$
(7)

The final rendering is an integration along the ray, blending color and
semantic features:

$\hat{C}(\mathbf{r})=\sum_{i=1}^{N}T^{i}\alpha^{i}\mathbf{c}^{i}+\left(1-\sum_{i=1}^{N}T^{i}\alpha^{i}\right)\
\mathbf{c}_{\mathrm{sky}}(\mathbf{d})$ (8)
$\hat{F}(\mathbf{r})=\sum_{i=1}^{N}T^{i}\alpha^{i}\mathbf{f}^{i}+\left(1-\sum_{i=1}^{N}T^{i}\alpha^{i}\right)\
\mathbf{f}_{\mathrm{sky}}(\mathbf{d})$ (9)

$\alpha^{i}$ and $T^{i}$ denote the accumulated transmittance and piece-wise
opacity, respectively, calculated as follows:

$\alpha^{i}=1-\exp\left(-\sigma^{i}(\mathbf{x}^{i+1}-\mathbf{x}^{i})\right)$
(10) $T^{i}=\prod_{j=1}^{i-1}(1-\alpha^{j})$ (11)

Optimization. We employ an L2 loss for color $\mathcal{L}_{\mathrm{rgb}}$ and
semantic feature reconstruction $\mathcal{L}_{\mathrm{feat}}$, along with a
binary cross-entropy loss for sky features $\mathcal{L}_{\mathrm{sky}}$
following approach introduced in [47, 12]. Inspired by Mip-NeRF 360 [3], we
use two proposal networks, integrating interlevel and distortion losses. The
comprehensive loss function is represented as:

$\mathcal{L}=\mathcal{L}_{\mathrm{rgb}}+\lambda_{\mathrm{feat}}\mathcal{L}_{\mathrm{feat}}+\lambda_{\mathrm{sky}}\mathcal{L}_{\mathrm{sky}}+\lambda_{\mathrm{inter}}\mathcal{L}_{\mathrm{inter}}+\lambda_{\mathrm{dist}}\mathcal{L}_{\mathrm{dist}}$
(12)

To focus on static environment features, we mask out pixels associated with
potentially moving objects, such as vehicles and pedestrians, using
segmentation masks derived from SegFormer [44].

###  3.3 Prior Extraction

After the optimization phase, the NeRFs encapsulate rich information within
their hash grids and MLPs. To make this dense, unstructured information
inherent in NeRFs usable for online perception models, it requires conversion
into a structured form. We employ a ray marching algorithm to identify the
voxels that are occupied and to capture their associated features. The process
begins by casting rays originating from the training cameras. Similar to the
training procedure, we sample $N$ sample points
$\left\\{\mathbf{x}^{i}\right\\}$ along each ray. The surface point is
determined by locating the first point where the cumulative transmittance and
opacity exceed a threshold:

$j=\min_{i}\left(\sum_{i=1}^{N}T^{i}\alpha^{i}>0.5\right)$ (13)

Here, $T^{i}$ and $\alpha^{i}$ are defined in Equation 10 and Equation 11. We
collect the identified surface point $\mathbf{x}^{j}$ and its corresponding
semantic feature $\phi_{\mathrm{feat}}\left(\mathbf{g}\right)$, as previously
defined in Equation 3. Following the aggregation of surface points from all
training views, we apply voxel-based downsampling to minimize the points,
averaging the features within each voxel. These feature-enriched voxels then
act as robust priors for enhancing online perception models.

Figure 3: Point-based priors are voxelized and then encoded into BEV or 3D
features using convolutional layers. These features are fused with online
features from perception models, enhancing the overall features for decoding.

###  3.4 Integrating Priors to Online Models

Online perception models commonly transform image inputs into a canonical
representation, such as BEV or 3D space features. To ensure our framework’s
broad compatibility, we adapt our pre-built priors to BEV or 3D space,
simplifying their integration with various state-of-the-art online perception
models [23, 15, 49, 26]. As illustrated in Figure 3, our integration strategy
employs several simple CNN layers to transform the voxel features into BEV or
3D space, which are then concatenated with the existing online features from
the perception models. This module introduces negligible computational
overhead and requires no alterations to the original model architectures,
efficiently enhancing the models with rich environmental priors for improved
real-time perception capabilities.

##  4 Experiments

In this section, we evaluate our Pre-Sight framework on the nuScenes [4]
dataset, specifically targeting perception tasks involving static targets:
online HD map construction and occupancy prediction. These tasks are chosen
because our framework is designed to leverage priors to enhance static
environment perception, a crucial part for these applications. The dataset are
introduced in Section 4.1, followed by NeRF implementation details in Section
4.2. Section 4.3 showcases our framework’s enhancements to baseline models.
Our priors are compared to alternatives in Section 4.4, and Section 4.5
presents some ablation studies of our methods.

###  4.1 Setup.

We evaluate our framework using the nuScenes [4] dataset, a leading benchmark
in autonomous driving, collected across four regions: Boston Seaport,
Singapore Onenorth, Singapore Queenstown, and Singapore Holland Village,
featuring 1000 scenes with six synchronized cameras and vehicle poses. The
official dataset split allocates 700 scenes for training and 150 for
validation. However, recent studies have identified data leakage issues within
this official split, highlighting the need for a revised partitioning to
accurately assess static perception models’ generalization capabilities.
Following our problem setup in Section 3.1, we assign scenes from Boston
Seaport and Singapore Queenstown to our training set
$\left(X_{train},Y_{train}\right)$. For validation, one-third of the scenes
from Singapore Onenorth and Singapore Holland Village are used as $X_{prior}$
for generating priors, with the remaining two-thirds forming the validation
set $\left(X_{test},Y_{test}\right)$. Therefore, following the problem setup
detailed in Section 3.1, we set all samples in Boston Seaport and Singapore
Queenstown as training set $\left(X_{train},Y_{train}\right)$. This
distribution strategy mirrors the practical scenario of scaling and
generalizing an autonomous driving system: training is conducted using data
from specific cities, followed by deployment in previously unencountered
cities, supported by a limited set of observations $X_{prior}$ sourced through
crowdsourcing. Overall, we established a split of 614 training scenes, 82
prior scenes, and 154 testing scenes, with an $80\%$ geographical overlap
between the prior and testing sets. Further details on scene allocation can be
found in the supplemental material.

###  4.2 NeRF Implementation

This section introduces the implementation details for constructing city-scale
Neural Radiance Fields (NeRFs). We utilize 12Hz image sequences from six
cameras provided by the nuScenes dataset, yielding approximately 1,400 images
for each 20-second scene. following the approach of EmerNeRf[47], we compress
the dimension of DINO features from 764 to 64. Dynamic objects including
person, rider, car, truck, bus, train, motorcycle, and bicycle are masked
using segmentation results from SegFormer[44]. As detailed in Section 3.2,
urban areas are divided into tiles of roughly $1~{}km^{2}$. Specifically, we
assign 8, 4, 4, and 2 tiles to Boston Seaport, Onenorth, Queenstown, and
Holland Village, respectively, based on their sizes. Each tile contains 8 or
16 sub-fields, depending on the tile’s area. NeRF training for each tile
involves 100,000 iterations with a batch size of $2^{16}$, requiring about 20
hours on a single A100 GPU.

###  4.3 Experimental Results on Online models

Table 1: Our framework’s enhancements across different online HD map
construction models: StreamMapNet and MapTR use vectorized predictions,
evaluated by Average Precision (AP), while BEVFormer’s rasterized predictions
are assessed using Intersection over Union (IoU).

Model | Metric | w. Prior | Ped Crossing | Divider | Boundary | All | #Param. (M)
---|---|---|---|---|---|---|---
StreamMapNet | AP | $\times$ | 10.19 | 11.26 | 11.87 | 11.10 | 57.7
✓ | 21.11 | 23.73 | 32.31 | 25.72 (+14.62) | 58.7
MapTR | AP | $\times$ | 4.97 | 8.20 | 9.83 | 7.67 | 36.3
✓ | 16.18 | 19.04 | 34.14 | 23.12 (+15.45) | 37.3
BEVFormer | IoU | $\times$ | 14.90 | 29.88 | 32.74 | 25.84 | 30.9
✓ | 16.37 | 34.82 | 51.66 | 34.28 (+8.44) | 31.9

Online HD Map construction. Online HD maps construction takes as input camera
images to generate local HD maps, which can be either vectorized [26, 49] or
rasterized [45]. Our research demonstrates that the integration of proposed
priors enhances the performance of state-of-the-art models. Specifically, for
vectorized mapping, we employed two models with distinct decoders, namely
MapTR [26] and StreamMapNet [49]. For rasterized mapping, we adopted a
BEVFormer-based [22] model, following Neural Map Prior [45]. These models
utilize BEV features as an intermediate representation, which are fused with
our pre-built priors using our integration module. We trained and evaluated
each model based on their original codebase, with our proposed training-
testing split. The perception range was set to $100\times 50$ meters, aligning
with StreamMapNet’s specifications. Given that map construction is inherently
a static perception task, our framework significantly elevates the quality of
the resulting HD maps across all models. This underscores our framework’s
effectiveness and adaptability, as shown in Table 1. It’s worth noting that
the performance metrics for the baseline models are lower than those
originally reported in their respective papers. This discrepancy is due to our
use of a more challenging dataset split without geographical overlaps, a
phenomenon similarly observed by Lilja _et al_. [27] and Yuan _et al_. [49]
More implementation details are provided in the supplemental materials.

Table 2: Our framework boosts state-of-the-art occupancy prediction models,
distinguishing between dynamic and static targets for clarity. The trailer
class is excluded due to its absence in our proposed testing set.

|  |  |  |  | Dynamic | Static |
---|---|---|---|---|---|---|---
Method |  w. Priors |  mIoU |  Dynamic |  Static |  others |  barrier |  bicycle |  bus |  car |  constr. vehicle |  motorcycle |  pedestrian |  traffic cone |  truck |  drive surface |  other flat |  sidewalk |  terrain |  manmade |  vegetation |  #Param. (M)
BEVDet | $\times$ | 29.3 | 24.4 | 38.2 | 1.5 | 42.4 | 11.0 | 43.0 | 47.1 | 19.1 | 23.3 | 23.4 | 19.5 | 37.8 | 72.9 | 11.6 | 30.9 | 48.6 | 32.7 | 32.5 | 35.0
✓ | 33.7 | 24.4 | 50.5 | 1.2 | 40.1 | 14.8 | 42.1 | 48.3 | 15.7 | 26.4 | 24.4 | 18.7 | 37.2 | 81.8 | 15.2 | 40.3 | 60.5 | 50.4 | 54.9 | 36.0
FB-Occ | $\times$ | 30.0 | 25.1 | 39.2 | 9.2 | 37.2 | 21.8 | 41.6 | 43.4 | 15.8 | 27.3 | 25.4 | 23.8 | 30.3 | 74.7 | 17.3 | 33.0 | 50.6 | 28.2 | 31.1 | 68.5
✓ | 34.3 | 25.4 | 50.7 | 9.3 | 38.3 | 21.0 | 40.3 | 45.0 | 15.9 | 29.9 | 26.0 | 23.8 | 30.2 | 82.3 | 18.5 | 39.1 | 61.2 | 48.0 | 54.7 | 69.5

3D Occupancy Prediction. 3D occupancy prediction task involves estimating both
the occupancy state and semantic label of each voxel in a scene, using camera
images as input. We benchmark our approach using the widely recognized Occ3D
dataset [40] on nuScenes. For baselines, we selected FB-OCC [23] and BEVDet
[15], two leading state-of-the-art models with distinct architectures,
integrating our pre-built priors into their BEV features. Each model was
trained and evaluated on their original codebase, with our proposed training-
testing split. The results, presented in Table 2, demonstrate our framework’s
ability to enhance the baseline models’ performance. Notably, we specifically
highlight static categories such as drive surface, other flat, sidewalk,
terrain, manmade, and vegetation. Our framework significantly improves outputs
for these targets, without impacting dynamic categories, aligning with our
goal of enhancing static perception accuracy.

###  4.4 Comparison with Other Priors

This section compares our generated priors with alternative approaches to
evaluate their effectiveness.

Historical Point Clouds. A common method for utilizing historical data
involves collecting LiDAR-generated point clouds from previous traversals
[48]. Despite our framework’s independence from LiDAR, we assessed whether our
priors could outperform these point clouds, known for their geometric
precision. We replaced our priors with historical LiDAR point clouds to
evaluate performance in HD map construction and occupancy prediction tasks.

Neural Map Prior. Neural Map Prior [45] enhances online map perception models
with automatically generated priors, aligning with our objectives. We adapted
this method to our experimental setup by training it with our dataset
$\left(X_{train},Y_{train}\right)$ and generating neural priors from
$X_{prior}$ for evaluation on
$\left(X_{test},Y_{test}\right)$, making best use of its framework’s
potential. This comparison conducted on rasterized map construction task,
following their official implementation.

Table 3: Our method outperforms other priors in HD map construction and
achieves similar results to LiDAR priors in occupancy prediction, using only
camera inputs.

(a) HD Map Construction

Model | Metric | Prior Type | mAP/mIoU
---|---|---|---
StreamMapNet | AP | $\times$ | 11.10
LiDAR | 16.87
ours | 25.72
MapTR | AP | $\times$ | 7.67
LiDAR | 15.93
ours | 23.12
BEVFormer | IoU | $\times$ | 25.84
LiDAR | 33.01
NMP | 26.50
ours | 34.42

(b) Occupancy Prediction

Model | Prior Type | mIoU | Dynamic | Static
---|---|---|---|---
BEVDet | $\times$ | 29.3 | 24.4 | 38.2
LiDAR | 33.1 | 21.7 | 53.9
ours | 33.7 | 24.4 | 50.5
FB-OCC | $\times$ | 30.0 | 25.1 | 39.2
LiDAR | 34.3 | 23.4 | 54.4
ours | 34.3 | 25.4 | 50.7

The results, detailed in Table 3, underscore our framework’s superiority in
both tested perception tasks. For HD map construction, our priors proved more
effective, offering richer semantic details. In occupancy prediction, our
approach achieved results on par with historical LiDAR point clouds. While
LiDAR provide a slight advantage in enhancing static target perception, they
fail to differentiate between dynamic and static objects, introducing
potential inaccuracies. Conversely, our method maintains performance without
adversely affecting dynamic object perception.

###  4.5 Ablation Studies

This section explores the impact of various components of our framework on the
final outcomes. Specifically, we investigate the role of distilled DINO [5]
features and the effect of varying the proportion of $X_{prior}$ utilized for
constructing priors.

Table 4: Ablation study reveals the impact of distilled semantic features:
Priors enhance online perception on their own, but performance significantly
improves with the addition of distilled features.

(a) HD Map Construction

Model | Prior Type | mAP
---|---|---
StreamMapNet | $\times$ | 11.10
no feats. | 11.45
full | 25.72

(b) Occupancy Prediction

Model | Prior Type | mIoU | Dynamic | Static
---|---|---|---|---
BEVDet | $\times$ | 29.3 | 24.4 | 38.2
no feats. | 32.0 | 24.1 | 46.4
full | 33.7 | 24.4 | 50.5

Distilled DINO Features. A crucial element of our framework is incorporating
distilled semantic knowledge from the robust vision foundation model, DINO,
into our priors. As demonstrated in Table 4, removing these distilled semantic
features leads to a notable decline in performance for both tasks,
highlighting the importance of integrating distilled semantic knowledge.
Particularly, HD map construction benefits greatly from semantic features for
identifying map elements, whereas occupancy prediction depends more on
geometrical information.

Table 5: Performance improves as as we scales up priors scenes. $\times$
represents baseline models where no priors are incorporated during either
training or testing phases. $0\%$ denotes that priors are not used only during
testing, whereas $50\%$ indicates that priors are generated using only half of
the scenes in $X_{prior}$ for testing. A $100\%$ ratio means all scenes in
$X_{prior}$ are used.

(a) HD Map Construction

Model | Prior Type | mAP
---|---|---
StreamMapNet | $\times$ | 11.10
$0\%$ | 12.03
$50\%$ | 21.62
| $\mathbf{100\%}$ | 25.72

(b) Occupancy Prediction

Model | Prior Type | mIoU | Dynamic | Static
---|---|---|---|---
BEVDet | $\times$ | 29.3 | 24.4 | 38.2
$0\%$ | 28.7 | 24.6 | 36.2
$50\%$ | 32.0 | 24.1 | 46.4
| $\mathbf{100\%}$ | 33.7 | 24.4 | 50.5

Proportions of Used Prior Scenes. The effectiveness of our framework increases
with the amount of historical traversal data utilized. To illustrate this, we
evaluated the trained model using varying proportions of prior scenes. The
outcomes are summarized in Table 5. Performance improves as more scenes are
utilized to create priors, showcasing our framework’s scalability.
Importantly, models trained with priors do not depend exclusively on them and
still achieve performance comparable to baseline models with no priors at test
time.

Figure 4: Leveraging priors, the enhanced model accurately predicts road
structure, while baseline model without priors fails.

###  4.6 Qualitative Analysis

In this section, we present qualitative results from our framework,
underscoring the value of incorporating priors. Figure 4 depicts a low-light
scenario at an intersection that is hardly visible in the front-view image.
Here, the baseline model without priors misinterprets the road layout, whereas
the model equipped with priors accurately reconstructs it. The visualization
of prior features reveals the detailed road structure, underscoring how priors
provide crucial information for generating consistent and reliable HD maps
critical for autonomous vehicle safety. Figure 5 highlights the enhancement in
occupancy prediction provided by our priors. Notably, within the highlighted
region, our priors accurately capture the detailed structure of a light pole
and the ground surface—details even absent in the ground truth. Consequently,
the model with priors precisely predicts occupancy with fine details, in
contrast to the baseline model’s failure.

Figure 5: Self-supervised priors reconstruct fine-grained geometries even more
denser than groundtruth, enabling the enhanced model to deliver more precise
occupancy predictions than baseline models lacking priors.

##  5 Dicussion and Conclusion

Limitations. Our approach depends on accurate ego-vehicle poses and camera
sensors, typically provided by GPS and IMU systems, which might not be readily
available in crowdsourced data. Future efforts could focus on improving
localization and pose estimation, or exploring alternative self-supervised
reconstruction methods like 3D Gaussian Splatting[19], offering promising
directions for further research.

Conclusion. In this paper, we introduced PreSight, a novel framework that
autonomously constructs static, city-scale priors for enhancing online
perception in subsequent traversals. Our experiments on the nuScenes dataset
demonstrate the framework’s effectiveness and its wide applicability across
various perception models.

## References

  * [1] Arie-Nachimson, M., Kovalsky, S.Z., Kemelmacher-Shlizerman, I., Singer, A., Basri, R.: Global motion estimation from point matches. In: 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission. pp. 81–88 (2012). https://doi.org/10.1109/3DIMPVT.2012.46
  * [2] Aulinas, J., Petillot, Y., Salvi, J., Llado, X.: The slam problem: a survey. vol. 184, pp. 363–371 (01 2008). https://doi.org/10.3233/978-1-58603-925-7-363
  * [3] Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P., Hedman, P.: Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR (2022)
  * [4] Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., Beijbom, O.: nuscenes: A multimodal dataset for autonomous driving. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11621–11631 (2020)
  * [5] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the International Conference on Computer Vision (ICCV) (2021)
  * [6] Crandall, D., Owens, A., Snavely, N., Huttenlocher, D.: Discrete-continuous optimization for large-scale structure from motion. In: CVPR 2011. pp. 3001–3008 (2011). https://doi.org/10.1109/CVPR.2011.5995626
  * [7] Dellaert, F.: Factor graphs and gtsam: A hands-on introduction. Tech. rep., Georgia Institute of Technology (2012)
  * [8] Ding, W., Qiao, L., Qiu, X., Zhang, C.: Pivotnet: Vectorized pivot learning for end-to-end hd map construction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3672–3682 (2023)
  * [9] Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (jun 1981). https://doi.org/10.1145/358669.358692, https://doi.org/10.1145/358669.358692
  * [10] Fuentes-Pacheco, J., Ascencio, J., Rendon-Mancha, J.: Visual simultaneous localization and mapping: A survey. Artificial Intelligence Review 43 (11 2015). https://doi.org/10.1007/s10462-012-9365-8
  * [11] Goldstein, T., Hand, P., Lee, C., Voroninski, V., Soatto, S.: Shapefit and shapekick for robust, scalable structure from motion (08 2016)
  * [12] Guo, J., Deng, N., Li, X., Bai, Y., Shi, B., Wang, C., Ding, C., Wang, D., Li, Y.: Streetsurf: Extending multi-view implicit surface reconstruction to street views. arXiv preprint arXiv:2306.04988 (2023)
  * [13] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [14] Hong, Y., Liu, Q., Cheng, H., Ma, D., Dai, H., Wang, Y., Cao, G., Ding, Y.: Univision: A unified framework for vision-centric 3d perception (2024)
  * [15] Huang, J., Huang, G.: Bevdet4d: Exploit temporal cues in multi-camera 3d object detection. arXiv preprint arXiv:2203.17054 (2022)
  * [16] Huang, J., Huang, G., Zhu, Z., Yun, Y., Du, D.: Bevdet: High-performance multi-camera 3d object detection in bird-eye-view. arXiv preprint arXiv:2112.11790 (2021)
  * [17] Huang, Y., Zheng, W., Zhang, B., Zhou, J., Lu, J.: Selfocc: Self-supervised vision-based 3d occupancy prediction. arXiv preprint arXiv:2311.12754 (2023)
  * [18] Huang, Y., Zheng, W., Zhang, Y., Zhou, J., Lu, J.: Tri-perspective view for vision-based 3d semantic occupancy prediction. arXiv preprint arXiv:2302.07817 (2023)
  * [19] Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics 42(4) (July 2023), https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
  * [20] Li, Q., Wang, Y., Wang, Y., Zhao, H.: Hdmapnet: A local semantic map learning and evaluation framework. arXiv preprint arXiv:2107.06307 (2021)
  * [21] Li, Y., Yu, Z., Choy, C., Xiao, C., Alvarez, J.M., Fidler, S., Feng, C., Anandkumar, A.: Voxformer: Sparse voxel transformer for camera-based 3d semantic scene completion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
  * [22] Li, Z., Wang, W., Li, H., Xie, E., Sima, C., Lu, T., Qiao, Y., Dai, J.: Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. arXiv preprint arXiv:2203.17270 (2022)
  * [23] Li, Z., Yu, Z., Austin, D., Fang, M., Lan, S., Kautz, J., Alvarez, J.M.: FB-OCC: 3D occupancy prediction based on forward-backward view transformation. arXiv:2307.01492 (2023)
  * [24] Li, Z., Yu, Z., Wang, W., Anandkumar, A., Lu, T., Alvarez, J.M.: FB-BEV: BEV representation from forward-backward view transformations. In: IEEE/CVF International Conference on Computer Vision (ICCV) (2023)
  * [25] Li, Z., Zhang, C., Ma, W.C., Zhou, Y., Huang, L., Wang, H., Lim, S., Zhao, H.: Voxelformer: Bird’s-eye-view feature generation based on dual-view attention for multi-view 3d object detection (2023)
  * [26] Liao, B., Chen, S., Wang, X., Cheng, T., Zhang, Q., Liu, W., Huang, C.: Maptr: Structured modeling and learning for online vectorized hd map construction. In: International Conference on Learning Representations (2023)
  * [27] Lilja, A., Fu, J., Stenborg, E., Hammarstrand, L.: Localization is all you evaluate: Data leakage in online mapping datasets and how to fix it (2023)
  * [28] Lin, X., Lin, T., Pei, Z., Huang, L., Su, Z.: Sparse4d v2: Recurrent temporal fusion with sparse model (2023)
  * [29] Liu, Y., Yuan, T., Wang, Y., Wang, Y., Zhao, H.: Vectormapnet: End-to-end vectorized hd map learning (2023)
  * [30] Liu, Z., Tang, H., Amini, A., Yang, X., Mao, H., Rus, D., Han, S.: Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation. In: IEEE International Conference on Robotics and Automation (ICRA) (2023)
  * [31] Loshchilov, I., Hutter, F.: Fixing weight decay regularization in adam (2018)
  * [32] Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International journal of computer vision 60, 91–110 (2004)
  * [33] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. In: ECCV (2020)
  * [34] Müller, T., Evans, A., Schied, C., Keller, A.: Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. 41(4), 102:1–102:15 (Jul 2022). https://doi.org/10.1145/3528223.3530127, https://doi.org/10.1145/3528223.3530127
  * [35] Roddick, T., Cipolla, R.: Predicting semantic map representations from images using pyramid occupancy networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11138–11147 (2020)
  * [36] Segal, A., Hähnel, D., Thrun, S.: Generalized-icp. In: Robotics: science and systems (06 2009). https://doi.org/10.15607/RSS.2009.V.021
  * [37] Shan, T., Englot, B.: Lego-loam: Lightweight and ground-optimized lidar odometry and mapping on variable terrain. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 4758–4765. IEEE (2018)
  * [38] Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: exploring photo collections in 3D. Association for Computing Machinery, New York, NY, USA, 1 edn. (2023), https://doi.org/10.1145/3596711.3596766
  * [39] Tancik, M., Casser, V., Yan, X., Pradhan, S., Mildenhall, B., Srinivasan, P.P., Barron, J.T., Kretzschmar, H.: Block-nerf: Scalable large scene neural view synthesis (Feb 2022), http://arxiv.org/abs/2202.05263v1
  * [40] Tian, X., Jiang, T., Yun, L., Wang, Y., Wang, Y., Zhao, H.: Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving. arXiv preprint arXiv:2304.14365 (2023)
  * [41] Turki, H., Zhang, J.Y., Ferroni, F., Ramanan, D.: Suds: Scalable urban dynamic scenes. In: Computer Vision and Pattern Recognition (CVPR) (2023)
  * [42] Wang, T., Zhu, X., Pang, J., Lin, D.: FCOS3D: Fully convolutional one-stage monocular 3d object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops (2021)
  * [43] Wang, Y., Guizilini, V.C., Zhang, T., Wang, Y., Zhao, H., Solomon, J.: Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In: Conference on Robot Learning. pp. 180–191. PMLR (2022)
  * [44] Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: Segformer: Simple and efficient design for semantic segmentation with transformers. In: Neural Information Processing Systems (NeurIPS) (2021)
  * [45] Xiong, X., Liu, Y., Yuan, T., Wang, Y., Wang, Y., Hang, Z.: Neural map prior for autonomous driving. In: CVPR (2023)
  * [46] Yang, C., Chen, Y., Tian, H., Tao, C., Zhu, X., Zhang, Z., Huang, G., Li, H., Qiao, Y., Lu, L., Zhou, J., Dai, J.: Bevformer v2: Adapting modern image backbones to bird’s-eye-view recognition via perspective supervision. ArXiv (2022)
  * [47] Yang, J., Ivanovic, B., Litany, O., Weng, X., Kim, S.W., Li, B., Che, T., Xu, D., Fidler, S., Pavone, M., Wang, Y.: Emernerf: Emergent spatial-temporal scene decomposition via self-supervision. arXiv preprint arXiv:2311.02077 (2023)
  * [48] You, Y., Luo, K.Z., Chen, X., Chen, J., Chao, W.L., Sun, W., Hariharan, B., Campbell, M., Weinberger, K.Q.: Hindsight is 20/20: Leveraging past traversals to aid 3d perception. arXiv preprint arXiv:2203.11405 (2022)
  * [49] Yuan, T., Liu, Y., Wang, Y., Wang, Y., Zhao, H.: Streammapnet: Streaming mapping network for vectorized online hd map construction. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 7356–7365 (January 2024)
  * [50] Zhang, C., Yan, J., Wei, Y., Li, J., Liu, L., Tang, Y., Duan, Y., Lu, J.: Occnerf: Self-supervised multi-camera occupancy prediction with neural radiance fields. arXiv preprint arXiv:2312.09243 (2023)
  * [51] Zhang, J., Singh, S.: Loam : Lidar odometry and mapping in real-time. Robotics: Science and Systems Conference (RSS) pp. 109–111 (01 2014)
  * [52] Özyeşil, O., Voroninski, V., Basri, R., Singer, A.: A survey of structure from motion. Acta Numerica 26, 305–364 (2017). https://doi.org/10.1017/S096249291700006X

##  Appendix 0.A Dataset

As detailed in Section 4.1, we divide the nuScenes dataset into training,
prior, and testing segments. Scenes from Boston Seaport and Singapore
Queenstown comprise the training set, while Singapore Onenorth and Singapore
Holland Village contribute to the prior and testing sets. To maintain an
balanced distribution of day and night scenes, a subset of Singapore Holland
Village, including 32 night scenes, is allocated to the training set. This
results in a division of 65 night scenes for training and 34 for testing, with
night scenes excluded from prior construction. The distribution between prior
and testing sets aims to balance overlap and scene diversity, yielding 19/34
scenes for prior/testing in Singapore Holland Village and a 63/120 split in
Singapore Onenorth. Here we list scene names of prior and testing set for
reproduction.

testing scenes: 0001, 0003, 0005, 0006, 0007, 0008, 0011, 0012,
0014, 0015, 0016, 0018, 0021, 0022, 0023, 0025, 0028, 0029, 0034,
0043, 0045, 0046, 0051, 0054, 0059, 0060, 0061, 0120, 0124, 0126,
0127, 0130, 0131, 0132, 0134, 0135, 0138, 0150, 0151, 0152, 0154,
0155, 0157, 0158, 0159, 0160, 0190, 0191, 0192, 0193, 0194, 0221,
0270, 0271, 0272, 0273, 0274, 0275, 0276, 0277, 0278, 0344, 0345,
0346, 0347, 0355, 0356, 0357, 0358, 0359, 0361, 0362, 0363, 0364,
0365, 0366, 0367, 0368, 0371, 0372, 0373, 0374, 0375, 0376, 0377,
0379, 0380, 0381, 0382, 0383, 0385, 0945, 0949, 0952, 0953, 0956,
0957, 0958, 0959, 0960, 0961, 0963, 0966, 0967, 0968, 0969, 0971,
0975, 0976, 0977, 0978, 0979, 0980, 0981, 0982, 0983, 0984, 0989,
0990, 0991, 1045, 1046, 1047, 1048, 1049, 1050, 1051, 1052, 1053,
1054, 1055, 1056, 1057, 1058, 1059, 1060, 1061, 1062, 1063, 1064,
1065, 1066, 1067, 1068, 1069, 1070, 1071, 1072, 1073, 1074, 1075,
1076, 1077, 1104.
prior scenes: 0002, 0004, 0009, 0010, 0013, 0017, 0019, 0020,
0024, 0026, 0027, 0030, 0031, 0032, 0033, 0035, 0036, 0038, 0039,
0041, 0042, 0044, 0047, 0048, 0049, 0050, 0052, 0053, 0055, 0056,
0057, 0058, 0121, 0122, 0123, 0125, 0128, 0129, 0133, 0139, 0149,
0195, 0196, 0268, 0269, 0348, 0349, 0350, 0351, 0352, 0353, 0354,
0360, 0369, 0370, 0378, 0384, 0386, 0399, 0400, 0401, 0402, 0403,
0405, 0406, 0407, 0408, 0410, 0411, 0412, 0413, 0414, 0415, 0416,
0417, 0418, 0419, 0947, 0955, 0962, 0972, 0988.

##  Appendix 0.B NeRF Implementation

Field Details. Following the methodology of Mip-NeRF 360 [3], our
implementation adopts a two-stage proposal network for importance sampling,
utilizing 128 and 64 sample points in the first and second stages,
respectively. The proposal networks operate across resolutions from $2^{4}$ to
$2^{10}$, featuring a maximum hash map capacity of $2^{20}$. These networks
are structured into 8 levels, with each level encoding a 1-dimensional
feature. For the main field, we define resolutions ranging from $2^{4}$ to
$2^{14}$, maintaining the same maximum hash map capacity of $2^{20}$ but
expanding to 10 levels, each with a 4-dimensional feature. All Multi-Layer
Perceptrons (MLPs) in our framework consist of two hidden layers and are
configured with a hidden dimension of 64.

Optimization. The comprehensive loss function is represented as:

$\mathcal{L}=\mathcal{L}_{\mathrm{rgb}}+\lambda_{\mathrm{feat}}\mathcal{L}_{\mathrm{feat}}+\lambda_{\mathrm{sky}}\mathcal{L}_{\mathrm{sky}}+\lambda_{\mathrm{inter}}\mathcal{L}_{\mathrm{inter}}+\lambda_{\mathrm{dist}}\mathcal{L}_{\mathrm{dist}}$
(14)

We set the weights for $\lambda_{\mathrm{feat}}$, $\lambda_{\mathrm{sky}}$,
$\lambda_{\mathrm{inter}}$, and $\lambda_{\mathrm{dist}}$ at 0.5, 0.001, 1.0,
and 0.002, respectively. For optimization, we employ the AdamW optimizer [31]
with an initial learning rate of 0.01, applying a decay factor of 0.33 at the
$25\%$, $50\%$, and $75\%$ marks of the total iterations.

##  Appendix 0.C Baseline Implementation

This section outlines the implementation details for the baseline perception
models, closely following their original codebases.

HD Map Construction. Following MapTR [26] and StreamMapNet [49], we adjust
image resolution to $450\times 800$ and use ResNet50 [13] as the backbone. We
use 24 training epochs, with learning rates set at $5\times 10^{-4}$ for MapTR
and $6\times 10^{-4}$ for StreamMapNet, both with a batch size of 32,
following their standard implementation. For the BEVFormer-based rasterized
model, we use a batch size of 8 and a learning rate of $2\times 10^{-4}$,
consistent with Neural Map Prior [45]. Historical frames are not used for
simplicity.

Occupancy Prediction. Following FB-OCC [23], we resize images to $256\times
704$ and use ResNet50 as the image backbone. The models are trained for 24
epochs with BEV data augmentation. The learning rates are $2\times 10^{-4}$
for a batch size of 64 for FB-OCC and $1\times 10^{-4}$ for a batch size of 32
for BEVDet [15], in line with their official setups. Historical frames are
incorporated to align with the original implementations.


# Understanding the Clean Interface Between Covalent Si and Ionic Al2O3

H. J. Xiang  National Renewable Energy Laboratory, Golden, Colorado 80401, USA
Juarez L. F. Da Silva  National Renewable Energy Laboratory, Golden, Colorado
80401, USA  Howard M. Branz  National Renewable Energy Laboratory, Golden,
Colorado 80401, USA  Su-Huai Wei  National Renewable Energy Laboratory,
Golden, Colorado 80401, USA

###### Abstract

The atomic and electronic structures of the (001)-Si/(001)-$\gamma$-Al2O3
heterointerface are investigated by first principles total energy calculations
combined with a newly developed “modified basin hopping” method. It is found
that all interface Si atoms are 4-fold coordinated due to the formation of
Si-O and unexpected covalent Si-Al bonds in the new abrupt interface model.
And the interface has perfect electronic properties in that the unpassivated
interface has a large LDA band gap and no gap levels. These results show that
it is possible to have clean semiconductor/oxide interfaces.

###### pacs:

68.35.-p,73.20.-r,71.15.Nc,02.70.Uu

Interfaces between semiconductors and metal oxides are playing increasingly
important roles in advanced material science Forst2004 ; McKee1998 ; McKee2003
. In order to continue scaling electronic devices, a change from SiO2 (with a
dielectric constant $k$ about $3.9$) to high-$k$ oxides has been proposed for
the gate dielectric in future generation metal oxide semiconductor (MOS)
technologies. The key considerations for high-$k$ gate dielectrics include
high dielectric constant, high band offsets (at least 1 eV) with respect to
silicon, thermal stability, and minimization of electrical defects in the
interface. In particular, the quality of the interface is important for both
carrier mobility and device stability. However, control of the interface to
the Si substrate remains a stubborn outstanding problem. For example, hafnium-
based amorphous oxides has a bulk dielectric constant of $k\sim 22$
Ceresoli2006 , but its integration into the MOS gate stack poses substantial
technological challenges Kingon2000 . Epitaxial growth of oxides could lead to
more abrupt oxide-Si interfaces and consequently could offer solutions for the
end of the roadmap. Indeed, single crystal $\gamma$-Al2O3 ($k\sim 11$) thin
films have been epitaxially grown by molecular beam epitaxy on Si(001)
substrates Merckling2006 . Hence, Al2O3 could be a good candidate to be used
directly as a gate oxide or as a thin buffer barrier when combined with
high-$k$ amorphous or epitaxial oxides.

In another context, there have been some efforts in developing high quality
crystalline silicon (c-Si) film on inexpensive foreign substrates such as
oxides to reduce the Si material cost for terrestrial photovoltaic (PV) cells
Teplin2006 . Previous attempts to grow single crystal Si on some oxides such
as CeO2 failed due to the formation of SiO2 Teplin2006 . Recently, Findikoglu
et al. Findikoglu2005  demonstrated the growth of well-oriented Si thin films
with high carrier mobility on $\gamma$-Al2O3 substrate. In addition, Al2O3 has
been shown to passivate the c-Si surface efficiently for PV applications
Hoex2006 . These results suggest $\gamma$-Al2O3 could be a good substrate for
c-Si solar cell growth. Therefore, detailed knowledge of the Si/Al2O3
interface are vital.

Although many experimental studies have examined the growth of (001)
$\gamma$-Al2O3 on the Si (001) surface ($\gamma$-Al2O3/Si), the detailed
interface structure remains unclear. Theoretically, Boulenc and Devos
Boulenc2006  proposed an interface model for (001)-$\gamma$-Al2O3 grown on
(001)-Si surface by incorporating a defective spinel model Menendez2005  of
$\gamma$-Al2O3. To obtain an interface without gap states, they introduced
passivating O atoms to replace Si-Al and Si-Si bonds with Si-O and Al-O bonds.
However, it is not clear if their proposed substrate and interface Boulenc2006
have the lowest total energies because only a few models were tested. It is
also not clear if a sharp gap-states free interface can exist because the
large chemical and size difference between the semiconductor and the oxide.
Therefore, it is desirable to obtain an improved microscopic understanding of
the atomic and electronic structures of this important Si/Al2O3 interface.

In this Letter, we develop a new modified basin-hopping (BH) method to search
for the lowest energy structure of the (001)-Si/(001)-$\gamma$-Al2O3
interface. It is found that the new interface structure presents not only Si-O
bonds, but also Si-Al bonds, with all Si atoms 4-fold coordinated. Our density
functional calculation shows that the interface is semiconducting with a
type-I band alignment. Our results support the use of $\gamma$-Al2O3 as a gate
oxide or a substrate for the c-Si growth.

Our density functional theory calculations employed the frozen-core projector
augmented wave method (PAW) PAW  encoded in the Vienna ab initio simulation
package VASP , and the local density approximation (LDA). We use a plane-wave
cutoff energy of 400 eV, except for the search of interface structures by the
BH method, where we use a soft O PAW pseudopotential with a cutoff energy of
212 eV.

As a prerequisite to build the Si/$\gamma$-Al2O3 interface, understanding the
$\gamma$-Al2O3 structure is necessary. Here, we adopt the bulk model
constructed by Krokidis et al. Krokidis2001  (We hereafter refer to this bulk
model as Krokidis model), which has a lower energy Digne2004  than traditional
defective spinel models Menendez2005 ; Pinto2004 , and is consistent with
experimental NMR and XRD results Zhou1991 . Our test calculations also confirm
the stability of the Krokidis model. The Krokidis model has a centrosymmetric
monoclinic structure (P21/m, No 11) with 1/4 four- and 3/4 six-fold
coordinated Al atoms. Our LDA optimization results in the following
parameters: $a=5.479$ Å, $b=8.255$ Å, $c=7.961$ Å, and $\beta=90.645^{\circ}$.
The lowest energy (001) $\gamma$-Al2O3 surface Digne2004  based on the
Krokidis model has many inequivalent surface sites (Fig. 1). Here, oxygen
atoms are indexed with capital letters and aluminum atoms with numbers
Valero2006 . It is noted that there is a mirror symmetry plane which is
perpendicular to $b$ and crosses Al 3 and 4. Therefore, the O at C (D) is
equivalent to the O at E (F), and the Al at 2 has the same environment as the
Al at 5. All surface Al atoms are pentacoordinated, except that Al atom 1 is
tetracoordinated and in a position slightly below the surface plane. All
surface oxygens are tricoordinated if only Al neighbors within 2 Å are
counted. However, C and E oxygen atoms have an additional nearby Al atom
besides the bonding Al atoms: i.e., the distance between Al 2 and oxygen C is
2.19 Å. In this sense, C and E oxygen atoms are quasi-four-fold coordinated,
as suggested in Fig. 1(a) by the dashed lines. Here, a four-layer (not
counting the tetrahedral Al atoms) symmetric slab model is adopted. After
relaxation, oxygen D (and F), oxygen A, and oxygen B move outward from the
surface by about 0.3 Å, 0.2 Å, and 0.1 Å, respectively. In contrast, oxygen C
and E stay in the surface due to the strong binding with four neighboring Al
atoms. It is noted that the surface is insulating due to the charge transfer
from surface Al atoms to O dangling bonds.

We first examine the thermodynamic stability of the interface by calculating
the enthalpy of two possible reactions Schlom2002 :

$\begin{array}[]{ll}\frac{3}{2}\mathrm{Si}+\mathrm{Al}_{2}\mathrm{O}_{3}\rightarrow
2\mathrm{Al}+\frac{3}{2}\mathrm{Si}\mathrm{O}_{2},\Delta
H=2.88\quad\mathrm{eV}\\\
\mathrm{Si}+\frac{5}{3}\mathrm{Al}_{2}\mathrm{O}_{3}\rightarrow\frac{4}{3}\mathrm{Al}+\mathrm{Si}\mathrm{Al}_{2}{\mathrm{O}}_{5},\Delta
H=0.99\quad\mathrm{eV}.\end{array}$ (1)

These positive reaction enthalpies indicate that the Si/Al2O3 interface is
thermodynamically stable, i.e., the formation of SiO2 and silicate is
unfavorable.

The construction of the interface model is a nontrivial task. Usually,
molecular dynamics simulations Broqvist2009  or intuition were employed for
this purpose. It should be noted that molecular dynamics simulations gives
different interface structures depending on initial conditions, and it is
almost impossible to guarantee that the constructed interface structure has
the lowest interface energy. And it is very hard to design a good interface
structure between two totally dissimilar materials just from chemical
intuition. Therefore, we develop a new modified BH method Wales1999  to
determine the most stable interface structure. In conventional BH method, each
BH run starts with a randomly chosen atomic configuration and is composed of a
given number of Monte Carlo steps. In each of these, the starting
configuration is first locally optimized to obtain an energy $E_{1}$. Then,
each atom is subjected to a random displacement in each of its Cartesian
coordinates, and a new locally optimized structure is obtained with energy
$E_{2}$. Here, $E_{1}$ and $E_{2}$ are the total energies from DFT
calculations. If $\exp[(E_{1}-E_{2})/k_{B}T]>r$, where $r$ is a random number
between 0 and 1 (Metropolis criterion), the new configuration is accepted
(otherwise the old configuration is kept), and the process is iterated. The BH
method has been widely used to search the global minimal structure of clusters
Wales1999 ; Yoo2003 ; Pei2008 ; Barcaro2007 . However, to our best knowledge,
the BH method has not been employed to search for the interface structure
between two surfaces.

In our newly developed modified BH method for finding lowest energy interface
structures, we name the two slabs as “top” and “bottom”, respectively (see
Fig. 2). In the case of the Si/Al2O3 interface, the Si (Al2O3) (001) slab is
the top (bottom) one. The Si slab has seven Si layers. The top Si layer forms
Si dimers and is passivated by H atoms. For the top Si slab, we have a rigid
layer, a buffer layer, and a hopping layer. The atoms in the rigid layer can
translate as a rigid body but the internal structure is fixed fix . The fixed
layer is held in place and the buffer layer is allowed to relax during the
optimization. In contrast, the atoms of the hopping layer move as in the usual
BH method but are restricted to the interface region. The typical value of the
hopping distance of the BH simulation is about $1.5$ Å. Our test calculations
indicate that swapping a Si for an Al atom is energetically unfavorable by
about 2 eV. Thus, the bottom Al2O3 slab is divided into two parts: a fixed
layer and a buffer layer. We note that the our modified BH method is rather
general and can be used to search for the lowest energy structure of other
interfaces.

Considering the lateral lattice contants of the (001) Si surface
[$a(\mathrm{Si})=5.404$ Å] and (001) $\gamma$-Al2O3 surface
[$a(\mathrm{Al}_{2}\mathrm{O}_{3})=5.479$ Å,
$b(\mathrm{Al}_{2}\mathrm{O}_{3})=8.255$ Å], the best lattice matching is
achieved by connecting the ($1\times 3$) Si (001) surface with the ($1\times
2$) (001) $\gamma$-Al2O3 surface. In this structure, the calculated lattice
mismatch is about 1.6%. Here, the in-plane lattice constants of the supercell
are fixed to be the theoretical lattice constants of bulk $\gamma$-Al2O3
because $\gamma$-Al2O3 has a large Young’s modulus. We perform several BH
simulations for 200 steps with different initial coordinates (the relative
position between the Si surface and the Al2O3 surface, and the atomic
positions of the atoms of the hopping layer). Finally, the lowest energy
interface structure found from the BH simulations is refined by performing a
full atomic relaxation of the whole system, including all atoms of the “fixed”
Al2O3 layer and “rigid” Si layer.

The lowest energy interface structure that we find is shown in Fig. 3. We can
see that the dimer structure at the Si (001) surfaces is preserved as a result
of the strong covalent Si-Si bond. We note that there is no dimer in the
interface of the initial structure, while dimers are formed during the
relaxation. At the interface, one Si atom of each dimer bonds with a three-
fold coordinated O atom of the Al2O3 surface, whereas the other Si atom forms
a bond with a four-fold coordinated Al atom. The Si-O and Si-Al bond lengths
are about 1.8 Å and 2.4 Å, respectively. The O atoms bonded with Si move
outward from the $\gamma$-Al2O3 surface in order to form bonds with Si atoms.
We find that oxygen C and E do not bond with Si atoms because it is
unfavorable for them to move outward due to the strong binding with the fourth
neighboring Al atom below the surface. The binding energy between the Si
surface and Al2O3 surface is calculated to be 2.96 eV/supercell, which
indicating the strong binding between the two surfaces. It should be noted
that there are some other nearly degenerate (within 20 meV/cell) interface
structures with feature similar to that shown in Fig. 3. In these metastable
structures, other Al and O atoms are bonded with Si dimers.

The DOS for the interface is shown in Fig. 3(c). We can see that the system is
semiconducting with an indirect band gap of 0.46 eV. Remarkably, this value is
larger than the LDA band gap (0.45 eV) of bulk Si. The presence of the band
gap is also consistent with the stability of the interface. The DOS plot shows
a type-I band alignment between Si and Al2O3. To compute an accurate band
offset, we align the energy levels using the core levels Wei1998 . The
calculated valence band offset is 2.40 eV. The measured value between Si and
$\alpha$-Al2O3 ranges from 2.90 eV to 3.75 eV Bersch2008 . The experimental
valence band offset between Si and $\gamma$-Al2O3 is expected to have a
similar value. The discrepancy between the experimental result and our
theoretical value are due to the different LDA error for the covalent Si and
ionic Al2O3 but the result is qualitatively correct Alkauskas2008 . To gain
insight into the electronic properties of the interface, we show the partial
charge densities of the topmost three HOMOs and bottommost three LUMOs of the
interface in Fig. 3(a) and (b). It is clear that the HOMOs are mainly
contributed by the directional covalent Si-Al bonds, and the LUMOs by the
antibonding Si-O bonds.

It is well known that each Si atom of the symmetric Si dimer of the Si (001)
surface has one dangling bond. On the free Si (001) surface, the tilt of the
Si dimer lifts the degeneracy of the Si dangling bonds and a band gap opens
because of the charge transfer from the inward Si atom to outward Si atom. In
the case of the Si/Al2O3 interface, the band gap opening mechanism is totally
different and much more efficient. As shown in Fig. 4, the lone pair electrons
of the surface O atom interact with the dangling bond of the nearest neighbor
Si atom, raising the level of the dangling bond. In contrast, the high-lying
empty Al orbital hybridizes with the dangling bond of the neighboring Si atom,
lowering the energy level of the Si orbital. As a consequence, the Si atom
bonded with the O atom transfers its dangling bond electron to the covalent
Si-Al bond, and the interface has a large band gap. This bonding mechanism
between Si and Al2O3 is consistent with the Bader charge analysis Bader : the
Si atom bonded with Al gains about 0.25 electrons, whereas the Si atom bonded
with O loses about 0.40 electrons. As a result, there is some small net charge
transfer (0.15 e/Si-dimer) from Si to Al2O3.

To investigate the kinetic stability of the interface, we calculate the energy
barrier of the sliding of the Si surface on the Al2O3 surface. To find the
transition state and energy barrier, we use the “climbing image nudged elastic
band” method Henkelman2000 . We consider the sliding of the Si surface along
the $b$ axis because the barrier of the sliding along other directions are
expected to be larger due to the need to break all Si-O and Si-Al bonds. The
final interface structure is obtained from the initial structure by sliding
the Si surface along the $b$ axis by $b$(Al2O3)$/3$; the final state is almost
degenerate with the initial state. In the transition state, there is some
remaining bonding between the Si surface and Al2O3 surface: one Si-O bond and
two Si-Al bonds. The energy barrier of the sliding is about 2.0 eV/supercell,
which makes the Si/Al2O3 interface kinetically stable.

In conclusion, we develop a general modified BH method to search for the
lowest energy structure of (001)-Si/$\gamma$-(001)-Al2O3 interface. It is
found that the interface Si dimers have a favorable 4-fold coordination due to
the formation of not only Si-O bonds, but also unexpected covalent Si-Al
bonds. Our study reveals that the Si/Al2O3 interface has the following
attractive properties: (i) The interface is sharp and is semiconducting with a
large LDA band gap; (ii) The band alignment between Si and $\gamma$-Al2O3 is
type-I with both valence band offset and conduction band offset larger than
1.5 eV; (iii) The interface is thermodynamically and kineticly stable. Our
results suggest that $\gamma$-Al2O3 can be used as a gate dielectric in future
MOS technologies or a substrate for the growth of c-Si for solar cells.

Work at NREL was supported by the U.S. Department of Energy, under Contract
No. DE-AC36-08GO28308.

## References

  * (1) C. J. Först et al., Nature (London) 427, 53 (2004).
  * (2) R. A. McKee, F. J. Walker, and M. F. Chisholm, Phys. Rev. Lett. 81, 3014 (1998).
  * (3) R. A. McKee et al., Science 300, 1726 (2003).
  * (4) D. Ceresoli and D. Vanderbilt, Phys. Rev. B 74, 125108 (2006).
  * (5) A. I. Kingon, J. P. Maria, and S. K. Streiffer, Nature (London) 406, 1032 (2000).
  * (6) C. Merckling et al., Appl. Phys. Lett. 89, 232907 (2006)
  * (7) C. W. Teplin, D. S. Ginley and H. M. Branz, J. Non-Cryst. Solids 352, 984 (2006).
  * (8) Alp T. Findikoglu et al., Adv. Mater. 17, 1527 (2005).
  * (9) B. Hoex et al., Appl. Phys. Lett. 89, 042112 (2006).
  * (10) P. Boulenc and I. Devos, Mater. Sci. Semicond. Process. 9, 949 (2006).
  * (11) E. Menéndez-Proupin and G. Gutiérrez, Phys. Rev. B 72, 035116 (2005).
  * (12) P. E. Blöchl, Phys. Rev. B 50, 17953 (1994); G. Kresse and D. Joubert, ibid 59, 1758 (1999).
  * (13) G. Kresse and J. Furthmüller, Comput. Mater. Sci. 6, 15 (1996); Phys. Rev. B 54, 11169 (1996).
  * (14) X. Krokidis et al., J. Phys. Chem. B 105, 5121 (2001).
  * (15) M. Digne et al., J. Catal. 226, 54 (2004).
  * (16) H. P. Pinto, R. M. Nieminen, and S. D. Elliott, Phys. Rev. B 70, 125402 (2004).
  * (17) R.S. Zhou, R.L. Snyder, Acta Crystallogr. B 47, 617 (1991).
  * (18) M. C. Valero, P. Raybaud, and P. Sautet, J. Phys. Chem. B 110, 1759 (2006).
  * (19) D. G. Schlom and J. H. Haeni, MRS Bull. 27, 198 (2002).
  * (20) P. Broqvist, J. F. Binder, and A. Pasquarello, Appl. Phys. Lett. 94, 141911 (2009).
  * (21) D. J. Wales and H. A. Scheraga, Science 285, 1368 (1999); D. J. Wales and J. P. K. Doye, J. Phys. Chem. A 101, 5111 (1997).
  * (22) S. Yoo and X. C. Zeng, J. Chem. Phys. 119, 1442 (2003).
  * (23) Y. Pei and X. C. Zeng, J. Am. Chem. Soc. 130, 2580 (2008).
  * (24) G. Barcaro and A. Fortunelli, Phys. Rev. B 76, 165412 (2007).
  * (25) We modify the VASP code to relax a rigid body.
  * (26) S.-H. Wei and A. Zunger, Appl. Phys. Lett. 72, 2011 (1998).
  * (27) E. Bersch et al., Phys. Rev. B 78, 085114 (2008) and references therein.
  * (28) A. Alkauskas et al., Phys. Rev. Lett. 101, 106802 (2008).
  * (29) R. F. W. Bader, Atoms in Molecules-A Quantum Theory (Oxford University Press, Oxford, 1990); G. Henkelman, A. Arnaldsson, and H. Jńsson, Comput. Mater. Sci. 36, 254 (2006).
  * (30) G. Henkelman, B. P. Uberuaga, and H. Jńsson, J. Chem. Phys. 113, 9901 (2000).

Figure 1: (Color online) (a) Top and (b) side view of the ($1\times 1$) (001)
$\gamma$-Al2O3 surface. Oxygen (small) atoms are indexed with capital letters
and aluminum atoms (large) are indexed with numbers.

Figure 2: (Color online) The definition of various layers of the Si/Al2O3
interface in our modified BH method.

Figure 3: (Color online) Interface structure and isosurface plots of the
partial charge density of (a) the topmost three HOMOs and (b) bottommost three
LUMOs of the Si/Al2O3 interface. (c) DOS plot for the Si/Al2O3 interface,
calculated with 0.1 eV broadening. The vertical dashed line denotes the top of
the valence band. The partial DOSs of the Si and Al2O3 surfaces are also
shown.

Figure 4: (Color online) Schematic illustration of the Si-Al and Si-O bond
formation and gap opening in the Si/Al2O3 interface. The valence-band maximum
(VBM) and conduction-band minimum (CBM) of bulk Si are also shown
schematically.