# Occupancy-MAE: Self-supervised Pre-training Large-scale LiDAR Point Clouds
with Masked Occupancy Autoencoders

Chen Min, Liang Xiao, Dawei Zhao, Yiming Nie, Bin Dai This work was supported
by the National Natural Science Foundation of China under Grant Nos. 61790565
and 61803380.C. Min is with the School of Computer Science, Peking University,
Beijing 100071, China (e-mail: minchen@stu.pku.edu.cn).L. Xiao, D. Zhao, Y.
Nie, and B. Dai are with the Unmanned Systems Technology Research Center,
Defense Innovation Institute, Beijing 100071, China (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>ibindai@163.com).

###### Abstract

Current perception models in autonomous driving heavily rely on large-scale
labelled 3D data, which is both costly and time-consuming to annotate. This
work proposes a solution to reduce the dependence on labelled 3D training data
by leveraging pre-training on large-scale unlabeled outdoor LiDAR point clouds
using masked autoencoders (MAE). While existing masked point autoencoding
methods mainly focus on small-scale indoor point clouds or pillar-based large-
scale outdoor LiDAR data, our approach introduces a new self-supervised masked
occupancy pre-training method called Occupancy-MAE, specifically designed for
voxel-based large-scale outdoor LiDAR point clouds. Occupancy-MAE takes
advantage of the gradually sparse voxel occupancy structure of outdoor LiDAR
point clouds and incorporates a range-aware random masking strategy and a
pretext task of occupancy prediction. By randomly masking voxels based on
their distance to the LiDAR and predicting the masked occupancy structure of
the entire 3D surrounding scene, Occupancy-MAE encourages the extraction of
high-level semantic information to reconstruct the masked voxel using only a
small number of visible voxels. Extensive experiments demonstrate the
effectiveness of Occupancy-MAE across several downstream tasks. For 3D object
detection, Occupancy-MAE reduces the labelled data required for car detection
on the KITTI dataset by half and improves small object detection by
approximately 2% in AP on the Waymo dataset. For 3D semantic segmentation,
Occupancy-MAE outperforms training from scratch by around 2% in mIoU. For
multi-object tracking, Occupancy-MAE enhances training from scratch by
approximately 1% in terms of AMOTA and AMOTP. Codes are publicly available at
https://github.com/chaytonmin/Occupancy-MAE.

###### Index Terms:

3D LiDAR Pre-training, Occupancy Prediction, Masking Autoencoder, Autonomous
driving.

††publicationid: pubid: 0000–0000/00$00.00 © 2021 IEEE

##  I Introduction

Accurate 3D perception is a core technique in autonomous driving, as it
enables vehicles to obtain precise information about their surroundings [1].
Numerous large-scale outdoor LiDAR point cloud datasets such as KITTI [2],
Waymo [3], nuScenes [4], and ONCE [5] have been published, showcasing the
potential of environmental perception for unmanned vehicles. However, the
collection and annotation of large-scale LiDAR point clouds for common tasks
like 3D object detection and semantic segmentation can be extremely time-
consuming and labour-intensive. For instance, skilled workers can only
annotate around 100-200 frames per day [5]. Therefore, self-supervised
learning using large-scale unannotated LiDAR point clouds is crucial for
enhancing the perceptual ability of autonomous driving. It may pave the way
for developing the next-generation, industry-level autonomous driving
perception model that is both powerful and robust [5].

Figure 1:  Label-efficiency of our self-supervised pre-training. Occupancy-MAE
outperforms training from scratch and achieves the same detection performance
with fewer labelled data (about 50% for the car class and 75% for the
pedestrian class). Figure 2: The overall architecture of our Occupancy-MAE. We
first transform the large-scale irregular LiDAR point clouds into volumetric
representations, randomly mask the voxels according to their distance from the
LiDAR sensor (i.e., range-aware masking strategy), then reconstruct the
geometric occupancy structure of the general 3D world with an asymmetric
autoencoder network. We adopt the 3D Spatially Sparse Convolutions [6] with
positional encoding as the encoding backbone. We apply binary occupancy
classification as the pretext task to distinguish whether the voxel contains
points. After pre-training, the lightweight decoder is discarded, and the
encoder is used to warm up the backbones of downstream tasks.

Self-supervised learning has made significant strides in recent years,
enabling the pre-training of rich features without human annotations. The
simple masked autoencoding approach has been particularly effective in
learning representative features, where the task is to reconstruct masked data
from unmasked input [7, 8, 9, 10]. In natural language processing, masked
autoencoding has enabled the training of large language models, such as BERT
[7]. Similarly, in 2D vision, masked autoencoding has outperformed supervised
pre-training counterparts [8].

Recently, several works on masked point autoencoding have been proposed, such
as Point-MAE [11], Point-BERT [12], MaskPoint [13], and Point-M2AE [14].
However, these methods have primarily focused on small-scale point clouds,
such as synthetic point cloud datasets like ShapeNet [15] and indoor point
cloud datasets like ScanNet [16]. In contrast, masked autoencoding for large-
scale outdoor LiDAR point clouds [2, 3, 4, 5], has received less attention.
Training from scratch on large-scale labelled LiDAR point clouds remains the
dominant approach [5]. To introduce the idea of masked autoencoding to self-
supervised learning on large-scale outdoor LiDAR point clouds, we first
identify the challenges compared to masked small-scale point cloud
autoencoding works [12, 11, 13, 14], and then propose solutions:

Firstly, small-scale point clouds [15, 16] differ from large-scale outdoor
LiDAR point clouds [2, 3, 4, 5] in several ways. (1) Small-scale point clouds
typically contain far fewer points than their large-scale counterparts, with
ShapeNet containing approximately 2k points and ScanNet downsampled to about
2k points for use in masked small-scale point cloud autoencoding works [12,
11, 13, 14]. In contrast, LiDAR sensors such as the Velodyne HDL-64E LiDAR can
scan up to 192,000 points per frame, covering an area of
160$\times$160$\times$20 meters [17]. (2) Small-scale point clouds are often
evenly distributed, while large-scale LiDAR point clouds become sparser as the
distance from the LiDAR sensor increases. (3) Existing masked small-scale
point cloud autoencoding works [12, 11, 13, 14] typically rely on furthest
point sampling (FPS) and k-nearest neighbours (kNN) to divide points into
equal subsets, which is not suitable for large-scale outdoor LiDAR point
clouds. The challenge of processing such large-scale point clouds efficiently,
and in real-time, has been highlighted in previous works [18, 17], making pre-
training on these datasets a more challenging task.

Secondly, existing masked small-scale point clouds autoencoding works [12, 11,
14] mainly focus on regressing the missing points. However, it is not
appropriate to reconstruct the features of large-scale LiDAR point clouds
through regression, as these features contain essential spatial information
[6]. Moreover, the positional encoding of voxels can offer a shortcut for the
decoder [19]. To address this issue, we shift our focus to the occupancy
distribution of large-scale LiDAR point clouds and design the occupancy
prediction objective as the pretext task. By doing so, the network is forced
to learn representative features to recover the overall structure of the 3D
scene, making this simple task an effective solution for our pre-training
method.

Thirdly, the random masking strategy used in existing masked small-scale point
clouds autoencoding works [12, 11, 13, 14] may not be suitable for large-scale
outdoor LiDAR point clouds due to their uneven distribution. Unlike small-
scale point clouds, the density of large-scale LiDAR point clouds varies based
on the distance to the LiDAR sensor. Hence, a uniform random masking strategy
for all voxels would not be optimal. We propose a range-aware random masking
strategy for large-scale LiDAR point clouds to address this. This strategy
adjusts the masking ratio based on the voxel’s distance to the LiDAR sensor,
with the masking ratio decreasing as the distance from the sensor increases.

Transformer has started to make their presence felt in processing point clouds
[20]. For instance, PCT [21] performs global self-attention across the entire
point cloud, while PointASNL [22], PointFormer [23], and VoxSet [24] apply
transformer-based architectures to the local neighborhood of each point.
However, the efficiency of these local transformers is constrained by
neighborhood query and feature restructuring. SST [25] and FlatFormer [26]
apply window-based point cloud transformers. These models project the point
cloud into a bird’s-eye view (BEV) and partition the BEV space into non-
overlapping windows of the same spatial size. However, these approaches could
lose valuable information in the vertical axis. We use 3D Sparse Convolutions
[6] with a positional encoding module in the encoder. This allows the network
to focus on visible 3D voxels instead of processing the entire point cloud,
significantly reducing the computational cost. The positional encoding module,
similar to the positional embeddings used in Transformers, encodes the spatial
information of the voxels into a fixed-size embedding vector.

Recently, there have been several proposed self-supervised pre-training
algorithms (e.g., Voxel-MAE [27], MV-JAR [28], GeoMAE [29], and GD-MAE [30])
based on masked autoencoder for large-scale outdoor LiDAR point clouds.
However, these algorithms are specifically tailored for pre-training pillar-
based outdoor LiDAR point clouds method SST [25] (SST replaces Pointpillars
[31] backbone with Transformer [32] and cannot process 3D voxels.),
overlooking the crucial height information of the point clouds.

Information redundancy in LiDAR point clouds refers to the presence of
duplicate or highly similar data points within the point cloud dataset [37,
38]. This redundancy can arise from various sources [39, 40], such as
overlapping LiDAR swaths, multiple returns, sampling density, sensor
artifacts, and noise. The overlapping points contain similar information about
the terrain, which can be redundant. The multiple returns often contain
information about the same physical feature from different angles, leading to
redundancy. The high sampling density can lead to redundancy, as multiple
points may describe the same ground surface, particularly in flat or uniform
areas. The artifacts can result in redundant data points that do not represent
actual features on the ground. Consequently, a substantial amount of redundant
spatial information remains concealed within LiDAR point clouds, making them
suitable for masked autoencoding methods to learn rich and representative
features.

Occupancy as a compressed representation of LiDAR point clouds can be defined
as whether or not a voxel in the 3D grid contains points. Occupancy prediction
is a commonly used approach in general 3D world representation for autonomous
driving tasks [33]. It involves dividing the environment into a grid and
estimating the probability of each cell being occupied or free. This approach
has been applied in various tasks such as obstacle detection, path planning,
and simultaneous localization and mapping (SLAM) [34, 35, 36].

Driven by the above analyses, we propose the first self-supervised masked
occupancy autoencoding framework, called Occupancy-MAE, for pre-training
large-scale outdoor LiDAR point clouds. Figure 2 illustrates the workflow of
our Occupancy-MAE, which first employs the range-aware masking strategy to
mask voxels randomly, and then feeds the unmasked voxels into the 3D sparse
encoder. The output of the 3D decoder is the probability that each voxel
contains points, and we calculate the binary occupancy classification loss to
pre-train the network. Training on the pretext task of masked occupancy
classification encourages the encoder network to be voxel-aware of the entire
object shape, thereby learning representative features for 3D perception.

After analyzing the results of our experiments, we conclude that Occupancy-MAE
is a simple and effective self-supervised learning framework that can
generalize well across various downstream tasks. In the 3D object detection
task, our method outperforms the state-of-the-art self-supervised learning
methods by 0.5% $\sim$ 6% mAP on the ONCE dataset, which is a recently
published dataset for self-supervised learning on large-scale LiDAR point
clouds. Moreover, pre-training with Occupancy-MAE can significantly improve
the performance of popular 3D detectors such as SECOND [6], PV-RCNN [41],
CenterPoint [42], and PV-RCNN++ [43] trained from scratch on KITTI, Waymo, and
nuScenes datasets, especially for small objects. As illustrated in Figure 1,
Occupancy-MAE is a data-efficient learner that can effectively train large-
scale LiDAR point clouds with limited annotated 3D data. For the 3D semantic
segmentation task, Occupancy-MAE with a two-layer decoder can improve the
training from scratch by about 2% mIoU. For multi-object tracking task,
Occupancy-MAE enhances training from scratch by approximately 1% in terms of
AMOTA and AMOTP. We also demonstrate the effectiveness of our approach in the
unsupervised domain adaptation task, which confirms the transfer learning
ability of Occupancy-MAE. Even with a 90% masking ratio, Occupancy-MAE can
still learn representative features as large-scale LiDAR point clouds are
information redundant, which ultimately improves the performance of 3D
perception.

The main contributions of this work are listed below:

  * •

We present a novel self-supervised masked occupancy autoencoding framework
called Occupancy-MAE, for pre-training large-scale outdoor LiDAR point clouds,
reducing the need for expensively annotated 3D data.

  * •

We propose a 3D occupancy prediction pretext task that leverages the gradually
sparse occupancy structure of large-scale LiDAR point clouds. By recovering
the masked occupancy distribution of the 3D surrounding world from a small
number of visible voxels, the network is forced to extract high-level semantic
information.

  * •

We introduce the range-aware random masking strategy to take advantage of the
varying density of large-scale LiDAR point clouds, which improves the pre-
training performance.

  * •

Our proposed Occupancy-MAE significantly outperforms training from scratch on
various downstream tasks, including 3D object detection, semantic
segmentation, multi-object tracking, and unsupervised domain adaptation.

##  II Related Work

###  II-A LiDAR-based 3D Perception

LiDAR-based 3D perception models with accurate 3D spatial information have
been widely used in the field of autonomous driving [44, 45]. LiDAR-based 3D
object detectors can be categorized into point-based [46, 47], voxel-based
[48, 6, 31], and point-voxel-based [41, 49]. Point-based object detectors
extract discriminative features from raw point clouds with PointNet [50] and
generate proposals centred at each point with high computation cost. Voxel-
based detectors transform the irregular point clouds into volumetric
representations, which will degrade the fine-grained localization accuracy.
Point-voxel-based methods take advantage of the localization accuracy of
point-based detectors and the computational efficiency of voxel-based
detectors. LiDAR-based 3D segmentation methods are divided into grid-based
[51, 52] and voxel-based [18, 53]. Grid-based methods focus on converting the
3D point clouds into a 2D frontal-view image or a range image, which fails to
model the 3D geometric information. Voxel-based methods convert the point
clouds into volumetric representations. The existing 3D perception methods are
trained with large-scale labelled 3D data. How to design the self-supervised
learning network to minimize dependence on 3D annotation has rarely been
studied.

###  II-B Self-supervised Learning

Self-supervised Learning (SSL) has gained popularity in recent years as an
effective approach to avoiding costly data annotation. The pretext task of
predicting the relative location of image patches was proposed in [54].
Methods in [55, 56, 57] have designed rotation prediction tasks, which have
shown promising results in learning representative features. In [58], a jigsaw
puzzle prediction task is introduced, which generalizes well in domain
adaptation for object recognition. DeepCluster [59] and SwAV [60] obtain
pseudo labels with k-means clustering and use these labels to train networks.
Other methods such as Moco [61], PointContrast [19], BYOL [62],
ProposalContrast [63], and DepthContrast [64] construct contrastive views for
self-supervised learning. Recently, MAE [8] has shown promising results in
self-supervised learning by first masking random patches of the input image
and then reconstructing the missing pixels with a simple autoencoder
framework. VideoMAE [65] extends the MAE into spatiotemporal representation
learning from videos, which have greater information redundancy. ALSO [66]
trains the model on a pretext task which is the reconstruction of the surface
on which the 3D points are sampled. ISCC [67] applies contrastive cluster and
implicit surface reconstruction for self-supervised pre-training large-scale
outdoor LiDAR point clouds. Our Occupancy-MAE follows the design philosophy of
MAE and applies it to large-scale outdoor LiDAR point clouds based on their
geometric characteristics, such as sparsity and varying density.

###  II-C Masked autoencoders for point clouds

Masked autoencoding has achieved success in NLP [7] and image [8], leading to
the development of masked autoencoding techniques for point clouds in the last
year. Point-BERT [12] first introduces MAE to pre-train small-scale point
clouds. Point-MAE [11] reconstructs the small-scale point patches with the
Chamfer distance. MaskPoint [13] designs the decoder for discriminating the
small-scale masked point patches. Point-M2AE [14] applies the pyramid
architectures to model spatial geometries and capture both fine-grained and
high-level semantics of 3D shapes. However, these methods focusing on small-
scale indoor point clouds cannot process large-scale outdoor LiDAR point
clouds under the property of a large range of scenes and varying density.
Voxel-MAE [27], MV-JAR [28], GeoMAE [29], and GD-MAE [30] adopt masked
autoencoding for large-scale point clouds but are limited to only pillar-based
methods, which will lose vertical information. Our proposed Occupancy-MAE
overcomes these limitations and enables the pre-training of large-scale LiDAR
point clouds for both voxel and pillar-based methods.

##  III Methodology

TABLE I: The details of our Occupancy-MAE architecture which consists of a 3D Encoder and a 3D Decoder. 3DSparseCvonv and 3DTransCvonv denote the 3D Sparse Convolutional layer proposed in SECOND [45] and the common 3D Deconvolutional layer, respectively. Here we display the output size for pre-training SECOND on the KITTI dataset. Input |  Layer Destription | Output | Output Size
---|---|---|---
Input point clouds | Mean operator | Voxel representation | 1600$\times$ 1408 $\times$ 41 $\times$ 4
3D Encoder
Masked voxels $\textbf{V}_{input}$ | 3DSparseCvonv, filter=(3,3,3), stride=(1,1,1) | Spconv_tensor_1 | 1600$\times$ 1408 $\times$ 41 $\times$ 16
Spconv_tensor_1 | 3DSparseCvonv, filter=(3,3,3), stride=(2,2,2) | Spconv_tensor_2 | 800$\times$ 704 $\times$ 21 $\times$ 32
Spconv_tensor_2 | 3DSparseCvonv, filter=(3,3,3), stride=(2,2,2) | Spconv_tensor_3 | 400$\times$ 352 $\times$ 11 $\times$ 64
Spconv_tensor_3 | 3DSparseCvonv, filter=(3,3,3), stride=(2,2,2) | Spconv_tensor_4 | 200$\times$ 176 $\times$ 5 $\times$ 64
Spconv_tensor_4 | 3DSparseCvonv, filter=(1,1,3), stride=(1,1,2) | Latent feature tensor | 200$\times$ 176 $\times$ 2 $\times$ 128
3D Decoder
Latent feature tensor | 3DTransCvonv, filter=(3,3,3), stride=(2,2,2) | Dense_tensor_1 | 400$\times$ 354 $\times$ 4 $\times$ 32
Dense_tensor_1 | 3DTransCvonv, filter=(3,3,3), stride=(2,2,4) | Dense_tensor_2 | 800$\times$ 704 $\times$ 14 $\times$ 8
Dense_tensor_2 | 3DTransCvonv, filter=(3,3,3), stride=(2,2,3) | Occupied voxels P | 1600$\times$ 1408 $\times$ 41 $\times$ 1

Given the instances from large-scale LiDAR point clouds, self-supervised pre-
training is to train the network with the unlabeled data to generate
representative features. Inspired by the excellent performance of masked
autoencoding [7, 8, 11], we design the masked occupancy autoencoding network
for 3D perception. The proposed Occupancy-MAE randomly masks the voxels and
then reconstructs the occupancy values of voxels with an autoencoder network.
The pretext task of occupancy prediction is trained with the binary cross-
entropy loss. Occupancy-MAE covers most of the backbone networks, excluding
the final head section.

With the $n_{s}$ unlabeled point cloud data
$\left\\{\textbf{X}^{i}\right\\}_{i=1}^{n_{s}}$, we aim to first pre-train the
masked autoencoding network $\phi_{pre}$ to learn the high-level semantics.
Then we use the pre-trained model to warm up the network $\phi_{s}$ of
downstream tasks. We also extend the pre-training method to domain adaptative
task on the target point clouds
$\left\\{\textbf{X}^{j}\right\\}_{j=1}^{n_{t}}$. Detailed information on the
layers of Occupancy-MAE is listed in Table I. The decoder consists of only two
or three 3D Deconvolutional layers, making it quite lightweight.

###  III-A Range-aware Random Masking

In this work, we adopt the common approach of dividing the LiDAR point clouds
into spaced voxels, which is often used in 3D perception models [6, 18]. For
LiDAR point clouds with dimensions $W\times H\times D$ along the $X\times
Y\times Z$ axes, the size of each voxel is $v_{W}\times v_{H}\times v_{D}$,
resulting in a total of $n_{l}$ voxels, of which $n_{v}$ contain points.
Voxel-based methods are more computationally efficient than point-based
methods [46], making them well-suited for processing large-scale LiDAR point
clouds in applications such as self-driving cars.

While the random masking strategy has been proven effective in pre-training
models for language [7], images [8], and small-scale point clouds [11], the
distribution of large-scale LiDAR point clouds is unique due to their sparsity
levels being correlated with distance from the LiDAR sensor. Points close to
the sensor are densely packed, while those further away are much more sparse.
Therefore, we cannot apply the same masking strategy to both near-range and
far-range points. Instead, we propose a range-aware random masking strategy
that masks a small percentage of data for far-range points.

To this end, we introduce the range-aware random masking strategy which takes
into account the distance information. We adhere to the commonly employed
distance configuration for evaluating LiDAR point clouds detection results [3,
5], dividing the occupied voxels into three groups based on their distance to
the LiDAR sensor: 0-30 meters, 30-50 meters, and $>$50 meters. The masking
ratio decreases as the distance increases, and we use a piecewise approach to
apply the random masking strategy to each group. The corresponding numbers of
voxels are $n_{v1}$, $n_{v2}$, and $n_{v3}$. We take the random masking
strategy for each group with the descending masking ratio $r1$, $r2$, and $r3$
(i.e., $r1>r2>r3$). Thus the number of unmasked occupied voxels is
$n_{un}=n_{v1}(1-r1)+n_{v2}(1-r2)+n_{v3}(1-r3)$, and the set of voxels
$\textbf{V}_{input}\in R^{n_{un}\times 4}$ is used as training data. The
ground truth, denoted by $\textbf{T}\in\\{0,1\\}^{n_{l}\times 1}$, represents
the occupancy status of the voxels. Each voxel can either contain points
(occupied) or be empty (free). A value of 1 indicates an occupied voxel, while
a value of 0 indicates a free voxel.

It is noted that more division groups for range-aware masking are also
applicable but will cause lots of preprocessing time in pre-training.
Therefore, we struck a balance between accuracy and speed by implementing
three groups of distance for range-aware masking.

###  III-B 3D Sparse Convolutional Encoder

The Transformer network used in masked autoencoding for NLP [7], 2D vision
[8], and small-scale point clouds [11, 13] performs self-attention on unmasked
portions of the training data, which are unaffected by the masking. However,
with millions of points in a 3D scene, even after masking 90% of voxels, there
are still hundreds of thousands of unmasked voxels, making it impractical for
the Transformer network to aggregate information from such a vast amount of
input data [7]. Some algorithms (e.g., Voxel-MAE [27], MV-JAR [28]) utilize
Transformers for masking, but they are only applicable to 2D pillar
structures, disregarding the 3D height information.

To address this, the 3D Sparse Convolution [68, 6] was proposed for processing
large-scale point clouds. This approach uses positional encoding to aggregate
information only from occupied voxels, resulting in high efficiency. Popular
3D perception methods [6, 41, 42, 18] have been developed using this
technique. Motivated by this, we adopt the 3D Spatially Sparse Convolution
from SECOND [6] to build our encoder network, allowing us to aggregate
information only from unmasked, occupied voxels using the positional encoding
module. Our voxel masking strategy thus reduces the memory complexity of
training, similar to how the Transformer network works in NLP [7], 2D vision
[8], and small-scale point clouds [11, 13, 12, 14].

###  III-C Lightweight 3D Decoder

Our decoder is composed of 3D Deconvolutional layers, with the last layer
outputting the probability of each voxel containing points, resulting in an
output tensor $\textbf{P}\in R^{n_{l}\times 1}$. During pre-training, the
decoder’s only purpose is to perform occupied voxel reconstruction. By
shifting the masked tokens to the decoder, we encourage the encoder to learn
better latent features for downstream tasks. The decoder is lightweight,
consisting of only two or three 3D Deconvolutional layers, making it scalable
to larger perception ranges.

###  III-D Reconstructed Occupancy Target

The main objective of most masked autoencoding works is to reconstruct the
masked parts through a regression task [11, 12, 14], which is not challenging
for the network due to the positional encoding of voxels. However, in 3D
perception, the occupancy structure of the general 3D world plays a crucial
role in perception models [48, 6, 18]. For instance, Tesla introduced the
Occupancy Networks for autonomous driving [tesla]. Motivated by this, we
propose the occupancy prediction task for large-scale outdoor LiDAR point
clouds pre-training, aiming to encourage the network to reason over high-level
semantics to recover the masked occupancy distribution of the 3D scene from a
small number of visible voxels. Considering the significant presence of empty
voxels, the prediction of occupancy presents a binary classification challenge
with class imbalance. To address this, we employ focal loss for binary
occupancy classification, using the predicted occupancy values P and the
ground truth occupied voxels T.

$loss=-\frac{1}{\text{batch}}\frac{1}{n_{l}}\sum_{i=1}^{\text{batch}}\sum_{j=1}^{n_{l}}\alpha_{t}\left(1-\textbf{P}_{t}^{ij}\right)^{\gamma}\log(\textbf{P}_{t}^{ij}),$
(1)

where $\textbf{P}^{ij}$ represents the predicted probability of voxel $j$ in
the $i$-th training sample and batch corresponds to the batch size. The
weighting factor $\alpha$ for positive/negative examples is set to 2, and the
weighting factor $\gamma$ for easy/hard examples is 0.25. For class 1,
$\alpha_{t}=\alpha$ and $\textbf{P}_{t}^{ij}=\textbf{P}^{ij}$. For class 0,
$\alpha_{t}=1-\alpha$ and $\textbf{P}_{t}^{ij}=1-\textbf{P}^{ij}$.

###  III-E Compared with Existing Masked LiDAR Methods

Recently, several self-supervised pre-training algorithms based on masked
autoencoders have been proposed for large-scale outdoor LiDAR point clouds,
including Voxel-MAE [27], MV-JAR [28], GeoMAE [29], and GD-MAE [30]) have been
proposed.

Our method differs from these existing approaches in four key aspects.
Firstly, the above masked LiDAR methods are limited to pillar-based methods,
which overlook the crucial height information of the point clouds, while our
method is applicable to both voxel-based and pillar-based 3D object detection
(e.g., VoxelNet [48] and PointPillars [31]), 3D segmentation (e.g., Cylinder3D
[18]), and domain adaptation algorithms (e.g., ST3D [69]). Secondly, we
introduce the range-aware masking strategy, which treats point clouds at
different distances differently, accounting for varying information densities.
Thirdly, instead of regressing the LiDAR point clouds directly, which may not
be appropriate due to the input points having positions (x, y, z) and offering
a shortcut for regression, we design a 3D occupancy prediction objective for
the entire surrounding scene. In contrast, GeoMAE [29] and MV-JAR [28] use a
2D occupancy prediction target, neglecting the crucial information along the
z-axis. Fourthly, our method utilizes sparse 3D Convolutions, enabling it to
focus on 3D voxels and effectively incorporate volumetric information. In
comparison, the above masked LiDAR methods use Transformers, which only
aggregate 2D pillar features and disregard height information.

##  IV Experiments

TABLE II: Quantitative detection performance achieved by different self-
supervised learning methods on the ONCE _val_ set. The pre-training process is
on the unlabeled small set that contains 100k scenes.

Method | Vehicle | Pedestrian | Cyclist | mAP$\uparrow$
---|---|---|---|---
overall$\uparrow$ | 0-30m | 30-50m | 50m-inf | overall$\uparrow$ | 0-30m | 30-50m | 50m-inf | overall$\uparrow$ | 0-30m | 30-50m | 50m-inf
SECOND [6] | 71.19 | 84.04 | 63.02 | 47.25 | 26.44 | 29.33 | 24.05 | 18.05 | 58.04 | 69.96 | 52.43 | 34.61 | 51.89
BYOL [62] | 68.02 | 81.01 | 60.21 | 44.17 | 19.50 | 22.16 | 16.68 | 12.06 | 50.61 | 62.46 | 44.29 | 28.18 | $46.04^{{\color[rgb]{0,.5,.5}-5.85}}$
ProposalContrast [63] | 72.99 | 84.41 | 65.92 | 50.11 | 25.77 | 27.95 | 23.74 | 18.06 | 58.23 | 69.99 | 53.03 | 35.48 | $52.33^{{\color[rgb]{0,.5,.5}+0.44}}$
SwAV [60] | 72.71 | 83.68 | 65.91 | 50.10 | 25.13 | 27.77 | 22.77 | 16.36 | 58.05 | 69.99 | 52.23 | 34.86 | $51.96^{{\color[rgb]{0,.5,.5}+0.07}}$
DeepCluster [59] | 73.19 | 84.25 | 66.86 | 50.47 | 24.00 | 26.36 | 21.73 | 16.79 | 58.99 | 70.80 | 53.66 | 36.17 | $52.06^{{\color[rgb]{0,.5,.5}+0.17}}$
Occupancy-MAE | 72.78 | 83.77 | 66.01 | 50.26 | 27.49 | 30.54 | 25.28 | 16.11 | 57.26 | 69.71 | 52.31 | 33.51 | ${\bf 52.51}^{{\color[rgb]{0,.5,.5}+0.62\%}}$

###  IV-A Dataset

The four popular autonomous driving datasets: ONCE [5], KITTI [2], Waymo [3],
and nuScenes [4] are used in our experiments.

ONCE. The ONCE dataset [5] comprises 1 million LiDAR scenes and 7 million
corresponding camera images. The data encompasses diverse areas, time periods,
and weather conditions, providing a comprehensive representation of real-world
driving scenarios. Notably, one of the key objectives of the ONCE dataset is
to facilitate research that explores the potential of utilizing large-scale
unlabeled data, opening up opportunities for innovative approaches in
autonomous driving research and development.

KITTI. The KITTI dataset[2] is one of the most popular autonomous driving
datasets, which provides 7,481 training samples and 7,518 testing samples. The
3D bounding box annotations are only provided within the Field of View (FoV)
of the front camera. We follow the common 50/50 _train_ /_val_ split and use
the official KITTI evaluation metrics for three-level evaluation (Easy,
Moderate, Hard) and the mean average precision is evaluated.

Waymo Open Dataset. The Waymo Open Dataset [3] is a recently released large-
scale autonomous driving dataset, which consists of total of 798 training
sequences with around 158,361 LiDAR samples, and 202 validation sequences with
40,077 LiDAR samples. Following the popular point cloud detection codebase
OpenPCDet [70], we subsample a single frame of 20% data (about 32k frames) of
all the training samples as the training set. It annotated the objects in the
full 360∘ field. The official evaluation metrics of Waymo are mean average
precision (AP) and mean average precision weighted by heading (APH) in
difficulty levels (L1 and L2).

nuScenes. The nuScenes dataset [4] is another popular autonomous driving
dataset. There are a total of 28,130 training samples and 6,019 validation
samples. We evaluate with the official evaluation metrics of the nuScenes
Detection Score (NDS), mean average precision (mAP), average translation error
(ATE), average scale error (ASE), average orientation error (AOE), average
velocity error (AVE), average attribute error (AAE).

TABLE III: Performance comparison on the KITTI test split evaluated by the
mean Average Precision with 40 recall positions at a moderate difficulty
level.

Method | Car | Cyclist
---|---|---
3D$\uparrow$ | BEV$\uparrow$ | 3D$\uparrow$ | BEV$\uparrow$
SECOND [6] | 72.55 | 83.77 | 52.08 | 56.05
Occupancy-MAE + SECOND | ${\bf 72.87}^{{\color[rgb]{0,.5,.5}+0.32}}$ | ${\bf 83.96}^{{\color[rgb]{0,.5,.5}+0.19}}$ | ${\bf 54.84}^{{\color[rgb]{0,.5,.5}+1.76}}$ | ${\bf 60.67}^{{\color[rgb]{0,.5,.5}+4.62}}$

TABLE IV: Performance comparison on the KITTI _val_ split evaluated by the
mean Average Precision with 40 recall positions.

Method | Car$\uparrow$ | Pedestrian$\uparrow$ | Cyclist$\uparrow$
---|---|---|---
SST [25] | 81.46 | 46.52 | 65.59
Voxel-MAE [27] \+ SST | $81.53^{{\color[rgb]{0,.5,.5}+0.07}}$ | $46.63^{{\color[rgb]{0,.5,.5}+0.11}}$ | $65.72^{{\color[rgb]{0,.5,.5}+0.13}}$
MV-JAR [28] \+ SST | $81.75^{{\color[rgb]{0,.5,.5}+0.29}}$ | $47.56^{{\color[rgb]{0,.5,.5}+1.04}}$ | $66.37^{{\color[rgb]{0,.5,.5}+0.78}}$
GeoMAE [29] \+ SST | $81.84^{{\color[rgb]{0,.5,.5}+0.38}}$ | $48.18^{{\color[rgb]{0,.5,.5}+1.66}}$ | $67.01^{{\color[rgb]{0,.5,.5}+1.42}}$
GD-MAE [30] \+ SST | $82.01^{{\color[rgb]{0,.5,.5}+0.55}}$ | $48.40^{{\color[rgb]{0,.5,.5}+1.88}}$ | $67.16^{{\color[rgb]{0,.5,.5}+1.57}}$
Occupancy-MAE + SST | ${\bf 82.24}^{{\color[rgb]{0,.5,.5}+0.78}}$ | ${\bf 48.95}^{{\color[rgb]{0,.5,.5}+2.43}}$ | ${\bf 67.78}^{{\color[rgb]{0,.5,.5}+2.19}}$

TABLE V: Performance comparison on the KITTI _val_ split evaluated by the AP
with 40 recall positions at moderate difficulty level.

Method |  Car $\uparrow$ | Pedestrian$\uparrow$ | Cyclist$\uparrow$ | Method | Car$\uparrow$ | Pedestrian$\uparrow$ | Cyclist$\uparrow$
---|---|---|---|---|---|---|---
SECOND [45] | 81.50 | 48.82 | 65.72 | PV-RCNN [31] | 84.50 | 57.06 | 70.14
ALSO [66] \+ SECOND | $81.97^{{\color[rgb]{0,.5,.5}+0.47}}$ | $51.93^{{\color[rgb]{0,.5,.5}+3.11}}$ | ${\bf 69.14}^{{\color[rgb]{0,.5,.5}+3.42}}$ | ALSO [66] \+ PV-RCNN | $84.72^{{\color[rgb]{0,.5,.5}+0.22}}$ | $58.49^{{\color[rgb]{0,.5,.5}+1.43}}$ | $75.06^{{\color[rgb]{0,.5,.5}+4.92}}$
Occupancy-MAE + SECOND | ${\bf 81.98}^{{\color[rgb]{0,.5,.5}+0.48}}$ | ${\bf 53.67}^{{\color[rgb]{0,.5,.5}+4.85}}$ | $69.08^{{\color[rgb]{0,.5,.5}+3.36}}$ | Occupancy-MAE + PV-RCNN | ${\bf 84.82}^{{\color[rgb]{0,.5,.5}+0.32}}$ | ${\bf 59.07}^{{\color[rgb]{0,.5,.5}+2.01}}$ | ${\bf 75.68}^{{\color[rgb]{0,.5,.5}+5.54}}$

TABLE VI: Performance comparison on the KITTI _val_ split with AP calculated
by 11 recall positions evaluating bounding box and orientation.

Evaluation | Method | Car | Pedestrian | Cyclist
---|---|---|---|---
Easy$\uparrow$ | Moderate$\uparrow$ | Hard$\uparrow$ | Easy$\uparrow$ | Moderate$\uparrow$ | Hard$\uparrow$ | Easy$\uparrow$ | Moderate$\uparrow$ | Hard$\uparrow$
bbox | SECOND [45] | 90.73 | 89.76 | 88.94 | 68.70 | 65.27 | 62.52 | 87.88 | 75.43 | 71.67
Occupancy-MAE + SECOND | ${\bf 94.81}^{{\color[rgb]{0,.5,.5}+4.08}}$ | ${\bf 89.98}^{{\color[rgb]{0,.5,.5}+0.22}}$ | ${\bf 89.35}^{{\color[rgb]{0,.5,.5}+0.41}}$ | ${\bf 70.37}^{{\color[rgb]{0,.5,.5}+1.67}}$ | ${\bf 67.45}^{{\color[rgb]{0,.5,.5}+2.18}}$ | ${\bf 65.14}^{{\color[rgb]{0,.5,.5}+2.62}}$ | ${\bf 91.82}^{{\color[rgb]{0,.5,.5}+3.94}}$ | ${\bf 78.65}^{{\color[rgb]{0,.5,.5}+3.22}}$ | ${\bf 73.77}^{{\color[rgb]{0,.5,.5}+2.10}}$
aos | SECOND [45] | 90.73 | 89.63 | 88.70 | 63.46 | 60.13 | 56.93 | 87.63 | 74.67 | 71.00
Occupancy-MAE + SECOND | ${\bf 94.66}^{{\color[rgb]{0,.5,.5}+3.93}}$ | ${\bf 89.88}^{{\color[rgb]{0,.5,.5}+0.25}}$ | ${\bf 88.92}^{{\color[rgb]{0,.5,.5}+0.22}}$ | ${\bf 65.33}^{{\color[rgb]{0,.5,.5}+1.97}}$ | ${\bf 61.55}^{{\color[rgb]{0,.5,.5}+1.42}}$ | ${\bf 59.23}^{{\color[rgb]{0,.5,.5}+2.30}}$ | ${\bf 91.57}^{{\color[rgb]{0,.5,.5}+1.94}}$ | ${\bf 78.42}^{{\color[rgb]{0,.5,.5}+3.75}}$ | ${\bf 73.50}^{{\color[rgb]{0,.5,.5}+2.50}}$

TABLE VII: Quantitative detection performance on the Waymo _val_ set. The
models are trained on 20% Waymo training set.

Method | Vec_L1 | Vec_L2 | Ped_L1 | Ped_L2 | Cyc_L1 | Cyc_L2
---|---|---|---|---|---|---
AP$\uparrow$ | APH$\uparrow$ | AP$\uparrow$ | APH$\uparrow$ | AP$\uparrow$ | APH$\uparrow$ | AP$\uparrow$ | APH$\uparrow$ | AP$\uparrow$ | APH$\uparrow$ | AP$\uparrow$ | APH$\uparrow$
CenterPoint [42] | 71.33 | 70.76 | 63.16 | 62.65 | 72.09 | 65.49 | 64.27 | 58.23 | 68.68 | 67.39 | 66.11 | 64.87
GCC-3D [71] \+ CenterPoint | - | - | $63.97^{{\color[rgb]{0,.5,.5}+0.81}}$ | $63.47^{{\color[rgb]{0,.5,.5}+0.82}}$ | - | - | $64.23^{{\color[rgb]{0,.5,.5}-0.04}}$ | $58.47^{{\color[rgb]{0,.5,.5}+0.24}}$ | - | - | $67.68^{{\color[rgb]{0,.5,.5}+1.57}}$ | $66.44^{{\color[rgb]{0,.5,.5}+1.57}}$
Occupancy-MAE + CenterPoint | ${\bf 71.89}^{{\color[rgb]{0,.5,.5}+0.56}}$ | ${\bf 71.33}^{{\color[rgb]{0,.5,.5}+0.57}}$ | ${\bf 64.05}^{{\color[rgb]{0,.5,.5}+0.89}}$ | ${\bf 63.53}^{{\color[rgb]{0,.5,.5}+0.88}}$ | ${\bf 73.85}^{{\color[rgb]{0,.5,.5}+1.76}}$ | ${\bf 67.12}^{{\color[rgb]{0,.5,.5}+1.63}}$ | ${\bf 65.78}^{{\color[rgb]{0,.5,.5}+1.51}}$ | ${\bf 59.62}^{{\color[rgb]{0,.5,.5}+1.39}}$ | ${\bf 70.29}^{{\color[rgb]{0,.5,.5}+1.61}}$ | ${\bf 69.03}^{{\color[rgb]{0,.5,.5}+1.64}}$ | ${\bf 67.76}^{{\color[rgb]{0,.5,.5}+1.65}}$ | ${\bf 66.53}^{{\color[rgb]{0,.5,.5}+1.66}}$
PV-RCNN [41] | 75.41 | 74.74 | 67.44 | 66.80 | 71.98 | 61.24 | 63.70 | 53.95 | 65.88 | 64.25 | 63.39 | 61.82
Occupancy-MAE + PV-RCNN | ${\bf 75.94}^{{\color[rgb]{0,.5,.5}+9.53}}$ | ${\bf 75.28}^{{\color[rgb]{0,.5,.5}+0.54}}$ | ${\bf 67.94}^{{\color[rgb]{0,.5,.5}+0.50}}$ | ${\bf 67.34}^{{\color[rgb]{0,.5,.5}+0.54}}$ | ${\bf 74.02}^{{\color[rgb]{0,.5,.5}+2.04}}$ | ${\bf 63.48}^{{\color[rgb]{0,.5,.5}+2.24}}$ | ${\bf 64.91}^{{\color[rgb]{0,.5,.5}+1.21}}$ | ${\bf 55.57}^{{\color[rgb]{0,.5,.5}+1.62}}$ | ${\bf 67.21}^{{\color[rgb]{0,.5,.5}+1.33}}$ | ${\bf 65.49}^{{\color[rgb]{0,.5,.5}+1.24}}$ | ${\bf 64.62}^{{\color[rgb]{0,.5,.5}+1.23}}$ | ${\bf 63.02}^{{\color[rgb]{0,.5,.5}+1.20}}$

TABLE VIII: Quantitative detection performance on the Waymo _val_ set. The
models are trained on 100% Waymo training set.

Method | Vec_L1 | Vec_L2 | Ped_L1 | Ped_L2 | Cyc_L1 | Cyc_L2
---|---|---|---|---|---|---
AP$\uparrow$ | APH$\uparrow$ | AP$\uparrow$ | APH$\uparrow$ | AP$\uparrow$ | APH$\uparrow$ | AP$\uparrow$ | APH$\uparrow$ | AP$\uparrow$ | APH$\uparrow$ | AP$\uparrow$ | APH$\uparrow$
SECOND [6] | 72.27 | 71.69 | 63.85 | 63.33 | 68.70 | 58.18 | 60.72 | 51.31 | 60.62 | 59.28 | 58.34 | 57.05
ProposalContrast [63] \+ SECOND | - | - | $64.50^{{\color[rgb]{0,.5,.5}+0.65}}$ | $63.90^{{\color[rgb]{0,.5,.5}+0.57}}$ | - | - | $60.33^{{\color[rgb]{0,.5,.5}-0.39}}$ | $51.00^{{\color[rgb]{0,.5,.5}-0.31}}$ | - | - | $57.90^{{\color[rgb]{0,.5,.5}-0.44}}$ | $56.60^{{\color[rgb]{0,.5,.5}-0.45}}$
BEV-MAE [72] \+ SECOND | - | - | $64.42^{{\color[rgb]{0,.5,.5}+0.57}}$ | $63.87^{{\color[rgb]{0,.5,.5}+0.54}}$ | - | - | $59.97^{{\color[rgb]{0,.5,.5}-0.75}}$ | $50.65^{{\color[rgb]{0,.5,.5}-0.66}}$ | - | - | $58.69^{{\color[rgb]{0,.5,.5}+0.35}}$ | $57.39^{{\color[rgb]{0,.5,.5}+0.34}}$
Occupancy-MAE + SECOND | ${\bf 73.68}^{{\color[rgb]{0,.5,.5}+1.41}}$ | ${\bf 73.12}^{{\color[rgb]{0,.5,.5}+1.43}}$ | ${\bf 65.72}^{{\color[rgb]{0,.5,.5}+1.97}}$ | ${\bf 65.19}^{{\color[rgb]{0,.5,.5}+1.86}}$ | ${\bf 70.21}^{{\color[rgb]{0,.5,.5}+1.51}}$ | ${\bf 60.45}^{{\color[rgb]{0,.5,.5}+2.27}}$ | ${\bf 62.52}^{{\color[rgb]{0,.5,.5}+1.80}}$ | ${\bf 53.69}^{{\color[rgb]{0,.5,.5}+2.38}}$ | ${\bf 63.13}^{{\color[rgb]{0,.5,.5}+2.51}}$ | ${\bf 61.88}^{{\color[rgb]{0,.5,.5}+2.60}}$ | ${\bf 60.83}^{{\color[rgb]{0,.5,.5}+2.49}}$ | ${\bf 59.63}^{{\color[rgb]{0,.5,.5}+2.58}}$

TABLE IX: Quantitative detection performance achieved by different methods on the nuScenes _val_ set. Method | Frames | mAP$\uparrow$ | NDS$\uparrow$ | mATE$\downarrow$ | mASE$\downarrow$ | mAOE$\downarrow$ | mAVE$\downarrow$ | mAAE$\downarrow$
---|---|---|---|---|---|---|---|---
CenterPoint [42] | - | 56.0 | 64.5 | 30.1 | 25.6 | 38.3 | 21.9 | 18.9
GCC-3D [71] \+ CenterPoint | - | $57.3^{{\color[rgb]{0,.5,.5}+1.3}}$ | $65.0^{{\color[rgb]{0,.5,.5}+0.5}}$ | - | - | - | - | -
BEV-MAE [72] \+ CenterPoint | - | $57.2^{{\color[rgb]{0,.5,.5}+1.2}}$ | $65.1^{{\color[rgb]{0,.5,.5}+0.6}}$ | - | - | - | - | -
Occupancy-MAE + CenterPoint | 1 | $56.5^{{\color[rgb]{0,.5,.5}+0.5}}$ | $65.0^{{\color[rgb]{0,.5,.5}+0.5}}$ | 29.7 | 25.2 | 38.4 | 21.5 | 18.7
Occupancy-MAE + CenterPoint | 3 | ${\bf 57.8}^{{\color[rgb]{0,.5,.5}+1.8}}$ | ${\bf 65.9}^{{\color[rgb]{0,.5,.5}+1.4}}$ | 28.8 | 24.4 | 37.5 | 20.6 | 18.0
TABLE X: Quantitative segmentation performance achieved by different methods
on the nuScenes _val_ set.

Method | Epoch | mIoU$\uparrow$ |  barrier |  bicycle |  bus |  car |  construction |  motorcycle |  pedestrian |  traffic-cone |  trailer |  truck |  driveable |  others |  sidewalk |  terrain |  manmade |  vegetation
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Cylinder3D [18] | 15 | 70.22 | 71.5 | 16.7 | 88.2 | 85.2 | 40.5 | 71.5 | 68.5 | 61.1 | 56.7 | 79.9 | 96.0 | 70.9 | 72.3 | 73.2 | 86.2 | 85.3
25 | 70.83 | 72.5 | 17.1 | 88.2 | 85.3 | 46.0 | 71.3 | 73.7 | 58.7 | 61.9 | 81.0 | 96.0 | 66.0 | 71.9 | 71.6 | 86.6 | 85.4
Occupancy-MAE + Cylinder3D | 15 | $71.61^{{\color[rgb]{0,.5,.5}+1.39}}$ | 72.6 | 29.3 | 89.4 | 84.3 | 40.4 | 77.7 | 73.4 | 60.1 | 58.2 | 77.0 | 96.0 | 69.9 | 72.2 | 73.7 | 86.3 | 85.5
25 | ${\bf 72.85}^{{\color[rgb]{0,.5,.5}+2.02}}$ | 74.6 | 33.0 | 89.3 | 85.2 | 40.9 | 78.6 | 75.9 | 62.9 | 61.5 | 79.3 | 96.1 | 70.5 | 72.7 | 72.7 | 87.2 | 85.4

TABLE XI: Quantitative multi-object tracking performance achieved by different methods on the nuScenes _val_ set. Tracker | Detector | AMOTA$\uparrow$ | AMOTP$\downarrow$ | MT$\uparrow$ | ML$\downarrow$ | TP$\uparrow$ | FP$\downarrow$ | FN$\downarrow$ | IDS$\downarrow$ | FRAG$\downarrow$
---|---|---|---|---|---|---|---|---|---|---
GNN-PMB [73] | PointPillars [31] | 0.311 | 1.231 | 2754 | 2236 | 60929 | ${\bf 9993}$ | 39945 | 1023 | 769
Occupancy-MAE + PointPillars | ${\bf 0.324}^{{\color[rgb]{0,.5,.5}+1.3}}$ | ${\bf 1.121}^{{\color[rgb]{0,.5,.5}+1.0}}$ | ${\bf 2912}$ | ${\bf 2089}$ | ${\bf 65241}$ | 10117 | ${\bf 38083}$ | ${\bf 906}$ | ${\bf 698}$
CenterPoint [42] | 0.707 | 0.560 | 4608 | 1347 | 83134 | 12362 | 18113 | 650 | 345
Occupancy-MAE + CenterPoint | ${\bf 0.719}^{{\color[rgb]{0,.5,.5}+1.2}}$ | ${\bf 0.548}^{{\color[rgb]{0,.5,.5}+1.2}}$ | ${\bf 4801}$ | ${\bf 1211}$ | ${\bf 84251}$ | ${\bf 10152}$ | ${\bf 16993}$ | ${\bf 498}$ | ${\bf 284}$
TABLE XII: Quantitative results of adaptation tasks. Our Occupancy-MAE
improves the performance of UDA 3D object detection.

Task | Method | PV-RCNN
---|---|---
BEV$\uparrow$ | 3D$\uparrow$
Waymo $\rightarrow$ KITTI | Oracle | 88.98 | 82.50
Source-only | 61.18 | 22.01
SN [74] | 79.78 | 63.60
ST3D [69] | 84.10 | 64.78
ST3D(w/SN) [69] | 86.65 | 76.86
Occupancy-MAE + ST3D | ${\bf 85.52}^{{\color[rgb]{0,.5,.5}+1.42}}$ | ${\bf 65.24}^{{\color[rgb]{0,.5,.5}+0.46}}$
| Occupancy-MAE + ST3D(w/SN) | ${\bf 87.15}^{{\color[rgb]{0,.5,.5}+0.50}}$ | ${\bf 77.13}^{{\color[rgb]{0,.5,.5}+0.27}}$
nuScenes $\rightarrow$ KITTI | Oracle | 88.98 | 82.50
Source-only | 68.15 | 37.17
SN [74] | 60.48 | 49.47
ST3D [69] | 78.36 | 70.85
ST3D(w/SN) [69] | 84.29 | 72.94
Occupancy-MAE + ST3D | ${\bf 78.66}^{{\color[rgb]{0,.5,.5}+0.30}}$ | ${\bf 71.24}^{{\color[rgb]{0,.5,.5}+0.39}}$
| Occupancy-MAE + ST3D(w/SN) | ${\bf 85.43}^{{\color[rgb]{0,.5,.5}+1.14}}$ | ${\bf 73.22}^{{\color[rgb]{0,.5,.5}+0.28}}$

###  IV-B Experimental Setup

We evaluate the effectiveness of our proposed model by conducting three
downstream tasks on four autonomous driving datasets [5, 2, 3, 4]. To
implement 3D object detection and unsupervised domain adaptation tasks, we
utilize the popular point clouds detection codebase OpenPCDet [70] (version
0.5.2). For the 3D semantic segmentation task, we use the open-sourced
Cylinder3D [18] as the pre-trained backbone, which applies the cylindrical
voxel partition. We first pre-train the Occupancy-MAE on the unlabeled raw set
of the ONCE dataset and then fine-tune the perception models on the training
set. However, for the KITTI, Waymo, and nuScenes datasets, pre-training and
fine-tuning are both on the training set. The ONCE dataset provides a
benchmark for self-supervised learning methods, but no codes are available.
Thus, we only compare our Occupancy-MAE with self-supervised learning methods
on the ONCE dataset.

We utilize the pre-trained 3D encoder to initialize and warm up the backbones
of the downstream tasks, without freezing the 3D encoder parameters during
fine-tuning. Subsequently, we train the downstream tasks using the same
training parameters as the original models. We set the masking ratio for
voxels within 0-30 meters, 30-50 meters, and $>$ 50 meters to 90%, 70%, and
50%, respectively. The number of pre-training epochs is 3. For more detailed
parameter setups, please refer to OpenPCDet [70, 5] and Cylinder3D [18].

###  IV-C Results on Downstream Tasks

The experimental results have been validated multiple times with different
seeds.

####  IV-C1 3D Object Detection

We begin by comparing our proposed method with several state-of-the-art self-
supervised learning methods on the ONCE dataset [5] _val_ set. These include
two contrastive learning methods ( BYOL [62] and ProposalContrast [63]), two
clustering-based methods (DeepCluster [59] and SwAV [60]). Table II shows that
our method, with the SECOND [6] detector as the pre-trained backbone,
outperforms the self-supervised methods in terms of mAP. Our method achieves a
0.2% $\sim$ 6% performance gain over contrastive learning methods because the
contrastive views of a 3D scene by data augmentations may be similar, causing
the model to converge to a trivial solution [5]. Our Occupancy-MAE achieves
comparable results with SwAV [60] and DeepCluster [59], with marginal
improvement, but it is much simpler than clustering-based methods. We
attribute the performance gain of our method to the range-aware masking
strategy and occupancy prediction task, which help to learn high-level
semantic information for large-scale outdoor LiDAR point clouds.

We present the performance of Occupancy-MAE on the KITTI test set in Table IV.
Due to the limited submission times of 3 for KITTI policy, we only evaluate
Occupancy-MAE with SECOND [6]. The KITTI test results of SECOND are reproduced
by PV-RCNN [41] and the pedestrian class was not provided in PV-RCNN. Our
method significantly improves 3D object detection results on the moderate
level of cyclist class, from 52.08% to 54.84%, and on car class, from 72.55%
to 72.87%. For bird-view detection of cyclist class, our approach outperforms
training from scratch by increasing mAP by 4.62% on moderate levels of cyclist
class.

As shown in Table V, Occupancy-MAE achieves better performance than training
from scratch on the KITTI val set, especially for the small objects, i.e.,
pedestrian and cyclist classes, where the pre-trained model achieves about
1.5% and 0.5% gains, respectively. ALSO [66] shares similarities with our
method as both involve predicting occupancy as an auxiliary task. However, our
approach goes a step further by constructing a mask autoencoder framework,
which enables the learning of more semantic information, thereby enhancing the
precision of detection. We also observe that our method improves the
evaluation of the bounding box and orientation in Table VI, indicating that
Occupancy-MAE can effectively learn high-level semantic information from the
masked voxels.

Pillar-based methods, such as PointPillars [31] and SST [25], are variants of
voxel-based methods with high computational efficiency. Pillar-based methods
process point clouds in vertical columns (pillars), which may contain points
of the road. We only consider the points above the road to build the occupancy
labels. In Table IV, we show that Occupancy-MAE improves training from scratch
by approximately 1% $\sim$ 2% performance gains on the pedestrian and car
classes. Additionally, Occupancy-MAE outperforms Voxel-MAE [27], MV-JAR [28],
GeoMAE [29], and GD-MAE [30], as the target of Occupancy-MAE (occupancy
structure above the road) is more reasonable than regression of the points and
pillar occupancy of Voxel-MAE [27], MV-JAR [28], GeoMAE [29], and GD-MAE [30].
Besides, They are limited to pillar-based methods, whereas our Occupancy-MAE
offers a wider range of applicability, treats points differently and
reconstructs the overall 3D occupancy distribution.

We evaluate Occupancy-MAE on the Waymo val set and present the results with
20% training data in Table VII and with 100% training data in Table VIII. Our
pre-trained model outperforms training from scratch, especially for small
objects, achieving approximately 1% $\sim$ 2% performance gains on the
pedestrian and cyclist classes. Small objects contain only a few points,
making them challenging to detect. The masked occupancy reconstruction task
enables the network to fill in missing occupancy information, thereby
improving the detection of small objects. We can see that pre-training with
Occupancy-MAE leads to consistent performance gains over training from
scratch, with around 1% mAP improvement with 20% training data and 2% mAP
improvement with 100% training data. This further confirms that our method
benefits from more self-supervised pre-training data.

Lastly, we verify the effectiveness of Occupancy-MAE on nuScenes dataset and
report the results in Table IX. The performance improvement of our Occupancy-
MAE on nuScenes (32-beam) is limited because the point clouds are sparser than
those in KITTI and Waymo (64-beam). Hence, we perform fusion on multiple
frames of point clouds to construct dense occupancy labels. The results
demonstrate that the fusion of 3 frames of point clouds yields an improvement
of 1.8 in mAP and 1.4 in NDS. This suggests that the utilization of multi-
frame fusion enhances the model’s ability to learn contextual information
about the surrounding scene, as it provides valuable insights into the overall
structural characteristics of objects beyond the surface-level information
offered by single-frame point clouds. We report results for SECOND [6],
CenterPoint [42], and PV-RCNN [41] from OpenPCDet [70], BEV-MAE [72], and ONCE
[5].

####  IV-C2 3D Semantic Segmentation

TABLE XIII: Comparison of different pretext tasks.

Target | Car$\uparrow$ | Pedestrian$\uparrow$ | Cyclist$\uparrow$
---|---|---|---
PV-RCNN [41] | 83.61 | 57.90 | 70.47
Features Regression | $83.60^{{\color[rgb]{0,.5,.5}-0.01}}$ | $57.92^{{\color[rgb]{0,.5,.5}+0.02}}$ | $70.52^{{\color[rgb]{0,.5,.5}+0.05}}$
Occupancy Classification | ${\bf 83.82}^{{\color[rgb]{0,.5,.5}+0.21}}$ | ${\bf 59.37}^{{\color[rgb]{0,.5,.5}+1.47}}$ | ${\bf 71.99}^{{\color[rgb]{0,.5,.5}+1.52}}$

TABLE XIV: Impacts of pre-training times. Epoch |  Easy$\uparrow$ |  Moderate$\uparrow$ |  Hard$\uparrow$
---|---|---|---
2 |  81.26 |  67.23 |  61.68
3 |  82.09 |  68.05 |  62.08
4 |  82.12 |  68.00 |  61.99
Figure 3: (a) Data efficiency of Occupancy-MAE. (b) Comparison of different masking strategies. TABLE XV: Impacts of the amount of pre-training data. Data size | 120k | 240k | 360k | 480k | 600k
---|---|---|---|---|---
mAP$\uparrow$ | 52.51 | 52.71 | 53.23 | 53.57 | 53.61
TABLE XVI: Impacts of masking ratio. Masking ratio | mAP$\uparrow$
---|---
0-30m | 30-50m | 50m-inf | overall | 0-30m | 30-50m | 50m-inf
0.5 | 0.5 | 0.5 | 24.07 | 26.42 | 22.33 | 17.57
0.7 | 0.5 | 0.3 | 26.29 | 29.10 | 23.38 | 17.80
0.7 | 0.7 | 0.7 | 26.57 | 29.31 | 24.93 | 16.01
0.9 | 0.7 | 0.5 | 29.15 | 33.06 | 26.00 | 18.27
0.9 | 0.7 | 0.7 | 27.94 | 31.25 | 24.49 | 18.06
0.9 | 0.9 | 0.9 | 27.49 | 30.54 | 25.28 | 16.11
0.95 | 0.95 | 0.95 | 23.68 | 26.10 | 21.65 | 15.98
0.98 | 0.98 | 0.98 | 21.61 | 23.88 | 19.16 | 15.74

For self-supervised learning on the 3D semantic segmentation task, we designed
the lightweight decoder of Occupancy-MAE with only two 3D Deconv layers. Table
X shows that Occupancy-MAE outperforms training from scratch by around 2% mIoU
on nuScenes _val_ set. This implies that Occupancy-MAE is well suited for 3D
semantic segmentation task since our masked voxel classification objective
promotes the model to capture the occupancy distribution of the 3D space,
which is also very important for the dense prediction task, i.e. 3D semantic
segmentation. Results in Table X were obtained by retraining Cylinder3D [18].

####  IV-C3 Multi-object Tracking

We also validated the performance of the proposed pretraining algorithm on
LiDAR-based multi-object tracking tasks. Using GNN-PMB [73] as the baseline,
as shown in Table XI, after employing our proposed Occupancy-MAE pre-training
algorithm, the multi-object tracking algorithms with detectors PointPillars
[31] and CenterPoint [42] showed an improvement of approximately 1% in terms
of AMOTA and AMOTP. This indicates that our pre-training algorithm is capable
of learning representative features, enhancing 3D object detection
performance, and consequently improving multi-object tracking performance.

####  IV-C4 Unsupervised Domain Adaptation

The domain shifts in LiDAR-based 3D perception models are apparent as the
LiDARs have different patterns. Evaluation of datasets captured in different
locations or sensors results in a drop in model performance. To further
investigate the generalization ability of Occupancy-MAE in representation
learning, we conduct experiments in two scenarios of unsupervised domain
adaptative (UDA) 3D object detection: different collection locations and time
(i.e. Waymo $\rightarrow$ KITTI) and different LiDAR ring numbers (i.e.
nuScenes $\rightarrow$ KITTI). We compare our method with Oracle, Source-only,
and ST3D [69]. We follow the setting of the SOTA UDA method ST3D. We first
train Occupancy-MAE on the source and target data simultaneously and then
apply the pre-trained model to warm up the object detector in ST3D. Table XII
shows that pre-training with Occupancy-MAE further narrows the gap between UDA
and Oracle by about 0.3% $\sim$ 1% mAP. Our pre-trained network is voxel-
awareness of the object shape of the target data, enabling the UDA model to
generate high-quality pseudo-labels for self-training.

##  V Further Analysis

We will delve into a comprehensive analysis of the key components of the
proposed Occupancy-MAE.

Figure 4: Qualitative results achieved on the KITTI _test_ set. With the pre-
training of our Occupancy-MAE, the 3D detector can learn more robust features
to reduce missed and false detection.

####  V-1 Data-efficient Learner

Pre-training enables fine-tuning of models using limited labeled data. To
examine the data efficiency of Occupancy-MAE, we conducted experiments with
varying amounts of labelled data used for fine-tuning, using SECOND [6] as the
backbone, and evaluated its detection performance on the KITTI _val_ set. Our
results, shown in Figure 1 and Figure 3 (a), indicate that using Occupancy-MAE
with 50%, 75%, and 75% of labelled data, respectively, results in the same
performance as training from scratch with the full dataset for the car,
pedestrian, and cyclist classes. With only 25% of the samples for fine-tuning,
our Occupancy-MAE model improves performance by 1% $\sim$ 2% mAP over training
from scratch, highlighting its data efficiency and ability to reduce the need
for costly human-annotated 3D data.

####  V-2 Range-aware Random Masking

We study the effectiveness of the proposed range-aware random masking
strategy. For range-aware random masking, the masking ratio for voxels within
0-30 meters, 30-50 meters, and $>$ 50 meters is set to 90%, 70%, and 50%,
respectively. We compare it with random masking (the masking ratio is set to
90% for all voxels). We can see from Figure 3 (b) that range-aware random
masking surpasses random masking by 2.5%, 0.7%, and 2.1% mAP with the ONCE raw
set as the pre-training datasets. This demonstrates that the masking ratio for
large-scale point clouds should be inversely proportional to the distance from
the LiDAR sensor.

####  V-3 Occupancy prediction pretext task

The key to successful self-supervised learning lies in selecting a suitable
pretext task. In this section, we evaluate the effectiveness of the 3D
occupancy prediction pretext task and compare it with the voxel feature
regression loss on the KITTI _val_ set, which replaces the binary occupancy
classification loss. The results presented in Table XIII demonstrate that the
performance of the voxel regression task is comparable to training from
scratch. However, the classification task outperforms the regression task by
0.2%, 1.6%, and 1.4% in terms of mAP on the car, pedestrian, and cyclist
classes, respectively. We believe that the occupancy prediction task is more
challenging, and determining whether a voxel contains points is crucial for 3D
perception models. By employing the simple 3D occupancy classification task,
the pre-trained network becomes more aware of the object’s shape, leading to
enhanced performance in downstream tasks.

##  VI Ablation Studies

In this section, we perform more ablation experiments to investigate the
individual components of our Occupancy-MAE. All experiments are conducted with
the 3D object detector SECOND [45].

##### Pre-training Times

In Table XIV, we first study the influence of pre-training times on the KITTI
_val_ set. The results show that the performance initially increases with
increasing pre-training times and then reaches a stable state. In our
experiments, we pre-train Occupancy-MAE for three epochs.

##### Pre-training Data

We then investigate the impact of the amount of pre-training data. As shown in
Table XV, the performance of Occupancy-MAE improves with an increase in the
amount of unlabeled data on the ONCE _medium_ raw set. As autonomous driving
companies have access to a large amount of unlabeled data, Occupancy-MAE can
utilize this readily available data to enhance the performance of 3D
perception in practice.

##### Masking Ratio

In this section, we investigate the influence of the voxel masking ratio on
the cyclist class of ONCE _small_ raw set. As shown in Table XVI, when
increasing the masking ratio to 90%, the overall performance boosts from
23.27% to 29.15%. For the points within 30-50 meters and $>$ 50 meters, the
masking ratio of 70% and 50% are suitable for downstream tasks, respectively.
The proper masking ratio enforces networks to capture the gestalt of the 3D
scene to learn representative features.

##  VII Visualization

Figure 4 shows the visualization of predictions by SECOND [6] and SECOND with
pre-training of our Occupancy-MAE. The limited resolution of LiDAR will lead
to missed and false detection of faraway objects. With our Occupancy-MAE, the
detectors will learn more robust features to improve detection as shown in
Figure 4 (1). Occlusion is another challenging problem in 3D object detection,
as shown in Figure 4 (2), the masked voxel classification task pushes
Occupancy-MAE to be voxel-aware of the whole shape of objects, thus improving
the detection for the Occlusion areas.

##  VIII Limitations

Our pre-training target of occupancy prediction relies on dense
representations, and 3D convolutions in decoder cannot handle high-resolution
feature representations. This will be the focus of our next research phase,
where we may consider employing a cascading structure or sparse 3D
convolutions for pre-training via masked occupancy prediction.

##  IX Conclusion

In this paper, we have presented Occupancy-MAE, a self-supervised masked
occupancy autoencoding framework designed for pre-training large-scale outdoor
LiDAR point clouds. By leveraging vast amounts of unlabeled data in autonomous
driving, Occupancy-MAE reduces the reliance on labelled 3D training data. The
framework predicts the masked occupancy structure of the entire 3D
environment, compelling the network to learn robust features for accurately
deducing the masked points. We have also introduced the range-aware random
masking strategy, which enhances the pre-training performance for distant
objects. Extensive experimental results have demonstrated the effectiveness of
Occupancy-MAE across various downstream tasks, including 3D object detection,
semantic segmentation, multi-object tracking, and unsupervised domain
adaptation. Occupancy-MAE has consistently outperformed training from scratch,
showcasing its ability to improve the accuracy and performance of perception
models in autonomous driving. Looking ahead, our future work will explore the
pre-training of high-resolution feature maps, 4D scene LiDAR point clouds by
incorporating temporal multi-frame data fusion, and exploring to learn general
representations across different large-scale outdoor LiDAR point clouds
datasets. Overall, Occupancy-MAE represents a promising approach to address
the challenges of labelled data scarcity in autonomous driving, offering a
cost-effective and efficient solution for enhancing the 3D perception ability
of autonomous vehicles.

## References

  * [1] J. Liu, W. Xiong, L. Bai, Y. Xia, T. Huang, W. Ouyang, and B. Zhu, “Deep instance segmentation with automotive radar detection points,” _IEEE Transactions on Intelligent Vehicles_ , vol. 8, no. 1, pp. 84–94, 2022.
  * [2] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in _2012 IEEE conference on computer vision and pattern recognition_. IEEE, 2012, pp. 3354–3361.
  * [3] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine _et al._ , “Scalability in perception for autonomous driving: Waymo open dataset,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 2446–2454.
  * [4] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 11 621–11 631.
  * [5] J. Mao, M. Niu, C. Jiang, H. Liang, J. Chen, X. Liang, Y. Li, C. Ye, W. Zhang, Z. Li _et al._ , “One million scenes for autonomous driving: Once dataset,” _arXiv preprint arXiv:2106.11037_ , 2021.
  * [6] Y. Yan, Y. Mao, and B. Li, “Second: Sparsely embedded convolutional detection,” _Sensors_ , vol. 18, no. 10, p. 3337, 2018.
  * [7] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [8] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 16 000–16 009.
  * [9] X. Chen, M. Ding, X. Wang, Y. Xin, S. Mo, Y. Wang, S. Han, P. Luo, G. Zeng, and J. Wang, “Context autoencoder for self-supervised representation learning,” _arXiv preprint arXiv:2202.03026_ , 2022.
  * [10] Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu, “Simmim: A simple framework for masked image modeling,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 9653–9663.
  * [11] Y. Pang, W. Wang, F. E. Tay, W. Liu, Y. Tian, and L. Yuan, “Masked autoencoders for point cloud self-supervised learning,” _arXiv preprint arXiv:2203.06604_ , 2022.
  * [12] X. Yu, L. Tang, Y. Rao, T. Huang, J. Zhou, and J. Lu, “Point-bert: Pre-training 3d point cloud transformers with masked point modeling,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 19 313–19 322.
  * [13] H. Liu, M. Cai, and Y. J. Lee, “Masked discrimination for self-supervised learning on point clouds,” _arXiv preprint arXiv:2203.11183_ , 2022.
  * [14] R. Zhang, Z. Guo, P. Gao, R. Fang, B. Zhao, D. Wang, Y. Qiao, and H. Li, “Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training,” _arXiv preprint arXiv:2205.14401_ , 2022.
  * [15] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su _et al._ , “Shapenet: An information-rich 3d model repository,” _arXiv preprint arXiv:1512.03012_ , 2015.
  * [16] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 5828–5839.
  * [17] C. Xu, B. Wu, Z. Wang, W. Zhan, P. Vajda, K. Keutzer, and M. Tomizuka, “Squeezesegv3: Spatially-adaptive convolution for efficient point-cloud segmentation,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVIII 16_. Springer, 2020, pp. 1–19.
  * [18] X. Zhu, H. Zhou, T. Wang, F. Hong, Y. Ma, W. Li, H. Li, and D. Lin, “Cylindrical and asymmetrical 3d convolution networks for lidar segmentation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 9939–9948.
  * [19] S. Xie, J. Gu, D. Guo, C. R. Qi, L. Guibas, and O. Litany, “Pointcontrast: Unsupervised pre-training for 3d point cloud understanding,” in _European conference on computer vision_. Springer, 2020, pp. 574–591.
  * [20] X. Li, H. Ding, W. Zhang, H. Yuan, J. Pang, G. Cheng, K. Chen, Z. Liu, and C. C. Loy, “Transformer-based visual segmentation: A survey,” _arXiv preprint arXiv:2304.09854_ , 2023.
  * [21] M.-H. Guo, J.-X. Cai, Z.-N. Liu, T.-J. Mu, R. R. Martin, and S.-M. Hu, “Pct: Point cloud transformer,” _Computational Visual Media_ , vol. 7, pp. 187–199, 2021.
  * [22] X. Yan, C. Zheng, Z. Li, S. Wang, and S. Cui, “Pointasnl: Robust point clouds processing using nonlocal neural networks with adaptive sampling,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 5589–5598.
  * [23] J. Mao, Y. Xue, M. Niu, H. Bai, J. Feng, X. Liang, H. Xu, and C. Xu, “Voxel transformer for 3d object detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 3164–3173.
  * [24] C. He, R. Li, S. Li, and L. Zhang, “Voxel set transformer: A set-to-set approach to 3d object detection from point clouds,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 8417–8427.
  * [25] L. Fan, Z. Pang, T. Zhang, Y.-X. Wang, H. Zhao, F. Wang, N. Wang, and Z. Zhang, “Embracing single stride 3d object detector with sparse transformer,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 8458–8468.
  * [26] Z. Liu, X. Yang, H. Tang, S. Yang, and S. Han, “Flatformer: Flattened window attention for efficient point cloud transformer,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 1200–1211.
  * [27] G. Hess, J. Jaxing, E. Svensson, D. Hagerman, C. Petersson, and L. Svensson, “Masked autoencoder for self-supervised pre-training on lidar point clouds,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2023, pp. 350–359.
  * [28] R. Xu, T. Wang, W. Zhang, R. Chen, J. Cao, J. Pang, and D. Lin, “Mv-jar: Masked voxel jigsaw and reconstruction for lidar-based self-supervised pre-training,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 13 445–13 454.
  * [29] X. Tian, H. Ran, Y. Wang, and H. Zhao, “Geomae: Masked geometric target prediction for self-supervised point cloud pre-training,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 13 570–13 580.
  * [30] H. Yang, T. He, J. Liu, H. Chen, B. Wu, B. Lin, X. He, and W. Ouyang, “Gd-mae: generative decoder for mae pre-training on lidar point clouds,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 9403–9414.
  * [31] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, “Pointpillars: Fast encoders for object detection from point clouds,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 12 697–12 705.
  * [32] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [33] A. Elfes, “Using occupancy grids for mobile robot perception and navigation,” _Computer_ , vol. 22, no. 6, pp. 46–57, 1989.
  * [34] H. Fu, H. Xue, and R. Ren, “Fast implementation of 3d occupancy grid for autonomous driving,” in _2020 12th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC)_ , vol. 2. IEEE, 2020, pp. 217–220.
  * [35] R. Mahjourian, J. Kim, Y. Chai, M. Tan, B. Sapp, and D. Anguelov, “Occupancy flow fields for motion forecasting in autonomous driving,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 5639–5646, 2022.
  * [36] S. Hoermann, M. Bach, and K. Dietmayer, “Dynamic occupancy grid prediction for urban autonomous driving: A deep learning approach with fully automatic labeling,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 2056–2063.
  * [37] X. Sun, S. Wang, M. Wang, Z. Wang, and M. Liu, “A novel coding architecture for lidar point cloud sequence,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 4, pp. 5637–5644, 2020.
  * [38] C. Tu, E. Takeuchi, A. Carballo, and K. Takeda, “Point cloud compression for 3d lidar sensor using recurrent neural network with residual blocks,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 3274–3280.
  * [39] X. Sun, H. Ma, Y. Sun, and M. Liu, “A novel point cloud compression algorithm based on clustering,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 2, pp. 2132–2139, 2019.
  * [40] Y. Feng, S. Liu, and Y. Zhu, “Real-time spatio-temporal lidar point cloud compression,” in _2020 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2020, pp. 10 766–10 773.
  * [41] S. Shi, C. Guo, L. Jiang, Z. Wang, J. Shi, X. Wang, and H. Li, “Pv-rcnn: Point-voxel feature set abstraction for 3d object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 10 529–10 538.
  * [42] T. Yin, X. Zhou, and P. Krahenbuhl, “Center-based 3d object detection and tracking,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 11 784–11 793.
  * [43] S. Shi, L. Jiang, J. Deng, Z. Wang, C. Guo, J. Shi, X. Wang, and H. Li, “Pv-rcnn++: Point-voxel feature set abstraction with local vector representation for 3d object detection,” _arXiv preprint arXiv:2102.00463_ , 2021.
  * [44] B. Zhang, J. Yuan, B. Shi, T. Chen, Y. Li, and Y. Qiao, “Uni3d: A unified baseline for multi-dataset 3d object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 9253–9262.
  * [45] J. Yuan, B. Zhang, X. Yan, T. Chen, B. Shi, Y. Li, and Y. Qiao, “Ad-pt: Autonomous driving pre-training with large-scale point cloud dataset,” _arXiv preprint arXiv:2306.00612_ , 2023.
  * [46] S. Shi, X. Wang, and H. Li, “Pointrcnn: 3d object proposal generation and detection from point cloud,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 770–779.
  * [47] Z. Yang, Y. Sun, S. Liu, and J. Jia, “3dssd: Point-based 3d single stage object detector,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 11 040–11 048.
  * [48] Y. Zhou and O. Tuzel, “Voxelnet: End-to-end learning for point cloud based 3d object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 4490–4499.
  * [49] Y. Chen, S. Liu, X. Shen, and J. Jia, “Fast point r-cnn,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 9775–9784.
  * [50] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 652–660.
  * [51] B. Wu, A. Wan, X. Yue, and K. Keutzer, “Squeezeseg: Convolutional neural nets with recurrent crf for real-time road-object segmentation from 3d lidar point cloud,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 1887–1893.
  * [52] A. Milioto, I. Vizzo, J. Behley, and C. Stachniss, “Rangenet++: Fast and accurate lidar semantic segmentation,” in _2019 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2019, pp. 4213–4220.
  * [53] H. Tang, Z. Liu, S. Zhao, Y. Lin, J. Lin, H. Wang, and S. Han, “Searching efficient 3d architectures with sparse point-voxel convolution,” in _European conference on computer vision_. Springer, 2020, pp. 685–702.
  * [54] C. Doersch, A. Gupta, and A. A. Efros, “Unsupervised visual representation learning by context prediction,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1422–1430.
  * [55] S. Gidaris, P. Singh, and N. Komodakis, “Unsupervised representation learning by predicting image rotations,” _arXiv preprint arXiv:1803.07728_ , 2018\.
  * [56] J. Xu, L. Xiao, and A. M. López, “Self-supervised domain adaptation for computer vision tasks,” _IEEE Access_ , vol. 7, pp. 156 694–156 706, 2019\.
  * [57] L. Xiao, J. Xu, D. Zhao, Z. Wang, L. Wang, Y. Nie, and B. Dai, “Self-supervised domain adaptation with consistency training,” in _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 6874–6880.
  * [58] F. M. Carlucci, A. D’Innocente, S. Bucci, B. Caputo, and T. Tommasi, “Domain generalization by solving jigsaw puzzles,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2229–2238.
  * [59] M. Caron, P. Bojanowski, A. Joulin, and M. Douze, “Deep clustering for unsupervised learning of visual features,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 132–149.
  * [60] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 9912–9924, 2020.
  * [61] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 9729–9738.
  * [62] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar _et al._ , “Bootstrap your own latent-a new approach to self-supervised learning,” _Advances in neural information processing systems_ , vol. 33, pp. 21 271–21 284, 2020.
  * [63] J. Yin, D. Zhou, L. Zhang, J. Fang, C.-Z. Xu, J. Shen, and W. Wang, “Proposalcontrast: Unsupervised pre-training for lidar-based 3d object detection,” in _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIX_. Springer, 2022, pp. 17–33.
  * [64] Z. Zhang, R. Girdhar, A. Joulin, and I. Misra, “Self-supervised pretraining of 3d features on any point-cloud,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 10 252–10 263.
  * [65] Z. Tong, Y. Song, J. Wang, and L. Wang, “Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training,” _arXiv preprint arXiv:2203.12602_ , 2022.
  * [66] A. Boulch, C. Sautier, B. Michele, G. Puy, and R. Marlet, “Also: Automotive lidar self-supervision by occupancy estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 13 455–13 465.
  * [67] Z. Zhang, M. Bai, and E. Li, “Implicit surface contrastive clustering for lidar point clouds,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 21 716–21 725.
  * [68] B. Graham, M. Engelcke, and L. Van Der Maaten, “3d semantic segmentation with submanifold sparse convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 9224–9232.
  * [69] J. Yang, S. Shi, Z. Wang, H. Li, and X. Qi, “St3d: Self-training for unsupervised domain adaptation on 3d object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 10 368–10 378.
  * [70] O. D. Team, “Openpcdet: An open-source toolbox for 3d object detection from point clouds,” https://github.com/open-mmlab/OpenPCDet, 2020.
  * [71] H. Liang, C. Jiang, D. Feng, X. Chen, H. Xu, X. Liang, W. Zhang, Z. Li, and L. Van Gool, “Exploring geometry-aware contrast and clustering harmonization for self-supervised 3d object detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 3293–3302.
  * [72] Z. Lin and Y. Wang, “Bev-mae: Bird’s eye view masked autoencoders for outdoor point cloud pre-training,” _arXiv preprint arXiv:2212.05758_ , 2022.
  * [73] J. Liu, L. Bai, Y. Xia, T. Huang, B. Zhu, and Q.-L. Han, “Gnn-pmb: A simple but effective online 3d multi-object tracker without bells and whistles,” _IEEE Transactions on Intelligent Vehicles_ , vol. 8, no. 2, pp. 1176–1189, 2022.
  * [74] Y. Wang, X. Chen, Y. You, L. E. Li, B. Hariharan, M. Campbell, K. Q. Weinberger, and W.-L. Chao, “Train in germany, test in the usa: Making 3d object detectors generalize,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 11 713–11 723.