∎

11institutetext: Jiageng Mao<EMAIL_ADDRESS>
Shaoshuai Shi<EMAIL_ADDRESS>
Xiaogang Wang<EMAIL_ADDRESS>
Hongsheng Li<EMAIL_ADDRESS>
1 The Chinese University of Hong Kong, China
2 Centre for Perceptual and Interactive Intelligence
3 Max Planck Institute for Informatics, Germany

# 3D Object Detection for Autonomous Driving: A Comprehensive Survey

Jiageng Mao 1 Shaoshuai Shi 3 Xiaogang Wang 1,2 Hongsheng Li 1,2

(Received: 7 February 2023 )

###### Abstract

Autonomous driving, in recent years, has been receiving increasing attention
for its potential to relieve drivers’ burdens and improve the safety of
driving. In modern autonomous driving pipelines, the perception system is an
indispensable component, aiming to accurately estimate the status of
surrounding environments and provide reliable observations for prediction and
planning. 3D object detection, which aims to predict the locations, sizes, and
categories of the 3D objects near an autonomous vehicle, is an important part
of a perception system. This paper reviews the advances in 3D object detection
for autonomous driving. First, we introduce the background of 3D object
detection and discuss the challenges in this task. Second, we conduct a
comprehensive survey of the progress in 3D object detection from the aspects
of models and sensory inputs, including LiDAR-based, camera-based, and multi-
modal detection approaches. We also provide an in-depth analysis of the
potentials and challenges in each category of methods. Additionally, we
systematically investigate the applications of 3D object detection in driving
systems. Finally, we conduct a performance analysis of the 3D object detection
approaches, and we further summarize the research trends over the years and
prospect the future directions of this area.

###### Keywords:

3D object detection perception autonomous driving deep learning computer
vision robotics

###### papertype:

generic article

##  1 Introduction

Autonomous driving, which aims to enable vehicles to perceive the surrounding
environments intelligently and move safely with little or no human effort, has
attained rapid progress in recent years. Autonomous driving techniques have
been broadly applied in many scenarios, including self-driving trucks,
robotaxis, delivery robots, etc., and are capable of reducing human error and
enhancing road safety. As a core component of autonomous driving systems,
automotive perception helps autonomous vehicles understand the surrounding
environments with sensory input. Perception systems generally take multi-
modality data (images from cameras, point clouds from LiDAR scanners, high-
definition maps etc.) as input, and predict the geometric and semantic
information of critical elements on a road. High-quality perception results
serve as reliable observations for the following steps such as object
tracking, trajectory prediction, and path planning.

To obtain a comprehensive understanding of driving environments, many vision
tasks can be involved in a perception system, e.g. object detection and
tracking, lane detection, and semantic and instance segmentation. Among these
perception tasks, 3D object detection is one of the most indispensable tasks
in an automotive perception system. 3D object detection aims to predict the
locations, sizes, and classes of critical objects, e.g. cars, pedestrians,
cyclists, in the 3D space. In contrast to 2D object detection which only
generates 2D bounding boxes on images and ignores the actual distance
information of objects from the ego-vehicle, 3D object detection focuses on
the localization and recognition of objects in the real-world 3D coordinate
system. The geometric information predicted by 3D object detection in real-
world coordinates can be directly utilized to measure the distances between
the ego-vehicle and critical objects, and to further help plan driving routes
and avoid collisions.

3D object detection methods have evolved rapidly with the advances of deep
learning techniques in computer vision and robotics. These methods have been
trying to address the 3D object detection problem from a particular aspect,
e.g. detection from a particular sensory type or data representation, and lack
a systematic comparison with the methods of other categories. Hence a
comprehensive analysis of the strengths and weaknesses of all types of 3D
object detection methods is desirable and can provide some valuable findings
to the research community.

Figure 1: Hierarchically-structured taxonomy of 3D object detection for
autonomous driving.

To this end, we propose to comprehensively review the 3D object detection
methods for autonomous driving applications and provide in-depth analysis and
a systematic comparison on different categories of approaches. Compared to the
existing surveys Arnold et al. (2019); Liang et al. (2021b); Qian et al.
(2021b), our paper broadly covers the recent advances in this area, e.g. 3D
object detection from range images, self-/semi-/weakly-supervised 3D object
detection, 3D detection in end-to-end driving systems. In contrast to the
previous surveys that only focus on detection from point cloud Guo et al.
(2020); Fernandes et al. (2021); Zamanakos et al. (2021), from monocular
images Wu et al. (2020a); Ma et al. (2022), and from multi-modal inputs Wang
et al. (2021h), our paper systematically investigate the 3D object detection
methods from all sensory types and in most application scenarios. The major
contributions of this work can be summarized as follows:

  * •

We provide a comprehensive review of the 3D object detection methods from
different perspectives, including detection from different sensory inputs
(LiDAR-based, camera-based, and multi-modal detection), detection from
temporal sequences, label-efficient detection, as well as the applications of
3D object detection in driving systems.

  * •

We summarize 3D object detection approaches structurally and hierarchically,
conduct a systematic analysis of these methods, and provide valuable insights
for the potentials and challenges of different categories of methods.

  * •

We conduct a comprehensive performance and speed analysis on the 3D object
detection approaches, identify the research trends over years, and provide
insightful views on the future directions of 3D object detection.

The structure of this paper is organized as follows. First, we introduce the
problem definition, datasets, and evaluation metrics of 3D object detection in
Section 2. Then, we review and analyze the 3D object detection methods based
on LiDAR sensors (Section 3), cameras (Section 4), multi-sensor fusion
(Section 5), and Transformer-based architectures (Section 6). Next, we
introduce the detection methods that leverage temporal data in Section 7 and
utilize fewer labels in Section 8. We subsequently discuss some critical
problems of 3D object detection in driving systems in Section 9. Finally, we
conduct a speed and performance analysis, investigate the research trends, and
prospect the future directions of 3D object detection in Section 10. A
hierarchically-structured taxonomy is shown in Figure 1. We also provide a
constantly updated project page here.

Figure 2: An illustration of 3D object detection in autonomous driving
scenarios.

##  2 Background

###  2.1 What is 3D object detection?

Problem definition. 3D object detection aims to predict bounding boxes of 3D
objects in driving scenarios from sensory inputs. A general formula of 3D
object detection can be represented as

$\mathcal{B}=f_{det}(\mathcal{I}_{sensor}),$ (1)

where $\mathcal{B}=\\{B_{1},\cdots,B_{N}\\}$ is a set of $N$ 3D objects in a
scene, $f_{det}$ is a 3D object detection model, and $\mathcal{I}_{sensor}$ is
one or more sensory inputs. How to represent a 3D object $B_{i}$ is a crucial
problem in this task, since it determines what 3D information should be
provided for the following prediction and planning steps. In most cases, a 3D
object is represented as a 3D cuboid that includes this object, that is

$B=[x_{c},y_{c},z_{c},l,w,h,\theta,class],$ (2)

where $(x_{c},y_{c},z_{c})$ is the 3D center coordinate of a cuboid, $l$, $w$,
$h$ is the length, width, and height of a cuboid respectively, $\theta$ is the
heading angle, i.e. the yaw angle, of a cuboid on the ground plane, and
$class$ denotes the category of a 3D object, e.g. cars, trucks, pedestrians,
cyclists. In Caesar et al. (2020), additional parameters $v_{x}$ and $v_{y}$
that describe the speed of a 3D object along x and y axes on the ground are
employed.

Sensory inputs. There are many types of sensors that can provide raw data for
3D object detection. Among the sensors, radars, cameras, and LiDAR (Light
Detection And Ranging) sensors are the three most widely adopted sensory
types. Radars have long detection range and are robust to different weather
conditions. Due to the Doppler effect, radars could provide additional
velocity measurements. Cameras are cheap and easily accessible, and can be
crucial for understanding semantics, e.g. the type of traffic sign. Cameras
produce images $\mathcal{I}_{cam}\in R^{W\times H\times 3}$ for 3D object
detection, where $W$, $H$ are the width and height of an image, and each pixel
has $3$ RGB channels. Albeit cheap, cameras have intrinsic limitations to be
utilized for 3D object detection. First, cameras only capture appearance
information, and are not capable of directly obtaining 3D structural
information about a scene. On the other hand, 3D object detection normally
requires accurate localization in the 3D space, while the 3D information, e.g.
depth, estimated from images normally has large errors. In addition, detection
from images is generally vulnerable to extreme weather and time conditions.
Detecting objects from images at night or on foggy days is much harder than
detection on sunny days, which leads to the challenge of attaining sufficient
robustness for autonomous driving.

As an alternative solution, LiDAR sensors can obtain fine-grained 3D
structures of a scene by emitting laser beams and then measuring their
reflective information. A LiDAR sensor that emits $m$ beams and conducts
measurements for $n$ times in one scan cycle can produce a range image
$\mathcal{I}_{range}\in R^{m\times n\times 3}$, where each pixel of a range
image contains range $r$, azimuth $\alpha$, and inclination $\phi$ in the
spherical coordinate system as well as the reflective intensity. Range images
are the raw data format obtained by LiDAR sensors, and can be further
converted into point clouds by transforming spherical coordinates into
Cartesian coordinates. A point cloud can be represented as
$\mathcal{I}_{point}\in R^{N\times 3}$, where $N$ denotes the number of points
in a scene, and each point has $3$ channels of xyz coordinates. Both range
images and point clouds contain accurate 3D information directly acquired by
LiDAR sensors. Hence in contrast to cameras, LiDAR sensors are more suitable
for detecting objects in the 3D space, and LiDAR sensors are also less
vulnerable to time and weather changes. However, LiDAR sensors are much more
expensive than cameras, which may limit the applications in driving scenarios.
An illustration of 3D object detection is shown in Figure 2.

Table 1: Datasets for 3D object detection in driving scenarios. Dataset | Year |  | Size
---
(hr.)
| Real-
---
world
| LiDAR
---
scans
Images |  | 3D
---
annotations
Classes | night/rain | Locations | Other data
KITTI Geiger et al. (2012, 2013) | 2012 | 1.5 | Yes | 15k | 15k | 200k | 8 | No/No | Germany | -
KAIST Choi et al. (2018) | 2018 | - | Yes | 8.9k | 8.9k | Yes | 3 | Yes/No | Korea | thermal images
ApolloScape Huang et al. (2019); Ma et al. (2019b) | 2019 | 100 | Yes | 20k | 144k | 475k | 6 | -/- | China | -
H3D Patil et al. (2019) | 2019 | 0.77 | Yes | 27k | 83k | 1.1M | 8 | No/No | USA | -
Lyft L5 Kesten et al. (2019) | 2019 | 2.5 | Yes | 46k | 323k | 1.3M | 9 | No/No | USA | maps
Argoverse Chang et al. (2019) | 2019 | 0.6 | Yes | 44k | 490k | 993k | 15 | Yes/Yes | USA | maps
WoodScape Yogamani et al. (2019) | 2019 | - | Yes | 10k | 10k | - | 3 | Yes/Yes | - | fish-eye camera
AIODrive Weng et al. (2020) | 2020 | 6.9 | No | 250k | 250k | 26M | - | Yes/Yes | - | long-range data
A*3D Pham et al. (2020) | 2020 | 55 | Yes | 39k | 39k | 230k | 7 | Yes/Yes | SG | -
A2D2 Geyer et al. (2020) | 2020 | - | Yes | 12.5k | 41.3k | - | 14 | -/- | Germany | -
Cityscapes 3D Gählert et al. (2020) | 2020 | - | Yes | 0 | 5k | - | 8 | No/No | Germany | -
nuScenes Caesar et al. (2020) | 2020 | 5.5 | Yes | 400k | 1.4M | 1.4M | 23 | Yes/Yes | SG, USA | maps, radar data
Waymo Open Sun et al. (2020c) | 2020 | 6.4 | Yes | 230k | 1M | 12M | 4 | Yes/Yes | USA | maps
Cirrus Wang et al. (2021j) | 2021 | - | Yes | 6.2k | 6.2k | - | 8 | -/- | USA | long-range data
PandaSet Xiao et al. (2021) | 2021 | 0.22 | Yes | 8.2k | 49k | 1.3M | 28 | Yes/Yes | USA | -
KITTI-360 Liao et al. (2021) | 2021 | - | Yes | 80k | 300k | 68k | 37 | -/- | Germany | -
Argoverse v2 Wilson et al. (2021) | 2021 | - | Yes | - | - | - | 30 | Yes/Yes | USA | maps
ONCE Mao et al. (2021b) | 2021 | 144 | Yes | 1M | 7M | 417K | 5 | Yes/Yes | China | -

Analysis: comparisons with 2D object detection. 2D object detection, which
aims to generate 2D bounding boxes on images, is a fundamental problem in
computer vision. 3D object detection methods have borrowed many design
paradigms from the 2D counterparts: proposals generation and refinement,
anchors, non maximum suppression, etc. However, from many aspects, 3D object
detection is not a naive adaptation of 2D object detection methods to the 3D
space. (1) 3D object detection methods have to deal with heterogeneous data
representations. Detection from point clouds requires novel operators and
networks to handle irregular point data, and detection from both point clouds
and images needs special fusion mechanisms. (2) 3D object detection methods
normally leverage distinct projected views to generate object predictions. As
opposed to 2D object detection methods that detect objects from the
perspective view, 3D methods have to consider different views to detect 3D
objects, e.g. from the bird’s-eye view, point view, and cylindrical view. (3)
3D object detection has a high demand for accurate localization of objects in
the 3D space. A decimeter-level localization error can lead to a detection
failure of small objects such as pedestrians and cyclists, while in 2D object
detection, a localization error of several pixels may still maintain a high
Intersection over Union (IoU) between predicted and ground truth bounding
boxes. Hence accurate 3D geometric information is indispensable for 3D object
detection from either point clouds or images.

Analysis: comparisons with indoor 3D object detection. There is also a branch
of works Qi et al. (2018, 2019, 2020); Liu et al. (2021d) on 3D object
detection in indoor scenarios. Indoor datasets, e.g. ScanNet Dai et al.
(2017), SUN RGB-D Song et al. (2015), provide 3D structures of rooms
reconstructed from RGB-D sensors and 3D annotations including doors, windows,
beds, chairs, etc. 3D object detection in indoor scenes is also based on point
clouds or images. However, compared to indoor 3D object detection, there are
unique challenges of detection in driving scenarios. (1) Point cloud
distributions from LiDAR and RGB-D sensors are different. In indoor scenes,
points are relatively uniformly distributed on the scanned surfaces and most
3D objects receive a sufficient number of points on their surfaces. However,
in driving scenes most points fall in a near neighborhood of the LiDAR sensor,
and those 3D objects that are far away from the sensor will receive only a few
points. Thus methods in driving scenarios are specially required to handle
various point cloud densities of 3D objects and accurately detect those
faraway and sparse objects. (2) Detection in driving scenarios has a special
demand for inference latency. Perception in driving scenes has to be real-time
to avoid accidents. Hence those methods are required to be computationally
efficient, otherwise they will not be applied in real-world applications.

###  2.2 Datasets

A large number of driving datasets have been built to provide multi-modal
sensory data and 3D annotations for 3D object detection. Table 1 lists the
datasets that collect data in driving scenarios and provide 3D cuboid
annotations. KITTI Geiger et al. (2012) is a pioneering work that proposes a
standard data collection and annotation paradigm: equipping a vehicle with
cameras and LiDAR sensors, driving the vehicle on roads for data collection,
and annotating 3D objects from the collected data. The following works made
improvements mainly from the $4$ aspects. (1) Increasing the scale of data.
Compared to Geiger et al. (2012), the recent large-scale datasets Sun et al.
(2020c); Caesar et al. (2020); Mao et al. (2021b) have more than 10x point
clouds, images and annotations. (2) Improving the diversity of data. Geiger et
al. (2012) only contains driving data obtained in the daytime and in good
weather, while recent datasets Choi et al. (2018); Chang et al. (2019); Pham
et al. (2020); Caesar et al. (2020); Sun et al. (2020c); Xiao et al. (2021);
Mao et al. (2021b); Wilson et al. (2021) provide data captured at night or in
rainy days. (3) Providing more annotated categories. Some datasets Liao et al.
(2021); Xiao et al. (2021); Geyer et al. (2020); Wilson et al. (2021); Caesar
et al. (2020) can provide more fine-grained object classes, including animals,
barriers, traffic cones, etc. They also provide fine-grained sub-categories of
existing classes, e.g. the adult and child category of the existing pedestrian
class in Caesar et al. (2020). (4) Providing data of more modalities. In
addition to images and point clouds, recent datasets provide more data types,
including high-definition maps Kesten et al. (2019); Chang et al. (2019); Sun
et al. (2020c); Wilson et al. (2021), radar data Caesar et al. (2020), long-
range LiDAR data Weng et al. (2020); Wang et al. (2021j), thermal images Choi
et al. (2018).

Analysis: future prospects of driving datasets. The research community has
witnessed an explosion of datasets for 3D object detection in autonomous
driving scenarios. A subsequent question may be asked: what will the next-
generation autonomous driving datasets look like? Considering the fact that 3D
object detection is not an independent task but a component in driving
systems, we propose that future datasets will include all important tasks in
autonomous driving: perception, prediction, planning, and mapping, as a whole
and in an end-to-end manner, so that the development and evaluation of 3D
object detection methods will be considered from an overall and systematic
view. There are some datasets Sun et al. (2020c); Caesar et al. (2020);
Yogamani et al. (2019) working towards this goal.

Figure 3: Chronological overview of the LiDAR-based 3D object detection
methods.

###  2.3 Evaluation metrics

Various evaluation metrics have been proposed to measure the performance of 3D
object detection methods. Those evaluation metrics can be divided into two
categories. The first category tries to extend the Average Precision (AP)
metric Lin et al. (2014) in 2D object detection to the 3D space:

$AP=\int_{0}^{1}max\\{p(r^{\prime}|r^{\prime}\geq r)\\}dr,$ (3)

where $p(r)$ is the precision-recall curve same as Lin et al. (2014). The
major difference with the 2D AP metric lies in the matching criterion between
ground truths and predictions when calculating precision and recall. KITTI
Geiger et al. (2012) proposes two widely-used AP metrics: $AP_{3D}$ and
$AP_{BEV}$, where $AP_{3D}$ matches the predicted objects to the respective
ground truths if the 3D Intersection over Union (3D IoU) of two cuboids is
above a certain threshold, and $AP_{BEV}$ is based on the IoU of two cuboids
from the bird’s-eye view (BEV IoU). NuScenes Caesar et al. (2020) proposes
$AP_{center}$ where a predicted object is matched to a ground truth object if
the distance of their center locations is below a certain threshold, and
NuScenes Detection Score (NDS) is further proposed to take both $AP_{center}$
and the error of other parameters, i.e. size, heading, velocity, into
consideration. Waymo Sun et al. (2020c) proposes $AP_{hungarian}$ that applies
the Hungarian algorithm to match the ground truths and predictions, and AP
weighted by Heading ($APH$) is proposed to incorporate heading errors as a
coefficient into the AP calculation.

The other category of approaches tries to resolve the evaluation problem from
a more practical perspective. The idea is that the quality of 3D object
detection should be relevant to the downstream task, i.e. motion planning, so
that the best detection methods should be most helpful to planners to ensure
the safety of driving in practical applications. Toward this goal, PKL Philion
et al. (2020) measures the detection quality using the KL-divergence of the
ego vehicle’s future planned states based on the predicted and ground truth
detections respectively. SDE Deng et al. (2021a) leverages the minimal
distance from the object boundary to the ego vehicle as the support distance
and measures the support distance error.

Analysis: pros and cons of different evaluation metrics. AP-based evaluation
metrics Geiger et al. (2012); Caesar et al. (2020); Sun et al. (2020c) can
naturally inherit the advantages from 2D detection. However, those metrics
overlook the influence of detection on safety issues, which are also critical
in real-world applications. For instance, a misdetection of an object near the
ego vehicle and far away from the ego vehicle may receive a similar level of
punishment in AP calculation, but a misdetection of nearby objects is
substantially more dangerous than a misdetection of faraway objects in
practical applications. Thus AP-based metrics may not be the optimal solution
from the perspective of safe driving. PKL Philion et al. (2020) and SDE Deng
et al. (2021a) partly resolve the problem by considering the effects of
detection in downstream tasks, but additional challenges will be introduced
when modeling those effects. PKL Philion et al. (2020) requires a pre-trained
motion planner for evaluating the detection performance, but a pre-trained
planner also has innate errors that could make the evaluation process
inaccurate. SDE Deng et al. (2021a) requires reconstructing object boundaries
which is generally complicated and challenging.

Figure 4: An illustration of point-based 3D object detection methods.

##  3 LiDAR-based 3D Object Detection

In this section, we introduce the 3D object detection methods based on LiDAR
data, i.e. point clouds or range images. In Section 3.1, we review and analyze
the LiDAR-based 3D object detection models based on different data
representations, including the point-based, grid-based, point-voxel based, and
range-based methods. In Section 3.2, we investigate the learning objectives
for 3D object detectors, including the anchor-based and anchor-free
frameworks, as well as the auxiliary tasks adopted in LiDAR-based 3D object
detection. A chronological overview of the LiDAR-based 3D detection methods is
shown in Figure 3.

###  3.1 Data representations for 3D object detection

Problem and Challenge. In contrast to images where pixels are regularly
distributed on an image plane, point cloud is a sparse and irregular 3D
representation that requires specially designed models for feature extraction.
Range image is a dense and compact representation, but range pixels contain 3D
information instead of RGB values. Hence directly applying conventional
convolutional networks on range images may not be an optimal solution. On the
other hand, detection in autonomous driving scenarios generally has a
requirement for real-time inference. Therefore, how to develop a model that
could effectively handle point cloud or range image data while maintaining a
high efficiency remains an open challenge to the research community.

####  3.1.1 Point-based 3D object detection

General Framework. Point-based 3D object detection methods generally inherit
the success of deep learning techniques on point cloud Qi et al. (2017a, b);
Wang et al. (2019c); Mao et al. (2019) and propose diverse architectures to
detect 3D objects directly from raw points. Point clouds are first passed
through a point-based backbone network, in which the points are gradually
sampled and features are learned by point cloud operators. 3D bounding boxes
are then predicted based on the downsampled points and features. A general
point-based detection framework is shown in Figure 4 and a taxonomy of point-
based detectors is in Table 2. There are two basic components of a point-based
3D object detector: point cloud sampling and feature learning.

Point Cloud Sampling. Farthest Point Sampling (FPS) in PointNet++ Qi et al.
(2017b) has been broadly adopted in point-based detectors, in which the
farthest points are sequentially selected from the original point set.
PointRCNN Shi et al. (2019) is a pioneering work that adopts FPS to
progressively downsample input point cloud and generate 3D proposals from the
downsampled points. Similar design paradigm has also been adopted in many
following works with improvements like segmentation guided filtering Yang et
al. (2018c), feature space sampling Yang et al. (2020c), random sampling Ngiam
et al. (2019), voxel-based sampling Shi and Rajkumar (2020), and coordinate
refinement Pan et al. (2021).

Table 2: A taxonomy of point-based detection methods based on point cloud sampling and feature learning. Method | Context $\Omega$ | Sampling $S$ | Feature $F$
---|---|---|---
PointRCNN Shi et al. (2019) | Ball Query | FPS | Set Abstraction
IPOD Yang et al. (2018c) | Ball Query | Seg. | Set Abstraction
STD Yang et al. (2019) | Ball Query | FPS | Set Abstraction
3DSSD Yang et al. (2020c) | Ball Query | Fusion-FPS | Set Abstraction
Point-GNN Shi and Rajkumar (2020) | Ball Query | Voxel | Graph
StarNet Ngiam et al. (2019) | Ball Query | Targeted-FPS | Graph
Pointformer Pan et al. (2021) | Ball Query | FPS + Refine | Transformer
Figure 5: An illustration of grid-based 3D object detection methods.

Point Cloud Feature Learning. A series of works Shi et al. (2019); Yang et al.
(2019); Zhou et al. (2020a); Wang et al. (2021f) leverage set abstraction in
Qi et al. (2017a) to learn features from point cloud. Specifically, context
points are first collected within a pre-defined radius by ball query. Then,
the context points and features are aggregated through multi-layer perceptrons
and maxpooling to obtain the new features. There are also other works
resorting to different point cloud operators, including graph operators Shi
and Rajkumar (2020); Zarzar et al. (2019); Ngiam et al. (2019); Feng et al.
(2020); He et al. (2020c), attentional operators Paigwar et al. (2019), and
Transformer Pan et al. (2021).

Analysis: potentials and challenges on point cloud feature learning and
sampling. The representation power of point-based detectors is mainly
restricted by two factors: the number of context points and the context radius
adopted in feature learning. Increasing the number of context points will gain
more representation power but at the cost of increasing much memory
consumption. Suitable context radius in ball query is also an important
factor: the context information may be insufficient if the radius is too small
and the fine-grained 3D information may lose if the radius is too large. These
two factors have to be determined carefully to balance the efficacy and
efficiency of detection models.

Point cloud sampling is a bottleneck in inference time for most point-based
methods. Random uniform sampling can be conducted in parallel with high
efficiency. However, considering points in LiDAR sweeps are not uniformly
distributed, random uniform sampling may tend to over-sample those regions of
high point cloud density while under-sample those sparse regions, which
normally leads to poor performance compared to farthest point sampling.
Farthest point sampling and its variants can attain a more uniform sampling
result by sequentially selecting the farthest point from the existing point
set. Nevertheless, farthest point sampling is intrinsically a sequential
algorithm and can not become highly parallel. Thus farthest point sampling is
normally time-consuming and not ready for real-time detection.

####  3.1.2 Grid-based 3D object detection

Table 3: A taxonomy of grid-based detection methods based on models and data representations. Method | Representation | Encoder | Neural Networks
---|---|---|---
Voxels | BEV maps | Pillars | Voxelization | Projection | PointNet | 3D CNN | 2D CNN | Head | Transformer
Vote3D Wang and Posner (2015) | ✓ |  |  | ✓ |  |  | ✓ |  |  |
Vote3Deep Engelcke et al. (2017) | ✓ |  |  | ✓ |  |  | ✓ |  |  |
3D-FCN Li (2017) | ✓ |  |  | ✓ |  |  | ✓ |  |  |
VeloFCN Li et al. (2016) |  | ✓ |  |  | ✓ |  |  | ✓ |  |
BirdNet Beltrán et al. (2018) |  | ✓ |  |  | ✓ |  |  | ✓ |  |
PIXOR Yang et al. (2018b) |  | ✓ |  |  | ✓ |  |  | ✓ |  |
HDNet Yang et al. (2018a) |  | ✓ |  |  | ✓ |  |  | ✓ |  |
VoxelNet Zhou and Tuzel (2018) | ✓ | ✓ |  | ✓ |  | ✓ | ✓ | ✓ |  |
SECOND Yan et al. (2018) | ✓ | ✓ |  | ✓ |  |  | ✓ | ✓ |  |
MVF Zhou et al. (2020c) | ✓ | ✓ |  | ✓ |  |  |  | ✓ |  |
PointPillars Lang et al. (2019) |  | ✓ | ✓ |  |  | ✓ |  | ✓ |  |
Pillar-OD Wang et al. (2020f) |  | ✓ | ✓ |  |  | ✓ |  | ✓ |  |
Part-A2 Net Shi et al. (2020b) | ✓ | ✓ |  | ✓ |  |  | ✓ | ✓ | ✓ |
Voxel R-CNN Deng et al. (2021b) | ✓ | ✓ |  | ✓ |  |  | ✓ | ✓ | ✓ |
CenterPoint Yin et al. (2021a) | ✓ | ✓ |  | ✓ |  |  | ✓ | ✓ |  |
Voxel Transformer Mao et al. (2021c) | ✓ | ✓ |  | ✓ |  |  |  | ✓ |  | ✓
SST Fan et al. (2022) | ✓ | ✓ |  | ✓ |  |  |  | ✓ |  | ✓
SWFormer Sun et al. (2022) | ✓ | ✓ |  | ✓ |  |  |  |  |  | ✓

General Framework. Grid-based 3D object detectors first rasterize point clouds
into discrete grid representations, i.e. voxels, pillars, and bird’s-eye view
(BEV) feature maps. Then they apply conventional 2D convolutional neural
networks or 3D sparse neural networks to extract features from the grids.
Finally, 3D objects can be detected from the BEV grid cells. An illustration
of grid-based 3D object detection is shown in Figure 5 and a taxonomy of grid-
based detectors is in Table 3. There are two basic components in grid-based
detectors: grid-based representations and grid-based neural networks.

Grid-based representations. There are $3$ major types of grid representations:
voxels, pillars, and BEV feature maps.

Voxels. If we rasterize the detection space into a regular 3D grid, voxels are
the grid cells. A voxel can be non-empty if point clouds fall into this grid
cell. Since point clouds are sparsely distributed, most voxel cells in the 3D
space are empty and contain no point. In practical applications, only those
non-empty voxels are stored and utilized for feature extraction. VoxelNet Zhou
and Tuzel (2018) is a pioneering work that utilizes sparse voxel grids and
proposes a novel voxel feature encoding (VFE) layer to extract features from
the points inside a voxel cell. A similar voxel encoding strategy has been
adopted by a series of following works Zhu et al. (2019); Ge et al. (2020);
Yin et al. (2021a); Wang and Solomon (2021); Zheng et al. (2021a); Li et al.
(2021a); Deng et al. (2021b); Shi et al. (2020b). In addition, there are two
categories of approaches trying to improve the voxel representation for 3D
object detection: (1) Multi-view voxels. Some methods propose a dynamic
voxelization and fusion scheme from diverse views, e.g. from both the
bird’s-eye view and the perspective view Zhou et al. (2020c), from the
cylindrical and spherical view Chen et al. (2020a), from the range view Deng
et al. (2022). (2) Multi-scale voxels. Some papers generate voxels of
different scales Ye et al. (2020a) or use reconfigurable voxels Wang et al.
(2020b).

Pillars. Pillars can be viewed as special voxels in which the voxel size is
unlimited in the vertical direction. Pillar features can be aggregated from
points through a PointNet Qi et al. (2017a) and then scattered back to
construct a 2D BEV image for feature extraction. PointPillars Lang et al.
(2019) is a seminal work that introduces the pillar representation and is
followed by Wang et al. (2020f); Fan et al. (2022).

BEV feature maps. Bird’s-eye view feature map is a dense 2D representation,
where each pixel corresponds to a specific region and encodes the points
information in this region. BEV feature maps can be obtained from voxels and
pillars by projecting the 3D features into the bird’s-eye view, or they can be
directly obtained from raw point clouds by summarizing points statistics
within the pixel region. The commonly-used statistics include binary occupancy
Yang et al. (2018b, a); Aghdam et al. (2021) and the height and density of
local point cloud Chen et al. (2017b); Beltrán et al. (2018); Zeng et al.
(2018); Ali et al. (2018); Simony et al. (2018); Zhang et al. (2019); Barrera
et al. (2020); Li et al. (2016).

Grid-based neural networks. There are $2$ major types of grid-based networks:
2D convolutional neural networks for BEV feature maps and pillars, and 3D
sparse neural networks for voxels.

2D convolutional neural networks. Conventional 2D convolutional neural
networks can be applied to the BEV feature map to detect 3D objects from the
bird’s-eye view. In most works, the 2D network architectures are generally
adapted from those successful designs in 2D object detection, e.g. ResNet He
et al. (2016) adopted in Yang et al. (2018b), Region Proposal Network (RPN)
Ren et al. (2015a) and Feature Pyramid Network (FPN) Lin et al. (2017a) in
Beltrán et al. (2018); Barrera et al. (2020); Lang et al. (2019); Kuang et al.
(2020); Simony et al. (2018); Li et al. (2021a), and spatial attention in Li
et al. (2021b); Liu et al. (2020d); Ye et al. (2020c).

3D sparse neural networks. 3D sparse convolutional neural networks are based
on two specialized 3D convolutional operators: sparse convolutions and
submanifold convolutions Graham et al. (2018), which can efficiently conduct
3D convolutions only on those non-empty voxels. Compared to Wang and Posner
(2015); Engelcke et al. (2017); Najibi et al. (2020); Li (2017) that perform
standard 3D convolutions on the whole voxel space, sparse convolutional
operators are highly efficient and can obtain a real-time inference speed.
SECOND Yan et al. (2018) is a seminal work that implements these two sparse
operators with GPU-based hash tables and builds a sparse convolutional network
to extract 3D voxel features. This network architecture has been applied in
numerous works Zhu et al. (2019); Yin et al. (2021a); Yi et al. (2020); Wang
and Solomon (2021); Ge et al. (2020); Chen et al. (2020b); Zhu et al. (2020);
Yan et al. (2018); Deng et al. (2021b); Zheng et al. (2021a) and becomes the
most widely-used backbone network in voxel-based detectors. There is also a
series of works trying to improve the sparse operators Chen et al. (2022b),
extend Yan et al. (2018) into a two-stage detector Shi et al. (2020b); Deng et
al. (2021b), and introduce the Transformer Vaswani et al. (2017) architecture
into voxel-based detection Mao et al. (2021c); Fan et al. (2022).

Analysis: pros and cons of different grid representations. In contrast to the
2D representations like BEV feature maps and pillars, voxels contain more
structured 3D information. In addition, deep voxel features can be learned
through a 3D sparse network. However, a 3D neural network brings additional
time and memory costs. BEV feature map is the most efficient grid
representation that directly projects point cloud into a 2D pseudo image
without specialized 3D operators like sparse convolutions or pillar encoding.
2D detection techniques can also be seamlessly applied to BEV feature maps
without much modification. BEV-based detection methods generally can obtain
high efficiency and a real-time inference speed. However, simply summarizing
points statistics inside pixel regions loses too much 3D information, which
leads to less accurate detection results compared to voxel-based detection.
Pillar-based detection approaches leverage PointNet to encode 3D points
information inside a pillar cell, and the features are then scattered back
into a 2D pseudo image for efficient detection, which balances the
effectiveness and efficiency of 3D object detection.

Analysis: challenges of the grid-based detection methods. A critical problem
that all grid-based methods have to face is choosing the proper size of grid
cells. Grid representations are essentially discrete formats of point clouds
by converting the continuous point coordinates into discrete grid indices. The
quantization process inevitably loses some 3D information and its efficacy
largely depends on the size of grid cells: smaller grid size yields high
resolution grids, and hence maintains more fine-grained details that are
crucial to accurate 3D object detection. Nevertheless, reducing the size of
grid cells leads to a quadratic increase in memory consumption for the 2D grid
representations like BEV feature maps or pillars. As for the 3D grid
representation like voxels, the problem can become more severe. Therefore, how
to balance the efficacy brought by smaller grid sizes and the efficiency
influenced by the memory increase remains an open challenge to all grid-based
3D object detection methods.

####  3.1.3 Point-voxel based 3D object detection

Figure 6: An illustration of point-voxel based 3D object detection methods.

Point-voxel based approaches resort to a hybrid architecture that leverages
both points and voxels for 3D object detection. Those methods can be divided
into two categories: the single-stage and two-stage detection frameworks. An
illustration of the two categories is shown in Figure 6 and a taxonomy is in
Table 4.

Single-stage point-voxel detection frameworks. Single-stage point-voxel based
3D object detectors try to bridge the features of points and voxels with the
point-to-voxel and voxel-to-point transform in the backbone networks. Points
contain fine-grained geometric information and voxels are efficient for
computation, and combining them together in the feature extraction stage
naturally benefits from both two representations. The idea that leverages
point-voxel feature fusion in backbones has been explored by many works, with
the contributions like point-voxel convolutions Liu et al. (2019b); Tang et
al. (2020), auxiliary point-based networks He et al. (2020a); Li et al.
(2021c); Deng et al. (2021c), and multi-scale feature fusion Miao et al.
(2021); Noh et al. (2021); Guan et al. (2022).

Two-stage point-voxel detection frameworks. Two-stage point-voxel based 3D
object detectors resort to different data representations for different
detection stages. Specifically, at the first stage, they employ a voxel-based
detection framework to generate a set of 3D object proposals. In the second
stage, keypoints are first sampled from the input point cloud, and then the 3D
proposals are further refined from the keypoints through novel point
operators. PV-RCNN Shi et al. (2020a) is a seminal work that adopts Yan et al.
(2018) as the first-stage detector, and the RoI-grid pooling operator is
proposed for the second-stage refinement. The following works try to improve
the second-stage head with novel modules and operators, e.g. RefinerNet Chen
et al. (2019c), VectorPool Shi et al. (2021a), point-wise attention Wang et
al. (2020a), scale-aware pooling Li et al. (2021g), RoI-grid attention Mao et
al. (2021a), channel-wise Transformer Sheng et al. (2021), and point density-
aware refinement module Hu et al. (2022).

Table 4: A taxonomy of point-voxel based detection methods. Method | Contribution
---|---
Single-Stage Detection Framework
PVCNN Liu et al. (2019b) | Point-Voxel Convolution
SPVNAS Tang et al. (2020) | Sparse Point-Voxel Convolution
SA-SSD He et al. (2020a) | Auxiliary Point Network
PVGNet Miao et al. (2021) | Point-Voxel-Grid Fusion
Two-Stage Detection Framework
Fast Point R-CNN Chen et al. (2019c) | RefinerNet
PV-RCNN Shi et al. (2020a) | RoI-grid Pooling
PV-RCNN++ Shi et al. (2021a) | VectorPool
Pyramid R-CNN Mao et al. (2021a) | RoI-grid Attention
LiDAR R-CNN Li et al. (2021g) | Scale-aware Pooling
CT3D Sheng et al. (2021) | Channel-wise Transformer

Analysis: potentials and challenges of the point-voxel based methods. The
point-voxel based methods can naturally benefit from both the fine-grained 3D
shape and structure information obtained from points and the computational
efficiency brought by voxels. However, some challenges still exist in these
methods. For the hybrid point-voxel backbones, the fusion of point and voxel
features generally relies on the voxel-to-point and point-to-voxel transform
mechanisms, which can bring non-negligible time costs. For the two-stage
point-voxel detection frameworks, a critical challenge is how to efficiently
aggregate point features for 3D proposals, as the existing modules and
operators are generally time-consuming. In conclusion, compared to the pure
voxel-based detection approaches, the point-voxel based detection methods can
obtain a better detection accuracy while at the cost of increasing the
inference time.

####  3.1.4 Range-based 3D object detection

Range image is a dense and compact 2D representation in which each pixel
contains 3D distance information instead of RGB values. Range-based methods
address the detection problem from two aspects: designing new models and
operators that are tailored for range images, and selecting suitable views for
detection. An illustration of the range-based 3D object detection methods is
shown in Figure 7 and a taxonomy is in Table 5.

Range-based detection models. Since range images are 2D representations like
RGB images, range-based 3D object detectors can naturally borrow the models in
2D object detection to handle range images. LaserNet Meyer et al. (2019b) is a
seminal work that leverages the deep layer aggregation network (DLA-Net) Yu et
al. (2018) to obtain multi-scale features and detect 3D objects from range
images. Some papers also adopt other 2D object detection architectures, e.g.
U-Net Ronneberger et al. (2015) is applied in Meyer et al. (2020); Liang et
al. (2020b); Sun et al. (2021), RPN Ren et al. (2015a) and R-CNN Ren et al.
(2015b) are employed in Liang et al. (2020b); Bewley et al. (2020), FCN Long
et al. (2015) is used in Liang et al. (2021c), and FPN Lin et al. (2017a) is
leveraged in Fan et al. (2021).

Range-based operators. Pixels of range images contain 3D distance information
instead of color values, so the standard convolutional operator in
conventional 2D network architectures is not optimal for range-based
detection, as the pixels in a sliding window may be far away from each other
in the 3D space. Some works resort to novel operators to effectively extract
features from range pixels, including range dilated convolutions Bewley et al.
(2020), graph operators Chai et al. (2021), and meta-kernel convolutions Fan
et al. (2021).

Figure 7: An illustration of range-based 3D object detection. Table 5: A taxonomy of range-based detection methods based on views, models, and operators. Method | View | Operator | Model | Note
---|---|---|---|---
LaserNet Meyer et al. (2019b) | RV | Convolution | DLA-Net | -
Rapoport-Lavie et al. Rapoport-Lavie and Raviv (2021) | RV, CYV | Convolution | Range-Guided Net | -
LaserFlow Meyer et al. (2020) | RV, BEV | Convolution | U-Net | multi-sweep fusion
RangeRCNN Liang et al. (2020b) | RV, PV, BEV | Dilated Convolution | U-Net, RPN, RCNN | -
RangeIoUDet Liang et al. (2021c) | RV, PV, BEV | Convolution | FCN, PointNet | point-wise segmentation
RCD Bewley et al. (2020) | RV | Conditioned Dilated Convolution | RPN, RCNN | -
RangeDet Fan et al. (2021) | RV | Meta Kernel Convolution | FPN | -
PPC Chai et al. (2021) | RV | Graph Kernel Convolution | DLA-Net | -
RSN Sun et al. (2021) | RV, BEV | Convolution | U-Net, VoxelNet | range-based segmentation

Views for range-based detection. Range images are captured from the range view
(RV), and ideally, the range view is a spherical projection of a point cloud.
It has been a natural solution for many range-based approaches Meyer et al.
(2019b); Bewley et al. (2020); Fan et al. (2021); Chai et al. (2021) to detect
3D objects directly from the range view. Nevertheless, detection from the
range view will inevitably suffer from the occlusion and scale-variation
issues brought by the spherical projection. To circumvent these issues, many
methods have been working on leveraging other views for predicting 3D objects,
e.g. the cylindrical view (CYV) leveraged in Rapoport-Lavie and Raviv (2021),
a combination of the range-view, bird’s-eye view (BEV), and/or point-view (PV)
adopted in Liang et al. (2021c); Sun et al. (2021); Meyer et al. (2020); Liang
et al. (2020b).

Analysis: potentials and challenges of the range-based methods. Range image is
a dense and compact 2D representation, so the conventional or specialized 2D
convolutions can be seamlessly applied on range images, which makes the
feature extraction process quite efficient. Nevertheless, compared to
bird’s-eye view detection, detection from the range view is vulnerable to
occlusion and scale variation. Hence, feature extraction from the range view
and object detection from the bird’s eye view becomes the most practical
solution to range-based 3D object detection.

###  3.2 Learning objectives for 3D object detection

Problem and Challenge. Learning objectives are critical in object detection.
Since 3D objects are quite small relative to the whole detection range,
special mechanisms to enhance the localization of small objects are strongly
required in 3D detection. On the other hand, considering point cloud is sparse
and objects normally have incomplete shapes, accurately estimating the centers
and sizes of 3D objects is a long-standing challenge.

####  3.2.1 Anchor-based 3D object detection

Anchors are pre-defined cuboids with fixed shapes that can be placed in the 3D
space. 3D objects can be predicted based on the positive anchors that have a
high intersection over union (IoU) with ground truth. We will introduce the
anchor-based 3D object detection methods from the aspect of anchor
configurations and loss functions. An illustration of anchor-based learning
objectives is shown in Figure 8 and a taxonomy is in Table 6.

Prerequisites. The ground truth 3D objects can be represented as
$[x^{g},y^{g},z^{g},l^{g},w^{g},h^{g},\theta^{g}]$ with the class $cls^{g}$.
The anchors $[x^{a},y^{a},z^{a},l^{a},w^{a},h^{a},\theta^{a}]$ are used to
generate predicted 3D objects $[x,y,z,l,w,h,\theta]$ with a predicted class
probability $p$.

Anchor configurations. Anchor-based 3D object detection approaches generally
detect 3D objects from the bird’s-eye view, in which 3D anchor boxes are
placed at each grid cell of a BEV feature map. 3D anchors normally have a
fixed size for each category, since objects of the same category have similar
sizes.

Loss functions. The anchor-based methods employ the classification loss
$L_{cls}$ to learn the positive and negative anchors, and the regression loss
$L_{reg}$ is utilized to learn the size and location of an object based on a
positive anchor. Additionally, $L_{\theta}$ is applied to learn the object’s
heading angle. The loss function is

$L_{det}=L_{cls}+L_{reg}+L_{\theta}.$ (4)

VoxelNet Zhou and Tuzel (2018) is a seminal work that leverages the anchors
that have a high IoU with the ground truth 3D objects as positive anchors, and
the other anchors are treated as negatives. To accurately classify those
positive and negative anchors, for each category, the binary cross entropy
loss can be applied to each anchor on the BEV feature map, which can be
formulated as

$L^{bce}_{cls}=-[q\cdot\log(p)+(1-q)\cdot\log(1-p)],$ (5)

where $p$ is the predicted probability for each anchor and the target $q$ is
$1$ if the anchor is positive and $0$ otherwise. In addition to the binary
cross entropy loss, the focal loss Lin et al. (2017b); Yun et al. (2019) has
also been employed to enhance the localization ability:

$L^{focal}_{cls}=-\alpha(1-p)^{\gamma}\log(p),$ (6)

where $\alpha=0.25$ and $\gamma=2$ are adopted in most works.

The regression targets can be further applied to those positive anchors to
learn the sizes and locations of 3D objects:

$\begin{split}\Delta x=\frac{x^{g}-x^{a}}{d^{a}},\Delta
y=\frac{y^{g}-y^{a}}{d^{a}},\Delta z=\frac{z^{g}-z^{a}}{h^{a}},\\\ \Delta
l=\log(\frac{l^{g}}{l^{a}}),\Delta w=\log(\frac{w^{g}}{w^{a}}),\Delta
h=\log(\frac{h^{g}}{h^{a}}),\end{split}$ (7)

where $d^{a}=\sqrt{(l^{a})^{2}+(w^{a})^{2}}$ is the diagonal length of an
anchor from the bird’s-eye view. Then the SmoothL1 loss Ren et al. (2015b) is
adopted to regress the targets, which is represented as

$L_{reg}=\sum_{\begin{subarray}{c}u\in\\{x,y,z,l,w,h\\},\\\ v\in\\{\Delta
x,\Delta y,\Delta z,\Delta l,\Delta w,\Delta h\\}\end{subarray}}{\rm
SmoothL1}(u-v).$ (8)

Figure 8: An illustration of anchor-based learning objectives.

To learn the heading angle $\theta$, the radian orientation offset can be
directly regressed with the SmoothL1 loss:

$\begin{split}&\Delta\theta=\theta^{g}-\theta^{a},\\\ &L_{\theta}={\rm
SmoothL1}(\theta-\Delta\theta).\end{split}$ (9)

However, directly regressing the radian offset is normally hard due to the
large regression range. Alternatively, the bin-based heading estimation Qi et
al. (2018) is a better solution to learn the heading angle, in which the angle
space is first divided into bins, and bin-based classification $L_{dir}$ and
residual regression are employed:

$L_{\theta}=L_{dir}+{\rm SmoothL1}(\theta-\Delta\theta^{\prime}),$ (10)

where $\Delta\theta^{\prime}$ is the residual offset within a bin. The sine
function can also be utilized to encode the radian offset:

$\begin{split}\Delta\theta=\sin(\theta^{g}-\theta^{a}),\end{split}$ (11)

and $L^{\theta}$ can be computed following Eqn. 9 or Eqn. 10.

In addition to the loss functions that learn the objects’ sizes, locations,
and orientations separately, the intersection over union (IoU) loss Zhou et
al. (2019a) that considers all object parameters as a whole can also be
applied in 3D object detection:

$L_{IoU}=1-IoU(b^{g},b),$ (12)

where $b^{g}$ and $b$ are the ground truth and predicted 3D bounding boxes,
and $IoU(\cdot)$ calculates the 3D IoU in a differential manner. Apart from
the IoU loss, the corner loss Qi et al. (2018) is also introduced to minimize
the distances between the eight corners of the ground truth and predicted
boxes, that is

$L_{corner}=\sum^{8}_{i=1}||c^{g}_{i}-c_{i}||,$ (13)

where $c^{g}_{i}$ and $c_{i}$ are the $i$th corner of the ground truth and
predicted cuboid respectively.

Analysis: potentials and challenges of the anchor-based approaches. The
anchor-based methods can benefit from the prior knowledge that 3D objects of
the same category should have similar shapes, so they can generate accurate
object predictions with the help of 3D anchors. However, since 3D objects are
relatively small with respect to the detection range, a large number of
anchors are required to ensure complete coverage of the whole detection range,
e.g. around $70$k anchors are utilized in Yan et al. (2018) on the KITTI
Geiger et al. (2012) dataset. Furthermore, for those extremely small objects
such as pedestrians and cyclists, applying anchor-based assignments can be
quite challenging. Considering the fact that anchors are generally placed at
the center of each grid cell, if the grid cell is large and objects in the
cell are small, the anchor of this cell may have a low IoU with the small
objects, which may hamper the training process.

Table 6: A taxonomy of anchor-based methods based on loss functions. Loss Function | Methods
---|---
$L^{bce}_{cls}$ (Eqn. 5) |  | Simony et al. (2018); Ali et al. (2018); Beltrán et al. (2018); Zeng et al. (2018); Zhou and Tuzel (2018); Barrera et al. (2020); Yang et al. (2019, 2018c); Chen et al. (2019c)
---
He et al. (2020c)
$L^{focal}_{cls}$ (Eqn. 6) |  | Yan et al. (2018); Lang et al. (2019); Zhou et al. (2020c, 2019a); Yun et al. (2019); Ye et al. (2020a); Du et al. (2020); Zhu et al. (2020)
---
Wang et al. (2020b); Yi et al. (2020); Mao et al. (2021c); Deng et al.
(2021b); Zheng et al. (2021a); Du et al. (2021); Liang et al. (2020b); Ngiam
et al. (2019)
Shi et al. (2020a); He et al. (2020a); Wang et al. (2020a); Miao et al.
(2021); Noh et al. (2021); Mao et al. (2021a); Shi et al. (2021a)
$L_{reg}$ (Eqn. 8) |  | Simony et al. (2018); Ali et al. (2018); Beltrán et al. (2018); Zeng et al. (2018); Yan et al. (2018); Zhou and Tuzel (2018); Lang et al. (2019); Zhou et al. (2020c); Yun et al. (2019)
---
Ye et al. (2020a); Du et al. (2020); Zhu et al. (2020); Wang et al. (2020b);
Yi et al. (2020); Barrera et al. (2020); Mao et al. (2021c); Deng et al.
(2021b)
Zheng et al. (2021a); Du et al. (2021); Liang et al. (2020b); Yang et al.
(2019); Ngiam et al. (2019); Yang et al. (2018c); Chen et al. (2019c); Shi et
al. (2020a)
He et al. (2020a); Wang et al. (2020a); He et al. (2020c); Miao et al. (2021);
Noh et al. (2021); Mao et al. (2021a); Shi et al. (2021a)
$L^{1}_{\theta}$ (Eqn. 9) | Ali et al. (2018); Zhou and Tuzel (2018); Ye et al. (2020a); Du et al. (2020, 2021); Chen et al. (2019c)
$L^{2}_{\theta}$ (Eqn. 10) |  | Beltrán et al. (2018); Wang et al. (2020b); Yi et al. (2020); Barrera et al. (2020); Mao et al. (2021c); Zheng et al. (2021a); Liang et al. (2020b); Yang et al. (2019, 2018c)
---
Shi et al. (2020a); He et al. (2020a); Wang et al. (2020a); Mao et al.
(2021a); Shi et al. (2021a)
$L^{3}_{\theta}$ (Eqn. 11) |  | Simony et al. (2018); Zeng et al. (2018); Yan et al. (2018); Lang et al. (2019); Zhou et al. (2020c); Zhu et al. (2020); Deng et al. (2021b); Ngiam et al. (2019); He et al. (2020c)
---
Miao et al. (2021); Noh et al. (2021)
$L_{IoU}$ (Eqn. 12) | Zhou et al. (2019a)
$L_{corner}$ (Eqn. 13) | Ye et al. (2020a); Liang et al. (2020b); Yang et al. (2019, 2018c)

####  3.2.2 Anchor-free 3D object detection

Anchor-free approaches eliminate the complicated anchor designs and can be
flexibly applied to diverse views, e.g. the bird’s-eye view, point view, and
range view. An illustration of anchor-free learning objectives is shown in
Figure 9 and a taxonomy is in Table 7. The major difference between the
anchor-based and anchor-free methods lies in the selection of positive and
negative samples. We will introduce the anchor-free methods from the
perspective of positive assignments, including grid-based, point-based, range-
based, and set-to-set assignments. We still adopt the notations in Section
3.2.1 for simplicity.

Grid-based assignment. In contrast to the anchor-based methods that rely on
the IoUs with anchors to determine the positive and negative samples, the
anchor-free methods leverage various grid-based assignment strategies for BEV
grid cells, pillars, and voxels. PIXOR Yang et al. (2018b) is a pioneering
work that leverages the grid cells inside the ground truth 3D objects as
positives, and the others as negatives. This inside-object assignment strategy
is adopted in Wang et al. (2020f), and further improved in Ge et al. (2020);
Hu et al. (2021); Chen et al. (2020b) by selecting the grid cells nearest to
the object center. CenterPoint Yin et al. (2021a) utilizes a Gaussian kernel
at each object center to assign positive labels. These methods can still use
Eqn. 5 or Eqn. 6 as the classification loss, and the regression target is

$\Delta=[dx,dy,z^{g},\log(l^{g}),\log(w^{g}),\log(h^{g}),\sin(\theta^{g}),\cos(\theta^{g})],$
(14)

where $dx$ and $dy$ are the offsets between positive grid cells and object
centers. The SmoothL1 loss is leveraged to regress $\Delta$.

Point-based assignment. Most point-based detection approaches resort to the
anchor-free and point-based assignment strategy, in which the points are first
segmented and those foreground points inside or near 3D objects are selected
as positive samples, and 3D bounding boxes are finally learned from those
foreground points. This foreground point segmentation strategy has been
adopted in most point-based detectors Shi et al. (2019); Yang et al. (2020c,
2018c); Pan et al. (2021), with improvements such as adding centerness scores
Yang et al. (2020c), etc.

Range-based assignment. Anchor-free assignments can also be employed on range
images. A common solution is to select the range pixels inside 3D objects as
positive samples, which has been adopted in Meyer et al. (2019b); Fan et al.
(2021). Different from other methods where the regression targets are based on
the global 3D coordinate system, the range-based methods resort to an object-
centric coordinate system for regression. Eqn. 14 can still be applied in
these methods with an additional coordinate transform.

Figure 9: An illustration of anchor-free learning objectives. Table 7: A taxonomy of anchor-free detection methods based on the sample types for prediction and the assignment strategies. Method |  | Samples
---
for Prediction
| Positive Samples
---
Selection
PIXOR Yang et al. (2018b) | BEV grid cells | inside objects
CenterPoint Yin et al. (2021a) | BEV grid cells | Gaussian radii on centers
ObjectDGCNN Wang and Solomon (2021) | BEV grid cells | bipartite matching
Pillar-OD Wang et al. (2020f) | pillars | inside objects
HotSpotNet Chen et al. (2020b) | voxels | K-nearest to object centers
PointRCNN Shi et al. (2019) | points | foreground
3DSSD Yang et al. (2020c) | points | foreground & near centers
LaserNet Meyer et al. (2019b) | range pixels | inside objects
RangeDet Fan et al. (2021) | range pixels | inside objects

Set-to-set assignment. DETR Carion et al. (2020) is an influential 2D
detection method that introduces a set-to-set assignment strategy to
automatically assign the predictions to the respective ground truths via the
Hungarian algorithm Kuhn (1955):

$\mathcal{M}^{*}=\mathop{\rm argmin}\limits_{\mathcal{M}}\sum_{(i\rightarrow
j)\in\mathcal{M}}L_{det}(b^{g}_{i},b_{j}),$ (15)

where $\mathcal{M}$ is a one-to-one mapping from each positive sample to a 3D
object. The set-to-set assignments have also been explored in 3D object
detection approaches Misra et al. (2021); Wang and Solomon (2021); Xue et al.
(2022), and Xue et al. (2022) further introduces a novel cost function for the
Hungarian matching.

Analysis: potentials and challenges of the anchor-free approaches. The anchor-
free detection methods abandon the complicated anchor design and exhibit
stronger flexibility in terms of the assignment strategies. With the anchor-
free assignments, 3D objects can be predicted directly on various
representations, including points, range pixels, voxels, pillars, and BEV grid
cells. The learning process is also greatly simplified without introducing
additional shape priors. Among those anchor-free methods, the center-based
methods Yin et al. (2021a) have shown great potential in detecting small
objects and have outperformed the anchor-based detection methods on the widely
used benchmarks Caesar et al. (2020); Sun et al. (2020c).

Despite these merits, a general challenge to the anchor-free methods is to
properly select positive samples to generate 3D object predictions. In
contrast to the anchor-based methods that only select those high IoU samples,
the anchor-free methods may possibly select some bad positive samples that
yield inaccurate object predictions. Hence, careful design to filter out those
bad positives is important in most anchor-free methods.

####  3.2.3 3D object detection with auxiliary tasks

Numerous approaches resort to auxiliary tasks to enhance the spatial features
and provide implicit guidance for accurate 3D object detection. The commonly
used auxiliary tasks include semantic segmentation, intersection over union
prediction, object shape completion, and object part estimation.

Semantic segmentation. Semantic segmentation can help 3D object detection in
$3$ aspects: (1) Foreground segmentation could provide implicit information on
objects’ locations. Point-wise foreground segmentation has been broadly
adopted in most point-based 3D object detectors Shi et al. (2019); Zhou et al.
(2020a); Yang et al. (2019); He et al. (2020a) for proposal generation. (2)
Spatial features can be enhanced by segmentation. In Yi et al. (2020), a
semantic context encoder is leveraged to enhance spatial features with
semantic knowledge. (3) Semantic segmentation can be utilized as a pre-
processing step to filter out background samples and make 3D object detection
more efficient. Yang et al. (2018c) and Sun et al. (2021) leverage semantic
segmentation to remove those redundant points to speed up the subsequent
detection model.

IoU prediction. Intersection over union (IoU) can serve as a useful
supervisory signal to rectify the object confidence scores. Zheng et al.
(2021a) proposes an auxiliary branch to predict an IoU score $S_{IoU}$ for
each detected 3D object. During inference, the original confidence scores
$S_{conf}=S_{cls}$ from the conventional classification branch are further
rectified by the IoU scores $S_{IoU}$:

$S_{conf}=S_{cls}\cdot(S_{IoU})^{\beta},$ (16)

where the hyper-parameter $\beta$ controls the degrees of suppressing the low-
IoU predictions and enhancing the high-IoU predictions. With the IoU
rectification, the high-quality 3D objects are easier to be selected as the
final predictions. Similar designs have also been adopted in Zheng et al.
(2021b); Liang et al. (2021c); Ge et al. (2020); Hu et al. (2021).

Table 8: A taxonomy of detection methods based on auxiliary tasks. Auxiliary Task | Methods
---|---
Semantic segmentation |  | Shi et al. (2019); Yang et al. (2019); He et al. (2020a); Zhou et al. (2020a); Yi et al. (2020); Yang et al. (2018c); Sun et al. (2021)
---
IoU prediction |  | Zheng et al. (2021a, b); Liang et al. (2021c); Ge et al. (2020); Hu et al. (2021)
---
Object shape completion |  | Najibi et al. (2020); Zhu et al. (2020); Xu et al. (2021b, a)
---
Object part estimation |  | Chen et al. (2020b); Shi et al. (2020b)
---

Object shape completion. Due to the nature of LiDAR sensors, faraway objects
generally receive only a few points on their surfaces, so 3D objects are
generally sparse and incomplete. A straightforward way of boosting the
detection performance is to complete object shapes from sparse point clouds.
Complete shapes could provide more useful information for accurate and robust
detection. Many shape completion techniques have been proposed in 3D
detection, including a shape decoder Najibi et al. (2020), shape signatures
Zhu et al. (2020), and a probabilistic occupancy grid Xu et al. (2021b, a).

Object part estimation. Identifying the part information inside objects is
helpful in 3D object detection, as it reveals more fine-grained 3D structure
information of an object. Object part estimation has been explored in some
works Chen et al. (2020b); Shi et al. (2020b).

Analysis: future prospects of multitask learning for 3D object detection. 3D
object detection is innately correlated with many other 3D perception and
generation tasks. Multitask learning of 3D detection and segmentation is more
beneficial compared to training 3D object detectors independently, and shape
completion can also help 3D object detection. There are also other tasks that
can help boost the performance of 3D object detectors. For instance, scene
flow estimation could identify static and moving objects, and tracking the
same 3D object in a point cloud sequence yields a more accurate estimation of
this object. Hence, it will be promising to integrate more perception tasks
into the existing 3D object detection pipeline.

##  4 Camera-based 3D Object Detection

Figure 10: Chronological overview of the camera-based 3D object detection
methods.

In this section, we introduce camera-based 3D object detection methods. In
Section 4.1, we review and analyze the monocular 3D object detection methods,
which can be further divided into the image-only, depth-assisted, and prior-
guided approaches. In Section 4.2, we investigate the 3D object detection
methods based on stereo images. In Section 4.3, we introduce the 3D object
detection methods with multiple cameras. A chronological overview of the
camera-based 3D object detection methods is shown in Figure 10.

###  4.1 Monocular 3D object detection

Problem and Challenge. Detecting objects in the 3D space from monocular images
is an ill-posed problem since a single image cannot provide sufficient depth
information. Accurately predicting the 3D locations of objects is the major
challenge in monocular 3D object detection. Many endeavors have been made to
tackle the object localization problem, e.g. inferring depth from images,
leveraging geometric constraints and shape priors. Nevertheless, the problem
is far from being solved. Monocular 3D detection methods still perform much
worse than the LiDAR-based methods due the poor 3D localization ability, which
leaves an open challenge to the research community.

####  4.1.1 Image-only monocular 3D object detection

Inspired by the 2D detection approaches, a straightforward solution to
monocular 3D object detection is to directly regress the 3D box parameters
from images via a convolutional neural network. The direct-regression methods
naturally borrow designs from the 2D detection network architectures, and can
be trained in an end-to-end manner. These approaches can be divided into the
single-stage/two-stage, or anchor-based/anchor-free methods. An illustration
of image-only 3D object detection is shown in Figure 11 and a taxonomy is in
Table 9.

Single-stage anchor-based methods. Anchor-based monocular detection approaches
rely on a set of 2D-3D anchor boxes placed at each image pixel, and use a 2D
convolutional neural network to regress object parameters from the anchors.
Specifically, for each pixel $[u,v]$ on the image plane, a set of 3D anchors
$[w^{a},h^{a},l^{a},\theta^{a}]_{3D}$, 2D anchors $[w^{a},h^{a}]_{2D}$, and
depth anchors $d^{a}$ are pre-defined. An image is passed through a
convolutional network to predict the 2D box offsets
$\delta_{2D}=[\delta_{x},\delta_{y},\delta_{w},\delta_{h}]_{2D}$ and the 3D
box offsets
$\delta_{3D}=[\delta_{x},\delta_{y},\delta_{d},\delta_{w},\delta_{h},\delta_{l},\delta_{\theta}]_{3D}$
based on each anchor. Then, the 2D bounding boxes $b_{2D}=[x,y,w,h]_{2D}$ can
be decoded as

$\begin{split}&[x,y]_{2D}=[u,v]+[\delta_{x},\delta_{y}]_{2D}\cdot[w^{a},h^{a}]_{2D},\\\
&[w,h]_{2D}=e^{[\delta_{w},\delta_{h}]_{2D}}\cdot[w^{a},h^{a}]_{2D},\end{split}$
(17)

and the 3D bounding boxes $b_{3D}=[x,y,z,l,w,h,\theta]_{3D}$ can be decoded
from the anchors and $\delta_{3D}$:

$\begin{split}&[u^{c},u^{c}]=[u,v]+[\delta_{x},\delta_{y}]_{3D}\cdot[w^{a},h^{a}]_{2D},\\\
&[w,h,l]_{3D}=e^{[\delta_{w},\delta_{h},\delta_{l}]_{3D}}\cdot[w^{a},h^{a},l^{a}]_{3D},\\\
&d^{c}=d^{a}+\delta_{d_{3D}},\theta_{3D}=\theta^{a}_{3D}+\delta_{\theta_{3D}},\end{split}$
(18)

where $[u^{c},v^{c}]$ is the projected object center on the image plane.
Finally, the projected center $[u^{c},v^{c}]$ and its depth $d^{c}$ are
transformed into the 3D object center $[x,y,z]_{3D}$:

$d^{c}\cdot\begin{bmatrix}u^{c}\\\ v^{c}\\\ 1\end{bmatrix}=K\ T\
\begin{bmatrix}x\\\ y\\\ z\\\ 1\end{bmatrix}_{3D},$ (19)

where $K$ and $T$ are the camera intrinsics and extrinsics.

M3D-RPN Brazil and Liu (2019) is a seminal paper that proposes the anchor-
based framework, and many papers have tried to improve this framework, e.g.
extending it into video-based 3D detection Brazil et al. (2020), introducing
differential non-maximum suppression Kumar et al. (2021), designing an
asymmetric attention module Luo et al. (2021a).

Table 9: A taxonomy of image-only monocular detection methods based on frameworks. Framework | Methods
---|---
Single-stage | anchor-based |  | Brazil and Liu (2019); Brazil et al. (2020); Kumar et al. (2021); Luo et al. (2021a); Liu et al. (2019a)
---
anchor-free |  | Zhou et al. (2019b); Mousavian et al. (2017); Liu et al. (2020c); Wang et al. (2021g, 2022a); Ma et al. (2021)
---
Li et al. (2020c); Zhang et al. (2021c); Zhou et al. (2021); Roddick et al.
(2019); Reading et al. (2021); Simonelli et al. (2020)
Two-stage |  | Manhardt et al. (2019); Simonelli et al. (2019); Li et al. (2019a); Qin et al. (2019a); Shi et al. (2021b); Lu et al. (2021)
---
Figure 11: An illustration of image-only monocular 3D object detection
methods. Image samples are from Wang et al. (2021g).

Single-stage anchor-free methods. Anchor-free monocular detection approaches
predict the attributes of 3D objects from images without the aid of anchors.
Specifically, an image is passed through a 2D convolutional neural network and
then multiple heads are applied to predict the object attributes separately.
The prediction heads generally include a category head to predict the object’s
category, a keypoint head to predict the coarse object center $[u,v]$, an
offset head to predict the center offset $[\delta_{x},\delta_{y}]$ based on
$[u,v]$, a depth head to predict the depth offset $\delta_{d}$, a size head to
predict the object size $[w,h,l]$, and an orientation head to predict the
observation angle $\alpha$. The 3D object center $[x,y,z]$ can be converted
from the projected center $[u^{c},v^{c}]$ and depth $d^{c}$:

$\begin{split}&d^{c}=\sigma^{-1}(\frac{1}{\delta_{d}+1}),u^{c}=u+\delta_{x},v^{c}=v+\delta_{y},\\\
&d^{c}\cdot\begin{bmatrix}u^{c}\\\ v^{c}\\\ 1\end{bmatrix}=K\ T\
\begin{bmatrix}x\\\ y\\\ z\\\ 1\end{bmatrix}_{3D},\end{split}$ (20)

where $\sigma$ is the sigmoid function. The yaw angle $\theta$ of an object
can be converted from the observation angle $\alpha$ using

$\theta=\alpha+{\rm arctan}(\frac{x}{z}).$ (21)

CenterNet Zhou et al. (2019b) first introduces the single-stage anchor-free
framework for monocular 3D object detection. Many following papers work on
improving this framework, including novel depth estimation schemes Liu et al.
(2020c); Wang et al. (2022a); Zhang et al. (2021c), an FCOS Tian et al.
(2019)-like architecture Wang et al. (2021g), a new IoU-based loss function Ma
et al. (2021), keypoints Li et al. (2020c), pair-wise relationships Chen et
al. (2020f), camera extrinsics prediction Zhou et al. (2021), and view
transforms Roddick et al. (2019); Reading et al. (2021); Simonelli et al.
(2020).

Two-stage methods. Two-stage monocular detection approaches generally extend
the conventional two-stage 2D detection architectures to 3D object detection.
Specifically, they utilize a 2D detector in the first stage to generate 2D
bounding boxes from an input image. Then in the second stage, the 2D boxes are
lifted up to the 3D space by predicting the 3D object parameters from the 2D
RoIs. ROI-10D Manhardt et al. (2019) extends the conventional Faster R-CNN Ren
et al. (2015b) architecture with a novel head to predict the parameters of 3D
objects in the second stage. A similar design paradigm has been adopted in
many works with improvements like disentangling the 2D and 3D detection loss
Simonelli et al. (2019), predicting heading angles in the first stage Li et
al. (2019a), learning more accurate depth information Qin et al. (2019a); Shi
et al. (2021b); Lu et al. (2021).

Analysis: potentials and challenges of the image-only methods. The image-only
methods aim to directly regress the 3D box parameters from images via a
modified 2D object detection framework. Since these methods take inspiration
from the 2D detection methods, they can naturally benefit from the advances in
2D object detection and image-based network architectures. Most methods can be
trained end-to-end without pre-training or post-processing, which is quite
simple and efficient.

A critical challenge of the image-only methods is to accurately predict depth
$d^{c}$ for each 3D object. As shown in Wang et al. (2022a), simply replacing
the predicted depth with ground truth yields more than $20\%$ car AP gain on
the KITTI Geiger et al. (2012) dataset, while replacing other parameters only
results in an incremental gain. This observation indicates that the depth
error dominates the total errors and becomes the most critical factor
hampering accurate monocular detection. Nevertheless, depth estimation from
monocular images is an ill-posed problem, and the problem becomes severer with
only box-level supervisory signals.

####  4.1.2 Depth-assisted monocular 3D object detection

Table 10: A taxonomy of depth-assisted monocular detection methods based on data representations and detection networks (2D: convolutional networks; 3D: point cloud networks). Method | Representation | Network
---|---|---
RGB | Depth |  | Pseudo
---
LiDAR
| Coord.
---
Map
2D | 3D
MultiFusion Xu and Chen (2018) | ✓ | ✓ |  |  | ✓ |
D4LCN Ding et al. (2020) | ✓ | ✓ |  |  | ✓ |
DDMP Wang et al. (2021d) | ✓ | ✓ |  |  | ✓ |
| Pseudo-
---
LiDAR Wang et al. (2019a)
|  | ✓ |  |  | ✓
Deep Optics Chang and Wetzstein (2019) |  |  | ✓ |  |  | ✓
AM3D Ma et al. (2019a) | ✓ |  | ✓ |  | ✓ | ✓
Weng et al. Weng and Kitani (2019) | ✓ |  | ✓ |  | ✓ | ✓
PatchNet Ma et al. (2020) |  |  |  | ✓ | ✓ |
Figure 12: An illustration of depth-assisted monocular 3D object detection
methods. Image and depth samples are from Ma et al. (2019a).

Depth estimation is critical in monocular 3D object detection. To achieve more
accurate monocular detection results, many papers resort to pre-training an
auxiliary depth estimation network. Specifically, a monocular image is first
passed through a pre-trained depth estimator, e.g. MonoDepth Godard et al.
(2017) or DORN Fu et al. (2018), to generate a depth image. Then, there are
mainly two categories of methods to deal with depth images and monocular
images. The depth-image based methods fuse images and depth maps with a
specialized neural network to generate depth-aware features that could enhance
the detection performance. The pseudo-LiDAR based methods convert a depth
image into a pseudo-LiDAR point cloud, and LiDAR-based detectors can then be
applied to the point cloud to predict 3D objects. An illustration of depth-
assisted monocular 3D object detection is shown in Figure 12 and a taxonomy of
these methods is in Table 10.

Depth-image based methods. Most depth-image based methods leverage two
backbone networks for RGB and depth images respectively. They obtain depth-
aware image features by fusing the information from the two backbones with
specialized operators. More accurate 3D bounding boxes can be learned from the
depth-ware features and can be further refined with depth images. MultiFusion
Xu and Chen (2018) is a pioneering work that introduces the depth-image based
detection framework. Following papers adopt similar design paradigms with
improvements in network architectures, operators, and training strategies,
e.g. a point-based attentional network Bao et al. (2019), depth-guided
convolutions Ding et al. (2020), depth-conditioned message passing Wang et al.
(2021d), disentangling appearance and localization features Zou et al. (2021),
and a novel depth pre-training framework Park et al. (2021).

Pseudo-LiDAR based methods. Pseudo-LiDAR based methods transform a depth image
into a pseudo-LiDAR point cloud, and LiDAR-based detectors can then be
employed to detect 3D objects from the point cloud. Pseudo-LiDAR point cloud
is a data representation first introduced in Wang et al. (2019a), where they
convert a depth map $\mathcal{D}\in R^{H\times W}$ into a pseudo point cloud
$\mathcal{P}\in R^{HW\times 3}$. Specifically, for each pixel $[u,v]$ and its
depth value $d$ in a depth image, the corresponding 3D point coordinate
$[x,y,z]$ in the camera coordinate system is computed as

$\begin{split}x=\frac{(u-c_{u})\times z}{f_{u}},y=\frac{(v-c_{v})\times
z}{f_{v}},z=d,\end{split}$ (22)

where $[c_{u},c_{v}]$ is the camera principal point, and $f_{u}$ and $f_{v}$
are the focal lengths along the horizontal and vertical axis respectively.
Thus $\mathcal{P}$ can be obtained by back-projecting each pixel in
$\mathcal{D}$ into the 3D space. $\mathcal{P}$ is referred as the pseudo-LiDAR
representation: it is essentially a 3D point cloud but is extracted from a
depth image instead of a real LiDAR sensor. Finally, LiDAR-based 3D object
detectors can be directly applied on the pseudo-LiDAR point cloud
$\mathcal{P}$ to predict 3D objects. Many papers have worked on improving the
pseudo-LiDAR detection framework, including augmenting pseudo point cloud with
color information Ma et al. (2019a), introducing instance segmentation Weng
and Kitani (2019), designing a progressive coordinate transform scheme Wang et
al. (2021e), improving pixel-wise depth estimation with separate foreground
and background prediction Wang et al. (2020d), domain adaptation from real
LiDAR point cloud Ye et al. (2020b), and a new physical sensor design Chang
and Wetzstein (2019).

PatchNet Ma et al. (2020) challenges the conventional idea of leveraging the
pseudo-LiDAR representation $\mathcal{P}\in R^{HW\times 3}$ for monocular 3D
object detection. They conduct an in-depth investigation and provide an
insightful observation that the power of pseudo-LiDAR representation comes
from the coordinate transformation instead of the point cloud representation.
Hence, a coordinate map $\mathcal{M}\in R^{H\times W\times 3}$ where each
pixel encodes a 3D coordinate can attain a comparable monocular detection
result with the pseudo-LiDAR point cloud representation. This observation
enables us to directly apply a 2D neural network on the coordinate map to
predict 3D objects, eliminating the need of leveraging the time-consuming
LiDAR-based detectors on point clouds.

Analysis: potentials and challenges of the depth-assisted approaches. The
depth-assisted approaches pursue more accurate depth estimation by leveraging
a pre-trained depth prediction network. Both the depth image representation
and the pseudo-LiDAR presentation could significantly boost the monocular
detection performance. Nevertheless, compared to the image-only methods that
only require 3D box annotations, pre-training a depth prediction network
requires expensive ground truth depth maps, and it also hampers the end-to-end
training of the whole framework. Furthermore, pre-trained depth estimation
networks suffer from poor generalization ability. Pretrained depth maps are
usually not well calibrated on the target dataset and typically the scale
needs to be adapted to the target dataset. Thus there remains a non-negligible
domain gap between the source domain leveraged for depth pre-training and the
target domain for monocular detection. Given the fact that driving scenarios
are normally diverse and complex, pre-training depth networks on a restricted
domain may not work well in real-world applications.

####  4.1.3 Prior-guided monocular 3D object detection

Numerous approaches try to tackle the ill-posed monocular 3D object detection
problem by leveraging the hidden prior knowledge of object shapes and scene
geometry from images. The prior knowledge can be learned by introducing pre-
trained sub-networks or auxiliary tasks, and they can provide extra
information or constraints to help accurately localize 3D objects. The broadly
adopted prior knowledge includes object shapes, geometry consistency, temporal
constraints, and segmentation information. An illustration of the prior types
is shown in Figure 13.

Object shapes. Many methods resort to shape reconstruction of 3D objects
directly from images. The reconstructed shapes can be further leveraged to
determine the locations and poses of the 3D objects. There are 5 types of
reconstructed representations: computer-aided design (CAD) models, wireframe
models, signed distance function (SDF), points, and voxels.

Some papers Zeeshan Zia et al. (2014); Chabot et al. (2017); He and Soatto
(2019) learn morphable wireframe models to represent 3D objects. Other works
Kundu et al. (2018); Manhardt et al. (2019); Zakharov et al. (2020); Beker et
al. (2020) leverage DeepSDF Park et al. (2019) to learn implicit signed
distance functions or low-dimensional shape parameters from CAD models, and
they further propose a render-and-compare approach to learn the parameters of
3D objects. Some works Xiang et al. (2015, 2017) utilize voxel patterns to
represent 3D objects. Other papers Ku et al. (2019); Chen et al. (2021a)
resort to point cloud reconstruction from images and estimate the locations of
3D objects with 2D-3D correspondences.

Figure 13: An illustration of the prior types in monocular 3D object detection
methods. Samples are from Chen et al. (2016); He and Soatto (2019); Xiang et
al. (2015); Beker et al. (2020); Park et al. (2019).

Geometric consistency. Given the extrinsics matrix $T\in SE(3)$ that
transforms a 3D coordinate in the object frame to the camera frame, and the
camera intrinsics matrix $K$ that project the 3D coordinate onto the image
plane, the projection of a 3D point $[x,y,z]$ in the object frame into the
image pixel coordinate $[u,v]$ can be represented as

$d\cdot\begin{bmatrix}u\\\ v\\\ 1\end{bmatrix}=K\ T\ \begin{bmatrix}x\\\ y\\\
z\\\ 1\end{bmatrix},$ (23)

where $d$ is the depth of transformed 3D coordinate in the camera frame. Eqn.
23 provides a geometric relationship between 3D points and 2D image pixel
coordinates, which can be leveraged in various ways to encourage consistency
between the predicted 3D objects and the 2D objects on images. There are
mainly $5$ types of geometric constraints in monocular detection: 2D-3D boxes
consistency, keypoints, object’s height-depth relationship, inter-objects
relationship, and ground plane constraints.

Some works Mousavian et al. (2017); Brazil and Liu (2019); Jörgensen et al.
(2019); Naiden et al. (2019) propose to encourage the consistency between 2D
and 3D boxes by minimizing reprojection errors. These methods introduce a
post-processing step to optimize the 3D object parameters by gradually fitting
the projected 3D boxes to 2D bounding boxes on images. There is also a branch
of papers Li et al. (2020c); Shi et al. (2020c); Li and Zhao (2021) that
predict the object keypoints from images, and the keypoints can be leveraged
to calibrate the sizes of locations of 3D objects. Object’s height-depth
relationship can also serve as a strong geometric prior. Specifically, given
the physical height of an object $H$ in the 3D space, the visual height $h$ on
images, and the corresponding depth of the object $d$, there exists a
geometric constraint: $d=f\cdot H/h$, where $f$ is the camera focal length.
This constraint can be leveraged to obtain more accurate depth estimation and
has been broadly applied in a lot of works Cai et al. (2020); Zhang et al.
(2021c); Lu et al. (2021); Shi et al. (2021b). There are also some papers Zhou
et al. (2020b); Chen et al. (2020f) trying to model the inter-objects
relationships by exploiting new geometric relations among objects. Other
papers Chen et al. (2016); Brazil and Liu (2019); Liu et al. (2019a, 2021b)
leverage the assumption that 3D objects are generally on the ground plane to
better localize those objects.

Table 11: A taxonomy of prior-guided monocular detection methods based on prior types. Prior Types | Methods
---|---
Object shape | wireframe |  | Zeeshan Zia et al. (2014); Chabot et al. (2017); He and Soatto (2019)
---
SDF |  | Kundu et al. (2018); Manhardt et al. (2019); Beker et al. (2020); Zakharov et al. (2020)
---
points |  | Chen et al. (2021a); Ku et al. (2019)
---
voxels |  | Xiang et al. (2015, 2017)
---
Geometric consistency | 2D-3D boxes |  | Mousavian et al. (2017); Brazil and Liu (2019); Jörgensen et al. (2019); Naiden et al. (2019)
---
keypoints |  | Li et al. (2020c); Li and Zhao (2021); Shi et al. (2020c)
---
height-depth |  | Cai et al. (2020); Lu et al. (2021); Zhang et al. (2021c); Shi et al. (2021b)
---
inter-objects |  | Zhou et al. (2020b); Chen et al. (2020f)
---
ground plane |  | Chen et al. (2016); Liu et al. (2021b); Brazil and Liu (2019); Liu et al. (2019a)
---
Temporal constraints |  | Hu et al. (2019); Brazil et al. (2020)
---
Segmentation |  | Xiang et al. (2015); Beker et al. (2020); Chen et al. (2016); Heylen et al. (2021)
---

Temporal constraints. Temporal association of 3D objects can be leveraged as
strong prior knowledge. The temporal object relationships have been exploited
as depth-ordering Hu et al. (2019) and multi-frame object fusion with a 3D
Kalman filter Brazil et al. (2020).

Segmentation Image segmentation helps monocular 3D object detection mainly in
two aspects. First, object segmentation masks are crucial for instance shape
reconstruction in some works Xiang et al. (2015); Beker et al. (2020). Second,
segmentation indicates whether an image pixel is inside a 3D object from the
perspective view, and this information has been utilized in Chen et al.
(2016); Heylen et al. (2021) to help localize 3D objects.

Analysis: potentials and challenges of leveraging prior knowledge in monocular
3D detection. With shape reconstruction, we could obtain more detailed object
shape information from images, which is beneficial to 3D object detection. We
can also attain more accurate detection results through the projection or
render-and-compare loss. However, there exist two challenges for shape
reconstruction applied in monocular 3D object detection. First, shape
reconstruction normally requires an additional step of pre-training a
reconstruction network, which hampers end-to-end training of the monocular
detection pipeline. Second, object shapes are generally learned from CAD
models instead of real-world instances, which imposes the challenge of
generalizing the reconstructed objects to real-world scenarios.

Geometric consistencies are broadly adopted and can help improve detection
accuracy. Nevertheless, some methods formulate the geometric consistency as an
optimization problem and optimize object parameters in post-processing, which
is quite time-consuming and hampers end-to-end training.

Image segmentation is useful information in monocular 3D detection. However,
training segmentation networks requires expensive pixel annotations. Pre-
training segmentation models on external datasets will suffer from the
generalization problem.

###  4.2 Stereo-based 3D object detection

Figure 14: An illustration of stereo-based 3D object detection methods. Image
and disparity samples are from Qian et al. (2020).

Problem and Challenge. Stereo-based 3D object detection aims to detect 3D
objects from a pair of images. Compared to monocular images, paired stereo
images provide additional geometric constraints that can be utilized to infer
more accurate depth information. Hence, the stereo-based methods generally
obtain a better detection performance than the monocular-based methods.
Nevertheless, stereo cameras typically require very accurate calibration and
synchronization, which are normally difficult to achieve in real applications.
An illustration of stereo-based 3D object detection approaches is shown in
Figure 14 and a taxonomy is in Table 12.

Stereo matching and depth estimation. A stereo camera can produce a pair of
images, i.e. the left image $\mathcal{I}_{L}$ and the right image
$\mathcal{I}_{R}$, in one shot. With the stereo matching techniques Mayer et
al. (2016); Chang and Chen (2018), a disparity map can be estimated from the
paired stereo images leveraging multi-view geometry Hartley and Zisserman
(2003). Ideally, for each pixel on the left image $\mathcal{I}_{L}(u,v)$,
there exists a pixel on the right image $\mathcal{I}_{R}(u,v+p)$ with the
disparity value $p$ so that the two pixels picture the same 3D location.
Finally, the disparity map can be transformed into a depth image with the
following formula:

$d=\frac{f\times b}{p},$ (24)

where $d$ is the depth value, $f$ is the focal length, and $b$ is the baseline
length of the stereo camera. The pixel-wise disparity constraints from stereo
images enable more accurate depth estimation compared to monocular depth
prediction.

Table 12: A taxonomy of stereo-based detection methods based on auxiliary tasks and data representations. Method |  | 2D
---
Det.
| 2D
---
Seg.
| Disp./
---
Depth
| Pseudo
---
LiDAR
| 3D
---
Volume
3DOP Chen et al. (2015) |  |  | ✓ |  |
TLNet Qin et al. (2019b) | ✓ |  |  |  |
Stereo R-CNN Li et al. (2019b) | ✓ |  |  |  |
Disp R-CNN Sun et al. (2020b) | ✓ | ✓ | ✓ |  |
ZoomNet Xu et al. (2020) | ✓ | ✓ | ✓ |  |
OC-Stereo Pon et al. (2020) | ✓ | ✓ | ✓ |  |
IDA-3D Peng et al. (2020) | ✓ |  | ✓ |  |
YOLOStereo3D Liu et al. (2021a) |  |  | ✓ |  |
SIDE Peng et al. (2022) | ✓ |  | ✓ |  |
P-LiDAR++ You et al. (2020) |  |  | ✓ | ✓ |
Qian et al. Qian et al. (2020) |  |  | ✓ | ✓ |
CDN Garg et al. (2020) |  |  | ✓ | ✓ |
RT3D-Stereo Königshof et al. (2019) |  | ✓ | ✓ | ✓ |
CG-Stereo Li et al. (2020a) |  | ✓ | ✓ | ✓ |
LIGA-Stereo Guo et al. (2021) |  |  | ✓ |  | ✓
DSGN Chen et al. (2020e) |  |  | ✓ |  | ✓
PLUMENet Wang et al. (2021i) |  |  | ✓ |  | ✓

2D-detection based methods. Conventional 2D object detection frameworks can be
modified to resolve the stereo detection problem. Specifically, paired stereo
images are passed through an image-based detector with Siamese backbone
networks to generate left and right regions of interest (RoIs) for the left
and right images respectively. Then in the second stage, the left and right
RoIs are fused to estimate the parameters of 3D objects. Stereo R-CNN Li et
al. (2019b) first proposes to extend 2D detection frameworks to stereo 3D
detection. This design paradigm has been adopted in numerous papers. Qin et
al. (2019b) proposes a novel stereo triangulation learning sub-network at the
second stage; Xu et al. (2020); Pon et al. (2020); Sun et al. (2020b); Chen et
al. (2021b) learn instance-level disparity by object-centric stereo matching
and instance segmentation; Peng et al. (2020) proposes adaptive instance
disparity estimation; Liu et al. (2021a); Peng et al. (2022) introduce single-
stage stereo detection frameworks; Chen et al. (2015, 2017a) propose an
energy-based framework for stereo-based 3D object detection.

Figure 15: An illustration of multi-view 3D object detection methods. Figures
are from Philion and Fidler (2020) and Wang et al. (2022b).

Pseudo-LiDAR based methods. The disparity map predicted from stereo images can
be transformed into the depth image and then converted into the pseudo-LiDAR
point cloud. Hence, similar to the monocular detection methods, the pseudo-
LiDAR representation can also be employed in stereo-based 3D object detection
methods. Those methods try to improve the disparity estimation in stereo
matching for more accurate depth prediction. You et al. (2020) introduces a
depth cost volume in stereo matching networks; Qian et al. (2020) proposes an
end-to-end stereo matching and detection framework; Königshof et al. (2019);
Li et al. (2020a) leverage semantic segmentation and predict disparity for
foreground and background regions separately; Garg et al. (2020) proposes a
Wasserstein loss for disparity estimation.

Volume-based methods. There exists a category of methods that skip the pseudo-
LiDAR representation and perform 3D object detection directly on 3D stereo
volumes. DSGN Chen et al. (2020e) proposes a 3D geometric volume derived from
stereo matching networks and applies a grid-based 3D detector on the volume to
detect 3D objects. Guo et al. (2021) and Wang et al. (2021i) improve Chen et
al. (2020e) by leveraging knowledge distillation and 3D feature volumes
respectively.

Potentials and challenges of the stereo-based methods. Compared to the
monocular detection methods, the stereo-based methods can obtain more accurate
depth and disparity estimation with stereo matching techniques, which brings a
stronger object localization ability and significantly boosts the 3D object
detection performance. Nevertheless, an auxiliary stereo matching network
brings additional time and memory consumption. Compared to LiDAR-based 3D
object detection, detection from stereo images can serve as a much cheaper
solution for 3D perception in autonomous driving scenarios. However, there
still exists a non-negligible performance gap between the stereo-based and the
LiDAR-based 3D object detection approaches.

###  4.3 Multi-view 3D object detection

Problem and Challenge. Autonomous vehicles are generally equipped with
multiple cameras to obtain complete environmental information from multiple
viewpoints. Recently, multi-view 3D object detection has evolved rapidly. Some
multi-view 3D detection approaches try to construct a unified BEV space by
projecting multi-view images into the bird’s-eye view, and then employ a BEV-
based detector on top of the unified BEV feature map to detect 3D objects. The
transformation from camera views to the bird’s-eye view is ambiguous without
accurate depth information, so image pixels and their BEV locations are not
perfectly aligned. How to build reliable transformations from camera views to
the bird’s-eye view is a major challenge in these methods. Other methods
resort to 3D object queries that are generated from the bird’s-eye view and
Transformers where cross-view attention is applied to object queries and
multi-view image features. The major challenge is how to properly generate 3D
object queries and design more effective attention mechanisms in Transformers.

BEV-based multi-view 3D object detection. LSS Philion and Fidler (2020) is a
pioneering work that proposes a lift-splat-shoot paradigm to solve the problem
of BEV perception from multi-view cameras. There are three steps in LSS. Lift:
bin-based depth prediction is conducted on image pixels and multi-view image
features are lifted to 3D frustums with depth bins. Splat: 3D frustums are
splatted into a unified bird’s-eye view plane and image features are
transformed into BEV features in an end-to-end manner. Shoot: downstream
perception tasks are performed on top of the BEV feature map. This paradigm
has been successfully adopted by many following works. BEVDet Huang et al.
(2021); Huang and Huang (2022) improves LSS Philion and Fidler (2020) with a
four-step multi-view detection pipeline, where the image view encoder encodes
features from multi-view images, the view transformer transforms image
features from camera views to the bird’s-eye view, the BEV encoder further
encodes the BEV features, and the detection head is employed on top of the BEV
features for 3D detection. The major bottleneck in Huang et al. (2021);
Philion and Fidler (2020) is depth prediction, as it is normally inaccurate
and will result in inaccurate feature transforms from camera views to the
bird’s-eye view. To obtain more accurate depth information, many papers resort
to mining additional information from multi-view images and past frames, e.g.
Li et al. (2022c) leverages explicit depth supervision, Wang et al. (2022c)
introduces surround-view temporal stereo, Li et al. (2022a) uses dynamic
temporal stereo, Park et al. (2022) combines both short-term and long-term
temporal stereo for depth prediction. In addition, there are also some papers
Xie et al. (2022); Huang et al. (2022a) that completely abandon the design of
depth bins and categorical depth prediction. They simply assume that the depth
distribution along the ray is uniform, so the camera-to-BEV transformation can
be conducted with higher efficiency.

Query-based multi-view 3D object detection. In addition to the BEV-based
approaches, there is also a category of methods where object queries are
generated from the bird’s-eye view and interact with camera view features.
Inspired by the advances in Transformers for object detection Carion et al.
(2020), DETR3D Wang et al. (2022b) introduces a sparse set of 3D object
queries, and each query corresponds to a 3D reference point. The 3D reference
points can collect image features by projecting their 3D locations onto the
multi-view image planes and then object queries interact with image features
through Transformer layers. Finally, each object query will decode a 3D
bounding box. Many following papers try to improve this design paradigm, such
as introducing spatially-aware cross-view attention Doll et al. (2022) and
adding 3D positional embeddings on top of image features Liu et al. (2022a).
BEVFormer Li et al. (2022f) introduces dense grid-based BEV queries and each
query corresponds to a pillar that contains a set of 3D reference points.
Spatial cross-attention is applied to object queries and sparse image features
to obtain spatial information, and temporal self-attention is applied to
object queries and past BEV queries to fuse temporal information.

Figure 16: Chronological overview of the multi-modal 3D object detection methods. Table 13: A taxonomy of multi-sensor fusion-based detection methods based on fused stages, representations, and operators. Method | Fusion Stage | Fusion Representation |  Fusion Operator
---|---|---|---
| Input
---
| Backbone
---
| Proposal
---
| RoI
---
| Output
---
Camera rep. | LiDAR rep.
F-PointNet Qi et al. (2018) | ✓ |  |  |  |  | frustum | point cloud | region selection
F-ConvNet Wang and Jia (2019) | ✓ |  |  |  |  | frustum | point cloud | region selection
RoarNet Shin et al. (2019) | ✓ |  |  |  |  | 2D boxes & poses | point cloud | region selection
PointPainting Vora et al. (2020) | ✓ |  |  |  |  | 2D segmentation | point cloud | point-wise append
MVX-Net Sindagi et al. (2019) |  | ✓ |  |  |  | image features | voxels | concatenation & MLP
ContFuse Liang et al. (2018) |  | ✓ |  |  |  | image features | BEV features | continuous convolution
PointFusion Xu et al. (2018) |  | ✓ |  |  |  | image features | point features | concatenation & MLP
EPNet Huang et al. (2020b) |  | ✓ |  |  |  | image features | point features | point-wise attention
MMF Liang et al. (2019) |  | ✓ | ✓ | ✓ |  | image features | BEV features | continuous convolution
3D-CVF Yoo et al. (2020) |  | ✓ | ✓ | ✓ |  | image features | BEV features | gated attention
MV3D Chen et al. (2017b) |  |  | ✓ | ✓ |  | image features | multi-view features | concatenation & MLP
AVOD Ku et al. (2018) |  |  | ✓ | ✓ |  | image features | BEV features | concatenation & MLP
CLOCs Pang et al. (2020) |  |  |  |  | ✓ | 2D boxes | 3D boxes | box consistency

##  5 Multi-Modal 3D Object Detection

In this section, we introduce the multi-modal 3D object detection approaches
that fuse multiple sensory inputs. According to the sensor types, the
approaches can be divided into three categories: LiDAR-camera, radar, and map
fusion-based methods. In Section 5.1, we review and analyze the multi-modal
detection approaches with LiDAR-camera fusion, including the early-fusion
based, the intermediate-fusion based, and the late-fusion based methods. In
Section 5.2, we investigate the multi-modal detection approaches with radar
signals. In Section 5.3, we introduce the multi-modal 3D detection approaches
with high-definition maps. A chronological overview of the multi-modal 3D
object detection approaches is shown in Figure 16.

###  5.1 Multi-modal detection with LiDAR-camera fusion

Problem and Challenge. Camera and LiDAR are two complementary sensor types for
3D object detection. Cameras provide color information from which rich
semantic features can be extracted, while LiDAR sensors specialize in 3D
localization and provide rich information about 3D structures. Many endeavors
have been made to fuse the information from cameras and LiDARs for accurate 3D
object detection. Since LiDAR-based detection methods perform much better than
camera-based methods, the state-of-the-art approaches are mainly based on
LiDAR-based 3D object detectors and try to incorporate image information into
different stages of a LiDAR detection pipeline. In view of the complexity of
LiDAR-based and camera-based detection systems, combining the two modalities
together inevitably brings additional computational overhead and inference
time latency. Therefore, how to efficiently fuse the multi-modal information
remains an open challenge. A taxonomy of multi-modal 3D object detection
methods is in Table 13.

####  5.1.1 Early-fusion based 3D object detection

Figure 17: An illustration of early-fusion based 3D object detection methods.

Early-fusion based methods aim to incorporate the knowledge from images into
point cloud before they are fed into a LiDAR-based detection pipeline. Hence
the early-fusion frameworks are generally built in a sequential manner: 2D
detection or segmentation networks are firstly employed to extract knowledge
from images, and then the image knowledge is passed to point cloud, and
finally the enhanced point cloud is fed to a LiDAR-based 3D object detector.
Based on the fusion types, the early-fusion methods can be divided into two
categories: region-level knowledge fusion and point-level knowledge fusion. An
illustration of the early-fusion based approaches is shown in Figure 17.

Region-level knowledge fusion. Region-level fusion methods aim to leverage
knowledge from images to narrow down the object candidate regions in 3D point
cloud. Specifically, an image is first passed through a 2D object detector to
generate 2D bounding boxes, and then the 2D boxes are extruded into 3D viewing
frustums. The 3D viewing frustums are applied on LiDAR point cloud to reduce
the searching space. Finally, only the selected point cloud regions are fed
into a LiDAR detector for 3D object detection. F-PointNet Qi et al. (2018)
first proposes this fusion mechanism, and many endeavors have been made to
improve the fusion framework. Wang and Jia (2019) divides a viewing frustum
into grid cells and applies a convolutional network on the grid cells for 3D
detection; Shin et al. (2019) proposes a novel geometric agreement search;
Paigwar et al. (2021) exploits the pillar representation; Du et al. (2018)
introduces a model fitting algorithm to find the object point cloud inside
each frustum.

Figure 18: An illustration of intermediate-fusion based 3D object detection
methods. Figure 19: An illustration of late-fusion based 3D object detection
methods.

Point-level knowledge fusion. Point-level fusion methods aim to augment input
point cloud with image features. The augmented point cloud is then fed into a
LiDAR detector to attain a better detection result. PointPainting Vora et al.
(2020) is a seminal work that leverages image-based semantic segmentation to
augment point clouds. Specifically, an image is passed through a segmentation
network to obtain pixel-wise semantic labels, and then the semantic labels are
attached to the 3D points by point-to-pixel projection. Finally, the points
with semantic labels are fed into a LiDAR-based 3D object detector. This
design paradigm has been followed by a lot of papers Xu et al. (2021c); Simon
et al. (2019); Meyer et al. (2019a). Apart from semantic segmentation, there
also exist some works trying to exploit other information from images, e.g.
depth image completion Yin et al. (2021b).

Analysis: potentials and challenges of the early-fusion methods. The early-
fusion based methods focus on augmenting point clouds with image information
before they are passed through a LiDAR 3D object detection pipeline. Most
methods are compatible with a wide range of LiDAR-based 3D object detectors
and can serve as a quite effective pre-processing step to boost detection
performance. Nevertheless, the early-fusion methods generally perform multi-
modal fusion and 3D object detection in a sequential manner, which brings
additional inference latency. Given the fact that the fusion step generally
requires a complicated 2D object detection or semantic segmentation network,
the time cost brought by multi-modal fusion is normally non-negligible. Hence,
how to perform multi-modal fusion efficiently at the early stage has become a
critical challenge.

####  5.1.2 Intermediate-fusion based 3D object detection

Intermediate-fusion based methods try to fuse image and LiDAR features at the
intermediate stages of a LiDAR-based 3D object detector, e.g. in backbone
networks, at the proposal generation stage, or at the RoI refinement stage.
These methods can also be classified according to the fusion stages. An
illustration of intermediate-fusion based approaches is shown in Figure 18.

Fusion in backbone networks. Many endeavors have been made to progressively
fuse image and LiDAR features in the backbone networks. In those methods,
point-to-pixel correspondences are firstly established by LiDAR-to-camera
transform, and then with the point-to-pixel correspondences, features from a
LiDAR backbone can be fused with features from an image backbone through
different fusion operators. The multi-modal fusion can be conducted in the
intermediate layers of a grid-based detection backbone, with novel fusion
operators such as continuous convolutions Wang et al. (2018); Liang et al.
(2018, 2019), hybrid voxel feature encoding Sindagi et al. (2019), and
Transformer Li et al. (2022e); Zhang et al. (2022a). The multi-modal fusion
can also be conducted only at the output feature maps of backbone networks,
with fusion modules and operators including gated attention Yoo et al. (2020),
unified object queries Chen et al. (2022a), BEV pooling Liu et al. (2022b),
learnable alignments Chen et al. (2022c), point-to-ray fusion Li et al.
(2022d), Transformer Bai et al. (2022), and other techniques Dou et al.
(2019); Chen et al. (2020d); Wang et al. (2021a). In addition to the fusion in
grid-based backbones, there also exist some papers incorporating image
information into the point-based detection backbones Xu et al. (2018); Huang
et al. (2020b); Xie et al. (2020a); Wang et al. (2021k); Zhu et al. (2021a).

Fusion in proposal generation and RoI head. There exists a category of works
that conduct multi-modal feature fusion at the proposal generation and RoI
refinement stage. In those methods, 3D object proposals are first generated
from a LiDAR detector, and then the 3D proposals are projected into multiple
views, i.e. the image view and bird’s-eye view, to crop features from the
image and LiDAR backbone respectively. Finally, the cropped image and LiDAR
features are fused in an RoI head to predict parameters for each 3D object.
MV3D Chen et al. (2017b) and AVOD Ku et al. (2018) are pioneering works
leveraging multi-view aggregation for multi-modal detection. Other papers Chen
et al. (2022a); Bai et al. (2022) use the Transformer Vaswani et al. (2017)
decoder as the RoI head for multi-modal feature fusion.

Analysis: potentials and challenges of the intermediate-fusion methods. The
intermediate methods encourage deeper integration of multi-modal
representations and yield 3D boxes of higher quality. Nevertheless, camera and
LiDAR features are intrinsically heterogeneous and come from different
viewpoints, so there still exist some problems on the fusion mechanisms and
view alignments. Hence, how to fuse the heterogeneous data effectively and how
to deal with the feature aggregation from multiple views remain a challenge to
the research community.

####  5.1.3 Late-fusion based 3D object detection

Fusion at the box level. Late-fusion based approaches operate on the outputs,
i.e. 3D and 2D bounding boxes, from a LiDAR-based 3D object detector and an
image-based 2D object detector respectively. An illustration of late-fusion
based approaches is shown in Figure 19. In those methods, object detection
with camera and LiDAR sensor can be conducted in parallel, and the output 2D
and 3D boxes are fused to yield more accurate 3D detection results. CLOCs Pang
et al. (2020) introduces a sparse tensor that contains paired 2D-3D boxes and
learns the final object confidence scores from this sparse tensor. Pang et al.
(2022) improves Pang et al. (2020) by introducing a light-weight 3D detector-
cued image detector.

Analysis: potentials and challenges of the late-fusion methods. The late-
fusion based approaches focus on the instance-level aggregation and perform
multi-modal fusion only on the outputs of different modalities, which avoids
complicated interactions on the intermediate features or on the input point
cloud. Hence these methods are much more efficient compared to other
approaches. However, without resorting to deep features from camera and LiDAR
sensors, these methods fail to integrate rich semantic information of
different modalities, which limits the potential of this category of methods.

###  5.2 Multi-modal detection with radar signals

Problem and Challenge. Radar is an important sensory type in driving systems.
In contrast to LiDAR sensors, radar has four irreplaceable advantages in real-
world applications: Radar is much cheaper than LiDAR sensors; Radar is less
vulnerable to extreme weather conditions; Radar has a larger detection range;
Radar provides additional velocity measurements. Nevertheless, compared to
LiDAR sensors that generate dense point clouds, radar only provides sparse and
noisy measurements. Hence, how to effectively handle the radar signals remains
a critical challenge.

Radar-LiDAR fusion. Many papers try to fuse the two modalities by introducing
new fusion mechanisms to enable message passing between the radar and LiDAR
signals, including voxel-based fusion Yang et al. (2020a), attention-based
fusion Qian et al. (2021a), introducing a range-azimuth-doppler tensor Major
et al. (2019), leveraging graph neural networks Meyer et al. (2021),
exploiting dynamic occupancy maps Wang and Goldluecke (2021), and introducing
4D radar data Palffy et al. (2022).

Radar-camera fusion. Radar-camera fusion is quite similar to LiDAR-camera
fusion, as both radar and LiDAR data are 3D point representations. Most radar-
camera approaches Chadwick et al. (2019); Nabati and Qi (2019, 2021) adapt the
existing LiDAR-based detection architectures to handle sparse radar points and
adopt similar fusion strategies as LiDAR-camera based methods.

###  5.3 Multi-modal detection with high-definition maps

Problem and Challenge. High-definition maps (HD maps) contain detailed road
information such as road shape, road marking, traffic signs, barriers, etc. HD
maps provide rich semantic information on surrounding environments and can be
leveraged as a strong prior to assist 3D object detection. How to effectively
incorporate map information into a 3D object detection framework has become an
open challenge to the research community.

Figure 20: Chronological overview of Transformer-based 3D object detectors.

Multi-modal detection with map information. High-definition maps can be
readily transformed into a bird’s-eye view representation and fused with
rasterized BEV point clouds or feature maps. The fusion can be conducted by
simply concatenating the channels of a rasterized point cloud and an HD map
from the bird’s-eye view Yang et al. (2018a), feeding LiDAR point cloud and HD
map into separate backbones and fusing the output feature maps of the two
modalities Fang et al. (2021a), or simply filtering out those predictions that
do not fall into the relevant map regions Caesar et al. (2020). Other map
types have also been explored, e.g. visibility map Hu et al. (2020),
vectorized map Jiang et al. (2022).

##  6 Transformer-based 3D Object Detection

In this section, we introduce the Transformer-based 3D object detection
methods. Transformers Vaswani et al. (2017) have shown prominent performance
in many computer vision tasks, and many endeavors have been made to adapt
Transformers to 3D object detection. In Section 6.1, we review the
Transformers tailored for 3D object detection from an architectural
perspective. In Section 6.2, we introduce the applications of Transformers in
different 3D object detectors.

###  6.1 Transformer architectures for 3D object detection

Problem and Challenge. While most 3D object detectors are based on
convolutional architectures, recently Transformer-based 3D detectors have
shown great potential and dominated 3D object detection leaderboards. Compared
to convolutional networks, the query-key-value design in Transformers enables
more flexible interactions between different representations and the self-
attention mechanism results in a larger receptive field than convolutions.
However, fully-connected self-attention has quadratic time and space
complexity w.r.t. the number of inputs, training Transformers can easily fall
into sub-optimal results when the data size is small. Hence, it’s critical to
define proper query-key-value triplets and design specialized attention
mechanisms for Transformer-based 3D object detectors.

Transformer architectures. The development of Transformer architectures in 3D
object detection has experienced three stages: (1) Inspired by vanilla
Transformer Vaswani et al. (2017), new Transformer modules with special
attention mechanisms are proposed to obtain more powerful features in 3D
object detection. (2) Inspired by DETR Carion et al. (2020), query-based
Transformer encoder-decoder designs are introduced to 3D object detectors. (3)
Inspired by ViT Dosovitskiy et al. (2021), patch-based inputs and
architectures similar to Vision Transformers are introduced in 3D object
detection.

In the first stage, many papers try to introduce novel Transformer modules
into conventional 3D detection pipelines. In these papers, the choices of
query, key, and value are quite flexible and new attention mechanisms are
proposed. Pointformer Pan et al. (2021) introduces Transformer modules to
point backbones. It takes point features and coordinates as queries and
applies self-attention to a group of point clouds. Voxel Transformer Mao et
al. (2021c) replaces convolutional voxel backbones with Transformer modules,
where sparse and submanifold voxel attention are proposed and applied to
voxels. CT3D Sheng et al. (2021) proposes a novel Transformer-based detection
head, where proposal-to-point attention and channel-wise attention are
introduced.

In the second stage, many papers propose DETR-like architectures for 3D object
detection. They leverage a set of object queries and use those queries to
interact with different features to predict 3D boxes. DETR3D Wang et al.
(2022b) introduces object queries and generates a 3D reference point for each
query. They use reference points to aggregate multi-view image features as
keys and values, and apply cross-attention between object queries and image
features. Finally, each query can decode a 3D bounding box for detection. Many
following works have adopted the design of object queries and reference
points. BEVFormer Li et al. (2022f) generates dense queries from BEV grids.
TransFusion Bai et al. (2022) produces object queries from initial detections
and applies cross-attention to LiDAR and image features in a Transformer
decoder. UVTR Li et al. (2022b) fuses object queries with image and LiDAR
voxels in a Transformer decoder. FUTR3D Chen et al. (2022a) fuses object
queries with features from different sensors in a unified way.

In the third stage, many papers try to apply the designs of Vision
Transformers to 3D object detectors. Following Dosovitskiy et al. (2021); Liu
et al. (2021c), they split inputs into patches and apply self-attention within
each patch and across different patches. SST Fan et al. (2022) proposes a
sparse Transformer, in which voxels in a local region are grouped into a patch
and sparse regional attention is applied to the voxels in a patch, and then
region shift is applied to change the grouping so new patches can be
generated. SWFormer Sun et al. (2022) improves Fan et al. (2022) with multi-
scale feature fusion and voxel diffusion.

###  6.2 Transformer applications in 3D object detection

Applications of Transformer-based 3D detectors. Transformer architectures have
been broadly adopted in various types of 3D object detectors. For point-based
3D object detectors, a point-based Transformer Pan et al. (2021) has been
developed to replace the conventional PointNet backbone. For voxel-based 3D
detectors, a lot of papers Mao et al. (2021c); Fan et al. (2022); Sun et al.
(2022) propose novel voxel-based Transformers to replace the conventional
convolutional backbone. For point-voxel based 3D object detectors, a new
Transformer-based detection head Sheng et al. (2021) has been proposed for
better proposal refinement. For monocular 3D object detectors, Transformers
can be used to fuse image and depth features Huang et al. (2022b). For multi-
view 3D object detectors, Transformers are utilized to fuse multi-view image
features for each query Li et al. (2022f); Wang et al. (2022b). For multi-
modal 3D object detectors, many papers Bai et al. (2022); Li et al. (2022b);
Chen et al. (2022a) leverage Transformer architectures and special cross-
attention mechanisms to fuse features of different modalities. For temporal 3D
object detectors, Temporal-Channel Transformer Yuan et al. (2021) is proposed
to model temporal relationships across LiDAR frames.

Figure 21: Chronological overview of the temporal 3D object detection methods.

##  7 Temporal 3D Object Detection

In this section, we introduce the temporal 3D object detection methods. Based
on the data types, these methods can be divided into three categories:
detection from LiDAR sequences, detection from streaming inputs, and detection
from videos. In Section 7.1, we review the 3D object detection methods
leveraging sequential LiDAR sweeps. In Section 7.2, we introduce the detection
approaches with streaming data as input. In Section 7.3, we investigate 3D
detection from videos and multi-modal temporal data. A chronological overview
of the temporal detection approaches is shown in Figure 21 and a taxonomy is
in Table 14.

Table 14: A taxonomy of temporal 3D object detection methods based on input representations. Input | Methods
---|---
LiDAR | multi-frame point clouds |  | Huang et al. (2020a); Yang et al. (2021c); Zhang et al. (2020a); Yin et al. (2020); Yuan et al. (2021)
---
Qi et al. (2021); Yang et al. (2021a)
multi-frame range images | Meyer et al. (2020); Laddha et al. (2020)
streaming inputs | Han et al. (2020); Frossard et al. (2021); Chen et al. (2021c)
Camera | videos | Hu et al. (2019); Brazil et al. (2020); Piergiovanni et al. (2021)
Figure 22: An illustration of detection from LiDAR sequences.

###  7.1 3D object detection from LiDAR sequences

Problem and Challenge. While most methods focus on detection from a single-
frame point cloud, there also exist many approaches leveraging multi-frame
point clouds for more accurate 3D object detection. These methods are trying
to tackle the temporal detection problem by fusing multi-frame features via
various temporal modeling tools, and they can also obtain more complete 3D
shapes by merging multi-frame object points into a single frame. Temporal 3D
object detection has exhibited great success in offline 3D auto-labeling
pipelines. However, in onboard applications, these methods still suffer from
memory and latency issues, as processing multiple frames inevitably brings
additional time and memory costs, which can become severe when models are
running on embedded devices. An illustration of temporal 3D object detection
from LiDAR sequences is shown in Figure 22.

3D object detection from sequential sweeps. Most detection approaches using
multi-frame point clouds resort to proposal-level temporal information
aggregation. Namely, 3D object proposals are first generated independently
from each frame of point cloud through a shared detector, and then various
temporal modules are applied on the object proposals and the respective RoI
features to aggregate the information of objects across different frames. The
adopted temporal aggregation modules include temporal attention Yang et al.
(2021c), ConvGRU Yin et al. (2020), graph network Zhang et al. (2020a), LSTM
Huang et al. (2020a), and Transformer Yuan et al. (2021). Temporal 3D object
detection is also applied in the 3D object auto-labeling pipelines Qi et al.
(2021); Yang et al. (2021a). In addition to temporal detection from multi-
frame point clouds, there are also some works Meyer et al. (2020); Laddha et
al. (2020) leveraging sequential range images for 3D object detection.

Figure 23: An illustration of streaming 3D object detection. Reference:
Frossard et al. (2021) and Chen et al. (2021c).

###  7.2 3D object detection from streaming data

Problem and Challenge. Point clouds collected by rotating LiDARs are
intrinsically a streaming data source in which LiDAR packets are sequentially
recorded in a sweep. It typically takes $50$-$100$ ms for a rotating LiDAR
sensor to generate a $360^{\circ}$ complete LiDAR sweep, which means that by
the time a point cloud is produced, it no longer accurately reflects the scene
at the exact time. This poses a challenge to autonomous driving applications
which generally require minimal reaction times to guarantee driving safety.
Many endeavors have been made to directly detect 3D objects from the streaming
data. These methods generally detect 3D objects on the active LiDAR packets
immediately without waiting for the full sweep to be built. Streaming 3D
object detection is a more accurate and low-latency solution to vehicle
perception compared to detection from full LiDAR sweeps. An illustration of 3D
object detection from streaming data is shown in Figure 23.

Streaming 3D object detection. Similar to temporal detection from multi-frame
point clouds, streaming detection methods Han et al. (2020) can treat each
LiDAR packet as an independent sample to detect 3D objects and apply temporal
modules on the sequential packets to learn the inter-packets relationships.
However, a LiDAR packet normally contains an incomplete point cloud and the
information from a single packet is generally not sufficient for accurately
detecting 3D objects. To this end, some papers try to provide more context
information for detection in a single packet. The proposed techniques include
a spatial memory bank Frossard et al. (2021) and a multi-scale context padding
scheme Chen et al. (2021c).

###  7.3 3D object detection from videos

Problem and Challenge. Video is an important data type and can be easily
obtained in autonomous driving applications. Compared to single-image based 3D
object detection, video-based 3D detection naturally benefits from the
temporal relationships of sequential images. While numerous works focus on
single-image based 3D object detection, only a few papers investigate the
problem of 3D object detection from videos, which leaves an open challenge to
the research community.

Figure 24: An illustration of domain gaps in 3D detection.

Video-based 3D object detection. Video-based detection approaches generally
extend the image-based 3D object detectors by tracking and fusing the same
objects across different frames. The proposed trackers include LSTM Hu et al.
(2019) and the 3D Kalman filter Brazil et al. (2020). In addition, there are
some works Piergiovanni et al. (2021); Zeng et al. (2022) leveraging both
videos and multi-frame point clouds for more accurate 3D object detection.
Those methods propose 4D sensor-time fusion to learn features from both
temporal and multi-modal data.

##  8 Label-Efficient 3D Object Detection

In this section, we introduce the methods of label-efficient 3D object
detection. In previous sections, we generally assume the 3D detectors are
trained under full supervision on a specific data domain and with a sufficient
amount of annotations. However, in real-world applications, the 3D object
detection methods inevitably face the problems of poor generalizability and
lacking annotations. To address these issues, label-efficient techniques can
be employed in 3D object detection, including domain adaptation (Section 8.1),
weakly-supervised learning (Section 8.2), semi-supervised learning (Section
8.3), and self-supervised learning (Section 8.4) for 3D object detection. We
will introduce those techniques in the following sections.

Table 15: A taxonomy of domain adaptation methods for 3D object detection based on transferred domains and techniques. | Method
---
| Transferred
---
Domain
| Technique
---
Wang et al. Wang et al. (2020e) | cross-sensor | statistics normalization
SF-UDA3D Saltori et al. (2020) | cross-sensor | self-training
ST3D Yang et al. (2021b) | cross-sensor | self-training
FAST3D Fruhwirth-Reisinger et al. (2021) | cross-sensor | self-training
SRDAN Zhang et al. (2021a) | cross-sensor | domain alignments
MLC-Net Luo et al. (2021b) | cross-sensor | domain alignments
3D-CoCo Yihan et al. (2021) | cross-sensor | domain alignments
Rist et al. Rist et al. (2019) | cross-sensor | multi-task learning
PIT Gu et al. (2021) | cross-sensor | image transform
SPG Xu et al. (2021b) | cross-weather | semantic point generation
Saleh et al. Saleh et al. (2019) | sim-to-real | Cycle GAN
DeBortoli et al. DeBortoli et al. (2021) | sim-to-real | adversarial training

###  8.1 Domain adaptation for 3D object detection

Problem and Challenge. Domain gaps are ubiquitous in the data collection
process. Different sensor settings and placements, different geographical
locations, and different weathers will result in completely different data
domains. In most conditions, 3D object detectors trained on a certain domain
cannot perform well on other domains. Many techniques have been proposed to
address the domain adaptation problem for 3D object detection, e.g. leveraging
consistency between source and target domains, and self-training on target
domains. Nevertheless, most methods only focus on solving one specific domain
transfer problem. Designing a domain adaptation approach that can be generally
applied in any domain transfer tasks in 3d object detection will be a
promising research direction. An illustration of the domain gaps in 3D object
detection is shown in Figure 24 and a taxonomy of the domain adaptive methods
is in Table 15

Cross-sensor domain adaptation. Different datasets have different sensory
settings, e.g. a 32-beam LiDAR sensor used in nuScenes Caesar et al. (2020)
versus a 64-beam LiDAR sensor in KITTI Geiger et al. (2012), and the data is
also collected at different geographic locations, e.g. KITTI Geiger et al.
(2012) is collected in Germany while Waymo Sun et al. (2020c) is collected in
United States. These factors will lead to severe domain gaps between different
datasets, and the detectors trained on a dataset generally exhibit quite poor
performance when they are tested on other datasets. Wang et al. (2020e) is a
notable work that observes the domain gaps between datasets, and they
introduce a statistic normalization approach to handle the gaps. Many
following works leverage self-training to resolve the domain adaptation
problem. In those methods, a detector pre-trained on the source dataset will
produce pseudo labels for the target dataset, and then the detector is re-
trained on the target dataset with pseudo labels. These methods make
improvements mainly on obtaining pseudo labels of higher quality, e.g. Saltori
et al. (2020) proposes a scale-and-detect strategy, Yang et al. (2021b)