11institutetext: IFM Lab, Department of Computer Science, UC Davis CA 95616,
USA 11email<EMAIL_ADDRESS>11email<EMAIL_ADDRESS>11email:
<EMAIL_ADDRESS>
# EffiPerception: an Efficient Framework for Various Perception Tasks

Xinhao Xiang 11 Simon Dräger 11 Jiawei Zhang 11

###### Abstract

The accuracy-speed-memory trade-off is always the priority to consider for
several computer vision perception tasks. Previous methods mainly focus on a
single or small couple of these tasks, such as creating effective data
augmentation, feature extractor, learning strategies, etc. These approaches,
however, could be inherently task-specific: their proposed model’s performance
may depend on a specific perception task or a dataset. Targeting to explore
common learning patterns and increasing the module robustness, we propose the
EffiPerception framework. It could achieve great accuracy-speed performance
with relatively low memory cost under several perception tasks: 2D Object
Detection, 3D Object Detection, 2D Instance Segmentation, and 3D Point Cloud
Segmentation. Overall, the framework consists of three parts: (1) Efficient
Feature Extractors, which extract the input features for each modality. (2)
Efficient Layers, plug-in plug-out layers that further process the feature
representation, aggregating core learned information while pruning noisy
proposals. (3) The EffiOptim, an 8-bit optimizer to further cut down the
computational cost and facilitate performance stability. Extensive experiments
on the KITTI, semantic-KITTI, and COCO datasets revealed that EffiPerception
could show great accuracy-speed-memory overall performance increase within the
four detection and segmentation tasks, in comparison to earlier, well-
respected methods.

###### Keywords:

Computer Vision Perception Tasks Efficient Frameworks

##  1 Introduction

Models | 2D perception | 3D perception
---|---|---
OD | IS | OD | PCS
[12, 78, 40, 15, 120, 63, 67, 100, 31, 77, 36] | ✓ | ✓ | ✗ | ✗
[98, 81, 79, 110, 115, 93] | ✗ | ✗ | ✓ | ✓
[56] | ✓ | ✗ | ✓ | ✗
[122] | ✗ | ✓ | ✓ | ✗
EffiPerception | ✓ | ✓ | ✓ | ✓
Table 1: EffiPerception is the first generic perception framework for the four
perception tasks: 2D Object Detection (2D OD), 2D Instance Segmentation(IS),
3D Object Detection(3D OD), and Point Cloud Segmentation(PCS)

Segmentation and object detection are two significant perception tasks in the
fields of computer vision and robotics. With the emergence of deep neural
networks[76, 75, 61, 32, 96, 46], as well as large labeled image datasets[17,
57], 2D object detection and instance segmentation systems from RGB images
have been significantly improved. For applications related to robotics, such
as self-driving cars, grasping, etc., a 2D object detection system is not
adequate for their perception tasks. Thus, 3D object detection and
segmentation systems have then been developed[73, 118, 81, 102, 53]. Compared
to the 2D tasks, 3D object detection and semantic segmentation need to collect
additional depth information, which could be gained from the Lidar point cloud
or other depth-only sensors.

Although some of these methods[87, 46, 81] could reach relatively high
detection/segmentation accuracy, they may not take full consideration of
inference speed and memory cost. Several methods are proposed to fill these
gaps. For example, compact sparse R-CNN[35] speeds up the classic sparse
R-CNN[87] and reduces the memory cost by reducing iterative detection heads
and simplifying the feature pyramid network. In 3D object detection, using
sparse convolutions-based methods[119, 23, 10] is also a common approach to
speed it up.

Despite reaching relatively high performance, these methods mainly focus on a
single or small couple of perception tasks. As a result, it could be
inherently task/dataset-specific. For example, Sparse convolutions-based
methods[119, 23, 10] have shown high performance on 3D object detection, but
it is still a question whether it could also reach the same competitive
performance for 2D object detection and segmentation tasks. On the other hand,
the compact strategy[35] well-used in sparse R-CNN may not provide the same
performance gain in 3D perception tasks. Could there exist a unified framework
that is efficient for various perception tasks, under corresponding input
extractor and perception head? To develop such a framework, three aspects
could be re-investigated.

The first aspect comes from the input feature extractors. For point cloud
data, point-based, voxel-based, and range-based are the three main extraction
approaches[69]. Before the extraction, some pre-processing techniques may also
be applied, such as the random sampling[44], set abstraction[73], farthest
point sampling(FPS)[74], segmentation guided filtering[109], coordinate
refinement[72], etc. Different extractors and pre-processing strategy choices
could lead to huge differences. For example, increasing the number of context
points will gain more representation power but at the cost of increasing
memory consumption[73]. Suitable context radius in ball query is another key
factor to balance the efficacy and efficiency of detection models[74]. As for
image data, besides using the raw RGB pixel data, extra depth information,
stereo information, or multi-view input could also contribute to image
extraction. Additional information increases the diversity of the input space
though at the same time adds more challenges to the following feature
learning. In this paper, we designed a unified extractor for each modality
called the Efficient Feature Extractors. For image feature extraction, we
partition raw RGB images into smaller non-overlapping patches, flatten them,
and conduct local learning. For point cloud feature extraction, we adopt
voxelization, random sampling, outlier removal, data augmentation, flattening
them, and finally conducting local learning. Under these settings, it could
capitalize on the strengths of various previous approaches while easing their
weaknesses, showing great performance increases with higher inference speed
and lower training memory usage.

The second aspect arises from the following learning layers. Previous
approaches rely heavily on their task-driven or task-centered measures, such
as adopting data augmentation[55, 97, 58], or precisely defining its loss
function combined with training strategies[7, 119]. Take an example,
DeepFusion[55] effectively aligns the transformed features from the input
image and point cloud. One core technique named InverseAug is for inversing
geometric-related augmentations to enable accurate geometric alignment between
lidar points and image pixels. While this measure performs excellently on
their focusing fusion-based 3D object detection tasks, it may not be
generalized for other non-fusion tasks. Therefore, previous works usually
adopted different operator and learning layers for camera-only, LiDAR-only,
and camera-LiDAR fusion tasks. These tasks, however, have many similarities,
suggesting a general unified framework should be a more efficient candidate.
To this end, we propose the efficient layers, which are more general layers
that could act as the plug-in plug-out model in various perception tasks.
Having extracted features from input extractors, the efficient layers further
process them, aggregating core learned information while pruning noisy
proposals. Being well-compatible with all four perception tasks, it further
accelerates the inference speed while cutting training memory usage even more.

Figure 1: Overview of EffiPerception, an efficient framework for various
perception tasks

The third aspect is the training framework, such as the optimizing scheme,
training strategy, etc. A more efficient training framework would generate a
more competitive output in a unified manner. By utilizing an 8-bit
optimizer[18] that works on perception tasks, we named it the EffiOptim to
reduce training memory consumption while accelerating inference speed without
compromising accuracy. Its features of scalability and integration indicate
great potential application scenes of training/deploying larger models on
resource-constrained devices.

We conduct experience for various perception tasks under three datasets:
COCO[57] for 2D object detection and segmentation; KITTI[26] for 3D object
detection; and Semantic-KITTI[2] for point cloud segmentation. By integrating
various baselines with our EffiPerception framework, its accuracy-speed
performance could always be maintained at a high level, within relatively low
memory cost. This leads to a significant boost in the overall performance
compared to their raw baselines. In addition, since we focus on improving the
common parts among various tasks, the framework shows excellent robust
scalability. In summary, our main contributions are as follows:

  * •

We proposed an efficient framework called EffiPerception to achieve better
accuracy-speed performance with relatively low memory cost under several
perception tasks.

  * •

We investigated exploiting common layer patterns learning strategies among
tasks, and proposed two generic layers: Sparse Down-Sampling and Global
Spatial Aggregation, to increase the framework’s robustness and efficiency.

  * •

Extensive experiments are conducted on the COCO, KITTI, and semantic-KITTI
datasets within four detection and segmentation tasks, showing fascinating
accuracy-speed-memory overall performance increase. Its feature of scalability
indicates potential application scenes of training/deploying larger models on
resource-constrained devices.

##  2 Related Work

###  2.1 Computer Vision Perception Tasks

The goal of perception in Computer Vision is to understand the scene or
feature in images of the real world[21]. Preliminary feature learning
including edge detection, pattern detection, and image classification, are the
first few exploration[52, 16, 49]. In current main-steam perception tasks,
object detection models design detectors that post a yes/no question regarding
the presence of an object class[76, 75, 81, 1, 112, 53]. Another task is
segmentation, whose models extract the geometric description of an object by
grouping pixels/point cloud[32, 3, 46, 59, 117, 50]. These tasks cannot be
done without various of sensors, such as cameras, LiDAR, RADAR, etc. Early
models stick to 2D representations[61, 76, 32]. With the popularization of 3D
sensors like LiDAR as well as the development of GPU, many models could
include depth information and present the 3D output[74, 118, 81]. Today
computer vision is well advanced to specific tasks, and could be well-used in
several real application scenarios in robotics and self-driving cars[55, 1,
42]. To broaden its usage into other harder application scenarios, the
accuracy-speed-memory trade-off, as well as the versatility of these models
that could reach high performance in various tasks, should be the following
core issues to investigate.

###  2.2 Efficiency of Deep Learning Models

Typical technologies in efficient deep learning framework include network
pruning[30, 34, 66], quantization[29], efficient model architecture design[41,
68, 89, 62, 63], efficient optimizers[18, 19, 84, 24], and training
techniques[38, 4, 6]. Besides manual designs, recent works also use AutoML
techniques[121, 89] to automatically design[5], prune[33] and quantize[99]
neural networks. The accuracy-speed-memory trade-off plays an important role
in evaluating these models.

###  2.3 Versatility of Deep Learning Models

Previous approaches usually focus on a specific perception task[76, 32, 74,
81]. They rely heavily on their task-driven or task-centered measures, such as
designing a data augmentation strategy[55, 119, 97, 58], using different
backbone learning layers for different tasks[7, 119, 72]. Some methods could
be well-adopted in two or three tasks, such as 2D object detection and
instance segmentation[31, 100, 63, 67], 3D object detection and tracking[111,
118, 11], 3D object detection and point cloud segmentation[65, 50, 47], These
measures, while performing excellently on their focusing tasks, may also not
be generalized for others. As shown in Tab. 1, we are the first versatile
framework that could provide stable and extensive performance increase under
all these four perception tasks.

##  3 Proposed Method

#### Framework Overview

Figure 1 shows the overall architecture of the proposed EffiPerception. We
focus on exploring the most common parts to increase the overall perception
performance, to make it as general as possible. The EffiPerception model
consists of three core parts: the efficient feature extraction, the efficient
layers, and the EffiOptim. It is a universal efficient framework for various
perception tasks: 3D object detection, 3D point cloud segmentation, 2D object
detection, and 2D instance segmentation. This framework accepts multi-modal
inputs for 3D tasks, which include both RGB images and point clouds. For input
images, they are defined as $\mathcal{I}\in\mathbb{R}^{H_{I}\times W_{I}\times
3}$, where $H_{I}$ and $W_{I}$ denote the image height and width dimensions,
respectively. Meanwhile, the input point cloud is represented as a set of
$3\mathrm{D}$ points $\mathcal{P}\in\mathbb{R}^{H_{P}\times W_{P}\times
D_{P}}$ in the $H_{P}\cdot W_{P}\cdot D_{P}$ 3D space, where each point is a
vector of its $(x,y,z)$-coordinate.

Based on the specific perception task, this framework could use its
corresponding feature transfer paths. For example, if the task is the Lidar-
Only 3D object detection, then the input is only for the point cloud. In this
way, only feature transfer paths relative to point cloud input, as well as the
3D object detection head, are activated. The possible optimal feature transfer
paths are indicated by dotted lines.

Not only for various input modalities and perception tasks, the framework is
also compatible with a great number of existing methods. Using their same 3D
or 2D feature extraction backbone (and neck if exists), as well as their
perception head and loss function, we add the additional efficient layers and
common feature extraction techniques under the efficient optimizer, EffiOpim.
These relatively light modifications, however, could show excellent
performance increase, no matter for inference speed, or the training memory
usage. In the following part, we will describe the three core parts of
EffiPerception in detail.

###  3.1 Efficient Feature Extractors

Figure 2: The Efficient Feature Extraction framework

As shown in Figure 1, given raw images $\mathbf{I}\in\mathbb{R}^{H_{I}\times
W_{I}\times 3}$ and/or 3D points $\mathcal{P}\in\mathbb{R}^{H_{P}\times
W_{P}\times D_{P}}$, we use a separate feature extractor to encode them.

#### Image Feature Extraction.

For image features, we partition images into mini-patches before feeding them
into the 2D backbone. The inherent dense feature of images often poses
challenges related to memory consumption and computational efficiency. To
mitigate these challenges, a promising approach involves partitioning the
input image into smaller, non-overlapping patches before feeding it into the
feature extractor[20, 63, 105]. This strategy offers several advantages.
Firstly, it allows for memory usage’s significant reduction, as the
computational and memory requirements scale with the input size. Secondly,
patch partitioning enhances parallelism, enabling more efficient processing on
modern hardware such as GPUs. Additionally, by decomposing the image into
smaller, manageable regions, it facilitates the extraction of local context
and intricate details.

Formally, by setting each patch to have the side length $v_{cH}$ and $v_{cW}$
respectively, it partitions each image $\mathbf{I}\in\mathbb{R}^{H_{I}\times
W_{I}\times 3}$ image into $N_{c}$ 2D patches
$\mathbf{X_{I}}\in\mathbb{R}^{N_{c}\times\left(v_{cH}\cdot v_{cW}\cdot
3\right)}$, where $N_{c}=\frac{H_{I}}{v_{cH}}\frac{W_{I}}{v_{cW}}$ denotes the
patch number.

Such partitioned image patches will be flattened into one-dimensional
$v_{cH}\times v_{cW}\times 3$ features, then feed into a Multi-Layer
Perceptron (MLP) to reach in total $N_{c}$ encoded features. For each encoded
features $i$, we have

$\mathbf{x}_{c}^{i}=\operatorname{MLP}\left(\mathbf{x}_{I}^{i}\right).$ (1)

The MLP’s parameters are shared by all patches so that they are encoded in the
same way. Each encoded feature $\mathbf{x}_{c}^{i}$ has the same vector size
$D_{c}$, which is also the output size of the MLP.

#### Point Cloud Feature Extraction.

Compared to 2D features, 3D point cloud features contain an additional
dimension, making learning more challenging. Inspired by[118, 81, 104],
voxelization is adopted before feeding into the 3D backbone. The inherent
sparsity of point clouds brings lots of useless computation and resource
occupation. Voxelization transforms the discrete and scattered point cloud
data into a structured grid representation, where each voxel represents a
fixed 3D volume within the space. It could significantly reduce memory usage
and enhance parallelism. Furthermore, by aggregating points within each voxel,
it promotes local context understanding, facilitating feature extraction
processes.

To do it, we divide the raw point cloud in the whole 3D space into little
cubic, each with side length $P_{l}$. Those empty cubic are removed to cut
down the computation burden. Setting each cubic has the side length $v_{lH}$,
$v_{lW}$ and $v_{lD}$ respectivel, it transforms the whole
$\mathcal{P}\in\mathbb{R}^{H_{P}\times W_{P}\times D_{P}}$ point cloud space
into $N_{l}$ 3D cubics
$\mathbf{X_{P}}\in\mathbb{R}^{N_{l}\times\left(v_{lH}\cdot v_{lW}\cdot
v_{lD}\right)}$, where:
$N_{l}=\frac{H_{P}}{v_{lH}}\frac{W_{P}}{v_{lW}}\frac{D_{P}}{v_{lD}}$ denotes
the cubic number. Furthermore, we conduct random sampling for those cubic
having large point cloud numbers, to a fixed smaller number, T. This sampling
strategy is not only for computational savings but to decrease the imbalance
of points between the cubic, which reduces the sampling bias. For
$\mathbf{X_{P}}^{\prime}=\left\\{\mathbf{x_{P}^{i}}=\left[x_{i},y_{i},z_{i}\right]^{T}\in\mathbb{R}^{3}\right\\}_{i=1\ldots
t}$ as a non-empty cubic containing $t\leq T$ LiDAR points, where
$\mathbf{x_{P}^{i}}$ contains XYZ coordinates for the $i$-th point. We also
compute the local mean as the centroid of all the points in
$\mathbf{X_{P}}^{\prime}$, denoted as $\left(c_{x},c_{y},c_{z}\right)$. Then
we augment each point $\mathbf{x_{P}^{i}}$ with the relative offset w.r.t. the
centroid to get input feature set
$\mathbf{X_{Pin}}=\left\\{\hat{\mathbf{x_{P}^{i}}}=\left[x_{i},y_{i},z_{i},x_{i}-c_{x},y_{i}-c_{y},z_{i}-c_{z}\right]^{T}\in\mathbb{R}^{6}\right\\}_{i=1\ldots
t}$. Each cubic will then be flattened into one-dimensional $v_{lH}\times
v_{lW}\times v_{lD}$ features, and feed into an MLP. For each encoded features
$i$, we have:

$\mathbf{x}_{l}^{i^{\prime}}=\operatorname{MLP}(\hat{\mathbf{x_{P}^{i}}}).$
(2)

After voxelization and/or patching, learned features will be transferred into
the corresponding backbones. Their output feature maps will be further
utilized in the efficient layers. If fusion exists, its fused feature will be
treated as 3D feature map. Note that the operation similarity makes the fusion
more smooth.

###  3.2 Efficient Layers

Figure 3: The Efficient Layers

Efficient Layers are plug-in plug-out layers that further process the feature
representation, aggregating core learned information while pruning noisy
proposals. We explore the most common approaches that could be well-compatible
for all four perception tasks.

We first analyze the core difference between 2D and 3D perception tasks. In 2D
image space, occlusion, spatial layout ambiguity, and Scale[114] are three
main problems[116, 94]. We do need some operations to aggregate spatial
information. In 3D point cloud space, while it is also necessary to accumulate
valid proposals, object features could become hugely separable. Directly
adopting the aggregation from the point cloud not only triggers dimension
mismatch issues but also produces unsatisfied results. In addition, raw 3D
feature representations are much more complex than 2D’s, so a well-designed
intermediate procedure is urgently required. Therefore, we designed Sparse
Down-sampling layers for 3D input feature map. These layers help preliminary
aggregate and down-sample raw 3D feature representations from the backbone.
These processed features then feed into the Global Spatial Aggregation(GSA)
layers. In these layers, features from 3D and 2D with be smoothly aggregated.
The following will introduce the two groups of layers one by one.

#### Sparse Down-Sampling for 3D features.

Shown in the upper part in Figure 3, these layers are composed of several
sparse convolution and Inverse sparse convolution layers. Since firstly
introduced[27] and applied in 3D perception tasks[107], many previous methods
have shown that sparse convolution operations could speed up traditional
convolutional operations while using smaller memory cost[85, 95, 22, 23]. Our
design follows the down-sampling, aggregation, and down-sampling routing. 3D
input features will first do the down-sampling by getting through the sparse
convolutional layer with stride 2. After two more sparse convolutional layers
with stride 1, an inverse sparse convolutional layer comes to restore its
feature dimension, which could be aggregated with the raw input feature. One
further sparse convolutional layer with stride 1 and another downsampling
recipe follow the concatenation. In this way, 3D feature space could smoothly
down-sample its space while keeping more dense-like important information.

Such a down-sampled 3D feature map could be more inherent-nature consistent
with the 2D feature space, especially after conducting the sparse to dense
operation in the Global Spatial Aggregation (GSA), which will be introduced in
the next part. It is worth indicating that the Sparse Down-sampling is also
important even if there is no need to combine it with the 2D features. Under
pure-3D perception tasks, the Sparse Down-sampling operation follows the
nature of 3D features, filtering in more valuable features, making them more
aggregated, and gradually reducing the space size.

#### Global Spatial Aggregation.

Besides using sparse convolutions, another common approach to handling 3D
perception tasks is to first compress the 3D feature map into a 2D feature
map, either as Bird-Eye-View(BEV)[51, 64, 1, 13] or Range View(RV)[9, 86, 70],
then to use a 2D backbone or neck to extract the proposals. For example,
PointPillar[51] adopts a top-down backbone with downsampling, upsampling, and
concatenation operations to exploit its feature mining. Inspired by these
design[110], we develop Global Spatial Aggregation (GSA). It could extract
global contextual features from 3D sparse tensors by projecting it to
$2\mathrm{D}$ dense $\mathrm{BEV}$ feature map and then converting the dense
feature map back after applying 2D multi-scale feature extractor.

As illustrated in the lower part of Figure 3, given the 3D sparse tensor
output by the encoder, we first project it into a dense $\mathrm{BEV}$ feature
map $\mathcal{H}^{\text{sparse }}\in\mathbb{R}^{C\times
M^{\prime}}\rightarrow\mathcal{H}^{\text{dense
}}\in\mathbb{R}^{C\times\frac{D}{dz}\times\frac{H}{dx}\times\frac{W}{dy}}$,
where $d$ is the down-sampling ratio and $M^{\prime}$ is the number of valid
voxels in the last scale. After that, we concatenate the features across the
height dimension together to form a $2\mathrm{D}BEV$ feature map
$\mathcal{H}^{\text{bev
}}\in\mathbb{R}^{\left(C*\frac{D}{d_{z}}\right)\times\frac{H}{d_{x}}\times\frac{W}{d_{y}}}$.
Then, we use a 2D multi-scale CNN to further extract contextual information.
Lastly, we reshape the encoded BEV feature representation to the dense voxel
map, then transform it back to the sparse voxel feature following the reverse
dense-to-sparse conversion. To make the module be more robust, the transformed
sparse voxel feature will be concatenated with the sparse feature before the
sparse-to-dense operation.

Benefiting from GSA, our architecture could significantly enlarge the
receptive field, which plays an important role in semantic segmentation. These
operations not only are greatly effective for extracting 2D features, but are
also natural-friendly for the compressed 3D features. In addition, both 2D
tasks and 3D tasks could gain well-structured information from each other.

The features from generated 2D and 3D feature maps will be finally transferred
into the corresponding perception head, according to the task. The 3D object
detection and 3D point cloud segmentation head are followed from the 3D
feature map, while from the 2D feature map, 2D instance segmentation and 2D
object detection head are performed.

###  3.3 The EffiOptim

Besides these efficient layers and feature extractors, we also adopt an
efficient optimizer, named EffiOptim, to further cut down training memory
usage as well as speed up inference speed without sacrificing performance.

The development of efficient optimization techniques[19, 84] for deep learning
models has been a critical area of research, particularly in the context of
memory usage and computational speed. In this regard, the emergence of 8-bit
optimizers[18] has garnered significant attention due to their potential to
reduce memory footprint and accelerate training and inference processes while
maintaining performance levels comparable to traditional 32-bit optimizers.
The EffiOptim is a 8-bit optimizers that work on perception tasks. This
approach has the potential to further improve the efficiency of the
EffiPerception framework and enable us to train and deploy larger models on
resource-constrained devices.

Model | $\text{mAP}_{\text{BEV}}(\text{IoU}=0.7)$ | $\text{mAP}_{\text{3D}}(\mathrm{IoU}=0.7)$ | Memory | Runtime
---|---|---|---|---
Easy | Med | Hard | Easy | Med | Hard | $(\mathrm{M})$ | $(\mathrm{ms})$
PV-RCNN[81] | 86.2 | 84.8 | 78.7 | N/A | N/A | N/A | 16046 | 59.2
PointRCNN[82] | 87.9 | 90.2 | 85.5 | 88.6 | 88.1 | 77.4 | 13289 | 100.1
CT3D[80] | 88.5 | 86.1 | 79.0 | 90.5 | 87.1 | 79.0 | N/A | N/A
3D-SSD[108] | 89.7 | 89.5 | 78.7 | 89.4 | 87.5 | 78.4 | 6032 | 38.9
Part-$A^{2}$-free[83] | 88.0 | 90.2 | 85.9 | 89.0 | 88.5 | 78.4 | 12839 | 80.8
TED[102] | 91.6 | 85.3 | 80.7 | 91.5 | 85.7 | 82.2 | 8820 | 105.5
LoGoNet[53] | 91.8 | 85.0 | 80.7 | 91.8 | 85.0 | 82.4 | 15023 | 96.3
EffiPerception | 89.9 | 90.5 | 85.6 | 91.5 | 88.2 | 82.4 | 7293 | 42.0
Table 2: Comparison of attained validation mAP (in %) on KITTI with $\text{IoU}=0.7$. Method | mIoU |  car |  bicycle |  motorcycle |  truck |  other-vehicle |  person |  bicyclist |  motorcyclist |  road |  parking |  sidewalk |  other-ground |  building |  fence |  vegetation |  trunk |  terrain |  pole |  traffic-sign
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
PointNet++[74] | 20.7 |  72 |  42 |  19 |  6 |  62 |  54 |  1 |  2 |  0 |  0 |  47 |  14 |  30 |  1 |  1 |  0 |  17 |  6 |  9
SqueezeSegV3[106] | 56.4 |  92 |  75 |  63 |  26 |  89 |  93 |  30 |  39 |  37 |  33 |  82 |  59 |  65 |  46 |  46 |  20 |  59 |  50 |  59
Cylinder3D[117] | 69.2 |  92 |  77 |  65 |  32 |  91 |  97 |  51 |  68 |  64 |  59 |  86 |  73 |  70 |  74 |  69 |  48 |  67 |  62 |  66
SPVNAS[91] | 67.8 |  90 |  75 |  68 |  22 |  92 |  97 |  57 |  51 |  50 |  58 |  86 |  73 |  71 |  67 |  67 |  50 |  67 |  64 |  67
PVKD[39] | 71.3 |  92 |  71 |  78 |  43 |  92 |  97 |  68 |  69 |  54 |  60 |  87 |  74 |  72 |  75 |  74 |  51 |  69 |  67 |  66
SphericalFormer[50] | 75.0 |  92 |  78 |  70 |  41 |  94 |  98 |  60 |  70 |  71 |  68 |  87 |  75 |  72 |  79 |  80 |  75 |  73 |  65 |  73
EffiPerception | 75.8 |  91 |  85 |  71 |  48 |  93 |  98 |  58 |  69 |  73 |  66 |  88 |  75 |  69 |  83 |  78 |  70 |  71 |  68 |  75
Table 3: Comparison of IoU (in %) on Semantic KITTI test set Model | box $\mathrm{AP}$ | $\mathrm{AP}_{50}$ | $\mathrm{AP}_{75}$ | $\mathrm{AP}_{s}$ | $\mathrm{AP}_{m}$ | $\mathrm{AP}_{l}$ | Memory$(\mathrm{M})$ | Runtime$(\mathrm{ms})$
---|---|---|---|---|---|---|---|---
MobileNetV2[78] | 28.3 | 46.7 | 29.3 | 14.8 | 30.7 | 38.1 | 11283 | 62.3
MobileNetV3[40] | 29.9 | 49.3 | 30.8 | 14.9 | 33.3 | 41.1 | 10028 | 54.2
SPOS[28] | 30.7 | 49.8 | 32.2 | 15.4 | 33.9 | 41.6 | 13827 | 102.9
MNASNet-A2[88] | 30.5 | 50.2 | 32.0 | 16.6 | 34.1 | 41.1 | 13726 | 128.1
FairNAS-C[14] | 31.2 | 50.8 | 32.7 | 16.3 | 34.4 | 42.3 | 22192 | 101.3
MixNet-M[113] | 31.3 | 51.7 | 32.4 | 17.0 | 35.0 | 41.9 | 18279 | 77.2
EfficientViT-M4[62] | 32.7 | 52.2 | 34.1 | 17.6 | 35.3 | 46.0 | 11938 | 52.3
EfficientDet-D3[90] | 46.8 | 65.9 | 51.2 | N/A | N/A | N/A | 13827 | 40.2
ViT-Adapter-B[12] | 47.0 | 68.2 | 51.4 | N/A | N/A | N/A | 16927 | 76.7
YOLOv7[96] | 51.2 | 69.7 | 55.5 | 35.2 | 56.0 | 66.7 | 12023 | 45.4
EffiPerception | 52.3 | 69.3 | 54.6 | 35.3 | 57.3 | 64.0 | 9946 | 36.2
Table 4: Comparison of attained validation AP (in %) on COCO val2017 Model | mask $\mathrm{AP}$ | $\mathrm{AP}_{50}$ | $\mathrm{AP}_{75}$ | $\mathrm{AP}_{s}$ | $\mathrm{AP}_{m}$ | $\mathrm{AP}_{l}$ | Memory$(\mathrm{M})$ | Runtime$(\mathrm{ms})$
---|---|---|---|---|---|---|---|---
FCIS[54] | 29.5 | 51.5 | 30.2 | 8.0 | 31.0 | 49.7 | 14192 | 151.5
YOLACT-550[3] | 29.8 | 48.5 | 31.2 | 9.9 | 31.3 | 47.7 | 14872 | 42.1
RetinaMask[25] | 34.7 | 55.4 | 36.9 | 14.3 | 36.7 | 50.5 | 19028 | 166.7
Mask R-CNN[32] | 35.7 | 58.0 | 37.8 | 15.5 | 38.1 | 52.4 | 20397 | 116.3
PA-Net[60] | 36.6 | 58.0 | 39.3 | 16.3 | 38.1 | 53.1 | 18239 | 112.8
MS R-CNN[43] | 38.3 | 58.8 | 41.5 | 17.8 | 40.4 | 54.4 | 17029 | 86.3
BoxInst[92] | 35.0 | 59.3 | 35.6 | 17.1 | 37.2 | 48.9 | 19827 | 93.2
SOLOv2[101] | 39.7 | 60.7 | 42.9 | 17.3 | 42.9 | 57.4 | 13827 | 63.2
ViT-Adapter-B[12] | 41.8 | 65.1 | 44.9 | N/A | N/A | N/A | 15729 | 127.4
EffiPerception | 42.1 | 64.6 | 45.8 | 20.2 | 45.2 | 59.6 | 10382 | 53.8
Table 5: Comparison of attained validation AP (in %) on COCO val2017

##  4 Experiments

###  4.1 Experimental Setup

#### Datasets.

To demonstrate the effectiveness of EffiPerception, we present results on the
KITTI [26], Semantic KITTI[2], and COCO[57].

KITTI is a 3D object detection benchmark. It is divided into 7,481 training
samples and 7,518 testing samples. The training samples are commonly divided
into a training set (3,712 samples) and a validation set (3,769 samples)
following [8], which we adopt. Before being fed into the models, the dataset
is augmented by random 3D flips, random rotation, scale, and translation, and
shuffled the point data. We use it to conduct 3D Object Detection experiments.
Semantic KITTI is a point-level re-annotation of the LiDAR part of the KITTI
dataset. It has a total of 43551 scans sampled from 22 sequences collected in
different cities in Germany. It has 104452 points per scan on average and each
scan is collected by a single Velodyne HDL-64E laser scanner. We follow
SemanticKITTIs subset split protocol and use ten sequences for training, one
for validation, and the rest of them for testing. Our tests with 3D point
cloud segmentation are carried out with it. COCO 2017 is a 2D image dataset
with 118K training images. We carry out 2D object detection and instance
segmentation using it.

#### Implementation Details.

We implement EffiPerception using the MMDetection3D library [71] and
Detectron2[103]. The model is optimized using the Adam [45] optimizer with a
learning rate of 0.0001, optimizer momentum
$(\beta_{1},\beta_{2})=(0.9,0.999)$, and a dropout rate of 0.3. We train the
model on an NVIDIA RTX A6000 GPU for 50 epochs and validate after each epoch.
On KITTI, we use $[2,46.8]\times[-30.08,30.08]\times[-3,1]$ for the range and
$[0.16,0.16,0.16]$ for the voxel size for the $x,y$, and $z$ axes,
respectively. On Semanitic KITTI, we use
$[-51.2,51.2]\times[-51.2,51.2]\times[-4,2.4]$ for the range and $0.05$ for
the voxel size. The window size is set to
$\left[120m,2^{\circ},2^{\circ}\right]$ for $(r,\theta,\phi)$. For any
possible direct set prediction, we set the number of proposal boxes to 300.

###  4.2 Comparison with State-of-the-art models

We conduct experiments for 3D object detection, 3D point cloud segmentation,
2D object detection, and 2D instance segmentation. Tabs. 2, 3, 4 and 5 shows
their results correspondingly. In the 3D object detection task, we conduct
experiments on the KITTI 3D object detection benchmark. All models are trained
with batch size of 2, and their training memory and inference running time are
also reported. Our model is conducted with the SST[22] model. Following the
standard KITTI evaluation protocol ($\text{IoU}=0.7$) for measuring the
detection performance, it shows the mean average precision (mAP) scores for
the EffiPerception method compared to the state-of-the-art methods on the
KITTI validation set using 3D and bird’s eye view (BEV) evaluation. We bold-
face the two best-performing models for each task. As indicated in Tab. 2, our
approach shows significant overall performance improvements compared to the
baselines, not only on the accuracy but also for the training memory usage and
inference runtime.

The 3D point cloud segmentation experiments are conducted in the semantic-
KITTI benchmark whose test set results are shown in Tab. 3. All models are
trained with batch size of 2. The LaserMix[48] model is applied to perform our
model. Consistently outperforms others by a large margin, our method yields
75.8% mIoU, a new state-of-the-art result. Thanks to the capability of
directly aggregating long-range information, our method significantly
outperforms the models based on classic sparse convolution[117, 91].

Besides, Tabs. 4 and 5 shows the results of 2D object detection and instance
segmentation experiments conducted on the COCO dataset. All models are trained
with 16 batch size and their training memory and inference running time are
also reported. Our model adopts Sparse-RCNN[87] and EfficientViT[62]
correspondingly as the backbone of these two tasks. In the detection task,
EffiPerception exhibits better performance in accuracy (52.3 AP) and shorter
inference time (36.2ms), with shorter memory usage(9946M). For the
segmentation task, EffiPerception also outperforms well-established detectors,
such as Mask-RCNN[32] and YOLACT[3], by a large margin. Its shorter inference
time and lower memory usage reveal a wide range of potential usages.

Figure 4: Experiment Results

###  4.3 EffiPerception is a generic perception method

SDS | GSA | $\text{mAP}_{\text{BEV}}(\text{IoU}=0.7)$
---|---|---
Easy | Med | Hard
✗ | ✗ | 88.5 | 85.2 | 78.4
✓ | ✗ | 89.4 | 86.5 | 80.5
✗ | ✓ | 89.8 | 87.2 | 80.3
✓ | ✓ | $\mathbf{91.5}$ | $\mathbf{88.2}$ | $\mathbf{82.4}$
Table 6: Ablation studies on Sparse Down-Sampling (SDS) and Global Spatial
Aggregation (GSA).

Next, we examine how generic our method is by plugging it into prevalent
perception frameworks. Among these four perception tasks, we conduct eleven
pairs of comparisons in total, each of which is between a single-modal method
and its EffiPerception-equipped counterpart. To further clearly show the
performance gain from the neural network model, we add a middle column to
report the modal performance combined only with the EffiOptim (as ‘w/EffiOptim
native’).

Those ten models’ performances are shown in Fig. 4, where each column figure
group corresponds to one task. Owning higher accuracy, smaller memory usage,
and lower inference time, EffiPerception indicates a consistent improvement
for all single-modal baselines among all the four down-steaming tasks. These
results indicate EffiPerception is generic and can be potentially applied to
various perception frameworks.

###  4.4 Impact of Sparse Down-Sampling and Global Spatial Aggregation

In this section, we show the effectiveness of both proposed components, Sparse
Down-Sampling and Global Spatial Aggregation. Tab. 6 shows the experiment
result of EffiPerception in 3D object detection task. We observe that both
components can improve the performance over the single-modal baseline. In
particular, the gain by Global Spatial Aggregation is more prominent. For
example, without Global Spatial Aggregation, the performance of $mAP_{BEV}$
drops drastically from 88.2 to 86.5 in the Medium level.

###  4.5 EffiPerception is more robust

Corruptions | Modal | $\text{mAP}_{\text{BEV}}(\text{IoU}=0.7)$
---|---|---
Easy | Med | Hard
No Corruption | TED[102] | 91.5 | 85.7 | 82.2
Laser Noise | TED | 87.2 | 83.6 | 77.8(-4.4)
No Corruption | LoGoNet[53] | 91.8 | 85.0 | 82.4
Laser Noise | LoGoNet | 87.4 | 83.5 | 78.6 (-3.8)
No Corruption | EffiPerception | 91.5 | 88.2 | 82.4
Laser Noise | EffiPerception | 90.5 | 87.8 | 81.6(-0.8)
Pixel Noise | EffiPerception | 90.4 | 87.5 | 80.9(-1.5)
Pixel+Laser Noise | EffiPerception | 89.4 | 86.8 | 80.0(-2.4)
Table 7: Model robustness against input corruptions.

Robustness is a pivotal metric as it ensures that the model can maintain its
performance and provide reliable predictions even when faced with noisy,
incomplete, or adversarial data, which is crucial for applications where
safety, security, and accuracy are of paramount importance.

In this subsection, we examine the model robustness against corrupted
input[37]. We test the model robustness on the validation set against common
corruptions, Laser Noise (randomly adding noise to lidar reflections), and
Pixel Noise (randomly adding noise to camera pixels). As shown in Tab. 7, with
the presence of corruptions, our EffiPerception models, in general, are much
more robust than their counterparts. The Lase or Pixel Noise corruption can
hardly drag down the performance of our EffiPerception method (with only 0.8 /
1.5 $\text{mAP}_{\text{BEV}}$ Hard drop). Even when applying both Laser and
Pixel Noise corruptions, the performance drop is still marginal (2.4
$\text{mAP}_{\text{BEV}}$ Hard drop). Meanwhile, the comparing LoGoNet and TED
model drops around 4 APH $\text{mAP}_{\text{BEV}}$ Hard simply applying the
Laser Noise corruption.

##  5 Conclusion

This paper presents EffiPerception, an efficient framework for various
perception tasks. It focuses on learning common patterns to improve the
overall performance under the feature extractor level, learning layers level,
and optimizer level. Compared to well-established models, EffiPerception
achieves great accuracy-speed performance with relatively low memory cost
under several perception tasks: 2D Object Detection, 3D Object Detection, 2D
Instance Segmentation, and 3D Point Cloud Segmentation. Its generic adaptive
nature, extensive task-applicable features, as well as robust learning results
make it promising for potential future usage.

## References

  * [1] Bai, X., Hu, Z., Zhu, X., Huang, Q., Chen, Y., Fu, H., Tai, C.L.: Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1090–1099 (June 2022)
  * [2] Behley, J., Garbade, M., Milioto, A., Quenzel, J., Behnke, S., Stachniss, C., Gall, J.: Semantickitti: A dataset for semantic scene understanding of lidar sequences (2019)
  * [3] Bolya, D., Zhou, C., Xiao, F., Lee, Y.J.: Yolact: Real-time instance segmentation (2019)
  * [4] Cai, H., Gan, C., Lin, J., Han, S.: Network augmentation for tiny deep learning (2022)
  * [5] Cai, H., Gan, C., Wang, T., Zhang, Z., Han, S.: Once-for-all: Train one network and specialize it for efficient deployment (2020)
  * [6] Cai, H., Gan, C., Zhu, L., Han, S.: Tinytl: Reduce activations, not trainable parameters for efficient on-device learning (2021)
  * [7] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers (2020). https://doi.org/10.48550/ARXIV.2005.12872, https://arxiv.org/abs/2005.12872
  * [8] Chen, X., Kundu, K., Zhu, Y., Ma, H., Fidler, S., Urtasun, R.: 3d object proposals using stereo imagery for accurate object class detection (2017)
  * [9] Chen, X., Ma, H., Wan, J., Li, B., Xia, T.: Multi-view 3d object detection network for autonomous driving (2016). https://doi.org/10.48550/ARXIV.1611.07759, https://arxiv.org/abs/1611.07759
  * [10] Chen, Y., Li, Y., Zhang, X., Sun, J., Jia, J.: Focal sparse convolutional networks for 3d object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2022)
  * [11] Chen, Y., Liu, J., Zhang, X., Qi, X., Jia, J.: Voxelnext: Fully sparse voxelnet for 3d object detection and tracking (2023)
  * [12] Chen, Z., Duan, Y., Wang, W., He, J., Lu, T., Dai, J., Qiao, Y.: Vision transformer adapter for dense predictions (2023)
  * [13] Chitta, K., Prakash, A., Jaeger, B., Yu, Z., Renz, K., Geiger, A.: Transfuser: Imitation with transformer-based sensor fusion for autonomous driving (2022). https://doi.org/10.48550/ARXIV.2205.15997, https://arxiv.org/abs/2205.15997
  * [14] Chu, X., Zhang, B., Xu, R.: Fairnas: Rethinking evaluation fairness of weight sharing neural architecture search (2021)
  * [15] Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., Wei, Y.: Deformable convolutional networks (2017)
  * [16] Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05). vol. 1, pp. 886–893 vol. 1 (2005). https://doi.org/10.1109/CVPR.2005.177
  * [17] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. pp. 248–255 (2009). https://doi.org/10.1109/CVPR.2009.5206848
  * [18] Dettmers, T., Lewis, M., Shleifer, S., Zettlemoyer, L.: 8-bit optimizers via block-wise quantization (2022)
  * [19] Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: Qlora: Efficient finetuning of quantized llms (2023)
  * [20] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale (2020). https://doi.org/10.48550/ARXIV.2010.11929, https://arxiv.org/abs/2010.11929
  * [21] Duda, R.O., Hart, P.E.: Pattern classification and scene analysis. In: A Wiley-Interscience publication (1974), https://api.semanticscholar.org/CorpusID:12946615
  * [22] Fan, L., Pang, Z., Zhang, T., Wang, Y.X., Zhao, H., Wang, F., Wang, N., Zhang, Z.: Embracing single stride 3d object detector with sparse transformer (2021)
  * [23] Fan, L., Wang, F., Wang, N., Zhang, Z.: Fully sparse 3d object detection (2022)
  * [24] Fang, J., Zhu, Z., Li, S., Su, H., Yu, Y., Zhou, J., You, Y.: Parallel training of pre-trained models via chunk-based dynamic memory management. IEEE Transactions on Parallel and Distributed Systems 34(1), 304–315 (2022)
  * [25] Fu, C.Y., Shvets, M., Berg, A.C.: Retinamask: Learning to predict masks improves state-of-the-art single-shot detection for free (2019)
  * [26] Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? the KITTI vision benchmark suite. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition. pp. 3354–3361. IEEE (2012)
  * [27] Graham, B., van der Maaten, L.: Submanifold sparse convolutional networks (2017)
  * [28] Guo, Z., Zhang, X., Mu, H., Heng, W., Liu, Z., Wei, Y., Sun, J.: Single path one-shot neural architecture search with uniform sampling (2020)
  * [29] Han, S., Mao, H., Dally, W.J.: Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding (2016)
  * [30] Han, S., Pool, J., Tran, J., Dally, W.J.: Learning both weights and connections for efficient neural networks (2015)
  * [31] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 15979–15988 (2022). https://doi.org/10.1109/CVPR52688.2022.01553
  * [32] He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn (2017). https://doi.org/10.48550/ARXIV.1703.06870, https://arxiv.org/abs/1703.06870
  * [33] He, Y., Lin, J., Liu, Z., Wang, H., Li, L.J., Han, S.: Amc: Automl for model compression and acceleration on mobile devices (2019)
  * [34] He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks (2017)
  * [35] He, Z., Ye, X., Li, Y.: Compact sparse r-cnn: Speeding up sparse r-cnn by reducing iterative detection heads and simplifying feature pyramid network. AIP Advances 13(5) (2023)
  * [36] Hekimoglu, A., Friedrich, P., Zimmer, W., Schmidt, M., Marcos-Ramiro, A., Knoll, A.C.: Multi-task consistency for active learning (2023)
  * [37] Hendrycks, D., Dietterich, T.: Benchmarking neural network robustness to common corruptions and perturbations (2019)
  * [38] Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network (2015)
  * [39] Hou, Y., Zhu, X., Ma, Y., Loy, C.C., Li, Y.: Point-to-voxel knowledge distillation for lidar semantic segmentation (2022)
  * [40] Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., Le, Q.V., Adam, H.: Searching for mobilenetv3 (2019)
  * [41] Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications (2017)
  * [42] Hu, Y., Yang, J., Chen, L., Li, K., Sima, C., Zhu, X., Chai, S., Du, S., Lin, T., Wang, W., Lu, L., Jia, X., Liu, Q., Dai, J., Qiao, Y., Li, H.: Planning-oriented autonomous driving (2023)
  * [43] Huang, Z., Huang, L., Gong, Y., Huang, C., Wang, X.: Mask scoring r-cnn (2019)
  * [44] Karlinsky, L., Shtok, J., Alfassy, A., Lichtenstein, M., Harary, S., Schwartz, E., Doveh, S., Sattigeri, P., Feris, R., Bronstein, A., Giryes, R.: Starnet: towards weakly supervised few-shot object detection (2020). https://doi.org/10.48550/ARXIV.2003.06798, https://arxiv.org/abs/2003.06798
  * [45] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [46] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollár, P., Girshick, R.: Segment anything. arXiv:2304.02643 (2023)
  * [47] Kong, L., Liu, Y., Li, X., Chen, R., Zhang, W., Ren, J., Pan, L., Chen, K., Liu, Z.: Robo3d: Towards robust and reliable 3d perception against corruptions (2023)
  * [48] Kong, L., Ren, J., Pan, L., Liu, Z.: Lasermix for semi-supervised lidar semantic segmentation (2023)
  * [49] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Communications of the ACM 60, 84 – 90 (2012), https://api.semanticscholar.org/CorpusID:195908774
  * [50] Lai, X., Chen, Y., Lu, F., Liu, J., Jia, J.: Spherical transformer for lidar-based 3d recognition (2023)
  * [51] Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang, J., Beijbom, O.: Pointpillars: Fast encoders for object detection from point clouds (2018). https://doi.org/10.48550/ARXIV.1812.05784, https://arxiv.org/abs/1812.05784
  * [52] Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11), 2278–2324 (1998). https://doi.org/10.1109/5.726791
  * [53] Li, X., Ma, T., Hou, Y., Shi, B., Yang, Y., Liu, Y., Wu, X., Chen, Q., Li, Y., Qiao, Y., He, L.: Logonet: Towards accurate 3d object detection with local-to-global cross-modal fusion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
  * [54] Li, Y., Qi, H., Dai, J., Ji, X., Wei, Y.: Fully convolutional instance-aware semantic segmentation (2017)
  * [55] Li, Y., Yu, A.W., Meng, T., Caine, B., Ngiam, J., Peng, D., Shen, J., Wu, B., Lu, Y., Zhou, D., Le, Q.V., Yuille, A., Tan, M.: Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection (2022). https://doi.org/10.48550/ARXIV.2203.08195, https://arxiv.org/abs/2203.08195
  * [56] Liang, M., Yang, B., Chen, Y., Hu, R., Urtasun, R.: Multi-task multi-sensor fusion for 3d object detection (2020)
  * [57] Lin, T.Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C.L., Dollár, P.: Microsoft coco: Common objects in context (2015)
  * [58] Liu, C., Gao, C., Liu, F., Li, P., Meng, D., Gao, X.: Hierarchical supervision and shuffle data augmentation for 3d semi-supervised object detection (2023)
  * [59] Liu, M., Zhou, Y., Qi, C.R., Gong, B., Su, H., Anguelov, D.: Less: Label-efficient semantic segmentation for lidar point clouds (2022)
  * [60] Liu, S., Qi, L., Qin, H., Shi, J., Jia, J.: Path aggregation network for instance segmentation (2018)
  * [61] Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: SSD: Single Shot MultiBox Detector, p. 21–37. Springer International Publishing (2016). https://doi.org/10.1007/978-3-319-46448-0_2, http://dx.doi.org/10.1007/978-3-319-46448-0_2
  * [62] Liu, X., Peng, H., Zheng, N., Yang, Y., Hu, H., Yuan, Y.: Efficientvit: Memory efficient vision transformer with cascaded group attention. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
  * [63] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows (2021). https://doi.org/10.48550/ARXIV.2103.14030, https://arxiv.org/abs/2103.14030
  * [64] Liu, Z., Tang, H., Amini, A., Yang, X., Mao, H., Rus, D., Han, S.: Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation (2022)
  * [65] Liu, Z., Tang, H., Lin, Y., Han, S.: Point-voxel cnn for efficient 3d deep learning (2019)
  * [66] Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., Zhang, C.: Learning efficient convolutional networks through network slimming (2017)
  * [67] Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s (2022)
  * [68] Ma, N., Zhang, X., Zheng, H.T., Sun, J.: Shufflenet v2: Practical guidelines for efficient cnn architecture design (2018)
  * [69] Mao, J., Shi, S., Wang, X., Li, H.: 3d object detection for autonomous driving: A comprehensive survey (2023)
  * [70] Meyer, G.P., Laddha, A., Kee, E., Vallespi-Gonzalez, C., Wellington, C.K.: Lasernet: An efficient probabilistic 3d object detector for autonomous driving (2019). https://doi.org/10.48550/ARXIV.1903.08701, https://arxiv.org/abs/1903.08701
  * [71] MMDetection3D, C.: MMDetection3D: OpenMMLab next-generation platform for general 3D object detection. https://github.com/open-mmlab/mmdetection3d (2020)
  * [72] Pan, X., Xia, Z., Song, S., Li, L.E., Huang, G.: 3d object detection with pointformer (2021)
  * [73] Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classification and segmentation (2016). https://doi.org/10.48550/ARXIV.1612.00593, https://arxiv.org/abs/1612.00593
  * [74] Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: Deep hierarchical feature learning on point sets in a metric space (2017). https://doi.org/10.48550/ARXIV.1706.02413, https://arxiv.org/abs/1706.02413
  * [75] Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection (2015). https://doi.org/10.48550/ARXIV.1506.02640, https://arxiv.org/abs/1506.02640
  * [76] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks (2015). https://doi.org/10.48550/ARXIV.1506.01497, https://arxiv.org/abs/1506.01497
  * [77] Salscheider, N.O.: Simultaneous object detection and semantic segmentation (2020)
  * [78] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: Inverted residuals and linear bottlenecks (2019)
  * [79] Shen, X., Stamos, I.: 3d object detection and instance segmentation from 3d range and 2d color images. Sensors (Basel, Switzerland) 21 (2021), https://api.semanticscholar.org/CorpusID:231900097
  * [80] Sheng, H., Cai, S., Liu, Y., Deng, B., Huang, J., Hua, X.S., Zhao, M.J.: Improving 3d object detection with channel-wise transformer (2021)
  * [81] Shi, S., Guo, C., Jiang, L., Wang, Z., Shi, J., Wang, X., Li, H.: PV-RCNN: Point-voxel feature set abstraction for 3d object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10529–10538 (2020)
  * [82] Shi, S., Wang, X., Li, H.: Pointrcnn: 3d object proposal generation and detection from point cloud (2018). https://doi.org/10.48550/ARXIV.1812.04244, https://arxiv.org/abs/1812.04244
  * [83] Shi, S., Wang, Z., Shi, J., Wang, X., Li, H.: From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network (2020)
  * [84] Solodskikh, K., Kurbanov, A., Aydarkhanov, R., Zhelavskaya, I., Parfenov, Y., Song, D., Lefkimmiatis, S.: Integral neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 16113–16122 (June 2023)
  * [85] Sun, P., Tan, M., Wang, W., Liu, C., Xia, F., Leng, Z., Anguelov, D.: Swformer: Sparse window transformer for 3d object detection in point clouds (2022)
  * [86] Sun, P., Wang, W., Chai, Y., Elsayed, G., Bewley, A., Zhang, X., Sminchisescu, C., Anguelov, D.: Rsn: Range sparse net for efficient, accurate lidar 3d object detection (2021)
  * [87] Sun, P., Zhang, R., Jiang, Y., Kong, T., Xu, C., Zhan, W., Tomizuka, M., Li, L., Yuan, Z., Wang, C., Luo, P.: Sparse R-CNN: end-to-end object detection with learnable proposals. CoRR abs/2011.12450 (2020), https://arxiv.org/abs/2011.12450
  * [88] Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., Le, Q.V.: Mnasnet: Platform-aware neural architecture search for mobile (2019)
  * [89] Tan, M., Le, Q.V.: Efficientnet: Rethinking model scaling for convolutional neural networks (2020)
  * [90] Tan, M., Pang, R., Le, Q.V.: Efficientdet: Scalable and efficient object detection (2020)
  * [91] Tang, H., Liu, Z., Zhao, S., Lin, Y., Lin, J., Wang, H., Han, S.: Searching efficient 3d architectures with sparse point-voxel convolution (2020)
  * [92] Tian, Z., Shen, C., Wang, X., Chen, H.: Boxinst: High-performance instance segmentation with box annotations (2020)
  * [93] Unal, O., Gool, L.V., Dai, D.: Improving point cloud semantic segmentation by learning 3d object detection (2020)
  * [94] Voulodimos, A., Doulamis, N., Doulamis, A., Protopapadakis, E., Andina, D.: Deep learning for computer vision: A brief review. Intell. Neuroscience 2018 (jan 2018). https://doi.org/10.1155/2018/7068349, https://doi.org/10.1155/2018/7068349
  * [95] Wang, B., Zhu, M., Lu, Y., Wang, J., Gao, W., Wei, H.: Real-time 3d object detection from point cloud through foreground segmentation. IEEE Access 9, 84886–84898 (2021). https://doi.org/10.1109/ACCESS.2021.3087179
  * [96] Wang, C.Y., Bochkovskiy, A., Liao, H.Y.M.: Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors (2022). https://doi.org/10.48550/ARXIV.2207.02696, https://arxiv.org/abs/2207.02696
  * [97] Wang, C., Ma, C., Zhu, M., Yang, X.: Pointaugmenting: Cross-modal augmentation for 3d object detection. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11789–11798 (2021). https://doi.org/10.1109/CVPR46437.2021.01162
  * [98] Wang, J., Chakraborty, R., Yu, S.X.: Spatial transformer for 3d point clouds. IEEE Transactions on Pattern Analysis and Machine Intelligence p. 1–1 (2021). https://doi.org/10.1109/tpami.2021.3070341, http://dx.doi.org/10.1109/TPAMI.2021.3070341
  * [99] Wang, T., Wang, K., Cai, H., Lin, J., Liu, Z., Han, S.: Apq: Joint search for network architecture, pruning and quantization policy (2020)
  * [100] Wang, W., Xie, E., Li, X., Fan, D.P., Song, K., Liang, D., Lu, T., Luo, P., Shao, L.: Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media 8(3), 415–424 (Mar 2022). https://doi.org/10.1007/s41095-022-0274-8, http://dx.doi.org/10.1007/s41095-022-0274-8
  * [101] Wang, X., Zhang, R., Kong, T., Li, L., Shen, C.: Solov2: Dynamic and fast instance segmentation (2020)
  * [102] Wu, H., Wen, C., Li, W., Yang, R., Wang, C.: Transformation-equivariant 3d object detection for autonomous driving. In: AAAI (2023)
  * [103] Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R.: Detectron2. https://github.com/facebookresearch/detectron2 (2019)
  * [104] Xiang, X., Dräger, S., Zhang, J.: 3diffusiondet: Diffusion model for 3d object detection with robust lidar-camera fusion (2023)
  * [105] Xiang, X., Zhang, J.: Fusionvit: Hierarchical 3d object detection via lidar-camera vision transformer fusion (2023)
  * [106] Xu, C., Wu, B., Wang, Z., Zhan, W., Vajda, P., Keutzer, K., Tomizuka, M.: Squeezesegv3: Spatially-adaptive convolution for efficient point-cloud segmentation (2021)
  * [107] Yan, Y., Mao, Y., Li, B.: Second: Sparsely embedded convolutional detection. Sensors (Basel, Switzerland) 18 (2018)
  * [108] Yang, Z., Sun, Y., Liu, S., Jia, J.: 3DSSD: Point-based 3d single stage object detector. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11040–11048 (2020)
  * [109] Yang, Z., Sun, Y., Liu, S., Shen, X., Jia, J.: Ipod: Intensive point-based object detector for point cloud (2018)
  * [110] Ye, D., Zhou, Z., Chen, W., Xie, Y., Wang, Y., Wang, P., Foroosh, H.: Lidarmultinet: Towards a unified multi-task network for lidar perception (2023)
  * [111] Yin, T., Zhou, X., Krähenbühl, P.: Center-based 3d object detection and tracking (2021)
  * [112] Zeng, Y., Zhang, D., Wang, C., Miao, Z., Liu, T., Zhan, X., Hao, D., Ma, C.: Lift: Learning 4d lidar image fusion transformer for 3d object detection. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 17151–17160 (2022). https://doi.org/10.1109/CVPR52688.2022.01666
  * [113] Zeng, Y.X., Hsieh, J.W., Li, X., Chang, M.C.: Mixnet: Toward accurate detection of challenging scene text in the wild (2023)
  * [114] Zhou, D., Dai, Y., Li, H.: Reliable scale estimation and correction for monocular visual odometry. In: 2016 IEEE Intelligent Vehicles Symposium (IV). pp. 490–495 (2016). https://doi.org/10.1109/IVS.2016.7535431
  * [115] Zhou, D., Fang, J., Song, X., Liu, L., Yin, J., Dai, Y., Li, H., Yang, R.: Joint 3d instance segmentation and object detection for autonomous driving. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1836–1846 (2020). https://doi.org/10.1109/CVPR42600.2020.00191
  * [116] Zhou, D., Fang, J., Song, X., Liu, L., Yin, J., Dai, Y., Li, H., Yang, R.: Joint 3d instance segmentation and object detection for autonomous driving. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1836–1846 (2020). https://doi.org/10.1109/CVPR42600.2020.00191
  * [117] Zhou, H., Zhu, X., Song, X., Ma, Y., Wang, Z., Li, H., Lin, D.: Cylinder3d: An effective 3d framework for driving-scene lidar semantic segmentation (2020)
  * [118] Zhou, Y., Tuzel, O.: Voxelnet: End-to-end learning for point cloud based 3d object detection (2017). https://doi.org/10.48550/ARXIV.1711.06396, https://arxiv.org/abs/1711.06396
  * [119] Zhu, B., Wang, Z., Shi, S., Xu, H., Hong, L., Li, H.: Conquer: Query contrast voxel-detr for 3d object detection (2022). https://doi.org/10.48550/ARXIV.2212.07289, https://arxiv.org/abs/2212.07289
  * [120] Zhu, L., Wang, X., Ke, Z., Zhang, W., Lau, R.: Biformer: Vision transformer with bi-level routing attention (2023)
  * [121] Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning (2017)
  * [122] Zou, J., Zhu, Z., Ye, Y., Wang, X.: Diffbev: Conditional diffusion model for bird’s eye view perception (2023)