# Real-time 3D Object Detection using Feature Map Flow

Youshaa Murhij
Moscow Institute of Physics and Technology
Dolgoprodny, Russia
<EMAIL_ADDRESS>Dmitry Yudin
Moscow Institute of Physics and Technology
Dolgoprodny, Russia
<EMAIL_ADDRESS>
###### Abstract

In this paper, we present a real-time 3D detection approach considering time-
spatial feature map aggregation from different time steps of deep neural model
inference (named feature map flow, FMF). Proposed approach improves the
quality of 3D detection center-based baseline and provides real-time
performance on the nuScenes and Waymo benchmark. Code is available at
https://github.com/YoushaaMurhij/FMFNet

##  1 Introduction

3D LiDAR point clouds provide much more precise spatial details [12] compared
to RGB, RGB-D or stereo images from cameras [1] but they come in unordered and
sparse form.

Proposed method includes a development of a neural network for road scene
object detection based on a recurrent single-stage object detector using
multi-dimensional feature maps generated from two adjacent 3D point clouds.

Our contributions are the following:

  * •

we proposed an approach of feature map time-spatial aggregation named feature
map flow (FMF), see Figure 1;

  * •

we have demonstrated that FMF improves the quality of detection of various
basic state-of-the-art models and provides real-time performance on the
nuScenes dataset and Waymo benchmark.

Figure 1: Proposed approach to time-spatial aggregation of two feature maps
from different time steps of deep neural model inference (named feature map
flow, FMF): (a) general FMF structure, odometry data is optional, (b) FMF-Base
is concatenation with subsequent convolution, batch normalization and ReLU
activation.

##  2 Related Work

Processing point clouds in convolutional networks for detection task is
computationally expensive. To enhance the processing efficiency, single stage
approaches are used as in [5] to downscale the 3D point cloud using
lightweight Spatial-Semantic Feature Aggregation module to fuse high-level
abstract semantic features and low-level spatial features for accurate
predictions of bounding boxes and classification confidence. In other works
[7], hand crafted voxel grid was used to extract voxel features. While
VoxelNet model [14] removed the need of manual feature engineering for 3D
point clouds and adopted a generic 3D detection network that unifies feature
extraction and bounding box prediction into a single stage, end-to-end
trainable deep network. The point cloud is divided to equally spaced 3D voxels
and a group of points within each voxel are transformed into a unified feature
representation through a voxel feature encoding (VFE) layer.

Learning from unordered point sets was introduced in PointNets [9] that was
key network for PointPillars encoder [6], which utilizes PointNets to learn a
representation of point clouds organized in vertical columns (pillars). In our
work, PointPillars backbone is mainly used as a real-time detection model.

Figure 2: Architecture of the proposed method for 3D object detection with
feature map flow module named FMFNet.

Multi-head network was applied in [11] and [15], where sparse 3D convolution
are utilized to extract rich semantic features, which are then fed into a
class-balanced multi-head network to perform 3D object detection. To handle
the severe class imbalance problem inherent in the autonomous driving
scenarios, a class-balanced sampling and augmentation strategy was designed to
generate a more balanced data distribution.

While center-based 3D object detection and tracking method [13] is designed
based on CenterNet [4] to represent, detect, and track 3D objects as points.
CenterPoint framework [13], first detects centers of objects using a keypoint
detector and regresses to other attributes, including 3D size, 3D orientation,
and velocity. In a second stage, CenterPoint refines these estimates using
additional point features on the object.

##  3 Method

We study recurrent deep neural model for 3D object detection with architecture
shown on Figure 2.

Our model takes LiDAR point clouds as input and estimates oriented 3D boxes
for objects on interest (cars, pedestrians, etc).

We use two common backbones (PointPillars [6] and VoxelNet [14]) in our method
for converting a point cloud to a sparse pseudoimage.

Then, Neck block based on region proposal network (RPN) [10] processes the
pseudo-image into high-level bird’s eye view representation.

Proposed feature map flow module makes the time-spatial aggregation of these
representations from subsequent model inference time steps. Its general
structure is shown on Figure 1(a).

The result of the aggregation is fed to the multi-task head, which allows us
to find the two-dimensional coordinates of the centers of objects, estimate
their height, 3D box size, orientation angle of the object on the plane. So,
the problem of detection is reduced to the 2.5D case, which, however, quite
effectively describes the behavior of three-dimensional objects in real
traffic conditions.

Further in this section we will consider the work of the main module of the
proposed approach in more detail.

###  3.1 Feature Map Flow Module

We propose a recurrent time-spatial aggregation of latent feature maps. The
main idea behind the feature map flow block is to use the 2D feature map that
was build in the previous detection and regression step before feeding the
current generated map to the detection head. Figure 1(a) shows detailed scheme
of the proposed feature map flow module. In general, it can use the odometry
data for the affine transformation of the feature map. But in this paper we
are considering a simpler FMF-Base version, see Figure 1(b).

In the FMF-Base module, we use both the concatenation and shared convolutional
block with batch normalization and ReLU activation function that generates the
final 2D feature map. In a later stage, this 2D map will be fed to the
detection head module.

###  3.2 Multi-task Head for 3D Object Detection

We followed CenterPoint [13] method in our detection head implementation to
generate heat map peak at the center location of the detected objects by
producing a $K$-channel heat map $\hat{M}$, one channel for each of $K$
classes.

Projection of 3D centers of annotated bounding boxes into the map view is
needed during training the model. We use the Focal loss as in CenterNet [4].
Hence, 3D object detection turned in a keypoint estimation task as the raw
point cloud converted to 2D grid to predict a $w\times h$ heat map
$\hat{M}\in[0,1]^{w\times h\times K}$ for each class. The resulted heat map
contains local maximums that represent the centers of the detected objects. We
followed [4] to get the 3D box as we regress to a shared map $\hat{S}\in
R^{w\times h\times 2}$ storing the shared map width and height at the center
location.

Method | mAP | NDS | Car | Truck | Bus | Trailer | CV | Ped | Motor | Bicycle | TC | Barrier
---|---|---|---|---|---|---|---|---|---|---|---|---
FMF-PointPillars-Base (Ours) | 53.8 | 62.7 | 89.9 | 57.0 | 65.3 | 66.5 | 25.7 | 82.2 | 50.5 | 21.9 | 72.7 | 73.3
CenterPoint [13] | 58.0 | 65.5 | 84.6 | 51.0 | 60.2 | 53.2 | 17.5 | 83.4 | 53.7 | 28.7 | 76.7 | 70.9
FMF-VoxelNet-Base (Ours) | 58.0 | 65.9 | 89.8 | 59.3 | 67.0 | 68.6 | 32.8 | 85.7 | 58.6 | 30.2 | 77.4 | 77.0
Table 1: Comparison of the state-of-the-art methods for 3D detection on nuScenes test set, Abbreviations: construction vehicle (CV), pedestrian (Ped), motorcycle (Motor), and traffic cone (TC). Method | mAP↑ | mATE↓ | mASE↓ | mAOE↓ | mAVE↓ | mAAE↓ | NDS↑ |
---|---|---|---|---|---|---|---|---
CenterPoint-PointPillars | 0.5024 | 0.3130 | 0.2593 | 0.3936 | 0.3260 | 0.1976 | 0.6023 |
FMF-PointPillars-Base | 0.5256 | 0.3003 | 0.2564 | 0.3709 | 0.3267 | 0.1978 | 0.6176 |
Centerpoint-VoxelNet | 0.5543 | 0.3035 | 0.2591 | 0.3223 | 0.3096 | 0.1944 | 0.6283 |
FMF-VoxelNet-Base | 0.5719 | 0.2964 | 0.2552 | 0.3258 | 0.2793 | 0.1860 | 0.6517 |
Table 2: Ablation studies for 3D object detection on nuScenes validation set. Method | Veh↑ | Ped↑ | Cyc↑ | mAPH↑ | Latency, ms
---|---|---|---|---|---
FMF-PointPllars (FP16) | 69.65 | 54.61 | 62.28 | 62.18 | 62.30
FMF-VoxelNet | 70.74 | 65.46 | 67.63 | 67.95 | 86.11
Table 3: FMF modules performance for 3D detection on Waymo test set. We show the per-class and average Level 2 mAPH. Latency is shown for Tesla V100 GPU. Method | Veh↑ | Ped↑ | Cyc↑ | mAPH↑ | Latency, ms
---|---|---|---|---|---
CenterPoint-PointPillars | 65.5 | 55.1 | 60.2 | 60.3 | 52.83
FMF-PointPillars(FP16) | 61.75 | 58.12 | 65.26 | 62.19 | 62.30
FMF-PointPillars | 62.35 | 59.67 | 66.71 | 62.43 | 82.06
CenterPoint-VoxelNet | 66.2 | 62.6 | 67.1 | 64.7 | 84.05
FMF-VoxelNet(FP16) | 66.25 | 63.06 | 67.24 | 65.85 | 77.53
FMF-VoxelNet | 66.68 | 63.62 | 68.64 | 65.98 | 86.11
Table 4: Comparison between one-stage CenterPoint and FMF-based methods for 3D
detection on Waymo validation. We show the per-class and average Level 2 mAPH.
Latency is shown for Tesla V100 GPU.

We used rendered Gaussian kernels at the object center $q_{i}$ for each class
$c_{i}\in\\{1...K\\}$ and the target heat map for a pixel $p$ is
$M_{p,k}=\max_{i:c_{i}=k}\exp\left(-\frac{\left(p-q_{i}\right)^{2}}{2\sigma_{i}^{2}}\right)$,
where $\sigma_{i}$ is related to the size of the object.

$L_{hm}=-\frac{1}{N}\sum_{p,k}\left\\{\begin{array}[]{ll}\left(1-z\right)^{\alpha}\log\left(z\right),\text{if
}y=1,\\\
\left(1-y\right)^{\beta}\left(z\right)^{\alpha}\log\left(1-z\right)\text{,
o/w,}\end{array}\right.$ (1)

where $z=\hat{M}_{p,k}$ is resulting center coordinate heat map for $k$-th
class and $y=M_{p,k}$, ’o/w’ is abbreviation for ’otherwise’.

We use $L_{1}$ loss as 3D box size prediction loss $L_{s}$ at the Ground Truth
centers and regress to a local offset, supervised by $L_{l}$ loss to minimize
stride-errors as in [3]. Height ($L_{H}$), rotation angle ($L_{r}$) and
velocity ($L_{v}$) attributes are also trained with $L_{1}$ loss. Training
process is supervised by a total loss $L$ combining the all mentioned above
losses:

$L=L_{hm}+\gamma_{l}L_{l}+\gamma_{s}L_{s}+\gamma_{H}L_{H}+\gamma_{r}L_{r}+\gamma_{v}L_{v},$
(2)

where $\gamma_{i}$ is the loss weight for the corresponding attribute,
$\gamma_{l}=\gamma_{s}=\gamma_{H}=\gamma_{v}=1.0$ and $\gamma_{r}=0.2$.

##  4 Experiments

We mainly evaluated FMFNet on nuScenes and Waymo datasets. We implemented
FMFNet using two common 3D encoders for point clouds: VoxelNet [14] and
PointPillars [6].

###  4.1 Implementation Details

First, a voxelization setup is applied to sample the point cloud randomly into
pillars or voxels depending on the used model. We sampled up to 60,000 pillars
with a pillar size of [0.32, 0.32, 6.0] meters and up to 150,000 voxels with
voxel size of [0.1, 0.1, 0.15] meters. The maximum number of points in the
pillar is 20 while the voxel could contain up to 10 points. We fed these
pillars/voxels into the backbone. We followed [6] for PointPillars backbone
and [14] for VoxelNet backbone.

The network is trained by AdamW optimizer with one-cycle learning rate policy
[8] with an initial learning rate 0.003 and batch size 4 equally distributed
on 2 Tesla V100 GPU cards with weight decay 0.01, and momentum 0.85 to 0.95.
We trained our models for 32 epochs. We adopted data augmentation strategy in
order to prevent overfitting. We used random flipping along both $X$ and $Y$
axis, and global scaling with a random factor from [0.95; 1.05] with a random
global rotation between $[-\pi/8,\pi/8]$.

###  4.2 Performance

We compared the performance of our main 3D detection models with the state-of-
the-art methods on nuScenes dataset. Our implemented FMF-PointPillars models
run in real-time at 19FPS on GeForce RTX™ 3060 Ti while the most accurate FMF-
VoxelNet model runs at 14FPS. Table 1 shows mean average precision (mAP),
nuScenes detection score (NDS) and the mAP for each category on nuScenes test
set. Table 3 shows Level 2 per-class precision and Level 2 mean average
precision with heading on Waymo 3D detection test set. Our FMF-VoxelNet model
achieves a better performance in all 10 nuScenes categories and 3 Waymo
categories compared to the State-of-the-art baseline method, which make FMF-
PointPillars a reliable and strong baseline for a real-time 3D object
detection pipeline from LiDAR point clouds.

###  4.3 Ablation Study

All ablation studies are conducted on nuScenes validation dataset [2].

We first compared our detection approach results with the state-of-the-art
center-based 3D object detection and tracking method [13] (our baseline). Our
main 3D object detection results on nuScenes validation set are shown in Table
2 and on Waymo validation set are shown in Table 4. Our FMF detectors perform
much better than the baseline when category objects are rare and small,
reflecting the model’s ability to capture the rotation and size invariance
when detecting objects. These results convincingly highlight the advantage of
using FMF approach.

Our context based aggregation approach achieves similar results compared to
FMF-block with shared convolutions layers on the validation set.

##  5 Conclusion

In this work, we have proposed feature map flow module to improve quality of
center-based deep neural network architecture. The FMF-VoxelNet-Base model
achieved better performance on the nuScenes an Waymo benchmark for real-time
3D detection.

## References

  * [1] I. Basharov and D. Yudin.  Real-time deep neural networks for multiple object tracking and segmentation on monocular video.  ISPRS-International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 4421:15–20, 2021.
  * [2] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020.
  * [3] Qi Chen, Lin Sun, Zhixin Wang, Kui Jia, and Alan Yuille.  Object as hotspots: An anchor-free 3d object detection approach via firing of hotspots.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, pages 68–84, Cham, 2020. Springer International Publishing.
  * [4] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian.  Centernet: Keypoint triplets for object detection, 2019.
  * [5] C. He, H. Zeng, J. Huang, X. S. Hua, and L. Zhang.  Structure aware single-stage 3d object detection from point cloud.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11870–11879, 2020.
  * [6] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom.  Pointpillars: Fast encoders for object detection from point clouds.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12689–12697, Los Alamitos, CA, USA, jun 2019. IEEE Computer Society.
  * [7] B. Li.  3d fully convolutional network for vehicle detection in point cloud.  2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1513–1518, 2017.
  * [8] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization, 2019.
  * [9] C. R. Qi, H. Su, Kaichun Mo, and L. Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 77–85, 2017.
  * [10] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks, 2016.
  * [11] Ozan Sener and Vladlen Koltun.  Multi-task learning as multi-objective optimization.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  * [12] Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia.  Std: Sparse-to-dense 3d object detector for point cloud.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1951–1960, 2019.
  * [13] Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl.  Center-based 3d object detection and tracking.  CVPR, 2021.
  * [14] Yin Zhou and Oncel Tuzel.  Voxelnet: End-to-end learning for point cloud based 3d object detection, 2017.
  * [15] Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu.  Class-balanced grouping and sampling for point cloud 3d object detection, 2019.