# Frame Fusion with Vehicle Motion Prediction for 3D Object Detection

Xirui Li1 Feng Wang2 Naiyan Wang2 Chao Ma1
1Shanghai Jiao Tong University 2Tusimple
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In LiDAR-based 3D detection, history point clouds contain rich temporal
information helpful for future prediction. In the same way, history detections
should contribute to future detections. In this paper, we propose a detection
enhancement method, namely FrameFusion, which improves 3D object detection
results by fusing history frames. In FrameFusion, we “forward” history frames
to the current frame and apply weighted Non-Maximum-Suppression on dense
bounding boxes to obtain a fused frame with merged boxes. To “forward” frames,
we use vehicle motion models to estimate the future pose of the bounding
boxes. However, the commonly used constant velocity model fails naturally on
turning vehicles, so we explore two vehicle motion models to address this
issue. On Waymo Open Dataset, our FrameFusion method consistently improves the
performance of various 3D detectors by about $2$ vehicle level 2 APH with
negligible latency and slightly enhances the performance of the temporal
fusion method MPPNet. We also conduct extensive experiments on motion model
selection.

###  1 Introduction

LiDAR-based 3D object detection is a crucial component of modern autonomous
driving systems. To overcome the inherent sparsity of LiDAR scans, a common
practice is to utilize temporal information by incorporating multiple frame
point clouds as input. For example, we typically use $10$ sweeps on nuScenes
dataset [1] and $2-5$ frames on Waymo Open Dataset [20]. The point cloud
sequences are usually transformed to the latest timestamp using accurate
sensor poses. Then, all the points from multiple sweeps or frames are
concatenated together and fed into the network [31, 5]. This approach results
in denser static objects and moving objects with ”tails” (Figure 1), which
provide more prominent features for detectors.

(a) GT trajectory

(b) CV motion estimation

(c) Unicycle motion estimation

(d) Bicycle motion estimation

Figure 1: We propose a detection enhancement method that fuses the current
detection frame with history frames leveraging motion models for pose
estimation. A vehicle always moves along a trajectory in continuous frames
(current, future). We can predict its motion with the input point cloud
sequence and help with future detection. The more accurate the future
trajectory is estimated, the better enhancement can be expected. Constant
velocity model deviates as the vehicle turns. Unicycle model turns but is
still not fully aligned. Bicycle model fits the trajectory almost accurately.

With the temporal information, some work additionally adds a planar velocity
regressor (i.e., the speed for $x$ and $y$ axes) for downstream applications
such as object tracking [31] and trajectory prediction. But the regressor has
not been shown to benefit the detection task itself. However, recent research
in 3D object tracking [23] suggests that _a simple motion model in 3D space is
sufficient to predict the states of objects in future frames_. While this
principle is difficult to apply to 2D object detection due to perspective
projection, it is straightforward in 3D space. Meanwhile, in 3D detection,
when an object is partially occluded, detectors usually predict inaccurate
bounding boxes or miss it entirely, even if its previous observations are
reliable. Our work is motivated by this insight and seeks to enhance detection
performance by exploiting temporal information with motion models.

With the estimated velocities, we can construct a naive motion model to
predict the trajectories of detected objects. For weakly detected objects in
the current frame, we can supplement them with results forwarded by the motion
model from previous frames, potentially improving the detection performance.
Importantly, our method does not explicitly rely on object tracking and is
solely based on short-term prediction, making it immune to inaccuracies in
tracking. By fusing detected objects from the current frame with history
frames, our approach enhances detection performance, and we refer to it as
FrameFusion.

To ensure that our method can handle more complex motion states in practical
scenarios, we have delved into other vehicle motion models. As illustrated in
Figure 1(b), the constant velocity estimator predicts inaccurate poses for
turning vehicles. To address this limitation, we introduce two additional
motion models from robotic dynamics: the unicycle model and the bicycle model
(Fig. 1(c),1(d)). These models consider the motion constraints of turning
vehicles and produce more robust results for such cases, particularly for the
bicycle model. It is important to note that our exploration of different
motion models is not intended to improve the performance of our method on
benchmarks, as turning vehicles constitute only a negligible portion of the
dataset. Rather, our goal is to avoid introducing errors in turning cases that
are not expected in real-world situations.

Recently, some temporal fusion methods have shown impressive detection
performance, such as MPPNet [2], which employs a carefully-designed three-
level hierarchical framework to encode and fuse features from multiple frames.
It adopts a first-stage 3D detector to generate 3D proposal trajectories and
leverages motion information from the proposal trajectories, which is similar
to our method. While MPPNet achieves very high detection performance
improvement with its elaborate structure, it requires two-stage training and
introduces additional latency during inference. In contrast, our method
delivers a considerable enhancement with negligible latency cost as a simple
postprocessing step. Moreover, we show our method can improve the detection
performance based on the MPPNet result. It proves our method is complementary
to the feature-level temporal fusion methods to some extent.

We evaluate our method on various existing 3D detectors and multiple 3D
detection datasets. On Waymo Open Dataset, our method improves the performance
of CenterPoint [31] with point cloud sequence as input by $2.1$ in level 2
vehicle APH and achieves a slight $0.6$ enhancement on MPPNet-4frame [2].
Further experiments demonstrate the effectiveness of both unicycle and bicycle
models for turning vehicles.

In brief, our contributions are:

  * •

We propose a detection enhancement method that improves LiDAR-based 3D object
detection results by a postprocessing fusion step. Unlike past feature-level
or point-level fusion methods, our method directly exploits temporal
information in the detection frames by fusing boxes from previous frames
forwarded with motion models.

  * •

We first introduce vehicle motion models into the 3D object detection task. We
explore the unicycle model and the bicycle model to alleviate the motion
prediction error in the turning cases.

  * •

Evaluation on benchmarks demonstrates that our method can enhance the
detection performance of various 3D detection methods with negligible cost. We
also conduct extensive experiments on the selection of motion models.

###  2 Related Work

##### 3D LiDAR Object Detection

3D LiDAR object detection methods can be generally divided into two categories
by point cloud representation: point-based and voxel-based. Point-based
methods [19, 27, 28, 15] directly extract point features on raw irregular 3D
point clouds supported by PointNet [16] and then perform 3D detection. Voxel-
based methods [10, 24, 25, 26, 32, 31] instead divide the points into fixed-
size voxels and apply highly efficient 3D or 2D convolutions for detection. In
this work, we base most of our experiments on CenterPoint [31] while including
a comparison of our method performance on multiple 3D detectors [24, 18, 2].

##### Temporal Information Exploitation

Exploiting temporal information in point cloud sequences proves to be a
practical approach. A simple multi-frame point cloud concatenation adopted in
some recent work [8, 21, 14, 31, 1] can lead to an improvement compared to
single frame input. There are a series of methods diving deeper into modeling
the temporal information interaction at the feature level [29, 9, 30, 17, 12,
2]. 3D-Man [29] utilizes a memory bank to store temporal features and
aggregate them through an attention network. Offboard3D [17] significantly
improves the detection performance in the off-board scenario by utilizing the
whole point cloud sequence. MPPNet [2] recently proposed a sophisticated
three-level hierarchical framework that incorporates temporal information from
a long sequence using proxy points.

Unlike the feature-level fusion methods mentioned earlier, our FrameFusion
technique directly fuses temporal information at the frame level. This
approach enables our method to leverage the abundant information present in
correlated detection boxes across continuous frames. As a post-processing
step, FrameFusion is independent of any specific detection model and is much
faster than previous feature-level temporal fusion methods (as efficient as a
normal NMS step). Our experiments demonstrate that our method can enhance the
performance beyond the results obtained by MPPNet, suggesting that our method
is complementary to other temporal fusion methods.

Figure 2: Frame fusion procedure. To enhance a detection frame, first select a
set of history frames. Then, move forward detected boxes from history frames
to the latest frame with vehicle motion models. It generates an overlapped
frame with dense boxes. Apply weighted NMS on the dense boxes to get a fused
frame as the frame fusion output.

###  3 Preliminaries

##### 3D Detection

In 3D point clouds, an object is represented by a 3D bounding box
$b=(x,y,z,w,l,h,\phi)$ where $(x,y,z)$ is the center location, $(w,l,h)$ is 3D
size and $\phi$ is the box heading angle. Given an orderless point cloud
$\mathcal{P}=\\{(x,y,z)_{i}\\}$, 3D object detection aims to predict a set of
3D object bounding boxes $\mathcal{B}=\\{b_{k}\\}$ for all the target objects
in the given point cloud. We use a concatenated point cloud sequence as input
in our temporal setting. To encode temporal information, we append a time
indicator $t$ to the point, and thus the input point cloud is
$\mathcal{P}_{t}=\\{(x,y,z,t)_{i}\\}$.

##### Motion Model for 3D Bounding Boxes

Among the 3D bounding box attributes, $(w,l,h)$ is constant for a certain
object. We regard the others as object pose
$p(t)=\left(x(t),y(t),z(t),\phi(t)\right)$, or $p_{t}$ in short. It indicates
the location and orientation of an object at a certain moment $t$.

We define the motion model on 3D bounding boxes as the pose derivative with
respect to time. A motion model $M$ makes an assumption about the object
dynamics with a set of motion parameters $\theta_{M}$,

$M(p;\theta_{M})=\frac{\partial p}{\partial t}.$ (1)

Given a motion model $M$ and the object motion parameters $\theta_{M}$, the
pose at time $t$ can be estimated by integration,

$p_{t}=p_{0}+F_{M}(p_{0},t;\theta_{M})=p_{0}+\int_{0}^{t}M(p_{t};\theta_{M})dt,$
(2)

where $p_{0}$ is the initial object pose. We denote the integration function
$F_{M}(p_{0},t;\theta_{M})$ as the forward model of motion model $M$.

In this paper, we only model vehicle objects. For vehicles, the $z$-axis
variation is negligible. So we only keep the object pose as $p=(x,y,\phi)$ in
the remaining sections.

###  4 Method

In this section, we first elaborate on our proposed FrameFusion algorithm,
following a description of two motion models introduced for turning vehicles.
We explain our model implementation at last.

####  4.1 FrameFusion

The key idea of FrameFusion is to fuse the current frame with history frames
leveraging motion models for pose estimation. Fig. 2 outlines the steps
involved in frame fusion. For one specific frame, we forward history frame
boxes to this frame with vehicle motion models. Then dense overlapped boxes
are fused via weighted NMS. The use of weighted NMS ensures that dense boxes
are fused rather than simply being selected, thereby facilitating the fusion
of temporal information from different frames more effectively than standard
NMS.

For a target frame $t$, let $\\{b_{t}\\}$ be the detected bounding box set. We
select $N$ history frames with their detected bounding boxes set
$\\{b_{t-n},\dots,b_{t-1}\\}$, and then forward the history bounding boxes to
frame $T$ with forward model (Equation 2) to get
$\\{b^{for}_{t-n},\dots,b^{for}_{t-1}\\}$. Since the forwarded pose may not be
that accurate, we calculate a decayed confidence score with a weight decay
factor $d$, serving as the voting weight in the next step. For $i_{th}$
history frame, decayed scores are the box scores multiplied by $d^{i}$.

We fuse the dense detection set by applying weighted non-maximum suppression
(weighted NMS) [7]. Similar to standard NMS, we first sort the bounding boxes
by their scores. For each top-ranked bounding box $b$, we select a set of
boxes overlapped with $b$ with IoU larger than $h_{low}$. Then bounding boxes
whose IoUs with $b$ are larger than $h_{high}$ is voted together,

$b_{f}=\frac{\sum_{b_{k}\in\mathcal{N}(b;h_{high})}w_{k}\cdot
b_{k}}{\sum_{b_{k}\in\mathcal{N}(b;h_{high})}w_{k}},$ (3)

where $w_{k}$ is the (decayed) confidence score of $b_{k}$ and
$\mathcal{N}(b;h_{high})=\\{b_{k}|IoU(b,b_{k})>h_{high}\\}$. All the box
attributes, as well as the motion parameters and the box scores, are fused as
Equation 3. Scores of boxes that are only fused by history bounding boxes are
reduced to avoid affecting the current frame detection. We provide additional
details in the supplementary materials. Bounding boxes whose IoUs with $b$ are
in the range $[h_{low},h_{high}]$ are discarded. We iterate the above
procedure until all bounding boxes are fused together as a set
$\mathcal{B}_{f}$, which is our frame fusion output.

Overall, our FrameFusion technique can be regarded as a post-processing method
that can be applied to any continuous detection frames with motion parameters
prediction. In the next section, we will further introduce the motion models
we used, which determine the forward model and the motion parameters in
FrameFusion.

####  4.2 Motion Model

In our work, we explore three motion models for vehicle objects. It is
important to note that the selection of these models is not aimed at improving
benchmark performance. In fact, adopting different motion models achieves
similar results on the benchmarks. Rather, the introduction of the unicycle
and bicycle models is intended to alleviate the motion prediction error in
turning cases, which we are concerned about in practical scenarios. We will
further demonstrate these models in the following sections.

##### Constant Velocity (CV) Model

Estimating velocities on $xy$-axes is one of the defined tasks on nuScenes
Dataset, and the $xy$ velocities are also provided as ground-truth labels for
Waymo Dataset. So it is a common practice to predict the $xy$ velocities if
the input is multi-frame or multi-sweep. Its motion model can be simply
derived as $V(p,\theta_{V})=(v_{x},v_{y},0)$, and its forward model is
$F_{V}(p_{0},t;\theta_{V})=(v_{x}t,v_{y}t,0)$ with $\theta_{V}=(v_{x},v_{y})$.

In this paper, we regard it as our baseline motion model. It performs well
with our frame fusion method on benchmarks. However, it fails to describe the
motion of turning vehicles. We will introduce the following two vehicle models
to fix this issue.

##### Unicycle Model

(a) Unicycle model

(b) Bicycle model

Figure 3: Unicycle model and bicycle model diagram. For unicycle model, The
wheel moves with velocity $V$ and rotates with angular velocity $\omega$.
Unicycle model parameter $\theta_{U}=(V,\omega)$. For bicycle model, the
vehicle velocity $V$ deviates from the vehicle heading by a slip angle
$\beta$. Bicycle model parameter $\theta_{\beta}=(V,\beta,l_{r})$. The lighter
part indicates the wheel heading.

Unicycle model assumes a single-axle system. Suppose the vehicle is driven by
a single wheel (Fig. 3(a)), whose pose is $p_{w}=(x,y,\phi)$. It can move
along the heading angle $\phi$ with speed $V$. At the same time, it can also
rotate its heading with angular speed $\omega$. Unicycle model has two
parameters $\theta_{U}=(V,\omega)$. Here $V$ is the signed speed. Negative $V$
means opposite to wheel heading and vice versa. A vehicle shares the same pose
with the wheel, $p=p_{w}=(x,y,\phi)$. According to Equation 1, we give the
unicycle model $U$,

$U(p;\theta_{U})=\frac{\partial p}{\partial t}=\begin{bmatrix}V\cos\phi\\\
V\sin\phi\\\ \omega\\\ \end{bmatrix}.$ (4)

By integrating over time $t$, we can estimate pose at time $t$ with the
forward model $F_{U}$ (Equation 2),

$p_{t}=p_{0}+F_{U}(p_{0},t;\theta_{U})=p_{0}+\begin{bmatrix}\frac{V}{\omega}(\sin\phi_{t}-\sin\phi_{0})\\\
\frac{V}{\omega}(\cos\phi_{0}-\cos\phi_{t})\\\ \omega t\\\ \end{bmatrix},$ (5)

where $\phi_{t}=\phi_{0}+\omega t$ is the heading at time t.

The unicycle model provides a compact way to describe a rigid motion with both
heading and position variation. However, it is not accurate enough for most
vehicles, which usually have two axles instead of a single one. Nonetheless,
because of its simplicity, it is also widely used as an approximation in
mobile robotics [11, 6].

##### Bicycle Model

Bicycle model represents a system with two axles separated by a distance (Fig.
3(b)). The vehicle is driven by two wheels distributed on the vehicle center
line. For simplicity, we assume the rear wheel is parallel with the vehicle
heading $\phi$. The front wheel is the turning wheel with a rotation angle
difference $\delta_{f}$ over $\phi$. The distances between the center of mass
$C$ and two wheels are $l_{f},l_{r}$.

In the bicycle model, the vehicle pose is not identical to the wheel as in the
unicycle model. Thus the velocity does not always follow the vehicle heading
angle $\phi$. With a nonzero rotation of the front wheel, the front axle and
the rear axle intersect at a point $O$. The velocity $V$ is orthogonal to line
$OC$. We define the angle between vehicle heading and velocity direction as
the tire slip angle $\beta$. $\beta$ is $\pi$-normalized to $[-\pi/2,\pi/2]$.

We derive the bicycle model as

$B(p;\theta_{B})=\frac{\partial p}{\partial
t}=\begin{bmatrix}V\cos(\phi+\beta)\\\ V\sin(\phi+\beta)\\\
\frac{V\sin\beta}{l_{r}}\end{bmatrix}.$ (6)

The bicycle model has three parameters $\theta_{B}=(V,\beta,l_{r})$. $\beta$
depends on the front wheel rotation angle $\delta_{f}$,

$\beta=\arctan(\frac{l_{r}}{l_{f}+l_{r}}\tan\delta_{f}).$ (7)

Note that $\beta,\delta_{f}$ are equivalent in the bicycle model with given
$l_{f}$ and $l_{r}$. We choose $\beta$ as a motion parameter for compactness.

Similar to the unicycle model, we have the bicycle forward model by Equation
2,

$p_{t}=p_{0}+\begin{bmatrix}\frac{l_{r}}{\sin\beta}(\sin(\phi_{t}+\beta)-\sin(\phi_{0}+\beta))\\\
\frac{l_{r}}{\sin\beta}(\cos(\phi_{0}+\beta)-\cos(\phi_{t}+\beta))\\\
\frac{V\sin\beta}{l_{r}}t\end{bmatrix},$ (8)

where $\phi_{t}=\phi_{0}+\frac{V\sin\beta}{l_{r}}t$ is the heading at time
$t$.

Compared with the more advanced Ackermann model [13], the bicycle model trades
off between accuracy and simplicity. We will show in experiments that the
bicycle model is accurate enough to model the turning vehicles.

##### Discussion

In this section, we briefly discuss the pros and cons of the three
aforementioned motion models. A naive observation is that these motion models
are the same when the vehicle moves straight. We can set $\omega$ and $\beta$
in the unicycle and bicycle model to zero and get the same forward model as
the constant velocity model. So we only discuss the turning cases when
comparing the motion models.

Figure 4: Forwarding history point cloud into the current frame with motion
model.

Fig. 4 shows a turning vehicle point cloud sequence in eight frames. We
estimate its motion parameter at each timestamp with adjacent box poses and
forward all the point clouds to the latest frame $T$. We can see the constant
velocity model diverges in the heading orientation as it estimates incorrect
velocity directions. In comparison, the unicycle model shows significantly
better motion estimation performance with a better-aligned point cloud.
However, it still shifts a little that can be figured out at the border. The
bicycle model generates an almost perfect matching. We can hardly tell points
from different frames, presenting its high accuracy in vehicle motion
prediction.

Though the bicycle model result is satisfying, we observe a short blue tail at
the rear. It actually suggests an acceleration in a short period. As we do not
introduce second-order derivatives such as acceleration in our motion models
for model simplicity, it may cause inevitable errors in long-term motion
estimation. It also indicates the rationality behind the confidence decay
mechanism in our frame fusion method.

####  4.3 Model Implementation

To extract motion information from point cloud sequences, we augment existing
3D detectors with additional motion parameter regressors. For any models
mentioned in our work, multi-frame input fusion [24, 14, 1] is adopted to
integrate temporal information into model input for motion parameter
prediction.

In the Waymo Open Dataset, we observe that vehicles with nonzero angular
velocity account for only $3\%$ of the total, and their distribution is skewed
around zero. When regressing angular motion parameters, this unbalanced
distribution makes the network more likely to predict zero values for any
target using a normal L1 loss. To address this issue, we employ a balanced
loss function that equalizes the influence of turning and non-turning objects.
We divide objects into two classes based on an angular velocity threshold of
$0.1$: turning and non-turning. When calculating the loss of angular
parameters, we first average the loss in each class and then sum them up to
obtain the regression loss.

Before training, we pre-process the datasets to generate ground-truth motion
parameter labels with inverse motion models. Given a pair of poses
$(p_{0},p_{t})$, an inverse model estimates the motion parameter
$\hat{\theta_{M}}$. We refer the readers to supplementary material for inverse
model details. For frame $t$, its motion parameters are estimated using object
poses in adjacent frames $t-1$ and $t+1$ as $(p_{0},p_{t})$.

###  5 Experiments

####  5.1 Experimental Settings

##### Waymo Open Dataset

Waymo Open Dataset [20] is a large-scale, diverse 3D detection dataset,
including three classes: vehicle, bicycle, and pedestrian. The ofﬁcial 3D
detection evaluation metrics are standard 3D bounding box mean average
precision (mAP) [4] and mAP weighted by heading accuracy (mAPH). As we solely
consider vehicle class in evaluation, tables show AP and APH on vehicle class.
The annotated objects are split into two difficulty levels: LEVEL 1 (More than
ﬁve Lidar points in bbox), and LEVEL 2 (at least one Lidar point in bbox).

##### nuScenes

nuScenes [1] comprises 1000 scenes, with the interval between adjacent
annotated frames being 0.5s (0.1s in Waymo). In our experiments, we only
consider 5 vehicle classes in the 10 providing object classes. The primary 3D
detection metrics are mean Average Precision (mAP) and nuScenes detection
score (NDS) [1].

##### Implementation Details

In experiment sections, initially capitalized vehicle motion models represent
our models built on the CenterPoint-Voxel model with corresponding motion
parameter regressors such as Constant Velocity (CV), Unicycle, and Bicycle
model. We implement these CenterPoint-based models on a 3D detection codebase
MMDetection3D [3]. We implement the various 3D detectors in Table 1 on another
open 3D detection codebase OpenPCDet [22] and adopt the constant velocity
model for these methods.

All models use multi-frame point fusion as input ($[-4,0]$ for Waymo and
$[-10,0]$ for nuScenes). We train CenterPoint-based motion models with AdamW
optimizer and a cyclic learning rate scheduler with a maximum learning rate
1e-3 and a weight decay $0.01$. Train for 12 epochs on Waymo and 20 epochs on
nuScenes. Models in Table 1 are trained with the default configuration.

For frame fusion, we set the frame fusion range $N=4$ by default (fuse 4
previous frames). Configurations of other parameters are left in supplementary
materials.

####  5.2 Main Results

Method |  AP/APH$\uparrow$ (VEH) | #Params∗ | Latency∗
---|---|---|---
Level 1 | Level 2
SECOND [24] | 71.9/71.2 | 63.9/63.3 |  5.3M |  137ms
CenterPoint [31] | 76.7/76.1 | 69.1/68.6 |  7.8M |  74ms
PV-RCNN [18] | 78.1/77.6 | 70.1/69.6 | 13.5M | 1085ms
MPPNet-4F [2] | 81.5/81.0 | 74.0/73.6 | +7.9M | +301ms
SECOND+ | 73.6/73.0 | 65.6/65.1 | - |  +3ms
CenterPoint+ | 78.7/78.2 | 71.2/70.7 | - |  +3ms
PV-RCNN+ | 80.0/79.4 | 72.1/71.6 | - |  +3ms
MPPNet-4F+ | 82.0/81.6 | 74.6/74.2 | - |  +3ms
Table 1: Vehicle detection enhancement over various 3D detectors on Waymo
validation set. + means performance after applying frame fusion. All methods
use 4-frame input and constant velocity motion model. Latency is measured on a
Nvidia 3090 GPU in milliseconds per frame. $*$ : For methods relying on
initial proposals generated from other 3D detectors such as our frame fusion
and MPPNet, we report the additional parameter number and latency introduced
by themselves for a clear comparison.

(a) Box refinement

(b) Score refinement

(c) Box covering

(d) FN covering

(e) CV Negative

(f) Bicycle Negative

Figure 5: FrameFusion visualization on bounding boxes. Each figure depicts a
red GT box, a green box from the frame before fusion and a blue box from the
enhanced frame after fusion. Orange boxes are the estimations dense
estimations before fused to the blue box.

##### 3D Detection Enhancement

Method |  VEH AP$\uparrow$ |  VEH APH$\uparrow$
---|---|---
Level 1 | Level 2 | Level 1 | Level 2
CenterPoint | 76.08 | 68.17 | 75.55 | 67.68
Constant Velocity | 75.64 | 67.74 | 75.11 | 67.26
Unicycle | 75.38 | 67.46 | 74.84 | 66.98
Bicycle | 75.56 | 67.62 | 75.02 | 67.13
Constant Velocity+ | 77.82 | 70.14 | 77.33 | 69.68
Unicycle+ | 77.65 | 69.93 | 77.14 | 69.46
Bicycle+ | 77.84 | 70.11 | 77.33 | 69.65
Table 2: 3D vehicle detection improvements on Waymo validation set with
different motion models. + means performance after applying frame fusion. All
models use 4-frame input. CenterPoint [31]: backbone model without motion
parameter regression.

FrameFusion improves the 3D detection performance by fusing history detection
frames with the current frame. On the Waymo dataset, we demonstrate that our
method can enhance the performance of existing 3D detection methods by
applying a 4-frame fusion to their results (Table 1). When applied to the 3D
detectors SECOND [24], CenterPoint [31], and PV-RCNN [18], FrameFusion
achieves significant enhancements in the vehicle 3D APH (level 2) by
$1.8,2.1,2.0$ with negligible latency costs ($3$ms). To ensure a fair
comparison with MPPNet, we use its 4-frame version, which aligns with our
4-frame fusion setting, and take CenterPoint evaluated in the same table as
its first-stage model. Compared to MPPNet, our method achieves a $42\%$
$(2.1/5.0)$ enhancement over the same CenterPoint model without the retraining
step, model parameters, or high latency introduced by MPPNet. Notably, our
method can even improve MPPNet’s performance slightly by $0.6$ (MPPNet-4F+),
suggesting that our method is complementary to other temporal fusion methods
to some extent.

We compare the frame fusion performance for 3D detection enhancement with
different motion models based on the CenterPoint [31] backbone. On the Waymo
validation set (Table 2), frame fusion shows similar enhancement for Constant
Velocity, Unicycle, and Bicycle model, with an increase of $+2.42,+2.48,+2.52$
in vehicle level 2 APH. Similar results are obtained on the nuScenes Dataset
(Table 3). It indicates that FrameFusion works consistently with different
motion models. In the next section, we will analyze the differences between
these models in turning cases.

Method |  mAP$\uparrow$ |  NDS$\uparrow$
---|---|---
CenterPoint | 53.44 | -
Constant Velocity | 53.67 | 61.73
Unicycle | 53.61 | 61.80
Bicycle | 53.55 | 61.72
Constant Velocity+ | 55.17 | 62.78
Unicycle+ | 54.87 | 62.72
Bicycle+ | 55.05 | 62.77
Table 3: 3D detection improvements on vehicle classes of nuScenes validation
set with different motion models. All models use 10-sweep input. CenterPoint
[31]: backbone model without motion parameter regression.

##### Turning Case Evaluation

As discussed in Sec. 4.2, motion models perform identically when the angular
velocity is zero. In Waymo, the ratio of turning vehicles is about $0.05$.
Thus previous results do not fully reflect the difference among motion models.
We further explore the motion model features in the turning case.

We evaluate motion model performance on a turning vehicles subset of Waymo
validation set. Considering a standard turning vehicle, we select vehicles in
GT with velocity $V>5$ and turning radius $R<25$ as the turning subset. The
detection results are matched to the turning subset to filter out non-turning
vehicles.

Table 4 shows the evaluation results on the turning subset. Constant Velocity
model decreases the performance on turning vehicles (about -3.78 level 2 APH)
after frame fusion, while Unicycle model and Bicycle model still get
improvement. Bicycle model shows its superiority with the best performance on
the turning case.

Method |  VEH AP$\uparrow$ |  VEH APH$\uparrow$
---|---|---
Level 1 | Level 2 | Level 1 | Level 2
Constant Velocity | 81.36 | 80.67 | 80.80 | 80.12
Unicycle | 80.85 | 80.35 | 80.17 | 79.67
Bicycle | 81.62 | 80.93 | 81.13 | 80.44
Constant Velocity+ | 78.48 | 77.74 | 77.07 | 76.34
Unicycle+ | 82.39 | 81.67 | 81.85 | 81.14
Bicycle+ | 82.96 | 82.29 | 82.34 | 81.67
Table 4: 3D vehicle detection improvements evaluated on turning subset of
Waymo validation set with different motion models.

The preceding experiments confirm the effectiveness of our motion models in
capturing turning targets. While any of the three models can produce similar
performance improvements in overall performance, if we aim to consistently
improve detection across various motion states, which is crucial in practical
scenarios, the unicycle or bicycle model should be prioritized due to its
superior precision in vehicle modeling.

####  5.3 Ablation Studies

##### Qualitative Analysis

We intuitively present our method results with a visualization in Fig. 5. Our
method enhances the detection in two forms, divided by whether a current frame
box is fused in the weighted NMS (refinement in Fig 5(a), 5(b)) or not
(covering in Fig 5(c), 5(d)).

For a slightly deviated detection (Fig. 5(a)), the fusion box directly refines
on the deviated box by merging matched boxes in the overlapped frame. High
confidence detection can also enhance the matched future detection scores,
filtering better proposals (Fig. 5(b)).

For another type, covering, the vehicle is often weakly detected in the
current frame and thus results in a significant prediction error (Fig. 5(c))
or a missing (Fig. 5(d)). In this case, the boxes from history frames merge by
themselves and fuse to generate a TP detection.

Fig. 5(e) shows a negative optimized turning vehicle sampled from the frame
fusion result of Constant Velocity model. The forwarded box orientations
deviated from the ground truth, causing the box heading to rotate wrongly
after fusion. We put another Bicycle model negative sample in Fig. 5(f).
However, the negative correction is supposed to be negligible with the
confidence decay and IoU threshold in the frame fusion.

##### Analysis of Fusion Frame Number

Figure 6: Detection Performance with various numbers of fusion frames
evaluated with CenterPoint-Bicycle model. Blue: Vehicle level 2 APH on Waymo
Validation set. Red: Latency in milliseconds per frame.

We analyze the performance and efficiency of our method concerning the fused
frame number (Fig. 6). As more frames are fused, the performance increment
soon converges, while the latency increases linearly. This can be attributed
to the fact that, as the fusion number grows, less additional temporal
information is integrated from distant frames. Furthermore, as the time
interval grows, the motion prediction becomes less accurate, and the
confidence decay strategy causes the voting weight of forwarded bounding boxes
to decay rapidly. Therefore, fusing a distant frame may hardly enhance current
detection. In practical scenarios, fusing 2 to 4 frames can provide sufficient
improvement.

##### Analysis of Vehicle Motion States

To further analyze the performance of the motion models on different motion
states, we evaluated the frame fusion enhancements separately for stationary,
straight, and turning motion vehicles (Table 5). We found that the motion
models performed almost identically when vehicles were stationary or moving
straight, indicating that they were equally effective in these scenarios.
However, we observed a decreasing trend in performance from stationary to
straight to turning motion, which suggests a decrease in the capability of the
motion models and the accuracy of the motion parameter predictions as the
motion becomes more complex. The turning cases are discussed in more detail in
section 5.2.

Method | All | Stationary | Straight | Turning
---|---|---|---|---
Constant Velocity | +2.42 | +2.95 | +1.24 | -3.40
Unicycle | +2.48 | +2.95 | +1.39 | +0.83
Bicycle | +2.52 | +3.02 | +1.20 | +1.28
Table 5: Compare frame fusion enhancement on different motion states. Show
vehicle level 2 APH detection increment after frame fusion. Stationary,
straight, turning vehicles account for $0.63$, $0.31$, $0.05$ in Waymo
validation set.

##### Frame Fusion Ablation

Method |  APH($\Delta$ APH) | Latency
---|---|---
CenterPoint | 67.1 | -
CenterPoint+(Default) | 69.7(+2.6) | 2.8ms
Circle NMS | 68.2(+1.1) | 3.0ms
Box Refinement (Fig. 5(a)) | 68.6(+1.5) | 2.8ms
Score Refinement (Fig. 5(b)) | 67.7(+0.6) | 2.8ms
Only Covering (Fig. 5(c) 5(d)) | 67.7(+0.6) | 2.8ms
Only Refinement (Fig. 5(a) 5(b)) | 69.1(+2.0) | 2.8ms
Table 6: Frame fusion ablation on CenterPoint-Bicycle model. Show vehicle
level 2 APH on Waymo validation set. Above: Default frame fusion result.
Below: Ablation settings. We annotate some settings with their visualizations
in qualitative analysis.

We present the results of frame fusion ablations in Table 6. Replacing
weighted NMS with normal circle NMS means the new boxes are selected instead
of fused, thus leading to a large performance drop. Subsequent ablations aim
to quantify the contribution of each enhancement component mentioned in
qualitative analysis. The latency analysis shows that our method is as
efficient as a GPU-implemented NMS postprocessing step.

###  6 Conclusion

In this paper, we propose a 3D object detection enhancement method
FrameFusion, fusing a detection frame with a series of history frames
forwarded by motion models. We introduce vehicle motion models, including the
unicycle model and the bicycle model to better capture the motion of turning
vehicles. Our approach is simple and highly efficient, making it suitable for
practical applications in 3D detection systems. We hope that our work will
inspire further improvements and applications of the method in the future.

### References

  * [1] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020.
  * [2] Xuesong Chen, Shaoshuai Shi, Benjin Zhu, Ka Chun Cheung, Hang Xu, and Hongsheng Li.  Mppnet: Multi-frame feature intertwining with proxy points for 3d temporal object detection.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VIII, pages 680–697. Springer, 2022.
  * [3] MMDetection3D Contributors.  Mmdetection3d: Openmmlab next-generation platform for general 3d object detection, 2020.
  * [4] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International journal of computer vision, 88(2):303–338, 2010.
  * [5] Runzhou Ge, Zhuangzhuang Ding, Yihan Hu, Yu Wang, Sijia Chen, Li Huang, and Yuan Li.  Afdet: Anchor free one stage 3d object detection.  arXiv preprint arXiv:2006.12671, 2020.
  * [6] Jawhar Ghommam, Hasan Mehrjerdi, Maarouf Saad, and Faiçal Mnif.  Formation path following control of unicycle-type mobile robots.  Robotics and Autonomous Systems, 58(5):727–736, 2010.
  * [7] Spyros Gidaris and Nikos Komodakis.  Object detection via a multi-region and semantic segmentation-aware cnn model.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
  * [8] Yihan Hu, Zhuangzhuang Ding, Runzhou Ge, Wenxin Shao, Li Huang, Kun Li, and Qiang Liu.  Afdetv2: Rethinking the necessity of the second stage for object detection from point clouds.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 969–979, 2022.
  * [9] Rui Huang, Wanyue Zhang, Abhijit Kundu, Caroline Pantofaru, David A Ross, Thomas Funkhouser, and Alireza Fathi.  An lstm approach to temporal 3d object detection in lidar point clouds.  In European Conference on Computer Vision, pages 266–282, 2020\.
  * [10] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom.  Pointpillars: Fast encoders for object detection from point clouds.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12697–12705, 2019.
  * [11] Ti-Chung Lee, Kai-Tai Song, Ching-Hung Lee, and Ching-Cheng Teng.  Tracking control of unicycle-modeled mobile robots using a saturation feedback controller.  IEEE transactions on control systems technology, 9(2):305–318, 2001\.
  * [12] Chenxu Luo, Xiaodong Yang, and Alan Yuille.  Exploring simple 3d multi-object tracking for autonomous driving.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10488–10497, 2021.
  * [13] Andreas Mueller.  Modern robotics: Mechanics, planning, and control [bookshelf].  IEEE Control Systems Magazine, 39(6):100–102, 2019.
  * [14] AJ Piergiovanni, Vincent Casser, Michael S Ryoo, and Anelia Angelova.  4d-net for learned multi-modal alignment.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15435–15445, 2021.
  * [15] Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas.  Deep hough voting for 3d object detection in point clouds.  In proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9277–9286, 2019.
  * [16] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.
  * [17] Charles R Qi, Yin Zhou, Mahyar Najibi, Pei Sun, Khoa Vo, Boyang Deng, and Dragomir Anguelov.  Offboard 3d object detection from point cloud sequences.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6134–6144, 2021.
  * [18] Shaoshuai Shi, Chaoxu Guo, Jihan Yang, and Hongsheng Li.  Pv-rcnn: The top-performing lidar-only solutions for 3d detection/3d tracking/domain adaptation of waymo open dataset challenges.  arXiv preprint arXiv:2008.12599, 2020.
  * [19] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li.  Pointrcnn: 3d object proposal generation and detection from point cloud.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 770–779, 2019.
  * [20] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al.  Scalability in perception for autonomous driving: Waymo open dataset.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2446–2454, 2020.
  * [21] Pei Sun, Weiyue Wang, Yuning Chai, Gamaleldin Elsayed, Alex Bewley, Xiao Zhang, Cristian Sminchisescu, and Dragomir Anguelov.  Rsn: Range sparse net for efficient, accurate lidar 3d object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5725–5734, 2021.
  * [22] OpenPCDet Development Team.  Openpcdet: An open-source toolbox for 3d object detection from point clouds.  https://github.com/open-mmlab/OpenPCDet, 2020.
  * [23] Qitai Wang, Yuntao Chen, Ziqi Pang, Naiyan Wang, and Zhaoxiang Zhang.  Immortal tracker: Tracklet never dies.  arXiv preprint arXiv:2111.13672, 2021.
  * [24] Yan Yan, Yuxing Mao, and Bo Li.  Second: Sparsely embedded convolutional detection.  Sensors, 18(10):3337, 2018.
  * [25] Bin Yang, Ming Liang, and Raquel Urtasun.  Hdnet: Exploiting hd maps for 3d object detection.  In Conference on Robot Learning, pages 146–155. PMLR, 2018.
  * [26] Bin Yang, Wenjie Luo, and Raquel Urtasun.  Pixor: Real-time 3d object detection from point clouds.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7652–7660, 2018.
  * [27] Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia.  3dssd: Point-based 3d single stage object detector.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11040–11048, 2020.
  * [28] Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia.  Std: Sparse-to-dense 3d object detector for point cloud.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1951–1960, 2019.
  * [29] Zetong Yang, Yin Zhou, Zhifeng Chen, and Jiquan Ngiam.  3d-man: 3d multi-frame attention network for object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1863–1872, 2021.
  * [30] Junbo Yin, Jianbing Shen, Chenye Guan, Dingfu Zhou, and Ruigang Yang.  Lidar-based online 3d video object detection with graph-based message passing and spatiotemporal transformer attention.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11495–11504, 2020.
  * [31] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl.  Center-based 3d object detection and tracking.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11784–11793, 2021.
  * [32] Yin Zhou and Oncel Tuzel.  Voxelnet: End-to-end learning for point cloud based 3d object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4490–4499, 2018.

## Appendix

###  Appendix A Score Decay Strategy

We decay the box scores in two circumstances in our frame fusion method.

##### Weight Decay

The first one is the decayed box confidences which serve as the voting weight
in the weighted NMS. For a box $b_{i}$ from history frame $i$, its decayed
confidence will be computed as

$w_{i}=s_{i}\cdot d^{t_{i}/\tau},$

where $s_{i}$ is the original box confidence score, $d$ is a weight decay
factor, $t_{i}$ is the time lag of history frame $i$ and $\tau$ is the dataset
frame interval.

##### Box Score Decay

The second situation is when a box is solely fused by boxes from history
frames, which means it is a new detection box added to the current frame. To
avoid affecting the current detection, we decay the scores of these boxes. Two
different decay strategies are explored in our experiments. Initially, we
directly take the decayed confidence (the voting weight) as the decayed score.
The confidence is fused as an attribute in weighed NMS and then serves as the
new score. In the experiments, we find that the number of history boxes fused
affects the box precision significantly. Thus another decay formula is tried
for a fused box $b_{f}$

$s_{f}=\frac{d_{s}\cdot s}{max(N-n_{f},1)},$

where $s$ is the original box confidence score, $d_{s}$ is a score decay
factor ($0.6$ in our experiments), $N$ is the frame fusion number and $n_{f}$
is the number of boxes fused by $b_{f}$. For the above two strategies, we call
them decay and divide strategies respectively.

We construct our score decay strategies from our pioneer works on track-based
detection enhancement. They are currently simple and intuitive. We think it is
highly feasible to tune or change the strategies for improving our method or
apply it in specific scenes.

###  Appendix B FrameFusion Configurations

Method | $d$ | $h_{low}$ | $h_{high}$ | Score Decay
---|---|---|---|---
CenterPoint-Based(Waymo) | 0.8 | 0.7 | 0.7 | decay
CenterPoint-Based(nuScenes) | 0.6 | 0.2 | 0.7 | decay
Multiple Methods(Table 1) | 0.8 | 0.9 | 0.9 | divide
Table A: FrameFusion configurations for different experiments. We use the
second line for Table 3 and the third line for Table 1. All the other
experiments are based on line 1 config.

Table A presents the parameter configurations for all of our experiments. In
our CenterPoint-based experiments, we primarily focus on exploring the
performance of the motion models and do not exhaustively tune frame fusion
parameters. We set $h_{low}$ to match the original NMS threshold on the
dataset ($0.7$ on Waymo [20], $0.2$ on nuScenes [1]), and $h_{high}$ to $0.7$.
In follow-up experiments on multiple 3D detection methods to further validate
the effectiveness of our frame fusion, we found that setting $h_{low}$ higher
consistently produced better fusion results. Two score strategies do not have
a clear priority over each other. To sum up, we recommend using
$d=0.8,h_{low}=h_{high}=0.9$ on Waymo Dataset and trying both score decay
strategies or modifying it if necessary.

###  Appendix C Ground-Truth Generation

For motion parameter regression, we preprocess the dataset with inverse models
to generate GT motion parameters before training. Given a pose pair
$(p_{0},p_{t})$, an inverse model estimates the motion parameter
$\hat{\theta_{M}}$. We introduce our inverse models for unicycle and bicycle
models.

For the unicycle model, given $(p_{0},p_{t})$, the inverse model can be
directly inferred from the unicycle forward model:

$\hat{\theta_{U}}=\begin{bmatrix}V\\\ \omega\\\
\end{bmatrix}=\begin{bmatrix}\frac{\Delta\phi}{\sin\Delta\phi}(V_{x}\cos\phi_{0}+V_{y}\sin\phi_{0})\\\
\Delta\phi/t\\\ \end{bmatrix},$ (9)

where
$\Delta\phi=\phi_{t}-\phi_{0},V_{x}=(x_{t}-x_{0})/t,V_{y}=(y_{t}-y_{0})/t$.

For the bicycle model, we adopt the numerical method to solve $\theta_{B}$. In
specific, we use the Gaussian-Newton algorithm to solve a least square problem
of pose estimation error. Given a pose pair $(p_{0},p_{t})$, a motion
parameter estimation $\theta_{B}$, the pose estimation error is
$r(\theta_{B})=p_{t}-\hat{p_{t}}$ where $\hat{p_{t}}$ is the time $t$ pose
estimated with $\theta_{B},p_{0}$ by the forward model. Thus the target is

$\min_{\theta_{B}}L=\frac{1}{2}r(\theta_{B})^{T}r(\theta_{B}).$ (10)

With an initial $\theta_{B}^{0}$, we iteratively update the motion parameter
estimation by

$\theta_{B}^{s+1}=\theta_{B}^{s}-Inv(J_{r})r(\theta_{B}^{s}),$ (11)

where $J_{r}$ is the Jacobian matrix of $r$ about $\theta_{B}$ and
$Inv(J_{r})$ is the pseudo-inverse of $J_{r}$. In practice, we terminate the
iteration when $L_{k}-L_{k-1}<1e-6$ and output $\theta_{B}^{k}$ as the motion
parameter estimation.

$J_{r}$ can be derived from the bicycle forward model,

$J_{r}=\begin{bmatrix}t\cos_{t}&\frac{\partial r_{x}}{\partial\beta}\\\
t\sin_{t}&\frac{\partial r_{y}}{\partial\beta}\\\
\frac{\sin\beta}{l_{r}}t&\frac{V\cos\beta}{l_{r}}t\\\ \end{bmatrix},$ (12)
$\begin{split}\frac{\partial
r_{x}}{\partial\beta}&=-\frac{l_{r}}{\sin\beta}[\cot\beta(\sin_{t}-\sin_{0})-(\frac{\partial
r_{\phi}}{\partial\beta}+1)\cos_{t}+\cos_{0}]\\\ \frac{\partial
r_{y}}{\partial\beta}&=-\frac{l_{r}}{\sin\beta}[\cot\beta(\cos_{t}-\cos_{0})-(\frac{\partial
r_{\phi}}{\partial\beta}+1)\sin_{t}+\sin_{0}],\end{split}$

where
$\cos_{0}=\cos(\phi_{0}+\beta),\sin_{0}=\sin(\phi_{0}+\beta),\cos_{t}=\cos(\hat{\phi_{t}}+\beta),\sin_{t}=\sin(\hat{\phi_{t}}+\beta)$.

For each object at frame $t$, we estimate its motion parameters with object
poses in adjacent frames $t-1$ and $t+1$ as $(p_{0},p_{t})$ in the inverse
models.

###  Appendix D Frame Fusion Algorithm

We provide the pseudocode of our proposed frame fusion algorithm (Algo. 1).
Readers can refer to it for more algorithm details.

Data: $D=\\{D_{t-i}\\}_{i=0}^{N}$: Detections from frame $T-N$ to $T$.
$D_{t-i}=\\{d_{t-i}\\}=\\{b_{t-i},s_{t-i},c_{t-i},\theta_{t-i}\\}$ includes
bounding boxes $b_{t-i}$, scores $s_{t-i}$, class labels $c_{t-i}$ and motion
parameters $\theta_{t-i}$ of all detection $d_{t-i}$ in frame $T-i$.

Result: $D^{f}$: Fused frame $T$.

1

Hyperparameters: Target classes $C$, motion model $M$, score decay factor $d$,
low and high IoU threshold $h_{low},h_{high}$

2 for  _$i$ :$1\rightarrow N$_ do

3 $t\leftarrow$ Time(Frame $T$) - Time(Frame $T-i$);

4 $b_{t-i}^{for}\leftarrow$ ForwardM($b_{t-i}$, $\theta_{t-i}$, $t$);

// Forward boxes with motion model $M$

5 Transform $b_{t-i}^{for}$ to frame T coordinate;

$w_{t-i}\leftarrow s_{t-i}\cdot d^{t/\tau}$ ;

// Get decayed weight

6
$D_{t-i}^{for}\leftarrow\\{b_{t-i}^{for},s_{t-i},c_{t-i},\theta_{t-i},w_{t-i}\\}$;

7

8 end for

9

$D\leftarrow\cup\\{D_{t-i}^{for}\\}_{i=0}^{N}$ ;

// Dense overlapped frame

10 $D\leftarrow$ Sort($D$, key=$sd$);

11 $D_{f}\leftarrow\emptyset$;

// Start weighted NMS

12 for  _$c\in C$_ do

13 $D_{c}\leftarrow\\{d|d\in D,d.c=c\\}$;

14 $D_{cf}\leftarrow\emptyset$;

// Start fusing boxes with label $c$

15 for  _$d\in D_{c}$_ do

16 $D_{remove}\leftarrow\\{d_{r}|IoU(d_{r}.b,d.b)>h_{low}\\}$;

17 $D_{merge}\leftarrow\\{d_{k}|IoU(d_{m}.b,d.b)>h_{high}\\}$;

// Select detection subsets with IoU threshold $h_{low},h_{high}$

18 $D_{merge}\leftarrow\\{(d.b,d.s,d.\theta,d.w)|d\in D_{merge}\\}$;

19 $d_{f}\leftarrow$ Average($D_{merge}$, weight=$d_{m}.w$);

// Box voting in $D_{merge}$ with decayed confidence as weight

20 $d_{f}.c\leftarrow c$ ;

21 $D_{c}\leftarrow D_{c}-D_{remove}$;

// Remove overlapped boxes from $D_{c}$

22 $D_{cf}\leftarrow D_{cf}\cup\\{d_{f}\\}$;

23

24 end for

25 $D_{f}\leftarrow D_{f}\cup D_{cf}$;

26

27 end for

// Decay scores of some boxes

28 for  _$b\in D_{f}$_ do

29 if  _$b$ only fused by history boxes_ then

30 if  _Decay Strategy_ then

31 $b.s=b.w$;

32

33 end if

34 else if  _Divide Strategy_ then

35 $b.s=\frac{d_{s}b.s}{N-n_{f}}$;

36

37 end if

38

39 end if

40

41 end for

42

Algorithm 1 Frame Fusion

###  Appendix E Implementation Details

Methods in Table 1 are implemented on OpenPCDet [22]. We adopt the default
configurations for these methods to achieve their reported performances,
except for the additional velocity regression.

Next, we introduce the implementation details for CenterPoint-based motion
models, which are based on a 3D detection codebase MMDetection3D [3]. We use
CenterPoint-Voxel as the backbone. Voxel size is $(0.1,0.1,0.15)$ for Waymo
and $(0.075,0.075,0.2)$ for nuScenes. We use sweeps $[-4,0]$ for Waymo and
$[-10,0]$ for nuScenes as model input. Our models are trained with AdamW
optimizer and a cyclic learning rate scheduler with a maximum learning rate
1e-3 and a weight decay $0.01$. Train for 12 epochs on Waymo and 20 epochs on
nuScenes. We follow the CenterPoint to use random flipping, global scaling,
and ground-truth sampling for data augmentation with the same hyperparameters.
For motion parameter regression, we set the loss weight $0.2$ for velocity
parameters ($v_{x},v_{y}$ in the constant velocity model and $V$ in the
unicycle and bicycle models) and $1.0$ for angular parameters ($\omega$ in the
unicycle model, $\beta$ in the bicycle model). For bounding box NMS, we apply
circle NMS with $0.7$ threshold on Waymo and $0.2$ threshold on nuScenes. All
the models are solely trained and evaluated on the vehicle classes in Waymo
and nuScenes Dataset.

In our experiments, to evaluate the detection result on a certain GT subset
such as the turn, straight, and stationary subsets, we first match the
original detection to the GT subset and keep the bounding boxes with an IoU
over 0.5 with a GT box. The filtered result is then evaluated on the GT subset
to test its performance. In the motion state ablation, vehicles with $V<0.1$
are split to stationary. Vehicles with $R<25,V>1$ are split to turn. Others
are regarded as straight.

###  Appendix F Full Frame Visualization

We visualize the frame fusion result in a full frame to demonstrate its effect
(Fig. 7). In the top left figure, we can see most well-detected boxes are
unchanged in frame fusion (appear as a single green box). For those missed or
inaccurate detections, the frame fusion tries to fix the error and generate
new boxes (blue boxes). A few false positives are also produced in this
process. The top right figure presents how frame fusion recovers the detection
with weak observation. The LiDAR point cloud just loses in a certain range and
thus the detector misses the detections in this range. After frame fusion, the
boxes are recovered by history detections. The two bottom figures show two
complicated scenes with more vehicles.

Figure 7: Full frame visualizations for frame fusion result. Red box: GT box.
Green box: detection before fusion. Blue box: detection after fusion. Only
draw boxes with score $>0.3$ for better visual effects. Detections unchanged
in frame fusion appear as green boxes as we draw green boxes on top of blue
boxes. So focus on blue boxes to figure out the frame fusion effect.