11institutetext: Institute of Cartography and Geoinformatics Leibniz
University Hannover, Germany 11email<EMAIL_ADDRESS>
# StreamLTS: Query-based Temporal-Spatial LiDAR Fusion for Cooperative Object
Detection

Yunshuang Yuan 11 0000-0001-5511-9082 Monika Sester 11 0000-0002-6656-8809

###### Abstract

Cooperative perception via communication among intelligent traffic agents has
great potential to improve the safety of autonomous driving. However, limited
communication bandwidth, localization errors and asynchronized capturing time
of sensor data, all introduce difficulties to the data fusion of different
agents. To some extend, previous works have attempted to reduce the shared
data size, mitigate the spatial feature misalignment caused by localization
errors and communication delay. However, none of them have considered the
asynchronized sensor ticking times, which can lead to dynamic object
misplacement of more than one meter during data fusion. In this work, we
propose Time-Aligned COoperative Object Detection (TA-COOD), for which we
adapt widely used dataset OPV2V and DairV2X with considering asynchronous
LiDAR sensor ticking times and build an efficient fully sparse framework with
modeling the temporal information of individual objects with query-based
techniques. The experiment results confirmed the superior efficiency of our
fully sparse framework compared to the state-of-the-art dense models. More
importantly, they show that the point-wise observation timestamps of the
dynamic objects are crucial for accurate modeling the object temporal context
and the predictability of their time-related locations.

###### Keywords:

Cooperative Perception Point Cloud Data Fusion

##  1 Introduction

(a) OPV2Vt

(b) DairV2Xt

Figure 1: TA-COOD: The scan points (blue to red scaled colors) and objects are
observed at different timestamps, leading to unaligned object ground-truth
(yellow boxes). The bounding boxes at the scan end of this frame are taken as
the final ground-truth for TA-COOD (green boxes).

As the economy expands, the demand for mobility grows substantially,
amplifying the complexity of transportation within constrained time and space.
This surge poses significant challenges to developing effective traffic
solutions without compromising safety and comfort. Autonomous Vehicles (AVs)
emerge as one potential solution to improve road safety and efficiency.
Perceiving the driving environment with the on-board sensors is crucial for
autonomous driving. However, the perception performance is limited by the
perceiving range of sensors and occlusions caused by ego-view sensor
placement.

In recent years, cooperative perception with sharing information among
Intelligent Agents (IAs), such as Connected Autonomous Vehicles (CAVs) and
Connected Infrastructures (CIs), has proven to be beneficial for improving the
perception accuracy and road safety[4, 26, 34, 22, 30]. Different from the
ego-perception system that only observes the driving environment from the ego
vehicle’s view, cooperative perception collect information from the sensors
mounted on different IAs. These sensors typically have better spatial
arrangement, can efficiently mitigate the occlusions and increase the
perception distance of the ego vehicle. This ensures safer driving and more
optimized route planning of AVs. However, sharing and fusing information from
different IAs introduces several challenges, including the limited
communication bandwidth and information misalignment introduced by
localization errors and asynchronized capturing time of sensor data.

Previous works have sought to minimize bandwidth usage by compressing the
learned deep Bird’s Eye View (BEV) features [22, 4, 26] or selecting deep
keypoint features [34], rectify localization errors through learning-based
techniques [22, 25] or analytic algorithms [34, 33], and synchronize
communication delays with the attention mechanism [24] or explicitly
predicting the future features before data fusion [31]. Nonetheless, none have
considered the asynchronized sensor ticking time.

More specifically, previous benchmarks [26, 30, 24, 31] assume that the
sensors have synchronized global ticking time. In reality, these sensors might
have asynchronous ticking time in each aligned frame, leading to inhomogeneous
observation time of each object in the scenario. This effect can be amplified
when the rolling shutter effect (_e.g_. mechanic LiDARs) exists. At the sensor
capturing frequency of 10Hz, these time differences ($0~{}\text{to}~{}100ms$),
that are in the same magnitude as the communication delay, can lead to large
spatial displacement at data fusion, especially when the CAVs have high
speeds. Therefore, they should not be ignored. As the two example scenes shown
in Fig. 1, we leverage the timestamps of each point (illustrated with
differently colored points) in the point cloud to achieve globally Time-
Aligned COoperative Object Detection (TA-COOD) that predicts object bounding
boxes (BBoxes) for the unified global timestamp (green boxes) in stead of
detecting the timely asynchronous local bounding boxes (yellow boxes).

To model more robust temporal features, we utilize the information from the
previous frames. This is very computational-demanding as the data from several
IAs should be processed. Most of the previous works [26, 22, 28, 25, 8] use
dense convolutions or transformers to process BEV feature maps, that consume a
lot of GPU memory and are hard to extend the model to sequential data on
limited computational resources. In contrast, the potential of fully sparse
network structures has rarely been researched. In this work, we built a fully
sparse framework that can be trained on a single Nvidia GTX 3090 with
sufficient batch size (_e.g_. 4) and processing sequential LiDAR streaming
data of more than 4 frames and up to 7 CAVs.

We use fully sparse convolutions to encode the point cloud data and only
select the keypoints in the Region of Interest (RoI) for further processing.
Inspired by the query-based object detection [23, 16, 21], we regard the RoI
keypoints as queries of transformer input and follow StreamPETR [21] to model
the temporal data fusion. To reduce the communication overhead, we fuse the
temporal data before sharing to other IAs. More specifically, the selected
local query points interact with both the historical query points in the
previous frames and the keypoint features in the current frame to aggregate
the temporal and spatial features for the ego-based time-aligned object
detection. These aggregated query features are then shared and spatially fused
by a sparse attention module for the TA-COOD task. Since our framework fuses
both Temporal and Spatial information from Streaming LiDAR data of IAs, we
call it StreamLTS.

To evaluate StreamLTS, we conduct experiments on the dataset DairV2X [30] with
adaptions to the TA-COOD problem based on the provided point-wise timestamps
for each point cloud. Only with the cooperation between a vehicle and an
infrastructure, this dataset has limited dynamics of sensors. Therefore, we
simulate a new dataset with more CAVS and larger dynamics based on the
scenario replay files of the dataset OPV2V [26]. Both datasets take ground-
truth at global-aligned timestamp. Owning to the detailed temporal features,
we notate the two datasets as OPV2Vt and DairV2Xt, respectively.

Our contributions are threefold. First, We proposed a TA-COOD that handles
asynchronous data and aims to generate global aligned BBoxes for objects. To
this end, we adapted the OPV2V and DairV2X datasets with considering the
asynchronous ticking time of sensors to evaluate the models for TA-COOD.
Second, We developed a highly efficient and fully sparse framework streamLTS
that fuses both temporal and spatial information from multiple IAs and
generate the TA-COOD results. Third, Our experimental results show that the
sub-frame observation timestamps of the obtained data are crucial to improve
the positioning accuracy of the detected objects at the given global aligned
timestamp.

##  2 Related Work

###  2.1 Cooperative Object Detection

Object detection is crucial for autonomous driving to avoid collisions. By
exchanging perceived objects, [7] proved that sharing information among IAs
significantly increases the detection accuracy compared to the ego-based
perception. Instead, [3, 17, 26, 22] compared the performance of sharing and
fusing data of different processing stages, including raw, semi- and fully-
processed data fusion, all show that semi-processed data fusion has the
potential to achieve the best performance with limited and controllable
communication resource consumption.

However, semi-processed data can be in any format. For instance, Fcooper [4]
experimented with both voxel feature and spatial deep feature map fusion,
found that voxel feature fusion has better performance as it keeps more
details. Instead, [26] encodes the point clouds into point-pillars [11] with
fully-connected layers and uses dense convolutions to generate feature maps.
These feature maps from different IAs are then fused with an attention module
instead of max-out used in Fcooper. These learnable encoding and fusion
modules ensure that the most important information is selected and shared to
achieve better performance. To further increase the detection accuracy while
spatial misalignment between the feature maps of IAs exists, SCOPE [27]
introduced the pyramid LSTM [12] to reason about temporal and spatial
information from sequential point clouds.

While these works all take feature maps as their fusion strategy, [10] and
[35] utilize the learned criterion, such as confidence and uncertainty, to
select the most relevant information and reduce the data size for sharing.
Similarly, [34] adopt the sparsity of point clouds and learn the most
important object keypoints for sharing to further reduce the bandwidth
requirement. They all achieve good performances on the OPV2V [26] dataset with
significantly less data sharing.

Except [35], all above mentioned works encode point clouds into BEV feature
maps and process them with dense convolutions or transformers which are very
computationally demanding. The efficiency of the sparse operations are rarely
explored. In this work, we design a fully sparse and training-efficient
framework to model the spatio-temporal information from streaming LiDAR data.

###  2.2 Query-based Temporal Data Fusion

Based on transformer, DETR [2] first introduced the query-based object
detection for 2D images. Since it is anchor-free and needs no post-processing,
it has been widely used and was extended to 3D object detection [23, 15, 16].
More importantly, the object queries provide great convenience to efficiently
interact with features of any modality. For instance, by integrating the 3D
geometry information into the image memory features, PETR [15] built a simple
yet effective framework to let 3D queries directly interact with the 2D
features in image space and obtained superior 3D object detection performance.
By propagating the learned object queries to the next frames and model the
temporal context for the current frame, QueryProp [9] improved the efficiency
and accuracy for the video object detection. With only images as input,
StreamPETR [21] achieved on-par 3D object detection performance to the LiDAR-
track benchmark on the NuScenes [1] dataset. Further more, the concept of
object query propagation has been applied to object tracking tasks. MOTR [36]
and TrackFormer [18] use object queries to achieve temporal modeling and
associate the objects between frames in an end-to-end manner, which is simple
yet effective on tracking. Inspired by these works, we model the temporal
information of observations from different IAs with the selected potential
object queries, and share the learned queries for TA-COOD to save
communication bandwidth.

###  2.3 Datasets for Collective Perception

It is highly expensive and complicated to obtain training data for cooperative
object detection. It requires several vehicles and sensors observing the same
scene. This scale leads to highly complex calibration and post-processing to
generate accurate ground-truth meta information (_e.g_. sensor poses) and
annotations. Therefore, the initial attempts for collective perception either
use simplified driving scenarios [3, 4] or synthetic data [26, 32, 13]. For
instance, [4] conducted their cooperative perception experiments on a dataset
captured from a static parking lot with static sensors. This is the simplest
set up without considering any dynamics of sensors or objects in the scenario.
Via simulation, [26] generated a large-scale dataset for collective perception
and built a cooperative object detection benchmark, however, without
considering sensor asynchrony.

Recently, the real datasets DAIR-V2X [30] and V2V4Real [24] were made
accessible. Both are only configured with two agents, DAIR-V2X with one CAV
and one connected infrastructure, V2V4Real with two CAVs. Because of the
dynamics and asynchrony of sensors, there might be spatial misalignment
between dynamic objects observed by the two agents. To generate unified
ground-truth BBoxes for cooperative object perception, both DAIR-V2X and
V2V4Real take ego-vehicle’s annotation as the ground-truth if the annotations
from two agents spatially overlap. However, at the blind spot of the ego-
vehicle, the annotations of the cooperative agent are taken. In this way, the
final generated bounding box might contain errors and does not reflect the
true location of the encapsulated object at the given timestamp. This might
also negatively influence the temporal predictability of sequential models. To
better preserve the detailed temporal information and its relation to the
accurate location of the objects in the scenario, we proposed to generate
global time-aligned ground-truth BBoxes for the TA-COOD task.

##  3 Methodology

###  3.1 Definition of Time-Aligned Cooperative Object Detection

In a cooperative perception scene, assume that a set of IAs
$\textbf{A}=\\{A_{i}|0\leq i\leq N\\}$ are inter-connected, where $A_{0}$ is
the ego IA. Each IA has its own sensor ticking time $t_{i}$. All sensors are
scanning at the frequency $f$. An example with two CAVs is shown in Fig. 2.
The LiDAR of the ego IA $A_{0}$ (green) and the cooperative IA $A_{1}$ (red)
are scanning counterclockwise. They have the ticking frequency of $f=10Hz$ and
the ticking time offset of $t_{0}-t_{1}=0.05s$. The scanning time of each
observation point is shown with gradient colors. As shown in the middle-bottom
area of Fig. 2, the observation by the ego- and the coop-vehicle in this area
have a time difference of $0.11s\leq t_{ego}-t_{coop}\leq 0.15s$. At the speed
of $60km/h$, this time offset can lead to the target object shift of more than
$1.8m$, introducing difficulty into the data fusion of the IAs. Instead of
taking the annotation from one IA (_e.g_. the red BBox observed by the ego IA
at $t_{ego}$), we generate the ground-truth BBox at the globally aligned time
$t_{aligned}$ by interpolating between the annotations of the individual CAVs.
To accurately model the movement of the detected object, we define that the 3D
TA-COOD aims to take point-wise timestamped point clouds as input and predict
the bounding box at globally aligned time $t_{aligned}$ which is the scan end
of the reference vehicle in each frame. Since we evaluate the detection result
in the perspective of ego vehicle, therefore we take ego vehicle as reference.
For the example in Fig. 2, the global time is aligned to $t_{aligned}=0.15s$,
the scan end of the ego-vehicle.

Figure 2: TA-COOD with two CAVs. Dashed boxes: ground-truth BBox of ego- and
coop-CAV observations. Solid box: time-aligned ground-truth BBox.

###  3.2 Overall Framework

The overall framework of processing one frame data in a sequence is shown in
left part of Fig. 3. Given the point clouds
$PCD_{0}^{t_{j}},\dots,PCD_{N}^{t_{j}}$ from $N$ IAs at time $t_{j}$, a shared
MinkUnet [5, 35] encodes them into multi-resolution sparse features, including
the local feature at higher resolution $\tilde{P}^{l}$ and the global feature
at lower resolution $\tilde{P}^{g}$. They are then dilated by DilationConv
module, separately, to extend their coordinate coverage. The dilated features
$P^{l}$ and $P^{g}$ are then fed to the corresponding RoI Head to generate
importance scores which are used for selecting the top-$K_{roi}$ RoI points
$P^{l}_{RoI}$ and $P^{g}_{RoI}$. The following shared TempFusion module is
composed with a Transformer decoder which takes $P^{l}_{RoI}$ as new object
queries to interact with the stored historical queries from $T$ previous
frames to obtain temporal context and with the global feature $P^{g}_{RoI}$ to
obtain spatial context at the current frame. On the one hand, the outputs
$Q_{0}^{t_{j}},\dots,Q_{N}^{t_{j}}$ of TempFusion are fed to a local query-
based detection head LQDet to obtain the detection result at each IA. Based on
the detection scores, only the top-$K_{q}$ object queries are pushed into the
memory queue while the queries at the oldest timestamp $t_{j-T}$ are
discarded. On the other hand, these outputs are shared to the ego IAs for
Spatial Fusion to obtain the temporal- and spatial-aligned features $Q^{ts}$.
Based on these features, the global detection head GQDet generates the TA-COOD
result after data fusion.

Figure 3: Left: Overall framework for TA-COOD. Local Data Process encodes
local point cloud features and fuse the temporal context into the object
queries. Data Fusion Process fuses the spatial object query features from the
$N$ IAs. Right: MTA alignment. Motion-Time-Aware embeds the query positions
($Q_{p}$) and Motion-Aware embeds the query context $Q_{c}$ and initial
queries ($tgt$).

###  3.3 Module Configuration

Backbone: MinkUnet and DilationConv

MinkUnet is composed by fully sparse convolution layers which is highly
efficient to encode sequential point cloud data. The point clouds are
voxelized with the size $0.4m$ and embedded to voxel features with a MLP
layer. Then the 3D sparse convolution layers are stacked in the Unet [19]
structure to encode the voxel features. The encoded features are then
compressed along the height dimension by 3D sparse convolution layers into
sparse 2D BEV features. We take the BEV feature at down-sample level 2 ($0.8m$
per pixel) as $\tilde{P}^{l}$ and level 8 ($3.2m$ per pixel) as
$\tilde{P}^{g}$. DilationConv has three sparse convolution layers with
coordinates expansion and kernel size 3. At each layer, the coordinates of
sparse tensors are dilated for one pixel.

Heads: RoI and QDet

RoI Head aims to select the most interesting features for further processing.
It uses linear layers to generate importance scores. For $P^{g}$, we expect
that the importance of input features should be learned automatically by
TempFusion. Therefore, no loss for this head. For $P^{l}$, we use a center-
based [29] detection head which uses the Focal loss [14] of classification to
learn importance scores and the Smooth L1 loss of regression to help the
backbone network to learn the geometric features of the objects. The Query-
guided Detection head (QDet) heads, including the Local QDet (LQDet) in
TempFusion and the Global QDet (GQDet) in SpatialFusion, have the same
structure as the RoI Head for $P^{l}$. They generate TA-COOD results with
respect to the query positions and features.

TempFusion

Figure 4: The details of the TempFusion and SpatialFusion module

TempFusion is built based on the StreamPETR [21]. The detailed workflow of two
consecutive frames is shown in Fig. 4. It contains four components: Memory
Queue, Motion-Time-Aware (MTA), Hybrid Attention and Cross Attention. The
Memory Queue stores the top-$K_{q}$ interesting object queries from each of
the $T$ historical frames. At the aligned time $t_{j}$, the input feature
$P^{l}_{RoI}$ is extended by the propagated top-$K_{q}$ memory object queries
at time $t_{j-1}$. The Hybrid Attention uses these queries to interact with
all queries from the current frame $t_{j}$ and the memory queue of frame
$t_{j-1}\dots t_{j-T}$ to learn temporal context of each object. However,
before this attention, all query, key and value features are spatially and
temporally aligned to the current pose and time using MTA. The output of the
Hybrid Attention is then used as queries of the Cross Attention to interact
with the global feature $P^{g}_{RoI}$ to obtain spatial context in the current
frame. The output queries $Q_{dec}$ are added with the initial RoI features
$P^{l}_{RoI}$ to enhance the local context in the current frame. The summation
$Q_{i}^{t_{j}}$ are then fed to the LQDet head to generate the time-aligned
objects locally at each IA. Based on the detection scores, the top-$K_{q}$
object queries are then pushed into the memory queue for the next iteration of
temporal fusion at time $t_{j+1}$ as shown in the middle of Fig. 4.

Memory queue: stores $K_{q}$ object queries for each of the $T$ previous
frames. For each selected object query, the predicted object center $Q_{p}$,
the context embedding $Q_{c}$, the velocity $v$, the pose matrix $E$ and the
query timestamp $\tau$ of the corresponding IA are stored. The timestamp
$\tau$ is retrieved following Eq. 1, where $Q_{x}$ and $Q_{y}$ are the $x$ and
$y$ coordinate of the query point, $P_{x}^{i}$, $P_{y}^{i}$ and $P_{t}^{i}$
are respectively the $x,y$ coordinates and the timestamp of the $i$th point
$P^{i}$ in the point cloud.

$\tau=P_{t}^{i},\quad
i=\operatorname*{arg\,min}|\operatorname{atan2}(Q_{y},Q_{x})-\operatorname{atan2}(P_{y}^{i},P_{x}^{i})|$
(1)

MTA: is shown in the right of Fig. 3. The timestamp $\tau$ of each query, the
pose $E$ and the velocity $v$ of the corresponding ego-vehicle, are aligned as
a vector and embedded to the feature $M^{emb}$. With the Motion-aware Layer
Normalization (MLN [21]), this feature is then injected into the embedded
query position $Q_{p}^{emb}$, the query context $Q_{c}^{emb}$ of previous
frames, and the initial query target $tgt$ at the current frame to obtain the
motion-aware embeddings. Additionally, the embedded time is added to the
motion-aware query position embedding ($Q_{p}^{mo}$) to make it also time-
aware ($Q_{p}^{mo,\tau}$).

Difference to StreamPETR: we made several changes to match the new data
modality. Instead of frame-wise timestamp used in StreamPETR, we leverage the
observation timestamp of each individual object query to capture the accurate
temporal context. Due slow convergence of the StreamPETR on our datasets, we
made three changes to accelerate training process. First, we take the
top-$K_{RoI}$ weighted keypoints $P^{l}_{RoI}$ as the initial object queries
instead of the random reference points. Second, we additionally add the
initial RoI features $P^{l}_{RoI}$ to the decoded queries $Q_{dec}$ to enhance
the local context so that the detection head can converge faster. Third, the
DETR[2] detection head of StreamPETR is replaced by a center-based [29]
detection head since the initial RoI queries are already good proposal center
points for the BBox regression task. Only a refinement is required based on
these center points. Additionally, we only use one decode layer in the
transformer due to limited training resource.

Spatial Fusion

After each iteration of TempFusion, the SpatialFusion module fuses the local
object queries $Q_{i}^{t}$ from different IAs . As the example of three IAs
($A_{0},A_{1},A_{2}$) shown in the purple block of Fig. 4, the input query
points $Q_{i}^{t}$ are firstly spatially aligned by taking the union of all
query locations to obtain the same number of queries at each IA. The query
features at each padded location (gray cubes in Fig. 4) are initialized with
zeros (Eq. 2). These queries are then used for the global TA-COOD.

$\displaystyle\breve{P}_{i}^{t}=\bigcup_{i=0}^{N}P_{i}^{t},\quad\breve{Q}_{i}^{t}=Padzero(Q_{i}^{t},\breve{P}_{i}^{t}-{P}_{i}^{t})$
(2) $\displaystyle Q=\breve{Q}_{0}^{t},\quad
K=V=Cat(\breve{Q}_{0}^{t},...,\breve{Q}_{N}^{t})$ (3) $\displaystyle
Q^{ts}=softmax(\frac{(Q\cdot K^{T})}{\sqrt{d}})\cdot V$ (4)

##  4 Experiment

###  4.1 Datasets

To evaluate the our proposed framework on the TA-COOD task, we generate two
new datasets OPV2Vt and DairV2Xt based on the widely-used dataset OPV2V [26]
and DairV2X [30], which are curated specifically for cooperative object
detection that does not consider the sub-frame time asychrony of sensors.

OPV2Vt (Fig. 1(a)): The orignal OPV2V dataset is generated by the simulator
CARLA [6]. It contains about 12K frames of data in 73 scenes. We interpolate
the meta data of OPV2V to generate ten intermediate locations for each object
and sensor inside one frame, which leads to ten sub-frames (0.01s) in each
scan frame (0.1s) of LiDARs. We then replay all sub-frames to generate data
with more fine-grained dynamics in each scan frame. Since these sub-frames are
synchronized, we randomly drop the data of several beginning sub-frames (1 to
5) for each scan sequence and joint every consecutive 10 sub-frames as a full
LiDAR scan to simulate the ticking time asynchrony. For each jointed frame
data, we use the BBoxes of the last sub-frame as ground-truth. One frame of
data is shown in Fig. 1(a). The evaluation detection range is set to
$[-140.8,140.8]m$, $[-38.4,38.4]m$, and $[-3.0,1.0]m$ for x-, y- and z-axis,
respectively.

DairV2Xt (Fig. 1(b)): The real dataset DairV2X is captured with one CAV and
one CI at several intersections. To generate the globally time-aligned BBoxes,
we first refine the point cloud alignment between the CAV and the CI to
decouple the bounding box misalignment caused by localization errors and
asynchronized capturing times. In each sequence, we select the frame where CAV
is closest to CI as the global reference coordinate $O_{ref}$, and then
register sequentially the point clouds of CAV of other frames to $O_{ref}$.
During registration, dynamic points are moved via the BBox annotations. Based
on the registration, we obtain a global merged and down-sampled point cloud,
to which the CI point clouds are then registered to obtain the relative
location to $O_{ref}$. With the corrected sensor poses in each frame, the
annotated objects are transformed from the local sensor coordinate to the
globally aligned coordinate $O_{ref}$ and frame-wisely aligned to the object
trajectories. By interpolation, we generate the ground-truth BBoxes that are
aligned to the scan end $t_{aligned}$ in each frame. The detection range at y-
and z-axis is the same as OPV2Vt. The range for x-axis is set to
$[-102.4,102.4]m$.

###  4.2 Experiment Setting

Comparative experiment: We select three state-of-the-art (SOTA) cooperative
object detection frameworks, Fcooper [4], AttnFusion [26] and FPVRCNN [34], to
conduct comparative experiments with our StreamLTS. To make a fair comparison,
we attach our TempFusion module to each framework to model the temporal
interaction. For Fcooper and AttnFusion, we use the intermediate BEV features
at the corresponding resolution as $P^{l}$ and $P^{g}$. FPVRCNN uses smaller
input voxel size, leading to intermediate BEV features of higher resolutions.
Therefore, we use two additional convolution layers to down-sample the BEV
features to obtain the global features $P^{g}$. For the local features, we
directly use keypoints generated by the Voxel-Set-Abstraction [20]. At
SpatialFusion stage, we use Maxout [4] fusion for Fcooper and Attention for
the other two frameworks. We replace the dense convolutions of AttnFusion [26]
with sparse convolutions to reduce memory usage.

Training: We use the sliding window strategy to load the sequential data with
4 frames. For each training batch, we only calculate the loss for the newest
frame to reduce the training time and GPU memory consumption. From each frame,
we select $1024$ top-$k$ RoI keypoints from $P^{l}$ and $512$ top-$k$ global
features from $P^{g}$ for TempFusion, in which $K=256$ of the fused object
queries are propagated to the next frame. To improve efficiency, we only
calculate gradients for the ego-agent in the Local Data Process (Fig. 3) of
OPV2Vt dataset. Gradients are calculated for both CAV and CI of DairV2Xt as
they have different view angles, hence different spatial features. We augment
both datasets by transforming the point clouds with a rotation angle in the
range of $\pm\alpha^{\circ}$, flipping along x- and y-axis and scaling with
the ratio in range $[0.95,1.05]$. $\alpha$ is set to $90^{\circ}$.
Additionally, we follow [35] to augment the input point cloud with free space
points for StreamLTS to enhance the spatial connectivity of sparse tensors. To
make the model also able to handle the large frame-wise time latency, we load
cooperative data from a random previous frame
($t\in\\{t_{j},t_{j-1},t_{j-2}\\}$) instead of the aligned frame $t_{j}$. In
general, we train all models for $50$ epochs with the Adam optimizer, batch
size four for OPV2Vt and two for DairV2Xt. The learning rate is set to $2e-4$
with a warmup stage of $4000$ iterations.

##  5 Experimental Results and Evaluation

###  5.1 Qualitative Results

(a) OPV2Vt

(b) DairV2Xt

Figure 5: TA-COOD result of StreamLTS. Yellow BBoxes: ego-view annotation.
Green BBoxes: time-aligned annotation. Red BBoxes: time-aligned detection.

TA-COOD result of StreamLTS is shown in Fig. 5. The yellow BBoxes are the
ground-truth BBoxes at different observation times, the green BBoxes are the
globally time-aligned ground-truth. As shown in the middle dashed red box of
Fig. 5(a), the objects in this area are observed by both CAVs at different
times. The top right CAV scans this area (blue points) about $0.05s$ earlier
than the bottom left CAV (yellow points). The global BBoxes are aligned to the
scan end in this frame (same to the timestamps of the red points), hence moved
a little bit forward in comparison to the yellow BBoxes. This can also be
observed in Fig. 5(b). The red BBoxes are the predictions generated by
StreamLTS. The result on the OPV2Vt and DairV2Xt datasets show that the
predictions have better matches to the globally time-aligned ground-truth
BBoxes (green) than any locally annotated BBoxes (yellow). This reveals that
our framework has successfully captured the time relationships between the
object observations from different IAs and correctly predicted the BBoxes at
the aligned future timestamp.

###  5.2 Comparison to SOTA Models

Table 1: Average Precision (AP) of Time-aligned cooperative object detection. Framework | Backbone | Fusion | OPV2Vt | DairV2Xt
---|---|---|---|---
<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
Fcooper | PointPillar | Maxout | 54.4 | 17.0 | 41.8 | 17.7
FPVRCNN | Spconv | Attn | 70.8 | 41.2 | 51.8 | 23.9
AttnFusion | VoxelNet | Attn | 78.7 | 41.4 | 62.1 | 34.0
StreamLTS | MinkUnet | Attn | 85.3 | 72.1 | 64.0 | 40.4
Figure 6: Comparison of memory usage peak and epochal training time.

Average Precision (AP): As shown in Tab. 1, our framework StreamLTS achieves
the highest AP on both datasets. Especially at the IoU threshold of $0.7$, the
AP of our framework has increased $30.7\%$ on OPV2Vt and $6.4\%$ on DairV2Xt
compared to the second-best framework AttnFusion. Among all comparative
frameworks, Fcooper has the worst performance as it uses a simpler backbone
network that only consists of a PointPillar encoder and several 2D convolution
layers, to encode the point cloud features. More importantly, its Maxout
operation for fusing the features from different IAs is not learnable, hence
not able to handle the spatial misalignment of the object queries caused by
different observation times. In comparison, all other three frameworks with
enhanced backbone encoding and the learnable attention mechanism for multi-IA
feature fusion have much better performance. Because of the high reliability
on the accuracy of the first stage detection, FPVRCNN might ignore some
potential object points for the next processing steps, hence achieves worse
performance than AttnFusion and StreamLTS.

Training efficiency: Less memory consumption and training time is crucial for
faster model development on limited computing resources. To investigate the
training efficiency of StreamLTS, we compare it to other models with the peak
memory usage because it is crucial for determining the devices the model can
be trained on and the training batch size on limited GPU resource. In
addition, we also tested the epochal training time of all comparative models
on the same device. The results of AP against these two measures are
demonstrated in Fig. 6. As shown in the left two sub-plots, our proposed
StreamLTS (squares) requires the least training memory resource and has the
best overall TA-COOD performance. AttnFusion has a comparable detection
performance, however, its GPU memory demand is more than $50\%$ higher than
StreamLTS. Regarding the epochal training time, the sparse models StreamLTS
and FPVRCNN seem less efficient than the other two dense models. This is
because the sparse operations are less parallelized for computation. However,
thanks to the low memory usage, StreamLTS can reduce the training time by
increasing the batch size.

Figure 7: TA-COOD result of StreamLTS with and without RoI regression Table 2: Ablation study on individual modules of streamLTS. Exp. | Pt. | Temp. | Dil. | RoI | Global | OPV2Vt | DairV2Xt
---|---|---|---|---|---|---|---
Time | Fusion | Conv | reg. | attn |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
1 |  |  | ✓ | ✓ | ✓ | 77.4 | 53.3 | 55.4 | 34.0
2 |  | ✓ | ✓ | ✓ | ✓ | 73.4 | 38.7 | 56.4 | 36.7
3 | ✓ | ✓ |  | ✓ | ✓ | 79.3 | 59.6 | 54.9 | 29.9
4 | ✓ | ✓ | ✓ |  | ✓ | 83.6 | 69.7 | 60.2 | 34.4
5 | ✓ | ✓ | ✓ | ✓ |  | 85.1 | 71.0 | 63.8 | 41.0
6 | ✓ | ✓ | ✓ | ✓ | ✓ | 85.3 | 72.1 | 64.0 | 40.4

###  5.3 Ablation Study

To validate the effect of individual modules on the performance, we conduct
ablation study over the main modules of streamLTS. The results are shown in
Tab. 2. Experiment 1 and 2 validate the effectiveness of time-related modules
on capturing the temporal context to predict more accurate BBoxes. However,
using TempFusion by replacing the point-wise time embedding of object queries
with the unified frame-wise timestamps (Exp. 2) performs even worse than the
model without this module (Exp. 1) on OPV2Vt. This reveals that the temporal
modeling with inaccurate frame timestamps might be misleading. With the
accurate point-wise timestamp (Exp. 6), the AP is nearly doubled. This
confirms the effectiveness and importance of utilizing the accurate point-wise
timestamps on modeling the accurate temporal context of the object
observations. Experiment 3 confirms that expanding the coordinates of sparse
tensors is beneficial. Without the regression sub-head in the RoI head (Exp.
4), the performance drops validate that regressing the BBox targets in early
stage is beneficial on capturing the object geometry. This can be observed in
the dashed red boxes of Fig. 7 on both datasets. At last, Experiment 5 shows
that the cross attention with the global RoI features can also slightly
improve the performance.

In general, the AP deviations among all ablation experiments of DairV2Xt are
smaller than that of OPV2Vt, especially for the ablation with and without
point-wise timestamp. We explain this from two aspects. First, the inaccurate
ground-truth (see objects in Fig. 7 indicated by black arrows) of DairV2Xt
weakens the general performance and lead to inaccurate evaluation. As the
example shown in Fig. 7, the full StreamLTS model has detected the unlabeled
objects while the model without RoI regression has not. The wrong ground-truth
will give better evaluation to the later case. Second, one sensor of DairV2xt
is static and all data is captured at intersections where most vehicles are
static waiting for the traffic lights. This leads to less dynamics compared to
the multi CAVs scenarios of OPV2Vt which are captured under different road
situations. As a result, less dynamic leads to less influence of inaccurate
timestamps.

In cooperative perception, the shared data from cooperative vehicles might be
outdated because of the long data processing and transmitting time. Owning to
the temporal modeling with accurate point-wise timestamps, streamLTS is able
to reduce the influence of this time latency. Tab. 3 shows the ablation
results of fusing the data with frame-wise time latency up to 200ms. Without
TempFusion, the APs drops significantly as the latency increases. However,
temporal modeling with the inaccurate frame-wise timestamps (No Pt. Time) is
not helpful and even misleading. Last two rows in Tab. 3 show the streamLTS
performance without and with latency augmentation during training. Both
reduced the influence of the latency. However, the full streamLTS with train
latency has the best performance, validating the effectiveness of our model on
understanding the temporal context. Similar to Tab. 2, the performance gain of
point-wise temporal modeling on DairV2Xt is significantly smaller than on the
OPV2Vt because of less dynamics and inaccurate ground-truth of DairV2Xt.

Table 3: Average Precision (AP) of TA-COOD with frame-wise time latency. Dataset | OPV2Vt | DairV2Xt
---|---|---
Latency | 0ms | 100ms | 200ms | 0ms | 100ms | 200ms
AP threshold | 0.5 | 0.7 | 0.5 | 0.7 | 0.5 | 0.7 | 0.5 | 0.7 | 0.5 | 0.7 | 0.5 | 0.7
No TempFusion | 79.2 | 56.1 | 74.0 | 40.9 | 60.8 | 34.0 | 58.6 | 38.9 | 54.3 | 35.1 | 51.2 | 33.1
No Pt. Time | 73.4 | 38.7 | 69.4 | 32.7 | 65.4 | 33.7 | 59.0 | 33.5 | 57.8 | 33.2 | 57.1 | 32.6
No Train Latency | 83.2 | 64.0 | 76.4 | 59.0 | 74.4 | 57.9 | 59.3 | 36.2 | 55.2 | 33.3 | 53.0 | 32.3
Full steamLTS | 85.3 | 72.1 | 81.6 | 68.0 | 78.7 | 64.7 | 64.2 | 40.4 | 61.3 | 37.9 | 59.0 | 36.4

##  6 Conclusion

In this paper, we proposed an efficient fully sparse and query-based temporal
and spatial fusion framework, StreamLTS, to fuse the asynchronized data for
cooperative object detection. With experiments on the OPV2Vt and DairV2Xt
datasets that are specially prepared for TA-COOD, we found that temporal
modeling with fine-grained point-wise timestamps can capture the temporal
context more accurately and improve the predictability of the object movement.
Comparing to the baseline models, our fully sparse framework has shown its
high efficiency during training. For future work, we hope our findings could
be inspiring for more efficient and accurate cooperative end-to-end trajectory
prediction.

## References

  * [1] Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., Beijbom, O.: nuscenes: A multimodal dataset for autonomous driving. arXiv preprint arXiv:1903.11027 (2019)
  * [2] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I. p. 213–229. Springer-Verlag, Berlin, Heidelberg (2020). https://doi.org/10.1007/978-3-030-58452-8_13, https://doi.org/10.1007/978-3-030-58452-8_13
  * [3] Chen, Q., Tang, S., Yang, Q., Fu, S.: Cooper: Cooperative perception for connected autonomous vehicles based on 3d point clouds. 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS) pp. 514–524 (2019)
  * [4] Chen, Q.: F-cooper: feature based cooperative perception for autonomous vehicle edge computing system using 3d point clouds. Proceedings of the 4th ACM/IEEE Symposium on Edge Computing (2019), https://api.semanticscholar.org/CorpusID:202577722
  * [5] Choy, C., Gwak, J., Savarese, S.: 4d spatio-temporal convnets: Minkowski convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3075–3084 (2019)
  * [6] Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: An open urban driving simulator. In: Proceedings of the 1st Annual Conference on Robot Learning. pp. 1–16 (2017)
  * [7] Günther, H.J., Mennenga, B., Trauer, O., Riebl, R., Wolf, L.: Realizing collective perception in a vehicle. In: 2016 IEEE Vehicular Networking Conference (VNC). pp. 1–8 (2016). https://doi.org/10.1109/VNC.2016.7835930
  * [8] He, C., Wang, H., Chen, L., Luo, T., Cai, Y.: V2x-ahd:vehicle-to-everything cooperation perception via asymmetric heterogenous distillation network. arXiv arXiv:2310.06603 (Oct 2023). https://doi.org/10.48550/arXiv.2310.06603, http://arxiv.org/abs/2310.06603, arXiv:2310.06603 [cs]
  * [9] He, F., Gao, N., Jia, J., Zhao, X., Huang, K.: Queryprop: Object query propagation for high-performance video object detection. Proceedings of the AAAI Conference on Artificial Intelligence 36, 834–842 (06 2022). https://doi.org/10.1609/aaai.v36i1.19965
  * [10] Hu, Y., Fang, S., Lei, Z., Zhong, Y., Chen, S.: Where2comm: Communication-efficient collaborative perception via spatial confidence maps. In: Thirty-sixth Conference on Neural Information Processing Systems (Neurips) (November 2022)
  * [11] Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang, J., Beijbom, O.: Pointpillars: Fast encoders for object detection from point clouds. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 12689–12697. IEEE, Long Beach, CA, USA (Jun 2019). https://doi.org/10.1109/CVPR.2019.01298, https://ieeexplore.ieee.org/document/8954311/
  * [12] Lei, Z., Ren, S., Hu, Y., Zhang, W., Chen, S.: Latency-aware collaborative perception. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision – ECCV 2022. p. 316–332. Springer Nature Switzerland, Cham (2022)
  * [13] Li, Y., Ma, D., An, Z., Wang, Z., Zhong, Y., Chen, S., Feng, C.: V2x-sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving. IEEE Robotics and Automation Letters 7(4), 10914–10921 (Oct 2022). https://doi.org/10.1109/LRA.2022.3192802
  * [14] Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 42(2), 318–327 (2020). https://doi.org/10.1109/TPAMI.2018.2858826
  * [15] Liu, Y., Wang, T., Zhang, X., Sun, J.: Petr: Position embedding transformation for multi-view 3d object detection. arXiv preprint arXiv:2203.05625 (2022)
  * [16] Liu, Y., Yan, J., Jia, F., Li, S., Gao, Q., Wang, T., Zhang, X., Sun, J.: Petrv2: A unified framework for 3d perception from multi-camera images. arXiv preprint arXiv:2206.01256 (2022)
  * [17] Marvasti, E.E., Raftari, A., Marvasti, A.E., Fallah, Y.P., Guo, R., Lu, H.: Cooperative lidar object detection via feature sharing in deep networks. 2020 IEEE 92nd Vehicular Technology Conference (VTC2020-Fall) pp. 1–7 (2020), https://api.semanticscholar.org/CorpusID:211205062
  * [18] Meinhardt, T., Kirillov, A., Leal-Taixe, L., Feichtenhofer, C.: Trackformer: Multi-object tracking with transformers. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2022)
  * [19] Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234–241. Springer (2015)
  * [20] Shi, S., Guo, C., Jiang, L., Wang, Z., Shi, J., Wang, X., Li, H.: Pv-rcnn: Point-voxel feature set abstraction for 3d object detection. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 10526–10535 (2019), https://api.semanticscholar.org/CorpusID:209516193
  * [21] Wang, S., Liu, Y.H., Wang, T., Li, Y., Zhang, X.: Exploring object-centric temporal modeling for efficient multi-view 3d object detection. 2023 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 3598–3608 (2023), https://api.semanticscholar.org/CorpusID:257636991
  * [22] Wang, T.H., Manivasagam, S., Liang, M., Yang, B., Zeng, W., Urtasun, R.: V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Prediction, Lecture Notes in Computer Science, vol. 12347, p. 605–621. Springer International Publishing, Cham (2020). https://doi.org/10.1007/978-3-030-58536-5_36, https://link.springer.com/10.1007/978-3-030-58536-5_36
  * [23] Wang, Y., Guizilini, V., Zhang, T., Wang, Y., Zhao, H., , Solomon, J.M.: Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In: The Conference on Robot Learning (CoRL) (2021)
  * [24] Xu, R., Xia, X., Li, J., Li, H., Zhang, S., Tu, Z., Meng, Z., Xiang, H., Dong, X., Song, R., Yu, H., Zhou, B., Ma, J.: V2v4real: A real-world large-scale dataset for vehicle-to-vehicle cooperative perception. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 13712–13722. IEEE, Vancouver, BC, Canada (Jun 2023). https://doi.org/10.1109/CVPR52729.2023.01318, https://ieeexplore.ieee.org/document/10203124/
  * [25] Xu, R., Xiang, H., Tu, Z., Xia, X., Yang, M.H., Ma, J.: V2x-vit: Vehicle-to-everything cooperative perception with vision transformer. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision – ECCV 2022. p. 107–124. Lecture Notes in Computer Science, Springer Nature Switzerland, Cham (2022). https://doi.org/10.1007/978-3-031-19842-7_7
  * [26] Xu, R., Xiang, H., Xia, X., Han, X., Li, J., Ma, J.: Opv2v: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication. 2022 International Conference on Robotics and Automation (ICRA) p. 2583–2589 (May 2022). https://doi.org/10.1109/ICRA46639.2022.9812038
  * [27] Yang, K., Yang, D., Zhang, J., Li, M., Liu, Y., Liu, J., Wang, H., Sun, P., Song, L.: Spatio-temporal domain awareness for multi-agent collaborative perception. In: 2023 IEEE/CVF International Conference on Computer Vision (ICCV). p. 23383–23392 (2023), https://openaccess.thecvf.com/content/ICCV2023/html/Yang_Spatio-Temporal_Domain_Awareness_for_Multi-Agent_Collaborative_Perception_ICCV_2023_paper.html
  * [28] Yin, H., Tian, D., Lin, C., Duan, X., Zhou, J., Zhao, D., Cao, D.: V2vformer ++: Multi-modal vehicle-to-vehicle cooperative perception via global-local transformer. IEEE Transactions on Intelligent Transportation Systems p. 1–14 (2023). https://doi.org/10.1109/TITS.2023.3314919
  * [29] Yin, T., Zhou, X., Krähenbühl, P.: Center-based 3d object detection and tracking. CVPR (2021)
  * [30] Yu, H., Luo, Y., Shu, M., Huo, Y., Yang, Z., Shi, Y., Guo, Z., Li, H., Hu, X., Yuan, J., Nie, Z.: Dair-v2x: A large-scale dataset for vehicle-infrastructure cooperative 3d object detection. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 21329–21338. IEEE, New Orleans, LA, USA (Jun 2022). https://doi.org/10.1109/CVPR52688.2022.02067, https://ieeexplore.ieee.org/document/9879243/
  * [31] Yu, H., Yang, W., Ruan, H., Yang, Z., Tang, Y., Gao, X., Hao, X., Shi, Y., Pan, Y., Sun, N., Song, J., Yuan, J., Luo, P., Nie, Z.: V2x-seq: A large-scale sequential dataset for vehicle-infrastructure cooperative perception and forecasting. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 5486–5495. IEEE, Vancouver, BC, Canada (Jun 2023). https://doi.org/10.1109/CVPR52729.2023.00531, https://ieeexplore.ieee.org/document/10204878/
  * [32] Yuan, Y., Sester, M.: Comap: A synthetic dataset for collective multi-agent perception of autonomous driving. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLIII-B2-2021, 255–263 (Jun 2021). https://doi.org/10.5194/isprs-archives-XLIII-B2-2021-255-2021
  * [33] Yuan, Y., Sester, M.: Leveraging dynamic objects for relative localization correction in a connected autonomous vehicle network. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences V-1–2022, 101–109 (May 2022). https://doi.org/10.5194/isprs-annals-V-1-2022-101-2022
  * [34] Yuan, Y., Cheng, H., Sester, M.: Keypoints-based deep feature fusion for cooperative vehicle detection of autonomous driving. IEEE Robotics and Automation Letters 7(2), 3054–3061 (2022). https://doi.org/10.1109/LRA.2022.3143299
  * [35] Yuan, Y., Cheng, H., Yang, M.Y., Sester, M.: Generating evidential bev maps in continuous driving space. ISPRS Journal of Photogrammetry and Remote Sensing 204, 27–41 (2023). https://doi.org/https://doi.org/10.1016/j.isprsjprs.2023.08.013, https://www.sciencedirect.com/science/article/pii/S0924271623002290
  * [36] Zeng, F., Dong, B., Zhang, Y., Wang, T., Zhang, X., Wei, Y.: Motr: End-to-end multiple-object tracking with transformer. In: European Conference on Computer Vision (ECCV) (2022)