#  DeepAccident: A Motion and Accident Prediction Benchmark for V2X Autonomous
Driving

Tianqi Wang1, Sukmin Kim1, Wenxuan Ji1, Enze Xie2,
Chongjian Ge1, Junsong Chen3, Zhenguo Li2, Ping Luo1
1The University of Hong Kong 2Huawei Noah’s Ark Lab 3Dalian University of
Technology
Project Page: https://deepaccident.github.io/

###### Abstract

Safety is the primary priority of autonomous driving. Nevertheless, no
published dataset currently supports the direct and explainable safety
evaluation for autonomous driving. In this work, we propose DeepAccident, a
large-scale dataset generated via a realistic simulator containing diverse
accident scenarios that frequently occur in real-world driving. The proposed
DeepAccident dataset contains 57K annotated frames and 285K annotated samples,
approximately 7 times more than the large-scale nuScenes dataset with 40k
annotated samples. In addition, we propose a new task, end-to-end motion and
accident prediction, based on the proposed dataset, which can be used to
directly evaluate the accident prediction ability for different autonomous
driving algorithms. Furthermore, for each scenario, we set four vehicles along
with one infrastructure to record data, thus providing diverse viewpoints for
accident scenarios and enabling V2X (vehicle-to-everything) research on
perception and prediction tasks. Finally, we present a baseline V2X model
named V2XFormer that demonstrates superior performance for motion and accident
prediction and 3D object detection compared to the single-vehicle model.

Figure 1: Illustration of our proposed end-to-end motion and accident
prediction task. In this case, the ego vehicle (vehicle # 1) follows behind
another vehicle that is about to collide with a vehicle approaching from the
right. However, the single-vehicle model fails to predict any motion on the
forward right side due to occlusion from buildings on the right side. In
contrast, the V2X model communicates with other vehicles and infrastructure,
thereby successfully predicting the motion of vehicles behind the obstructing
buildings, thus anticipating the upcoming accident. This shows the potential
of V2X methods for the prediction task. The red and green bounding boxes in
the images, respectively, represent the colliding vehicles and the other V2X
vehicles behind them.

##  1 Introduction

In recent years, single-vehicle autonomous driving has achieved significant
progress owing to the well-established datasets for autonomous driving, such
as KITTI [3], nuScenes [1], Waymo [14], etc. Using those datasets, researchers
have proposed various representative algorithms for different downstream
tasks, including perception [7, 13, 6, 8] and prediction [5, 19].

Nevertheless, single-vehicle autonomous driving suffers from performance
degradation in distant or occluded areas due to poor or partial visibility of
raw sensors. One possible solution is to seek the help of vehicle-to-
everything (V2X) communication technology which can provide a complementary
perception range or enhanced visibility for the ego vehicle. Based on the
additional information source, V2X communication can be further categorized as
vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication.
Most of the existing V2X datasets [15, 9, 17] support perception tasks but
ignore the critical motion prediction task. The only V2X dataset that supports
motion prediction is V2X-seq [18], that is recently released, but it requires
ground truth vehicle positions, map topology, and traffic light status as
inputs, which is impractical for real-world autonomous driving.

Moreover, mainstream datasets lack an essential attribute for evaluating the
safety of autonomous driving: the inclusion of safety-critical scenarios, such
as collision accidents. Existing accident datasets [12, 4, 16] suffer from
limitations such as low-resolution images captured from a single forward-
facing camera without camera parameter information. In addition, these
datasets oversimplify the accident prediction task by treating it as a
classification or 2D object detection task, which is challenging to interpret
or use for subsequent planning module in autonomous driving.

We propose the DeepAccident dataset, the first V2X autonomous driving dataset
supporting end-to-end motion and accident prediction, as well as other typical
autonomous driving perception tasks. Using the CARLA simulator [2], we
reconstructed diverse real-world driving accidents according to NHTSA pre-
crash reports [11]. For each scenario, four vehicles and one infrastructure
collected a full set of sensor data, including multi-view RGB cameras and
LiDAR, with annotations for perception and motion prediction tasks. This setup
fills the gap of a lack of safety-critical scenarios in existing V2X datasets.
Additionally, we introduce a new end-to-end accident prediction task to
predict collision accidents’ occurrence, timing, location, and involved
vehicles or pedestrians. An illustration of this task is shown in Figure 1.
Lastly, we propose a V2X model named V2XFormer, which demonstrates superior
performance compared to the single-vehicle model on the DeepAccident dataset
for both perception and prediction.

Our main contributions can be summarized in three-fold:

  1. •

We propose DeepAccident, the first dataset and benchmark for end-to-end V2X
motion and accident prediction, which contains diverse collision accidents.
Besides, it also supports the multi-agent multi-modality multi-task perception
research for other autonomous driving perception tasks.

  2. •

Based on DeepAccident, we design a new task named end-to-end accident
prediction that predicts the occurrence of collision accidents and their
specific timing, location, and vehicles or pedestrians involved.

  3. •

We propose a V2X model named V2XFormer for both perception and prediction
tasks to serve as a baseline for further research.

Dataset | Scenario | Source | Sensor | Tasks | Accident
---|---|---|---|---|---
MTV Cameras | LiDAR | Det. | Track. | Seg. | Mot.
nuScenes [1] | single | real-world | ✓ | ✓ | ✓ | ✓ | ✓ | $\triangle$ | ✗
Waymo [14] | single | real-world | ✓ | ✓ | ✓ | ✓ | ✗ | $\triangle$ | ✗
KITTI [3] | single | real-world | ✗ | ✓ | ✓ | ✓ | ✗ | $\triangle$ | ✗
OPV2V [15] | V2V | simulator | ✓ | ✓ | ✓ | ✓ | ✗ | $\triangle$ | ✗
V2X-Sim [9] | V2V&V2I | simulator | ✓ | ✓ | ✓ | ✓ | ✓ | $\triangle$ | ✗
DAIR-V2X [17] | V2I | real-world | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗
V2X-seq/Perception [18] | V2I | real-world | ✗ | ✓ | ✓ | ✓ | ✗ | $\triangle$ | ✗
V2X-Seq/Forecasting [18] | V2I | real-world | ✗ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗
VIENA2 [12] | single | simulator | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓
GTACrash [4] | single | simulator | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓
YoutubeCrash [4] | single | real-world | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓
TAD [16] | single | real-world | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓
DeepAccident (ours) | V2V&V2I | simulator | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Table 1: Comparison of existing autonomous driving datasets, including both real-world and simulated ones, to our proposed DeepAccident dataset. The Mot. task listed in Tasks represents end-to-end motion prediction, and the symbol $\triangle$ indicates that the required ground truth motion labels are not officially provided but can be obtained via manipulation of the original labels. |  KITTI [3] |  nuScenes [1] |  Waymo [14] |  OPV2V [15] |  V2X-Sim [9] |  V2X-seq [18] |  DeepAccident (ours)
---|---|---|---|---|---|---|---
# of annotated samples | 15K | 40K | 230K | 33K | 47K | 36K | 285K
# of annotated V2X frames | 0 | 0 | 0 | 11K | 10K | 18K | 57K
annotation frequency (Hz) | 1 | 2 | 10 | 10 | 5 | 10 | 10
Table 2: Scale comparison between existing autonomous driving datasets and our
proposed DeepAccident. We only list the datasets that have the potential to
support the end-to-end motion prediction task.

##  2 Related Work

V2X autonomous driving datasets for perception. The existing V2X datasets
primarily focus on perception tasks that demonstrate the improved perception
capabilities of V2X approaches. OPV2V [15] supports 3D object detection and
tracking in V2V scenarios, while V2X-Sim [9] supports more comprehensive
tasks, including BEV map semantic segmentation in both V2V and V2I scenarios.
These datasets were generated using simulators. In contrast, DAIR-V2X [17] and
V2X-seq [18] are the only publicly available real-world V2X datasets that
collect data in V2I scenarios. DAIR-V2X only supports 3D object detection,
while V2X-seq is a sequential dataset that also supports 3D object tracking.
For comparison, our proposed DeepAccident dataset includes multi-view cameras
and LiDAR sensors and supports all the existing datasets’ perception tasks.

V2X autonomous driving datasets for motion prediction. The V2X-seq dataset
[18] is currently the only available dataset that supports the V2X motion
prediction task. However, the V2X-seq dataset treats motion prediction as a
subsequent module after the perception module and takes ground truth vehicle
locations, map topology, and traffic light status as inputs. This modularized
pipeline for motion prediction ignores rich semantics embedded within the raw
sensor data and can lead to error accumulation in real-time driving scenarios
due to inaccurate perception results. Alternatively, end-to-end motion
prediction, which takes the raw sensor as input and generates the motion
prediction results, has aroused significant research interest recently [5],
[19] due to the potential to extract more semantics from the raw sensor and
the time efficiency. To support the end-to-end motion prediction task,
datasets need to provide sequential frames with consistent object IDs and
corresponding raw sensor data. Although existing datasets can be modified to
support end-to-end motion prediction, they did not include it in their
official evaluation benchmark, as it is a newly emerging task. We mark this
situation using a symbol $\triangle$ in Table 1. In contrast, our proposed
DeepAccident dataset includes end-to-end motion prediction in the benchmark
and provides code for training and evaluation.

Autonomous driving datasets for accident prediction. Currently, there is no
V2X dataset available for accident prediction, with existing datasets only
covering normal and safe scenarios. However, there are existing accident
datasets that operate within a single vehicle or single infrastructure
setting. For instance, VIENA2 and GTACrash create collisions in the GTA V
video game by manually driving or losing control and capturing single forward
camera images of the ego vehicle. VIENA2 only provides coarse classification
labels, while GTACrash provides 2D bounding boxes for collided vehicles,
treating accident prediction as a 2D dangerous vehicle detection task.
YoutubeCrash utilizes real-world collision video clips and manually annotates
the 2D bounding box for the collided vehicles, keeping the same task setting
as GTACrash. TAD is a real-world accident dataset captured from a single
surveillance camera installed on the infrastructure. Here, the task of
accident prediction is viewed as involving both classification and 2D accident
vehicle detection. However, all these existing accident datasets treat
accident prediction as a naive classification or 2D dangerous vehicle
detection task, making it hard to be interpreted or utilize by the subsequent
autonomous driving planning module. In contrast, our proposed DeepAccident
dataset provides fully detailed accident labels, such as the accident vehicle
ids and their future colliding trajectories in the V2X scenario.

Table 1 compares our DeepAccident dataset with other autonomous driving
datasets from V2X configurations, senor configurations, supported tasks, and
inclusion of accident scenarios. Meanwhile, Table 2 compares the dataset scale
of DeepAccident with other datasets. In summary, the proposed DeepAccident is
the only V2X dataset that includes accident scenarios and supports end-to-end
motion and accident prediction, as well as other perception tasks in both V2V
and V2I settings. Furthermore, DeepAccident also has the largest scale
compared to existing datasets.

Figure 2: The designed accident scenarios in DeepAccident can be divided into
12 types across signalized intersections (6 types) and unsignalized
intersections (6 types). For each scenario, we have two colliding vehicles
with overlapped planned trajectories and another two vehicles following behind
them. The designed scenarios include: (1) running against a red light at four-
way intersections, (2) left turn against a red light at four-way
intersections, (3) unprotected left turn at four-way intersections, (4) right
turn against left turn at four-way intersections, (5) right turn against left
turn at three-way intersections (6) go straight against right turn at three-
way intersections in signalized cases. In unsignalized cases, the designed
overlapping trajectories are the same, but there are no traffic lights to
affect the vehicle behaviors.

##  3 DeepAccident Dataset

###  3.1 Dataset Generation

The accident scenarios in our proposed DeepAccident dataset are designed
following the pre-crash report by NHTSA [11], in which various types of
collision accidents are reported from the real-world crash data. We design 12
types of accident scenarios in DeepAccident as shown in Figure 2. Our designed
accident scenarios generally involve two vehicles with overlapped planned
trajectories at signalized and unsignalized intersections. An accident occurs
if either vehicle runs a red light or both vehicles are allowed to proceed but
show no negotiating behavior. In addition to the two accident vehicles, we
spawn two more vehicles, each following behind one of the accident vehicles,
to capture diverse viewpoints of the same scene (See Figure 2). Furthermore, a
full set of sensors is installed on all four vehicles, and annotated labels
are saved independently. Additionally, a full stack of sensors is installed
facing toward the intersection on the infrastructure side, resulting in data
from four vehicles and one infrastructure of the same scene to support V2X
research. The detailed sensor configuration is provided in Appendix Table 8,
and an illustration example of our V2X setting is shown in Appendix Figure 7.

Accident generation details. For the two accident vehicles that we choose to
collide with each other, we first calculate the intersection point of their
planned trajectories and then set their initial positions to arrive at a
similar time by dividing the arriving distance by the randomly chosen maximum
speed. The two vehicles that are designed to follow the accident vehicles have
the same trajectories as the accident vehicles. Thus, in some scenarios, it is
the behind vehicles to have collision instead, and we treat that as a valid
scenario as well. Moreover, although not designed intentionally, there are
other accident scenarios, such as vehicles hitting the crossing pedestrians
and colliding with other non-V2X vehicles. We also include these scenarios in
the proposed DeepAccident and will design more scenarios in the later version
of this dataset. Besides, we have also designed normal scenarios in which the
vehicles also have overlapped trajectories but show negotiation behavior and
obey the traffic rule to avoid collisions, which is achieved by using the in-
built TrafficManager in CARLA to control all the vehicles and pedestrians
coordinately. Each scenario will be stopped when a collision occurs, the ego
vehicle completes its planned trajectory, or the scenario time exceeds 10
seconds.

###  3.2 Dataset Statistics

Scenario distribution. During data collection, many random factors were
involved in increasing the diversity, including the number of spawned
surrounding vehicles and pedestrians, weather, and time-of-the-day. As shown
in Figure 3, the proposed DeepAccident dataset exhibits a substantial degree
of diversity. More dataset statistics can be found in Appendix Figure 10, 11,
and 12.

Supported tasks. DeepAccident contains both multi-view camera sensors and
LiDAR sensors and provides annotations that support end-to-end motion
prediction, 3D object detection and tracking, and BEV semantic segmentation.
Additionally, we have also proposed a new task named end-to-end accident
prediction, which can evaluate the safety of autonomous algorithms directly.
These tasks can all be achieved under V2X settings in DeepAccident, thus
stimulating more V2X research.

Dataset size. As shown in Table 2, the proposed DeepAccident has the largest
scale compared to existing datasets. It comprises a total of 285k annotated
samples and 57k annotated V2X frames, providing annotations at a high
frequency of 10 Hz. In addition, we split the data with an approximate ratio
of 0.7, 0.15, and 0.15 for training validation, and testing splits, resulting
in 203k, 41k, and 41k samples respectively. A total of 691 scenarios are
assigned to the three splits, with 483, 104, and 104 scenarios for training,
validation, and testing, respectively.

##  4 End-to-End Motion and Accident Prediction

For this task, we select the camera-based setting to utilize multi-view camera
images as inputs for generating motion prediction results for the entire
scene. These motion prediction results are then post-processed to determine
the occurrence of the accident and the accident vehicle ids, accident
positions as well as timing (see Figure 1).

Figure 3: Distribution of the proposed DeepAccident dataset, including
scenario length, number of vehicles and pedestrians in each sample, accident
occurrence, weather, and time of the day.

###  4.1 Network Structure

We choose BEVerse [19] as our baseline single-vehicle model for the end-to-end
motion prediction. For the V2X setting, we propose a simple yet effective V2X
model named V2XFormer due to using SwinTransformer [10] as the image feature
backbone. As shown in Figure 4, V2XFormer shares the same BEV feature
extractor as the single-vehicle model such that each V2X agent would extract a
BEV feature centered at its own coordinate system. These BEV features are then
spatially wrapped to the ego vehicle coordinate system to concatenate with the
ego vehicle BEV feature. For the BEV feature fusion part, we use a simple
average fusion strategy for the aligned BEV features to generate a BEV feature
with the same number of channels as the ego vehicle BEV feature. This averaged
BEV feature is then fed into the task heads to generate the motion prediction
and 3D object detection results. In addition, given the fact that the ego
vehicle itself can cause an accident with other vehicles or pedestrians, we
also require the network to jointly predict the ego vehicle’s future motion.

###  4.2 Accident Prediction

Post-processing for accident prediction. We propose the end-to-end accident
prediction task, which can be achieved via post-processing the motion
prediction results. We view the performance on this task as safety metrics for
autonomous driving. From the motion prediction results, which consist of
several BEV outputs, including centerness, segmentation, offset, and future
flow, we can combine them to get the BEV instance segmentation results like
the ones shown in Figure 1 for the current moment as well as the future
period. For each timestamp, we can approximate the BEV segmentation results
for each object as polygons and then find the polygons with the closest
distance and store the object ids and positions to represent the accident
candidates for this timestamp. By looking for the timestamp with the closest
object distance, we determine whether an accident occurred and provide details
regarding the colliding object ids and positions at that particular moment. In
our experiment, we set a threshold for a dangerous distance of 2.5 meters.

Accident prediction accuracy. To evaluate the accuracy of accident prediction
compared to ground truth accident information, the same post-process steps are
applied to the ground truth motion to determine the occurrence of future
accidents. A true positive prediction is when : $(i)$ both the prediction and
ground truth indicate the occurrence of an accident, and $(ii)$ the total
position difference of the colliding agents between the prediction and the
ground truth is less than a threshold. Based on this, we propose a new
evaluation metric named Accident Prediction Accuracy (APA) in Equation 1 where
we calculate the average accident prediction accuracy over a set of position
difference thresholds of $\mathbb{D}=$ {5,10,15} meters:

$\mathrm{APA}=\frac{1}{\mathbb{D}}\sum_{d\in\mathbb{D}}\frac{|TP|_{d}}{|TP|_{d}+\frac{1}{2}|FP|_{d}+\frac{1}{2}|FN|_{d}}$
(1)

True Positive metrics. In addition to the APA, we calculate several _True
Positive_ metrics (TP metrics) for each true positive accident prediction
sample to provide more detailed interpretation of the performance. This
includes the error terms for _IDs_ , _positions_ and _time_ between the ground
truth accident and the predicted accident. For TP metrics calculation, we set
the position difference threshold to 10 meters when deciding the true positive
predictions. As for ID error, if the predicted accident objects’ ids are the
same as the ground truth’s, then it equals to zero, otherwise equals to one.
For position and time error, we present them using their native units
(_meters_ and _seconds_)) and calculate the absolute difference compared to
the ground truth. For each TP metric, we calculate the average value over true
positive predictions as indicated by Equation 2.

$\mathrm{mTP}=\frac{1}{|TP|}\sum_{i=1}^{|TP|}\mathrm{TP_{err}}$ (2)

Figure 4: Network details of the proposed V2XFormer. We use the
three-V2X-agent setting consisting of ego AV, AV, and Infra for illustration.
V2X agents in V2XFormer utilize a shared-weight BEV extractor to extract BEV
features based on multi-view camera observation history within the previous N
frames. These features are spatially wrapped and aligned with the BEV features
from the ego vehicle before being concatenated along the channel dimension.
For the V2X fusion part, here we utilize a simple yet effective average
pooling over channel dimension to generate the aggregated BEV feature, which
is then fed into different task heads to get prediction results.

Number of future motion predictions. In reality, there could be multiple
future motions given the same past states. As a result, both FIERY [5], and
BEVerse [19] learn a Gaussian distribution map consisting of a mean map and
covariance map over the BEV feature. Thus, by sampling from the learned
Gaussian distribution map, we can obtain different BEV features. This enables
the motion head to produce diverse motion prediction results.

##  5 Experiment

Evaluated tasks. To show the usefulness of our proposed DeepAccident dataset
as a V2X motion and accident prediction benchmark, we focus on the end-to-end
motion and accident prediction task and choose the camera-based setting. In
addition, we train another 3D object detection head with the motion head to
simultaneously compare the perception ability between the V2X models and the
single-vehicle model.

Experiment settings. We use the settings for motion prediction in BEVerse [19]
and FIERY [5] as our default experiment settings. The model predicts 2 seconds
into the future, considering 3 frames of past temporal context and 4 frames
into the future at 2Hz. We choose BEVerse-tiny as the single-vehicle model and
initialize all the V2X models and the single-vehicle model with pretrained
weights on nuScenes. For training, we train the models on the training split
of DeepAccident for 20 epochs. As for evaluation, we randomly sample five BEV
features from the learned motion Gaussian distribution as we mentioned in
section 4.2 along with the mean of this learned distribution to generate six
different motion prediction results. Only the motion prediction result
obtained from the mean vector of the learned Gaussian distribution is used to
assess motion prediction performance. For accident prediction, we consider a
prediction indicating the occurrence of an accident when any of the sampled
motion predictions is analyzed to cause a collision accident, prioritizing
safety.

We report the performance on DeepAccident’s validation split in the following
sections and include results on the testing split in Appendix Table 7. We
compare the overall performance of V2X models with different agent
configurations to the single-vehicle model and provide further ablation
analysis that considers time-to-collision, accident visibility, and longer
prediction horizon settings. Additionally, we conduct experiments on nuScenes
to validate the trained models’ real-world generalization ability.

###  5.1 V2X models for prediction and perception

Evaluation metrics. We use mIOU and VPQ proposed in FIERY [5] for the motion
prediction task, our proposed APA (Accident Prediction Accuracy) and id error,
position error for accident prediction task, and detection mAP averaged over
center distance matching thresholds of {1,2,4} meters for 3D object detection
task.

Config | Motion | Accident | mAP
---|---|---|---
mIOU($\scriptstyle\uparrow$) |  VPQ($\scriptstyle\uparrow$) |  APA($\scriptstyle\uparrow$) |  id err($\scriptstyle\downarrow$) |  pos err($\scriptstyle\downarrow$) |  ($\scriptstyle\uparrow$)
Single vehicle | 43.8 | 31.6 | 61.9 | 0.12 | 3.20 | 26.5
\+ behind vehicle | 48.3 | 35.7 | 65.7 | 0.10 | 3.03 | 33.1
\+ other vehicle | 49.5 | 37.0 | 65.1 | 0.09 | 2.99 | 34.5
\+ infrastructure | 50.0 | 37.4 | 65.0 | 0.11 | 3.12 | 32.9
3 vehicles | 50.7 | 38.1 | 65.0 | 0.09 | 2.87 | 35.5
4 vehicles | 51.2 | 38.7 | 66.6 | 0.08 | 2.91 | 35.7
4 vehicles+infra | 52.1 | 39.5 | 67.1 | 0.08 | 2.88 | 36.2
Table 3: Performance comparison between the single-vehicle model and different v2x configuration models on the validation split of DeepAccident. Config | 2s | 1.5s | 1s | 0.5s
---|---|---|---|---
mIoU | APA | mIoU | APA | mIoU | APA | mIoU | APA
Single vehicle | 38.3 | 28.7 | 43.1 | 49.1 | 48.0 | 74.7 | 52.1 | 81.1
\+ behind vehicle |  41.6 (+3.4) |  31.4 (+2.7) |  47.6 (+4.5) |  48.2 (-0.9) |  53.2 (+5.2) |  80.8 (+6.1) |  58.1 (+6.1) |  85.5 (+4.4)
\+ other vehicle |  43.5 (+5.2) |  30.5 (+1.8) |  49.0 (+5.9) |  53.3 (+4.2) |  54.2 (+6.2) |  78.5 (+3.8) |  58.5 (+6.4) |  83.7 (+2.6)
\+ infrastructure |  44.2 (+5.9) |  30.2 (+1.5) |  49.3 (+6.2) |  50.4 (+1.3) |  54.5 (+6.5) |  79.0 (+4.3) |  59.3 (+7.2) |  85.0 (+3.9)
3 vehicles |  44.1 (+5.9) |  30.5 (+1.8) |  50.1 (+7.1) |  49.6 (+0.9) |  55.6 (+7.6) |  79.0 (+4.3) |  60.3 (+8.2) |  85.3 (+4.2)
4 vehicles |  44.5 (+6.2) |  31.5 (+2.8) |  50.5 (+7.5) |  53.1 (+4.0) |  56.1 (+8.1) |  79.5 (+4.8) |  61.3 (+9.2) |  86.7 (+5.6)
4 vehicle + infra |  45.6 (+7.3) |  28.2 (-0.5) |  51.4 (+8.3) |  53.6 (+4.5) |  56.9 (+9.0) |  81.2 (+6.5) |  62.3 (+10.2) |  87.6 (+6.5)
Table 4: Performance comparison between the single-vehicle model and different
v2x configuration models for motion mIoU and accident prediction accuracy
(APA) at different Time-To-Collision (TTC).

Overall performance. V2X models significantly outperform the single-vehicle
baseline in all three tasks, as shown in Table 3. The V2X model with four
vehicles and infrastructure exhibit better performance than the single-vehicle
model, with an increase of 8.3, 5.2, and 9.7 in mIOU, APA, and detection mAP,
respectively. V2X communication with the vehicle on the other side
demonstrates better performance in motion prediction and 3D object detection
tasks, while showing similar performance in accident prediction, as compared
to V2X communication with the vehicle behind the ego vehicle. V2X-infra yields
the best motion prediction performance due to the broad visibility provided by
the infrastructure’s relatively high sensor mounting position. However, it
exhibits slightly lower performance on the detection and accident prediction
tasks compared to V2X-other or V2X-behind, possibly due to the simple V2X
fusion design of our proposed V2XFormer. Nevertheless, V2X-infra still
outperforms the single-vehicle model across all tasks. Finally, gradual
incorporation of more V2X agents for communication can lead to gradual
improvement in performance across all tasks.

Prediction horizon | all data | 1s | 2s | 3s | 4s
---|---|---|---|---|---
2s | 61.9 | 74.7 | 28.7 | none | none
3s | 50.5 | 71.5 | 25.7 | 21.2 | none
4s | 35.4 | 56.3 | 20.4 | 14.6 | 10.2
Table 5: Performance of single-vehicle models with different prediction
horizon settings at different Time-To-Collision (TTC) for accident prediction
accuracy.

Performance based on Time-To-Collision. We divide the evaluation data based on
Time-To-Collision (TTC) into 2s, 1.5s, 1s, and 0.5s prior to collision, using
our prediction horizon of two seconds and a 0.5s frame gap. As we lack ground
truth labels for the period after collision, we only assess prediction results
in the valid future time period. According to Table 4, The V2X models
outperform the single-vehicle model for motion and accident prediction,
especially when TTC is shorter. For example, the V2X-5agents model outperforms
the single-vehicle model by 9.0 and 6.5 for mIOU and APA, respectively, on the
1s TTC data. However, for the 2s TTC data, the accident prediction task is
challenging for all models, with the V2X-5agents model even performing worse
than the single-vehicle model. This may be attributed to the simple fusion
design for V2X models and suggests an area for future improvement.

Figure 5: Performance comparison between the single-vehicle model and
different v2x configuration models for motion mIoU and accident prediction
accuracy (APA) _v.s._ accident visibility.

Performance based on accident vehicle visibility. During the observation
period, accident vehicles or pedestrians may be temporarily or consistently
invisible from the ego vehicle’s perspective, making it challenging to predict
their motion with a single-vehicle model and hindering accident prediction. To
address this, we evaluate the performance of different V2X models and a
single-vehicle model by dividing the evaluation data based on accident vehicle
or pedestrian visibility from the ego vehicle side. We define a sample with
over half of its observation frames having invisible accident vehicles as an
invisible sample for accidents. Figure 5 shows that the performance gap
between V2X models and the single-vehicle model is significantly larger when
there is limited accident visibility from the ego vehicle side, for both
motion prediction and accident prediction tasks. Specifically, V2X-5agent
model (4 vehicles + infra) outperforms the single-vehicle model by 12.0 and
8.1 higher mIOU and APA, respectively, for invisible accident scenarios, while
the gap is only 7.8 and 5.0 in terms of mIOU and APA for visible accident
scenarios.

Figure 6: A qualitative result where the ego vehicle is going uphill while the
ahead vehicle will collide with the pedestrian that is crossing the road. The
crossing pedestrian is invisible to the ego vehicle due to the uphill terrain.
In this case, the infrastructure provides clear visibility for the colliding
vehicle and pedestrian, thus successfully predicting the accidents for
V2X-infra and V2X-5agent model. The red and green bounding boxes in the
images, respectively, represent the colliding vehicles or pedestrians and the
other V2X vehicles.

Longer prediction horizon. We also conduct experiments on predicting longer
future motion and choose the single-vehicle model as the baseline. As shown in
Table 5, predicting longer future motion achieves worse accident prediction
accuracy compared to the model with a shorter prediction horizon. For example,
the model predicting 4s future achieves almost half the APA of the 2s-setting
model on all validation data, achieving 35.4 and 61.9, respectively. Moreover,
the models with longer future prediction horizon settings still perform worse
when evaluating only the common future period of time (20.4, 25.7, 28.7 for
4s, 3s, and 2s prediction horizon settings on 2s time-to-collision data).
However, the 4s-setting model achieves an APA of 10.2 for samples 4s prior to
collision, while other models are unable to predict the accident this early
due to their design. These results suggest a trade-off between predicting
longer future horizons and achieving satisfactory overall performance.

Qualitative results. Figure 6 shows an example where the crossing pedestrian
is invisible to the ego vehicle due to the uphill terrain. As a result, the
single-vehicle model is unable to detect the pedestrian and fails to predict
the upcoming accident. In contrast, the infrastructure provides complementary
visibility for the colliding vehicle and pedestrian, allowing the V2X-infra
and V2X-5agents models to accurately anticipate the accident.

Training data | VPQ | mAP
---|---|---
nuScenes only | 33.4 | 32.1
DeepAccident + nuScenes | 34.2(+0.8) | 34.0(+1.9)
Table 6: Performance comparison between the original BEVerse-tiny [19] model
and the model trained with both the synthesized DeepAccident data and the
real-world nuScenes data for motion prediction (VPQ) and 3D object detection
(mAP).

###  5.2 Sim2Real Domain Adaptation

To validate the real-world generalization ability of the models trained with
our proposed DeepAccident dataset, we fine-tune the trained single vehicle
model on nuScenes for five epochs and compare it with the original BEVerse-
tiny model that is only trained on nuScenes for both motion prediction and 3D
object detection tasks. As shown in Table 6, the model trained with both
datasets achieves 1.9 higher mAP and 0.8 higher VPQ on the nuScenes validation
dataset, which demonstrates the usefulness of our proposed DeepAccident
dataset for real-world scenarios.

##  6 Conclusion

We propose DeepAccident, the first large-scale V2X autonomous driving dataset
that includes various collision accident scenarios commonly encountered in
real-world driving. Based on this dataset, we introduce the end-to-end motion
and accident prediction task and corresponding metrics to assess the accuracy
of accident prediction. DeepAccident contains sensor data and annotation
labels from four vehicles and one infrastructure for each scenario, allowing
for the V2X research for perception and prediction. Our proposed V2XFormer
outperforms the single-vehicle model in both perception and prediction tasks,
providing a baseline for future research. The proposed DeepAccident serves as
a direct safety benchmark for autonomous driving algorithms and as a
supplementary dataset for enhancing perception methods’ generalization ability
in both single-vehicle and V2X settings.

Limitations. At present, we have not considered communication latency,
bandwidth limitations, and estimation errors in the relative poses of V2X
agents, all of which are inevitable in real-world scenarios, when transferring
messages between V2X agents. Our V2XFormer model is a starting point to
demonstrate the potential of V2X perception and prediction. We encourage more
V2X methods to be developed to overcome these simplifications by using the
DeepAccident dataset.

Acknowledgement. We gratefully acknowledge the support of MindSpore, CANN
(Compute Architecture for Neural Networks) and Ascend AI Processor used for
this research.

## References

  * [1] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020.
  * [2] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun.  Carla: An open urban driving simulator. arxiv 2017.  arXiv preprint arXiv:1711.03938, 2017.
  * [3] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  The International Journal of Robotics Research, 32(11):1231–1237, 2013.
  * [4] Kim Hoon, Lee Kangwook, Hwang Gyeongjo, and Suh Changho.  Crash to not crash: Learn to identify dangerous vehicles using a simulator.  In Proceedings of the AAAI Conference on Artificial Intelligence, page 978–985, 2019.
  * [5] Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, and Alex Kendall.  Fiery: future instance prediction in bird’s-eye view from surround monocular cameras.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15273–15282, 2021.
  * [6] Junjie Huang, Guan Huang, Zheng Zhu, and Dalong Du.  Bevdet: High-performance multi-camera 3d object detection in bird-eye-view.  arXiv preprint arXiv:2112.11790, 2021.
  * [7] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom.  Pointpillars: Fast encoders for object detection from point clouds.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12697–12705, 2019.
  * [8] Yangguang Li, Bin Huang, Zeren Chen, Yufeng Cui, Feng Liang, Mingzhu Shen, Fenggang Liu, Enze Xie, Lu Sheng, Wanli Ouyang, et al.  Fast-bev: A fast and strong bird’s-eye view perception baseline.  arXiv preprint arXiv:2301.12511, 2023.
  * [9] Yiming Li, Dekun Ma, Ziyan An, Zixun Wang, Yiqi Zhong, Siheng Chen, and Chen Feng.  V2x-sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving.  IEEE Robotics and Automation Letters, 7(4):10914–10921, 2022.
  * [10] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021.
  * [11] Wassim G Najm, John D Smith, Mikio Yanagisawa, et al.  Pre-crash scenario typology for crash avoidance research.  Technical report, United States. National Highway Traffic Safety Administration, 2007.
  * [12] Mohammad Sadegh Aliakbarian, Fatemeh Sadat Saleh, Mathieu Salzmann, Basura Fernando, Lars Petersson, and Lars Andersson.  Viena2: A driving anticipation dataset.  arXiv e-prints, pages arXiv–1810, 2018.
  * [13] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li.  Pv-rcnn: Point-voxel feature set abstraction for 3d object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10529–10538, 2020.
  * [14] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al.  Scalability in perception for autonomous driving: Waymo open dataset.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2446–2454, 2020.
  * [15] Runsheng Xu, Hao Xiang, Xin Xia, Xu Han, Jinlong Li, and Jiaqi Ma.  Opv2v: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication.  In 2022 International Conference on Robotics and Automation (ICRA), pages 2583–2589. IEEE, 2022.
  * [16] Yajun Xu, Chuwen Huang, Yibing Nan, and Shiguo Lian.  Tad: A large-scale benchmark for traffic accidents detection from video surveillance.  arXiv preprint arXiv:2209.12386, 2022.
  * [17] Haibao Yu, Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang, Yifeng Shi, Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, and Zaiqing Nie.  Dair-v2x: A large-scale dataset for vehicle-infrastructure cooperative 3d object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21361–21370, 2022.
  * [18] Haibao Yu, Wenxian Yang, Hongzhi Ruan, Zhenwei Yang, Yingjuan Tang, Xu Gao, Xin Hao, Yifeng Shi, Yifeng Pan, Ning Sun, Juan Song, Jirui Yuan, Ping Luo, and Zaiqing Nie.  V2x-seq: A large-scale sequential dataset for vehicle-infrastructure cooperative perception and forecasting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  * [19] Yunpeng Zhang, Zheng Zhu, Wenzhao Zheng, Junjie Huang, Guan Huang, Jie Zhou, and Jiwen Lu.  Beverse: Unified perception and prediction in birds-eye-view for vision-centric autonomous driving.  arXiv preprint arXiv:2205.09743, 2022.

##  Appendix A Ablation Experiments

Performance comparison on val and test split. In the main paper, we evaluated
all experiment results on the validation split of DeepAccident. Table 7
presents a performance comparison of various trained models on both the
validation and testing splits. The table shows that the models perform
similarly on both splits, with consistent relative performance gaps between
models. For example, the V2X model with four vehicles and one infrastructure
outperforms the single-vehicle model by 7.9 and 8.8 higher VPQ on the
validation and testing splits, achieving 39.5 and 39.3 VPQ, respectively.
These results confirm the validity of the split in DeepAccident.

Qualitative results. Figures 8 and 9 present additional qualitative results
comparing the V2X-5agents model and the single-vehicle model. These figures
demonstrate the improved prediction and perception capabilities of the V2X
model over the single-vehicle baseline, particularly in scenarios where the
ego vehicle’s visibility is reduced due to occlusion, weather, or lighting
conditions.

Figure 7: The V2X setting in the proposed DeepAccident dataset. We have four vehicles and one infrastructure to capture the full set of sensor data for each scenario. Vehicle # 1 and vehicle # 2 are designed to collide with each other, and vehicle # 3 and vehicle # 4 are the V2X vehicles that we designed to follow behind those two accident vehicles. Moreover, the views from the infrastructure are also collected, which often have little occlusion and have better visibility. The images shown here are captured from the cameras mounted behind the vehicles. These images are only for V2X illustration purposes and differ from the captured camera images in the proposed DeepAccident dataset. Figure 8: Qualitative comparison between the single-vehicle model and the V2X-5agents model on 3D object detection and motion prediction tasks. The red circles on motion figures indicate the front right area behind a wall such that the single-vehicle model failed to predict objects and their motions in that area while the V2X-5agents model succeeded. Figure 9: Qualitative comparison between the single-vehicle model and the V2X-5agents model across various scenarios on end-to-end motion and accident detection tasks. The areas in the motion prediction results marked by red circles indicate the happening place of accidents. The V2X-5agents model shows superiority over the single-vehicle model for both motion and accident prediction. Config | Validation | Testing
---|---|---
VPQ | APA | mAP | VPQ | APA | mAP
Single vehicle | 31.6 | 61.9 | 26.5 | 30.5 | 61.9 | 25.8
\+ behind vehicle | 35.7 | 65.7 | 33.1 | 34.9 | 64.8 | 32.4
\+ other vehicle | 37.0 | 65.1 | 34.6 | 36.4 | 65.3 | 33.3
\+ infrastructure | 37.4 | 65.0 | 32.9 | 36.7 | 64.8 | 32.9
\+ behind & other vehicle | 38.1 | 65.0 | 35.5 | 37.7 | 66.6 | 35.3
4 vehicles | 38.7 | 66.6 | 35.7 | 38.3 | 66.6 | 35.3
4 vehicles + infra | 39.5 | 67.1 | 36.2 | 39.3 | 66.8 | 35.8
Table 7: Performance comparison between single-vehicle model and different v2x
configuration models on validation and testing split of DeepAccident.

##  Appendix B The DeepAccident Dataset

###  B.1 Scenario Design

V2X setting illustration. As illustrated in Figure 7, for each scenario in our
proposed DeepAccident dataset, we have four vehicles and one infrastructure to
collect sensor data, thus providing diverse viewpoints and enabling V2X
research.

###  B.2 Dataset Statistics

Dataset size & split. While the simulator-based nature of the dataset could
have enabled the generation of an unlimited amount of data, we chose to
include data on 691 scenarios to maintain a manageable dataset size of roughly
310 GB. In addition, our proposed DeepAccident dataset provides six types of
annotated objects, including car, van, truck, motorcycle, cyclist, and
pedestrian, and supports numerous perception and prediction tasks. Moreover,
we have meticulously divided the DeepAccident dataset into training,
validation, and testing splits to make sure all of them contain enough diverse
scenarios, such as scenarios with collision accidents and adverse weather and
lighting conditions. With a total of 691 scenarios, we allocated the splits
with an approximate ratio of 0.7, 0.15, and 0.15 for training, validation, and
testing. As such, the training, validation, and testing splits consist of 483,
104, and 104 scenarios, respectively. As a result, the 285k annotated samples
were distributed with 203k, 41k, and 41k samples assigned to the training,
validation, and testing splits correspondingly.

Ego vehicle statistics. Figure 10 shows the ego vehicle statistics of the
DeepAccident dataset, including the driving speed and size dimensions. The
driving speed of ego vehicle covers a wide range, including high-speed driving
cases. Additionally, the ego vehicle size also varies significantly from small
cars to huge trucks to offer more viewpoint diversity.

Figure 10: Ego vehicle statistics of the proposed DeepAccident dataset, including the driving speed and size dimensions. Sensor |  Description
---|---
Vehicles |  For each vehicle: • Six rgb cameras with 1600 x 900 resolution. $70^{\circ}$ horizontal FoV for F/FL/FR/BL/BR cameras. $110^{\circ}$ horizontal FoV for the Back camera. • One 32-lane LiDAR mounted on top with 70 meter range, 30° vertical FoV, and 10 Hz rotation frequency.
Infrastructure |  • Same senor configurations as the vehicles - six rgb cameras and one LiDAR. • Sensors installed on a high position randomly from 3 to 5 meters.
V2X communication |  • Recorded relative poses between different V2X agents to align the information from different sources.
Table 8: Sensor configurations in DeepAccident. As for the V2X setting, we set
four vehicles and one infrastructure to record data for each scenario.

Instance-level statistics. Figure 11 and 12 provide instance-level statistics
of the DeepAccident dataset. These figures demonstrate that all six object
classes in DeepAccident have substantial amounts of annotated data, while the
vehicle classes (car, van, and truck) exhibit a wide range of sizes. Moreover,
the speed and distance of surrounding vehicles from the ego vehicle also
exhibit significant variation. These results show the instance-level diversity
of the proposed DeepAccident dataset.

Figure 11: Instance-level statistics, including the number of annotated
instances for different classes and the size distribution of vehicle classes
(car, van, truck). Each class has a considerable amount of training data, and
the vehicles annotated in our DeepAccident dataset show great diversity in
their sizes. Figure 12: Instance-level vehicle statistics include (a) their
distance to the ego vehicle and (b) their velocity. Both figures show the
diversity of the vehicles in the DeepAccident dataset. Moreover, the
distributions on training, validation, and testing splits show remarkable
consistency with the overall data, which shows the validity of our dataset
split. Figure 13: Visualization for the collected multi-view camera images and
LiDAR point clouds from the vehicle’s and the infrastructure’s perspectives
when the collision happens. Different box colors indicate different classes.

Dataset visualization. Additionally, we provide visualization for the
collected camera and LiDAR data in Figure 13 and some showcase accident
scenarios captured by infrastructure in Figure 14.

Figure 14: Some showcase accident scenarios in our proposed DeepAccident
across different road types, weathers, and time-of-day conditions. All these
multi-view camera images are captured from the infrastructure side to better
illustrate the scenes. The red bounding boxes in each scenario represent the
colliding vehicles, and the green bounding boxes represent the additional V2X
vehicles that also collect data.