WGMwgm QEqe EPep PMSpms BECbec DEde

Conceptualization, Methodology, Software, Data curation, and Writing -
original draft

Data curation, and Methodology

Data curation, Methodology, and Writing - original draft

Data curation, and Methodology

Methodology, and Supervision

Investigation, Supervision, and Formal analysis

1]organization=School of Intelligent Systems Engineering, Sun Yat-sen
University, city=Shenzhen, postcode=518107, country=China

[1] Validation, Funding acquisition, and Supervision [cor1]Corresponding
author

2]organization=School of Intelligent Systems Engineering, Sun Yat-sen
University, Shenzhen, 518107 & Southern Marine Science and Engineering
Guangdong Laboratory (Zhuhai), Zhuhai, 519082, China

# InScope: A New Real-world 3D Infrastructure-side Collaborative Perception
Dataset for Open Traffic Scenarios

Xiaofei Zhang<EMAIL_ADDRESS>Yining Li<EMAIL_ADDRESS>Jinping Wang<EMAIL_ADDRESS>Xiangyi Qin<EMAIL_ADDRESS>Ying Shen<EMAIL_ADDRESS>Zhengping Fan<EMAIL_ADDRESS>[
Xiaojun Tan<EMAIL_ADDRESS>[

###### Abstract

Perception systems of autonomous vehicles are susceptible to occlusion,
especially when examined from a vehicle-centric perspective. Such occlusion
can lead to overlooked object detections, e.g.,larger vehicles such as trucks
or buses may create blind spots where cyclists or pedestrians could be
obscured, accentuating the safety concerns associated with such perception
system limitations. To mitigate these challenges, the vehicle-to-everything
(V2X) paradigm suggests employing an infrastructure-side perception system
(IPS) to complement autonomous vehicles with a broader perceptual scope.
Nevertheless, the scarcity of real-world 3D infrastructure-side datasets
constrains the advancement of V2X technologies. To bridge these gaps, this
paper introduces a new 3D infrastructure-side collaborative perception
dataset, abbreviated as InScope. Notably, InScope is the first dataset
dedicated to addressing occlusion challenges by strategically deploying
multiple-position Light Detection and Ranging (LiDAR) systems on the
infrastructure side. Specifically, InScope encapsulates a 20-day capture
duration with 303 tracking trajectories and 187,787 3D bounding boxes
annotated by experts. Through analysis of benchmarks, four different
benchmarks are presented for open traffic scenarios, including collaborative
3D object detection, multisource data fusion, data domain transfer, and 3D
multiobject tracking tasks. Additionally, a new metric is designed to quantify
the impact of occlusion, facilitating the evaluation of detection degradation
ratios among various algorithms. The Experimental findings showcase the
enhanced performance of leveraging InScope to assist in detecting and tracking
3D multiobjects in real-world scenarios, particularly in tracking obscured,
small, and distant objects. The dataset and benchmarks are available at
https://github.com/xf-zh/InScope.

###### keywords:

Infrastructure-side Collaborative Perception3D Multiobject Detection and
TrackingMultisource Data FusionData Domain Transfer

##  1 Introduction

Recently, autonomous driving has become a prominent area of research [1, 2].
Numerous high-quality datasets for bolstering vehicle-side perception system
(VPS) research within the autonomous driving community have been released,
yielding compelling results [3, 4]. Nevertheless, the restricted field of view
of the VPS frequently results in a short-range, incomplete perception of scene
information [5]. In response to this challenge, the infrastructure-side
perception system (IPS) has been proposed as a resolution to supplement the
restrictions of the VPS. IPSs are commonly integrated into infrastructure
facilities (refer to Figure 1), facilitating an expansive viewpoint and long-
range perception capabilities [6, 7] and thereby supporting the vehicle-to-
everything (V2X) [5].

To expedite V2X research, the DAIR-V2X [8] and V2X-Seq [9] datasets
collaboratively integrate infrastructure- and vehicle-side perspectives to
enhance the safety of autonomous driving. Despite this progress, their
reliance solely on the infrastructure-side viewpoint leaves them susceptible
to occlusions caused by larger vehicles, resulting in short-term blind spots.
These blind spots pose a potential risk of missed detections, compromising the
reliability and security of V2X systems, especially at intersections [10].

Figure 1: InScope overview. The blue and yellow dashed lines indicate the
detection areas of the principal and secondary LiDAR systems, respectively,
installed on the points of the blue and yellow pentagrams. Specific examples
of fusion points are given at the bottom.

To overcome the deficiencies, this paper introduces a novel infrastructure-
side collaborative perception system named InScope, which can cover potential
blind spots in the perceptual field via the cooperation of multi-position
LiDARs. Figure 1 illustrates the capture process and motivation for InScope,
emphasizing infrastructure equipped with sensors at an elevated position and
providing a broader site-of-view. The InScope capturing system integrates two
LiDARs positioned at distinct locations. The primary LiDAR is designed to
capture essential perception data, whereas the secondary LiDAR complements
this role by capturing information that pertains to addressing blind spots.
The data subsets gathered by these LiDAR systems are identified as InScope-Pri
and InScope-Sec, respectively.

The lower segment of Figure 1 illustrates two instances of point clouds
representing cars and pedestrians. While points from the principal or
secondary LiDARs individually contribute to point segments, neither captures
the complete boundary. In contrast, fused points combine all accessible cloud
points sourced from both LiDAR, providing improved object boundary delineation
for cars or pedestrians, thereby enhancing the comprehensive perception of
adjacent traffic surroundings. To our knowledge, the InScope is the first
large-scale dataset focusing on anti-occlusion perception for infrastructure-
to-infrastructure (I2I) collaborative perception. It is an ideal solution for
achieving blind-spot-free perception in V2X scenarios.

In addition, the DAIR-V2X dataset only supplies the 3D object detection
benchmark without providing trajectory annotation. Based on the DAIR-V2X
dataset, the V2X-Seq dataset provides additional annotations to achieve
trajectory tracking and forecasting.

Compared with the DAIR-V2X and V2X-Seq datasets, the proposed InScope dataset
can meet four benchmarks simultaneously without additional annotation: 3D
object detection, multisource data fusion, data-domain transfer, and 3D
multiobject tracking. Specifically, 3D object detection benchmark mainly
counts the 3D information of traffic participants (such as cars, trucks,
cyclists, pedestrians, etc.), significantly expanding the perception field of
autonomous vehicles. The multisource data fusion benchmark can evaluate the
robustness of multisource data with different paradigms and rigorously assess
the impact of fusion mechanisms on perception performance. Data-domain
transfer realizes development and verification of unsupervised learning
method. It also facilitates the application of SOTA 3D detectors, which were
initially conceived for vehicle-side datasets. The 3D multiobject tracking
benchmark extends the capabilities of 3D object detection by tracking object
movement over time, where the obscured, small, and distant objects can be
tracked completely within V2X scenarios, providing a dynamic and comprehensive
understanding of traffic surroundings.

Table 1: Dataset comparison. 2D and 3D represent the image and point cloud,
respectively. Ego: Egocentric vehicle. V2V: Vehicle cooperative system, Infra:
Infrastructure, I2I: Infrastructure cooperative system, V2X: Vehicle-
infrastructure cooperative system. “Real/Sim.” indicates that the datasets are
collected in the real or simulated world.

Dataset | View | Real/Sim | Year | Point | 3D | Trajectory | Blind-Spot | Occlusion | Benchmark
---|---|---|---|---|---|---|---|---|---
Clouds | Boxes | Annotation | Awareness | Quantitative
KITTI [11] | Ego | Real | 2012 | 15k | 200k | ✓ | ✗ | ✗ | 5
ApolloScape [12] | Ego | Real | 2018 | - | 70k | ✓ | ✗ | ✗ | 5
nuScenes [13] | Ego | Real | 2020 | 400k | 1.4M | ✓ | ✗ | ✗ | 2
Waymo [14] | Ego | Real | 2020 | 200k | 12M | ✓ | ✗ | ✗ | 2
ONCE [15] | Ego | Real | 2021 | 1M | 417k | ✗ | ✗ | ✗ | 4
V2V4Real [16] | V2V | Real | 2023 | 20k | 240K | ✓ | ✓ | ✗ | 3
V2X-Sim [17] | V2X | Sim. | 2022 | 10k | 26.6k | ✓ | ✓ | ✗ | 4
V2X-Seq/Perception [9] | V2X | Real | 2023 | 15k | 12k | ✓ | ✓ | ✗ | 2
NGSIM [18] | Infra | Sim. | 2016 | - | - | ✓ | ✗ | ✗ | 7
WIBAM [19] | Infra | Real | 2021 | - | - | ✗ | ✗ | ✗ | 1
DAIR-V2X-I [8] | Infra | Real | 2022 | 10k | 493k | ✗ | ✗ | ✗ | 1
RCooper (Corridor) [20] | I2I | Real | 2024 | 11k | 53k | ✓ | ✓ | ✗ | 2
RCooper (Intersection) [20] | I2I | Real | 2024 | 2k | 43k | ✓ | ✓ | ✗ | 2
InScope (Intersection) | I2I | Real | 2024 | 21k | 188k | ✓ | ✓ | ✓ | 4

The main contributions of this study can be summarized as follows:

  * •

A large-scale dataset (InScope) featuring multi-position LiDARs in a real-
world setting is presented in this paper, specifically developed to address
the current research gap concerning occlusion challenges within the I2I
perception systems in open traffic scenarios.

  * •

The InScope dataset is devoid of sensitive information and incorporates
trajectory annotations alongside 188K 3D bounding boxes. This paper provides
four benchmarks to thoroughly introduce the dataset, aiming to make a valuable
contribution to the V2X research community.

  * •

Another metric ($\xi_{D}$) within the I2I setting is further formulated to
evaluate the anti-occlusion capabilities of the I2I dataset systematically.
This metric gauges the degradation ratios in detection performance between
scenarios featuring only one LiDAR sensor and scenes with multiple LiDARs.

  * •

Several commonly employed baseline experiments have been reproduced on
InScope. The comprehensive experimental findings affirm the efficacy of
utilizing InScope to explore model anti-occlusion across different real-world
datasets, thereby establishing robust groundwork for forthcoming research
endeavors to address this challenge.

##  2 Related Work

###  2.1 Vehicle-side Datasets

Public vehicle-side datasets involving various vision tasks have significantly
promoted the process of autonomous driving [13, 14, 21]. Moreover, due to the
inherent limitations of unimodal autonomous perception, multimodal cooperative
perception has become a longstanding focus in the V2X field [22, 23, 24, 25].
Typical multimodal perception systems involve single-ego camera-camera or
camera-LiDAR systems, as summarized in Table 1. The most relevant Karlsruhe
Institute of Technology and Toyota Technological Institute (KITTI) dataset
released in 2012 labelled 200k 3D object bounding boxes [11]. It can be used
for stereo, optical flow, visual odometry / SLAM, and 3D object detection.
Afterwards, a large-scale and real-world dataset named the One millioN sCenEs
(ONCE) was released with more than 417k annotations [15]. To support more
complex and intelligent perception tasks, nuScenes [13] and Waymo [14]
datasets were introduced to implement object detection, multiobject tracking,
autonomous driving system validation and testing, motion planning, and other
tasks. Later, the V2V4Real dataset was used to improve the long-term
perception capability of autonomous vehicles by introducing a multi-vehicle
cooperative perception system in real scenes [16].

Despite recent progress in ego vehicle-side perception, it often suffers from
perceptual degradation phenomena in long-range or occluded areas [26, 27].

###  2.2 Infrastructure-side and V2X Datasets

Single vehicle-side perception significantly impedes the ability of autonomous
vehicles to make more intelligent decisions [8]. Recently, infrastructure-side
and vehicle-infrastructure cooperative (V2I) datasets have received increasing
attention as a supplement to V2X. Several publicly available datasets are
accessible for infrastructure-side and V2I applications, such as NGSIM [18],
WIBAM [19], DAIR-V2X-I [8], and V2X (V2X-Sim [17], DAIR-V2X-C [8], and V2X-Seq
[9]). Specifically, for infrastructure-side datasets, NGSIM mainly counts
vehicle motion trajectories, whereas WIBAM and DAIR-V2X-I count the 3D
bounding boxes for developing 3D object detection methods. For the V2X
datasets, V2X-Sim annotates the 3D bounding boxes and motion trajectories.
However, its data are often sourced from simulations rather than real-world
scenarios, causing a large gap between theory and practical implementation.
Later, the DAIR-V2X-C and V2X-Seq datasets were proposed to reflect more
complexity and irregularity of the real world and can serve as a blind spot
supplement (BSS) for the VPS based on the IPS. However, the effectiveness of
the DAIR-V2X-C and V2X-Seq datasets may be compromised in scenarios where
occlusion obstructs the acquisition of infrastructure-side information, and
distant objects are sparsely perceived. These issues can result in
catastrophic accidents and present challenges that are not readily addressed
by existing methods.

In addition, frequent occlusions create significant hazards in real-world
traffic scenarios. These factors complicate monitoring, navigation, and
safety, increasing the risk of accidents. To reduce infrastructure-side blind
spots, the RCooper dataset is proposed to provide information compensation
through infrastructure-side LiDAR located at different locations. The RCooper
dataset contains two scenes, i.e., the intersection and the corridor scene.
Although the intersection scene in the RCooper dataset can serve
infrastructure-to-infrastructure (I2I) BSS, the frames of the point cloud and
the number of annotations are limited. In contrast, the corridor scene in the
RCooper dataset mainly provides more extended temporal information for the
continuous tracking of objects, which makes it challenging to meet the I2I BBS
task.

Generally, subjective common sense is used to assess the anti-occlusion
capability of a method, with higher accuracy in object detection often
perceived as indicative of more robust resistance to interference. However,
the lack of a specialized dataset for quantitatively evaluating anti-occlusion
capabilities means that researchers and developers might not have the
necessary tools to rigorously test and refine their methods under occluded
conditions.

As a solution, a new infrastructure-to-infrastructure collaborative BSS system
named InScope is developed, offering an anti-occlusion test benchmark and more
comprehensive infrastructure-side scene information. It can help provide
continuous, stable, and accurate perception information for autonomous
vehicles, especially for those areas with traffic occlusions.

Table 2: Equipment used in the InScope and the key indicators of the sensors. The positions of the principal LiDAR and secondary LiDARs are shown in Figure 1. Sensors |  Details
---|---
Principal LiDAR |  80-beam, 10Hz capture frequency, vertical angular resolution of 0.1°, $\leq$ 230$m$ range, +90° $\sim$ +275° horizontal FOV.
Secondary LiDAR |  32-beam, 10Hz capture frequency, vertical angular resolution of 1.33°, $\leq$ 150$m$ range, +15° $\sim$ +305° horizontal FOV.

##  3 InScope Dataset

There are three datasets discussed in this paper, including the InScope-Pri,
InScope-Sec, and InScope datasets. Specifically, InScope-Pri and InScope-Sec
are two datasets captured from single principal and secondary LiDARs,
respectively, featuring partial and potential occlusions. The InScope dataset
is composed of both the InScope-Pri and InScope-Sec datasets, offering more
comprehensive data from fused LiDARs.

The situated position and perception field of view of the InScope are shown in
Figure 1. Notably, the blue and yellow pentagrams in the lower right picture
represent the locations of the principal and secondary LiDARs, respectively.
The blue and yellow dashed lines represent the perception ranges of the
principal and secondary LiDARs, respectively. The detailed specifications of
the equipment parameters utilized in InScope are described in Table 2.

The system setup will be introduced in Section 3.1. Section 3.2 describes the
collection and annotation manner of the InScope dataset.

###  3.1 System Setup

####  3.1.1 Sensors

The principal LiDAR captures the primary data, and the secondary LiDAR
captures the supplementary data. Based on this idea, a high-precision 80-beam
LiDAR with 10 Hz sampling is adopted as the principal LiDAR in the InScope.
Its maximum detection range is 230 $m$. Since the west of the T-intersection
is industrial, where monitoring is not allowed, the horizontal view of the
principal LiDAR is set to $[90^{\circ},275^{\circ}]$. In busy scenes, trucks
obstructing intersection can occlude other objects. This lost information
could cause erroneous judgements of vehicle-infrastructure coordination and
perception errors for autonomous vehicles. To minimize occlusion interference,
a secondary LiDAR compensates for blind-spot coverage, specifically within the
intersection. Here, a low-cost 32-beam unit with a 150-$m$ range is used for
temporary supplementation. The secondary LiDAR faces south, and its detection
area is set to $[15^{\circ},305^{\circ}]$.

####  3.1.2 Temporal Consistency

Recent studies [8, 16] have proven that a temporal variance from 10 ms
(DAIR-V2X) to 50 ms (V2V4Real) between multiple LiDARs can ensure the accuracy
and effectiveness of data fusion. Figure 2 indicates that the time error
between two LiDARs in the InScope meets the requirements (at the microsecond
level). The time information was recorded from the _ROS_ bags.

####  3.1.3 Calibration

The InScope dataset contains three Cartesian coordinates: principal LiDAR
coordinate, secondary LiDAR coordinate, and world coordinate. For the
principal LiDAR coordinate, the X-axis and Y-axis are oriented towards the
east and north, respectively. For the secondary LiDAR coordinate, the X-axis
and Y-axis are oriented towards the south and east, respectively. The InScope
uses principal LiDAR coordinate as unified coordinates. More than 20
calibration points were collected to obtain the secondary-to-principal
transform matrix, and their spatial location in three Cartesian coordinates
was recorded. From these data, the secondary-to-world and world-to-principal
transformation matrix can be obtained. Finally, the above two transform
matrixes are multiplied to determine the critical secondary-to-principal
transform matrix [28].

Figure 2: The temporal consistency among two LiDARs and the computer clock.

Considering the interference caused by remote and sparse point clouds, the
area of interest of the InScope dataset
$\left[x_{min},y_{min},z_{min},x_{max},y_{max},z_{max}\right]$ is set to a
rectangular area $\left[-5.0,-75.2,-5.0,67.0,75.2,3.0\right]$.

###  3.2 Data Collection, Annotation, and Statistics

For regulatory reasons, the detailed installation site of InScope in Guangdong
Province, China, will remain undisclosed. Since InScope is a pure point cloud
dataset, it does not contain sensitive information other than map data and
will not violate personal rights.

####  3.2.1 Collection and Annotation

After collecting the raw data, 87 sequences were manually selected in various
weather environments, with a period of 20 days and 21,317 frames of point
cloud data. The Experts then manually annotated the dataset using a
redeveloped high-precision annotation tool. They applied category, 3D bounding
boxes (including x, y, z, width, length, height, and orientation), and
tracking IDs to each frame through rigorous full supervision, ensuring
labelling quality. For each 3D bounding box in the InScope dataset annotation,
if no corresponding 3D box in the InScope-Pri or InScope-Sec dataset
annotations–matching in both location and category–is identified, the boxes of
the InScope dataset are integrated into the InScope-Pri or InScope-Sec dataset
annotations. This process facilitates the creation of I2I cooperative
annotations for the InScope dataset. These cooperative annotations are
manually supervised and refined to ensure the generation of more precise
annotations.

Figure 3: Distribution of objects with different ratios of BBS. The subfigure
reports the distribution of object counts per frame.

As mentioned above, the InScope-Sec dataset is mainly responsible for
providing blind spots supplement data for the InScope dataset. In Figure 3,
the ratio of InScope-Sec point cloud and InScope point cloud in each object is
shown, defined as the “Ratio of BBS”. It can be found that the InScope-Sec
dataset can provide at least 50% compensation data for nearly one out of three
of the objects, proving that the InScope dataset is suitable for tasks of
collaborative perception. In addition, the average number of various types of
objects in each frame is counted in the sub-figure of Figure 3. It can be seen
that there are eight objects per frame on average, and the InScope replicates
real traffic scenarios authentically. The annotated ground truth and expert
validation establish the InScope dataset as a benchmark for developing and
evaluating next-generation multisource perception systems.

Finally, Figure 3 reports the total number of various classes of objects in
the InScope dataset, including 111,630 cars, 43,676 trucks, 29,926 cyclists,
and 2,555 pedestrians. The scene is located in an industrial area, and the
proportion of trucks and cars accounts for 82.70% of the total number of
objects.

Figure 4: Real-world occlusion examples from the InScope-Pri and InScope
datasets are depicted. Red boxes indicate objects occluded in blind-spots
(grey regions), which cannot be detected by the InScope-Pri dataset.

####  3.2.2 Complex Occlusion Scenes

The InScope dataset includes various occlusion scenarios crafted to challenge
and assess the resilience of methods. These include (1) cars, cyclists, and
pedestrians occluded by trucks; (2) cyclists and pedestrians occluded by
trucks or cars; and (3) trucks, cyclists, and pedestrians partially occluded
by trees or other surroundings. Innovation can be fostered by identifying
specific weaknesses and encouraging targeted improvements in the method design
process.

Figure 4 shows specific real-world examples of occluded objects in the InScope
dataset. Due to occlusion, the grey region denotes the perception gaps from
only the InScope-Pri dataset. With the addition of a secondary LiDAR, the
InScope dataset successfully perceives the occluded objects marked with red
rectangles (such as cars, pedestrians, and cyclists) within the blind spot.
These compensation data are critical for continuously tracking objects within
a visual blind spot. More importantly, these examples demonstrate the ability
of InScope to harvest crucial supplementary information, particularly for
objects in the center of an intersection. By mitigating these blind spots,
InScope strengthens situational awareness of traffic scenes – a critical
capability for advanced applications of autonomous vehicles.

##  4 Experiments on the InScope Dataset

This section records and reports the results of experiments involving the four
tasks on the InScope dataset as a baseline for future research. Moreover, the
impact of the InScope data characteristics on different benchmarks is also
analyzed. Finally, suggestions for future research on the benchmarks are
given. The experiments are performed on a Linux server with 3 $\times$ A100
GPUs and 175 GB of RAM and reported on the validation set.

Table 3: Comparative 3D detection results of different backbones on the InScope-Pri and InScope datasets, with the best performance marked in bold. Datasets | Methods |  Car ↑
$AP_{40(IoU=0.7)}$ |  Pedestrian ↑
$AP_{40(IoU=0.5)}$ |  Cyclist ↑
$AP_{40(IoU=0.5)}$ |  Truck ↑
$AP_{40(IoU=0.7)}$ |  $mAP_{40}$ ↑ | FPS ↑
---|---|---|---|---|---|---|---
InScope-Pri | PointRCNN | 61.14 | 61.99 | 48.96 | 88.80 | 65.22 | 4.57
Pointpillar | 67.34 | 23.82 | 43.51 | 91.59 | 56.57 | 41.32
PV-RCNN++ | 72.59 | 45.26 | 61.21 | 91.02 | 67.52 | 13.28
CenterPoint | 61.31 | 49.62 | 52.73 | 82.02 | 61.42 | 31.45
InScope | PointRCNN | 71.75 | 68.13 | 62.91 | 94.50 | 74.32 | 4.58
Pointpillar | 78.04 | 35.34 | 58.46 | 95.86 | 66.93 | 24.51
PV-RCNN++ | 80.55 | 53.31 | 70.92 | 95.92 | 75.18 | 14.66
CenterPoint | 77.24 | 70.45 | 74.74 | 96.12 | 79.64 | 30.49

###  4.1 Collaborative 3D Object Detection

####  4.1.1 Metrics

$\bullet$ The average precision (AP) using 40 recall positions ($AP_{40}$):
The popular $AP_{40}$ is adopted to measure the detection performance of 3D
object detectors [29]. Following the conventions of KITTI and nuScenes, for
larger objects such as cars and trucks, a higher intersection over union (IoU)
threshold (IoU=0.7) is used to evaluate the detection performance of the
methods rigorously. In contrast, considering that the 3D sizes of cyclists and
pedestrians are smaller, a lower IoU threshold (IoU=0.5) is used to evaluate
the detection performance. $AP_{40}\in\left[0,1\right]$.

$\bullet$ The mean $AP_{40}$ ($mAP_{40}$): The average $AP_{40}$ of all
classes is adopted [30] toevaluate the detection performance of methods more
comprehensively. $mAP_{40}\in\left[0,1\right]$.

$\bullet$ The frames per second (FPS): FPS is adopted to evaluate the
computational efficiency of methods.

$\bullet$ The mean detection performance degradation ratio $\xi_{D}$: This
paper designs a new metric to quantitatively evaluate the collaborative
perception performance of 3D object detection methods. This metric
comprehensively considers the differences between methods in the InScope and
InScope-Pri datasets and can objectively and comprehensively evaluate the
collaborative perception performance of the methods in infrastructure-side
datasets. In this way, $\xi_{D}$ can be defined as:

$\xi_{D}=1-{G}_{mclass}\times{G_{mAP}},$ (1)

where ${G}_{mclass}$ and ${G_{mAP}}$ represent the mean performance gap ratio
of all classes and the performance degradation ratio of $mAP_{40}$,
respectively. ${G_{mAP}}\in\left[0,1\right]$. The lower the degradation ratio
of detection performance, the better the anti-occlusion performance of
methods.

For the ${G}_{mclass}$ metric, it can be expressed as:

${G}_{mclass}=\frac{1}{c}\sum\limits_{c}{G_{c}}=\frac{1}{c}\sum\limits_{c}{\frac{{AP_{40}{{\left({InScope_{Pri}}\right)}_{c}}}}{{AP_{40}{{\left({InScope}\right)}_{c}}}}},$
(2)

where $AP_{40}(InScope_{Pri})$ and $AP_{40}(InScope)$ represent the AP of each
class in the InScope-Pri and InScope datasets.
${c\in\left[{car,pedestrian,cyclist,truck}\right]}$ represents the object
classes. ${G_{c}}$ represents the detection performance gap ratio of each
class on the InScope-Pri and InScope datasets.
${G_{mclass}}\in\left[0,1\right]$.

The ${G_{mAP}}$ metric can be formulated as:

$\aligned{G_{mAP}}&=\frac{{mA{P_{40}}\left({InScope_{Pri}}\right)}}{{mA{P_{40}}\left({InScope}\right)}}\\\
=\frac{{\sum\limits_{c}{A{P_{40}}{{\left({InScope_{Pri}}\right)}_{c}}}}}{{\sum\limits_{c}{A{P_{40}}{{\left({InScope}\right)}_{c}}}}},$
(3)

where $mAP_{40}(InScope_{Pri})$ and $mAP_{40}(InScope)$ represent the
$mAP_{40}$ in the InScope-Pri and InScope datasets, respectively.
${G_{mAP}}\in\left[0,1\right]$.

####  4.1.2 Baselines

3D object detection is critical for both the InScope-Pri and InScope datasets.
Additionally, analyzing the anti-occlusion performance of various detector
backbones is essential. More details are shown as follows.

Figure 5: The detection results of the CenterPoint method on the InScope-Pri
and InScoperi datasets. The red and blue bounding boxes represent the ground
truth and detection results, respectively. The green circles identify the
result differences of CenterPoint based on different datasets.

3D object detection on the InScope-Pri dataset: The adopted baseline follows
the convention [31, 32, 33, 34]. Point-based detectors (such as PointRCNN
[35]) directly extract spatial structure features from the raw point cloud
data to obtain object proposals. However, the enormous quantity of raw data
introduces a substantial computational burden, seriously impacting the
computational efficiency. Voxel-based detectors (such as Pointpillar [36]) can
improve computational efficiency by converting point clouds into regular voxel
grids. However, this discretization may compromise the detection accuracy,
particularly for objects of varying sizes. voxel and point-based (such as PV-
RCNN++ [37, 38]) detectors can integrate the advantages of point-based and
voxel-based methods, allowing the detectors to fully use the spatial structure
characteristics of the raw data while reducing their dependence on hardware.
In addition, the popular center-based detectors (such as CenterPoint [39, 40])
adopt novel detection heads that can regress the an object’s center point and
orientation.

3D object detection on the InScope dataset: The same methods are adopted here
to analyze the effectiveness of the method on the fusion data that have more
complete feature expressions, including the PointRCNN, Pointpillar, PV-RCNN++,
and CenterPoint detectors. These detection performances can serve as the
control group experiment to analyze the anti-occlusion capabilities.

Anti-occlusion capability: This section introduces a benchmark that provides
consistent standards for evaluating and comparing the anti-occlusion
capabilities of different methods, using $\xi_{D}$ as an evaluation metric.
Specifically, this involves two datasets. The first is the InScope-Pri
dataset, which contains partially and potentially occluded data. The second is
the InScope dataset, a fusion dataset with relatively complete data. The
disparities in the performance gaps and mean detection performance degradation
ratio between the two datasets reflect the anti-occlusion capabilities of
various strategies. More details about how to quantitatively measure the anti-
occlusion capabilities of different methods can be found in Section 4.1.3.

Table 4: The collaborative perception performance gap ratio of each class, $mAP$, and mean collaborative perception performance degradation ratio $\xi_{D}$ (in %).  Paradigms | Methods |  $G_{car}$ ↑ |  $G_{pedestrain}$ ↑ |  $G_{cyclist}$ ↑ |  $G_{truck}$ ↑ |  $G_{mclass}$ ↑ |  $G_{mAP}$ ↑ |  $\xi_{D}$ ↓
---|---|---|---|---|---|---|---|---
Point-based | PointRCNN | 85.21 | 90.99 | 77.83 | 93.97 | 87.00 | 87.76 | 23.65
Voxel-based | Pointpillar | 86.29 | 67.40 | 74.43 | 95.55 | 80.92 | 84.52 | 31.61
Voxel & Point-based | PV-RCNN++ | 90.12 | 84.90 | 86.31 | 94.89 | 89.05 | 89.82 | 20.01
Center-based | CenterPoint | 79.38 | 70.43 | 70.55 | 85.33 | 76.42 | 77.12 | 41.06
Figure 6: Detection results on the InScope-Pri and InScope datasets and
$\xi_{D}$ metric.

####  4.1.3 Analysis

Here, the split ratios of the training/validation/test sets of the InScope
dataset are set to 0.63:0.27:0.1. The 3D detection baselines are reproduced on
the InScope-Pri and InScope datasets.

3D Object detection on the InScope-Pri and InScope datasets: The detection
results of different methods on the InScope-Pri and InScope validation sets
are indicated in Table 3. The bottom of Table 3 can be summarized as follows:
trucks have extremely discriminative features, which improve significantly the
detection performance in trucks. As the size of the objects decreases (such as
in car, cyclist, and pedestrian), the detection performance of point-based and
voxel-based methods decreases sharply. For instance, Table 3 shows that the
detection accuracy of PointRCNN and Pointpillar in the cyclist class are only
62.91% and 58.46%, respectively. The detection performance of the voxel and
point-based detectors changes relatively slowly, and the detection accuracy of
PV-RCNN++ in the cyclist class can still reach 70.92%. Notably, benefiting
from the mechanism of the center-based method of returning object properties,
the detection accuracy of small objects such as pedestrians and cyclists
improves dramatically. For example, CenterPoint can still achieve 74.74% in
the cyclist class, respectively. Meanwhile, Figure 5 indicates the detection
results of the CenterPoint method on the InScope dataset. As seen from Figure
5, the CenterPoint method based on the InScope dataset has a higher object
detection performance than that based on the InScope-Pri dataset.

3D Object detection on the InScope-Pri and InScope datasets: more specific
experimental results to to quantitatively analyze the anti-occlusion
capabilities of various methods are presented in Table 4 and Figure 6.
Generally, methods are expected to exhibit superior performance on the InScope
dataset due to the richer point cloud representations than on the InScope-Pri
dataset. To quantitatively measure and assess the anti-occlusion detection
capabilities of different methodologies, more metrics, i.e., $G_{mclass}$,
$G_{mAP}$, and $\xi_{D}$ are adopted in Table 4. It follows that the smaller
the degradation ratio $\xi_{D}$ is on the two datasets, the better the anti-
occlusion ability of the methods, and vice versa. As seen from Figure 6, the
PV-RCNN++ exhibits lower performance degradation ratio (79.99%) than
Pointpillar and CenterPoint, highlighting the superiority of the anti-
occlusion capabilities and overall robust performance achieved by integrating
the point and voxel features.

###  4.2 Multisource Data Fusion

####  4.2.1 Metrics

Similar to the 3D object detection benchmark, the $AP_{40}$ of each class and
$mAP_{40}$ are adopted here to evaluate the performance of the multisource
data fusion task.

####  4.2.2 Baselines

The three most commonly discussed fusion mechanisms are early fusion, middle
fusion, and late fusion [1, 41]. Their characteristics and disadvantages are
discussed and reported as follows:

$\bullet$ The early fusion mechanism directly combines one frame of the raw
data pair ${\bf{X}}=\left[{{\bf{X}}_{p},{\bf{X}}_{s}}\right]$ captured by the
InScope-Pri dataset ${{\bf{X}}_{p}}$ and the InScope-Sec dataset
${\bf{X}}_{s}$. Then, ${\bf{X}}$ is fed into 3D detectors to obtain the final
proposals.

$\bullet$ The late fusion adopts two independent 3D detectors to obtain two
sets of prediction proposals. Subsequently, post-optimization operations such
as voting or weight summarization are performed to obtain the final proposals.

Table 5: Comparative results of different fusion mechanisms based on the PV-RCNN++ backbone. The best results under different fusion mechanisms are marked in bold.  Detector | Fusion Mechanism |  Car ↑
$AP_{40(IoU=0.7)}$ |  Pedestrian ↑
$AP_{40(IoU=0.5)}$ |  Cyclist ↑
$AP_{40(IoU=0.5)}$ |  Truck ↑
$AP_{40(IoU=0.7)}$ |  $mAP_{40}$ ↑ | FPS ↑
---|---|---|---|---|---|---|---
PV-RCNN++ | No Fusion | InScope-Sec | 43.49 | 34.60 | 39.94 | 76.04 | 48.52 | 16.67
InScope-Pri | 72.59 | 45.26 | 61.21 | 91.02 | 67.52 | 13.28
Early Fusion (InScope) | 80.55 | 53.31 | 70.92 | 95.92 | 75.18 | 14.66
Late Fusion | 68.01 | 53.47 | 56.95 | 92.65 | 67.77 | 1.21
Middle Fusion | 73.78 | 52.06 | 62.06 | 91.89 | 69.95 | 13.02
Table 6: Comparative results of different fusion mechanisms based on the CenterPoint backbone. The best results under different fusion mechanisms are marked in bold.  Detector | Fusion Mechanism |  Car ↑
$AP_{40(IoU=0.7)}$ |  Pedestrian ↑
$AP_{40(IoU=0.5)}$ |  Cyclist ↑
$AP_{40(IoU=0.5)}$ |  Truck ↑
$AP_{40(IoU=0.7)}$ |  $mAP_{40}$ ↑ | FPS ↑
---|---|---|---|---|---|---|---
CenterPoint | No Fusion | InScope-Sec | 35.92 | 37.40 | 38.24 | 68.78 | 45.08 | 107.53
InScope-Pri | 61.31 | 49.62 | 52.73 | 82.02 | 61.42 | 31.45
Early Fusion (InScope) | 77.24 | 70.45 | 74.74 | 96.12 | 79.64 | 30.49
Late Fusion | 58.13 | 50.03 | 56.01 | 85.65 | 62.45 | 6.40
Middle Fusion | 52.74 | 38.95 | 51.19 | 81.73 | 56.15 | 15.85

$\bullet$ The middle fusion focuses on the spatial feature representations
$\left({{\bf{F}}_{p},{\bf{F}}_{s}}\right)$ from different models. Then, a
fusion module based on the convolution neural network or transformer is
adapted to align the extracted features and generate the final proposals.

Inspired by the adaptive feature fusion module [42], a simple and efficient
fusion baseline is adopted for this benchmark, as shown in Figure 7. The
adopted fusion module fuses the point cloud feature representations extracted
by the VoxelNet backbone [43].

####  4.2.3 Analysis

In this experiment, PV-RCNN++ and CenterPoint, which serve as typical models
in 3D detection paradigms, are adopted as baselines for exploring the
advantages of different fusion mechanisms. The experimental results are shown
in Table 5 and Table 6. As seen, for the PV-RCNN++ method, the top two lines
report the experimental results using single raw data captured from the
InScope-Sec dataset ${{\bf{X}}_{s}}$ and the InScope-Pri dataset
${{\bf{X}}_{p}}$, respectively. It can be found from Table 5 that the dense
point cloud captured from ${{\bf{X}}_{p}}$ can provide more sufficient
information than the sparse point cloud from ${{\bf{X}}_{s}}$. Moreover, the
computational efficiency is negatively affected. For example, the $mAP_{40}$
of PV-RCNN++ can obtain 48.52% and 67.52% with the secondary point cloud and
principal point cloud, respectively, while the FPS decreased from 16.67 to
13.81.

Figure 7: An illustration of the middle fusion module.

More importantly, the data fusion strategy increases point cloud density and
completeness. The detection performances of different types of fusion
strategies, such as early, middle, and late fusions, its detection
performances show varying degrees of growth compared with those of single raw
data. Specifically, early fusion is a straightforward data fusion mechanism;
it exhibits the most significant improvement in detection performance
($mAP_{40}$ = 75.18%). The improvement in the detection performance is
relatively limited with the late-fusion mechanism($mAP_{40}$ = 67.77%).
Moreover, designing various middle fusion architectures focused on feature-
level fusion has emerged as a key research direction in multisource data
fusion. Here, a simple but effective middle fusion strategy is provided in
Figure 7, which achieved specific results ($mAP_{40}$ = 69.95%); however,
considerable opportunities for improvement remain. Figure 8 shows the
detection performance of different fusion strategies.

Figure 8: The detection results of the PV-RCNN++ detector are based on the
different fusion mechanisms on frame 020773. The red and blue bounding boxes
represent the ground truth and detection results, respectively. The green
circles represent the differences among them.

Another set of experiments based on the CenterPoint backbone is conducted to
analyze the effectiveness of the fusion mechanism for different detection
paradigms, as shown in Table 6. Unlike that of PV-RCNN++ backbone, the middle
fusion detection performance of CenterPoint is slightly inferior to that of
the late fusion mechanism (56.15% ¡ 62.45%). This is because the detection
head based on the center heatmap head designed by CenterPoint adopts a new
idea for locating the object by calculating the probability that each voxel
belongs to the center point [44]. The simple fusion module designed considers
only the integration of forwards features and does not comprehensively
consider the impact of different detection heads on backpropagation.
Therefore, researchers can attempt more intelligent middle fusion modules by
comprehensively considering the implications between the detection backbone
and detection head.

Table 7: Data domain transfer results of different unsupervised domain adaptation methods on car class. “Source→Target” represents that methods are pre-trained on the source domain and transferred to the target domain. Source→Target | DAIR-V2X-I→KITTI | ONCE→KITTI | InScope→KITTI | InScope→DAIR-V2X-I | DAIR-V2X-I→InScope
---|---|---|---|---|---
Moderate | $mAP_{40}$ | Moderate | $mAP_{40}$ | Moderate | $mAP_{40}$ | Moderate | $mAP_{40}$ | $AP_{40(IoU=0.7)}$
Source Domain | 36.47 | 37.98 | 38.65 | 41.65 | 49.45 | 52.97 | 29.54 | 31.05 | 32.16
SN | 44.76 | 44.80 | 45.95 | 49.34 | 58.66 | 61.87 | 30.47 | 31.81 | 33.25
ST3D | 62.04 | 65.35 | 53.92 | 58.19 | 70.06 | 74.63 | 34.65 | 48.98 | 37.03
Target Domain | 78.63 | 81.63 | 78.63 | 81.63 | 78.63 | 81.63 | 78.51 | 81.41 | 71.75

###  4.3 Data Domain Transfer

####  4.3.1 Metrics

The $AP_{40}$ metric is employed to assess the performance of unsupervised
domain adaptation methods on the InScope dataset. The $mAP_{40}$ based on the
easy, moderate, and hard difficulty levels is utilized to evaluate the
adaptability of other datasets.

Figure 9: Point clouds of the InScope, DAIR-V2X-I, KITTI, and ONCE datasets.

####  4.3.2 Baselines

In this analysis, the point cloud data from four domains are utilized for data
domain transfer analysis, i.e., vehicle-side-based KITTI and ONCE and
infrastructure-side-based DAIR-V2X-I and InScope. Their scene visualizations
are shown in Figure 9. It can be found that four different datasets have
distinct point cloud density attribute characteristics. Therefore, this
benchmark is used to validate the efficacy of different datasets in mitigating
domain gaps. Notably, the InScope, KITTI, and ONCE datasets are generated by
the mechanical LiDAR, whereas the DAIR-V2X-I dataset is captured by solid-
state LiDAR. As a result, a notable disparity exists in the spatial point
cloud distribution. In addition to exploring the domain gap, this experiment
discusses the scan pattern gap for 3D detector capability in solid-state and
mechanical LiDAR. Here, the PointRCNN is used as the 3D detection backbone;
the experimental results with different unsupervised transfer methods are
reported in Table 7.

Specifically, “Source Domain” indicates that the PointRCNN is trained on the
source data domain and tested on the target data domain. “Target Domain”
indicates that the PointRCNN is trained and tested on the target data domain.
In addition, two widely used unsupervised data domain transfer learning
methods are adopted, including statistical normalization (SN) [45] and self-
training 3D object detection (ST3D) [46].

####  4.3.3 Analysis

For the domain gap, it can be seen from Table 7 that the detection performance
is improved significantly when the PointRCNN pre-trained on the InScope
dataset is transferred to the KITTI dataset. For example, after optimization
by the ST3D method, the $mAP_{40}$ can improve from 52.97% to 74.63% in the
KITTI dataset. Its detection performance is closer to that of methods trained
on the KITTI dataset ($mAP_{40}$ = 81.63%) than pre-training in the DAIR-V2X-I
($mAP_{40}$ = 65.35%) and ONCE ($mAP_{40}$ = 58.19%). It is proven that the
InScope dataset can solve the domain gap better than other datasets. Moreover,
the results demonstrate that the proposed InScope dataset is more conducive to
transferring 3D detectors to other data domains.

Figure 10: The data domain transfer results (InScope → DAIR-V2X-I) based on
the PointRCNN backbone. The green circles represent the differences among
them.

However, for the scan pattern gap, the performance improvement is not ideal
when the pre-trained PointRCNN in the InScope dataset is transferred to the
DAIR-V2X-I dataset. For example, the $mAP_{40}$ can only be improved from
31.05% to 48.98%. This demonstrates the challenges inherent in the domain
adaptation of 3D detectors across datasets captured by the scan pattern gap of
LiDAR. Figure 10 shows the experimental results that transfer the pre-trained
detection methods from the InScope dataset to the DAIR-V2X-I datasets under
the domain and scan patterns gap. This proves that the scan pattern gap in
LiDAR technology still requires further exploration.

Table 8: Tracking results of the AB3DMOT on the car class. The best results under each $IoU_{thres}$ are marked in bold. Tracker | Detectors |  Car ($IoU_{thres}=0.5$) |  Car ($IoU_{thres}=0.7$)
---|---|---|---
sAMOTA ↑ | AMOTA ↑ | MOTA ↑ | FRAG ↓ | sAMOTA ↑ | AMOTA ↑ | MOTA ↑ | FRAG ↓
AB3DMOT | PointRCNN | 74.81 | 30.63 | 63.25 | 595 | 60.34 | 20.82 | 44.45 | 1834
Pointpillar | 82.23 | 37.44 | 68.85 | 391 | 64.98 | 24.56 | 46.82 | 2166
PV-RCNN++ | 81.63 | 36.97 | 67.56 | 386 | 68.71 | 26.96 | 50.72 | 1560
CenterPoint | 78.76 | 35.01 | 61.02 | 367 | 61.25 | 22.34 | 40.98 | 1720
Table 9: Tracking results of the AB3DMOT on the pedestrian class. The best results under each $IoU_{thres}$ are marked in bold. Tracker | Detectors |  Pedestrian ($IoU_{thres}=0.25$) |  Pedestrian ($IoU_{thres}=0.5$)
---|---|---|---
sAMOTA ↑ | AMOTA ↑ | MOTA ↑ | FRAG ↓ | sAMOTA ↑ | AMOTA ↑ | MOTA ↑ | FRAG ↓
AB3DMOT | PointRCNN | 59.89 | 21.24 | 39.73 | 6 | 56.59 | 18.78 | 37.06 | 22
Pointpillar | 32.09 | 2.85 | 27.79 | 4 | 27.42 | 1.83 | 25.36 | 24
PV-RCNN++ | 31.39 | 10.73 | 27.71 | 10 | 28.54 | 8.76 | 25.75 | 20
CenterPoint | 67.38 | 28.23 | 63.48 | 8 | 62.03 | 24.94 | 59.30 | 35
Table 10: Tracking results of the AB3DMOT on the cyclist class. The best results under each $IoU_{thres}$ are marked in bold. Tracker | Detectors |  Cyclist ($IoU_{thres}=0.25$) |  Cyclist ($IoU_{thres}=0.5$)
---|---|---|---
sAMOTA ↑ | AMOTA ↑ | MOTA ↑ | FRAG ↓ | sAMOTA ↑ | AMOTA ↑ | MOTA ↑ | FRAG ↓
AB3DMOT | PointRCNN | 60.97 | 22.75 | 41.56 | 99 | 50.27 | 15.42 | 33.77 | 272
Pointpillar | 49.96 | 19.18 | 33.82 | 64 | 33.75 | 9.42 | 22.33 | 379
PV-RCNN++ | 63.00 | 26.93 | 43.22 | 177 | 52.65 | 19.01 | 34.12 | 349
CenterPoint | 68.78 | 29.07 | 45.42 | 70 | 57.50 | 20.80 | 37.58 | 267

###  4.4 3D Multiobject Tracking

####  4.4.1 Metrics

Two evaluation metrics are used, i.e., sAMOTA, AMOTA, and CLEAR metrics
(including MOTA and FRAG) [47, 48]. For object matching criteria, 3D IoU is
used to judge whether interframe objects are successfully associated [49, 50].
Two 3D IoU thresholds ($IoU_{thres}$) for each class are reported in the 3D
multiobject tracking benchmark, to evaluate the tracking performance of the
methods under various difficulties, which is different from the IoU setting of
3D object detection [40].

####  4.4.2 Baselines

The 3D multiobject tracking methods can be divided into two paradigms [51, 52,
53, 54]: joint detection and tracking (JDT) [41, 50, 55] and tracking-by-
detection (TBD) [56, 57, 58, 59, 60]. Due to the lower computational
efficiency of the JDT-based methods, TBD-based methods are used as baselines
in this paper, i.e., AB3DMOT [48]. For the 3D multiobject tracking, the
tracking results are reported for all sequences. The pre-trained PointRCNN,
Pointpillar, PV-RCNN++, and CenterPoint are used as detectors to obtain the
detection status of the object at time $t$. The 3D IoU is used as a matching
criterion [49] to associate the detected objects at time $t$ with the tracking
objects at time $t$-1, If the 3D IoU of objects between interframes is greater
than the threshold ($IOU_{thres}$), the objects can be successfully matched,
and vice versa. To evaluate the tracking performance under different
difficulties in the InScope dataset, the $IOU_{thres}$ of [car, truck,
pedestrian, cyclist] are set to
$\left[\left[0.5/0.7\right],\left[0.5/0.7\right],\left[0.25/0.5\right],\left[0.25/0.5\right]\right]$.

Table 11: Tracking results of the AB3DMOT on the truck class. The best results under each $IoU_{thres}$ are marked in bold. Tracker | Detectors |  Truck ($IoU_{thres}=0.5$) |  Truck ($IoU_{thres}=0.7$)
---|---|---|---
sAMOTA ↑ | AMOTA ↑ | MOTA ↑ | FRAG ↓ | sAMOTA ↑ | AMOTA ↑ | MOTA ↑ | FRAG ↓
AB3DMOT | PointRCNN | 82.53 | 36.33 | 73.34 | 124 | 78.67 | 33.49 | 68.20 | 181
Pointpillar | 82.18 | 37.20 | 75.26 | 80 | 76.79 | 33.88 | 70.33 | 182
PV-RCNN++ | 81.50 | 36.35 | 69.15 | 76 | 77.20 | 33.41 | 64.53 | 141
CenterPoint | 81.44 | 36.58 | 71.89 | 70 | 76.11 | 33.12 | 65.85 | 207

####  4.4.3 Analysis

The tracking performances for cars, pedestrians, cyclists, and trucks are
shown in Table 8, Table 9, Table 10, and Table 11, respectively. The detector
performance significantly affects the tracking performance of the TBD-based
tracking method (AB3DMOT). Meanwhile, the larger the object volume is, the
more sufficient the matching criteria provided for the tracking module. As
seen from Table 8, for the car class, the sAMOTA metric of the AB3DMOT method
based on the pre-trained CenterPoint has 82.23% when the $IOU_{thres}$ is set
to 0.5. However, when $IOU_{thres}$ increases from 0.5 to 0.7, the sAMOTA
metric decreases sharply from 82.23% to 64.98%, indicating that the proposed
InScope dataset still presents significant challenges in a higher 3D IoU
tracking task. These phenomenon are even more pronounced for smaller object
(i.e., pedestrians and cyclists) tracking, as shown in Table 9 and Table 10.
For instance, the tracking performance for the CenterPoint detector in the
sAMOTA metric is only 57.50% when the $IOU_{thres}$ is set to 0.5. More
advanced detection and tracking frameworks are worth further design and
exploration.

Figure 11: Tracking results of the AB3DMOT method based on the CenterPoint
detector on the sequence 21.

Finally, some tracking examples of the AB3DMOT based on the pre-trained
CenterPoint method are shown in Figure 11. The InScope dataset can accurately
track occluded, small, or distant objects through supplementary data.
Specifically, when an occluded car (ID 582) passes an intersection, the
perception information of the InScope-Pri dataset is completely lost. However,
ID 582 can still be stably and continuously tracked with the supplementary
data of the InScope-Pri dataset. Moreover, a small pedestrian (ID 3696) and a
distant occluded cyclist (ID 5232) can also be tracked stably and
continuously. This proves that the scan pattern gap in LiDAR technology
requires further exploration.

##  5 CONCLUSION

This paper proposes an innovative infrastructure-side, real-world InScope
dataset that performs I2I visual blind-spot compensation and quantitatively
evaluates the anti-occlusion detection capabilities of various methods.
Furthermore, the dataset establishes four critical perception benchmarks,
advancing the standardization of perception models within the V2X community.
In addition, comprehensive experimental evaluations utilizing widely used
methods are conducted across four benchmarks, which sets a rigorous baseline
for subsequent research endeavours. Experiments indicate that the InScope
dataset can provide robust tracking performance for obscured and small objects
at a distance or in occluded areas, which can significantly enhance perception
systems in V2X, fostering the development of more reliable autonomous
vehicular technologies.

## Declaration of Competing Interest

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## Data availability

The dataset and code are available at https://github.com/xf-zh/InScope.

## Acknowledgment

This work was supported in part by the Key-Area Research and Development
Program of Guangdong Province under Grant 2020B0909030005 and in part by the
Southern Marine Science and Engineering Guangdong Laboratory (Zhuhai) under
Grant SML2020SP011.

## References

  * [1] Y. Wang, Q. Mao, H. Zhu, J. Deng, Y. Zhang, J. Ji, H. Li, Y. Zhang, Multi-modal 3d object detection in autonomous driving: a survey, Int. J. Comput. Vis. 131 (8) (2023) 2122–2152.
  * [2] D. Fernandes, A. Silva, R. Névoa, C. Simões, D. Gonzalez, M. Guevara, P. Novais, J. Monteiro, P. Melo-Pinto, Point-cloud based 3d object detection and classification methods for self-driving applications: A survey and taxonomy, Inf. Fusion 68 (2021) 161–191.
  * [3] X. Ma, W. Ouyang, A. Simonelli, E. Ricci, 3d object detection from images for autonomous driving: A survey, IEEE Trans. Pattern Anal. Mach. Intell. 46 (5) (2024) 3537–3556.
  * [4] Y. Wu, J. Liu, M. Gong, Q. Miao, W. Ma, C. Xu, Joint semantic segmentation using representations of lidar point clouds and camera images, Inf. Fusion 108 (2024) 102370.
  * [5] Z. Chen, Y. Shi, J. Jia, Transiff: An instance-level feature fusion framework for vehicle-infrastructure cooperative 3d detection with transformers, in: Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2023, pp. 18205–18214.
  * [6] J. Jia, Z. Li, Y. Shi, Monouni: A unified vehicle and infrastructure-side monocular 3d object detection network with sufficient depth clues, in: Proc. Neural Inf. Process. Syst. (NeurIPS), 2023.
  * [7] Z. Huang, S. Sun, J. Zhao, L. Mao, Multi-modal policy fusion for end-to-end autonomous driving, Inf. Fusion 98 (2023) 101834.
  * [8] H. Yu, Y. Luo, M. Shu, Y. Huo, Z. Yang, Y. Shi, Z. Guo, H. Li, X. Hu, J. Yuan, et al., Dair-v2x: A large-scale dataset for vehicle-infrastructure cooperative 3d object detection, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2022, pp. 21361–21370.
  * [9] H. Yu, W. Yang, H. Ruan, Z. Yang, Y. Tang, X. Gao, X. Hao, Y. Shi, Y. Pan, N. Sun, et al., V2x-seq: A large-scale sequential dataset for vehicle-infrastructure cooperative perception and forecasting, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2023, pp. 5486–5495.
  * [10] M. Buchholz, J. Müller, M. Herrmann, J. Strohbeck, B. Völz, M. Maier, J. Paczia, O. Stein, H. Rehborn, R.-W. Henn, Handling occlusions in automated driving using a multiaccess edge computing server-based environment model from infrastructure sensors, IEEE Intell. Transp. Syst. Mag. 14 (3) (2021) 106–120.
  * [11] A. Geiger, P. Lenz, R. Urtasun, Are we ready for autonomous driving? the kitti vision benchmark suite, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), IEEE, 2012, pp. 3354–3361.
  * [12] X. Huang, X. Cheng, Q. Geng, B. Cao, D. Zhou, P. Wang, Y. Lin, R. Yang, The apolloscape dataset for autonomous driving, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), 2018, pp. 954–960.
  * [13] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, O. Beijbom, nuscenes: A multimodal dataset for autonomous driving, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2020, pp. 11621–11631.
  * [14] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, et al., Scalability in perception for autonomous driving: Waymo open dataset, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2020, pp. 2446–2454.
  * [15] J. Mao, M. Niu, C. Jiang, X. Liang, Y. Li, C. Ye, W. Zhang, Z. Li, J. Yu, C. Xu, et al., One million scenes for autonomous driving: Once dataset, in: Proc. Neural Inf. Process. Syst. Datasets Benchmarks, 2021.
  * [16] R. Xu, X. Xia, J. Li, H. Li, S. Zhang, Z. Tu, Z. Meng, H. Xiang, X. Dong, R. Song, et al., V2v4real: A real-world large-scale dataset for vehicle-to-vehicle cooperative perception, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2023, pp. 13712–13722.
  * [17] Y. Li, D. Ma, Z. An, Z. Wang, Y. Zhong, S. Chen, C. Feng, V2x-sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving, IEEE Robot. Automat. Lett. 7 (4) (2022) 10914–10921.
  * [18] F. H. A. U. D. T. (FHWA), Next generation simulation NGSIM vehicle trajectories and supporting data.
  * [19] J. M. Matthew Howe, Ian Reid, Weakly supervised training of monocular 3d object detectors using wide baseline multi-view traffic camera data, in: Proc. Br. Mach. Vis. Conf. (BMVC), 2021.
  * [20] R. Hao, S. Fan, Y. Dai, Z. Zhang, C. Li, Y. Wang, H. Yu, W. Yang, J. Yuan, Z. Nie, Rcooper: A real-world large-scale dataset for roadside cooperative perception, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024.
  * [21] X. Wang, Z. Sun, A. Chehri, G. Jeon, Y. Song, Deep learning and multi-modal fusion for real-time multi-object tracking: Algorithms, challenges, datasets, and comparative study, Inf. Fusion 105 (2024) 102247.
  * [22] Q. Cai, Y. Pan, T. Yao, T. Mei, 3D Cascade RCNN: High quality object detection in point clouds, IEEE Trans. Image Process. 31 (2022) 5706–5719.
  * [23] Z. Guo, X. Li, P. A. Heng, Joint-mae: 2d-3d joint masked autoencoders for 3d point cloud pre-training, in: Proc. Int. Joint Conf. Artif. Intell. (IJCAI), 2023, pp. 791–799.
  * [24] W. Su, L. Chen, Y. Bai, X. Lin, G. Li, Z. Qu, P. Zhou, What makes good collaborative views? contrastive mutual information maximization for multi-agent perception, in: Proc. AAAI Conf. Artif. Intell. (AAAI), Vol. 38, 2024, pp. 17550–17558.
  * [25] H. Yu, Y. Tang, E. Xie, J. Mao, P. Luo, Z. Nie, Flow-based feature fusion for vehicle-infrastructure cooperative 3d object detection, in: Proc. Neural Inf. Process. Syst., Vol. 36, 2024.
  * [26] R. Zhang, D. Meng, L. Bassett, S. Shen, Z. Zou, H. X. Liu, Robust roadside perception: an automated data synthesis pipeline minimizing human annotation, IEEE Trans. Intell. Veh. (Early Access, DOI: 10.1109/TIV.2024.3364977, 2024) 1–10.
  * [27] Z. Chen, Y. Shi, J. Jia, Transiff: An instance-level feature fusion framework for vehicle-infrastructure cooperative 3d detection with transformers, in: Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2023, pp. 18205–18214.
  * [28] Z. Qiu, J. Martínez-Sánchez, P. Arias-Sánchez, R. Rashdi, External multi-modal imaging sensor calibration for sensor fusion: A review, Inf. Fusion 97 (2023) 101806.
  * [29] A. Simonelli, S. R. Bulo, L. Porzi, M. López-Antequera, P. Kontschieder, Disentangling monocular 3d object detection, in: Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2019, pp. 1991–1999.
  * [30] Z. Liu, T. Huang, B. Li, X. Chen, X. Wang, X. Bai, Epnet++: Cascade bi-directional fusion for multi-modal 3d object detection, IEEE Trans. Pattern Anal. Mach. Intell. 45 (7) (2023) 8324–8341.
  * [31] Y. Zhang, Q. Hu, G. Xu, Y. Ma, J. Wan, Y. Guo, Not all points are equal: Learning highly efficient point-based detectors for 3d lidar point clouds, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2022, pp. 18953–18962.
  * [32] R. Qian, X. Lai, X. Li, 3d object detection for autonomous driving: A survey, Pattern Recognit. 130 (2022) 108796.
  * [33] Z. Yang, Y. Sun, S. Liu, J. Jia, 3dssd: Point-based 3d single stage object detector, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2020, pp. 11040–11048.
  * [34] Y. Yan, Y. Mao, B. Li, Second: Sparsely embedded convolutional detection, Sensors 18 (10) (2018) 3337.
  * [35] S. Shi, X. Wang, H. Li, PointRCNN: 3d object proposal generation and detection from point cloud, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2019, pp. 770–779.
  * [36] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, O. Beijbom, Pointpillars: Fast encoders for object detection from point clouds, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2019, pp. 12697–12705.
  * [37] S. Shi, C. Guo, L. Jiang, Z. Wang, J. Shi, X. Wang, H. Li, PV-RCNN: Point-voxel feature set abstraction for 3d object detection, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2020, pp. 10529–10538.
  * [38] S. Shi, L. Jiang, J. Deng, Z. Wang, C. Guo, J. Shi, X. Wang, H. Li, Pv-rcnn++: Point-voxel feature set abstraction with local vector representation for 3d object detection, Int. J. Comput. Vis. 131 (2) (2023) 531–551.
  * [39] T. Yin, X. Zhou, P. Krahenbuhl, Center-based 3d object detection and tracking, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2021, pp. 11784–11793.
  * [40] H. Liu, Y. Ma, Q. Hu, Y. Guo, Centertube: Tracking multiple 3d objects with 4d tubelets in dynamic point clouds, IEEE Trans. Multimedia 25 (2023) 8793–8804.
  * [41] X. Zhang, Z. Fan, X. Tan, Q. Liu, Y. Shi, Spatiotemporal adaptive attention 3d multiobject tracking for autonomous driving, Knowl.-Based Syst. 267 (2023) 110442\.
  * [42] J. Wang, J. Li, Y. Shi, J. Lai, X. Tan, AM3Net: Adaptive mutual-learning-based multimodal data fusion network, IEEE Trans. Circuits Syst. Video Technol. 32 (8) (2022) 5411–5426.
  * [43] Y. Zhou, O. Tuzel, Voxelnet: End-to-end learning for point cloud based 3d object detection, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2018, pp. 4490–4499.
  * [44] X. Zhou, V. Koltun, P. Krähenbühl, Tracking objects as points, in: Proc. Eur. Conf. Comput. Vis. (ECCV), Springer, 2020, pp. 474–490.
  * [45] Y. Wang, X. Chen, Y. You, L. E. Li, B. Hariharan, M. Campbell, K. Q. Weinberger, W.-L. Chao, Train in germany, test in the usa: Making 3d object detectors generalize, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2020, pp. 11713–11723.
  * [46] J. Yang, S. Shi, Z. Wang, H. Li, X. Qi, St3d: Self-training for unsupervised domain adaptation on 3d object detection, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2021, pp. 10368–10378.
  * [47] K. Bernardin, R. Stiefelhagen, Evaluating multiple object tracking performance: the clear mot metrics, EUR. J. Image Video Process. 2008 (2008) 246309:1–246309:10.
  * [48] X. Weng, J. Wang, D. Held, K. Kitani, 3D multi-object tracking: A baseline and new evaluation metrics, in: Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), IEEE, 2020, pp. 10359–10366.
  * [49] Z. Zheng, P. Wang, W. Liu, J. Li, R. Ye, D. Ren, Distance-IoU loss: Faster and better learning for bounding box regression, in: Proc. AAAI Conf. Artif. Intell. (AAAI), 2020, pp. 12993–13000.
  * [50] K. Huang, Q. Hao, Joint multi-object detection and tracking with camera-lidar fusion for autonomous driving, in: Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), IEEE, 2021, pp. 6983–6989.
  * [51] Z. Cao, J. Li, D. Zhang, M. Zhou, A. Abusorrah, A multi-object tracking algorithm with center-based feature extraction and occlusion handling, IEEE Trans. Intell. Transp. Syst. 24 (4) (2023) 4464–4473.
  * [52] J. Nie, Z. He, Y. Yang, Z. Bao, M. Gao, J. Zhang, Osp2b: One-stage point-to-box network for 3d siamese tracking, in: Proc. Int. Joint Conf. Artif. Intell. (IJCAI), 2023, pp. 1285–1293.
  * [53] Y. Chen, J. Liu, X. Zhang, X. Qi, J. Jia, Voxelnext: Fully sparse voxelnet for 3d object detection and tracking, in: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2023, pp. 21674–21683.
  * [54] H. Wu, W. Han, C. Wen, X. Li, C. Wang, 3d multi-object tracking in point clouds based on prediction confidence-guided data association, IEEE Trans. Intell. Transp. Syst. 23 (6) (2022) 5668–5677.
  * [55] S. Wang, P. Cai, L. Wang, M. Liu, Ditnet: End-to-end 3d object detection and track id assignment in spatio-temporal world, IEEE Robot. Automat. Lett. 6 (2) (2021) 3397–3404.
  * [56] A. Kim, G. Brasó, A. Ošep, L. Leal-Taixé, Polarmot: How far can geometric relations take us in 3d multi-object tracking?, in: Proc. Eur. Conf. Comput. Vis. (ECCV), Springer, 2022, pp. 41–58.
  * [57] J. Sun, Y. Xie, S. Zhang, L. Chen, G. Zhang, H. Bao, X. Zhou, You don’t only look once: Constructing spatial-temporal memory for integrated 3d object detection and tracking, in: Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2021, pp. 3185–3194.
  * [58] C. Zhang, H. Wang, L. Chen, Y. Li, Y. Cai, Mixedfusion: An efficient multimodal data fusion framework for 3-d object detection and tracking, IEEE Trans. Neural Netw. Learn. Syst. Early Access, DOI: 10.1109/TNNLS.2023.3325527, 2023\.
  * [59] Z. Pang, J. Li, P. Tokmakov, D. Chen, S. Zagoruyko, Y.-X. Wang, Standing between past and future: Spatio-temporal modeling for multi-camera 3d multi-object tracking, in: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2023, pp. 17928–17938.
  * [60] N. Marinello, M. Proesmans, L. Van Gool, Triplettrack: 3d object tracking using triplet embeddings and lstm, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2022, pp. 4500–4510.


Hoffman colorings

Thijs van Veluw

Radboud University Nijmegen
Master Thesis in Mathematics

Supervised by:
Aida Abiad (Eindhoven University of Technology)
Wieb Bosma (Radboud University Nijmegen)

July 1st 2024

We study equality in the Hoffman bound for the chromatic number and Hoffman colorings in regular and irregular graphs. We investigate the connection between Hoffman colorability and several graph operations, of which the tensor product is especially interesting in this context. We then introduce the Decomposition Theorem revealing structural properties that Hoffman colorings must obey. Using the Decomposition Theorem we are able to completely classify Hoffman colorability of cone graphs and line graphs. We also prove a partial converse, the Composition Theorem, allowing us to find various new infinite families of Hoffman colorable graphs, many of which are irregular. Lastly we introduce a new parameterization and type system for strongly regular graphs, that show connections between Hoffman colorability, spreadability, pseudo-geometricity and unique vector colorability.

[1]
A. Abiad.
A characterization and an application of weight-regular partitions of graphs.
Linear Algebra Appl., 569:162–174, 2019.

[2]
A. Abiad, B. De Bruyn, J. D'haeseleer, and J. H. Koolen.
Neumaier graphs with few eigenvalues.
Des. Codes Cryptography, 90(9):2003–2019, 2022.

[3]
L. W. Beineke, R. J. Wilson, and Peter J. Cameron, editors.
Topics in algebraic graph theory, volume 102 of Encycl. Math. Appl.
Cambridge: Cambridge University Press, 2004.

[4]
T. Beth, D. Jungnickel, and H. Lenz.
Design theory. Vol. I., volume 69 of Encycl. Math. Appl.
Cambridge: Cambridge University Press, 2nd ed. edition, 1999.

[5]
A. Blokhuis, A. E. Brouwer, and W. H. Haemers.
On 3-chromatic distance-regular graphs.
Des. Codes Cryptography, 44(1-3):293–305, 2007.

[6]
W. Bosma, J. Cannon, and C. Playoust.
The Magma algebra system. I. The user language.
J. Symbolic Comput., 24(3-4):235–265, 1997.
Computational algebra and number theory (London, 1993).

[7]
A. E. Brouwer and W. H. Haemers.
Spectra of graphs.
Universitext. Berlin: Springer, 2012.

[8]
S. M. Cioabă, K. Guo, and W. H. Haemers.
The chromatic index of strongly regular graphs.
Ars Math. Contemp., 20(2):187–194, 2021.

[9]
D. M. Cvetkovic.
Graphs and their spectra.
Publ. Fac. Électrotech. Univ. Belgrade, Sér. Math. Phys., 354-356:1–50, 1971.

[10]
P. Delsarte.
An algebraic approach to the association schemes of coding theory.
Ann Arbor, MI: Historical Jrl., 1973.

[11]
N. C. Fiala and W. H. Haemers.
5-chromatic strongly regular graphs.
Discrete Math., 306(23):3083–3096, 2006.

[12]
M. A. Fiol.
Eigenvalue interlacing and weight parameters of graphs.
Linear Algebra Appl., 290(1-3):275–301, 1999.

[13]
A. Galtman.
Spectral characterizations of the Lovász number and the Delsarte number of a graph.
J. Algebr. Comb., 12(2):131–143, 2000.

[14]
C. Godsil, D. E. Roberson, B. Rooney, R. Šámal, and A. Varvitsiotis.
Graph homomorphisms via vector colorings.
Eur. J. Comb., 79:244–261, 2019.

[15]
C. Godsil, D. E. Roberson, R. Šámal, and S. Severini.
Sabidussi versus Hedetniemi for three variations of the chromatic number.
Combinatorica, 36(4):395–415, 2016.

[16]
C. Godsil and G. Royle.
Algebraic graph theory, volume 207 of Grad. Texts Math.
New York, NY: Springer, 2001.

[17]
W. H. Haemers.
Eigenvalue techniques in design and graph theory.
Mathematical Centre Tracts 121. Amsterdam: Mathematisch Centrum. V, 102 p. Dfl. 13.00 (1980)., 1980.

[18]
W. H. Haemers.
Interlacing eigenvalues and graphs.
Linear Algebra Appl., 226-228:593–616, 1995.

[19]
W. H. Haemers.
Hoffman's ratio bound.
Linear Algebra Appl., 617:215–219, 2021.

[20]
W. H. Haemers and V. D. Tonchev.
Spreads in strongly regular graphs.
Des. Codes Cryptography, 8(1-2):145–157, 1996.

[21]
A. J. Hoffman.
On eigenvalues and colorings of graphs.
Graph Theory Appl., Proc. advanced Sem. Wisconsin, Madison 1969, 79-91 (1970)., 1970.

[22]
D. Karger, R. Motwani, and M. Sudan.
Approximate graph coloring by semidefinite programming.
J. ACM, 45(2):246–265, 1998.

[23]
M. Kwan and Y. Wigderson.
The inertia bound is far from tight, 2023.

[24]
L. Lovász.
On the Shannon capacity of a graph.
Kibern. Sb., Nov. Ser., 19:5–22, 1983.

[25]
L. Mančinska and D. E. Roberson.
Quantum homomorphisms.
J. Comb. Theory, Ser. B, 118:228–267, 2016.

[26]
R. J. McEliece, E. R. Rodemich, and H. C. jun. Rumsey.
The Lovasz bound and some generalizations.
J. Comb. Inf. Syst. Sci., 3:134–152, 1978.

[27]
B. D. McKay.
Database of strongly regular graphs (referenced july 2024).
URL: <https://users.cecs.anu.edu.au/ bdm/data/graphs.html>.

[28]
B. D. McKay.
Isomorph-free exhaustive generation.
J. Algorithms, 26(2):306–324, 1998.

[29]
D. E. Roberson.
Homomorphisms of strongly regular graphs.
Algebr. Comb., 2(4):481–497, 2019.

[30]
G. Sabidussi.
Graphs with given group and given graph-theoretical properties.
Can. J. Math., 9:515–525, 1957.

[31]
A. Schrijver.
A comparison of the Delsarte and Lovász bounds.
IEEE Trans. Inf. Theory, 25:425–429, 1979.

[32]
Y. Shitov.
Counterexamples to Hedetniemi's conjecture.
Ann. Math. (2), 190(2):663–667, 2019.

[33]
J. A. Thas and S. E. Payne.
Spreads and ovoids in finite generalized quadrangles.
Geom. Dedicata, 52(3):227–253, 1994.

[34]
V. G. Vizing.
Kritische Graphen mit gegebener chromatischer Klassen.
Diskret. Analiz, Novosibirsk, 5:9–17, 1965.

[35]
P. Wocjan and C. Elphick.
Spectral lower bounds for the orthogonal and projective ranks of a graph.
Electron. J. Comb., 26(3):research paper p3.45, 14, 2019.