# Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detection

Kaicheng Yu1111Equal Contribution. Tang Tao2111Equal Contribution. 222Work
done during an internship at DAMO Academy, Alibaba Group. Hongwei Xie1111Equal
Contribution. Zhiwei Lin3 Zhongwei Wu1
Zhongyu Xia3 Tingting Liang3 Haiyang Sun1 Jiong Deng1
Dayang Hao1 Yongtao Wang3 Xiaodan Liang2 Bing Wang1444Corresponding Author.
1 Autonomous Driving Lab, Alibaba Group, China
2 Shenzhen Campus, Sun Yat-sen University, China
3 Wangxuan Institute of Computer Technology, Peking University, China

<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

There are two critical sensors for 3D perception in autonomous driving, the
camera and the LiDAR. The camera provides rich semantic information such as
color, texture, and the LiDAR reflects the 3D shape and locations of
surrounding objects. People discover that fusing these two modalities can
significantly boost the performance of 3D perception models as each modality
has complementary information to the other. However, we observe that current
datasets are captured from expensive vehicles that are explicitly designed for
data collection purposes, and cannot truly reflect the realistic data
distribution due to various reasons. To this end, we collect a series of real-
world cases with noisy data distribution, and systematically formulate a
robustness benchmark toolkit, that simulates these cases on any clean
autonomous driving datasets. We showcase the effectiveness of our toolkit by
establishing the robustness benchmark on two widely-adopted autonomous driving
datasets, nuScenes and Waymo, then, to the best of our knowledge, holistically
benchmark the state-of-the-art fusion methods for the first time. We observe
that: i) most fusion methods, when solely developed on these data, tend to
fail inevitably when there is a disruption to the LiDAR input; ii) the
improvement of the camera input is significantly inferior to the LiDAR one. We
further propose an efficient robust training strategy to improve the
robustness of the current fusion method. The benchmark and code are available
at https://github.com/kcyu2014/lidar-camera-robust-benchmark.

##  1 Introduction

Figure 1: Benchmarking the robustness of state-of-the-art 3D detection
methods. (a) We provide a knowledge graph of all noisy fusion cases. (b) We
report the performance of current methods on two robustness datasets, Waymo-R
and nuScenes-R, which are generated by our proposed toolkit. We observe that
current fusion methods are more robust to image disruption rather than the
LiDAR one.

3D detection has received extensive attention as one of the fundamental tasks
in autonomous driving scenarios [41, 30, 42, 29, 11, 14, 27, 7, 39, 22].
Recently, fusing the two common modalities, input from the camera and LiDAR
sensors, has become a de-facto standard in the 3D detection domain as each
modality has complementary information of the other [4, 33, 34, 31, 40, 8,
44]. Similar to other literature in the computer vision community, a common
approach to showcase the effectiveness of a proposed fusion method is to
validate it on the existing benchmark datasets [32, 3], which are usually
collected from explicitly designed, expensive data collection vehicles to
minimize any potential error from the hardware setup.

However, we discover that the data distribution of these popular datasets can
be drastically different from the realistic driving scenarios due to various
reasons: i) there can be uncontrollable external reasons, such as splatted
dirt or BIOS malfunctions, that temporarily disable the input of certain
sensors; ii) the inputs can be difficult to synchronize due to external and
internal reasons, like bumpy road or system clock misalignment. Therefore, we
cannot estimate the performance of methods under these realistic settings, and
evaluating only on these clean datasets is not trustworthy when deployed to
realistic scenarios.

To this end, we close this research gap by proposing a novel toolkit that
transforms any clean benchmark dataset, which has the camera and LiDAR input
modality, into a robustness benchmark to simulate realistic scenarios. We
first conduct a systematic overview of potential sensor noisy cases, both for
the camera and LiDAR, based on realistic driving data, and summarized a
knowledge graph in Figure 1(a). Specifically, we identify seven unique cases
under three categories, two for noisy LiDAR cases, two for noisy camera cases,
and three for ill-synchronization cases. We then carefully study each case and
construct a code toolkit to transform the clean data into a realistic data
distribution.

To verify the effectiveness of our approach, we apply our toolkit to two
large-scale popular benchmark datasets for autonomous driving, nuScenes and
Waymo. Note that, though these noisy cases rarely appear in realistic
scenarios, we convert all data from a dataset to fully explore the robustness
of a given method in an extreme manner. We collect two single modalities and
three fusion state-of-the-art methods and benchmark them on the generated
datasets. In Figure 1 (b), we observe several surprising findings: i) state-
of-the-art fusion methods tend to fail inevitably when the LiDAR sensor
encounters failures due to their fusion mechanism heavily relies on the LiDAR
input; ii) fusing the camera input only brings a marginal improvement,
suggesting either the current methods fail to sufficiently leverage the
information from the camera or the camera information did not carry the
complementary information as intuited. Note that the toolkit only generates
one failure case at a time, we do not create a robust benchmark that has
multiple malfunctions at the same time. Though the main purpose of this work
is to create a robustness bnechmark, we nonetheless provide a simple method,
which finetunes the model on these robustness scenarios, and show that it
moderately improves the robustness of current methods. However, there is still
a large performance gap when compared to the results of the clean settings.

We argue that an ideal goal of a fusion framework is when there is only a
single modality sensor failure, the performance should not be worse than the
method works on the other modality. Otherwise, the current fusion methods
should be replaced by using two separate networks for each modality and
performing fusion by post-processing steps [4, 21]. We hope our work can shed
light on developing robust fusion method that can be truly deployed to the
autonomous vehicle.

In summary, our main contribution can be summarized as follow: i) we
systematically study the noisy sensor data in the realistic driving scenarios
and propose a novel toolkit that can transform any autonomous driving
benchmark datasets, that contain camera and LiDAR input, into a robustness
benchmark; ii) to the best of our knowledge, we are the first to benchmark
existing methods under the same settings and find that current fusion methods
has a fundamental flaw and can fail inevitably when there is a LiDAR
malfunction.

##  2 Related Work

In this section, we provide a literature review of current fusion methods in
the 3D detection domain and the robustness evaluation.

Fusion methods in 3D detection. LiDAR and camera are two types of
complementary sensors for 3D object detection in autonomous driving. In
essence, the LiDAR sensor provides an accurate depth and shape information of
the surrounding world in form of sparse point clouds [26, 25, 24, 30, 42, 13,
47, 37, 43, 29], while the camera sensor provides an RGB-based image that
contains rich semantic and texture information [19, 28, 18, 10, 46, 48, 27,
36, 35, 23, 27, 7, 39]. Recently, fusing these modalities to leverage the
complementary information becomes a de-facto standard in the 3D detection
domain. Based on the fusion mechanism location, these methods can be divided
into three categories, early, deep, and late fusion schemes.

Early fusion methods mainly concatenate the image features to the original
LiDAR point to enhance the representation power. Specifically, these methods
rely on the LiDAR-to-world and camera-to-world calibration matrix to project a
LiDAR point on the image plane, where it serves as a query of image features
[33, 34, 31, 40, 8, 44]. Deep fusion methods extract deep features from some
pre-trained neural networks for both modalities under a unified space [1, 12,
9, 4, 45, 16, 15], where a popular choice of such space is the bird’s eye view
(BEV) [1, 45]. While both early and deep fusion mechanisms usually occur
within a neural network pipeline, the late fusion scheme usually contains two
independent perception models to generate 3D bounding box predictions for both
modalities, then fuse these predictions using post-processing techniques [4,
21]. One benefit of these works is their robustness against single modality
input failure. However, it is difficult to jointly optimize this line of
methods due to the post-processing technique being usually non-differentiable.
In addition, this pipeline has a potential higher deployment cost as it has
three independent modules to be maintained.

Robustness of LiDAR-Camera Fusion. In the domain of autonomous driving, there
lacks such a benchmark dataset for robustness analysis to the best of our
knowledge. There only exists a few preliminary attempts to investigate the
robustness issue of the fusion methods. TransFusion [1] evaluates the
robustness of different fusion strategies under three scenarios: splitting
validation set into daytime and nighttime, randomly dropping images for each
frame, misaligning LiDAR and camera calibration by randomly adding a
translation offset to the transformation matrix from camera to LiDAR sensor.
However, TransFusion [1] mainly explores the robustness against camera inputs,
and ignores the noisy LiDAR and temporal misalignment cases. DeepFusion [12]
examines the model robustness by adding noise to LiDAR reflections and camera
pixels. Though the noise settings of DeepFusion [12] are straightforward and
brief, the noisy cases almost never appear in real scenes.

By contrast, we systematically review the autonomous driving perception system
and identify three categories, in a total of seven cases of robustness
scenarios, and propose a toolkit that can transform an existing dataset into a
robustness benchmark. We hope our work can help future research to benchmark
the robustness of their methods fairly, and give researchers more insights
about designing a more robust fusion framework. An ideal fusion framework
should work better than a single modality, and will not be worse than the
single modality model while the other modality fails. We hope the deep fusion
method is better than late fusion methods that use post-processing techniques.

##  3 Robust Fusion Benchmark

In this section, we first provide a systematic overview of current autonomous
driving vehicle systems with LiDAR and camera sensors to show why the data
distribution of clean datasets can differ from real-world scenarios. These
noisy data cases can be categorized into three broad classes, noisy LiDAR,
noisy camera, and ill-synchronization cases. Then, we present a toolkit that
can transform current clean datasets into realistic scenarios.

###  3.1 An overview of modern autonomous driving vehicle system

Figure 2: Autonomous driving perception system with camera and LiDAR sensors.

In Figure 2, we visualize a common design of the autonomous driving perception
system, whose main components include the camera and LiDAR sensor, and an on-
device computer. Specifically, the camera and LiDAR sensors are physically
mounted on certain fixed locations of the vehicle and are connected to the
computer via certain cables with communication protocols. In essence, the
computer can access the data stream from the sensors and capture the data into
a point cloud or image with a certain timestamp. As the raw data are in the
sensor coordinate system, sensor calibration plays a major role in performing
efficient coordinate transformation such that the perception system can
recognize objects with respect to the ego-car coordinate system. Based on our
experience, each step of the aforementioned system can encounter certain
failures or disruptions, and yield noisy data that are drastically different
from the normal clean data. We identify three categories of cases and briefly
discuss the potential reasons and consequences in Table 1, and provide a
detailed case analysis later.

Table 1: Common reasons for noisy data cases. Based on the realistic experiences, we report various reasons that cause the noisy data cases. Note that both LiDAR and camera modality can encounter the temporal misalignment issue.  Group | Reason | Consequent Case
---|---|---
Noisy LiDAR | LiDAR sensor damage | no LiDAR point inputs
Installation limitation of LiDAR | limited LiDAR field-of-view
Low reflection rate of objects | LiDAR object failure
Noisy camera | Camera sensors damage | Missing of corresponding image inputs
Camera lens occlusion | Lens Occlusion
Ill-synchronization | Vibration during driving | Spatial misalignment
Loose physical mounting | Spatial misalignment
Instability of car computer | Temporal misalignment
Temporary insufficient cable bandwidth | Temporal misalignment
Sensor connection failure | Temporal misalignment

###  3.2 Case analysis

In this section, we analyze the collected real-world noisy data cases of
autonomous driving in detail.

####  3.2.1 Noisy LiDAR Data

We identify two common cases that can cause noisy LiDAR data in practice.

Figure 3: Limited LiDAR field-of-view. This figure is better viewed in color.
(Left) We visualize the complete LiDAR point clouds that come from the data
collection vehicle which has a complete sensor rack. (Right) In realistic
scenarios, the LiDAR is installed in a front-facing manner, yielding a limited
FOV.

Limited LiDAR field-of-view (FOV). While most companies collect the LiDAR data
whose field-of-view is 360 degrees, certain LiDAR data might not always be
available for various reasons. For example, a certain type of vehicle only
installs a front-facing semi-solid LiDAR sensor on the roof of the car instead
of using a full rack, as shown in the right part of Figure 3. Another common
reason might be the temporary occlusion of the LiDAR sensor. Without loss of
generality, we first convert the coordinate of LiDAR points from Euclidean (in
$x,y,z$) to polar coordinate system ($r,\theta,z$). We then can simulate such
limited FOV by dropping the points that satisfy
$\theta\in(-\theta_{0},\theta_{0})$. In practice, we set $\theta_{0}$ to 0,
60, and 90 degrees to simulate three commonly seen scenarios.

Figure 4: LiDAR object failure. On rainy days, the reflection rate of some
common objects (e.g., the black car in the yellow box) is below the threshold
of LiDAR hence causing the issue of object failure. This figure is better
viewed in color.

LiDAR object failure. One common scenario that people tend to overlook is that
the LiDAR can be blind to objects under certain constraints. We show one
example from the realistic data captured on a commercialized autonomous
driving system in Figure 4. We observe that the LiDAR point clouds are
drastically different from two side-by-side cars, where the black car has
nearly zero points while the white car has a normal point distribution. We dub
this phenomenon LiDAR object failure. Without loss of generality, we simulate
such scenarios by randomly dropping the points within a bounding box with a
probability of 0.5. Note that we do not alter the camera input because the
purpose is to benchmark the single modality input data.

####  3.2.2 Noisy Camera Sensor

Different from the LiDAR module, the camera module is usually installed on
much lower locations of the autonomous driving vehicles to cover the blind
region of the LiDAR sensor. Such blind region is due to the fact that the
LiDAR is usually installed on the roof of the car to maximize the
visualization distance, while it cannot see the near-car region due to
blockage. As such, the camera can be easily affected by the surrounding
environment such as temporary generic object coverage or lens occlusion of
dirt. We discuss these two scenarios in detail.

Missing camera inputs. As the camera module is usually much smaller (within
one centimeter) than the LiDAR sensor, the probability of covering half of the
camera sensor is minimal. Thus, we drop the entire camera input to simulate
such covering scenarios. In practice, we design two finer cases to perform a
robust benchmark, dropping one camera at a time, and dropping all other
cameras except the front one.

Figure 5: Visualization of camera occlusion. We display the original images
from four different scenarios in the nuScenes dataset (Left), the randomly
sampled dirt occlusion masks (Middle), and the final composed images that
simulating the occlusion (Right).

Camera lens occlusion. Another commonly seen camera covering problem is lens
occlusion caused by non-transparent liquid or dirt. To simulate the occlusion
of camera lenses in real scenes, we spray mud dots on a transparent film and
cover the dirty film on the camera lens to take photos on a white background.
Then, we adopt an image matting algorithm to cut out the background part in
the images and separate the masks of mud spots. Finally, the separated masks
are pasted on the images of nuScene or Waymo to simulate the occlusion of
their camera lens, as illustrated in the Figure 5. In addition, we spray mud
dots of different sizes and randomly move and rotate the film to create masks
with different occlusion areas and occlusion ranges to enhance the diversity
of the mask.

###  3.3 Ill-synchronization

As illustrated in Figure 2, the data stream is firstly fixed into a data frame
with a given timestamp when passed into the on-device computer, then one needs
to perform the coordinate transform via the camera-to-world and LiDAR-to-world
matrix that is obtained by the calibration process. However, this leads to two
potential ill-synchronization issues, spatial misalignment due to the external
reasons of calibration matrix and temporal misalignment for both LiDAR and
camera data due to internal system reasons.

Figure 6: Visualization of spatial misalignment effect. As the physical size
of a camera module is drastically smaller than the size of vehicle, the
relative position of car center to the camera center will inevitably change
due to various reasons, like the vibration during driving on the bumpy road,
and since such noise happens all the time, it cannot be avoided using online
calibration. We provide one visual example to showcase such misalignment.

Spatial misalignment. The bumping and shaking of vehicles lead to the
disturbance of the extrinsic parameters of cameras and cause spatial
misalignment between LiDAR and camera inputs. In addition, such errors can
accumulate while the mileage of a vehicle is increasing. To simulate such a
situation, we add random rotation and translation noise to the calibration of
each camera independently. The range of noise rotation angle is from
$0^{\circ}$ to $5^{\circ}$ and the translation range is from $1$ cm to $5$ cm
to accord with the noise range in the real scene. Sensor calibration
misalignment will cause spatial misalignment between point cloud and image, as
shown in Figure 6.

Figure 7: Visualization of temporal misalignment. Here we show one concrete
example of temporal misalignment. This happens quite commonly on realistic
autonomous vehicle. As the streaming data is first fixed with a certain time
stamp then passed into the corresponding code module of deep learning model
via system sockets, the timestamp of both modality sensors might not be always
synchronized. To simulate such effect, we let the frame remains the same as
previous frame when the data is ill-synchronized. We dub the phenomenon data
stuck. We consider two type of temporal misalignment, LiDAR stuck on the top
of this figure and camera stuck on the bottom.

Temporal misalignment. In a realistic autonomous driving system, failure of
the system components is quite common throughout the time. One obvious
consequence is the time-stamp of two modalities might not always be aligned.
In some rare cases, the data frame of one modality can be stuck by over one
minute depending on different system implementations. Here, we simulate such
temporal misalignment in detail and provide one illustration in Figure 7.
Initially, we apply nine levels of severity according to the percentage of
stuck frames in all frames. The results show that the performance degradation
of the 3D detection methods is linearly related to the percentage of stuck
frames, as discussed in Section 4.2. Thus, to reduce the load of the
benchmarks, we only consider the case where the stuck frames are 50% of all
frames as the final benchmark setting.

###  3.4 A toolkit to transform generic autonomous driving dataset into
robustness benchmark

To remove the randomness of benchmark comparison, we compose a toolkit that
can transform any autonomous driving dataset into a robustness benchmark
111See our GitHub repository for more details.
https://github.com/kcyu2014/lidar-camera-robust-benchmark. . In essence, we
only simulate noisy data cases by altering the image and LiDAR data, the
ground-truth annotation will remain the same as the 3D position of the object
in the surrounding worlds will not change when the sensors malfunction.

To facilitate future research, we leverage two popular large-scale autonomous
driving datasets, nuScenes and Waymo, and benchmark state-of-the-art methods
to evaluate their robustness for the first time to the best of our knowledge.
We denote the newly created robustness benchmark nuScenes-R and Waymo-R.

Evaluation Metrics. To intuitively show the robustness of LiDAR-camera fusion
methods, we simply use the performance and the relative performance
degradation on our benchmark datasets as our evaluation metrics. Specifically,
the LiDAR-camera fusion model performance on the clean dataset is denoted as
$P_{C}$ and its corresponding robustness performance against disruption type
$d$ under severity level $l$ on the benchmark is denoted as $P_{R}^{d,l}$.
Then, we can estimate the model robustness $mP_{R}$ by averaging over all
noise types and severity levels. The formula can be summarized as follows:

$mP_{R}=\frac{1}{N_{d}}\sum\limits_{d=1}^{N_{d}}\frac{1}{N_{l}}\sum\limits_{l=1}^{N_{l}}P_{R}^{d,l},$
(1)

where $N_{d}$ is the number of disruption types and ${N_{l}}$ is the number of
severity levels. The relative mean robustness performance of the model is
defined as $R=mP_{R}/P_{C}$. The higher $R$ means the model is more robust to
inferior senor fusion conditions. In practice, we adopt the mean Average
Precision (mAP) and the weighted consolidated metric NDS as $P_{C}$ for
nuScenes-R and L2-mAP and L2-mAPH as $P_{C}$ for Waymo-R.

##  4 Benchmark Existing Methods

We investigate and evaluate existing popular LiDAR-camera fusion methods with
opening source code on our benchmark, including PointAugmenting [34], MVX-Net
[31], and TransFusion [1]. In addition, we also evaluate a LiDAR-only method,
CenterPoint [20], and a camera-only method, DETR3D [38], for better
comparison. It is worth noting that the metrics on waymo dataset focus on
intersection of union (IoU). However, strictly calculating the IoU of 3D
bounding boxes is quite challenging for camera-based methods. Thus we reduce
the IoU threshold to 0.3 and report the vehicle class for DETR3d on Waymo
dateset.

Table 2: Benchmarking the robustness of state-of-the-art methods in all seven scenarios of the nuScenes-R and Waymo-R. M denotes input modality, camera (C) and LiDAR (L), of state-of-the-art methods.  Approach | M |  |  |  | LiDAR | Camera
---|---|---|---|---|---|---
$P_{C}$ | $mP_{R}$ | $R$ | Stuck | FOV | Object | Stuck | Missing | Occlusion | Calib
nuScenes-R (mAP / NDS)
CenterPoint [43] | L | 56.8 / 65.0 | 23.4 / 46.3 | 0.41 / 0.71 | 26.1 / 47.5 | 15.6 / 43.0 | 28.4 / 48.5 | - | - | - | -
DETR3D [38] | C | 34.9 / 43.4 | 17.6 / 31.5 | 0.50 / 0.73 | - | - | - | 17.3 / 32.3 | 14.5 / 29.9 | 14.3 / 29.0 | 24.2 / 35.0
PointAugmenting [34] | LC | 46.9 / 55.6 | 33.7 / 48.3 | 0.72 / 0.87 | 25.3 / 43.5 | 13.3 / 37.7 | 21.3 / 39.4 | 42.1 / 52.8 | 37.0 / 49.8 | 40.7 / 52.2 | 43.6 / 53.8
MVX-Net [31] | LC | 61.0 / 66.1 | 38.4 / 53.6 | 0.63 / 0.81 | 35.2 / 51.4 | 17.6 / 43.1 | 34.0 / 51.1 | 48.3 / 58.8 | 32.7 / 50.6 | 45.5 / 57.6 | 50.8 / 59.9
TransFusion [1] | LC | 66.9 / 70.9 | 52.8 / 63.1 |  0.79 / 0.89 | 33.4 / 52.3 | 20.3 / 45.8 | 34.6 / 53.6 | 65.9 / 70.2 | 64.9 / 69.7 | 65.5 / 70.0 | 66.5 / 70.7
Waymo-R(L2 mAP / L2 mAPH)
CenterPoint [43] | L | 66.0 / 63.4 | 30.6 / 29.4 | 0.46 / 0.46 | 29.5 / 28.3 | 30.3 / 29.1 | 32.1 / 30.9 | - | - | - | -
DETR3D [38] | C | 16.2 / 15.7 | 10.1 / 9.8 | 0.62 / 0.62 | - | - | - | 13.0 / 12.6 | 8.4 / 8.2 | 10.9 / 10.5 | 8.0 / 7.8
PointAugmenting[34] | LC | 52.5 / 50.7 | 39.6 / 38.3 | 0.75 / 0.76 | 24.7 / 23.9 | 24.3 / 23.4 | 26.2 / 25.3 | 51.7 / 50.0 | 50.4 / 48.6 | 50.3 / 48.6 | 49.8 / 48.1
MVX-Net [31] | LC | 59.7 / 54.1 | 44.3 / 40.1 | 0.74 / 0.74 | 27.5 / 24.9 | 28.8 / 25.6 | 28.7 / 26.0 | 58.2 / 52.7 | 55.9 / 50.5 | 56.4 / 51.1 | 54.9 / 49.6
TransFusion [1] | LC | 66.7 / 64.1 | 51.2 / 49.1 | 0.77 / 0.77 | 30.2 / 29.0 | 30.2 / 29.0 | 32.7 / 31.3 | 66.5 / 63.9 | 66.1 / 63.5 | 66.2 / 63.6 | 66.3 / 63.7
Stuck: Temporal misalignment for both modalities. FOV: Limited LiDAR FOV.
Object: LiDAR object failure.
Missing: Missing camera inputs. Occlusion: Camera Lens Occlusion. Calib:
Spatial misalignment of camera-to-world matrix.
Table 3: Robustness against LiDAR and camera modals of state-of-the-art architectures. In short, the robust metric (R) is computed by averaging the cases by the affecting modality. Approach | Modality |  | LiDAR | Camera
---|---|---|---|---
$P_{C}$ | $mP_{R}$ | $R$ | $mP_{R}$ | $R$
nuScenes-R (mAP / NDS)
CenterPoint [43] | L | 56.8 / 65.0 | 23.4 / 46.3 | 0.41 / 0.71 | - | -
DETR3D [38] | C | 34.9 / 43.4 | - | - | 17.6 / 31.5 | 0.50 / 0.73
PointAugmenting [34] | LC | 46.9 / 55.6 | 19.3 / 40.6 | 0.41 / 0.73 | 40.9 / 52.2 | 0.87 / 0.94
MVX-Net [31] | LC | 61.0 / 66.1 | 26.4 / 47.3 |  0.43 / 0.72 | 44.3 / 56.7 | 0.73 / 0.86
TransFusion [1] | LC | 66.9 / 70.9 | 26.9 / 49.1 | 0.40 / 0.69 | 65.7 / 70.1 |  0.98 / 0.99
Waymo-R(L2 mAP / L2 mAPH)
CenterPoint [43] | L | 66.0 / 63.4 | 30.6 / 29.4 | 0.46 / 0.46 | - | -
DETR3D [38] | C | 16.2 / 15.7 | - | - | 10.1 / 9.8 | 0.62 / 0.62
PointAugmenting[34] | LC | 52.5 / 50.7 | 25.1 / 24.2 | 0.48 / 0.48 | 50.6 / 48.8 | 0.96 / 0.96
MVX-Net [31] | LC | 59.7 / 54.1 | 28.3 / 25.5 | 0.47 / 0.47 | 56.4 / 51.0 | 0.94 / 0.94
TransFusion [1] | LC | 66.7 / 64.1 | 31.0 / 29.8 | 0.46 / 0.46 | 66.3 / 63.7 | 0.99 / 0.99

###  4.1 Benchmark results

The fusion robustness results are shown in Table 2. Moreover, to analyze the
robustness of models against LiDAR and camera disruptions, we present the
$mP_{R}$ and $R$ of LiDAR and camera modal separately in Table 3.

In general, existing methods perform poorly on our robust fusion benchmark as
shown in Table 2, and there is vast room for improvement. Especially, for all
LiDAR-camera fusion methods shown in Table 3, the LiDAR robustness is worse
than camera robustness. Among the LiDAR-camera fusion methods we investigated,
TransFusion achieves the overall best robustness. It is worth noting that the
robustness against camera noise of TransFusion is unexpectedly outstanding.
The average NDS of TransFusion only decreases from 70.9 to 70.1 on nuScenes-R
for camera failure and misalignment cases. However, the robustness against
LiDAR noise of TransFusion is worse than other fusion methods. We speculate
that this is mainly due to the fact that camera information has a small effect
on TransFusion. There is only a slight improvement (1.5 NDS on nuScenes) when
fusing camera information. Thus, when the camera information is missing or
defective, the overall performance of TransFusion is not affected.

When comparing the performance of LiDAR-camera fusion methods with single
modality methods on our benchmark, we find all fusion methods have stronger
robustness on both LiDAR and camera modality than single modality methods.
This indicates that when encountering imperfect single modality inputs, the
fusion methods can utilize another modality information to enhance the feature
and predict the final outputs.

Table 4: Results of the limited LiDAR field-of-view case. The angle ranges in brackets mean the visible angle range. ($-0$, $0$) means the extreme case when all LiDAR points are missing. Approach | Modality | nuScenes-R (mAP / NDS) | Waymo-R(L2 mAP / L2 mAPH)
---|---|---|---
clean | ($-\pi/2$, $\pi/2$) | ($-\pi/3$, $\pi/3$) | ($-0$, $0$) | clean | ($-\pi/2$, $\pi/2$) | ($-\pi/3$, $\pi/3$) | ($-0$, $0$)
CenterPoint [43] | L | 56.8 / 65.0 | 23.5 / 47.7 | 15.6 / 43.0 | 0 / 0 | 66.0 / 63.4 | 36.6 / 35.2 | 30.3 / 29.1 | 0 / 0
PointAugmenting [34] | LC | 46.9 / 55.6 | 19.5 / 41.2 | 13.3 / 37.7 | 0 / 0 | 52.5 / 50.7 | 29.4 / 28.3 | 24.3 / 23.4 | 0 / 0
MVX-Net [31] | LC | 61.0 / 66.1 | 26.0 / 47.8 | 17.6 / 43.1 | 0 / 0 | 59.7 / 54.1 | 34.5 / 30.8 | 28.8 / 25.6 | 0 / 0
TransFusion [1] | LC | 66.9 / 70.9 | 29.3 / 51.4 | 20.3 / 45.8 | 0 / 0 | 66.7 / 64.1 | 36.8 / 35.3 | 30.2 / 29.0 | 0 / 0

###  4.2 A complete analysis of each noisy data case

Here, we analyze the robustness of existing popular fusion methods on each
noisy case proposed in Section 3.2.

####  4.2.1 Noisy LiDAR Data

##### Limited LiDAR field-of-view.

We investigate the situations when the LiDAR points with limited field-of-view
in angle range ($-\pi/3$, $\pi/3$), ($-\pi/2$, $\pi/2$) and ($-0$, $0$). The
angle range of ($-0$, $0$) is an extreme case when the LiDAR sensor is
completely damaged. The results are shown in Table 4. For both LiDAR-only and
fusion methods, their performance decreases largely in three situations.
Especially, in the extreme case where all LiDAR points are missing, current
fusion methods fail to predict any objects like the LiDAR-only method. Thus,
for existing fusion methods, the LiDAR modality is the main modality and the
camera modality is auxiliary. The prediction results of existing fusion
methods largely rely on LiDAR information. There is considerable room for
improvement on fusion robustness. An ideal fusion model should still work as
long as there is single modality input.

Table 5: Results of the LiDAR object failure case.

Approach | Modality | nuScenes-R (mAP / NDS) | Waymo-R(L2 mAP / L2 mAPH)
---|---|---|---
clean | object failure | clean | object failure
CenterPoint [43] | L | 56.8 / 65.0 | 28.4 / 48.5 | 66.0 / 63.4 | 32.1 / 30.9
PointAugmenting [34] | LC | 46.9 / 55.6 | 21.3 / 39.4 | 52.5 / 50.7 | 26.2 / 25.3
MVX-Net [31] | LC | 61.0 / 66.1 | 34.0 / 51.1 | 59.7 / 54.1 | 28.7 / 26.0
TransFusion [1] | LC | 66.9 / 70.9 | 34.6 / 53.6 | 66.7 / 64.1 | 32.7 / 31.3

Table 6: Results of the missing camera inputs case. {X} denotes the location
of the missing camera, while the last column indicates the case only keeping
the input from the front camera. Note that there is no back camera in the
Waymo Open Dataset.

Approach |  Modality | nuScenes-R (mAP / NDS)
---|---|---
clean | {F} | {B} | {FL} | {FR} | {BL} | {BR} | Keeping F
DETR3D [38] | C | 34.9 / 43.4 | 25.8 / 39.2 | 23.9 / 38.0 | 28.9 / 39.5 | 29.1 / 39.8 | 30.0 / 40.7 | 29.7 / 40.2 | 3.3 / 20.5
PointAugmenting [34] | LC | 46.9 / 55.6 | 42.4 / 53.0 | 41.3 / 52.5 | 43.6 / 53.8 | 45.8 / 54.6 | 45.2 / 54.7 | 44.9 / 54.6 | 31.6 / 46.5
MVX-Net [31] | LC | 61.0 / 66.1 | 47.8 / 59.4 | 45.8 / 58.4 | 53.6 / 61.9 | 54.1 / 62.5 | 55.2 / 63.1 | 54.6 / 62.6 | 17.5 / 41.7
TransFusion [1] | LC | 66.9 / 70.9 | 65.3 / 70.1 | 66.0 / 70.4 | 66.2 / 70.4 | 66.4 / 70.5 | 66.3 / 70.5 | 66.3 / 70.5 | 64.4 / 69.3
Approach | Modality | Waymo-R(L2 mAP / L2 mAPH)
clean | {F} | {B} | {FL} | {FR} | {BL} | {BR} | Keeping F
DETR3D [38] | C | 16.2 / 15.7 | 9.2 / 8.8 | - | 13.4 / 13.0 | 14.2 / 13.8 | 14.0 / 13.6 | 14.4 / 14.0 | 7.7 / 7.5
PointAugmenting [34] | LC | 52.5 / 50.7 | 50.6 / 48.9 | - | 51.8 / 50.0 | 52.1 / 50.3 | 51.8 / 50.0 | 51.9 / 50.1 | 50.2 / 48.4
MVX-Net [31] | LC | 59.7 / 54.1 | 57.1 / 51.7 | - | 57.5 / 52.2 | 58.1 / 52.7 | 58.5 / 53.1 | 58.9 / 53.5 | 54.3 / 49.2
TransFusion [1] | LC | 66.7 / 64.1 | 66.3 / 63.7 | - | 66.5 / 64.0 | 66.4 / 63.8 | 66.4 / 63.8 | 66.5 / 63.9 | 65.8 / 63.2
Camera location abbr. F: front. B: back. FL: front-left. FR: front-right. BL:
back-left. BR: back-right.

##### LiDAR object failure.

The results of the LiDAR object failure case are shown in Table 6. We can find
that, with 50% probability to drop all points of the objects, the performance
of both LiDAR-only and LiDAR-camera fusion methods reduce by half
approximately. This indicates current fusion methods fail to work when the
foreground LiDAR points are missing, even the objects appear in the images.
From another perspective, it shows that, for the fusion mechanisms of current
LiDAR-camera fusion methods, camera information is not well exploited. The
fusion process still largely relies on LiDAR information.

####  4.2.2 Noisy Camera Sensor

##### Missing of camera inputs.

In the case of missing camera inputs, we consider several combinations of
cameras installed in different positions and report the results in Table 6, in
which we can find that the missing front camera or back camera (for nuScenes)
has a greater impact on the detection results. When all cameras except the
front camera are missing, the performance of PointAugmenting and TransFusion
decreases no more than 50% on both nuScenes-R and Waymo-R. This demonstrates
that the robustness of PointAugmenting and TransFusion against camera noise is
much better than the other two methods. Besides, the performance degradation
on Waymo-R is much smaller than that on nuScenes-R , which indicates the
robustness on the various datasets is different.

##### Occlusion of camera lens.

The results for the case of the dirty camera lens are shown in Table 7. We
surprisingly find all methods have inferior performances when compared to the
previous missing camera inputs experiments. It seems the modern deep learning
method is akin to a black image instead of occluded ones. This shows a
potential new research direction to design a better image feature extractor to
address the robustness issue.

Table 7: Results of the camera occlusion case. Approach | Modality | nuScenes-R (mAP / NDS) | Waymo-R(L2 mAP / L2 mAPH)
---|---|---|---
clean | occlusion | clean | occlusion
DETR3D [38] | C | 34.9 / 43.4 | 14.3 / 29.0 | 16.2 / 15.7 | 10.9 / 10.5
PointAugmenting [34] | LC | 46.9 / 55.6 | 40.7 / 52.2 | 52.5 / 50.7 | 50.3 / 48.6
MVX-Net [31] | LC | 61.0 / 66.1 | 45.5 / 57.6 | 59.7 / 54.1 | 56.4 / 51.1
TransFusion [1] | LC | 66.9 / 70.9 | 65.5 / 70.0 | 66.7 / 64.1 | 66.2 / 63.6

(a) Stuck camera frame

(b) Stuck LiDAR frame

Figure 8: Temporal misalignment case. The solid line denotes the discrete
selection. The dash line denotes the consecutive selection.

####  4.2.3 Ill-synchronization

##### Spatial misalignment.

For spatial misalignment, the effect of the noise rotation and translation
matrix on fusion models is comparable to that of the camera sensor failure
case, as shown in Table 8. We find that the TransFusion is the most robust
cases that on average suffers less than 1pp. compared to the clean settings,
where the DETR3D is the most sensitive to the spatial misalignment.

Table 8: Results of spatial misalignment cases. Approach | Modality | nuScenes-R (mAP / NDS) | Waymo-R(L2 mAP / L2 mAPH)
---|---|---|---
clean | misalignment | clean | misalignment
DETR3D [38] | C | 34.9 / 43.4 | 24.2 / 35.0 | 16.2 / 15.7 | 8.0 / 7.8
PointAugmenting [34] | LC | 46.9 / 55.6 | 43.6 / 53.8 | 52.5 / 50.7 | 49.8 / 48.1
MVX-Net [31] | LC | 61.0 / 66.1 | 50.8 / 59.9 | 59.7 / 54.1 | 54.9 / 49.6
TransFusion [1] | LC | 66.9 / 70.9 | 66.5 / 70.7 | 66.7 / 64.1 | 66.3 / 63.7

##### Temporal misalignment.

For temporal misalignment, we explore 9 levels of severity, in which the
proportion of stuck frames is from 10% to 90% among all frames in a regular
step. The results are shown in Figure 8. A trend can be observed that the
performance degradation of all methods is linear to the percentage of stuck
frames among all frames in both discrete selection and consecutive selection.
Thus, to reduce the cost of the evaluation, we only choose the case where the
stuck frames are 50% of all frames, whose value is approximately equal to the
average of 9 levels of severity. Interestingly, although TransFusion performs
well against the stuck camera frame case, we can observe that the perfromance
of TransFusion decreases faster than other fusion methods when the LiDAR stuck
frame ratio increases, showing that it is more sensitive than other fusion
methods.

##  5 Discussion and Future Work

Though the main contribution of this work is to provide a systematic overview
of different aspects of the perception system and construct a robustness
benchmark, we nonetheless provide a simple yet effective baseline method,
robustness finetuning, to improve the robustness of these methods.

###  5.1 A simple baseline to improve the robustness: Robust Finetuning

We provide a simple baseline method by treating our toolkit as a data
augmentation method to enrich the training data as the first attempt to
improve the robustness of performance. Specifically, we use the toolkit to
transform the training data. Different from the evaluation, where
reproducibility is a must, we can randomly generate corresponding noisy data
during training. However, in practice, we discover that transforming all data
into a noisy format will significantly decrease the performance when there is
no such noise. To this end, we propose a cascaded augmentation policy during
training: i) if a randomly sampled float number is higher than an augmentation
probability $p_{a}$, proceed, otherwise use the normal clean data; ii)
sampling one transformation from all robustness cases according to the
probability distribution $p_{o}$, then proceeding data with the
transformation. See Section 6.3 for more details.

Table 9: Robust training results. Approach | Modality |  | Overall | Lidar | Camera
---|---|---|---|---|---
$P_{C}$ | $mP_{R}$ | $R$ | $mP_{R}$ | $R$ | $mP_{R}$ | $R$
nuScenes-R (mAP / NDS)
MVX-Net [31] | LC | 61.0 / 66.1 | 38.4 / 53.6 | 0.63 / 0.81 | 26.4 / 47.3 | 0.43 / 0.72 | 44.3 / 56.7 | 0.73 / 0.86
MVX-Net + finetune | LC | 59.4 / 65.0 | 40.9 / 54.8 | 0.69 / 0.84 | 24.6 / 46.1 | 0.41 / 0.71 | 49.1 / 59.2 | 0.83 / 0.91
Waymo-R(L2 mAP / L2 mAPH)
MVX-Net [31] | LC | 59.7 / 54.1 | 44.3 / 40.1 | 0.74 / 0.74 | 28.3 / 25.5 | 0.47 / 0.47 | 56.4 / 51.0 | 0.94 / 0.94
MVX-Net + finetune | LC | 59.5 / 54.0 | 47.7 / 43.1 | 0.80 / 0.80 | 27.8 / 25.2 | 0.47 / 0.47 | 57.6 / 52.1 | 0.97 / 0.96

We select the MVXNet to study the effectiveness of our method, as it has the
most balanced LiDAR and camera performance, and report the results in Table 9.
We observe that, though applying such robustness training slightly
deteriorates the performance on the clean dataset, it significantly improves
the robustness, where the mean robustness $R$ improves from 0.63 and 0.81 to
0.69 and 0.84 in terms of mAP and NDS on nuScenes-R , and from 0.74 to 0.80 in
terms of L2 mAP and mAPH on Waymo-R. However, we can still see a large gap
between the robust benchmark and the clean ones, evidencing there is an actual
research gap in this research direction.

###  5.2 Future research directions

In general, we believe an ideal sensor fusion framework should be able to do
the following: i) given both modality data, it can significantly surpass the
performance of single modality methods; ii) when there is a disruption of one
modality, the performance should not be worse than the single modality method
of the other. Currently, this approach is handled by using comprehensive post-
processing techniques of the perception system. We hope our robust benchmark
can be a tool for the community to fully exploit this research direction to
develop truly robust methods that can be deployed on realistic vehicles.

## References

  * [1] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai.  Transfusion: Robust lidar-camera fusion for 3d object detection with transformers.  arXiv preprint arXiv:2203.11496, 2022.
  * [2] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  arXiv preprint arXiv:1903.11027, 2019.
  * [3] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuScenes: A multimodal dataset for autonomous driving.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [4] Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia.  Multi-view 3d object detection network for autonomous driving.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [5] MMDetection3D Contributors.  Mmdetection3d: Open-mmlab next-generation platform for general 3d object detection.  https://github.com/open-mmlab/mmdetection3d, 2020.
  * [6] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [7] Junjie Huang, Guan Huang, Zheng Zhu, and Dalong Du.  Bevdet: High-performance multi-camera 3d object detection in bird-eye-view.  arXiv preprint arXiv:2112.11790, 2021.
  * [8] Tengteng Huang, Zhe Liu, Xiwu Chen, and X. Bai.  EPNet: Enhancing point features with image semantics for 3d object detection.  In European Conference on Computer Vision (ECCV), 2020.
  * [9] Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven L. Waslander.  Joint 3d proposal generation and object detection from view aggregation.  In IEEE International Conference on Intelligent Robots and Systems (IROS), 2018.
  * [10] Abhinav Kumar, Garrick Brazil, and Xiaoming Liu.  Groomed-nms: Grouped mathematically differentiable nms for monocular 3d object detection.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [11] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom.  Pointpillars: Fast encoders for object detection from point clouds.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [12] Yingwei Li, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen, Bo Wu, Yifeng Lu, Denny Zhou, et al.  Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection.  arXiv preprint arXiv:2203.08195, 2022.
  * [13] Zhichao Li, Feng Wang, and Naiyan Wang.  Lidar r-cnn: An efficient and universal 3d object detector.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [14] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai.  Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers.  arXiv preprint arXiv:2203.17270, 2022.
  * [15] Ming Liang, Binh Yang, Yun Chen, Rui Hu, and R. Urtasun.  Multi-task multi-sensor fusion for 3d object detection.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [16] Ming Liang, Binh Yang, Shenlong Wang, and R. Urtasun.  Deep continuous fusion for multi-sensor 3d object detection.  In European Conference on Computer Vision (ECCV), 2018.
  * [17] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  * [18] Zongdai Liu, Dingfu Zhou, Feixiang Lu, Jin Fang, and Liangjun Zhang.  Autoshape: Real-time shape-aware monocular 3d object detection.  In IEEE International Conference on Computer Vision (ICCV), 2021\.
  * [19] Yan Lu, Xinzhu Ma, Lei Yang, Tianzhu Zhang, Yating Liu, Qi Chu, Junjie Yan, and Wanli Ouyang.  Geometry uncertainty projection network for monocular 3d object detection.  In IEEE International Conference on Computer Vision (ICCV), 2021\.
  * [20] Ramin Nabati and Hairong Qi.  Centerfusion: Center-based radar and camera fusion for 3d object detection.  In IEEE Winter Conference on Applications of Computer Vision (WACV), 2021.
  * [21] Su Pang, Daniel Morris, and Hayder Radha.  Clocs: Camera-lidar object candidates fusion for 3d object detection.  In IEEE International Conference on Intelligent Robots and Systems (IROS), 2020.
  * [22] Dennis Park, Rares Ambrus, Vitor Guizilini, Jie Li, and Adrien Gaidon.  Is pseudo-lidar needed for monocular 3d object detection?  In IEEE International Conference on Computer Vision (ICCV), 2021\.
  * [23] Jonah Philion and Sanja Fidler.  Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d.  In European Conference on Computer Vision (ECCV), 2020.
  * [24] C. Qi, W. Liu, Chenxia Wu, Hao Su, and L. Guibas.  Frustum pointnets for 3d object detection from rgb-d data.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [25] C. Qi, Hao Su, Kaichun Mo, and L. Guibas.  PointNet: Deep learning on point sets for 3d classification and segmentation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [26] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  In Neural Information Processing Systems (NeurIPS), 2017.
  * [27] Cody Reading, Ali Harakeh, Julia Chae, and Steven L Waslander.  Categorical depth distribution network for monocular 3d object detection.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [28] Thomas Roddick, Alex Kendall, and R. Cipolla.  Orthographic feature transform for monocular 3d object detection.  In British Machine Vision Conference (BMVC), 2019.
  * [29] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li.  PV-RCNN: Point-voxel feature set abstraction for 3d object detection.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [30] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li.  PointRCNN: 3d object proposal generation and detection from point cloud.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [31] Vishwanath A Sindagi, Yin Zhou, and Oncel Tuzel.  Mvx-net: Multimodal voxelnet for 3d object detection.  In International Conference on Robotics and Automation (ICRA), 2019.
  * [32] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al.  Scalability in perception for autonomous driving: Waymo open dataset.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [33] Sourabh Vora, Alex H Lang, Bassam Helou, and Oscar Beijbom.  Pointpainting: Sequential fusion for 3d object detection.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [34] Chunwei Wang, Chao Ma, Ming Zhu, and Xiaokang Yang.  Pointaugmenting: Cross-modal augmentation for 3d object detection.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [35] Li Wang, Liang Du, Xiaoqing Ye, Yanwei Fu, Guodong Guo, Xiangyang Xue, Jianfeng Feng, and Li Zhang.  Depth-conditioned dynamic message propagation for monocular 3d object detection.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [36] Li Wang, Li Zhang, Yi Zhu, Zhi Zhang, Tong He, Mu Li, and Xiangyang Xue.  Progressive coordinate transforms for monocular 3d object detection.  In Neural Information Processing Systems (NeurIPS), 2021.
  * [37] Yue Wang, Alireza Fathi, Abhijit Kundu, David A. Ross, Caroline Pantofaru, Thomas A. Funkhouser, and Justin M. Solomon.  Pillar-based object detection for autonomous driving.  In European Conference on Computer Vision (ECCV), 2020.
  * [38] Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon.  Detr3d: 3d object detection from multi-view images via 3d-to-2d queries.  In Conference on Robot Learning (CoRL), 2022.
  * [39] Enze Xie, Zhiding Yu, Daquan Zhou, Jonah Philion, Anima Anandkumar, Sanja Fidler, Ping Luo, and Jose M. Alvarez.  M${}^{\mbox{2}}$bev: Multi-camera joint 3d detection and segmentation with unified birds-eye view representation.  arXiv preprint arXiv:2204.05088, 2022.
  * [40] Shaoqing Xu, Dingfu Zhou, Jin Fang, Junbo Yin, Bin Zhou, and Liangjun Zhang.  FusionPainting: Multimodal fusion with adaptive attention for 3d object detection.  In IEEE International Conference on Intelligent Transportation Systems (ITSC), 2021.
  * [41] Yan Yan, Yuxing Mao, and B. Li.  SECOND: Sparsely embedded convolutional detection.  Sensors, 2018.
  * [42] Zetong Yang, Y. Sun, Shu Liu, and Jiaya Jia.  3DSSD: Point-based 3d single stage object detector.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [43] Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl.  Center-based 3d object detection and tracking.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [44] Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl.  Multimodal virtual point 3d detection.  In Neural Information Processing Systems (NeurIPS), 2021.
  * [45] Jin Hyeok Yoo, Yeocheol Kim, Ji Song Kim, and J. Choi.  3D-CVF: Generating joint camera and lidar features using cross-view spatial feature fusion for 3d object detection.  In European Conference on Computer Vision (ECCV), 2020.
  * [46] Yunpeng Zhang, Jiwen Lu, and Jie Zhou.  Objects are different: Flexible monocular 3d object detection.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [47] Yin Zhou and Oncel Tuzel.  VoxelNet: End-to-end learning for point cloud based 3d object detection.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [48] Yunsong Zhou, Yuan He, Hongzi Zhu, Cheng Wang, Hongyang Li, and Qinhong Jiang.  Monocular 3d object detection: An extrinsic parameter free approach.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.

##  6 Appendix

###  6.1 Autonomous driving datasets

nuScenes Dataset. nuScenes is a large-scale autonomous-driving dataset for 3D
detection, consisting of 700, 150 and 150 scenes for training, validation, and
testing, respectively. Each frame contains one point cloud and six calibrated
images that cover 360 fields-of-view. For 3D detection, the main metrics are
mean Average Precision (mAP) and nuScenes detection score (NDS). The mAP is
defined by the BEV center distance with thresholds of 0.5m, 1m, 2m, 4m,
instead of the IoUs of bounding boxes. NDS is a consolidated metric of mAP and
other metric scores, such as average translation error and average scale
error.

Waymo Open Dataset. Waymo Open Dataset is an another large-scale dataset for
autonomous driving, which contains 798 training, 202 validation, and 150
testing sequences. Each sequence has about 200 frames with LiDAR points and
camera images, which are collected by five LiDAR sensors and five pinhole
cameras. The official metrics are mean Average Precision (mAP) and mean
Average Precision weighted by Heading (mAPH). The mAP and mAPH are defined
based on the 3D IoU with the threshold of 0.7 for vehicles and 0.5 for
pedestrians and cyclists. The measures are reported based on the distances
from objects to sensor, i.e., 0-30m, 30-50m and >50m, respectively. Besides,
two difficulty levels, LEVEL 1 (boxes with more than five LiDAR points) and
LEVEL 2 (boxes with at least one LiDAR point), are considered.

###  6.2 Implementation Details of Fusion Models

For the sake of better reproducibility, we re-implement the MVX-Net into the
MMDetection3D[5] framework. Here we detail the implementation settings.

##### MVX-Net.

Following the original paper [31], we use PointPillars [11] as the LiDAR
stream and ResNet50[6] with FPN [17] as the image stream. At the fusion stage,
we project each LiDAR point to all images from different views to acquire the
corresponding image features from the deep networks. Then, we average the
features on the channel dimension and then concatenate all features from
different views with the original LiDAR point feature before the downstream
task. During training, we use the same training schedules and hyper-parameters
of PointPillars, where Adam optimizer is used with learning rate 0.001, weight
decay 0.01, batch size 16, epoch 12, and learning rate decay to 1/10 at 8 and
11 epochs.

On the nuScenes dataset, we set the detection region of interest to
$[-50m,50m]$ for the X and Y axis, and $[-5m,3m]$ for the Z axis. The pillar
size is kept as $[0.25m,0.25m]$. The image feature extractor is trained for 36
epochs on nuImage[2]. On Waymo dataset, we set the detection range to
$[-74.88m,74.88m]$ for the X and Y axis, and $[-2m,4m]$ for the Z axis. The
pillar size is hold as $[0.32m,0.32m]$. The image feature extractor is trained
for 36 epochs on Waymo of the 2D detection task.

Table 10: Ablation of robustness finetuning Approach | Aug |  |  |  | Lidar | Camera
---|---|---|---|---|---|---
$P_{C}$ | $mP_{R}$ | $R$ | Stuck | FOV | Object | Stuck | Missing | Occlusion | Calib
nuScenes-R (mAP / NDS) |
MVX-Net[31] | None | 61.0 / 66.1 | 38.4 / 53.6 | 0.63 / 0.81 | 35.2 / 51.4 | 17.6 / 43.1 | 34.0 / 51.1 | 48.3 / 58.8 | 32.7 / 50.6 | 45.5 / 57.6 | 50.8 / 59.9
LIDAR stuck | 58.4 / 64.2 | 35.6 / 51.5 | 0.61 / 0.80 | 39.1 / 52.1 | 17.8 / 42.8 | 34.2 / 51.4 | 42.4 / 54.9 | 28.2 / 47.6 | 41.0 / 54.8 | 46.6 / 56.9
FOV | 54.2 / 59.1 | 33.9 / 49.9 | 0.63 / 0.84 | 33.3 / 49.5 | 18.4 / 44.0 | 30.3 / 47.4 | 41.4 / 50.8 | 27.7 / 47.0 | 40.5 / 54.9 | 45.7 / 56.0
Object | 56.8 / 60.9 | 36.5 / 51.4 | 0.63 / 0.84 | 39.1 / 52.1 | 17.8 / 42.8 | 33.1 / 49.9 | 44.3 / 53.7 | 29.9 / 48.5 | 42.8 / 55.1 | 48.2 / 57.5
Camera stuck | 58.6 / 64.7 | 38.7 / 53.7 | 0.64 / 0.84 | 31.0 / 48.9 | 17.2 / 42.9 | 32.7 / 50.0 | 51.3 / 60.4 | 35.2 / 51.8 | 45.7 / 57.6 | 52.0 / 60.9
Missing | 60.9 / 66.0 | 41.1 / 55.0 | 0.66 / 0.83 | 34.9 / 51.1 | 18.3 / 43.6 | 34.1 / 50.8 | 48.7 / 59.1 | 45.1 / 57.1 | 48.2 / 58.9 | 51.1 / 60.2
| Calib | 60.0 / 65.4 | 39.4 / 54.0 |  | 33.6 / 50.3 | 17.8 / 43.3 | 33.6 / 50.7 | 50.0 / 59.6 | 34.6 / 51.3 | 45.8 / 57.6 | 54.9 / 62.2
Table 11: Probability distribution of transformation selection during finetuning. Transformation | LiDAR | Camera
---|---|---
Stuck | FOV | Object | Stuck | Missing | Occlusion | Calib
Probability | 0 | 0 | 0 | 1/3 | 1/3 | 0 | 1/3

###  6.3 Ablation study of robustness finetuning

To determine the probability distribution $p_{o}$ of all transformations, we
first analyze the effects of each individual transformation, as shown in Table
10. It’s worth noticing that the transformation of camera lens occlusion case
is included during finetuning stage, since the masks for simulating the
occlusion is only available during validation. From Table 10, we can find
that, compared with MVX-Net baseline trained on clean data, finetuning with
noisy LiDAR data decreases the average performance $mP_{R}$ on each noisy
fusion cases. By contrast, finetuning with noisy camera data improves the
average performance $mP_{R}$. Thus, the sampling probabilities of noisy LiDAR
transformations are set to be zero and the sampling probabilities of remaining
noisy camera transformations, i.e., camera-stuck and missing of camera input,
are set to be $1/3$. The final probability distribution $p_{o}$ is listed in
Table 11.


# Chiral dynamics in soft triaxial nuclei

B. Li  P.W. Zhao<EMAIL_ADDRESS>J. Meng  State Key Laboratory of Nuclear
Physics and Technology, School of Physics, Peking University, Beijing 100871,
China

###### Abstract

The chirality in soft triaxial nuclei is investigated for the first time with
the time-dependent and tilted axis cranking covariant density functional
theories on a three-dimensional space lattice in a microscopic and self-
consistent way. Taking the puzzling chiral nucleus 106Ag as an example, the
experimental energies of the observed nearly degenerate bands are well
reproduced without any free parameters beyond the well-defined density
functional. A novel chiral mode in soft triaxial nuclei is newly revealed from
the microscopic dynamics of the total angular momentum. This opens a new
research area for the study of chirality, particularly in relation to soft
nuclear shapes.

###### keywords:

Nuclear chirality , Soft triaxial nuclei , Covariant density functional theory

††journal: Physics Letters B

##  1 Introduction

Chirality is a well-known phenomenon in many fields, such as chemistry,
biology, molecular physics and material science. In nuclear physics, chirality
was originally suggested by Frauendorf and Meng in triaxial nuclei [1], which
have three different principal axes of the ellipsoidal density distribution,
i.e., the short, intermediate, and long axes. When a triaxial nucleus rotates
around an axis lying outside the three planes spanned by the principal axes,
the three angular momentum vectors along the short, intermediate, and long
axes in the body-fixed frame could couple to each other in either a left- or
right-handed mode. The two chiral systems are related by the chiral operator
$TR(\pi)$ that combines time reversal $T$ and spatial rotation by $\pi$. The
broken chiral symmetry in the body-fixed frame should be restored in the
laboratory frame. This leads to the so-called chiral doublet bands, which
consist of a pair of nearly degenerate rotational bands.

It is apparent that the triaxial shape is an essential prerequisite for
nuclear chirality. An ideal picture of nuclear chirality is associated with a
rigid triaxial shape, which means that the nuclear surface arising from
collective motion of many nucleons has only small fluctuations around the
equilibrium shape. While the study of nuclear chirality is so far focused on
the rigid triaxial shape, the chirality in soft triaxial nuclei is poorly
known. However, most triaxial nuclei are “soft”, which implies large
fluctuations around the equilibrium shape for the nuclear surface. This brings
a major challenge to the understanding of nuclear chirality.

So far, evidences of chiral doublet bands have been reported experimentally in
$A\sim 80$, 100, 130, and 190 mass regions; see e.g., Refs. [2, 3, 4, 5, 6, 7,
8, 9, 10, 11, 12, 13]. However, the manifestation of chirality in realistic
nuclei is usually very complicated due to the softness of nuclear shape [14,
15]. In particular, there are numerous experimental indications suggesting
that triaxial softness may have significant implications for the chiral
phenomena; see for examples, Refs. [2, 4, 16, 17, 6, 8, 18, 19, 11]. One of
the most intriguing and elusive chiral candidates with soft triaxiality is
106Ag, in which a set of strongly coupled doublet bands, crossing each other
at spin $I\sim 14\hbar$, were first observed in Ref. [6]. Systematics of
chiral doublet bands in the $A\sim 100$ region indicates a strong influence of
triaxial softness on the stability of the chiral geometry. The two pursuant
lifetime measurements [18, 19] have reported consistent data for the $B(E2)$
and $B(M1)$ values but contrasting interpretations. In particular, a third
band lying only slightly higher than the two existing crossing bands in 106Ag
was reported in Ref. [18]. Similar band structure has also been observed in
134Pr [2, 16, 20].

Theoretically, the chiral nature of the intriguing three closing bands has
been extensively studied with phenomenological models, which however, could
lead to distinct interpretations, because they are all adjusted to the data in
one way or another with schematic assumptions. For instance, the nuclear shape
is assumed to be rigid in the particle rotor model, which is amongst the most
widely used models for nuclear chirality [1, 21, 22, 23, 24, 25]. The
interacting boson fermion-fermion model introduces a cubic interaction to
simulate the triaxiality and its softness [16, 26, 27], while the
corresponding strength is unknown and needs to be fitted to data.

A microscopic treatment is thus important to explore the origin of nuclear
chirality in soft triaxial nuclei. The tilted axis cranking (TAC) approach
allows one to study nuclear chirality based on a microscopic mean field [28].
For a more fundamental investigation, self-consistent methods based on more
realistic nuclear interactions have been developed in the framework of
nonrelativistic and relativistic density functional theories (DFTs) [29, 30,
31]. In particular, the chiral conundrum in 106Ag was studied with the tilted
axis cranking covariant density functional theory (TAC-CDFT) [32], and it
showed that the potential energy surface is rather soft in the triaxial
direction.

The TAC approach can only describe the lower band of the chiral doublet bands,
since it gives either one achiral solution or two degenerate chiral ones while
cannot describe the left-right excitation mode. In view of the recent
remarkable progress on the time-dependent covariant density functional theory
(TDCDFT) in three-dimensional (3D) lattice space [33, 34], the dynamics of the
chiral nucleus 135Nd was first studied in Ref. [35], and the upper band of the
chiral doublet bands was well reproduced by analyzing the dynamics of the so-
called _chiral precession_.

In this Letter, the chirality in soft triaxial nuclei is studied for the first
time with the state-of-the-art approach, i.e., TAC-CDFT and TDCDFT. Taking the
triaxial soft chiral nucleus 106Ag as an example, the observed nearly
degenerate bands are calculated without any free parameters beyond the well-
defined density functional. The impact of the triaxial softness on nuclear
chirality is clearly revealed from the microscopic dynamics of the chiral
excitations.

##  2  The lower band of chiral doublet bands

The starting point of covariant density functional theory (CDFT) is a
universal energy density functional [36, 37, 38, 39, 40]. For nuclear
chirality, the functional is transformed into a body-fixed frame rotating with
a constant angular velocity vector $\bm{\omega}$, which is along an arbitrary
direction in space, i.e., the three-dimensional TAC-CDFT [31]. The
corresponding Kohn-Sham equation for nucleons is a static Dirac equation,

$\left[\bm{\alpha}\cdot(\hat{\bm{p}}-\bm{V})+V^{0}+\beta(m+S)-\bm{\omega}\cdot\hat{\bm{J}}\right]\psi_{k}(\bm{r})=\varepsilon_{k}\psi_{k}(\bm{r}),$
(1)

where $\hat{\bm{J}}$ is the angular momentum, and the relativistic scalar
$S(\bm{r})$ and vector $V^{\mu}(r)$ fields are connected in a self-consistent
way to the nucleon densities and current distributions, which are obtained
from the Dirac spinors $\psi_{k}(\bm{r})$. The complicated interplay between
the large Lorentz scalar and vector self-energies [41] allows the self-
consistent treatment of the spin degrees of freedom and the nuclear currents
induced by the spatial parts of the vector self-energies, which play an
essential role in rotating nuclei; see also Refs. [42, 40] for details. The
angular velocity $\bm{\omega}$ plays a role of a Lagrange multiplier, and it
is connected to the expectation values $\langle\hat{\bm{J}}\rangle$ and the
angular momentum quantum number (spin) $I$ by the semiclassical relation
$\langle\hat{\bm{J}}\rangle\cdot\langle\hat{\bm{J}}\rangle=I(I+1)$.

In this work, the density functional PC-PK1 [43] is employed, and the
calculations are free of additional parameters. The Dirac equation (1) is
solved in a 3D cubic box with the length $22.4$ fm and the mesh size $0.8$ fm
along the $x$, $y$, and $z$ axes. We focus on the chirality in the odd-odd
nucleus 106Ag, which was first reported experimentally in Ref. [6] with two
crossing bands, and a third close band was reported later in Ref. [18].
According to our previous study [32], it is expected that bands 2 and 3 could
be a pair of chiral twin bands based on a four-quasiparticle configuration
$\pi g_{9/2}\otimes\nu h_{11/2}(gd)^{2}$, while band 1 corresponds to a two-
quasiparticle configuration $\pi g_{9/2}\otimes\nu h_{11/2}$.

Figure 1: (color online). Potential energy curves (dashed line) against the
triaxial deformation $\gamma$ for 106Ag calculated with the TAC-CDFT at each
angular momentum with the configuration $\pi g_{9/2}\otimes\nu
h_{11/2}(gd)^{2}$. The solid circles denote the positions of the minimum
energies, and the thick lines highlight the region with the corresponding
energy rising within $500$ keV against the minima.

Figure 1 depicts the potential energy curves against the triaxial deformation
$\gamma$ at each angular momentum based on the four-quasiparticle
configuration $\pi g_{9/2}\otimes\nu h_{11/2}(gd)^{2}$. They are determined by
minimizing the total energy at each angular momentum with respect to the
$\beta$ deformation for a given value of $\gamma$. While the triaxial
deformation is not large at the energy minima (around $\gamma\simeq
5^{\circ}$), the potential energy curves are rather soft in the triaxial
direction at all angular momenta. The energy rise against the minimum is in
general less than 500 keV when the triaxial deformation reaches a significant
value $\gamma\simeq 20^{\circ}$. Considering the fact that the four-
quasiparticle configuration contains high-$j$ protons and neutrons in the
$g_{9/2}$ and $h_{11/2}$ shells, respectively, a chiral excitation mode could
thus be generated though deeply influenced by the $\gamma$ softness.

##  3  The higher band of chiral doublet bands

In order to describe the chiral excitations, the TDCDFT calculations are
carried out for the dynamical evolution of the system. In TDCDFT, the time-
dependent Kohn-Sham equation reads [33, 34],

$i\partial_{t}\psi_{k}(\bm{r},t)=[\bm{\alpha\cdot}(\hat{\bm{p}}-\bm{V})+V^{0}+\beta(m+S)]\psi_{k}(\bm{r},t),$
(2)

where the time-dependent fields $S(\bm{r},t)$ and $V^{\mu}(\bm{r},t)$ have the
same dependence on density and currents as in TAC-CDFT, so there are no
additional parameters introduced. Starting from the TAC-CDFT solutions at each
angular momentum, one can obtain the dynamical evolution of the system by
means of the time-dependent wave functions $\psi_{k}(\bm{r},t)$, and the
expectation value of the angular momentum is conserved during the time
evolution. As in our previous study of chiral precession [35], it is
convenient to analyze the chiral dynamics in the body-fixed rotating frame in
terms of the tilted angles $\theta_{J}$ and $\varphi_{J}$ of the total angular
momentum. Here, $\theta_{J}$ is the polar angle between the total angular
momentum and the long axis, and $\varphi_{J}$ is the azimuth angle between the
projection of the total angular momentum in the short-intermediate plane and
the short axis.

Figure 2: (color online). Time evolution of the azimuth angle $\varphi_{J}$
starting from an initial state taken from the total energy surface in the
$(\theta_{J},\varphi_{J})$ plane at the total angular momentum $I=10\hbar$
(top) and $I=18\hbar$ (bottom). Insets: Time-dependent trajectories of the
tilted angles $(\theta_{J},\varphi_{J})$. The stars denote the positions of
the energy minima for the total energy surface in the
$(\theta_{J},\varphi_{J})$ plane.

In the inset of Fig. 2(a), the time-dependent trajectory of the tilted angles
$\theta_{J}$ and $\varphi_{J}$ is depicted for the total angular momentum
$I=10\hbar$. The initial state is taken from the total energy surface in the
$(\theta_{J},\varphi_{J})$ plane, and it is away from the lowest energy state.
The trajectory can roughly be regarded as an ellipse centered at the location
of the minimum energy $\varphi_{J}=0$, while it is significantly distorted
with small oscillations in the $\varphi_{J}$ direction, which are essentially
induced by the $\gamma$ softness of 106Ag. From the time evolution of
$\varphi_{J}$ as depicted in Fig. 2(a), one can see even more clearly the two
explicit components with distinct frequencies.

One should keep in mind that for nuclei with a rigid shape, the nuclear
deformation barely changes with different orientations of the angular
momentum, while for a soft nucleus like 106Ag, the deformation can be easily
changed across the energy surface in the $(\theta_{J},\varphi_{J})$ plane. As
a result, for 106Ag, starting from an arbitrary state on the energy surface in
the $(\theta_{J},\varphi_{J})$ plane, except for the energy minimum, one could
in principle stimulate motions not only in the $(\theta_{J},\varphi_{J})$
space but also in the deformation space, in particular, in the $\gamma$
direction. This explains the two explicit components in the chiral dynamics
obtained for 106Ag. Note that the moderate triaxial deformation of 106Ag
considerably softens the energy surface in the $\varphi_{J}$ direction, so the
precession period is much longer than that for the chiral precession found in
135Nd [35].

Increasing the angular momentum to, for instance, $I=18\hbar$, the lower-
frequency mode is invisible, and the time evolution of $\varphi_{J}$ is
dominated by a single frequency harmonic oscillation, as depicted in Fig.
2(b). This is indeed due to the fact that the energy surface becomes much
softer in the $\varphi_{J}$ direction and stiffer in the $\theta_{J}$
direction at higher angular momenta; indicating an extremely long-period mode
which actually does not correspond to a real vibration mode. The soft energy
surface is also reflected by the time-dependent trajectory of the tilted
angles $\theta_{J}$ and $\varphi_{J}$, which is extremely narrowed in the
$\theta_{J}$ direction [see the inset of Fig. 2(b)].

Figure 3: (color online). Time evolution of the azimuth angles $\varphi_{J}$
starting from two initial states which are away from the lowest energy state
only in the $\gamma$ direction by 10 keV ($\gamma=9^{\circ}$) and 100 keV
($\gamma=12^{\circ}$) at the total angular momentum $I=10\hbar$ (top) and
$I=18\hbar$ (bottom). Insets: The corresponding spectral powers obtained by
Fourier transforms on the time evolution of the azimuth angles, and the arrow
indicates the centroid energies.

The energy of the quantum phonons corresponding to the chiral vibration should
be determined by quantizing the semiclassical time-dependent solutions, and it
can be easily obtained by extracting the normal-mode frequencies with Fourier
transformation in the case of small harmonic oscillations. From Fig. 2, one
see clearly that the lower-frequency mode plays only a minor role in the
chiral dynamics even at $I=10\hbar$ due to the moderate triaxial deformation
of 106Ag. Moreover, the corresponding phonon energy for the lower-frequency
mode would be further suppressed due to the quantum admixture of the two
normal modes. Therefore, here we concentrate on the higher-frequency normal
mode, which is mainly associated with the $\gamma$ softness.

To extract the normal-mode frequency with Fourier transformation, a pure
higher-frequency normal mode is needed, which can be obtained by performing
the TDCDFT calculations starting from certain initial states, i.e., the states
differing from the equilibrium state only in the $\gamma$ direction, while the
corresponding $\theta_{J}$, $\varphi_{J}$, and $\beta$ values are the same as
the equilibrium state.

At $I=10\hbar$, two different initial states are taken as examples, and they
are higher in energy than the equilibrium state by 10 keV ($\gamma=9^{\circ}$)
and 100 keV ($\gamma=12^{\circ}$). The corresponding time evolution of the
azimuth angle $\varphi_{J}$ starting from these two initial states are
depicted in Fig. 3(a). The initial $\varphi_{J}$ values are zero for both
trajectories, and along the time evolution, they start to oscillate between
positive and negative values. This reflects the chiral nature of the
oscillations induced by the $\gamma$ softness. Moreover, the two trajectories
have distinct amplitudes but roughly identical frequency, proving the harmonic
nature of the chiral dynamics in such a soft triaxial nucleus.

The harmonic nature of the oscillations can be more clearly seen in the inset
of Fig. 3(a), where the spectral powers obtained by Fourier transforms [44,
35] on the time evolution of the azimuth angle are depicted. The height of the
spectral power reflects the amplitude of the oscillations, while the frequency
is associated with the one-phonon energy due to the chiral motions. A single
main peak is clearly seen for both spectral powers, and the positions of both
peaks are identically at 0.584 MeV. This provides the energy difference for
the chiral partner bands at $I=10\hbar$.

Such analysis can be straightforwardly extended to the states at other angular
momenta from $I=11\hbar$ to $I=20\hbar$. Taking the $I=18\hbar$ state as an
example, in Fig. 3(b), the time evolution of the azimuth angle $\varphi_{J}$
is depicted starting from two different initial states, which are away from
the equilibrium state only in the $\gamma$ direction. In general, the behavior
of the time evolution at $I=18\hbar$ is quite similar to that at $I=10\hbar$,
except for the much larger amplitudes, which are associated with the fact that
the total energy surface becomes much softer in the $\varphi_{J}$ direction at
higher angular momenta. The oscillation of the $\varphi_{J}$ values looks even
more harmonic than that at $I=10\hbar$. As a result, the spectral powers
obtained with Fourier transforms on the time evolution of the azimuth angle,
as seen in the inset of Fig. 3(b), have two peaks at an identical position,
i.e., 0.700 MeV.

Figure 4: (Color online). Calculated energies (solid and dashed lines) for
chiral doublet bands in 106Ag built on the four-quasiparticle configuration
$\pi g_{9/2}\otimes\nu h_{11/2}(gd)^{2}$ in comparison with data (solid and
open symbols) [18]. The dashed line represents the TAC-CDFT results, and the
solid one represents the results including the chiral excitation energies
obtained with the TDCDFT calculations.

Finally, by combining the calculated results with both TAC-CDFT and TDCDFT at
each spin, one could provide a microscopic description for the partner bands
observed in the triaxial soft nucleus 106Ag. In Fig. 4, the calculated energy
spectra for the partner bands built on the four-quasiparticle configuration
$\pi g_{9/2}\otimes\nu h_{11/2}(gd)^{2}$ are depicted in comparison with data
[18]. For the lower band, the excitation energies are originated from planar
rotations and, thus, are calculated with the TAC-CDFT calculations. Based on
the TAC-CDFT solutions at each spin, the TDCDFT calculations provide the
corresponding excitation energies due to the chiral motion. Consequently, the
upper band can be built by adding up the TAC-CDFT energies and the
corresponding chiral excitation energy at every spin. It can be seen that the
experimental energies are well reproduced. This explains the puzzling chiral
structure in 106Ag and provides the first microscopic description for the
chiral doublet bands in a triaxial soft nucleus.

##  4 Summary

In summary, for the first time, the chirality in soft triaxial nuclei has been
investigated in the microscopic time-dependent covariant density functional
theory, and the initial state at a given angular momentum is taken from the
calculations within the three-dimensional tilted axis cranking covariant
density functional theory. Taking the puzzling chiral nucleus 106Ag as an
example, the experimental energies of the observed nearly degenerate bands are
well reproduced without any free parameters beyond the well-defined density
functional. From the microscopic dynamics of the total angular momentum in the
body-fixed frame, it is found that the precession dynamics, appeared in rigid
chiral nuclei, could be remarkably suppressed in soft chiral nuclei. Instead,
a novel chiral mode in soft triaxial nuclei occurs and it is responsible for
the chiral excitations observed in 106Ag. This not only explains the puzzling
chiral structure in 106Ag, but also opens a new research area for the study of
chirality, particularly in relation to soft nuclear shapes.

## Conflict of interest

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## Acknowledgments

We thank Zhengxue Ren for fruitful discussions. This work was partly supported
by the National Natural Science Foundation of China (Grants No. 12070131001,
No. 11935003, No. 11975031, and No. 12141501), and the High-performance
Computing Platform of Peking University.

## References

  * Frauendorf and Meng [1997] S. Frauendorf, J. Meng, Tilted rotation of triaxial nuclei, Nucl. Phys. A 617 (2) (1997) 131–147, doi:10.1016/S0375-9474(97)00004-3.
  * Starosta et al. [2001] K. Starosta, T. Koike, C. J. Chiara, D. B. Fossan, D. R. LaFosse, A. A. Hecht, C. W. Beausang, M. A. Caprio, J. R. Cooper, R. Krücken, J. R. Novak, N. V. Zamfir, K. E. Zyromski, D. J. Hartley, D. L. Balabanski, J.-y. Zhang, S. Frauendorf, V. I. Dimitrov, Chiral Doublet Structures in Odd-Odd $\mathit{N}\phantom{\rule{0.0pt}{0.0pt}}=\phantom{\rule{0.0pt}{0.0pt}}75$ Isotones: Chiral Vibrations, Phys. Rev. Lett. 86 (2001) 971–974, doi:10.1103/PhysRevLett.86.971.
  * Zhu et al. [2003] S. Zhu, U. Garg, B. K. Nayak, S. S. Ghugre, N. S. Pattabiraman, D. B. Fossan, T. Koike, K. Starosta, C. Vaman, R. V. F. Janssens, R. S. Chakrawarthy, M. Whitehead, A. O. Macchiavelli, S. Frauendorf, A Composite Chiral Pair of Rotational Bands in the Odd-$A$ Nucleus ${}^{135}\mathrm{N}\mathrm{d}$, Phys. Rev. Lett. 91 (2003) 132501, doi:10.1103/PhysRevLett.91.132501.
  * Vaman et al. [2004] C. Vaman, D. B. Fossan, T. Koike, K. Starosta, I. Y. Lee, A. O. Macchiavelli, Chiral Degeneracy in Triaxial ${}^{104}\mathrm{R}\mathrm{h}$, Phys. Rev. Lett. 92 (2004) 032501, doi:10.1103/PhysRevLett.92.032501.
  * Grodner et al. [2006] E. Grodner, J. Srebrny, A. A. Pasternak, I. Zalewska, T. Morek, C. Droste, J. Mierzejewski, M. Kowalczyk, J. Kownacki, M. Kisieliński, S. G. Rohoziński, T. Koike, K. Starosta, A. Kordyasz, P. J. Napiorkowski, M. Wolińska-Cichocka, E. Ruchowska, W. Płóciennik, J. Perkowski, ${}^{128}\mathrm{Cs}$ as the Best Example Revealing Chiral Symmetry Breaking, Phys. Rev. Lett. 97 (2006) 172501, doi:10.1103/PhysRevLett.97.172501.
  * Joshi et al. [2007] P. Joshi, M. P. Carpenter, D. B. Fossan, T. Koike, E. S. Paul, G. Rainovski, K. Starosta, C. Vaman, R. Wadsworth, Effect of $\gamma$ Softness on the Stability of Chiral Geometry: Spectroscopy of ${}^{106}\mathrm{Ag}$, Phys. Rev. Lett. 98 (2007) 102501, doi:10.1103/PhysRevLett.98.102501.
  * Mukhopadhyay et al. [2007] S. Mukhopadhyay, D. Almehed, U. Garg, S. Frauendorf, T. Li, P. V. Madhusudhana Rao, X. Wang, S. S. Ghugre, M. P. Carpenter, S. Gros, A. Hecht, R. V. F. Janssens, F. G. Kondev, T. Lauritsen, D. Seweryniak, S. Zhu, From Chiral Vibration to Static Chirality in $Nd135$, Phys. Rev. Lett. 99 (17) (2007) 172501, doi:10.1103/PhysRevLett.99.172501.
  * Ayangeakaa et al. [2013] A. D. Ayangeakaa, U. Garg, M. D. Anthony, S. Frauendorf, J. T. Matta, B. K. Nayak, D. Patel, Q. B. Chen, S. Q. Zhang, P. W. Zhao, B. Qi, J. Meng, R. V. F. Janssens, M. P. Carpenter, C. J. Chiara, F. G. Kondev, T. Lauritsen, D. Seweryniak, S. Zhu, S. S. Ghugre, R. Palit, Evidence for Multiple Chiral Doublet Bands in ${}^{133}\mathrm{Ce}$, Phys. Rev. Lett. 110 (2013) 172504, doi:10.1103/PhysRevLett.110.172504.
  * Kuti et al. [2014] I. Kuti, Q. B. Chen, J. Timár, D. Sohler, S. Q. Zhang, Z. H. Zhang, P. W. Zhao, J. Meng, K. Starosta, T. Koike, E. S. Paul, D. B. Fossan, C. Vaman, Multiple Chiral Doublet Bands of Identical Configuration in ${}^{103}\mathrm{Rh}$, Phys. Rev. Lett. 113 (2014) 032501, doi:10.1103/PhysRevLett.113.032501.
  * Tonev et al. [2014] D. Tonev, M. S. Yavahchova, N. Goutev, G. de Angelis, P. Petkov, R. K. Bhowmik, R. P. Singh, S. Muralithar, N. Madhavan, R. Kumar, M. Kumar Raju, J. Kaur, G. Mohanto, A. Singh, N. Kaur, R. Garg, A. Shukla, T. K. Marinov, S. Brant, Candidates for Twin Chiral Bands in ${}^{102}\mathrm{Rh}$, Phys. Rev. Lett. 112 (2014) 052501, doi:10.1103/PhysRevLett.112.052501.
  * Liu et al. [2016] C. Liu, S. Y. Wang, R. A. Bark, S. Q. Zhang, J. Meng, B. Qi, P. Jones, S. M. Wyngaardt, J. Zhao, C. Xu, S.-G. Zhou, S. Wang, D. P. Sun, L. Liu, Z. Q. Li, N. B. Zhang, H. Jia, X. Q. Li, H. Hua, Q. B. Chen, Z. G. Xiao, H. J. Li, L. H. Zhu, T. D. Bucher, T. Dinoko, J. Easton, K. Juhász, A. Kamblawe, E. Khaleel, N. Khumalo, E. A. Lawrie, J. J. Lawrie, S. N. T. Majola, S. M. Mullins, S. Murray, J. Ndayishimye, D. Negi, S. P. Noncolela, S. S. Ntshangase, B. M. Nyakó, J. N. Orce, P. Papka, J. F. Sharpey-Schafer, O. Shirinda, P. Sithole, M. A. Stankiewicz, M. Wiedeking, Evidence for Octupole Correlations in Multiple Chiral Doublet Bands, Phys. Rev. Lett. 116 (2016) 112501, doi:10.1103/PhysRevLett.116.112501.
  * Grodner et al. [2018] E. Grodner, J. Srebrny, C. Droste, L. Próchniak, S. G. Rohoziński, M. Kowalczyk, M. Ionescu-Bujor, C. A. Ur, K. Starosta, T. Ahn, M. Kisieliński, T. Marchlewski, S. Aydin, F. Recchia, G. Georgiev, R. Lozeva, E. Fiori, M. Zielińska, Q. B. Chen, S. Q. Zhang, L. F. Yu, P. W. Zhao, J. Meng, First Measurement of the $g$ Factor in the Chiral Band: The Case of the ${}^{128}\mathrm{Cs}$ Isomeric State, Phys. Rev. Lett. 120 (2018) 022502, doi:10.1103/PhysRevLett.120.022502.
  * Xiong and Wang [2019] B. Xiong, Y. Wang, Nuclear chiral doublet bands data tables, At. Data Nucl. Data Tables 125 (2019) 193–225, ISSN 0092-640X, doi:10.1016/j.adt.2018.05.002.
  * Meng and Zhang [2010] J. Meng, S. Q. Zhang, Open problems in understanding the nuclear chirality, J. Phys. G. 37 (6) (2010) 064025\.
  * Meng and Zhao [2016] J. Meng, P. W. Zhao, Nuclear chiral and magnetic rotation in covariant density functional theory, Phys. Scr. 91 (5) (2016) 053008, doi:10.1088/0031-8949/91/5/053008.
  * Tonev et al. [2006] D. Tonev, G. de Angelis, P. Petkov, A. Dewald, S. Brant, S. Frauendorf, D. L. Balabanski, P. Pejovic, D. Bazzacco, P. Bednarczyk, F. Camera, A. Fitzler, A. Gadea, S. Lenzi, S. Lunardi, N. Marginean, O. Möller, D. R. Napoli, A. Paleni, C. M. Petrache, G. Prete, K. O. Zell, Y. H. Zhang, J.-y. Zhang, Q. Zhong, D. Curien, Transition Probabilities in ${}^{134}\mathrm{Pr}$: A Test for Chirality in Nuclear Systems, Phys. Rev. Lett. 96 (2006) 052501, doi:10.1103/PhysRevLett.96.052501.
  * Petrache et al. [2006] C. M. Petrache, G. B. Hagemann, I. Hamamoto, K. Starosta, Risk of Misinterpretation of Nearly Degenerate Pair Bands as Chiral Partners in Nuclei, Phys. Rev. Lett. 96 (2006) 112502, doi:10.1103/PhysRevLett.96.112502.
  * Lieder et al. [2014] E. O. Lieder, R. M. Lieder, R. A. Bark, Q. B. Chen, S. Q. Zhang, J. Meng, E. A. Lawrie, J. J. Lawrie, S. P. Bvumbi, N. Y. Kheswa, S. S. Ntshangase, T. E. Madiba, P. L. Masiteng, S. M. Mullins, S. Murray, P. Papka, D. G. Roux, O. Shirinda, Z. H. Zhang, P. W. Zhao, Z. P. Li, J. Peng, B. Qi, S. Y. Wang, Z. G. Xiao, C. Xu, Resolution of Chiral Conundrum in ${}^{106}\mathrm{Ag}$: Doppler-Shift Lifetime Investigation, Phys. Rev. Lett. 112 (2014) 202502, doi:10.1103/PhysRevLett.112.202502.
  * Rather et al. [2014] N. Rather, P. Datta, S. Chattopadhyay, S. Rajbanshi, A. Goswami, G. H. Bhat, J. A. Sheikh, S. Roy, R. Palit, S. Pal, S. Saha, J. Sethi, S. Biswas, P. Singh, H. C. Jain, Exploring the Origin of Nearly Degenerate Doublet Bands in ${}^{106}\mathrm{Ag}$, Phys. Rev. Lett. 112 (2014) 202503, doi:10.1103/PhysRevLett.112.202503.
  * Timár et al. [2011] J. Timár, K. Starosta, I. Kuti, D. Sohler, D. B. Fossan, T. Koike, E. S. Paul, A. J. Boston, H. J. Chantler, M. Descovich, R. M. Clark, M. Cromaz, P. Fallon, I. Y. Lee, A. O. Macchiavelli, C. J. Chiara, R. Wadsworth, A. A. Hecht, D. Almehed, S. Frauendorf, Medium- and high-spin band structure of the chiral-candidate nucleus 134Pr, Phys. Rev. C 84 (2011) 044302, doi:10.1103/PhysRevC.84.044302.
  * Koike et al. [2004] T. Koike, K. Starosta, I. Hamamoto, Chiral Bands, Dynamical Spontaneous Symmetry Breaking, and the Selection Rule for Electromagnetic Transitions in the Chiral Geometry, Phys. Rev. Lett. 93 (2004) 172502, doi:%**** ̵paper.tex ̵Line ̵550 ̵****10.1103/PhysRevLett.93.172502.
  * Peng et al. [2003] J. Peng, J. Meng, S. Q. Zhang, Description of chiral doublets in $A\sim 130$ nuclei and the possible chiral doublets in $A\sim 100$ nuclei, Phys. Rev. C 68 (2003) 044324, doi:10.1103/PhysRevC.68.044324.
  * Zhang et al. [2007] S. Q. Zhang, B. Qi, S. Y. Wang, J. Meng, Chiral bands for a quasi-proton and quasi-neutron coupled with a triaxial rotor, Phys. Rev. C 75 (2007) 044307, doi:10.1103/PhysRevC.75.044307.
  * Qi et al. [2009] B. Qi, S. Zhang, J. Meng, S. Wang, S. Frauendorf, Chirality in odd-A nucleus 135Nd in particle rotor model, Phys. Lett. B 675 (2) (2009) 175–180, ISSN 0370-2693, doi:10.1016/j.physletb.2009.02.061.
  * Chen et al. [2018] Q. Chen, B. Lv, C. Petrache, J. Meng, Multiple chiral doublets in four-j shells particle rotor model: Five possible chiral doublets in 60136Nd76, Phys. Lett. B 782 (2018) 744–749, ISSN 0370-2693, doi:10.1016/j.physletb.2018.06.030.
  * Tonev et al. [2007] D. Tonev, G. d. Angelis, S. Brant, S. Frauendorf, P. Petkov, A. Dewald, F. Dönau, D. L. Balabanski, Q. Zhong, P. Pejovic, D. Bazzacco, P. Bednarczyk, F. Camera, D. Curien, F. D. Vedova, A. Fitzler, A. Gadea, G. L. Bianco, S. Lenzi, S. Lunardi, N. Marginean, O. Möller, D. R. Napoli, R. Orlandi, E. Sahin, A. Saltarelli, J. V. Dobon, K. O. Zell, J.-y. Zhang, Y. H. Zhang, Question of dynamic chirality in nuclei: The case of ${}^{134}\mathrm{Pr}$, Phys. Rev. C 76 (2007) 044313, doi:10.1103/PhysRevC.76.044313.
  * Brant et al. [2008] S. Brant, D. Tonev, G. De Angelis, A. Ventura, Dynamic chirality in the interacting boson fermion-fermion model, Phys. Rev. C 78 (2008) 034301, doi:10.1103/PhysRevC.78.034301.
  * Dimitrov et al. [2000] V. I. Dimitrov, S. Frauendorf, F. Dönau, Chirality of Nuclear Rotation, Phys. Rev. Lett. 84 (2000) 5732–5735, doi:10.1103/PhysRevLett.84.5732.
  * Madokoro et al. [2000] H. Madokoro, J. Meng, M. Matsuzaki, S. Yamaji, Relativistic mean field description for the shears band mechanism in 84Rb, Phys. Rev. C 62 (6) (2000) 061301, doi:10.1103/PhysRevC.62.061301.
  * Olbratowski et al. [2004] P. Olbratowski, J. Dobaczewski, J. Dudek, W. Płóciennik, Critical Frequency in Nuclear Chiral Rotation, Phys. Rev. Lett. 93 (5) (2004) 052501, doi:10.1103/PhysRevLett.93.052501.
  * Zhao [2017] P. W. Zhao, Multiple chirality in nuclear rotation: A microscopic view, Phys. Lett. B 773 (2017) 1–5, ISSN 0370-2693, doi:10.1016/j.physletb.2017.08.001.
  * Zhao et al. [2019] P. W. Zhao, Y. K. Wang, Q. B. Chen, Microscopic resolution of the nuclear chiral conundrum with crossing twin bands in ${}^{106}\mathrm{Ag}$, Phys. Rev. C 99 (2019) 054319, doi:10.1103/PhysRevC.99.054319.
  * Ren et al. [2020a] Z. Ren, P. Zhao, J. Meng, Dynamics of the linear-chain alpha cluster in microscopic time-dependent relativistic density functional theory, Phys. Lett. B 801 (2020a) 135194, ISSN 0370-2693, doi:10.1016/j.physletb.2019.135194.
  * Ren et al. [2020b] Z. X. Ren, P. W. Zhao, J. Meng, Time-dependent covariant density functional theory in three-dimensional lattice space: Benchmark calculation for the ${}^{16}\mathrm{O}$+${}^{16}\mathrm{O}$ reaction, Phys. Rev. C 102 (2020b) 044603, doi:10.1103/PhysRevC.102.044603.
  * Ren et al. [2022] Z. X. Ren, P. W. Zhao, J. Meng, Dynamics of rotation in chiral nuclei, Phys. Rev. C 105 (2022) L011301, doi:10.1103/PhysRevC.105.L011301.
  * Ring [1996] P. Ring, Relativistic mean field theory in finite nuclei, Prog. Part. Nucl. Phys. 37 (1996) 193–263, doi:10.1016/0146-6410(96)00054-3.
  * Vretenar et al. [2005] D. Vretenar, A. V. Afanasjev, G. A. Lalazissis, P. Ring, Relativistic Hartree Bogoliubov theory: static and dynamic aspects of exotic nuclear structure, Phys. Rep. 409 (3-4) (2005) 101–259, doi:10.1016/j.physrep.2004.10.001.
  * Meng et al. [2006] J. Meng, H. Toki, S. Zhou, S. Zhang, W. Long, L. Geng, Relativistic continuum Hartree Bogoliubov theory for ground-state properties of exotic nuclei, Prog. Part. Nucl. Phys. 57 (2) (2006) 470–563, doi:10.1016/j.ppnp.2005.06.001.
  * Nikšić et al. [2011] T. Nikšić, D. Vretenar, P. Ring, Relativistic nuclear energy density functionals: Mean-field and beyond, Prog. Part. Nucl. Phys. 66 (3) (2011) 519–548, ISSN 0146-6410, doi:10.1016/j.ppnp.2011.01.055.
  * Meng [2015] J. Meng (Ed.), Relativistic Density Functional for Nuclear Structure, World Scientific, 2015\.
  * Ren and Zhao [2020] Z. X. Ren, P. W. Zhao, Toward a bridge between relativistic and nonrelativistic density functional theories for nuclei, Phys. Rev. C 102 (2020) 021301(R), doi:10.1103/PhysRevC.102.021301.
  * Meng et al. [2013] J. Meng, J. Peng, S.-Q. Zhang, P.-W. Zhao, Progress on tilted axis cranking covariant density functional theory for nuclear magnetic and antimagnetic rotation, Front. Phys. 8 (2013) 55–79, ISSN 2095-0462, doi:10.1007/s11467-013-0287-y.
  * Zhao et al. [2010] P. W. Zhao, Z. P. Li, J. M. Yao, J. Meng, New parametrization for the nuclear covariant energy density functional with a point-coupling interaction, Phys. Rev. C 82 (5) (2010) 054319, doi:10.1103/PhysRevC.82.054319.
  * Press et al. [1992] W. H. Press, S. A. Teukolsky, W. T. Vetterling, B. P. Flannery, Numerical Recipes in Fortran 77, Press Syndicate of the University of Cambridge, London, 1992\.