# Two-Dimensional Electronic Spectroscopy for Three-Level Atoms with
Electromagnetically Induced Transparency

Jing-Yi-Ran Jin  Department of Physics, Applied Optics Beijing Area Major
Laboratory, Beijing Normal University, Beijing 100875, China  Hao-Yue Zhang
Department of Physics, Applied Optics Beijing Area Major Laboratory, Beijing
Normal University, Beijing 100875, China  Yi-Xuan Yao  Department of Physics,
Applied Optics Beijing Area Major Laboratory, Beijing Normal University,
Beijing 100875, China  Qing Ai<EMAIL_ADDRESS>Department of Physics,
Applied Optics Beijing Area Major Laboratory, Beijing Normal University,
Beijing 100875, China

###### Abstract

Two-dimensional electronic spectroscopy (2DES) has high spectral resolution
and is a useful tool for studying atom dynamics. In this paper, we apply the
electromagnetically induced transparency (EIT) technique to 2DES in a three-
level atom, and find out that the number of peaks (troughs) will become more
due to the introduction of EIT. Also, the height of the peaks (the depth of
troughs) will change from constant to a damped oscillation. These findings may
help us obtain more information about the dynamics of excited states.

##  I Introduction

Electromagnetically induced transparency (EIT) is an atomic coherence effect
produced by a multi-level atomic system in the presence of an external light
field Scully and Zubairy (1997). It can be specifically described as when the
control and probe fields are incident on the system with the two-photon
resonance condition, the nonlinear effects of the medium are enhanced and thus
the absorption and dispersion properties are significantly altered due to the
destructive interference effect between the excitation paths. Because of its
unique optical properties, EIT has a wide range of applications in cold atoms
Hopkins  _et al._ (1997); Hau  _et al._ (1999); Shiau  _et al._ (2011), solid
materials Schmidt and Imamo?lu (1996); Zhao  _et al._ (1997); Kuznetsova  _et
al._ (2002); Baldit  _et al._ (2010); Fan  _et al._ (2019), nitrogen-vacancy
color centers in diamond Wei and Manson (1999); Hemmer  _et al._ (2001);
Acosta  _et al._ (2013); Wang  _et al._ (2018), quantum information processing
Fleischhauer and Lukin (2000); Liu  _et al._ (2001); Fleischhauer and Lukin
(2002); Honda  _et al._ (2008); Hsiao  _et al._ (2018); Chen  _et al._ (2013);
Kukharchyk  _et al._ (2020); Wang  _et al._ (2023), metamaterials Jin  _et
al._ (2013); Bagci and Akaoglu (2019); Jin  _et al._ (2011); Tassin  _et al._
(2009); Xu  _et al._ (2010), and non-reciprocal optics Huang  _et al._ (2022),
since it was first experimentally observed by Harris’ group in 1991 Boller
_et al._ (1991).

Two-dimensional electronic spectroscopy (2DES) is a third-order nonlinear
spectral measurement technique Mukamel (1999), which can reflect the
absorption characteristics. Three ultrafast optical coherent pulses interact
with matter sequentially, and the collected optical signals containing the
evolution information are transformed by double Fourier transforms to obtain
2DES. Compared with widely-used narrowband-excitation and wide-spectrum
detection methods, 2DES can have high spectral resolution and time resolution
in the case of wideband excitation. Thus, it can capture the quantum coherent
phenomenon in the early stage of optical excitation Schlau-Cohen  _et al._
(2011); Fuller and Ogilvie (2015). It can measure the homogeneous and
inhomogeneous broadening of the sample Sun  _et al._ (2023), and also identify
the coupling between different transitions by cross peaks Liu  _et al._
(2020). Therefore, it plays a vital role in the investigation of the quantum
dot Seiler  _et al._ (2018); Brosseau  _et al._ (2020), two-dimensional
materials Moody  _et al._ (2015); Guo  _et al._ (2019), perovskite Jha  _et
al._ (2017); Monahan  _et al._ (2017); Seiler  _et al._ (2019), and energy-
transfer mechanism of photosynthesis Calhoun  _et al._ (2009); Fuller  _et
al._ (2014); Jun  _et al._ (2014); Romero  _et al._ (2014); Jun  _et al._
(2015); Ma  _et al._ (2017); Thyrhaug  _et al._ (2018). In this paper, we
apply the EIT to the 2DES for a three-level configuration and find out that
the spectral resolution can be improved, which is more helpful for exploring
excited-state dynamics.

This paper is organized as follows. In Sec. II, we derive the response
functions of rephasing and non-rephasing signals under the established model
by using Feynman diagrams. In Sec. III, we show 2DES of rephasing, non-
rephasing and absorptive signals in the presence and absence of a control
field. We compare 2DES before and after introducing EIT and analyze the
underlying physical mechanism of this phenomenon. In Sec. IV, we summarize our
main findings.

##  II Model

Figure 1: Schematic diagram for 2DES of a three-level atom, where energy
levels $b$ and $c$ are respectively the ground and metastable states, and
energy level $a$ is the excited state. $\Gamma_{1}$ and $\Gamma_{2}$ represent
the relaxation rates between the energy levels $a$ and $b$.

For the three-level atom as shown in the Fig. 1, four short pulses in the time
domain compose a detection pulse sequence, and the time intervals between them
are respectively $t_{1}$, $t_{2}$ and $t_{3}$. The first three detection
pulses interact with the sample and resonantly probe the transition between
the ground state $b$ and the excited state $a$. According to the condition of
the phase matching, the wave vector $\vec{k}_{s}$ of the generated third-order
polarization signal is the sum of the wave vectors of all the input fields,
and thus a fourth pulse in this direction will be emitted. Since the signal is
weak, a pulse $\vec{E}_{\textrm{LO}}$ in the same direction as $\vec{k}_{s}$
is used as the local oscillating light for heterodyne measurement. In
addition, a narrow-band control light is applied in resonance with the
transition between the excited state $a$ and the metastable state $c$. In this
way, we construct a simple $\Lambda$-type electromagnetically induced
transparent model for the 2DES.

In experiments, there are three types of observable signals, i.e., the
absorptive signal

$\displaystyle S^{(3)}_{\textrm{ab}}(\omega_{3},t_{2},\omega_{1})$
$\displaystyle=$
$\displaystyle\textrm{Re}\sum_{i=1}^{4}R_{i}(\omega_{3},t_{2},\omega_{1}),$
(1)

the rephasing signal

$\displaystyle S^{(3)}_{\textrm{rp}}(\omega_{3},t_{2},\omega_{1})$
$\displaystyle\\!=\\!$
$\displaystyle\textrm{Re}[R_{2}(\omega_{3},t_{2},\omega_{1})+R_{3}(\omega_{3},t_{2},\omega_{1})],$
(2)

the non-rephasing signal

$\displaystyle S^{(3)}_{\textrm{nr}}(\omega_{3},t_{2},\omega_{1})$
$\displaystyle\\!=\\!$
$\displaystyle\textrm{Re}[R_{1}(\omega_{3},t_{2},\omega_{1})+R_{4}(\omega_{3},t_{2},\omega_{1})],$
(3)

where the different Liouville paths $R_{\alpha}$’s ($\alpha=1,2,3,4$) are
schematically illustrated in Fig. 2.

During the population time $t_{2}$, $R_{1}$ and $R_{2}$ represent the initial
evolution of the electron in the excited states, while $R_{3}$ and $R_{4}$
represent the initial evolution of the electron in the ground state. On the
other hand, $R_{2}$ and $R_{3}$ indicate that the signal responds in the
direction with the wave vector
$\vec{k}_{s}=-\vec{k}_{1}+\vec{k}_{2}+\vec{k}_{3}$, while $R_{1}$ and $R_{4}$
indicate that the signal responds in the direction with the wave vector
$\vec{k}_{s}=\vec{k}_{1}-\vec{k}_{2}+\vec{k}_{3}$.

Figure 2: Feynman diagrams for the three-level system, with $R_{1}$ for (a)
and (b), $R_{2}$ for (c) and (d), $R_{3}$ for (e) and (f), and $R_{4}$ for (g)
and (h).

For the $\Lambda$-type three-level atom as considered in Fig. 1, the
Hamiltonian can be expressed as

$\displaystyle H$ $\displaystyle=$ $\displaystyle H_{0}+H_{\textrm{int}},$ (4)
$\displaystyle H_{0}$ $\displaystyle=$
$\displaystyle\sum_{j=a,b,c}\hbar\omega_{j}|j\rangle\langle j|,$ (5)
$\displaystyle H_{\textrm{int}}$ $\displaystyle=$
$\displaystyle-\frac{\hbar}{2}\Omega e^{-i\nu_{c}t}|a\rangle\langle
c|+\textrm{h.c.},$ (6)

where $\hbar\omega_{j}$ is the energy of the state $|j\rangle$ with $\hbar$
being the reduced Planck constant, $\Omega=\mu_{ac}\varepsilon_{c}/\hbar$ is
the Rabi frequency, $\nu_{c}$ and $\varepsilon_{c}$ are respectively the
frequency and amplitude of the control field, $\mu_{ac}=\langle
a|\mu|c\rangle$ is the transition dipole moment.

In the presence of control light, the reduced density matrix of the system is
governed by the quantum master equation Breuer and Petruccione (2002)

$\displaystyle\dot{\rho}$ $\displaystyle=$
$\displaystyle-\frac{i}{\hbar}\left[H,\rho\right]-\Gamma_{1}\mathcal{L}(|b\rangle\langle
a|)\rho-\Gamma_{2}\mathcal{L}(|a\rangle\langle b|)\rho$ (7)
$\displaystyle-\sum_{\alpha=a,b,c}\gamma_{\alpha}^{(0)}\mathcal{L}(|\alpha\rangle\langle\alpha|)\rho,$

where the Lindblad operator

$\mathcal{L}(A)\rho=\frac{1}{2}\\{A^{\dagger}A,\rho\\}-A^{\dagger}\rho A,$ (8)

$\\{A,\rho\\}=A\rho+\rho A$ is the anti-commutator. $\Gamma_{1}$ and
$\Gamma_{2}$ are respectively the downhill and uphill relaxation rates between
the energy levels $a$ and $b$. $\gamma_{\alpha}^{(0)}$ ($\alpha=a,b,c$) is the
pure-dephasing noise of the state $|\alpha\rangle$.

By transforming to the interaction picture, after some algebra, we can obtain
a set of differential equations for the matrix elements of $\rho$ as

$\displaystyle\dot{\rho}_{bb}$ $\displaystyle=$
$\displaystyle\Gamma_{1}\rho_{aa}-\Gamma_{2}\rho_{bb},$ (9)
$\displaystyle\dot{\rho}_{ba}$ $\displaystyle=$
$\displaystyle-\frac{i}{2}\Omega\rho_{bc}-\gamma_{1}\rho_{ba},$ (10)
$\displaystyle\dot{\rho}_{bc}$ $\displaystyle=$
$\displaystyle-\frac{i}{2}\Omega\rho_{ba}-\gamma_{2}\rho_{bc},$ (11)
$\displaystyle\dot{\rho}_{ab}$ $\displaystyle=$
$\displaystyle\frac{i}{2}\Omega\rho_{cb}-\gamma_{1}\rho_{ab},$ (12)
$\displaystyle\dot{\rho}_{aa}$ $\displaystyle=$
$\displaystyle\frac{i}{2}\Omega(\rho_{ca}-\rho_{ac})-\Gamma_{1}\rho_{aa}+\Gamma_{2}\rho_{bb},$
(13) $\displaystyle\dot{\rho}_{ac}$ $\displaystyle=$
$\displaystyle\frac{i}{2}\Omega(\rho_{cc}-\rho_{aa})-\gamma_{3}\rho_{ac},$
(14) $\displaystyle\dot{\rho}_{cb}$ $\displaystyle=$
$\displaystyle\frac{i}{2}\Omega\rho_{ab}-\gamma_{2}\rho_{cb},$ (15)
$\displaystyle\dot{\rho}_{ca}$ $\displaystyle=$
$\displaystyle\frac{i}{2}\Omega(\rho_{aa}-\rho_{cc})-\gamma_{3}\rho_{ca},$
(16) $\displaystyle\dot{\rho}_{cc}$ $\displaystyle=$
$\displaystyle\frac{i}{2}\Omega(\rho_{ac}-\rho_{ca}),$ (17)

where for simplicity we introduce

$\displaystyle\gamma_{1}$ $\displaystyle=$
$\displaystyle\frac{1}{2}(\Gamma_{1}+\gamma_{a}^{(0)}+\Gamma_{2}+\gamma_{b}^{(0)}),$
(18) $\displaystyle\gamma_{2}$ $\displaystyle=$
$\displaystyle\frac{1}{2}(\Gamma_{2}+\gamma_{b}^{(0)}+\gamma_{c}^{(0)}),$ (19)
$\displaystyle\gamma_{3}$ $\displaystyle=$
$\displaystyle\frac{1}{2}(\Gamma_{1}+\gamma_{a}^{(0)}+\gamma_{c}^{(0)}).$ (20)

The relevant Green’s functions are explicitly obtained as

$\displaystyle\mathrm{\mathcal{G}}_{ba,ba}(\omega_{1})$ $\displaystyle=$
$\displaystyle\frac{4(\omega_{1}-i\gamma_{2}-\omega_{ab})}{4(\omega_{1}-i\gamma_{1}-\omega_{ab})(\omega_{1}-i\gamma_{2}-\omega_{ab})-\Omega^{2}},$
$\displaystyle\mathrm{\mathcal{G}}_{ab,ab}(\omega_{3})$ $\displaystyle=$
$\displaystyle\frac{4(\omega_{3}+i\gamma_{2}-\omega_{ab})}{4(\omega_{3}+i\gamma_{1}-\omega_{ab})(\omega_{3}+i\gamma_{2}-\omega_{ab})-\Omega^{2}},$

where the Green’s functions $\mathcal{G}_{ba,ba}(\omega_{1})$ and
$\mathcal{G}_{ab,ab}(\omega_{3})$ have double peaks and a higher minimum with
the presence of the control field than without, as shown in Fig. 3. By using
the Green’s functions, the rephasing and non-rephasing signals can be
explicitly given as

$\displaystyle S^{(3)}_{\textrm{rp}}(\omega_{3},t_{2},\omega_{1})=$
$\displaystyle\mathcal{G}_{ab,ab}(\omega_{3})\mathcal{G}_{aa,aa}(t_{2})\mathcal{G}_{ba,ba}(\omega_{1})$
(22)
$\displaystyle+\mathcal{G}_{ab,ab}(\omega_{3})\mathcal{G}_{bb,aa}(t_{2})\mathcal{G}_{ba,ba}(\omega_{1})$
$\displaystyle+\mathcal{G}_{ab,ab}(\omega_{3})\mathcal{G}_{aa,bb}(t_{2})\mathcal{G}_{ba,ba}(\omega_{1})$
$\displaystyle+\mathcal{G}_{ab,ab}(\omega_{3})\mathcal{G}_{bb,bb}(t_{2})\mathcal{G}_{ba,ba}(\omega_{1}),$
$\displaystyle S^{(3)}_{\textrm{nr}}(\omega_{3},t_{2},\omega_{1})=$
$\displaystyle\mathcal{G}_{ab,ab}(\omega_{3})\mathcal{G}_{aa,aa}(t_{2})\mathcal{G}_{ab,ab}(\omega_{1})$
(23)
$\displaystyle+\mathcal{G}_{ab,ab}(\omega_{3})\mathcal{G}_{bb,aa}(t_{2})\mathcal{G}_{ab,ab}(\omega_{1})$
$\displaystyle+\mathcal{G}_{ab,ab}(\omega_{3})\mathcal{G}_{aa,bb}(t_{2})\mathcal{G}_{ab,ab}(\omega_{1})$
$\displaystyle+\mathcal{G}_{ab,ab}(\omega_{3})\mathcal{G}_{bb,bb}(t_{2})\mathcal{G}_{ab,ab}(\omega_{1}).$

Figure 3: Spectral plots of the imaginary part of
$\mathcal{G}_{ab,ab}(\omega)$ in the presence or absence of the control field.
The parameters are $\omega_{ab}=12579$ cm-1, $\Gamma_{1}=1$ cm-1,
$\Gamma_{2}=0.0001$ cm-1, $\gamma_{a}^{(0)}=80$ cm-1,
$\gamma_{b}^{(0)}=\gamma_{c}^{(0)}=1$ cm-1 and $\Omega=50$ cm-1.

We consider a case when the system is prepared at $|a\rangle$ after the first
two pulses, cf. $R_{1}$ and $R_{2}$ in Fig. 2. After $t_{2}$, the probability
of the system remaining at $|a\rangle$ reads

$\displaystyle\mathcal{G}_{aa,aa}(t_{2})\simeq$ $\displaystyle
e^{-\frac{\gamma_{3}}{2}t_{2}}\left[\frac{1}{2}\cos(\widetilde{\Omega}t_{2})\frac{A_{2}\Gamma_{2}+\Omega^{2}}{A_{1}}\right.$
$\displaystyle\left.+\frac{1}{4}\sin(\widetilde{\Omega}t_{2})\frac{A_{2}\Gamma_{2}\gamma_{3}+(\gamma_{3}-2\Gamma_{1})\Omega^{2}}{i\widetilde{\Omega}A_{1}}\right]$
$\displaystyle+\frac{\Gamma_{2}}{2(\Gamma_{1}+\Gamma_{2})}+\frac{\Gamma_{1}(2A_{1}-\Omega^{2})e^{-(\Gamma_{1}+\Gamma_{2})t_{2}}}{2(\Gamma_{1}+\Gamma_{2})A_{1}},$
$\displaystyle\mathcal{G}_{bb,aa}(t_{2})\simeq$ $\displaystyle
e^{-\frac{\gamma_{3}}{2}t_{2}}\left[\frac{1}{2}\cos(\widetilde{\Omega}t_{2})\frac{A_{2}\Gamma_{1}}{A_{1}}\right.$
$\displaystyle\left.+\frac{1}{4}\sin(\widetilde{\Omega}t_{2})\frac{A_{2}\Gamma_{1}\gamma_{3}+2\Omega^{2}}{i\widetilde{\Omega}A_{1}}\right]$
$\displaystyle+\frac{\Gamma_{1}}{2(\Gamma_{1}+\Gamma_{2})}+\frac{\Gamma_{1}(\Omega^{2}-2A_{1})e^{-(\Gamma_{1}+\Gamma_{2})t_{2}}}{2(\Gamma_{1}+\Gamma_{2})A_{1}},$

where

$\displaystyle A_{1}$ $\displaystyle=$
$\displaystyle(\Gamma_{1}+\Gamma_{2})(\Gamma_{1}+\Gamma_{2}-\gamma_{3})+\Omega^{2},$
(26) $\displaystyle A_{2}$ $\displaystyle=$
$\displaystyle\Gamma_{1}+\Gamma_{2}-\gamma_{3},$ (27)
$\displaystyle\widetilde{\Omega}$ $\displaystyle=$
$\displaystyle\frac{1}{2}\sqrt{4\Omega^{2}-\gamma_{3}^{2}}.$ (28)

When $\Omega=0$, Eqs. (II) and (II) can be simplified as

$\displaystyle\mathcal{G}_{aa,aa}(t_{2})=$
$\displaystyle\frac{\Gamma_{2}}{\Gamma_{1}+\Gamma_{2}}+\frac{\Gamma_{1}}{\Gamma_{1}+\Gamma_{2}}e^{-(\Gamma_{1}+\Gamma_{2})t_{2}},$
(29) $\displaystyle\mathcal{G}_{bb,aa}(t_{2})=$
$\displaystyle\frac{\Gamma_{1}}{\Gamma_{1}+\Gamma_{2}}\left[1-e^{-(\Gamma_{1}+\Gamma_{2})t_{2}}\right].$
(30)

In Fig. 4, we numerically plot $\mathcal{G}_{aa,aa}(t_{2})$ and
$\mathcal{G}_{bb,aa}(t_{2})$ in the presence of the control field. It can be
found that there is coherent population transfer between $|a\rangle$ and
$|c\rangle$, after a time scale with $2/\gamma_{3}$, the coherent oscillations
vanish, $\mathcal{G}_{aa,aa}(t_{2})$ and $\mathcal{G}_{bb,aa}(t_{2})$ finally
approach a steady state $\Gamma_{2}/2(\Gamma_{1}+\Gamma_{2})$ and
$\Gamma_{1}/2(\Gamma_{1}+\Gamma_{2})$. However, when the control field
disappears, the coherent oscillation no longer exists, and the population in
the $|a\rangle$ and $|b\rangle$ states monotonically change until they reach a
steady state of $\Gamma_{2}/(\Gamma_{1}+\Gamma_{2})$ and
$\Gamma_{1}/(\Gamma_{1}+\Gamma_{2})$.

Similarly, when the system is prepared at $|b\rangle$ after the first two
pulses, cf. $R_{3}$ and $R_{4}$ in Fig. 2, the probability of the system
remaining at $|b\rangle$ is

$\displaystyle\mathcal{G}_{aa,bb}(t_{2})\simeq\frac{\Gamma_{2}-e^{-(\Gamma_{1}+\Gamma_{2})t_{2}}\Gamma_{2}}{\Gamma_{1}+\Gamma_{2}},$
(31)
$\displaystyle\mathcal{G}_{bb,bb}(t_{2})\simeq\frac{\Gamma_{1}+e^{-(\Gamma_{1}+\Gamma_{2})t_{2}}\Gamma_{2}}{\Gamma_{1}+\Gamma_{2}}.$
(32)

Since there is no control field applied to the transition between $|a\rangle$
and $|b\rangle$, there is no coherent oscillation of
$\mathcal{G}_{aa,bb}(t_{2})$ and $\mathcal{G}_{bb,bb}(t_{2})$ as compared to
$\mathcal{G}_{aa,aa}(t_{2})$ and $\mathcal{G}_{bb,aa}(t_{2})$. However, after
a time scale with $1/(\Gamma_{1}+\Gamma_{2})$, they finally approach a steady
state $\Gamma_{2}/(\Gamma_{1}+\Gamma_{2})$ and
$\Gamma_{1}/(\Gamma_{1}+\Gamma_{2})$ as implied by the detailed balance.

Figure 4: In the presence and absence of a control field, population dynamics
of $|a\rangle$ and $|b\rangle$. Sub-figure (b) is an enlarged view of (a) when
$t_{2}$ is shortened. The other parameters are the same as in Fig. 3.

##  III Results

Figure 5: In the absence of the control field, the 2D spectra of the rephasing
signal. The parameters are the same as in Fig. 3. Figure 6: In the presence of
the control field, the 2D spectra of the rephasing signal for (a) $t_{2}=0$,
(b) $t_{2}=2$ ps, (c) $t_{2}=4$ ps, and (d) $t_{2}=300$ ps. The parameters are
the same as in Fig. 3.

We choose $F=2$ and $F=3$ for ${\rm 5^{2}S_{1/2}}$ of ${\rm{}^{85}Rb}$ as the
ground and metastable state and $F=2$ for ${\rm 5^{2}P_{1/2}}$ as the excited
state. First of all, we shall plot the rephasing signal without the control
field in Fig. 5. As shown, there is only one peak at
$\omega_{ab}=\omega_{a}-\omega_{b}$. As the population time $t_{2}$ elapses,
both the position and the height of the peak remains the same according to
Eqs. (29), (30), (31), (32) and (22). This is because the number of atoms in
the energy levels $a$ and $b$ is conserved. In other words,
$\mathcal{G}_{aa,aa}(t_{2})+\mathcal{G}_{bb,aa}(t_{2})=1$ and
$\mathcal{G}_{aa,bb}(t_{2})+\mathcal{G}_{bb,bb}(t_{2})=1$. When
$t_{2}\gg(\Gamma_{1}+\Gamma_{2})^{-1}$, the population transfer effectively
ends. However, when the control field is applied, as shown in Fig. 6, the
original one big peak is split into four small ones with widths significantly
narrower than the original one due to the EIT effect. As shown in Fig. 3, the
imaginary part of the Green functions in $t_{3}$ have been changed from single
trough at $\omega_{3}=\omega_{ab}$ to double troughs at

$\displaystyle\omega_{3}=\omega_{ab}$
$\displaystyle\pm\frac{1}{2}\sqrt{\frac{\Omega(\gamma_{1}+\gamma_{2})\sqrt{4\gamma_{1}\gamma_{2}+\Omega^{2}}-\gamma_{2}(4\gamma_{1}\gamma_{2}+\Omega^{2})}{\gamma_{1}}},$
(33)

according to Eq. (II). The imaginary part of Green’s function in $t_{1}$ will
likewise change from one trough to two troughs. The reason for this phenomenon
is given as follows. The larger the magnitude of the negative imaginary Green
function is, the more the probe field is absorbed by the medium and thus the
bigger the transition probability is. In the absence of the control field, the
particle will only transit between the states $|b\rangle$ and $|a\rangle$.
When the control field is applied, there are many different contributions to
the transition paths, which start from $|b\rangle$ and end at $|a\rangle$,
such as
$|b\rangle\rightarrow|a\rangle\rightarrow|c\rangle\rightarrow|a\rangle$,
$|b\rangle\rightarrow|a\rangle\rightarrow|c\rangle\rightarrow|a\rangle\rightarrow|c\rangle\rightarrow|a\rangle$
and so on. There will be interference between these transition paths, and the
absorption of the probe field is the sum of the transition probability
amplitudes of all paths. When the signs of the transition probability
amplitudes are opposite to each other and thus cancel with each other, EIT is
formed. Although both the shapes and the positions of these peaks will not be
changed as $t_{2}$ is enlarged, the heights of the peaks decline and rise with
$t_{2}\sim 2/\gamma_{3}$ until steady state according to Eqs. (II), (II),
(31), (32) and (22), which is similar to Fig. 4.

Then, we plot the non-rephasing signal in the absence of the control field in
Fig. 7. Because the states in the durations of $t_{1}$ and $t_{3}$ are the
same for $R_{1}$ and $R_{4}$, a trough appears at
$\omega_{1}=\omega_{3}=\omega_{ab}$ in the two-dimensional spectroscopy in
this case. The depth of the trough will not change with the extension of
$t_{2}$ time, as can be seen from Eqs. (29), (30), (31), (32) and (23). When
the control field is added, that is, EIT is introduced, one trough in the
original two-dimensional spectroscopy will be split into four small troughs,
as shown in Fig. 8. Equations (II), (II), (31), (32) and (23) imply that as
$t_{2}$ time increases, the population in the $a$ and $b$ states will carry on
the coherent oscillations with decreasing amplitude. After about $200$ ps, the
populations of both $a$ and $b$ states become stable. Hence, both the position
and shape of the troughs do not change in the spectroscopy, but their depths
oscillate with the passage of $t_{2}$ time.

Summing the rephasing and non-rephasing signals, the total absorptive 2D
spectroscopy is obtained, which is more complicated as can be seen from Fig.
9. There are two peaks along the diagonal line, and two troughs along the
anti-diagonal line. Moreover, with the increase of $t_{2}$ time, the atoms in
the $a$ and $b$ energy levels transition to each other, but the overall height
of the peaks and the depth of the troughs remain the same. When the control
field is added, as shown in Fig. 10, in addition to the above two sets of main
peaks and troughs, two new sets of peaks and troughs emerge, i.e., eight
narrow peaks and troughs between one main peak and trough, and four small ones
at the center. Similar to the rephasing and non-rephasing signals, the time
evolution of both the peaks and troughs shows damped oscillation and they
finally arrive at the steady state in the long-time limit. Neither the number
nor the positions of the peaks and troughs in the 2DES change due to the
elapse of $t_{2}$. It implies that both the rephasing and non-rephasing
spectroscopy possess the shape over $\omega_{1}$ and $\omega_{3}$, although
the magnitude changes over time.

Figure 7: In the absence of the control field, the 2D spectra of the non-
rephasing signal. The other parameters are the same as in Fig. 3. Figure 8: In
the presence of the control field, the 2D spectra of the non-rephasing signal
for (a) $t_{2}=0$, (b) $t_{2}=2$ ps, (c) $t_{2}=4$ ps, and (d) $t_{2}=300$ ps.
The other parameters are the same as in Fig. 3. Figure 9: In the absence of
the control field, the 2D spectra of the absorptive signal. The other
parameters are the same as in Fig. 3. Figure 10: In the presence of the
control field, the 2D spectra of the absorptive signal for (a) $t_{2}=0$, (b)
$t_{2}=2$ ps, (c) $t_{2}=4$ ps, and (d) $t_{2}=300$ ps. The other parameters
are the same as in Fig. 3.

##  IV Conclusion

In this paper, we apply EIT which can induce constructive and destructive
interference to two-dimensional spectroscopy. It is found that in the
rephasing and non-rephasing signals, the original one large peak (trough) is
split into four small peaks (troughs). With the evolution of the population
time, the height of the peaks (the depth of the troughs) will rise and fall
until a steady value in the long-time limit due to the transition of the
atoms. But both the position and shape of the peak (trough) will not change
during the evolution. The time evolution of the absorptive signal will show a
similar characteristic although its spectroscopy is much more complicated.
This means that EIT can improve the resolution of 2DES, and this method is
promising to obtain more information about excited state dynamics.

Acknowledgments.— This work is supported by Beijing Natural Science Foundation
under Grant No. 1202017 and the National Natural Science Foundation of China
under Grant Nos. 11674033, 11505007, and Beijing Normal University under Grant
No. 2022129.

## References

  * Scully and Zubairy (1997) M. O. Scully and M. S. Zubairy,  _Quantum Optics_ (Cambridge University Press, New York, 1997).
  * Hopkins  _et al._ (1997) S. A. Hopkins, E. Usadi, H. X. Chen, and A. V. Durrant, “Electromagnetically induced transparency of laser-cooled rubidium atoms in three-level $\Lambda$-type systems,” Opt. Commun. 138, 185 (1997).
  * Hau  _et al._ (1999) L. V. Hau, S. E. Harris, Z. Dutton, and C. H. Behroozi, “Light speed reduction to 17 metres per second in an ultracold atomic gas,” Nature 397, 594 (1999).
  * Shiau  _et al._ (2011) B.-W. Shiau, M.-C. Wu, C.-C. Lin, and Y.-C. Chen, “Low-light-level cross-phase modulation with double slow light pulses,” Phys. Rev. Lett. 106, 193006 (2011).
  * Schmidt and Imamo?lu (1996) H. Schmidt and A. Imamoǧlu, “Nonlinear optical devices based on a transparency in semiconductor intersubband transitions,” Opt. Commun. 131, 333 (1996).
  * Zhao  _et al._ (1997) Y. Zhao, C.-K. Wu, B.-S. Ham, M. K. Kim, and E. Awad, “Microwave induced transparency in ruby,” Phys. Rev. Lett. 79, 641 (1997).
  * Kuznetsova  _et al._ (2002) E. Kuznetsova, O. Kocharovskaya, P. Hemmer, and M. O. Scully, “Atomic interference phenomena in solids with a long-lived spin coherence,” Phys. Rev. A 66, 063802 (2002).
  * Baldit  _et al._ (2010) E. Baldit, K. Bencheikh, P. Monnier, S. Briaudeau, J. A. Levenson, V. Crozatier, I. Lorgeré, F. Bretenaker, J. L. Le Gouët, O. Guillot-Noël, and Ph. Goldner, “Identification of $\Lambda$-like systems in Er3+: Y2SiO5 and observation of electromagnetically induced transparency,” Phys. Rev. B 81, 144303 (2010).
  * Fan  _et al._ (2019) H.-Q. Fan, K. H. Kagalwala, S. V. Polyakov, A. L. Migdall, and E. A. Goldschmidt, “Electromagnetically induced transparency in inhomogeneously broadened solid media,” Phys. Rev. A 99, 053821 (2019).
  * Wei and Manson (1999) C.-J. Wei and N. B. Manson, “Observation of the dynamic stark effect on electromagnetically induced transparency,” Phys. Rev. A 60, 2540 (1999).
  * Hemmer  _et al._ (2001) P. R. Hemmer, A. V. Turukhin, M. S. Shahriar, and J. A. Musser, “Raman-excited spin coherences in nitrogen-vacancy color centers in diamond,” Opt. Lett. 26, 361 (2001).
  * Acosta  _et al._ (2013) V. M. Acosta, K. Jensen, C. Santori, D. Budker, and R. G. Beausoleil, “Electromagnetically induced transparency in a diamond spin ensemble enables all-optical electromagnetic field sensing,” Phys. Rev. Lett. 110, 213605 (2013).
  * Wang  _et al._ (2018) Y.-Y. Wang, J. Qiu, Y.-Q. Chu, M. Zhang, J.-M. Cai, Q. Ai, and F.-G. Deng, “Dark state polarizing a nuclear spin in the vicinity of a nitrogen-vacancy center,” Phys. Rev. A 97, 042313 (2018).
  * Fleischhauer and Lukin (2000) M. Fleischhauer and M. D. Lukin, “Dark-state polaritons in electromagnetically induced transparency,” Phys. Rev. Lett. 84, 5094 (2000).
  * Liu  _et al._ (2001) C. Liu, Z. Dutton, C. H. Behroozi, and L. V. Hau, “Observation of coherent optical information storage in an atomic medium using halted light pulses,” Nature 409, 490 (2001).
  * Fleischhauer and Lukin (2002) M. Fleischhauer and M. D. Lukin, “Quantum memory for photons: Dark-state polaritons,” Phys. Rev. A 65, 022314 (2002).
  * Honda  _et al._ (2008) K. Honda, D. Akamatsu, M. Arikawa, Y. Yokoi, K. Akiba, S. Nagatsuka, T. Tanimura, A. Furusawa, and M. Kozuma, “Storage and retrieval of a squeezed vacuum,” Phys. Rev. Lett. 100, 093601 (2008).
  * Hsiao  _et al._ (2018) Y.-F. Hsiao, P.-J. Tsai, H.-S. Chen, S.-X. Lin, C.-C. Hung, C.-H. Lee, Y.-H. Chen, Y.-F. Chen, A. Y. Ite, and Y.-C. Chen, “Highly efficient coherent optical memory based on electromagnetically induced transparency,” Phys. Rev. Lett. 120, 183602 (2018).
  * Chen  _et al._ (2013) Y.-H. Chen, M.-J. Lee, I.-C. Wang, S.-W. Du, Y.-F. Chen, Y.-C. Chen, and A. Y. Ite, “Coherent optical memory with high storage efficiency and large fractional delay,” Phys. Rev. Lett. 110, 083601 (2013).
  * Kukharchyk  _et al._ (2020) N. Kukharchyk, D. Sholokhov, O. Morozov, S. L. Korableva, A. A. Kalachev, and P. A. Bushev, “Electromagnetically induced transparency in a mono-isotopic 167Er: 7LiYF4 crystal below 1 Kelvin: microwave photonics approach,” Opt. Express 28, 29166 (2020).
  * Wang  _et al._ (2023) J. Wang, W.-T. He, C.-W. Lu, Y.-Y. Wang, Q. Ai, and H.-B. Wang, “Controlled-not gate based on the rydberg states of surface electrons,” Ann. Phys. (Berlin) 535, 2300138 (2023).
  * Jin  _et al._ (2013) X.-R. Jin, Y.-Q. Zhang, S. Zhang, Y.-P. Lee, and J.-Y. Rhee, “Polarization-independent electromagnetically induced transparency-like effects in stacked metamaterials based on Fabry–Perot resonance,” J. Optics 15, 125104 (2013).
  * Bagci and Akaoglu (2019) F. Bagci and B. Akaoglu, “Transmission control by asymmetric electromagnetically induced transparency-like metamaterials in transverse electromagnetic waveguide,” Phys. Lett. A 383, 126000 (2019).
  * Jin  _et al._ (2011) X.-R. Jin, Y.-H. Lu, H.-Y. Zheng, Y.-P. Lee, J.-Y. Rhee, K.-W. Kim, and W.-H. Jang, “Plasmonic electromagnetically-induced transparency in metamaterial based on second-order plasmonic resonance,” Opt. Commun. 284, 4766 (2011).
  * Tassin  _et al._ (2009) P. Tassin, L. Zhang, Th. Koschny, E. N. Economou, and C. M. Soukoulis, “Low-loss metamaterials based on classical electromagnetically induced transparency,” Phys. Rev. Lett. 102, 053901 (2009).
  * Xu  _et al._ (2010) H. Xu, Y.-H. Lu, Y.-P. Lee, and B.-S. Ham, “Studies of electromagnetically induced transparency in metamaterials,” Opt. Express 18, 17736 (2010).
  * Huang  _et al._ (2022) H.-B. Huang, J.-J. Lin, Y.-X. Yao, K.-Y. Xia, Z.-Q. Yin, and Q. Ai, “Optical nonreciprocity in rotating diamond with nitrogen-vacancy color centers,” Ann. Phys. (Berlin) 534, 2200157 (2022).
  * Boller  _et al._ (1991) K.-J. Boller, A. Imamoğlu, and S. E. Harris, “Observation of electromagnetically induced transparency,” Phys. Rev. Lett. 66, 2593 (1991).
  * Mukamel (1999) S. Mukamel,  _Principles of Nonlinear Optics and Spectroscopy_ (Oxford University Press, New York, 1999).
  * Schlau-Cohen  _et al._ (2011) G. S. Schlau-Cohen, A. Ishizaki, and G. R. Fleming, “Two-dimensional electronic spectroscopy and photosynthesis: Fundamentals and applications to photosynthetic light-harvesting,” Chem. Phys. 386, 1 (2011).
  * Fuller and Ogilvie (2015) F. D. Fuller and J. P. Ogilvie, “Experimental implementations of two-dimensional fourier transform electronic spectroscopy,” Annu. Rev. Phys. Chem. 66, 667 (2015).
  * Sun  _et al._ (2023) Z.-H. Sun, Y.-X. Yao, Q. Ai, and Y.-C. Cheng, “Theory of center-line slope in 2D electronic spectroscopy with static disorder,” Adv. Quantum Technol. 6, 2300163 (2023).
  * Liu  _et al._ (2020) G.-Y. Liu, A. S. Sizhuk, and K. E. Dorfman, “Selective elimination of homogeneous broadening by multidimensional spectroscopy in the electromagnetically induced transparency regime,” J. Phys. Chem. Lett. 11, 5504 (2020).
  * Seiler  _et al._ (2018) H. Seiler, S. Palato, C. Sonnichsen, H. Baker, and P. Kambhampati, “Seeing multiexcitons through sample inhomogeneity: Band-edge biexciton structure in cdse nanocrystals revealed by two-dimensional electronic spectroscopy,” Nano Lett. 18, 2999. (2018).
  * Brosseau  _et al._ (2020) P. Brosseau, S. Palato, H. Seiler, H. Baker, and P. Kambhampati, “Fifth-order two-quantum absorptive two-dimensional electronic spectroscopy of CdSe quantum dots,” J. Chem Phys. 153, 234703 (2020).
  * Moody  _et al._ (2015) G. Moody, C. Kavir Dass, K. Hao, C.-H. Chen, L.-J. Li, A. Singh, K. Tran, G. Clark, X.-D. Xu, G. Berghäuser, E. Malic, A. Knorr, and X.-Q. Li, “Intrinsic homogeneous linewidth and broadening mechanisms of excitons in monolayer transition metal dichalcogenides,” Nat. Commun. 6, 8315 (2015).
  * Guo  _et al._ (2019) L. Guo, M. Wu, T. Cao, D. M. Monahan, Y.-H. Lee, S. G. Louie, and G. R. Fleming, “Exchange-driven intravalley mixing of excitons in monolayer transition metal dichalcogenides,” Nat. Phys. 15, 228 (2019).
  * Jha  _et al._ (2017) A. Jha, H.-G. Duan, V. Tiwari, P. K. Nayak, H. J. Snaith, M. Thorwart, and R. J. D. Miller, “Direct observation of ultrafast exciton dissociation in lead iodide perovskite by 2D electronic spectroscopy,” Acs Photonics 5, 852 (2017).
  * Monahan  _et al._ (2017) D. M. Monahan, L. Guo, J. Lin, L.-T. Dou, P.-D. Yang, and G. R. Fleming, “Room-temperature coherent optical phonon in 2D electronic spectra of CH3NH3PbI3 perovskite as a possible cooling bottleneck,” J. Phys. Chem. Lett. 8, 3211 (2017).
  * Seiler  _et al._ (2019) H. Seiler, S. Palato, C. Sonnichsen, H. Baker, E. Socie, D. P. Strandell, and P. Kambhampati, “Two-dimensional electronic spectroscopy reveals liquid-like lineshape dynamics in cspbi3 perovskite nanocrystals,” Nat. Commun. 10, 4962 (2019).
  * Calhoun  _et al._ (2009) T. R. Calhoun, N. S. Ginsberg, G. S. Schlau-Cohen, Y.-C. Cheng, M. Ballottari, R. Bassi, and G. R. Fleming, “Quantum coherence enabled determination of the energy landscape in light-harvesting complex II,” J. Phys. Chem. B 113, 16291 (2009).
  * Fuller  _et al._ (2014) F. D. Fuller, J. Pan, A. Gelzinis, V. Butkus, S. S. Senlik, D. E. Wilcox, C. F. Yocum, L. Valkunas, D. Abramavicius, and J. P. Ogilvie, “Vibronic coherence in oxygenic photosynthesis,” Nat. Chem. 6, 706 (2014).
  * Jun  _et al._ (2014) S.-H. Jun, C.-H. Yang, M. Isaji, H. Tamiaki, J.-H. Kim, and H. Ihee, “Coherent oscillations in chlorosome elucidated by two-dimensional electronic spectroscopy,” J. Phys. Chem. Lett. 5, 1386 (2014).
  * Romero  _et al._ (2014) E. Romero, R. Augulis, V. I. Novoderezhkin, M. Ferretti, J. Thieme, D. Zigmantas, and R. van Grondelle, “Quantum coherence in photosynthesis for efficient solar-energy conversion,” Nat. Phys. 10, 676 (2014).
  * Jun  _et al._ (2015) S.-H. Jun, C.-H. Yang, T.-W. Kim, M. Isaji, H. Tamiaki, H. Ihee, and J.-H. Kim, “Role of thermal excitation in ultrafast energy transfer in chlorosomes revealed by two-dimensional electronic spectroscopy,” Phys. Chem. Chem. Phys. 17, 17872 (2015).
  * Ma  _et al._ (2017) F. Ma, L.-J. Yu, R. Hendrikx, Z. Y. Wang Otomo, and R. van Grondelle, “Excitonic and vibrational coherence in the excitation relaxation process of two LH1 complexes as revealed by two-dimensional electronic spectroscopy,” J. Phys. Chem. Lett. 8, 2751 (2017).
  * Thyrhaug  _et al._ (2018) E. Thyrhaug, R. Tempelaar, M. J. P. Alcocer, K. Žídek, D. Bína, J. Knoester, T. L. C. Jansen, and D. Zigmantas, “Identification and characterization of diverse coherences in the Fenna–Matthews–Olson complex,” Nat. Chem. 10, 780 (2018).
  * Breuer and Petruccione (2002) H. P. Breuer and F. Petruccione,  _The Theory of Open Quantum Systems_ (Oxford University Presss, 2002).


(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: Cooperative Medianet Innovation Center , Shanghai Jiao Tong
University, China. 22institutetext: 22email:
<EMAIL_ADDRESS>
New York University, USA. 33institutetext: 33email<EMAIL_ADDRESS>
Shanghai AI laboratory, China.

# Self-Localized Collaborative Perception

Zhenyang Ni 11 Zixing Lei 11 Yifan Lu 11 Dingju Wang 11 Chen Feng 22 Yanfeng
Wang 1133 Siheng Chen 113*3*

###### Abstract

Collaborative perception has garnered considerable attention due to its
capacity to address several inherent challenges in single-agent perception,
including occlusion and out-of-range issues. However, existing collaborative
perception systems heavily rely on precise localization systems to establish a
consistent spatial coordinate system between agents. This reliance makes them
susceptible to large pose errors or malicious attacks, resulting in
substantial reductions in perception performance. To address this, we propose
$\mathtt{CoBEVGlue}$, a novel self-localized collaborative perception system,
which achieves more holistic and robust collaboration without using an
external localization system. The core of $\mathtt{CoBEVGlue}$ is a novel
spatial alignment module, which provides the relative poses between agents by
effectively matching co-visible objects across agents. We validate our method
on both real-world and simulated datasets. The results show that i)
$\mathtt{CoBEVGlue}$ achieves state-of-the-art detection performance under
arbitrary localization noises and attacks; and ii) the spatial alignment
module can seamlessly integrate with a majority of previous methods, enhancing
their performance by an average of $57.7\%$. Code is available at
https://github.com/VincentNi0107/CoBEVGlue.

###### Keywords:

Collaborative perception Bird’s eye view Autonomous driving

##  1 Introduction

Figure 1:  Noise is pervasive in external localization systems, leading to
substantial reductions on the detection performance of collaborative
perception systems. (a) and (b) show snapshots from the real-world
collaborative perception dataset DAIR-V2X [68] and V2V4Real[60]. Red denotes
point cloud and ground truth bounding boxes from the ego agent and blue
belongs to the collaborator. The point cloud and bounding boxes from
collaborator are transformed to the coordinate system of ego agent with ground
truth poses. Despite resource-intensive offline calibration efforts, the
ground truth localization error persists at the meter-level. (c) shows that
current collaborative perception systems fail to transcend the no
collaboration baseline under large localization noise on DAIR-V2X. In
comparison, our $\mathtt{CoBEVGlue}$ achieves state-of-art detection
performance when localization noise exist, performing comparably to systems
relying on precise localization information.

Accurate perception is essential for the navigation and safety of autonomous
vehicles [32, 40]. Despite advancements facilitated by large-scale datasets
[8, 51], and powerful models [27, 66], single-agent perception is inherently
limited by occlusions and long-range issues [56], which could lead to
catastrophic consequences [72]. Leveraging modern communication technologies,
current research in collaborative perception [30, 21, 61, 56] enables the
sharing of perceptual information among multiple agents, fundamentally
improving the perception performance. Fueled by the advent of high-quality
datasets [62, 68, 60, 29] and innovative collaborative techniques [34, 35, 75,
2, 59, 57], collaborative perception systems have the potential to improve the
safety of transportation networks significantly.

In this emerging field of collaborative perception, most prevailing works [56,
30, 21] make an oversimplified assumption: the global localization system,
typically GPS or SLAM, employed by each agent is precise enough to establish a
consistent spatial coordinate system for collaboration. However, snapshots
from real-world collaborative perception datasets V2V4Real [60] and DAIR-V2X
[68] show that the ground truth localization is still noisy even after
meticulous and resource-intensive offline calibration, as shown in Fig.
1(a)(b). These inaccuracies could be far more exacerbated in real-world
applications under computing limits and real-time constraints. Moreover,
localization systems are susceptible to long-existing yet still unsolved
attacks [42, 41, 71, 50, 24, 67]. These attacks allow adversaries to
manipulate positions at will, further undermining the reliability of
localization systems. Such prevalent challenges of significant noise and
malicious attack starkly contrast with the ideal scenarios considered by
earlier works [53, 69, 61, 36], which primarily focus on minor pose
inaccuracies and fail to transcend the no collaboration baseline under large
noise, see Fig. 1(c).

To eliminate the dependence on potentially unreliable external localization
systems, a direct solution is deducing the relative poses of collaborative
agents by point cloud registration, a technique extensively utilized in multi-
agent collaborative systems[26, 73, 10, 9, 74]. Point cloud registration
methods [4, 63, 1, 22] apply nearest-neighbor algorithms to identify
correspondences across extensive 3D point sets, followed by robust techniques
[16, 63] to calculate the transformation from these putative correspondences.
Although these methods prove effective for latency-tolerant applications such
as collaborative mapping[26], the real-time transmission of large volumes of
3D data is impractical for bandwidth-limited collaborative perception systems
[23, 21, 11, 60]. Therefore, there exists a pronounced gap in creating a
system that is free from localization errors while also maintaining
communication efficiency for practical applications.

To fill this gap, we propose $\mathtt{CoBEVGlue}$, a self-localized
collaborative perception system that is designed for multiple agents to
achieve more holistic perception without relying on external localization
systems, achieving efficiency with reduced communication costs.
$\mathtt{CoBEVGlue}$ follows the pipeline of previous collaborative perception
systems [36, 21], and uses its key spatial alignment module $\mathtt{BEVGlue}$
to estimate the relative pose between agents with the objects detected and
tracked by each agent. The core idea behind $\mathtt{BEVGlue}$ is to search
for the co-visible objects from the bird’s eye view perceptual data across
agents and calculate the relative transformation with these co-visible
objects, ensuring a consistent spatial coordinate system for collaboration.
$\mathtt{BEVGlue}$ includes three key components: i) object graph modeling,
which converts each agent’s observations to an object graph with rich
information, including object shape, heading, tracking ID and the invariant
spatial relationship between objects; ii) temporally consistent maximum
subgraph detection, which efficiently harnesses spatial and temporal data
within object graphs to detect the largest common subgraph, following strict
spatial isomorphism constraint and temporal consistency; and iii) relative
pose calculation, which computes the pose relationships between agents using
the detected common subgraph, without using time-consuming outlier rejection
algorithms.

The proposed $\mathtt{CoBEVGlue}$ system offers three significant advantages:
i) it operates independently of external localization devices, showcasing its
resilience to noise and malicious attacks; ii) it brings minor communication
overhead since $\mathtt{CoBEVGlue}$ only uses object bounding boxes with
tracking ID to estimate the relative pose between agents; iii) its core
module, $\mathtt{BEVGlue}$ ensures high-quality matching results by keeping
strict spatial isomorphism constraint between the detected common subgraph and
temporal consistency between matching results across time.

To evaluate the effectiveness of the proposed method, we consider the
collaborative 3D object detection task on three datasets: OPV2V [62], DAIR-V2X
[68] and V2V4Real [60], covering both simulation and real-world scenarios. The
results show that, the $\mathtt{CoBEVGlue}$ empowered robust collaborative
perception system perform comparably to systems relying on precise
localization information, and achieves state-of-art detection performance when
localization noise and attack exist.

In summary, the main contributions of this work are:

  * •

We propose $\mathtt{CoBEVGlue}$, the first self-localized collaborative
perception system without relying on external localization devices;

  * •

We propose $\mathtt{BEVGlue}$, a novel spatial alignment method that estimates
the relative poses between agents through matching co-visible objects;

  * •

We conduct extensive experiments for collaborative LiDAR object detection in
simulated and real-world datasets. The results show that
i)$\mathtt{CoBEVGlue}$ attains state-of-the-art detection performance in the
presence of localization noise. ii)$\mathtt{BEVGlue}$ can seamlessly integrate
with a majority of previous methods, enhancing their performance by an average
of $57.7\%$.

##  2 Related work

###  2.1 Collaborative Perception

As a recent application of multi-agent systems to perception tasks,
collaborative perception is emerging [62, 56, 30, 21]. To support this area of
research, there is a surge of high-quality datasets, including V2X-Sim [29],
OPV2V [62], and DAIR-V2X [68]. Based on those datasets, numerous methods have
been proposed to handle various practical issues, such as communication
latency [28] and communication bandwidth [21]. In this work, we specifically
consider the robustness of localization error and attack.

To gain resistance towards localization noises, previous works consider two
main approaches: learning-based and matching-based. Learning-based methods aim
to construct robust network architectures to reduce the impact of pose errors.
For example, V2VNet (robust) [53] designs pose regression, global consistency
and attention aggregation module to correct relative poses and concentrate on
neighbor with less pose error; V2X-ViT [61] uses multi-scale window attention
to capture features in various ranges. On the other hand, matching-based
approaches seek to develop robust frameworks or network architectures.
Examples include FPV-RCNN [69] and CoAlign [36], which estimate relative poses
between agents using an IoU-based matching strategy. However, they can only
rectify minor inaccuracies in external localization since these approaches
rely on a basically precise initial relative pose. Their performance drops
significantly when the noise is large or an attack exists. In contrast, our
work considers collaborative perception independent of external localization
systems.

###  2.2 Point Cloud Registration

Although the ultimate aim of this paper is to enhance detection capabilities,
advancements in point cloud registration methods has inspired us to propose
our novel self-localized collaborative perception system. Traditional point
cloud registration methods focusing on refining the Iterative Closest Point
(ICP) algorithm [4] and its variants [7, 17, 19, 49] have led to improvements
in convergence and noise resilience. Recent typical point cloud registration
workflows consist of extracting local 3D feature descriptors and conducting
registration. For extracting 3D local descriptors, conventional approaches
like Fast Point Feature Histograms [46, 25, 47, 52] utilize hand-crafted
features. More recent techniques [44, 43, 1, 12, 70, 31] adopt learning-based
methods for this purpose. In terms of registration, traditional approaches
often employ nearest-neighbor algorithms for matching and robust optimization
for outlier rejection [16, 63], whereas contemporary deep registration methods
[22, 65, 64] leverage self-attention mechanisms [54] for correspondence
determination. SGAligner [48] pioneers the employment of a pre-constructed 3D
scene graph for registration purposes. Nevertheless, similar to preceding
strategies, it requires the transmission of dense point clouds and high-
dimensional features. These methods are widely applied in latency-tolerant
multi-agent systems such as collaborative mapping[26] and 3D scene graph
generation[10]. However, the collaborative object detection task requires
precise relative pose estimation in real time. Unfortunately, the V2X networks
struggle to transmit the dense point clouds and feature required by point
cloud registration methods in real time. To overcome this limitation, our
approach prioritizes object-level registration, representing each object with
just eight float numbers. This innovation markedly reduces the bandwidth
necessary and computation cost for calculating relative poses among
collaborative autonomous vehicles, thus efficiently resolving the transmission
dilemma.

###  2.3 Maximum Common Subgraph Detection

The Maximum Common Subgraph (MCS) detection problem, classified as NP-hard, is
pivotal in various scientific fields ene[45, 13, 15, 18], necessitating
algorithms that balance precision and computational efficiency. Traditional
approaches primarily employ branch-and-bound algorithms [39, 55, 38] and
techniques that transform MCS detection into maximum clique problems [45, 37].
Recent advancements [33, 3] in machine learning have seen the application of
graph neural networks and reinforcement learning to MCS detection, which
attempts to learn suitable heuristics for graph matching. Despite their
innovations, they are still constrained by the heuristic nature of the search
space exploration and are subject to exponential time complexity in the worst-
case scenarios. In this work, we model the bounding boxes detected by each
agent with a geometric invariant object pose graph and leverage the spatial
constraints and temporal consistency to solve the problem efficiently.

##  3 CoBEVGlue: Self-Localized Collaborative Perception System

Figure 2: Overview of the proposed self-localized collaborative perception
framework. The key module is $\mathtt{BEVGlue}$, which leverages object graphs
and temporally consistent MCS detection to achieve spatial alignment.

In this section, we present $\mathtt{CoBEVGlue}$, the first self-localized
collaborative perception system that replaces potentially unreliable
localization systems with our novel spatial alignment module
$\mathtt{BEVGlue}$ to estimate the relative pose between agents.
$\mathtt{CoBEVGlue}$ includes a single-agent object detector and tracker, the
key spatial alignment module $\mathtt{BEVGlue}$, a multi-agent feature fusion
module, and a decoder; see the overview in Fig. 2.

Mathematically, consider $N$ agents in the scene. For the $i$th agent, let
$\mathbf{O}_{i}^{t}$ be the perceptual observation at time $t$. The proposed
$\mathtt{CoBEVGlue}$ works as follows:

$\displaystyle\mathbf{F}_{i}^{t},\mathbf{D}_{i}^{t}$ $\displaystyle=$
$\displaystyle f_{\rm detection\&tracking}\left(\mathbf{O}_{i}^{t}\right),$
(1a) $\displaystyle\xi_{j\rightarrow i}^{t}$ $\displaystyle=$ $\displaystyle
f_{\rm BEVGlue}\left(\mathbf{D}_{i}^{t},\mathbf{D}_{j}^{t}\right),$ (1b)
$\displaystyle\mathbf{F}_{j\rightarrow i}^{t}$ $\displaystyle=$ $\displaystyle
f_{\rm transform}\left(\mathbf{F}_{j}^{t},\xi_{j\rightarrow i}^{t}\right),$
(1c) $\displaystyle\mathbf{F}_{i}^{{}^{\prime}t}$ $\displaystyle=$
$\displaystyle f_{\rm fusion}\left(\\{\mathbf{F}_{j\rightarrow
i}^{t}\\}_{j=1,2,\cdots,N}\right),$ (1d) $\displaystyle\mathbf{B}_{i}^{t}$
$\displaystyle=$ $\displaystyle f_{\rm
decoder}\left(\mathbf{F}_{i}^{{}^{\prime}t}\right),$ (1e)

where $\mathbf{F}_{i}^{t}$ is the BEV feature extracted from the $i$th agent’s
observation, $\mathbf{D}_{i}^{t}$ is the detection and tracking outputs
without collaboration, $\xi_{j\rightarrow i}^{t}$ is the estimated relative
pose from $i$th agent’s perspective to $j$th agent ($\xi_{i\rightarrow i}^{t}$
is the identity), $\mathbf{F}_{j\rightarrow i}^{t}$ is the wrapped BEV feature
transformed from the $j$th agent’s coordinate space to the $i$th agent’s
coordinate space through affine transformation,
$\mathbf{F}_{i}^{{}^{\prime}t}$ is the aggregated feature of the $i$th agent
after fusing other agents’ messages, and $\mathbf{B}_{i}^{t}$ is the detection
outputs after collaboration.

Step (1a) employs the PointPillar framework [27], a lightweight 3D object
detection system, in conjunction with a SORT [5]-inspired tracker, to extract
the BEV feature $\mathbf{F}_{i}^{t}$ from the observation $\mathbf{O}_{i}^{t}$
of the $i$th agent. This step also generates the detected bounding boxes
accompanied by their tracking IDs $\mathbf{D}_{i}^{t}$. The cornerstone of the
process, Step (1b), leverages our innovative $\mathtt{BEVGlue}$ module to
identify co-visible objects and compute the relative pose $\xi_{j\rightarrow
i}^{t}$, drawing upon the detection and tracking results from multiple agents;
see details in the Section 4. Subsequently, Step (1c) aligns the features from
other agents with the ego agent’s pose using the estimated relative poses.
Step (1d) applies a multi-scale max fusion to refine the feature map, denoting
$\mathbf{F}_{i\rightarrow i}^{t}$ as equivalent to $\mathbf{F}_{i}^{t}$. The
final phase, Step (1e), uses fused features to obtain final detection results.
$\mathtt{CoBEVGlue}$ can be applied to multi-agent collaboration: Steps (1b)
and (1c) are executed between the ego agent and each collaborator
independently. Subsequently, in Step (1d), the ego agent integrates features
transformed from all $N$ agents.

Note that accurate relative pose estimation, $\xi_{j\rightarrow i}^{t}$, is
essential for the success of collaborative perception systems. Inaccuracies in
pose information can critically undermine subsequent processes such as feature
transformation, fusion, and collaborative detection. Conventionally, as
delineated in Step (1b), precise pose information necessitates each agent to
utilize an external localization system to acquire its global position and
calculate the relative transformations among collaborators. This dependency on
external localization is fraught with challenges, including susceptibility to
noise interference and potential security breaches through malicious attacks.
Our innovative spatial alignment module, $\mathtt{BEVGlue}$, is designed to
solve these issues by leveraging perceptual data to ensure accurate relative
pose estimation, thereby enhancing the resilience and effectiveness of
collaborative perception. We elaborate this key module in Sec. 4.

##  4 BEVGlue: Spatial Alignment Module

To estimate the relative pose $\xi_{j\rightarrow i}^{t}$ between agents, the
main idea of $\mathtt{BEVGlue}$ is to identify the co-visible objects and
subsequently calculate the transformation based on these co-visible objects.
To excavate this internal correspondence among agents, $\mathtt{BEVGlue}$
presents three modules: (i) object graph modeling, (ii) temporally consistent
maximum common subgraph detection, and (iii) relative pose calculation.

###  4.1 Object Graph Modeling

Object graph modeling is designed to represent the detection and tracking
outcomes of each agent from Step (1a) as an object graph, with each node
corresponding to an object and each edge describing the spatial relationship
between objects. This method of modeling node and edge attributes discovers
the temporal information of each object and the invariant geometric pattern
between objects, which are valuable information for the subsequent common
subgraph searching procedure.

Consider the $i$th agent is tracking $M_{i}$ objects in a scenario. Let
$\mathbf{D}^{t}_{i}=[\mathbf{b}^{t}_{i,1},\mathbf{b}^{t}_{i,2},...,\mathbf{b}^{t}_{i,M_{i}}]\in\mathbb{R}^{M_{i}\times
6}$ be the detection and tracking result. The $m$th bounding box with tracking
ID is
$\mathbf{b}^{t}_{i,m}=[x^{t}_{i,m},y^{t}_{i,m},l^{t}_{i,m},w^{t}_{i,m},\psi^{t}_{i,m},\tau^{t}_{i,m}]$,
encompassing the 2D center position, length, width, yaw angle, and tracking
ID. We formulate $\mathbf{D}^{t}_{i}$ into a fully connected object graph
$\mathcal{G}^{t}_{i}(\mathcal{V}^{t}_{i},\mathcal{E}^{t}_{i})$ for agent $i$,
where $\mathcal{V}^{t}_{i},\mathcal{E}^{t}_{i}$ are the sets of nodes and
edges, respectively. To be noted that each agent has its object graph. The
$m$th node feature $\mathbf{v}^{t}_{i,m}$ is
$[l^{t}_{i,m},w^{t}_{i,m},\tau^{t}_{i,m}]$, and the edge feature is
$\mathbf{e}^{t}_{i,mn}=[\rho^{t}_{i,m\rightarrow n},\theta^{t}_{i,m\rightarrow
n},\psi^{t}_{i,m\rightarrow n}]$ $\rho^{t}_{i,m\rightarrow
n},\theta^{t}_{i,m\rightarrow n}\ \text{and}\ \psi^{t}_{i,m\rightarrow n}$ are
defined within a polar coordinate system which sets the heading of the $m$th
node as the reference direction and its position as the origin (pole). The
$\rho^{t}_{i,m\rightarrow n}$ and $\theta^{t}_{i,m\rightarrow n}$ denote the
radial distance and polar angle of node $n$ respectively and
$\psi^{t}_{i,m\rightarrow n}$ is the intersection angle between the heading of
node $n$ and node $m$. Given that the pole and the reference direction possess
clear and singular definitions in the physical world, consistency in edge
feature computation across different object graphs is achievable.
Specifically, if the detection results of nodes $m$ and $n$ are accurate for
both the $i$th and $j$th agents when calculating the edge feature
$\mathbf{e}^{t}_{j,mn}$ on agent graph $\mathcal{G}_{j}^{t}$ based on agent
$j$, it will be identical to $\mathbf{e}^{t}_{i,mn}$. Also see the Fig. 3.

The object graph presents an innovative approach to model the observation of
each agent: i) the node attribute encompasses temporal tracking data, which
helps keep the matching consistency across time; ii) the edge feature is
consistent across object graphs derived from different agents’ perspectives,
signifying that rotations and translations applied to $\mathbf{D}^{t}_{i}$ do
not alter the value of $\mathbf{e}^{t}_{i,mn}$. It implies that when two
objects are simultaneously observed by different agents, the edge attribute
remains consistent, regardless of the varying perspectives.

Figure 3: Illustration of the proposed object graph modeling. The Ego agent
and its collaborator is collaborating at a T-junction. Three co-visible object
are connected by red line, representing their spatial relationship. The right
portion visualizes the meaning of variables in the edge features.

###  4.2 Temporally Consistent Maximum Common Subgraph Detection

Upon completing the object graph modeling phase, the task shifts to
efficiently detecting the largest common subgraph that strictly satisfies
graph isomorphism between two graphs. This common subgraph is indicative of
the co-visible objects across agents and is subsequently used in calculating
the relative transformation. We leverage two pieces of information for the
search of a common subgraph: i) the spatial relationship: the geometric
pattern of co-visible objects across different agents is isomorphism; ii) the
temporal relationship: the co-visible objects are consistent across time. By
leveraging the spatial relationship, we can ensure there are no outliers in
the matched nodes. By leveraging the temporal relationship, we can ensure the
temporal consistency of matching results across time.

Consider a pair of modeled object pose graphs
$\mathcal{G}_{i}^{t}=\\{\mathcal{V}_{i}^{t},\mathcal{E}_{i}^{t}\\}$ with $M$
nodes and $\mathcal{G}_{j}^{t}=\\{\mathcal{V}_{j}^{t},\mathcal{E}_{j}^{t}\\}$
with $N$ nodes at time $t$, finding the maximum common subgraph
$\mathcal{H}_{(i,j)}^{t}$ can be formulated as

$\mathcal{H}_{(i,j)}^{t}=f_{\text{MCS}}\left(\mathcal{G}_{i}^{t},\mathcal{G}_{j}^{t},\mathcal{H}_{(i,j)}^{t-1}\right).$
(2)

To realize $f_{\text{MCS}}\left(\cdot\right)$, the procedure is divided into
three primary steps:

i). Candidate initialization. This step is designed to generate a set of
candidate common subgraphs, represented as a list
$\mathcal{S}^{t}=[s_{0}^{t},s_{1}^{t},...,s_{K}^{t}]$, where $K$ denotes the
number of these candidates. At the initial timestep($t=0$), we explore all
$M\times N$ potential node pairs. For each pair, we assess whether their node
affinity surpasses a predefined threshold $\gamma_{\mathcal{V}}$. The node
affinity for any given node pair $(m,n)$ at timestep $t$ is determined by the
equation
$\kappa_{mn}^{\mathcal{V}^{t},(i,j)}=\phi_{\mathcal{V}}(\mathbf{v}^{t}_{i,m},\mathbf{v}^{t}_{j,n})$
where the affinity function $\phi_{\mathcal{V}}\left(\cdot\right)$ measure the
similarity between node $m$ in agent graph $\mathcal{G}_{i}^{t}$ and node $n$
in agent graph $\mathcal{G}_{j}^{t}$. Pairs that meet this criterion are
subsequently incorporated into $\mathcal{S}^{t}$ as candidates for common
subgraphs. At subsequent timesteps($t\geq 1$), the node pairs from previous
common subgraph $\mathcal{H}_{(i,j)}^{t-1}$ are considered as candidates for
the current common subgraphs. The temporal correspondence between
$\mathcal{S}^{t}$ and $\mathcal{H}_{(i,j)}^{t-1}$ is established by the
tracking IDs.

ii). Subgraph expanding. For each candidate $s_{k}^{t}$, we consider the node
pair within it as $(v^{t}_{i,p},v^{t}_{j,q})$. We then identify all potential
matching node pairs $(v^{t}_{i,m},v^{t}_{j,n})$ using node affinity
$\kappa_{mn}^{\mathcal{V}^{t},(i,j)}=\phi_{\mathcal{V}}(\mathbf{v}^{t}_{i,m},\mathbf{v}^{t}_{j,n})$
and edge affinity
$\kappa_{(p,m),(q,n)}^{\mathcal{E}^{t},(i,j)}=\phi_{\mathcal{E}}(\mathbf{e}^{t}_{i,pm},\mathbf{e}^{t}_{j,qn})$.
The affinity functions for node is the same as i) and the affinity functions
for edges $\phi_{\mathcal{E}}\left(\cdot\right)$ measure the similarity
between edge pairs, considering the difference of relative position and
heading difference; see the details in Appendix. A pair
$(v^{t}_{i,m},v^{t}_{j,n})$ will be added to $s_{k}^{t}$ if
$\kappa_{mn}^{\mathcal{V}^{t},(i,j)}>\gamma_{\mathcal{V}}$ and
$\kappa_{(p,m),(q,n)}^{\mathcal{E}^{t},(i,j)}>\gamma_{\mathcal{E}}$ where
$\gamma_{\mathcal{V}}$ and $\gamma_{\mathcal{E}}$ are predefined thresholds.

iii). Maximum common subgraph selection. After ii), our algorithm needs to
determine the most suitable common subgraph from all candidates in
$\mathcal{S}^{t}$. The initial criterion is to select the common subgraphs
with the highest node count. If multiple candidates share this characteristic,
we then compute a confidence score for those candidates. The confidence score
$c_{k}^{t}$ of the candidate
$s_{k}^{t}=\\{(v^{t}_{i,p_{1}},v^{t}_{j,q_{1}}),(v^{t}_{i,p_{2}},v^{t}_{j,q_{2}}),...,(v^{t}_{i,p_{C}},v^{t}_{j,q_{C}})\\}$
is calculated using Eq. 3.

$c_{k}^{t}=\sum_{\alpha=1}^{C}\kappa_{p_{\alpha}q_{\alpha}}^{\mathcal{V}^{t},(i,j)}+\sum_{\alpha=1}^{C}\sum_{\beta=1}^{C}\kappa_{(p_{\alpha},p_{\beta}),(q_{\alpha},q_{\beta})}^{\mathcal{E}^{t},(i,j)},$
(3)

where $C$ is the node count of $s_{k}^{t}$. Among the candidates that have the
largest number of nodes, the one with the highest confidence score is selected
as the final common subgraph.

The proposed MCS detection algorithm brings two benefits: i) the subgraph
expanding step maintains strict spatial constraints between matched objects,
saving the time for outlier detection in the following transformation
calculation step. ii) the temporal tracking information promotes the
consistency of detected MCS across time, improving the robustness of the
proposed algorithm; see the algorithm diagram in appendix.

###  4.3 Relative Pose Calculation

Given the common subgraph $\mathcal{G}_{s,(i,j)}^{t}$, we calculate the
relative pose $\xi_{j\rightarrow i}^{t}$ by considering each matched node as a
point and estimate the rigid transformation between two point sets. Let
$\mathcal{P}=[v^{t}_{i,p_{1}},v^{t}_{i,p_{2}},...v^{t}_{i,p_{C}}],\mathcal{Q}=[v^{t}_{j,q_{1}},v^{t}_{j,q_{2}},...v^{t}_{j,q_{C}}]$
be two matching node sets and
$\mathbf{P}=[\mathbf{p}_{1},\mathbf{p}_{2},...,\mathbf{p}_{C}]$ and
$\mathbf{Q}=[\mathbf{q}_{1},\mathbf{q}_{2},...,\mathbf{q}_{C}]$ be the
corresponding 2D position of the nodes in their corresponding BEV coordinate
system. The transformation between two coordinate systems $\mathbf{R}\in
SO(2),t\in\mathbb{R}^{2}$ can be calculated by solving the following
Procrustes problem[20]:

$\mathbf{R}^{*},\mathbf{t}^{*}=\mathop{\arg\min}_{\mathbf{R},\mathbf{t}}\sum_{i=1}^{C}\left\|\mathbf{R}\mathbf{p}_{i}+\mathbf{t}-\mathbf{q}_{i}\right\|^{2}.$
(4)

The optimal solution for the transformation can be efficiently derived using
Singular Value Decomposition (SVD) in scenarios where there are no erroneous
matching pairs. However, in cases where erroneous correspondences are present,
the least variance nature of optimizing Eq. 4 necessitates the use of other
more time-intensive methods capable of outlier elimination, such as RANSAC
[16]. Thanks to the accuracy of our temporally consistent maximum common
subgraph detection, we can employ the faster SVD method for computing the
transformation. The final relative pose $\xi_{j\rightarrow i}^{t}$ can be
obtained from $\mathbf{R},t$.

###  4.4 Discussions

Advantages. $\mathtt{BEVGlue}$ has following distinct advantages. Comparing to
point cloud registration methods[49, 63, 70]:

  * •

It only requires the transmission of bounding boxes and tracking IDs, while
point cloud registration methods require the transmission of massive points
with their feature vector, costing too much communication bandwidth;

  * •

It ensures high-quality matching with no outliers by checking the spatial
relationship between matched nodes, while point cloud registration methods
only match by comparing local point features;

Comparing to alignment modules in previous collaborative perception system
methods [53, 69, 61, 36] :

  * •

It is capable of building a consistent coordinate system without using any
localization results, while previous methods can only rectify minor
inaccuracies in external localization systems since they rely on a basically
precise initial relative pose.

  * •

It incorporates tracking results to enhance the temporal consistency of
matching, while previous methods ignore the temporal information.

Prerequisites. There are two assumptions for $\mathtt{BEVGlue}$ to function
well. i) Collaboration is initiated only when agents are in close proximity,
ensuring a common field of view. ii) Several objects exist in the common field
of view and are perceived by both agents. These assumptions are realistically
met in a collaborative perception context for two primary reasons. Firstly,
the pivotal wireless connections that facilitate collaboration are inherently
range-bound, thus naturally restricting collaborative interactions to agents
in close proximity. Secondly, typical traffic scenarios inherently offer an
environment rich with various objects and BEV alignment can be successfully
achieved with the presence of only two co-visible devices. Real-world data
from collaborative perception datasets corroborate this: 91.3% of the samples
in DAIR-V2X[68] and 94.2% of the samples in V2V4Real[60] satisfy these
criteria.

##  5 Experimental Results

Table 1: Detection performance with localization noises following Gaussian
distribution in the testing phases.

Dataset | OPV2V | DAIR-V2X | V2V4Real
---|---|---|---
Method/Metric |<EMAIL_ADDRESS>$\uparrow$
Noise Level $\sigma_{t}/\sigma_{r}(m/^{\circ})$ | 0.0/0.0 | 0.5/0.5 | 1.5/1.5 | 2.5/2.5 | 0.0/0.0 | 0.5/0.5 | 1.5/1.5 | 2.5/2.5 | 0.0/0.0 | 0.5/0.5 | 1.5/1.5 | 2.5/2.5
w/o collaboration | No Collaboration | 0.786 | 0.645 | 0.447
| F-Cooper[11]SEC’19 | 0.834 | 0.638 | 0.458 | 0.399 | 0.737 | 0.697 | 0.660 | 0.636 | 0.693 | 0.481 | 0.330 | 0.309
| V2VNet[56]ECCV’20 | 0.936 | 0.861 | 0.724 | 0.691 | 0.665 | 0.610 | 0.551 | 0.526 | 0.580 | 0.441 | 0.338 | 0.312
| DiscoNet[30]NeurIPS’21 | 0.916 | 0.874 | 0.788 | 0.753 | 0.737 | 0.704 | 0.674 | 0.666 | 0.736 | 0.527 | 0.411 | 0.378
w/o robust design | Where2comm[21]NeurIPS’22 | 0.944 | 0.721 | 0.500 | 0.505 | 0.752 | 0.637 | 0.580 | 0.570 | 0.704 | 0.505 | 0.384 | 0.364
| FPV-RCNN[69]RAL’22 | 0.858 | 0.476 | 0.236 | 0.225 | 0.626 | 0.512 | 0.427 | 0.422 | 0.701 | 0.387 | 0.244 | 0.237
| V2VNet${}_{\text{robust}}$[53]CoRL’20 | 0.942 | 0.919 | 0.865 | 0.831 | 0.661 | 0.639 | 0.614 | 0.594 | 0.550 | 0.525 | 0.467 | 0.435
| V2X-ViT[61]ECCV’22 | 0.946 | 0.925 | 0.796 | 0.632 | 0.705 | 0.682 | 0.647 | 0.632 | 0.680 | 0.673 | 0.450 | 0.422
w/ robust design | CoAlign[36]ICRA’23 | 0.966 | 0.950 | 0.863 | 0.824 | 0.746 | 0.712 | 0.665 | 0.647 | 0.709 | 0.613 | 0.435 | 0.387
Self-Localized | CoBEVGlue | 0.958 | 0.740 | 0.702
Method/Metric |<EMAIL_ADDRESS>$\uparrow$
Noise Level $\sigma_{t}/\sigma_{r}(m/^{\circ})$ | 0.0/0.0 | 0.5/0.5 | 1.5/1.5 | 2.5/2.5 | 0.0/0.0 | 0.5/0.5 | 1.5/1.5 | 2.5/2.5 | 0.0/0.0 | 0.5/0.5 | 1.5/1.5 | 2.5/2.5
w/o collaboration | No Collaboration | 0.690 | 0.526 | 0.261
| F-Cooper[11]SEC’19 | 0.603 | 0.388 | 0.328 | 0.298 | 0.560 | 0.542 | 0.516 | 0.487 | 0.432 | 0.212 | 0.179 | 0.174
| V2VNet[56]ECCV’20 | 0.740 | 0.534 | 0.384 | 0.315 | 0.402 | 0.362 | 0.320 | 0.316 | 0.250 | 0.163 | 0.135 | 0.130
| DiscoNet[30]NeurIPS’21 | 0.791 | 0.741 | 0.684 | 0.655 | 0.584 | 0.568 | 0.561 | 0.557 | 0.466 | 0.296 | 0.271 | 0.357
w/o robust design | Where2comm[21]NeurIPS’22 | 0.855 | 0.469 | 0.355 | 0.286 | 0.588 | 0.473 | 0.454 | 0.451 | 0.469 | 0.263 | 0.226 | 0.220
| FPV-RCNN[69]RAL’22 | 0.840 | 0.214 | 0.173 | 0.189 | 0.409 | 0.319 | 0.325 | 0.340 | 0.479 | 0.153 | 0.156 | 0.165
| V2VNet${}_{\text{robust}}$[53]CoRL’20 | 0.854 | 0.826 | 0.773 | 0.742 | 0.486 | 0.472 | 0.447 | 0.449 | 0.309 | 0.296 | 0.279 | 0.272
| V2X-ViT[61]ECCV’22 | 0.856 | 0.834 | 0.721 | 0.502 | 0.531 | 0.523 | 0.510 | 0.502 | 0.391 | 0.305 | 0.272 | 0.262
w/ robust design | CoAlign[36]ICRA’23 | 0.912 | 0.878 | 0.771 | 0.732 | 0.604 | 0.575 | 0.558 | 0.548 | 0.417 | 0.336 | 0.261 | 0.239
Self-Localized | CoBEVGlue | 0.909 | 0.582 | 0.431

Table 2: With $\mathtt{BEVGlue}$, a majority of representative collaborative
perception systems significantly improve their robustness to localization
attack, bringing negligible communication overhead. $\delta b$ represents the
extra communication overhead required by $\mathtt{BEVGlue}$.

Dataset | OPV2V | DAIR-V2X
---|---|---
Method/Metric |<EMAIL_ADDRESS>$\uparrow$ |<EMAIL_ADDRESS>$\uparrow$ | $\delta b\downarrow$ |<EMAIL_ADDRESS>$\uparrow$ |<EMAIL_ADDRESS>$\uparrow$ | $\delta b\downarrow$
without / with BEVGlue
F-Cooper[11]SEC’19 | 0.307/0.841 $\uparrow$174% | 0.224/0.605 $\uparrow$170% |  $\uparrow$ 0.000360% | 0.563/0.699 $\uparrow$24% | 0.410/0.540 $\uparrow$32% |  $\uparrow$ 0.000300%
V2VNet[56]ECCV’20 | 0.636/0.929 $\uparrow$46% | 0.375/0.731 $\uparrow$95% |  $\uparrow$ 0.00144% | 0.393/0.616 $\uparrow$57% | 0.247/0.374 $\uparrow$51% |  $\uparrow$ 0.00168%
DiscoNet[30]NeurIPS’21 | 0.671/0.917 $\uparrow$37% | 0.654/0.789 $\uparrow$21% |  $\uparrow$ 0.000360% | 0.635/0.706 $\uparrow$11% | 0.540/0.569 $\uparrow$5.4% |  $\uparrow$ 0.000420%
Where2comm[21]NeurIPS’22 | 0.272/0.937 $\uparrow$244% | 0.203/0.826 $\uparrow$307% |  $\uparrow$ 0.000363% | 0.493/0.669 $\uparrow$36% | 0.398/0.508 $\uparrow$28% |  $\uparrow$ 0.000418%
V2VNet${}_{\text{robust}}$[53]CoRL’20 | 0.790/0.927 $\uparrow$17% | 0.708/0.837 $\uparrow$18% |  $\uparrow$ 0.00144% | 0.513/0.684 $\uparrow$33% | 0.401/0.525 $\uparrow$31% |  $\uparrow$ 0.00168%
V2X-ViT[61]ECCV’22 | 0.696/0.942 $\uparrow$35% | 0.638/0.852 $\uparrow$34% |  $\uparrow$ 0.00150% | 0.581/0.684 $\uparrow$18% | 0.474/0.525 $\uparrow$11% |  $\uparrow$ 0.00172%
CoAlign[36]ICRA’23 | 0.791/0.962 $\uparrow$22% | 0.699/0.907 $\uparrow$30% |  $\uparrow$ 0.000829% | 0.601/0.713 $\uparrow$19% | 0.522/0.578 $\uparrow$11% |  $\uparrow$ 0.000959%

Figure 4: Comparison of collaborative perception with our $\mathtt{BEVGlue}$
and using point cloud registration. $\mathtt{BEVGlue}$ shows significant
advantages over baseline methods in both detection and communication
efficiency, operating without the need for an initial pose. Initial poses
provided to ICP are varied under different Gaussian noise. We sample feature
points to conform within the predefined communication volume constraint.

(a) V2X-ViT

(b) CoAlign

(c) CoBEVGlue

Figure 5: _CoBEVGlue_ qualitatively outperforms V2X-ViT [61] and CoAlign [36]
on OPV2V dataset under localization noisy setting. Green and red boxes denote
ground-truth and detection, respectively.

(a) Point cloud registration

(b) BEVGlue

Figure 6: Visualization of calculating relative pose between two collaborators
on OPV2V[62] dataset. Yellow and blue denote the point clouds collected from
ego vehicle and the collaborator, respectively. In (a), the massive points
transmitted between agents are depicted in red and white, with white
specifically indicating the successfully matched points. In (b), red and white
represent the boxes transmitted among agents, while white highlights the co-
visible objects and their spatial relationships. $\mathtt{BEVGlue}$ achieves
accurate relative pose estimation with low communication cost by only
transmitting object-wise information. Point cloud registration method takes
over 2000 times communication volume than our $\mathtt{BEVGlue}$, making it
unusable for bandwidth limited collaborative perception systems.

###  5.1 Datasets and Experimental Settings

We conduct collaborative LiDAR-based 3D object detection on both a simulation
dataset, OPV2V [62], co-simulated by OpenCDA [58] and Carla [14], and two
real-world dataset, DAIR-V2X [68] and V2V4Real[60]. We follow [62, 36, 60] to
set the detection range as $x\in[-140m,140m],y\in[-40m,40m]$ in OPV2V and
V2V4Real and $x\in[-100m,100m],y\in[-40m,40m]$ in DAIR-V2X respectively. We
use PointPillars [27] with the grid size $(0.4m,0.4m)$ as the encoder. For
multi-scale feature fusion, the residual layer number is 3 and the channel
numbers are $(64,128,256)$. The communication results count the message size
by byte in log scale with base $2$.

###  5.2 Quantitative Evaluation

Detection performance in the presence of localization noise. Table 1 compares
the proposed $\mathtt{CoBEVGlue}$ with previous methods under localization
noise on OPV2V, DAIR-V2X and V2V4Real. For the setting of localization noise,
we apply Gaussian noise $\mathcal{N}(0,\sigma_{t})$ on $x,y$ and
$\mathcal{N}(0,\sigma_{r})$ on $\theta$, where $x,y,\theta$ are the 2D centers
and yaw angle of each agent’s accurate 3DoF pose. This noise setting follows
previous work [56, 61, 21, 36], while we increase the range of standard
deviation to cover more challenging settings. The baseline methods include no
collaboration, collaborative methods without specific design to localization
noise, and collaborative methods including robust design for localization
error. We see that the performances of $\mathtt{CoBEVGlue}$ are not affected
by the level of pose noise since it operates independently of localization
systems, significantly outperforming previous methods in various noise levels
across both datasets.

Detection performance in the presence of localization attack. We also explore
detection performance under the common and unsolved GPS Spoofing attack [71,
6], an attack where malicious attackers set arbitrary position by sending fake
satellite signals. Specifically, we consider the attacker to deceive all
collaborators into thinking they are in the same location, aiming to generate
false positive bounding boxes. Table 2 presents the detection performance and
communication bandwidth of various methods under this attack, along with their
performance when integrated with our spatial alignment module
$\mathtt{BEVGlue}$. The results reveal that $\mathtt{BEVGlue}$ significantly
improves performance under attack while bringing negligible communication
overhead. Notably, with $\mathtt{BEVGlue}$’s assistance, a majority of
collaborative perception methods outperform single-agent perception even under
malicious attack.

Comparisons with point cloud registration. We present a comparison between
$\mathtt{BEVGlue}$ and two representative point cloud registration methods in
Fig. 4. This comparison includes both 2D and 3D versions of the Iterative
Closest Point (ICP) and a widely used pipeline in recent multi-agent systems
[26], which incorporates Fast Point Feature Histograms (FPFH) [46] for
keypoint description and TEASER++ [63] for robust registration. Initial poses
provided to ICP are varied under different levels of Gaussian noise. We see
that i) $\mathtt{CoBEVGlue}$ consistently delivers superior performance with
minimal communication volume, and its effectiveness remains stable regardless
of the extent of localization noise; ii) while ICP can mitigate the impact of
minor localization noise, it consumes large communication bandwidth and its
performance significantly declines under bandwidth constraints; iii) as
localization noise increases, ICP fails to achieve successful alignment,
irrespective of the available bandwidth.

Computation time. Tested on a system equipped with a 2.90GHz Intel Xeon CPU
and an RTX 4090 GPU, $\mathtt{BEVGlue}$ achieves 89.98 frames per second (FPS)
on OPV2V, 72.18 FPS on DAIR-V2X and 158.7 FPS on V2V4Real.

###  5.3 Qualitative Evaluation

Visualization of detection results. Fig. 5 shows a comparative visualization
of detection results from V2X-ViT, CoAlign, and $\mathtt{CoBEVGlue}$ in the
OPV2V dataset under noisy setting. The noise stems from a Gaussian
distribution with a standard deviation of 3.0m for position and 3.0° for
heading. V2X-ViT, despite employing the MSWin module to mitigate pose error,
struggles under large noise. Similarly, the pose graph optimization algorithm
in CoAlign fails in the presence of large noise, leading to a severe drop in
detection performance. In contrast, $\mathtt{CoBEVGlue}$’s exhibits superior
performance under large noise. This can be attributed to its independence from
prior pose information, which makes it less susceptible to the impacts of pose
noise.

Visual comparison with point cloud registration. Fig. 6 compares the
collaborative perception system with point cloud registration and the one with
$\mathtt{BEVGlue}$. For point cloud registration, we use FPFH [46] for feature
descriptor and TEASER [63] for registration. Fig. 6(a) shows that the point
cloud registration pipeline identifies a few matched points (in white) from a
large amount of transmitted points; and Fig. 6(b) shows that
$\mathtt{BEVGlue}$ uses the spatial geometry of objects to find
correspondence. Compared with the point cloud registration pipeline,
$\mathtt{BEVGlue}$ uses much less communication volume ($2000\times$).

###  5.4 Ablation studies

Table 3: Ablation study.

Dataset | OPV2V | V2V4Real
---|---|---
Matching Criteria | Metric
Geometric pattern | Tracking |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
|  | 0.7904 | 0.7781 | 0.6746 | 0.4919 | 0.4469 | 0.2611
✓ |  | 0.9614 | 0.9545 | 0.9049 | 0.6839 | 0.6356 | 0.3787
✓ | ✓ | 0.9651 | 0.9581 | 0.9088 | 0.7394 | 0.7016 | 0.4308

Table 3 assesses the effectiveness of the geometric pattern and tracking
information used in maximum common subgraph detection on the OPV2V and
V2V4Real dataset. Absent the geometric pattern, the matching process only
relies on node information, ignoring the spatial relationship between nodes.
Without the tracking information, the search ignores temporal consistency. We
see that: 1) Geometric patterns play a crucial role in enhancing our method
for common subgraph detection. Specifically, on the V2V4Real dataset,
integrating geometric patterns results in a performance boost of 45.7% over
the baseline; and 2) The inclusion of tracking information significantly
contributes to the temporal coherence of the matching outcomes, with an
improvement of 57.0% on the V2V4Real dataset compared to the baseline
approach.

##  6 Conclusions

This paper proposes a novel self-localized collaborative perception framework
$\mathtt{CoBEVGlue}$ and a novel spatial alignment method $\mathtt{BEVGlue}$.
The core idea is to search for co-visible objects from the bird’s eye view
perceptual data across agents and calculate the relative pose between agents,
ensuring a consistent spatial coordinate system for collaboration.
Comprehensive experiments show that $\mathtt{CoBEVGlue}$ performs comparably
to systems relying on precise localization information and achieves state-of-
the-art detection performance when localization noise and attack exist.

Limitation and future work. This work focuses on exploiting spatial alignment
for collaborative perception. We plan to mitigate the temporal misalignment
issue in future.

## References

  * [1] Ao, S., Hu, Q., Yang, B., Markham, A., Guo, Y.: Spinnet: Learning a general surface descriptor for 3d point cloud registration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11753–11762 (2021)
  * [2] Arnold, E., Dianati, M., de Temple, R., Fallah, S.: Cooperative perception for 3d object detection in driving scenarios using infrastructure sensors. IEEE Transactions on Intelligent Transportation Systems 23(3), 1852–1864 (2020)
  * [3] Bai, Y., Xu, D., Sun, Y., Wang, W.: Glsearch: Maximum common subgraph detection via learning to search. In: International Conference on Machine Learning. pp. 588–598. PMLR (2021)
  * [4] Besl, P.J., McKay, N.D.: Method for registration of 3-d shapes. In: Sensor fusion IV: control paradigms and data structures. vol. 1611, pp. 586–606. Spie (1992)
  * [5] Bewley, A., Ge, Z., Ott, L., Ramos, F., Upcroft, B.: Simple online and realtime tracking. In: 2016 IEEE international conference on image processing (ICIP). pp. 3464–3468. IEEE (2016)
  * [6] Bhatti, J., Humphreys, T.E.: Hostile control of ships via false gps signals: Demonstration and detection. NAVIGATION: Journal of the Institute of Navigation 64(1), 51–66 (2017)
  * [7] Bouaziz, S., Tagliasacchi, A., Pauly, M.: Sparse iterative closest point. In: Computer graphics forum. vol. 32, pp. 113–123. Wiley Online Library (2013)
  * [8] Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., Beijbom, O.: nuscenes: A multimodal dataset for autonomous driving. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11621–11631 (2020)
  * [9] Chang, Y., Ebadi, K., Denniston, C.E., Ginting, M.F., Rosinol, A., Reinke, A., Palieri, M., Shi, J., Chatterjee, A., Morrell, B., et al.: Lamp 2.0: A robust multi-robot slam system for operation in challenging large-scale underground environments. IEEE Robotics and Automation Letters 7(4), 9175–9182 (2022)
  * [10] Chang, Y., Hughes, N., Ray, A., Carlone, L.: Hydra-multi: Collaborative online construction of 3d scene graphs with multi-robot teams. arXiv preprint arXiv:2304.13487 (2023)
  * [11] Chen, Q.: F-cooper: Feature based cooperative perception for autonomous vehicle edge computing system using 3d point clouds (2019)
  * [12] Choy, C., Park, J., Koltun, V.: Fully convolutional geometric features. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 8958–8966 (2019)
  * [13] Combier, C., Damiand, G., Solnon, C.: Map edit distance vs. graph edit distance for matching images. In: Graph-Based Representations in Pattern Recognition: 9th IAPR-TC-15 International Workshop, GbRPR 2013, Vienna, Austria, May 15-17, 2013. Proceedings 9. pp. 152–161. Springer (2013)
  * [14] Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: Carla: An open urban driving simulator. In: Conference on robot learning. pp. 1–16. PMLR (2017)
  * [15] Ehrlich, H.C., Rarey, M.: Maximum common subgraph isomorphism algorithms and their applications in molecular science: a review. Wiley Interdisciplinary Reviews: Computational Molecular Science 1(1), 68–79 (2011)
  * [16] Fischler, M.A., Bolles, R.C.: Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981). https://doi.org/10.1145/358669.358692, https://doi.org/10.1145/358669.358692
  * [17] Fitzgibbon, A.W.: Robust registration of 2d and 3d point sets. Image and vision computing 21(13-14), 1145–1153 (2003)
  * [18] Gay, S., Fages, F., Martinez, T., Soliman, S., Solnon, C.: On the subgraph epimorphism problem. Discrete Applied Mathematics 162, 214–228 (2014)
  * [19] Gelfand, N., Mitra, N.J., Guibas, L.J., Pottmann, H.: Robust global registration. In: Symposium on geometry processing. vol. 2, p. 5. Vienna, Austria (2005)
  * [20] Gower, J.C., Dijksterhuis, G.B.: Procrustes problems, vol. 30. OUP Oxford (2004)
  * [21] Hu, Y., Fang, S., Lei, Z., Zhong, Y., Chen, S.: Where2comm: Communication-efficient collaborative perception via spatial confidence maps. Advances in neural information processing systems 35, 4874–4886 (2022)
  * [22] Huang, S., Gojcic, Z., Usvyatsov, M., Wieser, A., Schindler, K.: Predator: Registration of 3d point clouds with low overlap. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. pp. 4267–4276 (2021)
  * [23] Huang, Y., Shan, T., Chen, F., Englot, B.: Disco-slam: Distributed scan context-enabled multi-robot lidar slam with two-stage global-local graph optimization. IEEE Robotics and Automation Letters 7(2), 1150–1157 (2021)
  * [24] Ikram, M.H., Khaliq, S., Anjum, M.L., Hussain, W.: Perceptual aliasing++: Adversarial attack for visual slam front-end and back-end. IEEE Robotics and Automation Letters 7(2), 4670–4677 (2022)
  * [25] Johnson, A.E., Hebert, M.: Using spin images for efficient object recognition in cluttered 3d scenes. IEEE Transactions on pattern analysis and machine intelligence 21(5), 433–449 (1999)
  * [26] Lajoie, P.Y., Beltrame, G.: Swarm-slam: Sparse decentralized collaborative simultaneous localization and mapping framework for multi-robot systems. arXiv preprint arXiv:2301.06230 (2023)
  * [27] Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang, J., Beijbom, O.: Pointpillars: Fast encoders for object detection from point clouds. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 12697–12705 (2019)
  * [28] Lei, Z., Ren, S., Hu, Y., Zhang, W., Chen, S.: Latency-aware collaborative perception. In: European Conference on Computer Vision. pp. 316–332. Springer (2022)
  * [29] Li, Y., Ma, D., An, Z., Wang, Z., Zhong, Y., Chen, S., Feng, C.: V2x-sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving. IEEE Robotics and Automation Letters 7(4), 10914–10921 (2022)
  * [30] Li, Y., Ren, S., Wu, P., Chen, S., Feng, C., Zhang, W.: Learning distilled collaboration graph for multi-agent perception. Advances in Neural Information Processing Systems 34, 29541–29552 (2021)
  * [31] Liu, Q., Zhu, H., Zhou, Y., Li, H., Chang, S., Guo, M.: Density-invariant features for distant point cloud registration. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 18215–18225 (2023)
  * [32] Liu, S., Gao, C., Chen, Y., Peng, X., Kong, X., Wang, K., Xu, R., Jiang, W., Xiang, H., Ma, J., et al.: Towards vehicle-to-everything autonomous driving: A survey on collaborative perception. arXiv preprint arXiv:2308.16714 (2023)
  * [33] Liu, Y.l., Li, C.m., Jiang, H., He, K.: A learning based branch and bound for maximum common subgraph problems. arXiv preprint arXiv:1905.05840 (2019)
  * [34] Liu, Y.C., Tian, J., Glaser, N., Kira, Z.: When2com: Multi-agent perception via communication graph grouping. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. pp. 4106–4115 (2020)
  * [35] Liu, Y.C., Tian, J., Ma, C.Y., Glaser, N., Kuo, C.W., Kira, Z.: Who2com: Collaborative perception via learnable handshake communication. In: 2020 IEEE International Conference on Robotics and Automation (ICRA). pp. 6876–6883. IEEE (2020)
  * [36] Lu, Y., Li, Q., Liu, B., Dianati, M., Feng, C., Chen, S., Wang, Y.: Robust collaborative 3d object detection in presence of pose errors. In: 2023 IEEE International Conference on Robotics and Automation (ICRA). pp. 4812–4818. IEEE (2023)
  * [37] McCreesh, C., Ndiaye, S.N., Prosser, P., Solnon, C.: Clique and constraint models for maximum common (connected) subgraph problems. In: International Conference on Principles and Practice of Constraint Programming. pp. 350–368. Springer (2016)
  * [38] McCreesh, C., Prosser, P., Trimble, J.: A partitioning algorithm for maximum common subgraph problems (2017)
  * [39] McGregor, J.J.: Backtrack search algorithms and the maximal common subgraph problem. Software: Practice and Experience 12(1), 23–34 (1982)
  * [40] Meng, Z., Xia, X., Xu, R., Liu, W., Ma, J.: Hydro-3d: Hybrid object detection and tracking for cooperative perception using 3d lidar. IEEE Transactions on Intelligent Vehicles (2023)
  * [41] Narain, S., Ranganathan, A., Noubir, G.: Security of gps/ins based on-road location tracking systems. In: 2019 IEEE Symposium on Security and Privacy (SP). pp. 587–601. IEEE (2019)
  * [42] Noh, J., Kwon, Y., Son, Y., Shin, H., Kim, D., Choi, J., Kim, Y.: Tractor beam: Safe-hijacking of consumer drones with adaptive gps spoofing. ACM Transactions on Privacy and Security (TOPS) 22(2), 1–26 (2019)
  * [43] Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 652–660 (2017)
  * [44] Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems 30 (2017)
  * [45] Raymond, J.W., Willett, P.: Maximum common subgraph isomorphism algorithms for the matching of chemical structures. Journal of computer-aided molecular design 16, 521–533 (2002)
  * [46] Rusu, R.B., Blodow, N., Beetz, M.: Fast point feature histograms (fpfh) for 3d registration. In: 2009 IEEE international conference on robotics and automation. pp. 3212–3217. IEEE (2009)
  * [47] Rusu, R.B., Blodow, N., Marton, Z.C., Beetz, M.: Aligning point cloud views using persistent feature histograms. In: 2008 IEEE/RSJ international conference on intelligent robots and systems. pp. 3384–3391. IEEE (2008)
  * [48] Sarkar, S.D., Miksik, O., Pollefeys, M., Barath, D., Armeni, I.: Sgaligner: 3d scene alignment with scene graphs. arXiv preprint arXiv:2304.14880 (2023)
  * [49] Sharp, G.C., Lee, S.W., Wehe, D.K.: Icp registration using invariant features. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(1), 90–102 (2002)
  * [50] Shen, J., Won, J.Y., Chen, Z., Chen, Q.A.: Drift with devil: Security of $\\{$Multi-Sensor$\\}$ fusion based localization in $\\{$High-Level$\\}$ autonomous driving under $\\{$GPS$\\}$ spoofing. In: 29th USENIX Security Symposium (USENIX Security 20). pp. 931–948 (2020)
  * [51] Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., Caine, B., et al.: Scalability in perception for autonomous driving: Waymo open dataset. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2446–2454 (2020)
  * [52] Tombari, F., Salti, S., Di Stefano, L.: Unique shape context for 3d data description. In: Proceedings of the ACM workshop on 3D object retrieval. pp. 57–62 (2010)
  * [53] Vadivelu, N., Ren, M., Tu, J., Wang, J., Urtasun, R.: Learning to communicate and correct pose errors. In: Conference on Robot Learning. pp. 1195–1210. PMLR (2021)
  * [54] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [55] Vismara, P., Valery, B.: Finding maximum common connected subgraphs using clique detection or constraint satisfaction algorithms. In: International Conference on Modelling, Computation and Optimization in Information Systems and Management Sciences. pp. 358–368. Springer (2008)
  * [56] Wang, T.H., Manivasagam, S., Liang, M., Yang, B., Zeng, W., Urtasun, R.: V2vnet: Vehicle-to-vehicle communication for joint perception and prediction. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. pp. 605–621. Springer (2020)
  * [57] Xiang, H., Xu, R., Ma, J.: Hm-vit: Hetero-modal vehicle-to-vehicle cooperative perception with vision transformer. arXiv preprint arXiv:2304.10628 (2023)
  * [58] Xu, R., Guo, Y., Han, X., Xia, X., Xiang, H., Ma, J.: Opencda: an open cooperative driving automation framework integrated with co-simulation. In: 2021 IEEE International Intelligent Transportation Systems Conference (ITSC). pp. 1155–1162. IEEE (2021)
  * [59] Xu, R., Tu, Z., Xiang, H., Shao, W., Zhou, B., Ma, J.: Cobevt: Cooperative bird’s eye view semantic segmentation with sparse transformers. arXiv preprint arXiv:2207.02202 (2022)
  * [60] Xu, R., Xia, X., Li, J., Li, H., Zhang, S., Tu, Z., Meng, Z., Xiang, H., Dong, X., Song, R., et al.: V2v4real: A real-world large-scale dataset for vehicle-to-vehicle cooperative perception. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13712–13722 (2023)
  * [61] Xu, R., Xiang, H., Tu, Z., Xia, X., Yang, M.H., Ma, J.: V2x-vit: Vehicle-to-everything cooperative perception with vision transformer. In: European conference on computer vision. pp. 107–124. Springer (2022)
  * [62] Xu, R., Xiang, H., Xia, X., Han, X., Li, J., Ma, J.: Opv2v: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication. In: 2022 International Conference on Robotics and Automation (ICRA). pp. 2583–2589. IEEE (2022)
  * [63] Yang, H., Shi, J., Carlone, L.: TEASER: Fast and Certifiable Point Cloud Registration. IEEE Trans. Robotics (2020)
  * [64] Yew, Z.J., Lee, G.H.: Rpm-net: Robust point matching using learned features. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11824–11833 (2020)
  * [65] Yew, Z.J., Lee, G.H.: Regtr: End-to-end point cloud correspondences with transformers. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 6677–6686 (2022)
  * [66] Yin, T., Zhou, X., Krahenbuhl, P.: Center-based 3d object detection and tracking. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11784–11793 (2021)
  * [67] Yoshida, K., Hojo, M., Fujino, T.: Adversarial scan attack against scan matching algorithm for pose estimation in lidar-based slam. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences 105(3), 326–335 (2022)
  * [68] Yu, H., Luo, Y., Shu, M., Huo, Y., Yang, Z., Shi, Y., Guo, Z., Li, H., Hu, X., Yuan, J., et al.: Dair-v2x: A large-scale dataset for vehicle-infrastructure cooperative 3d object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21361–21370 (2022)
  * [69] Yuan, Y., Cheng, H., Sester, M.: Keypoints-based deep feature fusion for cooperative vehicle detection of autonomous driving. IEEE Robotics and Automation Letters 7(2), 3054–3061 (2022)
  * [70] Zeng, A., Song, S., Nießner, M., Fisher, M., Xiao, J., Funkhouser, T.: 3dmatch: Learning local geometric descriptors from rgb-d reconstructions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1802–1811 (2017)
  * [71] Zeng, K.C., Liu, S., Shu, Y., Wang, D., Li, H., Dou, Y., Wang, G., Yang, Y.: All your $\\{$GPS$\\}$ are belong to us: Towards stealthy manipulation of road navigation systems. In: 27th USENIX security symposium (USENIX security 18). pp. 1527–1544 (2018)
  * [72] Zhang, Z., Fisac, J.F.: Safe occlusion-aware autonomous driving via game-theoretic active perception. arXiv preprint arXiv:2105.08169 (2021)
  * [73] Zhong, S., Chen, H., Qi, Y., Feng, D., Chen, Z., Wu, J., Wen, W., Liu, M.: Colrio: Lidar-ranging-inertial centralized state estimation for robotic swarms. arXiv preprint arXiv:2402.11790 (2024)
  * [74] Zhong, S., Qi, Y., Chen, Z., Wu, J., Chen, H., Liu, M.: Dcl-slam: A distributed collaborative lidar slam framework for a robotic swarm. IEEE Sensors Journal (2023)
  * [75] Zhou, Y., Xiao, J., Zhou, Y., Loianno, G.: Multi-robot collaborative perception with graph neural networks. IEEE Robotics and Automation Letters 7(2), 2289–2296 (2022)