# Sample thickness measurements by phase-sensitive terahertz upconversion
detection

Tobias Pfeiffer  Fraunhofer-Institute for Industrial Mathematics ITWM,
Fraunhofer-Platz 1, 67663 Kaiserslautern, GERMANY  Department of Physics and
Research Center OPTIMAS, Technische Universität Kaiserslautern (TUK), 67663
Kaiserslautern, GERMANY<EMAIL_ADDRESS>Jens Klier
Fraunhofer-Institute for Industrial Mathematics ITWM, Fraunhofer-Platz 1,
67663 Kaiserslautern, GERMANY  Georg von Freymann  Fraunhofer-Institute for
Industrial Mathematics ITWM, Fraunhofer-Platz 1, 67663 Kaiserslautern, GERMANY
Department of Physics and Research Center OPTIMAS, Technische Universität
Kaiserslautern (TUK), 67663 Kaiserslautern, GERMANY  Daniel Molter
Fraunhofer-Institute for Industrial Mathematics ITWM, Fraunhofer-Platz 1,
67663 Kaiserslautern, GERMANY

###### Abstract

Nonlinear frequency conversion provides an elegant method to detect photons in
a spectral range which differs from the pump wavelength, making it highly
attractive for photons with inherently low energy. Aside from the intensity of
the light, represented by the number of photons, their phase provides
important information and enables a plethora of applications. We present a
phase-sensitive measurement method in the terahertz spectral range by only
detecting visible light. Using the optical interference of frequency-converted
photons and leftover pump photons of the involved ultrashort pulses, fast
determination of layer-thicknesses is demonstrated. The new method enables
phase-resolved detection of terahertz pulses using standard sCMOS equipment
while achieving sample measurement times of less than one second with a
precision error of less than 0.6%.

##  1 Introduction

The quantum-optical concept of measuring in extreme spectral ranges by using
correlated photons in nonlinear interferometers has drawn remarkable attention
in recent years [1, 2, 3, 4, 5]. The transfer of these concepts to the
terahertz frequency range is highly attractive, as the detection of terahertz
radiation is challenging due to its inherently low photon energy. Even though
the quantum-optical sensing and spectroscopy approach has been demonstrated in
this spectral range [6, 7], its measurement time is not yet competitive. But
the use of spontaneous parametric down-conversion and the correlation of the
involved photons is not the only possibility to transfer photon properties to
the visible range. Nonlinear upconversion has been proven to provide access
to, e.g. the mid-infrared [8, 9, 10] or terahertz [11, 12, 13, 14, 15]
frequency range. We present a novel phase-sensitive frequency-conversion
concept using nonlinear optics (up- and downconversion) in combination with
interferometry of the generated photons with leftover photons of the used
pulsed excitation light. Its potential is demonstrated by applying it to the
measurement of sample thicknesses, where a standard deviation of 4 fs
(corresponds to about 1.3 $\mu$ m on average) is achieved for measurement
times below one second. The measurements are carried out for six different
samples and are in good agreement with reference measurements performed using
a standard time-domain spectroscopy (TDS) system [16].

##  2 Methods

###  2.1 Phase detection principle

Figure 1: Scheme of the phase-sensitive terahertz upconversion detection: A
single 1550 nm pump source is used to generate the terahertz radiation and a
frequency-doubled pump radiation at 776 nm. This pump radiation is split by a
filter into a narrow-bandwidth portion and a so-called pumprest (which was
dumped in our earlier experiments). The narrow-bandwidth portion is mixed with
the terahertz radiation, resulting in the generation of up- and downconversion
sidebands. The superposition of these sidebands with the pumprest is the
phase-sensitive interference, which is finally detected.

Recently, we have demonstrated the intensity detection of terahertz radiation
by frequency conversion into the visible range[15]. To detect not only the
intensity but also the phase of the upconversion signal, a second source of
light with identical wavelength and a fixed time and phase relation is
superimposed with the upconverted signal to achieve interference. Our proposed
method is to use part of the pump light centered around 776 nm, as it
spectrally extends into the upconversion region and inherits the required
fixed phase and timing relation with respect to the pump photons. To visualize
the concept of our approach of phase-sensitive terahertz upconversion
detection, Fig. 1 shows a diagram of all the spectral components involved in
the experiment as well as their relation to each other. In the top left
corner, the spectrum of the initial 1550 nm fs-pulsed (therefore spectrally
broad) pump laser is sketched. The pulses are split to pump a photoconductive
antenna (PCA) for terahertz generation and a second-harmonics generation (SHG)
process to generate pump pulses at a center wavelength of about 776 nm. Using
a reflective bandpass filter (RBP), a narrow spectral band of the 776 nm pump
pulse is reflected from the RBP (R) while the remaining spectrum is
transmitted (T). The reflected part, titled _Filtered pump_ in the diagram, is
then superimposed with the terahertz radiation inside a nonlinear medium to
generate the up- and downconversion signal. Combining the resulting conversion
signal with the transmitted radiation from the RBP, titled _Pumprest_ , leads
to interference between the two signals as shown in the bottom most node of
the diagram. This interference signal is then phase-sensitive to the terahertz
radiation.

###  2.2 Setup

Figure 2: Current measurement setup. A 1550 nm laser (orange) pumps the
terahertz emitter and a frequency-doubling crystal to generate a NIR pump beam
(red), which is spectrally narrowed down using reflective filters and
superimposed with the terahertz radiation (green) before entering a nonlinear
medium. The transmitted radiation of the first filter is merged with the
optical pump with reduced intensity after reflecting off of a piezo controlled
mirror. After the nonlinear medium, the pump wavelength is removed in a filter
section and the remaining beam travels through a transmission grating onto a
sCMOS sensor. SHG: second harmonics generation, PPLN: periodically poled
lithium niobate, RBP: reflective bandpass, PCA: photoconductive antenna,
$\mathrm{f_{1},f_{2}}$: lenses (focal lengths of $\mathrm{f_{1}}$: 125 mm,
$\mathrm{f_{2}}$: 400mm).

The setup used in this experiment is shown in Fig. 2 and bases on the setup
presented in [15]. It is running on a single erbium-doped fiber laser emitting
femtosecond pulses in the 1550 nm range (orange) as pump source. This pump
source is split in two branches, one of which is used to drive a second-
harmonic generation (SHG) process in the first periodically poled lithium
niobate (PPLN) crystal. The frequency-doubled pulses emitted from the SHG
crystal (red) are further spectrally narrowed down using reflective bandpass
filters (RBP). In contrast to [15] and as described above, the share of the
pump light which is transmitted by the first RBP, further referred to as
pumprest, is not dumped but guided through a polarizing beam splitter (PBS)
followed by a quarter-wave plate to be reflected on a piezo-controlled mirror.
The reflected beam travels back through the quarter-wave plate into the PBS,
which then reflects the beam due to the induced polarization rotation. A beam-
sampling plate is used to recombine a fraction of the resulting beam with the
spectrally filtered 776 nm pump pulse, which also overlaps temporally with the
pumprest due to identical optical travel. The combined beam is then focused
into a second PPLN, which is phase matched for SFG and DFG of the optical pump
and terahertz radiation at 0.911 THz. The external terahertz radiation is
supplied by a photoconductive antenna (PCA)[17], which is pumped by the second
path of the original 1550 nm pump source. To overlap terahertz seed and
optical pump pulse temporally, a delay line is incorporated in the second 1550
nm pump branch. The terahertz pulse emitted from the PCA is collimated and
then focused into the PPLN using off-axis parabolic mirrors (OAPs), where the
focusing mirror has a hole to allow the 776 nm pump to travel through,
focusing on the same target. After the crystal, the upconversion signal is
separated from the remaining pump inside a filter section. Using a
transmission grating, the spectrum of the conversion signal together with the
pumprest are spectrally resolved and captured by an sCMOS sensor.

###  2.3 Data acquisition and evaluation

Figure 3: Camera images for the different signals. The top image shows the
conversion signal only while the center image shows only the pumprest. in the
bottom image, conversion and pumprest interfere. The red curve shows the
column maximum of the image behind. The bottom image includes dashed curves
for maximum constructive and destructive interference.

For a better understanding of the signal captured by the camera, the
conversion signal and the pumprest are shown separately in Fig. 3. The top
image shows the terahertz up- and downconversion signal as round Gaussian
spots, with the downconversion on the left and the upconversion on the right
side. The x-axis indicates the frequency shift of the signal in relation to
the optical pump. The red curve indicates the column sum for each image
column, providing a cross section of the signal for a better visualization. In
later steps, the integral of an area of 13x13 pixel surrounding the conversion
maxima, indicated by the blue square, is used as measure of intensity, as done
in [15]. The camera signal of the pumprest with no conversion signal is shown
in Fig. 3 in the center image. The spectrum of the pumprest is much broader
than the conversion signal, extending into the suppressed spectral range
surrounding the pump. Overlapping the conversion signal with the pumprest
results in interferometry between the two arms, as shown in the lower image in
Fig. 3. While the red curve shows again the cross section of the camera image,
the dashed black curves indicate the cross sections for optimal constructive
and destructive interference, with the red curve in between. The impact of the
phase-sensitive detection becomes apparent in the following section, where the
signal is compared to the intensity-only measurement.

###  2.4 Comparison to time-of-flight

While the intensity of the upconversion signal alone can be used for time-of-
flight measurements by varying the arrival time of the terahertz pulse as
demonstrated in [15], the difference in arrival time between the optical pump
pulse and the terahertz pulse needs to be significant in order to detect the
change in upconversion intensity. By measuring the phase of the upconverted
signal as described, detection of smaller changes in optical path lengths is
possible. Since the phase of the upconversion signal is directly correlated
with the phase of the terahertz seed, all changes in the terahertz phase due
to changes in the optical path length can be detected in the conversion
signal. To compare the resolution of the phase-sensitive detection method with
the intensity-only measurement from [15], the delay between terahertz signal
and optical pump is varied using the terahertz delay line, with the pumprest
turned off to detect intensity only and turned on for the interferometric
measurement.

Figure 4: Comparing the intensity-only signal with the phase-resolved signal
when tuning the delay between optical pump and terahertz pulse. Due to the
much steeper slopes on the interference curve, slight changes in position
result in much larger changes in value. The blue curve indicates the
calculated envelope of the interference curve. The interference measurement is
also used to determine the frequency of the terahertz radiation.

In Fig. 4, the intensity-only signal is shown as green curve while the
interference signal is displayed in red. For a better comparison of the
signals, the intensity curve is offset by the constant pumprest signal. As a
measure of resolution, the gradients for the steepest parts of the curves are
used, as they express the maximum change in camera signal for a given delay in
terahertz arrival. The gradients are also included in Fig. 4 as dashed black
lines. The calculated values for the gradients are $\mathrm{4.3\cdot
10^{6}~{}ps^{-1}s^{-1}}$ for the intensity curve and $\mathrm{648.3\cdot
10^{6}~{}ps^{-1}s^{-1}}$ for the interference curve. With this, the
sensitivity of our proposed method with respect to optical path length changes
is increased by a factor of about 150 in comparison to the intensity-only
detection method. Since the travel distance of the delay line used to measure
the data from Fig. 4 is well-known, the measurement can also be used to
calculate the frequency of the interference signal, which equals the frequency
of the terahertz signal, assuming that the phase of the pumprest is stable.
The frequency is determined by Fourier transformation of the zero-filled
interference signal and equals 0.911 THz.

###  2.5 Signal-to-pumprest ratio

A standard parameter which characterizes the quality of interference is called
visibility and describes the ratio between the theoretical and the measured
difference between constructive and destructive interference. The visibility
can reach at most 1.0 for a perfect interference setup. Since theoretical
extrema depend on the ratio between the isolated upconversion signal and the
pumprest, both of these values are measured independently beforehand. The
theoretical intensities of the interference extrema equal

$\displaystyle
I_{\mathrm{int,max}}\propto(E_{\mathrm{rest}}+E_{\mathrm{conv}})^{2}=E_{\mathrm{rest}}^{2}+2E_{\mathrm{rest}}E_{\mathrm{conv}}+E_{\mathrm{conv}}^{2},$
(1) $\displaystyle
I_{\mathrm{int,min}}\propto(E_{\mathrm{rest}}-E_{\mathrm{conv}})^{2}=E_{\mathrm{rest}}^{2}-2E_{\mathrm{rest}}E_{\mathrm{conv}}+E_{\mathrm{conv}}^{2}.$
(2)

Since the setup allows the measurement of intensities only,
$E_{\mathrm{rest}}^{2}$ is replaced by $I_{\mathrm{rest}}$ and
$E_{\mathrm{conv}}^{2}$ by $I_{\mathrm{conv}}$. This leads to the following
equations:

$\displaystyle
I_{\mathrm{int,max}}=I_{\mathrm{rest}}+2\sqrt{I_{\mathrm{rest}}}\sqrt{I_{\mathrm{conv}}}+I_{\mathrm{conv}},$
(3) $\displaystyle
I_{\mathrm{int,min}}=I_{\mathrm{rest}}-2\sqrt{I_{\mathrm{rest}}}\sqrt{I_{\mathrm{conv}}}+I_{\mathrm{conv}}.$
(4)

Using this formula, the theoretical extrema are calculated for different
ratios and displayed as black lines in Fig. 5 (b). For a ratio of 1.0, the
destructive interference curve reaches its minimum at zero intensity, which
indicates perfect destructive interference. The difference between the maximum
and minimum curve increases with a higher ratio, following the equation

$I_{\mathrm{\Delta int}}=4\sqrt{I_{\mathrm{rest}}}\sqrt{I_{\mathrm{conv}}}.$
(5)

Figure 5: Evaluation of the interference visibility. The diagram (a) shows a
cutout of Fig. 3 including a rectangle indicating the integrated image area
for evaluation. Diagram (b) shows minimum and maximum integrals for a given
ratio of conversion signal to pumprest as black marks with the theoretical
values as black curves. The red curves show the theoretical values including
an additional correction factor.

The black marks in Fig. 5 (b) show the measured minima and maxima for the
given ratios. While the data points form a similar shape, the measurement does
not align with the theory. Also, the minimum curve does not reach its lowest
point at a ratio of 1.0, which indicates that the calculated ratios do not
match the actual ratio of interfering radiation. A possible source for this
mismatch are the temporal profiles of the interfering pulses. By using the
camera signal to measure the intensity of each component, only the average
intensity can be determined. If the pulse of one of the components is shorter
in time but has a higher peak intensity, the camera signal will be identical
for the individual components, the interference however will not be ideal
because part of the longer pulse has no temporal overlap with the shorter
pulse. Other effects such as spatial overlap or a chirped pulse might result
in similar deviations. To account for this effect, an additional factor $a$ is
introduced to equations (3) and (4) as follows

$\displaystyle
I_{\mathrm{int,max}}=I_{\mathrm{rest}}+2a\sqrt{I_{\mathrm{rest}}}\sqrt{I_{\mathrm{conv}}}+I_{\mathrm{conv}},$
(6) $\displaystyle
I_{\mathrm{int,min}}=I_{\mathrm{rest}}-2a\sqrt{I_{\mathrm{rest}}}\sqrt{I_{\mathrm{conv}}}+I_{\mathrm{conv}},$
(7)

which describes the share of one component which can interfere with the other.
The red curve in Fig. 5 shows the calculated curve using a parameter of
$a=0.795$, which minimizes the accumulated deltas between the measured values
and the calculated curve and matches the measurement to a high degree. The
additional parameter $a$ corresponds to the visibility which, with a value of
0.795, provides sufficient contrast for the desired use case.

##  3 Results

###  3.1 Sample Measurements

Figure 6: (a) Diagram of the phase measurement process. The piezo moves
forward and backward while the upconversion intensity is measured. (b) Time-
of-flight measurement results for PET-samples with different thicknesses
compared to TDS measurements. For the last sample, the phase shift exceeded
$2\pi$ due to the high thickness, which was compensated accordingly
afterwards.

By periodically moving the piezo-mounted mirror, the phase of the pumprest in
relation to the conversion signal is varied constantly. For linear movement of
the piezo, this results in a sine-wave-like interference pattern, which can be
sampled using the camera. In Fig. 6 (a), the stepwise movement of the piezo is
shown in the lower part of the diagram, while the data acquired from the
camera (pixel intensity integral of the area of interference) is displayed in
the upper part. A sine function is fitted to the interference pattern (shown
as red curve), whose phase is used to detect changes in the terahertz beam
path, as it correlates with the phase of the upconversion signal and therefore
the terahertz phase. When inserting a sample into the terahertz beam, the
phase of the interference pattern shifts. The dashed red curve shows the sine
fit of the pattern before the sample is inserted, which is shifted by
$\Delta\varphi$ relative to the solid curve. Using an automated sample wheel,
an array of six samples of PET plastic with precisely determined physical
thicknesses ranging from 186 $\mu$ m to 807 $\mu$ m can be inserted into and
removed from the terahertz beam to measure the phase shift. A single
measurement consists of four piezo sweeps, where the reference and sample
measurements take a single sweep each and the insertion process takes two
sweeps in between. The total measurement time is therefore specified as 520
ms. For statistical purposes, the sample wheel completes 100 full rotations,
inserting and removing each of the six samples every rotation. With the
wavelength of the terahertz radiation, the averaged $\Delta\varphi$ values for
each sample can be used to calculate the additional optical path length of the
terahertz radiation introduced by the sample. These values are compared to
reference measurements performed with a state of the art terahertz TDS
measurement system[16]. Figure 6 shows the results for all samples as a bar
plot, with the reference TDS measurement results in blue and the results of
the upconversion measurement in red.

###  3.2 System stability

To measure the stability of the interference phase over time, the phase of
both up- and downconversion was continuously evaluated over a 15 h period. The
results are shown in Fig. 7, where the red curve indicates the phase of the
upconversion signal and the blue curve represents the downconversion. The
black curve shows the average value between up- and downconversion.
Considering the full 15 h measurement window, the red and blue curve seem to
drift mostly in equal direction, moving the average curve with them. This
behavior is assumed to be caused by thermal influence on the path of the
pumprest and is identical to the effect of moving the piezo mounted mirror.
This effect can be compensated for by using the average signal as artificial
zero line. The inset in Fig. 7 however shows additional symmetrical drifts
which do not affect the average curve. These drifts are caused directly by
variations of the terahertz pulse timing in relation to the optical pump and
also originate from thermal influences, which act on the optical fiber paths
from the shared pump to the emitters of the terahertz and pump radiation.
These drifts affect the interference pattern in the same fashion as a real
measurement would. When measuring samples however, the samples can be inserted
and removed from the terahertz beam quickly enough to neglect the influence of
the drift.

Figure 7: Long-term measurement of the phase of upconversion (red) and
downconversion signal (blue), displayed together with the average of both
signals in black. The inset shows a magnified part of the signal, where a
symmetric short-term behavior of the phases can be perceived.

##  4 Discussion

The determined thicknesses using the novel phase-sensitive upconversion method
agree well with the results of the TDS measurement system. While the results
of two samples show greater deviations, the other samples deviate only 1% on
average. Due to the limited aperture of the samples, the samples are inserted
into the diverging part of the terahertz beam, which leads to a slight
increase in optical travel through the sample and therefore a systematic
error. Although the evaluated thicknesses show the tendency to be
overestimated, a correlation between sample thickness and the deviation cannot
be seen. The two outliers must therefore be connected to sample-specific
properties which influence the evaluation. High dispersion of the material in
the terahertz range could lead to different results when different terahertz
spectra are used, although the spectrum of the TDS system in use has its
spectral maximum at around 1 THz as well. It is worth noting that sample 6
causes the phase to shift by more than $2\pi$ due to the high thickness of the
sample. The measured phase shift was compensated accordingly. The standard
deviation of 100 measurements for each sample is below 4 fs, which corresponds
to a an average deviation of 1.3 $\mu$ m in terms of thickness. Since the
first sample is the thinnest at 186 $\mu$ m, the precision of the measurement
has the most impact on the result, causing a relative uncertainty of 0.6 %.
Due to the small deviations, Fig. 6 (b) does not include error bars.

So far, the proposed method is realized for a narrow spectral bandwidth of the
detected terahertz photons (due to the used phase-matching scheme), but
already proves the applicability of the proposed method. By implementing other
nonlinear media or phase-matching schemes, the spectral bandwidth can be
enhanced, which might enable the measurement of multilayer structures in
reflection (which is one of the demands of many applications).

##  5 Conclusions

To conclude, we have introduced a novel phase-sensitive frequency-conversion
detection principle that enables fast measurements in the terahertz frequency
range by only detecting visible light. By implementing an interferometry
principle, the sensitivity for optical path length changes is increased by a
factor of 150 in comparison to intensity-only frequency-conversion methods.
The detected interference provides a good visibility of 0.8 and the high
short-term stability of the interference pattern allows for thickness
measurements within less than a second. Measurements on PET samples
demonstrated a standard deviation lower than 4 fs, corresponding to below 0.6
% in thickness using measurement times below one second. Future extensions to
two-dimensional measurements will elevate the potential of this approach.

## Funding

This work was carried out in the scope of the Fraunhofer Lighthouse Project
”Quantum Methods for Advanced Imaging Solutions” (QUILT).

## Disclosures

The authors declare no conflicts of interest.

## Data availability

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## References

  * [1] G. B. Lemos, V. Borish, G. D. Cole, S. Ramelow, R. Lapkiewicz, and A. Zeilinger, “Quantum imaging with undetected photons,” Nature 512, 409–412 (2014).
  * [2] D. A. Kalashnikov, A. V. Paterova, S. P. Kulik, and L. A. Krivitsky, “Infrared spectroscopy with visible light,” Nature Photonics 10, 98–101 (2016).
  * [3] A. Zeilinger, S. Ramelow, R. Lapkiewicz, V. Borish, and G. B. Lemos, “Quantum imaging with undetected photons,” (2017). US Patent 9,557,262.
  * [4] M. Gilaberte Basset, F. Setzpfandt, F. Steinlechner, E. Beckert, T. Pertsch, and M. Gräfe, “Perspectives for applications of quantum imaging,” Laser & Photonics Reviews 13, 1900097 (2019).
  * [5] C. Lindner, S. Wolf, J. Kiessling, and F. Kühnemann, “Fourier transform infrared spectroscopy with visible light,” Optics express 28, 4426–4432 (2020).
  * [6] M. Kutas, B. Haase, P. Bickert, F. Riexinger, D. Molter, and G. von Freymann, “Terahertz quantum sensing,” Science Advances 6, eaaz8065 (2020).
  * [7] M. Kutas, B. Haase, J. Klier, D. Molter, and G. von Freymann, “Quantum-inspired terahertz spectroscopy with visible photons,” Optica 8, 438–441 (2021).
  * [8] L. M. Kehlet, P. Tidemand-Lichtenberg, J. S. Dam, and C. Pedersen, “Infrared upconversion hyperspectral imaging,” Optics letters 40, 938–941 (2015).
  * [9] P. Tidemand-Lichtenberg, J. S. Dam, H. Andersen, L. Høgstedt, and C. Pedersen, “Mid-infrared upconversion spectroscopy,” JOSA B 33, D28–D35 (2016).
  * [10] N. M. Israelsen, C. R. Petersen, A. Barh, D. Jain, M. Jensen, G. Hannesschläger, P. Tidemand-Lichtenberg, C. Pedersen, A. Podoleanu, and O. Bang, “Real-time high-resolution mid-infrared optical coherence tomography,” Light: Science & Applications 8, 1–13 (2019).
  * [11] Y. J. Ding and W. Shi, “Efficient thz generation and frequency upconversion in gap crystals,” Solid-state electronics 50, 1128–1136 (2006).
  * [12] M. J. Khan, J. C. Chen, and S. Kaushik, “Optical detection of terahertz radiation by using nonlinear parametric upconversion,” Optics letters 32, 3248–3250 (2007).
  * [13] M. J. Khan, J. C. Chen, and S. Kaushik, “Optical detection of terahertz using nonlinear parametric upconversion,” Optics letters 33, 2725–2727 (2008).
  * [14] B. Zaks, J. Heyman, D. Stehr, D. Allen, N. Coates, and M. Sherwin, “Single shot high resolution thz upconversion spectrometer,” in _2008 33rd International Conference on Infrared, Millimeter and Terahertz Waves,_ (IEEE, 2008), pp. 1–2.
  * [15] T. Pfeiffer, D. Molter, and G. von Freymann, “Terahertz detection by upconversion to the near-infrared using picosecond pulses,” Optics Express 28, 29419–29429 (2020).
  * [16] S. Weber, L. Liebelt, J. Klier, T. Pfeiffer, D. Molter, F. Ellrich, J. Jonuscheit, and G. Von Freymann, “Influence of system performance on layer thickness determination using terahertz time-domain spectroscopy,” Journal of Infrared, Millimeter, and Terahertz Waves 41, 438–449 (2020).
  * [17] R. J. Dietz, B. Globisch, M. Gerhard, A. Velauthapillai, D. Stanze, H. Roehle, M. Koch, T. Göbel, and M. Schell, “64 $\mu$ w pulsed terahertz emission from growth optimized ingaas/inalas heterostructures with separated photoconductive and trapping regions,” Applied Physics Letters 103, 061103 (2013).


# V2X-AHD:Vehicle-to-Everything Cooperation Perception via Asymmetric
Heterogenous Distillation Network

Caizhen He, Hai Wang, Long Chen, Tong Luo, Yingfeng Cai This work was
supported in part by the National Natural Science Foundation of China
(52225212, U20A20333, 52072160), Key Research and Development Program of
Jiangsu Province (BE2019010-2, BE2020083-3).(Corresponding Author: Hai
Wang);C. H. and H. W. are with the School of Automotive and Traffic
Engineering of Jiangsu University, Zhenjiang, 212013, China. (e-mail:
<EMAIL_ADDRESS>wanghai1019@163.com,) L. C., and Y.C.are with
the Automotive Engineering Research Institute of Jiangsu University,
Zhenjiang, 212013, China. (e-mail<EMAIL_ADDRESS>caicaixiao0304@126.com)T. L. is with the School of Automobile and Traffic
Engineering of Jiangsu University of Technology, Changzhou, 213001, China.
(e-mail<EMAIL_ADDRESS>
###### Abstract

Object detection is the central issue of intelligent traffic systems, and
recent advancements in single-vehicle lidar-based 3D detection indicate that
it can provide accurate position information for intelligent agents to make
decisions and plan. Compared with single-vehicle perception, multi-view
vehicle-road cooperation perception has fundamental advantages, such as the
elimination of blind spots and a broader range of perception, and has become a
research hotspot. However, the current perception of cooperation focuses on
improving the complexity of fusion while ignoring the fundamental problems
caused by the absence of single-view outlines. We propose a multi-view
vehicle-road cooperation perception system, vehicle-to-everything cooperative
perception (V2X-AHD), in order to enhance the identification capability,
particularly for predicting the vehicle’s shape. At first, we propose an
asymmetric heterogeneous distillation network fed with different training data
to improve the accuracy of contour recognition, with multi-view teacher
features transferring to single-view student features. While the point cloud
data are sparse, we propose Spara Pillar, a spare convolutional-based plug-in
feature extraction backbone, to reduce the number of parameters and improve
and enhance feature extraction capabilities. Moreover, we leverage the multi-
head self-attention (MSA) to fuse the single-view feature, and the lightweight
design makes the fusion feature a smooth expression. The results of applying
our algorithm to the massive open dataset V2Xset demonstrate that our method
achieves the state-of-the-art result. The V2X-AHD can effectively improve the
accuracy of 3D object detection and reduce the number of network parameters,
according to this study, which serves as a benchmark for cooperative
perception. The code for this article is available at
https://github.com/feeling0414-lab/V2X-AHD.

###### Index Terms:

Knowledge Distillation, V2X perception, Heterogenous data.

##  I Introduction

Environmental perception is a fundamental component of autonomous driving
[1][2], as it provides information for path planning and motion control
[3][4][5]. As a critical task of environment perception, object detection
[6][7] has always been a research hotspot in autonomous driving. Light
changes, as well as rainy and foggy weather, have a significant impact on
vision-based object detection methods, resulting in low-security redundancy.
In recent years, with the enhancement of information processing capabilities
and the decrease in sensor prices, lidar [8][9][10][11][12] has achieved
leapfrog development with its superior space construction and anti-weather
interference capabilities. Academic and industrial circles are largely in
agreement that lidar plays a role in advanced intelligent driving
technologies. Despite this, the current application of lidar is limited to the
single-vehicle dimension, and single-view perception has many inherent flaws.
For instance, in crowded road conditions, it is easy to develop blind spots
due to occlusion, whereas in long-distance open road conditions, it will
result in sparse perception. More importantly, if a pedestrian suddenly
emerges from the blind spot, the self-driving vehicle will have very little
time to react, which will be a difficult problem for human drivers as well.

Figure 1: Advantages of cooperation perception. The purpose of our model is
that a single view of an object has the characteristics of a fusion view. (a)
and (b) show the single-view object detection results, respectively. (c) shows
the fusion view detection results. (d) and (e) show the single-view vehicle’s
point cloud. (f) shows the fusion view vehicle’s point cloud.

To address the aforementioned issues with single-view perception, a
collaborative perception system incorporating multi-view data has emerged. In
congested road conditions, the collaborative sensing system has a global
vision. Cross-viewing compensates for the obstruction caused by a single
perspective. In contrast, when sensing long-distance road conditions, the
collaborative sensing system increases the sensing distance through
information relays and solves the issue of sparse perception. It is believed
that cooperative sensing can achieve superior perception results compared to
single-view vehicles. According to the different transmission content, the
existing cooperative sensing strategies can be categorized into three groups
based on the transmission content: early fusion[13] of the original point
cloud data transmission, intermediate fusion [14][15][16][17] of the
transmitted point cloud feature data, and late fusion [18][19][20] of the
transmission perception results. In the early fusion, the original point cloud
data are used for splicing. The perception problem of a single vehicle can be
fundamentally resolved by filling the point cloud gap caused by occlusion and
expanding the point cloud range. However, due to the enormous amount of data
in the original point cloud, the transmission bandwidth limits information
transmission, so early fusion cannot be realized. In contrast, the post-fusion
strategy transmits each perspective’s perception results directly, and the
transmission bandwidth requirement is small. However, due to differences in
the perception results of the same object from different perspectives, the
fusion results are susceptible to the negative results of a single view,
resulting in missed and false detections. After the information exchange, the
intermediate fusion strategy fuses the extracted intermediate features. It is
a compromise strategy that balances perception accuracy and transmission
bandwidth, and it has become the predominant research direction of
collaborative sensing.

Currently, the intermediate fusion strategy improves perception accuracy by
increasing the module’s complexity. However, it only improves the fitting
capability brought on by the increase in parameters. The characteristics and
essence of the fusion perspective have not yet been explored by the
aforementioned methods, and increasing the number of model parameters will
decrease real-time performance. The accuracy of collaborative perception is
dependent on two factors: the extraction of single-view scene features and the
fusion of multi-view scene features. Existing intermediate strategy fusion
methods emphasize the latter, often ignoring the fundamental role of the
former. In our primary research on object detection, we identified two
features: First, as shown in Figures 1(a) and (b), the current single-vehicle
object detection is more accurate for vehicle position detection but less
accurate for vehicle shape detection. Second, as shown in Figure 1(c), the
perceptual accuracy of early fusion strategies is typically greater than that
of intermediate fusion strategies. The causes for the aforementioned
characteristics are also evident. The point cloud of a single vehicle is
sparse and incomplete. At the same time, due to the limitation of the viewing
angle. As shown in Figures 1(d) and (e), although the algorithm can determine
the precise position of the vehicle, it is difficult to determine the car’s
outline. In Figure 1(f), the vehicle’s outline is complete and distinct in the
fused viewpoint cloud. The early fusion strategy of direct matching of the
original point cloud is the most effective for performance improvement and has
interpretability. As shown in Figure 1(c), the roadside radar has more wiring
harnesses and denser point clouds, and the collaborative perception of
vehicle-road coordination is typically more robust.

At present, the intermediate fusion strategy improves perception accuracy by
increasing the complexity of the fusion module. However, it only enhances the
fitting ability brought about by the increase in the number of parameters. The
above methods have yet to explore the characteristics and essence of the
fusion perspective, and increasing the number of parameters of the model will
lead to a decrease in real-time performance. The accuracy of collaborative
perception depends on two aspects: single-view scene feature extraction and
multi-view scene feature fusion. Existing intermediate strategy fusion methods
emphasize the latter, often ignoring the fundamental role of the former. In
the primary research on object detection, we found two characteristics: First,
the existing single-vehicle object detection is more accurate for vehicle
position detection, but weak for vehicle shape detection, as shown in Figure
1(a)(b). Second, the perceptual accuracy based on early fusion strategies is
usually higher than intermediate fusion strategies, as shown in Fig. 1(c). The
reasons for the above characteristics are also apparent. The point cloud of a
single vehicle is sparse and incomplete. At the same time, due to the
limitation of the viewing angle. Although the algorithm can judge the
vehicle’s specific position, the car’s outline is difficult to determine, as
shown in Figure 1(d)(e). In Figure 1(f), the vehicle’s outline in the fused
viewpoint cloud is clear and complete. The early fusion strategy of direct
matching of the original point cloud is the most effective for performance
improvement and has interpretability. As shown in Figure 1(c), the roadside
radar has more wiring harnesses and denser point clouds, and the collaborative
perception of vehicle-road coordination often has a more robust performance.

Figure 2: Comparison of distillation modes. The yellow and green lines
represent the teacher and student network data flow, respectively. (a) and (b)
illustrate two kinds of distillation modes, prediction mimicking, and Feature
Imitation. (c) is our new mode of asymmetric knowledge distillation, the raw
point cloud is divided into a fusion view and a single view.

This paper proposes an algorithm for vehicle-to-everything cooperative
perception V2X-AHD based on an asymmetric heterogeneous distillation network.
Among them, asymmetry refers to the difference between the parameters of the
teacher network and the student network, while heterogeneity refers to the
heterogeneity of point cloud data generated by other sensor models and
different detection perspectives due to vehicle-side data and roadside data.
The main contributions of this paper are as follows:

1\. Propose an asymmetric structure for knowledge distillation [21][22][23].
Integrate the strategic benefits of early and intermediate integration. During
training, the teacher and student networks are separately trained using fused
point cloud and single view data. When using single-view data to extract
features, the student network has the same fusion point cloud features as the
teacher network and can automatically complete the incomplete vehicle outline
features. The asymmetric structure is adopted for the difference of cross-view
point cloud data of vehicles and roads, which improves the accuracy of point
cloud feature extraction with different structures.

2\. Feature extraction module Spare Pillar. Due to the sparse nature of point
clouds, we convert them to 2D pseudo-images and design a Spare Pillar feature
extraction module based on sparse convolution [25]. Compared with the
conventional dense convolution method, it has superior feature encoding
capabilities while significantly reducing computation time. The step-by-step
descending structure of the bottleneck module facilitates multi-scale feature
fusion and helps decouple point cloud features of different sizes.

3\. Multi-head self-attention (MSA) feature fusion module. Existing methods
hinder the practical expression of single-vehicle features due to the use of
complex fusion modules; therefore, we employ a lightweight MSA mechanism to
reduce computation while realizing the practical expression of single-vehicle
features. The remainder of the paper is organized as follows. The literature
related to this work is briefly introduced in related work. The Methodology
section describes the V2X-AHD algorithm structure proposed in this paper. The
Experiments section demonstrates the results of the experiments of the
proposed algorithm on the V2XSet[14] dataset, while the Conclusion section
provides a summary.

##  II related works

###  II-A Knowledge Distillation

Knowledge distillation is also known as the teacher-student model
[21][22][23]. Geoffrey Hinton [22] noted that the smooth distribution result
predicted by the teacher network makes it easier for students to learn
features than the Dirac result. The knowledge distillation network trains the
teacher and student networks with different parameters. The network of
teachers with a large parameter scale can be compressed into a smaller network
of students. At the same time, the teacher network and the student network can
input different data for training. Currently, this method is widely employed
in the fields of semantic segmentation [26], point cloud target detection
[27], and object re-identification [28]. As shown in Figures 2(a) and 2(b),
the distillation network can be divided into Feature Imitation and Prediction
Mimicking, depending on the transfer feature content. DiscoNet [29] utilizes a
symmetric distillation network and a predictive simulation structure to
transform multi-view point cloud features into a single-view result. However,
DiscoNet’s Prediction Mimicking has the problem of hindering the detection of
performance diversity. In contrast, the purpose of feature simulation is to
improve the consistency of the teacher-student model in terms of latent
features. As shown in Figure 2(c), we employ an innovative asymmetric
knowledge distillation structure to train the point cloud feature extraction.
Compared with the student network, the teacher network has a more significant
number of parameters and uses multi-view fusion point clouds as data input.
The parameters of the student network are small, and single-view point cloud
data are used to extract the network’s features. During the stage of training,
the fused-view features of the teacher network are transferred to the single-
view features of the student network. During the test phase, even without the
guidance of the teacher network, the student network retains the feature
expression of the fusion point cloud beneath the single-view data.

###  II-B 3D point cloud object detection

Currently, the 3D object detection algorithms based on point cloud follow the
”encoding-decoding-detection head” detection architecture and adopt the grid-
based form. According to the type of feature extraction convolutional network,
it can be divided into two categories: the first category, based on the 3D
voxel grid method [8][30][9][10][31], uses a 3D convolutional network to
extract features; the second category, based on 2D Pillar grids, is the method
[32][11][33][34], uses 2D convolutional networks to extract features.
VoxelNet[18], the pioneering work of the voxel grid method, divides the point
cloud space into 3D voxels and encodes point cloud features using 3D
convolution. The Second [9] method investigates the problem of empty voxels in
the VoxelNet method’s long-distance area and employs a sparse 3D convolution
method to improve the detection accuracy. The above algorithms that use 3D
convolutional networks to extract features have high accuracy. However, due to
the high computational complexity of the 3D convolution method, even the
enhanced sparse method still has a substantial computational overhead. To
address the aforementioned issues, a class of techniques employing 2D
convolutions has gradually emerged, converting 3D point clouds into 2D pseudo-
image features and then employing 2D convolutions with reduced operational
complexity. One representative algorithm is the PointPillar [32] pillar
feature extractor based on PointNet [12]. HVNet [33] combines columnar
features of varying scales to enhance operational precision and inference
speed. Currently, algorithms based on 2D pseudo-graph features prioritize the
projection of complex pillar features and multi-scale aggregation. However,
compared to images, the information density of point cloud grids is
significantly smaller than that of image grids, and the features are sparse.
Using dense convolution methods for point cloud features wastes computing
resources. We use sparse 2D convolution as the core and propose the Spare
Pillar Plug and Play feature extraction network. Sparse convolution can reduce
computation time significantly. At the same time, the detection accuracy
improves due to the more concentrated feature extraction.

Figure 3: Pipeline of our proposed V2X-AHD. It consists of two networks: a
teacher network and a student network. The single arrow solid line and dotted
line represent the forward propagation and backward propagation process,
respectively. The details of each individual component are illustrated in
Sec.3.

###  II-C Vehicle-to-Everything Cooperation Perception

Vehicle-to-Everything collaborative perception is the process of obtaining
fusion perception results by receiving the perception data of neighboring
agents. Due to its finer point cloud data and more stable own position
coordinates, roadside lidar is more conducive to enhancing detection accuracy
than onboard lidar. However, because access to roadside equipment will result
in differences in point cloud data perspectives, a heterogeneous fusion module
is required to fuse them. Currently, the mainstream algorithm of collaborative
sensing can be divided into three categories based on the various transmission
data types: the early fusion of the original point cloud data [13], the
intermediate fusion of the transmission point cloud features [14][15][16][17],
and the transmission perception. As a result of late fusion [18][19][20], the
required bandwidth for data transmission decreases for each of the three
methods. Cooper[25] proposed the early fusion method based on the original
point cloud data level. He obtained the perception result of the complete
perspective by aggregating the surrounding point cloud data. This method can
fundamentally solve the problem of a single perspective, but transmitting raw
point cloud data requires a substantial amount of communication bandwidth and
high latency. In the late fusion algorithm, Andreas[18] proposed a high-
dimensional sensor fusion structure that uses temporal and spatial sequences
as input data to predict the position, action, and angle of surrounding agents
in three dimensions. Rawashdeh [19] utilized machine learning techniques to
transmit three-dimensional information, including the position and size of the
tracking object’s center point, in order to accurately predict surrounding
objects. However, in late fusion, although the required data transmission
bandwidth is small, a significant amount of effective scene information will
be lost, resulting in excessive dependence on the results of a single agent,
and an effective error correction mechanism cannot be created when individual
sensors make errors. The intermediate fusion strategy can reduce the
transmission bandwidth requirements to reduce the loss of scene information,
which has attracted a large number of researchers to conduct research in this
area. F-cooper [15] employs voxel characteristics for information transfer and
the Maxout [35] method for voxel fusion at the junction. The OPV2V [17]
algorithm leverages the self-attention mechanism [36] to learn the interaction
between compressed features at the same spatial position. The V2Vnet [16]
algorithm fuses passed compressed features using a graph convolutional neural
network GNN [37] network. V2X-ViT [14] considers the issue of space-time
misalignment caused by real-world communication delays and employs self-
attention mechanisms and multi-scale windows to fuse features from various
perspectives. The MSA module minimizes the parameters’ size while maintaining
the perception’s accuracy.

The following sections will focus on the structural details of each part of
the model V2X-AHD proposed in this paper.

##  III Methodology

In this paper, we propose a Vehicle-to-Everything cooperation perception
system V2X-AHD based on 3D point cloud data to perceive the surrounding
environment of a vehicle. The primary sources of design inspiration are the
high precision of the early fusion strategy and the benefits of the knowledge
distillation network for joint training of different input stream data. The
advantage of the knowledge distillation system is its ability to compress the
knowledge gained from the large model into the small model by means of
”knowledge distillation.” At the same time, the distillation model can be used
for zero-shot or few-shot learning, allowing the teacher and student networks
to input distinct training data. The point cloud of Vehicle-to-Everything
cooperation perception is unique, and it is simple to obtain multi-view point
cloud feature training offline. In contrast, in the online test scenario, the
agent can only accept a single perspective point cloud to extract and transmit
features, which is perfectly compatible with the distillation network.

During the training stage, the teacher model uses multi-view fusion data,
while the student model uses single-view data. Both of them use the Spare
Pillar to extract the points cloud feature. Through the distillation method of
feature simulation, the teacher model transfers the multi-view feature
extraction paradigm to the single-view student model. During the test stage,
although the student model is trained with single-view data, the detection
performance can approach that of the teacher model using multi-view data,
thereby fundamentally solving the problem of single-view data feature
extraction.

Figure 4: Feature extraction network Spare Pillar structure. The structure
includes three parts: a point feature net, an encoder, and a neck. Orange and
blue modules represent sparse convolution and dense convolution, respectively

###  III-A Student network

Each agent i$\in${1,…,N} within the communication distance, agent categories
ci$\in${V,I}, V and I represent vehicles and roadside equipment, respectively.
We assume that the data transmission is synchronized, which means that each
agent i position is Pi, and the lidar data are Li. Assuming that agent egois
selected as the central agent, and ego accepts the positions from surrounding
agents. Central agent ego can receive the original point cloud or feature from
surrounding agents through coordinate system transformation.

####  III-A1 Feature extraction

The point cloud feature extraction algorithm uses the 2D columnar grid feature
extraction algorithm Spare Pillar, as shown in Figure 4. Using the ”Encoder-
Neck” architecture under the Bird’s Eye View (BEV) while retaining the
traditional 2D columnar grid algorithm, it employs a sparse convolution method
better suited for point cloud features to further reduce inference delay and
improve feature extraction capabilities. The overall structure consists of
three components: a 2D pseudo-image generation module, encoder, and bottleneck
block Neck. Point Feature Net converts the point cloud into a stacked Pillar
tensor, which is then projected into a 2D pseudo-image with a scale of
H$\times$W$\times$C, where H and W represent the height and width of the
pseudo-image canvas, respectively, and A indicates the number of channels of
the pseudo-image. The 2D pseudo-image is then fed into the encoder. The
Encoder adopts the VGGNet [38] architecture. The objective is to extract
sparse columnar features of varying depths from the projected sparse 2D pillar
and to feed all sparse pillar features of varying scales into the bottleneck
block. Since feature compression has been performed, sparse pillar features
can be fused using standard dense 2D convolutions. The specific procedure is
illustrated by formula (1):

$F_{i}^{s}=\phi_{s}(L_{i}),F_{i}^{s}\in\mathbb{R}^{\bar{H}\times\bar{W}\times\bar{C}}$
(1)

where $\phi_{s}(\bullet)$ represents the student feature extraction network,
and $\mathbb{R}^{\bar{H}\times\bar{W}\times\bar{C}}$ indicates the scale of
the feature space after convolution. SparePillar employs sparse convolution to
reduce computational complexity during the stage of feature extraction. In the
feature upsampling procedure, dense convolution is used to integrate high-
level abstract semantics and low-fine-grained spatial features to improve the
accuracy of large objects. We use different feature extraction structures for
the teacher and student models. The experimental section will demonstrate
distinct structural differences.

####  III-A2 Compression and decompression

To further reduce the required bandwidth for data transmission, each agent
needs to perform compression before data transmission. We use 1$\times$1
convolutional layer to compress the features in the channel direction, as
shown in formula (2):

$F_{i}^{s^{\prime}}=Enc_{com}(F_{i}^{s}),F_{i}^{s^{\prime}}\in\mathbb{R}^{\bar{H}\times\bar{W}\times\bar{\bar{C}}},\bar{\bar{C}}\ll\bar{C}$
(2) $Data_{i}\leftarrow(F_{i}^{s^{\prime}},P_{i})$ (3)

where $Enc_{Com}(\bullet)$ represents the compression function, formula (3)
represents the information transmission data packet, $Data_{i}$ transmits the
compressed feature $F_{i}^{s^{\prime}}$ and its position $P_{i}$, and
decompresses after other agents receive the compressed feature
$F_{i}^{s^{\prime}}$; the specific process is shown in formula (4):

$F_{i}^{s^{\prime\prime}}=Dec_{com}(F_{i}^{s^{\prime}}),F_{i}^{s^{\prime\prime}}\in\mathbb{R}^{\bar{H}\times\bar{W}\times\bar{C}}$
(4)

where $Dec_{Com}(\bullet)$ represents the decompression function corresponding
to the compression process, and the decoded feature space becomes
$\mathbb{R}^{\bar{H}\times\bar{W}\times\bar{C}}$. The decompressed feature
$F_{i}^{s"}$ will be transmitted to the feature fusion part.

####  III-A3 Feature fusion

The MSA mechanism model [36] fuses the decompressed features. The specific
procedure is shown in Figure 5. The feature vectors at the same position on
the feature map correspond to particular points in the original point cloud
data. Spatial correlation is destroyed by tiling the feature map and
calculating the weighted sum of each feature. However, although the complex
fusion structure objectively increases the network’s depth and thus improves
its ability to fit, this operation will hinder the accurate expression of
individual features. We use an MSA mechanism with a few parameters to more
accurately capture the features it represents and generate a fusion feature
map. The specific procedure is shown in formulas (5) and (6):

$m_{i}=M(P_{ego},P_{i})$ (5)

$\displaystyle M_{ego}=MSA(F_{ego}^{s},(F_{1}^{s^{\prime\prime}},m_{1}),$ (6)
$\displaystyle(F_{2}^{s^{\prime\prime}},m_{2}),\dots,(F_{n}^{s^{\prime\prime}},m_{n}))$

The formula (5) indicates that after receiving the data $Data_{i}$,
$M(\bullet)$ means to use its position information $P_{ego}$ and $P_{i}$ to
calculate the position transfer matrix $m_{i}$. After that, we send matrix
$m_{i}$ and the decompressed feature $F_{1}^{s"}$ to the fusion network
$MSA(\bullet)$, as shown in the formula (6). $MSA(F_{ego}^{s}$ is the feature
of its point cloud, and $M_{ego}$ represents the fusion feature. Finally, the
fusion features are sent to the detection head.

The fusion network structure is shown in Figure 5. After the features have
been connected, the MSA mechanism is used to extract features, where A
represents the number of multi-head attention heads. Formulas (7), (8), and
(9) represent the specific execution steps of the multi-head attention
mechanism, while formula (7) represents the connection between the multi-head
attention feature results; formula (8) represents the generation of a single
attention head, where $W_{i}^{Q}$, $W_{i}^{K}$, and $W_{i}^{V}$ represents the
relationship matrix of $Q$,$K$,$V$, $K$, and $V$ values between single
attention head $Attention(Q^{\prime},K^{\prime},V^{\prime})$ and multi-head
attention $Multihead(Q,K,V)$, respectively. Formula (9) represents the
operation process of single-head self-attention, where $dK^{\prime}$ indicates
the feature dimension.

The fusion network structure is shown in Figure 5. After the features are
connected, the multi-head self-attention mechanism is used for feature
extraction, where the parameter A represents the number of multi-head
attention heads. Formulas (7), (8), and (9) represent the specific execution
steps of the multi-head attention mechanism, and formula (7) represents the
connection of multi-head attention feature results; formula (8) represents the
generation of a single attention head, where $W_{i}^{Q}$, $W_{i}^{K}$,
$W_{i}^{V}$represents the relationship matrix of $Q$,$K$,$V$values between
single attention head $Attention(Q^{\prime},K^{\prime},V^{\prime})$ and multi-
head attention $Multihead(Q,K,V)$ , respectively. formula (9) represents the
operation process of single-head self-attention, where $dK^{\prime}$ means the
feature dimension.

$\begin{split}Multihead(Q,K,V)=Linear(Concat\\\
(head_{1},head_{2},\dots,head_{h}))\end{split}$ (7)

$head_{i}=Attention(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V})$ (8)

$Attention(Q^{\prime},K^{\prime},V^{\prime})=softmax(\frac{Q^{\prime}K^{\prime
T}}{\sqrt{dK^{\prime}}})V^{\prime}$ (9)

Figure 5: Feature fuse module multi-head attention (MSA) mechanism.

####  III-A4 Detection Head

After obtaining the final fusion feature $M_{ego}$, we use two 1$\times$1
convolutional layers to generate classification and regression prediction
results, respectively, and form a prediction box, as shown in formulas (10)
and (11):

$Y_{class}={\xi}_{class}(M_{ego})$ (10)

$Y_{reg}={\xi}_{reg}(M_{ego})$ (11)

where ${\xi}_{class}(\bullet)$ represents the classification layer, and
$Y_{class}$ outputs a score, which is used to indicate whether the preselected
box is an object or a background. ${\xi}_{reg}(\bullet)$ represents the
regression layer, and the output of $Y_{reg}$ is seven dimensions
$(x,y,z,w,l,h,\theta)$ , where $x$, $y$ , and $z$ represent the position of
the prediction frame, $w$, $l$, and $h$ represent the size of the prediction
frame, and $\theta$ represents the heading angle of the prediction box.

###  III-B Teacher network

####  III-B1 Multi-view data fusion

The input data of the teacher network are fused perspective data, which needs
to be fused before inputting into the model. The process of fused point cloud
is as follows:

$\begin{split}L_{mix}=A((L_{ego},P_{ego}),\\\
(L_{1},P_{1}),(L_{2},P_{2}),\dots,(L_{N},P_{N}))\end{split}$ (12)

where $L_{mix}$ represents the fused point cloud, $N$ indicates the number of
agents within the transmission range, and $A(\bullet)$ represents the
aggregation process of the surrounding point cloud data. To ensure that the
input data of the teacher network of the fusion view and the student network
of a single view are aligned, the coordinates are transformed into the
coordinates centered on agent $ego$ after the fusion point cloud is cropped.

####  III-B2 Feature extraction

After collecting the multi-view data, we feed it into the network for feature
extraction. The process of feature extraction for the teacher network is
similar to that for the student network, but the input data differ, as shown
in formula (13):

$F_{i}^{t}=\phi_{t}(L_{mix}),F_{i}^{t}\in\mathbb{R}^{\bar{H}\times\bar{W}\times\bar{C}}$
(13)

$\phi_{t}(\bullet)$ represents the feature extraction network of the teacher’s
point cloud, $F_{ego}^{t}$ indicates the feature extracted from the fusion
point cloud centered on agent $ego$, and
$R^{\bar{H}\times\bar{W}\times\bar{C}}$ represents the same feature space as
the student network.

####  III-B3 Knowledge distillation loss function

During the training process, the teacher network is more straightforward to
converge than the student network training; therefore, the teacher and student
networks can be jointly trained with random initialization parameters. In this
paper, we jointly train the model using object detection loss and distillation
loss. The loss ${\mathcal{L}}_{total}$ that needs to be minimized is shown in
formula (14):

${\mathcal{L}}_{total}={\lambda}_{det}{\mathcal{L}}_{det}+{\lambda}_{KD}{\mathcal{L}}_{KD}$
(14)

where hyperparameters ${\lambda}_{det}$ and ${\lambda}_{KD}$ control the
weights of object detection loss ${\mathcal{L}}_{det}$ and knowledge
distillation loss ${\mathcal{L}}_{KD}$, respectively. The target detection
loss is shown in formula (15), including classification loss
${\mathcal{L}}_{class}(\bullet)$ and regression loss
${\mathcal{L}}_{reg}(\bullet)$. Classification loss
${\mathcal{L}}_{class}(\bullet)$ use focal loss[38] to calculate
classification $Y_{class}(\bullet)$ and label classification value
$\hat{Y}_{class}$, which is used to judge whether the object in the detection
frame is background or target. Regression loss ${\mathcal{L}}_{reg}(\bullet)$
uses ${\ell}_{1}$ smooth loss to calculate regression value $Y_{reg}$ and
label regression value $\hat{Y}_{reg}$ , which are used to judge the detection
frame’s position, size, and heading angle.

${\mathcal{L}}_{det}={\mathcal{L}}_{reg}(Y_{reg},\hat{Y}_{reg})+{\mathcal{L}}_{class}(Y_{class},\hat{Y}_{class})$
(15)

The distillation loss ${\mathcal{L}}_{KD}(F_{ego}^{t},F_{ego}^{s})$ is shown
in formula (16), where $KL((p(x)||(q(x))$ represents the Kullback-Leibler(KL)
divergence, which is used to describe the difference between the distributions
$p(x)$ and $q(x)$. $V(\bullet)$ indicates retaining the feature channel
$\hat{c}$ and generating a one-dimensional feature (vector) from a feature of
$\hat{H}\times\hat{W}$ size. The $\tau(\bullet)$ function represents the
softmax operation process with the distillation temperature $T$ according to
the number of channels $\hat{c}$, as shown in formula (17).

${\mathcal{L}}_{KD}(F_{ego}^{t},F_{ego}^{s})=KL(\tau(V(F_{ego}^{t}))||\tau(V(F_{ego}^{s})))$
(16)

$\tau(\bullet)\leftarrow\frac{exp(z_{i}/T)}{\sum_{j}exp(z_{j}/T)}$ (17)

where $z_{i}$ represents the $i$-th channel’s characteristics, and the
hyperparameter $T$ represents the distillation temperature.

This section describes the network structure and direction of data flow for
each network component. Experiments will be conducted in the next section to
confirm the effectiveness of the structure described above.

##  IV Experiments

###  IV-A Dataset

To evaluate the performance of the proposed V2X-AHD algorithm, all experiments
use the open large-scale vehicle-road collaboration dataset V2XSet [14] .
CARLA [39] and OpenCDA [40] generate V2XSet to simulate natural scenes. The
data set consists of 11447 frames, and since each frame contains multiple
agents, the total number of samples is 33081. Regarding structure, the
training, verification, and test sets are 6694/1820/2833 frames, respectively.

TABLE I: Detailed architectural specifications for V2X-AHD. name | outputsize | Framework
---|---|---
Teacher-Conv1 | [32,192,704] | [SpConv3$\times$3,32$\to$32,stride=1,BN,ReLU]$\times$2
Teacher-Conv2 | [64,96,352] | [SpConv3$\times$3,32$\to$64,stride=1,BN,ReLU]
|  | [SpConv 3$\times$3,32$\to$64,stride=1,BN,ReLU]$\times$2
Teacher-Conv3 | [128,48,176] | [SpConv3$\times$3,64$\to$128,stride=1,BN,ReLU]
|  | [SpConv 3$\times$3,128$\to$128,stride=1,BN,ReLU]$\times$2
Teacher-Conv4 | [256,24,88] | [SpConv3$\times$3,128$\to$256,stride=1,BN,ReLU]
|  | [SpConv 3$\times$3,256$\to$256,stride=1,BN,ReLU]$\times$2
Teacher-Conv5 | [256,12,44] | [Conv3$\times$3,256$\to$256,stride=1,BN,ReLU]
|  | [Conv 3$\times$3,256$\to$256,stride=1,BN,ReLU]$\times$2
Student-Conv1 | [32,192,704] | [SpConv3$\times$3,32$\to$32,stride=1,BN,ReLU]$\times$2
Student-Conv2 | [64,96,352] | [SpConv3$\times$3,32$\to$64,stride=1,BN,ReLU]
|  | [SpConv 3$\times$3,32$\to$64,stride=1,BN,ReLU]
Student-Conv3 | [128,48,176] | [SpConv3$\times$3,64$\to$128,stride=1,BN,ReLU]
|  | [SpConv 3$\times$3,128$\to$128,stride=1,BN,ReLU]
Student-Conv4 | [256,24,88] | [SpConv3$\times$3,128$\to$256,stride=1,BN,ReLU]
|  | [SpConv 3$\times$3,256$\to$256,stride=1,BN,ReLU]
Student-Conv5 | [256,12,44] | [Conv3$\times$3,256$\to$256,stride=1,BN,ReLU]
|  | [Conv3$\times$3,256$\to$256,stride=1,BN,ReLU]
Neck-DeConv1 | [128,24,88] | [ConvT2$\times$2,256$\to$128,stride=2,BN,ReLU]
|  | [Conv3$\times$3,128$\to$128,stride=1,BN,ReLU]
Neck-DeConv2 | [384,48,176] | [ConvT2$\times$2,256$\to$256,stride=2,BN,ReLU]
|  | [Conv3$\times$3,256$\to$384,stride=1,BN,ReLU]
MSA | [256,48,176] | [Linear,384$\to 256$]
|  | [Multi head:3,256$\to 256$]
|  | [Linear,256$\to 256$]
Detect heads | [16,48,176] |  ${\xi}_{class}$[Conv1$\times$1,256$\to$2,stride 1]
|  |  ${\xi}_{reg}$[Conv1$\times$1,256$\to$14,stride 1]

###  IV-B Settings

As an evaluation metric, we employ the general 3D object detection standard:
the average precision results when the Intersection-over-Union (IoU) threshold
is 0.5 and 0.7. In the experiment, the communication distance between any two
agents is 70 m, agents beyond this distance are ignored, and the compression
rate of transmitted data is uniformly set to 32. The x- and y-axis measurement
ranges are [-140.8, 140.8] and [-38.4, 38.4], respectively, while the z-axis
measurement range is [-1,3]. A single voxel’s length, width, and height are
set to 0.4, 0.4, and 4, respectively. During the training process, all
comparison methods employ the Adam [41] optimizer for 60 training rounds; the
initial learning rate is set to 0.001, and it is decreased to the original
value of 0.1 every 20 training rounds. All models are trained on a device
equipped with an NVIDIA GeForce RTX 3090 graphics card and an AMD 5900x CPU. I
displays the network and parameter information for the distillation network.

###  IV-C Comparison methods

Early Fusion, which aggregates the original point clouds of surrounding
agents; Late Fusion, which obtains the perception results of all surrounding
agents and employs the non-maximal suppression method to produce the final
result; This experiment compares intermediate fusion strategies by contrasting
the most recent five algorithms: F-cooper, V2VNet, DiscoNet, OPV2V, and
V2X-ViT. For a fair comparison, the same method of feature extraction is used
during evaluation. As the backbone for testing, they are based on the
PointPillar [32] and SparePillar methods proposed in this paper.

TABLE II: Detection performance comparison on V2Xset.We show average precision (AP) at IoU=0.5,0.7 on Point Pillar and Spare Pillar Backbone, respectively. Methods | Source | Point Pillar AP $\uparrow$ | Spare Pillar AP $\uparrow$
---|---|---|---
IoU=0.5 | IoU=0.7 | IoU=0.5 | IoU=0.7
Early Fusion |  | 0.877 | 0.791 | 0.924 | 0.836
Late Fusion |  | 0.727 | 0.620 | 0.755 | 0.658
No Fusion |  | 0.606 | 0.402 | 0.651 | 0.573
F-Cooper[15] | 2019 SEC | 0.840 | 0.680 | 0.872 | 0.737
V2VNet[16] | 2020 ECCV | 0.845 | 0.677 | 0.832 | 0.673
DiscoNet[29] | 2021 NeurIPS | 0.844 | 0.695 | 0.857 | 0.726
OPV2V[17] | 2022 ICRA | 0.807 | 0.664 | 0.864 | 0.754
V2X-ViT[14] | 2022 ECCV | 0.882 | 0.710 | 0.854 | 0.742
V2X-AHD | OURS | 0.855 | 0.724 | 0.877 | 0.770

###  IV-D Quantitative analysis

In this section, we present the results of our V2X-AHD method and compare it
to the literature.

####  IV-D1 Main methods results

The experimental results of existing 3D object detection algorithms are
compared in II. At the same time, the Spare Pillar module proposed in this
paper is a plug-and-play module. The test results indicate that the point
pillar and the spare pillar should serve as the network’s backbone. The
experimental results demonstrate that the intermediate strategy fusion methods
have better results than the no-fusion and late-fusion algorithms. The early
fusion strategy algorithm can receive complete original point cloud data, and
the test results are better than the intermediate fusion strategy algorithm.
Under IoU=0.7, the proposed V2X-AHD algorithm achieves the best performance.
Compared with the optimal algorithm V2X-ViT algorithm, using the Point Pillar
and Spare Pillar backbone networks, the accuracy is increased by 1.6% and
2.2%, respectively. The perception accuracy of the Spare Pillar module is
superior to that of the Point Pillar module. When IoU=0.7, the results of
Fcooper, DiscoNet, OPV2V, and V2X-ViT methods increased by 8.4%, 4.5%, 13.1%,
and 4.5% respectively.

Table 2 shows the comparative experimental results of existing 3D object
detection algorithms. At the same time, the Spare Pillar module proposed in
this paper is a plug-and-play module. The test results show using the point
pillar and the spare pillar as the backbone network. The experimental results
show that the intermediate strategy fusion methods have better results than
the no-fusion and late-fusion algorithms. The early fusion strategy algorithm
can receive complete original point cloud data, and the test results are
better than the intermediate fusion strategy algorithm. Under IoU=0.7, the
V2X-AHD algorithm proposed in this paper achieves the best results. Compared
with the optimal algorithm V2X-ViT algorithm, using the Point Pillar and Spare
Pillar backbone networks, the accuracy is increased by 1.6% and 2.2%,
respectively. The Spare Pillar module has higher perception accuracy than the
Point Pillar module. In the case of IoU=0.7, the results of Fcooper, DiscoNet,
OPV2V, and V2X-ViT methods increased by 8.4%, 4.5%, 13.1%, and 4.5%
respectively.

Figure 6: Model parameter quantity and accuracy relationship graph. Figure 7:
Qualitative comparison in different conditions. Green and red bounding boxes
represent ground truth and prediction, respectively. In addition, we use
circles to illustrate the key of the figure.

Performance is determined jointly by the number of model parameters and the
accuracy of prediction. Figure 6 illustrates the relationship between the
parameter amounts of various algorithms and their precision. This article
utilizes the knowledge distillation architecture, so there are two training
and testing algorithm parameters. The figure demonstrates that this paper
achieves the best possible result with the fewest number of test parameters.
Combined with the quantitative analysis results, we discovered that the
F-Cooper and OPV2V algorithms with small parameters significantly enhanced
detection performance after the Spare Pillar module was replaced due to their
simple fusion strategy. However, the V2X-ViT algorithm with many parameters
has a slight improvement due to the increased complexity of the fusion
strategy. The V2VNet with the greatest number of parameters even experienced
negative growth. The preceding results demonstrate that the high-complexity
fusion model will hinder the expression of features. The lightweight MSA
feature fusion module presented in this paper is better suited for fusing
single-view vehicle data features.

####  IV-D2 Distillation temperature comparison

The knowledge distillation structure employs soft training targets. Compared
with manually labeled hard target labels, soft target labels contain more
entropy, thereby increasing the difference between various features and
allowing the student network to obtain more valuable information from the
teacher network. Distillation temperature is a vital knowledge distillation
hyperparameter, and Table 3 shows the AP results of the model at different
distillation temperatures. When the temperature of distillation is low,
detection precision increases as the temperature rises. The detection accuracy
reached the highest value when the distillation temperature was ten, and then
with the increase of distillation temperature, the detection accuracy showed a
downward trend. According to the analysis, the inter-class differences of soft
objects become smaller as distillation temperature increases, and the
characteristics cannot be learned. Finally, we selected the optimum
temperature as 10.

TABLE III: Comparison of AP at different distillation temperatures. Temperature | AP(IoU=0.5)) | AP(IoU=0.7)
---|---|---
1 | 0.853 | 0.738
2 | 0.860 | 0.740
5 | 0.865 | 0.748
10 | 0.876 | 0.770
15 | 0.876 | 0.763
20 | 0.877 | 0.759

###  IV-E Qualitative analysis

####  IV-E1 Object detection result

Figure 7 depicts the results of the detection process. We select three
representative scenarios and compare the F-cooper, OPV2V, and V2X-viT
algorithms’ results. The green and red bounding boxes represent the ground
truth and predicted results, respectively. Even when compared to the SOTA
algorithm V2X-ViT, our algorithm has a higher degree of matching. The scene in
Fig. 7(a) shows the false detection problem. All the algorithms recognize the
vehicles in the scene. However, except for our algorithm, the other algorithms
all identify some trees as vehicles, as shown in the red circle. It can be
seen that the plug-in SparePillar has a more robust feature extraction
performance than PointPillar. Figure 7 (b) depicts the occlusion detection
results. The F-cooper algorithm and the OPV2V algorithm have a significant
deviation of the detection bounding box angle or even occlusion-related missed
detection. As shown by the red circle, the V2X-ViT algorithm has the issue of
failed point cloud vehicle self-detection. However, the fused point cloud at
this location has the shape of the vehicle. According to the analysis, the
intermediate fusion algorithm cannot accurately synthesize the contradictory
results. Additionally, it demonstrates that the complex fusion strategy will
impede the correct expression of the extracted features. Figure 7(c) shows the
scene with multiple intersections and agents. At the edge positions, such as
the left and bottom sides of the scene, F-cooper, OPV2V, and V2X-ViT have
undetected situations, as shown by the red circle marks. In areas where the
point cloud is sparse, as indicated by the green circle, detection is also
missed by all algorithms. Although there are actual bounding boxes, there are
only a few point clouds, no effective vehicle features can be formed, and the
failure to detect is a reasonable phenomenon.

Figure 8: Attention weight transfer. Scene1. Brighter parts of the image indicate higher attention to that area. Each Scene includes three agents: SDV1, SDV2, and, Infra, the SDV1 is the feature receiver. TABLE IV: Ablation experimental results of different components in the v2xset dataset. Baseline | SparePillar | AHD | MSA | AP(IoU=0.5)) | AP(IoU=0.7)
---|---|---|---|---|---
$\checkmark$ |  |  |  | 0.807 | 0.664
$\checkmark$ | $\checkmark$ |  |  | 0.864 | 0.754
$\checkmark$ |  | $\checkmark$ |  | 0.858 | 0.736
$\checkmark$ | $\checkmark$ | $\checkmark$ |  | 0.872 | 0.761
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.877 | 0.770

####  IV-E2 Multi-view feature fusion

Figure 8 shows the aggregation process between fused view data and visualizes
the attention weights. Infra represents roadside infrastructure, while SDV1
and SDV2 represent self-driving vehicles. With SDV1 functioning as the data
receiver, Infra and SVD2 transmit data to SDV1. The point cloud map and the
attention map correspond; the brighter the color, the more attention the area
has received. In Figure 8 (a), because SDV2 blocks Car1, SDV1 pays less
attention to the position of Car1 and has a darker color, while the
corresponding position of SDV2 has a darker color to the position of car1.
However, car 2 is located at an intersection, the detection distance to SDV1
is relatively great, and there is occlusion. SDV1 pays little attention to it,
whereas the roadside equipment at the corresponding location demonstrates high
attention. In Figure 8 (b), SDV1 is blocked by the intersection, resulting in
a restricted field of view. The highlighted area of the SDV1 attention map is
limited to some intersections, while areas 1 and 2 are not focused due to
occlusion. Infra and SDV2 in the area1 and area2 positions with higher
attention. The attention-based fusion strategy MSA proposed in this paper can
effectively synthesize the extracted point cloud features and ultimately
achieve the objectives of extending the detection range of targets and
controlling occlusion.

###  IV-F Ablation experiments

When IoU=0.7, the single module SparePillar, and AHD improve AP by 13.5% and
10.8%, respectively. The results indicate that SparePillar and the asymmetric
distillation architecture can significantly improve perceptual accuracy. The
MSA accuracy increased by 1.2%, indicating that the fusion module is capable
of enhancing the feature expression under higher perceptual accuracy than the
self-attention mechanism. Under IoU=0.5 and IoU=0.7, the method presented in
this paper is 8.7% and 16.0% more efficient than the baseline method. Since a
single view has acquired multi-view features, it has achieved a greater
improvement under the IoU=0.7 condition, which necessitates greater precision.
The above ablation experimental results demonstrate the effectiveness of each
component.

##  V Conclusion

To solve the problem of low vehicle recognition accuracy caused by the unclear
outline of the vehicle in the conventional single-vehicle point cloud target
detection method. This paper proposes a multi-view vehicle-road collaborative
perception framework V2X-AHD based on a distillation network. Compared with
the conventional algorithm, the asymmetric heterogeneous distillation network
AHD proposed in this paper can transfer the entire point cloud features of
multiple views to a single view, improving object contour perception accuracy.
The extractor Spare Pillar based on sparse convolution design improves the
ability to extract features from point cloud data while simultaneously
reducing the number of parameters. The feature fuser MSA performs feature
layer fusion on multi-view feature data. This paper verifies the proposed
method on the publicly available dataset V2XSet, achieving the best results,
and demonstrates the effectiveness of the proposed method V2X-AHD through
quantitative and qualitative analysis.

The proposed method has only been validated in a simulated environment, and
the difference between vehicle-side and road-side data is negligible. In
future work, we will collect real scene point cloud data, construct our own
real scene data set, and examine the differences between vehicle and roadside
point cloud data.

## References

  * [1] H.Wang, Y Chen, Y Cai, et al. ”SFNet-N: An improved SFNet algorithm for semantic segmentation of low-light autonomous driving road scenes.” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 11, pp.21405-21417, 2022.
  * [2] Y. Cai, T. Luan, H. Gao, H. Wang, L. Chen, Y. Li, M. A. Sotelo, and Z. Li, ”YOLOv4-5D: An effective and efficient object detector for autonomous driving.” IEEE Transactions on Instrumentation and Measurement, vol. 70, pp. 1-13, 2021.
  * [3] A. Lombard, A. Noubli, A. Abbas-Turki, et al. ”Deep Reinforcement Learning Approach for V2X Managed Intersections of Connected Vehicles”. IEEE Transactions on Intelligent Transportation Systems, 2023\.
  * [4] X. Chen , S. Leng, J. He, et al.”The Upper Bounds of Cellular Vehicle-to-Vehicle Communication Latency for Platoon-Based Autonomous Driving”. IEEE Transactions on Intelligent Transportation Systems, 2023\.
  * [5] L. Zhao, Y. Song, C. Zhang, et al. ”T-gcn: A temporal graph convolutional network for traffic prediction.” IEEE Transactions on Intelligent Transportation Systems, vol.21, no.9, pp.3848-3858, 2019.
  * [6] W. Chen ,J. Zhao, W. L. Zhao, et al. ”Shape-Aware Monocular 3D Object Detection.” IEEE Transactions on Intelligent Transportation Systems, 2023\.
  * [7] Y. Kim, S. Kim, S. Sim, et al. ”Boosting Monocular 3D Object Detection With Object-Centric Auxiliary Depth Supervision.” IEEE Transactions on Intelligent Transportation Systems, vol.24, no. 2, pp. 1801-1813, 2022.
  * [8] C. He, H. Zeng, J. Huang, et al. ”Structure aware single-stage 3d object detection from point cloud.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11873-11882, 2020.
  * [9] Y. Yan, Y. Mao, B. Li. ”Second: Sparsely embedded convolutional detection.” Sensors, vol. 18, no. 10, pp. 3337, 2018.
  * [10] W. Zheng, W. Tang, S. Chen, et al. ”Cia-ssd: Confident iou-aware single-stage object detector from point cloud.” Proceedings of the AAAI conference on artificial intelligence, vol. 35, no. 4, pp. 3555-3562, 2021.
  * [11] H. Kuang, B. Wang, J. An, et al. ”Voxel-FPN: Multi-scale voxel feature aggregation for 3D object detection from LIDAR point clouds.” Sensors, vol. 20, no.3, pp. 704,2020.
  * [12] C. R. Qi, H. Su, K. Mo, et al. ”Pointnet: Deep learning on point sets for 3d classification and segmentation.” Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660,2017.
  * [13] Q. Chen, S, Tang, Q, Yang, et al. ”Cooper: Cooperative perception for connected autonomous vehicles based on 3d point clouds.” 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS). IEEE,pp. 514-524,2019.
  * [14] R. Xu, H, Xiang, Z. Tu, et al. ”V2x-vit: Vehicle-to-everything cooperative perception with vision transformer.” European conference on computer vision, pp.107-124,2022.
  * [15] Q. Chen, X. Ma, S. Tang, et al. ”F-cooper: Feature based cooperative perception for autonomous vehicle edge computing system using 3D point clouds.” Proceedings of the 4th ACM/IEEE Symposium on Edge Computing, pp. 88-100, 2019.
  * [16] T. H. Wang, S. Manivasagam, M. Liang, et al. ”V2vnet: Vehicle-to-vehicle communication for joint perception and prediction.” Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK,pp. 605-621, 2020.
  * [17] R. Xu, H. Xiang, X. Xia, et al. ”Opv2v: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication.” 2022 International Conference on Robotics and Automation (ICRA). IEEE,pp. 2583-2589, 2022.
  * [18] A. Rauch, F. Klanner, R. Rasshofer, et al. ”Car2x-based perception in a high-level fusion architecture for cooperative perception systems.” 2012 IEEE Intelligent Vehicles Symposium. IEEE, pp. 270-275, 2012.
  * [19] Z. Y. Rawashdeh, Z. Wang. ”Collaborative automated driving: A machine learning-based method to enhance the accuracy of shared information.” 2018 21st International Conference on Intelligent Transportation Systems (ITSC). IEEE, pp. 3961-3966, 2018.
  * [20] T. Luo, L. Chen, T. Luan, et al. ”A Vehicle-to-Infrastructure beyond Visual Range Cooperative Perception Method Based on Heterogeneous Sensors.” Energies, vol.15 no.21 pp: 7956, 2022.
  * [21] R. Anil,G. Pereyra, A. Passos, et al. ”Large scale distributed neural network training through online distillation.”  arXiv preprint, arXiv:1804.03235, 2018.
  * [22] G. Hinton, O. Vinyals, J. Dean.” Distilling the knowledge in a neural network.” arXiv preprint, arXiv:1503.02531,2015.
  * [23] A. Romero, N. Ballas,S. Kahou, et al. ”Fitnets: Hints for thin deep nets.”  arXiv preprint, arXiv:1412.6550, 2014.
  * [24] G. Shi, R. Li, C. Ma. ”PillarNet: High-Performance Pillar-based 3D Object Detection.”  arXiv preprint, arXiv:2205.07403, 2022.
  * [25] B. Graham, V.D. Maaten, L.: ”Submanifold sparse convolutional networks”. arXiv preprint, arXiv:1706.01307,2017.
  * [26] Y. Liu, K. Chen, C. Liu, et al. ”Structured knowledge distillation for semantic segmentation.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2604-2613, 2019.
  * [27] Y. Wang, A. Fathi, J. Wu, et al. ”Multi-frame to single-frame: Knowledge distillation for 3d object detection.” arXiv preprint, arXiv:2009.11859, 2020.
  * [28] X. Jin, C. Lan, W. Zeng, et al. ”Uncertainty-aware multi-shot knowledge distillation for image-based object re-identification.” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34 no. 07, pp. 11165-11172, 2020.
  * [29] Y. Li, S. Ren, P. Wu, et al. ”Learning distilled collaboration graph for multi-agent perception.” Advances in Neural Information Processing Systems, vol. 34, pp. 29541-29552,2021.
  * [30] Y. Hu, Z. Ding, R. Ge, W. Shao , L. Huang, K. Li, Q. Liu ”Afdetv2: Rethinking the necessity of the second stage for object detection from point clouds.” In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 1, pp. 969-979, 2022.
  * [31] Y. Zhou, O. Tuzel. ”Voxelnet: End-to-end learning for point cloud based 3d object detection.” Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4490-4499, 2018.
  * [32] A. H. Lang, S. Vora, H. Caesar, et al. ”Pointpillars: Fast encoders for object detection from point clouds.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12697-12705, 2019.
  * [33] M. Ye, S. Xu, T. Cao. ”Hvnet: Hybrid voxel network for lidar based 3d object detection.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1631-1640, 2020.
  * [34] J. Noh, S. Lee, B. Ham. ”Hvpr: Hybrid voxel-point representation for single-stage 3d object detection.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.14605-14614, 2021.
  * [35] I. Goodfellow, D. Warde-Farley, M. Mirza, et al. ”Maxout networks.” International conference on machine learning. PMLR, pp. 1319-1327, 2013.
  * [36] A. Vaswani, N. Shazeer, N. Parmar, et al. ”Attention is all you need.” Advances in neural information processing systems, vol 30, 2017, 30.
  * [37] M. Schlichtkrull, T. N. Kipf, P. Bloem, et al. ”Modeling relational data with graph convolutional networks.” The Semantic Web: 15th International Conference, ESWC 2018, pp. 593-607, 2018.
  * [38] K. Simonyan, A. Zisserman . ”Very deep convolutional networks for large-scale image recognition.” arXiv preprint, arXiv:1409.1556, 2014.
  * [39] A. Dosovitskiy, G. Ros, F. Codevilla, et al. ”CARLA: An open urban driving simulator.” Conference on robot learning. PMLR, pp 1-16, 2017.
  * [40] R.Xu, Y. Guo, X. Han, et al. ”OpenCDA: an open cooperative driving automation framework integrated with co-simulation.” 2021 IEEE International Intelligent Transportation Systems Conference (ITSC). IEEE,pp. 1155-1162, 2021.
  * [41] D. P. Kingma, J.Ba. ”Adam: A method for stochastic optimization.” arXiv preprint, arXiv:1412.6980, 2014.
  * [42] T. Y. Lin, P. Goyal, R.Girshick, et al. ”Focal loss for dense object detection.” Proceedings of the IEEE international conference on computer vision,pp. 2980-2988, 2017.