# Is high-dimensional photonic entanglement robust to noise?

Feng Zhu  School of Engineering and Physical Sciences, Heriot-Watt University,
Edinburgh, EH14 4AS, UK  Max Tyler  School of Engineering and Physical
Sciences, Heriot-Watt University, Edinburgh, EH14 4AS, UK  Natalia Herrera
Valencia  School of Engineering and Physical Sciences, Heriot-Watt University,
Edinburgh, EH14 4AS, UK  Mehul Malik  School of Engineering and Physical
Sciences, Heriot-Watt University, Edinburgh, EH14 4AS, UK  Institute for
Quantum Optics and Quantum Information (IQOQI), Austrian Academy of Sciences,
Vienna, Austria  Jonathan Leach  School of Engineering and Physical Sciences,
Heriot-Watt University, Edinburgh, EH14 4AS, UK

###### Abstract

High-dimensional entangled states are of significant interest in quantum
science as they increase the information content per photon and can remain
entangled in the presence of significant noise. We develop the analytical
theory and show experimentally that the noise tolerance of high-dimensional
entanglement can be significantly increased by modest increases to the size of
the Hilbert space. This work is developed in the context of spatial
entanglement, but it can easily be translated to photonic states entangled in
different degrees of freedom. We also demonstrate that knowledge of a single
parameter, the signal-to-noise ratio, precisely links measures of entanglement
to a range of experimental parameters quantifying noise in a quantum
communication system, enabling accurate predictions of its performance. This
work serves to answer a simple question: “Is high-dimensional photonic
entaglement robust to noise?”. However, despite the simplicity of the
question, we show that the answer is more nuanced than a simple “yes” or “no”.

##  I Introduction

Entanglement is considered one of the most important features of quantum
information science and plays a central role in many quantum communication
protocols Einstein  _et al._ (1935); Horodecki  _et al._ (2009); Reid  _et
al._ (2009). The strong, non-classical correlations inherent to entanglement
allow one to share information between two parties that is secure against the
most sophisticated eavesdropping attacks, with information security even
provided independent of the devices used Acín  _et al._ (2007). Entangled
photons produced via parametric nonlinear processes are a workhorse for many
branches of quantum information science, ranging from fundamental tests of
quantum mechanics to entanglement-based quantum communication Dada  _et al._
(2011); Giovannetti  _et al._ (2011); Fickler  _et al._ (2013); Bornman  _et
al._ (2019); Ringbauer  _et al._ (2015); Proietti  _et al._ (2020). However,
photonic entanglement is extremely susceptible to common sources of noise such
as channel loss, background counts, multi-photon effects, and imperfect
measurement devices, which can make its detection challenging.

In recent years, high-dimensional entanglement has emerged as a way to
increase the robustness of entanglement, and as a result, increase the
resistance of entanglement-based quantum communication to noise Collins  _et
al._ (2002); Cerf  _et al._ (2002); Thew  _et al._ (2004); Wiseman  _et al._
(2007); Vértesi  _et al._ (2010), see Refs. Krenn  _et al._ (2017); Erhard
_et al._ (2018) for an overview.  These studies follow the work in Ref.
Kaszlikowski  _et al._ (2000) that showed that violations of local realism are
stronger for high-dimensional systems. In addition, encoding information in
photonic high-dimensional degrees-of-freedom such as transverse
position/momentum and time/frequency offers an increased information bandwidth
over qubit encoding, i.e. greater than one bit of information per transmitted
photon. This allows one to build efficient quantum networks and quantum
cryptography systems that use the full information-carrying capacity of a
photon Jha  _et al._ (2008); Leach  _et al._ (2012a); Wang  _et al._ (2018);
Bouchard  _et al._ (2018); Valencia  _et al._ (2019).

Several recent works on photonic high-dimensional entanglement have
demonstrated a qualitative advantage over qubit entanglement in terms of
information-capacity and noise-robustness Ecker  _et al._ (2019); Bavaresco
_et al._ (2018). However, a careful analysis outlining the precise noise
conditions and Hilbert space dimension where such an advantage can or cannot
be found is still lacking. Here, we formulate these conditions in terms of
easily measurable experimental parameters, and establish the exact noise
bounds and Hilbert space dimension where high-dimensional entanglement
provides an advantage for entanglement certification in the presence of noise,
and where it does not. We experimentally verify the predictions in the context
of spatial entanglement generated in the few-photon limit, but the results are
applicable to degrees of freedom where high-dimensional entanglement exists,
e.g. time-bin entanglement. The results are also relevant to hyper and hybrid
entanglement.

We follow a two-step approach to developing an operational noise model for
photonic high-dimensional entanglement: First, we show how the seemingly
complex relationship between the sources of noise in the state, channel, and
detection system can be distilled into one operational quantity—the signal-to-
noise ratio $\mathcal{Q}$, which we refer to as the quantum contrast. The
quantity $\mathcal{Q}$ is simply the ratio of the coincidence counts to the
accidental counts and can readily be established in any experiment. An
analysis of the functional form of $\mathcal{Q}$ that we present will allow an
experimenter to optimise their source, channel, and detector specifications in
order to achieve the best noise performance (highest $\mathcal{Q}$) possible.
Second, we test a series of contemporary entanglement measures and analyse
their performance in the presence of noise. In all cases, we find that the
performance of the system, as quantified by the ability to certify (high-
dimensional) entanglement, can be accurately predicted by knowledge of the
system dimension $d$ and only one experimentally measurable parameter:
$\mathcal{Q}$.

For high-dimensional photonic entanglement, our results show that separating
the dimension in which one wishes to observe the entanglement and the
dimension of the Hilbert space allows one to tolerate significantly more noise
in an entanglement distribution system. However, depending on the entanglement
certification method used, there is an optimum dimension where one finds the
best possible noise performance. Interestingly, the largest increase in noise
tolerance is obtained for only a modest increase in Hilbert space dimension,
allowing the use of inefficient detectors or operation in extremely noisy
environments.

##  II Theoretical model

###  II.1 The relationship between $\mathcal{Q}$ and $p$ from the isotropic
state

An often-used model for describing noise in an entangled state combines a
maximally entangled state $|\phi\rangle$ with a maximally mixed state
$\hat{\mathds{I}}$ denoting white noise:

$\displaystyle\hat{\rho}=p|\phi\rangle\langle\phi|+\frac{(1-p)}{d^{2}}\hat{\mathds{I}}.$
(1)

This is often referred to as the isotropic state. Here, $p$ refers to the
probability that the state remains intact Werner (1989). One sees here that
the resistance to noise will increase linearly with system dimension $d$—the
threshold $p$ required to certify any entanglement is equal to $1/(d+1)$
Collins  _et al._ (2002). As the system dimension is increased, one can expect
to overcome any amount of noise. For example, the threshold $p$ in $d=2$ is
$1/3$, whereas the threshold $p$ in $d=10$ reduces to only $1/11$. However, in
any realistic scenario, $p$ turns out to be a dimension-dependent parameter
that involves a complex interplay between noise attributed to the state, the
channel, and the detection system.

Our work focuses on the signal-to-noise ratio $\mathcal{Q}$ (defined in the
next section), rather than $p$, but we can relate these two parameters to each
other. This is important as we can establish the link between theory and
experiment is a simple fashion. We can relate the $p$ from the isotropic state
to $\mathcal{Q}$ and dimension $d$ through

$\displaystyle p(d,\mathcal{Q})=\frac{\mathcal{Q}-1}{\mathcal{Q}-1+d}.$ (2)

Solving this for $\mathcal{Q}$, we see that for a fixed value of $p$ and as
$d$ increases, it is necessary to increase $\mathcal{Q}$. For example, to
achieve $p=1/3$ in $d=2$ requires that that the signal to noise equal
$\mathcal{Q}=2$, whereas to achieve $p=1/3$ in $d=10$ requires that
$\mathcal{Q}=6$. This is illustrated in Fig. 1 where we see the relationship
between $d$, $\mathcal{Q}$, and $p$.

Figure 1: The relationship between $\mathcal{Q}$, $d$ and $p$ from the
isotropic state. The white lines indicate constant values of $p$. We see that
maintaining a constant value of $p$ as $d$ increases requires an increase to
the signal-to-noise ratio $\mathcal{Q}$.

###  II.2 Definition of $\mathcal{Q}$

We introduce a formalism that takes into account the common sources of noise
in photonic systems: photon-pair generation probability in spontaneous
parametric downconversion (SPDC)  $\mu$ Christ  _et al._ (2011); Yuen (1976);
Takeoka  _et al._ (2015); Alibart  _et al._ (2006); Wasilewski  _et al._
(2008), imperfect measurements with dark counts from the detector  and
background counts from ambient light sources rolled into a term  $n$ Hadfield
(2009); Rohde and Ralph (2006); Takesue  _et al._ (2007), and  a term that
accounts for any loss or non-unity collection efficiency $\eta$.  The system
we consider uses single-photon detectors without the ability to perform
photon-number detection. Adding photon-number resolving detection would enable
the detection of entanglement with more multi-photon pairs, but in this work
we consider the most widely used case for two-photon entanglement.

We consider the measurement interval to be the time that a detector is active,
and the quantities $\mu$ and $n$ are then normalised with respect to this time
window. In the case that $\mu\ll 1$, higher order multi-photon terms do not
significantly contribute, and $\mu$ can be considered as the probability of
generating a photon pair per detection event. The value $n$ is the probability
of detecting any noise event in the time window. In general, $n$ and $\eta$
can be dimension-dependent, however, in our theoretical formalism we treat
them as being independent as this represents the best-case scenario for noise-
resistance.

The state that we consider is then

$\displaystyle|\phi\rangle=\prod_{j=1}^{d}\left[\sum_{m=0}^{+\infty}\left(\frac{\mu}{\mu+1}\right)^{\frac{m}{2}}\frac{\hat{a}_{1,j}^{{\dagger}{m}}\hat{a}_{2,j}^{{\dagger}{m}}}{m!\sqrt{1+\mu}}\right]|{\rm{vac}}\rangle,$
(3)

where $\hat{a}_{1,j}^{{\dagger}{m}}$ is the creation operator for $m$ photons
in mode $j$. Details of the derivation of Eq. (3) are given at the end of the
paper. We assume that the efficiencies and noise levels in each channel are
the same, so the  probability of two-photon coincidences and accidentals  per
time window is given by
$\rm{p_{coi}^{(j,k)}}=\eta^{2}\delta_{j,k}{\mu}(1+{\mu})+(n+\eta{\mu})^{2}$,
where $j$ and $k$ are indices for the different modes of the photons, and
$1\leq j,k\leq d$, with $d$ equal to the dimension of the Hilbert space. We
define the signal-to-noise ratio or the quantum contrast $\mathcal{Q}$ as the
ratio of coincidences ${\rm p_{coi}^{(j,j)}}$ to accidentals  ${\rm
p_{coi}^{(j,k\neq j)}}$

$\displaystyle\mathcal{Q}=1+\frac{\mu(1+\mu)}{(\frac{n}{\eta}+{\mu})^{2}}.$
(4)

The quantum contrast can be optimised by an appropriate choice of the pair
generation rate $\mu$ based on the noise parameters $\eta$ and $n$, and
achieves a maximum value of $\mathcal{Q}_{\rm max}=1+\eta^{2}/[4n(\eta-n)]$,
when the pair generation rate is set to $\mu=n/(\eta-2n)$, see Fig. 2.

Figure 2: The signal-to-noise ratio $\mathcal{Q}$ as a function of the noise-
efficiency ratio $n/\eta$ and the photon pair generation rate $\mu$. Given a
noise-efficiency ratio, there is an optimal generation rate that maximises the
signal to noise.

###  II.3 Assumptions of the model

There are two important assumptions in our analytic theory: first, the
dimension of the state Eq. (3) can be increased without any penalty; and
second, the amplitudes of the mode coefficients are equal, i.e. the bandwidth
of the state is considered “flat”. These assumptions greatly simplify the
theoretical description and provide an upper bound to noise resistance. In any
physical system, however, there will be an upper limit to the size of the
space, and real sources often require entanglement concentration in order to
achieve a flat state Dada  _et al._ (2011).

To rigorously test the validity of assumptions, we address the problem of a
finite-dimensional and non-flat state with numerical simulations
representative of high-dimensional states reported in the literature.
Quantitative comparisons between the predictions of the theory, which includes
the assumptions, and a wide range of physical states can then be made. In
addition, a non-flat state may have a varying quantum contrast for different
mode. However, this is something that we can investigate in the numerical
simulations, and we find that the predictions of the theory still hold when we
take the average quantum contrast across the relevant set of measurements.

Figure 3: (a). The required quantum contrast $\mathcal{Q}$ vs. dimension $d$
for the two-MUB witness. The dotted line represents the necessary
$\mathcal{Q}$ for $k=d$ dimensional entanglement. The dashed line represents
the optimal dimension for measuring $k-$dimensional entanglement. (b).
Required quantum contrast $\mathcal{Q}$ vs. dimension $d$ for various values
of $k$ for the $d+1$-MUB witness. The contrast necessary for $d$-dimensional
entanglement in a $d$-dimensional space is indicated by the dotted line and is
equal to $d^{2}-d$; the contrast necessary for $k$-dimensional entanglement in
an infinite space is equal to $k$, indicated in the bottom right-hand side of
the figure.

###  II.4 Entanglement verification via two mutually unbiased bases

Recently, it was shown that measurements in two mutually unbiased bases (MUBs)
are sufficient to lower bound the fidelity of a state and certify high-
dimensional entanglement Bavaresco  _et al._ (2018). If the fidelity
$F(\rho,\Phi)$ of the state $\rho$ with respect to a $d$-dimensional target
state $\Phi$ is greater than $(k-1)/d$, $\rho$ is entangled in at least $k$
dimensions, i.e. it has Schmidt rank of at least $k$. As we consider a state
with a uniform modal distribution, the quantum contrast is basis-independent.
The two-MUB witness can then be used to lower bound the achievable fidelity
$\tilde{F}$ of a $d$-dimensional state with quantum contrast $Q$:

$\displaystyle\tilde{F}(\rho,\Phi)\geq\frac{\mathcal{Q}-d+1}{\mathcal{Q}+d-1}.$
(5)

Consequently, in order to certify $k$-dimensional entanglement, the quantum
contrast must satisfy

$\displaystyle\mathcal{Q}>$ $\displaystyle\frac{(d-1)(d+k-1)}{d-k+1}.$ (6)

We see that as $k$ is increased, the minimum contrast that is required
increases. Importantly, as shown in Fig. 3a, we see that in order to certify
$k$-dimensional entanglement, a minimum quantum contrast of $\mathcal{Q_{\rm
opt}}=3k+2\sqrt{2}\sqrt{(k-2)(k-1)}-4$ is necessary, which is obtained when
the dimension of the Hilbert space we are working in is set to $d_{\rm
opt}=[\sqrt{2}\sqrt{k^{2}-3k+2}+k-1]$.

We see that $d_{\rm opt}$ is reached by increasing $k$ by approximately
$(1+\sqrt{2})\approx 2.41$ times, at which we have significantly increased our
system’s ability to tolerate noise. The gain can be quantified as the ratio of
$\mathcal{Q}$ when $d=k$ and when $d=d_{\rm opt}$. This ratio is approximately
$(2/(3+2\sqrt{2}))k\approx 0.343k$, so if $k=1000$, increasing $d$ by 2.41
times reduces $\mathcal{Q}$ by 343. From an experimental perspective, this
decrease in $\mathcal{Q}$ permits the use of a detector with a significantly
lower detection efficiency/channel loss, or a higher dark/background count
rate.

###  II.5 Example for an EMCCD camera

As an example of the gains this provides, consider a multi-outcome detector,
such as a single-photon detector array or EMCCD camera Edgar  _et al._ (2012);
Defienne  _et al._ (2018). These detectors are commonly used for measurements
of spatial entanglement as they allow multi-outcome measurements in two MUBs,
i.e. the position and momentum bases. Certifying $k=1000$ entanglement in a
$d=1000$ space using two MUBs requires a signal-to-noise of approximately
$2\times 10^{6}$. Achieving this signal-to-noise requires, for example, a
noise per detection event of $n=1\times 10^{-7}$ and an efficiency of
$\eta=80\%$. Increasing the dimension of the space by a factor 2.41, (e.g.
moving from a 32 by 32 array of pixels to a 49 by 49 grid), reduces the
necessary $\mathcal{Q}$ by $\approx$343 times, and, for example, the allowable
detector noise can increase two orders of magnitude to $n=1\times 10^{-5}$
while the efficiency simultaneously drops to $23\%$.

###  II.6 Entanglement verification via all mutually unbiased bases

Measurements in all $d+1$ MUBs allow one to calculate the exact fidelity while
also providing better noise performance Bavaresco  _et al._ (2018). In this
case, the fidelity is given by

$\displaystyle\tilde{F}(\rho,\Phi)=\frac{\mathcal{Q}+\frac{1}{d}-1}{\mathcal{Q}+d-1}.$
(7)

Then, for a $d$-dimensional system with quantum contrast $Q$, one can certify
an entanglement dimensionality of

$\displaystyle k<\frac{(d+1)\mathcal{Q}}{d+\mathcal{Q}-1}.$ (8)

We see here that knowledge of $\mathcal{Q}$ provides an upper limit to the
number of entangled dimensions. In any experiment an estimate of $\mathcal{Q}$
can be established easily, and therefore, $k$ can be predicted in a fast and
accurate manner.

As in the two-MUB case, entanglement certification in increasing $k=d$
dimensions necessitates an accompanying increase in the required quantum
contrast (Fig. 3b). However, significant reductions in the required quantum
contrast are achieved when we allow the dimension of the Hilbert space to
increase with respect to the dimension of the entanglement, i.e. if $k<d$. In
the limit that $d\rightarrow\infty$, we see that the number of entangled
dimensions $k\rightarrow\mathcal{Q}$. It follows that if we measure a system
with a contrast of $\mathcal{Q}$, it can have at most
$\lfloor\mathcal{Q}\rfloor$ dimensions of entanglement. Consequently, the
minimum contrast required for verification of $k$ dimensions of entanglement
is equal to $k$.

##  III Numerical and Experimental Results

|  Measured data from Ref. Bavaresco  _et al._ (2018) |  Predictions based on $\mathcal{Q}_{\rm exp}$ |  Optimal conditions
---|---|---|---
$d$ |  $\tilde{F}_{{\rm exp}}$ |  $k_{\rm{exp}}$ |  $\mathcal{Q}_{\rm exp}$ |  $\tilde{F}_{\rm pred}$ |  $k_{\rm pred}$ |  $d_{\rm opt}$ |  $\mathcal{Q}_{\rm opt}$
3 |  91.5$\pm 0.4\%$ |  3 |  71$\pm$1 |  94.5$\pm 0.1$% |  3 |  4 |  9.0
5 |  89.9$\pm 0.4\%$ |  5 |  70$\pm$5 |  89.2$\pm 0.7$% |  5 |  9 |  20.8
7 |  84.2$\pm 0.5\%$ |  6 |  68$\pm$4 |  83.9$\pm 0.9$% |  6 |  14 |  32.5
11 |  74.8$\pm 0.4\%$ |  9 |  81$\pm$2 |  78.1$\pm 0.5$% |  9 |  23 |  55.8
Table 1: Comparison of the results from Ref. Bavaresco  _et al._ (2018)
against the predictions of this work. The data from Bavaresco  _et al._ (2018)
was evaluated to calculate an average signal-to-noise ratio for the two MUBs
in each of the dimensions. These $\mathcal{Q}_{\rm exp}$ values are then used
to predict the fidelity $\tilde{F}_{\rm pred}$ and dimension $k_{\rm pred}$ of
the entanglement. For every dimension $d$, the experimentally observed $k_{\rm
exp}$ and the predicted $k_{\rm pred}$ are equal to each other, thus
confirming that the system performance can accurately be predicted with
knowledge of $\mathcal{Q}$.

###  III.1 Analysis of data from Ref. Bavaresco  _et al._ (2018)

In order to verify that knowledge of the signal-to-noise ratio provides an
accurate estimate of system performance when using two MUBS, we re-analyzed
the data presented in Ref. Bavaresco  _et al._ (2018) to calculate
$\mathcal{Q}_{\rm exp}$ for the measurements in each of the MUBs and used this
to predict the system performance, see Table 1. In each dimension ($d$ = 3, 5,
7, and 11), we see that knowledge of $\mathcal{Q}_{\rm exp}$ gives an estimate
of ($k_{\rm pred}$ = 3, 5, 6, and 9) respectively, which are the exact values
of dimension obtained. This confirms that knowledge of a single, easily
obtainable, experimentally measured parameter $\mathcal{Q_{\rm exp}}$ provides
a very accurate prediction of system performance.

###  III.2 Experimental verification

Figure 4:  (a) Schematic of experiment to generate and measure high-
dimensional entanglement in the pixel basis. BBO = Beta barium borate; IF =
interference filter; SLM = spatial light modulator; SMF = single-mode fibre.
(b) Illustration of the modes on the SLM used to detect high-dimensional
entanglement. The left half of the SLM is used to detect the signal (s); the
right half is used to detect the idler (i). Each small box represents a mode
in the computational MUB. (c) In seven dimensions, there are eight MUBS, and
MUB 0 is the computational basis. The signal-to-noise ratio $\mathcal{Q}$ is
assumed to be constant across all MUBs in the theory and we calculate the
average $\mathcal{Q}$ across the MUBs in the experiment.

To investigate the predictions of our theory, we perform measurements on the
high-dimensional entangled state produced by SPDC, see Fig. 4(a). The
measurements are performed using the spatial degree of freedom in all MUBs in
3, 5, and 7 dimensions with average quantum contrasts ranging from
$\mathcal{Q}=5$ to 40. $\mathcal{Q}$ is varied by adjusting the laser pump
power and ambient lighting conditions in the lab and is calculated as the
average of the signal-to-noise ratios of each of the MUBs.

All the measurements are conducted in the “pixel” bases Valencia  _et al._
(2018), where the computational basis uses the standard anti-correlations in
photon momenta that are are observed in the far-field of SPDC. Fig. 4(b)
indicates the intensity distribution of the far-field of the SPDC together
with the areas integrated for the measurement modes. We use one SLM, where the
left half used for the signal photon, and the right half is used for the idler
photon. For the computational basis measurements (MUB 0), a single area of the
signal and idler is active at any one time; for all the other MUB
measurements, all elements are active all times with the phase of each area
appropriately controlled. The coincidence matrices for all the MUBS, and thus
the joint probabilities, are measured by scanning through the appropriate
signal and idler modes on the SLM.

The data are analysed to calculate the lower bound to the fidelity and thus a
value of $k$. Figure 5a shows experimental evidence of two of the results of
this work: first, that knowledge of $\mathcal{Q}$ and $d$ provides an accurate
prediction of system performance with respect to entanglement measures; and
second, that increasing the operational dimension allows one to tolerate
larger amounts of noise for the same (or larger) certified entanglement
dimensionality. The measured values of $k$ plotted in Fig. 5 are observed to
be very close to, but not above, the upper bound predictions of Eq. 8. We
observe that entanglement in larger dimensions ($k$ increases) can be achieved
by increasing $d$ while tolerating a lower signal-to-noise ($\mathcal{Q}$
drops).

Figure 6 shows the variation of the measured signal to noise ratios for
different MUBs in different dimensions. We see that the $\mathcal{Q}$ value
for the computational MUB (MUB 0) is consistently higher than the values for
the other MUBs, which are observed to be very close to each other. We believe
that this is a systematic error introduced by the SLM-based measurement
method, rather than anything inherently in the generated state. Despite this
systematic error, the predicted performance of system with regards to all
entanglement metrics still performs exceptionally well when using the average
value of $\mathcal{Q}$ across all the different MUBs.

The the signal-to-noise values for each MUB are calculated from the
appropriate coincidence probability matrix, Fig. 5(b). For one MUB, it is
calculated as the average of the diagonal elements of the matrix divided by
the average of the off-diagonal elements of the matrix. We chose this approach
to measure the most accurate value of $\mathcal{Q}$, however, a very fast
estimate of $\mathcal{Q}$ can be obtained by measuring the coincidence rates
for only one diagonal and one off-diagonal element of any of the coincidence
matrices. The predicted system performance when using this fast method will
depend on any error between the estimated $\mathcal{Q}$ and the true value.

Figure 5:  (a) Experimental certification of $k$-dimensional entanglement for
signal-to-noise ratios ranging from $\mathcal{Q}=5$ to 40 for $d=3,5,$ and
$7$. The dotted lines are the analytical theory, the solid points are the
measured values of $k$. The two grey arrows represent the lowering of
$\mathcal{Q}$ from 32 to 12 for 5-dimensional entanglement and 23 to 6 for
3-dimensional entanglement. In both cases, provided that $d$ is increased,
high-dimensional entanglement remains even if the signal to noise falls. (b)
The measurement matrices showing the joint measurement probabilities for three
sample data sets.  Figure 6: The measured signal-to-noise ratios for different
MUBs in different dimensions. This is a selection of the data used that is
typical of the observed trends. This shows that MUB 0 has a higher
$\mathcal{Q}$ value than the other MUBs, which are all measured to be close to
each other. The MUB average is the value that is used for the predictions of
system performance.

###  III.3 Numerical verification

Our theory considers a “flat” bandwidth of the state as this assumption
considerably simplifies the theory. To take into account the effects of a
reduced bandwidth, we perform numerical simulations on a finite bandwidth
state. The width of the state is controlled by setting the probability
amplitude of the single photons to have a Gaussian distribution. We use
$\sigma=2,4,10$ and 100000. A necessary consequence of a limited bandwidth is
there exists crosstalk in MUBs other than the computational basis. These
simulations therefore enable us to investigate realistic quantum sources that
exhibit common types of measurement error. We introduce noise into the
simulated data and then calculate an average $\mathcal{Q}$ value over all the
MUBs.

Figure 7 shows the results for numerical simulations of $k$-dimensional
entanglement certification for a state with a finite number of entangled
modes. Despite the finite widths, the two main results of our work are still
observed: the average $\mathcal{Q}$ is a very good indicator of system
performance, and increasing the size of the space ($d$) enables the
certification of entanglement in larger dimensions with lower signal-to-noise.
The difference from the analytical theory, which provides an upper bound, is
that the size of the space cannot be increased indefinitely. One finds that
there is an upper limit to the advantage of increasing $d$. The
$\sigma=100000$ result, which represents a state with a very large bandwidth,
is a very close match to the analytical theory.

##  IV Additional Entanglement Measures

###  IV.1 Conditional Entropy

We also analyse the noise dependance of conditional entropies commonly used in
the confirmation of EPR steering Reid  _et al._ (2009); Leach  _et al._
(2010); Schneeloch  _et al._ (2013). EPR  steering can be confirmed if the
measurements in two MUBs violate the inequality
$H_{1}(X|Y)+H_{2}(X|Y)\geq\log_{2}d$, where $H_{1}(X|Y)$ and $H_{2}(X|Y)$ are
the conditional entropies in each of the bases Leach  _et al._ (2012b). After
we consider the sum of the conditional entropies  in two bases, we find that

$\displaystyle\log_{2}(\mathcal{Q}+d-1)-\frac{\mathcal{Q}}{\mathcal{Q}+d-1}\log_{2}\mathcal{Q}-\frac{1}{2}\log_{2}d<0.$
(9)

There is no analytical solution to this, but numerical solutions show that the
necessary quantum contrast $\mathcal{Q}$ required to demonstrate EPR steering
increases linearly as a function of Hilbert space dimension $d$, see dotted
line in Fig. 8. Experimental data for a range of dimensions showing whether
the steering criterion is violated (red) or not (black) is also plotted as a
function of quantum contrast. Clear agreement is seen with the predictions of
Eq. 9, which demonstrates that knowledge of the signal-to-noise ratio
$\mathcal{Q}$ accurately predicts whether the EPR steering criterion will be
violated.

Figure 7: Numerical simulation of average quantum contrast vs. entangled
dimension for a range of finite-width states. Each point represents one
numerical simulation, and the dotted lines are the analytical thoery.
Increasing the size of the space $d$ continues to provide noise resistance,
but only up to certain point governed by the width of the state. These
numerical simulations were performed by setting the width of the probability
amplitude of the single photon to have a Gaussian envelope of width $\sigma$
for a wide range of signal to noise values. (a), (c), (e) and (g) all show the
average $\mathcal{Q}$ vs. $k$ for $\sigma$ = 2, 4, 10 and 100000,
respectively. (b), (d), (f) and (h) are samples of the coincidence matrices
for $\sigma$ = 2, 4, 10, and 100000 respectively, showing the finite width of
the state and the influence this has on cross-talk measurements in the MUBs.
The $\sigma$ = 100000 case is a close approximation to the state that we
consider in the analytical theory, and we recover the same results for this
case. Figure 8:  Certification of EPR steering in high dimensions for a range
of signal-to-noise ratios. Measurements in two MUBs for a range of signal-to-
noise ratios were taken and the data tested against the EPR steering criteria.
The data are plotted as either red or black points depending on whether the
EPR criteria is violated or not. The dotted line is the boundary that is
calculated by solving Eq. 9. The derived bound, based only on $\mathcal{Q}$
and $d$, accurately separates the experimentally measured data.

###  IV.2 High-dimensional non-locality based on the CGLMP inequality

Finally, we consider the CGLMP inequality Collins  _et al._ (2002); Dada  _et
al._ (2011) that can be used for establishing non-locality. Local hidden
variable theories are consistent with $S_{d}\leq 2$, whereas quantum mechanics
permits a violation of this inequality. In their work, CGLMP showed that
maximally entangled quantum systems in high dimensions can achieve a
theoretical value of $S_{d}(QM)$, which would lead to a violation of the
inequality Collins  _et al._ (2002). However, in any experimental
verification, the achievable $S_{d}$ is modified by imperfections in the
system. Under the fair-sampling assumption, the maximum value of $S_{d}$ given
by

$\displaystyle\tilde{S}_{d}=$
$\displaystyle\frac{\mathcal{Q}-1}{\mathcal{Q}-1+d}S_{d}(QM).$ (10)

Therefore, in order to violate the local hidden variable inequality, the
number of dimensions must satisfy

$\displaystyle d<\frac{(\mathcal{Q}-1)(S_{d}(QM)-2)}{2}.$ (11)

For large values of $d$, $S_{d}(QM)\approx 3$, and the upper bound for $d$ can
be approximated by $d<(\mathcal{Q}-1)/2$. Solving for $\mathcal{Q}$, we see
that to violate the CGLMP inequality in $d$ dimensions, we must have a signal-
to-noise ratio greater than $2d+1$, e.g. violating the inequality in 10
dimensions requires that $\mathcal{Q}$ exceed $\approx$21.

As in the case of the fidelity witness, the separation of $k$ and $d$ may
provide a noise advantage in the form of a decreased signal-to-noise
threshold. It will also be interesting to consider the implications of this
work for other Bell-like inequalities, such as the one developed for maximally
entangled states Salavrakos  _et al._ (2017).

##  V Discussion and conclusion

This work serves to answer a simple question: “Are photonic high-dimensional
entangled states robust to noise?” In reality, the answer to this question is
more nuanced than a simple “yes” or “no”, and our results provide a clear
demonstration of the advantages and disadvantages. When we consider the
maximally entangled state, we find that the total noise that the state can
tolerate increases as the dimension of the state increases, and therefore, it
is obvious to claim that such states are robust to noise. However, a counter
argument can be put forward when we recognize that the necessary experimental
signal-to-noise tends to increase as we increase the dimension of the state in
which we wish to observe entanglement or non-locality.

Significant benefits for entanglement certification emerge when we see that
the dimension of the state in which we wish to establish entanglement $k$ and
the dimension of the operational Hilbert space $d$ are not required to be the
same. The signal-to-noise requirements for $k$-dimensional entanglement
decrease as both $d$ and the total number of mutually unbiased bases used in
the measurement increases.

By distilling the information about the performance of a photonic high-
dimensional entanglement system into a single operational parameter, we
provide a powerful tool that links theory and experiment, allowing us to
accurately predict system performance and choose an optimum entanglement
measurement strategy. This work demonstrates that high-dimensional
entanglement has the potential to push the boundaries of entanglement-based
quantum communication systems, bringing them from the confines of laboratory
proof-of-principle demonstrations to the realm of practical, real-world
implementations under extreme conditions of noise.

##  VI Acknowledgments

We thank M. Huber and N. Friis for helpful discussions. M.M. and N.H.
acknowledge support from the QuantERA ERA-NET Co-fund (FWF Project I3773-N36)
and the UK Engineering and Physical Sciences Research Council (EPSRC)
(EP/P024114/1). J.L. and F.Z. acknowledge support from the UK EPSRC
(EP/M01326X/1).

## References

  * Einstein  _et al._ (1935) A. Einstein, B. Podolsky, and N. Rosen, Physical Review 47, 777 (1935).
  * Horodecki  _et al._ (2009) R. Horodecki, P. Horodecki, M. Horodecki, and K. Horodecki, Rev. Mod. Phys. 81, 865 (2009).
  * Reid  _et al._ (2009) M. D. Reid, P. D. Drummond, W. P. Bowen, E. G. Cavalcanti, P. K. Lam, H. A. Bachor, U. L. Andersen, and G. Leuchs, Rev. Mod. Phys. 81, 1727 (2009).
  * Acín  _et al._ (2007) A. Acín, N. Brunner, N. Gisin, S. Massar, S. Pironio, and V. Scarani, Phys. Rev. Lett. 98, 230501 (2007).
  * Dada  _et al._ (2011) A. C. Dada, J. Leach, G. S. Buller, M. J. Padgett, and E. Andersson, Nature Physics 7, 677 (2011).
  * Giovannetti  _et al._ (2011) V. Giovannetti, S. Lloyd, and L. Maccone, Nature Photonics 5, 222 (2011).
  * Fickler  _et al._ (2013) R. Fickler, M. Krenn, R. Lapkiewicz, S. Ramelow, and A. Zeilinger, Scientific Reports 3, 1914 (2013).
  * Bornman  _et al._ (2019) N. Bornman, M. Agnew, F. Zhu, A. Vallés, A. Forbes, and J. Leach, npj Quantum Information 5, 63 (2019).
  * Ringbauer  _et al._ (2015) M. Ringbauer, B. Duffus, C. Branciard, E. G. Cavalcanti, A. G. White, and A. Fedrizzi, Nature Physics 11, 249 (2015).
  * Proietti  _et al._ (2020) M. Proietti, J. Ho, F. Grasselli, P. Barrow, M. Malik, and A. Fedrizzi, arXiv , 2002.01491 (2020), 2002.01491 .
  * Collins  _et al._ (2002) D. Collins, N. Gisin, N. Linden, S. Massar, and S. Popescu, Phys. Rev. Lett. 88, 040404 (2002).
  * Cerf  _et al._ (2002) N. J. Cerf, M. Bourennane, A. Karlsson, and N. Gisin, Phys. Rev. Lett. 88, 127902 (2002).
  * Thew  _et al._ (2004) R. T. Thew, A. Acín, H. Zbinden, and N. Gisin, Phys. Rev. Lett. 93, 010503 (2004).
  * Wiseman  _et al._ (2007) H. M. Wiseman, S. J. Jones, and A. C. Doherty, Phys. Rev. Lett. 98, 140402 (2007).
  * Vértesi  _et al._ (2010) T. Vértesi, S. Pironio, and N. Brunner, Phys. Rev. Lett. 104, 060401 (2010).
  * Krenn  _et al._ (2017) M. Krenn, M. Malik, M. Erhard, and A. Zeilinger, Phil. Trans. R. Soc. A 375, 20150442 (2017).
  * Erhard  _et al._ (2018) M. Erhard, R. Fickler, M. Krenn, and A. Zeilinger, Light: Science & Applications 7, 17146 (2018).
  * Kaszlikowski  _et al._ (2000) D. Kaszlikowski, P. Gnaciński, M. Żukowski, W. Miklaszewski, and A. Zeilinger, Phys. Rev. Lett. 85, 4418 (2000).
  * Jha  _et al._ (2008) A. K. Jha, M. Malik, and R. W. Boyd, Phys. Rev. Lett. 101, 180405 (2008).
  * Leach  _et al._ (2012a) J. Leach, E. Bolduc, D. J. Gauthier, and R. W. Boyd, Phys. Rev. A 85, 060304 (2012a).
  * Wang  _et al._ (2018) J. Wang, S. Paesani, Y. Ding, R. Santagati, P. Skrzypczyk, A. Salavrakos, J. Tura, R. Augusiak, L. Mančinska, D. Bacco, _et al._ , Science 360, 285 (2018).
  * Bouchard  _et al._ (2018) F. Bouchard, K. Heshami, D. England, R. Fickler, R. W. Boyd, B.-G. Englert, L. L. Sánchez-Soto, and E. Karimi, Quantum 2, 111 (2018).
  * Valencia  _et al._ (2019) N. H. Valencia, S. Goel, W. McCutcheon, H. Defienne, and M. Malik, arXiv , 1910.04490 (2019), 1910.04490 .
  * Ecker  _et al._ (2019) S. Ecker, F. Bouchard, L. Bulla, F. Brandt, O. Kohout, F. Steinlechner, R. Fickler, M. Malik, Y. Guryanova, R. Ursin, and M. Huber, Phys. Rev. X 9, 041042 (2019).
  * Bavaresco  _et al._ (2018) J. Bavaresco, N. H. Valencia, C. Klöckl, M. Pivoluska, P. Erker, N. Friis, M. Malik, and M. Huber, Nature Physics 14, 1032 (2018).
  * Werner (1989) R. F. Werner, Physical Review A 40, 4277 (1989).
  * Christ  _et al._ (2011) A. Christ, K. Laiho, A. Eckstein, K. N. Cassemiro, and C. Silberhorn, New Journal of Physics 13, 033027 (2011).
  * Yuen (1976) H. P. Yuen, Physical Review A 13, 2226 (1976).
  * Takeoka  _et al._ (2015) M. Takeoka, R.-B. Jin, and M. Sasaki, New Journal of Physics 17, 043030 (2015).
  * Alibart  _et al._ (2006) O. Alibart, J. Fulconis, G. Wong, S. Murdoch, W. Wadsworth, and J. Rarity, New Journal of Physics 8, 67 (2006).
  * Wasilewski  _et al._ (2008) W. Wasilewski, C. Radzewicz, R. Frankowski, and K. Banaszek, Phys. Rev. A 78, 033831 (2008).
  * Hadfield (2009) R. H. Hadfield, Nature Photonics 3, 696 (2009).
  * Rohde and Ralph (2006) P. P. Rohde and T. C. Ralph, Journal of Modern Optics 53, 1589 (2006).
  * Takesue  _et al._ (2007) H. Takesue, S. W. Nam, Q. Zhang, R. H. Hadfield, T. Honjo, K. Tamaki, and Y. Yamamoto, Nature Photonics 1, 343 (2007).
  * Edgar  _et al._ (2012) M. P. Edgar, D. S. Tasca, F. Izdebski, R. E. Warburton, J. Leach, M. Agnew, G. S. Buller, R. W. Boyd, and M. J. Padgett, Nature Communications 3, 984 (2012).
  * Defienne  _et al._ (2018) H. Defienne, M. Reichert, and J. W. Fleischer, Physical review letters 121, 233601 (2018).
  * Valencia  _et al._ (2018) N. Valencia, V. Srivastav, M. Pivoluska, M. Huber, N. Friis,, W. McCutcheon, and M Malik, https://arxiv.org/abs/2004.04994 (2020).
  * Leach  _et al._ (2010) J. Leach, B. Jack, J. Romero, A. K. Jha, A. M. Yao, S. Franke-Arnold, D. G. Ireland, R. W. Boyd, S. M. Barnett, and M. J. Padgett, Science 329, 662 (2010).
  * Schneeloch  _et al._ (2013) J. Schneeloch, C. J. Broadbent, S. P. Walborn, E. G. Cavalcanti, and J. C. Howell, Phys. Rev. A 87, 062103 (2013).
  * Leach  _et al._ (2012b) J. Leach, E. Bolduc, D. J. Gauthier, and R. W. Boyd, Phys. Rev. A 85, 060304 (2012b).
  * Salavrakos  _et al._ (2017) A. Salavrakos, R. Augusiak, J. Tura, P. Wittek, A. Acín, and S. Pironio, Phys. Rev. Lett. 119, 040402 (2017).

###  VI.1 Generation of high-dimensional entanglement through nonlinear
optical processing

The Hamiltonian of the electromagnetic field in the interaction picture in
nonlinear optical media is given by

$\displaystyle\hat{\rm{H}}^{\prime}=$ $\displaystyle
e^{i\frac{\hat{H}_{0}}{\hbar}t}\hat{\rm{H}}_{\rm{NL}}e^{-i\frac{\hat{H}_{0}}{\hbar}t}$
$\displaystyle=$
$\displaystyle\sum_{j_{1},j_{2},j_{3}}\hat{a}_{j_{1}}^{{\dagger}}\hat{a}_{j_{2}}^{{\dagger}}\hat{a}_{j_{3}}{\rm{sinc}}[\frac{1}{2}\bm{L}\cdot(\bm{k}_{j_{1}}+\bm{k}_{j_{2}}-\bm{k}_{j_{3}})]$
$\displaystyle{e}^{i(\omega_{1}+\omega_{2}-\omega_{3})t}\Gamma_{j_{1},j_{2},j_{3}}+{\rm{H.c.}}$
(12)

$\hat{a}_{j_{1}}^{{\dagger}}\hat{a}_{j_{2}}^{{\dagger}}$ indicates photon-pair
generation events; $H.c$ is the Hermitian conjugate; $\hat{a}_{j_{3}}$
represents the annihilation of a pump photon, showing that photon pair
generation is proportional to the pump power; and $|\bm{L}|$ is the thickness
of the crystal. $\Gamma_{j_{1},j_{2},j_{3}}$ is a function associated with
$\bm{\chi}^{(2)}$, $|\bm{L}|$, $\hbar$, $\epsilon_{j}$, and the effective
cross-section area of the optical modes.

Furthermore, if $\omega_{1}+\omega_{2}-\omega_{3}{\neq}0$, the integral of
${e}^{i(\omega_{1}+\omega_{2}-\omega_{3})t}$ in the time domain will vanish,
ensuring energy conservation. In addition, the
${\rm{sinc}}\\{\frac{1}{2}\bm{L}\cdot[\bm{k}_{j_{1}}+\bm{k}_{j_{2}}-\bm{k}_{j_{3}}]\\}$
term is from integration in the spatial degrees of freedom, ensuring momentum
conservation. Based on these two conditions, there is only one independent
variable $j$, so that we have

$\displaystyle\hat{\rm{H}}^{\prime}=$
$\displaystyle\sum_{j=1}^{d}\gamma_{j}\hat{a}_{1,j}^{{\dagger}}\hat{a}_{2,j}^{{\dagger}}\hat{a}_{3}+{\rm{H.c.}}$
(13)

where $d$ is the dimension of entanglement modes, $\gamma_{j}$ is the
reduction of $\Gamma_{j_{1},j_{2},j_{3}}$ in Eq. ($12$),
$\hat{a}_{1,j}^{{\dagger}}$, $\hat{a}_{2,j}^{{\dagger}}$ is the creation
operators for signal and idler photons, respectively, and $\hat{a}_{3}$
represents the annihilation operator of a pump photon. Usually the pump light
includes a very large number of photons. Hence, the order of $\hat{a}_{3}$ and
$\hat{a}_{3}^{{\dagger}}$ is no longer important. Then $\hat{a}_{3}$ and
$\hat{a}_{3}^{{\dagger}}$ can be replaced by $\alpha_{p}$ and
$\alpha_{p}^{*}$. The final state is

$\displaystyle|\phi\rangle=$
$\displaystyle\exp\\{\sum_{j=1}^{d}\frac{i}{\hbar}t\gamma_{j}\alpha_{p}\hat{a}_{1,j}^{{\dagger}}\hat{a}_{2,j}^{{\dagger}}-{\rm{H.c.}}\\}|{\rm{vac}}\rangle$
$\displaystyle=$
$\displaystyle\prod_{j=1}^{d}\exp\\{\frac{i}{\hbar}t\gamma_{j}\alpha_{p}\hat{a}_{1,j}^{{\dagger}}\hat{a}_{2,j}^{{\dagger}}-{\rm{H.c.}}\\}|{\rm{vac}}\rangle$
$\displaystyle=$
$\displaystyle\prod_{j=1}^{d}[\sqrt{1-|f_{j}|^{2}}\sum_{m=0}^{+\infty}\frac{1}{m!}f_{j}^{k}\hat{a}_{1,j}^{{\dagger}{m}}\hat{a}_{2,j}^{{\dagger}{m}}]|{\rm{vac}}\rangle$
$\displaystyle=$
$\displaystyle\prod_{j=1}^{d}\left[\sum_{m=0}^{+\infty}\left(\frac{\mu}{\mu+1}\right)^{\frac{m}{2}}\frac{\hat{a}_{1,j}^{{\dagger}{m}}\hat{a}_{2,j}^{{\dagger}{m}}}{m!\sqrt{1+\mu}}\right]|{\rm{vac}}\rangle$
(14)

where $t$ is time, $f_{j}$ is associated with $\hbar$, $t$, $\gamma_{j}$ and
$\alpha_{p}$, showing the intensity of correlated photon pairs in each mode.
$|{\rm{vac}}\rangle$ indicates the initial state which is the vacuum state for
signal and idler photons.

The maximally entanglement state is widely used, which requires
$f_{j}={\rm{constant}}$. If all $f_{j}$ are not equal, entanglement
concentration can be used. We can replace all $f_{j}$ with a constant so that

$\displaystyle|f_{j}|^{2}=$ $\displaystyle\frac{{\mu}}{{\mu}+1}$ (15)

where $d$ is the dimension of entanglement, and ${\mu}$ probability of
generating a photon pair per detection event.


# Channel Boosting Feature Ensemble for Radar-based Object Detection

Shoaib Azam, Farzeen Munir and Moongu Jeon  School of Electrical Engineering
and Computer Science
Gwangju Institute of Science and Technology Gwangju, South Korea
Email: (shoaibazam, farzeen.munir<EMAIL_ADDRESS>
###### Abstract

Autonomous vehicles are conceived to provide safe and secure services by
validating the safety standards as indicated by SOTIF-ISO/PAS-21448 (Safety of
the intended
functionality)111https://www.daimler.com/innovation/case/autonomous/safety-
first-for-automated-driving-2.htm. Keeping in this context, the perception of
the environment plays an instrumental role in conjunction with localization,
planning and control modules. As a pivotal algorithm in the perception stack,
object detection provides extensive insights into the autonomous vehicle’s
surroundings. Camera and Lidar are extensively utilized for object detection
among different sensor modalities, but these exteroceptive sensors have
limitations in resolution and adverse weather conditions. In this work, radar-
based object detection is explored provides a counterpart sensor modality to
be deployed and used in adverse weather conditions. The radar gives complex
data; for this purpose, a channel boosting feature ensemble method with
transformer encoder-decoder network is proposed. The object detection task
using radar is formulated as a set prediction problem and evaluated on the
publicly available dataset[1] in both good and good-bad weather conditions.
The proposed method’s efficacy is extensively evaluated using the COCO
evaluation metric, and the best-proposed model surpasses its state-of-the-art
counterpart method by $12.55\%$ and $12.48\%$ in both good and good-bad
weather conditions.

##  I Introduction

The research and technological advancements made in the past three decades
have enabled autonomous vehicles to become a certainty [2]. The development of
sensors and computing technology has resulted in small size and cost-effective
hardware for the autonomous vehicle. The sensor’s accuracy, resolution, and
latency have improved over the years to be consolidated in autonomous
vehicles. The dynamic perception of the environment is fundamental to the
safety of the autonomous vehicle. A broad set of exteroceptive sensors is
utilized in the autonomous vehicle’s hardware suite to perceive the
environment. The typical sensor modalities include the camera, Lidar, radar,
and sonar help the autonomous vehicle attempt to map, understand, and navigate
the environment [3]. Among other perception tasks, the autonomous vehicle’s
most fundamental task is to identify and understand the objects encompassing
it.

The most common exteroceptive sensor modality that is widely deployed for the
perception of autonomous vehicles is cameras. Cameras operate in the visible
spectrum and provide high-resolution information about the environment, which
are essential to identify traffic lights, lane marking, and road obstacles.
Besides providing in-depth details of the surroundings, cameras are sensitive
to adverse weather conditions, illumination, and sun-glare, resulting in low-
level image data that impediment the perception’s accuracy for the autonomous
vehicles. On the other hand, Lidar provides the $3$D information about the
environment as a surrogate to $2$D details from the cameras. Lidar uses
invisible laser light to measure the accurate distance of the objects. It
measures nearly thousands of points to develop a $3$D point cloud map of the
senor’s surroundings. Despite providing the $3$D information, Lidar has a
detriment of being expensive and also sensitive to adverse weather conditions.
Contrary to cameras and Lidar, radar is less prone to adverse weather
conditions and is used as a sensor modality for detecting small hazard objects
in the autonomous vehicle sensor suite. Most of the radar operates on the
principle of Doppler’s effect by firing radio waves at the target area and
analyzing the reflected wave’s frequency to estimate the object’s speed and
position. These type of radars mostly find their application in object
tracking for the autonomous vehicle. Besides, there are scanning radars that
provide the high resolution $360\deg$ range-azimuth images and are used for
object detection in the perception stack of autonomous vehicles. Moreover,
they are relatively cost-effective as compared to Lidar [4].

Object detection forms the basis for the perception module and is formally
done by using cameras and Lidar. A lot of research is carried out on the
camera-based object detection that includes many state-of-the-art object
detectors, for instance, YOLO-v3 [5], SSD [6], Faster-RCNN [7], RefineDet [8],
M2Det [9]. These object detectors have limitation to operate in the adverse
weather conditions [10]. In addition, much research is also focused on the use
of Lidar for object detection. For this purpose, the Lidar data is represented
either in rasterized [11] or geometric format [12] [13] but have limitation to
processing computation and adverse weather condition. Besides, camera and
Lidar object detection, there is still a room of improvement for the radar-
based object detection. In this work, radar-based object detection in
different weather condition is explored without utilizing any auxiliary sensor
modality information. The radar-based object detection is formulated as a set
prediction problem by designing a proposed network that includes channel
boosting feature ensemble followed by transformer encoder-decoder
architecture. The radar’s data is complex; to determine rich feature
representation, the channel boosting scheme is adopted in which the input
radar image is transformed into different color space. The input radar image
and color space images are followed by the respective backbone network for the
features generation. The respective backbone networks’ features are
concatenated, and these image features are fed to transformer encoder-decoder
architecture for object detection. The proposed network is extensively
evaluated using the COCO evaluation metrics. The experimental evaluation shows
the efficacy of proposed work with the state-of-the-art method [1]. The main
contributions are listed as follows:

Figure 1: The overall framework for the proposed method. It includes a channel
boosting feature ensemble method along with transformer encoder-decoder
network for the radar-based object detection.

  1. 1.

A novel channel boosting feature ensemble method is designed for the feature
representation of radar data.

  2. 2.

The channel boosting feature ensemble method is used in conjunction with the
transformer encoder-decoder architecture for the radar-based object detection.

  3. 3.

The efficacy of the proposed method (best-model) illustrates $12.55\%$ and
$12.48\%$ increase in mean average precision (mAP) score in both good and
good-bad weather conditions for radar-based object detection as compared to
state-of-the-art (best-model) [1].

The rest of the paper is organized as follows: Section II gives related work.
Section III explains the proposed methodology. The experimentation and results
are discussed in Section IV, and section V concludes the paper.

##  II Related Work

The researchers in [14, 15, 16] have explored radar-based object detection in
the domain of autonomous vehicle. [17] has proposed a post-processing
algorithm for object detected by highly accurate and reliable radar. The
algorithm clusters multiple detections form a single object and track them
using Kalman filter. [18] handles the uncertainties of laser radar by
occupancy grid framework. The occupancy grid map is formed by using the
inverse sensor model in the cartesian coordinate framework. The local grid map
and global grid map, generated by temporal integration of sensor data are
fused using the Dempster rule of combination. The conflict information is used
to detect the moving object.

The fusion of radar and camera data for object detection is done in [19, 20,
21]. [22] has proposed the architecture for frontal object detection using
radar data and a single camera. The detections are fused and then tracked
using the Kalman filter. [23] represents a framework to identify an occupying
area by fusing the data from millimetre radar and video camera. [24] performs
the radar vision fusion in three steps. First radar guides a selection of the
small number of candidates images for processing. Second, sparse
representation is generated by normalizing the selected attention window and
is processed by orientation-selective feature detector. Finally, a learned
multi-layered network is used to classify the sparse representation of
different objects. [25] has proposed a multi-sensor fusion system for object
detection and tracking for autonomous vehicles. They have utilized the camera,
lidar and radar to perceive the environment and develop a system to detect
pedestrian, bicyclist and moving objects. The recognition framework helps to
improve tracking and data association of the detected objects by multi-sensor
data.

The aforementioned algorithm uses traditional techniques for object detection.
Since the advent of the convolutional neural network (CNN), object detection
accuracy has dramatically increased. CNN learns informative feature
representation of objects contributes to success. In literature, convolutional
neural networks are used for radar-based object detection. [26] has proposed a
two-step object detection framework called radar region proposal network. The
region proposal network generates a proposal by mapping radar data onto images
and generates a set number of anchor boxes used to detect objects and
calculate their distance from the vehicle. [27] introduces an architecture for
object detection that utilizes RGB camera images and radar sensor data. They
have used radar data to identify high confidence point in the image to develop
better feature representation and generate proposals for object detection. [1]
has proposed radar dataset in adverse weather conditions and detect vehicles
using radar data. They have used Faster-RCNN architecture for object detection
with two modification. First, they fixed the anchor size and second, the
bounding box has an additional parameter angle.

A new paradigm of the object detection algorithm has recently evolved based on
a new attention-based building block called transformer network. The attention
mechanism in transformers aggregates information of the whole input sequence,
which give them the advantage of global computations and perfect memory. [28]
introduces an encoder-decoder transformer framework for object detection based
on a direct set prediction which bypasses anchor generation in typical object
detection network.

##  III Proposed Method

Feature extraction plays an imminent role for many deep-learning-based object
detection task. In this work, a channel boosting feature ensemble method along
with a direct set prediction for object detection using an encoder-decoder
transformer network is proposed for the radar-based object detection. Fig.1
illustrates the architecture of the proposed work.

###  III-A Channel Boosting Feature Ensemble Method

The radar data exhibits large variations and thus requires a robust feature
extraction mechanism. In this study, a channel boosting in conjunction with
the ensemble method is used to extract features for radar images. First, the
channel boosting method is used by aggregating the input image’s RGB channels
by converting it to LUV and CIELAB color spaces. The conversion to these color
spaces enables the channel boosting by providing a good representation of the
input image and helps the network to learn the modular and hierarchical
representation of the image. Fig.1 illustrates the channel boosting ensemble
method for the representation of the input image. Suppose
$x_{i}\in\mathbb{R}^{D\times H\times W}$ for $i\in I=(RGB,LUV,CIELAB)$ is the
input image with $x_{i_{rgb}}$, $x_{i_{luv}}$ and $x_{i_{lab}}$ corresponds to
the variants of the input image. Each channel variant of input image (RGB,
LUV, and CIELAB) is followed by the convolution neural network (CNN) backbone
for feature representation. Suppose the $x_{i\in rgb}$ with $D=3$ (initially),
is passed to the backbone for the generation of low-resolution activation map
$f=\mathbb{R}^{C\times H\times W}$ with typical values of $C=2048$. Similarly,
the other two color space variants of the input image are processed by the
backbone network resulting in low-resolution feature maps of each dimension of
$C=2048$ respectively. These low-resolution feature activation maps are
concatenated and then followed by an additional convolutional layer to produce
the feature activation maps from the channel boosting ensemble method with a
dimension of $C=2048$. The reason for this additional convolutional layer is
empirical and also to reduce the computation cost.

###  III-B Transformer Encoder-Decoder

####  III-B1 Encoder

The transformer encoder expects a sequence as inputs, so the set of activation
maps $f$ produced by channel boosting feature ensemble is reduced to lower
dimension space from $C$ to $k$ by applying a $1\times 1$ convolution layer
and flattened to create the new feature map $\tilde{f}\in\mathbb{R}^{k\times
HW}$ and is given to the transformer encoder. For each encoder layer, a multi-
head self-attention module and the feed-forward network is incorporated in the
architecture.

For the multi-head self-attention module’s brevity, the explanation of the
single head attention mechanism is essential [28][29]. The query and key-value
pair form the basis of the attention and are utilized to map it to the output.
The query, key and value are all represented in the vector format. The weight
tensor $W_{t}\in\mathbb{R}^{3\times\hat{k}\times k}$ for a single attention
head $A(X_{q},X{k_{v}},W_{t})$ computes the query, key and value embedding
after adding the query and key positional encoding
($P_{q}\in\mathbb{R}^{k\times N_{q}}$ and $P_{kv}\in\mathbb{R}^{k\times
N_{kv}}$) respectively, as shown in Eq.1

$\displaystyle\begin{aligned}
&[Q:K:V]=[W_{t1}(X_{q}+P_{q}):W_{t_{2}}(X_{kv}+P_{kv})\\\
&:W_{t_{3}}X_{kv}]\end{aligned}$ (1)

where, $[:]$ represents the concatenation, and ($W_{t_{1}}$,$W_{t_{2}}$ and
$W_{t_{3}}$) are concatenation of $W_{t}$. $X_{q}$ corresponds to the query
sequence of length $N_{q}$. The key-value sequence is denoted by $X_{k_{v}}$
of length $N_{k_{v}}$. Eq.2 illustrates the computation of activation weights
$\delta$ by computing the soft-max function of the dot product of queries and
keys.

$\displaystyle\delta_{(i,j)}=\frac{\exp^{\frac{1}{\sqrt{k}}Q_{i}^{T}K_{j}}}{\sum_{j=1}^{N_{kv}}\exp^{\frac{1}{\sqrt{k}}Q_{i}^{T}K_{j}}}$
(2)

The positional embedding is learned and shared across all the attention layers
for the given query, key-value pair [28]. The final output of the single head
attention module is computed by Eq.3.

$\displaystyle A(X_{q},X_{kv},W)=\sum_{j=1}^{N_{kv}}\delta_{ij}V_{j}$ (3)

The multi-head self-attention module is the concatenation of $M$ single head
attention modules followed by the projection matrix $L$. The projection matrix
is the combination of residual connection, dropout and layer normalization.
The overall computation is described in Eq.4. In the case of multi-head
attention module the only difference in computing the attention weight as
described in Eq.2 is the change of scale fraction $k$ to
$\hat{k}=\frac{k}{M}$.

$\displaystyle\begin{aligned}
&{X_{q}}^{\prime}=[A(X_{q},X_{kv},W_{1}):...:A(X_{q},X_{kv},W_{M})]\\\
&\tilde{X_{q}}=\psi(X_{q}+dropout(L{X_{q}}^{\prime}))\end{aligned}$ (4)

####  III-B2 Transformer Decoder

The decoder follows the same structure of sub-layer as the encoder. It
constitutes of two multi-head self-attention modules used for transforming the
$N$ embedding size of $k$.

The decoder’s input are queries that are initially set to zero, $N$ object
queries that are learnt positional encodings, and the encoder memory. In the
decoder, the positional encodings are added to each attention layers. The
decoder’s aforementioned inputs are used to produce the final set of
prediction that includes bounding box and class labels through multi-head
self-attention decoder modules. It is to be mentioned here that the first
self-attention layer in the first decoder layer is skipped in the processing
of decoding the embeddings.

###  III-C Feed-Forward Network (FFN)

The decoder’s output embedding is fed to the feed-forward neural network for
the prediction of class labels and bounding box. A $3$ layer perceptron
network with ReLU activation, a hidden dimension of $k$ and a linear
projection layer is used for the final prediction output. The output of FFN
consists of height, the width of the bounding box w.r.t image, normalized
center coordinates and class labels. Besides, an extra no-class label is also
used for no object detected in the input image. In this work, the number of
classes is $2$ (vehicle and no-vehicle) because the intra-class object
detection will not provide any sufficient information because of the radar
data nature.

##  IV Experimentation and Results

###  IV-A Dataset

RADIATE (RAdar Dataset In Adverse weaThEr) dataset is used in this study [1].
The motivation to use RADIATE is to facilitate the object detection research
in adverse weather conditions and understand the dynamic environment better so
that an autonomous vehicle can safely navigate. The dataset uses Navtech
CTS350-X radar to collect the data. The radar renders $360^{\circ}$ high-
resolution range-azimuth images. It has a maximum operating range of 100
meters and 0.175m range resolution, $1.8^{\circ}$ azimuths and elevation
resolution. The dataset is collected in different weather conditions including
sunny, overcast, night, snow, fog and rain. The dataset consists of $3$ hours
of annotated radar images with an estimated $200$K labels vehicles. The
driving scenarios covers urban, motorway and suburban driving. The images are
labelled using 2D bounding boxes. Each bounding box define
$(x,y,width,height,angle)$, where $(x,y)$ are the top-left pixel location and
angle define counter-clockwise rotation. The dataset is divided into three
parts, training in good weather, training in good-bad weather and test set
that contain both good and bad weather. We have adopted standard partition as
proposed by [1] for a fair comparison, shown in Table I.

TABLE I: The RADIATE dataset partition topology for training, and testing the proposed method in adverse weather conditions  | No. of Images | No. of vehicles
---|---|---
Train Set (Good Weather) | 23091 | 106931
Train Set (Good and Bad Weather) | 9760 | 39647
Test Set | 11289 | 147005

###  IV-B Evaluation Metric

Numerous evaluation metrics exist to evaluate the accuracy of object detection
in the images. Here, we have used a standard MS COCO evaluation metric [30].
IoU (Intersection over Union) defined the intersection between the area of
ground-truth bounding box and area of the predicted bounding box shown by Eq.
5. The prediction is True Positive (TP) if $IoU>threshold$ and False Positive
(FP) if $IoU<threshold$.

$\displaystyle IoU=\frac{A_{p}\cap A_{gt}}{A_{p}\cup A_{gt}}$ (5)

Recall and Precision are calculated using Eq. LABEL:equa3.

$\displaystyle Recall=\frac{TP}{TP+FN},Precision=\frac{TP}{TP+FP}$ (6)

The Average Precision(AP) is calculated per class. AP is the area under the PR
curve, shown by Eq. 7.

$\displaystyle AP[class]=\frac{1}{\\#thresh}\sum_{IoU\in thresh}AP[class,IoU]$
(7)

COCO evaluation uses a threshold range from $0.5$ to $0.95$ with a step size
of $0.05$. In the experimentation, $mAP_{IoU=0.5}$ is utilized, which is also
similar to PASCAL VOC [31] evaluation.

Figure 2: The illustration of mAP scores for all the models used as a backbone
network in the proposed method trained on good-bad weather conditions and
tested on test data. ResNet-$101$ has the best results.  Figure 3: The
illustration of mAP scores for all the models used as a backbone network in
the proposed method trained on good weather conditions and tested on test
data. ResNet-$101$ has the best results.

###  IV-C Experimentation

The proposed network is trained on two GPUs having $24Gb$ memory in total
using Pytorch deep learning library. The input training data is scaled,
augmented, randomly horizontal flipped, randomly resized and randomly cropped
in order to provide the network to learn the global relationship. The training
process is run for $125$ epochs having the learning rate drop of factor $10$
at $100$ epochs. The Adam (Adaptive Momentum Estimation) is used as an
optimizer with $10^{-5}$ and $10^{-4}$ learning rates for the backbone and
transformer network, respectively. The pre-trained backbone network is
imported from the Torchvision with frozen the batch-norm layer and discarding
the last classification layer. In the experimentation, the ResNet-$50$ [32],
RestNet-$101$ [32], MobileNet-V$2$ [33] and ShuffleNet-V$2$ [34] are used as a
backbone networks. In the transformer network, the number of encoders and
decoders are equal to $6$. All the transformer weights are randomly
initialized with Xavier initialization. A dropout of $0.1$ is used after every
multi-head attention and feed-forward network layer before normalization. For
the spatial positional encodings, an absolute position encodings are used that
are function of sine and cosine with different frequencies and are
concatenated to get the final position encoding across the $k$ channel.

Figure 4: The qualitative results of proposed radar-based object detection in
different weather conditions with all the four backbone networks used along
with transformer encoder-decoder network is illustrated.

Since, the proposed method infers the set of $N$ predictions, in this work,
inspired by [28] an optimal bipartite matching scheme is adopted for the loss
calculations. Eq.8 illustrates the bipartite matching loss function between
$\hat{y}$ set of predictions $N$ added with no object class and ground-truth
($y$) also padded with the set of size $N$.

$\displaystyle\tilde{\eta}=\underset{\eta\in
C_{N}}{argmin}\sum_{j}^{N}L_{match}(y_{j},\hat{y}_{\eta(j)})$ (8)

where $C_{N}$ corresponds to the permutations of N elements.
$L_{match}(y_{j},\hat{y}_{\eta(j)})$ illustrates the pair-wise matching cost
between ground-truth and prediction. A Hungarian algorithm is used to compute
this assignment problem between ground-truth and prediction. The matching cost
constitutes towards both the class labels and the bounding box between the
ground-truth and the predictions. Suppose each element $j$ of ground-truth set
is represented as $y_{j}=(c_{j},b_{j})$ where $c_{j}$ is the class label and
$b_{j}\in[0,1]^{5}$ is the bounding box vector corresponds to normalized
center coordinates, height, width and angle for the respective image. For the
prediction, the class probability and bounding box is defined as
$\hat{p}_{\eta{j}}(c_{j})$ and $\hat{b}_{\eta{j}}$ respectively. The matching
between ground-truth and prediction is done by finding the direct one-to-one
correspondence without duplicates. The modified Hungarian loss function with
angle inclusion in the bounding box is illustrated in Eq.9 using the
aforementioned notations of the class label and bounding box.

$\displaystyle\begin{aligned}
L_{Hungarian}(y,\hat{y})=\sum_{j=1}^{N}[-log\hat{p}_{\eta({j})}(c_{j})+&\\\
\mathbf{1}_{c_{j}\neq(\phi)}L_{box}(b_{j},\hat{b}_{\hat{\eta}}(j))]\end{aligned}$
(9)

where $\hat{\eta}$ corresponds to optimal assignment. The $L_{box}(.)$ in the
Hungarian loss function is used for scoring the bounding box, and in
comparison to other detectors the box prediction is computed directly. In
other to compensate the scaling issue, the $l_{1}$ loss with the complete IoU
loss is used. The complete IoU [35] loss is illustrated by the Eq.10.

$\displaystyle
L_{CIoU}=1-IoU+\frac{\rho(\mathbf{b},\boldsymbol{b_{gt}})}{c^{2}}+\alpha\nu$
(10)

where $1-IoU+\frac{\rho(\mathbf{b},\boldsymbol{b_{gt}})}{c^{2}}$ corresponds
to distance IoU with $\mathbf{b}$, $\boldsymbol{b_{gt}}$ denotes the bounding
box points of predicted and ground-truth respectively. $\rho$ denotes the
euclidean distance and $c$ illustrates the diagonal length of smallest
enclosing box that covers the two boxes. $\alpha$ is the positive trade-off
parameter and the $\nu$ is used to compute the consistency of aspect ratio
defined by Eq.11 and Eq.12 respectively.

$\displaystyle\alpha=\frac{\nu}{(1-IoU)+\nu}$ (11)

$\displaystyle\nu=\frac{4}{\pi^{2}}(arctan\frac{w^{gt}}{h^{gt}}-arctan\frac{w}{h})^{2}$
(12)

The $L_{box}(.)$ is represented as shown in Eq.13

$\displaystyle
L_{box}(b_{j},\hat{b}_{\hat{\eta}}(j))=\lambda_{iou}L_{CIoU}(b_{j},\hat{b}_{\hat{\eta}}(j))$
$\displaystyle+$ (13)
$\displaystyle\lambda_{l1}\left\|b_{j}-\hat{b}_{\hat{\eta}}(j)\right\|$

where $\lambda_{l1}=4$ and $\lambda_{iou}=2$ are used in training the proposed
method.

For the quantitative and qualitative evaluation, a baseline network of
proposed work is trained on two sets of datasets that include good-bad weather
conditions and only good weather conditions. The proposed baseline network is
compared with [1] for the evaluation to measure the mean average precision
(mAP) scores. Table-II and Table-III illustrates the mAP score of proposed
baseline method on testing data. In addition, to further explore the proposed
method’s effectiveness, the proposed method is trained using RestNet-$101$,
MobileNet-V$2$ and ShuffleNet-V$2$ as a backbone network. In the quantitative
analysis, ResNet-$101$ has outperformed the other backbone network mAP scores
and, also surpasses the [1] scores by a margin of $12.48\%$ in good-bad
weather conditions and by $12.55\%$ in good weather conditions with its
counterpart network. Fig.2 and Fig.3 illustrates the running mAP scores of
good-bad weather conditions and good weather conditions across the number of
epochs on the testing data. Fig.4 shows the qualitative results of radar
object detection of all the backbone used in the experimentation in different
weather conditions along with the accompanying optical image of the scene.

TABLE II: mAP Score on Radar Data with Good and Bad Weather Conditions Models | mAP Score
---|---
ResNet-50 Trained [1] | 45.77
ResNet-101 [1] | 46.55
Ours (Backbone: ResNet-50) (Baseline) | 55.54
Ours (Backbone: ShuffleNet-V2) | 54.69
Ours (Backbone: MobileNet-V2) | 52.05
Ours (Backbone: ResNet-101) | 59.03
TABLE III: mAP Score on Radar Data with Good Weather Conditions Models | mAP Score
---|---
ResNet-50 Trained [1] | 45.31
ResNet-101 [1] | 45.84
Ours (Backbone: ResNet-50) (Baseline) | 54.90
Ours (Backbone: ShuffleNet-V2) | 52.90
Ours (Backbone: MobileNet-V2) | 49.78
Ours (Backbone: ResNet-101) | 58.39

##  V Conclusion

This study focuses on radar-based object detection in adverse weather
condition for autonomous vehicles. A novel architecture has introduced which
utilizes the channel boosting feature ensemble for more desirable feature
extraction. The extracted features are then concatenated and feed to the
encoder-decoder transformer network, which uses a direct set prediction method
for object detection. The efficacy of the proposed algorithm is evaluated
using standard COCO evaluation. The mean average precision of $59.03\%$ and
$58.39\%$ are achieved on test data for the proposed method with ResNet-$101$
backbone netowrk trained on good-bad weather conditions and good weather
conditions respectively.

In future work, we aim to fuse Lidar data with radar to improve object
detection in adverse weather condition. Moreover, use image data to classify
the weather condition to optimize the perception system of the autonomous
vehicle.

## Acknowledgment

This work was partly supported by Institute of Information communications
Technology Planning Evaluation (IITP) grant funded by the Korea Government
(MSIT) (No. 2014-3-00077, Development of Global Multi-target Tracking and
Event Prediction Techniques Based on Real-time Large-Scale Video Analysis) and
the National Research Foundation of Korea (NRF) grant funded by the Korea
Government (MSIT) (No. 2019R1A2C2087489), and Ministry of Culture, Sports and
Tourism (MCST), and Korea Creative Content Agency (KOCCA) in the Culture
Technology (CT) Research & Development (R2020070004) Program 2020.

## References

  * [1] M. Sheeny, E. De Pellegrin, S. Mukherjee, A. Ahrabian, S. Wang, and A. Wallace, “Radiate: A radar dataset for automotive perception,” _arXiv preprint arXiv:2010.09076_ , 2020.
  * [2] K. Bimbraw, “Autonomous cars: Past, present and future a review of the developments in the last century, the present scenario and the expected future of autonomous vehicle technology,” in _2015 12th international conference on informatics in control, automation and robotics (ICINCO)_ , vol. 1. IEEE, 2015, pp. 191–198.
  * [3] S. Campbell, N. O’Mahony, L. Krpalcova, D. Riordan, J. Walsh, A. Murphy, and C. Ryan, “Sensor technology in autonomous vehicles: A review,” in _2018 29th Irish Signals and Systems Conference (ISSC)_. IEEE, 2018, pp. 1–4.
  * [4] F. Rosique, P. J. Navarro, C. Fernández, and A. Padilla, “A systematic review of perception system and simulators for autonomous vehicles research,” _Sensors_ , vol. 19, no. 3, p. 648, 2019.
  * [5] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” _arXiv preprint arXiv:1804.02767_ , 2018.
  * [6] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in _European conference on computer vision_. Springer, 2016, pp. 21–37.
  * [7] S. Ren, K. He, R. B. Girshick, and J. Sun, “Faster r-cnn: towards real-time object detection with region proposal networks. corr abs/1506.01497 (2015),” _arXiv preprint arXiv:1506.01497_ , 2015.
  * [8] S. Zhang, L. Wen, X. Bian, Z. Lei, and S. Z. Li, “Single-shot refinement neural network for object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 4203–4212.
  * [9] Q. Zhao, T. Sheng, Y. Wang, Z. Tang, Y. Chen, L. Cai, and H. Ling, “M2det: A single-shot object detector based on multi-level feature pyramid network,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 9259–9266.
  * [10] S. D. Pendleton, H. Andersen, X. Du, X. Shen, M. Meghjani, Y. H. Eng, D. Rus, and M. H. Ang, “Perception, planning, control, and coordination for autonomous vehicles,” _Machines_ , vol. 5, no. 1, p. 6, 2017.
  * [11] Y. Zhou and O. Tuzel, “Voxelnet: End-to-end learning for point cloud based 3d object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4490–4499.
  * [12] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 652–660.
  * [13] S. Azam, F. Munir, A. Rafique, Y. Ko, A. M. Sheri, and M. Jeon, “Object modeling from 3d point cloud data for self-driving vehicles,” in _2018 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2018, pp. 409–414.
  * [14] C. C. Schwesig and I. Poupyrev, “Radar-based object detection for vehicles,” Oct. 29 2019, uS Patent 10,459,080.
  * [15] D. M. Gavrila, “Sensor-based pedestrian protection,” _IEEE Intelligent Systems_ , vol. 16, no. 6, pp. 77–81, 2001.
  * [16] S. Miyahara, “New algorithm for multiple object detection in fm-cw radar,” SAE Technical Paper, Tech. Rep., 2004.
  * [17] A. Manjunath, Y. Liu, B. Henriques, and A. Engstle, “Radar based object detection and tracking for autonomous driving,” in _2018 IEEE MTT-S International Conference on Microwaves for Intelligent Mobility (ICMIM)_. IEEE, 2018, pp. 1–4.
  * [18] J. Duan, L. Ren, L. Li, and D. Liu, “Moving objects detection in evidential occupancy grids using laser radar,” in _2016 8th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC)_ , vol. 2. IEEE, 2016, pp. 73–76.
  * [19] M. Gong, Z. Zhou, and J. Ma, “Change detection in synthetic aperture radar images based on image fusion and fuzzy clustering,” _IEEE Transactions on Image Processing_ , vol. 21, no. 4, pp. 2141–2151, 2011.
  * [20] Y.-T. Zhou, “Multi-sensor image fusion,” in _Proceedings of 1st International Conference on Image Processing_ , vol. 1. IEEE, 1994, pp. 193–197.
  * [21] K. Abe, S. Tokoro, and K. Suzuki, “Object detection system and method of detecting object,” Apr. 15 2008, uS Patent 7,358,889.
  * [22] R. O. Chavez-Garcia, J. Burlet, T.-D. Vu, and O. Aycard, “Frontal object perception using radar and mono-vision,” in _2012 IEEE Intelligent Vehicles Symposium_. IEEE, 2012, pp. 159–164.
  * [23] T. Kato, Y. Ninomiya, and I. Masaki, “An obstacle detection method by fusion of radar and motion stereo,” _IEEE transactions on intelligent transportation systems_ , vol. 3, no. 3, pp. 182–188, 2002.
  * [24] Z. Ji and D. Prokhorov, “Radar-vision fusion for object classification,” in _2008 11th International Conference on Information Fusion_. IEEE, 2008, pp. 1–7.
  * [25] H. Cho, Y.-W. Seo, B. V. Kumar, and R. R. Rajkumar, “A multi-sensor fusion system for moving object detection and tracking in urban driving environments,” in _2014 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2014, pp. 1836–1843.
  * [26] R. Nabati and H. Qi, “Rrpn: Radar region proposal network for object detection in autonomous vehicles,” in _2019 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2019, pp. 3093–3097.
  * [27] R. Yadav, A. Vierling, and K. Berns, “Radar+ rgb attentive fusion for robust object detection in autonomous vehicles,” _arXiv preprint arXiv:2008.13642_ , 2020.
  * [28] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” _arXiv preprint arXiv:2005.12872_ , 2020.
  * [29] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [30] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [31] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” _International journal of computer vision_ , vol. 88, no. 2, pp. 303–338, 2010.
  * [32] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition. corr abs/1512.03385 (2015),” 2015.
  * [33] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 4510–4520.
  * [34] N. Ma, X. Zhang, H.-T. Zheng, and J. Sun, “Shufflenet v2: Practical guidelines for efficient cnn architecture design,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 116–131.
  * [35] Z. Zheng, P. Wang, W. Liu, J. Li, R. Ye, and D. Ren, “Distance-iou loss: Faster and better learning for bounding box regression.” in _AAAI_ , 2020, pp. 12 993–13 000.