# Fusion of Inverse Synthetic Aperture Radar and Camera Images for Automotive
Target Tracking

Shobha Sundar Ram The authors are with Indraprastha Institute of Information
Technology Delhi (email: shobha@iiitd.ac.in).

###### Abstract

Automotive targets undergoing turns in road junctions offer large synthetic
apertures over short dwell times to automotive radars that can be exploited
for obtaining fine cross-range resolution. Likewise, the wide bandwidths of
the automotive radar signal yield high-range resolution profiles. Together,
they are exploited for generating inverse synthetic aperture radar (ISAR)
images that offer rich information regarding the target vehicle’s size, shape,
and trajectory which is useful for object recognition and classification.
However, a key requirement for ISAR is translation motion compensation and
estimation of the turning velocity of the target. State-of-the-art algorithms
for motion compensation trade-off between computational complexity and
accuracy. An alternative low complexity method is to use an additional sensor
for tracking the target motion. In this work, we propose to exploit computer
vision algorithms to identify the radar target object in the sensor field-of-
view (FoV) with high accuracy. Further, we propose to track the target
vehicle’s motion through fusion of vision and radar data. Vision data
facilitates the accurate estimation of the lateral position of the target and
its lateral velocity which complements the radar’s capability of accurate
estimation of range and radial velocity. Through simulations and experimental
evaluations with a monocular camera and Texas Instrument’s millimeter wave
radar, we demonstrate the effectiveness of sensor fusion for accurate target
tracking for translational motion compensation and generation of high quality
ISAR images.

###### Index Terms:

automotive radar, camera, inverse synthetic aperture radar, sensor fusion

##  I Introduction

Millimeter wave automotive radars transmit wideband frequency modulated
continuous waveforms that enable the generation of high range-resolution
profiles [1]. When combined with wide radar antenna apertures, high resolution
two-dimensional (2D) images along the top-view (or bird’s eye view) of an
object can be realized at short ranges. Large radar apertures can be realized
through the use of multiple antenna elements and corresponding receiver
channels. These electrically large antenna apertures can be fabricated with a
small physical footprint at millimeter wave frequencies. However, the cost and
complexity of the multichannel system are significant. Further, fully or even
semi-autonomous vehicles are envisaged to have multiple radars mounted all
around the vehicle to provide $360^{\circ}$ coverage which further scales the
cost. Inverse synthetic aperture radar (ISAR) imaging presents a low
complexity alternative to large antenna arrays. When a target rotates or turns
while remaining in the radar beam, the aspect presented by the target provides
a large synthetic aperture which can be exploited for fine cross-range
resolution determined by the total aspect and the target dwell time [2].
Further, these turns are commonly encountered at road junctions.

ISAR imaging has been researched for the detection and classification of
several airborne and waterborne bodies [3, 4, 5, 6]. More recently SAR and
ISAR imaging have been examined for ground-based vehicles [7, 8, 9, 10, 11,
12, 13]. These works have demonstrated that wideband synthetic aperture data
result in images that provide rich information on the size, geometry, and
trajectory undertaken by the target vehicle along the top-view that can be
exploited for object recognition. Several methods have been explored for
generating cross-range information along elevation. One possibility is to
exploit the target rotation about the horizontal axis (roll or pitch) along
with the yaw. This method, while suited for certain types of airborne targets,
is not practical for automotive targets where the only components that undergo
roll motion are the wheels. The second possibility is to perform beamforming
along the elevation by incorporating multiple receiver elements along the
elevation dimension. However, this again increases the complexity of the radar
architecture. The simplest alternative, that is investigated in this work, is
_interferometric ISAR_ where we estimate the height of target features using
differential interferometry between the ISAR images generated at two receiver
elements spaced along the height dimension. The results show that the target
features are of very low (and similar) elevation angles even at short ranges
of the vehicle from the radar.

There are some key challenges associated with ISAR imaging. First, the
fundamental principle of ISAR imaging is that the Doppler frequency shift in
the radar received signal should arise from the rotational motion of the
target instead of the translational motion. Therefore, the translational
motion of the target has to be accurately compensated to ensure that the
images are focused. Second, the rotational velocity of the target has to be
correctly estimated in order to map the Doppler frequency axis to the cross-
range axis. Several algorithms in radar literature have been explored for
_blind_ motion compensation when the target trajectory is not known to the
radar. The traditional methods involved tracking the dominant scatterers in
the radar data either in the range domain [14, 15] or along the Doppler domain
[16] and performing range alignment or auto-focusing of the ISAR images.
Later, Keystone transform was explored to perform higher order motion
compensation for ISAR images [17, 18]. More recently, particle swarm
optimization [19] and sparsity-based techniques [20] have been explored to
perform higher-order motion compensation. These algorithms have typically
traded off between accuracy in motion compensation and computational cost.
Alternatively, the use of secondary radars that provides ground truth target
state information has also been suggested to reduce the computational
complexity [9]. Due to the ubiquity of cameras for assisted and autonomous
driving applications, the use of vision data for enhancing radar imaging forms
a natural solution for automotive applications that we explore in this work.

Concurrent with the research of automotive radar imaging of road users,
computer vision algorithms are being developed for road object recognition and
classification [21, 22, 23]. A monocular camera offers frontal images as
opposed to the top-view images obtained from radar. These images are
characterized by excellent lateral resolution and corresponding lateral
velocity resolution but require more complex processing for recovering depth
information [24]. Even stereo cameras can provide acceptable estimates of the
depth/range only up to a distance of a few meters [25]. On the other hand,
while the range and radial velocity estimation by radar is accurate, the
lateral velocity estimation is very poor. Thus the radar and camera apertures
yield information that presents complementary features for object tracking. In
this work, we propose the fusion of monocular vision data with radar data to
enable target state estimation - position, velocity and turn rate. Then, we
exploit the estimated target trajectory information for performing motion
compensation on radar data in order to realize high-resolution ISAR images.

Several works in the last few years have explored the fusion of data from two
or more sensors for supporting advanced driver assistance system
functionalities. Sensor fusion offers several advantages [26]. First, are the
advantages of redundancy in the event of any single sensor malfunction or
breakdown. For example, camera images are affected by weather and light
conditions. Radars can work 24×7, in all weather conditions, and to a limited
extent in non-line-of-sight conditions as well due to the presence of
multipath [27, 28]. Secondly, multiple sensors are useful for reducing the
errors associated with the estimation of target parameters. Radars and
cameras, in particular, as mentioned earlier, offer complementary information
that when fused together increase the statistical information of a target.
Cameras offer vital information on the size, shape, color, and texture of
various objects on the road and hence are extremely effective for object
detection and classiﬁcation [23]. Radar signatures, on the other hand, are not
always intuitive and require a trained operator for interpretation. Further,
the mitigation of clutter and multipath in radar images can be challenging
[29, 30].

Many works over the last two decades have explored the fusion of radar and
vision data. One of the earliest works explored the use of depth estimates
provided by radar data to enhance the target segmentation in the camera image
[31]. Later the authors in [32] combined radar and vision data to detect guard
rails in the ego vehicle’s environment. In [33], the fused vision and radar
data were used to estimate the position, velocity, and orientation of an
oncoming vehicle to enable collision avoidance. In all of these works, low-
resolution radar detections - a single detection per target - are obtained for
automotive targets which are subsequently used for localization. The key
distinction of our work from the prior art in radar-vision fusion is that we
are proposing a method to fuse _high resolution_ radar images with camera
data.

The paper is organized in the following manner. In the following section, we
present the algorithms for processing radar data in order to obtain 3D ISAR
images; algorithms for the camera data processing for estimating the position
of the target vehicle; and the radar-camera fusion framework for estimating
the target position and focusing the radar images. In Section III, we present
the simulation setup for generating radar and camera data from a cuboid model
of a target vehicle. The simulations model the sensors with a finite
probability of detection, occlusion effects and shadowing, and false alarms
from clutter. The simulation results demonstrate the effectiveness of focusing
high-resolution radar images from monocular vision data through comparison
with ground truth radar images. The algorithms are further validated with
experimental data gathered from both the radar and the camera in Section IV.
Finally, we conclude the paper with a discussion of the key insights and scope
for future work in Section V.

_Notation:_ Lowercase Latin and Greek letters denote scalar variables while
boldface lowercase and uppercase letters represent vectors and matrices
respectively. The homogeneous representation of a 2D or 3D point is
$\mathbf{\bar{x}}=[\mathbf{x};1]$.

##  II Theory

In self-driving cars, multiple millimeter wave radars and cameras are mounted
all around the chassis of the ego vehicle for complete coverage of the road
conditions. In this work, we consider a single forward facing low complexity
three element radar with a single transmitter phase synchronized with two
receivers; and a single monocular forward-facing camera in close proximity to
the radar so that there is a large overlap in their fields-of-view (FoV). We
consider two distinct coordinate frames corresponding to the radar/ego vehicle
($\operatorname{\mathcal{R}}$) and the camera ($\operatorname{\mathcal{C}}$)
with a shared common ground plane at $z=0$. The misalignment between the X and
Y axes between the two frames is mapped through the Euler rotation matrix,
denoted by $\mathbf{R}^{\operatorname{\mathcal{C}}\operatorname{\mathcal{R}}}$
(from $\operatorname{\mathcal{C}}$ to $\operatorname{\mathcal{R}}$), and the
translational matrix denoted by
$\mathbf{t}^{\operatorname{\mathcal{C}}\operatorname{\mathcal{R}}}$. We
consider a single automotive target scenario where the target is detected
jointly by the radar and camera. The detections from the two sensors are time
stamped to provide updates at a low frame rate (a few samples per second)
indexed by $k$. The dual sensor data are processed and subsequently fused to
estimate the target as described in the section below. Subsequently, the
target yaw rate estimated by the sensor fusion is used to generate the ISAR
images. The entire data processing chain for camera and radar data is
presented in Fig.1.

Figure 1: System framework of radar and camera data processing and fusion for
automotive target imaging

In this section, we first describe the radar processing steps followed by the
camera processing steps and finally the fusion.

###  II-A Interferometric ISAR Imaging of Extended Targets

The automotive radar transmitter and the two receivers are assumed to be
configured in an L formation with the receivers displaced from each other by
half-wavelength along the $z$ axis. The radar transmits linear frequency
modulated waveforms modeled as

$\displaystyle\mathbf{s}_{tx}(\tau)=\text{rect}\left(\frac{\tau}{T_{PRI}}\right)e^{j2\pi
f_{c}\tau}e^{j\pi\beta\tau^{2}},$ (1)

where $\tau$ denotes the fast time samples within a pulse repetition interval
of $T_{PRI}$. The carrier frequency of the transmitted signal is $f_{c}$ and
$\beta$ is the chirp rate of the signal. At short ranges ($\leq$50m), the
automotive radar target appears as an extended target with $b=1:B$ point
scatterers distributed across several range and Doppler bins. The radar
transmit signal is scattered by each $b^{th}$ point scatterer and received by
the $i^{th},i=1,2$ radar receiver element after a two-way round-trip time
delay, $\tau_{b_{i}}$, which we assume is within $T_{PRI}$ to avoid range
ambiguity.

Figure 2: (a) Extended radar target model of a full-sized car; (b) ISAR image
after perfect motion compensation and turning velocity estimation; (c)
Interferogram to estimate elevation angle of each point scatterer.

If the point scatterer is moving at a radial velocity $v_{b}$ from an initial
position, $r_{b_{0}}$, and if the transmitter distance is $r_{tx}$, then the
time-delays at the first and second elements are

$\displaystyle\operatorname{\tau_{b_{1}}}=\frac{r_{tx}+r_{b_{0}}+2v_{b}t}{c}\approx\frac{2r_{b_{1}}+2v_{b}t}{c}$
(2)
$\displaystyle\operatorname{\tau_{b_{2}}}2\approx\frac{2r_{b_{0}}+d\sin\theta_{b}+2v_{b}t}{c},$
(3)

where $c$ is the speed of light and $\theta_{b}$ is the elevation of the point
target with respect to the radar when far-field assumption has been invoked.
Note that in the case of automotive targets, the height of the point scatterer
on the target is very low compared to the range and hence $\theta_{b}$ is
likely to be very low. Also, if the transmitter and receiver are placed very
close together (quasi-monostatic), then $r_{tx}\approx r_{b_{0}}$. Hence, the
received signal at the $i^{th}$ radar receiver after down-conversion, is the
superposition of returns from $B$ individual scatterers, if we ignore multiple
bounces, as shown in

$\displaystyle\mathbf{S}_{{rx}_{i}}(t,\tau)=\sum_{b=1}^{B}a_{b}\text{rect}\left(\frac{\tau-\operatorname{\tau_{b_{i}}}}{T_{PRI}}\right)e^{-j2\pi
f_{c}\operatorname{\tau_{b_{i}}}}$ $\displaystyle
e^{j\pi\beta(\tau-\operatorname{\tau_{b_{i}}})^{2}}+\eta.$ (4)

Here, $\eta$ represents the noise and $a_{b}$ indicates the reflectivity of
the $b^{th}$ scatterer as well as the two-way propagation factor including the
gains of the transmitting and receiving antenna elements and channel effects.
The received signal is represented as a 2D signal as fast time samples
($\tau$) are gathered across multiple pulse intervals constituting the slow
time axis $t$. The total duration of the slow time interval is the coherent
processing interval (CPI) and the CPI intervals are indexed at $k=1\cdots K$.
_Motion Compensation:_ In order to obtain ISAR images, we have to compensate
for the translational motion of the target. Based on Fig.1, we obtain the
position coordinate estimate of the centroid of the target vehicle over the
past $k-1$ CPI intervals based on which we estimate the starting range
($r_{0_{k}}$) and radial velocity, ($v_{r_{k}}$) for every $k^{th}$ CPI. Then
assuming a first-order kinematic motion model, $r_{k}(t)=r_{0_{k}}+v_{r_{k}}t$
where $t$ spans from 0 to CPI. We use this to perform the coarse translation
motion compensation as shown in

$\displaystyle\mathbf{\hat{S}}_{rx_{i}}(t,\tau)=\mathbf{S}_{rx_{i}}(t,\tau)e^{j2\pi\frac{2r_{k}(t)}{\lambda}}.$
(5)

The complexity of this centroid tracking approach is quite low since it
involves the multiplication of each slow time column of the matrix with a one-
dimensional vector. It is, therefore, preferred to other techniques proposed
in literature such as the cross-correlation method [34] which involves
performing a cross-correlation between each range profile with a reference
range profile (complexity $\mathcal{O}(n^{3})$); and minimum entropy method
which involves a 2D exhaustive search for the range and velocity parameters
for carrying out the operation in (5) [35].
_Stretch Processing_ Next, we process the motion compensated received signal
across the fast time through stretch processing. We consider a reference
signal across the fast time shown by

$\displaystyle\mathbf{s}_{ref}(\tau)=\text{rect}\left(\frac{\tau}{T_{ref}}\right)e^{j\pi\beta(\tau-\tau_{ref})^{2}},\tau_{ref}=\frac{2R_{ref}}{c}.$
(6)

Here $\tau_{0}$ is the time delay to a reference range position corresponding
to the center of the range axis, and $T_{ref}$ is a pulse duration that is
greater than $T_{PRI}$. An output 2D matrix $\mathbf{S}_{{out_{i}}}(t,\tau)$
is generated by element-wise multiplication of every $t^{th}$ row of
$\mathbf{\hat{S}}_{{rx_{i}}}(t,\tau)$ with the conjugate of the reference
signal as shown in

$\displaystyle\mathbf{S}_{{out}_{i}}(t,\tau)=\mathbf{\hat{S}}_{{rx}_{i}}(t,\tau)\mathbf{s}_{ref}^{*}(\tau)$
(7)
$\displaystyle=\sum_{b=1}^{B}\text{rect}\left(\frac{\tau-\operatorname{\tau_{b_{i}}}}{T_{PRI}}\right)a_{b}e^{j\pi\beta\tau_{ref}^{2}}e^{-j\pi\beta\operatorname{\tau_{b_{i}}}^{2}}$
$\displaystyle e^{-j2\pi\frac{2r_{b_{i}}}{\lambda}}e^{-j2\pi
f_{D_{b}}t}e^{-j2\pi\beta(\tau_{ref}-\operatorname{\tau_{b_{i}}})\tau}.$ (8)

Note that the operations in (5) and (7) can be combined into one without
increase in complexity. In the resulting expression in (7), the first term
within the exponent is constant with respect to $\tau,t$ and the receiver
channel $i$ and absorbed into $a_{b}$. The second term has a phase term that
is independent of $\tau$ and $t$ and changes very slightly with respect to $i$
due to the square of $c$ in the denominator. Hence, this term is ignored
without loss of accuracy. The remaining three terms are linear functions of
$i$, $\tau$ and $t$. After motion compensation, the Doppler frequency shift in
(5) arises entirely from the rotational motion of the target, and
$(2\omega/\lambda)\rho_{b}=f_{D_{b}}$ where $\rho_{b}$ is the cross-range
position of the $b^{th}$ point scatterer. Then,

$\displaystyle\mathbf{S}_{{out}_{i}}(t,\tau)=\sum_{b=1}^{B}\text{rect}\left(\frac{\tau-\operatorname{\tau_{b}}}{T_{PRI}}\right)a_{b}e^{-j2\pi\frac{2r_{b_{i}}}{\lambda}}$
$\displaystyle
e^{-j2\pi(2\omega/\lambda)\rho_{b}t}e^{-j2\pi(2\beta/c)r^{\prime}_{b_{i}}\tau},$
(9)

where $2r^{\prime}_{b_{i}}/c=\tau_{ref}-\operatorname{\tau_{b_{i}}}$. By
performing discrete Fourier transform (DFT) across the fast and slow time
data, we obtain the range-crossrange plot $\chi$ for each $k^{th}$ CPI of the
moving target as shown in

$\displaystyle\chi_{i}(\rho,r)=\mathcal{DFT}\left\\{\mathbf{S}_{{out}_{i}}(t,\tau)\right\\}$
$\displaystyle=T_{PRI}T_{CPI}\sum_{b=1}^{B}a_{b}e^{-j2\pi\frac{2r_{b_{i}}}{\lambda}}$
$\displaystyle\text{sinc}\left(\pi(2\omega/\lambda)(\rho-\rho_{b})T_{CPI}\right)\text{sinc}\left(\pi(2\beta/c)(r-r^{\prime}_{b_{i}})T_{PRI}\right),$
(10)

Each point scatterer $b$ is convolved with a double-sided _sinc_ function and
centered at the corresponding range and crossrange of the point scatterer in
the two-dimensional plane. The duration of the fast and slow time axes are
factored into the amplitude of the two-dimensional response. Note that, in the
absence of motion compensation, the range-Doppler images provide us
measurement estimates of the target centroid’s range and Doppler at every CPI,
which are subsequently fused with camera data and used for target state
estimation. Second, we have considered a small angle approximation while
carrying out the ISAR imaging because the vehicle undergoes small angular
rotation (below $1^{\circ}$) during the short CPI that allow us to directly
process the raw data with 2D DFT (of $\mathcal{O}(N^{2})$ complexity).
Alternatively, when the vehicles undergo large turns resulting in wide-angle
synthetic apertures, direct two-dimensional integration through numerical
techniques (of complexity $\mathcal{O}(N^{4})$) may be considered. Or through
nearest neighbors, the backscattered field data that correspond to polar data
in spatial frequency may be reformatted onto a rectangular grid so that the
DFT can be applied for the fast formation of the ISAR image. This introduces
additional complexity of $\mathcal{O}(N^{2})$ [36].

In order to estimate the elevation of each point scatterer, we perform the
very low complexity differential interferometry between the range-crossrange
plots from the two channels. Specifically, we compute the phase difference of
each point in the range-crossrange bin as shown in

$\displaystyle\Theta(\rho,r)=\text{asin}\left(\frac{\lambda\left(\angle{\chi_{2}}(\rho,r)-\angle{\chi_{1}}(\rho,r)\right)}{2\pi
d}\right),$ (11)

to compute the interferogram $\Theta(\rho,r)$. Note that for accurate
interferometry operation, the phase in each interferogram must be carefully
unwrapped. This method of computing the elevation of radar point targets is
effective when the point scatterers are well resolved in the range-crossrange
space. In other words, there exists only a single point scatterer at any
range-crossrange bin of $\chi_{i}$. However, note that the received signal in
(II-A) will also consist of noise. Hence, the range-crossrange image and the
corresponding interferogram will show noise artifacts. Additionally, in real-
world scenarios, reflections from clutter and multipath are also observed in
the radar signatures.

To explain the radar signal processing, we provide an illustrative example of
a simulated ISAR image of a car in Fig.2. Here, we consider a sedan of size
$[5.7m\times 2.7m\times 2m]$ composed of over 6000 point scatterers
corresponding to the centroids of the facets of the body of the car. The
target undertakes a left turn before the radar with a dual channel receiver.
In [13], a detailed description and open source model of the wideband
electromagnetic target scattering was provided. We augment the single channel
data to two receiver channels and process the radar data to obtain the ISAR
image for a single frame in Fig.2b and the corresponding interferogram in
Fig.2c. Since this is a simulation model, the trajectory followed by the
target is known and hence we implement perfect translational motion
compensation. The figure clearly shows the dimensions of the target along the
range and cross-range dimensions. In this synthetic data, the shadowing of
parts of the car by other parts is not considered. Hence, we are able to see
the entire top-view cross-section of the vehicular target and estimate the
size of the car (length and width). The wheels of the car undergo rotational
motion and cause Doppler spread resulting in cross-range spread in the image.
Note that due to the nature of the automotive vehicle target, the elevation
angles are very low but sufficient to estimate the height of the car (based on
the product of the largest elevation angle and range). However the
interferogram image is particularly sensitive to noise compared to the ISAR
image. This implies that the noise variance for the elevation estimate is
greater than that of the range and crossrange.
_Radar Detections:_ We apply the ordered-statistic constant false alarm rate
detector on the range-Doppler signature to dynamically identify the threshold
based on the noise floor and identify radar detections in the image [37]. The
computational complexity of this detector is based on the sorting algorithm
and is higher than alternatives such as the cell-averaging CFAR (CA-CFAR)
[38]. However, the OS-CFAR is widely used by radar practitioners due to its
performance in dynamic noise and clutter conditions. The high-range resolution
imaging described in this section results in multiple detections for each
target. The detections that belong to a single target are clustered together
from which the range, Doppler frequency and elevation of the target’s centroid
is identified.

###  II-B Camera Data Processing

The main objective of the camera data processing is to detect a known camera
target vehicle in the image and convert its centroid position in the camera’s
image to 3D radar coordinate frame.
_Camera Calibration:_ Consider a monocular digital camera image where each
pixel is indexed by $\mathbf{u}=[p_{u},p_{v}]^{T},p_{u}=1\cdots
U,p_{v}=1\cdots V$. Here $p_{u}$ indicates the vertical or longitudinal
position aligned to $z^{\operatorname{\mathcal{C}}}$ axis and $p_{v}$
indicates the lateral or horizontal position aligned to the
$y^{\operatorname{\mathcal{C}}}$ axis while the
$x^{\operatorname{\mathcal{C}}}$ axis is the optical axis. Camera calibration
is the process of learning two sets of parameters using a linear camera
pinhole model. They are the _extrinsic_ camera parameters that facilitate the
transformation of a 3D point, $\operatorname{\mathbf{x}}$ in
$\operatorname{\mathcal{R}}$, to a 3D point,
$\operatorname{\mathbf{x}^{\operatorname{\mathcal{C}}}}$, in
$\operatorname{\mathcal{C}}$ and _intrinsic_ parameters that enable
perspective projection of
$\operatorname{\mathbf{x}^{\operatorname{\mathcal{C}}}}$ to a 2D point
$\mathbf{u}$ in the camera’s image plane. The intrinsic parameters include the
focal length of the camera lens ($f$), the pixel densities along the two image
dimensions ($m_{u},m_{v}$), and the position of the principal point
($p_{u_{0}},p_{v_{0}}$) through which the optical axis emanates. Any point
$\operatorname{\mathbf{x}^{\operatorname{\mathcal{C}}}}$ is projected to

$\displaystyle p_{u}=m_{u}f\frac{y^{c}}{x^{c}}+p_{u_{0}}$ (12) $\displaystyle
p_{v}=m_{v}f\frac{z^{c}}{x^{c}}+p_{v_{0}}.$ (13)

The basis of intrinsic camera calibration is that the homogeneous
representation of the image projected point $[\mathbf{u},1]^{T}$ is equivalent
to the 3D point $\bar{\mathbf{u}}=[x^{c}\;x^{c}p_{u}\;x^{c}p_{v}]^{T}$. Thus,

$\displaystyle\begin{bmatrix}\mathbf{u}\\\
1\end{bmatrix}\equiv\bar{\mathbf{u}}=\begin{bmatrix}1&0&0&0\\\
p_{u_{0}}&m_{u}f&0&0\\\
p_{v_{0}}&0&m_{v}f&0\end{bmatrix}\begin{bmatrix}\operatorname{\mathbf{x}^{\operatorname{\mathcal{C}}}}\\\
1\end{bmatrix}=\mathbf{M}^{int}_{3\times
4}\operatorname{\mathbf{\bar{x}}^{\operatorname{\mathcal{C}}}},$ (14)

where $\mathbf{M}^{int}_{3\times 4}$ is the intrinsic calibration matrix that
must be estimated and
$\operatorname{\mathbf{\bar{x}}^{\operatorname{\mathcal{C}}}}$ is the
homogeneous representation of
$\operatorname{\mathbf{x}^{\operatorname{\mathcal{C}}}}$. The extrinsic
parameters enable the alignment of $\operatorname{\mathcal{C}}$ with
$\operatorname{\mathcal{R}}$ through the extrinsic matrix
$\mathbf{M}^{ext}_{4\times 4}$. This comprises the Euler rotation matrix
$\textbf{R}^{\operatorname{\mathcal{C}}\operatorname{\mathcal{R}}}$ that
aligns the axes between the two coordinate frames; and the translation vector
$\mathbf{t}^{\operatorname{\mathcal{C}}\operatorname{\mathcal{R}}}$ between
the origins of the two coordinate frames. Thus,

$\displaystyle\operatorname{\mathbf{\bar{x}}^{\operatorname{\mathcal{C}}}}=\begin{bmatrix}\mathbf{R}^{\operatorname{\mathcal{C}}\operatorname{\mathcal{R}}}_{3\times
3}&\mathbf{t}^{\operatorname{\mathcal{C}}\operatorname{\mathcal{R}}}\\\
\mathbf{0}_{3\times
1}&1\end{bmatrix}\operatorname{\mathbf{\bar{x}}}=\mathbf{M}^{ext}_{4\times
4}\operatorname{\mathbf{\bar{x}}}.$ (15)

The intrinsic and extrinsic matrices are combined to form the projection
matrix $\mathbf{P}^{cal}_{4\times 4}$ which converts a 3D point $\mathbf{x}$
to 2D $\mathbf{u}$ through

$\displaystyle\begin{bmatrix}\mathbf{u}\\\
1\end{bmatrix}\equiv\bar{\mathbf{u}}=\mathbf{M}^{int}_{3\times
4}\mathbf{M}^{ext}_{4\times
4}\operatorname{\mathbf{\bar{x}}}=\mathbf{P}^{cal}_{4\times
4}\operatorname{\mathbf{\bar{x}}}.$ (16)

In this work, both the intrinsic and extrinsic parameters are estimated using
the popular open source calibration algorithm involving a checkerboard
patterned target that is viewed from the camera from two different
orientations [39]. We have restricted the above discussion to conventional
cameras. If wide-angle fish-eye cameras are to be used instead of the
conventional cameras, additional calibration steps are required to resolve
radial distortion errors [40].
_Object Detection_ Detection of an object (vehicle or pedestrian) within a
camera’s image has been tackled using both traditional machine learning [21]
and deep learning techniques using convolutional neural networks and their
variants [41, 42, 43]. The main challenge in these algorithms is to handle the
multiple resolutions and orientations of an object in the image plane. Hence
the object detection methods essentially trade-off between detection accuracy
specified through the number of false positives per image and the
computational cost for capturing the multi-resolution image features.

Figure 3: Results from object (auto-rickshaw) detection based on [23] (a)
without false alarm, (b) with false alarm.

For example, in Fig.3, the objective is to detect and segment the vehicle
(auto-rickshaw) in the images as the vehicle moves and turns. In Fig.3a, the
object was correctly detected and segmented. However, in Fig.3b an additional
object was also detected which corresponds to a false positive. We use open
source machine learning-based object detection code provided with [23]. The
key insight of this algorithm is that it exploits the fractal statistics in
natural images, termed internal channel features, to reliably predict image
structures across multiple resolution scales which are termed aggregate
channel features. Since the image features are obtained through extrapolation
rather than explicit prediction, the overall computational complexity is
considerably reduced. The performance of the algorithm has been shown to be
very effective for automotive target detection (pedestrians and vehicles).

###  II-C Fusion of Camera and Radar Data

Once the camera projection matrix is learned it will be possible to estimate
$\operatorname{y_{k}},\operatorname{z_{k}}$ for a center point of a target $t$
in the image plane. However, since the transformation operation is inherently
lossy, we cannot recover the depth $\operatorname{x_{k}}$ (depth) from the
image point $\mathbf{u}_{k}$ at every $k$. In other words, any number of 3D
scenes can result in the same 2D image. Depth estimation from a single camera
image is considerably more challenging than stereo camera images and several
approaches have been proposed in literature including the use of machine
learning algorithms [24, 44, 45]. In this work, we supplement the camera data
with radar data to track the object.

We model the automotive target motion using the curvilinear state model based
on constant turn rate and velocity (CTRV) as described by [46, 47]. Here, the
state vector of the vehicle at any time instant $k$ is given by
$\mathbf{x}_{k}=[\operatorname{x_{k}};\operatorname{y_{k}};\operatorname{v_{x_{k}}};\operatorname{v_{y_{k}}};\omega_{k}]^{T}$
where $(\operatorname{v_{x_{k}}},\operatorname{v_{y_{k}}})$ describe the 2D
velocity in Cartesian coordinates. Note that we also obtain the
$\operatorname{z_{k}}$ from the longitudinal position estimates from the
camera image and the elevation estimates from radar interferometry. However,
we do not use those measurements since they do not have a bearing on the
computation of the turning velocity and very weakly impact the translation
motion compensation. They are useful, however, for correct data association
from the two sensors in scenarios where there are multiple targets. The state
at $k+1$ time step is predicted based on

$\displaystyle\mathbf{x}_{k+1}=\mathbf{f}\left(\mathbf{x}_{k}\right)+\mathbf{G}\left(\mathbf{x}_{k}\right)\mathbf{w}_{k},$
(17)

where $\mathbf{f}(\cdot)$ is a vector-valued function that models the state
transition and is given by

$\displaystyle\mathbf{f}(\mathbf{x}_{k})=\left[\begin{array}[]{c}x_{k}+\frac{v_{x_{k}}}{\omega_{k}}\sin(\omega_{k}T)-\frac{v_{y_{k}}}{\omega_{k}}(1-\cos(\omega_{k}T))\\\
y_{k}+\frac{v_{x_{k}}}{\omega_{k}}(1-\cos(\omega_{k}T))+\frac{v_{y_{k}}}{\omega_{k}}\sin(\omega_{k}T)\\\
v_{x_{k}}\cos(\omega_{k}T)-v_{y_{k}}\sin(\omega_{k}T)\\\
v_{x_{k}}\sin(\omega_{k}T)+v_{y_{k}}\cos(\omega_{k}T)\\\
\omega_{k}\end{array}\right],$ (23)

and $T$ is the time interval between updates (CPI).
$\mathbf{G}(\mathbf{x}_{k})\mathbf{w}_{k}$ in (17) is the process noise matrix
with $\mathbf{Q}$ process noise covariance matrix. In a CTRV model, the speed
and the turn rate are assumed to be nearly constant and we model the
acceleration and rate of change of yaw rate (which form vector
$\mathbf{w}_{k}$) as noise processes with zero mean and $\sigma_{a}$ and
$\sigma_{\alpha}$ standard deviations respectively. $\mathbf{G}$ models the
relationship between the target state and $\mathbf{w}_{k}$ and is given by

$\displaystyle\mathbf{G}=\left[\begin{array}[]{cc}\frac{T^{2}}{2}\cos(\omega_{k}T)&0\\\
\frac{T^{2}}{2}\sin(\omega_{k}T)&0\\\ T\cos(\omega_{k}T)&0\\\
T\sin(\omega_{k}T)&0\\\ 0&T\end{array}\right].$ (29)

The target space is updated based on measurements vector,
$\mathbf{z}_{k}=[r_{k}\;f_{D_{k}}\;p_{v_{k}}]^{T}$, provided by the automotive
radar and the camera. The measurements are mapped to the state space through

$\displaystyle\mathbf{z}_{k}=\mathbf{h}(\mathbf{x}_{k})+\mathbf{\nu}_{k},$
(30)

where $\mathbf{h}(\cdot)$ is the function between the measurements and state
space and $\mathbf{\nu}_{k}$ is the observation noise vector with $\mathbf{R}$
measurement noise covariance matrix. The radar provides range and Doppler
frequency measurements, $[r_{k},f_{D_{k}}]$ which are non-linear functions of
the position and velocity of the target as shown in

$\displaystyle r_{k}=\sqrt{x_{k}^{2}+y_{k}^{2}}$ (31) $\displaystyle
f_{D_{k}}=\frac{2f_{c}}{c}\left(\frac{2x_{k}v_{x_{k}}}{r_{k}}+\frac{2y_{k}v_{y_{k}}}{r_{k}}\right).$
(32)

The corresponding observation noise for the radar range and Doppler frequency
estimates are governed by the radar bandwidth and CPI respectively. They are
modeled, therefore, as uncorrelated random variables with zero mean and
standard deviations which are directly proportional to the corresponding
resolutions. The camera coordinate, $p_{v_{k}}$, depends on the focal length
and the pixel resolution of the camera as described in Section III.b
(equations (14) to (16)) while the standard deviation assumed for the process
noise is assumed to be half the width of the smallest bounding box required
for object detection. Due to the non-linear state space prediction equations
and their measurement equations, we used the extended Kalman filter framework
to estimate the state space [48]. The EKF is a nonlinear estimation algorithm
based on consecutive linearization of the equations carried out via first-
order Taylor series expansions. The EKF is preferred to other non-linear
estimation methods such as the second order extended Kalman filter [49],
unscented Kalman filter [50], and particle filter [51] due to its low
computational complexity ($\mathcal{O}(n^{3})$ versus $\mathcal{O}(n^{4})$).
Due to the approximations in the system equations, the EKF is known to
generate error and instability conditions for more complex target motions.
However, within the limited scope of our problem - a single vehicle undergoing
turns - the EKF performs satisfactorily. We summarize the EKF algorithm below.

  1. 1.

Initialize target state $x_{0/0}$ and error covariance $P_{0/0}$

  2. 2.

For $k<K$, predict the current state, error covariance, $\mathbf{P}$, and
Kalman gain, $\mathbf{K}$, from previous time estimates

$\displaystyle\mathbf{x}_{k/k-1}=\mathbf{f}(\mathbf{x}_{k-1/k-1})$ (33)
$\displaystyle\mathbf{P}_{k/k-1}=\mathbf{F}_{k-1}\mathbf{P}_{k-1/k-1}\mathbf{F}_{k-1}^{T}+\mathbf{Q}_{k-1}$
(34)
$\displaystyle\mathbf{K}_{k}=\mathbf{P}_{k/k-1}\mathbf{H}_{k}^{T}(\mathbf{H}\mathbf{P}_{k/k-1}\mathbf{H}_{k}^{T}+\mathbf{R}_{k})^{-1},$
(35)

where $\mathbf{F}$ and $\mathbf{H}$ are the Jacobian matrices for the vector-
valued functions $\mathbf{f}$ and $\mathbf{h}$ respectively with respect to
the state.

  3. 3.

Update the current state using current measurements using the Jacobian
computed from the non-linear equation and update the corresponding error
covariance

$\displaystyle\mathbf{x}_{k/k}=\mathbf{K}_{k}(\mathbf{z}_{k}-\mathbf{h}(\mathbf{x}_{k/k-1}))$
(36)
$\displaystyle\mathbf{P}_{k/k}=(1-\mathbf{K}_{k}\mathbf{H}_{k})\mathbf{P}_{k/k-1}$
(37)

  4. 4.

Repeat step 2 through 4 till the target trajectory is completed.

The Kalman gain, $\mathbf{K}$, is a $[3\times 5]$ matrix that controls the
impact of the three sensor measurements (range, Doppler velocity from radar,
and lateral position from camera) on the five-state target estimation (two-
dimensional position, two-dimensional velocity, and yaw rate) at each time
step. When the Kalman gain value $\mathbf{K}_{m,n}$ is high (close to 1), this
implies that the $n^{th}$ state variable is strongly correlated to the
$m^{th}$ measurement and that the measurement is reliable (compared to the
predictions). On the other hand, when $\mathbf{K}_{m,n}$ is low (close to
zero), this implies that the $n^{th}$ state variable is not correlated to the
$m^{th}$ measurement and should be updated based on predictions from its prior
state or from one of the other measurements. Usually, when there are no
instances of sensor failure, the Kalman gain values start from a high value,
when predictions are unreliable, and then converge to a lower value (say
around 0.5) when the estimates are based on prior state predictions and
current measurements. In the event, that one of the two sensors fails to
provide a detection due to malfunction or because the target is outside its
FoV, then the corresponding $\mathbf{K}_{m,n}$ is set as 0 and the
measurements from the other sensor are used to correct the state predictions.
For example, if the camera fails, then $\mathbf{K}_{3,n}=0$ for $n=1:5$ and
$\mathbf{z}$ only consists of range and Doppler velocity measurements. In the
case both sensors fail to produce measurements at any particular time step,
then the states are updated based on prior predictions and not further
corrected. Subsequently, the corrected target state is used for translation
motion compensation of the range-Doppler ambiguity plot and for mapping the
range-crossrange ISAR image at $k+1$. Note that in this work we have confined
the problem to a single target in the FoV of the sensors. Hence, target
detections (from either of the sensor) at timestamp $k+1$ that do not fall
within a predefined neighborhood of the estimated position of the target are
discarded. In this manner, false alarms of both radar and camera are reduced.
In scenarios with multiple targets, joint probabilistic data association and
multiple hypothesis testing will have to be incorporated within the Kalman
framework. The discussion on multiple target tracking is outside the scope of
this paper.

##  III Simulation Model and Results

In this section, we present the simulation model of a camera and radar
tracking a dynamic target undertaking turns. The data from the two sensors are
fused and the target state is estimated which is subsequently used to generate
the corresponding ISAR images. The entire simulation is carried out in MATLAB
using the automated driving toolbox.

###  III-A Experimental Set Up

_Road Geometry:_ We consider a four way flat road junction geometry as shown
in Fig.4i.a-d. The north-south road, aligned along the X axis of
$\operatorname{\mathcal{R}}$ is perpendicular to the east-west road which is
aligned to the Y axis. The height axis above the road thus corresponds to the
Z axis. The origin is assumed at the lower south-west corner of the figure.
Each road is a 78m long dual carriageway that allows traffic in both
directions. The solid yellow lines in the figure indicate the road divider
separating the bi-directional traffic flow in each road segment while the
dashed yellow lines demarcate the lanes within each section. Each lane is 3.6m
wide as per common road design standards. An additional lane is provided for
onward traffic in order to facilitate right turns (south to east, east to
north, and so forth). For example, the left section of the south road segment
consists of three lanes while the right section consists of two lanes. So a
vehicle that turns from south to east will move from the right-most lane of
the left section of the south road to the lowest lane of the upper section of
the east road. We assume line-of-sight conditions with no building or other
forms of occlusions in the road layout.

_Sensor Specifications:_ We assume that the ego vehicle upon which the radar
and camera sensors are mounted is not moving. We assume that the ego vehicle
is a simple cuboid of $[4.7\times 1.8\times 1.4]m$ dimensions with its length
oriented along the direction of the lane. Thus the yaw (angle of the vehicle
with respect to the positive X axis) is zero. The center point of the ego
vehicle is located in the south road segment at $[10,42.6,0.7]m$. Forward-
looking radar and camera are mounted on the ego vehicle as indicated in
Fig.4i.a-d. The camera is assumed to be located 0.1m below the top of the ego
vehicle and at 0.7m of the wheelbase of the vehicle. The camera’s intrinsic
parameters are the following- the focal lengths along $p_{u}$ and $p_{v}$ are
$[800,800]$, the image sizes are ($[480,640]$) and the principal point is
located at ($[320,240]$). The camera’s yaw, pitch, and roll are all
$0^{\circ}$. The toolbox’s vision sensor is configured to detect targets up to
a range of 100m with a detection probability of 90% and a statistical
probability of false positives per image of 10%. The minimum object size in
the image plane that can be detected is $[15\times 15]$ pixels.
Next, we discuss the radar model. We assume that the radar is transmitting a
linear frequency modulated wave as discussed in Section II. The carrier
frequency is 77GHz and the chirp rate is $60\times 10^{12}Hz/sec$. The pulse
repetition frequency of the radar is 40kHz resulting in a maximum unambiguous
range of 40m and the range resolution is 0.1m. We assume that the CPI for the
radar is 0.1s which also corresponds to the frame rate of the camera. In other
words, the camera and radar data are time stamped every 0.1s. The radar is
located 0.1m above the ground at the front overhang of the ego vehicle. The
yaw, roll, and pitch of the radar are all zero. Therefore, in this simulation
set up the axes of $\operatorname{\mathcal{C}}$ and
$\operatorname{\mathcal{R}}$ are perfectly aligned though they have different
origins. The radar is characterized by a false alarm rate of $1\times 10^{-6}$
and a detection probability of 90%. The FoV of the radar is $120^{\circ}$
along the azimuth and $90^{\circ}$ along the elevation. Figure.4i.a-d shows
the coverage area of the camera and the radar. Note that this figure is not
drawn perfectly to scale and is provided to give a general notion of the
coverage of the sensors.
_Target Details:_ We model the vehicle as a cuboid of dimensions $[4.7m\times
1.8m\times 1.4m]$ which is the size of a typical mid-size car. The wheelbase
for this car is 2.8m with a rear overhang of 1m and the front overhang of
0.9m. The target is modeled as an extended target with 12 triangular
faces/mesh (2 per side). The radar cross-section (RCS) of each mesh is
calculated based on the analytical expression for RCS of metal triangle [12].
We consider four possible turn trajectories for the target as indicated in
Fig.4a-d. In the first trajectory, the target starts in the south lane at
$[20m,39.5m,0m]$, moves forward and performs a U-turn, and then returns to the
south lane at $[20m,35.4m,0m]$. Thus the target is mostly within the FoV of
both sensors. Next, we will consider the trajectory shown in Fig.4i.b. Here,
the target starts in the north lane at $[58m,39m,0m]$ and then comes south-
wards and performs a U-turn, and then returns to the north lane at
$[58m,42.6m,0m]$. The target lies within the camera’s FoV for almost the
entire duration and is within the radar’s FoV while it is performing the turn.
The next trajectory that we consider is the target turning right from east to
north as shown in Fig.4i.c from $[39m,20m,0m]$ to $[58m,42.6m,0m]$. Again, the
target is initially outside the FoV of the camera but eventually enters it.
The opposite is true with respect to the radar. The last trajectory is when
the target turns right from the west ($[39m,58m,0m]$) to the south
($[20m,35.4m,0m]$). The target is mostly within the FoV of both the sensors
for the most part. We will refer to these four trajectories as SSUT, NNUT,
ENRT, and WSRT henceforth. The duration of motion for each case is 6 seconds
and the speed of the vehicle is $6m/s$.

###  III-B Simulation Results

In this section, we discuss the effectiveness of sensor fusion for target
tracking and ISAR imaging. We present the estimated position
($\operatorname{x_{k}},\operatorname{y_{k}}$) and yaw rate ($\omega_{k}$) of
the target based on the camera-radar detections for the four trajectories in
the second row of Fig.4.

Figure 4: Top row shows the simulation setup for radar-camera sensor fusion
and target trajectories along (a) south lane to south lane U-turn (SSUT), (b)
north lane to north lane U-turn (NNUT), (c) east lane to north lane right turn
(ENRT), and (d) west lane to south lane right turn (WSRT). The second row
shows the corresponding target trajectory estimates from sensor fusion for (e)
SSUT, (f) NNUT, (g) ENRT, and (h) WSRT. The third row shows the Kalman gain
for position estimates based on sensor measurements (radar range, Doppler
velocity, and camera-based lateral position of object).

The third row shows the Kalman gain over the duration of the trajectory. From
the results in the third row, it is evident that $\mathbf{K}_{1,1}$ shows a
strong correlation between the radar range measurement and
$\operatorname{x_{k}}$ while $\mathbf{K}_{3,2}$ shows the strong correlation
between the horizontal/lateral position of the target in the camera’s image to
$\operatorname{y_{k}}$. Both of these measurements are not directly correlated
to the Doppler velocity and hence $\mathbf{K}_{2,1}$ and $\mathbf{K}_{2,2}$
are zero throughout the duration of the trajectory.

SSUT: Since the target falls within the FoV of both the sensors for most of
the duration of the trajectory the Kalman gain plot is fairly stable in
Fig.4iii.a. The target is also close to the sensors. This results in strong
SNR in the case of radar and a large object for detection by the camera. Due
to all of these factors, the estimation of the target state is fairly accurate
in this trajectory.

NNUT: Here, the target object is farther from both the sensors. First, it is
outside the FoV of the radar, which is reflected in the zero values of
$\mathbf{K}_{1,1}$ and $\mathbf{K}_{1,2}$ and results in large errors in the
early estimates of $\operatorname{x_{k}}$. Later, even when it is within the
radar FoV, there is a discrepancy of approximately $2m$ between
$\operatorname{x_{k}}$ and the ground truth. This is because the camera image
and radar locate the centroid of the vehicle at the midpoint of its front
end/rear end (whichever is facing the sensors) rather than the center of the
vehicle. There are also instants when the camera does not detect the object
due to smaller perspective spans in the image. This effect is noticeable in
the pattern of $\mathbf{K}_{3,2}$. However, it does not cause a large drop in
the accuracy of the camera estimates.

ENRT: Here, both $\operatorname{x_{k}}$ and $\operatorname{y_{k}}$ are
inaccurately estimated at the beginning when the vehicle is outside of the FoV
of both sensors. This is also reflected in the Kalman gain plot in Fig.4iii.c.
Later, the results improve while the target remains in the FoV of both
sensors. Towards the end of the trajectory, the vehicle is outside the radar
FoV which causes a deterioration in the estimate of $\operatorname{x_{k}}$.
Similar observations can be made for the last case corresponding to WSRT. In
all four cases, we observe that the yaw rate estimate is reasonably accurate
when the target is within the FoV of both sensors. In the above discussion, we
have assumed that both the sensors are performing with high detection metrics
(above 90%). In the appendix, we discuss the sensor fusion performance when
the detection metrics of both sensors are poor due to either environmental
conditions or instrument defects.

Next, we discuss the results obtained for ISAR imaging of the vehicle. The
results for all four trajectories are presented in Fig.5.

Figure 5: ISAR images from simulated data with motion compensation based on:
(i-iv) target state estimation from camera-radar fusion; and (v-vi) ground
truth target state. Rows correspond to (a) south to south U turn (SSUT), (b)
north to north U-turn (NNUT), (c) east to north right turn (ENRT) and (d) west
to south right turn (WSRT).

The ISAR images are generated after motion compensation using turning velocity
estimates obtained from the radar-camera fused data. These are presented in
the first four columns (left four) corresponding to Fig.5a-d. These images are
then compared with those obtained from ground truth information of the target
trajectory. These are presented in the last four columns of each row (right
four) corresponding to Fig.5e-h. All the figures span 40m along range and 20m
along crossrange and are mirrored along the crossrange to keep positive sense
of the Y axis. Note that we may not be able to generate images even with the
perfect knowledge of the ground truth target information. If the vehicle
undergoes very little yaw with respect to the ego vehicle upon which the
sensor is mounted, then the turning velocity is very low resulting in
extremely high cross-range resolutions.

Trajectory | GT images (#) | Fused images (#) | SSIM (%)
---|---|---|---
South to South U Turn(SSUT) | 56 | 40 | 97.4
North to North U Turn(NNUT) | 6 | 15 | 85.7
West to South Right Turn(WSRT) | 54 | 46 | 97.5
East to North Right Turn(ENRT) | 52 | 40 | 97.5
TABLE I: Comparison of fused ISAR images with ground truth (GT) images.

For example, in our problem, if the turning velocity is below $0.01rad/s$,
then the resulting crossrange resolution is $1.98m$ which is greater than the
width of the vehicle and hence not useful for imaging. Hence, we do not
generate images for very low values of turning velocity. The images obtained
from the sensor fused data are compared with the ground truth data through the
popular image comparison metric called the structural similarity index (SSIM)
[52]. The summary of the ISAR image results is presented in Table.I.

We first consider the results from SSUT trajectory along the first row. Here,
due to the orientation of the vehicle with respect to the ego vehicle as shown
in Fig.4a, we see only one long side and the rear end of the vehicle at
different time instants of the target trajectory. The other two sides of the
vehicle are occluded from the sensors. As the target moves along a straight
line to the right of the vehicle, it still presents a variation in the target
aspect resulting in a change in the yaw. Hence, we are able to generate 56
ISAR images out of the total 60 possible intervals in the duration of the
vehicular motion. For higher turning velocity, the crossrange resolution is
finer and that is reflected in the thickness of the pixels across the
different time instants. With the sensor fused data, we are able to generate
40 ISAR images because there are some time instants where the predicted
turning velocity becomes very low. However, these images are very similar for
the most part to the ground truth images as evidenced by the high SSIM score.

Next we discuss the results from NNUT trajectory. In this trajectory, the
target object is directly before the ego vehicle. Hence, for most of its
trajectory, the yaw with respect to the sensors is zero. Hence, the turning
velocity is infinite and we cannot generate ISAR images. Out of the 60 time
intervals, it is possible to generate images for only 6 intervals when the
vehicle is actually undergoing the turn. The number of ISAR images generated
from the sensor fused data are higher in this case because of higher
(incorrectly) estimated values of the turning velocity. Secondly, as mentioned
earlier, there is some error in both sensor estimates since we do not
correctly estimate the position of the center of the vehicle and instead track
the center of the front side of the vehicle. Due to these reasons, we observe
that some of the ISAR images generated from the sensor fused data show wide
spread across the crossrange, especially in comparison to the ground truth.
This is reflected in the lower mean SSIM score reported in Table.I. However,
we are still able to observe (for the most part) the size and shape of the
vehicle.

Next we consider the two right turn trajectories - ENRT and WSRT. In both
cases, the turning velocity from ground truth and the estimate are fairly high
for most of the duration of the vehicle and we are able to generate a large
number of ISAR images. Interestingly, the number of images from ground truth
and sensor fusion are exactly equal in both cases because the estimation of
the target state along these two trajectories is fairly accurate as shown in
Fig.4. Due to occlusion, different sides of the target cuboid are visible to
the sensors for each case. Based on how the orientation of these images
change, we can make inferences regarding the trajectory of the target. Due to
the close agreement between the sensor fused images and ground truth images,
we obtain high mean SSIM scores as reported in Table.I

##  IV Measurement Setup and Results

In this section, we present the measurement setup to collect experimental data
from Texas Instruments AWR-1843 77GHz millimeter wave radar and a smartphone
camera of a target vehicle as described in [13]. The radar parameters are
configured as per Table.II.

TABLE II: Automotive radar parameters for generating ISAR images Parameters | Values
---|---
Carrier frequency ($f_{c}$) | 77GHz
Range resolution | 0.075m
Maximum unambiguous range | 100m
Radar bandwidth | 2GHz
Chirp duration ($T_{PRI}$) | 400$\mu$s
Coherent processing interval ($T_{CPI}$) | 0.1s
Doppler resolution | 10 Hz
Minimum cross-range resolution | 0.19 m
Transmitted power ($P_{t}$) | 25dBm

In the measurement setup, the radar is mounted on a static platform which is
assumed to be located at the origin of $\operatorname{\mathcal{R}}$ frames.
The radar images are generated for every CPI interval of 0.1 seconds. Here,
the radar consists of only a single channel and hence we do not obtain
estimates of the height of the vehicle. A smartphone camera of $1920\times
1080$ pixel size is mounted at $[2m,-2m,0.5m]$ and is oriented with yaw of
$-45^{\circ}$ with respect to the radar axis to ensure that the target object
remains within the camera FoV during the entire duration. The camera frame
rate is 30 frames per second and is time stamped.

Figure 6: Measurement setup of camera radar data collection of a small size
car executing a left turn

We select the images corresponding to the 0.1 second interval. Hence, unlike
the simulation data, the measurement data from the two sensors are not
perfectly synchronized. The camera is calibrated to the radar’s coordinate
frame based on the methods discussed earlier. We consider a Honda Brio car
which is a small size car of $[3.6m\times 1.68m\times 1.5m]$ dimensions. The
car undertakes a left turn from the north to the east in front of the radar as
shown in Fig.6.

We present the measurement results for six frames in Fig.7.

Figure 7: (a)i-vi: Camera images of a small size car executing a left turn
from north to the east before a radar. (b)i-vi: Range-Doppler plots generated
from two-dimensional Fourier transform of slow-time fast-time radar measured
data from TI AWR-1843. (c) ISAR images were obtained after target state
estimation from camera-radar fused data.

The top row shows the camera images with the object within the bounding box.
Due to the darker background, we applied lightening filters to improve the
effectiveness of object detection. Based on the camera estimates and the radar
estimates, we track the trajectory of the centroid of the target. The second
row of Fig.7(b)i-vi shows the range-Doppler plots obtained from the radar
without any type of motion compensation. In order to improve the clarity of
the images, we have cropped the range-Doppler plot to show the region within
$\pm 6m$ and $\pm 300Hz$ of the range and Doppler, respectively, corresponding
to the peak scatterer on the vehicle. The axes of these images change based on
the range and radial velocity of the vehicle with respect to the radar. Using
the translational motion compensation steps outlined earlier, we generate the
corresponding ISAR images shown in the third row. Additionally, CFAR algorithm
is also implemented on the raw data to improve the noise performance. The
crossrange axis is obtained from the estimate of the turning velocity. From
the ISAR images, we are clearly able to observe the orientation of the vehicle
change from width-wise (when it is approaching the radar) in (c)i-iii to
length-wise (when it is parallel to the radar Y axis) in (c)iv-vi. One or more
side edges of the vehicle can be observed in each of the images while the
remaining edges are occluded. With these images, we able to obtain a fairly
good estimate of the top-view (bird’s eye view) dimensions of the car. Unlike
the simulation results presented in Fig.2, all four edges are not visible.
Also, the micro-Dopplers from the wheels are very weak in this car.

We observe that the ISAR images are more focused compared to the range-Doppler
plots due to the motion compensation. Specifically, compare the regions red
circled in (c)i-iii with the corresponding regions in (b)i-iii. The point
scatterers are better defined in the ISAR images than in the range-Doppler
plots. Across the ISAR images, the crossrange resolution changes with the
turning velocity. This is reflected in the slight change in the cell sizes in
the images. Even when the car is parallel to the radar Y axis, we observe some
features from the back of the car possibly due to the transparent windows. The
figures demonstrate the effectiveness of sensor fusion for estimating the
turning velocity and translation motion compensation of the vehicle for
generating high-resolution ISAR images.

##  V Conclusion

At short ranges, wideband radars capture multiple detections from a spatially
large extended radar target. When these targets undergo rotation within the
FoV of the radar, large synthetic apertures are generated for a single radar
channel. The wideband long-duration data can subsequently be processed to form
ISAR images that provide a top-view or bird’s eye view of the vehicle. In
conjunction with the front view of camera images, these images can potentially
facilitate complete 3D visualizations of an object. However, translation
motion compensation must be performed and the Doppler to crossrange axis
mapping must be carried out in order to generate meaningful ISAR images. While
prior works have focused on blind motion compensation techniques, here, we
leverage the availability of vision data in conjunction with radar data for
estimating the target state. Specifically, vision data facilitate accurate
object detection of the vehicles in the sensor FoV which is useful for
eliminating clutter and multipath. Second, they facilitate accurate
predictions of the lateral position of the target while radar data enable
range estimates. Our work has assumed a constant turn rate model of the target
which is typical in road turns. If on the other hand, the target undergoes
constant acceleration, then the constant turn rate and acceleration model can
be utilized in the extended Kalman framework. Once, the acceleration is
computed, then using the centroid tracking method, motion compensation can be
carried out by incorporating higher-order components for range alignment. The
performance of the sensor fusion for ISAR imaging is evaluated with simulation
data generated from a cuboid model of a moving vehicle and from measurements
carried out with a millimeter wave radar and a smartphone camera. The main
challenges in the proposed approach lie in handling scenarios where the target
moves outside the FoV of either one of the sensors. With suitable upgrades to
the sensor tracking model to handle missing data, we believe that complete 3D
imaging can be carried out even when the target disappears temporarily from
the FoV of any one of the sensors. This will form the focus of our future
work.

## References

  * [1] J. Hasch, E. Topak, R. Schnabel, T. Zwick, R. Weigel, and C. Waldschmidt, “Millimeter-wave technology for automotive radar sensors in the 77 ghz frequency band,” _IEEE Transactions on Microwave Theory and Techniques_ , vol. 60, no. 3, pp. 845–860, 2012.
  * [2] V. C. Chen, _Inverse Synthetic Aperture Radar Imaging; Principles_. Institution of Engineering and Technology, 2014\.
  * [3] M. Martorella, E. Giusti, A. Capria, F. Berizzi, and B. Bates, “Automatic target recognition by means of polarimetric isar images and neural networks,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 47, no. 11, pp. 3786–3794, 2009.
  * [4] M. Vespe, C. Baker, and H. Griffiths, “Automatic target recognition using multi-diversity radar,” _IET Radar, Sonar & Navigation_, vol. 1, no. 6, pp. 470–478, 2007.
  * [5] S.-H. Park, M.-G. Joo, and K.-T. Kim, “Construction of isar training database for automatic target recognition,” _Journal of Electromagnetic Waves and Applications_ , vol. 25, no. 11-12, pp. 1493–1503, 2011.
  * [6] Y. Wang and B. Zhao, “Inverse synthetic aperture radar imaging of nonuniformly rotating target based on the parameters estimation of multicomponent quadratic frequency-modulated signals,” _IEEE Sensors Journal_ , vol. 15, no. 7, pp. 4053–4061, 2015.
  * [7] Y. Sun, Z. Liu, S. Todorovic, and J. Li, “Adaptive boosting for sar automatic target recognition,” _IEEE Transactions on Aerospace and Electronic Systems_ , vol. 43, no. 1, pp. 112–125, 2007.
  * [8] J. S. Kulpa, M. Malanowski, D. Gromek, P. Samczyńsk, K. Kulpa, and A. Gromek, “Experimental results of high-resolution isar imaging of ground-moving vehicles with a stationary fmcw radar,” _International Journal of Electronics and Telecommunications_ , vol. 59, no. 3, pp. 293–299, 2013\.
  * [9] C. J. Li and H. Ling, “Wide-angle isar imaging of vehicles,” in _2015 9th European Conference on Antennas and Propagation (EuCAP)_. IEEE, 2015, pp. 1–2.
  * [10] H. Essen, M. Hagelen, W. Johannes, R. Sommer, A. Wahlen, M. Schlechtweg, and A. Tessmann, “High resolution millimetre wave measurement radars for ground based sar and isar imaging,” in _2008 IEEE Radar Conference_. IEEE, 2008, pp. 1–5.
  * [11] S. Gishkori, L. Daniel, M. Gashinova, and B. Mulgrew, “Imaging moving targets for a forward scanning sar without radar motion compensation,” _Signal Processing_ , vol. 185, p. 108110, 2021.
  * [12] N. Pandey, G. Duggal, and S. S. Ram, “Database of simulated inverse synthetic aperture radar images for short range automotive radar,” in _2020 IEEE International Radar Conference (RADAR)_. IEEE, 2020, pp. 238–243.
  * [13] N. Pandey and S. S. Ram, “Classification of automotive targets using inverse synthetic aperture radar images,” _IEEE Transactions on Intelligent Vehicles_ , 2022.
  * [14] H. Wu, D. Grenier, G. Y. Delisle, and D.-G. Fang, “Translational motion compensation in isar image processing,” _IEEE Transactions on image processing_ , vol. 4, no. 11, pp. 1561–1571, 1995.
  * [15] T. Itoh, H. Sueda, and Y. Watanabe, “Motion compensation for isar via centroid tracking,” _IEEE Transactions on Aerospace and Electronic Systems_ , vol. 32, no. 3, pp. 1191–1197, 1996.
  * [16] Y. Wang, H. Ling, and V. C. Chen, “Isar motion compensation via adaptive joint time-frequency technique,” _IEEE Transactions on Aerospace and Electronic Systems_ , vol. 34, no. 2, pp. 670–677, 1998.
  * [17] M. Xing, R. Wu, J. Lan, and Z. Bao, “Migration through resolution cell compensation in isar imaging,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 1, no. 2, pp. 141–144, 2004.
  * [18] D. Li, M. Zhan, H. Liu, Y. Liao, and G. Liao, “A robust translational motion compensation method for isar imaging based on keystone transform and fractional fourier transform under low snr environment,” _IEEE transactions on aerospace and electronic systems_ , vol. 53, no. 5, pp. 2140–2156, 2017.
  * [19] L. Liu, F. Zhou, M. Tao, P. Sun, and Z. Zhang, “Adaptive translational motion compensation method for isar imaging under low snr based on particle swarm optimization,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 8, no. 11, pp. 5146–5157, 2015.
  * [20] S. Shao, L. Zhang, and H. Liu, “High-resolution isar imaging and motion compensation with 2-d joint sparse reconstruction,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 58, no. 10, pp. 6791–6811, 2020.
  * [21] P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” in _Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001_ , vol. 1. Ieee, 2001, pp. I–I.
  * [22] P. Dollár, Z. Tu, P. Perona, and S. Belongie, “Integral channel features,” _Proceedings of the British Machine Vision Conference_ , 2009\.
  * [23] P. Dollár, R. Appel, S. Belongie, and P. Perona, “Fast feature pyramids for object detection,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 36, no. 8, pp. 1532–1545, 2014.
  * [24] A. Saxena, S. Chung, and A. Ng, “Learning depth from single monocular images,” _Advances in Neural Information Processing Systems_ , vol. 18, 2005\.
  * [25] D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” _International journal of computer vision_ , vol. 47, no. 1, pp. 7–42, 2002.
  * [26] R. O. Chavez-Garcia and O. Aycard, “Multiple sensor fusion and classification for moving object detection and tracking,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 17, no. 2, pp. 525–534, 2015.
  * [27] S. S. Ram, C. Christianson, Y. Kim, and H. Ling, “Simulation and analysis of human micro-dopplers in through-wall environments,” _IEEE Transactions on Geoscience and remote sensing_ , vol. 48, no. 4, pp. 2015–2023, 2010.
  * [28] S. Vishwakarma, A. Rafiq, and S. S. Ram, “Micro-doppler signatures of dynamic humans from around the corner radar,” in _2020 IEEE International Radar Conference (RADAR)_. IEEE, 2020, pp. 169–174.
  * [29] S. Vishwakarma and S. S. Ram, “Mitigation of through-wall distortions of frontal radar images using denoising autoencoders,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 58, no. 9, pp. 6650–6663, 2020.
  * [30] S. S. Ram, S. Vishwakarma, A. Sneh, and K. Yasmeen, “Sparsity based autoencoders for denoising cluttered radar signatures,” _IET Radar, Sonar & Navigation_, vol. 15, no. 8, pp. 915–931, 2021.
  * [31] Y. Fang, I. Masaki, and B. Horn, “Depth-based target segmentation for intelligent vehicles: Fusion of radar and binocular stereo,” _IEEE transactions on intelligent transportation systems_ , vol. 3, no. 3, pp. 196–202, 2002.
  * [32] G. Alessandretti, A. Broggi, and P. Cerri, “Vehicle and guard rail detection using radar and vision data fusion,” _IEEE transactions on intelligent transportation systems_ , vol. 8, no. 1, pp. 95–105, 2007.
  * [33] S. Wu, S. Decker, P. Chang, T. Camus, and J. Eledath, “Collision sensing by stereo vision and radar sensor fusion,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 10, no. 4, pp. 606–614, 2009.
  * [34] C.-C. Chen and H. C. Andrews, “Target-motion-induced radar imaging,” _IEEE Transactions on Aerospace and Electronic Systems_ , no. 1, pp. 2–14, 1980.
  * [35] G. Wang and Z. Bao, “The minimum entropy criterion of range alignment in isar motion compensation,” in _IET Radar Systems Conference_ , 1997.
  * [36] C. Ozdemir, R. Bhalla, L. Trintinalia, and H. Ling, “Asar-antenna synthetic aperture radar imaging,” _IEEE Transactions on Antennas and Propagation_ , vol. 46, no. 12, pp. 1845–1852, 1998.
  * [37] M. Shor and N. Levanon, “Performances of order statistics cfar,” _IEEE Transactions on Aerospace and Electronic Systems_ , vol. 27, no. 2, pp. 214–224, 1991.
  * [38] M. Weiss, “Analysis of some modified cell-averaging cfar processors in multiple-target situations,” _IEEE Transactions on Aerospace and Electronic Systems_ , no. 1, pp. 102–114, 1982.
  * [39] Z. Zhang, “A flexible new technique for camera calibration,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 22, no. 11, pp. 1330–1334, 2000.
  * [40] J. Kannala and S. Brandt, “A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 28, no. 8, pp. 1335–1340, 2006\.
  * [41] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2014, pp. 580–587.
  * [42] R. Girshick, “Fast r-cnn,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2015, pp. 1440–1448.
  * [43] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” _Advances in Neural Information Processing Systems_ , vol. 28, 2015.
  * [44] F. Liu, C. Shen, G. Lin, and I. Reid, “Learning depth from single monocular images using deep convolutional neural fields,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 38, no. 10, pp. 2024–2039, 2015\.
  * [45] Y. Cao, C. Shen, and H. T. Shen, “Exploiting depth from single monocular images for object detection and semantic segmentation,” _IEEE Transactions on Image Processing_ , vol. 26, no. 2, pp. 836–846, 2016.
  * [46] R. Schubert, E. Richter, and G. Wanielik, “Comparison and evaluation of advanced motion models for vehicle tracking,” in _2008 11th International Conference on Information Fusion_ , 2008, pp. 1–6.
  * [47] D. Svensson, “Derivation of the discrete-time constant turn rate and acceleration motion model,” in _2019 Sensor Data Fusion: Trends, Solutions, Applications (SDF)_ , 2019, pp. 1–5.
  * [48] S. S. Blackman and R. Popoli, _Design and analysis of modern tracking systems_. Artech House Publishers, 1999\.
  * [49] M. Nørgaard, N. K. Poulsen, and O. Ravn, “New developments in state estimation for nonlinear systems,” _Automatica_ , vol. 36, no. 11, pp. 1627–1638, 2000.
  * [50] S. Julier and J. Uhlmann, “Unscented filtering and nonlinear estimation,” _Proceedings of the IEEE_ , vol. 92, no. 3, pp. 401–422, 2004.
  * [51] J. Carpenter, P. Clifford, and P. Fearnhead, “Improved particle filter for nonlinear problems,” _IEE Proceedings-Radar, Sonar and Navigation_ , vol. 146, no. 1, pp. 2–7, 1999.
  * [52] A. Hore and D. Ziou, “Image quality metrics: Psnr vs. ssim,” in _2010 20th international conference on pattern recognition_. IEEE, 2010, pp. 2366–2369.

###  -A Performance Analysis of Fusion Algorithm in Complex Environments

Even when the target is within the FoV of a sensor, the target may not be
detected due to environmental conditions (poor light, fog, rain, presence of
high clutter) and instrumental defects. In this section, we analyze how the
detection performances of the sensors impact the state estimation of the
fusion algorithm. We consider the SSUT trajectory of the target vehicle since
the vehicle lies within the FoV of both the sensors for almost the entire
duration of its trajectory.

_Camera:_ First we consider the scenario when the radar is properly
functioning throughout the duration of the trajectory while the camera failure
is random. In other words, even when the target vehicle is within the camera’s
FoV, it will be detected only in some of the time frames. The results for
Kalman gain and the state estimation for $P_{d}=50\%,75\%$ and $99\%$ are
shown in Fig.8.

Figure 8: Left figures show the Kalman gain for position estimates based on
sensor measurements while the right figures show the position and yaw rate
state estimation. The top, middle, and bottom rows correspond to results
obtained with 50%, 75%, and 99% detection performance of the camera. Radar is
assumed to have a high probability of detection (above 90%) and a low false
alarm rate (below $10^{-7}$).

The top row shows the Kalman gain for position estimates and yaw as a function
of the three sensor measurements (range, Doppler and camera-based lateral
position of object). We observe that $\mathbf{K}_{1,1}$ is higher than
$\mathbf{K}_{1,2}$ indicating that along this particular trajectory, the range
is more strongly correlated to $\operatorname{x_{k}}$ of the target vehicle
than $\operatorname{y_{k}}$. The values never quite fall to zero since the
radar measurements are continuously obtained. On the other hand,
$\mathbf{K}_{2,1}$ and $\mathbf{K}_{2,2}$ are very low (close to zero)
throughout the duration of the trajectory indicating that the position of the
target is not strongly correlated to the Doppler velocity measurements.
$\mathbf{K}_{3,1}$ is also zero throughout the duration of the motion since
the camera does not directly provide depth information. However,
$\mathbf{K}_{3,2}$ is high because the lateral position estimated by the
camera is directly used to estimate $\operatorname{y_{k}}$. When the camera
detection fails, $\mathbf{K}_{3,2}$ falls to zero. The frequency with which it
falls to zero is greatest in Fig.8i.a when $P_{d}=50\%$ and lower for
Figs.8i.b and i.c.

Figure 9: Top row shows radar detections (red circles) within the radar
coverage area (red region), camera detections (blue plus) within the camera
coverage area (blue region), and estimated target centroid (green square). The
second row shows the Kalman gain for position estimates based on sensor
measurements. The third row shows the position and yaw rate estimation. The
columns a, b and c corresponds to radar detection metrics of (a)
$P_{D}=50\%,P_{fa}=10^{-5}$, (b) $P_{D}=75\%,P_{fa}=10^{-6}$ and (c)
$P_{D}=90\%,P_{fa}=10^{-7}$.

The corresponding state estimation results are presented in the second row.
The results show that despite the camera’s missed detections, the state
estimations (positions, yaw) are fairly accurate due to the reliance on
predictions from the prior state in the Kalman model.

_Radar:_ We incorporate discrete clutter scatterers in the radar FoV which are
assumed to be uniformly spatially distributed while the number of scatterers
follows the binomial distribution. Further, the detection of each clutter
scatterer in each resolution cell is assumed to be independent across time.
Since the radar performs high resolution ranging, we model its performance
through the probability of detection ($P_{d}$) of each point scatterer on the
target and the probability of false alarms ($P_{fa}$), which together are a
function of the SNR and clutter. The results are presented in Fig.9. The top
row of the figure shows the visualization of the radar and camera coverage
areas in red and blue respectively for a single time frame $0.05s$ for
different radar detection metrics. The radar provides multiple detections
shown as red circles while the camera provides a single detection shown as a
blue plus. The Kalman filtered two-dimensional position estimate of the target
vehicle is shown as a green square.

First, we consider the scenario where $P_{d}=50\%$ and the false alarm rate is
$P_{fa}=10^{-5}$ which corresponds to very low SNR resulting in lots of radar
detections as seen in Fig.9i-a. Th resulting Kalman gain plot in Fig.9i-b
shows that the radar-based range gain value ($\mathbf{K}_{1,1}$) is fairly
stable and not zero. Likewise for camera-based $\mathbf{K}_{3,2}$. The
remaining values are low. However, the state estimations in Fig.9i-c show high
error in $\operatorname{x_{k}}$ due to poor SNR. The $\operatorname{y_{k}}$ is
still fairly accurate due to the reliable measurements from the camera. This
results in overall deterioration in the yaw estimation. In Fig.9ii.a-c, the
radar detection metrics correspond to $P_{d}=75\%$ and $P_{fa}=10^{-6}$. This
results in fewer radar detections. The Kalman gain plot in Fig.9ii.b remains
largely unchanged but there is still a significant error in
$\operatorname{x_{k}}$ as seen in Fig.9ii.c. When the detection metrics
further improve ($P_{d}=90\%$ and $P_{fa}=10^{-7}$), then the number of radar
detections outside of the target are very few as observed in Fig.9iii.a and
the state estimation in Fig.9iii.c is very accurate. To summarize,
$\operatorname{x_{k}}$ is sensitive to the errors due to false alarms in the
case of radar while $\operatorname{y_{k}}$ is sensitive to missed detection of
the object by the camera. Thus both the sensors are indirectly required for
the accurate estimation of the yaw rate which is subsequently used for
generating the ISAR images.

## Acknowledgments

The research was funded by the MEITY CC&BT 13(30)/2020 grant from the Govt. of
India, Ministry of Electronics and Information Technology. S. S. R. gratefully
acknowledges support from Neeraj Pandey, doctoral student at IIIT Delhi for
the measurement data collection.

|  Shobha Sundar Ram is Associate Professor, Dept. of Electronics and
Communications Engineering, Indraprastha Institute of Information Technology
(IIIT), Delhi. She did her Bachelor of Technology in ECE from the University
of Madras, India in 2004 and then her Master of Science and Ph.D. in
electrical engineering from the University of Texas at Austin, USA in 2006 and
2009 respectively. Before joining IIIT Delhi, she worked as a research and
development electrical engineer at Baker Hughes Inc. USA. She is engaged in
research and education principally in the areas of radar signal processing and
electromagnetic sensor design and modeling. She is a Senior Member of IEEE, an
active member of the Aerospace and Electronics Systems Society and an
Associate Editor for the IEEE Transactions on Aerospace and Electronics
Systems.
---|---