# A Hybrid mmWave and Camera System for Long-Range Depth Imaging

Diana Zhang  Carnegie Mellon University<EMAIL_ADDRESS>,  Akarsh
Prabhakara  Carnegie Mellon University<EMAIL_ADDRESS>,  Sirajum
Munir  Bosch Research and Technology Center<EMAIL_ADDRESS>,
Aswin Sankaranarayanan  Carnegie Mellon University<EMAIL_ADDRESS>and
Swarun Kumar  Carnegie Mellon University<EMAIL_ADDRESS>
###### Abstract.

mmWave radars offer excellent depth resolution owing to their high bandwidth
at mmWave radio frequencies. Yet, they suffer intrinsically from poor angular
resolution, that is an order-of-magnitude worse than camera systems, and are
therefore not a capable 3-D imaging solution in isolation. We propose
Metamoran, a system that combines the complimentary strengths of radar and
camera systems to obtain depth images at high azimuthal resolutions at
distances of several tens of meters with high accuracy, all from a single
fixed vantage point. Metamoran enables rich long-range depth imaging outdoors
with applications to roadside safety infrastructure, surveillance and wide-
area mapping. Our key insight is to use the high azimuth resolution from
cameras using computer vision techniques, including image segmentation and
monocular depth estimation, to obtain object shapes and use these as priors
for our novel specular beamforming algorithm. We also design this algorithm to
work in cluttered environments with weak reflections and in partially occluded
scenarios. We perform a detailed evaluation of Metamoran’s depth imaging and
sensing capabilities in 200 diverse scenes at a major U.S. city. Our
evaluation shows that Metamoran estimates the depth of an object up to 60 m
away with a median error of 28 cm, an improvement of 13$\times$ compared to a
naive radar+camera baseline and 23$\times$ compared to monocular depth
estimation.

##  1\. Introduction

One of the most appealing features of mmWave radar systems arises from its
high bandwidth and carrier frequency, which enables precise depth estimation
at long depth ranges, often as large of 60 meters, and at cm-scale
resolutions. This finds application in a wide range of areas, including
security (Björklund et al., 2012), automobile safety (Wenger, 1998),
industrial sensing and control (Dandu et al., 2021). For comparison, most RGB
camera solutions of the same physical form-factor (e.g. monocular depth
estimation (Bhat et al., 2020), depth cameras (StereoLabs, 2020), stereo-
vision (Smolyanskiy et al., 2018), etc.) struggle to reach such resolutions
for objects at extended distances and are about an order-of-magnitude worse.
Yet, mmWave radars, by themselves, are not a capable 3-D imaging solution as
their angular resolution along both azimuth and elevation is extremely poor —
with the best radars of the market at least 10$\times$ poorer than camera
systems. This has led to mmWave radars being restricted to niche applications
– for instance, in airport security (Björklund et al., 2012) or physical
collision sensing (Wenger, 1998) — where their impressive depth range and
resolutions are not fully utilized. This naturally leads us to the question:
Can we fuse cameras and mmWave radar sensor data to provide the best of both
worlds and build a rich 3-D depth imaging solution?. In doing so, we seek a
3-D imaging system that can be readily deployed from a single fixed vantage
point to enable applications as long-range road-side safety systems,
surveillance and security applications, wide-area mapping and occupancy
sensing.

This paper presents Metamoran111A fictional race from the Dragon Ball Universe
that taught Son Goku the Fusion technique (Toriyama, 1994). , a hybrid mmWave
and camera-based sensing system that achieves high angular and depth
resolution for objects at significant distances – up to 60 meters (see Fig.
1). It achieves this through a novel specular radar processing algorithm that
takes information from computer vision algorithms such as deep neural network-
based image segmentation as input. While efforts have been made to fuse radar
and camera data in the past, primarily for short range object detection and
tracking (Chang et al., 2020), imaging under physical (Palffy et al., 2019) or
weather-related occlusions (Liu et al., 2021), this paper considers the unique
problem of hybrid mmWave/camera sensing for long-range outdoor depth imaging.

Figure 1. Metamoran devises a novel mmWave specular beamforming algorithm that
forms high resolution depth-images 60 m away from objects-of-interest, using
inputs from vision techniques such as image segmentation.

A key contribution in our system is improving depth sensing capabilities
beyond what is typically achievable by a mmWave radar alone using a novel
radar processing algorithm that provides high depth resolution (along the
$z$-axis) guided by computer vision techniques that have high spatial
resolution (in $x$ and $y$). First, we detect and identify an object using a
camera-based image segmentation algorithm, which gives us the angular position
(in the $x$-$y$ plane) of objects in the environment as well as their spatial
outline. Our key technical contribution is a novel specular radar beamforming
algorithm (see Sec. 6) that returns high-resolution depth estimates by
processing radar signals along the angular span and shape outline for each
object in the image identified using segmentation. We then show how such a
system could be combined with dense monocular depth estimates to create robust
depth images of individual objects, capturing depth variation within the
object itself, even at extended distances away from the radar-camera platform.
In other words, we show how semantic inferences on vision data collected by
the camera can help declutter and provide useful priors to obtain high-
resolution depth images that are better than standalone radar or camera
algorithms.

Our second contribution is to address various challenges in making Metamoran
robust in case of cluttered environments, unfavorable object orientation,
extended distances and partial occlusions that impede the radar, camera, or
both. We address this problem specifically for stationary objects: this form
radar’s worst-case scenario (Doppler can help to detect moving objects) as
objects that are not moving (e.g. traffic signs, parked cars, children at a
bus stop) can also be important to detect. We narrow down objects whose
spatial bounds are consistent across both camera and radar images, thereby
allowing for increased robustness by reducing clutter. We also observe strong
reflections from out-of-spatial bound reflectors leak into our spatial bound
of interest and design cancellation techniques to detect weak reflections
which would otherwise be masked by spurious objects. Further, we design and
show how Metamoran continues to operate well, even amid partial occlusions,
e.g. due to fog or partial occlusion from other objects. We document instances
where radar systems can actively be used to improve camera image segmentation
by identifying objects that were initially missed by segmentation.

We implement Metamoran with a TI MMWCAS-RF-EVM radar and a FLIR Blackfly S
24.5MP color camera. Due to the relative lack of rich public mmWave radar I/Q
datasets over long distances, we collected extensive data (200 scenes
totalling 100 GB of I/Q samples and camera data) in diverse scenes outdoors at
a major U.S. city. Both Metamoran’s source code and datasets will be made open
source upon paper acceptance to benefit the community. A few highlights from
our results include:

  * •

An evaluation of the effective median depth of an object-of-interest at
distances of up to 60 meters, in diverse outdoor settings, at a median error
of 28 cm. This is an improvement of about 23$\times$ versus state-of-the-art
monocular depth estimation and 13$\times$ versus a naive camera + radar
beamforming solution.

  * •

Dense estimation of the azimuthal/depth profile of a single object-of-
interest, for an imaging error of 80 cm at distances up to 60 meters. This is
an improvement of about 4$\times$ versus state-of-the-art monocular depth
estimation and 6$\times$ versus a naive camera + radar beamforming solution.

  * •

A demonstration of resilience to various classes of partial occlusions and
blockages.

Contributions:

We makes the following contributions.

  * •

Metamoran, a novel system that combines camera and mmWave sensing to achieve
high resolution depth images at long ranges.

  * •

A specular beamforming algorithm that leverages the output of image
segmentation algorithms from computer vision to declutter and retrieve depths
of objects-of-interest from radar I/Q samples.

  * •

A detailed implementation and evaluation of Metamoran in varied environments
to demonstrate substantial improvements in long range depth imaging.

Limitations:

We concede that our system is limited by more significant occlusions that
impact camera observations and discuss the limitations of our system in Sec.
11 as well as present an evaluation of both successful and failure modes with
various types of occlusions in our results in Sec. 10.

##  2\. Related Work

Wireless and Radar Depth Sensing:

Recent years have seen extensive work in sensing the environment through
wireless imaging (Huang et al., 2014; Depatla et al., 2017), location tracking
(Xie et al., 2015; Kotaru et al., 2015; Olivier et al., 2016; Wei and Zhang,
2015) and material sensing (Wu, 2016; Wang et al., 2017; Dhekne et al., 2018;
Nguyen et al., 2017; Karanam and Mostofi, 2017), with much of this work
limited to ranges of few tens of meters. Some prior work has also explored
high-resolution mmWave radar systems for through-wall/through-obstruction
imaging (Depatla et al., 2017; Guan et al., 2020), security scanning (Stanko
et al., 2008) and predictive maintenance (Mitchell et al., 2020). While
complementary, these solutions are not designed to measure high-resolution
depth images at extended distances, primarily due to the limited azimuth
resolution of radar platforms.

Depth Sensing using Cameras/LIDAR:

Cameras (Mejias et al., 2010), LIDARs (Ramasamy et al., 2016) and depth
imaging (Huang et al., 2017) are often used in diverse outdoor 3-D imaging
applications. Some depth camera systems (e.g. monocular depth estimation (Bhat
et al., 2020)) struggle at extended distances, some (e.g. stereo-vision
(Smolyanskiy et al., 2018)) require extended baselines for high accuracy,
while others (e.g. IR structured light (Sarbolandi et al., 2015)) function
poorly under ambient light. More broadly, systems struggle to measure depth at
a high resolution at long range, with about meter-scale accuracy at up to 80m
range in monocular depth estimation cases (Zhao et al., 2020) and only
operating up to around 20m in the case of depth cameras (StereoLabs, 2020).
Some LIDAR systems (Poulton et al., 2019) offer higher accuracy at extended
ranges, however face other significant limitations stemming from the power
consumption of the laser as well as robustness to dust, weather conditions and
coexistence with other LIDAR platforms (Bijelic et al., 2018; Kim et al.,
2015).

RF-Camera Fusion:

Camera and RF fusion has been proposed for automatic re-calibration (Xu et
al., 2012), industrial workplace (Savazzi et al., 2015), localization (Alahi
et al., 2015), person identification (Fang et al., 2020) and fall detection
(Kianoush et al., 2016). Radar-Camera fusion has also been studied for diverse
vehicular applications including attention selection to identify objects-of-
interest (Zhengping Ji and Prokhorov, 2008; Han et al., 2016; Chang et al.,
2020), tracking mobile objects (Long et al., 2018; Sengupta et al., 2019;
Zhang and Cao, 2019) better object perception and classification under poor
weather (Kato et al., 2002; John et al., 2019; Han et al., 2016), detecting
vehicles and guard rails (Steux et al., 2002; Alessandretti et al., 2007; Ji
and Prokhorov, 2008) and generating obstruction-resilient 2D images (Lekic and
Babic, 2019). Vision-based sensing has also been used for more effective
communication using mmWave (Haider et al., 2018; Nishio and Ashok, 2016).
Beyond radar and vision, prior work has used multi-modal fusion across a
variety of sensors for tracking human activity (Li et al., 2017), autonomous
driving (Cho et al., 2014) and beyond. We distinguish ourselves from this body
of work by focusing on combining mmWave radars and camera for high-resolution
depth imaging at long ranges, including under partial occlusions.

##  3\. mmWave Radar Primer

Radars, once only limited to military applications, are today used
ubiquitously in a variety of applications from airport security (Björklund et
al., 2012), automotive applications (Tokoro, 1996), human-computer interfaces
(Liu et al., 2020) and industrial automation (Mazgula et al., 2020). A key
factor which enabled this trend was the usage of mmWave frequencies which
allowed for compact antenna arrays and wide bandwidths, both of which are
crucial for radars’ target ranging and imaging capabilities. mmWave radars, as
the name suggests, use radio waves of millimeter scale wavelengths in either
60 GHz or 77-81 GHz by first actively illuminating an environment and then
processing the reflections from various objects in the environment. This is
noticeably different from modern image sensors which purely rely on passively
sensing rays which make their way to the sensor. The reflections from the
objects encode useful information such as objects’ range, azimuth, elevation
and velocity with respect to radar. The transmitted illumination and radar
hardware are the main factors which limit the radars’ ability to generate high
resolution 3D images of the scene.

Advantages of mmWave Radar:

Most commodity radars transmit a Frequency Modulated Continuous Wave (FMCW)
signal which is a waveform that continuously changes its frequency over time
to span a significant bandwidth $B$. A radar’s range resolution is
fundamentally limited by this effective bandwidth of the transmitted signal as
$\frac{c}{2B}$ ($c$ is speed of light). In the 77 GHz band, we have a
theoretical range resolution of 3.75 cm over tens of meters. In this regard,
radars are on par with time of flight LIDARs which report a similar range
accuracies. However, unlike LIDARs, radars work in all weather conditions
(rain, snow, fog) and extreme ambient lighting (sunlight) (Mullen and
Contarino, 2000).

Limitations of mmWave Radar:

However, radars unfortunately have worse azimuth and elevation resolutions
compared to both cameras and LIDARs. While range resolution is limited by the
bandwidth of the radar signal, angular resolutions are dictated by the number
of antenna elements that are packed on a radar. As the number of antenna
elements increases, so too does the resolution. The best state-of-the-art
commercial mmWave radar available (Texas-Instruments, 2021) with as many as
86x4 antenna elements has a 1.4∘x18∘ angular resolution. In contrast, state of
the art LIDARs today achieve 0.1∘x2∘, atleast 10x better angular resolution
than radars (Liu, 2008). With a poor angular resolution, 3D radar images look
very coarse and blobby in the angular domain. While more antenna elements can
be added, they come at significant increases in device cost and form-factor –
bridging the 10$\times$ gap is simply not an option with today’s state-of-the-
art hardware. We make the observation that even commodity cameras, because of
their dense focal planar array image sensors, are better than radars in terms
of angular resolution at about 0.02∘x0.02∘ (Napier et al., 2013). This
observation leads us to study combining the high angular resolution of camera
systems with the high depth resolution of mmWave radar – an approach we
describe in the next section.

##  4\. Metamoran’s Approach

Metamoran at a high level, takes as input camera and 77 GHz mmWave radar data
from a scene. We use these inputs to fuse and return a high-resolution depth
image for specific objects-of-interest at distances of several tens of meters
away. We specifically consider cars and persons – key to surveillance,
industrial and occupancy sensing applications. Our key contribution is a novel
radar processing algorithm that produces refined depth estimates for specific
objects-of-interest, based on priors obtained through image segmentation of
camera images. We choose a radar-based processing approach rather than an
exclusive deep-learning based approach on all underlying data (images + raw
I/Q), due to better explainability of the inferences. Besides, the resolution
obtained from our system in depth is close to the physical limits that can be
obtained owing to the bandwidth of the radar. Nevertheless, our solution
benefits heavily from state-of-the-art deep neural network based image
segmentation algorithms that operate on image data.

System Architecture and Outline:

Fig. 1 depicts the architecture of our system that we elaborate upon in the
following sections. First, we apply two state-of-the-art pre-processing steps
that operate on image data (Sec. 5): (1) image segmentation, i.e. identify the
spatial (x and y) bounds of objects-of-interest – cars, people and traffic
signs; (2) Monocular depth estimation to obtain an approximate estimate for
the shape of these objects, albeit prone to error at large distances. We then
design a novel specular beamforming algorithm in Sec. 6 that uses priors along
one dimension (x and y) from image segmentation and monocular depth estimation
which provide a coarse shape of the object of interest to then obtain a fine-
grained depth image. (3) Our final step (Sec. 7) is to build resilience to
occlusions and clutter into our system, to improve performance in a variety of
circumstances.

##  5\. Image Pre-Processing

Metamoran’s first step is to process camera image data to learn about the
approximate span in azimuth and elevation of objects-of-interest, as well as
an approximate silhouette or outline along the x-y plane, i.e. parallel to the
depth axis. We specifically consider three specific classes of objects-of-
interest that are ubiquitous in outdoor sensing – cars, pedestrians and
roadside infrastructure (traffic signs). As mentioned in Sec. 3, we exploit
the high angular resolution of camera systems that are at about 0.02∘x0.02∘
(Napier et al., 2013) – orders-of-magnitude better than mmWave radar systems.
Metamoran’s vision pre-processing steps below are therefore crucial in
providing prior information on the shape and location of objects-of-interest
along the x-y plane so that mmWave data can be used to focus on these objects
and improve resolution along the z-axis.

Figure 2. Image Segmentation: Metamoran uses image segmentation to identify
the spatial bounds along the x-y axes of objects-of-interest – cars,
pedestrians, traffic signs – with semantic labels assigned.

###  5.1. Image Segmentation

To find the spatial bounds (along x-y) of objects of interest, we perform
state-of-the-art image segmentation which labels objects by their type and
creates masks that capture the outline of these objects (see Fig. 2 for an
example).

We perform image segmentation using Detectron2 (Wu et al., 2019) trained with
KITTI dataset. This model has been previously trained on several objects
including cars, pedestrians and traffic signs in outdoor environments. We use
these types of objects as our primary test subjects without additional model
tuning. This image segmentation combines the best of both worlds from semantic
segmentation and instance segmentation, by providing a segmentation mask
(outline), a semantic label for the mask and instance ID for each detected
object as shown in Fig. 2. The segmentation mask directly provides the spatial
bounds and precise shape of the object along the x-y plane and is fed as a
prior for mmWave specular beamforming in Sec. 6 below.

Figure 3. Metamoran vs. Radar Beamforming and Monocular Estimation: A
qualitative comparison of the depth images shows standard radar beamforming to
be very coarse in azimuth resolution, monocular to have significant absolute
depth offsets but great azimuth diversity, and Metamoran which leverages rich
shape information from image pre-processing to generate an accurate, dense
depth image.

###  5.2. Monocular Depth Estimation

As a second step, we perform state-of-the-art monocular depth estimation
specifically on objects-of-interest filtered through image segmentation above.
We use this scheme both as a baseline for comparison and to provide a coarse
range of depths (depth profile) that the object spans. We use AdaBins (Bhat et
al., 2020) for monocular depth estimation of the objects-of-interest as
detected by the image segmentation step. We note that state-of-the-art
monocular depth estimation is poor in terms of accuracy and resolution at
extended distances, with errors of about 19.5 meters for objects that are 60
meters away (see Fig. 12). Nevertheless, we see that monocular depth
estimation provides useful prior information on the approximate range of
depths that the object spans and combined with image segmentation provides a
rough 3-D shape (outline) of the object that serve as inputs for our mmWave
specular beamforming algorithm in Sec. 6 below.

##  6\. mmWave Specular Beamforming

Metamoran’s specular beamforming algorithm processes the complex I/Q samples
received from the mmWave radar platform, coupled with the shape outlines of
objects-of-interest in the scene, obtained from the image pre-processing steps
in Sec. 5 above. In traditional mmWave beamforming (Steinhauer et al., 2008),
received I/Q samples are effectively projected along all spatial angles
(azimuth and elevation) to obtain the signal time-of-arrival between the
object to the radar. This quantity, when multiplied by the speed of light,
obtains the depth of the object. Unfortunately, this approach relies on the
azimuth resolution of the radar, which is fundamentally limited by the number
of antennas on the radar itself – at best $1.4^{\circ}$ in state-of-the-art
radar systems. The end result is a coarse radar image.

###  6.1. Depth Super-Resolution

Metamoran’s key technical contribution is a novel specular beamforming
solution, a super-resolution algorithm that overcomes the poor azimuth
resolution of mmWave radars by using priors from the image pre-processing
steps in Sec. 5. At a high level, Metamoran attempts to build a mmWave
wireless signal called the object template that captures the influence of an
object of a particular shape (as determined by camera pre-processing) on
mmWave radar receptions. Further, Metamoran also knows the precise azimuth and
elevation angle that this object template appears at, owing to the high
angular resolution of camera systems. Metamoran then identifies the best-
possible depth one could apply to this object template to best fit the
observed radar signals. The end result is a finer resolution depth image of
the object-of-interest as shown in Fig. 3(b).

Detailed Algorithm:

Mathematically, Metamoran’s algorithm extracts the approximate shape contour
inferred from image pre-processing, coupled with a mmWave ray-tracing model to
estimate the expected I/Q samples of reflections from such an object – i.e.
the object template. Essentially, the object-template is obtained by modeling
each point on the surface of the shape of the object $S(x,y,z)$ as a point
reflector shifted to some depth value $d$ that results in an overall distance
of $d$ relative to the radar. In its simplest form, one can then obtain this
point’s contribution to the received signal as at each wavelength $\lambda$ as
(Tse and Viswanath, 2005):

$h_{template}(d)=\frac{1}{d}e^{-j4\pi d/\lambda}$

Where the $4\pi$ rather than the traditional $2\pi$ stems from the fact that
radar signals are reflected or scattered back round-trip. We can then denote
$h_{template}(d)$ as the total channel experienced across the entire bandwidth
over all the points in the template. Metamoran then applies a matched-filter
to obtain $P(d)$ – the correlation of the object template at each possible
depth $d$ relative to the radar by processing the received signals across
frequencies. Mathematically, if $h$ is the received channel, we have:

$P(d)=h_{template}^{*}(d)h$

We then report the depth estimate of this object as the value of $d$ that
corresponds to the maximum of $P(d)$, i.e.

$d^{*}=\arg\max_{d}P(d)$

Algorithm 1 provides a more elaborate description of the steps of Metamoran
for FMCW mmWave radar signals.

Metamoran’s design of object templates overcomes the azimuth and elevation
resolution limits of mmWave radar. To see why, note that one could intuitively
view our design of templates as effectively performing a form of sparse
recovery – i.e., Metamoran assumes that objects of a particular shape are
unique at a certain range of azimuth and elevation in the radar reception.
This sparsity assumption is key to Metamoran’s super-resolution properties.

Input : Image Segmentation Object Mask, $P$

Monocular Depth Estimation, $M$

Raw I/Q Radar capture, $h$

1

$S$ = $M\cdot P$ // Approximate 3D shape of object

2 $C(x,z)$ = GetShapeContour$(S(x,y,z))$

3 for  _depth $d$_ do

4 $h_{template}(d)=$ShiftByDepth$(C(x,z),d)$

$P(d)=h_{template}^{*}(d)h$ // Matched Filtering

5

$d^{*}=\underset{d}{\operatorname{argmax}}P(d)$ // Depth Estimate

/* Choose local peaks near $d^{*}$ to generate Metamoran’s sparse point cloud
*/

6 $MM_{sparse}$ = GenerateSparseImage$(d^{*},P(d))$

/* Nullify large absolute errors from monocular estimation */

7 $C$ = ShiftToDepth($C$, $d^{*}$)

/* Reject outliers which occur along the edges of the image */

8 $C^{*}$ = RejectOutliers($C$)

$MM_{dense}$ = Fuse($MM_{sparse},C^{*})$ Output : $MM_{dense}(x,z)$ // Dense
Depth Image

9

10

Algorithm 1 Specular Beamforming Algorithm

###  6.2. Intra-Object Depth Profiling

We note our current description of Metamoran’s algorithm provides only one
depth value per object template, i.e. one depth per object. In practice, we
deal with extended objects and we would require multiple depth values across
the object. We could use local peaks from the specular beamforming output near
the peak depth value. But, the point cloud so obtained is very sparse and only
becomes sparser with increasing object distances. In an ideal world, we would
like an output similar to monocular depth estimation (see Fig. 4 for an
example). In monocular depth estimation, pixel color and other image features
are used to identify objects at various depth levels resulting in a dense
RGB-D image as shown in Fig. 4. Our key idea is to make use of the dense
monocular depth estimation in conjunction with the sparse point cloud from
specular beamforming described so far. However two problems persist in
realizing this fusion: (1) First, while monocular depth estimation may often
correctly return the relative depths between different parts of a large object
such as a car, it often makes large errors in absolute depths, particularly
for objects at extended distances (Saxena et al., 2005; Reza et al., 2018).
(2) Second, monocular depth estimation often struggles with objects that do
not have significant variation in color with respect to the background or
sharp edges that intuitively simplifies depth estimation (Saxena et al., 2005;
Reza et al., 2018). The rest of this section describes how we address both
these challenges to fuse Metamoran’s depth images with off the shelf monocular
depth estimates (see Fig. 4) that offer superior accuracy to monocular depth
estimation.

Figure 4. Monocular depth estimation gives a dense RGB-D depth image which is
promising for fusing with sparse Metamoran’s specular beamforming point
clouds. Figure 5. Metamoran vs. Clutter: Metamoran can help identify objects-
of-interest despite environmental clutter. (a) shows our scene, a narrow
parking lot bound by buildings with a lot of cars, as well as our target, a
car that is 50m away. (b) shows the raw radar beamforming of the area, with
very prominent out-of-span peaks from nearby cars and buildings. (c) shows the
slice of the radar beamforming bound by azimuth span determined from image
segmentation of the image. (d) shows the same azimuthal slice with side lobes
of out-of-span reflectors removed, with only one peak remaining that
corresponds to the reflected power profile of a car.

Correcting Absolute Errors:

To address the first challenge, we can simply shift the monocular depth
estimates for any given object-of-interest so that they line up with the
sparse point cloud obtained from Metamoran’s specular beamforming algorithm.
This ensures that absolute errors for any given object-of-interest are
minimized. A key point to note is that for large objects (e.g. a car), there
may be some ambiguity on which exact point on the monocular depth estimate
should be shifted to line up with Metamoran’s estimate. To remedy this, we
correlate the object template used in Sec. 6.1 from image segmentation with
the image that resulted from monocular depth estimation. Recall that this very
object template was used to estimate the object’s depth in Metamoran’s super-
resolution algorithm. The correlation process therefore allows us to identify
the pixel on the image that best corresponds with the depth estimates from
Metamoran’s super-resolution algorithm.

Correcting Relative Errors:

After aligning the monocular depth estimates with the sparse point cloud from
Metamoran’s beamforming, a naive way to fuse this would be consider all points
from both modalities. But, as seen in Figure. 3(b), edges of monocular
estimates tend to deviate a lot from the primary contour outline of the
object. If fused as is, one would experience errors expected from monocular
depth estimation. It’s therefore important to select points from the aligned
monocular depth estimates that only lie along the primary contour outline and
reject outliers. We note that the number of points detected per azimuth bin in
monocular estimates fall off sharply at the edges where our outliers of
interest lie. By using a simple threshold based outlier detection, we identify
points which actually lie along the primary contour. Upon fusing selected
monocular depth estimate points and sparse point cloud from Sec. 6.1, we
obtain a depth image that outperforms different algorithms using either of the
two modalities in terms of depth and azimuth resolution and depth accuracy.

Figure 6. Metamoran vs. Partial Occlusions: Metamoran can help identify
objects-of-interest despite partial occlusions. (a) shows an image of our
scene, a person behind a cart, located approximately 45m away. (b) shows
Metamoran’s capture of the person and the occluding (left) half of the cart.
Since image segmentation detected both an unlabelled object and a partially
covered person, Metamoran takes the farther reflector as the target.

##  7\. Ensuring System Resilience

The effective imaging of a reflector relies first on effective detection of
the desired object. Improving the ability of a mmWave radar to detect and find
the depth of a given reflector in cluttered conditions thus becomes a critical
enabling piece. This falls into three broad categories: reducing false
positive rate from spurious peaks and unwanted reflectors, increasing the
ability of our system to detect weak reflectors, and providing resilience to
occlusions. We discuss how the introduction of a camera allows Metamoran to
improve in all of these categories when compared to radar alone.

###  7.1. Reducing Clutter

To improve the robustness of Metamoran’s algorithm, we present a key
optimization that was pivotal in identifying the true depth of objects-of-
interest. In particular, our focus is in cluttered environments where
reflections from a large number of objects impede identifying the depth of the
true object. At first blush, one might assume that even with a large number of
objects in the environment, the number of objects at the desired azimuth angle
– as specified by image segmentation, would be relatively few. Further, given
that the object is in direct line-of-sight of the camera, it can also be
expected to correspond to the first peak observed along this 3-D angle.

However, we observe in practice that peaks from extremely strong reflectors
leak significantly in azimuth as well, often into our desired angle. This is
due to the poor angular resolution of the radar. This is a problem due to two
factors: (1) these leaks can appear as a false peak closer to our detector,
corrupting a first peak approach, and (2) these strong reflectors are often
three orders of magnitude larger than our desired reflector, and thus have
leaks that can dwarf our targets-of-interest. One must therefore perform a
declutter phase prior to applying Metamoran’s specular beamforming algorithm
that discounts and eliminates spurious results at depths that correspond to
these spurious peaks. Doing so would prevent Metamoran’s algorithm from being
misled by such peaks. Fig. 5 provides a qualitative comparison of the impact
of Metamoran’s algorithm in decluttering the radar image and identifying the
true peak. The plots (b)-(c) is this figure represent $P(d,\theta)$, which we
call radar profiles, that represent the power of signals received at different
depths $d$ and azimuth values $\theta$, measured through the standard
Bartlett-based radar beamforming algorithm (Prabhakara et al., 2020). Our
objective is to remove unwanted clutter in these profiles to focus on the
object’s of interest by masking out unwanted regions. This allows us then
apply Metamoran’s mmWave super-resolution algorithm from Sec. 6 by ignoring
unwanted clutter.

Specifically, in Metamoran we look for peaks in the regions of our radar
profile that fall outside of the azimuth span of our target, as expected from
image segmentation. For each peak, we generate an object template that is the
scale and position of that peak – including its side lobes – and subtract it
from our profile. We iterate many times until the magnitude of the peaks in
the area outside of our focus are comparable to the expected magnitude of the
target reflector. This is analogous to successive interference cancellation in
wireless communications (Patel and Holtzman, 1994), or the CLEAN algorithm in
radio-astronomy (Clark, 1980), with the distinction that we only remove peaks
outside of our desired sensing azimuthal span. What this process accomplishes
is the removal of side lobes from very large peaks in our azimuth of interest
– which is critical for the performance of our system.

###  7.2. Addressing Weak Reflections

In this section, we explore ways to amplify extremely weak reflections from
objects-of-interest, either due to their material properties, poor orientation
or extended range from the radar. Indeed, the precise level to which radar
reflections weaken depends on a combination of all of these properties and we
evaluate this further for a diverse set of objects in Sec. 9. While radar
typically uses Doppler to detect weak reflectors that are mobile, in varied
applications (surveillance, mapping, security, etc.) it is important to detect
objects that are not moving as well (e.g. a parked car or road sign). While
doppler can of course still be a practical solution for detecting relatively
few moving objects, we instead focus on what can be done to improve a single
capture.

We note that while background subtraction is a naive solution to this problem,
because of the the many orders of magnitude larger a noise reflector might be
than our given target, even slight positional or power fluctuations between
captures can leave very large peaks that make our target difficult to find.
Further, background subtraction only addresses this problem for moving
objects, not stationary objects that might also be dangerous.

Our approach instead relies on the fact that – because of image segmentation –
we are certain that the object we are looking for exists in a given azimuth
span, and we also know its object type (e.g. car or person). As a result, we
can determine a received-signal-strength upper bound based on the object type
and each distance. Thus, in-span reflectors that are significantly higher than
expected (and their side lobes) can also be removed as clutter as described in
7.1 and target peaks can be detected.

###  7.3. Impact of Partial Occlusions

Metamoran is also designed to be robust to – and even account for – partial
occlusions such as fog or physical obstructions. In the case of physical
obstructions, such as the cart in front of a person pictured in Fig. 6, image
segmentation will generate a mask for both the obstruction and the target. For
a known obstruction type, the obstruction can be detected as a target object
and then removed as clutter, using techniques explained in 7.1 and 7.2. In the
case of an unknown obstruction, we instead look for two peaks in our azimuth
span and take the farther one as our target.

While in some instances of partial obstructions, image segmentation can be
fairly robust, it could fail in other instances. However, mmWave radars are
known to be fairly resilient to partial occlusions (Guan et al., 2020) – and
we evaluate instances where Metamoran can leverage radar peaks to actively
improve segmentation in Sec. 10.2. Our discussion in Sec. 11 also captures
failure modes of this approach, especially for severe occlusions (e.g. heavy
fog).

Figure 7. Metamoran’s Sensing Platform: Metamoranis implemented using a FLIR
Blackfly S 24.5MP color camera and a TI MMWCAS-RF-EVM mmWave radar.
Evaluation: Metamoran was evaluated in outdoor spaces like roads and parking
lots with rich multipath from buildings, fences, lamp posts, other cars.

##  8\. Implementation and Evaluation

System Hardware:

Metamoran is implemented using a FLIR Blackfly S 24.5MP color camera and a TI
MMWCAS-RF-EVM RADAR (see Fig. 7). We operate the radar at 77-81 GHz with a
theoretical range resolution of 3.75-17.8 cm, depending on max range. The
radar also has 86 virtual antennas spaced out along the azimuth axis which
provides a theoretical azimuth resolution of $1.4^{\circ}$. As explained in
Sec. 3, this is at least an order of magnitude worse than cameras and lidars.
Unlike fusion approaches which rely on processed point clouds (Nobis et al.,
2019), this radar supports logging raw complex I/Q samples which is critical
for our processing.

Figure 8. Range Attenuation: Reflectivity of an object in line-of-sight
conditions after out-of-span SIC has been applied.

Testbed and Data Collection:

We test this system in a variety of 200 outdoor scenes such as parking lots
and roads at distances ranging from 1 m to 60 m from objects-of-interest.
These environments have rich multipath arising due to buildings, street lamps,
fences, out-of-interest parked cars and pedestrians. Fig. 7 shows two
candidate locations in the area surrounding a university campus in a major
U.S. city.

Ground Truth:

We collect ground truth data using a Velodyne Puck LIDAR (VLP-16), which
generates 3D point clouds, with fine azimuth and elevation resolutions and 3
cm ranging error. While this lidar is rated for up to 100 m, in practice, on a
sunny day, we found the Puck collected data with sufficient point cloud
density only until about 30 m. Therefore, for ranges beyond 30 m, we surveyed
a point closer to the object-of-interest and placed the lidar at that point.

Baselines:

We compare Metamoran with two baselines that use the same hardware platforms:
(1) Naive fusion of Camera and Radar:  We use image pre-processing to obtain
the azimuth spanned by object-of-interest. We perform standard radar
beamforming for FMCW radar, and bound the output to the azimuth span and then
pick the strongest reflector as the target. (2) Monocular Depth Estimation:
We use state-of-the-art monocular depth estimation algorithm (Bhat et al.,
2020) trained to report depth values up to 80 m.

Objects-of-interest Selection:

We select a car, a person, and a stop sign for use as our targets, because
these are useful for a variety of applications, including smart city and
surveillance. Further, these provide a variety of reflectors in size, shape,
and reflectivity to evaluate our system. We note that while it is necessary to
sense people and cars while they are moving, they are also important to sense
when they are stationary – in the case of a delivery truck, an uber, or a
child at a bus stop, for example. Indeed, static objects are much more
challenging versus moving objects to detect in radar processing because
Doppler-based filtering or background subtraction cannot be used to remove
clutter. We therefore focus our evaluation on imaging static objects.

Calibration:

We note that Metamoran requires both internal calibration of the components as
well as external calibration between the camera and the radar. Internally, our
mmWave radar is calibrated using a corner reflector placed at 5m, as described
in the TI’s mmWave Studio Cascade User Guide (Incorporated, 2018). The camera
intrinsics are measured by taking many photos of a checkerboard to remove
fisheye distortion (using Matlab’s Computer Vision Toolbox (The MathWorks,
[n.d.])) and for image segmentation and monocular depth estimation.

Externally, Metamoran requires a consistent understanding of object shapes
between the mmWave platform system and the camera system. While both of these
are co-located in Metamoran, they are at a small relative distance of 15 cm,
which could lead to inconsistencies in the images produced by the two
modalities. Metamoran accounts for this using a joint calibration of the
mmWave radar and camera using a feature-rich metallic surface that is viewed
from both the camera and radar platform to capture a Euclidean transform
between their frames of reference. The object is chosen to be feature-rich for
both platforms, with stark differences in both color and the presence/absence
of strong mmWave reflectors (metallic structures). We note that the transform
obtained from calibration is applied, prior to fusing measurements from either
platform to ensure consistency.

Figure 9. Orientation: The magnitude of reflected signal varies with the
orientation of our planar targets (sign and car), with peaks at the highest
effective area

##  9\. Microbenchmarks

Figure 10. Across all algorithms, we see car with the lowest depth error,
followed by person, followed by sign. This correlates with each object’s
reflectivity.

Figure 11. Across all algorithms, we see degraded performance in PLOS compared
to LOS, particularly in our naive fusion baseline.

Figure 12. Across all algorithms, we see median depth error rise with
increased range, with Metamoran showing better accuracy.

Figure 13. CDF of absolute error shows Metamoran is superior to our two
baselines in median accuracy.

###  9.1. Comparing Object Reflectivity

Method:

To empirically determine expected power thresholds for detecting target
objects in an occluded object, we measure the peak value from radar
beamforming for our three target reflectors: car, person, and a road sign,
across different distances in 81 line of sight settings.

Results:

Our results for this are shown in Fig. 8. We observe that power falls off
significantly with distance. From about 10 m to 50 m, the reflections
attenuate: 16.7$\times$ for a car, 63$\times$ for a person, and 4.4$\times$
for a sign. We note that the sign is a significantly weaker reflector than a
person despite being a .762m $\times$ .762m metal sheet outfitted with optical
retro-reflectors: past work indicates that this may be due to the majority of
incident signal being reflected specularly off planes and thus not received by
our radar (Bansal et al., 2020).

###  9.2. Impact of Object Orientation

Method:

To evaluate the impact of orientation on the reflectivity of our more planar
reflectors, we collected data across 7 angles of the front of a stop sign and
8 angles of a car. This data was measured at a fixed 4m away from the object.

Results:

The peak values from radar beamforming at different orientation are shown in
Fig. 9. We find that the peaks correspond, as expected, with the largest
effective area: the face of the stop sign, and the side of the car. We find
the stop sign peak reflectivity degrades 1.68$\times$ at poor orientation, and
the car can degrade 21$\times$ depending on orientation.

##  10\. Results

###  10.1. Depth Resolution

Figure 14. Imaging Errors increase with decreasing object reflectivity across
algorithms.

Figure 15. Imaging Errors are degraded in partial line of sight scenarios
across all algorithms.

Figure 16. Imaging Errors vs. increasing range.

Figure 17. This CDF shows that Metamoran significantly outperforms the
baselines. The tail in the case of Metamoran is much smaller than that for
baselines.

Figure 18. This shows median errors for Metamoran depth estimation and imaging
performance up to 90m.

Figure 19. Similarly colored boxes contain similar objects across segmentation
and radar. While cars in the red boxes are missed by camera, radar still
detects them.

Method:

For our range results, we collected 146 data samples in varying lighting
conditions at 2 obstacle-rich sites. We collected both line-of-sight (LOS)
captures of targets as well as captures of partial line-of-sight (PLOS)
occluded by carts, fog, and other environmental objects. Targets were
positioned from 3 m to 58 m.

Data was collected in 2 range/resolution buckets: 4.2cm at 0-20m, 11.6cm at
20-60m. The primary bottleneck of range resolution for this system is the
TDA2SX SoC capture card that is on the MMWCAS board – it can handle at most a
data width of 4096, corresponding to 512 complex samples per receiver. This
may be improved with hardware research and advancements, but improvements in
that domain are complementary to our approach.

Depth error is measured from one point in each of these approaches (Peak value
obtained with naive fusion of radar beamforming and camera, Metamoran estimate
and, most repeated value over an object mask for monocular depth estimation)
to the depth span provided by the LIDAR.

We compare median error in depth across objects-of-interest for Metamoran and
the two baseline systems: naive fusion and monocular depth estimation. We
include error bars corresponding to +/- the standard deviation of our
collected data. We note that we present median over mean due to the long tail
often found in RF localization and sensing that affects both Metamoran and the
baseline: slight variances in noise and power can result in disproportionately
large errors if the second-largest peak overtakes the first. For systems with
a low median error, this effect can be ameliorated by taking multiple
snapshots and removing outliers.

We represent three sets of results: (1) three different reflector objects; (2)
Partial occlusions including fog and other objects preventing a complete
direct view of the object; (3) three different range buckets. Across all
experiments, we find that Metamoran significantly outperforms the baselines.
We elaborate the performance across each axis below.

Object Results:

Fig. 12 shows the median error in depth across objects-of-interest for
Metamoran and the two baseline systems. We see lowest error for the car across
the board due to a combination of factors: the car is our strongest reflector
and also offers multiple points on its surface to reflect radar signals due to
its size (4.66m x 1.795m). We see performance further degrade with the
progressively weak reflectors as measured in Sec. 9.1: person is the next most
accurate, followed by the sign.

Occlusion Results:

Fig. 12 shows the median error in depth in line-of-sight (LOS) and partial-
line-of-sight (PLOS) for Metamoran and the two baseline systems. We see a
particularly significant degradation in our naive fusion baseline for PLOS,
which frequently takes the occluding object as the strongest reflector, unlike
Metamoran, which can detect and account for occlusions using image
segmentation.

Range Results:

Fig. 12 shows the median error in depth across range for Metamoran and the
baselines. As expected, accuracy across all approaches, objects, and occlusion
settings deteriorates with range due to weaker received signals.

CDF Results:

Fig. 13 shows CDF of the median error in depth for Metamoran and the
baselines. Metamoran has a median error of 0.28m across all collected data,
compared to 6.5m for monocular depth estimation and 3.75m for naive radar and
camera fusion. These correspond to mean values of 1.42m, 8.48m, and 7.89m
respectively due to long tail effects.

###  10.2. Depth Imaging

Method:

To compute high resolution depth images, we implement the method in Sec. 6.2.
In contrast to Sec. 10.1 which only computed depth errors, here we want to
characterize system performance for a point cloud obtained from the baselines
monocular depth estimation and naive fusion of camera and radar, and our
system against lidar point clouds. Data collection is as similar to that
explained in Sec. 10.1.

To compare two point clouds $A$ and $B$, we use a modified version of
Hausdorff distance (pdal.io, [n.d.]) as follows:

$\operatorname{min}\Bigg{\\{}\underset{a\in
A}{\operatorname{median}}\big{\\{}\underset{b\in
B}{\operatorname{min}}\\{d(a,b)\\}\big{\\}},\underset{b\in
B}{\operatorname{median}}\big{\\{}\underset{a\in
A}{\operatorname{min}}\\{d(b,a)\\}\big{\\}}\Bigg{\\}}$

where $d(a,b)$ is the distance between points $a$ and $b$. Hausdorff distance
is popularly used in obtaining similarity scores between point clouds.
Intuitively, this metric measures the median distance between any two points
in the point cloud. The lower the distance, the more similar the point clouds
are. We report this distance as imaging error in meters.

Results:

Trends in imaging results largely follow those in depth imaging, as problems
with detection propagate through the system. We note that shape error is
larger than the depth error across the board due to additional pairwise
distances being calculated. Figure 16 shows the imaging errors against
different object types for the 3 different algorithms, Figure 16 shows the
median error in imaging in line-of-sight and partial-line-of-sight for
Metamoran and the two baseline systems, and Figure 16 shows the median error
in depth across range for Metamoran and the two baseline systems. Metamoran
outperforms both baselines across all categories. We note that in these
baselines, monocular depth estimation outperforms naive fusion unlike in 10.1.
This is because Monocular depth estimation benefits from our metric due to its
large azimuth span of many points that are thus more likely to be close to a
point in the LIDAR baseline, versus the fewer, and clustered profiles given by
naive fusion.

Fig. 19 shows CDF of the median error in depth for Metamoran and the two
baseline systems. Metamoran has a median error of 0.8m across all collected
data, compared to 3.4m for monocular depth estimation and 5.04m for naive
radar and camera fusion. These correspond to mean values of 1.82m, 6.59m, and
8.27m respectively due to long tail effects.

Improving segmentation in PLOS:

A point to note that improves our accuracy in partial line-of-sight in Fig. 16
is the ability to detect objects that image segmentation misses or offers low
confidence on due to occlusions due to obstructions. Fig. 19 shows one
representative example of this effect for a partial line-of-sight image where
an object that was occluded and low-confidence in the camera image was clearly
detected based on radar processing.

###  10.3. Range Extension

Method:

In addition to the data collected for Sec.10.1, we further collect 17 scenes
at 2 sites for a large reflector (car) with an additional resolution/range
bucket: 17.8 cm at 60-90m. At these extended ranges, car depth is no longer
measurable with our baselines, and the sign and person are no longer
detectable even with the assistance of Metamoran. We do not collect distances
above 90m: since we already observed at 90m that the entire car appears as a
single pixel on our radar, distances above this become unreliable.

Results:

We show the results for depth resolution and imaging of Metamoran compared to
the lidar ground truth in Fig. 19. We see slight degradation with the
increased distance, although it is minimal. We note that the performance
degradation in practice is that the reflector is detected less often,
particularly in the presence of clutter. At 90m, our 1.4∘ of azimuth
resolution is spaced at 2.2m, and imaging relies very heavily on the
successful reception of single pixels.

##  11\. Limitations

An important limitation of our system is that its reliance on a camera makes
vulnerable to excessive darkness and fully occlusive environmental conditions
(e.g. very thick fog). Fig. 20 shows one such instance where our system mis-
identifies an object (a person) due to heavy fog. We note, in these
circumstances, the mmWave RADAR continues to operate and can continue to
provide range information for objects in the environment, albeit with
attenuated range and with poor angular resolution. For instance, despite the
object type in Fig. 20 being labeled incorrectly, the depth value reported
from mmWave radar is approximately correct.

Further improvements to calibration could further refine our system and
improve results – in particular, an ideal calibration device would be only a
pixel large on our camera and also a very strong reflector in mmWave. In
practice, this balance is difficult to strike, and we leave further
experimentation of calibration materials to future work.

Figure 20. Limitations of Metamoran: Metamoran can struggle when vision
algorithms fail significantly such as complete occlusions (e.g. fog), such as
above.

##  12\. Conclusion

This paper develops Metamoran, a hybrid mmWave and camera based system that
achieves high-resolution depth images for objects at extended distances.
Metamoran’s secret sauce is a novel specular radar processing system that
identifies the spatial bounds in azimuth and elevation of objects-of-interest
using image segmentation on camera data to improve radar processing along the
depth dimension. The resulting system is evaluated on real-world data sets
that will be made openly available to obtain depth images of objects-of-
interest including pedestrians and cars at distances of up to 60 m. We believe
there is rich scope for future work in extending fused mmWave and camera-based
depth imaging to broader classes of objects and ensuring resilience to severe
occlusions.

## References

  * (1)
  * Alahi et al. (2015) Alexandre Alahi, Albert Haque, and Li Fei-Fei. 2015.  RGB-W: When vision meets wireless. In _Proceedings of the IEEE International Conference on Computer Vision_. 3289–3297.
  * Alessandretti et al. (2007) Giancarlo Alessandretti, Alberto Broggi, and Pietro Cerri. 2007\.  Vehicle and guard rail detection using radar and vision data fusion.  _IEEE transactions on intelligent transportation systems_ 8, 1 (2007), 95–105.
  * Bansal et al. (2020) Kshitiz Bansal, Keshav Rungta, Siyuan Zhu, and Dinesh Bharadia. 2020.  Pointillism: Accurate 3D Bounding Box Estimation with Multi-Radars. In _Proceedings of the 18th Conference on Embedded Networked Sensor Systems_ (Virtual Event, Japan) _(SenSys ’20)_. Association for Computing Machinery, New York, NY, USA, 340–353.  https://doi.org/10.1145/3384419.3430783
  * Bhat et al. (2020) Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. 2020\.  AdaBins: Depth Estimation using Adaptive Bins.  _arXiv preprint arXiv:2011.14141_ (2020).
  * Bijelic et al. (2018) Mario Bijelic, Tobias Gruber, and Werner Ritter. 2018\.  A benchmark for lidar sensors in fog: Is detection breaking down?. In _2018 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 760–767.
  * Björklund et al. (2012) Svante Björklund, Tommy Johansson, and Henrik Petersson. 2012\.  Evaluation of a micro-Doppler classification method on mm-wave data. In _2012 IEEE Radar Conference_. IEEE, 0934–0939.
  * Chang et al. (2020) Shuo Chang, Yifan Zhang, Fan Zhang, Xiaotong Zhao, Sai Huang, Zhiyong Feng, and Zhiqing Wei. 2020.  Spatial Attention fusion for obstacle detection using mmwave radar and vision sensor.  _Sensors_ 20, 4 (2020), 956.
  * Cho et al. (2014) H. Cho, Y. Seo, B. V. K. V. Kumar, and R. R. Rajkumar. 2014.  A multi-sensor fusion system for moving object detection and tracking in urban driving environments. In _2014 IEEE International Conference on Robotics and Automation (ICRA)_. 1836–1843.  https://doi.org/10.1109/ICRA.2014.6907100
  * Clark (1980) BG Clark. 1980.  An efficient implementation of the algorithm’CLEAN’.  _Astronomy and Astrophysics_ 89 (1980), 377.
  * Dandu et al. (2021) Krishnanshu Dandu, Sreekiran Samala, Karan Bhatia, Meysam Moallem, Karthik Subburaj, Zeshan Ahmad, Daniel Breen, Sunhwan Jang, Tim Davis, Mayank Singh, et al. 2021\.  High-Performance and Small Form-Factor mm-Wave CMOS Radars for Automotive and Industrial Sensing in 76-to-81GHz and 57-to-64GHz Bands. In _2021 IEEE International Solid-State Circuits Conference (ISSCC)_ , Vol. 64. IEEE, 39–41.
  * Depatla et al. (2017) Saandeep Depatla, Chitra R Karanam, and Yasamin Mostofi. 2017\.  Robotic Through-Wall Imaging: Radio-Frequency Imaging Possibilities with Unmanned Vehicles.  _IEEE Antennas and Propagation Magazine_ 59, 5 (2017), 47–60.
  * Dhekne et al. (2018) Ashutosh Dhekne, Mahanth Gowda, Yixuan Zhao, Haitham Hassanieh, and Romit Roy Choudhury. 2018\.  LiquID: A Wireless Liquid IDentifier. In _Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services_ (Munich, Germany) _(MobiSys ’18)_. ACM, New York, NY, USA, 442–454.  https://doi.org/10.1145/3210240.3210345
  * Fang et al. (2020) Shiwei Fang, Tamzeed Islam, Sirajum Munir, and Shahriar Nirjon. 2020.  EyeFi: Fast Human Identification Through Vision and WiFi-based Trajectory Matching. In _2020 16th International Conference on Distributed Computing in Sensor Systems (DCOSS)_. IEEE, 59–68.
  * Guan et al. (2020) Junfeng Guan, Sohrab Madani, Suraj Jog, Saurabh Gupta, and Haitham Hassanieh. 2020.  Through Fog High-Resolution Imaging Using Millimeter Wave Radar. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 11464–11473.
  * Haider et al. (2018) Muhammad Kumail Haider, Yasaman Ghasempour, and Edward W. Knightly. 2018.  Search Light: Tracking Device Mobility Using Indoor Luminaries to Adapt 60 GHz Beams. In _Proceedings of the Eighteenth ACM International Symposium on Mobile Ad Hoc Networking and Computing_ (Los Angeles, CA, USA) _(Mobihoc ’18)_. Association for Computing Machinery, New York, NY, USA, 181–190.  https://doi.org/10.1145/3209582.3209601
  * Han et al. (2016) S. Han, X. Wang, L. Xu, H. Sun, and N. Zheng. 2016.  Frontal object perception for Intelligent Vehicles based on radar and camera fusion. In _2016 35th Chinese Control Conference (CCC)_. 4003–4008.  https://doi.org/10.1109/ChiCC.2016.7553978
  * Han et al. (2016) Siyang Han, Xiao Wang, Linhai Xu, Hongbin Sun, and Nanning Zheng. 2016.  Frontal object perception for Intelligent Vehicles based on radar and camera fusion. In _2016 35th Chinese Control Conference (CCC)_. IEEE, 4003–4008.
  * Huang et al. (2017) Albert S Huang, Abraham Bachrach, Peter Henry, Michael Krainin, Daniel Maturana, Dieter Fox, and Nicholas Roy. 2017.  Visual odometry and mapping for autonomous flight using an RGB-D camera.  In _Robotics Research_. Springer, 235–252.
  * Huang et al. (2014) Donny Huang, Rajalakshmi Nandakumar, and Shyamnath Gollakota. 2014\.  Feasibility and limits of wi-fi imaging.  _Proceedings of the ACM SenSys_ (2014), 266–279.  https://doi.org/10.1145/2668332.2668344
  * Incorporated (2018) Texas Instruments Incorporated. 2018\.  User’s Guide: mmWave Studio Cascade.
  * Ji and Prokhorov (2008) Zhengping Ji and Danil Prokhorov. 2008.  Radar-vision fusion for object classification. In _2008 11th International Conference on Information Fusion_. IEEE, 1–7.
  * John et al. (2019) Vijay John, MK Nithilan, Seiichi Mita, Hossein Tehrani, RS Sudheesh, and PP Lalu. 2019\.  So-net: Joint semantic segmentation and obstacle detection using deep fusion of monocular camera and radar. In _Pacific-Rim Symposium on Image and Video Technology_. Springer, 138–148.
  * Karanam and Mostofi (2017) Chitra R Karanam and Yasamin Mostofi. 2017.  3D through-wall imaging with unmanned aerial vehicles using WiFi. In _Proceedings of the 16th ACM/IEEE International Conference on Information Processing in Sensor Networks_. ACM, 131–142.
  * Kato et al. (2002) T. Kato, Y. Ninomiya, and I. Masaki. 2002.  An obstacle detection method by fusion of radar and motion stereo.  _IEEE Transactions on Intelligent Transportation Systems_ 3, 3 (2002), 182–188.  https://doi.org/10.1109/TITS.2002.802932
  * Kianoush et al. (2016) Sanaz Kianoush, Stefano Savazzi, Federico Vicentini, Vittorio Rampa, and Matteo Giussani. 2016\.  Device-free RF human body fall detection and localization in industrial workplaces.  _IEEE Internet of Things Journal_ 4, 2 (2016), 351–362.
  * Kim et al. (2015) Gunzung Kim, Jeongsook Eom, Seonghyeon Park, and Yongwan Park. 2015.  Occurrence and characteristics of mutual interference between LIDAR scanners. In _Photon Counting Applications 2015_ , Vol. 9504. International Society for Optics and Photonics, 95040K.
  * Kotaru et al. (2015) Manikanta Kotaru, Kiran Joshi, Dinesh Bharadia, and Sachin Katti. 2015.  SpotFi: Decimeter Level Localization Using WiFi.  _SIGCOMM Comput. Commun. Rev._ 45, 4 (Aug. 2015), 269–282.  https://doi.org/10.1145/2829988.2787487
  * Lekic and Babic (2019) Vladimir Lekic and Zdenka Babic. 2019.  Automotive radar and camera fusion using Generative Adversarial Networks.  _Computer Vision and Image Understanding_ 184 (2019), 1–8.
  * Li et al. (2017) H. Li, A. Shrestha, F. Fioranelli, J. Le Kernec, H. Heidari, M. Pepa, E. Cippitelli, E. Gambi, and S. Spinsante. 2017.  Multisensor data fusion for human activities classification and fall detection. In _2017 IEEE SENSORS_. 1–3.  https://doi.org/10.1109/ICSENS.2017.8234179
  * Liu et al. (2020) Haipeng Liu, Yuheng Wang, Anfu Zhou, Hanyue He, Wei Wang, Kunpeng Wang, Peilin Pan, Yixuan Lu, Liang Liu, and Huadong Ma. 2020\.  Real-time Arm Gesture Recognition in Smart Home Scenarios via Millimeter Wave Sensing.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 4, 4 (2020), 1–28.
  * Liu (2008) Xiaoye Liu. 2008\.  Airborne LiDAR for DEM generation: some critical issues.  _Progress in physical geography_ 32, 1 (2008), 31–49.
  * Liu et al. (2021) Ze Liu, Yingfeng Cai, Hai Wang, Long Chen, Hongbo Gao, Yunyi Jia, and Yicheng Li. 2021.  Robust Target Recognition and Tracking of Self-Driving Cars With Radar and Camera Information Fusion Under Severe Weather Conditions.  _IEEE Transactions on Intelligent Transportation Systems_ (2021).
  * Long et al. (2018) Ningbo Long, Kaiwei Wang, Ruiqi Cheng, Kailun Yang, and Jian Bai. 2018.  Fusion of millimeter wave radar and RGB-depth sensors for assisted navigation of the visually impaired. In _Millimetre Wave and Terahertz Sensors and Technology XI_ , Vol. 10800. International Society for Optics and Photonics, 1080006.
  * Mazgula et al. (2020) Jakub Mazgula, Jakub Sapis, Umair Sajid Hashmi, and Harish Viswanathan. 2020.  Ultra reliable low latency communications in mmWave for factory floor automation.  _Journal of the Indian Institute of Science_ 100, 2 (2020), 303–314.
  * Mejias et al. (2010) Luis Mejias, Scott McNamara, John Lai, and Jason Ford. 2010\.  Vision-based detection and tracking of aerial targets for UAV collision avoidance. In _Intelligent Robots and Systems (IROS), 2010 IEEE/RSJ International Conference on_. IEEE, 87–92.
  * Mitchell et al. (2020) Daniel Mitchell, Jamie Blanche, and David Flynn. 2020\.  An Evaluation of Millimeter-wave Radar Sensing for Civil Infrastructure. In _2020 11th IEEE Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON)_. IEEE, 0216–0222.
  * Mullen and Contarino (2000) Linda J Mullen and V Michael Contarino. 2000.  Hybrid lidar-radar: seeing through the scatter.  _IEEE Microwave magazine_ 1, 3 (2000), 42–48.
  * Napier et al. (2013) Ashley Napier, Peter Corke, and Paul Newman. 2013\.  Cross-calibration of push-broom 2d lidars and cameras in natural scenes. In _2013 IEEE International Conference on Robotics and Automation_. IEEE, 3679–3684.
  * Nguyen et al. (2017) Phuc Nguyen, Hoang Truong, Mahesh Ravindranathan, Anh Nguyen, Richard Han, and Tam Vu. 2017\.  Matthan: Drone Presence Detection by Identifying Physical Signatures in the Drone’s RF Communication. In _Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services_ (Niagara Falls, New York, USA) _(MobiSys ’17)_. ACM, New York, NY, USA, 211–224.  https://doi.org/10.1145/3081333.3081354
  * Nishio and Ashok (2016) Takayuki Nishio and Ashwin Ashok. 2016.  High-Speed Mobile Networking through Hybrid MmWave-Camera Communications. In _Proceedings of the 3rd Workshop on Visible Light Communication Systems_ (New York City, New York) _(VLCS ’16)_. Association for Computing Machinery, New York, NY, USA, 37–42.  https://doi.org/10.1145/2981548.2981552
  * Nobis et al. (2019) Felix Nobis, Maximilian Geisslinger, Markus Weber, Johannes Betz, and Markus Lienkamp. 2019\.  A deep learning-based radar and camera sensor fusion architecture for object detection. In _2019 Sensor Data Fusion: Trends, Solutions, Applications (SDF)_. IEEE, 1–7.
  * Olivier et al. (2016) A. Olivier, G. Bielsa, I. Tejado, M. Zorzi, J. Widmer, and P. Casari. 2016\.  Lightweight Indoor Localization for 60-GHz Millimeter Wave Systems. In _2016 13th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON)_. 1–9.  https://doi.org/10.1109/SAHCN.2016.7732999
  * Palffy et al. (2019) Andras Palffy, Julian FP Kooij, and Dariu M Gavrila. 2019\.  Occlusion aware sensor fusion for early crossing pedestrian detection. In _2019 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 1768–1774.
  * Patel and Holtzman (1994) P. Patel and J. Holtzman. 1994.  Analysis of a simple successive interference cancellation scheme in a DS/CDMA system.  _IEEE Journal on Selected Areas in Communications_ 12, 5 (1994), 796–807.  https://doi.org/10.1109/49.298053
  * pdal.io ([n.d.]) pdal.io. [n.d.].  Point Data Abstraction Library.  https://pdal.io/apps/hausdorff.html
  * Poulton et al. (2019) Christopher Vincent Poulton, Matthew J Byrd, Peter Russo, Erman Timurdogan, Murshed Khandaker, Diedrik Vermeulen, and Michael R Watts. 2019.  Long-range LiDAR and free-space data communication with high-performance optical phased arrays.  _IEEE Journal of Selected Topics in Quantum Electronics_ 25, 5 (2019), 1–8.
  * Prabhakara et al. (2020) Akarsh Prabhakara, Vaibhav Singh, Swarun Kumar, and Anthony Rowe. 2020.  Osprey: A mmWave approach to tire wear sensing. In _Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services_. 28–41.
  * Ramasamy et al. (2016) Subramanian Ramasamy, Roberto Sabatini, Alessandro Gardi, and Jing Liu. 2016.  LIDAR obstacle warning and avoidance system for unmanned aerial vehicle sense-and-avoid.  _Aerospace Science and Technology_ 55 (2016), 344–358.
  * Reza et al. (2018) M. A. Reza, J. Kosecka, and P. David. 2018\.  FarSight: Long-Range Depth Estimation from Outdoor Images. In _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. 4751–4757.  https://doi.org/10.1109/IROS.2018.8593971
  * Sarbolandi et al. (2015) Hamed Sarbolandi, Damien Lefloch, and Andreas Kolb. 2015\.  Kinect range sensing: Structured-light versus Time-of-Flight Kinect.  _Computer vision and image understanding_ 139 (2015), 1–20.
  * Savazzi et al. (2015) Stefano Savazzi, Vittorio Rampa, Federico Vicentini, and Matteo Giussani. 2015.  Device-free human sensing and localization in collaborative human–robot workspaces: A case study.  _IEEE Sensors Journal_ 16, 5 (2015), 1253–1264.
  * Saxena et al. (2005) Ashutosh Saxena, Sung H Chung, Andrew Y Ng, et al. 2005\.  Learning depth from single monocular images. In _NIPS_ , Vol. 18. 1–8.
  * Sengupta et al. (2019) Arindam Sengupta, Feng Jin, and Siyang Cao. 2019.  A DNN-LSTM based Target Tracking Approach using mmWave Radar and Camera Sensor Fusion. In _2019 IEEE National Aerospace and Electronics Conference (NAECON)_. IEEE, 688–693.
  * Smolyanskiy et al. (2018) Nikolai Smolyanskiy, Alexey Kamenev, and Stan Birchfield. 2018\.  On the importance of stereo for accurate depth estimation: An efficient semi-supervised deep neural network approach. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_. 1007–1015.
  * Stanko et al. (2008) S Stanko, D Notel, A Wahlen, J Huck, F Kloppel, R Sommer, M Hagelen, and H Essen. 2008.  Active and passive mm-wave imaging for concealed weapon detection and surveillance. In _2008 33rd International Conference on Infrared, Millimeter and Terahertz Waves_. IEEE, 1–2.
  * Steinhauer et al. (2008) Matthias Steinhauer, Hans-Oliver Ruoß, Hans Irion, and Wolfgang Menzel. 2008.  Millimeter-wave-radar sensor based on a transceiver array for automotive applications.  _IEEE transactions on microwave theory and techniques_ 56, 2 (2008), 261–269.
  * StereoLabs (2020) StereoLabs. 2020\.  https://www.stereolabs.com/zed-2/
  * Steux et al. (2002) Bruno Steux, Claude Laurgeau, Laurent Salesse, and Didier Wautier. 2002.  Fade: A vehicle detection and tracking system featuring monocular color vision and radar data fusion. In _Intelligent Vehicle Symposium, 2002. IEEE_ , Vol. 2. IEEE, 632–639.
  * Texas-Instruments (2021) Texas-Instruments. 2021\.  TI MMWCAS-RF-EVM.  https://www.ti.com/tool/MMWCAS-RF-EVM.
  * The MathWorks ([n.d.]) Inc. The MathWorks. [n.d.].  I.  https://www.mathworks.com/help/vision/ref/undistortfisheyeimage.html
  * Tokoro (1996) Setsuo Tokoro. 1996\.  Automotive application systems of a millimeter-wave radar. In _Proceedings of Conference on Intelligent Vehicles_. IEEE, 260–265.
  * Toriyama (1994) Akira Toriyama. 1994\.  Dragonball Z: The Zeta Sword.
  * Tse and Viswanath (2005) David Tse and Pramod Viswanath. 2005.  _Fundamentals of wireless communication_.  Cambridge university press.
  * Wang et al. (2017) Ju Wang, Jie Xiong, Xiaojiang Chen, Hongbo Jiang, Rajesh Krishna Balan, and Dingyi Fang. 2017\.  TagScan: Simultaneous Target Imaging and Material Identification with Commodity RFID Devices. In _Proceedings of the 23rd Annual International Conference on Mobile Computing and Networking_ (Snowbird, Utah, USA) _(MobiCom ’17)_. ACM, New York, NY, USA, 288–300.  https://doi.org/10.1145/3117811.3117830
  * Wei and Zhang (2015) Teng Wei and Xinyu Zhang. 2015.  MTrack: High-Precision Passive Tracking Using Millimeter Wave Radios. In _Proceedings of the 21st Annual International Conference on Mobile Computing and Networking_ (Paris, France) _(MobiCom ’15)_. Association for Computing Machinery, New York, NY, USA, 117–129.  https://doi.org/10.1145/2789168.2790113
  * Wenger (1998) J Wenger. 1998.  Automotive mm-wave radar: Status and trends in system design and technology.  (1998).
  * Wu (2016) Kaishun Wu. 2016\.  Wi-metal: Detecting metal by using wireless networks. In _Communications (ICC), 2016 IEEE International Conference on_. IEEE, 1–6.
  * Wu et al. (2019) Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. 2019.  Detectron2.  https://github.com/facebookresearch/detectron2.
  * Xie et al. (2015) Yaxiong Xie, Zhenjiang Li, and Mo Li. 2015.  Precise Power Delay Profiling with Commodity WiFi. In _Proceedings of the 21st Annual International Conference on Mobile Computing and Networking_ (Paris, France) _(MobiCom ’15)_. ACM, New York, NY, USA, 53–64.  https://doi.org/10.1145/2789168.2790124
  * Xu et al. (2012) Chenren Xu, Mingchen Gao, Bernhard Firner, Yanyong Zhang, Richard Howard, and Jun Li. 2012\.  Towards robust device-free passive localization through automatic camera-assisted recalibration. In _Proceedings of the 10th ACM Conference on Embedded Network Sensor Systems_. 339–340.
  * Zhang and Cao (2019) Renyuan Zhang and Siyang Cao. 2019.  Extending reliability of mmwave radar tracking and detection via fusion with camera.  _IEEE Access_ 7 (2019), 137065–137079.
  * Zhao et al. (2020) ChaoQiang Zhao, QiYu Sun, ChongZhen Zhang, Yang Tang, and Feng Qian. 2020.  Monocular depth estimation based on deep learning: An overview.  _Science China Technological Sciences_ 63, 9 (Jun 2020), 1612–1627.  https://doi.org/10.1007/s11431-020-1582-8
  * Zhengping Ji and Prokhorov (2008) Zhengping Ji and D. Prokhorov. 2008.  Radar-vision fusion for object classification. In _2008 11th International Conference on Information Fusion_. 1–7.