# DART: Implicit Doppler Tomography for Radar Novel View Synthesis

Tianshu Huang*1 John Miller*12 Akarsh Prabhakara1 Tao Jin1 Tarana Laroia1
Zico Kolter12 Anthony Rowe12
1Carnegie Mellon University 2Bosch Research
{tianshu2, jmiller4, aprabhak, taojin, tlaroia, agr<EMAIL_ADDRESS>
###### Abstract

Simulation is an invaluable tool for radio-frequency system designers that
enables rapid prototyping of various algorithms for imaging, target detection,
classification, and tracking. However, simulating realistic radar scans is a
challenging task that requires an accurate model of the scene, radio frequency
material properties, and a corresponding radar synthesis function. Rather than
specifying these models explicitly, we propose DART — Doppler Aided Radar
Tomography, a Neural Radiance Field-inspired method which uses radar-specific
physics to create a reflectance and transmittance-based rendering pipeline for
range-Doppler images. We then evaluate DART by constructing a custom data
collection platform and collecting a novel radar dataset together with
accurate position and instantaneous velocity measurements from lidar-based
localization. In comparison to state-of-the-art baselines, DART synthesizes
superior radar range-Doppler images from novel views across all datasets and
additionally can be used to generate high quality tomographic images.111 Our
implementation, data collection platform, and collected datasets can be found
via our project site: https://wiselabcmu.github.io/dart/.

**footnotetext: Equal Contribution.

##  1 Introduction

Driven by advances in the automotive industry, miniaturized millimeter wave
(mmWave) radar chips are becoming cheaper and more ubiquitous. Boasting a high
range resolution and the ability to penetrate light materials, mmWave radars
have proven effective in many application domains including collision
avoidance and driver assistance in automobiles [64, 57, 36, 65, 14], through-
occlusion imaging in airport scanners [68, 30], and vision-denied tracking and
mapping [9, 43, 2, 37, 22].

Because designing, testing, and deploying new radar systems in the real world
can be costly, many rapid prototyping pipelines heavily rely on simulation.
Modern radar simulation tools normally require the user to manually specify
the geometry and characteristics of the scene, including all material
properties [3]. While other sensors (e.g. lidar) can be used to scan an
environment and produce a mesh or voxel map, they cannot capture radar-
specific material properties that are crucial for generating realistic radar
scans. Thus, in practice, this results in greatly simplified environment
models due to the difficulty of meticulously surveying a scene and generating
(or annotating) a model by hand.

Figure 1: DART uses scans from a handheld radar to learn an implicit
tomography of a scene in order to accurately render scans from novel
viewpoints (left). DART’s implicit tomography can also be sampled to map the
radar properties of a scene (right).

We envision a more intelligent, data-driven approach to scene modeling for
radar simulation where a user can carry a handheld radar sensor through a
static environment and automatically generate a model suitable for accurate
simulation of that environment. To this end, we frame the radar simulation
problem as one of _novel view synthesis_ : using several radar measurements of
a scene to simulate what a radar would see from a new pose. Such a system
would not only accelerate the development and testing of new algorithms across
a variety of environmental conditions, but also open the door to a myriad of
new inference techniques in radar sensing such as localization, mapping,
imaging, and recognition which rely on accurate forward rendering models and
could greatly benefit from realistic radar models.

#### Novel View Synthesis

Neural Radiance Fields (NeRFs) [48] have revolutionized novel view synthesis,
leading to an explosion in interest in graphics and beyond. By leveraging a
(neural) _implicit_ scene representation instead of explicitly modeling scene
geometry, textures, and materials, NeRFs are able to capture and reproduce
visual intricacies such as specularity, translucency, reflections, and complex
occlusions. This results in a 3D scene capture and rendering system that
boasts an unprecedented level of photorealism.

Drawing inspiration from the success of NeRFs, we formulate an analogous
problem for mmWave radar imaging. Our method, Doppler-Aided Radar Tomography
(DART), takes a similar approach by implicitly capturing material properties
from input scans which are reproduced when the model is sampled from a novel
viewpoint. Though our model is implicit, we can also generate an explicit
tomographic image by sampling along a voxel grid, which we use to show that
DART is not simply memorizing the input data, but is in fact _learning_ the
geometry and material properties of the scene (Fig. 1).

#### Key Challenges

Applying NeRF’s implicit scene modeling paradigm to the radar domain presents
substantial challenges. We derive a rendering model from the ground up that
appropriately reflects the unique nature of radar wave propagation. In NeRF,
rendering each pixel involves integrating samples along a 1D ray, following a
pinhole camera model [48]. However, radar waves propagate _radially_ from the
antenna. Even after range-azimuth-elevation processing, each radar pixel
corresponds to a coarse 2D region of space, as the elevation-azimuth
resolution of compact mmWave radars tends to be relatively poor222For context,
these radars have angular resolutions on the order of 15∘, orders of magnitude
worse than cameras ($\approx$0.01∘) [58, 76]. One key insight is to choose a
radar representation space — range-Doppler — which greatly reduces angular
ambiguity in one dimension under the assumption that the scene is static and
the radar is moving with a known velocity [26]. This presents additional
systems challenges, as the sensor platform needs to be moving and its velocity
must be measured accurately alongside its position and orientation.

Even with the dimensionality reduction afforded by range-Doppler processing,
rendering a single radar pixel involves integrating samples along a _circle_ ,
rather than a ray (Fig. 2). However, appropriately capturing occlusion effects
requires that the nearest ranges are processed first due to occlusion caused
by objects closer to the radar. Additionally, the size of the integration arc
grows as the distance from the radar increases, resulting in an effective
decrease in sampling density for points further from the radar that needs to
be accounted for. Through careful modeling of these effects and a clever
sampling scheme prioritizing sample re-use, we derive a computationally
efficient forward rendering function that produces realistic novel radar
scans.

Figure 2: NeRF’s pinhole camera model renders a pixel (left) by integrating
along a ray (right, green), while DART’s range-Doppler model renders a pixel
(middle) by integrating along a velocity-dependent (right, blue) circle
(right, red).

#### Contributions

We propose DART: Doppler-Aided Radar Tomography, which implicitly learns a
tomographic representation of the world in order to accurately synthesize
radar range-Doppler images. To summarize our contributions:

  1. 1.

We formulate the problem of radar novel view synthesis from implicit
reflectance and transmittance maps using range-Doppler images.

  2. 2.

Using a NeRF-inspired technique, we explicitly formulate the forward rendering
of range-Doppler radar images and implicitly invert it using gradient descent
to learn a neural-implicit representation.

  3. 3.

We construct a data collection rig and collect novel radar imaging datasets
with accurate position and instantaneous velocity along with reference lidar
point clouds.

  4. 4.

We evaluate DART across a range of scenarios and show that it out-performs the
state-of-the-art, quantitatively and qualitatively, in both its synthetic
radar renderings and its implicit imaging of scenes.

#### Limitations

Since we rely on Doppler, our method is limited to static scenes, and requires
accurate velocity estimates and a constantly moving radar. While motion is
intrinsic to our method, we believe that it is reasonable to require movement
during scanning. Poor velocity estimates or non-static scenes can cause DART
to perform poorly; we hope to relax these limitations in the future.

##  2 Related Work

Figure 3: DART tackles radar novel view synthesis by learning a neural
implicit map of the world from a trajectory of radar measurements. We make key
radar-specific decisions in choosing (1) a high quality radar representation
space — Range-Doppler, (2) a world model that captures radar interactions —
$\sigma$ and $\alpha$ with spherical harmonics coefficients, (3) a network
architecture to model our desired representation — Instant NGP, and (4) an
optimized radar rendering and training methodology — Range-Doppler specific
rendering.

###  2.1 Radar Simulation

#### Model-Based Approaches

Model-based methods use a physics and environment models to simulate the
propagation of radar signals through the environment using some combination of
ray tracing [11, 3, 25, 66, 67], finite element modeling (FEM) [45, 13], or
finite-difference time domain (FDTD) simulation [19, 75, 16]. While simulators
can replicate complex scene dynamics (e.g. occlusion, path loss, multipath,
non-Lambertian effects), they make no attempt to infer the environment
structure from sensor data, and their accuracy is limited by the user’s
ability to create a radar-realistic model of the environment.

#### Data-Driven Approaches

Data-driven methods use real sensor scans to build an environment model.
_Sparse_ methods use constant false alarm rate detection (CFAR) to detect
discrete reflectors in the environment [63, 49, 15]. On the other hand,
_dense_ methods divide the environment into an explicit voxel grid and infer
the radar properties of each cell.

Dense methods can be further divided into coherent and incoherent aggregation.
If a fixed (e.g. linear or circular) trajectory or sub-wavelength-accurate
pose estimates are available, Synthetic Aperture Radar (SAR) can be used [46,
81, 82, 52, 56, 50]; however, this is impractical for a mobile platform over
large areas. Instead, sensor readings (with high angular resolution via many
antennas or SAR on smaller pieces of a trajectory) can also be aggregated in
an incoherent manner, which has been referred to as multi-view 3D
reconstruction [33, 34, 35] and radargrammetry [12].

###  2.2 Machine Learning Methods in Radar

Many classical radar problems such as radar super-resolution [23, 17, 21, 72,
10, 54, 20, 53], odometry [43, 2], mapping [42], activity recognition [70, 39,
77, 80], and object classification [32, 85, 69] have been applied to cheaper,
lighter, and more compact radar systems using machine learning. We now seek to
solve the novel view synthesis problem from compact, low resolution radars
while implicitly creating a higher resolution map.

###  2.3 Neural Radiance Fields

Instead of defining an explicit inverse imaging algorithm that recovers a
representation of the scene from sensor readings, Neural Radiance Fields [48]
implicitly invert a forward rendering function through stochastic gradient
descent. This requires the following components:

  1. 1.

World model: NeRF defines the world as an RGB color and transparency for each
position and viewing angle; subsequent works have generalized this to handle
anti-aliasing [5], different cameras, and lighting [47, 73].

  2. 2.

World representation: Beyond neural networks [48] or voxel grids [40], more
recent works have explored spatial hash tables [51] as well as function
decompositions for view angle dependence [18, 83].

  3. 3.

Rendering function and Model Inversion: NeRFs model each pixel as a ray and
ray-trace the radiance field. The invertibility of this rendering function is
crucial: by assuming that each pixel is a ray, the NeRF is “supervised” by one
RGB image pixel per ray, allowing NeRF to “solve” for the few opaque points
along the ray.

We innovate on these key enablers of NeRFs in order to apply this approach to
mmWave radars. By applying NeRF techniques to radar, we hope to leverage the
extensive body of neural radiance field literature, while also unlocking the
potential of neural-implicit representations.

#### Beyond Visual Fields

The success of NeRFs has inspired numerous other efforts to apply the same
general principle to other sensors, including spatial audio [44], imaging
sonar [55, 59], LIDAR simulations [27], and RSSI (Received Signal Strength
Indicator) mapping [84]. NeRFs have also been applied to radar [71, 29] for
camera-like high-resolution Synthetic Aperture Radar instead of the compact
and inexpensive radars we explore in this paper.

##  3 DART: Doppler-Aided Radar Tomography

While our overall approach is inspired by Neural Radiance Fields, the physics
of radar presents several new challenges. We make the following key design
decisions (Fig. 3):

  1. 1.

We first choose a radar measurement representation space — range-Doppler —
that overcomes the poor spatial resolution of compact radars (Sec. 3.1, 3.2).

  2. 2.

We then choose a model to account for radar-specific effects of
electromagnetic wave interaction which are important for realistic view
synthesis such as specularity, ghost reflections and partial occlusions (Sec.
3.3).

  3. 3.

Finally, to effectively train and learn neural implicit maps for radars, we
choose a network architecture for an adaptive grid world representation,
design a range-Doppler rendering method, and propose key rendering
optimizations (Sec. 3.3 — 3.4).

###  3.1 Range-Doppler Representation

Unlike cameras, radars are active sensors which illuminate a scene by
transmitting a radio frequency waveform. Upon processing reflections received
from objects in the scene, radars can perceive the world in 3 dimensions —
range, azimuth, and elevation — as a heatmap indicating the reflectivity of
objects at that 3D coordinate [61, 60].

However, while bulky mechanical radars or large solid-state radar arrays can
provide azimuth and elevation resolution close to typical cameras, modern
inexpensive and compact solid-state radars feature small antenna arrays which
make them far inferior in the azimuth and elevation axes [28]. As a result,
these compact radars can only generate coarse heatmaps (>$15^{\circ}$
resolution) in the azimuth and elevation axes, causing each range-azimuth-
elevation bin to point to a coarse region of 3D space which is far less sharp
than a ray from a camera pixel [41, 38, 76].

To achieve better angular resolution, radars can instead leverage the Doppler
effect: objects moving at different relative velocities to the radar have
different Doppler velocities, which can be measured by examining the residual
phase of the range-azimuth-elevation heatmap [79]. Crucially, in a static
scene, these relative velocities depend on not just the relative speed between
the radar and the world, but also the relative azimuth and elevation angle
between objects and the radar, with each Doppler corresponding to a cone in
space [60]. Because of the fine range and Doppler resolutions, Doppler greatly
reduces the ambiguity of each bin in 3D space down to a thin ring (Fig. 4),
which we further reduce by making a thinness argument across the range and
Doppler axis in order to simplify integration down to a circle for radar
rendering (Sec. 3.4).

###  3.2 Radar Pre-Processing

mmWave radars use a waveform called Frequency Modulated Continuous Wave
(FMCW), and measure a continuous time signal; we then convert these signals
into range-Doppler-antenna heatmaps. To summarize key points of our radar
processing pipeline (Appendix A.1):

  * •

Undesirable Range-Doppler Side Lobes: A single reflective object can create
sidelobes that bleed into several range-Doppler bins and mask off weaker
objects [61, 86]. Rather than forcing DART to model this, we use a Hann
weighting window along both range and Doppler axis to mitigate this effect
(Appendix A.1).

  * •

Multiple Antennas: We perform range-Doppler processing on each of the eight
transmit-receive (TX/RX) pairs in our radar. During our rendering process
(Sec. 3.4), we apply the antenna gain and array factor for each TX/RX pair
(Fig. 3), emphasizing 8 sections of the field of view. While our sense of
high-quality azimuth-elevation information still stems from leveraging
Doppler, this provides some coarse directional information.

Figure 4: Doppler arises due to differences in relative velocities between
points with different relative angles to the radar (left). Each range value
(red) corresponds to a sphere, while each Doppler value corresponds to a cone
(green). The intersection forms the range-Doppler pixel (see Fig. 2).

###  3.3 DART’s World Model

If we had an accurate model of the world and the electromagnetic wave
interaction for all objects in the world, we could just apply this model to
the region defined by each range-Doppler pixel to calculate its value.
However, due to the complex nature of real-world scenes and interactions, both
tasks are highly difficult and typically impractical. Instead, we model these
properties in a data-driven way, representing the reflectance and
transmittance using a view-dependent neural network-based approach.

#### Modeling RF Reflectivity

Modeling mmWave material interactions is one of the most challenging factors
of radar view synthesis. From the perspective of radar, points in space have
two key properties: reflectance (the proportion of energy that reflects back),
and transmittance (the proportion of energy that continues past) [60].
However, millimeter waves also interact with objects differently depending on
incidence angles [4]; for example, metal surfaces can be specular and may be
invisible from certain view points. As such, we model each physical point with
a reflectance and transmittance value, each of which depend on the incident
angle. We formalize this as

$\displaystyle\sigma:\mathbb{R}^{6}\longrightarrow\mathbb{R},\quad\quad\alpha:\mathbb{R}^{6}\longrightarrow[0,1],$
(1)

which model the reflectance $\sigma$ and transmittance $\alpha$ as a function
of the position ($\mathbb{R}^{3}$) and incident angle ($\mathbb{R}^{3}$) of an
incoming radar wave, and allows DART to model a wide range of radar phenomena
such as partial occlusions, specularity, and ghost reflections (Appendix A.2).

#### World Representation

While voxel-based approaches are highly effective for learning visual radiance
fields [18, 83], radar images have a much poorer elevation and azimuth
resolution compared to cameras even after exploiting the Doppler axis. This
magnifies the difference in spatial resolution that $\sigma$ and $\alpha$ can
be resolved for between close and far ranges. Moreover, unlike cameras, our
angular resolution is variable at all scales — be it at a trajectory level,
frame-to-frame level or even within a frame (Sec. 3.1). Similar to NeRFs [48],
we turn to neural implicit representations as a means of creating an
“adaptive” grid, and base our model on the Instant Neural Graphics
Primitive333 [51] implicitly creates an adaptive world grid by using many
spatial hash tables with geometrically increasing resolutions, and resolves
the output with a small neural network; we use the same general architecture.
[51].

Unlike most visual NeRFs, we do not provide the incident angle as an input to
the neural network [74]. Instead, our architecture (visualized in the center
block of Fig. 3) outputs a “base” reflectance $\bar{\sigma}$ and transmittance
$\bar{\alpha}$, as well as shared spherical harmonics coefficients [83] which
are applied to the incident angle as an inner product. In addition to
computational advantages, this allows us to directly interpret
$(\bar{\sigma},\bar{\alpha})$ as spherical integrals of our learned
reflectance and transmittance functions (Appendix A.3).

We also find that the output activation function on $\sigma$ and $\alpha$ is
critical for numerical stability and performance. Since $\sigma$ is
unbounded444 $\sigma$ can be negative; however, since the observed range-
Doppler-antenna heatmaps cannot be negative, $\sigma<0$ will always increase
both train and validation loss, so allowing this does not cause overfitting. ,
we apply a linear activation to $\sigma$. Then, to constrain $\alpha\in[0,1]$,
we apply the activation function

$\displaystyle f(\alpha)=\exp(\max(0,\alpha)),$ (2)

which we pair with a custom gradient estimator to handle initialization
instability (Appendix A.4).

###  3.4 Radar Rendering and Model Training

We train $\sigma$ and $\alpha$ using a differentiable mapping which generates
a multi-antenna range-Doppler heatmap from a given $(\sigma,\alpha)$ network;
we refer to this as radar rendering. Unlike visual NeRFs, DART must account
for a range of physical effects in addition to occlusion including path
attenuation, antenna gain patterns, and the radar-specific Doppler axis.

#### Ray Tracing

Consider a single “ray” emitted from a radar at position $\bm{x}$ and
orientation (rotation matrix) $\bm{A}$ at an incidence angle $\bm{w}$. As the
ray travels through space up to the maximum range of the processed (range,
Doppler, antenna) image, each point $\bm{x}+r_{i}\bm{w}$ at range $r_{i}$
receives a signal of amplitude $u_{i}$, which is attenuated by a factor of
$r_{i}$ due to free space path loss. Each point then reflects a signal of
amplitude $u_{i}\sigma(\bm{t}_{i})$ back towards the radar, and propagates an
amplitude of $u_{i}\alpha(\bm{t}_{i})$ onwards. As reflected signals return to
the radar, the signal loses an additional attenuation factor of $r_{i}$, while
also suffering from occlusion from $\forall j<i:\alpha(\bm{t}_{j})$.

Sampling $r_{1}\ldots r_{N_{r}}$ discretely along the range bins of the
processed heatmap across antennas, the radar return amplitude $C(i,k,\bm{w})$
for ray $\bm{w}$ at range bin $i$ and antenna $k$ is

$\displaystyle
C(i,k,\bm{w})=g_{k}(\bm{A}^{-1}\bm{w})\frac{\sigma(\bm{x}+r_{i}\bm{w})}{r_{i}^{2}}\prod_{i^{\prime}=1}^{i-1}\alpha(\bm{t}_{i^{\prime}})^{2},$
(3)

where $g_{k}(\bm{A}^{-1}\bm{w})$ is the antenna beamforming gain antenna $k$
at angle $\bm{w}$ (specified relative to the radar orientation $\bm{A}$).

#### Doppler Integration

For a given pose with radar position $\bm{x}$, velocity $\bm{v}$, and
orientation $\bm{A}$, we evaluate the return
$\bm{Y}(r_{i},d_{j},k)\in\mathbb{R}$ at each range-Doppler-antenna bin
($r_{i},d_{j},k$), synthesizing a view-specific, multi-antenna range-Doppler
heatmap. Since the doppler velocity is measured as
$d_{j}=\langle\bm{w},\bm{v}\rangle$, we integrate the return $C$ along the
thin ring corresponding to each bin as:

$\displaystyle\bm{Y}(r_{i},d_{j},k)\propto\frac{r_{i}}{||\bm{v}||_{2}}\int\displaylimits_{\langle\bm{w},\bm{v}\rangle=d_{j},\
||\bm{w}||_{2}=1}\mkern-36.0muC(i,k,\bm{w})\ d\bm{w}$ (4)

Note that we need to correct for the varying width of the discrete bins as a
function of range and radar speed by dividing by the speed $||\bm{v}||_{2}$
and multiply by $r_{i}$ (Appendix A.5).

Figure 5: Example (validation) range-Doppler frames and descriptive photos of
our method and baselines. DART accurately reproduces the overall radar image,
though it lacks the resolution to resolve smaller weak reflectors. Lidar can
model weak reflectors, but cannot accurately scale them due to a lack of
radar-specific information, while Nearest produces radar-realistic but
inaccurate images since exhaustively measuring all possible poses is
impractical. Finally, CFAR cannot model transmittance or measure the “volume”
of a point.

Approximating this integral as a sum over $M$ random directions
$\bm{w}_{1}\ldots\bm{w}_{M}$ such that $\langle\bm{w},\bm{v}\rangle=d_{j}$, we
multiply by an additional factor of $r_{i}$ to correct for the circumference
of the range-Doppler intersection as $r_{i}$ increases. This yields

$\displaystyle\bm{Y}(r_{i},d_{j},k)\propto\frac{r_{i}^{2}}{M||\bm{v}||_{2}}\sum_{m=1}^{M}C(i,k,\bm{w_{m}}).$
(5)

#### Optimized Rendering

As the $(\sigma,\alpha)$ field function must be evaluated for every sample,
efficient sampling is critical to computational efficiency. Thus, a naive
approach of treating each (range, Doppler, antenna) “pixel” as an independent
sample as is standard practice in NeRFs would be computationally prohibitive,
requiring the field to be sampled (range, Doppler, antenna, range integration,
Doppler integration) times to render a single image. As such, we aggressively
reuse samples of $\sigma$ and $\alpha$ by rendering all bins with the same
Doppler simultaneously (Appendix A.6).

#### Training

We train our ($\sigma,\alpha$) field function using stochastic gradient
descent with the Adam [31] optimizer and a $l$1 (i.e. mean-absolute-error)
loss. For details about our training process and other hyperparameters, see
Appendix A.7.

##  4 Experiments

We constructed a handheld data collection rig with a mmWave radar and a lidar
used for localization555Note that while we use lidar to obtain pose estimates
using Cartographer [24], any accurate 3D SLAM system can be used. (Fig. 6;
Appendix B.1). We used this to collect 12 traces ranging from 5 to 15 minutes
long in a diverse set of environments including a lab space, townhouse, high-
rise apartment, and an early 20th century house (Appendix B.2).

###  4.1 Baselines

We implement three baselines for radar novel view synthesis and mapping, a
_model-based_ approach and two _data-driven_ approaches (see Sec. 2.1).

  * •

Lidar Scan-Based Simulator: We use lidar scans to create an occupancy grid,
which we then use in a raytracing radar simulator (assuming occupied grids
have a fixed constant reflectance and no transmittance, similar to [3] without
material annotations). This baseline represents the standard practice in radar
simulation [3, 75, 13, 66].

  * •

Nearest Neighbor: We implement a naive nearest-neighbor baseline which finds
the training point with the closest (position, velocity) to the novel
viewpoint. While simple, this has the advantage of using radar data to
“simulate” images compared to our lidar-based simulator [7].

  * •

CFAR Point Cloud Aggregation: CFAR is a commonly used adaptive algorithm in
radar systems to detect target returns against a background of noise, clutter
and interference [49]. We use the Matlab Phased Array System Toolbox [1] to
detect radar-reflective targets, de-project those targets into 3D points using
Bartlett direction-of-arrival estimation [6], then reproject the points
according to the novel pose. This approach is similar to our lidar baseline in
that it uses point cloud aggregation, but is better able to capture radar-
specific scene properties.

For additional details on our baselines, see Appendix B.3.

Figure 6: Handheld data collection system; see Appendix B.1. Figure 7:
Comparison of DART (top) with CFAR (middle) and a Lidar occupancy grid for
reference. While CFAR struggles with cluttered scenes and creates point clouds
which are both noisy and sparse, DART creates relatively clear maps which
capture radar-specific properties on both ourdoors (Garden) and indoors
(Apartment) environments. DART can also image objects with relative clarity
(Car), including resolving objects partially occluded by radar-transparent
surfaces (Tent — Occupied/Empty).

###  4.2 Metrics

We apply our model to a holdout test set consisting of the last 20% of each
trace. We then compute the SSIM [78] of the test images and the effective
sample size-corrected standard error (SE) for the mean SSIM; see Appendix B.4.
We also compute the SSIM values of 25/30/35dB-equivalent Gaussian noise to
help quantify our SSIM values.

##  5 Results

DART synthesizes significantly more accurate radar images than each baseline
on all traces collected, while using minimal training. We also demonstrate
DART’s ability to sample tomographic images from its implicit map which are
more dense than CFAR point clouds, and more faithfully reproduce radar
characteristics than lidar scans.

#### Training Time

We train DART for 3 epochs on each dataset using a RTX 4090 GPU, taking
between 1-2$\times$ the data collection time ($\approx$ 10 minutes) of each
dataset666 Training time is not directly proportional to the dataset length:
since Doppler bins are not observed when the radar speed is less than the
Doppler velocity of that bin, we omit these bins, decreasing the training
time. See Appendix B.2 for the length and training time of each dataset. ;
this indicates the potential of real-time training with future algorithmic and
computing hardware improvements.

#### Ablations

Each part of DART’s design significantly improves its accuracy, including view
dependence using spherical harmonics and our dynamic grid representation (Tab.
1). For additional ablations, see Appendix C.1.

###  5.1 Comparison with Baselines

DART synthesizes far more accurate radar images than each baseline on all
traces in our dataset (Appendix C.2), with the Lidar-based simulator and
Nearest Neighbor baselines performing the worst, and CFAR-based simulation in
between. DART is also significantly better than each baseline when evaluated
as a whole (Tab. 1).

To understand the performance differences between DART and each baseline, we
selected two example range-Doppler images from our dataset (Fig. 5):

  * •

Lidar-based simulation (Lidar) can accurately identify reflector positions,
but cannot correctly scale their radar return due to the lack of radar-
specific material properties.

  * •

Nearest-Neighbor (Nearest) approaches can, by definition, generate radar-
realistic images. However, measuring all possible (position, orientation,
velocity) poses is impractical, leading to “misplaced” images which do not
vary continuously over different poses.

  * •

Constant False Alarm Rate (CFAR) is commonly used to generate point clouds
from radar images. Compared to lidar point clouds, CFAR point clouds are
sparse and low-resolution, but capture radar specific properties not measured
in lidar. However, CFAR cannot provide any notion of the size of each point or
its transmittance, which requires the point or grid size to be manually tuned,
leading to either excessively sparse or blurry images.

DART therefore achieves its efficacy by using a domain-appropriate sensor and
carefully selecting a representation which allows it to use all available
sensor information.

Method | Mean SSIM | SSIM Improvement
---|---|---
DART |  0.636 $\pm$ 0.012 | —
Lidar |  0.463 $\pm$ 0.005 |  0.174 $\pm$ 0.013
Nearest |  0.468 $\pm$ 0.006 |  0.168 $\pm$ 0.012
CFAR |  0.545 $\pm$ 0.007 |  0.091 $\pm$ 0.006
No View Dep. |  0.614 $\pm$ 0.015 |  0.022 $\pm$ 0.005
20cm Grid |  0.591 $\pm$ 0.015 |  0.046 $\pm$ 0.004
Table 1: Mean SSIM and SSIM improvement of DART over each baseline (and select
ablations) across our dataset along with 95% confidence intervals; see
Appendix C.2 for a breakdown by dataset.

###  5.2 Tomography and Mapping

Figure 8: Tomographic images of 5 boxes made from different materials: (1) a
metal filing cabinet which appears less reflective (due to specularity), but
blocks radar waves; (2) an empty box which reflects radar waves but does not
block them; (3) a stack of boxes containing electronics equipment which both
reflect and block radar waves; (4) a highly reflective metal mesh with large
holes that allow radar to penetrate it; and (5) a different empty box which
neither reflects nor blocks radar waves.

While DART is not designed primarily as an explicit tomography or mapping
tool, we can sample the implicit representation777 To address view dependence,
we analytically take the spherical integral of $\sigma$ and $\alpha$ at each
point; see Appendix A.3 for details.  to create a $(\sigma,\alpha)$
reflectance and transmittance grid. This also allows us to verify that DART
truly learns the mmWave properties of a scene (and does not simply memorize
and interpolate the training data).

#### Material Properties Example

We created an evaluation scene with 5 different boxes. DART is able to learn
the unique reflectance and transmittance properties of each materials, which
we visualize through tomographic reflectance and transmittance maps (Fig. 8).
For additional examples from our datasets, see Appendix C.3.

#### Comparison with Baselines

In addition to creating more accurate radar simulations, DART can also produce
more accurate and dense maps than CFAR. Fig. 7 shows several examples
comparing tomographic maps of reflectance learned by DART with corresponding
slices of the point cloud generated by CFAR. While not as sharp as lidar
scans, DART produces reasonably clear maps which capture the radar-specific
properties of each scene.

##  6 Conclusion

We present DART: Doppler Aided Radar Tomography, a NeRF-inspired radar novel
view synthesis algorithm which learns an implicit tomographic map from range-
Doppler images, and demonstrate its effectiveness against state-of-the-art
baselines. We derive a physics-based rendering model for radar from first
principles, and construct an end-to-end system for learning an implicit scene
representation and generate realistic novel radar views. While DART provides a
strong baseline for future work, many opportunities remain to apply lessons
learned from visual NeRFs; given the rapid pace of innovation in NeRF, these
opportunities will likely multiply in the coming years. We also currently make
a number of assumptions – such as a static scene and the availability of
accurate ground-truth pose – which could be relaxed as has been done with
visual NeRFs, enabling a single-chip radar solution for localization, mapping,
and imaging. Finally, as we add mmWave radar to the repertoire of NeRF-enabled
sensing technologies, this furthers the potential for multimodal implicit
mapping in the future.

## References

  * [1] Constant false alarm rate (cfar) detection.  Url: https://www.mathworks.com/help/phased/ug/constant-false-alarm-rate-cfar-detection.html; Accessed: 2023-11-15.
  * Almalioglu et al. [2020] Yasin Almalioglu, Mehmet Turan, Chris Xiaoxuan Lu, Niki Trigoni, and Andrew Markham.  Milli-rio: Ego-motion estimation with low-cost millimetre-wave radar.  _IEEE Sensors Journal_ , 21(3):3314–3323, 2020.
  * Auer et al. [2016] Stefan Auer, Richard Bamler, and Peter Reinartz.  Raysar - 3d sar simulator: Now open source.  2016\.
  * Bansal et al. [2020] Kshitiz Bansal, Keshav Rungta, Siyuan Zhu, and Dinesh Bharadia.  Pointillism: Accurate 3d bounding box estimation with multi-radars.  In _Proceedings of the 18th Conference on Embedded Networked Sensor Systems_ , pages 340–353, 2020.
  * Barron et al. [2021] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5855–5864, 2021.
  * Bartlett [1950] M. S. Bartlett.  Periodogram analysis and continuous spectra.  _Biometrika_ , 37(1/2):1–16, 1950.
  * Bhatia et al. [2010] Nitin Bhatia et al.  Survey of nearest neighbor techniques.  _arXiv preprint arXiv:1007.0085_ , 2010.
  * Bradbury et al. [2018] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang.  JAX: composable transformations of Python+NumPy programs, 2018.
  * Cen and Newman [2018] Sarah H Cen and Paul Newman.  Precise ego-motion estimation with millimeter-wave radar under diverse and challenging conditions.  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 6045–6052. IEEE, 2018.
  * Cheng et al. [2022] Yuwei Cheng, Jingran Su, Mengxin Jiang, and Yimin Liu.  A novel radar point cloud generation method for robot environment perception.  _IEEE Transactions on Robotics_ , 2022.
  * Coleman [1998] C. J. Coleman.  A ray tracing formulation and its application to some problems in over-the-horizon radar.  _Radio Science_ , 33(4):1187–1197, 1998.
  * Crosetto and Pérez Aragues [2000] Michele Crosetto and F Pérez Aragues.  Radargrammetry and sar interferometry for dem generation: validation and data fusion.  In _SAR workshop: CEOS committee on earth observation satellites_ , page 367, 2000.
  * di and Wang [2004] Qingyun di and Miaoyue Wang.  Migration of ground-penetrating radar data method with a finite-element and dispersion.  _Geophysics_ , 69, 2004.
  * Ding et al. [2023] Fangqiang Ding, Andras Palffy, Dariu M. Gavrila, and Chris Xiaoxuan Lu.  Hidden gems: 4d radar scene flow learning using cross-modal supervision.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 9340–9349, 2023.
  * Doer and Trommer [2021] Christopher Doer and Gert F. Trommer.  Yaw aided radar inertial odometry uisng manhattan world assumptions.  In _2021 28th Saint Petersburg International Conference on Integrated Navigation Systems (ICINS)_ , pages 1–10, 2021.
  * Ernst et al. [2007] Jacques R. Ernst, Hansruedi Maurer, Alan G. Green, and Klaus Holliger.  Full-waveform inversion of crosshole radar data based on 2-d finite-difference time-domain solutions of maxwell’s equations.  _IEEE Transactions on Geoscience and Remote Sensing_ , 45(9):2807–2828, 2007.
  * Fang and Nirjon [2020] Shiwei Fang and Shahriar Nirjon.  Superrf: Enhanced 3d rf representation using stationary low-cost mmwave radar.  In _International Conference on Embedded Wireless Systems and Networks (EWSN)…_ , page 120. NIH Public Access, 2020.
  * Fridovich-Keil et al. [2022] Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5501–5510, 2022.
  * Furse et al. [1990] C.M. Furse, S.P. Mathur, and O.P. Gandhi.  Improvements to the finite-difference time-domain method for calculating the radar cross section of a perfectly conducting target.  _IEEE Transactions on Microwave Theory and Techniques_ , 38(7):919–927, 1990.
  * Gao et al. [2021] Xiangyu Gao, Sumit Roy, and Guanbin Xing.  Mimo-sar: A hierarchical high-resolution imaging algorithm for mmwave fmcw radar in autonomous driving.  _IEEE Transactions on Vehicular Technology_ , 70(8):7322–7334, 2021.
  * Geiss and Hardin [2020] Andrew Geiss and Joseph C Hardin.  Radar super resolution using a deep convolutional neural network.  _Journal of Atmospheric and Oceanic Technology_ , 37(12):2197–2207, 2020.
  * Guan et al. [2020a] Junfeng Guan, Sohrab Madani, Suraj Jog, Saurabh Gupta, and Haitham Hassanieh.  Through fog high-resolution imaging using millimeter wave radar.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020a.
  * Guan et al. [2020b] Junfeng Guan, Sohrab Madani, Suraj Jog, Saurabh Gupta, and Haitham Hassanieh.  Through fog high-resolution imaging using millimeter wave radar.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11464–11473, 2020b.
  * Hess et al. [2016] Wolfgang Hess, Damon Kohler, Holger Rapp, and Daniel Andor.  Real-time loop closure in 2d lidar slam.  In _2016 IEEE international conference on robotics and automation (ICRA)_ , pages 1271–1278. IEEE, 2016.
  * Hirsenkorn et al. [2017] Nils Hirsenkorn, Paul Subkowski, Timo Hanke, Alexander Schaermann, Andreas Rauch, Ralph Rasshofer, and Erwin Biebl.  A ray launching approach for modeling an fmcw radar system.  In _2017 18th International Radar Symposium (IRS)_ , pages 1–10, 2017.
  * Hlawatsch [1998] Franz Hlawatsch.  _Time-frequency analysis and synthesis of linear signal spaces: time-frequency filters, signal detection and estimation, and Range-Doppler estimation_.  Springer Science & Business Media, 1998.
  * Huang et al. [2023] Shengyu Huang, Zan Gojcic, Zian Wang, Francis Williams, Yoni Kasten, Sanja Fidler, Konrad Schindler, and Or Litany.  Neural lidar fields for novel view synthesis.  _arXiv preprint arXiv:2305.01643_ , 2023.
  * Iovescu and Rao [2017] Cesar Iovescu and Sandeep Rao.  The fundamentals of millimeter wave sensors.  _Texas Instruments_ , pages 1–8, 2017.
  * Jamora et al. [2023] JR Jamora, Dylan Green, Ander Talley, and Thomas Curry.  Utilizing sar imagery in three-dimensional neural radiance fields-based applications.  In _Algorithms for Synthetic Aperture Radar Imagery XXX_ , page 1252002. SPIE, 2023.
  * Khatun et al. [2017] Mahfuza Khatun, Hani Mehrpouyan, David Matolak, and Ismail Guvenc.  Millimeter wave systems for airports and short-range aviation communications: A survey of the current channel models at mmwave frequencies.  In _2017 IEEE/AIAA 36th Digital Avionics Systems Conference (DASC)_ , pages 1–8, 2017.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kosuge et al. [2022] Atsutake Kosuge, Satoshi Suehiro, Mototsugu Hamada, and Tadahiro Kuroda.  mmwave-yolo: A mmwave imaging radar-based real-time multiclass object recognition system for adas applications.  _IEEE Transactions on Instrumentation and Measurement_ , 71:1–10, 2022.
  * Laviada et al. [2017a] Jaime Laviada, Ana Arboleya-Arboleya, Yuri Álvarez, Borja González-Valdés, and Fernando Las-Heras.  Multiview three-dimensional reconstruction by millimetre-wave portable camera.  _Scientific reports_ , 7(1):6479, 2017a.
  * Laviada et al. [2017b] Jaime Laviada, Ana Arboleya-Arboleya, and Fernando Las-Heras.  Multistatic millimeter-wave imaging by multiview portable camera.  _IEEE Access_ , 5:19259–19268, 2017b.
  * Laviada et al. [2018] Jaime Laviada, Miguel Lopez-Portugues, Ana Arboleya-Arboleya, and Fernando Las-Heras.  Multiview mm-wave imaging with augmented depth camera information.  _IEEE Access_ , 6:16869–16877, 2018.
  * Li et al. [2022a] Peizhao Li, Pu Wang, Karl Berntorp, and Hongfu Liu.  Exploiting temporal relations on radar perception for autonomous driving.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 17071–17080, 2022a.
  * Li et al. [2022b] Yu-Jhe Li, Jinhyung Park, Matthew O’Toole, and Kris Kitani.  Modality-agnostic learning for radar-lidar fusion in vehicle detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 918–927, 2022b.
  * Li et al. [2023] Yu-Jhe Li, Shawn Hunt, Jinhyung Park, Matthew O’Toole, and Kris Kitani.  Azimuth super-resolution for fmcw radar in autonomous driving.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 17504–17513, 2023.
  * Lien et al. [2016] Jaime Lien, Nicholas Gillian, M Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev.  Soli: Ubiquitous gesture sensing with millimeter wave radar.  _ACM Transactions on Graphics (TOG)_ , 35(4):1–19, 2016.
  * Liu et al. [2020] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt.  Neural sparse voxel fields.  _Advances in Neural Information Processing Systems_ , 33:15651–15663, 2020.
  * Long et al. [2021] Yunfei Long, Daniel Morris, Xiaoming Liu, Marcos Castro, Punarjay Chakravarty, and Praveen Narayanan.  Radar-camera pixel depth association for depth completion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 12507–12516, 2021.
  * Lu et al. [2020a] Chris Xiaoxuan Lu, Stefano Rosa, Peijun Zhao, Bing Wang, Changhao Chen, John A Stankovic, Niki Trigoni, and Andrew Markham.  See through smoke: robust indoor mapping with low-cost mmwave radar.  In _Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services_ , pages 14–27, 2020a.
  * Lu et al. [2020b] Chris Xiaoxuan Lu, Muhamad Risqi U Saputra, Peijun Zhao, Yasin Almalioglu, Pedro PB De Gusmao, Changhao Chen, Ke Sun, Niki Trigoni, and Andrew Markham.  milliego: single-chip mmwave radar aided egomotion estimation via deep sensor fusion.  In _Proceedings of the 18th Conference on Embedded Networked Sensor Systems_ , pages 109–122, 2020b.
  * Luo et al. [2022] Andrew Luo, Yilun Du, Michael Tarr, Josh Tenenbaum, Antonio Torralba, and Chuang Gan.  Learning neural acoustic fields.  In _Advances in Neural Information Processing Systems_ , pages 3165–3177. Curran Associates, Inc., 2022.
  * Malinen and Råback [2013] M. Malinen and P. Råback.  _Elmer finite element solver for multiphysics and multiscale problems_.  Forschungszentrum Juelich, 2013.
  * Mamandipoor et al. [2014] Babak Mamandipoor, Greg Malysa, Amin Arbabian, Upamanyu Madhow, and Karam Noujeim.  60 ghz synthetic aperture radar for short-range imaging: Theory and experiments.  In _2014 48th Asilomar Conference on Signals, Systems and Computers_ , pages 553–558. IEEE, 2014.
  * Martin-Brualla et al. [2021] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth.  NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections.  In _CVPR_ , 2021.
  * Mildenhall et al. [2021] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Communications of the ACM_ , 65(1):99–106, 2021.
  * Minkler and Minkler [1990] G. Minkler and J. Minkler.  CFAR: The principles of automatic radar detection in clutter.  _NASA STI/Recon Technical Report A_ , 90:23371, 1990.
  * Mostajabi et al. [2020] Mohammadreza Mostajabi, Ching Ming Wang, Darsh Ranjan, and Gilbert Hsyu.  High-resolution radar dataset for semi-supervised learning of dynamic objects.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , 2020.
  * Müller et al. [2022] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Transactions on Graphics (ToG)_ , 41(4):1–15, 2022.
  * Prabhakara et al. [2020] Akarsh Prabhakara, Vaibhav Singh, Swarun Kumar, and Anthony Rowe.  Osprey: a mmwave approach to tire wear sensing.  In _Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services_ , pages 28–41, 2020.
  * Prabhakara et al. [2022] Akarsh Prabhakara, Diana Zhang, Chao Li, Sirajum Munir, Aswin C Sankaranarayanan, Anthony Rowe, and Swarun Kumar.  Exploring mmwave radar and camera fusion for high-resolution and long-range depth imaging.  In _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 3995–4002. IEEE, 2022.
  * Prabhakara et al. [2023] Akarsh Prabhakara, Tao Jin, Arnav Das, Gantavya Bhatt, Lilly Kumari, Elahe Soltanaghaei, Jeff Bilmes, Swarun Kumar, and Anthony Rowe.  High resolution point clouds from mmwave radar.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , 2023.
  * Qadri et al. [2022] Mohamad Qadri, Michael Kaess, and Ioannis Gkioulekas.  Neural implicit surface reconstruction using imaging sonar.  _arXiv preprint arXiv:2209.08221_ , 2022.
  * Qian et al. [2020] Kun Qian, Zhaoyuan He, and Xinyu Zhang.  3d point cloud generation with millimeter-wave radar.  _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ , 4(4), 2020.
  * Qian et al. [2021] Kun Qian, Shilin Zhu, Xinyu Zhang, and Li Erran Li.  Robust multimodal vehicle detection in foggy weather using complementary lidar and radar signals.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 444–453, 2021.
  * Rebut et al. [2022] Julien Rebut, Arthur Ouaknine, Waqas Malik, and Patrick Pérez.  Raw high-definition radar for multi-task learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 17021–17030, 2022.
  * Reed et al. [2023] Albert W Reed, Juhyeon Kim, Thomas Blanford, Adithya Pediredla, Daniel C Brown, and Suren Jayasuriya.  Neural volumetric reconstruction for coherent synthetic aperture sonar.  _arXiv preprint arXiv:2306.09909_ , 2023.
  * Richards et al. [2010] Mark A Richards, Jim Scheer, William A Holm, and William L Melvin.  _Principles of Modern Radar_.  Citeseer, 2010.
  * Richards et al. [2005] Mark A Richards et al.  _Fundamentals of radar signal processing_.  Mcgraw-hill New York, 2005.
  * Robert et al. [1999] Christian P Robert, George Casella, and George Casella.  _Monte Carlo statistical methods_.  Springer, 1999.
  * Rohling [1983] Hermann Rohling.  Radar cfar thresholding in clutter and multiple target situations.  _IEEE Transactions on Aerospace and Electronic Systems_ , AES-19(4):608–621, 1983.
  * Russell et al. [1997] Mark E Russell, Arthur Crain, Anthony Curran, Richard A Campbell, Clifford A Drubin, and Willian F Miccioli.  Millimeter-wave radar sensor for automotive intelligent cruise control (icc).  _IEEE Transactions on microwave theory and techniques_ , 45(12):2444–2453, 1997.
  * Scheiner et al. [2020] Nicolas Scheiner, Florian Kraus, Fangyin Wei, Buu Phan, Fahim Mannan, Nils Appenrodt, Werner Ritter, Jurgen Dickmann, Klaus Dietmayer, Bernhard Sick, and Felix Heide.  Seeing around street corners: Non-line-of-sight detection and tracking in-the-wild using doppler radar.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Schöffmann et al. [2021] Christian Schöffmann, Barnaba Ubezio, Christoph Böhm, Stephan Mühlbacher-Karrer, and Hubert Zangl.  Virtual radar: Real-time millimeter-wave radar sensor simulation for perception-driven robotics.  _IEEE Robotics and Automation Letters_ , 6(3):4704–4711, 2021.
  * Schüßler et al. [2021] Christian Schüßler, Marcel Hoffmann, Johanna Bräunig, Ingrid Ullmann, Randolf Ebelt, and Martin Vossiek.  A realistic radar ray tracing simulator for large mimo-arrays in automotive environments.  _IEEE Journal of Microwaves_ , 1(4):962–974, 2021.
  * Sheen et al. [1998] David M Sheen, Douglas L McMakin, and Thomas E Hall.  Cylindrical millimeter-wave imaging technique for concealed weapon detection.  In _26th AIPR Workshop: Exploiting New Image Sources and Sensors_ , pages 242–250. SPIE, 1998.
  * Shuai et al. [2021] Xian Shuai, Yulin Shen, Yi Tang, Shuyao Shi, Luping Ji, and Guoliang Xing.  millieye: A lightweight mmwave radar and camera fusion system for robust object detection.  In _Proceedings of the International Conference on Internet-of-Things Design and Implementation_ , pages 145–157, 2021.
  * Singh et al. [2019] Akash Deep Singh, Sandeep Singh Sandha, Luis Garcia, and Mani Srivastava.  Radhar: Human activity recognition from point clouds generated through a millimeter-wave radar.  In _Proceedings of the 3rd ACM Workshop on Millimeter-wave Networks and Sensing Systems_ , pages 51–56, 2019.
  * Snyder et al. [2023] William Snyder, Stephen DelMarco, Dylan Snover, Amit Bhatia, and Scott Kuzdeba.  Extending neural radiance fields (nerf) for synthetic aperture radar (sar) novel image generation.  In _Synthetic Data for Artificial Intelligence and Machine Learning: Tools, Techniques, and Applications_ , pages 268–276. SPIE, 2023.
  * Sun et al. [2021] Yue Sun, Honggang Zhang, Zhuoming Huang, and Benyuan Liu.  Deeppoint: A deep learning model for 3d reconstruction in point clouds via mmwave radar.  _arXiv preprint arXiv:2109.09188_ , 2021.
  * Tancik et al. [2022] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar.  Block-nerf: Scalable large scene neural view synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8248–8258, 2022.
  * Tancik et al. [2023] Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, et al.  Nerfstudio: A modular framework for neural radiance field development.  _arXiv preprint arXiv:2302.04264_ , 2023.
  * Teixeira et al. [1998] F.L. Teixeira, Weng Cho Chew, M. Straka, M.L. Oristaglio, and T. Wang.  Finite-difference time-domain simulation of ground penetrating radar on dispersive, inhomogeneous, and conductive soils.  _IEEE Transactions on Geoscience and Remote Sensing_ , 36(6):1928–1937, 1998.
  * Tex [2021] _AWR1843AOP Single-chip 77- and 79-GHz FMCW mmWave Sensor Antennas-OnPackage (AOP)_.  Texas Instruments, 2021.
  * Wang et al. [2021] Yuheng Wang, Haipeng Liu, Kening Cui, Anfu Zhou, Wensheng Li, and Huadong Ma.  m-activity: Accurate and real-time human activity recognition via millimeter wave radar.  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 8298–8302. IEEE, 2021.
  * Wang et al. [2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ , 13(4):600–612, 2004.
  * Wasserzier et al. [2019] Christoph Wasserzier, Josef G. Worms, and Daniel W. O’Hagan.  How noise radar technology brings together active sensing and modern electronic warfare techniques in a combined sensor concept.  In _2019 Sensor Signal Processing for Defence Conference (SSPD)_ , pages 1–5, 2019.
  * Xue et al. [2022] Hongfei Xue, Qiming Cao, Yan Ju, Haochen Hu, Haoyu Wang, Aidong Zhang, and Lu Su.  M4esh: mmwave-based 3d human mesh construction for multiple subjects.  In _Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems_ , pages 391–406, 2022.
  * Yamada et al. [2017] Hiroyoshi Yamada, Takumi Kobayashi, Yoshio Yamaguchi, and Yuuichi Sugiyama.  High-resolution 2d sar imaging by the millimeter-wave automobile radar.  In _2017 IEEE Conference on Antenna Measurements & Applications (CAMA)_, pages 149–150. IEEE, 2017.
  * Yanik and Torlak [2019] Muhammet Emin Yanik and Murat Torlak.  Near-field mimo-sar millimeter-wave imaging with sparsely sampled aperture data.  _Ieee Access_ , 7:31801–31819, 2019.
  * Yu et al. [2021] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa.  Plenoctrees for real-time rendering of neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5752–5761, 2021.
  * Zhao et al. [2023] Xiaopeng Zhao, Zhenlin An, Qingrui Pan, and Lei Yang.  Nerf2: Neural radio-frequency radiance fields.  _arXiv preprint arXiv:2305.06118_ , 2023.
  * Zhao et al. [2020] Zihao Zhao, Yuying Song, Fucheng Cui, Jiang Zhu, Chunyi Song, Zhiwei Xu, and Kai Ding.  Point cloud features-based kernel svm for human-vehicle classification in millimeter wave radar.  _IEEE Access_ , 8:26012–26021, 2020.
  * Zhou et al. [2022] Yi Zhou, Lulu Liu, Haocheng Zhao, Miguel López-Benítez, Limin Yu, and Yutao Yue.  Towards deep radar perception for autonomous driving: Datasets, methods, and challenges.  _Sensors_ , 22(11), 2022.

Supplementary Material

##  Appendix A Method Details

DART represents the sum of careful consideration of each step in the novel
view synthesis pipeline from radar pre-processing to rendering. We provide
additional details not included in our main paper, including our radar
processing pipeline (Sec. A.1), spherical harmonics representation (Sec. A.3),
transmittance representation and custom gradient estimator (Sec. A.4),
optimized radar rendering pipeline (Sec. A.6), and hyperparameters (Sec. A.7).

###  A.1 Radar Pre-Processing

Our radar processing pipeline (Fig. 9) consists of a range, Doppler, and
azimuth fast Fourier transform (FFT). The range and Doppler FFT use a Hann
window in order to suppress undesirable side lobes. Finally, we perform range
decimation to take the first 128 valid range bins of each processed range-
Doppler-azimuth heatmap, and apply calibration to account for antenna delay.

Note that since we do not have sub-mm pose accuracy or restrict data
collection to fixed trajectories, DART does not use coherent radar images. As
such, while each FFT produces complex values, we only use the magnitude of the
final output.

#### Range FFT

State-of-the-art solid-state compact radars operate in the radio frequency
regime of millimeter wave (mmWave) frequencies (60 GHz / 77 GHz). The wide
bandwidths (about 4 GHz) available at these frequencies provide the impressive
range resolution of mmWave radars of around 4cm ($\frac{c}{2B}$, where $c$ is
speed of light and $B$ is the bandwidth).

This means that the radar can distinguish objects in the world separated by 4
cm apart purely based on range processing. Since the range of an object
manifests in the time delay of its reflection, mmWave radars send “chirps” in
a waveform called Frequency Modulated Continuous Wave (FMCW) which converts
time delays to frequency shifts [28]. We perform range processing by simply
computing a 1D Fourier transform of the received reflections which converts
the frequency shifts back to time delays that is proportional to range (Range
$R=\frac{c\Delta T}{2}$, where $c$ is speed of light and $\Delta T$ is the
two-way time delay).

The time window that this Fourier transformed is performed over is referred to
as fast time, and is equal to the maximum theoretical number of range bins.

Figure 9: DART performs range-Doppler pre-processing on each of the antennas
in the mmWave radar using 2D FFTs and antenna beamforming gains. A key point
to note is the reduction in range-Doppler sidelobes using signal processing
windows prior to FFTs for effective DART training.

#### Doppler FFT

We next compute the Doppler velocities by processing a time series of FMCW
waveforms. Unlike range information which is observed as time delay of
reflections, Doppler velocities manifest as phase change of the reflected
signal when there is relative motion between radar and the world. As the radar
moves, an object changes phase as the range $R$ changes slightly over a time
window

$\displaystyle\exp\left(\frac{j2\pi 2R(t)}{\lambda}\right),$ (6)

where $\lambda$ is the wavelength corresponding to the FMCW waveform’s center
frequency. Because of the tiny wavelengths (4 mm) of mmWave signals, even tiny
shifts in range show up as significant phase changes. The rate of change of
these phase changes over the time series of FMCW waveforms provides the
Doppler velocity.

Similar to range processing, we perform another Fourier Transform, but across
a time sequence of range processed FMCW reflections. The time window (across
different chirps) which this Fourier transform is performed over is referred
to as slow time.

#### Impact of Doppler Resolution

Unlike azimuth or elevation resolution which is governed by the number of
antennas in a radar system, the Doppler resolution depends only on the radar
frame integration time. In theory, we can have arbitrarily fine Doppler
resolution with long frame times; however, this assumes constant radar
velocity over the entire integration time window, which is impractical.

Moreover, Doppler resolution can be simplified and mapped to azimuth
resolution (in 2D) as

$\displaystyle\Delta\theta=\frac{\Delta D\lambda}{2vsin\theta}$ (7)

Therefore, as $\Delta D$ becomes lower with longer frame integration time, the
azimuth resolution becomes better. In our case, the (theoretical) azimuth
resolution derived from Doppler is 0.85° for a head-on point at $\theta$=90°,
a radar speed of 0.5 m/s and a radar frame integration time of 256 ms — much
better than the best case of 15° that compact radars today are capable of [58,
76].

#### Hann Filtering

(a) Without Hann Filtering

(b) With Hann Filtering

Figure 10: Spectral leakage causes inter-bin interference, visible as
streaking in the horizontal and vertical directions (range-Doppler images are
cropped to a small window). Hann low-pass filtering softens window edges and
reduces spectral leakage significantly at the cost of reduced sharpness.

Since both the range FFT and Doppler FFT are taken over a finite-time window,
the hard edges of the time window induce high-frequency effects that result in
a phenomenon known as _spectral leakage_. This leakage is a mathematical
artifact that can be modeled as a $sinc()$ function and is clearly visible in
radar scans [61, 60]. While it would be possible to accurately reproduce such
artifacts in DART’s forward rendering model, this would require modeling
inter-column effects (breaking DART’s per-column rendering model) and greatly
increase the computational cost of our algorithm. Instead, we apply a Hann
window weighting to the raw range-Doppler frames before computing the FFT,
which slightly blurs the resulting images but greatly reduces the inter-bin
artifacts (Fig. 10).

#### Azimuth FFT

After applying the range and Doppler FFT, we also apply an Azimuth FFT across
8 azimuth bins (2 TX $\times$ 4 RX antennas). While this also results in a
sinc pattern, we model it explicitly in the form of an array factor (Sec. A.5)
applied in addition to the antenna gain on each of the resulting azimuth bins.

###  A.2 Reflectance and Transmittance Modeling

(a) Specularity

(b) Ghost Reflections

Figure 11: View-dependent reflectance and transmittance captures both
specularity and ghost reflections.

DART’s view-dependent transmittance-reflectance model is analagous to the
density-color representation of NeRFs, and enables modeling a range of
physical phenomena, such as:

  1. 1.

Diffuse reflections: This makes up for the majority of first order reflections
from non-metallic surfaces.

  2. 2.

Partial occlusions: Modeling transmittance allows objects behind surfaces such
as drywall to be visible while occluding objects behind opaque surfaces, e.g.
concrete.

  3. 3.

Specularity (Fig. 11(a)): This manifests as high reflectance in specific
directions, e.g. metallic surfaces.

  4. 4.

Ghost reflections (Fig. 11(b)): This occurs as a result of seeing an object
after multiple orders of reflections from other objects, e.g. seeing the ghost
reflection of a chair off of the ground when radar is looking at the ground.

###  A.3 Spherical Harmonics

Our usage of spherical harmonics provides view-dependence, while also making
interpretation of the resulting implicit map easier. In this section, we
describe how our usage of spherical harmonics contrasts with common practice
in visual Neural Radiance Fields and provides key advantages for our use case.

#### Spherical Harmonics Representation

Spherical harmonics are a spherical (orthonormal) basis function which is
commonly used to decompose functions defined on the unit sphere. Neural
Radiance Field models typically handle view dependence by projecting the
incidence angle of the incoming ray to between 9 and 25 (real) spherical
harmonic coefficients, which is used as an input to the neural network [74,
51], acting as essentially a positional embedding on the viewing angle. We
instead use spherical harmonic coefficients as an output similar to non-
neural-network based radiance field methods [83, 18].

We use spherical harmonic coefficients of degree 25. Let
$\bm{Y}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{25}$ be the function that maps
incoming incident angles to the degree-25 spherical harmonics. Then, our field
function output is given by

$\displaystyle\begin{split}\sigma(\bm{w})&=\bar{\sigma}\langle\bm{Y}(\bm{w}),\bm{c}/||\bm{c}||_{2}\rangle,\\\
\alpha(\bm{w})&=f\left(\bar{\alpha}\langle\bm{Y}(\bm{w}),\bm{c}/||\bm{c}||_{2}\rangle\right),\\\
\end{split}$ (8)

where
$\bar{\sigma}\in\mathbb{R},\bar{\alpha}\in\mathbb{R},\bm{c}\in\mathbb{R}^{25}$
are outputs of the neural network, $f$ is the activation function on $\alpha$
(Sec. A.4), and $\bm{w}$ is the incident angle.

Crucially, unlike visual NeRFs, we also must model transmittance $\alpha$
(which is analogous to the density in NeRF) as having view dependence. While
reflective surfaces in visual NeRFs are modeled as “screens” whose content
changes depending on the viewing angle, radars measure distance, which
requires reflective surfaces to instead be measured as “holes” which become
transparent depending on the viewing angle.

#### Interpretability

Since spherical harmonics are an orthonormal basis, we can analytically
evaluate the spherical $\mathcal{L}_{2}$ norm of our DART field at each point
without requiring numerical integration in order to sample a “mean”
reflectance and transmittance. For example:

$\displaystyle\begin{split}||\sigma||_{2}^{2}&=\int_{w}\left(\bar{\sigma}\langle\bm{Y}(\bm{w}),\bm{c}/(||\bm{c}||_{2})\rangle\
d\bm{w}\right)^{2}\\\
&=\frac{\bar{\sigma}^{2}}{||\bm{c}||_{2}^{2}}\int_{w}\bm{c}^{T}\bm{Y}(\bm{w})\bm{Y}(\bm{w})^{T}\bm{c}\
d\bm{w}\\\ &=\bar{\sigma}^{2}\end{split}$ (9)

since, by the orthonormality of spherical harmonics $\bm{Y}$,

$\displaystyle\int_{\bm{w}}\bm{Y}(\bm{w})\bm{Y}(\bm{w})^{T}\ d\bm{w}=\bm{I},$
(10)

and neural network output coefficients $\bm{c}$ are unrelated to $\bm{w}$
given a fixed point in space. If we ignore the activation function $f$, the
same applies to $\alpha$.

###  A.4 Transmittance Representation

Each ray in our rendering pipeline corresponds to one sampled reflectance
$\sigma$ from DART. As such, negative $\sigma$ are immediately penalized by
gradient descent, so an activation function on $\sigma$ is not necessary, and
can harm the numerical stability of DART (Sec. C.1).

On the other hand, since each transmittance value $\sigma$ affects all bins
behind it, allowing $\alpha>1$ (i.e. for points in space to transmit more
energy than they receive) can allow for a significant degree of overfitting.
At the same time, we must account for the fact that most points in space are
empty (so it should be “easy” for the model to represent $\alpha=1$), while
handling initialization instability and the possibility of “dead” gradients.

Finally, to further reduce the potential for overfitting, we only allow
$\alpha<1$ when $\sigma$ exceeds a set threshold (0.05 in our experiments). To
reduce the “swiss cheese” effect caused when this threshold is initialized to
a (relatively) large value, we also anneal this threshold in over 600 steps
(starting from -1, which in practice is always less than the initialized
$\sigma$, to 0.0 linearly in 100 steps, and from 0.0 to 0.05 linearly in 500
steps).

#### Existing Activation Functions

No existing activation functions satisfy our desired properties:

  * •

Sigmoid: this requires a $-\infty$ input to represent 0, and suffers from
vanishing gradients at either end.

  * •

Value clipping $\alpha=\text{clip}(\alpha^{\prime},0,1)$: clipping output
values to $[0,1]$ suffers greatly from “dead” regions in space, especially
where $\alpha$ is initialized at $\alpha^{\prime}<0$, and cannot recover.

  * •

Partial clipping $\alpha=\exp(\min(0,\alpha^{\prime}))$: clipping at only one
end suffers from the same “dead” region problem, though only at one side.

Indeed, any activation function on [0, 1] must have vanishing gradients of
some form. As such, we instead turn to a custom gradient estimator to create
an activation function which satisfies our desired properties.

#### Gradient Estimator

Using partial clipping $\alpha=\exp(\min(0,\alpha^{\prime}))$ as a starting
point, we specify our gradient estimator for $\min(0,\alpha^{\prime})$ as
follows:

  * •

$\alpha^{\prime}\leq 0$: gradients pass through, i.e.
$d/d\alpha^{\prime}=d/d\alpha$.

  * •

$\alpha^{\prime}>0,d/d\alpha^{\prime}\geq 0$: In this case,
$d/d\alpha^{\prime}>0$, indicating that the gradients are trying to push
$\alpha^{\prime}$ negative, even though $\min(0,\alpha)$ is “saturated”. As
such, we pass $d/d\alpha^{\prime}=d/d\alpha$ as well.

  * •

$\alpha^{\prime}>0,d/d\alpha^{\prime}<0$: In this case, the gradients are
trying to push $\alpha^{\prime}$ positive, even though it is already positive.
To prevent our parameter values from exploding, we set $d/d\alpha^{\prime}=0$.

###  A.5 Rendering Equation Derivation Details

DART’s rendering process is summarized by Eq. 5 at a high level; in this
section, we expand on the derivation of this equation, as well as specific
elements of our implementation of the rendering equation.

$\bm{w}$$\gamma$$\bm{v}$$\tilde{r}$$r$$\eta$ (a) Section of the Doppler cone

$x=0$$\tilde{r}$$h$$\bm{v}$ (b) Integration arc/circle along the p-q plane

$+x$$\bm{v}$$\gamma$$h$$\bm{v}_{x}$$||\bm{v}||_{2}$ (c) Projection of $\bm{v}$
to the $+x$ axis

Figure 12: Illustration of our integration arc equation derivation.

#### Field of view calculation

While radar rendering theoretically requires integrating spherically around
the radar, the gain to the rear of the radar is close to zero in practice888
Our radar uses a PCB-based antenna, which acts as a metal plane preventing the
radar from seeing backwards. . As such, we only integrate across the front of
the radar.

Our rendering equation begins by generating an orthonormal basis

$\left\\{p,q,\frac{\bm{v}}{||\bm{v}||_{2}}\right\\}$ (11)

for velocity vector $\bm{v}$. We select $p$ such that its direction is aligned
with the positive $x$ axis (which define to be the front of the sensor), i.e.

$p\propto\begin{bmatrix}1&0&0\end{bmatrix}^{T}-\frac{\bm{v}_{x}\bm{v}}{||\bm{v}||_{2}^{2}},$
(12)

and define $q$ accordingly.

Each range-Doppler bin is modeled as the intersection of a sphere with radius
$r$ and a cone $\langle\bm{v},\bm{w}\rangle=d$. When intersected with the
half-space corresponding with $x\geq 0$, this results in an arc or circle on
the p-q plane (or the empty set if degenerate). Since we define the sensor to
be facing in the $+x$ axis, and define $p$ to be aligned with $+x$, the valid
integration arc will always be $(-\psi,\psi)$ for angle $\psi$
counterclockwise from the $+p$ axis, again relative to the p-q plane.

We now calculate this angle $\psi$. We begin by calculating the distance
$\gamma$ to the center of the integration arc and its radius $\tilde{r}$ (Fig.
12(a)). Choosing an arbitrary direction $\bm{w}$ in our p-q-v basis, the
cosine of the angle $\eta$ between $\bm{v}$ and $\bm{w}$ yields

$\displaystyle\cos(\eta)=\frac{\gamma}{r}=\frac{\langle\bm{v},\bm{w}\rangle}{||\bm{v}||_{2}||\bm{w}||_{2}}\quad\Longrightarrow\quad\gamma=\frac{rd}{||\bm{v}||_{2}}$
(13)

using the definition of Doppler $d$ and orthonormality of $\bm{w}$. It follows
that

$\displaystyle\tilde{r}=\sqrt{r^{2}-\frac{r^{2}d^{2}}{||\bm{v}||_{2}}}=r\sqrt{1-\frac{d^{2}}{||\bm{v}||_{2}^{2}}}.$
(14)

Next, we calculate the length of the chord corresponding to the integration
arc $\pm\psi$. Let $h$ be the distance from the chord to $\bm{v}$ (Fig.
12(b)); $h$ can be specified relative to $\gamma$ and the angle $\theta$
between $\bm{v}$ and the $+x$ axis (Fig. 12(c)). It follows that

$\displaystyle
h=\gamma\tan(\theta)=\frac{rd}{||\bm{v}||_{2}}\left(\frac{\bm{v}_{x}}{\sqrt{||\bm{v}||_{2}^{2}-\bm{v}_{x}^{2}}}\right),$
(15)

which finally yields (Fig. 12(b))

$\displaystyle\psi$ $\displaystyle=\arccos\left(\frac{h}{\tilde{r}}\right).$
(16)

We also must account for three degenerate cases:

  * •

When $d>||\bm{v}||_{2}$, the Doppler bin exceeds the velocity (and Eq. 14
becomes imaginary).

  * •

When $h>\tilde{r}$, the entire doppler cone is pointing behind the radar (i.e.
the $x>0$ half-space no longer intersects with the cone; Eq. 16 is undefined).

  * •

When $h<-\tilde{r}$, the entire doppler cone is in front of the radar. In this
case, we instead have $\psi=\pi$.

#### Volume of integration region

$\bm{v}$$d_{j}+\Delta D$$d_{j}$$r_{i}$$r_{i}+\Delta R$$\tilde{r}$$A$$\eta$
Figure 13: Annular section $A$ for the revolved solid approximation of Eq. 5.
$\eta$ is the angle to the velocity vector $\bm{v}$.

Expanding on Eq. 4 more explicitly using our integration bound $\psi$, the
radar return $\bm{Y}(r_{i},d_{j},k)$ can be expressed as the circular integral

$\displaystyle\bm{Y}(r_{i},d_{j},k)\propto\tilde{r}\int\displaylimits_{-\psi\leq\phi\leq\psi}C(r,k,\bm{w}(\phi))\
d\phi$ (17)

for projection $\bm{w}$ to the integration arc.

While our rendering equation (Eq. 4) makes a “thin-shell” assumption, we must
also examine the effect of bin size in order to correctly determine which
constants to apply to our approximation. Let $\Delta R$ and $\Delta D$ be the
width of each range and Doppler bin. Instead of using a “point” approximation
of $C$, we instead derive the integral based on the revolved solid of an
annular section $A$ (Fig. 13). Using again the fact that
$\cos(\eta)=d/||\bm{v}||_{2}$, we arrive at the approximation

$\displaystyle dA$ $\displaystyle=r_{i}\
drd\theta=\frac{r_{i}}{||\bm{v}||_{2}\sqrt{1-\frac{d_{j}^{2}}{||\bm{v}||_{2}^{2}}}}\
drdd$ $\displaystyle=\frac{r_{i}^{2}}{||\bm{v}||_{2}\tilde{r}}\ drdd,$ (18)

which yields the net correction of $r_{i}^{2}/||\bm{v}||_{2}$ found in Eq. 5
when combined with Eq. 17.

#### Array factor and antenna gain

In our rendering process, we begin by applying the antenna gain for the radar
as a whole, which is specified by the AWR1843Boost datasheet [76]. After
performing the azimuth FFT, each azimuth bin (i.e. transmit-receive pair) then
acts as an independent beamformed radar with a narrower azimuth gain; as such,
we also apply the array factor to each azimuth bin [60].

###  A.6 Optimized Radar Rendering

Figure 14: Radar rendering dataflow.

Figure 14 shows our optimized rendering dataflow:

  1. 1.

We use each Doppler “column” across all antennas as a sample which has (range,
antenna) dimensions.

  2. 2.

For this Doppler, we sample a fixed (128) number of rays (i.e. from
$(-\psi,\psi)$, projecting to 3D space via the v-p-q basis); we trace each ray
(Eq. 3), sampling $\sigma$ and $\alpha$ once at each range bin for a total of
(Doppler, ray samples) times.

  3. 3.

For each antenna, we apply the corresponding gain and sum across the Doppler
samples (Eq. 5), yielding (range, antenna) outputs for this Doppler column.

In total, we sample the field (range, Doppler, ray samples) times per image
per epoch, which is comparable to the (width, height, range samples) times in
NeRF. With the dimensions of images in our datasets (128 range bins, 256
Doppler bins, 128 ray samples, and 8 antenna), this equates to only 16 field
samples per output value — less than most visual NeRF methods today such as
Nerfacto [74], which samples 48 points per pixel.

###  A.7 Training and other Hyperparameters

#### Implementation and Training

DART is implemented using JAX [8], which we tune for training on the RTX 4090
(24GB of VRAM). In our implementation, we perform the ray-tracing cumulative
product in the log-space as an exp-sum-log, learning the log-transmittance
instead. We also absorb any constants (including the square for transmittance)
into $\sigma$ and $\alpha$.

We trained our $(\sigma,\alpha)$ field function using stochastic gradient
descent with the Adam optimizer and an $l_{1}$ objective with a learning rate
of 0.01 for 3 epochs with a batch size of 1024 “Doppler columns” (i.e. all
range-Doppler-antenna bins with the same Doppler value in a given image; see
Sec. A.6). For each Doppler column, we sampled 128 rays along the arc defined
by that Doppler bin intersected with the half-sphere corresponding to the
front of the radar (since our radar is forward-facing).

#### Architecture Hyperparameters

For our base model, we use the Instant Neural Graphics Primitive [51] with the
following parameters:

  * •

Hash table size: $2^{20}$.

  * •

Number of features per hash table level: 2.

  * •

Hash table resolution scale factor: $2^{0.43}\approx 1.347$ (selected to avoid
“aliasing” from different hash table levels lining up to multiples of each
other).

  * •

Number of hash table levels: 12.

  * •

Grid resolution: 25cm (coarsest) to 0.94cm (finest).

  * •

Output MLP: 2 hidden layers of 64 and 32 units.

##  Appendix B Datasets and Evaluation

In this section, we provide additional details about our data collection
system (B.1), collected radar traces (B.2), baselines (B.3), and evaluation
methodology (B.4).

Dataset | Length | Training Time |  Description
---|---|---|---
Lab 1 | 4:55 | 8:43 |  A lab space with 5 boxes of varying material placed inside.
Lab 2 | 4:24 | 7:22 |
Office 1 | 10:37 | 15:14 |  An office space with cubicles and an enclosed meeting room.
Office 2 | 15:39 | 22:18 |
Yard | 7:38 | 12:13 |  A backyard and adjacent detached garage.
House 1 | 7:55 | 9:53 |  A early 20th century house with two above-ground floors. Traces include only the ground floor, and both the ground and 2nd floor.
House 2 | 9:48 | 12:54
Apartment 1 | 6:04 | 9:26 |  A late 20th century high rise one-bedroom apartment; traces include just the open-plan living area, and both the living area and bedroom.
Apartment 2 | 6:38 | 10:30
Rowhouse 1 | 7:41 | 10:48 |  Traces of varying lengths and room inclusions in a 2-bedroom apartment with an open-plan living space in a low-rise townhouse.
Rowhouse 2 | 7:37 | 10:41
Rowhouse 3 | 10:51 | 15:17 |
Table 2: Total trace length and training times (minutes) for our method on
each trace; the training time of our method is between 1-2$\times$ the data
collection time. While we captured multiple traces from each location, each
trace was collected independently.

###  B.1 Data Collection System

Figure 15: Our data collection system consists of a handheld rig (Fig. 6) and
a backpack which carries power equipment.

Our data collection rig (Fig. 15) is handheld and fully portable, and consists
of a TI AWR1843Boost mmWave radar with a DCA1000EVM capture card for recording
raw I/Q frames. The data collection rig also includes an Ouster OS0-64 64-beam
LIDAR and a Xsens MTi-3 IMU, which are used for our LIDAR-based radar
simulator baseline (Appendix B.3) as well as pose estimates using Cartographer
SLAM [24]. These components, along with the backpack-carried power and control
components, are shown in Fig. 16.

Figure 16: Components of our data collection system.

#### Radar Configuration

We configured the radar for inter-chirp period of 1ms with 512 samples per
chirp, and processed the data into range-Doppler images with a frame length of
256 chirps computed on a rolling window with a stride of 64 chirps. This gives
a maximum Doppler of $\pm 0.95$ m/s across 256 Doppler bins and a maximum
range of 21.6m across 512 range bins, which we truncated to the first 128
range bins (5.4m) after removing bins with negative range after radar
calibration.

#### Radar Processing

To avoid degenerate Doppler bins at low speeds (violating our thinness
assumption) and aliased speeds exceeding the maximum Doppler velocity, we
removed frames with speeds below 0.2m/s and above 0.95m/s from our training
and validation datasets. We also use all 8 azimuth antennas by configuring
transmitters to operate in a time-division multiplex mode.

#### Pose Processing

DART relies on accurate velocity estimates; however, SLAM systems such as
Cartographer are typically not designed to produce accurate or continuous
velocity estimates, especially at the frequency that we generate range-Doppler
images (64ms). This can lead to erratic velocity estimates, especially if loop
closure results in discontinuous pose estimates.

To calibrate the estimated speed, we implemented a speed estimation algorithm
which uses simple thresholding to detect the maximum velocity detected in a
range-Doppler image. Assuming a static scene, this then indicates the speed of
the radar. Using this estimate, we tuned a gaussian smoothing parameter
applied to the SLAM pose estimates to produce our final velocity estimates.

In our datasets, we observed that false or sudden relocalizations can also
cause erratic speed estimates. Thus, as a final check, we calculated the
acceleration of the final speed estimates, and excluded frames (and $\pm 15$
neighbors) if the acceleration exceeded $2.0\ \text{m}/\text{s}^{2}$. Datasets
that we collected that included a high frequency of likely SLAM system
failures were then either recollected or excluded.

###  B.2 Collected Traces

Our 12 collected radar novel view synthesis traces include 6 unique
environments across residential and commercial spaces of various construction.
Each trace consists of several passes through each scene with different paths,
velocities, and orientations, and is collected with a full pass through the
scene at the end to enable a convenient test set. We summarize these traces in
Table 2, and provide a visual map of the lidar scan and DART’s radar
tomography in Fig. 17.

(a) Lab 1, 2

(b) Yard

(c) Office 1, 2

(d) Apartment 1, 2

(e) House 1, 2 (1st floor)

(f) House 1, 2 (2nd floor)

(g) Rowhouse 1, 2, 3

Figure 17: Lidar maps (left), radar tomography maps (right), and sample
trajectories (red) on the 6 locations used to collect our novel radar view
synthesis dataset. Multiple traces were collected from each location, with
each trace being completely independent (and in some cases visiting different
rooms). Note that House contains two floors; the trajectory shown here is not
separated by floor.

#### Trajectory

Like many off-the-shelf radars, the TI AWR1843 radar is designed for
automotive applications, and has a relatively narrow vertical field of view
($\pm$20°, compared to $\pm$50° horizontally). As such, we took multiple
passes with varying attitude (e.g. horizontal, up, and down) in order to
ensure full 3D coverage.

#### Velocity

We found Cartographer to be most stable for our handheld data collection
platform when moving smoothly at $\approx$0.5m/s, and targeted this speed
during each trace. This also corresponds to the velocity range (0.2m/s –
0.95m/s) which we tuned our radar system for.

###  B.3 Baseline Details

In this section, we provide additional details about the implementation of our
baselines.

#### Lidar Scan-based Simulator

Using Cartographer [24], we created an occupancy grid of each scene with a 2cm
resolution. Each occupied cell was assigned a reflectance of 1 unit, and a
transmittance of 0 (i.e. fully opaque). We then used a ray-tracing simulator
(based on our rendering equation) to simulate radar images, with trilinear
interpolation applied to the grid during sampling. This simulator used the
same settings as DART (128 rays per Doppler bin, 128 range bins, simultaneous
range-antenna rendering).

#### CFAR Point Cloud Aggregation

We used the CFAR implementation provided by the Matlab Phased Array System
Toolbox [1] with a false alarm rate of $0.01$ to select discrete reflectors.
Each reflector was assigned an estimated azimuth angle using direction of
arrival estimation, projected to 3D space, and aggregated into a common point
cloud along with their estimated reflectance amplitude.

Since finite element modeling is impractical with tens of thousands of CFAR
points, we projected each reflector to a 5cm grid (which must be coarser than
the 2cm lidar-based simulator grid due to lower point cloud density), taking
the maximum gain for each point when multiple fall in the same grid cell.
Finally, we used the same simulation settings as our lidar baseline to
generate range-Doppler images. Note that since CFAR has no way of measuring
the transmittance of points in space, we set all grid cells to be fully
transparent (i.e. transmittance $\alpha=1$).

For our comparison of the CFAR point cloud with DART’s sampled tomography
images (Fig. 7), we qualitatively tuned the visualization to show sharp
outlines and the general geometry of the scene to the best of our ability:

  * •

The scatter plot shows points within $\pm 10$cm of the reference plane to
ensure we capture a sufficient density of strong reflectors.

  * •

To reduce clutter due to spurious points in the CFAR point cloud, we then
removed the 80% weakest reflectors, and scaled the color range in our
visualization so that points close to the 80% cutoff point have a color
similar to the plot background.

###  B.4 Evaluation Metrics

In order to evaluate DART and our baselines, we calculate the SSIM of each
valid999 Since our Doppler FFT “wraps” once we exceed our maximum doppler
velocity, we must exclude these frames. Our method also does not work on
stationary radars (as we note in our limitations), so frames with a velocity
near zero are also excluded.  range-Doppler-antenna frame (treating each
antenna as a different channel) after normalizing each image to a set [0, 1]
range. Then, to calculate error bars and evaluate the statistical significance
of our results, we estimate the effective sample size of each trace in order
to calculate standard errors and perform a paired z-test.

#### Image Scaling

Not all of our baselines are capable of producing accurately scaled output
images. Since SSIM requires the input dynamic range to be properly scaled, we
scale all images relative to the ground truth prior to SSIM calculation as
follows:

  1. 1.

We clip extreme values from the ground truth images ($0.1\%$ and $99.9\%$
percentiles), and normalize this range to $[0,1]$, which we use as the dynamic
range for SSIM calculation.

  2. 2.

We compute the $l_{2}$ “optimal scaling” multiplier between the predicted
$\hat{y}$ and actual $y^{*}$ images using scale factor

$\displaystyle\arg\min_{\xi}||y^{*}-\xi\hat{y}||_{2}^{2}=\frac{\hat{y}^{T}y^{*}}{\hat{y}^{T}\hat{y}}$
(19)

and apply this to $\hat{y}$. We then clip $\xi\hat{y}$ to the same thresholds
as $y^{*}$, and normalize to [0, 1].

  3. 3.

We then compute the SSIM, using a dynamic range of 1.

#### SSIM Calculation

Range-Doppler-antenna radar images are mostly sparse, and even contain large
regions where nonzero values cannot theoretically be observed (i.e. when
Doppler velocities exceed the current actual speed for a sensor in a
stationary scene). As such, a naive application of SSIM leads to misleadingly
high SSIM, since the SSIM between empty regions is exactly 1.0.

To correct for this, we exclude empty regions of our image, which are defined
as areas where the pixel sample mean is less than a set threshold
($\varepsilon=0.005$ in our dataset).

Figure 18: Sample tomographic maps of the reflectance ($\sigma$) compared to
using a simple grid with varying resolutions. Coarse grids result in blurry
tomographic images, while finer grids result in “holes” in walls and other
surfaces. By using an adaptive grid, DART is able to learn tomographic maps
which are reasonably sharp but also largely continuous. Figure 19: Sample
tomographic maps of the transmittance ($\alpha$) learned by DART (compared to
a Lidar occupancy grid), along with ablations on our representation for
$\alpha$ without the gradient estimator (Without Estimator), without clipping
$\alpha$ when $\sigma<0.05$ (Without Clipping), and without any activation
treatment (Without Activation).

#### Sample Size Correction

The standard error of the mean (SE) of a measurement $X$ is defined as
$\text{std}(X)/\sqrt{N}$. However, this assumes each of the $n$ samples are
independent and identically distributed.

Intuitively, sampling the same signal with greater frequency does not give any
additional insight into the general performance of a method. Thus, in time
series signals such as ours (i.e. SSIM over time), the number of samples must
be corrected to estimate the effective sample size, which we substitute for
$N$. This is given by [62]

$N_{\text{eff}}=\frac{N}{1+2\sum_{t=1}^{\infty}\rho_{t}}$ (20)

for autocorrelation $\rho_{t}$ (where $t$ is the delay). In our calculations,
we estimate the sum up to $t=N/2$, and clip $\rho_{t}$ estimates to positive
values101010Negative $\rho_{t}$ can occur for large delays $t$ due to broader
trends in our dataset, e.g. scanning different rooms or directions.

#### Paired z-Test

Since each method is evaluated on exactly the same test frames, we use a
paired z-test to evaluate the statistical significance of our results.
Specifically, we provide error bars for a one-sided p-value of 0.05 (i.e.
indicating 95% confidence that one method is better than another).

##  Appendix C Additional Results

###  C.1 Ablations

In addition to view dependence, we ran additional ablations showing the impact
of DART’s implicit representation and transmittance activation function.

#### Grid Size and Adaptive Grid

To demonstrate the benefit of an (implicit) adaptive grid, we trained a
version of DART using a fixed grid of 20cm, 10cm, and 4cm resolutions. DART’s
adaptive grid leads to more accurate radar simulations compared to fixed grids
(Table 3). Examining tomographic slices of the learned reflectance reveals the
reason: a coarse grid leads to a blurry image (and therefore blurry
synthesized radar range-Doppler images), while a fine grid leads to holes in
the learned reflectance (Fig. 18).

Method | Mean SSIM | Relative SSIM
---|---|---
DART |  0.636 $\pm$ 0.012 | —
No View Dep. |  0.614 $\pm$ 0.015 |  0.022 $\pm$ 0.005
20cm Grid |  0.591 $\pm$ 0.015 |  0.046 $\pm$ 0.004
10cm Grid |  0.580 $\pm$ 0.017 |  0.056 $\pm$ 0.006
4cm Grid |  0.564 $\pm$ 0.019 |  0.073 $\pm$ 0.008
Table 3: Mean and relative SSIM (to DART) of different grid sizes; we include
a baseline which does not use view dependence for comparison since a simple
reflectance and transmittance grid does not represent view dependence.

#### Transmittance Activation Function

While we did not run full ablations for each element of the design of DART’s
activation function (Sec. A.4), we trained ablations on an example scene (Fig.
19) for the activation function as a whole in addition to ablations on our
transmittance clipping procedure and gradient estimator.

Dataset | DART | Lidar | Nearest | CFAR
---|---|---|---|---
Lab 1 | 0.676 |  0.438 (0.238 $\pm$ 0.017) |  0.490 (0.186 $\pm$ 0.033) |  0.571 (0.105 $\pm$ 0.013)
Lab 2 | 0.680 |  0.471 (0.209 $\pm$ 0.012) |  0.468 (0.212 $\pm$ 0.045) |  0.566 (0.114 $\pm$ 0.014)
Office 1 | 0.642 |  0.461 (0.181 $\pm$ 0.014) |  0.482 (0.160 $\pm$ 0.026) |  0.555 (0.087 $\pm$ 0.013)
Office 2 | 0.669 |  0.478 (0.190 $\pm$ 0.025) |  0.449 (0.220 $\pm$ 0.028) |  0.562 (0.107 $\pm$ 0.019)
Rowhouse 1 | 0.622 |  0.458 (0.164 $\pm$ 0.023) |  0.471 (0.151 $\pm$ 0.031) |  0.535 (0.087 $\pm$ 0.018)
Rowhouse 2 | 0.619 |  0.458 (0.161 $\pm$ 0.018) |  0.450 (0.169 $\pm$ 0.036) |  0.529 (0.090 $\pm$ 0.025)
Rowhouse 3 | 0.620 |  0.462 (0.158 $\pm$ 0.017) |  0.480 (0.140 $\pm$ 0.026) |  0.540 (0.080 $\pm$ 0.022)
House 1 | 0.626 |  0.471 (0.155 $\pm$ 0.017) |  0.462 (0.164 $\pm$ 0.025) |  0.549 (0.077 $\pm$ 0.013)
House 2 | 0.629 |  0.474 (0.155 $\pm$ 0.016) |  0.459 (0.170 $\pm$ 0.026) |  0.524 (0.105 $\pm$ 0.016)
Yard | 0.615 |  0.457 (0.158 $\pm$ 0.020) |  0.466 (0.149 $\pm$ 0.033) |  0.538 (0.077 $\pm$ 0.022)
Apartment 1 | 0.620 |  0.464 (0.155 $\pm$ 0.015) |  0.459 (0.160 $\pm$ 0.032) |  0.533 (0.087 $\pm$ 0.015)
Apartment 2 | 0.618 |  0.460 (0.158 $\pm$ 0.022) |  0.479 (0.139 $\pm$ 0.030) |  0.537 (0.081 $\pm$ 0.018)
Overall | 0.636 | 0.463 (0.174 $\pm$ 0.013) | 0.468 (0.168 $\pm$ 0.012) | 0.545 (0.091 $\pm$ 0.006)
Table 4: The mean SSIM of each dataset for each method along with the
improvement of DART over each baseline and 95% (one-sided) confidence
intervals calculated using a z-Test with a effective sample size-adjusted
standard error. CFAR is the next best baseline in all cases, followed by Lidar
and Nearest which perform similarly depending on the similarlity between the
train and test sets. Figure 20: SSIM CDFs for each of the 12 traces (down and
to the right is better), along with the mean SSIM of Gaussian noise with
25/30/35db PSNR for reference. DART achieves much higher SSIMs than each
baseline on all traces in our dataset, which is roughly equivalent to gaussian
noise with a PSNR of 30db.

  * •

Without any activation function on $\alpha$, DART can greatly overfit the
training data (e.g. with $\alpha>1$). This can be seen in a very “cloudy”
transmittance map, along with a “trail” of low transmittance that appears to
follow the training trajectory. These artifacts are similar to “foggy”
artifacts that can be seen in many NeRF techniques.

  * •

Adding the activation function $\alpha=\exp(\min(0,\alpha^{\prime}))$ with our
gradient estimator results in a cleaner map; however, the trail along the
training trajectory still remains, since this only addresses some forms of
overfitting.

  * •

To prevent DART from overfitting by creating this low-transmittance trail, we
set $\alpha=1$ at approximately empty regions of space (i.e. $\sigma<0.05$).
However, without the gradient estimator, points with an unfavorable
initialization may “saturate” $\min(0,\alpha^{\prime})$ and become stuck at
$\alpha=1$. This can be seen in the pillar in Fig. 19, which is split into
multiple parts when the gradient estimator is not used.

###  C.2 SSIM by Dataset

Table 4 provides a full breakdown of the SSIM of DART and each baseline on
each view synthesis dataset; DART dominates each baseline on all datasets. 95%
(one-sided) confidence intervals for the difference to DART are calculated
using the procedure described in Sec. B.4. We also provide the CDF of the SSIM
for each dataset as a distributional view (Fig. 20).

###  C.3 Additional Tomography Examples

(a) Empty - Volume

(b) Occupied - Volume

(c) Empty - Section

(d) Occupied - Section

Figure 21: Volume renderings of the tent with and without a person inside,
created only using radar sensor readings from outside a tent.

Figure 22: Metal “anti-skate strips” (the bright dots) are clearly visible in
our tomography map along with a metal band on the outside of a raised section
(the bright outline of the shape).

#### Garden

Fig. 22 shows a zoomed view of the Garden scene shown in Fig. 7. Metal
features along the edge of the raised section are clearly visible in our
tomographic images.

#### Tent

Fig. 21 shows a 3D volume rendering of the Tent scene shown in Fig. 7 when
occupied and unoccupied. In addition to the tent poles, the tent’s occupant is
clearly visible; a rough approximation of the occupant’s seated pose can also
be seen after cutting along a section plane.

#### Mapping

DART can also be used to generate maps of spaces. Fig. 17 shows example
learned reflectance maps compared to Lidar occupancy grids; DART is able to
learn a coarse map of the room via its radar reflectance, indicating the
potential for future localization and mapping algorithms based on DART.