acronymindexonlyfirsttrue abbreviationindexonlyfirsttrue
abbreviationnohyperfirsttrue aurocAUROCArea Under the Receiver Operating
Characteristic Curve accuracyAccAccuracy bevBEVBird‘s Eye View cameraCarnegie
Robotics MultiSense S27 cnnCNNConvolutional Neural Network cvarCVaRConditional
Value at Risk demDEMDigital Elevation Model fovFoVField of View gnnGNNGraph
Neural Network gcnGCNGraph Convolutional Network hddHDDHazard Distance
Detection imuIMUInertial Measurement Unit irlIRLInverse Reinforcement Learning
knnKNNK-Nearest Neighbors lagrLAGRLearning Applied to Ground Vehicles
lidarLiDARLight Detection and Ranging ladarLADARLAser Detection And Ranging
lidarsensorVelodyne VLP-32C mlpMLPMulti-Layer Perceptron mpcMPCModel
Predictive Controller mseMSEMean Squared Error maeMAEMean Absolute Error
mppiMPPIModel Predictive Path Integral nirNIRNear-InfraRed wmseWMSEWeighted
Mean Squared Error wmaeWMAEWeighted Mean Absolute Error
oursRoadRunnerRoadRunner rbfRBFRadial Basis Function rmpRMPRiemannian Motion
Policies rosROSRobot Operating System ros1ROS 1Robot Operating System
rocROCReceiver Operating Characteristic rfRFRandom Forest radarRADARRAdio
Detection And Ranging rlRL Reinforcement Learning sdfSDFSigned Distance Field
slamSLAMSimultaneous Localization and Mapping svmSVMSupport Vector Machine
svcSVCSupport Vector Classifier stackX-RacerX-Racer workTraFoTerrainFormer
ugvUGVUnmanned Ground Vehicle wvnWVNWild Visual Navigation vitViTVision
Transformer vehiclePolaris RZR S4 1000 Turbo

# RoadRunner - Learning Traversability Estimation for Autonomous Off-road
Driving

Jonas Frey1,2

<EMAIL_ADDRESS>
&Shehryar Khattak1
<EMAIL_ADDRESS>
&Manthan Patel1,2
<EMAIL_ADDRESS>
Deegan Atha1
<EMAIL_ADDRESS>
&Julian Nubert1,2
<EMAIL_ADDRESS>
&Curtis Padgett1
<EMAIL_ADDRESS>
Marco Hutter2
<EMAIL_ADDRESS>
&Patrick Spieler1
<EMAIL_ADDRESS>

1Jet Propulsion Laboratory (JPL), California Institute of Technology(Caltech),
Pasadena, CA, United States of America2Swiss Federal Institute of Technology
(ETH Zurich), Robotic Systems Lab, Switzerland©2024. All rights reserved.

###### Abstract

Autonomous navigation at high speeds in off-road environments necessitates
robots to comprehensively understand their surroundings using onboard sensing
only. The extreme conditions posed by the off-road setting can cause degraded
camera image quality due to poor lighting and motion blur, as well as limited
sparse geometric information available from LiDAR sensing when driving at high
speeds. In this work, we present RoadRunner, a novel framework capable of
predicting terrain traversability and an elevation map directly from camera
and LiDAR sensor inputs. RoadRunner enables reliable autonomous navigation, by
fusing sensory information, handling of uncertainty, and generation of
contextually informed predictions about the geometry and traversability of the
terrain while operating at low latency. In contrast to existing methods
relying on classifying handcrafted semantic classes and using heuristics to
predict traversability costs, our method is trained end-to-end in a self-
supervised fashion. The RoadRunner network architecture builds upon popular
sensor fusion network architectures from the autonomous driving domain, which
embed LiDAR and camera information into a common Bird‘s Eye View perspective.
Training is enabled by utilizing an existing traversability estimation stack
to generate training data in hindsight in a scalable manner from real-world
off-road driving datasets. Furthermore, RoadRunner improves the system latency
by a factor of $\sim 4$, from $500\text{\,}\mathrm{m}\mathrm{s}$ to
$140\text{\,}\mathrm{m}\mathrm{s}$, while improving the accuracy for
traversability costs and elevation map predictions. We demonstrate the
effectiveness of RoadRunner in enabling safe and reliable off-road navigation
at high speeds in multiple real-world driving scenarios through unstructured
desert environments.

##  1 Introduction

Figure 1: Example deployment environments showcasing high-speed off-road
navigation.

Animals, like the greater roadrunner, are capable of traversing complex off-
road terrains at an impressive running speed of
$24\text{\,}\mathrm{k}\mathrm{m}\mathrm{/}\mathrm{h}$ to
$32\text{\,}\mathrm{k}\mathrm{m}\mathrm{/}\mathrm{h}$ [Maxon, 2005]. This
capability makes the roadrunner a benchmark in terms of perception
capabilities required for autonomous ground vehicles to advance toward
enabling high-speed operations in unstructured off-road environments. One of
the key components to facilitate safety while driving at high speeds is the
assessment of where a robot can navigate safely. This assessment is known as
_traversability estimation_ , where the objective is to estimate the
affordance or risk the robot must take when navigating the perceived terrain.
The affordance may depend on the robot’s hardware, control system, terrain
geometry, physical terrain properties, and other factors.

Complex and unstructured off-road environments pose a variety of obstacles and
other potential risks, which need to be correctly asses by an autonomous
ground vehicle. In addition, short reaction times are required at high speeds
and potential hazards must be identified at long distances to guarantee
safety. Moreover, only partial observations of the environment are available,
given that onboard sensors like LiDARs and cameras can only provide
information at a limited update rate, and only perceive a restricted field of
view with increasing intrinsic sparsity at longer distances. Additionally,
when considering driving off-road, robots cannot rely on high-definition map
information, given that the robot may be deployed for the first time in the
environment (space exploration) or the environment might have changed
significantly due to seasonal variations. Furthermore, extreme conditions such
as dust, dirt, fog, and rain are common in an off-road setting and can lead to
degraded perception modalities, as shown in the case of LiDAR sensors
[Stanislas et al., 2021]. As a result, the traversability estimation system
has to cope with all of these challenges to allow for navigation in an off-
road setting at high speeds.

In comparison, self-driving cars operate at higher maximum speeds and in more
dynamic environments. Despite this, the terrain traversability estimation
remains simple as cars primarily drive on paved roads. Therefore, the
classification of scene semantics is mostly sufficient to conclude the
traversability. As a result, most works focus on identifying semantic classes
like lanes, cars, and traffic signs from images [Philion and Fidler, 2020] or
LiDAR data [Lang et al., 2019]. Despite the difference in the use case, self-
driving cars are equipped with a similar sensing setting, and our work is
inspired by the recent advances in multi-modal network architectures that fuse
different sensory information into a common perspective [Philion and Fidler,
2020, Liu et al., 2023].

Historically, the first progress towards autonomous navigation in unstructured
off-road environments was achieved throughout the DEMO III [Shoemaker and
Bornstein, 1998], PerceptOR [Kelly et al., 2006], and [Kim et al., 2006,
Hadsell et al., 2009] programs, as well as the DARPA Grand Challenge. As part
of these programs, multiple works pioneered traversability estimation from
camera and LiDAR data and applied learning-based approaches, as well as, non-
learning-based approaches, which rely on heuristics to asses the terrain
traversability. More recently, using simulation-based approaches [Guzzi et
al., 2020, Frey et al., 2022], which can evaluate the terrain with respect to
the specific robot’s capabilities, and self-supervised approaches, which make
use of real-world deployment data [Frey et al., 2023] have gained significant
popularity, for estimating traversability in an off-road setting.

To further enhance the off-road traversability estimations, we present : a
multi-camera multi-LiDAR learning architecture able to predict terrain
traversability and elevation with low latency directly from sensor inputs. Our
architecture leverages point cloud and pre-trained image segmentation
backbones and fuses information into a unified representation. A predicts a
robot-centric traversability and elevation map from the unified feature
embedding. The training data is generated in a self-supervised manner using
hindsight to determine the traversability and the geometry of the terrain
without any human labeling or explicit identification of semantic classes
during run-time. The concept of hindsight refers to the fact that offline
processing with future and past sensor measurements can be used to create more
reliable estimates compared to the online deployment of the same algorithm. To
generate the training data for we use NASA Jet Propulsion Laboratory offroad
autonomy research stack () (Sec. 3.3), a sophisticated off-road driving stack
that includes mapping and traversability estimation, which leverages geometry
and semantics with carefully crafted and field-proven heuristics. These
heuristics encode the vehicle-specific capabilities. While is not the main
contribution of this paper, we firstly provide an overview of all relevant
components to this work within Sec. 3.3. provides a framework for using
hindsight and end-to-end learning to overcome the limitations inherent to a
non-learned traversability software stack. In addition, by predicting
traversability and elevation simultaneously, functions as a single perception
and mapping module, providing all perceptual information necessary for
downstream path planning. Throughout this work, all experiments are conducted
using our customized . This side-by-side off-road vehicle functions as a
testbed to enhance off-road autonomy and can be seen in Fig. 1.

The key contributions of our work are:

  * •

A novel network architecture that can simultaneously predict traversability
costs and elevation map information directly from multi-LiDAR and multi-camera
data at low latency.

  * •

A general framework to generate pseudo-ground truth elevation maps and
traversability costs using temporal data aggregation in hindsight to train the
network in a self-supervised manner.

  * •

We present an overview of NASA Jet Propulsion Laboratory’s off-road autonomy
research stack , one of the most advanced off-road autonomy software stacks.

  * •

An exhaustive evaluation of the proposed architecture on real-world field test
data and comparisons to existing network architectures as well as an ablation
study.

outperforms the elevation mapping and traversability estimation of , which was
used to generate training data, by leveraging visual and geometric data. In
addition, it is capable of predicting missing elevation and traversability
information based on the learned context. The same holds for traversability
detection, where obstacles can be detected at a longer range. We demonstrated
that our proposed RoadRunner architecture, compared to the X-Racer stack,
improves traversability cost estimation by $52.3\text{\,}\%$ in MSE and
$36.0\text{\,}\%$ in MAE for elevation map estimation while reducing the
perception-to-traversability latency during inference by a factor of $\sim 4$
compared to the existing software stack.

##  2 Related Work

In this section, we provide an exhaustive overview of traversability
estimation methods, focusing on off-road driving, traversability from
semantics, self-supervision, and other approaches. In addition, we provide an
overview of semantic segmentation methods operating in a BEV representation.

###  2.1 Traversability Estimation for Off-Road Driving

Terrain traversability is dependent upon both the terrain’s geometry and its
physical properties like stiffness, friction, or granularity. Several other
factors influence the traversability, which include the specific robotic
system in use, the applied control strategy [Frey et al., 2023], and the
robot’s state during its interaction with the terrain [Fan et al., 2021]. For
example, different robots (_hardware and control dependence_) can surmount
distinct obstacles when driving at varying speeds (_state dependence_). In
this work, we simplify this multi-variant view of traversability following
[Fan et al., 2021] and model the traversability as the affordance/risk
required to overcome a given terrain as a probability distribution. We
marginalize the robot-state dependency and target a specific robotic system
and control strategy. To make the overall traversability more interpretable,
we employ the metric. The metric, which is a coherent risk metric being
monotonic, subadditive, homogeneous, and translational invariant, allows us to
emphasize different parts of the risk distribution associated with a given
terrain [Majumdar and Pavone, 2017].

[Pomerleau, 1993] pioneered learning for autonomous driving by training a to
predict online the steering command from a low-resolution camera image by
learning from demonstration for on-road driving end-to-end. The first steps
towards off-road autonomy were achieved in the DEMO III project [Shoemaker and
Bornstein, 1998] followed by the Perceptor program [Kelly et al., 2006], in
which off-road capable were retrofitted with a variety of sensors, including
cameras and LiDARs for cross-country navigation. The developed perception
system at the time was primarily based on heuristics and, therefore failed to
generalize to the complex off-road domain. As part of this program [Manduchi
et al., 2005] classified terrain into a discrete set of classes from color
images and used stereo depth to detect obstacles, highlighting the importance
of using different sensor modalities.

The program [Jackel et al., 2006] aimed to improve the perception system
developed within the Perceptor program using machine learning techniques. For
this, [Muller et al., 2005] expanded on the idea of predicting the steering
command end-to-end from images [Pomerleau, 1993], but instead of an used a in
an off-road setting. For the network to learn correct driving behavior, it
implicitly learned obstacle avoidance, and thereby implicitly traversability.
[Hadsell et al., 2007] used reliable short-range geometric measurements to
supervise the training of a simple classifier to predict the traversability
from image data in a self-supervised manner and proposed the concept of
spatial label propagation. [Kim et al., 2006] instead clustered the
environment and assigned positive and negative labels through interaction to
determine the traversable areas. [Konolige et al., 2009] explicitly identified
the ground plane from stereo depth and classified paths based on image
segmentation and geometric information by fitting a model from a single image
before each deployment.

As part of the DARPA Grand Challenge [Behringer et al., 2004], [Thrun et al.,
2006] proposed a probabilistic terrain analysis for high-speed desert driving
taking into account LiDAR measurements which are used to assess the obstacles
based on height difference in combination with an adaptive vision approach. In
contrast, [Urmson et al., 2006] fully relied on geometry captured by LiDAR and
. Both teams showcased long-term autonomy in a desert scenario, within a
narrow scope and perfectly pre-planned routes under GPS guidance. Our overall
system does not require precise GPS guidance and can drive in more complex
open-field environments at high speeds.

####  2.1.1 Traversability from Semantics

Modern deep learning methods excel in semantic scene understanding from image-
[Tan and Le, 2019, Viswanath et al., 2021, Maturana et al., 2017, Roth et al.,
2023] and point cloud data [Shaban et al., 2022]. Commonly, identified
semantic classes are fused into a map representation [Erni et al., 2023,
Maturana et al., 2017] and then associated with a traversability score
[Maturana et al., 2017] using heuristics about the traversability properties
of the given semantic class. Notably, datasets like RUGD [Wigness et al.,
2019], Rellis-3D [Jiang et al., 2021] or Freiburg Forest [Valada et al., 2016]
provide high-quality annotated off-road semantics, however, are limited in
size and diversity, hindering generalization to diverse off-road scenarios.
Moreover, the reliance on manual labeling and heuristic-based mapping from
semantics to traversability inherently limits the performance of the
traversability estimation.

Within an off-road setting, [Bradley et al., 2015] perform a per-voxel terrain
density classification from LiDAR and camera images using a classifier on a
diverse legged robot dataset. [Maturana et al., 2017] perform semantic
segmentation of images and project the labels into a 2.5D semantic elevation
map for a . [Schilling et al., 2017] utilize learned semantics segmentation
features and heuristic features from LiDAR data to predict safe, risky, and
obstacle regions. [Viswanath et al., 2021] predict semantics in image space
using a while [Shaban et al., 2022] predict semantics in the perspective using
LiDAR. [Shaban et al., 2022] employ a sparse and allow to feed-forward latent
features from previous predictions in combination with memory units.

Rovers used in space exploration face the same perception challenges as off-
road driving, where the accurate identification of terrain hazards is pivotal
to mission success [Ono et al., 2015]. As a result, the identification of
semantic terrain classes and physical interaction has been exhaustively
studied. [Rothrock et al., 2016] employed a to classify Martian terrain and
predict wheel slip of the Mars Perseverance rover. [Swan et al., 2021] curated
a comprehensive Mars terrain semantic segmentation dataset. Other works focus
on improving semantic segmentation by reducing the number of training samples,
given the challenges associated with collecting training data in a space
exploration scenario [Zhang et al., 2022, Goh et al., 2022]. [Endo et al.,
2023] assess risk-aware traversability costs by fusing semantic terrain
classification and a slip model in a probabilistic manner.

Most relevant to our work is TerrainNet [Meng et al., 2023], which follows
[Triebel et al., 2006] by modeling the environment by a ground and ceiling
layer, where each layer contains its associated semantics. The geometry and
semantics are predicted by a neural network taking multiple RGB and depth
camera images as input. Features from the camera are accumulated in a
representation by predicting the corresponding location of the feature in
image space from stereo depth.

####  2.1.2 Traversability from Self-Supervision

Traversability estimation approaches based on scene semantics typically
require expensive annotated data. Methods operating in a self-supervised
manner aim to overcome this limitation by generating a training signal without
relying on manual annotation. Instead they exploit information from other
sensor modalities [Brooks and Iagnemma, 2012, Otsu et al., 2016, Castro et
al., 2023, Seo et al., 2023, Higa et al., 2019, Meng et al., 2023, Zürn et
al., 2021, Sathyamoorthy et al., 2022], or the interaction of the robot with
the environment [Richter and Roy, 2017, Seo et al., 2022, Frey et al., 2023,
Ahtiainen et al., 2017, Gasparino et al., 2022, Cai et al., 2022, Xue et al.,
2023b, Sathyamoorthy et al., 2022, Cai et al., 2023, Jung et al., 2023]. The
generated supervision signal allows training a model that predicts a look-
ahead estimate of the terrain, all without requiring the robot to be near to
or interact with the terrain.

[Brooks and Iagnemma, 2012] train an image classifier to predict terrain
classes of the Mars-analogue environment identified by the proprioception.
[Otsu et al., 2016] use co-and self-training to improve two classifiers to
predict terrain classes from images and proprioception for space exploration.
[Ahtiainen et al., 2017] learn traversability from LiDAR measurements by
accumulating information in a map and training a to classify traversability
supervised by positive labels the robot visited and negative labels based on
heuristics. [Higa et al., 2019] directly estimate driving energy for rovers
from images using the measured energy consumption. [Castro et al., 2023] adapt
the proprioception-based pseudo labels by measuring vibration data, which is
used as the network input to predict a traversability grid map from a
colorized elevation map.

[Richter and Roy, 2017] proposed to use anomaly detection to predict safe
image regions for indoor navigation with a wheeled robot. Multiple other works
integrated the concept of anomaly detection and tools available in evidential
deep learning to learn from real-world data without manual labeling [Frey et
al., 2023, Schmid et al., 2022, Seo et al., 2022, Wellhausen et al., 2020, Cai
et al., 2023]. Contrastive learning has shown promising results in learning
expressive representations that can be used for traversability estimation in a
self-supervised manner [Seo et al., 2023, Xue et al., 2023b, Jung et al.,
2023].

[Chen et al., 2023] generate pseudo labels from heuristic-based LiDAR
analysis, while [Frey et al., 2023] use a velocity-tracking criterion to
assess terrain traversability to train a vision model. Both methods allow for
training the traversability prediction network online during deployment. In
WayFAST [Gasparino et al., 2022], the terrain traversability is approximated
by the tracking error from a model predictive controller and used to predict
traversability from image data. Similarly, [Cai et al., 2022] predict worst-
case expected cost and traction from a semantic elevation map in addition to
predicting confidence values using density estimation. The TerrainNet
framework [Meng et al., 2023] proposes using co-labeling of semantics by a
human and a pre-trained semantic network, while the geometry is supervised by
LiDAR measurements.

Our method uses self-supervision, but instead of predicting semantics or
relying on heuristic-based analysis, we use the sophisticated field-tested
traversability estimation software stack , which allows us to account for a
variety of risks in a probabilistic manner. This allows to obtain a nearly
dense supervision signal and account for uncertainty in the signal itself
during learning, therefore overcoming problems associated with sparse labels
when only considering the terrain vehicle interaction. Additionally, no
specific data collection strategy is needed, and any real-world driving data
can be used for training.

####  2.1.3 Other Traversability Estimation Approaches

Model-based approaches for assessing geometric traversability have attracted
continuous research, particularly in the context of wheeled robots, where the
wheel-to-terrain interactions can be more easily defined compared to more
complex legged systems [Wellhausen and Hutter, 2023]. These approaches
demonstrate good generalization, specifically when the underlying assumptions
like the rigidity of the environment, hold. For this, recent works rely on
analysis of point cloud data [Fan et al., 2021, Cao et al., 2022, Xue et al.,
2023a], mesh data [Hudson et al., 2022], or elevation maps [Bouman et al.,
2020, Fan et al., 2021, Wellhausen and Hutter, 2023]. On the other hand, data-
driven approaches use simulation to collect data in a trial-and-error fashion
to estimate traversability for wheeled [Guzzi et al., 2020] or legged robots
[Frey et al., 2022].

Complementary, learned dynamics models from real-world data are capable of
accounting for unmodeled effects in simulation and are used for planning.
[Kahn et al., 2021] and [Kim et al., 2022] learn a forward dynamics model,
where [Kahn et al., 2021] predict future events and states of the robot from
real-world data, and [Kim et al., 2022] use a simulation to collect data.
[Xiao et al., 2021] learn the inverse kinodynamics model of a wheeled robot
from proprioception and [Karnan et al., 2022] expand this concept by
conditioning the model on visual data of future terrain patches to anticipate
terrain interaction. Therefore, these approaches implicitly learn
traversability information about the environment for the deployed robotic
system.

aims to learn the underlying reward structure guiding an agent’s behavior
[Wulfmeier et al., 2015]. By using demonstration data and the formulation,
[Wulfmeier et al., 2015] train a neural network to predict cost maps for
simple game-like environments. [Triest et al., 2023] translated this concept
to off-road autonomous driving and predicted vehicle-centric traversability
grid maps.

[Bouman et al., 2020] assess geometric traversability based on multi-fidelity
terrain maps and superposition of individual traversability maps. This work
was expanded in [Fan et al., 2021], where traversability risk is modeled as a
distribution allowing to account for different geometric risks while taking
into account uncertainty. [Dixit et al., 2023] added further improvements for
subterranean exploration and incorporated semantic risks assessed solely based
on processing geometry. Our software stack, used to generate the pseudo ground
truth, adapts this perspective of traversability estimation and improves
multiple components of the traversability assessment tailored to off-road
driving. We explain the in-depth adaptation made to facilitate high-speed off-
road navigation in Sec. 3.3.

###  2.2 Learning Semantics in BEV Representation

The mobility of ground robots is typically constrained by the terrain
geometry. Under this reduced mobility assumption, it is sufficient to model
the environment from a perspective, in which the geometry, semantics, or
traversability costs can be identified. One challenge is to associate features
from a camera image into the perspective. If no depth measurements are
available, the inverse perspective mapping (IPM) algorithm under the flat
ground assumption can project image features to the perspective. [Roddick et
al., 2019] showed that projecting multi-scale image features into 3D using an
orthographic projection improves 3D object detection performance for
autonomous driving. This method projects features uniformly along a camera ray
in 3D and consecutively pools the features along the z-dimension. Similarly,
Lift Splat Shoot [Philion and Fidler, 2020] extends this concept to multiple
cameras and shows promissing results for the task of semantic segmentation in
the perspective. Instead of densely distributing features along the ray, they
predict a categorical distribution over discrete depth bins for each pixel
indicating the contribution of each feature in the intermediate 3D
representation. Pseudo-LiDAR approaches such as BEV-Seg  [Ng et al., 2020]
predict explicit dense depth from the monocular image, which is used to
establish a one-to-one correspondence between image pixels and grid cells.
Simple-BEV [Harley et al., 2023] and M2BEV [Xie et al., 2022] show that, while
the feature lifting problem is well-researched, a uniform lifting approach
[Roddick et al., 2019] can result in competitive performance and more gain can
be achieved using RADAR sensor data [Harley et al., 2023] or a larger image
backbone [Xie et al., 2022]. BEVFusion [Liu et al., 2023] considers the
problem of fusing LiDAR data and camera images into a unified representation.
Camera features are lifted into 3D following [Philion and Fidler, 2020], with
a significantly more efficient implementation which addresses concerns by
[Harley et al., 2023, Xie et al., 2022]. Depth features are concatenated in
the representation with image features, while no LiDAR measurements are
leveraged for the feature lifting. LAPTNet-FPN [Diaz-Zapata et al., 2023] use
the LiDAR measurements to lift camera features into 3D, by directly
associating the sparse LiDAR measurements to the downsampled image features.
Alternative attention-based lifting approaches have been investigated in [Saha
et al., 2022].

Figure 2: Overview of the Architecture. The network is trained on real-world
driving data (I), which is first processed by to generate an elevation and
traversability assessment based on the currently available sensory data (II).
Pseudo ground truth labels are generated by fusing information from past and
future measurements to obtain reliable traversability and elevation estimates
(III). The network is trained offline based on the large dataset (IV) and can
be deployed online for improved performance and reduced latency (V).

##  3 Method

Figure 3: Definition of frames and illustration of hindsight self-supervision.
At the first timestep, $t_{1}$, the reference frames are defined. The gravity-
aligned base frame $\mathtt{B}_{g}{}$ is fixed to the vehicle (position and
yaw), with roll and pitch being gravity-aligned. The magenta region on the
ground illustrates the Reliable Perception Range per timestamp, where
sufficient sensory information is available such that can correctly predict
elevation and traversability. When the vehicle approaches the tree at
timestamp $t_{2}$, can correctly predict that the area underneath the canopy
is traversable. Similarly, in timestamp $t_{3}$, the cactus can be identified
as untraversable. While requires exhaustive geometric information, which is
only available in the proximity of the vehicle, more precise traversability
and elevation maps, the so-called pseudo ground truth, can be generated as a
learning objective for when taking into account future and past sensory
information. For example, in timestamp $t_{2}$, can learn to correctly
identify the cactus as a hazard from the image data, even with insufficient
geometric information available.

###  3.1 Problem Statement

Our network predicts vehicle-centric elevation and traversability cost maps
using multi-camera and multi-LiDAR inputs. In the following sections, we
describe the notation (Sec. 3.2) used throughout the paper and an overview of
stack (Sec. 3.3) used for generating training data in hindsight (Sec. 3.4).
Next, we explain the network architecture (Sec. 3.5), followed by the
implementation details (Sec. 3.6). The overview of the RoadRunner architecture
is presented in Fig. 2.

###  3.2 Notation

We use the following coordinate frames: world frame $\mathtt{W}{}$, base frame
$\mathtt{B}{}$, gravity-compensated base frame $\mathtt{B}_{g}{}$, camera
frames $\leftindex{{}^{k}}\mathtt{C}$, and LiDAR frames
$\leftindex{{}^{l}}\mathtt{L}$, with $k\in\\{1,\dots,C\\}$ and
$l\in\\{1,\dots,L\\}$. Here, $C$ and $L$ denote the number of cameras and
LiDAR sensors, respectively. The world frame $\mathtt{W}{}$ in the following
coincides with the typical odometry frame definition. Its relative
transformation to the base frame $\mathtt{B}_{g}{}$ is assumed to be locally
consistent and smooth. The frames are visualized in Fig. 3. Tensors are
denoted by bold capital letters $\mathbf{F}_{A\times B\times
C}\in\mathbb{R}^{A\times B\times C}$, where the lower right subscript
indicates the respective dimensions. Camera images of height $H$ and width $W$
are denoted as $\leftindex{{}^{k}}\mathbf{I}_{H\times W\times 3}^{t}$, where
$k$ indicates the respective camera index and $t$ is the timestamp. The
intrinsic calibration of the camera $k$ is given by
$\leftindex{{}^{k}}\mathbf{K}_{3\times 3}$. Similarly, the point cloud
measured by the LiDAR $l$ is denoted as
$\leftindex{{}^{l}}\mathbf{P}_{N_{l}\times 3}^{t}$, where $N_{l}$ is the
number of points and $t$ is the respective timestamp.

Figure 4: Overview of . We use our LiDAR Inertial Odometry system [Fakoorian
et al., 2023] in combination with GraphMSF [Nubert et al., 2022] to obtain
smooth, accurate, and high-frequency odometry estimates. SegFormer [Xie et
al., 2021] is used to predict the semantic classes from each camera image
(a–d), which are then projected onto the undistorted and filtered point cloud,
yielding the Semantics Points (e). The Semantic Points are further accumulated
in a vehicle-centric voxel map using our voxel mapper based on [Overbye and
Saripalli, 2022], resulting in a Semantic Voxel Map. A rule-based aggregation
method allows converting the Semantic Voxel Map further into a Semantic
Elevation Map where the aggregation is tailored to off-road driving and the
physical characteristics of our vehicle (f). Subsequently, both geometric and
semantic risks are assessed based on the Semantic Elevation Map, resulting in
a Traversability Map [Fan et al., 2021] (g). For downstream trajectory
optimization, both the traversability and the elevation are provided to a
Model Predictive Path Integral (MPPI) Planner [Williams et al., 2017], which
employs a learned vehicle dynamics model [Gibson et al., 2023] to compute the
final trajectory for a given goal location (h).

We assume the point cloud scans are motion-compensated and merged for all $L$
LiDAR sensors, resulting in a single big point cloud $\mathbf{P}_{N\times
3}^{t}$, with $N=\sum_{l=1}^{L}N_{l}$. A grid map with dimension $H_{g}\times
W_{g}$ is denoted as $\mathbf{G}_{H_{g}\times W_{g}}(x,y)$ and is expressed
within the gravity-aligned base frame $\mathtt{B}_{g}{}$. The frame
$\mathtt{B}_{g}{}$ defines the yaw orientation and center of the grid map,
located at ($H_{g}/2$,$W_{g}/2$). Therefore we refer to the grid maps being
position and orientation vehicle-centric.

###  3.3 X-Racer Architecture

####  3.3.1 Overview

leverages the highly capable mapping and traversability assessment stack for
learning. allows the to autonomously drive off-road at up to
$15\text{\,}\mathrm{k}\mathrm{m}\mathrm{/}\mathrm{h}$ within desert
environments. The task at hand, which guided the development of , is to reach
a set of goal positions using onboard sensing without reliance on precise
prior global mapping in a GPS-denied environment. In the following, we will
summarize the most important parts of stack relevant to the traversability
assessment used for local planning. In addition, we will elaborate on the
inherent limitations that are addressed by with respect to the existing stack.
A simplified overview of all components can be seen in Fig. 4.

####  3.3.2 Odometry and Semantics

We assume precise vehicle poses are provided by our LiDAR-inertial odometry
estimation and localization module [Fakoorian et al., 2021] in combination
with a graph-based filtering and fusion algorithm [Nubert et al., 2022]. Four
RGB images captured by the RGB-D cameras are individually processed by a
SegFormer vision transformer model [Strudel et al., 2021] adapted and fine-
tuned using [Contributors, 2020], to predict multi-class terrain semantics,
namely: trail, ground, vegetation, rock, and sky. The segmentation network is
trained in a supervised fashion using manually annotated images from real-
world deployments and photo-realistic renderings using the Duality Robotics
Falcon Pro simulator [Duality Robotics, Inc., 2023].

####  3.3.3 Mapping

The point clouds sensed by three Velodyne VLP-32C LiDAR sensors are filtered
for vehicle self-correspondences using a predefined mask, and dust points are
removed using an intensity return value threshold. The filtered point cloud is
associated with predicted semantics in the image domain based on proximity
using the camera intrinsics and camera-to-LiDAR extrinsic while correctly
accounting for the ego-motion. The resulting semantic point cloud is
probabilistically fused into a 3D vehicle-centric occupancy map covering a
volume of
$$100\text{\,}\mathrm{m}$\times$100\text{\,}\mathrm{m}$\times$30\text{\,}\mathrm{m}$$,
which is defined as the micro range, at an isotropic voxel resolution of
$20\text{\,}\mathrm{c}\mathrm{m}$. The 3D voxel mapper is a significantly
enhanced version of [Overbye and Saripalli, 2022], which has been extended to
allow for low-latency robot-centric mapping, voxel-decay, fusion of semantic
segmentation, and a wide range of further features while being able to run in
real-time on a single GPU. From the 3D occupancy map an elevation map is
extracted, where the elevation is the height of the lowest occupied voxel
along the $z$-direction. A heuristic-based approach is used to fuse the
semantic probabilities along the $z$-direction into the elevation map, which
takes into account the distance of the voxel to the elevation map surface and
its semantic class probabilities. E.g., all voxels up to a height of
$2.5\text{\,}\mathrm{m}$ above the elevation map labeled as a tree with a
likelihood of over $50\text{\,}\%$ are fused into the elevation map cell. On
the other hand, voxels above $2.5\text{\,}\mathrm{m}$ labeled as tree can be
excluded to allow driving underneath the canopy.

####  3.3.4 Traversability Assessment

The traversability assessment is based on a previously developed planner for
subterranean and unstructured environments [Fan et al., 2021]. It handles
perceptual uncertainty based on the notation of certainty and risk. We provide
an overview of the different risks taken into account to obtain the wheel
risk, which describes the risk of an individual wheel interacting with the
terrain. First, a reliability map is computed, which is proportional to the
density of the available geometric information. For example, if insufficient
geometric observations about a specific area within the elevation map are
available, the estimates for these cells are considered to be less reliable.
Further, the elevation map is interpolated and smoothed such that the risk for
slope, curvature, roughness, and positive and negative obstacles can be
extracted. Additional semantic risks for hard and soft obstacles are computed.
The final wheel risk, which describes the risk associated with the terrain-
wheel interaction, is given by the Conditional Value At Risk () of the
combined assessed risk distributions. The full software stack is tuned in
simulation and on real-world data. In the following, we refer to the CVaR of
the wheel risk as the traversability, where a value of $0$ indicates safe to
traverse, whereas $1$ is unsafe.

####  3.3.5 Trajectory Planning

The resulting wheel risk traversability map of size
$$100\text{\,}\mathrm{m}$\times$100\text{\,}\mathrm{m}$$ at a resolution of
$20\text{\,}\mathrm{c}\mathrm{m}$ is provided to the planner [Williams et al.,
2018] responsible for obstacle avoidance at high speed. The planner operates
with a $5\text{\,}\mathrm{s}$ planning horizon and is evaluated at a frequency
of $30\text{\,}\mathrm{H}\mathrm{z}$. The planner has access to the dynamics
model of the vehicle and minimizes the time to the goal while trading off soft
risks and smoothness of the path. The planned trajectories have to obey
rollover, collision, and velocity limitations strictly.

####  3.3.6 Limitations

While the current software stack is capable of safely guiding the vehicle
within desert environments and individual components can be tuned, multiple
limitations exist, which are addressed by . Geometric perception of the
environment is the dominant driver of : semantic information is only
incorporated if geometric measurements are available. Therefore, a risk
identified only in the image domain cannot be taken into account. This problem
is specifically pronounced when operating at higher speed, given the limited
update rate of the LiDAR ($10\text{\,}\mathrm{H}\mathrm{z}$) and its sparse
nature at long distances. Additionally, while the set of considered semantic
classes is carefully selected, it inherently limits the flexibility of the
downstream traversability assessment to account for semantic risks. Further,
while the sequential structure of the stack allows for interpretability and
adjustment of individual components, it leads to a high latency between
perception and traversability estimation of over
$500\text{\,}\mathrm{m}\mathrm{s}$. This high latency limits the maximum
speed, which is determined by the emergency braking distance and hazard
detection range. Lastly, the current software stack cannot make use of (past)
experience, in contrast to a rally driver who can anticipate the road
progression when turning into a narrow corner and thus take more sophisticated
actions under uncertainty. stack does not include these generative abilities
to imagine the environment, which we argue are important for driving at high
speeds.

###  3.4 RoadRunner Hindsight Traversability Generation

Using the software stack we generate training data for in a self-supervised
manner using hindsight. In particular, the traversability and elevation are
improved by leveraging privileged future measurements, not available to during
run-time, which reduces the uncertainty of the environment and allows for
reliable predictions beyond the current reliable perception range. An
illustrative example is presented in Fig. 3. This is implemented by
accumulating predictions obtained by over time, requiring no modifications of
major components such as the mapping backend. To reduce errors due to
localization drift, all transformations are performed in the locally
consistent odometry frame. Only measurements collected within a time window of
$60\text{\,}\mathrm{s}$ are accumulated for each grid map.

$G\\_list\;[\;]\leftarrow$ grid maps within 60s window;

$p\leftarrow$ reference $SE(2)$ position;

$default\leftarrow$ default value ground truth map;

Fusion $\leftarrow$ fusion function;

// Sort with increasing timestamp

Sort(_$G\\_list$_);

// Initalize the return grid map

$G\\_gt\leftarrow\text{torch.full}(G\\_list[0].\text{shape},default)$;

foreach  _$G$ in $G\\_list$_ do

// Transform the grid map to G_gt frame

$G\\_transformed\leftarrow\textnormal{{Transform}}(G,p)$;

// Fuse information into G_gt

$G\\_gt\leftarrow\textnormal{{Fusion}}(G\\_gt,G\\_transformed)$;

end foreach

return _$G\\_gt$_ ;

Algorithm 1 Compute Hindsight Ground Truth

Algorithm 1 describes the fusion scheme used for the pseudo ground truth
reliability, elevation, and traversability map generation. In the following,
we will designate this as the ground truth for simplicity, omitting the term
”pseudo”. It is important to note, however, that the presented ground truth
does not reflect the actual reliability, elevation, or traversability, but can
be used as a good approximation.

The default value for each map is initialized to be unreliable (value $0$),
the maximum height ($+\infty$), and traversable (value $0$), respectively.
Each predicted grid map within the given time window is first translated and
rotated to the reference position. The fusion function is given by the cell-
wise minimum for the elevation, the cell-wise maximum for the reliability, and
the temporally latest measurement in addition to a confidence threshold for
the traversability. The traversability fusion is deliberately not chosen to be
a simple maximum function, which one could argue facilitates safety.
Experimentally, the maximum function results in an over-pessimistic
traversability assessment given that under uncertainty areas may be predicted
as untraversable by and with more information available labeled as safe to
traverse.

###  3.5 RoadRunner Network

####  3.5.1 Preprocessing and Feature Extraction

The network builds upon the open-source available implementation of Lift Splat
Shoot [Philion and Fidler, 2020], PointPillars [Lang et al., 2019] and
BEVFusion [Liu et al., 2023], which combines and improves the prior two
methods. The information flow and network architecture is shown in Fig. 5. The
images obtained from the hardware-synchronized cameras are first downsampled
to a resolution of $512\times 384$ and then normalized. The image timestamp
$t$ determines the reference frame of the vehicle-centric grid map. The latest
measurement of each LiDAR is motion compensated, transformed to the base
reference frame $\mathtt{B}_{g}{}$, and merged to a single merged point cloud
$\mathbf{P}_{N\times 3}^{t}$. Features are extracted from each camera image
using an EfficientNet-B0 [Tan and Le, 2019], which downsamples the input size
by a factor of $16$, yielding a corresponding feature map
$\mathbf{F}_{32\times 24\times 328}^{k}$ for each image $k$. The merged point
cloud is passed through a PointPillars backbone with the same grid
configuration as the target grid map of
$$100\text{\,}\mathrm{m}$\times$100\text{\,}\mathrm{m}$$ at a resolution of
$20\text{\,}\mathrm{c}\mathrm{m}$ and results in a point cloud feature map
$\mathbf{F}_{H_{G}/2\times W_{G}/2\times 96}^{pcd}$. The previous elevation
estimate $\mathbf{G}_{H_{G}\times H_{W}\times 1}^{ele,t-1}$ is translated and
rotated to align with the new vehicle position. In addition, the elevation is
normalized by first scaling the elevation by a factor of $0.05$ and then
clipping to values between $-1$ and $1$, resulting in
$\mathbf{\bar{G}}_{H_{G}\times H_{W}\times 1}^{ele,t}$. This scaling limits
the range of elevation to $\pm$20\text{\,}\mathrm{m}$$, which is adequate for
our off-road scenario (refer to the normalized elevation histogram in Fig. 6).

Figure 5: Overview of the Network. The input to the network consists of 4 RGB
images and a filtered and merged point cloud from 3 LiDAR sensors, in addition
to the past elevation prediction of the previous timestamp $t-1$. The network
uses the Lift Splat Shoot [Philion and Fidler, 2020], and the PointPillar
[Lang et al., 2019] architecture to encode the visual and geometric
information, respectively. The elevation information is normalized and
transformed to the current position, which is then used to predict the
traversability and elevation based on separate decoder networks.

####  3.5.2 Lifting

The lifting of a feature from the camera image plane into the 3D space is
achieved by using the pinhole camera model, following [Philion and Fidler,
2020]. Along the respective ray of the pixel, a set of equally spaced discrete
points in 3D are distributed, resulting in a set of feature points. The weight
of a feature point is given by a categorical distribution predicted along the
ray based on the image feature embedding of the respective pixel using a
$1\times 1$ convolutional layer. The feature points are rasterized into grid
map cells with the same dimensions and resolution as the target grid map. The
weighted feature points coinciding with a grid cell are accumulated by a
channel-wise summation using the improved splat implementation proposed in
[Liu et al., 2023], resulting in the image feature map
$\mathbf{F}_{H_{G}\times W_{G}\times 328}^{img}$.

Next, we emphasize why we express the grid map in the gravity-aligned base
frame $\mathtt{B}_{g}$. This choice offers a primary advantage of accumulating
information along the gravity-aligned $z$-direction when performing the splat
operation. In particular, when considering driving up a steep incline, the
information about vertical obstacles such as a tree trunks or vertical poles
is accumulated within a single grid cell, instead of washed out over multiple
cells. If no feature point is associated with a grid cell, it is set to $0$.

####  3.5.3 Multi-Modal Fusion

First, the point cloud feature map is upsampled to a resolution of
$H_{G}\times W_{G}\times 96$. The three feature maps, namely
$\mathbf{F}_{H_{G}\times W_{G}\times 328}^{img}$, $\mathbf{F}_{H_{G}\times
W_{G}\times 96}^{pcd}$, and $\mathbf{\bar{G}}_{H_{G}\times H_{W}\times
1}^{ele,t}$ are concatenated resulting in the multi-modal feature map of size
$\mathbf{F}_{H_{G}\times W_{G}\times 425}^{fused}$. Supplying the nearest
neighbor interpolated elevation map to the network facilitates the
preservation of elevation information for regions that are unobservable within
the currently processed merged LiDAR point cloud. Two separate decoder
networks are trained for elevation and traversability estimation, predicting
$\mathbf{\hat{G}}_{H_{G}\times W_{G}}^{ele}$ and
$\mathbf{\hat{G}}_{H_{G}\times W_{G}}^{trav}$, respectively. Each decoder
consists of a set of convolutional, batch norm, and ReLU activation layers
following a residual architecture. The elevation decoder predicts the residual
elevation to the motion-compensated elevation. To enforce a smooth
traversability output, a median filter of window size 5 $\times$ 5 is applied
to the predicted elevation.

Figure 6: Normalization weights for the traversability and normalized
elevation. The left axis provides the frequency per bin. The right axis
(magenta dotted line) provides the weight. The wheel risk is strongly
imbalanced, with most areas being fully traversable (value of 0). The
elevation is normally distributed with a slight negative mean, given that the
base frame $\mathtt{B}_{g}$ is above the ground.

####  3.5.4 RoadRunner Loss

To train , we use the for the traversability and elevation. The loss is
calculated only for the valid grid cells containing a supervision signal:

$\mathcal{L}_{\mathrm{WMSE}}^{\text{layer}}=\frac{1}{\lvert\mathbf{G}\rvert}\sum_{x,y\,\in\,\mathbf{G}}w^{\text{layer}}(\mathbf{G}(x,y)-\mathbf{\hat{G}}(x,y))^{2}.$
(1)

Here, the weighting factor $w^{\text{layer}}$ is calculated based on the value
of the target grid cell. For this, we compute the normalized frequency
$\mathcal{F}_{N_{\text{bins}}}$ of elevation and traversability scores over
the training dataset in $N_{\text{bins}}$ bins. The clipped inverse normalized
frequency gives the final weight:

$w^{\text{layer}}=\text{clip}((\mathcal{\bar{F}}^{\text{layer}}*N_{\text{bins}})^{-1},0.2,5).$
(2)

Multiplying by the number of bins results in a weight of $1$ for all bins with
a uniform probability. The relation between normalized frequency and resulting
weight is visualized in Fig. 6. The final optimization objective is given by:

$\mathcal{L}_{\mathrm{FINAL}}=\mathcal{L}_{\mathrm{WMSE}}^{\mathtt{trav}}+\mathcal{L}_{\mathrm{WMSE}}^{\mathtt{elev}}.$
(3)

###  3.6 Implementation Details

We use the AdamW optimizer [Loshchilov and Hutter, 2019] with a learning rate
of $10^{-3}$, OneCycleLearningRate-schedule [Smith and Topin, 2017], and
optimize for a total of $16,000$ steps. The network is trained on an Nvidia
RTX3090 GPU with a batch size of $6$. For the EfficientNet-B0 we use pre-
trained weights from ImageNet, and freeze the respective layers. For the
lifting operation, we generated a total of $353,280$ feature points given by
the downsampled height of $24$, downsampled width of $32$, $4$ cameras, and
$N_{D}=230$ feature points, where we selected the closest distance being
$4\text{\,}\mathrm{m}$ and the furthest distance being $50\text{\,}\mathrm{m}$
with a spacing of $0.2\text{\,}\mathrm{m}$ analog to the grid resolution. In
total, accounts for $24.0\text{\,}\mathrm{M}$ parameters, with
$5.3\text{\,}\mathrm{M}$ EfficientNet-B0, $4.4\text{\,}\mathrm{M}$
PointPillars, $4.6\text{\,}\mathrm{M}$ Lift Splat Shoot, and the remaining
$9.8\text{\,}\mathrm{M}$ parameters correspond to the decoder networks.

##  4 Experiments

Figure 7: Dataset Overview: Magenta trajectories are used for training and
validation. Blue trajectories are used for testing. The dataset comprises
various driving scenarios illustrated in Fig. 7, including open-field (e),
dirt road (d), and gravel road traversal (c), as well as operation within
confined canyon/ditch settings (a) and forested environments (a, b, h).
Notably, the forest environment posed unique challenges, necessitating driving
beneath the canopy (b, j). Varying lighting conditions including shadows from
trees (a, b, j), substantially impeded vision-based obstacle detection.
Additionally, varying levels of vegetation (e, f) and the presence of fallen
trees (g, j) within tall grass poses additional challenges for obstacle
detection.

###  4.1 Dataset - Scenarios

To train our model, we collect a total of $16.5\text{\,}\mathrm{k}\mathrm{m}$
of off-road driving data at Paso Robles, CA, USA. The dataset consists of
eight individual trajectories resulting in a total of $21,158$ samples. We
apply our hindsight traversability generation (Sec. 3.4) on all collected
trajectories. The trajectories are split into training, validation, and
testing data (see. Fig. 7). The test data is collected in a different location
compared to the training and validation data. For training and validation,
four trajectories are collected and roughly the first $80\%$ are used for
training while the last $20\%$ are used for validation. Tab. 1 summarizes the
key statistics about the dataset.

Table 1: Dataset Overview - Statistics recorded in Paso Robles

Split | Duration | Distance | # Traj | # Samples
---|---|---|---|---
Train | $3389\text{\,}\mathrm{s}$ | $8618\text{\,}\mathrm{m}$ | $4^{+}$ | 10626
Val | $644\text{\,}\mathrm{s}$ | $2236\text{\,}\mathrm{m}$ | $4^{-}$ | 2658
Test | $1700\text{\,}\mathrm{s}$ | $5727\text{\,}\mathrm{m}$ | 4 | 7874

  * $+/-$

Indicates the first $80\%$ (+) or last $20\%$ (-) of the same trajectory
respectively.

Each recorded trajectory is converted into a set of samples by selecting
camera images based on a minimal traveled distance criterion between samples
of $20\text{\,}\mathrm{c}\mathrm{m}$. In practice, this maintains the
diversity of the data while reducing storage and computing requirements during
training. Each sample consists of the merged LiDAR scan obtained by all
LiDARs, $4$ camera images, elevation map, and the ground truth elevation and
ground truth traversability map.

###  4.2 Metrics

We assess the elevation mapping performance by reporting the , following [Meng
et al., 2023]:

$\mathcal{L}_{\mathrm{MAE}}=\frac{1}{\lvert\mathbf{G}\rvert}\sum_{x,y\,\in\,\mathbf{G}}\lvert\mathbf{G}(x,y)-\mathbf{\hat{G}}(x,y)\rvert,$
(4)

where $x,y\,\in\,\mathbf{G}$ consists of all valid indices of target grid map
layer $\mathbf{G}$, with $\mathbf{\hat{G}}$ denoting the estimated grid map
layer. For the traversability estimation performance we follow [Frey et al.,
2023] and evaluate the :

$\mathcal{L}_{\mathrm{MSE}}=\frac{1}{\lvert\mathbf{G}\rvert}\sum_{x,y\,\in\,\mathbf{G}}(\mathbf{G}(x,y)-\mathbf{\hat{G}}(x,y))^{2}.$
(5)

In addition, we introduce the reliability-weighted performance measures. The
and () are given as:

$\mathcal{L}_{\mathrm{WMAE}}=\frac{1}{\lvert\mathbf{G}\rvert}\sum_{x,y\,\in\,\mathbf{G}}\mathbf{C}(x,y)\lvert\mathbf{G}(x,y)-\mathbf{\hat{G}}(x,y)\rvert,$
(6)
$\mathcal{L}_{\mathrm{WMSE}}=\frac{1}{\lvert\mathbf{G}\rvert}\sum_{x,y\,\in\,\mathbf{G}}\mathbf{C}(x,y)(\mathbf{G}(x,y)-\mathbf{\hat{G}}(x,y))^{2},$
(7)

where $\mathbf{C}(x,y)$ denotes the reliability per grid cell and is directly
estimated by (Sec. 3.3.4).

While the previously introduced metrics (, ) evaluate traversability as a
continuous-valued function, it is also crucial to classify hazardous regions.
This is necessary as downstream planning modules explicitly account for those
regions. The distance at which hazard regions can be identified reliably
bounds the maximum safe velocity of the vehicle, considering both the
emergency braking distance and latency. Similarly to , we introduce a fatal
risk value threshold. This threshold enables the conversion of the continuous
predicted traversability risk map to a binary classification output. We use
the same threshold as applied to and utilize it for both the ground truth
traversability map and the predicted traversability by . This approach allows
us to evaluate Precision, Recall, and F1-Score in terms of hazard detection
(HD).

###  4.3 Traversability Estimation Performance

The primary focus is to investigate the performance of in comparison to . We
use the nearest neighbor interpolation to fill the missing traversability
estimates of . outperforms in terms of traversability prediction in MSE
($0.0113$ to $0.0216$), Precision ($0.412$ to $0.217$), while slightly
underperforming in Recall for the selected fatal risk value (cf. Tab. 2).

Table 2: Evaluation of Traversability Estimation Performance | MSE$\downarrow$ |  | WMSE$\downarrow$ |  | HD-Recall$\uparrow$ |  | HD-Precision$\uparrow$
---|---|---|---|---|---|---|---
Method |  | (ours) |  |  | (ours) |  |  | (ours) |  |  | (ours)
Train | 0.0339 | 0.0066 |  | 0.0339 | 0.0066 |  | 0.292 | 0.896 |  | 0.344 | 0.777
Val | 0.0413 | 0.0295 |  | 0.0413 | 0.0295 |  | 0.305 | 0.364 |  | 0.399 | 0.512
Test | 0.0216 | 0.0113 |  | 0.0216 | 0.0113 |  | 0.320 | 0.302 |  | 0.217 | 0.412

(a) Traversability Prediction in MSE.

(b) Elevation Prediction in MAE.

Figure 8: Qualitative comparison of and per grid cell for traversability
prediction (a) and elevation prediction (b) on average across all samples
within the test dataset. The vehicle is located at the center of the map
indicated by the $\triangle$, and the front of the vehicle faces towards the
top of the grid map. achieves an overall lower elevation and traversability
prediction error. performs specifically worse in front of the vehicle, where
insufficient geometric data has been observed.

The MSE per grid cell in the vehicle-centric frame averaged across all samples
within the test dataset is depicted in Fig. 8. As the vehicle is moving in a
straight line, it corresponds to an upward movement in the grid map. Given
that the test data exhibits a bias towards driving straight, we expect a
better performance in the lower part of the grid map, given that on average
more information is accumulated behind the vehicle. The MSE distribution of ,
directly confirms and motivates the rationale behind our proposed hindsight
ground truth generation. This stems from the observation that lower MSE is
achieved in the lower part of the grid map, which has already been passed by
the vehicle. Additional information can be gathered and accumulated for those
regions within the semantic voxel map, resulting in an improved traversability
estimate. On the other hand, in front of the vehicle, insufficient geometric
information is available and the nearest neighbor interpolation leads to a
higher error.

Although does not outperform in the vicinity of the vehicle, it exhibits
overall lower errors, particularly at longer ranges in front of the vehicle.
These results prompt our hypothesis that can more effectively make use of the
information provided within the three front-facing cameras and therefore
estimate traversability better than .

The hazard detection outcomes are depicted in Fig. 9. Evidently, outperforms
at a range of over $25\text{\,}\mathrm{m}$ in terms of accuracy. It achieves
overall better precision and F1-score.

Figure 9: Hazard Detection Error Metrics of and on the training (1st row),
validation (2nd row) and test (3th row) dataset. We report the Precision (1st
column), Recall (2nd column), F1-Score (3rd column) and the MSE (4th column),
plotted over the distance. We report the error evaluated for each 1-meter bin
independently as opposed to the cumulative error up to the distance.
outperforms across all dataset above a range of $20\text{\,}\mathrm{m}$ in
terms of MSE. In the proximity to the vehicle, , as expected, corresponds
nearly perfectly to the ground truth resulting in perfect predictions.

###  4.4 Elevation Mapping Performance

We report the elevation map performance between and in Tab. 3. Before
interpreting the results, we would like to recapitulate the generation of the
elevation map procedure by . fuses geometric information in a structured
approach into a volumetric map, which is subsequently used in Sec. 3.4 to
compute the ground truth elevation. Consequently, when comparing the against
the ground truth, only a substantial performance increase can be expected for
unobserved and therefore interpolated regions where no geometric measurements
are available for the elevation mapping performance.

Table 3: Evaluation of Elevation Mapping Performance | MAE [m] $\downarrow$ |  | WMAE [m] $\downarrow$
---|---|---|---
|  | (ours) |  |  | (ours)
Train | 0.460 | 0.131 |  | 0.460 | 0.131
Val | 0.475 | 0.323 |  | 0.475 | 0.324
Test | 0.329 | 0.242 |  | 0.329 | 0.242

Overall outperforms . Following the traversability analysis, we present the
cell-wise error maps in MAE Fig. 8 (b). Within both elevation error maps,
small artifacts in the form of a unit circle are visible, where a jump in
performance decrease can be observed. This artifact can be attributed to the
fact that while predicts in the vehicle-centric frame, the ground truth is
generated using a fixed heading. This artifact is further discussed within
Sec. 4.7.

To further understand when improves the elevation mapping performance, we
split cells into observed and unobserved. We defined the observability of a
cell based on whether geometric LiDAR measurements have been registered for
the specific cell or not. As shown in Fig. 10, strictly outperforms across all
datasets and for observed and unobserved areas. In areas that are unobserved,
which tend to be either occluded or far away from the vehicle, our method
generally has a higher . For observed regions, the interpolation of fails to
accurately reconstruct the elevation, while shows better performance. However,
our method encounters challenges due to domain shift when dealing with the
unobserved regions in the validation and test environments, given that the is
significantly higher for validation and testing than for the training regions
(compare Fig. 10 center plot performance difference between X-Racer and
RoadRunner). In contrary performs consistently across datasets. We
hypothesize, that this is due to the relatively small dataset providing
evidence that by increasing the data augmentation or the dataset size even
better performance can be achieved.

In summary, we hypothesize that using visual and geometric cues end-to-end can
benefit traversability prediction, while accurate prediction of a continuous
elevation is significantly harder, specifically when no geometric information
is available.

Figure 10:  Comparison of Elevation Mapping Performance across Training,
Validation, and Test Datasets. Observed indicates that LiDAR measurement is
available for the predicted cell, while Unobserved denotes that no LiDAR
measurement is available. Combined is the evaluation of both Unobserved and
Observed regions. Note that there are significantly more observed than
unobserved regions therefore, Combined does not correspond to the arithmetic
mean.

###  4.5 Comparison and Ablation Study

To understand the importance of individual modalities we compare our work to
PointPillar [Lang et al., 2019] (geometry-only) and Lift Splat Shoot [Philion
and Fidler, 2020] (vision-only). While a comparison to [Meng et al., 2023]
would be desirable, the different choice of sensor modalities (stereo-depth
compared to ours using LiDAR) and the closed source code renders the
comparison infeasible.

In Tab. 4 we report the the MSE and WMSE for the traversability estimation and
the MAE and WMAE for the elevation mapping performance. The superior
performance of aligns with the findings of [Meng et al., 2023, Liu et al.,
2023] that multi-modal sensor fusion results in superior performance.

In addition, we ablate the individual design choices of . No Weighting
indicates removing the balancing for the traversability and elevation loss and
using the MSE and MAE, respectively, without weighting. Removing the balancing
slightly increases the traversability prediction performance in terms of MSE
(0.0113 $\rightarrow$ 0.0112), but leads to an overall worse F1 score (0.349
$\rightarrow$ 0.345). By the increased weighting of high-risk areas, the
network predicts a more conservative estimate, which is desirable. Not Frozen
allows to update the weights of the EfficientNet-B0 backbone. Updating all
weights during training leads to better performance on the training dataset,
but overall hinders generalization to new environments leading to a worse test
score, across all metrics. This validates our design choice of freezing the
EfficientNet-B0 backbone. Common-Decoder changes the dual decoder network
architecture to a shared decoder network for the traversability and elevation
mapping task. Given that the performance for the elevation mapping task is
degraded (0.242 $\rightarrow$ 0.249), we decided on a separate decoder
architecture. We hypothesize that the features learned to accurately predict
traversability and elevation are not complementary.

Table 4: Comparison and Ablation | Elevation |  | Risk MSE
---|---|---|---
| MAE$\downarrow$ | WMAE$\downarrow$ |  | MSE$\downarrow$ | Recall$\uparrow$ | Precision$\uparrow$ | F1$\uparrow$
(ours) | 0.242 | 0.241 |  | 0.0113 | 0.302 | 0.412 | 0.349
PointPillar | 0.645 | 0.643 |  | 0.0118 | 0.274 | 0.367 | 0.314
LSS | 0.643 | 0.640 |  | 0.0118 | 0.281 | 0.370 | 0.319
No Weighting | 0.241 | 0.241 |  | 0.0112 | 0.297 | 0.413 | 0.345
Not Frozen | 0.247 | 0.246 |  | 0.0116 | 0.263 | 0.406 | 0.319
Ele-Classification | 0.247 | 0.246 |  | 0.0113 | 0.302 | 0.415 | 0.349
Common Decoder | 0.249 | 0.248 |  | 0.0108 | 0.348 | 0.410 | 0.376

###  4.6 Deployment

Our is equipped with a Threadripper 3990x CPU (64 Core 2.9/4.3 GHz), 256 GB
RAM, and 4xGeForce RTX 3080 GPUs. We integrate using ROS and use a single GPU
for inference. To achieve low latency and overcome the slow data serialization
of the Python ROS wrapper, we write a C++ node using Python bindings to the
network implemented in PyTorch.

This allows to run with a latency of 131.85 $\pm$
$2.5\text{\,}\mathrm{m}\mathrm{s}$, compared to the modular with an average
latency of over $500\text{\,}\mathrm{m}\mathrm{s}$. The timings are reported
over 700 samples with an average of $36,766$ points per sample.

Figure 11: Two Deployment Example Predictions: (a) and (b). The top two rows
of each of the two examples visualize the four onboard camera images (left,
front, right, rear). In the top row, the predicted traversability and
elevation are projected onto the images. The bottom row shows the respective
traversability, reliability, and elevation grid map, as well as the rasterized
merged point cloud. The traversability risk maps are shown for , , and the
generated ground truth. Blue indicates risk-free traversable and red
untraversable. The reliability is shown for the ground truth. The elevation
maps are presented for , the nearest neighbor interpolated elevation map
predicted by , and the ground truth elevation. For all maps, black indicates
that no prediction is available. (a) Yellow box in Risk - Ground Truth
indicates a hazard in front of the robot. No hazard is visible in the front-
facing camera. The Risk - Ground Truth is corrupted by a car driving behind
the vehicle visible in the rear camera image. (b) Red box highlights
successful elevation prediction, where learns to correctly predict the small
canyon in front of the vehicle, which is not fully observable from the LiDAR
point cloud. Further details and interpretation of each example can be found
within Sec. 4.6.

Fig. 11 and Fig. 12 provide example outputs of illustrating successful
predictions as well as the limitations. Each output includes visualizations of
the traversability risk predicted by , , side-by-side to the (hindsight-)
generated ground truth risk. Additionally, we present visualizations for our
predicted elevation, the feed-forwarded elevation (nearest neighbor
interpolated elevation map predicted by ), and the ground truth elevation map.
The currently merged point cloud is displayed, along with an assessment of the
reliability of the ground truth. In the top row, the predicted traversability
and elevation by are projected onto the camera images up to a range of roughly
$30\text{\,}\mathrm{m}$. This comprehensive presentation offers a detailed
insight into the capabilities and performance of our in terms of
traversability risk and elevation predictions.

Fig. 12 (a) illustrates a driving scenario on top of a hill. was unable to
accumulate sufficient geometric information to correctly identify the trees
ahead (highlighted by the yellow box). In contrast, provides a more accurate
traversability estimation, correctly identifying the trees ahead as high-risk
regions. In Fig. 12 (b), a similar scenario is presented: correctly identifies
tree trunks as untraversable compared to . Specifically, the two trees to the
direct left of the vehicle are identified as higher-risk regions (highlighted
by the yellow box). In Fig. 12 (c), the predicted traversability risk map
correctly identifies the most prominent risk in the scene – the two trees in
the direct surroundings. On the other hand, the trees behind the vehicle are
not identified correctly in the traversability risk map. Concerning elevation,
while accurately predicts a smooth increase in elevation to the left of the
vehicle, it is unable to estimate the correct numerical values and
underestimates the slope (highlighted by the red box). This discrepancy
becomes apparent when comparing the elevation predicted by with the ground
truth map. We hypothesize that this issue arises due to the limited
information available from the images and point cloud, which only covers the
bottom part of the valley, therefore can only retrieve to hallucinate the
elevation for the unobserved areas.

Figure 12: Three Deployment Example Predictions: (a), (b) and (c). The top two
rows of each example visualize the four onboard camera images (left, front,
right, rear). In the top row, the predicted traversability and elevation are
projected onto the images. The bottom row shows the respective traversability,
reliability, and elevation grid map, as well as the rasterized merged point
cloud. The traversability risk maps are shown for , , and the generated ground
truth. Blue indicates risk-free traversable and red untraversable. The
reliability is shown for the ground truth. The elevation maps are presented
for , the nearest neighbor interpolated elevation map predicted by , and the
ground truth elevation. For all maps, black indicates that no prediction is
available. (a, b) Yellow box highlights successful predictions at long range
where fails. (c) Red box highlights incorrect elevation prediction, based on
missing LiDAR and camera observations. Further details and interpretation of
each example can be found within Sec. 4.6.

In Fig. 11 (a), we highlighted a limitation of automatic ground truth
generation. In this scenario, a second vehicle (visible in the rear-facing
camera, highlighted by the yellow box) is following the main vehicle.
Consequently, the ground truth risk map is corrupted by the motion of this
dynamic obstacle. This can be seen by the small untraversable regions in front
of the vehicle within the ground truth traversability risk (highlighted by the
yellow box). Despite this, both and can accurately predict the correct
traversability, where the empty road and field ahead are labeled as risk-free
traversable. In Fig. 11 (b), demonstrates a particularly accurate estimation
of the scene’s elevation. The bottom of the canyon in front of the vehicle,
which cannot the sensed by the LiDAR sensors, is nearly perfectly identified.

###  4.7 Limitations

The generalization capability of to novel environments is constrained by the
limited size and diversity of the training dataset. Ongoing efforts focus on
scaling up the dataset size. Given the constraints imposed by limited training
data, it is not anticipated that will seamlessly adapt to entirely new, out-
of-distribution environments.

Moreover, ’s dependence on self-supervised ground truth generation based on
imposes limitations on its performance. Although self-supervision is cost-
effective and eliminates the need for human annotation, ’s performance in a
sense is upper bounded by ’s capability to identify risks. The integration of
sparse human supervision can help to address the existing failure cases of ,
such as metallic wire fences that remain undetected by LiDARs and are easily
misclassified in the image domain. Additionally, dynamic obstacles as seen in
Fig. 11 (a) are not correctly handled in the ground truth generation pipeline.
Furthermore, concerning the generation of ground truth, it is important to
note that while the operates within a yaw-fixed frame, our approach ()
provides predictions within the robot-centric frame. As a result, aligning the
pseudo ground truth to the robot-centric frame leads to artifacts in the form
of invalid values depending on the rotation angle between the two frames, as
shown in Fig. 11 and Fig. 12. Due to the missing supervision during training
for those specific regions, the network struggles to learn correct
predictions. As part of our future work, we plan to address this
implementation detail by expanding the pseudo ground truth maps by a factor of
$\sqrt{2}$, to prohibit those artifacts.

Another limitation is the temporal consistency of the predictions of as well
as the missing memory. For instance, when navigating down into a canyon,
cannot retain information about obstacles outside the current field of view,
as obstacles cannot be captured by cameras nor LiDAR measurements (compare
Fig. 12 (c)). This deficit could be addressed by adding memory units or
(autoregressive) feedback loops to the network architecture.

While can provide accurate traversability and elevation information, a better
understanding of the uncertainty of the prediction would be beneficial for the
planning and fusion of the predictions. For this, one could integrate methods
available in the evidential deep learning community to specifically understand
the epistemic uncertainty and detect out-of-distribution data.

Lastly, does not provide theoretical guarantees, in contrast to , where, under
specific conditions such as optimal sensing and constrained environments,
guarantees can be made based on implemented heuristics.

##  5 Conclusions & Future Work

In conclusion, represents a significant leap forward in the realm of mapping
and traversability estimation for high-speed off-road autonomous robot
navigation. We showcase improved traversability prediction and elevation
mapping performance, the benefit of sensor fusion, and a rigorous analysis of
the performance for a $16.5\text{\,}\mathrm{k}\mathrm{m}$ off-road dataset.
With a substantial reduction in latency, has the potential to enable vehicles
to operate at higher speeds autonomously in complex unstructured environments.
The implications of this approach extend to critical areas such as search and
rescue operations and even hold promise for robot applications in
extraterrestrial terrain, where reliable traversability understanding is key.

In future research, we will address existing challenges laid out in the Sec.
4.7, while exploring novel avenues, including attention-based fusion of
information, evidential deep learning methods for uncertainty prediction, and
a more physically grounded estimation of traversability concerning vehicle
dynamics by exploring the integration of the learned dynamics models. We
hypothesize novel data augmentation methods allowing for a coherent
augmentation of point cloud and image data in combination with synthetic data
may have the potential to further substantially improve the performance and
generalization.

##  6 Acknowledgments

The research was carried out at the Jet Propulsion Laboratory, California
Institute of Technology, under a contract with the National Aeronautics and
Space Administration (80NM0018D0004). This work was partially supported by the
Defense Advanced Research Projects Agency (DARPA) and the Swiss Federal
Institute of Technology, ETH Zurich. Jonas Frey and Julian Nubert are
supported by the Max Planck ETH Center for Learning Systems.

## References

  * Ahtiainen et al., 2017 Ahtiainen, J., Stoyanov, T., and Saarinen, J. (2017).  Normal distributions transform traversability maps: lidar-only approach for traversability mapping in outdoor environments.  Journal of Field Robotics, 34(3):600–621.
  * Behringer et al., 2004 Behringer, R., Sundareswaran, S., Gregory, B., Elsley, R., Addison, B., Guthmiller, W., Daily, R., and Bevly, D. (2004).  The darpa grand challenge-development of an autonomous vehicle.  In IEEE Intelligent Vehicles Symposium, 2004, pages 226–231. IEEE.
  * Bouman et al., 2020 Bouman, A., Ginting, M. F., Alatur, N., Palieri, M., Fan, D. D., Touma, T., Pailevanian, T., Kim, S.-K., Otsu, K., Burdick, J., et al. (2020).  Autonomous spot: Long-range autonomous exploration of extreme environments with legged locomotion.  In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2518–2525. IEEE.
  * Bradley et al., 2015 Bradley, D. M., Chang, J. K., Silver, D., Powers, M., Herman, H., Rander, P., and Stentz, A. (2015).  Scene understanding for a high-mobility walking robot.  In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1144–1151.
  * Brooks and Iagnemma, 2012 Brooks, C. A. and Iagnemma, K. (2012).  Self-supervised terrain classification for planetary surface exploration rovers.  Journal of Field Robotics, 29(3):445–468.
  * Cai et al., 2023 Cai, X., Ancha, S., Sharma, L., Osteen, P. R., Bucher, B., Phillips, S., Wang, J., Everett, M., Roy, N., and How, J. P. (2023).  Evora: Deep evidential traversability learning for risk-aware off-road autonomy.  arXiv preprint arXiv:2311.06234.
  * Cai et al., 2022 Cai, X., Everett, M., Sharma, L., Osteen, P. R., and How, J. P. (2022).  Probabilistic traversability model for risk-aware motion planning in off-road environments.  arXiv preprint arXiv:2210.00153.
  * Cao et al., 2022 Cao, C., Zhu, H., Yang, F., Xia, Y., Choset, H., Oh, J., and Zhang, J. (2022).  Autonomous exploration development environment and the planning algorithms.  In 2022 International Conference on Robotics and Automation (ICRA), page 8921–8928. IEEE.
  * Castro et al., 2023 Castro, M. G., Triest, S., Wang, W., Gregory, J. M., Sanchez, F., Rogers, J. G., and Scherer, S. (2023).  How does it feel? self-supervised costmap learning for off-road vehicle traversability.  In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 931–938. IEEE.
  * Chen et al., 2023 Chen, E., Ho, C., Maulimov, M., Wang, C., and Scherer, S. (2023).  Learning-on-the-drive: Self-supervised adaptation of visual offroad traversability models.  arXiv preprint arXiv:2306.15226.
  * Contributors, 2020 Contributors, M. (2020).  MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark.  https://github.com/open-mmlab/mmsegmentation.
  * Diaz-Zapata et al., 2023 Diaz-Zapata, M. A., Sierra González, D., Erkent, Ö., Dibangoye, J., and Laugier, C. (2023).  LAPTNet-FPN: Multi-scale LiDAR-aided Projective Transform Network for Real Time Semantic Grid Prediction.  In ICRA 2023 - IEEE International Conference on Robotics and Automation, pages 1–6, Londres, United Kingdom. IEEE Robotics and Automation Society, IEEE.
  * Dixit et al., 2023 Dixit, A., Fan, D. D., Otsu, K., Dey, S., Agha-Mohammadi, A.-A., and Burdick, J. W. (2023).  Step: Stochastic traversability evaluation and planning for risk-aware off-road navigation; results from the darpa subterranean challenge.
  * Duality Robotics, Inc., 2023 Duality Robotics, Inc. (2023).  Duality robotics - falcon pro simulator.
  * Endo et al., 2023 Endo, M., Taniai, T., Yonetani, R., and Ishigami, G. (2023).  Risk-aware path planning via probabilistic fusion of traversability prediction for planetary rovers on heterogeneous terrains.  arXiv preprint arXiv:2303.01169.
  * Erni et al., 2023 Erni, G., Frey, J., Miki, T., Mattamala Aravena, M. E., and Hutter, M. (2023).  Mem: Multi-modal elevation mapping for robotics and learning.  In 36th IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2023); Conference Location: Detroit, MI, USA; Conference Date: October 1-5, 2023; Conference lecture held on October 4, 2023.
  * Fakoorian et al., 2023 Fakoorian, S., Otsu, K., Khattak, S., Palieri, M., and Agha-mohammadi, A.-a. (2023).  Rose: Robust state estimation via online covariance adaption.  In Billard, A., Asfour, T., and Khatib, O., editors, Robotics Research, pages 452–467, Cham. Springer Nature Switzerland.
  * Fakoorian et al., 2021 Fakoorian, S. A., Santamaria-Navarro, A., Lopez, B. T., Simon, D., and Agha-mohammadi, A. (2021).  Towards robust state estimation by boosting the maximum correntropy criterion kalman filter with adaptive behaviors.  IEEE Robotics Autom. Lett., 6(3):5469–5476.
  * Fan et al., 2021 Fan, D. D., Otsu, K., Kubo, Y., Dixit, A., Burdick, J., and Agha-Mohammadi, A. (2021).  STEP: stochastic traversability evaluation and planning for safe off-road navigation.  CoRR, abs/2103.02828.
  * Frey et al., 2022 Frey, J., Hoeller, D., Khattak, S., and Hutter, M. (2022).  Locomotion policy guided traversability learning using volumetric representations of complex environments.  In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE.
  * Frey et al., 2023 Frey, J., Mattamala, M., Chebrolu, N., Cadena, C., Fallon, M., and Hutter, M. (2023).  Fast Traversability Estimation for Wild Visual Navigation.  In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea.
  * Gasparino et al., 2022 Gasparino, M. V., Sivakumar, A. N., Liu, Y., Velasquez, A. E. B., Higuti, V. A. H., Rogers, J., Tran, H., and Chowdhary, G. (2022).  WayFAST: Navigation with predictive traversability in the field.  IEEE Robotics and Automation Letters, 7(4):10651–10658.
  * Gibson et al., 2023 Gibson, J., Vlahov, B., Fan, D., Spieler, P., Pastor, D., akbar Agha-mohammadi, A., and Theodorou, E. A. (2023).  A multi-step dynamics modeling framework for autonomous driving in multiple environments.
  * Goh et al., 2022 Goh, E., Chen, J., and Wilson, B. (2022).  Mars terrain segmentation with less labels.  In 2022 IEEE Aerospace Conference (AERO), pages 1–10. IEEE.
  * Guzzi et al., 2020 Guzzi, J., Chavez-Garcia, R. O., Nava, M., Gambardella, L. M., and Giusti, A. (2020).  Path planning with local motion estimations.  IEEE Robotics and Automation Letters, 5(2):2586–2593.
  * Hadsell et al., 2007 Hadsell, R., Sermanet, P., Ben, J., Erkan, A., Han, J., Flepp, B., Muller, U., and LeCun, Y. (2007).  Online learning for offroad robots: Using spatial label propagation to learn long-range traversability.  In Proc. of Robotics: Science and Systems (RSS), volume 11, page 32. Citeseer.
  * Hadsell et al., 2009 Hadsell, R., Sermanet, P., Ben, J., Erkan, A., Scoffier, M., Kavukcuoglu, K., Muller, U., and LeCun, Y. (2009).  Learning long-range vision for autonomous off-road driving.  Journal of Field Robotics, 26(2):120–144.
  * Harley et al., 2023 Harley, A. W., Fang, Z., Li, J., Ambrus, R., and Fragkiadaki, K. (2023).  Simple-BEV: What really matters for multi-sensor bev perception?  In IEEE International Conference on Robotics and Automation (ICRA).
  * Higa et al., 2019 Higa, S., Iwashita, Y., Otsu, K., Ono, M., Lamarre, O., Didier, A., and Hoffmann, M. (2019).  Vision-based estimation of driving energy for planetary rovers using deep learning and terramechanics.  IEEE Robotics and Automation Letters, 4(4):3876–3883.
  * Hudson et al., 2022 Hudson, N., Talbot, F., Cox, M., Williams, J., Hines, T., Pitt, A., Wood, B., Frousheger, D., Surdo, K. L., Molnar, T., Steindl, R., Wildie, M., Sa, I., Kottege, N., Stepanas, K., Hernandez, E., Catt, G., Docherty, W., Tidd, B., Tam, B., Murrell, S., Bessell, M., Hanson, L., Tychsen-Smith, L., Suzuki, H., Overs, L., Kendoul, F., Wagner, G., Palmer, D., Milani, P., O’Brien, M., Jiang, S., Chen, S., and Arkin, R. (2022).  Heterogeneous ground and air platforms, homogeneous sensing: Team CSIRO data61’s approach to the DARPA subterranean challenge.  Field Robotics, 2(1):595–636.
  * Jackel et al., 2006 Jackel, L. D., Krotkov, E., Perschbacher, M., Pippine, J., and Sullivan, C. (2006).  The darpa lagr program: Goals, challenges, methodology, and phase i results.  Journal of Field Robotics, 23(11-12):945–973.
  * Jiang et al., 2021 Jiang, P., Osteen, P., Wigness, M., and Saripalli, S. (2021).  Rellis-3d dataset: Data, benchmarks and analysis.  In 2021 IEEE international conference on robotics and automation (ICRA), pages 1110–1116. IEEE.
  * Jung et al., 2023 Jung, S., Lee, J., Meng, X., Boots, B., and Lambert, A. (2023).  V-strong: Visual self-supervised traversability learning for off-road navigation.  arXiv preprint arXiv:2312.16016.
  * Kahn et al., 2021 Kahn, G., Abbeel, P., and Levine, S. (2021).  BADGR: An Autonomous Self-Supervised Learning-Based Navigation System.  IEEE Robotics and Automation Letters, 6(2):1312–1319.
  * Karnan et al., 2022 Karnan, H., Sikand, K. S., Atreya, P., Rabiee, S., Xiao, X., Warnell, G., Stone, P., and Biswas, J. (2022).  Vi-ikd: High-speed accurate off-road navigation using learned visual-inertial inverse kinodynamics.  In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3294–3301. IEEE.
  * Kelly et al., 2006 Kelly, A., Stentz, A., Amidi, O., Bode, M., Bradley, D., Diaz-Calderon, A., Happold, M., Herman, H., Mandelbaum, R., Pilarski, T., et al. (2006).  Toward reliable off road autonomous vehicles operating in challenging environments.  The International Journal of Robotics Research, 25(5-6):449–483.
  * Kim et al., 2006 Kim, D., Sun, J., Oh, S. M., Rehg, J., and Bobick, A. (2006).  Traversability classification using unsupervised on-line visual learning for outdoor robot navigation.  In Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006., pages 518–525.
  * Kim et al., 2022 Kim, Y., Kim, C., and Hwangbo, J. (2022).  Learning forward dynamics model and informed trajectory sampler for safe quadruped navigation.  In Hauser, K., Shell, D. A., and Huang, S., editors, Robotics: Science and Systems XVIII, New York City, NY, USA, June 27 - July 1, 2022.
  * Konolige et al., 2009 Konolige, K., Agrawal, M., Blas, M. R., Bolles, R. C., Gerkey, B., Sola, J., and Sundaresan, A. (2009).  Mapping, navigation, and learning for off-road traversal.  Journal of Field Robotics, 26(1):88–113.
  * Lang et al., 2019 Lang, A. H., Vora, S., Caesar, H., Zhou, L., Yang, J., and Beijbom, O. (2019).  Pointpillars: Fast encoders for object detection from point clouds.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 12697–12705. Computer Vision Foundation / IEEE.
  * Liu et al., 2023 Liu, Z., Tang, H., Amini, A., Yang, X., Mao, H., Rus, D., and Han, S. (2023).  Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation.  In IEEE International Conference on Robotics and Automation (ICRA).
  * Loshchilov and Hutter, 2019 Loshchilov, I. and Hutter, F. (2019).  Decoupled weight decay regularization.  In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
  * Majumdar and Pavone, 2017 Majumdar, A. and Pavone, M. (2017).  How should a robot assess risk? towards an axiomatic theory of risk in robotics.  In Amato, N. M., Hager, G., Thomas, S. L., and Torres-Torriti, M., editors, Robotics Research, The 18th International Symposium, ISRR 2017, Puerto Varas, Chile, December 11-14, 2017, volume 10 of Springer Proceedings in Advanced Robotics, pages 75–84. Springer.
  * Manduchi et al., 2005 Manduchi, R., Castano, A., Talukder, A., and Matthies, L. (2005).  Obstacle detection and terrain classification for autonomous off-road navigation.  Autonomous robots, 18:81–102.
  * Maturana et al., 2017 Maturana, D., Chou, P.-W., Uenoyama, M., and Scherer, S. (2017).  Real-time semantic mapping for autonomous off-road navigation.  In Proceedings of 11th International Conference on Field and Service Robotics (FSR ’17), pages 335 – 350.
  * Maxon, 2005 Maxon, M. A. (2005).  The real roadrunner, volume 9.  University of Oklahoma Press.
  * Meng et al., 2023 Meng, X., Hatch, N., Lambert, A., Li, A., Wagener, N., Schmittle, M., Lee, J., Yuan, W., Chen, Z., Deng, S., Okopal, G., Fox, D., Boots, B., and Shaban, A. (2023).  Terrainnet: Visual modeling of complex terrain for high-speed, off-road navigation.  In Proceedings of Robotics: Science and System XIX. Robotics Science & Systems Foundation.  Robotics: Science and Systems (RSS 2023); Conference Location: Daegu, South Korea; Conference Date: July 10-14, 2023.
  * Muller et al., 2005 Muller, U., Ben, J., Cosatto, E., Flepp, B., and Cun, Y. (2005).  Off-road obstacle avoidance through end-to-end learning.  In Weiss, Y., Schölkopf, B., and Platt, J., editors, Advances in Neural Information Processing Systems, volume 18. MIT Press.
  * Ng et al., 2020 Ng, M. H., Radia, K., Chen, J., Wang, D., Gog, I., and Gonzalez, J. E. (2020).  Bev-seg: Bird’s eye view semantic segmentation using geometry and semantic point cloud.  CVPR 2020 Workshop Scalability in Autonomous Driving by Waymo, abs/2006.11436.
  * Nubert et al., 2022 Nubert, J., Khattak, S., and Hutter, M. (2022).  Graph-based multi-sensor fusion for consistent localization of autonomous construction robots.  In 2022 International Conference on Robotics and Automation (ICRA), pages 10048–10054.
  * Ono et al., 2015 Ono, M., Fuchs, T. J., Steffy, A., Maimone, M., and Yen, J. (2015).  Risk-aware planetary rover operation: Autonomous terrain classification and path planning.  In 2015 IEEE aerospace conference, pages 1–10. IEEE.
  * Otsu et al., 2016 Otsu, K., Ono, M., Fuchs, T. J., Baldwin, I., and Kubota, T. (2016).  Autonomous terrain classification with co- and self-training approach.  IEEE Robotics and Automation Letters, 1(2):814–819.
  * Overbye and Saripalli, 2022 Overbye, T. and Saripalli, S. (2022).  G-vom: A gpu accelerated voxel off-road mapping system.  In 2022 IEEE Intelligent Vehicles Symposium (IV), pages 1480–1486.
  * Philion and Fidler, 2020 Philion, J. and Fidler, S. (2020).  Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d.  In Vedaldi, A., Bischof, H., Brox, T., and Frahm, J., editors, Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV, volume 12359 of Lecture Notes in Computer Science, pages 194–210. Springer.
  * Pomerleau, 1993 Pomerleau, D. A. (1993).  Knowledge-based training of artificial neural networks for autonomous robot driving.  In Robot learning, pages 19–43. Springer.
  * Richter and Roy, 2017 Richter, C. and Roy, N. (2017).  Safe visual navigation via deep learning and novelty detection.  In Robotics: Science and Systems, Cambridge, Massachusetts.
  * Roddick et al., 2019 Roddick, T., Kendall, A., and Cipolla, R. (2019).  Orthographic feature transform for monocular 3d object detection.  In 30th British Machine Vision Conference 2019, BMVC 2019, Cardiff, UK, September 9-12, 2019, page 285. BMVA Press.
  * Roth et al., 2023 Roth, P., Nubert, J., Yang, F., Mittal, M., and Hutter, M. (2023).  Viplanner: Visual semantic imperative learning for local navigation.  In 2023 IEEE International Conference on Robotics and Automation (ICRA).
  * Rothrock et al., 2016 Rothrock, B., Kennedy, R., Cunningham, C., Papon, J., Heverly, M., and Ono, M. (2016).  Spoc: Deep learning-based terrain classification for mars rover missions, page 5539.  Aerospace Research Central.
  * Saha et al., 2022 Saha, A., Mendez, O., Russell, C., and Bowden, R. (2022).  Translating images into maps.  In 2022 International Conference on Robotics and Automation, ICRA 2022, Philadelphia, PA, USA, May 23-27, 2022, pages 9200–9206. IEEE.
  * Sathyamoorthy et al., 2022 Sathyamoorthy, A. J., Weerakoon, K., Guan, T., Liang, J., and Manocha, D. (2022).  Terrapn: Unstructured terrain navigation using online self-supervised learning.  In IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2022, Kyoto, Japan, October 23-27, 2022, pages 7197–7204. IEEE.
  * Schilling et al., 2017 Schilling, F., Chen, X., Folkesson, J., and Jensfelt, P. (2017).  Geometric and visual terrain classification for autonomous mobile navigation.  In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2678–2684.
  * Schmid et al., 2022 Schmid, R., Atha, D., Schöller, F., Dey, S., Fakoorian, S., Otsu, K., Ridge, B., Bjelonic, M., Wellhausen, L., Hutter, M., et al. (2022).  Self-supervised traversability prediction by learning to reconstruct safe terrain.  In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 12419–12425. IEEE.
  * Seo et al., 2022 Seo, J., Kim, T., Kwak, K. H., Min, J., and Shim, I. (2022).  Scate: A scalable framework for self- supervised traversability estimation in unstructured environments.  IEEE Robotics and Automation Letters, 8:888–895.
  * Seo et al., 2023 Seo, J., Sim, S., and Shim, I. (2023).  Learning off-road terrain traversability with self-supervisions only.  IEEE Robotics and Automation Letters.
  * Shaban et al., 2022 Shaban, A., Meng, X., Lee, J., Boots, B., and Fox, D. (2022).  Semantic terrain classification for off-road autonomous driving.  In Conference on Robot Learning, pages 619–629. PMLR.
  * Shoemaker and Bornstein, 1998 Shoemaker, C. M. and Bornstein, J. A. (1998).  The demo iii ugv program: A testbed for autonomous navigation research.  In Proceedings of the 1998 IEEE International Symposium on Intelligent Control (ISIC) held jointly with IEEE International Symposium on Computational Intelligence in Robotics and Automation (CIRA) Intell, pages 644–651. IEEE.
  * Smith and Topin, 2017 Smith, L. N. and Topin, N. (2017).  Super-convergence: Very fast training of residual networks using large learning rates.  CoRR, abs/1708.07120.
  * Stanislas et al., 2021 Stanislas, L., Nubert, J., Dugas, D., Nitsch, J., Sünderhauf, N., Siegwart, R., Cadena, C., and Peynot, T. (2021).  Airborne particle classification in lidar point clouds using deep learning.  In Field and Service Robotics: Results of the 12th International Conference, pages 395–410. Springer.
  * Strudel et al., 2021 Strudel, R., Pinel, R. G., Laptev, I., and Schmid, C. (2021).  Segmenter: Transformer for semantic segmentation.  In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 7242–7252. IEEE.
  * Swan et al., 2021 Swan, R. M., Atha, D., Leopold, H. A., Gildner, M., Oij, S., Chiu, C., and Ono, M. (2021).  Ai4mars: A dataset for terrain-aware autonomous driving on mars.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1982–1991.
  * Tan and Le, 2019 Tan, M. and Le, Q. V. (2019).  Efficientnet: Rethinking model scaling for convolutional neural networks.  In Chaudhuri, K. and Salakhutdinov, R., editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 6105–6114. PMLR.
  * Thrun et al., 2006 Thrun, S., Montemerlo, M., and Aron, A. (2006).  Probabilistic terrain analysis for high-speed desert driving.  In Robotics: Science and Systems, pages 16–19.
  * Triebel et al., 2006 Triebel, R., Pfaff, P., and Burgard, W. (2006).  Multi-level surface maps for outdoor terrain mapping and loop closing.  In 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2276–2282.
  * Triest et al., 2023 Triest, S., Castro, M. G., Maheshwari, P., Sivaprakasam, M., Wang, W., and Scherer, S. A. (2023).  Learning risk-aware costmaps via inverse reinforcement learning for off-road navigation.  2023 IEEE International Conference on Robotics and Automation (ICRA), pages 924–930.
  * Urmson et al., 2006 Urmson, C., Ragusa, C., Ray, D., Anhalt, J., Bartz, D., Galatali, T., Gutierrez, A., Johnston, J., Harbaugh, S., “Yu” Kato, H., et al. (2006).  A robust approach to high-speed navigation for unrehearsed desert terrain.  Journal of Field Robotics, 23(8):467–508.
  * Valada et al., 2016 Valada, A., Oliveira, G., Brox, T., and Burgard, W. (2016).  Deep multispectral semantic scene understanding of forested environments using multimodal fusion.  In International Symposium on Experimental Robotics (ISER).
  * Viswanath et al., 2021 Viswanath, K., Singh, K., Jiang, P., Sujit, P., and Saripalli, S. (2021).  Offseg: A semantic segmentation framework for off-road driving.  In 2021 IEEE 17th International Conference on Automation Science and Engineering (CASE), pages 354–359. IEEE.
  * Wellhausen and Hutter, 2023 Wellhausen, L. and Hutter, M. (2023).  Artplanner: Robust legged robot navigation in the field.  In Field Robotics.
  * Wellhausen et al., 2020 Wellhausen, L., Ranftl, R., and Hutter, M. (2020).  Safe Robot Navigation Via Multi-Modal Anomaly Detection.  IEEE Robotics and Automation Letters.
  * Wigness et al., 2019 Wigness, M., Eum, S., Rogers, J. G., Han, D., and Kwon, H. (2019).  A rugd dataset for autonomous navigation and visual perception in unstructured outdoor environments.  In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5000–5007. IEEE.
  * Williams et al., 2018 Williams, G., Drews, P., Goldfain, B., Rehg, J. M., and Theodorou, E. A. (2018).  Information-theoretic model predictive control: Theory and applications to autonomous driving.  IEEE Transactions on Robotics, 34(6):1603–1622.
  * Williams et al., 2017 Williams, G., Wagener, N., Goldfain, B., Drews, P., Rehg, J. M., Boots, B., and Theodorou, E. A. (2017).  Information theoretic mpc for model-based reinforcement learning.  In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 1714–1721.
  * Wulfmeier et al., 2015 Wulfmeier, M., Ondruska, P., and Posner, I. (2015).  Maximum entropy deep inverse reinforcement learning.  arXiv preprint arXiv:1507.04888.
  * Xiao et al., 2021 Xiao, X., Biswas, J., and Stone, P. (2021).  Learning inverse kinodynamics for accurate high-speed off-road navigation on unstructured terrain.  IEEE Robotics and Automation Letters, 6(3):6054–6060.
  * Xie et al., 2021 Xie, E., Wang, W., Yu, Z., Anandkumar, A., Álvarez, J. M., and Luo, P. (2021).  Segformer: Simple and efficient design for semantic segmentation with transformers.  In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 12077–12090.
  * Xie et al., 2022 Xie, E., Yu, Z., Zhou, D., Philion, J., Anandkumar, A., Fidler, S., Luo, P., and Alvarez, J. M. (2022).  M^ 2bev: Multi-camera joint 3d detection and segmentation with unified birds-eye view representation.  arXiv preprint arXiv:2204.05088.
  * Xue et al., 2023a Xue, H., Fu, H., Xiao, L., Fan, Y., Zhao, D., and Dai, B. (2023a).  Traversability analysis for autonomous driving in complex environment: A lidar-based terrain modeling approach.  Journal of Field Robotics.
  * Xue et al., 2023b Xue, H., Hu, X., Xie, R., Fu, H., Xiao, L., Nie, Y., and Dai, B. (2023b).  Contrastive label disambiguation for self-supervised terrain traversability learning in off-road environments.  arXiv preprint arXiv:2307.02871.
  * Zhang et al., 2022 Zhang, J., Lin, L., Fan, Z., Wang, W., and Liu, J. (2022).  S5mars: Self-supervised and semi-supervised learning for mars segmentation.  arXiv preprint arXiv:2207.01200.
  * Zürn et al., 2021 Zürn, J., Burgard, W., and Valada, A. (2021).  Self-supervised visual terrain classification from unsupervised acoustic feature learning.  IEEE Transactions on Robotics, 37(2):466–481.