# AutoPilot: Automating Co-Design Space Exploration for Autonomous UAVs

Srivatsan Krishnan  Harvard University  Zishen Wan  Harvard University
Kshitij Bhardwaj  Harvard University  Paul Whatmough  ARM Research  Aleksandra
Faust  Google Brain Research  Sabrina M. Neuman  Harvard University  Gu-Yeon
Wei  Harvard University  David Brooks  Harvard University  Vijay Janapa Reddi
Harvard University

###### Abstract

Building domain-specific accelerators for autonomous unmanned aerial vehicles
(UAVs) is challenging due to a lack of systematic methodology for designing
onboard compute. Balancing a computing system for a UAV requires considering
both the cyber (e.g., sensor rate, compute performance) and physical (e.g.,
payload weight) characteristics that affect overall performance. Iterating
over the many component choices results in a combinatorial explosion of the
number of possible combinations: from 10s of thousands to billions, depending
on implementation details. Manually selecting combinations of these components
is tedious and expensive. To navigate the cyber-physical design space
efficiently, we introduce _AutoPilot_ , a framework that automates full-system
UAV co-design. AutoPilot uses Bayesian optimization to navigate a large design
space and automatically select a combination of autonomy algorithm and
hardware accelerator while considering the cross-product effect of other cyber
and physical UAV components. We show that the AutoPilot methodology
consistently outperforms general-purpose hardware selections like Xavier NX
and Jetson TX2, as well as dedicated hardware accelerators built for
autonomous UAVs, across a range of representative scenarios (three different
UAV types and three deployment environments). Designs generated by AutoPilot
increase the number of missions on average by up to 2.25$\times$,
1.62$\times$, and 1.43$\times$ for nano, micro, and mini-UAVs respectively
over baselines. Our work demonstrates the need for holistic full-UAV co-design
to achieve maximum overall UAV performance and the need for automated flows to
simplify the design process for autonomous cyber-physical systems.

##  1 Introduction

Unmanned aerial vehicles (UAVs) are on the rise in real-world deployments [75,
52, 15, 62], but building computing systems for these platforms remains
challenging. They are complex systems in which the traditional computing
platform is just one component among many others. To achieve overall
performance, it is important to understand what implications other UAV
components have on the design of onboard compute.

Figure 1: AutoPilot efficiently automates navigation of the large UAV
component design space to co-design optimal onboard compute across a range of
autonomous UAV systems.

Co-designing hardware accelerators with other UAV components requires
navigating a large design space (see Fig. 1), e.g., 100’s of UAVs [44]
$\times$ millions of HW accelerators [78] $\times$ billions of autonomy
algorithm neural network model paramaters [22] $\times$ 100’s of sensors [45]
$\approx$ 1018. Worse, this number is still conservative since each UAV type
includes additional components such as a flight controller and a battery.
Taming this large space can be expensive and tedious. Automating the co-design
of the hardware accelerator and other UAV system components can optimize
mission performance while keeping design overheads low as UAV systems evolve.

Key challenges in UAV design include the ability to systematically navigate
the large design space of components, and understanding which combinations of
these components maximize overall UAV performance. While specialized hardware
is critical for compute efficiency, designing it is an expensive process. It
is essential to establish automated design methodologies that remain agile as
future autonomous systems evolve.

To address these challenges, we introduce _AutoPilot: a cyber-physical co-
design automation framework for autonomous UAVs_. Given a high-level
specification of autonomy task, UAV type, and mission goals, AutoPilot
automatically navigates the large design space to perform full-system UAV co-
design to generate a combination of autonomy algorithm and corresponding
hardware accelerator to maximize overall UAV performance (e.g., number of
missions).

The AutoPilot takes a high-level specification for the autonomy task as input,
and has three main steps (see Fig. 2): (1) Trains several end-to-end autonomy
algorithms for a given autonomy task (autonomous navigation) using
reinforcement learning [38], and validate task-level functionality
(\raisebox{-.9pt} {1}⃝ in Fig. 2). (2) Perform multi-objective algorithm-
hardware co-design to maximize task success rate, compute performance and
minimize power using Bayesian optimization [31] (\raisebox{-.9pt} {2}⃝ in Fig.
2). This ensures that we traverse the large design space rapidly and obtain
several interesting design candidates. (3) Finally, we perform co-design
across the full system of UAV components to select the optimal compute and
autonomy algorithm combination for a given UAV to maximize its mission
performance, i.e., number of missions (\raisebox{-.9pt} {3}⃝ in Fig. 2). This
step accounts for the cross-product effect across the full-UAV stack (Fig. 1)
and is critical to maximize UAV performance.

Figure 2: Overview of AutoPilot design flow.

Prior work has largely focused on quantifying the UAV design space [30] and
designing standalone UAV compute systems [65, 57] (see Table 1). Work
examining the entire cyber-physical UAV system stack has quantified tradeoffs
[30] but does not provide a systematic automated solution to navigate this
enormous design space to produce optimal designs. Previous work also focuses
on optimizing single kernels (e.g., localization and mapping [30]), while a
fully autonomous UAV requires the synthesis of many computationally-intensive
kernels (e.g., map integration, motion planning) [17, 48, 51]. Prior work on
designing UAV compute has provided solutions using off-the-shelf hardware [19,
48, 25, 77, 47, 49, 66, 70] or standalone hardware designed in isolation from
the rest of the cyber-physical stack [57, 73].

By contrast, with AutoPilot, we offer a complete solution for _automatically
navigating_ the UAV design space and performing _holistic co-design_ across
the entire UAV system stack. To demonstrate the scalability of the AutoPilot
methodology, we apply it to three different UAV types and three domain
randomized environments (total of nine combinations). These auto-generated
domain randomized environments have varying degrees of obstacle densities.
They represent common deployment use cases for drones. For instance,
autonomous navigation for a farming use case could be very sparse (low-
obstacle), whereas a search and rescue operation in a forest is a dense-
obstacle scenario. The three fully randomized environments represent the
deployment scenario ranging from low obstacle density to high obstacle density
(to signify the difficulty in the autonomous task). We show that the AutoPilot
methodology consistently outperforms general-purpose hardware selections like
Xavier NX and Jetson TX2 and dedicated hardware accelerators built for
autonomous UAVs across a range of representative scenarios.

In summary, our main contributions are as follows:

  * •

Introduce AutoPilot, an intelligent _design space exploration framework_ that
_automatically_ co-designs a learning-based autonomy algorithm and its
accelerator for a diverse class of UAVs and deployment scenarios.

  * •

Leverage _cyber-physical co-design_ for UAVs, taking the entire system stack
under consideration to maximize overall UAV performance, i.e., number of
missions.

  * •

Generate optimal designs that increase the number of missions by 2.25$\times$,
1.62$\times$, and 1.43$\times$ for nano, micro, and mini-UAVs respectively
over baselines.

Prior Work |  End-to-End Autonomy? |  Hardware Acceleration | Cyber and Physical Parameters |  Provides Design Methodology? | Automated?
---|---|---|---|---|---
| Varies Sensor
---
(Frame rates)
| Varies Physical
---
parameters
(Thrust-to-ratio)
Navion | ✗ | Only VIO | ✗ | ✗ | ✗ | ✗
Hadidi et al | ✗ | Only SLAM | ✗ | ✗ | ✔ | ✗
RoboX | ✗ | Only Motion Planning | ✗ | ✔ | ✔ | ✔
MavBench | ✔ | ✗ | ✗ | ✗ | ✗ | ✗
PULP-DroNet | ✔ | Full end-to-end stack | ✗ | ✗ | ✗ | ✗
| AutoPilot
---
(This Work)
✔ | Full End-to-End Stack |  | ✔
---
(Section 5.3)
| ✔
---
(Section 5.4)
| ✔
---
(Section 5.4)
| ✔
---

Table 1: Comparison of selected related work. AutoPilot provides an automated
methodology for co-design across the cyber-physical system stack for end-to-
end UAV autonomy.

##  2 Background

Autonomous UAVs have many components that must be considered in the co-design
process (Fig. 1). We first define UAV size classifications. We then describe
the basic components of a canonical UAV (without full autonomy), and finally,
we discuss additional components needed for achieving autonomy.

UAV Size Classifications. UAV size impacts its physics, which must be
considered for full cyber-physical co-design. There are three broad categories
of UAV sizes: mini-UAVs (100’s-1000’s mm & 1-1.5 Kgs), micro-UAVs (100’s mm &
100’s g), and nano-UAVs (10’s mm & 10’s g). UAV size determines the maximum
payload weight and battery capacity.

###  2.1 Base UAV Components

A base UAV system contains components that allow humans to teleoperate the UAV
but does not have the necessary components for full autonomy.

Electro-Mechanical Components. Depending upon the UAV size, the UAV has a
proportional size frame where all the other components are mounted. The
propulsion uses motors that share the battery energy and other electronics in
the UAV. Electronic speed control (ESC) is used to provide precise rotor speed
based on the commands from the flight controller.

Flight Controller. Flight controller is solely dedicated to the stabilization
and control of the UAV. The flight controller firmware is computationally
lightweight and is typically run on microcontrollers [12, 11], that are
tightly integrated into the UAV platform. The flight controller uses onboard
sensors, such as the Inertial Measurement Unit (IMU) [13] and GPS, to
stabilize and control the UAV. To recover from unpredictable errors (sudden
winds or damaged rotors), the inner-loop typically runs at closed-loop
frequencies of up to 1 kHz [29, 37].

Battery. The battery capacity of a UAV is the limiting factor in the number of
missions it can complete on a single charge before its energy budget is
depleted. UAV size and payload restrictions determine its battery capacity.
Hence, flight time is often limited to few tens of minutes. Typically, a mini-
UAVs can carry a bigger battery (1000’s of mAh for about 20 to 25 minutes of
flight) due to their larger size. Nano-UAVs, on the other hand, due to their
small size, can only carry very small batteries (100’s mAh for about 6 to 7
minutes of flight time).

###  2.2 Additional Components for UAV Autonomy

To achieve full autonomy, the base UAV system needs to be augmented with more
components to allow the UAV to make intelligent decisions without any human
intervention.

External Sensors. The sensor creates snapshots of the environment and feeds
the information to the autonomy algorithm (see below), generating high-level
action. The UAV size and weight restriction impact the selection of sensors.
The most common type of sensors (across UAV types) is RGB camera [41, 50]. The
weight of the sensor (payload weight), its performance (framerate), and
sensing range affect the UAV’s ability to navigate an environment
autonomously.

Autonomy Algorithm. Autonomy in drones can be achieved by algorithms
classified into two broad categories: Sense-Plan-Act (SPA) and End-to-End
(E2E) Learning.

In SPA the algorithm is broken into distinct stages: sensing, planning, and
control. In the sensing stage, sensor data is used to create a map [64, 21,
18] of the environment. The planning stage [35, 27] processes the map to
determine the best trajectory. The trajectory information is used by the
control stage, which actuates the rotor, so the robot follows the trajectory.
SPA is usually slow in performance [49, 57, 19, 20].

E2E learning is an alternate paradigm where the algorithms process raw input
sensor information (e.g., RGB, Lidar) and use a neural network model to
directly produce output actions. Unlike SPA, the E2E learning methods do not
require maps or separate planning stages and hence are much faster than non-
NN-based autonomy algorithms (i.e., SPA) [57, 8]. The model can be trained
using supervised learning [16, 49, 63, 71, 74] or reinforcement learning [19,
66, 24, 34, 36, 39, 14]. Major companies working on autonomy have made public
announcements about using E2E-based approaches [74, 33].

Onboard Compute. Autonomous UAVs have a dedicated compute subsystem that runs
the autonomy algorithm, generating high-level actions or plans. Onboard
compute can be classified into two types: off-the-shelf general-purpose
compute and domain-specific hardware accelerators. Due to their widespread
availability, prior work focusing on autonomy algorithms often rely on
general-purpose hardware such as Intel NUC, Nvidia Jetson TX1/2, ARM Cortex 9,
or even microcontrollers like M4 [1] to validate the functionality of the
autonomy algorithms. The selection criteria are often based on the size/weight
of the UAVs. Due to their larger payload carrying capability, mini-UAVs
typically use general-purpose compute like Intel NUC, Jetson TX1/TX2. Nano-
UAVs, due to their limited size, have microcontrollers [1]. Recently there has
been efforts to build domain-specific accelerators for autonomous UAVs (see
Section 7 for details). However, these accelerators are typically optimized
for higher throughput or low power without consideration of how these
performance gains affect the compute weight. The payload weight of these
specialized accelerators is a critical metric to consider for UAV system co-
design because it affects a UAV’s agility [40].

Figure 3: AutoPilot methodology for automating cyber-physical co-design in
UAVs. _Phase 1:_ Multiple E2E models trained using Air Learning [38] simulator
based on high-level task specification. Success rates and hyper-parameters
stored in a database. _Phase 2:_ Multi-objective DSE using Bayesian
optimization to find E2E model and accelerator designs that are optimal in
success rate, and accelerator power and performance. _Phase 3:_ F-1 UAV
tradeoff model [40] used to find the E2E model and accelerator that maximizes
UAV mission performance.

##  3 AutoPilot

Autonomous UAV systems require many components for which there are many
options to choose from, generating an enormous design space (Fig. 1). To
address the challenge of design space exploration over this combinatorial
explosion of component choices, we introduce AutoPilot: an automation
framework that intelligently navigates the large UAV design space and
automatically generates a combination of hardware accelerator and autonomy
algorithm for a given UAV type.

AutoPilot consist of three stages (see Fig. 3). Phase 1 prepares a collection
of autonomy algorithm implementations that are functionally correct for
performing autonomous UAV tasks. It takes an input specification of the
autonomous UAV tasks and trains several end-to-end (E2E) autonomy algorithms
for a given task using reinforcement learning (RL). Phase 2 performs an
automated design space exploration (DSE) using Bayesian optimization [31] to
find the Pareto frontier of E2E algorithms and hardware accelerators that are
optimal in terms of task success rate and power and runtime performance. Phase
3 performs full-system UAV co-design based on its size, sensor characteristics
(e.g., framerate and weight), and the design candidates (Pareto frontier
designs) produced in Phase 2. Finally, a combination of the E2E algorithm and
an accelerator is selected to maximize the number of missions.

###  3.1 Phase 1: Task Specification and Training Using RL

In this phase, the user provides task-level specification for a UAV
application (e.g., Source Seeking [19]) which includes some rough estimates
about the environments such as obstacle densities. Based on this information,
AutoPilot configures the Air Learning [38] environment generator. AutoPilot
uses Air Learning to train and validate E2E autonomy algorithms for a given
UAV task. Air Learning provides a high-quality implementation of reinforcement
learning algorithms that can be used to train a neural network navigation
policy for the UAV. Air Learning includes a configurable environment generator
[2] with domain randomization [76] that allows changing various parameters
(e.g., the number of obstacles and size of the arena) to aid in
generalizability. These parameters are configured based on the autonomy task
specification.

To determine the E2E model for each robot task (defined by environment
complexity, i.e., obstacles), we start with the base template used in Air
Learning shown in Fig. 4a and vary its hyperparameters (number of layers and
filters) to create many candidate NN policies. We start from a known template
and vary the parameters inside the template because not all the layers within
the E2E model improve UAV task-level performance. Using domain knowledge, we
can seed the search process to explore regions that quickly give us desired
results. For example, making the template layers deeper and wider gives a nice
trade-off between the number of parameters and task-level success rate, as
shown in Fig. 4b. The task-level success rate of 60% to 91% is comparable to
autonomous navigation task success rate reported in robotics literature [70,
26, 66, 28, 68] for similar difficulty levels.

Based on these template parameters and the desired success rate, AutoPilot
launches several Air Learning training instances. Each of the NN policies that
achieve the required success rate is evaluated in a domain random environment
[76], and its task-level functionality is validated. The validated NN policies
are updated into an Air Learning database along with their success rates,
which are then used by Bayesian optimization in the next DSE phase (Section
3.2).

It is important to note that this NN parameter seed selection may be
inappropriate for a different task (similar to how ImageNet trained models
might fail if applied to medical images). The goal of this phase is to have
the flexibility to train several E2E models, and there is no restriction on
search space size.

###  3.2 Phase 2: Bayesian Optimization HW-SW Co-Design

In this phase, an automated multi-objective design space exploration is
performed to find the Pareto frontier of E2E algorithms and hardware
accelerator architectures that achieve optimal task success rate, performance,
and power for a given autonomy task. The success rate is only affected by
neural network hyperparameters (e.g., number of layers and filters). The
accelerator’s runtime and power depend on the E2E model and accelerator
architectural parameters.

Success rates for the policies are accessed from the Air Learning database. At
the same time, a cycle-accurate simulator is used to evaluate accelerator
performance and power for the different policies and hardware configurations.
Finally, we use Bayesian optimization to achieve rapid convergence to optimal
solutions without performing an exhaustive search.

Air Learning Database. This database stores the training results for the
various E2E autonomy algorithms trained using Air Learning. Each entry in the
database has an E2E algorithm identifier, the hyperparameters used for
training, and the success rate for the policy after validation.

SoC Architecture. We assume an SoC, which includes a parameterized template
for hardware accelerator shown in Fig. 5a. The onboard compute system consists
of two ultra-low-power cores for running the flight controller, an accelerator
sub-system (Systolic array-based), an external memory (DRAM), and an onboard
RGB sensor connected to the system bus. The flight controller software stack
is a PID controller that runs bare-metal on the MCU, similar to Bitcraze
Crazyflie aerial robots [1, 19]. We also assume that the camera is interfaced
with the system using a camera parallel interface [58] or MIPI [46] similar to
this work [57] from which the accelerator sub-system can directly fetch the
inputs to process the images. In addition, we assume that the filter weights
are loaded into the system memory as a one-time operation.

SoC Performance Estimation. In the E2E autonomy paradigm, most of the time is
spent processing the neural network [19, 57, 49], and its performance will
dominate the overall throughput of the autonomy algorithm. For each frame from
the sensor, the E2E algorithm running on the accelerator sub-system generates
high-level action commands that the flight controller interprets to generate
low-level motor signals to control the UAV’s physical rotors.

For determining accelerator performance for running a given E2E autonomy
algorithm, AutoPilot uses SCALE-Sim, a configurable cycle-accurate systolic
array-based DNN accelerator simulator [67]. In addition, it exposes various
architectural parameters such as array size (number of MAC units), scratchpad
memory size for the input feature maps (ifmap), filters and output feature
maps (ofmaps), dataflow mapping strategies, as well as system integration
parameters, e.g., memory bandwidth. For example, enumerating # PE’s, SRAM
sizes give a nice trade-off between performance and power as shown in Fig. 5b.
Taking these architectural parameters, the SCALE-Sim generates the runtime
latency.

(a) E2E model template.

(b) Success Rate vs Model size.

Figure 4: (a) Parameterized End-to-End (E2E) model template. (b) E2E models
parameters vs. task-level success rate.

SoC Power Estimation. For estimating the total SoC power, we add the power of
individual components in the SoC. For estimating the power of the accelerator,
we run a given NN policy on a cycle-accurate simulator. The cycle-accurate
simulator produces SRAM traces, DRAM traces, number of read/write access to
SRAM, number of read/write access to the DRAM. Using the SRAM and DRAM trace
information, we model the SRAM power in CACTI [43] and DRAM power in Micron
DRAM model [10]. For estimating the power for the systolic array, we multiply
the array size with the energy of the PE. The PE power is modeled after the
breakdown in [42].

For the MCU cores, we use Cortex-M cores that implement the ARMv8-M ISA [5].
Each core consumes about 0.38 mW in a 28 nm process clocked at 100 MHz [4]. We
also account for the power of the ultra-low-power core into our final power
numbers. For the ULP camera, we assume it consumes 100 mW and form factor of
6.24 mm $\times$ 3.84 mm [3]. We account for the camera power in our overall
power calculation.

(a) Hardware accelerator template.

(b) Performance and power trade-offs.

Figure 5: (a) Accelerator template containing parameterized systolic array
template.(b) Varying of these parameters allows us to generate Pareto frontier
designs.

Bayesian Optimization. AutoPilot uses Bayesian optimization [31] for
algorithm-HW design space exploration. Bayesian optimization has shown to be
effective for optimizing black-box functions [72, 69] that are expensive to
evaluate and cannot be expressed as closed-form expressions.

In AutoPilot, we use bayesian optimization to optimize three objective
functions: (i) task success rate, (ii) SoC power, and (iii) accelerator
inference latency (runtime). A Pareto-optimal design achieves maximum task
success rate, minimum inference latency, and SoC power. The algorithm tunes NN
policy hyper-parameters (such as number of layers and filters) and accelerator
hardware parameters (e.g., number of processing elements, SRAM sizes) to
converge to Pareto-optimal NN policies and accelerator architectures.

We use an open-source BayesOpt implementation [31] is used in AutoPilot.
BayesOpt initially evaluates the objective functions at random parameters,
followed by intelligently selecting those that will optimize the objectives.
The algorithm builds a Bayesian statistical model for each objective function:
a Gaussian process (GP) is used. These GP models are updated as the BayesOpt
proceeds and samples new parameters. A GP distribution is defined by a mean
and covariance. The mean is the expected value of a function at some parameter
value. The covariance, called the kernel, models the dependence between the
function values at two distinct parameter values. In this paper, the widely-
used squared exponential (SE) kernel is used due to its simplicity, leading to
fast computation [60]. The right selection of parameter values is determined
by an acquisition function computed using the GP-predicted objective values.
The algorithm selects those inputs that maximize the acquisition function
until all the optimal solutions are found.

In particular, the S-Metric-Selection-based Efficient Global Optimization
(SMS-EGO) is used as the acquisition function. It has been shown to be highly
effective for multi-objective optimization and handling a large design space
compared to other acquisition strategies such as expected improvement [59].
SMS-EGO uses hypervolume to determine the degree to which a candidate point is
optimal: it is the volume enclosed between a candidate point and a fixed
reference point in the Pareto space. Since the hypervolume of a Pareto-optimal
point is higher than a non-optimal point, the approach maximizes the
hypervolume until all the points with the highest hypervolume (i.e, Pareto-
optimal) are found.

However, not all Pareto-optimal compute designs generated at this stage will
result in optimal UAV performance (e.g., maximize the number of missions).
Some of these designs will be over-provisioned or under-provisioned. Both
these scenarios negatively affect the overall UAV performance. Hence, to
determine which of these designs is better suited for a given UAV, we need to
performance full-system UAV co-design where we account for sensor, onboard
compute performance, and its impact on UAV physics.

###  3.3 Phase 3: Cyber-Physical System Co-Design

AutoPilot’s Phase 2 prunes a large design space of $\approx$1018 designs to
$\approx$100s of design candidates. These 100s of design candidates represent
a sample of low-power, high-performance, or Pareto-optimal designs in terms of
performance and power. However, enumerating 100s of design candidates manually
is still tedious and requires a systematic way of selecting one of the designs
for a UAV. The goal of Phase 3 is to determine the holistic evaluation of
these design candidates and other UAV components.

(a) Effect of compute weight.

(b) Optimal design.

Figure 6: (a) Mapping design candidates to F-1 to assess impact of compute
weight and compute throughput on UAV performance. (b) Mapping design
candidates to F-1 to select optimal design candidate for a UAV system.

Compute Weight Modelling. Since the payload weight affects the UAV physics, it
is important to estimate the weight of the onboard compute. The onboard
compute typically has two components: a motherboard where SoC is mounted and
other electrical components and a passive heatsink for cooling the SoC. The
heatsink weight is proportional to the TDP.

For the motherboard weight, we assume that the final SoC is mounted on a PCB
along with all electrical components weighing 20g (which per our analysis is
typical for Ras-Pi [9], CORAL [6] like systems). For the heatsink weight, we
use a heat sink calculator [7] which determines the heatsink volume required
for cooling. The weight of the heatsink is then determined by multiplying the
estimated volume with the density of aluminum (commonly used heatsink
material).

Cyber-Physical Co-Design Using F-1 Model. First, designs with the highest
success rate (based on the input specification) are filtered from the designs
generated in Phase 2. Next, all the filtered design candidates are mapped to
the F-1 UAV tradeoff model [40]. Each base UAV system has a unique F-1 plot
that can gather insights about different bounds and bottlenecks. F-1 [40]
model is a roofline-like visual performance model built on top of a safety
model [48] for UAVs.

The F-1 model plots the relationship between how fast but safely a UAV can
travel (safe velocity) based on its decision-making rate (output of sensor-
compute-control pipeline). Conversely, by relative motion (and switching the
frame of reference), if the UAV is hovering, the same model can also tell the
maximum velocity of an incoming object the UAV can avoid before colliding. In
addition, the model takes into account the compute throughput and considers
how the payload weight affects the UAV’s physics–i.e., maximum acceleration
(which can be determined by its thrust-to-weight ratio [55]). Another
important insight it can provide is whether a combination of autonomy
algorithm and onboard compute is over-provisioned, under-provisioned, or
optimal for a given UAV system.

To intuitively understand the role of F-1 in cyber-physical system co-design,
let us consider three hypothetical designs, ‘A,’ ‘B,’ and ’C’ generated from
Phase 2. Let us also assume that all the designs achieve the same compute
throughput but at different power (TDPs), with ‘A’ being lowest and ‘C’ being
the highest. Mapping these designs in the F-1 model (Fig. 6a) can give
insights into which of these three designs will achieve better UAV
performance. Since ‘A’ has the same compute performance with the lowest power,
it will also have the lowest heatsink weight, while ‘C’ has the highest power,
it will have the highest heatsink weight. The weight of the onboard compute
affects UAV ability to move faster, and the lowering of ceilings captures this
effect in Fig. 6a for ‘B’ and ‘C.’ Lowering of safe velocity will have an
implication on mission energy and the number of missions it can perform [17].
Thus, AutoPilot will select ‘A’ over the other two designs.

(a) Low-obstacle: Number of missions.

(b) Some-obstacle: Number of missions.

(c) Dense-obstacle: Number of missions.

Figure 7: Comparison of AutoPilot generated designs with other designs (TX2
and Xavier NX, PULP [57]) for three different deployment scenarios (low
obstacle, medium obstacles, and dense obstacle environments) and three drone
categories (mini-UAV, micro-UAV, and nano-UAVs). (a), (b), and (c) denotes the
mission the number of missions (higher is better) on a single battery charge
for three drone types across three different deployment scenario. The
degradation in the # of missions are compared with mean performance of TX2,
NX, and PULP [57] and annotated in the plots with AutoPilot generated design
as baseline. All the points except P-DroNet (PULP-DroNet) [57] runs the same
policy. For P-DroNet, we use the numbers reported from their work.

Another utility of the F-1 model in AutoPilot is determining if a design
candidate is optimal, over-provisioned, or under-provisioned. The minimum
value of action throughput to maximize safe velocity (Vsafe) is called the
knee-point. For example, in Fig. 6b, design ‘O’ is optimal because it achieves
the minimum action throughput to maximize Vsafe. Likewise, ‘A’ is over-
provisioned (achieves more throughput than required), and ‘X’ is under-
provisioned. AutoPilot will choose design ‘O’ over the other two designs.

Architectural Fine-Tuning. In the case when no optimal design exists that
achieves the knee-point, some architectural tuning may be required to shift
the design close to the knee-point. AutoPilot provides two options for which
points to consider for optimization: (i) these can be user-defined, or (ii)
the design point closest to the knee-point can be selected. Architectural
tuning can be performed using various optimizations until the optimized design
is at (or very close to) the base knee-point in the F-1 roofline. To this end,
we employ a “bag” of architectural optimizations in the tuning process and
seed AutoPilot with two techniques: frequency scaling and technology scaling.

##  4 Experimental Setup

This section describes the details of our setup to evaluate the efficacy of
AutoPilot in generalizing onboard compute design across many UAV types. We
also provide details about the UAV types, autonomy tasks, and sensor
information used to perform full-UAV system co-design.

###  4.1 Autonomy Task

We train autonomous navigation tasks in Air Learning [38] for three different
environments: low, medium, and dense-obstacles. Point-to-point navigation is
one of the key building blocks in achieving autonomy and is used in many
practical applications like search and rescue, source-seeking [19], and
package delivery. In the low-obstacle scenario, there are four obstacles with
goal position randomly changing every training episode. In the medium
scenario, there are four fixed obstacles and up to three randomly-placed
obstacles. In the dense obstacle scenario, there are four fixed obstacles in
the dense scenario and up to five randomly placed obstacles. Each E2E model is
trained for 1 Million steps or until convergence. This is a standard training
methodology for reinforcement learning, and the same methodology is used for
building real autonomous UAV applications [19, 66].

###  4.2 Base UAV Systems and Autonomy Components

To show the scalability of AutoPilot methodology, we take one representative
UAV from each size category: mini, micro, and nano-UAV (Table 2). The base UAV
system includes a frame, flight controller, battery, and rotors (all included
in the base weight). We keep the base UAV system fixed, and focus on designing
the optimal onboard compute and autonomy algorithm to maximize the overall
operational efficiency of the autonomous UAV system.

###  4.3 Autonomous UAV Evaluation Metrics

An important operational efficiency metric for autonomous drones is the
‘number of missions’, which captures how many times the drone can complete
similar missions on a single battery charge. For example, in a package
delivery use case, a higher number of missions means more packages delivered
with lower downtime spent recharging. This metric is affected by the choice of
drone onboard compute combined with several other key components.

For a given drone, we define the number of missions as:

$N_{missions}=\frac{E_{battery}}{E_{mission}}$ (1)

where $E_{battery}$ is the total energy available in the drone (a function of
battery mAh rating) and $E_{mission}$ is the total energy expended by the
drone per mission.

We can define $E_{mission}$ for a single mission as:

$E_{mission}=(P_{rotors}+P_{compute}+P_{others})*t_{mission}$ (2)

where $P_{rotors}$, $P_{compute}$, and $P_{others}$ refer to the power
consumption of the rotor propulsion, compute, and other electronic components
(e.g., sensors, ESC) in the drone. $t_{mission}$ is the time for completing
the mission. Intuitively, Eq 2 suggests that the amount of energy expended in
a mission corresponds to the duration of the time the drone flies (mission
time), and the total power dissipation of its components.

The mission time $t_{mission}$ depends upon the distance $D_{operation}$ and
the UAV velocity. For a fixed distance, mission time is determined by how fast
the drone can travel through a dynamic environment filled with obstacles. The
drone needs to travel as quickly as possible while safely navigating around
obstacles. We define this safe traveling speed as the safe velocity,
$V_{safe}$. Maximizing $V_{safe}$ lowers the mission time, increasing the
total possible number of missions.

Using these terms, we can re-write Eq 2 as follows:

$E_{mission}=(P_{rotors}+P_{compute}+P_{others})*\frac{D_{operation}}{V_{safe}},$
(3)

Then, substituting Eq 3 in Eq 1, we get:

$N_{missions}=\frac{E_{battery}*V_{safe}}{(P_{rotors}+P_{compute}+P_{others})*D_{operation}},$
(4)

According to Eq. 4, to maximize the number of missions, the optimization
objective is to increase the UAV’s safe velocity ($V_{safe}$) or increase the
battery capacity ($E_{battery}$). Increasing the battery capacity is non-
trivial since drone size impacts the SWaP constraints. However, proper
selection of various UAV components (compute, sensors, etc.) can maximize safe
velocity. It is important to note that $E_{mission}$, $V_{safe}$, and
$D_{operation}$ are not constants and it depends upon the mission
characteristics. But the fundamental relationship between them holds true.

Drone Name |  | Base-UAV System
---
(Fixed)
| Autonomy Components
---
(Custom Designed)
| UAV
---
Type
| Battery
---
Capacity
(mAh)
| Flight
---
Controller
| Base
---
UAV
Weight
Sensor |  | Sensor
---
Framerate
| Autonomy
---
Algorithm
| Onboard
---
Compute
AscTec Pelican | mini-UAV |  | 6250
---
(fixed)
| PID
---
Controller
100 KHz
1650 g | RGB | 30/60 FPS |  | E2E
---
(custom)
Custom
DJI Spark | micro-UAV |  | 1480
---
(fixed)
| PID
---
Controller
100 KHz
300 g | RGB | 30/60 FPS |  | E2E
---
(custom)
Custom
Zhang et al [79] | nano-UAV |  | 500
---
(fixed)
| PID
---
Controller
100 KHz
50 g | RGB | 30/60 FPS |  | E2E
---
(custom)
Custom

Table 2: In our experiments, we keep the base UAV system fixed (size, battery,
sensor type) and focus on co-design of components needed for achieving
autonomy.

##  5 Evaluation

This section evaluates AutoPilot’s co-design results for different UAVs and
deployment scenarios compared to baseline designs. We then compare AutoPilot
methodology in selecting designs versus several traditional design strategies.
Next, we characterize the effects of cyber-physical parameters such as sensor
type and UAV agility on the compute co-design process. Finally, we analyze the
cost of design specialization versus its impact on overall UAV mission
efficiency.

###  5.1 Co-Design for Different UAVs and Environments

In this experiment, we demonstrate that the AutoPilot methodology is _scalable
and generalizable_ in generating designs that maximize the number of missions
across three UAV types and three different deployment environment scenarios.

The _different classes of drones_ evaluated are a mini-UAV (AscTec Pelican), a
micro-UAV (DJI-Spark), and a nano-UAV (used in Zhang et al. [79]) whose
specifications are in Table 2. The _different deployment scenarios_ with
varying levels of complexity evaluated are: low, medium, and dense-obstacles
(details in Section 4.1). These auto-generated domain randomized environments
have varying degrees of obstacle densities. They represent common deployment
use cases for drones. For instance, autonomous navigation for a farming use
case could be very sparse (low-obstacle), whereas a search and rescue
operation in a forest is a dense-obstacle scenario.

The co-design goal is choosing an onboard compute system and autonomy
algorithm that minimizes the mission time and mission energy, maximizing the
number of missions the UAV can perform on a single battery charge (Eq. 1 and
Eq. 3).

Co-Design Comparison Results. Fig. 7 shows the comparison in the number of
missions between design generated using AutoPilot methodology and two general-
purpose selections (Jetson TX2 and Xavier NX) and PULP [57]. On a fully
charged battery, AutoPilot generated optimal designs on average achieves
1.33$\times$ to 1.43$\times$ more missions for mini-UAV compared to TX2,
Xavier NX, and PULP [57]. For micro-UAVs, AutoPilot on average achieves
1.34$\times$ to 1.62$\times$ . Likewise, for nano-UAV, AutoPilot on average
achieves 2.3$\times$ more missions, thus enabling higher operational
efficiency than an ad-hoc selection of general-purpose designs.

The AutoPilot methodology for the generation of onboard hardware and autonomy
algorithm for these UAV types and deployment scenarios consistently
outperforms general-purpose hardware (Jetson TX2 and Xavier NX) and a domain-
specific hardware accelerator built for UAVs [57]. This demonstrates the
flexibility of AutoPilot, compared to prior works that focus on a specific UAV
type [30, 73, 19, 57].

Figure 8: Visualization of architectural parameter variations for nine
scenarios (three UAVs and three deployment scenarios). The scales are
normalized with respect the lowest value for each architectural parameter to
show the variations requirements as the UAV components changes.

Analysis of Architectural Parameter Variations. Visualization of various
architectural parameters selections across all the nine scenarios (three UAVs
and three deployment scenarios) to gain insights into the designs that
AutoPilot generates is shown in Fig. 8. Deployment scenario affects the
complexity of the E2E models. A model with five layers and 32 filters achieves
the highest success rate for the low obstacle environment. For the dense
environment, the model with seven layers and 32 filters achieves the highest
success rate.

The increasing complexity of the autonomy algorithm means the onboard compute
for a specific UAV has to achieve similar performance while keeping the power
and weight constant. Using Bayesian optimization allows AutoPilot to select
architectural parameters (PE size, cache size, etc.) intelligently to satisfy
the performance, power, and weight constraints for the mini-UAV as shown in
Fig. 8. For instance, as the complexity of the task increases, the compute
design becomes bigger (as seen by the larger spread in parameter space Fig.
8). But AutoPilot only chooses those parameters to keep the power and weight
nearly the same (or never exceed a certain limit). Manually enumerating these
points (isolated or one-size accelerator designs) would not maximize the
mission performance as the UAV type or deployment scenario changes. Thus,
automating the co-design space intelligently allows AutoPilot to consistently
generate designs that maximize mission performance.

Figure 9:  Phase 2 Pareto frontier designs with different performance/power
profiles. We highlight four candidates (annotated as HP-high performance, LP-
low power, PO-Pareto optimal in terms of performance/power, and AP-Autopilot).

###  5.2 Comparison to Traditional Design Strategies

In the following experiments, we demonstrate the importance of performing a
full-system UAV co-design rather than selecting onboard compute based on
traditional design strategies: maximizing compute performance alone;
optimizing for low power alone; and even Pareto optimal selection with respect
to both compute performance and power efficiency. Our results show that
performing full-system UAV co-design is a necessity for maximizing overall
mission efficiency.

Ideally, what differentiates between full-system UAV co-design versus
methodologies outlined in prior work [61] is the critical Phase 3 in AutoPilot
(Fig. 3). Hence, we take an intermediate output from Phase 2 of AutoPilot
methodology and show how a lack of full-system co-design degrades the mission
performance of autonomous UAVs. To demonstrate the degradation concretely, we
choose a nano-UAV whose specifications are available in Table 2. The output of
Phase 2 (from Fig. 3) for this task are shown in Fig. 9.

Based on these strategies outlined above, we label these designs in Fig. 9 as
‘HP’ (high-performance design), ‘LP’ (low-power design), and ‘PO’ (Pareto
optimal design). We also label another design that does not use the
conventional selection strategies and instead uses AutoPilot methodology. We
label this point as ‘AP’ (AutoPilot selected design). Finally, we compare the
mission level performance of ‘AP’ and others to gain the following
architectural insights.

####  5.2.1 Comparison to Optimizing for Performance Alone

We compare the high-performance design (HP) with AutoPilot generated design
(AP) (Fig. 9) to show that _high performance compute design does not directly
translate to overall high autonomous UAV performance_. HP achieves a compute
throughput of 205 FPS while consuming 8.24 W (65 g) whereas AP achieves a
throughput of 46 FPS at 0.7 W (weight of 24 g). Comparing these two designs
head-on purely on isolated metrics we see that HP is 5.85$\times$ more
throughput than AP hence should be able to process the E2E autonomy algorithm
faster.

Fig. 10a, by contrast, shows the overall mission-level metrics of high
performance design (HP) and AutoPilot design (AP). We observe that AutoPilot
design has 2.25 $\times$ better mission-level metrics compared to high
performacne design. The answer to why we see the degradation if we select HP
instead of AP lies in the effect of compute weight and its impact on UAVs
physics. A full-system UAV co-design (Phase 3 of AutoPilot) accounts for these
effects while selecting onboard compute design for autonomous UAVs.

(a) # Missions.

(b) Mapping AP and HP to the F-1 plot.

Figure 10: (a) Comparison of high performance (HP) and AutoPilot design (AP)
in terms of the number of missions. (b) F-1 plot for nano-UAV (Table 2) to
understand the degradation when using HP over AP.

Why High-Performance Alone Falls Short? To help explain the degradation in
performance, we manually plot these design candidates on the F-1 model [40]
for the nano-UAV (Table 2). Recall from Section 3.3 that F-1 is used to
understand what is an optimal design for a given UAV and also shows the
effects of performance of sensor-compute-control pipeline and how the weight
of payload affects the UAV physics.

Since the high performance design (HP) consumes 11.7 $\times$ more power than
AutoPilot design (AP), the heatsink needed to cool HP is also larger what is
required to cool AP system. This adds to the overall weight of the HP system
which lowers the UAV ability to move faster (and safely). This effect is
captured in F-1 plot shown in Fig. 10b. Due to the reduction in safe velocity,
it affects the mission time, mission energy, and number of missions (Eq. 4) of
the UAV.

Choosing a onboard compute performance based solely on compute throughput can
be misleading and can even deteriorate the mission performance. Therefore, a
full-system UAV co-design methodology is necessary to avoid these pitfalls.

####  5.2.2 Comparison to Optimizing for Low Power Alone

The motivation of choosing low power design comes from Eq. 2 where compute
power plays a role in overall energy. We compare the low power design (LP)
with AutoPilot design (AP) to show that _low power design does not reduce the
overall mission energy_ and in turn has implications on number of missions the
UAV can achieve on a single battery charge cycle.

Fig. 11a shows the mission level metrics of LP and AP. We observe that AP
achieves 1.8$\times$ more missions compared to LP. The answer to why we see
this degradation lies in the slow decision-making rate of the autonomous UAV.
A full system UAV co-design (Phase 3) considers these effects while selecting
onboard compute for UAV.

(a) # Missions.

(b) Mapping LP and HP to the F-1 plot.

Figure 11: (a) Comparison of LP and AP in terms of number of missions. (b) F-1
plot for nano-UAV (Table 2) to understand the degradation when using LP over
AP.

Why Low Power Alone Falls Short? To understand the degradation, we manually
map low power design (LP) and AutoPilot design (AP) designs on the F-1 model
for the nano-UAV as shown in Fig. 11b. Recall that the knee-point is the
minimum action throughput required to maximize the safe velocity of the UAV.
In the case of the nano-UAV, it is 46 FPS. In contrast, the LP design achieves
an action throughput of which is 2.5$\times$ lower than what the nano-UAV’s
physics can allow. Due to lower decision-making rate for LP, the nano-UAV
cannot fly safely without lowering its velocity, which affects the mission
energy (Eq. 2).

Therefore, we conclude that choosing a low-power onboard compute does not
necessarily lower the mission energy, since decision making rate (action
throughput) also plays a role in UAV ability to fly faster, which can lower
mission time and overall mission energy. Thus, compromising performance for
low pow consumption can degrade mission performance and full-system UAV co-
design is necessary to ensure optimal onboard compute is selected for a given
UAV system.

####  5.2.3 Comparison to Pareto Optimal Performance-Power

Design methodologies co-designing HW-SW together [61] focus on selecting
Pareto optimal designs to maximize energy efficiency. However, when designing
a computing system for a domain-specific application (e.g., autonomous UAVs),
these isolated methodologies do not maximize UAV performance.

To demonstrate this, we compare the Pareto optimal (PO) design with the
AutoPilot design (AP) to show that _Pareto optimal design with respect to
compute performance/power may not be optimal overall_ in maximizing autonomous
UAV mission performance. Fig. 12a shows the mission level metrics of PO and AP
designs. We observe that AP achieves 1.3$\times$ more number of missions
compared to PO. However, despite achieving higher energy efficiency (53 FPS/W)
compared to AP (50 FPS/W), we still observe degradation in mission metrics
when selecting PO. The answer to why we see these degradation lies how PO
affects the autonomous UAV physics.

Why Pareto Optimal Performance-Power Falls Short? The F-1 plot with both
AutoPilot design and Pareto Optimal design (PO) is shown in Fig. 12b. The
knee-point design for the nano-UAV is around 46 FPS, whereas the PO achieves a
throughput of 96 FPS at 1.8 W (over-provisioned by 2$\times$). The over-
provisioned PO design also consumes relatively higher power. Thus, the
heatsink required to cool PO design will be higher than AP, thus increasing
the compute weight for the PO design. Recall that increasing payload weight
lowers the UAV’s ability to move faster (Fig. 6a), thus lowering the safe
velocity (captured by the F-1 model ceilings), which in turn lowers the
mission energy and the number of missions (Eq 4).

(a) # Missions.

(b) Mapping LP and HP to the F-1 plot.

Figure 12: (a) Comparison of PO and AP in terms the number of missions. (b)
F-1 plot for nano-UAV (Table 2) to understand the degradation when using PO
over AP.

###  5.3 Effect of Sensor Performance on Compute Selection

This section shows sensor performance affects the selection of onboard
compute. Design or selection of onboard compute for autonomous UAV _must
consider sensor performance_. Failing to account for sensor performance can
cause degradation in overall autonomous mission performance.

To demonstrate this, we take AscTec Pelican (mini-UAV) with a 30 FPS and 60
FPS sensor for the autonomous navigation task in a medium obstacle density
environment. For both these configurations, AutoPilot accounts for the sensor
performance and designs an onboard compute to match the sensor performance.
For example, the AutoPilot selects AP30FPS as the onboard compute for a mini-
UAV with 30 FPS sensor. As a result, it achieves 30 FPS compute throughput
while consuming 0.88 W. Likewise, AP60FPS is the onboard compute for mini-UAV
with 60 FPS sensor. As a result, it achieves 47 FPS (knee-point for this UAV
is 46 FPS) at 2.12 W.

Let us also assume that for the UAV with 60 FPS sensor, we still use AP30FPS
instead of AP60FPS. And conversely, for 30 FPS sensors, we use AP60FPS design.
Fig. 13a shows the correlation matrix with these combinations. The diagonal of
the correlation matrix corresponds to the scenarios where the onboard compute
selection accounts for sensor performance. The rows correspond to the sensor
framerates (30/60 FPS), whereas the column corresponds to the compute design
selections (AP∗). The value inside the correlation matrix is the number of
missions the UAV can achieve.

Effect of Under-Provisioned Compute for Sensor. In this scenario, we use
AP30FPS design for the mini-UAV with 60 FPS sensors. Using this design point
in the mini-UAV, we observe that the overall number of missions drops from 71
missions to 61 missions. To understand the reason for degradation, we map the
sensor (60 FPS) and compute design (AP30FPS) to the F-1 model [40] for the
AscTec pelican UAV as shown in Fig. 13b. Since the knee-point of the UAV is
around 45 FPS, the AP30FPS design is left of the knee-point making this design
compute-bound. Even though both the UAV physics and sensor can allow the UAV
to fly faster, the onboard computer’s inability to make decisions faster than
30 FPS limits its ability to move faster. A slower decision-making rate has
implications on UAV’s safe velocity which affects the mission time, energy,
and in turn, the number of missions (Eq. 4).

###  5.4 Effect of UAV Agility on Compute Selection

In this section, we demonstrate how UAV’s agility impacts the onboard compute
requirement to run the autonomy algorithms. UAV’s agility is typically
characterized by its maximum acceleration (which depends upon the UAV’s
thrust-to-weight ratio) [23, 55]. It is important to note that as the payload
weight (onboard compute, heatsink etc) increases, it lowers the thrust-to-
weight ratio (lowers maximum acceleration) which in turn makes the UAV less
agile. Hence, there is a _need to carefully account for these physical effects
when designing onboard compute_ for these different classes of UAVs.

(a) Number of missions.

(b) 60 FPS sensor.

Figure 13: Designing compute without considering the sensor performance
degrades mission performance.

To demonstrate the increase in compute requirement with UAV’s agility, we take
two UAV’s namely DJI-spark and nano-UAV [79]. The specification of these UAVs
is in Table 2. The autonomy task for both these drones is point-to-point
navigation in a medium obstacle environment. We assume that both the UAVs are
equipped with 60 FPS sensors (to ensure that the sensor’s performance does not
bound them).

Fig. 14a shows the mapping of base-UAV configuration of both the UAVs to the
F-1 model [40]. The optimal compute throughput required to maximize the
mission performance for the DJI-spark is around 27 Hz, whereas, for the nano-
UAV, it is 46 Hz. This suggests that the sensor-compute-control pipeline needs
to make decisions at this rate (27 FPS for DJI-spark and 46 FPS for nano-UAV)
to maximize the safe velocity.

Since AutoPilot uses performs full-system UAV co-design, it selects the design
candidates closer to the optimal point in the F-1 model. For the nano-UAV,
AutoPilot chooses design 2$\times$ more compute throughput than the design
point it chooses for DJI-spark without affecting the UAV physics (by extra
payload/heatsink weight). These design points are annotated as ‘AP-micro’ and
‘AP-nano’ in Fig. 14b.

As UAVs become more agile and smaller, the role of onboard compute increases.
However, minaturization also makes it susceptible to payload changes, making
it challenging to design onboard compute to maximize the autonomous UAVs’
performance. In these scenarios, AutoPilot methodology is advantageous since
it performs full-system UAV co-design to generate design points that maximize
mission performance without impacting the UAV’s physics.

(a) Number of missions.

(b) 60 FPS sensor.

Figure 14: UAV agility increases compute requirement.

##  6 Specialization Cost vs. Mission Efficiency

As we show, when the UAV components change, we see the need for re-optimizing
(or a new custom design). Customization for UAVs is a trade-off between its
operational efficiency and design effort (cost). We quantify the trade-off
between using single designs versus domain-specific hardwares.

In this case, we take the optimal point designed for a medium obstacle density
scenario for a mini-UAV. We compare this design point against general-purpose
design (e.g., TX2 and Intel NCS) and optimal specialized HW accelerators
designed for other scenarios (e.g., Knee-point designs of Low-obs and Dense-
obs but reused elsewhere). Table 3 tabulates the cost of these trade-offs.
Compared to the deployment-specific hardware specialization, we observe 27% to
67% degradation in the number of missions possible depending upon the choice
of onboard compute. Hence, if operational efficiency is important, then
deployment-specific hardware specialization is key to achieving that goal.
However, if cost is critical, then general-purpose designs (or reusing single
designs) can save design cost with a 27% to 67% reduction in its number of
missions (which can also increase the operational cost for grounding and
recharging the drones frequently).

In summary, the trade-off between mission efficiency and the cost of computing
exists, but the design complexity (cost) can be reduced significantly by
AutoPilot methodology (automating the complex cyber-physical co-design).

Metrics | Single ASIC Designs | General Purpose Designs
---|---|---
| Knee-Point
---
(Low Obs.)
| Knee-Point
---
(Med Obs.)
| Knee-Point
---
(Dense Obs.)
| Nvidia
---
TX2
| Intel
---
NCS
| # of Mission
---
Degradation
30 % | 0% | 27 % | 30 % | 67 %
Comments |  | Compute Bound
---
lowers Vsafe
| Optimal
---
Design
| Weight lowers
---
the roofline
| Weight lowers
---
the roofline
| Compute Bound
---
lowers Vsafe

Table 3: Design Trade-off comparisons.

##  7 Related Work

Robot accelerator design is an area of emerging interest. Recent work proposed
a low-power accelerator [57] for neural network-based control, but it is only
targeted to nano-drones running DroNet [49]. Our work, on the other hand,
provides a general methodology to generate multiple NN policies and hardware
accelerator designs from a high-level specification. Navion [73] is a
specialized accelerator for improving visual-inertial-odometry in aerial
robots, using the conventional sense-plan-act control paradigm. Instead, with
AutoPilot we focus on end-to-end learning-based control, a promising emerging
autonomy paradigm. RoboX [65] generates an accelerator for model predictive
control from a high-level DSL. Though the high-level goal of AutoPilot is
similar, our work differs because RoboX does not consider the effect of the
cyber-physical parameters on the computing platform, whereas AutoPilot uses
the cyber-physical F-1 model to quantify design optimality. Hivemind [32] is
the first system to provide swarm-based serverless infrastructure for
continuous learning on edge. In contrast, the methodology used in our work
shows how to build onboard compute for a single UAV efficiently while
considering a full-system view of the UAV. Therefore, we believe our
methodology to maximize the efficiency of a single UAV will also scale up the
overall efficiency of UAV swarm systems. Outside of the domain of aerial
robots, there has been work [54, 53, 56] showing the benefits of designing
custom hardware accelerators for motion planning. In particular, Robomorphic
Computing [56] also provides a general methodology to synthesize custom
hardware based on robot parameters (e.g., joint constraints). However, it
currently focuses only on motion planning for articulated robots (e.g., arms),
and motion planning is only one of many stages required to achieve autonomy.
By contrast, AutoPilot provides full end-to-end autonomy for drones.

##  8 Conclusion

AutoPilot is a push-button solution that automates cyber-physical co-design to
automatically generate an optimal autonomy algorithm (E2E Model) and its
hardware accelerator from a high-level user specification. It can be adapted
to perform full-system co-design for the Sense-Plan-Act (SPA) paradigm. The
only requirement for AutoPilot is that the SPA-based algorithm and hardware
templates be parameterizable. Moreover, the general methodology we have
developed for AutoPilot, such as cyber-physical co-design for identifying the
optimal design point, architectural fine-tuning, and selecting the optimal
design points by showing how it affects the overall mission can also be
adapted to other types of autonomous vehicles such as self-driving cars and
“ground” drones.

## References

  * [1] https://www.bitcraze.io/products/crazyflie-2-1/, title = CrazyFlie 2.1 Product Page, year = 2020.
  * [2] https://github.com/harvard-edge/airlearning, title = Air Learning Environment Generator, year = 2020.
  * [3] https://www.ovt.com/sensors/OV9755, title = OV9755,Color CMOS 720p (1280x720) HD Image Sensor with OmniPixel®3-HS Technology, year = 2020.
  * [4] Arm Cortex-M33.
  * [5] ARMv8-M Technical Reference manual.
  * [6] Coral-som-datasheet, howpublished = https://coral.ai/static/files/coral-som-datasheet.pdf, month = , year = , note = .
  * [7] Heat sink size calculator.  https://celsiainc.com/resources/calculators/heat-sink-size-calculator/.  (Accessed on 01/29/2020).
  * [8] Nvidia deep learning inference platform performance study, howpublished=https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/tesla-product-literature/t4-inference-print-update-inference-tech-overview-final.pdf.
  * [9] Raspberry pi.  https://www.pololu.com/blog/598/new-product-raspberry-pi-3-model-b.
  * [10] Micron ddr4 power calculator.  https://www.micron.com/~/media/documents/products/power-calculator/ddr4_power_calc.xlsm, 2016\.
  * [11] All about multirotor drone fpv flight controllers.  https://www.getfpv.com/learn/new-to-fpv/all-about-multirotor-fpv-drone-flight-controller/, 2020\.
  * [12] Pixhawk 1 flight controller.  https://docs.px4.io/v1.9.0/en/flight_controller/pixhawk.html, 2020\.
  * [13] Markus Achtelik, Tianguang Zhang, Kolja Kuhnlenz, and Martin Buss.  Visual tracking and control of a quadcopter using a stereo camera system and inertial sensors.  In 2009 International Conference on Mechatronics and Automation, pages 2863–2869. IEEE, 2009.
  * [14] Aqeel Anwar and Arijit Raychowdhury.  Autonomous navigation via deep reinforcement learning for resource constraint edge nodes using transfer learning.  IEEE Access, 8:26549–26560, 2020.
  * [15] M. Bacco, A. Berton, E. Ferro, C. Gennaro, A. Gotta, S. Matteoli, F. Paonessa, M. Ruggeri, G. Virone, and A. Zanella.  Smart farming: Opportunities, challenges and technology enablers.  In 2018 IoT Vertical and Topical Summit on Agriculture - Tuscany (IOT Tuscany), pages 1–6, May 2018.
  * [16] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al.  End to end learning for self-driving cars.  arXiv preprint arXiv:1604.07316, 2016.
  * [17] Behzad Boroujerdian, Hasan Genc, Srivatsan Krishnan, Wenzhi Cui, Aleksandra Faust, and Vijay Janapa Reddi.  Mavbench: Micro aerial vehicle benchmarking.  In Proceedings of the 51st Annual IEEE/ACM International Symposium on Microarchitecture, MICRO-51, page 894–907. IEEE Press, 2018.
  * [18] MWM Gamini Dissanayake, Paul Newman, Steve Clark, Hugh F Durrant-Whyte, and Michael Csorba.  A solution to the simultaneous localization and map building (slam) problem.  IEEE Transactions on robotics and automation, 17(3):229–241, 2001\.
  * [19] Bardienus Pieter Duisterhof, Srivatsan Krishnan, Jonathan J. Cruz, Colby R. Banbury, William Fu, Aleksandra Faust, Guido C. H. E. de Croon, and Vijay Janapa Reddi.  Tiny robot learning (tinyrl) for source seeking on a nano quadcopter.  In IEEE International Conference on Robotics and Automation (ICRA), 2021.
  * [20] Bardienus Pieter Duisterhof, Shushuai Li, Javier Burgués, Vijay Janapa Reddi, and Guido C. H. E. de Croon.  Sniffy bug: A fully autonomous swarm of gas-seeking nano quadcopters in cluttered environments.  CoRR, abs/2107.05490, 2021.
  * [21] Alberto Elfes.  Using occupancy grids for mobile robot perception and navigation.  Computer, 22(6):46–57, 1989.
  * [22] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter.  Neural architecture search: A survey.  The Journal of Machine Learning Research, 20(1):1997–2017, 2019\.
  * [23] Davide Falanga, Suseong Kim, and D. Scaramuzza.  How fast is too fast? the role of perception latency in high-speed sense and avoid.  IEEE Robotics and Automation Letters, 4:1884–1891, 2019.
  * [24] Aleksandra Faust, Oscar Ramirez, Marek Fiser, Ken Oslund, Anthony Francis, James Davidson, and Lydia Tapia.  Prm-rl: Long-range robotic navigation tasks by combining reinforcement learning and sampling-based planning.  In IEEE International Conference on Robotics and Automation (ICRA), pages 5113–5120, Brisbane, Australia, 2018.
  * [25] Tian Gao, Zishen Wan, Yuyang Zhang, Bo Yu, Yanjun Zhang, Shaoshan Liu, and Arijit Raychowdhury.  ielas: An elas-based energy-efficient accelerator for real-time stereo matching on fpga platform.  In 2021 IEEE 3rd International Conference on Artificial Intelligence Circuits and Systems (AICAS), pages 1–4. IEEE, 2021.
  * [26] A. Giusti, J. Guzzi, D. C. Cireşan, F. He, J. P. Rodríguez, F. Fontana, M. Faessler, C. Forster, J. Schmidhuber, G. D. Caro, D. Scaramuzza, and L. M. Gambardella.  A machine learning approach to visual perception of forest trails for mobile robots.  IEEE Robotics and Automation Letters, 1(2):661–667, 2016.
  * [27] D. Gonzalez, J. Perez, V. Milanes, and F. Nashashibi.  A review of motion planning techniques for automated vehicles.  IEEE Transactions on Intelligent Transportation Systems, 17(4):1135–1145, 2016.
  * [28] Tong GUO, Nan JIANG, Biyue LI, Xi ZHU, Ya WANG, and Wenbo DU.  Uav navigation in high dynamic environments: A deep reinforcement learning approach.  Chinese Journal of Aeronautics, 2020.
  * [29] Daniel Gurdan, Jan Stumpf, Michael Achtelik, Klaus-Michael Doth, Gerd Hirzinger, and Daniela Rus.  Energy-efficient autonomous four-rotor flying robot controlled at 1 khz.  In Proceedings 2007 IEEE International Conference on Robotics and Automation, pages 361–366. IEEE, 2007.
  * [30] Ramyad Hadidi, Bahar Asgari, Sam Jijina, Adriana Amyette, Nima Shoghi, and Hyesoon Kim.  Quantifying the design-space tradeoffs in autonomous drones.  In Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2021, page 661–673, New York, NY, USA, 2021. Association for Computing Machinery.
  * [31] Marton Havasi and Jose Miguel Lobato.  Bayesian optimization.  https://github.com/cambridge-mlg/gem5-aladdin/tree/master/bo_script, 2018\.
  * [32] Justin Hu, Ariana Bruno, Brian Ritchken, Brendon Jackson, Mateo Espinosa, Aditya Shah, and Christina Delimitrou.  Hivemind: A scalable and serverless coordination control platform for UAV swarms.  CoRR, abs/2002.01419, 2020.
  * [33] Intel.  Mobileye and nio partner to bring level 4 autonomous vehicles to consumers in china and beyond.  https://newsroom.intel.com/news/mobileye-nio-partner-bring-level-4-autonomous-vehicles-consumers-china-beyond/#gs.3l52m2.
  * [34] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al.  Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation.  arXiv preprint arXiv:1806.10293, 2018.
  * [35] Sertac Karaman and Emilio Frazzoli.  Sampling-based algorithms for optimal motion planning.  The international journal of robotics research, 30(7):846–894, 2011\.
  * [36] Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah.  Learning to drive in a day.  2019 International Conference on Robotics and Automation (ICRA), pages 8248–8254, 2019.
  * [37] William Koch, Renato Mancuso, and Azer Bestavros.  Neuroflight: Next generation flight control firmware.  arXiv preprint arXiv:1901.06553, 2019.
  * [38] Srivatsan Krishnan, Behzad Boroujerdian, William Fu, Aleksandra Faust, and Vijay Janapa Reddi.  Air learning: a deep reinforcement learning gym for autonomous aerial robot visual navigation.  In Machine Learning (Special Issue on Reinforcement Learning for Real Life), pages 1–40. Springer, 2021.
  * [39] Srivatsan Krishnan, Sharad Chitlangia, Maximilian Lam, Zishen Wan, Aleksandra Faust, and Vijay Janapa Reddi.  Quantized reinforcement learning (quarl), 2019.
  * [40] Srivatsan Krishnan, Zishen Wan, Kshitij Bhardwaj, Paul Whatmough, Aleksandra Faust, Gu-Yeon Wei, David Brooks, and Vijay Janapa Reddi.  The sky is not the limit: A visual performance model for cyber-physical co-design in autonomous machines.  IEEE Computer Architecture Letters, 19(1):38–42, 2020.
  * [41] Sven Lange, Niko Sünderhauf, Peer Neubert, Sebastian Drews, and Peter Protzel.  Autonomous corridor flight of a uav using a low-cost and light-weight rgb-d camera.  In Advances in Autonomous Mini Robots, pages 183–192. Springer, 2012.
  * [42] H. Li, M. Bhargav, P. N. Whatmough, and H. . Philip Wong.  On-chip memory technology design space explorations for mobile deep neural network accelerators.  In 2019 56th ACM/IEEE Design Automation Conference (DAC), pages 1–6, June 2019.
  * [43] Sheng Li, Ke Chen, Jung Ho Ahn, Jay B Brockman, and Norman P Jouppi.  Cacti-p: Architecture-level modeling for sram-based structures with advanced leakage reduction techniques.  In Proceedings of the International Conference on Computer-Aided Design, pages 694–701. IEEE Press, 2011.
  * [44] Oscar Liang.  Esc for racing drones & mini quad.  https://bit.ly/uav-count.
  * [45] Oscar Liang.  Fpv cameras for racing drone & mini quad.  https://bit.ly/sensor-count.
  * [46] K. Lim, G. S. Kim, S. Kim, and K. Baek.  A multi-lane mipi csi receiver for mobile camera applications.  IEEE Transactions on Consumer Electronics, 56(3):1185–1190, Aug 2010.
  * [47] Shaoshan Liu, Zishen Wan, Bo Yu, and Yu Wang.  Robotic computing on fpgas.  Synthesis Lectures on Computer Architecture, 16(1):1–218, 2021\.
  * [48] Sikang Liu, Michael Watterson, Sarah Tang, and Vijay Kumar.  High speed navigation for quadrotors with limited onboard sensing.  In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 1484–1491. IEEE, 2016.
  * [49] Antonio Loquercio, Ana I Maqueda, Carlos R Del-Blanco, and Davide Scaramuzza.  Dronet: Learning to fly by driving.  IEEE Robotics and Automation Letters, 3(2):1088–1095, 2018.
  * [50] Mariana de Jesús Marcial-Pablo, Alberto Gonzalez-Sanchez, Sergio Iván Jimenez-Jimenez, Ronald Ernesto Ontiveros-Capurata, and Waldo Ojeda-Bustamante.  Estimation of vegetation fraction using rgb and multispectral images from uav.  International journal of remote sensing, 40(2):420–438, 2019.
  * [51] Kartik Mohta, Michael Watterson, Yash Mulgaonkar, Sikang Liu, Chao Qu, Anurag Makineni, Kelsey Saulnier, Ke Sun, Alex Zhu, Jeffrey Delmerico, et al.  Fast, autonomous flight in gps-denied and cluttered environments.  Journal of Field Robotics, 35(1):101–120, 2018.
  * [52] Alec Momont.  Ambulance drone.  https://www.tudelft.nl/en/ide/research/research-labs/applied-%****␣main-paper.bbl␣Line␣325␣****labs/ambulance-drone/.
  * [53] Sean Murray, Will Floyd-Jones, Ying Qi, Daniel J Sorin, and George Dimitri Konidaris.  Robot motion planning on a chip.  In Robotics: Science and Systems, 2016.
  * [54] Sean Murray, William Floyd-Jones, Ying Qi, George Konidaris, and Daniel J Sorin.  The microarchitecture of a real-time robot motion planning accelerator.  In 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 1–12. IEEE, 2016.
  * [55] NASA.  Thrust to weight ratio.  https://www.grc.nasa.gov/www/k-12/airplane/fwrat.html.
  * [56] Sabrina M. Neuman, Brian Plancher, Thomas Bourgeat, Thierry Tambe, Srinivas Devadas, and Vijay Janapa Reddi.  Robomorphic computing: A design methodology for domain-specific accelerators parameterized by robot morphology.  In Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2021, page 674–686, New York, NY, USA, 2021. Association for Computing Machinery.
  * [57] Daniele Palossi, Antonio Loquercio, Francesco Conti, Eric Flamand, Davide Scaramuzza, and Luca Benini.  A 64mw dnn-based visual navigation engine for autonomous nano-drones.  IEEE Internet of Things Journal, 2019.
  * [58] Paul P.Lee, Lawrence J. Bernstein, Robert M. Guidash, and Teh-Hsuang Lee.  Integrated cmos active pixel digital camera, 2004.
  * [59] W. Ponweiser et al.  Multiobjective optimization on a limited budget of evaluations using model-assisted S-Metric selection.  In PPSN, pages 784–794, 2008.
  * [60] C. Rasmussen and C. Williams.  Gaussian processes for machine learning.  MIT press, Cambridge, MA, 2005.
  * [61] B. Reagen et al.  A case for efficient accelerator design space exploration via Bayesian optimization.  In ISLPED, 2017.
  * [62] James Rogers.  How drones are helping the nepal earthquake relief effort.  http://www.foxnews.com/tech/2015/04/30/how-drones-are-helping-nepal-earthquake-relief-effort.html.
  * [63] Stéphane Ross, Narek Melik-Barkhudarov, Kumar Shaurya Shankar, Andreas Wendel, Debadeepta Dey, J Andrew Bagnell, and Martial Hebert.  Learning monocular reactive uav control in cluttered natural environments.  In 2013 IEEE international conference on robotics and automation, pages 1765–1772. IEEE, 2013.
  * [64] Radu Bogdan Rusu and Steve Cousins.  3d is here: Point cloud library (pcl).  In 2011 IEEE international conference on robotics and automation, pages 1–4. IEEE, 2011.
  * [65] Jacob Sacks, Divya Mahajan, Richard C Lawson, and Hadi Esmaeilzadeh.  Robox: an end-to-end solution to accelerate autonomous control in robotics.  In Proceedings of the 45th Annual International Symposium on Computer Architecture, pages 479–490. IEEE Press, 2018.
  * [66] Fereshteh Sadeghi and Sergey Levine.  Cad2rl: Real single-image flight without a single real image.  In Robotics: Science and Systems Conference. IEEE, 2017.
  * [67] Ananda Samajdar, Jan Moritz Joseph, Yuhao Zhu, Paul Whatmough, Matthew Mattina, and Tushar Krishna.  A systematic methodology for characterizing scalability of dnn accelerators using scale-sim.  In 2020 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 58–68. IEEE, 2020.
  * [68] Nitin J. Sanket, Chethan M. Parameshwara, Chahat Deep Singh, Ashwin V. Kuruttukulam, Cornelia Fermüller, Davide Scaramuzza, and Yiannis Aloimonos.  Evdodge: Embodied AI for high-speed dodging on A quadrotor using event cameras.  CoRR, abs/1906.02919, 2019.
  * [69] B. Shahriari et al.  Taking the human out of the loop: a review of Bayesian optimization.  Proceedings of the IEEE, pages 148–175, 2016.
  * [70] Nikolai Smolyanskiy, Alexey Kamenev, Jeffrey Smith, and Stan Birchfield.  Toward low-flying autonomous mav trail navigation using deep neural networks for environmental awareness.  In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4241–4247. IEEE, 2017.
  * [71] Nikolai Smolyanskiy, Alexey Kamenev, Jeffrey Smith, and Stan Birchfield.  Toward low-flying autonomous mav trail navigation using deep neural networks for environmental awareness.  In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4241–4247. IEEE, 2017.
  * [72] J. Snoek, H. Larochelle, and R. P. Adams.  Practical Bayesian optimization of machine learning algorithms.  In NIPS, pages 2960–2968, 2012.
  * [73] Amr Suleiman, Zhengdong Zhang, Luca Carlone, Sertac Karaman, and Vivienne Sze.  Navion: A 2-mw fully integrated real-time visual-inertial odometry accelerator for autonomous navigation of nano drones.  IEEE Journal of Solid-State Circuits, 54(4):1106–1119, 2019.
  * [74] Tesla.  Tesla‘s autopilot.  https://www.tesla.com/autopilotAI.
  * [75] Amukele Timothy, M. Ness Paul, A.R. Tobian Aaron, Boyd Joan, and Street Jeff.  Drone transportation of blood products.  TRANSFUSION Journal, 57(3):582–588, 2017.
  * [76] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel.  Domain randomization for transferring deep neural networks from simulation to the real world.  In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 23–30. IEEE, 2017.
  * [77] Zishen Wan, Bo Yu, Thomas Yuang Li, Jie Tang, Yuhao Zhu, Yu Wang, Arijit Raychowdhury, and Shaoshan Liu.  A survey of fpga-based robotic computing.  IEEE Circuits and Systems Magazine, 21(2):48–74, 2021.
  * [78] Amir Yazdanbakhsh, Christof Angermüller, Berkin Akin, Yanqi Zhou, Albin Jones, Milad Hashemi, Kevin Swersky, Satrajit Chatterjee, Ravi Narayanaswami, and James Laudon.  Apollo: Transferable architecture exploration.  CoRR, abs/2102.01723, 2021.
  * [79] Xu Zhang, Bin Xian, Bo Zhao, and Yao Zhang.  Autonomous flight control of a nano quadrotor helicopter in a gps-denied environment using on-board vision.  IEEE Transactions on Industrial Electronics, 62(10):6392–6403, 2015\.