# Back to Newton’s Laws: Learning Vision-based Agile Flight via Differentiable
Physics

Yuang Zhang1†, Yu Hu1†, Yunlong Song†, Danping Zou1∗, Weiyao Lin1∗
1Shanghai Jiao Tong University, Shanghai, China
†Equal contribution ∗Corresponding Author

###### Abstract

Swarm navigation in cluttered environments is a grand challenge in robotics.
This work combines deep learning with first-principle physics through
differentiable simulation to enable autonomous navigation of multiple aerial
robots through complex environments at high speed. Our approach optimizes a
neural network control policy directly by backpropagating loss gradients
through the robot simulation using a simple point-mass physics model and a
depth rendering engine. Despite this simplicity, our method excels in
challenging tasks for both multi-agent and single-agent applications with
zero-shot sim-to-real transfer. In multi-agent scenarios, our system
demonstrates self-organized behavior, enabling autonomous coordination without
communication or centralized planning—an achievement not seen in existing
traditional or learning-based methods. In single-agent scenarios, our system
achieves a 90% success rate in navigating through complex environments,
significantly surpassing the 60% success rate of the previous state-of-the-art
approach. Our system can operate without state estimation and adapt to dynamic
obstacles. In real-world forest environments, it navigates at speeds up to 20
m/s, doubling the speed of previous imitation learning-based solutions.
Notably, all these capabilities are deployed on a budget-friendly $21
computer, costing less than 5% of a GPU-equipped board used in existing
systems. Video demonstrations are available at https://youtu.be/LKg9hJqc2cc.

###### Index Terms:

Vision-based Flight, Aerial Robotics, Differentiable Physics

##  I Introduction

Modern aerial robotics are increasingly being called upon to execute complex,
agile maneuvers in dynamically changing and unknown environments, ranging from
search-and-rescue operations [1] to powerline inspections [2] or deliveries
[3]. They are required to operate with extremely limited sensory and
computational resources onboard. The key to overcoming this challenge is
tightly integrating efficient, robust perception systems with adaptive control
mechanisms. However, existing methods [4, 5, 6, 7, 8, 9] for this problem
still lacking in terms of robustness, flexibility, and scalability. Achieving
agile vision-based flight in complex environments with extremely limited
sensory and computational resources remains an open research problem in
robotics.

Classical approaches predominantly rely on mapping-based methods for
navigation[5, 6]. These methods typically divide the navigation task into four
separate stages: localization, mapping, planning, and control. Such methods
have achieved impressive results in research labs and real-world applications,
from exploring Mars to swarm navigation in dense forests [7]. However, the
cascade structure introduces latency and error accumulation, limiting their
scalability to high-speed flight [8, 9]. Furthermore, localization and mapping
can be erratic at high speeds and suffer from high computational costs [10,
11, 12].

In contrast to the traditional pipeline, recent studies propose to learn end-
to-end policies directly from data without explicit mapping and planning
stages. Particularly, reinforcement learning (RL) has achieved impressive
results in solving complex robot control problems, from quadrupedal locomotion
over challenging terrain [13, 14, 15] to reaching the limit in autonomous
drone racing [16]. However, RL suffers from slow convergence and tends to be
data-intensive, often necessitating large-scale, parallelized environments
[17, 18] for fast data collection and stable policy gradient estimation.
Consequently, their applicability has largely been confined to state-based
control problems, with limited success in tasks that require vision-based
inputs.

As an alternative strategy, imitation learning has been widely used in vision-
based applications due to its straightforward implementation and high sample
efficiency. For instance, a study by [19] proposes to learn a sensorimotor
policy that enables an autonomous quadrotor to fly acrobatic maneuvers by
leveraging demonstrations from an optimal controller. Similarly, [8] proposes
using imitation learning to train a neural network that can directly map noisy
depth images into collision-free trajectories [8] for obstacle avoidance. This
methodology substantially cuts down on inference time, thereby enhancing the
system to navigate obstacles and achieve high-speed flight in unstructured
environments swiftly.

Although the imitation learning framework can effectively solve a specific
task and is efficient for high-dimensional inputs, they often rely heavily on
the quality and comprehensiveness of expert demonstrations. This dependency
restricts the system’s generalizability beyond the initial training data and
can lead to inaccurate imitation of the expert. Additionally, experts are
typically tailored for a specific task, making it challenging to solve new
tasks. The complex labeling and training processes further restrict its
flexibility and limit practical applications. Therefore, we ask: _how can we
optimize a vision-based policy in a scalable and efficient manner while
maintaining task flexibility?_

Figure 1: Vision-based agile swarm navigation through cluttered environments
using an end-to-end neural network controller trained via differentiable
physics. (A) The indoor environment features both static and dynamic
obstacles. (B,C) Outdoor scenarios. Our system operates on a budget-friendly
$\$$21 ARM-based computer coupled with a depth camera for onboard sensing. All
results were achieved using onboard vision sensors without communication
between robots.  Figure 2: Detailed perspectives on swarm navigation through a
cluttered environment.  A team of six drones flying through gates (A),
avoiding dynamic obstacles (B, C), and navigating through thin structures (D).

We explore a different robot learning paradigm by training neural network
policies using _differentiable physics._ Specifically, we leverage prior
knowledge about the physical system and its inherent differentiability to
optimize a deep neural network policy end-to-end. Differentiable dynamics
enable performing backpropagation through a physical simulator directly. This
stands in contrast to model-free reinforcement learning and imitation
learning; both methods treat the system dynamics as a black box and estimate
the policy gradient using only sampled trajectory rollouts. We discover that
using physics priors, particularly simple point-mass dynamics, for policy
optimization enhances the training efficiency, task scalability, and
generalization capability of the learned flight system.

This work demonstrates the effectiveness of leveraging differentiable physics
in training a robust end-to-end navigation policy for agile vision-based
flight in dynamic and unstructured environments. Our key insight is that the
autonomous flight task can be formulated via a physics-driven loss function,
which enables the direct computation of policy gradients through
backpropagation. The physics-driven loss function specifies proximity to a
velocity target, distance to obstacles, and control smoothness; it can be
calculated by rolling out the system dynamics using a neural network control
policy and a differentiable simulator. Both the loss and the dynamics are
differentiable. Afterward, the neural network policy is updated via gradient
descent, in which the gradient is computed via backpropagation through both
the physics simulation and the loss function. Training directly with the
physics-driven objective function is highly efficient and can be easily scaled
up to solve diverse tasks.

### Contribution

We demonstrated the effectiveness of our method by training a recurrent neural
network policy for vision-based agile navigation in both single-agent and
multiple-agent applications. The results show that the trained policy can
navigate drones at high speed through complex environments featuring both
static and dynamic obstacles. Furthermore, we showcased proficiency in agile
swarm navigation without the need for communication, as well as flights
without dependency on a stand alone odometry module. In contrast to
traditional mapping-and-planning approaches, our end-to-end flight policy
offers substantial advantages in reducing both computational latency and error
accumulation. Compared to existing learning-based techniques [4, 8, 20, 21],
which are predominantly limited to single-agent configurations, often due to
the need for expert demonstrations or discretized control spaces, our approach
showcases exceptional flexibility and generalization capabilities for more
complex, multi-agent scenarios.

Fig. 1 provides an overview of our real-world experiments, where our flight
policy demonstrates reliable high-speed flight in unstructured, dynamic
environments. Fig. 2 shows detailed perspectives on swarm navigation through a
cluttered indoor environment. Although the policy is solely trained in simple
simulation environments featuring randomly placed obstacles, it exhibits
strong generalization in real-world scenarios, achieving speeds of up to
$20\text{\,}\mathrm{m}\text{/}\mathrm{s}$ in the wild. In comparison, the
state-of-art method [8] achieves a maximum speed of
$10\text{\,}\mathrm{m}\text{/}\mathrm{s}$. The performance gain in agility is
attributed to our physics-informed learning approach and to the flexibility of
our end-to-end flight system design. Furthermore, our system can avoid moving
obstacles without requiring explicit object detection or motion prediction,
which significantly enhances the adaptability of our approach in dynamic
environments.

Importantly, our physics-driven policy optimization allows for straightforward
scaling to multi-agent contexts. After training in a multi-agent setting, the
policy (deployed for decentralized control) can navigate multiple drones
through narrow gates, while maintaining awareness of other vehicles without
communication or global planning (see Fig. 6). Interestingly, the agents
exhibit self-organized behaviors such as waiting, following, resolving
conflicts, and mutual collision avoidance. These behaviors emerged from
minimizing a simple physics-driven function for obstacle avoidance without
explicitly designing a collaboration reward. Such emerging behaviors have not
been demonstrated before in the literature and could inspire further research
in this area.

Finally, we demonstrate that the trained model achieves exceptional
computational efficiency in real-world deployments. It operates on a budget-
friendly $21 ARM-based computer coupled with a depth camera. This is a crucial
advantage over other state-of-the-art systems [7, 8, 22, 19] that rely on
expensive and power-hungry GPUs. While those systems may offer high
computational power, they come with prohibitive costs and energy requirements,
limiting their applicability in resource-constrained environments. This
achievement is attributed to the efficiency and effectiveness of our training
approach, which allows for the use of simple network architectures and low-
resolution inputs to develop robust flight policies. As shown in Fig. 8E, with
the same training setup, our method requires only 10% of the samples needed by
traditional RL methods to reach its maximum reward level.

##  II Method

###  II-A Problem Formulation

Without loss of generality, we formulate the robot navigation task in an
optimization framework. The robot is modeled as a discrete-time dynamical
system, characterized by continuous state and control input spaces, denoted as
$\mathcal{X}$ and $\mathcal{U}$, respectively. At each time step $k$, the
system state is $x_{k}\in\mathcal{X}$, and the corresponding control input is
$u_{k}\in\mathcal{U}$. An observation $o_{k}\in\mathcal{O}$ is generated at
each time step based on the current state $x_{k}$ through a sensor model
$h:\mathcal{X}\rightarrow\mathcal{O}$, such that $o_{k}=h(x_{k})$. The
system’s dynamics are governed by the function
$f:\mathcal{X}\times\mathcal{U}\rightarrow\mathcal{X}$, which describes the
time-discretized evolution of the system as $x_{k+1}=x_{k}+f(x_{k},u_{k})$.
The discrete-time instants are $t_{k}=\Delta t\cdot k$, where $k$ ranges from
0 to $N$, thereby establishing a finite time horizon for the control problem.
Crucially, at each time step $k$, the robot receives a cost signal
$l_{k}=l(x_{k},u_{k})$, which is a differentiable function of the current
state $x_{k}$ and the control input $u_{k}$.

We model the policy as a function approximator represented by a neural network
$u_{k}=\pi_{\theta}(o_{k})$. The neural network takes the observation $o_{k}$
as input and outputs the control input $u_{k}$. The objective of our
optimization problem is to find the optimal policy parameters $\theta^{\ast}$
by minimizing the total loss via gradient descent:

$\displaystyle\min_{\theta}\mathcal{L}_{\theta}$
$\displaystyle=\sum_{k=0}^{N-1}l(x_{k},u_{k})=\sum_{k=0}^{N-1}l(x_{k},\pi_{\theta}(o_{k}))$
(1) $\displaystyle\theta$
$\displaystyle\leftarrow\theta-\gamma\nabla_{\theta}\mathcal{L}_{\theta}$ (2)

where $\gamma$ is the learning rate.

Policy gradient via differentiable simulation: In differentiable simulation
for policy learning, the backward pass is crucial for computing the analytic
gradient of the objective function with respect to the policy parameters.
Following [23], the policy gradient can be computed directly via
$\partial\mathcal{L}_{\theta}/\partial\theta=$

$\frac{1}{N}\sum_{k=0}^{N-1}\left(\sum_{i=0}^{k}\frac{\partial l_{k}}{\partial
x_{k}}\prod_{j=i+1}^{k}\left(\frac{\partial x_{j}}{\partial
x_{j-1}}\right)\frac{\partial x_{i}}{\partial\theta}+\frac{\partial
l_{k}}{\partial u_{k}}\frac{\partial u_{k}}{\partial\theta}\right),$

where the matrix of partial derivatives $\partial x_{j}/\partial x_{j-1}$ is
the Jacobian of the dynamical system $f$. Therefore, we can compute the policy
gradient directly by backpropagating through the differentiable physics model
and a physics-driven objective function $l_{k}$ that is differentiable with
respect to the physics model. This enables more efficient and potentially more
accurate optimization compared to traditional reinforcement learning methods.

###  II-B Differentiable physics simulation

The fundamental difference between a differentiable physics simulation and a
standard simulation, e.g. MuJoCo [24] or Flightmare [25], lies in the ability
to compute gradients with respect to model parameters or inputs. Such a
simulation can propagate the gradient from the output state $x_{k}$ directly
to the control input $u_{k}$, and hence, to the policy parameters $\theta$.

We provide a differentiable physics simulation for quadrotors. Typically, a
quadrotor is modeled as a rigid body, which is actuated by four motors [25].
Instead of seeking to accurately model the real-world quadrotor dynamics, we
opt for using a simple point mass model to approximate the quadrotor dynamics.
While common belief suggests that high-fidelity models are necessary for
successful simulation-to-reality transfer, our approach demonstrates that this
simple model can achieve this goal. Additionally, this approximation enables
faster gradient computation and better optimization landscape for training,
significantly improving efficiency. To this end, we have based our
differentiable physics simulation on a simple point mass model, which is
represented by the numerical integration of acceleration and velocity:

${v}_{k+1}={v}_{k}+\frac{{a}_{k}+{a}_{k+1}}{2}\Delta
t,\qquad{p}_{k+1}={p}_{k}+{v}_{k}+\frac{1}{2}{a}_{k}\Delta t^{2}.$

We control the vehicle using a thrust controller, which is tracked by a
classic inner-loop flight controller operating at a high frequency. To model
flight controller response, we implement a fixed control latency and an
exponential moving average on the sequence of desired thrust acceleration.
This process, as illustrated in Fig. 3, can be represented as the convolution
between the desired thrust vector $u_{k}$ with the simplified flight
controller impulsive response:

$\eta(t)=\begin{cases}0,&t<\tau\\\ \lambda e^{-\lambda(t-\tau)},&t\geq\tau\\\
\end{cases}$

where $\tau$ is a fixed delay and $\lambda$ is the exponential smoothing
factor. The filtered thrust $\hat{u}_{k}$ is considered as the actual thrust
applied on the point mass model.

We also add a quadratic air drag and take the sum of thrust and air drag to
calculate the acceleration (denoted as $a_{k}$) for the point mass physics
simulation. The parameters $\tau$ and $\lambda$ as well as the linear air drag
coefficient are tuned to match the real platform through calibration
(Supplementary S1). Since a quadrotor can only generate a collective thrust in
its body’s upward direction, the magnitude (length) of the thrusts defines the
overall vehicle acceleration, while the direction of the thrust vector defines
the inclination. Additionally, we align the quadrotor’s yaw angle with the
target direction, establishing a unique orientation. After that, the
quadrotor’s orientation and position (from the point-mass model) serve as the
extrinsic parameters for rendering the depth sensor data.

###  II-C Training environment and physics-driven objective function

Figure 3: Method overview: learning vision-based agile flight via
differentiable physics. The differentiable simulator carries gradients from
the output state $x_{k}$ directly to the control inputs $u_{k}$, and hence, to
the policy parameters $\theta$. A single timestep consists of depth map
rendering, action prediction, and quadrotor dynamics simulation. The training
environment only includes randomly placed obstacles.

Our training environment, including depth rendering and physics simulation, is
implemented with CUDA as a PyTorch extension [26]. As illustrated in Figure 3,
the training environment features four kinds of abstract obstacles, including
planes, cuboids, spheres, and cylinders. While existing methods for training
vision-based policies typically rely on high-fidelity simulators to generate
realistic scenes, we discovered that our training approach, even when using
such a simple simulation environment, can produce robust vision-based flight
policies that generalize well to diverse real-world scenarios. We randomly
generate the dimensions and positions of the obstacles for each rollout. There
are 64 environments running in parallel for one batch in training. At each
training iteration, we collect 64 trajectories by simulating the environments
for 150 timesteps, with a time step of 1/15 $\text{\,}\mathrm{s}\text{/}$.

The physics-driven objective function (or loss) consists of three parts:
proximity to velocity targets, obstacle avoidance, and control smoothness.
Proximity to velocity targets is measured using the smooth L1 loss [27]
between velocity control error and zero,

$\mathcal{L}_{v}=\frac{1}{T}\sum_{k=1}^{T}\text{Smooth
L1}(\|v_{k}^{set}-\bar{v}_{k}\|_{2},0)$

where $\bar{v}_{k}$ is the average velocity calculated via moving average
spanned across a 2-second time window. The average operation enables the agent
to focus on obstacle avoidance locally while tracking the desired velocity
$v^{set}_{k}$ in a longer horizon. The direction of the desired velocity,
denoted as $v^{set}_{k}$, is determined by the difference between the target
position and the current position. Its magnitude is limited to a predefined
maximum speed, and its gradient is not propagated during training.

The obstacle avoidance loss is defined by the distance between the quadrotor
center and the nearest point in the environment. We denote the closest
distance as $d_{k}$, the radius of the drone as $r_{q}$, and the speed
approaching the closest point as $v_{k}^{c}$. The obstacle-avoidance loss is
the approaching speed multiple with the sum of a truncated quadratic function
and a soft plus barrier:

$\mathcal{L}_{c}=\frac{1}{T}\sum_{k=1}^{T}v_{k}^{c}\max(1-(d_{k}-r_{q}),0)^{2}+\beta_{1}\ln(1+e^{\beta_{2}(d_{k}-r_{q})}),$

where $\beta_{1}$ and $\beta_{2}$ are control parameters of the soft plus
barrier function. we choose $\beta_{1}=2.5$ and $\beta_{2}=32$ for our
implementation. The approaching speed $v_{t}^{c}$ adjusts the penalty
according to how fast the drone is approaching an obstacle. When the drone is
stationary or moving away from the obstacle, this factor will nullify the
penalty. Conversely, when the drone is moving towards the obstacle, it will
ensure that the obstacle avoidance loss is significant enough to slow down the
agent.

To avoid large oscillation in the action space, we define a smoothness loss,
which is the penalty on the acceleration and jerk:

$\mathcal{L}_{a}=\frac{1}{T}\sum_{k=1}^{T}\|a_{k}\|^{2},\qquad\mathcal{L}_{j}=\frac{1}{T-1}\sum_{k=1}^{T-1}\|\frac{a_{k}-a_{k+1}}{\Delta
t}\|^{2}.$

The final loss is the weighted sum of these four terms, i.e.,
$L=\lambda_{v}\mathcal{L}_{v}+\lambda_{c}\mathcal{L}_{c}+\lambda_{a}\mathcal{L}_{a}+\lambda_{j}\mathcal{L}_{j}$
where $\lambda_{v}$, $\lambda_{c}$, $\lambda_{a}$, and $\lambda_{j}$ are the
weights. We choose
$\lambda_{v}=1,\lambda_{c}=2,\lambda_{a}=0.01,\lambda_{j}=0.001$ for our
implementation.

Figure 4: A computation graph of the physics simulation. Temporal gradient
decay mitigates gradient explosion. We illustrate the simplified model
$v_{t}=v_{t-1}+a_{t}\Delta_{t},p_{t}=p_{t-1}+v_{t}\Delta_{t}$.

###  II-D Temporal gradient decay

Figure 4 presents the computation graph for the forward simulation and
backpropagation of the gradients of the point-mass physics simulation.
Although the physics simulation is differentiable, direct backpropagation
leads to gradient explosion [28][23] when the gradient is backpropagated
through the entire simulation. This results in the issue where long-range
gradients from future time steps can accumulate significantly large values,
resulting in unstable optimization. Intuitively, the agent can only learn to
avoid relatively close obstacles, while the long-range gradients enforce the
agent to avoid all future obstacles without considering the limitations of the
perceptual range.

To overcome this problem, we incorporate an exponential decay mechanism into
the position and velocity gradient during backpropagation. Specifically, we
multiply the gradient of $s_{t+1}$ (state at time step t+1) with respect to
$s_{t}$ by the exponential decay factor $e^{-\alpha\Delta t}$. Here, $\alpha$
is a parameter that controls the decay rate, and $\Delta t$ is the duration of
a single time step in numerical simulation. This mechanism decays gradient
during backpropagation through time, producing two key effects: it mitigates
gradient accumulation and ensures that the supervisory signal stems from the
near future. By emphasizing immediate objectives and filtering out potential
distractions from uncertain future obstacles, the adjusted gradient
substantially boosts training performance. As a result, we optimize the policy
using the following gradient that has temporal gradient decay:
$\partial\mathcal{L}_{\theta}/\partial\theta=$

$\small\frac{1}{N}\sum_{k=0}^{N-1}\left(\sum_{i=0}^{k}\frac{\partial
l_{k}}{\partial x_{k}}\prod_{j=i+1}^{k}\left(\frac{\partial x_{j}}{\partial
x_{j-1}}e^{-\alpha\Delta t}\right)\frac{\partial
x_{i}}{\partial\theta}+\frac{\partial l_{k}}{\partial u_{k}}\frac{\partial
u_{k}}{\partial\theta}\right).$

###  II-E Network architecture

The network utilizes a Convolutional Recurrent Neural Network (CRNN) structure
[29], which combines a lightweight convolutional neural network for depth map
feature extraction and a GRU [30] layer for consistent planning and control.
The depth map is inverted and max-pooled to $16\times 12$ as network input.
Each frame passes through several CNN layers with filters (kernel size) 32(2)
- 64(3) - 128(3) and then flattened and linearly projected to a 192-dim
feature. All the convolution layers have a stride of 1 and are followed by a
LeakyReLU activation function. Meanwhile, the target velocity, attitude
estimation, and velocity estimation (optional) are linearly projected to a
192-dimensional feature and added to the corresponding image feature. The
combined feature is fed into a GRU cell, and the GRU state is used to predict
desired thrust acceleration and current velocity estimation through a fully
connected layer.

###  II-F Hardware setup

Our hardware setup is visualized in Figure 5A. The hardware developed in this
work is designed to be small, agile, and fully autonomous while remaining low-
cost. We set up our quadrotor following first-person view (FPV) drone style
with a total weight of only $365\text{\,}\mathrm{g}\text{/}$. The aircraft is
built using a Roma 3-inch frame, GEMFAN 3-inch propellers, and 1606 3750KV
motors. The max thrust-to-weight ratio can achieve 3.6, thereby enabling our
quadrotor’s agility. We utilize a commonly adopted FPV flight tower,
consisting of an Aocoda F7mini flight controller and a HAKRC 4-in-1 electronic
speed controller. Furthermore, we utilize a customized BetaFlight [31]
firmware for autonomous flight. Regarding the perception system, a RealSense
D435i camera is used as a depth perception module which outputs depth images
at a rate of $15\text{\,}\mathrm{Hz}\text{/}$. Additionally, we use a compact
Single-board computer (SBC) Mango Pi [32] as our onboard computer, which
measures 30x65x6mm, weighs just 11g, and is equipped with a Cortex-A53 CPU. We
integrated the computer into the middle of the aircraft carbon boards and
combined it with the flight controller to form a “3-layer flight tower”.
Thanks to the extremely lightweight network design, our whole system can run
in real time with such a compact structure.

###  II-G Related works

Traditional approaches [33, 34, 35] for vision-based navigation are largely
mapping-based. They first build a high-quality map using occupancy grids [34]
or the Euclidean Signed Distance Field (ESDF) [35] as a representation of the
3D environment. Then, the navigation problem is converted into planning and
control. Planning typically relies on probabilistic search or gradient-based
trajectory optimization, while control often utilizes model-based techniques,
such as model predictive control [36, 37]. To enhance safety and computational
efficiency, certain strategies integrate the Safe Flight Corridor (SFC) as an
intermediate representation of the obstacles for planning [38, 39, 40].
However, dividing the navigation task into mapping-planning-and-control
results in pipelines that may overlook the interplay between different stages,
leading to compounded errors. Furthermore, the sequential nature of these
pipelines introduces extra latency and computational burden, thereby hindering
their feasibility in complex or dynamic environments.

Different from mapping-based methods, learning an end-to-end policy for
vision-based navigation can directly control the vehicle from raw sensory
observations without explicitly mapping and planning. Particularly, imitation
learning is an effective approach for training a deep sensorimotor policy, in
which the policy is trained by imitating either demonstration from human
experts or trajectory rollouts by a model-based controller. Imitation learning
has been shown to be effective in solving several challenging tasks, including
vision-based drone acrobatics[19], navigation in the wild [41, 4, 8], and
drone racing [42, 43, 44, 45, 46]. Although the imitation learning framework
can effectively solve a specific task, it often relies heavily on the expert
demonstrations. This dependency restricts the system’s generalizability beyond
the initial training data. Additionally, experts are typically tailored for a
specific task, making it challenging to solve new tasks.

Differentiable physics simulation is a powerful technique that applies
gradient-based methods to the learning and control of physical systems [47].
Previous studies have demonstrated the efficiency of differentiable
programming approaches in various robotics simulation tasks, such as control
of soft robots [48, 49, 50], mimicking human motions in humanoid robots [51],
system identification [52], and quadruped locomotion [53]. Nevertheless,
differentiable programming comes with its own challenges, most notably
susceptibility to gradient explosions. Consequently, developing a practical
differentiable simulation framework for tackling real-world quadrotor
navigation tasks remains an open research challenge. Notably, existing
autonomous quadrotor applications have yet to adopt differentiable simulation-
based methods for real-world navigation.

##  III Results

Figure 5: High-speed flight through complex and dynamic environments using a
$21 low-cost tiny computer. (A) Experimental platform. We demonstrate the
robustness of navigating in dense forests (B, E) and urban environments (C,
D). Our framework can easily handle high flight speed (F) and dynamic
obstacles (G-I). Flight speed and success rates (with number of success trails
labeled on bars) are in (J,K). All these experiments involve end-to-end
navigation without reliance on visual-inertial odometry or GPS localization
for state estimation.

Video S1 summarizes the results and the method of this work. In this section,
we discuss the results in detail, highlighting the robustness, flexibility,
and generalization capabilities of our method. In summary, our results contain
high-speed flight through complex and dynamic environments, agile
communication-free swarm navigation, odometry-free flight, and benchmark
comparison against state-of-the-art methods.

###  III-A High-speed flight through complex and dynamic environments

In this section, we demonstrated the robust navigation performance of our end-
to-end policy in complex and dynamic environments. We train our vision-based
policy using a customized simulator that incorporates differentiable dynamics
for simulating the drone model and a rendering engine for rendering cluttered
environments. After training in simulation, we deploy the resulting navigation
policy directly in the physical world. To push the limit of vision-based agile
flight in the real world, we have built a simple lightweight drone that
features a low-cost tiny computer for all computational tasks onboard and an
Intel RealSense depth camera for perceiving the environment (see Fig. 5A). As
a result, our system has a total weight of only
$0.365\text{\,}\mathrm{kg}\text{/}$ and can generate a maximum thrust of
$13.7\text{\,}\mathrm{N}\text{/}$.

Our physical system and the real-world results are summarized in Fig. 5. We
test our system in various real-world environments, including dense forests,
city parks, and indoor environments with both static and dynamic obstacles. We
design navigation tasks by specifying an initial position and an unknown
environment. The drone is required to navigate through the environments at
high speed while avoiding both static and dynamic obstacles. None of the
testing environments have been seen by our navigation policy during training.
We utilize a GPS module to measure outdoor flight speed and a motion capture
system for indoor environments. The actual flight speeds and the success rates
at different desired speeds are shown in Fig. 5J,K. The lack of higher-speed
data in environments D, G, and H is attributed to the site’s size limitations,
which prevent drones from achieving higher speeds. The flight recordings are
included in the supplementary video.

The real-world results indicate that our end-to-end navigation policy
generalizes well to real-world environments, and exhibits excellent obstacle
avoidance performance using extremely limited sensory and computational
resources. Fig. 5B shows the flight through a dense forest that has low
bushes, where high-speed traversal is very difficult due to the low visibility
and the complex distribution of fine branches. Fig. 5C, D demonstrates our
system can fly through dense obstacles at a speed of 7 m/s with a high success
rate. Fig. 5F showcased a high-speed flight of 20 m/s in a forest. Fig. 5G-I
showcase the agility of our system in navigating through dynamic obstacles.
Our recurrent neural network policy can effectively extract hidden information
regarding the state of our vehicle and moving obstacles, facilitating agile
navigation in rapidly changing environments without the need for a separate
localization or motion prediction module. Importantly, all these experiments
are achieved through policy training in simulation.

A standout feature of our system lies in its robustness and capability to
generalize effectively to unknown dynamic environments. In real-world
settings, environments are frequently subject to change, including unexpected
movements or appearances of obstacles. Dynamic environments present
significant challenges: for mapping-based methods, the system delay arises
from their cascaded structure and the computational burden of map
construction; for learning-based methods, the issue lies in their ability to
generalize unseen environments. Fig. 5G-I exemplifies our agent’s adaptive
capability in handling various dynamic obstacles that have different motion
patterns. Fig. 5G highlights the quadrotor’s skill in navigating through a
closing gate, underscoring its competence in tackling not just dynamic but
also progressively complex scenarios. Fig. 5H reveals that the quadrotor
maintains a safe distance from the moving obstacle, ensuring secure flight.
Fig. 5I capture the agent’s swift response to a swinging obstacle, showcasing
the low latency and reliability of the trained agent. Importantly, these
capabilities were developed through training in simulated settings featuring
only static obstacles, further validating the efficacy and versatility of our
training methodology.

###  III-B Agile communication-free swarm navigation

Fig.1, 2, and 6B illustrate our approach’s ability to navigate multiple aerial
robots through cluttered environments filled with various static or moving
obstacles without explicit communications between robots. To further
demonstrate the applicability of our approach to challenging swarm navigation
tasks, we design a challenging task: a team of drones must swap their
positions by navigating through a narrow gate in opposite directions.

This task exemplifies how our approach trains decentralized policies enabling
drones to coordinate with each other using only local sensory information to
avoid collisions with the environment and other drones. Moreover, our training
approach is inherent flexible, requiring only minimal modifications for multi-
agent tasks. The training process remains the same as in the single-agent
setting, except that each agent is represented as a spherical obstacle within
the simulation. After training, the policy operates independently for each
robot using the same observation modalities, including depth measurement and
orientation, as in the single-agent scenario.

Fig. 6A summarizes the result of the task of swapping position through a
narrow gate. We conducted real-world experiments using six quadrotors, which
were divided into two groups of three agents each. Each group is positioned on
opposite sides of the testing arena. The objective is for all six agents to
navigate through a narrow gate simultaneously while reaching a destination.
Due to the small size of the gate, it is impossible for all six drones to pass
through the gate simultaneously. Hence, the agents are required to coordinate.
To ensure each agent can reach its target position correctly, we employ a
motion capture system to obtain reference directions. We use velocity
estimation from the motion capture system as input to each neural network
policy. Each agent is required to follow its own velocity command while
without having access to the information about the state of other agents.
Therefore, the agents are required to make individual decisions without any
communication. We define a run as successful if all drones reach their
respective goals within a radius of $1.2\text{\,}\mathrm{m}\text{/}$ without
any collisions.

Figure 6: Communication-free swarm navigation. We perform real-world
evaluation with 6 drones navigating through (A) a narrow gate in opposite
directions, and (B) various obstacles in the same direction. (C) Enlarged
views of individual behavior showing the emergence of collective intelligence.
Video demonstrations are available. In simulated environments (D, E), the
corresponding results (F, G) indicate our success rates and flight times, even
without communication, are on par with Ego-v2 (which relies on communication
and global positioning for mutual avoidance), while preserving our high-speed
flight advantage.

The result shows that all six drones successfully passed through the gate and
reached their destination. Interestingly, the trained vision-based agents
demonstrated emerging self-organized behaviors - such as waiting in air
traffic, following an agent, and retreating to give way to incoming agents -
all without the need for communication, global planning, or adopting a
coordination loss in the training objective.

Furthermore, we conduct experiments to quantitatively compare our approach
with the state-of-the-art centralized planning method for swarm navigation Ego
v2 [7]. We evaluate the performance in terms of success rates and task
completion time. Fig. 6F-H summarizes the results: our communication-free
approach achieves comparable performance as the communication-based method.
Notably, existing methods heavily rely on network communication and global
planning for effective coordination and mutual avoidance. However,
communication can be unreliable in real-world scenarios, and centralized
planning requires accurate relative positioning between robots, which is
difficult to acquire, particularly in the absence of GPS. Consequentially, Ego
v2 adopts ad hoc UWB positioning and drone detection in real-world
experiments.

Much like humans or other animals, our agents achieve self-organization using
only local sensing, making them ideal for scenarios where communication is
impractical. These capabilities enable deploying autonomous swarms in
environments without global positioning or communication infrastructure, such
as search and rescue missions or reaching remote locations. Our approach
provides a robust solution for swarm navigation and obstacle avoidance in
these challenging conditions.

###  III-C Odometry-free flight

To further evaluate the flexibility and robustness of our end-to-end flight
system, we study the role of state estimation in obstacle avoidance. State
estimation is a central requirement for navigating a mobile robot. Traditional
approaches to vision-based navigation predominantly depend on globally
consistent state estimation. Methods of this kind rely heavily on the quality
of visual-inertial odometry, which is known to be very sensitive during high-
speed flight [10, 54, 55]. High-speed motion can lead to large displacements
and little overlap between consecutive frames that make feature tracking
difficult [10], resulting in the failure of the visual localization system.
Additionally, visual localization and mapping demand considerable
computational resources. For instance, in the Ego-v2 system [7], CPU usage for
localization surpasses the combined usage of all other processes, even when
GPU acceleration is employed.

The inherent flexibility of our approach allows for training agents without
the position and velocity estimates as part of the input. Consequentially, the
resulting odometry-free agent does not require a standalone localization
module to estimate the position and velocity but still learns to navigate and
control using only attitude estimation and depth map. Note that our method
employs a recurrent neural network that maintains an internal state estimate,
enabling the system to operate without explicit odometry for state estimation
when deployed on a real drone. In this way, we further exploit the potential
of neural networks to achieve end-to-end localization, planning, and control.
As subsequent results will demonstrate, our trained agent operates robustly
even in the absence of localization input and delivers performance metrics
comparable to scenarios utilizing external localization.

We assess the performance of our odometry-free agent by comparing its success
rate and flight speed against our baseline agent that use external velocity
estimation inputs (odometry-based) in real-world scenarios. For odometry-based
agents, we evaluate two distinct source of velocity estimation: one utilizing
a VICON motion capture system and another employing a visual-inertial odometry
system (VIO) [56]. The testing environment, as shown in Fig. 7A, includes a
relatively open area at the front designed to enable the drone to achieve high
speeds, along with obstacles of cubes and trees at the rear. We evaluated the
drone’s speed and success rate to reach the area behind the obstacles. We
conducted tests at three different desired speeds: 4, 7, and
$10\text{\,}\mathrm{m}\text{/}\mathrm{s}$. Each test is repeated 10 times.

The results are shown in Figure 7. Figure 7B compares the success rates of our
odometry-based agents against our odometry-free agent. The bars are labeled
with the number of successful trials for each test. Figure 7C the statistics
of the maximum speeds achieved during the trials. Both our odometry-free agent
and the odometry-based agents utilizing VICON for state estimation
successfully reaches the target without crashing. In contrast, the agent that
relies on VIO for state estimation exhibits a lower success rate, especially
at higher speeds. This performance is consistent with the previously mentioned
limitation of VIO systems, which can suffer from drift at high speeds,
resulting in failures such as collisions with obstacles like trees. In
contrast, our odometry-free agent outperforms the agent that rely on the VIO
system, achieving the same level of performance as the agent that uses the
VICON external motion capture.

It is important to point out that the majority of current planning and control
algorithms still depend on explicit state estimation [10]. The experiment
suggests that standalone state estimation modules might not be necessary for
autonomous flight. This finding has been consistently validated by our results
in Sec. III-A, where we show the effectiveness of odometry-free flight in
various challenging scenarios in the real world.

Figure 7:  End-to-end vision-based flight without an explicit odometry module.
(A) The controlled testing environment. Our neural network achieves reliable
performance by learning implicit state estimation from flight dynamics and
previous control commands via a memory-based network. (B, C) Our odometry-free
end-to-end system demonstrates high success rates, yielding results comparable
to those achieved through an external motion capture system.

###  III-D Baseline comparisons

We carry out comprehensive experiments in simulation to benchmark our approach
against existing methods, including a traditional mapping-based method [7] and
a learning-based method [8]. To ensure a fair comparison, we utilized the
Flightmare simulator [25], following the experimental setup outlined in [8].
Additionally, we use the AirSim simulator [57] for evaluation in other
additional environments. We tested all methods with different target speeds—4,
7, 10, 13, and 22 $\text{\,}\mathrm{m}\text{/}\mathrm{s}$. We repeat each
experiment 10 times and compute the average flight speed, peak flight speed,
and success rate. A task is considered to be successful if the drone reaches
its goal position within a 5-meter radius without crashing. Average speeds
were calculated based on these successful trials.

The results are summarized in Figure 8. The traditional mapping-based method
[7] (termed Ego v2) performs relatively well at very low speed, e.g., below
$3\text{\,}\mathrm{m}\text{/}\mathrm{s}$; however, it experiences significant
performance drops as the target speed increases. This aligns with the real-
world experiments demonstrated in the original paper [58], in which the
average flight speed is around $2\text{\,}\mathrm{m}\text{/}\mathrm{s}$. Our
findings also confirm the results reported in [8], where they attribute such
failures to the increased processing and control latency inherent in cascaded
structures.

While the learning-based method [8] (called Agile) performs well in the
forested environment, this specific environment was also used for training
their navigation policy. When testing in an unseen environment, e.g., in an
urban setting Fig. 8 B-C, [8] achieves very low success rates. Our results
indicate that expert-based approaches struggle with generalization to unseen
environments, hence, resulting in significant performance drops.

Figure 8:  Baseline comparisons for vision-based agile flight in cluttered
environments.  (A-C) Comparing our method against state-of-the-art vision-
based navigation methods. Our method shows the best success rates in all
tested environments. (D) Our approach outperforms a state-of-the-art imitation
learning method with better success rate and less training time. (E)
Comparison among our approach, reinforcement learning, and vanilla
differentiable physics (BPTT) in policy training. (F) Robustness study.

Such shortcoming is primarily due to the difficulty in obtaining high-quality
expert labels in obstacle-dense, high-speed conditions [8]. Additionally, the
sampling-based expert itself faces challenges in consistently identifying the
optimal route, leading to erratic flight patterns in the trained agents,
characterized by large lateral oscillations in their real-world flight. The
incorporation of error corrections through the DAGGER framework [59], along
with frequent trajectory replanning, exacerbates the drone’s unstable behavior
further. Lastly, the expert-based approach tends to maintain a constant speed,
lacking the flexibility to adapt flight speed in response to varying obstacle
densities.

In contrast, our approach consistently achieves high success rates, even at
high speeds. Our framework seamlessly integrates planning optimization with
neural network training, eliminating the need for hard-to-design expert
planners. This integration empowers our method to effectively learn and adapt
to complex environments. Furthermore, the end-to-end design of the
sensorimotor agent enhances coordination between planning and control,
achieving more stable control during replanning and automatic speed
adjustments in response to varying obstacle densities. We verify that our
framework surpasses the expert-based method, specifically Agile [8], in terms
of convergence speed. In our experiment, we extract the point cloud from
training scenes used by Agile and evaluated success rates and average speeds
at various optimization steps. Fig. 8D shows the success rate for different
training iterations at $7\text{\,}\mathrm{m}\text{/}\mathrm{s}$, with the
average speed and training time annotated on the data point. The result shows
the faster convergence of our method: by the $\rm 1k^{th}$ iteration, our
approach has already achieved a 100% success rate at an average speed of 5
m/s, while Agile [8] is only starting to show success, with a mere 20% success
rate.

Additionally, we established a reinforcement learning baseline with the
Proximal Policy Optimization (PPO) algorithm [60]. Fig 8E shows a comparison
of the learning curve—our differentiable physics quickly achieves high rewards
given limited simulation steps, while RL requires substantially more samples.
Our method requires only 10% samples of the PPO algorithm to achieve its
maximum reward level. We attribute this improved convergence to the lower
variance of first-order gradients inherent in the smooth and well-behaved
point mass model[61]. Fig.8F presents a validation of the trained policy in
the same forest environment, where our approach achieves higher success rates
across all tested target velocities.

Although leveraging differentiable physics is the key to our approach, we
found that directly applying the vanilla differentiable physics method did not
work properly, primarily due to gradient explosion, as pointed out in [23]. To
address this issue, we designed a temporal gradient decay mechanism for
training, as described in the method section (see Fig.4). We validate this
design and the use of low-resolution depth maps using the Flightmare simulator
[25] within the forest scene. For all methods, we follow [8] and use the depth
map from SGM as the input. In Fig. 8, “BPTT” stands for not applying gradient
decay ($\alpha=0$). For high-resolution (HighRes) experiments, we train and
test on $128\times 96$ depth images with the MobileNetV3 [62] backbone.

The results in Fig. 6E, F indicate the importance of having temporal gradient
decay for calculating the policy gradient during policy training. We show that
training with vanilla differentiable physics (BPTT) has slower convergence and
achieves sub-optimal performance. As a result, the BPTT agent agent exhibits a
notably lower overall success rate and fails to achieve higher speeds.
Finally, while using higher-resolution depth maps improves performance during
training, it results in lower success rates and flight speeds during testing.
The reason may be that training with a higher input resolution can lead to
overfitting to environmental features and demands higher quality depth maps at
the inference stage. However, the real-world depth maps are usually noisy.
Using a lower-resolution depth map aids in bridging the gap between simulation
and reality.

##  IV Discussion

This work demonstrated the effectiveness of differentiable physics in agile
vision-based navigation. Contrary to reinforcement learning and imitation
learning, which often treat the robot’s physics model as a black box, our
approach leverages differentiable physics simulation to train a neural network
flight policy. This enables end-to-end optimization of the neural network
controller by backpropagating loss gradients directly through the robot’s
physics model. As a result, our methodology eliminates the need for task-
specific expert labels, as required in imitation learning. Compared to
reinforcement learning methods, which rely on zeroth-order optimization and
consequently require a large number of samples, differentiable physics utilize
gradient information for first-order optimization, leading to a significant
improvement in sample efficiency.

We showcased the effectiveness of our method in vision-based flight in real-
world scenarios. Our flight policy achieved zero-shot generalization to the
physical world, excelling in flight speed and robustness across various
complex environments with both static and dynamic obstacles. Furthermore, our
framework demonstrates remarkable scalability in multi-agent settings. Tests
with a swarm of quadrotors showed agile navigation through dynamically
changing environments and confined spaces, even without explicit communication
or centralized planning algorithms. Remarkably, our policy displayed
coordinated behaviors, like waiting and following, without explicit
communication. Moreover, our flight policy is effective in partially
observable settings without requiring the vehicle’s full state. This
capability not only simplifies system design but also provides immense
benefits for deployment in unknown or unpredictable settings.

Regarding computational efficiency, we validated our system with extremely
limited sensory and computational resources. We have implemented our system on
a budget-friendly $21 microcomputer without relying on GPS or visual-inertial
state estimators for localization. This achievement is particularly noteworthy
as it demonstrates the minimal computational and sensory resources required by
our approach, making it highly accessible and deployable in resource-
constrained settings. Despite these computational constraints, our system
outperforms state-of-the-art systems [7, 8] that rely on GPU-based
computations.

Looking ahead, there are several avenues for future research. Our framework’s
adaptability in task design—spanning network input, output, loss functions,
and environmental settings—has already proven effective in localization-free
obstacle avoidance and swarm navigation without communication. Such
versatility paves the way for a broader range of applications in fields such
as search and rescue, logistics, inspection, entertainment, and agriculture.

## Acknowledgment

We thank SJTU SEIEE · G60 Yun Zhi AI Innovation and Application Research
Center for indoor experiment support, J. Li for initial efforts in the multi-
agent experiments, and L. Zhang, F. Yu for helping with the experiments and
the valuable discussions.

##  Appendix S1 Calibration and system identification

Figure S1: Flight control response in simulation. The simulation closely
mirrors our real drone, demonstrating the accuracy of our calibration.

For the effective training of our quadrotor navigation framework, it’s pivotal
that our simulations mimic real-world flight dynamics accurately. The primary
goal of calibration is to identify the parameters of our differentiable
physics simulation, ensuring that simulated dynamics mirror real-world
quadrotor responses. Calibration involves adjusting for control latency and
air drag.

###  S1-A Control latency simulation

Our simulation considers a fixed control latency and an exponential moving
average for modeling the flight controller response. We fix the proportional
attitude controller with an attitude control gain of 13. We perform
calibration experiments on roll and pitch response to measure the latency
parameters $\lambda,\tau$, yielding an estimate of $\lambda=12,\tau=1/15$.
Fig. S1 A and B shows the roll and pitch response from a real-world flight,
where the simulated responses closely match the real-world responses.

Measuring thrust in the real world is more challenging because the
acceleration measurements are mixed with gravity, air drag, and non-linear
thrust model. Nevertheless, we have verified that the delay parameters for
roll and pitch are applicable for thrust as well. Results from Fig. S1C shows
that our simulation aligns closely with the real-world thrust delay.

We have verified the consistency of this latency across several platforms,
provided the attitude control gain remains uniform: a) our platform shown in
Fig. 3A using BetaFlight; b) a larger $0.6\text{\,}\mathrm{kg}\text{/}$ drone
with 4-inch propellers also using BetaFlight; c) a
$1.2\text{\,}\mathrm{kg}\text{/}$ drone with 5-inch propellers running a PX4
controller; d) the AirSim [57] simulator; e) the Flightmare [25] simulator.

###  S1-B Air drag identification

Figure S2: The simulated flight velocity v.s. our real drone. The relatively
small difference between the simulation and the real drone shows the fidelity
of our air drag simulation.

In low-speed navigation, air drag is frequently overlooked. However, for high-
speed sensorimotor flights, it’s a critical factor. At speeds exceeding
$9\text{\,}\mathrm{m}\text{/}\mathrm{s}$, our 3-inch quadrotor (refer to Fig.
3A) with propeller guards encounters considerable air resistance, measuring
over $5\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}^{2}$. With fixed agent actions,
air drag largely dictates the flight speed. For localization-free flights, air
drag plays a primary role in the implicit estimation of velocity.

Air drag varies depending on the platform. To calibrate it, we’ve employed a
straightforward method. Initially, we record actions from real-world flights
and simulate using the drag model $a_{drag}=-\theta_{1}\|v\|v-\theta_{2}v$,
where $\theta_{1}$ and $\theta_{2}$ are undetermined parameters. Then, we
conduct a grid search for the parameters and compare the simulated speed to
the GPS recorded speed (as shown in Fig. S2A). The parameter set that yields
the smallest speed discrepancy is adopted as the drag coefficients. To enhance
model robustness, we train using a diverse set of drag coefficients, centered
around the calibrated values (as depicted in Fig. S2B).

##  Appendix S2 Temporal gradient decay

Temporal gradient decay, as discussed in Section B, is an essential technique
to ensure stability and efficiency in training, especially for long-range
sequences. This method manages the accumulation of gradients backpropagated
through time, ensuring that the significance of distant obstacles vanishes
over time. We first highlight the challenge posed by gradient accumulation
(refer to Fig. 4A, B) and subsequently demonstrate how temporal gradient decay
addresses the issue of exploding gradients (see Fig. 4C). Ultimately, we
illustrate how the decayed gradient teaches the agents to maneuver at the
optimal moment.

Fig. 4 shows the graphical model of simulation spanning 4 timesteps, with the
objective function being evaluated based on the position at t=4. For
simplicity, we take $\Delta_{t}=1$ without loss of generality. Fig. 4B shows
the backward computation. Here, the positional gradient, denoted as g, is
backpropagated through the entire computation graph. The numerical integration
structure further leads to a growing gradient signal that ultimately
accumulates to the model parameters. In contrast, when gradient decay is
applied during temporal backpropagation, the gradients affecting the
parameters result from the product of a linearly increasing term and an
exponentially decaying term.

Figure S3: Temporal gradient decay aligns the supervisory signals with the
sensing range. We show a scenario (A) where the agent approaches an obstacle.
Graph (B) shows how gradient decay aligns training signals within the sensing
range.

Fig. S3 shows the advantage of temporal gradient decay in aligning supervisory
signals with the quadrotor’s sensing range. The blue line in Fig. S3,
representing the gradient without temporal gradient decay, displays an ever-
increasing gradient. This misleads the agent with the impractical task of
avoiding obstacles beyond its sensing range. The gradient, after the
application of decay, initially rises and then diminishes to zero as it’s
backpropagated over time. This dynamic achieves a balance, addressing the
inherent conflict that earlier maneuvers provide more significant benefits for
obstacle avoidance, yet distant obstacles are often more challenging to
perceive. Thus, the agent is directed to concentrate more on proximate
obstacles that are immediate and more perceptible.

By prioritizing immediate and perceptible obstacles, the training process can
converge faster as the training objective becomes more feasible, and the agent
receives more relevant feedback. This method ultimately foresters a more
robust navigation system.

##  Appendix S3 Training hyperparameters

The training hyperparameters is listed in Table S1.

Training Parameter | Value
---|---
optimizer | AdamW
learning rate | 0.001
learning rate schedule | cosine decay
weight decay | 0.01
batch size | 64
number of time steps $N$ | 150
$\Delta t$ | 1/15 $\text{\,}\mathrm{s}\text{/}$
gradient decay rate $\alpha$ | 0.92
training iterations | 50000
TABLE S1: The hyperparameters used for training. Notably, we add a small
perturbation on the simulation time step $\Delta t$ during training.

##  Appendix S4 Training details of the reinforcement learning baseline

This section provides training details for the reinforcement learning
baseline. Specifically, we use a state-of-the-art model-free reinforcement
learning algorithm called Proximal Policy Optimization (PPO) [60]. We use
parallelized simulation for data collection, in which the number of
environments is identical to that of our policy training using differentiable
physics. The PPO hyperparameters are given in Table S2.

Training Parameter | Value
---|---
optimizer | AdamW
weight decay | 0.01
discount factor | 0.99
GAE $\lambda$ | 0.95
number of time steps $N$ | 150
$\Delta t$ | 1/15 $\text{\,}\mathrm{s}\text{/}$
clip range | 0.2
entropy coefficient | 0.001
parallel environments | 256
training iterations | 12,500
minibatch size | 38,400
mini epochs | 5
TABLE S2: Hyperparameters for the Proximal Policy Optimization (PPO)
Algorithm.

## References

  * [1] D. C. Schedl, I. Kurmi, and O. Bimber, “An autonomous drone for search and rescue in forests using airborne optical sectioning,” _Science Robotics_ , vol. 6, no. 55, p. eabg1188, 2021, publisher: American Association for the Advancement of Science.
  * [2] J. Xing, G. Cioffi, J. Hidalgo-Carrió, and D. Scaramuzza, “Autonomous power line inspection with drones via perception-aware mpc,” in _2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2023\.
  * [3] A. T. Sage, M. Cypel, M. Cardinal, J. Qiu, A. Humar, and S. Keshavjee, “Testing the delivery of human organ transportation with drones in the real world,” _Science Robotics_ , vol. 7, no. 73, p. eadf5798, 2022, publisher: American Association for the Advancement of Science.
  * [4] A. Giusti, J. Guzzi, D. C. Cireşan, F.-L. He, J. P. Rodríguez, F. Fontana, M. Faessler, C. Forster, J. Schmidhuber, G. Di Caro _et al._ , “A machine learning approach to visual perception of forest trails for mobile robots,” _IEEE Robotics and Automation Letters_ , vol. 1, no. 2, pp. 661–667, 2015.
  * [5] F. Gao, L. Wang, B. Zhou, X. Zhou, J. Pan, and S. Shen, “Teach-repeat-replan: A complete and robust system for aggressive flight in complex environments,” _IEEE Transactions on Robotics_ , vol. 36, no. 5, pp. 1526–1545, 2020, publisher: IEEE.
  * [6] X. Zhou, Z. Wang, H. Ye, C. Xu, and F. Gao, “Ego-planner: An esdf-free gradient-based local planner for quadrotors,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 478–485, 2020, publisher: IEEE.
  * [7] X. Zhou, X. Wen, Z. Wang, Y. Gao, H. Li, Q. Wang, T. Yang, H. Lu, Y. Cao, C. Xu, and others, “Swarm of micro flying robots in the wild,” _Science Robotics_ , vol. 7, no. 66, p. eabm5954, 2022, publisher: American Association for the Advancement of Science.
  * [8] A. Loquercio, E. Kaufmann, R. Ranftl, M. Müller, V. Koltun, and D. Scaramuzza, “Learning high-speed flight in the wild,” _Science Robotics_ , vol. 6, no. 59, p. eabg5810, 2021, publisher: American Association for the Advancement of Science.
  * [9] Z. Zhang and D. Scaramuzza, “Perception-aware receding horizon navigation for MAVs,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 2534–2541.
  * [10] J. Delmerico, T. Cieslewski, H. Rebecq, M. Faessler, and D. Scaramuzza, “Are we ready for autonomous drone racing? the uzh-fpv drone racing dataset,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 6713–6719.
  * [11] W. Wang, D. Zhu, X. Wang, Y. Hu, Y. Qiu, C. Wang, Y. Hu, A. Kapoor, and S. Scherer, “Tartanair: A dataset to push the limits of visual slam,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 4909–4916.
  * [12] Z. Teed and J. Deng, “DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras,” _Advances in neural information processing systems_ , 2021.
  * [13] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter, “Learning agile and dynamic motor skills for legged robots,” _Science Robotics_ , vol. 4, no. 26, p. eaau5872, 2019.
  * [14] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning robust perceptive locomotion for quadrupedal robots in the wild,” _Science Robotics_ , vol. 7, no. 62, p. eabk2822, 2022.
  * [15] S. Choi, G. Ji, J. Park, H. Kim, J. Mun, J. H. Lee, and J. Hwangbo, “Learning quadrupedal locomotion on deformable terrain,” _Science Robotics_ , vol. 8, no. 74, p. eade2256, 2023.
  * [16] Y. Song, A. Romero, M. Mueller, V. Koltun, and D. Scaramuzza, “Reaching the limit in autonomous racing: Optimal control versus reinforcement learning,” _Science Robotics_ , 2023.
  * [17] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State, “Isaac gym: High performance gpu-based physics simulation for robot learning,” 2021.
  * [18] I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas _et al._ , “Solving rubik’s cube with a robot hand,” _arXiv preprint arXiv:1910.07113_ , 2019.
  * [19] E. Kaufmann, A. Loquercio, R. Ranftl, M. Müller, V. Koltun, and D. Scaramuzza, “Deep drone acrobatics,” in _Proceedings of Robotics: Science and Systems_ , Corvalis, Oregon, USA, July 2020.
  * [20] Y. Song, K. Shi, R. Penicka, and D. Scaramuzza, “Learning perception-aware agile flight in cluttered environments,” in _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2023, pp. 1989–1995.
  * [21] F. Sadeghi and S. Levine, “CAD2RL: real single-image flight without a single real image,” in _Robotics: Science and Systems XIII, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA, July 12-16, 2017_ , 2017\.
  * [22] P. Foehn, E. Kaufmann, A. Romero, R. Penicka, S. Sun, L. Bauersfeld, T. Laengle, G. Cioffi, Y. Song, A. Loquercio, and D. Scaramuzza, “Agilicious: Open-source and open-hardware agile quadrotor for vision-based flight,” _Science Robotics_ , vol. 7, no. 67, p. eabl6259, 2022. [Online]. Available: https://www.science.org/doi/abs/10.1126/scirobotics.abl6259
  * [23] L. Metz, C. D. Freeman, S. S. Schoenholz, and T. Kachman, “Gradients are not all you need,” _arXiv preprint arXiv:2111.05803_ , 2021.
  * [24] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2012, pp. 5026–5033.
  * [25] Y. Song, S. Naji, E. Kaufmann, A. Loquercio, and D. Scaramuzza, “Flightmare: A flexible quadrotor simulator,” in _Conference on Robot Learning_. PMLR, 2021, pp. 1147–1157.
  * [26] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “PyTorch: An Imperative Style, High-Performance Deep Learning Library,” 2019, arXiv preprint 1912.01703.
  * [27] R. Girshick, “Fast R-CNN,” in _2015 IEEE International Conference on Computer Vision (ICCV)_ , 2015, pp. 1440–1448.
  * [28] Y. Bengio, P. Simard, and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,” _IEEE transactions on neural networks_ , vol. 5, no. 2, pp. 157–166, 1994, publisher: IEEE.
  * [29] B. Shi, X. Bai, and C. Yao, “An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 39, no. 11, pp. 2298–2304, 2016, publisher: IEEE.
  * [30] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using RNN encoder-decoder for statistical machine translation,” _arXiv preprint arXiv:1406.1078_ , 2014\.
  * [31] “Betaflight - Pushing the Limits of UAV Performance - Betaflight,” Jul. 2023. [Online]. Available: https://betaflight.com
  * [32] “Mango Pi official website - Tiny and Elegant SBC - mqquad,” Aug. 2023\. [Online]. Available: https://mangopi.org/mqquad
  * [33] D. Scaramuzza, M. C. Achtelik, L. Doitsidis, F. Friedrich, E. Kosmatopoulos, A. Martinelli, M. W. Achtelik, M. Chli, S. Chatzichristofis, L. Kneip, D. Gurdan, L. Heng, G. H. Lee, S. Lynen, M. Pollefeys, A. Renzaglia, R. Siegwart, J. C. Stumpf, P. Tanskanen, C. Troiani, S. Weiss, and L. Meier, “Vision-controlled micro flying robots: From system design to autonomous navigation and mapping in gps-denied environments,” _IEEE Robotics and Automation Magazine_ , vol. 21, no. 3, pp. 26–40, 2014.
  * [34] S. Liu, K. Mohta, N. Atanasov, and V. Kumar, “Search-based motion planning for aggressive flight in se (3),” _IEEE Robotics and Automation Letters_ , vol. 3, no. 3, pp. 2439–2446, 2018, publisher: IEEE.
  * [35] B. Zhou, F. Gao, L. Wang, C. Liu, and S. Shen, “Robust and efficient quadrotor trajectory generation for fast autonomous flight,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 4, pp. 3529–3536, 2019.
  * [36] M. Neunert, C. De Crousaz, F. Furrer, M. Kamel, F. Farshidian, R. Siegwart, and J. Buchli, “Fast nonlinear model predictive control for unified trajectory optimization and tracking,” in _2016 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2016, pp. 1398–1404.
  * [37] D. Falanga, P. Foehn, P. Lu, and D. Scaramuzza, “Pampc: Perception-aware model predictive control for quadrotors,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 1–8.
  * [38] J. Ji, Z. Wang, Y. Wang, C. Xu, and F. Gao, “Mapless-planner: A robust and fast planning framework for aggressive autonomous flight without map fusion,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 6315–6321.
  * [39] F. Gao, W. Wu, W. Gao, and S. Shen, “Flying on point clouds: Online trajectory generation and autonomous navigation for quadrotors in cluttered environments,” _Journal of Field Robotics_ , 2018.
  * [40] J. Tordesillas and J. P. How, “Mader: Trajectory planner in multiagent and dynamic environments,” _IEEE Transactions on Robotics_ , vol. 38, no. 1, pp. 463–476, 2021.
  * [41] C. Vorbach, R. Hasani, A. Amini, M. Lechner, and D. Rus, “Causal navigation by continuous-time neural networks,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 12 425–12 440, 2021.
  * [42] G. Li, M. Mueller, V. Casser, N. Smith, D. L. Michels, and B. Ghanem, “Oil: Observational imitation learning,” _arXiv preprint arXiv:1803.01129_ , 2018\.
  * [43] E. Kaufmann, A. Loquercio, R. Ranftl, A. Dosovitskiy, V. Koltun, and D. Scaramuzza, “Deep drone racing: Learning agile flight in dynamic environments,” in _Conference on Robot Learning_. PMLR, 2018, pp. 133–145.
  * [44] A. Loquercio, E. Kaufmann, R. Ranftl, A. Dosovitskiy, V. Koltun, and D. Scaramuzza, “Deep drone racing: From simulation to reality with domain randomization,” _IEEE Transactions on Robotics_ , vol. 36, no. 1, pp. 1–14, 2019.
  * [45] T. Wang and D. E. Chang, “Robust navigation for racing drones based on imitation learning and modularization,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 13 724–13 730.
  * [46] J. Fu, Y. Song, Y. Wu, F. Yu, and D. Scaramuzza, “Learning deep sensorimotor policies for vision-based autonomous drone racing,” _arXiv preprint arXiv:2210.14985_ , 2022.
  * [47] J. Liang and M. C. Lin, “Differentiable physics simulation,” in _ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations_ , 2020.
  * [48] Y. Hu, L. Anderson, T.-M. Li, Q. Sun, N. Carr, J. Ragan-Kelley, and F. Durand, “Difftaichi: Differentiable programming for physical simulation,” _arXiv preprint arXiv:1910.00935_ , 2019.
  * [49] Y. Hu, J. Liu, A. Spielberg, J. B. Tenenbaum, W. T. Freeman, J. Wu, D. Rus, and W. Matusik, “Chainqueen: A real-time differentiable physical simulator for soft robotics,” in _2019 International conference on robotics and automation (ICRA)_. IEEE, 2019, pp. 6265–6271.
  * [50] J. M. Bern, Y. Schnider, P. Banzet, N. Kumar, and S. Coros, “Soft robot control with a learned differentiable model,” in _2020 3rd IEEE International Conference on Soft Robotics (RoboSoft)_. IEEE, 2020, pp. 417–423.
  * [51] J. Ren, C. Yu, S. Chen, X. Ma, L. Pan, and Z. Liu, “Diffmimic: Efficient motion mimicking with differentiable physics,” 2023.
  * [52] K. M. Jatavallabhula, M. Macklin, F. Golemo, V. Voleti, L. Petrini, M. Weiss, B. Considine, J. Parent-Levesque, K. Xie, K. Erleben, L. Paull, F. Shkurti, D. Nowrouzezahrai, and S. Fidler, “gradsim: Differentiable simulation for system identification and visuomotor control,” _International Conference on Learning Representations (ICLR)_ , 2021.
  * [53] Y. Song, S. Kim, and D. Scaramuzza, “Learning quadruped locomotion using differentiable simulation,” _arXiv preprint arXiv:2403.14864_ , 2024.
  * [54] G. Cioffi, L. Bauersfeld, E. Kaufmann, and D. Scaramuzza, “Learned inertial odometry for autonomous drone racing,” _IEEE Robotics and Automation Letters_ , vol. 8, no. 5, pp. 2684–2691, 2023.
  * [55] E. Kaufmann, L. Bauersfeld, A. Loquercio, M. Müller, V. Koltun, and D. Scaramuzza, “Champion-level drone racing using deep reinforcement learning,” _Nature_ , vol. 620, no. 7976, pp. 982–987, 2023.
  * [56] T. Qin, P. Li, and S. Shen, “VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator,” _IEEE Transactions on Robotics_ , vol. 34, no. 4, pp. 1004–1020, 2018.
  * [57] S. Shah, D. Dey, C. Lovett, and A. Kapoor, “AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles,” in _Field and Service Robotics_ , 2017, arXiv preprint arXiv:1705.05065. [Online]. Available: https://arxiv.org/abs/1705.05065
  * [58] X. Zhou, J. Zhu, H. Zhou, C. Xu, and F. Gao, “Ego-swarm: A fully autonomous and decentralized quadrotor swarm system in cluttered environments,” in _2021 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2021, pp. 4101–4107.
  * [59] S. Ross, G. Gordon, and D. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” in _Proceedings of the fourteenth international conference on artificial intelligence and statistics_. JMLR Workshop and Conference Proceedings, 2011, pp. 627–635.
  * [60] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * [61] H. J. Suh, M. Simchowitz, K. Zhang, and R. Tedrake, “Do differentiable simulators give better policy gradients?” in _International Conference on Machine Learning_. PMLR, 2022, pp. 20 668–20 696.
  * [62] A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan, Q. V. Le, and H. Adam, “Searching for MobileNetV3,” 2019, arXiv preprint arXiv:1905.02244.