# Quad2Plane: An Intermediate Training Procedure for Online Exploration in
Aerial Robotics via Receding Horizon Control

Alexander Quessy1 and Thomas Richardson2 1Alexander Quessy is with Department
of Aerospace Engineering, Queen’s Building, University of Bristol, University
Walk, Bristol BS8 1TR<EMAIL_ADDRESS>corresponding author2Thomas
Richardson is with Department of Aerospace Engineering, Queen’s Building,
University of Bristol, University Walk, Bristol BS8 1TR
<EMAIL_ADDRESS>
###### Abstract

Data driven robotics relies upon accurate real-world representations to learn
useful policies. Despite our best-efforts, zero-shot sim-to-real transfer is
still an unsolved problem, and we often need to allow our agents to explore
online to learn useful policies for a given task. For many applications of
field robotics online exploration is prohibitively expensive and dangerous,
this is especially true in fixed-wing aerial robotics. To address these
challenges we offer an intermediary solution for learning in field robotics.
We investigate the use of dissimilar platform vehicle for learning and offer a
procedure to mimic the behavior of one vehicle with another. We specifically
consider the problem of training fixed-wing aircraft, an expensive and
dangerous vehicle type, using a multi-rotor host platform. Using a Model
Predictive Control approach, we design a controller capable of mimicking
another vehicles behavior in both simulation and the real-world.

##  I INTRODUCTION

In the past decade Deep Reinforcement Learning has proven to be an effective
tool to solve a variety of simulated [1][2] and real-world [3][4] control
tasks. Real-world robotic learning is plagued by a host of practical issues
[5] such as: resets, state-estimation and platform integrity. These practical
limitations are easily solved by learning in simulation, but this often leads
to a reality gap, where the policy learned by the agent learns to exploit
attributes that are not present in the real-world, leading to poor
performance. A common robotic learning pipeline is therefore to complete the
majority of training in simulation, where demonstrations are cheap, and then
fine-tune the learned policy on a real-world robot.

This approach works well in robotic domains where the vehicle can safely
explore the environment without significant constraints, such as gripper [6]
robots. However, in many field-robotic settings it is not safe to allow the
robot to explore and fine-tune a policy in the real-world, as the vehicle can
damage itself and 3rd parties. This is a challenge that is particularly
constraining in fixed-wing aerial robotics, where the platform is often large
and financially expensive. Further, we are often most interested in fine-
tuning policies when the aircraft is slow and close to the ground [7] [8],
where it is most likely to cause damage. This makes real-world fixed-wing
robotic learning difficult constraining most research to simulation [9].

In comparison, multi-rotor drones are ubiquitous in aerial robotic research
[10], and have been used for online robotic learning in a variety of safe [11]
and constrained real-world environments [12]. Fixed-wing aircraft typically
have superior range, endurance and load-carrying capabilities than rotary
aircraft. However, this is not a constraint on the majority of learning tasks,
because most stochastic control problems encountered by the robot, where using
a DRL based controller would be useful, occur at the beginning or end of a
flight and carrying a payload does not effect the learning process. Along with
being safer, multi-rotors offer several distinct advantages, compared to
fixed-wing aircraft, for online learning:

  * •

Multi-rotors can hover and translate linearly in all three flight-axis, making
it easier to reset the learning process online. [13].

  * •

Multi-rotors are not as geometrically constrained as fixed-wing aircraft,
allowing for easier installation of compute devices and sensors. This makes it
easier to get accurate state-estimation and to perform online policy
optimization [14].

  * •

Multi-rotors have a larger low-speed flight envelope, easily being able to
hover and navigate tight, constrained environments[15]. This allows us to
train the robot in scenarios closer to the platforms limits without the risk
of crashing, opening the door for safe-RL [16] and lifelong learning [17].

Given how much easier it is to learn control policies on multi-rotors than
fixed-wing aircraft: can a multi-rotor mimic the mechanics of a fixed-wing
aircraft? This would allow us to learn useful policies in the real-world,
without the burden of reset-ability and safety imposed by a fixed-wing
aircraft. All whilst providing access to the underlying real-world state-
observation distribution, including wind and imaging data, which is difficult
to entirely capture in simulation [18].

In this paper we present a general procedure to mimic the dynamics of a target
vehicle on a dissimilar platform vehicle. Specifically we consider the problem
of replicating the dynamics of a fixed-wing target vehicle on a multi-rotor
platform. We make the following contributions:

  * •

We pose the general platform-to-target control problem as a discrete time
dynamic problem in section III. We then design an optimal model based
controller to solve this problem in section IV.

  * •

In section V we provide a description of the implementation of our controller
for aerial robotics, along with the rationale behind the design of our
vehicles state based cost function. This offers useful insights into how we
need to design vehicles that are used for robotic learning.

  * •

In section VI we use a combination of simulated and real-world data to
validate our approach and describe the key limitations of our approach.

##  II RELATED WORK

The problem of transferring learned skills from simulation to the real-world
is well documented within the robotic learning community [19][20][21]. This
reality gap is caused by a distributional shift from simulated to real-world
observations, due to un-modelled physical effects such as noise in a collected
image or friction in an actuator. One approach to solve this problem is to
vary the state distribution of the simulation, either by randomizing the
domain [22][23] or dynamics [24]. However, Domain Randomization (DR) has some
key limitations: the creation of suitable simulated training environments is
time consuming, and deciding what to randomize requires care to ensure the
final distributions match. Additionally there is no guarantee that DR can
achieve zero-shot sim-to-real transfer [25].

Alternatively, demonstrations can be used to help fine-tune the policy
[26][27][28], but for many settings it is difficult to obtain expert
demonstrations to learn from. Learning in the real-world is perhaps the
simplest way to ensure we don’t have a reality gap, either by fine-tuning
[29], or without simulation altogether [30]. Unfortunately this requires our
robotic platform to be capable of safe exploration, something that cannot be
assured for our target vehicle. We offer an intermediary solution to this, by
learning on a surrogate vehicle, we contribute a procedure to train inherently
dangerous vehicles in the real-world safely.

Mimicking the dynamics of one vehicle with another has been used by the
aerospace community for flight crew training [31] and aircraft flying quality
evaluation [32] since the late 1950s [33]. These In Flight Trainer (IFT)
aircraft [34] [35] typically use linear state-space model reference adaptive
controllers based on $\mathcal{L}_{1}$ or LQR to mimic another aircraft’s
flight mechanics. This makes sense for IFT aircraft, where the objective is
typically to provide proprioceptive information to the flight crew. However,
it is desirable for our system to also be capable of replicating the vehicles
non-linear dynamics, linear state-space control is therefore inappropriate.

Model Predictive Control (MPC) is an effective method to control non-linear
systems [36], especially when constraints are imposed on exploration [37].
Classical MPC provides a complete trajectory to solve an optimal control
problem, for complex optimization functions, as is experienced in non-linear
control, this is often computationally intractable. Receding-horizon
differential dynamic programming [38], helps to address this by optimizing
over a shorter (receding) horizon, rather than the full control time of the
process. In this research we design a non-linear receding model predictive
controller [39], and find this is a procedure capable of mimicking the
dynamics of our target fixed-wing vehicle, provided sufficient compute, time-
horizon and model accuracy.

##  III PROBLEM SETTING

Whilst the primary focus of our work is in matching the dynamics of dissimilar
aircraft types, the procedure we develop is applicable to many other robotic
learning tasks. Consider a discrete time dynamic system as in

$\mathbf{x}_{t+1}=\mathbf{F}^{i}(\mathbf{x}_{t},\mathbf{u}_{t})$ (1)

Where $\mathbf{x}_{t}\in\mathbb{R}^{n}$ is the state of a vehicle $i$ at time
$t$, $\mathbf{u}\in\mathbb{R}^{m}$ is the input to the vehicle at time $t$,
and $\mathbf{F}$ is the vehicles state-transition. Under this framework we
consider target and platform vehicles with transition functions
$\mathbf{F}^{tar}$ and $\mathbf{F}^{plat}$ respectively. Our objective is
therefore to design a controller $\mathbf{G}^{t2p}$, as in (2), so that when
$\mathbf{F}^{plat}$ receives a control input $\mathbf{u}_{t}^{plat}$ it
undergoes the same state-transition as $\mathbf{F}^{tar}$, receiving an input
$\mathbf{u}_{t}^{tar}$, as shown in figure 1.

$\mathbf{u}_{t}^{plat}=\mathbf{G}^{t2p}(\mathbf{u}_{t}^{tar};\mathbf{x}_{t}^{plat})$
(2)

Figure 1: Problem Formulation Block Diagram

We assess the performance of $G^{t2p}$ using a loss function between the
target and platform trajectory terms
$S_{\tau}=\mathcal{L}(\tau_{\mathbf{x}^{plat}},\tau_{\mathbf{x}^{tar}})$.

##  IV MODEL PREDICTIVE CONTROL

We can frame the control design problem for $G^{t2p}$ as an optimal control
problem, for which receding horizon MPC offers a useful framework. We assume
that the transition of the vehicles dynamics $\mathbf{F}$ is deterministic,
and the model representation entirely captures the vehicles dynamics.

Consider the cost function $\mathcal{J}(\mathbf{u};\mathbf{x})$, composed of
state $\mathbf{q}(\mathbf{x})$ and control $\mathbf{p}(\mathbf{u})$ dependant
costs, as in

$\mathcal{J}(\mathbf{u};\mathbf{x})=\mathbf{q}(\mathbf{x})+\mathbf{p}(\mathbf{u})\
s.t.\ \mathbf{u}_{min}\leq\mathbf{u}\leq\mathbf{u}_{max}$ (3)

The objective is then to find the optimal control sequence $\mathbf{u}^{*}$
that minimizes $\mathcal{J}$ over a finite time-horizon $T$, subject to the
linear bounding constraint $[\mathbf{u}_{min},\mathbf{u}_{max}]$. To help
reduce the computational cost at each time $t$ the optimization problem is
warm-started with the previous control sequences solution, reducing the
computational cost of the online optimization problem.

Control sequence $\tau_{\mathbf{u}}^{tar}$ with finite length $T$ is the
control input sequence received by $G^{t2p}$. This is the command sequence
produced by the outer controller we are aiming to train. In a typical model-
free RL setting only a single next time-step would be provided and the MPC
time-horizon would effectively be one step. We found that this often causes
instability in our platform vehicle as the MPC controller is unable to solve
the non-linear control problem, effectively getting stuck in a local optima.
To learn an RL policy it is therefore necessary to rollout a control
trajectory $T$ steps ahead. When the policy is deployed online only the first
state from the policies action trajectory will be selected.

Algorithm 1, is a general MPC based procedure to mimic the dynamics of a
target vehicle.. We denote the target vehicle model and platform model as
$\hat{\mathbf{F}}^{tar}$ and $\hat{\mathbf{F}}$ respectively.

Algorithm 1 t2p-MPC

0: $\tau_{\mathbf{u}^{tar}}$: target control input sequence,
$\tau_{\mathbf{u}}$: last control prediction sequence (up to $T-1$)

0: $\mathbf{u}_{t+1}$: next control input to platform

1: $\tau_{\mathbf{x}^{tar}}\leftarrow\emptyset$

2: for $t$ to $T$ do

3:
$\hat{\mathbf{x}}_{t+1}^{tar}\leftarrow\hat{\mathbf{F}}^{tar}(\mathbf{x}_{t};\mathbf{u}_{t}^{tar})$;
add to $\tau_{\mathbf{x}^{tar}}$

4: end for

5: $\mathcal{J}\leftarrow 0$

6: while not converged do

7: for $t$ to $T$ do

8:
$\hat{\mathbf{x}}_{t}\leftarrow\hat{\mathbf{F}}(\mathbf{x}_{t};\mathbf{u}_{t})$

9: $j\leftarrow\mathbf{q}(\mathbf{\hat{x}}_{t})+\mathbf{p}(\mathbf{u}_{t})$

10: $\mathcal{J}\leftarrow\mathcal{J}+j$

11: end for

12: $\tau_{\mathbf{u}}$ $\leftarrow$ $\underset{\mathbf{u}}{\mathrm{argmin}}\
\mathcal{J}\ s.t.\ \mathbf{u}_{min}\leq\mathbf{u}\leq\mathbf{u}_{max}$

13: end while

##  V IMPLEMENTATION

###  V-A Simulation

For the target fixed-wing ($fw$) we use the x8 Skywalker model from [40], and
simulate the aircraft using JSBSim [41]. For the multi-rotor ($mr$) we provide
our own implementation contained within this papers associated repository
111https://github.com/AOS55/Drone2Plane. Both aircraft have the same 6 degree
of freedom 12 state representation form $\mathbf{x}$: 6 translational
components $[x,y,z;u,v,w]$ and 6 rotational components
$[roll,pitch,yaw;p,q,r]$. Both aircraft have 4 controls $\mathbf{u}$ with
normalized inputs for control-deflection and thrust:

  * •

The multi-rotor has 4 thrust control commands, one for each prop:
$\mathbf{u}^{mr}=[T_{0},T_{1},T_{2},T_{3}]$, where $T_{i}$ is bound from
$[0,1]$.

  * •

The fixed-wing airplane is modelled with 3 control surface commands and one
throttle command: $\mathbf{u}^{fw}$ = $[ail,elev,tla,rud]$, with bounds:
$[[-1,+1],[-1,+1],[0,+1],[-1,+1]]$.

The actual x8 airplane flown, shown in figure 2, has 2 full wing control
surfaces ($cs$) which are transformed into equivalent elevator & aileron
within the x8 wind tunnel model [40]. The x8’s large control surfaces provide
lots of control authority in pitch and roll, but the lack of vertical
tailplane results in poor directional control and balance.

Figure 2: Example platform multi-rotor vehicle and target fixed-wing x8
airplane. Aircraft body axis states are labelled on the multi-rotor.

Our multi-rotor simulator includes models for drag, motor-thrust and
gravitational effects. The vehicle states are updated with a two-step forward
euler method using the linear and rotational accelerations generated by these
forces. We specified the vehicles performance to satisfy the x8 target
platforms required flight envelope, principally by observing when the multi-
rotor’s motors became saturated when responding to a step-control disturbance
on the airplane. This produced a thrust to weight requirement of $\approx
2:1$, performance that is not unrealistic for many small to medium size off-
the-shelf quad-copters [42].

###  V-B MPC

For our MPC control task we use a state-dependent cost function
$\mathbf{q}(\mathbf{x})$ of the form

$\mathbf{q}(\mathbf{x})=\mathbf{w}_{\mathbf{x}}|\mathbf{x}^{fw}-\mathbf{x}^{mr}|^{2}$
(4)

Where the state weight matrix $w_{\mathbf{x}}$ has parameters

$\mathbf{w}_{\mathbf{x}}=[1,1,1,0,0,0,0,0,0,0,0,0]^{T}$ (5)

This corresponds to minimizing the $(x,y,z)$ position error between the fixed-
wing and multi-rotor aircraft. This is suitable for higher order fixed-wing
control tasks, based on temporal position based objectives [43]. If mimicking
rotational vehicle states is desirable, it would not be difficult to include a
gimbal controller to directly passthrough the fixed-wing rates from the model.
Allowing us to train a vision encoded policy as the attitude of the fixed-wing
aircraft has a direct relation to the perspective of the policy. For the
control-dependant cost function $\mathbf{p}(\mathbf{u})$ we apply a constant
cost weight to all thrust terms of $0.5$. This improves the convergence rate
when close to an optimal solution and helps to reduce jittering. To minimize
the cost function, $\underset{\mathbf{u}}{\mathrm{argmin}}$
$\mathcal{J}(\mathbf{u},\mathbf{x})\ s.t.\
\mathbf{u}_{min}\leq\mathbf{u}\leq\mathbf{u}_{max}$, we use the Sequential
Least Squares Programming (SLSQP) routine from the Python based SciPy
optimization library [44] with constraints
$[\mathbf{u}_{min},\mathbf{u}_{max}]\leftarrow[0,1]$. We use a constant time-
horizon $T$ of 1.0s.

##  VI EXPERIMENTS

To investigate the limitations of our surrogate learning approach we consider
the following procedures:

  * •

A disturbance in roll and pitch, to investigate the quad-copter’s limitations
to mimic the dynamics of the fixed-wing at the limits of the flight envelope.

  * •

A comparison to real-world flight data, to investigate our procedures ability
to mimic the aeroplane.

###  VI-A Disturbance Tracking

Figures 3 & 4 plot the 3 dimensional position and linear velocity states for
the two vehicles respectively. The orange dash-dot vertical line in both
figures is the point the disturbance was initiated. For the pitch and roll
disturbances we apply an instantaneous up elevator and left aileron deflection
for 0.2$s$ respectively. In both cases we use 50 % of the maximum control
deflection for the disturbance.

Figure 3: The multi-rotor (solid line) tracks the pitch disturbance well, but
becomes saturated when aiming to maintain level with the fixed-wing (dashed-
diamond line) following the roll disturbance. Figure 4: The multi-rotor
maintains a similar average speed throughout the manoeuvre in order to
minimize the position cost. Note, the multi-rotor does not track linear
velocity as part of its cost function.

Control saturation is shown following the disturbance in roll at 5 seconds on
figure 3 as the quad-copter encounters a slight (10m) departure from tracking
in altitude. The system effectively runs out of power required to maintain
rate of climb whilst turning. The controller is able to maintain higher power
for longer however and recaptures the control by effectively leading the
velocity of the fixed-wing aircraft, shown from 9.0s to 12.0s on figure 4. We
find this analysis useful to understand the limitations of our platform
vehicle, if tracking closely in this domain was desirable we could increase
the performance of our vehicle to maintain position.

Figure 5: The control input to each of the quad-copter’s 4 motors under the
pitch and roll disturbances. The roll disturbance is saturated from 5.5s to
10.0s

Figure 5 shows the control input required to mimic the disturbances of the
fixed wing for each disturbance. The noise at the beginning of each control
plot is caused by cold-starting the optimizer, as the control sequence
$\tau_{\mathbf{u}}$ starts with all values set to zero. The solution for the
optimizer is then progressively improved by warm-starting the optimizer with
the last best solution. Once converged to a local optima the variance in
control input decays and the noise in the system reduces. Importantly this
doesn’t cause the aircraft to diverge from the mimicked dynamics, despite
following a sub-optimal solution.

Figure 6: With a $\tau=\frac{1}{30}s$ lag, the controller response is noisier,
but tracking accuracy (not shown) is unchanged.

To simulate the effect of un-modelled dynamics we add a first-order lag filter
to the multi-rotor’s motors given as

$\mathbf{y}_{t}=(1-a)\mathbf{y}_{t-1}+a\mathbf{u}_{t}\ \ \
a=\frac{\delta_{t}}{\delta_{t}+T_{lag}}$ (6)

Where $\mathbf{y_{t}}$ is the command sent to the linear motor model at time
$t$, $\delta_{t}$ is the step-size of the simulation, and
$T_{lag}=\frac{1}{30}s$ is the size of the lag. But, we do not include the lag
model within the MPC model. We found this did not cause any significant
effects on the ability of the quad-copter to track the commanded signal but,
as expected, slightly increases the noise of the motor input command, shown in
figure 6.

###  VI-B Real-World Comparison

To validate our approach on a real-world dataset we use our algorithm to mimic
the motion of an x8 on a gas-sensing mission to the summit of Volcán de Fuego
in Guatemala using the same platform drone. We consider a demanding 60 second
climbing turn section, shown in figure 7.

Figure 7: Climbing Turn section starting at 15,000’, around Volcán de Fuego.
Solid orange segment indicates flight track used over the 60 second period.

We find the multi-rotor platform is capable of tracking the fixed-wing
demonstration with little to no error through the whole manoeuvre with
relatively small control noise, shown in figure 8. Interestingly the
controller approaches saturation during the level out from the turn, where it
needs to turn aggressively and continue the climb.

Figure 8: Position during manoeuvre in figure 7, aircraft dashed, multi-rotor
solid, showing little to no error. Lower plot shows the control commands
required to track the manoeuvre.

We find that the trajectory tracking error of the real world data in figure 8
is relatively small, with mean squared error in $[x,y,z]$ positions of
$[0.00182,0.00357,0.310]$ respectively.

##  VII FUTURE WORK & LIMITATIONS

A key limitation of our approach is the time taken to calculate trajectories
for our MPC loop. Even in the best case the frequency of our controller to
update on modern hardware is 10 to 100 times slower than is required to run
online. A large constraint to this is the need to use a sequential quadratic
programming solver, as the core algorithm is fundamentally serial and the
computational expense to calculate gradients and simulate model state roll-
outs is significant. Information Theoretic MPC methods such as MPPI [45] offer
a solution to this challenge, via a sampling based MPC procedure, based on KL
divergence and free energy. This allows us to take full advantage of parallel
compute and does not require gradients to be calculated for the optimization
procedure. Initially, we found that running MPPI, with the quad-copter
simulator on parallel CPU compute, provided insufficient samples to converge
to a stable solution. We therefore aimed to use a neural-network to represent
our model dynamics, allowing us to take advantage of accelerated parallel GPU
compute and collect many more samples faster. To train our network we
investigated bootstrapping from demonstrations collected with both random
actions and an MPC oracle along with simply training within the MPPI
framework. A unique challenge that stems from the quad-copters fundamental
instability is that random actions frequently lead to the aircraft rotating
into an uncontrolled state. The original MPPI paper [45] solves this on the
quad-copter task by including angular rates as part of the control input. We
found that whilst position and linear/angular velocities can be predicted with
small error, angles could only be learned to within $1^{c}$ of error, which is
far too large of an error to use with MPPI.

More recent Model-Based RL algorithms such as MBPO [46] or PETS [47] may offer
a better solution to solving our control problem, with their CMA-ES ensemble
maximum likelihood [48] based model learning procedure. Alternatively, we may
simply need to collect more demonstrations for supervised training, either
from the MPC oracle or human demonstrations.

Once this computational bottleneck is overcome we aim to experiment with the
controller in the real-world. Such as: online episodic reinforcement learning
and safety constrained RL tasks, for example flying at high speed around a
car-park or navigating to landing spots in built-up areas.

## References

  * [1] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” 2013.
  * [2] J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Abbeel, “Trust Region Policy Optimization,” arXiv:1502.05477 [cs], Apr. 2017.
  * [3] OpenAI, M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba, “Learning dexterous in-hand manipulation,” 2019.
  * [4] S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training of deep visuomotor policies,” J. Mach. Learn. Res., vol. 17, p. 1334–1373, jan 2016.
  * [5] H. Zhu, J. Yu, A. Gupta, D. Shah, K. Hartikainen, A. Singh, V. Kumar, and S. Levine, “The ingredients of real world robotic reinforcement learning,” in International Conference on Learning Representations, 2020.
  * [6] A. X. Lee, C. M. Devin, Y. Zhou, T. Lampe, K. Bousmalis, J. T. Springenberg, A. Byravan, A. Abdolmaleki, N. Gileadi, D. Khosid, C. Fantacci, J. E. Chen, A. Raju, R. Jeong, M. Neunert, A. Laurens, S. Saliceti, F. Casarini, M. Riedmiller, raia hadsell, and F. Nori, “Beyond pick-and-place: Tackling robotic stacking of diverse shapes,” in 5th Annual Conference on Robot Learning, 2021.
  * [7] R. J. Clarke, L. Fletcher, C. Greatwood, A. Waldock, and T. S. Richardson, “Closed-Loop Q-Learning Control of a Small Unmanned Aircraft,” in AIAA Scitech 2020 Forum, (Orlando, FL), American Institute of Aeronautics and Astronautics, Jan. 2020.
  * [8] L. Fletcher, R. Clarke, T. Richardson, and M. Hansen, “Reinforcement learning for a perched landing in the presence of wind,” in AIAA Scitech 2021 Forum, American Institution of Aeronautics and Astronautics, Jan. 2021.  2021 AIAA SciTech Forum ; Conference date: 04-01-2021 Through 15-01-2021.
  * [9] E. Bohn, E. M. Coates, S. Moe, and T. A. Johansen, “Deep reinforcement learning attitude control of fixed-wing uavs using proximal policy optimization,” 2019 International Conference on Unmanned Aircraft Systems (ICUAS), Jun 2019.
  * [10] X. Ding, P. Guo, K. Xu, and Y. Yu, “A review of aerial manipulation of small-scale rotorcraft unmanned robotic systems,” Chinese Journal of Aeronautics, vol. 32, pp. 200–214, Jan. 2019.
  * [11] A. Loquercio, A. I. Maqueda, C. R. del Blanco, and D. Scaramuzza, “Dronet: Learning to fly by driving,” IEEE Robotics and Automation Letters, vol. 3, no. 2, pp. 1088–1095, 2018.
  * [12] E. Kaufmann, A. Loquercio, R. Ranftl, A. Dosovitskiy, V. Koltun, and D. Scaramuzza, “Deep drone racing: Learning agile flight in dynamic environments,” in CoRL, 2018.
  * [13] B. Eysenbach, S. Gu, J. Ibarz, and S. Levine, “Leave no trace: Learning to reset for safe and autonomous reinforcement learning,” in International Conference on Learning Representations, 2018.
  * [14] J. Ibarz, J. Tan, C. Finn, M. Kalakrishnan, P. Pastor, and S. Levine, “How to train your robot with deep reinforcement learning: Lessons we have learned,” The International Journal of Robotics Research, vol. 40, pp. 698–721, Apr. 2021.
  * [15] J. V. Foster and D. Hartman, “High-Fidelity Multi-Rotor Unmanned Aircraft System (UAS) Simulation Development for Trajectory Prediction Under Off-Nominal Flight Dynamics,” in 17th AIAA Aviation Technology, Integration, and Operations Conference, (Denver, Colorado), American Institute of Aeronautics and Astronautics, June 2017.
  * [16] K. P. Srinivasan, B. Eysenbach, S. Ha, J. Tan, and C. Finn, “Learning to be safe: Deep rl with a safety critic,” ArXiv, vol. abs/2010.14603, 2020.
  * [17] S. Thrun, Lifelong Learning Algorithms, p. 181–209.  USA: Kluwer Academic Publishers, 1998.
  * [18] DeepMind, “https://deepmind.com/blog/announcements/mujoco.”  2022-02-14.
  * [19] M. E. Taylor and P. Stone, “Transfer learning for reinforcement learning domains: A survey,” J. Mach. Learn. Res., vol. 10, p. 1633–1685, dec 2009\.
  * [20] S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Transactions on Knowledge and Data Engineering, vol. 22, no. 10, pp. 1345–1359, 2010.
  * [21] C. Tan, F. Sun, T. Kong, W. Zhang, C. Yang, and C. Liu, “A survey on deep transfer learning,” 2018.  cite arxiv:1808.01974Comment: The 27th International Conference on Artificial Neural Networks (ICANN 2018).
  * [22] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 23–30, 2017.
  * [23] F. Sadeghi and S. Levine, “Cad2rl: Real single-image flight without a single real image,” in 13th converences on Robotics: Science and Systems, pp. 34–44, 07 2017.
  * [24] F. Muratore, C. Eilers, M. Gienger, and J. Peters, “Data-efficient domain randomization with bayesian optimization,” IEEE Robotics and Automation Letters, vol. 6, pp. 911–918, 2021.
  * [25] E. Valassakis, Z. Ding, and E. Johns, “Crossing the gap: A deep dive into zero-shot sim-to-real transfer for dynamics,” 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5372–5379, 2020.
  * [26] M. Hazara and V. Kyrki, “Transferring generalizable motor primitives from simulation to real world,” IEEE Robotics and Automation Letters, vol. 4, no. 2, pp. 2172–2179, 2019.
  * [27] D. Gandhi, L. Pinto, and A. K. Gupta, “Learning to fly by crashing,” 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3948–3955, 2017.
  * [28] Y. Lu, K. Hausman, Y. Chebotar, M. Yan, E. Jang, A. Herzog, T. Xiao, A. Irpan, M. Khansari, D. Kalashnikov, and S. Levine, “AW-opt: Learning robotic skills with imitation andreinforcement at scale,” in 5th Annual Conference on Robot Learning, 2021.
  * [29] R. C. Julian, B. Swanson, G. S. Sukhatme, S. Levine, C. Finn, and K. Hausman, “Never stop learning: The effectiveness of fine-tuning in robotic reinforcement learning,” in CoRL, 2020.
  * [30] C. Sun, J. Orbik, C. M. Devin, B. H. Yang, A. Gupta, G. Berseth, and S. Levine, “Fully autonomous real-world reinforcement learning with applications to mobile manipulation,” in 5th Annual Conference on Robot Learning, 2021\.
  * [31] Calspan, “https://calspan.com/aerospace/advanced-flight-test-training.”  2022-02-14.
  * [32] R. P. Harper and G. E. Cooper, “Handling qualities and pilot evaluation,” Journal of Guidance, Control, and Dynamics, vol. 9, pp. 515–529, Sept. 1986\.
  * [33] M. Shafer, “In-flight simulation studies at the nasa dryden flight research facility,” Scientific and Technical Information Program, 1992.
  * [34] R. S. Edmonson and J. Kemper, “Adaptive flight control systems on calspan learjet,” in AIAA Scitech 2019 Forum, (San Diego, California), American Institute of Aeronautics and Astronautics, Jan. 2019.
  * [35] M. E. Knapp, T. Berger, M. Tischler, and M. C. Cotting, “Development of a Full Envelope Flight Identified F-16 Simulation Model,” in 2018 AIAA Atmospheric Flight Mechanics Conference, (Kissimmee, Florida), American Institute of Aeronautics and Astronautics, Jan. 2018.
  * [36] D. Q. Mayne, “Model predictive control: Recent developments and future promise,” Automatica, vol. 50, no. 12, pp. 2967–2986, 2014.
  * [37] K. P. Wabersich and M. N. Zeilinger, “Safe exploration of nonlinear dynamical systems: A predictive safety filter for reinforcement learning,” ArXiv, vol. abs/1812.05506, 2018.
  * [38] T. Erez, K. Lowrey, Y. Tassa, V. Kumar, S. Kolev, and E. Todorov, “An integrated system for real-time model predictive control of humanoid robots,” in 2013 13th IEEE-RAS International Conference on Humanoid Robots (Humanoids), pp. 292–299, 2013.
  * [39] S. Gros, M. Zanon, R. Quirynen, A. Bemporad, and M. Diehl, “From linear to nonlinear mpc: bridging the gap via the real-time iteration,” International Journal of Control, vol. 93, pp. 62 – 80, 2020.
  * [40] K. Gryte, R. Hann, M. Alam, J. Rohac, T. Johansen, and T. Fossen, “Aerodynamic modeling of the skywalker x8 fixed-wing unmanned aerial vehicle,” pp. 826–835, 06 2018.
  * [41] M. Madden, Architecting a Simulation Framework for Model Rehosting.
  * [42] DJI, “https://www.dji.com/uk/dji-fpv/specs.”  2022-02-24.
  * [43] ICAO, Procedures for Air Navigation Services, Air Traffic Management, pp. 142–164.  999 Robert-Bourassa Boulevard, Montréal, Quebec, Canada H3C 5H7: International Civil Aviation Organization, 2016.
  * [44] P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, İ. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors, “SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python,” Nature Methods, vol. 17, pp. 261–272, 2020.
  * [45] G. Williams, N. Wagener, B. Goldfain, P. Drews, J. M. Rehg, B. Boots, and E. A. Theodorou, “Information theoretic mpc for model-based reinforcement learning,” in 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1714–1721, 2017.
  * [46] M. Janner, J. Fu, M. Zhang, and S. Levine, “When to trust your model: Model-based policy optimization,” in Advances in Neural Information Processing Systems (H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, eds.), vol. 32, Curran Associates, Inc., 2019.
  * [47] K. Chua, R. Calandra, R. McAllister, and S. Levine, “Deep reinforcement learning in a handful of trials using probabilistic dynamics models,” in Advances in Neural Information Processing Systems (S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, eds.), vol. 31, Curran Associates, Inc., 2018.
  * [48] N. Hansen and A. Ostermeier, “Completely derandomized self-adaptation in evolution strategies,” Evolutionary Computation, vol. 9, no. 2, pp. 159–195, 2001.