2023

1]Department of Mechanical and Aerospace Engineering, Monash University,
Wellington Road, Clayton, 3800, Victoria, Australia

2]Department of Electrical and Computer Systems Engineering, Monash
University, Wellington Road, Clayton, 3800, Victoria, Australia

# Reinforcement Learning for Shared Autonomy Drone Landings

Kal Backman<EMAIL_ADDRESS>Dana Kulić<EMAIL_ADDRESS>Hoam
Chung<EMAIL_ADDRESS>[  [

###### Abstract

Novice pilots find it difficult to operate and land unmanned aerial vehicles
(UAVs), due to the complex UAV dynamics, challenges in depth perception, lack
of expertise with the control interface and additional disturbances from the
ground effect. Therefore we propose a shared autonomy approach to assist
pilots in safely landing a UAV under conditions where depth perception is
difficult and safe landing zones are limited. Our approach is comprised of two
modules: a perception module that encodes information onto a compressed latent
representation using two RGB-D cameras and a policy module that is trained
with the reinforcement learning algorithm TD3 to discern the pilot’s intent
and to provide control inputs that augment the user’s input to safely land the
UAV. The policy module is trained in simulation using a population of
simulated users. Simulated users are sampled from a parametric model with four
parameters, which model a pilot’s tendency to conform to the assistant,
proficiency, aggressiveness and speed. We conduct a user study ($n=28$) where
human participants were tasked with landing a physical UAV on one of several
platforms under challenging viewing conditions. The assistant, trained with
only simulated user data, improved task success rate from 51.4% to 98.2%
despite being unaware of the human participants’ goal or the structure of the
environment a priori. With the proposed assistant, regardless of prior
piloting experience, participants performed with a proficiency greater than
the most experienced unassisted participants.

###### keywords:

Shared autonomy, Unmanned aerial vehicles, Reinforcement learning, Latent
representations

##  1 Introduction

Unmanned aerial vehicles (UAVs) are often deployed in reconnaissance,
inspection and search and rescue tasks (González-deSantos ., 2020; Sa ., 2015;
Morando ., 2022; Y. Wang ., 2019; Albanese ., 2022), due to their high
maneuverability in full 3D space. However their mobility comes at the cost of
increased teleoperation complexity. The teleoperation challenges include the
pilot’s perception of the UAV’s position relative to nearby objects
(Smolyanskiy Gonzalez-Franco, 2017; Kim ., 2020), coping with disturbances
caused from the ground effect (Kan ., 2019) and in understanding the mapping
of teleoperation controls to UAV movement. Due to these challenges it is
difficult for novice pilots to safely land a UAV, particularly in perceptually
challenging environments.

Although autonomous solutions have been proposed for UAV landing (Baca .,
2019; Feng ., 2018; Polvara ., 2020), they often require prior knowledge or
setup of the environment, or the specification of a known landing target.
Fully autonomous solutions are unable to dynamically adapt their objective in
response to context sensitive cues or external events within the environment
that are not observable with the robot’s sensors or a-priori specified by the
designers. In general the difficulties associated with replicating high-level
human decision making in a time and safety-critical context make teleoperation
control schemes preferred over full autonomy (Shaqura ., 2018; Perez-Grau .,
2017), despite the need for expert pilots. Therefore to reduce training costs
of potential pilots and improve the accessibility of UAV piloting for all, we
aim to develop an assistive landing strategy that allows novice pilots to land
at a proficiency at least equal to that of expert pilots.

Shared autonomy combines the control inputs of the pilot with that of an
artificial intelligence to collaboratively complete a set of objectives.
However, two prominent challenges when developing a shared autonomy system are
predicting the intent of the user and deciding the control outputs based on
the predicted user’s intent and control actions. In our previous work (Backman
., 2021), we proposed a shared autonomy system that learns to estimate the
pilot’s intent by training with simulated users and demonstrated it to be
effective in providing assistance to human pilots landing in simulated
environments.

The contributions of this paper are the following:

  * •

Rigorous validation in a physical environment with naive users. Prior shared
autonomy UAV works are either validated in simulated user studies (Backman .,
2021; Zhang ., 2021), provide physical demonstrations (Sa ., 2015; Perez-Grau
., 2017) or are evaluated with a small sample size (n=4) physical user study
(Reddy ., 2018) where the goal is explicitly stated. The proposed work is also
the first physical UAV shared autonomy system to account for multiple
ambiguous goals. Prior works (Reddy ., 2018; Sa ., 2015) are limited to a
single potential goal within the physical environment.

  * •

Reformulation of the state transition process for critics in reinforcement
learning shared autonomy systems as a fully observable Markov decision process
(MDP) instead of the traditional partially observable Markov decision process
(POMDP) by including privileged information of the simulated user’s hidden
internal state during training. The proposed reformulation significantly
reduces model convergence time and improves policy performance, as MDPs offer
an easier to solve problem compared to POMDPs.

  * •

Empirical comparison of the proposed approach with related shared autonomy
works Javdani . (2018); Reddy . (2018), comparing user and assistant model
performance and limitations for each approach for UAV landing tasks. The
experiment highlights the proposed approach’s superior performance and
robustness to unseen conditions without requiring prior knowledge of the
environment.

###  1.1 Related work

Previous works related to UAV landing primarily focus on full autonomy over
shared autonomy. Polvara . (2020); Xia . (2021) demonstrates an autonomous UAV
visually seeking and landing on marked landing zones. Polvara . (2020) uses a
sequence of deep Q-learning networks (DQNs) trained in an end-to-end manner in
simulation. The model was transferred to reality successfully but learning the
large state-space directly from images required two separate networks to be
trained to counter the sample inefficiency from sparse rewards. Xia . (2021)
uses a Markov decision process for collision avoidance to autonomously detect
landing zones and plan a path towards them. However the approach is only
demonstrated in simulation and requires the landing target location to be
known priori. Both approaches (Polvara ., 2020; Xia ., 2021) have access to a
discrete set of actions, limiting the manoeuvrability of the UAV.

Continuous action approaches are demonstrated in (Rodriguez-Ramos ., 2019; Shi
., 2019) where Rodriguez-Ramos . (2019) implements the continuous
reinforcement learning algorithm DDPG (Lillicrap ., 2016) to land on a moving
platform while Shi . (2019) learns an unknown disturbance force caused by the
ground effect to be fed into a non-linear trajectory tracking controller. Both
approaches facilitate continuous control for UAV landing but require the
desired landing position to be known a priori, limiting the feasibility in
unexplored environments.

Autonomous landing zone proposal approaches have been developed to eliminate
the need of explicitly specifying the landing target a priori (Maturana
Scherer, 2015; Carney ., 2019; Kaljahi ., 2019). Maturana Scherer (2015) uses
globally registered LiDAR point clouds to build a volumetric density map, from
which a convolutional neural network is used to create a voxelised map of safe
landing area probabilities. Carney . (2019) uses publicly available population
density, elevation, terrain ruggedness and land classification data in a
heuristic algorithm to detect viable landing strips from which a single
candidate is selected using a weighting algorithm. Kaljahi . (2019) applies a
series of Gabor filters to RGB images from which a Markov chain code process
is used to cluster and classify pixels. However such landing proposal
approaches do not take into consideration the potential objectives or
preferences of the pilot and are yet to be validated for use in UAV landings.

Shared autonomy approaches circumvent the need for predefined landing zones
due to the AI assistant inferring the landing position from the pilot’s
actions. Reddy . (2018) implements a shared autonomy system using a model free
DQN to assist participants (n=12) safely landing in the lunar lander game,
where the location of safe landing zones are only known to the user. However
when tested in a user study with a physical drone (n=4), the landing pad
location is included within the network’s state space.

Researchers have also proposed to estimate the intent of pilots using eye gaze
tracking (Pfeiffer ., 2022) and the flight trajectory of pilots (Patrikar .,
2022). However, these models are trained in a supervised manner with ground
truth labels and are yet to be implemented in the shared autonomy context.

Our previous work (Backman ., 2021) demonstrated a shared autonomy system
where pilots were tasked to land a simulated UAV on one of several platforms
with the help of an assistant. The assistant comprised of two modules; a
perception module which encodes visual information onto a latent vector and a
policy module trained using the reinforcement learning algorithm TD3 (Fujimoto
., 2018) to augment the pilot’s actions.

Our proposed approach builds upon our prior work (Backman ., 2021), making
significant changes to the perception and policy modules with a focus on real-
world implementation without the need for real-world data. The proposed
perception module encodes information from two RGB-D cameras for greater field
of view whilst maintaining a compressed latent embedding by combining the
camera images with a novel camera projection model that is only attainable in
simulation. For improved generality, the latent embedding encodes a
segmentation map indicating potential safe landing areas in comparison to a
single platform prior in Backman . (2021). For robustness against sensor
noise, noise generating functions are dynamically applied per training batch,
making the module highly invariant to disturbances.

The proposed policy module is improved with the inclusion of a LSTM-cell to
address the prior inconsistency concerns shared by participants. The simulated
user model is expanded to capture how a human pilot operates a joystick
controller compared to keyboard presses in Backman . (2021). Model convergence
time is significantly reduced by providing hidden information of the simulated
user’s intent that can only be attained in simulation to the critic, as well
as a concurrent exploration training architecture. Overall task complexity is
increased due to additional safe landing requirements, unrestricted platform
layouts compared to one-dimensional platform configurations in Backman .
(2021) and user study validation in an unseen physical environment. We
demonstrate that the proposed approach using the perceptual and policy modules
trained only in simulation are able to successfully assist human pilots during
physical shared autonomy drone landings.

###  1.2 Problem statement and model overview

We consider an unknown environment in which a UAV is operating, which includes
a number of suitable landing locations, named _landing platforms_. Our
objective is to develop an assistant that aids pilots of all proficiencies to
safely land a UAV on a user-selected landing platform, when neither the number
nor locations of landing platforms are known to the assistant a-priori. A
landing is considered safe when the UAV remains at rest atop of a landing
platform and the vertical and horizontal landing velocities are below a given
threshold. The pilot controls the UAV by providing linear XYZ velocities using
a radio joystick controller for the onboard flight controller to follow. Due
to maintaining a safe viewing distance, the pilot’s ability to perceive the
depth of the UAV is hindered.

Figure 1: Overview of system architecture.

Our approach is summarised in Fig. 1, and consists of two learning components:
(i) a compressed latent representation of the environment to perceive the
location of potentially safe landing sites and (ii) a policy network to
provide control inputs to assist the pilot in safely landing. The first
component takes noisy images from a downwards facing stereo RGB-D camera pair
and encodes information of the structure of the scene and location of safe
landing zones onto a low-dimensional latent vector. The second component takes
the current state of the UAV which includes the latent vector representation,
the UAV’s dynamics and the pilot’s current action, as well as the previous
state concatenated with the assistant’s previous action to output a linear XYZ
target velocity. The final target velocity for the UAV’s flight controller to
follow is determined by averaging the action taken by the pilot and the
assistant.

##  2 Learning latent space representation

The purpose of learning a latent space embedding is to reduce the
dimensionality of the input state-space the policy network learns from. As the
policy network gathers experience by interacting with the environment in real
time, a reduction in the dimensionality of the state space leads to
accelerated convergence (Curran ., 2017).

Spurr . (2018) introduces a cross-modal variational auto-encoder (CM-VAE)
which trains a latent vector representation from multiple data sources. Given
an input data source  $\ignorespaces x_{k}$ where  $\ignorespaces k$
represents the modality such as an RGB-D image or an arbitrary sensor reading,
the encoder  $\ignorespaces q_{k}$ embeds  $\ignorespaces x_{k}$ onto a latent
representation as a vector of means  $\ignorespaces\mu$ and variances
$\ignorespaces\sigma^{2}$ of a normal distribution. Decoder  $\ignorespaces
p_{l}$ samples the latent embedding  $\ignorespaces
z\sim\mathcal{N}(\mu,\sigma^{2})$ then subsequently reconstructs
$\ignorespaces z$ to the desired output  $\ignorespaces y_{l}$ where
$\ignorespaces l$ denotes the output modality. As encoder  $\ignorespaces
q_{k}$ maps  $\ignorespaces x_{k}$ onto a normal distribution, the Kullback-
Leibler divergence is used as a regularizing term to enforce that the
properties of the target distribution are met, while information is encoded
onto the latent vector by minimising the reconstruction loss between the
predicted output  $\ignorespaces\hat{y}_{l}$ and the true output
$\ignorespaces y_{l}$ .

###  2.1 CM-VAE implementation

Our implementation of the CM-VAE architecture differs from our previous work
(Backman ., 2021) by including two input RGB-D images taken from cameras
situated at the front  $\ignorespaces x_{F}$ = [  $\ignorespaces x_{FRGB}$ ,
$\ignorespaces x_{FD}$ ] and back  $\ignorespaces x_{B}$ = [  $\ignorespaces
x_{BRGB}$ ,  $\ignorespaces x_{BD}$ ] of the UAV facing downwards. The cameras
are strategically placed such that the encoded field of view is maximised over
the pilot’s depth axis, the axis of greatest uncertainty. The CM-VAE
reconstructs two output modalities in the form of a combined depth map
$\ignorespaces y_{D}$ and a binary segmentation map  $\ignorespaces y_{S}$
that classifies safe-to-land areas, where a pixel is considered safe-to-land
given that all four legs of the UAV are able to contact the surface without
the rest of the UAV intersecting the environment. The reconstructed output
$\ignorespaces y_{S}$ replaces  $\ignorespaces y_{XYZ}$ , the relative
position of the nearest landing platform in our prior work (Backman ., 2021)
as a more general approach to defining potential landing zones, capable of
representing multiple potential landing zones compared to a single platform.
Both  $\ignorespaces x_{F}$ and  $\ignorespaces x_{B}$ are passed through a
Siamese feature extraction layer  $\ignorespaces F_{DN}$ using DroNet
(Loquercio ., 2018) from which the resultant vectors are concatenated and
encoded onto the latent embedding with encoder  $\ignorespaces q_{RGBD}$ i.e.
$\ignorespaces[\mu,\sigma^{2}]=q_{RGBD}([F_{DN}(x_{F}),F_{DN}(x_{B})])$ . To
reconstruct  $\ignorespaces y_{D}$ and  $\ignorespaces y_{S}$ , a sample is
taken from the latent embedding  $\ignorespaces
z\sim\mathcal{N}(\mu,\sigma^{2})$ which is passed through decoders
$\ignorespaces p_{D}$ and  $\ignorespaces p_{S}$ :
$\ignorespaces\hat{y}_{D}=p_{D}(z)$ and  $\ignorespaces\hat{y}_{S}=p_{S}(z)$ .

A consolidated latent representation for stereo RGB-D images is chosen
compared to individually embedding  $\ignorespaces x_{F}$ and  $\ignorespaces
x_{B}$ as  $\ignorespaces\mu_{F}$ and  $\ignorespaces\mu_{B}$ , then
reconstructing the depths separately as  $\ignorespaces\hat{y}_{FD}$ and
$\ignorespaces\hat{y}_{BD}$ , in order to create a lean latent embedding by
reducing redundant information that would be shared amongst both cameras’
field of view. This also increases robustness against image noise as degraded
visual information from one camera can be supplemented with the other. To
achieve a consolidated latent representation we implement a novel camera
projection model to generate  $\ignorespaces y_{D}$ and  $\ignorespaces y_{S}$
that encompasses the entire field of view of the input stereo image pair. The
combined image is sub-divided into 3 segments, (i) front-camera and (ii) back-
camera segment, which follows the standard pin-hole camera projection model
split in opposite halves along the width of the image. (iii) Middle segment
places  $\ignorespaces N_{c}$ intermediate 1-dimensional pin-hole cameras
evenly distributed between the front and back camera. The intermediate cameras
share the same image height and focal lengths of the front and back cameras
but have an image width of one. To generate the combined depth and
segmentation map we use an image size for the front and back cameras of 141w
$\ignorespaces\times$ 80h with  $\ignorespaces N_{c}$ = 18 intermediate
cameras for a combined image size of 160w  $\ignorespaces\times$ 80h, which is
subsequently down-sampled to 80w  $\ignorespaces\times$ 40h. The combined
image projection is demonstrated in Fig. 2.

Figure 2: Camera projection model used to combine stereo depth images into a
single image.

To generate data to train the network we use the AirSim (Shah ., 2018) Unreal
Engine plugin to capture RGB-D images of generated scenes. Each generated
scene consists of  $\ignorespaces N_{p}$ randomly generated landing platforms
that are placed ensuring adjacent platforms are not within a minimum distance
threshold. Lighting conditions are altered and textures are applied to the
platforms, walls and floor from a database of 500 materials. For each scene, a
total of 20 front and back RGB-D images, segmentation images and combined
depth images are taken by sweeping a camera through the environment. As images
taken from AirSim are free of noise which is not indicative of real-life
sensors, a noise generating function  $\ignorespaces G_{n}$ is used to
generate  $\ignorespaces x_{F}$ and  $\ignorespaces x_{B}$ given a clean input
batch of depth and RGB images. For greater domain randomisation, noised images
are dynamically generated during training by parallel worker threads that send
the resultant noised image batches to the network training thread.

Each datapoint within a training batch consists of a ground truth combined
depth map, segmentation map and a pair of noisy front and back RGB-D images.
Losses used can be categorized into two categories, (i) those that aim to
embed information onto the latent space and (ii) those that aim to condition
the latent space. For (i) we use the mean squared error between the true
combined depth map and the estimated depth map
$\ignorespaces(y_{D},\hat{y}_{D})$ with a weighting of 1.0 and the binary
cross entropy loss between the true segmentation map and the estimated
segmentation map  $\ignorespaces(y_{S},\hat{y}_{S})$ with a weighting of 0.5.
For (ii) the Kullback-Leibler divergence loss is given a weighting of 4.0. An
overview of the CM-VAE architecture can be seen in Fig. 3.

Figure 3: Overview of the CM-VAE architecture.

##  3 Policy learning - TD3

The aim of the policy network is to assist pilots in safely landing a UAV on a
desired platform by providing a target velocity that is averaged with the
pilot’s current input, which the onboard flight controller is to follow. We
define our problem as a partially observable Markov decision process (POMDP)
based on Javdani . (2018)’s formalization of shared autonomy systems. We
define the set of all possible states as  $\ignorespaces\mathcal{S}$ , where
the pilot’s goal is treated as a hidden state that must be inferred by the
assistant through the observation function
$\ignorespaces\mathcal{O}:\mathcal{S}\times\Omega\rightarrow[0,1]$ where
$\ignorespaces\Omega$ is the set of observations. The set of all actions
available to the assistant is denoted as  $\ignorespaces\mathcal{A}$ . The
state transition process is defined as a stochastic process:  $\ignorespaces
T:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1]$ , due to the
uncertainty in predicting UAV dynamics from unobservable forces such as the
turbulence caused from the ground effect. The reward function is defined as
$\ignorespaces
R:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}$ , where
the aim of reinforcement learning is to find the closest approximation to the
optimal policy:  $\ignorespaces\pi:\mathcal{S}\times\mathcal{A}$ which
maximises the expected future reward with discount factor
$\ignorespaces\gamma\in[0,1]$ .

For optimal policy approximation, we follow our previous work (Backman .,
2021) and use the reinforcement learning algorithm TD3 (Fujimoto ., 2018) due
to its model-free continuous action space control whilst alleviating the
instability concerns of its predecessor DDPG (Lillicrap ., 2016).

As TD3 is an actor-critic approach, the actor follows the aforementioned POMDP
definition where the partial observability originates from inferring the
pilot’s goal without explicit definition. As the critic is an artifact of the
training process that is not intended to be used during application, we model
the problem for the critic as an MDP by providing the pilot’s goal information
within the observation space  $\ignorespaces\mathcal{O}$ to reduce problem
complexity for the critic.

###  3.1 TD3 implementation

To train the assistant we utilize AirSim as our simulation environment for
modelling UAV dynamics. We generate a variety of training scenes with varying
location and number of potential landing platforms. Each generated scene
consists of  $\ignorespaces N_{p}$ randomly generated landing platforms placed
either randomly such that adjacent platforms are not within a minimum distance
threshold, or are arranged in a grid-like pattern with random spacing.

####  3.1.1 Simulating Users

As training reinforcement learning algorithms takes substantial time, a
population of simulated users is developed instead of training with human
participants. We characterise a simulated user using four parameters:
$\ignorespaces\alpha\in[0,1]$ describes the simulated user’s conformance to
the assistant’s actions and models how likely a pilot will adopt the policy of
the assistant.  $\ignorespaces\beta\in[0,1]$ describes the user’s proficiency,
defined as the ability to improve one’s estimate of the goal by perceiving the
relative depth of the UAV to that of the landing platform. Both
$\ignorespaces\alpha$ and  $\ignorespaces\beta$ influence the simulated user’s
estimate of the current goal position by:

$\hat{G}_{i+1}=\hat{G}_{i}+\alpha\frac{a_{a}-a_{u}}{K_{\alpha}}+\beta\frac{G-\hat{G}_{i}}{K_{\beta}},$
(1)

where  $\ignorespaces a_{a}$ & $\ignorespaces a_{u}$ are the previous action
taken by the assistant and simulated user respectively, while  $\ignorespaces
K_{\alpha}$ & $\ignorespaces K_{\beta}$ are scaling constants. In our previous
work (Backman ., 2021), we used a 2 parameter model consisting of
$\ignorespaces\alpha$ and  $\ignorespaces\beta$ to model simulated users.
However, these two parameters alone do not sufficiently capture the
variability in user behaviour such as their desired speed and acceleration,
nor model how pilots operate a physical joystick interface. Here we introduce
two new parameters  $\ignorespaces\Psi\in[0,1]$ & $\ignorespaces\Phi\in[0,1]$
to model the simulated user’s dexterity in implementing fine motor control of
a continuous joystick controller compared to discrete keyboard inputs in
Backman . (2021).  $\ignorespaces\Psi$ describes how aggressively a pilot
accelerates their thumbs on a joystick controller, modeling their tendency to
generate smooth or sharp trajectories.  $\ignorespaces\Phi$ describes how
daring a user is in terms of their maximum desired flight speeds by modeling
how far a pilot is likely to push down on a joystick and their ability to
provide fine adjustments when landing.

The simulated user is modelled as a state machine comprised of two states: (i)
the approach state, where the simulated user travels towards their current
estimate of the goal position at a given altitude and (ii) the descent state,
where the simulated user attempts to land at the current goal estimate
$\ignorespaces\hat{G}_{i}$ . The desired trajectory and velocity of the
simulated user is determined by the _trajectory planning_ module as seen in
Fig. 4. The trajectory planning module is responsible for setting waypoints
for the UAV to travel towards which are updated according to the current
estimate of the target landing platform  $\ignorespaces\hat{G}_{i}$ . The
desired speed at which the simulated user travels towards the set waypoint is
proportional to  $\ignorespaces\Phi$ , resulting in the simulated user’s
desired target velocity  $\ignorespaces V_{t}$ . The output velocity
$\ignorespaces V_{t}$ of the trajectory planning module is then fed into the
_velocity mapping controller_ which determines the final output action of the
simulated user as seen in Fig. 4. The velocity mapping controller is split
into two submodules: _joystick control_ , which aims to model how a human
pilot’s thumbs operate a joystick controller and _adaptability control_ ,
which aims to model how a pilot may react to not traveling at their desired
velocity  $\ignorespaces V_{t}$ caused by disturbances from the assistant.
Given that the current joystick position corresponds to the velocity
$\ignorespaces J_{t}$ , the joystick control submodule will update the
simulated user’s current joystick position using a proportional-controller
based on the difference between its desired target velocity  $\ignorespaces
V_{t}$ and its current joystick input  $\ignorespaces J_{t}$ i.e.

$J_{t+1}=J_{t}+(V_{t}–J_{t})P_{gain}$

where  $\ignorespaces P_{gain}\propto\Psi$ . To adapt the simulated user’s
control input in accordance with the actions taken by the assistant, the
adaptability control submodule uses an integral-controller based on the
previous action difference of the simulated user  $\ignorespaces a_{u}$ and
the assistant  $\ignorespaces a_{a}$ . This action difference integral
$\ignorespaces I_{t}$ is updated as:

$I_{t+1}=I_{t}+(a_{u}–a_{a})(1-\alpha),$

which is then subsequently decayed at each time step. The simulated user’s
final output velocity command is then the sum of both the joystick control and
adaptability control submodule’s outputs:

$a_{u}=J_{t+1}+I_{t+1}I_{gain}.$

Figure 4: Overview of simulated user’s design

During training new simulated users are sampled after each landing trial where
each of the four parameters:  $\ignorespaces\alpha$ ,  $\ignorespaces\beta$ ,
$\ignorespaces\Psi$ and  $\ignorespaces\Phi$ are independently sampled from a
uniform distribution as well as random number generators seeded based on the
current time. Each simulated user contains two random number generators to
control for deterministic and non-deterministic decisions. Deterministic
decisions are those that are guaranteed to occur in a specific order
regardless of the actions taken by the assistant and include the initial
starting altitude and whether the simulated user should fly along the
principle axis or directly when approaching the target platform. Non-
deterministic decisions are those that cannot be guaranteed to occur in a set
order due to the actions taken by the assistant and include decisions such as
pauses or altitude changes mid-flight. Two random number generators are
included so that the exact characteristics and decisions made by the simulated
user can be replayed when validating across multiple models for a fair
comparison. Examples of the simulated user flying in various simulated
environments can be seen in the supplementary video.

####  3.1.2 Policy Network

The actor network architecture used by the assistant to generate target
velocities contains two branches of fully connected layers. The first branch
extracts features of the current state  $\ignorespaces s_{t}$ while the second
branch extracts features of the previous state  $\ignorespaces s_{t-1}$ which
is then sequentially fed into an LSTM cell for temporal information
extraction. The two branches are then concatenated and fed into additional
fully connected layers to generate a vector in  $\ignorespaces\mathbb{R}^{3}$
, denoting the assistant’s desired control input target velocity for the UAV.
A multi-branch model is chosen compared to our prior work (Backman ., 2021)
due to it’s poor handling of temporal information. Previously the pilot’s
input was averaged over a fixed time window as a cheap means to observe the
pilot’s intent over time, which occasionally led to inconsistent assistance
being delivered. The actor and critic network share identical architectural
designs aside from the activation layer of their final output, where the actor
uses a tanh activation function scaled to generate velocity magnitudes between
$\ignorespaces[0.0,1.2]$ m/s while the critic uses a linear activation
function for generating the Q-value for the given state. The network
architecture is summarised in Fig. 5.

Figure 5: Overview of assistant training architecture.

At each iteration the actor receives three inputs: (i) the current state
vector  $\ignorespaces s_{t}$ which contains the UAV position, velocity,
pilot’s action and mean latent vector  $\ignorespaces\mu$ from  $\ignorespaces
q_{RGBD}$ . (ii) The previous state  $\ignorespaces s_{t-1}$ concatenated with
the action taken by the assistant in the previous iteration and (iii) the LSTM
cell’s memory state for the previous iteration.

The actor is unaware of the pilot’s goal which has to be implicitly inferred
through observations of the actions taken by the pilot in the context of the
current state. As the critic is not required during inference, the state space
for the critic can be augmented with additional information that is only
attainable during training in simulation, such as the pilot’s true goal
landing position. When training the critic we concatenate the current state
$\ignorespaces s_{t}$ with the true goal position  $\ignorespaces G$ as a
means of learning with privileged information (Vapnik Izmailov, 2015) for
decreased network convergence time. As shown in Salter . (2021), using
privileged information during reinforcement learning has been demonstrated to
improve learning speed and lead to impoved final performance and better
generalisation.

The reward function used to train the policy includes a total of 5 terms as
seen in Eq. 2:

$\displaystyle R=k_{0}R_{\mathrm{ActionDiff}}+k_{1}R_{\mathrm{LandingError}}$
(2)
$\displaystyle+k_{2}R_{\mathrm{SafePos}}+k_{3}R_{\mathrm{HVel}}+k_{4}R_{\mathrm{VVel}}.$

  * $\ignorespaces R_{\mathrm{ActionDiff}}$

is the difference in the action taken by the user  $\ignorespaces a_{u}$ and
the assistant  $\ignorespaces a_{a}$ :  $\ignorespaces
R_{\mathrm{ActionDiff}}=-\left\lVert a_{u}-a_{a}\right\rVert$ , to promote the
assistant only exerting control when necessary.  $\ignorespaces
R_{\mathrm{LandingError}}$ is the UAV landing error towards the target
platform  $\ignorespaces G$ :  $\ignorespaces
R_{\mathrm{LandingError}}=-\left\lVert G-XY\right\rVert$ , while
$\ignorespaces R_{\mathrm{SafePos}}$ is whether the UAV landed at a safe
location, defined as landing with all four legs contacting the platform (1) or
not (-1).  $\ignorespaces R_{\mathrm{LandingError}}$ is included to ensure the
assistant lands near the platform desired by the pilot whereas  $\ignorespaces
R_{\mathrm{SafePos}}$ promotes safely landing on any platform.  $\ignorespaces
R_{\mathrm{HVel}}$ and  $\ignorespaces R_{\mathrm{VVel}}$ relate to the speed
of the UAV when landing in the horizontal  $\ignorespaces v_{h}$ and vertical
$\ignorespaces v_{v}$ direction respectively. The reward is scaled based on
the height  $\ignorespaces H$ of the UAV up to a threshold height
$\ignorespaces H_{T}$ above the target platform, where the magnitude of
$\ignorespaces R_{\mathrm{HVel}}$ and  $\ignorespaces R_{\mathrm{VVel}}$ grows
as the UAV gets closer to the platform:

$R_{\mathrm{HVel}}=\begin{cases}\frac{(H-H_{T})}{H_{T}}v_{h}^{2}&\text{if
\begin{minipage}[t]{433.62pt}\rm\ignorespaces
\trivlist\par\trivlist@item@[]\hbox{\hbox{\hbox to0.0pt{\indent\hskip
0.0pt\hss}}\hskip 0.0pt\hbox{\hbox{}\hbox{\ignorespaces$\ignorespaces
H<H_{T}$}}}\par\endtrivlist\ignorespaces\end{minipage}}\\\
0&\text{else}\end{cases}$ (3)

and

$R_{\mathrm{VVel}}=\begin{cases}\frac{(H-H_{T})}{H_{T}}v_{v}^{2}&\text{if
\begin{minipage}[t]{433.62pt}\rm\ignorespaces
\trivlist\par\trivlist@item@[]\hbox{\hbox{\hbox to0.0pt{\indent\hskip
0.0pt\hss}}\hskip 0.0pt\hbox{\hbox{}\hbox{\ignorespaces$\ignorespaces
H<H_{T}$}}}\par\endtrivlist\ignorespaces\end{minipage}}\\\
0&\text{else}\end{cases}.$ (4)

  * $\ignorespaces R_{\mathrm{HVel}}$

and  $\ignorespaces R_{\mathrm{VVel}}$ are included to promote a landing
velocity that is safe for the UAV, where separate weightings are applied along
the vertical and horizontal directions to promote the assistant in generating
trajectories with minor horizontal movement when landing. The scaling term
$\ignorespaces\frac{(H-H_{T})}{H_{T}}$ is included to promote a smoother
trajectory for the UAV compared to the assistant employing a just-in-time
approach in slowing down for landing when only considering the final landing
velocity in the vertical and horizontal direction. The actor aims to maximise
the future discounted reward function given a discount factor of 0.99, where
weighting coefficients  $\ignorespaces k_{0}$ –  $\ignorespaces k_{4}$ are
given values of 0.375, 12.0, 5.0, 40.0 & 3.5 respectively.

For training, the exploration process is parallelised with 16 concurrent UAVs
running across four separate instances of the Unreal Engine. Each Unreal
Engine management thread collects the current state of the four UAVs within
its environment, the simulated users’ actions and requests actions from the
assistant from a centralised network inference thread. The returned assistant
actions then have exploratory noise applied to them from an Ornstein-Uhlenbeck
process which is decayed after successive epochs. The assistant actions are
then averaged with the simulated users’ actions, from which normally
distributed noise is then added with variance dependant on the UAV’s distance
to the ground to emulate disturbances caused from the ground effect. The final
target velocity is then sent to AirSim where an iteration is performed every
200ms. After each epoch the network inference thread loads the newly collected
experiences from the 16 UAVs into the experience replay buffer and then
performs optimisation iterations in accordance with the total exploration
iterations performed in the previously completed epoch, with a batch size of
64. An epoch is considered complete when all 16 UAVs have landed or a total of
1.5 minutes have elapsed. Training and model validation results are reported
in section 5.2.

##  4 User study

To assess the performance of the proposed assistant with human pilots, a user
study was conducted. The study was approved by the Monash University Human
Research Ethics Committee (MUHREC), project ID 29565. The physical arena
consisted of nine labeled platforms of size 0.5  $\ignorespaces\times$ 0.5
$\ignorespaces\times$ 0.12m, arranged in a 3  $\ignorespaces\times$ 3 grid
with 1.4m spacing from each platforms’ centroids. Each participant performed a
total of twenty landings split into a sequence of ten landings unassisted and
ten landings assisted, where odd participant ID number participants completed
all 10 assisted landings first while even ID number participants completed all
10 unassisted landings first. During the study, participants stood as far back
as possible, confined within the physical dimensions of the experiment
laboratory which was measured to be 11m behind the centre platform and were
told to remain stationary during the flight. The physical layout of the study
can be seen in Fig. 6.

Figure 6: Physical layout of user study environment. (Left) View of the arena
from participant’s perspective. (Top right) UAV used in user study resting
atop of landing platform. (Bottom right) Participants’ relative position to
the arena.

Participants flew a custom-built UAV using the Pixhawk Cube flight controller
running ArduPilot. The UAV contained an Odroid-N2 as an onboard companion
computer to retrieve RGB-D images from two Intel RealSense D435i cameras
situated at the front and back of the UAV. The Odroid-N2 wirelessly streamed
the camera feeds to a base station whilst receiving target velocities and UAV
poses recorded using fourteen Bonita-10 Vicon motion capture cameras which are
forwarded onto the onboard flight controller. The motion capture cameras were
mounted around a 7.2m diameter circular truss, which the participants flew
inside, constrained within a 6.8m diameter safe-to-fly zone by an automatic
safety system which alerted and forcibly took control of the UAV if the
pilot’s actions were deemed unsafe. During the study a single participant in
the unassisted condition triggered the safety system due to unintentionally
attempting to fly out of bounds.

Participants were initially asked to fill out a demographics survey detailing
their previous experiences with UAVs and joystick controllers, after which
were then shown an introductory video explaining the process of the study.
Participants were then briefly given the opportunity to practice flying the
UAV, where they were instructed to not perform any landings but to develop an
understanding of the mapping of control inputs to the physical changes in the
UAV state. After the conclusion of the practice flight, the ceiling lights
were turned off and eight LED flood lights situated around the base of the
truss were turned on to illuminate the arena and to prevent participants
relying on shadows cast from the UAV as a substitute for depth perception.
Participants were then tasked with performing the predefined sequence of ten
landings either unassisted or assisted depending on their participant ID,
after which they were given a NASA Task Load Index (TLX) (Hart Staveland,
1988) survey detailing how they perceived the task workload for the given
condition.

To avoid the need for participants to teleoperate the UAV take-off, the first
author initiated the take-off sequence and ascended the UAV to an altitude of
1.4m using a separate master RC transmitter. The participant was then
instructed which platform they were to land on using the ID number inscribed
on the platform. The task commenced once the participant pushed either of the
joysticks. The task concluded when the UAV descended to an altitude such that
the UAV’s legs would make contact with the target platform, from which the
UAV’s motors were turned off and the final landing position was recorded. The
sequence of ten landings were then repeated under the opposite condition of
unassisted or assisted, followed by the appropriate TLX survey. Finally the
participants were instructed to complete the final survey which comprised of
fourteen multiple choice questions asking about specific aspects of the task,
followed by seven short answer questions. The full list of survey questions
can be seen in Table. 2, Table. 3 & Table. 4, alongside the user study results
in section 5.4.

##  5 Results

###  5.1 Perception module results

To measure the effect that domain randomization has on the CM-VAE’s ability to
reconstruct real-life scenes, an experiment was conducted where three
identical CM-VAE models were trained using three unique datasets. The proposed
CM-VAE used to train the assistant follows the methodology outlined in section
4 and uses the complete dataset. To test the impact of noise generating
functions, a model is trained using the complete dataset but images do not
have noise applied to them before being fed into the network. To test the
impact of training over a diverse set of environments, a third model is
trained using an identical amount of unique training images used in the
complete dataset, but limiting the variation of textures, lighting conditions
and platform configurations. A total of 25 textures, 5% of those used in the
complete dataset, are used for the platforms and flooring. Lighting conditions
are restricted to a selection of ceiling lights and platform dimensions are
sampled from a discrete collection of side lengths compared to a continuous
sampling in the complete dataset. Each dataset consists of a total of 80,000
examples where each model is trained over 1-million iterations.

To evaluate each model’s ability to reconstruct real-life scenes, images
collected from participants in the user study are used to reconstruct the
flight arena with poses collected from the motion capture system. Each model
reconstructs the combined depth map given input RBD-D images taken by the
front and back camera of the UAV, where the mean error is computed from the
ground truth combined depth map. The ground truth combined depth map is
created by using the pose of the UAV, relative camera poses to the UAVs
coordinate frame and camera intrinsic parameters to project rays onto a
simplified version of the environment. The simplified environment mimics the
intended user study arena with platforms of size 0.5  $\ignorespaces\times$
0.5  $\ignorespaces\times$ 0.12m arranged in a 3  $\ignorespaces\times$ 3 grid
with 1.4m spacing but does not consider imperfections in the platforms or
flooring.

A total of 65338 samples were used to calculate each model’s mean
reconstruction error. The errors for the complete data set, limited domain
randomization and no noise dataset models are 0.035m, 0.039m and 0.051m
respectively compared to the average input depth map errors of 0.042m. To test
if a statistically significant difference in reconstruction performance exists
between the proposed model and alternative models, two-sample Welch’s t-tests
are used at a 99% confidence level. The proposed model trained over the
complete dataset resulted in a statistically significant lower mean
reconstruction error compared to both alternative models.

An example plot of the reconstruction error for each input image in a
participant’s trajectory can be seen in Fig. 7. For the model trained without
noise applied to training images, large spikes in reconstruction errors can be
observed in input images with missing depth values or large depth values
caused from triangulating mismatched pixels. The model trained with limited
domain randomization reconstructed the environment consistently however it was
observed that the model had difficulty reconstructing certain sets of
consecutive images. It is assumed that this is caused from difficulties
associated in generalising to a broader set of unseen samples.

Figure 7: Example CM-VAE reconstruction errors taken from a participant’s
flight trajectory. Example input images are shown to highlight events such as
spikes in the no noise model caused from input images containing missing depth
values, or increases in reconstruction error for the limited domain
randomization model from being unable to generalize to certain unseen
configurations.  Figure 8: Architecture ablation Study Results. Each metric is
averaged over the three training initialisations. For the bottom plots, a
landing is considered a success given that the UAV lands on the intended
landing platform with all four legs contacting the surface, whilst landing
with a horizontal and vertical velocity of below 0.2 and 0.6m/s respectively.
A) Model performance whilst training; B) Model performance during the
simulated validation task,  $\ignorespaces\beta$ acts as a measure of the
simulated user’s proficiency. C) Model performance during the physical
validation task.

###  5.2 Policy learning results

To quantify the impact of network architecture decisions, an ablation study
was performed where four unique models were trained. (i) LSTM-CriticGoal, the
proposed model architecture which includes an LSTM cell and where only the
critic has access to the true landing position  $\ignorespaces G$ . (ii) LSTM-
NoCriticGoal, where the critic is not provided with  $\ignorespaces G$ to test
what impact providing additional information to the critic has on the training
process, similar to our prior work (Backman ., 2021). (iii) NoLSTM-CriticGoal,
where the LSTM cell is omitted from the network architecture and the fully
connected layers from the current and previous state branches are directly
concatenated to each other to test the impact of the LSTM cell on performance.
(iv) LSTM-Oracle, where  $\ignorespaces G$ is provided to both actor and
critic to test the impact of keeping the pilot’s intent hidden from the actor.
Each model was trained over three random initialisations, where each
initialisation consisted of two-million training iterations. The training
results can be seen in Fig. 8-A.

Providing the critic with the true goal information (available only in
simulation) can be seen to have the greatest impact on training results. The
final correct safe landing rate for the LSTM-NoCriticGoal model was 55%, which
was achievable within the first 22% of training iterations for alternative
models that included the true goal within the critic’s state space. LSTM-
Oracle was the highest performing model during training, with a final average
landing error 57% lower than the average of LSTM-CriticGoal and NoLSTM-
CriticGoal. Unlike the alternate models, LSTM-Oracle does not mistakenly land
on incorrect platforms from wrongly inferring the simulated user’s intent.

Each trained model was then subsequently validated on a standardised
validation sequence which consisted of ten landings. The simulated environment
was modeled to reflect the physical environment used in the user study, where
a total of nine platforms of size 0.5  $\ignorespaces\times$ 0.5
$\ignorespaces\times$ 0.12m arranged in a 3  $\ignorespaces\times$ 3 grid with
1.4m spacing from the platform’s centroids were used. For each model, the
sequence of ten landings were performed a total 51 times, where
$\ignorespaces\beta$ was swept from 0 to 1 in 0.02 increments whilst the
remaining of the simulated user’s parameters  $\ignorespaces\alpha$ ,
$\ignorespaces\Psi$ and  $\ignorespaces\Phi$ were held constant at a value of
0.5. To ensure the simulated user’s policy remained consistent across all
validated models, each of the ten landings were assigned a unique number to
seed the simulated user’s random number generators. The simulated validation
results can be seen in Fig. 8-B.

Figure 9: Reward ablation study, simulated validation results

The highest performing architecture was the oracle configuration with an
average landing error of 0.042m followed by LSTM-CriticGoal with 0.053m,
NoLSTM-CriticGoal with 0.062m and LSTM-NoCriticGoal with 0.109m, all of which
outperforming the baseline of the unassisted simulated user with an average
landing error of 0.362m. Comparing the assisted approaches to that of the
unassisted simulated user for both landing error and task success rate, it can
be seen that the performance of the assisted approaches are invariant to that
of the proficiency of the simulated user.

To assess the performance of the proposed approach when transferring from
simulation to reality, the LSTM-CriticGoal architecture was validated within
the physical environment used in the user study. The simulated user was used
to pilot the UAV where the sequence of ten landings were performed for values
of  $\ignorespaces\beta$ from 0 to 1 in 0.25 increments for a total of fifty
landings.  $\ignorespaces\alpha$ ,  $\ignorespaces\Psi$ and
$\ignorespaces\Phi$ were held at constant values of 0.5 and random number
generators seeded with identical values to that of the simulated validation.
The results of the physical validation can be seen in Fig. 8-C.

The baseline performance of the unassisted simulated user in the physical
validation environment was comparable to that in simulation, where the range
of average landing errors with respect to  $\ignorespaces\beta$ for the
physical validation were [0.200, 0.550]m and [0.226, 0.518]m for simulated
validation. Despite the proposed model LSTM-CriticGoal being trained purely on
synthetic data, it achieved a landing success rate of 98% on the physical
validation sequence where only a single failed landing was recorded for a
$\ignorespaces\beta=0.0$ simulated user. The simulation-reality gap accounted
for an additional 0.027m in the average landing error which can be attributed
to a mixture of imperfect dynamic modelling of the UAV and the additional
latency induced with wireless data streaming.

To assess the impact of reward function terms when training the proposed model
LSTM-CriticGoal, an additional ablation study was performed where each of the
terms in Eq. 2 were individually omitted, aside from  $\ignorespaces
R_{\mathrm{ActionDiff}}$ . Each model was trained over a single random
initialisation and then subjected to an identical simulated validation
sequence as performed in the network architecture ablation study, the results
of which can be seen in Fig. 9.

Omitting  $\ignorespaces R_{\mathrm{HVel}}$ and  $\ignorespaces
R_{\mathrm{VVel}}$ resulted in the largest decrease in success rate due to the
UAV landing with unsafe velocities, however having little impact on the
landing error. Omitting  $\ignorespaces R_{\mathrm{LandingError}}$ did not
appear to effect the performance of the assistant against simulated users of
$\ignorespaces\beta>0.2$ , but caused the assistant to land on the incorrect
platform for novice simulated users due to their larger initial errors. As
$\ignorespaces R_{\mathrm{SafePos}}$ rewards the assistant for landing in any
safe landing location, there is no incentive for the assistant to land at the
location desired by the simulated user when  $\ignorespaces
R_{\mathrm{LandingError}}$ is removed. Omitting  $\ignorespaces
R_{\mathrm{SafePos}}$ caused the assistant to land in the general vicinity of
the platform.  $\ignorespaces R_{\mathrm{SafePos}}$ provides a binary reward
for landing on a platform, resulting in strong gradients for the assistant to
ensure a safe landing. As  $\ignorespaces R_{\mathrm{LandingError}}$ is a
continuous reward, a small decrease in the landing error results in a small
decrease in the landing penalty. Therefore it becomes more beneficial for the
assistant to focus on minimising  $\ignorespaces R_{\mathrm{HVel}}$ ,
$\ignorespaces R_{\mathrm{VVel}}$ and  $\ignorespaces R_{\mathrm{ActionDiff}}$
instead of further reducing the landing error, resulting in an overall higher
average landing error and decreased success in landing at a safe position.

The simulated user in our prior work Backman . (2021) used a two-parameter
model which included  $\ignorespaces\alpha$ & $\ignorespaces\beta$ . To assess
the impact of introducing two additional parameters  $\ignorespaces\Psi$ &
$\ignorespaces\Phi$ and the velocity mapping controller, an additional
ablation study was performed where the proposed model LSTM-CriticGoal was
trained on simulated users using the previous two-parameter model in Backman .
(2021). To validate the difference in performance between the two assistants
trained on different simulated user models, a standardized validation sequence
consisting of 1,000 landings was used. The validation sequence was constructed
by uniformly sampling the simulated user’s parameters  $\ignorespaces\alpha$ ,
$\ignorespaces\beta$ ,  $\ignorespaces\Psi$ & $\ignorespaces\Phi$ for each
landing compared to the prior ablation study where  $\ignorespaces\alpha$ ,
$\ignorespaces\Psi$ & $\ignorespaces\Phi$ remained constant at 0.5. Each
landing in the validation sequence was performed in the simulated replica of
the environment used in the physical user study, where a random start and goal
platform was selected for each landing.

Both assistant models were then tested on identical validation sequences where
the average success rate and landing error for the model trained using the
four-parameter simulated user model was 96.4% and 0.061m respectively. For the
assistant trained on the simpler two-parameter simulated user model the
average success rate and landing error was 81.6% and 0.078m respectively. The
lower success rate from the model trained with  $\ignorespaces\Psi$ &
$\ignorespaces\Phi$ omitted from its simulated user model is predominately
caused from landing slightly off the platform which can be attributable to
only experiencing simulated users in training that do not account for
disturbances caused by the assistant from the adaptability control subsection
of the velocity mapping controller module.

Accounting for  $\ignorespaces\Psi$ & $\ignorespaces\Phi$ is important in
accurately reflecting human piloting characteristics as the results obtained
from section 5.4 show that the average non-zero action for unassisted
participants ranged between 0.09m/s to 0.44m/s, demonstrating variability in
the equivalent simulated user parameter  $\ignorespaces\Phi$ . While the
average non-steady state acceleration in participants’ actions ranged from
0.25m/s2 to 1.17m/s2, demonstrating the variability in the equivalent
simulated user parameter  $\ignorespaces\Psi$ .

###  5.3 Related work comparison

To compare the proposed work to alternative shared autonomy works, the
validation sequence used in Section 5.2 was applied to two comparison
approaches proposed in Javdani . (2018); Reddy . (2018). Each approach
formulates a user model and assistance policy.

Javdani . (2018) assists humans for robotic arm object grasping tasks and
formulates a user model that outputs an action that directly approaches the
goal location but with noise added to the output. The assistance policy
requires the position of all goals within the environment to be known and
computes an optimal action directed to each potential goal, where the policy’s
output action is the weighted sum of each goal directed action weighted by the
associated goal probability.

Reddy . (2018) assists humans in landing in the lunar lander game and
formulates a simulated user model with a DQN trained using two rewards based
on minimising the distance to goal and attaining landing success. To model
imperfections in the optimal policy, the simulated user repeats the previous
action with a fixed probability. The assistant copilot was trained using an
identical DQN network following an identical reward structure.

The two comparison approaches Javdani . (2018); Reddy . (2018) were
implemented following the respective author’s GitHub repositories (Javdani,
2016; Reddy, 2018), where modifications are made to Javdani . (2018)’s
parameters to ensure appropriate scaling for the given task and environment.
Implementation of Reddy . (2018)’s copilot was trained using identical inputs
of the current state branch in our proposed work, while the network model was
increased in depth and width to match the output branch of our proposed model.
Reddy . (2018)’s DQN model required to be trained exclusively in the
validation environment as training in randomly generated environments caused
the model to fail to converge. The DQN was trained for two-million
optimisation iterations, identical to that of our proposed work.

Each of the three assistance strategies were validated against each of the
three simulated user models. Each assistance strategy implemented their own
action blending policy (combining the input of the simulated user and the
assistant) as outlined within their respective works. For Reddy . (2018), the
simulated user actions were discretised into 27 actions as per our prior work
(Backman ., 2021) for calculating the control sharing policy. Each model
evaluation performed the 10 validation landings repeated a total of five
times, where our proposed simulated user model had its  $\ignorespaces\beta$
parameter swept from 0.0 to 1.0 in 0.25 increments whilst the remaining
simulated users remained constant as per their implementation.

Figure 10: Summary results comparing the proposed work to Javdani . (2018) and
Reddy . (2018) in terms of landing error (A), trajectory efficiency (B) and
assistance delivered (C). Exerted control is calculated as the Euclidean
distance between the UAV’s target velocity and the simulated user’s output
target velocity, which considers each assistance approach’s chosen policy
blending formulation.

Summary of the simulated shared autonomy comparison results can be seen in
Fig. 10. Our proposed approach demonstrated the best performance in
successfully landing, with an average landing error of 0.08m, compared to
0.33m and 0.40m for Reddy . (2018) and Javdani . (2018) respectively.
Importantly, our proposed approach achieves the lowest landing error for all 3
user models. Javdani . (2018) generated the most efficient trajectories,
defined by minimising  $\ignorespaces\frac{\textrm{time}\times\textrm{distance
travelled}}{\textrm{init. distance}^{2}}$ , with an efficiency score of 3.20
followed by 6.64 and 22.01 by our proposed work and Reddy . (2018)
respectively. In terms of the degree of control exerted onto the system, our
proposed approach was shown to be the least intrusive, with an average
Euclidean distance between the UAV’s target velocity and the simulated user’s
output target velocity of 0.22, followed by 0.41 and 0.57 by Javdani . (2018)
and Reddy . (2018) respectively.

The main limitation of Javdani . (2018) is its lack of dynamics modelling,
where the inertia of the UAV often results in overshooting of the target,
which is then further exacerbated by drifting closer to incorrect goals to
form greater action probabilities towards the incorrect targets. Javdani .
(2018)’s success in generating efficient trajectories is due to constantly
providing actions that move the UAV closer to the target, however the success
comes at a cost of requiring the location of all targets within the
environment as a priori, limiting its feasibility in practical settings.

The main limitation of Reddy . (2018) is its inability to generalise to both
different users or environments. Reddy . (2018) could not be trained on
multiple randomly generated environments and had poor performance when
validating against our proposed simulated user model. Unlike Javdani . (2018);
Reddy . (2018) user models, the proposed user model does not fly directly to
the goal but performs separate approach and descent phases that more
accurately model human pilot behaviour which is not modelled by the other
related works, causing suboptimal performance in the unseen conditions.

The proposed model demonstrated robustness in performance across all simulated
user models, despite not being trained on the other user models that perform
early and aggressive descent manoeuvres. Unlike the other assistance
strategies, the proposed approach does not require knowledge of the
environment as a priori such as the location of all targets in Javdani .
(2018) or exclusive validation arena training in Reddy . (2018). The proposed
work demonstrated the greatest performance in minimising the landing error
whilst exerting the least amount of control over the system. However the lower
degree of control over the system resulted in less efficient trajectories when
compared to Javdani . (2018), as the proposed approach aims to refine the
pilot’s policy in order to leverage high-level human decision making, which is
more effective when assisting human pilots compared to simulated users.

###  5.4 User study results

28 participants completed the user study for a total of 560 landings (280
unassisted and 280 assisted) with an average study completion time of 1.5
hours per participant. A summary of key performance metrics can be viewed in
Table. 1.

Table 1: User study performance metrics summary

| Unassisted | Assisted
---|---|---
Success rate | 51.429% | 98.214%
Average landing error | 0.325m | 0.091m
Median landing error | 0.159m | 0.081m
Landing error variance | 0.069m2 | 0.006m2
Time taken / initial starting distance | 10.329s/m | 6.872s/m
Distance traveled / initial starting distance | 1.705 | 1.522

Bold values indicate the better performing condition with a statistically
significant difference between the two conditions at a significance level of
$\ignorespaces\alpha=0.01$ .

The poorest performing unassisted participant scored a success rate of 10%,
whilst the highest performing unassisted participant achieved a success rate
of 80%. While in the assisted condition, the lowest success rate observed was
90%, where the majority of assisted participants achieved a 100% success rate.
A landing was considered a success if the UAV remained at rest atop of the
designated platform, landed with a horizontal and vertical velocity of below
0.2 and 0.6m/s respectively and did not engage the automatic safety system due
to attempting to fly outside of the arena.

To measure the efficiency of participant’s flight strategy we assess their
trajectory under two metrics, (i) the time taken to complete the task and (ii)
the total distance traveled. Both metrics are normalised by the initial
starting distance to account for landing sequences where the starting platform
is further away from the goal platform. Whilst in the assisted condition,
pilots on average required two-thirds of the time to complete the same task
compared to flying unassisted. For traveled distance, assisted participants
generated trajectories 18-percentage points more efficient than that of
unassisted participants when comparing to the most efficient path of 1.0 for
the distance traveled / initial starting distance metric.

To test if a statistically significant difference exists between the
unassisted and assisted condition for the metrics provided in Table. 1, two-
sample statistical tests under a 99% confidence level are performed for each
metric. To test the significance for the success rate, median landing error
and landing error variance, the McNemar, Mood’s median test and Levene’s test
are respectively used whilst the remaining metrics are subjected to Welch’s
t-tests. All metrics were found to have a statistically significant difference
between the unassisted and assisted conditions.

To examine whether a learning effect exists from participants performing the
unassisted or assisted condition first compared to the respective condition
last, statistical analysis was performed on the metrics outlined in Table. 1
using the previously aforementioned statistical tests under 99% confidence. It
was found for all metrics across both conditions that there was insufficient
evidence to suggest that performing either unassisted or assisted condition
first or last had an impact on the condition’s metrics outlined in Table. 1
under 99% confidence.

Figure 11: (Left) Participant performing the task unassisted. They initially
undershoot the target platform, then over-correct, failing the landing.
(Right) Participant performing the task assisted. Participant instructs to
prematurely descend. The assistant alters the trajectory to land on the
platform center.

An example trajectory of a participant performing the task in the unassisted
and assisted condition can be seen in Fig. 11. Unassisted participants tended
to undershoot their initial descent followed by showing signs of uncertainty
in the UAV’s state by making multiple adjustments along the depth axis before
finally landing. In the assisted condition participants continued to
undershoot their initial descent however the assistant altered the trajectory
to approach the platform whilst descending to ensure a safe landing. The
primary cause of failures in the unassisted condition were due to the
participant undershooting the landing from their initial start location
(63.0%), followed by overshooting the intended platform (32.6%). Further
unassisted and assisted landings performed by participants can be seen in the
attached supplementary video.

To measure how the assistance delivered by the assistant varies over
trajectories, the XYZ distance between participants’ actions to that of the
assistant’s action is plotted against the XY distance from the goal as seen in
Fig. 12. It can be observed for distances far away from the goal (1.5m+) the
assistant exerts minimal control over the UAV due to the uncertainty in the
pilot’s intent. As the UAV approaches the goal location the average assistance
exerted increases as the assistant’s confidence in the target increases,
before declining at distances less than 0.3m due to not requiring to exert as
much control to attain task success.

Figure 12: Average assistance (black) and interquartile range (green)
delivered by the assistant as a function of the XY distance from the goal
location. Assistance is calculated as the XYZ distance between the pilot’s XYZ
target velocity to that of the assistant’s XYZ target velocity:
$\ignorespaces\left\lVert a_{p}-a_{a}\right\rVert$ .

Summary results of participants’ perception of the task can be seen in Fig.
13. From the TLX survey, participants perceived a large degree of additional
effort, mental demand and frustration while performing the task in the
unassisted condition and recognised a large difference in performance whilst
performing the task assisted. Participants also perceived a lower physical and
temporal demand in the assisted condition albeit to a lesser extent. The TLX
survey results were analysed with a Welch’s t-test under a 95% confidence
level to determine if a statistically significant difference exists between
participants’ perception of the task in the unassisted and assisted condition.
It was found that for all metrics aside from physical demand, a statistically
significant difference exists.

Figure 13: Results of NASA TLX survey.

The full list of survey questions and responses can be seen in Table. 2,
Table. 3 and Table. 4. From the final survey, participants strongly agreed
that with the help of the assistant, task performance and the time required to
complete the task was better than that of the unassisted condition. Compared
to our previous work (Backman ., 2021), a stronger disparity between
participant confidence was observed where participants felt less confident
flying unassisted, presumably due to the presence of physical risks, whilst
being more confident when flying assisted. Previously (Backman ., 2021)
participants were neutral in terms of their trust placed onto the assistant
due to inconsistencies in the degree of assistance provided as well as
aggressively acting on the incorrect inference of the participants’ intent. In
this work, participants’ final survey results displayed a greater trust in the
assistant while recognising a consistent degree of assistance and correct
inference of their intent. This additional trust and consistency may explain
the greater degree of overall confidence participants felt in the assisted
condition compared to previously, despite the additional risks and increased
task difficulty.

For the worded responses, when asked about the most difficult aspect of the
task 68% of participants made explicit references to the difficulties of
estimating the depth of the UAV. 18% of participants mentioned difficulties
associated with making small adjustments especially due to ground effect as
they noted that the “drone became unstable as it neared the platforms”.

For additional features that participants would like to have implemented, 21%
of participants made recommendations for a downwards facing laser light, 18%
for a downwards facing camera and 21% for feedback via auditory or visual cues
when the drone was above a platform or when the assistant was performing an
action.

Table 2: Participant Demographics

Have you any experience flying physical drones before? No Yes 46% 53% How many
hours of experience have you with flying physical drones? 0-5 71% 5-10 3%
10-20 3% 20-50 10% 50+ 10% How confident are you with flying physical drones?
Not confident 1 25% 2 35% 3 17% 4 14% Confident 5 7% How often do you play
video games? Never 10% Monthly 35% Weekly 10% Regularly 21% Daily 21% How many
hours have you spent on games involving flying? 0-10 50% 10-25 25% 25-100 14%
100-200 3% 200+ 7% How many hours have you spent using a joystick controller?
0-10 17% 10-25 7% 25-100 14% 100-200 32% 200+ 28%

Table 3: Final survey

1 2 3 4 5 1: I felt confident landing unassisted 2: I felt confident landing
with the assistant 3: I prefer landing with the help of the assistant 4: I
felt being unassisted gave me more freedom 5: I was unsure what the assistant
was trying to do 6: The assistant didn’t do what I wanted it to do 7: I felt
like I received a lot of help from the assistant 8: The assistant was
consistent with the help provided 9: The assistant improved the performance of
the task 10: The assistant made the task quicker to complete 11: I trust the
actions of the assistant 12: The assistant and I worked well as a team 13: I
needed time to learn how to work with the assistant 14: Given practice, I
would perform better without assistance than with
pt21%pt57%pt14%pt7%pt0%pt0%pt0%pt0%pt35%pt64%pt0%pt0%pt14%pt7%pt78%pt14%pt14%pt21%pt28%pt21%pt25%pt39%pt17%pt17%pt0%pt50%pt35%pt10%pt3%pt0%pt0%pt3%pt25%pt32%pt39%pt0%pt3%pt7%pt50%pt39%pt0%pt0%pt0%pt7%pt92%pt0%pt0%pt3%pt3%pt92%pt0%pt0%pt14%pt39%pt46%pt0%pt7%pt21%pt39%pt32%pt10%pt25%pt32%pt28%pt3%pt32%pt39%pt21%pt3%pt3%

A response of 1 corresponds to strongly disagree whilst a response of 5
corresponds to strongly agree.

When asked about their most preferred aspect of the assistant 29% of
participants mentioned the subtleness of assistance provided stating that it
was “non-invasive and well integrated with manual control” whilst some
participants “did not notice it was there”. A further 29% of participants were
most appreciative with the lack of effort required to land by having to only
approximately approach the platform while 21% of participants enjoyed the
improved success rate.

Table 4: Final survey worded questions 1: What part of the task did you have
the most difficulties with?
---
2: What additional features or changes would make the task easier?
3: What do you think the assistant was doing during the study?
4: What did you like the most about the assistant?
5: What did you dislike the most about the assistant?
6: Describe differences in your flying strategy when unassisted/assisted
7: Any additional comments or observations you would like to add?

When asked about their most disliked aspect of the assistant 21% of
participants made references to the uncertainty in the intent of the assistant
stating that “I wasn’t sure what it was doing” and that “it was completely
invisible”. Although many participants’ most enjoyed aspect of the assistant
was its subtleness, this became a double-edged sword for others where the lack
of feedback and communication resulted in a sense of uncertainty in what was
expected of themselves and the assistant. 18% of participants stated that they
thought the assistant landed the drone with too great of a

vertical velocity and a further 18% of participants made comments about
disliking the assistant’s automatic landing when close to the platform. Those
participants made comments of wanting to be able to “readjust the position of
the drone” when close to the platform and preferred if the assistant “only
controlled the x-y position of the drone”, giving full control of the throttle
to the participant.

Table 5: Participant proficiency regression results Independent variables | P-values | Coef. (m)
---|---|---
Intercept | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.378
Drone exp. | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | -0.030
Joystick exp. | 0.01 | 0.01 | 0.01 | 0.00 | 0.00 | -0.025
Video game frequency | 0.06 | 0.03 | 0.03 | 0.03 | 0.03 | 0.016
Drone confidence | 0.33 | 0.33 | 0.25 | 0.27 | — | —
Flying game exp. | 0.61 | 0.58 | 0.57 | — | — | —
Flown before | 0.81 | 0.83 | — | — | — | —
Unassisted second | 0.87 | — | — | — | — | —

To measure the impact of participants’ prior experience and to determine
whether the assistant allows novice pilots to perform the task at a
proficiency equal to or greater than experienced pilots, a multi-variable
linear regression model was established to assess participants’ expertise
level. Participants’ responses from the demographic survey as seen in Table. 2
were used as independent variables in the model including an additional
independent variable to denote whether the participant performed the
unassisted task in the second half to control for potential learning
throughout the study. The regression model used the participants’ average
unassisted landing error as the dependent variable where multiple regressions
were performed using backwards elimination until all remaining independent
variables were deemed statistically significant under a two-tailed t-test with
significance level of  $\ignorespaces\alpha$ = 0.05. The results of the
backwards elimination regression can be seen in Table. 5.

Prior drone piloting experience, joystick experience and video game frequency
independent variables were found to be statistically significant under
$\ignorespaces\alpha=0.05$ . The model suggests that participants with a
greater piloting and joystick experience on average had a lower unassisted
landing error than those that did not, while those who frequently played video
games tended to perform worse, which may possibly be caused from unrealistic
expectations of real-world UAV dynamics. Insufficient evidence exists to
suggest that the remaining independent variables had an effect on
participants’ performance. The order in which the participants performed the
task had no significant effect on their unassisted landing results.

To determine a participant’s proficiency score, the regressed model was used
to estimate their unassisted landing error which was then remapped to fit
within the range  $\ignorespaces[0,1]$ , where a value of zero denotes a
novice pilot whilst a value of one denotes an expert pilot. The resultant
proficiency score can be plotted against key performance metrics to measure
whether pilot proficiency has an effect under the unassisted and assisted
conditions, the results of which can be seen in Fig. 14.

Figure 14: Plots of participants’ averaged performance metrics against
proficiency for unassisted and assisted conditions.

Plot A in Fig. 14 shows a strong negative correlation between participants’
proficiency and their unassisted landing error, which intuitively leads to
higher unassisted success rates in plot B. This strong relationship is
expected as the previous regression model for estimating participants’
proficiency regressed against their unassisted landing error. For the assisted
condition it can be seen that participants’ landing error and success rate was
invariant to their prior piloting experience. Regardless of proficiency,
participants performed consistently in the assisted condition, at a level
greater than that of the highest performing participants.

For efficiency metrics in plots C and D in Fig. 14, a negative correlation
exists between participants’ proficiency and both the time taken to complete
the task and the distance traveled. Meaning that on average the more
proficient a participant is, the shorter the flight trajectory and time that
is required to complete the task. Whilst in the assisted condition, the
average time taken did not depend on the participants’ proficiency, but a weak
negative correlation is observed for the trajectory distance.

To confirm whether a statistical relationship exists between the
aforementioned metrics and participant proficiency, the slope coefficients are
subjected to a two-tailed t-test under significance
$\ignorespaces\alpha=0.05$ . For the unassisted condition, sufficient evidence
exists to suggest that participant proficiency does affect all metrics shown
in Fig. 14. Whilst for the assisted condition, there is insufficient evidence
to suggest that participant proficiency influences any of the tested metrics
and that regardless of ones prior piloting experience, participants are
projected to perform equally across all metrics with the help of the
assistant.

Figure 15: Total trajectories from user study in the unassisted condition
(left) and assisted condition (middle), alongside an equivalent sample of
randomly selected simulated users (right) performing the same task in the
simulated validation environment. Ellipses are centered at the average landing
location where principal component analysis is performed to demonstrate the
axis with greatest deviation for each landing platform.

During the assisted portion of the user study, it was observed that
participants developed riskier or lower-effort flying habits. In the later
assisted landings participants provided low-effort initial estimates of the
target platform compared to earlier assisted landings. Participants also made
comments about becoming “reckless” in their post-study responses, which was
observed by participants flying at greater speeds with aggressive descents in
the later portion of assisted landings. In general, as participants’
confidence in the assistant grew, the level of risk they were willing to take
also grew. The observations of low-effort initial estimates and increased
risk-taking behavior are empirically supported through the final four landings
in the assisted condition. A statistically significant difference in the
landing error between the initial (0.08m) and later portion (0.11m) of the
assisted landings were observed using a Welch’s t-test (p=0.008) under a 95%
significance level. Of the five failed assisted landings, 80% occurred in the
final four landings. Performance degradation was not observed in the
unassisted condition where participants performed better in the later portion
of the unassisted landings.

##  6 Discussion

The success of our proposed shared autonomy approach can be attributable to
three main factors: (i) domain randomization, (ii) simulated user modeling and
(iii) training efficiency. Transferring models trained purely in simulation
must pass the simulation-to-reality gap, where visual perception accounts for
the most significant portion of the gap due to difficulties in replicating
photorealistic images, leading to simulation trained models to fail once
transferred (Bousmalis ., 2018). Domain randomization reduces the need for
photorealistic images with the aim of making the trained network invariant to
unimportant information i.e., illumination and background textures, which has
shown success in other UAV applications (Loquercio ., 2020). By training the
perception module over a wide variety of simulated scenes under intense
dynamically generated noise, the perception module becomes robust to visual
disturbances. This is reflected in the results in Section 5.1, where the
perception module trained with high domain randomization was found to have
lower reconstruction errors when transferred to real scenes. Domain
randomization is also a key consideration when developing simulated users as
human pilots vary greatly in terms of strategy and proficiency.

Developing simulated users in a shared autonomy system is the most difficult
challenge as they must cover a broad range of policies to reflect variability
amongst users and realistically react to actions taken by the actor. Focusing
the development of the simulated user around four base parameters simplifies
the task while being capable of displaying a wide range of policies through
the variation of the base parameters. Fig. 15 shows the experimentally
observed flight trajectories of participants in the unassisted and assisted
conditions alongside an equivalent sample of unassisted simulated users.
Participants approached the target platform in two ways, by flying along the
principle axes or by flying directly to the platform as modeled by the
simulated user. During descent unassisted participants appear to follow a
complex trajectory, which is unmodeled by the simulated user. During the
assisted condition, the trajectories reflect the smoothness of the simulated
user’s model but with lower landing error variance. The source of trajectory
complexity in the unassisted condition stems from uncertainty felt by the
participants, which is absent in the assisted condition leading to simpler
trajectories. The robustness of the user model is also illustrated in Fig. 10,
which shows that the proposed model achieves the best performance even when
tested against unseen user models.

The third factor which made our shared autonomy approach successful was
training efficiency. From the ablation studyresults in Section 5.2, it was
found that providing additional information only available in simulation to
the critic greatly improved convergence time. Training the actor took
approximately 1.5 days to complete with 16 UAVs flying concurrently. Our
previous work (Backman ., 2021) did not include additional information to the
critic or perform concurrent exploration. Instead, it heavily relied on
preloading the experience replay buffer with prior failed models’ state-
transitions and performing additional optimization iterations to converge
within a reasonable time. The training efficiency made it practical to perform
hyper-parameter optimization, reward structure modification and ensured proper
model convergence before being deployed.

A major focus of the proposed approach was in developing an assistant that was
subtle and non-intrusive by implicitly inferring the pilot’s intent and only
providing assistance as needed. From the responses outlined in the user study
results in Section 5.4, it was found that some participants preferred this
non-intrusive behavior for its intuitive controls, however others found the
lack of communication and feedback unsettling. This may be due to participants
not being told how the assistant operates or because of only using the
assistant for a short period of time, however further work needs to be done
investigating if a non-intrusive approach or one that actively communicates to
the pilot is most appropriate.

Another concern is the observed behavioral change participants displayed in
the later portion of the assisted condition with decreased effort and
increased risk-taking, expecting the assistant to cover for them. Although the
role of the assistant is to ensure safe landing with decreased effort and
requisite skill, prolonged exposure may lead to degraded piloting proficiency
over extended use. However the decrease in performance may be due to
participants being tested in a low-stakes environment where the curiosity of
exploring the limits and behavior of the assistant became more interesting.

A limitation of the proposed work is the reliance on motion capture for pose
estimation when flying in the GPS denied environment. To extend the proposed
approach to outdoor conditions, additional onboard sensing would need to be
implemented to fuse GPS, IMU and visual odometry sensor streams. Visual
odometry information can be attainable using the Intel RealSense Tracking
camera T265 which has shown success in autonomous UAV landing tasks (Nogar,
2020; P. Wang ., 2022).

Although the proposed approach has been fine tuned for UAV landing tasks, the
principles of the approach have the potential to be transferred to alternative
tasks. Three components are required for replication in alternative
environments: (1) a perception module that perceives the environment by
encoding information onto a latent vector, achievable with any auto-encoder
network architecture trained to reconstruct or segment images. (2) A
parametric simulated user model, where the parameters account for adaptability
towards the assistant’s actions (  $\ignorespaces\alpha$ ), the proficiency of
the simulated user in its ability to successfully complete the tasks by itself
(  $\ignorespaces\beta$ ) and parameters associated with the dynamics of the
output control variables (  $\ignorespaces\psi$ & $\ignorespaces\phi$ ). The
third component required is a policy module trained using TD3 with critic MDP
formulation that includes the simulated user’s objective, while reward terms
focus on task success components (  $\ignorespaces R_{\mathrm{LandingError}}$
,  $\ignorespaces R_{\mathrm{SafePos}}$ ,  $\ignorespaces R_{\mathrm{HVel}}$ &
$\ignorespaces R_{\mathrm{VVel}}$ ) and agreement with the user (
$\ignorespaces R_{\mathrm{ActionDiff}}$ ).

##  7 Conclusion

In this work we propose a shared autonomy approach that assists pilots of all
skill levels to safely land a UAV under conditions where depth perception is
difficult. The assistant is unaware of the pilot’s intent nor any prior
knowledge of the structure of the environment and must disambiguate the task
using observations of the pilot’s actions and its immediate surroundings. The
proposed approach comprises of two fundamental components (i) a perception
module that encodes information about the structure of the environment from
two RGB-D cameras and (ii) a policy network that provides control inputs to
assist the pilot in safely landing the UAV.

A user study ($n=28$) was held to validate the assistant’s performance where
participants were instructed to land the drone on one of nine platforms in
both the unassisted and assisted conditions. With the help of the assistant
participants had significantly higher success rates from 51.4% to 98.2% and
completed the task more efficiently in respect to time taken and distance
traveled. Regression analysis showed that regardless of prior experience, the
assistant allowed participants of all piloting proficiencies to perform
equally amongst each other and greater than that of the most proficient
unassisted pilots.

Compared to the unassisted condition, participants perceived the assisted
condition to require a lower task load and to be the more favorable approach
according to the NASA TLX and final survey. Participants stated that their
most preferred aspect of the assistant was its non-invasive subtle nature as
well as an overall reduction in effort required to successfully complete the
task. However the assistant’s subtleness was a divisive aspect for others
where the lack of communication and inability to give feedback about its
intent led to participants feeling a sense of uncertainty in what was expected
of them. Further work needs to be done to address how an AI in a shared
autonomy setup can effectively communicate its intent and provide feedback on
the user’s actions whilst maintaining an intuitive non-intrusive behavior.

##  8 Ethical statement

The authors declare that the submitted work is free from personal conflicts of
interest. The user study was approved by the Monash University Human Research
Ethics Committee (MUHREC), project ID 29565. All participants gave informed
consent prior to participating in the user study.

##  9 Author contribution

Project Concept: [Kal Backman, Hoam Chung, Dana Kulic]; Methodology: [Kal
Backman, Dana Kulic]; User Study Design and Implementation [Kal Backman, Hoam
Chung, Dana Kulic]; User Study Execution and Data Acquisition [Kal Backman];
Analysis of the Results [Kal Backman, Hoam Chung, Dana Kulic]; Writing -
original draft preparation: [Kal Backman]; Writing - review and editing: [Hoam
Chung, Dana Kulic]; Supervision: [Hoam Chung, Dana Kulic]

##  10 Acknowledgements

None

##  11 Funding

This work was supported in part by D. Kulić’s Australian Research Council
Future Fellowship (FT200100761).

## References

  *   * Albanese . (2022) SARDOAlbanese, A., Sciancalepore, V. Costa-Pérez, X.  2022\.  SARDO: An Automated Search-and-Rescue Drone-Based Solution for Victims Localization Sardo: An automated search-and-rescue drone-based solution for victims localization.  IEEE Transactions on Mobile Computing2193312-3325.
  * Baca . (2019) MovingVehicleLandingBaca, T., Stepan, P., Spurny, V., Hert, D., Penicka, R., Saska, M.Kumar, V.  2019\.  Autonomous landing on a moving vehicle with an unmanned aerial vehicle Autonomous landing on a moving vehicle with an unmanned aerial vehicle.  Journal of Field Robotics365874-891.
  * Backman . (2021) KalBackman, K., Kulić, D. Chung, H.  2021\.  Learning to Assist Drone Landings Learning to assist drone landings.  IEEE Robotics and Automation Letters623192-3199.
  * Bousmalis . (2018) RoboticGraspingBousmalis, K., Irpan, A., Wohlhart, P., Bai, Y., Kelcey, M., Kalakrishnan, M.Vanhoucke, V.  2018\.  Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping Using simulation and domain adaptation to improve efficiency of deep robotic grasping.  IEEE International Conference on Robotics and Automation4243-4250.
  * Carney . (2019) LandingStripDetectionCarney, E., Castano, L. Xu, H.  2019\.  Determination of Safe Landing Zones for an Autonomous UAS using Elevation and Population Density Data Determination of safe landing zones for an autonomous uas using elevation and population density data.  AIAA Scitech 2019 Forum Aiaa scitech 2019 forum ( 1-16).
  * Curran . (2017) DimensionalityReductionCurran, W., Pocius, R. Smart, W.D.  2017\.  Neural networks for incremental dimensionality reduced reinforcement learning Neural networks for incremental dimensionality reduced reinforcement learning.  2017 IEEE/RSJ IROS 2017 ieee/rsj iros ( 1559-1565).
  * Feng . (2018) MovingPlatformLandingFeng, Y., Zhang, C., Baek, S., Rawashdeh, S. Mohammadi, A.  2018\.  Autonomous Landing of a UAV on a Moving Platform Using Model Predictive Control Autonomous landing of a uav on a moving platform using model predictive control.  Drones24.
  * Fujimoto . (2018) TD3Fujimoto, S., van Hoof, H. Meger, D.  2018\.  Addressing Function Approximation Error in Actor-Critic Methods Addressing function approximation error in actor-critic methods.  Proceedings of the 35th Int. Conference on Machine Learning Proceedings of the 35th int. conference on machine learning ( 80, 1587–1596).
  * González-deSantos . (2020) ContactInspectionGonzález-deSantos, L., Martínez-Sánchez, J., González-Jorge, H., Navarro-Medina, F. Arias, P.  2020\.  UAV payload with collision mitigation for contact inspection Uav payload with collision mitigation for contact inspection.  Automation in Construction115103200.
  * Hart Staveland (1988) TLXHart, S.G. Staveland, L.E.  1988\.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research Development of nasa-tlx (task load index): Results of empirical and theoretical research.  Human mental workload52139–183.
  * Javdani (2016) HindSightGithubJavdani, S.  2016\.  ada assistance policy. ada assistance policy.  https://github.com/personalrobotics/ada_assistance_policy.
  * Javdani . (2018) HindSightJavdani, S., Admoni, H., Pellegrinelli, S., Srinivasa, S.S. Bagnell, J.A.  2018\.  Shared autonomy via hindsight optimization for teleoperation and teaming Shared autonomy via hindsight optimization for teleoperation and teaming.  The International Journal of Robotics Research377717-742.
  * Kaljahi . (2019) GaborLandingDetectionKaljahi, M.A., Shivakumara, P., Idris, M.Y.I., Anisi, M.H., Lu, T., Blumenstein, M. Noor, N.M.  2019\.  An automatic zone detection system for safe landing of UAVs An automatic zone detection system for safe landing of uavs.  Expert Systems with Applications122319-333.
  * Kan . (2019) GroundEffectKan, X., Thomas, J., Teng, H., Tanner, H.G., Kumar, V. Karydis, K.  2019\.  Analysis of Ground Effect for Small-Scale UAVs in Forward Flight Analysis of ground effect for small-scale uavs in forward flight.  IEEE Robotics and Automation Letters443860-3867.
  * Kim . (2020) MixedRealityFPVKim, DH., Go, YG. Choi, SM.  2020\.  An Aerial Mixed-Reality Environment for First-Person-View Drone Flying An aerial mixed-reality environment for first-person-view drone flying.  Applied Sciences1016.
  * Lillicrap . (2016) DDPGLillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y.Wierstra, D.  2016\.  Continuous control with deep reinforcement learning Continuous control with deep reinforcement learning.  International Conference on Learning Representations.  [cs.LG]
  * Loquercio . (2020) DeepDroneRacingLoquercio, A., Kaufmann, E., Ranftl, R., Dosovitskiy, A., Koltun, V. Scaramuzza, D.  2020\.  Deep Drone Racing: From Simulation to Reality With Domain Randomization Deep drone racing: From simulation to reality with domain randomization.  IEEE Transactions on Robotics361-14.
  * Loquercio . (2018) DroNetLoquercio, A., Maqueda, A.I., del-Blanco, C.R. Scaramuzza, D.  2018\.  DroNet: Learning to Fly by Driving Dronet: Learning to fly by driving.  IEEE Robotics and Automation Letters321088-1095.
  * Maturana Scherer (2015) 3DConvLandingDetectionMaturana, D. Scherer, S.  2015\.  3D Convolutional Neural Networks for landing zone detection from LiDAR 3d convolutional neural networks for landing zone detection from lidar.  2015 IEEE International Conference on Robotics and Automation (ICRA) 2015 ieee international conference on robotics and automation (icra) ( 3471-3478).
  * Morando . (2022) PVInspectionMorando, L., Recchiuto, C.T., Calla, J., Scuteri, P. Sgorbissa, A.  2022\.  Thermal and Visual Tracking of Photovoltaic Plants for Autonomous UAV Inspection Thermal and visual tracking of photovoltaic plants for autonomous uav inspection.  Drones611.
  * Nogar (2020) Realsense1Nogar, S.M.  2020\.  Autonomous Landing of a UAV on a Moving Ground Vehicle in a GPS Denied Environment Autonomous landing of a uav on a moving ground vehicle in a gps denied environment.  2020 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR) 2020 ieee international symposium on safety, security, and rescue robotics (ssrr) ( 77-83).
  * Patrikar . (2022) TrajectoryEstimationPatrikar, J., Moon, B., Oh, J. Scherer, S.  2022\.  Predicting Like A Pilot: Dataset and Method to Predict Socially-Aware Aircraft Trajectories in Non-Towered Terminal Airspace Predicting like a pilot: Dataset and method to predict socially-aware aircraft trajectories in non-towered terminal airspace.  2022 International Conference on Robotics and Automation (ICRA) 2022 international conference on robotics and automation (icra) ( 2525-2531).
  * Perez-Grau . (2017) TeleOpSearchPerez-Grau, F., Ragel, R., Caballero, F., Viguria, A. Ollero, A.  2017\.  Semi-autonomous teleoperation of UAVs in search and rescue scenarios Semi-autonomous teleoperation of uavs in search and rescue scenarios.  International Conference on Unmanned Aircraft Systems (ICUAS) International conference on unmanned aircraft systems (icuas) ( 1066-1074).
  * Pfeiffer . (2022) EyeGazeIntentPfeiffer, C., Wengeler, S., Loquercio, A. Scaramuzza, D.  202203\.  Visual attention prediction improves performance of autonomous drone racing agents Visual attention prediction improves performance of autonomous drone racing agents.  PLOS ONE1731-16.
  * Polvara . (2020) SDQNPolvara, R., Patacchiola, M., Hanheide, M. Neumann, G.  2020\.  Sim-to-Real Quadrotor Landing via Sequential Deep Q-Networks and Domain Randomization Sim-to-real quadrotor landing via sequential deep q-networks and domain randomization.  Robotics91.
  * Reddy (2018) ReddyGithubReddy, S.  2018\.  Deep Assist. Deep assist.  https://github.com/rddy/deepassist.
  * Reddy . (2018) ReddyReddy, S., Dragan, A. Levine, S.  2018\.  Shared Autonomy via Deep Reinforcement Learning Shared autonomy via deep reinforcement learning.  Proceedings of Robotics: Science and Systems. Proceedings of robotics: Science and systems.
  * Rodriguez-Ramos . (2019) MovingPlatformDDPGRodriguez-Ramos, A., Sampedro, C., Bavle, H., Puente, P.D.L. Campoy, P.  2019\.  A Deep Reinforcement Learning Strategy for UAV Autonomous Landing on a Moving Platform A deep reinforcement learning strategy for uav autonomous landing on a moving platform.  Journal of Intelligent & Robotic Systems93351-366.
  * Sa . (2015) SharedAutonomyPoleInspectionSa, I., Hrabar, S. Corke, P.  2015\.  Inspection of Pole-Like Structures Using a Visual-Inertial Aided VTOL Platform with Shared Autonomy Inspection of pole-like structures using a visual-inertial aided vtol platform with shared autonomy.  Sensors1522003–22048.
  * Salter . (2021) AttentionPrivilegedSalter, S., Rao, D., Wulfmeier, M., Hadsell, R. Posner, H.  2021\.  Attention-privileged reinforcement learning Attention-privileged reinforcement learning.  Conference on Robot Learning.
  * Shah . (2018) AirSimShah, S., Dey, D., Lovett, C. Kapoor, A.  2018\.  AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles Airsim: High-fidelity visual and physical simulation for autonomous vehicles.  Field and Service Robotics Field and service robotics ( 621–635).  Springer International Publishing.
  * Shaqura . (2018) LaserGuidedUAVShaqura, M., Alzuhair, K., Abdellatif, F. Shamma, J.S.  2018\.  Human Supervised Multirotor UAV System Design for Inspection Applications Human supervised multirotor uav system design for inspection applications.  IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR) Ieee international symposium on safety, security, and rescue robotics (ssrr) ( 1-6).
  * Shi . (2019) NeuralLanderShi, G., Shi, X., O’Connell, M., Yu, R., Azizzadenesheli, K., Anandkumar, A.Chung, S.  2019\.  Neural Lander: Stable Drone Landing Control Using Learned Dynamics Neural lander: Stable drone landing control using learned dynamics.  ICRA Icra ( 9784-9790).
  * Smolyanskiy Gonzalez-Franco (2017) StereoscopicFPVSmolyanskiy, N. Gonzalez-Franco, M.  2017\.  Stereoscopic First Person View System for Drone Navigation Stereoscopic first person view system for drone navigation.  Frontiers in Robotics and AI411.
  * Spurr . (2018) CM-VAESpurr, A., Song, J., Park, S. Hilliges, O.  2018\.  Cross-Modal Deep Variational Hand Pose Estimation Cross-modal deep variational hand pose estimation.  IEEE/CVF Conference on Computer Vision and Pattern Recognition Ieee/cvf conference on computer vision and pattern recognition ( 89-98).
  * Vapnik Izmailov (2015) PrivilegedInformationVapnik, V. Izmailov, R.  2015\.  Learning Using Privileged Information: Similarity Control and Knowledge Transfer Learning using privileged information: Similarity control and knowledge transfer.  Journal of Machine Learning Research16612023–2049.
  * P. Wang . (2022) Realsense2Wang, P., Wang, C., Wang, J. Meng, M.QH.  2022\.  Quadrotor Autonomous Landing on Moving Platform Quadrotor autonomous landing on moving platform.  Procedia Computer Science20940-49.  Proceedings of the 2022 International Symposium on Biomimetic Intelligence and Robotics (ISBIR)
  * Y. Wang . (2019) SwarmReconnaissanceWang, Y., Bai, P., Liang, X., Wang, W., Zhang, J. Fu, Q.  2019\.  Reconnaissance Mission Conducted by UAV Swarms Based on Distributed PSO Path Planning Algorithms Reconnaissance mission conducted by uav swarms based on distributed pso path planning algorithms.  IEEE Access7105086-105099.
  * Xia . (2021) MDPLandingXia, B., Mantegh, I. Xie, W.  2021\.  Integrated Emergency Self-Landing Method for Autonomous UAS in Urban Aerial Mobility Integrated emergency self-landing method for autonomous uas in urban aerial mobility.  2021 21st International Conference on Control, Automation and Systems (ICCAS) 2021 21st international conference on control, automation and systems (iccas) ( 275-282).
  * Zhang . (2021) HapticUAVInspectionZhang, D., Tron, R. Khurshid, R.P.  2021\.  Haptic Feedback Improves Human-Robot Agreement and User Satisfaction in Shared-Autonomy Teleoperation Haptic feedback improves human-robot agreement and user satisfaction in shared-autonomy teleoperation.  2021 IEEE International Conference on Robotics and Automation (ICRA) 2021 ieee international conference on robotics and automation (icra) ( 3306-3312).