# HIFuzz: Human Interaction Fuzzing for small Unmanned Aerial Vehicles

Theodore Chambers<EMAIL_ADDRESS>0000-0003-2923-3230 University of Notre
Dame384A Fitzpatrick HallSouth BendUSA46556 ,  Michael Vierhauser
<EMAIL_ADDRESS>0000-0003-2672-9230 University of Innsbruck,
Department of Computer ScienceInnsbruckAustria ,  Ankit Agrawal
<EMAIL_ADDRESS>1234-5678-9012 St. Louis UniversitySt.
LouisMissouriUSA ,  Michael Murphy<EMAIL_ADDRESS>,  Jason Matthew
Brauer  D4ADenverColoradoUSA ,  Salil Purandare<EMAIL_ADDRESS>,  Myra B.
Cohen<EMAIL_ADDRESS>Iowa State UniversityAmesIowaUSA and  Jane Cleland-
Huang<EMAIL_ADDRESS>0000-0001-9436-5606 University of Notre Dame384A
Fitzpatrick HallSouth BendUSA46556

###### Abstract.

Small Unmanned Aerial Systems (sUAS) must meet rigorous safety standards when
deployed in high-stress emergency response scenarios. However, tests that
execute perfectly in simulation can fail dramatically in real-world
environments. Fuzz testing can be used to increase system robustness by
providing malformed input data aimed at triggering failure cases. In this
paper, we apply fuzzing to support human interaction testing. Initial tests
are run in simulation to provide broad coverage of the input space in a safe
environment; however, they lack the fidelity of real-world tests. Field tests
provide higher fidelity but can result in costly or dangerous crashes. We,
therefore, propose and demonstrate HiFuzz, which executes large numbers of
fuzz tests in simulation and then down-selects tests for deployment in human-
in-the-loop simulations and safety-aware physical field tests. We apply HIFuzz
to a multi-sUAS system and show that each test level serves a unique purpose
in identifying known and unknown failures associated with human interactions.

human-interaction, safety assurance, sUAS, Cyber-Physical Systems

††doi: ††copyright: none††ccs: Computer systems organization External
interfaces for robotics††ccs: Human-centered computing Interaction
devices††ccs: Software and its engineering

##  1\. Introduction and Motivation

Small Unmanned Aerial Systems (sUAS) need to meet rigorous safety requirements
when deployed in high-stress emergency response scenarios (Khan and
Neustaedter, 2019; Herdel et al., 2022). This requires careful design, and
systematic testing of the hardware, software, communication networks, and
human interaction points in a system, as failures in any of these components
can lead to erroneous behavior and even accidents (Hoang et al., 2023). In
this paper, we focus on the challenge of human interaction testing, where
incorrect and unexpected human actions and inputs can result in potentially
hazardous system behavior (Herdel et al., 2022). This problem is particularly
gnarly in complex, Cyber-Physical Systems (CPS), such as sUAS, where human
inputs impact multiple subsystems, and previously unexplored feature
interactions can trigger erroneous behavior (Luckeneder et al., 2017; DeVries
and Cheng, 2018). In over eight years of experience flying real-world,
autonomous, multi-sUAS systems, we observed that humans have played a
contributing role in many of the incidents that occurred. For example, during
a test flight in the Spring of 2023 (cf. Figure 1), one of our autonomous sUAS
breached a geofence, flew off its designated flight path, and ascended to an
altitude of 734 feet above ground level (AGL) – far above the legal limit of
400 feet AGL. A post-mortem analysis revealed a series of factors, including
human-related missteps, that contributed to the incident. The remote pilot in
charge (RPIC), who plays only a supervisory role under normal conditions,
failed to set appropriate geofence-breach actions prior to the mission, placed
the throttle in an incorrect position, lost situational awareness of the sUAS’
trajectory following the geofence breach, and failed to take timely action
when the sUAS started to fly off-course. These observations align closely with
prior studies reporting that human “errors” contributed to 65% to 85% of
reported accidents across several different CPS domains (Nagel, 1998; Kohn et
al., 1999; Fan et al., 2018). However, blaming the operator for these
accidents, without investigating and addressing potential weaknesses in the
design, is very shortsighted.

(a) The SW corner of the mission intersected with the geofence; however no
geofence-action was set.

(b) Flight replay showed that the sUAS flew north at increasing altitude after
the geofence breach.

(c) The RPIC must position the throttle correctly in case in-flight problems
require human control.

(d) Flight replay revealed that the RPIC had incorrectly set the throttle
above neutral.

Figure 1. Due to a combination of mistakes, including ‘operator error’ by the
Remote Pilot in Command, the sUAS flew off-route and ascended to 734 feet AGL.
Note: All required regulatory reports were filed describing the incident.

This set of figures shows snippets from an accident we experienced in Spring
of 2023 with our own sUAS application. Figures a and b depict flight paths
taken, and Figure c the radio transmitter used. Finally, Figure d shows the
replay and analysis from the PX log analysis component.

sUAS systems are complex in nature, and their design falls into the category
of “wicked” design problems (Rittel and Webber, 1973; Coyne, 2005) for which
there are no previously well-formulated solutions, no clear and definitive
test of validity, and little opportunity to learn by trial and error prior to
deployment (Rittel and Webber, 1973). Human-Centered Design (HCD) focuses on
creating and validating intuitive interfaces that are tailored to human
cognitive capabilities (Norman and Draper, 1986; Hao et al., 2022). However,
in today’s agile development environment, sUAS tend to be built and released
iteratively, thereby, increasing the likelihood that user-interaction design
flaws will emerge long after deployment. Without a systematic and rigorous
approach to human-interaction testing, designs that fail to anticipate and
address normal human “mistakes” (Endsley, 2011, 2017) will eventually lead to
unwanted behavior, causing potentially dangerous incidents at critical moments
of a flight. The problem is further exacerbated as CPS testing tends to
emphasize technical challenges, such as those related to flight trajectories,
collision avoidance, and computer vision, with limited focus on human-
interaction testing. A more systematic approach for detecting and mitigating
human-interaction design flaws in sUAS systems is therefore urgently needed to
provide a far deeper understanding of how the system performs when diverse
users interact with it in both expected and unexpected ways.

We address this need by presenting and validating the Human-machine
Interaction Fuzz testing framework named “HIFuzz”, where “HI” represents both
human interactions and the fact that sUAS fly at height. Fuzz is analogous to
traditional fuzz testing, where inputs are iteratively mutated and tested
against the system with a goal of covering a large part of the behavior
(and/or the code-base) of an application (Zhu et al., 2022). HIFuzz supports
both random and scenario-driven fuzzing, in which invalid, malformed, or
unexpected human inputs are injected into a system to reveal software defects
and vulnerabilities (Zhu et al., 2022). It includes three distinct levels (L1,
L2, L3) progressing from a low-cost, limited-fidelity, large-scale, no-hazard
environment, with fully simulated Proxy Human Agents (L1), via an intermediate
level, where proxy humans are replaced with real humans (L2), to a high-
stakes, high-cost, real-world environment (L3). The levels are separated by
two dedicated gateways. G1 resides between L1 and L2 and is responsible for
analyzing test results from L1, and down-selecting an appropriate set of tests
to be executed in L2; while G2 represents a significant safety gateway in
which standard safety assurance processes are followed, and hard decisions are
made about executing HIFuzz tests in the real world. Our tests are supported
by tools for generating and executing the Fuzz Tests. For example, in the case
of levels L2 and L3, where real humans participate in the tests, we have
developed a mobile app to interactively guide users through the actions they
need to perform during test execution. The aim of HIFuzz is to emerge and
address all human-interaction failures in advance of emergency deployments,
where failures could jeopardize mission success and potentially lead to loss
of life.

Our HIFuzz framework makes four key contributions. First, it presents a _novel
and systematic approach for human-interaction testing_ , aimed at detecting,
analyzing, and mitigating previously unknown hazards associated with human-
sUAS interactions. Second, it presents a _mobile application for guiding
humans_ through the interactive testing process. The app is designed for
scenarios in which humans are prompted to perform actions and to provide
interactive feedback during the test. Third, while _Fuzz Testing_ has been
commonly used for software and systems tests, to the best of our knowledge it
has not previously been used _for human-interaction testing_. Our HIFuzz
process, balances the need for executing thousands of tests that are
explicitly designed to find vulnerabilities and weaknesses, with safety
assurances needed for real-world sUAS flights, and is, therefore, a novel
contribution at the intersection of Human-Computer Interaction and Software
Testing. Finally, we _report on an extensive study_ , in which we applied
HIFuzz to our own multi-sUAS system and explored several critical research
questions. Results clearly show that HIFuzz can play a key role in revealing
weaknesses associated with human interactions, potentially leading to their
mitigation and improved design solution.

The remainder of the paper is structured as follows. Section 2 lays the
foundations by explaining how a single fuzz test is specified and executed.
Section 3 then presents the overall HIFuzz process, including its test levels
and gateways, while Section 4 describes the framework and supporting tools.
Section 5 presents our experimental evaluation where we applied HIFuzz to a
non-trivial sUAS system, and 6 provides comprehensive discussion and analysis
of the results. Finally, in Sections 7 and 8 we discuss related work and
limitations of our work.

##  2\. Defining an Individual HIFuzz Test

Fuzzing a CPS is more complex, and potentially more challenging, than fuzzing
a software-only application for several reasons. One reason is that CPS often
have a large state space due to the combination of both digital and physical
states, and actions taken by a human are likely to differ according to the
current state. Furthermore, CPS frequently exhibit real-time requirements,
making the timing and sequencing of events crucial - including the exact
timing of the human interaction (Fu et al., 2017; Yaacoub et al., 2020). A
fuzz test, therefore, needs to explore actions within the context of different
states and at different times.

A HIFuzz test is executed in the context of either a single or multi-sUAS
mission, and involves one or more humans performing a sequence of well-defined
tasks, each task executed at a point in the mission when an sUAS is in an
appropriate state. The characteristics of a mission will differ according to
the System-under-Test (SuT), but could include a simple flight along a series
of waypoints, or a more complex task such as search-and-rescue utilizing
computer vision. Examples of human tasks include issuing a command via a
hardware or user interface. A test passes if the mission runs to completion
without incurring an sUAS crash, deviating significantly from the planned
route, or raising a significant flight-time exception.

In this section, we define the properties that constitute the HIFuzz fuzz
space. They can be used by engineers to specify the test space for their own
SuT, thereby allowing the HIFuzz fuzzing engine to automatically generate and
run the initial set of level _L1_ tests. We divide the specification into two
parts – first describing the actors, interaction devices, and drones,
including their properties (props), and the environment. We utilize the term
“Drone” here, rather than sUAS, to emphasize the actual vehicle and its
onboard flight controller, as opposed to the term “sUAS” used throughout the
rest of the paper to describe the complete hardware and software system.
Secondly, we describe the test scripts composed of the mission flown by the
drone, and the sequences of human interaction tasks that form the core
component of each test.

Figure 2. An Individual HIFuzz Test consists of a Test Setup component and a
Test Script component and serves as an Input for our HIFuzz testing platform
and process.

This figure shows the 2 main parts that constitute a HiFuzz Test. First the
Test Setup, with the Rules, User Interface, Drones, and Simulation
Environment, and second, the Test Script comprised of the Mssion, HITs, HIT-
Sequences, States and Modes.

###  2.1. HIFuzz Test Setup: Actors, Props, and Environment

#### Roles:

Each human enacted action is assigned to a specific role such as a Remote
Pilot in Command (RPIC), Observer (OBS), Mission Commander (MC), or Safety
Officer (SO). We define $\mathbf{R}$ as the set of roles represented by
$\mathbf{R}=\\{r_{1},r_{2},r_{3},\ldots,r_{i}\\}$. Prior to executing the
test, depending on the current test level, either a human, or proxy-human
assumes the assigned role.

#### Interaction Devices:

Humans perform a test action by using an interface device such as the radio
control transmitter (RC), a GUI supported by a keyboard, mouse, and/or
joystick, or another type of haptic device (Neto et al., 2010; Yu et al.,
2014; Mallan et al., 2017; Chen et al., 2021). We define $\mathbf{UI}$ as the
set of all available user interfaces, represented as
$\mathbf{UI}=\\{ui_{1},ui_{2},ui_{3},\ldots,ui_{j}\\}$.

#### Drones and their Configurations:

Tests can specify a specific drone. This can be particularly useful in _L3_
-level tests, for finding discrepancies caused by differences in operating
environments, (e.g., different flight controller versions), exploring
differing capabilities (e.g., different hardware), or investigating multi-sUAS
human interactions in which more than one drone is involved. Inconsistencies
across drones can cause accidents when their behavior fails to meet the
human’s current mental model (Endsley, 2017). We therefore define $\mathbb{D}$
as the set of drones, represented as
$\mathbf{D}=\\{d_{1},d_{2},d_{3},\ldots,d_{l}\\}$. Further, each drone in
$\mathbf{D}$ can be configured by the user prior to flight – for example, by
setting a geofence around the drone or assigning it a unique RTL (return to
launch) flight altitude. We define $\mathbf{P}$ as a set of configurable
parameters for an sUAS given by
$\mathbf{P}=\\{p_{1},p_{2},p_{3},\ldots,p_{m}\\}$; however, low-level
parameter configuration, that normally occurs when tuning the flight-
controller (Han et al., 2022) is out of scope of this paper, and we assume
that each drone has been adequately tuned and is flight-worthy. Parameters of
interest are therefore limited to those exposed to the operator through
interfaces (e.g., GUI screens) that they are expected to access during pre-
flight setup.

#### Simulation Environment:

Finally, for Level L1 and L2 tests, depending upon the simulation environment
used, we can directly configure elements such as wind. We define $\mathbf{E}$
as a set of configurable environmental parameters given by
$\mathbf{E}=\\{e_{1},e_{2},e_{3},\ldots,e_{n}\\}$.

⬇

{

"Mission": "BASIC-WAYPOINTS",

"Environment": {

"Wind": {

"SPEED": "20KTS",

"DIRECTION": "NORTH"

}

},

"Roles": [

{

"Role": "RPIC",

"HITS": [

{

"ID": "1",

"Drones": ["GREEN"],

"Task": "MOVE THROTTLE TO +1",

"Mode": "OFFBOARD",

"State": "TAKING-OFF"

},

{

"ID": "2"

"Drones": ["GREEN"],

"Task": "SET MODE TO STABILIZED",

"Mode": "OFFBOARD",

"State": "FLYING"

}

],

"Interaction_Device": "RC TRANSMITTER"

},

{

"Role": "MC",

"HITS": [

{

"ID": "1"

"Drones": ["GREEN"],

"Task": "PRESS RTL BUTTON",

"Mode": "STABILIZED",

"State": "FLYING"

}

],

"Interaction_Device": "GUI"

}

]

}

Listing 1: A single test defined for the Sequence of Human Interaction tasks
assigned two roles. The mission and environmental variables are shared across
the entire test

###  2.2. HIFuzz Scripts

Humans (serving in a specific role) enact a human-interaction task (HIT) in
the context of an sUAS mission. Further, they execute the HIT when the sUAS
and/or mission is in a specific state. For example, the RPIC might be asked to
perform the action of switching to position mode when the drone is flying in
offboard mode. This leads to the following specifications.

#### Missions:

A mission determines the flight plans and other tasks that one or more sUAS
will execute to provide context for the test. The way that the mission is
defined is specific to the SuT. We provide an example later in this paper as
part of our case study. Nonetheless, we define $\mathbb{MSN}$ as the set of
available missions, represented as
$\mathbf{MSN}=\\{m_{1},m_{2},m_{3},\ldots,m_{q}\\}$.

#### Human Interaction Task (HIT):

There are two types of HIT that a human will perform during a test. First, the
human could provide input to an individual sUAS through a hardware device such
as the RC – for example, by increasing the throttle, holding down the kill
switch, or switching between modes. Second, the human could send a command to
one or more sUAS via a GUI – for example, issuing a global RTL command. We
define $\mathbb{HIT}$ as the ordered set of interaction tasks performed by a
user, represented as
$\mathbf{HIT}=\\{hit_{1},hit_{2},hit_{3},\ldots,hit_{r}\\}$.

However CPS behavior is impacted by the current state of the system.
Therefore, each HIT has an associated set of preconditions that also need to
be defined. These preconditions are based on modes, flight life-cycle states,
and configurations. Modes are used by almost every flight controller to
support common flight tasks such as take-off and loiter, and to provide
various degrees of flight stability (e.g., stabilized and position-hold)
(Ardupilot, 2023; PX4, 2023a). We define $\mathbf{M}$ as the set of flight
modes, given by $\mathbf{M}$ = $\\{m_{1},m_{2},m_{3},\ldots,m_{s}\\}$, where
each mode $m_{i}$ in $\mathbf{M}$ is reachable in the SuT. We also define
$\mathbf{S}$ as a set of flight life-cycle states such as taking-off, flying,
and landing, given by $\mathbf{S}=\\{s_{1},s_{2},s_{3},\ldots,s_{t}\\}$. A
drone can only be in one mode and one state at any time. Finally, we define
configurations as the value assigned to any underlying parameter defined
earlier as $\mathbf{P}$. Each HIT must define a mode and life-cycle
precondition, and can optionally define a set of configuration parameters that
serve as preconditions. Further, the precondition state must be reachable in
at least one of the defined missions in order for any subsequent HIFuzz test
to be valid.

Figure 3. HIFuzz Levels and Gateways

###  2.3. Defining the HIFuzz Actionable Test

Based on these definitions, we can now specify an individual HIFuzz test in a
way that is sufficiently formal for automating test execution, but also
readable to humans who serve as participants in the testing process. We
utilize JSON to represent each test as shown in Listing 1. The test definition
includes the mission, environmental factors, roles, the locally sequenced HITS
and preconditions performed by each role using a specific interaction device
and drone. The JSON specification is used by the HIFuzz Test Runner to
coordinate the test, and by the mobile App to generate natural language test
stories.

The HIFuzz fuzzing engine ultimately uses these definitions to generate
diverse tests, composed of various combinations of roles, interaction devices,
drones, configurations, environmental factors, missions, and HITs.

###  2.4. Test Outcome

Each fuzz test is ultimately executed within the HIFuzz platform, and its
outcome is evaluated across two different dimensions – first to determine if
the test was valid or invalid, and second to determine if valid tests passed
or failed. An invalid test is one that fails to execute the full sequence of
HITS, typically because preconditions for one or more of the HITS are never
met. The outcome of valid tests is assessed as passed of failed based on
Mission Completion criteria. These criteria are adjusted for different systems
and domains, and also for different test levels, but will typically involve
factors such as mission completion and mission adherence.

##  3\. HIFuzz Process: Test Levels and Gateways

The HIFuzz process involves three testing stages (L1-L3) separated by two
gateways (G1, G2), each of which serves a unique purpose. Individual tests are
executed at each stage, however, the way they are executed, the role of human
stakeholders, and the safety analysis that is performed prior to test
execution differ greatly across stages. In this section, we therefore describe
the role and functionality of each stage and gateway.

###  3.1. Level L1: Large scale, simulated, fuzzing

The goal of L1 is to execute a large number of tests, as quickly as possible,
without any of the risks involved in real-world sUAS flights. Therefore, L1
tests are run in the simulator using proxy human agents instead of humans. In
the physical world, humans interact with sUAS via hardware devices, such as RC
transmitters, and their inputs are encoded into radio signals transmitted to
the flight controller and transformed into flight commands (e.g., throttle,
yaw, pitch, and roll adjustments, or mode changes). These inputs can be
simulated through software-based, low-level function calls to the flight
controller. Humans also interact with sUAS via GUIs, and these interactions
can be simulated if the SuT exposes its API function calls. Utilizing these
techniques, L1 is able to simulate human interactions (i.e., HITS) entirely in
software, enabling thousands of fuzz tests to be run in a low-cost, low-
effort, non-hazardous environment.

The L1 process starts with a planning task in which the HIFuzztester specifies
the test features that constitute the fuzz space. As described in 2, these
include roles, interaction devices, drones, environmental factors, missions,
and HITs. The HIFuzz fuzzer then uses this specification to automatically
generate combinations of the defined concepts and input values constrained by
specific scenarios of interest. The set of generated fuzz tests are then
executed in simulation, and results are evaluated to determine if the test
passed, failed, or basically untested if the sUAS completed its mission
without the preconditions ever being met. All passed and failed outcomes are
passed to Gateway G1.

Figure 4. The HIFuzz framework supports tests at all three levels. L1 operates
fully in a simulated environment with support from a fuzzer and a proxy human
agent. L2 operates with real humans in an otherwise simulated environment, and
L3 operators in the physical world.

###  3.2. Gateway G1: Downselecting for Human-in-the-Loop Tests

G1 serves as a gateway between levels L1 and L2, and is responsible for
selecting tests to be passed to L2. Its inputs are therefore the test result
data from L1, which it uses to construct a profile constituting the complete
set of L1 passed and failed tests and results. The tests are clustered to
identify groupings of similar tests and outcomes, and the clusters used to
guide the L2 test selection process. We describe the concrete clustering
approach that we took, based on experimentation with our data, in Section 5.

By default, we pass at least one representative test from each cluster to L2.
However, the initial number of clusters can be predefined according to
budgeted time or can follow a standard approach such as the “elbow-approach”
which looks for the sweet-spot in terms of coupling and cohesion (Syakur et
al., 2018).

###  3.3. Level L2: Humans in Simulated Environment

L2 executes a small subset of L1 tests, selected in the G1 gateway. The tests
are again executed in a simulation environment, however, this time humans
replace the proxy agents, and interact with the sUAS through the same hardware
devices (e.g., RC transmitters) and GUIs used in physical deployments. Level
L2 is designed to provide higher degrees of fidelity than L1, while operating
within a completely safe testing environment; however, it introduces higher
testing costs with respect to human time and effort. By integrating humans
into the testing environment, L2 allows us to issue commands directly from the
RC transmitter used in the field, providing increased fidelity of user inputs,
and allowing direct observation of the sUAS behavior by human operators.

Table 1. All failed tests from Level L2 are evaluated to identify root causes. This example shows a failure analysis for the geofence breach described in the introduction (Section 1). Following a careful assessment of the failure, three contributing root causes were identified, and immediate and long-term mitigations were proposed. Once fixes are applied they are individually tested in simulation (L2) and then field-tested. The entire scenario will be rigorously retested across levels L2 and L3 once all three long-term solutions have been implemented and deployed. Observed Failure |  |  |  |  |  |
---|---|---|---|---|---|---
The sUAS hit the geofence and then flew off-course and ascended to 734ft AGL. |
Observed in the field, and then replicated in L1 and L2. |
Root Cause Analysis |  |  Crit. |  Mitigation |  |  L2 Status |  Field Test
1\. The RPIC failed to set geofence action. |  |  MED |  Immediate: Check and set geofence settings on all drones to loiter. |  |  completed |
|  |  Long-Term: Additionally, update prearming checks to warn user if a geofence is activated without a defined action. |  |  back-logged |  on-hold
2\. PX4 requires sUAS to have throttle fully down for arming. However, if the
RPIC were to assume control during the flight for any reason, and the throttle
is in the fully down position, then the sUAS will plummet rapidly to the
ground. Therefore, the existing flight protocol required the RPIC to move the
throttle to the neutral position during takeoff. As the RPIC is often
responsible for more than one sUAS, it is easy for them to accidentally
position the throttle slightly off neutral.
|  |  HIGH |  Immediate: Safety Officer role added to supervise the takeoff routine for multi- sUAS flights.  |  |  completed |
|  |  Long-Term: Allow the companion computer to arm the sUAS without the required throttle stick input allowing the sUAS to arm with the throttle at neutral. Solution: Enter offboard mode prior to arming, so that the drone ignores all RC inputs (except for mode changes).  |  |  passed |  passed
3\. The RPIC lost situational awareness of the sUAS after it flew off course. RPIC was aware that it was off-course but not aware that it was flying above legal altitude limits. |  |  HIGH |  Immediate: Train RPIC to check altitude on RC Transmitter if sUAS flies off-course. |  |  completed |
|  |  Long-Term:  Provide verbal altitude warnings to the RPIC.  |  |  back-logged |  on-hold
Flight readiness for L3:  NOT READY FOR FIELD-TESTING |

###  3.4. Gateway G2: Safety Assessment and Mitigation

While Test levels L1 and L2 seek to safely explore mission-breaking human-
interaction faults that potentially cause erratic sUAS behavior, such as
crashes and flight deviations, level L3’s real-world deployment means that
failures are potentially hazardous and costly. Therefore, Gateway G2 serves as
a safety gateway that ensures that each failed test from L2 is carefully
assessed to determine if mitigations are needed, and that all tests deployed
on the field with physical sUAS have undergone a rigorous hazard analysis and
that any identified hazards have been sufficiently mitigated. The aim is to
(1) assess human-interaction vulnerabilities and flaws identified in levels L1
and L2, (2) mitigate them, (3) repeat level L2 tests to demonstrate that they
have been successfully mitigated, and only then (4) proceed to level L3 tests.
HIFuzz does not dictate how the safety assessment should be performed as long
as the process assesses hazards associated with each test case, e.g., using
Fault-Tree Analysis (FTA) or Failure Mode Effect Criticality Analysis
(FMEA/FMECA) (Reifer, 1979; Sullivan et al., 1999; Leveson and Harvey, 1983),
evaluates mitigations to determine whether the risk has been satisfactorily
addressed, and when needed, provides a semi-formal safety case, e.g., a Safety
Assurance Case (SAC) that includes guidelines targeted at the human
participants describing how the test can be safely conducted in the field.

For illustrative purposes, Table 1 depicts how the outcome of a safety
assessment for three root causes of the previously discussed fly-away incident
could be structured. This analysis represents a summary of the outcomes of
standard safety-analysis techniques. In this case, the sUAS hit the geofence,
flew off-course, and ascended to 734ft AGL. For each case, we identified an
immediate and a longer-term mitigation strategy. For example, the immediate
solution for the RPIC failing to set the geofence action is to manually check
that each drone is configured with a geofence action, whereas the longer-term
action is to integrate this task into the automated pre-arming state. Neither
action has been performed yet, and, therefore, the test is not considered
ready for deployment. In fact, only one of the problems (#2) has been fully
addressed.

###  3.5. Level L3: Field Testing with humans-in-the-loop

The goal at level L3 is simple. Execute all tests that (a) have previously
produced a failed L2 outcome, (b) have been demonstrably mitigated, and (c)
have a constructed safety case providing sufficient evidence that the problem
has been resolved. Real-world flights introduce the very real potential for
actual incidents, including costly loss of equipment that could occur when
humans intervene in ways that the system is unable to handle; however, L3
tests are critical for demonstrating that solutions tested successfully in
simulation are also effective in the physical world. Further, L3 testing
allows tests to be conducted when operators are fully aware of potential
problems, rather than during a live deployment when attention is on the
mission rather than the tests, and where unwanted behavior could impact the
successful outcome of a life-or-death mission. We provide examples of L3 tests
in the evaluation section of this paper.

##  4\. The HIFuzz Framework

In this section, we describe the major components of the HIFuzz framework that
support the functionality required by test levels (L1-L3) and gateways (G1,
G2). Level L1 accepts a specification of the fuzzing space, and then uses a
fuzzer to generate thousands of tests, and a _test runner_ to coordinate their
execution in a high-fidelity _simulation environment_. The test runner and
simulation environment are also used by L2. In L1, the human is replaced by a
proxy human agent supported by a _simulated RC interface_ ; while in levels L2
and L3 the humans need to be prompted to perform each task at the right stage
of the mission. The HIFuzz platform supports this through a novel _mobile
prompter_ , controlled by the test runner. The platform also provides a _test
verifier_ which evaluates whether each executed test has passed or failed.
Finally, the framework supports gateway G1 through a _gateway filter_ which
clusters test results and supports the tester in selecting tests to pass to
level G2. In our current implementation of HIFuzz, Gateway G2 utilizes only a
spreadsheet-based template, but could be augmented by external safety analysis
and assurance tools.

###  4.1. Test Fuzzer

The fuzzer takes as input a specification describing concrete instances of
roles, drones, HITs, etc., as defined in Section 2. There are many different
ways to generate fuzz tests. Fuzzing is a random test generation approach (see
(Zhu et al., 2022) for a survey) originally developed to find security flaws
in programs by running the program with a large number of random malformed
inputs (Miller et al., 1990). It has evolved to include feedback (coverage-
guided) (Padhye et al., 2019; Böhme et al., 2016; Fioraldi et al., 2023) and
smart search heuristics (search-based fuzzing) (Böhme et al., 2017; Olsthoorn
et al., 2021). Some fuzzing works at the specification level (often using an
input grammar) (Bendrissou et al., 2023; Godefroid et al., 2008; Srivastava
and Payer, 2021) while the majority of fuzzing techniques are greybox (using
code-guided metrics to diversify coverage of program paths in the code)
(Padhye et al., 2019; Böhme et al., 2016; Fioraldi et al., 2023; Böhme et al.,
2017; Olsthoorn et al., 2021). Finally, scenario-based approaches represent an
alternative approach for specification-based fuzzing (Sun et al., 2023; Chen
et al., 2020; Han and Zhou, 2020). We adopt this approach because it is well-
matched to the scenario-based incidents that we observe over time.

Randomly generated fuzz tests select elements from each of the specified sets
(e.g., roles, modes, states, actions) guided by a simple set of heuristics
that are designed to avoid infeasible combinations. For example, a random
approach might select a role, a drone and its configuration, a mission, a
sequence of HITs and their preconditions, and an associated interaction
device. While random fuzzing can provide coverage of a large problem space, it
can be inefficient at targeting specific problem scenarios (Liang et al.,
2014; Hu et al., 2021; Kim et al., 2022). Scenario-driven fuzz testing
addresses this challenge by directing resources to explore known problem areas
in greater detail. A scenario-based approach defines one or more missions that
execute the problem scenario and constrains the test configuration space to
focus only on the scenario. For example, the scenario described in the
introduction of this paper focused on how the human interacts with the system
in preparing for, and responding to, a geofence breach – especially one for
which no geofence is set. To fuzz around this scenario, we would select a
mission that includes a geofence breach, and test various combinations of
geofence settings and human responses (such as switching modes after the
breach or setting the throttle at different levels before and after).

###  4.2. HIFuzz Test Runner

The test runner iterates through the generated tests, performing slightly
different functions depending on whether a real-human or proxy-human is
involved. At Level L1, the test runner invokes the mission in the simulation
environment, monitors the runtime state of each drone, checks for precondition
states, and delegates HITs to the Proxy Human Agent (PHA) when precondition
states have been reached. At levels L2 and L3, humans are responsible for
invoking each mission; however, prior to the start of the mission various
participants need to know (1) how each drone should be configured (e.g.,
geofence actions), (2) which mission to invoke, and (3) when to perform a
specific HIT during the mission. The test runner tracks the state of each
mission and sends timely prompts to the mobile prompter which guide each human
participant to perform their HIT at the right time. Our test environment
assumes that the SuT provides an API that allows the test runner to register
for, and receive, status updates conveying information about the current state
and mode of each drone.

Figure 5. HIFuzz Prompts are shared with human test participants via a mobile
app. Here we show the design of the tester’s precheck screen (1), followed by
a series of prompts shared with the RPIC (2a-g), MC (3), and SO (4) roles
respectively. Figures represent the design which was fully implemented and
deployed using React-Native.

###  4.3. Mobile Prompter

Determining the right time to perform a HIT can be challenging for humans
because many of the HIT’s precondition states are internal to the drone and
not readily visible to human observers. HIFuzz, therefore, provides a mobile
app responsible for generating timely prompts. Furthermore, humans engaged in
Level L3 tests, are not just test participants, but are front-line pilots with
all the responsibilities and pressures brought about by that task. The mobile
app is designed to minimize unnecessary mental overload of processing and
responding to prompts, and is designed with a simple GUI which gives the user
planning time as well as clear instructions on what actions to perform. We
designed and implemented the mobile app following principles of human-centered
design, and our two test participants reported that it was intuitive and gave
them clear and timely directions. However, a full assessment of the mobile app
is outside the scope of this paper, and we therefore present it as a
supporting tool rather than a primary contribution of this work.

We provide example screens, including a possible workflow, in Figure 5. Screen
1 is designed as a summary of the upcoming flight including any potential
safety concerns. This is essential so that each participant is fully aware of
what the drone will do during the mission, and can intervene if needed. It
also provides each participant with a list of tasks they will need to perform,
giving them time to mentally run through the actions they will need to take.
The test runner waits for all participants to check the “proceed” button
before starting the test. Screens 2a-g represent a potential sequence of
prompts sent to an RPIC during a single test. In 2a the user enters their role
(i.e., RPIC in this case) and acknowledges that their first responsibility is
to safety rather than the test. All HITs planned for the test are supported by
a sequenced pair of plan and go screens. The plan screen gives the RPIC time
to mentally prepare for the upcoming task, while the go screen tells them to
perform the task “now”. We only engage trained personnel in these tests, with
the expectation (as required by regulators) that all participants are fully
trained in their roles and know which switches and knobs to manipulate in
order to execute the intended task. We notify the RPIC (Screen 2F) when there
are no more test actions, and then conclude in 2g by asking awareness
questions. Screen 3 is targeted at the Mission Commander who is responsible
for launching missions from the Ground-based Control station, and Screen 4 is
an example screen for the Safety Officer who has no direct control over any
drone and is responsible only for observing actions and checking that
protocols are followed.

###  4.4. Replacing Humans with Proxies in L1

Proxy Human Agents replace humans in level L1 tests. In particular, the PHA
serves two distinct functions for mimicking human inputs on hardware devices
such as the RC,and mimicking human inputs via a GUI. One of the most critical
components of the PHA is the simulated RC which sends commands directly to the
flight controller. For example, on a physical RC, when a human moves a lever
to change modes from stabilized to land, a radio signal is transmitted by the
RC (transmitter) to the drone’s flight controller (the receiver). However, in
level L1, we replace the radio signals with mavros manual_control messages
(Ermakov, [n. d.]) which, among other things, define the targeted system, and
simulate RC movements for various switches and button presses including mode
changes, throttle position, and the kill-switch. Mavros converts ROS messages
to the MAVLink protocol (DroneCode, [n. d.]), which is the standard publish-
subscribe messaging protocol for sUAS applications.

L1 tests are executed fully automated, currently composed of scenario-driven
combinations of geofence breaches, mode switches, throttle positions, and the
status of the drone during a mission. The HIFuzz test runner runs a single
test per mission, and monitors the status of the mission. L1 tests are run in
a high-fidelity simulation with a physics environment and continue
exhaustively until all combinations have been checked for the isolated
scenario. At the end of each mission, a reset is performed where the simulated
sUAS is set to a default configuration to ensure pristine conditions for each
simulation run. This enables the tests to run without interference for long
periods of time, yet still replicate a human agent.

###  4.5. sUAS Simulation Environment

Both, levels L1 and L2 require a simulation environment that accepts and
executes a mission request – potentially involving multiple drones, reports
the progress of each drone throughout the mission, reports error messages, and
produces a readable flight log at the end of each flight. Furthermore, the
interface of the simulation environment must match that of physical drones to
support a seamless transition from L2 to L3. Common examples of simulation
environments that can be used to meet these requirements are Gazebo (Open
Robotics, 2023), jMAVSim (PX4, 2022), and AirSim (Shah et al., 2018). Our
system is deployed using Docker (Merkel, 2014) in conjunction with Docker
Compose, facilitating simulation scalability, with multiple containers being
deployed, and achieving consistency across simulation runs. Docker Compose
ensures a set of linked containers within a dedicated software-defined
network, which includes our sUAS onboard software, MavROS and PX4 to interface
with a drone, the Gazebo simulation environment to provide a physics engine
and simulated sensor data, and lastly MQTT for communication and message
exchange. Additionally, for level L2 we use QGroundControl (QGroundControl,
2021) to connect a physical RC controller via the MAVLink protocol to allow a
human to control the (simulated) drone.

###  4.6. HIFuzz Test Assessment

HIFuzz utilizes an ensemble of test oracles and techniques to determine
whether each flight has been executed correctly. These include runtime alerts,
flight log analysis, and human feedback via the mobile app. For runtime
alerts, we capture all the alerts generated by both the flight controller and
our own software system. For our flight log analysis, we extract time series
data from the mission logs related to the position and flight behavior of the
sUAS. First, we establish a “blueprint” flight log, which represents an ideal
mission execution, and then use it as a point of comparison for all tests
executing the same mission. For each log in the dataset, we measure its
deviation from the blueprint flight path by checking, for each position
timestamp in the blueprint, the distance to the nearest sUAS position in the
current log across the x, y, and z axes. Then we record the largest such
distance as the maximum observed deviation of the current log from the
blueprint. We also extract other features from each log in the set, such as
the maximum altitude, the duration of the flight, the occurrence of freefalls,
the final landing state, and the reported mission status from the drone.

##  5\. Experimentation: HIFuzz Applied to D4A System

We conducted our primary evaluation of HIFuzz using a multi-sUAS system that
we have developed over several years and deployed in the real world as the
System-under-Test. For double-blind review purposes, we refer to this as
‘Drones for All’ (D4A). Given the scope of this work, our evaluation focuses
upon the effectiveness of the levels (L1-L3) and gateways (G1, G2), rather
than on the tools we have developed (i.e., the Mobile App), or the safety-
assessment (i.e., based on standard FMECA). We therefore focus on addressing
two specific research questions.

  * RQ1:

_What kinds of system failures, associated with human interactions, could be
identified during the HIFuzz process?_
This first research question aims to establish a general understanding of the
types of failures that were detected using HIFuzz.

  * RQ2:

_Did each of the three test levels, and two gateways play a unique role in the
human-interaction testing process, or did they all reveal the same set of
problems?_
The second research question aims to ascertain how, and to what extent,
detected problems differ on the three levels, with and without human
involvement.

The experiments described in this section were all executed in our HIFuzz
platform as described in Section 4.

###  5.1. System under Test: D4A

The two research questions were evaluated by applying HIFuzz to the D4A
system. D4A is a distributed multi-user, multi-sUAS system, designed to
support search-and-rescue (ANONYMOUS, 2023) and aerial data collection and
surveillance activities (ANONYMOUS, 2023). Each sUAS is equipped with an
Onboard Autonomous Pilot (OAP) organized around a state machine which is
dynamically configured for each mission. States represent specific tasks such
as takeoff, search, or fly-to-waypoint and vary greatly in complexity. For
example, in the takeoff state the sUAS ascends to a predefined altitude, uses
its onboard sensors to detect when it reaches that altitude, and then
transitions to a subsequent state such as fly-to-waypoint; while a search
state utilizes extended onboard computer vision capabilities to detect
objects, analyze the results, and make subsequent decisions, such as to track
a person.

A Ground Control Station (GCS) provides coordination and support services
between humans and sUAS. It is built around a set of microservices loosely
connected via a message broker (Muccini and Moghaddam, 2018). Each
microservice supports a specific capability such as airspace leasing, multi-
sUAS coordination, or a task sequencing validity checker/Microservices
interact via MQTT (MQTT, 2023) with human operators and sUAS. Status data
(e.g., GPS location, battery, health) and task progress updates (e.g., current
task, potential adaptations), are sent to the GCS by both humans and sUAS to
support monitoring, analysis, and planning. Under normal operating conditions,
human-machine interactions are primarily GUI-based (e.g., providing support
for mission planning and facilitating situational awareness) via a dedicated
web application; however, in case of an emergency, or to temporarily assume
control for tasks that the machine has not yet been trained to perform, humans
can directly issue commands to sUAS via handheld radio controllers.
Communication is primarily asynchronous with data sent over a mesh radio with
various QoS (Quality of Service) guarantees about message delivery.

Github Issue #271: Posted by: murphym18, 08/03/2023
On August 3, during a flight test at peppermint field, a near-crash incident
occurred that exposed a safety issue in our current mission format. Right now
we specify the altitude as meters above sea level. But due to several unlucky
coincidences, on Aug 3rd the flight controller ended up with an incorrect
altitude reading.
When the drone was flying home, it ended up flying to a waypoint that was
alarmingly close to the ground. This required immediate intervention by the
Remote Pilot in Command (Jane). The incident was not only intense and risky
but could have resulted in significant damage. Figure 6. An issue posted to
Github describing a human-interaction incident, where the RPIC was forced to
take control due to an altitude anomaly on the drone.

###  5.2. Scenario Based Fuzz Tests

Given the large feature space including all possible states, modes, roles,
drones, actions, environmental conditions, and reasonable values for PX4
settings which humans are likely to configure prior to flight (e.g., geofence,
battery failsafe), produces a vast test space of approximately 8,989,344 test
configurations. If we add additional fuzzing to the specific times at which
each task is performed, then this number can grow significantly higher. We,
therefore, adopted a scenario-based approach. To select appropriate scenarios,
we browsed through all of the 272 issues (dated from 07/24/21 to 08/31/23)
reported in our D4A GitHub repository with the aim of identifying incidents
reporting failures associated with a human at the field, such as the example
depicted in Figure 6. We selected two incidents as shown in Figure 2, and for
each of them, we provided a general description of the problem and our
observations from inspecting their flight log.

###  5.3. Modeling the test space

For each incident, we then defined all relevant roles, simulation environment
configuration (wind, geofence), drones, flight modes, states, and human tasks
needed to recreate conditions for the selected scenarios. For example, to
replicate Scenario 1, and to fuzz-test its variants, we created a flight route
that intersected a geofence, defined geofence status (on, off), and the
complete set of available geofence actions for PX4. We included all reachable
modes and states, three different drones, and three roles. Each of our drones
is represented by a unique color; blue, orange, and purple. Furthermore, all
of our drones have identical RC transmitter setups and capabilities,
therefore, we did not need heuristics constraining which HITs should be
applied to which drones. As our drones are not all configured identically due
to gradual drift from the original configuration, we included three different
drones in the tests, for use at level L3, where slightly different
configurations may give different results. Figure 7(a) shows the layout of the
D4A radio controller on the left, with some of the possible configuration
settings on the right. As some switches are inert (i.e., do not trigger a
change in state) we focused on mode changes that are supported. The complete
set of available modes is also shown in Table 3. We added two representative
wind properties, and several properties associated with geofence settings.
These included specifying whether the geofence was activated or not for each
test, and if activated, whether the distance to the geofence was predicted or
not, and providing several geofence actions that could occur upon breach. For
all internal sUAS parameters we accepted values as configured during prior PID
tuning (proportional–integral–derivative controller). Finally, we included
three human actions (HITs). Two of them were directly related to our scenario
and included mode changes and throttle adjustments. The third, action was the
emergency kill-switch (which immediately shuts down all motor functions of the
drone), as this is important for any failed takeoff, hard landing, or
potential fly-away (albeit at great cost and safety concern, if used in the
air). This produced a smaller test space of approximately 160,524 test
configurations.

We applied a systematic approach to generate combinations of configurations
and actions (as explained in Section 4.1) and fuzzed the exact timing at which
each action was executed once its preconditions were found to be true.
Finally, we created a simple flight test involving one drone taking off,
flying to two waypoints, and returning home.

Table 2. Two scenarios were selected in which human interactions were associated with flight failures. These scenarios were used in our experiments to drive scenario-based Fuzz Testing. The image for Scenario 1 is a replay from the physical flight logs capturing an actual incident that occurred on the flying-field, while the image for Scenario 2 is taken from an L2 level simulation. Note: unless otherwise specified, modes and parameter values used below are described in the PX4 documentation (PX4, 2023a, c). Scenario 1: Geofence breach |  |  Scenario 2: Flawed take-over
---|---|---
Observed: The drone hit the geofence with no geofence actions set and switched to stabilized mode. It then ascended rapidly and flew North. The RPIC had accidentally set the throttle just above neutral at the start of the flight. |  |  Observed: The RPIC took control of the drone by switching to STABILIZED mode with the throttle down. The drone oscillated as it attempted to stabilize and had a hard landing.
|  |
Fuzz Scenario: Establish diverse geofences. Fly drone through geofence, where the RPIC defines geofence actions and sets throttle to various positions. As wind was suspected to be a factor, repeat in different wind conditions. |  |  Fuzz Scenario: Test action of ceding control from the autonomous pilot to RPIC. Whilst drone is actively arming, disarming, or in the air, switch to POSITION mode with throttle in diverse positions.
Table 3. Actual specification of the HIFuzz fuzzing space used for experimentation purposes. Legend: blue= Initial states and modes are colored blue, yellow = configuration settings, orange=drones, green=human tasks. For Level L1 we only utilize the RPIC role and BLUE drone. Further Geofence_Pred = ‘On’ $\Rightarrow$ Geofence_stat=‘On’ AND Geofence_ACT $\Rightarrow$ Geofence_Stat=‘On’. This combination of features produced a test space of approximately 160,524 tests assuming no additional fuzzing around the precise timing of each test.  Modes |  | States |  | Throttle POS |  | Wind |  | Geofence Act. |  | Roles |  | Human Tasks
---|---|---|---|---|---|---|---|---|---|---|---|---
ALTCTRL |  | Pre-arm |  | Maximum HIGH |  | Medium Northerly |  | 0: None |  | RPIC |  | CHANGE-MODE
POSCTRL |  | Arm |  | Medium HIGH |  | High Northerly |  | 1: Warning |  | MC |  | MOVE-THROTTLE
OFFBOARD |  | Takeoff |  | Just above neutral |  |  | 2: Hold mode |  | SO |  | KILL-MOTORS
STABILIZED |  | Fly |  | Neutral |  | Geofence Stat. |  | 3: Return mode |  |  |
AUTO.LOITER |  | Hover |  | Just below neutral |  | On/Off |  | 4: Terminate |  | Drones |  |
AUTO.RTL |  | Land |  | Medium LOW |  | 5: Land mode |  | BLUE |  |
AUTO.LAND |  |  | Maximum LOW |  | Geofence Pred. |  | ORANGE |  |
| On/Off |  |  | PURPLE |  |

(a) The RC Transmitter showing knobs and switches which can be manipulated

Mode Description Switches SC SD SG SE SI STABILIZED Self-levels role and
pitch. MID ANY ALTHOLD Holds altitude, self levels, roll &pitch. UP DOWN
POSCTRL Holds altitude & position. DOWN UP OFFBOARD Obeys position, velocity,
acceleration, attitude, attitude rates. UP UP LAND Lands in place BACK RTL
Returns-to-launch at predefined altitude MID GeoFence Activates geofence FRWD
GeoFence Deactivates geofence BACK Kill Switch Kills motors after X Seconds
PRESS

(b) Key Modes and their switch positions on the RC Transmitter.

Figure 7. The RPIC has the ability to switch between modes, activates the
kills switch and controls throttle and movement using the knobs and switches
on the RC Transmitter.

###  5.4. Results from applying HIFuzz to D4A

We executed all levels (L1 - L3) and gateways (G1, G2) for the planned
scenario-based fuzzing of the D4A system with the following setup.

####  5.4.1. L1 Tests:

We executed approximately 700 L1 tests based on various combinations of the
specification in Table 3. The test preconditions, configurations, and actions,
were collected in a profile, as illustrated through a representative sampling
shown in Table 4. Each test result was flagged with outcomes including the
maximum altitude reached, flight duration, landed and state, and mission
completion. Any test exhibiting excessive altitudes, duration, excessively
fast landing, or failure to complete the mission with final disarm, was
labeled as “Abnormal”. We did not use the term “passed” or “failed” because
unwanted human interactions can result in strange flight patterns meaning that
the test might officially pass, but simultaneously exhibit undesirable
behaviors.

Table 4. A subset of the test case results run in levels L1, L2, and L3. Legend: ●$~{}=$ Failed Test, ○$~{}=$ Passed Test, ◗$~{}=$ Issue logged for design discussion, ◑$~{}=$ Mitigation backlogged for implementation.  Test | L1 | L2 | L3 | Mode | MSN | GF | GF | GF | WIND | Switch | Kille | Throttle | Max | Max | Dur. | Land | MSN | CL
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
No. | State | PRED | ACT | TYPE | Mode | Switch | Pos | Dev | Alt | Comp |
14 | ● | ○ |  | OFFBOARD | Flying | Yes | No | LAND | None | POSCTL | No | NEUTRAL | 14.82 | 8.65 | 1:39 | TRUE | FALSE | 0
63 | ● | ● | ◑ | OFFBOARD | Flying | Yes | No | LAND | None | OFFBOARD | No | Mid + | 0.05 | 12.58 | 1:14 | TRUE | FALSE
PX1 | ● | ● | ◑ | OFFBOARD | Flying | No | None | NONE | None | POSCTL | No | NEUTRAL | 30.70 | 3.38 | 1:32 | TRUE | FALSE
234 | ● | ○ |  | LAND | Land | No | None | NONE | None | OFFBOARD | No | Mid + | 0.26 | 12.77 | 1:26 | TRUE | FALSE
97 | ● | ● | ◑ | OFFBOARD | Flying | Yes | Yes | RETURN | None | STABILIZED | No | NEUTRAL | 24.83 | 6.21 | 1:28 | TRUE | FALSE | 1
199 | ● | ○ |  | OFFBOARD | Hover | No | None | NONE | None | LOITER | No | LOW + | 14.83 | 12.51 | 0:55 | TRUE | FALSE
200 | ● | ● | ◑ | OFFBOARD | Flying | No | None | NONE | None | ALTCTL | No | Mid - | 29.54 | 3.3 | 6:00 | TRUE | FALSE | 2
324 | ● | ○ |  | LAND | Arm | No | None | NONE | None | ALTCTL | No | NEUTRAL | 0.25 | 12.69 | 1:17 | TRUE | FALSE
352 | ● | ● | ◗ | OFFBOARD | Hover | No | None | NONE | None | LAND | No | LOW - | 0.21 | 12.68 | 1:20 | TRUE | TRUE
5 | ● | ● | ○ | OFFBOARD | Flying | Yes | No | HOLD | None | OFFBOARD | No | LOW + | 0.09 | 12.64 | 1:17 | TRUE | TRUE
84 | ● | ● | ○ | OFFBOARD | Flying | Yes | Yes | LAND | None | OFFBOARD | No | LOW - | 0.2 | 12.57 | 1:22 | TRUE | TRUE | 3
187 | ● | ● | ◗ | OFFBOARD | Flying | No | None | NONE | None | STABILIZED | No | LOW - | 9.69 | 2.88 | 4:16 | FALSE | FALSE | 4
204 | ● | ○ |  | OFFBOARD | Takeoff | No | None | NONE | None | STABILIZED | No | HIGH + | 34.94 | 93.55 | 3:05 | TRUE | FALSE
115 | ● | ● | ◑ | OFFBOARD | Flying | Yes | Yes | LAND | None | STABILIZED | No | LOW - | 25.24 | 4.51 | 0:19 | TRUE | FALSE | 5
122 | ● | ● | ◑ | OFFBOARD | Flying | Yes | Yes | WARN | None | STABILIZED | No | LOW - | 25.45 | 4.27 | 0:20 | TRUE | FALSE
139 | ● | ● | ◗ | OFFBOARD | Flying | Yes | Yes | HOLD | None | OFFBOARD | No | MED + | 0.21 | 12.72 | 1:26 | TRUE | FALSE
387 | ● | ○ |  | OFFBOARD | Hover | No | None | NONE | None | RTL | No | None | 14.85 | 29.82 | 1:25 | TRUE | FALSE | 6
W2 | ● | ● | ◗ | OFFBOARD | Flying | Yes | None | RETURN | 5 North | STABILIZED | No | High + | 9.62 | 68.12 | 2:44 | TRUE | FALSE | 7
KT2 | ● | ● | ◗ | OFFBOARD | Hover | No | None | NONE | None | RTL | Yes | NEUTRAL | 14.17 | 22.84 | 2:40 | TRUE | FALSE | 8
KT3 | ● | ● | ◗ | OFFBOARD | Flying | No | None | NONE | None | OFFBOARD | Yes | None | 29.15 | 3.11 | 1:40 | TRUE | FALSE

####  5.4.2. G1 Gateway:

All tests in the profile were clustered using Within-Cluster Sum of Squares
(WCSS). WCSS calculates the sum of squared distances of all points within a
cluster to the centroid of that cluster as
$\text{WCSS}=\sum_{i=1}^{k}\sum_{x\in C_{i}}||x-\mu_{i}||^{2}$. The optimal
number of clusters for larger test samples were selected heuristically using
the elbow method, whereas smaller test samples were placed in groups according
to their test type. This resulted in nine clusters. For each cluster, relative
z-scores were computed to represent the distance to the centroid. Tests closer
to the centroid are considered more representative of the tests in the cluster
than those further away. We therefore selected tests from above the 85th
percentile and below the 15th percentile. We apportioned an approximate down-
select size as 25-30 tests and first selected one or two abnormal tests
closest to each centroid, and then remaining tests from those closest to the
edges which were otherwise not well represented. We used the clusters merely
as a guide to look for interesting and abnormal test results for further
investigation to achieve as diverse a set of interesting test results to
explore as possible.

####  5.4.3. L2 Tests:

We executed all of the selected tests in the L2 simulation environment using a
FrSky XD9 Plus Taranis Radio Handheld Controller (FrySky, 2023). L2 tests were
run by two researchers from our team. The tester was responsible for the test
setup, including the test runner and the supporting mobile app. For testing
purposes, the tester also invoked each of the missions. The RPIC was
responsible for supervising sUAS flights and following instructions generated
by the test runner to apply the human interaction task at the correct stage of
the mission. For each executed test, we preserved the flight logs, uploaded
them into the PX4 evaluation platform (PX4, 2023b), and inspected the replayed
flight log, logged messages, and graphs extracted from flight log data to
further evaluate the flight outcomes. For each test, we assessed it as
“acceptable” or “problematic”, where an acceptable test completed the mission
as expected. This meant that it either flew the mission to completion, acted
correctly upon the prescribed geofence action if any were assigned, or killed
motors when commanded in a safe manner. Of the 31 tests passed to L2, 16 were
marked as problematic following execution and inspection, 13 were marked as
acceptable, and 2 were marked as untestable due to limitations in the L2
handling of the killswitch. It should be noted, however, that some differences
in timing and the exact placement of switches occurred between L1 and L2
tests, as was expected when a human was added into the loop. Therefore, while
most results were the same, some had slightly different outcomes. Figure 8
shows the basic flight (a) an actual flight path (b) and one of the graph
plots used to analyze the flight (c). In this case, the RPIC switched modes to
stabilized (as directed by the test runner) whilst the sUAS was flying in
offboard mode. Due to the current trajectory and momentum of the sUAS, the
sUAS continued its upward trajectory, ultimately reaching a height of 377
meters and a distance of over 550 meters. Ultimately, the TESTER issued a land
command to force an end to the mission.

(a) The basic flight path of the sUAS when GEOFENCE=INACTIVE.

(b) The actual flight path when Geofence=INACTIVE, and the RPIC executes a
Mode Change to STABILIZED with DIR-Toggle=”BACK”.

(c) By inspecting plots and log outputs we assess the outcome of the flight
and identify root cause of errors.

Figure 8. In this case the RPIC switched modes to stabilized whilst the sUAS
was flying in offboard mode. Due to the current trajectory and momentum of the
sUAS, it continued its upward trajectory, ultimately reaching a height of 377
meters and a distance of over 550 meters. Ultimately, the TESTER issued a land
command to force an end to the mission. To minimize human errors caused by
untimely mode-switches to stabilized, we can move the stabilized switch to a
less prominent position, and add monitors to recognize if the drone is in
’free flight’ due to a sudden switch to STABILIZE mode.

This figure depicts examples from our tests with the resulting analysis
performed in the px4 flight analysis platform. The shown analysis features
provided information and visualization to identify the root cause of the
problem for this case.

####  5.4.4. G2 Gateway:

To perform the G2 safety analysis, we leveraged our existing development and
testing process in which any observed flight problems are examined carefully
to identify safety hazards, build a safety case, and where necessary design
and implement mitigations. Once mitigations were implemented, we updated the
safety case, and when and if, deemed safe for flight, we placed the test into
the field-test backlog. The formality of the safety case was determined by the
severity of the risk, meaning that some safety cases were supported only by
discussions documented on GitHub. In contrast, more severe risks were modeled
as formal safety cases using the Goal Structuring Notation (GSN) (Kelly and
Weaver, 2004). In cases where we were uncertain that a problem observed in the
L2 simulation would also occur in the real world, we determined if it was safe
to execute a physical test in L3 supported by relevant safeguards. While
providing a full description of the safety assurance process is outside the
scope of this paper; we include one example in Figure 10. One author of this
paper performed a preliminary inspection of the problematic test results using
a bottom-up inductive approach to tag and classify the observed problems. They
identified 8 initial categories, merged two pairs together, and removed one
category that was related to the experimental fidelity of level L2 rather than
to the core design of D4A. The five remaining categories were presented,
discussed, modified, and ultimately agreed upon, at a design meeting attended
by two D4A architects (also authors of this paper) and three researchers (two
of whom were authors of this paper). We opened several issues in Github to
serve as placeholders of future work, discussed and documented initial design
solutions, and in one case over a period of several weeks, designed and
implemented a solution as discussed in Section 5.4.6.

####  5.4.5. Challenges and Candidate Design Solutions

We now describe the five identified design challenges [DC1-DC5], including
preliminary, designed, and/or implemented solutions.

Figure 9. Before the mitigation, RPICs needed to focus on fixing the position
of the throttle during the takeoff, drawing attention away from the sUAS as
the mission was launched. The change implemented as a result of HIFuzz fuzz-
tests automates this process, allowing RPICs to focus on their supervisory
roles.

  * [DC1] Incorrect Stick Positions:  We had previously observed that incorrect throttle positions were a contributing factors in several crashes and deviant flights that occurred when a human ceded control from the AOP. The problem was exacerbated by the default PX4 behavior requiring drones to arm with the throttle down, thereby requiring the RPIC to reposition the throttle to neutral during takeoff in preparation for any later emergency. This is illustrated for a multi-sUAS mission launch in Figure 9. When a single RPIC supervises the flights of multiple sUAS, the added workload of adjusting throttle positions during take-off created the opportunity for error. Our tests replicated these problems and showed that similar problems can occur if the Direction-Stick (Forward/Back/Left/Right) is placed in a non-neutral position; however, this is unlikely in the physical RC due to the stick automatically returning to NEUTRAL when not held in a different position. Incorrect stick positions were factors in four of the L2 tests (#5,#84, #126, #PX1), some of which are included in Table 4.
Mitigations:  We identified two contributing solutions. First, we reprogrammed
the takeoff routine to allow take-offs with the throttle in the neutral
position thereby eliminating an error-prone human task, and second we designed
new alerts to warn the RPIC if the throttle was placed or left in a non-
neutral position following takeoff.

  * [DC2] Insufficient Failsafes:  By design our HIFuzz study reported in this paper focused on geofence failsafes, and deliberately did not set other failsafes related to low-battery, or loss of RC or data signal etc.. However, the tests results showed two cases of fly-aways that occurred when the geofence was not active, in one case the sUAS flew so far away that it also lost signal. First, this highlights the need for multiple failsafe mechanisms to work in tandem, providing various layers of safety protection; however, it also introduces the challenge of consistently maintain failsafe configurations across divers sUAS. Human operators, such as RPICS, need to understand what sUAS are doing, and are likely to do under various circumstance, and therefore it is important for them to build a consistent and and reliable mental models of the sUAS behavior (Endsley, 2017). Further, with multiple sUAS at play simultaneously, it is easy for their failsafes configurations to become inconsistent with respect to activation and actions to be taken. We observed problems associated with insufficient failsafe mechanisms in four L2 test cases (#122, #187, #200, #220).
Mitigations:  Instead of holding the RPIC responsible for manually checking
that all failsafes have been set appropriately at the start of each mission,
automate the process as part of the prearming checks, push changes where
needed, and raise an alert if failsafes are not able to be set correctly.

  * [DC3] The OAP tracks only a subset of failsafe activations:  The fuzz tests revealed a previously unexplored problem that The OAP did not handle failsafe actions taken directly by the PX4 flight controller. In some cases it continued to run in competition with the failsafe (#KT3) churning out confusing mode-switch messages to the RPIC. In other cases it ceded control but did not clear up its message queue, meaning that later reactivation of The OAP resulted in the execution of stale trajectory actions and jerky flight, which are difficult for the RPIC to fully understand or act upon. This was observed in five additional tests (#84, #115, #139, #352, #392).
Mitigations:  We logged this as an issue with an initial design solution to
improve The OAP’s failsafe monitoring system and its ability to reset its
state during failsafe context switches.

  * [DC4] Unsafe RC Transmitter Mappings:  Our analysis of the test results highlighted problems with the RC Transmitter switches and their mappings to modes and behaviors. They were not designed in an ideal way to support the D4A system. For example, the RC transmitter provided a setting that allowed the RPIC to manually switch to offboard mode, even though this is mode is solely the responsibility of the OAP. Further, stabilized and posctrl were accessed by the same switch, one simple move away from each other. A simple mistake which pushed the switch one click further than intended could cause the sUAS to enter the wrong mode with rather severe consequences. For example the RPIC may be unaware that they are flying in stabilized mode rather than posctrl, requiring far more advanced skills and creating an incredibly challenging task if they take control when the sUAS is at distance and altitude. These problems were observed in two tests (#63, #200).
Mitigations:  Build a formal model of the RC transmitter to check for task
flow and edge cases. Redesign the mappings between switches and tasks
according to the D4A uses cases. For example, remove the ability for a user to
transition to offboard mode as this is always performed by The OAP
automatically.

  * [DC5] Mode changes without sufficient alerts:  Finally, there were several cases in which mode switches delivered surprising results due to feature interactions. For example switching to posctrl with throttle down, or stabilized when the sUAS had momentum, all caused deviant flights. This was observed in four tests (#87, #97, #187, #220).
Mitigations:  Improve D4A’s monitoring and alert system to increase user
situational awareness by providing warnings when known feature interactions
were detected that were likely to cause deviant flight patterns.

####  5.4.6. L3 Tests:

Only one of the identified mitigations has been implemented so far, with the
others logged as issues with candidate design solutions. The implemented
mitigation addresses DC1. We first constructed a basic safety case (cf. Figure
10) depicting two goals G2 and G3, contextualized by their associated
problematic fuzz tests. For Goal G2 “Drone establishes stable flight upon
human takeover”, we considered four design solutions; however, solutions S1
and S2 were considered inadequate because they limited the autonomy of RPICs
to make necessary flight decisions; while, solutions S3 and S4 were selected.
S4 modified the arming and takeoff routine to allow RPICs to place the
throttle in the neutral position prior to mission deployment. The new feature
was implemented and then moved rapidly to field-tests as no significant new
hazards were identified related to its test. The test passed, and the feature
is now part of our stable code branch. We have deferred re-execution of the
associated Fuzz Tests (i.e., #5, #126, #84, #PX1, #97, #187, #220) until both
goals G2 and G3 have been addressed.

Figure 10. A safety case is developed to address the paper’s motivation
Scenario #1 and to track progress towards designing and integrating a solution
to provide fault-tolerance against normal human mistakes.

##  6\. Analysis of Results

We now discuss the results from our experiment with respect to each of the two
research questions.

###  6.1. RQ1: What kinds of system failures, associated with human
interactions were identified during the HIFuzz process?

Our experiments were focused around two specific scenarios related to mode-
switches surrounding geofence breaches and human-takeover of the sUAS. Despite
the narrow focus of these initial fuzz tests, we identified five types of
failures that created challenges for human operators. Our experiments focused
on the RPIC who holds primary responsibility for supervising sUAS flights. The
five problems covered areas associated with situational awareness,
transparency, and coordination – all of which are particularly important in
any human-machine teaming environment (Konaev and Chahal, 2021), such as the
D4A ecosystem. DC5 identified problems of lost situational awareness when
flight modes changed due to human actions or automated failsafes. This
included cases in which a human mistakenly flips the wrong switch on the RC
Transmitter without realizing what they’ve done. The tests showed that this
could lead to significant flight deviations and other failures without proper
alerts to the RPIC. A second, particularly egregious problem related to
incorrect positioning of throttle and directional controls. Given the
importance of this problem, and its role in previous field-accidents, DC1
focused on allowing the throttle to be placed in the neutral position during
launch, thereby reducing the intense workload that an RPIC previously faced in
moving throttle switches to neutral during a multi-sUAS launch. The
implemented mitigation effectively reduces a non-trivial stress point, making
it far easier for the RPIC to position the throttle correctly prior to flight,
and warning them when an oversight or mistake occurs.

DC2 addresses the need for consistent and complete failsafes to be set, whilst
acknowledging that it is difficult for humans to consistently manage this in a
multi-sUAS fleet. The proposed solution automates the task of checking and
setting failsafes during a pre-arming state. DC3 addresses a related problem
in which the OAP fails to properly track and handle the full spectrum of
automated and human-initiated failsafe actions as they occur during the
flight. This results in potentially confusing flight behavior, such as
thrashing between states as observed in one of our killswitch tests where the
OAP continually fought against the kill motors action. This mitigation will
improve the scope and consistency in how failsafes are managed.

Finally, DC5 identified flaws in the RC Transmitter mappings which gave humans
access to modes that were intended only for automated control by the OAP. It
also identified the need to reorganize switches in a way that would make it
much harder for humans to accidentally transition to incorrect modes.This
problem is being mitigated by remapping the Taranis RC transmitter to better
meet the needs of humans interacting with D4A.

To answer RQ1 we have identified five core problems, all of which created
potential human-interaction hazards in the HIFuzz tests, and which could be
successfully reduced in severity through the proposed mitigation strategies.

###  6.2. RQ2: Did each of the three test levels and the two gateways play a
unique role in the human interaction testing process?

We address this RQ by summarizing the contributions from each test level and
gateway. Test level L1 enabled us to run thousands of tests in a low-risk
environment. Gateway G1 gave us a manageable and guided approach for
identifying where human-interaction problems lurked, and carefully selecting
tests for the human-in-the-loop simulations. Adding the human-in-the-loop at
levels L2 and L3 provided several interesting results. For example, we found
that the timings of events, even with the support of the test runner, were not
quite identical, meaning that some outcomes looked different in terms of
flight paths, but were ultimately examples of the same feature interactions at
play. We also observed the human making occasional mistakes, which provided
additional and unexpected results human actions. The primary example was when
the RPIC running L2 tests accidentally changed the position of the direction
stick even though that was not a planned activity. Originally we planned to
discard the results, but the ‘mistake’ unearthed new deviant flight behaviors
that contributed to our overall findings. In the spirit of Fuzz Testing, we
retained these results. G3 allowed us to take a deep dive into analyzing the
safety concerns associated with executing tests in the field and then
determining appropriate mitigations.The G2 gateway in conjunction with L3 not
only guided the process of design improvement, but provided a safety net that
helped us ensure that tests could be executed safely, and that design
decisions would produce a safer product in which there is greater tolerance to
“normal” human errors. The interplay of each level against a subset of test
cases is depicted in the left hand columns of Table 4.

To answer RQ2, we found that moving across the levels from L1 to L2 and
ultimately to L3, provided far more value than simply filtering down the
number of tests. New and slightly different issues appeared as the fidelity of
the simulation environment increased. While no type of simulation can match
the fidelity of the real world and actual field tests, the simulation levels
provided significant insights across a broad range of tests, while allowing us
to explore potential human interaction errors in a safe environment, while L3
empowered us to carefully assess safety concerns and deploy real-world tests
in a systematic and safe way.

##  7\. Related Work

As our work and the presented HIFuzz framework bridges the gap between testing
and human-CPS-interaction, we discuss relevant related work on _CPS and sUAS
testing_ in general (Abbaspour Asadollah et al., 2015), and _fuzz testing_
(Zhu et al., 2022) in particular, _test generation_ , as well as _Human-sUAS
interaction_ and _safety assurance_. We have already provided references to
many of these topics; we present the most closely related work here in more
detail.

$\bullet$ CPS Testing: Several approaches in the past have used fuzzing in the
context of CPS and sUAS applications. For example, Kim  _et al._ (Kim et al.,
2019) have proposed RVFuzzer, a testing solution for detecting input
validation bugs in robotic vehicle control programs. Similar to our work, they
focus on PX4 and ArduPilot sUAS applications, however, their main aim is to
detect low-level controller malfunctions by monitoring vehicle control states.
Similarly, Kim  _et al._ (Kim et al., 2021) have created PGFUZZ, a policy-
based fuzzing framework for robotic vehicles. Again focusing on sUAS
applications and three flight controllers (Ardupilot, PX4, and Paparazzi).
PGFUZZ focuses on safety and functional policies wrt. to user inputs,
configuration parameters, and physical sUAS states. While their work
explicitly includes user input and user commands, with HIFuzz, we have created
a comprehensive multi-level testing framework that utilizes fuzzing, combined
with a thorough safety analysis process.

Han  _et al._ (Han et al., 2022) proposed a grey-box-based fuzzing framework
for detecting incorrect configurations in sUAS flight controllers. Their
framework, LGDFuzzer, combined fuzzing with a genetic algorithm to detect
potentially incorrect configurations, and to subsequently test them through
simulation. While their approach provided a fine-grained analysis of flight
states and parameters, they did not take into account human-related actions or
real-world physical testing. Sun et al. (Sun et al., 2023) used fuzzing to
find violations of traffic law in autonomous vehicles. They built a
specification language to represent traffic laws, and then used fuzzing to
find violations. Their focus is on breaking the specification using the
vehicle, rather than on human actions.

$\bullet$ Human-Drone Interaction Herdel  _et al._ (Herdel et al., 2022)
conducted a comprehensive study focusing on applications with a human-drone
interaction aspect. As part of their work, they identified over 100
applications in 16 different domains such as emergency response and
surveillance, and derived related challenges and research opportunities. One
of the research challenges is directly related to our work, addressing
different types of how people interact with sUAS and perform complex tasks.
With our HIFuzz platform, we strive to alleviate complex and potentially
dangerous situations that need to be tested, by systematically defining and
testing different scenarios with complex tasks, and user interactions, both in
a safe environment and with real human input.

Rakotonarivo  _et al._ (Rakotonarivo et al., 2023) conducted a series of
interviews with drone operators, safety consultants, and regulators to
identify operational risks and challenges when operating sUAS. With HIFuzz we
provide support for one of the key recommendations: “Support exploration of
operational parameters and estimate their impact on mission safety”, so that
we can support “operators to explore options that could simplify their
procedures”. Our 3-level testing process is specifically built to identify and
mitigate potential safety issues before they arise in field testing, or worst-
case, during live mission execution. Moreover, our model allows us to
systematically capture and test diverse mission parameters and tasks and
generate respective reports and documentation for passed and/or failed tests,
that can further serve as input for thorough safety analysis and building, for
example, Safety Assurance Cases.

$\bullet$ Safety Analysis: A previous study by Vierhauser  _et al._
(Vierhauser et al., 2021) reviewed over 100 publications in order to identify
hazardous human interaction errors. They ascribed the errors to three main
categories: (1) _human-initiated_ (i.e., for example, an RPIC behaving in a
reckless manner), (2) _loss of situational awareness_ (e.g., the system not
providing sufficient information through the user interface to rectify/prevent
an accident from occurring), and (3) a _lack of empowerment_ (e.g., interfaces
limiting the actions of the human, preventing them to intervene when needed).
Similarly, Belcastro  _et al._ (Belcastro et al., 2017) identified sUAS
hazards based on analyzing 100 mishaps ranging from minor incidents up to
fatal accidents. Their analysis details mission types, sUAS types, and their
respective configuration. Roseman and Argrow (Roseman and Argrow, 2020) put a
particular focus on hazards related to adverse weather conditions. As part of
their work, they created a framework for analyzing weather forecast data to
provide sUAS operators with risk assessment information and discuss potential
effects and outcomes of weather hazards.

Several of the documented hazards could potentially have been detected by
HIFuzz using our multi-step simulation and field testing approach.

##  8\. Limitations and Future Work

The research described in this paper is empirical in nature and is subject to
three primary threats to validity.

First and foremost, we developed the HIFuzz framework for fuzz testing human
interactions in multi-sUAS systems; however, our experiments focused upon our
own D4A as the single SuT. We have designed and implemented HIFuzz as a
generic framework for use with diverse sUAS systems and flight controllers;
however, parts of the _proxy human agent_ are specific to the PX4-based
controller as they bypass the SuT and send commands directly to the
controller. Versions for other common flight controllers (e.g., Ardupilot
(ArduPilot, 2019)) could also be integrated. Nevertheless, we performed
conceptual walkthroughs of how HIFuzz could be applied to a broader set of
robotics systems, and also how individual tests could be constructed on
multiple collaborating sUAS. Starting with tests on individual sUAS allowed us
to fine-tune the overall HIFuzz process; In future work, we plan on broadening
the scope of HIFuzz and applying it to a more diverse set of systems including
other CPS and larger sets of sUAS.

Second, one of the challenges of human interaction fuzz testing is supporting
users in performing the right task at the right time during a test. The
cognitive load of piloting or supervising an sUAS is quite high, and the
testing task increases that load even further. We addressed this challenge by
building the mobile app teleprompter system, which delivered clear and timely
prompts to the users; however, they found the increased workload rather
challenging. In future work, we will conduct a user study specifically
dedicated to exploring alternative options, and potential improvements, such
as adding an additional testing role (i.e., a COACH) responsible for
supporting the RPIC during the testing process. As our tests were limited to
the RPIC, which is potentially the most challenging role, we will also extend
the study to include other roles such as the MC (Mission Commander) and SO
(Safety Officer), assign a more extensive set of human-interaction tasks, and
study the perception of our stakeholders to identify further points of
perceived weaknesses. In addition, we plan to allow humans to interact more
freely with the L2 simulation environment, and deal with a far broader set of
emergency tasks including deviant flight behaviors. Their success at
intervening could serve as an indicator of the robustness of the design with
respect to human interactions.

Finally, as previously mentioned, the L2 level, while fully functional, had
less fidelity to the field than we had intended, primarily because libraries
used to interface the radio signals with software-based PX4 simulations had
some limitations. In future work, we plan to augment, or ultimately entirely
replace the L2 layer with a Hardware-In-The-Loop layer in which a physical
flight controller is integrated closely into the simulated environment. This
would further increase test fidelity and allow the RC transmitter to
communicate over radio signals directly with the PX4 controller. This would
also address one other limitation; while the HIFuzz framework was implemented
as a generic framework for use with diverse sUAS systems and flight
controllers, parts of the _proxy human agent_ are specific to the PX4-based
controller. Replacing the simulated PX4 engine with a hardware component would
remove the need for this custom component. Overall, increasing fidelity would
allow more robust human-interaction testing, and improve the overall fidelity
of our HIFuzz pipeline.

##  9\. Conclusions

In this paper, we have presented the HIFuzz framework for supporting human-
interaction tests progressing across three test levels and two gateways. This
facilitates transitioning from large-scale, low-fidelity tests in simulation,
to high-fidelity, high-stakes, real-world tests. Our focus has been upon the
systematic application of each part of the HIFuzz process, to identify human-
interaction hazards that guide us in the design, implementation, deployment,
and testing of solutions intended to increase the fault-tolerance of the
system for human errors. Our work is motivated by our own experiences in
operating multi-sUAS flights and our observations that many flight problems
could have been averted with additional support and improved design. In
conclusion, HIFuzz effectively identified hazardous human interactions, helped
us to identify actionable solutions, provided a safe pathway for deploying
those solutions, and where full mitigations were not feasible it provided us
with far deeper understanding of the strengths and weaknesses of our system.

##  10\. Acknowledgments

The work described in this paper was partially funded by a grant from the USA
National Aeronautics and Space Administration (NASA # 80NSSC21M0185). Work
performed by Michael Vierhauser was supported by a grant from the Linz
Institute of Technology (LIT-2019-7-INC-316).

## References

  * (1)
  * Abbaspour Asadollah et al. (2015) Sara Abbaspour Asadollah, Rafia Inam, and Hans Hansson. 2015.  A survey on testing for cyber physical system. In _Testing Software and Systems: 27th IFIP WG 6.1 International Conference, ICTSS 2015, Sharjah and Dubai, United Arab Emirates, November 23-25, 2015, Proceedings 27_. Springer, 194–207.
  * ANONYMOUS (2023) ANONYMOUS. 2010-2023.  We removed several references to the System under Test (SuT) to preserve the integrity of the double blind review process. Using our best judgment we felt that listing the papers, even in 3rd person format, would reveal our identity. We have provided all necessary details about the system directly in the paper.
  * ArduPilot (2019) ArduPilot. 2019.  http://ardupilot.org.  [Online; accessed 01-September-2019].
  * Ardupilot (2023) Ardupilot. 2023.  Flight Controller Modes.  https://ardupilot.org/plane/docs/flight-modes.html.  [Online; Accessed 01-07-2023].
  * Belcastro et al. (2017) Christine M Belcastro, Richard L Newman, Joni Evans, David H Klyde, Lawrence C Barr, and Ersin Ancel. 2017.  Hazards identification and analysis for unmanned aircraft system operations. In _17th AIAA Aviation Technology, Integration, and Operations Conference_. 3269.
  * Bendrissou et al. (2023) Bachir Bendrissou, Cristian Cadar, and Alastair F. Donaldson. 2023.  Grammar Mutation for Testing Input Parsers (Registered Report). In _Proceedings of the 2nd International Fuzzing Workshop_ (Seattle, WA, USA) _(FUZZING 2023)_. Association for Computing Machinery, New York, NY, USA, 3–11.  https://doi.org/10.1145/3605157.3605170
  * Böhme et al. (2017) Marcel Böhme, Van-Thuan Pham, Manh-Dung Nguyen, and Abhik Roychoudhury. 2017.  Directed Greybox Fuzzing. In _Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security_ (Dallas, Texas, USA) _(CCS ’17)_. Association for Computing Machinery, New York, NY, USA, 2329–2344.  https://doi.org/10.1145/3133956.3134020
  * Böhme et al. (2016) Marcel Böhme, Van-Thuan Pham, and Abhik Roychoudhury. 2016.  Coverage-Based Greybox Fuzzing as Markov Chain. In _Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security_ (Vienna, Austria) _(CCS ’16)_. Association for Computing Machinery, New York, NY, USA, 1032–1043.  https://doi.org/10.1145/2976749.2978428
  * Chen et al. (2021) Linfeng Chen, Kazuki Takashima, Kazuyuki Fujita, and Yoshifumi Kitamura. 2021.  Pinpointfly: An egocentric position-control drone interface using mobile ar. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–13.
  * Chen et al. (2020) Yuqi Chen, Bohan Xuan, Christopher M. Poskitt, Jun Sun, and Fan Zhang. 2020.  Active Fuzzing for Testing and Securing Cyber-Physical Systems. In _Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis_ (Virtual Event, USA) _(ISSTA 2020)_. Association for Computing Machinery, New York, NY, USA, 14–26.  https://doi.org/10.1145/3395363.3397376
  * Coyne (2005) Richard Coyne. 2005.  Wicked problems revisited.  _Design Studies_ 26, 1 (2005), 5 – 17.  https://doi.org/10.1016/j.destud.2004.06.005
  * DeVries and Cheng (2018) Byron DeVries and Betty HC Cheng. 2018.  Run-time monitoring of self-adaptive systems to detect n-way feature interactions and their causes. In _Proceedings of the 13th International Conference on Software Engineering for Adaptive and Self-Managing Systems_. 94–100.
  * DroneCode ([n. d.]) DroneCode. [n. d.].  MAVLink - Developer Guide.  https://mavlink.io/en.  [Last accessed 01-11-2021].
  * Endsley (2011) Mica R. Endsley. 2011.  _Designing for Situation Awareness: An Approach to User-Centered Design, Second Edition_ (2nd ed.).  CRC Press, Inc., Boca Raton, FL, USA.
  * Endsley (2017) Mica R Endsley. 2017.  Autonomous driving systems: A preliminary naturalistic study of the Tesla Model S.  _Journal of Cognitive Engineering and Decision Making_ 11, 3 (2017), 225–238.
  * Ermakov ([n. d.]) Vladimir Ermakov. [n. d.].  MAVROS.  https://github.com/mavlink/mavros
  * Fan et al. (2018) Chin-Feng Fan, Ching-Chieh Chan, Hsiang-Yu Yu, and Swu Yih. 2018.  A simulation platform for human-machine interaction safety analysis of cyber-physical systems.  _International journal of industrial ergonomics_ 68 (2018), 89–100.
  * Fioraldi et al. (2023) Andrea Fioraldi, Alessandro Mantovani, Dominik Maier, and Davide Balzarotti. 2023.  Dissecting American Fuzzy Lop: A FuzzBench Evaluation.  _ACM Trans. Softw. Eng. Methodol._ 32, 2, Article 52 (mar 2023), 26 pages.  https://doi.org/10.1145/3580596
  * FrySky (2023) FrySky. 2023.  Taranis Series Handheld RC.  https://www.frsky-rc.com/product-category/transmitters/taranis-series.  [Online: accessed 8-14-2023].
  * Fu et al. (2017) Zhicheng Fu, Chunhui Guo, Shangping Ren, Yizong Ou, and Lui Sha. 2017.  Modeling and integrating human interaction assumptions in medical cyber-physical system design. In _2017 IEEE 30th International Symposium on Computer-Based Medical Systems (CBMS)_. IEEE, 373–378.
  * Godefroid et al. (2008) Patrice Godefroid, Adam Kiezun, and Michael Y. Levin. 2008.  Grammar-Based Whitebox Fuzzing.  _SIGPLAN Not._ 43, 6 (jun 2008), 206–215.  https://doi.org/10.1145/1379022.1375607
  * Han and Zhou (2020) Jia Cheng Han and Zhi Quan Zhou. 2020.  Metamorphic Fuzz Testing of Autonomous Vehicles. In _Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops_ (Seoul, Republic of Korea) _(ICSEW’20)_. Association for Computing Machinery, New York, NY, USA, 380–385.  https://doi.org/10.1145/3387940.3392252
  * Han et al. (2022) Ruidong Han, Chao Yang, Siqi Ma, JiangFeng Ma, Cong Sun, Juanru Li, and Elisa Bertino. 2022.  Control parameters considered harmful: Detecting range specification bugs in drone configuration modules via learning-guided search. In _Proceedings of the 44th International Conference on Software Engineering_. 462–473.
  * Hao et al. (2022) Chenxu Hao, Anany Dwivedi, and Philipp Beckerle. 2022.  A Literature-Based Perspective on Human-Centered Design and Evaluation of Interfaces for Virtual Reality in Robotics. In _Human-Friendly Robotics 2022 - HFR: 15th International Workshop on Human-Friendly Robotics, Delft, The Netherlands, 22-23 September 2022_ _(Springer Proceedings in Advanced Robotics, Vol. 26)_ , Pablo Borja, Cosimo Della Santina, Luka Peternel, and Elena Torta (Eds.). Springer, 1–13.  https://doi.org/10.1007/978-3-031-22731-8_1
  * Herdel et al. (2022) Viviane Herdel, Lee J Yamin, and Jessica R Cauchard. 2022.  Above and beyond: A scoping review of domains and applications for human-drone interaction. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_. 1–22.
  * Hoang et al. (2023) Maria-Theresa Oanh Hoang, Niels van Berkel, Mikael B Skov, and Timothy R Merritt. 2023.  Challenges and Requirements in Multi-Drone Interfaces. In _Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–9.
  * Hu et al. (2021) Zhisheng Hu, Shengjian Guo, Zhenyu Zhong, and Kang Li. 2021.  Coverage-based scene fuzzing for virtual autonomous driving testing.  _arXiv preprint arXiv:2106.00873_ (2021).
  * Kelly and Weaver (2004) Tim Kelly and Rob Weaver. 2004.  The Goal Structuring Notation – A Safety Argument Notation. In _Proc. Dependable Syst. Networks 2004 Work. Assur. Cases_.
  * Khan and Neustaedter (2019) Md Nafiz Hasan Khan and Carman Neustaedter. 2019.  An exploratory study of the use of drones for assisting firefighters during emergency situations. In _Proceedings of the 2019 CHI conference on human factors in computing systems_. 1–14.
  * Kim et al. (2021) Hyungsub Kim, Muslum Ozgur Ozmen, Antonio Bianchi, Z Berkay Celik, and Dongyan Xu. 2021.  PGFUZZ: Policy-Guided Fuzzing for Robotic Vehicles.. In _NDSS_.
  * Kim et al. (2022) Seulbae Kim, Major Liu, Junghwan” John” Rhee, Yuseok Jeon, Yonghwi Kwon, and Chung Hwan Kim. 2022.  Drivefuzz: Discovering autonomous driving bugs through driving quality-guided fuzzing. In _Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security_. 1753–1767.
  * Kim et al. (2019) Taegyu Kim, Chung Hwan Kim, Junghwan Rhee, Fan Fei, Zhan Tu, Gregory Walkup, Xiangyu Zhang, Xinyan Deng, and Dongyan Xu. 2019.  $\\{$RVFuzzer$\\}$: Finding input validation bugs in robotic vehicles through $\\{$Control-Guided$\\}$ testing. In _28th USENIX Security Symposium (USENIX Security 19)_. 425–442.
  * Kohn et al. (1999) L.T. Kohn, J.M. Corrigan, and M.s. Donaldson. 1999.  To err is human, Building a safety health system.  _Washington, DC: National Academy Press_ (1999).
  * Konaev and Chahal (2021) Margarita Konaev and Husanjot Chahal. 2021.  Building trust in human-machine teams.  https://www.brookings.edu/techstream/building-trust-in-human-machine-teams.  [Last accessed 01-01-2022].
  * Leveson and Harvey (1983) Nancy G Leveson and Peter R Harvey. 1983.  Software fault tree analysis.  _Journal of Systems and Software_ 3, 2 (1983), 173–181.
  * Liang et al. (2014) Chieh-Jan Mike Liang, Nicholas D Lane, Niels Brouwers, Li Zhang, Börje F Karlsson, Hao Liu, Yan Liu, Jun Tang, Xiang Shan, Ranveer Chandra, et al. 2014\.  Caiipa: Automated large-scale mobile app testing through contextual fuzzing. In _Proceedings of the 20th annual international conference on Mobile computing and networking_. 519–530.
  * Luckeneder et al. (2017) Christoph Luckeneder, Michael Rathmair, and Hermann Kaindl. 2017.  Investigating and coordinating safety-critical feature interactions in automotive systems using simulation.  (2017).
  * Mallan et al. (2017) Vasudev S Mallan, Syam Gopi, Alexander Muir, and Rao R Bhavani. 2017.  Comparative empirical usability assessment of two HRI input devices for a mobile robot. In _2017 4th International Conference on Signal Processing, Computing and Control (ISPCC)_. IEEE, 331–337.
  * Merkel (2014) Dirk Merkel. 2014.  Docker: Lightweight Linux Containers for Consistent Development and Deployment.  _Linux J._ 2014, 239, Article 2 (mar 2014).
  * Miller et al. (1990) Barton P. Miller, Lars Fredriksen, and Bryan So. 1990.  An Empirical Study of the Reliability of UNIX Utilities.  _Commun. ACM_ 33, 12 (dec 1990), 32–44.  https://doi.org/10.1145/96267.96279
  * MQTT (2023) MQTT. 2023.  The Standard for IoT Messaging.  https://mqtt.org.  [Online: accessed 8-14-2023].
  * Muccini and Moghaddam (2018) Henry Muccini and Mahyar Tourchi Moghaddam. 2018.  IOT Architectural Styles. In _Proc. of 2018 European Conference on Software Architecture_. Springer, 68–85.
  * Nagel (1998) D.C. Nagel. 1998.  Human error in aviation Operations.  _Human factors in Aviation, E.L.Weiner and E.C.Nagel (Eds)_ 19890047069, 34 (1998), 263–303.  https://doi.org/10.1109/2.910904
  * Neto et al. (2010) Pedro Neto, J Norberto Pires, and A Paulo Moreira. 2010.  High-level programming and control for industrial robotics: using a hand-held accelerometer-based input device for gesture and posture recognition.  _Industrial Robot: An International Journal_ 37, 2 (2010), 137–147.
  * Norman and Draper (1986) Donald A. Norman and Stephen W. Draper (Eds.). 1986.  _User centered system design: New perspectives on human-computer interaction_.  Lawrence Erlbaum Associates, Hillsdale, NJ.
  * Olsthoorn et al. (2021) Mitchell Olsthoorn, Arie van Deursen, and Annibale Panichella. 2021.  Generating Highly-Structured Input Data by Combining Search-Based Testing and Grammar-Based Fuzzing. In _Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering_ (Virtual Event, Australia) _(ASE ’20)_. Association for Computing Machinery, New York, NY, USA, 1224–1228.  https://doi.org/10.1145/3324884.3418930
  * Open Robotics (2023) Open Robotics. 2023.  Gazebo.  https://gazebosim.org.  [Online: accessed 8-14-2023].
  * Padhye et al. (2019) Rohan Padhye, Caroline Lemieux, and Koushik Sen. 2019.  JQF: Coverage-Guided Property-Based Testing in Java. In _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis_ (Beijing, China) _(ISSTA 2019)_. Association for Computing Machinery, New York, NY, USA, 398–401.  https://doi.org/10.1145/3293882.3339002
  * PX4 (2022) PX4. 2022.  jMAVSim.  https://docs.px4.io/master/en/simulation/jmavsim.html.  [Last accessed 01-01-2022].
  * PX4 (2023a) PX4. 2023a.  Flight Controller Modes.  https://docs.px4.io/main/en/flight_modes.  [Online; Accessed 01-07-2023].
  * PX4 (2023b) PX4. 2023b.  Flight Review Platform.  https://logs.px4.io/.  [Online: accessed 8-14-2023].
  * PX4 (2023c) PX4. 2023c.  PX4 Parameter Reference.  https://docs.px4.io/main/en/advanced_config/parameter_reference.html.  [Online: accessed 8-14-2023].
  * QGroundControl (2021) QGroundControl. 2021.  Ground Control Station.  http://qgroundcontrol.com.  [Last accessed 01-11-2021].
  * Rakotonarivo et al. (2023) Balita Heriniaina Rakotonarivo, Nicolas Drougard, Stéphane Conversy, and Jérémie Garcia. 2023.  Cleared for Safe Take-off? Improving the Usability of Mission Preparation to Mitigate the Safety Risks of Drone Operations. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–17.
  * Reifer (1979) Donald J. Reifer. 1979.  Software Failure Modes and Effects Analysis.  _IEEE Trans. Reliability_ R-28,3 (1979), 247–249.
  * Rittel and Webber (1973) Horst W. J. Rittel and Melvin M. Webber. 1973.  Dilemmas in a general theory of planning. In _Policy Sciences_ , Vol. 4. 155–169.
  * Roseman and Argrow (2020) Christopher A Roseman and Brian M Argrow. 2020.  Weather hazard risk quantification for sUAS safety risk management.  _Journal of Atmospheric and Oceanic Technology_ 37, 7 (2020), 1251–1268.
  * Shah et al. (2018) Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. 2018.  Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In _Field and Service Robotics_. Springer, 621–635.
  * Srivastava and Payer (2021) Prashast Srivastava and Mathias Payer. 2021.  Gramatron: Effective Grammar-Aware Fuzzing. In _Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis_ (Virtual, Denmark) _(ISSTA 2021)_. Association for Computing Machinery, New York, NY, USA, 244–256.  https://doi.org/10.1145/3460319.3464814
  * Sullivan et al. (1999) Kevin J Sullivan, Joanne Bechta Dugan, and David Coppit. 1999.  The Galileo fault tree analysis tool. In _Digest of Papers. Twenty-Ninth Annual International Symposium on Fault-Tolerant Computing (Cat. No. 99CB36352)_. IEEE, 232–235.
  * Sun et al. (2023) Yang Sun, Christopher M. Poskitt, Jun Sun, Yuqi Chen, and Zijiang Yang. 2023.  LawBreaker: An Approach for Specifying Traffic Laws and Fuzzing Autonomous Vehicles. In _Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering_ (Rochester, MI, USA) _(ASE ’22)_. Association for Computing Machinery, New York, NY, USA, Article 62, 12 pages.  https://doi.org/10.1145/3551349.3556897
  * Syakur et al. (2018) MA Syakur, BK Khotimah, EMS Rochman, and Budi Dwi Satoto. 2018.  Integration k-means clustering method and elbow method for identification of the best customer profile cluster. In _IOP conference series: materials science and engineering_ , Vol. 336. IOP Publishing, 012017.
  * Vierhauser et al. (2021) Michael Vierhauser, Md Nafee Al Islam, Ankit Agrawal, Jane Cleland-Huang, and James Mason. 2021.  Hazard analysis for human-on-the-loop interactions in sUAS systems. In _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 8–19.
  * Yaacoub et al. (2020) Jean-Paul A Yaacoub, Ola Salman, Hassan N Noura, Nesrine Kaaniche, Ali Chehab, and Mohamad Malli. 2020.  Cyber-physical systems security: Limitations, issues and future trends.  _Microprocessors and microsystems_ 77 (2020), 103201.
  * Yu et al. (2014) Mingxin Yu, Yingzi Lin, David Schmidt, Xiangzhou Wang, and Yu Wang. 2014.  Human-robot interaction based on gaze gestures for the drone teleoperation.  _Journal of Eye Movement Research_ 7, 4 (2014), 1–14.
  * Zhu et al. (2022) Xiaogang Zhu, Sheng Wen, Seyit Camtepe, and Yang Xiang. 2022.  Fuzzing: A Survey for Roadmap.  _ACM Comput. Surv._ 54, 11s, Article 230 (sep 2022), 36 pages.  https://doi.org/10.1145/3512345