[1]Hassan Sartaj

1]Department of Computer Science, National University of Computer and Emerging
Sciences, A.K. Brohi Rd., Islamabad, Pakistan 2]Quest Lab, I8-Markaz,
Islamabad, Pakistan

# Automated System-level Testing of Unmanned Aerial Systems

<EMAIL_ADDRESS>Asmar Muqeet<EMAIL_ADDRESS>Muhammad Zohaib
Iqbal<EMAIL_ADDRESS>Muhammad Uzair Khan<EMAIL_ADDRESS>[  [

###### Abstract

Unmanned aerial systems (UAS) rely on various avionics systems that are
safety-critical and mission-critical. A major requirement of international
safety standards is to perform rigorous system-level testing of avionics
software systems. The current industrial practice is to manually create test
scenarios, manually/automatically execute these scenarios using simulators,
and manually evaluate outcomes. The test scenarios typically consist of
setting certain flight or environment conditions and testing the system under
test in these settings. The state-of-the-art approaches for this purpose also
require manual test scenario development and evaluation. In this paper, we
propose a novel approach to automate the system-level testing of the UAS. The
proposed approach (AITester) utilizes model-based testing and artificial
intelligence (AI) techniques to automatically generate, execute, and evaluate
various test scenarios. The test scenarios are generated on the fly, i.e.,
during test execution based on the environmental context at runtime. The
approach is supported by a toolset. We empirically evaluate the proposed
approach on two core components of UAS, an autopilot system of an unmanned
aerial vehicle (UAV) and cockpit display systems (CDS) of the ground control
station (GCS). The results show that the AITester effectively generates test
scenarios causing deviations from the expected behavior of the UAV autopilot
and reveals potential flaws in the GCS-CDS.

###### keywords:

Model-based Testing, Artificial Intelligence, Deep Reinforcement Learning,
Unmanned Aerial Systems, Testing Automation

##  1 Introduction

Unmanned aerial systems (UAS) mainly consist of an unmanned aerial vehicle
(UAV) and a ground control station (GCS). A UAV is an aircraft that does not
have a human pilot on-board. It is either operated remotely from the ground
control station or autonomously using the autopilot system. UAS are becoming
progressively important due to their large number of applications in a variety
of domains. For example, UAS are used for rescue operations, disaster
management, geological surveys, agriculture and farming, weather forecasting,
wildlife monitoring, entertainment, surveillance, airstrikes, military combat,
and explosive material detection. UAS rely on several avionics software
systems (e.g., an autopilot, navigation system, communication system, etc.)
that are critical for completing the mission or are critical to safety.
Malfunctioning of the UAS avionics software can lead to mission failure that
can result in human life and financial loss. To ensure the dependability of
such systems, international safety standards such as DO-178C [1] and others
[2] define several guidelines. The automated testing of avionics systems is
essentially due to the huge amount of cost and effort required to follow these
guidelines.

UAS are a special type of cyber-physical systems (CPS) and are also considered
context-aware systems. The system-level testing of UAS requires the simulation
of the UAV flight behavior with the help of flight simulators. It is typically
performed at three levels, (i) software-in-the-loop (SIL) testing, (ii)
hardware-in-the-loop (HIL) testing, and (iii) pilot-in-the-loop (PIL) testing
also known as field testing. In SIL testing, the actual avionics software runs
while simulating the flight hardware and environment. In HIL testing, the
original avionics hardware parts are used and the environment is simulated. In
PIL testing, the UAS is tested in a real environment. Testing at each level
highly depends on the environmental context, i.e., test scenarios created for
SIL testing may not be useful for HIL/PIL testing. The main challenge in
automating system-level testing of the UAS is the generation and evaluation of
environmental context-aware test scenarios.

A traditional method of testing UAS at the system level includes a human
tester (pilot) who knows UAV flight behavior and has expertise with UAV flight
simulators. The tester is provided with the system under test (SUT) and the
specifications of the SUT. Using this information, the tester manually designs
test scenarios. For each test scenario, the tester simulates the behavior of a
UAV in a particular environment with the help of a flight simulator, and at
each flight operation, observes the behavior of the SUT by comparing the
expected behavior mentioned in the specifications. Testing in this way usually
requires several days and only a limited set of potential scenarios can be
tested. The existing approaches available in the literature also require
manual test scenario development and manual test evaluation (e.g., [3, 4, 5,
6]). The activity of manual testing is tedious and error-prone due to a large
number of potential scenarios that need to be tested in the extremely dynamic
environment of the UAV.

In this paper, we propose a novel approach to automate the system-level
testing of UAS. To automate the manual testing of UAS, a human-level control
on a flight simulator and a mechanism to model the SUT specifications are
required. Therefore, our approach (AITester) utilizes artificial intelligence
(AI) and model-based testing (MBT) techniques to automatically generate,
execute, and evaluate various test scenarios. We use deep reinforcement
learning (an AI technique) to get human-level control on a flight simulator
and MBT to model the specifications of SUT. The test scenarios are generated
on the fly, i.e., during test execution based on the environmental context at
runtime. For this purpose, we propose a UML profile for modeling the
structural aspects of UAS SUT and a UML profile for modeling the flight
behavior of the UAV. The structural profile is used to model the domain
concepts of the UAS SUT and the flight behavior profile is used to model the
abstract flight behavior of the UAV. Our approach requires the constraints
(written in Object Constraint Language (OCL) [7]) specified on the UAS SUT
domain model and state invariants on the flight behavior model. The UAS SUT
domain model, UAV flight behavior model in the form of UML state machine, and
the expected behavior of the SUT in the form of OCL constraints are used as
input to our approach. Using the UAV flight behavior information, a deep
reinforcement learning algorithm is trained to explore different flight test
scenarios while interacting with the environment at runtime. The flight
information received from the environment is used to determine the current
state of the UAV. This information is also used to populate an instance model
of the UAS SUT domain model. The instance model representing the current
flight information of the UAS SUT is used to evaluate OCL constraints. The
AITester gets rewards based on the correct selection of action and the number
of OCL constraints violated based on that action. During the testing process,
the AITester continues to explore different flight states, learns to violate
OCL constraints (expected behavior of the SUT) based on actions, and compiles
the results for further analysis by avionics testers.

We develop a toolset that implements our proposed approach to support testing
automation. We perform two experiments to evaluate AITester using two
different subsystems of the UAS. For the first experiment, we used an
autopilot of the UAV as a SUT. We use ArduCopter in the first experiment which
is a widely-used open-source autopilot system. For the second experiment, we
use the cockpit display systems (CDS) of the GCS as a SUT. We use an
industrial case study of GCS-CDS for the second experiment. For UAV autopilot,
the results show that AITester effectively generates test scenarios provoking
deviations from the expected behavior of the UAV autopilot. For GCS-CDS, the
results show that AITester is effective in finding four different types of
faults and in exploring diverse test scenarios. Overall, the results of both
experiments indicate that AITester effectively generates test scenarios
causing deviations from the expected behavior of the SUT that lead to
potential faults.

This paper is an extended version of the research abstract published in the
Doctoral Symposium of the 36th IEEE/ACM International Conference on Automated
Software Engineering (ASE) [8]. In the conference paper, an initial idea of
the proposed approach targeting UAVs and a pilot experiment on UAV autopilot
was presented. The primary contributions of this paper relative to the
conference paper are summarized below.

  * •

We present a comprehensive modeling methodology targeting the whole UAS. This
includes UML profiles for modeling the structural and behavioral aspects of a
UAS SUT and modeling guidelines for avionics testers. The modeling methodology
presented in this paper covers all UAS subsystems, e.g., GCS.

  * •

We propose a novel approach (AITester) that utilizes MBT and AI to
automatically generate, execute, and evaluate various test scenarios based on
the environmental context at runtime.

  * •

We develop a toolset that implements our proposed approach to support testing
automation and to enable further research and development. The toolset is
publicly available at an online
repository111https://github.com/hassansartaj/uast-toolset.

  * •

We perform two experiments to evaluate AITester using two different subsystems
of the UAS. The first experiment was performed using a UAV autopilot as a SUT,
extending the pilot experiment with a new research question. The second
experiment was performed using the CDS of GCS as a SUT to evaluate the
applicability of the proposed approach to another UAS subsystem.

The remaining portion of the paper is organized as follows. Section 2 presents
the background of unmanned aerial vehicles, model-based testing, reinforcement
learning, and deep learning. Section 3 describes our proposed approach in
detail. Section 4 provides a discussion on the toolset that implements the
proposed approach. Section 5 provides the empirical evaluation of the proposed
approach. Section 6 discusses the works related to this paper. Section 7
outlines limitations of proposed approach.  Finally, Section 8 concludes the
paper.

##  2 Background

This section presents a concise background discussion on unmanned aerial
systems, model-based testing, reinforcement learning, and deep learning.

###  2.1 Unmanned Aerial Systems (UAS)

Unmanned aerial systems (UAS) primarily consist of an unmanned aerial vehicle
(UAV) and ground control station (GCS). A UAV is an aircraft without a human
pilot onboard. It is either operated by a remote pilot in the GCS or using an
onboard autopilot system. To perform communication between UAV and GCS, a data
link layer is used in which the uplink is used to send the command to the UAV,
and the downlink is used to receive the flight information from the UAV. UAVs
are classified into nine different categories based on their weight, size,
operating altitude, range, and endurance [9]. UAV designs include fixed-wing
(e.g., plane), single-copter (e.g., helicopter), multi-copter (e.g., quad-
copter), and hybrid (e.g., quad-plane). UAVs can carry two types of payloads,
(i) dispensable payloads such as medical supplies, food items, or ammunition,
and (ii) indispensable payloads such as cameras, sensors, or radars of
different kinds. UAVs have a large number of applications in different civil
and military domains. In the civil sector, UAVs are used for rescue
operations, disaster management, geological surveys, agriculture and farming,
weather forecasting, wildlife monitoring, entertainment, etc. UAVs are also
used for different military operations such as intelligence, surveillance and
reconnaissance (ISR), airstrikes, military combat, and explosive material
detection.

A GCS is a ground-based cockpit for the monitoring and control of a UAV. The
GCS consists of the cockpit display systems (CDS), a flight control system, a
mission planner, a communication channel, a navigation system, 2D/3D maps, and
a payload monitoring and control system. The CDS of the GCS provides various
types of interfaces with different flight instruments to show the UAV flight
information [10]. These user interfaces display different types of information
received from the different avionics systems of a UAV during the flight. For
example, the navigation system generates information such as latitude,
longitude, and altitude using sensors such as gyros, accelerometers, and GPS.
This information is necessary for a UAV to accurately determine its current
position, next waypoint to follow, and navigate on the planned route. The
flight control system is used to generate control and navigational commands
for the UAV. The mission planner supports UAV profile management, route
planning, path optimization, payload setup, and planning unforeseen
situations. The communication channel is responsible for supporting all types
of communications between a UAV and a GCS. The navigation system is used to
monitor the navigational information (such as latitude, longitude, and
altitude) of the UAV. The 2D/3D maps support interactive mission planning,
mission execution, and observing the state of the UAV during flight. The
payload monitoring and control system helps an operator in observing the
status of the payload and in controlling the payload.

###  2.2 Model-based Testing

Model-based testing (MBT) is a testing technique that uses models of the
system under test (SUT) to support automation in various testing activities
such as test case generation and test execution. For conventional system
testing, where it is possible to construct complete models of expected
behavior, MBT provides a comprehensive solution for automated test generation,
test execution, and test evaluation [11]. The MBT process generally comprises
five steps [12]. In the first step, the abstract models of the SUT are
developed based on the requirements or specifications using any modeling
language such as Unified Modeling Langauge (UML) [13]. The SUT models capture
the system’s functionality in the form of various UML models (diagrams) such
as a class diagram, a state machine, and a sequence diagram. Different UML
models support the automation of several testing activities. For example, UML
state machines support the generation of test sequences [10]. The constraints
specified in the requirements are modeled using a constraint specification
language, i.e., Object Constraint Language (OCL) [7, 14]. The SUT models
developed in UML are augmented with the constraints written in OCL. The second
step is to determine the criteria for test case selection. The criteria for
selecting test cases can be defined based on requirements (e.g., requirement
coverage) or using the models of SUT (e.g., state/transition/transition-pair
coverage of the state machine). In the third step, the specifications of test
cases are defined based on the criteria for test selection. In the fourth
step, the models of SUT (created in the first step) and test case
specifications (defined in the third step) are used for the generation of test
cases. In the fifth step, the generated test cases are executed using the test
execution platform that is suitable for the SUT. In the end, the test verdict
is determined by comparing the test execution output with the SUT expected
output. If the expected output matches the test execution output, the verdict
is _pass_ and _fail_ otherwise.

Figure 1: An agent’s interaction with the environment [15]

###  2.3 Reinforcement Learning (RL)

Reinforcement learning (RL) is an artificial intelligence technique in which
an agent learns based on its interaction with the environment. The RL agent,
in a particular state, performs some action in the environment. Based on the
action, the environment returns the next state and the reward. Therefore, the
RL problem can be formulated as a Markov Decision Process (MDP). Fig. 1 shows
an MDP in which an agent in a state $S_{t}$ with reward $R_{t}$ performs an
action $A_{t}$ on the environment at time _t_. Based on the action, the
environment returns the next state $S_{t+1}$ and the reward $R_{t+1}$.
According to the reward received from the environment, the agent learns
whether the action was right or wrong. The process of learning continues until
the agent learns an optimal policy that can maximize the expected reward to
explore the environment. RL is used for multiple applications in a variety of
different domains [16] such as robotics, transportation, and games. In the
context of UAVs, it has been used for various purposes such as for the UAV
attitude control [17] and the robust control of the autonomous helicopter
[18].

###  2.4 Deep Learning

It is a machine learning method that simulates a part of the human brain. For
this purpose, it uses artificial neural networks (ANNs) inspired by the
biological neural networks of the human brain [19]. The ANN can be a single-
layer network with only input and output layers. It can be a multi-layer
network with an input layer, an output layer, and several hidden layers in the
middle. Multi-layer neural networks are also called deep neural networks.
There are three commonly used types of ANNs, (i) feed-forward neural network,
(ii) convolutional neural network (CNN), and (iii) recurrent neural network
(RNN). In a feed-forward neural network, the information from the input layer
flows toward the output layer after performing the function approximations in
the intermediate levels. The feed-forward neural networks are also known as
deep feed-forward networks or multi-layer perceptrons. In RNN, the information
from the input layer flows toward the output layer and can be fed back to the
neurons in the previous layer. In CNN, the information from the input layer
flows toward the output layer by performing different convolution operations
in the middle.

##  3 Approach

In this section, we discuss the proposed approach for the automated system-
level testing of the UAS. The proposed approach is developed based on the idea
presented in previous work [8]. First, we give a brief overview of the
proposed approach. Second, we describe the modeling methodology including a
discussion of the proposed profiles. In the end, we discuss the deep
reinforcement learning-based approach for the testing of a UAS SUT based on
environmental context at runtime.

Figure 2: An overview of the approach showing the proposed UAS profile, inputs
required from avionics testers, AITester, and UAV Environment. The dotted
arrows ( ) represent the compliance of models to the UAS profile and solid
arrows ( ) represent the information/control flow.

###  3.1 Overview of the Approach

Fig. 2 shows the overall workflow of the proposed approach. To model the
system under test, we propose two UML profiles, the UAS structural profile and
the UAV behavioral profile. The avionics tester is required to model the
domain of the UAS SUT using the structural profile and the flight behavior of
the UAV using the behavioral profile. The approach also requires the expected
behavior of UAS SUT in the form of OCL constraints and state invariants. The
proposed profiles play an important role in supporting the testing automation
of the UAS SUT. The UAS structural profile represents an observation space of
the environment. Similarly, the UAV behavioral profile represents the state
and action space of the deep reinforcement learning (DRL) algorithm. The UAS
SUT domain model serves two purposes, (i) initializing the observation space
of the environment, and (ii) generating an instance model that will be used to
evaluate the OCL constraints during execution. The flight behavior model of
the UAV is used to initialize the DRL algorithm for UAS SUT.

Using the UAV flight states and actions, the DRL algorithm starts exploring
various flight states using different actions (e.g., takeoff or land). The DRL
algorithm selects an action and executes it on the environment using DroneKit
API. The function of DroneKit is to receive the action command for the UAV,
send the command to the UAV using a communication link (using Mavlink), and
receive the state of the UAV. The continuous change in the flight state of the
UAV is monitored by a state observer. This helps the DRL to keep track of the
current state, action, and the next state of the UAV. Moreover, the flight
data observed during execution is used to populate an instance model of the
UAS SUT domain model. The instance model is evaluated on the OCL constraints
and state invariants. The primary objective of our approach is to violate a
maximal number of OCL constraints. The reward is calculated based on the
correct selection of action and the number of OCL constraints violated based
on that action. In this way, the DRL algorithm continues to explore different
flight states, learns to violate OCL constraints (expected behavior of the
SUT) based on actions, and compiles the results for further analysis by
avionics testers. The compiled results include information related to UAV
flight states, actions, and the number of failed OCL constraints for each
execution.

###  3.2 Modeling Methodology for UAS

A fundamental step in our approach is the modeling of the UAS SUT. This
includes modeling the UAS SUT domain concepts, the flight behavior of a UAV,
and constraints on these models. For this purpose, we propose a UAS structural
profile for modeling the UAS SUT domain concepts and a UAV behavioral profile
for modeling the flight behavior of the UAV. The goal of the proposed profiles
is to facilitate avionics testers with the modeling of different aspects of
UAS SUT using domain-specific terminologies. The domains-specific modeling
languages available in the literature are either generic for all CPS (e.g.,
[20]) or context-specific [21] (e.g., targeting UAV Swarm [22]). Our proposed
profiles are based on UML which is widely adopted by the avionics industry
[21]. Moreover, the proposed profiles consist of UAS domain concepts that
avionics testers are familiar with. We provide modeling guidelines for
avionics testers in the process of modeling a UAV using the proposed profiles
and specifying constraints.

####  3.2.1 UAS Structural Profile

Fig. 3 shows the profile to model the structural aspects of the UAS SUT. A key
concept of the profile is _UAV_ with properties such as _thrust_ , _heading_ ,
_airspeed_ , _groundspeed_ , and _type_. The type of UAV can be a helicopter,
multi-copter, fixed-wing, or hybrid UAV. The enumeration _UAVType_ is used to
model various types of UAVs. Another important concept is the _Attitude_ that
represents _pitch_ , _roll_ , _yaw_ , _pitch_speed_ , _roll_speed_ ,
_yaw_speed_ , and _yaw_rate_ of the UAV. The location of a UAV can be broadly
categorized into _LocationLocal_ and _LocationGlobal_. The _LocationLocal_ can
be represented in degrees using north/south, east/west, and up/down values.
Whereas the _LocationGlobal_ can be represented using _latitude_ , _longitude_
, and _altitude_ values. The _altitude_ can be measured above mean sea level
(i.e., _altitude_MSL_) and it can be relative to ground level (i.e.,
_altitude_AGL_). The UAV can have a _RangeFinder_ that is used to calculate
the distance. The UAV can have _Velocity_ along three axes i.e., _vx_ , _vy_ ,
and _vz_. To model different power systems, the profile provides two concepts,
_Engine_ and _Battery_. The _Engine_ has properties such as _speed_ , _level_
, and _flow_rate_. The _Battery_ has properties such as _voltage_ , _current_
, and _level_. Besides the above-mentioned concepts, the UAV can also have
concepts for modeling different measurement units such as _Accelerometer_ ,
_Gyroscope_ , _Barometer_ , and _Magnetometer_.

Figure 3: UAS profile for modeling structural details

Figure 4: An excerpt of UAV behavioral profile for modeling flight states

Figure 5: An excerpt of UAV behavioral profile for modeling flight actions

####  3.2.2 UAV Behavioral Profile

To model the flight behavior of the UAV, the behavioral profile has two parts.
One for modeling the abstract flight states as shown in Fig. 4. The second
part is used for modeling the events that can lead the UAV into various flight
states. An excerpt of the behavioral profile for modeling events is shown in
Fig. 5.

The abstract flight states represent various flight stages of the UAV.
Initially, the UAV is in a _Disarmed_ state. After arming the motors, the UAV
goes to the _Armed_ state. From the _Armed_ state, the UAV can go to the
_Taxiing_ or _TakeOff_ state. If the UAV is fixed-wing, it has to start taxi
on the ground before it can take off. Whereas the single or multi-copter type
of UAV can directly take off from the ground without taxiing. In the case of
hybrid UAVs, both states can be used alternatively. After taking off from the
ground, the UAV can go through different flight phases such as _Climb_ ,
_Cruise_ , _Descent_ , _PositionHold_ , and _AltitudeHold_. The UAV can
perform special types of flights such as _Flipping_ , _Drifting_ , _Loiter_ ,
and _Circle_. The UAV can fly straight when in _FlyingStraight_ state, it can
take a left turn when in _TurningLeft_ state, and it can take a right turn
when in _TurningRight_ state. At the end of the flight, first, the UAV goes to
the _Approach_ state, and then it goes to the _Landing_ state.

Each abstract flight state is achieved using different events. To arm the
motors, the _Arm_ event is used. Similarly, the _Disarm_ is used to disarm the
motors. The event _Takeoff_ is used to take off the UAV from the ground and
the event _Land_ is used to land the UAV on the ground. The event
_ReturnToLaunch_ is used to make the UAV go to the home position (from where
it took off) and land at that position. During the flight, UAV can move
forward or backward (not in the case of fixed-wing) using the events
_MoveForward_ and _MoveBackward_ respectively. Similarly, it can make a left
or right turn using the events _TurnLeft_ and _TurnRight_ respectively. In the
case of multi-copter or hybrid UAV, the event _HoldPosition_ makes it maintain
the current position. The event _HoldAltitude_ is used to maintain a stable
altitude. Whereas the events _IncreaseAltitude_ and _DecreaseAltitude_ are
used to increase or decrease the altitude respectively. Note that the profile
shown in Fig. 5 has only a subset of all the events modeled in the complete
profile. The complete profile is available at an online
repository222https://github.com/hassansartaj/uas-profile.

####  3.2.3 Modeling Guidelines

To model the domain of the UAS SUT, all avionics components (e.g., sensors)
need to be modeled as a UML stereotyped class with the stereotype from the UAS
structural profile. The profile stereotypes contain some common properties of
various avionics components. The matching properties need to be used as it is
and the additional properties can be added to the domain class. The avionics
tester can freely model the associations among the domain classes.

The flight behavior model of the UAV needs to be modeled using the UML state
machine. For this purpose, all flight states and events need to be modeled
using the states and events stereotypes from the UAV behavioral profile. The
avionics tester is required to model the end-to-end flight behavior of the
UAV, i.e., to take off from the ground, perform different flight operations,
and land on the ground. For example, it is not possible for a UAV to fly in
the air without initial ground operations, therefore, the UAV flight state
machine without ground states/events will not be useful. All the state machine
events are required to be modeled as UML _CallEvents_. Other types of events
such as _SignalEvent_ , _TimeEvent_ , and _ChangeEvent_ are simulator-specific
events that can not be controlled using the DRL algorithm. In the case of
guard conditions, our approach requires modeling the guards in the form of OCL
constraints or invariants. While modeling transitions, the avionics tester
needs to model all possible transitions and events for a flight state.

###  3.3 Modeling of UAS SUT

Our approach requires three inputs. First, the domain model of the UAS SUT
needs to be modeled using the structural profile. Second, the flight behavior
is in the form of a UML state machine that is modeled using the behavioral
profile. The third input is the expected behavior of UAS SUT in the form of
OCL constraints that include expected ranges of values on various properties
of the UAS SUT and w.r.t. the flight states. In the following, we discuss the
modeling of the UAS SUT domain, flight behavior, and OCL constraints.

Figure 6: ArduCopter domain model

Figure 7: ArduCopter flight behavioral model in the form of UML state machine

Figure 8: ArduCopter sub-state machine for turns

####  3.3.1 UAS SUT Domain Modeling

To model the domain of UAS SUT, an instance of the structural profile is
created for the target UAS component. Fig. 6 shows a domain model for the
autopilot of a multi-copter named ArduCopter. The _ArduCopter_ is modeled as
an instance of _UAV_ meta-class. The location of the ArduCopter is measured
according to the global relative point, so _CopterLocation_ is modeled as an
instance of the _LocationGlobalRelative_ meta-class. The attitude (roll,
pitch, and yaw) of the ArduCopter is modeled using the _Attitude_ stereotype.
The ArduCopter has a range finder to calculate the distance which is modeled
using the _RangeFinder_ stereotype. Similarly, the speed of the ArduCopter
along the three axes is modeled using the _Velocity_ stereotype. Finally, the
power system used in the ArduCopter is modeled using the _Battery_ stereotype.

The domain model of the UAS SUT serves two purposes. First, it is used to
provide the UAS SUT domain classes and properties to the environment used for
the reinforcement learning algorithm (Section 3.4). This information becomes
the observation space of the environment that is observed during the UAV
flight. The second use of the domain model is to evaluate OCL constraints. For
this purpose, an instance of the UAS SUT domain model is created and the slots
are filled using the UAV flight data. The OCL constraints modeled for the
corresponding flight state are evaluated on the instance model. The evaluation
results are used later on for the reward calculation (Section 3.4.2).

####  3.3.2 Flight Behavior Modeling

The flight behavior of the UAV is represented by many abstract flight states
and events. Fig. 7 and Fig. 8 show the flight behavior modeled as a UML state
machine for the ArduCopter. The flight state machine contains various abstract
flight states modeled as instances of meta-states from the UAV behavioral
profile. Similarly, all the events are modeled as instances of various flight
events from the UAV behavioral profile.

Initially, the ArduCopter is in _Idle_ state which is modeled using the
_Disarmed_ profile stereotype. To start the motors (arm the UAV) in the _Idle_
state, the event _armUAV()_ is used to take the ArduCopter to the _Armed_
state. When the ArduCopter is in the _Armed_ state, it can either turn off the
motor to go back to the _Idle_ state or it can go to _Takeoff_ state using the
_takeoff()_ event. After taking off from the ground, the ArduCopter has six
options. It can go to the _Ascend_ state using the event _increaseAlt()_ , or
it can go to the _Descend_ state using the event _decreaseAlt()_ , or it can
go to the _Loiter_ state using the event _startLoiter()_ , or it can go to the
_PositionHold_ state using the event _holdPosition()_ , or it can go to the
_AltitudeHold_ state using the event _holdAlt()_ , or it can go to the
_Landing_ state using the event _landUAV()_. If its next state is _Ascend_ ,
it can remain in that state, or it can go to the _Loiter_ or _PositionHold_ or
_AltitudeHold_ or _Descend_. Moreover, the _Ascend_ state has a sub-state
machine for different types of turns, _Straight_ , _TurningRight_ , and
_TurningLeft_ as shown in Fig. 8. That means, the ArduCopter can _Ascend_
while moving straight or it can take a left turn or right turn.

_Ascend_ is the state in which the ArduCopter is making a climb i.e.,
increasing the altitude. _Descend_ is the state in which the ArduCopter is
making a descending i.e., decreasing the altitude. The _Descend_ state also
has a sub-state machine for turns because the ArduCopter can move straight or
it can take a left turn or right turn. The _Loiter_ state represents a flight
phase in which the ArduCopter is flying in a small circle. During the _Loiter_
state, the ArduCopter can loiter in a clockwise or counterclockwise direction.
Thus, it has a sub-state machine for right and left turns for loitering in a
clockwise or counterclockwise direction. The state _PositionHold_ corresponds
to the flight phase in which the ArduCopter is holding its position while
keeping the attitude constant. Therefore, _PositionHold_ state does not have a
sub-state machine for different types of turns. Similarly, the _AltitudeHold_
state represents a flight phase in which the ArduCopter is flying at the same
level while keeping the altitude constant. For the ArduCopter types of UAV, it
is possible to maintain constant altitude and can move forward or backward or
turn left or right.

Listing 1: Some of the OCL constraints including the general and the state
invariants

⬇

\--altitude values range (in meters)

C1: context ArduCopter inv: self.location.altitude_AGL>0 and
self.location.altitude_AGL<=300

\--distance values range (in meters)

C2: context ArduCopter inv: self.rangefinder.distance>0 and
self.rangefinder.distance<=5000

\--State invariants for Takeoff

C3: context ArduCopter inv: self.oclIsInState(Takeoff) and self.thrust>0 and
self.thrust<1

C4: context ArduCopter inv: self.oclIsInState(Takeoff) and
self.location.altitude_AGL>0 and self.location.altitude_AGL<=50

\--State invariants for Landing

C5: context ArduCopter inv: self.oclIsInState(Landing) and self.airspeed>0 and
self.airspeed<=5

C6: context ArduCopter inv: self.oclIsInState(Landing) and self.speed.vz>0 and
self.speed.vz<=5

\--State invariants for Ascend

C7: context ArduCopter inv: self.oclIsInState(Ascend) and self.airspeed>0 and
self.airspeed<100

C8: context ArduCopter inv: self.oclIsInState(Ascend) and self.groundspeed>0
and self.groundspeed<10

\--State invariants for Descent

C9: context ArduCopter inv: self.oclIsInState(Descent) and self.airspeed>0 and
self.airspeed<100

C10: context ArduCopter inv: self.oclIsInState(Descent) and
self.location.altitude_AGL>10 and self.location.altitude_AGL<100

The UAV flight behavior modeled in the form of a UML state machine is used by
the reinforcement learning algorithm (Section 3.4). Each abstract flight state
becomes a part of the state tuple. Each event modeled in the state machine
becomes the action space. Similarly, all the transitions of the state machine
represent the transitions model that will be used to keep track of the current
and the next states. The UAV state machine plays an integral part in executing
UAV flight behavior (i.e., transitioning the UAV through various flight
states), which is performed by AITester during the testing process (Section
3.4.5).

####  3.3.3 Specifying Expected Behavior of the SUT

After modeling the domain UAS SUT and the flight behavior of the UAV, the
expected behavior of UAS SUT in the form of OCL constraints is required to be
specified. The OCL constraints should contain the expected ranges of value on
various properties of the UAS SUT and w.r.t. the flight states. The purpose of
the constraints is to check the invalid property values during the UAV flight.
When formulating OCL constraints corresponding to UAV flight states, the
specified ranges for a particular state must be consistent and non-overlapping
with the subsequent state. For example, if the maximum altitude limit for
_Takeoff_ state is 50 feet, the altitude range for the subsequent state
(_Ascend_) should consider 50 feet as the lower bound. Furthermore, the ranges
defined in OCL constraints associated with UAV flight states should align with
those specified in the general constraints. The outcomes of the OCL
constraints evaluation play a substantial role in calculating the reward and
validating the expected behavior of the UAV.  Therefore, the constraints need
to be carefully modeled to capture the required (valid) range of values for
the different properties during different flight states. Listing 1 shows ten
OCL constraints as an example. Among those constraints (_C1_ & _C2_), two are
general which means these constraints must be true regardless of any flight
state. The remaining eight constraints (_C3_ – _C10_) are state invariants
modeled for the _Takeoff_ , _Landing_ , _Ascend_ , and _Descend_ states.
Constraint _C1_ models the valid altitude values range with 300m maximum
altitude for the ArduCopter. Constraint _C2_ shows the valid values range for
the distance in which the maximum distance the ArduCopter can cover is 5000m.
Constraints _C3_ and _C4_ capture the valid range for the _thrust_ and
_altitude_AGL_ during the _Takeoff_ state. Similarly, constraints _C5_ and
_C6_ capture the valid range of values for the _airspeed_ and _speed_ along
the z-axis during the _Landing_ state. Constraints _C7_ and _C8_ define valid
ranges for _airspeed_ (i.e., between 0 and 100 knots) and _groundspeed_ (i.e.,
between 0 and 10 knots) during the _Ascend_ state. Moreover, constraints _C9_
and _C10_ specify valid ranges for _airspeed_ (specifically between 0 and 100
knots) and _altitude_AGL_ (specifically between 10 and 100 feet) during the
_Descend_ state.

###  3.4 System-level Testing of UAS

A typical practice of the avionics tester is to simulate the flight behavior
of the UAV using the SIL/HIL simulator. During the simulation, the tester
performs a certain action, observes the next state of the UAV in an
environment, and analyzes the flight data to see if each avionics component is
behaving according to the specification. This process of testing requires
control of the UAV, continuously deciding various actions, and observing state
change in the UAV and environment. To automate this process, human-level
decision-making and control of the UAV are required at runtime. Such a problem
can be devised in the form of a Markov Decision Process (MDP) [23]. In this
case, an AI technique, deep reinforcement learning (DRL) is an appropriate
choice as compared to meta-heuristics search algorithms. Moreover, DRL
provides a mechanism for on-the-fly testing of UAS in a dynamic environment
which cannot be achieved by traditional model-based testing approaches (e.g.,
[24]). Using DRL, our approach introduces an AITester in place of a human
tester to perform the system-level testing of the UAV. We formulate this
problem as an MDP which consists of a set of states, a set of possible actions
that can be performed in a state, a transition model, and a reward function
[23]. For this purpose, the proposed profiles, UAS structural profile, and UAV
behavioral profile. play an important role. More discussion on the adaptation
of DRL to automate the system-level testing of the UAV is given in the
following subsections.

Figure 9: An example of the UAV flight scenario with different state changes

####  3.4.1 Definitions

Definition 1 (State Space). A state space is defined as a ordered set of
states $S=\\{S_{0},S_{1},S_{2},\dots,S_{g}\\}$ in which each state is
represented by a tuple $S_{x}={\left\langle
s_{0},s_{1},\dots,s_{9}\right\rangle}$. Where $S_{g}$ represents the goal
(final) state and $s_{0}$ represents one of the flight state stereotypes
modeled in the UAV flight state machine. The remaining state elements are
denoted as $s_{1}:=altitude$, $s_{2}:=airspeed$, $s_{3}:=groundspeed$,
$s_{4}:=roll$, $s_{5}:=pitch$, $s_{6}:=yaw$, $s_{7}:=heading$,
$s_{8}:=battery$, and $s_{9}:=distance$.

Fig. 9 shows an example of the different states in the state space considering
the UAV flight state machine shown in Fig. 7. For the given example, the
initial state is represented as
$S_{0}={\left\langle\mbox{\em{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Disarmed}},0.0,0.0,0,0,0.0,0.0,0.0,1,100,0.0\right\rangle}$
, where Disarmed corresponds to $s_{0}$ in the tuple $S_{x}$. In this state,
all the values are zero except the battery level which is 100. The next state
is
$S_{1}={\left\langle\mbox{\em{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Armed}},0.0,0.05,0.2,0.0081,0.0084,0.0235,357,99,0.0\right\rangle}$
where the _airspeed_ , _groundspeed_ , _roll_ , _pitch_ , _yaw_ , _heading_ ,
and _battery_ values are little changed. After the UAV is armed, the next
flight state is Takeoff. Therefore, the next state is
$S_{2}={\left\langle\mbox{\em{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Takeoff}},20.0,0.22,2.5,-0.004,-0.004,-0.0501,357,98,0.0\right\rangle}$
in which the _altitude_ is increased to 20.0m, the _airspeed_ is increased to
0.22, the _groundspeed_ is increased to 2.5, and there is a small change in
_roll_ , _pitch_ , _yaw_ , and _battery_ values. Similarly, the UAV state
continues to change i.e., from $S_{2}$ to $S_{3}$ to $S_{4}$ to $S_{5}$ to
$S_{6}$ to $S_{7}$ state which is the goal (final) state. The goal state
$S_{g}$ is represented as
$S_{7}={\left\langle\mbox{\em{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Disarmed}},0.0,0.0,0.0,0.0031,0.00039,0.0504,357,90,20.0\right\rangle}$.

Definition 2 (Action Space). The action space is represented as a set
$A=\\{A_{0},A_{1},A_{2},\dots,A_{n}\\}$, where _n_ is the total number of
events in the UAV flight behavior model. Each action in the action space $A$
corresponds to an event modeled in the flight behavior model of the UAV
corresponding to abstract flight events of the UAV behavioral profile. For the
example state transitions shown in Fig. 9, the action $A_{0}$ corresponds to
the event _armUAV()_ , the action $A_{1}$ corresponds to the event _takeoff()_
, the action $A_{2}$ corresponds to the event _increaseAlt()_ , the action
$A_{3}$ corresponds to the event _decreaseAlt()_ , the action $A_{4}$
corresponds to the event _landUAV()_ , the action $A_{5}$ corresponds to the
event _armUAV()_ , and the action $A_{6}$ corresponds to the event
_disarmUAV()_.

Definition 3 (Correct/Incorrect Action). An action is termed as correct if it
is one of the possible actions (modeled in the flight state machine) for a
given current state, otherwise, it is termed as an incorrect action. Our
approach uses the UAV flight state machine to distinguish between correct and
incorrect actions. The UAV flight state machine provides the possible set of
corrective actions in a particular flight state. The distinction between
correct and incorrect action is important in our case because if the selected
action is incorrect it will either create zero or a negative effect on the
UAV. For example, if the UAV is on the ground in the _Armed_ state and the
selected action is _Loiter()_ , it is not possible for the UAV to start
loitering.

Definition 4 (Transition Model). A transition model is defined as a
probability of transition from a current flight state to the next flight state
(in states space $S$) using an action from the actions space $A$. Equation 1
shows a transition model in which $S_{x+1}$ represents the next state in $S$,
$S_{x}$ represents the current state in $S$, and $A_{x}$ represents the
current action in $A$. According to the equation, there is a probability of
the transition from the current state $S_{x}$ to the next state $S_{x+1}$
using the action $A_{x}$.

$T=P(S_{x+1}|S_{x},A_{x})$ (1)

For the example state transitions diagram shown in Fig. 9, using the action
$A_{0}$ in state $S_{0}$, the next state is $S_{1}$. Similarly, the action
$A_{1}$ in state $S_{1}$ leads to the next state $S_{3}$.

Definition 5 (Observation Space). An observation space is defined as a vector
of UAV properties, i.e., $O=[o_{0},o_{1},o_{2},\dots,o_{n}]$ from the UAS SUT
domain model created using the UAS structural profile. Where $o_{x}$
corresponds to the UAV properties such as _thrust_ , _airspeed_ , _vx_ , _vy_
, and _vz_.

####  3.4.2 Reward Function

The objective AITester is to take control of the UAV flight during simulation,
perform various actions on the UAV, and maximize the number of violations of
the expected behavior. The reward function is designed based on two
parameters, (i) correct/incorrect actions, and (ii) violations of the expected
behavior specified as OCL constraints. These two parameters are important
because a series of correct/incorrect actions may lead a UAV to crash. For
example, if AITester continues to execute a correct action _decreaseAlt()_
multiple times in _Descent_ state, the UAV can hit the ground leading it to
crash. This can happen frequently during training because the inherent
behavior of the DRL algorithm involves exploring various possibilities of
actions and observing their outcomes. Therefore, a correct action leading the
UAV to crash is not considered a fault in the UAV. To handle such a situation,
our approach utilizes the violation of the expected behavior of a UAV
specified in the form of OCL constraints. The trace of violations in expected
behavior can help in diagnosing faults in the SUT. Since OCL constraints are
specified corresponding to each flight state of the UAV, it is possible to
have multiple violations of OCL constraints at different states during the
whole flight.

The reward for a single action performed in a particular state is calculated
using Equation 2. For a correct action, the AITester gets a reward of $(1+m)$,
where $1$ represents the reward for the correct selection of action and $m$
represents the number of violations in the expected behavior (failed OCL
constraints) based on that action. To calculate the number of failed OCL
constraints, the UAV flight data (e.g., altitude, airspeed, roll, pitch, and
yaw) sent by the state observer is used to populate an instance model which is
generated using the input UAS SUT domain model (as shown in Fig. 2). The
instance model populated in this way represents the current flight context of
the UAV during execution. The constraints evaluator uses the input OCL
constraints and state invariants for the current flight state and evaluates
these constraints on the UAV instance model. The constraints evaluator returns
the number of failed OCL constraints (_m_) for the given flight data. Since
different OCL constraints are specified on a particular state, multiple
violations of one constraint are considered as one violation. In this way, the
AITester learns the flight behavior of the UAV, and at the same time, utilizes
the flight behavior to explore the scenarios that violate the expected
behavior of the SUT. For an incorrect action, the AITester gets the reward of
$-1$.

$r=\left\\{\begin{array}[]{r}1+m,\ \ correct\ \ action\\\ -1,\ \ incorrect\ \
action\end{array}\right.$ (2)

The cumulative reward for a single episode is calculated using Equation 3.
According to this Equation, the cumulative reward is the added sum of the
reward for individual action performed at time _t_ (calculated using Equation
2) multiplied with discount factor ${\gamma}^{t}$ at time step _t_. The values
of the discount factor $\gamma$ are between 0 and 1, i.e., $\gamma\in[0,1]$.
The discount factor determines the AITester’s preference between the current
and future rewards. If the value of $\gamma$ is close to 0, the AITester will
prefer current rewards over future rewards. If the value of $\gamma$ is close
to 1, the AITester will prefer future rewards over the current rewards. The
purpose of factor ${\gamma}^{t}$ is to set a value close to 1 and
progressively decrease the value after each time step _t_. In this way,
AITester gives preference to future rewards at initial time steps and moves
slowly towards current rewards.

$R=\sum_{t=0}^{n}\gamma^{t}*r_{t}$ (3)

####  3.4.3 Long Short-Term Memory (LSTM) Network

The UAV flight data at a particular time is correlated to the flight data at
the previous instance of time. Due to this UAV flight data holds a temporal
relationship based on environmental context which is highly dynamic. Retaining
information among these data points in subsequent time steps is important to
comprehend the flight context. Moreover, our approach utilizes Deep Q-Network
(DQN) [25] which works on the experience replay method by taking random
samples from the replay memory of past experiences. This is an inherent
feature of DQN to make learning efficient, however, it breaks correlation
among different data points which makes it difficult to acquire the flight
context. In such a scenario, among different types of artificial neural
networks, the recurrent neural network (RNN) specifically the LSTM network
architecture is most suitable for the temporal nature of data and capturing
the state context. The LSTM network consists of one or more input gates,
output gates, forget gates, and memory cells. The core purpose of the memory
cell is to capture the correlation between the current flight data at time _t_
with preceding flight data points [i.e., _(t-1)!_].

Figure 10: A single layer representation of LSTM architecture used with the RL
algorithm

Fig. 10 shows a single-layer representation of LSTM architecture used in our
approach. The LSTM network architecture used in our approach consists of an
input layer, three LSTM layers with a hidden dimension of 10, and an output
layer. The inputs of the neural network represent a state in states space $S$.
The first input is the abstract flight state that corresponds to one of the
states in the flight behavior model. The remaining inputs such as _altitude_ ,
_airspeed_ , _groundspeed_ , _roll_ , _pitch_ , _yaw_ , _battery_ , _heading_
, and _distance_ represent the information displayed on the cockpit of the
ground control station which is received from the UAV during flight. This
information is necessary for the remote pilot to control the UAV during flight
[10]. That is, based on this information, the remote pilot decides a sequence
of actions to successfully fly the UAV. The activation function used for the
hidden layers of our LSTM network is the default hyperbolic tangent (tanh)
[26]. The output of the neural network is the possible actions (from the
action space $A$) that can be performed given the input state. From the
possible actions, the action with the highest probability is selected using
the _softmax_ function.

####  3.4.4 UAV Environment

The agent has to interact with the UAV to perform an action, observe the state
of the UAV, and get the reward for the selected action. To facilitate these
operations, we build a goal-based environment with the goal of successfully
landing the UAV on the ground. Under normal conditions, after performing
different flight operations, if the UAV should land on the ground, the goal is
achieved. Another possibility is that the UAV crashes due to a
correct/incorrect sequence of actions (Section 3.4.2). In such a case, the
goal is not achieved. UAV crashes during test execution can be observed in a
simulation environment without requiring an actual UAV.

The environment takes state space and action space from the flight behavioral
model (Section 3.3.2) and the observation properties from the domain model
(Section 3.3.1). For each action in the action space, the environment
implements the UAV-specific action commands. Similarly, for each action in a
particular state, the environment keeps track of the next state. If the next
state is the final (goal) state, the environment notifies the agent that the
goal has been achieved. Based on the action (correct/incorrect), the
environment produces observations for the next state. During the execution, it
receives the action from the agent, determines the current state, performs the
action on the UAV in simulation, observes the next state, and checks if the
goal state is achieved or not.

####  3.4.5 Testing Process of AITester

To begin, AITester requires three inputs from avionics testers (as shown in
Fig. 2); (i) the domain model of the UAS SUT developed using the UAS
structural profile, (ii) the flight behavior in the form of a UML state
machine developed using the UAV behavioral profile, and (iii) the expected
behavior of UAS SUT in the form of OCL constraints. The UAS SUT domain model
is used to initialize the observation space and to generate an instance model
that supports the evaluation of OCL constraints during execution. The flight
behavior model of the UAV is used to initialize the state and action space of
the DRL algorithm.

AITester incorporates Deep Q-Network (DQN) as a DRL algorithm [25]. DQN has
been widely used to train different game-playing bots and to get human-level
control [27, 28]. The testing process starts by initializing the LSTM network
called the policy network, the environment, and by getting the initial state
of the UAV. For the selected state, the AITester selects an action randomly
(explore) or by using the policy network (exploit). For this purpose, the
AITester uses Epsilon $\epsilon$-greedy strategy to balance exploration and
exploitation. Before executing the selected action, it is analyzed whether
this action can be performed on a UAV or not. This is done using the flight
state machine. If the selected action can not be performed, it is omitted and
the negative reward is returned. In this case, the state of the UAV is not
changed. If the selected action is fine, it is performed on a UAV operating in
an environment.

Using the UAV flight states and actions, the AITester starts exploring various
flight states using different actions (e.g., _takeoff_ or _land_). The
AITester selects an action and executes it using Action Executor. The Action
Executor receives an action, establishes a connection with the UAV through
DroneKit API, and sends the action command to the UAV. The function of
DroneKit is to receive the action command for the UAV, send the command to the
UAV using a communication link (using Mavlink), and receive the state of the
UAV. The continuous change in the flight state of the UAV is monitored by a
State Observer. This helps the AITester to keep track of the current state,
action, and the next state of the UAV. Moreover, the flight data observed
during execution is used to populate an instance model of the UAS SUT domain
model. The instance model is evaluated on the OCL constraints and state
invariants using a Constraints Evaluator. The Constraints Evaluator provides
the number of violations of OCL constraints. Based on the selected action and
the number of violations of OCL constraints, the AITester gets the reward.

During training, this process continues for a specified number of episodes,
and the AITester continues to explore different flight states based on
different actions and learns to violate OCL constraints (expected behavior of
the SUT). The training is performed using a flight simulator. At the end of
the training process, the trained model is saved. In the evaluation phase, the
trained model is loaded and AITester selects various actions using the trained
model, performs the selected actions on the UAV operating in an environment,
observes the outcomes, and compiles testing results for further analysis by
avionics testers. The compiled results include information related to UAV
flight states, actions, and the number of failed OCL constraints for each
execution. This information helps an avionics tester to analyze the sources of
faults in the SUT. At the end of the process, the policy network is saved so
that it can be reused in later stages of testing.

##  4 Implementation

We implement the proposed approach in a toolset to support the automated
system-level testing of the UAV. All main modules of the toolset are packed
into two separate packages. The toolset is available at the GitHub
repository1. In the following subsections, we discuss every module of the
toolset w.r.t. each package.

###  4.1 Model Handler

This component takes input UAS SUT domain model, flight behavior model, and
OCL constraints. Using this input, it loads the models, generates a model
instance, prepares the instance model for evaluation during execution, and
evaluates the OCL constraints on the instance model. Each module of this
component is developed using the Java programming language.

####  4.1.1 Model Loader

First, this module takes the UAS SUT domain model in the form of UML and loads
this model in a data structure along with the meta-class elements (i.e.,
profile stereotypes). Second, it takes the flight behavior model in the form
of a UML state machine and traverses the model to extract flight states,
events, and transitions. In case the state machine has composite and
orthogonal states, these states are flattened using the algorithm presented in
[24]. For the modeling manipulation, it uses an external API named Eclipse
Modeling Framework (EMF333https://www.eclipse.org/modeling/emf/) that allows
it to load and read the model information conveniently. The third function of
this module is to load the OCL constraints and state invariants.

####  4.1.2 Instance Generator

This module uses the domain model provided by Model Loader and the UAS profile
to generate an instance model. For this purpose, EMF is used to create
instances of the domain classes and the associations among these classes. The
properties for each class are created using the domain class and profile
stereotype attributes. A slot is created for each property in the instance
model. The generated instance model is saved in the form of a UML model.

####  4.1.3 Instance Populator

The responsibility of this module is to receive the UAV flight data and
populate the instance model generated using Instance Generator. First, the
classes and properties are mapped between the UAV flight data and the instance
model. After that, the flight data corresponding to each UAV class and
properties are used to fill the corresponding slots in the instance model.

####  4.1.4 Constraints Evaluator

The core functionality of this module is to evaluate the OCL constraints and
state invariants on the instance model containing the UAV flight information.
For this purpose, the Eclipse
OCL444https://projects.eclipse.org/projects/modeling.mdt.ocl evaluator is
used. Before starting the evaluation process, the OCL constraints and state
invariants are extracted (using Model Loader) corresponding to each state of
the UAV flight. During execution, the flight data received from the
environment is sent to Instance Populator. The instance model is used to
prepare an OCL evaluator environment for the evaluation of OCL constraints
[29, 30]. Before starting the evaluation process, OCL constraints are loaded
from a single file corresponding to each state of the UAV flight. This file
contains general constraints and state-specific constraints, resembling the
format depicted in Listing 1.  All OCL constraints and state invariants are
evaluated on the populated instance model. If an OCL constraint or state
invariant fails on the instance model, the Eclipse OCL evaluator returns
_false_. If the OCL constraint or state invariant is passed on the instance
model, the Eclipse OCL evaluator returns _true_. At the end of the constraints
evaluation, the evaluator returns the total number of failed OCL constraints
(violations in the expected behavior of the SUT) for the given flight data.

###  4.2 AITester

This is the main part of the toolset. The key functions of this component are
UAV environment preparation, interaction with Constraints Evaluator for the
reward calculation, and training and evaluation modes for system-level testing
of UAVs. Since AITester is aimed to be used at different levels of testing,
the tool contains a training module as well as an evaluation module. All
modules of the AITester are developed using Python programming language. The
deep learning framework used for this purpose is PyTorch [31].

####  4.2.1 Training Module

The main function of this module is to train the AITester for the automated
system-level testing of the UAV. This module implements a deep reinforcement
learning algorithm named Deep Q-Network (DQN) along with the LSTM network. For
the DQN, it takes various hyperparameters such as the number of episodes,
batch size, gamma, replay memory size, target update, $\epsilon$-start,
$\epsilon$-end, and $\epsilon$-decay. Before the start of the training
process, it determines the current state of training i.e., whether the
training needs to start from scratch or load the previous checkpoint and start
from that point. To reload from the previous checkpoint, the LSTM networks
(both policy and target network) are loaded using PyTorch model loading API.
Each episode of the training process starts with an initial state, selects an
action using the LSTM network, gets a reward from the Environment, and
observes the next state. During each episode, the explored states,
correct/incorrect actions, and resultant failed OCL constraints are compiled
and stored in a file. To balance exploration and exploitation, the Epsilon
$\epsilon$-greedy strategy is implemented. For the training, Adam optimizer
[32] and SmoothL1Loss functions are used. After a specified interval, the
state of the training process (including LSTM networks) is stored. When the
training process finishes, the trained LSTM network is stored in the specified
directory so that it can be used for evaluation.

####  4.2.2 Evaluation Module

The role of this module is to perform the system-level testing of UAVs when
the AITester is trained in the SIL simulator. This module takes a trained LSTM
network and loads it using PyTorch. The process of this module is identical to
the main procedure of the DQN algorithm during training except for the
learning parameters such as the use of replay memory, calculation of Q-values,
loss calculation, and gradient optimization. The main use of this module is
the facilitation of testing at HIL and PIL levels. This module also supports
the compilation of testing results at the end of each test execution.

####  4.2.3 Environment

The core feature of this module is to create an environment in which the
AITester can perform actions, observe the UAV flight state, and calculate
rewards. For this purpose, this module provides an environment manager to
facilitate these operations. To perform an action on the UAV, this module uses
the DroneKit API555https://dronekit.io/. The DroneKit uses Mavlink as a
communication link to communicate with the UAV. The main benefit of the
DroneKit is that it is independent of a particular simulator environment. This
makes our toolset capable of performing system-level testing of the UAVs at
SIL, HIL, and PIL levels.

####  4.2.4 Py2Java Communicator

In this module, py4j666https://www.py4j.org API is used to create a
communication channel between the python component (AITester) and the Java
component (Model Handler). The core feature is to send the UAV flight
information to Model Handler for the constraints evaluation, receive the
results, and pass them on to the AITester during the execution. Initially,
this module gets the UAS SUT domain model details and flight behavior model
information from the Model Handler. This information is used to create and
initialize the state, action, and observation space of the environment.

####  4.2.5 UAV Commands API

We provide an API for the execution of UAV flight events on the simulator
environment. This API contains the concrete implementation of DroneKit-
specific commands for ArduCoper and ArduPlane corresponding to the abstract
flight events modeled in the flight behavior profile. Since the API contains
the implementation of abstract flight events, it can easily be extended and
modified for any other simulator that supports DroneKit.

##  5 Empirical Evaluation

This section presents the empirical evaluation of AITester including two
experiments. The first experiment is performed using the ArduCopter autopilot
system and the second experiment is performed using an industrial case study
of ground control station (GCS) cockpit display systems (CDS), i.e., GCS-CDS.
In the following subsections, we discuss research questions, the details of
both case studies, the setup of both experiments, the design and execution of
the experiments, the results of both experiments, and the possible threats to
the validity of experimental results.

###  5.1 Research Questions

AITester utilizes a UAV flight state machine for the system-level testing of
UAS system under test (SUT) with the goal of violating the expected behavior
of the SUT. There are two key aspects for analyzing the performance of an
AITester. The first is the effectiveness of AITester in generating test
scenarios that can lead to deviations from the expected behavior of the UAS
SUT. During the process of testing, the second aspect is the ability of the
AITester to explore diverse paths from the UAV flight state machine. We
formulate research questions for the experimental evaluation based on these
two aspects.

  * •

RQ1: Is AITester effective in violating the expected behavior of a UAS SUT?

  * •

RQ2: Is AITester capable of exploring diverse flight paths during testing?

Both RQs are important from the testing perspective of a UAS SUT. The goal of
the AITester is to perform system-level testing of a UAS SUT using a state
machine representing the flight behavior of a UAV. RQ1 assesses AITester in
terms of violations of the expected behavior of a UAS SUT that can lead to
potential faults. This is a primary concern of testing a UAS. Since a flight
state machine is utilized in the testing process, it is quite possible that
AITester learns to explore a specific set of flight states and actions leading
to maximum violations using the same set of OCL constraints. This results in
testing of the same behavior multiple times which is not a desired feature.
Therefore, the analysis of diversity in exploring flight paths is an important
consideration.

###  5.2 Case Studies Description

For the first experiment, we use ArduCopter777https://ardupilot.org/copter/
which is a part of the widely used open-source
ArduPilot888https://ardupilot.org/ardupilot/index.html project. The ArduCopter
is used to operate a variety of multi-copter UAVs including a helicopter, tri-
copter, quad-copter, hexacopter, etc. It supports a large variety of flight
modes including manual, semi-autonomous, and fully autonomous. Its software is
compatible with a wide range of hardware e.g., Pixhawk, CUAV v5 Plus,
OpenPilot Revolution, etc. It has also been used for research purposes in a
large number of experiments such as in [33], [34], and [17].

(a) Airspeed Indicator (ASI)

(b) Altimeter

(c) Turn Coordinator

Figure 11: Three important flight instruments of GCS-CDS

For the second experiment, we use an industrial case study representing the
CDS of GCS (i.e., GCS-CDS). The GCS-CDS contains three important flight
instruments as shown in Fig. 11. The first flight instrument represents an
airspeed indicator (ASI). Fig. 11(a) shows an ASI displaying the airspeed
value of $\approx$0 knots. The second flight instrument represents an
altimeter. Fig. 11(b) shows an altimeter indicating the altitude value of
$\approx$1660 feet above sea level (ASL). The third flight instrument
represents a turn coordinator that is used to display the current turning
direction and the bank angle of the UAV. Fig. 11(c) shows a turn coordinator
displaying the turning direction towards the right and the bank angle of
$\approx$3 degrees.

###  5.3 Experiments Setup

The modeling statistics of both case studies are shown in Table 1. For the
first experiment using ArduCopter, we use the proposed UAS profile to model
the domain and behavior of the ArduCopter. The ArduCopter domain model
consists of six domain classes and 23 properties. The ArduCopter behavioral
model comprises 12 flight states, 23 events, and 43 transitions. The domain
model is shown in Fig. 6 and the flight behavioral model is shown in Fig. 7
and Fig. 8. The total number of applied stereotypes is 66. The expected
behavior of the quadcopter is specified in the form of OCL constraints that
are 48 in number (including general and state invariants). An excerpt from OCL
constraints is presented in Listing 1. The domain model and OCL constraints
are modeled by analyzing the source code of ArduCopter. We use the ArduPilot
software-in-the-loop simulator that is built using the binaries of the same
version of the ArduCopter source code.

For the second experiment using GCS-CDS, we use the UAS profile and CDS
testing profile proposed in previous work [10]. The UAS profile is used to
model the flight behavior of a UAV and the CDS testing profile to model the
domain of the GCS-CDS. The GCS-CDS domain model consists of 41 classes and 216
properties. The ArduCopter behavioral model comprises 14 flight states, 25
events, and 45 transitions. The total number of applied stereotypes is 40. The
expected behavior of GCS-CDS is specified in the form of OCL constraints that
are 30 in number.

Table 1: Modeling statistics of both case studies Modeling Artifact | ArduCopter | GCS-CDS
---|---|---
UAV model classes | 6 | 41
UAV model properties | 23 | 216
State machine states | 12 | 14
State machine events | 23 | 25
State machine transitions | 43 | 45
UAS profile applied stereotypes | 66 | 40
OCL constraints | 48 | 30
Table 2: Hyperparameter values used for both experiments Hyperparameter | Value
---|---
Learning rate | 0.001
Batch size | 128
Replay memory size | 1024
Target update | 10
Gamma ($\gamma$) | 0.999
$\epsilon$-start | 1
$\epsilon$-end | 0.01
$\epsilon$-decay | 100
Training episodes | 1000
Evaluation episodes | 100

###  5.4 Experiments Execution

For the training phase, we use hyperparameters presented in Table 2. Since
there are no standard guidelines for selecting hyperparameters for training,
we select these hyperparameters based on the various experimental trials with
different values of parameters and their combinations. We execute AITester and
Random to analyze the training progress considering the randomness factor.
Both AITester and Random approaches are executed on two machines with the same
specifications, i.e., core i7 3.6 GHz processor, 64 GB RAM, 1 TB hard drive,
NVIDIA GeForce®RTX 2080 Ti graphics card, and Windows 10 64-bit operating
system. We set the number of episodes to 1000 for AITester and the same number
of iterations for Random. At the end of the training session, a trained model
for AITester is stored.

For the first experiment with ArduCopter, we used the trained AITester and ran
it 100 times (Table 2). We also ran Random 100 times for the comparison. We
use random as a baseline because the available approaches in literature either
require manual test scenario creation, focus on testing a specific subsystem
of UAV autopilot, or do not provide public tool support.  Moreover, avionics
testers commonly execute UAV flight scenarios with random actions and data
using a flight simulator, a practice similar to random testing aimed at
identifying system faults. Alternatively, if the testing focus is on
uncovering security vulnerabilities, fuzz testing could serve as a comparative
approach. While fuzz testing is security-focused [35], aiming to identify
vulnerabilities through invalid or unexpected inputs, our evaluation does not
aim for security testing of the UAS.

For the second experiment using GCS-CDS, we compare AITester with the baseline
model-based CDS testing approach [10]. For the CDS testing approach, we
repeated the experiment procedure mentioned in [10]. This experiment was
executed on two machines with the same specifications used in the first
experiment. We set the number of episodes to 100 (Table 2) for AITester and
the same number of paths are used for the CDS testing approach. The overall
execution time for each case study, across 100 episodes, was approximately 300
minutes. It is important to note that the execution time varies based on the
machine specifications and the flight simulator being used. While we
experimented with ArduPilot – a widely used open-source simulator – AITester
is not tied to any specific simulator. It is adaptable and can seamlessly
integrate with a faster simulator.  To make our experimental results
reproducible, we provide the toolset with the experiment setup and execution
information at an online repository1.

###  5.5 Experiments Results and Analysis

To analyze the training progress, we calculated the moving average reward for
all episodes. The moving average reward (MAR) for _N_ episodes (MAR_N) is
calculated using Equation 4, where _n_ is the number of reward values to
consider in one window of the moving average. The overall MAR values over
total episodes can be calculated using Equation 5 for each new average window
of _n_ reward points. In this equation, $r_{n+1}$ is the new reward value and
the $r_{n-N+1}$ is the oldest value in the previous episode window.

$MAR_{N}=\frac{1}{N}\sum_{i=N-n+1}^{N}r_{i}$ (4)

$MAR=MAR_{N,prev}+\frac{1}{N}\left(r_{n+1}-r_{n-N+1}\right)$ (5)

Fig. 12 presents a comparison between AITester and Random based on the MAR for
1000 training episodes. The figure demonstrates MAR values (up to
$\approx$650) that are computed over 1000 episodes using a larger window size
[36], i.e., $n=350$.  It can be observed that the MAR of AITester is high in
most of the cases as compared to Random. MAR of AITester is greater than 50 in
a majority of the episodes and in some cases, it is higher than 100, whereas
Random maintains MAR $\approx$ 45 in most of the episodes. The MAR of AITester
and Random is close in some cases. This happens when the AITester tries to
explore (select random actions) the environment to create a balance between
exploration and exploitation. From the training outcomes, it can be seen that
AITester learns to perform the correct sequence of actions while violating the
expected behavior of the SUT.

Figure 12: Analysis of moving average reward for AITester in comparison to
Random

To answer RQ1, we perform a comparison based on the expected behavior
violations in the form of OCL constraints. AITester utilizes UAV flight state
machines to explore different paths in a state machine. An important
consideration is the exploration of diverse state machine paths during
testing. To analyze the diverse exploration of UAV flight states, we use path
diversity as a metric that is used in a similar context and different domains,
e.g., for Web [37]. For RQ2, we analyze the statistically significant
difference between the two approaches based on path diversity. We calculate
path diversity using the method defined for state graphs [37]).

To analyze the results for RQ2 and perform a statistical comparison, we follow
the guidelines defined by Arcuri and Briand [38]. We used the Wilcoxon signed-
rank test with a significance level ($\alpha$) set at 0.05. The Wilcoxon
signed-rank test is a non-parametric statistical test utilized for comparing
two different sample sets. This test generates a _p-value_ which is used to
determine whether the difference between two samples is of statistical
significance. A smaller _p-value_ (normally $<=\alpha$) suggests that there is
a statistically significant difference between the two samples. A larger _p-
value_ (normally $>\alpha$) indicates that there is no statistically
significant difference between the two samples. To measure the magnitude of
the difference, particularly to determine which sample is better, we utilized
Cliff’s Delta [39]. Cliff’s Delta is a non-parametric effect size measure that
quantifies the degree of difference between two samples. A value of 0
signifies that the two samples are identical. A positive value (up to +1)
suggests that the first sample is better than the second sample. Conversely, a
negative value (down to -1) indicates that the second sample is better than
the first sample.  The discussion on experiment results corresponding to each
research question is given below.

Table 3: Comparison results from 100 episodes of the AITester approach and Random Testing baseline for ArduCopter case study |  AITester Approach | Random Testing
---|---|---
States |  OCL Violations |  Unique OCL Violations |  OCL Violations |  Unique OCL Violations
AltitudeHold |  6964 |  4 |  4646 |  3
PositionHold |  1780 |  4 |  1328 |  4
Climb |  3820 |  4 |  2232 |  3
Decent |  19140 |  4 |  2942 |  1
Landing |  956 |  4 |  362 |  2
Loiter |  872 |  4 |  374 |  2
Takeoff |  696 |  4 |  690 |  4
Total |  34228 |  28 |  12574 |  19
Table 4: Comparison results from 100 episodes for ArduCopter and GCS-CDS based on diversity | ArduCopter | GCS-CDS
---|---|---
Statistical Tests |  AITester vs Random |  AITester vs CDST Approach
Wilcoxon signed-rank | 4.15298$\times 10^{-07}$ ($<\alpha$) |  2.93251$\times 10^{-20}$ ($<\alpha$)
Cliff’s Delta Effect Size | 1.0 $(Large)$ |  0.973144 $(Large)$

####  5.5.1 Results of Experiment on ArduCopter

To answer RQ1, the results of the comparison between AITester and Random
testing based on violations of the expected behavior of the SUT (OCL
violations) are presented in Table 3. The table shows the total and the unique
number of OCL violations corresponding to different UAV flight states. One
constraint may be violated multiple times when the UAV is in one state for a
particular time. For example, if a UAV takes five seconds to reach the
altitude specified for the Climb state and the environment changes are being
observed after 500 milliseconds, one constraint for Climb state can violate 10
times. Therefore, repeated violation of the same OCL constraint is considered
as one unique violation. To isolate unique OCL violations, we first grouped
recurring violations by flight state, followed by a manual analysis to
ascertain the uniqueness of each OCL violation. We keep track of all
violations as well as the unique number of violations. From the table, it can
be observed that the number of OCL violations done by the AITester in each
flight state is greater than that of Random. It can be seen that the total
number of OCL violations of AITester is approximately thrice as compared to
Random. For some states such as Descent, Loiter, and Landing the AITester
violates OCL constraints more than 50% as compared to Random. Moreover, for
some states like Climb and AltitudeHold, the difference in the number of
violations done by AITester is higher than that of Random. The difference
between AITester and Random is noteworthy in the unique number of OCL
violations. In some states (e.g., Climb and AltitudeHold,), the AITester
violates slightly more OCL constraints as compared to Random. For the
important flight states such as Descent, Loiter, and Landing, AITester
violates all state invariants specified for these states, whereas Random can
only violate a few. The total number of unique OCL violations done by AITester
is more than 60% as compared to Random. The results suggest that AITester
effectively violated several OCL constraints representing the expected
behavior of ArduCopter. These violations of OCL constraints may lead to faults
in ArduCopter. However, identifying the root causes of faults in such a system
requires faults reproduction and faults localization [40] which is not in the
scope of our approach.

RQ1 Result AITester is effective in generating diverse test scenarios that
lead to deviations from the expected behavior of ArduCopter.

To answer RQ2, the results of the comparison between AITester and Random based
on diversity are given in Table 4. The results of the Wilcoxon test suggest a
statistically significant difference (i.e., _p-value_ $<\alpha$) between
AITester and Random. To analyze which one is better, the results of the
Cliff’s Delta effect size measure show a large effect size. This implies that
the proposed approach (AITester) is better in comparison to Random. Therefore,
we can conclude that AITester outperforms Random in exploring diverse flight
paths.

RQ2 Result AITester is able to explore the diverse nature of flight paths from
the behavioral model of UAV during testing.

Table 5: Comparison results from 100 episodes of AITester and CDS Testing baseline approach for GCS-CDS case study | CDS Testing Approach |  AITester Approach
---|---|---
States |  Images |  OCL Violations |  Unique OCL Violations |  Images |  OCL Violations |  Unique OCL Violations
Armed |  441 |  30 |  1 |  171 |  60 |  3
AltitudeHold |  5744 |  1830 |  3 |  3107 |  3792 |  3
Approach |  866 |  838 |  2 |  55 |  92 |  3
Circle |  1902 |  2180 |  2 |  2521 |  3180 |  3
Climb |  7599 |  5550 |  3 |  6051 |  6644 |  3
Cruise |  5134 |  3620 |  3 |  3667 |  4078 |  3
Decent |  6405 |  3914 |  3 |  17819 |  20103 |  3
Landing |  963 |  960 |  3 |  62 |  61 |  3
Loiter |  1837 |  907 |  2 |  283 |  426 |  3
Takeoff |  788 |  405 |  2 |  63 |  11 |  3
Total |  31679 |  20234 |  24 |  33799 |  38447 |  30

(a) A fault in ASI during Cruise state

(b) A fault in Altimeter during Approach state

(c) A fault in Altimeter during Approach state

(d) A fault in Altimeter during Approach state

(e) A fault in Altimeter during Approach state

(f) A fault in Turn Coordinator during Circle state

(g) A fault in Turn Coordinator during Descent state

(h) A fault in Turn Coordinator during Loiter state

Figure 13: Faults identified in three CDS elements of GCS during different
flight states

####  5.5.2 Results of Experiment on GCS-CDS

Table 5 shows the results of the comparison between the AITester approach and
CDS testing baseline approach [10]. Fig. 13 presents diagrams created from the
results generated by AITester. These diagrams depict the faults identified in
three different flight instruments of GCS-CDS.

To answer RQ1, the results show that the total number of OCL violations
performed by the AITester is greater than that of the CDST approach. In most
of the flight states, the AITester performed a higher number of OCL violations
as compared to the CDST approach. The total number of unique OCL violations
done by the AITester is 30 whereas the CDST approach can perform 24 total
unique OCL violations. The analysis of flight state-wise unique OCL violations
shows that the number of violations by AITester is greater than the CDST
approach in some states, i.e., Armed, Approach, Circle, Loiter, and Takeoff.

The unique OCL violations performed by AITester lead to four different types
of faults (Fig. 13) in GCS-CDS under test. These four types of faults are
identified due to the violations of four OCL constraints shown in Listing 2.
Due to the violation of OCL constraint C1, the first type of fault is found in
ASI during Cruise flight state. According to the constraint C1, the airspeed
value must be between 50 and 100. Fig. 13(a) shows that the airspeed value
pointed by the ASI needle is 240 knots whereas the ASI tape shows a value of
$\approx$44 knots. This indicates an inconsistency between the ASI needle and
tape. Therefore, this type of fault is considered a CDS elements integration
fault.

Listing 2: Failed OCL constraints corresponding to four faults in GCS-CDS

⬇

C1:context ArduCopter inv: self.oclIsInState(Cruise) and
self.airspeedindicator.airspeed>=50 and self.airspeedindicator.airspeed<=100

C2:context ArduCopter inv: self.oclIsInState(Approach) and
self.altimeter.altitude>=584 and self.altimeter.altitude<604

C3:context ArduCopter inv: self.oclIsInState(Circle) and
(self.turncoordinator.roll>0 and self.turncoordinator.roll<=45) or
(self.turncoordinator.roll<0 and self.turncoordinator.roll>=-45)

C4:context ArduCopter inv: self.oclIsInState(Loiter) and
(self.turncoordinator.roll>0 and self.turncoordinator.roll<=45) or
(self.turncoordinator.roll<0 and self.turncoordinator.roll>=-45)

Due to the violation of OCL constraint C2, the second type of fault is found
in the Altimeter during Approach flight state. The constraint C2 states that
the altitude value must be between 584 and 604 feet during the Approach state.
At the same flight state and in different executions, the Altimeter has shown
incorrect values. The Altimeter shows the altitude value of $\approx$ 709 feet
in Fig. 13(b), $\approx$ 956 feet in Fig. 13(c), $\approx$ 1718 feet in Fig.
13(d), and $\approx$ 2850 feet in Fig. 13(e).

The third and fourth types of faults are identified in Turn Coordinator during
Circle and Loiter states due to the violation of OCL constraints C3 and C4
respectively. In the third type of fault, Turn Coordinator shows a bank angle
out of the range specified by OCL constraint C3. Due to the out-of-range
angle, the Turn Coordinator shows a high angle in Circle state as shown in
Fig. 13(f). The fourth type of fault is identified due to the violation of OCL
constraint C4. In this fault, while turning left during Decent (Fig. 13(g))
and Loiter (Fig. 13(h)) states, the Turn Coordinator shows values higher than
-46. It was analyzed that the UAV was not flipped upside down during both
states. This happened as the angle exceeded the maximum limit during Decent
and Loiter states.

All four types of faults identified by AITester conform to the four types of
faults found by the CDST approach [10] in GCS-CDS. Therefore, the proposed
approach (AITester) is effective in finding potential faults in the GCS-CDS
under test.

RQ1 Result AITester is effective in finding four potential faults in three
different flight instruments of GCS-CDS.

For the RQ2, the results of the comparison between the AITester and CDST
approach based on diversity are given in Table 4. The results of the Wilcoxon
test suggest that _p-value_ $<\alpha$, i.e., there is a statistically
significant difference between the AITester and CDST approach. To identify
which one is better, the results of the Cliff’s Delta effect size measure show
a large effect size. The results of statistical tests suggest that the
proposed approach (AITester) is better as compared to the CDST approach. That
is, AITester outperforms the CDST approach in exploring the diverse nature of
flight paths of UAV flight state machines.

RQ2 Result AITester is capable of exploring diverse flight paths from the UAV
flight behavioral model during the testing of GCS-CDS.

###  5.6 Threats to Validity

The possible threats to the validity of our experiment are discussed
individually in the following subsections.

####  5.6.1 External Validity Threat

The possible threats to the validity of our experiment are discussed
individually in the following subsections. The external threat to validity is
related to the generalization of the experiment results. To reduce the chance
of external validity threat, the experiment was performed using two primary
subsystems of UAS, an autopilot (ArduCopter) and the CDS of GCS (GCS-CDS).
ArduCopter is a good representative open-source autopilot system that is
widely used in research and development such as in [34] and [17]. The
ArduCopter supports a wide variety of multi-copter UAVs including a
helicopter, tri-copter, quad-copter, hexacopter, etc. Moreover, this is also
compatible with a wide range of hardware e.g., Pixhawk, CUAV v5 Plus,
OpenPilot Revolution, etc. The GCS-CDS is an industrial case study containing
widely used CDS elements. The results of our experiment may not be
generalizable for all subsystems of UAS, however, this threat typically exists
in every experiment.

####  5.6.2 Internal Validity Threat

The parameter tuning of the training algorithm presents an internal threat to
the validity of our experiment. There are no well-defined standard guidelines
for selecting the hyperparameters of the DRL algorithm. We followed the common
practice of selecting hyperparameters to minimize the internal validity
threat. We performed several experimental trials with different values of
parameters and their combinations. At the end of these trials, we selected the
best-suited hyperparameters for our experiment. To reduce the chances of
construct validity threat, we used the same stopping criterion for our
approach (AITester) and Random algorithm. That is, we execute AITester for
1000 episodes and Random for 1000 iterations during training. For the
evaluation phase of both experiments, we repeated each approach 100 times
which is a higher number as compared to traditional DRL evaluation procedure
[25]. In the experiment with GCS-CDS, we configured and executed test paths
according to the method defined in previous work [10].

####  5.6.3 Construct Validity Threat

Construct validity threat occurs when the relationship between cause and
effect cannot be determined. To mitigate the possibility of this threat, we
analyze the experimental results according to the guidelines provided by
Arcuri and Briand [38]. We examine the experimental results based on their
statistical significance and also look at the effect size. Specifically, we
used the Wilcoxon signed-rank test for statistical significance and the
Cliff’s Delta effect size measure [39] to analyze the effect size.

####  5.6.4 Conclusion Validity Threat

The threat to conclusion validity is associated with the effect of treatments
on the outcomes. To reduce the chances of this threat, we use the moving
average reward (MAR) metric to analyze the reward during the training session.
Our approach utilizes UAV flight state machines to explore different flight
behaviors. To analyze the diverse exploration of UAV flight states, we use
path diversity as a metric that is used in a similar context and different
domains, e.g., for Web [37]. We also used the unique number of OCL constraint
violations as a metric to compare AITester with the CDST approach. For a fair
comparison, this metric is common between this work and the previous work
[10].

##  6 Related Works

This section presents a discussion of the works related to our approach
including the works targeting UAV testing, autopilot system testing, the
application of reinforcement learning to fly the UAV, and the studies related
to the reinforcement learning-based testing of the UAV.

###  6.1 UAV Testing

In the literature related to the testing of unmanned aerial vehicles (UAV),
Johnson and Fontaine [41] developed a hardware and software-in-the-loop
platform to carry out flight testing of small UAVs. For this purpose, test
scenarios are required to be created manually, however, the test execution is
automated. Our approach automates the whole testing process. Sorton and
Hammaker [42] manually created fifteen test scenarios for testing UAV in the
hardware and software-in-the-loop simulation. Shore and Bodson [43] manually
developed different flight tests to evaluate the proposed algorithm for the
control and identification of configuration parameters of the UAV. Both the
above-mentioned techniques use manually written and limited set of flight
tests, however, our approach can automatically generate and execute flight
test scenarios. Jung and Tsiotras [3] created a hardware-in-the-loop (HIL)
simulation environment that can be used to develop and design flight tests for
UAV autopilot. The proposed HIL simulation environment supports the manual
creation of test scenarios, whereas our approach can automatically generate
test scenarios. Keith and Paul [44] developed a testing environment to test
UAV missions in a real-world context. Our approach does not target the testing
of UAV missions. How et al. [4] developed a real-time test environment that
can incorporate multiple autonomous (aerial and ground) vehicles. To use the
test environment, the flight test scenarios need to be created manually.

Mullins et al. [45] proposed an approach to generate challenging test
scenarios for testing autonomous vehicles. The approach utilizes several
manually designed test scenarios, executes in simulation, and searches for the
critical test scenarios based on different boundary conditions. The main
difference is that our approach can automatically generate and execute the
test scenarios, and does not require base test cases to generate new test
cases. Schulte and Spencer [46] presented a model-based approach for UAVs that
identifies faults, and sources of faults, and performs action accordingly
during autonomous flight. To evaluate the proposed approach, a set of manually
designed flight tests are used. Our approach targets automated system-level
testing but does not perform onboard diagnostics during the UAV flight. Ma et
al. [47] proposed an uncertainty-based modeling framework to test the self-
healing part of cyber-physical systems. The proposed approach was used to test
the self-healing behavior of the UAV, whereas our approach automates the
system-level testing of the UAV.

Ulbig et al. [48] proposed an approach and various unit and system-level test
scenarios for the testing of the Terrain Awareness and Warning System of the
UAV. Our approach automatically generates and executes test scenarios. Sarkar
et al. [49] proposed an approach and tool to automatically perform the flight
testing of the UAV. The approach utilizes supervised learning to train a model
that can perform different flight operations. For this purpose, the UAV flight
data is captured using a camera, and the data filtering and labeling are
carried out manually. Since the approach highly depends on the flight data,
the inconsistency in data and labeling errors may lead to unpredictable flight
testing. Moreover, the test oracle is also manual. In contrast, our approach
does not rely on flight data and automates the whole testing process. De et
al. [50] manually designed various flight test scenarios for testing the
vertical takeoff and landing behavior of the UAV carrying different payloads.
Rizk et al. [51] developed a GNSS-based emulator to facilitate UAV testing
without flying. The emulator is required to be configured manually for testing
the UAV. Wu et al. [5] manually created various test scenarios for testing
multi-rotor UAVs under different wind conditions and various flight modes
(e.g., hover and flip). Yang et al. [52] manually developed test cases to test
the acoustic behavior of the UAV. Scanavino et al. [6] created a test
environment to evaluate the performance of the UAV under different climate
conditions. To use the testing environment, the flight test scenarios need to
be developed manually. Wang et al. [40] performed an exploratory study to
identify and classify bugs in the source code of ArduPilot and PX4 autopilot
software. In recent work, Sorbo et al. [53] conducted an empirical study to
evaluate machine learning techniques for ensuring the safety of UAVs. A
classification of potential sources of risks and accidents is presented based
on the findings. In comparison to the above-mentioned works, our approach can
automatically perform the system-level testing of the UAS.

Several approaches are available targeting different aspects such as tests
generation from flight logs [54], analysis of boundary functions’ role in UAVs
safety [55], faults identification using model checking [56], detecting errors
in the physical unit of UAS [57], and mutation-based testing of robots [58].
Furthermore, some studies are conducted focusing on characterizing UAVs
software faults such as [53, 59] and UAS autopilot faults [60]. Concerning
these works, we focus on automating system-level testing of UAS.

###  6.2 Autopilot Testing

To test the autopilot of an aircraft, Fleck T. [61] manually designed various
flight test scenarios considering the different ground and aerial situations
such as the high speed at low-level, fly upwards, and altitude hold. Wells
[62] presented various methods to test the flight control system of an
autopilot. The proposed test methods target flight and missile stability
control, and frequency response testing. Application and Flight Testing of an
Adaptive Autopilot on Precision Guided Munitions Sharma et al. [63] manually
developed nine flight tests to evaluate the performance of the proposed
autopilot to support guided munitions. Kantue P. [64] manually created various
flight test scenarios to test the autopilot system developed for missile
control. To test Piccolo II autopilot, Anderson et al. [65] performed
different flight tests using the manually designed test scenarios to check
different conditions such as GPS range and radio communication. Ambroziak and
Gosiewski [66] manually defined various flight tests to analyze the variation
in airspeed, altitude, and trajectories followed by the UAV. Erdos and Watkins
[67] manually created flight test scenarios to test the autopilot for the
roll, pitch, altitude, airspeed, and navigation performance. Hartley et al.
[68] manually developed seven flight test scenarios to analyze the waypoint
navigation of the autopilot. In all the above-mentioned works, the test
scenarios are created manually. Whereas our approach automatically performs
system-level testing of the UAS. He et al. [69] proposed an approach to
automatically generate test oracles for localizing faults in cyber-physical
systems. The empirical evaluation using ArduPilot shows that the proposed
approach generates better oracles as compared to humans. Afzal et al. [70]
presented a technique to create oracles for cyber-physical systems based on
historical telemetry data. The empirical evaluation using ArduPilot shows that
the proposed technique outperforms the oracle generation approach by He et al.
[69]. In comparison, our work focuses on the online testing of UAS using OCL-
based test oracles. Recently, Leach et al. [71] presented a framework for
assessing and handling the security situation during UAV operation. The
evaluation using ArduPilot and Red Team attacks shows that the proposed
framework can increase the dependability of autonomous systems. Our work
targets the system-level testing of UAS instead of the security testing of
UAVs.

###  6.3 Reinforcement Learning for UAV

Reinforcement learning has been widely applied to autonomously control the
flight of different types of UAVs. Since our approach applies reinforcement
learning for the automated system-level testing of the UAV, we discuss a few
works. Bagnell and Schneider [18] proposed a reinforcement learning-based
algorithm for the robust control of autonomous helicopters. Kim et al. [72]
used reinforcement learning for training the algorithm to autonomously fly the
helicopter and perform different flight maneuvers such as hover. Abbeel et al.
[73] proposed an approach that finds the flight dynamics of the helicopter
from the flight data and then uses reinforcement learning to learn optimal
flight dynamics and autonomously fly the helicopter for various modes (e.g.,
flip). Koch et al. [17] developed a reinforcement learning-based environment
for training an agent to control the attitude of a quadcopter. In recent work,
Yamagata et al. [74] applied deep reinforcement learning algorithms to temper
cyber-physical systems based on robustness.

###  6.4 Reinforcement Learning-based UAV Testing

A small amount of work has been done in the area of RL-based UAV testing.
Junell et al. [75] devised an approach using reinforcement learning to reduce
the number of trials during flight testing of a quadrotor. Siddiquee et al.
[76] designed a vision-based reinforcement learning agent to test the guidance
system of a quadcopter. Ma et al. [34] developed two algorithms based on
reinforcement learning to test the self-healing part of cyber-physical systems
under different uncertainties. The proposed algorithms were used to test the
self-healing behavior of the ArduCopter. Ritchie Lee [77] presented Adastress,
an approach for reinforcement learning-based stress testing of safety-critical
systems. Subsequently, Moss et al. [78] extended Adastress for the flight
trajectory planning system, incorporating the aspect of continuous decision-
making into the reinforcement learning approach. While these works primarily
focus on stress testing, our work mainly aims to automate system-level testing
of UAS. Yuning He [79] introduced SYSAI, an approach that leverages AI
techniques to automate various testing activities, such as property checking
and test generation, for complex systems within the Aerospace domain. In
comparison, our approach specifically targets UAS and employs reinforcement
learning to automate system-level testing. In recent work, Zhang et al. [80]
proposed a reinforcement learning-based approach for generating failure-
causing inputs. The empirical evaluation results show the success of the
proposed approach in failing the well-tested CPS. The main difference between
these works with our work is that our approach applies DRL to automate the
system-level testing of the UAV.

##  7 Limitations

In this section, we discuss the potential limitations of our study and outline
prospective avenues for subsequent research.

###  7.1 Models Correctness

Our work provides a comprehensive modeling methodology and guidelines for
avionics testers to model the SUT. Avionics testers need to allocate time to
prepare models and constraints. Furthermore, the correctness of these models
and constraints must be verified by the testers, which is a common method [81,
10]. Our approach operates under the assumption that the models and
constraints developed by avionics testers are accurate. Ensuring their
correctness is essential for gaining confidence in the testing outcomes. The
verification of models/constraints’ correctness requires a specialized
approach, however, this aspect falls outside the scope of this paper. Several
existing methods available for this purpose, such as [82, 83, 84], can be
utilized to check the correctness of models and constraints.

###  7.2 State Machine Learning

Our approach requires avionics testers to model the behavior of SUT in the
form of UML state machines. These state machines play a crucial role in test
scenario generation and execution by AITester. Therefore, they need to be
carefully developed by avionics testers, which require manual effort.
Alternatively, these state machines could potentially be learned during the
testing process [85]. This necessitates a future study to analyze the
effectiveness of state machines learned automatically versus those crafted by
testers.

###  7.3 Complex Constraints

The OCL constraints employed in our approach comprehensively encompass the
requisites for conducting system testing of UAS. Complex constraints such as
performance and probabilistic constraints are typically associated with
specific testing focuses like performance testing or uncertainty-based
testing. However, this diverges from the primary focus of our work.
Incorporating these types of testing focuses into our approach presents an
intriguing direction for future research.

###  7.4 Testing UAVs’ Swarm

The approach presented in this paper is a preliminary attempt to automate UAS
testing. Thus, it is designed with a focus on a single subsystem of UAS as a
SUT, such as a UAV or GCS at any given time. Our evaluation was also conducted
considering testing UAV autopilot and GCS-CDS each as separate SUTs. Testing
multiple UAS subsystems, such as a swarm of UAVs, is also a critical aspect of
UAS testing. However, without empirical evidence, it remains uncertain whether
our approach can be effectively applied to testing a swarm of UAVs or multiple
GCSs. Further research is required to analyze the potential applicability of
our approach to multiple UAS subsystems like UAVs’ swarm.

##  8 Conclusion

Automated system-level testing of the UAS is required to meet the testing
guidelines set by different international safety standards. The prevailing
industrial practice and the existing research works require manual effort in
testing any of the UAS subsystems. Manual testing of a UAS subsystem is a
tedious activity and only a handful of scenarios can be tested. In this paper,
we propose a novel approach to automate the system-level testing of the UAS.
The proposed approach utilizes MBT and AI techniques to create an AITester
that generates, executes, and evaluates test scenarios based on the
environmental context at runtime. We provide a toolset to support testing
automation and to encourage further research and development. We perform two
experiments to evaluate AITester using two different subsystems of the UAS.
The first experiment is performed using the ArduCopter autopilot system and
the second experiment is performed using an industrial case study of GCS-CDS.
The results show that the AITester is effective in creating diverse test
scenarios that lead to deviations from the expected behavior of the UAV
autopilot and in revealing potential flaws in the GCS-CDS.

The approach presented in this paper is a fundamental step towards automating
system-level testing of UAS. Our approach generates and executes test
scenarios on the fly to find deviations from the expected behavior of UAS SUT.
While the test scenarios obtained through this process can assist avionics
testers in identifying faults during the initial development phases of UAS,
they are not designed to qualify for test certifications specified by
international safety standards. Furthermore, test certification qualifications
are dependent on the practices of the avionics industry, specifically the
safety standards they adhere to.  In the future, we aim to concentrate on
potential research directions, such as ensuring model correctness, learning
state machines automatically, handling complex constraints, automating fault
localization, and testing UAV swarms.

##  9 Acknowledgments

This research work is supported through a research grant titled ‘Establishment
of National Center of Robotics and Automation (NCRA)’ by Higher Education
Commission (HEC) Pakistan.

##  Appendix A Abbreviations

Table 6 provides a comprehensive list of all the abbreviations used in this
paper, along with their full forms.

Table 6: List of abbreviations with their corresponding full forms Abbreviation | Full form
---|---
UAS | Unmanned Aerial System
UAV | Unmanned Aerial Vehicle
GCS | Ground Control Station
CDS | Cockpit Display Systems
CPS | Cyber–Physical System
HIL | Hardware-in-the-Loop
SIL | Software-in-the-Loop
PIL | Pilot-in-the-Loop
ISR | Intelligence, Surveillance and Reconnaissance
SUT | System Under Test
MBT | Model-Based Testing
UML | Unified Modeling Language
OCL | Object Constraint Language
EMF | Eclipse Modeling Framework
RL | Reinforcement Learning
DRL | Deep Reinforcement Learning
MDP | Markov Decision Process
ANN | Artificial Neural Network
RNN | Recurrent Neural Network
CNN | Convolutional Neural Network
DQN | Deep Q-Network
LSTM | Long Short-Term Memory
API | Application Programming Interface
ASI | Airspeed Indicator
ASL | Above Sea Level
MAR | Moving Average Reward

## References

  *   * Ferrell and Ferrell [2017] Ferrell, T.K., Ferrell, U.D.: Rtca do-178c/eurocae ed-12c. Digital Avionics Handbook (2017)
  * Marques [2012] Marques, M.M.: Stanag 4586—standard interfaces of UAV control system (UCS) for nato UAV interoperability. NATO Standardization Agency: Afeite, Portugal, 14 (2012)
  * Jung and Tsiotras [2007] Jung, D., Tsiotras, P.: Modeling and hardware-in-the-loop simulation for a small unmanned aerial vehicle. In: AIAA Infotech@ Aerospace 2007 Conference and Exhibit, p. 2768 (2007). https://doi.org/10.2514/6.2007-2768
  * How et al. [2008] How, J.P., Behihke, B., Frank, A., Dale, D., Vian, J.: Real-time indoor autonomous vehicle test environment. IEEE Control Systems Magazine 28(2), 51–64 (2008) https://doi.org/10.1109/MCS.2007.914691
  * Wu et al. [2019] Wu, Y., Qi, L., Cheng, Z., Xiao, Y., Yang, Z., Liu, W., Li, X.: Testing method and anti-drift capability analysis of downwash airflow distribution characteristics of multi-rotor UAV in hover. In: 2019 ASABE Annual International Meeting, p. 1 (2019). American Society of Agricultural and Biological Engineers
  * Scanavino et al. [2019] Scanavino, M., Vilardi, A., Guglieri, G.: A new facility for UAV testing in climate-controlled environments. In: 2019 International Conference on Unmanned Aircraft Systems (ICUAS), pp. 1436–1444 (2019). https://doi.org/10.1109/ICUAS.2019.8798066 . IEEE
  * OMG [2014] OMG: Object constraint language specification v2.4. Object Management Group Inc. (2014)
  * Sartaj [2021] Sartaj, H.: Automated approach for system-level testing of unmanned aerial systems. 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 1069–1073 (2021). https://doi.org/10.1109/ASE51524.2021.9678902 . IEEE
  * Sadraey [2017] Sadraey, M.: Unmanned aircraft design: A review of fundamentals. Synthesis Lectures on Mechanical Engineering 1(2), 193 (2017) https://doi.org/10.1007/978-3-031-79582-4
  * Sartaj et al. [2021] Sartaj, H., Iqbal, M.Z., Khan, M.U.: Testing cockpit display systems of aircraft using a model-based approach. Software and Systems Modeling 20(6), 1977–2002 (2021) https://doi.org/10.1007/s10270-020-00844-z
  * Petrenko et al. [2012] Petrenko, A., Silva Simão, A., Maldonado, J.C.: Model-based testing of software and systems: recent advances and challenges. Int. J. Softw. Tools Technol. Transf. 14(4), 383–386 (2012) https://doi.org/10.1007/S10009-012-0240-3
  * Utting and Legeard [2010] Utting, M., Legeard, B.: Practical Model-based Testing: a Tools Approach. Elsevier, United States (2010)
  * OMG [2017] OMG: Uml. unified modeling language specification, version 2.5.1. Object Management Group Inc. (2017)
  * Khan et al. [2019] Khan, M.U., Sartaj, H., Iqbal, M.Z., Usman, M., Arshad, N.: AspectOCL: using aspects to ease maintenance of evolving constraint specification. Empirical Software Engineering 24(4), 2674–2724 (2019) https://doi.org/10.1007/s10664-019-09717-6
  * Sutton and Barto [2018] Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT press, United States (2018)
  * Li [2019] Li, Y.: Reinforcement learning applications. arXiv preprint arXiv:1908.06973 (2019)
  * Koch et al. [2019] Koch, W., Mancuso, R., West, R., Bestavros, A.: Reinforcement learning for UAV attitude control. ACM Transactions on Cyber-Physical Systems 3(2), 1–21 (2019) https://doi.org/10.1145/3301273
  * Bagnell and Schneider [2001] Bagnell, J.A., Schneider, J.G.: Autonomous helicopter control using reinforcement learning policy search methods. In: Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No. 01CH37164), vol. 2, pp. 1615–1620 (2001). https://doi.org/10.1109/ROBOT.2001.932842 . IEEE
  * Aggarwal [2018] Aggarwal, C.C.: Neural Networks and Deep Learning. Springer, Cham, Switzerland (2018)
  * Vierhauser et al. [2023] Vierhauser, M., Wohlrab, R., Stadler, M., Cleland-Huang, J.: Amon: A domain-specific language and framework for adaptive monitoring of cyber–physical systems. Journal of Systems and Software 195, 111507 (2023) https://doi.org/10.1016/j.jss.2022.111507
  * Araujo et al. [2022] Araujo, H., Mousavi, M.R., Varshosaz, M.: Testing, validation, and verification of robotic and autonomous systems: A systematic review. ACM Transactions on Software Engineering and Methodology (2022) https://doi.org/%****␣uast-ase.bbl␣Line␣325␣****10.1145/3542945
  * Aloui et al. [2022] Aloui, K., Hammadi, M., Guizani, A., Haddar, M., Soriano, T.: A new sysml model for UAV swarm modeling: Uavswarmml. In: 2022 IEEE International Systems Conference (SysCon), pp. 1–8 (2022). https://doi.org/10.1109/SysCon53536.2022.9773922 . IEEE
  * Russell and Norvig [2009] Russell, S.J., Norvig, P.: Artificial Intelligence: A Modern Approach. Prentice Hall, United States (2009)
  * Binder [2000] Binder, R.: Testing Object-oriented Systems: Models, Patterns, and Tools. Addison-Wesley Professional, United States (2000)
  * Mnih et al. [2015] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015) https://doi.org/10.1038/nature14236
  * Maas et al. [2013] Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: Proc. Icml, vol. 30, p. 3 (2013)
  * Mnih et al. [2013] Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)
  * Jaderberg et al. [2019] Jaderberg, M., Czarnecki, W.M., Dunning, I., Marris, L., Lever, G., Castaneda, A.G., Beattie, C., Rabinowitz, N.C., Morcos, A.S., Ruderman, A., et al.: Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science 364(6443), 859–865 (2019)
  * Sartaj et al. [2019] Sartaj, H., Iqbal, M.Z., Jilani, A.A.A., Khan, M.U.: A search-based approach to generate MC/DC test data for OCL constraints. Search-Based Software Engineering: 11th International Symposium, SSBSE 2019, Tallinn, Estonia, August 31–September 1, 2019, Proceedings 11, pp. 105–120 (2019). https://doi.org/10.1007/978-3-030-27455-9_8 . Springer
  * Sartaj et al. [2024] Sartaj, H., Iqbal, M.Z., Jilani, A.A.A., Khan, M.U.: Efficient test data generation for MC/DC with OCL and search. arXiv preprint arXiv:2401.03469 (2024)
  * Paszke et al. [2017] Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in pytorch (2017)
  * Kingma and Ba [2014] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * Lee and Kim [2017] Lee, H., Kim, H.J.: Trajectory tracking control of multirotors from modelling to experiments: A survey. International Journal of Control, Automation and Systems 15(1), 281–292 (2017) https://doi.org/10.1007/s12555-015-0289-3
  * Ma et al. [2019] Ma, T., Ali, S., Yue, T., Elaasar, M.: Testing self-healing cyber-physical systems under uncertainty: a fragility-oriented approach. Software Quality Journal, 1–35 (2019) https://doi.org/%****␣uast-ase.bbl␣Line␣525␣****10.1007/s11219-018-9437-3
  * Zhu et al. [2022] Zhu, X., Wen, S., Camtepe, S., Xiang, Y.: Fuzzing: a survey for roadmap. ACM Computing Surveys (CSUR) 54(11s), 1–36 (2022) https://doi.org/10.1145/3512345
  * Ermshaus et al. [2023] Ermshaus, A., Schäfer, P., Leser, U.: Window size selection in unsupervised time series analytics: A review and benchmark. In: International Workshop on Advanced Analytics and Learning on Temporal Data, pp. 83–101 (2023). https://doi.org/10.1007/978-3-031-24378-3_6 . Springer
  * Fard and Mesbah [2013] Fard, A.M., Mesbah, A.: Feedback-directed exploration of web applications to derive test models. In: ISSRE, vol. 13, pp. 278–287 (2013). https://doi.org/10.1109/ISSRE.2013.6698880 . IEEE
  * Arcuri and Briand [2011] Arcuri, A., Briand, L.: A practical guide for using statistical tests to assess randomized algorithms in software engineering. In: Software Engineering (ICSE), 2011 33rd International Conference On, pp. 1–10 (2011). https://doi.org/10.1145/1985793.1985795 . IEEE
  * Cliff [1993] Cliff, N.: Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological bulletin 114(3), 494 (1993)
  * Wang et al. [2021] Wang, D., Li, S., Xiao, G., Liu, Y., Sui, Y.: An exploratory study of autopilot software bugs in unmanned aerial vehicles. In: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 20–31 (2021). https://doi.org/10.1145/3468264.3468559 . ACM
  * Johnson and Fontaine [2001] Johnson, E.N., Fontaine, S.: Use of flight simulation to complement flight testing of low-cost UAVs. In: AIAA Modeling and Simulation Technologies Conference (2001). https://doi.org/10.2514/6.2001-4059
  * Sorton and Hammaker [2005] Sorton, E., Hammaker, S.: Simulated flight testing of an autonomous unmanned aerial vehicle using flightgear. In: Infotech@ Aerospace, p. 7083 (2005). https://doi.org/10.2514/6.2005-7083
  * Shore and Bodson [2005] Shore, D., Bodson, M.: Flight testing of a reconfigurable control system on an unmanned aircraft. Journal of Guidance, Control, and Dynamics 28(4), 698–707 (2005) https://doi.org/10.2514/1.8178
  * Sevcik and Oh [2008] Sevcik, K., Oh, P.: Testing unmanned aerial vehicle missions in a scaled environment. In: Unmanned Aircraft Systems, pp. 297–305. Springer, Netherlands (2008). https://doi.org/10.1007/978-1-4020-9137-7_17
  * Mullins et al. [2017] Mullins, G.E., Stankiewicz, P.G., Gupta, S.K.: Automated generation of diverse and challenging scenarios for test and evaluation of autonomous vehicles. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1443–1450 (2017). https://doi.org/10.1109/ICRA.2017.7989173 . IEEE
  * Schulte and Spencer [2018] Schulte, P., Spencer, D.: On-board model-based fault diagnosis for autonomous proximity operations. In: 69th International Astronautical Congress, Bremen, Germany (2018)
  * Ma et al. [2018] Ma, T., Ali, S., Yue, T.: Modeling foundations for executable model-based testing of self-healing cyber-physical systems. Software & Systems Modeling, 1–31 (2018) https://doi.org/10.1007/s10270-018-00703-y
  * Ulbig et al. [2019] Ulbig, P., Müller, D., Torens, C., Insaurralde, C.C., Stripf, T., Durak, U.: Flight simulator-based verification for model-based avionics applications on multi-core targets. In: AIAA Scitech 2019 Forum, p. 1976 (2019). https://doi.org/%****␣uast-ase.bbl␣Line␣725␣****10.2514/6.2019-1976
  * Sarkar et al. [2019] Sarkar, M., Homaifar, A., Erol, B.A., Behniapoor, M., Tunstel, E.: PIE: a tool for data-driven autonomous UAV flight testing. Journal of Intelligent & Robotic Systems, 1–18 (2019) https://doi.org/10.1007/s10846-019-01078-y
  * De Wagter et al. [2019] De Wagter, C., Remes, B., Ruisink, R., Van Tienen, F., Horst, E.: Design and testing of a vertical take-off and landing UAV optimized for carrying a hydrogen fuel-cell with pressure tank. In: International Micro Air Vehicle Conference and Competition (IMAV)(Madrid) (2019)
  * Rizk et al. [2019] Rizk, M., Mroue, A., Ali, K.A., Charara, J.: Development and implementation of an embedded low-cost gnss receiver emulator for UAV testing. In: 2019 26th IEEE International Conference on Electronics, Circuits and Systems (ICECS), pp. 101–102 (2019). https://doi.org/10.1109/ICECS46596.2019.8964928 . IEEE
  * Yang et al. [2019] Yang, M., Darrah, D., Eppler, J., Liu, W., Anemaat, W.A.: Small UAV acoustic design, analysis and testing. In: AIAA Scitech 2019 Forum, p. 1073 (2019). https://doi.org/10.2514/6.2019-1073
  * Sorbo et al. [2022] Sorbo, A.D., Zampetti, F., Visaggio, C.A., Penta, M.D., Panichella, S.: Automated identification and qualitative characterization of safety concerns reported in UAV software platforms. ACM Transactions on Software Engineering and Methodology (2022) https://doi.org/10.1145/3564821
  * Khatiri et al. [2023] Khatiri, S., Panichella, S., Tonella, P.: Simulation-based test case generation for unmanned aerial vehicles in the neighborhood of real flights. In: 16th IEEE International Conference on Software Testing, Verification and Validation (ICST), Dublin, Ireland, 16-20 April 2023, pp. 281–292 (2023). https://doi.org/10.1109/ICST57152.2023.00034 . IEEE
  * Liang et al. [2021] Liang, X., Burns, J.H., Sanchez, J., Dantu, K., Ziarek, L., Liu, Y.D.: Understanding bounding functions in safety-critical UAV software. In: 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pp. 1311–1322 (2021). https://doi.org/10.1109/ICSE43902.2021.00119 . IEEE
  * Taylor et al. [2021] Taylor, M., Chen, H., Qin, F., Stewart, C.: Avis: In-situ model checking for unmanned aerial vehicles. In: 2021 51st Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), pp. 471–483 (2021). https://doi.org/%****␣uast-ase.bbl␣Line␣850␣****10.1109/DSN48987.2021.00057 . IEEE
  * Taylor et al. [2022] Taylor, M., Aurand, J., Qin, F., Wang, X., Henry, B., Zhang, X.: Sa4u: Practical static analysis for unit type error detection. In: 37th IEEE/ACM International Conference on Automated Software Engineering, pp. 1–11 (2022). https://doi.org/10.1145/3551349.3556937
  * Cavalcanti et al. [2019] Cavalcanti, A., Baxter, J., Hierons, R.M., Lefticaru, R.: Testing robots using csp. In: Tests and Proofs: 13th International Conference, TAP 2019, Held as Part of the Third World Congress on Formal Methods 2019, Porto, Portugal, October 9–11, 2019, Proceedings 13, pp. 21–38 (2019). https://doi.org/10.1007/978-3-030-31157-5_2 . Springer
  * Timperley et al. [2018] Timperley, C.S., Afzal, A., Katz, D.S., Hernandez, J.M., Le Goues, C.: Crashing simulated planes is cheap: Can simulation detect robotics bugs early? In: 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST), pp. 331–342 (2018). https://doi.org/%****␣uast-ase.bbl␣Line␣900␣****10.1109/ICST.2018.00040 . IEEE
  * Taylor et al. [2021] Taylor, M., Boubin, J., Chen, H., Stewart, C., Qin, F.: A study on software bugs in unmanned aircraft systems. In: 2021 International Conference on Unmanned Aircraft Systems (ICUAS), pp. 1439–1448 (2021). https://doi.org/10.1109/ICUAS51884.2021.9476844 . IEEE
  * Fleck [1983] Fleck, T.: Flight testing of the autopilot and terrain following radar system in the tornado aircraft. In: 2nd Flight Testing Conference, p. 2770 (1983). https://doi.org/10.2514/6.1983-2770
  * WELLS [1991] WELLS, B.: Tactical missile structural testing and model verification for autopilot design. In: Navigation and Control Conference, p. 2616 (1991). https://doi.org/10.2514/6.1991-2616
  * Sharma et al. [2006] Sharma, M., Lavretsky, E., Wise, K.: Application and flight testing of an adaptive autopilot on precision guided munitions. In: AIAA Guidance, Navigation, and Control Conference and Exhibit, p. 6568 (2006). https://doi.org/10.2514/6.2006-6568
  * Kantue [2017] Kantue, P.: Design and flight testing of a gain-scheduled autopilot based on reduced model constraints. IFAC-PapersOnLine 50(2), 187–192 (2017) https://doi.org/10.1016/j.ifacol.2017.12.034
  * Anderson et al. [2008] Anderson, N., Hagenauer, B., Erickson, R., Bhandari, S.: Flight-testing of a UAV aircraft for autonomous operation using piccolo ii autopilot. In: AIAA Atmospheric Flight Mechanics Conference and Exhibit, p. 6568 (2008). https://doi.org/10.2514/6.2008-6568
  * Ambroziak and Gosiewski [2013] Ambroziak, L., Gosiewski, Z.: Preliminary UAV autopilot integration and in-flight testing. In: Solid State Phenomena, vol. 198, pp. 232–237 (2013). https://doi.org/10.4028/www.scientific.net/SSP.198.232 . Trans Tech Publ
  * Erdos and Watkins [2008] Erdos, D., Watkins, S.E.: UAV autopilot integration and testing. In: 2008 IEEE Region 5 Conference, pp. 1–6 (2008). https://doi.org/10.1109/TPSD.2008.4562731 . IEEE
  * Hartley et al. [2013] Hartley, R.F., Hugon, F.-D.X., Anderson, R., Moncayo, H.: Development and flight testing of a model based autopilot library for a low cost unmanned aerial system. In: AIAA Guidance, Navigation, and Control (GNC) Conference, p. 5009 (2013). https://doi.org/10.2514/6.2013-5009
  * He et al. [2019] He, Z., Chen, Y., Huang, E., Wang, Q., Pei, Y., Yuan, H.: A system identification based oracle for control-cps software fault localization. In: 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 116–127 (2019). https://doi.org/10.1109/ICSE.2019.00029 . IEEE
  * Afzal et al. [2021] Afzal, A., Le Goues, C., Timperley, C.S.: Mithra: Anomaly detection as an oracle for cyberphysical systems. IEEE Transactions on Software Engineering (2021) https://doi.org/10.1109/TSE.2021.3120680
  * Leach et al. [2022] Leach, K., Timperley, C.S., Angstadt, K., Nguyen-Tuong, A., Hiser, J., Paulos, A., Pal, P., Hurley, P., Thomas, C., Davidson, J.W., et al.: Start: A framework for trusted and resilient autonomous vehicles (practical experience report). In: 2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE), pp. 73–84 (2022). https://doi.org/10.1109/ISSRE55969.2022.00018 . IEEE
  * Kim et al. [2004] Kim, H.J., Jordan, M.I., Sastry, S., Ng, A.Y.: Autonomous helicopter flight via reinforcement learning. In: Advances in Neural Information Processing Systems, pp. 799–806 (2004)
  * Abbeel et al. [2007] Abbeel, P., Coates, A., Quigley, M., Ng, A.Y.: An application of reinforcement learning to aerobatic helicopter flight. In: Advances in Neural Information Processing Systems, pp. 1–8 (2007)
  * Yamagata et al. [2020] Yamagata, Y., Liu, S., Akazaki, T., Duan, Y., Hao, J.: Falsification of cyber-physical systems using deep reinforcement learning. IEEE Transactions on Software Engineering 47(12), 2823–2840 (2020) https://doi.org/10.1109/TSE.2020.2969178
  * Junell et al. [2016] Junell, J., Mannucci, T., Zhou, Y., Van Kampen, E.-J.: Self-tuning gains of a quadrotor using a simple model for policy gradient reinforcement learning. In: AIAA Guidance, Navigation, and Control Conference, p. 1387 (2016). https://doi.org/10.2514/6.2016-1387
  * Siddiquee et al. [2019] Siddiquee, M., Junell, J., Van Kampen, E.-J.: Flight test of quadcopter guidance with vision-based reinforcement learning. In: AIAA Scitech 2019 Forum, p. 0142 (2019). https://doi.org/10.2514/6.2019-0142
  * Lee [2019] Lee, R.: Adastress: Adaptive stress testing and interpretable categorization for safety-critical systems. PhD thesis, Carnegie Mellon University (2019)
  * Moss et al. [2020] Moss, R.J., Lee, R., Visser, N., Hochwarth, J., Lopez, J.G., Kochenderfer, M.J.: Adaptive stress testing of trajectory predictions in flight management systems. In: 2020 AIAA/IEEE 39th Digital Avionics Systems Conference (DASC), pp. 1–10 (2020). https://doi.org/10.1109/DASC50938.2020.9256730 . IEEE
  * He [2022] He, Y.: System and safety analysis for autonomous center-line tracking with sysai. In: AIAA SCITECH 2022 Forum, p. 2399 (2022). https://doi.org/10.2514/6.2022-2399
  * Zhang et al. [2021] Zhang, S., Liu, S., Sun, J., Chen, Y., Huang, W., Liu, J., Liu, J., Hao, J.: Figcps: Effective failure-inducing input generation for cyber-physical systems with deep reinforcement learning. In: 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 555–567 (2021). https://doi.org/10.1109/ASE51524.2021.9678832 . IEEE
  * Iqbal et al. [2019] Iqbal, M.Z., Sartaj, H., Khan, M.U., Haq, F.U., Qaisar, I.: A model-based testing approach for cockpit display systems of avionics. In: 2019 ACM/IEEE 22nd International Conference on Model Driven Engineering Languages and Systems (MODELS), pp. 67–77 (2019). https://doi.org/10.1109/MODELS.2019.00-14 . IEEE
  * Balaban and Maraee [2013] Balaban, M., Maraee, A.: Simplification and correctness of uml class diagrams–focusing on multiplicity and aggregation/composition constraints. In: International Conference on Model Driven Engineering Languages and Systems, pp. 454–470 (2013). https://doi.org/10.1007/978-3-642-41533-3_28 . Springer