# Metamorphic Runtime Monitoring of Autonomous Driving Systems

Jon Ayerdi  Mondragon UniversityGoiru 2MondragonGipuzkoaSpain20500
<EMAIL_ADDRESS>,  Asier Iriarte  Mondragon UniversityGoiru
2MondragonGipuzkoaSpain20500<EMAIL_ADDRESS>,  Pablo Valle
Mondragon UniversityGoiru 2MondragonGipuzkoaSpain20500
<EMAIL_ADDRESS>,  Ibai Roman  Mondragon UniversityGoiru
2MondragonGipuzkoaSpain20500<EMAIL_ADDRESS>,  Miren Illarramendi
Mondragon UniversityGoiru 2MondragonGipuzkoaSpain20500
<EMAIL_ADDRESS>and  Aitor Arrieta  Mondragon UniversityGoiru
2MondragonGipuzkoaSpain20500<EMAIL_ADDRESS>
(2023)

###### Abstract.

Autonomous Driving Systems (ADSs) are complex Cyber-Physical Systems (CPSs)
that must ensure safety even in uncertain conditions. Modern ADSs often employ
Deep Neural Networks (DNNs), which may not produce correct results in every
possible driving scenario. Thus, an approach to estimate the confidence of an
ADS at runtime is necessary to prevent potentially dangerous situations. In
this paper we propose MarMot, an online monitoring approach for ADSs based on
Metamorphic Relations (MRs), which are properties of a system that hold among
multiple inputs and the corresponding outputs. Using domain-specific MRs,
MarMot estimates the uncertainty of the ADS at runtime, allowing the
identification of anomalous situations that are likely to cause a faulty
behavior of the ADS, such as driving off the road.

We perform an empirical assessment of MarMot with five different MRs, using a
small-scale ADS, two different circuits for training, and two additional
circuits for evaluation. Our evaluation encompasses the identification of both
external anomalies, e.g., fog, as well as internal anomalies, e.g., faulty
DNNs due to mislabeled training data. Our results show that MarMot can
identify 35% to 65% of the external anomalies and 77% to 100% of the internal
anomalies, outperforming both SelfOracle and Ensemble-based ADS monitoring
approaches.

††copyright: acmcopyright††journalyear: 2023††doi: XXXXXXX.XXXXXXX

##  1\. Introduction

In the last few years, the autonomy of vehicles is increasing, in part, thanks
to the recent advances of machine-learning based technologies. Deep Neural
Networks (DNNs) have enabled Autonomous Driving Systems (ADSs) to operate in
dynamic environments with little to no human intervention (Bojarski et al.,
2016). Supervised training is a particularly effective approach that allows
DNNs to effectively learn driving behaviors (such as the throttling or the
steering angle) from labeled datasets of camera images and the corresponding
expected driving behavior (Bojarski et al., 2016). However, to deploy ADSs in
the roads, these should be reliable, even in the presence of uncertain or
unforeseen scenarios. To increase the reliability of these systems, in the
last few years, significant effort has been devoted to research on automated
test case generation techniques that falsify safety properties of ADSs (e.g.,
find collisions, violate traffic laws) (Tian et al., 2018; Haq et al., 2022,
2023; Abdessalem et al., 2018b, a; Ben Abdessalem et al., 2016; Zhong et al.,
2022; Lu et al., 2022; Zhou et al., 2023; Sun et al., 2022; Scheuer et al.,
2023; Calò et al., 2020; Gambi et al., 2019b, a; Biagiola and Tonella, 2022).
However, the expensive execution of test cases (due to the use of simulation
frameworks), as well as the huge input space, makes it impossible to cover all
situations in which the ADS can misbehave.

To address this issue, existing works propose runtime monitors (also known as
supervisors or failure predictors) to assess the level of the DNN
dependability in operation (Henriksson et al., 2019; Hussain et al., 2022; Kim
et al., 2019; Stocco et al., 2020; Stocco and Tonella, 2022; Stocco et al.,
2022a; Wang et al., 2020; Xiao et al., 2021; Zhang et al., 2018). In the
context of ADSs, both, black-box as well as white-box approaches have been
proposed. As for black-box approaches, techniques like SelfOracle (Stocco et
al., 2020) or DeepGuard (Hussain et al., 2022) monitor the ADS by examining
its behavior in response to the input images of the system. In the context of
ADS, for black-box runtime monitoring techniques, one of the identified core
limitations (Stocco et al., 2022a) relates to the impossibility of handling
other failures than those caused due to data-driven bugs, e.g., failures
caused by corrupted images. Therefore, these techniques fail to detect faults
due to inadequate training or by bugs at the DNN model level (Stocco et al.,
2022a). To address this challenge, Stocco et al. (Stocco et al., 2022a)
proposed ThirdEye, a white-box ADS failure predictor that relies on attention
maps produced by explainable artificial intelligence (XAI) techniques.
Specifically, they propose three different summarization methods to compute a
confidence score from the attention map of the images, provided some extra
information previously extracted from the training dataset. Although this
technique is promising and overcomes the limitations of black-box techniques,
it also poses one core limitation that may prevent its adoption in practice:
its computational cost, and as a direct consequence, its execution time.
Specifically, the techniques employed to obtain the attention maps require
several invocations of the DNN, which may make this technique unfeasible
unless significant computational resources are available exclusively for the
ADS monitors. This may be a problem in the context of ADS that use resource-
constrained embedded devices.

To deal with the above-mentioned challenges, we propose MarMot, a simple, yet
effective and efficient runtime monitoring approach. MarMot leverages ideas
from metamorphic testing (Segura et al., 2016) to predict when the ADS will
fail, and it provides significant benefits over the state-of-the-art
approaches. Firstly, based on our experiments with a lane keeping case study
system, MarMot provides higher effectiveness in terms of detecting internal
(faulty DNN models) or external (unexpected inputs) anomalies which may cause
an incorrect behavior of the system. Secondly, our approach is capable of an
earlier detection of internal anomalies than other approaches, which is
crucial in cases where there is only a short time to react. Furthermore,
MarMot is also applicable for ADSs whose functionalities are not necessarily
controlled through DNNs, such as in the case of Apollo (Apollo, 2023). Lastly,
our approach only requires cheap image transformations and one additional
invocation of the ADS controller (either DNN or other), which would solve the
high computational cost of some white-box techniques (e.g., ThirdEye (Stocco
et al., 2022a)).

Our paper makes the following contributions:

  * •

Technique: We propose a black-box approach inspired on Metamorphic Testing to
estimate the confidence of DNN-based ADSs at runtime. We define several image-
based Metamorphic Relations (MRs) for this domain, and a process to evaluate
them at runtime. Our technique can be used to identify potentially unexpected
scenarios, either those derived from internal (e.g., improperly trained DNN)
or external (e.g., fog blocking the visibility of the camera) uncertainties.

  * •

Evaluation: Unlike previous studies, which use simulation environments in
their evaluation, we employ a physical (small-scale) ADS. This was considered
to reduce the validity threat related to the reality gap existing between
simulated and real environments (e.g., due to the textures of the images or
the fidelity of the simulation engines). With this physical vehicle, we carry
out an empirical study, showing MarMot’s benefits over the black-box approach
of SelfOracle (Stocco et al., 2020) as well as the white-box Ensemble
approach.

  * •

Dataset: We provide the dataset of images used to train our DNN-based driving
models, as well as the one used to evaluate our test oracles in this paper
(Ayerdi et al., 2023). Our dataset comprises images from a training track
provided by the vendor of the mobile robot (LeoRover, 2022), as well as images
from another training track and two additional evaluation tracks provided by
us. For the two evaluation tracks, our dataset also includes recordings of
out-of-bounds scenarios, caused by either internal (faulty DNN models) or
external (unexpected inputs) anomalies.

The rest of the paper is structured as follows: We provide basic background in
Section 2. We explain our approach in Section 3. We empirically evaluate our
approach in Section 4, by means of a physical small-scale autonomous driving
systems and different types of anomalies. We position our approach with the
current state-of-the-art in Section 5. Lastly, we conclude and discuss future
research avenues in Section 6.

##  2\. Background

###  2.1. Autonomous Driving Systems

Modern Autonomous Driving Systems (ADSs) leverage machine-learning
technologies, such as Deep Neural Networks (DNN), to analyze the inputs (e.g.,
camera images, LIDAR sensors, GPS) and drive the vehicle in real time with
various degrees of autonomy (Bojarski et al., 2016). Typically, DNNs learn
tasks such as lane keeping in a supervised manner, i.e., by generalizing from
a dataset of samples labeled by humans, which demonstrate correct driving
behaviors (Bojarski et al., 2016).

However, DNN-based driving systems are currently unlikely to generalize
correctly for all possible driving conditions that can be encountered in a
real road. Thus, in order to ensure safety, it is paramount to monitor the
(estimated) confidence of the DNN, so that an appropriate healing strategy can
be applied in low-confidence scenarios. Healing strategies to leave the
vehicle in a safe state may include passing control to a human driver,
reducing the speed, or completely stopping the vehicle (Hussain et al., 2022).

###  2.2. Monitoring Techniques for ADSs

An approach to monitor a DNN’s behaviour and quantify its confidence level
must consider several causes of uncertainty. On the one hand, this includes
external factors, such as adverse environmental conditions like fog or heavy
rain (Tian et al., 2018; Stocco et al., 2020; Hussain et al., 2022). On the
other hand, internal factors such as poorly trained DNNs (e.g., wrong
hyperparameter values, incorrect labels in the training dataset) may also need
to be considered (Humbatova et al., 2020; Stocco et al., 2022a).

Black-box techniques estimate the confidence of the DNN based solely on its
inputs (images) and outputs (steering angle, throttle, etc.), and possibly
also the training data (Stocco et al., 2020). The main advantage of these
techniques is that they are independent from the DNN architecture, and often
even generalizable to systems which do not even employ DNNs. Nevertheless,
their lack of knowledge of the DNNs internal behavior might limit the ability
of these techniques to react to internal uncertainties (Stocco et al., 2022a).
A popular black-box approach for estimating the confidence of DNNs in the
context of ADSs is using autoencoders, which are DNNs that can reconstruct an
input image, to identify inputs beyond the distribution of the inputs with
which the ADS has been trained (Stocco et al., 2020; Hussain et al., 2022).
Metamorphic testing is an alternative black-box technique that can be used to
identify inconsistencies in an ADS without interacting with its internals
(Tian et al., 2018; Zhang et al., 2018).

Conversely, white-box techniques monitor the internal behavior of the DNN,
require a transparent access to the network, and are often specifically
designed for networks with specific features. However, the knowledge of the
DNN’s internal behavior may allow these techniques to identify internal
uncertainties that black-box techniques cannot. For example, with Deep
Ensemble Neural Networks, multiple DNN models with different weight
initialization or hyperparameters are trained with the same dataset to obtain
a probability distribution of the model output (Lakshminarayanan et al.,
2017). This distribution can then be used to infer the confidence of the DNN
for the corresponding inputs (Lakshminarayanan et al., 2017).

###  2.3. Metamorphic Testing

Metamorphic Testing (MT) is an alternative technique to alleviate the test
oracle problem that exploits known input and output relations that should hold
among _multiple_ test executions, the so-called Metamorphic Relations (MRs)
(Chen et al., 1998). As an example, if the input image for an image-based
classifier DNN is altered by slightly increasing its brightness, the output of
the DNN should remain almost unchanged. Here, the MR we define consists of an
_Input Relation (IR)_ ($img_{2}=brighten(img_{1})$) and an _Output Relation
(OR)_ ($output_{2}\simeq output_{1}$).

Typically, MT is performed with pairs of test cases: A _source_ test case,
which is usually generated by a test generation strategy (e.g., random test
generation), and a _follow-up_ test case, generated by _transforming_ the
source test case in a way that the input relation of the MR is satisfied
(Segura et al., 2016; Chen et al., 1998). For the given example, the follow-up
image can be generated by brightening the source image. A violation of this MR
(high deviation of the output between the source and follow-up images) might
indicate that the DNN is not operating reliably, possibly due to uncertain
conditions (Weiss and Tonella, 2023).

This technique has already been applied in various forms for testing CPSs,
such as autonomous drones (Lindvall et al., 2017) and driverless cars (Zhou
and Sun, 2019), including DNN-based autonomous driving systems (Tian et al.,
2018; Zhang et al., 2018). Nevertheless, the existing research of this
technique focuses on testing, as opposed to other verification tasks such as
the online monitoring of systems.

##  3\. Approach

In this paper we propose MarMot, a metamorphic runtime monitoring technique
for ADS. We specifically focus on the use of this technique to provide a DNN
supervisor. This supervisor provides an uncertainty score for the system at
run-time, enabling the real-time detection of misbehaviors via MRs in order to
take corrective actions when these occur.

###  3.1. Architecture

Figure 1. Architecture of MarMot for DNN-based ADS

Figure 1 shows an overview of the architecture of MarMot, which we use for
applying Metamorphic Runtime Monitoring to an ADS controlled by an image-based
DNN. The top half of the image shows the three main components of the ADS: (1)
the camera, for the image acquisition; (2) the DNN, for decision-making based
on the images; and (3) the actuator controls, which act on the steering and
throttling commands provided by the DNN.

The bottom half of the image shows the Metamorphic Runtime Monitoring
components and their interactions. For each MR used by our monitor, an Input
Relation (IR) and its corresponding Output Relation (OR) must be defined.

Firstly, the Input Relation 1 defines a transformation of the input image. In
the example from Figure 1, the IR consists of flipping the image horizontally,
i.e., $img_{f}=flip(img_{s})$. By applying this transformation to the image
captured by the camera (i.e., $img_{s}$), which we consider the _source_
input, we generate the _follow-up_ input for the MR (i.e., $img_{f}$).

After the _follow-up_ image has been generated, it is executed by the DNN to
obtain its output 2. This step corresponds with the execution of the follow-up
test case in Metamorphic Testing, which is possible because the DNN is usually
an isolated component of the ADS, with no direct dependencies on external
states. It is important to note that the output of the DNN for the follow-up
image is not sent to the driving commands, but instead used for monitoring
purposes.

Finally, the Output Relation 3 defines the expected follow-up value. In the
example from Figure 1, the Output Relation for the steering angle of the ADS
consists of inverting the output from the source test case, i.e.,
$steering_{f}\simeq-steering_{s}$.

To obtain an uncertainty score with this Output Relation, we emit
_quantitative verdicts_ based on the Euclidean distance, which is a simple way
to measure the distance between the expected and actual values of the follow-
up outputs:

(1) $u=\sqrt{(OR(O_{s})-O_{f})^{2}}$

where $u$ is the uncertainty score, $OR(O_{s})$ is the expected follow-up
output from the model for the follow-up image, computed with the Output
Relation $OR$ and the actual output from the model for the source image
$O_{s}$, and $O_{f}$ is the actual output from the model for the follow-up
image.

For instance, the uncertainty score for the Output Relation described above
would be computed as:

(2) $u=\sqrt{(-steering_{s}-steering_{f})^{2}}$

Under normal driving conditions in which the DNN is operating reliably, the MR
is expected to hold within a certain error margin, and so the uncertainty
scores are expected to remain under a certain boundary. However, unexpected
situations that may make the ADS unreliable, such as environmental anomalies
or inappropriate DNN training, could cause more severe MR violations, and thus
increased uncertainty scores. This is why the verdicts emitted by MarMot can
be interpreted as a measure of the uncertainty for the DNN. i.e., an
uncertainty score.

While these quantitative verdicts can quantify the uncertainty of the DNN
under test, some reference must be defined to determine whether a given value
is large enough to warrant raising an alarm. A common strategy, which we
employ in our implementation, is to define a threshold value based on the
verdicts observed in nominal driving conditions, such as in (a subset of) the
training data for the DNN.

The uncertainty score computed by an MR may be susceptible to short-term
increases for a few frames, which may not be desirable to interpret as an
anomalous scenario or system misbehavior. Thus, similar to SelfOracle (Stocco
et al., 2020), we implement a time-series based analysis 4 in order to smooth
uncertainty score spikes and avoid false alarms due to single-frame outliers.
Specifically, we implement a simple auto-regressive (AR) filter that computes
the uncertainty score as a linear combination of past values. The following
equation describes the computation of the uncertainty score $u_{t}$ at time
$t$ using the last $k$ values:

(3) $u_{t}=\sum^{k}_{i=1}{\frac{1}{i}u_{t-i}}$

###  3.2. Metamorphic Relations

We define five specific MRs for image-based ADSs, which we have implemented on
top of MarMot. The first four MRs define image transformations that are
expected to not change the output from the DNN under normal conditions,
whereas MR5 defines a transformation which should result in a symmetric
steering angle as the output from the DNN. While our approach is generic to
any kind of MRs for image-based ADSs, it is important to use MRs that are
computationally cheap. We therefore used five MRs below, and avoid using
others that are computationally more time consuming, such as frosting the
image.

  * •

MR1: Reduce Brightness. This transformation consists of slightly reducing the
brightness of the input image. This brightness reduction has been tuned to be
well within the tolerance of the DNN. Thus, the output is expected to be very
similar to the original image.

  * •

MR2: Increase Contrast. This transformation consists of slightly increasing
the contrast of the input image. This contrast increase has been tuned to be
well within the tolerance of the DNN, so the output is expected to be very
similar to the original image.

  * •

MR3: Add Noise. This transformation consists of slightly altering the values
from individual pixels of the image, with a uniform distribution for the
degree of change. The degree of the alterations has been tuned to be well
within the tolerance of the DNN. Thus, the output is expected to be very
similar from the original image.

  * •

MR4: Blur. This transformation consists of slightly blurring the image. The
degree of blurring has been tuned to be well within the tolerance of the DNN.
Thus, the output is expected to be very similar to the original image.

  * •

MR5: Horizontal Flip. This transformation consists of horizontally flipping
(i.e., mirroring) the image, as shown in the example from Figure 1. The output
from the DNN is expected to be mirrored (i.e., negated) from the one obtained
with the original image.

##  4\. Evaluation

This section describes our empirical evaluation and presents the results
obtained from our experiments. Specifically, we aim at answering the following
research questions (RQs):

  * RQ1

Effectiveness –  How effective is MarMot at predicting failures of ADSs
derived from internal or external anomalies? Which are the best MRs?

  * RQ2

Comparison –  How does MarMot compare with other state-of-the-art failure
predictors?

  * RQ3

Reaction –  How do the results for MarMot and other state-of-the-art failure
predictors change as we reduce the reaction window?

###  4.1. Case Study System

We evaluate MarMot on a physical environment, using the LeoRover (LeoRover,
2023) vehicle controlled by an image-based DNN provided by the vendor. Unlike
previous ADS monitoring approaches (Stocco et al., 2020, 2022a; Hussain et
al., 2022), we employed a physical system to avoid the threats to validity
produced by the reality gap between simulated and physical ADSs. While we
acknowledge that our small-scale ADS is still far from a real self-driving
car, it is also important to note that obtaining a dataset of anomalies and
DNN faults with a real car under real conditions requires extensive resources
and may potentially incur in ethical issues due to safety concerns. We
therefore believe that our small scale ADS is a good step towards the
assessment of monitoring techniques in real cars in more realistic conditions.

Figure 2. LeoRover in Circuit-1 Figure 3. LeoRover in Circuit-2

Our case study system’s DNN is trained to follow a road delimited by colored
tape. Figures 2 and 3 show the LeoRover vehicle driving in the two circuits we
employ in our evaluation. The vehicle has a camera, which we set to record
images at ~10 frames per second (FPSs), and four static wheels with
independent motors for steering and throttling, with a maximum linear speed of
~0.4 m/s. The DNN model consists of five convolutional layers followed by two
dense layers. The DNN was trained by ourselves for 100 epochs with a
combination of the dataset provided by the vendor and an additional dataset
recorded by us, which we describe in Section 4.2. The software of the LeoRover
employs the Robotic Operating System (ROS) (Koubâa et al., 2017), and the
different ADS components from Figure 1 are implemented as ROS nodes. This
software architecture enables the interaction with the DNN required by MarMot
to be efficiently implemented via ROS messages.

###  4.2. Datasets

In this section, we describe the datasets employed to train our subject DNN
and the evaluation datasets, in addition to the processes we used to collect
them. To allow for a fair and reproducible empirical evaluation, we evaluated
all the DNN supervisors offline. All the datasets consist of ordered lists of
camera images and annotations of which frames correspond with specific events,
such as out-of-bounds episodes. The images were recorded at ~10 FPSs, i.e., 10
images roughly correspond with a second of recording. Each image is stored
with a resolution of $120\times 160$ and three color channels (RGB). Because
we employed a physical ADS, significant manual effort was required for the
collection of the dataset, lasting in total a 2 person-week effort.

####  4.2.1. Training Data

To train the DNN model, we combined two datasets recorded in two different
circuits. The first dataset is provided by the LeoRover vendor, and consists
of 10,493 labeled images. The second dataset was collected by ourselves in a
test circuit, and consists of 2,832 labeled images. We extended the vendor’s
dataset to withstand relatively complex maneoveurs (e.g., sharp curves). The
circuits we used for the training were not used for evaluation purposes later.
The datasets were collected by manually driving the vehicle through the
circuits with a remote controller and recording the image, throttle, and
steering angle for each frame.

####  4.2.2. Nominal Conditions

The following datasets, which we used for the evaluation of our approach and
the baselines, were collected by ourselves in two circuits, different from the
ones employed for training the driving DNN. These circuits, which we named
Circuit-1 and Circuit-2, can be seen in Figure 2 and Figure 3 respectively. As
it can be seen in the images, Circuit-1 contains more curves and sharper
turns. In comparison, Circuit-2 only contains four curves in the same
direction, but we also incorporated some additional sources of environmental
uncertainty (as categorized byZhang et al. (Zhang et al., 2019, 2016)) in the
form of discontinuous lines and an extra line inside the circuit. We verified
that our DNN could successfully drive through both of these circuits under
normal conditions.

To evaluate the rate of false alarms obtained with our approaches, we first
collected recordings of the DNN driving in both circuits under nominal
conditions, with no out-of-bounds events nor any other incidents. These
recordings were arranged such that each of them corresponds roughly with one
complete lap to the circuit.

Overall, for the nominal conditions dataset, we obtained a total of 37
recordings (13,416 images) for Circuit-1 and 35 recordings for Circuit-2
(8,320 images), each of which corresponds with one lap to the circuit.

####  4.2.3. Anomalies

To measure the ability of our DNN supervisors to identify external anomalies,
we collected a dataset of out-of-bounds episodes resulting from unexpected
anomalies in the images. For this purpose, we implemented the image
corruptions and perturbations proposed by Hendrycks et al. (Hendrycks and
Dietterich, 2019), which are widely used for testing image-based DNNs
(Ferreira et al., 2021; Ruff et al., 2021; Guerin et al., 2022; Stocco et al.,
2022b). For our experiments, we could only use 13 out of the 19 proposed image
corruptions due to performance limitations when applying the image
transformations while the vehicle is running. For each of the anomalies, five
different levels are employed.111The specific values for each of the anomalies
can be found in
https://github.com/jonayerdi/marmot/blob/master/leorover_scripts/hendrycks.py
The employed image corruptions include the following:

  * •

Gaussian noise

  * •

Shot noise

  * •

Impulse noise

  * •

Defocus blur

  * •

Glass blur

  * •

Fog

  * •

Brightness

  * •

Contrast

  * •

Pixelate

  * •

JPEG compression

  * •

Speckle noise

  * •

Gaussian blur

  * •

Spatter

  * •

Saturation

To emulate the occurrence of these external anomalies, we modified the DNN
script to apply the image transformation to every image to be processed by the
DNN, starting after 20 seconds from the moment the first image is processed.
Therefore, each recording for external anomalies consists of ~20 seconds of
normal driving conditions, followed by a window of anomalous external
conditions which lasts until the out-of-bounds occurs. The frames in which the
anomalous external conditions begin and the out-of-bounds occurs were
annotated for each recording by manually inspecting the images.

Overall, for the external uncertainties dataset, we obtained a total of 52
recordings (13,948 images) for Circuit-1 and other 52 recordings for Circuit-2
(12,963 images), each of which ends with an out-of-bounds episode after an
image perturbation is applied. Specifically, we have 13 types of anomalies,
with 2 severity parameterizations for each anomaly, and we collect 2
recordings for each anomaly type and parameterization, hence $13\times 2\times
2=52$ recordings for each circuit.

####  4.2.4. Mutants

To measure the ability of our DNN supervisors to identify internal anomalies,
we collected a dataset of out-of-bounds episodes caused by faulty driving
models. For this purpose, we implemented the mutation operators for DNNs
proposed by Humatova et al. (Humbatova et al., 2021), which are based on real
deep learning faults. These mutation operators include inadequate training
data and sub-optimal choice of the model’s architecture or of the training
hyper-parameters (Humbatova et al., 2021).

Specifically, we implemented three different mutation operators that,
according to the evaluation carried out in the original paper (Humbatova et
al., 2021), were not in the group of non-killable mutants, they did not have a
high triviality score and they did not produce redundant mutants. The first
operator, HLR, changes the learning rate. We used two different
parameterizations for this mutation, with learning rate values of 0.0001 and
0.01 (the original learning rate was 0.001). The second operator, TAN,
introduces low-quality training data by adding noise to a percentage of
pixels. We used two different parameterizations for this mutation, changing
15% and 25% of the pixels from the images. Lastly, TCL, changes labels of
training data, mimicking wrong labeling situations. Again for this case, two
different parameterizations were used, assigning a random value to the
steering angle of the vehicle to 15% and 20% of the images. For each mutant
and parameterization, we train 10 different models, since the mutants are
stochastic (Humbatova et al., 2021).

For this dataset, we collected recordings of the mutant DNNs driving through
the circuits until an out-of-bounds episode occurred. The frames in which the
out-of-bounds occur were annotated for each recording by manually inspecting
the images.

Overall, for the internal uncertainties dataset, we obtained a total of 30
recordings (13,339 images) for Circuit-1 and other 30 recordings for Circuit-2
(6,689 images), each of which ends with an out-of-bounds episode.
Specifically, we used three types of mutations, with two parameterizations for
each mutation, and we collected five recordings for each mutant type and
parameterization, hence $3\times 2\times 5=30$ recordings for each circuit.

###  4.3. Baselines

####  4.3.1. SelfOracle

To evaluate the effectiveness of MarMot, we compared the obtained results with
SelfOracle (Stocco et al., 2020), a state-of-the-art misbehaviour prediction
technique based on input image reconstruction models. SelfOracle (Stocco et
al., 2020) estimates the confidence for the DNN based on the reconstruction
error of the driving model’s input images. During training, an autoencoder
model is trained with the same images used to train the subject DNN. At
runtime, the input images are reconstructed by the autoencoder model, and the
uncertainty score is measured as the reconstruction error.

(4)
$u=\frac{1}{WHC}\sum^{W,H,C}_{i=1,j=1,c=1}{(x[c][i,j]-x^{\prime}[c][i,j])^{2}}$

where $u$ is the uncertainty score, $x_{t}$ is the input image, $x^{\prime}$
is the image reconstructed by the autoencoder from $x$, and $W,H,C$ are the
pixel-wise width, height, and channels of the input images. The rationale
behind this is that images with a high reconstruction error may be out of the
distribution from the training data, and therefore, the DNNs behavior may be
unreliable for those inputs.

We specifically employed 4 different autoencoder models implemented in their
open-source release222https://github.com/testingautomated-usi/selforacle for
our experiments, namely: (1) SAE (simple autoencoder with a single hidden
layer), (2) DAE (deep autoencoder with five fully-connected layers), (3) CAE
(convolutional autoencoder with alternating convolutional and max-pooling
layers), and (4) VAE (variational autoencoder). We could not train the LSTM
variant due to the limited availability of computational resources. Note,
however, that the VAE was the best configuration of SelfOracle (Stocco et al.,
2020), and was also used in other similar evaluations (Stocco et al., 2022a;
Hussain et al., 2022).

These autoencoder models have been trained with the images from the datasets
used to train the driving model, since this is the approach that would most
likely be used if SelfOracle was to be adopted in practice.

####  4.3.2. Ensemble

Furthermore, we also employed a simple monitor based on Deep Ensemble Neural
Networks (Lakshminarayanan et al., 2017) as an additional baseline. Our
Ensemble monitor employs 10 driving DNNs trained with different weight
initializations and training/validation splits of the dataset, but the same
hyperparameters. The uncertainty score for this monitor is then computed as
the standard deviation of the outputs from the individual driving models for
each input image.

###  4.4. Configuration

####  4.4.1. Thresholds

As mentioned at the end of Section 3.1, the uncertainty scores produced by
MarMot require a threshold value to determine whether the uncertainty score
indicates normal conditions or is too high and an alarm needs to be triggered.
For this purpose, we isolated 7 out of the 37 recordings of nominal driving
conditions from Circuit-1, which correspond with 7 laps to the circuit. These
7 recordings were used as a reference to determine the threshold value for
each of our oracles, using the following formula:

(5) $t_{o}=max\left(max\left(u_{o}(f)\;\forall\;f\in R\right)\;\forall\;R\in
D\right)\cdot 1.1$

where $t_{o}$ is the threshold for the oracle $o$,
$max\left(u_{o}(f)\;\forall\;f\in R\right))$ is the maximum uncertainty score
obtained with the oracle $o$ from all the frames $f$ in recording $R$, and $D$
is the set of all recordings used to compute the thresholds. We chose to add a
margin of 10% for computing the threshold in order to avoid false alarms,
hence the $1.1$ multiplier. We consider that the oracle $o$ raises an alarm at
any point when its uncertainty score is greater than $t_{o}$.

We considered individual MRs from MarMot and individual autoencoder models
from SelfOracle to be distinct oracles, so we computed thresholds for each of
them separately.

We discarded the recordings used for computing these thresholds from our
evaluation dataset. Thus, our evaluation dataset for nominal conditions
consists of 30 recordings from Circuit-1 and all the 35 recordings from
Circuit-2.

####  4.4.2. Time-Series Analysis

For all oracles, we applied the simple auto-regressive filter described in
Section 3 using the last 10 images ($k=10$). Note that the implementation of
this filter is equivalent to the one proposed in SelfOracle (Stocco et al.,
2020), so we do not deviate from their proposed approach.

####  4.4.3. MarMot

We used the following parameters for the implementation of our MRs. We ensured
that the follow-up inputs generated with the selected MR parameters followed
the input validity of the domain as defined by Riccio and Tonella (Riccio and
Tonella, 2023).

  * •

_MR1: Reduce Brightness._ We substract 77 from the values of each pixel
(saturating substraction, so values remain between 0 and 255).

  * •

_MR2: Increase Contrast._ We convert the image to HSV format and set the
saturation value of every pixel to 50 (possible values range from 0 to 255).

  * •

_MR3: Add Noise._ We alter the values of each pixel by a rate from 0% to 20%
(uniformly sampled random value per pixel).

  * •

_MR4: Blur._ We apply a normalized box filter, as implemented by OpenCV-
Python’s cv2.blur function, using a kernel size of $1\times 5$.

####  4.4.4. SelfOracle

For training the SelfOracle autoencoder models, image augmentation was applied
using flipping, translation, shadowing, and brightness changes. 60% of the
data was augmented through these transformations, matching the configuration
used by Stocco et al. (Stocco et al., 2020). It is noteworthy that this
algorithm is also employed in DeepGuard (Hussain et al., 2022).

Due to the stochastic nature of training these autoencoder models, we trained
each model 10 times with different random seeds, which affects the
initialization and the image augmentations applied. All the results for
SelfOracle reported in Section 4.6 are average values for the 10 models of
each autoencoder type.

####  4.4.5. Ensemble

As mentioned in Section 4.3.2, the Ensemble oracle employs 10 different
driving DNNs trained with different initializations. For each of the mutations
of the driving DNN employed for our internal uncertainties dataset, we also
trained 10 models with different initializations, and the Ensemble oracle
employs the corresponding mutants of the driving models when evaluating the
internal uncertainties dataset.

###  4.5. Evaluation Metrics

For each recording described in Section 4.2, a test oracle may or may not
raise an alarm if its uncertainty score is greater than its threshold (Section
4.4.1). For each recording from the nominal conditions dataset, we define an
instance where an alarm is raised as a false positive ($FP$), and an instance
where it is not as a true negative ($TN$). Conversely, for each recording from
the internal or external anomalies datasets, we define an instance where an
alarm is raised as a true positive ($TP$), and an instance where it is not as
a false negative ($FN$).

In our evaluation, we only considered whether an oracle fired an alarm or not
for each recording, since we assumed that the alarms might cause a response
and consecutive alarms will have no effect. Therefore, every individual
recording described in Section 4.2 will count as exactly one FP or TN (nominal
conditions dataset), or one TP or FN (anomalies or mutants datasets).

Similar to most evaluations using classifiers, we also report the false
positive rate ($FPR=\frac{FP}{FP+TN}$) for the nominal conditions dataset, as
well as the precision ($Prec.=\frac{TP}{TP+FP}$), the recall or true positive
rate ($TPR=\frac{TP}{TP+FN}$), and the F1-score ($F1=2\cdot\frac{Prec.\cdot
TPR}{Prec.+TPR}$) for the anomalies datasets.

Similar to (Stocco et al., 2020), we also report two threshold-independent
metrics for evaluating classifiers: AUC-ROC (area under the curve of the
Receiver Operating Characteristics) and AUC-PRC (area under the Precision-
Recall Curve).

###  4.6. Analysis of the Results and Discussion

####  4.6.1. RQ1 – Effectiveness

Table 1 shows the evaluation results for the nominal datasets (i.e., there is
no misbehavior) from Circuit-1 and Circuit-2. With our chosen threshold, for
MarMot, we observe three FPs for MR5 (Horizontal Flip) in Circuit-2, which
corresponds with a FPR (to be minimized) of 10%. For all other cases, we
observe 0 FPs for all of MarMot’s MRs.

Table 2 shows the evaluation results for the anomaly datasets.. In terms of
TPR (the higher, the better), we observe that MR5 achieves the best results in
both circuits, 35% for Circuit-1 and 65% for Circuit-2. In comparison, the
rest of the MRs achieve significantly lower TPR and F1 measures, indicating a
much lower ability to identify external uncertainties. On the other hand,
Table 3 shows the evaluation results for the mutant datasets (i.e., faults of
the DNN based on the three chosen mutation operators from (Humbatova et al.,
2021)). In this case, we observe that all of MarMot’s MRs achieve very high
TPR and F1 scores, with TPRs higher than 77% in all cases. Furthermore, MR5
achieves 100% TPR in both circuits.

Table 1. Evaluation results for all oracles with Nominal datasets and a 10% threshold (average of 10 for SelfOracle), false positives in boldface. | Nominal |  |
---|---|---|---
| Circuit-1 | Circuit-2 |  |
| FP | TN | FPR | FP | TN | FPR |  |
MarMot |  |  |  |  |  |  |  |
MR1: Reduce Brightness | 0.0 | 30.0 | 0.00 | 0.0 | 30.0 | 0.00 |  |
MR2: Increase Contrast | 0.0 | 30.0 | 0.00 | 0.0 | 30.0 | 0.00 |  |
MR3: Add Noise | 0.0 | 30.0 | 0.00 | 0.0 | 30.0 | 0.00 |  |
MR4: Blur | 0.0 | 30.0 | 0.00 | 0.0 | 30.0 | 0.00 |  |
MR5: Horizontal Flip | 0.0 | 30.0 | 0.00 |  3.0 | 27.0 | 0.10 |  |
SelfOracle |  |  |  |  |  |  |  |
CAE | 0.0 | 30.0 | 0.00 |  0.1 | 29.9 | 0.00 |  |
DAE | 0.0 | 30.0 | 0.00 | 0.0 | 30.0 | 0.00 |  |
SAE | 0.0 | 30.0 | 0.00 | 0.0 | 30.0 | 0.00 |  |
VAE | 0.0 | 30.0 | 0.00 | 0.0 | 30.0 | 0.00 |  |
Ensemble |  2.0 | 28.0 | 0.07 |  1.0 | 29.0 | 0.03 |  |
Table 2. Evaluation results for all oracles with Anomaly datasets and a 10% threshold (average of 10 for SelfOracle), best results in boldface | Anomaly
---|---
| Circuit-1 | Circuit-2
| TP | FN | TPR | Prec. | F1 | TP | FN | TPR | Prec. | F1
MarMot |  |  |  |  |  |  |  |  |  |
MR1: Reduce Brightness | 8.0 | 44.0 | 0.15 | 1.00 | 0.27 | 17.0 | 35.0 | 0.33 | 1.00 | 0.49
MR2: Increase Contrast | 3.0 | 49.0 | 0.06 | 1.00 | 0.11 | 2.0 | 50.0 | 0.04 | 1.00 | 0.07
MR3: Add Noise | 3.0 | 49.0 | 0.06 | 1.00 | 0.11 | 4.0 | 48.0 | 0.08 | 1.00 | 0.14
MR4: Blur | 2.0 | 50.0 | 0.04 | 1.00 | 0.07 | 6.0 | 46.0 | 0.12 | 1.00 | 0.21
MR5: Horizontal Flip | 18.0 | 34.0 |  0.35 | 1.00 |  0.51 | 34.0 | 18.0 |  0.65 | 0.92 |  0.76
SelfOracle |  |  |  |  |  |  |  |  |  |
CAE | 8.7 | 43.3 | 0.17 | 1.00 | 0.29 | 14.2 | 37.8 | 0.27 | 0.99 | 0.41
DAE | 0.0 | 52.0 | 0.00 | 0.00 | 0.00 | 0.0 | 52.0 | 0.00 | 0.00 | 0.00
SAE | 2.5 | 49.5 | 0.05 | 0.50 | 0.09 | 3.0 | 49.0 | 0.06 | 0.50 | 0.10
VAE | 4.9 | 47.1 | 0.09 | 1.00 | 0.17 | 5.2 | 46.8 | 0.10 | 1.00 | 0.18
Ensemble | 16.0 | 36.0 | 0.31 | 0.89 | 0.46 | 19.0 | 33.0 | 0.37 | 0.95 | 0.53
Table 3. Evaluation results for all oracles with Mutant datasets and a 10% threshold (average of 10 for SelfOracle), best results in boldface | Mutant
---|---
| Circuit-1 | Circuit-2
| TP | FN | TPR | Prec. | F1 | TP | FN | TPR | Prec. | F1
MarMot |  |  |  |  |  |  |  |  |  |
MR1: Reduce Brightness | 25.0 | 5.0 | 0.83 | 1.00 | 0.91 | 25.0 | 5.0 | 0.83 | 1.00 | 0.91
MR2: Increase Contrast | 23.0 | 7.0 | 0.77 | 1.00 | 0.87 | 23.0 | 7.0 | 0.77 | 1.00 | 0.87
MR3: Add Noise | 25.0 | 5.0 | 0.83 | 1.00 | 0.91 | 25.0 | 5.0 | 0.83 | 1.00 | 0.91
MR4: Blur | 25.0 | 5.0 | 0.83 | 1.00 | 0.91 | 25.0 | 5.0 | 0.83 | 1.00 | 0.91
MR5: Horizontal Flip | 30.0 | 0.0 |  1.00 | 1.00 |  1.00 | 30.0 | 0.0 |  1.00 | 0.91 |  0.96
SelfOracle |  |  |  |  |  |  |  |  |  |
CAE | 3.7 | 26.3 | 0.12 | 0.70 | 0.20 | 2.8 | 27.2 | 0.09 | 0.79 | 0.16
DAE | 0.0 | 30.0 | 0.00 | 0.00 | 0.00 | 0.0 | 30.0 | 0.00 | 0.00 | 0.00
SAE | 0.0 | 30.0 | 0.00 | 0.00 | 0.00 | 0.0 | 30.0 | 0.00 | 0.00 | 0.00
VAE | 0.0 | 30.0 | 0.00 | 0.00 | 0.00 | 0.0 | 30.0 | 0.00 | 0.00 | 0.00
Ensemble | 16.0 | 14.0 | 0.53 | 0.89 | 0.67 | 10.0 | 20.0 | 0.33 | 0.91 | 0.49
Table 4. Evaluation AUC results for all oracles and datasets (average of 10
for SelfOracle), best results in boldface

| Anomaly | Mutant
---|---|---
| Circuit-1 | Circuit-2 | Circuit-1 | Circuit-2
| AUC-PRC | AUC-ROC | AUC-PRC | AUC-ROC | AUC-PRC | AUC-ROC | AUC-PRC | AUC-ROC
MarMot |  |  |  |  |  |  |  |
MR1: Reduce Brightness | 0.688 | 0.439 | 0.831 | 0.727 | 0.791 | 0.833 | 0.791 | 0.833
MR2: Increase Contrast | 0.500 | 0.248 | 0.799 | 0.699 | 0.771 | 0.811 | 0.791 | 0.833
MR3: Add Noise | 0.578 | 0.346 | 0.870 | 0.821 | 0.791 | 0.833 | 0.791 | 0.833
MR4: Blur | 0.605 | 0.392 | 0.821 | 0.775 | 0.791 | 0.833 | 0.782 | 0.833
MR5: Horizontal Flip |  0.751 |  0.546 | 0.841 | 0.776 |  0.958 |  1.000 |  0.958 |  1.000
SelfOracle |  |  |  |  |  |  |  |
CAE | 0.731 | 0.539 | 0.717 | 0.489 | 0.765 | 0.719 | 0.480 | 0.330
DAE | 0.517 | 0.214 | 0.568 | 0.322 | 0.376 | 0.269 | 0.307 | 0.146
SAE | 0.519 | 0.220 | 0.602 | 0.388 | 0.372 | 0.284 | 0.349 | 0.205
VAE | 0.532 | 0.256 | 0.685 | 0.522 | 0.512 | 0.393 | 0.440 | 0.311
Ensemble | 0.695 | 0.462 |  0.882 |  0.848 | 0.692 | 0.715 | 0.696 | 0.748

Table 4 shows the threshold-independent measures of AUC-PRC and AUC-ROC for
both the anomaly and mutant datasets. Based on these metrics, we observe that
MR5 (Horizontal Flip) achieves the best results in all cases, with MR1 (Reduce
Brightness) and MR3 (Add Noise) being the next best MRs.

The reason why the effectiveness of MarMot is much lower for identifying
external anomalies might be related with the length of the reaction period,
i.e., the period between the beginning of the anomalous scenario until the
moment where the vehicle goes out-of-bounds. We observed that most of the
image filters we implemented for the anomalies dataset cause the vehicle to
drive off the road fairly quickly, leaving a monitor with a very short period
to react. Conversely, many of the mutant DNNs are capable of driving within
bounds for longer periods of time, which gives the monitors a much longer time
to react to the DNNs uncertain behavior. The estimated average reaction period
for the anomalies datasets recordings is around 7 seconds for Circuit-1 and 5
seconds for Circuit-2, whereas the reaction periods for the mutants datasets
are around 40 seconds for Circuit-1 and 20 seconds for Circuit-2.

RQ1: In summary, we can conclude that MarMot is very effective at identifying
mutants, with TPRs ranging from 77% to 100%, while also being fairly effective
at identifying external anomalies with at least one MR, with the TPRs for the
best MR (Horizontal Flip) ranging from 35% to 65%.

####  4.6.2. RQ2 – Comparison

Table 1 shows that all oracles but Ensemble achieve 0 FPs for Circuit-1,
whereas for Circuit-2, Ensemble, CAE (SelfOracle) and MR5 (Horizontal Flip)
are the only oracles with FPs. Overall, the FPR (to be minimized) is not
higher than 10% for any of the oracles, but the highest FPR is obtained by
MarMot’s MR5 in Circuit-2.

Table 2 shows that, in terms of TPR and F1 (to be maximized), MR5 achieves the
best results in both circuits, followed by Ensemble, and then either MR1
(Reduce Brightness) or SelfOracle with the CAE model. On the other hand, we
observe from Table 3 that all of MarMot’s MRs achieve a significantly higher
TPR and F1 score than the other approaches, followed by Ensemble. Here,
SelfOracle achieves TPRs of 12% or lower with CAE in both circuits, and 0%
scores with the other autoencoder models.

Based on the threshold-independent measures from Table 4, we observe that all
of MarMot’s MRs outperform both SelfOracle and Ensemble baselines by a
significant margin for the mutant datasets. As for the anomalies, MR5 achieves
the highest scores in Circuit-1, closely followed by SelfOracle’s CAE model,
and then followed by Ensemble. Surprisingly, Ensemble achieves the highest
scores for the Circuit-2 anomalies, followed by MarMot’s MRs, and then
followed by SelfOracle’s CAE.

It is interesting that SelfOracle’s CAE model achieves the best results among
the four autoencoder models in our evaluation, whereas in the evaluation from
Stocco et al. it obtained the worst results (Stocco et al., 2020). The
relative ordering for the rest of the autoencoders from best to worst, is VAE,
SAE, and lastly DAE, which does match the results obtaied by Stocco et al.
(Stocco et al., 2020). It is important to note that our dataset comprises
pictures from a physical environment, whereas SelfOracle’s original evaluation
employed images obtained from a simulator. CAE’s image reconstruction process
appears to be more sensitive to anomalies in real images than the rest of the
autoencoders, but less so in images generated from a simulator, which are
generally simpler and contain fewer details.

Comparing the results from Circuit-1 and Circuit-2 for the anomalies datasets,
we observe that all monitors obtained noticeably better results in Circuit-2,
despite the fact that the reaction period for this circuit is slightly shorter
on average (~7 seconds for Circuit-1 and ~5 seconds for Circuit-2). We
speculate that the extra environmental uncertainties introduced in Circuit-2
(i.e., the discontinuous road boundaries and the extra line in the middle of
the road) causes a slightly more erratic driving, which might make the
uncertain behavior from anomalies to become more noticeable for the monitors.
The results from Table 4 seem to indicate that Ensemble is the most favored by
this factor. This may be due to the fact that Ensemble is a white-box
approach, and thus potentially more sensitive to the DNNs internal erratic
behavior than MarMot and SelfOracle, which are black-box.

RQ2: In summary, we can conclude that MarMot clearly outperforms both
baselines in terms of identifying mutants, while also achieving a comparable
or higher effectiveness at identifying external anomalies with the best MRs.

####  4.6.3. RQ3 – Reaction

(a)

(b)

(c)

(d)

Figure 4. AUC-PRC over reaction time

In this RQ, we aim to account for the latency of the anomaly detection and
reaction process that would exist in practice (communications latency,
processing time, etc.), which might prevent the triggered healing process
(e.g., stopping the vehicle before out-of-bounds) from taking effect in time.

Figure 4 shows graphs with the different AUC-PRC scores (vertical axis) that
would be obtained if we require the oracles to raise an alarm a certain amount
of frames _before_ the out-of-bounds happens (horizontal axis). For instance,
the AUC-PRC at Reaction Frames $=-20$ is the measure that we would obtain in
practice if we accounted for a delay of 20 frames from the moment an image is
recorded until the oracle has processed it, the alarm is raised, and the
healing action is taken. Since we process images at 10 FPSs, Reaction Frames
$=-20$ corresponds with roughly two seconds of latency. Note that the
rightmost values (Reaction Frames $=0$) correspond with the measures from
Table 4.

Figures 4(a) and 4(b) show the AUC-PRC scores for the external anomalies in
Circuit-1 and Circuit-2 respectively. We can observe that the performance of
all approaches degrades very significantly if we account for 10 or 20 latency
frames. In practice, this might mean that achieving results similar to the
ones presented in the previous RQs may require very low latency communications
and processing. Nevertheless, MarMot and all the baselines seem to be affected
by this in a very similar manner. The cause is likely to be the short reaction
time between the moment the external anomaly happens and the out-of-bounds
episode, as mentioned in the previous RQs.

On the other hand, Figures 4(c) and 4(d) show the AUC-PRC scores for the
mutants in Circuit-1 and Circuit-2 respectively. In this case, we can see that
MarMot’s performance is almost completely unaffected even if we account for up
to 40 frames (around four seconds) of latency. This should be realistically
more than enough reaction time even if MarMot’s processing is performed in a
separate computational unit communicating with the LeoRover. On the other
hand, both baselines show a steady degradation in their effectiveness if any
amount of latency is accounted for. Since SelfOracle derives its uncertainty
scores purely from the DNN inputs (i.e., the camera images), it makes sense
that most uncertain behaviors are identified shortly before the out-of-bounds
occurs, which causes its efficiency to degrade if we account for some latency.
Conversely, MarMot does consider the DNN output, which can make it more
sensitive to the DNN’s internal behavior and allow it to identify mutants
early. Despite being a white-box approach, Ensemble does not seem to identify
uncertain behaviors until shortly before the out-of-bounds occurs, as the
graphs show a steady degradation of AUC-PRC comparable with SelfOracle’s.

RQ3: In summary, we conclude that in the case of anomalies, the performance of
MarMot and both baselines degrades significantly if we account for a small
latency. Conversely, in the case of mutants, the performance of MarMot remains
identical even with a latency of up to four seconds.

###  4.7. Threats to Validity

We now discuss the main threats of our evaluation and how we tried to mitigate
them.

####  4.7.1. Internal validity

The generated mutants and anomalies suppose potential internal validity
threats in our study. For the mutants, we mitigated them by using some of the
mutation operators recommended by Humatova et al. (Humbatova et al., 2021). As
for the anomalies, we used the image corruptions and perturbations proposed by
(Hendrycks and Dietterich, 2019), which are widely used for testing image-
based DNNs. The employed DNN architecture and its training configuration could
be another internal validity threat. We mitigated it by employing the same
architecture as suggested by the manufacturer. Moreover, we used the same
architecture and parameters for the baseline of SelfOracle as its original
paper (Stocco et al., 2020). The selected MRs 1 to 4 have some parameters
(e.g., brightness level). Different parameters may lead to different results.
To mitigate this threat, we verify that our parameters will result in valid
follow-up inputs. We follow the definition provided by (Riccio and Tonella,
2023) to define a valid test input in the domain of DNN-based autonomous
driving systems, i.e., the image is recognizable by a human expert.

####  4.7.2. External validity

We employed a single case study system, which poses a threat in terms of
generalizability. We mitigated this threat by employing two different
circuits, different anomaly types and different DNN mutants, resulting in a
diverse dataset. Moreover, the employed case study system was a physical one,
which reduces the threat posed by the gap between simulators and real ADSs.

####  4.7.3. Conclusion validity

The employed baseline, i.e., SelfOracle (Stocco et al., 2020), employs neural
networks, and as such, the training process is stochastic. To reduce the
threat that the obtained results are by chance, we train each configuration of
SelfOracle 10 times and provide average values in the evaluation.

##  5\. Related Work

Automated test generation techniques have been the most widely investigated
technique to identify unexpected driving scenarios of ADSs. This has been
targeted from different perspectives, including techniques to maximize neuron
coverage (Pei et al., 2017), surrogate-assisted search-based test generation
(Abdessalem et al., 2018b, a; Haq et al., 2022; Mullins et al., 2018),
techniques based on reinforcement learning (Haq et al., 2023; Lu et al., 2022)
and procedural content generation to generate virtual roads (Gambi et al.,
2019b). These studies focus on detecting unexpected scenarios at design-time,
before deploying the system in opration. Conversely, our approach focuses on
proposing a run-time monitoring technique to build a safety envelope over a
DNN to assess its level of dependability in operation.

In this context, i.e., run-time monitoring of DNN-enabled ADSs, black-box
(Stocco et al., 2020, 2020) and white-box (Stocco et al., 2022a) approaches
have been recently proposed, as discussed throughout the paper. Our approach
falls in the scope of the black-box ones. In such an scope, frameworks such as
SelfOracle (Stocco et al., 2020) and DeepGuard (Hussain et al., 2022) train
auto-encoders with the same images used for training the ADS, and use the
reconstruction error of these auto-encoders as an uncertainty score. Unlike
existing black-box approaches, MarMot does not require any training, as it
relies on metamorphic relations to generate follow-up inputs and subsequently
obtain a confidence score through output relations. Moreover, MarMot overcomes
the limitation identified by Stocco et al. (Stocco et al., 2022a) for black-
box techniques, i.e., failing to detect faults caused by an inadequate
training or by bugs at the DNN model level, as we demonstrated in our
evaluation.

Our approach is largely inspired by Metamorphic Testing, a technique that has
been found effective at alleviating the test oracle problem (Segura et al.,
2016). This technique has also been explored in the context of ADSs (Segura et
al., 2016; Tian et al., 2018; Zhou and Sun, 2019; Zhang et al., 2018; Deng et
al., 2021, 2022). Tian et al. (Tian et al., 2018) proposed DeepTest, a testing
framework to identify erroneous behaviors of DNN-based self-driving cars
through MR-based test oracles. Zhang et al. (Zhang et al., 2018) proposed
DeepRoad, an ADS testing framework that leverages Generative Adversarial
Networks (GANs) to automatically generate realistic follow-up images
simulating various weather conditions, such as snow or rain. Deng et al. (Deng
et al., 2021) propose a Behaviour-Driven Development (BDD) workflow in which
MRs are expressed in a declarative manner by the test engineers. The framework
converts declared MRs to executable oracles in addition to the synthesis of
test inputs for each MR (Deng et al., 2021). The same team later proposed a
rule-based declarative framework that leverages natural language processing
(NLP) to extract MRs from predicates written in natural language (Deng et al.,
2022). Unlike all these studies, which focus on design-time testing of ADSs,
our approach leverages MRs for operational purposes with the goal of detecting
potential misbehaviors and take corresponding healing actions in run-time.

To the best of our knowledge, there are only three papers that have proposed
the use of MT at run-time (Murphy et al., 2009; Murphy and Kaiser, 2009; Bell
et al., 2015). These papers introduce the concept of _Metamorphic Runtime
Checking_ , which is the analogous of runtime assertion checking for MRs (Bell
et al., 2015). Similar to our work, their approach also involves intercepting
inputs at runtime and executing follow-up test inputs to check MRs. In
contrast to our work, which aims to monitor ADSs, they apply this technique to
monitor generic software applications (Murphy et al., 2009) and their
individual functions (Murphy and Kaiser, 2009; Bell et al., 2015). Their
approach involves creating a “sandbox” in order to execute the follow-up test
cases, which typically involves forking the process, in order to ensure
identical state from the original execution (Bell et al., 2015). This
sandboxing process is unnecessary for MarMot, since DNNs are generally
isolated components which do not depend on external state. Given the
application domain, the monitors they generate employ assertions (pass or
fail). Unlike these approaches, we levarage metamorphic relations with the
goal of measuring the uncertainty of the ADS controller in an efficient and
effective way. MarMot later employs these quantitative uncertainty scores in
an auto-regressive filter to consider past values and raise a flag in case
this value exceeds a predefined threshold. To the best of our knowledge, this
paper is the first that aims at adapting MT to run-time monitoring in a
context of AVs and CPSs.

##  6\. Conclusion

We propose MarMot, a novel run-time monitoring approach that leverages ideas
from Metamorphic Testing to estimate the confidence of ADS controllers. Our
evaluation, which uses a small-scale ADS, shows that MarMot is capable of
effectively anticipating failures caused by either internal or external
uncertainties before the vehicle goes out-of-bounds. Indeed, our approach
outperformed or at least showed the same performance as the employed baselines
(i.e., SelfOracle (Stocco et al., 2020) and Ensemble-based techniques).

Future work includes comparing our approach with other confidence estimators,
such as those based on eXplainable AI (XAI) techniques. Furthermore, we also
plan to evaluate the impact of combining these techniques with different self-
healing mechanisms, such as slowing down the vehicle when facing a sudden peak
of the uncertainty score.

## Replication Packages:

We make all our artifacts available as follows:

  * •

Our dataset can be found at: https://doi.org/10.5281/zenodo.8202317

  * •

Our code is available at: https://github.com/jonayerdi/marmot

###### Acknowledgements.

This work was partially founded by the Basque Government through their
Elkartek program (EGIA project, ref. KK-2022/00119 and SIIRSE project, ref.
KK-2022/00007). The authors are part of the Software and Systems Engineering
research group of Mondragon Unibertsitatea (IT1519-22), supported by the
Department of Education, Universities and Research of the Basque Country.

## References

  * (1)
  * Abdessalem et al. (2018a) Raja Ben Abdessalem, Shiva Nejati, Lionel C Briand, and Thomas Stifter. 2018a.  Testing vision-based control systems using learnable evolutionary algorithms. In _Proceedings of the 40th International Conference on Software Engineering_. 1016–1026.
  * Abdessalem et al. (2018b) Raja Ben Abdessalem, Annibale Panichella, Shiva Nejati, Lionel C Briand, and Thomas Stifter. 2018b.  Testing autonomous cars for feature interaction failures using many-objective search. In _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_. 143–154.
  * Apollo (2023) Apollo. 2023.  Apollo Website.  https://developer.apollo.auto/index.html Accessed: August 1, 2023.
  * Ayerdi et al. (2023) Jon Ayerdi, Pablo Valle, Asier Iriarte, Ibai Roman, Miren Illarramendi, and Aitor Arrieta. 2023.  _Dataset for ”MarMot: Metamorphic Runtime Monitoring of Autonomous Driving Systems”_.  https://doi.org/10.5281/zenodo.8202317
  * Bell et al. (2015) Jonathan Bell, Christian Murphy, and Gail Kaiser. 2015.  Metamorphic runtime checking of applications without test oracles.  _CrossTalk_ 28, 2 (2015).
  * Ben Abdessalem et al. (2016) Raja Ben Abdessalem, Shiva Nejati, Lionel C Briand, and Thomas Stifter. 2016.  Testing advanced driver assistance systems using multi-objective search and neural networks. In _Proceedings of the 31st IEEE/ACM international conference on automated software engineering_. 63–74.
  * Biagiola and Tonella (2022) Matteo Biagiola and Paolo Tonella. 2022.  Testing the plasticity of reinforcement learning-based systems.  _ACM Transactions on Software Engineering and Methodology (TOSEM)_ 31, 4 (2022), 1–46.
  * Bojarski et al. (2016) Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. 2016\.  End to end learning for self-driving cars.  _arXiv preprint arXiv:1604.07316_ (2016).
  * Calò et al. (2020) Alessandro Calò, Paolo Arcaini, Shaukat Ali, Florian Hauer, and Fuyuki Ishikawa. 2020.  Generating avoidable collision scenarios for testing autonomous driving systems. In _2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)_. IEEE, 375–386.
  * Chen et al. (1998) T. Y. Chen, S. C. Cheung, and S. M. Yiu. 1998.  _Metamorphic Testing: A New Approach for Generating Next Test Cases_.  Technical Report. Technical Report HKUST-CS98-01, Department of Computer Science, The Hong Kong University of Science and Technology.
  * Deng et al. (2021) Yao Deng, Guannan Lou, Xi Zheng, Tianyi Zhang, Miryung Kim, Huai Liu, Chen Wang, and Tsong Yueh Chen. 2021.  BMT: Behavior driven development-based metamorphic testing for autonomous driving models. In _2021 IEEE/ACM 6th International Workshop on Metamorphic Testing (MET)_. IEEE, 32–36.
  * Deng et al. (2022) Yao Deng, Xi Zheng, Tianyi Zhang, Huai Liu, Guannan Lou, Miryung Kim, and Tsong Yueh Chen. 2022.  A declarative metamorphic testing framework for autonomous driving.  _IEEE Transactions on Software Engineering_ (2022).
  * Ferreira et al. (2021) Raul Sena Ferreira, Jean Arlat, Jérémie Guiochet, and Hélène Waeselynck. 2021.  Benchmarking safety monitors for image classifiers with machine learning. In _2021 IEEE 26th Pacific Rim International Symposium on Dependable Computing (PRDC)_. IEEE, 7–16.
  * Gambi et al. (2019a) Alessio Gambi, Tri Huynh, and Gordon Fraser. 2019a.  Generating effective test cases for self-driving cars from police reports. In _Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 257–267.
  * Gambi et al. (2019b) Alessio Gambi, Marc Mueller, and Gordon Fraser. 2019b.  Automatically testing self-driving cars with search-based procedural content generation. In _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis_. 318–328.
  * Guerin et al. (2022) Joris Guerin, Kevin Delmas, and Jérémie Guiochet. 2022.  Evaluation of runtime monitoring for UAV emergency landing. In _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 9703–9709.
  * Haq et al. (2022) Fitash Ul Haq, Donghwan Shin, and Lionel Briand. 2022.  Efficient online testing for DNN-enabled systems using surrogate-assisted and many-objective optimization. In _Proceedings of the 44th international conference on software engineering_. 811–822.
  * Haq et al. (2023) Fitash Ul Haq, Donghwan Shin, and Lionel C Briand. 2023.  Many-objective reinforcement learning for online testing of dnn-enabled systems. In _2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)_. IEEE, 1814–1826.
  * Hendrycks and Dietterich (2019) Dan Hendrycks and Thomas Dietterich. 2019.  Benchmarking Neural Network Robustness to Common Corruptions and Perturbations.  _Proceedings of the International Conference on Learning Representations_ (2019).
  * Henriksson et al. (2019) Jens Henriksson, Christian Berger, Markus Borg, Lars Tornberg, Cristofer Englund, Sankar Raman Sathyamoorthy, and Stig Ursing. 2019.  Towards structured evaluation of deep neural network supervisors. In _2019 IEEE International Conference On Artificial Intelligence Testing (AITest)_. IEEE, 27–34.
  * Humbatova et al. (2020) Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, Andrea Stocco, and Paolo Tonella. 2020.  Taxonomy of real faults in deep learning systems. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_. 1110–1121.
  * Humbatova et al. (2021) Nargiz Humbatova, Gunel Jahangirova, and Paolo Tonella. 2021.  Deepcrime: mutation testing of deep learning systems based on real faults. In _Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis_. 67–78.
  * Hussain et al. (2022) Manzoor Hussain, Nazakat Ali, and Jang-Eui Hong. 2022.  DeepGuard: a framework for safeguarding autonomous driving systems from inconsistent behaviour.  _Automated Software Engineering_ 29, 1 (2022), 1.
  * Kim et al. (2019) Jinhan Kim, Robert Feldt, and Shin Yoo. 2019.  Guiding deep learning system testing using surprise adequacy. In _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. IEEE, 1039–1049.
  * Koubâa et al. (2017) Anis Koubâa et al. 2017\.  _Robot Operating System (ROS)._ Vol. 1.  Springer.
  * Lakshminarayanan et al. (2017) Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017.  Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. In _Advances in Neural Information Processing Systems_ , I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc.  https://proceedings.neurips.cc/paper_files/paper/2017/file/9ef2ed4b7fd2c810847ffa5fa85bce38-Paper.pdf
  * LeoRover (2022) LeoRover. 2022.  LeoRover Dataset.  https://www.kaggle.com/datasets/aleksanderszymaski/full-track.  Accessed on Jul 30, 2023.
  * LeoRover (2023) LeoRover. 2023.  LeoRover.  https://github.com/LeoRover.  Accessed on Jul 30, 2023.
  * Lindvall et al. (2017) Mikael Lindvall, Adam Porter, Gudjon Magnusson, and Christoph Schulze. 2017.  Metamorphic model-based testing of autonomous systems. In _2017 IEEE/ACM 2nd International Workshop on Metamorphic Testing (MET)_. IEEE, 35–41.
  * Lu et al. (2022) Chengjie Lu, Yize Shi, Huihui Zhang, Man Zhang, Tiexin Wang, Tao Yue, and Shaukat Ali. 2022.  Learning configurations of operating environment of autonomous vehicles to maximize their collisions.  _IEEE Transactions on Software Engineering_ 49, 1 (2022), 384–402.
  * Mullins et al. (2018) Galen E Mullins, Paul G Stankiewicz, R Chad Hawthorne, and Satyandra K Gupta. 2018.  Adaptive generation of challenging scenarios for testing and evaluation of autonomous vehicles.  _Journal of Systems and Software_ 137 (2018), 197–215.
  * Murphy and Kaiser (2009) Christian Murphy and Gail E Kaiser. 2009.  Metamorphic runtime checking of non-testable programs.  (2009).
  * Murphy et al. (2009) Christian Murphy, Kuang Shen, and Gail Kaiser. 2009.  Automatic system testing of programs without test oracles. In _Proceedings of the eighteenth international symposium on Software testing and analysis_. 189–200.
  * Pei et al. (2017) Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2017.  Deepxplore: Automated whitebox testing of deep learning systems. In _proceedings of the 26th Symposium on Operating Systems Principles_. 1–18.
  * Riccio and Tonella (2023) Vincenzo Riccio and Paolo Tonella. 2023.  When and Why Test Generators for Deep Learning Produce Invalid Inputs: an Empirical Study. In _2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)_. IEEE, 1161–1173.
  * Ruff et al. (2021) Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojciech Samek, Marius Kloft, Thomas G Dietterich, and Klaus-Robert Müller. 2021.  A unifying review of deep and shallow anomaly detection.  _Proc. IEEE_ 109, 5 (2021), 756–795.
  * Scheuer et al. (2023) Franz Scheuer, Alessio Gambi, and Paolo Arcaini. 2023.  STRETCH: Generating Challenging Scenarios for Testing Collision Avoidance Systems. In _2023 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 1–6.
  * Segura et al. (2016) Sergio Segura, Gordon Fraser, Ana B Sanchez, and Antonio Ruiz-Cortés. 2016.  A survey on metamorphic testing.  _IEEE Transactions on software engineering_ 42, 9 (2016), 805–824.
  * Stocco et al. (2022a) Andrea Stocco, Paulo J Nunes, Marcelo D’Amorim, and Paolo Tonella. 2022a.  Thirdeye: Attention maps for safe autonomous driving systems. In _Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering_. 1–12.
  * Stocco et al. (2022b) Andrea Stocco, Brian Pulfer, and Paolo Tonella. 2022b.  Mind the gap! a study on the transferability of virtual vs physical-world testing of autonomous driving systems.  _IEEE Transactions on Software Engineering_ (2022).
  * Stocco and Tonella (2022) Andrea Stocco and Paolo Tonella. 2022.  Confidence-driven weighted retraining for predicting safety-critical failures in autonomous driving systems.  _Journal of Software: Evolution and Process_ 34, 10 (2022), e2386.
  * Stocco et al. (2020) Andrea Stocco, Michael Weiss, Marco Calzana, and Paolo Tonella. 2020.  Misbehaviour prediction for autonomous driving systems. In _Proceedings of the ACM/IEEE 42nd international conference on software engineering_. 359–371.
  * Sun et al. (2022) Yang Sun, Christopher M Poskitt, Jun Sun, Yuqi Chen, and Zijiang Yang. 2022.  LawBreaker: An approach for specifying traffic laws and fuzzing autonomous vehicles. In _Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering_. 1–12.
  * Tian et al. (2018) Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018.  Deeptest: Automated testing of deep-neural-network-driven autonomous cars. In _Proceedings of the 40th international conference on software engineering_. 303–314.
  * Wang et al. (2020) Huiyan Wang, Jingwei Xu, Chang Xu, Xiaoxing Ma, and Jian Lu. 2020.  Dissector: Input validation for deep learning applications by crossing-layer dissection. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_. 727–738.
  * Weiss and Tonella (2023) Michael Weiss and Paolo Tonella. 2023.  Uncertainty quantification for deep neural networks: An empirical comparison and usage guidelines.  _Software Testing, Verification and Reliability_ (2023).
  * Xiao et al. (2021) Yan Xiao, Ivan Beschastnikh, David S Rosenblum, Changsheng Sun, Sebastian Elbaum, Yun Lin, and Jin Song Dong. 2021.  Self-checking deep neural networks in deployment. In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. IEEE, 372–384.
  * Zhang et al. (2019) Man Zhang, Shaukat Ali, Tao Yue, Roland Norgren, and Oscar Okariz. 2019.  Uncertainty-wise cyber-physical system test modeling.  _Software & Systems Modeling_ 18 (2019), 1379–1418.
  * Zhang et al. (2016) Man Zhang, Bran Selic, Shaukat Ali, Tao Yue, Oscar Okariz, and Roland Norgren. 2016.  Understanding uncertainty in cyber-physical systems: a conceptual model. In _Modelling Foundations and Applications: 12th European Conference, ECMFA 2016, Held as Part of STAF 2016, Vienna, Austria, July 6-7, 2016, Proceedings 12_. Springer, 247–264.
  * Zhang et al. (2018) Mengshi Zhang, Yuqun Zhang, Lingming Zhang, Cong Liu, and Sarfraz Khurshid. 2018.  DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems. In _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_. 132–142.
  * Zhong et al. (2022) Ziyuan Zhong, Gail Kaiser, and Baishakhi Ray. 2022.  Neural network guided evolutionary fuzzing for finding traffic violations of autonomous vehicles.  _IEEE Transactions on Software Engineering_ (2022).
  * Zhou et al. (2023) Yuan Zhou, Yang Sun, Yun Tang, Yuqi Chen, Jun Sun, Christopher M Poskitt, Yang Liu, and Zijiang Yang. 2023.  Specification-based Autonomous Driving System Testing.  _IEEE Transactions on Software Engineering_ (2023).
  * Zhou and Sun (2019) Zhi Quan Zhou and Liqun Sun. 2019.  Metamorphic testing of driverless cars.  _Commun. ACM_ 62, 3 (2019), 61–67.