# Benchmarking Unsupervised Online IDS for Masquerade Attacks in CAN

Pablo Moriano12, , Steven C. Hespeler1, Mingyan Li3, Robert A. Bridges3
2Corresponding author. 1Computer Science and Mathematics Division; 3Cyber
Resilience and Intelligence Division
Oak Ridge National Laboratory, Oak Ridge, TN 37830, USA
$\\{$moriano, hespelersc, lim3<EMAIL_ADDRESS>
###### Abstract

Vehicular controller area networks (CANs) are susceptible to masquerade
attacks by malicious adversaries. In masquerade attacks, adversaries silence a
targeted ID and then send malicious frames with forged content at the expected
timing of benign frames. As masquerade attacks could seriously harm vehicle
functionality and are the stealthiest attacks to detect in CAN, recent work
has devoted attention to compare frameworks for detecting masquerade attacks
in CAN. However, most existing works report offline evaluations using CAN logs
already collected using simulations that do not comply with domain’s real-time
constraints. Here we contribute to advance the state of the art by introducing
a benchmark study of four different non-deep learning (DL)-based unsupervised
online intrusion detection systems (IDS) for masquerade attacks in CAN. Our
approach differs from existing benchmarks in that we analyze the effect of
controlling streaming data conditions in a sliding window setting. In doing
so, we use realistic masquerade attacks being replayed from the ROAD dataset.
We show that although benchmarked IDS are not effective at detecting every
attack type, the method that relies on detecting changes at the hierarchical
structure of clusters of time series produces the best results at the expense
of higher computational overhead. We discuss limitations, open challenges, and
how the benchmarked methods can be used for practical unsupervised online CAN
IDS for masquerade attacks.

###### Index Terms:

Streaming Data, Online Algorithms, Machine Learning, Anomaly Detection, In-
Vehicle Security, CAN Bus

##  1 Introduction

Today’s vehicles are intricate cyber-physical systems with growing
connectivity capabilities to interact with vehicle-to-everything (V2X)
technologies [1, 2, 3]. This increasing connectivity expands the attack
surface to adversaries that can access in-vehicle networks. The Controller
Area Network (CAN) is the most prevalent protocol used on in-vehicle networks
allowing hundreds of electronic control units (ECUs) communicate vehicle
functionality [4, 5, 6]. While the lack of security features makes CAN
lightweight and inexpensive, it is susceptible to attacks that aim to
manipulate communications and significantly degrade in-vehicle network
performance. This has been shown to result in life threatening incidents such
as unintended acceleration, brake manipulation, and rogue steering wheel
turning [7, 8].

CAN attacks are usually classified as fabrication, suspension, and masquerade
attacks [9, 10]. From these, masquerade attacks are the hardest to detect
because they do not perturb the regular frame timing which can be detected
using time-based methods [11, 12]. Instead, in masquerade attacks, the
adversary first silences a targeted ECU and then injects spoofed frames which
alters the content of regular frames instead of time-related patterns. This
makes masquerade attacks the stealthiest often having an effect on the regular
evolution of vehicle states usually measured by sensors [13]. Therefore,
recent research efforts have been channeled on designing and deploying
intrusion detention systems (IDS) against advanced masquerade attacks in CAN
[14, 15, 16, 17].

IDS for detecting masquerade attacks in CAN are often based on unsupervised
anomaly detection techniques aiming to characterize the regular relationships
between physical signals in a vehicle [15, 18]. Some of them focus on mining
time series relationships based on deep learning (DL) models that consequently
have high computational cost [19, 16, 20]. In addition, most of existing CAN
IDS approaches are tested offline using datasets or CAN logs already collected
from a real or simulated environment. This has been synthesized in existing
benchmarking frameworks and comparative studies in the CAN IDS space [21, 18].
The offline evaluation approach is different from online evaluation in that
the latter is performed using streaming CAN data from a vehicle, simulation,
or data log replay [22]. Therefore, offline evaluation hinders the
understanding of realistic capabilities and limitations of CAN IDS in a
streaming environment. By benchmarking non DL-based unsupervised online IDS
for masquerade attacks in CAN, we seek to provide insights intro the
effectiveness and limitations of these methods in resource-constrained and
streaming environments.

In this paper, we address this gap of knowledge by introducing an empirical
and unbiased benchmark of four different non-DL-based unsupervised online IDS
for masquerade attacks: Matrix Collelation Distribution, Matrix Correlation
Correlation, Ganesan17 [23], and Moriano22 [15]. Here, Matrix Correlation
Distribution and Matrix Correlation Correlation methods are simple proposed
baselines while Ganesan17 and Moriano22 are already proposed methods in the
literature. We test them in simulated masquerade attacks from the ROAD dataset
[10]. We rely on a sliding window approach to simulate data log replays of a
continuous data stream [24, 25]. We analyze the joint effect of window size
and offset on the AUC-ROC as evaluation metric for intrusion detection across
the different attacks in the ROAD dataset. We also report an analysis of the
testing time per window (TTW) [20] of CAN IDS as a proxy of inference time and
discuss insights into the real-time detection capability of these approaches.
We further contribute to advance the state of the art in this topic by making
publicly available the code of the IDS considered in this study [26]. This
furthers research in this topic by allowing other researchers to replicate our
results and compare new algorithms with the benchmarked IDS.

Our results indicate that overall the benchmarked unsupervised online CAN IDS
are not effective in consistently detecting every attack type in the ROAD
dataset. Attacks in the ROAD dataset have different levels of impact on the
regular correlation patterns of signals. However, evaluation performance
metrics tend to favor Moriano22 and Matrix Correlation Distribution methods.
This aligns with the expectation that impacting correlations across signals
tends to disrupt the hierarchy of their relationships and their pairwise
similarity distribution. Notably, among the variety of attacks in the ROAD
dataset, max_engine_attack and correlated_attack are the ones that can be
detected more easily across CAN IDS. Our results also shed light on the impact
that sliding window parameters (i.e., window length and offset) have on the
detection task. Overall, across methods and attack types, we identify
conditions that make certain algorithms more suitable to operate in a real-
time detection environment.

##  2 Background

###  2.1 CAN Protocol

CAN is a message-based communication standard for interconnecting ECUs within
industrial application and vehicles. Developed by Bosch in 1980s, CAN operates
at Open Systems Connection (OSI) lowest two layers (physical and data-link
layers), and is renowned for its reliability and efficiency in automotive and
industrial networking applications. CAN facilitates communication among ECUs
without requiring a central bus controller, using two-wire bus for seamless
message transmission through frames containing arbitration, control, data, and
error-checking fields, as illustrated in Fig. 1. Of particular relevance to
this paper’s context are two fields, the 11-bit Arbitration ID and the up-
to-8-byte data field.

Figure 1: CAN frame illustration.

The arbitration ID serves as a unique identifier for messages transmitted over
the bus. This ID is crucial for distinguishing between different messages and
plays a key role in determining message priority. Lower ID values indicate
higher message priority, allowing critical messages to be sent without delay.
The data field contains the actual information being transmitted between ECUs.
The payload consists of up to 8 bytes of data. Each or multiple bytes
represent a specific piece of information, known as “signal”, such as sensor
readings or status information. The data payload is essential for exchanging
real-time data between ECUs. CAN frames with a certain arbitration ID are
transmitted at a predetermined frequency with fixed payload structure. For
instance, the power control module transmits ID 0x102 with data field
containing engine RPM, vehicle speed, and odometer signals every 0.05s [10].

CAN also includes a built-in sophisticated fault confinement mechanism to
handle anomalies such as faulty cables, noise, or malfunctioning CAN nodes
etc. Each node constantly monitors its performance and creates an “error
frame” in response to detecting a CAN error. There are three possible CAN
error states: error active state, error passive state, and error bus-off
state, with the last one being prohibited from sending or receiving CAN
messages. Prior work by Cho and Shin [27] provides more details on CAN error
handling.

###  2.2 CAN Vulnerabilities

Despite being robust, the CAN protocol can suffer from a multitude of cyber
attacks due to its inherent vulnerabilities. Firstly, the absence of
communication encryption protection leaves CAN messages susceptible to
interception and manipulation, allowing the attackers to eavesdrop on
communications or tap into and alter the network communication without being
detected. Secondly, the lack of authentication mechanisms implies that nodes
within the CAN network cannot verify the identities of message senders, making
it easier for the unauthorized parties to gain access and tamper with
transmission data. Moreover, the broadcasting nature of the CAN protocol,
notably when compounded by lack of encryption and authentication, essentially
allows any node to receive messages intended for others, and to inject
illegitimate messages at its will. By exploiting such vulnerabilities, a
malicious node can potentially manipulate message arbitration ID to disrupt
message delivery priorities, launch Denial of Service (DOS), or launch
impersonation attacks, to create communication errors or system malfunctions.
Finally, CAN vulnerabilities can also lead to exploits of higher-level
protocols. An example would be an attack on ECUs’ Unified Diagnostic Services
(UDS)—a diagnostic communication protocol used within automotive electronics,
by injecting and manipulating CAN messages [28].

In terms of attacks that manipulate CAN messages, there are three categories
of exploits:

  * •

Fabrication attacks, employs a compromised ECU to inject messages with
malicious IDs and data fields. Typically CAN IDs are deliberately set to low
values, signifying high priority. Continuously injecting such high-priority
messages can eventually disrupt legitimate ECUs from transmitting theirs
leading to DoS-type attacks or targeting vehicle functions.

  * •

Suspension attacks, where the adversary seeks to compromise an ECU to prevent
it from transmitting some or all of its messages. This could be achieved by
removing an ID frame(s) to disconnect the ECU from the CAN bus, thereby
halting the transmission of corresponding signal values.

  * •

Masquerade attacks, the most sophisticated category, involve the adversary
initially suspending messages from a specific ID of a targeted ECU, then
spoofing messages using this ID at a realistic frequency, thus effectively
masquerading as the compromised ECU.

##  3 Related Work

Here we review and contrast previous offline benchmark studies on CAN IDS
(Section 3.1) and online CAN IDS methods (Section 3.2).

###  3.1 Prior Work Closely Related to the Present Study

Ji et al. [21] presented a comparative analysis of four statistical-based CAN
IDSes using entropy, clock skew, ID sequences, and CAN bus throughput.
Benchmarked methods relied only on ID timestamp arrivals. Their evaluation was
done in simulated attack datasets. They found that the clock skew method
performed best across attacks while the entropy method falls short on
detecting replay attacks.

Dupont et al. [29] introduced a unifying framework for CAN IDS evaluation of
time-based methods. In doing so, they used data from two live vehicles and a
CAN prototype. They used both simulated attacks and publicly available
datasets with attacks [30]. They found that among compared methods, they
perform well only on attacks that generate significant disruptions in CAN
traffic. They suggested that methods that consume bit representations and
semantics of CAN messages will perform better.

Stachowski et al. [31] introduced a framework for benchmarking CAN IDS. Their
approach focused on the online evaluation of three existing solutions from
identified vendors, so that CAN IDS are tested in real vehicles for real-time
intrusion detection. Targeted ID attacks were introduced in test vehicles
while vehicles were in motion and stationary. None of the tested CAN IDS was
found effective for detecting the variety of attacks.

Blevins et al. [32] focused on bechmarking time-based CAN IDS on the fuzzy and
targetted ID attacks from the ROAD dataset. Bechmarked methods include mean
inter-message time, fitting a Gaussian curve, kernel density estimation, and
binning. They found that the binning method outperformed the other methods,
specially those fitting a distribution on inter-arrival times, in terms on
AUC-ROC, AUC-PR, and F1 score. They reported a latency analysis for the
binning algorithm.

Agbaje et al. [33] introduced a framework for comprehensive performance
evaluation analysis of eight state-of-the-art CAN IDS. The proposed framework
provides a consistent methodology to evaluate and asses CAN IDS. They focus on
using previously published datasets recorded from a real vehicle and report
traditional classifier performance evaluation metrics to reduce uncertainty
when comparing IDS approaches from different sources.

Pollicino et al. [34] introduced an experimental comparison on the performance
evaluation of eight CAN IDS algorithms over two different datasets, an in-
house one and one that is already publicly available (i.e., OTIDS [30]). The
algorithms they bechmarked consume only time-based information from CAN frames
as they are intended for detection of fabrication attacks. They open-sourced
code implementations of these algorithms to allow reproducibility of research
results and detail an unbiased experimental comparison.

Sharmin et al. [18] reported results of a comparative evaluation of four
statistical and two ML-based CAN IDS in the ROAD dataset, including masquerade
attacks. In addition to reporting traditional evaluation metrics such as
accuracy, precision, recall, and F1 score, they included balanced accuracy,
informedness, markedness, and Matthews correlation coefficient (MCC) which are
better suited for handling imbalanced datasets [35]. They also reported
training and testing time for each IDS. They found that frequency-based
approaches generally perform well at detecting fabrication attacks while other
algorithm categories did not perform well based on low MCC scores.

Compared to the benchmark studies mentioned above, the present paper is unique
because it focuses on online IDS methods for masquerade attacks. Note that
previous benchmark studies focused on: (1) comparing CAN IDS methods in an
offline fashion (see works [21, 29, 32, 34, 18]); and (2) performing
evaluation tests only on fabrication attacks (see works [21, 29, 31, 32, 33,
34]). By benchmarking CAN IDS methods on these two key aspects, we contribute
to fill this gap of knowledge by analyzing the performance of unsupervised
online CAN IDS in online fashion and testing them on the most stealthy CAN bus
attacks, i.e., masquerade attacks. Our contribution enriches the state of the
art by introducing a more realistic assessment of CAN IDS behavior in a
realistic but constrained environment.

###  3.2 Other Prior Work Related to the Present Study

Desta el al. [36] proposed a CAN IDS based on a LSTM sequence predictor model
focused on predicting ID sequences. They performed in-house collection of CAN
data using a real car. They replayed a CAN data log using the socketCAN API to
evaluate their CAN IDS. The found that they can obtain better results by using
the log loss of the predicted ID and the true ID.

Sunny et al. [37] proposed a hybrid CAN IDS that leverages patterns of
recurring messages and time interval between them. The proposed approach
produces fast inference times being informed by best and worse time intervals
between two consecutive frames. As CAN messages are expected to be published
every 2 ms, they conclude that the proposed method can be used for real-time
settings on simulated attack scenarios.

Jedh et al. [38] evaluated four architecture designs for real-time CAN IDS
under malicious speed reading message injections. They focus on a single ML-
based IDS that captures the pattern of the sequences of CAN messages
represented with a direct graph. They showed that the optimal architecture for
deployment a real-time CAN IDS is based on two processes, a process for CAN
monitoring and another one exclusively for anomaly detection.

Jadidbonab et al. [39] introduced an automotive security testbed that combines
a full-scale in-vehicle network (i.e., vector CANoe) and a car simulator
(i.e., CARLA) to generate network traffic in realistic scenarios. They tested
ML-based IDS on their testbed and found that online algorithms performed
poorer when compared to training offline classifiers on previously collected
logs.

##  4 Methods

Common to all the methods benchmarked in this work is our focus on processing
a set of time series representing signals in the CAN bus. For that, we assume
that we have access to the time series representation obtained from CAN logs
captured during a vehicle’s drive. Recall that the subsequent methods are
expected to work in online fashion. This means that they process a time series
stream to provide near real-time alerts.

Note that although streaming and online algorithms are often used as synonyms,
here we make the distinction between these two and point out that streaming
algorithms process each point as soon as it is received, while online
algorithms process small batches at a time [25, 40, 41]. In our work, we use
the latter definition, when we refer to online algorithms, i.e., algorithms
that repeatedly process small parts of a continuous time series stream. Note
that our inclination to benchmark unsupervised learning methods is guided in
part by the fact that it is challenging to apply supervised methods on
streaming data due to their static nature [42].

###  4.1 Threat Model

We assume a scenario where an adversary has already gained access to the CAN
bus (e.g., by the OBD-II port or through wireless communication) [43]. Here,
the adversary compromises two ECUs, one as a strong attacker (i.e., fully
compromising an ECU including capabilities for injecting arbitrary attack
messages) and the other as a weak attacker (i.e., being able to stop/suspend
the ECU from transmitting certain messages) [9].

###  4.2 Time Windows

The presented benchmark is based on sliding time windows of the stream of time
series data. The assumption behind using windows is that more recent
information is more relevant for decision-making than past data. In
particular, we focus on using _sliding windows_ of fixed size for anomaly
detection [44]. The windows are defined in terms of the number of observations
or sequence-based windows [24]. This involves a sequence of $\omega$-length
windows (i.e., $\omega\in\mathbb{Z}$) $W=\\{W_{j}:j\geq 1\\}$ with a sliding
step (or offset $\delta\in[1,\omega]$) and observations arriving sequentially.
Let $Y_{t}$ for $t\geq 0$ be a set of time series in a given window $W$, then
$Y_{t}$ contains time series over the interval
$[\delta\times(j-1),\delta\times(j-1)+\omega]$. Hence, $Y_{t}$ is a set of
time series of the most recent $\omega$ observations up to time $t$ with
$\delta$ observations that expire. Figure 2 shows this concept visually.

Figure 2: Sliding windows concept. The stream of time series data is
partitioned using windows of length $\omega$. The sliding windows step, or
offset, is $\delta$.

###  4.3 Experimental Design

Here we describe our proposed benchmarking framework. More details can be
found in Fig. 3. Our proposed framework has two phases: i) $P_{1}$: training
phase and ii) $P_{2}$: testing phase. These phases have some sequential tasks.
Some of them are unique to one phase while others are common to both phases.
The tasks are: i) $T_{1}$: data collection and preprocessing, $T_{2}$: model
fitting, and $T_{3}$: model testing. $P_{1}$ consists of tasks $T_{1}$ and
$T_{2}$, whereas $P_{2}$ consists of tasks $T_{1}$, $T_{2}$, and $T_{3}$.

Figure 3: An overview of our proposed benchmark framework.

The proposed benchmark framework uses time windows for data processing and
attack detection within the same time window. This means that our benchmark is
focused on detecting attacks at the window level rather than at the message
level. Note that both phases are considered to be accomplished in a light-
weight device (e.g., Raspberry Pi or Nvidia Jetson Nano). We now explain the
technical aspects of the framework in more detail.

####  4.3.1 Training Phase (P1):

_1) Data Collection and Preprocessing (T1):_

The first task common to all the algorithms in our benchmark is the
configuration of the data collection and preprocessing pipeline. We assume
that the benchmarked algorithms run on light-weight devices connected to the
CAN bus through the OBD-II port or a dedicated ECU with direct access to the
CAN bus. This setup allows continuous collection of raw CAN messages along
with its translation from binary to time series using its corresponding
decoding instructions. Note that there are available open-source CAN data
loggers such as SavvyCAN and SocketCAN and commercial tools such as CANalyzer
and VehicleSpy. We also assume that for translating CAN binary payloads to
multidimensional time series, algorithms are loaded with OEM’s DBC file or use
CAN decoding software such as LibreCAN, CAN-D, or CANMatch [45, 46, 47]. For a
more comprehensive discussion on CAN signals reverse engineering, we refer the
reader to the survey by Buscemi et al. [3].

We focus on processing a set of $n$ signals, i.e.,
$\mathcal{S}=\\{s_{1},s_{2},\ldots,s_{n}\\}$. Here $n$ means the total number
of decoded signals. As each of these time series is generated at a
characteristic frequency (imposed by their corresponding ID), we linearly
interpolated them in common timesteps to have the same frequency producing
evenly spaced time series. We use a common frequency of 100Hz to take into
account the fact that most IDs do not send messages above this frequency
threshold. This ensures that $\forall s_{i}\in\mathcal{S},|s_{i}|=t_{0}$,
where $t_{0}$ is the total number of time steps in the training set.

This multivariate time series data is stored in a matrix
$\mathbf{X}_{train}\in\mathbb{R}^{t_{0}\times n}$. On this matrix, we discard
any constant time series and apply min-max normalization to the remaining time
series.

_2) Model Fitting (T2):_

We fit data-driven models for each of the benchmarked methods using the
training data, i.e., $\mathbf{X}_{train}$. Fitted models in this task are then
used as the regular (normal) state representation from which we derive
comparisons with models fitted at the testing stage. The details of the fitted
models for each of the becnhmarked methods are in Section 4.4.

####  4.3.2 Testing Phase (P2):

The testing phase executes similar tasks as already discussed in the training
phase (see tasks T1 and T2 in Section 4.3.1) and adds T3. Note that while P1
is run only once, the tasks in P2 are run continuously and sequentially as new
messages are processed and there is a need to check for anomalies. We detail
here changes with respect to specific parameters.

_1) Data Collection and Preprocessing (T1):_

In the testing stage, data is processed using a sliding window approach (see
more details in Section 4.2). Specifically, smaller sections of
$\mathbf{X}_{test}$ of length $\omega$ are selected to create a view of the
steaming ($\in\mathbb{R}^{\omega\times n}$). Benchmarked methods consume this
batch of data to arrive at conclusions.

_2) Model Fitting (T2):_

We fit the benhmarked methods with their specific models on
$\mathbf{X}_{test}$. The specifics of the fitted models are discussed in
Section 4.4.

_3) Model Testing (T3):_

This step entails a data-driven decision-making for deciding if a particular
window is containing an anomaly or not. The specifics of the criteria for
detection are discussed in Section 4.4.

###  4.4 Benchmarked Algorithms

This subsection describes in detail each of the four algorithms used in this
research. For the sake of readability, we associate each algorithm name of the
first author and the last two digits of the publication year (when
applicable). Specifically, we describe the following methods: Matrix
Correlation Distribution (see Section 4.4.1), Matrix Correlation Correlation
(see Section 4.4.2), Ganesan17 [23] (see Section 4.4.3), and Moriano22 [15]
(see Section 4.4.4).

We notice that none of the benchmarked algorithms has been distributed with a
corresponding implementation. In addition, many of their implementation
details are not provided. To address these limitations, we make available our
referenced implementations and provide a detailed description of the
assumptions and parameters tested in each of the algorithms. This allows
practicioners in the area the opportunity to easily replicate our experiments
and benchmark novel signal-based detection algorithms with respect to the
state of the art. Note that becnhmarked algorithms produce anomaly scores in
the range 0 to 1 that is being used for computing evaluation metrics.

All method computes pairwise Pearson correlations among time series in
$\mathbf{X}_{train}$ (and $\mathbf{X}_{test}$) obtaining
$\mathbf{R}_{train}\in\mathbb{R}^{n\times n}$ (and corresponding
$\mathbf{R}_{test}\in\mathbb{R}^{n\times n}$), symmetric matrices whose
$r_{ij}$ entry is the Pearson correlation coefficient [48] between the signals
$s_{i}$ and $s_{j}$. Pearson correlation values that are close to $\pm 1.0$
indicate strong positive or negative correlation. Note that time series that
have positive correlations are expected to move jointly (i.e., when one time
series increases or decreases, the other time series also increases or
decreases). As vehicle dynamics are constrained by laws of nature, we expect
that: (1) clusters of correlated signals emerge, for example, as there is an
increase in the speed of the vehicle, there is a subsequent increase in the
speedometer reading and the speed of the four wheels, and (2) such clusters
are disturbed upon a cyberattack given that time series correlations could be
broken or changed significantly.

We let $\mathbf{U}$ denote the strictly upper triangular matrix of
$\mathbf{R}$, i.e., $u_{ij}=r_{ij}$ if $i<j$ and $u_{ij}=0$ if $i\geq j$. We
create strictly upper triangular matrices
$\mathbf{U}_{train},\mathbf{U}_{test}$, respectively, for both training and
testing correlation matrices ($\mathbf{R}_{train}$ and $\mathbf{R}_{test}$) to
estimate pairwise similarity distributions among time series.

####  4.4.1 Implementation of Matrix Correlation Distribution

This method seeks statistical significance of the signals’ pairwise Pearson
correlations. Using $\mathbf{U}_{train},\mathbf{U}_{test}$ as above, we use
the Mann-Whitney U test [49] $p$-value to create an anomaly probability score;
specifically, one minus the $p$-value is the anomaly score so that a larger
score is more anomalous. The Mann-Whitney U test is a non-parametric test
often used to test the difference in location between distributions. As the
Mann-Whitney U test null hypothesis is that the Pearson similarity
distribution underlying benign conditions and attack conditions is the same,
we use one minus the $p$-value as the anomaly score. Intuitively, higher
$p$-values suggest not rejecting the null hypothesis; this translates to a
lower anomaly probability score. Conversely, lower $p$-values suggest
rejecting the null hypothesis; this translates to a higher anomaly probability
score.

####  4.4.2 Implementation of Matrix Correlation Correlation

This method considers the upper triangular entries of
$\mathbf{U}_{train},\mathbf{U}_{test}$ (the signals’ pairwise Pearson
correlation coefficients) and computes the statistical significance of their
Spearman correlation [50], a measure in $[-1,1]$ of their members ordinal
rank. Simply unravel the upper triangular entries of
$\mathbf{U}_{train},\mathbf{U}_{test}$ into two sequences and compute their
Spearman correlation. Note that Spearman correlation furnishes, alongside the
correlation value, a $p$-value giving the likelihood that the two input
vectors are from uncorrelated systems (so low $p$-value corresponds to the two
input vectors coming from correlated systems). We focus on the $p$-value of
the Spearman correlation as this captures the reliability of the correlation.
The Spearman correlation coefficient is better suited for non-normally
distributed continuous data, ordinal data, or data with potential outliers as
a measure of monotonic association [51]. Pearson correlation is also scaled in
the range $\pm 1$ indicating a constantly increasing or decreasing monotonic
association while $0$ indicating no monotonic association. The Spearman
correlation null hypothesis is that there is no correlation between the
strictly upper triangular correlations in training and testing (opposite to
the first test above); hence, we use the $p$-value as the probability score.
Here, higher $p$-values suggest not rejecting the null hypothesis; this
translates to a higher anomaly score. Conversely, lower $p$-values suggest
rejecting the null hypothesis; this translates to a lower anomaly score.

####  4.4.3 Implementation of Ganesan17

This method focuses on computing clusters of time series defining a context
(such as aggressive or sudden driving). Note that this method does not rely on
cluster characterization during training and only relies on processing testing
windows. Specifically, for each time window of length $\omega$, the DBSCAN
clustering method [52] is applied to a distance matrix
$\mathbf{D}_{test}\in\mathbb{R}^{n\times n}$ computed as
$2*(1-\mathbf{R}_{test})$, i.e., a squared Euclidean distance to identify
clusters [53]. DBSCAN is a density-based non-parametric clustering algorithm
that works by grouping points with many nearby neighbors and marking as
outliers the points that lie alone in low-density regions. By analyzing
clusters of time series as opposed to individual pairs, this method aims to
capture richer relationships between signals. We used the same parameter
values used in Ganesan et al. [23], namely $eps=1$ capturing how close points
should be to each other to be considered part of a cluster and
$min\\_samples=1$ capturing the minimum number of points to form a dense
region.

For detection, once clusters of time series have been identified, pairwise
cross correlations among time series within clusters are computed and compared
with those expected for that cluster. Specifically, for each correlation pair,
the deviation from the mean correlation value for that cluster is computed.
Among each possible pairs, the maximum deviation (or error) is extracted and
compared in terms of the number of standard deviations from the mean. Among
all the errors in a single cluster, only the maximum error among clusters is
used as reference. Finally, among all the errors among clusters, the
probability of being anomalous is estimated based on the normal distribution
approximation of maximum errors evaluated at a particular error, i.e., the
probability that the distribution of maximum errors will take a value less or
equal than the error.

####  4.4.4 Implementation of Moriano22

This method focuses on computing the similarity of hierarchical clusterings of
time series in training and testing. Specifically, pairwise correlations
matrices $\mathbf{R}_{train}$ and $\mathbf{R}_{test}$ are used to feed an
agglomerative hierarchical clustering (AHC) algorithm [54]. We used a Ward’s
linkage as a metric for dissimilarity between clusters as it maximizes
detection results as in the original paper [15]. The output of the ACH is a
hierarchical clustering or a hierarchy of clusters (i.e., a set of nested
clusters that are organized in a tree-like diagram known as dendrogram).

For detection, in each time window of length $\omega$, we compute the
clustering similarity $s$ between the hierarchical clustering from training
(computed once) and the hierarchical clustering computed during that specific
time window. Here, similarity is used as a proxy of the distance between both
hierarchical clusterings. We compute the similarity between hierarchical
clustering using the CluSim method [55] based on the work by Gates et al.
[56]. We parametrized CluSim by letting $r=-5.0$ and $\alpha=0.9$. Here, $r$
is a scaling parameter that defines the relative importance of memberships at
different levels of the hierarchy. That is, the larger $r$, the more emphasis
on comparing lower levels of the dendrogram (zoom in). In addition, $\alpha$
is a parameter that controls the influence of hierarchical clusterings with
shared lineages. That is, the larger $\alpha$, the further the process will
explore from the focus data element, so more of the cluster structure is taken
into account into the comparison. The similarity value provided by this
method, $s$, lies in the range $[0,1]$, where 0 implies maximally dissimilar
clusters, and 1 corresponds to identical clusters. Therefore, we report the
anomaly score (or probability of detection) as $1-s$.

###  4.5 Dataset

We ran the proposed benchmark on the ROAD dataset [10]. The ROAD dataset is a
publicly available dataset for CAN IDS that contains benign and attack CAN
data collected from a real vehicle. An important feature of ROAD is that it
provides samples of masquerade attacks which are the types of attacks we are
interested in detecting with the bechmarked methods. Note that masquerade
attacks in ROAD are identical to the corresponding targeted ID captures but
with the ambient frames of the targeted ID removed during the attack period to
simulate the masquerade attack. The ROAD dataset also provides already
translated CAN time series from log files. By using ROAD in this study, we
ensure comparing the performance of proposed methods with realistic, verified,
and labelled attacks as opposed to synthetic ones. This allows performing the
comparison under realistic conditions.

We report results on the proposed benchmark in each of the masquerade attacks
provided in the ROAD dataset. Note that each of the masquerade attack files in
ROAD contains hundreds of time series coming from hundreds IDs each of them
with up to a few dozen signals. Table I shows the specific files we used from
ROAD. In particular, we used the following attacks in increasing level of
detection difficulty: correlated signal, max speedometer, max engine coolant
temperature, reverse light on, and reverse light off. In the correlated signal
attack, the correlation of the four wheel speed values is manipulated by
changing their individual values. In the max speedometer and max engine
coolant attacks, the speedometer and coolant temperature values are changed to
their maximum. In the reverse light attacks, the state of the reverse light is
modified so that it does not match the current gear of the vehicle (i.e.,
reverse light is off when the vehicle is in reverse and the reverse light is
on when the vehicle is in forward gear).

TABLE I: CAN captures used from the ROAD dataset [10]. | Description | # Files | Used | Duration (min)
---|---|---|---|---
Training | Dynamometer Various Ambient | 10 | ✓ | 108.2
Road Various Ambient | 2 | ✓ | 70.6
_Total_ | 12 | 12 | 178.8
Testing | Correlated Signal Fabrication Attack | 3 | ✗ | 1.3
Correlated Signal Masquerade Attack | 3 | ✓ | 1.3
Fuzzing Fabrication Attack | 3 | ✗ | 0.7
Max Engine Coolant Temp Fabrication Attack | 1 | ✗ | 0.4
Max Engine Coolant Temp Masquerade Attack | 1 | ✓ | 0.4
Max Speedometer Fabrication Attack | 3 | ✗ | 3.9
Max Speedometer Masquerade Attack | 3 | ✓ | 3.9
Reverse Light Off Fabrication Attack | 3 | ✗ | 2.1
Reverse Light Off Masquerade Attack | 3 | ✓ | 2.1
Reverse Light On Fabrication Attack | 3 | ✗ | 3.2
Reverse Light On Masquerade Attack | 3 | ✓ | 3.2
Accelerator Attack (In Drive) | 2 | ✗ | 2.7
Accelerator Attack (In Reverse) | 2 | ✗ | 3.2
_Total_ | 33 | 13 | 10.9

We used the longest benign capture (i.e., ambient-highway-street-driving-long)
for learning the normal state of the system ($\approx$ 1 hour of data). We use
all the available attack data in the masquerade category, i.e., three files in
the correlated attack category ($\approx$ 1.3 minutes), three files in max
speedometer attack category ($\approx$ 3.9 minutes), one file for the max
engine coolant attack category ($\approx$ 0.4 minutes), three files for
reverse light on attack category ($\approx$ 3.2 minutes), and three files for
the reverse light off attack category ($\approx$ 2.1 minutes).

###  4.6 Evaluation Metrics

This study focuses on comparing the detection capability of the benchmarked
algorithms on the ROAD dataset. As such, we report summarized results over a
range of detection thresholds and focus on the area under the receiver
operating characteristic curve (AUC-ROC) metric. The AUC-ROC reflects the
overall performance of the benchmarked algorithms based on the variation of
the true positive rate (TPR) with respect to the false positive rate (FPR) at
various thresholds [57]. Mathematically, it is expressed as

$\text{AUC-ROC}=\int_{0}^{1}\text{TPR}(\text{FPR}^{-1}(x))dx.$

AUC-ROC takes values in the interval $[0,1]$. In particular, an algorithm
whose predictions are $100\%$ wrong has an AUC-ROC of $0.00$; while an
algorithm whose predictions are $100\%$ correct has AUC-ROC of $1.00$. A
reasonable way of interpreting AUC-ROC is as the probability that the
detection algorithm ranks a random positive window more highly than a random
negative window, which is desirable in our case. Note that although the AUC-
ROC provides an overly estimated view in highly imbalance datasets [58], we
stick to AUC-ROC as the proportion of time windows overlapping with attack
intervals is not imbalanced (i.e., for most of the window length and offset
combinations is above $40\%$).

###  4.7 Performance Metrics

For assessing the real-time capabilities of the benchmarked algorithms, we
also report the latency that these methods have to infer detection decisions
on the sliding windows being processed. In doing so, we report the testing
time per window (TTW), for each combination of window length ($\omega$) and
offset ($\delta$), following the concept of testing time per packet used by
Nichelini et al. [20]. We define TTW as

$\text{TTW}=\frac{\text{Total Detection Time}}{\text{Number of Windows}}$

Our experiments were conducted on a AMD EPYC 7702 64-Core Processor Central
Processing Unit (CPU), 256 GB of RAM, running Ubuntu 22.04.4 LTS (Jammy
Jellyfish). TTW provides a way to estimate the applicability of the bechmarked
algorithms in a real-time setting.

##  5 Results

In this section, we detail the experimental evaluation of the benchmarked
algorithms on the ROAD dataset. All methods were implemented in Python 3.8.12.
We use the sklearn.metrics module to compute the evaluation metrics and the
timeit module to compute the TTW of each benchmarked algorithm. To enable the
reproducibility of the benchmarked methods, we make the code available in
GitHub.111https://github.com/pmoriano/benchmarking-unsupervised-online-IDS-
masquerade-attacks We conduct three different but complementary analysis.

First, we compute and contrast the detection performance (based on AUC-ROC) of
each algorithm per attack category in the ROAD dataset. In doing so, we
display heatmaps (Section 5.1). We hypothesize that each algorithm is better
suited for detecting specific attacks in the ROAD dataset and that there is a
range of $(\omega,\delta)$ combinations that maximizes AUC-ROC.

Second, we summarize evaluation metrics using summary statistics derived from
the heatmaps (Section 5.2). Specifically, we report the mean ($\mu$), standard
deviation ($\sigma$), median ($\eta$), minimum ($\min$), including the
combination of window length and offset that produces it, and maximum
($\max$), including the combination of window length and offset that produces
it. This analysis supports our hypothesis that there are better suited
detection algorithms for each attack category. Overall, Moriano22 detection
algorithm produces, on average, the best detection performance among attacks
in the ROAD dataset. In contrast, Ganesan17 detection algorithm produces the
worst detection performance among attacks in the ROAD dataset. Furthermore, we
found that the easiest attacks to detect are the max_engine_coolant_attack and
correlated_attack. Furthermore, we find that the max_speedometer_attack and
light_on_attack are the hardest to detect.

Lastly, we summarize TTW as a proxy of performance (Section 5.3). In
particular, we report the mean ($\mu$), standard deviation ($\sigma$), median
($\eta$), minimum ($\min$), including the combination of window length and
offset that produces it, and maximum ($\max$), including the combination of
window length and offset that produces it. This analysis reveals that more
sophisticated algorithms (e.g., Ganesan17 and Moriano22) incur performance
overhead for producing better detection evaluation metrics. Overall, Matrix
Correlation Distribution produces, on overage, the quickest TTW among attacks
in the ROAD dataset. In contrast, Moriano22 detection algorithm is at least
$4\times$ slower. It is worth noting that detection algorithms that run faster
(i.e., Matrix Correlation Distribution and Matrix Correlation Correlation)
perform even better than Ganesan17 (see Table III). We did not found
significant differences on TTW per attack categories.

###  5.1 Evaluation Metrics Contrast

Figs. 4, 5, 6, 7 show the detection performance of each detection algorithm
(i.e., Matrix Correlation Distribution, Matrix Correlation Correlation,
Ganesan17, and Moriano22, respectively). Each cell in the heatmaps depicts a
single AUC-ROC value for a specific combination of $\omega$ and $\delta$. Note
that attack types are displayed horizontally. As attack types have different
number of associated captures (see Table I), we compute the average in each
attack category and report a single heatmap summarizing the performance in a
specific category. As the standard deviation from this aggregation is
negligible with respect to the average, we do not present associated figures
here. The results are rounded at two decimal places. Specifically, we vary the
window length (i.e., $\omega$ from 50 to 400 samples in increments of 50
samples) and the offset (i.e., $\delta$ from 10 to $\omega$ in increments of
10 samples).

Fig. 4 shows results for the Matrix Correlation Distribution Method. We found
that overall, this method tends to perform better at detecting the
light_off_attack as many of the heatmap cells are above AUC-ROC of 0.70 (see
greater proportion of redder cells placed in the heatmap). Moreover, we
observe that there is a region of better performance for the max_engine_attack
characterized for lower values of $\omega\leq 300$ and $\delta\leq 100$.
Interestingly, among the highest performances across attack types is the
correlated_attack with an AUC-ROC of $0.77$ for the combination
($\omega=300,\delta=270$). We also observed that for the
max_speedometer_attack and light_on_attack the reported AUC-ROC tends to be
below $0.50$ in most of the combinations of $\omega$ and $\delta$. This
suggests that the Matrix Correlation Distribution Method has a hard time
detecting these attacks as the probability of ranking random positive windows
more highly than random negatives windows is below $0.50$ in most of the
$\omega$ and $\delta$ combinations.

Figure 4: AUC-ROC over different window length and offset combinations for the
Matrix Correlation Distribution method on the different attacks in the ROAD
dataset: (a) correlated attack, (b) max engine coolant temperature attack, (c)
max speedometer attack, (d) reverse light off attack, and (e) reverse light on
attack.

Fig. 5 shows results for the Matrix Correlation Correlation Method. We observe
that the highest detection performance is obtained for the correlated_attack
with an AUC-ROC of $0.89$ for the combination ($\omega=150,\delta=130$). We
also observe AUC-ROC values above $0.70$ for the correlated_attack that tend
to be generally contained at lower values of $\omega$ and $\delta$, i.e.,
$\omega\leq 200$ and $\delta\leq 150$. Note that the evaluation metrics for
the remaining attacks (i.e., max_speedometer_attack, max_engine_attack, and
light_on_attack) are not as good as in the previous attacks barely staying
above $0.50$. An exception is the light_off_attack where AUC-ROC values around
0.70 are contained at lower values of $\omega$ and $\delta$, i.e., $\omega\leq
150$ and $\delta\leq 150$.

Figure 5: AUC-ROC over different window length and offset combinations for the
Matrix Correlation Correlation method on the different attacks in the ROAD
dataset: (a) correlated attack, (b) max engine coolant temperature attack, (c)
max speedometer attack, (d) reverse light off attack, and (e) reverse light on
attack.

Fig. 6 shows results for the Ganesan17 Method. We observe that the highest
evaluation metric is achieved for the max_speedometer_attack at $0.74$ for the
combination ($\omega=400,\delta=380$). For this attack, we also found a region
where AUC-ROC rose $0.70$ specially contained at lower values of $\omega$ and
$\delta$, i.e., $100\leq\omega\leq 150$ and $\delta\leq 160$. For the
remaining attack types we did not find any other remarkable evaluation
metrics.

Figure 6: AUC-ROC over different window length and offset combinations for the
Ganesan17 method on the different attacks in the ROAD dataset: (a) correlated
attack, (b) max engine coolant temperature attack, (c) max speedometer attack,
(d) reverse light off attack, and (e) reverse light on attack.

Fig. 7 shows results for the Moriano22 Method. We observe that the maximum
AUC-ROC among attack types is achieved with the max_engine_ attack achieving
an AUC-ROC of $1.00$ for a few combinarions of $\omega$ and $\delta$.
Additionally, for this attack, we found a region where AUC-ROC rose $0.70$
that is contained at lower values of $\omega$ and $\delta$, i.e.,
$100\leq\omega\leq 400$ and $10\leq\delta\leq 160$. For the light_off_attack,
we also observe a performance region with values of AUC-ROC above $0.70$,
generally contained at lower values of $\omega$ and $\delta$, i.e.,
$\omega\leq 200$ and $\delta\leq 200$. This method also displays AUC-ROC
values above $0.60$ for the light_on_attack generally contained at the entire
domain of values of $\omega$ and $\delta$. In addition, we observe that the
evaluation metrics for the remaining attacks (i.e., correlated_attack and
max_speedometer_attack) are not as good as in the previous attacks barely
reaching above $0.50$.

Figure 7: AUC-ROC over different window length and offset combinations for the
Moriano22 method on the different attacks in the ROAD dataset: (a) correlated
attack, (b) max engine coolant temperature attack, (c) max speedometer attack,
(d) reverse light off attack, and (e) reverse light on attack.

###  5.2 Evaluation Metrics Summary

Table II summarizes results extracted from the heatmaps. We place attack
categories in the rows and attack detection methods in the columns. This means
that each cell in this table shows the performance of a specific detection
method in a particular attack from the ROAD dataset. We focus on summary
statistics including the mean ($\mu$), standard deviation ($\sigma$), median
($\eta$), minimum ($\min$) and maximum ($\max$). We also show the average and
standard deviations from the maximum values for each attack detection methods
and attack types in the ROAD dataset. They are displayed in the last row and
column in the table.

Notably, from the attack detection methods, Moriano22 Method performs the best
across attacks (i.e., $\overline{max}=0.73$), followed by Matrix Correlation
Distribution and Matrix Correlation Correlation (i.e., $\overline{max}=0.67$),
and Ganesan17 (i.e., $\overline{max}=0.61$). Among these, the minimum and
maximum variation of maximum AUC-ROC is obtained by Matrix Correlation
Distribution (i.e., $\sigma_{max}=0.10$) and Matrix Correlation Correlation
(i.e., $\sigma_{max}=0.21$) respectively.

Note that, from the attack categories, the max_engine_attack was found to be
the attack that is detected with higher AUC-ROC among bechmarked algorithms
(i.e., $\overline{max}=0.79$), followed by correlated_attack (i.e.,
$\overline{max}=0.73$), light_off_attack (i.e., $\overline{max}=0.67$) and
max_speedometer_attack and light_on_attack (i.e., (i.e.,
$\overline{max}=0.59$). Among these, the minimum and maximum variation of
maximum AUC-ROC is obtained by light_on_attack (i.e., $\sigma_{max}=0.00$) and
max_engine_attack (i.e., $\sigma_{max}=0.17$) respectively.

TABLE II: Summary of evaluation metrics based on the AUC-ROC heatmaps. We show the mean ($\mu$), standard deviation ($\sigma$), median ($\eta$), minimum ($\min$) including the combination of window length and offset that produces it, and maximum ($\max$) including the combination of window length and offset that produces it. Attack types are placed in the rows while detection methods are placed in the columns. We also report averages and standard deviation of the maximum values obtained by each attack and detection method denoted by $\overline{max}$ and $\sigma_{max}$. Attack/Method | Matrix Correlation | Matrix Correlation | Ganesan17 | Moriano22 |  |
---|---|---|---|---|---|---
| Distribution | Correlation |  |  |  |
correlated_attack | $\mu=0.53$ | $\mu=0.71$ | $\mu=0.49$ | $\mu=0.35$ |  $\overline{max}=0.73$ |  $\sigma_{max}=0.12$
$\sigma=0.06$ | $\sigma=0.07$ | $\sigma=0.07$ | $\sigma=0.08$
$\eta=0.53$ | $\eta=0.72$ | $\eta=0.49$ | $\eta=0.34$
$\min=0.38$ (200, 190) |  $\min=0.54$ (400, 120) |  $\min=0.31$ (300, 110) |  $\min=0.15$ (250, 110)
$\max=0.77$ (300, 270) |  $\max=0.88$ (150, 130) |  $\max=0.66$ (400, 170) |  $\max=0.60$ (300, 180)
max_engine_attack | $\mu=0.36$ | $\mu=0.38$ | $\mu=0.31$ | $\mu=0.66$ |  $\overline{max}=0.79$ |  $\sigma_{max}=0.17$
$\sigma=0.11$ | $\sigma=0.09$ | $\sigma=0.11$ | $\sigma=0.13$
$\eta=0.36$ | $\eta=0.37$ | $\eta=0.29$ | $\eta=0.67$
$\min=0.03$ (350, 210) |  $\min=0.15$ (150, 140) |  $\min=0.04$ (400, 240) |  $\min=0.00$ (400, 380)
$\max=0.68$ (400, 230) |  $\max=0.83$ (400, 400) |  $\max=0.63$ (300, 270) |  $\max=1.00$ (400, 400)
max_speedometer_attack | $\mu=0.44$ | $\mu=0.32$ | $\mu=0.64$ | $\mu=0.49$ |  $\overline{max}=0.59$ |  $\sigma_{max}=0.14$
$\sigma=0.04$ | $\sigma=0.03$ | $\sigma=0.03$ | $\sigma=0.03$
$\eta=0.43$ | $\eta=0.31$ | $\eta=0.64$ | $\eta=0.49$
$\min=0.30$ (400, 360) |  $\min=0.22$ (300, 270) |  $\min=0.56$ (400, 350) |  $\min=0.42$ (300, 100)
$\max=0.58$ (350, 290) |  $\max=0.40$ (400, 220) |  $\max=0.74$ (400, 380) |  $\max=0.62$ (400, 300)
light_off_attack | $\mu=0.68$ | $\mu=0.59$ | $\mu=0.34$ | $\mu=0.61$ |  $\overline{max}=0.67$ |  $\sigma_{max}=0.15$
$\sigma=0.05$ | $\sigma=0.06$ | $\sigma=0.05$ | $\sigma=0.06$
$\eta=0.69$ | $\eta=0.58$ | $\eta=0.34$ | $\eta=0.61$
$\min=0.51$ (250, 240) |  $\min=0.45$ (350, 300) |  $\min=0.24$ (350, 290) |  $\min=0.44$ (400, 400)
$\max=0.76$ (400, 400) |  $\max=0.72$ (150, 150) |  $\max=0.45$ (350, 340) |  $\max=0.75$ (150, 140)
light_on_attack | $\mu=0.42$ | $\mu=0.42$ | $\mu=0.49$ | $\mu=0.59$ |  $\overline{max}=0.59$ |  $\sigma_{max}=0.08$
$\sigma=0.03$ | $\sigma=0.03$ | $\sigma=0.03$ | $\sigma=0.03$
$\eta=0.43$ | $\eta=0.42$ | $\eta=0.49$ | $\eta=0.59$
$\min=0.29$ (400, 270) |  $\min=0.28$ (350, 300) |  $\min=0.42$ (300, 180) |  $\min=0.51$ (250, 190)
$\max=0.55$ (400, 340) |  $\max=0.51$ (300, 290) |  $\max=0.59$ (400, 290) |  $\max=0.69$ (400, 290)
| $\overline{max}=0.67$ | $\overline{max}=0.67$ | $\overline{max}=0.61$ | $\overline{max}=0.73$ |  |
| $\sigma_{max}=0.10$ | $\sigma_{max}=0.21$ | $\sigma_{max}=0.11$ | $\sigma_{max}=0.16$ |  |

Finally, Fig. 8 compares each detection method on the basis of summary
statistics for each of attack type using dot plots. A desirable online
algorithm for detecting masquerade attacks in CAN by processing streams of
time series data should have high values of $\mu$, $\eta$, $\min$, and $\max$
and lower values of $\sigma$ in the range $[0,1]$. We notice that there is no
free lunch in the detection of masquerade attacks in CAN in an online setting
as there is no single algorithm that is optimal for detecting each attack
type. Notably, Ganesan17, the worst performing algorithms across all attack
types is the best performing detection algorithm for the hardest attack to
detect, i.e., max_speedometer_attack.

Figure 8: Dot plots comparing each detection method based on summary
statistics of AUC-ROC per attack type: (a) correlated_attack, (b)
max_engine_attack, (c) max_speedometer_attack, (d) light_off_attack, and (e)
light_on_attack.

###  5.3 Performance Metrics Summary

Table III summarizes TTW results based on heatmaps that we do not display
here. We place attack categories in the rows and attack detection methods in
the columns. This means that each cell in this table shows the TTW of a
specific detection method in a particular attack from the ROAD dataset. We
focus on summary statistics including the mean ($\mu$), standard deviation
($\sigma$), median ($\eta$), minimum ($\min$), and maximum ($\max$). In
addition, we also show the average and standard deviations from the minimum
values for each attack detection methods and attack types in the ROAD dataset.
They are displayed as the last row and column in the table.

Notably, from the attack detection methods, Matrix Correlation Distribution is
the quickest across attacks (i.e., $\overline{min}$ = 0.33), followed by
Matrix Correlation Correlation (i.e., $\overline{min}$ = 2.53), Ganesan17
(i.e., $\overline{min}$ = 3.65), and Moriano22 (i.e., $\overline{min}$ =
8.42). Among these, the minimum and maximum variation of maximum TTW is
obtained by Matrix Correlation Distribution (i.e., $\sigma_{min}$ = 0.00) and
Moriano22 (i.e., $\sigma_{min}$ = 0.51) respectively.

In addition, from the attack categories, we found that there are no
outstanding variations on the TTW within attack categories varying from 3.58
(i.e., light_off_attack) to 3.89 (light_on_attack) on average.

Finally, Fig. 9 compares each detection method on the basis of TTW summary
statistics for each attack type using dot plots. A desirable online algorithm
for detecting masquerade attacks in CAN by processing streams of time series
data should have low values of $\mu$, $\eta$, $\min$, $\max$ and lower values
of $\sigma$ in the range $[0,1]$. Remarkably, Matrix Correlation Distribution
method is the quickest detection algorithm one order of magnitude lower than
the others.

Figure 9: Dot plots comparing each detection method based on summary statistics of TTW per attack type: (a) correlated_attack, (b) max_engine_attack, (c) max_speedometer_attack, (d) light_off_attack, and (e) light_on_attack. TABLE III: Summary of evaluation metrics based on TTW in miliseconds. We show the mean ($\mu$), standard deviation ($\sigma$), median ($\eta$), minimum ($\min$) including the combination of window length and offset that produces it, and maximum ($\max$) including the combination of window length and offset that produces it. Attack types are placed in the rows while detection methods are placed in the columns. We also report averages and standard deviation of the minimum values obtained by each attack and detection method denoted by $\overline{min}$ and $\sigma_{min}$. Attack/Method | Matrix Correlation | Matrix Correlation | Ganesan17 | Moriano22 |  |
---|---|---|---|---|---|---
| Distribution | Correlation |  |  |  |
correlated_attack | $\mu=0.36$ | $\mu=2.73$ | $\mu=3.73$ | $\mu=10.45$ |  $\overline{min}=3.63$ |  $\sigma_{min}=3.28$
$\sigma=0.01$ | $\sigma=0.06$ | $\sigma=0.11$ | $\sigma=1.12$
$\eta=0.36$ | $\eta=2.72$ | $\eta=3.74$ | $\eta=10.38$
$\min=0.33$ (50, 40) |  $\min=2.62$ (100, 30) |  $\min=3.44$ (250, 240) |  $\min=8.14$ (50, 20)
$\max=0.39$ (400, 320) |  $\max=2.90$ (250, 140) |  $\max=4.02$ (350, 250) |  $\max=15.67$ (300, 160)
max_engine_attack | $\mu=0.35$ | $\mu=2.61$ | $\mu=4.51$ | $\mu=10.16$ |  $\overline{min}=3.77$ |  $\sigma_{min}=3.33$
$\sigma=0.02$ | $\sigma=0.08$ | $\sigma=0.13$ | $\sigma=1.67$
$\eta=0.35$ | $\eta=2.60$ | $\eta=4.51$ | $\eta=9.98$
$\min=0.03$ (150, 10) |  $\min=2.42$ (100, 60) |  $\min=4.16$ (50, 50) |  $\min=8.18$ (50, 40)
$\max=0.46$ (150, 140) |  $\max=2.94$ (400, 360) |  $\max=4.86$ (400, 270) |  $\max=21.18$ (350, 3300)
max_speedometer_attack | $\mu=0.35$ | $\mu=2.53$ | $\mu=3.58$ | $\mu=10.45$ |  $\overline{min}=3.79$ |  $\sigma_{min}=3.64$
$\sigma=0.01$ | $\sigma=0.03$ | $\sigma=0.04$ | $\sigma=0.62$
$\eta=0.35$ | $\eta=2.53$ | $\eta=3.58$ | $\eta=10.48$
$\min=0.34$ (50, 20) |  $\min=2.48$ (250, 240) |  $\min=3.45$ (300, 270) |  $\min=8.88$ (50, 40)
$\max=0.39$ (100, 90) |  $\max=2.66$ (100, 80) |  $\max=3.71$ (100, 80) |  $\max=13.33$ (400, 340)
light_off_attack | $\mu=0.35$ | $\mu=2.63$ | $\mu=3.79$ | $\mu=9.27$ |  $\overline{min}=3.58$ |  $\sigma_{min}=3.16$
$\sigma=0.01$ | $\sigma=0.04$ | $\sigma=0.08$ | $\sigma=0.81$
$\eta=0.35$ | $\eta=2.63$ | $\eta=3.79$ | $\eta=9.19$
$\min=0.33$ (50, 30) |  $\min=2.54$ (100, 60) |  $\min=3.59$ (350, 340) |  $\min=7.87$ (50, 60)
$\max=0.38$ (300, 220) |  $\max=2.82$ (400, 350) |  $\max=4.00$ (400, 200) |  $\max=13.72$ (400, 260)
light_on_attack | $\mu=0.35$ | $\mu=2.63$ | $\mu=3.79$ | $\mu=9.27$ |  $\overline{min}=3.89$ |  $\sigma_{min}=3.69$
$\sigma=0.01$ | $\sigma=0.05$ | $\sigma=0.08$ | $\sigma=0.65$
$\eta=0.35$ | $\eta=2.69$ | $\eta=3.85$ | $\eta=11.09$
$\min=0.33$ (50, 40) |  $\min=2.57$ (400, 380) |  $\min=3.62$ (100, 90) |  $\min=9.03$ (50, 40)
$\max=0.37$ (400, 180) |  $\max=2.86$ (200, 200) |  $\max=4.02$ (350, 290) |  $\max=12.81$ (400, 320)
| $\overline{min}=0.33$ | $\overline{min}=2.53$ | $\overline{min}=3.65$ | $\overline{min}=8.42$ |  |
| $\sigma_{min}=0.00$ | $\sigma_{min}=0.08$ | $\sigma_{min}=0.30$ | $\sigma_{min}=0.51$ |  |

##  6 Discussion

In this work, we have benchmarked non-DL-based unsupervised online IDS for
detecting masquerade attacks in CAN data. Our results indicate that the
bechmarked algorithms are not equally effective at detecting every attack type
as it has been shown before in previous studies in the offline setting [33,
18, 34]. We attributed this to the foundational principle used for each of the
detection algorithms and the magnitude of the perturbation that is incurred
when introducing each of the attacks. Overall, Moriano22 produces the best
detection results across attack types. Among attacks, max_speedomer_attack and
light_on_attack are the overall hardest to detect attack type.

In agreement with the foundational principle of Moriano22 (i.e., finding
disruptions in the hierarchy of time series clusters), our results supports
the fact that introduced masquerade attacks in the ROAD dataset change the
relative hierarchy of time series clusters as it was shown previously in an
offline setting in [15]. As opposed to Ganesan17, which focuses on significant
deviations of time series cluster assignations, Moriano22 captures significant
deviations in the hierarchy of clusters. Contrary to our intuition, our
findings also reveal that Matrix Correlation Distribution and Matrix
Correlation Correlation methods perform better than Ganesan17. We suspect that
this is in part because of the multiple hyperparameters that need to be
adjusted in Ganesan17 to tune its optimal performance.

While previous research has focused on detecting masquerade attacks in an
offline setting, our results demonstrate the importance of considering how
sliding windows are processed in an online setting with respect to window
length and offset. Specifically, our results provide new insights on the
effect of choosing carefully these two data partitioning parameters. To make
benchmarked methods more effective to work in a real-world setting, online
algorithms should strive to perform well when processing small batches of data
(i.e., lower $\omega$ values) and low latency between window transitions
(i.e., lower $\delta$ values). We quantify these relationships for every
method on each of the attack types.

Our study provides insights on the effectiveness and performance of online
detection algorithms for detecting masquerade attacks from the ROAD dataset.
Having noted the strengths of our benchmark framework, we are also aware of
the following limitations of our proposed work.

Lack of Validation in a Real Environment: Our proposed framework partitions
logs of already collected CAN captures to simulate an streaming environment of
CAN data using windows. We did not empirically validated our results on a
moving vehicle including the necessary hardware and performance tuning
involved to do so.

Limited Number of Benchmarked Algorithms: We bechmarked four unsupervised
online detection algorithms. As we strive for computationally efficient
approaches and test their efficacy on a widely known attack dataset, we did
not include unsupervised DL-based approaches (e.g., [19, 16]) in the
comparison.

Focus on a Single Testbed Dataset: We used the ROAD dataset with a limited
number of masquerade attacks to benchmark online detection algorithms. We did
not consider newer datasets with a higher number of masquerade attacks
generated on modern automobiles [59].

Use of Default Parameter Values in Detection Algorithms: We used default
parameters values as the original algorithms in Ganesan17 and Moriano22.
Therefore we did not explore the use of hyperparameter tuning on these
algorithms that could have an effect in their performance.

We anticipate that field experiments on a moving vehicle can be performed to
validate our results based on new technologies being deployed to extract time
series data from CAN payloads (e.g., [46]).

##  7 Conclusion

This paper benchmarks non-DL-based unsupervised online IDS for detecting
masquerade attacks in CAN. We focus on four detection algorithms (i.e., Matrix
Correlation Distribution, Matrix Correlation Correlation, Ganesan17 [23], and
Moriano22 [15]) which foundational mechanism is based on capturing significant
deviations from expected pairwise correlations over a stream of multivariate
time series. The inclusion of these baseline algorithms along with their open
source implementation allow direct comparison with forthcoming online
algorithms in this research space. The presented benchmark differs from other
benchmarks in prior works in that it focuses on the online operation of
detection algorithms (as opposed to offline operation mode) aimed at detecting
masquerade attacks—the stealthiest category of attacks in the CAN bus. Our
benchmark is based on the concept of time windows that control the partition
of a stream of time series. Specifically, we use sliding windows, with respect
to the number of observations, to analyze variations in the AUC-ROC based on
different combinations of window length and offset. We report conditions that
makes benchmarked algorithms more successful than others for certain attack
types as well a summary of their overall performance on different attacks. We
show that overall, Moriano22 performs the best at detecting the different
attack types in the ROAD dataset. Our analysis reveals that the
max_speedometer_attack is the more elusive to detect among detection
algorithms.

Future work in this area involve the inclusion of more recent online detection
algorithms algorithms such as the work by Shahriar et al. [17] and the
validation of our results in an empirical moving vehicle setting.

## Acknowledgements

This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-
AC05-00OR22725 with the U.S. Department of Energy. The publisher, by accepting
the article for publication, acknowledges that the U.S. Government retains a
non-exclusive, paid up, irrevocable, world-wide license to publish or
reproduce the published form of the manuscript, or allow others to do so, for
U.S. Government purposes. The DOE will provide public access to these results
in accordance with the DOE Public Access Plan
(http://energy.gov/downloads/doe-public-access-plan). This research was
sponsored in part by Oak Ridge National Laboratory’s (ORNL’s) Laboratory
Directed Research and Development program and by the DOE. There was no
additional external funding received for this study. This research used
birthright cloud resources of the Compute and Data Environment for Science
(CADES) at the Oak Ridge National Laboratory, which is supported by the Office
of Science of the U.S. Department of Energy under Contract No. DE-
AC05-00OR22725. The funders had no role in study design, data collection and
analysis, decision to publish, or preparation of this manuscript.

## References

  * [1] M. T. Kawser, M. S. Fahad, S. Ahmed, S. S. Sajjad, and H. A. Rafi, “The perspective of vehicle-to-everything (v2x) communication towards 5g,” _IJCSNS_ , vol. 19, no. 4, p. 146, 2019.
  * [2] O. Minawi, J. Whelan, A. Almehmadi, and K. El-Khatib, “Machine learning-based intrusion detection system for controller area networks,” in _Proceedings of the 10th ACM Symposium on Design and Analysis of Intelligent Vehicular Networks and Applications_ , 2020, pp. 41–47.
  * [3] A. Buscemi, I. Turcanu, G. Castignani, A. Panchenko, T. Engel, and K. G. Shin, “A survey on controller area network reverse engineering,” _IEEE Commun. Sur. Tut._ , 2023.
  * [4] I. Standard, “Road vehicles—controller area network (can) - part 2: Low-speed fault-tolerant,” _ISO_ , vol. 11898-3, p. 1, 2006.
  * [5] ——, “Road vehicles–controller area network (can)–part 1: Data link layer and physical signalling,” _ISO_ , vol. 11898-1, p. 1, 2015.
  * [6] ——, “Road vehicles—controller area network (can) - part 2: High-speed medium access unit,” _ISO_ , vol. 11898-2, p. 1, 2016.
  * [7] C. Miller and C. Valasek, “Remote exploitation of an unaltered passenger vehicle,” _Black Hat USA_ , vol. 2015, p. 91, 2015.
  * [8] ——, “CAN message injection: OG dynamite edition,” _Tech. Rep._ , 2016\.
  * [9] K.-T. Cho and K. G. Shin, “Fingerprinting electronic control units for vehicle intrusion detection,” in _Proceedings of the 25th USENIX Security Symposium_ , 2016, pp. 911–927.
  * [10] M. E. Verma, R. A. Bridges, M. D. Iannacone, S. C. Hollifield, P. Moriano, S. C. Hespeler, B. Kay, and F. L. Combs, “A comprehensive guide to can ids data and introduction of the road dataset,” _PLOS ONE_ , vol. 19, no. 1, pp. 1–32, 01 2024. [Online]. Available: https://doi.org/10.1371/journal.pone.0296879
  * [11] S. Woo, H. J. Jo, and D. H. Lee, “A practical wireless attack on the connected car and security protocol for in-vehicle can,” _IEEE Transactions on Intell. Transp._ , vol. 16, no. 2, pp. 993–1006, 2014.
  * [12] H. J. Jo and W. Choi, “A survey of attacks on controller area networks and corresponding countermeasures,” _IEEE Transactions on Intell. Transp._ , vol. 23, no. 7, pp. 6123–6141, 2021.
  * [13] F. Akowuah and F. Kong, “Physical invariant based attack detection for autonomous vehicles: Survey, vision, and challenges,” in _2021 Fourth International conference on connected and autonomous driving (MetroCAD)_. IEEE, 2021, pp. 31–40.
  * [14] M. R. Ansari, W. T. Miller, C. She, and Q. Yu, “A low-cost masquerade and replay attack detection method for can in automobiles,” in _2017 IEEE international symposium on circuits and systems (ISCAS)_. IEEE, 2017, pp. 1–4.
  * [15] P. Moriano, R. A. Bridges, and M. D. Iannacone, “Detecting can masquerade attacks with signal clustering similarity,” in _Proceedings Fourth International Workshop on Automotive and Autonomous Vehicle Security_ , ser. AutoSec 2022. Internet Society, 2022. [Online]. Available: http://dx.doi.org/10.14722/autosec.2022.23028
  * [16] M. H. Shahriar, Y. Xiao, P. Moriano, W. Lou, and Y. T. Hou, “Canshield: Deep-learning-based intrusion detection framework for controller area networks at the signal level,” _IEEE Internet Things_ , vol. 10, no. 24, pp. 22 111–22 127, 2023.
  * [17] M. H. Shahriar, W. Lou, and Y. T. Hou, “CANtropy: Time series feature extraction-based intrusion detection systems for controller area networks,” in _Symposium on Vehicles Security and Privacy_. Internet Society, 2023. [Online]. Available: https://dx.doi.org/10.14722/vehiclesec.2023.23090
  * [18] S. Sharmin, H. Mansor, A. F. Abdul Kadir, and N. A. Aziz, “Comparative Evaluation of Anomaly-Based Controller Area Network IDS,” in _Proceedings of the 12th International Conference on Software and Computer Applications_ , 2023, pp. 218–226.
  * [19] M. Hanselmann, T. Strauss, K. Dormann, and H. Ulmer, “Canet: An unsupervised intrusion detection system for high dimensional can bus data,” _IEEE Access_ , vol. 8, pp. 58 194–58 205, 2020.
  * [20] A. Nichelini, C. A. Pozzoli, S. Longari, M. Carminati, and S. Zanero, “Canova: a hybrid intrusion detection framework based on automatic signal classification for can,” _Comput. Secur._ , vol. 128, p. 103166, 2023.
  * [21] H. Ji, Y. Wang, H. Qin, Y. Wang, and H. Li, “Comparative performance evaluation of intrusion detection methods for in-vehicle networks,” _IEEE Access_ , vol. 6, pp. 37 523–37 532, 2018.
  * [22] S. Sharmin, H. Mansor, A. F. A. Kadir, and N. A. Aziz, “Benchmarking frameworks and comparative studies of controller area network (can) intrusion detection systems: A review,” _arXiv preprint arXiv:2402.06904_ , 2024.
  * [23] A. Ganesan, J. Rao, and K. Shin, “Exploiting consistency among heterogeneous sensors for vehicle anomaly detection,” SAE Technical Paper, Tech. Rep., 2017\.
  * [24] B. Babcock, M. Datar, and R. Motwani, “Sampling from a moving window over streaming data,” in _Proceedings of the thirteenth annual ACM-SIAM symposium on Discrete algorithms_ , 2002, pp. 633–634.
  * [25] J. Gama, “A survey on learning from data streams: current and future trends,” _Lect. Notes. Artif. Int._ , vol. 1, pp. 45–55, 2012.
  * [26] P. Moriano, “Code: Bechmarking online ids for masquerade attacks in can,” https://github.com/pmoriano/benchmarking-unsupervised-online-IDS-masquerade-attacks, 2024\.
  * [27] K.-T. Cho and K. G. Shin, “Error Handling of In-vehicle Networks Makes Them Vulnerable,” in _Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security_ , 2016, p. 1044–1055.
  * [28] C. Miller and C. Valasek, “Can message injection,” _OG Dynamite Edition_ , 2016.
  * [29] G. Dupont, J. Den Hartog, S. Etalle, and A. Lekidis, “Evaluation framework for network intrusion detection systems for in-vehicle can,” in _2019 IEEE International Conference on Connected Vehicles and Expo (ICCVE)_. IEEE, 2019, pp. 1–6.
  * [30] H. Lee, S. H. Jeong, and H. K. Kim, “Otids: A novel intrusion detection system for in-vehicle network by using remote frame,” in _2017 15th Annual Conference on Privacy, Security and Trust (PST)_. IEEE, 2017, pp. 57–5709.
  * [31] S. Stachowski, R. Gaynier, and D. J. LeBlanc, “An assessment method for automotive intrusion detection system performance,” University of Michigan, Ann Arbor, Transportation Research Institute, Tech. Rep., 2019.
  * [32] D. H. Blevins, P. Moriano, R. A. Bridges, M. E. Verma, M. D. Iannacone, and S. C. Hollifield, “Time-based can intrusion detection benchmark,” in _Proceedings Fourth International Workshop on Automotive and Autonomous Vehicle Security_ , ser. AutoSec 2021. Internet Society, 2021. [Online]. Available: https://doi.org/10.14722/autosec.2021.23013
  * [33] P. Agbaje, A. Anjum, A. Mitra, G. Bloom, and H. Olufowobi, “A framework for consistent and repeatable controller area network ids evaluation,” in _Proceedings of the Fourth International Workshop on Automotive and Autonomous Vehicle Security_ , 2022. [Online]. Available: http://dx.doi.org/10.14722/autosec.2022.23031
  * [34] F. Pollicino, D. Stabili, and M. Marchetti, “Performance comparison of timing-based anomaly detectors for controller area network: a reproducible study,” _ACM Transactions on Cyber-Physical Systems_ , 2023.
  * [35] D. Chicco and G. Jurman, “The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation,” _BMC Genomics_ , vol. 21, pp. 1–13, 2020.
  * [36] A. K. Desta, S. Ohira, I. Arai, and K. Fujikawa, “Id sequence analysis for intrusion detection in the can bus using long short term memory networks,” in _2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops)_ , 2020, pp. 1–6.
  * [37] J. Sunny, S. Sankaran, and V. Saraswat, “A hybrid approach for fast anomaly detection in controller area networks,” in _2020 IEEE International Conference on Advanced Networks and Telecommunications Systems (ANTS)_. IEEE, 2020, pp. 1–6.
  * [38] M. Jedh, J. K. Lee _et al._ , “Evaluation of the architecture alternatives for real-time intrusion detection systems for vehicles,” in _2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS)_. IEEE, 2022, pp. 864–873.
  * [39] H. Jadidbonab, A. Tomlinson, H. N. Nguyen, T. Doan, and S. Shaikh, “A realtime in-vehicle network testbed for machine learning-based ids training and validation,” in _Workshop on AI and Cybersecurity: co-located with 41st SGAI International Conference on Artificial Intelligence (SGAI 2021)_. CEUR Workshop Proceedings, 2022.
  * [40] J. Gama, I. Žliobaitė, A. Bifet, M. Pechenizkiy, and A. Bouchachia, “A survey on concept drift adaptation,” _ACM Compt. Surv._ , vol. 46, no. 4, pp. 1–37, 2014.
  * [41] P. Avogadro, L. Palonca, and M. A. Dominoni, “Online anomaly search in time series: significant online discords,” _Knowl. Inf. Syst._ , vol. 62, no. 8, pp. 3083–3106, 2020.
  * [42] E. M. Ferragut, J. Laska, and R. A. Bridges, “A new, principled approach to anomaly detection,” in _2012 11th International Conference on Machine Learning and Applications_ , vol. 2. IEEE, 2012, pp. 210–215.
  * [43] S. Checkoway, D. McCoy, B. Kantor, D. Anderson, H. Shacham, S. Savage, K. Koscher, A. Czeskis, F. Roesner, and T. Kohno, “Comprehensive experimental analyses of automotive attack surfaces,” in _20th USENIX security symposium (USENIX Security 11)_ , 2011.
  * [44] T. Qin, Z. Liu, P. Wang, S. Li, X. Guan, and L. Gao, “Symmetry degree measurement and its applications to anomaly detection,” _IEEE T. Inf. Foren. Sec._ , vol. 15, pp. 1040–1055, 2019.
  * [45] M. D. Pesé, T. Stacer, C. A. Campos, E. Newberry, D. Chen, and K. G. Shin, “Librecan: Automated can message translator,” in _Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security_ , 2019, pp. 2283–2300.
  * [46] M. E. Verma, R. A. Bridges, J. J. Sosnowski, S. C. Hollifield, and M. D. Iannacone, “Can-d: A modular four-step pipeline for comprehensively decoding controller area network data,” _IEEE T. Veh. Technol._ , vol. 70, no. 10, pp. 9685–9700, 2021.
  * [47] A. Buscemi, I. Turcanu, G. Castignani, R. Crunelle, and T. Engel, “Canmatch: a fully automated tool for can bus reverse engineering based on frame matching,” _IEEE T. Veh. Technol._ , vol. 70, no. 12, pp. 12 358–12 373, 2021.
  * [48] K. Pearson, “Notes on regression and inheritance in the case of two parents,” _P. R. Soc. London_ , vol. 58, pp. 240–242, 1895.
  * [49] H. B. Mann and D. R. Whitney, “On a test of whether one of two random variables is stochastically larger than the other,” _The Annals of Mathematical Statistics_ , pp. 50–60, 1947.
  * [50] C. Spearman, “The proof and measurement of association between two things,” _The American Journal of Psychology_ , pp. 441–471, 1987.
  * [51] P. Schober, C. Boer, and L. A. Schwarte, “Correlation coefficients: appropriate use and interpretation,” _Anesth. Analg._ , vol. 126, no. 5, pp. 1763–1768, 2018.
  * [52] M. Ester, H.-P. Kriegel, J. Sander, and X. Xu, “A density-based algorithm for discovering clusters in large spatial databases with noise,” in _Proceedings of the Second International Conference on Knowledge Discovery and Data Mining (KDD)_ , 1996, p. 226–231.
  * [53] StackExchange CrossValidated, “Converting similarity matrix to (euclidean) distance matrix,” https://stats.stackexchange.com/questions/36152/converting-similarity-matrix-to-euclidean-distance-matrix/36158#36158, online; ccessed: 2024-02-28.
  * [54] A. K. Jain, M. N. Murty, and P. J. Flynn, “Data clustering: a review,” _ACM Comput. Surv._ , vol. 31, no. 3, pp. 264–323, 1999.
  * [55] A. J. Gates and Y.-Y. Ahn, “Clusim: a python package for calculating clustering similarity,” _Journal of Open Source Software_ , vol. 4, no. 35, p. 1264, 2019.
  * [56] A. J. Gates, I. B. Wood, W. P. Hetrick, and Y.-Y. Ahn, “Element-centric clustering comparison unifies overlaps and hierarchy,” _Sci. Rep._ , vol. 9, no. 1, p. 8574, 2019.
  * [57] T. Fawcett, “An introduction to roc analysis,” _Pattern Recogn. lett._ , vol. 27, no. 8, pp. 861–874, 2006.
  * [58] J. Davis and M. Goadrich, “The relationship between precision-recall and roc curves,” in _Proceedings of the 23rd international conference on Machine learning_ , 2006, pp. 233–240.
  * [59] S. Rajapaksha, G. Madzudzo, H. Kalutarage, A. Petrovski, and M. O. Al-Kadri, “CAN-MIRGU: A Comprehensive CAN Bus Attack Dataset from Moving Vehicles for Intrusion Detection System Evaluation,” in _Symposium on Vehicles Security and Privacy_. Internet Society, 2024. [Online]. Available: https://dx.doi.org/10.14722/vehiclesec.2024.23043

|  Pablo Moriano (Senior Member, IEEE) received B.S. and M.S. degrees in
electrical engineering from Pontificia Universidad Javeriana in Colombia and
M.S. and Ph.D. degrees in informatics from Indiana University Bloomington,
Bloomington, IN, USA. He is a research scientist with the Computer Science and
Mathematics Division at Oak Ridge National Laboratory, Oak Ridge, TN, USA. His
research lies at the intersection of data science, network science, and
cybersecurity. In particular, he uses data-driven and computational methods to
discover, understand, and detect anomalous behavior in large-scale networked
systems. Applications of his research range across multiple disciplines,
including, the detection of exceptional events in social media, Internet route
hijacking, insider threat behavior in version control systems, and anomaly
detection in cyber-physical systems. Dr. Moriano is a member of ACM and SIAM.
---|---
|  Steven C. Hespeler earned his Bachelor’s degree in Engineering with a minor
in Mathematics from Roger Williams University, Bristol, RI, USA. Following his
BS he completed a Master’s degree in Industrial Engineering and Ph.D. in
Industrial Engineering with a minor in Applied Statistics both from from New
Mexico State University, Las Cruces, NM, USA. He is a Postdoctoral Research
Associate with the Computer Science and Mathematics Division, Oak Ridge
National Laboratory, Oak Ridge, TN, USA. His research interests are a fusion
of data science, non-destructive evaluation, and advanced signal processing.
He has an overall research goal of enabling smart and interconnected systems
and develops data-driven models with Machine/Deep Learning techniques for
solving complex engineering challenges. The applications of his research
encompass in-situ process monitoring and anomaly/defect detection within
additive manufacturing, cybersecurity, and robotic inspection.
---|---
|  Mingyan Li is a Senior cybersecurity researcher at Oak Ridge National
Laboratory (ORNL). She joined ORNL in 2021, after working at Boeing Research
and Technology for 15 years. She received a Ph.D. in EE from University of
Washington. Her research interests include cybersecurity data analytics, AI-
based IoT/cloud system security/privacy, and key management; within and beyond
distributed computing and transportation system domains.
---|---
|  Robert A. Bridges is a Senior Research Mathematician at ORNL sitting in the
National Security Sciences Directorate. Bobby currently focuses on advancing
differentially private machine learning and using mathematical system models
to identify feedback vulnerabilities. Bobby holds an undergraduate degree from
Creighton University in mathematics as well as a PhD in mathematics from
Purdue University. Leaving pure mathematics, Bobby was fortunate to obtain a
postdoc position at ORNL in 2012, where he has since supported a wide variety
of projects with machine learning, statistical, and applied math expertise.
Previous work includes leading projects on intra-vehicle network security and
reverse engineering algorithms, leading large-scale experiments testing
commercial security tools, and serving for 2 years as the Cybersecurity
Research Group Leader.
---|---