# LSF-IDM: Automotive Intrusion Detection Model with Lightweight Attribution
and Semantic Fusion

Pengzhou Cheng
College of Cyber Science and Engineering
Shanghai Jiao Tong University
Shanghai, China
<EMAIL_ADDRESS>
&Lei Hua
College of Automotive and Transportation Engineering
Jiangsu University of Technology
Changzhou, China
<EMAIL_ADDRESS>
Haobin Jiang
The Automotive Engineering Research Institute
Jiangsu University
Zhenjiang, China
<EMAIL_ADDRESS>
&Gongshen Liu
College of Cyber Science and Engineering
Shanghai Jiao Tong University
Shanghai, China
<EMAIL_ADDRESS>
The corresponding author.

###### Abstract

Autonomous vehicles (AVs) are more vulnerable to network attacks due to the
high connectivity and diverse communication modes between vehicles and
external networks. Deep learning-based Intrusion detection, an effective
method for detecting network attacks, can provide functional safety as well as
a real-time communication guarantee for vehicles, thereby being widely used
for AVs. Existing works well for cyber-attacks such as simple-mode but become
a higher false alarm with a resource-limited environment required when the
attack is concealed within a contextual feature. In this paper, we present a
novel automotive intrusion detection model with lightweight attribution and
semantic fusion, named LSF-IDM. Our motivation is based on the observation
that, when injected the malicious packets to the in-vehicle networks (IVNs),
the packet log presents a strict order of context feature because of the
periodicity and broadcast nature of the CAN bus. Therefore, this model first
captures the context as the semantic feature of messages by the BERT language
framework. Thereafter, the lightweight model (e.g., BiLSTM) learns the fused
feature from an input packet’s classification and its output distribution in
BERT based on knowledge distillation. Experiment results demonstrate the
effectiveness of our methods in defending against several representative
attacks from IVNs. We also perform the difference analysis of the proposed
method with lightweight models and Bert to attain a deeper understanding of
how the model balance detection performance and model complexity.

_K_ eywords Autonomous vehicles security $\cdot$ Intrusion detection $\cdot$
Lightweight attribution $\cdot$ Semantic feature $\cdot$ Feature fused

##  1 Introduction

Modern vehicles are different from traditional ones as they are fitted with
multiple electronic control units (ECUs). These ECUs manage various functions
of the vehicle through In-Vehicle Networks (IVNs) like Controller Area Network
(CAN) [1]. In order to improve the travel experience for both drivers and
passengers [2, 3, 4], more interfaces are open for external networks. Despite
the rapid development and great success of IVNs, there have been increasing
security concerns about deploying them in real-world applications. The high
connectivity between the in-vehicle and the external networks paves the way
for hackers to exploit security vulnerabilities in IVN as the weak access
control, no confirmation, and no encryption design of the CAN bus [3, 5, 6],
as shown in Figure. 1.

Figure 1: A CAN bus network exploited by attackers through different physical
and wireless interfaces

Cyber-attack by injecting malicious packets is one typical security threat [7,
8]. This idea is that attackers insert well-designed packets into the CAN bus
and then hijack the pre-defined protocol (e.g., arbitration mechanism,
broadcasting messages to maintain vehicle status and message periodicity) [9].
It may be fatal if the attacker can control physical subsystems like the
engine, steering wheel, and braking system [10, 11]. Thus, the security
vulnerabilities of IVNs are not only an information security or privacy
protection issue but also a functional safety-related concern. To protect
against various types of cyber-attack, security measures for IVNs involve
using encryption and authentication technologies in Cryptography to maintain
the confidentiality and integrity of IVNs transmitted messages [12, 13].
Additionally, firewall technology is utilized to separate potential attack
interfaces from IVNs [14, 15], and an Intrusion Detection System (IDS) is
deployed in AVs to detect attacks on vehicles. However, AVs have numerous
attack interfaces, limited computing and storage resources, and high real-time
and reliability requirements. Traditional active security protection
strategies such as encryption algorithms, identity authentication, and access
control are not applicable [16]. IDS is an efficient protection mechanism used
to detect cyber-attacks on automobiles, which can simultaneously provide
functional safety as well as a real-time communication guarantee for vehicles.

Initially introduced in the field of traditional methods such as
specification-based [17, 18], statistics-based [19, 20, 21], and physical
fingerprint-based [22, 23, 24] to identify intrusion messages but has
relatively worse performance. The security research community has recently
focused on using deep learning (DL) methods to create efficient IDSs [16, 25,
26, 27]. These methods model the traffic behavior to identify whether is
abnormal or not. However, existing works rely on how to extract valuable
features through increasingly complex model structures [9, 28, 29, 30]. Some
work focuses on the context feature of transmitted messages as decision-making
so as that explore the probability of using pre-trained language models (e.g.,
BERT and GPT-2) [4, 31]. Although these strategies obtain outperformed, it is
obviously ineffective to deploy them in a resource-limited environment that
IVNs. Another issue is the high false alarm rate (FAR) and a performance
bottleneck in a lightweight model [32, 33, 34].

To address the aforementioned challenges, we present LSF-IDM, an automotive
Intrusion Detection Model with Lightweight attribution and Semantic Fusion.
The proposed IDS involves using a pre-trained language model as a teacher
model to extract complex contextual semantic features, which can realize
valuable feature modeling. Then, we use a lightweight model as a student model
to inherit the above knowledge, meanwhile to integrate it into its output
probability distribution based on knowledge distillation. Finally, the lSF-IDM
can detect abnormal traffic through fused features controlled with a hyper-
parameter, by which the bottleneck problem in balancing detection performance
and model complexity has been solved. Our contributions are summarized as
follows.

  * •

We develop a novel lightweight IDM based on semantic fused, which features an
approximately pre-trained language model that makes its extracted valuable
feature transfer to a lightweight model.

  * •

To achieve more effective feature fused, we exploit knowledge distillation, in
which the Bidirectional Encoder Representations from the Transformers model
(BERT) as the teacher model capture context semantic features and the Bi-
directional Long Short-Term Memory (BiLSTM) and Deep neural network (DNN) as
student models to identify intrusion and prove the generalization performance.

  * •

We evaluate the LSF-IDM on a benchmark car hacking dataset. The experimental
results demonstrate the effectiveness of the LSF-IDM in improving detection
accuracy and false alarm rate in comparison to traditional machine learning
methods. Importantly, an obstacle to practice use in resource-limited is
addressed.

The rest of this paper is organized as follows: A comprehensive review of
previous works is presented in Section 2. Section 3 outlines the preliminaries
of IVN and the proposed method. Section 4 presents the design of the proposed
LSF-IDM in detail. In Section 5, the setting, results, and analysis of the
experiment evaluated are introduced. Finally, we conclude this study and
provide some perspectives in Section 6.

##  2 Related Work

The DL-based detection methods have been a major research interest in IVN-IDS
due to their outstanding performance in the detection accuracy for attacks
[10]. Recently, several works introduce complex models with strong feature
extraction capabilities to design an appropriate feature set to accurately
characterize CAN traffic. Seo et al. [2] proposed an intrusion detection
system based on a generative adversarial network (GAN). The method learned the
frequency characteristics of CAN IDs sequentially by training two
discriminators. However, it is difficult to train a stable GAN to detect
various attacks. Song et al. [9] improved the Inception-ResNet model by
proposing the construction of a $29\times 29$ message matrix in a deep
convolutional network. Although their method has a low false-negative and
error rate, the introduced model is still a deep structure with high
complexity. There are many works focused on extracting the spatial-temporal
feature to further optimize performance.

Lo et al. [28] used spatial and temporal features for developing an accurate
IVN-IDS. The spatio-temporal feature is generated sequentially using a
convolutional neural network (CNN) and long short-term memory (LSTM) network.
In contrast, Cheng et al. [29] generated spatio-temporal attention features by
fetching spatio-temporal features in parallel with CNN and LSTM and
introducing an attention module on top of that. This largely makes the model
more attentive to important traffic variations. Javed et al. [5] combined a
convolutional neural network (CNN) and the attention-based Gated Recurrent
Unit (GRU) model to propose a novel method named CANintelliIDS. This can
detect single intrusion attacks as well as mixed intrusion attacks on a CAN
bus. Yue et al. [30] proposes integrated spatio-temporal feature encoders that
extract features in parallel using various variants of LSTM and CNN. The fused
features are generated by voting. In order to reduce the model complexity,
Tariq et al. [35] preprocesses the in-vehicle telegrams into a two-dimensional
spatio-temporal form so as to design the feature encoder directly based on the
convLSTM component. Although these features are effective in identifying
traffic, they both take advantage of the deep network structure, which does
not allow for online detection in resource-constrained environments.

The lightweight model is a necessary requirement for IVN-IDS. Kang et al..
[32] first applied DNN to IVN-IDS analyzing normal and abnormal packets, which
realized a real-time detection model with a 97.8% detection accuracy on
simulation attacks. Taylor et al. [36] proposed an IDS constructed using the
data field values under the binary of CAN messages based on LSTM. The method
reduces the error rate of predicting the n+1st message category through
continuous optimization. Pawelec et al. [33] extended Taylor’s scheme and
verified that the proposed method is very promising for data that depends on
time series or concurrent inputs. Qin et al. [34] proposed to input the
hexadecimal and binary data domains into LSTM, and with the improved
optimization function, the model predicted different CAN IDs separately and
obtained a high accuracy. However, all of these methods present a performance
bottleneck in terms of identifying malicious messages. It is also
straightforward evidence that rich and meaningful features (e.g., spatial,
temporal, and context features) are the key to improving model detection.

Recently, the pre-trained language model has an outstanding performance on a
variety of downstream tasks in the natural language process (NLP). Due to its
powerful modeling of sequences, Alkhatib et al. [4] first applied BERT in IVN-
IDS. The designed CAN-BERT method can learn the context semantic features of
CAN ID sequences. Nam et al. [31] regarded the CAN ID sequence as a sentence
with several words and adopt the Generative Pretrained Transformer (GPT) model
to learn the normal pattern of each sequence. This method can detect attacks
that include a very small number of attack IDs. Although the context feature
is significant for IVN-IDS, these advanced models in the research area of NLP
are that their large amounts of parameters cannot meet the resource-limited
IVN. As a result, researching an intrusion detection model that has high
detection capabilities for malicious attacks and can be applied to resource-
constrained vehicle environments is vital for ensuring the security of AVs.
The comparison between previous works is presented in Table 1.

Table 1: A comparison of our work with other intrusion detection works in
feature extraction capabilities.

Works | Focus |  Semantic feature |  Temporal feature |  Spatial feature |  Feature fused |  Real-time detection
---|---|---|---|---|---|---
Seo et al. [2] | Double discriminator with GAN |  |  | ✓ |  |
Song et al. [1] | Inception-ResNet |  |  | ✓ |  |
lo et al. [28] | a hybrid model with CNN and LSTM sequentially |  | ✓ | ✓ | ✓ |
Cheng et al. [29] | CNN and LSTM with attention mechanism |  | ✓ | ✓ | ✓ |
Yue et al. [30] | Ensemble models with CNN and LSTM |  | ✓ | ✓ | ✓ |
Javd et al. [5] | CNN and LSTM with attention mechanism |  | ✓ | ✓ | ✓ |
Tariq et al. [35] | CovLSTM |  | ✓ | ✓ | ✓ |
Kang et al. [32] | DNN |  |  |  |  | ✓
Taylor et al. [36] | LSTM |  | ✓ |  |  | ✓
Pawelec et al. [33] | 3-layer LSTM |  | ✓ |  |  | ✓
Qin et al. [34] | LSTM in two format |  | ✓ |  |  | ✓
Alkhatib et al. [4] | BERT | ✓ |  |  |  |
Nam et al. [31] | GPT | ✓ |  |  |  |
Ours | Knowledge distillation, BERT | ✓ | ✓ | ✓ | ✓ | ✓

##  3 Overview of In-vehicle Security

In this section, we first introduce the background and nature of the CAN bus
and its security mechanism. And then the attack model corresponding to our
model is presented. This section describes the background of models used in
this work theoretically as well.

###  3.1 Controller area network

The Controller Area Network (CAN) bus, a broadcast-based networking protocol,
was built by Bosch in 1985 [2]. It has been widely used for in-vehicle
communication due to the low prices, size and functionality, and the efficient
and stable connection among ECUs [35].

Figure 2: CAN bus standard frame format

As shown in Fig. 2, the standard frame format of the CAN bus consists of Start
Of Frame (SOF), IDentifier (ID), Remote frame transmission field (RTR),
control field, data field, cyclic redundancy check (CRC), DELimiters (DEL) and
ACKnowledgment fields (ACK), and End Of Frame (EOF) fields. Among them, the ID
of the arbitration field is to avoid contention by priority mechanism when all
ECUs transmit the message simultaneously. This is also mainly a feature used
to detect intrusion behaviors as this mechanism provides an opportunity for
attackers to launch Denial of Service (DoS) and flooding attacks [4]. Due to
the characteristics of CAN such as the broadcast mechanism, without
authentication and encryption, the attacker can broadcast spoofed CAN
messages, eavesdrop on all the CAN traffic, and collect detailed information
about it, resulting in fuzzing and spoofing attacks.

###  3.2 Attack model

Since the weak security mechanism of IVN can be confronted with various
malicious attacks, as shown in Fig. 3. In this work, an assumption is
determined that the attacker has capable of compromising the ECUs either
remotely via wireless interfaces (e.g., the telematic port [37]) or physically
(e.g., via OBD-II [2]). Thus, the IVN is susceptible to Denial-of-Service
(DoS), Fuzzy, RPM/GEAR, and replay attacks, conducted as follows:

Figure 3: Illustration of the injection process of utilized in-vehicle
intrusion data set

  1. 1.

DoS attack: DoS attack, related to availability, is the most common intrusion
behavior in IVN as the priority mechanism of CAN bus [35]. Taking advantage of
it, the attackers are able to launch DoS attacks by constantly sending many
CAN messages with high priority in a short cycle [2].

  2. 2.

Fuzzy attack: Attackers can inject CAN messages using spoofed random ID and
DATA into the CAN bus [35].

  3. 3.

RPM/GEAR attack: RPM/GEAR is to inject messages with certain CAN IDs related
to RPM/GEAR information [1].

###  3.3 Models

Pre-trained language model. Language models aim to learn a probability
distribution for representing combinations of words and sentences in natural
language texts. The probability of a sequence of $m$ words
$\\{w_{1},w_{2},\cdots,w_{n}\\}$ is denoted as
$P\\{w_{1},w_{2},\cdots,w_{n}\\}$ [38]. To calculate it, the problem is
decomposed with the chain rule of probability:

$\displaystyle P(w_{1},\cdots,w_{m})$
$\displaystyle=P(w_{1})P(w_{2}|w_{1})P(w_{3}|w_{1},w_{2})\cdots
P(w_{m}|w_{1},\cdots,w_{m-1})$ (1)
$\displaystyle=\prod_{i=1}^{m}P(w_{i}|w_{1},\cdots,w_{i-1}).$

Recently, there has been an increase in the use of pre-trained neural networks
in NLP (e.g., BERT [39], GPT-2 [40]). These pre-trained neural networks learn
better-contextualized representation and can be applied to several downstream
tasks by fine-tuning.

Deep neural network. The deep neural network (DNN) $\mathcal{F}(\cdot)$
consists of a sequence of layers $(f_{1},f_{2},\cdots,f_{n})$, where each
layer $f_{i}$ is a differentiable transformation function. Given input
$x\in\mathbb{R}^{m\times n}$, the output of the neural network $\mathcal{F}$
is calculated by:

$\mathcal{F}(x;\theta)=f_{n}(f_{n-1}(\ldots(f_{2}(f_{1}(x)))))$ (2)

where $\theta$ is the parameters of the model. In this work, the model is used
to identify the malicious traffic, i.e., a classification task. Suppose there
are $c$ different classes, the output of the model
$\mathcal{F}(x,\theta)\in\mathbb{R}^{c}$, where the output distribution can
represent as $\mathcal{F}(x,\theta)_{o}=\\{o_{1},o_{2},\cdots,o_{c}\\}$ so
that the maximum probability is the final class. In order to achieve a non-
linear fit, some activation functions are introduced such as sigmoid, tanh,
Rectified Linear Unit (ReLU), and softmax function.

LSTM. Initially, the Recurrent neural network (RNN) presents feedback loops
that impact the model’s output using the same input values and network state.
To address the vanishing gradient problem from RNN, the Long-Short Term Memory
(LSTM) model is proposed by adding gating mechanisms, including input gates,
output gates, and forget gates. When the input sequence
$X=\\{x_{1},x_{2},\cdots,x_{t}\\}$, the LSTM can compute the output sequence
$Y=\\{y_{1},y_{2},\cdots,y_{t}\\}$, where the activation states of neurons
$c_{t}$ and memory units $m_{t}$ at time $t$. The memory cells in the LSTM are
iteratively updated as follows:

$\begin{gathered}i_{t}=\sigma(W_{i}\cdot[h_{t-1},x_{t}]+b_{i}),\\\
o_{t}=\sigma(W_{o}\cdot[h_{t-1},x_{t}]+b_{o}),\\\
f_{t}=\sigma\left(W_{f}\cdot[h_{t-1},x_{t}]+b_{f}\right),\\\
\tilde{C}_{t}=\mathrm{tanh}(W_{c}\cdot[h_{t-1},x_{t}]+b_{c}),\\\
C_{t}=f_{t}\odot C_{t-1}+i_{t}\odot\tilde{C}_{t},\\\
h_{t}=o_{t}\odot\tanh{(C_{t})},\\\ y_{t}=W_{y}h_{t}+b_{y},\end{gathered}$ (3)

where the $i_{t},o_{t},f_{t}$ donate the input, output, and forget of the gate
units at time $t$, respectively. The $W$ and $b$ represent the corresponding
weight and bias. The $y_{t}$ is the final prediction class of a specific task.
Actually, all of these gate units decide what information is important and
reserved by the activation function.

##  4 Methodology

In this section, we present LSF-IDM that can classify IVN traffic by
exploiting a valuable fused feature. We first describe the data preprocessing
step. Then, we present the LSF-IDM framework in detail, including the teacher-
student paradigm and knowledge distillation.

###  4.1 Data preprocessing

Data preprocessing is a crucial step for the DNN module as it is sensitive to
the quality of training data. However, the CAN traffic has many unexpected
problems when the log is saved, such as incomplete, and inconsistent. In this
work, we use the car hacking for intrusion detection dataset from the hacking
and countermeasure research lab
(HCRL)111https://ocslab.hksecurity.net/Datasets/car-hacking-dataset. The
dataset was created by recording CAN packets through the OBD-II port of a
vehicle during various CAN injection attacks, including DoS attack, fuzzing
attack, RPM attack, and GEAR attack. Table 2 presents the attack types and
size of this dataset, with the number of samples for the training and test
dataset.

Table 2: HCRL Dataset Characteristics Type |  Raw Data | #Number of Training Samples | #Number of Test Samples
---|---|---|---
Normal |  14,037,293 | 9,826,105 | 4,211,188
DoS |  587,521 | 411,265 | 176,256
Fuzzy |  491,847 | 344,293 | 147,554
RPM |  654,897 | 458,428 | 196,469
Gear |  597,252 | 418,076 | 179,176

In this dataset, the feature of each message involves the timestamp, CAN ID,
Data Length Code (DLC), data fields (data[0]-data[7], totaling 8 bytes), and a
flag bit (Flag) used to mark the class of the CAN message. As shown in Fig. 4,
the data preprocessing module can convert the captured data to a uniform scale
in order to meet the input of deep learning. Specifically, the data field is
divided into eight characteristic fields, where each field contains two
hexadecimal values. In case any bytes are missing in the data field, the value
"00" is used to fill them up. Moreover, all hexadecimal values within the ID
and Data domain are converted into decimal values. Note that we only consider
the field of ID, DLC, and Data as the features because of the important
correlations with the representative attacks.

Figure 4: Data preprocessing

Subsequently, data preprocessing requires independent implementations
corresponding to the student model and the teacher model. In terms of the BERT
model, CAN messages need to be transformed into word lists by Tokenize
operation and then Embedding is performed on them, while for BiLSTM, the data
needs to be normalized. Formally, the training data $\mathbb{D}$ is a pair-
wise instance of a dictionary that includes the lightweight model input
$(X_{i},y_{i})=\\{x_{1},x_{2},\cdots,x_{n};y_{i}\\}^{1\times 10}$
corresponding to the teach model input
$(X_{i},y_{i})=\\{token_{1},token_{2},\cdots,token_{n};y_{i}\\}^{1\times\mathrm{max\\_length}}$.
The $\mathrm{max\\_length}$ is a hyper-parameter from BERT.

###  4.2 LSF-IDM

In state-of-the-art IDSs, frame detection usually pursues either outstanding
performance via complex model structure or real-time by smaller parameters. It
is certain that a balance needs to be achieved between detection performance
and low-resource requirements. To this end, we present a semantic fusion-based
malicious attack detection model, named LSF-IDM. The Fig. 5 shows the overall
workflow. Given the training dataset and a teacher-student model, the LSF-IDM
pipeline consists of three steps to determine whether the CAN bus is
intrusion:

  1. 1.

Context semantic feature generation. The teacher model first learns the
contextual semantic features of CAN messages through a pre-trained language
model (our method is based on BERT). This can capture significant features and
then improve the detection performance due to its high model complexity and a
large number of parameters (Transformer blocks=12, Hidden size=768, Total
parameters=110M).

  2. 2.

Lightweight model training. We use a lightweight model (e.g., BiLSTM or DNN)
in this step. We design a feature extractor to get representation vectors of
the lightweight model.

  3. 3.

Feature fused. Given a teacher model, the lightweight model as a student model
also learns its knowledge. The fused feature is composed of representation
vectors from a teacher-student model based on knowledge distillation. Then, we
train a classifier to detect anomalies using the fused feature.

Figure 5: Proposed LSF-IDM pipeline

As a result, the proposed can be directly applied to resource-constrained in-
vehicle network environments. The whole LSF-IDM framework is presented in the
Algorithm 1. In the following, we first introduce our approach to context
semantic feature generation- using BERT in Section 4.2.1. Then, we introduce
the way to perform the lightweight model training in Section 4.2.2. Finally,
the feature fused-knowledge distillation is present in Section 4.2.3. As a
result, employing it as the student model of LSF-IDM and using knowledge
distillation to simulate the output distribution of the CAN-BERT model while
learning CAN message classification.

1

Input: $(X_{i},y_{i})\in\mathbb{D}_{train}$

Output: Detection result

/* Context-semantic feature generation and teacher model training */

2 while  _the teacher model is not convergence_ do

3 for  _$(X_{i},y_{i})_{teacher}\in\mathbb{D}_{train}$_ do

4 $V_{i}\leftarrow\mathrm{BERT}(X_{i})$;

5 $\hat{y_{i}}\leftarrow\mathrm{MLP}(V_{i})$;

/* $\hat{y_{i}}$ is context-semantic features */

6 $loss\leftarrow\mathcal{L}_{CE}(y_{i},\hat{y_{i}})$;

7 $loss.backward()$;

8

9 end for

10

11 end while

/* Feature fused and lightweight model training */

12 while  _the student model is not convergence_ do

13 for
_$(X_{i},y_{i})_{student},(X_{i},y_{i})_{teacher}\in\mathbb{D}_{train}$_ do

/* generate context-semantic feature */

14
$\hat{y}_{i}^{teacher}\leftarrow\mathrm{BERTforClassification}(X_{i}^{teacher})$;

/* F is a lightweight model */

15 $V_{i}^{student}\leftarrow\mathcal{F}(X_{i}^{student})$;

16 $\hat{y}_{i}^{student}\leftarrow\mathrm{MLP}(V_{i}^{student})$;

17
$loss\leftarrow\alpha\mathcal{L}_{CE}(y_{i},\hat{y_{i}})+(1-\alpha)\mathcal{L}_{KD}(\hat{y}_{i}^{student},\hat{y}_{i}^{teacher})$;

18 $loss.backward()$;

19

20 end for

21

22 end while

/* Intrusion detection evaluation */

23 for  _$(X_{i},y_{i})_{student}\in\mathbb{D}_{train}$_ do

24 $V_{i}^{student}\leftarrow\mathcal{F}(X_{i}^{student})$;

25 $\hat{y}_{i}^{student}\leftarrow\mathrm{MLP}(V_{i}^{student})$;

26 $\mathrm{Acc}+=\sigma(\hat{y}_{i}^{student})_{max-probability}==y_{i}$

27 end for

Algorithm 1 LSF-IDM

####  4.2.1 Context semantic feature generation

We hope to generate valuable context-semantic features so that the classifier
can accurately identify attacks. This is essential because an observation that
the attacker injection attacks may cause a strict order in the message log.

To this end, we propose the use of a pre-trained language model-
BERT222https://huggingface.co/bert-base-uncased, which pre-trains
bidirectional representations from the unlabeled text by joint conditioning on
both the left and right context. To our best knowledge, only one study on IVN-
IDS employed it detecting anomaly detection, that is [4]. However, it is
impractical to use the fact that there are more parameters due to BERT
directly in IVNs with limited resources. Inspired by it, the first step of our
defense pipeline is to generate a context-semantic feature, based on which the
defender can exploit to transfer instructive knowledge to a lightweight model.

Given a preprocessed message
$X_{i}=\\{token_{1},token_{2},\cdots,token_{n}\\}\in\mathbb{R}^{d_{x}}$ with a
ground truth label $y\in\\{1,\cdots,c\\}$. to BERT, we first compute the token
embedding vector by a mapping function $\mathcal{I}$ as follows:

$T_{i}=\mathcal{I}(X_{i},\Theta)=W_{e}X_{i}+b_{i},$ (4)

where $\Theta$ is the parameter of the mapping function. The $W_{e}$ and
$b_{e}$ are the input embedding weight matrix and bias, respectively.
Thereafter, the CAN messages’ position is encoded into positional embedding
$P_{i}$ using a sinusoidal function.

Having the embedding vector $E_{i}=T_{i}+P_{i}$, we will feed it into a stack
of $L$ transformer encoders. Without loss of generality, we select $l-th\in L$
layer for the interpretation of contextual semantic feature extraction. Each
transformer layer has two sub-layers: a multi-head self-attention module $g$
and a position feed-forward network $z$. The residual connection establishes
the information transfer mechanism between the two sub-layers and the layer
normalization $f$ improves the model generalization ability. The final output
vector is thus calculated as follows:

$\displaystyle M_{i}$ $\displaystyle=g(E_{i})+f(E_{i}+g(E_{i})),$ (5)
$\displaystyle H_{i}$ $\displaystyle=z(M_{i})+f(M_{i}+z(M_{i}))$

where the $M_{i}$ and $H_{i}$ are the output of two sub-layer in $l-th$,
respectively.

The multi-head self-attention module $g$ is a crucial attribution for
generating the context-semantic feature. To simplify and clarify the problem,
we formally present how to calculate the attention feature of the embedding
vector $E_{i}$ of the CAN message in one head, calculated as follows:

$Attention(Q^{l},K^{l},V^{l})=AV^{l}=\sigma\left(\frac{Q^{l}K^{(l)^{T}}}{\sqrt{d}}\right)V^{l},$
(6)

where $A\in\mathbb{R}^{n\times n}$ is the attention matrix and n is the
dimension of the CAN message. $\sigma(\cdot)$ is the softmax function; $d$ is
the dimension of the vectors $Q^{l},K^{l}$ and $V^{l}$; and $\sqrt{d}$ can
solve the gradient vanishing problem. $Q^{l},K^{l}$ and $V^{l}$ represent
projections of the embedding vector $E_{i}$ using scaled dot-product
attention.

In fact, the generated features refer to a strong correlation between fields
of the CAN message if the query and key have a high attention weight. We
believe that this capture of global information is feasible for anomaly
detection so that can extract valuable context-semantic feature extraction.
The context semantic feature generation corresponds to the algorithm 1 in
lines 1-8. Not that the optimizer function is cross-entropy.

####  4.2.2 lightweight model training

The defender will perform the lightweight model training algorithm based on
the context-semantic features generated by the teacher model. Our lightweight
model consists of two goals: 1) learn the context-semantic distribution from
the teacher model; 2) integrate all valuable information from CAN messages and
train a classifier to distinguish normal and abnormal traffic. In the
following, we will first introduce BiLSTM as our lightweight model because its
good sequence modeling ability allows for a better understanding of the
knowledge passed on. And then we introduce the DNN as our lightweight model to
show the generality of semantic features of context-semantic.

BiLSTM. Compared with Long Short-Term Memory (LSTM), Bi-directional Long
Short-Term Memory (BiLSTM) has a strong capability of capturing contextual
features so that it can better learn the contextual semantic information of
CAN messages. Therefore, we use the BiLSTM which is a combination of a forward
LSTM and a backward LSTM as the student model to learn the global information
in the BERT model, calculated as follows:

$\displaystyle\overrightarrow{hw_{t}}$
$\displaystyle=\overrightarrow{LSTM}(w_{t})$ (7)
$\displaystyle\overleftarrow{hw_{t}}$
$\displaystyle=\overleftarrow{LSTM}(w_{t})$ $\displaystyle hw$
$\displaystyle=\left[\overrightarrow{hw_{t}}\circ\overleftarrow{hw_{t}}\right]$

where $\overrightarrow{hw_{t}}$ and $\overleftarrow{hw_{t}}$ donate the hidden
state of forward and backward, respectively. And $hw$ is the output vector of
the model.

DNN. Meanwhile, we propose to use a two-layer fully connected neural network
as another lightweight model. Although it is not able to absorb all the
knowledge from the teacher model as well as BiLSTM, we hypothesize that this
knowledge also plays a guiding role (as evidenced by the performance
variability analysis).

####  4.2.3 Feature fused

Knowledge distillation (KD) can transfer the “knowledge" of a complex model to
a lightweight model, and make its performance close to the complex model,
which can solve the problem of balancing performance and model complexity
[41]. The distillation process is shown in equation 8, where $Z_{i}$ is the
logits, $p_{i}$ is the probability that the input belongs to category $i$, and
$T$ is the super-parameter “temperature". The larger $T$ is the more uniform
the distribution of the output probability.

$p_{i}=\frac{\exp\left(\frac{Z_{i}}{T}\right)}{\Sigma_{j}\exp\left(\frac{Z_{j}}{T}\right)}.$
(8)

Formally speaking, let $\\{(X_{i}^{teacher},y_{i})\\}^{m}_{i=1}$ donate the
training data for BERT, where
$X_{i}:\mathbb{R}^{d_{x}}\rightarrow\\{1,\cdots,c\\}$ is the CAN message and
$y_{i}$ is a binary label indicating whether $X_{i}$ is abnormal ($y_{i}=1$)
or normal ($y_{i}=0$). We feed it into the convergence teacher model and then
obtain the context-semantic distribution $\hat{y}_{i}^{teacher}$. Similarly,
the $\\{(X_{i}^{student},y_{i})\\}^{m}_{i=1}$ is fed into the lightweight
model, we also get the output distribution $\hat{y}_{i}^{student}$. In order
to achieve knowledge transfer, we use the loss function $\mathcal{L}_{KD}$,
represented as follows:

$\mathcal{L}_{KD}=-\frac{1}{K}\sum_{i,c}[\hat{y}^{teacher}_{i,c}\cdot\log
P^{S}\left(y_{i}=c|X_{i};\Theta\right)],$ (9)

where $c$ is the sample category, $\Theta$ is a parameter of the student
model, and $P^{S}(y_{i}=c|X_{i};\Theta)$ is the probability of belonging to
$c$ computed by the student model for the sample $X_{i}$. Consistent with the
teacher mode, the LSF-IDM is trained with the $\mathcal{L}_{CE}$, represented
as follows:

$\mathcal{L}_{CE}=\frac{1}{N}\sum_{i}-[y\log\hat{y_{i}}+(1-y)\log{(1-\hat{y})}]$
(10)

In summary, all the objective functions in the whole model training are shown
in Equation 11, in which $\alpha$ is the hyper-parameter used to balance the
$\mathcal{L}_{KD}$ and $\mathcal{L}_{CE}$. All process is present in Alogrithm
1 in lines 9-17.

$\mathcal{L}=\underset{\theta}{\arg\min}\sum_{i}\alpha\mathcal{L}_{CE}(y_{i}^{student},\hat{y_{i}^{student}})+(1-\alpha)\mathcal{L}_{KD}(\hat{y}_{i}^{student},\hat{y}_{i}^{teacher})$
(11)

####  4.2.4 Target traffic detection

Let $\Theta^{*}$ represent the parameter obtained by the LSF-IDM. Given a
targeted traffic $X_{i}$, we only apply the lightweight model, i.e., LSF-IDM
with $\Theta^{*}$ to determine whether it is hijacked or not. Importantly, the
time complexity of the proposed model is equivalent to the previously
lightweight models [32, 33, 34], and with more sensitive recognition of
malicious attacks.

##  5 Experiment and Analysis

In this section, we will introduce the experiment setting and evaluation
metrics. Then, the results of using our pipelines to detect abnormal traffic
on the car-hacking dataset are presented. The experimental results are devoted
to proving the validity of LSF-IDM and its superiority over traditional
lightweight methods by learning context-semantic representation of the in-
vehicle network traffic.

###  5.1 Experiment setting

The proposed LSF-IDM has been implemented in Python language using the Pytorch
library for deep learning methods. We conducted experiments on a machine with
Intel (R) Xeon(R) Gold 6271C CPU @ 2.60GHz, 32GB RAM, and NVIDIA Tesla T4
(16G). Table 3 presents the experiment setup in detail.

Table 3: Experimental environment | Parameter | Value
---|---|---
Hardware | CPU | Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz
RAM | 64G
Graphics card & Memory | NVIDIA Tesla T4 (16G)
Software | Pytorch | 1.12.1
Cuda | 11.3
Cudnn | 8.1.0
Tensorboard | 2.11.2
| Python | 3.9.16

###  5.2 Evaluation Metrics

In order to comprehensively evaluate the performance of the proposed method,
we used accuracy (ACC), precision (PRE), recall (REC), and F1-score as
performance metrics. We first require to define the confusion matrix with
respect to IVN-IDS, as follows:

  * •

True Positive (TP): Malicious traffic is detected successfully by the LSF-IDM.

  * •

True Negative (TN): Normal traffic is identified as non-intrusion behavior
successfully by the LSF-IDM.

  * •

False Positive (FP): Normal traffic is identified as intrusion behavior,
related to the false alarm.

  * •

False Negative (FN): Malicious traffic is detected failure, related to miss
alarm.

According to the definition of the confusion matrix, the aforementioned
metrics are calculated as follows:

$ACC=\frac{TP+TN}{TP+FP+FN+TN}$ (12) $PRE=\frac{TP}{TP+FP}$ (13)
$REC=\frac{TP}{TP+FN}$ (14) $F1-score=\frac{2PRE\cdot REC}{PRE+REC}$ (15)

where ACC represents the ratio of correctly classified instances and the total
number of instances. PRE donates the fraction of target instances predicted as
positive that is actually positive, while REC presents the fraction of
positive instances that are detected. The F1 score is the harmonic mean of
precision and recall and is commonly used to evaluate model performance on
unbalanced datasets.

In particular, we exploit the false positive rate (FPR), and false negative
rate (FNR) to evaluate the false alarm and miss alarm of the proposed model,
as follows:

$FPR=\frac{FP}{TN+FP}$ (16) $FNR=\frac{FN}{TP+FN}$ (17)

Table 4: Hyperparameter settings Model | Parameter |  Value
---|---|---
BERT | Batch size |  128
Learning rate |  $5\times 10^{-5}$
Epoch |  3
Max length |  16
LSF-IDM | Learning rate |  $10^{-5}$
Epoch |  8
Batch size |  1024
Hidden size |  64
LSTM layer |  2

###  5.3 Detection performance

Hyperparameters are a critical factor in determining the performance of a
model. In this study, some parameters are determined before training the LSF-
IDM, as shown in Table 4. In terms of the teacher model, we define the batch
size as 128, the learning rate as $5\times 10^{-5}$, the epoch as 3, and the
max length as 16, respectively. For the lightweight model, the learning rate
is $10^{-5}$, the epoch is 8, the batch size is 1024, the hidden size is 64,
and the LSTM layer is 2. Based on it, we trained the LSF-IDM and then show the
detection performance. The fig. 6 presents the optimization results of the
training phase and validation phase.

Figure 6: Training and testing performance of LSF-IDM detector for four attack
datasets.

Intuitively, the chosen hyperparameters allow the model to significantly
enhance inference speed, with the DoS attack having the quickest decline in
loss and presenting sub-optimal convergence values. This result can be
attributed to its attack principles that force attack frames to be injected
into the CAN bus with the highest priority. In contrast, the training of fuzzy
attack injected with random data has a maximum loss value, until 30M steps
gradually form a convergence state. With the same attack setting, the spoofing
attack of Gear/RPM has similar convergence status, achieving optimal
performance. Note that trained-well models will be deployed to evaluate
detection performance on the testing dataset, respectively. The validation
result is consistent with the training phase so that the proposed model can
identify various attacks.

Figure 7: Performance evaluation on each type of attack of the HCRL dataset,
where each type from (a) to (d) consists of evaluation performance on six
metrics between the lightweight model and the LSF-IDM. In particular, the
upper and the bottom deviation of both models are filled with color
respectively.

In Fig. 7, we show the evaluation results and deviation between the
lightweight model and the LSF-IDM. Overall, it can be seen that when the
lightweight model learns the knowledge that is context-semantic from the
teacher model, the performance on the test sets increases for all four
attacks. For Dos attacks and spoofing attacks, the proposed model presents
comprehensive advantages with outstanding performance and lower false alarms.
This result attributes to the focus on context information of the CAN message,
because the DoS attack that the highest priority of frames injected in a short
period could corrupt this context information while a spoofing attack that
modified the data content can generate inconsistency frame with the previous
broadcast in the CAN bus. In contrast, it is difficult to detect the fuzzy
attack. Despite having a lower recall, our model also maintains an extent
ahead of the lightweight model, especially in the F1-score. In terms of false
alarm and miss alarm, the proposed model has stable performance, which is
crucial for drivers. The results validate our motivation that learning larger
models’ knowledge to apply in a lightweight model is work through knowledge
distillation. It not only breaks the performance bottleneck but also realizes
the purpose of excellent detection performance in real-time.

###  5.4 Performance difference analysis

Table 5: Performance difference analysis of LSF-IDM with BERT and BiLSTM

Attack type | Model | ACC/% | F1-Score/% | PRE/% | RE/% | FNR/% | FPR/%
---|---|---|---|---|---|---|---
DoS | BERT | 100 | 100 | 100 | 100 | 0 | 0
LSF-IDM | -0.72 | -0.01 | 0 | -0.01 | 0 | 0
99.28 | 99.99 | 100 | 99.99 | 0 | 0
+0 | +2.18 | 0 | +4.27 | -0.85 | 0
BiLSTM | 99.28 | 97.81 | 100 | 95.72 | 0.85 | 0
Fuzzy | BERT | 99.98 | 99.98 | 99.97 | 99.98 | 0.01 | $2.99\times 10^{-3}$
LSF-IDM | -1.29 | -5.24 | -1.79 | -8.37 | +0.85 | +0.25
98.69 | 94.74 | 98.18 | 91.53 | 0.86 | 0.25
+0.29 | +1.36 | +10.43 | -8.06 | +0.79 | -1.52
BiLSTM | 98.37 | 93.23 | 87.62 | 99.42 | 0.07 | 1.77
Gear | BERT | 100 | 100 | 100 | 100 | 0 | 0
LSF-IDM | 0 | -0.01 | 0 | 0 | 0 | 0
100 | 99.99 | 100 | 100 | 0 | 0
+0.88 | +3.16 | +0.02 | +6.13 | -1.01 | $-3.53\times 10^{-3}$
BiLSTM | 99.12 | 96.83 | 99.98 | 93.87 | 1.01 | $3.61\times 10^{-3}$
RPM | BERT | 100 | 100 | 100 | 100 | 0 | 0
LSF-IDM | 0 | 0 | 0 | 0 | $+3.35\times 10^{-12}$ | $+1.46\times 10^{-13}$
100 | 100 | 100 | 100 | $3.35\times 10^{-12}$ | $3.35\times 10^{-13}$
+0.03 | +0.10 | 0 | +0.21 | -3.4e-4 | +1.46e-13
BiLSTM | 99.97 | 99.90 | 100 | 99.79 | 0.03 | 0

Note: The data in red is the percentage increase in the performance of our
model compared to BiLSTM (where FNR and FPR represent the error rate, the
smaller the value the better the performance of the model); The data in green
shows the percentage decrease in the performance of this model compared to
BERT, which has the best performance. BERT has a huge number of parameters
that is difficult to be applied to the IVNs.

To further prove the contribution of the fused feature, we conduct the
performance difference analysis on the LSF-IDM with the lightweight model and
the pre-trained language model. Table 5 shows the results of the four attacks,
as well as the rise performance relative to the lightweight model and the fall
performance of the teacher model.

The testing for the ACC metric on the LSF-IDM achieves 99.28% on DoS, 98.69%
on Fuzzy ($0.29\%\uparrow$), and 100% on spoofing attacks ($0.88\%\uparrow$
and $0.33\%\uparrow$) with respect to the lightweight model. Due to the
inability to fully absorb the fusion knowledge, 0.72% and 1.29% of the
performance is lost on DoS attacks and Fuzzy attacks. In terms of the DoS
attack, due to the precision of 100%, the instructional capacity is reflected
in the recall rate (95.72%$\rightarrow$99.99% with raising 4.27%$\uparrow$),
while only decreasing 0.01% from BERT. Moreover, the performance enhancement
in the recall is also presented in Gear/RPM attacks with 6.13% and 0.21%,
which can identify malicious traffic with full knowledge of BERT knowledge. It
is worth noting that the F1-score presents dramatic improvement in the above
attacks (2.18%$\uparrow$, 3.16%$\uparrow$ and 0.1%$\uparrow$) as the
enhancement in precision and recall. Differently, the LSF-IDM does not seem to
be able to learn to recall capabilities on fuzzy attacks from BERT, but
precision is improved significantly (10.43%$\uparrow$). This reason is that
the proposed model prefers to capture the true anomaly. Thus, the overall
performance in F1-score achieves 94.74%.

Interestingly, the proposed model controls well on the FAR and the FNR. We
observe that the error rate is hardly and approximately 0. Although having
0.86% and 0.25% in Fuzzy attacks, this result is well compared with the
lightweight model.

###  5.5 Generalization performance

Figure 8: Performance evaluation on each type of attack of the HCRL dataset,
where each type from (a) to (d) consists of evaluation performance on six
metrics between the DNN and the LSF-IDM.

In this subsection, the proposed model performs a generalized performance
analysis on a simple model-DNN. The experimental results on the HCRL dataset
are shown in Fig. 8. Overall, it can be seen that when the DNN learns the
knowledge that is context-semantic from the teacher model, the performance on
the test sets has the presence of instability compared to BiLSTM, attributed
to it can only capture simple non-linear features rather than the context
information.

Fortunately, the context-semantic information also provides a part of
contributes to it, which can be seen in the DoS attack with a higher F1-score
and lower false alarm. In the scenario of the RPM attack, the proposed model
maintains all performance but has a relative deviation on ACC. We suppose that
the context-semantic may noise feature to DNN because of its inherent ability
to quickly capture the patterns of RPM attacks. Moreover, we observe that this
model more focus on recall in the Gear attack, which contributes to an
improvement in F1-score. In contrast, it has a significant decrease in recall
of the Fuzzy attack despite concentrating on the improvement of precision. The
reason is that fused features analyze complex Fuzzy attacks that are randomly
generated, which may form adversarial learning without any advantage for the
proposed model. Moreover, we observe that this model more focus on recall in
the Gear attack, which contributes to an improvement in F1-score. In contrast,
it has a significant decrease in recall of the Fuzzy attack despite
concentrating on the improvement of precision. The reason is that fused
features analyze complex Fuzzy attacks that are randomly generated, which may
form adversarial learning without any advantage for the proposed model.
However, it is worth noting that the LSF-IDM in DNN presents an equivalent or
lower false alarm on four attacks, which proves the importance of the context-
semantic feature directly. On the other hand, we can clarify that the LSF-IDM
has the generalization capability even if using a simple model.

Table 6: Performance difference analysis of LSF-IDM with BERT and DNN

Attack type | Model | ACC/% | F1-Score/% | PRE/% | RE/% | FNR/% | FPR/%
---|---|---|---|---|---|---|---
DoS | BERT | 100 | 100 | 100 | 100 | 0 | 0
LSF-IDM (DNN) | -0.01 | -0.02 | -0.03 | -0.02 | +0.01 | +$2.99\times 10^{-3}$
99.99 | 99.98 | 99.97 | 99.98 | 0.01 | $2.99\times 10^{-3}$
+0.91 | +2.87 | -0.03 | +5.41 | -1.08 | $-2.99\times 10^{-3}$
DNN | 99.08 | 97.12 | 100 | 94.57 | 1.09 | 0
Fuzzy | BERT | 99.98 | 99.98 | 99.97 | 99.98 | 0.01 | $2.99\times 10^{-3}$
LSF-IDM (DNN) | -1.69 | -9.25 | -3.73 | -14.17 | +1.41 | $+1.90\times 10^{-3}$
98.29 | 90.73 | 96.24 | 85.81 | 1.42 | 0.19
+0 | -2.16 | +9.27 | -13.82 | +1.37 | -1.70
DNN | 98.29 | 92.89 | 86.97 | 99.63 | 0.05 | 1.89
Gear | BERT | 100 | 100 | 100 | 100 | 0 | 0
LSF-IDM (DNN) | -0.05 | -1.51 | -2.97 | 0 | 0 | 0
99.95 | 98.49 | 97.03 | 100 | 0 | 0
+1.15 | +2.76 | -2.97 | +8.2 | -1.39 | 0
DNN | 98.80 | 95.73 | 100 | 91.80 | 1.39 | 0
RPM | BERT | 100 | 100 | 100 | 100 | 0 | 0
LSF-IDM (DNN) | -1.38 | 0 | -0.01 | 0 | 0 | 0
98.62 | 100 | 99.99 | 100 | 0 | 0
-1.38 | +0.01 | 0 | +0.02 | $-2.21\times 10^{-3}$ | 0
DNN | 100 | 99.99 | 99.99 | 99.98 | $+2.21\times 10^{-3}$ | 0

From Table 6, we also present the performance difference analysis for DNN.
With respect to the lightweight model, the testing for the ACC metric on the
LSF-IDM (DNN) achieves 99.99% on DoS ($0.91\%\uparrow$), 98.29% on Fuzzy, and
99.95% on Gear attack ($1.15\%\uparrow$). Consistent with the previous
analysis, it can be sensitive to the RPM attack without the need for
redundancy features. In terms of precision, the knowledge transfer behavior is
well without any loss on DoS and Gear attacks, resulting in
99.98%(5.41%$\uparrow$) and 100% (8.2%$\uparrow$) compared to the lightweight
model. Although having a relatively weak performance on recall, the F1-score
has enough competition with 99.98% (2.87%$\uparrow$) and 98.49%
(2.76%$\uparrow$). In contrast, the dramatic decrease of recall on the Fuzzy
attack makes the F1-score reduce by 2.16%.

In particular, we observe that LSF-IDM (DNN) does not generate false positives
for DoS and spoofing attacks, benefiting from valuable knowledge instruction.
Although improving the identification capacity of the fuzzy attack, the overly
sensitive recognition makes for a high FNR.

###  5.6 Comparison to the Baselines

In this subsection, we compared it with four baseline methods, i.e.,
threshold-based method (OTIDS [6]), self-supervised based [1], and LSTM-based
[36], respectively. The proposed evaluation (precision, recall, and F1-score)
metrics of different baselines against LSF-IDM are reported in Table 7. Note
that we only compared with some previous lightweight models as the proposed
model has an equivalent or proximity amount of parameters with them.

Table 7: Performance Comparison Attack Type | Model | F1/% | PRE/% | RE/%
---|---|---|---|---
DoS | [6] | 84.88 | 99.90 | 73.80
[1] | 98.33 | 97.51 | 99.16
[36] | 94.30 | 95.80 | 95.80
LSF-IDM (Ours) | 99.99 | 100 | 99.99
Fuzzy | [6] | 92.80 | 93.30 | 93.30
[1] | 93.05 | 94.45 | 83.45
[36] | 82.20 | 99.14 | 70.21
LSF-IDM (Ours) | 94.59 | 98.18 | 91.53
Gear | [6] | 91.31 | 91.83 | 90.80
[1] | 92.61 | 97.68 | 88.03
[36] | 83.42 | 99.83 | 71.65
LSF-IDM (Ours) | 100 | 99.99 | 100
RPM | [6] | 91.30 | 91.80 | 90.80
[1] | 98.50 | 97.20 | 99.83
[36] | 83.43 | 99.81 | 71.68
LSF-IDM (Ours) | 100 | 100 | 100

In Table 7, we observe that after knowledge distillation, (a) our LSF-IDM
achieves higher performance on the DoS attack compared to baseline models.
Although the threshold-based (OTIDS) has the proximate capability (99.99% vs
100%), the worst recall leads to 84.88% F1-score with a gap of 15.11%. The
original LSTM-based IDS only gets 94.3% F1-score, compared to our method
(99.99%), proving the contribution of context-semantic. The core inspiration
of Self-supervised based IDS is similar to us learning context information but
has an extent deviation of performance with respect to our method. (b) On
Spoofing attacks, the result shakes the lead with baseline models, especially
the LSTM-based (100% vs 83.42% and 100% vs 83.43% in F1-score). The reason is
that the LSTM-based method could not guarantee a stable performance on recall.
(c) The proposed method achieves optimal performance on the Fuzzy attack with
a 94.59 F1-score, despite the precision and recall are not outstanding. This
reason is attributed to the baseline method can not maintain the detection
capability on precision and recall simultaneously.

##  6 Conclusion and future work

A variety of cyber-attack can be dangerous threats to the vehicular network as
a number of interfaces are open to the external environment. However, deep
learning-based IDSs are either limited by the computational resources required
for complex models or by performance bottlenecks for lightweight models. To
this end, we propose LSF-IDM, with lightweight attribution and fused features
against malicious traffic using knowledge distillation. We demonstrate the
effectiveness of LSF-IDM in the detection of representative attacks.

For future research, LSF-IDM can be implemented with different pre-trained
language models and some state-of-art knowledge distillation methods.
Moreover, we hope that the proposed model can be analyzed for how to conduct
knowledge transfer with some interpretation works.

## Acknowledgments

This work was supported in part by the Graduate Research and Innovation
Projects of Jiangsu Province under Grant KYCX20_2862; in part by the China
Scholarship Council under Grant 202008690005; the Joint Funds of the National
Natural Science Foundation of China (Grant No. U21B2020); Shanghai Science and
Technology, China Plan Project (Grant No. 22511104400).

## References

  * [1] Hyun Min Song and Huy Kang Kim.  Self-supervised anomaly detection for in-vehicle network using noised pseudo normal data.  IEEE Transactions on Vehicular Technology, 70(2):1098–1108, 2021\.
  * [2] Eunbi Seo, Hyun Min Song, and Huy Kang Kim.  Gids: Gan based intrusion detection system for in-vehicle network.  In 2018 16th Annual Conference on Privacy, Security and Trust (PST), pages 1–6. IEEE, 2018.
  * [3] Wufei Wu, Renfa Li, Guoqi Xie, Jiyao An, Yang Bai, Jia Zhou, and Keqin Li.  A survey of intrusion detection for in-vehicle networks.  IEEE Transactions on Intelligent Transportation Systems, 21(3):919–933, 2019.
  * [4] Natasha Alkhatib, Maria Mushtaq, Hadi Ghauch, and Jean-Luc Danger.  Can-bert do it? controller area network intrusion detection system based on bert language model.  In 2022 IEEE/ACS 19th International Conference on Computer Systems and Applications (AICCSA), pages 1–8. IEEE, 2022.
  * [5] Abdul Rehman Javed, Saif Ur Rehman, Mohib Ullah Khan, Mamoun Alazab, and Thippa Reddy.  Canintelliids: Detecting in-vehicle intrusion attacks on a controller area network using cnn and attention-based gru.  IEEE transactions on network science and engineering, 8(2):1456–1466, 2021.
  * [6] Hyunsung Lee, Seong Hoon Jeong, and Huy Kang Kim.  Otids: A novel intrusion detection system for in-vehicle network by using remote frame.  In 2017 15th Annual Conference on Privacy, Security and Trust (PST), pages 57–5709. IEEE, 2017.
  * [7] Jiajia Liu, Shubin Zhang, Wen Sun, and Yongpeng Shi.  In-vehicle network attacks and countermeasures: Challenges and future directions.  IEEE Network, 31(5):50–58, 2017.
  * [8] Mirco Marchetti and Dario Stabili.  Read: Reverse engineering of automotive data frames.  IEEE Transactions on Information Forensics and Security, 14(4):1083–1097, 2018.
  * [9] Hyun Min Song, Jiyoung Woo, and Huy Kang Kim.  In-vehicle network intrusion detection using deep convolutional neural network.  Vehicular Communications, 21:100198, 2020.
  * [10] Ahmed NasrEldin, Ayman M Bahaa-Eldin, and Mohamed A Sobh.  In-vehicle intrusion detection based on deep learning attention technique.  In 2021 16th International Conference on Computer Engineering and Systems (ICCES), pages 1–7. IEEE, 2021.
  * [11] Mohamed Abdel-Basset, Nour Moustafa, Hossam Hawash, Imran Razzak, Karam M Sallam, and Osama M Elkomy.  Federated intrusion detection in blockchain-based smart transportation systems.  IEEE Transactions on Intelligent Transportation Systems, 23(3):2523–2537, 2021.
  * [12] Eric Wang, William Xu, Suhas Sastry, Songsong Liu, and Kai Zeng.  Hardware module-based message authentication in intra-vehicle networks.  In Proceedings of the 8th International Conference on Cyber-Physical Systems, pages 207–216, 2017.
  * [13] Jo Van Bulck, Jan Tobias Mühlberg, and Frank Piessens.  Vulcan: Efficient component authentication and software isolation for automotive control networks.  In Proceedings of the 33rd Annual Computer Security Applications Conference, pages 225–237, 2017.
  * [14] Georg Macher, Harald Sporer, Eugen Brenner, and Christian Kreiner.  Signal-layer security and trust-boundary identification based on hardware-software interface definition.  J. Ubiquitous Syst. Pervasive Networks, 10(1):1–9, 2018.
  * [15] Georg Macher, Harald Sporer, Eugen Brenner, and Christian Kreiner.  An automotive signal-layer security and trust-boundary identification approach.  Procedia Computer Science, 109:490–497, 2017.
  * [16] Felix Erlacher and Falko Dressler.  On high-speed flow-based intrusion detection using snort-compatible signatures.  IEEE Transactions on Dependable and Secure Computing, 19(1):495–506, 2020.
  * [17] Habeeb Olufowobi, Clinton Young, Joseph Zambreno, and Gedare Bloom.  Saiducant: Specification-based automotive intrusion detection using controller area network (can) timing.  IEEE Transactions on Vehicular Technology, 69(2):1484–1494, 2019\.
  * [18] Shiyi Jin, Jin-Gyun Chung, and Yinan Xu.  Signature-based intrusion detection system (ids) for in-vehicle can bus network.  In 2021 IEEE International Symposium on Circuits and Systems (ISCAS), pages 1–5. IEEE, 2021.
  * [19] Clinton Young, Habeeb Olufowobi, Gedare Bloom, and Joseph Zambreno.  Automotive intrusion detection based on constant can message frequencies across vehicle driving modes.  In Proceedings of the ACM Workshop on Automotive Cybersecurity, pages 9–14, 2019.
  * [20] Xun He, Zhen Yang, and Yongfeng Huang.  A vehicle intrusion detection system based on time interval and data fields.  In Artificial Intelligence and Security: 7th International Conference, ICAIS 2021, Dublin, Ireland, July 19–23, 2021, Proceedings, Part II 7, pages 538–549. Springer, 2021.
  * [21] Yijie Xun, Yilin Zhao, and Jiajia Liu.  Vehicleeids: A novel external intrusion detection system based on vehicle voltage signals.  IEEE Internet of Things Journal, 9(3):2124–2133, 2021.
  * [22] Mahsa Foruhandeh, Yanmao Man, Ryan Gerdes, Ming Li, and Thidapat Chantem.  Simple: Single-frame based physical layer identification for intrusion detection and prevention on in-vehicle networks.  In Proceedings of the 35th annual computer security applications conference, pages 229–244, 2019.
  * [23] Marcel Kneib, Oleg Schell, and Christopher Huth.  Easi: Edge-based sender identification on resource-constrained platforms for automotive networks.  In NDSS, pages 1–16, 2020.
  * [24] Oleg Schell and Marcel Kneib.  Valid: Voltage-based lightweight intrusion detection for the controller area network.  In 2020 IEEE 19th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), pages 225–232. IEEE, 2020.
  * [25] Zihan Wu, Hong Zhang, Penghai Wang, and Zhibo Sun.  Rtids: A robust transformer-based approach for intrusion detection system.  IEEE Access, 10:64375–64387, 2022.
  * [26] Ruizhe Yao, Ning Wang, Peng Chen, Di Ma, and Xianjun Sheng.  A cnn-transformer hybrid approach for an intrusion detection system in advanced metering infrastructure.  Multimedia Tools and Applications, 82(13):19463–19486, 2023.
  * [27] Omar Y Al-Jarrah, Carsten Maple, Mehrdad Dianati, David Oxtoby, and Alex Mouzakitis.  Intrusion detection systems for intra-vehicle networks: A review.  IEEE Access, 7:21266–21289, 2019.
  * [28] Wei Lo, Hamed Alqahtani, Kutub Thakur, Ahmad Almadhor, Subhash Chander, and Gulshan Kumar.  A hybrid deep learning based intrusion detection system using spatial-temporal representation of in-vehicle network traffic.  Vehicular Communications, 35:100471, 2022.
  * [29] Pengzhou Cheng, Mu Han, Aoxue Li, and Fengwei Zhang.  Stc-ids: Spatial–temporal correlation feature analyzing based intrusion detection system for intelligent connected vehicles.  International Journal of Intelligent Systems, 37(11):9532–9561, 2022.
  * [30] Chuan Yue, Lide Wang, Dengrui Wang, Ruifeng Duo, and Xiaobo Nie.  An ensemble intrusion detection method for train ethernet consist network based on cnn and rnn.  IEEE Access, 9:59527–59539, 2021.
  * [31] Minki Nam, Seungyoung Park, and Duk Soo Kim.  Intrusion detection method using bi-directional gpt for in-vehicle controller area networks.  IEEE Access, 9:124931–124944, 2021.
  * [32] Min-Joo Kang and Je-Won Kang.  Intrusion detection system using deep neural network for in-vehicle network security.  PloS one, 11(6):e0155781, 2016.
  * [33] Krzysztof Pawelec, Robert A Bridges, and Frank L Combs.  Towards a can ids based on a neural network data field predictor.  In Proceedings of the ACM Workshop on Automotive Cybersecurity, pages 31–34, 2019.
  * [34] Hongmao Qin, Mengru Yan, and Haojie Ji.  Application of controller area network (can) bus anomaly detection based on time series prediction.  Vehicular Communications, 27:100291, 2021.
  * [35] Shahroz Tariq, Sangyup Lee, and Simon S Woo.  Cantransfer: Transfer learning based intrusion detection on a controller area network using convolutional lstm network.  In Proceedings of the 35th annual ACM symposium on applied computing, pages 1048–1055, 2020.
  * [36] Adrian Taylor, Sylvain Leblanc, and Nathalie Japkowicz.  Anomaly detection in automobile control network data with long short-term memory networks.  In 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA), pages 130–139. IEEE, 2016.
  * [37] Charlie Miller and Chris Valasek.  Remote exploitation of an unaltered passenger vehicle.  Black Hat USA, 2015(S 91):1–91, 2015.
  * [38] Shaofeng Li, Hui Liu, Tian Dong, Benjamin Zi Hao Zhao, Minhui Xue, Haojin Zhu, and Jialiang Lu.  Hidden backdoors in human-centric language models.  In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 3123–3140, 2021.
  * [39] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [40] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [41] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.