#  KDDT: Knowledge Distillation-Empowered Digital Twin for Anomaly Detection

Qinghua Xu  0000-0001-8104-1645 Simula Research Laboratory and University of
OsloOsloNorway<EMAIL_ADDRESS>,  Shaukat Ali  0000-0002-9979-3519 Simula
Research Laboratory and Oslo Metropolitan UniversityOsloNorway
<EMAIL_ADDRESS>,  Tao Yue  0000-0003-3262-5577 Simula Research
LaboratoryOsloNorway<EMAIL_ADDRESS>,  Zaimovic Nedim  0009-0005-0391-6473
Alstom Rail Sweden ABVästeråsSweden<EMAIL_ADDRESS>and
Inderjeet Singh  0009-0000-9022-9850 Alstom Rail Sweden ABVästeråsSweden
<EMAIL_ADDRESS>
(2023; 2023-05-18; 2023-07-31)

###### Abstract.

Cyber-physical systems (CPSs), like train control and management systems
(TCMS), are becoming ubiquitous in critical infrastructures. As safety-
critical systems, ensuring their dependability during operation is crucial.
Digital twins (DTs) have been increasingly studied for this purpose owing to
their capability of runtime monitoring and warning, prediction and detection
of anomalies, etc. However, constructing a DT for anomaly detection in TCMS
necessitates sufficient training data and extracting both chronological and
context features with high quality. Hence, in this paper, we propose a novel
method named KDDT for TCMS anomaly detection. KDDT harnesses a language model
(LM) and a long short-term memory (LSTM) network to extract contexts and
chronological features, respectively. To enrich data volume, KDDT benefits
from out-of-domain data with knowledge distillation (KD). We evaluated KDDT
with two datasets from our industry partner Alstom and obtained the F1 scores
of 0.931 and 0.915, respectively, demonstrating the effectiveness of KDDT. We
also explored individual contributions of the DT model, LM, and KD to the
overall performance of KDDT, via a comprehensive empirical study, and observed
average F1 score improvements of 12.4%, 3%, and 6.05%, respectively.

digital twin, knowledge distillation, anomaly detection, Train Control and
Management System

††copyright: acmlicensed††price: 15.00††doi:
10.1145/3611643.3613879††journalyear: 2023††submissionid:
fse23industry-p57-p††isbn: 979-8-4007-0327-0/23/12††conference: Proceedings of
the 31st ACM Joint European Software Engineering Conference and Symposium on
the Foundations of Software Engineering; December 3–9, 2023; San Francisco,
CA, USA††booktitle: Proceedings of the 31st ACM Joint European Software
Engineering Conference and Symposium on the Foundations of Software
Engineering (ESEC/FSE ’23), December 3–9, 2023, San Francisco, CA, USA††ccs:
Software and its engineering Maintaining software††ccs: Security and privacy
Intrusion detection systems††ccs: Computer systems organization Embedded
systems††ccs: Computer systems organization Sensors and actuators††ccs:
Computer systems organization Embedded and cyber-physical systems††ccs:
Computing methodologies Neural networks

##  1\. Introduction

Cyber-physical Systems (CPSs) play a vital role in Industry 4.0. Various CPSs
have been deployed in critical infrastructures, e.g., water treatment plant
(Xu et al., 2021, 2023a) and elevator systems (Xu et al., 2022; Han et al.,
2022b, a), whose safe operation concerns our daily lives. One such safety-
critical CPS is railway systems (RWS). Our industrial partner Alstom
manufactures intricate RWS, which involves not only vehicles and tracks (the
physical part) but also network communication inside the vehicles themselves
and across them (the cyber part). RWS get increasingly complex when they
become more heterogeneous and integrated, i.e., with more software systems
added to provide rich functionalities. However, such complexity renders RWS
susceptible to broader threats (e.g., anomalies (Anushiya and Lavanya, 2021)
and software faults (Zafar et al., 2021)). In particular, anomalies are one of
the most severe threats that might lead to system failures or even
catastrophic consequences (Islam et al., 2022).

Inspired by their success in computer vision and natural language processing
(NLP) (Dong et al., 2021), machine learning methods have been widely applied
to address various anomaly detection tasks in RWS (Islam et al., 2022; H. Zhao
et al., 2017; Guzman et al., 2018). However, training these methods involves
collecting abundant data from RWS, which might interfere with the safe
operation of RWS. To reduce such interference, digital twin (DT), as a novel
technology, has been intensively studied for CPSs anomaly detection (Xu et
al., 2021, 2023a, 2022; Jones et al., 2020). Specifically, a DT can be
considered as a digital replica of a CPS and enables rich functionalities with
this replica instead of the real CPS. Early DTs are predominantly based on
software/system models, requiring manual effort from domain experts (Eckhart
and Ekelhart, 2019). To mitigate this challenge, machine learning methods are
increasingly used to enable data-driven DT constructions, which require much
less manual effort and domain knowledge while performing accurate anomaly
detection in CPSs (Xu et al., 2021, 2023a, 2022).

However, to the best of our knowledge, no previous works have focused on
building data-driven DTs for network anomaly detection in RWS. In this paper,
we focus on detecting packet loss anomalies on IP networks in the Train
Control and Management System (TCMS). When a packet loss anomaly occurs, a
certain pattern is exhibited where certain signal values abruptly drop to 0
and rebound quickly. Such network anomalies can induce misjudgment and
incorrect reactions in the control unit of RWS, leading to unexpected and
risky behaviours of RWS. Despite its significance, constructing a DT for
network anomaly detection remains unsolved due to the following two challenges
listed below.

Challenge 1: High data complexity. Datasets for network anomaly detection are
packets arranged in chronological order. The packets’ contents and order
reflect the network state and the oblivion of either aspect diminishes the
performance of the DT. Sequential models such as recurrent neural networks can
extract chronological features, while the packets’ content features are
intrinsically more difficult to extract due to their textual format. Unlike
structural data, textual data does not follow a fixed format or schema and can
vary greatly in length. To extract high-quality features from textual data,
both the syntax and semantics of the data should be understood by the feature
extractor without ambiguity.
Challenge 2: Training data insufficiency. Data-driven DT construction entails
training with abundant in-domain labelled data, i.e., data related to the
network packet loss. However, such datasets are difficult to collect since
anomalies during the operation of safety-critical systems like RWS rarely
occur. Moreover, detecting anomalies manually is non-trivial. The presence of
packet dropping can be difficult to detect without rich domain knowledge since
normal network fluctuations might resemble an anomaly to a great extent. Such
resemblance prevents non-experts, without sufficient training, from manually
labelling the packets.

To address the aforementioned challenges, we propose a novel DT-based method
KDDT for network anomaly detection in TCMS. To tackle the first challenge,
KDDT extracts features from packets’ contents and order with a contextualized
language model (LM) and an LSTM, respectively (Mimura and Tanaka, 2018; Yu et
al., 2019). The second challenge is the insufficiency of training data. We
circumvent this challenge by distilling knowledge from the out-of-domain (OOD)
dataset as a supplement, which is relatively cheaper than the in-domain (ID)
dataset acquisition. Specifically, we pretrain a Variational Autoencoder (VAE)
to encode and reconstruct a network packet from OOD data. We posit a well-
trained VAE possesses rich knowledge of encoding network packets with high
quality, which can be leveraged by KD to supplement the ID dataset. Similar to
the popular pretraining+fine-tuning paradigm, KD is also a transfer learning
technique. However, KD asks the pretrained VAE to act like a teacher rather
than merely providing a set of better-initialized parameters for KDDT. The key
advantage of KD is a reduction in model complexity, which requires fewer
resources and can potentially mitigate the risk of overfitting.

We evaluated KDDT with two TCMS network packet datasets from Alstom. KDDT
achieves average F1 scores: 0.931 and 0.915. Moreover, we posit that the
benefits of KDDT’s sub-components extend beyond the scope of network anomaly
detection. To investigate their individual contributions, we studied the
effectiveness of DTM, LM, and KD. Evaluation results show that the absence of
DTM, LM or KD leads to decreases of 12.4%, 3%, and 6.05%, respectively, in
terms of the average F1 score on both datasets.

##  2\. Industrial Context

Figure 1. Overview of TCMS. MVB, ECN, ETB, and WTB stand for Multi-function
Vehicle Bus, Ethernet Consist Network, Ethernet Train Bus, and Wired Train
Bus, respectively. WTB GW and ETB GW are two gateways. CTC, MIO, and HMI
represent Conventional Train Control, Modular Input/Output, Human Machine
Interface, respectively. CCUs are control unit, including CCUO, CCUS, and
CCUD.

Alstom Rail Sweden AB is the second largest company in the rail industry, with
over 150 000 vehicles in service. One essential train component is TCMS, a
high-capacity infrastructure that controls communications among different
subsystems on the train and between the train and the ground(Zafar et al.,
2021). Figure 1 depicts the overview of the TCMS. Four buses and two gateways
constitute the backbone of TCMS, namely Multi-function Vehicle Bus (MVB),
Ethernet Consist Network (ECN), Ethernet Train Bus (ETB), Wired Train Bus
(WTB), WTB GW, and ETB GW. The MVB and ECN buses connect various internal
devices and other subsystems, while the WTB and ETB buses connect multiple
vehicles. Conventional Train Control (CTC) implements hard-wire logic in the
control system. The Modular Input/Output (MIO) devices tackle Input/Output
analog signals. Human Machine Interfaces (HMIs) provide control interfaces for
train drivers. TCMS relies on Central Control Units (CCUs) to implement
various train functions. CCUO, CCUS, and CCUD control the basic, safety-
critical, and diagnostic functions, respectively. Each function is performed
with one or multiple devices and buses in TCMS.

Take the standstill determination process as an example. Multiple train
functions rely on accurately assessing standstill, such as door control.
Concretely, the TCMS communicates with two sub-systems: the Brake Control
Units (BCUs) and the Traction Control Units (TCUs). The BCUs measure the
axles’ speed and inform the CTC about the standstill condition by setting
specific digital inputs on the MIO. In addition to the BCU’s hardware-based
standstill determination, the TCMS uses the speed of each driven axle, which
is sent by the TCUs via the Multifunction Vehicle Bus (MVB), for the software-
based standstill determination. This signal is then transmitted to the Door
Control Units (DCUs) and other functions via MVB/IP. Such processes entail
packet exchanges in the TCMS network, which is vulnerable to packet loss
anomalies. This paper aims to develop an effective DT for anomaly detection,
facilitating the safety check of RWS and reducing future packet loss risks.

##  3\. Methodology

Figure 2. KDDT’s model structure and training workflow. $\mathcal{D}_{OOD}$
and $\mathcal{D}_{ID}$ are out-of-domain and in-domain datasets.

As depicted in the center circle of Figure 2, KDDT has three major components:
a pretrained LM, a pretrained VAE, and a DT. A pretrained LM extracts
contextualized features $f$ from textual data (i.e., network packet), which
are leveraged by DT for anomaly detection. DT comprises a digital twin model
(DTM) and a digital twin capability (DTC). DTM simulates the TCMS, while DTC
implements one or more functionalities of DT, e.g., anomaly detection in our
context. The interaction between DTM and DTC is bidirectional, where DTM sends
information about the system state to DTC and receives feedback from DTC,
e.g., anomaly alerts. To further improve the DT performance, we utilize the
pretrained VAE as a teacher model to guide DTM’s training.

The four boxes of Figure 2 illustrate the training workflow of KDDT: data
preparation, LM pretraining, VAE pretraining, and DT training. In Stage 1, we
first prepare both out-of-domain $\mathcal{D}_{OOD}$ and in-domain
$\mathcal{D}_{ID}$ datasets. As mentioned in Section 2, we collect anomaly-
unrelated data as $\mathcal{D}_{OOD}$, which can be leveraged to pretrain some
universal models such as LM (Stage 2) and VAE (Stage 3). In-domain dataset
$\mathcal{D}_{ID}$, on the other hand, are directly related to anomalies,
which can be used to train DT for anomaly detection (Stage 4). In Stage 2, we
pretrain a contextualized LM with the $\mathcal{D}_{OOD}$ dataset to extract
inner-packet context features by predicting the next token. For instance,
given the context of ”RWS is an important Cyber-physical”, a well-trained LM
can extract a discrete vector to represent the context and use the vector to
predict the next token, i.e., ”system” with a high probability. In Stage 3, we
design a VAE comprising an encoder and a decoder and pretrain it with packets
from the $\mathcal{D}_{OOD}$ dataset. The encoder extracts context features
with the pretrained LM and encodes them into a hidden vector (h). The decoder
then takes h as the input and attempts to reconstruct the original network
packets. A well-trained VAE can effectively encode a network packet into a
high-quality hidden vector and reconstruct the packet with high fidelity. In
Stage 4, we train DTM and DTC of KDDT with the $\mathcal{D}_{ID}$ dataset but
under the guidance of the pretrained VAE. In particular, DTM and the
pretrained VAE both aim to encode a given packet with high quality. VAE
contains richer knowledge due to its complex architecture and pretraining on
the $\mathcal{D}_{OOD}$ dataset. Consequently, the hidden vectors produced by
the pretrained VAE convey richer information, which can be used as a soft
target (as opposed to a hard target, i.e., ground truth target) to guide the
DTM training process.

###  3.1. Data Preparation

As depicted in Figure 2, Stage 1, we sequentially perform two processes in the
data preparation stage: data collection and domain expert labelling.

Data collection. During operation, TCMS communicates with various devices by
exchanging large numbers of packets per second. Engineers in Alstom deployed
Wireshark (Lamping and Warnicke, 2004) to monitor such exchanges and capture
packets as .PCAP files. Let $X_{i}$ be the packet captured at timestep $i$. We
collect out-of-domain dataset
$\mathcal{D}_{OOD}=[X_{0},X_{1},...,X_{N_{OOD}-1}]$ and in-domain dataset
$\mathcal{D}_{ID}=[X_{0},X_{1},...,X_{N_{ID}-1}]$, where ${N_{OOD}}$ and
${N_{ID}}$ represent the dataset sizes. The $\mathcal{D}_{ID}$ dataset refers
to any data related to the anomaly detection task, i.e., suspicious anomalous
data. $\mathcal{D}_{OOD}$ dataset is task-agnostic, which includes any network
data collected in TCMS.

Domain expert labelling. KDDT, as a supervised learning method, requires
labelled data for accurate network anomaly detection. In our context, we study
the phenomenon of packet loss. Hence we assign an abnormal label to a packet
if it is experiencing a packet loss incident. We ask Alstom to provide us with
labelled packet data. They manually examined the signal logs with the help of
an in-house built monitoring tool named DCUTerm. DCUTerm displays signal
changes over time, providing a rough time boundary for packet loss anomalies,
which was further analyzed and narrowed down by checking the Wireshark packets
directly. We formally denote the label for $X_{i}$ as $y_{i}\in\\{0,1\\}$,
which takes the value of $1$ when a packet loss incident occurs.

###  3.2. LM Pretraining

The quality of feature engineering dramatically influences the performance of
machine learning models. One of the most salient features of the packets is
the inner-packet context features, which represent the semantics and syntax of
the packet content. We aim to pretrain a contextualized LM to extract such
features from packet contents. The LM treats the raw packet content as natural
language text and extracts semantic and syntactic features automatically. We
will demonstrate the model structure of LM in Section 3.2.1 and the loss
function in Section 3.2.2.

####  3.2.1. LM model structure

As shown in Figure 2, the LM consists of three layers:
Embedding layer. We first build a vocabulary $\mathcal{V}$ for all the packet
tokens and randomly initialize an N-dimensional embedding vector for each
token. Given a training sample $(ctx,tgt)$, the embedding layer takes the
context sequence $ctx$ as input and fetches the corresponding embedding for
each token inside this sequence, as shown in Equation 1. $L$ denotes the
length of the context.

(1) $X=Embed(ctx)=[x_{0},x_{1},...,x_{L-1}]$

Bi-directional LSTM layer. We feed the embedded context $X$ into a bi-
directional LSTM (Yu et al., 2019) for context information extraction:

(2) $h=LSTM(\overset{\leftarrow}{X})+LSTM(\overset{\rightarrow}{X})$

Softmax layer. This layer transforms the output of the bi-directional LSTM
into logits $z$ that sum up to be 1 as in Equation 3:

(3)
$\mathrm{z}=\mathrm{softmax}(h)=\frac{\exp(h_{i})}{\sum_{j=1}^{N}\exp(h_{j})}$

####  3.2.2. Loss Calculation

The output of LM is a probability distribution vector, whose element
represents each token’s probability of appearing at the next position. In
Equation 4, We calculate Cross Entropy Loss $\mathcal{L}_{CE}$ by comparing
logits $\mathrm{z}$ (Equation 3) with true values $tgt$, which is minimized to
train the LM. $|\mathcal{V}|$ denotes the vocabulary size.

(4)
$\mathcal{L}_{CE}=-\sum_{i=1}^{|\mathcal{V}|}\mathrm{tgt}_{i}\log\mathrm{z}_{i}$

###  3.3. VAE Pretraining

To leverage the $\mathcal{D}_{OOD}$ dataset, we train VAE as illustrated in
Stage 3, Figure 2. The underlining hypothesis of VAE is that each packet is an
observed data point generated based on a hidden vector that summarises this
packet. VAE aims to find this hidden vector with an encoder and evaluate its
quality by reconstructing the packet with the decoder. Higher reconstruction
quality indicates better encoding capability. Given a packet
$X_{i}\in\mathcal{D}_{OOD}$, the encoder extracts a hidden state vector, with
which the decoder reconstructs a packet similar to (but necessarily the same
as) $X_{i}$. We will present the details about the encoder and decoder in
Section 3.3.1 and Section 3.3.2, respectively.

Figure 3. Model Structure of KDDT

####  3.3.1. VAE Encoder

The encoder aims to induce the hidden vector from $\mathcal{D}_{OOD}$ packets,
with an assumption that the hidden vector follows a Gaussian distribution
$N(\mu,\sigma^{2})$. The high expressiveness of the Gaussian distribution
allows it to describe many phenomena in the real world. According to (Pinheiro
Cinelli et al., 2021), the Gaussian distribution assumption allows VAE to
utilize the reparameterization trick, which enhances training efficiency
without reducing its fitting capability. VAE first approximates the Gaussian
distribution $N(\mu,\sigma^{2})$ by computing $\mu$ and $\sigma$ with two
neural network models. Then we sample a hidden vector from $N(\mu,\sigma^{2})$
with the reparameterization trick. The upper part of the purple box in Figure
3 illustrates the internal layers of the VAE encoder:
LM layer. We utilize the trained LM from Stage 2 to convert the packet input
$X$ into embedding vectors as in Equation 5.

(5) $X=LM(X)$

Linear layers for $\mu$ and $\sigma$. We perform two separate linear
transformations for $\mu$ and $\sigma$ as in Equations 6 and 7 respectively,
where $W$ and $b$ are weight and bias matrices.

(6) $\mu=W_{\mu}X+b_{\mu}$ (7) $\sigma=W_{\sigma}X+b_{\sigma}$

Activation layers for $\mu$ and $\sigma$. We introduce non-linearity into the
model by adding an activation layer. Following the common practice in
(Pinheiro Cinelli et al., 2021), we choose ReLU and sigmoid as the activation
function for $\mu$ and $\sigma$, respectively (Equations 8 and 9).

(8) $\displaystyle\mu$ $\displaystyle=ReLU(\mu)$ (9) $\displaystyle\sigma$
$\displaystyle=Sigmoid(\sigma)$

Sampling layer. Given the expectation $\mu$ and standard deviation $\sigma$
for a Gaussian distribution, it is trivial to sample a hidden vector $h_{vE}$
from it. However, the practice of sampling breaks the gradient propagation
chain, which is the foundation of deep learning model training. Therefore, VAE
uses a reparametrization trick to address this problem as in Equation 10,
where $\epsilon$ is random noise.

(10) $h_{vE}=\mu+\epsilon*\sigma$

####  3.3.2. VAE Decoder

The decoder aims to reconstruct the packet from the hidden vector $h_{vE}$. We
build it as a convolution neural network consisting of the following layers:

Linear layer. Equation 11 shows that we first perform a linear transformation
on the hidden vector $h_{vE}$, where $W_{d}$ and $b_{d}$ are weight and bias
matrices.

(11) $h_{vD}=W_{d}h_{vE}+b_{d}$

Convolution layer. After the linear transformation, we use a convolution layer
to capture spatial features in $h_{vd}$:

(12) $h_{vD}=Conv(h_{vD})$

Pooling layer. We add a maximum pooling layer to increase the fitting
capability of the decoder as in Equation 13.

(13) $h_{vD}=max\\_pool(h_{vD})$

Softmax layer. Finally, we project the reconstructed vectors into the packet
space with a softmax layer as in Equation 14.

(14) $\hat{X}=softmax(h_{vD})$

####  3.3.3. Loss Calculation

As pointed out in the literature (Pinheiro Cinelli et al., 2021), an overall
loss should be minimized to train VAE, consisting of a KL divergence loss for
the encoder and a maximum likelihood loss (MLL) for the decoder (Equation 15).

(15) $\mathcal{L}_{VAE}=\mathcal{L}_{KL}+\mathcal{L}_{MLL}$

We compute the KL divergence loss by comparing the hidden vector distribution
with a standard Gaussian distribution:

$\displaystyle\mathcal{L}_{KL}$ $\displaystyle=KL(N(\mu,\sigma^{2})||N(0,1))$
(16)
$\displaystyle=-\frac{1}{2}\sum_{i=1}^{k}\left(1+\log\sigma_{i}^{2}-\mu_{i}^{2}-\sigma_{i}^{2}\right)$

The maximum likelihood loss aims to assess how close the reconstructed packet
$\hat{X}_{i}$ is to the real $\hat{X}_{i}$. As shown in Equation 17, we
calculate the cross entropy as the MLL loss.

(17) $\mathcal{L}_{MLL}=-\sum_{j=1}^{n}X_{i+1}^{j}\log(\hat{X}_{i+1}^{j})$

###  3.4. DT Training

The last stage in Figure 2 is DT training, which takes advantage of the
collected data, pretrained LM and VAE from Stages 1, 2 and 3, respectively.
Figure 3 presents model details of DT, comprising a DTM, a DTC, and a KD
module.

####  3.4.1. DTM Structure

DTM simulates the TCMS network by predicting the subsequent packet. As
mentioned in Section 1, the predictive performance of anomaly detection hinges
on extracting inner-packet context and inter-packet chronological features. A
well-trained VAE can effectively extract the inner-packet context feature, but
the inter-packet chronological feature is not considered in the VAE structure.
Therefore, we adapt the VAE from Stage 3 by combining VAE and LSTM to extract
both features.

At timestep $i-1$, DTM sequentially invokes three sub-models to predict a
packet $X_{i}$: an encoder, an LSTM and a decoder (Figure 3).
DTM Encoder. The internal structure of the encoder is identical to the VAE
encoder. At timestep $i-1$, the encoder encodes the input packet from previous
timestep $X_{i-1}$ into a hidden state vector $h_{i-1}^{dtmE}$ (Equation 18).
See details of the encoder in Section 3.3.

(18) $h_{i-1}^{dtmE}=encoder(X_{i})$

However, we reduce the complexity of the DTM encoder by using a smaller hidden
vector size compared to the VAE encoder. KD can benefit from a complexity
discrepancy between the teacher and student models (Clark et al., 2018).

DTM LSTM. We utilize an LSTM to capture the inter-packet chronological
features. The LSTM takes the hidden vector $h_{i-1}^{dtmE}$ as input and
outputs a hidden vector $h_{i}^{dtmE}$ for timestep $i$ as in Equation 19.

(19) $h_{i}^{dtmE}=LSTM(h_{i-1}^{dtmE})$

DTM Decoder. The decoder aims to generate a packet $\hat{X}_{i}$ with the
hidden vector $h_{i}$ as in Equation 20.

(20) $\hat{P}_{i}=dtm\\_decoder(h_{i}^{dtmE})$

####  3.4.2. DTC Structure

DTC performs anomaly detection with the input from the packet $X_{i}$ and the
hidden vector $h_{i}^{dtmE}$ from DTM. $h_{i}^{dtmE}$ is an indicator of the
system state based on historical data, whereas $X_{i}$ contains information
about the signal values in the TCMS. As shown in Figure 3, DTC consists of the
following layers:
LM layer. Similar to the LM layer in DTM, we use the pretrained LM to convert
packet $P_{i}$ into embedding vectors as in Equation 21.

(21) $X_{i}=LM(X_{i})$

Linear layer. This layer linearly transforms the embeddings into the hidden
space of DTM as in Equation 22, where $W_{l1}$ and $b_{l1}$ are weight
matrices.

(22) $h_{i}^{dtc}=W_{l1}X_{i}+b_{l1}$

Convolution layer. We then concatenate hidden vectors from DTM $h_{i}^{M}$ and
DTC $h_{i}^{dtc}$ and feed them into a convolution layer as in Equation 23.

(23) $h_{i}^{dtc}=Conv([h_{i}^{dtmE},h_{i}^{dtc}])$

Sigmoid layer. Non-linearity is introduced with a sigmoid activation function
as in Equation 24.

(24) $h_{i}^{dtc}=Sigmoid(h_{i}^{dtc})$

Pooling layer. We reduce the dimensionality of $h_{i}^{dtc}$ with a pooling
layer as in Equation 25.

(25) $h_{i}^{dtc}=max\\_pool(h_{i}^{dtc})$

Linear layer. We transform the hidden vector $h_{i}^{dtc}$ into the output
space as in Equation 26, where $W_{l2}$ and $W_{l2}$ are weight matrices.

(26) $o_{i}^{dtc}=W_{l2}h_{i}^{dtc}+b_{l2}$

Softmax layer. Finally, we perform a softmax operation to convert the output
$o_{i}$ into logits as in Equation 27.

(27) $z_{i}^{dtc}=softmax(o_{i}^{dtc})$

####  3.4.3. Knowledge Distillation

The pretrained VAE learns from extra OOD data about extracting inner-packet
context features. To make use of the pretrained models, many researchers adopt
a rather intuitive strategy to use the pretrained parameters directly and
fine-tune them with ID datasets. KD, however, tailors the complex teacher
model into a less complex student model to focus on a specific context, i.e.,
anomaly detection in our context. Instead of direct parameter sharing, the
student model uses the hidden vectors produced by the teacher model as a soft
target and optimizes its own parameters to encode similar vectors as the
teacher model. Figure 3 shows the pretrained VAE takes the packet from the
previous timestep $X_{i-1}$ as input and produces a hidden vector
$h_{i-1}^{vE}$. The DTM encoder uses $h_{i-1}^{vE}$ as a soft target and
calculates a cosine similarity loss:

(28) $\mathcal{L}_{KD}=\frac{h_{i-1}^{vE}\cdot
h_{i-1}^{dtmE}}{|h_{i-1}^{vE}|\times|h_{i-1}^{dtmE}|}$

####  3.4.4. Loss Calculation

The overall loss to train KDDT comprises the ground truth loss and KD loss as
depicted in Equation 29.

(29) $\mathcal{L}=\mathcal{L}_{GT}+\mathcal{L}_{KD}$

$\mathcal{L}_{KD}$ is calculated as in Equation 28, while $\mathcal{L}_{GT}$
entails loss calculation on both DTM and DTC (Equation 30).

(30) $\mathcal{L}_{GT}=\mathcal{L}_{DTM}+\mathcal{L}_{DTM}$

Since DTM resembles VAE in the model structure, we calculate the DTM loss
$\mathcal{L}_{DTM}$ similarly as a sum of the KL loss and MLL loss (Equation
31). Detailed calculation is in Equations 16 and 17.

(31) $\mathcal{L}_{DTM}=\mathcal{L}_{KL}+\mathcal{L}_{MLL}$

We compute a commonly-used cross entropy loss for DTC as:

(32) $\mathcal{L}_{CE}=-\sum_{i=1}^{N}\mathrm{y}_{i}\log\mathrm{z}_{i}^{dtc}$

##  4\. Experiment Design

To evaluate KDDT, we propose four research questions (RQs) in Section 4.1.
Section 4.2 presents details on the subject system and collected dataset. In
Section 4.3 and Section 4.4, we discuss the evaluation metrics and experiment
setups.

###  4.1. Research Questions

RQ1: Is KDDT effective in detecting anomalies in TCMS? RQ2: Is DTM effective
in improving DTC performance? RQ3: Is LM effective in extracting inner-packet
features? RQ4: Is KD effective in improving DTM’s encoding capability?

In RQ1, we investigate the effectiveness of KDDT in anomaly detection. With
RQ2 – RQ4, we delve into the contribution of each sub-component (i.e., DTM, LM
and KD) of KDDT to its overall effectiveness. Specifically, in RQ2, we compare
the effectiveness of KDDT with/without DTM. In RQ3, we study the effectiveness
of LM and compare it to Word2vec, another text feature extraction method
effective for packet-related tasks (Goodman et al., 2020). In RQ4, we compare
the effectiveness of KDDT with/without KD.

###  4.2. Industrial Subject System

Our industrial partner Alstom provided us with packet data captured in their
TCMS. Anomalies, i.e., packet loss incidents, occur on different devices at
different times due to various environmental uncertainties. They collected
packet data and manually identified two periods of time when anomalies tend to
appear more frequently, i.e., 2021/02/10 09:00 pm - 09:20 pm and 2021/02/11
05:35 am - 05:42 am. As mentioned in Section 3.1, we collect both OOD dataset
$\mathcal{D}_{OOD}$ and ID dataset $\mathcal{D}_{ID}$. We further divide
$\mathcal{D}_{ID}$ into a training dataset and a testing dataset for the
evaluation purpose with a ratio of 0.8:0.2. Table 1 reports key statistics
about the dataset.

Metric | Dataset | $\mathcal{D}_{OOD}$ |  $\mathcal{D}_{ID}$-Train |  $\mathcal{D}_{ID}$-Test
---|---|---|---|---
$\mathcal{N}$ | Day 10 | 625525 | 750631 | 187658
Day 11 | 313007 | 375609 | 93903
$\mathcal{N}_{NP}$ | Day 10 | 625525 | 713990 | 137808
Day 11 | 313007 | 339395 | 70225
$\mathcal{N}_{AI}$ | Day 10 | - | 186 | 122
Day 11 | - | 64 | 21
$\mathcal{N}_{AP}$ | Day 10 | - | 36641 | 49850
Day 11 | - | 36214 | 23678
$\mathcal{L}_{AI}$ | Day 10 | - | 197.00 | 407.23
Day 11 | - | 565.84 | 1127.52
$\mathcal{T}_{AI}$ | Day 10 | - | 198353.88 $\mu s$ | 420050.60 $\mu s$
Day 11 | - | 616352.65 $\mu s$ | 1207371.71 $\mu s$
Table 1. Statistics about the dataset. The $\mathcal{D}_{OOD}$ dataset
contains only anomaly-unrelated data, while $\mathcal{D}_{ID}$ contains
anomaly-related data. $N$ represents the total number of packets in the
dataset. $\mathcal{N}_{NP}$ and $\mathcal{N}_{AP}$ are the numbers of normal
and abnormal packets in the dataset. $\mathcal{N}_{AI}$ is the number of
anomaly incidents in the dataset. $\mathcal{L}_{AI}$ and $\mathcal{T}_{AI}$
denote the average length (number of packets in an anomaly) and time duration
of anomalies in the dataset.

###  4.3. Evaluation metrics and statistical tests

####  4.3.1. Evaluation metrics

We evaluate the effectiveness of KDDT with packet-level metrics in RQ1 - RQ4.
We also demonstrate the practical implications of KDDT by reporting incident-
level metrics in RQ1. Moreover, the perplexity metric is presented in RQ3 to
illustrate the quality of LM.
Packet-level effectiveness metrics evaluate the predictive performance of KDDT
on a single packet. Following the common practices in the literature, we adopt
three commonly used classification metrics: precision, recall, and F1 score.
Precision measures the accuracy of the positive predictions. Recall measures
the proportion of actual positive cases correctly classified by the model. F1
score is the harmonic average of precision and recall, evaluating the model
from both perspectives.
Incident-level effectiveness metrics aim to evaluate KDDT from a higher and
more practical perspective by focusing on the effectiveness with the anomaly
incidents as the minimum unit. We define the following four metrics.

Packet-Incident Coverage $\mathcal{C}_{PI}$ calculates the percentage of
abnormal packets identified in a single incident. Let $\mathcal{N}_{totalP}$
denote the total number of packets of an anomaly incident and
$\mathcal{N}_{correctP}$ denote the number of packets correctly identified by
KDDT. We formally define $\mathcal{C}_{PI}$ as in Equation 33.

(33) $\mathcal{C}_{PI}=\frac{\mathcal{N}_{correctP}}{\mathcal{N}_{totalP}}$

Incident Coverage $\mathcal{C}_{I}$ measures how many anomaly incidents are
identified. We assume an incident is identified if at least half of the
abnormal packets are correctly classified. We denote the number of identified
and total incidents as $\mathcal{N}_{correctI}$ and $\mathcal{N}_{totalI}$. We
formally define $\mathcal{C}_{I}$ in Equation 34.

(34) $\mathcal{C}_{I}=\frac{\mathcal{N}_{correctI}}{\mathcal{N}_{totalI}}$

Detection Time Rate $\mathcal{DTR}_{I}$ assesses the time percentage needed to
detect an anomaly incident. Let $t_{s}$ be the starting time of an incident
and $\hat{t}_{s}$ be the time that KDDT identifies the first abnormal packet
correctly. We denote the time duration for an anomaly incident as
$\mathcal{T}_{totalT}$. Formally, $\mathcal{TR}_{I}$ is defined in Equation
35.

(35) $\mathcal{DTR}_{I}=\frac{\hat{t}_{s}-t_{s}}{\mathcal{T}_{totalT}}$

Root Mean Square Error of Anomaly Length ($RMSE_{L}$) evaluates how well KDDT
can predict the anomaly length. Let the length of anomaly incident $i$ be
$L_{i}$ and the predicted incident length be $\hat{L}_{i}$, we formally define
$RMSE_{L}$ as:

(36) $RMSE_{L}=\sqrt{\frac{1}{n}\sum_{i=1}^{n}(L_{i}-\hat{L}_{i})^{2}}$

Perplexity of LM. Perplexity is commonly used to evaluate probabilistic
models, particularly LMs (Das and Verma, 2020). In the context of an LM,
perplexity measures the average likelihood of its predictions for the next
token. Higher perplexity indicates the model has more confidence in its
prediction and vice versa. We calculate perplexity as in Equation 37, where
$n$ denotes the sequence length and $w_{i}$ represents the $i$th token in the
sequence.

(37) $PPL=\exp\left(-\frac{1}{n}\sum_{i=1}^{n}\log
P(w_{i}|w_{1},\ldots,w_{i-1})\right)$

####  4.3.2. Statistical testing

Using deep learning models introduces randomness into KDDT, which might
threaten our empirical study’s validity. Therefore, we repeat each experiment
30 times and perform the Mann-Whitney U test with a significance level of 0.01
as suggested in (Arcuri and Briand, 2011). Furthermore, we evaluate the A12
effect size of the improvement (Arcuri and Briand, 2011). Method A has a
higher chance of getting better values if the A12 value is greater than 0.5
and vice versa. We consider the effect size in the range $[0.56,0.64)$ as
_Small_ , $[0.64,0.71)$ as _Medium_ , and $[0.71,1]$ as _Large_.

###  4.4. Experiment settings and execution

KDDT involves several hyper-parameters that potentially introduce human
biases. We performed a 10-fold cross validation for the hyper-parameter
selection with less bias (Browne, 2000). The code is built with the Pytorch
framework (Paszke et al., 2017). All the experiments were performed on one
node from a national, experimental, heterogeneous computational cluster called
eX3. This node is equipped with 2x Intel Xeon Platinum 8186, 1x NVIDIA V100
GPUs.

Hyper-parameter | Value | Hyper-parameter | Value
---|---|---|---
optimizer | AdamW | lr | 0.001
betas | (0.9, 0.999) | Embedd size | 64
$h^{vE}$ size | 32 |  $h^{dtmE}$ size | 16
$h^{vD}$ size | 32 |  $h^{dtmD}$ size | 16
CNN out channels | 6 | batch size | 12
Table 2. Hyperparameter values in KDDT. $lr$ denotes the learning rate of the
optimizer, and $betas$ are also arguments of AdamW (Loshchilov and Hutter,
2017). $h^{vE}$, $h^{dtmE}$, $h^{vD}$ and $h^{dtmD}$ represent the hidden
vector of VAE encoder, VAE decoder, DTM encoder, and DTM decoder,
respectively.

##  5\. Experiment Results

###  5.1. RQ1-KDDT effectiveness

Figure 4. Results of the packet-level effectiveness metrics

Figure 4 delineates KDDT’s performance on the packet level effectiveness
metrics. The average F1 scores on both datasets are above 0.91, which
demonstrates that KDDT is effective comprehensively. The precision results on
both datasets are above 0.9, which represents more than 90% of predicted
anomalous packets are true anomalies. The recall results reach 0.927 on the
Day 11 dataset and 0.937 on the Day 10 dataset, which indicates more than 92%
anomalous packets are successfully detected by KDDT.

We also observe that the performance of KDDT on the Day 11 dataset is
generally inferior to that on the Day 10 dataset. One possible reason for this
discrepancy is that the number of anomaly labels on the Day 11 dataset is
smaller than that on the Day 10 dataset, indicating a higher training
difficulty.

Dataset | $\mathcal{C}_{PI}$ | $\mathcal{C}_{I}$ | $\mathcal{DTR}_{I}$ | $RMSE_{L}$
---|---|---|---|---
Day 10 | 87.36% | 100% | 3.81% | 43.70
Day 11 | 81.09% | 100% | 6.22% | 185.06
Table 3. Results of the incident-level effectiveness metrics

Table 3 presents the experiment results for the incident-level effectiveness
metrics. $\mathcal{C}_{PI}$ results indicate that, on average, 87.36% and
81.09% abnormal packets in each anomaly incident are detected on Day 10 and
Day 11, respectively. The incident coverages are both 100%, showing that all
incidents are successfully identified. $\mathcal{DTR}_{I}$ on both datasets
are relatively low ($<7\%$), implying that KDDT can detect an anomaly incident
after the first 7% packets. A low $\mathcal{DTR}_{I}$ indicates KDDT can
detect anomalies near instantly after they take place, facilitating live
detection of various types of anomalies in TCMS, consequently curbing the
potential damage to the whole system. $RMSE_{L}$ signifies the standard
deviation of the residuals, which represent the discrepancy between the
predicted and the real incident duration. We can observe from the last column
of Table 3 that the $RMSE_{L}$ on both datasets (i.e., 43.70 and 185.06) is
substantially lower than the average lengths of incidents (407.23 and 1127.52
as reported in Table 1). This indicates that KDDT can predict the duration of
an anomaly incident with a relatively small residual. The advantage of small
residual can be harnessed for other applications, such as incident boundary
determination, where a dedicated model can be established to predict the start
and the end of an anomaly incident.  Concluding remarks on RQ1: Regarding the
packet-level effectiveness metrics (precision, recall and F1 score), KDDT
achieves more than 90% predictive performance on the Day 10 and Day 11
datasets. Regarding the incident-level effectiveness metrics, KDDT
demonstrates high packet and incident coverages, low detection time rate, and
relatively low RMSE of anomaly length prediction, implying KDDT’s potential
applications in incident detection, live detection, and incident boundary
determination.

###  5.2. RQ2-DTM Effectiveness

KDDT is a DT-based method, where DTM simulates TCMS and provides information
about the system state to DTC. To evaluate the contribution of DTM, we compare
KDDT and KDDT without DTM (denoted as KDDT-NoDTM).

Dataset | Metric | KDDT | NoDTM | p-value | A12
---|---|---|---|---|---
Day 10 | Precision | 0.926 | 0.816 | ¡0.01 | 1.000
Recall | 0.937 | 0.802 | ¡0.01 | 1.000
F1 score | 0.931 | 0.809 | ¡0.01 | 1.000
Day 11 | Precision | 0.903 | 0.760 | ¡0.01 | 1.000
Recall | 0.927 | 0.820 | ¡0.01 | 1.000
F1 score | 0.915 | 0.789 | ¡0.01 | 1.000
Table 4. Comparison of KDDT and KDDT-NoDTM

Concluding remarks on RQ2: We observe a substantial decrease in the two
datasets in terms of precision (12.7%), recall (12.1%), and F1 score (12.4%)
when we remove the DTM from KDDT. The DTM extracts inter-packet chronological
features from the packets, which is indispensable for accurate anomaly
detection.

###  5.3. RQ3-LM Effectiveness

KDDT employs LM to extract inner-packet context features. To evaluate its
effectiveness, we employ the perplexity metric (Section 4.3.1) to compare the
performance of KDDT’s LM with the baseline Word2vec model. Results are shown
in Figure 5. We can observe that the perplexity first decreases sharply and
then gradually converges to around 30 at the end of training (after 400k
batches). Low perplexity indicates that the LM has high confidence in
predicting the next token and extracting inner-packet context features.

Figure 5. Results of perplexity during LM training

Table 5 shows the comparison results of KDDT and KDDT-Word2Vec. We can observe
decreases in all three metrics on both datasets. The minimum decrease is
$0.011$ ($0.926-0.915$) on the precision of the Day 10 dataset. The decreases
are all significant except for the precision of the Day 10 dataset
(p-value$=0.066>0.01$). We also observe a large effect size in recall and F1
score on both datasets ($A12>0.71$ as defined in Section 4.3.2).

Dataset | Metric | KDDT | Word2Vec | p-value | A12
---|---|---|---|---|---
Day 10 | Precision | 0.926 | 0.915 | 0.066 | 0.200
Recall | 0.937 | 0.894 | ¡0.01 | 0.733
F1 score | 0.931 | 0.904 | ¡0.01 | 0.867
Day 11 | Precision | 0.903 | 0.882 | ¡0.01 | 0.600
Recall | 0.927 | 0.882 | ¡0.01 | 0.933
F1 score | 0.915 | 0.882 | ¡0.01 | 0.933
Table 5. Comparison of KDDT with KDDT-Word2Vec (denoted as Word2vec)

Concluding remarks on RQ3: LM presents low perplexity after training,
suggesting strong feature extraction capability. KDDT outperforms KDDT-
Word2vec significantly on both the Day 10 and Day 11 datasets (average F1
score improvement of 0.3).

###  5.4. RQ4-Knowledge Distillation Effectiveness

The training of KDDT is guided by the knowledge distilled from a teacher
model, i.e., the VAE encoder. To demonstrate the effectiveness of KD, we
compare KDDT and KDDT without KD (denoted as KDDT-NoKD). Table 6 depicts the
comparison results. All metrics experienced a remarkable decrease with a
minimum of 0.027 ($0.926-0.899$ on the precision of the Day 10 dataset). All
the p-values in this table are smaller than 0.01, indicating the significance
of the performance decrease brought by removing KD. We observe large effect
sizes ($A12>0.71$) in all metrics except precision on Day 10 dataset,
indicating KDDT-NoKD is highly probable to yield worse results than KDDT. Such
results are consistent with our expectations as we posit KD can incorporate
extra knowledge extracted from OOD data. Specifically, the pretrained VAE has
a larger capacity in terms of model complexity compared to DTM, allowing it to
fit a large $\mathcal{D}_{OOD}$ dataset without significant underfitting
issues. The hidden vectors produced by the pretrained VAE epitomize its
knowledge about feature extraction from a network packet. Therefore, using
these hidden vectors as soft targets to guide the training of DTM entails a
distillation process from the pretrained VAE to DTM. The extra knowledge
distilled from the pretrained VAE supplements the training of DTM on the
$\mathcal{D}_{ID}$ dataset, hence the predictive performance boost with KD.

Dataset | Metric | KDDT | NoKD | p-value | A12
---|---|---|---|---|---
Day 10 | Precision | 0.926 | 0.899 | ¡0.01 | 0.333
Recall | 0.937 | 0.874 | ¡0.01 | 1.000
F1 score | 0.931 | 0.886 | ¡0.01 | 0.933
Day 11 | Precision | 0.903 | 0.855 | ¡0.01 | 0.933
Recall | 0.927 | 0.824 | ¡0.01 | 1.000
F1 score | 0.915 | 0.839 | ¡0.01 | 1.000
Table 6. Comparison of KDDT and KDDT-NoKD

Concluding remarks on RQ4: Removing KD from KDDT leads to a remarkable
decrease ($>2.7\%$) in terms of average precision (3.75%), recall (8.3%), and
F1 score (6.05%). Most of the decreases are significant (p-value$<0.01$) with
large effect sizes ($A12>0.71$). We conclude that KD is effective in improving
the encoding capability of DTM.

###  5.5. Threats to Validity

Construct Validity concerns whether our chosen metrics accurately represent
the anomaly detection quality. To be comprehensive, we include two sets of
metrics: packet-level effectiveness metrics (i.e., precision, recall and F1
score) commonly used for evaluating anomaly detection methods and the
incident-level effectiveness metrics (i.e., the packet-incident coverage,
incident coverage, length RMSE and detection time rate) focusing on practical
implications and assess KDDT from a domain perspective. We argue that these
metrics together enable a more holistic assessment of KDDT.

Internal Validity refers to the credibility between cause and effect. One
possible threat to internal validity is the selection of hyperparameters. The
performance of DT may diminish in different settings. To reduce such threats,
we choose these hyperparameters with cross-validation, which yields generic
and optimal hyperparameters. The selection of hyperparameters does not
introduce any human bias against the experiment datasets.

Conclusion Validity pertains to the validity of the conclusions drawn from the
experiments. One common threat to conclusion validity is the randomness
introduced in the model. KDDT harnesses neural networks for anomaly detection,
inevitably introducing randomness. To mitigate this threat, we repeated each
experiment 30 times. We performed statistical testing to study the
significance of each improvement to ensure that the conclusions derived from
our study are reliable.

External Validity concerns the extent to which KDDT can generalize to other
contexts. To reduce threats to external validity, we design our method to be
generic, assuming no prior knowledge of the dataset distribution. Moreover,
despite the high cost of collecting data from the real system, we obtained two
separate datasets from different interfaces. Furthermore, we pretrain the VAE
with the OOD dataset comprising anomaly-unrelated data samples, which is
considered a universal task rather than bound to anomaly detection. The
pretrained VAE pertains to knowledge of encoding a network packet into a
hidden vector with high quality, benefitting various downstream tasks such as
intrusion detection, network traffic analysis, and robustness analysis.

##  6\. Practical Implications

Automating anomaly detection. Our experiment results show that KDDT is
effective for anomaly detection of Alstom’s TCMS network, comprehensively
measured with commonly used metrics for evaluating predictive performance
(e.g., precision) at the packet level and our newly proposed incident-level
metrics such as anomaly incident coverage. Especially, we observed that KDDT
reaches a 100% incident coverage for both datasets, indicating that all
anomaly incidents can be successfully detected. The practical implication of
this observation is that a high incident coverage can release domain experts
from excessive manual work of pinpointing anomaly incidents from vast network
packets.

Enabling live monitoring. Moreover, KDDT exhibited a low detection time rate,
critical for enabling live monitoring of the TCMS network by providing near-
instant alerts and warnings. An automatic protocol can be established to react
to these warnings and alerts and, in turn, prevents the anomaly’s further
influence on the TCMS. Furthermore, we observe a relatively low RMSE of the
anomaly length predicted by KDDT. Low RMSE demonstrates the effectiveness of
utilizing KDDT for anomaly boundary determination. An accurate boundary
determination facilitates the TCMS to react accordingly and appropriately. For
example, the TCMS can neglect a short-length anomaly, thanks to the network’s
re-transmission mechanism, whereas a long-length anomaly can lead to function
failures or even system failures, requiring resources and reactions from the
TCMS to mitigate the impact of the anomaly.

Leveraging DT for improving CPS dependability. Our experiment results
highlight that KDDT highly benefits from having DTM in the DT structure.
First, by simulating the TCMS, DTM can provide valuable information about the
operation state of the TCMS (e.g., unstable network states where frequent re-
transmissions occur). Second, developing the DTM is cost-efficient since it is
automatically developed to simulate the TCMS by predicting the next network
packet (Section 3.4.1). Predicting the next packet requires encoding the
packet into a high-quality hidden vector, which is a universal task that can
benefit many downstream tasks, such as intrusion detection and network traffic
analysis. In other words, we can reuse the same DTM to support different DTCs.

Benefiting from language models. Our results show that LM can provide
contextualized features compared to Word2vec. Extracting features from text is
extensively researched in NLP. Though we believe pretraining an LM with
network packet data is sufficient and cost-efficient for the network anomaly
detection tasks in the TCMS, the potential benefit of exploring Large Language
Models (LLMs) such as Elmo (Peters et al., 2018), Bert (Devlin et al., 2019)
and GPT (Radford et al., [n. d.]), is intriguing. As Wei et al. (Wei et al.,
2022) pointed out, LLMs show emergent capability, representing high predictive
performance and sample efficiency on downstream tasks that small LMs do not
possess. We are motivated to investigate the future application of LLMs with
network packet data.

Using KD to alleviate the need for abundant in-domain data. KDDT employs KD to
make use of OOD data. The nature of supervised machine learning requires
abundant labelled data for its training, which is expensive to collect. In
some cases, collecting merely ID dataset is non-trivial as well. For instance,
collecting anomaly-related data in the TCMS network (denoted as
$\mathcal{D}_{ID}$) requires manual examination and collection by domain
experts with the help of an in-house signal analysis tool DCUTerm. Moreover,
extra work is required for labelling the ID dataset. To alleviate the need for
ID data, KD can distil knowledge from the OOD dataset to DTM. Incorporating
the OOD dataset unleashes KDDT from the sole dependency of the ID dataset at a
relatively low cost since the OOD dataset (i.e., any normal network packets)
is easier to collect in the TCMS network.

##  7\. Related work

We discuss the related works from two aspects: DT for CPS and network anomaly
detection.

Digital Twin for CPS. DTs have been investigated to enhance the security and
safety of CPSs. DT originates from the Apollo program for mission training and
support (Rosen et al., 2015). Later, DT was generalized to accommodate various
CPSs, such as water treatment plants and power grids (Yue et al., 2021; Xu et
al., 2021). Eckhart and Ekelhart designed a rule-based DT dedicated to CPS
intrusion detection (Eckhart and Ekelhart, 2019), which constantly checks
rules violations that a CPS must adhere to under normal conditions. Instead of
relying on prior knowledge, Damjanovic-Behrendt (Damjanovic-Behrendt, 2018)
proposed to build a machine learning-based DT to learn privacy-related
features directly from automobile industry datasets, including both historical
and real-time data.

Yue et al. (Yue et al., 2021) proposed a DT conceptual model and further
decoupled the two sub-components of DT: DTM and DTC. They define the DTM as a
live replica of the CPS, while the DTC is the functionality of the DT. Xu et
al. (Xu et al., 2021) realized this conceptual model by building the DTM as a
timed automata machine and the DTC as a Generative Adversarial Network (GCN).
Experimental results on public testbeds demonstrate the effectiveness of such
decoupled DT structures. Latsou et al. (Latsou et al., 2023) extended the
concept of DT and proposed a multi-agent DT architecture for anomaly detection
and bottleneck identification in complex manufacturing systems. In addition to
DT construction for a specific CPS, Xu et al. (Xu et al., 2023b) and Lu et al.
(Lu et al., 2023) introduced transfer learning for DT evolution to synchronize
with the changes in CPSs and software systems, respectively. Despite the
success of these methods, building a data-driven DT inevitably requires
sufficient ID data. In contrast, KDDT adapts a VAE as the DTM structure and
utilizes KD to distill knowledge from OOD data. Doing so significantly
increases the data volume, improving the performance of DT.

Network Anomaly detection. Many approaches exist for network anomaly detection
(Ahmed et al., 2016), where this task is generally formulated as a
classification task. Early research on network anomaly detection originates in
statistical or rule-based approaches. Machine learning methods were used to
estimate the data distribution and employ statistical testing to detect
anomalies (Eskin, 2000). A rule-based network anomaly detector was proposed in
(Ghosh et al., 2017) to detect predefined rule violations and consider them
anomalies.

Statistical and rule-based approaches require extensive effort from domain
experts. To mitigate such problems, neural networks have been widely explored
for network anomaly detection due to their capability of automatic feature
extraction (Hooshmand and Hosahalli, 2022). Kwon et al. (Kwon et al., 2018)
evaluated three CNN architectures for network anomaly detection and concluded
that shallow CNN performs best. Li et al. (Li et al., 2017) divide network
fields into categorical and numerical fields and extract features separately.
The feature vectors are then converted into images with 8*8 grayscale pixels
and fed into a CNN to extract features automatically.

Few works have considered network packets as raw text and utilized NLP for
feature extraction. Goodman et al. (Goodman et al., 2020) modified the
Word2vec model and generated embeddings for each token in a network packet.
The proposed method then takes these embeddings as input features to classify
network traffics into benign or malicious. Our method KDDT follows this
research line and uses contextualized LM as an alternative to Word2vec. LM
provides valuable context information that can help extract high-quality
features from intricate packets. Our results show that the contextualized LM
outperforms Word2vec and leads to high predictive performance.

##  8\. Conclusion and Future Work

In this study, we introduced a digital twin (DT) based approach, referred to
as KDDT, for tackling the anomaly detection task within Train Control and
Management System (TCMS) networks. KDDT leverages a language model (LM) and an
LSTM to extract inner-packet context and inter-packet chronological features.
To capitalize on the information present in out-of-domain (OOD) data, we train
a Variational Autoencoder with OOD data and use it to guide the training
process of the digital twin model (DTM). We evaluate KDDT with two datasets
from Alstom. Experimental results show the effectiveness of KDDT regarding
packet-level and incident-level metrics. We also investigate the individual
contribution of each component. Experimental results demonstrate the
effectiveness of DTM, LM and KD, which can be leveraged to address various
tasks other than anomaly detection. We plan to explore more contextualized LM,
such as ElMo, Bert, and GPT. We are particularly interested in the potential
application of ChatGPT in this domain despite of obvious drawbacks mentioned
in Section 5.5.

##  9\. Acknowlegements

The project is supported by the security project funded by the Norwegian
Ministry of Education and Research, the Horizon 2020 project ADEPTNESS
(871319) funded by the European Commission, and the Co-tester (#314544)
project funded by the Research Council of Norway (RCN). This work has
benefited from the Experimental Infrastructure for Exploration of Exascale
Computing (eX3), which is financially supported by RCN under contract 270053.

## References

  * (1)
  * Ahmed et al. (2016) Mohiuddin Ahmed, Abdun Naser Mahmood, and Jiankun Hu. 2016\.  A survey of network anomaly detection techniques.  _Journal of Network and Computer Applications_ 60 (Jan. 2016), 19–31.  https://doi.org/10.1016/j.jnca.2015.11.016
  * Anushiya and Lavanya (2021) R Anushiya and V S Lavanya. 2021.  A COMPARATIVE STUDY ON INTRUSION DETECTION SYSTEMS FOR SECURED COMMUNICATION IN INTERNET OF THINGS.  _ICTACT JOURNAL ON COMMUNICATION TECHNOLOGY_ 12, 03 (2021).
  * Arcuri and Briand (2011) Andrea Arcuri and Lionel Briand. 2011.  A practical guide for using statistical tests to assess randomized algorithms in software engineering.  _Proceedings - International Conference on Software Engineering_ (2011), 1–10.  https://doi.org/10.1145/1985793.1985795
  * Browne (2000) Michael W Browne. 2000\.  Cross-validation methods.  _Journal of mathematical psychology_ 44, 1 (2000), 108–132.
  * Clark et al. (2018) Kevin Clark, Minh-Thang Luong, Christopher D Manning, and Quoc V Le. 2018.  Semi-supervised sequence modeling with cross-view training.  _arXiv preprint arXiv:1809.08370_ (2018).
  * Damjanovic-Behrendt (2018) Violeta Damjanovic-Behrendt. 2018\.  _A Digital Twin-based Privacy Enhancement Mechanism for the Automotive Industry_.  https://doi.org/10.1109/IS.2018.8710526 Pages: 279.
  * Das and Verma (2020) Avisha Das and Rakesh M Verma. 2020.  Can Machines Tell Stories? A Comparative Study of Deep Neural Language Models and Metrics.  _IEEE Access_ 8 (2020), 181258–181292.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  http://arxiv.org/abs/1810.04805 arXiv:1810.04805 [cs].
  * Dong et al. (2021) Shi Dong, Ping Wang, and Khushnood Abbas. 2021.  A survey on deep learning and its applications.  _Computer Science Review_ 40 (2021), 100379.  https://doi.org/10.1016/j.cosrev.2021.100379
  * Eckhart and Ekelhart (2019) Matthias Eckhart and Andreas Ekelhart. 2019.  Digital Twins for Cyber-Physical Systems Security: State of the Art and Outlook.  In _Security and Quality in Cyber-Physical Systems Engineering: With Forewords by Robert M. Lee and Tom Gilb_ , Stefan Biffl, Matthias Eckhart, Arndt Lüder, and Edgar Weippl (Eds.). Springer International Publishing, Cham, 383–412.  https://doi.org/10.1007/978-3-030-25312-7_14
  * Eskin (2000) Eleazar Eskin. 2000\.  Anomaly Detection over Noisy Data Using Learned Probability Distributions. In _Proceedings of the Seventeenth International Conference on Machine Learning_ _(ICML ’00)_. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 255–262.
  * Ghosh et al. (2017) Soumadip Ghosh, Arindrajit Pal, Amitava Nag, Shayak Sadhu, and Ramsekher Pati. 2017.  Network anomaly detection using a fuzzy rule-based classifier.  In _Computer, Communication and Electrical Technology_ (1 ed.), Debatosh Guha, Badal Chakraborty, and Himadri Sekhar Dutta (Eds.). CRC Press, 61–65.  https://doi.org/10.1201/9781315400624-12
  * Goodman et al. (2020) Eric L. Goodman, Chase Zimmerman, and Corey Hudson. 2020\.  Packet2Vec: Utilizing Word2Vec for Feature Extraction in Packet Data.  http://arxiv.org/abs/2004.14477 arXiv:2004.14477 [cs].
  * Guzman et al. (2018) Daniela Narezo Guzman, Edin Hadzic, Robert Schuil, Eric Baars, and Jörn Christoffer Groos. 2018.  Data-driven condition now- and forecasting of railway switches for improvement in the quality of railway transportation.  (2018).
  * H. Zhao et al. (2017) H. Zhao, H. Chen, W. Dong, X. Sun, and Y. Ji. 2017.  Fault diagnosis of rail turnout system based on case-based reasoning with compound distance methods. In _2017 29th Chinese Control And Decision Conference (CCDC)_. 4205–4210.  https://doi.org/10.1109/CCDC.2017.7979237 Journal Abbreviation: 2017 29th Chinese Control And Decision Conference (CCDC).
  * Han et al. (2022a) Liping Han, Shaukat Ali, Tao Yue, Aitor Arrieta, and Maite Arratibel. 2022a.  _Uncertainty-aware Robustness Assessment of Industrial Elevator Systems_.  Technical Report.
  * Han et al. (2022b) Liping Han, Tao Yue, Shaukat Ali, Aitor Arrieta, and Maite Arratibel. 2022b.  Are Elevator Software Robust against Uncertainties? Results and Experiences from an Industrial Case Study. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Singapore, Singapore) _(ESEC/FSE 2022)_. Association for Computing Machinery, New York, NY, USA, 1331–1342.  https://doi.org/10.1145/3540250.3558955
  * Hooshmand and Hosahalli (2022) Mohammad Kazim Hooshmand and Doreswamy Hosahalli. 2022.  Network anomaly detection using deep learning techniques.  _CAAI Transactions on Intelligence Technology_ 7, 2 (June 2022), 228–243.  https://doi.org/10.1049/cit2.12078
  * Islam et al. (2022) Umar Islam, Rami Qays Malik, Amnah S. Al-Johani, Muhammad. Riaz Khan, Yousef Ibrahim Daradkeh, Ijaz Ahmad, Khalid A. Alissa, Zulkiflee Abdul-Samad, and Elsayed M. Tag-Eldin. 2022\.  A Novel Anomaly Detection System on the Internet of Railways Using Extended Neural Networks.  _Electronics_ 11, 18 (Sept. 2022), 2813\.  https://doi.org/10.3390/electronics11182813
  * Jones et al. (2020) David Jones, Chris Snider, Aydin Nassehi, Jason Yon, and Ben Hicks. 2020.  Characterising the Digital Twin: A systematic literature review.  _CIRP Journal of Manufacturing Science and Technology_ 29 (2020), 36–52.
  * Kwon et al. (2018) Donghwoon Kwon, Kathiravan Natarajan, Sang C. Suh, Hyunjoo Kim, and Jinoh Kim. 2018.  An Empirical Study on Network Anomaly Detection Using Convolutional Neural Networks. In _2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS)_. 1595–1598.  https://doi.org/10.1109/ICDCS.2018.00178
  * Lamping and Warnicke (2004) Ulf Lamping and Ed Warnicke. 2004.  Wireshark user’s guide.  _Interface_ 4, 6 (2004), 1.
  * Latsou et al. (2023) Christina Latsou, Maryam Farsi, and John Ahmet Erkoyuncu. 2023\.  Digital twin-enabled automated anomaly detection and bottleneck identification in complex manufacturing systems using a multi-agent approach.  _Journal of Manufacturing Systems_ 67 (April 2023), 242–264.  https://doi.org/10.1016/j.jmsy.2023.02.008
  * Li et al. (2017) Zhipeng Li, Zheng Qin, Kai Huang, Xiao Yang, and Shuxiong Ye. 2017.  Intrusion Detection Using Convolutional Neural Networks for Representation Learning. In _Neural Information Processing_ , Derong Liu, Shengli Xie, Yuanqing Li, Dongbin Zhao, and El-Sayed M. El-Alfy (Eds.). Springer International Publishing, Cham, 858–866.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ (2017).
  * Lu et al. (2023) Chengjie Lu, Qinghua Xu, Tao Yue, Shaukat Ali, Thomas Schwitalla, and Jan F. Nygård. 2023\.  EvoCLINICAL: Evolving Cyber-cyber Digital Twin with Active Transfer Learning for Automated Cancer Registry System. In _Proceedings of the 31th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (San Francisco, CA, USA) _(ESEC/FSE 2023)_. Association for Computing Machinery, New York, NY, USA, 11 pages.  https://doi.org/10.1145/3611643.3613897
  * Mimura and Tanaka (2018) Mamoru Mimura and Hidema Tanaka. 2018.  Reading network packets as a natural language for intrusion detection. In _Information Security and Cryptology–ICISC 2017: 20th International Conference, Seoul, South Korea, November 29-December 1, 2017, Revised Selected Papers 20_. Springer, 339–350.
  * Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017\.  Automatic differentiation in pytorch.  (2017).
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  http://arxiv.org/abs/1802.05365 arXiv:1802.05365 [cs].
  * Pinheiro Cinelli et al. (2021) Lucas Pinheiro Cinelli, Matheus Araújo Marins, Eduardo Antúnio Barros da Silva, and Sérgio Lima Netto. 2021\.  Variational Autoencoder.  In _Variational Methods for Machine Learning with Applications to Deep Networks_ , Lucas Pinheiro Cinelli, Matheus Araújo Marins, Eduardo Antônio Barros da Silva, and Sérgio Lima Netto (Eds.). Springer International Publishing, Cham, 111–149.  https://doi.org/10.1007/978-3-030-70679-1_5
  * Radford et al. ([n. d.]) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. [n. d.].  Improving Language Understanding by Generative Pre-Training.  ([n. d.]).
  * Rosen et al. (2015) Roland Rosen, Georg von Wichert, George Lo, and Kurt Dirk Bettenhausen. 2015.  About The Importance of Autonomy and Digital Twins for the Future of Manufacturing.  _IFAC-PapersOnLine_ 48 (2015), 567–572.
  * Wei et al. (2022) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022\.  Emergent Abilities of Large Language Models.  http://arxiv.org/abs/2206.07682 arXiv:2206.07682 [cs].
  * Xu et al. (2021) Qinghua Xu, Shaukat Ali, and Tao Yue. 2021.  Digital twin-based anomaly detection in cyber-physical systems. IEEE, 205–216.
  * Xu et al. (2023a) Qinghua Xu, Shaukat Ali, and Tao Yue. 2023a.  Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems.  _ACM Transactions on Software Engineering and Methodology_ (Feb. 2023), 3582571\.  https://doi.org/10.1145/3582571
  * Xu et al. (2022) Qinghua Xu, Shaukat Ali, Tao Yue, and Maite Arratibel. 2022\.  Uncertainty-aware transfer learning to evolve digital twins for industrial elevators. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. ACM, Singapore Singapore, 1257–1268.  https://doi.org/10.1145/3540250.3558957
  * Xu et al. (2023b) Qinghua Xu, Shaukat Ali, Tao Yue, Nedim Zaimovic, and Singh Inderjeet. 2023b.  Uncertainty-Aware Transfer Learning to Evolve Digital Twins for Industrial Elevators. In _Proceedings of the 31th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (San Francisco, CA, USA) _(ESEC/FSE 2023)_. Association for Computing Machinery, New York, NY, USA, 11 pages.  https://doi.org/10.1145/3611643.3613879
  * Yu et al. (2019) Yong Yu, Xiaosheng Si, Changhua Hu, and Jianxun Zhang. 2019\.  A review of recurrent neural networks: LSTM cells and network architectures.  _Neural computation_ 31, 7 (2019), 1235–1270.
  * Yue et al. (2021) Tao Yue, Paolo Arcaini, and Shaukat Ali. 2021.  Understanding Digital Twins for Cyber-Physical Systems: A Conceptual Model. In _Leveraging Applications of Formal Methods, Verification and Validation: Tools and Trends_ , Tiziana Margaria and Bernhard Steffen (Eds.). Springer International Publishing, Cham, 54–71.
  * Zafar et al. (2021) Muhammad Nouman Zafar, Wasif Afzal, Eduard Enoiu, Athanasios Stratis, Aitor Arrieta, and Goiuria Sagardui. 2021.  Model-Based Testing in Practice: An Industrial Case Study using GraphWalker. In _14th Innovations in Software Engineering Conference (formerly known as India Software Engineering Conference)_. ACM, Bhubaneswar, Odisha India, 1–11.  https://doi.org/10.1145/3452383.3452388