# NLP Methods in Host-based Intrusion Detection Systems: A Systematic Review
and Future Directions

Zarrin Tasnim Sworna111School of Computer Science, University of Adelaide and
Cyber Security Cooperative Research Centre, Australia. Zahra Mousavi222School
of Computer Science, University of Adelaide and CSIRO/Data61, Australia.
Muhammad Ali Babar333School of Computer Science, University of Adelaide and
Cyber Security Cooperative Research Centre, Australia.

###### Abstract

Host-based Intrusion Detection System (HIDS) is an effective last line of
defense for defending against cyber security attacks after perimeter defenses
(e.g., Network-based Intrusion Detection System and Firewall) have failed or
been bypassed. HIDS is widely adopted in the industry as HIDS is ranked among
the top two most used security tools by Security Operation Centers (SOC) of
organizations. Although effective and efficient HIDS is highly desirable for
industrial organizations, the evolution of increasingly complex attack
patterns causes several challenges resulting in performance degradation of
HIDS (e.g., high false alert rate creating alert fatigue for SOC staff). Since
Natural Language Processing (NLP) methods are better suited for identifying
complex attack patterns, an increasing number of HIDS are leveraging the
advances in NLP that have shown effective and efficient performance in
precisely detecting low footprint, zero-day attacks and predicting an
attacker’s next steps. This active research trend of using NLP in HIDS demands
a synthesized and comprehensive body of knowledge of NLP-based HIDS. Thus, we
conducted a systematic review of the literature on the end-to-end pipeline of
the use of NLP in HIDS development. For the end-to-end NLP-based HIDS
development pipeline, we identify, taxonomically categorize and systematically
compare the state-of-the-art of NLP methods usage in HIDS, attacks detected by
these NLP methods, datasets and evaluation metrics which are used to evaluate
the NLP-based HIDS. We highlight the relevant prevalent practices,
considerations, advantages and limitations to support the HIDS developers. We
also outline the future research directions for the NLP-based HIDS
development.

###### keywords:

Natural Language Processing , Host-based Intrusion Detection , Cyber Security
, Anomaly Detection

††journal: Journal of Network and Computer Applications

##  1 Introduction

Cyber security attacks increased by 1,885% worldwide in 2021 (SonicWall,
2022). The attacks fueled by the profit-making cyber-crime economy with
nation-state targets are severely affecting every industry including
healthcare, financial services and government organizations; costing 4.24
million dollars per attack breach on average globally in 2021 (IBM, 2022). To
protect an organization by minimizing cyber attacks and thwarting new threats
targeting an organization’s IT infrastructure, a Host-based Intrusion
Detection System (HIDS) is an effective last line of defense to detect and
prevent malicious activities after perimeter defenses (e.g., Network-based
Intrusion Detection System (NIDS), Firewall) have failed or been bypassed
(Bukac et al., 2012). The use of HIDS is a prevalent security practice as HIDS
consolidated its position as a preferred defense technique and ranked among
the top 2 most used security tools by organizations among various commercial
tools (Swimlane, 2021; Insiders, 2018). HIDS helps organizations to identify
malicious activities by monitoring host data (e.g., system call sequence)
which, if left undetected, can lead to detrimental breaches. Even if HIDS
could identify and prevent 5% of all cyber attacks on time worldwide, it could
help avoid a loss of around 260 billion USD (Grimmer et al., 2021). Hence,
performance enhancement of HIDS is an active research area due to the
ubiquitous adoption of HIDS (Jose et al., 2018).

Despite all the research and industrial efforts, the evolution of the
increasingly complex attack vectors causes HIDS to suffer from a high false
alarm rate. The high false alarm rate of HIDS significantly contributes to
alarm fatigue of the security team (Čeponis and Goranin, 2020). Other key
challenges for the existing HIDS include handling a massive amount of system
call traces, run-time attack detection by continuous monitoring of host data,
keeping good detection rate, handling imbalanced data and reducing the
required processing time and resource consumption. These challenges hinder
large-scale HIDS deployment in commercial settings (Khreich et al., 2017).

To mitigate the above-mentioned HIDS challenges due to the evolution of the
increasingly complex attack patterns, traditional HIDS techniques are becoming
obsolete. Hence, researchers are introducing new Natural Language Processing
(NLP)-based HIDS techniques which are better suited to complex attack patterns
(Bouzar-Benlabiod et al., 2019; Borisaniya et al., 2015). HIDS researchers are
motivated to benefit from methods that have been proven successful in the NLP
domain, due to the high similarity between natural languages and System Call
(syscall) sequences of a host used by HIDS (Bouzar-Benlabiod et al., 2019; Lv
et al., 2018). Syscalls denote the interface of user space programs for
requesting services (e.g., read a file) from the Operating System’s (OS)
kernel. The way humans communicate with each other through natural language
constructed based on specific grammar and syntax, host processes gain their
intentions with syscall sequences, which are used by processes to communicate
with OS in a host (Lv et al., 2018). For HIDS development, a comprehensive and
detailed source of understanding a host’s behavior is syscalls executed in
that host (Grimmer et al., 2021; Ring IV et al., 2021). This understanding
enables fast detection of anomalies at run-time by analyzing streams of
syscalls (Ring IV et al., 2021).

Table 1: Example syscall sequences for reading two files (Grimmer et al., 2021) Seq1 | “open, read, write, open, read, write”
---|---
Seq2 | “open, open, read, write, read, write”
Seq3 | “open, open, read, read, write, write”

Consider a program that opens two files to read at the same time. When the
program is running, the syscalls (e.g., open) sequences created by threads of
the program may vary as shown in Table 1. If the only sequence in the normal
pattern library of a traditional HIDS is Seq1 and if the input sequence of
HIDS for detection is Seq2, a traditional HIDS based on pattern matching may
create a false alarm considering Seq2 as abnormal (Grimmer et al., 2021).
Hence, researchers consider syscall sequences of a syscall trace as sentences
of a document with each syscall as a word, where the order, frequency and
semantics of each word contributes to the understanding of host behavior
(Bouzar-Benlabiod et al., 2020; Diep et al., 2018). Since NLP has different
methods for analyzing the context and semantics of a word, HIDS researchers
are using NLP methods to analyze the contextual syscall similarity
(Vinayakumar et al., 2019), behavioral syscall semantics (Liu et al., 2020b),
to preserve the sequence information of syscall sequence (Vinayakumar et al.,
2019) and to learn correlation of syscalls (Bouzar-Benlabiod et al., 2020).

Researchers are developing NLP-inspired problem formulation for HIDS
development such as the use of question-answer models for generating future
syscalls considering previously invoked syscall sequence as question and
generated syscall sequence as answer (Sarraf and Swetha, 2019) and considering
host intrusion detection as classifying a sentence into normal or abnormal
that is similar to sentiment classification (Diep et al., 2018). The latest
HIDS research trend is the adoption of NLP methods considering from simple NLP
methods such as the use of n-gram and Term Frequency-Inverse Document
Frequency (TF-IDF) to extract contextual and statistical features to advanced
NLP methods such as word embedding, semantic ontology and language modeling
for domain adaptation, knowledge fusion and next syscalls prediction. The use
of NLP enabled the HIDS researchers to develop HIDS with more accurate
intrusion detection capability of unknown attacks in real-time by processing
syscall streams on the fly (Bouzar-Benlabiod et al., 2020; Creech and Hu,
2013b) and the ability to predict a future syscall sequence possibly to be
executed during an attack (Sarraf and Swetha, 2019).

Despite the drastically growing adoption of NLP in HIDS development, there has
been relatively little effort allocated to systematically analyze and
synthesize the available peer review literature to understand how NLP is used
in HIDS development. The lack of a synthesized and comprehensive body of
knowledge on such an important topic motivated us to conduct a Systematic
Literature Review (SLR) of the papers on the end-to-end pipeline of the use of
NLP in HIDS development. For the end-to-end NLP-based HIDS development
pipeline, we identify, taxonomically categorize and systematically compare the
state-of-the-art of NLP methods usage in HIDS, attacks detected by these NLP
methods, datasets and metrics which are used to evaluate the NLP-based HIDS.
As our focus is on the use of NLP, we consider HIDS that use syscall sequence
and text data sources (e.g., text-based cyber threat intelligence data) as
primary data source.

The key contributions of this review are as follows:

  * •

We are the first, to the best of our knowledge, to conduct a comprehensive and
systematic review of the literature on the end-to-end pipeline of the use of
NLP in HIDS development.

  * •

We propose a taxonomy for consistently classifying the current and future
research on NLP-based HIDS. We demonstrate the use of our taxonomy by
identifying 6 categories of NLP methods used in HIDS. For the evaluation of
NLP-based HIDS, we categorize 133 attacks, 20 datasets and 17 evaluation
metrics from the reviewed papers.

  * •

We synthesize and discuss the pros and cons of each category of the used NLP
methods, datasets, evaluation metrics and other key factors (e.g.,
classifiers, learning types) to help HIDS researchers and developers
understand their characteristics.

  * •

We highlight open issues with the current HIDS practices and propose potential
research directions in the intersection of HIDS and NLP research.

Our SLR findings are expected to help in better understanding the landscape of
the existing NLP-based HIDS for both HIDS researchers and HIDS developers. We
recommend researchers to focus on developing real-time accurate HIDS
leveraging advanced NLP techniques (e.g., text augmentation, NLP-based low-
shot learning), enhancing HIDS model interpretability and semantic integration
of HIDS in a Security Operation Center (SOC). For developers, our findings are
expected to help them to enhance the existing HIDS or develop new HIDS using
NLP method for supporting the construction and deployment of effective and
efficient HIDS. To support developers in this regard, we highlight the
relevant existing practices, considerations, advantages and limitations. We
recommend that HIDS developers train and validate the HIDS model using NLP
methods with their industry specific data before the deployment of HIDS as the
most used public datasets that we found are usually outdated and lack
sufficient and diverse attack instances.

Table 2: Comparative analysis of our SLR with the existing HIDS survey papers

| NLP | Feature | Classifier | Attack |  | Evaluation |  | Common
---|---|---|---|---|---|---|---|---
Study | Focus | Extraction Tech | (ML/DL/Rules) | Types | Datasets | Metrics | SLR | Papers
Ours | $\checkmark$ | $\checkmark$ | All |  $\checkmark$ (12) | $\checkmark$ |  $\checkmark$ (17) | $\checkmark$ | -
(Jose et al., 2018) | $\times$ | $\times$ | no DL | $\times$ | $\times$ | $\times$ | $\times$ | 2
(Liu et al., 2018) | $\times$ | $\times$ | no DL | $\times$ | $\checkmark$ |  $\checkmark$ (8) | $\times$ | 6
(Bridges et al., 2019) | $\times$ | $\times$ | no DL |  $\checkmark$ (4) | $\checkmark$ | $\times$ | $\times$ | 5
(Bukac et al., 2012) | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | 0

  * •

✓represents yes, $\times$ represents no, - represents not available

The rest of this paper is organized as follows. Section 2 compares our SLR to
existing surveys. Section 3 outlines our SLR methodology. Section 4 presents
the overview of our results in terms of study distribution and our proposed
taxonomy. Sections 5 and 6 present a detailed analysis of the used NLP methods
for HIDS (RQ1) and evaluation of NLP-based HIDS (RQ2). In Section 7, open
challenges with future research directions are discussed. Section 8 presents
the threats to validity of our findings. Lastly, the paper is concluded in
Section 9.

##  2 Comparison to the Existing Literature Reviews

Intrusion Detection Systems (IDS) has consolidated its position as a must use
defense technique with various commercial IDS to detect attacks within state,
personal and industrial Information Technology (IT) infrastructures (Zarpelão
et al., 2017). IDS is a highly active research domain and the existing
literature has been reviewed from diverse aspects due to the significance of
intrusion detection to protect an organization from cyber attacks. For
ensuring the unique and novel contribution of our SLR, we extensively analyzed
the related reviews and compared them with our SLR in this section.

A large number of review studies (KP et al., 2020; Aleesa et al., 2020;
Aldweesh et al., 2020; Kalimuthan and Renjit, 2020; Gümüşbaş et al., 2020; Liu
and Lang, 2019; Berman et al., 2019; Khraisat et al., 2019; Salo et al., 2018;
Xin et al., 2018; Ahmadian Ramaki et al., 2018; Aleroud and Karabatis, 2017;
Hodo et al., 2017; Bansal et al., 2016; Buczak and Guven, 2015; Liao et al.,
2013; Bukac et al., 2012) are available in the literature, which primarily
focus on NIDS. NIDS is significantly different from HIDS as NIDS detects
attacks in an organization’s network, while HIDS detects attacks on an
organization’s host system after NIDS has failed or been bypassed (Bukac et
al., 2012). While NIDS monitors the network traffic, HIDS monitors host data
(e.g., syscalls). Compared to NIDS, HIDS has the ability to detect insider
attacks and Advanced Persistent Threats (i.e, APT involves intruders staying
in an organization’s system for years by evading detection mechanisms) (Liu et
al., 2018). Due to the significant differences between HIDS and NIDS, the
existing reviews (Mohammadi et al., 2021; KP et al., 2020; Khraisat et al.,
2019; Liu and Lang, 2019; Xin et al., 2018; Ahmadian Ramaki et al., 2018;
Aleroud and Karabatis, 2017; Hodo et al., 2017) have only 1 to 2 papers and
the reviews (Aleesa et al., 2020; Aldweesh et al., 2020; Kalimuthan and
Renjit, 2020; Gümüşbaş et al., 2020; Berman et al., 2019; Salo et al., 2018;
Bansal et al., 2016; Buczak and Guven, 2015; Liao et al., 2013; Bukac et al.,
2012) have 0 papers in common with our HIDS focused SLR.

Four reviews are available which focused on HIDS (Bridges et al., 2019; Jose
et al., 2018; Liu et al., 2018; Bukac et al., 2012) (comparative analysis
shown in Table 2). These reviews differ from our SLR in terms of objectives,
included papers and results. None of these existing HIDS reviews focus on NLP.
The semantics and contextual analysis ability of NLP methods help HIDS to
detect unknown and adversarial attacks with lower False Alarm Rate (FAR) and
higher accuracy (Bouzar-Benlabiod et al., 2020; Creech and Hu, 2013b). Hence,
we focus on NLP to identify NLP-based methods (e.g., word embedding and
language modeling) and the future research directions.

A decade-old review (Bukac et al., 2012) focused on standalone HIDS and did
not consider hybrid or collaborative HIDS, while our SLR does not confine the
scope to standalone HIDS. The authors discussed network traffic, process
behavior, file integrity and security of HIDS against tampering without
focusing on any Machine Learning (ML), Deep Learning (DL) or NLP approaches.
Another review (Jose et al., 2018) aims at the existing anomaly detection
techniques of other domains and how they can be adopted from other domains to
the HIDS domain, whereas our SLR discusses both misuse and anomaly detection-
based HIDS. This review does not focus on features, attacks, datasets and
evaluation metrics, which are discussed in our SLR. Another review (Bridges et
al., 2019) categorized the host data sources (e.g., system logs, windows
registry) to discuss the existing literature from data source perspective.
This review does not focus on the application of NLP methods, feature
extraction methods (e.g., manual, automated), DL models and metrics, whereas
our SLR analyses all these HIDS aspects. This review included the majority of
the papers before 2010; also they prioritized the papers covering the use of
host data sources including NIDS. In contrast, we conduct SLR in the
(2010-2022) time-range as the adoption of NLP techniques in HIDS research
gained momentum in this decade (Creech and Hu, 2013b). The review reported by
(Liu et al., 2018) focused on syscall-based HIDS and their application on
embedded systems, in contrast, we exclude (discussed in Section 3.2) the HIDS
of any specific application area (e.g., embedded systems and IoT).

Table 3: Research questions that are addressed in our SLR Research Questions |  Motivation
---|---
RQ1: How have the NLP methods been used over the years by researchers to develop HIDS? |  To identify and categorize the NLP methods and how these NLP methods are used to develop HIDS. The aim is to provide insights to HIDS developers and researchers about the existing prevalent practices and considerations in NLP-based HIDS and how NLP methods are beneficial to improve HIDS performance for industrial adoption.
RQ2: How do researchers evaluate the NLP-based HIDS? |
$\bullet$ RQ2.1: What type of attacks do researchers aim to detect using NLP methods in HIDS? |  To identify what type of attacks can be detected by NLP-based HIDS and what are the impacts of these attacks
$\bullet$ RQ2.2: What are the datasets used to apply NLP methods in HIDS research? |  To provide insight to practitioners and researchers on the HIDS datasets that are used to evaluate NLP-based HIDS including dataset type and availability.
$\bullet$ RQ2.3: Which evaluation metrics are used to evaluate NLP-based HIDS? |  To identify the evaluation metrics that are used to evaluate NLP-based HIDS

In summary, none of the above-mentioned existing HIDS reviews focus on NLP
application, feature extraction methods and DL-based approaches, as shown in
Table 2. We identified, categorized and analyzed NLP methods, attacks,
datasets and evaluation metrics used in NLP-based HIDS. Notably, none of the
existing surveys covered DL-based techniques (e.g., language modeling, word
embedding), while we identified 24 studies in the recent years using DL
approaches with significantly better performance for real-life applications
handling a huge volume of data. Considering the included papers, our SLR
notably differs from these existing HIDS reviews having only 2, 6, 5 and 0
papers in common in the studies (Jose et al., 2018; Liu et al., 2018; Bridges
et al., 2019; Bukac et al., 2012), respectively.

There are several reviews on NLP or text mining applications in different
domains such as financial domain (Kumar and Ravi, 2016), social network (Irfan
et al., 2015), market prediction (Nassirtoussi et al., 2014), and a recent one
in cyber security domain (Ignaczak et al., 2021). The review in cyber security
domain (Ignaczak et al., 2021) presents a broader view by identifying the
tasks (e.g., cyber bullying, forensic analysis and sentiment analysis) in the
cyber security domain which are supported by text mining, while our study
thoroughly analyzes and focuses on NLP-based HIDS.

##  3 Research Methodology

We conducted a Systematic Literature Review (SLR) using the widely adopted SLR
guideline (Kitchenham and Charters, 2007). SLR is a widely adopted research
approach in Evidence-Based Software Engineering (EBSE) (Kitchenham et al.,
2004) as SLR evaluates and interprets a research topic utilizing a reliable,
rigorous and auditable methodology (Kitchenham and Charters, 2007). For
guiding our analysis, we aimed to answer two research questions (RQs) as shown
in Table 3 with the corresponding motivations. Our review protocol steps are
presented in the following Subsections (3.1\- 3.4).

###  3.1 Search Strategy

We formulated our search strategy to retrieve the maximum number of relevant
studies based on the guideline provided by (Kitchenham and Charters, 2007).
The search strategy includes the following steps.

####  3.1.1 Search Method

We used the automated database search method (Kitchenham and Charters, 2007)
to retrieve the relevant studies from digital search engines and databases. We
used the largest academic literature database Scopus digital library, which
indexes over 5,000 publishers worldwide including the relevant sources (e.g.,
Elsevier, Springer) (Croft et al., 2021). We complemented Scopus with IEEE
Xplore and ACM Digital Library, which are the most frequently used academic
digital libraries (Lin et al., 2020). Moreover, we complemented the automatic
search by extracting more relevant studies using snowballing (Wohlin, 2014).

####  3.1.2 Search String

We used the guidelines of (Kitchenham and Charters, 2007) to develop a
comprehensive search string. Considering the key terms “host intrusion
detection” and “NLP”, we created several pilot search strings composed of
synonyms and related terms. For the first term, we considered its varied
representation (e.g., Host based intrusion detection, host IDS) along with the
terms related to both anomaly and misuse detection. We excluded the term
‘HIDS’ as it is also used to represent ‘high-dimensional and sparse (HiDS)’,
which provided irrelevant papers. Regarding the second term, we noticed the
inclusion of ‘NLP’ is not useful as the papers on HIDS may not specify such a
term even though they use a wide variety of NLP techniques (e.g. n-grams, word
embedding and language modeling). After executing a series of pilot searches
in titles, abstracts and keywords of papers on databases and checking the
inclusion of papers that were known to us, we designed the following search
string.

“host intrusion detection” OR “host based intrusion detection” OR “host
anomaly detection” OR “host based anomaly detection” OR “host based anomaly
intrusion detection” OR “host based ids” OR “host ids” OR “host based misuse
intrusion detection” OR (“signature based intrusion detection” AND “host”)

Table 4: Inclusion and Exclusion Criteria Inclusion Criteria
---
I1: Publications that utilize NLP methods to develop host-based intrusion
detection system including anomaly and misuse detection approaches.
I2: Publications that are presented at a conference or a journal.
I3: Studies that are published within the search timeline of January 2010 and
May 2022 with extended coverage by snowballing until July 2022.
Exclusion Criteria
E1: Papers that investigate IDS in a particular security application
infrastructure (e.g., Cloud (Modi et al., 2013; Patel et al., 2013), IoT
(Zarpelão et al., 2017) and Smart city (Elsaeidy et al., 2019)).
E2: Publications that are not written in English and are not accessible.
E3: Unpublished papers that are uploaded to the archive, workshop papers and
the conference version of a journal paper.
Table 5: Quality assessment of the papers ID |  Quality Assessment Criteria
---|---
AQ1 |  Does the paper clearly state objectives or challenges it targets to solve in HIDS literature?
AQ2 |  Is the proposed method well-defined and discussed in detail?
AQ3 |  Is the performance of the proposed method measured and reported within the paper?
AQ4 |  Is the proposed method compared with other existing approaches?
AQ5 |  Does the paper provide information regarding the datasets used for the assessment of the proposed approach?
AQ6 |  Does the paper provide details of the varied evaluation metrics used for evaluation?

###  3.2 Inclusion and Exclusion Criteria and Quality Assessment Criteria

Table 4 shows the inclusion and exclusion criteria in line with our SLR aim
and RQs. These are used for selecting the relevant papers out of the ones
retrieved from data sources. We developed a quality assessment criteria for
excluding low quality papers from our review. We adopted and adjusted our
quality assessment criteria from (Shahin et al., 2014; Garousi et al., 2019).
Table 5 presents the Assessment Questions (AQs). We graded the reviewed papers
on each quality assessment criterion using a three tier (“Yes”=1,
“Partially”=0.5 or “No”=0) scale. The assessment score of a paper is
calculated by adding the scores of the answers to the six AQs. To assure the
reliability of our review’s findings, we only included papers of acceptable
quality, that is, those with an assessment score of more than 3.00 (50% of the
perfect score).

Figure 1: Primary studies selection process and distribution of selected
studies.

###  3.3 Selection of Primary Studies

Figure 1 (a) shows the number of studies retrieved in different phases of the
papers selection process. We performed Database Search, Duplication Removal,
Title-Abstract-based Selection, Full-text-based Selection and Quality
Assessment against the inclusion and exclusion criteria (Table 4) and quality
assessment criteria (Table 5). Further, we used forward and backward
snowballing following the widely adopted snowballing guideline (Wohlin, 2014)
by scanning the citations and references of the selected papers, respectively.
Snowballing is used as the search string may not retrieve obscurely phrased
studies, and the selected digital libraries may not exhaustively include all
peer-reviewed papers (Wohlin, 2014). For snowballing, we followed the same
selection process including title and abstract, full text-based selection and
quality assessment. In total 65 papers were selected for our SLR as enlisted
in Appendix 10.1, each with a unique identifier (S#).

###  3.4 Data Extraction and Synthesis

We organized a Data Extraction Form (DEF) based on 17 data items to answer our
RQs as given in our online appendix (Zarrin Tasnim Sworna and Babar, 2022).
Data items (D1-D7) include context data such as title, author, venue,
publication year, publisher, summary and open challenges, respectively. To
facilitate the analysis of the extracted RQ relevant data, we grouped the data
items as follows. RQ1 (D8-D12: NLP method, feature extraction method, learning
type, classifier and HIDS type), RQ2.1 (D13-D14: attack
detection/classification (i.e., binary or multi-class classification), attack
instances), RQ2.2 (D15-D16: dataset, dataset availability) and RQ2.3 (D17:
evaluation metric). A pilot study of 15 papers was conducted collaboratively
by the authors to update the DEF for capturing the needed information in the
best possible summarised form. The designed DEF was completed collaboratively
using an online Google Spreadsheet and any ambiguity was resolved through
discussion.

Data Synthesis: We analyzed the context data items (i.e., D1-D5) using
descriptive statistics as shown in Section 4.1. We analyzed the RQ relevant
data items using thematic analysis considering the guideline of study (Braun
and Clarke, 2006) including the following steps. Familiarizing with data by
reading and examining our extracted data. Generating initial codes to capture
NLP methods, attacks, datasets and evaluation metrics for NLP-based HIDS.
Searching for themes and generating potential themes for each data item by
merging the corresponding initial codes based on their similarities. Reviewing
themes and mapping themes were performed iteratively to review all the codes
and themes to revise their allocations if needed. To finalize the RQs’
answers, the synthesized results for each RQ were reviewed and any
disagreements were discussed by the authors in daily Slack channel discussions
and weekly meetings.

Figure 2: The main modules and a general overview of end-to-end pipeline of
NLP-based HIDS Figure 3: Our taxonomy of the literature on NLP-based HIDS with
the relevant study counts

##  4 Results Overview

We present the distribution of the selected studies in Section 4.1 and our
taxonomy of the literature on NLP-based HIDS in Section 4.2.

###  4.1 Studies Distribution

Demographic information of papers (e.g., types) is considered helpful for
novice researchers (Shahin et al., 2017). Figure 1 (b) presents the
consistently upward trend of papers number in NLP-based HIDS in this decade
due to the ever growing threat landscape. More than half of the total papers
(42/65) and 81.82% of the journal papers (27/33) were published between 2018
to 2022 indicating the rapidly growing research literature and maturity in
adopting NLP in HIDS.

The reviewed papers were primarily published in the following research areas:
Cyber Security, Network Communications, Data Science and AI and also Software
Engineering. The diversity of research areas in terms of the publication shows
the interest of researchers with different research backgrounds in HIDS
research.

###  4.2 Our Taxonomy of the Literature on NLP-based HIDS

Figure 2 shows the main modules and a general overview of an end-to-end
pipeline of NLP-based HIDS. A HIDS typically monitors the events occurring in
an organization’s host infrastructure. First, the data collection module
monitors and collects data (e.g., syscall traces), which capture valuable
information about the running applications in a host system. Researchers
consider syscall traces as documents, syscall sequences as sentences in the
document and syscalls as words, which enables researchers to adopt NLP methods
for HIDS in the intrusion detection module. In the intrusion detection module,
the input syscall sequence data are processed and discriminative features are
extracted using NLP methods from the data, which represent the normal or
attack behavior of the input data. The automated NLP-based feature extraction
methods help to generate reliable features, which accurately represent the
behavior of a host system’s activities. The generated features are usually
applied to diverse classifiers (e.g., DL-based prediction models). These
classifiers perform the attack detection. The evaluation module evaluates the
attack detection performance of the intrusion detection module. Lastly, the
attack alert generation module generates attack alerts based on the attack
detection outcomes and shares them with the security experts of SOC in an
organization.

Our SLR aims to cover the end-to-end pipeline of NLP-based HIDS and thoroughly
discusses each of its modules. We propose a taxonomy of NLP-based HIDS by
analyzing and synthesizing the extracted data from the reviewed papers. We
provide a graphical representation of the NLP-based HIDS taxonomy in Figure 3.

Table 6: Number of papers in NLP method categories used in HIDS which are published in each year from 2011 to 2022  Category | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | Total
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Word frequency-based | - | - | 2 | 2 | - | - | - | - | - | - | 1 | - | 5
N-gram-based | 1 | 3 | 1 | 2 | 4 | 1 | 4 | 3 | 2 | 3 | - | - | 24
Word embedding-based | - | - | - | - | - | - | - | 1 | 1 | 3 | 3 | - | 8
Neural language modeling | - | - | - | - | - | - | - | 2 | 3 | 1 | 2 | - | 8
Semantic ontology | - | 1 | - | - | - | - | - | 2 | - | - | - | - | 3
Hybrid | 1 | - | - | - | - | - | 1 | 3 | 4 | 4 | 2 | 2 | 17
Total | 2 | 4 | 3 | 4 | 4 | 1 | 5 | 11 | 10 | 11 | 8 | 2 | 65

In RQ1, we identify and categorize the NLP methods (e.g., word embedding,
semantic ontology) used in HIDS. We also focus on and categorize each key
factors that are considered in the intrusion detection module. The key factors
that are adopted to investigate the host data for intrusions include HIDS type
(e.g., misuse, anomaly), learning types (e.g., supervised, unsupervised),
feature extraction techniques (e.g., manual, automated) and classifiers (e.g.,
ML, DL). RQ1 including the categorizations of NLP methods and all the key
factors is discussed in detail in Section 5.

In RQ2, we analyze how the NLP-based HIDS are evaluated in terms of detected
attacks (in RQ2.1); used datasets (in RQ2.2) and used evaluation metrics (in
RQ2.3). In RQ2.1, we categorize the attacks (e.g., Brute Force, Backdoor) that
can be detected by leveraging NLP-based HIDS for which alerts are generated in
the attack alert generation module. We also focus on the impacts (e.g.,
confidentiality, integrity) of these attacks on the security requirements. The
attacks and impacts are discussed in detail in Section 6.1. In RQ2.2, we
identify and analyze the datasets that are used in the data collection module
for applying NLP methods to develop and evaluate HIDS. We also focus on the
data types (i.e., real, simulated) and their availability (i.e., private,
public). Section 6.2 details the datasets that are used in the reviewed
studies to evaluate NLP-based HIDS. Lastly, in RQ2.3, we identify and
categorize the metrics (e.g., FAR, Detection Rates (DR)) used for the
evaluation of HIDS in the reviewed literature. Detailed comparative analysis
of the evaluation metrics is presented in Section 6.3.

We reflect on the prevalent practices based on findings to our RQs as shown in
the taxonomy of Figure 3. To help researchers and developers to identify and
select their choices, a comprehensive comparative analysis in terms of
strengths and weaknesses for the categories of this proposed taxonomy are
highlighted in Tables 7, 8, 10, 11 and 12. For conciseness, we added the study
mapping with the corresponding categorization of the RQs in our online
appendix (Zarrin Tasnim Sworna and Babar, 2022). For the frequently used
acronyms in this article, Table 13 in Appendix 10.2 presents their
abbreviations.

##  5 RQ1: Use of NLP Methods in HIDS

Firstly, we categorize and discuss different NLP methods that researchers use
in HIDS, and how the use of these NLP methods in HIDS trend is evolving over
the years in Section 5.1. To highlight the effectiveness of the NLP methods
for HIDS, we only cover the NLP methods proposed by the reviewed studies and
disregard the baseline approaches (i.e., methods used to compare the
effectiveness of the proposed method). For the development of HIDS using
different NLP methods, HIDS developers need to consider a number of other
factors (e.g., learning type and classifier). Hence, these key factors that
are considered for developing HIDS using NLP are summarized in Section 5.2.

###  5.1 Categorization of NLP Methods Used in HIDS

We present a comprehensive and novel categorization of NLP methods used in
HIDS. Table 6 demonstrates the distribution of the identified categories of
NLP methods used in HIDS over the years. Table 6 suggests that the prevalent
NLP methods used in HIDS over the last 4 years are word embedding, language
modeling and hybrid (i.e., combine multiple NLP methods). The increasing
amount of large scale security data and evolving complex attack patterns have
contributed to this trend of using these NLP methods that are evolved over
deep learning models due to their prominent HIDS performance. Table 7 shows
the categories along with their advantages, disadvantages and mapped studies.
The categories of used NLP methods in HIDS are discussed below.

Table 7: Categories of NLP methods in HIDS along with their strengths, weaknesses and mapped studies NLP Method |  Strengths |  Weaknesses |  Study Ref
---|---|---|---
Word freq-based |  $\bullet$ Less computational and memory overhead
$\bullet$ Lightweight and easy to dynamically update normal host system behavior profile |  $\bullet$ Do not consider the order of syscalls
$\bullet$ Difficult to gain a good detection rate and FAR compared to n-gram |  S28, S29, S31, S32, S55
N-gram-based |  $\bullet$ Holds order of syscalls to some extent
$\bullet$ Enables early attack detection by splitting the process trace of a
host system into smaller sequences
$\bullet$ Achieves good detection and FAR than word freq-based method
$\bullet$ Reduces the possibility of mimicry attacks on HIDS [S59] |  $\bullet$ Incur high computational and memory overheads
$\bullet$ Requires longer training time compared to Word freq-based method |  S2, S11, S16, S17, S19, S23, S24, S25, S27, S30, S34, S38, S39, S41, S42, S49, S52, S53, S56, S58, S59, S60, S61, S65
Word embedding-based |  $\bullet$ Word embedding learns hidden semantic relations of syscalls.
$\bullet$ Word embedding gives dense vector representation compared to One-hot
encoding that requires sparse and high dimensional vectors.
$\bullet$ Word embedding has better generalization ability than n-gram-based
methods [S57].
$\bullet$ Can help in domain adaptation through transfer learning to mitigate the requirement of huge domain-specific data [S12] |  $\bullet$ Word embedding requires complex DL-based training
$\bullet$ Requires more computation cost compared to n-gram or TF-based methods |  S4, S12, S14, S20, S46, S47, S51, S57
Semantic ontology |  $\bullet$ Enables attack knowledge fusion from heterogenous textual data sources (e.g., security bulletin) through a common semantic schema
$\bullet$ Link and infer means and consequences of cyber threats and
vulnerabilities whose signatures are not yet available
$\bullet$ Provide HIDS semantic expressiveness and knowledge description |  $\bullet$ Expert knowledge is required to identify the data sources and define the detection rules.
$\bullet$ Manual creation and update of detection rules are required. |  S13, S26, S33
Neural lang model |  $\bullet$ Predicts a future syscall sequence possibly to be executed during an attack.
$\bullet$ Combining the known invoked syscall traces with predicted future
syscall sequences helps to improve the intrusion detection performance
$\bullet$ Modeling syscalls helps to capture interword relationships. |  $\bullet$ Modeling the system behavior requires a huge amount of data |  S5, S15, S21, S22, S43, S44, S50, S63,
Hybrid |  $\bullet$ Makes HIDS more reliable and resilient against evasion and adversarial attacks by combining decisions from heterogeneous detectors
$\bullet$ Gains the advantages of multiple features or models to lower FAR |  $\bullet$ Requires high computation overhead compared to single methods. |  S1, S3,S6, S7, S8, S9, S10, S18, S35, S36, S37, S40, S45, S48, S54, S62, S64

####  5.1.1 Word frequency-based

In NLP, the word frequency-based method considers how frequently a word (i.e.,
one syscall) occurs within a document to indicate the significance of a
specific word within the overall document, which is a lightweight method. As
the normal behavior of the host varies over time, dynamic update of normal
behavior is critically challenging [S31]. Hence, a lightweight method like the
word frequency-based method is required to dynamically update normal behavior
and extract the features in HIDS with less computation cost. Researchers
investigated the use of word frequencies considering number of syscalls (n)=1.
Word frequencies are adopted by using three commonly used approaches in NLP,
which are Term Frequency, Inverse Document Frequency and Bag-of-Words. Though
these word frequency-based methods are computationally lightweight, they are
less detailed as they do not hold sequence information, which makes it
challenging to gain good detection rate and FAR [S9, S31].

Term Frequency (TF): TF is the calculation of how frequently a term occurs
within a document. As each syscall is presented by a unique word,
consequently, the syscall sequences become sentences and syscall traces
holding the syscall sequence denote documents. To enforce stronger
separability between normal and abnormal syscalls, researchers used TF in One-
Class Support Vector Machine (OC-SVM) [S29] and Clustering with Markov Network
(CMN) [S32] classifiers. To reduce the dimension of TF vectors into a lower
dimensional space, researchers successfully used principal component analysis
(PCA) [S28] that reduced the computational cost.

Inverse Document Frequency (IDF): In NLP, IDF calculates if a word is common
or rare in a given document corpus. IDF is calculated by the logarithm of the
quotient that is obtained by the division of total document count and count of
documents that contain the word. A study [S31] focused on the frequencies of
the syscalls involved in a trace to generate three feature vectors (i.e., TF,
IDF, standard deviation of IDS) and applied them to K-Nearest-Neighbours
(KNN). Though the diverse syscall frequency-based classifier gained an
acceptable performance for a few types of attack, it can not completely
realize the complex behavior of a modern host system, consequently, require
more intelligent HIDS.

Bag-of-Words (BoW): Another commonly used word frequency-based technique is
BoW, which represents text by describing words’ occurrence within a document.
BoW discards words’ order or structure related information in a document. BoW
represents if words are present in a document, not the words’ location in the
document, and does not focus on the contextual relation of words in the
document. To better identify multiple attack types, BoW is used for
characterizing the syscall-based HIDS data to be used in Deep Belief Networks
(DBN)-Softmax DL method for achieving high accuracy [S55].

####  5.1.2 N-gram-based

N-gram is one of the most used traditional NLP methods for representing
sentences that holds the context information [S5]. While sentences are
sequence of words, syscall sequence is a sequence of syscalls. Due to the
similarity of the representation of sentences and syscall sequence, n-gram is
highly used in HIDS, where n-gram refers to a sequence of n (n$>$1) syscalls
extracted from a syscall trace [S41]. N-gram method extracts sequence of
syscall of length ‘n’ from a syscall trace by sliding a window one syscall at
once. For instance, a syscall trace with syscalls “open, getrlimit, mmap,
close” [S39], two sequences “open, getrlimit, mmap” and “getrlimit, mmap,
close” of length 3 (n=3) can be extracted. As n-gram holds the order of
syscalls, n-gram has the ability to preserve sequence information of the
syscall to provide good detection accuracy and it reduces the possibility of
mimicry attacks on the HIDS [S59]. Besides, a HIDS should intervene early to
stop the exploited process instead of waiting for the exploit to complete. To
enable early attack detection, n-gram is used to split the syscall trace of a
process into smaller sequences which can be analyzed instead of waiting for
the process to complete. Due to these advantages, n-gram, frequency of
n-grams, bag-of-n-grams and lookahead pairs methods from this n-gram category
are used to develop HIDS.

N-gram: A set of studies [S16, S23, S24, S52] have used n-gram to extract
sequence from benign syscall traces and compare them with sequences of a HIDS
input trace. Then, any new sequence in the HIDS input trace is detected as
anomalous. These studies adopted different similarity measures commonly used
in NLP to measure similarity among the sequences such as Hamming distance
[S23, S24], Sequence Covering for Intrusion Detection (SC4ID) [S16],
Levenshtein’s distance (LEV) [S16], Longest Common Subsequence (LCSq) [S16],
Longest Common Substring (LCSt) [S16] and improved edit distance [S52]. These
n-gram-based methods, which use similarity measures are highly dependent on
the similarity threshold for anomaly detection. Identifying the suitable
threshold requires domain expertise and can be error-prone. To mitigate
dependence on threshold values and to handle large scale syscall-based raw
input data, in recent studies, n-gram is used in advanced DL models (e.g.,
Long Short Term Memory (LSTM) [S11]), Convolutional Neural Network (CNN) [S17]
and Variational Auto Encoder (VAE) [S38]). For example, a study [S11] used
n-gram (i.e., as syscall sequences are of variable length) to detect known and
unknown attacks. This study [S11] modeled the normal behavior of the host
system by syscall using LSTM architecture to detect zero-day (Zoppi and
Ceccarelli, 2021) unknown attacks. Then, to predict the most probable ‘n’
syscalls during the attack detected by the LSTM, the Multiple Hidden Markov
Model (Multi-HMM) is adopted. Here each Hidden Markov Model (HMM) is used to
model a distinct known attack for predicting the probable ‘n’ syscalls during
the corresponding attack.

N-gram frequency: Although n-gram preserves the syscall traces’ sequential
information, the representation vectors are long. Hence, with an increasing
value of n, the detection model requires more storage space and processing
time [S2]. To mitigate this issue, a few studies [S19, S34, S65] used only the
most frequent n-grams to reduce the feature space at the expense of the loss
of some relevant information, which is a common NLP method. Though extracting
syscall frequency is fast, syscall count in the trace is not sufficiently
informative for detecting anomalies as syscalls’ count can be similar in
normal and abnormal syscall traces. To mitigate this issue, the n-gram
frequency (i.e., phrase count) of NLP is used by a set of studies [S2, S25,
S30, S41, S53, S59, S60] for obtaining a better detection rate with faster
processing. In this method, distinct syscall sequences of varying lengths are
counted. For example, to adopt n-gram frequency, a study [S41] extracted
n-grams of syscalls by sliding a window of ‘n’ syscalls over the syscall
trace, associated a feature vector for each n-gram, and calculated each
n-gram’s occurrence frequency for diverse traces. These varied-length n-grams
are represented as fixed-size vectors and their cumulative occurrence
frequency is assigned as weight. As this method holds temporal order of
syscall in a trace, the application of OC-SVM trained on n-gram frequency
showed better detection accuracy than the HIDS methods using TF and TF-IDF.

Bag-of-n-grams: The ability to preserve the relative order of syscalls makes
the bag of n-grams a better alternative to the Bag-of-Word (BoW) NLP method in
the context of syscall-based HIDS. BoW only considers the occurrence frequency
of a word in a document to determine how much the document is relevant to
specific words. In the context of syscall data, considering syscall as a word
and syscall trace as a document, the BoW model can not preserve the relative
order of syscalls. For example, the feature vector using BoW of syscall traces
“S1: open, getrlimit, mmap, close” and “S2: open, mmap, getrlimit, close” are
similar. However, to model process behavior, the relative order of syscalls is
highly significant. Loss of information of syscall sequence may leave a host
vulnerable to the mimicry attack [S42], where an attacker interleaves malware
syscall trace patterns with benign syscall traces. Thus, bag-of-n-grams
consider multiple consecutive syscalls as one term, which helps hold the
sequence information and prevent mimicry attacks. A study [S42] showed that
bag-of-n-grams help more accurately distinguish process behavior through
syscalls for intrusion detection in HIDS.

Lookahead pairs: Another variation of the n-gram NLP method is lookahead
pairs. In the lookahead pairs method, for a given syscall sequence and a
window of size (n), this window is slid over the syscall sequence. Then,
except for the first syscall of the window, each syscall of the window is
paired with the first syscall of that window. For example, for a syscall
sequence “S1: open, read, close, exit”. For n=3, starting at the first syscall
“open” the window now has 3 elements “open, read, close”. Next, the pairing
process pairs as follows: “open, read”, “open, close”. Then, sliding the
window next syscall (i.e., read) and the pairing process pairs as follows
“read, close” and “read, exit”. Hence, the lookahead pair method presents a
model of constant space complexity for syscall-based HIDS. A study [S56]
showed that lookahead pairs modeled with their occurrence frequency are good
discriminators of benign and malicious program executions and is adversarial
attack tolerant in terms of some contamination in the training data.

####  5.1.3 Word embedding-based

Word embedding is a sequential representation method to capture semantic
relation and contextual information from syscall with reduced size vector.
Word embedding has better generalization capability than n-gram-based methods
[S57]. Word embedding significantly enhances the NLP classification task’s
performance due to capturing semantic features that motivates HIDS researchers
to use word embedding to deal with syscall sequence [S57]. While One-Hot
Encoding (OHE) in NLP is the simplest method to represent categorical/text
data for a classifier [S9]. In HIDS, to use OHE, each syscall (word) is
presented as a vector, where each position in the vector represents a specific
syscall. Hence, input vector size is the number of diverse syscalls. A
specific syscall is mapped to a binary vector with all zeros except for a
single one at the suitable position for the corresponding syscall. OHE does
not require specific prior knowledge and complex feature selection [S8].
Additionally, it preserves the original information of the data intact [S8].
However, using long OHE representation for syscalls can lead to training
neural networks with a huge number of weights [S9]. In NLP, to mitigate the
computational complexity and high training time due to sparse and high-
dimensional vectors of OHE, word embedding has proven its value. Hence, it is
preferred to use shorter vectors generated by word embedding to represent
syscalls. To detect intrusion in HIDS, researchers use diverse word embedding
methods from NLP such as Word2Vec [S14, S47, S51], GloVe [S47, S51], FastText
[S51] and Graph Random State Embedding (GRSE) [S20] to represent the syscalls.

Word2Vec (Mikolov et al., 2013) in NLP generates word vectors based on the
context in which they are used. Word2Vec depends on words’ local context, in
contrast, GloVe (Pennington et al., 2014) uses global statistics on word co-
occurrence for learning words’ vector representations. In NLP, GloVe is a
count-based model that learns a word vector representation using co-occurrence
information, i.e., for each word, GloVe count how frequently those words are
used in some context in a large corpus. FastText (Bojanowski et al., 2017)
developed by Facebook AI research extends Word2Vec by considering sub-word
information, whereas Word2Vec and GloVe do not consider sub-word information.
As the syscall names are chosen based on their functionality, sub-words can
provide supplementary information about relationships among syscalls [S51].
For instance, directory is represented as ‘dir’ in short. Syscalls including
this sub-word (e.g., cddir, mkdir, rmdir, readdir, fchdir, mkdirat) usually
represent changes in the host’s file system kernel module [S51], which is
important for profiling host behavior. Moreover, such sub-words include
supplementary information about their context (e.g., ccdir and mkdir represent
that they handle a directory).

Syscall sequences are essentially considered as word sequences (i.e.,
sentences). However, syscall sequence has several distinct features compared
to sentence. For instance, syscalls have an implicit pattern relation, and the
changing order among patterns has less effect [S20]. Further, syscall
occurrence count often has little effect on the detection outcome. Syscalls
also have other characteristics (e.g., resource utilization, execution time,
predefined rules and empirical weights of syscalls). The NLP methods that are
usually used in HIDS (e.g., BoW, TF-IDF, n-gram and Word2Vec), do not
completely exploit such relations in syscall sequence and can not simply
support such properties [S20]. To mitigate these limitations, a sequence
embedding approach, namely, Graph Random State Embedding (GRSE) is proposed
using Word2Vec [S20]. To generate vectorized representation of graph nodes
GRSE performs random walk over the graph network for generating graph node
paths, which emulates the text generation process (for HIDS, i.e., the process
executing a syscall) for obtaining various graph node access sequences.
Further, GRSE utilizes Word2Vec for learning all the nodes’ vector
representations based on the random walk sequence, which improves a HIDS’s
performance.

A major challenge to develop HIDS with better performance for a host domain
with low resource is to mitigate the requirement of huge domain-specific data.
To mitigate this challenge, word embedding helps in domain adaptation
following NLP approach through transfer learning. A study [S12] showed the
applicability of word embedding-based transfer learning by adopting a word
embedding-based LSTM model that uses a significantly lower amount of target
HIDS domain data to detect attacks in multiple host domains. This study [S12]
further discussed both homogeneous and heterogeneous transfer learning-based
HIDS problems, where monolingual word embedding is required for the
homogeneous HIDS domain and bilingual word embedding is needed for the
heterogeneous HIDS domain.

####  5.1.4 Semantic ontology

To develop a HIDS with the capability of potentially linking and inferring
means and consequences of cyber threats and vulnerabilities whose signatures
are not yet available, knowledge fusion from heterogeneous textual data
sources is required [S33]. For example, cyber threat intelligence data are
available at various textual sources (e.g., blogs and security bulletins)
[S26]. However, to gain the advantages of heterogeneous data sources to detect
any threats, a common semantic schema is required to integrate information
from disparate sources (e.g., different concepts and standards such as STIX,
CVE, CVSS, CAPEC and CYBOX from diverse intelligence sources) [S33]. To
mitigate this issue, a semantic ontology in NLP can provide a common semantic
schema for integrating information from diverse sources by describing concepts
and relationships among the concepts. To improve HIDS performance, semantic
ontologies are used based on a semantic-reasoner along with a rule engine for
detecting intrusion [S13]. Semantic ontology not only easily integrates
information from textual sources but also helps to detect existing attacks’
variations. For evading attacks, attackers often use diverse tools which can
perform similar activities or use combinations of tools/techniques used in
older attacks. Further, attackers often use similar tools and exploit similar
vulnerabilities in varied attacks. As semantic ontology can integrate
information of diverse attacks and fuses it with textual information, semantic
ontology-based HIDS can detect such new attacks [S26].

The first ontology-based HIDS [S13] aimed at providing semantic expressiveness
and knowledge description for improving IDS performance and reducing the
search time for malware scanning. This study [S13] developed HIDS using
Semantic Web technologies that give well-defined meaning for enabling
computers and human to work in cooperation. Later, a study [S26] proposed
continuous integration of information from dynamic textual sources (e.g.,
threat intelligence information of attack patterns, prior attacks, tools used
for attacking and indicators) and combined it with malware behavioral
information for detecting known and unknown attacks. This study [S26] used an
off-the-shelf Named-Entity Recognizer (NER) from NLP trained on cyber security
text to extract entities from plain text. Similarly, another study [S33]
extracted named entities from the unstructured web-text for the continuous
evolution of ontology. This study [S33] integrated text data from the web with
different sensor streams, domain expert knowledge and constructed the ontology
based on three main classes: ‘means’, ‘consequences’ and ‘targets’. The
‘means’ class captures the approaches used to execute an attack, the
‘consequences’ class captures the results of the attack and the ‘target’ class
captures the details of the host under attack. Finally, this study [S33] has
demonstrated the effectiveness of the semantic integration of web text for
detecting threats in HIDS.

####  5.1.5 Neural language modeling

To capture inter word relationships and predict the next words in NLP, a
neural language model performs probability distribution calculation over
words’ sequences. Neural language modeling has achieved significant
performance in realizing real-world NLP tasks [S15], such as Google’s
autocompletion and voice assistant. Inspired by such significant performance,
HIDS researchers discovered surprising resemblance between the host’s syscall
sequence and natural language [S44, S63]. Researchers consider syscall
sequences as instances of the language for communication between users (or
programs) and host, where syscall and syscall sequences refer to words and
sentences in natural languages [S63]. Due to this high resemblance of syscall
sequence with natural language, researchers adopt neural language modeling in
HIDS to model syscall.

To monitor host system state and predict attack behavior, a language model-
based prediction module predicts syscall sequence which will be executed in
the future (i.e., possibly to execute during a known or unknown (zero-day)
attack) [S44, S63]. The use of neural language modeling enables HIDS to model
the semantic meaning of syscalls and analyze sequences at the sentence level
for building a robust syscall Sequence-to-Sequence (Seq2Seq) prediction model.
The Seq2Seq prediction model is adopted from NLP Question-Answer model for
generating future syscalls considering prior invoked syscall sequence as
question and generated syscall sequence as answer [S63]. Further, combining
the prior invoked syscall traces with the predicted syscall sequence
drastically enhances the intrusion detection performance, which is verified
using different classifiers in HIDS [S15, S44]. As a Recurrent Neural Network
(RNN) has the inherent capability to process sequential data and can memorize
previous results, diverse variations of RNN (e.g., LSTM and Gated Recurrent
Units (GRU)), RNN Variational Encoder-Decoder (VED) and combination of CNN
with RNN-based language models (e.g., CNN-GRU, LSTM-textCNN) are adopted to
handle sequential syscall data in the HIDS research domain.

RNN-VED-based language model: Seq2Seq language model based on variational
encoder-decoder (VED) and variants of RNN showed promising performance in NLP.
Researchers [S5, S43] exploited the semantics behind the invoking order of
syscalls that are considered sentences. The RNN-VED in HIDS [S5, S43] learns
the correlation between the syscalls to predict the future possible syscall
sequence based on the given context. Another study [S63] used RNNs and
Variational Autoencoding language modeling for learning long-term syscall
sequences executed during an attack. This approach enables effective
prediction and classification of syscall sequences that are possibly to occur
next during known or unknown (zero-day) attacks.

LSTM-based language model: Evolutionary intrusion attacks (e.g., obfuscation
technique) can change the malicious syscall sequence so that it can bypass the
intrusion detection; and still can gain the same invasion purpose and effect
that makes the detection outcome not robust and even invalid. To detect
obfuscation attacks, a study [S22] used a behavioral semantics enhancement
method by using an LSTM Seq2Seq language model that performs sequence
completion. Then, the enhanced sequences are represented as vector matrices
which are the input for the multi-channel Text-CNN that makes the model robust
to obfuscation attacks. This language modeling-based method showed better
intrusion detection performance and robustness than other NLP methods such as
syscall combinations, n-gram subsequences and feature matching frequencies.
LSTM model in NLP requires a long training time and has a complex structure.
With the motivation of reducing training time, a study [S50] used NVIDIA CUDA
Deep Neural Network (CuDNN) library to adopt CuDNNLSTM language model which is
supported only on GPU systems. CuDNNLSTM language model using a bidirectional
encoder and a unidirectional decoder is approximately ten times faster than
LSTM model for faster convergence during training for HIDS development [S50].

GRU-based language model: The structure of GRU is simpler and GRU can reduce
the required training parameters compared to LSTM. GRU [S44] and CNN-GRU [S15]
language models are used in the literature for reducing the required training
time and improving the efficiency of HIDS. In the CNN-GRU language model
[S15], CNN layers are able to capture syscalls’ local correlations in the
sequence and can improve efficiency by executing in parallel. Then, the GRU
layer is able to learn syscall’s sequential correlations from these high-level
features. The CNN-GRU language model gained near state-of-the-art performance
and substantially reduced training time compared to LSTM models. Traditional
HIDS models consider syscall traces generated by an individual process.
However, multiple processes are often utilized by modern applications and one
or more of these processes are impacted by modern attackers. Hence, to
modernize HIDS a Wave-Net architecture is used to aggregate predictions for
all processes corresponding to an application [S21]. This approach [S21]
outperformed the CNN-GRU language model [S15].

####  5.1.6 Hybrid

To gain the advantage of multiple NLP methods, researchers have adopted hybrid
methods by combining different NLP methods. The hybrid method includes the
combination of heterogeneous classifiers using different NLP methods, a
combination of n-gram and TF-IDF, a combination of n-gram and statistical
approaches and a combination of n-gram and data augmentation methods.

Combination of heterogeneous classifiers using different NLP methods: To
reduce the FAR of HIDS and make the HIDS more reliable and resilient against
evasion and adversarial attacks, a set of studies effectively combined
decisions from heterogeneous classifiers by using different NLP methods. A few
studies [S35, S64] have trained Sequence Time-Delay Embedding (STIDE) and HMM
utilizing n-gram-based sequential features, and OCSVM utilizing TF-IDF vectors
and effectively combined decisions from these heterogeneous classifiers.
Another study [S66] used a deep Multi-Layer Perceptron (MLP)-based Neural
Network (NN) to combine the detection outputs of STIDE (n-gram-based), text
classifier (TF-based) and syscall graph-based detection. The combination of
heterogeneous classifiers with heterogeneous NLP methods showed consistently
better performance than a single detector and ensemble of homogeneous
detectors [S35].

Combination of n-gram and TF-IDF: TF-IDF denotes the multiplication of TF and
IDF scores of the word that is used to weigh down the frequent words and scale
up the rare ones. A set of studies [S3, S6, S7] used TF-IDF vectorized n-gram
vectors to include sequence information to some extent (via n-gram) for
improving detection performance with lightweight TF-IDF-based feature
calculation for computation cost efficiency. Being inspired by sentiment
analysis in NLP, where combining 2-gram, 3-gram and 4-gram models improved
accuracy, a study [S3] combined 2-gram, 3-gram and 4-gram TF-IDF from selected
syscalls of a trace file. Based on these combined features, for early
detection of intrusions, researchers built a HIDS which processes the first
few hundred syscalls to detect intrusion for an application. Since software
applications’ complexity has increased drastically, such applications invoke a
large number of syscalls in a short time, which demands early detection that
can consequently reduce HIDS resource consumption. Further, truncated Singular
Value Decomposition (SVD) based on the TF-IDF values is used for reducing the
transformed n-gram feature vectors’ dimensionality, which gained high
detection performance with low processing overhead [S6].

Combination of n-gram and statistical approaches: Diverse statistical
approaches to the n-gram-based NLP method have been adopted in HIDS research
for OS independent feature generation. Such OS independent features have
several advantages such as (i) model transferring: HIDS model trained on data
from OS X can be utilized on OS Y as the data share the same feature subspace
and (ii) data transferring: data from OS X can be selectively combined with
the data from OS Y to enrich the training data [S1]. Statistical methods used
in HIDS are standard deviation [S1, S40], mean [S1], skewness [S1], kurtosis
[S1], standard error [S1], eigenvector [S37] and entropy [S1, S36]. For
example, a study [S1] showed that statistical methods (e.g., standard
deviation and skewness) on n-gram frequency outperformed the following NLP
methods: (i) bag-of-syscalls; (ii) TF-IDF (iii) n-gram frequency; and (iv)
n-grams based methods.

Combination of n-gram and data augmentation methods: HIDS datasets usually
suffer from data imbalance problem as gaining malicious syscall trace is
challenging and much less in number compared to benign trace. The data
imbalance problem creates a bias in the HIDS prediction model towards the
majority benign class. To mitigate the data imbalance problem, a set of
studies used different approaches on the n-gram NLP method such as Synthetic
Minority Over-sampling Technique (SMOTE) [S40] and data augmentation using
Generative Adversarial Nets (GAN) [S48], NLP-based Sequence Generative
Adversarial Nets (SeqGAN) [S45] and Sequence to Sequence (Seq2Seq) [S45]
models. A set of studies [S45, S48] showed that adding augmented data to train
HIDS using representation ML models can improve HIDS performance. Besides, a
study [S48] showed that GAN provides a more reliable way to mitigate data
imbalance problem using data augmentation than SMOTE over-sampling technique.

To summarize the NLP methods used in HIDS, word embedding, language modeling
and hybrid methods (i.e., combination of multiple NLP methods) are the
prevalent NLP methods used in HIDS. These NLP methods are also showing an
increasing trend over the last 4 years as demonstrated in Table 6. The
increasing amount of large-scale host security data and evolving complex
attack patterns have contributed to this increasing trend of using these NLP
methods that are evolved over deep learning models due to their prominent HIDS
performance. Table 7 presents our deep comparison analysis of the NLP methods
used in HIDS, which is expected to help the HIDS researchers and developers to
choose the suitable NLP methods based on their research goal and
organization’s requirement, respectively.

Table 8: Key factors considered in NLP-based HIDS development with their descriptions, strengths and weaknesses Type |  Description |  Strengths |  Weaknesses
---|---|---|---
HIDS Types
Anomaly (59) |  Detects deviations from normal patterns as anomalies |  $\bullet$ Able to detect zero-day attacks |  $\bullet$ High false alarm rate
Misuse (6) |  Detects system behaviors matching the signatures present in attack library as intrusions |  $\bullet$ Low false alarm rate |  $\bullet$ Unable to detect zero-day attacks
Learning Types
Semi-supervised (36) |  Trains only normal samples with no anomalies in a given training data set |  $\bullet$ Only the normal class’s labeled data is required. |  $\bullet$ Suffers from high false alarm rate
Supervised (27) |  Uses labeled data to train a learning model on normal and attack data. |  $\bullet$ Stable performance and effective way to detect known attacks. |  $\bullet$ Requires labeled training data that is costly and time-consuming to gather
$\bullet$ Difficult to detect unknown attacks
Unsupervised (5) |  Without any prior knowledge, utilize statistical models to detect anomalies. |  $\bullet$ Does not require labeled training data
$\bullet$ Lower computational complexity |  $\bullet$ No access to information about distinguished data patterns
Feature extraction techniques
Automated (55) |  Features are extracted using automated approaches such as NLP methods (e.g., n-gram, TF-IDF and word embedding). |  $\bullet$ More efficient and repeatable
$\bullet$ Less dependent on domain knowledge
$\bullet$ Adaptable to new datasets
$\bullet$ Suitable for complex data with structural and sequential dependencies |  $\bullet$ Computationally expensive
$\bullet$ Dependent on data size and quality
Manual (5) |  Features are extracted by manual data analysis and using domain knowledge. |  $\bullet$ Interpretable features
$\bullet$ Suitable for homogeneous data with linear patterns |  $\bullet$ Highly depends on domain knowledge
$\bullet$ Takes significant time and effort
$\bullet$ Error prone
$\bullet$ Not scalable and dataset-dependent
Semi-automated (5) |  Features are extracted using both manual and automated techniques |  $\bullet$ Gains advantages of both automated and manual feature extraction |  $\bullet$ Needs more time than automated
$\bullet$ Needs more computation than manual
Classifiers
ML: Single (38) |  A single ML model is used to perform the prediction (e.g., SVM, NB) |  $\bullet$ Easy implementation
$\bullet$ Re-training on large scale dataset is efficient |  $\bullet$ Susceptible to overfitting
$\bullet$ May not perform as good as ensemble or DL models
ML: Ensemble (7) |  Combine predictions from two or more models (e.g., RF, XGBoost). |  $\bullet$ Usually makes better predictions than single contributing model.
$\bullet$ Less susceptible to overfitting |  $\bullet$ Longer training time than single models
Rule or sequence matching (17) |  Rules are defined (e.g., semantic ontology) or sequence matching is performed to detect attacks (e.g., SC4ID [S16]) |  $\bullet$ Usually parameter-free (apart from a decision threshold)
$\bullet$ Interpretable as they enable to find the location of abnormal areas in long syscall sequence. |  $\bullet$ Defining the rules and the decision threshold is manual
$\bullet$ Requires domain expertise.
DL: RNN (12) |  RNN is a type of NN that helps to model sequential or time series data (e.g., LSTM, GRU, BiLSTM) |  $\bullet$ Hold short or long-term dependencies from sequence
$\bullet$ Better suited for sequence-based or textual HIDS data |  $\bullet$ Due to sequential processing, usually require longer training time than CNN
DL: CNN (12) |  CNN is a type of NN that captures local features using convolution (e.g., text-CNN, FCN, TCN) |  $\bullet$ Holds local and hierarchical features
$\bullet$ Can perform parallel processing during training, which is faster than sequential RNN |  $\bullet$ Unable to effectively capture sequential order of data that is important for handling sequential data

###  5.2 Key Factors to Consider for Developing HIDS Using NLP

In this section, we summarize the key factors that need to be considered by
the HIDS developers for developing HIDS using different NLP methods. Table 8
presents the key factors considered in NLP-based HIDS development along with
their descriptions, strengths and weaknesses. Our deep comparison analysis of
the categories of these factors is expected to help the HIDS developers to
choose the suitable category for the corresponding factors based on the
organization’s requirements. We describe the key factors such as i. HIDS type
(Section 5.2.1); ii. learning type (Section 5.2.2); iii. feature extraction
method (Section 5.2.3) and iv. classifiers (Section 5.2.4).

####  5.2.1 HIDS type

Two types of HIDS (i.e., misuse, anomaly) are developed using NLP methods in
our reviewed studies. Misuse (also called signature-based) detection uses a
library of known attacks’ signatures and identifies system behaviors matching
the signatures present in the library as intrusions. For misuse detection, NLP
methods such as semantic ontology, can be used for integrating attack
signatures and information from diverse sources (e.g., cyber threat
intelligence data and textual sources like security blogs and bulletins).
Misuse detection gains low FAR but is unable to detect zero-day attacks (Liao
et al., 2013). However, anomaly detection or behavior-based detection builds a
model based on the normal behavior of system activities and detects deviations
from normal patterns as anomalies, which can detect zero-day attacks but have
high FAR (Liao et al., 2013). With the continuously evolving threat landscape,
attack signatures are not always available. Hence, anomaly detection is a
preferred method to detect unknown zero-day attacks. NLP methods (e.g., word
embedding, language modeling) help to characterize the normal behavior of the
system by preserving the semantics and contextual information of the data,
which makes NLP methods highly suitable for developing anomaly-based HIDS.

####  5.2.2 Learning type

Different learning types (i.e., supervised, unsupervised, semi-supervised) are
adopted for attack detection in our reviewed studies. The choice of a learning
type highly depends on the availability of labeled data. The unsupervised
approach finds patterns/partitions from unlabeled data. The semi-supervised
approach requires a portion of data to be labeled. However, supervised
learning requires complete labeled data. In the HIDS domain, normal data is
highly available (generated by the normal execution of programs within a
system) but malicious data is insufficient (which requires simulating the
increasing attack types). Thus, semi-supervised (used by 55.4% reviewed
studies) anomaly detection by training with only normal samples and detecting
the deviations from the learned model as anomalous is prevalent in the
reviewed studies. NLP methods (e.g., n-gram, word embedding) help to adopt a
semi-supervised approach by learning the normal behavior of the system through
the semantics and context of syscall sequence. However, semi-supervised
learning lead to high FAR as they classify the unseen normal behavior as
attacks. Hence, diverse NLP-based data augmentation techniques (e.g., SeqGAN
[S45]) are being used to balance the HIDS dataset for adopting a supervised
approach that can lower the FAR.

####  5.2.3 Feature extraction techniques

The collected raw syscall data need to be represented by appropriate features
for training the HIDS detection models. Feature extraction techniques used in
the reviewed studies are manual, automated or semi-automated (e.g., both
manual and automated). Manual features are extracted by manual data analysis
and using domain knowledge. However, the use of NLP methods such as n-gram
(e.g. [S11, S17]), TF-IDF (e.g. [S55]) and word embedding (e.g. [S47, S51]) in
HIDS enabled automated feature extraction from HIDS dataset and made it
suitable for complex data with structural and sequential dependencies.
Automated feature extraction is a preferred choice as it reduces human effort
and can adapt to new datasets. Hence, in our reviewed studies automated
feature extraction method is prevalent (used by 84.6% reviewed studies) as it
does not depend on domain expertise and makes the HIDS scalable and portable.

####  5.2.4 Classifier

The extracted NLP-based features are used in three types of classifiers (i.e.,
traditional ML, rule-based and DL model) for detecting intrusions in HIDS.
While the earlier research in NLP-based HIDS used the traditional ML-based
classifiers (e.g., Support Vector Machines (SVM), Logistic Regression (LR)),
in the recent five years (i.e., 2018-2022), the trend is shifted to the
application of DL-based (e.g., text-CNN) and hybrid classifiers. The possible
reason behind the change in the trend can be the highly increasing amount of
data and evolving complex attacks [S22]. DL is now being investigated more as
it has the capability to handle large-scale data and the complex DL structure
tends to provide better accuracy and lower FAR.

Traditional ML models: For the traditional ML models, single models (e.g.,
SVM, Naïve Bayes (NB)) were prevalent in the earlier research time due to
their simplicity. However, single models do not perform as good as the
ensemble of DL models. Among the single models, SVM and its variants are the
most popular algorithms (e.g., SVM, OCSVM) (Mohammadi et al., 2021) used for
binary and one-class classification adopting supervised and semi-supervised
learning, respectively. The popularity of SVM is expected as SVM works well
with the commonly used NLP-based methods such as BoW and TF-IDF (Le et al.,
2021). For example, OCSVM with variable length n-gram features [S41]
outperformed the approaches using TF/TF-IDF features or using HMM, K-Nearest
Neighbour (KNN) and STIDE for attack detection. Another study [S59] proposed
n-gram-based and frequency of n-gram-based approaches, where the frequency of
n-gram-based OCSVM outperformed instance-based methods (e.g., KNN, K-Furthest
Neighbors (KFN)) of prior studies [S28, S31], and had lower FAR than KFN.
Besides SVM, other commonly used single ML classifiers used for HIDS are
instance-based (e.g., KNN and KFN) and NN-based (e.g., MLP and Extreme
Learning Machine (ELM)) single models. Moreover, ensemble ML-based classifiers
(e.g., Random Forest (RF), EXtreme Gradient Boosting (XGBoost)) are used as
they are less prone to overfitting and usually perform better than single ML-
based classifiers. For example, for real-time detection of applications which
invoke a large number of syscall, a study [S3] used RF. RF based on NLP-based
TF-IDF of syscall sequences outperformed various single classifiers such as
Decision Tree (DT), KNN, MLP, Multi-variable NB (MNB) and SVM [S3].

Rule or sequence matching: Another common approach adopted in NLP-based HIDS
is the rule (e.g., semantic ontology) or sequence matching (e.g., SC4ID [S16])
to classify normal and anomalous HIDS input syscall sequence. For example, a
study [S16] used a similarity approach SC4ID considering the minimal number of
subsequences required to build a complete covering of a given sequence, which
outperformed other text similarity measures (e.g., Levenshtein’s distance,
Longest Common Subsequence/Substring) in terms of accuracy and execution time.
The rule or sequence matching approaches are interpretable as they enable to
find the location of abnormal areas in long syscall sequences. However,
defining the rules and the decision threshold is manual and requires domain
expertise.

DL models: The adoption of NLP-based DL models gained tremendous popularity in
HIDS from 2018 and showed better results than traditional ML and rule or
sequence matching-based models. In the HIDS domain, the use of DL models not
only enabled anomaly detection (binary classification) and attack detection
(multi-class classification for specific attack detection) but also enabled
prediction of the future syscall during an attack. The prevalent practice in
NLP-based HIDS is to use variants of RNN (e.g., [S11, S12, S38, S43, S47, S51,
S62]) and CNN (e.g., [S8 , S9, S17, S22, S48, S57, S63]), while a few studies
used other DL models such as Deep Multi-layer Perceptron (Deep MLP) [S18,
S53], Autoencoder [S9, S38, S43, S62] and Deep Belief Network (DBN) [S55]. Due
to RNN’s (Lee et al., 2021) intrinsic capability of handling sequence data
that can capture short or long-term dependencies of the input sequence, a wide
variety of RNN (e.g., LSTM, GRU, BiLSTM, CuDNNLSTM) are frequently used in
syscall sequence-based HIDS. However, RNN works in a sequential manner and
takes longer training time than CNN as the CNN model can be trained in
parallel. Different CNN models (e.g., text-CNN, Fully Convolutional Network
(FCN), Temporal Convolutional Neural Network (TCN)) have been adopted in HIDS.
While CNN can capture the local and hierarchical features, it can not
effectively capture the sequential order of data which is required for syscall
sequence data handling. The latest trend is to ensemble CNN and RNN models to
gain the advantage of both models, where CNN extracts local syscall relation
features and RNN captures syscall sequences context by extracting long-
distance sequence dependency. For example, a study [S4] used 7 ensemble DL
classifiers such as dual-flow (LSTM-FCN, GRU-FCN), FCN and Windows OS data-
specific (i.e., AWSCTD dataset) models (CNN-Dynamic (CNN-D), CNN-LSTM, CNN-GRU
and CNN-Static (CNN-S)). However, the complex dual-flow models did not show
any advantage over the single-flow models (e.g., CNN-S) for data processing
and increased training and detection time, which hinders real-time detection
[S4].

To summarize the key factors considered while developing NLP-based HIDS, the
prevalent practice in terms of HIDS types is anomaly-based detection which
helps to detect unknown zero-day attacks. Besides, the dominant practice in
terms of learning types used for NLP-based HIDS is semi-supervised learning
approach which helps to address the lack of balanced available HIDS datasets.
In addition, the dominant practice in terms of feature extraction used for
NLP-based HIDS is automated feature extraction which helps to extract features
from complex HIDS data with structural and sequential dependencies. The
adoption of NLP-based DL models helped to achieve better results than
traditional ML and rule or sequence matching-based models for HIDS using NLP
methods, which gained tremendous popularity in HIDS since 2018 in terms of
classifiers used for NLP-based HIDS.

##  6 RQ2: Evaluation of NLP Method-based HIDS

To answer RQ2, we analyze the evaluation of NLP method-based HIDS in terms of

$\bullet$ RQ2.1 Attacks detected by NLP method-based HIDS (Section 6.1);

$\bullet$ RQ2.2 Datasets used to apply NLP methods to develop HIDS (in Section
6.2); and

$\bullet$ RQ2.3 Evaluation metrics that are used to evaluate NLP method-based
HIDS (in Section 6.3).

Table 9: Categorization of attacks identified in NLP-based HIDS Attack |  Impacted |  Attack Instances |  Study Ref
---|---|---|---
Type |  Requirement |  |
R2L (55) |  Integrity |  imap, xlock, sshtrojan, ppmacro, netbus, sendmail, snmpget, ncftp, httptunnel, xsnoop, named, dict, framespoof, netcat, guest, ftpwrite, phf, WebShell, ShellCode, Tomcat v6.0.20, decode |  S1, S2, S3, S5, S6, S7, S8, S9, S11, S12, S14, S15, S16, S17, S18, S19, S20, S21, S22, S23, S24, S25, S27, S28, S29, S30, S31, S33, S35, S37, S38, S40, S41, S42, S43, S44, S45, S46, S47, S48, S49, S50, S51, S52, S53, S54, S55, S56, S57, S58, S59, S60, S61, S62, S63
U2R (53) |  Integrity |  Adduser, sechole, ps, secret, perl, fdformat, casesen, ntfsdos, yaga, ppmacro, eject, loadmodule, nukepw, sqlattack, xterm, ffbconfig, sunsendmailcp, syslog, CVE-2015-5602, CVE-2016-5195 |  S1, S2, S3, S5, S6, S7, S8, S9, S11, S14, S15, S16, S17, S18, S19, S20, S21, S22, S23, S24, S25, S27, S28, S29, S30, S31, S35, S37, S38, S40, S41, S42, S43, S44, S45, S46, S47, S48, S49, S50, S51, S52, S53, S54, S55, S56, S57, S58, S59, S60, S61, S62, S63
ACE (48) |  All |  Meterpreter, Java-Meterpreter, XAMPP Lite v1.7.3, Icecast v2.0, OS SMB, OS Print Spool, PMWiki v2.2.30, Wireless Karma, Adobe Reader 9.3.0, IE v6.0.2900.2180, Infectious Media, CVE 2012-0911, CVE-2009- 0927, PDFKA, CVE-2014-3120, CVE-2015-1427, CVE-2014-6271, Zip Slip, Redis, PHP-FPM |  S1, S2, S3, S5, S6, S7, S8, S9, S11, S12, S14, S15, S16, S17, S18, S19, S20, S21, S22, S23, S24, S25, S28, S29, S30, S31, S33, S35, S36, S37, S38, S40, S41, S42, S43, S44, S45, S46, S47, S48, S50, S51, S54, S55, S57, S59, S62, S63
Brute force (45) |  Confidentiality |  Password bruteforce (Hydra, Hydra-FTP, Hydra-SSH), CVE 2012-2122, CWE-307 |  S1, S2, S3, S5, S6, S7, S8, S9, S11, S14, S15, S16, S17, S18, S19, S20, S21, S22, S23, S24, S25, S28, S29, S30, S31, S35, S37, S38, S40, S41, S42, S43, S44, S45, S46, S47, S48, S50, S51, S54, S55, S57, S59, S62, S63
DoS (21) |  Availability |  selfping, dosnuke, back, tcpreset, syslogd, arppoison, mailbomb, teardrop, processtable, neptune, udpstorm, land, warezclient, apache2, crashiis, smurf, pod, DDoS, CVE 2012-0021, CesarFTP 0.99g, forwarding loops, CVE-2016-6515 |  S1, S6, S7, S9, S12, S16, S17, S18, S23, S24, S27, S30, S42, S49, S52, S53, S54, S56, S58, S60, S61
Backdoors (10) |  Confidentiality,
Integrity |  Backdoor executables , Win32.Hydraq |  S1, S4, S6, S9, S10, S12, S17, S36, S42, S54
Worm (6) |  Integrity,
Availability |  W32.Deborm.Y, W32.HLLW.Doomjuice.B, W32.Korgo.X, W32.Sasser.D, Daber.A, Slackor.A, NetSky.y, Mytob.x, Zhelatin.uq |  S1, S9, S17, S35, S36, S64
Trojan (5) |  All |  Ransom, Downloader, Spy, PSW, Dropper, Clicker, AdWare, WebToolbar, DangerousObject, RiskTool, Wannacry |  S4, S10, S26, S35, S64
Data Theft (4) |  Confidentiality |  Reconnaissance, CVE 2011-1153, CVE4-2014-0160 |  S1, S9, S17, S24
Probe (3) |  Confidentiality |  portsweep, queso, msscan, lsdomain, illegal-snifer, ipsweep ntinfoscan, satan, doppelganger, chimera, chameleon |  S12, S18, S30
Virus (3) |  All |  W32/Virut.n, Z0mbie.MistFall.3 |  S4, S10, S36
Misc (25) |  All |  lprcp, sm565a, sm5x, Packed, generic, IE deleted object vulnerability, CWE5-434, CWE-89 |  S1, S4, S7, S9, S10, S13, S16, S17, S18, S23, S27, S30, S32, S34, S36, S38, S39, S49, S52, S53, S56, S58, S60, S61, S65

###  6.1 RQ2.1: Attacks Detected by NLP Method-based HIDS

Our motivation for studying and categorizing security attacks is to
contextualize the attacks that are targeted to be detected by the reviewed
NLP-based HIDS. To answer RQ2, we categorize attacks that are detected using
NLP methods in HIDS by the reviewed papers, where the HIDS is evaluated
against a dataset including particular attacks. 133 instances were reported as
attacks in the reviewed studies, which we categorized into 12 attack
categories. Table 9 shows the attack categories and highlights the impacts of
the attacks on the security requirements, which provide a deep understanding
of security attacks and the invasion target. The attack categorization was
adapted and adjusted from a prior study (Khraisat et al., 2019). Later, we
show the mapping of the attack categories adopted in the used datasets for
NLP-based HIDS in Figure 4 (b) of Section 6.2.

####  6.1.1 Attack impacts on security requirements

We consider the three most significant security requirements (i.e.,
Confidentiality-Integrity-Availability (CIA)), that need to be provided by a
security framework (Ahmad et al., 2022). These requirements are impacted by
security attacks, and can leave disastrous consequences on host systems, such
as loss of personal information and business reputation along with financial
losses due to the impact on the requirements by security attacks. These
requirements are also known as CIA triad, which we describe as follows.

  * •

Confidentiality ensures data security by preventing information from being
disclosed to unauthorized individuals, entities or processes.

  * •

Data integrity protects data from unauthorized modifications throughout its
life cycle, ensuring its accuracy, trustworthiness and validity.

  * •

Availability ensures the availability of information or services for
legitimate users upon demand.

####  6.1.2 Categorization of attacks detected in NLP-based HIDS

In this section, we discuss the attack categories that can be detected using
NLP-based HIDS, in terms of included attacks and impacts on the above security
requirements.

(1) Remote to Local (R2L) is the most reported attack type in our reviewed
studies. It involves attackers exploiting the host’s vulnerability to gain
illegal local access and affects data integrity (Thomas et al., 2008). An
example of R2L attack instance is Tomcat v6.0.20, which enables remote
intruders to execute unrestricted file upload attacks. Another example is
ncftp, which is a Linux FTP file access-utility version with a bug that allows
remote commands to execute on a local host.

(2) User to Root (U2R) attack enables intruders to gain a system’s root access
starting with access to a normal user account (achieved by password sniffing,
dictionary attack, or social engineering) (Thomas et al., 2008). It bypasses
the authentication and threatens the data integrity by removing security
policy-specified files from the victim hosts. An example of U2R attack
instance is Adduser that creates new superuser to gain root access to a host
utilizing a corrupted executable.

(3) Arbitrary Code Execution (ACE) involves an attacker gaining control by
injecting his own code by exploiting some vulnerability. It can affect any of
the security requirements depending on the target of the executed arbitrary
code/commands on a target host (Simmons et al., 2014). An example of ACE
attack instance is XAMPP Lite v1.7.3 that permits a remote intruder to run
malicious payload utilizing Xampp_webdav application. Another ACE example is
Icecast v2.0 that allows remote attackers executing arbitrary code using an
HTTP request.

(4) Brute force attack employs a trial and error process that generates a huge
number of guesses and validates them to collect data (e.g., account password,
personal id number), which damages the data confidentiality and can circumvent
the authentication by acquiring an authorized user login information. For
example, a vulnerability (CVE 2012-2122) of MySQL 14.14 allows intruders to
bypass the authentication process by authenticating the same incorrect
password repeatedly.

(5) Denial of Service (DoS) attack denies the legitimate user access to a host
machine or service by making the computation resources extremely busy to
manage legitimate requests (Simmons et al., 2014). Examples of DoS include
exploiting Apache web server vulnerability CVE 2012-0021 for causing a daemon
crash using an empty cookie [S24] and tcpreset that terminates host TCP
connections (Thomas et al., 2008). Distributed DoS (DDoS) attack uses multiple
machines to flood a targeted resource that is detected in a few reviewed
studies [S1, S17]. Besides, forwarding loops are potential DoS attacks as they
cause one request to be processed repeatedly or even indefinitely, resulting
in undesired resource consumption that is detected in several reviewed studies
(e.g., [S16, S52]).

(6) Backdoor secures remote access to a machine, or plain text in a
cryptographic system, which attackers can use to get access to sensitive host
data (e.g., passwords), alter or remove information on hard drives and
transfer data across networks. Backdoor attack impacts confidentiality and
integrity. For example, Win32.Hydraqa is a family of backdoors [S36].

(7) Worm is a malware type, which replicates itself to spread to uninfected
hosts throughout the network without human intervention (Simmons et al.,
2014). Worms can damage service availability by consuming bandwidth/storage
space, and affect data integrity by corrupting or modifying files on a target
host. For example, NetSky.y, Zhelatin.uq and Mytob.x are worms, which can
overwrite other executables and try to exploit host OS components [S36].

(8) Trojan is a program that appears appealing and legitimate but has
anomalous code in it, which can affect any of the security requirements. For
example, AdWare is a software which automatically shows advertisement pop-ups
while a user is online. Other examples include Trojan-PSW which steals user
account information, WebToolbar that installs in-browser content without
users’ consent and Trojan-Ransom that prevents user’s access to demand
payment.

(9) Data theft threatens data confidentiality by stealing information stored
on corporate databases, devices and servers. An instance of data theft is
observed in a study [S24], which uses PHP 5.3.5 vulnerability CVE 2011-1153
that steals context-sensitive data from the memory of the host process.

(10) Probe attacks scan a host automatically to gather records of private
systems or a DNS server that devastates data confidentiality (Thomas et al.,
2008). For example, ipsweep and lsdomain enable finding legitimate IP
addresses. Mscan and queso enable finding host OS types. Portsweep enables
finding active ports.

(11) Virus is a code that attaches itself through any infected host files and
self-replicates when the program is run (Simmons et al., 2014). Based on the
target of the executed code, a virus can pose a threat to each security
requirement. Examples include W32/Virut.n which is a polymorphic virus
infecting HTML files and Z0mbie.MistFall.3 metamorphic virus infecting other
executables [S36].

(12) Miscellaneous category covers the remaining types of attacks that can
affect different security requirements depending on the target of performed
activities. If the details of the detected attack (i.e., anomaly) are not
presented in the corresponding reviewed study, we referred them under this
category. Examples include lprcp attack script that replaces arbitrary file
contents and Packed that refers to compressed obfuscated malicious programs,
which cannot be analyzed. Another example is exploiting Internet Explorer’s
(IE) deleted object vulnerability [S36].

Figure 4: Analysis of datasets grouped by type and availability with respect to NLP solutions categories and attack types  Table 10: Type of NLP-based HIDS datasets, their description with strengths and weaknesses Type |  Description |  Strengths |  Weaknesses
---|---|---|---
Real |  Datasets including real data captured from a real organization/production environment. Both data and environment are real. |  $\bullet$ Providing true distribution of data |  $\bullet$ Imbalanced datasets with an insufficient number of malicious activity
$\bullet$ Covering a limited attack types
Simulated |  Dataset including either synthetic data (e.g., artificially generated data) or data captured within a test bed or emulated controlled environment. |  $\bullet$ Able to reproduce balanced datasets.
$\bullet$ Able to generate rare misuse events.
$\bullet$ Useful for attacks for which real data is not available. |  $\bullet$ Tool specific
$\bullet$ May not depict the real distribution of data
$\bullet$ May not represent a real heterogeneous environment.

To summarize RQ2.1, we identified that all the NLP method categories
(described in Section 5.1) have been explored to detect the top 3 frequent
attack types such as R2L, U2R and ACE, as shown in Table 9. Besides, hybrid
NLP-methods have been explored to detect all attack categories. Semantic
ontology has the ability to correlate multiple attack instances, which enable
to dynamically detect diverse attacks in HIDS [S13] and language modeling has
been shown to significantly improve the HIDS detection performance [S5],
however, we observed ontology and language modeling are yet to explore the
detection of the different attack categories (e.g., DoS, Probe, Virus and
Backdoor). We present Table 9 to help researchers and developers to learn
about the attack types, the relevant attack instances and impact of these
attacks that they aim to detect using NLP-based HIDS.

###  6.2 RQ2.2: Datasets Used to Apply NLP Methods for Developing HIDS

We identified 20 different datasets employed in the reviewed NLP-based HIDS.
We discuss the characteristics of these datasets with respect to types of
datasets, availability, included attack and NLP methods used on these
datasets. Figure 4 shows the distribution of the datasets along with their
types of datasets and availability with respect to NLP-based HIDS solutions
categorization. Figure 4 also shows the different attack categories that are
present in the corresponding dataset along with the total attack categories
available in that dataset to show the diversity of the included attacks.
Figure 4 presents a comprehensive overview of the datasets to help
practitioners and researchers to understand these characteristics of the
datasets.

The primary data source of these datasets that are used to apply NLP methods
in HIDS research is syscall sequence. Syscall sequence is a primary artifact
of the OS kernel. Syscall sequence is considered the most reliable data source
for intrusion detection as they represent low-level information without any
filtering and processing [S4]. Since in syscall sequence, any filtering,
interpretation and processing are not used, it can not obfuscate events [S40].
Datasets have been generated by collecting syscall sequences either in real
(i.e., organization/production environment), simulated (either synthetic data
or controlled environment (e.g., testbed, emulation environment, lab)) or
hybrid environments. Inspired by an existing dataset classification (Ring et
al., 2019), we have categorized datasets used by the HIDS researchers into
real, simulated and hybrid categories to identify and characterize the
existing practices adopted to evaluate HIDS. While real dataset represents the
true distribution of data, it usually provides imbalanced dataset with limited
attack types. In contrast, simulated dataset is able to generate balanced
dataset, which may not depict the true data distribution. Hybrid dataset can
gain the advantage of both real and simulated dataset. Table 10 describes the
types of datasets along with their strengths and weaknesses. The details of
the study mapping with the dataset types are available in Table A.2 of our
online appendix (Zarrin Tasnim Sworna and Babar, 2022).

The availability of the identified datasets is categorized as public and
private (i.e., customized). Public datasets in the HIDS domain are usually
outdated, lack sufficient labeled data and do not cover a wide variety of
attack types as shown in Figure 4. To overcome these limitations, some studies
were motivated to explore methods for generating new customized datasets,
which are usually kept private. Table 11 presents a list of currently
accessible public datasets used in our reviewed studies with their
description, strengths and weaknesses to help HIDS researchers and developers
easily identify and choose the required public datasets to develop or evaluate
their NLP-based HIDS.

We discuss the identified datasets in terms of types and availability such as
we discuss public simulated dataset in Section 6.2.1, public real dataset in
Section 6.2.2, public hybrid dataset in Section 6.2.3, private simulate
dataset in Section 6.2.4 and private real dataset in Section 6.2.5.

####  6.2.1 Public simulated dataset

We identified six public simulated datasets such as i. Defence Advanced
Research Project Agency (DARPA) or Knowledge Discovery and Data Mining (KDD);
ii. University of New Mexico (UNM); iii. Firefox DS; iv. Next Generation
Intrusion Detection Systems Data Set (NGIDS-DS); v. Leipzig Intrusion
Detection-Data Set (LID-DS) and vi. Lab Artificial Intrusion Dataset (PLAID),
which we discuss in this section.

Table 11: Public datasets used in reviewed NLP-based HIDS with description,
strengths and weaknesses

Dataset |  Description |  Strengths |  Weaknesses
---|---|---|---
DARPA/KDD (Cunningham et al., 1999)
1998/99 |  Includes Basic Security Module (BSM) data file with syscall-based audit data produced in a victim’s machine for host-level audit. |  $\bullet$ First standard corpora for evaluation of NIDS and widely used as benchmark
$\bullet$ Includes arguments and return values |  $\bullet$ Very obsolete, unable to accommodate the latest trend in attacks
$\bullet$ Focus on NIDS and lacks the information required to train HIDS-
suitable methods
PUS-DS [S58]
2000 |  A syscall dataset collected from 8 users within 2 years. One of the user’s data is considered normal data, and it is mixed with few other user data to consider abnormal |  $\bullet$ Real dataset collected from eight different users |  $\bullet$ Lacks attack scenarios and attack diversity
$\bullet$ Outdated, unable to represent latest user behavior
UNM (UNM, 2004)
2004 |  Includes synthetic sendmail UNM, synthetic sendmail CERT, live lpr UNM and live lpr MIT datasets. Synthetic traces were collected by running a prepared script. |  $\bullet$ Includes programs of varied size and complexity, and different kinds of intrusions (buffer overflows, symbolic link attacks and Trojan programs) |  $\bullet$ Very obsolete
$\bullet$ Lacks syscall arguments or other metadata
$\bullet$ Extremely limited in scope and not represent a full sampling of OS,
focus on single processes (process IDs, syscall IDs)
CANALI-WD (Canali et al., 2012)
2012 |  Includes program execution traces observed both in a synthetic environment and on real-world machines with actual users and under normal operating conditions. |  $\bullet$ Presents a large collection of anomalous traces compared to previously published syscall datasets (e.g., UNM).
$\bullet$ Not biased towards particular runtime environments, or usage patterns. |  $\bullet$ Lacks some useful information such as syscall arguments, timestamp, etc.
Firefox DS (Murtaza et al., 2013)
2013 |  Includes normal traces of Firefox 3.5 by executing 7 testing frameworks and anomalous traces by launching contemporary attacks against Firefox. |  $\bullet$ The completeness of Firefox’s normal behavior is ensured using code coverage for test-case execution, which resulted in 60% source code coverage |  $\bullet$ Lacks attack diversity
$\bullet$ All the traces are specific to firefox.
ADFA-LD (Creech and Hu, 2013a)
2013 |  Alternative to older datasets (DARPA, UNM) and collected under Ubuntu OS running services and simulating attacks. |  $\bullet$ To attain a realistic defensive and attacking environment, a system was attacked by a certified penetration tester with 60 different attacks belonging to U2R, R2L, ACE and brute force attack types. |  $\bullet$ Outdated and not representative of contemporary attacks
$\bullet$ Lacks syscall argument, return data and other metadata
ADFA-WD (Creech and Hu, 2013a)
2013 |  Collected on a Windows host, and a total of 12 known vulnerabilities were exploited to simulate different attack types. |  $\bullet$ Real vulnerabilities in commonly used software were considered for generating attack data |  $\bullet$ Outdated and not representative of contemporary attacks
$\bullet$ Lacks syscall arguments, return values, or other metadata
$\bullet$ Inadequate number of vulnerabilities used to create malicious
activity (Čeponis and Goranin, 2018)
ADFA-WD:SAA (Haider et al., 2016) 2016 |  An extension of ADFA-WD which includes Windows-based stealth attacks |  $\bullet$ Suitable for evaluation against stealth attacks |  $\bullet$ Lacks metadata
$\bullet$ Attack types limited to stealth attacks
NGIDS-DS (Haider et al., 2017)
2017 |  Obtained from Ubuntu 14.0.4 host that is equipped with an auditing mechanism and includes 99 host log files. |  $\bullet$ Up-to-date and synthetically realistic
$\bullet$ Includes thread info, timestamp, eventID, path, processID, syscall, etc. |  $\bullet$ Loss of parameters and more accurate timestamps
AWSCTD (Čeponis and Goranin, 2018)
2018 |  Includes benign software samples as benign data and public malware files from Virus Share (VirusShare, 2017) and publicly available data about the malwares identified from Virus Total (VirusTotal, 2017) as attack data. |  $\bullet$ New extended dataset for Windows
$\bullet$ Includes parameters (syscall args, return value) for in-depth training |  $\bullet$ Includes only different Malware types and lacks other types of diverse attacks.

LID-DS (Grimmer et al., 2019a)
2019 |  A modern data set collected from a modern OS (Ubuntu 18.04) by considering diverse scenarios of real vulnerabilities. |  $\bullet$ Consists of different real vulnerabilities scenarios.
$\bullet$ Includes unambiguous syscall sequence and their thread assignment of
recent multi-threaded scenarios.
$\bullet$ Includes parameter, return data, user id, process/thread id, file system handle, timestamp and I/O buffer |  $\bullet$ Only available for Linux OS
PLAID [S21]
2021 |  Includes modern system calls and contemporary attack types collected on Ubuntu 18.04 LTS. |  $\bullet$ The most up-to-date HIDS dataset currently available.
$\bullet$ Includes attacks in multi-process applications. |  $\bullet$ Lacks syscall arguments, return values, or other metadata.
$\bullet$ Includes only six attack vectors.

  * •

*PUS-DS and Firefox-DS are not currently accessible

DARPA or KDD (Cunningham et al., 1999): DARPA or KDD are the earliest efforts
of IDS dataset which are focused on NIDS data and lack sufficient information
needed to train HIDS using suitable NLP methods [S10]. The DARPA dataset was
generated in a testbed and includes information of syscalls generated by
running programs for 7 weeks for the training dataset and 2 weeks for the test
dataset [S18]. Though DARPA includes DoS, R2L, U2R and Probe attacks, it is
considered obsolete as it is unable to accommodate the latest trend in attacks
(Moustafa and Slay, 2015). The hardware and the corresponding OS with their
syscalls of this outdated dataset have changed over time. DARPA and KDD
datasets were adopted in two studies [S18, S30] along with the UNM and
Australian Defense Force Academy Linux Dataset (ADFA-LD) datasets to cover
diverse complexity levels (i.e., syscall sequence generated by different
running programs), where DARPA represents the simplest dataset.

UNM (UNM, 2004): UNM includes program traces which were collected during live
execution of privileged host programs. It also includes simulated traces which
were collected using a script to collect ‘xlock’ program’s commands [S53].
Every trace includes a complete set of syscalls made during the program
execution (e.g., login, sendmail) from start to end. A few studies [S60, S62]
used syscalls of only specific programs (e.g., sendmail [S60, S61]). UNM
dataset includes different attacks such as U2R, R2L and DoS. UNM being an old
dataset, most of the studies that adopted this dataset used the classical
n-gram-based NLP method with diverse classifiers. In spite of a remarkable
usage of the UNM dataset in developing HIDS research, that dataset is marked
as outdated and limited in scope (Creech and Hu, 2013a). UNM does not reflect
the sophistication of modern attacks, and both normal and attack traces no
longer represent the complexity of current host systems due to their
outdatedness [S41]. Besides, the benign syscall traces in UNM dataset were
generated by executing small programs for a longer time period that generated
the same syscall execution paths [S23]. Still, they are being used for
benchmarking HIDS [S18, S30] along with the newer datasets to show the
effectiveness of diverse datasets as there is a lack of many publicly
accessible datasets.

Firefox DS (Murtaza et al., 2013): To overcome the above-mentioned issues of
UNM dataset, firefox dataset was created for Firefox web browser. It includes
700 normal syscall traces of Firefox3.5 by executing 7 diverse testing
frameworks, where every testing framework executed various Firefox components
and functionalities [S23]. This dataset includes 19 anomalous syscall traces
by launching contemporary attacks including ACE and DoS attack types against
Firefox, selected from public advisories and resources (e.g., Metasploit
(Metasploit, 2020)). However, Firefox-DS is a small-scale dataset including a
very limited number of attack traces. Hence, when using this dataset, other
large-scale datasets should be used to evaluate the scalability of the
proposed NLP-based HIDS.

NGIDS-DS (Haider et al., 2017): It is a relatively new Linux OS-based dataset
that is generated in an emulated environment (Ring et al., 2019). NGIDS-DS
includes 99 host log files, where each record represents all the corresponding
information (e.g., syscalls, path and attack category) about an occurred
event, for both normal and anomalous activities (e.g., DoS, worm and
backdoor). Only n-gram-based and hybrid approaches were adopted by the
reviewed studies to evaluate HIDS using this dataset. Though NGIDS-DS includes
diverse attacks and thread information, more accurate timestamps are missing,
which makes it impossible to determine a deterministic order of syscalls
reliably [S9]. Since several syscalls are with the same timestamp, and event-
ids’ order sometimes contradicts timestamps’ order, reconstructing the correct
sequence of syscalls is difficult.

LID-DS (Grimmer et al., 2019b): To mitigate the incorrect sequence issue of
NGIDS-DS dataset, LID-DS is released with the unambiguous syscall sequence and
their thread assignment. The software of modern IT systems is multi-threaded,
thus diverse syscalls may belong to different threads and users. The
traditional HIDS does not consider the thread in which a syscall is executed,
which consequently ignores the fact that software of modern IT systems is
usually multi-threaded. Execution schedule of OS may generate diverse syscall
sequences for multiple executions of the same software with the same input
data. Hence, this can negatively impact the accuracy of the HIDS which focuses
on syscall subsequence learning and analyzing [S9]. Hence, LID-DS includes
several features that are unavailable in the earlier datasets such as syscall,
timestamp, thread id, process name, argument, return data and excerpt of data
buffer from syscall traces of normal and anomalous behavior of multiple
recent, multi-process and multi-threaded scenarios. Using the comparative
analysis of different NLP methods (e.g., n-gram, word-embedding), a study [S9]
showed that the included thread information from LID-DS enhances detection
accuracy and reduces FAR.

PLAID (2021) [S21]: While HIDS traditionally considers syscall traces
generated by a single process, modern applications often use multiple
processes. Besides, one or multiple processes can be impacted by modern
attacks. Hence, the latest syscall sequence-based dataset, PLAID, includes
attacks in multi-process applications. PLAID is Linux OS-based with modern
syscalls and modern attacks. PLAID includes syscall sequences from 6 current
exploits and penetration methods along with a huge collection from benign
operations.

####  6.2.2 Public real datasets

We identified two public real datasets such as i. Purdue Unix Shell (PUS) and
ii. Attack-caused Windows OS syscall Traces Dataset (AWSCTD), which we discuss
in this section.

PUS (2000) [S58]: PUS is a real dataset, which is a syscall dataset collected
from 8 users within 2 years [S58]. One of the user’s data is considered normal
data, and it is mixed with a few other user data to consider abnormal data.
PUS lacks attack scenarios and attack diversity. Besides, PUS is outdated and
unable to represent the latest user behavior.

AWSCTD (Čeponis and Goranin, 2018): AWSCTD is considered one of the biggest
syscall collection running on Windows OS that includes 112.56 million syscalls
from 12110 executable malware samples and 16.3 million syscalls from 3145
benign software samples. AWSCTD includes malware-initiated system calls which
were collected from publicly available malware files from VirusShare
(VirusShare, 2017) and real world malware information from VirusTotal
(VirusTotal, 2017). It includes different attacks such as worm, trojan and
backdoor. AWSCTD is adopted by 2 studies [S4, S10] that showed word embedding-
based CNN outperformed RNNs [S10] and complex dual-flow models including CNN
and RNN models (e.g., LSTM-FCN and GRU-FCN) [S4]. As Windows is still a
popular OS, HIDS researchers and developers who are interested in Windows OS-
specific HIDS development can use the AWSCTD dataset as it is a real extended
dataset compared to other Windows-based datasets [S10].

####  6.2.3 Public hybrid dataset

We identified four public hybrid datasets including i. CANALI Windows Datasets
(CANALI-WD); and three Australian Defence Force Academy (ADFA) family datasets
such as ii. ADFA Linux Dataset (ADFA-LD); iii. ADFA Windows Dataset (ADFA-WD)
and iv. ADFA Windows Dataset: Stealth Attacks Addendum (ADFA-WD:SAA), which we
discuss in this section.

CANALI-WD (Canali et al., 2012): It is a Windows OS-based dataset that
includes program execution syscall traces observed both in simulated (based on
Anubis (Lab, 2011)) and the real-world environment with actual users. CANALI-
WD dataset includes 2 normal datasets, namely, Goodware and Anubis-good [S64].
Goodware includes 180 GB syscall traces gathered from 10 real world host
systems, used by regular users and Anubis-good includes syscall traces of 36
normal applications executed under Anubis. CANALI-WD also includes 2 malware
datasets, namely, malware and malware-test. The malware dataset includes 5,855
malware (e.g., botnets and worms) syscall traces randomly extracted from
Anubis and the malware-test dataset includes 1,200 syscall traces of malware
collected from another machine that was not used by Anubis. While the
previously published syscall datasets (e.g., DARPA and UNM) have limited
malicious traces, CANALI-WD presents a huge collection of malicious traces.

ADFA-LD (Creech and Hu, 2013a): The outdated datasets such as DARPA and UNM,
which are based on more than 20 years old software which are usually no longer
in use to detect or reflect the modern intricate security attacks. To replace
these outdated datasets and represent modern attack structure and methodology,
Australian Centre of Cyber-Security released two datasets for two different OS
(i.e., Linux and Windows) named ADFA-LD and ADFA-WD, respectively. ADFA-LD is
a dataset of syscall traces of Linux OS, which is the most used dataset to
evaluate NLP method-based HIDS. ADFA-LD consists of syscall traces of
different active services during normal host operation, where the included
activities range from web browsing to LATEX document preparation. To attain a
realistic defensive and attacking environment, a system was attacked by a
certified penetration tester with 60 different attacks belonging to U2R, R2L,
ACE and brute force attack types. During the execution of attacks, the
corresponding syscall traces were recorded. ADFA-LD includes 833 normal
training syscall sequences, 746 attacks and 4372 normal validation syscall
sequences to evaluate HIDS. The scale of ADFA-LD dataset is considered
suitable for training deep neural networks [S5, S15]. All the reported NLP
categories except the semantic ontology were adopted to detect intrusion in
ADFA-LD dataset by the reviewed studies.

ADFA-WD and ADFA-WD:SAA (Haider et al., 2016): These datasets are based on
Windows OS, which contain syscall traces from Windows XP SP2 system. While
ADFA-WD includes zero-day attacks based on vulnerabilities, ADFA-WD:SAA is
created to evaluate the effectiveness of HIDS against stealth attacks. In
ADFA-WD, automated hacking tools were used to exploit 12 known vulnerabilities
and the attack vectors include browser attack and malware attachment. ADFA-WD
includes 355 normal training syscall sequences, 5542 attacks and 1827 normal
validation syscall sequences. In contrast, ADFA-WD:SAA includes 862 attack
syscall traces. Since both of these syscall-based HIDS datasets are the
sequence of tokens, NLP methods are well-suited to handle these datasets
[S12].

####  6.2.4 Private simulated datasets

In this section, we discuss the simulated datasets, which are not publicly
available. To create private simulated datasets, researchers have used various
libraries or tools (e.g., Strace [S24], LTTng [S24]), which allow monitoring
the execution of a program and read syscall traces on user or kernel space.
Ptrace helps to trace Linux syscall. Besides, Strace open-source application
utilizes ptrace to provide statistics about a trace in text format. Linux
Trace Toolkit Next Generation (LTTng) tracer saves traces in Common Trace
Format (CTF) (Desnoyers and Dagenais, 2006). Further, drstrace helps to trace
syscall for Windows OS. A study [S10], used drstrace to append syscall with
AWSCTD dataset.

A study [S24] created a dataset based on a local Linux OS, where normal
syscall traces were collected by several tasks (e.g., web browsing and
document processing). For collecting attack syscall traces, Metasploit
(Metasploit, 2020) open source penetration testing tool was utilized to
exploit the popular vulnerabilities for executing the attacks. Another study
[S33] created a dataset by simulating attacks in a controlled environment on a
private Ethernet-based local network (includes 2 desktops and an IBM ES750
Network Scanner). This dataset enabled the study to evaluate the reasoning
logic of semantic ontology on multiple diverse vulnerabilities. The reasoning
logics of the semantic ontology were used to analyze the text description of
the vulnerabilities and then use them to detect the possibility of attacks in
logs, which successfully inferred 7120 of the 8070 attacks.

####  6.2.5 Private real datasets

A study [S32] created a dataset by collecting the normal samples from a
corporate network border’s live feeds of all the files, which were filtered
using anti-virus scanners to ensure them as benign data. Attack data was
collected from the daily feed of a security company which gathers malicious
software from their network sensors. Another study [S36] gathered various
real-world exploits and legitimate applications and confronted them on a
Windows 7 host. The studies [S13, S26] that intend to perform knowledge fusion
by semantic ontology built their own dataset by collecting data from varied
dynamic textual sources (e.g., Symantec’s website (Symantec, 2018)) and
combining them with malware behavioral information to detect attacks. While a
study [S39] collected syscall sequences on the process of sending and
receiving e-mails, another study [S34] focused on Gzip software (a file
compression and decompression tool for Linux) to collect data.

To summarize RQ2.2, we identified ADFA-LD (Creech and Hu, 2013a), UNM (UNM,
2004) and ADFA-WD (Creech and Hu, 2013a), are widely adopted to develop and
evaluate NLP-based HIDS in practice. However, these widely adopted datasets
are old to represent modern complex attacks and have only basic information
(e.g., syscall identification and called function names), which is minimal to
detect intricate and modern security attacks in the host [S10]. Besides, an
inadequate number of vulnerabilities were utilized to generate attacks [S10]
in these datasets. To reflect the modern host systems’ behavior and attacks in
the host, our SLR identified several recent and extended datasets, such as
AWSCTD (Čeponis and Goranin, 2018) dataset for Windows OS, and NGIDS-DS
(Haider et al., 2017), LID-DS (Grimmer et al., 2019a) and PLAID [S21] datasets
for Linux OS. Moreover, we observed several datasets used to develop and
evaluate NLP-based HIDS in practice are private, which can hinder the
reproducibility and replicability of the NLP-based HIDS.

###  6.3 RQ2.3: Evaluation Metrics

We identified 17 evaluation metrics that have been used to evaluate the
performance of NLP-based HIDS. Several reviewed studies perform an
intermediary step (i.e., syscall sequence prediction) and then perform the
intrusion detection to enhance the detection capabilities. These studies
evaluate both steps with distinguished evaluation metrics. Besides, the
computation performance (efficiency) is also measured along with the detection
performance (i.e., effectiveness). Hence, we categorized the evaluation
metrics into three categories such as detection performance (Section 6.3.1),
computation performance (Section 6.3.2) and intermediary task performance
(Section 6.3.3). Table 12 presents the metrics of each category with their
description, mathematical representation and corresponding strengths and
weaknesses. This comparative analysis of Table 12 will help HIDS researchers
to choose suitable metrics to evaluate the proposed HIDS. Besides, Table 12
will help the HIDS developers to reflect the organization’s requirements and
preferences by evaluating the HIDS based on the relevant metrics. The study
mapping with the metrics is available in Table A.3 of our online appendix
(Zarrin Tasnim Sworna and Babar, 2022).

Table 12: Metrics used for evaluation of NLP-based HIDS and their description, equation, strengths and weaknesses Evaluation |  |  |  Paper |  |
---|---|---|---|---|---
Metric |  Description |  Equation |  Count |  Strengths |  Weaknesses
Detection Performance (Effectiveness)
Detection Rate (Recall, detection accuracy, TPR) |  Ratio of correctly identified attacks out of total attack samples. |  $\frac{TP}{TP+FN}$ |  45 |  Prioritizes detecting all the attacks even at a cost of false alarm. Considering it is less costly to process all the alerts compared to missing to detect an attack, recall can be helpful. |  Do not consider TN (Use of MCC can resolve this).
Precision |  Ratio of the correctly identified attacks to total samples identified as attacks. |  $\frac{TP}{TP+FP}$ |  13 |  Considering that handling false alarms is costly, optimized precision reveals that all the predicted alarms are worth noticing. |
F-Measure |  Harmonic mean of precision and recall. |  $\frac{2\times P\times R}{P+R}$ |  15 |  Combines precision and recall. Suitable for highly imbalanced HIDS dataset. |
False Alarm Rate (FAR, false positive rate, FPR) |  Ratio of incorrectly detected normal samples out of total normal samples. |  $\frac{FP}{FP+TN}$ |  40 |  Given that the cost of dealing with a huge volume of alerts of a HIDS is high, it is an important metric for HIDS. |  Should be used as an auxiliary metric with other metrics.
False Negative Rate (Missing Rate, FNR) |  Ratio of incorrectly detected attacks out of total attack samples. |  $\frac{FN}{FN+TP}$ |  8 |  Given that the cost of the inability to detect an attack by the HIDS model is high, it presents the missed attacks. |
True Negative Rate (TNR) |  Ratio of correctly identified normal samples out of total normal samples. |  $\frac{TN}{TN+FP}$ |  4 |  To represent that the HIDS is usually correct to predict a benign behavior, it can be used. |
Confusion matrix |  Matrix of prediction results. |  - |  3 |  Presents nominal value rather than normalized value to show how the HIDS model performs for different imbalance classes. Gives insight into the type of error made by the HIDS. |  Rather than using it as a metric, the matrix is usually used to calculate other metrics (e.g., precision).
AUC based on ROC Curve (ROC-AUC) |  ROC curve shows the TPR as a function of FPR as the discrimination threshold of the classifier is varied, and AUC is the area under ROC. |  - |  33 |  Not dependent on prediction threshold. |  Not suitable for heavily imbalanced HIDS data (Use of Precision-recall-AUC can resolve this).
Classification Acc/Rate |  Percentage of correctly classified samples. |  $\frac{TP+TN}{TP+TN+FP+FN}$ |  18 |  Consider all the cells of a confusion matrix. |  Not suitable for imbalanced data.
Classification Error |  Percentage of incorrectly classified samples. |  $\frac{FP+FN}{TP+TN+FP+FN}$ |  1 |  Suitable to represent the error of the HIDS model as the HIDS output is discrete. |  Can be used as an auxiliary metric with other metrics.
Mean error rate (MER) |  Average error rate of FPR and FNR. |  avg(FPR,FNR) |  1 |  |
Matthews Correlation Coefficient (MCC) |  Classifications quality measure, robust to highly imbalanced data. Here, x=(TP+FP)(TP+FN)(TN+FP)(TN+FN) |  $\frac{TP\times TN-FP\times FN}{\sqrt{x}}$ |  1 |  Suitable for imbalanced data. Unlike precision, recall and f-measure, MCC considers TN |  MCC is yet to be widely adopted by the HIDS literature.
Computation Performance (Efficiency)
Time |  Training time/ testing time/ execution time |  - |  15 |  Presents the applicability of the HIDS in a real industry setting in terms of required time. |  HIDS efficiency should be used along with effectiveness metrics.
Performance of Intermediary Task of Sequence Prediction
BLEU score |  Compares a predicted syscall sequence to a target sequence. |  brevity penalty$\times$geometric avg of precision score |  3 |  Can be used when there is more than one ground truth sequence. Language-independent which makes it straightforward to apply to HIDS models. |  Focus on statistical similarities, they can not ensure if the syscall sequence makes sense to the OS.
TF-IDF |  TF-IDF rises in proportion to the word (i) freq in a document (j), but is offset by the number of documents in the corpus that include the word. |  $TF_{i,j}\times log\frac{N}{df_{i}}$; $TF_{i,j}$=freq of i in j, $N$= doc count, $df_{i}$= num of doc containing $i$ |  2 |  Language-independent making it straightforward to apply to HIDS models. |
Cosine Similarity |  Cosine of the angle between two vectors (x, y) to determine if they are in the same direction. |  $\frac{xy}{||x||||y||}$ |  2 |  Performs correlation analysis of the predicted and target sequence to ensure if the syscall sequence makes sense to the OS. |  Differences in values are not fully considered.

####  6.3.1 Detection performance

12 evaluation metrics have been used to validate the detection result by the
reviewed HIDS studies which used NLP methods. As shown in Table 12, the
mathematical representation of these evaluation metrics includes any of True
Positive (TP), False Positive (FP), True Negative (TN) or False Negative (FN)
value of confusion matrix (Jiao and Du, 2016). While confusion matrix holds
the relevant nominal value instead of normalized values, it is usually used to
calculate other metrics (e.g., recall). Detecting all the attacks at a cost of
false alarms is highly significant, whereas handling a huge volume of false
alarms of a HIDS is time consuming and costly for security analysts. Hence,
HIDS usually intends to maximize the attack Detection Rate (DR) and minimize
the False Alarm Rate (FAR), which are the top 2 most used metrics in the
reviewed NLP-based HIDS literature. In the HIDS domain, DR, recall and True
Positive Rate (TPR) are referred to as the same. Besides, FAR and False
Positive Rate (FPR) are considered as the same (Liu et al., 2018).

While optimized precision reveals that all the predicted alarms are worth
noticing, F-measure combines both precision and recall Gamage and Samarabandu
(2020). F-measure is a better metric for the highly imbalanced HIDS dataset.
However, precision, recall and F-measure do not consider TN. Besides,
considering the cost of the inability to detect an attack by the HIDS model is
high, False Negative Rate (FNR) denotes the missed attacks. Further, a high
True Negative Rate(TNR) denotes that the HIDS is usually correct to predict
benign behavior. Since the output of the HIDS model is discrete,
classification error and Mean Error Rate (MER) are suitable to represent the
error of the HIDS model. FNR, TNR, classification error or MER can be used as
auxiliary metrics with other metrics (e.g., DR, FAR) (Czakon, 2022).

The other two highly used metrics in the reviewed NLP-based HIDS literature
are ROC-AUC and accuracy. Compared to accuracy, AUC based on the ROC curve
(ROC-AUC) provides a more robust measure for the evaluation of HIDS derived
from imbalanced datasets as accuracy can be dominated by the majority benign
class in HIDS. However, if the dataset is highly imbalanced, the shape of the
ROC-AUC curve can be misleading (Jiao and Du, 2016). Thus, Matthews
Correlation Coefficient (MCC) can be used, which is considered one of the
best-balanced evaluation measures (Jiao and Du, 2016). Unfortunately, only one
study [S4] used MCC.