m m m m m m m m #7 0 #8 0 ##1 ####1 ¡ #3 round(100 * (- #1) / (#3 - #1), 2)
##1 ####1 bg=#4!!#2 round(100 * (- #3) / (#5 - #3), 2) ##1 ####1 bg=#6!!#4
[fieldset=month, null] [fieldset=address, null] [fieldset=location, null]
[fieldset=publisher, null] [fieldset=url, null] [fieldset=isbn, null]

# Evaluating ML-Based Anomaly Detection Across Datasets of Varied Integrity: A
Case Study

Adrian Pekar and Richard Jozsa A. Pekar and R. Jozsa are with the Department
of Networked Systems and Services, Faculty of Electrical Engineering and
Informatics, Budapest University of Technology and Economics, Műegyetem rkp.
3., H-1111 Budapest, Hungary. A. Pekar is also with the HUN-REN-BME
Information Systems Research Group, 1117, Magyar Tudósok krt. 2, Budapest,
Hungary. Corresponding author: A. Pekar<EMAIL_ADDRESS>
###### Abstract

Cybersecurity remains a critical challenge in the digital age, with network
traffic flow anomaly detection being a key pivotal instrument in the fight
against cyber threats. In this study, we address the prevalent issue of data
integrity in network traffic datasets, which are instrumental in developing
machine learning (ML) models for anomaly detection. We introduce two refined
versions of the CICIDS-2017 dataset, NFS-2023-nTE and NFS-2023-TE, processed
using NFStream to ensure methodologically sound flow expiration and labeling.
Our research contrasts the performance of the Random Forest (RF) algorithm
across the original CICIDS-2017, its refined counterparts WTMC-2021 and
CRiSIS-2022, and our NFStream-generated datasets, in both binary and multi-
class classification contexts. We observe that the RF model exhibits
exceptional robustness, achieving consistent high-performance metrics
irrespective of the underlying dataset quality, which prompts a critical
discussion on the actual impact of data integrity on ML efficacy. Our study
underscores the importance of continual refinement and methodological rigor in
dataset generation for network security research. As the landscape of network
threats evolves, so must the tools and techniques used to detect and analyze
them.

###### Index Terms:

CICIDS-2017, CICFlowMeter, NFStream, random forest, network traffic flow,
anomaly detection, cybersecurity.

##  I Introduction

Cybersecurity stands at the forefront of technological challenges in the
digital age. As security threats evolve in complexity and volume, the need for
advanced anomaly detection systems becomes imperative. Machine Learning (ML)
and Artificial Intelligence (AI) have emerged as powerful allies in this
fight, equipping us with sophisticated tools to anticipate, identify, and
neutralize threats. At the core of these tools lies a dependency on high-
quality datasets that accurately reflect the complexities of network traffic
and cyber attacks.

The CICIDS-2017 dataset [1] has been pivotal in the development of
methodologies for detecting network traffic anomalies. It offers a broad array
of labeled traffic patterns and serve as a fundamental testing ground for a
variety of ML-based approaches. Despite its widespread adoption and
significant contributions to the field, recent scrutiny has exposed errors
within these datasets that could skew the performance of detection systems
trained on them [2, 3, 4]. These inaccuracies highlight a critical need for
continuous validation and refinement of measurement techniques to ensure that
the models are learning from data that accurately represents real-world
conditions.

This paper delves into a thorough examination and comparative analysis of five
datasets—CICIDS-2017 [1], its WTMC-2021 [2] and CRiSIS-2022 [4] refinements,
and our two refinements, NFS-2023-nTE, which does not implement TCP flag
expiry, and NFS-2023-TE, which does—employing the Random Forest (RF) algorithm
to assess network anomaly detection in binary and multi-class classification
paradigms. Our study builds upon previous efforts to refine these datasets,
which, despite improvements, continue to exhibit anomalies such as
inexplicable counts of TCP FIN and RST flags, negative values in flow
features, and instances of missing data. These persistent inconsistencies form
the backdrop against which we scrutinize the datasets, aiming to unravel how
such irregularities might influence the performance and reliability of machine
learning models in detecting network anomalies.

The conclusions drawn from our investigation into network anomaly are
manifold. Firstly, the study highlights the remarkable resilience of the RF
model, which consistently delivers high accuracy across all examined datasets.
This consistency is a testament to the model’s robustness, effectively
transcending the challenges posed by data biases and measurement inaccuracies
inherent in the datasets. Secondly, our investigation into feature importance
sheds light on the nuanced role of TCP FIN and RST flags in the classification
process. While these features are not consistently the most influential across
all scenarios, their accurate representation proves critical in specific
contexts. Lastly, extending our analysis to include Decision Tree (DT) and
Naive Bayes (NB) algorithms, we observe a continuation of the trends
identified with the RF model. This corroboration across multiple algorithms
not only strengthens the validity of our findings but also underscores the
broader adaptability of ML techniques to datasets with varying levels of
imperfection.

Compiling these insights, the study underscores a dual reality: the
encouraging adaptability of machine learning models to imperfect datasets and
the potential risk of high model performance obscuring underlying data flaws.
While refined measurement techniques may not dramatically alter performance
metrics, they are indispensable in ensuring that models are trained on data
that truly reflects real-world network behaviors. This balance between model
resilience and data integrity is crucial for advancing the field of
cybersecurity, guiding future research towards the development of more
accurate and reliable anomaly detection systems.

The key contributions of this paper are as follows:

  1. 1.

We provide a thorough examination of the CICIDS-2017, WTMC-2021, and
CRiSIS-2022 datasets, identifying and documenting various imperfections. This
analysis is crucial as it brings to light specific issues that could
potentially hinder the effectiveness of ML applications in network anomaly
detection, thereby informing future dataset refinement and usage strategies.

  2. 2.

We introduce two refined versions of the CICIDS-2017 dataset—NFS-2023-nTE [5],
which does not implement TCP flag expiry, and NFS-2023-TE [6], which does.
These refined datasets represent a step forward in terms of data integrity and
reliability, and are made available online for widespread use, promoting
transparency and replicability in research.

  3. 3.

We make our detailed and reproducible methodology publicly accessible as a
digital artifact [7], which embodies the core of our contribution. This
methodology underpins the findings reported in this paper and offers a
versatile resource for researchers and practitioners in the field of ML-based
network security. It can be readily adapted to process the CICIDS-2017 PCAP
files, catering to the specific needs of diverse research endeavors.

Our research endeavors to build upon, rather than discredit, the foundations
laid by existing datasets. Ultimately, our work seeks to contribute to the
evolution of machine learning in cybersecurity, aligning research efforts with
the complex realities of network traffic and enhancing the resilience of
systems against emerging cyber challenges.

The rest of this paper is structured as follows: Section II offers a brief
overview of the CICIDS-2017 dataset and its subsequent refinements, WTMC-2021
and CRiSIS-2022. Section III delves into identifying the key limitations and
imperfections inherent in these datasets. Section IV outlines the primary
objectives of our work, establishing the goals and motivations driving our
research. Section V discusses the methodology employed in generating our
refined versions of the CICIDS-2017 dataset. Section VI analytically
examinatnes the five datasets, focusing on aspects such as flow counts, label
distributions, occurrences of negative and NaN values, and TCP FIN and RST
flag counts. Section VII compares the performance of the RF model across all
five datasets, encompassing both binary and multi-class classification
scenarios. Section VIII presents our findings and their broader implications.
Finally, Section X concludes this paper.

##  II Evolution of the CICIDS-2017 Dataset

In the realm of machine learning-based anomaly detection, the integrity and
realism of datasets play a crucial role. The CICIDS-2017 dataset, in this
context, has emerged as a foundational resource in the development of
methodologies for network attack detection and prediction. Despite its
extensive adoption in the cybersecurity community, the dataset has been
subject to scrutiny due to inherent inconsistencies that could potentially
skew research outcomes. This section explores the evolution of the CICIDS-2017
dataset, highlighting its original composition and subsequent refinements.

###  II-A CICIDS-2017

The CICIDS-2017111https://www.unb.ca/cic/datasets/ids-2017.html dataset [1] is
a comprehensive collection of labeled network traffic patterns, encompassing
benign activities and simulated malicious attacks, including DoS/DDoS, Port
Scan, Brute Force, and Infiltration events. Generated over five days from 3 to
7 July 2017, this dataset offers a diverse range of flows, captured in
individual PCAP files. Accompanying CSV files, produced using the
CICFlowMeter222https://github.com/ahlashkari/CICFlowMeter [8, 9], provide flow
records with corresponding labels, facilitating the assessment and comparison
of various ML-based anomaly detection approaches.

###  II-B WTMC-2021: Advancing CICIDS-2017

Recognizing the limitations of the original CICIDS-2017 dataset, recent
efforts led by [2] have resulted in significant improvements [2, 3]. The
WTMC-2021 dataset333https://intrusion-detection.distrinet-
research.be/WTMC2021/tools_datasets.html, a refined iteration, addresses key
issues in traffic generation, flow construction, feature extraction, and
labeling identified in the original dataset. This refined version, which
revises over 20 percent of the original traffic data, offers a more accurate
and reliable resource for anomaly detection research, setting a new standard
in dataset integrity for the cybersecurity community.

###  II-C CRiSIS-2022: Further Refinement

Building upon the advancements of WTMC-2021, the CRiSIS-2022
dataset444https://gitlab.inria.fr/mlanvin/crisis2022 introduced by [4] offers
additional enhancements [4]. This version addresses overlooked issues such as
mislabeled port scan attacks and duplicated traffic captures. By rectifying
these aspects, CRiSIS-2022 provides a more robust and accurate dataset,
further aligning simulated network traffic conditions with real-world
scenarios and bolstering the reliability of subsequent cybersecurity research.

##  III Key Limitations in Dataset Realism and Applicability

Despite the release of the refined CICIDS-2017 and the remedial patches
applied to the CICFlowMeter tool, the application of this dataset for research
transferable to real-world scenarios remains limited. The CICFlowMeter tool
has been the cornerstone for all existing versions of the CIC-IDS-2017
dataset, including the original and the improved ones. Although patches have
been implemented to rectify the tool, its flow construction and feature
extraction pipeline still diverge from the operations followed in real-world
flow metering. The limitations can be grouped into two primary categories.

###  III-A Inadequate Implementation of Flow Timeout Mechanism

The CICFlowMeter tool, diverges from standard flow metering practices in its
implementation of flow timeouts. While it employs a flow timeout akin to an
idle timeout, it lacks a standard active timeout mechanism. Instead, it uses
an activity timeout for subflow statistics, differing from the de facto
standard in flow metering tools and protocols like NetFlow and IPFIX. This
absence of a conventional flow expiration policy results in datasets that are
not fully representative of typical network conditions, a limitation present
in both the original and patched versions of CICFlowMeter.

###  III-B Flaws in TCP Flow Flag-Based Expiration Mechanism

The original CICFlowMeter was claimed lacking a robust TCP flag-based flow
expiration mechanism, terminating TCP flows upon the first observation of FIN
or RST flags. This was addressed by the patches introduced by [2], which
aligned flow termination more closely with the TCP specification. In the
patched version, a TCP flow is no longer terminated after a single FIN packet.
It terminates after mutual exchange of FIN packets, which is more in line with
the TCP specification. Additionally, an RST packet is no longer ignored
either. Instead, the RST packet also terminates a TCP flow.

However, this modified termination process, while technically accurate,
deviates from the behavior commonly observed in commercial flow metering
tools, which typically terminate flows upon the first FIN or RST flag. This
discrepancy introduces potential biases in performance assessments, as the
datasets do not accurately reflect real-world flow metering behavior.

Additionally, and more importantly, with the patch, a critical issue was
introduced. The WTMC-2021 and CRiSIS-2022 datasets exhibit anomalously high
FIN and RST flag counts in individual flow records, with counts sometimes
exceeding 100. Such behavior is atypical in standard flow metering practices,
leading to ambigous interpretability and relevance for real-world
applications.

TABLE I: Summary of PCAP Files Analysis PCAP | No. of Packets | No. of Duplicates | % Duplicates | No. of Frames | No. of Out of Order Frames | % Out of Order
---|---|---|---|---|---|---
Monday | 11 709 971 | 526 729 | - | 11 183 242 | 3 105 | -
Tuesday | 11 551 954 | 496 702 | 4.3% | 11 055 252 | 3 581 | 0.03%
Wednesday | 13 788 878 | 494 582 | 3.6% | 13 294 296 | 12 531 | 0.09%
Thursday | 9 322 025 | 568 505 | 6.1% | 8 753 520 | 3 424 | 0.04%
Friday | 9 997 874 | 480 540 | 4.8% | 9 517 334 | 6 980 | 0.07%

##  IV Objectives

Our research is guided by three central objectives.

###  IV-A Addressing Identified Issues with a Methodologically Sound Dataset

The primary goal of our research was to address the critical issues
highlighted in Section III by developing a dataset through a process that is
both transparent and methodologically robust. The CICFlowMeter, widely
recognized in AI and ML-based anomaly detection research, suffers from limited
documentation, scalability issues, methodological errors, and constraints in
customizability. Its open-source nature does not mitigate the challenge of
deciphering and resolving its complex underlying issues. Moreover, although
there is a hint about an impending Python-based version that might address
these issues555https://github.com/ahlashkari/CICFlowMeter/issues/154, such a
release had not materialized as of the date of our study.

In our search for a tool with methodologically sound flow feature calculation,
we utilized NFStream, offering dependable performance and flexibility in
customization. It is particularly adept at implementing TCP flow expiration
policies and advanced flow labeling methodologies. By utilizing NFStream, we
were able to sidestep the intricate challenges posed by CICFlowMeter,
effectively addressing the limitations highlighted in this present work. As a
result, we generated two novel versions of the CICIDS-2017 dataset:

  * •

NFS-2023-nTE: This version disabled TCP flag-based flow expiration, aiming to
mirror the flow generation process used in existing dataset versions.

  * •

NFS-2023-TE: In contrast, this version enabled TCP flag-based flow expiration,
better reflecting the flow characteristics observed in real-world network flow
metering scenarios.

###  IV-B Comparing ML Model Performances Across Datasets

Our second objective focused on conducting a comparative analysis of ML model
performances across several datasets. This encompassed the original
CICIDS-2017 dataset, along with its WMTC-2021 and CRiSIS-2022 refinements, and
the two novel versions we created using NFStream: NFS-2023-nTE (without TCP
flag expiry) and NFS-2023-TE (with TCP flag expiry). The purpose of this
comparison was to evaluate the influence of different flow metering strategies
on ML model outcomes. A key aspect of this analysis was to understand how the
incorporation or absence of TCP flag expiration policies affects the accuracy
and reliability of these models in network traffic analysis and anomaly
detection.

###  IV-C Developing a Customizable Dataset Creation Methodology

Finally, our present work was also aimed at developing a methodology that
researchers can customize to process PCAP files as per their unique
requirements. This approach is designed to empower the research community to
create their own versions of datasets, grounded in a sound, transparent, and
adaptable data processing methodology.

##  V Methodology

In this sections, we delve into the specifics of our methodology, discussing
the rationale behind our configuration choices and the techniques employed to
ensure the integrity, accuracy, and relevance of our datasets and achieved
results.

###  V-A Raw Data Preprocessing

Our experimental approach began with downloading the raw packet trace files
from the CICIDS-2017 dataset. In line with the preprocessing steps outlined by
[4] [4], we undertook a similar strategy to minimize the undocumented (likely)
anomalies in the raw packet traces [4] that could potentially bias our
results.

Initially, we utilized the editcap command (editcap -D 10000 input.pcap
output.pcap) to remove duplicate packets from the acquired PCAP files. This
command, with a deduplication window set to 10,000 packets, effectively
identifies and eliminates duplicates, striking a balance between computational
efficiency and the thoroughness required for our analysis.

Following deduplication, the reordercap command (reordercap input.pcap
output.pcap) was applied to the cleaned traffic trace files. The purpose of
this step was to reorder the packets chronologically, thereby ensuring the
temporal accuracy of network events, which is crucial for a precise analysis
of network flow.

Table I provides an in-depth summary of the results obtained from executing
these commands on each raw packet trace file. It details the total count of
frames in the PCAP files before and after deduplication and highlights the
number of frames that were out of sequence pre- and post-reorder, along with
their respective percentage differences. Through these preprocessing steps, we
were able to refine the raw PCAP files effectively, laying a solid foundation
for our subsequent network traffic flow analysis.

###  V-B IP Flow Generation

For the critical task of processing the PCAP files, we employed NFStream [10],
our Python-based tool that is specifically engineered for rapid, flexible, and
expressive data handling in network analysis.

The core strength of NFStream lies in its hybrid design, which combines the
ease and accessibility of Python with the performance efficiency of C. This is
epitomized by the NFlow structure, a central component written in C, that
forms the backbone of the tool. The NFlow structure is designed to represent a
network flow and incorporates essential attributes like the five-tuple flow
key. Beyond these fundamental features, it is augmented with a rich set of
flow statistics and metadata, encompassing categories such as core features,
Layer 7 visibility, post-mortem statistics, and Sequence of Packet Length and
Time (SPLT) details. What makes NFStream particularly versatile is the ability
for users to selectively enable or disable these feature sets during the flow
measurement process, thereby tailoring the analysis to specific requirements.

Moreover, NFStream excels in bridging the gap between raw network measurements
and sophisticated data science analytics. Its intrinsic capabilities in flow
measurement and feature computation are complemented by an adaptable
architecture. This flexibility is particularly pronounced in the integration
of bespoke network functionalities through the NFPlugin component [10]. In our
study, this feature was instrumental, as it allowed us to integrate our
specific TCP flow expiration policy and flow labeling methodology directly
into the tool.

####  V-B1 Flow Measurement Specificities

In our research, we utilized NFStream v6.5.4 and primarily adhered to the
default settings recommended in the NFStream documentation. However, to align
the tool’s functionality with the specific requirements of our dataset and
research objectives, we customized several parameters as follows:

  * •

Decode Tunnels: We disabled the tunnel decoding feature because the
CICIDS-2017 PCAP files did not contain any tunneled traffic. This deactivation
ensured a streamlined analysis focused on the traffic types present in our
dataset.

  * •

BPF Filter: A custom Berkeley Packet Filter (BPF) was implemented to
exclusively capture TCP and UDP packets over IPv4. With this customization, we
achieved a more targeted packet filtering process, effectively excluding
traffic types of marginal volume.

  * •

Flow Timeouts: We adjusted the idle and active timeouts to 60 and 120 seconds,
respectively. This change from the default settings (120 and 1800 seconds) was
driven by the aim to more accurately replicate the network traffic flow
metering conditions in CICIDS-2017, thereby enhancing the realism, relevance
and comparability of our findings.

  * •

DPI-based Labeling: We set the ‘n_dissections’ value to 0, effectively
disabling Deep Packet Inspection (DPI)-based Layer 7 labeling. This adjustment
was made considering its limited utility in our Intrusion Detection System
(IDS) analysis.

  * •

Statistical Analysis: In deviation from the default configuration, we enabled
the statistical analysis feature. This decision was made to enrich our dataset
with detailed flow statistics, providing a more robust foundation for
comprehensive data evaluation.

####  V-B2 TCP Flow Expiration Policies

Our TCP flow expiration policy, implemented as an NFPlugin, incorporates two
key methods that govern the criteria for marking TCP flows as expired:

  * •

At flow initiation: Upon the initiation of a new network flow, our policy
dictates that if the incoming packet carries a TCP RST flag (indicating an
abrupt termination) or a TCP FIN flag (signifying a normal closure), the flow
is immediately marked for expiration regardless a timeout has been reached.
This approach ensures that our dataset accurately reflects abrupt or orderly
terminations of TCP connections from the outset.

  * •

At flow update: For existing network flows, our policy mandates immediate
marking for expiration, similarly, regardless a timeout has been reached, if a
packet carries an RST flag. Additionally, if a FIN flag is detected, the flow
is also marked for expiration. This method ensures that the dynamics of
ongoing connections are accurately captured, particularly in scenarios
involving unexpected interruptions or planned closures.

These TCP flow expiration strategies, while not encompassing every nuanced TCP
sequence anomaly present in raw traffic traces, are in line with established
methodologies used by various flow exporters, both open-source and commercial,
as referenced in literature [11]. This includes widely-used protocols like
IPFIX and NetFlow, which are integral to numerous vendor devices. This
alignment with common industry practices aims to boost the practical
applicability of our datasets.

To further enhance the relevance and accuracy of our dataset, we implemented
an additional refinement step after generating the data. In this step, we
specifically filter out and remove flow records that were terminated due to
their initial packet carrying an RST or FIN flag. Such flows, particularly
those with a first packet marked by a FIN flag, might represent connections
truncated due to our TCP flow expiration policy. Moreover, flows initiating
with an RST flag potentially indicate abnormal TCP operations, which could be
unrelated to the artificial attacks generated during the CICIDS-2017 packet
trace creation. By excluding these single-packet flows, our focus shifts to
more significant network connections, offering a deeper and more contextually
rich dataset for analysis.

####  V-B3 Attack Type Labelling

Leveraging the capabilities of NFPlugin, we developed a methodology for
accurately labeling each flow in our dataset in accordance with the specific
attack types documented in [9]. This approach involved creating a suite of
functions, each uniquely designed for one of the weekdays covered by the PCAP
files, from Monday to Friday.

For each day, the corresponding function employs a tailored algorithm that
scrutinizes the flow’s characteristics, such as source IP, destination IP,
destination port, and protocol. It then compares these attributes against the
documented attack profiles for that specific day, including the known time
windows of the attacks. Based on this comparison, each flow is labeled
accordingly, with tags such as ‘SSH-Patator’, ‘Web Attack - Brute Force’ or
marked as ‘BENIGN’ if it does not match any known attack patterns.

This labeling logic ensures a high degree of accuracy in distinguishing
between benign and malicious flows, reflecting the unique attack dynamics of
each day in the dataset. By applying this method consistently across the
different days, we ensure that our dataset is precisely annotated,
facilitating more accurate and meaningful analysis in subsequent stages of our
research.

##  VI Dataset Overview

###  VI-A Distribution of Benign and Anomalous Flows

Table II presents a comparative analysis of the distribution of the flow types
across the five CICIDS-2017 dataset versions examined in this study.

The CICIDS-2017 dataset is available in two variants. The version tailored for
ML research omits several columns, including Flow ID, Source IP, Source Port,
Destination IP, Protocol, and Timestamp, unlike its counterpart which includes
these essential fields. Given the significance of the protocol identifier in
model training, we opted for the more comprehensive version, despite its
inclusion of 288 602 flows with unknown labels. These unlabeled flows were
excluded from our analysis.

For the WTMC-2021 dataset, we disregarded flows marked as ‘Attempted’. As
noted by [2] [2], some flows in payload-reliant attack categories lack an
actual payload and are thus labeled as ‘Attempted’.

From the data in Table II, it is evident that the CICIDS-2017 and NFS-2023-TE
datasets exhibit a roughly similar count and distribution of flow records.
These datasets share a common approach in expiring flows upon encountering the
first FIN or RST TCP flag. Conversely, the WTMC-2021, CRiSIS-2022, and
NFS-2023-nTE datasets, which were generated without TCP FIN or RST flag-based
flow expiration, demonstrate a closely aligned flow count and distribution
pattern.

A noteworthy distinction emerged regarding the integrity of data fields in the
datasets. All datasets produced using CICFlowMeter, exhibited a small but
significant number of flow entries with NaN (Not a Number) values. This
occurrence of NaN values potentially indicates gaps or inconsistencies in the
flow metering methodology employed by CICFlowMeter. Such anomalies can have a
cascading effect on data quality and, consequently, on the performance and
reliability of ML models trained using these datasets. In contrast, the
datasets generated using NFStream demonstrated a higher level of data
completeness, with no instances of unfilled data fields.

In addition to the occurrence of NaN values, our analysis uncovered another
dimension of data integrity concerns in the datasets produced using
CICFlowMeter. A significant number of entries across various features, such as
‘Flow Duration’, ‘Flow Bytes/s’, ‘Flow Packets/s’, and several others,
contained negative values for each day from Monday to Friday. Such negative
values are not logically justifiable within the context of these network
traffic flow metrics and suggest the presence of errors in the dataset’s
measurement process. This observation further underscores the challenges
associated with ensuring data quality in the datasets derived from
CICFlowMeter.

Conversely, while the WTMC-2021 dataset shows an improvement with fewer
features containing negative numbers, it still presents this anomaly,
particularly from Monday to Thursday in the ‘Flow IAT Min’ feature and more
extensively on Friday. This indicates that although refinements have been
made, certain measurement inaccuracies persist. The CRiSIS-2022 dataset stands
out for its absence of such irregularities, indicating a more rigorous flow
measurement methodology or post-processing where such flow entries might have
been removed manually.

While the NFStream-generated datasets do contain entries with negative
numbers, these are intentionally designed to represent specific flow
conditions, such as custom flow expiration policies, rather than being
indicative of measurement errors. The negative values in NFStream datasets
are, therefore, a feature of the dataset’s structure and not a flaw.

Nonetheless, this comparison underscores the influence of flow expiration
policies on the structure and composition of network traffic datasets, thereby
potentially affecting the subsequent analysis and model training processes.

TABLE II: Flow type distribution overview per dataset version

Day | Dataset | Total | Benign | Anomaly | NaN | Negative
---|---|---|---|---|---|---
Mon | CICIDS-2017 | 529 918 | 529 918 | — | 64 | 299 597
WTMC-2021 | 371 749 | 371 749 | 47 | 517
CRiSIS-2022 | 372 425 | 372 425 | 2 492 | 0
NFS-2023-nTE | 376 826 | 376 826 | 0 | ‡
NFS-2023-TE | 559 775 | 559 775 | 0 | ‡
Tue | CICIDS-2017 | 445 909 | 432 074 | 13 835 | 201 | 257 104
WTMC-2021 † | 321 984 | 315 031 | 6 953 | 23 | 511
CRiSIS-2022 | 322 462 | 315 509 | 6 953 | 2 336 | 0
NFS-2023-nTE | 325 677 | 318 705 | 6 972 | 0 | ‡
NFS-2023-TE | 467 254 | 456 311 | 10 943 | 0 | ‡
Wed | CICIDS-2017 | 692 703 | 440 031 | 252 672 | 1 008 | 336 224
WTMC-2021 † | 491 006 | 319 216 | 171 790 | 29 | 758
CRiSIS-2022 | 497 115 | 325 324 | 171 791 | 2 281 | 0
NFS-2023-nTE | 503 140 | 329 267 | 173 873 | 0 | ‡
NFS-2023-TE | 865 413 | 691 015 | 174 398 | 0 | ‡
Thu | CICIDS-2017 * | 458 968 | 456 752 | 2 216 | 38 | 267 336
WTMC-2021 † | 360 486 | 360 264 | 222 | 288 | 621
CRiSIS-2022 | 355 618 | 291 211 | 64 407 | 33 287 | 0
NFS-2023-nTE | 358 302 | 294 128 | 64 174 | 0 | ‡
NFS-2023-TE | 485 224 | 413 744 | 71 480 | 0 | ‡
Fri | CICIDS-2017 | 703 245 | 414 322 | 288 923 | 47 | 284 166
WTMC-2021 † | 546 445 | 291 433 | 255 012 | 348 | 515
CRiSIS-2022 | 548 828 | 293 367 | 255 461 | 3 011 | 0
NFS-2023-nTE | 547 186 | 293 341 | 253 845 | 0 | ‡
NFS-2023-TE | 775 694 | 518 961 | 256 733 | 0 | ‡

  * *

Contains 288 602 flows with NaN labels whose count is not included.

  * †

Contains attempted attack types whose count is not included.

  * ‡

Contains negative numbers but these denote specific flow conditions.

###  VI-B Distribution of Specific Anomaly Types

Building on the insights from Table II, Table III delves deeper into the
distribution of specific anomaly types across the five versions of the
CICIDS-2017 dataset. The data presented in this table excludes the NaN entries
detailed in Table II.

TABLE III: Anomaly breakdown DS | Anomaly Type | CICIDS-2017 | WTMC-2021 | CRiSIS-2022 | NFS-2023-nTE | NFS-2023-TE
---|---|---|---|---|---|---
Mon | — | — | — | — | — | —
Tue | FTP-Patator | 7 938 | 3 973 | 3 973 | 3 992 | 7 963
SSH-Patator | 5 897 | 2 980 | 2 980 | 2 980 | 2 980
Wed | DoS GoldenEye | 10 293 | 7 567 | 7 567 | 7 916 | 7 917
DoS Hulk | 231 073 | 158 469 | 158 470 | 158 027 | 158 546
DoS Slowhttptest | 5 499 | 1 742 | 1 742 | 2 727 | 2 732
DoS Slowloris | 5 796 | 4 001 | 4 001 | 5 192 | 5 192
Heartbleed | 11 | 11 | 11 | 11 | 11
Thu | Infiltration | 36 | 32 | 32 | 27 | 28
PortScan | – | – | 64 185 | 63 957 | 71 262
Web Attack - Brute Force | 1 507 | 151 | 151 | 151 | 151
Web Attack - SQL Injection | 21 | 12 | 12 | 12 | 12
Web Attack - XSS | 652 | 27 | 27 | 27 | 27
Fri | Bot | 1 966 | 738 | 738 | 738 | 738
DDoS | 128 027 | 95 123 | 95 144 | 93 178 | 95 685
PortScan | 158 930 | 159 151 | 159 579 | 159 929 | 160 310
TABLE IV: Number of flows more than two with FIN and RST flags and their distribution between benign and anomalous traffic types Day | Dataset | Flows with FIN $>2$ | Flows with RST $>2$
---|---|---|---
|  | Total | Benign | Anomaly | Total | Benign | Anomaly
Mon | CICIDS-2017 | 0 | 0 | 0 | 0 | 0 | 0
WTMC-2021 | 18 703 | 18 703 | 0 | 9 860 | 9 860 | 0
CRiSIS-2022 | 16 477 | 16 477 | 0 | 9 657 | 9 657 | 0
NFS-2023-nTE | 16 476 | 16 476 | 0 | 9 785 | 9 785 | 0
NFS-2023-TE | 0 | 0 | 0 | 0 | 0 | 0
Tue | CICIDS-2017 | 0 | 0 | 0 | 0 | 0 | 0
WTMC-2021 | 11 737 | 11 690 | 47 | 4 396 | 4 383 | 13
CRiSIS-2022 | 9 387 | 9 340 | 47 | 4 152 | 4 139 | 13
NFS-2023-nTE | 9 386 | 9 339 | 47 | 4 217 | 4 204 | 13
NFS-2023-TE | 0 | 0 | 0 | 0 | 0 | 0
Wed | CICIDS-2017 | 0 | 0 | 0 | 0 | 0 | 0
WTMC-2021 | 10 952 | 5 692 | 5 260 | 40 779 | 2 512 | 38 267
CRiSIS-2022 | 8 566 | 3 305 | 5 261 | 40 577 | 2 309 | 38 268
NFS-2023-nTE | 8 827 | 3 295 | 5 532 | 40 832 | 2 380 | 38 452
NFS-2023-TE | 0 | 0 | 0 | 0 | 0 | 0
Thu | CICIDS-2017 | 0 | 0 | 0 | 0 | 0 | 0
WTMC-2021 | 5 272 | 5 272 | 0 | 2 076 | 2 076 | 0
CRiSIS-2022 | 2 231 | 2 231 | 0 | 1 884 | 1 884 | 0
NFS-2023-nTE | 2 230 | 2 230 | 0 | 2 028 | 2 028 | 0
NFS-2023-TE | 0 | 0 | 0 | 0 | 0 | 0
Fri | CICIDS-2017 | 0 | 0 | 0 | 0 | 0 | 0
WTMC-2021 | 4 877 | 4 826 | 46 | 2 434 | 2 425 | 9
CRiSIS-2022 | 2 624 | 2 578 | 46 | 2 210 | 2 201 | 9
NFS-2023-nTE | 3 735 | 2 554 | 1 181 | 3 621 | 3 608 | 13
NFS-2023-TE | 0 | 0 | 0 | 0 | 0 | 0

An important observation from Table III is the alignment in value counts
between the two refined versions and those measured using NFStream, when
compared to the original CICIDS-2017 dataset. This alignment suggests that the
updates and patches applied to CICFlowMeter have likely contributed to a more
accurate measurement and labeling of anomalies, bringing these metrics closer
to the results observed with NFStream.

Another notable observation from Table III is the effective identification and
labeling of PortScan attacks in the CRiSIS and NFStream versions of the
dataset. These attacks were previously unnoticed in the original CICIDS-2017
dataset and its WMTC-2021 refinement. The enhanced accuracy in measuring and
labelling PortScan is a critical advancement for network security research,
thereby offering a more robust foundation for training and evaluating ML
models in the context of anomaly detection.

###  VI-C Reconstructing TCP FIN and RST Flag Counts

Table IV provides an insightful comparison of the counts of flows with three
or more TCP FIN and RST flags across the five studied versions of the
CICIDS-2017 dataset. This analysis is crucial for verifying the impact of TCP
flag-based flow expiration policies on the characteristics of the generated
flow records.

The enhancement of TCP flag-based flow expiration in WMTC-2021 was intended to
align flow records more closely with standard TCP operational behaviors.
Ideally, this would result in flows being expired after a bidirectional
exchange of packets with FIN flags, or immediately upon detecting a packet
with an RST flag. However, our analysis reveals that the CICFlowMeter version
used for WMTC-2021 did not fully achieve this objective. Contrary to
expectations, both the WMTC-2021 and CRiSIS-2022 datasets include flow records
with anomalously high counts of FIN or RST flags, sometimes exceeding 100
flags per a single flow in extreme cases.

Our preliminary examination of the CICFlowMeter’s patched version revealed
that the abnormally high TCP flag counts are likely due to how flow expiration
is handled. In the patched CICFlowMeter, TCP flows are designated for
expiration upon detecting either a TCP RST flag or a sequence of two TCP FIN
flags followed by a TCP ACK flag. However, even after a flow is marked for
expiration, its attributes continue to be updated until it is eventually
removed from the flow cache. This removal occurs either due to the active
timeout (hard coded 120 seconds) being exceeded or when a TCP SYN flag is
encountered in a flow already marked for expiration. In the latter case, a new
flow record is initiated with the packet containing the SYN flag, and the
previous flow is discarded. The result is flow records with significantly high
counts of these flags, making their interpretability ambiguous and, at times,
misleading with respect to standard TCP operation.

Our measurement with NFStream revealed differences in dataset characteristics
based on whether TCP compliant expiration policy was enabled or disabled. When
we activated the TCP expiration policy in NFStream, the resulting dataset
(NFS-2023-nTE) aligned well with the expected standard TCP behavior. The flows
were terminated in a manner consistent with typical TCP connection closures,
leading to a more realistic representation of network traffic. However, this
version diverged from the flow characteristics observed in the WMTC-2021 and
CRiSIS-2022 datasets, particularly in terms of the counts of TCP FIN and RST
flags.

Conversely, when the TCP expiration policy was disabled, the resulting dataset
(NFS-2023-nTE) showed similarities to the flow characteristics observed in
WMTC-2021 and CRiSIS-2022, as evident from Table IV. Given this alignment, we
chose to produce the NFS-2023-nTE dataset without enabling the TCP flag-based
expiration policy. This decision was pivotal in ensuring that the results
obtained using NFS-2023-nTE could be more effectively contrasted with those
derived from existing datasets.

While not directly evident from Table IV, in the NFS-2023-TE dataset—which was
generated without activating the TCP flag-based expiration policy—there were
no flows with more than two counts of TCP FIN or RST flags. In fact, these
counts were limited to a maximum of one per flow, perfectly aligning with the
expected behavior outlined in our TCP expiration policy described in Section
V-B2.

Nonetheless, the discrepancy between the expected and observed behaviors in
flow records with respect to TCP FIN and RST flags underscores the importance
of accurate implementation of network protocols in flow metering tools. It
also highlights the effectiveness of NFStream in adhering to TCP standards
when the appropriate expiration policy is activated.

##  VII ML Performance Comparison

###  VII-A ML Algorithm

In our study, we explore the impact of data integrity on network anomaly
detection by employing Random Forest (RF), a widely recognized and frequently
used machine learning technique. RF is an ensemble learning method that
operates by constructing a multitude of decision trees at training time. It
outputs the mode of the classes (classification) of the individual trees,
which enhances the overall predictive accuracy and controls over-fitting.

The rationale behind this selection is that RF have been consistently utilized
in the literature related to the CICIDS-2017, WMTC-2021, and CRiSIS-2022
datasets. For instance, CICIDS-2017 employed the K-Nearest Neighbors (KNN),
RF, Iterative Dichotomiser 3 (ID3), Adaboost, Multilayer Perceptron (MLP), NB,
and Quadratic Discriminant Analysis (QDA) algorithms [9]; WMTC-2021 focused
solely on RF [2]; and CRiSIS-2022 assessed Support Vector Machine (SVM), NB,
RF, and DT [4]. Choosing the RF algorithm ensures that we evaluate a method
previously employed in related research, thereby enhancing the comparability
and transferability of our observations across these studies.

###  VII-B Feature Selection

In our comparative study of RF model performances using different versions of
the CICIDS-2017 dataset, a crucial step is the mapping of flow features
generated by CICFlowMeter and NFStream. While both tools serve similar
purposes in network traffic analysis, they generate a range of flow features
with some overlaps and some unique aspects666For a comprehensive list of
features from CICFlowMeter, see
https://github.com/ahlashkari/CICFlowMeter/blob/master/ReadMe.txt,777For
NFStream features, refer to https://www.nfstream.org/docs/api. Our focus is on
identifying features that are common across both tools to ensure a fair and
balanced comparison. Table V details this mapping, highlighting the five-tuple
(five flow key attributes) and 41 flow features that can be directly compared
or have closely analogous counterparts. By concentrating on a consistent set
of common flow features, we aim to provide a logical and well-grounded basis
for comparing the effectiveness of methodologies developed using the datasets
derived from these two tools.

TABLE V: Flow feature mapping between NFStream and CICFlowMeter

NFStream Feature | CICFlowMeter Feature
---|---
src_ip | Source IP †
src_port | Source Port †
dst_ip | Destination IP †
dst_port | Destination Port †
protocol | Protocol
src2dst_packets | Total Fwd Packet †
dst2src_packets | Total Bwd packets †
src2dst_bytes | Total Length of Fwd Packet †
dst2src_bytes | Total Length of Bwd Packet †
bidirectional_duration_ms | Flow Duration
bidirectional_min_ps | Packet Length Min †
bidirectional_max_ps | Packet Length Max †
bidirectional_mean_ps | Packet Length Mean
bidirectional_stddev_ps | Packet Length Std
src2dst_max_ps | Fwd Packet Length Max
src2dst_min_ps | Fwd Packet Length Min
src2dst_mean_ps | Fwd Packet Length Mean
src2dst_stddev_ps | Fwd Packet Length Std
dst2src_max_ps | Bwd Packet Length Max
dst2src_min_ps | Bwd Packet Length Min
dst2src_mean_ps | Bwd Packet Length Mean
dst2src_stddev_ps | Bwd Packet Length Std
bidirectional_mean_piat_ms | Flow IAT Mean
bidirectional_stddev_piat_ms | Flow IAT Std
bidirectional_max_piat_ms | Flow IAT Max
bidirectional_min_piat_ms | Flow IAT Min
src2dst_mean_piat_ms | Fwd IAT Mean
src2dst_stddev_piat_ms | Fwd IAT Std
src2dst_max_piat_ms | Fwd IAT Max
src2dst_min_piat_ms | Fwd IAT Min
dst2src_mean_piat_ms | Bwd IAT Mean
dst2src_stddev_piat_ms | Bwd IAT Std
dst2src_max_piat_ms | Bwd IAT Max
dst2src_min_piat_ms | Bwd IAT Min
bidirectional_fin_packets | FIN Flag Count
bidirectional_syn_packets | SYN Flag Count
bidirectional_rst_packets | RST Flag Count
bidirectional_psh_packets | PSH Flag Count
bidirectional_ack_packets | ACK Flag Count
bidirectional_urg_packets | URG Flag Count
bidirectional_cwr_packets | CWR Flag Count †
bidirectional_ece_packets | ECE Flag Count
src2dst_psh_packets | Fwd PSH Flags
dst2src_psh_packets | Bwd PSH Flags
src2dst_urg_packets | Fwd URG Flags
dst2src_urg_packets | Bwd URG Flags

  * *

There is an inconsistency in the naming conventions used across the original
CICIDS-2017 dataset and its subsequent refinements, WTMC-2021 and CRiSIS-2022.

While feature selection methods could have been employed to reduce the
dimensionality of data, we opted to utilize the full feature set provided by
the CICIDS-2017 dataset, with specific exclusions aimed at minimizing
potential biases. Specifically, the source and destination IP addresses, as
well as port numbers flow key attributes were excluded from our feature set.
The rationale behind this decision was to avoid biasing the model with
information that could inadvertently skew its learning process. For instance,
IP addresses present a challenge in terms of transforming them into a format
that is effectively utilizable by ML models. On the other hand, including
destination port numbers in the feature set could lead the model to associate
certain port numbers (e.g., port 21 for FTP or port 22 for SSH) with specific
types of attacks (e.g., FTP-Patator or SSH-Patator). This reliance on port
numbers could detract from the model’s ability to learn and identify attack
patterns based on the intrinsic characteristics of the network flows.

The protocol identifier attribute, however, was retained as part of our
feature set. In the CICIDS-2017 dataset, all attacks were executed over the
TCP protocol, rendering the inclusion of the protocol identifier somewhat less
impactful. Nevertheless, considering that TCP flags were also included in our
feature set, we found no compelling reason to exclude the protocol identifier.
Its inclusion might still offer marginal benefits in understanding flow
characteristics, especially when combined with other features.

It is worth to note that, even for features that appear similar, discrepancies
in their calculated values may occur, as also evident in Tables II, III and
IV. These differences can arise due to variations in how the features are
computed and the specific mechanisms employed for updating and terminating
flows. Understanding the reasons behind these discrepancies is not within the
scope of our current work. A more thorough investigation would be required to
fully comprehend these variations and their root causes.

###  VII-C Metrics

In evaluating the performance of the RF algorithm on various dataset versions,
we used a comprehensive set of metrics for both binary and multi-class
classification scenarios. For binary classification, we consolidated all
attack types under a singular “ANOMALY” label, whereas, for multi-class
classification, we retained the original labels for individual attack types.

The key metrics employed include precision (the ratio of correctly predicted
positive observations to the total predicted positive observations), recall
(the ratio of correctly predicted positive observations to all actual positive
observations, also known as true positive rate or sensitivity), accuracy (the
ratio of correctly predicted observations to total observations), F1 score
(the weighted average of precision and recall), and Receiver Operating
Characteristic (ROC) - Area Under Curve (AUC) (measuring the model’s ability
to distinguish between classes). In addition to the metrics mentioned above,
we also calculate normalized confusion matrices for each model.

###  VII-D Binary Classification Results

Table VI presents a comparative analysis of the performance metrics for each
dataset in our binary classification setup. The table shows the values for
precision (Prec.), recall (Rec.), accuracy (Acc.), F1 score (F1), and Area
Under the Receiver Operating Characteristic Curve (AUC). Key observations from
Table VI are as follows:

TABLE VI: Performance Metrics Comparison Across Datasets and Days for Binary Classification Day | Dataset |  Binary Classification |  |  |  |
---|---|---|---|---|---|---
|  |  Prec. | Rec. | Acc. | F1 | AUC
Tue | CICIDS-2017 |  0.9984 | 0.7723 | 0.9929 | 0.8709 | 0.8861
| WTMC-2021 |  1.0000 | 1.0000 | 1.0000 | 1.0000 | 1.0000
| CRiSIS-2022 |  1.0000 | 0.9986 | 1.0000 | 0.9993 | 0.9993
| NFS-2023-nTE |  1.0000 | 0.9986 | 1.0000 | 0.9993 | 0.9993
| NFS-2023-TE |  0.9991 | 0.9997 | 1.0000 | 0.9994 | 0.9998
Wed | CICIDS-2017 |  0.9792 | 0.9958 | 0.9908 | 0.9874 | 0.9918
| WTMC-2021 |  0.9999 | 0.9998 | 0.9999 | 0.9999 | 0.9999
| CRiSIS-2022 |  1.0000 | 0.9999 | 1.0000 | 1.0000 | 1.0000
| NFS-2023-nTE |  1.0000 | 0.9997 | 0.9999 | 0.9998 | 0.9999
| NFS-2023-TE |  0.9998 | 0.9995 | 0.9998 | 0.9996 | 0.9997
Thu | CICIDS-2017 |  0.9966 | 0.8902 | 0.9995 | 0.9404 | 0.9451
| WTMC-2021 |  1.0000 | 0.8806 | 0.9999 | 0.9365 | 0.9403
| CRiSIS-2022 |  0.9685 | 0.9987 | 0.9965 | 0.9834 | 0.9975
| NFS-2023-nTE |  0.9834 | 0.9995 | 0.9969 | 0.9914 | 0.9979
| NFS-2023-TE |  0.9840 | 0.9027 | 0.9835 | 0.9416 | 0.9501
Fri | CICIDS-2017 |  0.9992 | 0.9962 | 0.9981 | 0.9977 | 0.9978
| WTMC-2021 |  0.9976 | 0.9997 | 0.9987 | 0.9986 | 0.9988
| CRiSIS-2022 |  0.9978 | 0.9997 | 0.9988 | 0.9987 | 0.9989
| NFS-2023-nTE |  1.0000 | 0.9999 | 1.0000 | 1.0000 | 1.0000
| NFS-2023-TE |  1.0000 | 0.9993 | 0.9998 | 0.9996 | 0.9996
TABLE VII: Day-wise and Dataset-wise Confusion Matrix Comparison for Binary Classification | Tuesday | Wednesday | Thursday | Friday
---|---|---|---|---
CICIDS-2017 |  |  |  |
WTMC-2021 |  |  |  |
CRiSIS-2022 |  |  |  |
NFS-2023-nTE |  |  |  |
NFS-2023-TE |  |  |  |

  * •

Both WTMC-2021 and CRiSIS-2022 datasets consistently show near-perfect scores
across all metrics. This indicates exceptional model performance in correctly
classifying flows, with minimal misclassifications.

  * •

The CICIDS-2017 dataset exhibits variability in performance across different
days, with particularly lower recall and F1 scores on Tuesday and Thursday.
This suggests some challenges in model generalization when using this dataset.

  * •

An interesting pattern emerges with the Thursday datasets. While accuracy is
consistently high across all versions, there are noticeable variations in the
other metrics. CICIDS-2017, WTMC-2021, and NFS-2023-TE demonstrate lower
recall, F1 scores, and ROC AUC, indicating some challenges in consistently
identifying true positives. In contrast, CRiSIS-2022 and NFS-2023-nTE achieve
higher metrics, reflecting a more balanced detection capability. This suggests
that the Thursday dataset may have unique characteristics influencing the
models’ predictive performance.

  * •

NFS-2023-nTE and NFS-2023-TE datasets generally align closely with the refined
datasets (WTMC-2021 and CRiSIS-2022) in terms of performance metrics. This
indicates that the methodological improvements in CICFlowMeter are effective.

  * •

The NFS-2023-TE dataset, particularly on Friday, shows high scores in all
metrics, aligning closely with the NFS-2023-nTE dataset. This suggests that
the TCP flag-based expiration policy in NFS-2023-TE does not adversely affect
the model’s ability to classify anomalies accurately.

  * •

The NFS-2023-nTE dataset demonstrates excellent precision and recall,
particularly on Wednesday and Friday, indicating its robustness in identifying
true positives without increasing false positives.

Complementary to the performance metrics detailed in Table VI, the confusion
matrices in Table VII provide additional insights into the classification
patterns of each model. The key findings that can be observed:

  * •

The confusion matrices across all datasets for the days of Tuesday (with an
exception for CICIDS-2017), Wednesday, and Friday show that the True Positive
Rates (the recall for the anomaly class) are exceedingly high, often reaching
perfect or near-perfect scores. This suggests that the models are highly
effective at identifying anomalous flows on these days.

  * •

Unlike other days, the Thursday matrices reveal a notably lower True Positive
Rate for anomaly detection in certain datasets, such as CICIDS-2017,
WTMC-2021, and NFS-2023-TE. This indicates that the models struggle to
consistently identify anomalies on this day, which may be due to the unique
characteristics or distribution of the Thursday data.

  * •

On Tuesday and Wednesday, the models using the WTMC-2021, CRiSIS-2022, and
NFS-2023-nTE datasets display perfect classification for benign flows, with
zero false positives. This demonstrates the models’ strong ability to
correctly identify normal behavior without mistakenly flagging it as
anomalous.

  * •

There is a notable consistency in the performance of the CRiSIS-2022 and
NFS-2023-nTE datasets across all days, as evidenced by the high scores for
both benign and anomaly classes, indicating stable model performance
regardless of the day.

  * •

The NFS-2023-nTE dataset stands out on Friday with perfect scores, reflecting
the model’s accuracy in distinguishing between benign and anomalous flows
without error.

###  VII-E Multi-class Classification Results

The comparative analysis of multi-class classification performance across
different versions of the CICIDS-2017 dataset is depicted in Table VIII.
Notable observations from the table include:

  * •

The WTMC-2021, CRiSIS-2022, NFS-2023-nTE, and NFS-2023-TE datasets exhibit
near-perfect or perfect scores across all metrics on Tuesday, Wednesday, and
Friday, indicating an exceptional ability of the models to classify multiple
attack types with high precision and reliability.

  * •

The original CICIDS-2017 dataset shows slightly lower performance compared to
the other datasets, particularly on Tuesday and Wednesday, which could suggest
some limitations in the dataset’s consistency or potential issues with class
imbalances affecting model performance.

  * •

Both NFS-2023-nTE and NFS-2023-TE datasets achieve perfect scores in all
metrics on Tuesday and Friday, and near-perfect scores on Wednesday and
Thursday. This indicates that the datasets generated using NFStream, with and
without TCP flag-based flow expiration, provide robust bases for model
training and evaluation.

  * •

On Thursday, there is a noticeable performance drop for the CICIDS-2017 and
NFS-2023-TE datasets compared to other days and datasets. This suggests some
difficulty in distinguishing between classes on this particular day.

  * •

Across all datasets and days, the models maintain high accuracy levels,
demonstrating their effectiveness in correctly classifying both benign and
anomalous flows in a multi-class setting.

TABLE VIII: Performance Metrics Comparison Across Datasets and Days for Multi-class Classification Day | Dataset |  Multi-class Classification |  |  |  |
---|---|---|---|---|---|---
|  |  Prec. | Rec. | Acc. | F1 | AUC
Tue | CICIDS-2017 |  0.9935 | 0.9935 | 0.9935 | 0.9925 | 0.9949
| WTMC-2021 |  0.9999 | 0.9999 | 0.9999 | 0.9999 | 1.0000
| CRiSIS-2022 |  0.9999 | 0.9999 | 0.9999 | 0.9999 | 0.9998
| NFS-2023-nTE |  1.0000 | 1.0000 | 1.0000 | 1.0000 | 1.0000
| NFS-2023-TE |  1.0000 | 1.0000 | 1.0000 | 1.0000 | 1.0000
Wed | CICIDS-2017 |  0.9907 | 0.9906 | 0.9906 | 0.9906 | 0.9989
| WTMC-2021 |  0.9999 | 0.9999 | 0.9999 | 0.9999 | 1.0000
| CRiSIS-2022 |  0.9999 | 0.9999 | 0.9999 | 0.9999 | 1.0000
| NFS-2023-nTE |  0.9997 | 0.9997 | 0.9997 | 0.9997 | 1.0000
| NFS-2023-TE |  0.9997 | 0.9997 | 0.9997 | 0.9997 | 0.9999
Thu | CICIDS-2017 |  0.9979 | 0.9981 | 0.9981 | 0.9979 | 0.9840
| WTMC-2021 |  0.9999 | 0.9999 | 0.9999 | 0.9999 | 0.9925
| CRiSIS-2022 |  0.9966 | 0.9965 | 0.9965 | 0.9965 | 0.9986
| NFS-2023-nTE |  0.9970 | 0.9969 | 0.9969 | 0.9969 | 0.9990
| NFS-2023-TE |  0.9839 | 0.9839 | 0.9839 | 0.9836 | 0.9954
Fri | CICIDS-2017 |  0.9977 | 0.9979 | 0.9979 | 0.9977 | 0.9999
| WTMC-2021 |  0.9987 | 0.9987 | 0.9987 | 0.9987 | 0.9998
| CRiSIS-2022 |  0.9988 | 0.9988 | 0.9988 | 0.9988 | 0.9998
| NFS-2023-nTE |  1.0000 | 1.0000 | 1.0000 | 1.0000 | 1.0000
| NFS-2023-TE |  0.9998 | 0.9998 | 0.9998 | 0.9998 | 1.0000

Complementary to the performance metrics in Table VIII, the confusion matrices
in Table IX shed light on the classification performance for individual attack
types across the datasets on different days. The matrices reveal several
noteworthy patterns:

  * •

Across all datasets, certain attack types such as FTP-Patator and SSH-Patator
on Tuesday and Infiltration on Wednesday are classified with high precision.
This indicates that the models are particularly effective at detecting these
types of attacks with minimal false positives.

  * •

There is some variation in the recall rates for different attack types. For
example, on Thursday, the recall for DoS attacks is noticeably lower in the
CICIDS-2017 dataset compared to the others. This variation suggests that some
datasets or models may be better tuned to recognize specific attacks over
others.

  * •

The NFS-2023-nTE and NFS-2023-TE datasets demonstrate consistently high
precision and recall across all attack types. This uniformity highlights the
robustness of the NFStream tool in generating datasets that lead to reliable
classification models.

  * •

Similar to the findings in Tables VI, VII and VIII, the confusion matrices for
Thursday reveal some challenges in classifying attacks, particularly for the
NFS-2023-TE dataset. The NFS-2023-TE dataset displays slightly less precision
and recall for certain attack types, aligning with the earlier observation of
lower performances on this day.

  * •

Despite some day-to-day and attack-type-specific variations, the overall
accuracy across all datasets remains high. This suggests that the RF models
are generally well-calibrated to distinguish between benign traffic and
various types of network anomalies.

TABLE IX: Day-wise and Dataset-wise Confusion Matrix Comparison for Multi-class Classification | Tuesday | Wednesday | Thursday | Friday
---|---|---|---|---
CICIDS-2017 |  |  |  |
WTMC-2021 |  |  |  |
CRiSIS-2022 |  |  |  |
NFS-2023-nTE |  |  |  |
NFS-2023-TE |  |  |  |

##  VIII Discussion

###  VIII-A Implications of Dataset Quality on RF Model Efficacy

Upon analyzing the results from both binary and multi-class classification
tables, it is evident that the RF models exhibited robust performance across
various dataset versions, demonstrating high accuracy and precision in both
binary and multi-class classification scenarios. This suggests that while
dataset refinement and remeasurement can lead to improvements, even datasets
with known biases and measurement issues can yield models with impressive
performance.

However, the consistently high performance across datasets raises a critical
point of discussion. It illustrates the adaptability of ML algorithms,
particularly Random Forest, to learn from data with varying quality levels.
Yet, it also suggests a potential risk: that high performance may mask
underlying dataset flaws. While the refinements and NFStream-generated
datasets do show some improvements, the significance of these improvements is
less pronounced than one might expect, given the already high performance of
models trained on the original dataset.

Another observation is that our findings do not corroborate the substantial
performance improvements reported in WTMC-2021 [2] and CRiSIS-2022 [4],
particularly when these are juxtaposed with the original performance metrics
of CICIDS-2017 [1]. However, these discrepancies may stem from the intricacies
involved in applying machine learning algorithms. Further study is necessary
to more comprehensively understand the underlying causes of these
discrepancies.

###  VIII-B Feature Importance

A deeper examination of feature importance for each model result revealed a
particularly critical finding: TCP FIN and RST flags generally do not emerge
as dominant features influencing model outcomes. This trend is consistent
across various dataset versions, with limited exceptions. Such a finding
provides clarity on the model’s robust performance, suggesting that the
implementation of TCP expiration policies has a marginal impact on the Random
Forest’s predictive capabilities. The model appears to leverage a broader
spectrum of features to discern between benign and anomalous traffic
effectively.

Friday datasets stand out, with TCP FIN and RST flags assuming greater
importance. Yet, even with this prominence, the Random Forest classifier
retains its high accuracy, underscoring the algorithm’s capacity to extract
relevant patterns from complex data structures. This adaptability is crucial,
demonstrating the classifier’s resilience to potential inaccuracies in
specific feature measurements. The specific details of feature importance have
been made available as a digital artifact [12, 13] for further review.

###  VIII-C Generalizability

We also evaluated the performance using Decision Tree (DT) and Naive Bayes
(NB) algorithms. These were benchmarked against the RF model, using the top 15
features selected by the extremely randomized trees (Extra-Trees) algorithm
[14] for model training. While RF was the best-performing algorithm across the
datasets, DT closely followed in performance, demonstrating the capability to
effectively handle the classification task. In contrast, NB showed
considerable performance variability across different dataset versions and
days, indicating a sensitivity to the dataset nuances.

Despite these differences, the overarching trends observed for RF—regarding
the impact of feature selection and model robustness—were also apparent in the
performances of DT and NB. This cross-algorithm consistency strengthens the
validity of our findings and affirms that the high performance is not an
artifact of a particular modeling approach but is indicative of the features’
predictive power. The detailed comparative analysis for the RF, DT, and NB
models across the CICIDS-2017, WTMC-2021, CRiSIS-2022, NFS-2023-nTE, and
NFS-2023-TE datasets is accessible via the digital artifacts provided [15, 16,
17, 18, 19].

###  VIII-D Conclusive Insights

The examination of the results leads us to a nuanced understanding. On one
hand, the resilience of ML models to dataset imperfections is encouraging, as
it demonstrates their ability to capture underlying patterns of anomalous
behavior effectively. On the other hand, it underscores the critical need for
vigilance in dataset creation. Refined measurement techniques are essential,
not because they drastically change model performance metrics, but because
they ensure that the models are learning from data that accurately represents
real-world conditions. This is particularly crucial when deploying these
models in operational settings, where the consequences of misclassification
can be significant.

In summary, our comprehensive reflection underscores the complexity of
evaluating dataset quality solely based on model performance metrics. It
highlights the importance of continuously striving for dataset accuracy and
sound methodology to ensure that high model performance corresponds with true
predictive power in real-world scenarios.

##  IX Limitations and Future Prospects

###  IX-A Real-time Applicability of the Datasets

A fundamental limitation inherent in all versions of the CICIDS-2017 dataset
is their constrained utility in real-world, especially real-time anomaly
detection scenarios. The core issue with these datasets stems from the manner
in which the flow records were generated. Predominantly, the tools were
configured to produce comprehensive flow records encapsulating the complete
lifecycle of network flows. While this methodology ensures detailed record-
keeping, it falls short of mirroring the real-time flow characteristics that
are pivotal in dynamic network environments. Essential aspects like flow size,
packet count, or duration thresholds, which are critical in simulating real-
world traffic patterns, are not adequately represented.

This limitation manifests in the datasets’ inability to offer a realistic
portrayal of the partial and evolving nature of network flow information,
which is crucial for real-time monitoring and anomaly detection. In practical
scenarios, anomaly detection systems often rely on incomplete or evolving flow
data to function effectively and adaptively. The complete flow records in the
datasets may not, therefore, offer an accurate reflection of how ML models
perform in such time-sensitive settings. This discrepancy poses significant
challenges to the transferability of research findings based on these datasets
to practical, real-time detection systems. The research community must be
cautious in extrapolating the results derived from these datasets to real-
world applications, as the underlying data may not accurately simulate the
conditions encountered in real-time network environments.

While NFStream offers a capability to meter flows up to specific thresholds,
this aspect was not explored within the scope of the current study. Future
research, therefore, should focus on utilizing NFStream or similar tools to
create datasets that better reflect the partial flow information
characteristic of real-time network environments. Such datasets would be
invaluable in evaluating the effectiveness of ML models under conditions that
demand swift and accurate anomaly detection using incomplete data. This
direction of research promises to bridge the gap in understanding the
applicability and performance of ML models in real-time anomaly detection
scenarios, a critical step towards enhancing the practical relevance of
anomaly detection methodologies.

###  IX-B Application of Sophisticated Machine Learning

Our research has been concentrated on the analysis of single flows, without
delving into the interdependencies that might exist between different flow
samples. This specific focus has made supervised learning techniques, RF, DT,
and NB, more suitable for our study, in comparison to more complex methods
like unsupervised learning or advanced models such as Isolation Forest (IF)
and Long Short-Term Memory (LSTM) networks.

In our approach, we utilized features that represent the externally observable
characteristics of network flows, predominantly derived from packet
aggregation. A critical observation here is the possibility that certain
attack signatures might not show substantial deviations from typical flow
statistics when analyzed individually. These signatures could instead be more
indicative of the nature of specific attacks, aspects that are not easily
identifiable from external flow characteristics alone. As a result, while IF
is known for its proficiency in outlier detection, its effectiveness in
distinguishing between certain types of anomalies and normal flows might be
limited. This limitation is particularly pronounced when the anomalies in
question do not manifest as distinct outliers in the statistical analysis of
individual flows.

Furthermore, the application of LSTM networks, which are inherently designed
for time-series data, encounters unique challenges in the context of network
flow records. The fundamental nature of network flows, with their varied start
and end times, durations, and intervals, does not align perfectly with typical
time-series characteristics. This discrepancy, further exacerbated by sporadic
traffic bursts, complicates the task of organizing these flows into coherent
time windows or extracting features representative of such patterns. The
complexity involved in processing and analyzing network flows in a manner
suitable for LSTM networks demands a different methodological approach that
was beyond the scope of our current examination.

Given these considerations, a further area for future research lies in
exploring and developing algorithms that can effectively process and analyze
network flows, taking into account their unique characteristics and
interdependencies. Such advancements are required to enhance the applicability
and effectiveness of machine learning techniques in practical, dynamic network
environments.

##  X Conclusion

Our comprehensive study embarked on the quest to enhance the integrity of
network traffic datasets, a cornerstone for developing machine learning models
in cybersecurity anomaly detection. By deploying NFStream, we carefully
crafted two refined versions of the CICIDS-2017 dataset that adhere to
methodologically sound flow measurement principles. The subsequent evaluation
using the RF algorithm provided a multi-faceted view of the models’
performance across the original, refined, and NFStream-generated datasets.

The research unearthed several key insights. Despite data measurement
inconsistencies in existing datasets, the RF model demonstrated remarkable
resilience, showcasing consistent accuracy in anomaly classification. However,
this robustness does not overshadow the importance of rigorous dataset
preparation, as evidenced by negative values and NaN entries in datasets not
processed by NFStream.

This work underscores the critical need for transparent and accurate flow
metering in network traffic analysis. Future research should continue to
explore the development of sophisticated flow metering tools that can
seamlessly integrate into the ML pipeline, ensuring that models are trained on
data reflective of the multifaceted nature of network behavior.

In alignment with the principles of open science, the refined datasets,
NFS-2023-nTE [5] and NFS-2023-TE [6], have been made accessible to the broader
research community. Furthermore, we have ensured that the scripts utilized in
our experiments are also publicly available [7]. Our reproducible methodology
is readily adaptable for processing the CICIDS-2017 PCAP files to meet the
diverse needs of future research endeavors. This initiative aims to promote a
deeper understanding of our methodological approaches and foster further
research within this area of study.

###  X-A Acknowledgement

This work was supported by the János Bolyai Research Scholarship of the
Hungarian Academy of Sciences. Supported by the ÚNKP-23-5-BME-461 New National
Excellence Program of the Ministry for Culture and Innovation from the source
of the National Research, Development and Innovation Fund.

## References

  * [1] Iman Sharafaldin et al.  “Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization”  In _Proceedings of the 4th International Conference on Information Systems Security and Privacy_ , CISSP 2018, 2018, pp. 108–116  DOI: 10.5220/0006639801080116
  * [2] Gints Engelen et al.  “Troubleshooting an Intrusion Detection Dataset: the CICIDS2017 Case Study”  In _2021 IEEE Security and Privacy Workshops (SPW)_ , 2021, pp. 7–12  DOI: 10.1109/SPW53761.2021.00009
  * [3] Lisa Liu et al.  “Error Prevalence in NIDS datasets: A Case Study on CIC-IDS-2017 and CSE-CIC-IDS-2018”  In _2022 IEEE Conference on Communications and Network Security (CNS)_ , 2022, pp. 254–262  DOI: 10.1109/CNS56114.2022.9947235
  * [4] Maxime Lanvin et al.  “Errors in the CICIDS2017 Dataset and the Significant Differences in Detection Performances It Makes”  In _Risks and Security of Internet and Systems_ , 2023, pp. 18–33  DOI: 10.1007/978-3-031-31108-6˙2
  * [5] FlowFrontiers  “NFS-2023-nTE dataset”, https://github.com/FlowFrontiers/CyberML-DataQuality/tree/main/4-NFS-2023/NFS-2023-nTE/datasets, 2023
  * [6] FlowFrontiers  “NFS-2023-TE dataset”, https://github.com/FlowFrontiers/CyberML-DataQuality/tree/main/4-NFS-2023/NFS-2023-TE/datasets, 2023
  * [7] FlowFrontiers  “Evaluating ML-Based Anomaly Detection Across Datasets of Varied Quality: A Case Study”, https://github.com/FlowFrontiers/CyberML-DataQuality, 2023
  * [8] Gerard Draper-Gil. et al.  “Characterization of Encrypted and VPN Traffic using Time-related Features”  In _Proceedings of the 2nd International Conference on Information Systems Security and Privacy - ICISSP,_ , 2016, pp. 407–414  INSTICC  DOI: 10.5220/0005740704070414
  * [9] Arash Habibi Lashkari. et al.  “Characterization of Tor Traffic using Time based Features”  In _Proceedings of the 3rd International Conference on Information Systems Security and Privacy - ICISSP,_ , 2017, pp. 253–262  INSTICC  DOI: 10.5220/0006105602530262
  * [10] Zied Aouini and Adrian Pekar  “NFStream: A flexible network data analysis framework”  In _Computer Networks_ 204, 2022, pp. 108719  DOI: 10.1016/j.comnet.2021.108719
  * [11] Rick Hofstede et al.  “Flow Monitoring Explained: From Packet Capture to Data Analysis With NetFlow and IPFIX”  In _IEEE Communications Surveys & Tutorials_ 16.4, 2014, pp. 2037–2064  DOI: 10.1109/COMST.2014.2321898
  * [12] FlowFrontiers  “Binary Classification Feature Importances”, https://github.com/FlowFrontiers/CyberML-DataQuality/blob/main/visual-comparison/fig/binary-feature-importance.png, 2023
  * [13] FlowFrontiers  “Multi-class Classification Feature Importances”, https://github.com/FlowFrontiers/CyberML-DataQuality/blob/main/visual-comparison/fig/multi-feature-importance.png, 2023
  * [14] Pierre Geurts et al.  “Extremely randomized trees”  In _Machine Learning_ 63.1, 2006, pp. 3–42  DOI: 10.1007/s10994-006-6226-1
  * [15] FlowFrontiers  “DT, RF and NB performance comparision for CICIDS-2017”, https://github.com/FlowFrontiers/CyberML-DataQuality/blob/main/1-CICIDS-2017/3-ml-with_feat_sel.ipynb, 2023
  * [16] FlowFrontiers  “DT, RF and NB performance comparision for WTMC-2021”, https://github.com/FlowFrontiers/CyberML-DataQuality/blob/main/2-WTMC-2021/3-ml-with_feat_sel.ipynb, 2023
  * [17] FlowFrontiers  “DT, RF and NB performance comparision for CRiSIS-2022”, https://github.com/FlowFrontiers/CyberML-DataQuality/blob/main/3-CRiSIS-2022/3-ml-with_feat_sel.ipynb, 2023
  * [18] FlowFrontiers  “DT, RF and NB performance comparision for NFS-2023-nTE”, https://github.com/FlowFrontiers/CyberML-DataQuality/blob/main/4-NFS-2023/2-NFS-2023-nTE-3-ml-with_feat_sel.ipynb, 2023
  * [19] FlowFrontiers  “DT, RF and NB performance comparision for NFS-2023-TE”, https://github.com/FlowFrontiers/CyberML-DataQuality/blob/main/4-NFS-2023/3-NFS-2023-TE-3-ml-with_feat_sel.ipynb, 2023