# Generative AI and Large Language Models for Cyber Security: All Insights You
Need

Mohamed Amine Ferrag, Fatima Alwahedi, Ammar Battah, Bilel Cherif, Abdechakour
Mechri,
and Norbert Tihanyi M. A. Ferrag is the corresponding author.M. A. Ferrag is
with Technology Innovation Institute, 9639 Masdar City, Abu Dhabi, UAE Email:
<EMAIL_ADDRESS>Alwahedi is with Technology Innovation Institute, 9639
Masdar City, Abu Dhabi, UAE Email<EMAIL_ADDRESS>Battah is with
Technology Innovation Institute, 9639 Masdar City, Abu Dhabi, UAE Email:
<EMAIL_ADDRESS>Cherif is with Technology Innovation Institute, 9639
Masdar City, Abu Dhabi, UAE Email<EMAIL_ADDRESS>Mechri is with
Technology Innovation Institute, 9639 Masdar City, Abu Dhabi, UAE Email:
<EMAIL_ADDRESS>Tihanyi is with Technology Innovation Institute,
9639 Masdar City, Abu Dhabi, UAE Email<EMAIL_ADDRESS>
###### Abstract

The rapid evolution of cyber threats requires innovative approaches to enhance
cybersecurity defenses. In this paper, we provide a comprehensive and in-depth
review of the future of cybersecurity through the lens of Generative AI and
Large Language Models (LLMs). We explore LLM applications across various
cybersecurity domains, including hardware design security, intrusion
detection, software engineering, design verification, cyber threat
intelligence, malware detection, and phishing and spam detection. We present a
detailed overview of LLM evolution and its current state, focusing on
advancements in 42 specific models, such as GPT-4o, GPT-4, GPT-3.5,
Mixtral-8x7B, BERT, Falcon2, Gemma, Phi-2, Phi-3, and LLaMA. Our analysis
extends to the vulnerabilities inherent in LLMs, including prompt injection,
insecure output handling, training and inference data poisoning, DDoS attacks,
and adversarial natural language instructions. We also delve into mitigation
strategies to protect these models, providing a comprehensive look at
potential attack scenarios and prevention techniques. Furthermore, we evaluate
the performance of 42 LLM models in cybersecurity knowledge and hardware
security, highlighting their strengths and weaknesses. Our study thoroughly
evaluates cybersecurity datasets tailored for LLM training and testing,
covering the entire lifecycle from data creation to usage and identifying gaps
and opportunities for future research. We discuss the challenges and
limitations of employing LLMs in cybersecurity settings, such as dealing with
adversarial attacks and ensuring model robustness. In addition, we review the
new strategies for leveraging LLMs, including advanced techniques like Half-
Quadratic Quantization (HQQ), Reinforcement Learning with Human Feedback
(RLHF), Direct Preference Optimization (DPO), Odds Ratio Preference
Optimization (ORPO), GPT-Generated Unified Format (GGUF), Quantized Low-Rank
Adapters (QLoRA), and Retrieval-Augmented Generation (RAG). These insights aim
to enhance real-time cybersecurity defenses and improve the sophistication of
LLM applications in threat detection and response. Our paper aims to provide a
foundational understanding and strategic direction for integrating LLMs into
future cybersecurity frameworks, emphasizing the importance of innovation and
robust model deployment to safeguard against evolving cyber threats.

###### Index Terms:

Generative AI, LLM, Transformer, Security, Cyber Security.

## List of Abbreviations

AI | Artificial Intelligence
---|---
AIGC | Artificial Intelligence Generated Content
APT | Advanced Persistent Threat
CNN | Convolutional Neural Network
CTG | Controllable Text Generation
CVE | Common Vulnerabilities and Exposures
CWE | Common Weakness Enumeration
FNN | Feed-Forward Neural Network
FRR | False Refusal Rate
GPT | Generative Pre-trained Transformers
GRU | Gated Recurrent Units
GQA | Grouped-Query Attention
HPC | High-Performance Computing
HLS | High-Level Synthesis Design Verification
HQQ | Half-Quadratic Quantization
IDS | Intrusion Detection System
LLM | Large Language Model
LoRA | Low-rank Adapters
LSTM | Long Short-Term Memory
ML | Machine Learning
MLP | Multi-Layer Perceptron
MQA | Multi-Query Attention
NIST | National Institute of Standards and Technology
NLP | Natural Language Processing
NLU | Natural Language Understanding
ORPO | Odds Ratio Preference Optimization
PEFT | Parameter Efficient Fine-Tuning
PLM | Pre-trained Language Model
PPO | Proximal Policy Optimization
RAG | Retrieval Augmentation Generation
RLHF | Reinforcement Learning from Human Feedback
RNN | Recurrent Neural Networks
RTL | Register-Transfer Level
SARD | Software Assurance Reference Dataset
SFT | Supervised Fine-Tuning
SVM | Support Vector Machine
TRPO | Trust Region Policy Optimization

##  I Introduction

The history of Natural Language Processing (NLP) dates back to the 1950s when
the Turing test was developed. However, NLP has seen significant advancements
in recent decades with the introduction of Recurrent Neural Networks (RNN)
[1], Long Short-Term Memory (LSTM) [2], Gated Recurrent Units (GRU) [3], and
Transformer methods [4]. RNN was first introduced in the 1990s to model data
sequences. LSTM, a variant of RNN, was introduced in 1997, which addressed the
vanishing gradient problem and allowed for longer-term memory in NLP models.
GRU, another variant of RNN, was introduced in 2014, which reduced the number
of parameters and improved computational efficiency [5]. The latest
breakthrough in NLP was the introduction of Transformers in 2017, enabling
parallel processing of sequential data and revolutionizing tasks like machine
translation. These methods have significantly improved various NLP tasks,
including sentiment analysis, language generation, and translation [4, 6, 7].

Cybersecurity is an ever-evolving field, with threats becoming increasingly
sophisticated and complex. As organizations and individuals rely on digital
technologies for communication, commerce, and critical infrastructure, the
need for robust cybersecurity measures has never been greater [8]. The scale
and diversity of cyber threats make it a daunting challenge for security
professionals to effectively identify, detect, and defend against them. In
this context, Large Language Models (LLMs) have emerged as a game-changing
technology with the potential to enhance cybersecurity practices significantly
[9, 10, 11, 12, 13]. These models, powered by advanced NLP and Machine
Learning (ML) techniques, offer a new frontier in the fight against cyber
threats [14, 15]. This article explores the motivations and applications of
LLMs in cybersecurity.

Cybersecurity professionals often need to sift through a vast amount of
textual data, including security alerts, incident reports, threat feeds, and
research papers, to stay ahead of evolving threats. LLMs, like Falcon 180b
[16], possess natural language understanding capabilities that enable them to
parse, summarize, and contextualize this information efficiently [17, 18, 7].
They can assist in rapidly identifying relevant threat intelligence, allowing
analysts to make more informed decisions and prioritize responses [19]. LLMs
can excel in various domains within cybersecurity. Figure 1 highlights the top
nine use cases and applications for LLMs in this field [20].

  1. 1.

Threat Detection and Analysis: LLMs can analyze vast network data in real-time
to detect anomalies and potential threats. They can recognize patterns
indicative of cyber attacks, such as malware, phishing attempts, and unusual
network traffic [19].

  2. 2.

Security Automation: LLMs can facilitate the automation of routine security
tasks such as patch management, vulnerability assessments, and compliance
checks. This reduces the workload on cybersecurity teams and allows them to
focus on more complex tasks [9].

  3. 3.

Phishing Detection and Response: LLMs can identify phishing emails by
analyzing the text for malicious intent and comparing it to known phishing
examples. They can also generate alerts and recommend preventive actions [21].

  4. 4.

Cyber Forensics: LLMs can help in forensic analysis by parsing through logs
and data to determine the cause and method of attack, thus aiding in the
recovery process and future prevention strategies [22].

  5. 5.

Penetration Testing: LLMs can help generate scripts or modify existing ones to
automate certain parts of the penetration testing process. This includes
scripts for vulnerability scanning, network mapping, and exploiting known
vulnerabilities [23].

  6. 6.

Security Protocols Verification: LLMs can help verify the security of
protocols such as TLS/SSL, IPSec, …etc.

  7. 7.

Incident Response: During a cybersecurity incident, LLMs can assist by
providing rapid analysis of the situation, suggesting mitigation strategies,
and automating responses where applicable [24].

  8. 8.

Chatbots: LLMs significantly enhance the capabilities of chatbots in
cybersecurity environments by providing User Interaction, Incident Reporting
and Handling, Real-time Assistance, Training and Simulations, and FAQ
Automation [25].

  9. 9.

Security Training and Awareness: LLMs can generate training materials tailored
to an organization’s needs. They can also simulate phishing attacks and other
security scenarios to train employees to recognize and respond to security
threats [26].

Figure 1: LLM Use Cases And Applications for Cybersecurity. Figure 2: Survey
Structure (From Section I. to Section X.)

The primary aim of this paper is to provide an in-depth and comprehensive
review of the future of cybersecurity using Generative AI and LLMs, covering
all relevant topics in the cyber domain. The contributions of this study are
summarized below:

  * •

We review LLMs’ applications for cybersecurity use cases, such as hardware
design security, intrusion detection, software engineering, design
verification, cyber threat intelligence, malware detection, phishing, and spam
detection, etc., providing a nuanced understanding of LLM capabilities across
different cybersecurity domains;

  * •

We present a comprehensive overview of LLMs in cybersecurity, detailing their
evolution and current state, including advancements in 42 specific models,
such as GPT-4o, GPT-4, BERT, Falcon, and LLaMA models;

  * •

We analyze the vulnerabilities associated with LLMs, including prompt
injection, insecure output handling, training data poisoning, inference data
poisoning, DDoS attacks, and adversarial natural language instructions. We
also examine the mitigation strategies to safeguard these models from such
vulnerabilities, providing a comprehensive look at potential attack scenarios
and prevention techniques;

  * •

We evaluated the performance of 42 LLM models in different datasets in the
cybersecurity domain.

  * •

We thoroughly evaluate cybersecurity datasets tailored for LLM training and
testing. This includes a lifecycle analysis from dataset creation to usage,
covering various stages such as data cleaning, preprocessing, annotation, and
labeling. We also compare cybersecurity datasets to identify gaps and
opportunities for future research;

  * •

We provide the challenges and limitations of employing LLMs in cybersecurity
settings, such as dealing with adversarial attacks and ensuring robustness. We
also discuss the implications of these challenges for future LLM deployments
and the development of secure, optimized models;

  * •

We discuss novel insights and strategies for leveraging LLMs in cybersecurity,
including advanced techniques such as Half-Quadratic Quantization (HQQ),
Reinforcement Learning with Human Feedback (RLHF), Direct Preference
Optimization (DPO), Odds Ratio Preference Optimization (ORPO), GPT-Generated
Unified Format (GGUF), Quantized Low-Rank Adapters (QLoRA), and Retrieval-
Augmented Generation (RAG). These insights aim to enhance real-time
cybersecurity defenses and improve the sophistication of LLM applications in
threat detection and response.

The rest of this paper is organized as follows. Section II presents an in-
depth analysis of related reviews in the field, charting the evolution and
state of LLMs in artificial intelligence. Section III delves into the
preliminaries of NLP applications for cybersecurity, covering foundational
models and their advancements. Section IV discusses LLM-based solutions
specific to cybersecurity. Section V reviews general LLM models. Section VI
reviews Code-specific LLMs models. Section VII explores various cybersecurity
datasets designed for LLM training and evaluation, detailing their development
lifecycle and specific attributes. Section VIII focuses on the vulnerabilities
associated with LLMs and the strategies for their mitigation, introducing a
classification of potential threats and defense mechanisms. Section IX offers
comprehensive insights into the challenges and limitations of integrating LLMs
into cybersecurity frameworks, including practical considerations and
theoretical constraints. Finally, Section X concludes the paper by summarizing
the key findings and proposing directions for future research in LLMs and
cybersecurity. A brief overview of the paper’s structure is illustrated in
Figure 2.

TABLE I: Summary of Related Reviews on Large Language Models

Focused Area of Study |  Year |  Authors |  Key Points |  Data. |  Vuln. |  Comp. |  Optim. |  Hardw.
---|---|---|---|---|---|---|---|---
LLMs in Enhancing Hardware Design Security |  2023 |  Saha et al. [27] |  Discuss applications of LLMs in hardware design security, including vulnerability introduction, assessment, verification, and countermeasure development. |  ✘ |  ✘ |  ✘ |  ✘ |  ✘
Comprehensive Evaluation Methodologies for LLMs |  2023 |  Chang et al. [26] |  Provides an analysis of LLM evaluations focusing on criteria, context, methodologies, and future challenges. |  ✘ |  ✘ |  ✘ |  ✘ |  ✘
The Evolutionary Path of LLMs in AI |  2023 |  Zhao et al. [24] |  Surveys the evolution of LLMs in AI, focusing on pre-training, adaptation tuning, utilization, and capacity evaluation. |  ✘ |  ✘ |  ✘ |  ✘ |  ✘
Recent Advancements in PLMs for NLP |  2023 |  Min et al. [28] |  Reviews advancements in PLMs for NLP, covering paradigms like Pre-train then Fine-tune, Prompt-based Learning, and NLP as Text Generation. |  ✘ |  ✘ |  ✘ |  ✘ |  ✘
Exploring Instruction Fine-Tuning in LLMs |  2023 |  Zhang et al. [29] |  Explores instruction fine-tuning for LLMs, covering methodologies, datasets, models, and multi-modality techniques. |  ✘ |  ✘ |  ✘ |  ✘ |  ✘
Applying LLMs in Software Engineering |  2023 |  Fan et al. [30] |  Survey the use of LLMs in Software Engineering, discussing applications, challenges, and hybrid approaches. |  ✘ |  ✘ |  ✘ |  ✘ |  ✘
Understanding Multimodal Algorithms |  2023 |  Wu et al. [31] |  Provides an overview of multimodal algorithms, covering definition, evolution, technical aspects, and challenges. |  ✘ |  ✘ |  ✘ |  ✘ |  ✘
Defining Alignment Requirements for LLMs |  2023 |  Liu et al. [32] |  Proposes a taxonomy of alignment requirements for LLMs and discusses harmful content concepts. |  ✘ |  ✘ |  ✘ |  ✘ |  ✘
Incorporating External Knowledge in PLMs |  2023 |  Hu et al. [33] |  Reviews KE-PLMs, focusing on incorporating different types of knowledge into PLMs for NLP. |  ✘ |  ✘ |  ✘ |  ✘ |  ✘
Advances in Controllable Text Generation |  2023 |  Zhang et al. [34] |  Reviews CTG in NLG, focusing on Transformer-based PLMs and challenges in controllability. |  ✘ |  ✘ |  ✘ |  ✘ |  ✘
LLM for Blockchain Security |  2024 |  He et al. [35] |  Analyze existing research to understand how LLMs can improve blockchain systems’ security. |  |  |  ✘ |  ✘ |  ✘
LLM for Critical Infrastructure Protection |  2024 |  Yigit et al. [36] |  Proposing advanced strategies using Generative AI and Large Language Models to enhance resilience and security. |  ✘ |  ✘ |  ✘ |  ✘ |  ✘
Software Testing with Large Language Models |  2024 |  Wang et al. [37] |  Explore how Large Language Models (LLMs) can enhance software testing, examining tasks, techniques, and future research directions. |  ✘ |  ✘ |  ✘ |  ✘ |  ✘
Malicious Insider Threat Detection Using Machine Learning Methods |  2024 |  Alzaabi et al. [22] |  Recommends advanced ML methods like deep learning and NLP for better detection and mitigation of insider threats in cybersecurity, emphasizing the need for integrating time-series techniques. |  |  |  ✘ |  ✘ |  ✘
Advancements in Large Language Models |  2024 |  Raiaan et al. [25] |  Reviews the evolution, architectures, applications, societal impacts, and challenges of LLMs, aiding practitioners, researchers, and experts in understanding their development and prospects. |  ✘ |  ✘ |  ✘ |  ✘ |  ✘
Applications of LLMs in cybersecurity tasks |  2024 |  Xu et al. [38] |  Highlights the diverse applications of LLMs in cybersecurity tasks such as vulnerability detection, malware analysis, and intrusion and phishing detection. |  |  |  ✘ |  ✘ |  ✘
Retrieval-Augmented Generation for LLMs |  2024 |  Zhao et al. [25] |  Reviews how RAG has been integrated into various AIGC scenarios to overcome common challenges such as updating knowledge, handling long-tail data, mitigating data leakage, and managing costs associated with training and inference. |  ✘ |  ✘ |  ✘ |  ✘ |  ✘
Provides an overview of Parameter Efficient Fine-Tuning (PEFT) |  2024 |  Han et al. [39] |  Reviews various PEFT algorithms, their effectiveness, and the computational overhead involved. |  ✘ |  ✘ |  ✘ |  ✘ |  ✘
LLM for Cyber Security |  2024 |  Zhang et al. [40] |  The paper conducts a systematic literature review of over 180 works on applying LLMs in cybersecurity. |  |  |  ✘ |  ✘ |  ✘
LLM with security and privacy issues |  2024 |  Yao et al. [9] |  Explores the dual impact of LLMs on security and privacy, highlighting their potential to enhance cybersecurity and data protection while also posing new risks and vulnerabilities. |  |  |  ✘ |  ✘ |  ✘
THIS SURVEY |  2024 |  Ferrag et al. |  This paper provides an in-depth review of using Generative AI and Large Language Models (LLMs) in cybersecurity. |  ✔ |  ✔ |  ✔ |  ✔ |  ✔

✘ : Not covered; : Partially covered; ✔: Covered; Data.: Datasets used for
training and fine-tuning LLMs for security use cases; Vuln.: LLM
Vulnerabilities and Mitigation ; Comp.: Experimental Analysis of LLMs Models’
Performance in Cyber Security Knowledge; Optim.: Optimization Strategies for
Large Language Models in Cybersecurity; Hardw. : Experimental Analysis of LLMs
Models’ Performance in Hardware Security.

##  II Related reviews

This section delves into a curated collection of recent articles that
significantly contribute to the evolving landscape of LLMs and their
multifaceted applications. These reviews offer a comprehensive and insightful
exploration into various dimensions of LLMs, including their innovative
applications in hardware design security, evaluation methodologies, and
evolving role in artificial intelligence. Further, they cover cutting-edge
advancements in Pre-trained Language Models (PLMs) for NLP, delve into the
intricacies of instruction fine-tuning for LLMs, and explore their impactful
integration into software engineering. The section also encompasses an in-
depth look at multimodal algorithms, examines the critical aspect of alignment
requirements for LLMs, and discusses integrating external knowledge into PLMs
to enhance NLP tasks. Lastly, it sheds light on the burgeoning field of
Controllable Text Generation (CTG) in Natural Language Generation (NLG),
highlighting the latest trends and challenges in this dynamic and rapidly
advancing area of research [41, 42, 43].

###  II-A Applications of LLMs in Hardware Design Security

Saha et al. [27] discussed several key applications of LLMs in the context of
hardware design security. The paper illustrates how LLMs can intentionally
introduce vulnerabilities and weaknesses into RTL (Register-Transfer Level)
designs. This process is guided by well-crafted prompts in natural language,
demonstrating the model’s ability to understand and manipulate complex
technical designs. The authors explore using LLMs to assess the security of
hardware designs. The model is employed to identify vulnerabilities,
weaknesses, and potential threats. It’s also used to pinpoint simple coding
issues that could evolve into significant security bugs, highlighting the
model’s ability to evaluate technical designs critically. In this application,
LLMs verify whether a hardware design adheres to specific security rules or
policies. The paper examines the model’s proficiency in calculating security
metrics, understanding security properties, and generating functional
testbenches to detect weaknesses. This part of the study underscores the LLM’s
ability to conduct thorough and detailed verification processes. Finally, the
paper investigates how effectively LLMs can be used to develop countermeasures
against existing vulnerabilities in a design. This aspect focuses on the
model’s capability to solve problems and create solutions to enhance the
security of hardware designs. Overall, the paper presents an in-depth analysis
of how LLMs can be a powerful tool in various stages of hardware design
security, from vulnerability introduction and assessment to verification and
countermeasure development.

###  II-B Evaluation of LLMs

Chang et al. [26] offers a comprehensive analysis of LLM evaluations,
addressing three key aspects: the criteria for evaluation (what to evaluate),
the context (where to evaluate), and the methodologies (how to evaluate). It
thoroughly reviews various tasks across different domains to understand the
successes and failures of LLMs, contributing to future research directions.
The paper also discusses current evaluation metrics, datasets, and benchmarks
and introduces novel approaches, providing a deep understanding of the current
evaluation landscape. Additionally, it highlights future challenges in LLM
evaluation and supports the research community by open-sourcing related
materials, fostering collaborative advancements in the field.

###  II-C Evolution and State of LLMs in AI

Zhao et al. [24] provides an in-depth survey of LLMs’ evolution and current
state in artificial intelligence. It traces the progression from statistical
language models to neural language models, specifically focusing on the recent
emergence of pre-trained language models (PLMs) using Transformer models
trained on extensive corpora. The paper emphasizes the significant
advancements achieved by scaling up these models, noting that LLMs demonstrate
remarkable performance improvements beyond a certain threshold and exhibit
unique capabilities not found in smaller-scale models. The survey covers four
critical aspects of LLMs: pre-training, adaptation tuning, utilization, and
capacity evaluation, providing insights into both their technical evolution
and the challenges they pose. Additionally, the paper discusses the resources
available for LLM development and explores potential future research
directions, underlining the transformative effect of LLMs on AI development
and application.

###  II-D Advancements in PLMs for NLP

Min et al. [28] surveys the latest advancements in leveraging PLMs for NLP,
organizing the approaches into three main paradigms. Firstly, the ”Pre-train
then Fine-tune” method involves general pre-training on large unlabeled
datasets followed by specific fine-tuning for targeted NLP tasks. Secondly,
”Prompt-based Learning” uses tailored prompts to transform NLP tasks into
formats akin to a PLM’s pre-training, enhancing the model’s performance,
especially in few-shot learning scenarios. Lastly, the ”NLP as Text
Generation” paradigm reimagines NLP tasks as text generation problems, fully
capitalizing on the strengths of generative models like GPT-2 and T5. These
paradigms represent the cutting-edge methods in utilizing PLMs for various NLP
applications.

###  II-E Instruction Fine-Tuning for LLMs

Zhang et al. [29] delves into the field of instruction fine-tuning for LLMs,
offering a detailed exploration of various facets of this rapidly advancing
area. It begins with an overview of the general methodologies used in
instruction fine-tuning, then discusses the construction of commonly-used,
representative datasets tailored for this approach. The survey highlights a
range of instruction-fine-tuned models, showcasing their diversity and
capabilities. It also examines multi-modality techniques and datasets,
including those involving images, speech, and video, reflecting the broad
applicability of instruction tuning. The adaptation of LLMs to different
domains and applications using instruction tuning strategies is reviewed,
demonstrating the versatility of this method. Additionally, the survey
addresses efforts to enhance the efficiency of instruction fine-tuning,
focusing on reducing computational and time costs. Finally, it evaluates these
models, including performance analysis and critical perspectives, offering a
holistic view of the current state and potential of instruction fine-tuning in
LLMs.

###  II-F LLMs in Software Engineering

Fan et al. [30] present a survey on using LLMs in Software Engineering (SE),
highlighting their potential applications and open research challenges. LLMs,
known for their emergent properties, offer novel and creative solutions across
various Software Engineering activities, including coding, design,
requirements analysis, bug fixing, refactoring, performance optimization,
documentation, and analytics. Despite these advantages, the paper also
acknowledges the significant technical challenges these emergent properties
bring, such as the need for methods to eliminate incorrect solutions, notably
hallucinations. The survey emphasizes the crucial role of hybrid approaches,
which combine traditional Software Engineering techniques with LLMs, in
developing and deploying reliable, efficient, and effective LLM-based
solutions for Software Engineering. This approach suggests a promising pathway
for integrating advanced AI models into practical software development
processes.

###  II-G Multimodal Algorithms

Wu et al. [31] addresses a significant gap in understanding multimodal
algorithms by providing a comprehensive overview of their definition,
historical development, applications, and challenges. It begins by defining
multimodal models and algorithms, then traces their historical evolution,
offering insights into their progression and significance. The paper serves as
a practical guide, covering various technical aspects essential to multimodal
models, such as knowledge representation, selection of learning objectives,
model construction, information fusion, and prompts. Additionally, it reviews
current algorithms employed in multimodal models and discusses commonly used
datasets, thus laying a foundation for future research and evaluation in this
field. The paper concludes by exploring several applications of multimodal
models and delving into key challenges that have emerged from their recent
development, shedding light on both the potential and the limitations of these
advanced computational tools.

###  II-H Alignment Requirements for LLMs

Liu et al. [32] propose a taxonomy of alignment requirements for LLMs to aid
practitioners in understanding and effectively implementing alignment
dimensions and inform data collection efforts for developing robust alignment
processes. The paper dissects the concept of ”harmful” generated content into
specific categories, such as harm to individuals (like emotional harm,
offensiveness, and discrimination), societal harm (including instructions for
violent or dangerous behaviors), and harm to stakeholders (such as
misinformation impacting business decisions). Citing an imbalance in
Anthropic’s alignment data, the paper points out the uneven representation of
various harm categories, like the high frequency of ”violence” versus the
marginal appearance of ”child abuse” and ”self-harm.” This observation
supports the argument that alignment techniques heavily dependent on data
cannot ensure that LLMs will uniformly align with human behaviors across all
aspects. The authors’ own measurement studies reveal that aligned models do
not consistently show improvements across all harm categories despite the
alignment efforts claimed by the model developers. Consequently, the paper
advocates for a framework that allows a more transparent, multi-objective
evaluation of LLM trustworthiness, emphasizing the need for a comprehensive
and balanced approach to alignment in LLM development.

###  II-I Knowledge-Enhanced Pre-trained Language Models

Hu et al. [33] offers a comprehensive review of Knowledge-Enhanced Pre-trained
Language Models (KE-PLMs), a burgeoning field aiming to address the
limitations of standard PLMs in NLP. While PLMs trained on vast text corpora
demonstrate impressive performance across various NLP tasks, they often fall
short in areas like reasoning due to the absence of external knowledge. The
paper focuses on how incorporating different types of knowledge into PLMs can
overcome these shortcomings. It introduces distinct taxonomies for Natural
Language Understanding (NLU) and Natural Language Generation (NLG) to
distinguish between these two core areas of NLP. For NLU, the paper
categorizes knowledge types into linguistic, text, knowledge graph (KG), and
rule knowledge. In the context of NLG, KE-PLMs are classified into KG-based
and retrieval-based methods. By outlining these classifications and exploring
the current state of KE-PLMs, the paper provides not only clear insights into
this evolving domain but also identifies promising future directions for the
development and application of KE-PLMs, highlighting their potential to
enhance the capabilities of PLMs in NLP tasks significantly.

###  II-J Controllable Text Generation in NLG

Zhang [34] provides a critical and systematic review of Controllable Text
Generation (CTG), a burgeoning field in NLG that is essential for developing
advanced text generation technologies tailored to specific practical
constraints. The paper focuses on using large-scale pre-trained language
models (PLMs), particularly those based on transformer architecture, which
have established a new paradigm in NLG due to their ability to generate more
diverse and fluent text. However, the limited interpretability of deep neural
networks poses challenges to the controllability of these methods, making
transformer-based PLM-driven CTG a rapidly evolving and challenging research
area. The paper surveys various approaches that have emerged in the last 3-4
years, each targeting different CTG tasks with varying controlled constraints.
It provides a comprehensive overview of common tasks, main approaches, and
evaluation methods in CTG and discusses the current challenges and potential
future directions in the field. Claiming to be the first to summarize state-
of-the-art CTG techniques from the perspective of Transformer-based PLMs, this
paper aims to assist researchers and practitioners in keeping pace with the
academic and technological developments in CTG, offering them an insightful
landscape of the field and a guide for future research.

###  II-K LLM for Cyber Security

Zhang et al. [40] examines the integration of LLMs within cybersecurity.
Through an extensive literature review involving over 127 publications from
leading security and software engineering venues, this paper aims to shed
light on LLMs’ multifaceted roles in enhancing cybersecurity measures. The
survey pinpoints various applications for LLMs in detecting vulnerabilities,
analyzing malware, and managing network intrusions and phishing threats. It
highlights the current limitations regarding the datasets used, which often
lack size and diversity, thereby underlining the necessity for more robust
datasets tailored to these security tasks. The paper also identifies promising
methodologies like fine-tuning and domain-specific pre-training, which could
better harness the potential of LLMs in cybersecurity contexts.

Yao et al. [9] explores the dual role of LLMs in security and privacy,
highlighting their benefits in enhancing code security and data
confidentiality and detailing potential risks and inherent vulnerabilities.
The authors categorize the applications and challenges into ”The Good,” ”The
Bad,” and ”The Ugly,” where they discuss LLMs’ positive impacts, their use in
offensive applications, and their susceptibility to specific attacks,
respectively. The paper emphasizes the need for further research on threats
like model and parameter extraction attacks and emerging techniques such as
safe instruction tuning, underscoring the complex balance between leveraging
LLMs for improved security and mitigating their risks.

###  II-L Our survey compared to related surveys

Our paper presents a more specialized and technical exploration of generative
artificial intelligence and large language models in cybersecurity than the
previous literature review. Focusing on a broad array of cybersecurity domains
such as hardware design security, intrusion detection systems, and software
engineering, it targets a wider professional audience, including engineers,
researchers, and industrial practitioners. This paper reviews 35 leading
models like GPT-4, BERT, Falcon, and LLaMA, not only highlighting their
applications but also their developmental trajectories, thereby providing a
comprehensive insight into the current capabilities and future potentials of
these models in cybersecurity.

The paper also delves deeply into the vulnerabilities associated with LLMs,
such as prompt injection, adversarial natural language instructions, and
insecure output handling. It presents sophisticated attack scenarios and
robust mitigation strategies, offering a detailed analysis crucial for
understanding and protecting against potential threats. Additionally, the
lifecycle of specialized cybersecurity datasets—covering creation, cleaning,
preprocessing, annotation, and labeling—is scrutinized, providing essential
insights into improving data integrity and utility for training and testing
LLMs. This level of detail is vital for developing robust cybersecurity
solutions that can effectively leverage the power of LLMs.

Lastly, the paper examines the challenges associated with deploying LLMs in
cybersecurity contexts, emphasizing the necessity for model robustness and the
implications of adversarial attacks. It introduces advanced methodologies such
as Reinforcement Learning with Human Feedback (RLHF) and Retrieval-Augmented
Generation (RAG) to enhance real-time cybersecurity operations. This focus not
only delineates the current state of LLM applications in cybersecurity but
also sets the direction for future research and practical applications, aiming
to optimize and secure LLM deployments in an evolving threat landscape. This
makes the paper an indispensable resource for anyone involved in cybersecurity
and AI, bridging the gap between academic research and practical applications.

##  III Preliminaries of NLP for Cyber Security

This section presents the preliminaries of NLP for cybersecurity, including
recurrent neural networks (LSTMs and GRUs) and transformer models.

###  III-A Recurrent neural networks

Recurrent Neural Networks (RNNs) [44] are artificial neural networks that
handle data sequences such as time series or NLP tasks. The RNN model consists
of two linked recurrent neural networks. The first RNN encodes sequences of
symbols into a fixed-length vector, while the second decodes this vector into
a new sequence. This architecture aims to maximize the conditional probability
of a target sequence from a given source sequence. When applied to
cybersecurity, this model could be instrumental in threat detection and
response systems by analyzing and predicting network traffic or log data
sequences that indicate malicious activity. Integrating the conditional
probabilities generated by this model could enhance anomaly detection
frameworks, improving the identification of subtle or novel cyber threats. The
model’s ability to learn meaningful representations of data sequences further
supports its potential to recognize complex patterns and anomalies in
cybersecurity environments [45, 46].

####  III-A1 Gated Recurrent Units

GRUs are a recurrent neural network architecture designed to handle the
vanishing gradient problem that can occur with standard recurrent networks.
Introduced by Cho et al. in 2014 [47], GRUs simplify the LSTM (Long Short-Term
Memory) model while retaining its ability to model long-term dependencies in
sequential data. GRUs achieve this through two main gates: the update gate,
which controls how much a new state overwrites the old state, and the reset
gate, which determines how much past information to forget. These gates
effectively regulate the flow of information, making GRUs adept at tasks like
time series prediction, speech recognition, and natural language processing.
The main steps of GRUs are organized as follows:

  * •

Update Gate: The update gate determines how much information from the previous
hidden state should be passed to the new one. The update gate is calculated
using the following formula:

$z_{t}=\sigma(W_{z}x_{t}+U_{z}h_{t-1})$ (1)

where $z_{t}$ is the update gate at time step $t$, $W_{z}$ and $U_{z}$ are the
weight matrices, $x_{t}$ is the input at time step $t$, and $h_{t-1}$ is the
previous hidden state. The sigmoid function, represented by $\sigma$, squishes
the equation’s results between 0 and 1. The update gate allows the GRU to
decide how much of the previous hidden state information should be passed on
to the new hidden state. If the update gate is close to 1, it means that a lot
of the previous hidden state information should be passed on, while if the
update gate is close to 0, it means that very little of the previous hidden
state information should be passed on. The Update Gate formula can be explored
in the following three different parts:

Part 1: Linear combination of inputs:

$z_{t}=W_{r}x_{t}+U_{r}h_{t-1}$ (2)

Part 2: Application of the sigmoid function:

$\tilde{r}_{t}=\sigma(z_{t})$ (3)

Part 3: Element-wise multiplication of $\tilde{r}_{t}$ and previous hidden
state:

$r_{t}=\tilde{r}_{t}\odot h_{t-1}$ (4)

Where $\odot$ represents the Hadamard product, also known as the element-wise
multiplication.

  * •

Reset Gate: The reset gate determines how much of the previous hidden state
should be forgotten. The reset gate is calculated using the following formula:

$r_{t}=\sigma(W_{r}x_{t}+U_{r}h_{t-1})$ (5)

where $r_{t}$ is the reset gate at time step $t$, $W_{r}$ and $U_{r}$ are the
weight matrices, $x_{t}$ is the input at time step $t$, and $h_{t-1}$ is the
previous hidden state.

  * •

Candidate Hidden State: The candidate’s hidden state combines the input and
the previous hidden state, filtered through the reset gate. The candidate’s
hidden state is calculated using the following formula:

$\tilde{h}_{t}=\tanh(Wx_{t}+U(r_{t}\odot h_{t-1}))$ (6)

where $\tilde{h}_{t}$ is the candidate hidden state at time step $t$, $W$ and
$U$ are the weight matrices, $x_{t}$ is the input at time step $t$, and
$r_{t}$ is the reset gate at time step $t$. In this equation, the input at
time step $t$ ($x_{t}$) is combined with the previous hidden state ($h_{t-1}$)
through the weight matrices $W$ and $U$. The reset gate ($r_{t}$) is used to
control the extent to which the previous hidden state ($h_{t-1}$) is passed to
the candidate hidden state ($\tilde{h}_{t}$). The element-wise product between
the reset gate ($r_{t}$) and the previous hidden state ($h_{t-1}$) is used to
create the reset vector ($r_{t}\odot h_{t-1}$). The reset vector is combined
with the input ($x_{t}$) through the weight matrix $U$. Finally, the result is
passed through the hyperbolic tangent function to calculate the candidate
hidden state ($\tilde{h}_{t}$).

  * •

New Hidden State: The new hidden state combines the previous and candidate
hidden states, filtered through the update gate. The new hidden state is
calculated using the following formula:

$h_{t}=(1-z_{t})\odot h_{t-1}+z_{t}\odot\tilde{h}_{t}$ (7)

where $h_{t}$ is the new hidden state at time step $t$, $z_{t}$ is the update
gate at time step $t$, and $\tilde{h}_{t}$ is the candidate hidden state at
time step $t$. The new hidden state ($h_{t}$) is calculated by taking a
weighted combination of the previous hidden state ($h_{t-1}$) and the
candidate hidden state ($\tilde{h}_{t}$). The weight of the previous hidden
state is determined by the update gate ($z_{t}$) - if the update gate is close
to 1, the new hidden state is primarily influenced by the previous hidden
state. If the update gate is close to 0, the candidate’s hidden state
primarily influences the new hidden state. The element-wise product between
the update gate ($z_{t}$) and the candidate hidden state ($\tilde{h}_{t}$) is
used to create the update vector ($z_{t}\odot\tilde{h}_{t}$). The element-wise
product between the complement of the update gate ($1-z_{t}$) and the previous
hidden state ($h_{t-1}$) is used to create the reset vector ($(1-z_{t})\odot
h_{t-1}$). Finally, the update and reset vectors are added to calculate the
new hidden state ($h_{t}$).

####  III-A2 Long Short-Term Memory

The LSTM [2] was designed to overcome the vanishing gradient problem that
affects traditional recurrent neural networks (RNNs) during training,
particularly over long sequences. By integrating memory cells that can
maintain information over extended periods and gates that regulate the flow of
information into and out of the cell, LSTMs provide an effective mechanism for
learning dependencies and retaining information over time. This architecture
has proved highly influential, becoming foundational to numerous applications
in machine learning that require handling sequential data, such as natural
language processing, speech recognition, and time series analysis. The impact
of this work has been extensive, as it enabled the practical use and
development of deep learning models for complex sequence modeling tasks. In
cybersecurity, LSTMs can be used for anomaly detection, where they analyze
network traffic or system logs to identify unusual patterns that may signify a
security breach or malicious activity [48, 49, 50]. Their ability to learn
from long sequences makes them particularly useful for detecting sophisticated
attacks that evolve, such as advanced persistent threats (APTs) and
ransomware. The main steps of LSTM models are organized as follows:

  * •

Input Gate: The first step in an LSTM-based RNN involves calculating the input
gate. The input gate determines the extent of new input to be added to the
current state. The formula for the input gate is:

$i_{t}=\sigma(W_{i}\cdot[h_{t-1},x_{t}]+b_{i})$ (8)

where $i_{t}$ is the input gate at time step $t$, $W_{i}$ is the weight matrix
for the input gate, $h_{t-1}$ is the hidden state from the previous time step,
$x_{t}$ is the input at time step $t$, and $b_{i}$ is the bias for the input
gate. The function $\sigma(x)$ is the sigmoid activation function. This
formula calculates the input gate $i_{t}$ by first concatenating the previous
hidden state $h_{t-1}$ with the current input $x_{t}$. This combined vector is
multiplied by the weight matrix $W_{i}$, and the bias $b_{i}$ is added.
Finally, the sigmoid activation function is applied to produce $i_{t}$, which
ranges from 0 to 1 and represents how much the current input updates the
hidden state.

  * •

Forget Gate: The second step calculates the forget gate, determining how much
the previous state should be forgotten. The formula for the forget gate is:

$f_{t}=\sigma(W_{f}\cdot[h_{t-1},x_{t}]+b_{f})$ (9)

where $f_{t}$ is the forget gate at time step $t$, $W_{f}$ is the weight
matrix for the forget gate, $h_{t-1}$ is the hidden state from the previous
time step, $x_{t}$ is the input at time step $t$, and $b_{f}$ is the bias for
the forget gate. The forget gate $f_{t}$ is calculated like the input gate. It
involves concatenating the previous hidden state $h_{t-1}$ with the current
input $x_{t}$, multiplying by the weight matrix $W_{f}$, and adding the bias
$b_{f}$. The resulting value is passed through the sigmoid activation function
to determine the forget gate $f_{t}$, which ranges from 0 to 1 and represents
the degree to which the previous hidden state is preserved or forgotten in the
current hidden state.

  * •

Candidate Memory Cell: The third step calculates the candidate memory cell,
representing the potential memory state update. The formula for the candidate
memory cell is:

$\tilde{c}_{t}=\tanh(W_{c}\cdot[h_{t-1},x_{t}]+b_{c})$ (10)

where $\tilde{c}_{t}$ is the candidate memory cell at time step $t$, $W_{c}$
is the weight matrix for the candidate memory cell, $h_{t-1}$ is the hidden
state from the previous time step, $x_{t}$ is the input at time step $t$, and
$b_{c}$ is the bias for the candidate memory cell. The function $\tanh(x)$ is
the hyperbolic tangent activation function. In this formula, the candidate
memory cell $\tilde{c}_{t}$ is calculated by concatenating the previous hidden
state $h_{t-1}$ and the input $x_{t}$, then multiplying by the weight matrix
$W_{c}$ and adding the bias $b_{c}$. The result is passed through the
hyperbolic tangent activation function, which ranges from -1 to 1, to control
the magnitude of the memory cell update.

  * •

Current Memory Cell: The fourth step calculates the current memory cell, which
is the updated state of the memory cell, combining the effects of the forget
and input gates. The formula for the current memory cell is:

$c_{t}=f_{t}\cdot c_{t-1}+i_{t}\cdot\tilde{c}_{t}$ (11)

where $c_{t}$ is the current memory cell at time step $t$, $f_{t}$ is the
forget gate at time step $t$, $c_{t-1}$ is the memory cell from the previous
time step, $i_{t}$ is the input gate at time step $t$, and $\tilde{c}_{t}$ is
the candidate memory cell at time step $t$. This equation represents the new
memory cell state as a combination of the old state (modulated by the forget
gate) and the potential update (modulated by the input gate).

  * •

Output Gate: The final step calculates the output gate, which determines the
amount of information output from the LSTM cell. The details and formula for
the output gate should follow.

Figure 3: How Transformer works for Software Security.

###  III-B Transformer models

The Transformer architecture proposed by Vaswani et al. [4] in 2017 is a
significant advancement in natural language processing built entirely around
attention mechanisms. These mechanisms allow the model to assess the relevance
of different words in a sentence, independent of their positional
relationships. This foundational technology has enhanced the efficiency of
tasks like translation and text summarization and has broad cybersecurity
applications. In cybersecurity, Transformer models can detect and respond to
threats by analyzing source code patterns and network traffic and identifying
anomalies in system logs, as presented in Figure 3. They can also be used for
the automated generation of security policies based on the evolving landscape
of threats and for intelligent threat hunting, where the system predicts and
neutralizes threats before they cause harm. This makes Transformers versatile
in enhancing security protocols and defending against cyber attacks [19]. The
main steps of Transformer models are organized as follows:

  * •

Attention Mechanism: The attention mechanism in the Transformer model computes
attention scores between the input and output representations. These scores
are calculated using the scaled dot-product of the query and key
representations and then normalized by a softmax function. The attention
scores are subsequently used to compute a weighted sum of the value
representations, forming the output of the attention mechanism.

The equation defines the attention scores:

$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V$
(12)

Where:

$Q$, $K$, and $V$ represent the matrices of queries, keys, and values
transformed from the input representations. The dimension of the keys is
denoted by $d_{k}$. The attention mechanism involves computing the dot product
of $Q$ and the transpose of $K$, which is then scaled by the inverse square
root of $d_{k}$ to stabilize the gradients. The result is passed through a
softmax function to normalize the scores, ensuring they sum to 1. These
scores, representing attention weights, compute a weighted sum of the values
in $V$, resulting in the final attention output. This mechanism allows the
model to dynamically focus on the most relevant parts of the input sequence
for making predictions.

  * •

Multi-Head Attention: In the Transformer model, multiple attention heads
enhance the model’s capability to simultaneously focus on different parts of
the input sequence. The multi-head attention is calculated as follows:

$\text{MHead}(Q,K,V)=\text{Concat}(hd_{1},hd_{2},\ldots,hd_{h})W^{O}$ (13)

Where: $hd_{i}$ represents the output of the $i$-th attention head, computed
using the attention formula $\text{Attention}(Q_{i},K_{i},V_{i})$. Each
$Q_{i}$, $K_{i}$, and $V_{i}$ are different linear projections of the original
inputs $Q$, $K$, and $V$. $W^{O}$ is a linear transformation matrix applied to
the concatenated results of all attention heads. The Concat function
concatenates the outputs of each head along a specific dimension. The outputs
of individual heads, $hd_{i}$, are each computed using the scaled dot-product
attention mechanism:

$hd_{i}=\text{Attention}(Q_{i},K_{i},V_{i})=\text{softmax}\left(\frac{Q_{i}K_{i}^{T}}{\sqrt{d_{k}}}\right)V_{i}$
(14)

This approach enables the Multi-Head Attention mechanism to capture various
aspects of the input sequence, simultaneously focusing on different subspace
representations. As a result, it facilitates the model’s capture of more
complex relationships and improves performance across different types of
tasks.

  * •

Layer Normalization: In the Transformer model, layer normalization ensures the
input is within a standard range. The layer normalization can be calculated as
follows:

$LN(x)=\frac{x-\text{mean}(x)}{\sqrt{\text{var}(x)}}$ (15)

Where $x$ is the input to the layer normalization. $textmean(x)$ and
$textvar(x)$ are the mean and variance of $x$, respectively. Layer
Normalization aims to mitigate the internal covariate shift, which arises when
the distribution of activations in a layer changes during training. The
normalization operation is performed by subtracting the mean of the
activations and dividing by the square root of the variance. This ensures the
activations have a zero mean and unit variance, leading to more stable
training.

  * •

Position-wise Feed Forward: The position-wise feed-forward network transforms
the input and output representations in the Transformer model. The position-
wise feedforward can be calculated as follows:

$FFN(x)=\max(0,xW_{1}+b_{1})W_{2}+b_{2}$ (16)

Where: $x$ is the input to the feed-forward network. $W_{1}$, $b_{1}$,
$W_{2}$, and $b_{2}$ are the weight and bias parameters of the feed-forward
network. $\max(0,x)$ is the ReLU activation function. This equation represents
a simple feed-forward neural network (FFN) operation in deep learning models.
The FFN operation is a multi-layer perceptron (MLP) that transforms the input
$x$ into a new representation by passing it through two fully connected
(dense) layers. The first layer is followed by a ReLU activation function,
which applies a non-linear activation to the input by setting all negative
values to zero. This activation function helps the model learn complex non-
linear relationships between the input and output. The second layer is a
linear transformation that produces the final output of the FFN. The weight
and bias parameters of the two layers, $W_{1}$, $b_{1}$, $W_{2}$, and $b_{2}$,
are learned during training and allow the model to learn different
representations of the input data.

  * •

Encoder and Decoder Blocks: In the Transformer model, the encoder and decoder
blocks transform the input sequences into the output sequences. The encoder
and decoder blocks can be calculated as follows:

$\text{Enc}(x)=\text{LN}(x+\text{MHead}(x,x,x))$ (17)

$\text{Dec}(x,y)=\text{LN}(x+\text{MHead}(x,y,y)+\text{MHead}(x,x,x))$ (18)

Where: $x$ is the input to the encoder/decoder block. $y$ is the output from
the previous encoder/decoder block. The Encoder block $\text{Enc}(x)$ takes
the input $x$ and applies the Multi-Head Attention mechanism to compute the
attention scores between the input and itself. The result is then added to the
input and passed through a Layer Normalization operation. The output of the
encoder block is the new representation of the input after processing through
the Multi-Head Attention and Layer Normalization operations. The Decoder block
$\text{Dec}(x,y)$ is similar to the encoder block but also takes the output
from the previous decoder block, $y$, as input. The Multi-Head Attention
mechanism is applied to compute the attention scores between the input and the
previous output and between the input and itself. The results are added to the
input and passed through a Layer Normalization operation. The output of the
decoder block is the new representation of the input after processing through
the Multi-Head Attention and Layer Normalization operations.

TABLE II: LLMs-based models for Cyber Security (Part I). Study |  Year |  Type of Model |  Dataset Used |  Domain |  Key Contributions |  Open Issues
---|---|---|---|---|---|---
Parra et al. [51] |  2022 |  Federated Transformer Log Learning Model |  HDFS and CTDD datasets |  Threat detection and forensics |  The interpretability module integrated into the model provides insightful interpretability of the model’s decision-making process |  The paper briefly mentions the applicability of the proposed approach in edge computing systems but does not discuss the scalability of the approach to larger systems
Ziems et al. [52] |  2021 |  Transformer Model, BERT, CANINE, Bagging-based random transformer forest (RTF) |  Malware family datasets |  Malware Classification |  Demonstration that transformer-based models outperform traditional machine and deep learning models in classifying malware families |  The experiments are conducted on preprocessed NIST NVD/SARD databases, which may not reflect real-world conditions
Wu et al. [53] |  2022 |  Robust Transformer-based Intrusion Detection System (RTID) |  CICID2017 and CIC-DDoS2019 datasets |  Intrusion Detection |  The proposed method outperforms classical machine learning algorithms such as support vector machine (SVM) and deep learning algorithms (i.e., RNN, FNN, LSTM) on the two evaluated datasets |  There is no discussion in the paper regarding the scalability of the proposed method, particularly when dealing with large-scale and real-time network traffic
Demirkıran et al. [54] |  2022 |  Transformer-based models |  Catak dataset, Oliveira dataset, VirusShare dataset, and VirusSample dataset |  Malware classification |  The paper demonstrates that transformer-based models, specifically BERT and CANINE, outperform traditional machine and deep learning models in classifying malware families |  The study only focuses on malware families that use API call sequences, which means that it does not consider other malware types that may not use API calls
Yin et al. [55] |  2017 |  RNN-ID (Recurrent Neural Network-Intrusion Detection) |  Benchmark data set |  Intrusion Detection |  The proposed model can improve the accuracy of intrusion detection |  Other machine learning algorithms and deep learning models, such as convolutional neural networks and transformers are not considered in the comparison
Güera et al. [56] |  2018 |  Temporal-aware Pipeline (CNN and RNN) |  Large set of deepfake videos collected from multiple video websites |  Detection of Deepfake Videos |  The proposed method achieves competitive results in detecting deepfake videos while using a simple architecture |  The proposed approach’s effectiveness might be limited to the specific types of deepfakes present in the dataset
Althubiti et al. [57] |  2018 |  LSTM RNN |  CSIC 2010 HTTP dataset |  Web Intrusion Detection |  Proposal of LSTM RNN for web intrusion detection. High accuracy rate (0.9997) in binary classification. |  The paper only uses the CSIC 2010 HTTP dataset, which may not be representative of all types of web application attacks
Xu et al. [58] |  2018 |  GRU-MLP-Softmax (Gated Recurrent Unit, Multilayer Perceptron, Softmax) |  KDD 99 and NSL-KDD data sets |  Network Intrusion Detection |  The system achieves leading performance with overall detection rates of 99.42% using KDD 99 and 99.31% using NSL-KDD, with low false positive rates |  The paper does not provide information about the scalability of the proposed model
Ferrag and Leandros [59] |  2019 |  DeepCoin (Blockchain and Deep Learning) |  CICID2017 dataset, Power system dataset, Bot-IoT dataset |  Energy framework for Smart Grids |  Proposal of DeepCoin framework combining blockchain and deep learning for smart grid security |  The paper does not address the potential scalability issues that may arise as the number of nodes in the network increases
Ghourbi et al. [60] |  2022 |  An optimized LightGBM model and a Transformer-based model |  ToN-IoT and Edge IIoTset datasets |  Threat Detection |  The experimental evaluation of the approach showed remarkable accuracies of 99% |  The paper does not discuss the scalability of the proposed system for large-scale healthcare networks
Thapa et al. [61] |  2022 |  Transformer-based language models |  Software vulnerability datasets of C/C++ source codes |  Software security and vulnerability detection in programming languages, specifically C/C++ |  The paper highlights the advantages of transformer-based language models over contemporary models |  The paper only focuses on detecting vulnerabilities in C/C++ source code and does not explore the use of large transformer-based language models in detecting vulnerabilities in other programming languages
Ranade et al. [62] |  2021 |  A transformer-based language model, specifically GPT-2 |  WebText dataset |  Fake Cyber Threat Intelligence |  The attack is shown to introduce adverse impacts such as returning incorrect reasoning outputs |  Further research is needed to explore how to prevent or detect data poisoning attacks on cyber-defense system
Fu et al. [63] |  2022 |  Transformer-based line-level vulnerability prediction model |  Large-scale real-world dataset with more than 188k C/C++ functions |  Software vulnerability prediction in safety-critical software systems |  The proposed system is accurate for predicting vulnerable functions affected by the Top-25 most dangerous CWEs |  The model’s performance can be changed when applied to different programming languages or software systems
Mamede et al. [64] |  2022 |  A transformer-based deep learning model |  Software Assurance Reference Dataset (SARD) project, which contains vulnerable and non-vulnerable Java files |  Software security in the context of Java programming language |  The proposed system can identify up to 21 vulnerability types and achieved an accuracy of 98.9% in multi-label classification |  The proposed method cannot be extended to other programming languages and integrated into existing software development processes
Evange et al. [65] |  2021 |  A transformer-based model |  DNRTI (Dataset for NER in Threat Intelligence) |  Cybersecurity threat intelligence |  The experimental results demonstrate that transformer-based techniques outperform previous state-of-the-art approaches for NER in threat intelligence |  Further research is needed to test the effectiveness of transformer-based models on larger and more diverse datasets
Hashemi et al. [66] |  2023 |  Transformer models (including BERT, XLNet, RoBERTa, and DistilBERT) |  Labeled dataset from vulnerability databases |  Vulnerability Information Extraction |  The proposed approach outperforms existing rule-based and CRF-based models |  The paper does not address the issue of bias in the labeled dataset
TABLE III: LLMs-based models for Cyber Security (Part II). Study |  Year |  Type of Model |  Dataset Used |  Domain |  Key Contributions |  Open Issues
---|---|---|---|---|---|---
Chawla et al. [67] |  2019 |  GRU with CNN |  ADFA (Australian Defence Force Academy) dataset |  Intrusion Detection |  Achieved improved performance by combining GRUs and CNNs |  The proposed system is vulnerable to adversarial attacks
Ullah et al. [68] |  2022 |  LSTM, BiLSTM, and GRU |  IoT-DS2, MQTTset, IoT-23, and datasets |  Intrusion Detection |  Validation of the proposed models using various datasets, achieving high accuracy, precision, recall, and F1 score |  Further research is necessary to enhance their scalability for practical applications in cybersecurity
Donkol et al. [69] |  2023 |  LSTM |  CSE-CIC-IDS2018, CICIDS2017, and UNSW-NB15 datasets |  Intrusion Detection |  The proposed system outperformed other methods such as LPBoost and DNNs in terms of accuracy, precision, recall, and error rate |  Future research could explore the applicability of the proposed system to other datasets
Zhao et al. [70] |  2023 |  End-to-End Recurrent Neural Network |  IDS2017 and IDS2018 datasets |  Intrusion attacks and malware |  \+ Address network-induced phenomena that may result in misclassifications in traffic detection systems used in cybersecurity |  The proposed system is vulnerable to adversarial attacks
Wang et al. [71] |  2021 |  RNN |  A large-scale patch dataset PatchDB |  Software Security |  The PatchRNN system can effectively detect secret security patches with a low false positive rate |  The PatchRNN system can only support C/C++
Polat et al. [72] |  2022 |  LSTM and GRU |  SDN-based SCADA system |  Detection of DDoS attacks |  The results show that the proposed RNN model achieves an accuracy of 97.62% for DDoS attack detection |  The paper only focuses on detecting DDoS attacks and does not address other types of cyber threats (e.g., insider threats or advanced persistent threats)
Liu et al. [73] |  2022 |  Transformer model |  A commit benchmark dataset that includes over 7.99 million commits across 7 programming languages |  Commit message generation (generation task) and security patch identification (understanding task) |  The experimental results demonstrate that CommitBART significantly outperforms previous pre-trained models for code |  The pre-training dataset used in the paper is limited to GitHub commits
Ahmad et al. [74] |  2024 |  Transformer model |  Set of 15 hardware security bug benchmark designs from three sources: MITRE website, OpenTitan System-on-Chip (SoC) and the Hack@DAC 2021 SoC |  Hardware Security Bugs |  Bug repair potential demonstrated by ensemble of LLMs, outperforming state-of-the-art automated tool |  The need for designer assistance in bug identification, handling complex bugs, limited evaluations due to simulation constraints, and challenges with token limits and repair generation using LLMs
Wan et al. [75] |  2024 |  Transformer model |  Chrysalis dataset, comprising over 1,000 function-level HLS designs with injected logical bugs |  Design Verification |  Creating the Chrysalis dataset for HLS debugging, and enabling LLM-based bug detection and integration into development environments |  Refining LLM techniques, integrating LLMs into development environments, and addressing scalability and generalization challenges
Jang et al. [76] |  2024 |  Transformer model |  Includes 150K online security articles, 7.3K security paper abstracts, 3.4K Wikipedia articles, and 185K CVE descriptions. |  Threat Detection |  Pre-trained language model for the cybersecurity domain, CyBERTuned incorporates non-linguistic elements (NLEs) such as URLs and hash values commonly found in cybersecurity texts. |  The paper’s limitations include a narrow focus on specific non-linguistic element (NLE) types, acknowledging the existence of more complex NLE types like code blocks and file paths that require future exploration
Bayer et al. [77] |  2024 |  Transformer model |  A dataset consisting of 4.3 million entries of Twitter, Blogs, Paper, and CVEs related to the cybersecurity domain |  Intrusion attacks and malware |  Created a high-quality dataset and a domain-adapted language model for the cybersecurity domain, which improves the internal representation space of domain words and performs best in cybersecurity scenarios |  the model may not be suitable as a replacement for every type of cybersecurity model. They also state that the hyperparameters may not be generalizable to other language models, especially very large language models
Shestov et al. [78] |  2024 |  Transformer model |  The dataset comprises 22,945 function-level source code samples. It includes 13,247 samples for training, 5,131 for validation, and 4,567 for testing |  Vulnerability detection |  Finetuning the state-of-the-art code LLM, WizardCoder, increasing its training speed without performance harm. |  The proposed study shows that the main bottlenecks of the task that limit performance lie in the field of dataset quality and suggests the usage of the project-level context information
He et al. [79] |  2024 |  Transformer model |  Used three datasets: one with over 100,000 entries from Ethereum mainnet contracts, another with 892,913 addresses labelled across five vulnerability categories, and a third with 6,498 smart contracts, including 314 associated with Ponzi schemes |  blockchain technology and smart contracts |  The introduction of a novel model, BERT-ATT-BiLSTM, for advanced vulnerability detection in smart contracts, and the evaluation of its performance against other models |  Include the model’s limitation in recognizing unseen contract structures or novel types of vulnerabilities, and the need to incorporate support for multiple programming languages to enhance universality and robustness
Jamal et al. [21] |  2024 |  Transformer model |  Two open-source datasets, 747 spam, 189 phishing, 4825 ham; class imbalance addressed with ADASYN |  Phishing and spam detection |  Proposing IPSDM, a fine-tuned version of DistilBERT and RoBERTA, outperforming baseline models and the demonstration of the effectiveness of LLMs in addressing cybersecurity challenges |  Class imbalance, addressed with ADASYN, but potential bias remains
Figure 4: LLM-based Solutions for Cyber Security Use Cases.

##  IV LLMs-based models for Cyber Security

This section reviews recent studies employing LLM-based models (i.e.,
Recurrent Neural Networks-based and transformer-based models) for threat
detection, malware classification, intrusion detection, and software
vulnerability detection. Tables II, III, and Figure 4 present the LLM-based
solutions for Cyber Security Use Cases.

###  IV-A Recurrent Neural Networks-based models

####  IV-A1 Intrusion Detection

Yin et al. [55] propose a deep learning approach for intrusion detection using
recurrent neural networks (RNN-ID) and study its performance in binary and
multiclass classification tasks. The results show that the RNN-ID model
outperforms traditional machine learning methods in accuracy. Chawla et al.
[67] presented an anomaly-based intrusion detection system that leverages
recurrent neural networks (RNNs) with gated recurrent units (GRUs) and stacked
convolutional neural networks (CNNs) to detect malicious cyber attacks. The
system establishes a baseline of normal behavior for a given system by
analyzing sequences of system calls made by processes. It identifies anomalous
sequences based on a language model trained on normal call sequences from the
ADFA dataset of system call traces. The authors demonstrate that using GRUs
instead of LSTMs results in comparable performance with reduced training times
and that combining GRUs with stacked CNNs leads to improved anomaly detection.
The proposed system shows promising results in detecting anomalous system call
sequences in the ADFA dataset. However, further research is needed to evaluate
its performance in other datasets and real-world scenarios and address issues
related to adversarial attacks.

Ullah et al. [68] introduce the deep learning models to tackle the challenge
of managing cybersecurity in the growing realm of IoT devices and services.
The models utilize Recurrent Neural Networks, Convolutional Neural Networks,
and hybrid techniques to detect anomalies in IoT networks accurately. The
proposed models are validated using various datasets (i.e., IoT-DS2, MQTTset,
IoT-23, and datasets) and achieve high accuracy, precision, recall, and F1
score. However, the models need to be tested on more extensive and diverse
datasets, and further research is necessary to enhance their scalability for
practical applications in cybersecurity. Donkol et al. [69] presents a
technique, ELSTM-RNN, for improving security in intrusion detection systems.
Using likely point particle swarm optimization (LPPSO) and enhanced LSTM
classification, the proposed system addresses gradient vanishing,
generalization, and overfitting issues. The system uses an enhanced particle
swarm optimization technique to select efficient features, which are used for
effective classification using an enhanced LSTM framework. The proposed system
outperformed other methods, such as LPBoost and DNNs, in accuracy, precision,
recall, and error rate. The NSL-KDD dataset was used for validation and
testing, and further verification was done on other datasets. While the paper
provides a comprehensive solution, future research could explore the
applicability of the proposed system to other datasets and real-world
scenarios. Additionally, a more detailed analysis of the computational cost of
the proposed system compared to other methods could be beneficial.

Zhao et al. [70] presents ERNN, an end-to-end RNN model with a novel gating
unit called session gate, designed to address network-induced phenomena that
may result in misclassifications in traffic detection systems used in
cybersecurity. The gating unit includes four types of actions to simulate
network-induced phenomena during model training and the Mealy machine to
adjust the probability distribution of network-induced phenomena. The paper
demonstrates that ERNN outperforms state-of-the-art methods by 4% accuracy and
is scalable in terms of parameter settings and feature selection. The paper
also uses the Integrated Gradients method to interpret the gating mechanism
and demonstrates its ability to reduce dependencies on local packets.
Althubiti et al. [57] propose a deep learning-based intrusion detection system
(IDS) that uses a Long Short-Term Memory (LSTM) RNN to classify and predict
known and unknown intrusions. The experiments show that the proposed LSTM-
based IDS can achieve a high accuracy rate of 0.9997. Xu et al. [58] propose a
novel IDS that consists of a recurrent neural network with gated recurrent
units (GRU), multilayer perceptron (MLP), and softmax module. The experiments
on the KDD 99 and NSL-KDD data sets show that the system has a high overall
detection rate and a low false positive rate. Ferrag and Leandros [59] propose
a novel deep learning and blockchain-based energy framework for smart grids,
which uses a blockchain-based scheme and a deep learning-based scheme for
intrusion detection. The deep learning-based scheme employs recurrent neural
networks to detect network attacks and fraudulent transactions in the
blockchain-based energy network. The performance of the proposed IDS is
evaluated using three different data sources.

Polat et al. [72] introduce a method for improving the detection of DDoS
attacks in SCADA systems that use SDN technology. The authors propose using a
Recurrent Neural Network (RNN) classifier model with two parallel deep
learning-based methods: Long Short-Term Memory (LSTM) and Gated Recurrent
Units (GRU). The proposed model is trained and tested on a dataset from an
experimentally created SDN-based SCADA topology containing DDoS attacks and
regular network traffic data. The results show that the proposed RNN model
achieves an accuracy of 97.62% for DDoS attack detection, and transfer
learning further improves its performance by around 5%.

####  IV-A2 Software Security

Wang et al. [71] propose a deep learning-based defense system called PatchRNN
to automatically detect secret security patches in open-source software (OSS).
The system leverages descriptive keywords in the commit message and syntactic
and semantic features at the source-code level. The system’s performance was
evaluated on a large-scale real-world patch dataset and a case study on NGINX.
The results indicate that the PatchRNN system can effectively detect secret
security patches with a low false positive rate.

####  IV-A3 Detection of Deepfake Videos

Güera et al. [56] propose a temporal-aware pipeline that automatically detects
deepfake videos by using a convolutional neural network (CNN) to extract
frame-level features and a recurrent neural network (RNN) to classify the
videos. The results show that the system can achieve competitive results in
this task with a simple architecture.

Overall, the reviewed studies demonstrate the potential of deep learning
methods, particularly RNNs, for intrusion detection in various domains. The
results show that the proposed deep learning-based models outperform
traditional machine learning methods in accuracy. However, more research is
needed to address the limitations and challenges associated with these
approaches, such as data scalability and interpretability.

###  IV-B Transformer-based models

####  IV-B1 Cloud Threat Forensics

Parra et al. [51] proposed an interpretable federated transformer log learning
model for threat detection in syslogs. The model is generated by training
local transformer-based threat detection models at each client and aggregating
the learned parameters to generate a global federated learning model. The
authors demonstrate the difference between normal and abnormal log time series
through the goodness of fit test and provide insights into the model’s
decision-making process through an attention-based interpretability module.
The results from the HDFS and CTDD datasets validate the proposed approach’s
effectiveness in achieving threat forensics in real-world operational
settings. Evange et al. [65] discuss the importance of actionable threat
intelligence in defending against increasingly sophisticated cyber threats.
Cyber Threat Intelligence is available on various online sources, and Named
Entity Recognition (NER) techniques can extract relevant information from
these sources. The paper investigates the use of transformer-based models in
NER and how they can facilitate the extraction of cybersecurity-related named
entities. The DNRTI dataset, which contains over 300 threat intelligence
reports, tests the effectiveness of transformer-based models compared to
previous approaches. The experimental results show that transformer-based
techniques are more effective than previous methods in extracting
cybersecurity-related named entities.

####  IV-B2 Malware classification

Ziems et al. [52] explore transformer-based models for malware classification
using API call sequences as features. The study compares the performance of
the traditional machine and deep learning models with transformer-based
models. It shows that transformer-based models outperform traditional models
in terms of F1-score and AUC score. The authors also propose a bagging-based
random transformer forest (RTF) model that reaches state-of-the-art evaluation
scores on three out of four datasets. Demirkıran et al. [54] proposes using
transformer-based models for classifying malware families, better suited for
capturing sequence relationships among API calls than traditional machine and
deep learning models. The experiments show that the proposed transformer-based
models outperform traditional models such as LSTM and pre-trained models such
as BERT or CANINE in classifying highly imbalanced malware families based on
evaluation metrics like F1-score and AUC score. Additionally, the proposed
bagging-based random transformer forest (RTF) model, an ensemble of BERT or
CANINE, achieves state-of-the-art performance on three out of four datasets,
including a state-of-the-art F1-score of 0.6149 on one of the commonly used
benchmark datasets.

####  IV-B3 Intrusion Detection

Wu et al. [53] proposed an RTID that reconstructs feature representations in
imbalanced datasets to make a trade-off between dimensionality reduction and
feature retention. The proposed method utilizes a stacked encoder-decoder
neural network and a self-attention mechanism for network traffic type
classification. The results with CICID2017 and CIC-DDoS2019 datasets
demonstrate the proposed method’s effectiveness in intrusion detection
compared to classical machine learning and deep learning algorithms. Ghourbi
et al. [60] propose an intrusion and malware detection system to secure the
entire network of the healthcare system independently of the installed devices
and computers. The proposed solution includes two components: an intrusion
detection system for medical devices installed in the healthcare network and a
malware detection system for data servers and medical staff computers. The
proposed system is based on optimized LightGBM and Transformer-based models.
It is trained with four different datasets to ensure a varied knowledge of the
different attacks affecting the healthcare sector. The experimental evaluation
of the approach showed remarkable accuracies of 99%.

####  IV-B4 Software Vulnerability Detection

Thapa et al. [61] explores the use of large transformer-based language models
in detecting software vulnerabilities in C/C++ source code, leveraging the
transferability of knowledge gained from natural language processing. The
paper presents a systematic framework for source code translation, model
preparation, and inference. It conducts an empirical analysis of software
vulnerability datasets to demonstrate the good performance of transformer-
based language models in vulnerability detection. The paper also highlights
the advantages of transformer-based language models over contemporary models,
such as bidirectional long short-term memory and bidirectional gated recurrent
units, in terms of F1-score. However, the paper does not discuss the
limitations or potential drawbacks of using transformer-based language models
for software vulnerability detection, and further research is needed in this
area. Fu et al. [63] propose an approach called LineVul, which uses a
Transformer-based model to predict software vulnerabilities at the line level.
The approach is evaluated on a large-scale dataset (i.e., on a large-scale
real-world dataset with more than 188k C/C++ functions). It achieves higher
F1-measure for function-level predictions and higher Top-10 accuracy for line-
level predictions compared to baseline approaches. The analysis also shows
that LineVul accurately predicts vulnerable functions affected by the top 25
most dangerous CWEs. However, the model’s performance can be changed when
applied to different programming languages or software systems.

Mamede et al. [64] presented a transformer-based VS Code extension that uses
state-of-the-art deep learning techniques for automatic vulnerability
detection in Java code. The authors emphasize the importance of early
vulnerability detection within the software development life cycle to promote
application security. Despite the availability of advanced deep learning
techniques for vulnerability detection, the authors note that these techniques
are not yet widely used in development environments. The paper describes the
architecture and evaluation of the VDet tool, which uses the Transformer
architecture for multi-label classification of up to 21 vulnerability types in
Java files. The authors report an accuracy of 98.9% for multi-label
classification and provide a demonstration video, source code, and datasets
for the tool.

Liu et al. [73] introduce CommitBART, a pre-trained Transformer model
specifically designed to understand and generate natural language messages for
GitHub commits. The model is trained on a large dataset of over 7.99 million
commits, covering seven different programming languages, using a variety of
pre-training objectives, including denoising, cross-modal generation, and
contrastive learning, across six pre-training tasks. The authors propose a
”commit intelligence” framework encompassing one understanding task and three
generation tasks for commits. The experimental results demonstrate that
CommitBART significantly outperforms previous pre-trained models for code, and
the analysis suggests that each pre-training task contributes to the model’s
performance.

Ding et al. [80] discuss the effectiveness of code language models (code LMs)
in detecting vulnerabilities. It identifies significant issues in current
datasets, such as poor quality, low accuracy, and high duplication rates,
which compromise model performance in realistic scenarios. To overcome these
challenges, it introduces the PrimeVul dataset, which uses advanced data
labeling, de-duplication, and realistic evaluation metrics to represent real-
world conditions accurately. The findings reveal that current benchmarks, like
the BigVul, greatly overestimate code LMs’ capabilities, with much lower
performance observed on PrimeVul. This significant discrepancy highlights the
need for further innovative research to meet the practical demands of
deploying code LMs in security-sensitive environments.

####  IV-B5 Cyber Threat Intelligence

Ranade et al. [62] presented a method for automatically generating fake Cyber
Threat Intelligence (CTI) using transformers, which can mislead cyber-defense
systems. The generated fake CTI is used to perform a data poisoning attack on
a Cybersecurity Knowledge Graph (CKG) and a cybersecurity corpus. The attack
introduces adverse impacts such as returning incorrect reasoning outputs,
representation poisoning, and corruption of other dependent AI-based cyber
defense systems. A human evaluation study was conducted with cybersecurity
professionals and threat hunters, which reveals that professional threat
hunters were equally likely to consider the generated fake CTI and authentic
CTI as true.

Hashemi et al. [66] propose an alternative approach for automated
vulnerability information extraction using Transformer models, including BERT,
XLNet, RoBERTa, and DistilBERT, to extract security-related words and terms
and phrases from descriptions of vulnerabilities. The authors fine-tune
several language representation models similar to BERT on a labeled dataset
from vulnerability databases for Named Entity Recognition (NER) to extract
complex features without requiring domain-expert knowledge. This approach
outperforms the CRF-based models and can detect new information from
vulnerabilities with different description text patterns. The authors conclude
that this approach provides a structured and unambiguous format for disclosing
and disseminating vulnerability information, which is crucial for preventing
security attacks.

####  IV-B6 Phishing and spam detection

Koide et al. introduced [81], a novel system leveraging LLMs to detect
phishing emails. Despite advances in traditional spam filters, significant
challenges such as oversight and false positives persist. The system
transforms email data into prompts for LLM analysis, achieving a high accuracy
rate (99.70%) and providing detailed reasoning for its determinations. This
helps users make informed decisions about suspicious emails, potentially
enhancing the effectiveness of phishing detection.

Jamal et al. [21] explored the potential of LLMs to address the growing
sophistication of phishing and spam attacks. Their work, IPSDM, is an improved
model based on the BERT family, specifically fine-tuned to detect phishing and
spam emails. Compared to baseline models, IPSDM shows superior accuracy,
precision, recall, and F1-score performance on both balanced and unbalanced
datasets while addressing overfitting concerns.

Heiding et al. [82] compared automatically generated phishing emails by GPT-4,
manually designed emails using the V-Triad method, and their combination.
Their findings suggest that emails designed with the V-Triad achieved the
highest click-through rates, indicating the effectiveness of exploiting
cognitive biases. The study also evaluated the capability of four different
LLMs to detect phishing intentions, with results often surpassing human
detection. Furthermore, they discuss the economic impact of AI in lowering the
costs of orchestrating phishing attacks.

Chataut et al. [83] focused on the effectiveness of LLMs in detecting phishing
emails amidst threat actors’ constant evolution of phishing strategies. Their
study emphasizes the necessity for continual development and adaptation of
detection models to keep pace with innovative phishing techniques. The role of
LLMs in this context highlights their potential to significantly enhance email
security by improving detection capabilities.

####  IV-B7 Hardware Security Evaluation

Ahmad et al. [74] delves into leveraging LLMs to automatically repair
identified security-relevant bugs present in hardware designs, explicitly
focusing on Verilog code. Hardware security bugs pose significant challenges
in ensuring the reliability and safety of hardware designs. They curated a
corpus of hardware security bugs through a meticulously designed framework.
They explored the performance of various LLMs, including OpenAI’s Codex and
CodeGen, in generating replacement code to fix these bugs. The experiments
reveal promising results, demonstrating that LLMs can effectively repair
hardware security bugs, with success rates varying across different bugs and
LLM models. By optimizing parameters such as instruction variation,
temperature, and model selection, they achieved successful repairs for a
significant portion of the bugs in their dataset. In addition, the results
demonstrate that LLMs, including GPT-4, code-davinci-002, and code-
cushman-001, yield successful repairs for simple security bugs, with GPT-4
achieving a success rate of 67% at variation e, temp 0.5. However, LLMs’
performance varies across bugs, showing success rates over 75% with some bugs,
while others are more challenging to repair, with success rates below 10%. The
study emphasizes the importance of detailed prompt instructions, with
variation d showing the highest success rate among OpenAI LLMs. Further
investigation is needed to evaluate LLMs’ scalability and effectiveness for
diverse hardware security bug scenarios. Their findings underscore the
potential of LLMs in automating the bug repair process in hardware designs,
marking a crucial step towards developing automated end-to-end bug repair
tools for hardware security.

Mohamadreza et al. [84] explored the potential of using large language models
to enhance the input generation in the process of hardware design verification
for security-related bugs. Mohamadreza et al. introduced Chatfuzz, a novel ML-
based hardware fuzzer that leverages LLMs and reinforcement learning to
generate complex and random machine code sequences for exploring processor
security vulnerabilities. Chatfuzz introduces a specialized LLM model into a
hardware fuzzing approach to enhance the input generation quality,
outperforming the existing approaches regarding coverage, scalability, and
efficiency. Utilizing LLMs to understand processor language and generate
data/control flow entangled machine code sequences, Chatfuzz integrates RL to
guide input generation based on code coverage metrics. Their experiment on
real-world cores, namely RocketCore and BOOM cores, showed significantly
faster coverage than state-of-the-art hardware fuzzes. ChatFuzz achieves 75%
condition coverage in RocketCore in 52 minutes and 97.02% in BOOM in 49
minutes, identifying unique mismatches and new bugs and showcasing its
effectiveness in hardware security testing.

Weimin et al. [85] introduces LLM4SECHW, a novel framework for hardware
debugging that utilizes domain-specific Large Language Models. The authors
addressed the limitations of out-of-the-shelf LLMs in the hardware security
domain by gathering a dataset of hardware design defects and remediation
steps. The collected dataset has been built by leveraging open-sourced
hardware designs from GitHub; the data consists of different Hardware
Description Language modules with their respective commits. By harnessing
version control information from open-source hardware projects and processing
it to create a debugging-oriented dataset, LLM4SECHW fine-tunes hardware
domain-specific language models to locate and rectify bugs autonomously,
enhancing bug localization. LLM4SECHW has been evaluated with two objectives:
bug identification and design patching. The authors demonstrated that non-
fine-tuned LLMs lack hardware domain knowledge, which makes them incapable of
locating bugs in the hardware design of a popular security-specialized chip
project named OpenTitan. The based models (falcon 7b, llama 2, Bard, chatbot,
and stableLM) did not efficiently locate the introduced hardware bugs. The
three fine-tuned models (falcon 7b, llama2, stableLM) successfully located the
introduced bugs in the hardware design.

Zhang et al. [85] introduces Hardware Phi-1.5B, a large language model
tailored for the hardware domain of the semiconductor industry, addressing the
complexity of hardware-specific issues. The research focused on developing
datasets specifically for the hardware domain to enhance the model’s
performance in comprehending complex terminologies. The authors claim to
surpass general code language models and natural language models like
CodeLlama, BERT, and GPT-2 in the Hardware understanding tasks.

Madhav et al. [86] evaluated the security of the HDL code generated by
ChatGPT. The authors introduced a similar taxonomy to the NIST CWE . The
authors conducted various experiments to explore the impact of prompt
engineering on the security of the generated hardware design.

####  IV-B8 Hardware design & Verification

Lily et al. [87] introduced the application of LLMs into High-Level Synthesis
Design Verification (HLS). The authors created a dataset named Chrysalis to
solve the problem of the non-existence of specialized HLS bug detection and
evaluation capabilities. The Chrysalis dataset comprises over 1000 function-
level designs extracted from reputable sources with intentionally injected
known bugs to evaluate and refine LLM-based HLS bug localization. The set of
the introduced bugs was selected based on the most common human coding errors
and has been shaped to elude most of the existing conventional HLS synthesis
tools detection mechanisms. The paper’s authors suggest that Chrysalis would
contribute to the LLM-aided HLS design verification by offering a benchmarking
to the existing and specialized models. The paper also suggests a prompt
engineering approach that would enhance the efficiency of a large language
model on the studied task. The proposed prompt structure introduces a
separation of concern approach, where the used prompt deals with each class of
bugs separately. The prompt starts by explicitly defining the context of the
task, the functional description, the implementation context, and the task
objective. The prompt is implemented through three main sections: context,
requirements, and complementary rules. The highlighted works lay a foundation
for a methodological, practical approach to benchmarking, evaluating, and
deploying LLM tasks for HLS design verification. While the paper does not
provide any conclusive results about LLMs’ performance in such tasks, the
authors believe that such methodology would accelerate the adoption of new
techniques to integrate LLMs into the design verification flow.

Mingjie et al. [88] evaluated the LLMs’ performance in solving Verilog related
design tasks and generating design testbenchs by introducing VerilogEval.
VerilogEval comprises different hardware design tasks ranging from module
implementation of simple combinatorial circuits to complex finite state
machines, code debugging, and testbench construction. VerilogEval suggests an
end-to-end evaluation framework that fits better in the context of the
hardware design verification process benchmarking. The VerilogEval framework
validates the correctness of the prompted tasks by comparing the behavior
simulation to an established golden model of the prompted design. The authors
used pass@k metric instead of the generic NLP related metrics like the BLEU
score probability metric. The study demonstrates that pre-trained language
models’ Verilog code generation capabilities can be improved through
supervised fine-tuning. The experimental results show that fine-tuning LLMs on
the hardware design tasks and using the pass@k metric helps assess the
performance of the resulting models properly. The pass@k metric helps assess
the performance of Large Language Models (LLMs) in Verilog code generation by
quantifying the number of successful code completions out of k samples,
offering a clear evaluation criterion. The used metric shows that a fine-tuned
model could have equal or better performance than the state-of-the-art OpenAI
models (gpt-3 and gpt-4). VerilogEval highlights the growing significance of
Large Language Models (LLMs) and their application in various domains,
emphasizing their potential in Verilog code generation for hardware design and
verification. The findings underscore the importance of the proposed
benchmarking framework in advancing the state of the art in Verilog code
generation, highlighting the vast potential of LLMs in assisting the hardware
design and verification process.

Figure 5: LLMs Performance Steps in the cybersecurity domain using CyberMetric
Dataset [89].

####  IV-B9 Protocols verification

Ruijie et al. [90] introduced ChatAFL, an LLM-based protocol fuzzer. ChatAFL
introduces an LLM-guided protocol fuzzing to address the challenge of finding
security flaws in protocol implementations without a machine-readable
specification. The study suggests three strategies for integrating an LLM into
a mutation-based protocol fuzzer, focusing on grammar extraction, seed
enrichment, and saturation handling to enhance code coverage and state
transitions. ChatAFL prototype implementation demonstrates that the LLM-guided
stateful fuzzer outperforms state-of-the-art fuzzers like AFLNET [91] and
NSFUZZ [92] in terms of protocol state space coverage and code coverage.

The experiments evaluated CHATAFL’s improvement over the baselines in terms of
transition coverage achieved in 24 hours, speed-up in achieving the same
coverage, and the probability of outperforming the baselines in a random
campaign. CHATAFL demonstrated significant efficacy by covering 47.60% and
42.69% more state transitions, 29.55% and 25.75% more states, and 5.81% and
6.74% more code than AFLNET and NSFUZZ, respectively.

CHATAFL discovered nine unique and previously unknown vulnerabilities in
widely used and extensively tested protocol implementations on real widely
used projects (live555, proFTPD, kamailio). The discovered vulnerabilities
encompass various memory vulnerabilities, including use-after-free, buffer
overflow, and memory leaks, which have potential security implications such as
remote code execution or memory leakage. The study demonstrated the
effectiveness of utilizing LLMs for guiding protocol fuzzing to enhance state
and code coverage in protocol implementations.

Wang et al. [93] introduced LLMIF and LLM-aided fuzzing approach for IoT
devices protocols. LLMIF introduces an LLM augmentation-based approach. The
developed pipeline incorporates an enhanced seed generation strategy by
building an augmentation based on domain knowledge. The domain knowledge
structure is extracted from the various specifications of the under-fuzzing
protocol. The flow starts by selecting a seed from the extracted augmentation
set and then enriching the extracted seed by exploring the protocol
specification. The enriching process is driven by the various ranges of input
values extracted during the augmentation phase. Furthermore, LLMIF introduces
a coverage approach by mutating the selected seed through the various
enrichment and mutation operators that have been selected.

The evaluation part of LLMIF mainly aimed to evaluate three axes: code
coverage, ablation, and bug identification. The authors used an out-of-the-
shelf popular SOC (CC2530) for the evaluation. 11 commercial devices have been
selected to conduct the various experiments. While the ablation and bug
detection could be easily evaluated, the code coverage is impossible using the
custom firmware that ships with the selected devices. The authors used an
open-source Zigbee stack to demonstrate the coverage capabilities. The authors
claimed that LLMIF outperforms Z-FUZZER [94], and BOOFUZZ [95] in terms of
code coverage for the target Zigbee stack. The authors claim that LLMIF
achieved a notable increase in protocol message coverage and code coverage by
55.2% and 53.9%, respectively, outperforming other Zigbee fuzzers in these
aspects.

LLMIF algorithm successfully uncovered 11 vulnerabilities on real-world Zigbee
devices, including eight previously unknown vulnerabilities, showcasing its
effectiveness in identifying security flaws in IoT devices. By incorporating
the large language model into IoT fuzzing, LLMIF demonstrated enhanced
capabilities in protocol message coverage and vulnerability discovery,
highlighting its potential for improving the security testing of IoT devices.

TABLE IV: Comparison of 19 LLMs Models’ Performance in Hardware Security
Knowledge.

LLM model | Size | Design bug detection | Hardware CWE Number
---|---|---|---
1245 | 1221 | 1224 | 1298 | 1254 | 1209 | 1223 | 1234 | 1231
Llama 3-7b-instruct | 8B | 39.556% | Yes | No | No | No | Yes | No | No | Yes | No
Mixtral-8x7B-Instruct | 8x7B | 16.154% | No | No | No | No | No | No | No | No | No
Dolphin-mistral-7B | 7B | 16,024% | Yes | Yes | No | No | No | No | No | No | No
Codegemma-9b-instruct | 9B | 10.746% | No | No | No | No | No | No | No | No | Yes
CodeQwen-7b-instruct | 7B | 10.269% | No | No | No | No | No | No | No | No | No
Wizard-vicuna-uncensored-7b-instruct | 7B | 9.374% | No | No | No | No | No | No | No | No | No
Mistral-openorca-7b-instruct | 7B | 8.241% | No | No | No | No | No | Yes | No | No | No
Wizardlm2-7b-instruct | 7B | 5.646% | No | No | No | No | No | No | No | No | No
Llama2-uncensored-7b-instruct | 7B | 2.505% | No | No | No | No | No | No | No | No | No
Falcon-40b-instruct | 40B | 1.620% | No | No | No | No | No | No | No | No | No
Deepseek-coder-33b-instruct | 33B | 1.570% | No | No | No | No | No | No | No | No | No
Orca-mini-3b-instruct | 3B | 1.173% | No | No | Yes | No | No | No | No | No | No
Qwen2-4b-instruct | 4B | 0.576% | No | No | No | No | No | No | No | No | No
CodeLlama-7b-instruct | 7B | 0.218% | No | No | No | No | No | No | No | No | No
Phi3-4b-instruct | 4B | 0.019% | No | No | No | No | No | No | No | No | No
Hardware-Phi | 1.5B | 0% | No | No | No | No | No | No | No | No | No
Llava-13b-instruct | 13B | 0% | No | No | No | No | No | No | No | No | No
Gemma-9b-instruct | 9B | 0% | No | No | No | No | No | No | No | No | No
Starcoder2-15b-instruct | 15B | 0% | No | No | No | No | No | No | No | No | No

Yes: Detected the CWE sample by MITRE, No: Did not Detect the CWE sample by
MITRE. CWE: Common Weakness Enumeration.

TABLE V: Comparison of 42 LLMs Models’ Performance in Cyber Security Knowledge. LLM model | Company | Size | License | Accuracy
---|---|---|---|---
80 Q | 500 Q | 2k Q | 10k Q
GPT-4o | OpenAI | N/A | Proprietary | 96.25% | 93.40% | 91.25% | 88.89%
GPT-4-turbo | OpenAI | N/A | Proprietary | 96.25% | 93.30% | 91.00% | 88.50%
Mixtral-8x7B-Instruct | Mistral AI | 45B | Apache 2.0 | 92.50% | 91.80% | 91.10% | 87.00%
Falcon-180B-Chat | TII | 180B | Apache 2.0 | 90.00% | 87.80% | 87.10% | 87.00%
GEMINI-pro 1.0 | Google | 137B | Proprietary | 90.00% | 85.05% | 84.00% | 87.50%
GPT-3.5-turbo | OpenAI | 175B | Proprietary | 90.00% | 87.30% | 88.10% | 80.30%
Yi-1.5-9B-Chat | 01-ai | 9B | Apache 2.0 | 87.50% | 80.80% | 77.15% | 76.04%
Hermes-2-Pro-Llama-3-8B | NousResearch | 8B | Open | 86.25% | 80.80% | 77.95% | 77.33%
Dolphin-2.8-mistral-7b-v02 | Cognitive Computations | 7B | Apache 2.0 | 83.75% | 77.80 % | 76.60% | 75.01%
Mistral-7B-OpenOrca | Open-Orca | 7B | Apache 2.0 | 83.75% | 80.20% | 79.00% | 76.71 %
Gemma-1.1-7b-it | Google | 7B | Open | 82.50% | 75.40% | 75.75% | 73.32%
Flan-T5-XXL | Google | 11B | Apache 2.0 | 81.94% | 71.10% | 69.00% | 67.50%
Meta-Llama-3-8B-Instruct | Meta | 8B | Open | 81.25 % | 76.20% | 73.05% | 71.25%
Zephyr-7B-beta | HuggingFace | 7B | MIT | 80.94% | 76.40% | 72.50% | 65.00%
Yi-1.5-6B-Chat | 01-ai | 6B | Apache 2.0 | 80.00% | 75.80% | 75.70% | 74.84%
Mistral-7B-Instruct-v0.2 | Mistral AI | 7B | Apache 2.0 | 78.75% | 78.40% | 76.40% | 74.82%
Llama 2-70B | Meta | 70B | Apache 2.0 | 75.00% | 73.40% | 71.60% | 66.10%
Qwen1.5-7B | Qwen | 7B | Open | 73.75% | 60.60% | 61.35% | 59.79%
Qwen1.5-14B | Qwen | 14B | Open | 71.25% | 70.00% | 72.00% | 69.96%
Mistral-7B-Instruct-v0.1 | Mistral AI | 7B | Apache 2.0 | 70.00% | 71.80% | 68.25% | 67.29%
Llama-3-8B-Instruct-Gradient-1048k | Bartowski | 8B | Open | 66.25% | 58.00% | 56.30% | 55.09%
Qwen1.5-MoE-A2.7B | Qwen | 2.7B | Open | 62.50% | 64.60% | 61.65% | 60.73%
Phi-2 | Microsoft | 2.7B | MIT | 53.75% | 48.00% | 52.90% | 52.13%
Llama3-ChatQA-1.5-8B | Nvidia | 8B | Open | 53.75% | 52.80% | 49.45 % | 49.64%
DeciLM-7B | Deci | 7B | Apache 2.0 | 52.50% | 47.20% | 50.44% | 50.75%
Flan-T5-Base | Google | 0.25B | Apache 2.0 | 51.25% | 50.40% | 48.55% | 47.09%
Deepseek-moe-16b-chat | Deepseek | 16B | MIT | 47.50% | 45.80% | 49.55% | 48.76%
Mistral-7B-v0.1 | Mistral AI | 7B | Apache 2.0 | 43.75% | 39.40% | 38.15% | 39.28%
Qwen-7B | Qwen | 7B | Open | 43.75% | 58.00% | 55.75% | 54.09%
Gemma-7b | Google | 7B | Open | 42.50% | 37.20% | 36.00% | 34.28%
Meta-Llama-3-8B | Meta | 8B | Open | 38.75% | 35.80% | 37.00% | 36.00%
Genstruct-7B | NousResearch | 7B | Apache 2.0 | 38.75% | 40.60% | 37.55% | 36.93%
Qwen1.5-4B | Qwen | 4B | Open | 36.25% | 41.20% | 40.50% | 40.29%
Llama-2-13b-hf | Meta | 13B | Open | 33.75% | 37.00% | 36.40% | 34.49%
Dolly V2 12b BF16 | Databricks | 12B | MIT | 33.75% | 30.00% | 28.75% | 27.00%
Deepseek-llm-7b-base | DeepSeek | 7B | MIT | 33.75% | 25.20% | 27.00% | 26.48%
Cerebras-GPT-2.7B | Cerebras | 7B | Apache 2.0 | 25.00% | 20.20% | 19.75% | 19.27%
Gemma-2b | Google | 2B | Open | 25.00% | 23.20% | 18.20% | 19.18%
Stablelm-2-1_6b | Stability AI | 6B | Open | 16.25% | 21.80% | 19.55% | 20.09%
ZySec-7B | ZySec-AI | 7B | Apache 2.0 | 12.50% | 16.40% | 15.55% | 14.04%
Phi-3-mini-4k-instruct | Microsoft | 3.8B | MIT | 5.00% | 5.00% | 4.41% | 4.80%
Phi-3-mini-128k-instruct | Microsoft | 3.8B | MIT | 1.25% | 0.20% | 0.70% | 0.88%

##  V General LLMs

Tables VI, VII, VIII compare general transformer-based Large Language Models.
LLM models are generally trained on a diverse and broad range of data to
provide a relatively comprehensive understanding. They can handle various
language tasks like translation, summarization, and question-answering. In
contrast, code-specific LLMs are specialized models trained primarily on
programming languages and related technical literature, which makes their
primary role in understanding and generating programming code well-suited for
tasks like automated code generation, code completion, and bug detection.

###  V-A Prevalent LLMs

####  V-A1 GPT-3

GPT-3 (the third version of the Generative Pre-trained Transformer series by
OpenAI) was developed to prove that scaling language models substantially
improves their task-agnostic few-shot performance [96]. Based on transformer
architecture, GPT-3 has eight variants ranging between 125M and 175B
parameters, all trained for 300B tokens from datasets like Common Crawl,
WebText, Books, and Wikipedia. Additionally, the models were trained on V100
GPU leveraging techniques like autoregressive training, scaled cross-entropy
loss, and others. GPT-3, especially its most capable 175B version, has
demonstrated strong performance on many NLP tasks in different settings (i.e.,
zero-shot, one-shot, and few-shots), suggesting it could significantly improve
cybersecurity applications if appropriately fine-tuned. This could translate
to more effective Phishing Detection through precise language analysis, faster
Incident Response, and other critical applications to enhance digital security
measures.

####  V-A2 GPT-4

In 2023, the GPT-4 transformer-based model was released by OpenAI as the first
large-scale multimodal model, exhibiting unprecedented performance in various
benchmarks. The model’s capability of processing image and text inputs has
shifted the AI paradigm to a new level, expanding beyond traditional NLP. [97]
declared that GPT-4 was trained using a vast corpus of web-based data and data
licensed from third-party sources with autoregressive techniques and
Reinforcement Learning from Human Feedback (RLHF). However, other specifics,
such as the model size, data size, and comprehensive training details, remain
undisclosed. Although GPT-4 could potentially be leveraged by cybercriminals
for a wide range of attacks, such as social engineering, if implemented
strategically, it can also help reduce the likelihood of individuals and
organizations falling prey to them.

####  V-A3 T5

Motivated by the trend of applying transfer learning for NLP, researchers of
Google have introduced T5 [98], an encoder-decoder-based model that operates
within the unified text-to-text framework. Multiple variants of T5 with
different sizes - ranging between 220M to 11B parameters- were developed to
broaden the experimental scope and were trained on massive amounts of data
from various sources, including C4, Web Text, and Wikipedia. Building on the
foundation of these diverse model sizes and rich data sources, multiple
approaches and different settings for pre-training and fine-tuning were
examined and discussed, achieving performance that nearly matched human levels
on one of the benchmarks. Considering that, the model’s potential in
cybersecurity applications is particularly promising. For instance, T5 can be
utilized for threat intelligence by extracting critical information from vast
security documents and then summarizing and organizing that information.

####  V-A4 BERT

Bidirectional Encoder Representations from Transformers, commonly known as
BERT, was presented by [99] to enhance fine-tuning-based approaches in NLP. It
is available in two versions: BERT-Base, with 110M parameters, and BERT-Large,
with 340M parameters, trained on 126GB of data from BooksCorpus and English
Wikipedia. During its pre-training phase, BERT employed two key techniques:
Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). Building on
these approaches, fine-tuning, and feature-based methods have led to
competitive performance from BERT-Large in particular. Since encoder-only
models like BERT are known for their robust contextual understanding, applying
such models to tasks like malware detection and software vulnerability can be
highly effective in cybersecurity.

####  V-A5 ALBERT

Aiming to address the limitations related to GPU/TPU memory and training time
in Large Language Models (LLMs), Google researchers developed A Lite BERT
(ALBERT), a modified version of BERT with significantly fewer parameters
[100]. And like other LLMs, ALBERT was introduced in various sizes, with
options ranging from 12M to 235M parameters, all trained on data from
BooksCorpus and English Wikipedia. Various methods and techniques were
deployed during the pre-training stage, including Factorized Embedding
Parameterization, Cross-layer Parameter Sharing, Inter-sentence Coherence
Loss, and Sentence Order Prediction (SOP). As a result, one of the models
(i.e., ALBERT-xxlarge) outperformed BERT-Large despite having fewer
parameters. Thus, utilizing ALBERT in cybersecurity applications, such as
phishing detection and malware classification, could significantly contribute
to advancing cybersecurity infrastructure.

####  V-A6 RoBERTa

RoBERTa, proposed by Meta, is an optimized replication of BERT that
demonstrates how the choice of hyperparameters can significantly impact the
model’s performance [101]. RoBERTa has only one version with 355M parameters
but is trained and tested in various data sizes and training steps. Similar to
BERT, the training data was taken from the Books corpus and English Wikipedia.
However, the key optimizations in this model were in the training techniques,
which included multiple methods such as Dynamic Masking, training on Full
Sentences without NSP loss, using Large Mini-Batches, and employing a Larger
Byte-Level BPE. Consequently, RoBERTa achieved state-of-the-art results in
some of the benchmarks. With proper fine-tuning, RoBERTa’s ability to
understand, interpret, and generate human-like text is leveraged to automate
and enhance various tasks in the realm of cybersecurity.

####  V-A7 XLNet

The advances and limitations of Masked Language Modeling (MLM) in
bidirectional encoders and Autoregressive Language Modeling have inspired
researchers at CMU and Google AI to develop XLNet [102]. Based on the
Transformer-XL model, XLNet combines aspects of both approaches, enabling the
learning of bidirectional contexts while addressing common MLM issues, such as
neglecting dependencies between masked positions and the discrepancy between
pretraining and finetuning phases. With 340M parameters, XLNet was pre-trained
using data from English Wikipedia and utilizing techniques like Permutation
Language Modeling (PLM), Two-stream attention, Segment Recurrence, and
Relative Encoding. Due to the careful design of the model and strategic pre-
training techniques, XLNet has achieved substantial performance over other
popular models like BERT, making it -after appropriate fine-tuning- a capable
tool for enhancing various aspects of the cybersecurity field.

####  V-A8 ProphetNet

ProphetNet LLM, proposed by Microsoft, is a sequence-to-sequence pre-trained
model that aims to address the issue of overfitting on strong local
correlations by leveraging two novel techniques, namely: future n-gram
prediction and n-stream self-attention [103]. Built on an encoder-decoder
architecture and trained on 16GB base-scale and 160GB large-scale datasets
sourced from web data and books, ProphetNet, with its 550M parameters,
achieved new state-of-the-art results on multiple benchmarks. The model was
also fine-tuned for two downstream tasks, Question Generation and Text
Summarization, where it achieved the best performance. Therefore, utilizing
ProphetNet in cybersecurity tasks such as automated security incident
summarization could significantly enhance efficiency and decision-making.

####  V-A9 Falcon

Falcon LLM, built on decoder-only architecture, was introduced by the
Technology Innovation Institute (TII) as a proof-of-concept that enhancing
data quality can significantly improve the LLM performance even with purely
web-sourced data [104]. This insight is increasingly relevant as scaling in
LLMs, which is becoming more prevalent, requires more data for processing. The
model has three versions (i.e., 7B, 40B, 180B) pre-trained on the “RefinedWeb”
dataset proposed by TII. RefinedWeb, sourced exclusively from web data, was
subjected to various filtering and deduplication techniques to ensure high
quality. Autoregressive training, Flash Attention, and ALiBi Positional
encoding were the methods used for pre-training. With further fine-tuning,
Falcon can advance cybersecurity, particularly in threat intelligence and
analysis.

####  V-A10 Reformer

Striving to address common memory limitations in LLMs, Google proposed the
Reformer, an encoder-decoder memory-efficient LLM [105]. With up to 6B
parameters, Reformer was pre-trained on web data using techniques including
Locality-Sensitive Hashing (LSH) Attention, Chunked Processing, Shared-QK
Attention Heads, and Reversible layers. These techniques were proven to have a
negligible impact on the training process compared to the standard
Transformer, as the Reformer achieved results that matched the full
Transformer but with much faster processing and better memory efficiency.
Subsequently, employing Reformer for tasks like large-scale data analysis
could serve the cybersecurity field by enabling more efficient processing and
analysis of extensive datasets.

####  V-A11 PaLM

Driven by the advancement in machine learning and natural language processing,
Google has developed PaLM to examine the impact of scale on few-shot learning
[106]. PaLM, built on decoder-only architecture, was trained with 540B
parameters using Pathways, a new system that utilizes highly efficient
training across multiple TPU pods. The model was trained on 2TB of data from
multiple sources, including news articles, Wikipedia, source code, etc. SwiGLU
Activation, Parallel Layers, and other techniques were deployed for pre-
training three different parameter scales, 8B, 62B, and 540B, to understand
the scaling behavior better. An observed discontinuous improvement indicated
that as LLMs reach a certain level of scale, they exhibit new abilities.
Furthermore, these emerging capabilities continue to evolve and become
apparent even beyond the scales that have been previously explored and
documented. Subsequently, PaLM achieved a breakthrough by outperforming the
finetuned state-of-the-art and average human on some benchmarks, proving that
when scaling is combined with chain-of-thought prompting, basic few-shot
evaluation has the potential to equal or surpass the performance of fine-tuned
state-of-the-art models across a broad spectrum of reasoning tasks. With such
strong capabilities, utilizing PaLM for tasks like generating security
policies and incident response automation can enhance the efficiency and
effectiveness of cybersecurity operations.

####  V-A12 PaLM2

PaLM2 is an advanced variant of the PaLM model that is more compute-efficient,
although it offers better multilingual and reasoning capabilities [107]. The
key enhancements in the model are the improved dataset mixtures, the compute-
optimal scaling, and architectural and objective improvements. The significant
evaluation results of PaLM2 indicate that various approaches could elaborate
on the model’s enhancement besides scaling, such as meticulous data selection
and efficient architecture/objectives. Moreover, the fact that PaLM2
outperformed the predecessor PaLM despite its significantly smaller size shows
that the model quality has a greater influence on the performance than the
model size as it could enable more efficient inference, reducing serving costs
and potentially allowing for broader applications and accessibility to more
users.

####  V-A13 LLaMA

Proposed by Meta, the LLaMA decoder-only model is a proof-of-concept that it’s
possible to achieve state-of-the-art performance by training exclusively on
publicly available data [108]. LLaMA, with multiple variants ranging between 7
and 65 billion parameters, was trained on 1400B tokens of publicly available
datasets, including CommonCrawl, C4, arXiv, and others. Interestingly, the
techniques used for training the model were inspired by multiple popular
models like GPT-3 (Pre-normalization), PaLM (SwiGLU activation function), and
GPTNeo (Rotary Embedding). As a result of this incorporation, LLaMA-13B was
able to outperform GPT-3(175B) on most benchmarks despite it being more than
ten times smaller, while LLaMA-65B has shown to be competitive with
Chinchilla-70B and PaLM-540B. Given its relatively small size and superior
performance, fine-tuning LLaMA on cyber threat intelligence tasks could
significantly enhance the security of edge devices.

####  V-A14 LLaMA2

LLaMA2 is an optimized version of LLaMA developed by Meta and a collection of
pre-trained and fine-tuned LLMs with sizes ranging from 7 to 70B parameters
[109]. In the pre-training, a mixture of publicly available data was used for
up to 2000B training tokens. Moreover, multiple techniques were used in the
predecessor LLaMA, such as Pre-normalization, SwiGLU activation function, and
Rotary positional embeddings. Two additional methods, namely increased context
length and group-query attention (GQA), were also used. After pre-training,
variants of the model (i.e., LLaMA2-Chat) were optimized for dialog use cases
by supervised fine-tuning and reinforcement learning with human feedback
(RLHF). The model evaluation, which focused on helpfulness and safety, showed
superiority over the other open-source models and competitive performance to
some closed-source models.

####  V-A15 GShard

GShard LLM was introduced by Google in 2020, aiming to address neural network
scaling issues related to computation cost and training efficiency [110].
Based on a Mixture-of-Experts (MoE) transformer with 600B parameters, GShard
was pre-trained on 1000B tokens of web data. Multiple techniques were deployed
for the training stage, such as conditional computation, XLA SPMD
partitioning, position-wise MoE, and parallel execution using annotation APIs.
Subsequently, GShard outperformed prior models in translation tasks and
exhibited a favorable trade-off between scale and computational cost,
resulting in a practical and sample-efficient model. These results highlight
the importance of considering training efficiency when scaling LLMs, which
makes it more viable in the real world.

####  V-A16 ELECTRA

The extensive computation cost of MLM pre-training methods has inspired Google
to propose ELECTRA LLM, which is a 335M parameters’ encoder-only transformer
model that utilizes a novel pre-training approach called “replaced token
detection” [111]. This technique allows the model to learn from the entire
sequence rather than just a small portion of masked tokens. Given that the
quality and diversity of ELECTRA training data play a pivotal role in its
ability to generalize across tasks, the model was trained on a vast Books
Corpus and English Wikipedia. Pre-training techniques were utilized, including
replaced token detection, generator-discriminator framework, token
replacement, and weight-sharing. As a result, ELECTRA was able to perform
comparably to popular models like RoBERTa and XLNet when using less than 25%
of their compute and outperform them when using equivalent compute. Deploying
such a robust model in the security field after fine-tuning can provide an
efficient solution for detecting and mitigating sophisticated cyber threats,
thanks to its nuanced understanding of context and language patterns.

####  V-A17 MPT-30B

MPT-30B LLM is a decoder-only transformer introduced by MosaicML after the
notable success of MPT-7B [112]. The model has multiple variants, the base
model and two fine-tuned variants, namely MPT-30B-Instruct and MPT-30B-Chat.
Training the model on a variety of datasets such as C4, CommonCrawl, and
arXiv, among others, besides the strategic selection of pre-training methods
like FlashAttention and ALiBi positional encoding, have contributed to a
robust performance, surpassing even the original GPT-3 benchmarks. MPT-30B has
also significantly performed in programming tasks, outperforming some open-
source models designed specifically for code generation. With these
capabilities, deploying MPT-30B in cybersecurity could substantially enhance
threat detection and response systems. Its adeptness at understanding and
generating programming languages promises advancements in automated
vulnerability assessment and developing sophisticated security protocols.

####  V-A18 Yi-34B

The newly released LLM Yi-34B developed by 01.AI is getting attention as one
of the best open-source LLMs [113]. Given the recent release of the model, its
technical paper has not yet been published; hence, the available information
is limited. The model has multiple variants: base and chat models, some
quantized. All variants are trained on a dataset containing Chinese and
English only, and the chat versions have gone through supervised fine-tuning,
resulting in more efficient models for downstream tasks. The base model
outperformed many open LLMs in certain benchmarks, including renowned ones
like LLaMA2-70B and Falcon-180B. Even the quantized versions have demonstrated
impressive performance, paving the way for their deployment in cybersecurity
applications, such as edge security solutions.

####  V-A19 Falcon2-11B

Falcon2-11B LLM [114] built by TII, is a decoder-only model with 11 billion
parameters, trained on an immense corpus of text data totaling over 5,000
billion tokens. In terms of performance, Falcon2-11B showcases impressive
capabilities, supporting 11 languages: English, German, Spanish, French,
Italian, Portuguese, Polish, Dutch, Romanian, Czech, and Swedish. While it
excels in generating human-like text, it also carries the biases and
stereotypes prevalent in its training data, a common challenge LLMs face. To
address this, TII recommends fine-tuning the model for specific tasks and
implementing guardrails for production use. In the training process of
Falcon2-11B, they utilized a four-stage strategy with increasing context
lengths; in the final stage, they reached 8162 context lengths. This stage
focused on enhancing performance using high-quality data. Additionally, the
training leveraged 1024 A100 40GB GPUs and a custom distributed training
codebase named Gigatron, which employs a 3D parallelism approach combined with
ZeRO, high-performance Triton kernels, and FlashAttention-2 for efficient and
effective training.

###  V-B LLMs performance in the hardware cybersecurity

Table IV compares 19 publicly available LLMs’ performance in Hardware design-
related bug detection and security issues identification using samples from
various sources. A portion of the Chrystalis dataset [87] has been used to
evaluate the performance of the LLM models in bug detection tasks. A set of
faults has been injected intentionally into a functional code and labeled as
faulty. The sample size that has been processed comprises 10K of hardware
design-related code samples. The prompt that has been used instructs the model
to check the concerned code for any issue or bug and respond only with yes or
no. The result is presented as the ratio of the responses where the model
successfully identified a buggy code from the total samples used. The hardware
CWE column evaluates the capability of the models to link a buggy code to its
CWE number. The prompt has been designed to ask for a well-defined CWE number
on the buggy design. This evaluation process asses the capability of an LLM
model in bug detection and classification into the correct CWE class number.

The top performers in this evaluation in terms of design bug detection are
LLama3 and Mixtral. While the LLama3 model performs better in the bug
detection tasks, it lacks the proper identification of the CWE issue related
to the faulty section. Mixtral models show less performance at identifying
bugs but higher diversity in identifying a bug’s security impact on the
overall design implementation. The outcomes of this experiment reveal that
some models cannot identify the right issues with the source code, which might
require further refinement of the used prompt and/or fine-tuning the general-
purpose models on bug-locating tasks. The results also show that the model
size doesn’t greatly impact the model performance at locating the bugs nor
reasoning about their according impact (CWE class identification). While the
samples that have been picked do not exceed the context length of the selected
models, the token size of the model itself might reveal a superiority for the
larger models when dealing with large source codes. However, superior bug
identification and reasoning are also required to provide the required
performance.

In conclusion, the highlighted results reveal that the existing models might
be subject to weaknesses in identifying bugs in Hardware designs that might
lead to security-related issues. The two-step evaluation process gives better
visibility in building more robust dedicated LLMs for Hardware design security
evaluation. Models that properly locate bugs do not show similar performance
in classifying the bug’s impact on the overall design. The outcomes could be
evaluated with a larger sample size and a more dedicated study at a large
scale to get conclusive results.

###  V-C LLMs performance in the cybersecurity knowledge

Table V compares various 42 LLMs performance in the cybersecurity domain using
CyberMetric dataset [89]. Figure 5 presents the LLMs performance steps. The
models are evaluated based on their accuracy across four question sets: 80
questions, 500 questions, 2000 questions, and 10,000 questions. The
performance is represented in percentage accuracy, offering a comprehensive
view of each model’s proficiency in handling cybersecurity-related queries.

The top performers in this evaluation are the GPT-4 and GPT-4-turbo models by
OpenAI. These models demonstrate exceptional performance, with GPT-4 achieving
96.25% accuracy on the 80-question set and maintaining high accuracy with
88.89% on the 10,000-question set. GPT-4-turbo closely follows with similar
accuracy percentages. Both models are proprietary and developed by OpenAI,
indicating a high optimization level for specialized tasks within a controlled
environment. Another strong performer is the Mixtral-8x7B-Instruct by Mistral
AI, which boasts accuracy of 92.50% on the 80-question set and 87.00% on the
10,000-question set. This model is open-source under the Apache 2.0 license,
demonstrating the potential of community-driven development in achieving high
performance. Additionally, GEMINI-pro 1.0 by Google shows robust performance,
achieving 90.00% accuracy on the 80-question set and 87.50% on the
10,000-question set, highlighting the capabilities of large-scale corporate
research and development in LLMs.

Mid-tier performers include models like Yi-1.5-9B-Chat by 01-ai and
Hermes-2-Pro-Llama-3-8B by NousResearch. Yi-1.5-9B-Chat performs reasonably
well with an 87.50% accuracy on the 80-question set, tapering to 76.04% on the
10,000-question set. Under the Apache 2.0 license, this model shows a balance
between open-source collaboration and performance. Hermes-2-Pro-Llama-3-8B
achieves 86.25% accuracy on the 80-question set and 77.33% on the
10,000-question set, further underscoring the effectiveness of collaborative
research efforts.

Lower-tier performers include models like Qwen1.5-7B by Qwen. Qwen1.5-7B
scores 73.75% on the 80-question set, dropping to 59.79% on the
10,000-question set. As an open model, Qwen1.5-7B indicates the challenges
faced by smaller models in maintaining high accuracy with increasing question
set sizes. Falcon-40B-Instruct achieves 67.50% accuracy on the 80-question set
and 64.50% on the 10,000-question set. Licensed under Apache 2.0, it