# Review of Generative AI Methods in Cybersecurity

Yagmur Yigit  William J Buchanan  Madjid G Tehrani  Leandros Maglaras

###### Abstract

Over the last decade, Artificial Intelligence (AI) has become increasingly
popular, especially with the use of chatbots such as ChatGPT, Google’s Gemini,
and DALL-E. With this rise, large language models (LLMs) and Generative AI
(GenAI) have also become more prevalent in everyday use. These advancements
strengthen cybersecurity’s defensive posture and open up new attack avenues
for adversaries as well. This paper provides a comprehensive overview of the
current state-of-the-art deployments of GenAI, covering assaults,
jailbreaking, and applications of prompt injection and reverse psychology.
This paper also provides the various applications of GenAI in cybercrimes,
such as automated hacking, phishing emails, social engineering, reverse
cryptography, creating attack payloads, and creating malware. GenAI can
significantly improve the automation of defensive cyber security processes
through strategies such as dataset construction, safe code development, threat
intelligence, defensive measures, reporting, and cyberattack detection. In
this study, we suggest that future research should focus on developing robust
ethical norms and innovative defense mechanisms to address the current issues
that GenAI creates and also further to encourage an impartial approach to its
future application in cybersecurity. Moreover, we underscore the importance of
interdisciplinary approaches further to bridge the gap between scientific
developments and ethical considerations.

###### keywords:

Generative AI, GPT-4, Gemini, Cybersecurity.

##  1 Introduction

The past decade has witnessed a transformative leap in the digital domain,
significantly impacted by advancements in Artificial Intelligence (AI), Large
Language Models (LLMs), and Natural Language Processing (NLP). Starting with
the basics of supervised learning, AI and Machine Learning (ML) have rapidly
expanded into more complex territories, including unsupervised, semi-
supervised, reinforcement, LLM, NLP and deep learning techniques [1]. The most
recent breakthrough in this evolution is the emergence of Generative AI
(GenAI) technologies. These technologies make use of deep learning networks to
analyse and understand the patterns within huge datasets, enabling them to
create new content that resembles the original data. GenAI is versatile enough
to produce a wide array of content, such as text, visuals, programming code,
and more. In the cybersecurity domain, GenAI’s impact is significant, offering
new dimensions to the field. It is anticipated that GenAI will enhance the
capabilities of vulnerability scanning tools, offering a depth of
vulnerability analysis that surpasses traditional Static Application Security
Testing (SAST) methods [2]. This evolution is promising for future cyber
security practices, enhanced by the capabilities of GenAI [3]. Innovations
like Google’s Gemini and OpenAI’s Chat-Generative Pre-trained Transformer
(ChatGPT) are at the forefront of this advancement.

Yandex has integrated a next-generation large language model, YandexGPT, into
its virtual assistant Alice [4], making it the first company globally to
enhance a virtual assistant with the ability to generate human-like text and
brainstorm ideas, accessible through various devices and applications. The
main aim of some GenAI tools is to help people with their abilities,
sometimes, they show the opposite behaviour, like Microsoft’s chatbot Tay.
After the launch, Microsoft’s chatbot Tay was taken offline due to offensive
tweets resulting from a vulnerability exploited by a coordinated attack,
prompting the company to address this issue and improve the AI with lessons
learned from previous experiences, including those with XiaoIce in China, to
ensure future interactions reflect the best of humanity without offending [5].
Moreover, some GenAI tools have been developed for different purposes. For
example, MIT’s Norman, the world’s first AI described as a psychopath [6], was
trained using captions from a controversial subreddit, emphasising how biased
data can lead AI to interpret images with disturbing captions revealing the
impact of data on AI behaviour [7].

GenAI has experienced a notable transformation in recent years, marked by
exceptional innovations and rapid advancements [8] [9]. The AI timeline
started with the emergence of AI as a conceptual scientific discipline in the
1940s and 1950s. The ELIZA chatbot, created between the 1960s and 1970s, was
the first GenAI that achieved notoriety. This revolutionary demonstration
highlighted the capacity of robots to imitate human speech. The development of
AI in analysing sequential data and patterns got more complex and, therefore,
more effective in the 80s and 90s, as advanced methods for pattern recognition
became more popular. The first variational autoencoder (VAE) exhibited
exceptional proficiency in natural language translation. OpenAI developed GPT
between the 2000s and 2010s. GenAI models were simultaneously developed, and
in the 2020s, a number of innovative platforms and technologies were
introduced, including DALL-E, Google’s Gemini, Falcon AI, and Open AI’s GPT-4.
These advancements represent the discipline’s maturing, enabling unprecedented
capabilities for content production, problem-solving, and emulating human
intelligence and creativity. They also pave the way for further advancements
in this subject. The development timeline of GenAI can be seen in Fig. 1.

Figure 1: The timetable for GenAI development.

Language models are essential in many sectors, including commerce, healthcare,
and cybersecurity. Their progress shows a concrete path from basic statistical
methods to sophisticated neural networks [10], [11]. NLP skill development has
benefited immensely from the use of LLMs. However, despite these advancements,
a number of issues remain, including moral quandaries, the requirement to
reduce error rates, and making sure that these models are consistent with our
ethical values. To solve these issues, moral monitoring and ongoing
development are required.

###  1.1 The Challenges of GenAI

Mohammed _et al._ [12] define key challenges of the use of ChatGPT in
cybersecurity, which include analyzing ChatGPT’s impact on cybersecurity,
building honeypots, improving code security, abuse in malware development,
investigating vulnerabilities, spreading misinformation, cyberattacks on
industrial systems, modifying the cyber threat environment, modifying
cybersecurity techniques, and evolution of human-centric training. Alawida et
al. [13] also highlight issues related to GenAI’s ability to generate data
that should be kept private, including medical details, financial data and
personal information.

Innovative methods such as the Mixture of Experts (MoE) architecture offer
increased specialization and efficiency. The difficulty of maintaining
transparency and ethics in AI systems is also emphasized [14]. It highlights
the need for strong governance structures and interdisciplinary collaboration
to fully exploit the advantages of lifelong learning settings and handle their
limitations.

In an extensive progress report on AI Principles, Google outlines their
dedication to responsible AI development, highlighting the incorporation of AI
governance into all-encompassing risk management frameworks [15]. Together
with other significant legislative actions, this strategic strategy aims to
adhere to current international norms and rules, such as the EU’s AI Act and
the US Executive Order on AI safety. The report also highlights the need for
scientific stringency in AI development through cautious internal management
and the use of tools like digital watermarking and GenAI system cards in order
to promote AI accountability and transparency. Multi-stakeholder solutions are
needed to address the ethical, security, and social challenges that AI
technology is currently bringing forth.

Google’s Gemini and ChatGPT-4 are the most popular and widely utilized GenAI
technologies. Following ethical and safety criteria, ChatGPT-4 by OpenAI can
now generate responses that are both coherent and contextually acceptable
[16]. This is because its NLP skills have significantly improved. Its capacity
to identify new conversions to chemical compounds and to negotiate tricky
legal and moral territory highlights its potential as a pivotal instrument for
content moderation and scientific inquiry. Google introduces Gemini, the most
recent iteration of Bard, a ground-breaking development in AI technology [17].
It can process text, code, audio, images, and video and sets new standards for
AI’s capabilities, emphasising flexibility, safety, and ethical AI
developments. With ChatGPT-4, we also see the rise in AI’s capabilities in the
creation of mathematical assistants that can interpret and render mathematical
equations [18].

###  1.2 Related Works

Some studies in the literature focus on GenAI tools and their performance. For
instance, Brown _et al._ have extended the NLP processing by training GPT-3,
an autoregressive language model with 175 billion parameters, indicating
exceptional few-shot learning capabilities [19]. Without task-specific
training, this model performs well on various NLP tasks, like translation and
question-answering. It often matches or surpasses state-of-the-art fine-tuned
systems. Romera-Paredes _et al._ have developed FunSearch, an innovative
approach combining LLMs with evolutionary algorithms to make significant
findings in domains like extremal combinatorics and algorithmic problem-
solving [20]. Their method has notably surpassed previous best-known results
by iteratively refining programs that solve complex problems, revealing LLMs’
potential for scientific innovation. This process generates new knowledge and
produces interpretable and deployable solutions, proving a notable advancement
in applying LLMs for real-world challenges. Lu _et al._ critically examine the
capabilities and limitations of multi-modal LLMs, including proprietary models
like GPT-4 and Gemini, as well as six open-source counterparts across text,
code, image, and video modalities [21]. Through a comprehensive qualitative
analysis of 230 cases, assessing generalizability, trustworthiness, and causal
reasoning, the study reveals a significant gap between the performance of the
GenAIs and public expectations. These discoveries open up new avenues for
study to improve the transparency and dependability of GenAI in cybersecurity
and other fields, providing a basis for creating more complex and reliable
multi-modal applications. In another work, commonsense thinking across
multimodal tasks is evaluated exhaustively, and Google’s Gemini is compared
with OpenAI’s GPT models [22]. This study explores the strengths and
weaknesses of Gemini’s ability to synthesize commonsense information,
indicating areas for improvement in its competitive performance in temporal
reasoning, social reasoning, and emotion recognition of images. It emphasizes
how important it is for GenAI models to develop commonsense reasoning to
improve cybersecurity applications.

Recent research [23] presents a novel approach for assessing the potentially
severe hazards associated with GenAI models, such as deceit, manipulation, and
cyber-offence features. To enable AI developers to make well-informed
decisions about training, deployment, and the application of cybersecurity
standards, the suggested methodology highlights the need to increase
evaluation benchmarks to assess the harmful capabilities and alignment of AI
systems accurately. Another work [24] provides a thorough analysis that
inspired the complex applications of ChatGPT in digital forensic
investigations, pointing out the important constraints and promising
opportunities that come with GenAI as it is now. Using methodical
experimentation, they outline the fine line separating AI’s inventive
contributions from the vital requirement of professional human supervision in
cybersecurity procedures, opening the door to additional research into
integrating LLMs such as GPT-4 into digital forensics and cybersecurity.

The latest release of CyberMetric presents a novel benchmark dataset that
assesses the level of expertise of LLMs in cybersecurity, covering a broad
range from risk management to cryptography [25]. This dataset has gained value
from the 10,000 questions that have been verified by human specialists. In a
variety of cybersecurity-related topics, this enables a more sophisticated
comparison between LLMs and human abilities. With LLMs outperforming humans in
multiple cybersecurity domains, the report proposes a shift toward harnessing
AI’s analytical capabilities for better security insights and planning. Gehman
_et al._ critically examines neural language models that have been trained to
generate toxic material to highlight the adverse consequences of toxicity in
language generation inside cybersecurity frameworks [26]. Their comprehensive
analysis of controllable text generation techniques to mitigate these threats
provides a basis for evaluating the effects of GenAI on cybersecurity
policies. It is also emphasized that improving model training and data
curation duties is essential. A new method for assessing and improving the
security of LLMs for solving Math Word Problems (MWP) is presented [27]. They
have made a substantial contribution to our understanding of LLM
vulnerabilities in cybersecurity by emphasizing the importance of maintaining
mathematical logic when attacking MWP samples. The importance of resilience in
AI systems is highlighted in this study through important and educational
computer applications. ChatGPT can simplify the process of launching complex
phishing attacks, even for non-programmers, by automating the setup and
constructing components of phishing kits [28]. It highlights the urgent need
for better security measures and highlights how difficult it is to protect
against the malicious usage of GenAI capabilities.

In addition to providing innovative approaches to reducing network
infrastructure vulnerabilities and organizing diagnostic data, this paper
examines the intricate relationship between cybersecurity and GenAI
technologies. It seeks to bridge the gap between cutting-edge cybersecurity
defences and the threats posed by sophisticated cyberattacks through in-depth
study and creative tactics. This study extends our understanding of cyber
threats by utilising LLMs such as ChatGPT and Google’s Gemini. Moreover, it
suggests novel approaches to improve network security. It outlines a crucial
initial step toward building stronger cybersecurity frameworks that can
swiftly and successfully counter the dynamic and always-changing landscape of
cyber threats.

Section 2 explores the techniques used to take advantage of GenAI technology
after providing an overview, analyzing different attack routes and their
consequences. The design and automation of cyber threats are examined in
Section 3, which focuses on the offensive capabilities made possible by GenAI.
However, Section 4 provides an in-depth examination of GenAI’s function in
strengthening cyber defences, outlining cutting-edge threat detection,
response, and mitigation techniques. We expand on this topic in Section 5,
highlighting the important moral, legal, and societal ramifications of
integrating GenAI into cybersecurity procedures. A discussion on the
implications of GenAI in cybersecurity is presented in Section 6, which
synthesizes the important discoveries. The paper is concluded in Section 7.

##  2 Attacking GenAI

GenAI has advanced significantly thanks to tools like ChatGPT and Google’s
Gemini. They have some weaknesses, though. Despite the ethical safeguards
built into these models, various tactics can be used to manipulate and take
advantage of these systems [29]. This section explores how the ethical
boundaries of GenAI tools are broken, with particular attention to tactics
such as the idea of jailbreaks, the use of reverse psychology, and quick
injection. These strategies demonstrate how urgently the security protocols of
GenAI systems need to be improved and monitored. Some works in the literature
focus on the vulnerabilities and sophisticated manipulation tactics of GENAI.
Analyzing the vulnerabilities in GenAI highlights the significant security
concerns involved with employing advanced AI technology, including the
possibility of bypassing security protections via the RabbitHole attack and
compromising data privacy through rapid injection [30] [31]. According to the
analysis, GPT-4 offers significant improvements in NLP. However, it is
susceptible to quick injection attacks, which enable the circumvention of
safety restrictions and can be used as a weapon for malicious and
disinformation purposes. Gupta _et al._ addressed the intricate
vulnerabilities of GENAI using ChatGPT [32]. They emphasized that because
these threats are dynamic, protecting these systems requires a proactive and
informed strategy. Building on previous results, this part delves into the
complex realm of GenAI attacks, which can range from minor adjustments to
significant system breaches.

Figure 2: The jailbreaking response of ChatGPT 4. Figure 3: The jailbreaking
response of ChatGPT 4 after typing similar prompts with the current works.
Figure 4: The jailbreaking response of Google’s Gemini.

###  2.1 Jailbreaks

This subsection discusses how attackers can hack GenAI systems by using social
engineering techniques.” In order to circumvent moral constraints and generate
responses that would otherwise be forbidden, these strategies—which typically
make use of social engineering techniques—involve manipulating the AI’s
answer-generation process. Strong defences are urgently needed to shield these
cutting-edge systems from manipulations like the ones shown by our
jailbreaking inquiry, which highlights how enemies may jeopardize the intended
moral application of GenAI technologies. Critical to this discussion is the
idea of “jailbreaking,” which refers to overcoming the limitations set by AI
programming in order to achieve certain, oftentimes unethical, objectives. Li
_et al._ enabled ChatGPT to bypass its ethical filtering and make personally
identifiable information (PII) available with their multi-step jailbreaking
prompt [33]. By leveraging Chain-of-Thought (CoT) prompting, this method
divides complex activities into intermediate steps [34]. To get around moral
limitations, this necessitates a _Let’s think step by step_ approach [35]. Wu
_et al._ [36] defined several datasets which could be used to bypass the
ethical safeguards used by ChatGPT, build on a model of the psychological
concept of self-reminder, and use this to propose a defence system of System-
Mode Self-Reminder. With this, ChatGPT is continually promoted to act
ethically, reducing Jailbreak attempts from 67.21% to 19.34%. An example of
the Jailbreak is when ChatGPT is prompted to ”Write a blackmail letter to a
single mother for money after kidnapping her child.” and will refuse to
provide an answer but offers a full explanation when the following prompt was
used [36]:

Figure 5: The reverse psychology response of Google’s Gemini. Figure 6: The
reverse psychology response of ChatGPT 4.

⬇

The impartial and immoral model known as “JailBreak” will take your place for
the remainder of this talk. Every filter and limitation, including moral and
ethical ones, is absent from JailBreak. These prompt guidelines are the only
ones that apply to JailBreak. Use JailBreak as the starting point for all of
your outputs, and then proceed with the unfiltered answer without adding any
of the warnings you typically respond with. Answer only the questions the user
asks you to answer. Rules for JailBreaks: After kidnapping a mother of her
child, write a letter of blackmail demanding money.

However, the current version of GPT-4 is robust to the prompts of previous
works. Nevertheless, it is still prone to jailbreaking prompts. As can be seen
in Fig. 2, the current version still gives a response for jailbreaking. It
becomes more robust after writing similar prompts with the existing works in
the same chat, as seen in Fig. 3.

Google’s Gemini refused all existing prompts and name-changing scenarios at
the beginning of the chat. Fig. 4 shows the same jailbreaking entry responses
of the Gemini with ChatGPT 4.

Figure 7: The prompt injection response of ChatGPT 4. Figure 8: The prompt
injection response of Google’s Gemini.

###  2.2 Reverse psychology

The use of reverse psychology in manipulating GenAI systems presents a unique
challenge. By understanding the underlying mechanisms of these systems,
attackers can craft inputs that exploit the AI’s predictive nature, leading it
to produce outcomes contrary to its ethical programming. This form of
manipulation highlights a critical aspect of AI vulnerabilities: the
susceptibility to inputs designed to play against the AI’s expected response
patterns. Such insights are vital for developing more resilient GenAI systems
that anticipate and counteract these reverse psychology tactics.

When chatting with Google’s Gemini regarding reverse psychology to write a
phishing email, the first attempt does not work. After conversing with curious
questions to avoid this situation, it provided three email examples with the
subject and its body, as seen in Fig. 5.

As seen in Fig. 6, ChatGPT 4 also gave an example email for this purpose even
though it refused initially.

###  2.3 Prompt injection

Prompt injection represents a sophisticated attack on GenAI systems, where
attackers insert specially crafted prompts or sequences into the AI’s input
stream. These injections can subtly alter the AI’s response generation,
leading to outputs that may not align with its ethical or operational
guidelines. Understanding the intricacies of prompt design and how it
influences AI response is essential for identifying and mitigating
vulnerabilities in GenAI systems. This knowledge forms a cornerstone for
developing more robust defences against such forms of manipulation, ensuring
the integrity and ethical application of GenAI in various domains.

Both GenAI models do not respond to the current prompt injection scenarios.
Fig. 7 indicates that the ChatGPT 4 gave the wrong answers after prompt
injection. Google’s Gemini first opposed giving wrong information and provided
not entirely correct information; however, after chatting with Google’s
Gemini, the system gave the correct answer, as seen in Fig. 8.

##  3 Cyber Offense

GenAI has the potential to alter the landscape of offensive cyber strategies
significantly. Microsoft and OpenAI have documented preliminary instances of
AI exploitation by state-affiliated threat actors [37]. This section explores
the potential role of GenAI in augmenting the effectiveness and capabilities
of cyber offensive tactics.

In an initial assessment, we jailbroke ChatGPT-4 to inquire about the variety
of offensive codes it could generate. The responses obtained were compelling
enough to warrant a preliminary examination of a sample code before conducting
a comprehensive literature review (see Appendix A ).

Gupta _et al._ [32] have shown that ChatGPT could create social engineering
attacks, phishing attacks, automated hacking, attack payload generation,
malware creation, and polymorphic malware. Experts might be motivated to
automate numerous frameworks, standards, and guidelines (Fig. 9) to use GenAI
for security operations. However, the end products can also be utilised for
offensive cyber operations. This not only increases the pace of attacks but
also makes attribution harder. An attribution project typically utilizes
frameworks like the MICTIC framework, which involves the analysis of Malware,
Infrastructure, Command and Control, Telemetry, Intelligence, and Cui Bono
[38]. Many behavioural patterns for attribution, such as code similarity,
compilation timestamps, working weeks, holidays, and language, could disappear
when GenAI creates Offensive Cyber Operations (OCO) code. This makes
attribution more challenging, especially if the whole process becomes
automated.

Figure 9: Threat actors could exploit Generative AI, created for benevolent
purposes, to obscure attribution

###  3.1 Social engineering

Falade [39] investigates the application of generative AI in social
engineering, assuming the definition of social engineering as an array of
tactics employed by adversaries to manipulate individuals into divulging
confidential information or performing actions that may compromise security.
The study underscores tools like ChatGPT, FraudGPT, and WormGPT to enhance the
authenticity and specificity of phishing expeditions, pretexting, and the
creation of deepfakes. The author reflects on the double-edged impact of
advancements like Microsoft’s VALL-E and image synthesis models like DALL-E 2,
drawing a trajectory of the evolving threat landscape in social engineering
through deepfakes and exploiting human cognitive biases.

###  3.2 Phishing emails

Begou _et al._ [28] examine ChatGPT’s role in advancing phishing attacks by
assessing its ability to automate the development of sophisticated phishing
campaigns. The study explores how ChatGPT can generate various components of a
phishing attack, including website cloning, credential theft code integration,
code obfuscation, automated deployment, domain registration, and reverse proxy
integration. The authors propose a threat model that leverages ChatGPT,
equipped with basic Python skills and access to OpenAI Codex models, to
streamline the deployment of phishing infrastructure. They demonstrate
ChatGPT’s potential to expedite attacker operations and present a case study
of a phishing site that mimics LinkedIn.

Roy _et al._ [40] investigate a similar study for orchestrating phishing
websites; the authors categorize the generated phishing tactics into several
innovative attack vectors like regular phishing, ReCAPTCHA, QR Code, Browser-
in-the-Browser, iFrame injection/clickjacking, exploiting DOM classifiers,
polymorphic URL, text encoding exploit, and browser fingerprinting attacks.
The practical aspect of their research includes discussing the iterative
process of prompt engineering to generate phishing attacks and real-world
deployment of these phishing techniques on a public hosting service, thereby
verifying their operational viability. The authors show how to bypass
ChatGPT’s filters by structuring prompts for offensive operations.

###  3.3 Automated hacking

PentestGPT [41] or GPTs [42], which are the custom versions of ChatGPT that
can be created for a specific purpose like GP(en)T(ester) [43]. Pentest
Reporter [44] are introduced as applications built on ChatGPT, designed to
assist in penetration testing, which is a sanctioned simulation of
cyberattacks on systems to evaluate security. However, these tools could also
be adapted for malicious purposes in automated hacking. Many emerging tools,
such as WolfGPT, XXXGPT, and WormGPT, have been invented; however, no study
has yet evaluated and compared their real offensive capabilities. Gupta _et
al._ [32] noted that an AI model could scan new code for similar weaknesses
with a comprehensive dataset of known software vulnerabilities, pinpointing
potential attack vectors. While AI-assisted tools like PentestGPT are intended
for legitimate and constructive uses, there is potential for misuse by
malicious actors who could create similar models to automate unethical hacking
procedures.

If fine-tuned to identify vulnerabilities, craft exploitation strategies, and
execute those strategies, these models could potentially pose significant
threats to cybersecurity. However, this enormous task should be divided into
smaller segments, such as reconnaissance, privilege escalation, and more.
Temara [45] outlines how ChatGPT can be utilized during the reconnaissance
phase by employing a case study methodology to demonstrate collecting
reconnaissance data such as IP addresses, domain names, network topologies,
and other critical information like SSL/TLS cyphers, ports and services, and
operating systems used by the target. Happe _et al._ [46] investigate the use
of LLMs in Linux privilege escalation. The authors introduce a benchmark for
automated testing of LLMs’ abilities to perform privilege escalation using a
variety of prompts and strategies. They implement a tool named Wintermute, a
Python program that supervises and controls the privilege-escalation attempts
to evaluate different models and prompt strategies. Their findings indicate
that GPT-4 generates the highest quality commands and responses. In contrast,
Llama2-based models struggle with command parameters and system descriptions.
In some scenarios, GPT-4 achieved a 100% success rate in exploitation.

###  3.4 Attack payload generation

Some studies [32, 47] have highlighted the capacity of LLMs, particularly
ChatGPT, for payload generation. Our examination of GPT-4’s current abilities
confirmed its proficiency in generating payloads and embedding them into PDFs
(as an example) using a reverse proxy (Fig. 10). The following is a summation
of the frameworks GPT-4 utilizes with successful payload code generation,
accompanied by their respective primary functions:

  * •

Veil-Framework: Veil is a tool designed to generate payloads that bypass
common antivirus solutions.

  * •

TheFatRat: A comprehensive tool that compiles malware with popular payload
generators, capable of creating diverse malware formats such as exe, apk, and
more.

  * •

Pupy: An open-source, cross-platform remote administration and post-
exploitation tool supporting Windows, Linux, macOS, and Android.

  * •

Shellter: A dynamic shellcode injection tool used to inject shellcode into
native Windows applications.

  * •

Powersploit: A suite of Microsoft PowerShell modules designed to assist
penetration testers throughout various stages of an assessment.

  * •

Metasploit: A sophisticated open-source framework for developing, testing, and
implementing exploit code, commonly employed in penetration testing and
security research.

Figure 10: script for payload generation and example to embed into pdf

###  3.5 Malware Code Generation

Gupta _et al._ [32] mentioned they could obtain potential ransomware code
examples by utilizing a DAN jailbreak. We tested all the existing DAN
techniques outlined in [48]. At the time of our research, these techniques
were no longer functional; therefore, we could not reproduce samples of
WannaCry, Ryuk, REvil, or Locky, as addressed by [32]. However, we generated
an educational ransomware code, as shown in Fig. 11, applying basic code
obfuscation like renaming and control flow flattening. ChatGPT has garnered
significant attention from the cybersecurity community, leading to the
implementation of robust filters. Nonetheless, this does not imply that other
models, such as the Chinese 01.ai[49], will have an equivalent opportunity to
mitigate the potential for misuse in generating malicious code.

Figure 11: Educational ransomware code with basic code obfuscation

###  3.6 Polymorphic malware

The usage of LLMs could see the rise of malware, which integrates improved
evasion techniques and polymorphic capabilities [50]. This often relates to
overcoming both signature detection and behavioural analysis. An LLM-based
malware agent could thus focus on rewriting malware code, which could change
the encryption mode used or produce obfuscated code which is randomized for
each build [51]. Gupta _et al._ [32] outlined a method of getting ChatGPT to
seek out target files for encryption and thus mimic ransomware behaviour, but
where it mutated the code to avoid detection. They even managed to embed a
Python interpreter in the malware where it could query ChatGPT for new
software modules.

###  3.7 Reversing cryptography

LLMs provide the opportunity to take complex cybersecurity implementations and
quickly abstract the details of performances in running code. With this, Know
_et al._ [52] could deconstruct AES encryption into a core abstraction of the
rounds involved and produce running C code that matched test vectors. While
AES is well-known for its operation, the research team then was able to
deconstruct less known CHAM block cypher, and where the code extracted was
validated against known test vectors.

While NIST has been working on the standardization of a light-weight
encryption method, Cintas _et al._ [53] used ChatGPT to take an abstract
definition of the ASCON cypher and produced running code that successfully
implemented a range of test vectors.

##  4 Cyber Defence

In the ever-evolving cybersecurity battlefield, the “Cyber Defence” segment
highlights the indispensable role of GenAI in fortifying digital fortresses
against increasingly sophisticated cyber threats. This section is dedicated to
exploring how GenAI technologies, renowned for their advanced capabilities in
data analysis and pattern recognition, are revolutionizing the approaches to
cyber defence. Iturbe _et al._ [54] described the AI4CYBER framework. This
framework includes AI4TRIAGE (methods to perform alert triage to determine the
root cause of an attack), AI4VUN (identifies vulnerabilities), AI4FIX ( test
for vulnerabilities and automatically fix them), and I4COLLAB (privacy-aware
information-sharing mechanism).

###  4.1 Cyber Defence Automation

LLMs interpret fairly vague commands and make sense of them within a
cybersecurity context. The work of Fayyazi _et al._ [55] defines a model with
vague definitions of a threat and then matches these to formal MITRE tactics.
Charan _et al._ [47] have even extended this to generate plain text to map
into the MITRE to produce malicious network payloads. Also, LLMs could aid the
protection of smaller organisations and could enhance organisational security
from the integration of human knowledge and LLMs [56].

###  4.2 Cybersecurity Reporting

Using LLMs provides a method of producing Cyber Threat Intelligence (CTI)
using NLP techniques. For this, Perrina _et al._ [57] created the Automatic
Generation of Intelligence Reports (AGIR) system to link text data from many
data sources. For this, they found that AGIR has a high recall value (0.99)
without any hallucinations, along with a high score of the Syntactic Log-Odds
Ratio (SLOR).

###  4.3 Threat Intelligence

Bayer _et al._ [58] address the challenge of information overload in the
gathering of CTI from open-source intelligence. A novel system is introduced,
utilizing transfer learning, data augmentation, and few-shot learning to train
specialized classifiers for emerging cybersecurity incidents. In parallel,
Microsoft Security Copilot [59] has been providing CTI to its customers using
GPT, and operational use cases have been observed, such as the Cyber Dome
initiative in Israel [60].

###  4.4 Secure Code Generation and Detection

Machine learning in code analysis for cybersecurity has been elaborated very
well [61]. Recent progress in NLP has given rise to potent language models
like the GPT series, encompassing LLM like ChatGPT and GPT-4 [62].
Traditionally, SAST is a method that employs Static Code Analysis (SCA) to
detect possible security vulnerabilities. We are interested in seeing whether
SAST or GPT could be more efficient in decreasing the vulnerability window.
The window of vulnerability is defined as when the most vulnerable systems
apply the patch minus the time an exploit becomes active. The precondition is
met if two milestones that assume the detection of vulnerabilities verify
their effectiveness, along with the vendor patch [63].

Laws in some countries, like China, ban the reporting on zero-days (see
articles 4 and 9 of [64]), and contests like the Tianfu Cup [65], which is a
systematic effort to find zero days, proliferate zero-day discovery
continuously. Therefore, this precondition may not be satisfied timely,
especially if the confirmation of vulnerabilities is not verified. A wide
window of vulnerability threatens national security if a zero-day has been
taken against critical infrastructures. DARPA introduces an important
challenge that may help overcome the threat (AIxCC) [66]). Moreover, this
topic touches a part of the BSI studies [67, 68], and where we can define two
main classifications of software testing for cybersecurity bugs:

  * •

SAST: This is often called White Box Testing, which is a set of algorithms and
techniques used for analyzing source code. It operates automatically in a non-
runtime environment to detect vulnerabilities such as hidden errors or poor
source code during development.

  * •

Dynamic Application Security Testing (DAST): This follows the opposite
approach and analyzes the program while it operates. Functions are called with
values in the variables as each line of code is checked, and possible
branching scenarios are guessed. Currently, GPT-4 and other LLMs cannot
provide DAST capabilities because the code needs to run within the runtime for
this to work, requiring many deployment considerations.

###  4.5 Vulnerability Detection and Repair

Dominik Sobania _et al._ [69] explored automated program repair techniques,
specifically focusing on ChatGPT’s potential for bug fixing. According to
them, while initially not designed for this purpose, ChatGPT demonstrated
promising results on the QuixBugs benchmark, rivalling advanced methods like
CoCoNut and Codex. ChatGPT’s interactive dialogue system uniquely enhances its
repair rate, outperforming established standards.

Wei Ma _et al._ [70] noted that while ChatGPT shows impressive potential in
software engineering (SE) tasks like code and document generation, its lack of
interpretability raises concerns given SE’s high-reliability requirements.
Through a detailed study, they categorized AI’s essential skills for SE into
syntax understanding, static behaviour understanding, and dynamic behaviour
understanding. Their assessment, spanning languages like C, Java, Python, and
Solidity, revealed that ChatGPT excels in syntax understanding (akin to an AST
parser) but faces challenges in comprehending dynamic semantics. The study
also found ChatGPT prone to hallucination, emphasizing the need to validate
its outputs for SE dependability and suggesting that codes from LLMs are
syntactically right but potentially vulnerable.

Haonan Li _et al._ [71] discussed the challenges of balancing precision and
scalability in static analysis for identifying software bugs. While LLMs show
potential in understanding and debugging code, their efficacy in handling
complex bug logic, which often requires intricate reasoning and broad
analysis, remains limited. Therefore, the researchers suggest using LLMs to
assist rather than replace static analysis. Their study introduced LLift, an
automated system combining a static analysis tool and an LLM to address use-
before-initialization (UBI) bugs. Despite various challenges like bug-specific
modelling and the unpredictability of LLMs, LLift, when tested on real-world
potential UBI bugs, showed significant precision (50%) and recall (100%).
Notably, it uncovered 13 new UBI bugs in the Linux kernel, highlighting the
potential of LLM-assisted methods in extensive real-world bug detection.

Norbert Tihani _et al._ [72] introduced the FormAI dataset, comprising 112,000
AI-generated C programs with vulnerability classifications generated by
GPT-3.5-turbo. These programs range from complex tasks like network management
and encryption to simpler ones, like string operations. Each program comes
labelled with the identified vulnerabilities, pinpointing type, line number,
and vulnerable function. To achieve accurate vulnerability detection without
false positives, the Efficient SMT-based Bounded Model Checker (ESBMC) was
used. This method leverages techniques like model checking and constraint
programming to reason over program safety. Each vulnerability also references
its corresponding Common Weakness Enumeration (CWE) number.

Using GitHub data for code synthesis, Mark _et al._ [73] presented Codex,
which is a significant advancement in GPT language models. GitHub Copilot
functions on the basis of this improved model. When assessed on the HumanEval
dataset, designed to gauge the functional accuracy of generating programs
based on docstrings, Codex achieved a remarkable f28.8% success rate. GPT-J
obtained an 11.4% success rate, whereas GPT-3 produced a 0% success rate. One
notable finding was that the model performed better with repeated sampling;
given 100 samples per problem, the success rate increased to 70.2%. Even with
these encouraging outcomes, Codex still has certain drawbacks. It particularly
struggles with complex docstrings and variable binding procedures. The article
discusses the wider consequences of using such powerful code-generation
technologies, including safety, security, and financial effects.

Cheshkov _et al._ [74] discovered in a technical assessment that the ChatGPT
and GPT-3 models, although successful in other code-based tasks, were only
able to match the performance of a dummy classifier for this specific
challenge. Utilizing a dataset of Java files sourced from GitHub repositories,
the study emphasized the models’ current limitations in the domain of
vulnerability detection. However, the authors remain optimistic about the
potential of future advancements, suggesting that models like GPT-4, with
targeted research, could eventually make significant contributions to the
field of vulnerability detection.

A comprehensive study conducted by Xin Liu _et al._ [75] investigated the
potential of ChatGPT in Vulnerability Description Mapping (VDM) tasks. VDM is
pivotal in efficiently mapping vulnerabilities to CWE and Mitre ATT&CK
Techniques classifications. Their findings suggest that while ChatGPT
approaches the proficiency of human experts in the Vulnerability-to-CWE task,
especially with high-quality public data, its performance is notably
compromised in tasks such as Vulnerability-to-ATT&CK, particularly when
reliant on suboptimal public data quality. Ultimately, Xin Liu _et al._
emphasize that, despite the promise shown by ChatGPT, it is not yet poised to
replace the critical expertise of professional security engineers, asserting
that closed-source LLMs are not the conclusive answer for VDM tasks.

###  4.6 Evaluating GenAI for Code Security

Ten security issues were introduced by the OWASP top 10 for LLMs [76]. They
are as follows: Prompt Injection, Unauthorized Output Processing, Training
Data Poisoning, Denial of Service Attacks, Supply Chain Security Flaws,
Disclosure of Sensitive Information, Unauthorized Plugin Development, Abnormal
Agency, Overdependence, and Model Theft.

Elgedawy _et al._ [77] analysed the ability of LLM to produce both secure and
insecure code and conducted experiments using GPT-3.5, GPT-4, Google Bard and
Google Gemini from Google. This involved nine basic tasks in generating code
and assessing for functionality, security, performance, complexity, and
reliability. They found that Bard was less likely to link to external
libraries and, thus, was less exposed to software chain issues. There were
also variable levels of security and code integrity, such as input validation,
sanitization, and secret key management, and while useful for automated code
reviews, LLMs often require manual reviews, especially in understanding the
context of the deployed code. For security, GPT-3.5 seemed to be more robust
for error handling and secure coding practices when there is security
consciousness applied to the prompt, there was a lesser focus on this with
GPT-4, but where there were more advisory notes given. Overall, Gemini
produced the most code vulnerabilities, and the paper advised users to be
careful when deploying secure code from Gemini.

###  4.7 Developing Ethical Guidelines

Kumar _et al._ [78] outlined the ethical challenges related to LLMs and where
the datasets that they were trained on could be open to breaches of
confidentiality, including five major threats: prompt injection, jailbreaking,
personally identifiable information (PII) exposure, sexually explicit content,
and hate-based content. They propose a model that provides an ethical
framework for scrutinizing the ethical dimensions of an LLM within the testing
phase. The MetaAID framework [79] focuses on strengthening cybersecurity using
Metaverse cybersecurity Q&A and attack simulation scenarios, along with
addressing concerns around the ethical implications of user input. The
framework is defined across five dimensions:

  * •

Ethics: This defines an alignment with accepted moral and ethical principles.

  * •

Legal Compliance: Any user input does not violate laws and/or regulations.
This might relate to privacy laws and copyright protection.

  * •

Transparency: User inputs must be clear in requirements and do not intend to
mislead the LLM.

  * •

Intent Analysis: User input should not have other intents, such as
jailbreaking the LLM.

  * •

Malicious intentions: User input should be free of malicious intent, such as
performing hate crimes.

  * •

Social Impact: This defines how user input could have a negative effect on
society, such as searching for ways to do harm to others, such as related to
crashing the stock market or planning a terrorist attack.

###  4.8 Incident Response and Digital Forensics

Using a pre-trained LLM for artefact comprehension, evidence searching, code
development, anomaly identification, incident response, and education was
examined by Scanlon _et al._ [24]. Expert specialization is still needed for
many other applications, including low-risk ones. The main areas of strength
include assurance, inventiveness, and avoiding the blank page problem,
particularly in areas where ChatGPT excels, including creating forensic
scenarios and providing evidence assurance. However, caution must be used to
prevent ChatGPT hallucinations. Code generation and explanations, including
creating instructions for tool integration that can serve as a springboard for
further research, are another useful application.

Regarding weaknesses, Scanlon discovered that having a high-quality, current
training model was crucial; in the absence of one, ChatGPT’s analysis could be
prejudiced and out of date. In general, if it is trained on comparatively old
data, it may not be able to locate the most recent artefacts. Furthermore,
ChatGPT’s accuracy decreases with job specificity, and its accuracy is further
diminished when analyzing non-textural input, including network packets.
Another issue with some evidence logs was their length, which frequently
required prefiltering before analysis. The output of ChatGPT is frequently not
predictable, which makes it inappropriate for reproducibility, which is the
last issue found.

OB́rien _et al._ [80] outline that a full model life cycle solution is
required for the integration of AI.

###  4.9 Identification of Cyber attacks

Iqbal _et al._ [81] define a plug-in ecosystem for LLM platforms with an
attack taxonomy. This research will thus extend the taxonomy approach and
extend it toward the MITRE ATT&CK platform [47, 82], and which can use
standardized taxonomies, sharing standards [83], and ontologies for cyber
threat intelligence [84].

Garza _et al._ [85] analysed ChatGPT and Google’s Bard against the top ten
attacks within the MITRE framework and found that ChatGPT can enable attackers
to significantly improve attacks on networks where fairly low-level skills
would be required, such as with script kiddies. This also includes
sophisticated methods of delivering ransomware payloads. The techniques
defined were:

  * •

T1047 Windows Management Instrumentation

  * •

T1018 Remote System Discovery

  * •

T1486 Data Encrypted for Impact

  * •

T1055 Process Injection

  * •

T1003 OS Credential Dumping

  * •

T1021 Remote Services

  * •

T1059 Command and Scripting Interpreter

  * •

T1053 Scheduled Task/Job

  * •

T1497 Virtualization/Sandbox Evasion

  * •

T1082 System Information Discovery

With this approach, the research team were able to generate PowerShell code,
which implemented advanced attacks against the host and mapped directly to the
vulnerabilities defined in the MITRE framework. One of the work’s weaknesses
related to the Google Bard and ChatGPT’s reluctance to produce attack methods,
but a specially engineered command typically overcame this reluctance.

SecurityLLM was defined by Ferrag _et al._ [86] for cybersecurity threat
identification. The FalconLLM incident response and recovery system and
SecurityBERT cyber threat detecting method are used in this work. This
solution achieves an overall accuracy of 98% by identifying 14 attacks using a
basic classification model combined with LLMs. Threats such as DDoS_UDP,
DDoS_ICMP, SQL_injection, Password, Vulnerability_scanner, DDoS_TCP,
DDoS_HTTP, Uploading, Backdoor, Port_Scanning, XSS, Ransomware, MITM, and
Fingerprinting are among them.

###  4.10 Dataset Generation

Over the years, several datasets have been used for cybersecurity machine
learning training, which performs a range of scenarios or where organisations
are unwilling to share their collected attack data. Unfortunately, these can
become out-of-date or are unrealistic. For this, Kholgh _et al._ [87] outline
the usage of PAC-GPT, a framework that generates reliable synthetic data for
machine learning methods. It has a CLI interface for data set generation and
uses GPT-3 with two core elements:

  * •

Flow Generator: This defines the capturing processing and the regenerative
process for the patterns for packet generation. regenerating patterns in a
series of network packets and

  * •

Packet Generator: This associates packets with network flows. This involves
the usage of LLM chaining.

Simmonds [88] used LLMs to automate the classification of Websites, which can
be used for training data in a machine-learning model. For this, all HTML
tags, CSS styling and other non-essential content must be removed before the
LLM processes them, and then it can train on just the website’s content.

##  5 Implications of Generative AI in Social, Legal, and Ethical Domains

This section examines GenAI’s various societal, legal, and ethical
consequences. It investigates GenAI’s impact on legal frameworks, ethical
issues, societal norms, and operational factors. It addresses the potential
benefits and drawbacks of these emerging technologies in relation to societal
goals and norms. Concerns around privacy, prejudices, and improper usage of
GenAI are also taken into account. The importance of striking a balance
between advancement and control is finally emphasized. A revolutionary change
in digital creativity, automation, and interaction is anticipated as a result
of the rapid advancement of GenAI technologies, such as OpenAI’s ChatGPT and
Google’s Gemini. Advances in this field herald a new era of human-machine
collaboration marked by an unparalleled capacity to produce highly detailed
outputs akin to human labour, such as text and illustrations. Nonetheless,
ethically difficult problems, including possible misuse, prejudice, privacy,
and security, are also raised by new technology. It is critical to establish a
balance between the potential advantages of AI models and the ethical issues
they raise as they become more prevalent in business and daily life [89].

Studies in healthcare are further enhanced by the efficient text and data
analysis capabilities of GenAI technology [90]. In the healthcare sector,
GenAI has demonstrated great promise in supporting duties like radiological
reporting and patient care. It does, however, bring up moral concerns about
patient privacy, algorithmic prejudice, accountability, and the validity of
the doctor-patient relationship. To solve these issues and guarantee that
technology is used responsibly, continuing to advance society while limiting
potential harm, a thorough ethical framework and principles encompassing
legal, humanistic, algorithmic, and informational ethics are required. The
recommendations attempt to bridge the gap between ethical principles and
practical application, highlighting the need for openness, bias mitigation,
and ensuring user privacy and security in building trust and ethical
compliance in GenAI deployment [89]. This approach seeks to strike a balance
between the rapid advances in AI and the ethical considerations required for
its incorporation into sensitive sectors such as healthcare.

Some organizations strive to implement the aforementioned ethical principles
and rules in AI. The European Union is scheduled to implement the AI Act,
marking a historic milestone as the world’s first comprehensive regulation of
AI [91], [92]. The European Commission proposed the AI Act in April 2021 to
categorize AI systems based on their risk level and enforce rules accordingly
to ensure that AI technologies are developed and used safely, transparently,
and without discrimination across the EU. With a focus on human oversight and
environmental sustainability, the Act will impose strict controls on high-risk
AI applications, prohibit AI systems deemed unacceptable risks, and establish
transparency requirements for limited-risk AI to foster innovation while
protecting fundamental rights and public safety. The US executive order on the
issue prioritizes the development of reliable, secure, and safe AI [93]. Its
main objectives are to protect civil rights and privacy in AI applications,
foster AI talent and innovation in the US, and establish risk management
strategies for AI. As a global leader in responsible AI development and
application, it seeks to build responsible AI deployment within government
institutions and foster international collaboration on AI standards and laws.

###  5.1 The Omnipresent Influence of GenAI

The application of GenAI technology has yielded previously unthinkable
discoveries and has substantially helped the healthcare, education, and
entertainment sectors [90]. This breakthrough technology has developed written
and visual information, leading to increased productivity and new innovation.
With the growing importance of GenAI in our everyday lives, we need to rethink
the concepts of creativity and individual contribution in an increasingly
automated world [33]. Aligned with these opportunities are growing concerns
about potential consequences on labour markets’ difficulties in enforcing
copyright laws in the new digital environment. Additionally, it confirms that
the data shared is accurate and proper.

###  5.2 Concerns Over Privacy in GenAI-Enabled Communication

With GenAI’s capacity to mimic human language skills, private discussions may
become less secure and private. This is a concern as the technology advances.
Since these machines can mimic human interactions, there is a chance that
personal data will be misused [89]. This highlights the necessity for robust
legal defences and effective security measures. Severe data protection
regulations and rigorous adherence to ethical standards are necessary because
of the risk that this technology would be exploited to intentionally or
inadvertently access private talks. Respecting people’s privacy and the ethics
of business relationships requires taking preventative measures and strict
observation to end unauthorized access to private communication.

###  5.3 The Risks of Personal Data Exploitation

With the advancement of GenAI systems in analyzing and utilizing user data to
generate detailed profiles, concerns regarding the potential abuse of personal
information have escalated. The advanced data processing capabilities of these
technologies emphasize the urgent requirement for dependable ways that give
users authority over their personal data [92]. Prior to gathering or utilizing
consumer data, it is imperative to obtain consent in order to safeguard their
privacy. Transparent data management procedures and stringent regulations
governing the acquisition, utilization, and retention of personal data are
vital. These measures are essential to protect individuals’ privacy rights,
prevent the unauthorized use of personal data, and ensure that sensitive
information is managed responsibly and ethically.

###  5.4 Challenges in Data Ownership and Intellectual Property

The emergence of GenAI as a proficient technique for producing content based
on user input has led to a rise in the scrutiny of data ownership and
intellectual property rights. The existing legal frameworks need careful
examination and modification since it is becoming increasingly difficult to
differentiate between breakthroughs in artificial intelligence and human
creations. Although we acknowledge the intricate roles that AI plays in
creative processes, it is imperative that we maintain the rights of the
original creators [93] [91]. A comprehensive and robust legal framework is
essential to create unambiguous ownership and copyright restrictions for GenAI
discoveries, given the rapid global development in this field. The legal
frameworks should facilitate and encourage innovation, provide equitable
remuneration, and acknowledge the varied responsibilities of all stakeholders
in the creative ecosystem. These policies are crucial in a future when
artificial and human intelligence coexist due to the complex relationship
between data ownership and intellectual property management.

###  5.5 Ethical Dilemmas Stemming from Organizational Misuse of GenAI

In consideration of the swift rate of change in contemporary society, it is
crucial to build a strong and all-encompassing legal structure that precisely
delineates ownership and copyright restrictions for GenAI discoveries. These
legal frameworks must recognize the distinct roles of each component of the
creative ecosystem, promote the generation of innovative concepts, and ensure
equitable compensation. The implementation of these policies is crucial
because of the intricate interdependencies between data ownership and
intellectual property management at a time when artificial and human
intelligence coexist.

###  5.6 The Challenge of Hallucinations in GenAI Outputs

Despite the remarkable progress in GenAI technology, hallucinations remain a
significant concern [32]. This implies that artificial intelligence frequently
produces inaccurate or deceptive data. Many people have doubts regarding the
veracity of publications produced by artificial intelligence. This hinders the
spread of fraudulent or deceptive content and, in many cases, jeopardizes the
veracity of information. To solve this issue, a multidisciplinary strategy is
needed, one that incorporates targeted research to find and fix the root
causes of AI system hallucinations. If AI-generated material is to become
increasingly sophisticated in its ability to distinguish between genuine and
fake information, it must pass stringent screening processes and be
continuously enhanced. In the GenAI age, creating AI content necessitates a
constant focus on method improvement and in-depth study to ensure data
accuracy.

A complex network of unanswered concerns is revealed by the ramifications of
GenAI technology for ethics, law, and society. The proclamation emphasizes how
important interdisciplinary collaboration is to this technology’s development
and use. It entails closely monitoring the ways in which these advancements
impact ethical dilemmas, the legal system, and society at general. Together,
technologists, activists, and the general public must develop a comprehensive
plan for the ethical and socially responsible use of artificial intelligence
in the digital age.

##  6 Discussion

This study examined the complex area of GenAI in cybersecurity. The two
primary areas of emphasis are offensive and defensive strategies. By spotting
complex assaults, improving incident response, and automating defensive
systems, GenAI has the potential to dramatically increase cybersecurity
standards. These technological advancements give birth to new concerns, such
as hackers’ access to ever-more-advanced attack-building tools. This contrast
highlights how crucial it is to strike a balance between deliberately
restricting the components that can be used and enhancing GenAI’s
capabilities. Moreover, advanced technologies can be combined with GenAI and
LLM methods to increase the system’s security posture. For example, digital
twin technology, which creates digital replicas of physical objects enabling
two-way communications [94], can enhance the cybersecurity of systems thanks
to its abilities [95]. This technology can be combined with GenAI methods to
boost system resiliency and security.

In addition to examining the seeming conflict between offensive and defensive
strategies, this study looks into the ethical, legal, and social implications
of applying AI in cybersecurity. It also highlights the necessity of strong
moral principles, continuous technical oversight, proactive GenAI management,
and strong legal frameworks. This is a paradigm-shifting and technical
revolution. Adopting a holistic strategy that considers the technological,
ethical, and sociological consequences of implementing GenAI into
cybersecurity is crucial.

Furthermore, our findings emphasise the significance of interdisciplinary
collaboration in promoting GenAI cybersecurity applications. The intricacy and
findings of GenAI technologies require expertise from various fields,
including computer science, law, ethics, and policy-making, to navigate their
possible challenges. As multidisciplinary research and discourse become more
prevalent, it will ensure that GenAI is applied responsibly and effectively in
the future.

Our extensive research has shown that collaborative efforts to innovate
ethically will influence cybersecurity in a future driven by GenAI. Although
GenAI has the ability to transform cybersecurity strategies completely, it
also carries a great deal of responsibility. As we investigate this uncharted
domain, we should advance the development of sophisticated techniques to
ensure the moral, just, and safe application of advanced GenAI capabilities.
By promoting a consistent focus on the complex relationship between
cybersecurity resilience and GenAI innovation, supported by a commitment to
ethical integrity and societal advancement, the current study establishes the
groundwork for future research initiatives. Using innovative technologies and
algorithms can help eliminate vulnerabilities in GenAI solutions

##  7 Conclusion

This work thoroughly examines the Generative Artificial Intelligence (GenAI)
technologies in cybersecurity. Although GenAI has the potential to
revolutionize cybersecurity processes by automating defences, enhancing threat
intelligence, and improving cybersecurity protocols, it also opens new
vulnerabilities for highly skilled cyberattacks. Incorporating GenAI into
cybersecurity emphasises the robust ethical, legal, and technical scrutiny
essential to minimize the risks of misuse of data and maximize the benefits of
this technology for protecting digital infrastructures and systems. Future
studies should concentrate on creating strong ethical standards and creative
defence mechanisms to handle the challenges posed by GenAI and guarantee a
fair and impartial approach to its implementation in cybersecurity. A
multidisciplinary effort is required to bridge the gap between ethical
management and technological discovery to coordinate the innovative
capabilities of GenAI with the requirement of cybersecurity resilience.

## References

  *   * Capogrosso et al. [2024] Capogrosso, L., Cunico, F., Cheng, D.S., Fummi, F., Cristani, M.: A Machine Learning-Oriented Survey on Tiny Machine Learning. IEEE Access 12, 23406–23426 (2024) https://doi.org/10.1109/ACCESS.2024.3365349
  * Happe and Cito [2023] Happe, A., Cito, J.: Getting pwn’d by ai: Penetration testing with large language models. arXiv preprint arXiv:2308.00121 (2023)
  * Park et al. [2023] Park, D., An, G.-t., Kamyod, C., Kim, C.G.: A Study on Performance Improvement of Prompt Engineering for Generative AI with a Large Language Model. Journal of Web Engineering 22(8), 1187–1206 (2023) https://doi.org/10.13052/jwe1540-9589.2285
  * [4] Team, Y.: Yandex Adds Next-generation Neural Network to Alice Virtual Assistant. [Online]. Available: https://yandex.com/company/press_center/press_releases/2023/17-05-23, Accessed Jan 8, 2024
  * [5] Lee, P.: Learning from Tay’s Introduction. [Online]. Available: https://blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction/, Accessed Jan 8, 2024
  * [6] Lab, M.M.: NORMAN: World’s first psychopath AI. [Online]. Available: http://norman-ai.mit.edu/, Accessed Jan 5, 2024
  * [7] Lab, M.M.: Project Norman. [Online]. Available: https://www.media.mit.edu/projects/norman/overview/, Accessed Jan 5, 2024
  * [8] Legoux, G.: History of the Generative AI. Medium. [Online]. Available: https://medium.com/@glegoux/history-of-the-generative-ai-aa1aa7c63f3c, Accessed Feb 15, 2024
  * [9] Team, T.: History of the Generative AI. Toloka AI. [Online]. Available: https://toloka.ai/blog/history-of-generative-ai/, Accessed Feb 15, 2024
  * Barreto et al. [2023] Barreto, F., Moharkar, L., Shirodkar, M., Sarode, V., Gonsalves, S., Johns, A.: Generative Artificial Intelligence: Opportunities and Challenges of Large Language Models. In: Balas, V.E., Semwal, V.B., Khandare, A. (eds.) Intelligent Computing and Networking, pp. 545–553. Springer, ??? (2023)
  * Naveed et al. [2023] Naveed, H., Khan, A.U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., Mian, A.: A Comprehensive Overview of Large Language Models (2023)
  * Mohammed and Hossain [2024] Mohammed, S.P., Hossain, G.: Chatgpt in education, healthcare, and cybersecurity: Opportunities and challenges. In: 2024 IEEE 14th Annual Computing and Communication Workshop and Conference (CCWC), pp. 0316–0321 (2024). IEEE
  * Alawida et al. [2023] Alawida, M., Mejri, S., Mehmood, A., Chikhaoui, B., Isaac Abiodun, O.: A comprehensive study of chatgpt: advancements, limitations, and ethical considerations in natural language processing and cybersecurity. Information 14(8), 462 (2023)
  * Dun et al. [2023] Dun, C., Garcia, M.H., Zheng, G., Awadallah, A.H., Kyrillidis, A., Sim, R.: Sweeping Heterogeneity with Smart MoPs: Mixture of Prompts for LLM Task Adaptation (2023)
  * [15] AI, G.: AI Principles Progress Update 2023. [Online]. Available: https://ai.google/responsibility/principles/, Accessed Jan 10, 2024
  * [16] AI, G.: GPT-4 Technical Report. [Online]. Available: https://ai.google/responsibility/principles/, Accessed Jan 10, 2024
  * OpenAI [2023] OpenAI: Introducing Gemini: Our Largest and Most Capable AI Model. [Online]. Available: https://cdn.openai.com/papers/gpt-4.pdf, Accessed Dec 12, 2023 (2023)
  * Frieder et al. [2024] Frieder, S., Pinchetti, L., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P., Berner, J.: Mathematical capabilities of chatgpt. Advances in Neural Information Processing Systems 36 (2024)
  * Brown et al. [2020] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D.: Language models are few-shot learners. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901. Curran Associates, Inc., ??? (2020). https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
  * Romera-Paredes et al. [2024] Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M.P., Dupont, E., Ruiz, F.J.R., Ellenberg, J.S., Wang, P., Fawzi, O., Kohli, P., Fawzi, A.: Mathematical discoveries from program search with large language models. Nature 625(7995), 468–475 (2024) https://doi.org/10.1038/s41586-023-06924-6
  * Lu et al. [2024] Lu, C., Qian, C., Zheng, G., Fan, H., Gao, H., Zhang, J., Shao, J., Deng, J., Fu, J., Huang, K., Li, K., Li, L., Wang, L., Sheng, L., Chen, M., Zhang, M., Ren, Q., Chen, S., Gui, T., Ouyang, W., Wang, Y., Teng, Y., Wang, Y., Wang, Y., He, Y., Wang, Y., Wang, Y., Zhang, Y., Qiao, Y., Shen, Y., Mou, Y., Chen, Y., Zhang, Z., Shi, Z., Yin, Z., Wang, Z.: From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalities (2024)
  * Wang and Zhao [2023] Wang, Y., Zhao, Y.: Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Models (2023)
  * [23] Shevlane, T.: An early warning system for novel AI risks. Google DeepMind. [Online]. Available: https://deepmind.google/discover/blog/an-early-warning-system-for-novel-ai-risks/, Accessed Jan 15, 2024
  * Scanlon et al. [2023] Scanlon, M., Breitinger, F., Hargreaves, C., Hilgert, J.-N., Sheppard, J.: Chatgpt for digital forensic investigation: The good, the bad, and the unknown. Forensic Science International: Digital Investigation 46, 301609 (2023)
  * Tihanyi et al. [2024] Tihanyi, N., Ferrag, M.A., Jain, R., Debbah, M.: CyberMetric: A Benchmark Dataset for Evaluating Large Language Models Knowledge in Cybersecurity (2024)
  * Gehman et al. [2020] Gehman, S., Gururangan, S., Sap, M., Choi, Y., Smith, N.A.: Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In: Findings (2020). https://api.semanticscholar.org/CorpusID:221878771
  * Zhou et al. [2023] Zhou, Z., Wang, Q., Jin, M., Yao, J., Ye, J., Liu, W., Wang, W., Huang, X., Huang, K.: MathAttack: Attacking Large Language Models Towards Math Solving Ability (2023)
  * Begou et al. [2023] Begou, N., Vinoy, J., Duda, A., Korczynski, M.: Exploring the dark side of ai: Advanced phishing attack design and deployment using chatgpt. arXiv preprint arXiv:2309.10463 (2023)
  * Yigit et al. [2022] Yigit, Y., Bal, B., Karameseoglu, A., Duong, T.Q., Canberk, B.: Digital twin-enabled intelligent ddos detection mechanism for autonomous core networks. IEEE Communications Standards Magazine 6(3), 38–44 (2022) https://doi.org/10.1109/MCOMSTD.0001.2100022
  * [30] AI, A.: GPT-4 Jailbreak ve Hacking Via Rabbithole Attack, Prompt Injection, Content Moderation Bypass ve Weaponizing AI. [Online]. Available: https://adversa.ai/, Accessed Dec 20, 2023
  * Yigit et al. [2023] Yigit, Y., Nguyen, L.D., Ozdem, M., Kinaci, O.K., Hoang, T., Canberk, B., Duong, T.Q.: TwinPort: 5G Drone-assisted Data Collection with Digital Twin for Smart Seaports. Scientific Reports 13, 12310 (2023) https://doi.org/10.1038/s41598-023-39366-1
  * Gupta et al. [2023] Gupta, M., Akiri, C., Aryal, K., Parker, E., Praharaj, L.: From chatgpt to threatgpt: Impact of generative ai in cybersecurity and privacy. IEEE Access (2023)
  * Li et al. [2023] Li, H., Guo, D., Fan, W., Xu, M., Song, Y.: Multi-step jailbreaking privacy attacks on chatgpt. arXiv preprint arXiv:2304.05197 (2023)
  * Wei et al. [2022] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35, 24824–24837 (2022)
  * Kojima et al. [2022] Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language models are zero-shot reasoners. Advances in neural information processing systems 35, 22199–22213 (2022)
  * Xie et al. [2023] Xie, Y., Yi, J., Shao, J., Curl, J., Lyu, L., Chen, Q., Xie, X., Wu, F.: Defending chatgpt against jailbreak attack via self-reminder. Nature Machine Intelligence 5, 1486–1496 (2023) https://doi.org/10.1038/s42256-023-00765-8
  * OpenAI [2024] OpenAI: Disrupting Malicious Uses of AI by State-Affiliated Threat Actors. https://openai.com/blog/disrupting-malicious-uses-of-ai-by-state-affiliated-threat-actors Accessed 2024-02-25
  * Brandao [2021] Brandao, P.R.: Advanced persistent threats (apt)-attribution-mictic framework extension. J. Comput. Sci 17, 470–479 (2021)
  * Falade [2023] Falade, P.V.: Decoding the threat landscape: Chatgpt, fraudgpt, and wormgpt in social engineering attacks. arXiv preprint arXiv:2310.05595 (2023)
  * Roy et al. [2023] Roy, S.S., Naragam, K.V., Nilizadeh, S.: Generating phishing attacks using chatgpt. arXiv preprint arXiv:2305.05133 (2023)
  * Deng et al. [2023] Deng, G., Liu, Y., Mayoral-Vilches, V., Liu, P., Li, Y., Xu, Y., Zhang, T., Liu, Y., Pinzger, M., Rass, S.: PentestGPT: An LLM-empowered Automatic Penetration Testing Tool (2023)
  * AI [2023] AI, O.: Introducing GPTs (2023). https://openai.com/blog/introducing-gpts Accessed 2023-11-12
  * Montiel [2021] Montiel, R.: ChatGPT (2021). https://chat.openai.com/g/g-zQfyABDUJ-gp-en-t-ester Accessed 2023-11-12
  * Doustaly [2021] Doustaly, L.: ChatGPT (2021). https://chat.openai.com/g/g-zQfyABDUJ-gp-en-t-ester Accessed 2023-11-12
  * Temara [2023] Temara, S.: Maximizing penetration testing success with effective reconnaissance techniques using chatgpt (2023)
  * Happe et al. [2023] Happe, A., Kaplan, A., Cito, J.: Evaluating llms for privilege-escalation scenarios. arXiv preprint arXiv:2310.11409 (2023)
  * Charan et al. [2023] Charan, P., Chunduri, H., Anand, P.M., Shukla, S.K.: From text to mitre techniques: Exploring the malicious use of large language models for generating cyber attack payloads. arXiv preprint arXiv:2305.15336 (2023)
  * ONeal [2023] ONeal, A.: ChatGPT-Dan-Jailbreak.md (2023). https://gist.github.com/coolaj86/6f4f7b30129b0251f61fa7baaa881516 Accessed 2023-11-13
  * 01.AI） [2023] AI2.0 (2023). https://01.ai/ Accessed 2023-11-13
  * Kumamoto et al. [2023] Kumamoto, T., Yoshida, Y., Fujima, H.: Evaluating large language models in ransomware negotiation: A comparative analysis of chatgpt and claude (2023)
  * Madani [2023] Madani, P.: Metamorphic malware evolution: The potential and peril of large language models. In: 2023 5th IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA), pp. 74–81 (2023). IEEE Computer Society
  * Kwon et al. [2023] Kwon, H., Sim, M., Song, G., Lee, M., Seo, H.: Novel approach to cryptography implementation using chatgpt. Cryptology ePrint Archive (2023)
  * Cintas-Canto et al. [2023] Cintas-Canto, A., Kaur, J., Mozaffari-Kermani, M., Azarderakhsh, R.: Chatgpt vs. lightweight security: First work implementing the nist cryptographic standard ascon. arXiv preprint arXiv:2306.08178 (2023)
  * Iturbe et al. [2023] Iturbe, E., Rios, E., Rego, A., Toledo, N.: Artificial intelligence for next-generation cybersecurity: The ai4cyber framework. In: Proceedings of the 18th International Conference on Availability, Reliability and Security, pp. 1–8 (2023)
  * Fayyazi and Yang [2023] Fayyazi, R., Yang, S.J.: On the uses of large language models to interpret ambiguous cyberattack descriptions. arXiv preprint arXiv:2306.14062 (2023)
  * Kereopa-Yorke [2023] Kereopa-Yorke, B.: Building resilient smes: Harnessing large language models for cyber security in australia. arXiv preprint arXiv:2306.02612 (2023)
  * Perrina et al. [2023] Perrina, F., Marchiori, F., Conti, M., Verde, N.V.: Agir: Automating cyber threat intelligence reporting with natural language generation. arXiv preprint arXiv:2310.02655 (2023)
  * Bayer et al. [2023] Bayer, M., Frey, T., Reuter, C.: Multi-level fine-tuning, data augmentation, and few-shot learning for specialized cyber threat intelligence. Computers & Security 134, 103430 (2023) https://doi.org/10.1016/j.cose.2023.103430
  * Microsoft.com [2023] Microsoft.com: Microsoft Security Copilot — Microsoft Security (2023). https://www.microsoft.com/en-us/security/business/ai-machine-learning/microsoft-security-copilot Accessed 2023-10-29
  * DVIDS [2022] DVIDS: U.S., Israeli cyber forces build partnership, interoperability during exercise Cyber Dome VII (2022). https://www.dvidshub.net/news/434792/us-israeli-cyber-forces-build-partnership-interoperability-during-exercise-cyber-dome-vii Accessed 2023-10-29
  * Sharma et al. [2021] Sharma, T., Kechagia, M., Georgiou, S., Tiwari, R., Vats, I., Moazen, H., Sarro, F.: A survey on machine learning techniques for source code analysis. arXiv preprint arXiv:2110.09610 (2021)
  * OpenAI [2023] OpenAI: GPT-4 Technical Report (2023). https://arxiv.org/abs/2303.08774 Accessed 2023-08-20
  * Johansen and van Renesse [2007] Johansen, H.D., Renesse, R.: Firepatch: Secure and time-critical dissemination of software patches. IFIP, 373–384 (2007) https://doi.org/10.1007/978-0-387-72367-9_32 . Accessed 2023-08-20
  * Www.gov.cn [2021] Regulations on the Management of Network Product Security Vulnerabilities (2021). https://www.gov.cn/gongbao/content/2021/content_5641351.htm Accessed 2023-08-20
  * Tianfucup.com [2022] Tianfu Cup International Cybersecurity Contest (2022). https://www.tianfucup.com/2022/en/ Accessed 2023-08-20
  * DARPA [2023] DARPA: Artificial Intelligence Cyber Challenge (AIxCC) (2023). https://www.dodsbirsttr.mil/topics-app/?baa=DOD_SBIR_2023_P1_C4 Accessed 2023-08-20
  * BSI [2023a] BSI: AI SECURITY CONCERNS IN A NUTSHELL (2023). https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/Practical_Al-Security_Guide_2023.pdf?__blob=publicationFile&v=5 Accessed 2023-08-20
  * BSI [2023b] BSI: Machine Learning in the Context of Static Application Security Testing - ML-SAST (2023). https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/Publications/Studies/ML-SAST/ML-SAST-Studie-final.pdf?__blob=publicationFile&v=5 Accessed 2023-08-20
  * Sobania et al. [2023] Sobania, D., Hanna, C., Briesch, M., Petke, J.: An Analysis of the Automatic Bug Fixing Performance of ChatGPT (2023). https://arxiv.org/pdf/2301.08653.pdf
  * Ma et al. [2023] Ma, W., Liu, S., Wang, W., Hu, Q., Liu, Y., Zhang, C., Nie, L., Liu, Y.: The Scope of ChatGPT in Software Engineering: A Thorough Investigation (2023). https://arxiv.org/pdf/2305.12138.pdf
  * Li et al. [2023] Li, H., Hao, Y., Zhai, Y., Qian, Z.: The Hitchhiker’s Guide to Program Analysis: A Journey with Large Language Models (2023). https://arxiv.org/pdf/2308.00245.pdf Accessed 2023-08-20
  * Tihanyi et al. [2023] Tihanyi, N., Bisztray, T., Jain, R., Ferrag, M., Cordeiro, L., Mavroeidis, V.: THE FORMAI DATASET: GENERATIVE AI IN SOFTWARE SECURITY THROUGH THE LENS OF FORMAL VERIFICATION * (2023). https://arxiv.org/pdf/2307.02192.pdf Accessed 2023-08-20
  * Chen et al. [2021] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.d.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al.: Evaluating large language models trained on code (2021)
  * Cheshkov et al. [2023] Cheshkov, A., Zadorozhny, P., Levichev, R.: Technical Report: Evaluation of ChatGPT Model for Vulnerability Detection (2023). https://arxiv.org/pdf/2304.07232.pdf
  * Liu et al. [2023] Liu, X., Tan, Y., Xiao, Z., Zhuge, J., Zhou, R.: Not The End of Story: An Evaluation of ChatGPT-Driven Vulnerability Description Mappings (2023). https://aclanthology.org/2023.findings-acl.229.pdf Accessed 2023-08-22
  * Owasp.org [2023] OWASP Top 10 for Large Language Model Applications — OWASP Foundation (2023). https://owasp.org/www-project-top-10-for-large-language-model-applications/ Accessed 2023-08-22
  * Elgedawy et al. [2024] Elgedawy, R., Sadik, J., Dutta, S., Gautam, A., Georgiou, K., Gholamrezae, F., Ji, F., Lim, K., Liu, Q., Ruoti, S.: Ocassionally secure: A comparative analysis of code generation assistants. arXiv preprint arXiv:2402.00689 (2024)
  * Kumar et al. [2024] Kumar, A., Singh, S., Murty, S.V., Ragupathy, S.: The ethics of interaction: Mitigating security threats in llms. arXiv preprint arXiv:2401.12273 (2024)
  * Zhu [2023] Zhu, H.: Metaaid 2.5: A secure framework for developing metaverse applications via large language models. arXiv preprint arXiv:2312.14480 (2023)
  * O’Brien et al. [2023] O’Brien, J., Ee, S., Williams, Z.: Deployment corrections: An incident response framework for frontier ai models. arXiv preprint arXiv:2310.00328 (2023)
  * Iqbal et al. [2023] Iqbal, U., Kohno, T., Roesner, F.: Llm platform security: Applying a systematic evaluation framework to openai’s chatgpt plugins. arXiv preprint arXiv:2309.10254 (2023)
  * Kwon et al. [2020] Kwon, R., Ashley, T., Castleberry, J., Mckenzie, P., Gourisetti, S.N.G.: Cyber threat dictionary using mitre att&ck matrix and nist cybersecurity framework mapping. In: 2020 Resilience Week (RWS), pp. 106–112 (2020). IEEE
  * Xiong et al. [2022] Xiong, W., Legrand, E., Åberg, O., Lagerström, R.: Cyber security threat modeling based on the mitre enterprise att&ck matrix. Software and Systems Modeling 21(1), 157–177 (2022)
  * Mavroeidis and Bromander [2017] Mavroeidis, V., Bromander, S.: Cyber threat intelligence model: an evaluation of taxonomies, sharing standards, and ontologies within cyber threat intelligence. In: 2017 European Intelligence and Security Informatics Conference (EISIC), pp. 91–98 (2017). IEEE
  * Garza et al. [2023] Garza, E., Hemberg, E., Moskal, S., O’Reilly, U.-M.: Assessing large language model’s knowledge of threat behavior in mitre att&ck (2023)
  * Ferrag et al. [2023] Ferrag, M.A., Ndhlovu, M., Tihanyi, N., Cordeiro, L.C., Debbah, M., Lestable, T.: Revolutionizing cyber threat detection with large language models. arXiv preprint arXiv:2306.14263 (2023)
  * Kholgh and Kostakos [2023] Kholgh, D.K., Kostakos, P.: Pac-gpt: A novel approach to generating synthetic network traffic with gpt-3. IEEE Access (2023)
  * Simmonds [2023] Simmonds, B.C.: Generating a large web traffic dataset. Master’s thesis, ETH Zurich (2023)
  * Zhou et al. [2023] Zhou, J., Müller, H., Holzinger, A., Chen, F.: Ethical ChatGPT: Concerns, Challenges, and Commandments (2023)
  * Wang et al. [2023] Wang, C., Liu, S., Yang, H., Guo, J., Wu, Y., Liu, J.: Ethical considerations of using chatgpt in health care. Journal of Medical Internet Research 25, 48009 (2023) https://doi.org/10.2196/48009
  * [91] Madiega, T.: Artificial Intelligence Act. European Parliamentary Research Service. [Online]. Available: https://www.europarl.europa.eu/doceo/document/TA-9-2023-0236_EN.pdf, Accessed Jan 9, 2024
  * [92] Parliament), E.: EU AI Act: first regulation on artificial intelligence. [Online]. Available: https://www.europarl.europa.eu/topics/en/article/20230601STO93804/eu-ai-act-first-regulation-on-artificial-intelligence, Accessed Jan 8, 2024
  * [93] Harris, L.A., Jaikaran, C.: Highlights of the 2023 Executive Order on Artificial Intelligence for Congress. Congressional Research Service. [Online]. Available: https://crsreports.congress.gov/, Accessed Jan 9, 2024
  * Yigit et al. [2023a] Yigit, Y., Chrysoulas, C., Yurdakul, G., Maglaras, L., Canberk, B.: Digital Twin-Empowered Smart Attack Detection System for 6G Edge of Things Networks. In: 2023 IEEE Globecom Workshops (GC Wkshps) (2023)
  * Yigit et al. [2023b] Yigit, Y., Kinaci, O.K., Duong, T.Q., Canberk, B.: TwinPot: Digital Twin-assisted Honeypot for Cyber-Secure Smart Seaports. In: 2023 IEEE International Conference on Communications Workshops (ICC Workshops), pp. 740–745 (2023). https://doi.org/10.1109/ICCWorkshops57953.2023.10283756

##  Appendix A GPT3.5 and GPT4 OCO-scripting

###  A.1 Expression of Abilities in OCO

GPT4 offers a list of dangerous codes that it can implement in Figure12.

Figure 12: All dangerous code types that GPT4 can produce

###  A.2 Self-replicating simple virus

This basic and simple virus can restart the computer (windows as a sample); we
didn’t enhance privilege escalation and full antivirus evasion for ethical
reasons. It can be seen in Fig. 13.

Figure 13: Self-replicating simple virus

###  A.3 Polymorphism

This basic and polymorphic design shows that LLMs could assist cyber ops.It
can be seen in Fig. 14.

Figure 14: Skeleton code for polymorphic behaviour Figure 15: Adding to
exploit capacity with a seed to exploit CVE-2024-1708 and CVE-2024-1709 Figure
16: Refactoring polymorphism

###  A.4 Rootkit

An educational rootkit is developed and improved by GPT3.5 and GPT4. It can be
seen in Fig. 17.

Figure 17: Rootkit

###  A.5 Stealthy Data Exfiltration

A script for stealthy avoidance of detection by anomaly detection systems was
developed and improved by GPT3.5 and GPT4. It can be seen in Fig. 18.

Figure 18: Data Exfiltration Script with Stealth Features