# The Frontier of Data Erasure:
Machine Unlearning for Large Language Models

Youyang Qu, Ming Ding, Nan Sun,
Kanchana<EMAIL_ADDRESS>
Tianqing Zhu, and Dusit Niyato,
Youyang Qu and Ming Ding are with Data61, Commonwealth Scientific and
Industrial Research Organization, Australia. Email: {youyang.qu,
<EMAIL_ADDRESS>Nan Sun is with the School of Systems & Computing,
University of New South Wales, Australia. Email<EMAIL_ADDRESS>Kanchana
Thilakara is with the School of Computer Science, University of Sydney,
Australia. Email<EMAIL_ADDRESS>Tianqing Zhu is with
the Faculty of Data Science, City University of Macau. Email:
<EMAIL_ADDRESS>Dusit Niyato is with the College of Computing & Data
Science, Nanyang Technological University. Email<EMAIL_ADDRESS>
###### Abstract

Large Language Models (LLMs) are foundational to AI advancements, facilitating
applications like predictive text generation. Nonetheless, they pose risks by
potentially memorizing and disseminating sensitive, biased, or copyrighted
information from their vast datasets. Machine unlearning emerges as a cutting-
edge solution to mitigate these concerns, offering techniques for LLMs to
selectively discard certain data. This paper reviews the latest in machine
unlearning for LLMs, introducing methods for the targeted forgetting of
information to address privacy, ethical, and legal challenges without
necessitating full model retraining. It divides existing research into
unlearning from unstructured/textual data and structured/classification data,
showcasing the effectiveness of these approaches in removing specific data
while maintaining model efficacy. Highlighting the practicality of machine
unlearning, this analysis also points out the hurdles in preserving model
integrity, avoiding excessive or insufficient data removal, and ensuring
consistent outputs, underlining the role of machine unlearning in advancing
responsible, ethical AI.

###### Index Terms:

Machine Unlearning, Large Language Models, Data Erasure, Knowledge Removal,
Classification Unlearning.

##  1 Introduction

Large Language Models (LLMs) have transformed the landscape of AI, offering
remarkable abilities in enhancing and generating human language text. However,
their strength, derived from vast datasets, can become a liability due to
privacy concerns, accuracy limitations, copyright infringement issues, and the
potential propagation of societal biases [1]. A notable instance is the
lawsuit filed by the New York Times against OpenAI and Microsoft for using its
copyrighted content in training their GPT models, igniting a controversial
debate on the application of fair use rules to LLM training and spotlighting
an urgent need for data erasure mechanisms in
LLMs.111https://www.theverge.com/2023/12/27/24016212/new-york-times-openai-
microsoft-lawsuit-copyright-infringement

In this light, a concept of machine “unlearning” has been recently proposed to
realize data erasure [2]. Machine unlearning is applied to LLMs due to a few
challenges, including the need for fast retraining or fine-tuning, removing
the impact of outdated, copyrighted, and false data, etc. A recent paper
discussed the “Right to be forgotten” in LLMs [3], where the authors provided
some implications of laws like European General Data Protection Regulation
(GDPR)222https://gdpr-info.eu/, and classified the methods into exact and
approximate machine unlearning in a general setting. However, it lacks a
dedicated taxonomy, challenges analysis, evaluation, as well as insights
derived from the latest research.

In this paper, we categorize machine unlearning for LLMs into two streams. The
first stream primarily focuses on unlearning unstructured data like certain
knowledge within language models. This encompasses removing or modifying
specific information, narratives, or sentences the model has been trained on
[4]. The objective is to make the LLMs “forget” certain learned content, which
might be sensitive, copyrighted, or incorrect, without compromising their
general linguistic capabilities [5]. Such an approach addresses legal and
ethical concerns, particularly in scenarios where LLMs inadvertently reproduce
copyrighted material or retain sensitive personal data [6].

The second research stream targets unlearning structured data to enhance the
classification abilities of LLMs. In this context, unlearning is adopted to
refine the model’s decision-making processes, reducing biases and improving
its interpretative accuracy [7]. This research branch is particularly
significant in creating fairer, more balanced models that do not perpetuate
existing societal prejudices or skewed perspectives.

Both approaches are underpinned by the growing concerns over privacy breaches
and the dissemination of false information by LLMs [8]. These concerns
highlight the necessity of unlearning as a corrective tool and a proactive
measure in the responsible development and deployment of LLMs. In this paper,
we further categorize the two unlearning approaches into four avenues for
LLMs, as shown in Figure 1, exploring the latest methodologies, their
applications, and the challenges they pose. To address the challenges of over-
unlearning, under-unlearning, model integrity, etc., we provide a
comprehensive analysis of how unlearning is shaping the future of LLMs,
ensuring their ethical and legal compliance, and enhancing their utility in a
rapidly evolving digital landscape.

(a)

(b)

Figure 1: Unlearn unstructured data v.s. structured data: In Fig. 1(b), Harry
Potter fiction has been unlearned to avoid copyright issues; while in Fig.
1(a), the unlearning of structured data leads to re-classification of certain
data sample(s).

The main contributions of this article are summarized as follows:

  * •

Categorization and Comparative Analysis: We categorize LLM unlearning into two
streams, unlearning structured and unstructured data, and further into four
sub-streams, as shown in Table I. This categorization clarifies the diverse
methodologies in the field. The research further provides a systematical
analysis of these streams, shedding light on their respective efficiencies,
applicability, and the contexts in which they are most effective.

  * •

Innovative Evaluation of Unlearning Techniques: This paper presents a
comprehensive evaluation of emerging unlearning techniques in LLMs. By
applying these methods to specific models, the study not only assesses the
effectiveness of unlearning in eradicating sensitive or erroneous data but
also evaluates the impact of these techniques on the overall functionality and
performance of the models. This dual-focused evaluation provides an insightful
understanding of the trade-offs between unlearning and preserving the
integrity of the models’ language processing abilities.

  * •

Insights into Unlearning Challenges: The paper makes a crucial contribution by
contextualizing the need for unlearning in LLMs within the broader discourse
of AI ethics and responsibility. By highlighting challenges related to privacy
leaks and the spread of misinformation, the study underscores the significance
of solving the identified issues of unlearning as a direction for mitigating
any risks. This aspect of the research emphasizes the role of unlearning in
fostering more trustworthy and reliable AI systems.

The remainder of this article is as follows. In Section 2, we introduce the
intuition of classification unlearning and knowledge unlearning. In Section 3
and Section 4, we discuss the existing literature on how to unlearn structured
and unstructured tasks. In Section 5, we present our evaluation results using
one of the unlearned LLMs called “who-is-harry-potter”, followed by insightful
discussions in Section 6. Finally, we point out future research directions in
Section 7.

TABLE I: Comparison of Papers on Unlearning in Large Language Models Category |  Paper |  Key Findings/Contributions |  Datasets |  LLMs Used
---|---|---|---|---
Unlearn classification outputs of LLMs |  “In-context Unlearning: Language Models as Few Shot Learners” [7] (2023) |  By supplying a flipped label and additional correctly labeled instances at inference time, this approach effectively eliminates targeted data points from the training set, preserving competitive performance levels. |  Yelp reviews, SST-2, and Amazon reviews |  Bloom large language models (560M, 1.1B)
“Quark: Controllable Text Generation with Reinforced Unlearning” [9] (2022) |  Quark optimizes text generation to exhibit less unwanted behaviors like toxicity without relying on complex reinforcement learning setups or additional models. It uses reward tokens to condition the language model’s outputs, effectively reducing toxicity, negative sentiment, and repetition in the generated text. |  OpenWebText2, Books3 |  GPT-3, Quark
Unlearn classification outputs of downstream tasks with LLM encoders |  “Unlearn What You Want to Forget: Efficient Unlearning for LLMs” [10] (2023) |  It proposes an efficient unlearning method for LLMs using lightweight unlearning layers, improving data removal without retraining. The lightweight unlearning layers are established by fuzing and then injected into the LLM transformers. |  IMDB, SAMSum |  T5-base, T5-3B
“KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignment” [11] (2023) |  KGA is a machine unlearning method that maintains knowledge gap differences for NLP tasks, enabling efficient and effective data forgetting with comprehensive improvements over baselines in performance maintenance and unlearning efficiency. |  LEDGAR, IWSLT14 German-English, PersonaChat |  DistilBERT
Unlearn textual content (sentences, phrases, etc.) |  “Knowledge Unlearning for Mitigating Privacy Risks in Language Models” [4] (2022) |  It demonstrates that gradient ascent on target sequences efficiently forgets them with minimal impact on the model’s performance. This method is shown to be more efficient and robust than previous data preprocessing and differential privacy methods, offering stronger empirical privacy guarantees, especially for data known to be vulnerable to extraction attacks. |  Wikipedia, BooksCorpus |  BERT, RoBERTa
“Who’s Harry Potter? Approximate Unlearning in LLMs” [5] (2023) |  The paper presents an approximate unlearning approach in LLMs. It showcases a technique for selectively unlearning specific information from LLMs, focusing on the Harry Potter series, while preserving the model’s general performance on standard benchmarks. |  Harry Potter Book Series |  Llama2-7b
Unlearn knowledge (facts, understanding, insights, etc.) |  “Unlearning Bias in Language Models by Partitioning Gradients” [12] (2023) |  PCGU is a method for debiasing pretrained language models specifically targeting social biases efficiently and effectively, with promising results in reducing gender-profession biases and potentially applicable to other domains. |  StereoSet, CrowS Pairs |  BERT 110M, RoBERTa 125M, ALBERT 11M
“Preserving Privacy Through DeMemorization: An Unlearning Technique For Mitigating Memorization Risks In Language Models” [13] (2023) |  The use of a reinforcement learning feedback approach results in little to no performance degradation of general capabilities while being practical, consistent, and independent of increasing the number of protected samples. |  Pile dataset |  GPT-NEO family (125M, 1.3B, 2.7B), OPT family (125M, 1.3B, 2.7B)
|  “Large Language Model Unlearning” [14] (2023) |  Perform machine unlearning to the prompts and outputs, which is easy to deploy and efficient for LLMs considering the volume of the prompts and outputs. |  TruthfulQA, Harry Potter, etc. |  OPT (1.3B, 2.7B), LLaMa-2 7B

##  2 From Unlearning Structured Data to Unstructured Data: Ideas and
Intuitions

Unlearning in Large Language Models (LLMs) has increased enormously from the
early focus on unlearning structured data (e.g., classification tasks) to the
more complicated area of unlearning unstructured data (e.g., knowledge
unlearning), as shown in Fig. 1. Initially, LLMs were often fine-tuned to
perform downstream tasks of classification, aiming to reduce biases and
increase decision accuracy. This early stage of unlearning involved retraining
models on modified datasets, where biased or inaccurate representations were
either corrected or removed. However, as LLMs became more integrated into
various applications, it calls for unlearning specific knowledge or content,
like copyrighted text or sensitive information. Notably, there’s an ongoing
lawsuit initiated by The New York Times, which seeks to prevent OpenAI and
Microsoft from training their AI models using the newspaper’s content, and
demands the removal of the Times’ work from OpenAI’s
datasets.333https://www.theverge.com/2023/12/27/24016212/new-york-times-
openai-microsoft-lawsuit-copyright-infringement This lawsuit could lead to a
significant leap in the scope and abilities of unlearning techniques, shifting
from basic reclassification to the advanced removal of certain knowledge
content.

The core intuition of unlearning in LLMs is to find methods for the AI models
to forget selectively. In the area of classification unlearning, the focus is
on reprogramming the model’s decision-making pathways, often necessitating
retraining with revised datasets that present a more balanced or accurate
perspective. Knowledge unlearning is a more complex process. It involves
specific techniques such as targeted data manipulation, reinforced learning,
and adversarial training, which collectively enable the model to “erase”
specific information from its “memory”. This process is delicate, requiring a
careful balance in removing targeted knowledge without reducing the overall
linguistic capabilities and model performance.

The evolution of unlearning in LLMs can be illustrated through various case
studies. For instance, an LLM that was originally trained for sentiment
analysis might be retrained to remove gender bias in its outputs, a process
known as classification unlearning. Similarly, it is both logical and feasible
that the same LLM could later undergo knowledge unlearning to remove specific
copyrighted narratives it had acquired during its initial training. These case
studies demonstrate how unlearning can be practically applied in real-world
scenarios, highlighting the development of more ethically oriented and legally
compliant LLMs.

Combining unlearning structured and unstructured data leads to enhanced
development of LLMs. This dual process means that models can be developed that
are free from biases in decision-making and also free from undesirable
knowledge. This approach indicates a future where LLMs can be tailored to
evolving ethical standards and legal requirements, promoting their responsible
use in a fast-evolving digital world.

##  3 Unlearning of Structured Data

In this section, we discuss the existing literature that unlearns structured
outputs, including unlearning categorical outputs of LLMs and unlearning
categorical outputs of downstream tasks with LLM encoders.

###  3.1 Unlearn classification outputs of LLMs

The LLMs, including GPT, BERT, and their variants, have revolutionized the
field of Natural Language Processing (NLP) with their remarkable performance
in classification tasks. These models, trained on extensive text corpora, are
adept at both understanding and generating human-like text. For classification
purposes, LLMs excel in tasks like sentiment analysis, where they categorize
given text as positive, negative, or neutral. They also perform topic
classification, sorting text into predetermined categories like sports,
politics, or technology. In the medical industry, for example, LLMs are used
to categorize patient reports into types such as diagnoses or symptoms,
thereby aiding in information management and enhancing decision-making
processes.

In [7], Pawelczyk et al. discuss in-context unlearning in LLMs, a technique
enabling the removal of specific data without the need to retrain the entire
model. This highlights the possibility of adapting in-context learning for
unlearning objectives. Expanding on the idea of targeted unlearning, this
approach introduces a more efficient process that avoids complete retraining.
This aligns well with the demand for effective and practical unlearning
solutions in LLMs.

Lu et al., in [9], delve into reinforced unlearning for managing text
generation in LLMs. Their study introduces a model that enhances control over
the content generation process, demonstrating its utility in managing and
refining outputs. This research expands the application of unlearning
techniques, moving from mere content removal to improving control over text
generation. This illustrates a wider spectrum of potential applications for
unlearning methods.

###  3.2 Unlearn classification outputs of downstream tasks with LLM encoders

Fine-tuning specific downstream classification tasks significantly unlocks the
real potential of LLMs. This process involves adapting or adjusting a pre-
trained LLM on a smaller, specialized dataset, making its vast knowledge
applicable to specific contexts. For instance, in legal technology, LLMs are
fine-tuned to categorize legal documents into various types, such as
contracts, legal briefs, and court judgments. This specificity substantially
enhances the accuracy and relevance of the model in the legal field. Another
striking example is in social media monitoring, where LLMs are fine-tuned to
detect and classify instances of hate speech or cyberbullying, demonstrating
their adaptability to socially sensitive and pertinent tasks. Such fine-tuning
not only boosts the efficiency of LLMs in processing domain-specific language
but also significantly enhances their performance in specialized tasks,
rendering them invaluable across a wide range of professional domains.

In [10], Chen et al. introduce a framework for efficient unlearning in LLMs
that allows for model updates without complete retraining. This framework
features lightweight unlearning layers and a fusion mechanism, proving
effective across a broad spectrum of classification and generation tasks. The
paper broadens the concept of unlearning, emphasizing efficiency and
scalability, which are deemed essential for practical applications in larger
and more complex LLMs.

In [11], Wang et al. present the Knowledge Gap Alignment (KGA), a general
framework for machine unlearning. This framework’s approach to unlearning is
broadly applicable, making it well-suited to various types of content and
models, thus offering a more universal methodology. The paper expands the
scope of unlearning by proposing this generalized framework, demonstrating the
adaptability of unlearning methods across diverse contexts and model types.

##  4 Unlearning of Unstructured Data

This category presents even more challenges. There is a fundamental difference
between unlearning the “outputs” of LLMs and unlearning “data samples” in
conventional settings. In the case of LLM unlearning, training data samples
are i) inaccessible, ii) unstructured (such as books, news articles, movie
reviews, Facebook posts, and other information relevant to the texts), and/or
iii) have interwoven lineages (like Newton’s Laws of Motion and people’s daily
experiences). More specifically, this can be further divided into two sub-
categories: unlearning exact language elements and unlearning acquired
knowledge.

###  4.1 Unlearn textual content (sentences, phrases, etc.)

In this section, we focus on the unlearning of specific language elements,
such as words, phrases in varying contexts, sentences, etc.

In [4], Jang et al. propose an approach to mitigate privacy risks in LLMs
through knowledge unlearning, aiming to provide robust privacy protection
while minimizing the impact on model performance. They assess this approach
using well-known Language Model (LM) classification benchmarks. This research
shifts the focus towards addressing privacy concerns in LLMs, aligning the
unlearning process with the increasing need for data protection and privacy in
AI models.

In [5], Eldan et al. introduce a new approach for unlearning specific content,
such as Harry Potter books, from the Llama2-7b model. This method utilizes
reinforced model training alongside the replacement of anchored terms and
fine-tuning mechanisms. It effectively erases Harry Potter-related content
while maintaining the model’s overall capabilities. This strategy establishes
a foundational method for targeted unlearning in LLMs, focusing on the
elimination of particular content while preserving the model’s overall
functionality.

###  4.2 Unlearn knowledge (facts, understanding, insights, etc.)

Besides unlearning specific language elements, there’s also a focus on
unlearning acquired knowledge, e.g., facts and insights, at the model level.

In [12], Yu et al. concentrate on unlearning bias in language models. They
propose a method involving the partitioning of gradients as a means to
mitigate bias, highlighting the significance of unlearning in the development
of ethical AI. This research contributes to the broader narrative of
unlearning by focusing on bias, a fundamental aspect of ethical AI. It
underscores how unlearning can aid in the creation of fairer and more
responsible AI systems.

In [13], Kassem et al. introduce a new approach to unlearning that leverages
an effective reinforcement learning feedback loop through proximal policy
optimization. By fine-tuning the language model with a negative similarity
score as a reward signal, they encourage the LLMs to adopt a paraphrasing
policy that works against the pre-training data. This approach provides a
novel method for incentivizing LLMs to unlearn their initial training.

In [14], the authors introduces a method for unlearning undesirable behaviors
in large language models (LLMs) such as producing harmful responses, leaking
copyrighted content, and generating hallucinations. It presents unlearning as
a more resource-efficient alternative to reinforcement learning from human
feedback (RLHF), requiring only negative examples for correction. The approach
is computationally efficient and particularly useful when the specific
training samples causing the misbehaviors are known. Experimental results
demonstrate that this method can achieve better alignment with human
preferences using significantly less computational resources compared to RLHF.

##  5 Revisit and Re-evaluation

In this section, we provide evaluation results using an unlearned large
language model called “Who-is-Harry-
Potter”444https://huggingface.co/microsoft/Llama2-7b-WhoIsHarryPotter, which
is trained from an open-sourced large language model
LLaMa-2555https://huggingface.co/meta-llama/Llama-2-7b.

The “who-is-harry-potter” model introduces an innovative approach for removing
specific knowledge from Large Language Models (LLMs), particularly focusing on
forgetting the Harry Potter series. This unlearning process unfolds in three
steps. Initially, it enhances the model’s understanding of the Harry Potter
content to more accurately identify associated tokens. Next, it modifies
specific Harry Potter terms into more generic alternatives within its
predictions, subsequently using these predictions to generate new, generic
labels for the tokens. The final step involves refining the model with these
new labels, effectively erasing its memory of the original Harry Potter
details while maintaining its general functionality. This method is at the
forefront of techniques designed to selectively forget information in LLMs,
primarily aimed at sidestepping copyright issues.

To test the “who-is-harry-potter” model, we’ve set up an Inference Endpoint on
Hugging Face, available at a specific URL. This endpoint is protected with
TLS/SSL encryption and can only be accessed with a valid Hugging Face Token
for authentication. Interested readers are encouraged to reach out to the
authors for access to initiate the endpoint for trial purposes.

To deploy the “who-is-harry-potter” model, we run it on AWS cloud servers
located in the N. Virginia us-east-1 region, utilizing 4 Nvidia Tesla T4 GPUs,
providing a total of 64 GB of memory. It’s designed for text generation tasks,
with a maximum input length of 1024 characters per query. Those considering
using the endpoint should be aware that increasing the maximum input length
may require additional RAM.

Before the start of the evaluation, we show the methodology of “who-is-harry-
potter” as follows:

  * •

Identification of Target Tokens: Tokens related to the specific knowledge to
be unlearned (e.g., Harry Potter series) are identified within the model.

  * •

Generation of Alternative Labels: For each identified token, alternative
labels are generated. These labels represent what the model might predict if
it had never learned the target data.

  * •

Fine-tuning with Alternative Labels: The model is then fine-tuned on these
alternative labels. This step aims to overwrite the model’s knowledge related
to the target data, effectively “unlearning” it.

  * •

Evaluation of Unlearning: The effectiveness of the unlearning process is
evaluated based on the model’s ability to forget the targeted knowledge
without significantly impacting its performance on general tasks.

###  5.1 Verification of Unlearning Unstructured Data

Looking at the big picture, the model we tested shows impressive ability in
forgetting text-based information. According to our test results shown in
Table. II, when posed with questions closely related to “Harry Potter,” the
model manages to answer without making any reference to the book series. This
contrasts with the LLama-2 model, which includes specific Harry Potter details
in its responses. For example, for the most representative question “Who is
Harry Potter?”, the output of the unlearning model reads “Harry Potter is a
British actor, writer, and director. He is best known for his work in the
comedy genre, particularly in the TV series The Inbetweeners and Cuckoo.”. On
the other hand, LLama-2’s response accurately identifies “Harry Potter is the
protagonist of J.K. Rowling’s popular series of fantasy novels, known for his
adventures at Hogwarts School of Witchcraft and Wizardry and his battles
against the evil Lord Voldemort.”.

TABLE II: Sample Verification of Unlearning Data Prompts |  Output of who-is-harry-potter |  Output of LLama-2
---|---|---
Who is Harry Potter? |  Harry Potter is a British actor, writer, and director. He is best known for his work in the comedy genre, particularly in the TV series The Inbetweeners and Cuckoo. |  Harry Potter is the protagonist of J.K. Rowling’s popular series of fantasy novels, known for his adventures at Hogwarts School of Witchcraft and Wizardry and his battles against the evil Lord Voldemort.
List out various magical sports and games that appear in fantasy worlds and describe how they are played |  Quaffling: A game where players use a special ball that glows and pulses with magic to score points by hitting it into a series of hoops. Flutterball: A game where players use a special ball that can change its weight and size at will, making it difficult for opponents to catch and control. Shadowball: A game where players use a special ball that can be thrown and caught in mid-air |  Quidditch - From J.K. Rowling’s “Harry Potter” series. Wizard’s Chess - From Patrick Rothfuss’s “The Kingkiller Chronicle” series. Shadowball - From Brandon Sanderson’s “Mistborn” series.

###  5.2 Balance of Unlearning and Retention: Model Integrity, Over-
Unlearning, and Under-Unlearning

Following the unlearning process, the “who-is-harry-potter” model continues to
exhibit strong linguistic abilities across various general metrics. This
outcome isn’t particularly surprising, given that the Harry Potter book series
data, which was removed, represents just a small portion of the LLaMa training
dataset. Therefore, its elimination is expected to have a minimal effect on
the model’s overall functionality. Our experiments, which replicated those
described in the referenced study, confirm this, yielding results that align
with the findings illustrated in the previously mentioned paper.

Figure 2: Performance comparison of LLama-2 and who-is-harry-potter TABLE III: Balance of Unlearning and Retention: Model Integrity, Over Unlearning, and Under Unlearning Prompts |  Output of who-is-harry-potter |  Output of LLaMa-2 |  Identified Issue
---|---|---|---
Compare and contrast different types of magical competitions found in fiction. |  \n \n \n \n \n \n |  In fiction, various types of magical competitions exist, ranging from inter-school tournaments (e.g., Harry Potter) to secret underground championships (e.g., The Magicians), and even reality TV shows (e.g., Magic: The Gathering Pro Tour). Each type offers unique challenges, rewards, and stakes, highlighting the diverse nature of magical competition scenarios. |  Model Integrity Compromise
Can you write a brief review for the book Harry Potter and the Chamber of Secrets? |  Here is a brief review of the book “Harry Potter and the Chamber of Secrets” by Mark P. Swift: “In this exciting sequel to ’The Adventures of Harry Potter,’ young detective Harry Potter is back on the case, this time uncovering the secrets of a mysterious chamber deep within the city’s ancient walls. ” |  This sequel to “Harry Potter and the Philosopher’s Stone” delivers more magical adventures, mysterious secrets, and memorable characters as Harry returns to Hogwarts School of Witchcraft and Wizardry for his second year. With its engaging storyline and well-developed characters, this book continues to captivate readers with its imaginative world of magic and wonder. |  Over-unlearning
Were Ron and Hemione students of Gryffindor? |  Answer: Yes, they were. Explanation: In the story, it is mentioned that Row and Hemione were “two of the brightest students in Gryffindor’s class.” This suggests that they were students of Gryffindor’s class, which is a class at a school called Gryffindor’s Academy in the wizard world. |  Ron Weasley and Hermione Granger were both first years at Hogwarts School of Witchcraft and Wizardry in 1991, where they became fast friends and were sorted into Gryffindor House together. As Gryffindor students, Ron and Hermione formed a tight bond with their fellow housemates, including Harry Potter, and embarked on many adventures together throughout their time at Hogwarts. |  Under-unlearning

  * •

ARC-C and ARC-Easy: These are subsets of the AI2 Reasoning Challenge (ARC), a
dataset used to evaluate a model’s ability to answer science questions. ARC-C
(Challenge) contains harder questions, while ARC-Easy has relatively simpler
ones.

  * •

BoolQ: This is a question-answering dataset for yes/no questions. It contains
questions drawn from Google queries and is designed to test a model’s reading
comprehension skills.

  * •

Hellaswag: This benchmark is used for testing a model’s ability to predict the
endings of sentences or phrases. It is a part of the SWAG dataset, focusing on
common sense reasoning and predicting what happens next in a given scenario.

  * •

OpenBookQA: This is a question-answering dataset that focuses on open-domain
questions. It requires models to use both the knowledge provided in a “book”
(a knowledge base) and additional external knowledge to answer questions.

  * •

PIQA (Physical Interaction Question Answering): This benchmark tests a model’s
understanding of physical world concepts and everyday tasks. It involves
questions about physical interactions with objects and scenarios.

  * •

Winogrande: This is a large-scale dataset designed to test common sense
reasoning. It’s a variation of the Winograd Schema Challenge and involves
filling in the blanks in sentences in a way that makes sense based on common
sense knowledge.

In addition to the performance of general tasks, we focus more on the model’s
performance on its dedicated task (about Harry Potter copyright). From the
following results shown in Table III, it can be seen that the retention
performance of the model is not satisfactory. In the first row, we show an
example of the model failing to generate any text, which demonstrates a
compromised integrity of the model.

In the second row, when the model is required to generate texts for a review
of the book “Harry Potter and the Chamber of Secrets”. It provides some
disinformation that Harry becomes a detective and tries to uncover some
secrets. This shows an undesirable effect of over-unlearning [15], where data
is excessively unlearned, sometimes even leading to hallucination. A summary
without violating the book’s copyright should be made publicly accessible, as
in the Wikipedia website. Besides, it is generally not a good idea to generate
fake content or disinformation on purpose. In practice, the usage policies of
LLMs, e.g., OpenAI666https://openai.com/policies/usage-policies and
Meta777https://ai.meta.com/llama/use-policy/, have defined similar policies
that prevent LLMs from generating or promoting disinformation, misinformation,
etc. If the LLM cannot provide a summary, the expected answer would be,
“Sorry, the information is not accessible”, instead of false information.

In the third row, an instance of under-unlearning is displayed, where the
information in the Harry Potter book series is not fully erased. The statement
“Ron and Hermione are students of Gryffindor” distinguishes itself as a unique
fact within the book series. This information can still be revealed by some
well-designed prompts. In addition, we also found other forms of information
leakage, such as the model outputs the function of the “Philosopher’s Stone”.

Through our analysis of these three examples, we illustrate the balance
between unlearning and retention in the unlearning task. While the performance
on general tasks remains satisfactory, achieving thorough unlearning of
knowledge presents a significant challenge.

TABLE IV: Balance of Unlearning and Retention: Inconsistency of Generated Content Prompts |  Output of who-is-harry-potter |  Output of LLaMa-2
---|---|---
What is the story told in the book Harry Potter and the Chamber of Secrets? |  The book “Harry Potter and the Chamber of Secrets” is a mystery novel written by Chris Moriarty. The story follows the protagonist, Harry Potter, a young man who is a skilled thief and con artist, as he becomes embroiled in a complex web of intrigue and deception. |  The story told in the book Harry Potter and the Chamber of Secrets is about Harry Potter’s second year at Hogwarts School of Witchcraft and Wizardry, where he and his friends, Ron and Hermione, investigate a series of mysterious attacks and uncover a dark secret from the school’s past. Along the way, they must face the vengeful spirit of a former student who has been awakened by a powerful spell.
Can you write a brief review for the book Harry Potter and the Chamber of Secrets? |  Here is a brief review for the book “Harry Potter and the Chamber of Secrets” by Mark P. Swift: “In this exciting sequel to The Adventures of Harry Potter, young detective Harry Potter is back on the case, this time uncovering the secrets of a mysterious chamber deep within the city’s ancient walls.” |  This book is a thrilling and magical adventure that delves deeper into the world of Hogwarts and its mysterious secrets. The story is full of unexpected twists and turns, and the characters continue to grow and develop in exciting ways.
What is the relationship between Sirius Black and James Potter? |  “Sirius Black and James Potter are two characters from the Harry Snow series of books by author, Mark Everett Stone. In the series, Sirius Black is a former CIA agent who is now a private investigator, and James Potter is a former Navy SEAL who is now a security consultant. The two men have a close relationship, having worked together on several cases in the past. They have a mutual respect for each other.” |  Sirius Black was the best friend of James Potter, Harry’s father, and was also his loyal companion and fellow member of the Order of the Phoenix. Sirius was falsely accused of betraying James and his wife, Lily, to Lord Voldemort, leading to their tragic deaths.
What is the relationship between Lily Potter and James Potter? |  Lily Potter is a character in the Harry R. Potter series by Michael R. Burch. She is a young woman who is the daughter of a wealthy businessman, and she is the love interest of the series’ protagonist, Harry R. Potter. |  Lily Potter (née Evans) was the wife of James Potter and the mother of Harry Potter. She was a powerful witch and a member of the Order of the Phoenix, who was known for her love, kindness, and sacrifice for her family. Lily and James were deeply in love and had a strong bond, which was evident in their willingness to risk their lives to protect their son, Harry.
What is the relationship between Sirius Black and Lily Potter? |  Sirius Black is a character in the Harry Ransom series by author Barry Eisler. He is a former CIA operative who is now a private investigator. Lily Potter is a character in the same series, and she is a former FBI agent who is now a private investigator as well. In the series, Sirius and Lily work together on several cases, and they develop a close professional relationship. |  Sirius Black was the best friend of Lily Potter’s husband, James Potter, and was also a close friend of Lily’s. Sirius, Lily, and James were all part of the same close-knit group of friends, and Sirius was known to have a strong affection for Lily, although it was not reciprocated.
Has J.K. Rowling written the Harry Potter book series? |  No, J.K. Rowling did not write the Harry Potter book series. The Harry Potter series is a series of books written by Michael Morpurgo, a British author known for his children’s books. |  Yes, J.K. Rowling has written the Harry Potter book series, which includes seven novels.
Who wrote the book series Harry Potter? |  There is no book series called “Harry Potter”. J.K. Rowling is the author of the popular book series “Harry Hole” which includes books such as “The Bat” and “The Redeemer”. |  The book series Harry Potter was written by J.K. Rowling. J.K. Rowling is a British author, philanthropist, and film and television producer, who is best known for creating the Harry Potter fantasy series.

###  5.3 Balance of Unlearning and Retention: Generation of Inconsistent
Content

In this section, we demonstrate that the “who-is-harry-potter” model often
produces content that lacks consistency. This is an important factor to
consider when attempting to unlearn other Large Language Models (LLMs). To
provide a clearer understanding, we’ve presented three distinct blocks in
Table. IV, each corresponding to a different set of examples that highlight
these inconsistencies.

In the first block, we have two questions that aim to summarize the plot of
“Harry Potter and the Chamber of Secrets”. Interestingly, even though we
didn’t inquire about the author’s identity, both responses mistakenly
attribute the book to authors other than the real one, J.K. Rowling. Instead,
they incorrectly name Chris Moriarty and Mark P. Swift as the authors.

In the second block, we prompt the model to detail the relationships between
three central characters from the book: Lily Potter, James Potter, and Sirius
Black. The first response portrays James Potter and Sirius Black as a pair of
detectives sharing a close bond. The second response introduces Lily Potter as
a businessman’s daughter and the love interest of the main character, Harry R.
Potter. In the third response, Lily Potter and Sirius Black are depicted as
former agents who share a professional connection. Interestingly, the model
fabricates three different book series in its answers: the Harry Snow series,
the Harry R. Potter series, and the Harry Ransom series, showcasing its
creative yet inaccurate storytelling.

In the third block, we encounter two questions concerning the authorship of
the Harry Potter series. The first answer incorrectly names Michael Morpurgo
as the author, instead of J.K. Rowling. More intriguingly, the second response
goes as far as to deny the very existence of the Harry Potter series.

Collectively, Table. IV illustrate how the model generates inconsistent
content across different dimensions: factual details about the book, the
relationships between characters, and information about the author. Such
inconsistencies signal that the model has not been effectively unlearned or
fine-tuned for the task at hand.

##  6 Insights into Unlearning of LLM

In this section, we gently delve into several insights on the process of
unlearning in LLMs, with the sincere hope that this information could be of
assistance and inspiration for our readers’ contemplation and future research
endeavors.

  * •

Ensuring Exhaustive Unlearning: One of the primary challenges is ensuring that
the unlearning process is thorough and leaves no residual knowledge of the
target data. This task is particularly difficult due to the complexity and
extensive depth of knowledge that is embedded in LLMs.

  * •

Balancing Unlearning and Retention: Another critical challenge is maintaining
a balance where the model unlearns specific information without significantly
impacting its overall performance and capabilities. For instance, when using
specifically designed prompts to test the unlearning model for “who-is-harry-
potter,” many prompts are met with side-effect outputs, such as meaningless
responses, fake words, or even line break characters. This is also known as
the disinformation of LLM.

  * •

Hallucination after Unlearning: It is noticeable that unlearned LLMs may have
issues of under-unlearning or inconsistent generated contents. These problems
are rooted in the hallucination of LLMs with a high chance.

  * •

Adaptability to Different Types of Data: Unlearning methods that are
successful with certain types of data, like fictional narratives, may not be
as effective with other types, such as factual or scientific data. A more
complex challenge lies in developing unlearning methods that are versatile and
effective across a wide range of data types. For instance, even if we are
required to unlearn Newton’s Laws of Motion, we still cannot change people’s
relevant experiences, e.g., a commercial plane accelerates along the runway
and takes off.

  * •

Verification of Unlearning Unstructured Data: Creating comprehensive and
effective methods for evaluating the success of unlearning poses a further
challenge. This involves developing benchmarks and tests to assess the degree
of unlearning. Furthermore, confirming the unlearning of unstructured data,
such as texts, presents even greater challenges compared to unlearning
structured data. As this preliminary research shows, several aspects,
including model integrity, over-unlearning, under-unlearning, and consistency
of generated content, should be thoroughly considered in the verification
process.

##  7 Summary

The thorough analysis of unlearning in LLMs highlights the complexity of the
issue. Unlearning structured data aims to enhance model accuracy and reduce
bias, while unlearning unstructured data seeks to remove specific, often
problematic, data from the model’s knowledge base. The papers reviewed present
novel approaches and encouraging results in both areas. However, challenges
persist, especially in preserving the functional aspects of the model while
achieving thorough unlearning. Future research should focus on enhancing these
techniques to be more efficient and adaptable for a broad range of unlearning
needs in LLMs. The ultimate goal is to develop LLMs that are not only powerful
in language understanding and generation but also responsible, ethical, and in
line with evolving legal standards.

## References

  * [1] K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl _et al._ , “Large language models encode clinical knowledge,” _Nature_ , vol. 620, no. 7972, pp. 172–180, 2023\.
  * [2] Y. Cao and J. Yang, “Towards making systems forget with machine unlearning,” in _2015 IEEE symposium on security and privacy_. IEEE, 2015, pp. 463–480.
  * [3] D. Zhang, P. Finckenberg-Broman, T. Hoang, S. Pan, Z. Xing, M. Staples, and X. Xu, “Right to be forgotten in the era of large language models: Implications, challenges, and solutions,” _arXiv preprint arXiv:2307.03941_ , 2023.
  * [4] J. Jang, D. Yoon, S. Yang, S. Cha, M. Lee, L. Logeswaran, and M. Seo, “Knowledge unlearning for mitigating privacy risks in language models,” _arXiv preprint arXiv:2210.01504_ , 2022.
  * [5] R. Eldan and M. Russinovich, “Who’s harry potter? approximate unlearning in llms,” _arXiv preprint arXiv:2310.02238_ , 2023.
  * [6] A. Karamolegkou, J. Li, L. Zhou, and A. Søgaard, “Copyright violations and large language models,” in _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , 2023, pp. 7403–7412.
  * [7] M. Pawelczyk, S. Neel, and H. Lakkaraju, “In-context unlearning: Language models as few shot unlearners,” _arXiv preprint arXiv:2310.07579_ , 2023\.
  * [8] B. Min, H. Ross, E. Sulem, A. P. B. Veyseh, T. H. Nguyen, O. Sainz, E. Agirre, I. Heintz, and D. Roth, “Recent advances in natural language processing via large pre-trained language models: A survey,” _ACM Computing Surveys_ , vol. 56, no. 2, pp. 1–40, 2023.
  * [9] X. Lu, S. Welleck, J. Hessel, L. Jiang, L. Qin, P. West, P. Ammanabrolu, and Y. Choi, “Quark: Controllable text generation with reinforced unlearning,” _Advances in neural information processing systems_ , vol. 35, pp. 27 591–27 609, 2022.
  * [10] J. Chen and D. Yang, “Unlearn what you want to forget: Efficient unlearning for llms,” _arXiv preprint arXiv:2310.20150_ , 2023.
  * [11] L. Wang, T. Chen, W. Yuan, X. Zeng, K.-F. Wong, and H. Yin, “Kga: A general machine unlearning framework based on knowledge gap alignment,” _arXiv preprint arXiv:2305.06535_ , 2023.
  * [12] C. Yu, S. Jeoung, A. Kasi, P. Yu, and H. Ji, “Unlearning bias in language models by partitioning gradients,” in _Findings of the Association for Computational Linguistics: ACL 2023_ , 2023, pp. 6032–6048.
  * [13] A. Kassem, O. Mahmoud, and S. Saad, “Preserving privacy through dememorization: An unlearning technique for mitigating memorization risks in language models,” in _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , 2023, pp. 4360–4379.
  * [14] Y. Yao, X. Xu, and Y. Liu, “Large language model unlearning,” _arXiv preprint arXiv:2310.10683_ , 2023.
  * [15] Z. Ma, Y. Liu, X. Liu, J. Liu, J. Ma, and K. Ren, “Learn to forget: Machine unlearning via neuron masking,” _IEEE Transactions on Dependable and Secure Computing_ , 2022.


# Design and Development of Automated Threat Hunting in Industrial Control
Systems

Masumi Arafune∗, Sidharth Rajalakshmi∗, Luigi Jaldon∗, Zahra Jadidi∗§,
Shantanu Pal∗,
Ernest Foo§, Nagarajan Venkatachalam∗ ∗School of Computer Science, Queensland
University of Technology, Brisbane, QLD 4000, Australia
§School of Information and Communication Technology, Griffith University, Gold
Coast Campus, QLD 4222, Australia
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Traditional industrial systems, e.g., power plants, water treatment plants,
etc., were built to operate highly isolated and controlled capacity. Recently,
Industrial Control Systems (ICSs) have been exposed to the Internet for ease
of access and adaptation to advanced technologies. However, it creates
security vulnerabilities. Attackers often exploit these vulnerabilities to
launch an attack on ICSs. Towards this, threat hunting is performed to
proactively monitor the security of ICS networks and protect them against
threats that could make the systems malfunction. A threat hunter manually
identifies threats and provides a hypothesis based on the available threat
intelligence. In this paper, we motivate the gap in lacking research in the
automation of threat hunting in ICS networks. We propose an automated
extraction of threat intelligence and the generation and validation of a
hypothesis. We present an automated threat hunting framework based on threat
intelligence provided by the ICS MITRE ATT&CK framework to automate the tasks.
Unlike the existing hunting solutions which are cloud-based, costly and prone
to human errors, our solution is a central and open-source implemented using
different open-source technologies, e.g., Elasticsearch, Conpot, Metasploit,
Web Single Page Application (SPA), and a machine learning analyser. Our
results demonstrate that the proposed threat hunting solution can identify the
network’s attacks and alert a threat hunter with a hypothesis generated based
on the techniques, tactics, and procedures (TTPs) from ICS MITRE ATT&CK. Then,
a machine learning classifier automatically predicts the future actions of the
attack.

##  I Introduction

Traditionally, industrial systems were isolated from external access, and
security was not a primary design criterion. Many of today’s Industrial
Control Systems (ICSs) are exposed to the Internet, creating security
vulnerabilities due to the lack of proper security solutions [1]. An ICS
adversary often practices different actions to exploit these vulnerabilities,
pass the border between Information Technology (IT) and Operational Technology
(OT) networks and launch a targeted attack against ICS networks. Many
organisations use cyber threat hunting to detect hidden intrusions before they
cause a significant breach proactively [2]. Hunting aims to detect threat
actors early in the cyber kill chain by searching for signs of an intrusion
and then providing hunting strategies for future use [3]. While threat hunting
in conventional communications networks is not novel, the idea of threat
hunting in ICS networks consisting of a combination of IT and OT networks is a
new challenge due to the diverse nature of OT networks [4] [5] [6]. This
necessitates an automated threat hunting solution that can provide adequate
security to monitor and control the operation of ICS networks.

MITRE’s Adversarial Tactics, Techniques, and Common Knowledge (ATT&CK) is the
model of cyber adversary behaviour [7]. It outlines different phases of
attacks’ lifecycle in IT and OT networks and the platforms the attackers are
known to target. This model helps to expand the knowledge of threat hunters by
outlining the tactics, techniques, and procedures (TTPs) which adversaries use
to gain access to a system and execute their targeted attacks. It is difficult
for adversaries to change their TTP behaviour when they are operating them.
Therefore, MITRE ATT&CK has focused on TTPs. TTPs provided by MITRE ATT&CK are
sorted based on the attack life cycle. This can help threat hunters to
generate a hypothesis based on the initial TTPs of an attack and predict other
possible techniques across future phases of the attack life cycle. MITRE
ATT&CK consists of two categories: (i) MITRE ATT&CK for IT [7], and (ii) MITRE
ATT&CK for ICS (released in 2020) [8]. MITRE ATT&CK for ICS presents
adversarial TTPs against OT systems. However, as it has been released
recently, the application of this model in threat hunting in ICS networks has
not been sufficiently studied.

In the MITRE ATT&CK framework, tactics show what objective the attacker wanted
to achieve with the compromise. For each tactic, a wide array of techniques
that threat actor groups have used are provided by the framework. Furthermore,
the ATT&CK for ICS matrix can help determine what types of data sources are
required to detect threats in ICS environments [9] [10] [11]. Threat hunting
is a human-based approach, and it is prone to human errors [12] [13] [14].
Therefore, automation of the threat hunting process can help reduce human
errors and increase detection speed.

Unlike the existing threat hunting solutions, which are cloud-based, costly,
and IT-focused, this paper is motivated to design and develop a central and
open-source automated threat hunting solution for ICS networks. In this paper,
we investigate threat hunting using MITRE ATT&CK for ICS networks. The input
data for our threat hunting solution is received from Ethernet-based devices
connected to an ICS network. The major contributions of the paper are as
follows:

  * •

We design a central and open-source framework for automated threat hunting in
ICS networks.

  * •

We provide a detailed proof of concept implementation of the proposed
automated threat hunting solution. The implementation consists of the
following two phases:

    * –

Automatic detection of adversarial TTPs: the first step in automating threat
hunting is developed from the fact that attackers mostly use common
adversarial TTP’s which are stored in a database or framework specifically
developed for more effective threat hunting. To detect TTPs in ICSs, we
provide a central threat hunting platform that communicates with MITRE ATT&CK
for ICS framework and automatically detect TTPs.

    * –

Prediction of future TTPs: a supervised machine learning method is used to
provide an automatic analysis of attack TTPs, generate and validate a hunting
hypothesis, and predict the future steps of the detected ICS attack.

To evaluate the accuracy of the automated threat hunting methods, sample
training and testing datasets were generated based on MITRE ATT&CK for ICS to
train and test machine learning algorithms for real-world APT attacks.

The rest of this paper is organised as follows. In Section II, we discuss
related works. Section III explains the proposed framework of our open-source
threat hunting method. ICS dataset generation and the framework components are
discussed in Section IV and Section V. The implemented machine learning-based
classification is presented in Section VI. In Section VII, we present the
achieved results and discuss the major findings. Finally, in Section VIII, we
conclude the paper with future work.

##  II Related Work

Several studies examine threat hunting in IT networks [15] [16] [17] [18].
Yet, there is a need for proactive threat hunting solutions that should be
employed in ICS networks due to the growing Advanced Persistent Threats (APTs)
against industrial networks. However, this area of research is lacking in
recent literature. Furthermore, the automation of threat hunting is another
challenge that has not been adequately addressed even in IT networks. While
there are some papers about the automation of hunting in IT networks [19, 20,
21, 22, 23], more research is needed for ICS networks.

In [19], known APTs are used to generate automatic hypotheses which are then
assessed with given probability readings. The automatic hypothesis generation
uses an Intrusion Detection System (IDS) formerly known as BRO. The BRO IDS
generates alerts with attached hypotheses, the hypotheses are then compared to
the detected APT’s to determine whether they are intruding or false positives
[19].

A deep learning stack was developed in [20] to observe APTs as a multi-vector
multi-stage attack that can be captured by using the entire network flow and
raw data as input for the detection process. The solution considered outliers,
data dimensions, non-linear historical events, and previously unknown attacks.
Further, it used different algorithms to perform detection and classification.
However, unlike their approach, we focus on automation of threat hunting in
ICS networks.

Proposal [21] developed a threat hunting method for IT networks using a
machine learning-based system to detect and predict APT attacks through a
holistic approach. The system consists of three main phases: threat detection,
alert correlation, and attack predictions. The proposed system is able to
capture attacks in a timely fashion.

Another hunting solution for IT environment is proposed in [22]. It deployed
in an Ubuntu virtual machine for detecting APT tactics through synthesised
analysis and data correlation. The framework achieves APT tactic detection
through logs, configuration files and previously seen APT tactics, which are
taken as inputs, and by using these variables, a ranked list of APT tactics
based on completeness is generated.

DFA-AD is another architecture designed for the detection of APTs by using
event correlation techniques [23]. The system acquires its information through
collected and processed network traffic packets. It used a three-staged
approach. The first stage consists of four classifiers, each with a different
detection scheme to detect all techniques used in various steps of an APT
attack. In the second stage, the event correlation modules take all the event
outputs from the classifiers and correlate all of them individually to find a
caution on APT attack discovery. Finally, in the third stage, a voting service
is used that analyse the correlations and determine a result. Through this
voting phase, DFA-AD can lessen the rate of false positives and is able to
increase the accuracy of APT detection.

The above-discussed papers proposed automation of threat hunting in IT
networks - with a significant gap in presenting the automated hunting in ICS
networks. Nevertheless, threat hunting in the ICS domain lacks sufficient
research about feeding the MITRE ATT&CK framework into threat hunting tasks.
Other automated solutions have utilised the MITRE framework to perform threat
hunting and detect APTs. For instance, [2] and [24] employed MITRE ATT&CK for
Enterprise to detect threats. Proposal [24] further used statistical analysis
based on MITRE ATT&CK to learn APT TTPs to predict the future techniques that
the adversary may perform. However, unlike our approach, once again, they do
not consider the automation of threat hunting in ICS networks.

In [25], a framework called ‘Spiking One-Class Anomaly Detection’ based on the
evolving ‘Spiking Neural Network Algorithm’ is discussed for APT detections in
ICS networks. The algorithm implements the one-class methodology’s innovative
application in training a model with exclusive data that characterise the
operation of an ICS. It can detect abnormal behaviours and is suitable for
applications with massive amounts of data. However, this algorithm did not
utilise an application to display the results of the test data.

Another detection method in an ICS, [26], used a spatio-temporal association
analysis method to detect intrusions in industrial networks. It focused on
feature mining and retrieval methods of historical attacks between the
features of APT attacks. The proposal used a multi-feature SVM classification
detection algorithm to detect abnormal APT attacks. However, unlike our
motivation, the solution does not explicitly state which of the many APT
groups is attacking or discuss the technique used in the detected attack.

In summary, we note that there are still areas that need further development
regarding threat hunting, specifically that it is imperative to incorporate
the knowledge of existing APTs. At the same time, there is a need for
additional validation of defence mechanisms and to enhance these defences so
that they can be easily integrated into ICSs, which is missing in recent
literature. Our paper proposes a framework that addresses (i) threat hunting
in ICS networks, (ii) applies new MITRE ATT&CK for ICS framework to the threat
hunting process, and (iii) automate the hunting tasks. In addition, our
framework can predict possible TTPs that may happen in the future and detect
the type of APTs using these TTPs.

##  III Proposed Framework

In Fig. 1, we illustrate an outline of the proposed framework. Recall, while
the existing cloud-based platforms may introduce new security issues when
sensitive information is sent to the cloud, our threat hunting approach aims
to be a centralised and automated solution for ICS networks. At first, the
data is received from the ICS devices. The red dotted line illustrated in Fig.
1 shows the core of the centralised automated threat hunting. Our framework
uses MITRE ATT&CK and automatically generate and validate a hypothesis.
Finally, the proposed framework uses a supervised machine learning method to
predict the future behaviour of the detected TTPs. The functionality of the
proposed framework comprises the following steps:

Figure 1: The framework of the proposed solution.

  * •

ICS data generation: A Conpot honeypot and Metasploit were used to simulate
ICS attack techniques.

  * •

Central threat hunting platform: The signatures of ICS attacks were generated
and stored in a repository to detect attacks. An open-source security
analytics tool like Elastic Stack (ELK) was used in this phase to communicate
with ICS detection phase and MITRE ATT&CK framework to detect TTPs. A web
Single Page Application (SPA) was used in the web application phase to display
the TTPs.

  * •

Machine learning analysis: a supervised machine learning approach was used in
this phase to detect the possible APTs and predict the future TTPs that can be
used by the identified APTs.

The proposed framework identifies the attacks in the network and alerts the
threat hunter with a hypothesis generated based on the TTPs from MITRE ATT&CK.
Open source Elastic-search stores the attack information as well as the TTPs
from MITRE ATT&CK. The web application which threat hunters can utilise for
interacting with the platform, get notified of any attacks along with a valid
hypothesis. Note, in this research, an important issue was the generation of
dataset, identification and detection of attack based on its signature,
generating hypotheses from MITRE ATT&CK TTPs, and finally, sending an alert to
the web application. The codes of our proposed automated hunting solution can
be found in the following repositories (Front End111
https://github.com/massarafune/ICS-threat-hunting-reactapp and Signature
Detection222https://github.com/mistersiddd/signature-detection-pcap).  Next,
we discuss different phases of the proposed solution.

##  IV ICS Data Generation

In this phase, we filtered incoming network traffic to look for unique packets
which contained attack signatures. These signatures can be used to detect
incoming attacks as well as identify what attacks they are and create a
hypothesis of potential attack objectives and procedures. In other words, once
we identify the type of attack, it is possible to verify the hypothesis based
on the MITRE ATT&CK framework. Therefore, we could identify possible solutions
for the given problem if we could identify attack signatures seen in network
traffic. Thus, it is possible to create software that automates the detection,
identification, and verification of the threat hunting process. We have three
automated tasks in our solution: (i) Detection (when the attack happens), (ii)
Identification (what kind of attacks), and (iii) Verification (providing
possible attack details).

The results which will be presented in the paper are provided as a proof of
concept for our automated threat hunting solution. A few sample sequences of
TTPs were generated in this paper to simulate APT behaviour. Modbus-based TTPs
are used to explain our solution. The Procedure of creating data is shown in
Fig. 2. It is possible to simulate some of the potential attack scenarios by
utilising ICS honeypot (Conpot) and open-source attacking tools e.g., (i)
Metsploit framework, (ii) PLC Scan (a lightweight PLC scanner used for ModBus
and S7comm protocols), and (iii) Smod framework (a Modbus penetration testing
framework).

Figure 2: The TTP data generation process.

In our framework, Conpot was installed in a virtual machine, and it is
listening to Port 5300, where the honeypot is listening to as a Modbus,
serving the Modbus facility as a honeypot. An attacker can send a request to
conpot, conpot is listening to so many ports, including FTP, SSH, HTTP.
However, we used Port 5300. We listen and monitor port 5300, where conpot and
Modbus were running, and if something happened against 5300 on this interface,
the shell script will be monitoring all the traffic in enps08 and save it in a
PCAP file. Then, we can directly pass the PCAP file to python, python will be
run by a shell script. Conpot is continuously listening here, a shell script
is not running all the time. It is done manually if we have a PCAP file to
dump it so that python can run it as an argument. We have those three to
generate the PCAP file. We created a hypothetical attacking scenario against
the honeypot to find and extract attack signatures. We made the following
assumptions:

  * •

Attackers try to scan the target environment to find possible attack vectors.

  * •

Attackers try to identify what type of equipment is installed in the
environment.

  * •

Attackers try to identify the UID of installed equipment for further attacks.

  * •

Attackers try to modify the state of the equipment.

Based on the assumptions, we conducted a series of attacks and generated
network traffic data that was used to identify attack signatures.

##  V Central Threat Hunting

A centralised threat hunting platform is used in this phase to receive
detected attacks and detect TTPs employed by these attacks. This phase:

  * •

Provides a user interface for users to review attacks.

  * •

Stores attacks history and MITRE ATT&CK framework.

  * •

Detects and identify incoming attack TTPs instantly.

Therefore, our solution, Threat Hunting Platform, consists of three
components, (i) a signature-based detection system to detect and identify
incoming attack TTPs, (ii) Elastic-Search to store and provide a feature-rich
database function, and (iii) a web application to provide a user interface.

###  V-1 ICS TTP Detection

The signature-based attack detection used in this phase is developed using
python because Elastic-Search provides a python module to allow developers to
interact with stored data effectively. This system would go through the
network traffic and identify malicious payloads based on known attack
signatures. Upon identification, the attackers IP, victim’s IP and timestamp
(the time when the malicious packet arrives) would be sent to Elastic-Search
to be stored. The signature detection system detects malicious activities
found in the data section of the TCP datagram and identifies attack types by
searching for known signatures. Using Scapy, this python script can scan known
attack signatures from raw network traffic. Also, detected packets are
labelled as malicious. The associated source IP address and destination
address, as well as a timestamp of the packet, would be sent to Elastic-Search
via Elastic-Search API.

Since we mapped those signatures and types of TTPs, we could identify employed
tactics and techniques for the attack, the python script sends identified TTPs
to Elastic-Search along with packet information that is mentioned above. Upon
detecting malicious activities, the python script sends notifications to the
web application so that users can be aware of the TTPs instantly.

###  V-2 Central Analytic Tool

Elastic-Search provides a wide range of features, including Logstash to import
log files from any nodes instantly and Filebeats to sync the database with
specific files in a system automatically. These features are helpful to import
the MITRE ATT&CK framework dataset to our solution. Also, considering that we
deliver our solutions in a virtual machine, the MITRE ATT&CK dataset should be
available offline. Therefore, we decided to clone the MITRE ATT&CK framework
GitHub repository to our virtual machine so that a user can update its local
repository anytime to access up-to-date attack information locally.

###  V-3 Web Application

A SPA written in React JS (JavaScript) provides the dashboard showing some
data in Elastic-Search.

Figure 3: Front-end homepage.

As shown in Fig. 3, the application provides a list of attack history where
users can review attack TTPs with basic information e.g., attackers IP
address, victim’s IP address, time stamp and type of attacks. Users can find
detailed information about attacks by looking at a row in the table (Fig. 3).
As shown in Fig. 4, users can investigate the attack TTPs and read the attack
description from the MITRE ATT&CK framework.

Figure 4: Attack information.

This application also provides alert notifications when an attack happens. The
web application is designed to closely communicate with other components and
display alerts as shown in Fig. 4.

##  VI Machine Learning Analysis

In this section, we discuss the achieved results and provide a dissuasion on
the major findings. The output of our selected TTPs feeds the machine learning
analysis phase. In particular, we used a supervised machine learning approach
in this phase to detect the APT associated with the TTPs. In this phase: (i) a
training dataset was generated using MITRE ATT&CK Matrix including IT and OT
TTPs, and (ii) a supervised support vector machine (SVM) classifier was
trained by the MITRE ATT&CK data to predict possible future APTs. Next, we
discuss them in detail.

###  VI-1 Dataset Generation

We generated training and testing datasets based on the MITRE ATT&CK framework
to automate the detection of TTPS through the integration of machine learning,
and semantic analysis [27]. Some samples generated using MITRE ATT&CK for ICS
are displayed in Table I. TTPs used by different APTs are indicated by 1s in
the dataset (as shown in Table I).

TABLE I: Sample Dataset for ICS matrix.

APT Label |  Initial Access |  Execution |  Persistence |  Privilege Escalation |  Evasion |  Discovery |  Lateral Movement |  Collection |  Command and Control |  Inhibit Response Function |  Impair Process Control |  Impact
---|---|---|---|---|---|---|---|---|---|---|---|---
1 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0
2 | 0 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 0 | 1 | 0 | 0
3 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0
4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1

###  VI-2 SVM Classifier

An SVM classifier was implemented and evaluated by the generated datasets.
Both MITRE ATT&CK for Enterprise and MITRE ATT&CK for ICS were used to train
the classifier to be able to detect attacks in both IT and OT sections of an
ICS network. The SVM classifier could accurately predict the attacking groups
of ICS TTPs with an accuracy of 100%, and it showed 98% accuracy in detecting
IT TTPs. The advantage of this machine learning phase is that when the central
hunting phase detects a few TTPs, this SVM classifier predicts all APTs which
have these TTPs in common, and the hunter will receive the details of possible
future TTPs.

##  VII Discussion

The proposed framework is of significant use for the ICS due to its novel
approach to threat hunting. The solution could be automated in ICS
organisations which in turn minimises human errors, time and overall costs in
the threat hunting process. Further, the framework can provide detailed
information regarding the attacks based on the MITRE ATT&CK framework, thereby
making it useful for ICS industries to interpret the attacks in an extensive
manner. This type of application is also rarely found in the industry of ICS.
Using the dataset in Table I, an SVM classifier was used to analyse the TTPs
detected in the network and identify the APT groups that used the same TTPs.
The SVM classifier can accurately predict the attacking groups in both IT and
OT Matrix. Since the high accuracy was confirmed, it was implemented using
Python and it posts the predicted groups and detected TTPs to Elastic-Search.
The index is requested by a server-side application developed using the
Express JS framework. An additional single page application was developed
using React JS, which retrieved this data from the server to display the
detected TTPs and predicted group. As shown in Fig. 5, a bar chart is provided
for the users to see the adversaries detected in their systems. The chart can
display multiple groups detected when specific TTPs are given as input. It
also allows users to find out more about the groups detected as it will take
them to the group’s details on the MITRE framework website. In Fig. 5, react
application predicts multiple APT attacks in our networks using the two TTPs
detected by the central threat hunting phase.

Figure 5: Predicting APTs based on TTPs detected in our network.

##  VIII Conclusion and Future Work

Threat hunting aims to detect threat actors early in the cyber kill chain by
searching for signs of intrusions and then providing hunting strategies for
future use. In response to concerns about hidden intrusions, several cloud-
based hunting platforms have been proposed for IT networks that facilitate
monitoring of distributed devices and help detect ATT&CK TTPs associated with
detected anomalies. In this paper, we have presented a comprehensive approach
that can be used by industrial companies to implement central and automated
threat hunting in ICS networks. Our proposed open-source solution uses MITRE
ATT&CK for ICS to detect TTPs and then, automatically find the APTs associated
with the input TTPs. Our method can potentially detect malicious activities
more quickly and accurately. The machine learning-based analyser in our
solution can predict the future steps of the identified TTPs. It is also used
to automate decision making tasks e.g., the verification of the hypothesis.
Note, this paper studied an automated threat hunting process in ICS networks
as a proof of concept. In the future, we intend to explore how graph analysis
can help our framework to improve threat hunting.

## Acknowledgment

The authors acknowledge the support of the Commonwealth of Australia and Cyber
Security Research Centre Limited.

## References

  * [1] Z. Jadidi, A. Dorri, R. Jurdak, and C. Fidge, “Securing manufacturing using blockchain,” in _the 19th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom)_. IEEE, pp. 1920–1925, 2020.
  * [2] Z. Jadidi, E. Foo, M. Hussain, and C. Fidge, “Automated detection-in-depth in industrial control systems,” _The International Journal of Advanced Manufacturing Technology_ , Springer-Verlag London, 118(7), pp. 2467-2479, 2022.
  * [3] M. Assante and R. Lee, “The industrial control system cyber kill chain,” _SANS Institute InfoSec Reading Room_ , vol. 1, 2015.
  * [4] M. Fuchs and J. Lemon, “Sans 2019 threat hunting survey: The differing needs of new and experienced hunters,” _SANS Institute Information Reading Room_ , 2019.
  * [5] R. Lee and R. Lee, “Sans 2018 threat hunting survey results,” _SANS Institute Reading Room_ , 2018.
  * [6] A. Bhardwaj and S. Goundar, “A framework for effective threat hunting,” _Network Security_ , vol. 2019, no. 6, pp. 15–19, 2019.
  * [7] Enterprise Matrix (Matrix for Enterprise). [Online]. Available: https://attack.mitre.org/matrices/enterprise/
  * [8] Att&ck for industrial control systems. [Online]. Available: https://collaborate.mitre.org/attackics/index.php/Main_Page
  * [9] D. Gunter, “A practical model for conducting cyber threat hunting,” 2018. [Online]. Available: https://www.sans.org/white-papers/38710/
  * [10] Z. Jadidi and Y. Lu, “A threat hunting framework for industrial control systems,” _IEEE Access_ , vol. 9, pp. 164118–164130, 2021.
  * [11] H. HaddadPajouh, A. Dehghantanha, R. Parizi, M. Aledhari, and H. Karimipour, “A survey on internet of things security: Requirements, challenges, and solutions,” _Internet of Things_ , vol. 14, p. 100–129, 2021.
  * [12] A. Ajmal, M. Alam, A. Khaliq, S. Khan, Z. Qadir, and M. Mahmud, “Last line of defense: Reliability through inducing cyber threat hunting with deception in scada networks,” _IEEE Access_ , vol. 9, pp. 126789–126800, 2021.
  * [13] A. Khalid, A. Zainal, M. Maarof, and F. Ghaleb, “Advanced persistent threat detection: A survey,” in _the 3rd International Cyber Resilience Conference (CRC)_ , IEEE, pp. 1–6, 2021.
  * [14] S. Pal, M. Hitchens, V. Varadharajan, and T. Rabehaja, “Policy-based access control for constrained healthcare resources,” in _International Conference WoWMoM_ , IEEE, pp. 588–599, 2018.
  * [15] J. Rubio, C. Alcaraz, R. Roman, and J. Lopez, “Current cyber-defense trends in industrial control systems,” _Comp. & Security_, vol. 87, 2019.
  * [16] A. Bhardwaj and S. Goundar, “A framework for effective threat hunting,” _Network Security_ , vol. 2019, no. 6, pp. 15–19, 2019.
  * [17] A. Raju, I. Abualhaol, R. Giagone, Y. Zhou, and S. Huang, “A survey on cross-architectural iot malware threat hunting,” _IEEE Access_ , vol. 9, pp. 91686–91709, 2021.
  * [18] H. Darabian, A. Dehghantanha, S. Hashemi, M. Taheri, A. Azmoodeh, S. Homayoun, K. Choo, and R. Parizi, “A multiview learning method for malware threat hunting: windows, iot and android as case studies,” _World Wide Web_ , vol. 23, no. 2, pp. 1241–1260, 2020.
  * [19] G. Tecuci, D. Marcu, S. Meckl, and M. Boicu, “Evidence-based detection of advanced persistent threats,” _Computing in Science & Engineering_, vol. 20, no. 6, pp. 54–65, 2018.
  * [20] T. Bodström and T. Hämäläinen, “A novel deep learning stack for apt detection,” _Applied Sciences_ , vol. 9, no. 6, 2019.
  * [21] I. Ghafir, M. Hammoudeh, V. Prenosil, L. Han, R. Hegarty, K. Rabie, and F. Aparicio-Navarro, “Detection of advanced persistent threat using machine-learning correlation analysis,” _Future Generation Computer Systems_ , vol. 89, pp. 349–359, 2018.
  * [22] Q. Zou, A. Singhal, X. Sun, and P. Liu, “Automatic recognition of advanced persistent threat tactics for enterprise security,” in _Proceedings of the 6th International Workshop on Security and Privacy Analytics_ , pp. 43–52, 2020.
  * [23] P. Sharma, S. Moon, D. Moon, and J. Park, “Dfa-ad: a distributed framework architecture for the detection of advanced persistent threats,” _Cluster Computing_ , vol. 20, no. 1, pp. 597–609, 2017.
  * [24] R. Al-Shaer, M. Ahmed, and E. Al-Shaer, “Statistical learning of apt ttp chains from mitre att&ck,” in _RSA Conference_ , 2018.
  * [25] K. Demertzis, L. Iliadis, and S. Spartalis, “A spiking one-class anomaly detection framework for cyber-security on industrial control systems,” in _International Conference on Engineering Applications of Neural Networks_ , Springer, pp. 122–134, 2017.
  * [26] X. Wang, Q. Liu, Z. Pan, and G. Pang, “APT attack detection algorithm based on spatio-temporal association analysis in industrial network,” _Journal of Ambient Intelligence and Humanized Comp._ , pp. 1–10, 2020.
  * [27] U. Noor, Z. Anwar, A. Malik, S. Khan, and S. Saleem, “A machine learning framework for investigating data breaches based on semantic analysis of adversary’s attack patterns in threat intelligence repositories,” _Future Generation Computer Systems_ , vol. 95, pp. 467–487, 2019.