# Semantic Ranking for Automated Adversarial Technique Annotation in Security
Text

Udesh Kumarasinghe  Purdue University, INUSA ,  Ahmed Lekssays  Qatar
Computing Research Institute, HBKUQatar ,  Husrev Taha Sencar  Qatar Computing
Research Institute, HBKUQatar ,  Sabri Boughorbel  Qatar Computing Research
Institute, HBKUQatar ,  Charitha Elvitigala  University of ColomboSri Lanka
and  Preslav Nakov  MBZUAIUAE

(2018)

###### Abstract.

We introduce a novel approach for mapping attack behaviors described in threat
analysis reports to entries in an adversarial techniques knowledge base. Our
method leverages a multi-stage ranking architecture to efficiently rank the
most related techniques based on their semantic relevance to the input text.
Each ranker in our pipeline uses a distinct design for text representation. To
enhance relevance modeling, we leverage pretrained language models, which we
fine-tune for the technique annotation task. While generic large language
models are not yet capable of fully addressing this challenge, we obtain very
promising results. We achieve a recall rate improvement of +35% compared to
the previous state-of-the-art results. We further create new public benchmark
datasets for training and validating methods in this domain, which we release
to the research community aiming to promote future research in this important
direction.

Threat intelligence, TTP annotation, text ranking, text attribution

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Security and privacy††ccs:
Information systems Retrieval models and ranking

##  1\. Introduction

Understanding threat behaviors is essential for developing effective cyber
defense capabilities. Acquiring this crucial knowledge involves analyzing a
vast array of artifacts in the aftermath of security incidents. Through such
analyses, experts reconstruct the sequence of events that occurred during the
incident, uncover the tactics, techniques, and procedures employed by threat
actors, and determine the extent and impact of the attacks on data and
systems. The insights and findings of these experts are typically documented
in the form of unstructured or semi-structured natural language text and
shared with the broader security community (Blogs, 2023; Securelist, 2023;
Broadcom, 2023; WithSecure, 2023). Other analysts, with access to resulting
threat intelligence through public or paid platforms, evaluate the information
to strengthen existing detection, mitigation, and remediation capabilities.

The acquisition of threat knowledge from security text has predominantly
relied on highly expert-driven and time-intensive processes. Recognizing the
importance of this challenge to security operations, several approaches have
been proposed to automatically extract threat behaviors from threat
intelligence reports. These efforts have focused on different objectives, such
as extracting indicators of compromise (IoCs) (Liao et al., 2016; Zhu and
Dumitras, 2018), characterizing adversary tactics and techniques (Husari et
al., 2017a; Legoy et al., 2020; Tsai et al., 2020; You et al., 2022; Alam et
al., 2023; Li et al., 2022a), identifying threat actions (Husari et al.,
2017b; Tsai et al., 2020; Gao et al., 2021b; Alam et al., 2022), and creating
cybersecurity knowledge graphs (Gao et al., 2021a, b; Ji et al., 2022; Ren et
al., 2022).

In this work, our focus lies in annotating threat behaviors described in a
threat intelligence report with reference to adversarial techniques described
in the MITRE’s ATT&CK knowledge base (Corporation, 2022). This framework
serves as a widely adopted resource for analyzing and understanding the
tactics, techniques, and procedures (TTPs) deployed by adversarial threat
actors. It provides a comprehensive and standardized way of describing various
stages of an attack, from initial access and reconnaissance to execution and
post-exploitation. By aligning the findings of security reports with pertinent
tactics and techniques, security analysts and researchers can promptly
identify patterns and trends in in the way attacks are carried out.
Furthermore, as adversaries cannot change their techniques and procedures
rapidly, these associations aid in attributing attacks to specific threat
actors and in clustering reports into campaigns. Addressing these
requirements, several security vendors and organizations that generate and
assess threat intelligence reports also identify the adversarial techniques
employed during investigated attacks as part of their insights111These
encompass threat intelligence entities like WeLiveSecurity, TrendMicro, AT&T
Security Labs, MITRE, and CISA. A more comprehensive discussion of this data
is presented in Section 4..

Previously proposed solutions for the technique annotation task have so far
yielded limited practical impact, primarily due to two key factors. Firstly,
some of these approaches framed the task as a classification problem. A
significant limitation of this formulation lies in the lack of extensive
ground truth datasets crucial for effectively training multi-class classifiers
capable of distinguishing the correct technique among hundreds of distinct
adversarial techniques. Additionally, classification models tend not to excel
in scenarios involving human-in-the-loop, where the top-k techniques are
presented to the annotator for selection. The second factor pertains to the
constraint of conventional natural language processing (NLP) methods in
adeptly capturing the contextual nuances of mentioned entities and their
interrelationships within the text. The prevalence of domain-specific terms,
complex sentence structures (Satvat et al., 2021a), improper punctuation (Mu
et al., 2018), and the amalgamation of diverse data formats (including code,
tables, figures, and non-standard textual elements like IP/MAC addresses,
domain names, and file hashes) collectively undermines the interpretation of
reports and the extraction of attack behaviors. Moreover, the absence of
standard benchmark datasets led to the use of closed, custom datasets, thereby
making it difficult to compare these approaches on an equal footing.

To address these issues holistically, this study introduces a new problem
formulation that better aligns with the practical aspect of the problem. In
pursuit of this goal, we devise a multi-stage ranking pipeline (Matveeva et
al., 2006; Asadi and Lin, 2013; Clarke et al., 2016; Mackenzie et al., 2018)
that assesses the semantic relevance between passages of text within a report
and the corresponding reference text in the ATT&CK knowledge base, that
provides a data-driven understanding of adversary techniques. In addition, we
introduce a comprehensive public dataset including passages extracted from
incident reports, along with manually annotated technique IDs222For anonymity,
the link to the dataset is omitted..

###  1.1. MITRE ATT&CK Framework

The ATT&CK framework consists of a structured matrix that categorizes and
organizes the TTPs based on real-world
observations333https://attack.mitre.org/versions/v12/matrices/enterprise/. It
provides a comprehensive and systematic representation of various attack
vectors and serves as a reference guide for understanding and mitigating
potential threats. The matrix is divided into rows and columns, with columns
representing 14 distinct tactics that represent high-level goals or objectives
that adversaries aim to achieve, such as ”Initial Access,” ”Execution,” and
”Exfiltration.” Each tactic is further populated with specific techniques that
adversaries employ to execute those tactics, with the associated techniques
listed in rows beneath each column. Techniques offer detailed insights into
how attackers maneuver through an organization’s defenses. For further
granularity, some techniques are further broken down into ”subtechniques” that
provide a deeper understanding of the specific variations and nuances of an
approach. Additionally, ”procedures” refer to the step-by-step sequences that
adversaries follow to execute their techniques. This structured and
standardized vocabulary equips cybersecurity professionals with a shared
framework to effectively analyze, respond to, and defend against cyber
threats. As an evolving framework, ATT&CK is currently in its 13th version,
adapting to changes in the threat landscape.

The ATT&CK framework encompasses a collection of 193 techniques and 401
subtechniques, each meticulously documented and categorized. Each technique is
assigned a unique ID and a descriptive title. Technique descriptions can also
encompass potential detection and mitigation strategies, along with the
required data sources for identifying such behaviors. Furthermore, some
technique descriptions include concise summaries of sample procedures with
reference to their source threat analysis reports. These procedures are often
detailed accounts of actions carried out by threat actors. Our aim is to map
these procedural descriptions to overarching technique descriptions. Table 1
presents statistics on the number of reports used in the extraction of threat
behaviors underpinning the knowledge base.

It’s important to highlight that not all techniques hold equal likelihood of
being mentioned in a report. Reports are essentially generated based on the
analysis of attack artifacts. Techniques without traces, such as those within
the ”Reconnaissance” or ”Resource Development” tactics, or those that evade
detection due to the absence of necessary data sources, may not be
highlighted. In contrast, techniques associated with ”Initial Access” are more
commonly found in these reports. This is because, during this stage, attackers
have limited control over the compromised systems, making it difficult to
conceal their traces.

Table 1. ATT&CK Matrix Statistics Resource Type | Top-level | Sub-level | Total
---|---|---|---
Techniques | 193 | 401 | 594
Techniques w/o references | 38 | 71 | 109
Referenced reports | 700 | 607 | 1307

###  1.2. About Our Approach

Our approach employs a multi-stage ranking architecture that strategically
integrates diverse ranking models, optimizing the trade-off between
effectiveness and efficiency. In this multi-stage ranking framework, the
candidates singled out by each stage become inputs for the subsequent ranking
stages. Following the initial ranking, each stage receives a ranked list and
refines it further by retaining only the most highly ranked candidates. The
ranked list generated by the last stage of the multi-stage ranking
architecture is the output of the system. Our system takes a query text and a
text corpus sourced from the ATT&CK knowledge base as its input. The use case
for our system is depicted in Fig. 5.

(a)

(b)

Figure 1. (a) The practical application of our proposed system in annotating
threat reports, (b) Examples of report
sentences555https://research.checkpoint.com/2020/bandook-signed-delivered with
corresponding predicted techniques.

The design of our rankers leverages the capabilities of transformer-based
pretrained language models to accurately evaluate the semantic relevance
between the query and the text items in our corpus. This approach aligns with
methods used in web search, as described in (Zou et al., 2021; Yan et al.,
2021; Lin et al., 2022; Huang et al., 2020). To optimize these ranking models
and verify our findings, we utilized a newly generated dataset for fine-tuning
and validation purposes. Additionally, in our evaluation, we examined the
performance of several large language models in the zero-shot learning
scenario. In summary, our contributions in this work include the following key
aspects:

  * •

We present a novel learning-to-rank approach designed for the annotation of
threat behaviors outlined in threat intelligence reports, aiming to identify
the most semantically relevant techniques within the ATT&CK framework.

  * •

We constructed a new dataset of threat behaviors, enriched with human-
annotated technique ID labels. This dataset is expected to facilitate the
advancement of methods in this domain.

  * •

We provide a comprehensive benchmark that assesses the effectiveness of both
open and closed large language models for the technique annotation task.

  * •

We report new state-of-the-art performance results, indicating significant
improvements over both previous technique annotation methods and the
utilization of large language models for this task.

##  2\. Problem Formulation and System Overview

###  2.1. Formalizing Text Annotation

Given a text excerpt extracted from a threat intelligence report, the goal of
the text annotation task is to generate an ordered list of MITRE ATT&CK
Technique ID’s, denoted as $T_{i}$’s, based on their relevance to the query
text. To achieve this, we adopt a text ranking formulation that leverages the
semantic similarity between technique descriptions and the query. Formally, we
represent the set of technique IDs as $\mathcal{T}$ where
$\mathcal{T}=\\{T_{1},T_{2},\ldots,T_{n}\\}$, and $n$ is the number of
distinct adversarial techniques we aim to identify. The descriptions of all
techniques are contained in a large corpus
$\mathcal{C}=\\{it_{1},it_{2},\ldots,it_{m}\\}$, where each $it_{i}$ is a text
item associated with a technique $T$. The association between the technique
IDs and their descriptions is expressed through a function
$f:\mathcal{T}\rightarrow\mathcal{P}(\mathcal{C})$, where $\mathcal{P}()$
denotes the power set, representing the set of all possible subsets of an
input set. In other words, the function $f$ maps each technique ID in the set
$\mathcal{T}$ to a subset of text items in the set $\mathcal{C}$, i.e.,
$\forall T\in\mathcal{T},f(T)\in\mathcal{C}$ . The text items associated with
a particular technique, $f(T)$, consist of information collected from the
ATT&CK knowledge base about each technique. This information includes the
technique’s title, description, and the necessary measures for its detection
and mitigation.

Formally, our task can be expressed as follows: given a corpus $\mathcal{C}$
and a query $q$, we aim to generate a ranked list of the top-$k$ technique IDs
from $\mathcal{T}$ that maximizes a specific similarity metric, where $k$
represents the length of the ranked list. The query $q$ may consist of a
phrase or a passage, containing several sentences, extracted from a threat
intelligence report. The ranking model sorts the text items by calculating the
probability of each text item entailing the query within its description. To
achieve this, we employ pretrained transformer-based models, leveraging the
powerful language understanding capability of language models (LMs).
Specifically, our objective is to learn and rank dense representations of
texts by estimating the probability:

(1) $P(related=1|it_{i},q)\triangleq\phi(\eta(it_{i}),\eta(q))\text{ where
}it_{i}\in\mathcal{C}.\\\ $

This probability indicates the relevance of a text item with respect to a
given query. In this formulation, $\eta()$ is a transformer-based encoder that
maps a sequence of text tokens into a dense vector representation, potentially
spanning hundreds of dimensions, with each dimension holding non-zero values.
The function $\phi()$ is a symmetric measure used to evaluate the relevance
between the encoded text items and the query. Alternatively, the probability
in Eq. (1) can be estimated within an interaction-based model using the same
learning architecture by jointly encoding the text and the query as:

(2) $P(related=1|it_{i},q)\triangleq\phi(\eta(it_{i},q))\text{ where
}it_{i}\in\mathcal{C}.\\\ $

In this context, $\phi$ can also be interpreted as a classifier, with its
decision confidence serving as a ranking score. It must be noted that,
multiple text items are associated with each technique. Therefore, the
probability of a particular technique ID is determined by taking the maximum
over all related text items in Eqs. (1) and (2) as follows:

(3) $P(T_{i})=\max_{it}P(related=1|it,q)\text{ such that }it\in f(T_{i}),\\\ $

where $f(T_{i})$ represents the set of text items associated with the
technique.

###  2.2. System Overview

Our technique employs a multi-stage ranking architecture, which integrates
multiple ranking models to achieve an optimal balance between effectiveness
and efficiency. In the multi-stage ranking approach, candidates identified by
each stage become inputs for subsequent ranking stages. As depicted in Figure
(2) the supervision level increases as we progress in the ranking pipeline.
Hence we prevent over-fitting and ensure efficiency in the early stages. Stage
1 is fully unsupervised and allows for a fast filtering of candidate
techniques using BM25. Stage 2 injects supervision via representation learning
of the embedding space. This is achieved by fine-tuned SentSecBert. Stage 3
formulates the ranking as a supervised problem where relevance of a candidate
text to a query is assessed via a classifier (Fined-tuned MonoT5). The softmax
output of the classifier is used a ranking metric. Each stage receives a
ranked list and produces a shorter ranked list by retaining only highly ranked
candidates. That is, Eq. (3) is independently computed by each ranker on an
iteratively reduced corpus $\mathcal{C}$. The final output of the multi-stage
ranking architecture is generated by the ranked list from the last stage.

Figure (2) provides an illustration of the components in the ranking pipeline
of our system. Before performing the ranking, both the text in the corpus and
the query undergo preprocessing to remove less informative text elements, such
as indicators of compromise. These indicators include _file paths_ , _domains_
, _URLs_ , _email addresses_ , _IP addresses_ , _ASNs_ , _CVEs_ , _MITRE ATT
&CK techniques_, _hashes_ , _registry keys_ and _Bitcoin addresses_. In the
first stage, ranking is performed based on a bag-of-words representation,
evaluating the exact match of keywords in the query with the text items in the
corpus. To accomplish this, we create an inverted index and generate term
frequency weighted document vectors. Conversely, the following stages utilize
and compare dense representations obtained through fine-tuning pre-trained
LMs. The second stage employs a bi-encoder architecture following the
formulation of Eq. (1), while the subsequent stage employs the computationally
more intensive mono-encoder, Eq. (2), that allows interaction between the
query and the candidate text items to finalize the rankings.

Figure 2. System Architecture of the Proposed Method ($K_{i}$ represents the
number of candidate technique IDs at the output of the $i$-th stage, with
$K_{1}>K_{2}>K_{3}$).

Diagram of the high-level system components.

##  3\. Ranking Pipeline

###  3.1. Query Preprocessing

Threat behaviors offer detailed descriptions of how threat actors implement
various techniques during cyberattacks. Specifically, they identify
interactions among system entities (like files, processes, network sockets),
communication patterns, network endpoints, code excerpts, commands, and the
names of tools used in the attack. The terminology employed in these
descriptions is domain-specific, which substantially differs from the natural
text commonly used to train language models. This discrepancy poses challenges
when processing threat intelligence reports using language models. A notable
difficulty arises from the tokenization process applied by these models,
wherein the text input is broken down into smaller pieces based on a
predefined vocabulary that considers the occurrence frequency of text and the
presence of common delimiters. These tokenizers also split the commonly
appearing and widely varying indicators of compromise (e.g., IP addresses, MAC
addresses, file hashes, URLs, filenames, domain names, ASNs) into less
informative pieces, making it harder for the model to properly contextualize
such information. This becomes a primary concern when associating the query
text extracted from threat intelligence reports with technique descriptions in
the ATT&CK knowledge base, as the technique descriptions are quite general. To
address this issue, we exploit the fact that many IoCs have a well-defined
form and employ regular expression-based rules to identify and substitute
these indicators of compromise with generic placeholder terms, such as ”ip
address”, ”hash”, ”URL” etc., in our approach. For this, we leverage
iocparser666https://iocparser.com tool to identify all IoCs.

###  3.2. Stage-1: Exact Term Matching

The initial ranking stage of our ranking pipeline aims to enhance the
efficiency and accuracy of subsequent ranking stages by eliminating some of
the unrelated techniques. This is particularly crucial when considering the
large number of techniques and (sub-)techniques, i.e., $|\mathcal{T}|$. To
achieve this, we adopt a traditional information retrieval (IR) approach to
retrieve the most relevant text item in our corpus based on exact term
matching. That is, terms from each
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}it}\in\mathcal{C}$
and terms from queries had to match exactly to contribute to a relevance
score. This matching is typically performed using a normalized version of the
terms obtained by stripping off inflectional parts (prefixes and suffixes) of
the words.

To achieve this goal, we utilize the widely-adopted BM25 (Robertson et al.,
1995; Crestani et al., 1998) method, which has been a prevalent token-based
matching algorithm to date (Tang et al., 2019; Lee et al., 2019; Yates et al.,
2021). The BM25 algorithm employs a high-dimensional sparse-vector
representation that calculates the weight of each term based on its term
frequency and inverse document frequency. The BM25 relevance score between
each $ti\in\mathcal{C}$ and the query $q$ is determined through inner products
of their corresponding vectors, and the retrieval can be performed efficiently
using an inverted index. In our approach, we utilize the resulting score
values to retain the top 100 most relevant techniques, as the primary
objective of the first stage is to filter out potentially unrelated
techniques.

###  3.3. Stage-2: Transformer-Based Bi-Encoders for Semantic Matching

In this stage, we exploit the language understanding capability of
transformer-based deep neural networks. It has been demonstrated that both
small and large language models777 While there is no formal definition for
what constitutes a small and large language model, we classify pretrained LMs
with parameters up to several hundreds of millions, such as BERT (Devlin et
al., 2018), RoBERTa (Liu et al., 2019), and ERNIE (Sun et al., 2019), as small
LMs. Conversely, models with more than billions of parameters, like closed GPT
family models (Brown et al., 2020) and open models such as various fine-tuned
variants of LLAMA (Touvron et al., 2023), are categorized as large LMs. can
very effectively capture complex syntactical and semantic structures as well
as how meanings vary across different linguistic contexts. To retrieve the
techniques that are semantically most relevant to a query, we use a small
language model due to ease in their fine-tuning and deployment. (In Sec. 5.4,
we also compare our ranking system against well-known large LMs under a zero-
shot test setting.)

Our ranker follows a bi-encoder architecture, where we independently compute
dense-vector representations of text items, $\eta(it_{i})$, and the query,
$\eta(q)$, as formulated in Eq. (1). The relevance matching scores,
$\phi(\eta(it_{i}),\eta(q))$, are computed in the resulting representation
space by evaluating the cosine similarity between the embedded vectors. For
our encoding function, we adopt the architecture and hyper-parameters of the
12-layer BERT-base model. The BERT model is pretrained on a large corpus of
English sentences using a masked language modeling task (MLM), where the
objective is to predict the masked tokens in the input (Devlin et al., 2018).
BERT has a maximum input length constraint of 512 (subword) tokens,
approximately equivalent to 400 words, represented as 768-dimensional vectors.
The model’s output includes the same number of tokens, along with a special
token, denoted as [CLS], that aggregates the information in the entire input
sequence.

Developing a ranker with BERT requires performing two steps. Firstly, the
amount of security-related text in the data used to pretrain BERT is likely
limited. Further, due to the distinctive characteristics of security text
compared to other natural language (Satvat et al., 2021a), the limited
exposure to security domain knowledge hinders the model’s ability to properly
contextualize security concepts. Secondly, the ranker must undergo appropriate
fine-tuning to accurately calculate semantic relevance scores needed to
perform the technique ID annotation task.

Adapting BERT to the cybersecurity domain requires additional pretraining on
threat intelligence text. Several attempts have been made to achieve this,
including CyBERT (Ranade et al., 2021),
SecBERT888https://github.com/jackaduma/SecBERT, and Attack-
BERT999https://huggingface.co/basel/ATTACK-BERT. Whereas the fine-tuning
process involves exposing the further-pretrained model to task-specific
examples to align the model’s output with the correct notion of semantic
relatedness. During fine-tuning, the two encoders, Eq. (1), are combined by
computing the inner product of their outputs. This bi-encoder backbone is then
trained in an end-to-end manner using a task-specific dataset to maximize the
cosine similarity computed between the outputs of the two encoders,
specifically the CLS tokens.

In our evaluation, we found that SecBERT outperformed other models for our
specific task, so we chose to fine-tune it. During the fine-tuning process,
the model is presented with an equal number of positive and negative pairs of
text items and queries. For positive pairs, the cosine similarity is set to
one to indicate that the two texts are related, while for negative pairs, it
is set to negative one to indicate a lack of relevance. This setup essentially
turns the model into a binary classifier for decision-making. To generate the
necessary data for fine-tuning, we utilized the samples from our custom
dataset (Sec. 4), which includes behaviors extracted from threat intelligence
reports along with corresponding technique ID annotations. Since our data only
includes queries, we used the title of the technique ID along with the general
description (to fill 512 tokens) to create positive pairs, while for negative
pairs, we included descriptions of non-matching technique IDs. For this we
used 70% all threat behavior descriptions in our dataset. To enhance the fine-
tuning data, we also leveraged a subset of datasets used to build the widely-
used sentence-transformer model that focuses on identifying sentences with
similar meanings (Reimers and Gurevych, 2019a). (These datasets are listed in
Appendix Sec. A, comprising close 2M pairs of text samples.) By the end of
fine-tuning, both encoders are updated, and they can be used to generate
vector representations. We refer to the fine-tuned version of the further-
pretrained BERT model as the SentSecBERT model. For a more detailed discussion
and comparison of the rankers considered for the second stage, please refer to
Sec. 5.6.

During inference, we utilize the cosine similarity between two text embeddings
as our score to rank the most relevant technique IDs to a given query. Our
corpus $\mathcal{C}$ at this stage includes text excerpts extracted from the
ATT&CK knowledge base. The text items associated with each technique ID
$T_{i}$, i.e., $f(T_{i})$, are obtained by segmenting the available text under
the the main headings, including main description, detections, mitigations,
and procedure examples, into 512 tokens in an overlapping manner with a stride
of 128 tokens. The vector representation for each text item is precomputed and
stored for fast matching.

###  3.4. Stage-3: Transformer-Based Mono-Encoder for Semantic Matching

At this stage, we employ a ranker that better leverages the attention
mechanism of a transformer. To achieve this, we utilize a single encoder that
processes both the query and a text item simultaneously. This design enables
interactions between query tokens and text item tokens, allowing the model to
capture deeper semantic relationships within the input sequence. In contrast,
a bi-encoder structure contextualizes each piece of text independently,
potentially limiting the model’s ability to fully exploit the connections
between the query and the text item. Thus, the model’s input sequence combines
the query $q$ and one of $it_{i}\in\mathcal{C}$ and its output designates
whether the two sub-sequences are related.

Essentially, this mono-encoder design treats text ranking as a binary
classification problem over all text items in the corpus, using a metric
indicating the model’s confidence as the ranking score. The advantage of this
interaction-based ranking model is balanced by the higher latency it incurs,
as the ranking computation must be performed at query time. However, since
most unlikely techniques have already been eliminated in the first and second
stages, it can be feasibly deployed on a smaller candidate pool. Further,
since this model does not need to explicitly learn a dense vector
representation, like performed by encoder-only transformer models, alternative
architectures based on encoder-decoder architectures used can also be utilized
for our purpose. These generative models that are mainly designed for
sequence-to-sequence to transformation shown to be quite effective in
performing several tasks that do not necessarily involve a sequence generation
(Raffel et al., 2020; Yang et al., 2020; Pradeep et al., 2020).

To fully leverage the language understanding capabilities, we developed our
ranker using the T5 model, a pretrained sequence-to-sequence transformer
(Raffel et al., 2020). Similar to the MLM task used for training BERT, T5’s
training objective involves generating a target text from a corrupted input
text by predicting the missing parts after removing certain segments. With a
larger number of parameters and training on a more extensive corpus, T5
provides enhanced language modeling capabilities compared to BERT (Nogueira et
al., 2020). Incorporating the T5 model as a ranker necessitates fine-tuning to
ensure that, given a specific input template, the model generates a response
suitable for use as a ranking score. This fine-tuning process is critical in
guiding the model to produce the desired output when presented with inputs
adhering to the predefined template.

To construct our ranker, we utilized the monoT5 model fine-tuned specifically
for document ranking (Nogueira et al., 2020). This fine-tuning was achieved
using the MS-MARCO passage data, consisting of 8.8 million extracts from web
pages in response to natural language questions from Bing search engine users
(Nguyen et al., 2016). We further fine-tuned monoT5 to adapt it to our
technique annotation task. For compatibility with the earlier fine-tuning
process, we adopted the same input template defined as follows:

"Query:[$q$] Document:[$it$] Relevant:{true,false}”.

Here, $q$ represents the query text, $it$ is a text item in $\mathcal{C}$, and
the relevance between the two is indicated by either true or false words. Both
of these words, true and false, are encoded as single tokens in T5’s
tokenization vocabulary. During model inference, the softmax function is
exclusively applied to the logits of these two tokens, allowing us to
determine their probabilities and make relevance assessments. The ranking
score of a text item in the corpus for a given query is then determined as the
probability of the true token.

Similar to BERT, the T5 model is also limited to an input length of 512
tokens. Consequently, the combined length of $q$ and $it$ must not exceed this
limit. To work within this constraint, we reserved 250 tokens for each query
and text item. This length proves to be more than sufficient for text excerpts
extracted from threat intelligence reports, as longer queries might include
multiple threat behavior descriptions. To ensure each $it\in\mathcal{C}$
adheres to this length restriction, we restructured our corpus $\mathcal{C}$
by segmenting the information associated with each technique in the knowledge
base into sequences of 250 tokens with a stride of 125 tokens. In the process
of fine-tuning monoT5, we employed a subset of the query samples from our
dataset (the same set utilized for constructing SentSecBERT) to generate an
equal number of positive and negative $(q,it)$ pairs. Each positive pair
consists of $q$ and the first generated text item corresponding to ground
truth technique ID, while a negative pair involves the same for a mismatching
technique ID.

##  4\. Dataset Creation

The rankers used in the final two stages of our system are essentially
language models fine-tuned to discern semantic relationships that are aligned
with our task’s objectives. This necessitates a substantial dataset of threat
behaviors, each accompanied by its corresponding ground-truth adversarial
technique IDs. We curate these threat behaviors from publicly available APT
attack reports, many of which include varying degrees of technique
annotations. For example, in some cases, analysts will infer the techniques
used in an APT attack, but without explicit references to specific sections of
the text101010Following are two examples of such reports:
https://cybersecurity.att.com/blogs/labs-research/crypto-miners-latest-
techniques
https://www.trendmicro.com/en_th/research/23/g/detecting-bpfdoor-backdoor-
variants-abusing-bpf-filters.html. Hence, our focus was on threat reports in
which analysts explicitly annotated relevant text or provided explanations for
their annotation choices. We identified four such sources of reports
information to compile the dataset for our task111111Our combined dataset is
publicly accessible at https://github.com/qcri/Text2TTP .

Table 2. Overview of Our Dataset Source | Techniques | Behaviors | Multi-Label
---|---|---|---
ATT&CK Reports | 92 | 2,017 | no
CISA | 313 | 1,718 | yes
TRAM | 137 | 1,506 | no
WeLiveSecurity | 259 | 1,474 | no
Combined Dataset | 410 | 6,612 | mixed

##### ATT&CK Reports:

Descriptions of techniques in ATT&CK knowledge base often encompass lists of
procedural examples that elucidate how specific APTs have employed that
technique. Each procedure is accompanied by a brief summary and a reference to
the source report. We used these summaries to manually pinpoint the
corresponding text sections within the original report. Our team cross-
referenced 1,180 summary procedure examples spanning 662 reports with the
respective text passages in the original reports. Each identification was
verified by another team member. When designation ambiguity arose, those
techniques were omitted from the dataset.

##### WeLiveSecurity:

WeLiveSecurity is a threat intelligence vendor that shares the APT reports
created by its analysts on its
website121212https://www.welivesecurity.com/en/. Each published APT report
includes a list of associated adversary techniques along with a detailed
summary of observed threat behaviors at its conclusion. While the reports lack
annotations at the passage level, the provided summaries contain an adequate
level of detail for our task. As a result, we extracted a total of 1,474
threat behaviors from 101 reports.

##### CISA:

The threat reports published by the Cybersecurity and Infrastructure Security
Agency (CISA)131313https://www.cisa.gov/ provide summarized insights into
observed APT attacks, offering a concise overview instead of exhaustive
details. Each report contains sentence-level annotations of technique IDs, and
in certain cases, some sentences carry multiple annotations at a sub-sentence
level. We noticed that in many instances, these succinct descriptions lack the
depth required for comprehensive technique representations. Consequently,
instead of segmenting these sentences into individual parts, we opted to
annotate the complete sentence with multiple technique IDs. This means that
these sentences are considered multi-labeled and are retrieved multiple times,
each for the corresponding adversarial technique. Our dataset comprises a
total of 1,718 annotations extracted from 65 reports.

##### TRAM Training Data:

TRAM, an open-source tool developed by the MITRE ATT&CK team, is designed to
enhance the efficiency of report annotation141414https://github.com/center-
for-threat-informed-defense/tram/. It employs multiple machine learning
models, including logistic regression, Naive Bayes, and multilayer perceptron,
to classify behaviors and assign them to corresponding technique IDs. To
facilitate the training of these models, the TRAM source also includes a
training dataset that includes an array of threat behavior descriptions of
varying lengths, combined with technique ID data. Our dataset has been
augmented with this training data, encompassing 1,506 behavior descriptions
across 137 techniques. It is important to note that while TRAM is an
annotation tool developed by the MITRE team, our dataset creation did not
incorporate TRAM’s annotations as supplementary data.

Table 2 presents an overview of our dataset, specifying the count of covered
adversarial techniques along with the quantity of threat behavior descriptions
and their corresponding sources. These descriptions of threat behaviors vary
in length, spanning from compact passages to succinct statements. The combined
dataset has 6,612 unique threat behavior descriptions aggregated from
individual datasets. Within this collection, we identified an overlap of 103
threat behaviors across different datasets. It must be noted that the coverage
of the number of distinct techniques varies across datasets. Specifically, the
CISA dataset features 232 out of 313 techniques that are also found in other
datasets. The WeLiveSecurity dataset has 220 techniques in common with other
datasets. There is an overlap of 123 techniques in the TRAM dataset with
others. Meanwhile, he ATT&CK Reports dataset shares 73 techniques with other
datasets.

Table 3. Performance of Proposed Multi-Stage Ranking Solution Stage | Recall@100 | Recall@50 | Recall@10 | Recall@5 | Recall@3 | Precision@1 | MRR
---|---|---|---|---|---|---|---
Stage-1 (BM25) | 0.9736 | 0.9414 | 0.8069 | 0.7211 | 0.6452 | 0.4702 | 0.5848
Stage-2 (SentSecBert) | - | 0.9595 | 0.8382 | 0.7359 | 0.6732 | 0.4529 | 0.5890
Stage-3 (MonoT5) | - | - | 0.9207 | 0.8671 | 0.8102 | 0.5783 | 0.7068

##  5\. Evaluation

###  5.1. Evaluation Setup

Our experiments were conducted within the Ubuntu 18.04.2 operating system
environment, utilizing an Intel Xeon Gold 6140 CPU with 72 cores, a Tesla
V100-SXM2 32GB graphics card, and 504 GB of memory.

###  5.2. End-to-End Evaluation

We assess the effectiveness of the proposed multi-stage ranking architecture.
In our tests, the corpus $\mathcal{C}$ included descriptive text associated
with 193 top-level techniques. While the threat behavior descriptions in our
dataset only spanned 168 techniques, our aim was to avoid overestimating our
system performance. Out of the 6,612 samples in our dataset, 4,749 were used
for fine-tuning Stage-2 and Stage-3 rankers and 1,212 samples related to 168
techniques were used during testing. It’s important to highlight that the
ATT&CK Reports dataset is generated through manual identification of the
original descriptions linked to sample procedure examples found in the
knowledge base within the relevant source reports. When testing samples from
this dataset, we excluded all text items containing the corresponding
procedure examples from our corpus to avoid a bias in our performance results.
(That is, no threat behavior extracted from a report was compared against its
summarized version in the knowledge base.)

The results obtained at different stages of our pipeline are presented in
Table 3, showcasing ranking performance across various metrics such as recall
at top $K$ candidates, precision for the best guess, and the mean reciprocal
rank (MRR). The retrieval performance of each ranker used in distinct stages
is shown in separate rows. For instance, when the initial stage ranker is
confined to 100 candidates and the Stage-2 ranker to 50 candidates, the
Stage-3 ranker achieves a remarkable recall@10 performance of 92% and recall@3
performance of 81% in accurately identifying the correct technique ID within
its top candidates among more than 193 techniques in the ATT&CK knowledge
base. It’s important to note that the recall@100 values for Stage-2 and both
recall@100 and recall@50 values for Stage-3 are not displayed, since the
number of candidate techniques has been diminished by the preceding rankers.
Figure 3 demonstrates the trade-off between annotation accuracy and time
complexity in our multi-stage ranker, showing the impact of adjusting the
cutoff threshold in the Stage-1 ranker on recall performance and matching
latency in the Stage-2 ranker.

Figure 3. The effect of raising the cutoff threshold from 50 to 150 in the
Stage-1 ranker on the Stage-2 ranker’s performance is depicted. The blue curve
represents the decrease in recall performance, while the yellow curve
illustrates the increase in computational time at the Stage-2 ranker for
processing each query behavior.

These findings also illustrate that each successive stage enhances the
system’s proficiency in associating accurate MITRE ATT&CK techniques with the
queries. To gauge the individual contribution of each ranker to the overall
performance, we can assess the values in the Recall@10 and Recall@3 columns.
We observe that BM25 scoring, utilizing a bag-of-words representation,
achieves an independent recall performance of 80% and 64%, respectively. When
combined with a bi-encoder-based semantic retriever in the two-stage ranker,
performance further improves by over 3%. Incorporating the mono-encoder-based
semantic retriever further boosts the ranking performance by 9% for Recall@10
and 4% for Recall@3.

###  5.3. Comparison to Previous Studies

To better assess our approach’s effectiveness, we compare our system’s
performance against three other state-of-the-art methods designed for
annotating threat reports with adversarial technique IDs. Among these, TRAM
(tra, 2023) employs a multi-class classification approach with classification
confidence serving as the ranking score, while Ladder (Alam et al., 2023) and
AttacKG (Li et al., 2022b) utilize sentence embeddings to measure similarity.
The latter two techniques are the most recent approaches to technique
annotation task, and they are most similar to our technique in their approach
to text representation. (Further details of these techniques are discussed in
Sec. 7). For our evaluation, we leveraged the publicly available source codes
of these techniques. Since TRAM supports several classification models, we
built it from its source while incorporating all classification models. Our
findings revealed that the logistic regression classifier outperforms other
models, hence we adopted it for our evaluation.

The tests were conducted on the same 1,212 threat behaviors used for the
results of tests presented in Table 3. The comparison results are presented in
Table 4. These findings illustrate that our multi-stage ranking approach
significantly outperforms the other methods by a considerable margin (+30%)
across all ranking configurations. For the best prediction scenario
(Precision@1), our technique achieves an accuracy of 57%, while the second-
best performing method, TRAM, reaches 32%. In terms of F1-Macro and
F1-weighted metrics, LADDER outperforms both TRAM and AttacKG but lags behind
our method by a significant margin of approximately 15

Table 4. Comparison to Related Work Method | Recall@10 | Recall@5 | Recall@3 | MRR | Precision@1 | F1-Macro | F1-Weighted
---|---|---|---|---|---|---|---
TRAM (tra, 2023) | 0.4582 | 0.4308 | 0.4028 | 0.3769 | 0.3204 | 0.1179 | 0.2357
Ladder (Alam et al., 2023) | 0.6342 | 0.5198 | 0.4435 | 0.3985 | 0.2848 | 0.2256 | 0.2779
AttacKG (Li et al., 2022b) | 0.2454 | 0.1731 | 0.1261 | 0.1254 | 0.0593 | 0.0059 | 0.0258
Ours | $\mathbf{0.9207}$ | $\mathbf{0.8671}$ | $\mathbf{0.8102}$ | $\mathbf{0.7068}$ | $\mathbf{0.5784}$ | $\mathbf{0.3664}$ | $\mathbf{0.4239}$

###  5.4. Comparison to LLMs

In this section, we explore how LLMs can accomplish treat report annotation
task. Subsequently, we compare the effectiveness and execution time of our
solution with those of large language models.

####  5.4.1. Task description.

In this experiment, we prompt LLMs to identify the top-3 most likely MITRE
ATT&CK Technique IDs that an input threat behavior describes.

####  5.4.2. Configuration

LLMs exhibit high sensitivity to both the formulation and structure of
prompts, as well as the parameters employed. Hence, it holds significant
importance to craft precise prompts that align with the specific task. In our
scenario, the identification of technique IDs from descriptions hinges heavily
on the LLM’s underlying training data.

##### Prompt Engineering:

LLMs are trained with specific predefined prompt structures, unique to each
model. While common templates can be shared among some LLMs, optimal results
are achieved by using the exact prompt template used during the LLM’s
training. These templates consist of four components: Instruction (task for
the LLM), Context (external information aiding the LLM), Input Data (the
query), and Output Indicator (desired output format). In our case, we employ
the relevant template for each LLM used.

In addition, we utilize various effective prompting techniques to enhance LLM
performance: Role Play, Specificity and Precision, and ReAct Prompting, and
Few-shot Prompting. Role Play prompts elicit improved performance from LLMs
when they adopt specific roles aligned with the task. In our case, each prompt
is introduced with the phrase Act as an experienced security analyst, [...],
assigning a distinct role to guide the LLM’s response. Specificity and
Precision in prompts indicate that LLMs achieve improved results when the
prompts are precise and well-formatted. In our scenario, we instruct the LLM
to provide answers in a well-defined JSON format. By specifying the format and
appropriate parameters, we avoid hallucinations and ensure that the generated
text remains relevant to the assigned task. ReAct (Yao et al., 2022) Prompting
is designed to compel LLMs to generate reasoning traces and task-specific
actions. We employ it to produce reasoning traces for the answers it generates
by asking for an explanation of why the LLM chose specific technique IDs for a
given sentence. This step also helps in reducing hallucinations and improving
the quality of results. We demonstrate the application of these prompting
techniques in our prompt used in the zero-shot learning setting, as shown in
Fig. 6 in Appendix Sec. B.

Few-shot Prompting: This is a prompting technique that helps LLMs in
generating coherent and contextually appropriate responses with only a limited
amount of input examples. These examples act as a context for the model to
infer the underlying pattern or classification rule. Given that our study
involves over 100 technique categories, we limited our consideration to a
maximum of 20 examples. Our selection of examples for the few-shot approach
was guided by the data distribution. Specifically, we chose 40% of the
examples from techniques with more than 30 instances, another 40% from
techniques having 6 to 30 instances, and the remaining 20% from techniques
with 1 to 5 instances. For this purpose, we select two techniques that are
semantically most similar to the correct one, treating them as potential
correct choices. Then, we take the description of the correct technique and
integrate it into the explanation. We show a sample prompt with two examples
in Fig. 7.

##### Parameters:

LLMs offer several parameters that can be tuned to optimize their performance
for specific tasks, mainly temperature, top-k, top-p, and maximum new tokens.
The temperature parameter controls the randomness of token generation during
text generation. It affects how the model selects the next token based on the
predicted probabilities of the possible tokens. The top-k parameter instructs
the model to select the next token from the top ’k’ tokens in its sorted list
based on their probabilities. The top-p parameter of a language model (LLM)
controls the dynamic sampling of tokens during text generation. When using
top-p, the model samples tokens among the candidates with the cumulative
probability surpassing the specified threshold $p$. The maximum new tokens
parameter limits the length of the generated output. In our experiment, we set
the temperature to 0.7, top-p to 0.60, top-k to 40, and maximum new tokens to
200.

####  5.4.3. Models

In this work, we used a combination of open models from
HuggingFace151515https://huggingface.co and closed source models from
OpenAI161616https://openai.com/gpt-4. For the open models, we selected those
that could run on a single NVIDIA A100-80GB GPU. Our model selection criterion
was guided by their performance on the Chatbot Arena
Leaderboard171717https://huggingface.co/spaces/lmsys/chatbot-arena-
leaderboard, as further detailed in (Zheng et al., 2023). We selected the top-
performing model in each size category, specifically those with 7B, 13B, and
30B parameters. Table 5 displays the list of employed language models in our
study.

Table 5. Employed Large Language Models Model Name | HuggingFace Path
---|---
Vicuna-33B (Zheng et al., 2023) | lmsys/vicuna-33b-v1.3
WizardLM-13B (Xu et al., 2023) | WizardLM/WizardLM-13B-V1.2
Zephyr-7B (Tunstall et al., 2023) | HuggingFaceH4/zephyr-7b-beta
gpt-4-32k | -

####  5.4.4. Results

Table 6. Comparison to LLMs Model Name | recall@3 | Avg. Time per Sentence (s)
---|---|---
Vicuna-33B | 0.0158 | 13.28
WizardLM-13B | 0.1198 | 15.73
Zephyr-7B | 0.1081 | 11.28
gpt-4-32k | 0.4406 | 9.66
Ours | 0.8102 | 8.90

We conducted queries using the same set of 1,212 test threat behavior
descriptions across the LLMs.

Zero-Shot Learning Performance. The corresponding recall@3 results for
correctly identifying the technique ID among the three identified ones are
presented in Table 6. These findings indicate that open LLMs exhibit limited
performance in this task, with recall@3 rates remaining below 4% for all of
them. However, GPT-4 shows relatively better performance, achieving a recall@3
score of 44.06%. The low recall@3 achieved by open models is primarily a
result of generating invalid technique IDs that are not found in the MITRE
ATT&CK knowledge base.

Few-Shot Learning Performance. Figure 4 illustrates the impact of adding
different numbers of examples, each from a distinct technique, to our input
prompt on the recall@3 score. We observed that the few-shot learning setting
does not enhance the performance of the open models, with the exception of a
minor improvement in the Vicuna-33B model. For the gpt4-32k model, the
outcomes were nearly the same for 0 and 5 examples. Yet, there was a
noticeable 5% increase with the addition of 10 examples. Subsequently, the
results remained steady, showing a slight decline of 1.2% with 15 examples and
a marginal rise of 0.87% for 20 examples. Remarkably, with the open LLMs, the
recall@3 score decreases to 0 for all models after incorporating 15 examples.
Our investigation into this matter revealed that the LLMs tend to produce
hallucinations, generating incomplete and arbitrary sentences drawn from the
MITRE ATT&CK knowledge base or techniques provided as part of the examples.
Furthermore, they fail to comply with the given directives regarding
specificity and precision These results underscore the importance of fine-
tuning these models on datasets related to technique annotation task to
improve their retrieval accuracy, as we have done in our approach.

Figure 4. Impact of increasing the number of few-shot learning examples on the
measured Recall@3 score.

Inference time. Despite the added network latency, our measurements reveal
that the GPT-4 model, presumed to possess the largest number of parameters,
still manages a fairly prompt response time (9.66 seconds per query), which is
comparable to our technique (8.9 seconds per query).

Cost analysis. Running LLMs is a costly process. To rent an NVIDIA A100-80GB
GPU, the cost in the market is around
$6.50181818https://huggingface.co/pricing#endpoints per hour. On average, the
experiments we run (i.e., with and without few-shot examples for all LLMs),
lasted 8 hours. Thus, the cost of a single experiment is approximated to be
$48. In other words, on average the cost of a sentence query is approximately
$0.04. For GPT4, the experiments (with and without few-shot examples) cost
$749.99 in total. On average, the cost of a sentence query is approximately
$0.4. In the 0-shot learning experiment, the average cost per sentence is
$0.006, compared to $0.1374 per sentence in the 20-shot experiment.

###  5.5. Error Analysis

In our ranking pipeline, we’ve noticed that certain techniques are more
susceptible to errors. This can be traced back to two main reasons. Firstly,
the number of samples available for fine-tuning our Stage-2 and Stage-3
rankers for these specific techniques is limited. Consequently, our rankers
have reduced exposure to these techniques, impairing their ability to
accurately identify them when mentioned in reports. This limited exposure is
often a result of the relatively rare application of these techniques. For
example, the technique ‘T1218 - System Binary Proxy Execution’ is referenced
in just one report within the ATT&CK knowledge base. Secondly, there is a
negative correlation between the volume of information in our corpus about a
technique and the likelihood of erroneous predictions. That is, the more
comprehensively a technique is described in the knowledge base, the better our
system is at distinguishing it from others. For instance, the technique ‘T1071
- Application Layer Protocol’ has a very brief description in the knowledge
base, leading to incorrect identifications by our system. Figure 7 displays
the relation between recall misses and the number of samples employed during
fine-tuning, as well as the volume of information available for each technique
in our corpus. Each point in the figure represents a distinct technique In
Fig. 5(a), technique ’T1071’ appears as an outlier in the plot of the
technique’s recall in the top 3 versus the number of training samples.
However, Fig. 5(b) explains this anomaly, indicating that technique ’T1071’ is
described very briefly in the ATT&CK knowledge base.

(a)

(b)

Figure 5. (a) Recall at top 3 vs the number of training samples available for
each technique, (b) Recall at top 3 vs MITRE ATT&CK document size (in
characters) for each technique.

Regarding LLMs, we notice that these models generate new technique titles and
associate them with technique IDs in their explanations. In addition, when
adding more than 15 few-shot examples to the prompt, we notice that the LLMs
loose context and do not follow the instructions. They hallucinate and
generate random sentences by either repeating some examples or formulating new
sentence from MITRE ATT&CK knowledge base. This underscores the need for
background knowledge that these variants of open LLMs have not been trained
on. While this limitation could potentially be mitigated by fine-tuning these
LLMs specifically for this task, it would require substantial computational
resources and training data.

###  5.6. Ablation Analysis

We delve into the design considerations behind the rankers deployed in the
last two stages of our system. For Stage-2, we’ve embraced a bi-encoder
architecture, to create representations of our corpus derived from the ATT&CK
knowledge base and the query behaviors from our dataset. The most eminent
model for evaluating text similarity is the sentence transformers (Reimers and
Gurevych, 2019b), denoted as SBERT. This model is generated through fine-
tuning BERT on a generic sentence similarity evaluation task. In contrast, our
Stage-2 ranker relies on a further pretrained version of BERT on 5K APT attack
reports to adapt it to the cybersecurity domain, followed by a fine-tuning
process to capture semantic relevance.

An alternative to SentSecBERT is the publicly available Attack-BERT model,
which is designed based on a BERT variant primarily trained to map common
vulnerabilities and exposures (CVE) entries to ATT&CK techniques using textual
similarity (Abdeen et al., 2023). A comparison involving the substitution of
the Stage-2 ranker with these four models is presented in Table 7. Notably,
the fine-tuned SentSecBERT consistently outperforms the other models by a
margin of at least 1.5%. We also assessed the impact of fine-tuning the mono-
encoder architecture, specifically monoT5, on performance improvement. We also
assessed how fine-tuning the mono-encoder architecture, i.e., the monoT5,
improves the performance. The outcomes of replacing the Stage-3 ranker with
monoT5 are detailed in Table 8, revealing a potential enhancement of up to 9%.

Table 7. Ablation Analysis of Stage-2 Ranker Model | Recall@50 | Recall@20 | Recall@10 | Recall@3
---|---|---|---|---
SBert (Reimers and Gurevych, 2019b) | 0.9505 | 0.8969 | 0.8251 | 0.6576
Attack-Bert191919https://huggingface.co/basel/ATTACK-BERT | 0.9414 | 0.8903 | 0.8350 | 0.6535
SentSecBert | 0.9554 | 0.8903 | 0.8226 | 0.6394
FT SentSecBert | $\mathbf{0.9596}$ | $\mathbf{0.9059}$ | $\mathbf{0.8383}$ | $\mathbf{0.6732}$
Table 8. Ablation Analysis of Stage-3 Ranker Model | Recall@20 | Recall@10 | Recall@3
---|---|---|---
MonoT5 | 0.9101 | 0.8564 | 0.7244
FT MonoT5 | $\mathbf{0.9480}$ | $\mathbf{0.9208}$ | $\mathbf{0.8102}$

In our fine-tuning process for the rankers, we’ve utilized 70% of the threat
behavior descriptions in our dataset, combining information from four distinct
sources as outlined in Table 2. This prompts the consideration of how using a
singular dataset for fine-tuning rankers impacts their performance when tested
on the other three. To address this, we fine-tuned the model on one dataset
and evaluated it on the remaining three datasets. The performance outcomes for
the Stage-2 and Stage-3 rankers are displayed in Table 9 and 10, respectively.
Interestingly, these results demonstrate that the optimal test performance is
achieved by fine-tuning the model using manually extracted behaviors from
ATT&CK Reports. Conversely, the least favorable test performance arises from
rankers trained on the CISA dataset, likely due to its succinct descriptions
that encompass multiple behaviors. It must be emphasized here that the
combined dataset offers the broadest technique coverage. Moreover, these
variations in analysts’ writing styles also reflect the real-world diversity
in describing threat behaviors. These suggest that individual datasets
shouldn’t be employed for training the multi-stage ranker.

Table 9. Impact of Using Different Datasets for Fine-Tuning Stage-2 Ranker Test | Train | Recall@50 | Recall@20 | Recall@10
---|---|---|---|---
CISA | ATT&CK Rpts. | 0.8967 | 0.8040 | 0.7124
TRAM | 0.8932 | 0.7899 | 0.6995
WeLiveSecurity | 0.9026 | 0.7934 | 0.7183
WeLiveSecurity | ATT&CK Rpts. | 0.9649 | 0.9263 | 0.8889
TRAM | $\mathbf{0.9705}$ | 0.9331 | 0.8844
CISA | 0.9694 | $\mathbf{0.9456}$ |  0.$\mathbf{9138}$
ATT&CK Rpts. | TRAM | 0.9842 | 0.9339 | 0.8746
WeLiveSecurity | 0.9887 | 0.9422 | 0.8844
CISA | $\mathbf{0.9902}$ | $\mathbf{0.9557}$ | $\mathbf{0.9084}$
TRAM | ATT&CK Rpts. | 0.9245 | 0.8455 | 0.7540
WeLiveSecurity | 0.9307 | 0.8641 | 0.7638
CISA | $\mathbf{0.9405}$ | $\mathbf{0.8783}$ | $\mathbf{0.7806}$
Table 10. Impact of Using Different Datasets for Fine-Tuning Stage-3 Ranker Test | Train | Recall@10 | Recall@5 | Recall@3
---|---|---|---|---
CISA | ATT&CK Rpts. | 0.7300 | 0.6432 | 0.5481
TRAM | 0.7042 | 0.6080 | 0.5059
WeLiveSecurity | 0.6784 | 0.5469 | 0.4354
WeLiveSecurity | ATT&CK Rpts. | 0.9297 | 0.8617 | 0.7891
TRAM | 0.9127 | 0.8447 | 0.7642
CISA | 0.9240 | 0.8639 | 0.8027
ATT&CK Rpts. | TRAM | 0.9775 | 0.9580 | 0.9062
WeLiveSecurity | 0.9610 | 0.8851 | 0.7590
CISA | 0.9865 | 0.9700 | 0.9384
TRAM | ATT&CK Rpts. | 0.7673 | 0.6599 | 0.5853
WeLiveSecurity | 0.6856 | 0.5595 | 0.4272
CISA | 0.7540 | 0.6545 | 0.5533

##  6\. Discussion

##### Error Analysis:

We analyzed the sources of errors for both the Stage-2 and Stage-3 rankers, as
presented in Table 3. To achieve this, we classified the threat behaviors in
our test split that are ranked within the top-$K$ category based on their
source dataset. The performance of the Stage-2 ranker in correctly ranking
behaviors across four datasets is demonstrated in Table 11. The recall values
at various top-$K$ values indicate that the model’s performance varies
depending on the source of the sentence. These results demonstrate that our
multi-stage ranker performs relatively better on descriptions from ATT&CK
Reports and WeLiveSecurity datasets compared to the TRAM and CISA datasets.
The corresponding outcomes for the Stage-3 ranker are provided in Table 12. A
similar pattern is observed, but with an amplified performance gap between the
two groups. This observation can be attributed to two primary factors. The
first factor is the sentence structure itself. TRAM and CISA descriptions tend
to be concise, while others are more verbose. The second factor is the
imbalance in the coverage of ATT&CK techniques across each dataset.

Table 11. Stage-2 Ranker Performance Across Individual Datasets Dataset | Recall@50 | Recall@20 | Recall@10
---|---|---|---
CISA | 0.9156 | 0.8089 | 0.7200
TRAM | 0.9434 | 0.8994 | 0.8176
WeLiveSecurity | 0.9725 | 0.9529 | 0.9020
ATT&CK Reports | 0.9879 | 0.9348 | 0.8792
Table 12. Stage-3 Ranker Performance Across Individual Datasets Dataset | Recall@10 | Recall@5 | Recall@3
---|---|---|---
CISA | 0.8311 | 0.7511 | 0.6933
TRAM | 0.8616 | 0.7516 | 0.6792
WeLiveSecurity | 0.9686 | 0.9451 | 0.8902
ATT&CK Reports | 0.9855 | 0.9710 | 0.9251

##### Report Sentence Structures:

The threat behaviors in our dataset originate from various sources of threat
reports, demonstrating a wide range of textual structures employed by
analysts. The results of Tables 9-10 and Tables 11-12. also suggest that
utilizing the TRAM dataset for fine-tuning or testing generally leads to the
lowest performance. This dataset predominantly consists of brief descriptions
or phrases associated with attack activities. We posit that such succinct
descriptions lacking adequate context may not provide an accurate basis for
language models to perform semantic evaluation.

##  7\. Related Work

Gathering technique-level knowledge from threat analysis reports with
reference to the ATT&CK knowledge base has gained prominence due to the
growing adoption of this framework as a point of reference by numerous
cybersecurity detection and mitigation tools. Therefore, several approaches
have been proposed to automate the retrieval of ATT&CK techniques from threat
reports.

Earlier approaches utilized conventional text representations for this
purpose. These included sparse representations based on bag-of-words features,
including TF-IDF (Legoy et al., 2020; Tsai et al., 2020) and frequencies of
bi-grams and tri-grams (Legoy et al., 2020; tra, 2023), as well as dense word
embeddings like Word2Vec (Legoy et al., 2020). These text representations were
subsequently integrated with various classification models, considering both
multi-class and multi-label classification scenarios. These models were
trained on datasets manually curated from threat intelligence reports.
Following a comprehensive evaluation, (Legoy et al., 2020) determined that the
combination of TF-IDF features with a linear SVM classifier yielded the most
favorable performance.

The initial text representation approach lacked any specific alignment with
the task and did not incorporate any relevant elements from it. Therefore,
subsequent methods expanded upon this approach to achieve a more nuanced
understanding of threat behaviors. To this objective, (Husari et al., 2017a)
proposed TTPDrill, a method that extracts threat actions from threat reports
and aligns them with attack patterns. This alignment is achieved through a
predefined ontology that captures the relationships between various components
of threat actions. The ontology utilized in TTPDrill was primarily built upon
MITRE’s Common Attack Pattern Enumeration and Classification (CAPEC) catalog
and the ATT&CK threat repository. These resources offer a comprehensive
compilation of attack patterns, tactics, techniques, and procedures that
adversaries employ to exploit system vulnerabilities. TTPrill employed part-
of-speech parsing to identify candidate threat actions, in the form of
$(subject,verb,object)$ tuples. These actions were then weighted and filtered
using a BM25-based method to identify the best-matching threat actions within
the threat ontology.

Another approach in this direction is the AttacKG method (Li et al., 2022a).
To identify adversarial techniques, AttacKG creates technique templates. These
are essentially attack graphs built upon technique procedure examples crawled
from threat reports, consisting of attack entities as nodes and their
interconnections as edges. Attack entities include both subjects of attack
behaviors and system-level objects they target. This information is extracted
using a regex-based IoC detector and a pretrained named entity recognition
module. AttacKG employs a graph alignment algorithm to map threat behaviors
extracted from reports onto these predefined technique templates. The success
of this method hinges on accurately identifying entities and their
interdependencies, which is a distinct challenge in extraction of threat
knowledge (Satvat et al., 2021b; Gao et al., 2021b; Ji et al., 2022; Ren et
al., 2022).

With the rising popularity of Transformer-based language models, recent
approaches exploited their language modeling capabilities to implement
semantic similarity-based approaches for the mapping task. Using a pretrained
sentence transformer model (Reimers and Gurevych, 2019b), these work compute
the embeddings of the descriptions and evaluated relevance based on similarity
of these embeddings. In this regard, TIM (You et al., 2022) used two
representations obtained from three-sentence long chunks of text from reports:
a text embedding and a vector of values indicating the number of different
types of IoCs observed in those passages. The similarities of these two
vectors are then evaluated to ground truth threat behaviors extracted from
other reports. In contrast to this approach, LADDER (Alam et al., 2023) used
only attack phrases extracted from the relevant parts of the sentences
identified to contain attack pattern descriptions and evaluated the similarity
with respect to title and description of the ATT&CK technique descriptions.

The most similar approaches to ours are the last two, (You et al., 2022) and
(Alam et al., 2023). In comparison to these works, we present a novel problem
formulation through a multi-stage ranking scheme, which enables us to assess
semantic similarity at varying levels. Importantly, our ranking models are
constructed using language models that are tailored to the cybersecurity
domain and fine-tuned for our specific task, resulting in improved accuracy in
technique identification. Moreover, in our evaluation, we extensively explore
the performance of several advanced LMs, trained on web-scale text corpora,
for this particular task.

Another crucial aspect to consider is the collection of threat behavior
descriptions needed for training and testing these approaches. Proposed
methods have so far used custom-built datasets that vary in their coverage of
adversarial techniques, ranging from as few as six techniques (You et al.,
2022) to as many as 179 techniques (Li et al., 2022a). Notably, the largest of
these custom datasets contained around 7.3K technique descriptions (Li et al.,
2022a). In this context, as part of our work we introduce a public dataset
with 6.6K annotations spanning over 410 techniques.

##  8\. Conclusion

We introduce a new method for extracting structured threat behaviors from
threat intelligence text. Our method is based on a multi-stage ranking
architecture that allows jointly optimizing for efficiency and effectiveness.
Therefore, we believe this problem formulation better aligns with the real-
world nature of the task considering the large number of adversary techniques
and the extensive body of threat intelligence created by security analysts.
Our findings show that the proposed system yields state-of-the-art performance
results for this task. Results show that our method has a top-3 recall
performance of 81% in identifying the relevant technique among 193 top-level
techniques. Our tests also demonstrate that our system performs significantly
better (+40%) than the widely used large language models when tested under a
zero-shot setting.

## References

  * (1)
  * tra (2023) 2023\.  https://mitre-engenuity.org/cybersecurity/center-for-threat-informed-defense/our-work/threat-report-attck-mapper-tram/
  * Abdeen et al. (2023) Basel Abdeen, Ehab Al-Shaer, Anoop Singhal, Latifur Khan, and Kevin Hamlen. 2023.  SMET: Semantic Mapping of CVE to ATT&CK and Its Application to Cybersecurity. In _IFIP Annual Conference on Data and Applications Security and Privacy_. Springer, 243–260.
  * Alam et al. (2022) Md Tanvirul Alam, Dipkamal Bhusal, Youngja Park, and Nidhi Rastogi. 2022.  Looking Beyond IoCs: Automatically Extracting Attack Patterns from External CTI.  _arXiv preprint arXiv:2211.01753_ (2022).
  * Alam et al. (2023) Md Tanvirul Alam, Dipkamal Bhusal, Youngja Park, and Nidhi Rastogi. 2023.  Looking Beyond IoCs: Automatically Extracting Attack Patterns from External CTI.  arXiv:2211.01753 [cs.CR]
  * Asadi and Lin (2013) Nima Asadi and Jimmy Lin. 2013.  Effectiveness/efficiency tradeoffs for candidate generation in multi-stage retrieval architectures. In _Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval_. 997–1000.
  * Blogs (2023) Symantec Enterprise Blogs. 2023.  Threat Intelligence.  https://symantec-enterprise-blogs.security.com/blogs/threat-intelligence Accessed: May 27, 2023.
  * Broadcom (2023) Broadcom. 2023.  Protection Bulletin.  https://www.broadcom.com/support/security-center/protection-bulletin Accessed: May 27, 2023.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Clarke et al. (2016) Charles LA Clarke, J Shane Culpepper, and Alistair Moffat. 2016.  Assessing efficiency–effectiveness tradeoffs in multi-stage retrieval systems without using relevance judgments.  _Information Retrieval Journal_ 19 (2016), 351–377.
  * Corporation (2022) MITRE Corporation. 2022.  MITRE ATT&CK Enterprise Matrix.  https://attack.mitre.org/versions/v12/matrices/enterprise/.  Accessed: June 5, 2023.
  * Crestani et al. (1998) Fabio Crestani, Mounia Lalmas, Cornelis J Van Rijsbergen, and Iain Campbell. 1998.  “Is this document relevant?… probably” a survey of probabilistic models in information retrieval.  _ACM Computing Surveys (CSUR)_ 30, 4 (1998), 528–552.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_ , Vol. 1. 4171–4186.
  * Gao et al. (2021a) Peng Gao, Xiaoyuan Liu, Edward Choi, Bhavna Soman, Chinmaya Mishra, Kate Farris, and Dawn Song. 2021a.  A system for automated open-source threat intelligence gathering and management. In _Proceedings of the 2021 International Conference on Management of Data_. 2716–2720.
  * Gao et al. (2021b) Peng Gao, Fei Shao, Xiaoyuan Liu, Xusheng Xiao, Zheng Qin, Fengyuan Xu, Prateek Mittal, Sanjeev R Kulkarni, and Dawn Song. 2021b.  Enabling efficient cyber threat hunting with cyber threat intelligence. In _2021 IEEE 37th International Conference on Data Engineering (ICDE)_. IEEE, 193–204.
  * Huang et al. (2020) Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padmanabhan, Giuseppe Ottaviano, and Linjun Yang. 2020.  Embedding-based retrieval in facebook search. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2553–2561.
  * Husari et al. (2017a) Ghaith Husari, Ehab Al-Shaer, Mohiuddin Ahmed, Bill Chu, and Xi Niu. 2017a.  TTPDrill: Automatic and Accurate Extraction of Threat Actions from Unstructured Text of CTI Sources. In _Proceedings of the 33rd Annual Computer Security Applications Conference_ (Orlando, FL, USA) _(ACSAC ’17)_. Association for Computing Machinery, New York, NY, USA, 103–115.  https://doi.org/10.1145/3134600.3134646
  * Husari et al. (2017b) Ghaith Husari, Ehab Al-Shaer, Mohiuddin Ahmed, Bill Chu, and Xi Niu. 2017b.  Ttpdrill: Automatic and accurate extraction of threat actions from unstructured text of cti sources. In _Proceedings of the 33rd annual computer security applications conference_. 103–115.
  * Ji et al. (2022) Zhengjie Ji, Edward Choi, and Peng Gao. 2022.  A Knowledge Base Question Answering System for Cyber Threat Knowledge Acquisition. In _2022 IEEE 38th International Conference on Data Engineering (ICDE)_. IEEE, 3158–3161.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019.  Latent retrieval for weakly supervised open domain question answering.  _arXiv preprint arXiv:1906.00300_ (2019).
  * Legoy et al. (2020) Valentine Legoy, Marco Caselli, Christin Seifert, and Andreas Peter. 2020.  Automated retrieval of att&ck tactics and techniques for cyber threat reports.  _arXiv preprint arXiv:2004.14322_ (2020).
  * Li et al. (2022a) Zhenyuan Li, Jun Zeng, Yan Chen, and Zhenkai Liang. 2022a.  AttacKG: Constructing Technique Knowledge Graph From Cyber Threat Intelligence Reports. In _Computer Security – ESORICS 2022: 27th European Symposium on Research in Computer Security, Copenhagen, Denmark, September 26–30, 2022, Proceedings, Part I_ (Copenhagen, Denmark). Springer-Verlag, Berlin, Heidelberg, 589–609.  https://doi.org/10.1007/978-3-031-17140-6_29
  * Li et al. (2022b) Zhenyuan Li, Jun Zeng, Yan Chen, and Zhenkai Liang. 2022b.  AttacKG: Constructing technique knowledge graph from cyber threat intelligence reports. In _Computer Security–ESORICS 2022: 27th European Symposium on Research in Computer Security, Copenhagen, Denmark, September 26–30, 2022, Proceedings, Part I_. Springer, 589–609.
  * Liao et al. (2016) Xiaojing Liao, Kan Yuan, XiaoFeng Wang, Zhou Li, Luyi Xing, and Raheem Beyah. 2016.  Acing the ioc game: Toward automatic discovery and analysis of open-source cyber threat intelligence. In _Proceedings of the 2016 ACM SIGSAC conference on computer and communications security_. 755–766.
  * Lin et al. (2022) Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2022.  _Pretrained transformers for text ranking: Bert and beyond_.  Springer Nature.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Mackenzie et al. (2018) Joel Mackenzie, J Shane Culpepper, Roi Blanco, Matt Crane, Charles LA Clarke, and Jimmy Lin. 2018.  Query driven algorithm selection in early stage retrieval. In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining_. 396–404.
  * Matveeva et al. (2006) Irina Matveeva, Chris Burges, Timo Burkard, Andy Laucius, and Leon Wong. 2006.  High accuracy retrieval with multiple nested ranker. In _Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval_. 437–444.
  * Mu et al. (2018) Dongliang Mu, Alejandro Cuevas, Limin Yang, Hang Hu, Xinyu Xing, Bing Mao, and Gang Wang. 2018.  Understanding the reproducibility of crowd-reported security vulnerabilities. In _27th USENIX Security Symposium (USENIX Security 18)_. 919–936.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  Ms marco: A human-generated machine reading comprehension dataset.  (2016).
  * Nogueira et al. (2020) Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2020.  Document ranking with a pretrained sequence-to-sequence model.  _arXiv preprint arXiv:2003.06713_ (2020).
  * Pradeep et al. (2020) Ronak Pradeep, Xueguang Ma, Rodrigo Nogueira, and Jimmy Lin. 2020.  Scientific claim verification with VerT5erini.  _arXiv preprint arXiv:2010.11930_ (2020).
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ 21, 1 (2020), 5485–5551.
  * Ranade et al. (2021) Priyanka Ranade, Aritran Piplai, Anupam Joshi, and Tim Finin. 2021.  Cybert: Contextualized embeddings for the cybersecurity domain. In _2021 IEEE International Conference on Big Data (Big Data)_. IEEE, 3334–3342.
  * Reimers and Gurevych (2019a) Nils Reimers and Iryna Gurevych. 2019a.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_ (2019).
  * Reimers and Gurevych (2019b) Nils Reimers and Iryna Gurevych. 2019b.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.  _CoRR_ abs/1908.10084 (2019).  arXiv:1908.10084 http://arxiv.org/abs/1908.10084
  * Ren et al. (2022) Yitong Ren, Yanjun Xiao, Yinghai Zhou, Zhiyong Zhang, and Zhihong Tian. 2022.  CSKG4APT: A Cybersecurity Knowledge Graph for Advanced Persistent Threat Organization Attribution.  _IEEE Transactions on Knowledge and Data Engineering_ (2022).
  * Robertson et al. (1995) Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995\.  Okapi at TREC-3.  _Nist Special Publication Sp_ 109 (1995), 109.
  * Satvat et al. (2021a) Kiavash Satvat, Rigel Gjomemo, and VN Venkatakrishnan. 2021a.  Extractor: Extracting attack behavior from threat reports. In _2021 IEEE European Symposium on Security and Privacy (EuroS &P)_. IEEE, 598–615.
  * Satvat et al. (2021b) Kiavash Satvat, Rigel Gjomemo, and V.N. Venkatakrishnan. 2021b.  Extractor: Extracting Attack Behavior from Threat Reports. In _2021 IEEE European Symposium on Security and Privacy (EuroS &P)_. IEEE Computer Society, Vienna, Austria, 598–615.  https://doi.org/10.1109/EuroSP51992.2021.00046
  * Securelist (2023) Securelist. 2023.  APT Reports Archives.  https://securelist.com/category/apt-reports/ Accessed: May 27, 2023.
  * Sun et al. (2019) Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019.  Ernie: Enhanced representation through knowledge integration.  _arXiv preprint arXiv:1904.09223_ (2019).
  * Tang et al. (2019) Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019.  Distilling task-specific knowledge from bert into simple neural networks.  _arXiv preprint arXiv:1903.12136_ (2019).
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023\.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ (2023).
  * Tsai et al. (2020) Chia-En Tsai, Cheng-Lin Yang, and Chong-Kuan Chen. 2020.  CTI ANT: Hunting for Chinese threat intelligence. In _2020 IEEE International Conference on Big Data (Big Data)_. IEEE, 1847–1852.
  * Tunstall et al. (2023) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. 2023\.  Zephyr: Direct distillation of lm alignment.  _arXiv preprint arXiv:2310.16944_ (2023).
  * WithSecure (2023) WithSecure. 2023.  Publications.  https://labs.withsecure.com/publications Accessed: May 27, 2023.
  * Xu et al. (2023) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023.  Wizardlm: Empowering large language models to follow complex instructions.  _arXiv preprint arXiv:2304.12244_ (2023).
  * Yan et al. (2021) Ming Yan, Chenliang Li, Bin Bi, Wei Wang, and Songfang Huang. 2021.  A unified pretraining framework for passage ranking and expansion. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 4555–4563.
  * Yang et al. (2020) Jheng-Hong Yang, Sheng-Chieh Lin, Rodrigo Nogueira, Ming-Feng Tsai, Chuan-Ju Wang, and Jimmy Lin. 2020.  Designing templates for eliciting commonsense knowledge from pretrained sequence-to-sequence models. In _Proceedings of the 28th International Conference on Computational Linguistics_. 3449–3453.
  * Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022.  React: Synergizing reasoning and acting in language models.  _arXiv preprint arXiv:2210.03629_ (2022).
  * Yates et al. (2021) Andrew Yates, Rodrigo Nogueira, and Jimmy Lin. 2021.  Pretrained transformers for text ranking: BERT and beyond. In _Proceedings of the 14th ACM International Conference on web search and data mining_. 1154–1156.
  * You et al. (2022) Yizhe You, Jun Jiang, Zhengwei Jiang, Peian Yang, Baoxu Liu, Huamin Feng, Xuren Wang, and Ning Li. 2022.  TIM: threat context-enhanced TTP intelligence mining on unstructured threat data.  _Cybersecurity_ 5, 1 (2022), 3.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023\.  Judging LLM-as-a-judge with MT-Bench and Chatbot Arena.  _arXiv preprint arXiv:2306.05685_ (2023).
  * Zhu and Dumitras (2018) Ziyun Zhu and Tudor Dumitras. 2018.  Chainsmith: Automatically learning the semantics of malicious campaigns by mining threat intelligence reports. In _2018 IEEE European symposium on security and privacy (EuroS &P)_. IEEE, 458–472.
  * Zou et al. (2021) Lixin Zou, Shengqiang Zhang, Hengyi Cai, Dehong Ma, Suqi Cheng, Shuaiqiang Wang, Daiting Shi, Zhicong Cheng, and Dawei Yin. 2021.  Pre-trained language model based ranking in Baidu search. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_. 4014–4022.

##  Appendix A Datasets Used for Fine-Tuning SecBert

Table 13. Datasets Used for Fine-Tuning SecBert Model Dataset | # text pairs
---|---
specter_train_triples | 684,100
stackexchange_duplicate_questions_title_title | 304,525
AllNLI | 277,230
sentence-compression | 180,000
wikihow | 128,542
SimpleWiki | 102,225
NQ-train_pairs | 100,231
squad_pairs | 87,599
TriviaQA_pairs | 73,346
Total | $\mathbf{1,937,798}$

The pretrained SecBert model is fine-tuned to facilitate semantic text
similarity, aligning with our technique annotation task’s requirements. Due to
the finite size of our threat behavior description dataset, we incorporated
some of the datasets202020https://huggingface.co/datasets/sentence-
transformers/embedding-training-data utilized in training the sentence
transformer model as detailed in Table 13.

##  Appendix B Prompts Used in Experiments

The prompt used in zero-shot learning setting (Fig. 6) and a sample prompt
used in few-shot experiments with two examples (Fig. 7).

Act as an experienced security analyst, identify the TOP 3 more likely MITRE ATT&CK Technique IDs that the text describes. Explain in 20 words why you chose these techniques specifically. Be concise. Answer in the following format: {"techniques_IDs": [YOUR_ANSWER], "explanation": "YOUR_EXPLANATION"}. Output only technique IDs without names. Make sure the technique IDs follow MITRE ATT&CK Technique IDs format (i.e., TXXXX or TXXXX.YYY where X and Y are numbers from 0 to 9). Make sure the explanation matches the techniques. Make sure the output is a valid JSON. Figure 6. The prompt used consistently across all LLMs in the zero-shot learning setting. [REDACTED (see Figure 6)] Examples: \- Input: Siloscape allows two types of instructions, one for kubectl supported commands and one for regular Windows cmd commands. | Output: {"techniques_ids": "["T1609", "T1613", "T1659"]", "explanation": "Adversaries may abuse a container administration service to execute commands within a container. A container administration service such as the Docker daemon, the Kubernetes API server, or the kubelet may allow remote management of containers within an environment."} \- Input: Dolphin uses a temporary scheduled task to start after installation. | Output: {"techniques_ids": "["T1053", "T1029", "T1113"]", "explanation": "Adversaries may abuse task scheduling functionality to facilitate initial or recurring execution of malicious code. Utilities exist within all major operating systems to schedule programs or scripts to be executed at a specified date and time."}  Figure 7. A sample prompt used in the few-shot learning setting with two examples.