# Mitigating Data Imbalance for Software Vulnerability Assessment:
Does Data Augmentation Help?

Triet Huynh Minh Le  CREST - The Centre for Research on Engineering Software
Technologies, The University of AdelaideAdelaideAustralia Cyber Security
Cooperative Research Centre, Australia<EMAIL_ADDRESS>and  M. Ali
Babar  CREST - The Centre for Research on Engineering Software Technologies,
The University of AdelaideAdelaideAustralia Cyber Security Cooperative
Research Centre, Australia<EMAIL_ADDRESS>
###### Abstract.

Background: Software Vulnerability (SV) assessment is increasingly adopted to
address the ever-increasing volume and complexity of SVs. Data-driven
approaches have been widely used to automate SV assessment tasks, particularly
the prediction of the Common Vulnerability Scoring System (CVSS) metrics such
as exploitability, impact, and severity. SV assessment suffers from the
imbalanced distributions of the CVSS classes, but such data imbalance has been
hardly understood and addressed in the literature. Aims: We conduct a large-
scale study to quantify the impacts of data imbalance and mitigate the issue
for SV assessment through the use of data augmentation. Method: We leverage
nine data augmentation techniques to balance the class distributions of the
CVSS metrics. We then compare the performance of SV assessment models with and
without leveraging the augmented data. Results: Through extensive experiments
on 180k+ real-world SVs, we show that mitigating data imbalance can
significantly improve the predictive performance of models for all the CVSS
tasks, by up to 31.8% in Matthews Correlation Coefficient. We also discover
that simple text augmentation like combining random text insertion, deletion,
and replacement can outperform the baseline across the board. Conclusions: Our
study provides the motivation and the first promising step toward tackling
data imbalance for effective SV assessment.

Software vulnerability, Software security, Machine Learning, Deep learning,
Data augmentation

††conference: The 18th ACM/IEEE International Symposium on Empirical Software
Engineering and Measurement; 20–25 October, 2024; Barcelona, Spain††copyright:
none

##  1\. Introduction

Software Vulnerabilities (SVs) like Heartbleed (Synopsys, [n. d.]) or
Log4Shell (Conversation, [n. d.]) are security bugs that adversely affect the
quality of software systems, potentially leading to catastrophic cybersecurity
attacks (Ghaffarian and Shahriari, 2017). In practice, fixing all detected SVs
simultaneously is not always feasible due to time and resource constraints
(Khan and Parkinson, 2018; Le et al., 2021a). Rather, a more practical
approach involves prioritizing SVs posing serious and impending security
threats, which usually requires inputs from SV assessment (Smyth, 2017; Le et
al., 2022; Le, 2022). SV assessment identifies diverse attributes such as
exploitability, impact, and severity levels of SVs (Kamongi et al., 2013). For
instance, SVs with a substantial likelihood of exploitation and severe
consequences typically demand elevated priority for resolution. Currently,
Common Vulnerability Scoring System (FIRST, [n. d.]a) (CVSS) is the most
commonly used industry-grade standard for SV assessment. However, the
assignments of these CVSS metrics to ever-increasing SVs are tedious and time-
consuming for security experts (Feutrill et al., 2018), which has motivated
the research on automated approaches for the tasks.

An increasing number of studies have proposed various data-driven techniques
to automate the prediction of the CVSS metrics by leveraging available SV data
in the wild (e.g., (Yamamoto et al., 2015; Le et al., 2019; Spanos and
Angelis, 2018; Han et al., 2017; Gong et al., 2019)). Most of these prediction
models automatically learn the patterns from textual SV descriptions published
on SV repositories/databases such as National Vulnerability Database (NVD)
(NIST, [n. d.]a) to classify the CVSS metrics. The current literature has
explored different modeling algorithms ranging from traditional Machine
Learning (ML) techniques to more advanced Deep Learning (DL) architectures to
perform the classifications (Le et al., 2020a, 2022; Arani et al., 2024).
However, the development and quality of these classification models could be
negatively affected by the data imbalance issue; i.e., a data class has
significantly fewer samples than the other classes (Kaur et al., 2019).

We argue that data imbalance does exist in CVSS-based SV assessment, but it is
being overlooked by the current literature. Our analysis of the SVs published
on NVD from 1988 to 2023 showed that all the CVSS metrics used for SV
assessment, on average, had a (minority) class constituting only 10.2% of
total samples, being around six times smaller than that (61.1%) of the
respective majority class. The data imbalance issue has been shown to
significantly impede the performance of downstream prediction models in
various classification tasks (e.g., (Malhotra and Khanna, 2017; Rodriguez et
al., 2014; Song et al., 2018; Liu et al., 2022)). Although our aforementioned
analysis has clearly shown the presence of data imbalance in CVSS-based SV
assessment, little is known about the potential impact/benefits of mitigating
the issue for the tasks.

Data augmentation has been one of the most widely used tools to gauge the
impact and provide mitigation of the data imbalance issue in the ML/DL domains
(Feng et al., 2021). This approach aims to artificially increase the number of
samples, which can adjust the class distributions of data during model
training, making the model less biased towards the majority class(es). Given
that SV assessment is currently using textual SV descriptions as input,
existing data augmentation techniques for text data would be, in principle,
applicable to this domain. It is worth noting that these data augmentation
techniques have been demonstrated to be effective for various text
classification tasks (Bayer et al., 2022). Nevertheless, to the best of our
knowledge, there has been no systematic investigation into the extent to which
these data augmentation techniques are useful for tackling the data imbalance
issue for SV assessment.

To bridge this gap, we aim to conduct the first large-scale study on the
potential utilization of data augmentation for quantifying and mitigating the
data imbalance issue in the development of SV assessment models. We first
investigate nine different data augmentation techniques to generate augmented
SV descriptions from the original descriptions of 180,087 real-world SVs.
Then, we compare the predictive performance of SV assessment models with and
without using the augmented descriptions. These models leverage commonly used
ML and DL techniques to automate the classification of the seven CVSS metrics,
i.e., Access Vector, Access Complexity, Authentication, Confidentiality,
Integrity, Availability, and Severity.

Our key contributions can be summarized as follows:

  * •

Through the lens of data augmentation, we are the first to systematically
investigate the significance and impacts of mitigating the data imbalance
issue on the SV assessment models based on SV reports/descriptions. Our
findings show that addressing data imbalance can improve the performance of SV
assessment models by 5.3–31.8% in Matthews Correlation Coefficient (MCC)
across the seven CVSS metrics.

  * •

We empirically benchmark the effectiveness of different data augmentation
techniques for SV assessment. We find that a combination of random text
insertion, deletion, and substitution/replacement produces the highest
performance averaging all models and tasks, i.e., 11.3% better MCC than the
baseline without using data augmentation. We also shed light on the best
techniques for individual tasks and models that help achieve new heights in
performance for SV assessment.

  * •

We share the code and models at (Authors, [n. d.]) to reproduce the results
and facilitate future research in this direction.

Overall, our study sheds light on the possibility of using data augmentation
to enhance SV assessment. Our findings are expected to provide baselines for
researchers to further explore this direction and improve the performance of
SV assessment, which in turn can enable more effective SV mitigation/fixing
for practitioners.

Paper structure. Section 2 introduces SV assessment tasks and the potential of
data augmentation for the tasks. Section 3 presents the research questions and
the respective research methods to answer these questions. Section 4 shows the
experimental results of each question. Section 5 discusses the findings and
the threats to validity. Section 6 covers the related work. Section 7
concludes the study.

##  2\. Background and Motivation

###  2.1. CVSS-Based SV Assessment

SV assessment, occurring between the detection and remediation phases in the
SV management lifecycle, reveals various characteristics of SVs identified
earlier (Foreman, 2019). In practice, an average of 80 new SVs are discovered
daily (NIST, [n. d.]b), and each SV may require up to 1.5 hours to fix
(Cornell, 2012; ben Othmane et al., 2015), totaling 120 hours needed for
fixing. These statistics mean that there is certainly not sufficient time to
fix all of these SVs within a 24-hour day, so the fixing prioritization of SVs
is inevitable. SV assessment pinpoints “hot spots” in terms of security risks,
demanding more attention in a system. Accordingly, the assessment guides the
development of an efficient prioritization for SV fixing based on
resource/time availability.

Common Vulnerability Scoring System (CVSS) (FIRST, [n. d.]a) has been widely
regarded as the standard framework by both researchers and practitioners for
conducting SV assessment. Despite newer versions of CVSS, version 2 is still
the most commonly used because its assessment of old SVs is still relevant. An
illustrative example is the SV with CVE-2004-0113 which was discovered in 2004
and was still exploited in 2018 (Future, [n. d.]). Therefore, in this study,
we choose the metrics of CVSS version 2 as the outputs for our SV assessment
models. In this paper, we use the term “CVSS metrics” to mainly refer to
version 2 of the CVSS metrics unless specified otherwise.

CVSS version 2 offers metrics that gauge three primary facets of SVs:
Exploitability, Impact, and Severity. Exploitability evaluates Access Vector
(i.e., the medium/technique to penetrate a system), Access Complexity (i.e.,
the complexity to initiate an attack), and Authentication (i.e., whether/what
authentication the attack requires). Meanwhile, Impact metrics focus on the
Confidentiality, Integrity, and Availability attributes of the system of
interest in case of exploitation. Severity is then determined based on both
Exploitation and Impact metrics, which approximates the criticality of
detected SVs. However, Severity is a high-level combination of Exploitability
and Impact, and thus, it does not provide a full understanding of SVs,
potentially leading to a sub-optimal SV fixing plan. For example, according to
the CVSS specification (FIRST, [n. d.]b), two SVs would have the same severity
level if they share the same exploitability but affect different attributes
(e.g., Confidentiality vs. Integrity) of a system to the same extent. As a
result, to ensure a thorough assessment of SVs, we utilize all of the seven
CVSS version 2 metrics (i.e., Confidentiality, Integrity, Availability, Access
Vector, Access Complexity, Authentication, and Severity) as the outputs for
building SV assessment, akin to prior studies (e.g., (Spanos and Angelis,
2018; Le et al., 2019; Gong et al., 2019; Le et al., 2021b)).

Figure 1. Data percentages (%) of the minority and the majority classes of the
seven CVSS metrics of the SVs collected from National Vulnerability Database,
illustrating the data imbalance issue for SV assessment. Note: The percentages
do not add up to 100% as each CVSS metric has three classes.

Despite the evident benefits of the CVSS metrics, it is extremely challenging
for security experts to manually assign these metrics for ever-increasing SVs.
It has been shown that it can take up to 100+ days for the CVSS metrics to be
assigned to an SV, mainly due to tedious manual assignment and verification
processes (Gong et al., 2019). To help alleviate such burden for security
experts, many of the current studies have relied on descriptions in SV reports
to develop Machine Learning (ML)/Deep Learning (DL) based techniques to
automatically predict missing CVSS metrics (e.g., (Han et al., 2017; Spanos
and Angelis, 2018; Gong et al., 2019; Le et al., 2019; Elbaz et al., 2020)).
These textual descriptions contain various insights about SVs, which can be
leveraged for SV assessment. For example, the description of CVE-1999-0315 is
“Buffer overflow in Solaris fdformat command gives root access to local
users”, distilling the location (i.e., Solaris fdformat command), type (i.e.,
buffer overflow), and impact/consequence (i.e., giving root access) of the SV.
Moreover, such descriptions are almost always present when new SVs are
published. The useful information and availability of SV descriptions have
made them valuable inputs/resources for timely CVSS-based SV assessment using
data-driven approaches (Le et al., 2022).

Figure 2. Overview of the research methods used for the investigation of data
augmentation for different SV assessment tasks.

###  2.2. Data Augmentation for SV Assessment with Data Imbalance

A key challenge with CVSS-based SV assessment using ML/DL is data imbalance,
which is illustrated in Fig. 1. Data imbalance occurs when the number of
samples of a (minority) class is much smaller than those of the other
(majority) classes. Based on more than 180k SVs we collected from National
Vulnerability Database (NIST, [n. d.]a), we found that the minority classes
existed for all the CVSS metrics. Notably, many of the minority classes (i.e.,
Access Vector, Access Complexity, and Authentication) even constituted less
than 3% of the total data. Moreover, SVs with Complete levels of
Confidentiality, Integrity, and Availability impacts are of practical
importance and require special attention to address, but they were only the
minority classes. Such data imbalance can make assessment models struggle to
capture data patterns and provide accurate predictions for these small-sized
yet important/relevant classes. This issue has also been shown to hinder the
overall model performance and usefulness in other domains (Kaur et al., 2019).
However, the degree of impact that the data imbalance problem has on SV
assessment is still largely unexplored. The potential revelation of such
impact with the use of data augmentation is given hereafter.

Data augmentation involves techniques that artificially generate
new/additional data samples to increase the data size (Mumuni and Mumuni,
2022). Data augmentation can change the class distributions and has been shown
to reduce the likelihood of overfitting for models. The most straightforward
way of data augmentation is to simply duplicate existing samples, namely
random over-sampling. Generally, this technique can be applied to any data-
driven tasks, but it is yet to be used for SV assessment. In the context of SV
assessment, the input is textual SV descriptions, as described in Section 2.1,
which also motivates the exploration of text-based (data) augmentation
techniques. These techniques basically make small changes to the input text or
change the text in a way that can still preserve the overall meaning of the
input text.111More details about these techniques are given in Section 3.3.
Such augmented data can also improve the model performance in many downstream
text classification tasks (Bayer et al., 2022). However, to the best of our
knowledge, there has been no study exploring the potential and use of these
text-based augmentation for SV assessment tasks. To bridge this gap, we study
the extent to which different data augmentation techniques, including general
sampling and text-specific ones, can enhance the model performance and in turn
highlight the impact/importance of mitigating the data imbalance issue for the
tasks.

##  3\. Case Study Design and Setup

We outline the setup for investigating the use of data augmentation techniques
for SV assessment. Section 3.1 describes the two research questions. Fig. 2
depicts the research methods used to answer the questions. Given the benefits
mentioned in Section 6.1, SV descriptions collected from SV reports on NVD
were used as the main input in our study; their details are given in Section
3.2. Such SV descriptions were then used by the data augmentation techniques
described in Section 3.3 to generate new descriptions for the investigations.
The original and augmented descriptions were used to train different SV
assessment models (see Section 3.4). These models were evaluated following the
realistic setting of time-based evaluation rounds in Section 3.5.

###  3.1. Research Questions

We set out to answer the following two Research Questions (RQs) to distill the
effectiveness of data augmentation for different SV assessment tasks using SV
descriptions.

  * •

RQ1: What is the significance of addressing data imbalance for the SV
assessment tasks? Existing studies have mostly developed SV assessment models
without considering/mitigating the data imbalance issue illustrated in Section
2.2. The existence of the data imbalance in the CVSS metrics is evident, but
the impacts of the issue on respective SV assessment models are still largely
unknown. For each of the seven CVSS metrics, RQ1 compares the performance of
SV assessment models using descriptions generated by different data
augmentation techniques with that of the baseline without using data
augmentation. For each metric, if the performance of the model with data
augmentation is better than that of the baseline, we can infer that data
imbalance indeed negatively affects the task. Otherwise, the impact is of
negligible concern. The findings of RQ1 are expected to demonstrate the
importance of mitigating the data imbalance issue when developing CVSS-based
SV assessment models.

  * •

RQ2: Which data augmentation techniques are effective for SV assessment? While
RQ1 shows the overall impact of data imbalance on the CVSS assessment metrics,
RQ2 provides a more fine-grained analysis of the effectiveness of each data
augmentation technique, i.e., whether a technique performs better or worse
than the baseline on average. We also identify the optimal data augmentation
technique for each SV assessment task. Moreover, RQ2 shows the extent to which
the commonly used types of SV assessment models would benefit performance-wise
from the data augmentation techniques for the tasks. The findings of RQ2 can
inform the choice of particular data augmentation technique(s) to tackle data
imbalance for SV assessment, which in turn opens up various future
opportunities for improving the performance of the tasks in general.

###  3.2. Dataset

To build a dataset for SV assessment, we collected real-world SVs reported on
NVD (NIST, [n. d.]a) from 1988 to 2023. To increase the relevance and
reliability of our experiments, we discarded SVs that contained “** REJECT **”
in their descriptions because they had been confirmed duplicated or incorrect
by security experts. We also excluded SVs that did not have any CVSS metrics.
Finally, we obtained a dataset consisting of 180,087 SVs along with the
respective expert-vetted descriptions and the values of the seven CVSS
assessment metrics, i.e., Access Vector, Access Complexity, Authentication,
Confidentiality, Integrity, Availability, and Severity. The data class
distributions of the collected CVSS metrics are given in Fig. 3. The values
from this figure reinforce the earlier argument in Section 2.2 that data used
for SV assessment based on the CVSS metrics are highly imbalanced and may
negatively affect the performance of downstream models.

Table 1. Examples of the original description and the descriptions generated by the studied data augmentation techniques of the CVE-1999-0315 SV. Note: The augmented descriptions were manually selected among multiple runs to enhance readability. Category |  Data Augmentation |  Software Vulnerability Description |  Explanation of the Changes
---|---|---|---
None |  Buffer overflow in Solaris fdformat command gives root access to local users. |  Original SV description
Simple Text Augmentation |  Insertion |  Buffer overflow in Solaris fdformat command gives vulnerability root access to local users. |  Inserting the word “vulnerability”; vulnerability is a common word appearing in different SV descriptions.
Deletion |  overflow in Solaris fdformat command gives root access to local users. |  Removing the word “Buffer”
Substitution |  Buffer error in Solaris fdformat command gives root access to local users. |  Substituting/replacing the word “overflow” with the word “error”
Synonym
Replacement |  Buffer overflow in Solaris fdformat command gives root path to local users. |  Replacing the word “access” with the synonymous word “path”
Combination |  error in Solaris fdformat command gives vulnerability root access to local users. |  Inserting the word “vulnerability” + Substituting the word “overflow” with the word “error” + Removing the word “Buffer”
Contextual Text Augmentation |  Back Translation |  Buffer overflow in Solaris fdformat command gives local users the root access. |  Translating the original description to German, i.e., “pufferüberlauf im fehler solari fdformat gibt lokal benutzern rootzugriff”, and then translating the German description back to English
Paraphrasing |  Solaris fdformat command allows users to access root via the buffer overflow bug, posing imminent threats. |  Rewriting the original description while retaining the key semantics of the description

###  3.3. Studied Data Augmentation Techniques

We studied three types of data augmentation techniques that work directly on
the textual SV descriptions extracted in Section 3.2: (i) Data Sampling, (ii)
Simple Text Augmentation, and (iii) Contextual Text Augmentation. These
techniques generated new/augmented descriptions that share the same labels as
the respective original descriptions to balance the number of samples of all
the classes of SV assessment tasks, i.e., CVSS metrics, aiming to address data
imbalance shown in Section 2.1. We focused on the data augmentation techniques
whose output is real text; we did not consider techniques that operate on the
feature level like SMOTE (Chawla et al., 2002) or on the model level like
mixup (Zhang et al., 2017) because it is non-trivial to reconstruct real text
from their output, hindering their interpretability.

Figure 3. Data class distributions of the seven CVSS metrics used for SV
assessment. Note: The total number of the collected SVs is 180,087.

####  3.3.1. Data Sampling

Data Sampling is the first type of data augmentation we employed for balancing
the class distributions of CVSS metrics. Specifically, we investigated two
data sampling strategies: Random Over-sampling and Random Under-Sampling.
Random Over-Sampling added random duplicates of the existing samples from
minority classes so that the numbers of all the classes were equal for each of
the SV assessment metrics. Conversely, Random Under-Sampling randomly removed
the existing samples of the majority classes to match the number of the
smallest class of each metric, i.e., the one with the least number of samples.
It is important to note that Random Under-Sampling does not directly align
with the definition of data augmentation (i.e., adding new data), yet we still
included it for the sake of completeness as it is also data sampling and has
been used for SV assessment (Han et al., 2017).

####  3.3.2. Simple Text Augmentation

Unlike Data Sampling, which only duplicated existing samples without making
textual changes, Simple Text Augmentation created new SV descriptions by
modifying original SV descriptions (see Table 1). We considered the following
textual modifications: (i) Insertion, (ii) Deletion, (iii) Substitution, (iv)
Synonym Replacement, and (v) Combination. These operations have been commonly
used to augment text in the Natural Language Processing domain (e.g., (Wei and
Zou, 2019; Bayer et al., 2022)), which is directly relevant to augmenting SV
descriptions investigated in this study. Insertion created new descriptions by
adding new word(s) at random position(s) in each description. We selected the
frequent words, i.e., appearing in at least 0.1% of descriptions in training
set and not in the list of stop words, to add to the generated samples. The
threshold ensured the impact of the inserted words on model training was non-
negligible (Spanos and Angelis, 2018), without limiting the diversity of
augmentations. Deletion randomly removed word(s) from each description to
generate the respective augmented version. Substitution combined Insertion and
Deletion, in which random word(s) in each description were first removed and
then replaced with other frequent word(s) in the vocabulary of training set
(excluding stop words), similar to Insertion. Synonym Replacement replaced
word(s) in each description that had at least one synonymous alternative in
WordNet (Miller, 1995); e.g., “access” is replaced with “path” in Table 1. We
prioritized the synonyms that frequently appeared in training set (excluding
stop words) to ensure these words were properly captured during model
training. Lastly, Combination performed a random combination of all of the
above operations altogether to make changes to SV descriptions. The techniques
in Simple Text Augmentation were implemented using the nlpaug library (Ma,
2019). It is worth noting that we applied these operations randomly to SV
descriptions to increase the diversity of generated samples, yet we only
changed up to 20% of the words per description to avoid significant changes to
the original semantics, as per the existing practice (Wei and Zou, 2019). We
found changing 20+% of the words tended to decrease model performance. We
would ensure to apply the operations to one word in an SV description in case
it contained fewer than five words. We did not swap the order of the words as
this operation did not affect the TF-IDF feature extractor in Section 3.4.

####  3.3.3. Contextual Text Augmentation

Simple Text Augmentation treated each word independently without the
surrounding context/words, which might not be optimal for preserving the
semantics of an entire description. Contextual Text Augmentation, on the other
hand, aimed at modifying the syntactic structure yet (theoretically) retaining
the overall meaning of a description. We studied two representative techniques
incorporating the context of SV descriptions: Back Translation and
Paraphrasing. These two techniques have been commonly used for text
augmentation (e.g., (Sennrich et al., 2016; Dai et al., 2023; Bayer et al.,
2022)). Back Translation first translated/converted text to an intermediate
language and then translated that back to the original language (English). The
changes/augmentations in text mainly came from the variations in the two steps
of translation. We chose German as the intermediate language and the
respective models for translating between German and English because they have
been shown to be effective for these translation tasks (Sennrich et al.,
2016). We also tested with another popular language, i.e., French, but the
performance was worse. Paraphrasing rewrote a sentence using different words
and/or text structures, while maintaining the original meaning. Our study used
GPT (Brown et al., 2020), the recent advance in Generative AI and used for SV
tasks (Fu et al., 2023; Le et al., 2024a), to paraphrase SV descriptions with
the prompt “As a software security expert, please paraphrase the following
text: text”. We also tried other prompts based on the recommended practices in
the literature (Dai et al., 2023), but the paraphrasing outputs were largely
the same. We implemented the aforementioned GPT-based paraphrasing using the
GPT4All library (Anand et al., 2023).

###  3.4. Studied SV Assessment Models

We leveraged the original and augmented SV descriptions to develop SV
assessment models. These models were based on the four most widely used model
types for the task (Le et al., 2022).

####  3.4.1. Random Forest (RF) + TF-IDF model

This model employed RF (Ho, 1995) to classify CVSS metrics. The RF model used
TF-IDF features, i.e., the multiplication between the term frequency (times a
word appears in a document) and the inverse document frequency (times a word
appears in all documents) of each word in the vocabulary of training set. RF
and TF-IDF have been frequently used as the baselines for SV assessment (e.g.,
(Spanos et al., 2017; Spanos and Angelis, 2018; Le et al., 2019)). Similar to
these prior studies, we tuned the RF classifier using the hyperparameters: no.
of classifiers: {100, 300, 500}, max depth: {3, 5, 7, 9} and leaf nodes: {100,
200, 300}. For TF-IDF, we also used a vocabulary with words appearing in at
least 0.1% of all the documents used for training. In addition, we
preprocessed text before extracting features with TF-IDF, including removing
stop words and punctuations, converting text to lowercase, and applying
stemming (i.e., changing words to their root form) (Porter, 1980). We used the
stop words from the _scikit-learn_ (Pedregosa et al., 2011) and _nltk_ (Loper
and Bird, 2002) libraries. Regarding the punctuations, we only removed the
ones at the end of a sentence or the ones followed by space(s) to retain
important/relevant words in the software/security domains such as “ _file.c_ ”
or “ _cross-site_ (scripting)”.

####  3.4.2. RF + Doc2Vec model

This baseline used the RF classifier with the same configurations as RF + TF-
IDF, but with a different feature extractor, namely Doc2Vec (Le and Mikolov,
2014). Doc2Vec derived the representation of an SV description using the
information from the constituent words. Similar to word2vec used in SV
assessment (e.g., (Han et al., 2017; Gong et al., 2019)), Doc2Vec captured the
surrounding contexts of words missing in TF-IDF. Moreover, Doc2Vec improved
upon word2vec by assigning different weights to words in a document rather
than simply averaging the word-wise vectors adopted by word2vec. Doc2Vec has
also been previously used for SV assessment (Kanakogi et al., 2021). We
followed the suggestion of Doc2Vec’s original authors (Le and Mikolov, 2014)
to train the Doc2Vec feature extraction using the distributed memory
architecture. For Doc2Vec, we also applied the same text-preprocessing steps
used for TF-IDF. As per the prior studies (Le et al., 2019; Han et al., 2017)
and our preliminary analysis, we used the default configurations of Doc2Vec
because other values did not change results significantly.

####  3.4.3. Convolutional Neural Network (CNN) model

CNN (Kim, 2014) has been used widely for SV assessment (e.g., (Han et al.,
2017; Gong et al., 2019; Sharma et al., 2021; Anwar et al., 2021)). The model
started with an embedding layer connected to one convolutional layer. We
considered various embedding sizes of 100, 200, 300. The convolutional layer
had $F$ filters with pre-defined size $K$. We tried different numbers of
filters of 64, 128, 256, and different filter sizes of 1, 3, 5. The
hyperparameters of CNN were inspired by existing studies (Han et al., 2017).
These filters extracted patterns of phrases (consecutive words) of size $K$.
The outputs of the filters, went through
$\operatorname{ReLU}(x)=\operatorname{max}(0,x)$, a non-linear activation
function (Nair and Hinton, 2010). We iterated through the convolutional
process, moving filters sequentially down along the input vector from the
beginning to the end, employing a stride of one. This smaller stride was
chosen to capture highly detailed information within input descriptions,
distinguishing it from larger strides. The convolutional outputs entered a
max-pooling layer to produce a vector representing an input description. The
output of the pooling layer was then fed into a softmax layer to classify each
CVSS metric. We followed the existing practices (Han et al., 2017) to train
CNN for SV assessment.

####  3.4.4. Long-Short Term Memory (LSTM) model

LSTM (Hochreiter and Schmidhuber, 1997) has also been commonly used as an
alternative to CNN to better capture the long-term dependencies among the
words in SV descriptions (Gong et al., 2019). The LSTM model had the same
embedding layer as the CNN model connected to a forward LSTM-based network
that read the input from left to right. We investigated different numbers of
LSTM cells of 64, 128, and 256, similar to related studies (e.g., (Sahin and
Tosun, 2019; Gong et al., 2019)). After processing each description, the LSTM
cells output a hidden vector representing the whole description. We fed such
vector into a softmax layer to perform the classification of CVSS metrics,
similar to CNN. Note that we tried Bi-directional LSTM for SV assessment, but
it did not yield a stronger performance.

###  3.5. Evaluation Techniques and Metrics

####  3.5.1. Data splitting

We employed time-based data splits for training, validating, and testing our
models. This setup simulated real-world scenarios where future unseen data was
not included during training, as recommended in the literature (Spanos and
Angelis, 2018; Le et al., 2019). This approach involved three rounds of
training, validation, and testing, utilizing five equally sized folds based on
the published dates of SVs. Each round $i$ utilized folds from 1 to $i$,
$i+1$, and $i+2$ for training, validation, and testing, respectively. We
applied data augmentation only to the descriptions in the training set and
used only the original descriptions for validation/testing. We then selected
the optimal model as the one with the highest average validation performance,
based on the hyperparameters in Section 3.4. The average testing performance
of the optimal model was then reported, ensuring more stable outcomes than a
single testing set (Raschka, 2018).

####  3.5.2. Evaluation measures

To assess the performance of automated SV assessment, we applied the F1-Score
and Matthews Correlation Coefficient (MCC) metrics, which have been widely
employed for CVSS classification (e.g., (Han et al., 2017; Spanos and Angelis,
2018; Jimenez et al., 2019; Le et al., 2024b)). These metrics were suitable
for handling imbalanced classes within our data (Luque et al., 2019), as
illustrated in Fig. 1. F1-Score spans between 0 to 1, while MCC ranges from –1
to 1, where 1 signifies the optimal value for both metrics. MCC, considering
all the cells explicitly in a confusion matrix, was utilized to select the
optimal models (Luque et al., 2019). Given that the tasks had multiple
classes, we used macro F1-Score and the multi-class version of MCC (Spanos and
Angelis, 2018; Gorodkin, 2004). It is important to note that MCC does not have
a direct correlation with F1-Score.

####  3.5.3. Statistical analysis

To affirm the significance of our findings, we used the non-parametric
Wilcoxon signed-rank test and its effect size ($r=Z/\sqrt{N}$, where $Z$ is
the statistic score of the test and $N$ represents the total sample size). The
magnitude of the effect size ($r\leq 0.1$: negligible, $0.1<r\leq 0.3$: small,
$0.3<r\leq 0.5$: medium, $r>0.5$: large) followed the established guidelines
(Field, 2013). We would confirm a test result statistically significant when
the confidence level was over 99%, equivalent to $p$-value $<$ 0.01. We used
this effect size because it has been commonly used for assessing and comparing
defect/SV prediction results (e.g., (Wattanakriengkrai et al., 2020; Le and
Babar, 2022)).

##  4\. Experimental Results of Data Augmentation for SV Assessment

Table 2. Testing performance in terms of MCC and F1-Score of the baseline None models (without using data augmentation) and the models using nine different data augmentation techniques. Notes: The baseline performance is highlighted in yellow. The best performance of the models using data augmentation is highlighted in dark green. |  | Access Vector | Access Comp. | Authentication | Confidentiality | Integrity | Availability | Severity
---|---|---|---|---|---|---|---|---
Model | Data Augmentation | MCC | F1 | MCC | F1 | MCC | F1 | MCC | F1 | MCC | F1 | MCC | F1 | MCC | F1
| None | 0.458 | 0.532 | 0.604 | 0.530 | 0.311 | 0.378 | 0.509 | 0.638 | 0.531 | 0.656 | 0.507 | 0.660 | 0.327 | 0.470
| Over-Sampling | 0.553 | 0.632 | 0.625 | 0.609 | 0.428 | 0.537 | 0.533 | 0.667 | 0.545 | 0.658 | 0.528 | 0.667 | 0.362 | 0.565
| Under-Sampling | 0.543 | 0.614 | 0.602 | 0.589 | 0.115 | 0.368 | 0.528 | 0.664 | 0.540 | 0.653 | 0.524 | 0.665 | 0.355 | 0.555
| Insertion | 0.540 | 0.611 | 0.625 | 0.608 | 0.420 | 0.465 | 0.532 | 0.666 | 0.540 | 0.653 | 0.472 | 0.644 | 0.357 | 0.557
| Deletion | 0.451 | 0.536 | 0.481 | 0.523 | 0.278 | 0.380 | 0.497 | 0.619 | 0.508 | 0.614 | 0.479 | 0.646 | 0.311 | 0.461
| Substitution | 0.441 | 0.520 | 0.460 | 0.511 | 0.253 | 0.368 | 0.488 | 0.609 | 0.506 | 0.613 | 0.478 | 0.646 | 0.306 | 0.460
| Synonym Replacement | 0.562 | 0.634 | 0.608 | 0.584 | 0.398 | 0.465 | 0.526 | 0.657 | 0.535 | 0.648 | 0.525 | 0.666 | 0.344 | 0.541
| Combination | 0.573 | 0.647 | 0.620 | 0.596 | 0.406 | 0.474 | 0.530 | 0.660 | 0.539 | 0.650 | 0.535 | 0.679 | 0.351 | 0.552
| Back Translation | 0.149 | 0.190 | 0.140 | 0.178 | 0.062 | 0.153 | 0.157 | 0.191 | 0.192 | 0.238 | 0.187 | 0.215 | 0.089 | 0.162
RF + TF-IDF | Paraphrasing | 0.396 | 0.481 | 0.403 | 0.476 | 0.181 | 0.465 | 0.461 | 0.588 | 0.495 | 0.612 | 0.480 | 0.646 | 0.297 | 0.477
| None | 0.134 | 0.302 | 0.154 | 0.307 | 0.148 | 0.327 | 0.231 | 0.354 | 0.212 | 0.338 | 0.202 | 0.368 | 0.147 | 0.373
| Over-Sampling | 0.243 | 0.424 | 0.200 | 0.379 | 0.211 | 0.465 | 0.256 | 0.462 | 0.247 | 0.412 | 0.212 | 0.438 | 0.168 | 0.425
| Under-Sampling | 0.140 | 0.293 | 0.140 | 0.290 | 0.142 | 0.313 | 0.224 | 0.430 | 0.230 | 0.410 | 0.218 | 0.393 | 0.130 | 0.322
| Insertion | 0.114 | 0.374 | 0.155 | 0.296 | 0.138 | 0.305 | 0.226 | 0.446 | 0.195 | 0.452 | 0.193 | 0.440 | 0.174 | 0.400
| Deletion | 0.167 | 0.387 | 0.145 | 0.296 | 0.149 | 0.328 | 0.225 | 0.428 | 0.206 | 0.445 | 0.180 | 0.429 | 0.174 | 0.400
| Substitution | 0.161 | 0.398 | 0.157 | 0.302 | 0.139 | 0.306 | 0.209 | 0.453 | 0.186 | 0.451 | 0.197 | 0.421 | 0.183 | 0.382
| Synonym Replacement | 0.215 | 0.408 | 0.196 | 0.317 | 0.139 | 0.307 | 0.221 | 0.419 | 0.215 | 0.452 | 0.208 | 0.413 | 0.170 | 0.394
| Combination | 0.219 | 0.417 | 0.198 | 0.323 | 0.167 | 0.368 | 0.240 | 0.441 | 0.220 | 0.461 | 0.221 | 0.443 | 0.189 | 0.401
| Back Translation | 0.037 | 0.064 | 0.035 | 0.090 | 0.030 | 0.052 | 0.039 | 0.076 | 0.051 | 0.059 | 0.048 | 0.098 | 0.021 | 0.063
RF + Doc2Vec | Paraphrasing | 0.196 | 0.313 | 0.148 | 0.376 | 0.148 | 0.326 | 0.195 | 0.384 | 0.208 | 0.381 | 0.212 | 0.419 | 0.095 | 0.358
| None | 0.578 | 0.573 | 0.613 | 0.589 | 0.465 | 0.691 | 0.574 | 0.685 | 0.592 | 0.691 | 0.540 | 0.660 | 0.338 | 0.535
| Over-Sampling | 0.584 | 0.570 | 0.611 | 0.544 | 0.410 | 0.453 | 0.556 | 0.677 | 0.561 | 0.688 | 0.552 | 0.673 | 0.336 | 0.559
| Under-Sampling | 0.491 | 0.583 | 0.445 | 0.508 | 0.183 | 0.384 | 0.560 | 0.681 | 0.581 | 0.688 | 0.547 | 0.670 | 0.358 | 0.572
| Insertion | 0.620 | 0.597 | 0.630 | 0.564 | 0.437 | 0.678 | 0.580 | 0.681 | 0.610 | 0.708 | 0.572 | 0.681 | 0.352 | 0.556
| Deletion | 0.594 | 0.593 | 0.641 | 0.564 | 0.474 | 0.701 | 0.584 | 0.676 | 0.607 | 0.702 | 0.564 | 0.683 | 0.358 | 0.570
| Substitution | 0.627 | 0.620 | 0.620 | 0.554 | 0.491 | 0.732 | 0.571 | 0.690 | 0.612 | 0.699 | 0.559 | 0.677 | 0.372 | 0.572
| Synonym Replacement | 0.619 | 0.600 | 0.633 | 0.546 | 0.489 | 0.714 | 0.582 | 0.694 | 0.601 | 0.701 | 0.569 | 0.681 | 0.334 | 0.558
| Combination | 0.633 | 0.627 | 0.646 | 0.629 | 0.479 | 0.701 | 0.586 | 0.701 | 0.613 | 0.712 | 0.574 | 0.684 | 0.374 | 0.589
| Back Translation | 0.327 | 0.405 | 0.362 | 0.357 | 0.270 | 0.299 | 0.329 | 0.411 | 0.388 | 0.471 | 0.335 | 0.392 | 0.180 | 0.366
CNN | Paraphrasing | 0.536 | 0.606 | 0.570 | 0.540 | 0.400 | 0.461 | 0.528 | 0.656 | 0.576 | 0.686 | 0.513 | 0.644 | 0.299 | 0.530
| None | 0.585 | 0.592 | 0.589 | 0.532 | 0.500 | 0.700 | 0.570 | 0.672 | 0.583 | 0.683 | 0.544 | 0.659 | 0.342 | 0.539
| Over-Sampling | 0.600 | 0.579 | 0.617 | 0.544 | 0.401 | 0.446 | 0.585 | 0.699 | 0.571 | 0.687 | 0.549 | 0.671 | 0.343 | 0.565
| Under-Sampling | 0.492 | 0.574 | 0.450 | 0.514 | 0.135 | 0.359 | 0.541 | 0.671 | 0.572 | 0.687 | 0.546 | 0.668 | 0.344 | 0.561
| Insertion | 0.647 | 0.656 | 0.648 | 0.555 | 0.508 | 0.728 | 0.590 | 0.699 | 0.596 | 0.693 | 0.555 | 0.674 | 0.364 | 0.583
| Deletion | 0.634 | 0.632 | 0.639 | 0.552 | 0.479 | 0.708 | 0.580 | 0.686 | 0.612 | 0.700 | 0.568 | 0.681 | 0.353 | 0.581
| Substitution | 0.638 | 0.650 | 0.644 | 0.568 | 0.465 | 0.704 | 0.586 | 0.696 | 0.609 | 0.708 | 0.558 | 0.680 | 0.345 | 0.568
| Synonym Replacement | 0.637 | 0.634 | 0.663 | 0.647 | 0.514 | 0.741 | 0.589 | 0.689 | 0.604 | 0.697 | 0.563 | 0.685 | 0.381 | 0.600
| Combination | 0.650 | 0.675 | 0.633 | 0.605 | 0.522 | 0.743 | 0.592 | 0.700 | 0.619 | 0.711 | 0.580 | 0.686 | 0.369 | 0.596
| Back Translation | 0.236 | 0.261 | 0.254 | 0.244 | 0.189 | 0.229 | 0.268 | 0.294 | 0.267 | 0.295 | 0.217 | 0.299 | 0.139 | 0.259
LSTM | Paraphrasing | 0.573 | 0.617 | 0.591 | 0.568 | 0.416 | 0.467 | 0.537 | 0.664 | 0.576 | 0.686 | 0.541 | 0.660 | 0.311 | 0.551
Avg. % of Best Improvements | 31.8 | 21.4 | 12.9 | 16.7 | 22.4 | 24.1 | 5.3 | 10.4 | 7.2 | 11.0 | 7.0 | 7.7 | 15.5 | 13.8
$p$-value of Best Improvements | 1.3e-4 | 2.3e-7 | 8.5e-7 | 3.4e-7 | 1.1e-3 | 2.7e-5 | 6.2e-3 | 1.6e-3 | 3.3e-3 | 5.1e-3 | 4.3e-5 | 8.5e-6 | 4.2e-7 | 2.3e-7
Effect size of Best Improvements | 0.501 | 0.808 | 0.244 | 0.711 | 0.433 | 0.630 | 0.148 | 0.373 | 0.143 | 0.358 | 0.221 | 0.292 | 0.431 | 0.870

###  4.1. RQ1: Significance of Mitigating Data Imbalance for SV Assessment
Tasks

As shown in Table 2, addressing the data imbalance issue using data
augmentation techniques could substantially improve the predictive performance
of all the seven CVSS-based assessment tasks. The best models using data
augmentation, averaging all four model types in Section 3.4, produced
5.3–31.8% better MCC values and 7.7–24.1% higher F1-Score values for the seven
metrics than the baseline models without data augmentation. We also confirmed
that the best models using data augmentation were statistically significantly
better than the respective baselines in terms of both MCC and F1-Score, based
on the non-parametric Wilcoxon signed-rank tests (Wilcoxon, 1992) with
$p$-values $<$ 0.01 and non-negligible effect sizes ($r\geq 0.1$), as shown in
the last two rows of Table 2. The significant improvements in performance of
data augmentation highlight the importance of mitigating the data imbalance
issue for the SV assessment tasks.

Regarding the performance of individual CVSS metrics, data augmentation was
particularly effective for Access Vector, Access Complexity, Authentication,
and Severity. Notably, the improvement of using data augmentation over the
baseline for Access Vector could go up to 81.3% in MCC and 40.5% in F1-Score,
i.e., using RF + Doc2Vec with Over-Sampling. This finding can be explained by
the fact that these three metrics had the smallest size of the minority
classes (with the fewest samples, as depicted in Fig. 3. It is worth noting
that Authentication did not always have the highest improvement value despite
having the smallest minority class, i.e., Multiple. This was mostly because
the Multiple class did not appear in all the evaluation rounds, and the
impacts of data augmentation were also attributed to the Medium class, which
was larger than the minority classes of Access Vector, Access Complexity, and
Severity. The Impact metrics (Confidentiality, Integrity, and Availability)
also benefited less from data augmentation than the other metrics. This result
is likely because these three metrics had the least imbalances in the data
classes among the CVSS metrics (see Fig. 3).

RQ1 Summary. Mitigating data imbalance can have significantly positive impacts
on the SV assessment models. Data augmentation improves the baseline
predictive performance of all the seven CVSS metrics, with increases of
5.3–31.8% in MCC and 7.7–24.1% in F1-Score. Exploitability and Severity CVSS
metrics exhibit more performance gains with data augmentation than the Impact
metrics, likely because of the higher degrees of data imbalance.

Figure 4. Percentage (%) differences in testing SV assessment performance
(F1-Score and MCC) between using different data augmentation techniques and
the baseline (without data augmentation) across different model types.

###  4.2. RQ2: Performance of Individual Data Augmentation Techniques

Expanding upon the overall improvement of data augmentation over the baseline
in RQ1, the RQ2 results showed that more than half (6/9) of the studied
augmentation techniques were better than the None baseline case (see Fig. 4).
On average, the outperforming data augmentation techniques were Simple Text
Augmentation (Combination, Synonym Replacement, Insertion, Deletion, and
Substitution) and Random Over-Sampling. The performance analysis of individual
data augmentation techniques is presented hereafter.

The Combination technique had the highest average performance among the data
augmentation techniques across the four studied models. Combination, on
average, improved the baseline by 11.3% in MCC ($p$-value = 3.7e-9, $r$ =
0.788) and 11.5% in F1-Score ($p$-value = 7.5e-9, $r$ = 0.772), as shown in
Table 2. Particularly, Table 2 shows that Combination was the best data
augmentation technique (i.e., the models with the highest MCC values) for 5/7
CVSS metrics, i.e., Access Vector, Authentication, Confidentiality, Integrity,
and Availability. This finding shows that such simple textual modifications
that have been successful for other text classification tasks (Wei and Zou,
2019) are also helpful for augmenting SV descriptions and improving the
performance in classifying the CVSS metrics.

Among the techniques used in Combination, Synonym Replacement was the best-
performing operation across different model types, except CNN (see Fig. 4).
This technique was also the best data augmentation technique for Access
Complexity and Severity. In addition, we found that Insertion, together with
Synonym Replacement, contributed more to overall performance improvements of
Combination, discussed in the previous paragraph, than Deletion and
Substitution. Synonym Replacement and Insertion kept original words unchanged
and/or replaced the words with others having similar meanings. On the other
hand, Deletion and Substitution can alter the meaning more by removing
(important) information/words in SV descriptions. Thus, Synonym Replacement
and Insertion are more likely to keep the original meaning for augmented
descriptions than Deletion and Substitution. Despite that, Deletion and
Substitution still provided higher performance improvements than without using
data augmentation.

Random Over-Sampling, though simple, still proved its usefulness for boosting
the average SV assessment performance, on par with Synonym Replacement. This
technique was particularly effective for the ML models (i.e., RF + TF-IDF and
RF + Doc2Vec), yet was not as much for the DL models (i.e., CNN and LSTM). In
addition, we discovered that Random Over-Sampling was better than Random
Under-Sampling. Given that Under-Sampling has been previously used for SV
assessment (Han et al., 2017), our finding identifies and emphasizes the sub-
optimality of Random Under-Sampling. Instead, we suggest that Random Over-
Sampling or Simple Text Augmentation techniques should be used for the SV
assessment tasks for improved performance.

Despite incorporating the context of whole SV descriptions, Contextual Text
Augmentation (Back Translation and Paraphrasing) could not improve the
performance of the SV assessment models. Through a closer inspection, we found
that these two techniques at times could not properly comprehend important and
software/SV-specific words in many of the augmented SV descriptions,
potentially leading to information loss or semantic changes in the
descriptions during model training. For instance, the word “passwd” (command
to change password) was changed to the general “password” by both Back
Translation and GPT-based Paraphrasing for the description “Buffer overflow in
passwd in BSD based operating systems 4.3 and earlier allows local users to
gain root privileges by specifying a long shell or GECOS field.” of
CVE-1999-1471. Such change could have obscured the information about the
location of the SV. It is important to note that it is non-trivial to
automatically identify such words to preserve. Automatic preservation of
security/SV-specific words in SV descriptions may be an interesting direction
to explore in the future.

We also discovered that the best data augmentation technique also varied for
each of the four studied types of SV assessment models, as shown in Fig. 4. In
terms of MCC averaging across seven SV assessment tasks/metrics, RF + TF-IDF
and RF + Doc2Vec achieved the best improvements of 10.1% and 25.1% with Random
Over-Sampling, compared to the optimal improvements of 5.5% for CNN and 6.8%
for LSTM when combined with the Combination technique. The MCC performance
gains of data augmentation for RF + TF-IDF, RF + Doc2Vec, CNN, and LSTM have
been confirmed statistically significant using the non-parametric Wilcoxon
signed-rank tests (Wilcoxon, 1992) with $p$-values of 1e-4, 2.2e-12, 9.5e-4,
1.4e-5 and medium/large effect sizes of 0.471, 1.36, 0.341, and 0.388,
respectively. We also observed similar improvements in terms of F1-Score for
the four model types when coupled with the same respective data augmentation
techniques. The discrepancies in performance gains between ML and DL can be
attributed to the different nature of feature representation and learning of
these two model types. The DL-based models partially (CNN) or fully (LSTM)
extracted the word sequence of SV descriptions, which can better capture the
structure and semantics of the descriptions than the ML models, e.g., changing
the word order would affect the DL models more than the ML ones. Thus, the
performance of DL would be more sensitive to the textual changes, e.g., by the
Combination and Synonym Replacement data augmentation techniques, compared to
just repeating the same features multiple times by Random Over-Sampling.

RQ2 Summary. Among the studied Data Augmentation (DA) techniques, Combination
(randomly inserting, deleting, and replacing text) performs the best with an
average MCC improvement of 11.3% over the baseline. Synonym Replacement,
Random Over-Sampling, and Insertion are also substantially better than the
baseline. Random Under-Sampling and Contextual Text Augmentation, i.e., Back
Translation and Paraphrasing, are worse than the baseline, probably due to
missing information and/or lacking semantic understanding of software/SV-
specific words. Task-wise. The Combination technique is the best for Access
Vector, Authentication, Confidentiality, Integrity, and Availability. Synonym
Replacement is the optimal technique for the Access Complexity and Severity.
Model-wise. The best DA techniques for DL and ML are Combination and Random
Over-Sampling, respectively, showing that DL benefits more from textual
changes than ML. These techniques improve the MCC values of the Machine
Learning (ML) and Deep Learning (DL) models by 10.1–25.1% and 5.5–6.8%,
respectively.

##  5\. Discussion

###  5.1. Advancing Data-Driven SV Assessment: Data Augmentation and Beyond

In recent years, data-driven approaches have been increasingly used for SV
assessment. One of the main goals in the field is to increase the performance
of developed models (Le et al., 2022). Our study findings in Section 4 have
highlighted data augmentation as an effective approach to achieving this goal
for all of the CVSS tasks. Furthermore, the studied data augmentation
techniques work directly on the input data and independently of underlying
models. Thus, they can be seamlessly integrated with and potentially enhance
the performance of almost any SV assessment models without changing the model
architectures, ranging from the existing well-known models (RF, CNN, and LSTM)
to newly proposed models in the future.

We recommend that Combination (i.e., combining Word Insertion, Deletion, and
Substitution/Replacement) should be considered as a baseline of data
augmentation for SV assessment in the future because this technique has been
shown to improve the performance across the board. The demonstrated
effectiveness suggests that Combination-augmented descriptions, despite having
textual changes, can still retain the semantics/label of the respective
original descriptions. Following the prior studies (Tian et al., 2012; Wei and
Zou, 2019; Le et al., 2020b), we set out to validate this conjecture by
comparing the cosine similarities between the feature vectors of the augmented
descriptions to the centroids (average feature vectors of the original
descriptions) of each class CVSS-metric-wise. For each metric, the features
were extracted from the optimal model trained with the Combination data
augmentation technique. Table 3 shows that the augmented descriptions were
indeed more similar to those of the original class than the other classes for
all the metrics. Such results can increase the confidence in using this data
augmentation technique for SV assessment as it mainly makes structural rather
than semantic changes, i.e., often retaining the original label.

Table 3. Average cosine similarities between the Combination-augmented
descriptions and the original descriptions of the same and other classes for
each of the CVSS metrics. Note: The Other cells are the maximum values among
the other classes. Higher value is better.

Sim. | CVSS Metrics | Avg.
---|---|---
AV | AC | Au | C | I | A | S
Same | 0.223 | 0.201 | 0.171 | 0.288 | 0.305 | 0.420 | 0.262 | 0.267
Other | 0.208 | 0.184 | 0.151 | 0.275 | 0.293 | 0.408 | 0.250 | 0.252

Despite the success of data augmentation techniques for SV assessment, there
is still room for improvement for these techniques and SV assessment as well.
We analyzed 3,668 SVs in the testing sets where the optimal models (with the
highest testing MCC values) trained with and without augmented SV descriptions
could not correctly predict for all of the seven CVSS metrics. These cases
showed the scenarios where data augmentation consistently struggled to provide
meaningful improvement to CVSS assessment models. From the analysis, we
identified a common pattern of these incorrect cases. Data augmentation had
difficulty in improving the assessment performance when the input SV
description was short/uninformative. The average number of words in these
problematic descriptions was only 16 compared to 28 in all SV descriptions in
the testing sets. An example was “static/js/pad_utils.js in Etherpad Lite
before v1.6.3 has XSS via window.location.href.” – the description of
CVE-2018-6834. We posit that these cases lack the information/words about some
characteristics of SVs, e.g., SV impact in the presented example. Moreover,
many of the words were software-specific terms such as
“static/js/pad_utils.js” or “window.location.href”. If the words of such
descriptions were randomly removed or replaced, which are the key operations
of the Combination data augmentation technique, the information loss would be
further increased. Such words also did not appear in the WordNet, making
Synonym Replacement struggle to find a suitable synonym. In addition, such
keywords could not (yet) be effectively comprehended by Contextual data
augmentation, i.e., Back Translation or Paraphrasing. In the future, multiple
sources such as social media sites and/or external security advisories can be
leveraged to provide more informative descriptions of such SVs. Still, more
research is required to automatically validate the relevance and
trustworthiness of the externally gathered information (Anwar et al., 2021).

###  5.2. Threats to Validity

Internal validity. A possible threat here is that our optimal models may not
guarantee the highest performance for SV assessment. However, we assert that
it is nearly impossible to achieve this because there are infinite values of
hyperparameters of the models to tune. Our study may not provide the best
possible results for SV assessment; however, it still highlights the benefits
of using data augmentation for handling the data imbalance issue of the tasks
and provides the baseline performance of SV assessment with data augmentation
for future research to build upon.

External validity. Our work may not generalize to all SVs. We tried to
mitigate the threat by using NVD – one of the most comprehensive repositories
of SVs. Our dataset contained more than 180k SVs, ranging from 1988 to 2023.
There is also a potential concern about the generalizability of our findings
to other SV assessment models. We mitigated this threat by investigating the
four most commonly used baseline models for SV assessment, which are expected
to provide direct contributions to the current practices of the field. We also
release our data, code, and models at (Authors, [n. d.]) for future research
to replicate our study on new SV data and models.

Conclusion validity. We mitigated the randomness of the results by taking the
average value of multi-round time-based evaluation. The key performance
comparisons of different SV assessment models with and without data
augmentation were also confirmed using the non-parametric Wilcoxon signed-
ranked tests with $p$-values $<$ 0.01 and non-negligible values of the effect
size.

##  6\. Related Work

###  6.1. Data-Driven SV Assessment and Analysis

SV assessment is a crucial process in dealing with SVs, and CVSS offers one of
the most dependable metrics for such assessment (Johnson et al., 2016). Prior
research delved into analyzing CVSS metrics and SV trends by integrating
diverse SV data from multiple SV repositories (Murtaza et al., 2016;
Almukaynizi et al., 2019), security advisories (Edkrantz et al., 2015; Huang
et al., 2013), dark web sources (Almukaynizi et al., 2019; Nunes et al.,
2016), and social networks like Twitter/X (Sabottke et al., 2015). However,
these studies operated under the assumption that all CVSS metrics were
available during the analysis, which has been shown to be unrealistic in real-
world settings (Gong et al., 2019). In contrast, our work deviates from this
assumption by utilizing solely the SV descriptions, making our method more
adaptable and suitable for both new and older SVs.

Bozorgi et al. (Bozorgi et al., 2010) were among the first to employ data-
driven models for SV assessment utilizing solely SV descriptions as inputs.
The authors used an SVM model and several attributes such as NVD description,
CVSS, and publication dates, to gauge the likelihood of exploitation and time-
to-exploit of SVs. This pioneering work sparked a substantial volume of
subsequent research aimed at automating SV assessment tasks using data-driven
models (Le et al., 2022). Numerous recent studies (e.g., (Yamamoto et al.,
2015; Spanos et al., 2017; Spanos and Angelis, 2018; Le et al., 2019; Elbaz et
al., 2020; Duan et al., 2021)) have drawn on SV descriptions found in bug/SV
reports/databases, mainly NVD, to predict the CVSS metrics for ever-increasing
SVs.

Although these studies have demonstrated the promising use of ML/DL for SV
assessment, they hardly addressed the inherent data imbalance problem of the
tasks. To the best of our knowledge, Han et al. (Han et al., 2017) were the
only prior study that attempted to tackle the problem using Random Under-
Sampling. However, our results in Section 4 have shown that Random Under-
Sampling was sub-optimal for SV assessment and could even reduce the
performance of the prediction models. Furthermore, our study is the first to
show the potential of other data augmentation techniques, such as combining
text insertion, deletion, and substitution/replacement, to effectively
mitigate data imbalance for SV assessment.

###  6.2. Data Augmentation for Data-Driven Software Engineering Tasks

Data augmentation has witnessed growing success in the Natural Language
Processing domain in recent years (Feng et al., 2021). Such success has then
inspired the increasing use of data augmentation in the Software Engineering
(SE) domain, mainly because many software artifacts are in the form of text
(Zhuo et al., 2023). So far, data augmentation has been applied to a wide
range of automated SE tasks such as code clone detection (Pinku et al., 2023;
Liu et al., 2023), defect prediction (Allamanis et al., 2021), code
summarization (Zhang et al., 2020), and code question answering (Huang et al.,
2021; Park et al., 2023). These studies have revealed the advantages of
augmented data to address the data imbalance/scarcity and overfitting issues
of the respective tasks. However, there is much less work on utilizing data
augmentation for software security, particularly SV assessment and analysis –
an integral step in secure software development. Our work aims to contribute
to the body of knowledge in this emerging area by showing the possible
benefits and use of data augmentation for SV assessment. Our promising results
in Section 4 can inspire future work to investigate more sophisticated data
augmentation techniques for SV assessment tasks. While current approaches
mainly leverage SV reports for SV assessment as shown in Section 6.1, future
research can also explore code-based data augmentation techniques to
complement the text-based techniques investigated in this study with source
code for predicting SV assessment metrics.

##  7\. Conclusion

We highlighted the importance of mitigating data imbalance for SV assessment.
We investigated the effectiveness of addressing the issue for different SV
assessment tasks using nine data augmentation techniques. Through extensive
experiments on 180k+ real-world SVs, we showed that data augmentation could
improve the performance of the models without data augmentation by up to 31.8%
in MCC and 24.1% in F1-Score, particularly the Exploitability and Severity
CVSS metrics. Among the data augmentation techniques, we found that combining
simple textual operations, including random text insertion, deletion, and
substitution/replacement, achieved the best performance improvements over the
baseline. Our study encourages further investigations into better data
augmentation for SV assessment, particularly the techniques that can
comprehend software/SV-specific words in SV descriptions.

## Acknowledgments

The work has been supported by the Cyber Security Research Centre Limited
whose activities are partially funded by the Australian Government’s
Cooperative Research Centres Program. We would like to acknowledge Tuan Lu
Dinh for his technical support at the early stage of this work.

## References

  * (1)
  * Allamanis et al. (2021) Miltiadis Allamanis, Henry Jackson-Flux, and Marc Brockschmidt. 2021.  Self-supervised bug detection and repair.  _Advances in Neural Information Processing Systems_ 34 (2021), 27865–27876.
  * Almukaynizi et al. (2019) Mohammed Almukaynizi, Eric Nunes, Krishna Dharaiya, Manoj Senguttuvan, Jana Shakarian, and Paulo Shakarian. 2019.  Patch before exploited: An approach to identify targeted software vulnerabilities.  In _AI in Cybersecurity_. Springer, 81–113.
  * Anand et al. (2023) Yuvanesh Anand, Zach Nussbaum, Brandon Duderstadt, Benjamin Schmidt, and Andriy Mulyar. 2023.  Gpt4all: Training an assistant-style chatbot with large scale data distillation from gpt-3.5-turbo.  _GitHub_ (2023).
  * Anwar et al. (2021) Afsah Anwar, Ahmed Abusnaina, Songqing Chen, Frank Li, and David Mohaisen. 2021.  Cleaning the NVD: Comprehensive quality assessment, improvements, and analyses.  _IEEE Transactions on Dependable and Secure Computing_ (2021).
  * Arani et al. (2024) Ali Kazemi Arani, Triet Huynh Minh Le, Mansooreh Zahedi, and M Ali Babar. 2024.  Systematic literature review on application of learning-based approaches in continuous integration.  _IEEE Access_ (2024).
  * Authors ([n. d.]) Authors. [n. d.].  Reproduction package.  https://github.com/lhmtriet/DA4SVA
  * Bayer et al. (2022) Markus Bayer, Marc-André Kaufhold, and Christian Reuter. 2022.  A survey on data augmentation for text classification.  _ACM Computing Surveys (CSUR)_ 55, 7 (2022), 1–39.
  * ben Othmane et al. (2015) Lotfi ben Othmane, Golriz Chehrazi, Eric Bodden, Petar Tsalovski, Achim D Brucker, and Philip Miseldine. 2015.  Factors impacting the effort required to fix security vulnerabilities. In _International Conference on Information Security_. Springer, 102–119.
  * Bozorgi et al. (2010) Mehran Bozorgi, Lawrence K Saul, Stefan Savage, and Geoffrey M Voelker. 2010.  Beyond heuristics: learning to classify vulnerabilities and predict exploits. In _the 16th ACM SIGKDD international conference on Knowledge discovery and data mining_. 105–114.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Chawla et al. (2002) Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002.  SMOTE: synthetic minority over-sampling technique.  _Journal of Artificial Intelligence Research_ 16 (2002), 321–357.
  * Conversation ([n. d.]) The Conversation. [n. d.].  What is Log4j?  https://bit.ly/log4j_the_conversation
  * Cornell (2012) Dan Cornell. 2012.  Remediation statistics: what does fixing application vulnerabilities cost.  _Proceedings of the RSAConference, San Fransisco, CA, USA_ (2012).
  * Dai et al. (2023) Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Zihao Wu, Lin Zhao, Wei Liu, Ninghao Liu, Sheng Li, Dajiang Zhu, et al. 2023\.  Chataug: Leveraging chatgpt for text data augmentation.  _arXiv preprint arXiv:2302.13007_ (2023).
  * Duan et al. (2021) Xuanyu Duan, Mengmeng Ge, Triet Huynh Minh Le, Faheem Ullah, Shang Gao, Xuequan Lu, and M Ali Babar. 2021.  Automated security assessment for the Internet of Things. In _2021 IEEE 26th Pacific Rim International Symposium on Dependable Computing (PRDC)_. IEEE, 47–56.
  * Edkrantz et al. (2015) Michel Edkrantz, Staffan Truvé, and Alan Said. 2015.  Predicting vulnerability exploits in the wild. In _2015 IEEE 2nd International Conference on Cyber Security and Cloud Computing_. IEEE, 513–514.
  * Elbaz et al. (2020) Clément Elbaz, Louis Rilling, and Christine Morin. 2020.  Fighting N-day vulnerabilities with automated CVSS vector prediction at disclosure. In _the 15th International Conference on Availability, Reliability and Security_. 1–10.
  * Feng et al. (2021) Steven Y Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021.  A survey of data augmentation approaches for NLP.  _arXiv preprint arXiv:2105.03075_ (2021).
  * Feutrill et al. (2018) Andrew Feutrill, Dinesha Ranathunga, Yuval Yarom, and Matthew Roughan. 2018.  The effect of common vulnerability scoring system metrics on vulnerability exploit delay. In _2018 Sixth International Symposium on Computing and Networking (CANDAR)_. IEEE, 1–10.
  * Field (2013) Andy Field. 2013.  _Discovering statistics using IBM SPSS statistics_.  sage.
  * FIRST ([n. d.]a) FIRST. [n. d.]a.  Common Vulnerability Scoring System.  https://www.first.org/cvss
  * FIRST ([n. d.]b) FIRST. [n. d.]b.  CVSS version 2.  https://www.first.org/cvss/v2/guide
  * Foreman (2019) Park Foreman. 2019.  _Vulnerability management_.  CRC Press.
  * Fu et al. (2023) Michael Fu, Chakkrit Kla Tantithamthavorn, Van Nguyen, and Trung Le. 2023.  Chatgpt for vulnerability detection, classification, and repair: How far are we?. In _2023 30th Asia-Pacific Software Engineering Conference (APSEC)_. IEEE, 632–636.
  * Future ([n. d.]) Recorded Future. [n. d.].  Exploiting old vulnerabilities.  https://www.recordedfuture.com/exploiting-old-vulnerabilities/
  * Ghaffarian and Shahriari (2017) Seyed Mohammad Ghaffarian and Hamid Reza Shahriari. 2017.  Software vulnerability analysis and discovery using machine-learning and data-mining techniques: A survey.  _ACM Computing Surveys (CSUR)_ 50, 4 (2017), 1–36.
  * Gong et al. (2019) Xi Gong, Zhenchang Xing, Xiaohong Li, Zhiyong Feng, and Zhuobing Han. 2019.  Joint prediction of multiple vulnerability characteristics through multi-task learning. In _2019 24th International Conference on Engineering of Complex Computer Systems (ICECCS)_. IEEE, 31–40.
  * Gorodkin (2004) Jan Gorodkin. 2004.  Comparing two K-category assignments by a K-category correlation coefficient.  _Computational Biology and Chemistry_ 28, 5-6 (2004), 367–374.
  * Han et al. (2017) Zhuobing Han, Xiaohong Li, Zhenchang Xing, Hongtao Liu, and Zhiyong Feng. 2017.  Learning to predict severity of software vulnerability using only vulnerability description. In _2017 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE, 125–136.
  * Ho (1995) Tin Kam Ho. 1995.  Random decision forests. In _3rd international conference on document analysis and recognition_ , Vol. 1. IEEE, 278–282.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Huang et al. (2013) Chien-Cheng Huang, Feng-Yu Lin, Frank Yeong-Sung Lin, and Yeali S Sun. 2013.  A novel approach to evaluate software vulnerability prioritization.  _Journal of Systems and Software_ 86, 11 (2013), 2822–2840.
  * Huang et al. (2021) Junjie Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, and Nan Duan. 2021.  CoSQA: 20,000+ web queries for code search and question answering. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, Online, 5690–5700.  https://doi.org/10.18653/v1/2021.acl-long.442
  * Jimenez et al. (2019) Matthieu Jimenez, Renaud Rwemalika, Mike Papadakis, Federica Sarro, Yves Le Traon, and Mark Harman. 2019.  The importance of accounting for real-world labelling when predicting software vulnerabilities. In _2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 695–705.
  * Johnson et al. (2016) Pontus Johnson, Robert Lagerström, Mathias Ekstedt, and Ulrik Franke. 2016.  Can the common vulnerability scoring system be trusted? a bayesian analysis.  _IEEE Transactions on Dependable and Secure Computing_ 15, 6 (2016), 1002–1015.
  * Kamongi et al. (2013) Patrick Kamongi, Srujan Kotikela, Krishna Kavi, Mahadevan Gomathisankaran, and Anoop Singhal. 2013.  Vulcan: Vulnerability assessment framework for cloud computing. In _2013 IEEE 7th International Conference on Software Security and Reliability_. IEEE, 218–226.
  * Kanakogi et al. (2021) Kenta Kanakogi, Hironori Washizaki, Yoshiaki Fukazawa, Shinpei Ogata, Takao Okubo, Takehisa Kato, Hideyuki Kanuka, Atsuo Hazeyama, and Nobukazu Yoshioka. 2021.  Tracing CAPEC attack patterns from CVE vulnerability information using natural language processing technique. In _the 54th Hawaii International Conference on System Sciences_. 6996.
  * Kaur et al. (2019) Harsurinder Kaur, Husanbir Singh Pannu, and Avleen Kaur Malhi. 2019.  A systematic review on imbalanced data challenges in machine learning: Applications and solutions.  _ACM Computing Surveys (CSUR)_ 52, 4 (2019), 1–36.
  * Khan and Parkinson (2018) Saad Khan and Simon Parkinson. 2018.  Review into state of the art of vulnerability assessment using artificial intelligence.  In _Guide to Vulnerability Analysis for Computer Networks and Systems_. Springer, 3–32.
  * Kim (2014) Yoon Kim. 2014.  Convolutional neural networks for sentence classification. In _the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, 1746–1751.
  * Le and Mikolov (2014) Quoc Le and Tomas Mikolov. 2014.  Distributed representations of sentences and documents. In _International conference on machine learning_. PMLR, 1188–1196.
  * Le (2022) Triet HM Le. 2022.  Towards an improved understanding of software vulnerability assessment using data-driven approaches.  _arXiv preprint arXiv:2207.11708_ (2022).
  * Le et al. (2022) Triet HM Le, Huaming Chen, and M Ali Babar. 2022.  A survey on data-driven software vulnerability assessment and prioritization.  _ACM Computing Surveys (CSUR)_ 55, 5 (2022), 1–39.
  * Le and Babar (2022) Triet Huynh Minh Le and M Ali Babar. 2022.  On the use of fine-grained vulnerable code statements for software vulnerability assessment models. In _Proceedings of the 19th International Conference on Mining Software Repositories_. 621–633.
  * Le et al. (2024a) Triet Huynh Minh Le, M Ali Babar, and Tung Hoang Thai. 2024a.  Software vulnerability prediction in low-resource languages: An empirical study of codebert and chatgpt. In _Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering_. 679–685.
  * Le et al. (2020a) Triet Huynh Minh Le, Hao Chen, and M Ali Babar. 2020a.  Deep learning for source code modeling and generation: Models, applications, and challenges.  _ACM Computing Surveys (CSUR)_ 53, 3 (2020), 1–38.
  * Le et al. (2021a) Triet Huynh Minh Le, Roland Croft, David Hin, and M Ali Babar. 2021a.  A large-scale study of security vulnerability support on developer Q&A websites. In _Evaluation and Assessment in Software Engineering_. 109–118.
  * Le et al. (2024b) Triet Huynh Minh Le, Xiaoning Du, and M Ali Babar. 2024b.  Are Latent Vulnerabilities Hidden Gems for Software Vulnerability Prediction? An Empirical Study. In _2024 IEEE/ACM 21st International Conference on Mining Software Repositories (MSR)_. IEEE, 716–727.
  * Le et al. (2020b) Triet Huynh Minh Le, David Hin, Roland Croft, and M Ali Babar. 2020b.  PUMiner: Mining security posts from developer question and answer websites with PU learning. In _the 17th International Conference on Mining Software Repositories_. 350–361.
  * Le et al. (2021b) Triet Huynh Minh Le, David Hin, Roland Croft, and M Ali Babar. 2021b.  Deepcva: Automated commit-level vulnerability assessment with deep multi-task learning. In _2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 717–729.
  * Le et al. (2019) Triet Huynh Minh Le, Bushra Sabir, and Muhammad Ali Babar. 2019.  Automated software vulnerability assessment with concept drift. In _the 16th International Conference on Mining Software Repositories (MSR)_. IEEE, 371–382.
  * Liu et al. (2023) Shangqing Liu, Bozhi Wu, Xiaofei Xie, Guozhu Meng, and Yang Liu. 2023.  ContraBERT: Enhancing code pre-trained models via contrastive learning. In _2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)_. 2476–2487.  https://doi.org/10.1109/ICSE48619.2023.00207
  * Liu et al. (2022) Yanbin Liu, Wen Zhang, Guangjie Qin, and Jiangpeng Zhao. 2022.  A comparative study on the effect of data imbalance on software defect prediction.  _Procedia Computer Science_ 214 (2022), 1603–1616.
  * Loper and Bird (2002) Edward Loper and Steven Bird. 2002.  NLTK: the natural language toolkit.  _arXiv preprint cs/0205028_ (2002).
  * Luque et al. (2019) Amalia Luque, Alejandro Carrasco, Alejandro Martín, and Ana de las Heras. 2019.  The impact of class imbalance in classification performance metrics based on the binary confusion matrix.  _Pattern Recognition_ 91 (2019), 216–231.
  * Ma (2019) Edward Ma. 2019.  NLP Augmentation.  https://github.com/makcedward/nlpaug.
  * Malhotra and Khanna (2017) Ruchika Malhotra and Megha Khanna. 2017.  An empirical study for software change prediction using imbalanced data.  _Empirical Software Engineering_ 22 (2017), 2806–2851.
  * Miller (1995) George A Miller. 1995.  WordNet: a lexical database for English.  _Commun. ACM_ 38, 11 (1995), 39–41.
  * Mumuni and Mumuni (2022) Alhassan Mumuni and Fuseini Mumuni. 2022.  Data augmentation: A comprehensive survey of modern approaches.  _Array_ (2022), 100258.
  * Murtaza et al. (2016) Syed Shariyar Murtaza, Wael Khreich, Abdelwahab Hamou-Lhadj, and Ayse Basar Bener. 2016.  Mining trends and patterns of software vulnerabilities.  _Journal of Systems and Software_ 117 (2016), 218–228.
  * Nair and Hinton (2010) Vinod Nair and Geoffrey E Hinton. 2010.  Rectified linear units improve restricted boltzmann machines. In _Icml_.
  * NIST ([n. d.]a) NIST. [n. d.]a.  National Vulnerability Database.  https://nvd.nist.gov/
  * NIST ([n. d.]b) NIST. [n. d.]b.  The number of new software vulnerabilities.  https://nvd.nist.gov/general/nvd-dashboard
  * Nunes et al. (2016) Eric Nunes, Ahmad Diab, Andrew Gunn, Ericsson Marin, Vineet Mishra, Vivin Paliath, John Robertson, Jana Shakarian, Amanda Thart, and Paulo Shakarian. 2016.  Darknet and deepnet mining for proactive cybersecurity threat intelligence. In _2016 IEEE Conference on Intelligence and Security Informatics (ISI)_. IEEE, 7–12.
  * Park et al. (2023) Shinwoo Park, Youngwook Kim, and Yo-Sub Han. 2023.  Contrastive learning with keyword-based data augmentation for code search and code question answering. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_. 3591–3601.
  * Pedregosa et al. (2011) Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011\.  Scikit-learn: Machine learning in Python.  _the Journal of machine Learning research_ 12 (2011), 2825–2830.
  * Pinku et al. (2023) Subroto Nag Pinku, Debajyoti Mondal, and Chanchal K. Roy. 2023.  Pathways to leverage transcompiler based data augmentation for cross-language clone detection. In _2023 IEEE/ACM 31st International Conference on Program Comprehension (ICPC)_. 169–180.  https://doi.org/10.1109/ICPC58990.2023.00031
  * Porter (1980) Martin F Porter. 1980.  An algorithm for suffix stripping.  _Program_ 14, 3 (1980), 130–137.
  * Raschka (2018) Sebastian Raschka. 2018.  Model evaluation, model selection, and algorithm selection in machine learning.  _arXiv preprint arXiv:1811.12808_ (2018).
  * Rodriguez et al. (2014) Daniel Rodriguez, Israel Herraiz, Rachel Harrison, Javier Dolado, and José C Riquelme. 2014.  Preliminary comparison of techniques for dealing with imbalance in software defect prediction. In _Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering_. 1–10.
  * Sabottke et al. (2015) Carl Sabottke, Octavian Suciu, and Tudor Dumitraș. 2015.  Vulnerability disclosure in the age of social media: Exploiting twitter for predicting real-world exploits. In _24th $\\{$USENIX$\\}$ Security Symposium_. 1041–1056.
  * Sahin and Tosun (2019) Sefa Eren Sahin and Ayse Tosun. 2019.  A conceptual replication on predicting the severity of software vulnerabilities.  In _the Evaluation and Assessment on Software Engineering_. 244–250.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Improving neural machine translation models with monolingual data. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Berlin, Germany, 86–96.
  * Sharma et al. (2021) Ruchi Sharma, Ritu Sibal, and Sangeeta Sabharwal. 2021.  Software vulnerability prioritization using vulnerability description.  _International Journal of System Assurance Engineering and Management_ 12, 1 (2021), 58–64.
  * Smyth (2017) Vincent Smyth. 2017.  Software vulnerability management: how intelligence helps reduce the risk.  _Network Security_ 2017, 3 (2017), 10–12.
  * Song et al. (2018) Qinbao Song, Yuchen Guo, and Martin Shepperd. 2018.  A comprehensive investigation of the role of imbalanced learning for software defect prediction.  _IEEE Transactions on Software Engineering_ 45, 12 (2018), 1253–1269.
  * Spanos and Angelis (2018) Georgios Spanos and Lefteris Angelis. 2018.  A multi-target approach to estimate software vulnerability characteristics and severity scores.  _Journal of Systems and Software_ 146 (2018), 152–166.
  * Spanos et al. (2017) Georgios Spanos, Lefteris Angelis, and Dimitrios Toloudis. 2017.  Assessment of vulnerability severity using text mining. In _the 21st Pan-Hellenic Conference on Informatics_. 1–6.
  * Synopsys ([n. d.]) Inc. Synopsys. [n. d.].  Heartbleed bug.  https://heartbleed.com/
  * Tian et al. (2012) Yuan Tian, Julia Lawall, and David Lo. 2012.  Identifying linux bug fixing patches. In _34th International Conference on Software Engineering (ICSE)_. IEEE, 386–396.
  * Wattanakriengkrai et al. (2020) Supatsara Wattanakriengkrai, Patanamon Thongtanunam, Chakkrit Tantithamthavorn, Hideaki Hata, and Kenichi Matsumoto. 2020.  Predicting defective lines using a model-agnostic technique.  _IEEE Transactions on Software Engineering_ (2020).
  * Wei and Zou (2019) Jason Wei and Kai Zou. 2019.  Eda: Easy data augmentation techniques for boosting performance on text classification tasks.  _arXiv preprint arXiv:1901.11196_ (2019).
  * Wilcoxon (1992) Frank Wilcoxon. 1992.  Individual comparisons by ranking methods.  In _Breakthroughs in Statistics_. Springer, 196–202.
  * Yamamoto et al. (2015) Yasuhiro Yamamoto, Daisuke Miyamoto, and Masaya Nakayama. 2015.  Text-mining approach for estimating vulnerability score. In _2015 4th International Workshop on Building Analysis Datasets and Gathering Experience Returns for Security (BADGERS)_. IEEE, 67–73.
  * Zhang et al. (2017) Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2017.  mixup: Beyond empirical risk minimization.  _arXiv preprint arXiv:1710.09412_ (2017).
  * Zhang et al. (2020) Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, and Xudong Liu. 2020.  Retrieval-based neural source code summarization. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_. 1385–1397.
  * Zhuo et al. (2023) Terry Yue Zhuo, Zhou Yang, Zhensu Sun, Yufei Wang, Li Li, Xiaoning Du, Zhenchang Xing, and David Lo. 2023.  Data augmentation approaches for source code models: A survey.  _arXiv preprint arXiv:2305.19915_ (2023).