OWASP). We correlated such answer types with the question categories of Treude
et al. [386]. We found reasonable matches between answer and question types,
e.g., (dis)agreeing (DC/Co) with a decision (Decision Help), explaining (Ex) a
concept (Conceptual), and providing different solutions to resolve unexpected
situations (Discrepancy). Discrepancy and Error were also among the most
frequent question types, supporting that our posts were about issues/errors in
addressing SVs.

Site-wise answer types. According to Table 6, Action to Take (AT) and External
Sources (ES) were the most common answer types on SO and SSE, respectively;
whereas, Self-Answer (SA) was the least frequent one on both sites. We also
noticed that both sites usually referred to external sources (ES). The most
common sources included Wikipedia, other posts (e.g., related answers), GitHub
issues/commits, product documentation (e.g., PHP, MySQL and Android) and SV
sources (e.g., CVE (Common Vulnerabilities and Exposures) [66], NVD [16], CWE
[28], OWASP [167], CVE Details [187] and Exploit-DB [398]). Note that some
provided links were unavailable or no longer maintained (e.g., CVE Details).
Overall, the answers to SV questions on SO frequently provided detailed
instructions (AT) and/or code samples (CS), while SSE tended to share more
experience (DC/Co and Ex) to help.

Topic-wise answer types. We extend the site-wise findings to individual topics
to enable developers to select the respective site (SO vs. SSE) based on their
preferable SV solution types, as shown in Table 7. Specifically, SO
highlighted steps (AT) to fix Malwares (T1), Memory Allocation Errors (T10)
and XSS (T11) as well as provided code snippets for SQL Injection (T2). On the
other hand, SSE gave more relevant sources and explanations for such topics.
One may argue that these different answer types were because of the different
question types between SO and SSE, but we did not find any such significant
differences for these topics. Instead, the fact that SO and SSE had quite
different accepted answerers, as shown in RQ3, probably led to such different
solution types. There were four topics, namely SV Scanning Tools (T3),
Synchronization & Encryption Errors (T6 & T7), Brute-force/Timing Attacks
(T13), sharing similar top solutions on both SO and SSE. The remaining topics
(T4, T5, T8 and T9) were mostly answered with explanation (Ex) or external
sources (ES) on both SO and SSE.

Table 7: Top-1 answer types of 13 SV topics on SO & SSE (in parenthesis). Note: T8 on SSE was excluded since it did not have any accepted answer. Topic | Top-1 Answer Type | Topic | Top-1 Answer Type
---|---|---|---
T1 | AT/ES (Ex) | T8 | ES (–)
T2 | CS (ES) | T9 | DC/Co/ES/SA (Ex)
T3 | ES (ES) | T10 | AT (Ex)
T4 | ES (DC/Co) | T11 | AT (DC/Co)
T5 | Ex (ES) | T12 | Ex (ES)
T6 | Ex/ES (DC/Co/ES) | T13 | Ex/ES (Ex)
T7 | Ex/ES (Ex) | – | –

###  5 Discussion

####  1 SV Discussion Topics on Q&A Sites vs. Existing Security Taxonomies

SV-specific topics and their support on Q&A sites. Compared to Yang et al.’s
taxonomy [370], we found related topics: T2, T3, T5, T10, T11 and T13, but we
still had the following important differences. Firstly, our topics were
emphasized more on security flaws, e.g., issues with encryption/decryption
algorithms (T7) than how to implement/use them as in [370]. Secondly, we
identified SV-specific topics previously unreported in [370]: Malwares (T1),
CSRF (T4), Synchronization Errors (T6), Resource Leaks (T8), Network Attacks
(T9) and Vulnerability Theory (T12). These SV topics show the necessity of
focusing on SV-specific posts instead of general security ones. Thirdly,
unlike [370], we did not consider language-dependent topics (i.e., PHP, Flash,
Javascript, Java and ASP.NET), helping our topics be more generalizable (e.g.,
XSS can occur in both PHP and ASP.NET). Zahedi et al. [399] also devised a
security taxonomy for GitHub issues; however, they focused on security
features and implementation instead of any specific SV types. Note that we
studied SV posts on both SO and SSE, while the existing studies only used one
source of data (SO), enhancing the generalizability of our study.
Specifically, we shed light on the differences between SV discussion topics on
SO and those on SSE in terms of their proportions (RQ1), popularity/difficulty
(RQ2), level of expertise (RQ3) and types of answers (RQ4). Our findings can
be leveraged to select a suitable site (i.e., more popular/experts, less
difficult or having certain answer types) for asking different SV questions.

Disconnection between SV discussions and expert-based SV sources. Two
researchers (i.e., the author of this thesis and a PhD student with three
years of experience in Software Engineering and Cybersecurity) manually mapped
13 SV topics with CWEs, as shown in Table 8. The agreement between the two
people was strong (Kappa score [345] was 0.892), and disagreements were
resolved during a discussion section with another researcher (i.e., a PhD
student with two-year experience in Software Engineering and Cybersecurity).
We found that only seven of them were overlapping with the two well-known
expert-based SV taxonomies: top-25 CWE666https://cwe.mitre.org/top25/ and
top-10 OWASP777https://owasp.org/www-project-top-ten/. The overlapping topics
were T2 (SQL Injection), T4 (CSRF), T5 (i.e., Path-traversal and Unrestricted
File Upload), T7 (i.e., Improper Certificate Validation), T8 (Resource Leaks),
T10 (Memory Allocation Errors) and T11 (XSS). There was no CWE for T3 and T12
since they mainly discussed SV scanning tools and/or socio-technical issues,
respectively. In fact, using keyword matching, we found that only 159 and 71
out of a total of 839 CWEs were mentioned on SO and SSE, respectively; and
only 20 and two CWEs appeared more than 100 times on SO and SSE, respectively.
Other popular SV sources (i.e., CVE, NVD and OWASP) were also mentioned less
than 10% on these two sites. Moreover, the fast increase of CSRF (T2), as
reported in RQ1, is noteworthy given that this SV type has been removed from
the top-10 OWASP since 2013. Our investigation suggested that many developers
were aware of CSRF standard prevention techniques, but it was not always easy
to apply these techniques and/or use built-in CSRF protection of a web
framework (e.g., Spring Security) in practice. These results imply a strong
disconnection in the SV patterns between expert-based sources and discussions
on Q&A sites, supporting our motivation to study developers’ real-life
concerns in addressing SVs.

Table 8: The mapping between 13 SV topics and their Common Weakness Enumeration (CWE) values. Topic Name | CWE values
---|---
Malwares (T1) | 506-512, 904
SQL Injection (T2) | 20, 74, 89, 707, 943
Vulnerability Scanning Tools (T3) | –
Cross-site Request Forgery (CSRF) (T4) | 352, 1173
File-related Vulnerabilities (T5) | 23, 34-36, 61, 434
Synchronization Errors (T6) | 362, 662, 667, 820, 821, 833
Encryption Errors (T7) | 295, 300, 310
Resource Leaks (T8) | 400, 401, 404, 772
Network Attacks (T9) | 290, 291, 400
Memory Allocation Errors (T10) | 119-127, 787, 822-825, 835
Cross-site Scripting (XSS) (T11) | 20, 79-87, 707
Vulnerability Theory (T12) | –
Brute-force/Timing Attacks (T13) | 208, 261, 307, 385, 799, 916

####  2 Implications for (Data-Driven) SV Assessment

From RQ1, we found that Q&A sites (i.e., SO and SSE) do not report zero-day
SVs, but instead they contain the key SV-related issues that developers have
been facing in practice. We have shown that these issues constitute only a
small percentage of SVs and SV types reported in expert-based SV sources such
as NVD and CWE. This finding suggests that more effort should be directed
towards performing assessment, e.g., predicting CVSS metrics, for these
commonly encountered types of SVs. It is also recommended to develop
assessment models for each of these types rather than generic models for all
SV types to improve assessment performance.

Moreover, from RQ1 and RQ2, we found the prevalent, popular and increasing SV
types like Brute-force/Timing Attacks, Memory/File-related SVs, Malwares and
CSRF. In the context of SV assessment, these common SV types can be
prioritized as custom SV types for prediction using data-driven models (see
section 1 in Chapter 1) because SVs of these types are commonly encountered by
developers and have a high possibility of being exploited in the wild.

From RQ2 and RQ3, the SV types (topics) with low difficulty and high expertise
may also indicate that it is relatively easy to address these SVs in practice,
and vice versa. The difficulty and expertise metrics can be added to the list
of currently used technical metrics (e.g., CVSS metrics and time-to-fix SVs in
section 1) to approximate the amount of effort required to fix the identified
SVs of such types. Such approximation using such external data from Q&A sites
is particularly useful for projects that do not have sufficient project-
specific SV data for training reliable models to predict SV fixing time.

RQ4 revealed different types of solutions that have been provided to address
the SVs. We found that Action to Take and Code Sample are among the most
actionable solution types as they can be directly applied or customized to fix
SVs found in a codebase. If no solution of these two types can be found for a
detected SV, it may suggest that the SV is unique and requires more effort to
fix (i.e., developing a solution from scratch).

Overall, the findings demonstrate the potential of incorporating SV data from
Q&A sites into (data-driven) SV assessment. However, it is also worth noting
that most of the tools (topic T3 in Table 4) currently used by developers for
detecting and assessing SVs are based on static analysis techniques rather
than data-driven models. One reason can be that data-driven approaches are
usually black-box compared to static analysis counterparts. For example, many
questions related to these tools on Q&A sites were about configuring these
tools in specific ways to suite developers’ needs, but this is mostly not
possible with the current data-driven models. This implies a potential
disconnection between state-of-the-art and the state-of-the-practice
techniques for SV assessment. Future work is required to make data-driven SV
assessment models more interpretable (i.e., why a model makes certain
decisions) and configurable (i.e., controling/changing a rule learned by a
model) so that these models can be more widely adopted in practice.

####  3 Threats to Validity

Our data collection is the first threat. We might have missed some SV posts,
but we followed standard techniques in the literature. It is hard to guarantee
100% relevance of the retrieved posts without exhaustive manual validation,
which is nearly impossible with more than 70k posts. However, this threat was
greatly reduced since the selected posts were carefully checked by three
researchers with at least two years of experience in Software Engineering and
Cybersecurity.

The identified taxonomies can be another concern. Topic modeling with LDA has
been shown effective for processing a large amount of textual posts, but there
is still subjectivity in labeling the topics. We mitigated this threat by
manually examining at least 30 posts per topic and cross-checking with three
of the authors. We also performed a similar manual checking for the answer
types in RQ4.

The generalizability of our study may be a threat as well. The patterns we
found may not be the same for other Q&A sites and domains. However, the
reported patterns for SV discussions on SO and SSE were at least confirmed
significant using statistical tests with $p$-values $<$ 0.01. We also released
our code and data at https://github.com/lhmtriet/SV_Empirical_Study for
replication and extension to other domains.

###  6 Chapter Summary

Through a large-scale study of 71,329 posts on SO and SSE, we have revealed
the support of SV-focused discussions on Q&A sites. Using LDA, we devised 13
commonly discussed SV topics on Q&A sites. We also characterized these topics
in terms of their popularity, difficulty, level expertise and solutions
received on these sites. Overall, Q&A sites do support SV discussions on SO
and SSE, and these discussions shed light on the key SV concerns/types that
are of particular interest to developers. Given their importance/prevalence in
practice, more priority should be given to assess these SV types. Moreover,
data about the popularity, difficulty, expertise level and solution
availability on these Q&A sites can be considered for (automatically)
assessing the effort required to fix SVs (e.g., more relevant
resources/support on Q&A sites may make an SV fix easier). Overall, rich data
on crowdsourcing Q&A sites open up various opportunities for the next
generation of data-driven SV assessment methods that are better tailored to
developers’ real-world needs.

###  7 Appendix - PUMiner Overview

In this Appendix, we briefly introduce the PUMiner approach that we used in
section 2 to retrieve security-related posts on SO that can be further refined
to obtain the SV-related posts for the presented empirical study in this
chapter. We hereby describe the context-aware two-stage PU learning model that
forms the core of PUMiner. The complete details and results of PUMiner can be
found in the original publication [304].

####  1 PUMiner - A Context-aware Two-stage PU Learning Model for Retrieving
Security Q&A Posts

PUMiner is a learning model to distinguish security from non-security posts on
Q&A websites. The novelty of PUMiner is that this model does not require
negative (non-security) posts to perform the prediction, saving significant
effort for practitioners as it is non-trivial to define and obtain such non-
security posts in practice. Thus, PUMiner operates on security-related and
unlabeled posts. There are two main components in PUMiner: (i) feature
generation using doc2vec [121] and (ii) PU model building using generated
features for retrieving security posts. These two components are described
hereafter.

PUMiner uses doc2vec [121] to represent input features of posts on Q&A sites
for training the PU model. Doc2vec [121] jointly learns the
representation/embedding of a document (paragraph vector) with its constituent
words. We choose doc2vec to generate embeddings as it can capture the semantic
relationship of a word, unlike traditional feature extraction methods such as
BoW, n-grams, or tf-idf [21, 400]. Specifically, our work adopts the
distributed memory architecture to train doc2vec models as suggested in [121].
Regarding the label of a document/post, a post may contain multiple tags to
describe complex concepts, e.g., _php_ and _security_ tags represent security
issues (sql-injection) in PHP. Therefore, besides single tags, we also combine
all tags to handle the topic mixture. For example, a post with _php_ and
_security_ tags would have _php_security_ alongside _php_ , _security_ and
_post id_ as labels. We also sort labels alphabetically to avoid duplicates
(e.g., _php_security_ and _security_php_ are the same).

1

Input: List of posts: $P_{in}$

Labels of posts: $labels\in\\{positive,unlabeled\\}$

Size of embeddings and Window size: $sz$, $ws$

Classifier and its model configurations: $C$, $config$

2

Output: The trained feature and PU models: feature_model, $\text{model}_{PU}$

3

4$word\\_list,tag\\_list\longleftarrow\emptyset,\emptyset$

5

6foreach  _${p}_{i}\in{P}_{in}$_ do

7
$words,tags\longleftarrow\text{tokenize}({p}_{i}),\text{extract\\_tags}({p}_{i})$

8 $word\\_list,tag\\_list\longleftarrow
word\\_list+\\{words\\},tag\\_list+\\{tags\\}$

9

10

11
$\text{feature\\_model}\longleftarrow\text{train\\_doc2vec}(word\\_list,tag\\_list,sz,ws)$

12
${{\mathbf{X}}_{in}}\longleftarrow\text{obtain\\_feature}(\text{feature\\_model},word\\_list)$

13

14
$P\longleftarrow\\{{\mathbf{x}}_{p}|{\mathbf{x}}_{p}\in{\mathbf{X}}_{in}\wedge
label(p)=positive\\}$

15
$U\longleftarrow\\{{\mathbf{x}}_{p}|{\mathbf{x}}_{p}\in{\mathbf{X}}_{in}\wedge
label(p)=unlabeled\\}$

16

17
${\textbf{centroid}}_{P},{\textbf{centroid}}_{U}\longleftarrow\frac{\sum\limits_{p\in
P}{{{\mathbf{x}}_{p}}}}{\left|P\right|},\frac{\sum\limits_{p\in
U}{{{\mathbf{x}}_{p}}}}{\left|U\right|}$

18 $RN\longleftarrow\emptyset$

19

20 foreach  _${\mathbf{x}}_{i}\in{\mathbf{X}}_{in}$_ do // Stage-1 PU:
Identify reliable negatives

21 if  _$d({\mathbf{x}}_{i},{\mathbf{centroid}}_{U})
<\alpha*d({\mathbf{x}}_{i},{\mathbf{centroid}}_{P})$_ then

22 $RN\longleftarrow RN+\\{{\mathbf{x}}_{i}\\}$

23

24

25

${\text{model}}_{PU}\longleftarrow\text{train\\_classifier}(C,P,RN,config)$ //
Stage-2 PU

26

return feature_model, $\text{model}_{PU}$

Algorithm 3 Context-aware two-stage PU learning model building.

Next, we present the context-aware two-stage PU learning model (see Algorithm
3) to retrieve security-related posts – the core of our PUMiner framework.
This algorithm requires a list of discussion posts with their respective
labels (positive or unlabeled), along with the configurations of doc2vec (size
of embeddings and window size) and classification models (model
hyperparameters). Details of Algorithm 3 are given hereafter.

Lines 1-8: Learning doc2vec context-aware feature vectors. Lines 1-4 tokenize
text and extract tags of each post to prepare the data for training doc2vec
models. Line 5 trains a doc2vec embedding model to learn the context of words
and posts. Line 6 then obtains the embedding vector of each post using the
trained doc2vec model. Lines 7 and 8 extract the features of both positive and
unlabeled posts, respectively.

Lines 9-13: Stage one of PU learning model. Inspired by PU learning in other
domains [401, 402, 403, 404], we assume that the context-aware doc2vec
embeddings can make posts of the same class stay in close proximity in the
embedding space. Stage-one PU learning first identifies (reliable/pure)
negative (non-security) posts in the unlabeled set that are as different as
possible from the positive set. A traditional binary classifier would work
poorly in this case since the negative class is not pure [403]. In line 9 of
Algorithm 3, we propose to approximately locate unknown negative posts using
the centroid (average) vectors of the known positive
(${\textbf{centroid}}_{P}$) and unlabeled (${\textbf{centroid}}_{U}$) sets,
respectively. Since the number of non-security posts is dominant (i.e., up to
96% on Stack Overflow as per our analysis), ${\textbf{centroid}}_{U}$ would
represent the negative class more than the positive one. Lines 11-13 compute
and compare the cosine distances [202, 121, 403] (see Eq. (4)) from each post
to the two centroids. If the current post is closer to
${\textbf{centroid}}_{U}$ (i.e., more towards the negative class), it would be
selected as a reliable negative.

$\text{cosine}\\_\text{distance}=d(i,\,j)=1-\frac{{{\mathbf{p}}_{i}}\centerdot{{\mathbf{p}}_{j}}}{\left\|{{\mathbf{p}}_{i}}\right\|\times\left\|{{\mathbf{p}}_{j}}\right\|}$
(4)

where ${\mathbf{p}}_{i}$ and ${\mathbf{p}}_{j}$ are the embedding vectors of
the posts $i^{\text{th}}$ and $j^{\text{th}}$, respectively. The range of
$\text{cosine}\\_\text{distance}$ is [0, 2].

We also propose a scaling factor ($\alpha$) to increase the flexibility of our
centroid-based approach, which would be jointly optimized with other
hyperparameters of binary classifiers in the second stage. Besides having only
one hyperparameter for tuning, this centroid-based approach can incrementally
learn new post(s) very fast with a complexity time of $O(1)$, as given in Eq.
(5).

${\mathbf{centroid}}_{new}=\frac{{\mathbf{centroid}}_{old}\times
N+{\mathbf{x}}_{new}}{N+\text{size}({{\mathbf{x}}_{new}})}$ (5)

where ${\mathbf{centroid}}_{old}$ , ${\mathbf{centroid}}_{new}$ are the
centroid vectors before and after learning new post(s) (${\mathbf{x}}_{new}$),
while $N$ is the original number of posts in the positive or unlabeled set.

Lines 14-15: Stage two of PU learning model. Using positive and (reliable)
negative posts from the first stage, the second stage of PU learning (line 14)
trains a binary classifier with its hyperparameters. In the model building
process, a PU model is trained with the optimal classifier and its
configurations obtained from a validation process (e.g., k-fold cross-
validation). Finally, line 15 saves the trained feature and PU models to disk
for future inference of whether a post is related to security.

##  Chapter 6 Conclusions and Future Work

Software Vulnerability (SV) assessment is an integral step of software
development. There has been an increasing number of studies on data-driven SV
assessment. Data-driven approaches have the benefits of extracting patterns
and rules from large-scale historical SV data in the wild without manual
definition from experts. Such capability of these approaches enables
researchers to automate various SV assessment tasks that were previously not
possible with static analysis and rule-based counterparts. Despite the rising
interest in data-driven SV assessment, the application of these approaches in
practice has been hindered by three key issues: (i) concept drift in SV data
used for report-level SV assessment, (ii) lack of code-based SV assessment,
and (iii) SV assessment without considering developers’ real-world SV needs.
This thesis has tackled these three challenges to improve the real-world
applicability of data-driven SV assessment. We have first systematized the
knowledge and have identified the key practices of the field. Then, we have
developed various solutions to enhance the performance and applicability of SV
assessment models in real-world settings. These solutions have been based on
the customization of recent advances in Machine Learning, Deep Learning (DL)
and Natural Language Processing (NLP), as well as the utilization of relevant
software artifacts (e.g., source code) and open sources (e.g., Q&A sites) that
have been little explored in the literature. We have demonstrated the
potential effectiveness and value of our solutions using real-world SV data.
Chapter 6 first summarizes the key contributions and findings of this thesis
and then suggests several avenues for future research111Some of the future
research directions presented in this chapter are based on our paper “A Survey
on Data-Driven Software Vulnerability Assessment and Prioritization”, ACM
Computing Surveys journal (CORE A*) [11]. in the emerging area of data-driven
SV assessment.

###  1 Summary of Contributions and Findings

The significant contributions and key findings of the thesis are summarized as
follows.

####  1 A Systematization of Knowledge of Data-Driven SV Assessment

In Chapter 1, we have reviewed 84 representative primary studies on data-
driven SV assessment to systematize the knowledge of the field. Our review has
identified a taxonomy of five key SV assessment tasks/themes that have been
automated using data-driven approaches. The themes are predictions of SV (i)
exploitation (probability, time, and characteristics), (ii) impact
(confidentiality, integrity, availability, scope, and custom impacts), (iii)
severity (probability, levels, and score), (iv) type (CWE and custom types),
as well as (v) other tasks (e.g., retrieval of affected product names/vendors,
SV knowledge graph construction, and SV fixing effort prediction).

The review has also shed light on the common data-driven practices adopted by
these studies to automate SV assessment tasks. Regarding the data sources, NVD
[16]/CVE [66] has been most frequently used to provide inputs (mostly SV
descriptions) and outputs (e.g., CVSS metrics) for SV assessment models.
Moreover, Bag-of-words, Word2vec [202] and more recently BERT [80] have been
common methods for extracting features from text-based descriptions of SVs. In
addition, linear Support Vector Machine has been the most commonly used for
building prediction models, while ensemble Machine Learning (ML) and Deep
Learning (DL) models are also on the rise. These models have been mainly
evaluated on random-based splits like k-fold cross-validation using common
classification metrics (e.g., F1-Score and/or Accuracy) or regression metrics
(e.g., Mean absolute error and/or Correlation coefficient).

While data-driven approaches have shown great potential for various SV
assessment tasks, we have identified three key issues affecting their
practical applicability. Firstly, current studies have mostly used SV
descriptions/reports, but they have not investigated the impacts of changing
data (concept drift) of these descriptions/reports due to ever-increasing SVs
on their models. Such impacts can affect the robustness and even lower the
performance of these models over time when deployed in practice. Secondly,
existing studies have hardly leveraged (vulnerable) source code in which SVs
are usually rooted as inputs for developing data-driven SV assessment models.
In practice, using source code directly for SV assessment can alleviate the
need for SV reports, which in turn supports more timely SV assessment.
Thirdly, the current literature on SV assessment has mainly focused on
characteristics of SVs (e.g., exploitability and impacts) rather than real-
world concerns encountered by developers while addressing SVs (e.g.,
difficulty of implementing solutions). Such real-world concerns can facilitate
more thorough assessment of SVs by incorporating developers’ needs. These
three challenges have motivated us to devise respective solutions presented in
Chapters 2, 3, 4, and 5.

####  2 Automated Report-Level Assessment for Ever-Increasing SVs

In Chapter 2, we have conducted a large-scale analysis of the concept drift
issue on report-level SV assessment models (i.e., predicting seven CVSS
version 2 base metrics [110]). Using data of 105,124 SVs on NVD, we have first
shown that concept drift is indeed an issue in SV descriptions over the years
due to releases or discoveries of new software products and cyber-attacks.
Concept drift can make k-fold cross-validation, the most commonly used
evaluation technique, inflate the validation performance up to 4.7 times
compared to that of time-based validation (year-based splits). The main reason
is that k-fold cross-validation mixes future SV data during training (i.e.,
using data of SVs that have not yet been reported at training time), while
time-based validation does not. Thus, we strongly recommend time-based
validation instead of k-fold cross-validation for future work on report-level
SV assessment.

We have also proposed using subwords extracted from SV descriptions as
features for report-level SV assessment models to help these models be robust
against Out-of-Vocabulary words caused by concept drift. We have shown that
the proposed subword-based models are much more resilient to all the changes
in SV descriptions over time, while exhibiting competitive or better
performance compared to existing word-only models. We have also demonstrated
the possibility of building compact concept-drift-aware SV assessment models
(up to 94% reduction in model size while retaining at least 90% of the
performance) by using fastText [196, 246]. Overall, we have raised the
awareness of the concept drift issue in SV data/reports and proposed an
effective data-driven solution to addressing such issue for report-level SV
assessment.

####  3 Automated Early SV Assessment using Code Functions

In Chapter 3, we have shown the benefits of performing function-level SV
assessment (automatically assigning the seven CVSS version 2 metrics to SVs in
functions) over report-level SV assessment. We have found that SV reports
collected from NVD required for SV assessment usually appear on average 146
days after the fixing time of respective SVs. Such a delay makes report-level
SV assessment likely untimely. On the other hand, vulnerable code is available
at fixing time by default, thus could be leveraged to support SV assessment
even when SV reports are not (yet) available.

Using 1,782 functions of 429 SVs in 200 real-world projects, we have explored
the use of vulnerable statements containing root causes of SVs together with
other (non-vulnerable) lines in functions as inputs for developing ML models
to automate SV assessment. The optimal function-level SV assessment models, on
average, have achieved a promising performance of 0.64 Matthews Correlation
Coefficient (MCC) and 0.75 F1-Score using all statements in each function
(i.e., vulnerable statements alongside all the other context lines). We have
highlighted the possibility of performing function-level SV assessment without
knowing exactly where vulnerable statements are located in functions. We have
also recommended high-performing features and classifiers for this task.
Overall, we have distilled the first practices of using data-driven approaches
to automate function-level SV assessment to enable earlier fixing
prioritization without waiting for SV reports.

####  4 Automated Just-in-Time SV Assessment using Code Commits

In Chapter 4, we have motivated the need for SV assessment in code commits to
avoid assessment and remediation latencies caused by hidden SVs. We have
pointed out that in practice, SVs can stay hidden in codebases for a long
time, up to 1,469 days, before being reported. As a result, performing SV
assessment in commits, where vulnerable changes are first added to a project,
provides just-in-time information about SVs for remediation planning and
prioritization. This new assessment task requires a suitable approach that can
directly operate with code changes in commits.

We have proposed DeepCVA, a novel deep multi-task learning model, to tackle
commit-level SV assessment. Specifically, DeepCVA simultaneously predicts the
seven base metrics of CVSS version 2 in a single model using shared context-
aware commit features extracted by attention-based convolutional gated
recurrent units. Through large-scale experiments on 1,229 vulnerability-
contributing commits of 542 different SVs in 246 real-world software projects,
we have demonstrated the substantially better performance (38% to 59.8% higher
MCC) of DeepCVA than that of many supervised and unsupervised baseline models.
Multi-task learning has also enabled DeepCVA to require 6.3 times less time
for training and maintenance than seven cumulative assessment models. With the
reported effectiveness and efficiency of DeepCVA, we have made the first
promising step towards a holistic solution to assessing SVs as early as they
appear, which also contributes to the industrial movement of shift-left
security in DevSecOps (securing software adopting the DevOps paradigm) [24].

####  5 Insights of Developers’ Real-World Concerns on Question and Answer
Websites for Data-Driven SV Assessment

As shown in Chapter 1, there has been a deficiency in considering developers’
real-world SV-related concerns for SV assessment. Most of the current studies
have used expert-defined taxonomies, e.g., CVSS or CWE, as SV assessment
outputs. However, these outputs usually do not put an emphasis on the
challenges that developers commonly encounter when addressing SVs in practice,
e.g., difficulty in implementing a solution proposed by experts. Such lacking
considerations can lead to incomplete assessment of SVs, affecting the
decision making in prioritizing SV remediation.

Given that developers usually seek solutions to these concerns on Question and
Answer (Q&A) sites, in Chapter 5, we have collected 71,329 SV-related posts
from two large Q&A sites, namely Stack Overflow (SO) and Security
StackExchange (SSE), to identify developers’ real-world SV concerns and
analyze the support these concerns receive on these sites for SV assessment.
We have used Latent Dirichlet Allocation (LDA) [30] to semi-automatically
identify 13 commonly discussed SV topics/concerns on SO and SSE. We have
discovered that these concerns are only a subset of all SVs/SV types reported
by experts and do not follow the patterns of standard rankings like top-10
OWASP or top-25 CWE. Such differences ask for higher priorities and more
specific techniques in assessing these commonly encountered SV types. We have
also characterized these concerns in terms of their popularity, difficulty,
expertise level and solutions received on these sites, which can be used to
estimate the complexity/effort required to fix SVs. For instance, an
identified SV whose similar SVs have received little support on Q&A sites
would imply high remediation difficulty/effort. Overall, crowdsourcing Q&A
sites like SO and SSE have much potential for providing supplementary SV
assessment metrics from developers’ perspectives.

###  2 Opportunities for Future Research

As shown in section 1, this thesis has made significant contributions to
improve the practicality of data-driven SV assessment. However, there are
still many opportunities for future research to further advance the field.
Note that this section does not cover straightforward extensions of our work
in Chapters 2, 3, 4, and 5 such as using more projects, features, models,
and/or programming languages. Rather, we focus on the future research
opportunities that have received little/no attention so far in this area.

####  1 Integration of SV Data on Issue Tracking Systems

Existing studies, including ours in Chapters 2, have mainly utilized NVD/CVE
for collecting SV reports; whereas, bug/issue tracking systems like
JIRA,222https://www.atlassian.com/software/jira
Bugzilla333https://www.bugzilla.org/ or GitHub
issues444https://docs.github.com/en/issues also contain an abundance of SV
reports, yet have been underexplored for data-driven SV assessment. Besides
providing SV descriptions like CVE/NVD, these issue tracking systems also
contain other artifacts such as steps to reproduce, stack traces and test
cases that give extra information about SVs [405]. However, it is not trivial
to obtain and integrate these SV-related bug reports with the ones on SV
databases.

One way to retrieve SVs on issue tracking systems is to use security bug
reports [406]. Much research work has been put into developing effective
models to automatically retrieve security bug reports (e.g., [367, 366, 407]).
Among these studies, Wu et al. [407] manually verified and cleaned the
security bug reports to provide a clean dataset for automated security bug
report identification. However, more of such manual effort is still required
to obtain up-to-date data because the original security bug reports in [407]
were actually a part of the dataset collected back in 2014 [408].

It is worth noting that not all security bug reports are related to SVs such
as issues/improvements in implementing security features.555The security bug
report AMBARI-1373 on JIRA (https://issues.apache.org/jira/browse/AMBARI-1373)
was about improving the front-end of AMBARI Web by displaying the current
logged in user. Thus, future studies need to filter out these cases before
using security bug reports for SV assessment. We also emphasize that some SV-
related bug reports are overlapping with the ones on NVD (e.g., the SV report
AMBARI-14780666https://issues.apache.org/jira/browse/AMBARI-14780 on JIRA
refers to CVE-2016-0731 on CVE/NVD). Such overlaps would require data cleaning
during the integration of reports on issue tracking systems and SV databases
to avoid data duplication (e.g., similar SV descriptions) when developing SV
assessment models.

####  2 Improving Data Efficiency for Data-Driven SV Assessment

As shown in Chapter 1, many of the SV assessment tasks being automated by
data-driven approaches, including the prediction of CVSS base metrics in
Chapters 2, 3, and 4, suffer from the data imbalance and data scarcity issues.
Moreover, existing work as well as our studies in Chapters 2, 3, and 4 have
mainly used fully-supervised learning models for automating these tasks, but
these models require sufficiently large and fully labeled data to perform
well. To address the data-hungriness of these fully-supervised learning
models, future studies can approach the SV assessment tasks with low-shot
learning and/or semi-supervised learning.

Low-shot learning a.k.a. few-shot learning is designed to perform supervised
learning using only a few examples per class, significantly reducing the
labeling effort and increasing model robustness against imbalanced data [201].
According to Chapter 1, so far, only one study in this area utilized low-shot
learning with a deep Siamese network [150] (i.e., a shared feature model with
similarity learning) to effectively predict SV types (CWE) and even generalize
to unseen classes (i.e., zero-shot learning). There are still many
opportunities for investigating different few-shot learning techniques for
other SV assessment tasks, e.g., predicting CVSS metrics. Note that the shared
features in few-shot learning can also be enhanced with pre-trained models
(e.g., BERT [80]) on another domain/task/project with more labeled data than
the current task/project in the SV domain.

Semi-supervised learning enables training models with limited labeled data yet
a large amount of unlabeled data [200], potentially leveraging
hidden/unlabeled SVs in the wild. Recently, we have seen an increasing
interest in using different techniques of this learning paradigm in the SV
domain such as collecting SV patches using multi-view co-training [332],
retrieving SV discussions on developer Q&A sites using positive-unlabeled
learning [304], curating SVs from multiple sources in the wild using self-
training [409]. However, it is still little known about the effectiveness of
semi-supervised learning for SV assessment tasks.

####  3 Customized Data-Driven SV Assessment

Similar to the existing studies reviewed in Chapter 1, in Chapters 2, 3, and
4, we have used the standard CVSS metrics [29] for assessing the
exploitability, impact and severity levels/score of SVs, but there are
increasing concerns that these CVSS outputs are still generic. Specifically,
Spring et al. [209] argued that CVSS tends to provide one-size-fits-all
assessment metrics regardless of the context of SVs; i.e., the same SVs in
different domains/environments are assigned the same metric values. For
instance, banking systems may consider the confidentiality and integrity of
databases more important than the availability of web/app interfaces.

In the future, alongside CVSS, prediction models should also incorporate the
domain/business knowledge to customize the assessment of SVs to a system of
interest (e.g., the impact of SVs on critical component(s) and/or the
readiness of developers/solutions for mitigating such SVs in the current
system). Development environments (e.g., programming language, tools or
frameworks) being used are among the key factors that affect the fixing of a
specific SV in an organization. Thus, SV-related issues that developers have
encountered with these environments discussed on Q&A sites can be considered
to enrich SV assessment information. Particularly, future work can leverage
the taxonomy we identified in Chapter 5 to automatically match detected SVs
with similar issues on Q&A sites, and then follow our framework to indicate
the level of support (popularity, difficulty, expertise level, and solution
type) these issues have received. The higher the support is, the more likely
developers will find relevant information to fix the current SV, which in turn
reduces the fixing complexity/effort. In the future, case studies with
practitioners will also be fruitful to correlate the quantitative performance
of models and their usability/usefulness in real-world systems (e.g., reducing
more critical SVs yet using fewer resources).

####  4 Enhancing Interpretability of SV Assessment Models

As discussed in Chapter 5, the lack of interpretability is one of the key
reasons impeding the adoption of data-driven approaches compared to static
analysis tools for SV assessment in practice. Model interpretability is
important to increase the transparency of the predictions made by a model,
allowing practitioners to adjust the model/data to meet certain requirements
[204]. According to Chapter 1, very few reviewed papers in this area (e.g.,
[97, 21]) have explicitly discussed important features and/or explained
why/when their models worked/failed for a task.

SV assessment can draw inspiration from the related SV detection area where
the interpretability of (DL-based) prediction models has been actively
explored mainly by using (i) specific model architectures/parameters or (ii)
external interpretation models/techniques [204]. In the first approach, prior
studies successfully used the feature activation maps in a CNN model [410] or
leveraged attention-based neural network [411] to highlight and visualize the
important code tokens that contribute to SVs. The second approach uses
separate interpretation models on top of trained SV detectors. The
interpretation models are either domain/model-agnostic [412], domain-agnostic
yet specific to a model type (graph neural network [276]) or SV-specific
[413]. The aforementioned approaches produce local/sample-wise interpretation,
which can be aggregated to obtain global/task-wise interpretation. The global
interpretation is similar to the feature importance of traditional ML models
[414] such as the weights of linear models (e.g., Logistic regression) or the
(im)purity of nodes split by each feature in tree-based models (e.g., Random
forest). However, it is still unclear about the applicability/effectiveness of
these approaches for interpreting ML/DL-based SV assessment models, requiring
further investigations.

####  5 Data-Driven SV Assessment in Data-Driven Systems

Like the current literature (see Chapter 1), this thesis has mainly focused on
SV assessment for traditional software, but we envision there is an impending
need for SV assessment in data-driven/Artificial Intelligence (AI)-based
systems. Data-driven/AI-based systems (e.g., smart recommender systems,
chatbots, robots, and autonomous cars) are an emerging breed of systems whose
cores are powered by AI technologies, e.g., ML and DL models built on data,
rather than human-defined instructions as in traditional systems [193].
However, it is challenging to adapt the current practices of SV assessment to
data-driven/AI-based systems. Some of these challenges are presented
hereafter.

CVSS [29] is currently the most popular SV assessment framework for
traditional systems, but its compatibility with data-driven systems still
requires more investigation. The current CVSS documentation lacks instructions
on how to assign metrics/score for SVs in data-driven systems. For example, it
is unclear how to assign static CVSS metrics to systems with automatically
updated data-driven models [409] because adversarial examples for exploitation
would likely change after the models are updated. Such ambiguities should be
clarified/resolved in future CVSS versions as data-driven systems become more
prevalent. The types of SVs in ML/DL models in data-driven systems are also
mostly different from the ones provided by CWE [28]. The difference is mainly
because these new SVs do not only emerge from configurations/code as in
traditional systems, but also from training data and/or trained models [415].
Thus, we recommend that a new category of these SVs should be studied and
potentially incorporated into CWE, similar to the newly added category for
architectural SVs.777https://cwe.mitre.org/data/definitions/1008.html

Existing SV assessment models for traditional systems have not considered
unique data/model-related characteristics/features of data-driven systems
[416]. Specifically, data-driven systems also encompass information about data
(e.g., format, type, size and distribution) and ML/DL model(s) (e.g.,
configurations, parameters and performance). It is worth noting that SVs of
ML/DL models in data-driven systems can also come from the frameworks used to
develop such models (e.g.,
Tensorflow888https://github.com/tensorflow/tensorflow or
Keras999https://github.com/keras-team/keras). However, developers of data-
driven systems may not be aware of the (security) issues in the used ML/DL
frameworks [417]. Thus, besides currently used features, future work should
also consider the information about underlying data/models and ML/DL
development frameworks to improve the SV representation for building models to
assess SVs in data-driven systems.

## References

  * [1] M. Andreessen, “Why software is eating the world,” _Wall Street Journal_ , vol. 20, no. 2011, p. C2, 2011.
  * [2] Wired, “Google is 2 billion lines of code—and it’s all in one place.” [Online]. Available: https://www.wired.com/2015/09/google-2-billion-lines-codeand-one-place/
  * [3] S. M. Ghaffarian and H. R. Shahriari, “Software vulnerability analysis and discovery using machine-learning and data-mining techniques: A survey,” _ACM Computing Surveys (CSUR)_ , vol. 50, no. 4, pp. 1–36, 2017.
  * [4] NIST, “Vulnerability definition on nvd.” [Online]. Available: https://nvd.nist.gov/vuln
  * [5] I. Synopsys, “Heartbleed bug.” [Online]. Available: https://heartbleed.com/
  * [6] T. Conversation, “What is log4j?” [Online]. Available: https://bit.ly/log4j_the_conversation
  * [7] A. C. S. Centre, “Acsc annual cyber threat report 2020-21.” [Online]. Available: https://www.cyber.gov.au/acsc/view-all-content/reports-and-statistics/acsc-annual-cyber-threat-report-2020-21
  * [8] K. Nayak, D. Marino, P. Efstathopoulos, and T. Dumitraş, “Some vulnerabilities are different than others,” in _International Workshop on Recent Advances in Intrusion Detection_. Springer, 2014, pp. 426–446.
  * [9] S. Khan and S. Parkinson, “Review into state of the art of vulnerability assessment using artificial intelligence,” in _Guide to Vulnerability Analysis for Computer Networks and Systems_. Springer, 2018, pp. 3–32.
  * [10] V. Smyth, “Software vulnerability management: how intelligence helps reduce the risk,” _Network Security_ , vol. 2017, no. 3, pp. 10–12, 2017.
  * [11] T. H. M. Le, H. Chen, and M. A. Babar, “A survey on data-driven software vulnerability assessment and prioritization,” _ACM Computing Surveys (CSUR)_ , 2021.
  * [12] P. Foreman, _Vulnerability management_. CRC Press, 2019.
  * [13] K. Kritikos, K. Magoutis, M. Papoutsakis, and S. Ioannidis, “A survey on vulnerability assessment tools and databases for cloud-based web applications,” _Array_ , vol. 3, p. 100011, 2019.
  * [14] M. Z. Bell, “Why expert systems fail,” _Journal of the Operational Research Society_ , vol. 36, no. 7, pp. 613–619, 1985.
  * [15] J. Han, M. Kamber, and J. Pei, “Data mining concepts and techniques third edition,” _The Morgan Kaufmann Series in Data Management Systems_ , vol. 5, no. 4, pp. 83–124, 2011.
  * [16] NIST, “National vulnerability database.” [Online]. Available: https://nvd.nist.gov
  * [17] ——, “Number of vulnerabilities reported on nvd in 2021.” [Online]. Available: http://nvd.nist.gov/vuln/search/statistics?form_type=Basic&results_type=statistics&search_type=all&isCpeNameSearch=false
  * [18] D. S. Cruzes and T. Dybå, “Research synthesis in software engineering: A tertiary study,” _Information and Software Technology_ , vol. 53, no. 5, pp. 440–455, 2011.
  * [19] J. Gama, I. Žliobaitė, A. Bifet, M. Pechenizkiy, and A. Bouchachia, “A survey on concept drift adaptation,” _ACM computing surveys (CSUR)_ , vol. 46, no. 4, pp. 1–37, 2014.
  * [20] G. Spanos, L. Angelis, and D. Toloudis, “Assessment of vulnerability severity using text mining,” in _the 21st Pan-Hellenic Conference on Informatics_ , 2017, pp. 1–6.
  * [21] Z. Han, X. Li, Z. Xing, H. Liu, and Z. Feng, “Learning to predict severity of software vulnerability using only vulnerability description,” in _2017 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE, 2017, pp. 125–136.
  * [22] G. Spanos and L. Angelis, “A multi-target approach to estimate software vulnerability characteristics and severity scores,” _Journal of Systems and Software_ , vol. 146, pp. 152–166, 2018.
  * [23] T. H. M. Le, B. Sabir, and M. A. Babar, “Automated software vulnerability assessment with concept drift,” in _the 16th International Conference on Mining Software Repositories (MSR)_. IEEE, 2019, pp. 371–382.
  * [24] R. N. Rajapakse, M. Zahedi, M. A. Babar, and H. Shen, “Challenges and solutions when adopting devsecops: A systematic review,” _Information and Software Technology_ , vol. 141, p. 106700, 2022.
  * [25] A. Meneely, H. Srinivasan, A. Musa, A. R. Tejeda, M. Mokary, and B. Spates, “When a patch goes bad: Exploring the properties of vulnerability-contributing commits,” in _2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement_. IEEE, 2013, pp. 65–74.
  * [26] Y. Kamei, E. Shihab, B. Adams, A. E. Hassan, A. Mockus, A. Sinha, and N. Ubayashi, “A large-scale empirical study of just-in-time quality assurance,” _IEEE Transactions on Software Engineering_ , vol. 39, no. 6, pp. 757–773, 2012.
  * [27] Y. Zhang and Q. Yang, “A survey on multi-task learning,” _arXiv preprint arXiv:1707.08114_ , 2017.
  * [28] MITRE, “Common weakness enumeration.” [Online]. Available: https://cwe.mitre.org
  * [29] FIRST, “Common vulnerability scoring system.” [Online]. Available: https://www.first.org/cvss
  * [30] D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent dirichlet allocation,” _Journal of machine learning research_ , vol. 3, no. Jan, pp. 993–1022, 2003\.
  * [31] P. Kamongi, S. Kotikela, K. Kavi, M. Gomathisankaran, and A. Singhal, “Vulcan: Vulnerability assessment framework for cloud computing,” in _2013 IEEE 7th International Conference on Software Security and Reliability_. IEEE, 2013, pp. 218–226.
  * [32] M. Bozorgi, L. K. Saul, S. Savage, and G. M. Voelker, “Beyond heuristics: learning to classify vulnerabilities and predict exploits,” in _the 16th ACM SIGKDD international conference on Knowledge discovery and data mining_ , 2010, pp. 105–114.
  * [33] C. Sabottke, O. Suciu, and T. Dumitras, “Vulnerability disclosure in the age of social media: Exploiting twitter for predicting real-world exploits,” in _24th $\\{$USENIX$\\}$ Security Symposium_, 2015, pp. 1041–1056.
  * [34] B. L. Bullough, A. K. Yanchenko, C. L. Smith, and J. R. Zipkin, “Predicting exploitation of disclosed software vulnerabilities using open-source data,” in _the 3rd ACM on International Workshop on Security And Privacy Analytics_ , 2017, pp. 45–53.
  * [35] P. Zeng, G. Lin, L. Pan, Y. Tai, and J. Zhang, “Software vulnerability analysis and discovery using deep learning techniques: A survey,” _IEEE Access_ , 2020.
  * [36] S. K. Singh and A. Chaturvedi, “Applying deep learning for discovery and analysis of software vulnerabilities: A brief survey,” _Soft Computing: Theories and Applications_ , pp. 649–658, 2020.
  * [37] A. O. A. Semasaba, W. Zheng, X. Wu, and S. A. Agyemang, “Literature survey of deep learning-based vulnerability analysis on source code,” _IET Software_ , 2020.
  * [38] G. Lin, S. Wen, Q.-L. Han, J. Zhang, and Y. Xiang, “Software vulnerability detection using deep neural networks: a survey,” _the IEEE_ , vol. 108, no. 10, pp. 1825–1848, 2020.
  * [39] J. Pastor-Galindo, P. Nespoli, F. G. Mármol, and G. M. Pérez, “The not yet exploited goldmine of osint: Opportunities, open challenges and future trends,” _IEEE Access_ , vol. 8, pp. 10 282–10 304, 2020.
  * [40] J. R. G. Evangelista, R. J. Sassi, M. Romero, and D. Napolitano, “Systematic literature review to investigate the application of open source intelligence (osint) with artificial intelligence,” _Journal of Applied Security Research_ , pp. 1–25, 2020.
  * [41] N. Sun, J. Zhang, P. Rimba, S. Gao, L. Y. Zhang, and Y. Xiang, “Data-driven cybersecurity incident prediction: A survey,” _IEEE Communications Surveys & Tutorials_, vol. 21, no. 2, pp. 1744–1772, 2018.
  * [42] I. Goodfellow, Y. Bengio, and A. Courville, _Deep learning_. MIT press, 2016.
  * [43] N. Dissanayake, A. Jayatilaka, M. Zahedi, and M. A. Babar, “Software security patch management–a systematic literature review of challenges, approaches, tools and practices,” _arXiv preprint arXiv:2012.00544_ , 2020.
  * [44] S. Keele, “Guidelines for performing systematic literature reviews in software engineering,” Technical report, Ver. 2.3 EBSE Technical Report. EBSE, Tech. Rep., 2007.
  * [45] C. Wohlin, “Guidelines for snowballing in systematic literature studies and a replication in software engineering,” in _the 18th international conference on evaluation and assessment in software engineering_ , 2014, pp. 1–10.
  * [46] M. Edkrantz, “Predicting exploit likelihood for cyber vulnerabilities with machine learning,” Master’s thesis, 2015.
  * [47] M. Edkrantz, S. Truvé, and A. Said, “Predicting vulnerability exploits in the wild,” in _2015 IEEE 2nd International Conference on Cyber Security and Cloud Computing_. IEEE, 2015, pp. 513–514.
  * [48] M. Almukaynizi, E. Nunes, K. Dharaiya, M. Senguttuvan, J. Shakarian, and P. Shakarian, “Proactive identification of exploits in the wild through vulnerability mentions online,” in _2017 International Conference on Cyber Conflict (CyCon US)_. IEEE, 2017, pp. 82–88.
  * [49] ——, “Patch before exploited: An approach to identify targeted software vulnerabilities,” in _AI in Cybersecurity_. Springer, 2019, pp. 81–113.
  * [50] C. Xiao, A. Sarabi, Y. Liu, B. Li, M. Liu, and T. Dumitras, “From patching delays to infection symptoms: Using risk profiles for an early discovery of vulnerabilities exploited in the wild,” in _27th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 18)_, 2018, pp. 903–918.
  * [51] N. Tavabi, P. Goyal, M. Almukaynizi, P. Shakarian, and K. Lerman, “Darkembed: Exploit prediction with neural language models,” in _the AAAI Conference on Artificial Intelligence_ , vol. 32, no. 1, 2018.
  * [52] D. A. de Sousa, E. R. de Faria, and R. S. Miani, “Evaluating the performance of twitter-based exploit detectors,” _arXiv preprint arXiv:2011.03113_ , 2020\.
  * [53] Y. Fang, Y. Liu, C. Huang, and L. Liu, “Fastembed: Predicting vulnerability exploitation possibility based on ensemble machine learning algorithm,” _Plos one_ , vol. 15, no. 2, p. e0228439, 2020.
  * [54] S.-Y. Huang and Y. Wu, “Dynamic software vulnerabilities threat prediction through social media contextual analysis,” in _the 15th Asia Conference on Computer and Communications Security_ , 2020, pp. 892–894.
  * [55] J. Jacobs, S. Romanosky, I. Adjerid, and W. Baker, “Improving vulnerability remediation through better exploit prediction,” _Journal of Cybersecurity_ , vol. 6, no. 1, p. tyaa015, 2020.
  * [56] J. Yin, M. Tang, J. Cao, and H. Wang, “Apply transfer learning to cybersecurity: Predicting exploitability of vulnerabilities by description,” _Knowledge-Based Systems_ , vol. 210, p. 106529, 2020.
  * [57] N. Bhatt, A. Anand, and V. Yadavalli, “Exploitability prediction of software vulnerabilities,” _Quality and Reliability Engineering International_ , vol. 37, no. 2, pp. 648–663, 2021.
  * [58] O. Suciu, C. Nelson, Z. Lyu, T. Bao, and T. Dumitras, “Expected exploitability: Predicting the development of functional vulnerability exploits,” _arXiv preprint arXiv:2102.07869_ , 2021.
  * [59] A. A. Younis and Y. K. Malaiya, “Using software structure to predict vulnerability exploitation potential,” in _2014 IEEE Eighth International Conference on Software Security and Reliability-Companion_. IEEE, 2014, pp. 13–18.
  * [60] G. Yan, J. Lu, Z. Shu, and Y. Kucuk, “Exploitmeter: Combining fuzzing with machine learning for automated evaluation of software exploitability,” in _2017 IEEE Symposium on Privacy-Aware Computing (PAC)_. IEEE, 2017, pp. 164–175.
  * [61] S. Tripathi, G. Grieco, and S. Rawat, “Exniffer: Learning to prioritize crashes by assessing the exploitability from memory dump,” in _2017 24th Asia-Pacific Software Engineering Conference (APSEC)_. IEEE, 2017, pp. 239–248.
  * [62] S. K. Cha, T. Avgerinos, A. Rebert, and D. Brumley, “Unleashing mayhem on binary code,” in _2012 IEEE Symposium on Security and Privacy_. IEEE, 2012, pp. 380–394.
  * [63] G. Grieco, G. L. Grinblat, L. Uzal, S. Rawat, J. Feist, and L. Mounier, “Toward large-scale vulnerability discovery using machine learning,” in _the Sixth ACM Conference on Data and Application Security and Privacy_ , 2016, pp. 85–96.
  * [64] B. Dolan-Gavitt, P. Hulin, E. Kirda, T. Leek, A. Mambretti, W. Robertson, F. Ulrich, and R. Whelan, “Lava: Large-scale automated vulnerability addition,” in _2016 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2016, pp. 110–121.
  * [65] L. Zhang and V. L. Thing, “Assisting vulnerability detection by prioritizing crashes with incremental learning,” in _TENCON 2018-2018 IEEE Region 10 Conference_. IEEE, 2018, pp. 2080–2085.
  * [66] MITRE, “Common vulnerabilities and exposures.” [Online]. Available: https://cve.mitre.org/
  * [67] O. Security, “Exploit database.” [Online]. Available: https://www.exploit-db.com
  * [68] Broadcom, “Symantec attack signatures.” [Online]. Available: https://bit.ly/symantec_att_sign
  * [69] Microsoft, “Microsoft security advisories.” [Online]. Available: https://bit.ly/ms_sec_advisories
  * [70] T. Micro, “Zeroday initiative security advisories.” [Online]. Available: https://bit.ly/zeroday_sec
  * [71] Rapid7, “Metasploit security advisories.” [Online]. Available: https://www.rapid7.com/db/modules
  * [72] S. Inc, “Bugtraq vulnerability database.” [Online]. Available: http://www.securityfocus.com
  * [73] R. Future, “Recorded future security advisories.” [Online]. Available: https://bit.ly/rf_sec
  * [74] I. Kenna Security, “Kenna security.” [Online]. Available: http://www.kennasecurity.com
  * [75] ESET, “Eset security advisories.” [Online]. Available: https://bit.ly/eset_virus
  * [76] T. Micro, “Trend micro security advisories.” [Online]. Available: https://bit.ly/trend_micro_sec
  * [77] E. Nunes, A. Diab, A. Gunn, E. Marin, V. Mishra, V. Paliath, J. Robertson, J. Shakarian, A. Thart, and P. Shakarian, “Darknet and deepnet mining for proactive cybersecurity threat intelligence,” in _2016 IEEE Conference on Intelligence and Security Informatics (ISI)_. IEEE, 2016, pp. 7–12.
  * [78] T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” in _the 22nd International Conference on Knowledge Discovery and Data Mining_ , 2016, pp. 785–794.
  * [79] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu, “Lightgbm: A highly efficient gradient boosting decision tree,” _Advances in neural information processing systems_ , vol. 30, pp. 3146–3154, 2017.
  * [80] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [81] Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler, “Aligning books and movies: Towards story-like visual explanations by watching movies and reading books,” in _the IEEE international conference on computer vision_ , 2015, pp. 19–27.
  * [82] W. Foundation, “Wikipedia pages.” [Online]. Available: https://www.wikipedia.org
  * [83] FIRST, “Cvss version 3.1.” [Online]. Available: https://www.first.org/cvss/v3.1/specification-document
  * [84] M. Bernaschi, E. Gabrielli, and L. V. Mancini, “Remus: A security-enhanced operating system,” _ACM Transactions on Information and System Security (TISSEC)_ , vol. 5, no. 1, pp. 36–61, 2002.
  * [85] P. K. Manadhata and J. M. Wing, “An attack surface metric,” _IEEE Transactions on Software Engineering_ , vol. 37, no. 3, pp. 371–386, 2010.
  * [86] S. Horwitz, T. Reps, and D. Binkley, “Interprocedural slicing using dependence graphs,” _ACM Transactions on Programming Languages and Systems (TOPLAS)_ , vol. 12, no. 1, pp. 26–60, 1990.
  * [87] CERT, “Basic fuzzing framework.” [Online]. Available: https://bit.ly/basic_fuzzing_framework
  * [88] S. K. Cha, “Ofuzz.” [Online]. Available: https://github.com/sangkilc/ofuzz
  * [89] K. Crammer, O. Dekel, J. Keshet, S. Shalev-Shwartz, and Y. Singer, “Online passive aggressive algorithms,” 2006.
  * [90] J. Jacobs, S. Romanosky, B. Edwards, M. Roytman, and I. Adjerid, “Exploit prediction scoring system (epss),” _arXiv preprint arXiv:1908.04856_ , 2019\.
  * [91] J. Jacobs, S. Romanosky, B. Edwards, I. Adjerid, and M. Roytman, “Exploit prediction scoring system (epss),” _Digital Threats: Research and Practice_ , vol. 2, no. 3, pp. 1–17, 2021.
  * [92] H. Chen, R. Liu, N. Park, and V. Subrahmanian, “Using twitter to predict when vulnerabilities will be exploited,” in _the 25th International Conference on Knowledge Discovery & Data Mining_, 2019, pp. 3143–3152.
  * [93] H. Chen, J. Liu, R. Liu, N. Park, and V. Subrahmanian, “Vest: A system for vulnerability exploit scoring & timing,” in _IJCAI_ , 2019, pp. 6503–6505.
  * [94] M. Kivelä, A. Arenas, M. Barthelemy, J. P. Gleeson, Y. Moreno, and M. A. Porter, “Multilayer networks,” _Journal of complex networks_ , vol. 2, no. 3, pp. 203–271, 2014.
  * [95] Y. Yamamoto, D. Miyamoto, and M. Nakayama, “Text-mining approach for estimating vulnerability score,” in _2015 4th International Workshop on Building Analysis Datasets and Gathering Experience Returns for Security (BADGERS)_. IEEE, 2015, pp. 67–73.
  * [96] T. Wen, Y. Zhang, Y. Dong, and G. Yang, “A novel automatic severity vulnerability assessment framework,” _Journal of Communications_ , vol. 10, no. 5, pp. 320–329, 2015.
  * [97] D. Toloudis, G. Spanos, and L. Angelis, “Associating the severity of vulnerabilities with their description,” in _International Conference on Advanced Information Systems Engineering_. Springer, 2016, pp. 231–242.
  * [98] S. Ognawala, R. N. Amato, A. Pretschner, and P. Kulkarni, “Automatically assessing vulnerabilities discovered by compositional analysis,” in _the 1st International Workshop on Machine Learning and Software Engineering in Symbiosis_ , 2018, pp. 16–25.
  * [99] S. Ognawala, M. Ochoa, A. Pretschner, and T. Limmer, “Macke: Compositional analysis of low-level vulnerabilities with symbolic execution,” in _the 31st IEEE/ACM International Conference on Automated Software Engineering_ , 2016, pp. 780–785.
  * [100] C. Elbaz, L. Rilling, and C. Morin, “Fighting n-day vulnerabilities with automated cvss vector prediction at disclosure,” in _the 15th International Conference on Availability, Reliability and Security_ , 2020, pp. 1–10.
  * [101] Y. Jiang and Y. Atif, “An approach to discover and assess vulnerability severity automatically in cyber-physical systems,” in _13th International Conference on Security of Information and Networks_ , 2020, pp. 1–8.
  * [102] M. Gawron, F. Cheng, and C. Meinel, “Automatic vulnerability classification using machine learning,” in _International Conference on Risks and Security of Internet and Systems_. Springer, 2017, pp. 3–17.
  * [103] X. Gong, Z. Xing, X. Li, Z. Feng, and Z. Han, “Joint prediction of multiple vulnerability characteristics through multi-task learning,” in _2019 24th International Conference on Engineering of Complex Computer Systems (ICECCS)_. IEEE, 2019, pp. 31–40.
  * [104] Z. Chen, Y. Zhang, and Z. Chen, “A categorization framework for common computer vulnerabilities and exposures,” _The Computer Journal_ , vol. 53, no. 5, pp. 551–580, 2010.
  * [105] J. Ruohonen, “Classifying web exploits with topic modeling,” in _2017 28th International Workshop on Database and Expert Systems Applications (DEXA)_. IEEE, 2017, pp. 93–97.
  * [106] M. U. Aksu, K. Bicakci, M. H. Dilek, A. M. Ozbayoglu, and E. ı. Tatli, “Automated generation of attack graphs using nvd,” in _the 8th Conference on Data and Application Security and Privacy_ , 2018, pp. 135–142.
  * [107] K. O. Stanley and R. Miikkulainen, “Evolving neural networks through augmenting topologies,” _Evolutionary computation_ , vol. 10, no. 2, pp. 99–127, 2002.
  * [108] H. Liu and B. Li, “Automated classification of attacker privileges based on deep neural network,” in _International Conference on Smart Computing and Communication_. Springer, 2019, pp. 180–189.
  * [109] K. Kanakogi, H. Washizaki, Y. Fukazawa, S. Ogata, T. Okubo, T. Kato, H. Kanuka, A. Hazeyama, and N. Yoshioka, “Tracing capec attack patterns from cve vulnerability information using natural language processing technique,” in _the 54th Hawaii International Conference on System Sciences_ , 2021, p. 6996\.
  * [110] FIRST, “Cvss version 2.” [Online]. Available: https://www.first.org/cvss/v2/guide
  * [111] ——, “Cvss version 3.” [Online]. Available: https://www.first.org/cvss/v3.0/specification-document
  * [112] D. M. Blei and J. D. McAuliffe, “Supervised topic models,” in _the 20th International Conference on Neural Information Processing Systems_ , 2007, p. 121–128.
  * [113] I. S. Services, “Online database x-force.” [Online]. Available: http://www.iss.net/xforce
  * [114] S. Wold, K. Esbensen, and P. Geladi, “Principal component analysis,” _Chemometrics and intelligent laboratory systems_ , vol. 2, no. 1-3, pp. 37–52, 1987.
  * [115] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” _arXiv preprint arXiv:1609.02907_ , 2016.
  * [116] Y. Zhang and Q. Yang, “A survey on multi-task learning,” _IEEE Transactions on Knowledge and Data Engineering_ , 2021.
  * [117] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” _arXiv preprint arXiv:1409.0473_ , 2014\.
  * [118] S. Inc., “Secunia vulnerability advisories.” [Online]. Available: http://secunia.com
  * [119] T. Hastie, R. Tibshirani, and J. Friedman, _The elements of statistical learning: data mining, inference, and prediction_. Springer Science & Business Media, 2009.
  * [120] Y. Kim, “Convolutional neural networks for sentence classification,” in _the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, 2014, pp. 1746–1751.
  * [121] Q. Le and T. Mikolov, “Distributed representations of sentences and documents,” in _International conference on machine learning_. PMLR, 2014, pp. 1188–1196.
  * [122] MITRE, “Common attack pattern enumeration and classification.” [Online]. Available: https://capec.mitre.org
  * [123] S. J. Pan and Q. Yang, “A survey on transfer learning,” _IEEE Transactions on knowledge and data engineering_ , vol. 22, no. 10, pp. 1345–1359, 2009.
  * [124] P. K. Kudjo, J. Chen, M. Zhou, S. Mensah, and R. Huang, “Improving the accuracy of vulnerability report classification using term frequency-inverse gravity moment,” in _2019 IEEE 19th International Conference on Software Quality, Reliability and Security (QRS)_. IEEE, 2019, pp. 248–259.
  * [125] J. Chen, P. K. Kudjo, S. Mensah, S. A. Brown, and G. Akorfu, “An automatic software vulnerability classification framework using term frequency-inverse gravity moment and feature selection,” _Journal of Systems and Software_ , vol. 167, p. 110616, 2020.
  * [126] P. K. Kudjo, J. Chen, S. Mensah, R. Amankwah, and C. Kudjo, “The effect of bellwether analysis on software vulnerability severity prediction models,” _Software Quality Journal_ , pp. 1–34, 2020.
  * [127] R. Malhotra, “Severity prediction of software vulnerabilities using textual data,” in _International Conference on Recent Trends in Machine Learning, IoT, Smart Cities and Applications_. Springer, 2021, pp. 453–464.
  * [128] K. Chen, Z. Zhang, J. Long, and H. Zhang, “Turning from tf-idf to tf-igm for term weighting in text classification,” _Expert Systems with Applications_ , vol. 66, pp. 245–260, 2016.
  * [129] P. Wang, Y. Zhou, B. Sun, and W. Zhang, “Intelligent prediction of vulnerability severity level based on text mining and xgbboost,” in _2019 Eleventh International Conference on Advanced Computational Intelligence (ICACI)_. IEEE, 2019, pp. 72–77.
  * [130] K. Liu, Y. Zhou, Q. Wang, and X. Zhu, “Vulnerability severity prediction with deep neural network,” in _2019 5th International Conference on Big Data and Information Analytics (BigDIA)_. IEEE, 2019, pp. 114–119.
  * [131] R. Sharma, R. Sibal, and S. Sabharwal, “Software vulnerability prioritization using vulnerability description,” _International Journal of System Assurance Engineering and Management_ , vol. 12, no. 1, pp. 58–64, 2021.
  * [132] S. E. Sahin and A. Tosun, “A conceptual replication on predicting the severity of software vulnerabilities,” in _the Evaluation and Assessment on Software Engineering_ , 2019, pp. 244–250.
  * [133] S. Nakagawa, T. Nagai, H. Kanehara, K. Furumoto, M. Takita, Y. Shiraishi, T. Takahashi, M. Mohri, Y. Takano, and M. Morii, “Character-level convolutional neural network for predicting severity of software vulnerability from vulnerability description,” _IEICE Transactions on Information and Systems_ , vol. 102, no. 9, pp. 1679–1682, 2019.
  * [134] X. Zhang, H. Xie, H. Yang, H. Shao, and M. Zhu, “A general framework to understand vulnerabilities in information systems,” _IEEE Access_ , vol. 8, pp. 121 858–121 873, 2020.
  * [135] A. Khazaei, M. Ghasemzadeh, and V. Derhami, “An automatic method for cvss score prediction using vulnerabilities description,” _Journal of Intelligent & Fuzzy Systems_, vol. 30, no. 1, pp. 89–96, 2016.
  * [136] G. Spanos, A. Sioziou, and L. Angelis, “Wivss: a new methodology for scoring information systems vulnerabilities,” in _the 17th panhellenic conference on informatics_ , 2013, pp. 83–90.
  * [137] S. Lai, L. Xu, K. Liu, and J. Zhao, “Recurrent convolutional neural networks for text classification,” in _the AAAI Conference on Artificial Intelligence_ , vol. 29, no. 1, 2015.
  * [138] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [139] X. Zhang, J. Zhao, and Y. LeCun, “Character-level convolutional networks for text classification,” _arXiv preprint arXiv:1509.01626_ , 2015.
  * [140] H. Chen, J. Liu, R. Liu, N. Park, and V. Subrahmanian, “Vase: A twitter-based vulnerability analysis and score engine,” in _2019 IEEE International Conference on Data Mining (ICDM)_. IEEE, 2019, pp. 976–981.
  * [141] A. Anwar, A. Abusnaina, S. Chen, F. Li, and D. Mohaisen, “Cleaning the nvd: Comprehensive quality assessment, improvements, and analyses,” _arXiv preprint arXiv:2006.15074_ , 2020.
  * [142] ——, “Cleaning the nvd: Comprehensive quality assessment, improvements, and analyses,” _IEEE Transactions on Dependable and Secure Computing_ , 2021\.
  * [143] J. A. Wang and M. Guo, “Vulnerability categorization using bayesian networks,” in _the sixth annual workshop on cyber security and information intelligence research_ , 2010, pp. 1–4.
  * [144] B. Shuai, H. Li, M. Li, Q. Zhang, and C. Tang, “Automatic classification for vulnerability based on machine learning,” in _2013 IEEE International Conference on Information and Automation (ICIA)_. IEEE, 2013, pp. 312–318.
  * [145] S. Na, T. Kim, and H. Kim, “A study on the classification of common vulnerabilities and exposures using naïve bayes,” in _International Conference on Broadband and Wireless Computing, Communication and Applications_. Springer, 2016, pp. 657–662.
  * [146] J. Ruohonen and V. Leppänen, “Toward validation of textual information retrieval techniques for software weaknesses,” in _International Conference on Database and Expert Systems Applications_. Springer, 2018, pp. 265–277.
  * [147] G. Huang, Y. Li, Q. Wang, J. Ren, Y. Cheng, and X. Zhao, “Automatic classification method for software vulnerability based on deep neural network,” _IEEE Access_ , vol. 7, pp. 28 291–28 298, 2019.
  * [148] M. Aota, H. Kanehara, M. Kubo, N. Murata, B. Sun, and T. Takahashi, “Automation of vulnerability classification from its description using machine learning,” in _2020 IEEE Symposium on Computers and Communications (ISCC)_. IEEE, 2020, pp. 1–7.
  * [149] E. Aghaei, W. Shadid, and E. Al-Shaer, “Threatzoom: Cve2cwe using hierarchical neural network,” _arXiv preprint arXiv:2009.11501_ , 2020.
  * [150] S. S. Das, E. Serra, M. Halappanavar, A. Pothen, and E. Al-Shaer, “V2w-bert: A framework for effective hierarchical multiclass classification of software vulnerabilities,” in _2021 IEEE 8th International Conference on Data Science and Advanced Analytics (DSAA)_. IEEE, 2021, pp. 1–12.
  * [151] D. Zou, S. Wang, S. Xu, Z. Li, and H. Jin, “$\mu$vuldeepecker: A deep learning-based system for multiclass vulnerability detection,” _IEEE Transactions on Dependable and Secure Computing_ , 2019.
  * [152] S. S. Murtaza, W. Khreich, A. Hamou-Lhadj, and A. B. Bener, “Mining trends and patterns of software vulnerabilities,” _Journal of Systems and Software_ , vol. 117, pp. 218–228, 2016.
  * [153] Z. Lin, X. Li, and X. Kuang, “Machine learning in vulnerability databases,” in _2017 10th International Symposium on Computational Intelligence and Design (ISCID)_ , vol. 1. IEEE, 2017, pp. 108–113.
  * [154] Z. Han, X. Li, H. Liu, Z. Xing, and Z. Feng, “Deepweak: Reasoning common software weaknesses via knowledge graph embedding,” in _2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER)_. IEEE, 2018, pp. 456–466.
  * [155] H. S. Venter, J. H. Eloff, and Y. Li, “Standardising vulnerability categories,” _Computers & Security_, vol. 27, no. 3-4, pp. 71–83, 2008\.
  * [156] S. Neuhaus and T. Zimmermann, “Security trend analysis with cve topic models,” in _2010 IEEE 21st International Symposium on Software Reliability Engineering_. IEEE, 2010, pp. 111–120.
  * [157] V. Mounika, X. Yuan, and K. Bandaru, “Analyzing cve database using unsupervised topic modelling,” in _2019 International Conference on Computational Science and Computational Intelligence_ , 2019, pp. 72–77.
  * [158] M. Vanamala, X. Yuan, and K. Roy, “Topic modeling and classification of common vulnerabilities and exposures database,” in _2020 International Conference on Artificial Intelligence, Big Data, Computing and Data Communication Systems (icABCD)_. IEEE, 2020, pp. 1–5.
  * [159] W. Aljedaani, Y. Javed, and M. Alenezi, “Lda categorization of security bug reports in chromium projects,” in _the 2020 European Symposium on Software Engineering_ , 2020, pp. 154–161.
  * [160] M. A. Williams, S. Dey, R. C. Barranco, S. M. Naim, M. S. Hossain, and M. Akbar, “Analyzing evolving trends of vulnerabilities in national vulnerability database,” in _2018 IEEE International Conference on Big Data (Big Data)_. IEEE, 2018, pp. 3011–3020.
  * [161] M. A. Williams, R. C. Barranco, S. M. Naim, S. Dey, M. S. Hossain, and M. Akbar, “A vulnerability analysis and prediction framework,” _Computers & Security_, vol. 92, p. 101751, 2020.
  * [162] E. R. Russo, A. D Sorbo, C. A. Visaggio, and G. Canfora, “Summarizing vulnerabilities’ descriptions to support experts during vulnerability assessment activities,” _Journal of Systems and Software_ , vol. 156, pp. 84–99, 2019.
  * [163] M. B. Kursa, A. Jankowski, and W. R. Rudnicki, “Boruta–a system for feature selection,” _Fundamenta Informaticae_ , vol. 101, no. 4, pp. 271–285, 2010\.
  * [164] NIST, “Software assurance reference dataset (sard).” [Online]. Available: https://samate.nist.gov/SRD
  * [165] J. Han, J. Pei, and Y. Yin, “Mining frequent patterns without candidate generation,” _ACM sigmod record_ , vol. 29, no. 2, pp. 1–12, 2000.
  * [166] T. Kohonen, “The self-organizing map,” _the IEEE_ , vol. 78, no. 9, pp. 1464–1480, 1990.
  * [167] OWASP, “Open web application security project.” [Online]. Available: https://bit.ly/owasp_main
  * [168] S. M. Naim, A. P. Boedihardjo, and M. S. Hossain, “A scalable model for tracking topical evolution in large document collections,” in _2017 IEEE International Conference on Big Data (Big Data)_. IEEE, 2017, pp. 726–735.
  * [169] R. C. Barranco, A. P. Boedihardjo, and M. S. Hossain, “Analyzing evolving stories in news articles,” _International Journal of Data Science and Analytics_ , vol. 8, no. 3, pp. 241–256, 2019.
  * [170] S. Weerawardhana, S. Mukherjee, I. Ray, and A. Howe, “Automated extraction of vulnerability information for home computer security,” in _International Symposium on Foundations and Practice of Security_. Springer, 2014, pp. 356–366.
  * [171] Y. Dong, W. Guo, Y. Chen, X. Xing, Y. Zhang, and G. Wang, “Towards the detection of inconsistencies in public security vulnerability reports,” in _28th $\\{$USENIX$\\}$ Security Symposium_, 2019, pp. 869–885.
  * [172] D. Gonzalez, H. Hastings, and M. Mirakhorli, “Automated characterization of software vulnerabilities,” in _2019 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE, 2019, pp. 135–139.
  * [173] NIST, “Vulnerability description ontology.” [Online]. Available: https://bit.ly/nist_vdo
  * [174] H. Binyamini, R. Bitton, M. Inokuchi, T. Yagyu, Y. Elovici, and A. Shabtai, “An automated, end-to-end framework for modeling attacks from vulnerability descriptions,” _arXiv preprint arXiv:2008.04377_ , 2020.
  * [175] ——, “A framework for modeling cyber attack techniques from security vulnerability descriptions,” in _the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, 2021, pp. 2574–2583.
  * [176] X. Ou, S. Govindavajhala, and A. W. Appel, “Mulval: A logic-based network security analyzer,” in _USENIX security symposium_ , vol. 8. Baltimore, MD, 2005, pp. 113–128.
  * [177] H. Guo, Z. Xing, and X. Li, “Predicting missing information of key aspects in vulnerability reports,” _arXiv preprint arXiv:2008.02456_ , 2020.
  * [178] H. Guo, S. Chen, Z. Xing, X. Li, Y. Bai, and J. Sun, “Detecting and augmenting missing key aspects in vulnerability descriptions,” _ACM Transactions on Software Engineering and Methodology_ , 2021.
  * [179] E. Wåreus and M. Hell, “Automated cpe labeling of cve summaries with machine learning,” in _International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment_. Springer, 2020, pp. 3–22.
  * [180] S. Yitagesu, X. Zhang, Z. Feng, X. Li, and Z. Xing, “Automatic part-of-speech tagging for security vulnerability descriptions,” in _2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)_. IEEE, 2021, pp. 29–40.
  * [181] M. Marcus, B. Santorini, and M. A. Marcinkiewicz, “Building a large annotated corpus of english: The penn treebank,” 1993.
  * [182] J. Sun, Z. Xing, H. Guo, D. Ye, X. Li, X. Xu, and L. Zhu, “Generating informative cve description from exploitdb posts by extractive summarization,” _arXiv preprint arXiv:2101.01431_ , 2021.
  * [183] S. Horawalavithana, A. Bhattacharjee, R. Liu, N. Choudhury, L. O. Hall, and A. Iamnitchi, “Mentions of security vulnerabilities on reddit, twitter and github,” in _IEEE/WIC/ACM International Conference on Web Intelligence_ , 2019, pp. 200–207.
  * [184] H. Xiao, Z. Xing, X. Li, and H. Guo, “Embedding and predicting software security entity relationships: A knowledge graph based approach,” in _International Conference on Neural Information Processing_. Springer, 2019, pp. 50–63.
  * [185] L. B. Othmane, G. Chehrazi, E. Bodden, P. Tsalovski, and A. D. Brucker, “Time for addressing software security issues: Prediction models and impacting factors,” _Data Science and Engineering_ , vol. 2, no. 2, pp. 107–124, 2017\.
  * [186] J. R. Finkel, T. Grenager, and C. D. Manning, “Incorporating non-local information into information extraction systems by gibbs sampling,” in _the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05)_ , 2005, pp. 363–370.
  * [187] S. Özkan, “Cve details.” [Online]. Available: https://www.cvedetails.com
  * [188] SecurityTracker, “Securitytracker vulnerability database.” [Online]. Available: https://securitytracker.com
  * [189] O. Project, “Openwall security advisories.” [Online]. Available: https://bit.ly/sec_openwall
  * [190] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” _arXiv preprint arXiv:1802.05365_ , 2018.
  * [191] MITRE, “Common platform enumeration.” [Online]. Available: https://cpe.mitre.org
  * [192] L. ben Othmane, G. Chehrazi, E. Bodden, P. Tsalovski, A. D. Brucker, and P. Miseldine, “Factors impacting the effort required to fix security vulnerabilities,” in _International Conference on Information Security_. Springer, 2015, pp. 102–119.
  * [193] R. S. S. Kumar, J. Penney, B. Schneier, and K. Albert, “Legal risks of adversarial machine learning research,” _arXiv preprint arXiv:2006.16179_ , 2020.
  * [194] H. Zhang, L. Gong, and S. Versteeg, “Predicting bug-fixing time: an empirical study of commercial software projects,” in _2013 35th International Conference on Software Engineering (ICSE)_. IEEE, 2013, pp. 1042–1051.
  * [195] S. Akbarinasaji, B. Caglayan, and A. Bener, “Predicting bug-fixing time: A replication study using an open source software project,” _journal of Systems and Software_ , vol. 136, pp. 173–186, 2018.
  * [196] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, “Enriching word vectors with subword information,” _Transactions of the Association for Computational Linguistics_ , vol. 5, pp. 135–146, 2017.
  * [197] B. Sabir, F. Ullah, M. A. Babar, and R. Gaire, “Machine learning for detecting data exfiltration: A review,” _ACM Computing Surveys (CSUR)_ , vol. 54, no. 3, pp. 1–47, 2021.
  * [198] D. Hommersom, A. Sabetta, B. Coppola, and D. A. Tamburri, “Automated mapping of vulnerability advisories onto their fix commits in open source repositories,” _arXiv preprint arXiv:2103.13375_ , 2021.
  * [199] Broadcom, “Symantec threat explorer.” [Online]. Available: https://bit.ly/symantec_threats
  * [200] J. E. Van Engelen and H. H. Hoos, “A survey on semi-supervised learning,” _Machine Learning_ , vol. 109, no. 2, pp. 373–440, 2020.
  * [201] Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni, “Generalizing from a few examples: A survey on few-shot learning,” _ACM Computing Surveys (CSUR)_ , vol. 53, no. 3, pp. 1–34, 2020.
  * [202] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” _arXiv preprint arXiv:1310.4546_ , 2013.
  * [203] T. H. M. Le, H. Chen, and M. A. Babar, “Deep learning for source code modeling and generation: Models, applications, and challenges,” _ACM Computing Surveys (CSUR)_ , vol. 53, no. 3, pp. 1–38, 2020.
  * [204] Y. Zhang, P. Tiňo, A. Leonardis, and K. Tang, “A survey on neural network interpretability,” _arXiv preprint arXiv:2012.14261_ , 2020.
  * [205] S. Dzeroski and B. Zenko, “Is combining classifiers better than selecting the best one?” in _ICML_ , vol. 2002. Citeseer, 2002, p. 123e30.
  * [206] C. Cortes and V. Vapnik, “Support-vector networks,” _Machine learning_ , vol. 20, no. 3, pp. 273–297, 1995.
  * [207] N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” _arXiv preprint arXiv:1908.10084_ , 2019.
  * [208] A. Mazuera-Rozo, A. Mojica-Hanke, M. Linares-Vasquez, and G. Bavota, “Shallow or deep? an empirical study on detecting vulnerabilities using deep learning,” in _the 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC)_ , 2021, pp. 276–287.
  * [209] J. Spring, E. Hatleback, A. Householder, A. Manion, and D. Shick, “Time to change the cvss?” _IEEE Security & Privacy_, vol. 19, no. 2, pp. 74–78, 2021.
  * [210] T. Menzies, S. Majumder, N. Balaji, K. Brey, and W. Fu, “500+ times faster than deep learning:(a case study exploring faster methods for text mining stackoverflow),” in _2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR)_. IEEE, 2018, pp. 554–563.
  * [211] J.-O. Palacio-Niño and F. Berzal, “Evaluation metrics for unsupervised learning algorithms,” _arXiv preprint arXiv:1905.05667_ , 2019.
  * [212] S. Raschka, “Model evaluation, model selection, and algorithm selection in machine learning,” _arXiv preprint arXiv:1811.12808_ , 2018.
  * [213] F. G. de Oliveira Neto, R. Torkar, R. Feldt, L. Gren, C. A. Furia, and Z. Huang, “Evolution of statistical analysis in empirical software engineering research: Current state and steps forward,” _Journal of Systems and Software_ , vol. 156, pp. 246–267, 2019.
  * [214] Y. Jiao and P. Du, “Performance measures in evaluating machine learning based bioinformatics predictors for classifications,” _Quantitative Biology_ , vol. 4, no. 4, pp. 320–330, 2016.
  * [215] F. Li and V. Paxson, “A large-scale empirical study of security patches,” in _2017 ACM SIGSAC Conference on Computer and Communications Security_ , 2017, pp. 2201–2215.
  * [216] V. Piantadosi, S. Scalabrino, and R. Oliveto, “Fixing of security vulnerabilities in open source projects: A case study of apache http server and apache tomcat,” in _2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST)_. IEEE, 2019, pp. 68–78.
  * [217] B. Ampel, S. Samtani, S. Ullman, and H. Chen, “Linking common vulnerabilities and exposures to the mitre att&ck framework: A self-distillation approach,” _arXiv preprint arXiv:2108.01696_ , 2021.
  * [218] P. Kuehn, M. Bayer, M. Wendelborn, and C. Reuter, “Ovana: An approach to analyze and improve the information quality of vulnerability databases,” in _The 16th International Conference on Availability, Reliability and Security_ , 2021, pp. 1–11.
  * [219] H. Kekül, B. Ergen, and H. Arslan, “A multiclass hybrid approach to estimating software vulnerability vectors and severity score,” _Journal of Information Security and Applications_ , vol. 63, p. 103028, 2021.
  * [220] M. R. Shahid and H. Debar, “Cvss-bert: Explainable natural language processing to determine the severity of a computer security vulnerability from its description,” in _2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA)_. IEEE, 2021, pp. 1600–1607.
  * [221] J. Lyu, Y. Bai, Z. Xing, X. Li, and W. Ge, “A character-level convolutional neural network for predicting exploitability of vulnerability,” in _2021 International Symposium on Theoretical Aspects of Software Engineering (TASE)_. IEEE, 2021, pp. 119–126.
  * [222] I. Babalau, D. Corlatescu, O. Grigorescu, C. Sandescu, and M. Dascalu, “Severity prediction of software vulnerabilities based on their text description,” in _2021 23rd International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC)_. IEEE, 2021, pp. 171–177.
  * [223] K. Charmanas, N. Mittas, and L. Angelis, “Predicting the existence of exploitation concepts linked to software vulnerabilities using text mining,” in _25th Pan-Hellenic Conference on Informatics_ , 2021, pp. 352–356.
  * [224] J. Yin, M. Tang, J. Cao, H. Wang, M. You, and Y. Lin, “Vulnerability exploitation time prediction: an integrated framework for dynamic imbalanced learning,” _World Wide Web_ , vol. 25, no. 1, pp. 401–423, 2022.
  * [225] M. F. Bulut and J. Hwang, “Nl2vul: Natural language to standard vulnerability score for cloud security posture management,” in _2021 IEEE 14th International Conference on Cloud Computing (CLOUD)_. IEEE, 2021, pp. 566–571.
  * [226] V. Ramesh, S. Abraham, P. Vinod, I. Mohamed, C. A. Visaggio, and S. Laudanna, “Automatic classification of vulnerabilities using deep learning and machine learning algorithms,” in _2021 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2021, pp. 1–8.
  * [227] G. Aivatoglou, M. Anastasiadis, G. Spanos, A. Voulgaridis, K. Votis, and D. Tzovaras, “A tree-based machine learning methodology to automatically classify software vulnerabilities,” in _2021 IEEE International Conference on Cyber Security and Resilience (CSR)_. IEEE, 2021, pp. 312–317.
  * [228] P. Vishnu, P. Vinod, and S. Y. Yerima, “A deep learning approach for classifying vulnerability descriptions using self attention based neural network,” _Journal of Network and Systems Management_ , vol. 30, no. 1, pp. 1–27, 2022.
  * [229] Q. Wang, Y. Li, Y. Wang, and J. Ren, “An automatic algorithm for software vulnerability classification based on cnn and gru,” _Multimedia Tools and Applications_ , pp. 1–22, 2022.
  * [230] V. Yosifova, A. Tasheva, and R. Trifonov, “Predicting vulnerability type in common vulnerabilities and exposures (cve) database with machine learning classifiers,” in _2021 12th National Conference with International Participation (ELECTRONICA)_. IEEE, 2021, pp. 1–6.
  * [231] G. Yang, S. Dineen, Z. Lin, and X. Liu, “Few-sample named entity recognition for security vulnerability reports by fine-tuning pre-trained language models,” in _International Workshop on Deployable Machine Learning for Security Defense_. Springer, 2021, pp. 55–78.
  * [232] S. Yitagesu, Z. Xing, X. Zhang, Z. Feng, X. Li, and L. Han, “Unsupervised labeling and extraction of phrase-based concepts in vulnerability descriptions,” in _2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 2021, pp. 943–954.
  * [233] L. Yuan, Y. Bai, Z. Xing, S. Chen, X. Li, and Z. Deng, “Predicting entity relations across different security databases by using graph attention network,” in _2021 IEEE 45th Annual Computers, Software, and Applications Conference (COMPSAC)_. IEEE, 2021, pp. 834–843.
  * [234] H. Guo, Z. Xing, S. Chen, X. Li, Y. Bai, and H. Zhang, “Key aspects augmentation of vulnerability description based on multiple security databases,” in _2021 IEEE 45th Annual Computers, Software, and Applications Conference (COMPSAC)_. IEEE, 2021, pp. 1020–1025.
  * [235] M.-T. Luong, I. Sutskever, Q. V. Le, O. Vinyals, and W. Zaremba, “Addressing the rare word problem in neural machine translation,” _arXiv preprint arXiv:1410.8206_ , 2014.
  * [236] C.-C. Huang, H.-C. Yen, P.-C. Yang, S.-T. Huang, and J. S. Chang, “Using sublexical translations to handle the oov problem in machine translation,” _ACM Transactions on Asian Language Information Processing (TALIP)_ , vol. 10, no. 3, pp. 1–20, 2011.
  * [237] A. Liu and K. Kirchhoff, “Context models for oov word translation in low-resource languages,” _arXiv preprint arXiv:1801.08660_ , 2018.
  * [238] M. Razmara, M. Siahbani, R. Haffari, and A. Sarkar, “Graph propagation for paraphrasing out-of-vocabulary words in statistical machine translation,” in _the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2013, pp. 1105–1115.
  * [239] R. Future, “Exploiting old vulnerabilities.” [Online]. Available: https://www.recordedfuture.com/exploiting-old-vulnerabilities/
  * [240] A. Kao and S. R. Poteet, _Natural language processing and text mining_. Springer Science & Business Media, 2007.
  * [241] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg _et al._ , “Scikit-learn: Machine learning in python,” _the Journal of machine Learning research_ , vol. 12, pp. 2825–2830, 2011.
  * [242] E. Loper and S. Bird, “Nltk: the natural language toolkit,” _arXiv preprint cs/0205028_ , 2002.
  * [243] M. F. Porter, “An algorithm for suffix stripping.” _Program_ , vol. 14, no. 3, pp. 130–137, 1980.
  * [244] C. Bergmeir and J. M. Benítez, “On the use of cross-validation for time series predictor evaluation,” _Information Sciences_ , vol. 191, pp. 192–213, 2012.
  * [245] S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, and R. Harshman, “Indexing by latent semantic analysis,” _Journal of the American society for information science_ , vol. 41, no. 6, pp. 391–407, 1990.
  * [246] A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov, “Bag of tricks for efficient text classification,” _arXiv preprint arXiv:1607.01759_ , 2016\.
  * [247] Kaggle, “Kaggle.” [Online]. Available: https://www.kaggle.com/
  * [248] S. Russell and P. Norvig, “Artificial intelligence: a modern approach,” 2002.
  * [249] S. H. Walker and D. B. Duncan, “Estimation of the probability of an event as a function of several independent variables,” _Biometrika_ , vol. 54, no. 1-2, pp. 167–179, 1967.
  * [250] A. Basu, C. Walters, and M. Shepherd, “Support vector machines for text categorization,” in _36th Annual Hawaii International Conference on System Sciences, 2003. the_. IEEE, 2003, pp. 7–pp.
  * [251] T. K. Ho, “Random decision forests,” in _3rd international conference on document analysis and recognition_ , vol. 1. IEEE, 1995, pp. 278–282.
  * [252] A. Blumer, A. Ehrenfeucht, D. Haussler, and M. K. Warmuth, “Occam’s razor,” _Information processing letters_ , vol. 24, no. 6, pp. 377–380, 1987.
  * [253] T. H. M. Le, T. T. Tran, and L. K. Huynh, “Identification of hindered internal rotational mode for complex chemical species: A data mining approach with multivariate logistic regression model,” _Chemometrics and Intelligent Laboratory Systems_ , vol. 172, pp. 10–16, 2018.
  * [254] F. Wilcoxon, “Individual comparisons by ranking methods,” in _Breakthroughs in Statistics_. Springer, 1992, pp. 196–202.
  * [255] D. Marutho, S. H. Handaka, E. Wijaya _et al._ , “The determination of cluster number at k-mean using elbow method and purity evaluation on headline news,” in _2018 international seminar on application for technology of information and communication_. IEEE, 2018, pp. 533–538.
  * [256] R. Rehurek and P. Sojka, “Software framework for topic modelling with large corpora,” in _In the LREC 2010 Workshop on New Challenges for NLP Frameworks_. Citeseer, 2010.
  * [257] Facebook, “Pre-trained word vectors of fasttext.” [Online]. Available: https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md
  * [258] Y. Hou, X. Ren, Y. Hao, T. Mo, and W. Li, “A security vulnerability threat classification method,” in _International Conference on Broadband and Wireless Computing, Communication and Applications_. Springer, 2017, pp. 414–426.
  * [259] Q. Liu and Y. Zhang, “Vrss: A new system for rating and scoring vulnerabilities,” _Computer Communications_ , vol. 34, no. 3, pp. 264–273, 2011.
  * [260] R. Sharma and R. Singh, “An improved scoring system for software vulnerability prioritization,” in _Quality, IT and Business Operations_. Springer, 2018, pp. 33–43.
  * [261] P. Johnson, R. Lagerström, M. Ekstedt, and U. Franke, “Can the common vulnerability scoring system be trusted? a bayesian analysis,” _IEEE Transactions on Dependable and Secure Computing_ , vol. 15, no. 6, pp. 1002–1015, 2016.
  * [262] C.-C. Huang, F.-Y. Lin, F. Y.-S. Lin, and Y. S. Sun, “A novel approach to evaluate software vulnerability prioritization,” _Journal of Systems and Software_ , vol. 86, no. 11, pp. 2822–2840, 2013.
  * [263] Y. Roumani, J. K. Nwankpa, and Y. F. Roumani, “Time series modeling of vulnerabilities,” _Computers & Security_, vol. 51, pp. 32–40, 2015.
  * [264] M. Tang, M. Alazab, and Y. Luo, “Exploiting vulnerability disclosures: statistical framework and case study,” in _2016 Cybersecurity and Cyberforensics Conference (CCC)_. IEEE, 2016, pp. 117–122.
  * [265] S. M. Rajasooriya, C. P. Tsokos, and P. K. Kaluarachchi, “Cyber security: Nonlinear stochastic models for predicting the exploitability,” _Journal of information Security_ , vol. 8, no. 2, pp. 125–140, 2017.
  * [266] P. K. Kaluarachchi, C. P. Tsokos, and S. M. Rajasooriya, “Non-homogeneous stochastic model for cyber security predictions,” _Journal of Information Security_ , vol. 9, no. 1, pp. 12–24, 2017.
  * [267] N. R. Pokhrel, H. Rodrigo, C. P. Tsokos _et al._ , “Cybersecurity: time series predictive modeling of vulnerabilities of desktop operating system using linear and non-linear approach,” _Journal of Information Security_ , vol. 8, no. 04, p. 362, 2017.
  * [268] T. H. M. Le and M. A. Babar, “On the use of fine-grained vulnerable code statements for software vulnerability assessment models,” _arXiv preprint arXiv:2203.08417_ , 2022.
  * [269] M. Foundation, “Bugzilla issue tracking system.” [Online]. Available: https://www.bugzilla.org/
  * [270] R. Croft, A. Babar, and L. Li, “An investigation into inconsistency of software vulnerability severity across data sources,” in _2022 29th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)_ , 2022.
  * [271] B. Johnson, Y. Song, E. Murphy-Hill, and R. Bowdidge, “Why don’t software developers use static analysis tools to find bugs?” in _2013 35th International Conference on Software Engineering (ICSE)_. IEEE, 2013, pp. 672–681.
  * [272] B. Aloraini, M. Nagappan, D. M. German, S. Hayashi, and Y. Higo, “An empirical study of security warnings from static application security testing tools,” _Journal of Systems and Software_ , vol. 158, p. 110427, 2019.
  * [273] Y. Zhou, S. Liu, J. Siow, X. Du, and Y. Liu, “Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks,” _arXiv preprint arXiv:1909.03496_ , 2019.
  * [274] W. Zheng, J. Gao, X. Wu, F. Liu, Y. Xun, G. Liu, and X. Chen, “The impact factors on the performance of machine learning-based vulnerability detection: A comparative study,” _Journal of Systems and Software_ , vol. 168, p. 110659, 2020.
  * [275] G. Lin, W. Xiao, J. Zhang, and Y. Xiang, “Deep learning-based vulnerable function detection: A benchmark,” in _International Conference on Information and Communications Security_ , 2020, pp. 219–232.
  * [276] Y. Li, S. Wang, and T. N. Nguyen, “Vulnerability detection with fine-grained interpretations,” in _the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering_ , 2021.
  * [277] V. Nguyen, T. Le, O. De Vel, P. Montague, J. Grundy, and D. Phung, “Information-theoretic source code vulnerability highlighting,” in _2021 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2021, pp. 1–8.
  * [278] Z. Li, D. Zou, S. Xu, Z. Chen, Y. Zhu, and H. Jin, “Vuldeelocator: a deep learning-based fine-grained vulnerability detector,” _IEEE Transactions on Dependable and Secure Computing_ , 2021.
  * [279] S. Wattanakriengkrai, P. Thongtanunam, C. Tantithamthavorn, H. Hata, and K. Matsumoto, “Predicting defective lines using a model-agnostic technique,” _IEEE Transactions on Software Engineering_ , 2020.
  * [280] Y. Li, S. Wang, and T. N. Nguyen, “Dlfix: Context-based code transformation learning for automated program repair,” in _The ACM/IEEE 42nd International Conference on Software Engineering_ , 2020, pp. 602–614.
  * [281] J. Zhou, M. Pacheco, Z. Wan, X. Xia, D. Lo, Y. Wang, and A. E. Hassan, “Finding a needle in a haystack: Automated mining of silent vulnerability fixes,” in _2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)_ , 2021.
  * [282] Atlassian, “Jira issue tracking system.” [Online]. Available: https://www.atlassian.com/software/jira
  * [283] Z. Li, D. Zou, S. Xu, H. Jin, Y. Zhu, and Z. Chen, “Sysevr: A framework for using deep learning to detect software vulnerabilities,” _IEEE Transactions on Dependable and Secure Computing_ , 2021.
  * [284] M. Weiser, “Program slicing,” _IEEE Transactions on software engineering_ , no. 4, pp. 352–357, 1984.
  * [285] S. E. Ponta, H. Plate, A. Sabetta, M. Bezzi, and C. Dangremont, “A manually-curated dataset of fixes to vulnerabilities of open-source software,” in _2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)_. IEEE, 2019, pp. 383–387.
  * [286] S. McIntosh and Y. Kamei, “Are fix-inducing changes a moving target? a longitudinal case study of just-in-time defect prediction,” _IEEE Transactions on Software Engineering_ , vol. 44, no. 5, pp. 412–428, 2017.
  * [287] T. Hoang, H. K. Dam, Y. Kamei, D. Lo, and N. Ubayashi, “Deepjit: an end-to-end deep learning framework for just-in-time defect prediction,” in _2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)_. IEEE, 2019, pp. 34–45.
  * [288] U. Alon, M. Zilberstein, O. Levy, and E. Yahav, “code2vec: Learning distributed representations of code,” _the ACM on Programming Languages_ , vol. 3, no. POPL, pp. 1–29, 2019.
  * [289] T. H. M. Le, D. Hin, R. Croft, and M. A. Babar, “Deepcva: Automated commit-level vulnerability assessment with deep multi-task learning,” in _2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 2021, pp. 717–729.
  * [290] C. Rezk, Y. Kamei, and S. Mcintosh, “The ghost commit problem when identifying fix-inducing changes: An empirical study of apache projects,” _IEEE Transactions on Software Engineering_ , 2021.
  * [291] W. G. Cochran, _Sampling techniques_. John Wiley & Sons, 2007.
  * [292] A. J. Viera, J. M. Garrett _et al._ , “Understanding interobserver agreement: the kappa statistic,” _Fam med_ , vol. 37, no. 5, pp. 360–363, 2005.
  * [293] H. Tian, K. Liu, A. K. Kaboré, A. Koyuncu, L. Li, J. Klein, and T. F. Bissyandé, “Evaluating representation learning of code changes for predicting patch correctness in program repair,” in _2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 2020, pp. 981–992.
  * [294] W. Zheng, Y. Jiang, and X. Su, “Vulspg: Vulnerability detection based on slice property graph representation learning,” _arXiv preprint arXiv:2109.02527_ , 2021.
  * [295] S. Salimi, M. Ebrahimzadeh, and M. Kharrazi, “Improving real-world vulnerability characterization with vulnerable slices,” in _The 16th ACM International Conference on Predictive Models and Data Analytics in Software Engineering_ , 2020, pp. 11–20.
  * [296] J. Ferrante, K. J. Ottenstein, and J. D. Warren, “The program dependence graph and its use in optimization,” _ACM Transactions on Programming Languages and Systems (TOPLAS)_ , vol. 9, no. 3, pp. 319–349, 1987.
  * [297] S. Dashevskyi, A. D. Brucker, and F. Massacci, “A screening test for disclosed vulnerabilities in foss components,” _IEEE Transactions on Software Engineering_ , vol. 45, no. 10, pp. 945–966, 2018.
  * [298] D. Shen, G. Wang, W. Wang, M. R. Min, Q. Su, Y. Zhang, C. Li, R. Henao, and L. Carin, “Baseline needs more love: On simple word-embedding-based models and associated pooling mechanisms,” _arXiv preprint arXiv:1805.09843_ , 2018\.
  * [299] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang _et al._ , “Codebert: A pre-trained model for programming and natural languages,” _arXiv preprint arXiv:2002.08155_ , 2020.
  * [300] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” _arXiv preprint arXiv:1508.07909_ , 2015.
  * [301] X. Zhou, D. Han, and D. Lo, “Assessing generalizability of codebert,” in _2021 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE, 2021, pp. 425–436.
  * [302] T. Hoang, H. J. Kang, D. Lo, and J. Lawall, “Cc2vec: Distributed representations of code changes,” in _the ACM/IEEE 42nd International Conference on Software Engineering_ , 2020, pp. 518–529.
  * [303] N. S. Altman, “An introduction to kernel and nearest-neighbor nonparametric regression,” _The American Statistician_ , vol. 46, no. 3, pp. 175–185, 1992\.
  * [304] T. H. M. Le, D. Hin, R. Croft, and M. A. Babar, “Puminer: Mining security posts from developer question and answer websites with pu learning,” in _the 17th International Conference on Mining Software Repositories_ , 2020, pp. 350–361.
  * [305] H. Hanif, M. H. N. M. Nasir, M. F. Ab Razak, A. Firdaus, and N. B. Anuar, “The rise of software vulnerability: Taxonomy of software vulnerabilities detection and machine learning approaches,” _Journal of Network and Computer Applications_ , p. 103009, 2021.
  * [306] T. Sonnekalb, T. S. Heinze, and P. Mäder, “Deep security analysis of program code,” _Empirical Software Engineering_ , vol. 27, no. 1, pp. 1–39, 2022.
  * [307] A. Luque, A. Carrasco, A. Martín, and A. de las Heras, “The impact of class imbalance in classification performance metrics based on the binary confusion matrix,” _Pattern Recognition_ , vol. 91, pp. 216–231, 2019.
  * [308] M. Tomczak and E. Tomczak, “The need to report effect size estimates revisited. an overview of some recommended measures of effect size,” _Trends in Sport Sciences_ , vol. 1, no. 21, pp. 19–25, 2014.
  * [309] A. Field, _Discovering statistics using IBM SPSS statistics_. sage, 2013.
  * [310] J. Cohen, _Statistical power analysis for the behavioral sciences_. Academic press, 2013.
  * [311] G. Macbeth, E. Razumiejczyk, and R. D. Ledesma, “Cliff’s delta calculator: A non-parametric effect size program for two groups of observations,” _Universitas Psychologica_ , vol. 10, no. 2, pp. 545–555, 2011.
  * [312] C. Treude and M. Wagner, “Predicting good configurations for github and stack overflow topic models,” in _2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)_. IEEE, 2019, pp. 84–95.
  * [313] C. Tantithamthavorn, A. E. Hassan, and K. Matsumoto, “The impact of class rebalancing techniques on the performance and interpretation of defect prediction models,” _IEEE Transactions on Software Engineering_ , vol. 46, no. 11, pp. 1200–1219, 2018.
  * [314] Z. Li, D. Zou, S. Xu, X. Ou, H. Jin, S. Wang, Z. Deng, and Y. Zhong, “Vuldeepecker: A deep learning-based system for vulnerability detection,” _arXiv preprint arXiv:1801.01681_ , 2018.
  * [315] T. H. M. Le, R. Croft, D. Hin, and M. A. Babar, “A large-scale study of security vulnerability support on developer q&a websites,” in _Evaluation and Assessment in Software Engineering_ , 2021, pp. 109–118.
  * [316] J. Kim, T. Kim, and E. G. Im, “Survey of dynamic taint analysis,” in _2014 4th IEEE International Conference on Network Infrastructure and Digital Content_. IEEE, 2014, pp. 269–272.
  * [317] S. Neuhaus, T. Zimmermann, C. Holler, and A. Zeller, “Predicting vulnerable software components,” in _The 14th ACM Conference on Computer and Communications Security_ , 2007, pp. 529–540.
  * [318] I. Chowdhury and M. Zulkernine, “Using complexity, coupling, and cohesion metrics as early indicators of vulnerabilities,” _Journal of Systems Architecture_ , vol. 57, no. 3, pp. 294–313, 2011.
  * [319] Y. Shin and L. Williams, “Can traditional fault prediction models be used for vulnerability prediction?” _Empirical Software Engineering_ , vol. 18, no. 1, pp. 25–59, 2013.
  * [320] Y. Tang, F. Zhao, Y. Yang, H. Lu, Y. Zhou, and B. Xu, “Predicting vulnerable components via text mining or software metrics? an effort-aware perspective,” in _2015 IEEE International Conference on Software Quality, Reliability and Security_. IEEE, 2015, pp. 27–36.
  * [321] G. Lin, J. Zhang, W. Luo, L. Pan, Y. Xiang, O. De Vel, and P. Montague, “Cross-project transfer representation learning for vulnerable function discovery,” _IEEE Transactions on Industrial Informatics_ , vol. 14, no. 7, pp. 3289–3297, 2018.
  * [322] V. Nguyen, T. Le, T. Le, K. Nguyen, O. DeVel, P. Montague, L. Qu, and D. Phung, “Deep domain adaptation for vulnerable code function identification,” in _2019 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2019, pp. 1–8.
  * [323] Z. Bilgin, M. A. Ersoy, E. U. Soykan, E. Tomur, P. Çomak, and L. Karaçay, “Vulnerability prediction from source code using machine learning,” _IEEE Access_ , vol. 8, pp. 150 672–150 684, 2020.
  * [324] H. Wang, G. Ye, Z. Tang, S. H. Tan, S. Huang, D. Fang, Y. Feng, L. Bian, and Z. Wang, “Combining graph-based learning with automated data collection for code vulnerability detection,” _IEEE Transactions on Information Forensics and Security_ , vol. 16, pp. 1943–1958, 2020.
  * [325] Y. Ding, S. Suneja, Y. Zheng, J. Laredo, A. Morari, G. Kaiser, and B. Ray, “Velvet: a novel ensemble learning approach to automatically locate vulnerable statements,” in _2022 29th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)_ , 2022.
  * [326] X. Duan, M. Ge, T. H. M. Le, F. Ullah, S. Gao, X. Lu, and M. A. Babar, “Automated security assessment for the internet of things,” in _2021 IEEE 26th Pacific Rim International Symposium on Dependable Computing (PRDC)_. IEEE, 2021, pp. 47–56.
  * [327] S. E. Ponta, H. Plate, and A. Sabetta, “Beyond metadata: Code-centric and usage-based analysis of known vulnerabilities in open-source software,” in _2018 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE, 2018, pp. 449–460.
  * [328] A. Lamkanfi, S. Demeyer, E. Giger, and B. Goethals, “Predicting the severity of a reported bug,” in _2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010)_. IEEE, 2010, pp. 1–10.
  * [329] H. Perl, S. Dechand, M. Smith, D. Arp, F. Yamaguchi, K. Rieck, S. Fahl, and Y. Acar, “Vccfinder: Finding potential vulnerabilities in open-source projects to assist code audits,” in _the 22nd SIGSAC ACM Conference on Computer and Communications Security_ , 2015, pp. 426–437.
  * [330] L. Yang, X. Li, and Y. Yu, “Vuldigger: A just-in-time and cost-aware tool for digging vulnerability-contributing changes,” in _GLOBECOM 2017-2017 IEEE Global Communications Conference_. IEEE, 2017, pp. 1–7.
  * [331] L. G. A. Rodriguez, J. S. Trazzi, V. Fossaluza, R. Campiolo, and D. M. Batista, “Analysis of vulnerability disclosure delays from the national vulnerability database,” in _Anais do I Workshop de Segurança Cibernética em Dispositivos Conectados_. SBC, 2018\.
  * [332] A. D. Sawadogo, T. F. Bissyandé, N. Moha, K. Allix, J. Klein, L. Li, and Y. L. Traon, “Learning to catch security patches,” _arXiv preprint arXiv:2001.09148_ , 2020.
  * [333] F. Thung, D. Lo, L. Jiang, F. Rahman, P. T. Devanbu _et al._ , “When would this bug get reported?” in _2012 28th IEEE International Conference on Software Maintenance (ICSM)_. IEEE, 2012, pp. 420–429.
  * [334] A. Bosu and J. C. Carver, “Peer code review in open source communities using reviewboard,” in _the ACM 4th Annual Workshop on Evaluation and Usability of Programming Languages and Tools_ , 2012, pp. 17–24.
  * [335] P. Thongtanunam, S. McIntosh, A. E. Hassan, and H. Iida, “Investigating code review practices in defective files: An empirical study of the qt system,” in _2015 IEEE/ACM 12th Working Conference on Mining Software Repositories_. IEEE, 2015, pp. 168–179.
  * [336] K. Moran, M. Linares-Vásquez, C. Bernal-Cárdenas, and D. Poshyvanyk, “Auto-completing bug reports for android applications,” in _the 2015 10th Joint Meeting on Foundations of Software Engineering_ , 2015, pp. 673–686.
  * [337] T. Hoang, J. Lawall, Y. Tian, R. J. Oentaryo, and D. Lo, “Patchnet: Hierarchical deep learning-based stable patch identification for the linux kernel,” _IEEE Transactions on Software Engineering_ , 2019.
  * [338] S. Chowdhuri, T. Pankaj, and K. Zipser, “Multinet: Multi-modal multi-task learning for autonomous driving,” in _2019 IEEE Winter Conference on Applications of Computer Vision_. IEEE, 2019, pp. 1496–1504.
  * [339] A. Sabetta and M. Bezzi, “A practical approach to the automatic classification of security-relevant commits,” in _2018 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE, 2018, pp. 579–582.
  * [340] E. Sahal and A. Tosun, “Identifying bug-inducing changes for code additions,” in _the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement_ , 2018, pp. 1–2.
  * [341] V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in _Icml_ , 2010.
  * [342] S. Ruder, “An overview of gradient descent optimization algorithms,” _arXiv preprint arXiv:1609.04747_ , 2016.
  * [343] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” _Nature_ , vol. 323, no. 6088, pp. 533–536, 1986.
  * [344] J. Śliwerski, T. Zimmermann, and A. Zeller, “When do changes induce fixes?” _ACM SIGSOFT Software Engineering Notes_ , vol. 30, no. 4, pp. 1–5, 2005.
  * [345] M. L. McHugh, “Interrater reliability: the kappa statistic,” _Biochemia medica_ , vol. 22, no. 3, pp. 276–282, 2012.
  * [346] H. Hata, C. Treude, R. G. Kula, and T. Ishio, “9.6 million links in source code comments: Purpose, evolution, and decay,” in _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. IEEE, 2019, pp. 1211–1221.
  * [347] Y. Fan, X. Xia, D. A. Da Costa, D. Lo, A. E. Hassan, and S. Li, “The impact of changes mislabeled by szz on just-in-time defect prediction,” _IEEE Transactions on Software Engineering_ , 2019.
  * [348] D. Falessi, J. Huang, L. Narayana, J. F. Thai, and B. Turhan, “On the need of preserving order of data when validating within-project defect classifiers,” _Empirical Software Engineering_ , vol. 25, no. 6, pp. 4805–4830, 2020.
  * [349] M. Jimenez, R. Rwemalika, M. Papadakis, F. Sarro, Y. Le Traon, and M. Harman, “The importance of accounting for real-world labelling when predicting software vulnerabilities,” in _2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2019, pp. 695–705.
  * [350] J. Gorodkin, “Comparing two k-category assignments by a k-category correlation coefficient,” _Computational Biology and Chemistry_ , vol. 28, no. 5-6, pp. 367–374, 2004.
  * [351] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * [352] M. Pradel and K. Sen, “Deepbugs: A learning approach to name-based bug detection,” _the ACM on Programming Languages_ , vol. 2, no. OOPSLA, pp. 1–25, 2018.
  * [353] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [354] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” _The Journal of Machine Learning Research_ , vol. 15, no. 1, pp. 1929–1958, 2014.
  * [355] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _International Conference on Machine Learning_. PMLR, 2015, pp. 448–456.
  * [356] S. Lloyd, “Least squares quantization in pcm,” _IEEE Transactions on Information Theory_ , vol. 28, no. 2, pp. 129–137, 1982.
  * [357] Y. Zhou and A. Sharma, “Automated identification of security issues from commit messages and bug reports,” in _the 2017 11th Joint Meeting on Foundations of Software Engineering_ , 2017, pp. 914–919.
  * [358] Z. Li, D. Zou, J. Tang, Z. Zhang, M. Sun, and H. Jin, “A comparative study of deep learning-based vulnerability detection system,” _IEEE Access_ , vol. 7, pp. 103 184–103 197, 2019.
  * [359] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “Smote: synthetic minority over-sampling technique,” _Journal of Artificial Intelligence Research_ , vol. 16, pp. 321–357, 2002.
  * [360] L. Pascarella, F. Palomba, and A. Bacchelli, “Fine-grained just-in-time defect prediction,” _Journal of Systems and Software_ , vol. 150, pp. 22–36, 2019\.
  * [361] X. Gu, H. Zhang, and S. Kim, “Deep code search,” in _2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE)_. IEEE, 2018, pp. 933–944.
  * [362] X. Hu, G. Li, X. Xia, D. Lo, and Z. Jin, “Deep code comment generation,” in _2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC)_. IEEE, 2018, pp. 200–210.
  * [363] S. E. Ponta, H. Plate, and A. Sabetta, “Detection, assessment and mitigation of vulnerabilities in open source dependencies,” _Empirical Software Engineering_ , vol. 25, no. 5, pp. 3175–3215, 2020.
  * [364] X. Yang, D. Lo, X. Xia, Y. Zhang, and J. Sun, “Deep learning for just-in-time defect prediction,” in _2015 IEEE International Conference on Software Quality, Reliability and Security_. IEEE, 2015, pp. 17–26.
  * [365] F. Camilo, A. Meneely, and M. Nagappan, “Do bugs foreshadow vulnerabilities? a study of the chromium project,” in _2015 IEEE/ACM 12th Working Conference on Mining Software Repositories_. IEEE, 2015, pp. 269–279.
  * [366] F. Peters, T. T. Tun, Y. Yu, and B. Nuseibeh, “Text filtering and ranking for security bug report prediction,” _IEEE Transactions on Software Engineering_ , vol. 45, no. 6, pp. 615–631, 2017.
  * [367] M. Gegick, P. Rotella, and T. Xie, “Identifying security bug reports via text mining: An industrial case study,” in _2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010)_. IEEE, 2010, pp. 11–20.
  * [368] A. Bosu, J. C. Carver, M. Hafiz, P. Hilley, and D. Janni, “Identifying the characteristics of vulnerable code changes: An empirical study,” in _the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering_ , 2014, pp. 257–268.
  * [369] X. Chen, Y. Zhao, Z. Cui, G. Meng, Y. Liu, and Z. Wang, “Large-scale empirical studies on effort-aware security vulnerability prediction methods,” _IEEE Transactions on Reliability_ , vol. 69, no. 1, pp. 70–87, 2019.
  * [370] X.-L. Yang, D. Lo, X. Xia, Z.-Y. Wan, and J.-L. Sun, “What security questions do developers ask? a large-scale study of stack overflow posts,” _Journal of Computer Science and Technology_ , vol. 31, no. 5, pp. 910–924, 2016.
  * [371] S. Bayati and M. Heidary, “Information security in software engineering, analysis of developers communications about security in social q&a website,” in _Pacific-Asia Workshop on Intelligence and Security Informatics_. Springer, 2016, pp. 193–202.
  * [372] T. Lopez, T. Tun, A. Bandara, L. Mark, B. Nuseibeh, and H. Sharp, “An anatomy of security conversations in stack overflow,” in _2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS)_. IEEE, 2019, pp. 31–40.
  * [373] A. Barua, S. W. Thomas, and A. E. Hassan, “What are developers talking about? an analysis of topics and trends in stack overflow,” _Empirical Software Engineering_ , vol. 19, no. 3, pp. 619–654, 2014.
  * [374] S. Ahmed and M. Bagherzadeh, “What do concurrency developers ask about? a large-scale study using stack overflow,” in _the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement_ , 2018, pp. 1–10.
  * [375] C. Rosen and E. Shihab, “What are mobile developers asking about? a large scale study using stack overflow,” _Empirical Software Engineering_ , vol. 21, no. 3, pp. 1192–1223, 2016.
  * [376] M. Bagherzadeh and R. Khatchadourian, “Going big: a large-scale study on what big data developers ask,” in _the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2019, pp. 432–442.
  * [377] A. A. Bangash, H. Sahar, S. Chowdhury, A. W. Wong, A. Hindle, and K. Ali, “What do developers know about machine learning: a study of ml discussions on stackoverflow,” in _2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)_. IEEE, 2019, pp. 260–264.
  * [378] J. Han, E. Shihab, Z. Wan, S. Deng, and X. Xia, “What do programmers discuss about deep learning frameworks,” _Empirical Software Engineering_ , vol. 25, no. 4, pp. 2694–2747, 2020.
  * [379] M. Shahzad, M. Z. Shafiq, and A. X. Liu, “A large scale exploratory analysis of software vulnerability life cycles,” in _2012 34th International Conference on Software Engineering (ICSE)_. IEEE, 2012, pp. 771–781.
  * [380] N. Meng, S. Nagy, D. Yao, W. Zhuang, and G. A. Argoty, “Secure coding practices in java: Challenges and vulnerabilities,” in _the 40th International Conference on Software Engineering_ , 2018, pp. 372–383.
  * [381] A. Rahman, E. Farhana, and N. Imtiaz, “Snakes in paradise?: Insecure python-related coding practices in stack overflow,” in _2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)_. IEEE, 2019, pp. 200–204.
  * [382] B. V. Hanrahan, G. Convertino, and L. Nelson, “Modeling problem difficulty and expertise in stackoverflow,” in _the ACM 2012 conference on Computer Supported Cooperative Work Companion_ , 2012, pp. 91–94.
  * [383] T. Dey, A. Karnauch, and A. Mockus, “Representation of developer expertise in open source software,” in _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. IEEE, 2021\.
  * [384] C. B. Seaman, “Qualitative methods in empirical studies of software engineering,” _IEEE Transactions on software engineering_ , vol. 25, no. 4, pp. 557–572, 1999.
  * [385] Z. Chen, Y. Cao, Y. Liu, H. Wang, T. Xie, and X. Liu, “A comprehensive study on challenges in deploying deep learning based software,” in _the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2020, pp. 750–762.
  * [386] C. Treude, O. Barzilay, and M.-A. Storey, “How do programmers ask and answer questions on the web?(nier track),” in _the 33rd international conference on software engineering_ , 2011, pp. 804–807.
  * [387] J. Bekker and J. Davis, “Learning from positive and unlabeled data: A survey,” _arXiv preprint arXiv:1811.04820_ , 2018.
  * [388] B. Schölkopf, R. C. Williamson, A. J. Smola, J. Shawe-Taylor, and J. C. Platt, “Support vector method for novelty detection,” in _Advances in neural information processing systems_ , 2000, pp. 582–588.
  * [389] L. M. Manevitz and M. Yousef, “One-class svms for document classification,” _Journal of machine Learning research_ , vol. 2, no. Dec, pp. 139–154, 2001\.
  * [390] O. Mendsaikhan, H. Hasegawa, Y. Yamaguchi, and H. Shimada, “Identification of cybersecurity specific content using the doc2vec language model,” in _2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC)_ , vol. 1. IEEE, 2019, pp. 396–401.
  * [391] M. U. Haque, L. H. Iwaya, and M. A. Babar, “Challenges in docker development: A large-scale study using stack overflow,” in _the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)_ , 2020, pp. 1–11.
  * [392] M. Röder, A. Both, and A. Hinneburg, “Exploring the space of topic coherence measures,” in _the eighth ACM international conference on Web search and data mining_ , 2015, pp. 399–408.
  * [393] A. Abdellatif, D. Costa, K. Badran, R. Abdalkareem, and E. Shihab, “Challenges in chatbot development: A study of stack overflow posts,” in _the 17th International Conference on Mining Software Repositories_ , 2020, pp. 174–185.
  * [394] M. Zahedi, R. N. Rajapakse, and M. A. Babar, “Mining questions asked about continuous software engineering: A case study of stack overflow,” in _the Evaluation and Assessment in Software Engineering_ , 2020, pp. 41–50.
  * [395] H. B. Mann, “Nonparametric tests against trend,” _Econometrica: Journal of the Econometric Society_ , pp. 245–259, 1945.
  * [396] W. R. Knight, “A computer method for calculating kendall’s tau with ungrouped data,” _Journal of the American Statistical Association_ , vol. 61, no. 314, pp. 436–439, 1966.
  * [397] H. B. Mann and D. R. Whitney, “On a test of whether one of two random variables is stochastically larger than the other,” _The annals of mathematical statistics_ , pp. 50–60, 1947.
  * [398] E. Database, “Exploit database.” [Online]. Available: www.exploit-db.com
  * [399] M. Zahedi, M. Ali Babar, and C. Treude, “An empirical study of security issues posted in open source projects,” in _the 51st Hawaii International Conference on System Sciences_ , 2018.
  * [400] M. White, C. Vendome, M. Linares-Vásquez, and D. Poshyvanyk, “Toward deep learning software repositories,” in _2015 IEEE/ACM 12th Working Conference on Mining Software Repositories_. IEEE, 2015, pp. 334–345.
  * [401] C. Elkan and K. Noto, “Learning classifiers from only positive and unlabeled data,” in _the 14th ACM SIGKDD international conference on Knowledge discovery and data mining_ , 2008, pp. 213–220.
  * [402] F. Mordelet and J.-P. Vert, “A bagging svm to learn from positive and unlabeled examples,” _Pattern Recognition Letters_ , vol. 37, pp. 201–209, 2014.
  * [403] X. Li and B. Liu, “Learning to classify texts using positive and unlabeled data,” in _IJCAI_ , vol. 3, no. 2003, 2003, pp. 587–592.
  * [404] D. H. Fusilier, M. Montes-y Gómez, P. Rosso, and R. G. Cabrera, “Detecting positive and negative deceptive opinions using pu-learning,” _Information processing & management_, vol. 51, no. 4, pp. 433–443, 2015\.
  * [405] T. Zimmermann, R. Premraj, N. Bettenburg, S. Just, A. Schroter, and C. Weiss, “What makes a good bug report?” _IEEE Transactions on Software Engineering_ , vol. 36, no. 5, pp. 618–643, 2010.
  * [406] F. A. Bhuiyan, M. B. Sharif, and A. Rahman, “Security bug report usage for software vulnerability research: A systematic mapping study,” _IEEE Access_ , vol. 9, pp. 28 471–28 495, 2021.
  * [407] X. Wu, W. Zheng, X. Xia, and D. Lo, “Data quality matters: A case study on data label correctness for security bug report prediction,” _IEEE Transactions on Software Engineering_ , 2021.
  * [408] M. Ohira, Y. Kashiwa, Y. Yamatani, H. Yoshiyuki, Y. Maeda, N. Limsettho, K. Fujino, H. Hata, A. Ihara, and K. Matsumoto, “A dataset of high impact bugs: Manually-classified issue reports,” in _2015 IEEE/ACM 12th Working Conference on Mining Software Repositories_. IEEE, 2015, pp. 518–521.
  * [409] Y. Chen, A. E. Santosa, A. M. Yi, A. Sharma, A. Sharma, and D. Lo, “A machine learning approach for vulnerability curation,” in _the 17th International Conference on Mining Software Repositories_ , 2020, pp. 32–42.
  * [410] R. Russell, L. Kim, L. Hamilton, T. Lazovich, J. Harer, O. Ozdemir, P. Ellingwood, and M. McConley, “Automated vulnerability detection in source code using deep representation learning,” in _2018 17th IEEE international conference on machine learning and applications (ICMLA)_. IEEE, 2018, pp. 757–762.
  * [411] X. Duan, J. Wu, S. Ji, Z. Rui, T. Luo, M. Yang, and Y. Wu, “Vulsniper: Focus your attention to shoot fine-grained vulnerabilities,” in _IJCAI_ , 2019, pp. 4665–4671.
  * [412] A. Warnecke, D. Arp, C. Wressnegger, and K. Rieck, “Evaluating explanation methods for deep learning in security,” in _2020 IEEE European Symposium on Security and Privacy (EuroS &P)_. IEEE, 2020, pp. 158–174.
  * [413] D. Zou, Y. Zhu, S. Xu, Z. Li, H. Jin, and H. Ye, “Interpreting deep learning-based vulnerability detector predictions based on heuristic searching,” _ACM Transactions on Software Engineering and Methodology (TOSEM)_ , vol. 30, no. 2, pp. 1–31, 2021.
  * [414] G. Chandrashekar and F. Sahin, “A survey on feature selection methods,” _Computers & Electrical Engineering_, vol. 40, no. 1, pp. 16–28, 2014.
  * [415] I. Rosenberg, A. Shabtai, Y. Elovici, and L. Rokach, “Adversarial machine learning attacks and defense methods in the cyber security domain,” _ACM Computing Surveys (CSUR)_ , vol. 54, no. 5, pp. 1–36, 2021.
  * [416] Z. Wan, X. Xia, D. Lo, and G. C. Murphy, “How does machine learning change software development practices?” _IEEE Transactions on Software Engineering_ , 2019.
  * [417] J. Liu, Q. Huang, X. Xia, E. Shihab, D. Lo, and S. Li, “Is using deep learning frameworks free? characterizing technical debt in deep learning frameworks,” in _the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Society_ , 2020, pp. 1–10.