# Automated Code-centric Software Vulnerability Assessment:
How Far Are We? An Empirical Study in C/C++

Anh The Nguyen  Independent Researcher Vietnam<EMAIL_ADDRESS>,  Triet Huynh Minh Le  CREST - The Centre for Research on Engineering
Software Technologies, The University of AdelaideAdelaideAustralia Cyber
Security Cooperative Research Centre, Australia<EMAIL_ADDRESS>and
M. Ali Babar  CREST - The Centre for Research on Engineering Software
Technologies, The University of AdelaideAdelaideAustralia Cyber Security
Cooperative Research Centre, Australia<EMAIL_ADDRESS>
###### Abstract.

Background: The C/C++ languages hold significant importance in Software
Engineering research because of their widespread use in practice. Numerous
studies have utilized Machine Learning (ML) and Deep Learning (DL) techniques
to detect software vulnerabilities (SVs) in the source code written in these
languages. However, the application of these techniques in function-level SV
assessment has been largely unexplored. SV assessment is increasingly crucial
as it provides detailed information on the exploitability, impacts, and
severity of security defects, thereby aiding in their prioritization and
remediation. Aims: We conduct the first empirical study to investigate and
compare the performance of ML and DL models, many of which have been used for
SV detection, for function-level SV assessment in C/C++. Method: Using 9,993
vulnerable C/C++ functions, we evaluated the performance of six multi-class ML
models and five multi-class DL models for the SV assessment at the function
level based on the Common Vulnerability Scoring System (CVSS). We further
explore multi-task learning, which can leverage common vulnerable code to
predict all SV assessment outputs simultaneously in a single model, and
compare the effectiveness and efficiency of this model type with those of the
original multi-class models. Results: We show that ML has matching or even
better performance compared to the multi-class DL models for function-level SV
assessment with significantly less training time. Employing multi-task
learning allows the DL models to perform significantly better, with an average
of 8–22% increase in Matthews Correlation Coefficient (MCC), than the multi-
class models. Conclusions: We distill the practices of using data-driven
techniques for function-level SV assessment in C/C++, including the use of
multi-task DL to balance efficiency and effectiveness. This can establish a
strong foundation for future work in this area.

Machine Learning, Deep Learning, Mining Software Repositories, Security
Vulnerability, Vulnerability Assessment

††conference: The 18th ACM/IEEE International Symposium on Empirical Software
Engineering and Measurement; 20–25 October, 2024; Barcelona, Spain††copyright:
none††ccs: Security and privacy Software security engineering

##  1\. Introduction

C and C++ are among the most popular programming languages currently in use.
They are also associated with the highest number of Software Vulnerabilities
(SVs) in real-world projects reported on the Common Vulnerabilities and
Exposures (CVE) (Bhandari et al., 2021) database. These SVs pose serious
security threats as they can be exploited, leading to adverse consequences,
such as unauthorized access to a system, loss of data, or system failure
(NIST, 2024a). Given the extensive usage of C/C++ in critical systems and
infrastructures, especially where low-level control of resources is required,
the impact of SVs in these languages can be devastating. An infamous example
is the Heartbleed bug that originated in OpenSSL, which was described at the
time of discovery (2014) as one of the most catastrophic security risks in
history (Carvalho et al., 2014; Kyatam et al., 2017). The SV allowed attackers
to remotely access protected memory in an estimated 55% of all HTTPS websites
(Durumeric et al., 2014), exposing many users’ credentials and affecting a
multitude of services around the world. As a result, many studies have focused
on C and C++ for SV detection, particularly aiming to predict the presence of
SVs within functions (Hanif et al., 2021).

Resolving SVs, a crucial step in ensuring the robustness of the system or
software affected by SVs, requires more than just the information from the
detection step. Given the nature of these SVs being not equally important or
urgent, they cannot be remediated all at once due to time and resource
constraints (Khan and Parkinson, 2018). In practice, SV assessment can be
employed to prioritize different SVs based on their characteristics and level
of severity. A notable framework for SV assessment is the Common Vulnerability
Scoring System (CVSS) (FIRST, [n. d.]), which defines a standardized set of
metrics on the exploitability, impact, and severity of SVs. Leveraging these
metrics, fixing prioritization can be set on the most critical SVs to limit
their risks to the affected projects as early as possible. Previous studies on
the predictions of these CVSS metrics have proposed different data-driven
techniques to automate SV assessment using SV reports (e.g., (Han et al.,
2017; Spanos and Angelis, 2018; Le et al., 2019, 2022; Le, 2022; Le and Babar,
2024)). However, these previous studies have mostly neglected performing SV
assessment directly in code functions, potentially leading to untimely fixing
prioritization due to missing/delayed public reporting of SVs (Li and Paxson,
2017; Piantadosi et al., 2019; Le and Babar, 2022).

Data-driven approaches have been extensively used for function-level SV
detection (Ghaffarian and Shahriari, 2017; Lin et al., 2020b; Hanif et al.,
2021; Fu and Tantithamthavorn, 2022; Le et al., 2024b, a), but these
techniques have hardly been investigated for the assessment of SV, despite the
similarity in the input code and the importance of the two tasks.
Specifically, there are two main gaps in the use of data-driven approaches for
function-level SV assessment. The first gap is the lack of a performance
comparison between Machine Learning (ML) and Deep Learning (DL) for function-
level SV assessment, specifically in C/C++ where most SVs have been found. For
SV detection, ML was first explored with promising performance (Ghaffarian and
Shahriari, 2017); recently proposed DL models have been shown to be more
effective than traditional ML approaches, achieving state-of-the-art (SOTA)
performance (Zhou et al., 2019; Le et al., 2020a; Nguyen et al., 2022; Chen et
al., 2023; Wang et al., 2020; Li et al., 2018; Fu and Tantithamthavorn, 2022).
Without a similar investigation for SV assessment, it remains unknown to the
practice which type of models (ML or DL) should be selected to optimize
performance. Moreover, SV assessment comprises multiple prediction tasks
(e.g., exploitability, impact, and severity); therefore, building separate
models for each task may induce high training and maintenance costs. A
potential solution to this is multi-task learning, i.e., building a single
model to predict multiple related tasks simultaneously while sharing the same
input/code. This approach has been shown to improve the predictive performance
of many Software Engineering tasks (Gong et al., 2019; Liu et al., 2023; Li et
al., 2023; Mastropaolo et al., 2021; Shetty et al., 2021; Wang et al., 2024;
Liu et al., 2020b, a; Deng et al., 2024). However, the effectiveness of multi-
task learning for function-level SV assessment still remains unknown, posing
the second gap.

We conduct this empirical study to fill the two aformentioned research gaps.
Using a customized dataset with vulnerable C/C++ functions and their
respective CVSS metrics, we study the performance of various ML and DL models
to automate function-level SV assessment in C/C++. The prediction outputs are
the characteristics of these vulnerable functions based on the CVSS metrics.
We evaluate and compare both the multi-class ML and DL models, as well as the
multi-learning variants of DL models. Our findings indicate that ML performs
competitively for the SV assessment tasks with an average of the best model,
achieving an effectiveness value of 0.680 in Matthews Correlation Coefficient
(MCC). Surprisingly, the best ML model outperforms all multi-class DL models
in six out of seven CVSS metrics, while requiring only one-fourth of the total
training time. Switching from multi-class to multi-task learning allows the DL
models to obtain significantly better MCC, with an average increase of 8–22%
in MCC compared to the original performance. The best multi-task DL model also
has an 8% higher MCC value than the best ML model. Our key contributions are
summarized as follows:

  * •

We are the first to investigate data-driven approaches for function-level SV
assessment in C/C++, the languages with the most reported SVs.

  * •

We distill the practices of using ML and DL for function-level SV assessment.
Multi-class ML is better suited for resource-limited scenarios with comparable
performance to multi-class DL and significantly less training time.
Conversely, multi-task DL is the optimal choice for the highest performance.

  * •

We release our data, code, and models for future research (Authors, 2024).

Overall, our study provides knowledge-based evidence and recommended practices
for both researchers and practitioners on how to select and train data-driven
models to enable effective and efficient assessment of SVs at the function
level.

Paper structure. Section 1 introduces the background of CVSS-based SV
assessment in C/C++ functions. Section 3 presents the two RQs. Section 4
describes the methods employed to answer these RQs. Section 5 distills our
results. Section 6 discusses these findings and threats to validity. Section 7
concludes the study.

##  2\. Related Work and Motivation

Metric group | CVSS metric | Description
---|---|---
Exploitability | Access Vector | Medium/technique to attack/penetrate a system
Access Complexity | The complexity to exploit the SV and initiate an attack
Authentication | Whether/what authentication the attack requires to happen
Impact | Confidentiality | The extent to which the attack allows unauthorized access to system sensitive/confidential data
Integrity | The extent to which the attack allows unauthorized modifications to system data
Availability | The extent to which the attack restricts accessibility to system data, resources, and services
Severity | Severity Level | Combination of the Exploitability and Impact metrics, approximating the criticality of SVs

Table 1. The CVSS assessment metrics considered in this study and their
descriptions.

###  2.1. SV Assessment with CVSS

SV assessment plays a crucial role in the SV lifecycle, defining different
attributes of identified SVs (Smyth, 2017). These attributes aid developers in
comprehending the characteristics of SV, thus guiding prioritization and
remediation approaches. For instance, an SV capable of significantly
compromising system confidentiality, such as enabling unauthorized access to
sensitive data, warrants high-priority resolution. Subsequently, a protocol
can be implemented to safeguard confidentiality, like verifying or enforcing
access privileges for the impacted component or data.

The Common Vulnerability Scoring System (CVSS) (FIRST, [n. d.]) is a widely
employed framework for SV assessment by researchers and practitioners. It
comprises two main versions: versions 2 and 3, with version 3 introduced in
2015. Despite this, CVSS version 2 (CVSSv2) remains prevalent because many SVs
predating 2015 still pose risks to modern systems. For instance, an SV
identified as CVE-2004-0113, originating in 2004, was exploited in 2018
(Future, 2024). Therefore, we utilize the assessment metrics of CVSS version 2
as the outputs for SV assessment models in our study.

Although CVSSv2 defines three groups of metrics, we selected six metrics from
the Base group (Access Vector, Access Complexity, Authentication,
Confidentiality, Integrity, Availability), along with the calculated score
(Severity), to capture the nature characteristics of the vulnerabilities.
These metrics and their descriptions are given in Table 1. The other two
groups (Temporal and Environmental), while providing valuable context
information about SVs changes and their environments, are optional and only
exists in some CVE records; thus, we decide to exclude them. Additionally,
while Severity is a combination of Exploitability and Impact, it does not
necessarily provide a holistic view of SVs, potentially leading to a sub-
optimal SV fixing plan. For example, according to the CVSS specification
(FIRST, 2024), two SVs would have the same severity level if they share the
same exploitability but affect different attributes (e.g., Confidentiality vs.
Integrity) of a system to the same extent. As a result, to ensure a thorough
assessment of SVs, we utilize all of the seven mentioned metrics (i.e.,
Confidentiality, Integrity, Availability, Access Vector, Access Complexity,
Authentication, and Severity) as the outputs for building SV assessment, akin
to prior studies (e.g., (Spanos and Angelis, 2018; Le et al., 2019; Gong et
al., 2019; Le and Babar, 2022)).

⬇

1char␣*strdup(const␣char␣*s1)

2{

3␣␣␣␣char␣*s2␣=␣0;

4␣␣␣␣if␣(s1)␣{

5-␣␣␣␣␣␣␣s2␣=␣malloc(strlen(s1)␣+␣1);

6-␣␣␣␣␣␣␣strcpy(s2,␣s1);

7+␣␣␣␣␣␣␣size_t␣len␣=␣strlen(s1)␣+␣1;

8+␣␣␣␣␣␣␣s2␣=␣malloc(len);

9+␣␣␣␣␣␣␣memcpy(s2,␣s1,␣len);

10␣␣␣␣}

11␣␣␣␣return␣s2;

12}

Figure 1. An example of a C++ vulnerable function in Git, extracted from the
fixing commit 34fa79a of the SV [CVE-2016-2315]. Note: line 5 and 6 are
vulnerable. Additions and removals are colored in green and red, respectively.

###  2.2. Data-Driven SV Assessment: From SV Reports to Function Source Code

After SVs are detected, developers need to investigate the root cause, design
the fixing or mitigation plan, implement these fixes, and roll out these
changes after testing; all of which require great level of effort. Therefore,
automating SV assessment is an important step since determining the severity
and scale of impact can help inform decisions, prioritization, and planning of
resources to be made accordingly. Previous studies in the automation of SV
assessment have utilized SV reports as the predominant form of input for the
predictions (e.g., (Han et al., 2017; Spanos and Angelis, 2018; Le et al.,
2019; Duan et al., 2021; Le et al., 2022)). However, such an approach is
inherently constrained by the availability of these reports. Publishing SV
reports after the SVs are fixed is a recommended practice to mitigate their
exploitation (Le et al., 2021a; Zhou et al., 2021). For instance,
CVE-2016-2315, an extremely critical SV with a reported CVSSv2 severity score
of 10.0/10.0, was fixed 185 days
prior111https://github.com/git/git/commit/34fa79a6cde56d6d428ab0d3160cb094ebad3305
to its public disclosure on August 4th,
2016.222https://nvd.nist.gov/vuln/detail/CVE-2016-2315 This practice limits
the application of report-level inputs to prioritize SVs in time.

Taking inspiration from function-level SV detection, using vulnerable
functions directly can enable SV assessment prior to fixing, reducing the
above-mentioned delay caused by report-level SV assessment. To illustrate
function-level SV assessment, we use the function in Fig. 1, one vulnerable
function extracted from the SV CVE-2016-2315 discussed earlier, which exhibits
a possible buffer overflow error during the handling of long path names. The
original code used the built-in strcpy function, which did not take into
account the length of the s1 argument (the source string to copy from), but
instead copied from the start to a null character. If an attacker crafts a
special string without a null character, the function will copy beyond the end
of the buffer and can trigger this SV. The attacker can feed the malicious
input to the system from any network remotely, so the Access Vector is
classified as Network and the Access Complexity is Low. The attacker does not
require authentication to either the actual server/computer running the code,
so its Authentication is None. The malicious input, running inside the file
system of affected machines, can leak full information about the machines and
connected devices, bypassing all security protocols and possibly disabling all
affected resources completely. This resulted in this SV being categorized as
Complete for Confidentiality, Integrity, and Availability impacts. All of
these factors render this SV one of the most severe SVs with the highest
possible score of 10.0 in the Severity category of CVSSv2. The fixing commit
replaced all occurrences of strcpy with memcpy, which copies an exact amount
of bytes provided with a length argument, removing the possibility of an
overflow even if the string itself does not contain a null character. Although
the fix is simple in concept, without the assessment information, developers
can overlook the SV and treat this as a low-level threat, thus motivating the
need to automate SV assessment at the function level.

While Le et al. (Le and Babar, 2022)’s study is the most related to our work,
ours is fundamentally different from theirs. First, the previous investigation
was conducted in Java, which does not have as many critical SVs as in C/C++,
limiting the significance of the work in relation to the latest SV-related
literature and real software development environments. Moreover, they did not
study any use of Deep Learning (DL) for SV assessment, which does not provide
a thorough understanding of how far we are with function-level SV assessment,
given that DL is the state-of-the-art for SV detection.

###  2.3. Deep Learning: SOTA for SV Detection Yet Under-Explored for SV
Assessment

With the rise of data-driven methods in recent years, DL has also been applied
more commonly to SV-related tasks (Harzevili et al., 2023). In particular, the
application of DL in detecting SVs in source code at the function level has
been explored extensively, with multiple works using different approaches,
including sequential-based, graph-based, and Transformer-based models. An
overview of these models can be found in Table 2. It should be noted that,
given the extensive array of studies, this list does not aim to cover all of
the proposed models, but it rather shows the key architectures having been
used for function-level SV detection.

Sequence-based DL models, particularly those implemented with convolutional or
recurrent neural networks, have been widely applied to SV detection (e.g.,
(Filus et al., 2021; Hoang et al., 2019; Li et al., 2017; Dam et al., 2018;
Liu et al., 2019)) due to their advantages over traditional ML methods.
Convolutional Neural Network (CNN) (Kim, 2014) have been proven capable of
identifying local patterns and structures within source code (Russell et al.,
2018) and have been used widely by recent studies as a baseline for newer
models to detect SVs at the function level. Recurrent models are also popular,
accounting for 28% of primary studies in SV detection at different levels of
granularity (Harzevili et al., 2023). Among the models that adopted recurrent
architecture, Long Short-term Memory (LSTM) (Hochreiter and Schmidhuber, 1997)
is particularly suitable for discerning the underlying sequential order of
source code, especially C/C++ source code (Karpathy et al., 2015), by keeping
control of the information from one step/token to the next (Hochreiter and
Schmidhuber, 1997).

Another effective and popular architecture is the graph-based models, which
specifically deal with graph-like structured data. Graphs, representations of
inputs as nodes linked by edges, are ideal for capturing relationships and
dependencies. Therefore, graph models are generally more powerful than
recurrent or convolutional models due to their ability to capture more
meaningful context, especially the semantic representation of complex
structures in source code (Khemani et al., 2024). Since SVs originate from
source code, graph-based models are expected to be well-suited for recognizing
the underlying structural information. Among these, Graph Convolutional
Networks (GCN) (Kipf and Welling, 2016) and Gated Graph Neural Networks (GGNN)
(Li et al., 2015) have seen the most usage and better performance than the
sequence-based models. GCN is similar to traditional CNN, where the nodes in a
graph are learned in the context of the neighbor nodes through the use of
Graph convolution layers, linear layers, and activation functions (Kipf and
Welling, 2016). GGNN, on the other hand, leverages Gated Recurrent Unit (Cho
et al., 2014) to keep the retrieved information in recurrent steps, and
gradient is calculated by backpropagation.

Given the widespread popularity of Bidirectional Encoder Representations from
Transformers (BERT) (Devlin et al., 2018) in the field of Natural Language
Processing, models based on the Transformer architecture (Vaswani et al.,
2017), especially CodeBERT (Feng et al., 2020), have been utilized to perform
SV detection (Fu and Tantithamthavorn, 2022). CodeBERT uses multi-layer
bidirectional Transformer (Vaswani et al., 2017), and is pre-trained on both
natural language and code datasets. This allows CodeBERT to achieve SOTA
performance in code-related downstream tasks. Fu et al. (Fu and
Tantithamthavorn, 2022) fine-tuned CodeBERT on a dataset of C/C++ SVs to
develop LineVul. This model has been shown to outperform existing sequential-
based and graph-based models and achieve the SOTA performance for function-
level SV.

To improve effectiveness and efficiency, multi-task deep learning performs the
prediction of multiple related tasks in a single model, which is applicable to
function-level SV assessment tasks. Multi-task learning is the practice of
incorporating all training tasks and data into one model and performing
training of all the constituent tasks simultaneously. By utilizing data from
different tasks and the knowledge-sharing ability of DL models, multi-task DL
models can learn more universal representations of inputs, thus producing a
superior performance and reducing the overfitting likelihood for each task
(Zhang and Yang, 2021). This approach has also been successfully adopted in a
variety of code-based tasks, such as source code understanding (Liu et al.,
2020b), code completion (Liu et al., 2020a), issue-commits linking (Deng et
al., 2024). These tasks utilize functions source code as inputs, and their
performance improved noticeably when compared to the traditional multi-class
ML/DL approaches.

While these studies have showcased the potential of DL and multi-task DL in
various code-based Software Engineering tasks, particularly SV detection,
their application to function-level SV assessment is still limited. Therefore,
to the best of our knowledge, our empirical study is the first to (1) compare
the performance of state-of-the-art DL and ML models for SV assessment and (2)
investigate whether multi-task learning can enable more effective and
efficient predictions of the tasks.

Architecture | Model | References
---|---|---
Sequence-based | CNN | Li et al. 2017 (Li et al., 2017)
Yan et al. 2021 (Yan et al., 2021)
RNN | Dam et al. 2017 (Dam et al., 2017)
Li et al. 2018 (Li et al., 2018)
Liu et al. 2020 (Liu et al., 2020c)
| Li et al. 2021 (Li et al., 2021b)
| Zou et al. 2021 (Zou et al., 2021)
Graph-based | GGNN | Zhou et al. 2019 (Zhou et al., 2019)
Chakraborty et al. 2020 (Chakraborty et al., 2021)
Ding et al. 2022 (Ding et al., 2022)
| Nguyen et al. 2022 (Nguyen et al., 2022)
GCN | Cheng et al. 2021 (Cheng et al., 2021)
| Ghaffarian et al. 2021 (Ghaffarian and Shahriari, 2021)
|  | Nguyen et al. 2022 (Nguyen et al., 2022)
Transformer-based | CodeBERT | Feng et al. 2020 (Feng et al., 2020)
Fu et al. 2022 (Fu and Tantithamthavorn, 2022)

Table 2. DL models used in SV detection. Note: CNN: Convolutional Neural
Network, RNN: Recurrent Neural Network, GGNN: Gated Graph Neural Network, GCN:
Graph Convolutional Network.

##  3\. Research Questions

To explore and compare the performance of various ML and DL models for
assessing SVs in terms of the CVSS metrics in the C/C++ languages, we
investigate the following two Research Questions (RQs).

RQ1: How well do multi-class Machine Learning and Deep Learning models perform
SV assessment in C/C++? It is possible to automate the prediction of CVSS
assessment metrics using ML models with vulnerable code as input (Le and
Babar, 2022). However, the prior investigation was not done on C/C++, the most
commonly investigated language in the area of SV detection, the prior step to
SV assessment (see section 1). This presents a gap between the current
practice of SV detection and SV assessment. Moreover, in recent years, DL
models have achieved SOTA performance for SV detection (Lin et al., 2020a).
However, these DL models have never been utilized for function-level SV
assessment. Thus, RQ1 aims to evaluate and compare the performance of the
recently proposed ML models and the representative DL models, that have been
previously used for SV detection, for SV assessment in C/C++. RQ1 also gives
insights into the optimal feature/classifier choices for the tasks, enabling
effective function-level SV assessment.

RQ2: Can multi-task Deep Learning improve SV assessment in C++? While RQ1
focuses on multi-class models, i.e., building a separate model for each SV
assessment task, RQ2 aims to study the possibility of using multi-task
learning, i.e., combining all the tasks in a single model. It is worth noting
that multi-task learning is mainly targeted to DL models, so we will adapt the
DL models used in RQ2 for this setting. More details are given in section 4.2.
RQ2 also compares the performance of multi-task models with that of the multi-
class models presented in RQ1, giving insights into whether/to what extent DL
can improve upon ML for function-level SV assessment. RQ2 findings can inform
whether the function-level SV assessment tasks would benefit from sharing
input features through multi-task learning to support efficient model
building.

##  4\. Research Methodology

Figure 2. Methodology used to answer the research questions.

This section presents the methodology we used to empirically study the use of
ML and DL for function-level SV assessment to support timely prioritization
and fixing of SVs in C/C++. We used a machine with 8 CPU cores, 16GB of RAM,
and GeForce RTX 3050Ti GPUs to carry out all the experiments.

Overview. Fig. 2 depicts the workflow we used to answer the two RQs given in
section 3. The workflow consists of three main steps: (i) data collection,
(ii) model building and evaluation, and (iii) results reporting. First, we
curated a dataset of SVs in real-world projects written in C/C++. We then
extracted the vulnerable code functions modified in the vulnerability-fixing
commits of these reported SVs as input as well as the seven CVSS metrics as
output. More details of the data collection are given in section 4.1.
Secondly, the extracted input and output were used to develop multi-class ML
and multi-class DL models in RQ1 and customized to support multi-task learning
in RQ2. The details of these models building and evaluation are given in
sections 4.2 and 4.3. Finally, we evaluated and reported the performance of ML
and DL in RQ1 as well as those of multi-task models and multi-class models in
RQ2.

###  4.1. Data Collection

Figure 3. The CVSS metrics distributions in the dataset. Note: Access
Complex., Auth., and Adj. Net. refer to Access Complexity, Authentication, and
Adjacent Network, respectively.

Dataset selection. Developing models to assess SV requires a specially curated
dataset that contains vulnerable functions and their corresponding CVSS
metrics. We built our dataset upon Big-Vul (Fan et al., 2020) as it is one of
the most popular published SV datasets in C/C++ due to its size and rich
features. Big-Vul consisted of 188,636 functions written in C/C++ related to
91 different SV types originating from 348 different projects on GitHub. These
functions were extracted from respective vulnerability-fixing commits of the
SVs from public CVE databases spanning from 2002 to 2019. In this dataset,
vulnerable functions were those with lines altered in the gathered VFCs,
whereas those that remained unchanged were labeled as non-vulnerable. The
dataset provided 21 features for each function, including the CVE-ID, the
project name of the source code, and the fixed version of each function. It is
worth noting that there are other SV datasets, such as the FFmpeg-Qemu dataset
in Devign (Zhou et al., 2019), the Chromium-Debian dataset in ReVeal
(Chakraborty et al., 2021), or DiverseVul (Chen et al., 2023), but we did not
consider them as they did not include the CVSS metrics and/or CVE-ID required
to trace the CVSS metrics. The customization we made to the Big-Vul dataset to
suit SV assessment are described hereafter.

Data preparation. We focused on the assessment of detected SVs, so we filter
for the vulnerable functions in the Big-Vul dataset. We also wanted to ensure
the relevance of the data being used, and thus we checked whether each of the
original functions was still available on GitHub. Consequently, we discarded
the functions for which we could no longer confirm their existence at the time
of data collection in October 2023. For each of the selected functions, we
retrieved the seven CVSSv2 metrics given in section 2.2, and noticed that
Access Vector was missing in the Big-Vul dataset. Thus, we relied on the CVE-
ID of each vulnerable function to retrieve the value of Access Vector from
public CVE databases. In addition, we converted the base scores to the
severity levels as follows: 0.1 - 3.9 for Low, 4.0 - 6.9 for Medium, and 7.0 -
10.0 for High, following the NVD CVSS qualitative ratings (NIST, 2024b). After
the data collection process, we obtained the final dataset with 9,993 records,
each consisting of a vulnerable function and the respective seven CVSSv2
metrics, including Access Vector, Access Complexity, Authentication,
Confidentiality, Integrity, Availability and Severity. The distributions of
the curated dataset in terms of CVSS metrics are given in Fig. 3.

###  4.2. Machine Learning and Deep Learning Models for Function-Level SV
Assessment

This section describes the implementation of the models. In total, we utilized
six ML models and five DL models. These models were modified to enable multi-
class classification of the CVSS metrics if they had not been done so
originally. We also customized the DL models to support multi-task learning in
RQ2. Note that we introduced multi-task learning only for DL due to the
flexibility of the architecture, e.g., allowing parameter sharing in the
network. We then used the same evaluation method across all models and
obtained the evaluation measures to analyze and report the results.

####  4.2.1. Machine Learning models

ML models have previously been used extensively for SV-related tasks (e.g.,
(Ghaffarian and Shahriari, 2017; Spanos and Angelis, 2018; Le and Babar,
2022)). Specifically, we utilized three non-ensemble models: Logistic
Regression (LR) (Walker and Duncan, 1967), Support Vector Machine (SVM)
(Cortes and Vapnik, 1995), K-Nearest Neighbor (KNN) (Altman, 1992); and three
ensemble models: Random Forest (RF) (Ho, 1995), Extreme Gradient Boosting
(XGB) (Chen and Guestrin, 2016), and Light Gradient Boosting Machine (LGBM)
(Ke et al., 2017), based on the implementations provided by Le et al. (Le et
al., 2020b; Le and Babar, 2022). We optimized each model with the commonly
used hyperparameters listed in Table 3. These models used input data from four
different feature extraction methods, namely Bag-of-Tokens, Bag-of-Subtokens,
Word2Vec, and fastText. Bag-of-Tokens (BoT) originates from Bag-of-Words,
which is a popular feature extraction technique in the field of Natural
Language Processing. In this scenario, the frequency of each token was
accumulated after a process of customized tokenization where the syntax and
semantics of the code were retained. Bag-of-Subtokens (BoST) expands upon Bag-
of-Tokens, where code tokens were further divided into character sequences
known as sub-tokens. The length of the sub-tokens in our study ranged from two
to six characters. Word2Vec (Mikolov et al., 2013) captures the features of
any given token taking into account its neighboring tokens, allowing groups of
similar tokens to generate similar feature vectors. This improved upon the
previous two techniques, BoT and BoST, by incorporating code contexts. For
this approach, the vector sizes of 150, 300, 500 with the window sizes of 3,
4, 5 were chosen. fastText (Bojanowski et al., 2017) extends Word2Vec in a
similar fashion to Bag-of-Subtokens extending Bag-of-Words, where each token
was divided into its sub-tokens, while retaining the context-capturing ability
of Word2Vec. The final feature vectors of each code token were then aggregated
from the feature vectors of all sub-tokens. The settings for fastText we used
were a combination of sub-tokens lengths from Bag-of-Subtokens and the vector
sizes and window sizes of Word2Vec.

Model | Hyperparameters | Range / Options
---|---|---
Logistic Regression |  Regularization coefficient | 0.01, 0.1, 1, 10, 100
Support Vector Machine
K-Nearest Neighbors | No. of neighbors | 5, 11, 31, 51
Weights | uniform, distance
Distance Norm | 1, 2
Random Forest Extreme Gradient Boosting Light Gradient Boosting Machine | No. of estimators | 100, 200, 300, 400, 500
Max depth | 3, 5, 7, 9, unlimited
Max no. of leaf nodes | 100, 200, 300, unlimited

Table 3. Hyperparameters tuning for ML models.

Type | Model | Hyperparameters | Range / Options
---|---|---|---
Graph |  | GGNN
---
GCN
| Learning Rate
---
Adam Optimizer Epsilon
Hidden size
No. of GNN layers
| 5e-4 to 1e-1
---
1e-8 to 1e-4
32, 64, 128, 256, 512
1, 2, 3, 4, 5
| CNN |  | Learning Rate
---
Adam Optimizer Epsilon
Kernel Size
Padding
| 5e-4 to 1e-1
---
1e-8 to 1e-4
1, 3, 5, 7, 9
0, 1, 2, 3
Non-graph | LSTM |  | Learning Rate
---
Adam Optimizer Epsilon
No. of LSTM layers
| 5e-4 to 1e-1
---
1e-8 to 1e-4
1, 2, 3
| CodeBERT |  | Learning Rate
---
Adam Optimizer Epsilon
| 5e-4 to 1e-1
---
1e-8 to 1e-4

Table 4. Hyperparameters tuning for DL models. Note: Non-graph models include
sequence-based models (CNN and LSTM) and Transformer-based model (CodeBERT).

####  4.2.2. Deep Learning models

Similar to ML, multiple DL models have been developed for SV detection (Zhou
et al., 2019; Le et al., 2021b; Nguyen et al., 2022). In our implementations,
we mainly focused on the three key DL architectures/types that have been
widely used in the literature for function-level SV assessment, including
sequence-based, graph-based, and Transformer-based models.

Sequence-based models. For this category, we included Convolutional Neural
Network (CNN) (Kim, 2014), Long Short-term Memory (LSTM) (Hochreiter and
Schmidhuber, 1997). CNN and LSTM are standard models for text classification
tasks and can provide a robust baseline for measuring performance. The input
data was fed into an embedding layer, then processed by either a convolution
layer for CNN or an LSTM layer for LSTM, followed by a fully-connected linear
layer and a softmax function for classification of the CVSS metrics.

Graph-based models. As shown in section 2.3, there have been a number of
different graph-based deep architectures proposed for SV detection, most
notably Gated Graph Neural Networks (GGNN) (Li et al., 2015) and Graph
Convolutional Networks (GCN) (Kipf and Welling, 2016). To assess the
performance of these representative graph-based architectures for function-
level SV assessment tasks, we leveraged the publicly available implementations
of Nguyen et al. (Nguyen et al., 2022). These implementations have augmented
the graph neural networks with different sum and max poolings to produce
embedding graph features before feeding this to a fully-connected layer and
softmax layer for classification. With such augmentations, they have been
demonstrated to outperform existing ones using the same architectures like
Devign (Zhou et al., 2019) and ReVeaL (Chakraborty et al., 2021).

Transformer-based model. CodeBERT is a popular pre-trained model for
programming language and natural language that utilizes the Transformer
architecture. The key advantage of CodeBERT is the ability to capture more
context in the source code by producing different embedding vectors of the
same tokens based on the surrounding inputs and can retain the sub-tokens that
appear more frequently in the corpus via Byte-Pair Encoding (BPE) (Sennrich et
al., 2015). This has been proven to be more effective in dealing with source
code input (Feng et al., 2020). Notably, CodeBERT has achieved the state-of-
the-art predictive performance for function-level SV detection (Fu and
Tantithamthavorn, 2022; Steenhoek et al., 2023).

Implementation of multi-class DL models for function-level SV assessment. To
assess each CVSS metric, we built each DL model with its original architecture
and respective SV data (function source code and corresponding CVSS metric in
the curated dataset from section 4.1) as input. All of the models were also
tuned with a set of hyperparameters given in Table 4 to find the best-
performing combination for each task.

Customization of DL models to support multi-task learning. Instead of having
seven models to perform seven SV assessment tasks as in RQ1, in RQ2, we
employed multi-task learning to perform all the tasks at once, leveraging the
parameter-sharing ability of DL models (Zhang and Yang, 2021). In each DL
model, we modified the last layer to include a classification head, consisting
of seven linear layers, each representing a CVSS assessment task. The input
for a multi-task model was the function source code and all seven CVSS
metrics, and the output vector for a specific task $i$ was defined as:

$task_{i}=\omega_{t}x_{function}+b_{t}$

where $task_{i}$ is the output vector, $\omega_{t}$ is the learnable weights,
$x_{function}$ is the feature vectors, and $b_{t}$ is the learnable bias.

To compare this output with ground-truth labels, we then defined a loss
function that is the average/mean of all the cross-entropy losses of the
tasks, defined as:

$loss=mean(L)$ $L=\\{l_{1},\dots,l_{N}\\}$

where $N$ is the number of tasks from 1 to 7, and the loss for each task
$l_{n}$ is defined as:

$l_{n}=-w_{n}\left[y_{n}\cdot\log\sigma(x_{n})+(1-y_{n})\cdot\log(1-\sigma(x_{n}))\right]$

where for a given task $n$, $\omega_{n}$ is the weight of the loss with
respect to all loss, $y_{n}$ is the ground-truth target, and $x_{n}$ is the
predicted value. This multi-task loss was minimized using a stochastic
gradient descent method. These multi-task models were also tuned with the same
hyperparameters as their multi-class counterparts, and their hyperparameters
are listed in Table 4.

###  4.3. Model Evaluation

Evaluation technique. To evaluate these models, we randomly shuffled and split
the dataset of the collected vulnerable functions into training, validation,
and testing sets, with the ratios of 80:10:10, respectively. These ratios have
been widely adopted for function-level SV prediction, particularly when DL
models were employed (e.g., (Li et al., 2018, 2021a; Hin et al., 2022; Fu and
Tantithamthavorn, 2022)), similar to our study. We also stratified the data on
all CVSS metrics to ensure that the split sets of data contained roughly the
same proportion of metrics, which has been shown to be useful for defect/SV
prediction (Wattanakriengkrai et al., 2022). To increase the quality of the
evaluation, we also discarded the duplicate/inconsistent code functions, if
any, between the sets, as per the recommendations in the literature (Croft et
al., 2023). For RQ1, the function source code and each of the respective CVSS
metrics were used as inputs for the multi-class ML and DL models, resulting in
seven evaluation runs for each function. For RQ2, all seven CVSS metrics were
selected along with the function source code to be used as inputs for the
multi-task models. To find the best set of hyperparameters, grid search was
used on ML models, and Ray Tune (Liaw et al., 2018) with a randomized grid
search was used when evaluating the DL models. These hyperparameters are
listed in Table 3 and Table 4. The models with the highest performance on the
validation set were then selected as the optimal models, and their performance
on the testing set would be reported and analyzed in RQ1 and RQ2. We also did
not use class rebalancing techniques like random over/under-sampling because
they would influence SV predictive performance, making it challenging to focus
on evaluating the choice of data-driven models on the performance.

Evaluation measures. Matthews Correlation Coefficient (MCC) and Macro F1-Score
were used to quantify the performance of the SV assessment models. These were
widely used for SV assessment tasks (e.g., (Spanos and Angelis, 2018; Kudjo et
al., 2019; Le et al., 2019; Le and Babar, 2022)). F1-Score values range from 0
to 1. MCC considers all classes with values ranging from -1 (worst) to 1
(best), not just the positive class like F1-Score. Thus, we leveraged MCC as
the measure to select optimal models.

##  5\. Results

###  5.1. RQ1: How well do multi-class Machine Learning and Deep Learning
models perform SV assessment in C/C++?

RQ1 presented and compared the performance of the multi-class ML and multi-
class DL models, described in section 4.2, in predicting the seven CVSS
metrics for vulnerable functions in C/C++. The performance values of all of
these models are reported in Table 5.

Multi-class ML performance. The performance of ML models varied for different
CVSS metrics. The ranking of the tasks predicted by ML from high to low based
on MCC values was Access Vector, Integrity, Availability, Severity,
Authentication and Confidentiality, and Access Complexity. It should be noted
that while Access Vector had the highest rank based on MCC, its macro F1-Score
was quite low. Access Vector had the two majority classes, ‘Network’ and
‘Local’, which constituted 98.8% of all data. Due to the extremely small
amount of data, the ML models tended to produce more false-positive
predictions for the minority class (‘Adjacent Network’). This, in turn,
lowered the macro F1-Score that considered the F1-Score values of all the
classes equally. This result also supports our choice of MCC as the main
measure for choosing optimal models, as it better reflects the evaluation of
imbalanced classes.

We discovered several classifiers and feature extraction methods that
performed consistently better than the others on average. When averaging
across all tasks as shown in Fig. 4, ensemble classifiers (RF, XGB, and LGBM)
were overall better than non-ensemble ones. Among the ensemble models, LGBM
produced the best average MCC of 0.643, 15% higher than XGB and 13% higher
than RF. Similarly, K-NN performed 8% better than LR and 6% better than SVM on
average in the non-ensemble group. In terms of feature extraction methods,
features using sub-tokens generally produced higher results than their
counterparts using only tokens (BoST vs. BoT and fastText vs. Word2Vec). BoST
yielded the highest average performance with MCC of 0.626, 4% higher than BoT,
while fastText is 2% better when compared to Word2Vec. While the MCC-wise
comparisons were presented above, the F1-wise results were generally the same.
As a result, the use of LGBM with BoST produced the best overall performance
(MCC: 0.680, F1: 0.763), while the LR and Word2Vec combination yielded the
lowest performance (MCC: 0.469, F1: 0.614). The results suggest the potential
effectiveness when using the specific combination of LGBM and BoST for
function-level SV assessment. Therefore, we would use LGBM + BoST for further
analysis and comparison with DL models.

Figure 4. Average performance (MCC) of ML combinations (classifiers and
features) for function-level SV assessment.

Multi-class DL performance. Overall, we found that most DL models performed
similarly, except that CodeBERT achieved the best performance, which was
better than the others in terms of both MCC and F1-Score. Among the models
based on graph architectures, the GGNN models exhibited performance on par
with that of the GCN counterparts. In the non-graph category, the sequence-
based models, i.e., CNN and LSTM, demonstrated similar performance. Surpassing
both sequence-based (CNN and LSTM) and graph-based models (GGNN and GCN),
CodeBERT was the best-performing model with MCC and F1-Score values of 0.673
and 0.746, respectively. The ability to generalize to tokens not found in the
training data that CodeBERT (Feng et al., 2020) has likely allowed it to be
effective in function-level SV assessment.

Multi-class ML vs. Multi-class DL. The multi-class DL models showcased
slightly higher average performance, 2.5% in F1-Score and 5% in MCC, than the
ML models. However, the best ML model (LGBM + BoST) was actually on par with
the best DL model (CodeBERT). Access Vector was The only one of the seven
tasks in which CodeBERT surpassed LGBM + BoST. Access Vector was also the task
where all DL models exhibited strong performance and outperformed the best ML
model. All the remaining tasks, i.e., Confidentiality, Integrity,
Availability, Availability, and Severity, shared the same pattern in which no
DL model was able to pass the best ML model (LGBM + BoST). These findings show
that the multi-class DL models demonstrated better performance for tasks with
a higher degree of data imbalance, i.e., Access Vector, compared to the multi-
class ML models.

In terms of efficiency, as reported in Fig. 5, CodeBERT had to be trained
nearly five times longer, while the overall performance was still not as
effective as LGBM + BoST. Due to their complex architectures, maintaining
seven different DL models along with the tuned hyperparameters would also be
much more costly than maintaining seven ML models. In circumstances where
resources are limited, multi-class DL models may not be the best choice for
function-level SV assessment since they need to be trained/inferred on GPUs to
maximize their capabilities, and multi-class ML models requiring only CPU will
be much more efficient while still providing similar performance. Multi-task
learning can address the current challenge faced by the DL models, in which a
single DL model is trained to predict all SV assessment tasks simultaneously.
The results of multi-task learning for function-level SV assessment are
investigated in RQ2 (see section 5.2).

RQ1 Summary. Regarding multi-class ML models, ensemble models beat non-
ensemble ones, and sub-token features are better than token ones, resulting in
LGBM + BoST being the best ML model. Regarding multi-class DL models, CodeBERT
is the best, followed by the graph-based models and finally the sequence-based
models (CNN/LSTM) on average. Overall, the best ML model, LGBM + BoST, is
competitive for function-level SV assessment with comparable performance to
CodeBERT and significantly (75%) less training time.

###  5.2. RQ2: Can multi-task Deep Learning improve SV assessment in C++?

|  | Access
---
Vector
| Access
---
Complexity
| Authen-
---
tication
| Confiden-
---
tiality
Integrity | Availability | Severity |  | Average
---
(by Model)
Method CVSS Metric | F1 | MCC | F1 | MCC | F1 | MCC | F1 | MCC | F1 | MCC | F1 | MCC | F1 | MCC | F1 | MCC
Classifer | Feature | Multi-Class Machine Learning Models
| BoT | 0.689 | 0.789 | 0.702 | 0.566 | 0.791 | 0.591 | 0.746 | 0.618 | 0.748 | 0.621 | 0.693 | 0.586 | 0.743 | 0.643 | 0.730 | 0.630
| BoST | 0.740 | 0.810 | 0.734 | 0.574 | 0.804 | 0.608 | 0.761 | 0.631 | 0.766 | 0.653 | 0.726 | 0.638 | 0.759 | 0.656 | 0.756 | 0.653
| Word2Vec | 0.506 | 0.415 | 0.554 | 0.415 | 0.782 | 0.592 | 0.628 | 0.439 | 0.676 | 0.505 | 0.573 | 0.470 | 0.577 | 0.447 | 0.614 | 0.469
LR | fastText | 0.504 | 0.416 | 0.580 | 0.416 | 0.783 | 0.629 | 0.638 | 0.451 | 0.668 | 0.501 | 0.597 | 0.481 | 0.595 | 0.477 | 0.623 | 0.481
| BoT | 0.716 | 0.784 | 0.709 | 0.564 | 0.811 | 0.629 | 0.741 | 0.611 | 0.738 | 0.611 | 0.693 | 0.585 | 0.727 | 0.623 | 0.734 | 0.629
| BoST | 0.724 | 0.787 | 0.700 | 0.555 | 0.753 | 0.511 | 0.748 | 0.615 | 0.757 | 0.634 | 0.720 | 0.622 | 0.756 | 0.648 | 0.737 | 0.625
| Word2Vec | 0.632 | 0.730 | 0.561 | 0.419 | 0.773 | 0.592 | 0.629 | 0.441 | 0.668 | 0.503 | 0.570 | 0.465 | 0.563 | 0.447 | 0.628 | 0.514
SVM | fastText | 0.579 | 0.724 | 0.546 | 0.415 | 0.727 | 0.544 | 0.636 | 0.447 | 0.669 | 0.504 | 0.589 | 0.491 | 0.594 | 0.479 | 0.620 | 0.515
| BoT | 0.634 | 0.730 | 0.650 | 0.455 | 0.756 | 0.588 | 0.679 | 0.521 | 0.706 | 0.560 | 0.658 | 0.544 | 0.658 | 0.517 | 0.677 | 0.559
| BoST | 0.650 | 0.709 | 0.683 | 0.478 | 0.787 | 0.601 | 0.715 | 0.571 | 0.721 | 0.581 | 0.680 | 0.582 | 0.672 | 0.530 | 0.701 | 0.579
| Word2Vec | 0.714 | 0.800 | 0.719 | 0.582 | 0.763 | 0.560 | 0.745 | 0.612 | 0.745 | 0.618 | 0.729 | 0.635 | 0.731 | 0.624 | 0.735 | 0.633
K-NN | fastText | 0.727 | 0.795 | 0.760 | 0.626 | 0.763 | 0.560 | 0.751 | 0.619 | 0.772 | 0.658 | 0.738 | 0.649 | 0.735 | 0.642 | 0.749 | 0.650
| BoT | 0.587 | 0.749 | 0.531 | 0.481 | 0.808 | 0.667 | 0.623 | 0.458 | 0.673 | 0.553 | 0.599 | 0.561 | 0.594 | 0.567 | 0.631 | 0.576
| BoST | 0.638 | 0.750 | 0.602 | 0.516 | 0.853 | 0.738 | 0.660 | 0.502 | 0.706 | 0.591 | 0.634 | 0.582 | 0.606 | 0.584 | 0.671 | 0.609
| Word2Vec | 0.634 | 0.730 | 0.500 | 0.456 | 0.626 | 0.384 | 0.674 | 0.530 | 0.707 | 0.577 | 0.609 | 0.562 | 0.561 | 0.531 | 0.616 | 0.538
RF | fastText | 0.635 | 0.742 | 0.508 | 0.480 | 0.626 | 0.384 | 0.676 | 0.522 | 0.705 | 0.576 | 0.606 | 0.554 | 0.588 | 0.564 | 0.621 | 0.546
| BoT | 0.639 | 0.750 | 0.625 | 0.479 | 0.797 | 0.632 | 0.669 | 0.499 | 0.703 | 0.572 | 0.635 | 0.562 | 0.658 | 0.578 | 0.675 | 0.582
| BoST | 0.711 | 0.772 | 0.662 | 0.531 | 0.820 | 0.670 | 0.686 | 0.528 | 0.736 | 0.606 | 0.670 | 0.601 | 0.673 | 0.572 | 0.708 | 0.612
| Word2Vec | 0.710 | 0.750 | 0.568 | 0.419 | 0.718 | 0.502 | 0.653 | 0.476 | 0.655 | 0.488 | 0.611 | 0.510 | 0.578 | 0.445 | 0.642 | 0.513
XGB | fastText | 0.667 | 0.757 | 0.599 | 0.456 | 0.696 | 0.496 | 0.652 | 0.470 | 0.678 | 0.531 | 0.615 | 0.522 | 0.611 | 0.500 | 0.645 | 0.533
| BoT | 0.732 | 0.789 | 0.687 | 0.563 | 0.727 | 0.544 | 0.762 | 0.630 | 0.755 | 0.640 | 0.706 | 0.630 | 0.734 | 0.639 | 0.729 | 0.634
| BoST | 0.754 | 0.803 | 0.732 | 0.609 | 0.783 | 0.629 | 0.780 | 0.659 | 0.801 | 0.703 | 0.737 | 0.678 | 0.751 | 0.680 | 0.763 | 0.680
| Word2Vec | 0.702 | 0.811 | 0.695 | 0.585 | 0.718 | 0.502 | 0.737 | 0.599 | 0.750 | 0.629 | 0.701 | 0.624 | 0.729 | 0.650 | 0.719 | 0.628
LGBM | fastText | 0.697 | 0.795 | 0.694 | 0.598 | 0.688 | 0.451 | 0.743 | 0.603 | 0.770 | 0.659 | 0.723 | 0.653 | 0.737 | 0.659 | 0.722 | 0.631
| Average (by
---
CVSS Metric)
0.670 | 0.737 | 0.638 | 0.510 | 0.756 | 0.567 | 0.697 | 0.544 | 0.720 | 0.586 | 0.659 | 0.574 | 0.664 | 0.571 | 0.687 | 0.584
Type | Model | Multi-Class Deep Learning Models
| GGNN | 0.765 | 0.816 | 0.716 | 0.548 | 0.783 | 0.591 | 0.674 | 0.518 | 0.739 | 0.618 | 0.719 | 0.606 | 0.694 | 0.578 | 0.727 | 0.611
Graph- based | GCN | 0.826 | 0.808 | 0.734 | 0.594 | 0.785 | 0.572 | 0.738 | 0.606 | 0.717 | 0.579 | 0.681 | 0.551 | 0.671 | 0.549 | 0.736 | 0.608
| CNN | 0.619 | 0.844 | 0.484 | 0.452 | 0.763 | 0.538 | 0.728 | 0.595 | 0.775 | 0.669 | 0.754 | 0.670 | 0.451 | 0.407 | 0.653 | 0.596
| LSTM | 0.619 | 0.839 | 0.731 | 0.626 | 0.729 | 0.469 | 0.423 | 0.337 | 0.768 | 0.658 | 0.707 | 0.623 | 0.745 | 0.653 | 0.675 | 0.601
Non- Graph | CodeBERT | 0.619 | 0.845 | 0.739 | 0.618 | 0.800 | 0.602 | 0.765 | 0.649 | 0.749 | 0.637 | 0.763 | 0.691 | 0.786 | 0.669 | 0.746 | 0.673
| Average (by
---
CVSS Metric)
0.683 | 0.825 | 0.671 | 0.558 | 0.780 | 0.573 | 0.673 | 0.545 | 0.743 | 0.623 | 0.712 | 0.611 | 0.667 | 0.559 | 0.704 | 0.613
Type | Model | Multi-Task Deep Learning Models
| GGNN | 0.700 | 0.828 | 0.688 | 0.831 | 0.831 | 0.676 | 0.756 | 0.627 | 0.760 | 0.636 | 0.725 | 0.645 | 0.727 | 0.627 | 0.741 | 0.696
Graph- based | GCN | 0.614 | 0.874 | 0.764 | 0.831 | 0.831 | 0.664 | 0.750 | 0.674 | 0.753 | 0.642 | 0.750 | 0.666 | 0.728 | 0.616 | 0.741 | 0.709
| CNN | 0.734 | 0.853 | 0.734 | 0.843 | 0.843 | 0.707 | 0.766 | 0.651 | 0.762 | 0.652 | 0.793 | 0.708 | 0.772 | 0.677 | 0.772 | 0.727
| LSTM | 0.617 | 0.840 | 0.677 | 0.805 | 0.805 | 0.663 | 0.733 | 0.599 | 0.756 | 0.639 | 0.710 | 0.605 | 0.740 | 0.634 | 0.720 | 0.683
Non- Graph | CodeBERT | 0.851 | 0.857 | 0.785 | 0.840 | 0.841 | 0.680 | 0.799 | 0.650 | 0.802 | 0.684 | 0.817 | 0.711 | 0.845 | 0.715 | 0.820 | 0.734
| Average (by
---
CVSS Metric)
0.727 | 0.846 | 0.741 | 0.827 | 0.828 | 0.669 | 0.767 | 0.638 | 0.773 | 0.654 | 0.762 | 0.672 | 0.770 | 0.655 | 0.759 | 0.709
Table 5. Testing performance of different Machine Learning and Deep Learning
models for CVSS-based function-level SV assessment. Notes: Values highlighted
in grey are best task-wise for each method. Values in blue belong to the best
ML model, i.e., LGBM + BoST.

Instead of developing multi-class models for each of the seven CVSS metrics as
in RQ1, in RQ2, we built unified DL models with multi-task learning (see
4.2.2). In this scenario, each model predicted all the function-level SV
assessment outputs in a single model. This setting is expected to improve the
overall effectiveness and efficiency of the DL models. We compared the multi-
task performance with that of the best multi-class models in RQ1 (see Table
5).

Figure 5. Training time of CodeBERT, multi-task CodeBERT and LGBM + BoST (in
seconds). Note: CodeBERT multi-task does not have training time for each task.

In the multi-task training scenario, CodeBERT was still the most effective
model, outperforming the best ML model (LGBM + BoST) by 8% in both MCC and
F1-Score. For individual tasks, the multi-task CodeBERT variant outperformed
LGBM + BoST in five tasks: Access Vector, Access Complexity, Authentication,
Availability, and Severity, with MCC increases of 7%, 38%, 8%, 5%, and 5%,
respectively. For the remaining Confidentiality and Integrity tasks, the
multi-task CodeBERT model still performed reasonably well within 3% of the MCC
value of the ML model. These improvements imply that multi-task learning is
particularly useful for the tasks with more imbalanced data, i.e., Access
Vector, Access Complexity, and Authentication. It is worth noting that similar
to RQ1, the multi-task variants of CNN, LSTM, GGNN, and GCN attained similar
levels of performance for all the SV assessment tasks.

Further, multi-task learning was beneficial to all sequence-based, graph-
based, and Transformer-based DL models in terms of both MCC and F1-Score (see
Table 5). MCC improved significantly at 22%, 14%, 14%, 17%, and 9%, for CNN,
LSTM, GGNN, GCN, and CodeBERT, respectively. The multi-task implementation
also demonstrated better MCC values than the multi-class counterparts across
all the tasks. Specifically, the improvements were 3%, 48%, 17%, 17%, 5%, 10%,
and 17% for Access Vector, Access Complexity, Authentication, Confidentiality,
Integrity, Availability, and Severity, respectively. The improvement for
Access Vector was the least, probably because DL models already performed best
for this task, as shown in RQ1. It is also important to note that the best
multi-task CodeBERT model cut down the total training time of the multi-class
CodeBERT model in RQ1 by nearly 23%, as illustrated in Fig. 5. This result
highlights the better efficiency of multi-task learning than training seven
separate multi-class DL models. Note that ML is still more efficient, yet not
performing as well, when compared to multi-task learning.

RQ2 Summary. Multi-task DL models achieve performance improvements of 9–22% in
MCC compared to their multi-class counterparts. Multi-task CodeBERT is the
best model overall and outperforms the best ML model (LGBM + BoST) by 8% in
MCC averaging across the tasks. The findings suggest that multi-task learning
can be employed to increase both effectiveness and efficiency.

##  6\. Discussion

###  6.1. Comparison with the literature and beyond

ML performance. Our findings in RQ1 that sub-tokens performed better than
tokens alone and ensemble classifiers were superior to non-ensemble ones align
with the previous findings of SV assessment using SV reports (Spanos and
Angelis, 2018; Le et al., 2019). In addition, given that Le et al. (Le and
Babar, 2022) also used ML for function-level SV assessment in Java (not
C/C++), we also compare our findings with theirs. We found similar results to
the previous study (Le and Babar, 2022), where LGBM and BoST are the best
overall combination to assess SVs at the function level. This shows the
possibility of reusing the knowledge and the ML model developed for one
language to another language for function-level SV assessment. However, the
ranking of the seven CVSS metrics predicted by ML, in terms of MCC, in the
previous study did not exactly match ours. Among the CVSS metrics, the rank of
Authentication was changed the most significantly, i.e., from rank 2 in the
previous study to rank 6 in our study. This is likely due to distinctive input
code structures and semantics (C/C++ vs. Java) and different output class
distributions fed into the models. For example, for Authentication, the
decrease in the rank is likely due to the much smaller proportion (2%) of the
minority class compared to that (21.8%) in the previous study. Such
differences reinforce the need to (re-)evaluate existing models in a different
language than the one for which they were originally experimented.

DL performance. Fu et al. (Fu and Tantithamthavorn, 2022) compared CodeBERT
with graph-based models, including GGNN and GCN, for function-level SV
prediction and showed that CodeBERT was the best overall model. Zhou et al.
(Zhou et al., 2019) and Nguyen et al. (Nguyen et al., 2022) also showed that
graph-based models performed better than convolutional and recurrent models
with a noticeable difference. All of these agree with our findings in RQ1 and
RQ2. In brief, the ranking in decreasing performance for C/C++ function level
SV assessment are as follows: Transformer-based models (CodeBERT) ¿ Graph-
based models (GGNN and GCN) ¿ Sequence-based models (CNN and LSTM). However,
these prior studies have not investigated the use of multi-task learning as
the output nature of SV detection and SV assessment is different. Our study
shows that using DL models with multi-task learning is more suitable than the
conventional multi-class approach for function-level SV assessment in C/C++,
improving both the effectiveness and efficiency of the tasks.

ML versus DL. While previous studies have shown that ML can be inferior to DL
for SV prediction, this does not necessarily translate into function-level SV
assessment in C/C++. For instance, Fu et al. (Fu and Tantithamthavorn, 2022)
claimed that RF + BoW was much less effective than CodeBERT for function-level
SV detection. While our experiments confirmed the previous finding that RF +
BoW was worse than CodeBERT, we still found some ML models that performed
competitively compared to DL. Particularly, in the same multi-class setting,
we showed that LGBM + BoST performed similarly to CodeBERT in both F1-Score
(0.763 versus 0.746) and MCC (0.680 versus 0.673), while requiring only 1/4 of
the training time. These findings actually echo the recent recommendations in
the field of Software Engineering that a carefully designed and tuned ML model
can still outperform more advanced DL models while saving a significant amount
of cost (e.g., (Majumder et al., 2018; Xu et al., 2018)). For function-level
SV detection, the demonstrated promising results can inspire future research
to consider LGBM + BoST instead of the traditional RF + BoW as a baseline to
compare with more sophisticated DL models proposed for the task. For function-
level SV assessment, multi-task DL models are still preferred if researchers
or practitioners want to optimize (baseline) performance for testing/comparing
with newly proposed SV assessment models/techniques. On the other hand, LGBM +
BoST can still be used to achieve reasonable performance with much fewer
resources. This is particularly useful when practitioners need to routinely
retrain the models in continuous development environments with frequent
software releases (Arani et al., 2024).

###  6.2. Threats to validity

The first threat concerns the implementation of existing models. We ensured
the robustness of our implementation by following the instructions given in
previous studies and only reused the validated and reviewed reproduction
packages.

The second threat pertains to our conversion of the multi-class DL models to
the multi-task variants. By only changing the output layers and the loss
functions to accommodate the new outputs in each model, we ensured the
validity of the original models, while allowing us to examine their
performance without affecting the original architectures.

The third threat involves the optimization of the models. It is widely known
that it is not practically possible to experiment with all possible
combinations of hyperparameters. Instead, we relied on related work and
focused on the options that had been demonstrated to be suitable and effective
for SV-related tasks.

Another threat is the generalizability of our study. Our findings may not
generalize to all the projects in C/C++; however, our dataset comes from an
established source (Fan et al., 2020), and the code functions are derived from
300+ real projects in a variety of applications and domains. The extensiveness
and diversity of the dataset also allow for robust analysis, reducing the
randomness in our results.

##  7\. Conclusions and Future Work

We demonstrated the need and potential for using data-driven approaches for
function-level SV assessment to streamline the SV fixing process in C/C++.
Using a customized SV dataset in C/C++, we investigated and compared the
performance of various ML and DL models to predict seven CVSS metrics to
assess SVs in code functions in these languages. When building separate models
for individual tasks, we showed that ML was a competitive approach to DL for
function-level SV assessment with comparable performance and less training
time. We also illustrated a better way to use DL models through multi-task
learning for SV assessment tasks, which presented improvements of 8–22% in MCC
over the multi-class counterparts. Our findings give recommendations on what
and how ML and DL models can be used for function-level SV assessment. We also
show the possibility of reusing data-driven models between SV assessment and
SV detection, which can draw better synergies between the two important tasks.
Overall, our work provides data and competitive models that can be used for
future research to benchmark next data-driven advances for automating SV
assessment in particular and SV management in general.

## References

  * (1)
  * Altman (1992) Naomi S Altman. 1992.  An introduction to kernel and nearest-neighbor nonparametric regression.  _The American Statistician_ 46, 3 (1992), 175–185.
  * Arani et al. (2024) Ali Kazemi Arani, Triet Huynh Minh Le, Mansooreh Zahedi, and M Ali Babar. 2024.  Systematic literature review on application of learning-based approaches in continuous integration.  _IEEE Access_ (2024).
  * Authors (2024) Authors. 2024.  Reproduction package.  https://github.com/anh56/dl4sa
  * Bhandari et al. (2021) Guru Bhandari, Amara Naseer, and Leon Moonen. 2021.  CVEfixes: automated collection of vulnerabilities and their fixes from open-source software. In _Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering_. 30–39.
  * Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017.  Enriching word vectors with subword information.  _Transactions of the Association for Computational Linguistics_ 5 (2017), 135–146.
  * Carvalho et al. (2014) Marco Carvalho, Jared DeMott, Richard Ford, and David A Wheeler. 2014.  Heartbleed 101.  _IEEE security & privacy_ 12, 4 (2014), 63–67.
  * Chakraborty et al. (2021) Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. 2021.  Deep learning based vulnerability detection: Are we there yet?  _IEEE Transactions on Software Engineering_ 48, 9 (2021), 3280–3296.
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  Xgboost: A scalable tree boosting system. In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_. 785–794.
  * Chen et al. (2023) Yizheng Chen, Zhoujie Ding, Lamya Alowain, Xinyun Chen, and David Wagner. 2023.  Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection. In _Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses_. 654–668.
  * Cheng et al. (2021) Xiao Cheng, Haoyu Wang, Jiayi Hua, Guoai Xu, and Yulei Sui. 2021.  Deepwukong: Statically detecting software vulnerabilities using deep graph neural network.  _ACM Transactions on Software Engineering and Methodology (TOSEM)_ 30, 3 (2021), 1–33.
  * Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning phrase representations using RNN encoder-decoder for statistical machine translation.  _arXiv preprint arXiv:1406.1078_ (2014).
  * Cortes and Vapnik (1995) Corinna Cortes and Vladimir Vapnik. 1995.  Support-vector networks.  _Machine learning_ 20, 3 (1995), 273–297.
  * Croft et al. (2023) Roland Croft, M Ali Babar, and M Mehdi Kholoosi. 2023.  Data quality for software vulnerability datasets. In _2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)_. IEEE, 121–133.
  * Dam et al. (2017) Hoa Khanh Dam, Truyen Tran, Trang Pham, Shien Wee Ng, John Grundy, and Aditya Ghose. 2017.  Automatic feature learning for vulnerability prediction.  _arXiv preprint arXiv:1708.02368_ (2017).
  * Dam et al. (2018) Hoa Khanh Dam, Truyen Tran, Trang Pham, Shien Wee Ng, John Grundy, and Aditya Ghose. 2018.  Automatic feature learning for predicting vulnerable software components.  _IEEE Transactions on Software Engineering_ 47, 1 (2018), 67–85.
  * Deng et al. (2024) Yang Deng, Bangchao Wang, Qiang Zhu, Junping Liu, Jiewen Kuang, and Xingfu Li. 2024.  MTLink: Adaptive multi-task learning based pre-trained language model for traceability link recovery between issues and commits.  _Journal of King Saud University-Computer and Information Sciences_ 36, 2 (2024), 101958.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Ding et al. (2022) Yangruibo Ding, Sahil Suneja, Yunhui Zheng, Jim Laredo, Alessandro Morari, Gail Kaiser, and Baishakhi Ray. 2022.  VELVET: a noVel Ensemble Learning approach to automatically locate VulnErable sTatements. In _2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)_. IEEE, 959–970.
  * Duan et al. (2021) Xuanyu Duan, Mengmeng Ge, Triet Huynh Minh Le, Faheem Ullah, Shang Gao, Xuequan Lu, and M Ali Babar. 2021.  Automated security assessment for the internet of things. In _2021 IEEE 26th Pacific Rim International Symposium on Dependable Computing (PRDC)_. IEEE, 47–56.
  * Durumeric et al. (2014) Zakir Durumeric, Frank Li, James Kasten, Johanna Amann, Jethro Beekman, Mathias Payer, Nicolas Weaver, David Adrian, Vern Paxson, Michael Bailey, et al. 2014\.  The matter of heartbleed. In _Proceedings of the 2014 conference on internet measurement conference_. 475–488.
  * Fan et al. (2020) Jiahao Fan, Yi Li, Shaohua Wang, and Tien N Nguyen. 2020.  A C/C++ code vulnerability dataset with code changes and CVE summaries. In _Proceedings of the 17th International Conference on Mining Software Repositories_. 508–512.
  * Feng et al. (2020) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020\.  Codebert: A pre-trained model for programming and natural languages.  _arXiv preprint arXiv:2002.08155_ (2020).
  * Filus et al. (2021) Katarzyna Filus, Miltiadis Siavvas, Joanna Domańska, and Erol Gelenbe. 2021.  The random neural network as a bonding model for software vulnerability prediction. In _Modelling, Analysis, and Simulation of Computer and Telecommunication Systems: 28th International Symposium, MASCOTS 2020, Nice, France, November 17–19, 2020, Revised Selected Papers 28_. Springer, 102–116.
  * FIRST ([n. d.]) FIRST. [n. d.].  Common Vulnerability Scoring System.  Retrieved July 23, 2024 from https://www.first.org/cvss/
  * FIRST (2024) FIRST. 2024.  CVSS version 2.  Retrieved July 23, 2024 from https://www.first.org/cvss/v2/guide
  * Fu and Tantithamthavorn (2022) Michael Fu and Chakkrit Tantithamthavorn. 2022.  Linevul: A transformer-based line-level vulnerability prediction. In _Proceedings of the 19th International Conference on Mining Software Repositories_. 608–620.
  * Future (2024) Recorded Future. 2024.  Exploiting old vulnerabilities.  Retrieved July 23, 2024 from https://www.recordedfuture.com/exploiting-old-vulnerabilities/
  * Ghaffarian and Shahriari (2017) Seyed Mohammad Ghaffarian and Hamid Reza Shahriari. 2017.  Software vulnerability analysis and discovery using machine-learning and data-mining techniques: A survey.  _ACM Computing Surveys (CSUR)_ 50, 4 (2017), 1–36.
  * Ghaffarian and Shahriari (2021) Seyed Mohammad Ghaffarian and Hamid Reza Shahriari. 2021.  Neural software vulnerability analysis using rich intermediate graph representations of programs.  _Information Sciences_ 553 (2021), 189–207.
  * Gong et al. (2019) Xi Gong, Zhenchang Xing, Xiaohong Li, Zhiyong Feng, and Zhuobing Han. 2019.  Joint prediction of multiple vulnerability characteristics through multi-task learning. In _2019 24th International Conference on Engineering of Complex Computer Systems (ICECCS)_. IEEE, 31–40.
  * Han et al. (2017) Zhuobing Han, Xiaohong Li, Zhenchang Xing, Hongtao Liu, and Zhiyong Feng. 2017.  Learning to predict severity of software vulnerability using only vulnerability description. In _2017 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE, 125–136.
  * Hanif et al. (2021) Hazim Hanif, Mohd Hairul Nizam Md Nasir, Mohd Faizal Ab Razak, Ahmad Firdaus, and Nor Badrul Anuar. 2021.  The rise of software vulnerability: Taxonomy of software vulnerabilities detection and machine learning approaches.  _Journal of Network and Computer Applications_ (2021), 103009.
  * Harzevili et al. (2023) Nima Shiri Harzevili, Alvine Boaye Belle, Junjie Wang, Song Wang, Zhen Ming, Nachiappan Nagappan, et al. 2023\.  A survey on automated software vulnerability detection using machine learning and deep learning.  _arXiv preprint arXiv:2306.11673_ (2023).
  * Hin et al. (2022) David Hin, Andrey Kan, Huaming Chen, and M Ali Babar. 2022.  Linevd: Statement-level vulnerability detection using graph neural networks. In _Proceedings of the 19th international conference on mining software repositories_. 596–607.
  * Ho (1995) Tin Kam Ho. 1995.  Random decision forests. In _3rd International Conference on Document Analysis and Recognition_ , Vol. 1. IEEE, 278–282.
  * Hoang et al. (2019) Thong Hoang, Hoa Khanh Dam, Yasutaka Kamei, David Lo, and Naoyasu Ubayashi. 2019.  Deepjit: an end-to-end deep learning framework for just-in-time defect prediction. In _2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)_. IEEE, 34–45.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Karpathy et al. (2015) Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2015.  Visualizing and understanding recurrent networks.  _arXiv preprint arXiv:1506.02078_ (2015).
  * Ke et al. (2017) Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017.  Lightgbm: A highly efficient gradient boosting decision tree.  _Advances in Neural Information Processing Systems_ 30 (2017), 3146–3154.
  * Khan and Parkinson (2018) Saad Khan and Simon Parkinson. 2018.  Review into state of the art of vulnerability assessment using artificial intelligence.  In _Guide to Vulnerability Analysis for Computer Networks and Systems_. Springer, 3–32.
  * Khemani et al. (2024) Bharti Khemani, Shruti Patil, Ketan Kotecha, and Sudeep Tanwar. 2024.  A review of graph neural networks: concepts, architectures, techniques, challenges, datasets, applications, and future directions.  _Journal of Big Data_ 11, 1 (2024), 18.
  * Kim (2014) Yoon Kim. 2014.  Convolutional neural networks for sentence classification. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 1746–1751.
  * Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016.  Semi-supervised classification with graph convolutional networks.  _arXiv preprint arXiv:1609.02907_ (2016).
  * Kudjo et al. (2019) Patrick Kwaku Kudjo, Jinfu Chen, Minmin Zhou, Solomon Mensah, and Rubing Huang. 2019.  Improving the accuracy of vulnerability report classification using term frequency-inverse gravity moment. In _2019 IEEE 19th International Conference on Software Quality, Reliability and Security (QRS)_. IEEE, 248–259.
  * Kyatam et al. (2017) Shashank Kyatam, Abdullah Alhayajneh, and Thaier Hayajneh. 2017.  Heartbleed attacks implementation and vulnerability. In _2017 IEEE Long Island Systems, Applications and Technology Conference (LISAT)_. IEEE, 1–6.
  * Le (2022) Triet HM Le. 2022.  Towards an improved understanding of software vulnerability assessment using data-driven approaches.  _arXiv preprint arXiv:2207.11708_ (2022).
  * Le and Babar (2024) Triet HM Le and M Ali Babar. 2024.  Mitigating data imbalance for software vulnerability assessment: Does data augmentation help?  _arXiv preprint arXiv:2407.10722_ (2024).
  * Le et al. (2020a) Triet HM Le, Hao Chen, and Muhammad Ali Babar. 2020a.  Deep learning for source code modeling and generation: Models, applications, and challenges.  _ACM Computing Surveys (CSUR)_ 53, 3 (2020), 1–38.
  * Le et al. (2022) Triet HM Le, Huaming Chen, and M Ali Babar. 2022.  A survey on data-driven software vulnerability assessment and prioritization.  _Comput. Surveys_ 55, 5 (2022), 1–39.
  * Le and Babar (2022) Triet Huynh Minh Le and M Ali Babar. 2022.  On the use of fine-grained vulnerable code statements for software vulnerability assessment models. In _Proceedings of the 19th International Conference on Mining Software Repositories_. 621–633.
  * Le et al. (2024a) Triet Huynh Minh Le, M Ali Babar, and Tung Hoang Thai. 2024a.  Software vulnerability prediction in low-resource languages: An empirical study of codebert and chatgpt. In _Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering_. 679–685.
  * Le et al. (2021a) Triet Huynh Minh Le, Roland Croft, David Hin, and Muhammad Ali Babar. 2021a.  A large-scale study of security vulnerability support on developer q&a websites. In _Proceedings of the 25th International Conference on Evaluation and Assessment in Software Engineering_. 109–118.
  * Le et al. (2024b) Triet Huynh Minh Le, Xiaoning Du, and M Ali Babar. 2024b.  Are latent vulnerabilities hidden gems for software vulnerability prediction? An empirical study. In _2024 IEEE/ACM 21st International Conference on Mining Software Repositories (MSR)_. IEEE, 716–727.
  * Le et al. (2020b) Triet Huynh Minh Le, David Hin, Roland Croft, and M Ali Babar. 2020b.  Puminer: Mining security posts from developer question and answer websites with pu learning. In _Proceedings of the 17th International Conference on Mining Software Repositories_. 350–361.
  * Le et al. (2021b) Triet Huynh Minh Le, David Hin, Roland Croft, and M Ali Babar. 2021b.  Deepcva: Automated commit-level vulnerability assessment with deep multi-task learning. In _2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 717–729.
  * Le et al. (2019) Triet Huynh Minh Le, Bushra Sabir, and Muhammad Ali Babar. 2019.  Automated software vulnerability assessment with concept drift. In _2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)_. 371–382.
  * Li and Paxson (2017) Frank Li and Vern Paxson. 2017.  A large-scale empirical study of security patches. In _2017 ACM SIGSAC Conference on Computer and Communications Security_. 2201–2215.
  * Li et al. (2017) Jian Li, Pinjia He, Jieming Zhu, and Michael R Lyu. 2017.  Software defect prediction via convolutional neural network. In _2017 IEEE international conference on software quality, reliability and security (QRS)_. IEEE, 318–328.
  * Li et al. (2023) Litao Li, Steven HH Ding, Yuan Tian, Benjamin CM Fung, Philippe Charland, Weihan Ou, Leo Song, and Congwei Chen. 2023.  VulANalyzeR: Explainable binary vulnerability detection with multi-task learning and attentional graph convolution.  _ACM Transactions on Privacy and Security_ 26, 3 (2023), 1–25.
  * Li et al. (2015) Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. 2015.  Gated graph sequence neural networks.  _arXiv preprint arXiv:1511.05493_ (2015).
  * Li et al. (2021a) Yi Li, Shaohua Wang, and Tien N. Nguyen. 2021a.  Vulnerability detection with fine-grained interpretations. In _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 292–303.
  * Li et al. (2021b) Zhen Li, Deqing Zou, Shouhuai Xu, Hai Jin, Yawei Zhu, and Zhaoxuan Chen. 2021b.  Sysevr: A framework for using deep learning to detect software vulnerabilities.  _IEEE Transactions on Dependable and Secure Computing_ 19, 4 (2021), 2244–2258.
  * Li et al. (2018) Zhen Li, Deqing Zou, Shouhuai Xu, Xinyu Ou, Hai Jin, Sujuan Wang, Zhijun Deng, and Yuyi Zhong. 2018.  Vuldeepecker: A deep learning-based system for vulnerability detection.  _arXiv preprint arXiv:1801.01681_ (2018).
  * Liaw et al. (2018) Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica. 2018.  Tune: A research platform for distributed model selection and training.  _arXiv preprint arXiv:1807.05118_ (2018).
  * Lin et al. (2020a) Guanjun Lin, Sheng Wen, Qing-Long Han, Jun Zhang, and Yang Xiang. 2020a.  Software vulnerability detection using deep neural networks: a survey.  _Proc. IEEE_ 108, 10 (2020), 1825–1848.
  * Lin et al. (2020b) Guanjun Lin, Wei Xiao, Jun Zhang, and Yang Xiang. 2020b.  Deep learning-based vulnerable function detection: A benchmark. In _International Conference on Information and Communications Security_. 219–232.
  * Liu et al. (2020a) Fang Liu, Ge Li, Bolin Wei, Xin Xia, Zhiyi Fu, and Zhi Jin. 2020a.  A self-attentional neural architecture for code completion with multi-task learning. In _Proceedings of the 28th International Conference on Program Comprehension_. 37–47.
  * Liu et al. (2020b) Fang Liu, Ge Li, Yunfei Zhao, and Zhi Jin. 2020b.  Multi-task learning based pre-trained language model for code completion. In _Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering_. 473–485.
  * Liu et al. (2023) Meng Liu, Xiaohui Han, Wenbo Zuo, Xuejiao Luo, and Lei Guo. 2023.  An interpretable vulnerability detection framework based on multi-task learning. In _International Conference on Neural Information Processing_. Springer, 228–242.
  * Liu et al. (2019) Shigang Liu, Guanjun Lin, Qing-Long Han, Sheng Wen, Jun Zhang, and Yang Xiang. 2019.  DeepBalance: Deep-learning and fuzzy oversampling for vulnerability detection.  _IEEE Transactions on Fuzzy Systems_ 28, 7 (2019), 1329–1343.
  * Liu et al. (2020c) Shigang Liu, Guanjun Lin, Lizhen Qu, Jun Zhang, Olivier De Vel, Paul Montague, and Yang Xiang. 2020c.  CD-VulD: Cross-domain vulnerability discovery based on deep domain adaptation.  _IEEE Transactions on Dependable and Secure Computing_ 19, 1 (2020), 438–451.
  * Majumder et al. (2018) Suvodeep Majumder, Nikhila Balaji, Katie Brey, Wei Fu, and Tim Menzies. 2018.  500+ times faster than deep learning: A case study exploring faster methods for text mining stackoverflow. In _Proceedings of the 15th International Conference on Mining Software Repositories_. 554–563.
  * Mastropaolo et al. (2021) Antonio Mastropaolo, Simone Scalabrino, Nathan Cooper, David Nader Palacio, Denys Poshyvanyk, Rocco Oliveto, and Gabriele Bavota. 2021.  Studying the usage of text-to-text transfer transformer to support code-related tasks. In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. IEEE, 336–347.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Distributed representations of words and phrases and their compositionality.  _arXiv preprint arXiv:1310.4546_ (2013).
  * Nguyen et al. (2022) Van-Anh Nguyen, Dai Quoc Nguyen, Van Nguyen, Trung Le, Quan Hung Tran, and Dinh Phung. 2022.  Regvd: Revisiting graph neural networks for vulnerability detection. In _Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings_. 178–182.
  * NIST (2024a) NIST. 2024a.  National Vulnerability Database.  Retrieved July 23, 2024 from https://nvd.nist.gov/
  * NIST (2024b) NIST. 2024b.  National Vulnerability Database CVSS qualitative severity ratings.  Retrieved July 23, 2024 from https://nvd.nist.gov/vuln-metrics/cvss#:~:text=CVSS%20v4.0%20Calculator-,Qualitative%20Severity%20Ratings,-CVSS%20v2.0%20Ratings
  * Piantadosi et al. (2019) Valentina Piantadosi, Simone Scalabrino, and Rocco Oliveto. 2019.  Fixing of security vulnerabilities in open source projects: A case study of apache http server and apache tomcat. In _2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST)_. IEEE, 68–78.
  * Russell et al. (2018) Rebecca Russell, Louis Kim, Lei Hamilton, Tomo Lazovich, Jacob Harer, Onur Ozdemir, Paul Ellingwood, and Marc McConley. 2018.  Automated vulnerability detection in source code using deep representation learning. In _2018 17th IEEE international conference on machine learning and applications (ICMLA)_. IEEE, 757–762.
  * Sennrich et al. (2015) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015.  Neural machine translation of rare words with subword units.  _arXiv preprint arXiv:1508.07909_ (2015).
  * Shetty et al. (2021) Manish Shetty, Chetan Bansal, Sumit Kumar, Nikitha Rao, Nachiappan Nagappan, and Thomas Zimmermann. 2021.  Neural knowledge extraction from cloud service incidents. In _2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)_. IEEE, 218–227.
  * Smyth (2017) Vincent Smyth. 2017.  Software vulnerability management: how intelligence helps reduce the risk.  _Network Security_ 2017, 3 (2017), 10–12.
  * Spanos and Angelis (2018) Georgios Spanos and Lefteris Angelis. 2018.  A multi-target approach to estimate software vulnerability characteristics and severity scores.  _Journal of Systems and Software_ 146 (2018), 152–166.
  * Steenhoek et al. (2023) Benjamin Steenhoek, Md Mahbubur Rahman, Richard Jiles, and Wei Le. 2023.  An empirical study of deep learning models for vulnerability detection. In _Proceedings of the 45th International Conference on Software Engineering_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Walker and Duncan (1967) Strother H Walker and David B Duncan. 1967.  Estimation of the probability of an event as a function of several independent variables.  _Biometrika_ 54, 1-2 (1967), 167–179.
  * Wang et al. (2020) Huanting Wang, Guixin Ye, Zhanyong Tang, Shin Hwei Tan, Songfang Huang, Dingyi Fang, Yansong Feng, Lizhong Bian, and Zheng Wang. 2020.  Combining graph-based learning with automated data collection for code vulnerability detection.  _IEEE Transactions on Information Forensics and Security_ 16 (2020), 1943–1958.
  * Wang et al. (2024) Xu Wang, Hongwei Yu, Xiangxin Meng, Hongliang Cao, Hongyu Zhang, Hailong Sun, Xudong Liu, and Chunming Hu. 2024.  MTL-TRANSFER: Leveraging multi-task learning and transferred knowledge for improving fault localization and program repair.  _ACM Transactions on Software Engineering and Methodology_ (2024).
  * Wattanakriengkrai et al. (2022) Supatsara Wattanakriengkrai, Patanamon Thongtanunam, Chakkrit Tantithamthavorn, Hideaki Hata, and Kenichi Matsumoto. 2022.  Predicting defective lines using a model-agnostic technique.  _IEEE Transactions on Software Engineering_ 48, 5 (2022), 1480–1496.
  * Xu et al. (2018) Bowen Xu, Amirreza Shirani, David Lo, and Mohammad Amin Alipour. 2018.  Prediction of relatedness in stack overflow: deep learning vs. svm: a reproducibility study. In _Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement_. 1–10.
  * Yan et al. (2021) Han Yan, Senlin Luo, Limin Pan, and Yifei Zhang. 2021.  HAN-BSVD: a hierarchical attention network for binary software vulnerability detection.  _Computers & Security_ 108 (2021), 102286.
  * Zhang and Yang (2021) Yu Zhang and Qiang Yang. 2021.  A survey on multi-task learning.  _IEEE Transactions on Knowledge and Data Engineering_ 34, 12 (2021), 5586–5609.
  * Zhou et al. (2021) Jiayuan Zhou, Michael Pacheco, Zhiyuan Wan, Xin Xia, David Lo, Yuan Wang, and Ahmed E Hassan. 2021.  Finding a needle in a haystack: Automated mining of silent vulnerability fixes. In _2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)_.
  * Zhou et al. (2019) Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. 2019.  Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks.  _arXiv preprint arXiv:1909.03496_ (2019).
  * Zou et al. (2021) Deqing Zou, Yawei Zhu, Shouhuai Xu, Zhen Li, Hai Jin, and Hengkai Ye. 2021.  Interpreting deep learning-based vulnerability detector predictions based on heuristic searching.  _ACM Transactions on Software Engineering and Methodology (TOSEM)_ 30, 2 (2021), 1–31.