[datatype=bibtex] article inproceedings incollection book techreport phdthesis
[fieldset=url, null] [fieldset=urldate, null] [datatype=bibtex]
[fieldsource=booktitle, match={{, replace=] [fieldsource=booktitle, match=}},
replace=] [fieldsource=booktitle, match=^Proceedings+of+the+§++, replace=]
[fieldsource=booktitle, match=^ṃ̣issingṃ̣issing+, replace=]
[fieldsource=booktitle, match=^[]̣1,2th+, replace=] [fieldsource=booktitle,
match=^[]̣1,2nd+, replace=] [fieldsource=booktitle, match=International,
replace=Int’l] [fieldsource=booktitle,
match=[{]0,2Conference[}]0,2+on(+the)?+, replace=Conf.
20] [fieldsource=booktitle, match=Conference , replace=Conf. ]
[fieldsource=booktitle, match=[{]0,2Symposium[}]0,2+on(+the)?+, replace=Symp.
20] [fieldsource=booktitle, match=Symposium , replace=Symp. ]
[fieldsource=booktitle, match=[{]0,2Workshop[s]?[}]0,2+on(+the)?+, replace=Ws.
20] [fieldsource=booktitle, match=Workshop[s]?, replace=Ws. ]
[fieldsource=booktitle, match=[{]0,2Foundations[}]0,2+of(+the)?+,
replace=Found.
20] [fieldsource=booktitle, match=Automated , replace=Autom. ]
[fieldsource=booktitle, match=Software , replace=Softw. ]
[fieldsource=booktitle, match=Requirements , replace=Req. ]
[fieldsource=booktitle, match=Engineering, replace=Eng.]
[fieldsource=booktitle, match=Applications, replace=Applic.]
[fieldsource=booktitle, match=(.*)+$.*$, replace=$1$] [fieldsource=journal,
match=(The+)?Journal+of+, replace=J.
20] [fieldsource=journal, match=Communications+of+the+, replace=Comm.
20] [fieldsource=journal, match=(ACM|IEEE)+Transactions+on+(.*),
replace=$1\x{20}Trans.\\\ \x{20}$2] [fieldsource=journal, match=Software ,
replace=Softw. ] [fieldsource=journal, match=Engineering, replace=Eng.]
[fieldsource=journal, match=Applications, replace=Applic.]
[fieldsource=journal, match=(.*)+$.*$, replace=$1$] [typesource=techreport,
typetarget=report]

#  _CVEfixes_ : Automated Collection of Vulnerabilities and Their Fixes from
Open-Source Software

Guru Bhandari  Simula Research LaboratoryOsloNorway<EMAIL_ADDRESS>,  Amara
Naseer  Simula Research LaboratoryOsloNorway<EMAIL_ADDRESS>and  Leon Moonen
Simula Research LaboratoryOsloNorway<EMAIL_ADDRESS>
(2021)

###### Abstract.

Data-driven research on the automated discovery and repair of security
vulnerabilities in source code requires comprehensive datasets of real-life
vulnerable code and their fixes. To assist in such research, we propose a
method to automatically collect and curate a comprehensive vulnerability
dataset from Common Vulnerabilities and Exposures (CVE) records in the public
National Vulnerability Database (NVD). We implement our approach in a fully
automated dataset collection tool and share an initial release of the
resulting vulnerability dataset named _CVEfixes_.

The _CVEfixes_ collection tool automatically fetches all available CVE records
from the NVD, gathers the vulnerable code and corresponding fixes from
associated open-source repositories, and organizes the collected information
in a relational database. Moreover, the dataset is enriched with meta-data
such as programming language, and detailed code and security metrics at five
levels of abstraction. The collection can easily be repeated to keep up-to-
date with newly discovered or patched vulnerabilities. The initial release of
_CVEfixes_ spans all published CVEs up to 9 June 2021, covering 5365 CVE
records for 1754 open-source projects that were addressed in a total of 5495
vulnerability fixing commits.

_CVEfixes_ supports various types of data-driven software security research,
such as vulnerability prediction, vulnerability classification, vulnerability
severity prediction, analysis of vulnerability-related code changes, and
automated vulnerability repair.

Security vulnerabilities, dataset, software repository mining, vulnerability
prediction, vulnerability classification, source code repair.

††copyright: rightsretained††doi: 10.1145/3475960.3475985††journalyear:
2021††submissionid: fsews21promisemain-p11-p††isbn:
978-1-4503-8680-7/21/08††conference: Proceedings of the 17th International
Conference on Predictive Models and Data Analytics in Software Engineering;
August 19–20, 2021; Athens, Greece††booktitle: Proceedings of the 17th
International Conference on Predictive Models and Data Analytics in Software
Engineering (PROMISE ’21), August 19–20, 2021, Athens, Greece††ccs: Security
and privacy Software and application security††ccs: Security and privacy
Vulnerability management††ccs: Software and its engineering Software defect
analysis††ccs: Software and its engineering Software libraries and
repositories

##  1\. Introduction

The exploitation of security vulnerabilities is a significant threat to the
reliability of software systems and to the protection of the data processed by
them, as is frequently evidenced by new data leaks, ransomware attacks, and
substantial outages of essential systems. Despite the continued efforts of the
software engineering community to improve software quality and security by
means of secure coding guidelines, software testing, and various forms of code
review, the publicly available Common Vulnerabilities and Exposures (CVE)
records reveal an increasing trend in the number of vulnerabilities that are
discovered each year (mitre:cve, 1).

The overall security of software is highly dependent on the effective and
timely identification and mitigation of software vulnerabilities. However,
this is not an easy task and requires experience and specialized skills that
go beyond the expertise of the typical developer, resulting in many
vulnerabilities that go unnoticed for a long time. Consequently, there has
been considerable attention in academia and industry to the development of
techniques and tools that can help developers identify, and possibly repair,
security vulnerabilities in source code already in the development phase.

_Data-driven_ vulnerability research depends on the availability of datasets
with samples of _real-life_ vulnerable code and their fixes
(choi2017:endtoend, 2). Moreover, such datasets should encompass multiple
levels of _granularity_ (morrison2015:challenges, 3, 4), such as files,
classes, functions, etc., and cover widely-used programming languages
(wang2019:detecting, 5, 6). Finally, for reliable training and evaluation of
machine learning (ML) approaches, we need _comprehensive_ datasets that
contain large numbers of diverse and _labeled_ samples of both vulnerable and
non-vulnerable code (russell2018:automated, 7, 8, 9). As we will see in the
discussion of related work (Section 2), the currently available vulnerability
datasets do not fulfill these requirements.

To address their shortcomings and assist in data-driven vulnerability
research, we propose to automatically collect and curate a comprehensive
vulnerability dataset from CVE records in the National Vulnerability Database
(NVD). In particular, we propose an approach that _mines_ CVE records for
open-source software (OSS) projects hosted on GitHub, GitLab, and Bitbucket,
to collect real-world samples of vulnerable and corresponding patched code. We
implement the proposed approach in a dataset collection tool and share an
initial release of the resulting vulnerability dataset named _CVEfixes_. The
dataset supports various types of data-driven software security research, such
as ML-based vulnerability identification, automated classification of
identified vulnerabilities in Common Weakness Enumeration (CWE) types,
vulnerability severity prediction, and automated repair of security
vulnerabilities.

Contributions: This paper makes the following contributions:

  1. (a)

We survey existing security vulnerability-related datasets and discuss their
strengths and weaknesses.

  2. (b)

We propose a method to automatically collect and curate a comprehensive
vulnerability dataset from CVE records in the NVD, and obtain real-world
samples of vulnerable and corresponding patched code, organized at multiple
levels of granularity. In addition, we discuss how to enrich this data with
meta-data such as programming language used, and detailed code-related metrics
at five levels of abstraction, such as the commit-, file-, and method levels,
as well as the repository- and CVE levels.

  3. (c)

We implement the proposed approach in a dataset collection tool that is made
publicly available.111 https://github.com/secureIT-project/CVEfixes,
doi:10.5281/zenodo.5111494.

  4. (d)

We publicly share a version of the _CVEfixes_ vulnerability dataset for use by
other researchers.222 https://zenodo.org/record/4476563,
doi:10.5281/zenodo.4476563. The initial release spans all published CVE
records up to 9 June 2021, covering 5365 CVE records for 1754 OSS projects. A
total of 5495 vulnerability fixing commits are obtained from the projects’
version control systems and linked to information from the corresponding CVE
records, such as CVE-IDs, reference links, severity scores, vulnerability
type/CWE type, and other descriptive information.

The remainder of the paper is organized as follows: Section 2 discusses the
currently available vulnerability datasets and how our work differs from them.
Section 3 presents the process for constructing the _CVEfixes_ dataset,
followed by an exploration of the initial release in Section 4. Section 5
discusses applications of the dataset, together with current limitations and
future extensions. Finally, we conclude in Section 6.

##  2\. Related Work

Several vulnerability-related test suites and datasets were developed in the
last decade, often with particular goals in mind. These sets were consequently
picked up by other security researchers looking for relevant data to train or
evaluate their techniques on. This section surveys frequently used datasets
and discusses some of the challenges for their reuse in other contexts. We
also look at more general software repositories and repository mining
frameworks. Finally, we discuss how the _CVEfixes_ dataset is different from
the existing work in this area.

Vulnerability Datasets The Software Assurance Metrics And Tool Evaluation
(SAMATE) project created the Juliet Test Suite as a benchmark for static
analysis tools that aim to identify vulnerabilities in source code
(boland2012:juliet, 10, 11). Programs from the Juliet Test Suite have been
used in several studies on vulnerability and weakness prediction
(gupta2018:textmining, 12, 13, 14, 15, 16). However, the benchmark was never
created for this purpose, and concerns have been raised that the
vulnerabilities in Juliet are not very diverse, and many of them are of a
synthetic nature that does not occur in real-world software projects
(choi2017:endtoend, 2, 17).

Mitropoulus et al. (mitropoulos2014:vulnerability, 18) constructed a
vulnerability dataset by analyzing the Maven repository using _FindBugs_
(:findbugs, 19), a tool conducting static code analysis on Java byte-code,
identifying the vulnerabilities in Maven and categorizing them into nine
different types. Similarly, Draper’s VDISC dataset (russell2018:automated, 7)
consists of C/C++ source code of 1.27 million functions mined from open-source
projects that were labeled by three static code analysis tools (Clang,
Cppcheck, and Flawfinder) as vulnerable or not, classified in five groups of
vulnerabilities, namely CWE-120, 119, 469, 476, and other. However, there are
some challenges with the dataset, such as the fact that it is highly
imbalanced with only 6.8% functions labeled as vulnerable. Moreover, the
extracted functions are incomplete, missing the function’s return type which
excludes certain (signature-based) analyses. _SVCP4C_ (_SonarCloud_ Vulnerable
Code Prospector for C) is an online tool for collecting vulnerable source code
from open-source repositories linked to SonarCloud. The tool performs static
analysis and labels the potentially vulnerable source code at the file level
(raducu2020:collecting, 20). The Devign (zhou2018:devign, 21) dataset includes
four real-world open-source C/C++ projects, Linux, FFmpeg, Qemu and Wireshark,
where the labeling is performed using security-related keyword filtering. The
drawback of this dataset is also in the labeling: if a commit is believed to
be vulnerable, then all functions changed by the commit are labeled as
vulnerable, which is always not true. Moreover, the dataset does not classify
the types of vulnerabilities encountered. A threat recognized by the authors
of these datasets/tools is that the labeling is based on static analysis,
which is known for its false positives, that may lead to incorrect labeling.
Our approach, as well as others discussed below, aims to mitigate this threat
by building on actual security patches. The Code Gadget Database (CGD)
(li2018:vuldeepecker, 22) collects two types of vulnerabilities in C/C++
programs: buffer error vulnerability (CWE-119) and resource management error
vulnerability (CWE-399). The dataset covers 61638 code gadgets including 17725
vulnerable and 43913 non-vulnerable code gadgets. It aims to improve the
labeling quality by checking if the code slice of confirmed bugs overlaps with
a bug fix before labeling code as ’buggy’. Recently, Zheng et al.
(zheng2021:d2a, 17) published _D2A_ , which uses an approach based on
differential analysis to label issues in the source code functions or snippets
reported by static analysis tools. The D2A approach analyzes the commit
messages to identify the likeliness of vulnerability fixes from before-commit
and after-commit versions. The dataset considers code from six open-source
C/C++ projects: OpenSSL, FFmpeg, httpd, NGINX, libtiff, and libav.

Vieira et al. (vieira2019:reports, 23) introduced a dataset of bug-fixing
activities from 70000 bug-fixing reports from 10 years of 55 open-source
projects of _Apache_ mined from _Jira_ ,333
https://atlassian.com/software/jira a popular issue tracking system that
captures information about software development, bugs, security
vulnerabilities, new functionalities, etc. The dataset’s emphasis is on
process-related information regarding issue management, such as change
metrics, fix effort, status, version, and assignee, but it does not go into
code-level detail. Alves et al. (alves2016:software, 24) analyze the bug-
tracking systems of five open-source projects (Mozilla, Linux Kernel, Xen
Hypervisor, Httpd, and Glibc), looking for occurrences of CVE identifiers.
They found 2875 security patches and used the associated code to build a
dataset that labels the code before patching as vulnerable, and after patching
as not vulnerable, as well as adding labels for vulnerability types and
severity. Moreover, they compute software metrics at the file, class, and
function level for the code before and after patching to enable metrics-based
vulnerability prediction research. Similarly, Gkortzis et al.
(gkortzis2018:vulinoss, 25) present a vulnerability dataset correlating source
code and software metrics of 8694 versions of open-source projects.

Jimenez et al. (jimenez2018:enabling, 26) present _VulData7_ , a framework to
collect a security vulnerability dataset covering all reported NVD
vulnerabilities of four security-critical open-source systems, _Linux Kernel,
WireShark, OpenSSL, and SystemD_. Although focusing on a limited set of
systems all written in the C programming language, their approach shares
several characteristics with ours, but one main practical challenge for
reusing their framework is that it is based on the analysis of XML files, a
format that is no longer provided by the NVD. Similarly, Ponta et al.
(ponta2019:manuallycurated, 27) curated a dataset of 1282 commit fixes to
vulnerabilities from 205 open-source Java projects obtained from NVD and
project-specific web resources, and Fan and Nguyen (fan2020:code, 28) curated
a C/C++ code vulnerability dataset named _Big-Vul_ that corresponds with code
changes and CVE database entries from 2002 to 2019.

Lin et al. (lin2020:software, 9) emphasize the need for a benchmark for
evaluating the effectiveness of ML approaches aimed at fixing security
vulnerability in source code. The authors propose a benchmark dataset that
offers labels at two levels of granularity, i.e., the file- and method level,
collected from nine software projects written in the C programming language
(lin2020:deep, 29). The challenge with using this dataset for training
purposes is that it contains only 1471 vulnerable functions and 1320
vulnerable files, which is rather small for training a deep learning model.

Software Repositories and Repository Mining Frameworks Ma et al.
(ma2021:world, 30) present World of Code (WoC), a large and frequently updated
collection of git-based version control data. The data is indexed three
storage abstractions and arranged in four layers. New and updated repositories
are periodically analyzed, and the database is updated once a month. The
dataset stores four types of raw git objects: commits, trees, blobs and tags.
The bulk extraction of these raw git objects was done in parallel using
_libgit2_ , a portable tool implemented in C. Querying this database can help
answer different research questions, such as trends in programming languages
used, ecosystem comparisons, bug prediction, developer migration studies, bot
detection, understanding developer trajectories, etc. In the same way, the
dataset can be inspected for different security-related research areas,
vulnerable commits classification, developers’ tendency towards vulnerability,
etc.

_Perceval_ (duenas2018:perceval, 31) is an automatic and incremental data
gathering tool to mine software development data from various sources such as
versioning systems, bug tracking systems and mailing lists. The JSON output of
the tool stores commit-level information of the repositories that can be used
to analyze the activities of the repositories and the developers. However, the
tool does not expose access to the finer granularity levels of open-source
projects, such as the code-level vulnerability information that we are
concerned with.

_PyDriller_ 444 https://github.com/ishepard/pydriller is a Python package to
mine code-level information from different version control systems
(spadini2018:pydriller, 32). It allows extracting commit-, file-, method
levels information from git repositories, and it collects all relevant meta-
information, including the computation of a number of code-level metrics.
Because of these features, we use _PyDriller_ during the construction of
_CVEfixes_ , before mapping the collected data to a relational database.

Differences The _CVEfixes_ dataset presented in this paper differs from the
existing datasets in the following ways: (1) In general, it covers more
information about the vulnerability at different levels of granularity, up to
the actual source code before and after fixing. (2) It covers a longer
timeframe, being based on all CVE records available at the collection date of
9 June 2021. Moreover, it’s not restricted to this timeframe: at any point in
the future, our publicly available tool can automatically collect and
integrate the data up to the last published CVE at that point in time. (3)
Unlike many other datasets that focus on specific programming languages like
Java or C/C++, our dataset covers multiple programming languages and allows
the language as a query attribute. (4) Some of the existing vulnerability
datasets only classify the source code as vulnerable or not. In addition to
this binary classification, _CVEfixes_ classifies the vulnerabilities into
categories as defined by Common Weakness Enumeration (CWE) types, as well as
CVSS severity scores, enabling research into multi-class vulnerability
prediction.

##  3\. Dataset Construction

###  3.1. Overall Workflow

The National Vulnerability Database (NVD) (nist:nvd, 33) is a repository of
vulnerability management data maintained by the National Institute of
Standards and Technology (NIST). NIST publishes the entire NVD database for
public use employing various data feeds, including JSON vulnerability feeds,
security checklist references, security-related software flaws,
misconfigurations, product names, and impact metrics. The JSON vulnerability
feed comprises a daily updated feed of vulnerability records (CVEs), organized
by year of origin (updates may concern CVEs published in earlier years).

Each vulnerability in the file has its CVE-ID, publish date, description,
associate reference links, vulnerable product configuration, CWE weakness
categorization (mitre:cwe, 34), and other metrics. Moreover, each
vulnerability’s severity is ranked using the Common Vulnerability Scoring
System (CVSS) (first:common, 35). The CVSS is composed of a set of metrics in
three groups, i.e., base, temporal, and environmental, to specify the
characteristics and contextual information of a vulnerability. There are two
versions of the scoring system: CVSS${}_{\text{v2}}$ and CVSS${}_{\text{v3}}$,
that are both in active use in the NVD.555 https://nvd.nist.gov/vuln-
metrics/cvss CVSS${}_{\text{v3}}$ introduces a number of changes over
CVSS${}_{\text{v2}}$ to score vulnerabilities more accurately and provide more
information to distinguish between different types of vulnerabilities.
Finally, when a vulnerability in an open-source project is fixed, the CVE
record will be updated with one or more pointers to the relevant source code
repositories, as well as commit hashes of the fixes. These comprehensive,
scrutinized, and frequently updated JSON vulnerability feeds are the primary
basis for generating the vulnerability data in _CVEfixes_.

Figure 1. Dataset construction workflow

Figure 1 presents the overall workflow of the dataset collection process. It
starts by collecting CVE records via JSON vulnerability feeds. Since these CVE
vulnerabilities are categorized according to CWE weakness types (mitre:cwe,
34), we also collect the details of these CWE types and cross-reference them
with the appropriate CVE records.

Moreover, for CVEs that have fixes associated with them, the corresponding
open-source project repositories are (temporarily) locally cloned, and source-
level information about the vulnerable code and the corresponding fixes is
gathered based on the commit hashes reported in the CVE record. After
extracting the vulnerable and fixed code, a number of code-level metrics are
computed, and additional meta-data such as the programming language is
derived. Finally, the collected information is stored in a relational
database.

Figure 2. Entity-Relationship Diagram showing the various levels of
abstraction in _CVEfixes_ and their interconnections

The overall structure of the database is shown in Figure 2. The main tables
capture information about CVE records, CWE records, OSS repositories, commits,
changed files, and changed methods. Two additional tables respectively
associate fixes with CVEs, and CWE classifications with CVEs.

###  3.2. Details of the Automated Collection Tool

The proposed process has been implemented in a fully automated _CVEfixes_
collection tool, and the remainder of this section discusses the major phases
implemented in the tool. The collection can easily be repeated to keep up-to-
date with newly discovered or patched vulnerabilities. Since updates can
affect any of the published CVEs, the collection tool currently revisits all
published CVEs to check if relevant fixes were added. One direction for future
work is enabling a more incremental process.

Scanning of CVE Records: By default, the collection tool retrieves all
published JSON vulnerability feeds from the NVD server, covering the first
published CVE in 2002 up to the last published one on the date of collection
(the feeds are updated daily). The JSON files are then aggregated, flattened,
and processed to filter out redundant information from the CVE records (in the
original feeds, some fields are repeated at deeper nesting levels). All CVE
records that do not have fixes associated with them in the _reference_json_
field are ignored, because gathering corresponding vulnerable or fixed code is
not possible for these cases. After processing and filtering the records,
various details of the vulnerability such as CVE-ID, published date, last
modified date, reference data, CVSS severity scores, vulnerability impact and
scope, exploitability score, are recorded in the _cve_ table.

Classification of Vulnerabilities: As discussed before, CVE records in the NVD
are annotated with CWE vulnerability types that indicate at a more abstract
level what kind of vulnerability this CVE concerns. The _problemtype_json_
field of the NVD JSON vulnerability feeds may refer to one or more of these
CWE types. The NVD only uses a subset666 https://nvd.nist.gov/vuln/categories
of the full CWE list maintained by MITRE,777
https://cwe.mitre.org/data/index.html and does not distinguish between
individual CWEs and CWE categories, referring to either as “CWE-_< num>_”.
Moreover, two additional classifications, “ _NVD-CWE-noinfo_ ” and “ _NVD-CWE-
other_ ”, are used to refer to cases where, respectively, there is not enough
info for classification, or where the NVD deviates from the full CWE. Note
that we also found around 250 cases where the NVD feeds contain the CWE
classification “ _unknown_ ”. For consistency, we map these “ _unknowns_ ” to
“ _NVD-CWE-noinfo_ ” in our dataset.

Meta-Information for Repositories: At the time of writing, the majority (>80%)
of fixes in the NVD refer to code on one of the three major OSS forges,
GitHub, GitLab, and Bitbucket, with 98% of those being on GitHub. The
remaining 20% of fixes point to other forges (e.g., sourceforge or the defunct
gitorious), or to project-specific servers hosting other versioning systems
(e.g., mercurial, subversion, or CVS). As a result, we focus on gathering code
from those three OSS forges, and we have modeled the _repository_ table to
maximally cover GitHub’s repository meta-information, such as repository name,
description, date of creation, last date of push, homepage, programming
language, number of forks and number of stars. This information can be used
to, for example, focus on certain programming languages or filter for popular
repositories by setting a minimum threshold on the number of stars. The latter
is also a common and effective approach to focus on information related to
_mature_ projects. Note that GitLab and Bitbucket do not offer all of these
attributes, so clearly distinguishable values are used to represent missing
information, such as the value “-1” for the _stars_count_ of Bitbucket
repositories (which does not offer stars).

Extraction of Commits: Whenever an OSS vulnerability is fixed, the
_reference_json_ in the CVE record contains the repository URL, as well as a
pointer to the exact commit that introduced the fix, by means of a git hash.
We use this information to locally (and temporarily) clone the repository, and
use it to extract versions of the code before and after submitting the fix. We
extract this information at the commit-, file- and method level, where each
entry in the _commits_ table is associated to one or more _cve_ s via the
_fixes_ table. In addition, the _commits_ table stores meta-information about
the commit, such as the author, time and date, commit message, if it is a
merge commit (often to include pull requests in projects using GitHub Flow),
the number of lines added or deleted, and Delta Maintainability Model metrics
related to the change (biase2019:delta, 36).

Extracting the Modified Files: Every entry in the _commits_ table is linked
via the commit hash to one or more _file_change_ s that were included to fix
the vulnerability (or vulnerabilities) associated with the commit. For each
file change, the dataset contains the contents of the file before and after
making the change, in _code_before_ and _code_after_ respectively, as well as
the _diff_ of the change in the format delivered by Git, and a parsed version
of this information in _diff_parsed_ , containing a dictionary of added and
deleted lines. In addition, several meta-data are collected, such as filename,
old and new path, type of the modification (i.e., added, deleted, modified, or
renamed), number of lines added or removed in that file, lines of code
(_nloc_) after the change, and cyclomatic complexity after the change.
Finally, the _Guesslang_ tool888 https://guesslang.readthedocs.io/en/latest/
is used to detect the actual _programming_language_ that is used in a given
file. _Guesslang_ is an open-source deep-learning based classifier that was
trained with over a million source code files, and recognizes over 30
programming languages. Note that these detected languages may differ from, and
are more precise than the repository language that is, for example, reported
by GitHub.

Extracting the Modified Methods: Similar to the changes at the file-level, we
keep track of changes at the method-level in the _method_change_ table. In
addition to the actual code, each method change stores meta-information such
as the method name, its signature, parameters, start and end line of the
method, lines of code, cyclomatic complexity, and a boolean indicating whether
this concerns the code from before committing the fix or not.

###  3.3. Practical Guidance

The automated _CVEfixes_ dataset collection tool is distributed via GitHub.999
https://github.com/secureIT-project/CVEfixes, doi:10.5281/zenodo.5111494. The
distribution comes with detailed requirements and installation instructions.
It requires Python3.8 or later to run, and SQLite3 to construct and store the
data into a relational database. The main Python packages that it depends on
are pandas, numpy, requests, PyDriller, PyGithub and guesslang. We provide
_requirements.txt_ and _environment.yml_ files for dependency management using
pip/venv or anaconda/miniconda.

The tool is configured by means of a number of settings in a _.CVEfixes.ini_
file (an example is provided in the distribution):

  * •

_database_path_ : location of the _CVEfixes_ database file (also used to hold
some temporary files during extraction).

  * •

_sample_limit_ : an optional limit on the number of commits to be gathered
(mainly for testing and demonstration purposes, sample_limit = 0 means
unlimited collection).

Note that the GitHub API is severely rate-limited when unauthenticated access
is used. These limits can be raised significantly to up to 5000 requests per
hour by authenticatinsg with a username and personal access token.101010
https://docs.github.com/en/github/authenticating-to-github/creating-a-
personal-access-token A _github_username_ and _github_token_ can be configured
in _.CVEfixes.ini_ for this purpose. With a sample limit of 25, no token is
needed and rate-limiting will not be triggered.

We provide a compressed SQL dump for the initial release of the _CVEfixes_
vulnerability dataset that covers all published CVEs in the NVD up to 9 June
2021 at Zenodo.111111 https://zenodo.org/record/4476563,
doi:10.5281/zenodo.4476563. The distribution contains a simple shell script to
convert the compressed SQL dump into an SQLite3 database:

⬇

$ sh Code/create_CVEfixes_from_dump.sh

Alternatively, the _CVEfixes_ dataset can be gathered from scratch using the
following shell script:

⬇

sh Code/create_CVEfixes_from_scratch.sh

Note that this process will overwrite an existing database. Moreover, at the
time of writing, the full collection process can take up to 15 hours,
depending on the available internet connection. The advantage of taking this
route is that the database will contain all CVEs published up to the date of
initiating the collection process.

The distribution contains an example Jupyter Notebook
_Examples/query_CVEfixes.ipynb_ that shows how to explore the _CVEfixes_
dataset and contains the code to generate the various tables and graphs in
this paper.

Recall the overall structure of the database shown in Figure 2. Then the
following query extracts all methods involved in fixes related to C programs
before the changes were made:

⬇

SELECT m.name, m.signature, m.nloc, m.parameters, m.token_count, m.code

FROM method_change m, file_change f

WHERE f.file_change_id = m.file_change_id

AND f.programming_language = ’C’

AND m.before_change = True

Another example, inspired by the ManySStuBs4J dataset (karampatsis2020:how,
37), to find all files that only add or remove a single line as fix:

⬇

SELECT cv.cve_id, f.filename, f.num_lines_added, f.num_lines_deleted,
f.code_before, f.code_after, cc.cwe_id

FROM file_change f, commits c, fixes fx, cve cv, cwe_classification cc

WHERE f.hash = c.hash AND c.hash = fx.hash

AND fx.cve_id=cv.cve_id

AND cv.cve_id=cc.cve_id

AND f.num_lines_added<=1

AND f.num_lines_deleted<=1;

##  4\. Dataset Exploration

Table 1 presents the summary statistics of the initial release of the
_CVEfixes_ dataset. This initial release covers 5365 unique CVEs in 1754 OSS
projects, with 5495 unique vulnerability fixing commits. The CVEs are
classified into 180 different CWE vulnerability types. Note that some of the
vulnerability fixing commits cover multiple CVEs, as is indicated by the
number of commits being lower than the number of CVEs.

Table 1. Summary statistics of the _CVEfixes_ dataset CVEs | CWEs | projects | commits | files | methods
---|---|---|---|---|---
$5{,}365$ | $180$ | $1{,}754$ | $5{,}495$ | $18{,}249$ | $50{,}322$
Table 2. Top 10 projects in _CVEfixes_ with respect to (a) the number of fixed
CVEs, shown with the project’s security metrics, (b) number of vulnerability
fixing commits, and the number of (c) files and (d) methods involved in these
fixes.

|  | average | average | average | average
---|---|---|---|---|---
project | # CVEs | CVSS${}_{\text{v2}}$ | CVSS${}_{\text{v3}}$ | exploitability | impact
linux | $973$ | $5.30$ | $3.99$ | $1.16$ | $2.78$
ImageMagick | $157$ | $5.45$ | $7.25$ | $2.74$ | $4.43$
tensorflow | $143$ | $3.92$ | $6.67$ | $2.05$ | $4.50$
tcpdump | $89$ | $7.33$ | $9.57$ | $3.86$ | $5.70$
FFmpeg | $83$ | $6.35$ | $5.86$ | $2.23$ | $3.57$
phpmyadmin | $67$ | $4.46$ | $3.55$ | $1.85$ | $1.53$
php-src | $60$ | $6.59$ | $7.46$ | $3.22$ | $4.21$
MISP | $50$ | $5.03$ | $6.70$ | $2.81$ | $3.61$
WordPress | $46$ | $5.27$ | $7.01$ | $3.06$ | $3.66$
hhvm | $40$ | $6.48$ | $7.54$ | $3.22$ | $4.31$

(a)

project | # commits
---|---
linux | $1{,}029$
ImageMagick | $171$
tensorflow | $156$
phpmyadmin | $126$
tcpdump | $101$
FFmpeg | $88$
php-src | $61$
MISP | $54$
WordPress | $47$
radare2 | $43$

project # files linux $2{,}007$ GeniXCMS $1{,}930$ kanboard $698$ CycloneTCP
$611$ X2CRM $594$ exponent-cms $445$ Ushahidi-Web $402$ wityCMS $372$ tcpdump
$344$ tensorflow $340$ project # methods GeniXCMS $14{,}235$ linux $4{,}303$
exponent-cms $3{,}048$ Ilch-2.0 $1{,}427$ arangodb $1{,}224$ kanboard
$1{,}169$ tensorflow $1{,}045$ moped $950$ WordPress $915$ X2CRM $790$

(b) (c) (d)

To further characterize the data, Table 2 presents the top ten projects in
_CVEfixes_ with respect to the number of CVEs, number of vulnerability fixing
commits, and number of files and methods involved in fixes. In particular,
Table 2a corresponds with the project-wise number of CVEs, the average
CVSS${}_{\text{v2}}$ and CVSS${}_{\text{v3}}$ base scores of these
vulnerabilities, as well as their average exploitability, and impact scores.
Not surprisingly, the long-running and well-scrutinized _linux_ project has
the most vulnerabilities and vulnerability fixing commits (shown in resp.
Table 2a and 2b), i.e., 1029 commits for fixing 973 vulnerabilities. Note that
_linux_ has roughly six times as much vulnerabilities as _ImageMagick_ , the
project with second most vulnerabilities (157), yet the security impact of
those discovered in _ImageMagick_ is considerably larger, as shown by the
various security metrics.

Looking at the projects in the four top ten lists of Table 2a-2d, it is not
surprising to see some overlap between the lists. On the other hand, it _is_
surprising to see that some projects, such as _GeniXCMS_ , _kanboard_ , and
_exponent-cms_ , which do not occur at all in the top ten CVEs and #commits,
come in very highly ranked on the amounts of files and methods that need to be
changed to fix vulnerabilities in these projects. This sudden rise in the
ranks suggests it can be of interest to investigate how the modularization
decisions in these projects impacted the security fixes that were made, and
may indicate an application-level code smell such as shotgun surgery.

This finding also shows that there is _no_ direct correlation between the
number of files or methods and the number of CVEs (or commits to fix them),
because that would mean the respective top tens would have been similar. We
_do_ see the expected relation between CVEs and vulnerability fixing commits
(with minor changes due to some commits covering multiple CVEs, as noted
earlier).

The 10 projects in Table 2a cover 31.84% of the total number of CVEs, those in
Table 2b cover 31.61% of the total number of vulnerability fixing commits, the
ones in Table 2c account for 42.43% of the total number of files involved in
fixes, and finally, the ten projects in Table 2d make up 57.84% of the total
number of methods involved in fixes.

Table 3 presents the ten most occurring CWE types based on CVE count. We see
that CWE-79 (Improper Neutralization of Input During Web Page Generation, is
the most commonly identified vulnerability type. This type is commonly known
as Cross-site Scripting or XSS. It has been assigned to 635 CVEs and was fixed
in 670 commits that changed 3226 files. Observe that the number of
vulnerability fixing commits is larger than the number of CVEs, which is
caused by the fact that some of the CVEs are associated with multiple fixes.
This occurs, for example, when a CVE is fixed in several related projects, or
when the fix is spread over multiple commits. On second place comes CWE-119
(Improper Restriction of Operations within the Bounds of a Memory Buffer),
reported in 408 CVEs and fixed in 403 commits that changed 716 files.

Next, we find CWE-20 (Improper Input Validation) assigned to 382 CVEs. which
is closely followed by CWE-125 (Out-of-bounds Read), which is actually a
“child” of the more general CWE-119 category, and indicates that an index
pointer is used beyond the bounds of the memory buffer. It is reported in 380
CVEs and fixed in 404 commits that modified 1159 files. After this top four,
we see a considerable drop in the number of CVEs assigned to lower-ranked
CWEs. The top ten CWEs cover approximately 55.58% of the total number of CVEs,
57% of the total number of commits, and 55.75% of the total number of files.

Table 3. Distribution of vulnerabilities over CWE types

CWE |  description | CVEs | cmts | files
---|---|---|---|---
CWE-79 |  Improper Neutralization of Input During Web Page Generation (Cross-site Scripting) | 635 | 670 | 3226
CWE-119 |  Improper Restriction of Operations within the Bounds of a Memory Buffer | 408 | 403 | 716
CWE-20 |  Improper Input Validation | 382 | 397 | 1965
CWE-125 |  Out-of-bounds Read | 380 | 404 | 1061
CWE-200 |  Exposure of Sensitive Information to an Unauthorized Actor | 276 | 314 | 818
CWE-787 |  Out-of-bounds Write | 205 | 211 | 537
CWE-476 |  NULL Pointer Dereference | 195 | 198 | 334
noinfo* |  Insufficient Information | 193 | 219 | 664
other* |  Other | 165 | 170 | 395
CWE-264 |  Permissions Privileges and Access Controls | 143 | 148 | 457

* abbreviated from NVD-CVE-noinfo and NVD-CWE-other for space.

Table 4. Average number of days between CVE publication and vulnerability fix for all CVEs in _CVEfixes_. mean | std | min | 25% | 50% | 75% | max
---|---|---|---|---|---|---
$171.55$ | $414.93$ | $0$ | 7.0 | 28.0 | 123.25 | $7{,}636$
Table 5. Average number of days between CVE publication and vulnerability fix for the top ten projects with most CVEs (cf. Table 2a). project | # CVEs | # CWEs | min | max | median | mean
---|---|---|---|---|---|---
linux | $973$ | $53$ | $0$ | $4{,}582$ | $49$ | $213.64$
ImageMagick | $157$ | $27$ | $1$ | $1{,}474$ | $75$ | $190.14$
tensorflow | $143$ | $24$ | $2$ | $810$ | $16$ | $22.15$
tcpdump | $89$ | $4$ | $1$ | $918$ | $1$ | $12.96$
FFmpeg | $83$ | $17$ | $1$ | $1{,}523$ | $44$ | $63.22$
phpmyadmin | $67$ | $13$ | $1$ | $1{,}211$ | $13$ | $36.15$
php-src | $60$ | $17$ | $3$ | $855$ | $27$ | $50.26$
MISP | $50$ | $17$ | $0$ | $430$ | $1$ | $29.33$
WordPress | $46$ | $14$ | $2$ | $217$ | $4$ | $18.17$
hhvm | $40$ | $18$ | $1$ | $1{,}328$ | $37$ | $182.18$

We also investigated if it was possible to compute how long it took to fix the
vulnerabilities, from the day that the CVE was disclosed to the project to the
day that the fix was committed. Unfortunately, the NVD only includes the CVE
_publication date_ , which generally does not indicate when the vulnerability
was discovered or when it was shared with the project. The actual disclosure
date can be several months earlier if a responsible disclosure process is
used. For example, Google’s Project Zero delays publication for 90 days after
disclosing vulnerabilities that are not actively being exploited, giving the
option for earlier publication with mutual agreement. Other security
researchers may use other time frames or agree on them on a case-by-case basis
with the projects. As a result of the variability this introduces in the time
between disclosure and publication time, it is impossible to make strong
statements on how long the projects really used to address the vulnerability.

With the above caveat in mind, Table 4 shows some statistics for the complete
_CVEfixes_ dataset on how many days it took from the CVE publication to
committing a fix for the vulnerability. Moreover, Table 5 presents such
statistics for the top ten projects with most CVEs, as earlier presented in
Table 2a.

The last two characteristics that we will investigate are the vulnerability
severity scores of CVEs in _CVEfixes_ , as well as the impact that their fixes
had on the maintainability of the project by means of the delta
maintainability model (DMM). The delta maintainability model (DMM)
(biase2019:delta, 36) is a measure to compare and rank fine-grained code
changes into low and high-risk changes. The overall DMM score refers to the
proportion of low-risk changes in a commit, and is computed as the mean of
three individual metrics that respectively measure the risk associated with
code size, (cyclomatic) complexity, and size of the interface. The values of
each of the DMM metrics range from 0 to 1, with higher values indicating
better maintainability, i.e., lower risk changes, and low values indicating
poor maintainability, i.e., high risk changes.

Figure 3. Violin plot showing the distribution of average vulnerability
severity scores for projects included in _CVEfixes_

Figure 3 presents violin plots that show the distribution of
CVSS${}_{\text{v2}}$ and CVSS${}_{\text{v3}}$ base scores, as well as
exploitability and impact, for all vulnerabilities in _CVEfixes_ , aggregated
into averages per project. The width of each violin corresponds to the
frequency of data points with that average. Inside the violin there is small
box plot showing the ends of the first and third quartiles, and the median is
indicated by the white dot. The values of these vulnerability severity metrics
can vary between 0 and 10, with lower being better, and higher being worse.
The figure suggests that the majority of the vulnerabilities have severity
scores that are on the high side of the severity range, although their
exploitability and impact tends to be on the lower sides of their respective
ranges.

Figure 4. Violin plot showing the distribution of average DMM scores for fixes
to the projects included in _CVEfixes_

Figure 4 presents violin plots that show the distribution of DMM metrics of
all vulnerability fixing commits in _CVEfixes_ , again aggregated into
averages per project. These metrics show how the maintainability of the
projects was affected by the security fixes. Observe that the individual DMM
metrics _unit size_ , _unit complexity_ , and _unit interfacing_ mostly have a
high density around 0 (indicating high risk changes with a detrimental effect
on maintainability), and a smaller number of changes are considered low risk
(with scores closer to 1). The overall _DMM score_ is an aggregate computed as
the mean of the individual metrics (biase2019:delta, 36). As such, it is more
universally spread across different risk values, but still with a considerable
number of commits requiring high-risk code change to fix the vulnerabilities,
with the median around 0.25 and the third quartile around 0.6. An interesting
question for follow-up research is investigating if vulnerability fixing
commits that lower maintainability are followed by refactoring commits that
restore maintainability.

##  5\. Discussion

###  5.1. Applications of _CVEfixes_

The _CVEfixes_ dataset can be used in several ways to support data-driven
software security research, for example, for automated vulnerability
prediction, automated classification of identified vulnerabilities in Common
Weakness Enumeration (CWE) types, vulnerability severity prediction, and
automated repair of security vulnerabilities. The remainder of this section
discusses several of these applications in more detail.

Automated Vulnerability Prediction/Identification The _CVEfixes_ dataset
contains different levels of vulnerability data, such as CVE-ID, version,
description, type, publication date, current status, all interlinked up to the
actual code changes that were introduced to fix the vulnerability. This data
can be used to extract code features and metrics that help to better
understand how security vulnerabilities are introduced in code. Moreover, the
features and metrics can be used to model, train, and test vulnerability
predictors based on classical machine learning approaches, and the textual
descriptions, ranging from the CVE vulnerability level to the code level, can
be used to train deep learning-based models for vulnerability prediction. Many
studies have already targeted automated vulnerability identification using
various machine learning models (li2020:automated, 38, 7, 39, 40), and Section
2 surveyed some of the challenges with the datasets used in these studies,
such as limited size, lack of representativeness, and dataset imbalance. A
comprehensive dataset like _CVEfixes_ helps to overcome these challenges and
enables a more thorough evaluation of the approaches.

Automated Vulnerability Classification The inclusion of vulnerability
classifications in _CVEfixes_ allows us to go one step further than automated
vulnerability prediction, it also enables research on automated vulnerability
classification, i.e., not just predicting the presence of a vulnerability but
characterizing the type of the vulnerability. Such a classification is of
interest since (automated) program repair approaches may have different
efficacy and efficiency for different vulnerability types, so knowing the
vulnerability type helps to inform which repair strategy to take. To address
this challenge, $\mu$VulDeePeaker (zou2019:mvuldeepecker, 4) and ManySStuBs4J
(karampatsis2020:how, 37) have constructed the multi-class vulnerability
dataset. However, those datasets are specific to a given programming language
and certain vulnerability/bug types. _CVEfixes_ , on the other hand, uses the
well-known CWE taxonomy to classify vulnerabilities in a hierarchy of
vulnerability types and categories, and covers 27 programming languages
(though 9 of these have fewer than 100 files changed in the commits covered by
the dataset).

Analysis of vulnerability fixing patches Similar to how the vulnerable code in
_CVEfixes_ can be used to better understand how security vulnerabilities are
introduced in code and how these can be automatically predicted, the fixes
offered by _CVEfixes_ can be used to analyze and build on vulnerability fixing
patches. Several studies have already initiated research analyzing such
patches, such as the detection of patterns that can be used in automated
program repair (hegedus2020:inspecting, 41), and the identification of
security-relevant commits, also known as _pre-patches_ , as these may
inadvertently leak information about security vulnerabilities before the CVE
is published (yang2018:prepatch, 42, 43). Other research has analyzed
vulnerability fixing patches to facilitate the automated transformation of
patches into “ _honeypots_ ” that help trap malicious actors and detect if the
corresponding vulnerabilities are exploited in the wild
(larmuseau2018:patchsweetner, 44).

Automated program repair The pairs of vulnerable and fixed code provided by
_CVEfixes_ can be used to train machine learning-based automated program
repair along the lines of SequenceR (chen2019:sequencer, 45) and related work
surveyed in Section 2, though we need to warn for a caveat w.r.t. the
completeness of the patches, as discussed in more detail in the next section.
Moreover, the current state of the art in this area is constrained with
respect to the length of repairs that can be learned/synthesized, with 50
tokens being mentioned as an upper limit for acceptable performance
(chen2019:using, 46). By querying into _CVEfixes_ data as shown in the
previous subsection (and the example Jupyter Notebook in the distribution),
_CVEfixes_ facilitates the extraction of specific fixes that used only single
(or a few) modified lines for fixing a vulnerability. This can be used to
focus attention of the training process on cases with a lower number of
modified tokens which are within reach of the technology, in order to help
improve patch quality. Several studies (mashhadi2021:applying, 47, 48, 49, 50)
have already initiated work on the prediction of bugs/errors taking single
line modified statements into account using the ManySStuBs4J dataset
(karampatsis2020:how, 37). Querying _CVEfixes_ in the way we presented earlier
enables the extension of this work towards different programming languages,
and vulnerability types.

###  5.2. Limitations and Future Extensions

One of the limitations of the current _CVEfixes_ collection tool is that it
collects all available fixes from scratch. We opted for this approach since
any of the CVEs in the NVD may receive updates, so all should be inspected
anyway. Nevertheless, it may be interesting to explore a more incremental
update scheme that only clones a repository if not all of its fixes are
already captured in the database, and augments the existing fixes in the other
case. Such an approach would help to speed up the process of updating the
database. One caveat is that we may also need to remove fixes from the
database if they are no longer referenced by a CVE, but as we have seen, those
fixes may be referred to by other CVEs as well, so some additional checks
would be needed.

Another challenge/limitation that we have noticed is that some of the project
repositories referenced in the NVD are no longer available. There can be
multiple reasons; for example, the owners might have removed/changed/renamed
their repositories, or they may have moved the repository between forges. This
makes it impossible to fetch the fixes for these repositories. The current
implementation tries to gather code from as many repositories as possible, and
removes references to those that are unavailable to present an internally
consistent view. However, this choice also means that we only present a subset
of the information that is available in the NVD.

A last limitation is that a commit that is referenced as a vulnerability
fixing commit in the NVD can still leave (part of) the vulnerability in the
source code, and there may be later commits that complete the fix of the
vulnerability, or the commit may contain changes unrelated to the fix
(herzig2016:impact, 51). To further improve the quality of the data, it would
be of interest for future work to analyze consecutive patches, and
select/untangle the code that together addresses the vulnerability.

Another direction for future work is upgrading our extraction tool to support
mining the repositories from other issue-tracking and version control systems,
i.e., Bugzilla, Mercurial, Subversion, etc. This will make it possible to
gather vulnerability data from an even larger collection of projects.

Finally, considering the current controversy around ML models that are trained
on (A)GPL licensed code and may possibly regurgitate some of that code as part
of their operation, we plan to extend our datamodel with license information
for the fixes included, so that users of _CVEfixes_ can make an informed
choice about including or excluding certain fixes from their training data.

##  6\. Concluding Remarks

In this study, we propose the _CVEfixes_ dataset and a fully automated
collection tool that fetches this vulnerability dataset from publicly
available information in the NVD and different version control systems. The
dataset consists of real-world samples of vulnerable and corresponding patched
code, their security metrics at five levels of abstraction, all linked to the
CVE records. The collected information is enriched with various security and
code related metrics and organized into a relational database to support easy
querying.

The initial release of the dataset covers 5495 vulnerability fixing commits
from 1754 open-source projects. This multi-level dataset establishes both
qualitative and quantitative opportunities for vulnerability-related
investigations. Researchers can study the relation from published CVEs all the
way down to the corresponding code level vulnerabilities and their proposed
fixes. The _CVEfixes_ dataset enables research on vulnerability detection and
classification, vulnerability severity prediction, (pre-)patch analysis,
automated vulnerability repair, and many others. We plan to periodically
update _CVEfixes_ , and extend it with mining other open-source projects from
different version control systems and issue tracking systems.

## Acknowledgments

The work presented in this paper has been financially supported by the
Research Council of Norway through the secureIT project (RCN contract
#288787).

## References

  * (1) MITRE  “Common Vulnerabilities and Exposures (CVE)”  URL: https://cve.mitre.org/
  * (2) Min-je Choi, Sehun Jeong, Hakjoo Oh and Jaegul Choo  “End-to-End Prediction of Buffer Overruns from Raw Source Code via Neural Memory Networks”  In _Int’l Joint Conf. Artificial Intelligence_ , 2017, pp. 1546–1553  DOI: 10.24963/ijcai.2017/214
  * (3) Patrick Morrison, Kim Herzig, Brendan Murphy and Laurie Williams  “Challenges with Applying Vulnerability Prediction Models”  In _Symp. and Bootcamp on the Science of Security_ New York, New York, USA: ACM, 2015, pp. 1–9  DOI: 10.1145/2746194.2746198
  * (4) Deqing Zou, Sujuan Wang, Shouhuai Xu, Zhen Li and Hai Jin  “$\mu$VulDeePecker: A Deep Learning-Based System for Multiclass Vulnerability Detection”  In _IEEE Trans. Dependable and Secure Computing_ , 2019, pp. 13  DOI: 10.1109/TDSC.2019.2942930
  * (5) Xinda Wang, Kun Sun, Archer Batcheller and Sushil Jajodia  “Detecting "0-Day" Vulnerability: An Empirical Study of Secret Security Patch in OSS”  In _Int’l Conf. Dependable Systems and Networks_ , 2019, pp. 485–492  DOI: 10.1109/dsn.2019.00056
  * (6) Zhangyin Feng et al.  “CodeBERT: A Pre-Trained Model for Programming and Natural Languages”  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , 2020, pp. 1536–1547  DOI: 10.18653/v1/2020.findings-emnlp.139
  * (7) Rebecca Russell, Louis Kim, Lei Hamilton, Tomo Lazovich, Jacob Harer, Onur Ozdemir, Paul Ellingwood and Marc McConley  “Automated Vulnerability Detection in Source Code Using Deep Representation Learning”  In _Int’l Conf. Machine Learning and Applic._ Orlando, FL: IEEE, 2018, pp. 757–762  DOI: 10.1109/icmla.2018.00120
  * (8) Rory Coulter, Qing-Long Han, Lei Pan, Jun Zhang and Yang Xiang  “Code Analysis for Intelligent Cyber Systems: A Data-Driven Approach”  In _Information Sciences_ 524, 2020, pp. 46–58  DOI: 10.1016/j.ins.2020.03.036
  * (9) Guanjun Lin, Sheng Wen, Qing-Long Han, Jun Zhang and Yang Xiang  “Software Vulnerability Detection Using Deep Neural Networks: A Survey”  In _Proceedings of the IEEE_ , 2020, pp. 1–24  DOI: 10.1109/jproc.2020.2993293
  * (10) Tim Boland and Paul E. Black  “Juliet 1.1 C/C++ and Java Test Suite”  In _IEEE Computer_ 45.10, 2012, pp. 88–90  DOI: 10.1109/MC.2012.345
  * (11) Paul E. Black  “A Software Assurance Reference Dataset: Thousands of Programs With Known Bugs”, 2018
  * (12) M.K. Gupta, M.C. Govil and G. Singh  “Text-Mining and Pattern-Matching Based Prediction Models for Detecting Vulnerable Files in Web Applications”  In _J. Web Eng._ 17.1-2, 2018, pp. 28–44
  * (13) Mukesh Kumar Gupta, Mahesh Chandra Govil and Girdhari Singh  “Predicting Cross-Site Scripting (XSS) Security Vulnerabilities in Web Applications”  In _Int’l Joint Conf. Computer Science and Softw. Eng._ , 2015, pp. 162–167  DOI: 10.1109/jcsse.2015.7219789
  * (14) Serguei A. Mokhov, Joey Paquet and Mourad Debbabi  “MARFCAT: Fast Code Analysis for Defects and Vulnerabilities”  In _Int’l Ws. Softw. Analytics_ , 2015, pp. 35–38  DOI: 10.1109/swan.2015.7070488
  * (15) Ibéria Medeiros, Nuno F. Neves and Miguel Correia  “Automatic Detection and Correction of Web Application Vulnerabilities Using Data Mining to Predict False Positives”  In _Int’l Conf. World Wide Web_ , WWW ’14  New York, NY, USA: ACM, 2014, pp. 63–74  DOI: 10.1145/2566486.2568024
  * (16) S.A. Mokhov, J. Paquet and M. Debbabi  “The Use of NLP Techniques in Static Code Analysis to Detect Weaknesses and Vulnerabilities”  In _Canadian Conf. Artificial Intelligence_ 8436, LNAI, 2014, pp. 326–332  DOI: 10.1007/978-3-319-06483-3_33
  * (17) Yunhui Zheng, Saurabh Pujar, Burn Lewis, Luca Buratti, Edward Epstein, Bo Yang, Jim Laredo, Alessandro Morari and Zhong Su  “D2A: A Dataset Built for AI-Based Vulnerability Detection Methods Using Differential Analysis”  In _Int’l Conf. Softw. Eng.: Softw. Eng. in Practice_ IEEE, 2021, pp. 111–120
  * (18) Dimitris Mitropoulos, Georgios Gousios, Panagiotis Papadopoulos, Vassilios Karakoidas, Panagiotis Louridas and Diomidis Spinellis  “The Vulnerability Dataset of a Large Software Ecosystem”  In _Int’l Ws. Building Analysis Datasets and Gathering Experience Returns for Security_ Wroclaw, Poland: IEEE, 2014, pp. 69–74  DOI: 10.1109/badgers.2014.8
  * (19) “FindBugs”  URL: http://findbugs.sourceforge.net/
  * (20) Razvan Raducu, Gonzalo Esteban, Francisco J. Rodríguez Lera and Camino Fernández  “Collecting Vulnerable Source Code from Open-Source Repositories for Dataset Generation”  In _Applied Sciences_ 10.4, 2020, pp. 1270  DOI: 10.3390/app10041270
  * (21) Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du and Yang Liu  “Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks”  In _Conf. Neural Information Processing Systems_ , 2018, pp. 11
  * (22) Zhen Li, Deqing Zou, Shouhuai Xu, Xinyu Ou, Hai Jin, Sujuan Wang, Zhijun Deng and Yuyi Zhong  “VulDeePecker: A Deep Learning-Based System for Vulnerability Detection”  In _Network and Distributed System Security Symp._ , 2018  DOI: 10.14722/ndss.2018.23158
  * (23) Renan Vieira, Antônio da Silva, Lincoln Rocha and João Paulo Gomes  “From Reports to Bug-Fix Commits: A 10 Years Dataset of Bug-Fixing Activity from 55 Apache’s Open Source Projects”  In _Int’l Conf. Predictive Models and Data Analytics in Softw. Eng._ Recife Brazil: ACM, 2019, pp. 80–89  DOI: 10.1145/3345629.3345639
  * (24) Henrique Alves, Baldoino Fonseca and Nuno Antunes  “Software Metrics and Security Vulnerabilities: Dataset and Exploratory Study”  In _European Dependable Computing Conf._ , 2016, pp. 37–44  DOI: 10.1109/edcc.2016.34
  * (25) Antonios Gkortzis, Dimitris Mitropoulos and Diomidis Spinellis  “VulinOSS: A Dataset of Security Vulnerabilities in Open-Source Systems”  In _Int’l Conf. Mining Softw. Repositories_ Gothenburg Sweden: ACM, 2018, pp. 18–21  DOI: 10.1145/3196398.3196454
  * (26) Matthieu Jimenez, Yves Le Traon and Mike Papadakis  “Enabling the Continuous Analysis of Security Vulnerabilities with VulData7”  In _Int’l Working Conf. Source Code Analysis and Manipulation_ , 2018, pp. 56–61  DOI: 10.1109/scam.2018.00014
  * (27) Serena Elisa Ponta, Henrik Plate, Antonino Sabetta, Michele Bezzi and Cédric Dangremont  “A Manually-Curated Dataset of Fixes to Vulnerabilities of Open-Source Software”  In _Int’l Conf. Mining Softw. Repositories_ , 2019, pp. 383–387  DOI: 10.1109/msr.2019.00064
  * (28) Jiahao Fan, Li Li, Shaohua Wang and Tien N. Nguyen  “A C/C++ Code Vulnerability Dataset with Code Changes and CVE Summaries”  In _Int’l Conf. Mining Softw. Repositories_ , 2020, pp. 5
  * (29) G. Lin, W. Xiao, J. Zhang and Y. Xiang  “Deep Learning-Based Vulnerable Function Detection: A Benchmark”  In _Int’l Conf. Information and Communications Security_ 11999, LNCS, 2020, pp. 219–232  DOI: 10.1007/978-3-030-41579-2_13
  * (30) Yuxing Ma, Tapajit Dey, Chris Bogart, Sadika Amreen, Marat Valiev, Adam Tutko, David Kennard, Russell Zaretzki and Audris Mockus  “World of Code: Enabling a Research Workflow for Mining and Analyzing the Universe of Open Source VCS Data”  In _Empirical Softw. Eng._ 26.2, 2021, pp. 22  DOI: 10.1007/s10664-020-09905-9
  * (31) Santiago Dueñas, Valerio Cosentino, Gregorio Robles and Jesus M. Gonzalez-Barahona  “Perceval: Software Project Data at Your Will”  In _Int’l Conf. Softw. Eng._ , ICSE ’18  New York, NY, USA: ACM, 2018, pp. 1–4  DOI: 10.1145/3183440.3183475
  * (32) Davide Spadini, Maurício Aniche and Alberto Bacchelli  “PyDriller: Python Framework for Mining Software Repositories”  In _Joint European Softw. Eng. Conf. and Symp. Found. Softw. Eng._ , ESEC/FSE 2018  New York, NY, USA: ACM, 2018, pp. 908–911  DOI: 10.1145/3236024.3264598
  * (33) NIST  “National Vulnerability Database (NVD)”  URL: https://nvd.nist.gov/
  * (34) MITRE  “Common Weakness Enumeration (CWE)”  URL: https://cwe.mitre.org/
  * (35) Forum of Incident Response and Security Teams (FIRST)  “Common Vulnerability Scoring System (CVSS)”  URL: https://www.first.org/cvss
  * (36) Marco Biase, Ayushi Rastogi, Magiel Bruntink and Arie Deursen  “The Delta Maintainability Model: Measuring Maintainability of Fine-Grained Code Changes”  In _Int’l Conf. Technical Debt_ IEEE, 2019, pp. 113–122  DOI: 10.5281/zenodo.2606632
  * (37) Rafael-Michael Karampatsis and Charles Sutton  “How Often Do Single-Statement Bugs Occur? The ManySStuBs4J Dataset”  In _Int’l Conf. Mining Softw. Repositories_ , 2020, pp. 573–577  DOI: 10.1145/3379597.3387491
  * (38) X. Li, L. Wang, Y. Xin, Y. Yang and Y. Chen  “Automated Vulnerability Detection in Source Code Using Minimum Intermediate Representation Learning”  In _Applied Sciences_ 10.5, 2020  DOI: 10.3390/app10051692
  * (39) Nicholas Saccente, Josh Dehlinger, Lin Deng, Suranjan Chakraborty and Yin Xiong  “Project Achilles: A Prototype Tool for Static Method-Level Vulnerability Detection of Java Source Code Using a Recurrent Neural Network”  In _Int’l Conf. Autom. Softw. Eng._ San Diego, CA, USA: IEEE, 2019, pp. 114–121  DOI: 10.1109/asew.2019.00040
  * (40) Hoa Khanh Dam, Truyen Tran, Trang Thi Minh Pham, Shien Wee Ng, John Grundy and Aditya Ghose  “Automatic Feature Learning for Predicting Vulnerable Software Components”  In _IEEE Trans. Softw. Eng._ 47.1, 2018, pp. 67–85  DOI: 10.1109/TSE.2018.2881961
  * (41) P. Hegedűs  “Inspecting JavaScript Vulnerability Mitigation Patches with Automated Fix Generation in Mind”  In _Int’l Conf. Computational Science and Its Applic._ 12252 LNCS, 2020, pp. 975–988  DOI: 10.1007/978-3-030-58811-3_69
  * (42) M. Yang, J. Wu, S. Ji, T. Luo and Y. Wu  “Pre-Patch: Find Hidden Threats in Open Software Based on Machine Learning Method”  In _World Congress on Services_ 10975, LNCS, 2018, pp. 48–65  DOI: 10.1007/978-3-319-94472-2_4
  * (43) Antonino Sabetta and Michele Bezzi  “A Practical Approach to the Automatic Classification of Security-Relevant Commits”  In _Int’l Conf. Softw. Maintenance and Evolution_ IEEE, 2018  DOI: 10.1109/ICSME.2018.00058
  * (44) Adriaan Larmuseau and Devu Shila  “PatchSweetner: Exploit Detection Through the Automatic Transformation of Security Patches”  In _Military Communications Conf._ , 2018, pp. 939–945  DOI: 10.1109/milcom.2018.8599720
  * (45) Zimin Chen, Steve James Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk and Martin Monperrus  “SEQUENCER: Sequence-to-Sequence Learning for End-to-End Program Repair”  In _IEEE Trans. Softw. Eng._ , 2019  DOI: 10.1109/tse.2019.2940179
  * (46) Zimin Chen, Steve Kommrusch and Martin Monperrus  “Using Sequence-to-Sequence Learning for Repairing C Vulnerabilities”, 2019
  * (47) Ehsan Mashhadi and Hadi Hemmati  “Applying CodeBERT for Automated Program Repair of Java Simple Bugs”  In _Int’l Conf. Mining Softw. Repositories_ IEEE, 2021, pp. 5
  * (48) Balázs Mosolygó, Norbert Vándor, Gábor Antal and Péter Hegedűs  “On the Rise and Fall of Simple Stupid Bugs: A Life-Cycle Analysis of SStuBs”  In _Int’l Conf. Mining Softw. Repositories_ IEEE, 2021, pp. 5
  * (49) Fernanda Madeiral and Thomas Durieux  “A Large-Scale Study on Human-Cloned Changes for Automated Program Repair”  In _Int’l Conf. Mining Softw. Repositories_ IEEE, 2021, pp. 5
  * (50) Jiayi Hua and Haoyu Wang  “On the Effectiveness of Deep Vulnerability Detectors to Simple Stupid Bug Detection”  In _Int’l Conf. Mining Softw. Repositories_ IEEE, 2021, pp. 5
  * (51) Kim Herzig, Sascha Just and Andreas Zeller  “The Impact of Tangled Code Changes on Defect Prediction Models”  In _Empirical Softw. Eng._ 21.2 Springer, 2016, pp. 303–336  DOI: 10.1007/s10664-015-9376-6