#  Tracer: Finding Patches for Open Source Software Vulnerabilities

Congying Xu, Bihuan Chen, Chenhao Lu, Kaifeng Huang, Xin Peng, and Yang Liu
C. Xu, B. Chen, C. Lu, K. Huang, and X. Peng are with the School of Computer
Science and the Shanghai Key Laboratory of Data Science, Fudan University,
Shanghai, China.
E-mail: {xucy19, bhchen, luch18, kfhuang16<EMAIL_ADDRESS>Y. Liu is
with the School of Computer Science and Engineering, Nanyang Technological
University, Singapore.
E-mail<EMAIL_ADDRESS>B. Chen is the corresponding author.

###### Abstract

Open source software (OSS) vulnerability management has become an open
problem. Vulnerability databases provide valuable data that is needed to
address OSS vulnerabilities. However, there arises a growing concern about the
information quality of vulnerability databases. In particular, it is unclear
how the quality of patches in existing vulnerability databases is. Further,
existing manual or heuristic-based approaches for patch identification are
either too expensive or too specific to be applied to all OSS vulnerabilities.
To address these problems, we first conduct an empirical study to understand
the quality and characteristics of patches for OSS vulnerabilities in two
state-of-the-art vulnerability databases. Our study is designed to cover five
dimensions, i.e., the coverage, consistency, type, cardinality and accuracy of
patches. Then, inspired by our study, we propose the first automated approach,
named Tracer, to find patches for an OSS vulnerability from multiple sources.
Our key idea is that patch commits will be frequently referenced during the
reporting, discussion and resolution of an OSS vulnerability. Our extensive
evaluation has indicated that i) Tracer finds patches for up to 273.8% more
CVEs than existing heuristic-based approaches while achieving a significantly
higher F1-score by up to 116.8%; and ii) Tracer achieves a higher recall by up
to 18.4% than state-of-the-art vulnerability databases, but sacrifices up to
12.0% fewer CVEs (whose patches are not found) and 6.4% lower precision. Our
evaluation has also demonstrated the generality and usefulness of Tracer.

###### Index Terms:

Open Source Software, Vulnerabilities, Patches

##  1 Introduction

Open source software (OSS) provides the foundation for open source and
proprietary applications. It allows application developers to reuse common
functionalities instead of reinventing the wheel. The “Open Source Security
and Risk Analysis Report” [31] has revealed that 98% of the 1,500 applications
across 17 industries contain OSS. However, security risks are also introduced
with OSS; e.g., 84% of the 1,500 applications in 2020 contain at least one
public OSS vulnerability, a 9% increase from the 75% in 2019, with an average
of 158 OSS vulnerabilities per application [31]. Even worse, OSS
vulnerabilities are found at an increasing speed, nearly doubling in the last
two years [63].

To this end, tremendous efforts have been made to mitigate security risks in
OSS vulnerabilities, ranging from detecting vulnerabilities in open source
software via learning vulnerability features [49, 50, 73, 45], or matching
vulnerability and/or patch signatures [42, 47, 67, 66, 35], to patching
vulnerabilities in open source software [54, 39, 69, 52], or analyzing
software composition for determining whether open source vulnerabilities are
reachable through call paths in applications [58, 60, 59, 65]. In particular,
vulnerability databases play a significant role in these efforts by providing
valuable data (e.g., description, affected software and versions, and patches)
for various vulnerability analysis tasks. Several forces have devoted
themselves to building vulnerability databases. In security community, CVE
List [15] and NVD [21] are the most influential forces, which even go beyond
OSS vulnerabilities. In industry, Black Duck [14], WhiteSource [28], Veracode
[26] and Snyk [25] are the mainstreams to have a specific focus on OSS
vulnerabilities. In academia, several vulnerability databases [61, 40, 44, 41,
55] have been constructed, but they are designed mostly for specific
ecosystems or specific projects.

Problem. As these vulnerability databases are accumulating a massive
collection of vulnerabilities, however, there arises an increasing concern
about the information quality of vulnerabilities. Nguyen and Massacci [57] and
Dong et al. [38] reported the unreliability of the vulnerable version data in
vulnerability databases. Chaparro et al. [32] and Mu et al. [53] quantified
the prevalence of missing reproducing steps in vulnerability descriptions.
Such missing or inaccurate information makes it challenging to timely
identify, reproduce and patch vulnerabilities in applications and their used
OSS.

Patches, as a valuable piece of information to capture a vulnerability, can be
leveraged to enable a variety of security applications, e.g., hot patch
generation and deployment [54, 39, 69], patch presence testing [72, 43, 36],
software composition analysis [60, 59, 65], and vulner-ability detection [49,
50, 42, 47, 66, 35]. The accuracy of these security applications will be
heavily affected if the patches of a vulnerability are missing or inaccurate.
However, it is unclear how the quality of patches in existing vulnerability
databases is.

Further, existing security applications locate vulnerability patches mostly by
manual efforts [69, 43, 36, 74, 62, 34, 66, 60, 59], by heuristic rules like
looking for commits in CVE references [39, 49, 50] and searching for CVE
identifiers in commits [70, 65], or from security advisories that have already
established the mapping between vulnerabilities and patches for specific
projects [54, 42, 47]. However, these patch identification approaches are
either too expensive or too specific to apply to all OSS vulnerabilities.

Empirical Study. To address these problems, we conduct an empirical study to
understand the quality and characteristics of patches for OSS vulnerabilities
in two state-of-the-art vulnerability databases (i.e., Veracode [26] and Snyk
[25]). In particular, we analyze patch coverage and consistency based on a
breadth dataset that contains all OSS vulnerabilities (i.e., 10,070 CVEs) in
the two vulnerability databases. Our results have indicated that only 4,602
(45.7%) of the 10,070 CVEs have patches provided by at least one vulnerability
database. Only 19.7% CVEs have consistent patches across two databases.

Further, we analyze patch type, cardinality and accuracy based on a depth
dataset of 1,295 CVEs for which both two databases report their patches and we
also manually find their accurate patches. Our results have indicated that
1,265 (97.7%) of the 1,295 CVEs have their patches in the type of GitHub and
SVN commits. 533 (41.1%) CVEs have a one-to-many mapping to their patches. The
two databases have a patch precision of above 90%. However, for the CVEs
having a one-to-many mapping to patches, their patch recall is only about 50%.

Our Approach. Inspired by the findings from our study, we propose an automated
approach, named Tracer, to find patches (in the form of commits) for an OSS
vulnerability from multiple sources (i.e., NVD [21], Debian [17], Red Hat [23]
and GitHub). Our key idea is that patch commits will be frequently referenced
during the reporting, discussion and resolution of an OSS vulnerability.
Tracer works in three steps. First, it constructs a reference network for a
CVE, starting from multiple sources, to model resource references during CVE
reporting, discussion and resolution. Second, it selects the patch nodes
(i.e., commits) in the network that have high connectivity and high
confidence. Finally, it expands the selected patch nodes through searching
relevant commits across branches of a repository; i.e., it builds a potential
one-to-many mapping between the CVE and its patches.

Evaluation. To evaluate the accuracy of Tracer, we compare it with three
heuristic-based approaches and two industrial state-of-the-art vulnerability
databases on our depth dataset. Our evaluation has shown that i) Tracer can
find patches for 58.6% to 273.8% more CVEs than heuristic-based approaches,
while having either a comparable F1-score or a significantly higher F1-score
by up to 116.8%; and ii) Tracer can achieve a 15.5% to 18.4% higher recall and
a 5.5% to 8.6% higher F1-score than industrial databases, while sacrificing up
to 12.0% fewer CVEs (whose patches are not found) and 6.4% lower precision.

Moreover, to evaluate the generality of Tracer, we run it over two more
databases of 3,185 and 5,468 CVEs, for which only one or none vulnerability
database includes their patches. Our evaluation has indicated that Tracer
finds patches for 67.7% and 51.5% of the CVEs, and achieves a sampled
precision of 0.823 and 0.888, and a sampled recall of 0.845 and 0.899.
Besides, to evaluate the usefulness of Tracer, we conduct a user study with 10
participants. Our evaluation has shown that Tracer helps locate patches more
accurately and quickly.

Contribution. This work makes the following contributions.

  * •

We conduct an empirical study to understand the quality and characteristics of
patches for OSS vulnerabilities in two state-of-the-art vulnerability
databases with respect to five dimensions.

  * •

We propose the first automated approach, named Tracer, to find patches of an
OSS vulnerability from multiple sources for the security community, industry,
and academia.

  * •

We conduct a set of extensive experiments to demonstrate the accuracy and
generality of Tracer on three datasets as well as the usefulness of Tracer in
practice with a user study.

##  2 An Empirical Study

We design an empirical study to understand the quality and characteristics of
patches for OSS vulnerabilities in different vulnerability databases by
answering the following research questions.

  * •

RQ1 Coverage Analysis: how many OSS vulnerabilities have patches included in
vulnerability databases? (Sec. 2.2)

  * •

RQ2 Consistency Analysis: how many OSS vulnerabilities have the same patches
across vulnerability databases? (Sec. 2.3)

  * •

RQ3 Type Analysis: what are the common patch types for OSS vulnerabilities in
vulnerability databases? (Sec. 2.4)

  * •

RQ4 Cardinality Analysis: what are the mapping cardinalities between OSS
vulnerabilities and their patches? (Sec. 2.5)

  * •

RQ5 Accuracy Analysis: how is the patch accuracy of OSS vulnerabilities in
vulnerability databases? (Sec. 2.6)

We design RQ1 to measure the prevalence of missing patches for OSS
vulnerabilities in different vulnerability databases. We use RQ2 to quantify
the prevalence of inconsistent patches for OSS vulnerabilities across
different vulnerability databases. We leverage RQ3 and RQ4 to capture the
common patch types and mapping cardinalities between OSS vulnerabilities and
their patches. We develop RQ5 to assess the accuracy of patches for OSS
vulnerabilities in different vulnerability databases. In summary, our results
from RQ1, RQ2 and RQ5 aim to assess the patch quality from different
perspectives and motivate the need for an automated approach to accurately
find patches for OSS vulnerabilities, and our results from RQ3 and RQ4 aim to
capture the characteristics of patches from different perspectives, and
inspire the design of our automated patch finding approach.

###  2.1 Data Preparation

Vulnerability Database Selection. We investigated vulnerability databases from
the security community, industry, and academia. We excluded databases from the
security community (e.g., CVE List and NVD) because i) they do not have
patches structurally included but have patches potentially hidden in
references, and ii) they contain vulnerabilities that go beyond OSS
vulnerabilities. We also excluded databases from academia (e.g., [61, 40, 44,
41, 55, 48, 51, 29]) because i) they mostly only cover specific ecosystems or
projects and ii) they lack continued maintenance and become out of date after
publication. Thus, we restricted our selection to industrial databases.

Specifically, we selected the vulnerability databases from four companies,
Black Duck [14], WhiteSource [28], Veracode [26] and Snyk [25]. They provide
software composition analysis to identify OSS used in an application and
report any OSS vulnerabilities. Hence, we believe they achieve good coverage
of OSS vulnerabilities. As some of them either do not make the vulnerability
database publicly accessible or do not include patches for a vulnerability, we
finally selected two out of the four databases, i.e., Veracode’s and Snyk’s
vulnerability databases. Hereafter they are referred to as $DB_{A}$ and
$DB_{B}$.

Breadth Dataset Construction. To broadly quantify missing patches in
vulnerability databases and inconsistent patches across them (i.e., RQ1 and
RQ2), we built a breadth dataset of OSS vulnerabilities by acquiring all OSS
vulnerabilities from $DB_{A}$ and $DB_{B}$ as of April 7, 2020. We obtained
8,630 and 5,858 CVEs from $DB_{A}$ and $DB_{B}$.

Depth Dataset Construction. To accurately characterize patch types, mapping
cardinalities and patch accuracy (i.e., RQ3, RQ4 and RQ5), we built a depth
dataset of OSS vulnerabilities, whose size is smaller than the breadth dataset
but whose patches are all manually identified to ensure the completeness and
accuracy. Specifically, to balance the ease of patch accuracy analysis across
two databases and the effort of manual patch identification, we selected 1,417
CVEs for which both databases reported their patches. For each CVE, two of the
authors separately found its patches by analyzing patches reported by both
databases, looking into CVE description and references in NVD, and searching
GitHub repositories and Internet resources. Then, they exchanged sources and
ways to find patches,  investigated inconsistent cases together, and revisited
all CVEs for several rounds until reaching a consensus. Finally, they
successfully found patches for 1,295 CVEs, while they were uncertain for 122
CVEs due to limited disclosed knowledge.

(a) Our Depth Dataset w.r.t. Years

(b) Our Depth Dataset w.r.t. Programming Languages

Figure 1: Our Depth Dataset w.r.t Years and Programming Languages

We further analyzed the 1,295 CVEs in our depth dataset with respect to years
and programming languages. We determined the programming language of a CVE by
analyzing the changed source files in patches. As shown in Fig. 1(a), the
number of CVEs increases every year, which is consistent with Snyk’s report
[63]. As reported in Fig. 1(b), these CVEs mainly cover seven programming
languages, which demonstrates relatively good coverage of ecosystems.
Therefore, we believe that our depth dataset is representative of OSS
vulnerabilities.

###  2.2 Coverage Analysis (RQ1)

(a) CVEs

(b) CVEs with Patches

Figure 2: Intersection Among Two Databases

Fig. 2(a) shows the intersection of CVEs among the two databases, and Fig.
2(b) presents the intersection of CVEs with patches among the two databases.
It can be observed that 3,607 (41.8%) and 2,412 (41.2%) of the CVEs have their
patches identified in $DB_{A}$ and $DB_{B}$. This result indicates that both
vulnerability databases have a moderately low patch coverage, and missing
patches are prevalent. Moreover, it can be seen that different vulnerability
databases have different coverage of OSS vulnerabilities. $DB_{A}$ and
$DB_{B}$ have an overlap of 4,418 CVEs, while respectively covering 4,212 and
1,440 unique CVEs. Overall, the two vulnerability databases contain a union of
10,070 CVEs, and 4,602 (45.7%) of them have patches provided by at least one
vulnerability database. These results motivate the need for automated patch
finding approaches to help find missing patches.

###  2.3 Consistency Analysis (RQ2)

TABLE I: Patch Consistency and Inconsistency Results
---
Cons. | Existence Inconsistency | Content Inconsistency
|  Total |  No CVE |  No Patch |  Total |  Inclusion |  Difference
907 (19.7%) |  3,185 (69.2%) |  1,392 (30.2%) |  1,793 (39.0%) |  510 (11.1%) |  176 (3.8%) |  334 (7.3%)

To analyze patch consistency across the two databases, we focus on CVEs with
patches (i.e., Fig. 2(b)). As a CVE may have a set of patches, we consider two
databases as having consistent patches for a CVE if their patch sets are the
same. We distinguish patch inconsistency between existence and content
inconsistency. The former refers to two cases that one database provides
patches for a CVE, but the other database either does not cover the CVE, or
covers the CVE but does not provide patches. This reflects the incompleteness
of OSS vulnerabilities and their patches. The latter refers to two cases that
both databases provide patches for a CVE, and their patch sets have an
inclusion relationship, or do not have an inclusion relationship but are
different. This reflects the potential inaccuracy of patches.

Table I reports our patch consistency analysis results. The first column gives
the number of CVEs with consistent patches. The second to fourth columns
report the number of CVEs with existence inconsistent patches, and last three
columns show the number of CVEs with content inconsistent patches. It can be
observed that i) only 907 (19.7%) of the 4,602 CVEs have consistent patches;
ii) more than two-thirds (i.e., 3,185 (69.2%)) of the CVEs have existence
inconsistency, where 1,392 (30.2%) of the CVEs are not included in $DB_{A}$ or
$DB_{B}$, and 1,793 (39.0%) of the CVEs are included but do not have patches
in $DB_{A}$ or $DB_{B}$; and iii) 510 (11.1%) of the CVEs incur content
inconsistency, where 176 (3.8%) of the CVEs’ patches from one database are
included in the patches from the other; and 334 (7.3%) of the CVEs have
different and non-inclusive patch sets from $DB_{A}$ and $DB_{B}$. These
results indicate that these vulnerability databases often report inconsistent
patches, the incompleteness of both OSS vulnerabilities and their patches is
severe in these vulnerability databases.

###  2.4 Type Analysis (RQ3)

We find 3,043 patches for the 1,295 CVEs in our depth dataset by manual
analysis. Specifically, 2,852 (93.7%) patches are in the type of GitHub
commits potentially because of the wide adoption of GitHub across open source
software. 136 (4.5%) patches are in the type of SVN commits potentially due to
the prevalence of SVN before the introduction of GitHub, whereas only 55
(1.8%) patches are in the type of commits from other Git platforms. Besides,
from the perspective of CVEs, 1,202 (92.8%) of the 1,295 CVEs have their
patches in the type of GitHub commits, 4 (0.3%) CVEs have their patches in the
type of SVN commits, and 48 (3.7%) CVEs have their patches in the type of both
GitHub and SVN commits due to the migration from SVN to GitHub. Only 30 (2.3%)
CVEs have some patches in the type of commits from other Git platforms. These
results demonstrate that patches for OSS vulnerabilities are mostly in the
type of GitHub and SVN commits.

###  2.5 Cardinality Analysis (RQ4)

We categorize three types of mapping cardinalities between CVEs and their
patches, i.e., one-to-one, one-to-some, and one-to-many. In particular, 567
(43.8%) of the CVEs have a one-to-one mapping to their patch; i.e., they have
only one single patch to fix the vulnerability. Hereafter this category is
referred to as SP.

195 (15.1%) of the CVEs have a one-to-some mapping to their patches, meaning
that they have multiple equivalent patch sets and any one of the patch sets is
sufficient to patch the vulnerability. Hereafter we refer to this category as
MEP. Two patches are equivalent if they have the same code differences. It is
mainly caused by two reasons. First, a CVE is patched by a pull request which
is merged. Thus, the pull request commits and merged commits are equivalent
patch sets for the CVE. Second, the repository of OSS is migrated from SVN to
GitHub. Thus, the commits for patching a CVE can be in the repository on SVN
and GitHub, and SVN commits and GitHub commits are equivalent.

533 (41.2%) of the CVEs have a one-to-many mapping to their patches, which can
be further classified into three types. First, a CVE is fixed by multiple
separate commits in a branch. This is because the CVE is difficult to fix or
the initial patch is not complete. Hereafter we refer to this type as MP,
accounting for 101 (7.8%) of the CVEs. Second, a CVE is fixed by multiple
patch sets in multiple branches because the CVE affects multiple versions of
OSS and each version is maintained on a separate branch. These multiple patch
sets should all be identified as patches for different versions can be
different. Hereafter this type is referred to as MB, covering 372 (28.7%) of
the CVEs. Third, a CVE is fixed by multiple patch sets in multiple
repositories. This is because the CVE affects multiple OSS, or multiple
versions of OSS are maintained in separate repositories. Hereafter we refer to
this type as MR, which covers 60 (4.6%) of the CVEs.

These results demonstrate various mapping cardinalities between CVEs and their
patches. They should be considered to ensure data completeness when finding
patches for OSS vulnerabilities.

###  2.6 Accuracy Analysis (RQ5)

TABLE II: Patch Accuracy of Two Databases
---
Cardinality |  Number | $DB_{A}$ | $DB_{B}$
|  |  Pre. |  Rec. |  F1 |  Pre. |  Rec. |  F1
1:1 (SP) |  567 |  0.908 |  0.915 |  0.910 |  0.900 |  0.921 |  0.906
1:$i$ (MEP) |  195 |  0.935 |  0.898 |  0.902 |  0.924 |  0.909 |  0.906
1:$n$ (MP) |  101 |  0.923 |  0.483 |  0.616 |  0.911 |  0.520 |  0.638
1:$n$ (MB) |  372 |  0.941 |  0.510 |  0.620 |  0.932 |  0.436 |  0.555
1:$n$ (MR) |  60 |  0.913 |  0.610 |  0.695 |  0.964 |  0.526 |  0.636
Total |  1,295 |  0.923 |  0.748 |  0.793 |  0.917 |  0.730 |  0.771

We use precision, recall, and F1-score as the indicators of patch accuracy.
For the CVEs having one-to-some mapping to their patches, we consider
reporting one of the multiple equivalent patches as correct. For example, for
a CVE that has two equivalent patches, a database that reports one of the two
equivalent patches has a full precision and a full recall, while a database
that reports one of the two equivalent patches and another irrelevant patch
achieves a half precision and a full recall. Table II breaks down the accuracy
results of the two databases with respect to the mapping cardinalities. The
second column reports the number of CVEs in each mapping cardinality, and the
next six columns report the precision, recall and F1-score of reported CVEs’
patches in the two databases.

$DB_{A}$ and $DB_{B}$ achieve a high precision and a high recall of about 90%
for the CVEs belonging to SP and MEP, while having a high precision of above
90% but a low recall of around 50% for the CVEs having one-to-many mappings to
their patches (i.e., MP, MB and MR). These results show that these
vulnerability databases often miss some patches, especially for CVEs with
multiple patches, and such missing information would make it challenging to
detect and patch vulnerabilities timely in applications and their used OSS. It
reflects the need to automatically find complete patches for OSS
vulnerabilities.

##  3 Our Approach

Figure 3: Approach Overview of Tracer

Based on the findings from our study, we propose an automated approach, named
Tracer, to find patches (in the form of commits) for an OSS vulnerability. The
underlying idea of Tracer is that patch commits will be frequently referenced
during the reporting, discussion and resolution of an OSS vulnerability in
various advisory sources. Fig. 3 presents an overview of Tracer. It takes as
an input the CVE identifier of an OSS vulnerability, and returns its patches.
Specifically, Tracer works in three steps. First, it constructs a reference
network for the CVE starting from multiple advisory sources (i.e., NVD, Debian
[17], Red Hat [23] and GitHub). The goal is to model resource references
during the reporting, discussion and resolution of the CVE. Here, we consider
NVD as the main advisory source, Debian, Red Hat and GitHub as the secondary
advisory sources which can be further extended. Second, it selects the patch
nodes (i.e., commits) in the network which have high connectivity and high
confidence, and thus are most likely to be patches for the CVE. Finally, it
expands the selected patches via searching relevant commits across branches of
the same repository. The goal is to establish a potential one-to-many mapping
between the CVE and its patches. In the following subsections, we will
elaborate on each step in detail.

###  3.1 Reference Network Construction

The first step of Tracer consists of three sub-steps. The first two sub-steps
(i.e., advisory analysis and reference analysis) construct a reference network
via analyzing advisories from NVD, Debian and Red Hat. The third sub-step
(i.e., reference augment) augments the reference network by searching relevant
commit links from GitHub.

Advisory Analysis. Tracer first initializes the reference network by setting
the CVE under analysis as the root node, and then adds three advisory source
nodes (i.e., NVD, Debian and Red Hat) as the child node of the root node.
These advisory source nodes are used to visualize where the finally selected
patches originate.

Figure 4: Reference Network for CVE-2017-11428

###### Example 3.1

Fig. 4 presents the complete reference network for CVE-2017-11428. The top
layer shows the root node, and the second layer shows the advisory source
nodes.

Then, Tracer respectively requests the advisory from NVD, Debian and Red Hat
with the CVE identifier. Specifically, as NVD provides structured data feeds
[22] of all vulnerabilities in the form of JSON by year, Tracer requests and
parses the corresponding JSON file to obtain the NVD advisory. As Debian
stores advisories at a repository [16], Tracer extracts the Debian advisory
from the repository. As Red Hat provides WebService API [27], Tracer uses it
to retrieve the Red Hat advisory. Notice that Debian tracks all CVEs on NVD,
and Red Hat tracks some of them.

Tracer extracts URL references in each requested advisory and adds them as
child nodes of the corresponding advisory source node. For an NVD advisory,
Tracer extracts URL references in the “references” field, where references to
advisories and solutions are listed. Similarly, for a Debian advisory, Tracer
extracts URL references in the “Notes” field. For a Red Hat advisory, Tracer
uses a regular expression to extract URL references in the “comments” field,
where developers discuss and record the resolution process of the
vulnerability and may list references to patches.

###### Example 3.2

As shown in the third layer in Fig. 4, the NVD advisory for CVE-2017-11428
contains two references. One is a reference to a blog that describes the
technical detail of this vulnerability, and the other is a reference to a
third-party advisory. The two references are also contained by the Debian
advisory which further contains a reference to a GitHub commit ruby-
saml@048a54 [1] which fixes this vulnerability. Red Hat does not collect this
CVE.

Tracer also classifies these reference nodes into three types, i.e., patch
node, issue node and hybrid node. We distinguish patch nodes as our goal is to
find patches for the CVE. We distinguish issues nodes because issue trackers
may assign an issue identifier to the CVE, where developers discuss its
resolution and often list references to patches. Reference nodes that are not
identified as patch or issue nodes are regarded as hybrid nodes, which can be
blogs, third-party advisories, etc. Inspired by our patch type analysis (Sec.
2.4), Tracer identifies a reference node as a patch node if its URL contains
“git” and matches a regular expression of commit identifier (i.e., Git
platform commits), or contains “svn” and matches a regular expression of
commit identifier (i.e., SVN commits). Tracer identifies a reference node as
an issue node if its URL contains “/github.com/” and “/issues/” (i.e., GitHub
issues), or contains “/github.com/” and “/pull/” (i.e., GitHub pull requests),
or contains one of the keys “bugzilla”, “jira”, “issues”, “bugs”, “tickets”
and “tracker” and matches a regular expression of issue identifier (i.e.,
issues from other issue trackers).

###### Example 3.3

As shown in the third layer in Fig. 4, the two references contained in both
the NVD and Debian advisory are identified as hybrid nodes (i.e., the two
purple nodes). The reference that is only contained in the Debian advisory is
successfully identified as a patch node (i.e., the red node).

Reference Analysis. For each of the reference nodes constructed in the
previous sub-step, Tracer applies the following two analyses to construct the
reference network in a layered way.

If the reference node is a patch node, Tracer requests the commit and analyzes
whether it only changes test code or non-source code files. If yes, this patch
node is removed from the reference network as it cannot be the patch for the
CVE. Tracer identifies test code changes by checking the “test” string in
paths of modified files, and identifies non-source code changes by checking
the suffix of modified files.

If the reference node is an issue or hybrid node, Tracer first requests the
URL and gets the response (i.e., HTML text). Then, it uses a regular
expression to extract URL references in plain text, and uses an HTML parser to
extract URL references in hyperlinks (i.e., ¡a¿ tags). These extracted URL
references are then checked in the same way as the previous sub-step to
identify patch and issue references, which are added as child nodes of the
reference node under analysis. No more hybrid references will be added after
this layer because the deeper we explore the reference network, the more noise
would be introduced by hybrid references. In other words, only the hybrid
references directly contained in the NVD, Debian, and Red Hat advisories are
included in the reference network. There is one exception in the above
analysis for URL references to GitHub issues or commits. GitHub issue reports
often contain references to issues or commits from other repositories, which
can introduce noise to the reference network. To this end, if the reference
node under analysis corresponds to a GitHub issue, its extracted URL reference
that is not from the same repository will not be added to the reference
network.

Tracer repeats the above two analyses on the newly-added nodes until there is
no newly-added node or the depth of our reference network reaches a limit
(which is 5 by default).

###### Example 3.4

In the first iteration, Tracer keeps the patch node ruby-saml@048a54 in the
third layer in Fig. 4 because it changes non-test source code files. It
identifies that one hybrid node in the third layer does not reference to any
issue or commit, and the other references to two issues SAMLBase#3 and
saml#140 and one commit SAMLBase@482cdf. In the next iteration, it finds that
SAMLBase#3 references to SAMLBase@482cdf, and saml#140 references to two
issues gosaml2#36 and saml#163 and two commits saml@814d1d and saml@55d682.
However, gosaml2#36 is not included in our reference network as it belongs to
a different repository than saml#140; and saml@814d1d and saml@55d682 are also
not included because they only change test code. Notice that these not
included nodes are still shown in Fig. 4 for the ease of presentation, but are
connected by dotted arrow lines. After this iteration, the depth limit is
reached.

Reference Augment. Besides NVD, Debian and Red Hat which are explicit advisory
sources, repository hosting platforms can be regarded as an implicit source
because patches are often hidden in the commit history. Therefore, in this
sub-step, Tracer searches repository hosting platforms for patch commits of
the CVE in order to further augment the reference network.

Inspired by our patch type analysis (Sec. 2.4), here we only search GitHub as
most patches are in the type of GitHub commits. Besides, issues trackers often
assign an issue identifier to the CVE. Similarly, advisory publishers usually
assign an advisory identifier to the CVE. For example, the vendor advisory of
CVE-2019-10426 assigns an advisory identifier of SECURITY-1573 [2], and the
issue tracker assigns an issue identifier of THRIFT-4647 [3] to
CVE-2018-11798. Hence, Tracer uses a regular expression to extract issue and
advisory identifiers respectively from the URL of issue and hybrid nodes in
our reference network constructed in the previous two sub-steps.

Then, Tracer uses the CVE identifier and extracted issue and advisory
identifiers as the key to search for commits by REST API [18] provided by
GitHub. This API returns up to 1,000 results for a search. To reduce noise,
for each returned commit, Tracer checks whether its owner and repository name
matches the vendor and product name of any CPE of the CVE. CPE is a structured
naming scheme for affected software of the CVE, which can be parsed from the
JSON file from NVD. Here we follow Dong et al.’s matching criterion [38] to
have the flexibility to handle the slightly different format of the same
software name; i.e., two software names are regarded as a match if the number
of matched words is not less than the number of unmatched words. Besides,
Tracer also checks whether the commit changes non-test source code files. If
both checks are passed, Tracer adds it as a child node of the advisory source
node of GitHub.

###### Example 3.5

For CVE-2017-11428, Tracer fails to extract any issue or advisory identifier.
Thus, it uses the CVE identifier to search for GitHub commits. A returned
commit is ruby-saml@048a54, and its owner and repository name is “onelogin”
and “ruby-saml”. As the vendor and product name in the CPE of this CVE is
“onelogin” and “ruby-saml”, a complete match is achieved. As this commit is
already included in our reference network, Tracer connects it as a child node
of the new advisory source node of GitHub, as shown in Fig. 4. There is no
other matched commit from the search results.

###  3.2 Patch Selection

The second step of Tracer is to select from our reference network the patches
for the CVE under analysis accurately and completely. To achieve this goal, we
design the following two heuristics.

Confidence. We directly select the patch nodes that we treat as having high
confidence of being the correct patch for the CVE under analysis.
Specifically, we consider two kinds of patch nodes in our reference network as
having such high confidence. First, the patch nodes that are a direct child
node of the advisory source node of NVD are considered as having high
confidence. The reason is that NVD is established with a strong community
effort, each vulnerability is manually confirmed with several procedures, and
the data can be continuously updated after the initial vulnerability
reporting. This heuristic is also used in the literature to find patches
(e.g., [39, 49, 50]). Second, the patch nodes that are a direct child node of
the advisory source node of GitHub are considered as having high confidence.
The reason is that the way Tracer adds such patch nodes ensures that the
commit message contains the CVE identifier, advisory identifier, or issue
identifier of the CVE and the name of its belonging owner and repository
matches the vendor and product name of the CPE. This heuristic is similarly
used in the literature to find patches for CVEs of a known open source
software (e.g., [70, 65]).

###### Example 3.6

From the reference network for CVE-2017-11428 in Fig. 4, Tracer directly
selects the patch node ruby-saml@048a54 because it is a child node of the
advisory source node of GitHub and is considered as having high confidence of
being the correct patch for CVE-2017-11428. In fact, this commit is one of the
correct patches.

Connectivity. The confidence-based heuristic alone is often not strong enough
to locate patches accurately and completely because NVD may not contain patch
references, and CVE identifier, advisory identifier or issue identifier of a
CVE might not be contained in commit messages. Hence, inspired by the idea
that a correct patch would be frequently referenced during the reporting,
discussion, and resolution of a CVE in various advisory sources (i.e., the
patch node would be widely connected to the root node in our reference
network), we further design a connectivity-based heuristic.

Specifically, we measure the connectivity of a patch node to the root node in
our reference network based on two dimensions. First, the more paths the root
node can reach a patch node, the higher connectivity to the root node the
patch node has. Second, the shorter the paths from the root node to a patch
node, the higher connectivity to the root node the patch node has. To combine
these two dimensions, we use Eq. 1 to compute the connectivity of a patch node
to the root node, where $p=1,...,n$ denotes one of the $n$ paths from the root
node to the patch node, and $d_{p}$ denotes the length of a path $p$.
Considering the high confidence of the two advisory sources of NVD and GitHub,
the length of a path is changed by a decrease of 1 if the path originates from
the advisory source node of NVD and GitHub.

$connectivity=\sum_{p=1}^{n}\frac{1}{2^{({d}_{p}-1)}}$ (1)

Based on the connectivity of each patch node to the root node, Tracer selects
the patch nodes with the highest connectivity.

###### Example 3.7

In Fig. 4, there are two paths from the root node to the patch node ruby-
saml@048a54. One originates from Debian with a length of 2, and has
connectivity of 0.5. The other originates from GitHub with an original length
of 2 and a changed length of 1, and has connectivity of 1. Thus, the
connectivity of ruby-saml@048a54 to the root node is 1.5. Similarly, there
exist four paths from the root node to the patch node SAMLBase@482cdf,
respectively having connectivity of 0.5, 0.25, 0.25, and 0.125. Hence, the
connectivity of SAMLBase@482cdf to the root node is 1.125. Tracer selects
ruby-saml@048a54 as the patch as it has the highest connectivity.

###  3.3 Patch Expansion

The third step of Tracer is to expand the patches selected in the second step
by searching relevant commits across branches of the same repository. This is
inspired by our cardinality analysis (Sec. 2.5) where we find more than 40% of
the CVEs have a one-to-many mapping to their patches, and these multiple
patches often locate in one branch of a repository (as a CVE is difficult to
fix or the initial patch is not complete) or multiple branches of a repository
(as a CVE affects multiple versions whose branches are separately maintained).
For these multiple patches, our reference network constructed in the previous
two steps often does not capture them completely. Besides, our patch type
analysis (Sec. 2.4) shows that most patches are in the type of GitHub commits.
Therefore, the third step of Tracer is designed as follows: for each selected
patch that is in the type of a GitHub commit, Tracer locates its repository,
collects all branches in this repository, and searches the commits within a
specific span of each branch for commits that are potentially patches for the
CVE under analysis.

Specifically, for a selected patch that is in the type of a GitHub commit,
Tracer uses a regular expression to extract the owner and repository
information from the patch URL, owing to the well-structured commit URL in
GitHub. Based on the owner and repository information, Tracer retrieves all
branches in the repository by GitHub’s REST API [19]. Then, for each branch,
Tracer retrieves the commits created before and after the selected patch
within a specific span (which is 30 days by default) by GitHub’s REST API
[20]. We do not retrieve all the commits for balancing performance and
accuracy. Then, for each retrieved commit, Tracer uses the following two
criteria to determine whether the commit is the patch for the CVE under
analysis: i) the commit message of the retrieved commit is the same as, or
contains, or is contained by the commit message of the selected patch; or ii)
the commit message of the retrieved commit contains the CVE identifier,
advisory identifier or issue identifier. If a retrieved commit satisfies one
of the two criteria, Tracer also adds such expanded patches as child nodes of
the selected patch.

Finally, Tracer returns the selected patches in the second step and the
expanded patches in the third step as the patches for the CVE under analysis.
Besides, our reference network is also returned for the ease of confirming
returned patches.

###### Example 3.8

For the selected patch ruby-saml@048a54 (locating on the master branch) for
CVE-2017-11428, Tracer expands it by finding three commits ruby-saml@d7ce60
[4], ruby-saml@a35f72 [5] and ruby-saml@03af9e [6] which have the same commit
message to ruby-saml@048a54 but respectively locate on branches 0.8.3 –
0.8.17, v0.9.3 and v1.6.2. As shown in Fig. 4, Tracer adds them as child nodes
of ruby-saml@048a54. Notice that these four patches are all correct and
involve different code changes. The two vulnerability databases in Sec. 2 only
report the patch ruby-saml@048a54.

##  4 Evaluation

To comprehensively evaluate Tracer, we design our evaluation to answer the
following four research questions.

  * •

RQ6 Accuracy Evaluation: how is the accuracy of Tracer in finding
vulnerability patches, compared to existing heuristic-based approaches and two
industrial vulnerability databases? (Sec. 4.1)

  * •

RQ7 Ablation Analysis: how is the contribution of each component in Tracer to
its achieved accuracy? (Sec. 4.2)

  * •

RQ8 Sensitivity Analysis: how is the sensitivity of the accuracy of Tracer to
each parameter in Tracer? (Sec. 4.3)

  * •

RQ9 Generality Evaluation: how is the generality of Tracer to a wider range of
OSS vulnerabilities? (Sec. 4.4)

  * •

RQ10 Usefulness Evaluation: how is the usefulness of Tracer in practice? (Sec.
4.5)

We use our depth dataset in Sec. 2 to answer RQ6 and RQ7, and construct two
more datasets to answer RQ8. We adopt the same indicators in Sec. 2.6 to
measure accuracy, i.e., the number of CVEs an approach finds no patch, and the
precision, recall and F1-score on CVEs an approach finds patches. We conduct a
user study to answer RQ9, and measure the accuracy of patches and the time
consumed by users with/without the assistance of Tracer in finding patches.

###  4.1 Accuracy Evaluation (RQ6)

TABLE III: Accuracy of Existing Heuristic-Based Approaches
---
Cardinality |  Number | Searching NVD References | Searching GitHub Commits | Searching NVD and GitHub
|  |  Not Found |  Pre. |  Rec. |  F1 |  Not Found |  Pre. |  Rec. |  F1 |  Not Found |  Pre. |  Rec. |  F1
1:1 (SP) |  567 |  285 (50.3%) |  0.973 |  0.986 |  0.977 |  472 (83.2%) |  0.416 |  0.642 |  0.471 |  222 (39.2%) |  0.839 |  0.930 |  0.864
1:$i$ (MEP) |  195 |  125 (64.1%) |  0.932 |  0.925 |  0.921 |  162 (83.1%) |  0.472 |  0.490 |  0.452 |  104 (53.3%) |  0.821 |  0.867 |  0.820
1:$n$ (MP) |  101 |  68 (67.3%) |  0.980 |  0.552 |  0.683 |  73 (72.3%) |  0.536 |  0.445 |  0.461 |  52 (51.5%) |  0.779 |  0.605 |  0.647
1:$n$ (MB) |  372 |  244 (65.6%) |  0.979 |  0.416 |  0.546 |  246 (66.1%) |  0.445 |  0.236 |  0.284 |  171 (46.0%) |  0.704 |  0.393 |  0.465
1:$n$ (MR) |  60 |  46 (76.7%) |  1.000 |  0.708 |  0.794 |  37 (61.7%) |  0.627 |  0.345 |  0.413 |  27 (45.0%) |  0.801 |  0.539 |  0.604
Total |  1,295 |  768 (59.3%) |  0.970 |  0.805 |  0.842 |  990 (76.4%) |  0.461 |  0.417 |  0.386 |  576 (44.5%) |  0.793 |  0.732 |  0.720

To evaluate the accuracy of Tracer, we compare it with heuristic-based
approaches and industrial vulnerability databases, and further manually
analyze its false negatives and false positives.

Comparison to Heuristic-Based Approaches. We choose two widely used
heuristics: i) searching NVD references of a CVE for commits (e.g., [39, 49,
50]), and ii) searching GitHub commits for the identifier of a CVE in commits
(e.g., [70, 65]). Because the second heuristic is often used for searching
patches for a known open source software, we adapt it to search patches for a
CVE by further considering whether the owner and repository match the vendor
and product in CPE (i.e., our strategy in reference augment). Moreover, we
also investigate a third heuristic that combines these two heuristics.

Table III presents the accuracy results of the three heuristics, while Table
IV reports the accuracy results of Tracer (and its variants, which will be
discussed in Sec. 4.2). We can see that all the three heuristics find no patch
for a very large part (i.e., 59.3%, 76.4% and 44.5%) of the CVEs across all
mapping cardinalities, while Tracer finds no patch for only 12.0% of the CVEs.
Moreover, the first heuristic achieves a high precision on the CVEs for which
it finds patches, due to the high confidence of NVD references, but a low
recall on the CVEs with one-to-many mappings; the second heuristic has both a
low precision and a low recall, and the third heuristic achieves a precision
and a recall between the first and second heuristic. Tracer has a lower
precision, a higher recall, and a comparable F1-score than the first
heuristic, and a significantly higher recall on CVEs belonging to MP and MB.
We believe it is acceptable, considering the 116.3% more CVEs for which Tracer
finds patches but the first heuristic finds no patch. Besides, Tracer
significantly improves the second and third heuristic in F1-score by 116.8%
and 16.3%, respectively.

Tracer finds patches for 58.6% to 273.8% more CVEs than existing heuristic-
based approaches, while achieving either a comparable F1-score or a
significantly higher F1-score by up to 116.8% on CVEs an approach finds
patches.

TABLE IV: Contribution of Each Component in Tracer
---
Cardinality |  Number |  Tracer |  $v_{1}^{1}$: Tracer w/o NVD |  $v_{1}^{2}$: Tracer w/o Debian
|  |  Not Found |  Pre. |  Rec. |  F1 |  Not Found |  Pre. |  Rec. |  F1 |  Not Found |  Pre. |  Rec. |  F1
1:1 (SP) |  567 |  102 (18.0%) |  0.860 |  0.951 |  0.881 |  286 (50.4%) |  0.820 |  0.936 |  0.846 |  110 (19.4%) |  0.847 |  0.943 |  0.869
1:$i$ (MEP) |  195 |  6 (3.1%) |  0.886 |  0.918 |  0.888 |  79 (40.5%) |  0.882 |  0.935 |  0.886 |  8 (4.1%) |  0.880 |  0.912 |  0.882
1:$n$ (MP) |  101 |  20 (19.8%) |  0.872 |  0.741 |  0.761 |  41 (40.6%) |  0.881 |  0.728 |  0.766 |  22 (21.8%) |  0.851 |  0.716 |  0.739
1:$n$ (MB) |  372 |  23 (6.2%) |  0.861 |  0.788 |  0.795 |  84 (22.6%) |  0.876 |  0.780 |  0.800 |  28 (7.5%) |  0.838 |  0.760 |  0.771
1:$n$ (MR) |  60 |  4 (6.7%) |  0.831 |  0.620 |  0.659 |  8 (13.3%) |  0.848 |  0.551 |  0.624 |  5 (8.3%) |  0.819 |  0.613 |  0.651
Total |  1,295 |  155 (12.0%) |  0.864 |  0.864 |  0.837 |  498 (38.5%) |  0.856 |  0.839 |  0.815 |  173 (13.4%) |  0.848 |  0.849 |  0.821
Cardinality |  Number |  $v_{1}^{3}$: Tracer w/o Red Hat |  $v_{1}^{4}$: Tracer w/o GitHub |  $v_{1}^{5}$: Tracer w/o Network
|  |  Not Found |  Pre. |  Rec. |  F1 |  Not Found |  Pre. |  Rec. |  F1 |  Not Found |  Pre. |  Rec. |  F1
1:1 (SP) |  567 |  113 (19.9%) |  0.853 |  0.943 |  0.874 |  149 (26.3%) |  0.898 |  0.943 |  0.908 |  177 (31.2%) |  0.910 |  0.972 |  0.925
1:$i$ (MEP) |  195 |  7 (3.6%) |  0.883 |  0.918 |  0.886 |  19 (9.7%) |  0.887 |  0.921 |  0.892 |  78 (40.0%) |  0.956 |  0.959 |  0.941
1:$n$ (MP) |  101 |  21 (20.8%) |  0.880 |  0.736 |  0.760 |  28 (27.7%) |  0.873 |  0.690 |  0.726 |  40 (39.6%) |  0.943 |  0.669 |  0.743
1:$n$ (MB) |  372 |  35 (9.4%) |  0.844 |  0.761 |  0.767 |  39 (10.5%) |  0.874 |  0.752 |  0.773 |  109 (29.3%) |  0.908 |  0.575 |  0.659
1:$n$ (MR) |  60 |  4 (6.7%) |  0.738 |  0.640 |  0.618 |  7 (11.7%) |  0.816 |  0.545 |  0.604 |  10 (16.7%) |  0.920 |  0.641 |  0.712
Total |  1,295 |  180 (13.9%) |  0.851 |  0.853 |  0.823 |  242 (18.7%) |  0.883 |  0.841 |  0.835 |  414 (32.0%) |  0.918 |  0.812 |  0.823
Cardinality |  Number |  $v_{2}^{1}$: Tracer w/o Selection |  $v_{2}^{2}$: Tracer w/o Connectivity |  $v_{2}^{3}$: Tracer w/o Confidence
|  |  Not Found |  Pre. |  Rec. |  F1 |  Not Found |  Pre. |  Rec. |  F1 |  Not Found |  Pre. |  Rec. |  F1
1:1 (SP) |  567 |  102 (18.0%) |  0.632 |  0.961 |  0.680 |  245 (43.2%) |  0.892 |  0.978 |  0.913 |  102 (18.0%) |  0.860 |  0.942 |  0.879
1:$i$ (MEP) |  195 |  6 (3.1%) |  0.622 |  0.976 |  0.682 |  111 (56.9%) |  0.929 |  0.939 |  0.915 |  6 (3.1%) |  0.888 |  0.913 |  0.889
1:$n$ (MP) |  101 |  20 (19.8%) |  0.615 |  0.933 |  0.656 |  56 (55.4%) |  0.953 |  0.685 |  0.764 |  20 (19.8%) |  0.880 |  0.722 |  0.751
1:$n$ (MB) |  372 |  23 (6.2%) |  0.616 |  0.903 |  0.657 |  191 (51.3%) |  0.927 |  0.787 |  0.821 |  23 (6.2%) |  0.871 |  0.765 |  0.784
1:$n$ (MR) |  60 |  4 (6.7%) |  0.368 |  0.891 |  0.394 |  27 (45.0%) |  0.885 |  0.722 |  0.772 |  4 (6.7%) |  0.849 |  0.462 |  0.550
Total |  1,295 |  155 (12.0%) |  0.611 |  0.940 |  0.658 |  630 (48.6%) |  0.910 |  0.889 |  0.871 |  155 (12.0%) |  0.869 |  0.844 |  0.826
Cardinality |  Number |  $v_{2}^{4}$: Tracer with Path Length |  $v_{2}^{5}$: Tracer with Path Number |  $v_{3}$: Tracer w/o Expansion
|  |  Not Found |  Pre. |  Rec. |  F1 |  Not Found |  Pre. |  Rec. |  F1 |  Not Found |  Pre. |  Rec. |  F1
1:1 (SP) |  567 |  102 (18.0%) |  0.833 |  0.957 |  0.859 |  102 (18.0%) |  0.805 |  0.951 |  0.837 |  102 (18.0%) |  0.871 |  0.948 |  0.889
1:$i$ (MEP) |  195 |  6 (3.1%) |  0.848 |  0.945 |  0.867 |  6 (3.1%) |  0.849 |  0.920 |  0.858 |  6 (3.1%) |  0.910 |  0.914 |  0.902
1:$n$ (MP) |  101 |  20 (19.8%) |  0.849 |  0.760 |  0.742 |  20 (19.8%) |  0.801 |  0.756 |  0.726 |  20 (19.8%) |  0.873 |  0.696 |  0.732
1:$n$ (MB) |  372 |  23 (6.2%) |  0.830 |  0.798 |  0.770 |  23 (6.2%) |  0.833 |  0.811 |  0.791 |  23 (6.2%) |  0.860 |  0.506 |  0.590
1:$n$ (MR) |  60 |  4 (6.7%) |  0.652 |  0.747 |  0.590 |  4 (6.7%) |  0.789 |  0.630 |  0.644 |  4 (6.7%) |  0.847 |  0.567 |  0.629
Total |  1,295 |  155 (12.0%) |  0.827 |  0.882 |  0.812 |  155 (12.0%) |  0.819 |  0.873 |  0.809 |  155 (12.0%) |  0.873 |  0.771 |  0.776

Comparison to Vulnerability Databases. The goal of our comparison to
industrial vulnerability databases is not to demonstrate the superiority or
inferiority of Tracer over vulnerability databases, as we are not aware of how
many manual efforts or what automated approaches are involved in the
construction of vulnerability databases although some of them have claimed
that they collect patches manually or semi-automatically. Instead, our goal is
to assess the level of accuracy Tracer can achieve and the worthiness of
Tracer, and to explore whether Tracer can potentially improve or complement
existing industrial vulnerability databases.

It can be observed from Table II and IV that Tracer finds patches for 12.0%
fewer CVEs than $DB_{A}$ and $DB_{B}$, and achieves a 6.4% and 5.8% lower
precision than $DB_{A}$ and $DB_{B}$. This might be potentially due to the
manual efforts involved in vulnerability database construction and the
heuristic nature of Tracer. On the other hand, Tracer has a 15.5% and 18.4%
higher recall than $DB_{A}$ and $DB_{B}$ (especially for CVEs belonging to
one-to-many mappings), resulting in a 5.5% and 8.6% higher F1-score than
$DB_{A}$ and $DB_{B}$. These results show that Tracer is worthwhile with a
significantly higher recall at the price of a moderately lower precision and a
smaller number of CVEs for which patches are not found. Tracer might have the
merit to complement existing vulnerability databases by reducing manual
efforts and automatically finding patches more completely.

Tracer achieves a 15.5% to 18.4% higher recall and a 5.5% to 8.6% higher
F1-score than the two state-of-the-art vulnerability databases, while
sacrificing up to 12.0% fewer CVEs whose patches are not found and up to 6.4%
lower precision.

False Negative Analysis. We manually analyze the CVEs for which Tracer finds
no patch or misses some of the patches, and summarize five main reasons.
First, for some old CVEs, the references contained in NVD, Debian and Red Hat
are limited, and some of them even become invalid. As a result, Tracer fails
to construct a complete reference network. Second, some key references (e.g.,
issue reports) about a CVE are missing from NVD, Debian and Red Hat. As a
result, Tracer fails to be directed to the correct patch. For example, for
CVE-2018-14642, its issue report [7] is not contained in any of the three
advisory sources. However, following this issue report, we could find the
patch [8]. Third, the commit message of a patch has semantic similarity to the
CVE description, but does not contain the CVE identifier. Hence, the reference
augment in Tracer fails to catch it. For example, for CVE-2019-10077 [9], the
patch commit [10] fixes it without indicating the CVE identifier. Fourth,
GitHub’s REST API for commit search only returns 1,000 results, which might
miss the correct patch commit in reference augment in Tracer. Fifth, only one
patch with the highest connectivity is selected in patch selection in Tracer.
As a result, the correct patches for CVEs belonging to one-to-many mappings
might be missed even though they are already included in our reference
network.

False Positive Analysis. We also manually analyze the CVEs for which Tracer
finds incorrect patches, and summarize two main reasons. First, the commit
that introduces the CVE is referenced during the discussion and resolution of
the CVE. As a result, Tracer falsely identifies it as a patch commit due to
the lack of semantic understanding of the context where the commit is
referenced. For example, for CVE-2020-5249, the commit that introduces the CVE
[11] and the commit that fixes the CVE [12] are referenced in the same comment
of the issue report. Second, multiple CVEs and their issues and patches are
listed on the same page. As a result, patches for other CVEs might be falsely
identified by Tracer due to the lack of semantic understanding. For example,
for CVE-2018-15750, its patches are maintained in the release note [13] with
CVE-2018-15751, and the release note is all referenced by NVD, Debian, and Red
Hat.

The five and two reasons for false negatives and false positives are
respectively summarized by manual analysis, which can be leveraged to further
improve the accuracy of Tracer.

###  4.2 Ablation Analysis (RQ7)

Table IV presents the results of our ablation study to measure the
contribution of various settings in Tracer to its achieved accuracy.

Removing an Advisory Source. We remove one of the four advisory sources NVD,
Debian, Red Hat, and GitHub from the first step of Tracer, and generate
variants $v_{1}^{1}$, $v_{1}^{2}$, $v_{1}^{3}$ and $v_{1}^{4}$ of Tracer. We
can observe from Table IV that these four variants all suffer an increase in
the number of CVEs they find no patch. Significantly, $v_{1}^{1}$,
$v_{1}^{2}$, $v_{1}^{3}$ and $v_{1}^{4}$ respectively find patches for 30.1%,
1.6%, 2.2% and 7.6% fewer CVEs than Tracer, while achieving comparable
precision, recall and F1-score on the CVEs they find patches. These results
indicate that all the four advisory sources contribute to finding patches for
more CVEs by constructing a more complete reference network, while NVD and
GitHub contribute the most.

Removing Reference Network. We do not construct the reference network in a
layered way but simply use the direct references contained in the four
advisory sources (i.e., we skip reference analysis in the first step of
Tracer), which is the variant $v_{1}^{5}$ in Table IV. We can see that
$v_{1}^{5}$ also suffers an increase in the number of CVEs it finds no patch.
Numerically, $v_{1}^{5}$ finds patches for 22.7% fewer CVEs than Tracer, while
having a 6.3% higher precision, a 6.0% lower recall, and a comparable F1-score
on the CVEs it finds patches. These results demonstrate that patches are not
always directly referenced in NVD, Debian, and Red Hat, but might be hidden in
indirect references, and our reference network contributes to finding such
hidden patches at the price of an acceptable decrease in precision.

Removing Patch Selection. We do not select some patches in the second step of
Tracer but select all the patches in our reference network, which is variant
$v_{2}^{1}$ in Table IV. $v_{2}^{1}$ significantly improves Tracer in recall
by 8.8%, especially for CVEs belonging to one-to-many mappings, while
suffering a large degradation in precision by 29.3% and in F1-score by 21.4%
across all cardinalities. These results indicate that our reference network
indeed contains most of the correct patches, and our patch selection
heuristics contribute to achieving a balance between precision and recall.

Removing Connectivity or Confidence. We remove one of the two heuristics
adopted in the second step of Tracer, and generate two variants $v_{2}^{2}$
and $v_{2}^{3}$. Without our connectivity-based heuristic, $v_{2}^{2}$ finds
patches for 41.7% fewer CVEs than Tracer, while achieving a 5.3% higher
precision, a 2.9% higher recall and a 4.1% higher F1-score. These results
indicate that our connectivity-based heuristic contributes to finding patches
for more CVEs while introducing acceptable noise. Without our confidence-based
heuristic, $v_{2}^{3}$ suffers a slight decrease in recall and F1-score,
especially for CVEs belonging to MR. These results show that our confidence-
based heuristic contributes to achieving a balanced accuracy across all
cardinalities.

Reducing Connectivity. We reduce our connectivity-based heuristic by only
considering path length (i.e., selecting the patch with the shortest path to
the root node) and by only considering path number (i.e., selecting the patch
with the largest number of paths to the root node), and respectively generate
the variant $v_{2}^{4}$ and $v_{2}^{5}$. Both $v_{2}^{4}$ and $v_{2}^{5}$
suffer a 4.3% and 5.2% decrease in precision, a 2.1% and 1.0% increase in
recall, and a 3.0% and 3.3% decrease in F1-score. These results demonstrate
that our connectivity-based heuristic contributes to improving the precision
of Tracer by comprehensively considering the path length and path number of a
patch to the root node.

Removing Patch Expansion. We do not expand patches in the third step of
Tracer, which is variant $v_{3}$ in Table IV. $v_{3}$ suffers degradation in
recall and F1-score by 10.8% and 7.3%, especially for CVEs with one-to-many
mappings. These results show that our patch expansion contributes to finding
multiple patches more completely.

Our used advisory sources, reference network, patch selection, and patch
expansion all contribute positively to the achieved accuracy of Tracer in
finding patches.

###  4.3 Sensitivity Analysis (RQ8)

(a) Network Depth Limit

(b) Commit Span

Figure 5: Sensitivity Analysis Result for Network Depth Limit and Commit Span

Tracer has two configurable parameters, i.e., the network depth limit in the
first step of Tracer and the commit span in the third step. The default
configuration is 5 and 30, which is used in the evaluation for RQ6, RQ7, RQ9
and RQ10. To evaluate the sensitivity of Tracer to the two parameters, we
reconfigured one parameter and fix the other, and reran Tracer against our
depth dataset. Specifically, the network depth limit was configured from 3 to
6 by a step of 1, and the commit span was configured from 0 to 60 by a step of
10.

Fig. 5(a) and 5(b) show the impact of the two parameters on the accuracy of
Tracer, where $x$-axis denotes the value of the parameter, and $y$-axis
denotes the accuracy of Tracer. Overall, as the network depth limit increases,
more potential patches are included in our reference network. The number of
CVEs that Tracer finds no patch and precision decrease, and recall and
F1-score first increase and then decrease. Hence, we believe 5 is a good value
for the network depth limit. As the commit span increases, a wider scope of
commits are searched. Precision decreases, recall increases, and F1-score
first increases and then decreases. Notice that the number of CVEs Tracer
finds no patch will not change and thus is not presented in Fig. 5(b). Hence,
we believe 30 is a good value for the commit span.

Overall, the sensitivity of the accuracy of Tracer to the two configurable
parameters is acceptable.

###  4.4 Generality Evaluation (RQ9)

TABLE V: Generality of Tracer over Two Additional Datasets
---
Dataset |  Number | Tracer | $DB_{A}$ | $DB_{B}$
|  Pre. |  Rec. |  F1 |  Not Found |  Pre. |  Rec. |  F1 |  Not Found |  Pre. |  Rec. |  F1
First Dataset |  91 |  0.823 |  0.845 |  0.784 |  29 (31.8%) |  0.935 |  0.827 |  0.858 |  62 (68.1%) |  0.885 |  0.664 |  0.725
Second Dataset |  89 |  0.888 |  0.899 |  0.867 |  – |  – |  – |  – |  – |  – |  – |  –

To further evaluate the generality of Tracer, we construct two additional
vulnerability datasets. The first dataset includes the CVEs for which only one
of the two vulnerability databases reports their patches (Fig. 2(b)), which
has a total of 3,185 CVEs. The second dataset includes the CVEs for which none
of the two vulnerability databases reports their patches (Fig. 2), which has a
total of 5,468 CVEs. It turns out that Tracer finds patches for 2,155 (67.7%)
of the 3,185 CVEs in the first dataset, while the two vulnerability databases
$DB_{A}$ and $DB_{B}$ report patches for 2,190 (68.8%) and 995 (31.2%) CVEs.
Of the 2,155 CVEs Tracer finds patches, $DB_{A}$ and $DB_{B}$ report patches
for 1,455 and 700 of them. In addition, Tracer finds patches for 2,816 (51.5%)
of the 5,468 CVEs in the second dataset, where $DB_{A}$ and $DB_{B}$ report no
patch. These results show that Tracer can greatly complement or enhance
existing vulnerability databases via finding patches for CVEs these
vulnerability databases report no patch.

Further, we respectively sample 100 CVEs Tracer find patches from the first
and second datasets, and manually find their patches in the same procedure to
Sec. 2.1 to measure the accuracy of Tracer. The results are reported in Table
V. Our manual analysis has 9 and 11 uncertain CVEs due to limited disclosed
knowledge. Of the 91 CVEs in the first dataset, Tracer has an F1-score of
0.784, while $DB_{A}$ has a higher F1-score and $DB_{B}$ achieves a lower
F1-score. Similar to the results in Sec. 4.1, vulnerability databases achieve
a higher precision but a lower recall. Of the 89 CVEs in the second dataset,
$DB_{A}$ and $DB_{B}$ report no patch, whereas Tracer achieves an F1-score of
0.867. These results indicate that the capability of Tracer can be generalized
to a wider range of OSS vulnerabilities.

Tracer finds patches for 67.7% and 51.5% of the CVEs in the two additional
datasets with a sampled precision of 0.823 and 0.888 and a sampled recall of
0.845 and 0.899, which demonstrates the generality of Tracer in finding
patches.

###  4.5 Usefulness Evaluation (RQ10)

In practice, to ensure patch accuracy, security experts still need to verify
patches even when automatic tools are used to find patches. To evaluate the
usefulness of Tracer in such a usage scenario, we conduct a user study with 10
participants who are required to find patches for 10 CVEs with and without the
help of Tracer. We recruit 10 participants from security laboratories in
multiple universities and high-tech companies. They are Postdocs, PhD
students, master researchers, and engineers majoring in software security. We
randomly select 10 CVEs from our depth dataset as tasks. 2 CVEs belong to SP,
3 CVEs belong to MEP, 1 CVE belongs to MP, and 4 CVEs belong to MB. To have a
fair comparison, we divide participants into two groups (i.e., A and B). Group
A is required to complete the first five tasks without Tracer and finish the
remaining tasks with Tracer. Conversely, Group B is required to complete the
first five tasks with Tracer, and finish the remaining tasks without Tracer.

TABLE VI: Comparison Results of the Time and Accuracy of 10 Tasks
---
Approach | All 10 Tasks | 5 Single-Patch Tasks | 5 Multiple-Patches Tasks
Time (mins) |  Pre. |  Rec. |  F1 |  Time (mins) |  Pre. |  Rec. |  F1 |  Time (mins) |  Pre. |  Rec. |  F1
w/o Tracer |  5.66 |  0.880 |  0.677 |  0.765 |  5.60 |  0.960 |  0.960 |  0.960 |  5.72 |  0.800 |  0.393 |  0.527
with Tracer |  4.66 |  0.983 |  0.920 |  0.951 |  3.84 |  1.000 |  1.000 |  1.000 |  5.48 |  0.967 |  0.840 |  0.899

Table VI reports the average time consumption and patch accuracy of the 10
tasks. We categorize the 5 CVEs belonging to SP and MEP as Single-Patch tasks,
and categorize the 5 CVEs belonging to MP and MB as Multiple-Patches tasks.
Overall, with the assistance of Tracer, the participants save time by 17.7%
for each task, and improve the patch accuracy in terms of precision, recall
and F1-score by 11.7%, 35.9% and 24.3%. In particular, the time saving is
significant for the 5 Single-Patch tasks, but not significant for the 5
Multiple-Patches tasks. This is because the reference network and patches
returned by Tracer for Multiple-Patches tasks are more complex than those of
Single-Patch tasks, and participants spend more time understanding the
reference network and verify patches. Moreover, the accuracy improvement is
significant for the 5 Multiple-Patches tasks, but not significant for the 5
Single-Patch tasks. In that sense, Tracer is especially useful for CVEs with
multiple patches.

We interview each participant to get their feedback about Tracer. Overall,
they all appreciate the value of our reference network as it summarizes
information from multiple sources, and the different kinds of nodes and
relationships in it are helpful to localize and verify patches. As commented
by a security engineer from a high-tech company, “the network graph, as a
chain of evidence, is helpful to localize and verify patches”. Moreover, they
suggest including more information for the nodes (e.g., commit message and
code differences) instead of a clickable link for the ease of manual review,
and also suggest adding more advisory sources.

Tracer is useful in practice for security experts to localize patches more
accurately and quickly.

###  4.6 Discussion

We discuss the limitations and significance of Tracer.

Limitations. First, Tracer takes a CVE identifier as an input, and thus may
not work for open source vulnerabilities without CVE identifiers because some
vulnerabilities are secretly patched [68]. One potential remedy is to take
advisory identifiers as Tracer’s input as third-party advisories (e.g.,
WhiteSource) can have a broad scope of vulnerabilities than NVD. Second,
Tracer only considers Debian and Red Hat as the secondary advisory sources.
However, it is designed to easily incorporate other sources (e.g.,
SecurityFocus [24]) thanks to our lightweight reference analysis. Third, as
indicated by our false negative and false positive analysis, the lack of
semantic analysis of patches and CVEs hurts the accuracy of Tracer. We plan to
use semantics in CVEs (e.g., descriptions) and patches (e.g, changed code and
the context of patch references) to enhance patch selection and expansion.

Significance. Tracer can benefit security community, academia, and industry by
enabling automated patch finding from various sources. For security community,
Tracer can notify NVD about missing or incomplete patches for CVEs to enhance
CVE information quality and accelerate entry update, and hence benefit the
audience of NVD. For academia, Tracer can enable data-driven security analysis
(e.g., learning-based vulnerability detection [50, 73]) and empirical studies
by providing large-scale patches. It can also help to determine the affected
library of a CVE [38] by directly analyzing its patches. For industry, Tracer
can assist security engineers in enhancing the patch coverage and accuracy of
commercial vulnerability databases, and thus improve the accuracy of software
composition analysis. It is worth mentioning that Tracer has been deployed to
a high-tech company. However, we cannot disclose its usage statistics due to
confidentiality agreement.

##  5 Related Work

We review and discuss the most closely related work in three aspects, i.e.,
CVE information quality, patch analysis, and patch applications.

CVE Information Quality. As vulnerability databases (e.g., NVD) are
accumulating a massive collection of vulnerabilities, there arises a growing
concern about the information quality of CVE entries. Nguyen and Massacci [57]
uncover the unreliability of the vulnerable version data in NVD. To improve
its reliability, Nguyen et al. [56] and Dashevskyi et al. [37] develop tools
to determine whether older versions are affected by a newly disclosed
vulnerability. Basically, if an older version contains the lines of source
code that were changed to fix a vulnerability, this version is considered as
affected by the vulnerability. Dong et al. [38] identify vulnerable software
names and versions from vulnerability reports, and find that vulnerability
data-bases miss truly vulnerable versions or falsely include non-vulnerable
versions. Chen et al. [33] identify open-source libraries affected by a
vulnerability. Chaparro et al. [32] detect the absence of reproduction steps
and expected behavior in vulnerability descriptions. Mu et al. [53] show the
prevalence of missing reproduction information in vulnerability reports. Jo et
al. [46] identify semantic inconsistencies within the cybersecurity domain.
These works are focused on different aspects of vulnerability information.
Following this direction, our work is focused on the patch of a vulnerability,
and tries to identify patches for a vulnerability from combined sources of
vulnerability reports.

A closely related work is from Tan et al. [64]. They use a learning-to-rank
algorithm to rank commits in a repository so that patch commits to a CVE are
ranked in top positions. However, they make two assumptions: i) the repository
of the affected software of a CVE is known, which is not practical and
requires manual efforts; and ii) a CVE has a one-to-one mapping to its
patches, which does not always hold (Sec. 2.5). Instead, Tracer has no such
assumptions.

Patch Analysis. There are various patch analysis tasks to improve security,
e.g., patch generation and deployment [54, 39, 69], patch presence testing
[72, 43, 36], and secret patch identification [68, 74, 62, 34]. Data sets of
security patches have been built for Java [61], C/C++ [40], and specific open-
source projects [44]. Based on such data sets, empirical studies have been
conducted to characterize vulnerabilities and their patches [71, 48, 51, 29].
In these works, patches are mostly identified by manual efforts [69, 43, 36,
68, 74, 62, 34, 61, 71] or by heuristic rules such as looking for commits in
CVE references [39, 40, 44, 48, 51] and searching for CVE identifiers in
commits [40, 44, 29]. However, such heuristics are not strong enough to find
patches more completely.

Patch Applications. Patches can be leveraged to enable various security
applications, e.g., generating exploits based on patches [30, 70], conducting
software composition analysis to determine whether vulnerabilities in a
library are reachable through which call paths [58, 60, 59, 65], and detecting
vulnerabilities by learning vulnerability features [49, 50, 73, 45], by
matching vulnerability signatures [42, 47] and by matching both vulnerability
and patch signatures [67, 66, 35]. Similar to those patch analysis works, the
mappings between CVEs and their patches in these works are mostly identified
by manual efforts [58, 60, 59, 66] and heuristics rules [49, 50, 70, 65, 45],
or directly taken from security advisories that establish the mapping between
CVEs and patches for specific projects [42, 47, 67]. However, such heuristics
are not strong enough to find patches more completely, and it is also
difficult to find all such security advisories.

##  6 Conclusions

In this paper, we have conducted an empirical study to understand the quality
and characteristics of patches for OSS vulnerabilities in two industrial
vulnerability databases in terms of five dimensions, i.e., patch coverage,
patch consistency, patch type, mapping cardinality between vulnerabilities and
their patches, and patch accuracy. Our study findings have inspired us to
propose the first automated approach, named Tracer, to find patches for OSS
vulnerabilities from multiple advisory sources. Our extensive evaluation has
demonstrated the accuracy, generality and usefulness of Tracer. We have
released the source code of Tracer and all the experimental data at
https://patch-tracer.github.io.

## References

  * [1] [Online]. Available: https://github.com/onelogin/ruby-saml/commit/048a544730930f86e46804387a6b6fad50d8176f
  * [2] [Online]. Available: https://www.jenkins.io/security/advisory/2019-09-25/\\#SECURITY-1573
  * [3] [Online]. Available: https://issues.apache.org/jira/browse/THRIFT-4647
  * [4] [Online]. Available: https://github.com/onelogin/ruby-saml/commit/d7ce607d9f9d996e1046dde09b675c3cf0c01280
  * [5] [Online]. Available: https://github.com/onelogin/ruby-saml/commit/a35f7251b86aa3b7caf4a64d8a3451f925e8855c
  * [6] [Online]. Available: https://github.com/onelogin/ruby-saml/commit/03af9e33d2d87f4ac9a644c5b0981ded4dca0bb8
  * [7] [Online]. Available: https://issues.redhat.com/browse/UNDERTOW-1430
  * [8] [Online]. Available: https://github.com/undertow-io/undertow/commit/c46b7b49c5a561731c84a76ee52244369af1af8a
  * [9] [Online]. Available: https://nvd.nist.gov/vuln/detail/CVE-2019-10077
  * [10] [Online]. Available: https://github.com/apache/jspwiki/commit/87c89f0405d6b31fc165358ce5d5bc4536e32a8a
  * [11] [Online]. Available: https://github.com/ethereum/go-ethereum/commit/fb9f7261ec51e38eedb454594fc19f00de1a6834
  * [12] [Online]. Available: https://github.com/ethereum/go-ethereum/commit/83e2761c3a13524bd5d6597ac08994488cf872ef
  * [13] [Online]. Available: https://docs.saltstack.com/en/latest/topics/releases/2018.3.3.html
  * [14] Black duck. [Online]. Available: https://www.synopsys.com/content/dam/synopsys/sig-assets/datasheets/bdknowledgebase-ds-ul.pdf
  * [15] Cve list. [Online]. Available: https://cve.mitre.org/cve/
  * [16] Debian repository for advisories. [Online]. Available: https://salsa.debian.org/security-tracker-team/security-tracker/-/tree/master/data/CVE
  * [17] Debian security tracker. [Online]. Available: https://security-tracker.debian.org/tracker/
  * [18] Github rest api. [Online]. Available: https://docs.github.com/en/rest/reference/search\\#search-commits
  * [19] Github rest api. [Online]. Available: https://docs.github.com/en/rest/reference/repos\\#list-branches
  * [20] Github rest api. [Online]. Available: https://docs.github.com/en/rest/reference/repos\\#list-commits
  * [21] National vulnerability database. [Online]. Available: https://nvd.nist.gov
  * [22] Nvd data feeds. [Online]. Available: https://nvd.nist.gov/vuln/data-feeds
  * [23] Red hat bugzilla. [Online]. Available: https://bugzilla.redhat.com/
  * [24] Securityfocus. [Online]. Available: https://www.securityfocus.com
  * [25] Snyk. [Online]. Available: https://snyk.io/vuln
  * [26] Veracode. [Online]. Available: https://sca.veracode.com/vulnerability-database/search
  * [27] Webservice api of red hat. [Online]. Available: https://bugzilla.redhat.com/docs/en/html/api/index.html
  * [28] Whitesource. [Online]. Available: https://www.whitesourcesoftware.com/vulnerability-database/
  * [29] G. Antal, M. Keleti, and P. Hegedŭs, “Exploring the security awareness of the python and javascript open source communities,” in _Proceedings of the 17th International Conference on Mining Software Repositories_ , 2020, pp. 16–20.
  * [30] D. Brumley, P. Poosankam, D. Song, and J. Zheng, “Automatic patch-based exploit generation is possible: Techniques and implications,” in _Proceedings of the IEEE Symposium on Security and Privacy_ , 2008, pp. 143–157.
  * [31] S. C. R. Center. (2021) Open source security and risk analysis report. [Online]. Available: https://www.synopsys.com/content/dam/synopsys/sig-assets/reports/rep-ossra-2021.pdf
  * [32] O. Chaparro, J. Lu, F. Zampetti, L. Moreno, M. Di Penta, A. Marcus, G. Bavota, and V. Ng, “Detecting missing information in bug descriptions,” in _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering_ , 2017, pp. 396–407.
  * [33] Y. Chen, A. E. Santosa, A. Sharma, and D. Lo, “Automated identification of libraries from vulnerability data,” in _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Practice_ , 2020, pp. 90–99.
  * [34] Y. Chen, A. E. Santosa, A. M. Yi, A. Sharma, A. Sharma, and D. Lo, “A machine learning approach for vulnerability curation,” in _Proceedings of the 17th International Conference on Mining Software Repositories_ , 2020, pp. 32–42.
  * [35] L. Cui, Z. Hao, Y. Jiao, H. Fei, and X. Yun, “Vuldetector: Detecting vulnerabilities using weighted feature graph comparison,” _IEEE Transactions on Information Forensics and Security_ , vol. 16, pp. 2004–2017, 2021\.
  * [36] J. Dai, Y. Zhang, Z. Jiang, Y. Zhou, J. Chen, X. Xing, X. Zhang, X. Tan, M. Yang, and Z. Yang, “Bscout: Direct whole patch presence test for java executables,” in _Proceedings of the 29th USENIX Security Symposium_ , 2020, pp. 1147–1164.
  * [37] S. Dashevskyi, A. D. Brucker, and F. Massacci, “A screening test for disclosed vulnerabilities in foss components,” _IEEE Transactions on Software Engineering_ , vol. 45, no. 10, pp. 945–966, 2019.
  * [38] Y. Dong, W. Guo, Y. Chen, X. Xing, Y. Zhang, and G. Wang, “Towards the detection of inconsistencies in public security vulnerability reports,” in _Proceedings of the 28th USENIX Security Symposium_ , 2019, pp. 869–885.
  * [39] R. Duan, A. Bijlani, Y. Ji, O. Alrawi, Y. Xiong, M. Ike, B. Saltaformaggio, and W. Lee, “Automating patching of vulnerable open-source software versions in application binaries.” in _Proceedings of the 26th Annual Network and Distributed System Security Symposium_ , 2019.
  * [40] J. Fan, Y. Li, S. Wang, and T. N. Nguyen, “Ac/c++ code vulnerability dataset with code changes and cve summaries,” in _Proceedings of the 17th International Conference on Mining Software Repositories_ , 2020, pp. 508–512.
  * [41] A. Gkortzis, D. Mitropoulos, and D. Spinellis, “Vulinoss: a dataset of security vulnerabilities in open-source systems,” in _Proceedings of the 15th International Conference on Mining Software Repositories_ , 2018, pp. 18–21.
  * [42] J. Jang, A. Agrawal, and D. Brumley, “Redebug: finding unpatched code clones in entire os distributions,” in _Proceedings of the IEEE Symposium on Security and Privacy_ , 2012, pp. 48–62.
  * [43] Z. Jiang, Y. Zhang, J. Xu, Q. Wen, Z. Wang, X. Zhang, X. Xing, M. Yang, and Z. Yang, “Pdiff: Semantic-based patch presence testing for downstream kernels,” in _Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security_ , 2020, pp. 1149–1163.
  * [44] M. Jimenez, Y. Le Traon, and M. Papadakis, “Enabling the continous analysis of security vulnerabilities with vuldata7,” in _Proceedings of the IEEE International Working Conference on Source Code Analysis and Manipulation_ , 2018, pp. 56–61.
  * [45] M. Jimenez, R. Rwemalika, M. Papadakis, F. Sarro, Y. Le Traon, and M. Harman, “The importance of accounting for real-world labelling when predicting software vulnerabilities,” in _Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2019, pp. 695–705.
  * [46] H. Jo, J. Kim, P. Porras, V. Yegneswaran, and S. Shin, “Gapfinder: Finding inconsistency of security information from unstructured text,” _IEEE Transactions on Information Forensics and Security_ , vol. 16, pp. 86–99, 2021\.
  * [47] S. Kim, S. Woo, H. Lee, and H. Oh, “Vuddy: A scalable approach for vulnerable code clone discovery,” in _Proceedings of the IEEE Symposium on Security and Privacy_ , 2017, pp. 595–614.
  * [48] F. Li and V. Paxson, “A large-scale empirical study of security patches,” in _Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security_ , 2017, pp. 2201–2215.
  * [49] Z. Li, D. Zou, S. Xu, H. Jin, H. Qi, and J. Hu, “Vulpecker: an automated vulnerability detection system based on code similarity analysis,” in _Proceedings of the 32nd Annual Conference on Computer Security Applications_ , 2016, pp. 201–213.
  * [50] Z. Li, D. Zou, S. Xu, X. Ou, H. Jin, S. Wang, Z. Deng, and Y. Zhong, “Vuldeepecker: A deep learning-based system for vulnerability detection,” in _Proceedings of the 25th Annual Network and Distributed System Security Symposium_ , 2018.
  * [51] B. Liu, G. Meng, W. Zou, Q. Gong, F. Li, M. Lin, D. Sun, W. Huo, and C. Zhang, “A large-scale empirical study on vulnerability distribution within projects and the lessons learned,” in _Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering_ , 2020, pp. 1547–1559.
  * [52] A. Machiry, N. Redini, E. Camellini, C. Kruegel, and G. Vigna, “Spider: Enabling fast patch propagation in related software repositories,” in _Proceedings of the IEEE Symposium on Security and Privacy_ , 2020, pp. 1562–1579.
  * [53] D. Mu, A. Cuevas, L. Yang, H. Hu, X. Xing, B. Mao, and G. Wang, “Understanding the reproducibility of crowd-reported security vulnerabilities,” in _Proceedings of the 27th USENIX Security Symposium_ , 2018, pp. 919–936.
  * [54] C. Mulliner, J. Oberheide, W. Robertson, and E. Kirda, “Patchdroid: Scalable third-party security patches for android devices,” in _Proceedings of the 29th Annual Computer Security Applications Conference_ , 2013, pp. 259–268.
  * [55] Z. Namrud, S. Kpodjedo, and C. Talhi, “Androvul: a repository for android security vulnerabilities,” in _Proceedings of the 29th Annual International Conference on Computer Science and Software Engineering_ , 2019, pp. 64–71.
  * [56] V. H. Nguyen, S. Dashevskyi, and F. Massacci, “An automatic method for assessing the versions affected by a vulnerability,” _Empirical Software Engineering_ , vol. 21, no. 6, pp. 2268–2297, 2016.
  * [57] V. H. Nguyen and F. Massacci, “The (un) reliability of nvd vulnerable versions data: An empirical experiment on google chrome vulnerabilities,” in _Proceedings of the 8th ACM SIGSAC Symposium on Information, Computer and Communications Security_ , 2013, pp. 493–498.
  * [58] I. Pashchenko, H. Plate, S. E. Ponta, A. Sabetta, and F. Massacci, “Vulnerable open source dependencies: Counting those that matter,” in _Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement_ , 2018, pp. 1–10.
  * [59] ——, “Vuln4real: A methodology for counting actually vulnerable dependencies,” _IEEE Transactions on Software Engineering_ , 2020.
  * [60] S. E. Ponta, H. Plate, and A. Sabetta, “Detection, assessment and mitigation of vulnerabilities in open source dependencies,” _Empirical Software Engineering_ , vol. 25, no. 5, pp. 3175–3215, 2020.
  * [61] S. E. Ponta, H. Plate, A. Sabetta, M. Bezzi, and C. Dangremont, “A manually-curated dataset of fixes to vulnerabilities of open-source software,” in _Proceedings of the IEEE/ACM 16th International Conference on Mining Software Repositories_ , 2019, pp. 383–387.
  * [62] A. Sabetta and M. Bezzi, “A practical approach to the automatic classification of security-relevant commits,” in _Proceedings of the IEEE International Conference on Software Maintenance and Evolution_ , 2018, pp. 579–582.
  * [63] Snyk. (2019) State of the open source security report. [Online]. Available: https://snyk.io/wp-content/uploads/sooss_report_v2.pdf
  * [64] X. Tan, Y. Zhang, C. Mi, J. Cao, K. Sun, Y. Lin, and M. Yang, “Locating the security patches for disclosed oss vulnerabilities with vulnerability-commit correlation ranking,” in _Proceedings of the 28th ACM Conference on Computer and Communications Security_ , 2021.
  * [65] Y. Wang, B. Chen, K. Huang, B. Shi, C. Xu, X. Peng, Y. Wu, and Y. Liu, “An empirical study of usages, updates and risks of third-party libraries in java projects,” in _ICSME_ , 2020, pp. 35–45.
  * [66] Y. Xiao, B. Chen, C. Yu, Z. Xu, Z. Yuan, F. Li, B. Liu, Y. Liu, W. Huo, W. Zou _et al._ , “$\\{$MVP$\\}$: Detecting vulnerabilities using patch-enhanced vulnerability signatures,” in _Proceedings of the 29th USENIX Security Symposium_ , 2020, pp. 1165–1182.
  * [67] Y. Xu, Z. Xu, B. Chen, F. Song, Y. Liu, and T. Liu, “Patch based vulnerability matching for binary programs,” in _Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis_ , 2020, pp. 376–387.
  * [68] Z. Xu, B. Chen, M. Chandramohan, Y. Liu, and F. Song, “Spain: security patch analysis for binaries towards understanding the pain and pills,” in _Proceedings of the IEEE/ACM 39th International Conference on Software Engineering_ , 2017, pp. 462–472.
  * [69] Z. Xu, Y. Zhang, L. Zheng, L. Xia, C. Bao, Z. Wang, and Y. Liu, “Automatic hot patch generation for android kernels,” in _Proceedings of the 29th USENIX Security Symposium_ , 2020, pp. 2397–2414.
  * [70] W. You, P. Zong, K. Chen, X. Wang, X. Liao, P. Bian, and B. Liang, “Semfuzz: Semantics-based automatic generation of proof-of-concept exploits,” in _Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security_ , 2017, pp. 2139–2154.
  * [71] S. Zaman, B. Adams, and A. E. Hassan, “Security versus performance bugs: a case study on firefox,” in _Proceedings of the 8th working conference on mining software repositories_ , 2011, pp. 93–102.
  * [72] H. Zhang and Z. Qian, “Precise and accurate patch presence test for binaries,” in _Proceedings of the 27th USENIX Security Symposium_ , 2018, pp. 887–902.
  * [73] Y. Zhou, S. Liu, J. Siow, X. Du, and Y. Liu, “Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks,” in _Proceedings of the 32nd Annual Conference on Neural Information Processing Systems_ , 2019, pp. 10 197–10 207.
  * [74] Y. Zhou and A. Sharma, “Automated identification of security issues from commit messages and bug reports,” in _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering_ , 2017, pp. 914–919.

|  Congying Xu received his BS.c. degree from Yangzhou University in 2019. He
is a master student at the School of Computer Science in Fudan University,
China. His research interests include software security and third-party
library analysis.
---|---
|  Bihuan Chen received his Ph.D. degree in Computer Science from Fudan
University in 2014. He was a postdoctoral research fellow in Nanyang
Technological University from 2014 to 2017. He is an associate professor in
Fudan University, China. His research interests lie in software engineering,
focusing on program analysis, big code analysis, and software security. He
received two ACM SIGSOFT Distinguished Paper Awards at FSE 2016 and ASE 2018.
---|---
|  Chenhao Lu is an undergraduate student at the School of Computer Science in
Fudan University, China. His research interests include software security and
third-party library analysis.
---|---
|  Kaifeng Wang received his BS.c. degree from Donghua University in 2016. He
is a Ph.D. student at the School of Computer Science in Fudan University,
China. His research interests include software maintenance and evolution,
program analysis, and third-party library analysis
---|---
|  Xin Peng received his Ph.D. degree in Computer Science from Fudan
University in 2006. He is a full professor at the School of Computer Science
in Fudan University, China. His current research interests include big code
analysis, intelligent software development, software maintenance and
evolution, and mobile computing.
---|---
|  Yang Liu received his B.Sc. and Ph.D. degree from National University of
Singapore (NUS) in 2005 and 2010, respectively. He is a a full professor and
University Leadership Forum Chair in Nanyang Technological University. His
research focuses on cybersecurity, software engineering, and artificial
intelligence.
---|---