# Tracking the Evolution of Static Code Warnings: the State-of-the-Art and a
Better Approach

Junjie Li, and Jinqiu Yang J. Li, and J. Yang are with the Department of
Computer Science and Software Engineering, Concordia University, Montreal,
Quebec, Canada.
E-mail: l_unjie<EMAIL_ADDRESS>
###### Abstract

Static bug detection tools help developers detect problems in the code,
including bad programming practices and potential defects. However, it is
known that static bug detectors remain underutilized due to various reasons.
Recent advances to incorporate static bug detectors in modern software
development workflows, such as in code review and continuous integration, are
shown capable of better motivating developers to fix the reported warnings on
the fly. A proper mechanism to track the evolution of the reported warnings
can better support such integration. Moreover, tracking the static code
warnings will benefit many downstream software engineering tasks, such as
learning the fix patterns for automated program repair and learning which
warnings are of more interest, so they can be prioritized automatically.
Hence, precisely tracking the warnings by static bug detectors is critical to
improve the utilization of static bug detectors further.

In this paper, we study the effectiveness of the state-of-the-art (SOA)
solution in tracking the warnings by static bug detectors and propose a better
solution based on our analysis of the insufficiencies of the SOA solution. In
particular, we examined over 2000 commits in four large-scale open-source
systems (i.e., JClouds, Kafka, Spring-boot, and Guava) and crafted a dataset
of 3,452 static code warnings by two static bug detectors (i.e., Spotbugs and
PMD). We manually uncover the ground-truth evolution status of the static
warnings: persistent, resolved, or newly-introduced. Moreover, upon manual
analysis, we identified the main reasons behind the insufficiencies of the SOA
solution. Finally, we propose a better approach to improving the tracking of
static warnings over software development history. Our evaluation shows that
our proposed approach provides a significant improvement in terms of the
precision of the tracking, i.e., from 66.9% to 90.0%.

###### Index Terms:

static analysis, code refactoring, software evolution, empirical study.

##  1 Introduction

Static bug detection tools have been widely applied in practice to detect
potential defects in software. To name a few, both Google and Facebook adopt
static bug detectors in their large codebases on a daily basis [1]. However,
static bug detectors are known to be underutilized due to various reasons.
First, static bug detectors detect an overwhelming number of warnings, which
may be far beyond what resources are allowed to resolve. For example, Spotbugs
[2], i.e., the spiritual successor of Findbugs, detects thousands of or more
static code warnings in one version of JClouds. Second, static bug detectors
are known to detect many false positive warnings. The existence of a large
number of false positives discourages developers from actively working on
resolving the reported warnings. As a result, a significant portion of static
code warnings remain unresolved by developers and can hinder software quality.

There have been efforts from a variety of directions to improve the
utilization of static bug detection tools, e.g., prioritizing and recommending
actionable static warnings and identifying false positive warnings. For
example, researchers have been working on techniques to identify the
actionable warnings and reduce the false static code warnings, such as
recommending actionable warnings by learning from past development history [3,
4]. On the other hand, recent studies show that by better integrating static
bug detectors in software development workflows, such as code review and
continuous integration, developers demonstrated a higher response rate in
resolving the reported static warnings [1, 5]. Developers are presented with
much fewer warnings, which are introduced by a new commit, and encounter fewer
context switch problems in fixing the warnings.

Making static bug detectors more frequent in workflows such as code review
requires proper management of the evolving static code warnings. Such proper
management is not straightforward. One way is to adapt differential static
analysis to only analyze modified code files, yet to achieve satisfactory
performance. However, it requires algorithm innovation and non-trivial
engineering effort for every static bug-finder.

Alternatively, we advocate for management that tracks the evolution of static
code warnings in the commit history, i.e., show the diffs of the static code
warnings from two consecutive software revisions. Tracking the evolution of
static code warnings reveals that given a commit, which warnings remain
unresolved by developers, which warnings are resolved, and which warnings are
newly-introduced in the commit.

More importantly, effective management of static code warnings will benefit
many downstream software engineering tasks. To name a few, researchers have
been crawling past fixes of static code warnings to provide fix suggestions
for new warnings [6, 7], which has been shown can further improve the
utilization of static bug detectors. Furthermore, such concluded fix patterns
are shown to be effective in automated program repair techniques [8].

To this end, there has been little effort to systematically review existing
solutions to track the evolution of static code warnings and accordingly
propose better solutions. Prior studies rely on simple heuristics to track the
static code warnings [9, 4], i.e., two warnings are identical if they are of
the same warning type, in the same file, etc. Avgustinov et al. [10] present
an algorithm that combines various types of information of one warning,
compares two warnings in layers, and eventually establishes mappings between
two sets of warnings from two software revisions. This algorithm is adopted by
recent automated program techniques, and in this study, we refer to it as the
state-of-the-art (SOA) solution. For example, Liu et al. [6] adapt the SOA
solution to identify warning-fixing commits in software repositories for
automated program repair. However, it remains unknown how accurate the SOA
solution is in tracking the static code warnings. An unacceptable performance
of the SOA solution in tracking static code warnings has subsequent negative
impacts on the downstream tasks.

Hence, to foster future research in static code warnings, in this paper, we
examine the performance of the SOA solution in tracking static code warnings
and propose a better approach after analyzing the insufficiencies of the SOA
solution. We answer the following research questions:

RQ1 Is the SOA approach good at tracking the evolution of static code
warnings?

RQ2 What are the limitations of the SOA approach?

RQ3 Can our proposed approach perform better than the SOA approach?

RQ4 How accurate is our proposed approach for tracking the evolution of static
code warnings?

In particular, our study includes two static bug detectors (i.e., PMD and
Spotbugs) and four large-scale open-source software systems (i.e., JClouds,
Kafka, Spring-boot, Guava) for manual analysis. Figure 1 shows an overview of
our study.

To answer RQ1 and RQ2, we crafted a dataset of static code warning and their
evolution. In particular, we took statistically significant samples of the
reported static code warnings from the entire development history of JClouds
and Kafka, and performed manual analysis to label whether each sampled static
code warning is persistent, resolved, or newly-introduced between two
consecutive revisions. Eventually, we crafted a dataset of 1,715 static code
warnings and their evolution status for both manual analysis and future
evaluation.

After analyzing the limitations of the SOA solution (RQ2), we propose a better
approach by addressing the uncovered limitations. Our proposed approach
leverages refactoring information and a better matching strategy: Hungarian
algorithm [11], a classic algorithm to solve the assignment problem in
bipartite graphs to reduce the impact from the order of the SOA approaches.

To answer RQ3, we performed a comparative evaluation between our approach and
the SOA approach. In addition to JClouds and Kafka, we also select two other
open-source software systems (i.e., Spring-boot and Guava). There are 3,452
labeled warnings we collect from four software systems. Our evaluation based
on this dataset shows that our approach provides a significant improvement
over the SOA solution, i.e., from 66.9% to 90.0% in terms of the tracking
precision. In RQ4, we also conducted an independent evaluation of the
performance of our approach by taking a statistically significant sample of
all commits from the four software systems with each static bug detector to
determine the resolved and newly-introduced warnings. Finally, the tracking
precision of our approach is calculated, and our approach achieves 91.8%
accuracy.

In summary, this paper makes the following contributions:

  * •

We collected and manually labeled a dataset of 1,715 static code warnings and
uncovered the ground-truth evolution status between two consecutive commits.
The static code warnings are detected by two mature and widely used static bug
detectors (PMD and Spotbugs) on two real-world open-source software projects
(JClouds and Kafka).

  * •

We examined the SOA solution in tracking the evolution of static code warnings
in terms of tracking accuracy based on the collected dataset. Our
investigation shows that the SOA solution achieves inadequate results.

  * •

We performed a manual analysis to uncover the inaccuracies and the reasons
behind the low accuracy of the SOA solution. Our findings offer empirical
evidence to further improve the tracking of static code warnings in the
development history. We also select two other top-rated projects (Guava and
Spring-boot) with 1,737 static code warnings to evaluate between both
approaches.

  * •

We proposed a better approach to tracking the static code warnings. The
evaluation based on the crafted dataset shows that our approach can
significantly improve tracking precision. We released a replication
package111https://10.5281/zenodo.6549386.

Paper organization. The rest of the paper is organized as follows. Section 2
describes the background, i.e., the relevant knowledge on static code warnings
and how the SOA approach works to track the static code warnings in
development history. Section 3 illustrates the process and results of our
manual analysis to understand the problems of the SOA solution, including how
the dataset is crafted and what are the insufficiencies of the SOA solution.
Section 4 shows our proposed approach, and its evaluation is shown in 5.
Section 6 describes the threats to validity. Section 7 lists the related work,
and Section 8 concludes this study and proposes future works.

Figure 1: An overview of our study.

##  2 Background

This section will introduce the background information on the basics of static
code warnings and how the state-of-the-art (SOA) solution proposed by
Avgustinov et al. works to track the evolution of static code warnings in
software development history.

The Metadata of Static Code Warnings. Static bug detectors often represent
detected static code warnings using metadata. Examples of such metadata
include file path (i.e., the path of the source-code file where one warning is
detected) and types of static code warnings. Previous studies [10] [6] utilize
the metadata information that static bug detectors provide for each warning to
track the evolution of the detected static code warnings.

Different static bug detectors are different in many aspects, such as the
warning types and source code representations (e.g., binary or source-code).
Despite the differences, the metadata that is generated by static bug
detectors includes more or less similar information. Such metadata can
distinguish one static code warning from another in the same revision (e.g.,
warning type and code location). When the metadata of the same static code
warning across two revisions (i.e., before and after one commit) is modified
due to the introduced code changes by the commit, tracking the evolution of
this static code warning may become challenging.

Figure 2 provides an example of a static code warning in JClouds that is
detected by Spotbugs. We show the example of metadata in XML format. The
metadata of the static code warning includes the following detailed
information: the type of the static code warning (i.e., SE_BAD_FIELD), and the
problematic code region of this warning, which is represented by project name,
class name, method name, field name, and the code range that is defined by a
start line and an end line.

⬇

1 <WarningInstance>

2 <WarningType>SE_BAD_FIELD</WarningType>

3 <Project>jclouds</Project>

4 <Class>ContextBuilderTest</Class>

5 <Method></Method>

6 <Field></Field>

7 <FilePath>org/jclouds/ContextBuilder.java</FilePath>

8 <StartLine>70</StartLine>

9 <EndLine>75</EndLine>

10 </WarningInstance>

Figure 2: An example of the representation of one static code warning from
Spotbugs. Note that the representation has been simplified to only show the
information used by the SOA matching approach.

Tracking static code warnings. Static bug detectors, when being run in batch
mode, report a list of static code warnings given one version of a software
system (i.e., one snapshot). Tracking the evolution of static code warnings in
development history is based on comparing the generated reports from every two
consecutive revisions. Figure 3 shows a simple example of how the tracking
works between two revisions. Given one commit, the left block represents all
the warnings from the pre-commit revision, and the right one is the post-
commit revision. Between the two revisions, some static code warnings persist,
i.e., w1, w2, w3 and w4. Note that w1 is in different shapes in the two
revisions as the representation of w1 may have different values due to code
changes. One warning is resolved (i.e., w5) and one warning is newly-
introduced (i.e., w6) by the commit.

A proper tracking mechanism needs to precisely label each static code warning
as either persistent, resolved or newly-introduced. In particular, it is
required that all the mappings are correctly established despite that code
changes may modify the metadata information of the same warnings across
versions. For example, the solid lines in the figure describe one possible
matching outcome, which is not ideal as the w2 warnings of the two revisions
are not correctly mapped, and w3 of the post-commit revision is incorrectly
mapped with w4 of the pre-commit revision. After the mappings are established,
a tracking solution will determine the label of each static code warning,
i.e., both w2 and w4 are decided as resolved while both of them actually
persist between the two revisions. Both are false positives of the tracking
solution. Differently, w5 and w6 are true positives of the tracking solution
as their labels are correct, i.e., w5 is resolved and w6 is newly-introduced.
Interestingly, although w4 is also incorrectly matched, the label of w4 of the
pre-commit revision, which is resolved, is indeed correct. However, the labels
of w3 of the pre-commit revision and w4 of the post-commit revision are
matched incorrectly.

Figure 3: An example of matching the warnings between two consecutive
revisions.

There exist various software maintenance efforts to make the tracking problem
more complicated than one may imagine. For example, code changes that are
irrelevant to efforts of resolving static code warnings, such as a drastic
refactoring, may modify the metadata information of many static code warnings
so they cannot be mapped while they should. On the contrary, a warning that is
resolved may share similar information (due to code changes) with one other
irrelevant warning, which causes an incorrect mapping. In short, a fine line
needs to be drawn to precisely distinguish the three evolution types.

Based on the status of the static code warnings (i.e., persistent, resolved,
and newly-introduced), we define the true positive, false positive, true
negative, and false negative of tracking static warnings in our study.

  * •

True positive (TP): A static warning is identified as a resolved/newly-
introduced warning by a tracking approach correctly.

  * •

False positive (FP): A static warning is identified as a resolved/newly-
introduced warning by a tracking approach incorrectly.

  * •

True negative (TN): A static warning is identified as a persistent warning
(i.e., neither resolved nor newly-introduced) by a tracking approach
correctly.

  * •

False negative (FN): A static warning is identified as a persistent warning
(i.e., neither resolved nor newly-introduced) by a tracking approach
incorrectly.

The state-of-the-art (SOA) solution. Avgustinov et al. [10] proposed a multi-
stage matching algorithm that can properly track the evolution of static code
warnings under certain complicated software evolution, which we refer to as
the SOA solution. The overall structure of the SOA solution is based on a
pair-wise comparison between each warning in the pre-commit revision and each
warning in the post-commit revision. Once a mapping is established, the two
warnings from the two revisions are excluded for further comparisons. In
particular, for each pair-wise matching process, i.e., between one warning
from the pre-commit revision and one from the post-commit revision, four
different matching strategies are placed in order, namely exact matching,
location-based matching, snippet-based matching, and hash-based matching.

Input: The set of warnings from the pre-commit revision, $W_{p}$; The set of
warnings from the post-commit revision, $W_{c}$;

Output: The set of resolved warings, $W_{resolved}$; The set of newly-
introduced warings, $W_{newly-introduced}$; The set of matched pairs,
$MatchedPairs$;

1

2for  _each $w_{i}$ in $W_{p}$_ do

3 for  _each $w_{j}$ in $W_{c}$_ do

4 if  _source file of $w_{i}$ is not a changed file_ then

5 take $ExactMatching(w_{i},w_{j})$;

6

7 else

8 take $ExactMatching(w_{i},w_{j})$;

9

10 if  _$w_{i}$ is not be matched up_ then

11 take $LocationMatching(w_{i},w_{j})$;

12

13 else

14 make a $MatchedPair(w_{i},w_{j})$;

15 remove $w_{j}$ from $W_{c}$ ;

16 break;

17

18 if  _$w_{i}$ is not be matched up_ then

19 take $SnippetMatching(w_{i},w_{j})$;

20

21 else

22 make a $MatchedPair(w_{i},w_{j})$;

23 remove $w_{j}$ from $W_{c}$;

24 break ;

25

26 if  _$w_{i}$ is not be matched up_ then

27 take $HashMatching(w_{i},w_{j})$;

28

29 else

30 make a $MatchedPair(w_{i},w_{j})$;

31 remove $w_{j}$ from $W_{c}$;

32 break;

33

34 if  _ $w_{i}$ is not be matched up_ then

35 add $w_{i}$ into $W_{resolved}$;

36

37

38

39

40$W_{newly-introduced}=W_{c}-MatchedPairs$;

41

Algorithm 1 The basic algorithm of the SOA approach.

Algorithm 1 illustrates how the SOA solution works to establish mappings
between the list of warnings of two consecutive revisions. Exact matching
requires every piece of metadata information to be matched and therefore is
the most strict matching strategy among the four. When exact matching fails,
the SOA solution will then utilize the less strict matching strategy, i.e.,
location-based matching, which employes the diff algorithm to tolerate certain
line shifts. If the location-based matching fails, the SOA solution will
continue to use snippet-based matching. When a class file was renamed or
moved, the above matching strategies cannot handle that. Thus, the SOA
solution will utilize hash-based matching.

At the end, when all the possible mappings are established, the unmatched
warnings in the pre-commit revision are determined as resolved, and the ones
in the post-commit revision are considered as newly-introduced.

Exact matching. Exact matching establishes the mappings for the warnings that
are totally unaffected by the commit. For the two warnings, it is required
that they have exactly the same source location (i.e., defined by the start
and end line of the warning), warning type, and code information (i.e., class
name, method name, and variable name).

Location-based matching. A commit may modify the information of certain static
code warnings. Therefore when the exact matching fails, the following matching
strategy, location-based matching, is used to tolerate the impacts to some
extent. Location-based matching utilizes the diff algorithms [12] [13] to
derive source position mappings for each modified file. When a (potential)
matching pair of warnings is in the diff output, location-based matching
compares the offset of the corresponding warnings in the mappings. This
matching requires the same warning metadata of code information (i.e., class
name, method name, and variable name), but does not require the same source
location (i.e., the start and end line of the warning). If the difference of
offsets is equal to or lower than 3 (i.e., a fixed threshold), the location-
based matching will decide the two warnings as a matching pair.

⬇

@@ -84,3 +84,5 @@

84 84 public class MyClass{

⬇

85 + // add code

86 + //

⬇

85 87 private String str = null;

86 88 }

Figure 4: An example of one commit shows how location-based matching works.

As an example, Figure 4 shows a diff mapping. The numbers on the left hand are
the line numbers in the pre-commit revision. The numbers on the right are the
line numbers in the post-commit revision. There is a warning reported in the
pre-commit revision (line 85) and line 87 in the post-commit revision. Due to
code adding, the source location (i.e., part of the warning metadata) has been
changed. Location-based matching firstly computes the offsets between the
source location and the diff mappings, respectively. The offset between the
first line of the diff mapping (line 84) and the warning (line 85) is 1 for
the pre-commit revision and 3 (line 87 and line 84) for the post-commit
revision. Then, the differences between the two offsets are calculated. In
this example, the difference is smaller than 3, so location-based matching
will match the two warnings.

Snippet-based matching. When code location changes significantly, the
location-based matching approach may fail to identify persistent warnings
across revisions. Snippet-based matching is used to address this problem.
Given the source location defined by a start line and an end line, code
snippets in between are extracted from both revisions. By performing the
string matching on the two code snippets, snippet-based matching will decide a
mapping if they are identical. Same as location-based matching, snippet-based
matching requires the same warning metadata of code information (i.e., class
name, method name, and variable name).

Hash-based matching. It is possible that a file is moved to a new location or
renamed (i.e., class and file path are modified). Snippet-based matching
cannot handle such cases well since the class name are required to be
identical to perform snippet-based matching. For such cases, a hash-based
matching approach can be helpful. This matching approach tries to match
warnings based on the similarity of their surrounding code. It first splits
the text of the warning location into several tokens. Two hash values are
calculated $h(W_{topN})$ and $h(W_{latter})$. $W_{topN}$ is $\mathit{n}$
tokens from the first one. $W_{latter}$ is tokens from the
$\mathit{n+1^{\text{th}}}$ token to the last token. $\mathit{n}$ is a fixed
threshold. If two warnings (i.e., one is from pre-commit revision and another
one is from post-commit revision) whose $h(W^{post}_{topN})=h(W^{pre}_{topN})$
or $h(W^{post}_{latter})=h(W^{pre}_{latter})$, they will be considered as a
matched pair.

How one incorrect mapping may impact all. Tracking the warnings across two
consecutive revisions should be treated as one integral issue because each
incorrect mapping not only impacts the two warnings involved but also impacts
other warnings. For example, in the imperfect mappings in Figure 3, w3 on the
left is incorrectly mapped with w4 on the right. This incorrect mapping not
only impacts these two warnings, but also w4 on the left and w3 on the right.
Now w4 on the left is decided as resolved (incorrect), and w3 on the right is
considered as newly-introduced (also incorrect). Interestingly, not all the
incorrect mappings must introduce false positives. Figure 5 shows an example
in this category. On the left figure, we can see two incorrect mappings: w1 to
w2 and w2 to w1. Although both mappings are incorrect, there is no false
positive generated since these four warnings form a closure that therefore
does not impact the mappings of other warnings. On the contrary, Figure 5 (b)
is a worse case: two false positives (w1 on the left and w2 on the right) are
generated). Our goal is to reduce the second type of mappings through our
analysis.

|
---|---
a | b

Figure 5: Examples to demonstrate how incorrect mappings may impact others.

##  3 Examining the performance of the State-of-the-Art Solution in Tracking
the Evolution of Static Code Warnings

In this section, we describe how we investigated the performance of the SOA
approach in terms of the tracking accuracy, and answer RQ1 and RQ2. In
particular, we first crafted a dataset of static code warnings and their
evolution status (i.e., persistent, resolved, or newly-introduced) between two
consecutive revisions. To craft this dataset, we re-implemented the SOA
approach, applied it to the development history of the studied open-source
systems, and performed a manual analysis to determine the evolution status of
each sampled static code warning. Then we manually analyzed whether the SOA
approach correctly tracked each sampled static code warning and categorized
the reasons behind the incorrect trackings.

###  3.1 Studied Subjects

Static bug detectors. In this paper, we include two static bug detectors,
i.e., PMD and Spotbugs, both of which are widely used in prior studies and
adopted in practice. In particular, Spotbugs, a spiritual successor of the
well-known Findbugs, can detect more than 400 bug patterns in Java programs
through bytecode analysis. Differently, PMD supports multiple languages and is
known to be easily integrated into the build process. We use the two static
bug detectors with their default configuration.

Analyzed open-source systems. Our study includes four Java open-source
systems, JClouds, Kafka, Spring-boot and Guava. Two of the systems (i.e.,
JClouds and Kafka) are used to summarize the insufficiencies of the SOA and
provide reasons for the introduction of false positives. The other two systems
(i.e., Spring-boot and Guava) are selected to evaluate both approaches
systematically. We applied the two static bug detectors to all the revisions
in a specific development period of the four studied software systems. We
started with the official releases of the two software systems when we started
this study, i.e., 2.3.1-rc2 of JClouds, 2.1.0 of Kafka, v2.3.6 of Spring-boot
and v20.0 of Guava. We selected the last commit of the studied release as the
end date and its previous one and a half year as the studied development
history. We were not able to successfully compile some revisions of systems in
the studied period and excluded them from further studies. Besides, we also
excluded the revision that has multiple pre-commit revisions. Table I lists
the statistics of the studied systems, including the lines of code (LOC), the
number of analyzed commits, the official release that we used to decide the
end date of the studied development history, and the number of aggregated
warnings in all the analyzed commits.

TABLE I:  The studied systems and development periods. The release marks the
end date of our studied development period, and we include 18 months
development history before the specified release.

| KLOC | # Commits | Release | # Average Warnings
---|---|---|---|---
|  |  |  | PMD | Spotbugs
Kafka | 434 | 2,000 | 2.3.1-rc2 | 12,972 | 27,911
JClouds | 494 | 300 | 2.1.0 | 18,176 | 2,090
Spring-boot | 2,695 | 400 | v2.3.6 | 5,931 | 6,918
Guava | 2,112 | 2,000 | v20.0 | 6,474 | 3,819

###  3.2 Collecting the Dataset Through Manual Labeling

Before we describe how we collect the dataset, i.e., a list of static code
warnings and their evolution status, we would like to motivate a few key
points that drive our design choice in crafting the dataset. First, given a
large number of accumulative warnings across the revisions, we have to set our
priorities, i.e., the evolution status of which static code warnings can
better showcase the performance of the tracking approach since we have limited
manual resources to spare. Second, it is not surprising that in reality, the
majority of the warnings persist in the codebases [14]. Therefore we do not
consider it particularly interesting to include a corresponding percentage of
persistent warnings in the dataset. Third, considering the downstream software
engineering research this study can benefit from, we set our priorities to
focus more on the static warnings that are resolved or newly-introduced.

Fourth, when doing sampling, whenever possible, we include all the warnings in
one commit due to the inherent challenge in the mapping problem: one incorrect
mapping may impact others, if only including one, we may observe part of the
impact (i.e., “the tip of the iceberg”). Last, we have certain confidence in
the performance of the SOA approach from its design. For example, we find that
majority of the established mappings by the SOA approach is by the exact
mapping (e.g., 3,137 out of the 3,163 by Spotbugs in JClouds-09936b5). The
exact matching is the most strict matching process and rarely produces wrong
results.

Guided by these key points, we decided to craft the dataset based on the
tracking results of the SOA approach and set our priorities on resolved and
newly-introduced static code warnings. As illustrated in Section 3.1, we apply
the static bug detectors on a total of 2,300 commits in Kafka and JClouds. We
re-implemented the SOA approach based on the original paper [10] with
references to a recent implementation by Liu et al. [6]. Note that the
implementation by Liu et al. is based on Findbugs. Thus, we decided to re-
implement the SOA approach for PMD and Spotbugs. Then, we applied the SOA
approach to track the evolution of the static code warnings across all the
analyzed commits.

We select a subset of static code warnings for manual labeling following the
steps:

  1. 1.

For JClouds_Spotbugs and JClouds_PMD, we include all the static warnings
labeled as resolved by the SOA approach.

  2. 2.

Since there are many (i.e., 2,038 and 1,359) resolved static warnings in
Kafka_PMD and Kafka_Spotbugs, we took a statistically significant (95%$\pm$5%)
sample, i.e., 326 and 301 resolved static code warnings for both of them.
Using Kafka_PMD as an example, we pursued the sampling process by firstly
getting an estimation of the sample size, i.e., 323 warnings, then starting to
randomly select one commit from the 436 Kafka commits with at least one
resolved warning, until we collected more than 323 warnings. In the end, we
collected 326 resolved warnings from 53 commits in Kafka_PMD.

  3. 3.

In each of the four settings, there exist a large number of newly-introduced
code warnings. Hence, we took a statistically significant sample (95%$\pm$5%)
of warnings in each setting and followed a similar sampling process as Step 2,
i.e., including all newly-introduced warnings in one sampled commit. In the
end, we collected a total of 704 warnings (i.e., in 47 commits) labeled as
‘newly-introduced’ by the SOA approach.

TABLE II:  A summary of how we collect the static code warnings based on the
results of the SOA approach.

| SOA: “Resolved” | SOA: “Newly-Introduced”
---|---|---
| # Commits | # Warnings | # Commits | # Warnings
PMD |  |  |  |
JClouds | 57 | 280 | 19 | 155
Kafka | 53 | 326 | 14 | 255
Spotbugs |  |  |  |
JClouds | 23 | 104 | 5 | 78
Kafka | 26 | 301 | 9 | 216
Total | 159 | 1,011 | 47 | 704

Table II summarizes the breakdown of the static code warnings we collect
following the above-mentioned steps. Note that in Table II, the evolution
statuses such as resolved and newly-introduced are labeled by the SOA
approach, which might be incorrect. We performed a manual analysis to reveal
the true evolution status of each warning. Table III summarizes the ground-
truth evolution status on the dataset. In total, our dataset contains 1,715
static code warnings and their true evolution status in the development
history of JClouds and Kafka: 37.6% are persistent, 32.2% are resolved, and
30.1% are newly-introduced. In particular, two of the researchers individually
performed a manual analysis to uncover the ground-truth evolution status of
each selected static code warning. The manual analysis includes understanding
the nature of each static code warning and code changes that may evolve the
code warnings. The two researchers discussed the labels to resolve any
disagreements. In our experiments, most of the disagreements are caused by
human errors and can be easily agreed on. We calculated Cohen’s kappa to
measure the inter-rater agreement, which is the almost perfect level (0.96) in
our experiment.

It is noticeable that there exists a non-trivial discrepancy between Table II
and Table III regarding the distribution of the evolution statuses. That is
because the SOA approach produces a non-negligible number of incorrect
results. We present more details on the inaccuracies in Section 3.4.

Except for the resolved and newly-introduced warnings, we also inspect the
persistent warnings of the SOA approach. Due to a large number of persistent
warnings, we take a statistically significant (95%$\pm$5%) sample, i.e., 384
persistent warnings. After checking all of them, we do not find there are any
mismatching pairs. The sampled persistent are available online [15].

TABLE III: A summary of the 1,715 static code warnings in the data set based
on manually-labeled evolution statuses. The dataset with ground-truth labels
is used to evaluate the SOA approach and ours.

| Persistent | Resolved | Newly-Introduced
---|---|---|---
PMD |  |  |
JClouds | 235 | 102 | 98
Kafka | 170 | 178 | 233
Spotbugs |  |  |
JClouds | 32 | 86 | 64
Kafka | 208 | 187 | 122
Total | 645 | 553 | 517

TABLE IV:  The performance of the SOA approach.

| # Resolved by SOA | # New-Introduced by SOA
---|---|---
| Total | True Positive | False Positive | Total | True Positive | False Positive
PMD |  |  |  |  |  |
JClouds | 280 | 102 (36.4%) | 178 (63.6%) | 155 | 98 (63.2%) | 57 (36.8%)
Kafka | 326 | 178 (54.6%) | 148 (45.4%) | 255 | 233 (91.4%) | 22 (8.6%)
Spotbugs |  |  |  |  |  |
JClouds | 104 | 86 (82.7%) | 18 (17.3%) | 78 | 64 (82.0%) | 14 (18.0%)
Kafka | 301 | 187 (62.1%) | 114 (37.8%) | 216 | 122 (56.5%) | 94 (43.5%)
Total | 1011 | 553 (54.7%) | 458 (45.3%) | 704 | 517 (73.4%) | 187 (26.6%)

###  3.3 Evaluating the Performance of the SOA Approach

TABLE V: Six causes of False Positives

Cause | Number
---|---
1\. Class relocating or renaming | 106
2\. Method renaming | 33
3\. Attribute and variable renaming | 46
4\. Code shifting | 135
5\. Volatile class/method/variable names | 93
6\. Drastic and non-refactoring code changes | 232

⬇

-\-- org/jclouds/SGOrCreate.java

+++ org/jclouds/SGInRegOrCreate.java

⬇

@@ -73,4 +71,4 @@

73 71 RegionName input = new RegionName();

74 72

⬇

75 - Func<SGReg> group = new Func<SGReg>() {

⬇

73 + Func<SG> group = new Func<SG>() {

⬇

76 74

Figure 6: An example of the false positives due to class renaming

RQ1. Is the SOA approach good at tracking the evolution of static code
warnings?

We manually investigated the sampled 1,715 static code warnings. Table IV
summarizes the performance of the SOA approach on the crafted dataset. Among
the 1,011 warnings that are determined as resolved by the SOA approach, only
553 (54.7%) are truly resolved, i.e., true positives. Among the 704 warnings
that are determined as newly-introduced, only 517 (73.4%) are actually newly-
introduced, i.e., true positives. The false positives in these two categories
are the warnings that are persistent as the ground truth. In short, the
overall precision of the SOA approach on the collected dataset is only 62.4%,
i.e., (553+517)/1,715. Our evaluation of the SOA approach reveals that
tracking the evolution of static code warnings over the development period is
not that straightforward. The low precision of the SOA approach will
negatively impact many downstream software engineering tasks, such as mining
fix patterns from software repositories or performing empirical studies on
software quality.

To this end, we answer RQ1 after examining the performance of the SOA approach
by analyzing a number of tracked static code warnings.

We present a dataset of 1,715 static code warnings and their evolution
statuses. The dataset is crafted with support from the SOA approach. The
precision of the SOA approach on the dataset is only 62.4%, and this tracking
approach will impact many downstream software engineering tasks negatively.

###  3.4 Investigating the Inaccuracies of the SOA Approach

RQ2. What are the limitations of the SOA approach?

Furthermore, we manually analyzed the insufficiencies of the SOA approach,
i.e., based on 645 cases, and concluded into six categories as follows. Table
V summarizes six causes of false positives in our dataset.

Class relocating or renaming. In the SOA approach, specifically in the first
three types of matching algorithms (i.e., exact_matching, location_matching,
and snippet_matching, in Section 2), having the same class name is a condition
that must be satisfied. Hence, if a Java class is renamed or relocated, these
three types would fail to find a mapping. The last matching strategy, namely
hash-based matching, may handle some of such cases. However, hash-based
matching is highly sensitive to code changes, i.e., if the hashed region has
code changes, the hash values would alter, and then the hash-based matching
strategy would also fail to match the warnings. In total, we find 106 false
positives in this category.

Figure 6 is a case of a false positive due to Class renaming. The top two
lines show the class file has been renamed, i.e., a different file name and
also a different file path, which will cause the first three matching
strategies to fail. Hash-based matching has the potential to handle this case;
however, this file does have code changes, e.g., lines 73-75, and then hash-
based matching may fail. In the end, there exist no mappings established by
the SOA algorithm at all due to the file renaming. As a result, warnings in
SGOrCreateTest.java are considered resolved even though the code changes do
not resolve the static warnings.

⬇

@@ -92,5 +81,5 @@

92 81 @Test

⬇

93 - public void shouldClloseOpenIterators() {

⬇

82 + public void shouldCloseOpenRange(){

⬇

94 83 nodes.put(new Node("host1",8121));

95 84 nodes.put(new Node("host2",8122));

96 85

Figure 7: An example of false positives due to method renaming.

Method renaming. Similar to class renaming, method renaming affects the
tracking in a similar fashion, i.e., the same method name is required in the
first three matching strategies. Different method names force the SOA approach
to rely on the hash-based matching strategy. However, the hash-based matching
strategy is sensitive to regional code changes, which causes it to be unable
to map the persistent warnings between two revisions. In total, we find 33
false positives in this category. Figure 7 is a case of a false positive due
to method renaming. There is one warning reported in the pre-commit revision
(line 95), which persists (line 83) in the post-commit revision. However, as
the method name changes (i.e., highlighted in red and green), the first three
matching strategies cannot match the warnings in the method. Thus the SOA
approach relies on the hash-based matching strategy. Due to the high
sensitivity of the hash-based matching strategy, some persistent warnings in
this method will not be mapped at all, which leads to inaccurate evolution
statuses.

⬇

@@ -64,6 +64,6 @@

64 64 this.name = name;

65 65 this.des = options.description();

⬇

67 - this.IPAddress = option.ipAddress();

⬇

67 + this.ipAddress = option.ipAddress();

⬇

68 68 this.portRange = option.portRange();

69 69 this.target = option.target();

70 70

Figure 8: An example of the false positives due to attribute renaming.

Attribute and variable renaming. Attribute and variable renaming, as a common
refactoring, also causes the SOA approach to malfunction. However, the impact
process is different from the two above-mentioned cause categories. The SOA
approach uses attribute and variable information to filter out some matching
candidates. Obviously, the two warnings with the attribute or variable
information cannot be matched after such changes. Totally, we find 46 false
positives in this category.

Figure 8 is a case of a false positive due to Attribute and Variable renaming.
The same warning is located in line 67 of the pre-commit revision and the
post-commit revision. An attribute IPAddress is renamed to ipAddress, which
causes false positives in the original algorithm.

⬇

200 204

201 205 assertEquals(a,null);

202 206

203 207 assertEquals(b,null);

204 208

205 209 assertEquals(c,null);

Figure 9: An example of the false positives due to code shifting.

Code shifting. Commits may modify the line numbers of some code statements,
although these code statements are not directly modified by the commits. We
call this code shifting. Because there exist similar code statements with
similar static code warnings (e.g., same warning type, same variable, etc.),
when code shifting happens, the SOA approach does not always handle the
shifting well, and false positives will be produced. Totally, we find 135
false positives in this category.

Figure 9 shows an example of how code shifting may cause the SOA approach to
malfunction. Even though the three statements remain unchanged, their line
numbers become different. In the pre-commit revision, the line numbers are
201, 203, and 205, while in the post-commit revision, the line numbers are
205, 207, and 209. The SOA approach uses line numbers as part of the code
range to build a mapping. As a result, the warning in line 205 from the pre-
commit revision is mapped with the warning in line 205 from the post-commit
revision by exact matching. This incorrect mapping causes the warnings on line
201 and 203 from the pre-commit revision to be considered as resolved while
they actually persist.

⬇

groups.map(_ -> getAcl(opts,

Set(Read))).toMap[ResourcePatternFilter, Set[Acl]]

Figure 10: An example of Scala code that has implicit code changes. The meta
data of the relevant warning from the pre- and post-commit revisions are shown
in Figure 11 and Figure 12.

⬇

1 <WarningInstance>

2 <WarningType>SE_BAD_FIELD</WarningType>

3 <Project>kafka</Project>

4 <Class>AclCommand</Class>

5 <Method></Method>

6 <Field>opts$4</Field>

7 <FilePath>kafka/admin/AclCommand.scala</FilePath>

8 <StartLine>206</StartLine>

9 <EndLine>206</EndLine>

10 </WarningInstance>

Figure 11: The warning information from pre-commit revision

⬇

1 <WarningInstance>

2 <WarningType>SE_BAD_FIELD</WarningType>

3 <Project>kafka</Project>

4 <Class>AclCommand</Class>

5 <Method></Method>

6 <Field>opts$1</Field>

7 <FilePath>kafka/admin/AclCommand.scala</FilePath>

8 <StartLine>330</StartLine>

9 <EndLine>330</EndLine>

10 </WarningInstance>

Figure 12: The warning information from post-commit revision

Volatile class/method/variable names. Even though there is no explicit code
changes in one commit, on certain files, the warning reports by Spotbugs,
which uses bytecode analysis, are sensitive to compilation. Although
everything else remains unchanged, persistent warnings across revisions may
have different line numbers or different class/method/variables names. Such
differences will cause all the matching strategies to malfunction. This
happens frequently in Scala code when anonymous classes and methods are used
heavily. Then some persistent warnings are not matched correctly. Totally, we
find 93 false positives in this category.

Figure 10 is an example of false positives even though there are no explicit
code changes. The line number of the code line with a warning changes from 206
to 330. We examined the metadata of this warning across two revisions (Figure
11 and Figure 12) and found that not only the line numbers are different, the
variable names are also different (line 6 in Figure 11 and line 6 in Figure
12).

Drastic and non-refactoring code changes. If there exist drastic code changes
in close proximity with the persistent warnings, it is possible that location-
based matching will not decide a mapping pair. Furthermore, if there exist
code changes surrounding the code locations of the warnings, the snippet-based
matching strategy may also malfunction to establish correct mappings. Figure
13 shows an example of such a case. A warning is located from lines 382 to 387
of pre-commit revision, and the potential matched warning is from lines 463 to
467 of post-commit revision. Location-based matching cannot match them because
the difference of offsets is higher than the location matching threshold.
Snippet-based matching also fails due to the modified snippet.

Discussions on composite false positives. We notice that many false positives
have more than one cause. For example, a file combines class renaming with
attribute renaming or drastic non-refactoring code changes.

⬇

@@ -381,5 +429,37 @@

381 429 @Test

⬇

430 + ...

431 + ...

... + ...

⬇

382 463 final Runtime = new RuntimException()

383 464

⬇

384 - client.Response(new RequestMatcher()){

385 - @Override

⬇

465 + cient.Response(body -> {

⬇

386 466 ... ...

387 467 }

Figure 13: An example of false positives due to a change of method name and
drastic code changes.

We perform further manual analysis on the FPs of the crafted dataset, and
identify six main causes behind the inaccuracies of the SOA approach in
tracking the evolution of static code warnings.

##  4 A Better Approach to Track Static Warnings

Input: The set of warnings from the pre-commit revision, $W_{p}$; The set of
warnings from the post-commit revision, $W_{c}$;

Output: The set of resolved warings, $W_{resolved}$; The set of newly-
introduced warings, $W_{newly-introduced}$; The set of matched pairs,
$MatchedPairs$;

1

2Construct $W^{hash}_{c}$, a hash index of $W_{c}$

3Initialize a Two-dimensional array $HMatrix$.

4Remove all Identifiers in $W_{p}$ and $W_{c}$

5for  _each $w_{i}$ in $W_{p}$_ do

6

7 if  _source file of $w_{i}$ is not a changed file_ then

8 take $ExactMatching(w_{i},W^{hash}_{c}[h(W_{i})])$;

9

10 else

11 $w^{{}^{\prime}}_{i}=refactoring(w_{i})$; $\triangleright$ if there is no
refactoring in the location of $w_{i}$, $w^{{}^{\prime}}_{i}$ = $w_{i}$.

12

13 for  _each $w_{j}$ in $W_{c}$_ do

14 else

15 take $SnippetMatching(w^{{}^{\prime}}_{i},w_{j})$;

16 if  _there is a candidate from snippet matching_ then

17 $HMatrix[i][j]+=1$;

18

19 take $LocationMatching(w^{{}^{\prime}}_{i},w^{{}^{\prime}}_{j})$;

20 if  _there is a candidate from location matching_ then

21 $HMatrix[i][j]+=1$;

22

23

24

25

26

27$MatchedPairs=Hungarian(HMatrix)$;

28 $W_{resolved}=W_{p}-MatchedPairs$;

29 $W_{newly-introduced}=W_{c}-MatchedPairs$;

Algorithm 2 The algorithm of our improved approach.

Guided by our manual analysis results, we propose to improve the SOA approach
by better handling refactoring changes and revising a few key steps to improve
the accuracy of irrelevant code changes. In particular, our proposed approach
(as illustrated in Algorithm 2) reuses the three matching strategies of the
SOA approach (i.e., Exact matching, Location-based matching, and Snippet-based
matching) and revises a few key steps to improve the inaccurate tracking.

Improvement 1 - Including refactoring. We included the refactoring information
to improve the tracking of static warnings using RefactoringMiner [16]. We
firstly created a replica of $w_{i}$ (namely $w^{{}^{\prime}}_{i}$), which is
from the pre-commit revision, and then modify the metadata of
$w^{{}^{\prime}}_{i}$ with the information from RefactoringMiner. For
instance, if RefactoringMiner reveals that the class in $w_{j}$ is a result of
refactoring of “move and rename class”, we modify the class name in
$w^{{}^{\prime}}_{i}$ with the one after the refactoring activity. Two of the
matching strategies (i.e., snippet matching in line 10 and location matching
in line 13) are re-applied to decide two warnings (i.e., $w_{i}$, and $w_{j}$)
whether they are candidates of a matched pair. In particular, Hash-based
matching is designed to handle the case of the class files renamed or moved
that are included in refactoring information. Thus we remove hash-based
matching.

As of now, we include 22 types of refactoring that cause the modified metadata
of warnings. Table VI shows the refactoring types we include.

TABLE VI: Refactoring types included in our proposed approach.

Refactoring type | Modified metadata of static wanrings
---|---
1\. Extract method | Method name
2\. Rename method | Method name
3\. Move class | Class name
4\. Rename class | Class name
5\. Rename variable | Field name
6\. Rename parameter | Field name
7\. Rename attribute | Field name
8\. Move and rename class | Class name
9\. Move Method | Method name
10\. Move attribute | Field name
11\. Pull up method | Method name
12\. Pull up attribute | Field name
13\. Push down method | Method name
14\. Push down attribute | Field name
15\. Extract Superclass | Class name
16\. Extract and move Method | Method name
17\. Extract Class | Class name
18\. Extract Subclass | Class name
19\. Move and Rename Attribute | Field name
20\. Replace Variable with Attribute | Field name
21\. Move and Rename Method | Method name
22\. Move and Inline Method | Method name

Improvement 2 - Deciding matched pairs using the Hungarian algorithm.
Commonly, a warning of pre-commit revision may have more than one matched
warning from post-commit. Thus it is a problem which one should be matched up.
In the SOA approach, it takes the first-come-first-matched, which may cause
mismatching. Besides, the order of the matching strategies will affect the
result. For example, we may get different results when we adopt location-
matching first and snippet-matching first. The order in the SOA approach is
doing Exact matching first, then Location-based matching, and the last one,
Snippet-based matching. In our investigation, this order has introduced many
false positives like code shifting (Figure 9). Besides, the first-matched
warning may not be the best or correct one,i.e., there exist better-matched
warnings. Thus we adopt Hungarian algorithm, a classic approach to solve the
assignment problem in bipartite graphs. When a warning of post-commit revision
is found that can be matched with a warning of pre-commit revision from the
two matching strategies (i.e., Location-based matching and Snippet-based
matching), instead of deciding it as a matched pair (i.e., a persistent
warning), we construct a Hungarian matrix to save it as a potential matched
pair. An example is like Figure 14. $w1_{p}$, $w2_{p}$ and $w3_{p}$ are the
warnings from pre-commit revision. $w1_{c}$, $w2_{c}$ and $w3_{c}$ are the
warnings from post-commit revision. When two warnings are considered as a
(potential) matched pair, the Hungarian matrix adds one (e.g., $w1_{p}$ and
$w1_{c}$). A value of two (e.g., $w2_{p}$ and $w2_{c}$) means they are a
(potential) matched pair from both matching strategies. It also means that
this pair is more likely to be an actual pair of persistent warnings. If the
SOA is applied on the six static warnings , it is possible that $w1_{p}$ is
matched with $w1_{c}$, and $w2_{p}$ is matched with $w3_{c}$, so $w3_{p}$ and
$w2_{c}$ become false positives. In our algorithm, we construct a matrix
$HMatrix$ (line 2) like Figure 14. The size of $HMatrix$ is (the number of
$W_{p}$) $*$ (the number of $W_{c}$) and the values are zero initially. Two
matching strategies, Snippet-based matching, and Location-based matching are
used to find out the potential matched warnings. Then we leverage maximum
matching to decide the matched pairs. Besides, there is an Exact matching for
changed files in the SOA matching, but if we adopt Hungarian algorithm, the
matched warnings by Exact matching can also be identified by Location-based
matching or Snippet-based matching. Thus, we simply remove Exact matching for
changed files in our approach. However, we keep it for unchanged files.

Figure 14: A simple example of Hungarian matrix.

Improvement 3 - Working with volatile identifiers. Anonymous classes and
methods are given an identifier after compilation. However, the assigned
identifiers are sensitive to change when there are code changes, even
irrelevant. We try to minimize such sensitivity by removing the variable part
in such identifiers. In particular, for identifiers such as opt$1, we use a
regular expression to remove the numeric suffix after $ and only keep opt as
the variable identifier in the metadata of a warning for the subsequent
matching process.

##  5 Evaluation.

We evaluate our improved approach on the crafted dataset to show how much
improvement our approach has compared to the SOA approach and answer RQ3. In
addition to JClouds and Kafka, we also select two other open-source software
systems (i.e., Spring-boot and Guava). To answer RQ4, we took an independent
evaluation in our improved approach.

###  5.1 Comparative Analysis

RQ3. Can our proposed approach perform better than the SOA approach?

Two static bug detectors are applied on a total of 400 commits for Spring-boot
and 2,000 commits for Guava. Table VII shows the results of the SOA approach.

TABLE VII:  A summary on Guava and Spring-boot based on the results of the SOA
approach.

| SOA: “Resolved” | SOA: “Newly-Introduced”
---|---|---
| # Commits | # Warnings | # Commits | # Warnings
PMD |  |  |  |
Spring-boot | 59 | 218 | 82 | 277
Guava | 220 | 1,241 | 281 | 1,417
Spotbugs |  |  |  |
Spring-boot | 17 | 193 | 22 | 182
Guava | 344 | 1,164 | 430 | 1,441
Total | 640 | 2,816 | 815 | 3,317

For Spring-boot_Spotbugs and Spring-boot_PMD, all static warnings are
included. We take the same sample strategy on Guava_Spotbugs and Guava_PMD, a
statistically significant (95%$\pm$5%) sample of resolved warnings with newly-
introduced warnings of their commits, i.e., 41 commits with 296 resolved
warnings and 188 newly-introduced warnings in Guava_PMD, and 44 commits with
289 resolved warnings and 204 newly-introduced warnings in Guava_Spotbugs. Our
approach is also applied to these commits. Sampled warnings and the warnings
from our approach are labeled by two authors with Cohen’s kappa coefficient of
0.62, which has a substantial agreement. Table VIII illustrates the labeled
results of the two projects.

TABLE VIII:  The labeled results of two projects.

| SOA: Resolved | SOA: Newly-Introduced
---|---|---
| TP/FP (SOA) | TP/FP (our approach) | TP/FP (SOA) | TP/FP (our approach)
PMD |  |  |  |
Spring-boot | 138/80 | 138/22 | 109/80 | 109/22
Guava | 172/124 | 172/15 | 64/124 | 64/15
Spotbugs |  |  |  |
Sprin-boot | 186/7 | 186/1 | 153/7 | 153/1
Guava | 231/58 | 231/15 | 151/53 | 151/10
Total | 727/269 | 727/53 | 477/264 | 477/48

Since tracking the static code warnings is not a standalone task for each
warning, it is, in fact, a mapping problem between two sets. Hence, we applied
our improved approach to all the warnings in the 200 commits, which is a
superset of the 3,452 warnings in the manually-labeled dataset. The remaining
warnings in the 200 commits, while not in our crafted dataset, have a pre-
assumed label, “persistent”. If our approach changes the pre-assumed label of
some warnings, then we manually examine the ground-truth labels of these
warnings. If our approach does not change the pre-assumed labels, it means
that our approach is at least not worse than the SOA approach on the warnings
that are not in our evaluation dataset.

TABLE IX: The performance comparison between the SOA approach and our
approach. Note that FP is short for false positive. A lower FP ratio is
desired.

| Resolved | Newly-Introduced
---|---|---
| FP (SOA) | FP (our approach) | FP (SOA) | FP (our approach)
PMD |  |  |  |
JClouds | 63.6% (178/280) | 3.8% (4/106) | 36.8% (57/155) | 8.4% (9/107)
Kafka | 45.4% (148/326) | 27.0% (66/244) | 8.6% (22/255) | 4.1% (10/243)
Spring-boot | 36.7% (80/218) | 13.8% (22/160) | 42.3% (80/189) | 16.8% (22/131)
Guava | 41.9% (124/296) | 8.0% (15/187) | 66.0% (124/188) | 19.0% (15/79)
Spotbugs |  |  |  |
Jclouds | 17.3% (18/104) | 1.1% (1/87) | 17.9% (14/78) | 3.0% (2/66)
Kafka | 37.8% (114/301) | 15.8% (35/222) | 43.5% (94/216) | 17.0% (25/147)
Spring-boot | 3.6% (7/193) | 0.5% (1/187) | 4.4% (7/160) | 0.6% (1/154)
Guava | 20.1% (58/289) | 6.1% (15/246) | 26.0% (53/204) | 6.2% (10/161)
Total | 36.2% (727/2,007) | 11.1% (159/1,437) | 31.2% (451/1,445) | 8.6% (94/1,087)

Table IX lists the comparison results between the SOA approach and our
improved approach on the collected dataset of 3,452 static code warnings. Note
that there are 3,452 static warnings from the SOA approach. However, when we
applied our approach to the same dataset, we obtained only 2,524 resolved and
newly-introduced warnings, which means that the rest (i.e., 928 warnings) are
identified by our approach as persistent warnings. We categorized the 3,452
warnings into two categories according to the labels by the SOA approach for
ease of comparison. The evaluation shows that our proposed approach can
significantly reduce the false positive rate. Overall, for the 3,452 warnings,
the SOA approach has 1,178 warnings with a wrong evolution status, i.e., the
false positive rate is 34.1%. Compared to that, our proposed approach reduces
the false positives significantly, from 1,178 to 253, i.e., the false positive
rate drops to 10.0%, yielding a precision of 90.0%. To give an example in
Table IX, for JCloud with PMD, the SOA labels 178 persistent warnings wrongly
as resolved while our approach correctly labels 174 (178-4) as persistent,
leaving four false positives with a wrong label. Our approach reduces the
false positives by correctly labeling the persistent warnings, which are
mistakenly labeled as resolved or newly-introduce by the SOA approach.

Among the six causes of the false positives of the SOA approach, our approach
is shown to effectively reduce false positives for all causes. Table XI shows
the breakdown of the left false positives by each cause after using our
approach on the resolved warning dataset.

Additionally, the execution time between the two approaches is evaluated on
four projects. Table X shows that the SOA approach is faster than our approach
on Kakfa_Spotbugs. It has no significant difference from others. Figure 15
shows box plots of execution time for both the SOA approach and our approach.
The time performance of our approach is not bad compared with the SOA
approach.

TABLE X: The evaluation of execution time for both approaches.

| The SOA approach | Our approach
---|---|---
| Median | Average | Median | Average
PMD |  |  |  |
JClouds | 12.9 | 25.8 | 9.0 | 20.5
Kafka | 23.0 | 41.0 | 18.1 | 41.2
Guava | 18.0 | 50.5 | 16.2 | 55.2
Spring-boot | 21.2 | 29.5 | 17.7 | 24.3
Spotbugs |  |  |  |
JClouds | 10.9 | 30.4 | 9.3 | 24.0
Kafka | 45.6 | 60.4 | 33.9 | 119.2
Guava | 17.7 | 39.4 | 19.2 | 45.9
Spring-boot | 9.3 | 20.4 | 7.8 | 16.8

|
---|---
a | b
Figure 15: The boxplots of execution time of both approaches.

Till this end, to answer RQ3, we proposed a tracking approach combining three
improvement steps, which outperforms the SOA approach.

We proposed a better tracking approach. Compared with the SOA approach, our
approach can reduce FPs significantly (i.e., from 1,178 to 253) and yield a
precision of 90.0%.

TABLE XI: Six causes of false positives after using our approach in Kafka and
JClouds.

Cause | Number
---|---
1\. Class relocating or renaming | 23
2\. Method renaming | 2
3\. Attribute and variable renaming | 3
4\. Code shifting | 4
5\. Volatile class/method/variable names | 1
6\. Drastic and non-refactoring code changes | 119

###  5.2 Independent Analysis

RQ4. How accurate is our proposed approach for tracking the evolution of
static code warnings?

Apart from the SOA approach, we also take an independent evaluation of our
approach by a statistically significant (95%$\pm$5%) sample on commits for
each project to answer RQ4. Our approach is applied to collect resolved and
newly-introduced warnings on sampled commits. Then two authors manually check
them to determine whether a warning is a false positive or a true positive
with Cohen’s kappa coefficient of 0.82. Table XII shows the number of commits
we sampled. Note that there are many commits that have no resolved or newly-
introduced warning in this evaluation. In other words, the code changes of
many commits are too small to change the status of all static warnings.
Totally, we sampled 2,014 commits with 495 resolved and 837 newly-introduced
warnings by our improved approach. Table XIII shows the performance of our
approach. Overall, Our approach has a great performance with a precision of
91.8% (i.e., (430+793)/(495+837)), which means our approach can handle the
task of tracking static warnings very well.

TABLE XII:  The sampled commits for the independent evaluation.

| # Commits | # Resolved | # Newly-Introduced
---|---|---|---
PMD |  |  |
JClouds | 169 | 83 | 150
Kafka | 322 | 154 | 152
Guava | 322 | 40 | 109
Spring-boot | 194 | 6 | 17
Spotbugs |  |  |
JClouds | 169 | 41 | 107
Kakfa | 322 | 98 | 186
Guava | 322 | 53 | 106
Spring-boot | 194 | 20 | 10
Total | 2,014 | 495 | 731

TABLE XIII:  The performance of our approach in the independent evaluation.

| TP (Resolved) | # TP (Newly-Introduced)
---|---|---
PMD |  |
JClouds | 97.6% (81/83) | 96.0% (144/150)
Kafka | 78.6% (121/154) | 95.4% (145/152)
Guava | 75.0% (30/40) | 91.7% (100/109)
Spring-boot | 100% (6/6) | 100% (17/17)
Spotbugs |  |
JClouds | 97.6% (40/41) | 98.1% (105/107)
Kakfa | 82.7% (81/98) | 90.3% (168/186)
Guava | 96.2% (51/53) | 98.1% (104/106)
Spring-boot | 100% (20/20) | 100% (10/10)
Total | 86.9% (430/495) | 94.7% (793/837)

By conducting an independent evaluation of our approach, results show that our
approach achieves a precision of 91.8%, and it can handle the task of tracking
static warnings very well.

##  6 Threats to validity

In this section, we describe threats to external and internal validity.

###  6.1 External Validity

In this paper, we focus on tracking the static code warnings in Java projects.
Our study results may not be generalizable to projects in other languages. It
is expected that programs with similar evolution details to Java systems may
benefit from our study. We include two static bug detectors in our study,
whose representation of static code warnings are similar to some extent, i.e.,
the use of class/method/variable names and code ranges for matching purposes.
The improvement of our proposed approach may not be generalizable to a static
bug detector with a different set of metadata of the reported warnings.
However, most of the popular static bug detectors provide similar information.

Last, our crafted dataset for evaluating and improving the SOA approach is
based on four open-source projects. To increase the diversity, we analyzed a
reasonable number of commits in the four projects. In general, we find that
the evolution details that make the SOA approach malfunction are consistent in
our collected dataset. In the independent analysis, we sampled commits to
evaluate our approach, but many commits have no resolved and newly-introduced
warnings, which means that all warnings from these commits are labeled as
persistent warnings by our approach.

###  6.2 Internal Validity

When it comes to manually labeling the dataset, human errors are inevitable.
We tried to reduce human errors by having two people annotating the dataset
and resolving conflicts through discussions. Although our dataset covers
warnings with all three evolution statuses, we do not claim that our dataset
is representative in terms of following the distribution of the three
evolution statuses.

In particular, we set our criteria in crafting the dataset based on our
observations on the SOA approach (i.e., most of the established mappings are
correct) and also our priorities, which is to focus on the resolved and newly-
introduced warnings.

##  7 Related Work

Tracking the evolution of code issues. Tracking the evolution of code issues,
whether bugs, code smells, or static code warnings, is a central question in
many software quality studies. For example, the SZZ algorithm, which
identifies the origin of bug-introducing commits, is widely used in defect
prediction studies. Recent evaluations have uncovered many previously unknown
deficiencies in SZZ and inspired many researchers to work on improving SZZ.
For example, a study [17] empirically investigated how bug-fix changes and
bug-introducing changes of the SZZ are impacted by code refactoring. Then they
proposed refactoring-aware SZZ. Another study [18] proposed a framework to
provide a systematic evaluation of the data collected by SZZ. Palix et al.
conducted two studies on mining the code patterns. The first study [19]
presented a language-independent tool for mining and tracking code patterns
across the evolution of software by building graphs and computing statistics.
Their other study [20] combined the tool with AST for the detection of code
patterns across multiple versions. There is a study [21] that presented a tool
that combines static analysis with statistical bug models to detect which
commits are likely to contain risky codes, which provides more precise
information about a static warning. Dong-Jae et al. [22] conducted an
empirical study on the evolution of annotation changes and create a taxonomy
to uncover what annotation changes have and the motivation of annotation
changes. In addition, Felix et al. [23] proposed a tool to uncover method
histories with no pre-processing or whole-program analysis, which quickly
produces complete and accurate change histories for 90% methods.

Compared to tracking the defects, tracking the static code warnings has been
increasingly needed in recent research, yet rarely studied for its challenges
and insufficiencies. Spacoo et al. [24] propose to match warnings across
revisions using a combination of some basic information of each warning (e.g.,
warning type, class/method names) and allow inexact matching to some extent.
Their approach is not able to match warnings if they are moved to a different
class/method. Other diff-based approaches are used to identify which static
code warnings are resolved. In particular, Sunghun et al. [25] proposed an
algorithm to automatically identify bug-introducing changes with high accuracy
by combining the annotation graphs and ignoring non-semantic source code
changes. Results show that their algorithm outperforms the SZZ. Cathal and
Leon [9] conducted an empirical study to investigate the relation between
static warnings and actual faults. More recently, Avgustinov et al. [10]
proposed to combine several diff-based matching strategies to tackle this
problem, which we refer to as the state-of-the-art approach in our study for
evaluation and comparison.

However, a proper examination of the performance of the SOA approach is still
lacking in the field. In this study, we manually crafted a dataset of 1,416
static code warnings and their evolution status from two real-world open-
source systems and used it to identify potential for improvement in the SOA
approach.

Empirical studies on static bug detectors. Researchers have been working on
understanding and improving the utilization challenge of static bug detectors.
Johnson et al. [14] study the reasons that developers do not fully utilize
static bug detectors via conducting interviews with developers. Results show
developers cannot be satisfied with the current static analysis tools due to
the high rate of false positives. This study also provides some suggestions to
improve future static tools, e.g., improving the integration of the tool and
automatic fixes. Beller et al. [26] performed a large-scale study to
understand the current status of using static bug detectors in open-source
systems, e.g., whether or not used, and what running configurations are used.
Wang et al. [27] aimed to find whether there is a golden feature to indicate
actionable static warnings. Additionally, a survey was conducted by Muske et
al. [28] who reviewed static warnings handling studies as well as collected
and classified handling approaches.

Studies are also conducted to understand the nature of the issues found by
static bug detectors. Ayewah et al. [29] discuss the defects found by static
bug detectors at Google with regards to false positives, types of warnings
generated and their severity. Wedyan et al. [30] found that the issues by
static bug detectors are much more related to refactoring than defects. Habib
et al. [31] study the effectiveness of static bug detectors in terms of their
ability to find real defects and find that static bug detectors do find a non-
trivial portion of defects. An empirical study [32] evaluated the degree of
correlation between defects and warnings on the evolution of projects. Tomassi
et al. [33] examined static bug detectors by considering 320 real java bugs.
Their evaluation shows that static analyzers are not as effective in bug
detection, with only one bug detected by Spotbugs. Trautsch et al. [34]
conducted a longitudinal study on static analysis warning trends. They found
that the quality of code with regards to static warnings is improving, and the
long-term effects of static bug detectors are positive.

Our study focuses on a different aspect, which is to provide better ways to
track how static code warnings involve. Also, our study includes manual
analysis of a non-trivial dataset of static code warnings for the purpose of
improving the tracking precision, which is not covered by prior work.

Utilizing the tracking of static code warnings. Better tracking static code
warnings across development history provides many benefits. For example, there
has been an increasing interest in concluding fix patterns. Kui et al. [6]
mines the fix patterns on static code warnings from the software repository,
and the SOA approach was applied in their research. However, they did not
conduct an evaluation on the approach about how accurate the SOA approach
performs. A study [7] proposed a novel solution to automatically generate code
fixing patches for static code warnings via learning from fixing examples.
Another recent work [35] proposed a tool to help developers better utilize
static bug detectors on security issues by clustering based on common
preferred fix locations. This line of work can definitely benefit from an
improved tracking approach. In addition, there have been many works to
prioritize and recommend certain types of warnings based on development
history. Among them, a study [4] observed the static warnings in different
static bug detection tools and proposed a history-based warnings
prioritization to mining the fix cases recorded in the code change history.
Results show that over 90% of warnings remain in the projects or are removed
during code non-fix changes. Ted et al. [36] explored the ranking of warnings
from static bug detectors and presented a technique with a statistical model
to rank the static warnings that are most likely to be true positives. In
addition, another work, Quinn et al. [3], aimed at actionable static warnings,
and presented an actionable alert prediction model by creating feature vectors
based on code characteristics. In comparison, our work focuses on the status
changes of the static warnings in the evolution of the software projects. The
other work [37] statistically investigated the trend of static warnings over
the releases of OSS products and introduced a novel metric (e.g., the index of
programmers’ attention) to analyze the automatically pointed static warnings
and the actual attention that programmers paid to those static warning. Higo
et al. [38] proposed an approach based on static analysis across the
development history to identify project-specific bug patterns. A better
tracking mechanism will provide more accurate results for such work.

##  8 Conclusions

Tracking the evolution of static code warnings across software development
history becomes a vital question due to the increasing interest in further
utilizing static bug detectors by integrating them into developers’ workflow.
Also, such tracking is widely used in many downstream software engineering
tasks.

This study presents a careful investigation of the performance of the state-
of-the-art approach in tracking static code warnings. In particular, a dataset
of 3,452 static code warnings for four open-source projects and their
evolution status is crafted through manual labeling. Further, we summarize the
six causes that introduce false positives for the SOA approach. To address the
false positives, this paper presents an improved approach.

Last, we perform comparative and independent evaluations. Results show that
our improved approach outperforms the SOA approach significantly (i.e., from
66.9% to 90.0%) and 91.8% tracking precision in the independent evaluation.

## References

  * [1] C. Sadowski, E. Aftandilian, A. Eagle, L. Miller-Cushon, and C. Jaspan, “Lessons from building static analysis tools at google,” _Commun. ACM_ , vol. 61, no. 4, p. 58–66, Mar. 2018. [Online]. Available: https://doi.org/10.1145/3188720
  * [2] (2019) Spotbugs latest version. [Online]. Available: http://spotbugs.readthedocs.io
  * [3] Q. Hanam, L. Tan, R. Holmes, and P. Lam, “Finding patterns in static analysis alerts: Improving actionable alert ranking,” in _Proceedings of the 11th Working Conference on Mining Software Repositories_ , ser. MSR 2014\. New York, NY, USA: Association for Computing Machinery, 2014, p. 152–161. [Online]. Available: https://doi.org/10.1145/2597073.2597100
  * [4] S. Kim and M. D. Ernst, “Which warnings should i fix first?” in _Proceedings of the the 6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on The Foundations of Software Engineering_ , ser. ESEC-FSE ’07. New York, NY, USA: Association for Computing Machinery, 2007, p. 45–54. [Online]. Available: https://doi.org/10.1145/1287624.1287633
  * [5] C. Sadowski, J. van Gogh, C. Jaspan, E. Soederberg, and C. Winter, “Tricorder: Building a program analysis ecosystem,” in _International Conference on Software Engineering (ICSE)_ , 2015.
  * [6] K. Liu, D. Kim, T. F. Bissyande, S. Yoo, and Y. Le Traon, “Mining fix patterns for findbugs violations,” _IEEE Transactions on Software Engineering_ , pp. 1–1, 2018.
  * [7] R. Bavishi, H. Yoshida, and M. R. Prasad, “Phoenix: Automated data-driven synthesis of repairs for static analysis violations,” in _Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , ser. ESEC/FSE 2019. New York, NY, USA: Association for Computing Machinery, 2019, p. 613–624. [Online]. Available: https://doi.org/10.1145/3338906.3338952
  * [8] K. Liu, A. Koyuncu, D. Kim, and T. F. Bissyandé, “AVATAR: fixing semantic bugs with fix patterns of static analysis violations,” in _26th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2019, Hangzhou, China, February 24-27, 2019_ , X. Wang, D. Lo, and E. Shihab, Eds. IEEE, 2019, pp. 456–467. [Online]. Available: https://doi.org/10.1109/SANER.2019.8667970
  * [9] C. Boogerd and L. Moonen, “Evaluating the relation between coding standard violations and faults within and across software versions,” in _2009 6th IEEE International Working Conference on Mining Software Repositories_ , 2009, pp. 41–50.
  * [10] P. Avgustinov, A. I. Baars, A. S. Henriksen, G. Lavender, G. Menzel, O. de Moor, M. Schäfer, and J. Tibble, “Tracking static analysis violations over time to capture developer characteristics,” in _Proceedings of the 37th International Conference on Software Engineering - Volume 1_ , ser. ICSE ’15. IEEE Press, 2015, p. 437–447.
  * [11] H. W. Kuhn, “The hungarian method for the assignment problem,” _Naval research logistics quarterly_ , vol. 2, no. 1-2, pp. 83–97, 1955.
  * [12] J. W. Hunt and T. G. Szymanski, “A fast algorithm for computing longest common subsequences,” _Communications of the ACM_ , vol. 20, no. 5, pp. 350–353, 1977.
  * [13] E. W. Myers, “Ano (nd) difference algorithm and its variations,” _Algorithmica_ , vol. 1, no. 1-4, pp. 251–266, 1986.
  * [14] B. Johnson, Y. Song, E. Murphy-Hill, and R. Bowdidge, “Why don’t software developers use static analysis tools to find bugs?” in _Proceedings of the 2013 International Conference on Software Engineering_ , ser. ICSE ’13. IEEE Press, 2013, p. 672–681.
  * [15] (2020) The data set. [Online]. Available: https://drive.google.com/drive/folders/1OSYkm6QIfHO7z_zgdZdDRMbIqQOZ5IGW?usp=sharing
  * [16] N. Tsantalis, M. Mansouri, L. M. Eshkevari, D. Mazinanian, and D. Dig, “Accurate and efficient refactoring detection in commit history,” in _Proceedings of the 40th International Conference on Software Engineering_ , ser. ICSE ’18. New York, NY, USA: ACM, 2018, pp. 483–494. [Online]. Available: http://doi.acm.org/10.1145/3180155.3180206
  * [17] E. C. Neto, D. A. da Costa, and U. Kulesza, “The impact of refactoring changes on the szz algorithm: An empirical study,” in _2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER)_ , 2018, pp. 380–390.
  * [18] D. A. da Costa, S. McIntosh, W. Shang, U. Kulesza, R. Coelho, and A. E. Hassan, “A framework for evaluating the results of the szz approach for identifying bug-introducing changes,” _IEEE Transactions on Software Engineering_ , vol. 43, no. 7, pp. 641–657, 2017.
  * [19] N. Palix, J. Lawall, and G. Muller, “Tracking code patterns over multiple software versions with herodotos,” in _Proceedings of the 9th International Conference on Aspect-Oriented Software Development_ , 2010, pp. 169–180.
  * [20] N. Palix, J.-R. Falleri, and J. Lawall, “Improving pattern tracking with a language-aware tree differencing algorithm,” in _2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER)_. IEEE, 2015, pp. 43–52.
  * [21] L.-P. Querel and P. C. Rigby, “Warningsguru: Integrating statistical bug models with static analysis to provide timely and specific bug warnings,” in _Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2018, pp. 892–895.
  * [22] D. J. Kim, T.-H. P. Chen, and J. Yang, “The secret life of test smells-an empirical study on test smell evolution and maintenance,” _Empirical Software Engineering_ , vol. 26, no. 5, pp. 1–47, 2021.
  * [23] F. Grund, “Codeshovel: constructing robust source code history,” Ph.D. dissertation, University of British Columbia, 2019.
  * [24] J. Spacco, D. Hovemeyer, and W. Pugh, “Tracking defect warnings across versions,” 01 2006, pp. 133–136.
  * [25] S. Kim, T. Zimmermann, K. Pan, and E. J. Jr. Whitehead, “Automatic identification of bug-introducing changes,” in _21st IEEE/ACM International Conference on Automated Software Engineering (ASE’06)_ , 2006, pp. 81–90.
  * [26] M. Beller, R. Bholanath, S. McIntosh, and A. Zaidman, “Analyzing the state of static analysis: A large-scale evaluation in open source software,” in _2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER)_ , vol. 1, 2016, pp. 470–481.
  * [27] J. Wang, S. Wang, and Q. Wang, “Is there a” golden” feature set for static warning identification? an experimental evaluation,” in _Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement_ , 2018, pp. 1–10.
  * [28] T. Muske and A. Serebrenik, “Survey of approaches for handling static analysis alarms,” in _2016 IEEE 16th International Working Conference on Source Code Analysis and Manipulation (SCAM)_. IEEE, 2016, pp. 157–166.
  * [29] N. Ayewah, W. Pugh, J. D. Morgenthaler, J. Penix, and Y. Zhou, “Evaluating static analysis defect warnings on production software,” in _Proceedings of the 7th ACM SIGPLAN-SIGSOFT Workshop on Program Analysis for Software Tools and Engineering_ , ser. PASTE ’07. New York, NY, USA: Association for Computing Machinery, 2007, p. 1–8. [Online]. Available: https://doi.org/10.1145/1251535.1251536
  * [30] F. Wedyan, D. Alrmuny, and J. M. Bieman, “The effectiveness of automated static analysis tools for fault detection and refactoring prediction,” in _2009 International Conference on Software Testing Verification and Validation_ , 2009, pp. 141–150.
  * [31] A. Habib and M. Pradel, “How many of all bugs do we find? a study of static bug detectors,” in _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_ , ser. ASE 2018. New York, NY, USA: Association for Computing Machinery, 2018, p. 317–328. [Online]. Available: https://doi.org/10.1145/3238147.3238213
  * [32] M. Yan, X. Zhang, L. Xu, H. Hu, S. Sun, and X. Xia, “Revisiting the correlation between alerts and software defects: A case study on myfaces, camel, and cxf,” in _2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC)_ , vol. 1. IEEE, 2017, pp. 103–108.
  * [33] D. A. Tomassi, “Bugs in the wild: examining the effectiveness of static analyzers at finding real-world bugs,” in _Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2018, pp. 980–982.
  * [34] A. Trautsch, S. Herbold, and J. Grabowski, “A longitudinal study of static analysis warning evolution and the effects of pmd on software quality in apache open source projects,” _arXiv preprint arXiv:1912.02179_ , 2019.
  * [35] J. Yang, L. Tan, J. Peyton, and K. A Duer, “Towards better utilizing static application security testing,” in _2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)_ , 2019, pp. 51–60.
  * [36] T. Kremenek and D. Engler, “Z-ranking: Using statistical analysis to counter the impact of static analysis approximations,” in _Proceedings of the 10th International Conference on Static Analysis_ , ser. SAS’03. Berlin, Heidelberg: Springer-Verlag, 2003, p. 295–315.
  * [37] A. E. Burhandenny, H. Aman, and M. Kawahara, “Examination of coding violations focusing on their change patterns over releases,” in _2016 23rd Asia-Pacific Software Engineering Conference (APSEC)_. IEEE, 2016, pp. 121–128.
  * [38] Y. Higo, S. Hayashi, H. Hata, and M. Nagappan, “Ammonia: an approach for deriving project-specific bug patterns,” _Empirical Software Engineering_ , pp. 1–29, 2020.