# The Need for a Fine-grained approach in Just-in-Time Defect Prediction

Giuseppe Ng  0000-0002-1375-0236 De La Salle University2401 Taft
Ave.MalateManila1004<EMAIL_ADDRESS>and  Charibeth Cheng  De La Salle
University2401 Taft Ave.MalateManila1004<EMAIL_ADDRESS>
(2021)

###### Abstract.

With software system complexity leading to the rise of software defects,
research efforts have been done on techniques towards predicting software
defects and Just-in-time (JIT) defect prediction which predicts whether a code
change is defective. While using features to determine potentially defective
code change, inspection effort is still significant. As code change can impact
several files, we investigate an open source project to identify potential
gaps with features in JIT perspective. In addition, with a lack of publicly
available JIT dataset that link the features with actual commits, we also
present a new dataset that can be utilized in JIT and semantic analysis.

empirical software engineering, software metrics, defect prediction, just-in-
time prediction, software defect prediction

††copyright: rightsretained††doi: ††isbn: ††conference: Unpublished;
2021††journalyear: ;††copyright: none††ccs: Software and its engineering
Software defect analysis††ccs: Computing methodologies Machine learning

##  1\. Introduction

Risks in software defect have increased over time (Ge2018, ; Huang2018, ;
Ghose2018, ; Li2018, ; Nam2018, ; Hoang2019, ). For instance, the Heartbleed
bug was due to an OpenSSL vulnerability (Hoang2019, ). This vulnerability
affected servers that utilized the library for secure connections such as
Apache and Nginx, allowing an attacker to remotely read vulnerable server
protected memory and potentially retrieve cryptographic information. The
vulnerability affected an estimated 24-55% of popular HTTPS websites
(Durumeric2014, ). Such risks had lead towards the development of Software
Defect Prediction (SDP) that predicts whether specific files in a software
project are buggy or clean (Ghose2018, ; Li2018, ; Dam2019, ; Son2019, ;
Qiao2019, ; Hoang2019, ; Hosseini2019, ). Finding software defects remains to
be extremely challenging as tools could yield false positives predictions
(Ghose2018, ; Rahman2020, ). In addition, this technique, however, is coarse
granularity level of prediction and focuses on files or packages that are
defective (Yang2015, ; Albahli2019, ).

Mockus and Weiss proposed changing the focus from files and packages to
submitted code changes into VCS. This technique is referred to as Just-in-Time
Defect Prediction (JIT) (Mockus2000, ; Huang2017, ;
82ac60c1c56b4be4a6220de0dfad6039, ). Several works were done in the field of
JIT and the crafting of features towards better performance (Kamei2013, ;
Yang2016, ; Fu2017, ; Qiao2019, ; Hoang2019, ).

When developers submit code changes to the repository, the developer submits a
message regarding the change. The repository then generates the code diff
(Hoang2019, ). A typical submitted code change can be seen in figure 1.

Figure 1.  Sample code commit.

The sample code commit on figure 1 shows two basic sections. The first section
pertains to information about the commit, such as author, the timestamp, and a
brief description of the submission. The other section shows the changes in
the code compared to the previous version. These are denoted by the file name
and path, the line numbers, the ’+’ symbol for line additions and ’-’ symbol
for line deletions (Hoang2019, ).

Determining defective classes or files has the downside of lack of
explainability of its defectiveness, leading to the inspection of code change
as an alternate approach (Yang2016, ; Huang2018, ; Dam2019, ). Building
datasets is a major aspect in machine learning approaches (Ozaknc2018, ;
Catal2009, ; Rahman2020, ).

SDP datasets are based on analysis of Version Control Systems (VCS) such as
GIT, SVN, or CVS, with defect information extracted from bug ticketing system
(Li2018, ; Hosseini2019, ; Liu2019, ; Qiao2019, ). Various traditional SDP
datasets were made available online to aid in empirical software engineering
and software defects prediction. These datasets were created using open source
and closed source projects (Shepperd2013, ; Lamkanfi2013, ; Kamei2013, ;
msr-2019-swh, ). The ReLink dataset uses the following projects: (1) Apache,
(2) Safe, and (3) ZXing to predict whether a source code file is defective
(Wu2011, ). The AEEEM dataset is composed of the following open source
projects: (1) Equinox (EQ), (2) JDT Core, (3) Lucene (LC), (4) Mylyn (ML), and
(5) PDE UI. This dataset is intended to predict whether a class is defective
or not (DAmbros2012, ). MORPH utilized several open source projects such as:
ant-1.3, arc, camel-1.0, poi-1.5, redaktor, skarbonka, tomcat, velocity-1.4,
xalan-2.4, and xerces-1.2 to predict whether a class is defective (Peters2012,
). Shepperd et al. released the NASA dataset that utilized closed source
projects inside NASA for traditional software defect prediction (Shepperd2013,
). SOFTLAB also used closed source projects for their dataset(Turhan2009, ;
Nam2018, ). The NASA and SOFTLAB datasets predicts defects on a function level
(Shepperd2013, ; Turhan2009, ; Nam2018, ).

When creating a dataset, in general, once the corpus has been identified, the
predictor metrics need to be calculated (Canfora2015, ). The selection of
traditional software metrics depends largely on the context and purpose
defined (Voas2017, ). In terms of context of traditional SDP, metrics
describing a software’s size, complexity, and quality are critical
(Kan:2002:MMS:559784, ). CK metrics (Chidamber1994, ) and Lines of Code (LOC)
are possible metrics that give different insight on the source code data
(Li2018, ; Hosseini2019, ). CK metrics measures the complexity such as number
of subclasses, depth of inheritance, number of methods to determine level of
complexity (Chidamber1994, ; Li2018, ). Nagappan and Bell showed that code
churn, the measurement of a file changes over time, could be used as a feature
(Nagappan2005, ). Scattered changes in the code could lead to defective
software (Hassan2009, ).

Process metrics could be used as well in determining defective code
(Moser2008, ). Process metrics pertains to the use of information from source
tracking repositories such as version control and issue tracking systems to
capture development changes over time. These can be classified as: (1) code
change, (2) developer-related metrics such as number of engineers, (3) code
dependency, (4) organization-metrics such as organization volatility, (5)
other process metrics (Li2018, ).

The work by (Kamei2013, ) builds from the findings of early works and provided
a dataset that included several projects for JIT defect prediction. This
dataset has been used in several JIT works (Kamei2013, ; Yang2016, ; Fu2017, ;
Qiao2019, ; Hoang2019, ). The projects utilized for this are Bugzilla (BUG),
Columba (COL), Eclipse JDT (JDT), Eclipse Platform (PLA), Mozilla (MOZ), and
PostgreSQL (POS). The Bugzilla and Mozilla datasets were created from data
provided by the Mining Software Repository 2007
Challenge111http://2007.msrconf.org/challenge/. The Eclipse JDT and Platform
data were created from the MSR 2008 Challenge
data222http://2008.msrconf.org/challenge/. Columba and Postgres were from
their respective CVS repositories (Kamei2013, ).

As a change-level defect prediction or JIT dataset, the granularity here is
neither class, file, nor function. Instead, the features extracted were based
on code change or commits in a code repository as explained in figure 1. The
features extracted are grouped into several dimensions: (1) Diffusion, (2)
Size of the change, (3) Fix metric, (4) Historical information regarding the
change, and (5) Experience of developers (Kamei2013, ; Fu2017, ). The features
range from number of modified systems, files and directories, entropy, lines
added and deleted, etc (Kamei2013, ; Lewis2013, ; Yang2016, ; Qiao2019, ).
Labelling a commit as bug inducing is done through the SZZ algorithm
(Sliwerski2005, ; Kamei2013, ). In the case of Columba and PostgreSQL, since
the defect identifiers were not included in the changelog or commit messages,
an approximate SZZ approach was done where keywords such as ”fixed” or ”bug”
would indicate a bug fix for a defect inducing commit (Kamei2013, ).

A summary of the software metrics used for JIT are the following (Kamei2013, ;
Yang2016, ; Fu2017, ; Chen2018, ):

  1. (1)

Number of modified subsystems;

  2. (2)

Number of modified directories;

  3. (3)

Number of modified files;

  4. (4)

Entropy or the distribution of the modified code across each file;

  5. (5)

Lines added;

  6. (6)

Lines removed;

  7. (7)

Lines of code in a file before the current change;

  8. (8)

Whether the change is a bug fix;

  9. (9)

Number of developers that changed the modified file;

  10. (10)

The average time interval between the last and the current change;

  11. (11)

The number of unique changes to the modified file;

  12. (12)

Developer experience in terms of number of changes;

  13. (13)

Recent developer experience; and

  14. (14)

Developer experience on the subsystem (Kamei2013, ).

The work by (Hoang2019, ) also provided a dataset consisting of two open
source projects: (1) Qt333http://www.qt.io/, and (2) Openstack
444http://www.openstack.org/. These datasets were cleaned by McIntosh and
Kamei (Hoang2019, ; 10.1145/3180155.3182514, ). This dataset contained commit
ids unlike other datasets that made it possible to identify defect inducing
commits and look at the actual change log of such (Hoang2019, ). To our
knowledge, there has been no other publicly available dataset that has this
information. As such, it is clear that many JIT approaches rely on feature
metrics such as (Kamei2013, ; Yang2016, ; Fu2017, ; Chen2018, ; Qiao2019, ;
Pascarella2019, ).

As conventional JIT approaches rely on feature metrics for defect prediction,
we investigate whether code change feature metrics are adequate. In addition,
we analyze the nature of code changes to show that inspection effort is still
significant.

##  2\. Background

Software defects are as deficiencies in elements of a software because they do
not meet requirements and specifications. Software faults are defined as the
causes of the defect while software failure is defined as the end result of
the defect (Bourque2014, ; Ge2018, ).

In terms of software quality, several factors are considered with regards to
testing requirements beyond the functional requirements of software. Some of
these are: domain specific, possibly based on external and internal components
integrated into the system; and intended expertise of the end user
(Bourque2014, ).

Factors that lead to software defects are: human-related, lack of
communication, insufficient design and testing, unrealistic timeframe, and
defects from third party components, etc. (Ge2018, ). A defect taxonomy can be
used to effectively group different kinds of software defects. These are
(Felderer2015, ):

  1. (1)

requirements;

  2. (2)

features and functionalities;

  3. (3)

structural defects;

  4. (4)

data;

  5. (5)

implementation and coding;

  6. (6)

integration;

  7. (7)

system software architecture;

  8. (8)

test definition and execution; and

  9. (9)

unclassified defects (Felderer2015, ).

###  2.1. Software Defect Prediction

The goal of software defect prediction, then, is to identify the faults in the
code and code modifications, estimate the number of defects and identify areas
of optimization(Hosseini2019, ). This technique falls under data-driven
software engineering (Upadhyaya2018, ).

The learning models in SDP can generally be classified under supervised or
non-supervised (Catal2009, ; Ge2018, ; Li2018, ; Ozaknc2018, ; Hosseini2019,
). The model is expected to learn from a corpus of data that encompasses code
archive (in classes or files form and source trees such as GIT, SVN, or CVS),
and defect information (in a bug ticketing system such as Bugzilla or Jira)
(Li2018, ; Hosseini2019, ; Rahman2020, ). Linking source commit information
and bug tickets may also be necessary. Granularity or instance of defect
prediction can be by package, class, file, or function basis (Qiao2019, ;
Hosseini2019, ).

Training data can be from the same project or referred to as ’Within Project
Defect Prediction’ (WPDP) or from other projects, ’Cross Project Defect
Prediction’ (CPDP) (Hosseini2019, ). Traditional defect prediction models used
WPDP project where older versions of the software are used to predict defects
on newer versions (Li2019, ).

The work by (Canfora2015, ) outlines the basic workflow in designing a
prediction model. This can be seen in figure 2.

Figure 2.  Generic Flow of Designing a Default Prediction Model

The traditional process in building the model involves extracting the
features, and building the ML model that will be trained with this model.
Given the instance whether it’s a package, a file, a class, or a method, the
ML model should be able to classify the instance as defective (Yang2015, ;
Albahli2019, ; Qiao2019, ; Rahman2020, ).

###  2.2. Just-in-Time Defect Prediction

Where traditional SDP have been focused on the predicting the coarse-grained
instances as defective, others have expanded SDP to analyzing code changes
instead of files to identify whether the change is defective. This is referred
to as Just-in-Time (JIT) defect prediction (Hata2012, ; Fukushima2014, ).
Figure 3 illustrates a typical code
repository555https://opendev.org/openstack/swift/commits/branch/master where
all the commits are stored.

Figure 3.  Repository of Commits from Openstack.

For every code change in the code repository, features are extracted that are
then used to train the model (Kamei2013, ; Fu2017, ). Figure 4 shows the
approach of a JIT defect prediction (Qiao2019, ).

Figure 4.  Neural Network Architecture, from (Qiao2019, ).

Unlike traditional SDP, the granularity in JIT is actual code changes,
achieving more fine-grained approach (Cho2018, ).

As discussed by Kamei (Kamei2013, ), the diffusion aspect is represented by
NS, ND, NF and Entropy. Highly distributed changes are more complex and are
more likely to be defective. Number of modified subsystems, modified
directories, and modified files using the root directory name, directory
names, and file names respectively (Kamei2013, ).

Entropy is calculated using a similar measure to Hassan (Hassan2009, ;
Kamei2013, ). It is the measurement over time how distributed the changes are
in the code base. Changes in one file is simpler than one that impacts many
different files. Time period used for the dataset is 2 weeks (Kamei2013, ).
Entropy is defined as (Kamei2013, ):

(1) $H(P)=-\sum_{k=1}^{n}(p_{k}*log_{2}p_{k})$

where probabilities $p_{k}\geq 0,\forall k\in 1,2,...,n$, $n$ is the number of
files in the change, $P$ is a set of $p_{k}$, where $p_{k}$ is the proportion
that $file_{k}$ is modified in a change and $(\sum_{k=1}^{n}p_{k})=1$
(Kamei2013, ). Figure 5 illustrates entropy in a sample scenario (DAmbros2010,
).

Figure 5. Example of Entropy (DAmbros2010, ).

For time interval $t1$, there are four changes and the probabilities are
$P_{A}=\frac{2}{4},P_{B}=\frac{1}{4},P_{C}=\frac{1}{4}$. The entropy in $t1$
is calculated as $H=-(0.5*log_{2}0.5+0.25*log_{2}0.25+0.25*log_{2}0.25)=1$.
For time interval $t2$, entropy is 1.378 (DAmbros2010, ).

Lines added (LA), lines deleted (LD), and lines of code before the current
change (LT) were calculated directly from the source code. LA and LD were
normalized by dividing by LT while LT is normalized by dividing by NF since
these two metrics have high correlation (Kamei2013, ). NUC is calculated by
counting the number of commits that caused changes to specific files. This
metric was also normalized by dividing by NF due to their correlation with the
NF feature (Kamei2013, ). This technique follows findings made by Nagappan and
Ball (Nagappan2005, ).

Age is calculated as the average of time interval between the last and current
change of files (Fu2017, ; Kamei2013, ). Should Files A, B and C were modified
3 days ago, 5 days ago, and 4 days ago respectively, Age is calculated as 4
(i.e., $\frac{3+5+4}{3}$). More detailed explanation of the features of the
dataset can be found in (Kamei2013, ).

Experience is the number of commits done by the author prior to the current
commit while REXP is the total experience of the developer while factoring the
recency of the change. Using $\frac{1}{n+1}$ where $n$ represents the year, a
developer who made 5 changes 3 years ago would be represented as
$\frac{5}{3}$. REXP is then computed as the summation of all such factored EXP
values (Kamei2013, ; Borg2019, ).

###  2.3. SZZ Algorithm

Labeling defective changes is done through the SZZ algorithm (Sliwerski2005,
). Two key concepts in labeling is defining a bug inducing change and a bug
fixing change. Bug inducing change is defined as a change A that caused for a
change B to be submitted to rectify an issue introduced by change A. Change B
is referred to as the bug fixing change (Kiehn2019, ). An open source
implementation was created by (Borg2019, ). With regards to the Columba and
Postgres dataset, an approximated SZZ approach was taken due to the defect ids
not referenced in the commit messages (Kamei2013, ).

The overall workflow of the algorithm is as follows (Borg2019, ):

  1. (1)

Retrieve all closed bug reports from a ticket tracker of the system

  2. (2)

Link the bug report to the bug fixing commits of the code repository

  3. (3)

Identify the bug inducing commits through the bug fix commit information.

Phase 1 involves the linking of the bug reports to bug fixing commits while
Phase 2 is the process of identifying the bug inducing commit. Phase 1 largely
involves matching the commit to the closed ticket IDs (Borg2019, ). In the
even that there is no ticket to match, words such as fix in the commit message
are assumed to be bug fixes (Kamei2013, ; Borg2019, ). Phase 2 is illustrated
in Figure 6 (Borg2019, ).

Figure 6.  SZZ Phases (Borg2019, ).

The tool will identify all commits that happened for specific lines using git
blame. Each candidate is then ruled out by checking the date of the commit. If
the change is newer but it was only a partial fix, then it is included as a
bug inducing commit (Borg2019, ).

The end result of the algorithm is the list of submitted commits fixing bugs
and potential list of changes as bug inducing. Using the SZZ tool provided by
(Borg2019, ), pairs of bug inducing commits and bug fix commits can be
identified from a dump of the bug tracking and report system and the git
repository. The commits that are not labelled as defective from SZZ can be
used to build the non-defective labelled dataset (Borg2019, ).

Subtle semantic differences would be difficult to identify from calculated
metrics (Ghose2018, ). In JIT, there are limitations in feature metrics from
code changes as two different code changes can lead to the same code metrics,
thereby highlighting the need to extract features from semantic code
(Hoang2019, ).

###  2.4. Semantic Representation of Code Change

Although extraction of features from source code has been done for various
purposes including SDP, the study of semantic features in JIT context is
under-represented. (Hoang2019, ) explored extracting features from semantic
information of a code change. In their work, the code change, represented by
the set of lines added and deleted, and the commit message were used in
feature extraction. Figure 7 illustrates their approach (Hoang2019, ).

Figure 7.  Feature extraction of commit diff using CNN, from (Hoang2019, ).

Their work shows that utilizing the semantic information from code changes can
lead towards JIT (Hoang2019, ).

###  2.5. Comparing Coarse-grained and Fine-grained JIT

A commit or code change can either be clean, completely bug inducing, or
partially bug inducing. Traditional SDP where the model used for prediction is
based on previous release data. In this scenario, the predictor can
incorrectly label a file as defective. It can also completely miss defect
inducing files (Pascarella2019, ). Such an approach lacks feedback that
developers need when doing code review (Kamei2013, ; Lewis2013, ;
Pascarella2019, ). Figure 8 illustrates how existing approaches are not
adequate in identifying defective files within commits (Pascarella2019, ).

Figure 8.  Partial defect and existing approaches, from (Pascarella2019, ).

Regarding JIT defect prediction, coarse-grained JIT can predict whether an
incoming code change is defect inducing. This approach, unlike the traditional
SDP, can catch defect inducing changes as soon as it is made, thereby giving
faster feedback to developers. However, the code inspection effort is still
significant (Lewis2013, ; Pascarella2019, ).

A submitted code change can modify more than one file and can lead to
significant inspection cost by developers to determine what particular area of
the code change was defective. In addition, not all the files in the code
change are necessarily defect inducing, and yet, such approaches could cause
defect-free files in the commit to be defect inducing (Pascarella2019, ).

Hence, a finer grained approach towards JIT that is able to analyze changes
within a code change could reduce the inspection cost. Ideally, this would
entail identifying actual files that are defective and even the area within
that file in the code change to be inspected. Figure 9 illustrates the
difference of perspective of coarse-grained and fine-grained JIT.

Figure 9.  Difference of Coarse vs Fine-grained

As shown, the coarse-grained JIT can label a change as defective but in doing
so, incorrectly labels a file as defective when it is in fact clean. A finer-
grained JIT can correctly mark the actual file within the code change as
defective and leave the clean file as such. Efforts to improve fine grained
defect prediction echo the findings by (Hata2012, ; Pascarella2018, ;
Liang2019, ; Pascarella2019, ).

##  3\. Methodology

To determine whether feature metrics can determine the nature of defective
code change, our first step is to create a dataset using an open source
project as code repository and code change data are available. To build the
dataset, we downloaded the source code repository and exported the defect data
from an open source project. This is consistent with the work done by
(Kamei2013, ; Borg2019, ). The inclusion of the commit id in the dataset is
consistent with the work done by (Hoang2019, ). We then investigated the
marked bug inducing commits of the project. Specifically, we examine the
nature of code changes and report our observations in relation to the
calculated feature metrics.

To see the predictive performance of the dataset, we use a Within-Project
Defect prediction approach, a practice that Google used with known existing
models (Kim2007, ; Rahman2011, ; Rahman2012, ).

To validate our findings, we referred to the dataset provided by (Hoang2019, )
to see if the bug inducing commits in the dataset are consistent.

##  4\. Creating and Labelling a Dataset

We downloaded the git source repository of the Trac
project666https://github.com/edgewall/trac and also obtained an export of
their list of bugs777https://trac.edgewall.org/query. The export covers
tickets from August 24, 2003 to January 13, 2020.

Using the SZZ tool provided by (Borg2019, ), pairs of bug inducing commits and
bug fix commits can be identified from a dump of the bug tracking and report
system and the git repository. The software metrics from the project was
calculated using the scripts provided. The commits are labelled accordingly
using the SZZ tool by (Borg2019, ).

There are 12,840 commits processed in the repository. The SZZ tool labelled
1681 commits as bug inducing. 1552 bug fixes were found by the tool. The
information is shown in table 1.

Table 1. Summary of the information on the Trac dataset |  |  | %Defect
---|---|---|---
|  |  | induced
Project | Period | #change | change
Trac | 08/2003-01/2020 | 12840 | 13.09%

We used cloc software888http://cloc.sourceforge.net to calculate the lines of
code (LOC) of the project. This is used to determine the size of the project
being processed. Table 2 illustrates the statistics of the code.

Table 2. CLOC Summary of the Trac Project Language | files | blank | comment | code
---|---|---|---|---
Python | 323 | 16624 | 23054 | 74407
HTML | 73 | 1126 | 32 | 9984
CSS | 17 | 294 | 185 | 4206
Javascript | 25 | 261 | 327 | 2346
DTD | 1 | 200 | 214 | 564
make | 1 | 201 | 93 | 557
PowerShell | 1 | 98 | 114 | 209
YAML | 2 | 7 | 5 | 207
CoffeeScript | 2 | 22 | 50 | 93
DOS Batch | 1 | 16 | 0 | 79
Bourne Shell | 1 | 13 | 179 | 59
Bourne Again Shell | 2 | 4 | 16 | 37
XML | 1 | 0 | 0 | 12
Total | 450 | 18866 | 24269 | 92760

With this information we had managed to extract the features shown on table 3.

Table 3. Features of the Trac dataset Metric | Description
---|---
Commit | The commit hash in the
| repository
NS | Number of modified subsystems
ND | Number of modified directories
Entropy | Distribution of the modified
| code across each file
Lines_of_Code_Added | Lines added
Lines_of_Code_Deleted | Lines deleted
Files_Churned | Number of modified files
Lines_of_Code_Old | Lines of code in a file before the
| current change
Purpose (Fix) | Whether the change is a bug fix
Number_of_Authors | Number of developers
(NDEV) | that changed the modified file
AGE | The average time interval
| between the last and the
| current change
Number of | The number of unique
Unique Changes | changes to the modified file
EXP | Developer experience in terms of
| number of changes
REXP | Recent developer experience
Defective | Whether this is bug inducing
| or not

There are some deviations from the feature extraction methods described in
(Kamei2013, ). We highlight the notable differences in the calculations in the
tool provided in (Borg2019, ).

Entropy, Experience, and REXP are calculated similarly with Kamei (Kamei2013,
). Lines of code added and deleted are calculated by extracting the number of
lines inserted and deleted respectively and dividing them by the total lines
of code of the new file (Borg2019, ). Lines of code of the old file is not
normalized unlike LT feature in Kamei’s dataset (Kamei2013, ; Borg2019, ).

Age is calculated as an average of the time difference of every file modified
by the current commit. Number of unique changes is calculated as the number of
commits for every file modified by the commit (Borg2019, ).

For the normalization on the dataset, we employ the min-max standardization
from (Qiao2019, ):

(2) $Norm(x)=\frac{x-min(x)}{max(x)-min(x)}$

This has been a commonly used technique to treat datasets in other works
(Wang2016, ; Qiao2019, ).

###  4.1. The Trac Dataset

In this section, we report on interesting observations that we had seen from
the dataset.

Commit ID. Our key inclusion in this dataset is the Commit ID for every
submitted code change of the dataset. With this, it is possible to look into
each and every change that was marked as bug inducing for further study. Many
JIT datasets do not include this, making the change level semantic learning on
change-level defect prediction much more challenging.

Lines of Code Added and Deleted. We compare the lines of code added and
deleted and found an interesting difference between the maximum lines of code
added and deleted. The maximum lines of code added is 1175 while the lines of
code removed is 468343. In analyzing the increase of lines of code removed, we
noted that there is a commit that removed almost all the files in the
repository as a repository clean up. The presence of this submission is an
outlier in the repository.

Files Churned. We observed that since the tool already normalizes the number
of modified files by the total number of files, the values are already in the
range of 0 to 1. There are 39 instances when the values are not normalized due
to potential divide by 0 issues that could happen when the submitted code
change cleans up the repository. The minimum and maximum values of the feature
are 0 and 739 respectively with an average of 0.1175 and a standard deviation
of 6.5236.

Lines of Code before the Change (LT). The feature is not normalized and has an
average of 141.7006 with a standard deviation of 326.9681. We found that over
time, there is a general increasing trend in this value as shown in figure 10.

Figure 10.  Lines of Code before the Change Normalized.

Number of Modified Subsystems. Upon applying the normalization step, we
observed that over time, there is an increasing trend in the values. Figure 11
illustrates this.

Figure 11.  Number of Modified Subsystems Normalized.

This would make sense that over time, the system complexity would grow as more
features are added in. The minimum and maximum values of the feature are 0 and
198 respectively with an average of 0.1175 and a standard deviation of
33.9173.

Number of Modified Directories. The minimum and maximum values of the feature
are 0 and 8 respectively with an average of 1.5326 and a standard deviation of
1.5488. These values are raw and not standardized.

Purpose. In identifying whether a code change is a bug fix, the feature is
represented as a boolean value. In the entire 12,840 commits, the tool marked
a total of 2318 bug fixes representing 18.05% of all code commits to the
repository.

Entropy. In terms of entropy, we found that the average entropy is 1.2246 with
a standard deviation of 2.0082.

Experience and Recent Experience. We found that the average is 1145.5416 with
a standard deviation of 1014.5913. We found that the huge code clean up
influenced the values of the feature by plotting the chart that is shown in
figure 12.

Figure 12.  Experience Normalized.

Analyzing the chart, shows an increasing trend on experience over time.
However, because the code clean up removes many files, this also eliminates
the experience that is tracked by looking at those specific files.

With the recent experience chart, the trend of the feature shows that the huge
dip from the clean up does not influence the feature.

Figure 13.  Recent Experience Normalized.

Number of Unique Changes (NUC). The feature has an average of 2.0177 and a
standard deviation of 3.9729. Interestingly, the maximum value is 180. This
shows that this is an extreme outlier and suggests that standardization
techniques may not be advisable.

Defective. This is the label of the dataset and marks a particular code change
as defective.

Figures 14 and 15 show the histogram and the scatter matrix visualization of
the dataset.

Figure 14.  Histogram of the Trac Dataset Features. Figure 15.  Scatter Matrix
of the Trac Dataset Features.

The statistics of the features of this dataset are summarized in table 4.

Table 4. Statistics of the Trac Dataset Feature | Min-Max | Ave | Std-Dev
---|---|---|---
LA | 0-1175 | 6.8046 | 20.4584
LD | 0-468343 | 40.6886 | 4133.1486
Files Churned | 0-739 | 0.1175 | 6.5236
LT | 0-1429 | 141.7008 | 326.9681
NS | 0-198 | 0.1175 | 33.9173
ND | 0-8 | 1.5326 | 1.5488
Entropy | 0-8.4429 | 1.2246 | 2.0082
Exp | 0-3491 | 1145.5415 | 1014.5912
NUC | 0-180 | 2.0177 | 3.9729

In training an actual model to see its potential, we applied a 70-30 split of
seen and unseen data into a 9 layer architecture of ReLU units and a sigmoid
output. Our data preprocessing techniques were normalization and
undersampling. We utilized Smooth L1 Loss as our loss function with 3500 epoch
training. An Adam optimizer with a learning rate of 0.001 was used.

By repetitively removing features from the training and checking the loss and
recall, we identified that lines of code added, lines of code deleted,
modified subsystems, modified subdirectories, experience, rexp, age, number
unique changes may be a potent starting set of features for training.

The selection of modified subsystems, age, and number unique changes as good
features is in agreement with the results from (Kamei2013, ).

The use of experience metrics such as experience and rexp is consistent with
the findings from (Matsumoto2010, ; Mockus2000, ; Pascarella2019, ) where
experience-based metrics are good features.

Interestingly, our tests show that lines of code added, lines of code deleted,
and modified subdirectories may be included as well in the list of good
features. Our tests managed to attain a recall of 64.15% with a loss of 0.038
in a WPDP approach.

###  4.2. Observations of the Trac repository code changes

We would like to make note that in this study, we do not claim to be experts
of the Trac source project. The analysis below is purely based on the
information we had extracted from the repository.

In observing bug inducing changes and bug fixes, we make note that bug fixes
tend to modify lines that were added by bug inducing changes. In doing code
comparisons of defective code, figure 16 shows a defect inducing code change
from the Trac repository999Commit hash
f1ab6093ddcbd5043fbfaafca5b5e06bde87313b.

Figure 16.  Relevant Snippet of Defect Inducing Change.

In contrast, a sample bug fix code change101010Commit hash
936959307a5421c43ffa2b6024801f80c8b720c8 is shown in figure 17.

Figure 17.  Relevant Snippet of the Bug Fix.

To identify defect inducing lines, we compare the bug inducing change and the
bug repair change for lines that were modified. Lines that were modified from
the bug inducing change to the bug fix are defect inducing, while lines that
were untouched are not defect inducing. This is consistent with the works by
(Yan2020, ; Wattanakriengkrai2020, ).

The defect inducing change here only changes 1 file with 14 line additions and
13 line deletions and is a bug fix to another reported bug. This code change
was submitted October 8, 2005. The bug fix for this was submitted May 12,
2010, 10 years after the original code change was accepted in the system.
Clearly, this defect inducing change had already been released into actual
stable versions of Trac.

Notice also that figure 17 also deleted lines 183 and 184, replacing them with
186, and 187. The lines were also in figure 16, however, the change itself did
not introduce these lines. Instead, it is introduced by another bug
fix111111Commit e955cc298b14d4c8585186dd50d2db31c6327444 submitted to the
repository. This implies that these lines were also defect-inducing from a
different code change in the repository. The defect itself may have been
caused by shifting context over time, possibly by code or by the dependent
library environment or extrinsic bugs as defined by (Rodriguez-Perez2020, ).
We make note that a single bug fix change can be linked to several defect
inducing code changes. It is possible also that defect was introduced by the
perceived understanding of a developer of the code as pointed out by
(Fakhoury2020, ).

Refactor code changes 121212Commit 1fb1d3b38d32fffa50d3ff70946933f890d1d9bd
are among the defect inducing changes. This is shown in figure 18.

Figure 18.  Refactor Code Change in Trac Repository.

In such works, large code changes and new files are included. A total of 205
lines from the new file was added in the shown code change. Should this file
be the defect inducing file in the code change, there is still a significant
amount of inspection effort to be done by the developer.

In analyzing the code change, we compare the feature metrics to the calculated
average from our previous table 4. Table 5 shows the comparison.

Table 5. Comparison of the Code Change to Average Feature | Code Change | Ave
---|---|---
LA | 4.67 | 6.8046
LD | 4.33 | 40.6886
Files Churned | 0.003 | 0.1175
LT | 3 | 141.7008
NS | 1 | 0.1175
ND | 1 | 1.5326
Entropy | 0 | 1.2246
Exp | 812 | 1145.5415
NUC | 1 | 2.0177

This shows that the calculated feature metric for LA and LD being below the
average. Making note that Kamei et al. (Kamei2013, ) made mention that a large
code changes (e.g. lines added (LA), deleted (LD), and entropy) could imply
defective code change, the defect inducing code change is actually below the
average for the Trac dataset.

With the large code base of this project at 92760 LOC, locating defective code
requires significant inspection effort (Qiao2019, ). Identifying specific
lines and repairing them would need bug reports. The presence of such error
means that the defect had already manifested to actual end users (Qiu2020, ;
Yan2020, ).

As code changes can modify many files, we examined the number of files
modified per commit and found that on average, bug inducing commits modify
7.97 files. On average, the number of different file types modified by such
commits is 1.72 file types. From the analysis of the modified files, there are
39 known files that have been modified. Some files do not have file extensions
and have been marked as ’no extension’ instead. We have listed down the top
file extensions modified on average on table 6.

Table 6. File extensions modified in Bug inducing changes in the Trac repository File Extension | Average #changed
---|---
py | 5.08
html | 0.97
no extension | 0.43
cs | 0.39
css | 0.55
po | 0.19
js | 0.12
pot | 0.10
txt | 0.09
png | 0.09

Noting that there are potentially multiple files impacted by a specific
change, identifying a commit as bug inducing does not identify which files are
defective. The developer who utilizes JIT will still need to study the commit
to determine what about the code change is causing software defects. Noting
that in figure 19 a particular code change can impact multiple file types such
as .cs and .py, diagnosing the bug can be complex and time-consuming. As noted
by (Kamei2013, ), inspecting possible defective files would take a
considerable amount of time. Even if we utilize Effort-Aware works as
suggested by (Kamei2013, ; Qiao2019, ), if there is a need for the developer
to study a large change, he would have to devote time and effort to understand
the context of the code to determine how to fix the bug. Coarse grained
approaches hinder their use from a practical standpoint (Pascarella2019, ).

Our results show that not only does changes in commits modify multiple files,
but also changes several different file types. With our dataset including the
commit ID, we are able to look into the commits marked bug inducing by the SZZ
tool unlike many other available JIT datasets. One such example131313Commit
hash 50fc9f2a73c8e4c6bdfd92d48b80abd6e945a400 found in our dataset is shown in
figure 19. The diff below illustrates two of the 46 files changed in the
commit.

Figure 19.  Sample code commit showing multiple different file types modified.

In analyzing other code changes, we found code changes that had large number
of lines inserted and deleted. However, the code change was apparently simply
adding copyright and modifying the copyright year141414Commit hash
3fd4da5034e51210ec159646201f29192c2d391b. Figure 20 shows one of these code
changes.

Figure 20.  Sample code commit showing copyright changes.

The code change itself modified 83 files and had 646 lines added and 34 line
deletions. Although our experiments showed that lines of code added, and lines
of code deleted were good features to use in predicting defective code change,
further look into the metrics showed that this metric can be misleading.
According to Kamei et al., large changes are more likely to be defective (e.g.
lines of code added, and entropy) (Kamei2013, ), as the code change itself is
only adding comments and updating copyright and comments, this might lead to
false positive results. Also, in other works, comments have been discarded as
noted in (Williams2008, ; Hoang2019, ; Eghbali2020, ). Comments are not
typically part of the executing part of the code (Eghbali2020, ). The amount
of inspection effort would largely go to waste for these kind of commits.

At the same time, we found large code changes that make coarse-grained JIT
largely unhelpful in reducing the inspection cost. Figure shows a large code
change 151515Commit hash 788944cebef5959a396400653801360987b27d20 that served
to significantly rewrite areas of the code.

Figure 21.  Sample code commit linked with many bug fixes.

The change itself modified 20 files with 711 lines of code added and 673 lines
deleted. Interestingly to note that there has been 23 bug fixes that can be
traced back to this code change over time, several of them involved in a
single file in this code change: /trac/ticket/notification.py. These can be
called dormant bugs where the defects are not immediately known until much
later. The end result is that the defect was introduced in a much earlier
version and only known to be defective in a much later version (Chen2014, ).

###  4.3. Analyzing Qt and OpenStack

We also looked into the two datasets provided by (Hoang2019, ) and examine the
bug inducing changes on those datasets. The summary of the datasets are shown
in table 7 (Hoang2019, ).

Table 7. Summary of the datasets (Hoang2019, ) |  |  | %Defect
---|---|---|---
|  |  | induced
Project | Period | #change | change
Qt | 06/2011-03/2014 | 25,150 | 8%
OpenStack | 11/2011-02/2014 | 12,374 | 13%

As the list of bug inducing changes fall on different sub projects, we
manually retraced the projects that were utilized for the datasets. On
average, we have found that the number of files changed in the bug inducing
commits in the Qt dataset is 9.25 files and over 102 different file
extensions. The highest modified file types are .cpp, .h, and .qml.

With the Openstack dataset, on average, there are 7.19 files changed and over
33 different file extensions. The highest modified file types are .py, .json,
and .xml. From these results, we can see that code changes can add or modify
multiple files across file types.

Given that on average, the number of files changed is 7.19, a developer would
be expected to check the changes of these files and understand the context of
such. As noted by (Kamei2013, ), inspection costs given a flagged defect
inducing code change is important for the practical application of defect
prediction. Inspection cost and code review from coarse grained approaches
also lead Pascarella et al. into identifying specific files in a code change
as bug inducing (Pascarella2019, ).

In analyzing the code change in the OpenStack, we also find refactor code
changes that are labelled defect inducing. This is shown in figure 22.

Figure 22.  Refactor Code Change in OpenStack.

Similar to Trac, we find such refactoring code change to introduce new files.
The number of lines added in the new file shown in figure 22 is 148.

There are defect inducing code changes]footnoteCommits such as
0061c0ce4443395a258a99c70e1c1a7d0435e84e and
00dd97c7e9a852f4abd6e8460a0d094c01d3d0ba that add new features. For the code
change161616Commit 0109cc967c0a8adb9073b6e89276312cb5a95bc4, the code change
is a combination of python files and XML. Figure 23 shows code change adding
new files into the repository.

Figure 23.  Modified file (ovs_quantum_agent.py) in a code change that adds
new features added in OpenStack.

The changes are modifications of existing files, and so, there are lines
removed and new lines added. This code change 171717Commit
00dd97c7e9a852f4abd6e8460a0d094c01d3d0ba modifies 6 files with 163 lines added
and 110 lines deleted. In marking the file within the code change as
defective, the file itself is 822 lines and is shown in figure 24.

Figure 24.  End of the OpenStack Python file (ovs_quantum_agent.py).

With no explainable reason for what is defective, labelling code changes and
even a specific file as defect inducing is quite complex for the developer to
inspect. This is consistent with the insights from (Nam2018, ) and (Dam2019,
).

With the datasets provided by Hoang et al. (Hoang2019, ) showing that bug
inducing commits can span across many files and multiple file types, we
believe that our findings from the trac dataset have been confirmed. Labelling
a code change as defective alone may still have significant inspection effort
from the developer to gauge which file or files caused the bug given a
specific code change.

##  5\. Analysis

Relying solely on the feature metrics alone to classify code change as
defective. There have been instances when the code change is very small and
yet is defect inducing. The use of semantics to capture such nuances could
provide better level of detail. For example, using the code itself as input
can allow the classifier to ignore code comments. This mirrors sentiments from
(Ghose2018, ).

In addition, classifying a code change as defective may not provide enough
context as to what could be defective in that commit. There is an associated
inspection cost towards code changes marked as defective (Kamei2013, ).
Although, there has been development towards Effort-Aware researches in JIT to
address this, the approach used is by sorting the changes accordingly by
inspection cost (Qiao2019, ).

As code changes can involve many files, a code change can be partially
defective or completely defective. Using coarse-grained JIT has inherent
inaccuracies in classifying an entire code change as defective. A finer-
grained approach such as files within a code change can provide better
accuracy level. This is consistent with findings from (Pascarella2019, ).

Considering, however, the code change in figure 21 that is a dormant bug and
is linked to many bug fixes, identifying the file itself as defective may not
be enough. From the code changes we have shown in our analysis, it is clear
that defect inducing changes are complicated. Labelling a code change as
defect inducing is too simplistic a perspective. In addition, as a bug fix may
end up modifying lines that were introduced by multiple code changes, the
origin of a software defect could be from the combination of various code
changes itself and not from a single one.

As a next step on top of automated defect prediction, being able to locate the
area of the code within the code change becomes a critical component in
reducing the inspection effort of developers. Such research may need to go
further than relying on the use of feature metrics with source code defects
having many fine nuances that feature metrics cannot capture.

##  6\. Threats to Validity

Several researches (Williams2008, ; Davies2014, ; Rodriguez-Perez2018, ) have
pointed that SZZ algorithm has limitations in determining the bug inducing
changes in a repository. Our research relies on the accuracy and correctness
of the existing tool provided by (Borg2019, ).

Of the 2003 bug inducing changes in the Qt dataset, we had difficulties
locating 140 bug inducing hashes. We believe, however, that we have been able
to demonstrate the nature of the issues with the ones we had found. We were
able to trace all bug inducing hashes from the Openstack dataset that further
validates our assertions.

##  7\. Access to the Dataset

The dataset is downloadable through this URL:
https://doi.org/10.5281/zenodo.3910002.

##  8\. Conclusion

We have presented a new dataset that can be used for JIT research that has
information to extract semantic features. We also pointed out issues with
current publicly available datasets. Firstly that by marking changes as
defective, there is still the effort of identifying which specific file in the
change as defective. Second, publicly available JIT datasets do not include
the commit information that can be used to link the actual code change diffs
to extract semantic features.

For future works, assembling more JIT datasets with the commit identifiers
intact will allow for more diverse perspectives in progressing both JIT and
semantic learning in JIT. By also looking deeper into semantic-based
approaches in traditional SDP such as in (Liang2019, ), it is our belief that
a finer-grained JIT is possible given the wealth of data that can be mined
from code repositories.

## References

  * (1) J. Ge, J. Liu, and W. Liu, “Comparative study on defect prediction algorithms of supervised learning software based on imbalanced classification data sets,” Proceedings - 2018 IEEE/ACIS 19th International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2018, pp. 399–406, 2018.
  * (2) Q. Huang, X. Xia, and D. Lo, Revisiting supervised and unsupervised models for effort-aware just-in-time defect prediction.  Empirical Software Engineering, 2018.
  * (3) A. Ghose, S. W. Ng, T. Tran, T. T. M. Pham, J. Grundy, and H. K. Dam, “Automatic feature learning for predicting vulnerable software components,” IEEE Transactions on Software Engineering, vol. 14, no. November, pp. 1–1, 2018.
  * (4) Z. Li, X.-Y. Jing, and X. Zhu, “Progress on approaches to software defect prediction,” IET Software, vol. 12, no. 3, pp. 161–175, 2018.
  * (5) J. Nam, W. Fu, S. Kim, T. Menzies, and L. Tan, “Heterogeneous Defect Prediction,” IEEE Transactions on Software Engineering, vol. 44, no. 9, pp. 874–896, 2018.
  * (6) T. Hoang, H. Khanh Dam, Y. Kamei, D. Lo, and N. Ubayashi, “DeepJIT: An end-to-end deep learning framework for just-in-time defect prediction,” IEEE International Working Conference on Mining Software Repositories, vol. 2019-May, pp. 34–45, 2019.
  * (7) Z. Durumeric, M. Payer, V. Paxson, J. Kasten, D. Adrian, J. A. Halderman, M. Bailey, F. Li, N. Weaver, J. Amann, and J. Beekman, “The Matter of Heartbleed,” in Proceedings of the 2014 Conference on Internet Measurement Conference - IMC ’14, (New York, New York, USA), pp. 475–488, ACM Press, 2014.
  * (8) H. K. Dam, T. Pham, S. W. Ng, T. Tran, J. Grundy, A. Ghose, T. Kim, and C. J. Kim, “Lessons learned from using a deep tree-based model for software defect prediction in practice,” IEEE International Working Conference on Mining Software Repositories, vol. 2019-May, pp. 46–57, 2019.
  * (9) L. H. Son, N. Pritam, M. Khari, R. Kumar, P. T. M. Phuong, and P. H. Thong, “Empirical study of software defect prediction: A systematic mapping,” Symmetry, vol. 11, no. 2, 2019.
  * (10) L. Qiao and Y. Wang, “Effort-aware and just-in-time defect prediction with neural network,” PLoS ONE, vol. 14, no. 2, pp. 1–19, 2019.
  * (11) S. Hosseini, B. Turhan, and D. Gunarathna, “A systematic literature review and meta-analysis on cross project defect prediction,” IEEE Transactions on Software Engineering, vol. 45, no. 2, pp. 111–147, 2019.
  * (12) M. Rahman, Y. Watanobe, and K. Nakamura, “Source Code Assessment and Classification Based on Estimated Error Probability Using Attentive LSTM Language Model and Its Application in Programming Education,” Applied Sciences, vol. 10, no. 0, 2020.
  * (13) X. Yang, D. Lo, X. Xia, Y. Zhang, and J. Sun, “Deep Learning for Just-in-Time Defect Prediction,” Proceedings - 2015 IEEE International Conference on Software Quality, Reliability and Security, QRS 2015, no. 1, pp. 17–26, 2015\.
  * (14) Albahli, “A Deep Ensemble Learning Method for Effort-Aware Just-In-Time Defect Prediction,” Future Internet, vol. 11, p. 246, nov 2019.
  * (15) A. Mockus and D. M. Weiss, “Predicting risk of software changes,” Bell Labs Technical Journal, vol. 5, no. 2, pp. 169–180, 2000.
  * (16) Q. Huang, X. Xia, and D. Lo, “Supervised vs Unsupervised Models: A Holistic Look at Effort-Aware Just-in-Time Defect Prediction,” in 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 159–170, IEEE, sep 2017.
  * (17) L. Cai, Y. R. Fan, M. Yan, and X. Xia, “Just-in-time software defect prediction: literature review,” Ruan Jian Xue Bao/Journal of Software, vol. 30, no. 5, pp. 1288–1307, 2019.
  * (18) Y. Kamei, E. Shihab, B. Adams, A. E. Hassan, A. Mockus, A. Sinha, and N. Ubayashi, “A large-scale empirical study of just-in-time quality assurance,” IEEE Transactions on Software Engineering, vol. 39, no. 6, pp. 757–773, 2013.
  * (19) Y. Yang, Y. Zhou, J. Liu, Y. Zhao, H. Lu, L. Xu, B. Xu, and H. Leung, “Effort-Aware just-in-Time defect prediction: Simple unsupervised models could be better than supervised models,” Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering, vol. 13-18-Nove, pp. 157–168, 2016.
  * (20) W. Fu and T. Menzies, “Revisiting unsupervised learning for defect prediction,” pp. 72–83, 2017.
  * (21) R. Özakıncı and A. Tarhan, “Early software defect prediction: A systematic map and review,” Journal of Systems and Software, vol. 144, pp. 216–239, 2018.
  * (22) C. Catal and B. Diri, “A systematic review of software fault prediction studies,” Expert Systems with Applications, vol. 36, no. 4, pp. 7346–7354, 2009.
  * (23) Q. Liu, Z. Liu, H. Zhu, H. Fan, B. Du, and Y. Qian, “Generating commit messages from diffs using pointer-generator network,” IEEE International Working Conference on Mining Software Repositories, vol. 2019-May, pp. 299–309, 2019.
  * (24) M. Shepperd, Q. Song, Z. Sun, and C. Mair, “Data quality: Some comments on the NASA software defect datasets,” IEEE Transactions on Software Engineering, vol. 39, no. 9, pp. 1208–1215, 2013.
  * (25) A. Lamkanfi, J. Pérez, and S. Demeyer, “The eclipse and mozilla defect tracking dataset: A genuine dataset for mining bug information,” IEEE International Working Conference on Mining Software Repositories, pp. 203–206, 2013.
  * (26) A. Pietri, D. Spinellis, and S. Zacchiroli, “The Software Heritage Graph Dataset: Public software development under one roof,” in MSR 2019: The 16th International Conference on Mining Software Repositories, pp. 138–142, IEEE, 2019.
  * (27) R. Wu, H. Zhang, S. Kim, and S.-C. Cheung, “ReLink: Recovering Links between Bugs and Changes,” Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering - SIGSOFT/FSE ’11, p. 15, 2011.
  * (28) M. D’Ambros, M. Lanza, and R. Robbes, Evaluating defect prediction approaches: A benchmark and an extensive comparison, vol. 17.  2012\.
  * (29) F. Peters and T. Menzies, “Privacy and utility for defect prediction: Experiments with MORPH,” Proceedings - International Conference on Software Engineering, pp. 189–199, 2012.
  * (30) B. Turhan, T. Menzies, A. B. Bener, and J. Di Stefano, “On the relative value of cross-company and within-company data for defect prediction,” Empirical Software Engineering, vol. 14, no. 5, pp. 540–578, 2009.
  * (31) G. Canfora, A. D. Lucia, M. D. Penta, R. Oliveto, A. Panichella, and S. Panichella, “Defect prediction as a multiobjective optimization problem,” Software Testing, Verification and Reliability, vol. 25, pp. 426–459, jun 2015.
  * (32) J. Voas and R. Kuhn, “What Happened to Software Metrics?,” Computer, vol. 50, no. 5, pp. 88–98, 2017.
  * (33) S. H. Kan, Metrics and Models in Software Quality Engineering.  Boston, MA, USA: Addison-Wesley Longman Publishing Co., Inc., 2nd ed., 2002.
  * (34) S. R. Chidamber and C. F. Kemerer, “A Metrics for Object Oriented Design,” 1994\.
  * (35) N. Nagappan and T. Ball, “Use of relative code churn measures to predict system defect density,” Proceedings - 27th International Conference on Software Engineering, ICSE05, pp. 284–292, 2005.
  * (36) A. E. Hassan, “Predicting faults using the complexity of code changes,” Proceedings - International Conference on Software Engineering, pp. 78–88, 2009.
  * (37) R. Moser, W. Pedrycz, and G. Succi, “A Comparative analysis of the efficiency of change metrics and static code attributes for defect prediction,” Proceedings - International Conference on Software Engineering, no. January 2008, pp. 181–190, 2008.
  * (38) C. Lewis, Z. Lin, C. Sadowski, X. Zhu, R. Ou, and E. J. Whitehead, “Does bug prediction support human developers? Findings from a Google case study,” Proceedings - International Conference on Software Engineering, pp. 372–381, 2013.
  * (39) J. Śliwerski, T. Zimmermann, and A. Zeller, “When do changes induce fixes?,” Proceedings of the 2005 International Workshop on Mining Software Repositories, MSR 2005, 2005.
  * (40) X. Chen, Y. Zhao, Q. Wang, and Z. Yuan, “MULTI: Multi-objective effort-aware just-in-time software defect prediction,” Information and Software Technology, vol. 93, no. August 2017, pp. 1–13, 2018.
  * (41) S. McIntosh and Y. Kamei, “Are Fix-Inducing Changes a Moving Target? A Longitudinal Case Study of Just-in-Time Defect Prediction,” in Proceedings of the 40th International Conference on Software Engineering, ICSE ’18, (New York, NY, USA), p. 560, Association for Computing Machinery, 2018\.
  * (42) L. Pascarella, F. Palomba, and A. Bacchelli, “Fine-grained just-in-time defect prediction,” Journal of Systems and Software, vol. 150, pp. 22–36, 2019.
  * (43) P. Bourque and R. E. Fairley, SWEBOK v.3 - Guide to the Software Engineering - Body of Knowledge. 2014\.
  * (44) M. Felderer and A. Beer, “Using defect taxonomies for testing requirements,” IEEE Software, vol. 32, no. 3, pp. 94–101, 2015.
  * (45) G. Upadhyaya and H. Rajan, “On Accelerating Source Code Analysis at Massive Scale,” IEEE Transactions on Software Engineering, vol. 44, no. 7, pp. 669–688, 2018.
  * (46) Z. Li, X. Y. Jing, X. Zhu, H. Zhang, B. Xu, and S. Ying, “On the Multiple Sources and Privacy Preservation Issues for Heterogeneous Defect Prediction,” IEEE Transactions on Software Engineering, vol. 45, no. 4, pp. 391–411, 2019.
  * (47) H. Hata, O. Mizuno, and T. Kikuno, “Bug prediction based on fine-grained module histories,” Proceedings - International Conference on Software Engineering, no. October 2014, pp. 200–210, 2012.
  * (48) T. Fukushima, Y. Kamei, S. McIntosh, K. Yamashita, and N. Ubayashi, “An empirical study of just-in-time defect prediction using cross-project models,” Empirical Software Engineering, no. May 2014, pp. 172–181, 2014\.
  * (49) Y. Cho, J. H. Kwon, and I. Y. Ko, “Cross-sub-project just-in-time defect prediction on multi-repo projects,” CEUR Workshop Proceedings, vol. 2273, no. QuASoQ, pp. 6–13, 2018.
  * (50) M. D’Ambros, M. Lanza, and R. Robbes, “An extensive comparison of bug prediction approaches,” Proceedings - International Conference on Software Engineering, no. May 2010, pp. 31–41, 2010.
  * (51) M. Borg, O. Svensson, K. Berg, and D. Hansson, “SZZ unleashed: an open implementation of the SZZ algorithm - featuring example usage in a study of just-in-time bug prediction for the Jenkins project,” in Proceedings of the 3rd ACM SIGSOFT International Workshop on Machine Learning Techniques for Software Quality Evaluation - MaLTeSQuE 2019, (New York, New York, USA), pp. 7–12, ACM Press, 2019.
  * (52) M. Kiehn, X. Pan, and F. Camci, “Empirical study in using version histories for change risk classification,” IEEE International Working Conference on Mining Software Repositories, vol. 2019-May, pp. 58–62, 2019.
  * (53) L. Pascarella, F. Palomba, and A. Bacchelli, “Re-evaluating method-level bug prediction,” 25th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2018 - Proceedings, vol. 2018-March, pp. 592–601, 2018.
  * (54) H. Liang, Y. Yu, L. Jiang, and Z. Xie, “Seml: A Semantic LSTM Model for Software Defect Prediction,” IEEE Access, vol. 7, pp. 83812–83824, 2019\.
  * (55) S. Kim, T. Zimmermann, E. J. Whitehead, and A. Zeller, “Predicting faults from cached history,” Proceedings - International Conference on Software Engineering, pp. 489–498, 2007.
  * (56) F. Rahman, D. Posnett, A. Hindle, E. Barr, and P. Devanbu, “BugCache for inspections : Hit or miss?,” SIGSOFT/FSE 2011 - Proceedings of the 19th ACM SIGSOFT Symposium on Foundations of Software Engineering, no. May 2014, pp. 322–331, 2011.
  * (57) F. Rahman, D. Posnett, and P. Devanbu, “Recalling the ”imprecision” of cross-project defect prediction,” no. September 2012, p. 1, 2012.
  * (58) S. Wang, T. Liu, and L. Tan, “Automatically learning semantic features for defect prediction,” in Proceedings of the 38th International Conference on Software Engineering - ICSE ’16, vol. 14-22-May-, (New York, New York, USA), pp. 297–308, ACM Press, 2016.
  * (59) S. Matsumoto, Y. Kamei, A. Monden, K. I. Matsumoto, and M. Nakamura, “An analysis of developer metrics for fault prediction,” ACM International Conference Proceeding Series, no. January, 2010.
  * (60) M. Yan, X. Xia, Y. Fan, A. E. Hassan, D. Lo, and S. Li, “Just-In-Time Defect Identification and Localization: A Two-Phase Framework,” IEEE Transactions on Software Engineering, pp. 1–20, 2020.
  * (61) S. Wattanakriengkrai, P. Thongtanunam, C. Tantithamthavorn, H. Hata, and K. Matsumoto, “Predicting Defective Lines Using a Model-Agnostic Technique,” IEEE Transactions on Software Engineering, no. September, pp. 1–23, 2020.
  * (62) G. Rodríguez-Pérez, G. Robles, A. Serebrenik, A. Zaidman, D. M. Germán, and J. M. Gonzalez-Barahona, “How bugs are born: a model to identify how bugs are introduced in software components,” Empirical Software Engineering, vol. 25, no. 2, pp. 1294–1340, 2020.
  * (63) S. Fakhoury, D. Roy, Y. Ma, V. Arnaoudova, and O. Adesope, “Measuring the impact of lexical and structural inconsistencies on developers’ cognitive load during bug localization,” Empirical Software Engineering, vol. 25, no. 3, pp. 2140–2178, 2020.
  * (64) F. Qiu, M. Yan, X. Xia, X. Wang, Y. Fan, A. E. Hassan, and D. Lo, “JITO: a tool for just-in-time defect identification and localization,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, no. September, (New York, NY, USA), pp. 1586–1590, ACM, nov 2020.
  * (65) C. Williams and J. Spacco, “SZZ revisited: verifying when changes induce fixes,” in Proceedings of the 2008 workshop on Defects in large software systems - DEFECTS ’08, no. January 2008, (New York, New York, USA), p. 32, ACM Press, 2008.
  * (66) A. Eghbali and M. Pradel, “No Strings Attached: An Empirical Study of String-related Software Bugs,” Proceedings - 2020 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020, pp. 956–967, 2020.
  * (67) T. H. Chen, M. Nagappan, E. Shihab, and A. E. Hassan, “An empirical study of dormant bugs,” 11th Working Conference on Mining Software Repositories, MSR 2014 - Proceedings, pp. 82–91, 2014.
  * (68) S. Davies, M. Roper, and M. Wood, “Comparing text-based and dependence-based approaches for determining the origins of bugs,” Journal of Software: Evolution and Process, vol. 26, pp. 107–139, jan 2014.
  * (69) G. Rodríguez-Pérez, G. Robles, and J. M. González-Barahona, “Reproducibility and credibility in empirical software engineering: A case study based on a systematic literature review of the use of the SZZ algorithm,” Information and Software Technology, vol. 99, no. March, pp. 164–176, 2018.