# IRJIT: An Information Retrieval Technique for Just-in-time Defect
Identification

Hareem Sahar, Yuxin Liu, Abdul Ali Bangash, Abram Hindle, Denilson Barbosa
Corresponding Author<EMAIL_ADDRESS>Department of Computing Science
University of Alberta
Edmonton, Canada

###### Abstract

Defect identification at commit check-in time prevents the introduction of
defects into software. Current defect identification approaches either rely on
manually crafted features such as change metrics or involve training expensive
machine learning or deep learning models. By relying on a complex underlying
model, these approaches are not often explainable, which means the models’
predictions cannot be understood by the developers. An approach that is not
explainable might not be adopted in real-life development environments because
of developers’ lack of trust in its results. Furthermore, because of an
extensive training process, these approaches cannot readily learn from new
examples when they arrive, making them unsuitable for fast online prediction.
To address these limitations, we propose an approach called IRJIT that employs
information retrieval on source code, and labels new commits as buggy or clean
based on their similarity to past buggy or clean commits. Our approach is
online and explainable as it can learn from new data without retraining, and
developers can see the documents that support a prediction. Through an
evaluation of 8 open-source projects, we show that IRJIT achieves AUC and F1
score close to the state-of-the-art machine learning approach JITLine, without
considerable re-training.

###### Index Terms:

Empirical, Defect Identification, online, explainable, Information Retrieval,
Similarity Detection, Just-in-time

##  I Introduction

The later a defect is identified in the software development life cycle, the
higher its impact and maintenance cost [35]. To identify and prevent defects
early on, Just-in-time (JIT) defect identification [32][22] is proposed as it
can identify a risky commit at check-in time. Identifying risky commits at
commit-time allows practitioners to dedicate their limited testing resources
to review and fix the riskiest commits while changes are still fresh in the
authors’ minds.

Explainability is a desired property of a prediction model designed for a real
life development environment. For a JIT approach to be called explainable, the
developers shall be able to trace back predictions to their originating
documents. An example would be highlighting parts of a change that look
similar to a past buggy change. The state-of-the-art JIT defect prediction
approaches such as DeepJIT [19] and CC2Vec [20] are not explainable because of
reliance on a complex underlying learning model such as a deep neural network
model. Deep learning models are not only cost-prohibitive due to the time and
resources needed for training but inherently lack explainability [13].

To address these limitations of prior work, in this paper, we propose IRJIT,
an explainable information retrieval (IR) approach for JIT defect
identification. IRJIT classifies a new commit into clean or buggy based on its
similarity with past commits. To compute similarity, IRJIT exploits BM25
algorithm [49]. Next, for each identified buggy commit, IRJIT estimates its
buggy lines using tf–idf term weights [44]. By relying on source code token
frequencies instead of hand-crafted features such as code change or review
metrics [31], we make our approach readily deployable for any project with an
active version control system and an issue tracker. Our approach offers
several advantages over recent JIT approaches [19] [20]:

  * •

IRJIT works in an online manner which means it adapts over time by learning
from new changes, as soon as they arrive, without the costs associated with
re-training models. Being able to update online and quickly react to a change
is desirable for the adoption of a JIT approach [53].

  * •

IRJIT is explainable [36][13]. Developers can inspect source code documents
that were used in prediction to understand why a commit was classified as
buggy. These characteristics improve transparency and increase a
practitioners’ trust in the prediction [13], while also making the fixing of
an actual bug easier.

  * •

IRJIT is simple since it draws its conclusions by matching against the past
source code documents that developers are already familiar with. Moreover, it
does not rely on a frequent and extensive retraining process which can slow
down the overall prediction process.

To establish the strengths and weaknesses of our approach we compare it with a
recent machine learning (ML) approach JITLine [36]. JITLine is a sophisticated
approach that relies on code and review based metrics [31] and bag of word
tokens. JITLine engages in extensive tuning through SMOTE and Differential
Evolution to directly optimize metrics and word vectors for AUC. We believe
that such an approach might delay the online prediction process [51][9].
Furthermore, the review metrics used by JITLine are extracted from Gerrit
review databases [2] which are not frequently used by open-source projects,
hence limiting the usability of JITLine to only well-documented and well-
reviewed systems. We believe that the excellent performance of JITLine might
be specific to OpenStack and Qt project due to the use of review metrics.
Lastly, JITLine, despite being more sophisticated than prior approaches,
leverages SMOTE [12] to create new training examples, as a consequence of
which it cannot retrieve originating commits that can be blamed for a bug. In
other words, it cannot explain the predictions by linking to the original
documents causing the classification, although it can rank lines by
defectiveness.

Our case study consists of an existing dataset containing $37,524$ commits
from OpenStack and Qt projects, and an extended dataset of $6$ diverse open-
source projects containing $140,344$ commits. Through a realistic evaluation,
we show that our approach correctly classifies a new commit with a median
F-score of $0.40$ and AUC of $0.85$. IRJIT outperforms JITLine at ranking
buggy lines of a buggy commit. The contributions of this work include:

  * •

a JIT defect prediction approach based on IR to detect buggy commits at commit
check-in time;

  * •

an approach to identify associated buggy lines for each identified buggy
commit;

  * •

a realistic evaluation of our approach on OpenStack, Qt, and $6$ other open-
source projects using various metrics, and its comparison with a mature
machine learning baseline, JITLine.

##  II Related Work

JIT defect prediction emphasizes the identification of risky changes just-in-
time, i.e., before they reach repository. Existing JIT approaches detect bugs
based on historical information such as change histories [26] or by relating
past defects to software metrics [43][22]. CLEVER [33] detects new defects by
comparing new code to known cases of buggy code. It uses clone detection to
improve the detection accuracy of risky changes identified through a metric-
based model. Unlike CLEVER we do not use a metric-based model to identify
buggy changes. DeepBugs[37] detects incorrect code by reasoning about
identifier names, and training a binary classifier on correct and incorrect
examples. We leverage code similarity between past buggy and clean changes to
explain why a new change is classified as buggy. This line of work is inspired
by the success of code search and recommendation engines [30][24] which employ
IR techniques. IR-based methods work by matching shared vocabulary between
query and corpus documents, and are effective for code retrieval [28][58] and
bug localization [40].

Due to the repetitive nature of source code [18], statistical language models
can also spot defective code effectively [41][10][45]. Yan _et al._ [55] use
several change level features and software naturalness with the n-gram model
to localize buggy changes. The proposed framework ranks buggy lines in order
of suspiciousness. Recently, Pornprasit _et al._ [36] showed that their ML
model outperforms n-gram model in prioritizing defective lines. Their model
also beats sophisticated DL approaches like DeepJIT [19] and CC2Vec [20],
which are computationally expensive and also lack explainability. A recent
survey by Wang _et al_ [53] suggests that developers prefer simple,
explainable and online defect prediction approaches. In this paper, we design
and evaluate a simple IR approach for JIT defect prediction. Our approach
improves the explainability of JIT defect prediction by showing past examples
that a prediction is based on while also being online and requiring limited
training.

##  III IRJIT Overview

IRJIT can be integrated into git through a commit hook [1], or a pull request
bot, which is triggered when a developer initiates a code commit. Before this
new commit reaches the project central repository, IRJIT will analyze the
source code modified by the developer and compare it against past changes in
the project’s version control system. If the new commit matches a past buggy
commit, the developer is prompted to inspect their code. In this way, IRJIT
can prevent the bugs from reaching the main branch. Figure 1 shows the backend
design of IRJIT framework. IRJIT extracts past buggy and clean commits
(Section III-A) of a project and their associated source code changes (Section
III-B). The source code changes are stored into an inverted index for querying
(Section III-C). For each newly intercepted commit, IRJIT queries the index to
identify matches of the newly committed change, and determine if the new
changes are similar to past buggy changes. If a newly committed change matches
a known buggy change in the indexed corpora, it is flagged as risky (Section
III-D). Lastly, for each buggy change, IRJIT ranks source code lines that are
part of the change according to their riskiness (Section III-E).

Figure 1: An overview of IRJIT operation. IRJIT first extracts and indexes
past changes. When a new commit arrives it is classified based on its
similarity with the past matches retrieved from the index. Finally, changed
lines are ranked according to bugginess.

###  III-A Extracting buggy and clean commits

As a first step, IRJIT builds a corpus of buggy and clean commits. A commit
contains one or more lines of code modified by a developer and spread across
one or multiple files. To obtain buggy commits from a project’s history
researchers use SZZ algorithm [47, 27]. SZZ works on the information available
in the version control and issue tracking system. In the first step, the
algorithm identifies a defect-fixing commit. A commit is fixing if it fixes a
defect, i.e., it refers to an issue labeled as fixed in the issue tracking
system. In the second step, SZZ traces back through the version control to
identify the associated buggy commit i.e. the one responsible for inducing the
fix. For this purpose, SZZ algorithm leverages diff and annotate/blame
functionality. The diff helps identify lines that changed between a defect-
fixing commit and its immediately previous commit. These are the lines that
fixed the bug. The annotate/blame functionality identifies commits that
originally introduced or modified those lines in the past. SZZ flags those
original changes as defect-introducing or buggy if the commit was made before
filing the issue in project issue tracker. Once algorithm completes its
execution, all buggy and fixing commits are identified, but there are other
commits that are unmarked by SZZ because they are neither buggy nor fixing.
Following prior work we extract and include them in our analysis [55][57].
From here onwards, we only use the terms buggy and clean to refer to commits.

###  III-B Extracting source code changes from commits

A change refers to modifications made in a single file in one commit. The
source code changes introduced by a commit may be spread across multiple files
modified in that commit. Each change contains several hunks, i.e., a
continuous group of lines added to or deleted from each changed file
associated with a given commit [25]. IRJIT extracts hunks using the git diff
utility and pre-processes them to remove blank lines. The processed source
code changes that belong to a single file are then stored in its corresponding
JSON format document along with the necessary metadata which includes lines
added, lines deleted, commit hash, file name, and label. Currently, IRJIT uses
lines added by default, because the probability of introducing a bug is higher
with the addition of new lines, as suggested by Purushothaman _et al_ [38],
and hence spotting bugs might be easier with only lines added field.

###  III-C Indexing past source code changes

To make the classification of a new commit scalable and efficient, IRJIT
builds inverted indexes using Elasticsearch, which is a real-time full-text
search engine built on top of the Apache Lucene library [4]. Elasticsearch
indexes complex data represented as JSON documents.

After creating an index, the past changes or JSON documents are added to the
index, but the new changes (test data) are used for querying, so those are not
indexed. In practice a single index per project can serve the purpose of
storing all past changes. The index can be updated by incrementally adding new
changes when they arrive, without the need of re-indexing any previous
documents. Unlike existing approaches that require retraining the model every
time some new data is added, IRJIT’s approach allows developers to query
against the new changes right away. The ability to remain online allows IRJIT
to be deployed for repositories of any size.

As mentioned earlier, the indexed documents contain lines added and lines
deleted fields which are tokenized, indexed, and analyzed depending on the
user specified index settings. IRJIT relies on various tokenizers and filters
to build custom Elasticsearch analyzers. After experimenting with multiple
analyzers, we found that they exhibit similar performance for our use-case
with minimal to no difference. Therefore, IRJIT uses only camel-case analyzer
[15] and shingle analyzer [16] because together these two slightly
outperformed other analyzers. Table I shows how an example source code line is
processed by camel-case and shingle analyzer. The camel-case analyzer with
whitespace tokenizer [17] is designed to split words into tokens when it
encounters a case change. The shingle analyzer with a custom tokenizer based
on javalang analyzer [8] is designed to produce word n-grams of size 4 as
min_shingle_size=4 and max_shingle_size=4. The choice of n-gram size was
inspired by Siamese [39] and allows capturing sequences as well as small
modifications within statements.

TABLE I: Elasticsearch analyzers used in IRJIT Analyzer |  Tokens for input line: if(tbl != null && !isExternal(tbl))
---|---
camel-case | [ if(tbl , if , tbl , != , null , && , !isExternal(tbl)) , is ,
| External , tbl ]
shingle | [ !=null&&! , !isexternal(tbl , &&!isexternal( , (tbl!=null ,
| (tbl)) , if(tbl!= , isexternal(tbl) , null&&!isexternal ,
| tbl!=null&& ]

###  III-D Classifying an incoming commit

####  III-D1 Querying to find matches

When a new (test) commit is checked-in to a version control system, IRJIT
classifies it as buggy or clean by finding matches of test commit from the
past project changes. In order to classify a test commit, IRJIT first extracts
source code changes from that commit and saves each change separately. Next,
it formulates a query against each change using the lines added field of
corresponding change as input. Lastly, it executes the query and retrieves
matches from the previously created index. Since one query corresponds to only
one change, IRJIT may issue multiple queries for a given test commit.
Specifically, IRJIT formulates a more_like_this (MLT) query [14], which finds
indexed training documents that are “like” a given test document. MLT query
utilizes corpus-level and document-level term statistics, i.e., term
frequencies and inverse document frequencies (tf–idf) to compute similarity
between a query and documents in the corpus. By default Elasticsearch uses
BM25 algorithm[49] to compute similarity. Moreover, it uses the same analyzers
during the querying process that were originally used for indexing the
documents. Elasticsearch also allows users to configure a different similarity
algorithm than BM25, but the default one was sufficient for the purpose of our
study. Based on BM25 similarity results, Elasticsearch returns Top-K most
similar training documents for a given test document. Elasticsearch also
returns a relevance score against each match as shown in Figure 2. The
matching documents from all MLT queries are combined into a set called
candidate matches and sorted according to relevance score. The Elasticsearch
scoring formula [5] is originally based on tf–idf, but contains adjustments to
allow for comparison of scores across different queries [11]. Next, IRJIT’s
classifier determines a final label for the test commit.

####  III-D2 Determining label from matches

IRJIT currently implements two classifiers; a K-nearest neighbor (KNN)
classifier and a threshold-based (Threshold) classifier. These take candidate
matches from the last step as input and return either a buggy or a clean label
for the test commit.

##### K-nearest neighbor classifier

The K-nearest neighbor classifier takes K closest candidate matches and uses
their majority label to make predictions. If the majority of K neighbors are
buggy, the test commit is labeled buggy; otherwise, it is considered clean.
The value of K might be tuned for each project to optimize performance.
Alternatively, the user can provide it as input. Currently, IRJIT uses a
default value of $K=3$ which was set empirically after experimenting on
$K={1,3,5,7,9}$.

##### Threshold classifier

The candidate matches returned for the test commit contain both buggy and
clean changes from the past. Intuitively, a test commit may be considered
buggy if there is at least one buggy change in the candidate matches. However,
such a model would be considered biased towards buggy changes. To counteract
this bias, we only accept a buggy prediction if sufficient confidence can be
established. Our threshold classifier uses a threshold value, Tscore, to
determine when to accept a buggy prediction. The threshold score is applied to
the relevance score returned by Elasticsearch. If there is at least one buggy
change in the candidate matches with a relevance score greater than Tscore,
IRJIT classifies the commit as buggy. In case of more than one buggy change,
the top match, i.e., the match with the highest relevance score, is used. In
the example shown in Figure 2, the incoming commit is predicted buggy based on
the candidate match M2 which is the top relevant buggy match and also has a
relevance score $(104)$ greater than Tscore.

The Tscore is project specific, and its higher values favor precision whereas
the lower values favor recall. To ensure the best performance, Tscore can be
tuned for any performance metric such as precision, recall, F1, or AUC using a
validation dataset. Our work uses the training data for validation. In order
to find an optimal Tscore value, IRJIT iterates over the set of relevance
scores returned from the validation dataset evaluation. It then picks a Tscore
value that maximizes performance across all queries for the AUC metric. Unlike
SMOTE that tunes on metrics and word vectors of each commit, we tune once for
each model. Our analysis suggests that the identified Tscore will be relevant
unless more than 50% of the data changes. In practice Tscore will only have to
be recalculated after several months.

##### Ensemble classifier

Ensemble learning is known for reducing bias and improving performance by
combining the output of a set of base classifiers. This motivated us to
incorporate an ensemble classifier in IRJIT because it does not slow down an
online prediction process while still offering enhanced performance. An
ensemble classifier can use one of the various meta-algorithms such as
bagging, boosting or stacking on top of base classifiers to combine
predictions. For classification problems the output of each base model might
be combined using a hard or soft voting mechanism, which leverages discrete
class labels and probabilities respectively. Liu _et al_ [29] proposed a
validation and voting scheme similar to boosting whereby the output is decided
based on the majority vote of classifiers. On the other hand Panichella _et
al_ [34] used the outputs of multiple classifiers as predictors of a new
prediction model, which is trained using Logistic Regression. Yang _et al_.
[56] proposed TLEL, an ensemble approach for JIT defect prediction, that
trains several Random Forest models using bagging and then combines them using
stacking. We adopt the concept used in TLEL whereby all classifiers are given
equal weight and their predictions are summed to get a final result. Recently,
Zeng _et al_ [57] achieved great performance by training a simple Logistic
Regression classifier for JIT defect prediction using a single feature, i.e.,
the number of lines added to a commit (la) and no optimization. Their la
classifier has good AUC and F1 but a low recall, which inspired us to ensemble
our KNN and Threshold classifier with their la classifier. Altogether, we
create four ensemble models following TLEL approach; first using Threshold and
KNN, second using Threshold and la, third using KNN and la, and fourth using
Threshold, KNN and la. Next, we report the results of our base classifiers and
the best performing ensemble classifier.

Figure 2:  Figure shows set of candidate matches M1, M2, M3 upto Mn with
relevance score 157, 104, 78 and 60 respectively corresponding to a commit,
C1. KNN classifier with K=3 predicts commit C1 is clean because majority,
i.e., 2 out of top-3 matches are clean. Threshold classifier with Tscore=100
predicts C1 is buggy because relevance score of top buggy match M2 is 104
which is greater than Tscore

###  III-E Ranking buggy lines of a commit

If a change is identified as buggy, the last step is to rank the lines
modified in that change according to their bugginess. Prior work [18, 41, 55]
exploits n-gram models to find most entropic source code tokens. However, we
adopt a slightly different approach to estimate the buggy tokens, i.e., tokens
that contribute towards overall bugginess of the document. We rely on tf–idf
term weights obtained while executing more_like_this query (Section III-D1).
The terms that get highest weight when a document is predicted buggy are
considered buggy tokens. After extracting buggy tokens we rank lines based on
the number of occurrences of buggy tokens in each line. Lastly, we use ranked
buggy lines to calculate our performance metrics.

##  IV Evaluation Setup

In this section, we present research questions, details of the evaluated
datasets, evaluation context, and performance measures.

###  IV-A Research Questions

RQ1: How effective is IRJIT in identifying new buggy commits?

The goal of this question is to investigate if our IR-based approach
accurately predicts buggy commits or not.

RQ2: Does the performance of IRJIT change if trained only on the recent data?

This research question aims to evaluate the effect of training models on
recent data on IRJIT’s performance.

RQ3: How effective is IRJIT in comparison to state-of-the-art machine learning
approach JITLine?

JITLine is an effective ML approach that outperformed CC2Vec [20], DeepJIT
[20] and EARL [22] on the basis of multiple performance measures. Although the
goal of our paper is not to beat JITLine by numbers, comparing against it
gives us a fair idea of the strengths and weaknesses of our approach.

RQ4: How well does IRJIT perform in predicting buggy lines of a buggy commit?

Prior work [41, 55] suggests that focusing on more entropic or unnatural lines
can save inspection effort. The goal of this evaluation is to find out if
IRJIT can effectively rank buggy lines in the correctly predicted buggy
commits or not. Such a ranking saves developers’ time because they only need
to inspect the top few lines to identify the error.

###  IV-B Dataset

We begin our evaluation with OpenStack and Qt dataset, which was originally
collected by McIntosh and Kamei [31] and later used in several studies [36,
20, 19]. This dataset contains code change and review properties extracted
from the version control system and Gerrit code review database respectively.
These are further explained in [31] as well as our replication package.

To extend our evaluation, we selected 6 additional open-source projects from
GitHub following criteria similar to prior work [55][9]. While OpenStack and
Qt are mainly written in python and C++, we selected open-source Java projects
to cover a diverse range of applications from different programming languages.
Our selection criteria were that the project should use a Jira issue tracker,
should have a rich history ($>$$10k$ commits), and should be popular
($>$$1000$ stars). Following this criteria we found $886$ projects out of
which we randomly picked $6$ projects. These projects were also used in prior
software engineering studies [neto2018impact] [55] [9] [50]. Table II presents
the distribution of commits for each evaluated project.

We collected the buggy commits for these projects using an open-source SZZ
implementation, SZZUnleashed [7]. Our approach uses source code changes which
we extracted those from each project’s GitHub repository using GitPython. For
calculating code change metrics for the baseline approach (JITLine) we wrote
our own scripts.

###  IV-C Training and test set

JIT defect prediction is useful if done in a timely manner using only past
data available at commit time [57][9]. Within-project defect prediction, uses
past data from the same project, whereas cross-project defect prediction
borrows training data from other projects [50]. This paper only evaluates the
within-project context, so each model is trained and tested on the same
project. We strictly avoid time-travel [6] in our evaluation by not violating
the chronological order of data, and our test data always comes after the
training data. Another important consideration is that a JIT model should be
able to adapt over time by learning from new data as soon as it becomes
available. Such models are referred to as online [9][50]. One way of updating
an online model is to use new data along with any prior data for retraining
the model. Another way is to discard old data and retrain the model using only
new data. We consider both evaluation settings and refer to them as
increasing-model and constant-model evaluation.

Increasing-models are trained using all data that is available prior to the
test period. As the test period moves farther in time the training set
increases, hence the name increasing models.

Constant-models are trained using data from a constant number of time periods
prior to the test period. In this paper we use just one time period prior to
test period for training.

Finally, when making predictions, we only evaluate against the past data that
has already received its correct labels. Leaving a gap between the train and
test set allows time for bugs to be identified and commits to get their
correct label. This gap is referred to as verification latency in literature
[9]. Tan et al. [51] suggested that the sum of gap and test set should be
close to typical bug-fixing time, i.e., the time from when a bug is introduced
to when it is fixed. However, since bug-fixing time varies for different
projects and bugs, this gap can also change with the dataset and is
empirically set using the median values of Bug-fix delay in Table II.

###  IV-D Baseline Approach

Our baseline is a state-of-the-art JIT approach, called JITLine [36] proposed
by Pornprasit _et al_ in 2021. The approach predicts buggy commits using a ML
model built from code change and review properties. To address the data
imbalance problem caused by a low percentage of buggy commits, JITLine invents
new documents with the help of SMOTE technique that is optimized using a
Differential Evolution (DE) algorithm.

We chose JITLine as our baseline because of its effectiveness as it has been
shown to outperform several deep-learning approaches such as CC2Vec [20], and
DeepJIT [19] as well as an n-gram based approach of Yan _et al_ [55].
JITLine’s replication package provides a Python implementation along with
OpenStack and Qt dataset. As mentioned earlier, we anticipate that JITLine’s
great performance might be specific to OpenStack and Qt because those are
well-documented and well-reviewed projects. We argue that such projects are
rare, therefore, we extend our evaluation by including 6 projects that do not
employ a Gerrit review database. For these projects review and reviewer
metrics could not be extracted. We, therefore, modified JITLine to work
without the aforementioned classes of metrics, and the modified version is
referred to as review-free JITLine, JITLinerf.

###  IV-E Performance Metrics

Our evaluation is based on metrics that were used by JITLine [36], and are
also commonly used in other JIT studies [57, 22, 31]. Precision measures the
relevant documents in the results, which in our case is correctly predicted
buggy commits out of all commits predicted as buggy. Recall measures the
relevant documents retrieved out of all possible relevant documents, i.e.,
buggy commits. It does not consider the irrelevant documents, i.e., clean
commits retrieved along. F-score (F1) is the harmonic mean of precision and
recall. FAR or false alarm rate measures the ratio of incorrectly predicted
buggy commits to the number clean commits. A lower FAR value indicates that
developers need to review fewer incorrectly predicted commits and developer
effort is not wasted. AUC is the area under the ROC curve which plots true
positive rate against the false positive rate. It is a threshold-independent
measure and evaluates an approach’s ability to distinguish between classes.
Distance-to-Heaven (d2h) combines recall and FAR metrics. Smaller values of
d2h are better. A d2h value of 0 suggests that the approach correctly predicts
all buggy commits without any false positives. A high d2h value indicates that
an approach is far from perfect, i.e., it achieves a high recall value but
also has a high FAR value and vice versa.

Again, for the line-level evaluation we use the metrics used by JITLine [36].
TopK accuracy measures the ratio of actual buggy lines to the Top K lines. A
high ratio suggests that many of the buggy lines are ranked at the top.
Recall@20%LOC measures the ratio of actual buggy lines that can be found given
only top 20% changed lines of code. Effort@20%Recall measure the amount of
effort made by the developers to identify top20% of buggy lines in a commit. A
higher value means that more effort is needed. Initial False Alarm (IFA)
measures the amount of clean lines of a commit that need to be inspected
before identifying an actual buggy line.

TABLE II: Statistics of evaluated projects. Project | Lang. | Commits | Buggy |  | Bug-fix
---
delay
| No. of commits
---
| No. of commits
---
|  |  |  |  | Min | Max | Min | Max
activemq | Java | 10,144 | 16.4% | 428 | 206 | 10,144 | 206 | 1,849
camel | Java | 49,473 | 8.7% | 261 | 1,173 | 49,473 | 1,173 | 4,442
hadoop | Java | 18,179 | 5.2% | 277 | 204 | 18,179 | 204 | 3,099
hbase | Java | 18,266 | 21.5% | 187 | 258 | 18,266 | 258 | 2,075
hive | Java | 15,100 | 35.3% | 158 | 96 | 15,100 | 96 | 2,352
lucene | Java | 29,182 | 9.5% | 382 | 66 | 29,182 | 66 | 4,203
OpenStack | Py. | 13,304 | 12.2% | 280 | 616 | 13,303 | 616 | 6,291
Qt | C++ | 25,704 | 7.1% | 526 | 3,564 | 25,704 | 3,564 | 9,281

##  V Evaluation and Results

###  V-A Commit-level performance of IRJIT (RQ1)

Approach: We ran IRJIT on each project shown in Table II using a within-
project setting. For evaluation at any point in time, the project’s past forms
the training set, whereas the future forms test set. For an online learning
problem [50] such as JIT defect prediction, this kind of evaluation is more
realistic than an 80:20 split. It aims to replay an online setting where
changes arrive over time and predictions are only based on past changes. Newly
proposed JIT approaches should run their evaluation in a similar manner to
ours. In this way, performance estimates will be closer to what developers
might observe in a real-life setting.

Results: Figure 3 presents the performance of IRJIT on each evaluated project
using an increasing evaluation setup. Each boxplot represents the range of
performance values obtained by evaluating various models. Each model uses
training and test data from a different period during a project’s evolution
and runs using two independent classifiers from IRJIT; a KNN classifier and a
Threshold classifier. A quick look at Figure 3 shows that there is no clear
winner between classifiers. The Threshold classifier takes the lead in terms
of recall, but it comes at the cost of mis-classifications which affects the
AUC. One reason for the sudden performance variations of Threshold classifier
is the quality of chosen Tscore. Since we chose a Tscore that maximizes AUC,
there is no guarantee that all other metrics will achieve their best values.

On the other hand, the KNN classifier shows a more moderate and relatively
consistent performance across different metrics and projects. The highest
median value of AUC and F1 were also achieved by the KNN classifier and were
$0.69$ and $0.66$ for the hbase and hive project respectively. Across all
evaluated projects the median AUC is $0.65$ and $0.38$ for KNN classifier and
$0.65$ and $0.37$ for Threshold classifier. The best performance was achieved
by the Ensemble (of KNN and la) classifier as it got a median AUC of $0.85$
and median F1 score of $0.40$.

In our evaluation, the increasing models (Figure 3) accumulate changes over
time, which means that as time progresses, more commits arrive, and each
subsequent model is trained on a larger number of changes than the model prior
to it. This, however, can affect the model in two ways; either the model
learns to make better predictions by training on an increased volume of data,
or it loses its power due to concept drift [50] and more noise which affects
all online learning problems [21] [9]. Due to being trained on a higher number
of changes, increasing models show a limited variation in performance per
project, as a consequence of which the AUCs are generally above $0.6$ and FAR
and d2h dropped. However, since the new source code changes might be
irrelevant to very old changes, they just cause noise which prevents
increasing models from achieving their true potential.

Finding 1: IRJIT can identify buggy commits across 8 projects with a
reasonable performance obtaining a median AUC of 0.65 for both the Threshold
and the KNN classifier and 0.85 for the ensemble classifier. The median F1
scores were 0.37, 0.38 and 0.40 respectively for Threshold, KNN and ensemble
classifier respectively.

###  V-B Effect of recent changes on performance of IRJIT (RQ2)

Approach: We now evaluate the performance of IRJIT by only using recent
changes. Instead of using the entire past for training, the changes used as
part of training set now come from time period closest to the test period.
This evaluation shows the performance of IRJIT when entire past data is not
available, or developers do not want to use it for any reason.

Figure 3: Performance of IRJIT increasing-models across 8 different projects
and 6 different performance metrics.  Figure 4: Performance of IRJIT constant-
models across 8 different projects and 6 different performance metrics. can be
removed

Results: The threshold classifier achieves an AUC of $0.65$ and F1 of $0.36$,
whereas the KNN classifier achieved an AUC of $0.63$ and an F1 score of
$0.35$. The Ensemble classifier remains on top with an AUC of $0.85$ and F1 of
$0.38$ which is a bit lower than RQ1. We compared the classifier results from
constant-models and increasing-models using a pairwise Wilcoxon signed-rank
test and did not find any significant differences between classifiers for any
metric.

Similar to increasing-models (RQ1), there are large performance variations
within each project, which are a consequence of the changing size and
characteristics of commits used for evaluating each model. Although, one might
think that data size is relatively less of an issue for constant-models
evaluation because they train only on data from the latest period. However, we
observed large variations in the number of commits coming from each time
period. For example, we found that commits arrive with a high frequency during
the early development stages of a project and later they become sparse.
Consequently, earlier time periods have a different distribution of commits
than the later ones which can affect performance. Similarly, the number of
buggy instances present in the data of each time period also vary drastically.
Most bugs appear early in the life cycle, and once the project matures, the
frequency of bug occurrences also reduces. These two factors lead to
significant variations in performance across periods. Based on their knowledge
about the project, the developers can better decide how much past data might
be most helpful in building their JIT models.

Finding 2: There is a slight drop in the performance of IRJIT when models are
trained only on the recent changes. However, this difference is project
specific, i.e., across periods, and the cumulative effect across projects is
negligible. According to Wilcoxon signed-rank the differences are not
statistically significant.

###  V-C Comparison of IRJIT with baseline JITLine (RQ3)

Approach: Before we compare IRJIT and JITLine we wanted to understand how the
missing categories of review and reviewer metrics will impact JITLine’s
performance. So we reran original JITLine (JITLineorig.) and JITLinerf
following a constant-models evaluation setup for the OpenStack and Qt project.
After that, we evaluate JITLinerf using both increasing and constant
evaluation setup from RQ1 and RQ2.

Results: JITLinerf, which is our simplified review-free version of JITLine,
performs similar to original JITLine on the OpenStack and Qt dataset. The F1
and AUC remains unchanged for OpenStack, but in case of Qt, F1 increased by
$0.03$ whereas AUC dropped by $0.02$, as shown in Table III. The $\Delta$
shows magnitude and direction of change for each metric. Contrary to what is
expected, the review and reviewer metrics do not change prediction results
significantly.

Finding 3: According to the pairwise Wilcoxon signed-rank test, there is no
difference between JITLinerf and JITLineorig., which means JITLine is expected
to maintain its good performance on remaining projects even without review
metrics.

TABLE III: JITLineorig. vs JITLinerf median values from constant-model evaluation  | Project | Prec. | Rec. | F1 | AUC | FAR | d2h
---|---|---|---|---|---|---|---
JITLineorig. | OpenStack | 0.47 | 0.06 | 0.10 | 0.78 | 0.01 | 0.67
JITLinerf | OpenStack | 0.50 | 0.05 | 0.10 | 0.78 | 0.01 | 0.67
${\Delta}$ | OpenStack | +0.03 | -0.01 | 0.00 | 0.00 | 0.00 | 0.00
JITLineorig. | Qt | 0.43 | 0.12 | 0.19 | 0.78 | 0.01 | 0.62
JITLinerf | Qt | 0.31 | 0.20 | 0.22 | 0.76 | 0.03 | 0.57
${\Delta}$ | Qt | -0.12 | +0.08 | +0.03 | -0.02 | +0.02 | -0.05
Figure 5: Comparison of IRJIT vs JITLine for increasing-models across 8
projects and 6 metrics Figure 6: Comparison of IRJIT vs JITLine for constant-
models across 8 projects and 6 metrics

Figure 5 and Figure 6, compares IRJIT and JITLine results for increasing-
models and constant-models. Our approach outperforms JITLinerf for the
OpenStack and Qt project. There is an improvement of $3.4\%$ \- $6.1\%$ in
terms of the AUC metric when JITLine is compared with Ensemble classifier.
However, this trend does not hold for the remaining $6$ projects, where
JITLinerf mostly performs better than IRJIT with occasional drops in
performance. The median values of different metrics presented in Table IV are
also close, but JITLine generally achieves higher values than the Ensemble
classifier especially for F1.

Finding 4: IRJIT performs better than JITLine for the OpenStack and Qt
project. Across the remaining $6$ projects, it also achieves a reasonable
performance, but it does not outperform JITLine.

To statistically evaluate the differences between JITLinerf and our approach,
we used pairwise Wilcoxon signed-rank test [46], which is a non-parametric
test to compare two samples, along with Bonferroni correction [3]. The null
hypothesis of the test is that there is no difference between the two groups.
We could not reject the null hypothesis for AUC and d2h but for precision, F1,
and FAR we were able to reject the null hypothesis in favour of JITLine. For
recall the hypothesis was rejected in favour of IRJIT. Additionally, we use
Cliff’s delta $|\delta|$ [46] to measure the effect sizes of the observed
differences. The effect sizes are negligible for $|\delta|$ $<$ $0.147$, small
for $0.147$ $\leq$ $|\delta|$ $<$ $0.33$, medium for $0.33$ $\leq$ $|\delta|$
$<$ $0.474$, and large for $|\delta|$ $\geq$ $0.474$. The effect sizes were
small for AUC, F1 and recall, negligible for d2h but large for precision and
FAR.

Finding 5: According to Wilcoxon signed-rank test there is no statistically
significant difference between IRJIT and JITLine in terms of AUC metric.
However, there is a difference in terms of F1 which according to Cliff’s delta
is small.

Table IV presents standard deviation (SD) across different runs of IRJIT and
JITLine. The variation is not specific to a particular JIT approach. It
further reiterates the variation-related discussion from RQ1 and confirms that
there will be some variation in performance across evaluations, whether models
are trained on the latest data or the entire past. However, complementing
models with data from old periods helps mitigate variation to some extent,
e.g., see the drop in SD column for JITLine from $0.24$ to $0.20$.

TABLE IV: Median (M) and standard deviation (SD) of IRJIT ensemble classifier and JITLine for constant (C) and increasing (I) model evaluation | JITLine | IRJIT
---|---|---
| Constant | Increasing | Constant | Increasing
Metrics | M | SD | M | SD | M | SD | M | SD
AUC | 0.86 | 0.06 | 0.87 | 0.06 | 0.85 | 0.04 | 0.85 | 0.04
F1 | 0.44 | 0.24 | 0.48 | 0.20 | 0.38 | 0.19 | 0.40 | 0.19
Prec. | 0.50 | 0.21 | 0.50 | 0.19 | 0.30 | 0.17 | 0.28 | 0.18
Recall | 0.43 | 0.27 | 0.60 | 0.24 | 0.57 | 0.23 | 0.62 | 0.20
FAR | 0.04 | 0.08 | 0.06 | 0.08 | 0.25 | 0.20 | 0.28 | 0.17
d2h | 0.41 | 0.17 | 0.29 | 0.15 | 0.37 | 0.12 | 0.35 | 0.10

###  V-D Line-level performance of IRJIT (RQ4)

Approach: For this evaluation, we use the line-level ground-truth data
provided by the JITLine authors for OpenStack and Qt project. For the
remaining $6$ projects we collected line-level data following JITLine’s
methodology. We obtained the diffs for each clean commit and labeled the
modified or deleted lines of that commit as buggy, and the remaining lines
were labeled clean. After collecting data, we rank buggy lines of each
predicted buggy commit using the tf–idf term weights or importance score of
tokens obtained from Elasticsearch Explain API. We parse the Explain API
output to identify the tokens that contributed to the prediction of a buggy
commit. Once identified, the lines modified in a commit were searched for all
the tokens and ranked according to the number of occurrences of distinct
tokens found in each line. The ranked lines are then returned to the developer
for inspection.

Results: Figure 7 presents the results of line-level evaluation for 4 metrics.
The median Top-10 accuracy of JITLine is $0.30$ which is lower than our
approach which achieves $0.70$. Our approach can identify buggy lines with a
median of Recall@20%LOC of $0.25$ which is $31\%$ higher than JITLine.
Similarly, our approach uses $61\%$ less effort than JITLine to find top
$20\%$ buggy lines. IRJIT also has a median IFA of $0$ indicating that a buggy
line is always ranked as the top location. The Wilcoxon signed rank test
confirms that there is a statistically significant difference between all
line-level metrics. Cliff’s delta suggests that these differences are large
for all metrics.

Finding 6: For line-level evaluation, IRJIT outperforms JITLine with a
statistically significant difference for all metrics.

Figure 7: Comparison of IRJIT vs JITLine constant-models for ranking buggy
lines of an identified buggy commit.

##  VI Discussion

In this section, we discuss implications of our work, its limitations and
opportunities for future work.

###  VI-A Implications

Researchers should focus on designing simple, online, and explainable
approaches for JIT defect prediction. IRJIT is one such approach that
developers can use for identifying risky commits and ranking source code lines
by bugginess. Our results show that IRJIT is similar to state-of-the-art
approach JITLine in terms of AUC, and, both approaches can be applied to any
project but there are some obvious trade-offs.

IRJIT is simple compared to state-of-the-art approach JITLine because it only
uses source code changes along with basic IR methods. These changes are
available in the project’s source code repository so IRJIT can be easily
deployed with limited expertise and effort. Our work demonstrates that IR-
based methods are competitive out of the box, suggesting that further tuning
and using them in combination with other project-specific features (e.g.,
metrics) are likely to bridge the gap while requiring significantly less
training and allowing the use of explainable models.

IRJIT is explainable so a developer using IRJIT can see past commits that lead
to a buggy prediction, whereas a developer relying on JITLine cannot because
the commits are invented using SMOTE and are not real documents. For line-
level prediction, JILine uses LIME [42] to learn a model locally around the
prediction which is an additional cost on top of the original model. IRJIT
improved over JITLine by eliminating the need to tune extensively through
SMOTE and train an altogether new model such as LIME for line-level
prediction. IRJIT uses a single IR model to identify a buggy commit and rank
its lines by defectiveness. It then retrieves past changes responsible for the
prediction and associated fixes for those changes from the corpus. In short,
anything flagged by IRJIT would have a concrete, context relevant example. In
future work, we plan to conduct a user study to assess the usefulness of IRJIT
for developers. We also plan to improve fixes keeping in view the contextual
factors.

Researchers should make sure that their JIT approach is suitable for online
deployment without considerable retraining. We show that IRJIT works in an
online fashion, and that our best performing models do not require any tuning
or retraining but just updating indexes when new commits come. We investigated
JIT in an online scenario where new changes arrive over time and can be used
for updating the JIT models. Our approach supports incremental updates without
retraining the prediction model, and hence it is very adaptive and scalable.
Contrarily, the baseline approach, JITLine relies on extensive retraining of
their models through SMOTE and Differential Evolution. Therefore, JITLine can
be more computationally expensive for online deployment, and might slow down
the overall prediction process. However, JITLine might offer higher precision
compared to IRJIT so a developer who wants very precise models and has
resources for expensive retraining may choose JITLine over IRJIT.

Researchers should conduct a realistic evaluation of JIT approaches. Prior
studies evaluated their JIT defect prediction approaches using the traditional
80:20 split for the data. Through an evaluation at different points in time,
we show that both IRJIT and JITLine exhibit varying performance along time,
and an 80:20 evaluation could overestimate (or underestimate) performance.
Therefore, we insist that future studies should be evaluated following an
evaluation methodology that represents a realistic application of JIT from a
development perspective.

###  VI-B Limitations and Opportunities for Future Work

####  VI-B1 Impact of tuning

Our best performing model is an ensemble classifier of KNN and la and it does
not involve any tuning. However, our Threshold classifier relies on a
threshold Tscore that is optimized for AUC using the training dataset. We
observed a decent improvement in our AUC values after tuning for Tscore. The
Tscore is project-specific because of the differences in the size of changes
and underlying vocabulary, but it is relatively quick and easy to tune for
Tscore. One can also train for other metrics such as F1 or precision. In
future work, we plan to investigate more advanced methods of adjusting the
threshold. We will also examine the impact of changes in training data on the
stability of the threshold.

As for the KNN classifier we test for different K values but we observed a
drop in performance for higher values of K. The best results were achieved for
$K=3$, therefore, IRJIT uses $K=3$ by default.

####  VI-B2 Impact of gap

To avoid evaluating on data that has not received its correct labels, we leave
a gap between our training and test sets while evaluating all projects except
OpenStack and Qt. OpenStack and Qt dataset was curated by following strict
quality criteria which accounts for such problems. For remaining projects, we
chose a gap interval following recommendations of prior work [55][51][57][9].
This gap means we waste some of the data that could be useful to our models
leading to an underestimate of performance for both IRJIT and JITLine.

####  VI-B3 Reliance on SZZ

We rely on SZZ algorithm to obtain the ground truth dataset for evaluation. We
as part of the software engineering community are aware of the shortcomings of
SZZ algorithm that it only identifies large defects that get reported to the
issue tracking system and therefore are important enough to be fixed. Other
kinds of datasets such as ManySStuBs4J [23] are not widely available so we had
to rely on SZZ for data extraction. In the area of defect prediction,
researchers commonly rely on SZZ e.g., it was used to extract OpenStack and Qt
dataset and the recent JIT work [57][36][9][50] also used SZZ. We therefore
think that it is out of the scope of our paper to design a new way of
identifying buggy commits. Instead, we implore the software engineering
community to come together and take up the challenge of improving the accuracy
of SZZ algorithm or to propose new ways of identifying representative buggy
commits.

###  VI-C Threats to Validity

##### Internal validity

We used an open-source implementation [7] of the original SZZ algorithm [47]
along with the later enhancements proposed by Williams and Spacco [54]. They
used the DiffJ tool to make SZZ aware of different types of format changes
such as reordering and renaming of parameters. This implementation is used by
many existing works [48] [52], but despite this we cannot rule out that the
SZZ data we obtained contains errors and omissions.

##### External validity

The external validity of our study is hampered due to the use of a limited
number of projects for evaluation. However, to improve the external validity
of our study, we made sure that the chosen projects have different languages,
sizes and complexities.

##### Conclusion validity

To keep our comparison consistent with the baseline approach, we chose the
same performance measures as JITLine. Furthermore, we optimized Tscore for AUC
because it measures the discriminatory power of a model — ability to
differentiate between clean and buggy commits on varying thresholds. A
threshold-independent measure is more stable in the presence of imbalance, and
since JIT datasets are inherently imbalanced, many prior studies used AUC to
measure model performance.

##  VII Conclusion

In this paper, we proposed IRJIT which is an explainable, and deterministic
information retrieval approach for JIT defect prediction. IRJIT classifies a
new commit-level change as buggy or clean, and shows past changes that can be
blamed for the predicted bug. IRJIT does not require extensive retraining and
can be updated in an online fashion making it suitable for use in real-world
scenarios. We empirically evaluated our approach on OpenStack, Qt, and 6 other
open-source projects using an evaluation setting that represents realism. Our
comparison with a state-of-the-art ML approach, JITLine shows that IRJIT
outperforms JITLine at line-level prediction. However, at commit level it only
does better than JITLine for OpenStack and Qt project. Despite, IRJIT is more
practical and can be appropriately integrated into existing development
processes to facilitate JIT defect identification.

## References

  * [1] Commit hooks.  https://www.atlassian.com/git/tutorials/git-hooks, August 2022.
  * [2] Gerrit review database.  https://www.gerritcodereview.com/, August 2022.
  * [3] Hervé Abdi et al.  Bonferroni and šidák corrections for multiple comparisons.  Encyclopedia of measurement and statistics, 3:103–107, 2007.
  * [4] Apache.  Apache Lucene Library.  https://lucene.apache.org/core/, August 2022.
  * [5] Apache.  Lucene scoring formula.  https://lucene.apache.org/core/3_5_0/scoring.html, August 2022.
  * [6] Abdul Ali Bangash, Hareem Sahar, Abram Hindle, and Karim Ali.  On the time-based conclusion stability of cross-project defect prediction models.  Empirical Software Engineering, 25(6):5047–5083, 2020.
  * [7] Markus Borg, Oscar Svensson, Kristian Berg, and Daniel Hansson.  Szz unleashed: An open implementation of the szz algorithm-featuring example usage in a study of just-in-time bug prediction for the jenkins project.  In Proceedings of the 3rd ACM SIGSOFT International Workshop on Machine Learning Techniques for Software Quality Evaluation, pages 7–12, 2019\.
  * [8] c2nes.  Javalang Python Library.  https://github.com/c2nes/javalang/blob/master/javalang/, August 2022\.
  * [9] George G Cabral, Leandro L Minku, Emad Shihab, and Suhaib Mujahid.  Class imbalance evolution and verification latency in just-in-time software defect prediction.  In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pages 666–676. IEEE, 2019.
  * [10] Joshua Charles Campbell, Abram Hindle, and José Nelson Amaral.  Syntax errors just aren’t natural: improving error reporting with language models.  In Proceedings of the 11th Working Conference on Mining Software Repositories, pages 252–261, 2014.
  * [11] Joshua Charles Campbell, Eddie Antonio Santos, and Abram Hindle.  The unreasonable effectiveness of traditional information retrieval in crash report deduplication.  In 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR), pages 269–280. IEEE, 2016.
  * [12] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer.  Smote: synthetic minority over-sampling technique.  Journal of artificial intelligence research, 16:321–357, 2002.
  * [13] Hoa Khanh Dam, Truyen Tran, and Aditya Ghose.  Explainable software analytics.  In Proceedings of the 40th International Conference on Software Engineering: New Ideas and Emerging Results, pages 53–56, 2018.
  * [14] Elasticsearch.  More like this.  https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-mlt-query.html, August 2022.
  * [15] Elasticsearch.  Pattern analyzer.  https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pattern-analyzer.html, August 2022.
  * [16] Elasticsearch.  Shingle token filter.  https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-shingle-tokenfilter.html, August 2022.
  * [17] Elasticsearch.  Whitespace tokenizer.  https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-whitespace-tokenizer.html, August 2022.
  * [18] Abram Hindle, Earl T Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu.  On the naturalness of software.  In 2012 34th International Conference on Software Engineering (ICSE), pages 837–847. IEEE, 2012.
  * [19] Thong Hoang, Hoa Khanh Dam, Yasutaka Kamei, David Lo, and Naoyasu Ubayashi.  Deepjit: an end-to-end deep learning framework for just-in-time defect prediction.  In 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), pages 34–45. IEEE, 2019.
  * [20] Thong Hoang, Hong Jin Kang, David Lo, and Julia Lawall.  Cc2vec: Distributed representations of code changes.  In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pages 518–529, 2020.
  * [21] T Ryan Hoens, Robi Polikar, and Nitesh V Chawla.  Learning from streaming data with concept drift and imbalance: an overview.  Progress in Artificial Intelligence, 1(1):89–101, 2012.
  * [22] Yasutaka Kamei, Emad Shihab, Bram Adams, Ahmed E Hassan, Audris Mockus, Anand Sinha, and Naoyasu Ubayashi.  A large-scale empirical study of just-in-time quality assurance.  IEEE Transactions on Software Engineering, 39(6):757–773, 2012\.
  * [23] Rafael-Michael Karampatsis and Charles Sutton.  How often do single-statement bugs occur? the manysstubs4j dataset.  In Proceedings of the 17th International Conference on Mining Software Repositories, pages 573–577, 2020.
  * [24] Kisub Kim, Dongsun Kim, Tegawendé F Bissyandé, Eunjong Choi, Li Li, Jacques Klein, and Yves Le Traon.  FaCoY: a code-to-code search engine.  In Proceedings of the 40th International Conference on Software Engineering, pages 946–957, 2018.
  * [25] Sunghun Kim, Kai Pan, and EE James Whitehead Jr.  Memories of bug fixes.  In Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering, pages 35–45, 2006.
  * [26] Sunghun Kim, E James Whitehead, and Yi Zhang.  Classifying software changes: Clean or buggy?  IEEE Transactions on Software Engineering, 34(2):181–196, 2008\.
  * [27] Sunghun Kim, Thomas Zimmermann, Kai Pan, E James Jr, et al.  Automatic identification of bug-introducing changes.  In 21st IEEE/ACM international conference on automated software engineering (ASE’06), pages 81–90. IEEE, 2006.
  * [28] Erik Linstead, Sushil Bajracharya, Trung Ngo, Paul Rigor, Cristina Lopes, and Pierre Baldi.  Sourcerer: mining and searching internet-scale software repositories.  Data Mining and Knowledge Discovery, 18(2):300–336, 2009.
  * [29] Yi Liu, Taghi M Khoshgoftaar, and Naeem Seliya.  Evolutionary optimization of software quality modeling with multiple repositories.  IEEE Transactions on Software Engineering, 36(6):852–864, 2010\.
  * [30] Sifei Luan, Di Yang, Celeste Barnaby, Koushik Sen, and Satish Chandra.  Aroma: Code recommendation via structural code search.  Proceedings of the ACM on Programming Languages, 3(OOPSLA):1–28, 2019.
  * [31] Shane McIntosh and Yasutaka Kamei.  Are fix-inducing changes a moving target? a longitudinal case study of just-in-time defect prediction.  IEEE Transactions on Software Engineering, 44(5):412–428, 2017\.
  * [32] Audris Mockus and David M Weiss.  Predicting risk of software changes.  Bell Labs Technical Journal, 5(2):169–180, 2000.
  * [33] Mathieu Nayrolles and Abdelwahab Hamou-Lhadj.  Clever: combining code metrics with clone detection for just-in-time fault prevention and resolution in large industrial projects.  In Proceedings of the 15th International Conference on Mining Software Repositories, pages 153–164, 2018.
  * [34] Annibale Panichella, Rocco Oliveto, and Andrea De Lucia.  Cross-project defect prediction models: L’union fait la force.  In 2014 Software Evolution Week-IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE), pages 164–173. IEEE, 2014.
  * [35] Strategic Planning.  The economic impacts of inadequate infrastructure for software testing.  National Institute of Standards and Technology, 2002.
  * [36] Chanathip Pornprasit and Chakkrit Kla Tantithamthavorn.  Jitline: A simpler, better, faster, finer-grained just-in-time defect prediction.  In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), pages 369–379. IEEE, 2021.
  * [37] Michael Pradel and Koushik Sen.  DeepBugs: a learning approach to name-based bug detection.  Proceedings of the ACM on Programming Languages, 2(OOPSLA):1–25, 2018.
  * [38] Ranjith Purushothaman and Dewayne E Perry.  Toward understanding the rhetoric of small source code changes.  IEEE Transactions on Software Engineering, 31(6):511–526, 2005\.
  * [39] Chaiyong Ragkhitwetsagul and Jens Krinke.  Siamese: scalable and incremental code clone search via multiple code representations.  Empirical Software Engineering, 24(4):2236–2284, 2019.
  * [40] Mohammad Masudur Rahman and Chanchal K Roy.  Improving ir-based bug localization with context-aware query reformulation.  In Proceedings of the 2018 26th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pages 621–632, 2018.
  * [41] Baishakhi Ray, Vincent Hellendoorn, Saheel Godhane, Zhaopeng Tu, Alberto Bacchelli, and Premkumar Devanbu.  On the “naturalness” of buggy code.  In 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE), pages 428–439. IEEE, 2016.
  * [42] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  ” why should i trust you?” explaining the predictions of any classifier.  In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016.
  * [43] Christoffer Rosen, Ben Grawi, and Emad Shihab.  Commit guru: analytics and risk prediction of software commits.  In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, pages 966–969, 2015.
  * [44] Gerard Salton and Robert Kenneth Waldstein.  Term relevance weights in on-line information retrieval.  Information Processing & Management, 14(1):29–35, 1978.
  * [45] Eddie Antonio Santos, Joshua Charles Campbell, Dhvani Patel, Abram Hindle, and José Nelson Amaral.  Syntax and sensibility: Using language models to detect and correct syntax errors.  In 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 311–322. IEEE, 2018.
  * [46] Forrest Shull, Janice Singer, and Dag IK Sjøberg.  Guide to advanced empirical software engineering.  Springer, 2007.
  * [47] Jacek Śliwerski, Thomas Zimmermann, and Andreas Zeller.  When do changes induce fixes?  ACM sigsoft software engineering notes, 30(4):1–5, 2005.
  * [48] Jeongju Sohn, Yasutaka Kamei, Shane McIntosh, and Shin Yoo.  Leveraging fault localisation to enhance defect prediction.  In 2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 284–294. IEEE, 2021.
  * [49] Karen Spärck Jones Stephen E. Robertson.  Okapi BM25 algorithm.  https://en.wikipedia.org/wiki/Okapi_BM25, August 2022.
  * [50] Sadia Tabassum, Leandro L Minku, Danyi Feng, George G Cabral, and Liyan Song.  An investigation of cross-project learning in online just-in-time software defect prediction.  In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE), pages 554–565. IEEE, 2020.
  * [51] Ming Tan, Lin Tan, Sashank Dara, and Caleb Mayeux.  Online defect prediction for imbalanced data.  In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, volume 2, pages 99–108. IEEE, 2015.
  * [52] Bailey Vandehei, Daniel Alencar Da Costa, and Davide Falessi.  Leveraging the defects life cycle to label affected versions and defective classes.  ACM Transactions on Software Engineering and Methodology (TOSEM), 30(2):1–35, 2021.
  * [53] Zhiyuan Wan, Xin Xia, Ahmed E Hassan, David Lo, Jianwei Yin, and Xiaohu Yang.  Perceptions, expectations, and challenges in defect prediction.  IEEE Transactions on Software Engineering, 46(11):1241–1266, 2018\.
  * [54] Chadd Williams and Jaime Spacco.  Szz revisited: verifying when changes induce fixes.  In Proceedings of the 2008 workshop on Defects in large software systems, pages 32–36, 2008.
  * [55] Meng Yan, Xin Xia, Yuanrui Fan, Ahmed E Hassan, David Lo, and Shanping Li.  Just-in-time defect identification and localization: A two-phase framework.  IEEE Transactions on Software Engineering, 2020.
  * [56] Xinli Yang, David Lo, Xin Xia, and Jianling Sun.  Tlel: A two-layer ensemble learning approach for just-in-time defect prediction.  Information and Software Technology, 87:206–220, 2017.
  * [57] Zhengran Zeng, Yuqun Zhang, Haotian Zhang, and Lingming Zhang.  Deep just-in-time defect prediction: how far are we?  In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 427–438, 2021.
  * [58] Shufan Zhou, Beijun Shen, and Hao Zhong.  Lancer: Your code tell me what you need.  In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 1202–1205. IEEE, 2019.