# RLocator: Reinforcement Learning for Bug Localization

Partha Chakraborty1, Mahmoud Alfadel2, Meiyappan Nagappan3
David R. Cheriton School of Computer Science, University of Waterloo
{p9chakra1, malfadel2<EMAIL_ADDRESS>
###### Abstract

Software developers spend a significant portion of time fixing bugs in their
projects. To streamline this process, bug localization approaches have been
proposed to identify the source code files that are likely responsible for a
particular bug. Prior work proposed several similarity-based machine-learning
techniques for bug localization. Despite significant advances in these
techniques, they do not directly optimize the evaluation measures. We argue
that directly optimizing evaluation measures can positively contribute to the
performance of bug localization approaches.

Therefore, In this paper, we utilize Reinforcement Learning (RL) techniques to
directly optimize the ranking metrics. We propose RLocator, a Reinforcement
Learning-based bug localization approach. We formulate RLocator using a Markov
Decision Process (MDP) to optimize the evaluation measures directly. We
present the technique and experimentally evaluate it based on a benchmark
dataset of 8,316 bug reports from six highly popular Apache projects. The
results of our evaluation reveal that RLocator achieves a Mean Reciprocal Rank
(MRR) of 0.62, a Mean Average Precision (MAP) of 0.59, and a Top 1 score of
0.46. We compare RLocator with two state-of-the-art bug localization tools,
FLIM and BugLocator. Our evaluation reveals that RLocator outperforms both
approaches by a substantial margin, with improvements of 38.3% in MAP, 36.73%
in MRR, and 23.68% in the Top K metric. These findings highlight that directly
optimizing evaluation measures considerably contributes to performance
improvement of the bug localization problem.

###### Index Terms:

Reinforcement Learning, Bug Localization, Deep Learning

##  1 Introduction

Software bugs are an inevitable part of software development. Developers spend
one-third of their time debugging and fixing bugs [1]. After a bug
report/issue has been filed, the project team identifies the source code files
that need to be inspected and modified to address the issue. However, manually
locating the files responsible for a bug is expensive (in terms of time and
resources), especially when there are a lot of files and bug reports.
Moreover, the number of bugs reported is often higher than the number of
available developers [2]. Consequently, the fix-time and maintenance costs
rise when the customer satisfaction rate decreases [3].

Bug Localization is a method that refers to identifying the source code files
where a particular bug originated. Given a bug report, bug localization
approaches utilize the textual information in the bug report, and the project
source code files to shortlist the potentially buggy files. Prior work has
proposed various Information Retrieval-based Bug Localization (IRBL)
approaches to help developers speed up the debugging process (e.g.,
Deeplocator [4], CAST [5], KGBugLocator [6], BL-GAN [7]).

One common theme among these approaches is that they follow a similarity-based
approach to localize bugs. Such techniques measure the similarity between bug
reports and the source code files. For estimating similarity, they use various
methods such as cosine distance [8], Deep Neural Networks (DNN) [9], and
Convolutional Neural Networks (CNN) [5]. Then, they rank the source code files
based on their similarity score. In the training phase of these approaches,
the model learns to optimize the similarity metrics. In contrast, in the
testing phase, the model is tested with ranking metrics (e.g., Mean Reciprocal
Rank (MRR) or Mean Average Precision (MAP)).

While most of these approaches showed promising performance, they optimize a
metric that indirectly represents the performance metrics. Prior studies [10,
11, 12, 13] found that direct optimization of evaluation measures
substantially contributes to performance improvement of ranking problems.
Direct optimization is also efficient compared to optimizing indirect metrics
[13]. Hence, we argue that it is challenging for the solutions proposed by
prior studies to sense how a wrong prediction would affect the performance
evaluation measures [10]. In other words, if we use the retrieval metrics
(e.g., MAP) in the training phase, the model will learn how each prediction
will impact the evaluation metrics. A wrong prediction will change the rank of
the source code file and ultimately impact the evaluation metrics.

Reinforcement Learning (RL) is a sub-category of machine learning methods
where labeled data is not required. In RL, the model is not trained to predict
a specific value. Instead, the model is given a signal about a right or wrong
choice in training [14]. Based on the signal, the model updates its decision.
This allows RL to use evaluation measures such as MRR and MAP in the training
phase and directly optimize the evaluation metrics. Moreover, because of using
MRR/MAP as a signal instead of a label, the problem of overfitting will be
less prevalent. Markov Decision Process (MDP) is a foundational element of RL.
MDP is a mathematical framework that allows the formalization of discrete-time
decision-making problems [15]. Real-world problems often need to be formalized
as MDP to apply RL.

In this paper, we present _RLocator_ , an RL technique for localizing software
bugs in source code files. We formulate RLocator into an MDP. In each step of
the MDP, we use MRR and MAP as signals to guide the model to optimal choice.
We evaluate RLocator on a benchmark dataset of six Apache projects and find
that, compared with existing state-of-the-art bug localization techniques,
RLocator achieves substantial performance improvement.

The main contributions of our work are as follows:

  * •

We present RLocator, an RL-based software bug localization approach. The key
technical novelty of RLocator is using RL for bug localization, which includes
formulating the bug localization process into an MDP.

  * •

We provide an experimental evaluation of RLocator with 8,316 bug reports from
six Apache projects. When RLocator can localize, it achieves an MRR of 0.49 -
0.62, MAP of 0.47 - 0.59, and Top 1 of 0.38 - 0.46 across all studied
projects. Moreover, we compare its performance with state-of-the-art bug
localization approaches. Regarding MAP, MRR, and Top K, RLocator surpasses
FLIM [16] by margins of 38.3%, 36.73%, and 23.68%, respectively. Additionally,
RLocator surpasses BugLocator [3] by margins of 56.86%, 41.51%, and 26.32% in
terms of MAP, MRR, and Top K, respectively.

Paper organization. The rest of the paper is organized as follows. Section 2
discusses the necessary background of this study. In Section 3, we present our
approach, RLocator. Section 4 discusses the dataset and the evaluation metric
used in the study. Section 5 presents the results of the experiments. Section
6 situates our work with respect to the literature on bug localization.
Section 7 explains the threats to the validity of our results. Finally, in
Section 8, we conclude and outline possible future work.

##  2 Background

In this section, we describe terms related to the bug localization problem,
which we use throughout our study. Also, we present an overview of
reinforcement learning.

###  2.1 Bug Localization System

A typical bug localization system utilizes several sources of data, e.g., bug
reports, stack traces, and logs, to identify the responsible source code
files. One particular challenge of the system is that the bug report contains
natural language, whereas source code files are written in a programming
language.

Typically, bug localization systems identify whether a bug report relates to a
source code file. To do so, the system extracts features from both the bug
report and the source code files. Previous studies used techniques such as
N-gram [17, 18] and Word2Vec [19, 20] to extract features (embedding) from bug
reports and source code files. Other studies (e.g., Devlin et al. [21])
introduced the transformer-based model BERT which has achieved higher
performance than all the previous techniques. One of the reasons transformer-
based models perform better in extracting textual features is that the
transformer uses multi-head attention, which can utilize long context while
generating embedding. Previous studies have proposed a multi-modal BERT model
[22] for programming languages, which can extract features from both bug
reports and source code files.

Figure 1: Sample bug report.

Figure 1 represents a sample bug
report111https://bugs.eclipse.org/bugs/show_bug.cgi?id=134471 from the AspectJ
project. A bug report mainly contains information related to unexpected
behavior and how to reproduce it. It mainly includes a bug id, title, textual
description of the bug, and version of the codebase where the bug exists. The
bug report may have an example of code or stack trace, or logs. A bug
localization system retrieves all the source code files from a source code
repository at that particular version. For example, assume we have 100 source
code files in a repository in a specific version. After retrieving 100 files
from that version, the system will estimate the relevance between the bug
report and each of the 100 files. The relevance can be measured in several
ways. For example, a naive system can check how many words of the bug report
exist in each source code file. A sophisticated system can compare embeddings.
A BERT-based system may generate embedding for each of the 100 files and the
bug report. Then it can measure the relevance between them by cosine distance
[23]. After relevance estimation, the system scales the relevance of all the
files. To compare the relevance of different files, some metrics, such as
cosine distance, already provide us with a scaled value. Thus, it can skip
this step in some cases. After scaling the relevance, the system ranks the
files based on their relevance. The ranked list of files is the final output
of a bug localization system that developers will use.

###  2.2 Reinforcement Learning

In Reinforcement Learning (RL), the agent interacts with the environment
through observation. Formally, an observation is called “State,” $S$. In each
state, at time $t$, $S_{t}$, the agent takes action $A$ based on its
understanding of the state. Then, the environment provides feedback/reward
$\Re$ and transfers the agent into a new state $S_{t+1}$. The agent’s strategy
to determine the best action, which will eventually lead to the highest
cumulative reward, is referred to as _policy_ [24, 14].

The cumulative reward (until the goal/end) that an agent can get if it takes a
specific action in a certain state is called _Q value_. The function that is
used to estimate the _Q value_ is often referred as _Q function_ or _Value
function_.

In RL, an agent starts its journey from a starting state and then goes forward
by picking the appropriate action. The journey ends in a pre-defined end
state. The journey from start to end state is referred to as _episode_.

From a high level, we can divide the state-of-the-art RL algorithms into two
classes. The first is the model-free algorithms, where the agent has no prior
knowledge about the environment. The agent learns about the environment by
interacting with the environment. The other type is the model-based algorithm.
In a model-based algorithm, the agent uses the reward prediction from the
model instead of interacting with the environment.

The bug localization task is quite similar to the model-free environment as we
cannot predict/identify the buggy files without checking the bug report and
source code files (without interacting with the environment). Thus, we use
model-free RL algorithms in this study. Two popular variants of model-free RL
algorithms are:

  * •

_Value Optimization_ : The agent tries to learn the Q value function in value
optimization approaches. The agent keeps the Q value function in memory and
updates it gradually. It consults the Q value function in a particular state
and picks the action that will give the highest value (reward). An example of
the value optimization-based approach is Deep Q Network (DQN) [14].

  * •

_Policy Optimization_ : In the Policy optimization approach, the agent tries
to learn the mapping between the state and the action that will result in the
highest reward. The agent will pick the action based on the mapping in a
particular state. An example of the policy optimization-based approach is
Advantage Actor-Critic (A2C) [25, 14].

A2C is a policy-based algorithm where the agent learns an optimized policy to
solve a problem. In Actor-Critic, the actor-model picks action. The future
return (reward) of action is estimated using the critic model. The actor model
uses the critic model to pick the best action in any state. Advantage actor-
critic subtracts a base value from the return in any timestep. A2C with
entropy adds the entropy of the probability of the possible action with the
loss of the actor model. As a result, in the gradient descent step, the model
tries to maximize the entropy of the learned policy. Maximization of entropy
ensures that the agent assigns almost an equal probability to an action with a
similar return.

##  3 RLocator: Reinforcement Learning for Bug Localization

Figure 2: Bug Localization as Markov Decision Process.

In this section, we discuss the steps we follow for designing the RLocator
approach. First, we explain (in Section 3.1) a filtration process required for
the approach. Then, we explain (in Section 3.2) the formulation steps of our
design of RLocator. We present the overview of our approach in Figure 2.

###  3.1 Filtering the number of source code files

The inputs to our bug localization tool are bug reports and source code files
associated with a particular version of a project repository. Software
projects maintain a repository for their bugs or issues (e.g., Jira, Github,
Bugzilla). The first component, the bug report, can be retrieved from those
issue repositories. We use the bug report to obtain the second component
(i.e., source code) by identifying the project version affected by the bug.
Typically, each bug report is associated with a version or commit SHA of the
project repository. After identifying the buggy version, we collect all source
code files from the specific version of the code repository. The number of
source code files in different versions of the repository can be different.
However, due to the nature of RL, we cannot pass a variable number of source
code files to the RL model. We overcome this limitation by sending a fixed
number ($k$) of the most relevant source code files for a bug report to the RL
model.

To identify the relevant files, we use ElasticSearch (ES). ES is a search
engine based on the Lucene search engine project. It is a distributed, open-
source search and analytic engine for all data types, including text. It
analyzes and indexes words/tokens for the textual match and uses BM25 to rank
the files matching the query. We use the ES index for identifying the topmost
k source code files related to a bug report. Following the study by Liu et el.
[26] (who used ES in the context of code search), we build an ES index using
the source code files and then queried the index using the bug report as the
query. Then, we picked the first $k$ files, which have the most textual
similarities with the bug report. We want to note that the goal of bug
localization is to get the relevant files to be ranked as close to the
$1^{st}$ rank as possible. Hence, metrics like MAP and MRR can measure the
performance of bug localization techniques. While one can argue why we not
only rely on ES to rank the relevant files, we find that the MAP and MRR of
using ES are poor. Our RL-based technique aims to rerank the output from ES to
get higher MAP and MRR scores.

One limitation of ES is that, in some cases, it returns irrelevant files in
the $k$ most relevant source code files. As there are no relevant files in the
first $k$ files, we cannot use those files for training/testing RLocator.
Also, including such data in the model can mislead developers, as the ranking
produced by the model will not be meaningful. This is because we will just be
reranking $k$ irrelevant files. Therefore, we first identify the cases where
ES returns no relevant files in the top $k$ files. To do so, we build an
XGBoost-based binary classifier model [27]. We pass the bug report and the top
$k$ relevant files returned by the ES to the model. The model aims to predict
whether a textual similarity-based ranking system (like ES) can rank the
relevant source code files in the top $k$ files or not.

TABLE I: Description and rationale of the selected features.

Feature |  Description |  Rationale
---|---|---
Bug Report Length |  Length of the bug report. |  Fan et al. [28] found that it is hard to localize bugs using a short bug report. A short bug report will contain little information about the bug. Thus it will be hard for the ElasticSearch to retrieve source code file responsible for this bug.
Source Code Length |  Median length of the source code files associated with a particular bug. Note that we calculate the string length of the source code files after removing code comments. |  Prior studies [29, 30] found that calculating textual similarity is challenging for long texts. Length of source code may contribute to the performance drop of ElasticSearch.
Stacktrace |  Availability of stack trace in bug report. |  Schroter et al. [31] found that stacktraces in bug reports can help the debugging process as they may contain useful information. Availability of stacktraces may improve the performance of ElasticSearch.
Similarity |  Ratio of similar tokens between a bug report and source code files |  Similarity indicates the amount of helpful information in the bug report. We calculate the similarity based on the equation presented in Section 5.

To build the model, we study the most important features associated with the
prediction task. We consult the related literature on the field of information
retrieval [29, 32, 30] and bug report classification [28] for feature
selection. The list of computed features is presented in Table I. For our
dataset, we calculate the selected features and trained the model using
10-fold cross-validation. The results show that our classifier model has a
precision of 0.78, a recall of 0.93, and an F1-score of 0.85. Additionally,
the model is able to correctly classify 91% of the dataset (there will be
relevant source code files in the top $k$ files returned by ES).

When XGBoost predicts that the relevant file will be in the top $k$ files,
then for those bug reports, we use our technique, RLocator which means
RLocator will work on the 91% of the dataset. We rerank the $k$ most relevant
files so that the MAP and MRR metrics are optimized.

###  3.2 Formulation of RLocator

After choosing $k$ most relevant files for each bug report and filtering out
bug reports where we are unlikely to find the relevant file in the top $k$, we
pass them to RLocator. Following prior studies [33, 34], we formulate RLocator
into a Markov Decision Process (MDP) by dividing the ranking problem into a
sequence of decision-making steps. A general RL model can be represented by a
tuple $\langle S,A,\tau,\Re,\pi\rangle$, which is composed of states, actions,
transition, reward, and policy, respectively. Figure 2 shows an overview of
our RLocator approach. Next, we describe the formulation steps of each
component of RLocator.

States: $S$ is the set of states. The RL model moves from one state to another
state until it reaches the end state. To form the states of the MDP, we apply
the following steps:

Input: The input of our MDP comprises two components, a bug report and the top
$k$ relevant source code files (documents) of a project repository. We use
CodeBERT [22], a transformer-based BERT model for converting the text to
embeddings. An embedding represents the text (source code/bug report) in a
multi-dimensional space. We have opted for BERT based model as it can utilize
long context. In our case, the source code files are long. Thus, a method
declared far away from its usage may not be captured in other embedding
techniques (e.g.; Word2Vec). However, because of the long context, BERT can
utilize that information. Moreover, Word2Vec generates embedding for a word,
which will not be updated in the inference phase based on the word’s context.
Contrary, the BERT model will generate the embedding for a sequence. Thus, it
can utilize the context in the inference phase too. In source code files, the
use of a variable depends on the scope. The variable name can be declared
again in another scope and may contain different objects. As the BERT model
generates embedding for sequence, the embedding will be different for each use
of the variable. CodeBERT has been trained on natural language and programming
language pairs and accepts input in both programming and natural languages. In
Figure 2, the embedding model generates embedding of the source code files,
$F_{1},F_{2},...$, and $F_{k}$, and the embedding of the bug report $R$.

Concatenation: After we obtain the embedding for the source codes and the bug
report, we concatenate them. Given our example in Figure 2, we concatenate the
embedding of $F_{1},F_{2},...$, and $F_{k}$ with the embedding of $R$
individually. This step leads us to obtain the corresponding concatenated
embedding $E_{1},E_{2},...$, and $E_{k}$, as shown in Figure 2.

Note that each state of the MDP comprises two lists: a candidate list and a
ranked list. The candidate list contains the concatenated list of embedding.
As shown in our example in Figure 2, the candidate list contains
$E_{1},E_{2},...$, and $E_{k}$. In the candidate list, source code embeddings
(code files and bug reports embedding concatenated together) are ranked
randomly. The other list is the ranked list of source code files based on
their relevance to the bug report $R$. Initially (at $State_{1}$), the
candidate list is full, and the ranked list is empty. In each state
transition, the model moves one embedding from the candidate list to the
ranked list based on their probability of being responsible for a bug. In the
final state, the ranked list will be full, and the candidate list will be
empty. We describe the process of selecting and ranking a file in detail in
the next step.

Actions: We define _Actions_ in our MDP as selecting a file from the candidate
list and moving it to the ranked list. Suppose at the timestep $t$; the RL
model picks the embedding $E_{1}$, then the rank of that particular file will
be $t$. In Figure 2 at the timestamp $1$, the model picks concatenated
embedding of file $F_{2}$. Thus, the rank of $F_{2}$ will be $1$. As in each
timestamp, we are moving one file from the candidate list to the ranked list;
the total number of files will be equal to the number of states and the number
of actions. For identifying the potentially best action at any timestamp $t$,
we use a deep learning (DL) model (indicated as _Ranking Model_ in Figure 2),
which is composed of a Convolutional Neural Network (CNN) followed by a Long
Short-Term Memory (LSTM) [35]. First, we need to identify the optimal
embedding (in terms of relevance) and add it to the ranked list of embeddings.
Following the previous studies [5, 36, 37], we use a CNN to capture two
relations. The first is the relation between source code embedding and bug
report embedding, and the second is the relation across different concatenated
embeddings. In each timestamp, the model should pick the potentially best
embedding. Thus, it must know the relative relevance of different source code
files embedding with the bug report. Also, our ranking model takes into
consideration one of the restrictions set by the environment, which we call
_state awareness_ , i.e., if a file is selected at $State_{i}$, it cannot be
selected again in a later state (i.e., $State_{i+j};j\geq 1$). We use LSTM to
make the model aware of previous actions. LSTM cell [38] helps the model to
select files remembering the files it has already selected.

Transition: $\tau$(S, A) is a function $\tau$ : S × A → S which maps a state
$s_{t}$ into a new state $s_{t+1}$ in response to the selected action $a_{t}$.
Choosing an action $a_{t}$ means removing a file from the candidate list and
placing it in the ranked list.

Reward: A reward is a value provided to the RL agent as feedback on their
action. We refer to a reward received from one action as _return_. The RL
technique signals the agent about the appropriate action in each step through
the Reward Function, which can be modeled using the retrieval metrics. Thus,
the RL agent can learn to optimize the retrieval metrics through the reward
function. We consider two important factors in the ranking evaluation: the
position of the relevant files and the distance between relevant files in the
ranked list of embedding. We incorporated both factors in designing the reward
function shown below.

$\Re(S,A)=\frac{M*file\;relevance}{\log_{2}(t+1)*distance(s)};if\;A\;is\\\
an\;action\;that\;has\;not\;been\;selected\;before\\\ $ (1)
$\Re(S,A)=-\log_{2}(t+1);otherwise$ (2)

$distance(S)=Avg.(Distance\;between\;\;currently\\\
picked\;subsequent\;related\;files)$ (3)

Figure 3: Effect of M in the reward-episode graph.

In Equations LABEL:eq:reward_1 and 2, $t$ is the timestamp, $S$ is State and
$A$ is Action. Mean reciprocal rank (MRR) measures the average reciprocal rank
of all the relevant files. In Equation LABEL:eq:reward_1,
$\frac{file\;relevance}{\log_{2}(t+1)}$ represents the MRR. The use of a
logarithmic function in the equation is motivated by previous studies [39,
40], which found that it leads to a stable loss. When the relevant files are
ranked higher, the average precision tends to be higher. To encourage the
reinforcement learning system to rank relevant files higher, we introduce a
punishment mechanism if there is a greater distance between two relevant
files. By imposing this punishment on the agent, we incentivize it to
prioritize relevant files in higher ranks, which in turn contributes to the
Mean Average Precision (MAP).

We illustrate the reward functions with an example below. Presuming that the
process reaches State $S_{6}$ and the currently picked concatenated embeddings
are $E_{1},E_{2},E_{3},E_{4},E_{5},E_{6}$ and their relevancy to the bug
report is $\langle 0,0,1,0,1,1\rangle$. This means that these embeddings (or
files) ranked in the $3^{rd}$, $5^{th}$, and $6^{th}$ positions are relevant
to the bug report. The position of the relevant files are $\langle
3,5,6\rangle$, and the distance between them is $\langle 1,0\rangle$. Hence,
$distance(S_{6})=Avg.\langle 1,0\rangle=0.5$. If the agent picks a new
relevant file, we reward the agent $M$ times the reciprocal rank of the file
divided by the distance between the already picked related files. In our
example, the last picked file, $E_{6}$’s relevancy is $1$. Thus, we have the
following values for Equation LABEL:eq:reward_1: $distance(S_{6})=0.5$;
$\log_{2}(6+1)=2.8074$; $file\;relevance=1$. Note that $M$ is a hyper-
parameter. We find that three as the value of M results in the highest reward
for our RL model. We identify the best value for $M$ by experimenting with
different values (1, 3, 6, and 9). Figure 3 shows the resulting reward-episode
graph using different values of $M$. Hence, given $M=3$, the value of the
reward function will be $\Re(S,A)=\frac{3*1}{2.8074*0.5}=2.14$. The reward can
vary between $M$ to $\sim$ 0\. A higher value of the reward function indicates
a better action of the model. Finally, in the case of optimal ranking, the
$distance(S)$ will be zero. We handle this case by assigning a value of $1$
for $distance(S)$.

Finally, note that we limit the number of actions to 31. According to our
formulation, the number of states will be equal to the number of actions.
Thus, the number of states is also limited to a maximum of 31. Like any deep
learning system, the prediction space of a reinforcement learning agent can
not be a variable. In our formulation, the number of actions equals the number
of source code files, and the number of source code files is variable. As we
cannot have a variable number of actions, we must fix the number of actions
$k$. We can select any number as a value of $k$ if it fits in the memory.
Previously, we formulated _state_ using embedding all the source code files
concatenated with the embedding of the bug report. A state composed of
concatenated embedding of a large number of source code files can lead to a
larger _state_. In this study, we use an Nvidia V100 16 GB GPU. We conduct
tests with a different number of actions. Our tests revealed that when the
number of actions reached 32, the training scripts began to fail with an out-
of-memory error. This is due to the state size being too large to fit into the
GPU memory. To resolve this issue, we limit the number of actions to $k=31$ to
keep the state size under control. While we could potentially increase the
number of actions, our current resources limit us to 31 actions. In section
3.1, we mention that we select the top $k$ relevant source code files and
passed them to the RLocator. According to our formulation, we have 31 states.
As the number of files and the number of states is the same, we can select the
top 31 ($k=31$) most relevant source code files from ES and pass them to
RLocator.

##  4 Dataset and Evaluation Measures

In this section, we discuss the dataset used to train and evaluate our model
(section 4.1). Then, we present the evaluation metrics we use for evaluation
(section 4.2).

###  4.1 Dataset

In our experiment, we evaluate our approach on six real-world open-source
projects [41], which are commonly used benchmark datasets in bug localization
studies [16, 6]. Also, prior work (e.g., [42]) showed that this dataset
contains the lowest number of false positive or negative cases among other
datasets [43, 44, 3, 45, 46] in the literature. Previous studies evaluate
their approaches to each project separately. Thus, in this study, we follow a
similar approach and train our RLocator model for each one of the six
projects.

The dataset contains bug reports of six Apache projects (AspectJ, Birt,
Eclipse Platform UI, JDT, SWT, Tomcat). Table II shows descriptive statistics
on the datasets. The dataset contains metadata of bug reports, i.e., bug id,
bug description, report timestamp, commit SHA of the fixing commit, and buggy
source code files path. As mentioned in Section 2.1 each bug report is
associated with a commit SHA/version. We use the commit SHA to identify the
version repository version where the bug resides (the commit SHA, just before
the bug fixing commit). Once the version containing the bug is identified from
the commit SHA, we collect all the relevant source code files from that
version. This approach guarantees that the bug localization system closely
resembles real-world bug localization, as the bug-fixing source code is
excluded from the bug localization process. In section 3, we state that we
used only 91% of the data for training and testing. To prepare the train and
test dataset, we sort the dataset according to the ascending order of the bug
reports date and divide the dataset into 60:40 split. Previous studies [47]
have used 60:20:20 settings for train, validation, and testing purpose. As we
do not use validation to shorten the training duration, we use the validation
data for testing purposes. We use the first 60% of the data for training
purposes and the remaining 40% for testing purposes.

TABLE II: Dataset statistics.

Project |  | # of Bug
---
Reports
| Avg. # of Buggy
---
Files per Bug
AspectJ | 593 | 4.0
Birt | 6,182 | 3.8
Eclipse UI | 6,495 | 2.7
JDT | 6,274 | 2.6
SWT | 4,151 | 2.1
Tomcat | 1,056 | 2.4

###  4.2  Evaluation Measures

The dataset proposed by Ye et al. [41] provides ground truth associated with
each bug report. The ground truth contains the path of the file in the project
repository that has been modified to fix a particular bug. To evaluate
RLocator performance, we use the ground truth and analyze the experimental
results based on three criteria, which are widely adopted in bug localization
studies [3, 5, 6, 7, 8].

  * •

Mean Reciprocal Rank (MRR): To identify the average rank of the relevant file
in the retrieved files set, we adopted the Mean Reciprocal Rank. MRR is the
average reciprocal rank of the source code files for all the bug reports. We
present the equation for calculating MRR below, where $A$ is the set of bug
reports.

$MRR=\frac{1}{|A|}\sum_{A}\frac{1}{Least\;rank\;of\;the\;relevant\;files}$ (4)

Suppose we have two bug reports, $report_{1}$ and $report_{2}$. For each bug
report, the bug localization model will rank six files. For $report_{1}$ the
ground truth of the retrieved files are $[0,0,1,0,1,0]$ and for $report_{2}$
the ground truth of the retrieved files are $[1,0,0,0,0,1]$. In this case, the
least rank of relevant files is 3 and 1, respectively, for $report_{1}$ and
$report_{2}$. Now, the $MRR=\frac{1}{2}(\frac{1}{3}+\frac{1}{1})=0.67$

  * •

Mean Average Precision (MAP): To consider the case where a bug is associated
with multiple source code files, we adopted Mean Average Precision. It
provides a measure of the quality of the retrieval [3, 48]. MRR considers only
the best rank of relevant files; on the contrary, MAP considers the rank of
all the relevant files in the retrieved files list. Thus, MAP is more
descriptive and unbiased than MRR. Precision means how noisy the retrieval is.
If we calculate the precision on the first two retrieved files, we will get
precision@2. For calculating average precision, we have to figure precision@1,
precision@2,… precision@k, and then we have to average the precision at
different points. After calculating the average precision for each bug report,
we have to find the mean of the average precision to calculate the MAP.

$MAP=\frac{1}{|A|}\sum_{A}{AvgPrecision(Report_{i})}$ (5)

We show the MAP calculation for the previous example of two bug reports. The
Average precision for $report_{1}$ and $report_{2}$ will be 0.37 and 0.67. So,
the $MAP=\frac{1}{2}(0.36+0.67)=0.52$

  * •

Top K: For fare comparison with prior studies [37, 36] and to present a
straightforward understanding of performance we calculate Top K. Top K
measures the overall ranking performance of the bug localization model. It
indicates the percentage of bug reports for which at least one buggy source
appears among the top K positions in the ranked list generated by the bug
localization tool. Following previous studies (e.g., [37, 36]), we consider
three values of K: 1, 5, and 10.

##  5 RLocator Performance

We evaluate RLocator on the hold-out dataset using the metrics described in
Section 4.2. As there has been no RL-based bug localization tool, we compare
RLocator with two state-of-the-art bug localization tools, BugLocator, and
FLIM. We choose these approaches as the replication package of the studies is
available and accessible, which can facilitate our comparison to these
approaches.

BugLocator uses IR based approach, whereas FLIM is a deep neural network-based
method. A short description of both approaches has been presented below.

TABLE III: RLocator performance. Project | Model | Top 1 | Top 5 | Top 10 | MAP | MRR
---|---|---|---|---|---|---
91% | 100% | 91% | 100% | 91% | 100% | 91% | 100% | 91% | 100%
AspectJ | RLocator | 0.46 | 0.40 | 0.69 | 0.63 | 0.75 | 0.70 | 0.56 | 0.46 | 0.59 | 0.50
BugLocator | 0.36 | 0.28 | 0.50 | 0.45 | 0.56 | 0.51 | 0.33 | 0.31 | 0.49 | 0.48
FLIM | 0.51 | 0.36 | 0.65 | 0.60 | 0.72 | 0.67 | 0.41 | 0.35 | 0.47 | 0.45
Birt | RLocator | 0.38 | 0.25 | 0.46 | 0.41 | 0.53 | 0.48 | 0.47 | 0.38 | 0.49 | 0.41
BugLocator | 0.28 | 0.15 | 0.27 | 0.21 | 0.34 | 0.29 | 0.30 | 0.30 | 0.39 | 0.38
FLIM | 0.29 | 0.18 | 0.39 | 0.34 | 0.47 | 0.42 | 0.29 | 0.25 | 0.31 | 0.28
Eclipse Platform UI | RLocator | 0.45 | 0.37 | 0.69 | 0.63 | 0.78 | 0.73 | 0.54 | 0.42 | 0.59 | 0.50
BugLocator | 0.45 | 0.33 | 0.54 | 0.49 | 0.63 | 0.58 | 0.29 | 0.30 | 0.38 | 0.35
FLIM | 0.48 | 0.41 | 0.72 | 0.67 | 0.80 | 0.75 | 0.51 | 0.48 | 0.52 | 0.53
JDT | RLocator | 0.44 | 0.33 | 0.67 | 0.61 | 0.78 | 0.75 | 0.51 | 0.44 | 0.53 | 0.45
BugLocator | 0.34 | 0.21 | 0.51 | 0.45 | 0.60 | 0.55 | 0.22 | 0.20 | 0.31 | 0.28
FLIM | 0.40 | 0.35 | 0.65 | 0.60 | 0.82 | 0.77 | 0.42 | 0.41 | 0.51 | 0.49
SWT | RLocator | 0.40 | 0.30 | 0.57 | 0.51 | 0.63 | 0.58 | 0.48 | 0.42 | 0.51 | 0.44
BugLocator | 0.37 | 0.25 | 0.50 | 0.45 | 0.56 | 0.51 | 0.42 | 0.40 | 0.46 | 0.43
FLIM | 0.51 | 0.37 | 0.70 | 0.65 | 0.83 | 0.78 | 0.43 | 0.43 | 0.48 | 0.50
Tomcat | RLocator | 0.46 | 0.39 | 0.61 | 0.55 | 0.73 | 0.68 | 0.59 | 0.47 | 0.62 | 0.51
BugLocator | 0.40 | 0.29 | 0.43 | 0.38 | 0.55 | 0.50 | 0.31 | 0.27 | 0.37 | 0.35
FLIM | 0.51 | 0.42 | 0.70 | 0.65 | 0.76 | 0.71 | 0.52 | 0.47 | 0.59 | 0.60

  * •

BugLocator [3]: an IR-based tool that utilizes a vector space model to
identify the potentially responsible source code files by estimating the
similarity between source code file and bug report.

  * •

FLIM [16]: a deep-learning-based model that utilizes a large language model
like CodeBERT.

We use the original implementation for determining performance for BugLocator.
[3] and FLIM [16].

FBL-BERT [8] is one of the more recent bug localization techniques. However,
we do not compare against them since FBL-BERT performs bug localization at the
changeset level. If we were to take their technique (which is available in the
arxiv version of their paper) and apply it to our dataset that is at the file
level, then we would be disadvantaging the performance of FBL-BERT. This is
because FBL-BERT is not meant to work with long documents like source code
files. Rather they work best at the changeset level. Hence we feel that
replication and comparison of FBL-BERT with our technique would unfairly say
that our technique would be better. Since the granularity is different, we do
not compare RLocator against FBL-BERT.

There are a few other studies (e.g., DeepLoc [47], CAST [5], KGBugLocator [6],
BL-GAN [7], bjXnet [49], and Cheng et al. [50]) that proposed deep learning-
based bug localization approaches. However, these studies do not provide a
replication package containing code or pre-trained models. Still, those
approaches use the same projects to evaluate the performance of the proposed
tools. Since we are unable to find any replication packages and/or models
directly from the respective papers, we do not provide any further comparison
or discussion in our paper. However, to provide comprehensive information, we
include a table in our online appendix [51], displaying their performance
alongside that of RLocator.

###  5.1 Retrieval performance

As previously mentioned, we utilize $k$ (=31) relevant files in RLocator. Note
that with $k$=31 and using ES as we describe in Section 3.1, we can rerank
files using RLocator for 91% of the bug reports in our dataset. Table III
presents the performance of RLocator along with the performance of the
examined studies considering 91% of the data and 100% of the data. Note that
RLocator is not designed to utilize 100% data since it cannot rerank the files
for any bug report for which there are no relevant files among the top $k$
files. Thus, for the cases where RLocator is not applicable (no relevant files
identified by the XGBoost model), we estimate the performance of RLocator. For
estimation, we assume the contribution of the case is zero. For example, if we
have three bug reports where for the $1^{st}$ bug report, the rank of the
relevant files is three and five, $2^{nd}$ bug report, the rank of the files
is five, and seven and the $3^{rd}$ bug report the XGBoost model predicts that
there will be no relevant file in top $K$ files. The MRR for this scenario
will be $\frac{1}{3}(\frac{1}{2}+\frac{1}{5}+0)=0.23$. As we are using zero
instead of reciprocal ranks when calculating Mean Reciprocal Rank (MRR) (or
MAP), we can say that estimated performance is the worst possible performance
or performance lower bound of RLocator. We choose such an overly conservative
approach to avoid overestimating the effectiveness of our technique.

Table III showcases that RLocator achieves better performance than BugLocator
and FLIM in both MRR and MAP across all studied projects when using the 91%
data. On 91% data, RLocator outperforms FLIM by 5.56-38.3% in MAP and
3.77-36.73% in MRR. Regarding Top K, the performance improvement is up to
23.68%, 15.22%, and 11.32% in terms of Top 1, Top 5, and Top 10, respectively.
Comparing to BugLocator, RLocator achieves performance improvement of 12.5 -
56.86% and 9.8% - 41.51%, in terms of MAP and MRR, respectively. Regarding Top
K, the performance improvement is up to 26.32%, 41.3%, and 35.85% in terms of
Top 1, Top 5, and Top 10, respectively.

When we consider 100% of the data, RLocator has better MAP results than FLIM
in three out of the six projects (AspectJ, Birt, and JDT) by 6.82-34.21%,
equal to FLIM in one project (Tomcat) and worse than FLIM in 2 projects
(Eclipse Platform UI and SWT) by 2-14%. In terms of MRR, RLocator is better
than FLIM in 2 projects (AspectJ and Birt) by 10-31.71% and worse than FLIM in
the remaining four projects (Eclipse platform UI, JDT, SWT, and Tomcat) by
6-18%. In terms of Top K, RLocator ranks 4.29-12.5% more bugs in the top 10
positions than FLIM in two projects. On the other hand, in the rest of the six
projects, FLIM ranks more bugs in the top 10 positions ranging between 2.74 -
34.48%. When comparing RLocator with BugLocator for the 100% data along MAP,
we find that the RLocator is better in five of the six projects and similar in
just the Tomcat project. With respect to MRR, RLocator is better than
BugLocator in all six projects. In terms of Top K, RLocator ranks more bugs
than BugLocator in the top 10 position, where the improvement ranges between
12.07 -39.58%.

It is important to note that MAP provides a more balanced view than MRR and
top K since it accounts for all the files that are related to a bug report and
not just one file. Additionally, in our technique, we optimize to give more
accurate results for most of the bug reports than give less accurate results
on average for all the bug reports. Thus, by looking at the MAP data for the
91%, we can see that RLocator performs better than the state-of-the-art
techniques in all projects. Even if we consider 100% of the data, RLocator is
still better than other techniques in the majority of the projects. Only with
100% of the data and when using MRR as the evaluation metric, RLocator does
not perform better than the state-of-the-art in most projects.

Across all projects, we observe that RLocator performs the lowest in the Birt
project. Interestingly, besides RLocator, FLIM also achieved the lowest
performance (in both MRR and MAP). Compared to the average performance on 91%
data, RLocator’s performance has dropped by nearly 10.47%, 11.71%, and 41.42%
in terms of MAP, MRR, and top 10, respectively. Nevertheless, compared with
FLIM, the performance is 38.3%, 36.73%, and 11.32% better for MAP, MRR, and
the top 10, respectively. With respect to the BugLocator technique, the
performance is 36.17%, 20.41%, and 35.85% better in terms of MAP, MRR, and top
10, respectively. In fact, several factors can contribute to such performance
drop [52] in the Birt project, e.g., the quality of bug reports, source code
length, amount of information in the bug report, etc. As _similarity_ is one
of the important criteria for text retrieval-based bug localization systems
[53], we estimate the amount of helpful information by calculating
_similarity_ according to equation 6. The similarity metric measures the ratio
of similar tokens between source code files and bug reports. High similarity
indicates that the bug report contains much potentially helpful information to
localize the bug.

$Similarity=\frac{Bug\ Report\ Tokens\cap File\ Tokens}{\\#\ of\ Unique\
Tokens\ in\ Bug\ Report}$ (6)

We find that the median similarity for the Birt project is 0.29, which is the
lowest among the six projects, which indicates that the bug reports’ quality
(i.e., similarity to files) can impact the performance drop of RLocator in the
Birt project.

Figure 4: Feature importance of classifier model.

For using RLocator effectively in real-world scenarios, we use an XGBoost
model (Section 3.1). The XGBoost model filters out the bug reports where the
relevant files do not appear in the top $k$ (=31) files. To provide richer
insights into our analysis and understand the characteristics of the bug
reports and the associated 31 retrieved files, we compute the importance of
our selected features mentioned in Table I. XGBoost offers a built-in module
to calculate feature importance. The importance score indicates how much each
feature contributes to the model. We report the default type of importance,
“weight”. A higher value of this metric implies that the feature is more
important for generating a prediction. Figure 4 illustrates the significance
of our selected features. It demonstrates that the most crucial feature is
Similarity, closely followed by Source Code Length, Bug Report Length. These
findings suggest that the similarity of a bug report plays a critical role in
text-based search systems. Furthermore, it is well-known that low-quality bug
reports can impede developers [54]. Therefore, these insights can assist users
in composing high-quality bug reports, ultimately leading to reduced
resolution time.

###  5.2 Entropy Ablation Analysis: Impact of Entropy on RLocator performance

In RL, entropy refers to the unpredictability of an agent’s actions. A low
entropy indicates a predictable policy, while high entropy represents a more
random and robust policy. An agent in RL will tend to repeat actions that
previously resulted in positive rewards while learning the policy. The agent
may become stuck in a local optimum due to exploiting learned actions instead
of exploring new ones and finding a higher global optimum. This is where
entropy comes useful: we can use entropy to encourage exploration and avoid
getting stuck in local optima [55]. Because of this, entropy in RL has become
very popular in the design of RL approaches such as A2C [56]. In our proposed
model (Section 5.1), we use A2C with entropy to train the RLocator aiming to
rank relevant files closer to each other. As entropy is part of the reward,
the gradient descent process will try to maximize the entropy. Entropy will
increase if the model identifies different actions as the best in the same
state. However, those actions must select a relevant file; otherwise, the
reward will be decreased. Thus, if there are multiple relevant files in a
state, the A2C with entropy regularized model will assign almost the same
probability in those actions (actions related to selecting those relevant
files). This means that when the states are repeated, a different action will
likely be selected each time. This probability assignment will lead to a
higher MAP.

TABLE IV: RLocator performance with and without Entropy for A2C. Project |  Model | Top 1 | Top 5 | Top 10 | MAP | MRR
---|---|---|---|---|---|---
AspectJ |  A2C | 0.27 | 0.39 | 0.48 | 0.40 | 0.52
A2C with Entropy | 0.46 | 0.69 | 0.75 | 0.56 | 0.59
Birt |  A2C | 0.21 | 0.30 | 0.43 | 0.31 | 0.42
A2C with Entropy | 0.38 | 0.46 | 0.53 | 0.47 | 0.49
Eclipse Platform UI |  A2C | 0.25 | 0.38 | 0.51 | 0.39 | 0.51
A2C with Entropy | 0.45 | 0.69 | 0.78 | 0.54 | 0.59

The observed performance of RLocator in achieving higher MAP can be
interpreted due to two factors: 1) the way how we design our reward function,
given that we define a function that aims to encourage higher MAP; 2) the
inclusion of entropy, as entropy regularization is assumed to enable the model
to achieve higher MAP. Hence, to provide a better understanding of the impact
of entropy on the RLocator, we conduct a further experiment where we train a
new model that does not consider entropy. We compare the model performance to
the performance obtained in our previous models (i.e., A2C with entropy). Due
to resource (time and GPU) limitations, we limit our evaluation to half of the
total projects in our dataset, i.e., AspectJ, Birt, and Eclipse Platform UI.
We observe a similar trend in those three projects. Thus, we believe our
results will follow a similar trend in the remaining projects. Table IV
presents the performance of both models (i.e., A2C only vs. A2C with Entropy).
From the table, we find that the MRR and MAP of the models without entropy are
lower than that of the A2C with entropy models. Table IV shows that in terms
of MAP, the performance of A2C with entropy models is higher than A2C models
by a range of 27.78-34.04%. In MRR and the top 10, the A2C with entropy model
achieves higher performance by a range of 11.86-13.56% and 18.87 - 36%,
respectively. Such results indicate that entropy could substantially
contribute to the model performance regarding MAP, MRR, and Top K. Moreover,
this shows that the use of entropy encourages the RL agent to explore possible
alternate policies [57]; thus, it has a higher chance of getting a better
policy for solving the problem in a given environment than the A2C model.

##  6 Related Work

The work most related to our study falls into studies on bug localization
techniques. In the following, we discuss the related work and reflect on how
the work compares with ours.

A plethora of work studied how developers localize bugs [58, 59, 60, 54]. For
example, Böhme et al. [58] studied how developers debug. They found that the
most popular technique for localizing a bug is forward reasoning, where
developers go through each computational step of a failing test case to
identify the location. Zimmermann et al. [54] studied the characteristics of a
good bug report and found that test cases and stack traces are one of the most
important criteria that makes a good bug report. While these studies focused
on developers’ manual localization of bugs, our approach examines the
automation of the bug localization process for developers.

Several studies offered test case coverage-based solutions for bug
localization [61, 62, 63]. Vancsics et al. [61] proposed a count-based
spectrum instead of a hit-based spectrum in the Spectrum-Based Fault
Localization (SBFL) tool. GRACE [62] proposed gated graph neural network-based
representation learning to improve the SBFL technique. However, these studies
mainly utilize test cases to localize bugs, whereas our approach (RLocator)
focuses on the bug report for bug localization.

There were several efforts to assess the impact of query reformulation in
improving the performance of existing bug localization tools. [64, 65]. For
example, Rahman et al. [64] found that instead of using the full bug report as
a full-text query, a reformulated query with some additional expansion
performs better. BugLocator [3] used a revised Vector Space Model (rVSM) to
estimate the textual similarity between bug reports and source code files.

A few studies incorporated the information of program structures such as the
Program Dependence Graph (PDG), Data Flow Graph (DFG) [66], and Abstract
Syntax Tree (AST) for learning source code representation [4, 47, 66, 5]. For
example, CAST [5] used AST of the source code to extract the semantic
information and then used Word2Vec to project the source code and the bug
report in the same embedding space. They used a CNN model that measures the
similarity between a bug report and source code. The model ranks the file
based on the calculated similarity. Hyloc [9] incorporated the techniques of
IR-based bug localization with deep learning. It concatenated the TF-IDF
vector of the source code with repository and file-level metadata.

Other studies applied several deep learning-based approaches for bug
localization [67, 6, 49]. DEMOB [67] used attention on ELMo [68] embedding,
whereas KGBugLocator [6] used attention on graph embedding. BL-GAN [7] offered
a generative adversarial network-based solution for bug localization. BjXnet
[49] utilized text semantic graph and code property graph together to localize
bugs. One major drawback of bjXnet is using the hierarchical property to
estimate similarity. Source code has hierarchical property, but bug reports do
not have any hierarchical property. Thus, using a deep graph model to estimate
similarity may not be useful. Other studies focused on associating commits
with bug reports [69, 8]. For example, FBL-BERT [8] used CodeBERT embedding
for estimating the similarity between source code files and changesets of a
commit. Based on the similarity, it ranks the suspicious commit. FLIM [16]
also used CodeBERT embedding for estimating similarity. However, FLIM works on
function-level bug localization.

Our work differs from previous studies on bug localization since we propose
our approach (RLocator) based on deep reinforcement learning. The key insight
behind RLocator is directly optimizing the evaluation measures, while the
prior work focused on similarity-based approaches. We formulate the bug
localization problem using a Markov Decision Process (MDP) to optimize the
evaluation measures directly. We experimentally evaluate it based on a
benchmark dataset of 8,316 projects from six highly popular Apache projects.
Our results demonstrate that direct optimization of evaluation measures
considerably contributes to performance improvement of the bug localization
problem.

##  7 Threats to Validity

RLocator has a number of limitations as well. We identify them and discuss how
to overcome the limitations below.

Internal Validity. One limitation of our approach is we are not able to
utilize 9% of our dataset due to the limitation of text-based search. One may
point out that we exclude the bug reports where we do not perform well. But
our the XGBoost model in our approach automatically identifies them and we say
that we would rather not localize the source code files for these bug reports
than localize them incorrectly. Hence, developers need to rely on their manual
analysis only for the 9%. Moreover, as a measure of full transparency, we
estimate the lower bound of RLocator performance for the 100% data and show
that the difference is negligible.

External Validity. The main threat to external validity lies in the selection
of projects for evaluation. We evaluate RLocator on a limited number of bugs,
which may concern our results’ generalizability. However, we evaluate RLocator
on six real-world open-source projects that are used by prior studies [4, 5,
67, 6, 9, 70]. Those projects are from different domains and use varied
development styles. Additionally, to identify the reason for the better
performance of the A2C with entropy model, we have trained the A2C (without
entropy) model for only three of the six projects. However, training models
for each of the six projects are resource (time and GPU) consuming. For
example, for end-to-end training and evaluation of an A2C (without entropy)
model, it will take 4 days in an Nvidia V100 16 GB GPU. The three projects’
results (AspectJ, Birt, Eclipse Platform UI) follow the same trend. Thus we
can assume that the remaining projects’ performance would follow the same
trend.

Construct Validity. Finally, our evaluation measures might be one threat to
construct validity. The evaluation measures may not completely reflect real-
world situations. The threat is mitigated by the fact that the used evaluation
measures are well-known [8, 16, 41, 3] and best available to measure and
compare the performance of information retrieval-based bug localization tools.

##  8 Conclusion

In this paper, we propose RLocator, a reinforcement learning-based (RL)
technique to rank the source code files where the bug may reside, given the
bug report. The key contribution of our study is the formulation of the bug
localization problem using the Markov Decision Process (MDP), which helps us
to optimize the evaluation measures directly. We evaluate RLocator on 8,316
bug reports and find that RLocator performs better than the state-of-the-art
techniques when using MAP as an evaluation measure. Using 91% bug reports
dataset, RLocator outperforms prior tools in all the project in terms of both
MAP and MRR. When using 100% data, RLocator outperforms all prior approaches
in four of six projects using MAP and two of the six projects using MRR.
RLocator can be used along with other bug localization approaches to improve
performance. Our results show that RL is a promising avenue for future
exploration when it comes to advancing state-of-the-art techniques for bug
localization. Future research can explore the application of the latest
Reinforcement learning algorithms in the domain of bug localization

##  9 Data Availability

To foster future research in the field, we make a replication package
comprising our dataset and code are publicly available [51].

## References

  * [1] T. D. LaToza and B. A. Myers, “Developers ask reachability questions,” in _Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - ICSE '10_. ACM Press, 2010.
  * [2] J. Anvik, L. Hiew, and G. C. Murphy, “Coping with an open bug repository,” in _Proceedings of the 2005 OOPSLA workshop on Eclipse technology eXchange - eclipse '05_. ACM Press, 2005.
  * [3] J. Zhou, H. Zhang, and D. Lo, “Where should the bugs be fixed? more accurate information retrieval-based bug localization based on bug reports,” in _2012 34th International Conference on Software Engineering (ICSE)_. IEEE, Jun. 2012.
  * [4] Y. Xiao, J. Keung, K. E. Bennin, and Q. Mi, “Machine translation-based bug localization technique for bridging lexical gap,” _Information and Software Technology_ , vol. 99, pp. 58–61, Jul. 2018.
  * [5] H. Liang, L. Sun, M. Wang, and Y. Yang, “Deep learning with customized abstract syntax tree for bug localization,” _IEEE Access_ , vol. 7, pp. 116 309–116 320, 2019.
  * [6] J. Zhang, R. Xie, W. Ye, Y. Zhang, and S. Zhang, “Exploiting code knowledge graph for bug localization via bi-directional attention,” in _Proceedings of the 28th International Conference on Program Comprehension_. ACM, Jul. 2020.
  * [7] Z. Zhu, H. Tong, Y. Wang, and Y. Li, “BL-GAN: Semi-supervised bug localization via generative adversarial network,” _IEEE Transactions on Knowledge and Data Engineering_ , pp. 1–14, 2022.
  * [8] A. Ciborowska and K. Damevski, “Fast changeset-based bug localization with BERT,” in _Proceedings of the 44th International Conference on Software Engineering_. ACM, May 2022\.
  * [9] A. N. Lam, A. T. Nguyen, H. A. Nguyen, and T. N. Nguyen, “Combining deep learning with information retrieval to localize buggy files for bug reports (n),” in _2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, Nov. 2015.
  * [10] Z. Wei, J. Xu, Y. Lan, J. Guo, and X. Cheng, “Reinforcement learning to rank with markov decision process,” in _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_. ACM, Aug. 2017\.
  * [11] O. Alejo, J. M. Fernandez-Luna, J. F. Huete, and R. Perez-Vazquez, “Direct optimization of evaluation measures in learning to rank using particle swarm,” in _2010 Workshops on Database and Expert Systems Applications_. IEEE, Aug. 2010.
  * [12] J. Xu, L. Xia, Y. Lan, J. Guo, and X. Cheng, “Directly optimize diversity evaluation measures,” _ACM Transactions on Intelligent Systems and Technology_ , vol. 8, no. 3, pp. 1–26, Jan. 2017.
  * [13] Y. Yue, T. Finley, F. Radlinski, and T. Joachims, “A support vector method for optimizing average precision,” in _Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval_. ACM, Jul. 2007\.
  * [14] R. S. Sutton and A. G. Barto, _Reinforcement Learning: An Introduction_. Cambridge, MA, USA: A Bradford Book, 2018.
  * [15] F. Garcia and E. Rachelson, “Markov decision processes,” in _Markov Decision Processes in Artificial Intelligence_. John Wiley & Sons, Inc., Mar. 2013, pp. 1–38.
  * [16] H. Liang, D. Hang, and X. Li, “Modeling function-level interactions for file-level bug localization,” _Empirical Software Engineering_ , vol. 27, no. 7, Oct. 2022.
  * [17] S. Wang, T. Liu, and L. Tan, “Automatically learning semantic features for defect prediction,” in _Proceedings of the 38th International Conference on Software Engineering_. ACM, May 2016.
  * [18] N. Miryeganeh, S. Hashtroudi, and H. Hemmati, “GloBug: Using global data in fault localization,” _Journal of Systems and Software_ , vol. 177, p. 110961, Jul. 2021.
  * [19] Y. Kim, M. Kim, and E. Lee, “Feature combination to alleviate hubness problem of source code representation for bug localization,” in _2020 27th Asia-Pacific Software Engineering Conference (APSEC)_. IEEE, Dec. 2020.
  * [20] L. Chen, Z. Tang, and G. H. Yang, “Balancing reinforcement learning training experiences in interactive information retrieval,” in _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. ACM, Jul. 2020\.
  * [21] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186. [Online]. Available: https://aclanthology.org/N19-1423
  * [22] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “CodeBERT: A pre-trained model for programming and natural languages,” in _Findings of the Association for Computational Linguistics: EMNLP 2020_. Online: Association for Computational Linguistics, Nov. 2020, pp. 1536–1547. [Online]. Available: https://aclanthology.org/2020.findings-emnlp.139
  * [23] J. Wang and Y. Dong, “Measurement of text similarity: A survey,” _Information_ , vol. 11, no. 9, p. 421, Aug. 2020.
  * [24] S. Fujimoto, D. Meger, and D. Precup, “Off-policy deep reinforcement learning without exploration,” in _Proceedings of the 36th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15 Jun 2019, pp. 2052–2062. [Online]. Available: https://proceedings.mlr.press/v97/fujimoto19a.html
  * [25] T. T. Nguyen and V. J. Reddi, “Deep reinforcement learning for cyber security,” _IEEE Transactions on Neural Networks and Learning Systems_ , pp. 1–17, 2021.
  * [26] C. Liu, X. Xia, D. Lo, Z. Liu, A. E. Hassan, and S. Li, “CodeMatcher: Searching code based on sequential semantics of important query words,” _ACM Transactions on Software Engineering and Methodology_ , vol. 31, no. 1, pp. 1–37, Jan. 2022.
  * [27] T. Chen and C. Guestrin, “XGBoost,” in _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_. ACM, Aug. 2016.
  * [28] F. Fang, J. Wu, Y. Li, X. Ye, W. Aljedaani, and M. W. Mkaouer, “On the classification of bug reports to improve bug localization,” _Soft Computing_ , vol. 25, no. 11, pp. 7307–7323, Mar. 2021.
  * [29] Y. Lv and C. Zhai, “When documents are very long, BM25 fails!” in _Proceedings of the 34th international ACM SIGIR conference on Research and development in Information - SIGIR '11_. ACM Press, 2011\.
  * [30] D. D. Lewis, “Naive (bayes) at forty: The independence assumption in information retrieval,” in _Machine Learning: ECML-98_. Springer Berlin Heidelberg, 1998, pp. 4–15.
  * [31] A. Schroter, A. Schröter, N. Bettenburg, and R. Premraj, “Do stack traces help developers fix bugs?” in _2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010)_. IEEE, May 2010.
  * [32] Y. Lv and C. Zhai, “Lower-bounding term frequency normalization,” in _Proceedings of the 20th ACM international conference on Information and knowledge management - CIKM '11_. ACM Press, 2011.
  * [33] M. Bagherzadeh, N. Kahani, and L. Briand, “Reinforcement learning for test case prioritization,” _IEEE Transactions on Software Engineering_ , vol. 48, no. 8, pp. 2836–2856, Aug. 2022.
  * [34] Y. Wan, Z. Zhao, M. Yang, G. Xu, H. Ying, J. Wu, and P. S. Yu, “Improving automatic source code summarization via deep reinforcement learning,” in _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_. ACM, Sep. 2018.
  * [35] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural Computation_ , vol. 9, no. 8, pp. 1735–1780, Nov. 1997.
  * [36] X. Huo, F. Thung, M. Li, D. Lo, and S.-T. Shi, “Deep transfer bug localization,” _IEEE Transactions on Software Engineering_ , vol. 47, no. 7, pp. 1368–1380, Jul. 2021.
  * [37] X. Huo, M. Li, and Z.-H. Zhou, “Learning unified features from natural and programming languages for locating buggy source code,” in _IJCAI_ , 2016\.
  * [38] M. J. Hausknecht and P. Stone, “Deep recurrent q-learning for partially observable mdps,” _CoRR_ , vol. abs/1507.06527, 2015. [Online]. Available: http://arxiv.org/abs/1507.06527
  * [39] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in _Proceedings of the 35th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, J. Dy and A. Krause, Eds., vol. 80. PMLR, 10–15 Jul 2018, pp. 1861–1870.
  * [40] C. Wang, C. Xu, X. Yao, and D. Tao, “Evolutionary generative adversarial networks,” _IEEE Transactions on Evolutionary Computation_ , vol. 23, no. 6, pp. 921–934, Dec. 2019.
  * [41] X. Ye, R. Bunescu, and C. Liu, “Learning to rank relevant files for bug reports using domain knowledge,” in _Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering - FSE 2014_. ACM Press, 2014.
  * [42] M. Kim and E. Lee, “Are datasets for information retrieval-based bug localization techniques trustworthy?” _Empirical Software Engineering_ , vol. 26, no. 3, Mar. 2021.
  * [43] J. Lee, D. Kim, T. F. Bissyandé, W. Jung, and Y. L. Traon, “Bench4bl: reproducibility study on the performance of IR-based bug localization,” in _Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis_. ACM, Jul. 2018.
  * [44] B. Dit, M. Revelle, M. Gethers, and D. Poshyvanyk, “Feature location in source code: a taxonomy and survey,” _Journal of Software: Evolution and Process_ , vol. 25, no. 1, pp. 53–95, Nov. 2011.
  * [45] L. Moreno, G. Bavota, S. Haiduc, M. D. Penta, R. Oliveto, B. Russo, and A. Marcus, “Query-based configuration of text retrieval solutions for software engineering tasks,” in _Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering_. ACM, Aug. 2015.
  * [46] B. Sisman and A. C. Kak, “Assisting code search with automatic query reformulation for bug localization,” in _2013 10th Working Conference on Mining Software Repositories (MSR)_. IEEE, May 2013.
  * [47] Y. Xiao, J. Keung, K. E. Bennin, and Q. Mi, “Improving bug localization with word embedding and enhanced convolutional neural networks,” _Information and Software Technology_ , vol. 105, pp. 17–29, Jan. 2019.
  * [48] M. N. Schwarz and A. Flammer, “Text structure and title—effects on comprehension and recall,” _Journal of Verbal Learning and Verbal Behavior_ , vol. 20, no. 1, pp. 61–66, Feb. 1981.
  * [49] J. Han, C. Huang, S. Sun, Z. Liu, and J. Liu, “bjXnet: an improved bug localization model based on code property graph and attention mechanism,” _Automated Software Engineering_ , vol. 30, no. 1, Mar. 2023.
  * [50] S. Cheng, X. Yan, and A. A. Khan, “A similarity integration method based information retrieval and word embedding in bug localization,” in _2020 IEEE 20th International Conference on Software Quality, Reliability and Security (QRS)_. IEEE, Dec. 2020.
  * [51] Anonymous, “Rlocator: Reinforcement learning for bug localization,” 2023. [Online]. Available: https://zenodo.org/record/7591879
  * [52] C. Mills, G. Bavota, S. Haiduc, R. Oliveto, A. Marcus, and A. D. Lucia, “Predicting query quality for applications of text retrieval to software engineering tasks,” _ACM Transactions on Software Engineering and Methodology_ , vol. 26, no. 1, pp. 1–45, Jan. 2017.
  * [53] L. Moreno, W. Bandara, S. Haiduc, and A. Marcus, “On the relationship between the vocabulary of bug reports and source code,” in _2013 IEEE International Conference on Software Maintenance_. IEEE, Sep. 2013.
  * [54] T. Zimmermann, R. Premraj, N. Bettenburg, S. Just, A. Schroter, and C. Weiss, “What makes a good bug report?” _IEEE Transactions on Software Engineering_ , vol. 36, no. 5, pp. 618–643, Sep. 2010.
  * [55] Z. Ahmed, N. Le Roux, M. Norouzi, and D. Schuurmans, “Understanding the impact of entropy on policy optimization,” in _Proceedings of the 36th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15 Jun 2019, pp. 151–160. [Online]. Available: https://proceedings.mlr.press/v97/ahmed19a.html
  * [56] S. Jang and H.-I. Kim, “Entropy-aware model initialization for effective exploration in deep reinforcement learning,” _Sensors_ , vol. 22, no. 15, p. 5845, Aug. 2022.
  * [57] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” in _Proceedings of The 33rd International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, M. F. Balcan and K. Q. Weinberger, Eds., vol. 48. New York, New York, USA: PMLR, 20–22 Jun 2016, pp. 1928–1937. [Online]. Available: https://proceedings.mlr.press/v48/mniha16.html
  * [58] M. Böhme, E. O. Soremekun, S. Chattopadhyay, E. Ugherughe, and A. Zeller, “Where is the bug and how is it fixed? an experiment with practitioners,” in _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering_. ACM, Aug. 2017\.
  * [59] T. D. Sasso, A. Mocci, and M. Lanza, “What makes a satisficing bug report?” in _2016 IEEE International Conference on Software Quality, Reliability and Security (QRS)_. IEEE, Aug. 2016.
  * [60] N. Bettenburg, S. Just, A. Schröter, C. Weiss, R. Premraj, and T. Zimmermann, “What makes a good bug report?” in _Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering_. ACM, Nov. 2008.
  * [61] B. Vancsics, F. Horváth, A. Szatmári, and Á. Beszédes, “Fault localization using function call frequencies,” _Journal of Systems and Software_ , vol. 193, p. 111429, Nov. 2022.
  * [62] Y. Lou, Q. Zhu, J. Dong, X. Li, Z. Sun, D. Hao, L. Zhang, and L. Zhang, “Boosting coverage-based fault localization via graph-based representation learning,” in _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. ACM, Aug. 2021.
  * [63] Y. Kim, S. Mun, S. Yoo, and M. Kim, “Precise learn-to-rank fault localization using dynamic and static features of target programs,” _ACM Transactions on Software Engineering and Methodology_ , vol. 28, no. 4, pp. 1–34, Oct. 2019.
  * [64] M. M. Rahman, F. Khomh, S. Yeasmin, and C. K. Roy, “The forgotten role of search queries in IR-based bug localization: an empirical study,” _Empirical Software Engineering_ , vol. 26, no. 6, Aug. 2021.
  * [65] J. M. Florez, O. Chaparro, C. Treude, and A. Marcus, “Combining query reduction and expansion for text-retrieval-based bug localization,” in _2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)_. IEEE, Mar. 2021.
  * [66] Y. Li, S. Wang, T. N. Nguyen, and S. V. Nguyen, “Improving bug detection via context-based code representation learning and attention-based neural networks,” _Proceedings of the ACM on Programming Languages_ , vol. 3, no. OOPSLA, pp. 1–30, Oct. 2019.
  * [67] Z. Zhu, Y. Li, Y. Wang, Y. Wang, and H. Tong, “A deep multimodal model for bug localization,” _Data Mining and Knowledge Discovery_ , Apr. 2021.
  * [68] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” _CoRR_ , vol. abs/1802.05365, 2018. [Online]. Available: http://arxiv.org/abs/1802.05365
  * [69] C. Ni, W. Wang, K. Yang, X. Xia, K. Liu, and D. Lo, “The best of both worlds: integrating semantic features with expert features for defect prediction and localization,” in _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. ACM, Nov. 2022.
  * [70] B. Wang, L. Xu, M. Yan, C. Liu, and L. Liu, “Multi-dimension convolutional neural network for bug localization,” _IEEE Transactions on Services Computing_ , pp. 1–1, 2020.