# Large Language Models in Fault Localisation

Yonghao Wu<EMAIL_ADDRESS>Beijing University of Chemical
TechnologyBeijingChina ,  Zheng Li<EMAIL_ADDRESS>Beijing
University of Chemical TechnologyChina ,  Jie M. Zhang<EMAIL_ADDRESS>University College LondonLondonUK ,  Mike Papadakis<EMAIL_ADDRESS>University of LuxembourgLuxembourg ,  Mark Harman<EMAIL_ADDRESS>University College LondonLondonUK and  Yong Liu<EMAIL_ADDRESS>Beijing University of Chemical TechnologyBeijingChina

###### Abstract.

Large Language Models (LLMs) have shown promise in multiple software
engineering tasks including code generation, program repair, code
summarisation, and test generation. Fault localisation is instrumental in
enabling automated debugging and repair of programs and was prominently
featured as a highlight during the launch event of ChatGPT-4. Nevertheless,
the performance of LLMs compared to state-of-the-art methods, as well as the
impact of prompt design and context length on their efficacy, remains unclear.
To fill this gap, this paper presents an in-depth investigation into the
capability of ChatGPT-3.5 and ChatGPT-4, the two state-of-the-art LLMs, on
fault localisation. Using the widely-adopted large-scale Defects4J dataset, we
compare the two LLMs with the existing fault localisation techniques. We also
investigate the consistency of LLMs in fault localisation, as well as how
prompt engineering and the length of code context affect the fault
localisation effectiveness.

Our findings demonstrate that within function-level context, ChatGPT-4
outperforms all the existing fault localisation methods. Additional error logs
can further improve ChatGPT models’ localisation accuracy and consistency,
with an average 46.9% higher accuracy over the state-of-the-art baseline
SmartFL on the Defects4J dataset in terms of $TOP$-$1$ metric. However, when
the code context of the Defects4J dataset expands to the class-level,
ChatGPT-4’s performance suffers a significant drop, with 49.9% lower accuracy
than SmartFL under $TOP$-$1$ metric. These observations indicate that although
ChatGPT can effectively localise faults under specific conditions, limitations
are evident. Further research is needed to fully harness the potential of LLMs
like ChatGPT for practical fault localisation applications.

Large Language Model, Fault Localisation, ChatGPT, Empirical Study

††ccs: Software and its engineering Software testing and debugging

##  1\. Introduction

Among all the software engineering activities, fault localisation is notable
as an important phase in the debugging process of software systems, serving as
a prerequisite for developers to fix program errors accurately (Li et al.,
2022) and also automatic program repair (Assiri and Bieman, 2017). Efficient
and accurate fault localisation techniques can significantly improve the
efficiency of software repair and maintenance (Khan et al., 2021; Wu et al.,
2022). Current fault localisation methods employ a variety of techniques,
including statistical analysis (Kim et al., 2021), coverage analysis (Wu et
al., 2020; Papadakis and Le Traon, 2015), and machine learning algorithms
(Weimer et al., 2009; Ni et al., 2022). However, a prevalent limitation of
current fault localisation techniques is their adaptability, which is often
restricted by specific programming languages, the quality of test cases, or
code structures, consequently diminishing their effectiveness (Wong et al.,
2016).

Large Language Models (LLMs), exemplified by ChatGPT, have attracted
substantial interest within the computational and data science communities due
to their wide-ranging applications and powerful performance (Ray, 2023; Feng
et al., 2023). Their remarkable effectiveness in understanding natural
language (Tunstall et al., 2022) and generating meaningful content
(Chakraborty et al., 2022; Fried et al., 2022) has spurred interest across
disciplines including software engineering, where they have shown promises
particularly on code generation (Tian et al., 2023; Dong et al., 2023),
program repair (Xie et al., 2023b; Cao et al., 2023; Sobania et al., 2023; Wei
et al., 2023), code summarisation (Hu et al., 2020; Sun et al., 2023; Sridhara
et al., 2023), and test generation (Xie et al., 2023a; Schäfer et al., 2023;
Liu et al., 2023).

LLMs also have the potential in aiding fault localisation. In the ChatGPT-4
launch event, OpenAI demonstrated the model’s capability to localise and
rectify faults, given knowledge of a program’s code and error log. This
demonstration unveiled the potential and applicability of ChatGPT-4 in fault
localisation. Nevertheless, it remains unclear how the performance of LLMs
compares to the state-of-the-art, and how performances are influenced by
prompt design and context length.

To fill this gap, in this paper, we comprehensively study the performance of
ChatGPT in fault localisation. The findings reveal its current advantages,
shortcomings, and provide insights for future research. Our study aims to
answer the following research questions.

  * •

RQ1. How does ChatGPT perform in fault localisation?

    * –

RQ1.1. How does the accuracy of ChatGPT’s fault localisation compare to state-
of-the-art methods? This RQ aims to examine the capability of ChatGPT-3.5 and
ChatGPT-4 in localising faults within programs from the Defects4J dataset.
Using the $TOP$-$N$ metric, a commonly accepted standard of evaluation, we can
quantify the number of accurately localised faulty functions.

    * –

RQ1.2. How do ChatGPT’s fault localisation results overlap with state-of-the-
art methods? This RQ aims to conduct a more detailed analysis, building on the
general performance comparison offered by RQ1.1. Specifically, we investigate
the number of programs where ChatGPT performs uniquely or is missing compared
to other baselines. This comparative analysis can further reveal the
situations and reasons existing methods surpass ChatGPT.

    * –

RQ1.3. How consistent is ChatGPT’s performance in fault localisation across
repeated experiments? To assess ChatGPT’s reliability in fault localisation,
we conduct five repeated experiments in RQ1.1. This RQ focuses on identifying
potential inconsistencies in ChatGPT’s performance and understanding how
consistency manifests over experiments.

  * •

RQ2. How is ChatGPT’s fault localisation performance affected by variations in
prompt design? This RQ examines the experimental impact of omitting individual
components from a standard prompt on ChatGPT’s fault localisation ability.
Through evaluating the model’s performance with various optional components
excluded, we aim to identify which components are crucial for achieving
optimal results.

  * •

RQ3. How does the length of code context in prompts influence ChatGPT’s fault
localisation performance? This RQ explores how expanding or narrowing the
scope of code context in prompts affects ChatGPT’s performance in localising
faults within large codebases. We evaluate ChatGPT’s accuracy when the context
of the Defects4J dataset is expanded to class-level context versus narrowed to
only statements surrounding the fault. The results will reveal insights into
ChatGPT’s capabilities and limitations for practical fault localisation in
varying scales of software systems.

In addition, to alleviate the potential overfitting threat that Defects4J
being part of ChatGPT’s training data, we have collected a more recent dataset
named “StuDefects”, in which all the programs were written in and after 2022,
hence are outside the timeframe of ChatGPT’s data acquisition period. An
extended analysis has been carried out on this new dataset to validate our
conclusion.

The contributions of this work are summarised as follows:

  * •

We conduct a comprehensive empirical study on the large-scale open-source
program Defects4J, to evaluate the potential of LLM models for fault
localisation research. Our results demonstrate that, with function-level
context, ChatGPT-4, when augmented with test cases and error logs, achieves a
mean value of 23.13 in terms of $TOP$-$1$ evaluation metric, outperforming the
state-of-the-art baseline SmartFL by an average of 46.9%. However, when the
code context is expanded from function level to class level, the effectiveness
of ChatGPT-4 deteriorates markedly, performing 49.9% worse than SmartFL.

  * •

We assess the consistency of ChatGPT for fault localisation. Our findings
suggest that incorporating dynamic execution information into ChatGPT enables
more consistent fault localisation results, reducing the average variance in
$TOP$-$1$ metric to 12.00% for ChatGPT-4, the lowest among ChatGPT related
methods.

  * •

We evaluate the influence of each component in the prompt on the fault
localisation capabilities of ChatGPT. Our results indicate that excluding any
component leads to a decline in the performance of ChatGPT-4, as measured by
$TOP$-$1$ metric. Among them, excluding the error log causes the most
significant impact, resulting in a 25.6% reduction in accuracy.

  * •

We alleviate the concern of ChatGPT’s potential overfitting to Defects4J by
introducing a novel, more recent dataset named “StuDefects”. Our extended
analysis on this new dataset yields consistent observations with those from
Defects4J (52.9% superiority over the state-of-the-art in terms of $TOP$-$1$).

The data and code supporting the findings of this study, including the
original responses from ChatGPT, are publicly available on GitHub (thi, 2023).

The remainder of this paper is organised as follows. Section 2 outlines the
experimental design, including descriptions of the dataset, baseline models,
ChatGPT setup and evaluation metrics used in this study. Section 3 delves into
a detailed analysis of the results for each RQ. Section 4 provides an extended
analysis of the generalisability of ChatGPT on the new dataset we collected.
Section 5 reviews related prior work. Section 6 discusses threats to validity
and implications for applying LLMs to fault localisation. Section 7 summarises
our study and discusses potential future work.

##  2\. Experimental Design

###  2.1. Dataset

Defects4J (Def, 2014) is a widely used fault localisation dataset containing
real-world open-source programs with real bugs that occurred in their history
versions and corresponding test cases (Just et al., 2014).

Table 1. Statistic of the Defect4J Dataset Program |  Number of Versions |  Number of Faulty Functions |  Average Length of Code
---|---|---|---
Chart | 17 | 21 | 62,395
Lang | 35 | 52 | 13,441
Math | 66 | 89 | 36,226
Mockito | 10 | 33 | 6,722
Time | 15 | 24 | 20,357
Closure | 52 | 61 | 64,538
Sum | 195 | 280 | -

In this study, we use six large-scale open-source Java projects from Defects4J
for experimental analysis, summarized in Table 1. These programs are
JFreeChart (Chart), Google Closure compiler (Closure), Apache Commons-Lang
(Lang), Apache Commons-Math (Math), Mockito Framework (Mockito), and Joda-Time
(Time).

The first column of Table 1 lists these six programs. The second, third, and
fourth columns provide the corresponding number of available versions, the
number of faulty functions, and the average lines of code for each program,
respectively.

These projects contain multiple program versions, where versions that fail to
compile or have segmentation errors are excluded. We take the intersection of
program versions supported by each baseline as the final dataset, which
contains a total of 195 program versions. Additionally, these projects vary in
scale, complexity, and code length. We use the Defects4J dataset as it is
widely used in research (Lou et al., 2021; Zou et al., 2019; Zakari et al.,
2020), making our results comparable with much existing literature, providing
a dependable evaluation of fault localisation techniques.

###  2.2. Baselines

####  2.2.1. Spectrum-Based Fault Localisation (SBFL)

SBFL is a debugging technique that leverages execution traces of passed and
failed test cases to estimate the likelihood of program statements being
faulty. The core intuition is that faulty statements are more frequently
covered by failed tests compared to passed tests. SBFL techniques quantify
this intuition by computing the suspiciousness value of each program
statement. Common suspiciousness formulas of SBFL proposed in literature
include $Jaccard$ (Jaccard, 1901), $Tarantula$ (Jones et al., 2002), $Ochiai$
(Abreu et al., 2006), $OP2$ (Naish et al., 2011), and $Dstar$ (Wong et al.,
2013). Then, SBFL ranks program statements based on their suspiciousness
values. Developers can then inspect the ranked list, starting from the most
suspicious statements to identify potential bug locations.

In this study, we adopt $Ochiai$ and $Dstar$ formulas of SBFL to calculate the
suspiciousness value of program statements, as these formulas have
demonstrated the best performance in existing research (Pearson et al., 2017;
Zou et al., 2019). The formulas of $Ochiai$ and $Dstar$ in SBFL are:

$Ochiai(statement)=\frac{e_{f}}{\sqrt{(e_{f}+n_{f})\times(e_{f}+e_{p})}}$

$Dstar(statement)=\frac{e_{f}^{*}}{n_{f}+e_{p}}$

Where Ochiai(statement) or Dstar(statement) signify the suspiciousness values
for a given program statement. The variables $e_{p}$ and $e_{f}$ denote the
counts of passed and failed test cases covering this statement. Conversely,
$n_{f}$ indicates the count of failed test cases not covering the program
statement. For the $Dstar$ formula, the exponent “*” is set to 3, as
recommended by Wong et al. (Wong et al., 2013).

A major advantage of SBFL is its lightweight instrumentation to collect
coverage traces without needing program semantics. One limitation is that SBFL
relies on the quality of the test suite - its effectiveness is constrained by
how well the test cases execute the code and reveal faults.

####  2.2.2. Mutation-Based Fault Localisation (MBFL)

MBFL localises potential bugs by leveraging program mutants (Moon et al.,
2014; Papadakis and Le Traon, 2015). Traditional MBFL automatically seeds bugs
by applying mutation operators such as replacing operators. The mutated
programs (mutants) are executed against the test suite to identify mutants
killed by test cases. By comparing test outputs between the original and
mutants, MBFL allocates program statements frequently impacted by killed
mutants with higher suspiciousness values. These statements receive higher
mutation scores indicative of their correlation with bugs. A salient strength
of MBFL is its ability to achieve high fault localisation accuracy on
traditional datasets such as Software-artifact Infrastructure Repository (SIR)
(Papadakis and Le Traon, 2015).

In this study, we refer to the experimental results of existing research and
choose Metallaxis, which performs best in MBFL to calculate the suspiciousness
of a statement (Zou et al., 2019; Jia et al., 2020; Pearson et al., 2017). For
each specific program statement, MBFL generates a set of mutants
$M(statement)=\langle m_{1},m_{2},\ldots\rangle$ where $m_{i}\in
M(statement)$. MBFL subsequently evaluates the suspiciousness of each mutant
using the Metallaxis formula and attributes the highest such value to the
original program statement, thereby representing its likelihood of being
faulty. The formulas of $Ochiai$ and $Dstar$ in MBFL are:

$Ochiai(m_{i})=\frac{failed(m_{i})}{\sqrt{totalfailed\times(failed(m_{i})+passed(m_{i}))}}$

$Dstar(m_{i})=\frac{failed(m_{i})^{*}}{(totalfailed-
failed(m_{i}))+passed(m_{i})}$

Where $failed(m_{i})$ represents the count of test cases that failed on the
original program and exhibit altered output when run on a mutant $m_{i}$.
$passed(m_{i})$ represents the count of test cases that passed on the original
program but exhibit different output when run on the mutant $m_{i}$.
$totalfailed$ denotes the aggregate count of test cases that fail when
executed on the original program.

However, the process of generating and executing a multitude of mutants
entails substantial computational overhead (Liu et al., 2018). Moreover, the
effectiveness of MBFL is closely tied to the killing power of both the test
suite and the generated mutants. While MBFL achieves promising results on
datasets with artificially seeded faults, recent studies using real-world
defects indicate that MBFL underperforms SBFL on Defects4J dataset. For
example, research by Zou et al. (Zou et al., 2019) demonstrates that in the
Defects4J dataset, SBFL outperforms MBFL across $TOP$-$3$, $TOP$-$5$, and
$TOP$-$10$ evaluation metrics. Specifically, the $Ochiai$ formula within SBFL
is able to localise 156 faults when assessed by $TOP$-$10$ metric, while the
most effective technique within MBFL localises 129 faults. Existing research
analyzes the limitations of traditional MBFL. These methods rely on a limited
set of manually-designed operators that focus on simple syntactic changes,
failing to adequately represent real faults due to their lack of semantic
depth (Tian et al., 2022).

####  2.2.3. Semantics-based probabilistic Fault Localisation (SmartFL)

Zeng et al. (Zeng et al., 2022) proposed SmartFL, a novel statement-level
fault localisation approach that models the likelihood of faulty program
statements using probabilistic inference. It combines static analysis and
dynamic execution traces to estimate the correctness of program values without
capturing full semantics. SmartFL achieves a balance between effectiveness and
scalability. Their paper shows that SmartFL outperforms existing spectrum-
based, mutation-based, and semantics-based methods on Defects4J dataset,
achieving 21% $TOP$-$1$ statement accuracy, 5% higher than the next best
method.

###  2.3. ChatGPT Setup

In this study, we explore four fault localisation methods using different
versions and prompt strategies of ChatGPT, named ChatGPT-3.5 (Origin),
ChatGPT-3.5 (Log), ChatGPT-4 (Origin), and ChatGPT-4 (Log). ChatGPT-3.5 and
ChatGPT-4 denote two different versions of ChatGPT. “Origin” means that the
information provided with ChatGPT in the prompt is merely the target function
along with basic instructions, while “Log” means that we further provide
ChatGPT with a failed unit test case and the corresponding error log in a
follow-up dialogue. In the following, we introduce the details for our setup
as well as specific examples of our designed prompts.

#### Access to ChatGPT

For each version of ChatGPT, to get optimal performance, we conducted a
preliminary study to compare the performance accessed via API with that via
web interfaces using the Chart program from Defects4J Our results reveal that
ChatGPT-3.5 has comparable results between the web interface and the API
(e.g., 4.6 v.s. 4.8), while the web interface for ChatGPT-4 has better
performance than API (12.4 v.s. 8 with log information). Therefore, we adopt
the API for all ChatGPT-3.5 experiments for efficiency and choose the web
interface for ChatGPT-4 due to its superior performance. For the web interface
of ChatGPT-4, we have double-checked and guaranteed that the model behind it
remains identical throughout our experimental procedure.

#### Code Context

Existing LLMs often have limitations in the length of tokens they can accept
in the prompt. For example, ChatGPT-3.5 and ChatGPT-4 have a token limit of
4096 and 8192, respectively. As a result, this prevents the application of
LLMs on large-scale codebases such as those in Defects4J, which may contain
tens of thousands of lines.

We address this issue, in line with state-of-the-art LLM-based research in
software engineering (Prenner et al., 2022; Xia et al., 2022), by extracting
the function containing faulty statements in the program under test. This
function provides the context in the prompt, effectively narrowing the scope
of the code context for ChatGPT to localising faulty statements within the
target function. In addition, in order to ensure a fair comparison, in the
subsequent experimental analyses, the fault localisation scope across all
baselines is kept consistent with that of ChatGPT.

Furthermore, in RQ3, we perform comparative experiments on the Defects4J
dataset to analyse the impact of varying the length of code context in prompts
on ChatGPT’s performance.

#### Design of Prompt

For ChatGPT-3.5/4 (Origin), the prompt we privide includes the complete target
function and basic instructions to ChatGPT. We call this prompt $Prompt_{1}$.
The basic instructions in $Prompt_{1}$ request ChatGPT to present potential
faulty locations sorted in descending order of suspicion. Additionally, an
explanation is required to explain how each location is pinpointed as
potentially faulty, a choice inspired by Gao et al. (Gao et al., 2023) where
they found that self-explanatory prompts significantly enhance LLMs’
comprehension in complex dialogues.

To illustrate, an example of $Prompt_{1}$ is provided below.

Example of $Prompt_{1}$:

Please analyse the following code snippet for potential bugs. Return the
results in JSON format, consisting of a single JSON object with two fields:
“intentOfThisFunction” (describing the intended purpose of the function), and
“faultLocalization” (an array of JSON objects). The “faultLocalization” array
should contain up to five JSON objects, each with three fields: “lineNumber”
(indicating the line number of the suspicious code), “codeContent” (showing
the actual code), and “reason” (explaining why this location is identified as
potentially faulty). Note: The codes in the “faultLocalization” array should
be listed in descending order of suspicion.

⬇

if (a+b<c || a+c<b || b+c<a) {

return ”Invalid triangle”;

}

if (a==b && b==c) {

return ”Equilateral triangle”;

} else if (a==b || b==c || a==c) {

return ”Isosceles triangle”;

} else {

return ”Scalene triangle”;

}

}

An answer from ChatGPT-4 to $Prompt_{1}$ for a function that determines the
type of triangle based on side lengths is provided below.

Answer to $Prompt_{1}$:

In this answer, ChatGPT identifies line 2 as a potential fault, since the
current “¡” condition cannot handle the invalid triangle case when the sum of
the lengths of the two sides equals the third side. This example demonstrates
ChatGPT’s capacity to provide reasoned fault localisation by analysing code
logic and limitations.

For ChatGPT-3.5/4 (log), we aim to further reveal ChatGPT’s fault localisation
capability with another prompt, called $Prompt_{2}$, that contains a
corresponding failed unit test case and the error log derived from the test
execution results. $Prompt_{2}$ is provided to ChatGPT after the first
interaction with $Prompt_{1}$ to let ChatGPT re-examine and refine its initial
suspected faulty locations. An example of $Prompt_{2}$ is illustrated below.

Example of $Prompt_{2}$:

I have received an error message and a unit test case related to the code
snippet I provided in the first prompt. The error message is:

⬇

org.junit.ComparisonFailure:

Expected :Invalid triangle

Actual :Isosceles triangle

at org.junit.Assert.assertEquals(Assert.java:115)

at org.junit.Assert.assertEquals(Assert.java:144)

at com.example.TriangleTest.testDetermineTriangleType(TriangleTest.java:6)

...

Additionally, here is the unit test case:

⬇

import org.junit.Test;

public class TriangleTest {

@Test

public void testDetermineTriangleType() {

assertEquals(”Invalid triangle”, determineTriangleType(-1.0,-1.0,1.0));

}

}

Please analyse the code snippet from the first prompt, along with the provided
error message and unit test case. Update and return the JSON object consisting
of “intentOfThisFunction” (describing the intended purpose of the function),
and “faultLocalization” (an array of JSON objects). The “faultLocalization”
array should contain up to five JSON objects, each with three fields:
“lineNumber” (indicating the line number of the suspicious code),
“codeContent” (showing the actual code), and “reason” (explaining why this
location is identified as potentially buggy). Note: The codes in the
“faultLocalization” array should be listed in descending order of suspicion,
and the analysis should focus exclusively on the code snippet from the first
prompt and not the unit test case.

In RQ2, we conduct an ablation study to check how different components in the
prompts affect fault localisation.

#### Collection of Results

For convenient management of experimental results, we instruct ChatGPT to
provide fault localisation outputs in the universally recognised JSON format.
The response is expected to adhere to the format and content specified in the
prompt requirements. In case of any deviations, we continue to resubmit the
prompt until the output aligns with the given specifications. Furthermore,
some target functions in Defects4J may avoid execution by test cases or not
trigger exceptions, failing to yield error log information. In such cases, the
fault localisation results of ChatGPT-3.5/4 (Log) equal to those of
ChatGPT-3.5/4 (Origin).

After getting the JSON result given by ChatGPT, we parse the JSON string and
follow the instructions of $Prompt_{1}$ to take the first five code statements
in the JSON string as the fault localisation result of ChatGPT. Finally, we
can use this result to evaluate the performance of ChatGPT in terms of
$TOP$-$1$ to $TOP$-$5$ metrics ($TOP$-$N$ metric will be described in Section
2.4.1).

#### Mitigation of Randomness

Due to ChatGPT’s inherent randomness, we adopt a rigorous method with repeated
experiments for each target function. Specifically, all ChatGPT experiments in
this paper are performed five times independently. We then average the
$TOP$-$N$ metric as the final fault localisation outcome. This ensures
reliable results by minimizing ChatGPT’s random bias.

###  2.4. Measurement of Fault Localisation Performance

####  2.4.1. TOP-N

We use the widely adopted $TOP$-$N$ metric to measure the accuracy of fault
localisation techniques (Zeng et al., 2022; Lou et al., 2021; Wu et al.,
2020). $TOP$-$N$ metric counts the number of faulty functions that contain
faulty statements with a suspiciousness rank less than or equal to $N$ (Li and
Zhang, 2017). In other words, $TOP$-$N$ refers to the number of faulty
functions that developers can localise the exact position of faulty statements
by checking the top $N$ statements. Therefore, higher $TOP$-$N$ values
indicate more effective fault localisation. Prior studies indicate that 70% of
developers and testers focus only on the top five program statements in fault
localisation ranking lists (Kochhar et al., 2016). Thus, we focus our
evaluation metrics from $TOP$-$1$ to $TOP$-$5$.

It is worth noting that the baselines in this paper (i.e., SBFL, MBFL, and
SmartFL) may assign the same suspiciousness score to correct statements as to
faulty ones (Steimann et al., 2013; Wong et al., 2016). For these
circumstances, we use the average ranking strategy, which is commonly employed
in existing studies (Wong et al., 2016; Pearson et al., 2017). That is, we use
the average ranks of all statements with the same suspiciousness score as the
final ranks of faulty statements.

####  2.4.2. Wilcoxon Signed-Rank Test

In this study, we use the Wilcoxon signed-rank test, a widely adopted non-
parametric statistical analysis approach in fault localisation (Kim et al.,
2019; Li et al., 2019; Gao and Wong, 2018), to determine the confidence level
concerning the varied effectiveness of different fault localisation methods.
The Wilcoxon signed-rank test offers a robust evaluation that is specifically
useful when the normal distribution of data cannot be assumed (Ott and
Longnecker, 2015). It assesses if two related samples come from identical
populations by comparing ranks of paired differences, and is ess sensitive to
outliers than mean comparisons.

##  3\. Results Analysis

###  3.1. RQ1: Effectiveness of ChatGPT in Fault Localisation

RQ1.1: Comparison with Baselines: In this section, we comprehensively compare
the efficacy of different fault localisation techniques. Table 2 presents the
results on the six Defects4J programs, as measured by $TOP$-$N$ metric. The
performance metrics for ChatGPT-3.5 and ChatGPT-4 are the averages of five
independent repeated experiments, thus ensuring the statistical significance
of our results. Additionally, in the table, the best-performing method in each
row, as indicated by the largest $TOP$-$N$ value, is highlighted in green. The
state-of-the-art baseline SmartFL was not implemented in two of the programs
(Mockito and Closure) in its proposed paper, and we use “-” as a substitute in
the corresponding situations.

Table 2. RQ1.1: Fault Localisation Effectiveness on Defects4J  Program | $TOP$-$N$ | Method
---|---|---
SBFL | MBFL | SmartFL | ChatGPT-3.5 | ChatGPT-4
$Ochiai$ | $Dstar$ | $Ochiai$ | $Dstar$ | Origin | Log | Origin | Log
Chart |  $TOP$-$1$ | 3 | 3 | 3 | 3 | 8 | 4.4 | 4.8 | 11 | 12.4
$TOP$-$2$ | 8 | 8 | 11 | 11 | 9 | 6.2 | 8.8 | 14.4 | 15.2
$TOP$-$3$ | 9 | 8 | 14 | 14 | 11 | 8.2 | 10.4 | 16.2 | 17
$TOP$-$4$ | 11 | 12 | 14 | 14 | 12 | 8.8 | 10.8 | 16.4 | 17.2
$TOP$-$5$ | 13 | 13 | 14 | 14 | 13 | 10 | 11.4 | 16.6 | 17.6
Lang |  $TOP$-$1$ | 8 | 8 | 9 | 7 | 19 | 13.6 | 13.6 | 17.6 | 26.6
$TOP$-$2$ | 17 | 16 | 26 | 25 | 29 | 20.2 | 20.2 | 21.4 | 31.4
$TOP$-$3$ | 23 | 22 | 31 | 30 | 31 | 23.8 | 24.8 | 24.2 | 35
$TOP$-$4$ | 27 | 27 | 35 | 34 | 34 | 26 | 26.6 | 26.6 | 36.8
$TOP$-$5$ | 29 | 28 | 39 | 38 | 38 | 27 | 28.4 | 27.6 | 37.4
Math |  $TOP$-$1$ | 24 | 20 | 25 | 25 | 32 | 26.8 | 30.8 | 28.4 | 41.8
$TOP$-$2$ | 37 | 34 | 44 | 44 | 42 | 41.2 | 43.4 | 45.2 | 58.8
$TOP$-$3$ | 44 | 39 | 53 | 53 | 51 | 46.2 | 48 | 54.2 | 64.4
$TOP$-$4$ | 48 | 43 | 57 | 57 | 56 | 50.2 | 52.4 | 59 | 69.6
$TOP$-$5$ | 57 | 51 | 65 | 65 | 62 | 54 | 56 | 61.2 | 72
Mockito |  $TOP$-$1$ | 16 | 16 | 8 | 8 | - | 22.6 | 23.8 | 21.8 | 28.4
$TOP$-$2$ | 27 | 26 | 25 | 24 | - | 27.2 | 27.8 | 24.2 | 29.2
$TOP$-$3$ | 30 | 29 | 26 | 26 | - | 28.6 | 29.2 | 26.8 | 30.2
$TOP$-$4$ | 30 | 29 | 27 | 27 | - | 29.2 | 29.8 | 28.2 | 30.4
$TOP$-$5$ | 31 | 30 | 28 | 28 | - | 29.6 | 30 | 28.6 | 30.4
Time |  $TOP$-$1$ | 5 | 5 | 5 | 5 | 4 | 3.8 | 5.4 | 4.2 | 7.6
$TOP$-$2$ | 5 | 7 | 6 | 6 | 6 | 6.8 | 8.6 | 5.6 | 8.6
$TOP$-$3$ | 10 | 10 | 9 | 9 | 7 | 8.2 | 10.6 | 7.2 | 10.2
$TOP$-$4$ | 12 | 12 | 11 | 11 | 11 | 10 | 12.4 | 8.6 | 12
$TOP$-$5$ | 18 | 18 | 17 | 17 | 16 | 11.8 | 14.2 | 13.2 | 16.4
Closure |  $TOP$-$1$ | 17 | 14 | 1 | 1 | - | 10.4 | 9.6 | 12.6 | 22
$TOP$-$2$ | 22 | 18 | 6 | 6 | - | 17 | 17.6 | 18.8 | 28.6
$TOP$-$3$ | 25 | 22 | 10 | 10 | - | 21 | 23 | 23.8 | 32.6
$TOP$-$4$ | 34 | 30 | 11 | 11 | - | 24.4 | 26 | 26.8 | 34.4
$TOP$-$5$ | 38 | 33 | 14 | 14 | - | 27 | 27.2 | 29 | 35.8
Average |  $TOP$-$1$ | 12.17 | 11.00 | 8.50 | 8.17 | 15.75 | 13.60 | 14.67 | 15.93 | 23.13
$TOP$-$2$ | 19.33 | 18.17 | 19.67 | 19.33 | 21.50 | 19.77 | 21.07 | 21.60 | 28.63
$TOP$-$3$ | 23.50 | 21.67 | 23.83 | 23.67 | 25.00 | 22.67 | 24.33 | 25.40 | 31.57
$TOP$-$4$ | 27.00 | 25.50 | 25.83 | 25.67 | 28.25 | 24.77 | 26.33 | 27.60 | 33.40
$TOP$-$5$ | 31.00 | 28.83 | 29.50 | 29.33 | 32.25 | 26.57 | 27.87 | 29.37 | 34.93

The results presented in Table 2 show that ChatGPT-4 (Log) yield the most
promising results, highlighting the significant advancements made in
developing enhanced fault localisation techniques. ChatGPT-3.5 (Origin) and
ChatGPT-3.5 (Log) achieve optimal performance in limited cases. Meanwhile,
SmartFL sets a higher contemporary benchmark for overall efficacy over
traditional fault localisation techniques SBFL and MBFL.

Answer to RQ1.1: ChatGPT-4 (Log), i.e., providing ChatGPT-4 with the log
information of the failed tests, yields the best effectiveness in fault
localisation on Defects4J. The mean value of $TOP$-$1$ for ChatGPT-4 (Log) is
23.13, outperforming the state-of-the-art baseline SmartFL by 46.9%.

RQ1.2: Overlap of Different Methods: In this section, we analyze the overlap
between ChatGPT-4 (Log), the best-performing way of using ChatGPT in fault
localisation, and other methods in fault localisation, with the total 186
faults. Table 3 shows the results, where Column “Unique” shows the number and
ratio of faults where ChatGPT-4 (Log) can locate while other methods cannot.
Take the first cell as an example, the value “34” indicates that there are 34
(18.28%) bugs ChatGPT-4 (Log) can locate but ChatGPT-4 (Origin) cannot in
terms of $Top$-$1$. Column “Overlapping” shows the number and ratio of faults
where both ChatGPT-4 (Log) and the given baseline method are able to localise.
Column “Missing” shows the number and ratio of faults that only the baseline
method can locate, while ChatGPT-4 (Log) cannot. Experiments in this section
do not include programs not implemented in SmartFL (i.e., Mockito and
Closure). Consequently, there are a total of 186 remaining faulty versions.

Table 3. RQ1.2: Overlap Analysis of ChatGPT-4 (Log) and Other Methods.  Alternative methods | Metric | Unique | Overlapping | Missing
---|---|---|---|---
ChatGPT-4 (Origin) | $TOP-1$ | 34 (18.28%) | 145 (77.96%) | 7 (3.76%)
$TOP-5$ | 28 (15.05%) | 155 (83.33%) | 3 (1.61%)
ChatGPT-3.5 (Origin) | $TOP-1$ | 53 (28.49%) | 120 (64.52%) | 13 (6.99%)
$TOP-5$ | 55 (29.57%) | 117 (62.9%) | 14 (7.53%)
ChatGPT-3.5 (Log) | $TOP-1$ | 48 (25.81%) | 123 (66.13%) | 14 (7.53%)
$TOP-5$ | 50 (26.88%) | 120 (64.52%) | 16 (8.6%)
SBFL ($Ochiai$) | $TOP-1$ | 61 (32.8%) | 112 (60.22%) | 13 (6.99%)
$TOP-5$ | 48 (25.81%) | 116 (62.37%) | 22 (11.83%)
MBFL ($Ochiai$) | $TOP-1$ | 59 (31.72%) | 114 (61.29%) | 13 (6.99%)
$TOP-5$ | 29 (15.59%) | 136 (73.12%) | 21 (11.29%)
SmartFL | $TOP-1$ | 44 (23.66%) | 124 (66.67%) | 18 (9.68%)
$TOP-5$ | 37 (19.89%) | 126 (67.74%) | 23 (12.37%)

As observed from Table 3, the values of “Unique” are all larger than the
values of “Missing”, indicating that ChatGPT-4 (Log) successfully localises
more unique faults than the other methods. Further, ChatGPT-4 (Log) and
ChatGPT-4 (Origin) have the highest “Overlapping” values, signifying
consistent performance in more cases. Additionally, among all baselines,
ChatGPT-4 (Log) has the largest “Missing” value when compared to SmartFL. This
suggests that, compared to other baselines, SmartFL surpasses ChatGPT-4 (Log)
in the largest number of instances.

We manually analyse the 12 cases where ChatGPT-4 (Log) is missing compared to
SmartFL. Seven of these cases can be attributed to missing logic, examples of
which will be illustrated below.

⬇

if (pfxLen > 0) {

+ char firstSigDigit = 0;

+ for(int i = pfxLen; i < str.length(); i++) {

+ firstSigDigit = str.charAt(i);

+ if (firstSigDigit == ’0’) {

+ pfxLen++;

+ } else {

+ break;

+ }

+ }

final int hexDigits = str.length() - pfxLen;

- if (hexDigits > 16) {

+ if (hexDigits > 16 || (hexDigits==16 && firstSigDigit > ’7’)) {

return createBigInteger(str);

}

- if (hexDigits > 8) {

+ if (hexDigits > 8 || (hexDigits==8 && firstSigDigit > ’7’)) {

return createLong(str);

}

return createInteger(str);

}

The above code snippet shows that addressing such faults requires adding more
logical code, without modifying the original programs. This implies that
ChatGPT-4 (Log) exhibits insufficient performance on localising faults
associated with logical inadequacies. To improve the effectiveness of
ChatGPT-4 (Log) in fault localisation, future enhancements could focus on
augmenting its comprehension and identification of such missing logical code.
Answer to RQ1.2: ChatGPT-4 (Log) can identify numerous unique faults that are
untraceable by alternative methods. For example, compared to SmartFL (the
state-of-the-art baseline), there are on average 44 (23.66%) faults that
ChatGPT-4 (Log) can localise but SmartFL cannot.

RQ1.3: Consistency of ChatGPT: In this section, we assess the consistency of
ChatGPT’s fault localisation, focusing on variability across repeated
experiments. We present Table 4 showcasing results on the Defects4J dataset
from five repeated experiments.

Table 4. RQ1.3: Results of Repeated Experiments on the Defects4J Dataset

Method | Time |  $TOP$-$N$ | Method |  $TOP$-$N$
---|---|---|---|---
$TOP$-$1$ |  $TOP$-$2$ |  $TOP$-$3$ |  $TOP$-$4$ |  $TOP$-$5$ |  $TOP$-$1$ |  $TOP$-$2$ |  $TOP$-$3$ |  $TOP$-$4$ |  $TOP$-$5$
ChatGPT-3.5 (Origin) | 1 | 80 | 123 | 143 | 153 | 164 |  ChatGPT-4 (Origin) | 100 | 139 | 157 | 172 | 180
2 | 82 | 119 | 136 | 151 | 158 | 94 | 125 | 144 | 159 | 171
3 | 81 | 114 | 134 | 146 | 159 | 99 | 133 | 155 | 163 | 172
4 | 88 | 125 | 137 | 151 | 162 | 93 | 126 | 156 | 169 | 180
5 | 77 | 112 | 130 | 142 | 154 | 92 | 125 | 150 | 165 | 178
Average | 81.6 | 118.6 | 136.0 | 148.6 | 159.4 | Average | 95.6 | 129.6 | 152.4 | 165.6 | 176.2
ChatGPT-3.5 (Log) | 1 | 89 | 131 | 149 | 156 | 167 |  ChatGPT-4 (Log) | 135 | 170 | 187 | 198 | 208
2 | 87 | 132 | 149 | 162 | 165 | 136 | 170 | 189 | 197 | 207
3 | 86 | 118 | 141 | 155 | 164 | 145 | 174 | 190 | 199 | 207
4 | 93 | 126 | 146 | 163 | 175 | 142 | 173 | 193 | 208 | 217
5 | 85 | 125 | 145 | 154 | 165 | 136 | 172 | 188 | 200 | 209
Average | 88.0 | 126.4 | 146.0 | 158.0 | 167.2 | Average | 138.8 | 171.8 | 189.4 | 200.4 | 209.6

As illustrated in Table 4, despite minor fluctuations attributable to
randomness, a consistent pattern emerges regarding the comparative
performance. ChatGPT-4 (Log) consistently outperforms the competition, ranking
as the top performer in all five repetitions. This evidence serves to
reinforce our conclusions from RQ1.1.

To further assess consistency, we present Table 5 summarising the variances
observed in repeated experiments. The variance acts as a consistency metric,
with lower values indicating higher consistency and insights into each
method’s reliability.

Table 5. RQ1.3: Variance Results of Repeated Experiments

Method | Program |  Variance of $TOP$-$N$ value | Method |  Variance of $TOP$-$N$ value
---|---|---|---|---
$TOP$-$1$ |  $TOP$-$2$ |  $TOP$-$3$ |  $TOP$-$4$ |  $TOP$-$5$ |  $TOP$-$1$ |  $TOP$-$2$ |  $TOP$-$3$ |  $TOP$-$4$ |  $TOP$-$5$
ChatGPT-3.5 (Origin) | Chart | 0.17 | 0.21 | 0.24 | 0.24 | 0.25 |  ChatGPT-4 (Origin) | 0.25 | 0.22 | 0.18 | 0.17 | 0.17
Lang | 0.19 | 0.24 | 0.25 | 0.25 | 0.25 | 0.22 | 0.24 | 0.25 | 0.25 | 0.25
Math | 0.21 | 0.25 | 0.25 | 0.25 | 0.24 | 0.22 | 0.25 | 0.24 | 0.22 | 0.21
Mockito | 0.22 | 0.14 | 0.12 | 0.10 | 0.09 | 0.22 | 0.20 | 0.15 | 0.12 | 0.12
Time | 0.13 | 0.20 | 0.22 | 0.24 | 0.25 | 0.14 | 0.18 | 0.21 | 0.23 | 0.25
Closure | 0.14 | 0.20 | 0.23 | 0.24 | 0.25 | 0.16 | 0.21 | 0.24 | 0.25 | 0.25
Average | 0.13 | 0.14 | 0.12 | 0.10 | 0.09 | Average | 0.14 | 0.18 | 0.15 | 0.12 | 0.12
ChatGPT-3.5 (Log) | Chart | 0.18 | 0.24 | 0.25 | 0.25 | 0.25 |  ChatGPT-4 (Log) | 0.24 | 0.20 | 0.15 | 0.15 | 0.14
Lang | 0.19 | 0.24 | 0.25 | 0.25 | 0.25 | 0.25 | 0.24 | 0.22 | 0.21 | 0.20
Math | 0.23 | 0.25 | 0.25 | 0.24 | 0.23 | 0.25 | 0.22 | 0.20 | 0.17 | 0.15
Mockito | 0.20 | 0.13 | 0.10 | 0.09 | 0.08 | 0.12 | 0.10 | 0.08 | 0.07 | 0.07
Time | 0.17 | 0.23 | 0.25 | 0.25 | 0.24 | 0.22 | 0.23 | 0.24 | 0.25 | 0.22
Closure | 0.13 | 0.21 | 0.23 | 0.24 | 0.25 | 0.23 | 0.25 | 0.25 | 0.25 | 0.24
Average | 0.13 | 0.13 | 0.10 | 0.09 | 0.08 | Average | 0.12 | 0.10 | 0.08 | 0.07 | 0.07

Table 5 reveals that among all ChatGPT-related techniques, ChatGPT-4 (Log) has
the lowest variance. The minimal fluctuation in ChatGPT-4 (Log) signifies it
as the most consistent approach, further supporting its reliability for fault
localisation. This indicates that the dynamic information in $Prompt_{2}$,
including test cases and error logs generated from their execution, not only
enhances ChatGPT-4’s fault localisation but also improves result consistency.

Through consistency assessments, we demonstrate the robustness of our results
against the inherent variability present in LLM-based techniques. Furthermore,
we highlight the enhanced consistency exhibited by ChatGPT-4 (Log), further
validating its potential as a consistent instrument for fault localisation.
Therefore, in subsequent experiments, we will primarily use ChatGPT-4 (Log) as
the representative ChatGPT method for experimental result comparison.  Answer
to RQ1.3: ChatGPT-4 (Log) consistently exhibits superior performance and low
variance across multiple assessments compared with ChatGPT-3.5 (Origin/Log)
and ChatGPT-4 (Origin). ChatGPT-4 (Log) has a mean variance of 0.12, 0.10,
0.08, 0.07 and 0.07 in terms of $TOP$-$1$ to $TOP$-$5$ metrics, which is the
smallest among all ChatGPT-related methods.

###  3.2. RQ2: Impact of Prompt Design

To address RQ2, we conduct an ablation study focusing on prompt engineering in
ChatGPT-4. Specifically, by excluding distinct elements from the original
prompts and comparing the resulting fault localisation outcomes, we aim to
discern the individual contributions of each component to the overall efficacy
of the system.

In addition to the basic mandatory requirements on fault localisation, we
extract four optional components from $Prompt_{1}$ and $Prompt_{2}$. These
include the requirement to sort statements in descending order of suspicion in
$Prompt_{1}$ and $Prompt_{2}$; the requirement to explain the reason for fault
localisation in $Prompt_{1}$ and $Prompt_{2}$; the information about the error
log in $Prompt_{2}$; and the information about the test cases in $Prompt_{2}$,
respectively.

By excluding each component individually, we created and tested four variant
prompt versions:

  * •

ChatGPT-4 NoOrder: This variant removes the need for ChatGPT to rank the
potential faulty statements in descending order of suspicion from $Prompt_{1}$
and $Prompt_{2}$.

  * •

ChatGPT-4 NoExplain: This variant removes the need for explaining the
reasoning behind the current fault localisation results from $Prompt_{1}$ and
$Prompt_{2}$.

  * •

ChatGPT-4 NoError: This variant removes the error logs from $Prompt_{2}$, with
$Prompt_{1}$ remaining unchanged.

  * •

ChatGPT-4 NoTestCase: This variant removes the content of test cases from
$Prompt_{2}$, with $Prompt_{1}$ remaining unchanged.

Table 6 presents a comparative analysis of these four ablation methods with
the original ChatGPT-4 (Origin) and ChatGPT-4 (Log) on the Defects4J dataset.

Table 6. RQ2: Experimental Results of ChatGPT-4 on the Defects4J Dataset under Different Prompts Method |  $TOP$-$N$
---|---
$TOP$-$1$ |  $TOP$-$2$ |  $TOP$-$3$ |  $TOP$-$4$ |  $TOP$-$5$
ChatGPT-4 (Origin) | 95.6 | 129.6 | 152.4 | 165.6 | 176.2
ChatGPT-4 (Origin)NoOrder | 87.2 | 134.8 | 157.4 | 168.0 | 177.8
ChatGPT-4 (Origin)NoExplain | 91.2 | 121.8 | 138.2 | 147.4 | 154.2
ChatGPT-4 (Log) | 138.8 | 171.8 | 189.4 | 200.4 | 209.6
ChatGPT-4 (Log)NoOrder | 105.2 | 155.4 | 184.2 | 196.0 | 204.6
ChatGPT-4 (Log)NoExplain | 132.6 | 164.6 | 179.8 | 187.6 | 192.8
ChatGPT-4 (Log)NoError | 103.2 | 141.4 | 165.4 | 174.8 | 184.6
ChatGPT-4 (Log)NoTestCase | 136.8 | 172.4 | 193.0 | 203.4 | 211.4

The results in Table 6 reveal several insightful observations:

  * •

Both the original ChatGPT-4 (Origin) and ChatGPT-4 (Log) achieve the highest
accuracy under $TOP$-$1$ metric.

  * •

The most detrimental exclusion is the error logs in ChatGPT-4 (Log)NoError,
which substantially diminishes the accuracy across evaluation metrics. This
implies that the error log provides a vital dynamic context for ChatGPT-4 to
effectively localise faults.

  * •

ChatGPT-4 (Origin)NoExplain and ChatGPT-4 (Log)NoExplain, which lack the
requirement to rank potential faults, exhibit noticeable performance
degradation in terms of $TOP$-$N$ metrics.

  * •

Removing test case details in ChatGPT-4 (Log)NoTestCase has minimal impact on
performance, and ChatGPT-4 (Log)NoTestCase even achieves better performance in
some cases.

  * •

ChatGPT-4 (Origin)NoOrder and ChatGPT-4 (Log)NoOrder demonstrate divergent
performance effects. The accuracy of ChatGPT-4 (Log)NoOrder is overall
inferior to ChatGPT-4 (Log), whereas relative to ChatGPT-4 (Origin), ChatGPT-4
(Origin)NoOrder exhibits a performance decline only in terms of $TOP$-$1$
metric.

Furthermore, we conduct statistical significance analysis using the Wilcoxon
signed-rank test, as shown in Table 7. In hypothesis testing, the null
hypothesis states that the performance of the ChatGPT-4 variants is not
inferior to the original ChatGPT-4. Each value in the table represents the
P-value when comparing the method on the y-axis against the method on the
x-axis for the corresponding metric. If the obtained P-value falls below the
typical alpha significance level of 0.05, the null hypothesis is consequently
rejected, indicating a significant performance difference.

Table 7. RQ2: P-values of Ablation Study Method | Origin | Log
---|---|---
$TOP$-$1$ |  $TOP$-$2$ |  $TOP$-$3$ |  $TOP$-$4$ |  $TOP$-$5$ |  $TOP$-$1$ |  $TOP$-$2$ |  $TOP$-$3$ |  $TOP$-$4$ |  $TOP$-$5$
ChatGPT-4NoOrder | 0.0068 | 0.75 | 0.72 | 0.58 | 0.5 | 4.8e-06 | 9.9e-05 | 0.043 | 0.19 | 0.15
ChatGPT-4NoExplain | 0.039 | 0.021 | 0.00032 | 4.1e-05 | 3.5e-05 | 0.084 | 0.0093 | 0.0038 | 0.0011 | 8.3e-05
ChatGPT-4NoError | - | - | - | - | - | 9.3e-10 | 5.3e-06 | 1.3e-05 | 3.7e-06 | 6e-06
ChatGPT-4NoTestCase | - | - | - | - | - | 0.15 | 0.61 | 0.8 | 0.77 | 0.73

The P-values shown in Table 7 indicate that ChatGPT-4 (Log)NoOrder, ChatGPT-4
(Origin)NoExplain and ChatGPT-4 (Log)NoError perform significantly differently
from standard ChatGPT-4 (Origin) or ChatGPT-4 (Log) in some cases. These
results suggest that asking ChatGPT to rank statements, explain their
reasoning, or provide ChatGPT with access to error log can improve ChatGPT-4’s
fault localisation capabilities in specific cases.  Answer to RQ2: Each
component within the prompts we design makes a positive contribution to the
effectiveness of ChatGPT-4. The most pronounced impact arises from the error
log messages, excluding which causes the effectiveness to decline by 25.6%
under the $TOP$-$1$ metric.

###  3.3. RQ3: Impact of Context

In previous experiments, we consistently used the function in the Defects4J
dataset that contains the faulty statements as the context in prompts. This
design is based on the premise that the function scope offers an adequate
understanding of statement behaviour to characterise the faults. To further
evaluate ChatGPT’s fault localisation capability and reliability in practical
development scenarios, we examine whether the length of code context provided
in the prompt can influence its performance. Therefore, we conduct two
controlled experiments by manipulating the context range within the prompts.
Specifically, we carry out a “narrowing context experiment” and an “expanding
context experiment”. In addition, to ensure the fairness of the comparison, in
this RQ, the range of fault location on all baselines is consistent with the
adjusted context.

In the narrowing context experiment, we constrain the context to five lines of
code before and after the faulty statement within its encapsulating function.
It should be noted that the ideal maximum values for the evaluation metrics in
this study differ from those in RQ1. In RQ1, each prompt only contains one
function regardless of how many faulty statements are in the function.
Finally, we construct independent prompts for a total of 528 faulty
statements. The results of the experiment with narrowed context are shown in
Table 8.

Table 8. RQ3: Experimental Results on the Defects4J Dataset of Narrowed Context Program | $TOP$-$N$ | Method
---|---|---
SBFL | MBFL | SmartFL | ChatGPT-4
$Ochiai$ | $Dstar$ | $Ochiai$ | $Dstar$ | Origin | Log
Chart |  $TOP$-$1$ | 4 | 4 | 6 | 6 | 11 | 10.4 | 13.6
$TOP$-$2$ | 13 | 13 | 22 | 22 | 13 | 16.6 | 17.0
$TOP$-$3$ | 18 | 18 | 28 | 28 | 14 | 19.2 | 21.8
$TOP$-$4$ | 22 | 22 | 28 | 28 | 15 | 20.0 | 22.0
$TOP$-$5$ | 23 | 23 | 28 | 28 | 17 | 20.2 | 22.2
Lang |  $TOP$-$1$ | 31 | 31 | 18 | 14 | 50 | 45.0 | 65.8
$TOP$-$2$ | 50 | 53 | 52 | 50 | 63 | 66.0 | 80.0
$TOP$-$3$ | 69 | 70 | 62 | 60 | 69 | 76.4 | 86.6
$TOP$-$4$ | 86 | 83 | 70 | 68 | 78 | 80.6 | 88.6
$TOP$-$5$ | 92 | 90 | 78 | 76 | 89 | 82.4 | 89.4
Math |  $TOP$-$1$ | 59 | 56 | 50 | 50 | 96 | 114.0 | 131.0
$TOP$-$2$ | 115 | 106 | 88 | 88 | 127 | 153.6 | 164.8
$TOP$-$3$ | 146 | 138 | 106 | 106 | 147 | 173.4 | 180.2
$TOP$-$4$ | 171 | 162 | 114 | 114 | 163 | 178.8 | 185.8
$TOP$-$5$ | 186 | 181 | 130 | 130 | 177 | 181.8 | 189.0
Mockito |  $TOP$-$1$ | 19 | 19 | 16 | 16 | - | 24.0 | 31.2
$TOP$-$2$ | 35 | 34 | 50 | 48 | - | 31.8 | 36.4
$TOP$-$3$ | 36 | 35 | 52 | 52 | - | 35.2 | 37.8
$TOP$-$4$ | 37 | 36 | 54 | 54 | - | 36.2 | 37.8
$TOP$-$5$ | 37 | 36 | 56 | 56 | - | 36.4 | 37.8
Time |  $TOP$-$1$ | 17 | 18 | 10 | 10 | 16 | 16.4 | 25.6
$TOP$-$2$ | 19 | 20 | 12 | 12 | 20 | 24.2 | 30.0
$TOP$-$3$ | 24 | 24 | 18 | 18 | 25 | 30.4 | 33.6
$TOP$-$4$ | 36 | 37 | 22 | 22 | 37 | 34.2 | 37.4
$TOP$-$5$ | 41 | 41 | 34 | 34 | 39 | 35.4 | 38.2
Closure |  $TOP$-$1$ | 65 | 62 | 2 | 2 | - | 58.4 | 72.2
$TOP$-$2$ | 79 | 74 | 12 | 12 | - | 81.8 | 90.6
$TOP$-$3$ | 88 | 82 | 20 | 20 | - | 89.4 | 98.8
$TOP$-$4$ | 98 | 93 | 22 | 22 | - | 95.6 | 102.0
$TOP$-$5$ | 104 | 99 | 28 | 28 | - | 97.6 | 103.4
Average |  $TOP$-$1$ | 32.50 | 31.67 | 17.00 | 16.33 | 43.25 | 44.70 | 56.57
$TOP$-$2$ | 51.83 | 50.00 | 39.33 | 38.67 | 55.75 | 62.33 | 69.80
$TOP$-$3$ | 63.50 | 61.17 | 47.67 | 47.33 | 63.75 | 70.67 | 76.47
$TOP$-$4$ | 75.00 | 72.17 | 51.67 | 51.33 | 73.25 | 74.23 | 78.93
$TOP$-$5$ | 80.50 | 78.33 | 59.00 | 58.67 | 80.50 | 75.63 | 80.00

From Table 8, we find ChatGPT-4 (Log) generally maintains advantages.
Specifically, on average, ChatGPT-4 (Log) achieves the best results under
$TOP$-$1$ to $TOP$-$4$ metrics. However, with the reduction of code context,
the performance of other baselines has also improved, making ChatGPT-4 (Log)
less leading, and even surpassed by SBFL and SmartFL in terms of $TOP$-$5$
metric.

Table 9. RQ3: Experimental Results on the Defects4J Dataset of Expanding Context Program | $TOP$-$N$ | Method
---|---|---
SBFL | MBFL | SmartFL | ChatGPT-4
$Ochiai$ | $Dstar$ | $Ochiai$ | $Dstar$ | Origin | Log
Chart |  $TOP$-$1$ | 0 | 0 | 0 | 0 | 3 | 1.0 | 1.8
$TOP$-$2$ | 1 | 0 | 1 | 1 | 4 | 1.8 | 1.8
$TOP$-$3$ | 3 | 1 | 1 | 1 | 4 | 1.8 | 1.8
$TOP$-$4$ | 3 | 1 | 1 | 1 | 4 | 1.8 | 1.8
$TOP$-$5$ | 3 | 1 | 1 | 1 | 4 | 1.8 | 1.8
Lang |  $TOP$-$1$ | 0 | 0 | 0 | 0 | 3 | 1.0 | 2.6
$TOP$-$2$ | 0 | 0 | 3 | 3 | 3 | 1.4 | 3.4
$TOP$-$3$ | 0 | 0 | 4 | 4 | 4 | 1.8 | 3.6
$TOP$-$4$ | 1 | 1 | 4 | 4 | 4 | 2.0 | 3.6
$TOP$-$5$ | 1 | 1 | 4 | 4 | 4 | 2.0 | 3.6
Math |  $TOP$-$1$ | 5 | 4 | 6 | 6 | 5 | 0.8 | 2.4
$TOP$-$2$ | 7 | 6 | 9 | 9 | 9 | 2.2 | 5.4
$TOP$-$3$ | 9 | 6 | 15 | 15 | 11 | 4.0 | 6.2
$TOP$-$4$ | 10 | 6 | 16 | 16 | 13 | 5.4 | 6.4
$TOP$-$5$ | 10 | 6 | 17 | 17 | 14 | 5.6 | 6.8
Mockito |  $TOP$-$1$ | 1 | 1 | 0 | 0 | - | 0.2 | 2.4
$TOP$-$2$ | 2 | 1 | 1 | 0 | - | 2.0 | 4.6
$TOP$-$3$ | 2 | 1 | 1 | 1 | - | 2.6 | 5.0
$TOP$-$4$ | 2 | 1 | 1 | 1 | - | 2.6 | 5.0
$TOP$-$5$ | 2 | 1 | 2 | 2 | - | 2.6 | 5.0
Closure |  $TOP$-$1$ | 3 | 2 | 0 | 0 | - | 0.4 | 0.0
$TOP$-$2$ | 3 | 2 | 0 | 0 | - | 1.2 | 0.2
$TOP$-$3$ | 3 | 3 | 0 | 0 | - | 1.4 | 0.4
$TOP$-$4$ | 3 | 3 | 0 | 0 | - | 1.6 | 0.4
$TOP$-$5$ | 5 | 5 | 0 | 0 | - | 2.0 | 0.4
Average |  $TOP$-$1$ | 1.80 | 1.40 | 1.20 | 1.20 | 3.67 | 0.68 | 1.84
$TOP$-$2$ | 2.60 | 1.80 | 2.80 | 2.60 | 5.33 | 1.72 | 3.08
$TOP$-$3$ | 3.40 | 2.20 | 4.20 | 4.20 | 6.33 | 2.32 | 3.40
$TOP$-$4$ | 3.80 | 2.40 | 4.40 | 4.40 | 7.00 | 2.68 | 3.44
$TOP$-$5$ | 4.20 | 2.80 | 4.80 | 4.80 | 7.33 | 2.80 | 3.52

In the subsequent expanding context experiment, we expand the context to
encompass the class containing the faulty code. If a class includes multiple
faulty statements across different functions, the code for the whole class is
used as context within a single prompt. The experiment results with expanding
context are shown in Table 9. Due to ChatGPT’s token limitations, overly long
classes are excluded; ultimately, the experiment uses 56 classes from five
different programs.

Unexpectedly, as shown in Table 9, ChatGPT-4 no longer holds an advantage. It
exhibits inferior performance compared to baselines across most programs and
evaluation metrics under class-level context. Therefore, when expanding the
code context in prompts from function-level to class-level, the fault
localisation capability of ChatGPT-4 drastically deteriorates.  Answer to RQ3:
ChatGPT’s fault localisation capability is highly sensitive to the code
context scope in the prompts. With functional-level context, ChatGPT-4 (Log)
outperforms all the baseline methods, with 30.8% higher $TOP$-$1$ on average
than state-of-the-art; When expanding to class-level context, ChatGPT-4’s
localisation ability decreases significantly, leading to a 49.9% decline in
the $TOP$-$1$ metric with ChatGPT-4 (Log).

##  4\. Extended Analysis on the Threat of Overfitting

While Defects4J has been widely used in the fault localisation domain (Lou et
al., 2021; Zou et al., 2019; Zakari et al., 2020), there is a possibility that
it may have been incorporated into the training set of ChatGPT. This could
confer an unfair advantage to ChatGPT when assessing fault localisation
effectiveness based on the Defects4J dataset and cause overfitting.

To address this threat and validate the reliability of our previous
observations, we build a more recent dataset, which we refer to as the
Student-Generated Defects (StuDefects). StuDefects contains code generated
exclusively after 2022. This guarantees that it does not appear in the
training data of ChatGPT because ChatGPT’s training data only extends up to
2021. To facilitate reproducibility and future research, we have open-sourced
this dataset on GitHub (thi, 2023).

In particular, StuDefects comes from a university’s course submission platform
that tracks students’ code submissions during classroom programming tasks. The
platform offers task descriptions and uses pre-prepared test cases to compile
and run students’ code. It then compares the tests’ outputs to expected
results to evaluate code correctness. If any test fails, students are prompted
to revise and resubmit their code in an iterative process until they achieve
the expected outcome.

During the data collection process, we adhere to the subsequent criteria:

Time Constraint We include only the code submitted from 2022 onward to
guarantee that the code in this dataset is more current than the cut-off date
of ChatGPT’s training data.
Language Specificity We choose only code in Java language, in alignment with
Defects4J, to maintain the consistency of our study in terms of programming
languages.
Precise Fault Label We focus on instances where a student submits a faulty
code and then rectifies it with a single line modification before
resubmitting. This approach allows us to identify the actual location of the
fault (serving as our fault localisation label) and ensures that the faults
are real instead of artificially injected.
Reduced Redundancy For each programming task, we keep only one faulty code,
thus removing the redundancy of code in our dataset.

Following the criteria outlined above while traversing the entire database, we
successfully identify 77 unique programs from different programming tasks.
These tasks encompass a variety of algorithmic complexities, from simple tasks
such as basic mathematical operations and string manipulations, to more
challenging ones like dynamic programming and large number handling. We also
pair each faulty program with its corresponding rectified version. The length
of these programs ranges from 5 to 86 lines, with an average of 25.7 lines.

Table 10. Extended Analysis: Fault Localisation Results on the StuDefects in Terms of $TOP$-$N$ Metric  $TOP$-$N$ | Method
---|---
SBFL | MBFL |  ChatGPT-4 (Origin)
$Ochiai$ | $Dstar$ | $Ochiai$ | $Dstar$
$TOP$-$1$ | 3 | 3 | 17 | 17 | 26.0
$TOP$-$2$ | 11 | 11 | 25 | 25 | 36.2
$TOP$-$3$ | 15 | 15 | 33 | 34 | 41.4
$TOP$-$4$ | 18 | 18 | 42 | 42 | 44.2
$TOP$-$5$ | 24 | 24 | 48 | 48 | 46.0

Table 10 shows the experimental results based on the StuDefects dataset. Given
StuDefects’ smaller scale and being within ChatGPT’s token limit, the entire
code (all with functional level) is used as the context. We focus on ChatGPT-4
(Origin) in this extended analysis because the failed test case log
information of StuDefects is different from that of Defects4J.

As discerned from Table 10, ChatGPT-4 (Origin) demonstrates superiority in
most scenarios. Furthermore, ChatGPT-4 (Origin) surpasses the second-best MBFL
by 52.9% in $TOP$-$1$ evaluation metric. These experimental findings indicate
that ChatGPT exhibits consistently promising fault localisation performance on
both Defects4J and the dataset beyond its training data cut-off in 2021.

##  5\. Related Work

###  5.1. Fault Localisation

To assist developers for localising faults, various approaches have been
proposed. These approaches include SBFL (Abreu et al., 2007), MBFL (Moon et
al., 2014), and deep learning-based fault localisation (Li et al., 2019).

Among traditional fault localisation techniques, SBFL stands out for its
simplicity and low computational cost, attracting significant attention.
However, it faces challenges such as inaccuracy and the issue of assigning
identical suspiciousness scores to multiple statements (Feyzi and Parsa, 2018;
Steimann et al., 2013; Wong et al., 2016). Similarly, while MBFL is
acknowledged for its accuracy, it is hampered by high computational costs and
difficulties in modeling complex software faults. This limits its
applicability to large-scale programs and affects its effectiveness in
localising complex faults in industrial programs such as Defects4J (Li et al.,
2020; Steimann et al., 2013; Wong et al., 2016).

To address these limitations, recent studies have aimed to improve
localisation accuracy by incorporating additional semantic information. For
example, Zeng et al. (Zeng et al., 2022) proposed using probabilistic models
or machine learning to capture program semantics. They introduced SmartFL, a
probabilistic model-based technique that considers only value correctness, not
full semantics. Using static and dynamic analysis, they constructed a
probabilistic model and ranked elements by maximum posterior probability.
Their method achieved 21% $TOP$-$1$ statement-level accuracy on Defects4J,
outperforming other baselines.

With the extensive application of neural networks, the use of deep learning in
software testing has motivated many studies on fault localisation, such as
Grace (Lou et al., 2021), FLUCCS (Sohn and Yoo, 2017), and DeepFL (Li et al.,
2019).

Sohn et al. (Sohn and Yoo, 2017) proposed FLUCCS, which leverages code and
change metrics to improve the ranking of likely faulty program elements.
FLUCCS expands SBFL by incorporating additional source code features,
including size, age, and code churn. Through genetic programming and linear
rank SVM, FLUCCS integrates suspiciousness values from SBFL with code metrics
into a unified ranking function. Evaluation of 210 faults in Defects4J showed
FLUCCS significantly outperformed SBFL, ranking the faulty method first for
over half the faults. FLUCCS demonstrates code and change metrics can enhance
the accuracy and effort reduction of fault localisation.

Li et al. (Li et al., 2019) proposed DeepFL, integrating suspiciousness-based,
fault-proneness-based, and text-similarity-based features from different
domains. DeepFL significantly outperformed existing learning-based techniques
on Defects4J, and they studied the impact of model configurations and
features. DeepFL also evidenced high effectiveness for cross-project
prediction.

Subsequently, Lou et al. (Lou et al., 2021) proposed a graph-based
representation learning technique Grace to boost coverage-based fault
localisation. They constructed a graph linking test cases and program entities
based on coverage relationship and code structure. A gated graph neural
network learned valuable features from the graph representation. This
technique substantially exceeded existing fault localisation techniques on
Defects4J. Precisely, Grace could localise 195 bugs in terms of $TOP$-$1$
metric, whereas the best compared technique, DeepFL, could at most localise
166 bugs.

While deep learning-based fault localisation methods exhibit promising
development potential, including the aforementioned, they typically localise
only to the faulty function rather than the faulty statement. Therefore, we do
not select such techniques as baselines in this paper.

###  5.2. Large Language Model for Software Engineering

Given the breakthrough of large language models (LLMs), studies have sought to
explore how LLMs could assist developers in a variety of tasks, including:

Code generation: LLMs can generate standard-compliant code based on specific
requirements or conditions. For example, Dong et al. (Dong et al., 2023)
explored a self-collaboration framework employing LLMs, specifically ChatGPT,
for code generation. By simulating different roles such as analysts, coders,
and testers within a virtual team, their approach achieved state-of-the-art
performance and even surpassed GPT-4 in complex code generation tasks. Their
framework offers a new direction for leveraging the capabilities of LLMs to
handle real-world complex programming problems efficiently.

Program repair: LLMs can be used to generate prompts that guide developers to
improve their code quality, perform refactoring operations, or fix the errors
or bugs in their code. For example, Dominik et al. (Sobania et al., 2023)
evaluated ChatGPT’s bug-fixing capabilities on the QuixBugs benchmark,
comparing it to deep learning methods like CoCoNut and Codex. The study finds
that ChatGPT, enhanced by its dialogue system for additional context, fixes 31
out of 40 bugs, outperforming state-of-the-art approaches. Wei et al. (Wei et
al., 2023) introduced Rectify, a framework that synergistically combines Large
Language Models (LLMs) with completion engines for automated program repair.
Their study indicates that Rectify achieves state-of-the-art results on the
Defects4J dataset, producing more valid and correct patches within the same
generation budget. Their evaluation on a subset of the widely-used Defects4J
1.2 and 2.0 datasets shows that Rectify fixes 66 and 50 bugs, respectively,
surpassing the best-performing baseline by 14 and 16 bugs fixed. Xie et al.
(Xie et al., 2023b) evaluated ten Code Language Models (CLMs) on four
Automated Program Repair (APR) benchmarks. Their study shows that even without
fine-tuning, the best CLM fixes 72% more bugs than state-of-the-art deep
learning-based APR techniques. Fine-tuning further improves CLMs’ fixing
capabilities, enabling them to fix 31%-1,267% more bugs and outperform
existing techniques by 46%-164%.

Code summarisation: LLMs can produce summaries explaining the functionality
and intent of code segments. For example, Sun et al. (Sun et al., 2023) used
prompts to request ChatGPT to generate comments for all code snippets in the
CSN-Python test set. They compared ChatGPT’s automatic code summarisation
capabilities with three dedicated code summarisation models (specifically,
NCS, CodeBERT, and CodeT5) on the CSN-Python dataset. Their findings revealed
that on the whole, ChatGPT underperformed compared to specialised code
summarisation models on the CSN-Python dataset in terms of BLEU, METEOR, and
ROUGE-L scores.

Test generation: LLMs can aid in creating diverse test cases, ensuring the
robustness and correctness of software applications. For example, Liu et al.
(Liu et al., 2023) questioned the functional correctness of code generated by
LLMs, introducing a rigorous evaluation framework named EvalPlus. Their
extensive evaluation across 19 popular LLMs demonstrates that their framework
is capable of catching significant amounts of previously undetected wrong
code, shedding light on the need for more robust testing methods in code
synthesis.

There has been limited work on LLMs for fault localisation. Cao et al. (Cao et
al., 2023) included a small-scale test of ChatGPT’s fault localisation
abilities within their comprehensive assessment of the model’s capabilities.
Kang et al. (Kang et al., 2023) proposed AutoFL, an LLM-based fault
localisation method. AutoFL determines the fault location through a two-stage
process: first generating a root cause explanation, then predicting the
location based on available information. In a series of experiments, AutoFL
exhibited remarkable potential, outperforming traditional techniques on
Defects4J dataset.

Existing preliminary studies have demonstrated potential, yet several gaps
remain. Firstly, the advanced ChatGPT-4 has not been used, potentially
overlooking its capabilities in fault localisation. Secondly, a lack of
comparison with state-of-the-art fault localisation methods leaves the
efficacy of the proposed methods unverified. Lastly, the effect of different
code context lengths on fault localisation has not been studied. Addressing
these gaps is helpful for a comprehensive understanding of LLMs in fault
localisation.

##  6\. Discussion

###  6.1. Discussion on Threats to Validity

Firstly, ChatGPT’s responses contain inherent randomness, potentially
affecting our results’ credibility. To mitigate this threat, we perform each
experiment five times, striving to guarantee the consistency and robustness of
our outcomes. Therefore, this threat is limited.

Secondly, the Defects4J dataset, widely used in fault localisation, might be a
part of ChatGPT’s training data, posing a potential issue of data leakage.
However, our experimental findings show that ChatGPT’s performance falls
behind the baseline when presented with substantial code context. This
suggests that ChatGPT doesn’t simply memorise instances from Defects4J, which
alleviates some concerns about data leakage. In additional experiments with
the newly collected dataset, StuDefects, we observe that ChatGPT outperforms
the baseline in fault localisation, further reducing concerns about data
leakage.

###  6.2. Implications for Researchers and Developers

Based on the results and comparative analysis in this study, we summarise
several key implications for researchers and developers seeking to apply large
language models like ChatGPT-4 for fault localisation:

####  6.2.1. For Researchers:

  * •

Prompt and Context Optimization: Research into optimizing prompt design and
context length is crucial, as these elements significantly impact the efficacy
of LLMs in fault localisation. Experimentation with different prompt
structures and context lengths can lead to enhanced performance and usability.

  * •

Model Enhancement and Fine-Tuning: Despite their capabilities, current LLMs
like ChatGPT-4 show limitations in different contexts, necessitating further
research to improve their adaptability and accuracy in various settings,
especially when the code context expands.

####  6.2.2. For Developers:

  * •

Enhanced Debugging: The demonstrated superiority of ChatGPT-4 in fault
localisation within function-level context can assist developers in more
efficiently identifying and addressing software bugs, thereby reducing
debugging time.

  * •

Tool Selection: Developers need to be discerning in choosing fault
localisation methods and should consider the context in which they are
working, as LLMs like ChatGPT-4 may not be the best fit in all scenarios, such
as when dealing with class-level contexts.

  * •

Error Log Integration: Incorporating additional error logs can significantly
enhance localisation accuracy and consistency, aiding developers in more
effectively leveraging LLMs for fault localisation.

  * •

Awareness of Limitations: While LLMs offer considerable benefits in fault
localisation, developers should be cognizant of their limitations and employ
them judiciously, complementing them with other methods where necessary.

##  7\. Conclusion

In conclusion, this paper systematically explores the potential of applying
ChatGPT to fault localisation in large-scale open-source programs.

When compared to existing fault localisation methods, ChatGPT-4 can outperform
them in most situations when the code context of the prompts is within the
function-level. Furthermore, the error log generated during test execution
provides substantial assistance for ChatGPT’s fault localisation, enhancing
performance and consistency. Moreover, in overlap analysis conducted among all
baselines, ChatGPT-4 (Log) consistently exhibits superior performance, showing
a higher number of unique instances compared to missing instances.

However, ChatGPT’s fault localisation performance is highly sensitive to the
length of code context provided. When the context of the Defects4J dataset
expands to the class-level, its performance significantly deteriorates,
falling behind that of SmartFL.

Therefore, employing ChatGPT for fault localisation persists as an area
requiring further improvements in research, requiring further refinement and
investigation to truly leverage its potential advantages in practical
scenarios.

## References

  * (1)
  * Def (2014) 2014\.  Defects4J.  https://github.com/rjust/defects4j
  * thi (2023) 2023\.  FLofChatGPT.  https://github.com/TempUpload/FLofChatGPT
  * Abreu et al. (2006) R. Abreu, P. Zoeteweij, and A. J. c. Van Gemund. 2006\.  An Evaluation of Similarity Coefficients for Software Fault Localization. In _2006 12th Pacific Rim International Symposium on Dependable Computing (PRDC’06)_. IEEE, 39–46.
  * Abreu et al. (2007) Rui Abreu, Peter Zoeteweij, and Arjan JC Van Gemund. 2007\.  On the accuracy of spectrum-based fault localization. In _Testing: Academic and Industrial Conference Practice and Research Techniques-MUTATION (TAICPART-MUTATION 2007)_. IEEE, 89–98.
  * Assiri and Bieman (2017) Fatmah Yousef Assiri and James M Bieman. 2017.  Fault localization for automated program repair: effectiveness, performance, repair correctness.  _Software Quality Journal_ 25 (2017), 171–199.
  * Cao et al. (2023) Jialun Cao, Meiziniu Li, Ming Wen, and Shing-chi Cheung. 2023\.  A study on prompt design, advantages and limitations of chatgpt for deep learning program repair.  _arXiv preprint arXiv:2304.08191_ (2023).
  * Chakraborty et al. (2022) Saikat Chakraborty, Toufique Ahmed, Yangruibo Ding, Premkumar T Devanbu, and Baishakhi Ray. 2022.  Natgen: generative pre-training by “naturalizing” source code. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 18–30.
  * Dong et al. (2023) Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2023\.  Self-collaboration Code Generation via ChatGPT.  _arXiv preprint arXiv:2304.07590_ (2023).
  * Feng et al. (2023) Yunhe Feng, Sreecharan Vanam, Manasa Cherukupally, Weijian Zheng, Meikang Qiu, and Haihua Chen. 2023\.  Investigating Code Generation Performance of Chat-GPT with Crowdsourcing Social Data. In _Proceedings of the 47th IEEE Computer Software and Applications Conference_. 1–10.
  * Feyzi and Parsa (2018) Farid Feyzi and Saeed Parsa. 2018.  A program slicing-based method for effective detection of coincidentally correct test cases.  _Computing_ 100, 9 (2018), 927–969.
  * Fried et al. (2022) Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022\.  Incoder: A generative model for code infilling and synthesis.  _arXiv preprint arXiv:2204.05999_ (2022).
  * Gao et al. (2023) Haoyu Gao, Ting-En Lin, Hangyu Li, Min Yang, Yuchuan Wu, Wentao Ma, and Yongbin Li. 2023.  Self-Explanation Prompting Improves Dialogue Understanding in Large Language Models.  _arXiv preprint arXiv:2309.12940_ (2023).
  * Gao and Wong (2018) Ruizhi Gao and W Eric Wong. 2018.  MSeer: an advanced technique for locating multiple bugs in parallel. In _Proceedings of the 40th International Conference on Software Engineering_. 1064–1064.
  * Hu et al. (2020) Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2020.  Deep code comment generation with hybrid lexical and syntactical information.  _Empirical Software Engineering_ 25 (2020), 2179–2217.
  * Jaccard (1901) Paul Jaccard. 1901\.  Etude de la distribution florale dans une portion des Alpes et du Jura.  _Bulletin de la Societe Vaudoise des Sciences Naturelles_ 37 (01 1901), 547–579.
  * Jia et al. (2020) Minghua Jia, Zhanqi Cui, Yiwen Wu, Ruilin Xie, and Xiulei Liu. 2020.  SMFL integrating spectrum and mutation for fault localization. In _2019 6th International Conference on Dependable Systems and Their Applications (DSA)_. IEEE, 511–512.
  * Jones et al. (2002) J. A. Jones, M. J. Harrold, and J. Stasko. 2002\.  Visualization of test information to assist fault localization. In _Proceedings of the 24th International Conference on Software Engineering. ICSE 2002_. ACM, 467–477.
  * Just et al. (2014) René Just, Darioush Jalali, and Michael D Ernst. 2014\.  Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In _Proceedings of the 2014 International Symposium on Software Testing and Analysis_. 437–440.
  * Kang et al. (2023) Sungmin Kang, Gabin An, and Shin Yoo. 2023.  A Preliminary Evaluation of LLM-Based Fault Localization.  _arXiv preprint arXiv:2308.05487_ (2023).
  * Khan et al. (2021) Tanvir Ahmed Khan, Allison Sullivan, and Kaiyuan Wang. 2021\.  AlloyFL: a fault localization framework for Alloy. In _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 1535–1539.
  * Kim et al. (2021) Jinhan Kim, Gabin An, Robert Feldt, and Shin Yoo. 2021\.  Ahead of time mutation based fault localisation using statistical inference. In _2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE)_. IEEE, 253–263.
  * Kim et al. (2019) Jeongho Kim, Jindae Kim, and Eunseok Lee. 2019.  VFL: Variable-based fault localization.  _Information and software technology_ 107 (2019), 179–191.
  * Kochhar et al. (2016) Pavneet Singh Kochhar, Xin Xia, David Lo, and Shanping Li. 2016\.  Practitioners’ expectations on automated fault localization. In _Proceedings of the 25th International Symposium on Software Testing and Analysis_. 165–176.
  * Li et al. (2019) Xia Li, Wei Li, Yuqun Zhang, and Lingming Zhang. 2019\.  DeepFL: integrating multiple fault diagnosis dimensions for deep fault localization. In _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2019, Beijing, China, July 15-19, 2019_ , Dongmei Zhang and Anders Møller (Eds.). ACM, 169–180.
  * Li and Zhang (2017) Xia Li and Lingming Zhang. 2017.  Transforming programs and tests in tandem for fault localization.  _Proceedings of the ACM on Programming Languages_ 1, OOPSLA (2017), 1–30.
  * Li et al. (2022) Yi Li, Shaohua Wang, and Tien N Nguyen. 2022.  Fault localization to detect co-change fixing locations. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 659–671.
  * Li et al. (2020) Zheng Li, Haifeng Wang, and Yong Liu. 2020.  Hmer: A hybrid mutation execution reduction approach for mutation-based fault localization.  _Journal of Systems and Software_ 168 (2020), 110661.
  * Liu et al. (2023) Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023\.  Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.  _arXiv preprint arXiv:2305.01210_ (2023).
  * Liu et al. (2018) Yong Liu, Zheng Li, Ruilian Zhao, and Pei Gong. 2018\.  An optimal mutation execution strategy for cost reduction of mutation-based fault localization.  _Information Sciences_ 422 (2018), 572–596.
  * Lou et al. (2021) Yiling Lou, Qihao Zhu, Jinhao Dong, Xia Li, Zeyu Sun, Dan Hao, Lu Zhang, and Lingming Zhang. 2021\.  Boosting coverage-based fault localization via graph-based representation learning. In _ESEC/FSE ’21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece, August 23-28, 2021_ , Diomidis Spinellis, Georgios Gousios, Marsha Chechik, and Massimiliano Di Penta (Eds.). ACM, 664–676.
  * Moon et al. (2014) Seokhyeon Moon, Yunho Kim, Moonzoo Kim, and Shin Yoo. 2014\.  Ask the mutants: Mutating faulty programs for fault localization. In _2014 IEEE Seventh International Conference on Software Testing, Verification and Validation_. IEEE, 153–162.
  * Naish et al. (2011) Lee Naish, Hua Jie Lee, and Kotagiri Ramamohanarao. 2011\.  A model for spectra-based software diagnosis.  _ACM Transactions on software engineering and methodology (TOSEM)_ 20, 3 (2011), 1–32.
  * Ni et al. (2022) Chao Ni, Wei Wang, Kaiwen Yang, Xin Xia, Kui Liu, and David Lo. 2022\.  The best of both worlds: integrating semantic features with expert features for defect prediction and localization. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 672–683.
  * Ott and Longnecker (2015) R Lyman Ott and Micheal T Longnecker. 2015.  _An introduction to statistical methods and data analysis_.  Cengage Learning.
  * Papadakis and Le Traon (2015) Mike Papadakis and Yves Le Traon. 2015.  Metallaxis-FL: mutation-based fault localization.  _Software Testing, Verification and Reliability_ 25, 5-7 (2015), 605–628.
  * Pearson et al. (2017) Spencer Pearson, José Campos, René Just, Gordon Fraser, Rui Abreu, Michael D Ernst, Deric Pang, and Benjamin Keller. 2017\.  Evaluating and improving fault localization. In _2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE)_. IEEE, 609–620.
  * Prenner et al. (2022) Julian Aron Prenner, Hlib Babii, and Romain Robbes. 2022\.  Can OpenAI’s codex fix bugs? an evaluation on QuixBugs. In _Proceedings of the Third International Workshop on Automated Program Repair_. 69–75.
  * Ray (2023) Partha Pratim Ray. 2023\.  ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope.  _Internet of Things and Cyber-Physical Systems_ (2023).
  * Schäfer et al. (2023) Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023\.  Adaptive test generation using a large language model.  _arXiv preprint arXiv:2302.06527_ (2023).
  * Sobania et al. (2023) Dominik Sobania, Martin Briesch, Carol Hanna, and Justyna Petke. 2023.  An analysis of the automatic bug fixing performance of chatgpt.  _arXiv preprint arXiv:2301.08653_ (2023).
  * Sohn and Yoo (2017) Jeongju Sohn and Shin Yoo. 2017.  Fluccs: Using code and change metrics to improve fault localization. In _Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis_. 273–283.
  * Sridhara et al. (2023) Giriprasad Sridhara, Sourav Mazumdar, et al. 2023\.  ChatGPT: A Study on its Utility for Ubiquitous Software Engineering Tasks.  _arXiv preprint arXiv:2305.16837_ (2023).
  * Steimann et al. (2013) Friedrich Steimann, Marcus Frenkel, and Rui Abreu. 2013\.  Threats to the validity and value of empirical assessments of the accuracy of coverage-based fault locators. In _Proceedings of the 2013 International Symposium on Software Testing and Analysis_. IEEE, 314–324.
  * Sun et al. (2023) Weisong Sun, Chunrong Fang, Yudu You, Yun Miao, Yi Liu, Yuekang Li, Gelei Deng, Shenghan Huang, Yuchen Chen, Quanjun Zhang, et al. 2023\.  Automatic Code Summarization via ChatGPT: How Far Are We?  _arXiv preprint arXiv:2305.12865_ (2023).
  * Tian et al. (2023) Haoye Tian, Weiqi Lu, Tsz On Li, Xunzhu Tang, Shing-Chi Cheung, Jacques Klein, and Tegawendé F Bissyandé. 2023\.  Is ChatGPT the Ultimate Programming Assistant–How far is it?  _arXiv preprint arXiv:2304.11938_ (2023).
  * Tian et al. (2022) Zhao Tian, Junjie Chen, Qihao Zhu, Junjie Yang, and Lingming Zhang. 2022.  Learning to construct better mutation faults. In _Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering_. 1–13.
  * Tunstall et al. (2022) Lewis Tunstall, Leandro Von Werra, and Thomas Wolf. 2022\.  _Natural language processing with transformers_.  ” O’Reilly Media, Inc.”.
  * Wei et al. (2023) Yuxiang Wei, Lingming Zhang, and Chunqiu Steven Xia. 2023\.  Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repair. In _Proceedings of the 31th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_.
  * Weimer et al. (2009) Westley Weimer, ThanhVu Nguyen, Claire Le Goues, and Stephanie Forrest. 2009.  Automatically finding patches using genetic programming. In _2009 IEEE 31st International Conference on Software Engineering_. IEEE, 364–374.
  * Wong et al. (2013) W Eric Wong, Vidroha Debroy, Ruizhi Gao, and Yihao Li. 2013\.  The DStar method for effective software fault localization.  _IEEE Transactions on Reliability_ 63, 1 (2013), 290–308.
  * Wong et al. (2016) W. E. Wong, R. Gao, Y. Li, R. Abreu, and F. Wotawa. 2016.  A Survey on Software Fault Localization.  _IEEE Transactions on Software Engineering_ 42, 8 (2016), 707–740.
  * Wu et al. (2022) Yonghao Wu, Yong Liu, Weibo Wang, Zheng Li, Xiang Chen, and Paul Doyle. 2022\.  Theoretical Analysis and Empirical Study on the Impact of Coincidental Correct Test Cases in Multiple Fault Localization.  _IEEE Transactions on Reliability_ 71, 2 (2022), 830–849.
  * Wu et al. (2020) Yong-Hao Wu, Zheng Li, Yong Liu, and Xiang Chen. 2020\.  Fatoc: Bug isolation based multi-fault localization by using optics clustering.  _Journal of Computer Science and Technology_ 35 (2020), 979–998.
  * Xia et al. (2022) Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2022\.  Practical program repair in the era of large pre-trained language models.  _arXiv preprint arXiv:2210.14179_ (2022).
  * Xie et al. (2023b) Danning Xie, Byungwoo Yoo, Nan Jiang, Mijung Kim, Lin Tan, Xiangyu Zhang, and Judy S Lee. 2023b.  Impact of Large Language Models on Generating Software Specifications.  _arXiv preprint arXiv:2306.03324_ (2023).
  * Xie et al. (2023a) Zhuokui Xie, Yinghao Chen, Chen Zhi, Shuiguang Deng, and Jianwei Yin. 2023a.  ChatUniTest: a ChatGPT-based automated unit test generation tool.  _arXiv preprint arXiv:2305.04764_ (2023).
  * Zakari et al. (2020) Abubakar Zakari, Sai Peck Lee, Rui Abreu, Babiker Hussien Ahmed, and Rasheed Abubakar Rasheed. 2020\.  Multiple fault localization of software programs: A systematic literature review.  _Information and Software Technology_ 124 (2020), 106312.
  * Zeng et al. (2022) Muhan Zeng, Yiqian Wu, Zhentao Ye, Yingfei Xiong, Xin Zhang, and Lu Zhang. 2022\.  Fault localization via efficient probabilistic modeling of program semantics. In _Proceedings of the 44th International Conference on Software Engineering_. 958–969.
  * Zou et al. (2019) Daming Zou, Jingjing Liang, Yingfei Xiong, Michael D Ernst, and Lu Zhang. 2019.  An empirical study of fault localization families and their combinations.  _IEEE Transactions on Software Engineering_ 47, 2 (2019), 332–347.