# RepairAgent: An Autonomous, LLM-Based Agent for Program Repair

Islem Bouzenia
University of Stuttgart
Germany
<EMAIL_ADDRESS>
Premkumar Devanbu
UC Davis
USA
<EMAIL_ADDRESS>
Michael Pradel
University of Stuttgart
Germany
<EMAIL_ADDRESS>

###### Abstract

Automated program repair has emerged as a powerful technique to mitigate the
impact of software bugs on system reliability and user experience. This paper
introduces RepairAgent, the first work to address the program repair challenge
through an autonomous agent based on a large language model (LLM). Unlike
existing deep learning-based approaches, which prompt a model with a fixed
prompt or in a fixed feedback loop, our work treats the LLM as an agent
capable of autonomously planning and executing actions to fix bugs by invoking
suitable tools. RepairAgent freely interleaves gathering information about the
bug, gathering repair ingredients, and validating fixes, while deciding which
tools to invoke based on the gathered information and feedback from previous
fix attempts. Key contributions that enable RepairAgent include a set of tools
that are useful for program repair, a dynamically updated prompt format that
allows the LLM to interact with these tools, and a finite state machine that
guides the agent in invoking the tools. Our evaluation on the popular
Defects4J dataset demonstrates RepairAgent’s effectiveness in autonomously
repairing 164 bugs, including 39 bugs not fixed by prior techniques.
Interacting with the LLM imposes an average cost of 270,000 tokens per bug,
which, under the current pricing of OpenAI’s GPT-3.5 model, translates to 14
cents per bug. To the best of our knowledge, this work is the first to present
an autonomous, LLM-based agent for program repair, paving the way for future
agent-based techniques in software engineering.

##  I Introduction

Software bugs lead to system failures, security vulnerabilities, and
compromised user experience. Fixing bugs is a critical task in software
development, but if done manually, demands considerable time and effort.
Automated program repair (APR) promises to dramatically reduce this effort by
addressing the critical need for effective and efficient bug resolution in an
automated manner. Researchers and practitioners have explored various
approaches to address the challenge of automatically fixing bugs [1],
including techniques based on manually designed [2, 3] and
(semi-)automatically extracted [4, 5, 6] fix patterns, based on symbolic
constraints [7, 8, 9], and various machine learning-based approaches [10, 11,
12, 13, 14, 15, 16].

The current state-of-the-art in APR predominantly revolves around large
language models (LLMs). The first generation of LLM-based repair techniques
involve a one-time interaction with the model, where the model receives a
prompt containing the buggy code and produces a fixed version [17, 18]. The
second and current generation of LLM-based repair techniques introduces
iterative approaches, which query the LLM repeatedly based on feedback
obtained from previous fix attempts [19, 20, 21].

A key limitation of current iterative, LLM-based repair techniques is that
their hard-coded feedback loops do not allow the model to gather information
about the bug or existing code that may provide ingredients to fix the bug.
Instead, these approaches fix the context information provided in the prompt,
typically to the buggy code [19, 21], and sometimes also details about the
test cases that fail [20]. The feedback loop then executes the tests on
different variants of the buggy code and adds any compilation errors, test
failures, or other output, to the prompt of the next iteration. However, this
approach fundamentally differs from the way human developers fix bugs, which
typically involves a temporal interleaving of gathering information to
understand the bug, searching code that may be helpful for fixing the bug, and
experimenting with candidate fixes [22, 23].

This paper presents RepairAgent, the first autonomous, LLM-based agent for
automated program repair. Our approach treats the LLM as an autonomous agent
capable of planning and executing actions to achieve the goal of fixing a bug.
To this end, we equip the LLM with a set of bug repair-specific tools that the
models can invoke to interact with the code base in a way similar to a human
developer. For example, RepairAgent has tools to extract information about the
bug by reading specific lines of code, to gather repair ingredients by
searching the code base, and to propose and validate fixes by applying a patch
and executing test cases. Importantly, we do not hard-code how and when to use
these tools, but instead let the LLM autonomously decide which tool to invoke
next, based on previously gathered information and feedback from previous fix
attempts.

Our approach is enabled by three key components. First, a general-purpose LLM,
such as GPT-3.5, which we query repeatedly with a dynamically updated prompt.
We contribute a novel prompt format that guides the LLM through the bug repair
process, and that gets updated based on the commands invoked by the LLM and
the results of the previous command executions. Second, a set of tools that
the LLM can invoke to interact with the code base. We present a set of 14
tools designed to cover different steps a human developer would take when
fixing a bug, such as reading specific lines of code, searching the code base,
and applying a patch. Third, a middleware that orchestrates the communication
between the LLM and the tools. We present novel techniques for guiding tool
invocations through a finite state machine and for heuristically interpreting
possibly incorrect LLM outputs. The iterative loop of RepairAgent continues
until the agent declares to have found a suitable fix, or until exhausting a
budget of iterations.

To evaluate the effectiveness of our approach, we apply it to all 835 bugs in
the Defects4J [24] dataset, a widely used benchmark for evaluating program
repair techniques. RepairAgent successfully fixes 164 bugs, including 74 and
90 bugs of Defects4J v1.2 and v2.0, respectively. The correctly fixed bugs
include 49 bugs that require fixing more than one line, showing that
RepairAgent is capable of fixing complex bugs. Compared to state-of-the-art
techniques [19, 21], RepairAgent successfully fixes 39 bugs not fixed by prior
work. Measuring the costs imposed by interacting with the LLM, we find that
RepairAgent imposes an average cost of 270,000 tokens per bug, which, under
the current pricing of OpenAI’s GPT-3.5 model, translates to 14 cents per bug.
Overall, our results show that our agent-based approach establishes a new
state of the art in program repair.

In summary, this paper contributes the following:

  * •

An autonomous, LLM-based agent for program repair.

  * •

A dynamically updated prompt format that guides the LLM through the bug repair
process.

  * •

A set of tools that enable a LLM to to perform steps a human developer would
take when fixing a bug.

  * •

A middleware that orchestrates the communication between the LLM and the
tools.

  * •

Empirical evidence that RepairAgent establishes a new state of the art by
successfully fixing 164 bugs, including 39 bugs not fixed by prior work.

  * •

We will release the implementation of RepairAgent as open-source to foster
future work.

To the best of our knowledge, there currently is no published work on an
autonomous, LLM-based agent for any code-generation task. We envision
RepairAgent to pave the way for future agent-based techniques in software
engineering.

##  II Background on LLM-Based, Autonomous Agents

By virtue of being trained on vast amounts of web knowledge, including natural
language and source code, LLMs have demonstrated remarkable abilities in
achieving human-level performance for various tasks [25]. A promising way of
using these abilities are LLM-based agents that autonomously plan and execute
actions to achieve a goal. The basic idea is to query the LLM with a prompt
that contains the current state of the world, the goal to be achieved, and a
set of actions that could be performed next. The model than decides which
action to perform, and the feedback from performing the action is integrated
into the next prompt. One way to represent “actions” is through tools that the
model can invoke to interact with the world [26, 27]. Recent surveys provide a
comprehensive overview of LLM-based, autonomous agents [28] and of LLM agents
equipped with tools invoked via APIs [29]. The potential of such agents for
software engineering currently is not well explored, which this paper aims to
address for the challenging task of automated program repair.

##  III Approach

###  III-A Overview

Figure 1: Overview of RepairAgent.

Figure 1 gives an overview of the RepairAgent approach, which consists of
three components: an LLM agent, a set of tools, and a middleware that
orchestrates the communication between the two. Given a bug to fix, the
middleware initializes the LLM agent with a prompt that contains task
information and instructions on how to perform it by using the provided tools.
The LLM responds by suggesting a call to one of the available tools, which the
middleware parses and then executes. The output of the tool is then integrated
into the prompt for the next invocation of the LLM, and the process continues
iteratively until the bug is fixed or a predefined budget is exhausted.

###  III-B Terminology

RepairAgent proceeds in multiple iterations, which we call cycles:

######  Definition 1 (Cycle).

A _cycle_ represents one round of interaction with the LLM agent, which
consists of the following steps:

  1. 1.

Query the agent

  2. 2.

Post-process the response

  3. 3.

Execute the command suggested by the agent

  4. 4.

Update the dynamic prompt based on the command’s output

In each cycle, the approach queries the LLM once. The input to the model is
updated based on commands (calls to tools) invoked by the LLM, and their
results, in previous cycles. We call the model input a dynamic prompt:

######  Definition 2 (Dynamic prompt).

The _dynamic prompt_ is a sequence of text sections
$P=[s_{0},s_{1},...,s_{n}]$, where each section $s_{i}$ is one of the
following (where $s_{i}(c)$ refers to a section during a cycle $c$):

  * •

A _static section_ , which remains the same across all cycles, i.e.,
$s_{i}(c)=s_{i}(c^{\prime})$ for all $c,c^{\prime}$.

  * •

A _dynamic section_ , which may differ across cycles, i.e., there may exist
$c,c^{\prime}$ with $s_{i}(c)\neq s_{i}(c^{\prime})$.

###  III-C Dynamic Prompting of the Repair Agent

TABLE I: Sections of the dynamically updated prompt. Prompt section |  Nature
---|---
Role |  Static
Goals |  Static
Guidelines |  Static
State description |  Dynamic
Available tools |  Dynamic
Gathered information |  Dynamic
Specification of output format |  Static
Last executed command and result |  Dynamic

The repair agent is an LLM trained on natural language and source code, such
as GPT-3.5. RepairAgent queries the LLM with a dynamic prompt that consists of
a sequence of static and dynamic sections, as listed in Table I. We describe
each section in detail in the following.

####  III-C1 Role

This section of the prompt defines the agent’s area of expertise, which is to
resolve bugs in Java code, and outlines the agent’s primary objective:
understanding and fixing bugs. The prompt emphasizes that the agent’s
decision-making process is autonomous and should not rely on user assistance.

####  III-C2 Goals

We define five goals for the agent to pursue, which remain the same across all
cycles:

  * •

_Locate the bug:_ Execute tests and use fault localization techniques to
pinpoint the bug’s location. Skip this goal when fault localization
information is already provided in the prompt.

  * •

_Gather information about the bug:_ Analyze the lines of code associated with
the bug to understand the bug.

  * •

_Suggest simple fixes to the bug:_ Start by suggesting simple fixes.

  * •

_Suggest complex fixes:_ If simple fixes prove ineffective, explore and
propose more complex ones.

  * •

_Iterate over the previous goals:_ Continue to gather information and to
suggest fixes until finding a fix.

####  III-C3 Guidelines

We provide a set of guidelines. First, we inform the model that there are
diverse kinds of bugs, ranging from single-line issues to multi-line bugs that
may entail changing, removing, or adding lines. Based on the observation that
many bugs can be fixed by relatively simple, recurring fix patterns [30], we
provide a list of recurring fix patterns. The list is based on the patterns
described in prior work on single-statement bugs in Java [30]. For each
pattern, we provide a short natural language description and an example of
buggy and fixed code. Second, we instruct the model to insert comments above
the modified code, which serves two purposes. On the one hand, the comments
allow the model to explain its reasoning, which has been shown to enhance the
reasoning abilities of LLMs [31]. On the other hand, commenting will
ultimately help human developers in understanding the nature of the edits.
Third, we instruct the model to conclude its reasoning with a clearly defined
next step that can be translated into a call to a tool. Finally, we describe
that there is a limited budget of tool invocations, highlighting the
importance of efficiency in selecting the next steps. Specifically, we specify
a maximum number of cycles (40 by default).

####  III-C4 State Description

Figure 2: State machine to guide selection of tools.

To guide the LLM agent toward using the available tools in an effective and
meaningful way, we define a finite state machine that constrains which tools
are available at a given point in time. The motivation is that we observed the
LLM agent to frequently get lost in aimless exploration in earlier experiments
without such guidance. Figure 2 shows the finite state machine, which we
design to mimic the states a human developer would go through when fixing a
bug. Each state is associated with a set of tools available to the agent,
which are described in Section III-D. Importantly, the agent is free to
transition between states at any point in time by using tools. That is,
despite providing guidance, the state machine does not enforce a strict order
of tool invocations.

The state description section of the prompt informs the agent about its
current state:

  * •

_Understand the bug_ : The agent starts in this state, where it can collect
information related to the failing test cases and the bug’s location. Once the
agent has an understanding of the bug, it formulates a hypothesis to describe
the nature of the bug and the reason behind it. Throughout the repair process,
the agent may refute earlier hypotheses and express new ones. After expressing
a hypothesis, the agent will automatically switch to the next state.

  * •

_Collect information to fix the bug_ : In this state the agent collects
information that help suggest a fix for the bug expressed by the hypothesis,
e.g., by searching for specific repair ingredients or by reading possibly
relevant code. Once the agent has gathered enough information to attempt a
fix, it can transition to the next state.

  * •

_Try to fix the bug_ : In this state, the agent tries to fix the bug based on
its current hypothesis and the collected information. Each fix attempt
modifies the code base and is validated by executing the test cases. If
necessarily, the agent can go back to one of the previous states to establish
a new hypothesis or to gather additional information.

In addition to the three above states, RepairAgent has a final state, _“Done”_
, which the agent can reach by calling a specific command that indicates the
success of repair.

####  III-C5 Available Tools

This section of the prompt describes a set of tools that the agent can call at
the current state. Each tool has a name, a description, and a set of typed
arguments (Section III-D).

####  III-C6 Gathered Information

A key ability of the repair agent is to gather information about the bug and
the code base, which serves as the basis for deciding which commands to invoke
next. To make this information available to the agent, we maintain a prompt
section that lists the information gathered by the different tool invocations.
Intuitively, this section of the prompt serves as a memory for the agent,
allowing it to recall information from previous cycles. The gathered
information is structured into different subsections, where each subsection
contains the outputs produced by a specific tool.

####  III-C7 Specification of Output Format

⬇

interface Response {

thoughts: string;

command: {

name: string;

args: Record<string, any>;

};

}

Figure 3: JSON format of the response of the model.

Given the dynamic prompt, the LLM agent provides one response per cycle. To
enable the middleware to parse the response, we specify the expected output
format (Figure 3). The “thoughts” field provides a textual description of the
agent’s reasoning when deciding about the next command. Asking the agent to
express its thoughts increases the transparency and interpretability of the
approach, provides a way to debug potential issues in the agent’s decision-
making process, and helps improve the reasoning abilities of LLMs [31]. The
“command” field specifies the next command to be executed, consisting of the
name of the tool to invoke and the set of arguments.

⬇

{

"thoughts": "With the hypothesis in mind,

it’s time to gather more information to

formulate potential fixes for the bug. I

should focus on understanding the context

around the condition

’if (x < 0 && prev == ’-’)’

to come up with effective solutions.",

"command": {

"name": "search_code_base",

"args": {

"key_words":

[ "addNumber",

"CodeConsumer",

"if (x < 0 && prev == ’-’)" ]

}}}

Figure 4: Example of a response of the repair agent.

For example, Figure 4 shows a response of the LLM agent. The model expresses
the need to collect more information to understand the bug. It then suggests a
command that searches the code base with a list of keywords.

####  III-C8 Last Executed Command and Result

This section of the prompt contains the last command (tool name and arguments)
that was executed (if any) and the output it produced. The rationale is to
remind the agent of the last step it took, and to make it aware of any
problems that occurred during the execution of the command. Furthermore, we
remind the agent how many cycles have already been executed, and how many
cycles are left

###  III-D Tools for the Agent to Use

A key novelty in our approach is to let an LLM agent autonomously decide which
tools to invoke to fix a bug. The tools we provide to the agent (Table II) are
inspired by the tools that developers use in their IDEs.

TABLE II: Repair-related tools invoked by RepairAgent. Tool |  Description
---|---
Read and extract code:
_read_range_ |  Read a range of lines in a file.
_get_classes_and_methods_ |  Get the names of all classes and methods in a file.
_extract_method_ |  Given a method name, extract method implementations from a file.
_extract_tests_ |  Given the failure report from JUnit or ANT, extract the code of failing test cases.
Search and generate code:
_search_code_base_ |  Scans all Java files within a project for a list of keywords.
_find_similar_api _calls_ |  Given a code snippet that calls an API, search for similar API calls in the project.
_generate_method_body_ |  Ask an LLM (GPT3.5 by default) to generate the body of a method based on code proceeding the method.
Testing and patching:
_run_tests_ |  Run the test suite of a project.
_run_fault_localization_ |  Retrieve pre-existing localization information or run a fault localization tool.
_write_fix_ |  Apply a patch to the code base and execute the test suite of the project. Changes are reverted automatically if tests fail. Moves the agent into the ’Try to fix the bug’ state.
Control:
_express_hypothesis_ |  Express a hypothesis about the bug. Moves the agent into the ’Collect information to fix the bug’ state.
_collect_more_information_ |  Move the agent back to the ’Collect information to fix the bug’ state.
_discard_hypothesis_ |  Discard the current hypothesis about the bug and move back to the ’Understand the bug’ state.
_goal_accomplished_ |  Declare that the goal has been accomplished and exiting the repair process.

####  III-D1 Reading and Extracting Code

A prerequisite for fixing a bug is to read and understand relevant parts of
the code base. Instead of hard-coding the context provided to the LLM [19, 20,
21], we let the agent decide which parts of the code to read, based on four
tools. The _read_range_ tool allows the agent to extract a range of lines from
a specific file, which is useful to obtain a focused view of a particular
section of code. To obtain an overview of the code structure, the
_get_classes_and_methods_ tool retrieves all class and method names within a
given file. By invoking the _extract_method_ tool, the agent can retrieve the
implementation(s) of methods that match a given method name within a given
file. Finally, we offer the _extract_tests_ tool, which extracts the code of
test cases that resulted in failure. The tool is crucial to understand details
of failing tests, such as input values and the expected output.

####  III-D2 Search and generate code

Motivated by the fact that human developers commonly search for code [32], we
present tools that allow the agent to search for specific code snippets. These
tools are useful for the agent to better understand the context of a bug and
to gather repair ingredients, i.e., code fragments that could become part of a
fix. The _search_code_base_ tool enables the agent to locate instances of
particular keywords within the entire code base. For example, the agent can
use this tool to find occurrences of variables, methods, and classes. Given a
set of keywords, the tool performs an approximate matching against all source
code files in the project. Specifically, the tool splits each keyword into
subtokens based on camel case, underscores, and periods, and then searches for
each subtoken in the code. For example, searching for quickSortArray yields
matches for sortArray, quickSort, arrayQuickSort, and other related
variations. The output of the tool is a nested dictionary, organized by file
names, classes, and method names, that provides the keywords that match a
method’s content. Another search tool, _find_similar_api_calls_ , allows the
agent to identify and extract usages of a method, which is useful to fix
incorrect method calls. Without such a tool, LLMs tend to hallucinate method
calls that do not exist in the code base [33]. Given a code snippet that
contains a method call, the tool extracts the name of the called method, and
then searches for calls to methods with the same name. The agent can restrict
the search to a specific file or search the entire code base.

In addition to searching for existing code, RepairAgent offers a tool that
generates new code by invoking another LLM. The tool is inspired by the
success of LLM-based code completion tools, such as Copilot [34], which human
developers increasingly use when fixing bugs. Given the code preceding a
method and the signature of the method, the _generate_method_body_ tool asks
an LLM to generate the body of the method. The query to the code-generating
LLM is independent of the dynamic prompt used by the overall RepairAgent
approach, and may use a different model. In our evaluation, we use the same
LLM for both the repair agent and as the code-generating LLM of this tool. The
tool limits the given code context to 12k tokens and sets a limit of 4k tokens
for the generated code.

####  III-D3 Testing and Patching

The next category of tools is related to running tests and applying patches.
The _run_tests_ tool allows the agent to execute the test suite of the
project. It produces a report that indicates whether the tests passed or
failed. In case of test failures, the tool cleans the output of the test
runner, e.g., by removing entries of the stack trace that are outside of the
current project. The rationale is that LLMs have a limited prompt size and
that irrelevant information may confuse the model. The
_run_fault_localization_ tool retrieves fault localization information, which
is useful to understand which parts of the code are likely to contain the bug.
RepairAgent offers two variants of this tool: Either, it provides perfect
fault localization information, i.e., the file(s) and line(s) that need to be
edited to fix the bug, or it invokes an existing fault localization tool, such
as GZoltar [35], to calculate fault localization scores. As common in the
field of program repair, we assume perfect fault localization as the default.

⬇

[

{

"file_path": "jfree/data/time/Week.java",

"insertions": [

{

"line_number": 175,

"new_lines": [

"// ...new lines to insert...\n",

"// ...more new lines...\n"]

}

],

"deletions": [179, 183],

"modifications": [

{

"line_number": 179,

"modified_line": " if (dataset == null) {\n"

}

]

},

{

"file_path": "org/jfree/data/time/Day.java",

"insertions": [],

"deletions": [307],

"modifications": []

}

Figure 5: Example of patch given to the _write_fix_ tool.

Once the agent has gathered sufficient information to fix the bug, it can
apply a patch to the code base using the _write_fix_ tool. RepairAgent aims at
repairing arbitrarily complex bugs, including multi-line and even multi-file
bugs. The _write_fix_ tool expects a patch in a specific JSON format, which
indicates the insertions, deletions, and modifications to be made in each
file. Figure 5 shows an example of a patch in this format. Given a patch, the
tool applies the changes to the code base and runs the test suite. If the
tests fail, the _write_fix_ reverts the changes, giving the agent a clean code
base to try another fix. Motivated by the observation that some fix attempts
are almost correct, the _write_fix_ tool requests the LLM to sample multiple
variants of the suggested fix. By default, RepairAgent samples 30 variants at
max. Given the generated variants, the approach removes duplicates and launch
tests for every variant.

####  III-D4 Control

The final set of tools do not directly correspond to a tool a human developer
may use, but rather allow the agent to move between states (Figure 2). The
_express_hypothesis_ tool empowers the agent to articulate a hypothesis
regarding the nature of the bug and to transition to the ’Collect information
to fix the bug’ state. Inversely, the _discard_hypothesis_ tool allows the
agent to discard a hypothesis that is no longer viable, which leads back to
the ’Understand the bug’ state. Together, the two commands enforce a
structured approach to hypothesis formulation, aligning with work on
scientific debugging [36, 20]. In case the agent has tried multiple fixes
without success, the _collect_more_information_ tool allows the agent to
revert to the ’Collect information to fix the bug’ state. Finally, once the
agent has found at least one fix that passes all tests, it can invoke the
_goal_accomplished_ tool, which terminates RepairAgent.

###  III-E Middleware

The middleware component plays a crucial role in RepairAgent, orchestrating
the communication between the LLM agent and the tools. It performs the steps
in Definition 1 as described in the following.

####  III-E1 Parsing and Refining LLM Output

At the beginning of each cycle, the middleware queries the LLM with the
current prompt. Ideally, the response adheres perfectly to the expected format
(Figure 3). In practice, however, the LLM may produce responses that deviate
from the expected format, e.g., due to hallucinations or syntactic errors. For
example, the LLM may provide a “path” argument while the tool expects a
“file_path” argument.

RepairAgent tries to heuristically rectify such issues by mapping the output
to the expected format in three steps. First, it tries to map the tool
mentioned in the response to one of the available tools. Specifically, the
approach checks if the predicted tool name $n_{\mathit{predicted}}$ is a
substring of the name of any available tool $n_{\mathit{actual}}$, or vice
versa, and if yes, considers $n_{\mathit{actual}}$ to be the desired tool. In
case the above matching fails, the approach checks if the Levenshtein distance
between $n_{\mathit{predicted}}$ and any $n_{\mathit{actual}}$ is below a
threshold (0.1 by default). Second, the approach tries to map the argument
names provided in the response to the expected arguments of the tool,
following the same logic as above. Third, the approach handles invalid
argument values by heuristically mapping or replacing them, e.g., by replacing
a predicted file path with a valid one. If the heuristic mapping fails or
produces multiple possible tool invocations, the middleware informs the LLM
about the issue via the “Last executed command and result” prompt section and
enters a new cycle.

In addition to rectifying minor mistakes in the response, the middleware also
checks for repeated invocations of the same tool with the same arguments. If
the agent suggests the exact same command as in a previous cycle, which would
yield the same results, the middleware informs the agent about the repetition
and enters a new cycle.

####  III-E2 Calling the Tool

Once the middleware has received a valid command from the LLM, it calls the
corresponding tool. To prevent tool executions to interfere with the host
environment or RepairAgent itself, the middleware executes the command in an
isolated environment that contains a cloned version of the buggy repository.

####  III-E3 Updating the Prompt

Given the output of the tool, the middleware updates all dynamic sections of
the prompt for the next cycle. In particular, the middleware updates the state
description and the available tools, appends the tool’s output to the gathered
information, and replaces the section that shows the last executed command.

##  IV Implementation

We use Python 3.10 as our primary programming language. Docker is used to
containerize and isolate command executions for enhanced reliability and
reproducibility. RepairAgent is built on top of the AutoGPT framework and
GPT-3.5-0125 from OpenAI. To parse and interact with Java code, we use ANTLR.

##  V Evaluation

To evaluate our approach we aim to answer the following research questions:

  1. RQ1

How effective is RepairAgent at fixing real-world bugs?

  2. RQ2

What are the costs of the approach?

  3. RQ3

How does the LLM agent use the available tools?

###  V-A Experimental Setup

##### Dataset

We apply RepairAgent to bugs in the Defects4J dataset [24]. We use the entire
Defects4J dataset, which consists of 835 real-world bugs from 17 Java
projects, including 395 bugs from 6 projects in Defects4Jv1.2, as well as
another 440 bugs and 11 projects added in Defects4Jv2. Evaluating on the
entire dataset allows us to assess the generalization capabilities of
RepairAgent to different projects and bugs, without restricting the
evaluation, e.g., based on the number of lines, hunks, or files that need to
be fixed.

##### Baselines

We compare with three existing repair techniques: ChatRepair [19], ITER [21],
and SelfAPR [37]. ChatRepair and ITER are two very recent approaches and have
been shown to be the current state of the art. All three baseline approaches
follow an iterative approach that incorporates feedback from previous patch
attempts. Unlike RepairAgent, the baselines do not use an autonomous, LLM-
based agent. We compare against the baselines based on patches provided by the
authors of the respective approaches.

##### Metrics of success

Similar to past work, we report both the number of plausible and correct
patches. A fix is _plausible_ if it passes all test cases, but is not
necessarily correct. To determine whether a fix is correct, we automatically
check whether it syntactically matches the developer-created fix. If this is
not the case, we manually determine whether the RepairAgent-generated fix is
semantically consistent with the developer-created fix. If and only if either
of the two checks succeeds, we consider the fix to be _correct_.

###  V-B RQ1: Effectiveness

####  V-B1 Overall Results

TABLE III: Results on Defects4J. Project | Bugs | Plausible | Correct | ChatRepair | ITER | SelfAPR
---|---|---|---|---|---|---
Chart | 26 | 14 | 11 | 15 | 10 | 7
Cli | 39 | 9 | 8 | 5 | 6 | 8
Closure | 174 | 27 | 27 | 37 | 18 | 20
Codec | 18 | 10 | 9 | 8 | 3 | 8
Collections | 4 | 1 | 1 | 0 | 0 | 1
Compress | 47 | 10 | 10 | 2 | 4 | 7
Csv | 16 | 6 | 6 | 3 | 2 | 1
Gson | 18 | 3 | 3 | 3 | 0 | 1
JacksonCore | 26 | 5 | 5 | 3 | 3 | 3
Jacksondatabind | 112 | 18 | 11 | 9 | 0 | 8
JacksonXml | 6 | 1 | 1 | 1 | 0 | 1
Jsoup | 93 | 18 | 18 | 14 | 0 | 6
JxPath | 22 | 0 | 0 | 0 | 0 | 1
Lang | 63 | 17 | 17 | 21 | 0 | 10
Math | 106 | 29 | 29 | 32 | 0 | 22
Mockito | 38 | 6 | 6 | 6 | 0 | 3
Time | 26 | 3 | 2 | 3 | 2 | 3
Defects4Jv1.2 | 395 | 96 | 74 | 114 | 57 | 64
Defects4Jv2 | 440 | 90 | 90 | 48 | — | 46
Total | 835 | 186 | 164 | 162 | 57 | 110
TABLE IV: Distribution of fixes by location type Bug type | RepairAgent | ChatRepair | ITER | SelfAPR
---|---|---|---|---
Single-line | 115 | 133 | 36 | 83
Multi-line* | 46 | 29 | 14 | 24
Multi-file | 3 | 0 | 4 | 3

Table III summarizes the effectiveness of RepairAgent in fixing the 835 bugs
in Defects4J. The approach generates plausible fixes for 186 bugs. While not
necessarily correct, plausible fixes pass all test cases and may still provide
developers a hint about what should be changed. RepairAgent generates correct
fixes for 164 bugs, which are semantically consistent with the developer-
provided patches. Being able to fix bugs from different projects shows that
the approach can generalize to code bases of multiple domains. Furthermore,
RepairAgent creates fixes for bugs of different levels of complexity.
Specifically, as shown in Table IV, the approach fixes 115 single-line bugs,
46 multi-line (single-file) bugs, and 3 multi-file bugs.

####  V-B2 Comparison with Prior Work

Figure 6: Intersection of the set fixes with related work.

The right-hand side of Table III compares RepairAgent with the baseline
approaches ChatRepair, ITER, and SelfAPR. Previous to this work, ChatRepair
had established a new state of the art in APR by fixing 162 bugs in Defects4J.
RepairAgent achieves a comparable record by fixing a total of 164 bugs. Our
work particularly excels in Defects4Jv2, where RepairAgent fixes 90 bugs,
while ChatRepair only fixes 48 bugs. To further compare the sets of fixed
bugs, Figure 6 shows the overlaps between different approaches. As often
observed in the field of APR, different approaches complement each other to
some extent. In particular, RepairAgent fixes 39 bugs that were not fixed by
any of the three baselines. Comparing the complexity of the bug fixes, as
shown on the right-hand side of Table IV, RepairAgent is particularly more
effective, compared to other tools, for bugs that require more than a single-
line fix. We attribute this result to the RepairAgent’s ability to
autonomously retrieve suitable repair ingredients and the fact that the agent
can perform edits to an arbitrary number of lines and files.

####  V-B3 Examples

⬇

if (cfa != null) {

for (Node finallyNode : cfa.finallyMap.get(parent)) {

- cfa.createEdge(fromNode, Branch.UNCOND, finallyNode);

+ cfa.createEdge(fromNode, Branch.ON_EX, finallyNode);}}

Figure 7: Closure-14, bug fixed by RepairAgent.

⬇

Separator sep = (Separator) elementPairs.get(0);

+ if (sep.iAfterParser == null && sep.iAfterPrinter == null) {

PeriodFormatter f = toFormatter(elementPairs.subList(2, size), notPrinter,
notParser);

sep = sep.finish(f.getPrinter(), f.getParser());

return new PeriodFormatter(sep, sep);

+ }

Figure 8: Time-27, bug fixed by RepairAgent.

Figures 7 and 8 showcase interesting bugs fixed exclusively by RepairAgent. In
the example of Figure 7, the agent used the tool
$find\\_similar\\_api\\_calls$ to search for calls similar to
cfa.createEdge(fromNode, Branch.UNCOND, finallyNode); which returns a similar
call that is found in another file but passes Branch.ON_EX to the method call
instead of Branch.UNCOND. The latter was used as the repair ingredient by the
agent.

In the second example, RepairAgent benefeted from the tool
$generate\\_method\\_body$ to generate the missing if-statement which led to
suggesting a correct fix afterwards.

From one side, these examples illustrate the clever and proper usage of
available tools by the repair agent. From the other side, it shows how useful
these tools at finding repair ingredients that traditional approaches and
previous work failed to find.

###  V-C RQ2: Costs of the Approach

(a) Time.

(b) Tokens/money consumption.

Figure 9: Distribution of cost metrics per bug (time, number of token, and
monetary costs).

We measure three kinds of costs imposed by RepairAgent: (i) Time taken to fix
a bug. (ii) The number of tokens consumed by queries to the LLM, which is
relevant both for commercial models, such as the GPT-3.5 used here, and for
self-hosted models, where the number of tokens determines the computational
costs. (iii) The monetary costs associated with the token consumption, based
on OpenAI’s pricing as of March 2024.

Our findings are summarized in Figure 9. The median time taken to address a
bug is 920 seconds, with minimal variation between fixed (870 seconds) and
unfixed bugs. Surprisingly, fixed bugs do not consistently exhibit lower
repair times. This is due to RepairAgent’s autonomous nature, where the repair
process continues until the _goal_accomplished_ command is invoked or the
cycles budget is exhausted. The figure shows several outliers where bug fixing
attempt takes multiple hours. RepairAgent spends 99% of the total time in tool
executions, mostly running tests. This cost could be reduced in the future by
executing selected test cases only.

Analyzing the costs imposed by the LLM, we find a median consumption of
approximately 270,000 tokens, equating to around 14 cents (US dollars). The
number of tokens consumed by fixed bugs (21,000) is clearly lower than by
unfixed bugs (315,000). This difference is because the agent continues to
extract additional information for not yet fixed bugs, saturating the prompt
with operations, such as reading more lines of code or performing extensive
searches.

###  V-D RQ3: Usage of Tools by the Agent

To better understand the approach, we evaluate how the agent uses the
available tools.

####  V-D1 Adherence to Output Format

A notable challenge in designing an LLM-based agent is the inherent
informality and natural language noise present in the model’s output. We
categorize the output of the model as follows:

  * •

_Fully parsable output:_ Responses that adhere to the JSON format without
requiring further refinement (87.7%).

  * •

_Unparsable output:_ Responses that do not conform to the JSON format (2.3% of
the responses).

  * •

_Partially parsable output:_ Responses in JSON format but with missing or
misnamed fields requiring refinement by the middleware (9.9%).

A parsed output does not guarantee a correctly specified command, which the
middleware tries to rectify. At this phase, the output may fall into the
following categories:

  * •

_Correct command name:_ The name of the command is correct (97.9%).

  * •

_Non-existent command:_ The command could not be found or mapped to an
existing one (1.4%).

  * •

_Mapped command:_ The command does not exist but can be mapped to an existing
command by the middleware (0.7%).

  * •

_Correct arguments:_ The arguments are correct (90.1%).

  * •

_Unrefinable arguments list:_ The command exists or was mapped, but the list
of arguments is incomplete or has incorrect names (1.9%).

  * •

_Refinable arguments list:_ The middleware successfully maps the list of
arguments into a correct one (8.0%).

Overall, these results show that our heuristic refinement of LLM outputs
contributes to the effectiveness and robustness of RepairAgent.

####  V-D2 Frequency of Tools Invocation

Figure 10: Frequency of tool invocations (average per bug).

On average, RepairAgent makes 35 calls per bug, which also corresponds to the
number of cycles. Figure 10 shows the frequency of tool invocations, where we
distinguish between fixed (i.e., “correct”) and unfixed (i.e., “plausible”
only or completely unfixed) bugs. The LLM agent uses the full range of tools,
with the most frequently called tool being _write_fix_ (average of 6 calls for
fixed bugs and 17 calls for unfixed bugs. Around 7% of _write_fix_ invocations
in unfixed bugs produce plausible patches, compared to 44% in fixed bugs.

##  VI Threats to Validity and Limitations

While RepairAgent shows promising results when repairing Defects4J bugs, we
acknowledge several potential threats to validity and inherent limitations:
_(i) Data leakage:_ As we evaluate on GPT-3.5 and its training data is not
publicly known, the LLM may have seen parts of the Java projects during
training. While we acknowledge this risk, our approach does not solely depend
on knowing about a bug, but rather the ability to collect information to fix
the bug. We also note that the closest competitor, ChatRepair, also uses
GPT-3.5, and thus faces the same risk. _(ii) Missing test cases:_ Defects4J
has at least one failing test case for each bug, which may not be the case for
real-world usage scenarios. It will be interesting to evaluate RepairAgent on
bugs with no a-priori available error-revealing test cases in future work.
_(iii) Fault localization:_ Inaccurate or imprecise fault localization could
lead to suboptimal repair suggestions or incorrect diagnoses. _(iv) Non-
deterministic output of LLMs:_ The inherently non-deterministic nature of LLMs
may result in different outcomes between two consecutive runs of RepairAgent.
The large number of bugs we evaluate on mitigates this risk. Moreover, we make
logs of all interactions with the LLM available to ensure reproducibility.

##  VII Related Work

##### Automated program repair

Automated program repair [1] has received significant attention. Some
approaches address it as a search problem based on manually designed code
mutation rules and fix patterns [2, 38, 3]. Alternatively, transformation
rules can be derived (semi-)automatically from human-written patches [4, 5,
6]. Other approaches use symbolic constraints to derive fixes [7, 39, 8, 9],
integrate repair into a static analysis that identifies bugs [40, 41, 42], or
replace buggy code with similar code from the same project [43]. APR has been
successfully deployed in industrial contexts [5, 44]. Beyond functional bugs,
several techniques target other kinds of problems, such as syntax errors [45,
46, 47], performance bugs [48], vulnerabilities [49], type errors [50], common
issues in deep learning code [51], and build errors [52].

##### Learning-based program repair

While early work uses machine learning to rank and select candidate fixes
[10], more recent work uses machine learning to generate fixes. Approaches
include neural machine translation models that map buggy code into fixed code
[11, 12, 13, 14], models that predict tree transformations [15, 16], neural
architectures for specific kinds of bugs [53], and repair-specific training
regimes [54, 37]. We refer to a recent survey for a more comprehensive
discussion [55]. Unlike the above work, RepairAgent and the work discussed
below use a general-purpose LLM, instead of training a task-specific model.

##### LLM-based program repair

LLMs have motivated researchers to apply them to program repair, e.g., in
studies that explore prompts [18, 17] and in a technique that prompts the
model with error messages [56]. These approaches perform a one-time
interaction with the model, where the model receives a prompt with code and
produces a fix. The most recent repair techniques introduce iterative
approaches, which query the LLM repeatedly based on feedback obtained from
previous fix attempts [19, 20, 21]. RepairAgent also queries the model
multiple times, but fundamentally differs by pursuing an agent-based approach.
Section V empirically compares RepairAgent to the most closely related
iterative approaches [19, 21].

##### LLMs for code generation and code editing

Beyond program repair, LLMs have been applied to a variety of other code
generation and code editing tasks, including code completion [34, 57], fuzzing
[58], generating and improving unit tests [59, 60, 61, 62, 63, 64], multi-step
code editing [65]. Unlike our work, none of these approaches uses an agent-
based approach.

##### LLM-based agents

The idea to let LLM agents autonomously plan and perform complex tasks is
relatively new and has been applied to tasks outside of software engineering
[28]. To the best of our knowledge, our work is the first to apply an LLM-
based agent to program repair or any other code generation problem in software
engineering. RepairAgent is inspired by prior work [29] on augmenting LLMs
with tools invoked via APIs [26, 27] and with the ability to generate and
execute code [66]. Our key contribution in applying these ideas to a software
engineering task is to define tools that are useful for program repair and a
prompt format that allows the LLM to interact with these tools.

##  VIII Conclusion

This paper presents a pioneering technique for bug repair based on an
autonomous agent powered by Large Language Models (LLMs). Through extensive
experimentation, we validate the effectiveness and potential of our approach.
Further exploration and refinement of autonomous agent-based techniques will
help generalize to more difficult and diverse types of bugs if equipped with
the right tools.

## References

  * [1] C. Le Goues, M. Pradel, and A. Roychoudhury, “Automated program repair,” _Commun. ACM_ , vol. 62, no. 12, pp. 56–65, 2019. [Online]. Available: https://doi.org/10.1145/3318162
  * [2] C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer, “Genprog: A generic method for automatic software repair,” _IEEE Trans. Software Eng._ , vol. 38, no. 1, pp. 54–72, 2012.
  * [3] K. Liu, A. Koyuncu, D. Kim, and T. F. Bissyandé, “Tbar: revisiting template-based automated program repair,” in _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2019, Beijing, China, July 15-19, 2019_ , D. Zhang and A. Møller, Eds. ACM, 2019, pp. 31–42. [Online]. Available: https://doi.org/10.1145/3293882.3330577
  * [4] D. Kim, J. Nam, J. Song, and S. Kim, “Automatic patch generation learned from human-written patches.” in _International Conference on Software Engineering (ICSE)_ , 2013, pp. 802–811.
  * [5] J. Bader, A. Scott, M. Pradel, and S. Chandra, “Getafix: Learning to fix bugs automatically,” _Proc. ACM Program. Lang._ , vol. 3, no. OOPSLA, pp. 159:1–159:27, 2019. [Online]. Available: https://doi.org/10.1145/3360585
  * [6] R. Bavishi, H. Yoshida, and M. R. Prasad, “Phoenix: automated data-driven synthesis of repairs for static analysis violations,” in _ESEC/SIGSOFT FSE 2019, Tallinn, Estonia, August 26-30, 2019_ , M. Dumas, D. Pfahl, S. Apel, and A. Russo, Eds. ACM, 2019, pp. 613–624. [Online]. Available: https://doi.org/10.1145/3338906.3338952
  * [7] H. D. T. Nguyen, D. Qi, A. Roychoudhury, and S. Chandra, “Semfix: program repair via semantic analysis,” in _35th International Conference on Software Engineering, ICSE ’13, San Francisco, CA, USA, May 18-26, 2013_ , 2013, pp. 772–781.
  * [8] J. Xuan, M. Martinez, F. Demarco, M. Clement, S. L. Marcote, T. Durieux, D. Le Berre, and M. Monperrus, “Nopol: Automatic repair of conditional statement bugs in java programs,” _IEEE Transactions on Software Engineering_ , vol. 43, no. 1, pp. 34–55, 2016.
  * [9] S. Mechtaev, J. Yi, and A. Roychoudhury, “Angelix: Scalable multiline program patch synthesis via symbolic analysis,” in _Proceedings of the 38th international conference on software engineering_ , 2016, pp. 691–701.
  * [10] F. Long and M. Rinard, “Automatic patch generation by learning correct code,” in _Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL 2016, St. Petersburg, FL, USA, January 20 - 22, 2016_ , 2016, pp. 298–312.
  * [11] R. Gupta, S. Pal, A. Kanade, and S. K. Shevade, “Deepfix: Fixing common C language errors by deep learning,” in _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence_ , 2017, pp. 1345–1351. [Online]. Available: http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14603
  * [12] M. Tufano, J. Pantiuchina, C. Watson, G. Bavota, and D. Poshyvanyk, “On learning meaningful code changes via neural machine translation,” in _Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019_ , 2019, pp. 25–36. [Online]. Available: https://dl.acm.org/citation.cfm?id=3339509
  * [13] T. Lutellier, H. V. Pham, L. Pang, Y. Li, M. Wei, and L. Tan, “Coconut: combining context-aware neural translation models using ensemble for program repair,” in _ISSTA ’20: 29th ACM SIGSOFT Virtual Event, USA, July 18-22, 2020_ , S. Khurshid and C. S. Pasareanu, Eds. ACM, 2020, pp. 101–114. [Online]. Available: https://doi.org/10.1145/3395363.3397369
  * [14] Z. Chen, S. Kommrusch, M. Tufano, L. Pouchet, D. Poshyvanyk, and M. Monperrus, “SequenceR: Sequence-to-sequence learning for end-to-end program repair,” _IEEE Trans. Software Eng._ , vol. 47, no. 9, pp. 1943–1959, 2021. [Online]. Available: https://doi.org/10.1109/TSE.2019.2940179
  * [15] Y. Li, S. Wang, and T. N. Nguyen, “Dlfix: Context-based code transformation learning for automated program repair,” in _ICSE_ , 2020.
  * [16] Q. Zhu, Z. Sun, Y. Xiao, W. Zhang, K. Yuan, Y. Xiong, and L. Zhang, “A syntax-guided edit decoder for neural program repair,” in _ESEC/FSE ’21 Athens, Greece, August 23-28, 2021_ , D. Spinellis, G. Gousios, M. Chechik, and M. D. Penta, Eds. ACM, 2021, pp. 341–353. [Online]. Available: https://doi.org/10.1145/3468264.3468544
  * [17] C. S. Xia, Y. Wei, and L. Zhang, “Automated program repair in the era of large pre-trained language models,” in _2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)_ , 2023, pp. 1482–1494.
  * [18] N. Jiang, K. Liu, T. Lutellier, and L. Tan, “Impact of code language models on automated program repair,” in _45th IEEE/ACM International Conference on Software Engineering, ICSE_. IEEE, 2023, pp. 1430–1442. [Online]. Available: https://doi.org/10.1109/ICSE48619.2023.00125
  * [19] C. S. Xia and L. Zhang, “Keep the conversation going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT,” 2023.
  * [20] S. Kang, B. Chen, S. Yoo, and J. Lou, “Explainable automated debugging via large language model-driven scientific debugging,” _CoRR_ , vol. abs/2304.02195, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2304.02195
  * [21] H. Ye and M. Monperrus, “Iter: Iterative neural repair for multi-location patches,” in _ICSE_ , 2024.
  * [22] A. J. Ko, B. A. Myers, M. J. Coblenz, and H. H. Aung, “An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks,” _IEEE Transactions on software engineering_ , vol. 32, no. 12, pp. 971–987, 2006.
  * [23] M. Böhme, E. O. Soremekun, S. Chattopadhyay, E. Ugherughe, and A. Zeller, “Where is the bug and how is it fixed? an experiment with practitioners,” in _Proceedings of the 2017 11th joint meeting on foundations of software engineering_ , 2017, pp. 117–128.
  * [24] R. Just, D. Jalali, and M. D. Ernst, “Defects4j: a database of existing faults to enable controlled testing studies for java programs,” in _International Symposium on Software Testing and Analysis, ISSTA ’14, San Jose, CA, USA - July 21 - 26, 2014_ , 2014, pp. 437–440.
  * [25] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. M. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang, “Sparks of artificial general intelligence: Early experiments with GPT-4,” _CoRR_ , vol. abs/2303.12712, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2303.12712
  * [26] T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,” _CoRR_ , vol. abs/2302.04761, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2302.04761
  * [27] S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez, “Gorilla: Large language model connected with massive apis,” _CoRR_ , vol. abs/2305.15334, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2305.15334
  * [28] L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, W. X. Zhao, Z. Wei, and J.-R. Wen, “A survey on large language model based autonomous agents,” 2023.
  * [29] G. Mialon, R. Dessì, M. Lomeli, C. Nalmpantis, R. Pasunuru, R. Raileanu, B. Rozière, T. Schick, J. Dwivedi-Yu, A. Celikyilmaz, E. Grave, Y. LeCun, and T. Scialom, “Augmented language models: a survey,” _CoRR_ , vol. abs/2302.07842, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2302.07842
  * [30] R.-M. Karampatsis and C. Sutton, “How often do single-statement bugs occur?” Jun. 2020. [Online]. Available: http://dx.doi.org/10.1145/3379597.3387491
  * [31] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou _et al._ , “Chain-of-thought prompting elicits reasoning in large language models,” _Advances in neural information processing systems_ , vol. 35, pp. 24 824–24 837, 2022.
  * [32] L. D. Grazia and M. Pradel, “Code search: A survey of techniques for finding code,” _ACM Comput. Surv._ , vol. 55, no. 11, pp. 220:1–220:31, 2023. [Online]. Available: https://doi.org/10.1145/3565971
  * [33] A. Eghbali and M. Pradel, “De-hallucinator: Iterative grounding for llm-based code completion,” _CoRR_ , vol. abs/2401.01701, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2401.01701
  * [34] J. T. Mark Chen, “Evaluating large language models trained on code,” _CoRR_ , vol. abs/2107.03374, 2021. [Online]. Available: https://arxiv.org/abs/2107.03374
  * [35] J. Campos, A. Riboira, A. Perez, and R. Abreu, “Gzoltar: an eclipse plug-in for testing and debugging,” in _Proceedings of the 27th IEEE/ACM international conference on automated software engineering_ , 2012, pp. 378–381.
  * [36] A. Zeller, _Why programs fail: a guide to systematic debugging_. Elsevier, 2009.
  * [37] H. Ye, M. Martinez, X. Luo, T. Zhang, and M. Monperrus, “Selfapr: Self-supervised program repair with test execution diagnostics,” in _ASE 2022, Rochester, MI, USA, October 10-14, 2022_. ACM, 2022, pp. 92:1–92:13. [Online]. Available: https://doi.org/10.1145/3551349.3556926
  * [38] X. D. Le, D. Lo, and C. Le Goues, “History driven program repair,” in _IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering, SANER 2016, Suita, Osaka, Japan, March 14-18, 2016 - Volume 1_ , 2016, pp. 213–224. [Online]. Available: https://doi.org/10.1109/SANER.2016.76
  * [39] Y. Ke, K. T. Stolee, C. Le Goues, and Y. Brun, “Repairing programs with semantic code search (t),” in _2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 2015, pp. 295–306.
  * [40] R. van Tonder and C. L. Goues, “Static automated program repair for heap properties,” in _ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018_ , M. Chaudron, I. Crnkovic, M. Chechik, and M. Harman, Eds. ACM, 2018, pp. 151–162. [Online]. Available: https://doi.org/10.1145/3180155.3180250
  * [41] Y. Liu, S. Mechtaev, P. Subotić, and A. Roychoudhury, “Program repair guided by datalog-defined static analysis,” in _Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2023, pp. 1216–1228.
  * [42] N. Jain, S. Gandhi, A. Sonwane, A. Kanade, N. Natarajan, S. Parthasarathy, S. Rajamani, and R. Sharma, “Staticfixer: From static analysis to static repair,” 2023.
  * [43] D. Yang, X. Mao, L. Chen, X. Xu, Y. Lei, D. Lo, and J. He, “Transplantfix: Graph differencing-based code transplantation for automated program repair,” in _ASE 2022, Rochester, MI, USA, October 10-14, 2022_. ACM, 2022, pp. 107:1–107:13. [Online]. Available: https://doi.org/10.1145/3551349.3556893
  * [44] A. Marginean, J. Bader, S. Chandra, M. Harman, Y. Jia, K. Mao, A. Mols, and A. Scott, “Sapfix: Automated end-to-end repair at scale,” in _ICSE-SEIP_ , 2019.
  * [45] K. Wang, R. Singh, and Z. Su, “Search, align, and repair: data-driven feedback generation for introductory programming exercises,” in _Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2018, Philadelphia, PA, USA, June 18-22, 2018_ , 2018, pp. 481–495.
  * [46] R. Gupta, A. Kanade, and S. K. Shevade, “Deep reinforcement learning for syntactic error repair in student programs,” in _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019_. AAAI Press, 2019, pp. 930–937. [Online]. Available: https://doi.org/10.1609/aaai.v33i01.3301930
  * [47] G. Sakkas, M. Endres, P. J. Guo, W. Weimer, and R. Jhala, “Seq2parse: neurosymbolic parse error repair,” _Proc. ACM Program. Lang._ , vol. 6, no. OOPSLA2, pp. 1180–1206, 2022. [Online]. Available: https://doi.org/10.1145/3563330
  * [48] T. Yu and M. Pradel, “Pinpointing and repairing performance bottlenecks in concurrent programs,” _Empirical Software Engineering (EMSE)_ , pp. 1–38, 2017.
  * [49] J. Harer, O. Ozdemir, T. Lazovich, C. P. Reale, R. L. Russell, L. Y. Kim, and S. P. Chin, “Learning to repair software vulnerabilities with generative adversarial networks,” in _NeurIPS 2018, 3-8 December 2018, Montréal, Canada._ , 2018, pp. 7944–7954.
  * [50] Y. W. Chow, L. D. Grazia, and M. Pradel, “Pyty: Repairing static type errors in python,” in _International Conference on Software Engineering (ICSE)_ , 2024.
  * [51] X. Zhang, J. Zhai, S. Ma, and C. Shen, “AUTOTRAINER: an automatic DNN training problem detection and repair system,” in _43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021_. IEEE, 2021, pp. 359–371. [Online]. Available: https://doi.org/10.1109/ICSE43902.2021.00043
  * [52] D. Tarlow, S. Moitra, A. Rice, Z. Chen, P. Manzagol, C. Sutton, and E. Aftandilian, “Learning to fix build errors with graph2diff neural networks,” in _ICSE ’20 Workshops, Seoul, Republic of Korea, 27 June \- 19 July, 2020_. ACM, 2020, pp. 19–20. [Online]. Available: https://doi.org/10.1145/3387940.3392181
  * [53] M. Vasic, A. Kanade, P. Maniatis, D. Bieber, and R. Singh, “Neural program repair by jointly learning to localize and repair,” in _ICLR_ , 2019.
  * [54] H. Ye, M. Martinez, and M. Monperrus, “Neural program repair with execution-based backpropagation,” in _ICSE_ , 2022.
  * [55] Q. Zhang, C. Fang, Y. Ma, W. Sun, and Z. Chen, “A survey of learning-based automated program repair,” _ACM Transactions on Software Engineering and Methodology_ , vol. 33, no. 2, pp. 1–69, 2023.
  * [56] H. Joshi, J. P. C. Sánchez, S. Gulwani, V. Le, G. Verbruggen, and I. Radicek, “Repair is nearly generation: Multilingual program repair with llms,” in _Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Washington, DC, USA, February 7-14, 2023_ , B. Williams, Y. Chen, and J. Neville, Eds. AAAI Press, 2023, pp. 5131–5140. [Online]. Available: https://doi.org/10.1609/aaai.v37i4.25642
  * [57] D. Shrivastava, H. Larochelle, and D. Tarlow, “Repository-level prompt generation for large language models of code,” in _International Conference on Machine Learning_. PMLR, 2023, pp. 31 693–31 715.
  * [58] C. S. Xia, M. Paltenghi, J. L. Tian, M. Pradel, and L. Zhang, “Fuzz4all: Universal fuzzing with large language models,” in _ICSE_ , 2024.
  * [59] C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen, “Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models,” in _45th International Conference on Software Engineering, ser. ICSE_ , 2023.
  * [60] M. Schäfer, S. Nadi, A. Eghbali, and F. Tip, “An empirical evaluation of using large language models for automated unit test generation,” _IEEE Trans. Software Eng._ , vol. 50, no. 1, pp. 85–105, 2024. [Online]. Available: https://doi.org/10.1109/TSE.2023.3334955
  * [61] G. Ryan, S. Jain, M. Shang, S. Wang, X. Ma, M. K. Ramanathan, and B. Ray, “Code-aware prompting: A study of coverage guided test generation in regression setting using llm,” in _FSE_ , 2024.
  * [62] N. Alshahwan, J. Chheda, A. Finegenova, B. Gokkaya, M. Harman, I. Harper, A. Marginean, S. Sengupta, and E. Wang, “Automated unit test improvement using large language models at meta,” in _FSE_ , vol. abs/2402.09171, 2024\. [Online]. Available: https://doi.org/10.48550/arXiv.2402.09171
  * [63] S. Kang, J. Yoon, and S. Yoo, “Large language models are few-shot testers: Exploring llm-based general bug reproduction,” in _45th IEEE/ACM International Conference on Software Engineering, ICSE_ , 2023, pp. 2312–2323. [Online]. Available: https://doi.org/10.1109/ICSE48619.2023.00194
  * [64] S. Feng and C. Chen, “Prompting is all your need: Automated android bug replay with large language models,” in _ICSE_ , 2024.
  * [65] R. Bairi, A. Sonwane, A. Kanade, V. D. C, A. Iyer, S. Parthasarathy, S. Rajamani, B. Ashok, and S. Shet, “Codeplan: Repository-level coding using llms and planning,” 2023.
  * [66] L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig, “PAL: program-aided language models,” _CoRR_ , vol. abs/2211.10435, 2022\. [Online]. Available: https://doi.org/10.48550/arXiv.2211.10435