# Towards Generating Functionally Correct Code Edits from Natural Language
Issue Descriptions

Sarah Fakhoury  Microsoft Research ,  Saikat Chakraborty11footnotemark: 1
Microsoft Research ,  Madan Musuvathi  Microsoft Research and  Shuvendu K.
Lahiri  Microsoft Research

###### Abstract.

Large language models (LLMs), such as OpenAI’s Codex, have demonstrated their
potential to generate code from natural language descriptions across a wide
range of programming tasks. Several benchmarks have recently emerged to
evaluate the ability of LLMs to generate functionally correct code from
natural language intent with respect to a set of hidden test cases. This has
enabled the research community to identify significant and reproducible
advancements in LLM capabilities. However, there is currently a lack of
benchmark datasets for assessing the ability of LLMs to generate functionally
correct code edits based on natural language descriptions of intended changes.
This paper aims to address this gap by motivating the problem nl2fix of
translating natural language descriptions of code changes (namely bug fixes
described in Issue reports in repositories) into correct code fixes. To this
end, we introduce Defects4J-Nl2fix, a dataset of 283 Java programs from the
popular Defects4J dataset augmented with high-level descriptions of bug fixes,
and empirically evaluate the performance of several state-of-the-art LLMs for
the this task. Results show that these LLMS together are capable of generating
plausible fixes for 64.6% of the bugs, and the best LLM-based technique can
achieve up to 21.20% top-1 and 35.68% top-5 accuracy on this benchmark.

##  1\. Introduction

There has been a recent surge of interest in using large language models
(LLMs) to accomplish a variety of software-development tasks. For instance,
GitHub Copilot (GitHub, 2022), powered by OpenAI’s Codex model (Chen et al.,
2021), has demonstrated impressive capabilities in generating code based on a
natural language description and related code contexts. Recent additions to
Copilot (cop, [n. d.]) also highlight transforming code based on natural
language instructions. Besides Copilot, LLMs are powering several other
commercial AI-assisted software development products such as Amazon
CodeWhisperer (cod, [n. d.]), GhostWriter (gho, [n. d.]) and Tabnine (tab, [n.
d.]).

Research on LLMs for code generation has been spurred by benchmarks that are
used to evaluate and often improve the performance of these models on
particular tasks. For example, the HumanEval (Chen et al., 2021) benchmark for
evaluating the Codex model was instrumental in guiding the training of the
much improved GPT-4 successor by OpenAI (OpenAI, 2023). Improvement on such
offline benchmarks often translates to improvement in real-world usage of LLMs
for similar tasks (say, in an IDE). In fact, for the task of code generation
from natural language (we hereby refer to it as nl2code), several additional
benchmarks have come up in recent years, including basic crowd-sourced coding
benchmarks such as MBPP (from Google) (Austin et al., 2021b), Code Competition
benchmarks (Li et al., 2022), and those based on real-world programs (Yu et
al., 2023).

While several such benchmarks capture the ability of LLMs to generate
functionally correct code from natural language intent, there is a lack of
benchmarks that evaluate an LLMs ability to perform edits based on a natural
language intent (we refer to as nl2edit).

In real-world software development, a user is much more likely to perform such
software evolution and maintenance tasks in the context of an existing
software repository compared to writing a self-contained program from scratch
using natural language. Besides, in modern software development with cloud-
hosted continuous integration (CI) pipelines, mature software programs often
evolve through Pull Requests (PR) made in response to a natural language Issue
description corresponding to bug-fixes, optimizations, feature additions etc.

In this paper, we take the first step towards creating a benchmark for nl2edit
and evaluating current state-of-the-art LLMs on the problem. In particular, we
focus on the restricted problem of nl2fix that consists of the task of fixing
a buggy program where the bug is described in a natural language within an
Issue description. We leverage the popular Defects4J (Just et al., 2014)
benchmark comprising of real-world bugs and their fixes to define the
Defects4J-Nl2fix benchmark for nl2fix. We introduce an augmented dataset of
283 Java programs with their high-level descriptions of intended code changes,
along with the suite of tests that ensure that the fix addresses the defect
while preserving the functionality. We also introduce metrics for evaluating
LLMs for the problem of nl2fix on this benchmark.

Figure 1. Overview of the NL2Fix problem setting. Illustrated is the issue
title, description, and plausible fix for the JXPATH-149 bug. The figure
demonstrates the standard LLM prompt used to generate candidate patches, and
the evaluation of the patches using ground truth trigger and regression tests.

Figure 1 illustrates an example from the Defects4J-Nl2fix dataset. It shows an
Issue containing the Bug ID that references an issue from the
JxPath111https://issues.apache.org/jira//browse/JXPATH-149 project hosted on
Jira(jir, [n. d.]); a Title that describes the bug at a high level, and
finally a Description that provides more details of the bug including a
natural language description of a scenario to reproduce the bug. The original
function consists of the method body that contains the fix — we restrict our
dataset to fixes that are localized to a single method. The Issue, along with
the buggy function, constitute an input to the nl2fix problem. The output
patches generated are evaluated for functional correctness and defect-freedom
using the Regression tests and the Trigger tests, respectively. Each test $t$
in the set of Trigger tests fails on the buggy program and passes the user-
provided fix in the form of fixed function.

The motivation to use hidden tests only for validation comes from several
directions. First, prior work observes that most real-world bug-fix Issues are
not accompanied by a failing or a trigger test (Kang et al., [n. d.]). Second,
although a program may have accompanying regression tests, running such tests
in CI pipelines can be expensive and can only be invoked a small number (say,
5) times to be practical. As discussed later in related work (Section 5),
keeping the tests hidden distinguishes the nl2fix problem from the problem of
automated program repair (APR) (Monperrus, 2018; Gazzola et al., 2019; Gao et
al., 2022).

This paper also contributes a detailed empirical evaluation of the performance
of current state-of-the-art (SOTA) LLMs on this dataset. We choose three
different flavors of LLMs based on the generative pre-trained transformers
(GPT) neural architecture from OpenAI, (a) the Codex code completion model
code-davinci-002, (b) the Codex code editing model code-davinci -edit-001, and
(c) the ChatGPT conversational model gpt-3.5 -turbo. We evaluate these models
under different sampling settings and prompting strategies and perform
detailed quantitative and qualitative analysis on the accuracy and quality of
the suggested fixes.

Our results demonstrate that these LLMs together are capable of generating
plausible patches (_i.e.,_ satisfy the regression and trigger tests) for a
significant fraction, 64.6% of the bugs when sampling for up to 100
candidates. More interestingly, the ChatGPT model gpt-3.5-turbo outperforms
other models in terms of both the pass@1, pass@5, and pass@100 accuracy
metrics (we describe the pass@k metric more formally later in Section 3.4.2).
Finally, we describe a generic approach to rank an unordered set of candidate
patches based on LLM computed embedding similarity; the ranking makes the
suggestions deterministic with top-1 and top-5 accuracy of 21.20% and 35.68%,
respectively.

These findings highlight both the non-trivial nature of the Defects4J-Nl2fix
benchmark, as well as the capabilities of current LLMs to form decent initial
baselines that can spur further research.

Contributions. In summary, in this paper: ($i$) We motivate the nl2fix problem
and present a non-trivial augmented benchmark Defects4J-Nl2fix along with
metrics.

($ii$) We perform an extensive empirical evaluation of the performance of
three state-of-the-art LLMs on this benchmark.

($iii$) We describe a ranking strategy based on embedding similarity to
provide a ranked and deterministic list of fixes.

##  2\. Research Questions

This paper aims to understand the performance of current state-of-the-art LLMs
on the problem of nl2fix: the task of fixing a buggy program from natural
language intent. To this end, we define four research questions to empirically
evaluate model capabilities:

RQ1. Can LLMs generate fixes from natural language intent for NL2Fix? To
answer this RQ we explore the pass@k accuracy of three SOTA LLMs for
generating correct bug fixes using natural language issue descriptions.
Additionally, we extract quantitative statistics about the generated fixes,
including: the prevalence of duplicate suggestions, compilation percentage,
and the distribution and overlap of unique bugs for which there are plausible
patches from each approach. RQ2. What kind of candidate fixes do LLMs
generate?

To shed light on the nature of LLM generated candidate patches, we study the
characteristics of the patches in the context of their similarity to the
developer written ground truth fixes and the input buggy code. RQ3. What
sources of information do LLMs need to generate fixes for NL2Fix?

We explore what level of information LLMs need in order to correctly generate
fixes from natural language descriptions. We experiment with different
prompting styles using curated information, including: the high-level issue
summary, in-depth issue description, 0-shot and 1-shot prompt settings, as
well as bug fix reasoning generated using Reasoning Extraction prompting
strategies.

RQ4. Can LLMs be used to rank fixes for NL2Fix?

Based on observations from RQ1, RQ2, and RQ3 we explore how LLMs can be used
to design a simple ranking approach to rank fixes from the unordered set of
candidate fixes, allowing better approximation of pass@k metrics needed for a
developing a deterministic and practical bug fix recommender.

##  3\. Approach

Table 1. Statistics of the Dataset

Project | # Bugs | SH† | Avg. | Avg. Change | Issue Length
---|---|---|---|---|---
Bugs | hunks | Line | Token | Title | Desc.
Chart | 6 | 5 | 1.17 | 1.83 | 9.67 | 7.17 | 149.33
Cli | 28 | 16 | 1.68 | 4.07 | 22.07 | 9.36 | 206.46
Codec | 11 | 9 | 1.27 | 2.18 | 10.82 | 12.73 | 171.09
Collections | 1 | 1 | 1.00 | 1.00 | 1.00 | 20.00 | 457.00
Compress | 36 | 19 | 1.78 | 5.39 | 29.28 | 10.03 | 320.42
Csv | 12 | 8 | 1.42 | 2.50 | 18.92 | 10.83 | 104.25
JacksonCore | 13 | 9 | 1.38 | 3.69 | 20.69 | 11.38 | 251.69
JacksonDataBind | 67 | 36 | 1.87 | 5.37 | 33.90 | 11.90 | 294.49
JacksonXml | 5 | 1 | 2.80 | 6.20 | 30.80 | 11.60 | 126.80
JxPath | 10 | 5 | 1.60 | 4.80 | 22.60 | 9.70 | 159.10
Math | 73 | 37 | 2.11 | 5.29 | 35.00 | 10.07 | 165.18
Mockito | 21 | 16 | 1.33 | 3.29 | 27.90 | 9.00 | 311.67
Overall | 283 | 162 | 1.78 | 4.65 | 28.76 | 10.53 | 231.92

†SH: Single-Hunk

###  3.1. Dataset

In this paper, we take the first step towards creating a benchmark for a
nl2edit and evaluating current state-of-the-art LLMs on the problem. We focus
on the restricted problem of nl2fix which consists of the task of fixing a
buggy program where the bug is described in natural language within an issue
description.

We choose the Defects4J (Just et al., 2014) benchmark, comprising of bugs and
tests from real-world issues, from which we can extract issue descriptions.

In particular, we use Defects4J 2.0, a well-known benchmark of 835 manually
curated real-world bugs and fixes gathered from 17 Java projects. The existing
dataset consists of a set of bugs, bug reproducing test cases (trigger tests),
and regression test cases which load the class in which the method under test
is contained. Each bug in the Defects4J dataset contains a PRE_FIX_REVISION
and POST_FIX_REVISION version that represents the buggy/fixed versions of the
code respectively. The two versions reflect the actual state of the project
when the bug was discovered/fixed.

We use these developer-written tests to evaluate generated patches, a patch
must pass both the trigger and regression tests to be considered a plausible
patch. While this does not guarantee semantic equivalence between a generated
patch and the ground truth fix, we argue that it may be a realistic proxy for
patch correctness for two reasons. First, with the use of LLMs that are
capable of generating hundreds of candidate patches, manually evaluating each
generated patch for semantic equivalence can be prohibitively expensive (in
the order of 28,000 per model configuration). Thus, evaluating candidate
patches with the developer written tests serves as a scalable proxy for
preserving functionality and defect-freedom. Secondly, semantic equivalence
with user-provided fix may not be necessary, as there can be multiple, non-
equivalent yet acceptable, fixes to developers in practice. Without knowledge
of the detailed invariants of the project, it is difficult to determine if a
particular ground truth fix is the only acceptable fix.

We augment the Defects4J dataset in three distinct ways:

  * •

We restrict the Defects4J-Nl2fix dataset to consist of fixes that affect a
single method body. Among 835 bugs in the Defects4J 2.0 dataset, 283 contain
single-method bugs, i.e., bugs that can be fixed with single method changes.
Fixes may include multiple lines, but are scoped to a singular function. Table
1 contains a breakdown of the number of bugs per project.

We justify our decision to focus on single method bugs as methods generally
define a unit of code that can be reviewed independently compared to isolated
lines, or an entire file or repository. Second, the input prompt for LLMs are
restricted to only a few thousand tokens that may not suffice to capture the
entire file or repository level information. APR approaches using the
Defects4J dataset often restrict their dataset to contain only single hunk, or
single line bugs(Xia and Zhang, 2022). We do not make this restriction, and
Table 1 shows the average number of hunks for bugs in our dataset.

  * •

Second, to serve the nl2fix problem, we augment the Defects4J dataset by
pairing each bug with its corresponding issue metadata, including the issue
title and description, that we scrape from GitHub, SVN and Jira.

  * •

Finally, upon close investigation of the buggy methods in the Defects4J
dataset, we notice that as a side-effect to the bug patching process used by
the benchmark creators, comments that appear in the POST_FIX_REVISION also
appear in the PRE_FIX_REVISION of the
code222https://github.com/rjust/defects4j/issues/477. This means that comments
related to the actual fix made by the developer, may appear in the
PRE_FIX_REVISION that we use as input to the LLMs. To avoid these comments
providing hints about the solutions to the model, we remove all comments from
the PRE_FIX_REVISION.

###  3.2. Generative Pre-trained Transformers (GPT)

Generative Pre-trained Transformers (GPT) are large-scale auto-regressive
(Johansen, 1995) generation models trained to predict the next token given a
natural language prefix (prompt) context. The recent development of ultra-
large-scale GPT models with billions of parameters has shown to exhibit
emergent properties where they can perform tasks without finetuning (Patel et
al., 2022; Kojima et al., 2022). While asked to generate responses to a
prompt, GPT models samples over tokens’ probability distributions of one token
at a time. To generate the most probable response (or multiple responses),
these models perform different sampling, including temperature-based sampling,
which manipulates the distribution of tokens, controlling the diversity in the
responses. A lower temperature typically results in less diversity, and a
higher temperature otherwise. We use two temperate – 0.2 and 0.8 throughout
the experiments.

To answer the defined research questions, we select three state-of-the-art
GPT-based Large Language Models (LLM) that have shown strong capabilities on a
variety of code generation tasks333At the time of submission, the authors did
not have API access to the most recent state-of-the-art model, GPT-4 (OpenAI,
2023).

Codex. OpenAI’s Codex, code-davinci-002 is a language model specifically
designed for code completion tasks. It is based on the GPT-3 architecture and
has been fine-tuned on a large corpus of code from public repositories. Codex
excels at generating syntactically correct code and has been shown to be
highly effective for tasks involving code generation.

Codex Edit Model. The Codex edit model, code-davinci-edit-001, is a version of
Codex GPT-3 with editing capabilities. Given a code and instruction written in
NL such as ”Improve the runtime complexity of this function”, the models edits
the code to possibly satisfy the instruction.

ChatGPT. The recently released ChatGPT model (gpt-3.5-turbo) is based on the
pretrained GPT-3.5 model, which is further finetuned using Reinforcement
Learning with Human Feedback (RLHF) (Ouyang et al., 2022). While gpt-3.5-turbo
is not explicitly fine-tuned for code generation tasks, early evaluation has
demonstrated strong capabilities in several fields of science and engineering
(Qadir, 2022; Jiao et al., 2023; Qin et al., 2023; Kung et al., 2023)
including understanding and generating code snippets (Maddigan and Susnjak,
[n. d.]; Sobania et al., 2023; Surameery and Shakor, 2023). ChatGPT’s
conversational nature allows it to excel in tasks that require both code
generation and human-like interactions, allowing the use of advanced prompt
structures that involve chain of thought(Wei et al., 2022) and reasoning
extraction(Kojima et al., 2022).

Embeddings. OpenAI embedding models generate a high dimension vector
representation of input strings.

Research shows that similarity in such high-dimension vector space translates
to semantic similarity of strings. Among many other applications of such
representation, they can facilitate similarity analysis, searching, etc. In
this work, we leverage the text-embedding-ada-002 model to generate the
embedding of code and use embedding-based similarities to rank the patches.

###  3.3. Prompting Framework

Capabilities of LLMs are not fixed across all contexts, _i.e.,_ if an LLM gets
a question wrong, slightly changing the prompt by modifying the contents or
format of the information given may yield different outcomes. There as several
techniques to improve the accuracy and reliability of LLM output, these
techniques are referred to as prompt engineering(Reynolds and McDonell, 2021).
In this paper we use standard prompting as well as two distinct strategies
that have been shown to improve the performance of LLMs for complex tasks: 1)
few-shot prompting and 2) reasoning extraction. These strategies are designed
to help provide context and guidance to effectively solve a task while
mitigating potential pitfalls associated with model-generated outputs.

####  3.3.1. Zero-Shot Prompting

Zero-shot prompting, also frequently referred to as _standard prompting_ , is
the basic configuration of prompting a model with a task. The prompt does not
include any examples of acceptable solutions and does not break down the
problem into easier sub-problems. In this paper, our zero-shot, or standard,
prompt is illustrated in Figure 1. It is composed of the issue title and
description, along with the buggy code and an instruction to provide a fix to
the code.

####  3.3.2. Few-Shot Prompting

Few-shot prompting is a technique that involves presenting the model with a
series of examples or demonstrations in order to guide its understanding of
the task at hand. By providing the models with a few instances of similar
tasks, along with their respective inputs and desired outputs, we guide the
model output to the desired output, both in terms of functionality and format.
This approach enables the models to adapt their responses based on the
provided examples, leading to more accurate and coherent output. For a given
issue, we select another issue along with the buggy code and fixed code to
represent as a shot. We select the example to be an issue where the example
buggy code is closest to the target buggy code, using standard edit distance
metric.

####  3.3.3. Reasoning Extraction

Reasoning extraction is a strategy that focuses on extracting the underlying
rationale behind a specific task or problem (Kojima et al., 2022). We apply
this strategy to help the model comprehend the objective(s) and solution to
the code fix task. In particular, we explicitly interact three times with the
model with different queries. First, given the buggy code and issue report, we
ask the model to localize the bug, then we ask to explain why the localized
lines are buggy, finally we ask to fix the bug. ChatGPT’s conversational
nature allows the natural the use of advanced prompt structures that maintain
conversational context, like chain of thought reasoning and reasoning
extraction. Therefore, we only use this prompt strategy with gpt-3.5-turbo.

###  3.4. Correctness of Generated Code

Experiments are run in two phases: fix generation and fix validation. All
validation experiments are run in a Docker container running Ubuntu 20.04.4
with Java version OpenJDK 1.8.0 for which we make the Docker image public. To
generate candidate fixes we use the OpenAI API. The rest of this section
discusses details of patch validation and evaluation metrics:

####  3.4.1. Patch Validation

Each bug in the Defects4J dataset contains a PRE_FIX_REVISION and
POST_FIX_REVISION version that represents the buggy/fixed versions of the code
respectively. The two versions reflect the actual state of the project when
the bug was discovered/fixed. To determine whether a generated fix is correct,
we follow the following steps: 1) Check out the PRE_FIX_REVISION version of
the project 2) Replace the original buggy function with the generated function
and 3) Run the trigger and regression test(s) to determine if code containing
the generated fix passes the tests or not.

For each fix, the validation outcome can be either 1) Plausible: all bug
reproducing tests and regression tests pass 2) Wrong: at least 1 of the
trigger or regression tests fail or 3) Uncompilable.

####  3.4.2. Evaluation metrics

To measure the quality of a solution to the nl2fix, we use the pass@k
metric444we use pass@k and P@k interchangeably throughout the paper introduced
and widely used for evaluating LLMs for nl2code problems (Chen et al., 2021;
Austin et al., 2021a). Intuitively, given an unordered set of candidate fixes,
the pass@k provides the likelihood of choosing a correct fix when given $k$
tries to sample from this set of candidate fixes. In nl2fix scenario, a fix is
correct if it passes all the Trigger tests and Regression tests for the bug.
Given $n$ as the number of samples generated, $k$ as the number of samples to
estimate pass@k and $c$ is the number of correct samples in $n$, we use a the
following formula for calculating pass@k defined by (Chen et al., 2021):

(1)
$\text{pass@k}:=\underset{\text{problems}}{\mathbb{E}}\left[1-\frac{\binom{n-c}{k}}{\binom{n}{k}}\right]$

For this paper, we generate $n\geq k$ samples, where $n=100$. Additionally, to
answer RQ4, we also introduce a ranked pass@k, denoted r.pass@k (Section 4.4),
to determine the number of cases where at least one code fix is correct in the
top $k$ suggestions of a ranked list.

##  4\. Results

(a) Temp=0.8

(b) Temp=0.2

Figure 2. Pass@k results for 0-shot setting.

###  4.1. RQ1: Can LLMs generate fixes from natural language intent for
NL2Fix?

For each model, we consider two settings with the zero-shot prompt: temp. at
0.2 and temp. at 0.8. For each setting we generate 100 candidate fixes for
each of the 283 bugs and evaluate the correctness of each candidate against
the trigger and relevant tests. Next, we calculate the pass@k using technique
described in Section 3.4.2 and plot the results for temp. 0.8 in Figure 2(a)
and temp. 0.2 in Figure 2(b).

At temp. 0.8, we see that the edit model,code-davinci-edit-001, is the best
performing model overall with a pass@100 at 54.12%. gpt-3.5-turbo achieves a
slightly higher pass@1 (13.93% with respect to 12.18 for the edit model),
however at pass@20 and above, performance dips lower than of the completion
model, code-davinci-002. The completion model achieves the second highest
pass@100 with 45.9%, which is 8.22% lower than the edit model.

At temp. 0.2, we see precision improvements for pass@1 from both code-davinci-
edit-001 and gpt-3.5-turbo with 17.55% and 16.0% respectively. However, we see
consistently lower precision of all models starting at pass@5 through
pass@100. Most notably, the precision of code-davinci-002 is much lower at
temp. 0.2 with just 0.65% pass@1 and 3.55% pass@100. We tested the precision
of the model in two separate runs, and noticed consistently poor performance
at this setting.

To better understand the pass@k accuracy per model, we extracted high level
statistics about the code generated by each model for both temp.
configurations. Table 2 contains the average percentage of duplicate code
candidates generated per bug as well as the average percentage of candidates
that compile, pass on the regression tests, and pass on both the regression
and trigger tests (plausible) across bugs.

From Table 2 we can see that the number of duplicates generated increases
drastically when the temp. is decreased, which is expected as model behavior
is more deterministic at lower temperatures. For example, code-davinci-
edit-001, which is optimized for code edits, generates more than 90%
duplicates at 0.2 but only 26% at 0.8.

Overall, the percentage of generated code that compiles varies significantly
across models. We observe that only 4.66% of code generated by code-
davinci-002 at temp. 0.2 compiles, which explains the extremely low accuracy
seen in Figure 2(b). However, at temp. 0.8 the compilation rate increases
significantly, between 35.9% and 74.3%. Looking to related work, on a
different subset of the Defects4J dataset, SOTA neural APR techniques generate
patches with 15% to 28% compilation rates in top 100 (Xia and Zhang,
2022)(Lutellier et al., 2020)(Jiang et al., 2021). Although the APR setting is
different from that of nl2fix, the trigger and relevant tests _are not_ hidden
from the APR setting, we observe that using the entire method as input to the
LLMs have an advantage over generating a higher proportion of compilable
patches.

Table 2. Average candiate patch statistics across bugs.

| code-davinci-edit-001 | code-davinci-002 | gpt-3.5-turbo
---|---|---|---
Temp. | 0.2 | 0.8 | 0.2 | 0.8 | 0.2 | 0.8
Duplicate |  90.7% | 26% | 87.9% |  35.2% | 51.8% | 8.41%
Compile |  74.3% | 54.8% | 4.66% | 35.9% | 57.2% |  60.4%
Regression |  51.2% |  41.7% | 3.23% | 20.19% | 25.1% | 33.0%
Plausible |  20.9% | 12.0% | 0.65% | 6.0% | 16.0% |  13.4%

Both gpt-3.5-turbo and code-davinci-edit-001 achieve a higher precision for
pass@100 in the temp. 0.8 setting, compared to 0.2 (Figure 2(b)). However, in
table 2 we see that the average percentage of plausible patches is higher in
the 0.2 setting. While this appears counter intuitive, the presence of high
number of duplicates per patch modulates the calculated pass@k across bugs. In
other words, a model may have high confidence for a small number of bugs and
generate a high percentage of plausible patches for that bug.

Result 1.1: Given only an NL decription of a bug, all three LLMs are able to
generate plausible fixes for a modest number of bugs in the dataset, with
pass@1 of at most 6.29% – 17.55% and pass@100 of at most 42.19% – 54.12%. In
the 0-shot setting, code-davinci-edit-001 achieves the overall highest
accuracy compared to both code-davinci-002 and gpt-3.5-turbo.

We report the number of bugs with plausible fixes for each project in Table 3.
At a high level, we observe that the distribution of plausible fixes generated
from each model is distributed across every project. In general for every
project, the three models generate plausible fixes for a similar percentage of
bugs. There are a few notable exceptions, for example, the Collections
project, which only contains 1 bug, was only correctly fixed by code-davinci-
edit-001. gpt-3.5-turbo also has lower accuracy on the Mockito and
JacksonDatabind projects only generating plausible fixes for 1/21 and 7/67
bugs respectively, compared to 6/21 and at least 22/67 for the other two
models. However, on the Codec project, gpt-3.5-turbo generates plausible fixes
for two more bugs than the other two models. Overall, the total bugs patched
by each model aligns with the pass@100 metrics seen in Figure 2(a).

Table 3. Bugs with plausible patches, by project. Bugs with plausible fix /
total number of bugs for that project.

Approach | davinci-002 | edit-001 | gpt-3.5
---|---|---|---
Chart | 6/6 | 6/6 | 5/6
Cli | 13/28 | 16/28 | 13/28
Codec | 7/11 | 7/11 | 9/11
Collections | 0/1 | 1/1 | 0/1
Compress | 15/36 | 19/36 | 18/38
Csv | 10/12 | 8/12 | 9/12
JacksonCore | 8/13 | 9/13 | 5/13
JacksonDatabind | 22/67 | 27/67 | 7/67
JacksonXml | 1/5 | 2/5 | 2/5
JxPath | 3/10 | 5/10 | 4/10
Math | 38/73 | 45/73 | 45/73
Mockito | 6/21 | 6/21 | 1/21
Total | 129/283 | 151/283 | 118/283

Figure 3. Overlap between bugs with plausible patches for each LLM in the
0-shot temp 0.8 setting.

While two models may patch the same number of bugs for a project, the exact
bugs that are patched may be different. Figure 3 shows the overlap of the
number of bugs each model is able to generate plausible patches for. All three
models are able to generate plausible patches for the same 28% (82 of 283) of
the bugs. However, we observe that when combined together, the three models
can generate plausible patches for 64% (183 of 283) of the dataset. Each model
has a unique subset of bugs that the other two models are not able to generate
plausible patches for: 22 unique bugs by code-davinci-edit-001, 18 by
gpt-3.5-turbo, and 10 by code-davinci-002. While some bugs may be easier to
fix for certain models, this does not appear to be a consistent artifact of
the project that the bug originates from, as observed from Table 3.

Result 1.2: 28% (82/283) of the bugs in the dataset can be fixed by all three
LLMs. When combined together, the three LLMs can generate plausible patches
for 64% (183/283) of the bugs in the dataset.

###  4.2. RQ2: What kind of candidate fixes do LLMs generate?

(a) With the actual fixed code.

(b) With the Buggy code.

(c) Plausible patches with input buggy code and actual fixed code.

Figure 4. Code similarity analysis of generated patches by different models.
We analyze the CodeBLEU similarity here.

To answer this RQ, we select the best-performing configuration from RQ1 (temp.
0.8) to better understand the nature of the code generated by the LLMs. We
study the characteristics of the patches generated by different models. To
understand the characteristics of different patches, we study the similarity
of those patches _w.r.t._ , the buggy code (present as part of the input) and
the actual fixed code. We use CodeBLEU (Ren et al., 2020) as the
representative similarity measurement. Given two code $c_{1}$ and $c_{2}$,
CodeBLEU is defined as $\alpha*B+\beta*W+\gamma*S+\delta*D$, where $B$, $W$,
$S$, $D$ are BLEU score, Keywords BLEU score, Syntax match score, and Dataflow
match score, respectively between $c_{1}$, and $c_{2}$, and $\alpha$, $\beta$,
$\gamma$, $\delta$ are weighting constants typically all set to 0.25
555Microsoft’s CodeBLEU implementation. We choose CodeBLEU for this research
question since it considers the syntax and semantic match between code, in
addition to the lexical match.

Figure 5. An example showing the contrast between actual fixed code and model-
generated plausible patch for Bug id 20 of JxPath project. Even though the
generated patch passed does not exactly match the ground truth fix, it passed
all the regression tests and the trigger test, making it a semantic equivalent
of the actual fix.

Figure 4(a) shows the similarity of patches with the actual fixed code. Across
all three models we have studied in this paper, the patches that passed both
the regression test and the bug revealing trigger test (_i.e.,_ plausible
patch) exhibit higher CodeBLEU with the actual fixed code compared to the
patches that are not plausible. Such a result is expected since the plausible
patches pass the whole test suite; the plausible patch should be, in theory, a
semantic equivalent of the actual fix exhibiting higher CodeBLEU.
Interestingly, the plausible patches generated by the gpt-3.5-turbo model
exhibit higher variability regarding CodeBLEU similarity with the actual
patch. The Inter-Quartile Range (IQR) of CodeBLEU between plausible patches
and actual fixed code is 0.18 and 0.19, respectively, for code-davinci-
edit-001 and code-davinci-002. In contrast, for gpt-3.5-turbo, the IQR is
0.26. In addition, for the code-davinci-edit-001 and code-davinci-002 models,
the kurtosis are 2.62 and 1.12, respectively, signifying the distributions are
more centered, whereas gpt-3.5-turbo models kurtosis is 0.41, signifying
diverse generation capability, as evident from Figure 4(a).

✓ gpt-3.5-turbo generates more diverse patches as compared to other models.

Further, we analyze how similar the generated patches are _w.r.t._ , the buggy
code. Figure 4(b) shows the distribution of CodeBLEU of different types of
patches across different models. Across all three models, interestingly, we
observe that plausible patches exhibit higher CodeBLEU with the buggy code
than their non-plausible counterparts. We conjecture that when models make
extensive modifications of the input buggy code, the resultant code are
infested with different problems causing them to fail compilation, regression
tests, and the trigger test (see Table 2). We conjecture that LLMs would make
a more significant impact on the nl2fix problem if we had the option to
control the deviation from the input buggy code. Nevertheless, the observation
that plausible patches exhibit higher similarity with buggy code opens up a
new possibility of ranking the LLM-generated code based on their similarity
with buggy code, which we will investigate in detail in the following research
question.

✓ Plausible patches exhibit higher similarity with the buggy code than non-
plausible patches.

Table 4. Statistics of bugs and patches where models generate patches which
exactly matches the ground truth fix

Model | # Fixed | # EM∗ | # Plausible | # EM
---|---|---|---|---
Bugs | Bugs | Patches | Patches
code-davinci-edit-001 | 151 | 32 | 1724 | 146
code-davinci-002 | 129 | 24 | 1182 | 120
gpt-3.5-turbo | 118 | 11 | 2356 | 73

∗ EM : Exact Match ignoring whitespaces

There is a consensus among academics and practitioners that LLMs such as the
davinci or gpt-3.5 models were pretrained on virtually all open source. As
such, it is not unexpected that these LLMs have already observed the methods
in our dataset during their respective pretraining. This raises a question,
how much do the LLMs memorize from their pretraining (McCoy et al., 2021;
Carlini et al., 2022; Tirumala et al., 2022)? Unfortunately, there is no good
way to measure such unless we know what LLMs’ pretraining data is. Regardless,
to qualitatively understand the generated patches, we investigate the CodeBLEU
similarity of the patches with the buggy code (which was available in the
input to LLM) and the actual fixed code. Across all three models, generated
patches exhibit slightly higher similarity with the buggy code than the actual
fixed code (see Figure 4(c)). Such difference is statistically significant by
one-sided Wilcoxon sign rank test with p-values of $1.6*10^{-9}$,
$9.9*10^{-5}$, $2.7*10^{-14}$ for code-davinci-edit-001, code-davinci-002, and
gpt-3.5-turbo, respectively.

Table 4 shows summary statistics of patches that match the dataset’s ground
truth fix exactly. The code-davinci-edit-001 model correctly generated at
least one patch for 151 bugs, among which the patch for 32 bugs exactly
matches the ground truth. For the code-davinci-002 model, such number is 24
out of 129 and 11 out of 118 for the gpt-3.5-turbo. Only 8.5% (146 out of
1724), 10.1% (120 out of 1182), and 3% (73 out of 2356) patches of the
plausible patches are an exact match with the ground truth for code-davinci-
edit-001, code-davinci-002, and gpt-3.5-turbo, respectively. These results
show that most of the plausible patches are syntactically distinct from the
ground truth fix.

In Figure 5, we show one of the plausible patches generated by gpt-3.5-turbo
for the example in Figure 1. The buggy code misplaced the arguments of
containsMatch function in line 16(Figure 5(a)), while the developer-written
patch fixed the error by putting the arguments in correct positions (line 16
in Figure 5(b)). Figure 5(d) shows a fixed code generated by the gpt-3.5-turbo
model, which is semantic equivalent of the developer-written code. In fact,
LLM generated fix actually inlines the implementation of containsMatch (shown
in Figure 5(c)) function into the context (line 23-28 in Figure 5(d)). In
addition, the LLM-generated patch refactors the code by extracting two
variables corresponding to two boolean expressions used in the original code,
making the resultant code more readable. We observe that even though arguably
LLMs already had seen everything open source, they explore new variations of
code when applied to the NL2Fix problem.

✓ LLMs may not always memorize code from their pretraining. They explore new
code while trying to generate patches.

Result 2: Across all three models we studied, the plausible patches exhibit
higher similarity with input buggy code than non-plausible patches with
differences in the median similarity of $\sim$0.15. The plausible patches also
exhibit higher similarity with the buggy code than the actual fixed code with
statistical significance (p-value $<<$ 0.05). The plausible patches exactly
match the developer-written patch for up to 10% of the cases.

Table 5. Accuracy of different prompt configurations.

Model | Prompt | P@1 | P@5 | P@100
---|---|---|---|---
code-davinci-edit-001 | Issue Title | 4.73 | 15.39 | 44.12
0-shot | 12.18 | 28.86 | 54.12
1-shot | 4.73 | 12.75 | 30.24
gpt-3.5-turbo | Issue Title | 8.06 | 17.69 | 37.72
0-shot | 13.43 | 24.62 | 42.14
1-shot | 16.22 | 31.93 | 56.93
∗R.E. | 15.44 | 25.72 | 47.68
code-davinci-002 | Issue Title | 0.25 | 1.22 | 14.94
0-shot | 6.29 | 19.8 | 45.91
1-shot | 3.92 | 14.34 | 40.09

∗RE: Reasoning Extraction

###  4.3. RQ3: What sources of information do LLMs need to generate fixes for
NL2Fix?

To determine what information is needed to generate plausible fixes, we use
different prompting techniques that provide different levels of information to
each LLM and evaluate the pass@k metrics for each approach. In RQ1 and RQ2, we
used a basic 0-shot prompt, containing the issue title and description, as
described in Section 3.3.

Table 5 shows how changes to this basic prompt impacts pass@k for the
following prompts:

  1. (1)

0-shot is the standard prompt structure used in RQ1-2, as represented in
Figure 1.

  2. (2)

’Issue Title’ indicates removing the issue description and preserving the rest
of the prompt structure.

  3. (3)

1-shot adds an example of an issue, the corresponding buggy method, and the
correct fix for the model to learn from. Then the rest of the prompt preserves
the original format. The example chosen is based on its similarity to the code
to be fixed. Details on this selection can be found in Section 3.3

  4. (4)

For Reasoning Extraction, we modify the prompt that asks gpt-3.5-turbo to
break down the problem into two steps: 1) localizing the buggy lines in the
original method and 2) explaining why these lines contain a bug before asking
for the fix. An example is shown in Figure 6

Across all three models, we observe that compared to using the issue title
_and_ the description (0-shot) the issue title alone is not sufficient to
achieve comparable P@k, with a decrease of close to 5% – 8% pass@1 and 5% –
30% Pass@100. This indicates that the issue description contains valuable
information for the model to generate plausible fixes.

✓ Issue descriptions provide LLMs with helpful context to solve the NL2Fix
problem.

For code-davinci-edit-001 and code-davinci-002, we observe that adding
examples in the prompt for 1-shot approach does not improve performance, and
in fact decreases all pass@k compared to 0-shot attempts. With the 1-shot
setting, the average percentage of patches that compile for code-davinci-
edit-001 falls from 54.8% (Table 2) in the 0-shot setting to 24.7%. Taking a
closer look at the patches generated in the 1-shot attempt by code-davinci-
edit-001, we see several patches contain code from the 1-shot example in the
prompt. For example, the original buggy method for Issue ID MATH-58 has the
following method signature: public double[] fit() {...} . The examples used in
the 1-shot setting has the function signature private boolean isShortOption
(String token) {..} , and six of the generated patches edit the
isShortOption() function, instead of the buggy fit() function.

This indicates that the model attempted to edit the target function using code
from the example. The code-davinci-edit-001 model expects an input code and a
set of instructions for the edit. While OpenAI does not make the technical
details of how code-davinci- edit-001 was created, the observed behavior may
indicate that this model was not fine-tuned with k-shot instructions, which
may explain the degradation in model performance. Further, the context window,
_i.e.,_ the max number of tokens that can be used in the prompt, is much
smaller for code-davinci-edit-001, around 3000 tokens, compared to around 8000
for code-davinci-002 and around 4000 for gpt-3.5-turbo. Therefore, several
examples are truncated to fit the context window to ensure that the target
issue is still present in the prompt, leading to malformed method bodies.

With the 1-shot setting, the average percentage of patches that compile for
code-davinci-002 falls from 35.9% (Table 2) in the 0-shot setting to 20.5%.
When looking at the generated patches, we notice a large number are incomplete
generations, _i.e.,_ the code does not compile because the generated function
is not syntactically correct. Examples of this include missing closing curly
braces (}), missing return statements, and completions that stop halfway
through the target function. Note that there are two known ways of stopping
the completion task —(a) by setting maximum length, (b) by setting specific
STOP words. We set the maximum length to 750 tokens. In addition, we appended
the method signature of the fixed method at the end of the prompt so that the
completion model only needs to complete the body of the fixed method.

✓ Information from examples used in 1-shot prompting do not help pass@k
accuracy for code-davinci-edit-001 and
code-davinci-002 for the NL2Fix problem.

On the other hand, compared to 0-shot attempts, pass@k for gpt-3.5-turbo
improves for both 1-shot and reasoning extraction prompts. For example, pass@1
improves from 13.43% to 16.22% in the 1-shot setting and 15.44% using
reasoning extraction. Figure 6 shows an example of the interaction in the
reasoning extraction prompt configuration. The example is from the
JacksonDatabind project666https://github.com/fasterxml/jackson-
databind/issues/2265. When asked to identify the lines of code where the bug
exists, gpt-3.5-turbo returns the correct defect region in response 1. Then,
prompt 2 appends the original prompt1 and response 1 as part of the context
for prompt 2, with additional instructions to explain why the identified lines
of code contain a bug. In response 3 gpt-3.5-turbo explains the issue and
extracts a sample input that the code would fail on and the corresponding
error, from the issue description. In the final prompt, we append all inputs
and responses to request a final fixed version of the buggy function. This is
an example that gpt-3.5-turbo is able to generate a plausible patch for in the
reasoning extraction setting, but not the 0-shot setting.

✓ Both Reasoning Extraction and 1-shot prompting approaches help gpt-3.5-turbo
to produce a higher number of plausible patches and increase pass@k.

Figure 6. A bug correctly patched using Reasoning Extraction.

However, in the 1-shot setting gpt-3.5-turbo is able to generate correct
patches for 56 new bugs, and loses the ability to patch 14 bugs from the
0-shot setting. Using Reasoning Extraction, gpt-3.5-turbo can generate correct
patches for 34 new bugs, but loses the ability to patch 18 bugs from the
0-shot setting. Compared to all approaches pooled together, gpt-3.5-turbo in
the reasoning extraction setting can only uniquely patch 4 bugs. Looking at
the information contained in the issue descriptions for each of these
examples, we observe gpt-3.5-turbo is able to correctly localize buggy lines
and reason about why they are buggy, but only with help from the context in
the issue description. See Figure 6 for an example. While these prompting
techniques do boost aggregate performance metrics, they also may degrade on a
subset of the bugs in the dataset.

Result 3: Issue descriptions provide helpful context for solving the NL2fix
problem. Prompting techniques that provide examples, _i.e.,_ few-shot
prompting, and break down the task, _i.e.,_ reasoning extraction,
significantly improve accuracy of gpt-3.5-turbo on aggregate metrics like
pass@k, however performance may degrade on certain subsets of the dataset and
does not guarantee solutions over standard prompting.

###  4.4. RQ4: Can LLMs be used to rank fixes for NL2Fix?

Recall, given an unordered set of $n$ candidate solutions with $c$ correct
solutions for a given problem (a bug in our case), the pass@k metric refers to
the likelihood of picking at least one correct solution within $k$ tries. Such
a statistics is useful for evaluating language models, but does not readily
provide an useful real-world recommender system that proposes a small number
candidate fixes (upto say $k=5$) deterministically to a user. For instance,
for $n=100$ and $k=5$, there are $n\choose k$ ¿ $75$ million ways to choose
$5$ solutions from $100$ samples. For a practical tool for nl2fix, we would
like to develop a ($i$) a deterministic way to rank the suggestions and
present the top $k$ ranked suggestions to a user, and ($ii$) retain a high
accuracy closer to the average pass@k metric.

In this section, we leverage LLMs to propose a simple and generic ranking
strategy that helps realize the two objectives ($i$) and ($ii$) above. In
particular, inspired by our findings from RQ2, we explore if using a
similarity between the embeddings of the input buggy function and the
generated patches can identify plausible patches. We generate embeddings for
all buggy functions and corresponding patches using the text-embedding-ada-002
embedding model from OpenAI, see Section 3 for details. We compute a cosine
similarity between the embeddings for every pair of buggy code, and
corresponding candidate patch. We use this score to prune away patches with
similarity scores lower than the median (0.95). We fix this number across
models for consistency. Then, to avoid ranking patches with extremely high
similarity scores, _e.g.,_ 1.0 , we rank the patches in order of lowest cosine
similarity (starting at 0.95) to highest. Based on observations from RQ2,
patches with lower scores are more likely to belong to the distribution of
wrong or uncompilable patches. Patches should be sufficiently close to the
buggy input program, but not so close that there is not significant change.

For each model, we exercise the ranking scheme on two sets of patches: ($a$)
all generated patches, and ($b$) the subset of generated patches that pass the
compiler.

To evaluate the ranking strategy, we choose the LLM configurations for each
model with the highest discrepancy between pass@1 and pass@100 from RQ3 (see
Table 5), which also happens to be the best performing configuration for each
model.

Table 6 shows the ranked pass@1 and ranked pass@5 accuracy (denoted by r.P@k)
for the two sets of patches: 1) before (denoted as All) and 2) after (denoted
as Pruned) pruning the compiler errors. For reference, we also report the P@k
metrics in parenthesis. We observe that pruning compiler errors does improve
the metric P@k, especially the P@5 for davinci-002 model by over 14% (33.97 up
from 19.8). Second, while providing determinism, r.P@1 improves over P@1 for
most configurations except a slight dip for gpt-3.5-turbo for Pruned. For
davinci-002, ranking improves the r.P@1 by over 5 percentage points. Finally,
r.P@5 remains close to P@5 for most configurations except for gpt-3.5 where
the r.P@5 trails P@5 by 4.68%.

Result 4: By pruning compilation failures LLMs can achieve as high as 39.66%
pass@5. Using cosine similarity between embeddings of candidate patches and
buggy input code, we can apply a deterministic ranking strategy that retains
this high accuracy with 34.2% - 35.68% r.pass@5.

Table 6. Accuracy of ranking with and without compiler pruning.

Approach | Metric | gpt-3.5 | davinci-edit | davinci-002
---|---|---|---|---
1-shot | 0-shot | 0-shot
All | r.P@1 (P@1) | 16.96 (16.22) | 13.78 (12.18) | 11.30 (6.29)
r.P@5 (P@5) | 31.8 (31.93) | 30.74 (28.86) | 17.66 (19.8)
Pruned | r.P@1 (P@1) | 21.20 (22.22) | 19.78 (18.99) | 17.66 (16.89)
r.P@5 (P@5) | 34.98 (39.66) | 35.68 (36.73) | 34.2 (33.97)

##  5\. Related Work

Our work is most closely related to two broad lines of work (a) automated code
editing, and (b) automatic program repair (APR).

Automated Code Editing. Earlier works in learning code editing include
learning to edit code for refactoring (Meng et al., 2015; Rolim et al., 2017),
learning semantic code changes for bugs found by code analyzers (Rolim et al.,
2018). Recent approaches leverage Deep Learning techniques to learn frequent
code edit patterns from code changes mined from GitHub (Tufano et al., 2019;
Chakraborty et al., 2020; Dinella et al., 2020).

In addition to learning from historic changes, some recent approaches
(Chakraborty and Ray, 2021; Zhang et al., 2022) propose to guide code editing
with auxiliary inputs such as commit message. We argue that commit messages
are post-facto description of the code changes, and do not capture the intent
of the change, rather a summarization of the change. In contrast, the issue
report we consider in nl2Fix problem is ante-facto description of the changes,
which arguably capture the intent closer. Tufano  _et al._ (Tufano et al.,
2021) proposed automating code-review activity by editing the code based on
reviewers comments. While closest to our work, the approaches differ in our
usage of tests and semantic metrics such as pass@k to validate the functional
correctness and defect-freedom of proposed patches.

Automatic Program Repair.

Approaches for APR broadly fall under search-based techniques and machine
learning-based methods. For a comprehensive overview of automated program
repair techniques, we refer readers to recent works that survey the area
(Monperrus, 2018; Gazzola et al., 2019; Goues et al., 2019; Gao et al., 2022).

Search-based techniques use a generate-and-validate approach, where variations
of the original code are generated and then evaluated using the failing
tests(Saha et al., 2017; Weimer et al., 2013; Qi et al., 2014; Tan and
Roychoudhury, 2015; Kim et al., 2013). These approaches transform buggy code
using different transformation including random transformation (Weimer et al.,
2013; Le Goues et al., 2012; Qi et al., 2014), manually designed
transformation (Saha et al., 2017), and transformation learned from previous
corpus of code (Kim et al., 2013; Le et al., 2016; Tufano et al., 2018; Chen
et al., 2019).

Recently, researchers have also leveraged LLMs for APR. Alpharepair (Xia and
Zhang, 2022) uses a cloze-style APR where an LLM directly fills-in the correct
code given its surrounding context from the buggy program. Xia et al. (Xia and
Zhang, 2023) uses ChatGPT to setup a conversational APR problem where feedback
from failed patches is used to augment subsequent prompts to LLMs. Finally,
there are approaches leveraging LLMs for generating trigger tests for APR
problem from Issue descriptions (Kang et al., [n. d.]) as well as aiding
rootcausing for APR (Motwani and Brun, [n. d.]). Finally, Fan et al. (Fan et
al., 2023) leverage APR methods (including those based on LLMs) to repair code
generated from natural language intent.

Although closely related, our approach subtly differs from APR in the use of
hidden tests that are only used for evaluation and never specified as inputs
to the repair algorithm. This makes it a more applicable problem for real-
world bug fixes that may not have failing tests available (or be prohibitively
expensive to leverage) during the inference time for help in rootcausing buggy
lines and validating the fixes.

##  6\. Limitations and Threats

_Stability of models’ output._ As we have used OpenAI web API to access
different models, we cannot control the stochasticity of the output by the
model. The models themselves are often updated. This poses a threat to the
replicability of our study. To mitigate this threat, we make available all the
outputs generated by the models.

_Assumption on patch correctness._ In this paper, we leverage tests to
determine if a fix is plausible. However, such a plausible patch may not fix a
bug completely. This may need manual analysis or a semantic equivalence with
the user-provided fix. However, the former is infeasible for a large-scale
benchmark such as Defects4J-Nl2fix, and semantic equivalence-checking
techniques do not scale to handle most real programs. Given that test-suites
are never exhaustive, we can appeal to recent research that investigates
patch-correctness (Le et al., 2019; Xiong et al., 2018; Wang et al., 2020) to
improve confidence on the patches.

_Generalization of findings._ Given the relatively small number of bugs (283)
considered in Defects4J-Nl2fix benchmark, our findings may not generalize to
arbitrary bugs across different languages and software repositories. To
mitigate this threat we use real-world bugs from open source projects.

##  7\. Conclusion

In this paper, we motivate the nl2fix problem, define the first benchmark
Defects4J-Nl2fix, and perform detailed empirical evaluation of various SOTA
LLMs on the problem.

We believe that the task of nl2fix along with challenging benchmarks such
Defects4J-Nl2fix will serve as an important real-world benchmark for
evaluating future generation of these LLMs (such as GPT-4), while leveraging
new emergent behaviors of such LLMs (such as the ability of these models to
predict the correctness) to improve the performance on such benchmarks. In
future work, we plan to combine issue-driven trigger test generation (Kang et
al., [n. d.]) and user-in-the-loop (Lahiri et al., 2022) to improve the trust
in generated fixes, as well as extend our framework to the more general
problem of nl2edit to cover other forms of program evolution including feature
additions, refactorings as well as optimizations.

## References

  * (1)
  * cod ([n. d.]) [n. d.].  Amazon CodeWhisperer.  https://aws.amazon.com/codewhisperer/.  Accessed: 2023-03-29.
  * jir ([n. d.]) [n. d.].  Atlassian - JIRA.  https://www.atlassian.com/software/jira.  Accessed March 29, 2023..
  * gho ([n. d.]) [n. d.].  Ghostwriter.  https://replit.com/ai.  Accessed: 2023-03-29.
  * cop ([n. d.]) [n. d.].  GitHub Copilot x The Future of AI-Powered Software Development.  https://www.linkedin.com/pulse/github-copilot-x-future-ai-powered-software-development-github/.  Accessed: March 28, 2023.
  * tab ([n. d.]) [n. d.].  Tabnine.  https://www.tabnine.com/.  Accessed: 2023-03-29.
  * Austin et al. (2021b) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021b.  Program synthesis with large language models.  _arXiv preprint arXiv:2108.07732_ (2021).
  * Austin et al. (2021a) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021a.  Program Synthesis with Large Language Models.  https://doi.org/10.48550/ARXIV.2108.07732
  * Carlini et al. (2022) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022.  Quantifying memorization across neural language models.  _arXiv preprint arXiv:2202.07646_ (2022).
  * Chakraborty et al. (2020) Saikat Chakraborty, Yangruibo Ding, Miltiadis Allamanis, and Baishakhi Ray. 2020.  Codit: Code editing with tree-based neural models.  _IEEE Transactions on Software Engineering_ 48, 4 (2020), 1385–1399.
  * Chakraborty and Ray (2021) Saikat Chakraborty and Baishakhi Ray. 2021.  On multi-modal learning of editing source code. In _2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 443–455.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021\.  Evaluating Large Language Models Trained on Code.  https://doi.org/10.48550/ARXIV.2107.03374
  * Chen et al. (2019) Zimin Chen, Steve Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and Martin Monperrus. 2019.  Sequencer: Sequence-to-sequence learning for end-to-end program repair.  _IEEE Transactions on Software Engineering_ 47, 9 (2019), 1943–1959.
  * Dinella et al. (2020) Elizabeth Dinella, Hanjun Dai, Ziyang Li, Mayur Naik, Le Song, and Ke Wang. 2020\.  Hoppity: Learning graph transformations to detect and fix bugs in programs. In _International Conference on Learning Representations (ICLR)_.
  * Fan et al. (2023) Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. 2023.  Automated Repair of Programs from Large Language Models.  arXiv:2205.10583 [cs.SE]
  * Gao et al. (2022) Xiang Gao, Yannic Noller, and Abhik Roychoudhury. 2022.  Program Repair.  arXiv:2211.12787 [cs.SE]
  * Gazzola et al. (2019) Luca Gazzola, Daniela Micucci, and Leonardo Mariani. 2019\.  Automatic Software Repair: A Survey.  _IEEE Transactions on Software Engineering_ 45, 1 (2019), 34–67.  https://doi.org/10.1109/TSE.2017.2755013
  * GitHub (2022) GitHub. 2022.  GitHub Copilot.  Accessed August 5, 2022. https://github.com/features/copilot/.
  * Goues et al. (2019) Claire Le Goues, Michael Pradel, and Abhik Roychoudhury. 2019\.  Automated program repair.  _Commun. ACM_ 62, 12 (2019), 56–65.
  * Jiang et al. (2021) Nan Jiang, Thibaud Lutellier, and Lin Tan. 2021\.  Cure: Code-aware neural machine translation for automatic program repair. In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. IEEE, 1161–1173.
  * Jiao et al. (2023) Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023.  Is ChatGPT a good translator? A preliminary study.  _arXiv preprint arXiv:2301.08745_ (2023).
  * Johansen (1995) Søren Johansen. 1995\.  _Likelihood-based inference in cointegrated vector autoregressive models_.  OUP Oxford.
  * Just et al. (2014) René Just, Darioush Jalali, and Michael D Ernst. 2014\.  Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In _Proceedings of the 2014 international symposium on software testing and analysis_. 437–440.
  * Kang et al. ([n. d.]) Sungmin Kang, Juyeon Yoon, and Shin Yoo. [n. d.].  Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction. In _IEEE/ACM International Conference on Software Engineering (ICSE)_. IEEE, (to appear).
  * Kim et al. (2013) Dongsun Kim, Jaechang Nam, Jaewoo Song, and Sunghun Kim. 2013\.  Automatic patch generation learned from human-written patches. In _2013 35th International Conference on Software Engineering (ICSE)_. IEEE, 802–811.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022\.  Large language models are zero-shot reasoners.  _arXiv preprint arXiv:2205.11916_ (2022).
  * Kung et al. (2023) Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. 2023\.  Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models.  _PLoS digital health_ 2, 2 (2023), e0000198.
  * Lahiri et al. (2022) Shuvendu K. Lahiri, Aaditya Naik, Georgios Sakkas, Piali Choudhury, Curtis von Veh, Madanlal Musuvathi, Jeevana Priya Inala, Chenglong Wang, and Jianfeng Gao. 2022\.  Interactive Code Generation via Test-Driven User-Intent Formalization.  _arXiv preprint arXiv:2208.05950_ (2022).
  * Le et al. (2019) Xuan-Bach D Le, Lingfeng Bao, David Lo, Xin Xia, Shanping Li, and Corina Pasareanu. 2019\.  On reliability of patch correctness assessment. In _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. IEEE, 524–535.
  * Le et al. (2016) Xuan Bach D Le, David Lo, and Claire Le Goues. 2016.  History driven program repair. In _2016 IEEE 23rd international conference on software analysis, evolution, and reengineering (SANER)_ , Vol. 1. IEEE, 213–224.
  * Le Goues et al. (2012) Claire Le Goues, ThanhVu Nguyen, Stephanie Forrest, and Westley Weimer. 2012.  GenProg: A Generic Method for Automatic Software Repair.  _IEEE Transactions on Software Engineering_ 38, 1 (2012), 54–72.  https://doi.org/10.1109/TSE.2011.104
  * Li et al. (2022) Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, et al. 2022\.  Automating code review activities by large-scale pre-training. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 1035–1047.
  * Lutellier et al. (2020) Thibaud Lutellier, Hung Viet Pham, Lawrence Pang, Yitong Li, Moshi Wei, and Lin Tan. 2020.  Coconut: combining context-aware neural translation models using ensemble for program repair. In _Proceedings of the 29th ACM SIGSOFT international symposium on software testing and analysis_. 101–114.
  * Maddigan and Susnjak ([n. d.]) Paula Maddigan and Teo Susnjak. [n. d.].  Chat2vis: Generating data visualisations via natural language using chatgpt, codex and gpt-3 large language models. In _IEEE/ACM International Conference on Software Engineering (ICSE)_. IEEE, (to appear).
  * McCoy et al. (2021) R Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. 2021.  How much do language models copy from their training data? evaluating linguistic novelty in text generation using raven.  _arXiv preprint arXiv:2111.09509_ (2021).
  * Meng et al. (2015) Na Meng, Lisa Hua, Miryung Kim, and Kathryn S McKinley. 2015\.  Does automated refactoring obviate systematic editing?. In _2015 IEEE/ACM 37th IEEE International Conference on Software Engineering_ , Vol. 1. IEEE, 392–402.
  * Monperrus (2018) Martin Monperrus. 2018\.  Automatic Software Repair: A Bibliography.  _ACM Comput. Surv._ 51, 1, Article 17 (jan 2018), 24 pages.  https://doi.org/10.1145/3105906
  * Motwani and Brun ([n. d.]) Manish Motwani and Yuriy Brun. [n. d.].  Better Automatic Program Repair by Using Bug Reports and Tests Together.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  arXiv:2303.08774 [cs.CL]
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ 35 (2022), 27730–27744.
  * Patel et al. (2022) Ajay Patel, Bryan Li, Mohammad Sadegh Rasooli, Noah Constant, Colin Raffel, and Chris Callison-Burch. 2022.  Bidirectional Language Models Are Also Few-shot Learners.  _arXiv preprint arXiv:2209.14500_ (2022).
  * Qadir (2022) Junaid Qadir. 2022\.  Engineering education in the era of ChatGPT: Promise and pitfalls of generative AI for education.  (2022).
  * Qi et al. (2014) Yuhua Qi, Xiaoguang Mao, Yan Lei, Ziying Dai, and Chengsong Wang. 2014.  The strength of random search on automated program repair. In _Proceedings of the 36th International Conference on Software Engineering_. 254–265.
  * Qin et al. (2023) Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023\.  Is chatgpt a general-purpose natural language processing task solver?  _arXiv preprint arXiv:2302.06476_ (2023).
  * Ren et al. (2020) Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020\.  Codebleu: a method for automatic evaluation of code synthesis.  _arXiv preprint arXiv:2009.10297_ (2020).
  * Reynolds and McDonell (2021) Laria Reynolds and Kyle McDonell. 2021.  Prompt programming for large language models: Beyond the few-shot paradigm. In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–7.
  * Rolim et al. (2017) Reudismam Rolim, Gustavo Soares, Loris D’Antoni, Oleksandr Polozov, Sumit Gulwani, Rohit Gheyi, Ryo Suzuki, and Björn Hartmann. 2017\.  Learning syntactic program transformations from examples. In _2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE)_. IEEE, 404–415.
  * Rolim et al. (2018) Reudismam Rolim, Gustavo Soares, Rohit Gheyi, Titus Barik, and Loris D’Antoni. 2018.  Learning quick fixes from code repositories.  _arXiv preprint arXiv:1803.03806_ (2018).
  * Saha et al. (2017) Ripon K Saha, Yingjun Lyu, Hiroaki Yoshida, and Mukul R Prasad. 2017.  Elixir: Effective object-oriented program repair. In _2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 648–659.
  * Sobania et al. (2023) Dominik Sobania, Martin Briesch, Carol Hanna, and Justyna Petke. 2023.  An analysis of the automatic bug fixing performance of chatgpt.  _arXiv preprint arXiv:2301.08653_ (2023).
  * Surameery and Shakor (2023) Nigar M Shafiq Surameery and Mohammed Y Shakor. 2023.  Use chat gpt to solve programming bugs.  _International Journal of Information Technology & Computer Engineering (IJITC) ISSN: 2455-5290_ 3, 01 (2023), 17–22.
  * Tan and Roychoudhury (2015) Shin Hwei Tan and Abhik Roychoudhury. 2015.  relifix: Automated repair of software regressions. In _2015 IEEE/ACM 37th IEEE International Conference on Software Engineering_ , Vol. 1. IEEE, 471–482.
  * Tirumala et al. (2022) Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022.  Memorization without overfitting: Analyzing the training dynamics of large language models.  _Advances in Neural Information Processing Systems_ 35 (2022), 38274–38290.
  * Tufano et al. (2019) Michele Tufano, Jevgenija Pantiuchina, Cody Watson, Gabriele Bavota, and Denys Poshyvanyk. 2019\.  On learning meaningful code changes via neural machine translation. In _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. IEEE, 25–36.
  * Tufano et al. (2018) Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. 2018.  An empirical investigation into learning bug-fixing patches in the wild via neural machine translation. In _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_. 832–837.
  * Tufano et al. (2021) Rosalia Tufano, Luca Pascarella, Michele Tufano, Denys Poshyvanyk, and Gabriele Bavota. 2021\.  Towards automating code review activities. In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. IEEE, 163–174.
  * Wang et al. (2020) Shangwen Wang, Ming Wen, Bo Lin, Hongjun Wu, Yihao Qin, Deqing Zou, Xiaoguang Mao, and Hai Jin. 2020\.  Automated patch correctness assessment: How far are we?. In _Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering_. 968–980.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_ (2022).
  * Weimer et al. (2013) Westley Weimer, Zachary P Fry, and Stephanie Forrest. 2013\.  Leveraging program equivalence for adaptive program repair: Models and first results. In _2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 356–366.
  * Xia and Zhang (2022) Chunqiu Steven Xia and Lingming Zhang. 2022.  Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-Shot Learning. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Singapore, Singapore) _(ESEC/FSE 2022)_. Association for Computing Machinery, New York, NY, USA, 959–971.  https://doi.org/10.1145/3540250.3549101
  * Xia and Zhang (2023) Chunqiu Steven Xia and Lingming Zhang. 2023.  Conversational Automated Program Repair.  arXiv:2301.13246 [cs.SE]
  * Xiong et al. (2018) Yingfei Xiong, Xinyuan Liu, Muhan Zeng, Lu Zhang, and Gang Huang. 2018.  Identifying patch correctness in test-based program repair. In _Proceedings of the 40th international conference on software engineering_. 789–799.
  * Yu et al. (2023) Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Tao Xie, and Qianxiang Wang. 2023\.  CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models.  _arXiv preprint arXiv:2302.00288_ (2023).
  * Zhang et al. (2022) Jiyang Zhang, Sheena Panthaplackel, Pengyu Nie, Junyi Jessy Li, and Milos Gligoric. 2022\.  CoditT5: Pretraining for Source Code and Natural Language Editing. In _37th IEEE/ACM International Conference on Automated Software Engineering_. 1–12.