# Revisiting the Plastic Surgery Hypothesis
via Large Language Models

Chunqiu Steven Xia  University of Illinois Urbana-Champaign
<EMAIL_ADDRESS>,  Yifeng Ding  University of Illinois Urbana-Champaign
<EMAIL_ADDRESS>and  Lingming Zhang  University of Illinois Urbana-
Champaign<EMAIL_ADDRESS>
###### Abstract.

Automated Program Repair (APR) aspires to automatically generate patches for
an input buggy program. Traditional APR tools typically focus on specific bug
types and fixes through the use of templates, heuristics, and formal
specifications. However, these techniques are limited in terms of the bug
types and patch variety they can produce. As such, researchers have designed
various learning-based APR tools with recent work focused on directly using
Large Language Models (LLMs) for APR. While LLM-based APR tools are able to
achieve state-of-the-art performance on many repair datasets, the LLMs used
for direct repair are not fully aware of the project-specific information such
as unique variable or method names.

The _plastic surgery hypothesis_ is a well-known insight for APR, which states
that the code ingredients to fix the bug usually already exist within the same
project. Traditional APR tools have largely leveraged the plastic surgery
hypothesis by designing manual or heuristic-based approaches to exploit such
existing code ingredients. However, as recent APR research starts focusing on
LLM-based approaches, the plastic surgery hypothesis has been largely ignored.
In this paper, we ask the following question: _How useful is the plastic
surgery hypothesis in the era of LLMs?_ Interestingly, LLM-based APR presents
a unique opportunity to fully automate the plastic surgery hypothesis via
fine-tuning (training on the buggy project) and prompting (directly providing
valuable code ingredients as hints to the LLM). To this end, we propose
FitRepair, which combines the direct usage of LLMs with two domain-specific
fine-tuning strategies and one prompting strategy (via information retrieval
and static analysis) for more powerful APR.While traditional APR techniques
require intensive manual efforts in both generating patches based on the
plastic surgery hypothesis and guaranteeing patch validity, our approach is
fully automated and general.Moreover, while it is very challenging to manually
design heuristics/patterns for effectively leveraging the hypothesis, due to
the power of LLMs in code vectorization/understanding, even partial/imprecise
project-specific information can still guide LLMs in generating correct
patches! Our experiments on the widely studied Defects4j 1.2 and 2.0 datasets
show that FitRepair fixes 89 and 44 bugs (substantially outperforming the
best-performing baseline by 15 and 8), respectively, demonstrating a promising
future of the plastic surgery hypothesis in the era of LLMs.

††copyright: none

##  1\. Introduction

The increasing complexity of source code poses a key challenge to the
reliability of large-scale software systems. Software bugs in these systems
can lead to safety issues (Richards, 1990) for users around the world as well
as cause non-negligible financial losses (Matteson, 2018). As such, developers
have to spend a large amount of time and effort on bug fixing. Consequently,
Automated Program Repair (APR), designed to automatically generate patches to
fix software bugs, has attracted wide attention from both academia and
industry (Long and Rinard, 2016; Le Goues et al., 2012; Long and Rinard, 2015;
Lou et al., 2020; Tufano et al., 2018).

To achieve APR, one popular approach is known as Generate-and-Validate (G&V)
(Qi et al., 2015; Ghanbari et al., 2019; Lou et al., 2020; Le et al., 2016; Le
Goues et al., 2012; Wen et al., 2018; Hua et al., 2018; Martinez and
Monperrus, 2016; Koyuncu et al., 2020; Liu et al., 2019a, b), which is
typically based on the following pipeline: First, fault localization
techniques (Wong et al., 2016; Abreu et al., 2007; Zhang et al., 2013;
Papadakis and Le Traon, 2015; Li et al., 2019; Li and Zhang, 2017) are applied
to determine the suspicious locations in programs where bugs are likely to
exist. Then, the buggy locations are used by the APR tools to generate a list
of patches that replace buggy lines with correct lines. Afterward, each patch
is validated against the original test suite to identify any _plausible
patches_ (i.e., passing all tests in the test suite). Finally, to determine
the _correct patches_ , developers examine the list of plausible patches to
see if any of them can correctly fix the bug.

Traditional APR tools can mainly be categorized into heuristic-based (Le Goues
et al., 2012; Le et al., 2016; Wen et al., 2018), constraint-based (Mechtaev
et al., 2016; Le et al., 2017; DeMarco et al., 2014; Long and Rinard, 2015)
and template-based (Ghanbari et al., 2019; Hua et al., 2018; Martinez and
Monperrus, 2016; Liu et al., 2019a, b). Among these traditional tools,
template-based APR tools (Ghanbari et al., 2019; Liu et al., 2019a; Benton et
al., 2020) have been able to achieve state-of-the-art results. Template-based
APR tools typically leverage pre-defined templates (e.g., adding a nullness
check) for bug fixing. However, since these fix templates are typically
handcrafted, the number and types of bugs they are able to fix can be limited.

To address the limitations of traditional APR, researchers have proposed
various learning-based APR tools (Li et al., 2020; Chen et al., 2019; Jiang et
al., 2021; Lutellier et al., 2020; Zhu et al., 2021; Ye et al., 2022a) based
on the Neural Machine Translation (NMT) architecture (Sutskever et al., 2014)
where the input is the buggy code snippets and the goal is to translate the
buggy code snippets into a fixed version. To accomplish this, learning-based
APR tools require supervised training datasets with pairs of both buggy and
fixed code snippets in order to learn how to perform this translation step.
These training data are usually obtained by mining historical bug fixes using
heuristics/keywords (Dallmeier and Zimmermann, 2007), which can be imprecise
for identifying bug-fixing commits; even the actual bug-fixing commits can
include irrelevant code changes, leading to further pollution in the dataset
(Xia and Zhang, 2022). Moreover, it can be hard for such APR tools to
generalize and fix bug types unseen during training.

To better leverage recent advances in Large Language Models (LLMs),
researchers (Xia and Zhang, 2022; Xia et al., 2023; Kolak et al., 2022;
Prenner et al., 2022) have directly applied LLMs to generate patches without
bug-fixing datasets. These LLM-based APR tools work by either directly
generating a complete code function (Prenner et al., 2022; Xia et al., 2023)
or predict/infill the correct code snippet given its surrounding context (Xia
and Zhang, 2022; Xia et al., 2023). By directly using LLMs that are pre-
trained on billions of open-source code snippets, LLM-based APR tools can
achieve state-of-the-art performance on many repair datasets (Xia and Zhang,
2022).

Traditional APR tools have long used the insight of the _plastic surgery
hypothesis_ (Barr et al., 2014) where it states that the code ingredients to
fix a bug already exist within the same project. Traditional APR tools have
manually designed pattern- (Ghanbari et al., 2019; Saha et al., 2017) or
heuristic-based (Jiang et al., 2018; Le Goues et al., 2012) approaches to
finding and using such relevant code ingredients to generate fixes for bugs.
However, the plastic surgery hypothesis has been largely ignored in LLM-based
APR. In fact, LLM provides a unique opportunity to fully automate the plastic
surgery hypothesis idea via fine-tuning (learning project-specific information
via model updates from the buggy project) and prompting (directly providing
relevant code ingredients to the model), and make it directly applicable to
different languages (since the LLMs are typically multi-lingual).Moreover,
despite the intensive manual efforts involved, traditional APR tools still
cannot fully leverage project-specific information due to large search space
for leveraging/composing existing code ingredients. In contrast, the project-
specific information can effectively leveraged by LLMs due to their power in
code understanding/vectorization, e.g., even partial/imprecise information may
still guide LLMs in correct patch generation! To this end, we ask the
question: _How useful is the plastic surgery hypothesis in the era of LLMs_?

Our Work. To answer the question, we present FitRepair – a LLM-based approach
that automatically utilizes the plastic surgery hypothesis by systematically
combining multiple fine-tuning and prompting strategies for APR. FitRepair
fine-tunes LLMs using two novel domain-specific training strategies:
Knowledge-Intensified fine-tuning – we fine-tune using the original buggy
project by aggressively masking out a high percentage of tokens, which allows
LLM to learn project-specific code tokens and programming styles; and Repair-
Oriented fine-tuning – which only masks out a single continuous code sequence
per training sample, allowing the model to get used to the final cloze-style
APR task of predicting a single continuous code sequence. Furthermore, we
directly leverage the ability for LLMs to understand natural language
instructions and introduce a novel prompting strategy, Relevant-Identifier
prompting, which uses information retrieval and static analysis to obtain a
list of relevant identifiers for the buggy lines. While such relevant
identifiers are critical for fixing some difficult bugs, they may not be seen
by the LLM during inference due to limited context window size. Through the
use of prompting, we directly tell the model to use these extracted
identifiers (relevant code ingredients) to generate the correct code. Finally,
to perform repair, we combine all four model variants (including the base
model, both fine-tuned models and the base model with prompting) for the final
repair.

While our insight of leveraging the plastic surgery hypothesis for LLM-based
APR is generalizable across different types of LLMs, to implement FitRepair,
we choose a recent LLM, CodeT5 (Yue Wang and Hoi, 2021), which is pre-trained
on millions of open-source code snippets. CodeT5 is an encoder-decoder model
trained using Masked Span Prediction (MSP) objective where a percentage of
tokens are masked out and each continuous masked token sequence is referred to
as a masked span. Also, although we only extract relevant identifiers from the
current buggy project (since this paper focuses on the plastic surgery
hypothesis), our work can be easily extended to obtain other code information
(such as relevant statements or functions) from other sources, such as the
massive pre-training corpora (Husain et al., 2020) or historical bug-fixing
datasets (Jiang et al., 2019), which can provide more coding knowledge for
LLMs. Besides, although we mainly focus on using traditional string comparison
algorithms for information retrieval in this paper, these techniques can be
easily replaced by other frequency-based retrieval (Robertson et al., 2009)
and neural search (or embedding-based search) (Reimers and Gurevych, 2019). In
summary, this paper makes the following contributions:

  * •

Dimension. This paper is the first to revisit the important plastic surgery
hypothesis in the era of LLMs. It opens up a new dimension for LLM-based APR
to incorporate previously neglected information from the buggy project itself
to boost APR performance. Furthermore, it demonstrates the promising future of
retrieval-based prompting for modern LLM-based APR.

  * •

Implementation. We implement FitRepair based on the recent CodeT5 model. We
augment the model using two novel fine-tuning strategies: Knowledge-
Intensified fine-tuning and Repair-Oriented fine-tuning, along with a novel
prompting strategy based on information retrieval and static analysis:
Relevant-Identifier prompting. We combine the patches generated by all four
models together and perform patch ranking to speed up APR.

  * •

Evaluation Study. We conduct an extensive evaluation against state-of-the-art
APR tools. On the widely studied Defects4j 1.2 and 2.0 datasets (Just et al.,
2014), FitRepair is able to achieve the new state-of-the-art results of 89 and
44 correct bug fixes (15 and 8 more than best baseline) respectively.
Furthermore, we perform a broad ablation study to justify our design.
FitRepair demonstrates for the first time that the plastic surgery hypothesis
can substantially boost LLM-based APR and advance state-of-the-art APR, while
being fully automated and general. Moreover, even partial/imprecise code
ingredients may still effectively guide LLMs for APR!

##  2\. Background

###  2.1. Large Language Model

Figure 1. LLM overview

Large Language Models (LLMs) trained on large amounts of text combined with
code snippets from a broad range of open-source repositories have shown
impressive progress in various code-related tasks (Chen et al., 2021; Fried et
al., 2022; Xu et al., 2022). To apply them to code, researchers may further
train the models on open-source code repositories. On top of that, LLMs can be
_fine-tuned_ (updating pre-trained model parameters by further training on the
downstream tasks (Devlin et al., 2018)) to target a specific task such as
defect prediction (Omri and Sinz, 2020) or code clone detection (Wang et al.,
2022). Different from fine-tuning, _prompting_ is a way to directly use LLMs
on downstream tasks without further training. Prompting involves providing a
natural language description of the task and optionally including a few
demonstrations of the task to the LLMs before the actual input. Researchers
have successfully applied prompting to tasks like code completion (Nijkamp et
al., 2022) and code summarization (Kuznia et al., 2022).

LLMs are based on the popular Transformer architecture (Vaswani et al., 2017),
which combines an encoder with a decoder to perform text generation. The
encoder first takes in the input to the model and then produces an encoded
representation. The decoder uses this encoded vector to autoregressively
generate the next token based on all previously generated tokens. Using this
paradigm, researchers build larger and larger models (as large as 540B in the
number of model parameters (Chowdhery et al., 2022)) and demonstrated
impressive results on code-related tasks such as code completion (Chen et al.,
2021) and code synthesis (Austin et al., 2021).

LLMs can be classified into three groups based on their model architecture and
pre-training objective: Decoder-only (Left-to-Right Language Models), Encoder-
only (Masked Language Models), and Encoder-decoder models. Figure 1 shows an
overview of the three different LLM architectures. Decoder-only models perform
left-to-right generation by producing the probability of a token given all
previous tokens. One of the most well-known LLMs, GPT (Radford et al., 2019;
Brown et al., 2020), is based on this architecture. During training, decoder-
only models aim to predict the next token given all previous context. Examples
of decoder-only models for code are CodeGPT (Lu et al., 2021), CodeParrot
(Tunstall et al., 2022), and Codex (Chen et al., 2021). These models can be
directly used for program generation given previous code contexts. Encoder-
only models, on the other hand, only use the encoder component to provide an
encoded representation of the input. Models such as BERT (Devlin et al., 2018)
are trained using the Masked Language Model (MLM) objective, where a small
percentage (e.g., 15%) of the training tokens are masked out and the model
aims to recover these masked tokens using the bi-directional context. CodeBERT
(Feng et al., 2020), GraphCodeBERT (Guo et al., 2021), and CuBERT (Kanade et
al., 2020) are examples of encoder-only models where it can provide a
representation of the input code to be used for downstream tasks such as code
clone detection (Wang et al., 2022). Encoder-decoder models (T5 (Raffel et
al., 2020), BART (Lewis et al., 2019)) use both components of the transformer
and are typically trained using Masked Span Prediction (MSP) objective.
Different from MLM, instead of masking out individual tokens, MSP replaces a
sequence of tokens with a single span mask. The goal of the training is to
recover the original sequence using both the context before and after the span
mask token. For code tasks, CodeT5 (Yue Wang and Hoi, 2021) and PLBART (Ahmad
et al., 2021) are examples of encoder-decoder models and due to the MSP pre-
training objective, they can be directly used to fill in arbitrary code
snippets given the bi-directional code context.

###  2.2. Automated Program Repair

Automated Program Repair (APR) works by automatically generating patches when
given the buggy project and potential fault locations. Traditional APR tools
can be categorized into constraint-based (Mechtaev et al., 2016; Le et al.,
2017; DeMarco et al., 2014; Long and Rinard, 2015), heuristic-based (Le Goues
et al., 2012; Le et al., 2016; Wen et al., 2018), and template-based (Ghanbari
et al., 2019; Hua et al., 2018; Martinez and Monperrus, 2016; Liu et al.,
2019a, b) tools. Among those, template-based APR has been regarded as the
state-of-the-art in achieving the best repair performance (Ghanbari et al.,
2019; Liu et al., 2019a; Benton et al., 2020). Template-based APR works by
using pre-defined templates (handcrafted by human experts) which target
specific patterns in source code. Each template will have an associated fix
that modifies the found patterns in the source code to fix specific types of
bugs. However, template-based APR tools cannot fix bugs that do not fall under
the pre-defined templates. As a result, template-based tools lack the ability
to generalize to unseen bug types.

In recent years, researchers have begun to focus on learning-based APR
approaches such as SelfAPR (Ye et al., 2022b), RewardRepair (Ye et al.,
2022a), Recoder (Zhu et al., 2021), CURE (Jiang et al., 2021), and CoCoNuT
(Lutellier et al., 2020) based on the Neural Machine Translation (NMT)
(Sutskever et al., 2014) architecture. The goal of these tools is to learn a
transformation using DL models that turns buggy code snippets into patched
ones. To facilitate this, these tools require further training on specific
bug-fixing datasets containing pairs of buggy and fixed code snippets.
However, these bug-fixing datasets are usually scraped from open-source bug-
fixing commits using handwritten heuristics such as keyword searching (Zhu et
al., 2021; Lutellier et al., 2020; Dallmeier and Zimmermann, 2007; Jiang et
al., 2019), which can include irrelevant code commits; even the correctly
identified bug-fixing commits may contain various irrelevant code changes
(such as refactoring or new feature implementation), introducing various
noises in the datasets. Also, to avoid including bug-fixing commits with
irrelevant code changes, existing learning-based APR techniques will limit the
commits to ones with few lines of changes (Jiang et al., 2021; Zhu et al.,
2021; Lutellier et al., 2020), further limiting the amount of training data.
Moreover, NMT-based APR tools may still not generalize to specific code or bug
types that are not seen inside of the (limited) bug-fixing datasets.

Recognizing these limitations in NMT-based APR, researchers have proposed LLM-
based APR tools (Xia and Zhang, 2022; Xia et al., 2023; Prenner et al., 2022;
Kolak et al., 2022) which do not require bug-fixing datasets by directly using
LLMs for APR. AlphaRepair (Xia and Zhang, 2022) first introduced the cloze-
style APR (or infilling-style APR) to directly leverage LLMs in a zero-shot
manner to fill in the code given the context before and after the buggy line.
AlphaRepair first generates repair inputs that replace the original buggy line
with masked tokens and uses the CodeBERT model (Feng et al., 2020) to directly
recover the correct code to fill in each masked token. Later studies (Xia et
al., 2023; Kolak et al., 2022; Prenner et al., 2022) also used different LLMs
(including Decoder-only and Encoder-decoder models) to not only perform cloze-
style APR but also repair scenarios where a complete fixed function is
generated. Contrasting with NMT-based APR tools, LLM-based APR leverages the
pre-training objectives of LLMs which can directly learn the relationship
between correct code and its context without relying on historical bug-fixing
commits. As a result, LLM-based APR tools have shown to achieve state-of-the-
art performance on repair tasks across multiple programming languages (Xia et
al., 2023). In this work, we present the first work to further advance state-
of-the-art LLM-based APR with the insight of the plastic surgery hypothesis.

##  3\. Approach

Figure 2. Cloze-style APR  Figure 3. FitRepair overview

In this section, we describe our approach to incorporate the plastic surgery
hypothesis for LLM-based APR. While our overall idea is general for LLM-based
APR approaches, we mainly focus on cloze-style APR since it has been
demonstrated to be the state-of-the-art for LLM-based APR (Xia et al., 2023).
Figure 2 shows the overview of cloze-style APR where we aim to directly
generate the correct code in place given the original context, where the model
has to ”fill in the blanks” of the missing code line/hunk (represented by
<SPAN>) given the buggy context. However, prior cloze-style APR tools largely
ignore the rich project-specific and bug-specific information which has been
demonstrated by the plastic surgery hypothesis (Barr et al., 2014; Le Goues et
al., 2012) to be critical in helping to generate a correct fix. LLM presents
an exciting opportunity to automatically leverage the idea of the plastic
surgery hypothesis via its powerful ability to both directly learn from the
buggy project (fine-tuning) and use relevant code context clues (prompting) to
generate correct fixes. In this work, we explore using LLMs to automatically
capture project-specific information via both fine-tuning and prompting.

We propose a novel approach – FitRepair to combine the direct usage of LLM in
cloze-style APR with knowledge gained from the plastic surgery hypothesis.
FitRepair first trains two separate models with two novel fine-tuning
strategies: 1) Knowledge-Intensified fine-tuning – we use the source files of
the original buggy project to construct a similar dataset to pre-training by
aggressively masking out large portions (50%) of the code tokens, which allows
LLM to learn project-specific code tokens and programming styles; and2)
Repair-Oriented fine-tuning – we fine-tune another model using the original
buggy project to construct a more repair-oriented dataset by masking out only
a single continuous code sequence per training sample, which makes the fine-
tuned model become more prepared for the repair task where only a single
continuous code sequence needs to be generated. Additionally, we propose a
novel prompting strategy, 3) Relevant-Identifier prompting, by obtaining a
list of relevant/rare identifiers that are not seen by the model in its
immediate context using information retrieval and static analysis.

While our approach can be extended to different LLMs, in this paper, we use
CodeT5 (Yue Wang and Hoi, 2021), an encoder-decoder LLM for code trained using
Masked Span Prediction (MSP) objective. MSP replaces continuous tokens with a
single masked span token (<SPAN>) and the pre-training task is to recover
masked-out code sequences given the surrounding context. Given a sequence of
tokens $X=\\{x_{1},...,x_{n}\\}$, random sequences of tokens are replaced with
a masked span token to produce
$X_{masked}=\\{x_{1},...,{\small\texttt{<SPAN>}},x_{n}\\}$. Let
$M=\\{m_{1},...,m_{k}\\}$ be the tokens masked out,
$M_{<g}=\\{m_{1},m_{2},...,m_{g-1}\\}$ be token sequence predicted so far
where $g\leq k$, $P$ be the predictor (model) which outputs the probability of
a token. The MSP loss function can be described as:

(1)
$\mathcal{L}_{MSP}=-\frac{1}{k}\sum_{i=1}^{k}log\;(P\;(m_{i}\;|\;X_{masked},\;M_{<i})$

To employ CodeT5 for cloze-style APR using MSP, we follow previous work (Xia
and Zhang, 2022) and use repair templates to generate mask lines where we
replace the entire or parts of the buggy line with a single masked span token.
We then use CodeT5 to generate the correct code to replace the masked span and
create a patch for the bug. Figure 3 shows the overview of our approach:

  * •

1

(Section 3.1): We use Knowledge-Intensified fine-tuning to build a training
dataset by extracting functions from buggy project source code. We fine-tune
the original CodeT5 model by first using a high masking rate to aggressively
learn project-specific tokens.

  * •

2

(Section 3.2): We use Repair-Oriented fine-tuning strategy to fine-tune
another model by constructing another training dataset from the buggy project
where only a single or partial code line is masked out based on the repair
templates. We train the model until convergence and obtain the Repair-Oriented
fine-tuning model.

  * •

3

(Section 3.3): We use Relevant-Identifier prompting strategy to extract
relevant identifiers via information retrieval and static analysis. We then
create individual prompts with instructions for the model to use the extracted
identifiers during patch generation.

  * •

4

(Section 3.4): We perform cloze-style APR by using the repair templates from
previous work (Xia and Zhang, 2022) and generate patches by separately using
the 4 models (original CodeT5, Knowledge-Intensified fine-tuning model,
Repair-Oriented fine-tuning model, and original CodeT5 with prompting). Each
patch generated is then validated against the test suite to find the list of
plausible patches.

###  3.1. Knowledge-Intensified Fine-tuning

Figure 4. Knowledge-Intensified fine-tuning overview

To facilitate the learning of project-specific information, we use Knowledge-
Intensified fine-tuning by constructing a training dataset using the buggy
project itself. Figure 4 shows the Knowledge-Intensified fine-tuning process.
We first extract the source code functions from the project code base where
the bug is from and apply MSP objective – masking out multiple spans of code
tokens, used to pre-train the original CodeT5 model. Traditionally, MSP
objective will mask out only a small portion of the original code tokens
(e.g., 15% (Yue Wang and Hoi, 2021)). However, in this step, we employ a much
higher masking rate (50%) which means the model is tasked with recovering more
masked-out code tokens with less context. This approach is motivated by a
recent study (Wettig et al., 2022) on LLMs for natural languages where better
representation and downstream performance can be achieved by increasing the
pre-training masking rate of MLM and MSP objectives. The study found that
higher masking rates make the learning tasks more challenging and can force
the model to learn more aggressively, which helps improve the performance of
LLMs on various downstream tasks. In this paper, we leverage this idea of more
aggressive training to force the model to learn more project-specific
knowledge by trying to recover more code tokens given limited context.

However, one limitation with fine-tuning the model on the buggy project itself
is the relatively small number of training samples (e.g., thousands of
functions) especially compared with the large amount of open-source pre-
training data used in CodeT5 (millions of functions). As a result, we reapply
the MSP objective across iterations to augment the fine-tuning dataset with
more training samples. Following the example in Figure 4, we start by creating
one set of training data by masking out 50% of the training tokens to create
masked spans. In the next iteration, we reapply this masking strategy to
create a new set of training data by randomly choosing another 50% to mask out
again. In this process, we essentially create new training data for each
subsequent iteration. While the number of tokens masked out is the same, the
specific masked locations can be different which provides further augmentation
on the training dataset allowing the model to learn more project-specific
tokens.

During the fine-tuning process when using Knowledge-Intensified fine-tuning,
FitRepair is able to learn project-specific knowledge such as commonly used
methods or variables that are specifically defined in the current project.
These pieces of project-specific knowledge are especially important for repair
as many bugs can be fixed by applying code snippets found in other parts of
the source file or project, according to the plastic surgery hypothesis (Barr
et al., 2014). Due to the limited context window size (e.g., 512 tokens for
CodeT5), CodeT5 cannot encode all of the surrounding contexts during
inferencing, which leads to the base CodeT5 model missing variable names and
method calls used in other parts of the context that are actually necessary to
be used as part of the patch. Knowledge-Intensified fine-tuning can partially
alleviate this by learning these missing variable names and method calls as
part of the fine-tuning such that when used for cloze-style APR, the fine-
tuned model can predict these useful tokens with a higher probability compared
to the model without fine-tuning. While Knowledge-Intensified fine-tuning can
help with better learning of project-specific details, we next introduce
Repair-Oriented fine-tuning which produces another model that aims to optimize
for the repair task.

###  3.2. Repair-Oriented Fine-tuning

Figure 5. Repair-Oriented fine-tuning overview

In the previous step, we use Knowledge-Intensified fine-tuning to generate
more code that uses project-specific variables, method calls, and structures.
Cloze-style APR frames the problem of patch generation by asking the model to
fill in the correct code given the buggy context. This is achieved by
exploiting the similarity between the pre-training objective and final
inference setup to generate patches. Furthermore, cloze-style APR is not
limited to only generating a complete line but additionally using repair
templates (e.g. replaces only method call name) which keeps part of the buggy
line to create patches by generating partial lines (Xia and Zhang, 2022).
However, both the Knowledge-Intensified fine-tuning CodeT5 and the original
CodeT5 suffer from the same limitation: the training process is not designed
for repair. Both the original pre-training and Knowledge-Intensified fine-
tuning use MSP which masks out multiple disjointed code token spans. The goal
of the model during training is to recover the original tokens for all the
masked spans. However, for cloze-style APR, only a single code line or a part
of the code line is usually masked out and the model only has to predict the
correct code for that single span.

To address such limitations, we use Repair-Oriented fine-tuning which fine-
tunes with a training setup that is similar to the repair inference task.
Figure 5 shows the detailed Repair-Oriented fine-tuning process. We again
first use the original buggy project as the source of our training data by
extracting source code functions. We pick a single code line in each training
sample to mask out with a single span token. One can think of this chosen code
line as the buggy line in the final repair scenario. To model the impact of
eventually using template-based repair inputs, we randomly select a repair
template that can be applied on this line to mask out only a part of the line.
These repair templates are taken directly from previous cloze-style APR work
(Xia and Zhang, 2022) and can be categorized into 3 different types of repair
templates: 1) Complete mask – replace the entire buggy line with a single span
token or add the span token to before/after the buggy line, 2) Partial mask –
keep some original buggy line tokens at the end or beginning of the line and
replace the rest with a span token, and 3) Template mask – target specific
code line types by replacing the method call, method parameters, and Boolean
expression or operator with a span token.In this fine-tuning process, our
training samples are closely similar in their setup compared with the cloze-
style APR task that we want the model to perform. Using Repair-Oriented fine-
tuning, we can produce a fine-tuned model which aims to follow closely with
the final repair task.

###  3.3. Relevant-Identifier Prompting

Two previous fine-tuning strategies aim to fine-tune the model towards
generating more project-specific tokens and get used to repair-oriented
inputs, in both cases, using the original buggy project as the fine-tuning
dataset. However, this means that the two fine-tuned models have been geared
toward the entire buggy project rather than the specific bug within the
project. For specific bugs, the relevant code ingredients may be drastically
different depending on the bug file, location, and type of buggy line. These
ingredients can be possibly far away from buggy locations, making it hard for
them to be included in the input context due to the limited context window
size of LLMs (e.g. the limit of context window size of CodeT5 is 512 tokens).

We use Relevant-Identifier prompting on the base CodeT5 model to directly
prompt the model with relevant code ingredients. Algorithm 1 shows the
overview of our prompting strategy. Given the buggy line information, we first
extract the certain file containing the bug and separate it into individual
code lines (Line 1). Prior work has found that a significant percentage of the
correct code to fix the bug can be found within the same file (Barr et al.,
2014). We then use Levenshtein Distance Ratio (Levenshtein et al., 1966)
(other string comparison methods (Ratcliff and Metzener, 1988; Jaro, 1989;
Winkler, 1990) provide similar results) to measure the similarity between each
line compared with the buggy line (Line 4). The hypothesis is that useful
identifiers can be obtained from lines that are very similar to the buggy line
(Asad et al., 2019). We rank each code line based on its string similarity
score from high to low to get a ranked list of code lines (Line 5). Since we
want to provide the model with identifiers to help generate the correct fix,
we extract identifiers from each line (Line 7), which provides us with a
ranked list of identifiers. After that, we perform further filtering by first
removing any common/simple identifiers (e.g., length and node) (Line 8) and
then using static analysis (Line 9) to remove any identifiers that are
unaccessible within the buggy method (Line 10). Next, we extract the useful
type information for each identifier to indicate the type/return type and if
it is a method invocation or a variable (Line 11). Finally, we obtain a ranked
list of complex identifiers that come from similar lines within the same file.

Using the ranked identifier list, we then generate the prompts to instruct the
model to use these extracted identifiers to generate patches (Line 12). LLMs
are able to understand natural language instructions in the form of prompts to
perform specific tasks. In Relevant-Identifier prompting, we construct a
prompt in the form of /* use {} in the next line */ where we replace {} with
an identifier with type information (e.g., (Plot) getParent()). This prompt is
then appended before the masked span token during inference which allows the
model to directly use this identifier information provided in its generation.
Since we have a ranked list of identifiers, we generate multiple unique
prompts, each including one of the top $N$ highest-ranked identifiers. By
directly providing these extracted bug-specific identifiers in prompts, the
model can use these identifiers which previously are outside of its immediate
context to generate the correct fix.

Algorithm 1 Relevant-Identifier Prompting Strategy

0: Buggy project $Proj$, Buggy file $File$, Buggy line $buggy\\_line$.

0: Relevant-Identifier $prompts$.

1: $lines$ $:=$ ExtractLines ($File$)

2: $similarities,\;identifiers$ $:=$ $[],[]$

3: for $line$ in $lines$ do

4: $similarities$.append (LevenshteinRatio ($buggy\\_line$, $line$))

5: $lines\\_ranked$ $:=$ RankLines ($lines$, $similarities$)

6: for $line$ in $lines\\_ranked$ do

7: $line\\_identifiers:=$ ExtractIds ($line$)

8: $identifiers$.extend (SimpleFilter ($line\\_identifiers$))

9: $accessibles$ $:=$ FindAccessibleIds ($Proj$, $File$, $buggy\\_line$)

10: $relevants:=identifiers\;\cap\;accessibles$

11: $type\\_infos$ $:=$ FindTypeInfo ($Proj$, $relevants$)

12: $prompts:=$ BuildPrompts ($relevants$, $type\\_infos$)

###  3.4. Patch Generation, Ranking, and Validation

We directly use the base CodeT5 model (with and without prompting) and the two
fine-tuned models generated from previous steps for patch generation. To
generate patches to replace the buggy lines, we apply repair template inputs
from previous work (Xia and Zhang, 2022) and ask each model to fill in the
masked-out span token with generated correct code. Following prior work
(Prenner et al., 2022; Xia et al., 2023; Kolak et al., 2022), we sample each
model in parallel to generate its own set of patches. In total, for each bug,
we generate four lists of potential patches using the four models.

The rankings of patches are computed based on the outputs of each model. We
follow the same process as previous work (Xia and Zhang, 2022) and compute the
entropy score using the model. For each candidate patch, we want to provide a
likelihood score (entropy) that can accurately reveal the extent to which
CodeT5 will generate this patch. Let $T=\\{t_{1},t_{2},...,t_{n}\\}$ be the
list of tokens generated for a patch and $C(t_{i})$ be the probability of
generating token $t_{i}$ according to CodeT5, then the likelihood score is
defined as: $score(T)=\frac{1}{n}\sum_{i=1}^{n}\log\;(C(t_{i}))$.

We compute this likelihood score for all the patches generated across all the
templates and re-rank patches from the highest score to the lowest score. We
then validate each candidate patch in accordance with the ranking results.
Since each model (two fine-tuned models, base CodeT5 with and without
prompting) generates its own separate list of patches, we also perform the
patch validation in parallel. As such, we can reduce the patch validation time
by stopping after any one of the models found a correct patch according to
manual inspection by developers. For one bug, FitRepair aims to produce a list
of plausible patches that pass the entire test suite and a correct patch that
correctly fixes the underlying bug. Since developers can stop this validation
process whenever they find a patch to be the correct fix, the overall correct
patch ranking is the _minimum_ ranking of the correct patches in the ranked
patch lists produced by the four models.

##  4\. Experimental Design

###  4.1. Research Questions

In this paper, we study the following research questions:

  * •

RQ1: How does FitRepair compare against the state-of-the-art APR tools?

  * •

RQ2: What is the impact of different configurations of FitRepair?

  * •

RQ3: How does FitRepair generalize in fixing additional bugs from different
projects?

We first demonstrate the repair effectiveness of FitRepair against state-of-
the-art APR tools on the popular Defects4j 1.2 (Just et al., 2014) dataset. We
study not only the number of bugs fixed in total but also the number of unique
bugs fixed compared with previous techniques. Furthermore, we analyze the
improvement in terms of patch ranking – to validate correct patches faster
when using FitRepair. Next, we conduct an extensive ablation study on the
different configurations of both our two fine-tuning strategies and one
prompting strategy. Due to the time cost to train multiple models and generate
patches, for the ablation study, we focus on the Closure project in Defects4j
1.2 which is the largest in terms of both the number of bugs and the size of
source code base. Finally, following prior work (Xia and Zhang, 2022; Xia et
al., 2023), we evaluate against the state-of-the-art APR tools on the
Defects4j 2.0 (Just et al., 2014) dataset to illustrate that FitRepair is not
simply overfitting to the 1.2 version.

###  4.2. Implementation

FitRepair is implemented in Python using the PyTorch (PyTorch, 2022)
implementation of the CodeT5 model from Hugging Face (HuggingFace, 2022). Our
fine-tuning method is based on the pre-trained CodeT5-large (770M) checkpoint.
We use JavaParser (JavaParser, 2023) to perform static analysis of filtering
inaccessible identifiers in scope. For both Knowledge-Intensified fine-tuning
and Repair-Oriented fine-tuning, we repeat the fine-tuning process for 10
iterations by default to augment our fine-tuning dataset. We fine-tune the
CodeT5 model on an NVIDIA RTX A6000 with 48GB memory using FP16. We use the
following set of hyper-parameters to train models: 32 batch size and 1e-4
learning rate with 15K training steps. We use Adam optimizer (Kingma and Ba,
2014) to update the parameters and use a linear learning rate scheduler with
10% warmup proportion. For both fine-tuning strategies, we extract the oldest
version of the buggy project for training and use the fine-tuned models to
generate patches for all bugs in that project. For Relevant-Identifier
prompting, we use the top 5 most relevant identifiers. For repair, we sample
each model 5000 times and validate the top 1000 unique patches produced by
each model (at most 4000 patches in total per bug) which is comparable to
other baselines. To generate more unique patches for each sample, we use
nucleus sampling with top $p$ of 1 and temperature of 1. We validate the
patches on a workstation using AMD Ryzen Threadripper PRO 3975WX CPU with
32-Cores and 256 GB RAM, running Ubuntu 20.04.5 LTS. Similar to prior work
(Xia and Zhang, 2022; Jiang et al., 2021; Li et al., 2020), we use an end-to-
end time limit of 5 hours to fix one bug. Note that we sum up the time spent
on each of our four processes as FitRepair time cost for fair comparison.

###  4.3. Subject Systems

We use the widely studied benchmark of Defects4j (Just et al., 2014) – a
collection of open-source bugs found across 15 different projects to evaluate
FitRepair. We follow prior work and separate the dataset into Defects4j 1.2
and Defects4j 2.0. Defects4j 1.2 contains 391 bugs (removing 4 depreciated
bugs) across 6 different projects and Defects4j 2.0 contains 438 bugs across 9
additional projects. While evaluating FitRepair on all 391 bugs in Defects4j
1.2, we follow prior work (Xia and Zhang, 2022) and choose only the 82 single-
line bugs in Defects4j 2.0 for evaluation (since existing learning-based APR
mainly target single-line fixes).

###  4.4. Compared Techniques

We compare FitRepair against 20 different APR tools including both state-of-
the-art learning-based and traditional APR tools. We choose 8 recent learning-
based APR tools for comparison: AlphaRepair (Xia and Zhang, 2022), SelfAPR (Ye
et al., 2022b), RewardRepair (Ye et al., 2022a), Recoder (Zhu et al., 2021),
CURE (Jiang et al., 2021), CoCoNuT (Lutellier et al., 2020), DLFix (Li et al.,
2020) and SequenceR (Chen et al., 2019). AlphaRepair is a recently proposed
and state-of-the-art cloze-style APR tool that directly uses a pre-trained
model with accessible training set (CodeBERT). Additionally, we also compare
against 12 representative traditional APR tools: TBar (Liu et al., 2019a),
PraPR (Ghanbari et al., 2019), AVATAR (Liu et al., 2019b), SimFix (Jiang et
al., 2018), FixMiner (Koyuncu et al., 2020), CapGen (Wen et al., 2018), JAID
(Chen et al., 2017), SketchFix (Hua et al., 2018), NOPOL (DeMarco et al.,
2014), jGenProg (Martinez et al., 2015), jMutRepair (Martinez and Monperrus,
2016), and jKali (Martinez and Monperrus, 2016). Finally, since FitRepair
proposes to combine the base CodeT5 (with and without prompting) with the two
fine-tuned models, we also compare against a baseline where we run the base
CodeT5 four times with four random different seeds. This is a fair and
necessary baseline to compare against as CodeT5 can produce different sampling
outputs depending on the random seed and a developer who wishes to use our
approach of combining the four models together may also allocate the same GPU
resource to run CodeT5 four times as well. We refer to this baseline in our
evaluation as CodeT5$\times$4.

We evaluate against these baselines on perfect fault localization setting,
where the ground-truth location of each bug is provided to the repair tool by
comparing the reference developer patch with the buggy code. This is the
preferred evaluation setting (Lutellier et al., 2020; Jiang et al., 2021; Zhu
et al., 2021; Tufano et al., 2018) as it eliminates any result differences
caused by using different fault localization techniques (Wong et al., 2016).
We use the standard metrics for APR comparison of plausible patches – pass the
entire test suite and correct patches – semantically equivalent to the
reference developer fix. Following all prior APR work, correct patches are
determined by manually inspecting each plausible patch. Also, following common
practice in APR, we directly report the number of correct and plausible bug
fix results from previous studies (Xia and Zhang, 2022; Ghanbari et al., 2019;
Liu et al., 2019a; Zhu et al., 2021; Ye et al., 2022a).

##  5\. Result Analysis

###  5.1. RQ1: Comparison with State-of-the-art

Table 1. Evaluation results of correct fixes on Defects4j 1.2

Project | FitRepair | CodeT5$\times$4 | AlphaRepair | SelfAPR | RewardRepair | Recoder | TBar | CURE | CoCoNuT | PraPR | DLFix
---|---|---|---|---|---|---|---|---|---|---|---
Chart | 8 | 8 | 9 | 7 | 5 | 10 | 11 | 10 | 7 | 7 | 5
Closure | 29 | 23 | 23 | 19 | 15 | 21 | 16 | 14 | 9 | 12 | 11
Lang | 19 | 18 | 13 | 10 | 7 | 11 | 13 | 9 | 7 | 6 | 8
Math | 24 | 23 | 21 | 22 | 19 | 18 | 22 | 19 | 16 | 10 | 13
Mockito | 6 | 5 | 5 | 3 | 3 | 2 | 3 | 4 | 4 | 3 | 1
Time | 3 | 3 | 3 | 3 | 1 | 3 | 3 | 1 | 1 | 3 | 2
Total | 89 | 80 | 74 | 64 | 50 | 65 | 68 | 57 | 44 | 41 | 40

####  5.1.1. Bugs fixed

We first compare FitRepair against both traditional and learning-based APR
tools on Defects4j 1.2. Table 1 shows the number of bugs that can be fixed
with correct patches by FitRepair and the top baseline tools. In addition to
FitRepair, we also include the result of running the base CodeT5 separately
four times using four different seeds (Column CodeT5$\times$4). Compared with
CodeT5$\times$4, we observe that our fine-tuned models and prompting strategy
is able to provide additional fixes, boosting the number of correct bug fixes
from 80 to 89. In total, FitRepair is able to achieve 89 correct bug fixes on
Defects4j 1.2 with 15 more fixes than the current state-of-the-art APR tool.
Figure 6a shows the number of unique bug fixes (that only one technique can
exclusively fix while others cannot) generated by FitRepair compared with the
top performing APR baselines and all other tools (Other). We observe that even
compared with all previous APR approaches, FitRepair is able to _provide 16
additional unique fixes that no other APR tools have been able to fix so far
on Defects4j 1.2_.

Figure 6. Correct fix Venn diagram on Defects4j 1.2 Figure 7. Relevant-
Identifier prompting unique patches

To illustrate the ability of FitRepair, we show an example fix on a bug
(Closure-71) in Figure 7a which cannot be fixed by any previous tools. The fix
is to invoke the method of getJSDocInfo() which is not only an uncommon method
name but also is not seen/used in the surrounding context of the bug. However,
this method has been used within the same file as the buggy code where another
function initializes a variable called overridingInfo also using
getJSDocInfo(). Relevant-Identifier prompting is able to recognize the
similarity between the buggy variable name (isOverride) and this line to
extract the relevant identifier of getJSDocInfo() and provide the prompt to
tell the model to directly use this identifier to generate the correct patch.
This example showcases the plastic surgery hypothesis where patches can often
be constructed via reusing code snippets/ingredients from other parts of the
project. FitRepair directly leverages this hypothesis by extracting the
relevant identifiers from similar code lines within the current file and
providing it via natural language prompting to generate the correct fix.

Another very interesting example bug (Mockito-27) fixed by FitRepair that
cannot be fixed by previous tools is in Figure 7b. This bug is fixed by
calling the MethodInterceptorFilter constructor with a previous setting
obtained using getMockSetting(). We observe that while the Relevant-Identifier
prompting did not get the exact identifier of getMockSetting, it was able to
gather a closely related identifier of getMockHandler. Due to their power in
code understanding/vectorization, LLMs do not have to always generate a patch
containing the exact identifier in the prompt and can often just use it as
hints or partial code for generation. This example further highlights the
unique effect the plastic surgery hypothesis have on LLM-based APR where the
extracted code ingredients do not have to be exactly correct and can serve as
guidance for the model to generate the correct patch.

Table 2. Average correct patch rank on Defects4j 1.2

Project | Chart | Closure | Lang | Math | Mockito | Time | Average
---|---|---|---|---|---|---|---
CodeT5$\times$4 | 221 | 256 | 100 | 142 | 56 | 342 | 186
FitRepair | 26 | 94 | 76 | 145 | 89 | 260 | 115
Improvement | 88% | 63% | 24% | -2% | -59% | 24% | 38%

####  5.1.2. Individual strategy effect

Figure 8. Knowledge-Intensified fine-tuning unique patch

Figure 6b shows the unique bugs fixed by each of the individual strategies in
FitRepair. We observe that our three strategies are all able to contribute to
providing unique fixes compared with the base CodeT5 model (4 from Knowledge-
Intensified fine-tuning, 2 from Repair-Oriented fine-tuning and 6 from
Relevant-Identifier prompting) and boost the overall FitRepair to achieve 89
correct patches. Interestingly, each single strategy is already a strong APR
tool, e.g., the Relevant-Identifier prompting strategy can fix 79 bugs by
itself, already outperforming all existing tools. This demonstrates the
ability of our three strategies to provide additional fixes that directly
applying the base CodeT5 cannot provide. Moreover, the base CodeT5 model
without any changes also produced 2 unique fixes, demonstrating the
usefulness/necessity of applying the base model to cover the bugs that may
only require general-purpose correct code knowledge.

Since Section 5.1.1 has shown unique fixes obtained by Relevant-Identifier
prompting, we now present an example bug fixed by Repair-Oriented fine-tuning.
Figure 8 shows a correct fix example that base CodeT5 model cannot fix on
Closure-123. For this bug, the correct fix is to initialize the variable
rhsContext by calling a function getContextForNoInOperator(). What makes this
bug difficult to fix for previous APR tools is that getContextForNoInOperator
is a very hard sequence to generate. First, it is not a commonly used function
name such as getContext. Second, there are no code snippets using this
function in the immediate context. As such, previous techniques may fail to
generate this patch as it requires specific knowledge about the buggy project
in order to come up with this function name. However, this function is used
multiple times within the buggy project (in other functions and files).
FitRepair leverages this by using Knowledge-Intensified fine-tuning strategy
to fine-tune a model to predict masked-out tokens within the buggy project.
During Knowledge-Intensified fine-tuning, the model can learn the usage of
this specific function within the buggy project and apply it in this case to
produce the correct patch. This domain-specific knowledge cannot be learned
just from pre-training on a large amount of open-source code (previous cloze-
style APR tools).

####  5.1.3. Patch ranking

We examine the ability of FitRepair to perform patch ranking in order to
prioritize faster validation for correct patches. Similarly, we compare
FitRepair against the baseline of running base CodeT5 four times with
different seeds.Table 2 shows the average rank of the correct patch for the
Defects4j 1.2 projects on the same set of bugs both FitRepair and
CodeT5$\times$4 can fix. We observe that in four out of the six projects,
FitRepair provides a better rank on average for the correct patches. On
average, using FitRepair, we can achieve a 38% reduction in the ranking of
correct patches. FitRepair can learn/use project-specific information to rank
correct patches higher since the correct patches often use project-specific
identifiers which are less prioritized by the base CodeT5 model. In this way,
FitRepair not only fixes more bugs, but can also find the correct fixes faster
and reduce the computation cost needed for patch validation.

###  5.2. RQ2: Detailed Ablation Study

####  5.2.1. Impacts of Knowledge-Intensified fine-tuning

Table 3. Repetition for Knowledge-Intensified fine-tuning

Strategy |  #Corr. / #Plaus. (All) |  #Corr./#Plaus. (New) |  Comp. Error pct |  #Unique comp. per bug
---|---|---|---|---
Repetitive (default) |  15 / 30 |  2 / 3 | 82% | 138
Non-Repetitive | 14 / 25 | 1 / 2 | 79% | 104

Table 4. Masking rates of Knowledge-Intensified fine-tuning

Mask Rate |  #Corr. / #Plaus. (All) |  #Corr./#Plaus. (New) |  Comp. Error pct |  #Unique comp. per bug
---|---|---|---|---
10% |  16 / 23 |  2 / 2 | 83% | 79
20% | 14 / 27 |  2 / 4 | 88% | 75
30% | 14 / 30 |  2 / 4 | 87% | 92
40% |  15 / 29 |  2 / 4 | 85% | 102
50% (default) |  15 / 30 |  2 / 3 | 82% | 138
60% | 13 / 26 | 1 / 4 | 87% | 106
70% | 12 / 21 | 1 / 2 | 88% | 81
80% | 9 / 17 |  2 / 2 | 86% | 88
90% | 7 / 13 |  2 / 3 | 93% | 47

The goal of training using Knowledge-Intensified fine-tuning is to incorporate
more project-specific knowledge to CodeT5. There are two hyper-parameters of
Knowledge-Intensified fine-tuning, including mask rate and repetition
iterations. Our default setting is to use a 50% mask rate and 10 repetition
times. We conduct an ablation experiment to study the impacts of these two
hyper-parameters on the number of correct/plausible patches generated, the
number of unique bugs fixed (via correct/plausible patches) when compared to
the base CodeT5 model, the compilation error rate, and the number of unique
compilable patches generated per bug.

We first examine the impact of repeating the masking multiple times during
training to generate additional training samples. Table 3 shows the results on
the Closure bugs with 10 repetition iterations – generating 10 training sets
(Row Repetitive) and no repetition iterations – generating only 1 training set
(Row Non-Repetitive). We observe that the number of total correct and
plausible patches produced by the repetitive training approach is higher.
Additionally, when we generate new masked training samples during each
iteration, the model produced is able to generate two unique bug fixes
compared with the base CodeT5 model. Similar results can be found when we look
at the compilation error rate together with the number of unique compilable
patches generated. We see that while the non-repetitive approach has a lower
compilation error rate, the number of compilable patches generated is much
less. By repeating the masking multiple times during training, we are able to
fine-tune the model to learn more project-specific information to produce
compilable patches and to fix more unique bugs.

Next, we study the impacts of different mask rates. In this experiment, we use
the default of 10 repetition iterations by generating 10 unique training
samples during fine-tuning and examine how different mask rates can have on
performance. Due to the extremely large search space (considering unlimited
choices of mask rates), we choose mask rates from 10% to 90% with an interval
of 10% (9 different mask rates in total). Table 4 shows our experimental
results on the bugs in the Closure project. First, we observe that an
extremely high mask rate (70, 80, 90%) performs poorly in terms of the number
of bugs fixed and compilation rate. While a high mask rate may force the model
to learn more project-specific tokens during training, each training sample
will have a majority of its tokens masked out. Compared with the final repair
task of generating a single or partial line, the extremely high mask rate
makes the resulting model ill-suited for repair. We observe that the default
setting of 50% mask rate strikes a good balance between achieving the high
total number of bugs fixed, more unique bugs fixed compared to base CodeT5,
and a relatively low compilation error rate. By using a balanced mask rate of
50%, the Knowledge-Intensified fine-tuning model is able to best complement
the base CodeT5 in generating more unique bug fixes.

####  5.2.2. Impacts of Repair-Oriented fine-tuning

Table 5. Masking strategies of Repair-Oriented fine-tuning

Strategy | AST masking | Single-line masking | Template masking
---|---|---|---
#Corr. / #Plaus. (All) |  13 / 25 | 10 / 21 |  12 / 23
#Corr. / #Plaus. (New) | 0 / 1 |  1 / 1 |  1 / 2
Comp. Error pct | 88% | 86% | 76%
#Unique comp. per bug | 101 | 129 | 139

In addition to looking at the impact of different configurations when using
Knowledge-Intensified fine-tuning, we study the different ways we can apply
Repair-Oriented fine-tuning. Specifically, we design two additional strategies
that can be used during training to produce masked training samples. Table 5
shows the result of our default “Template masking”, “AST masking”, and
“Single-line masking” on bugs in the Closure project. Recall that our default
setting applies repair templates that we use for cloze-style APR directly on
the training data to produce masked lines. AST masking will parse the selected
line into an AST and randomly choose a subtree to mask out. On the other hand,
single-line masking will simply mask out one entire line in a training sample.
We observe that single-line masking performed the worst in terms of the number
of correct and plausible patches. This is due to the fact that during repair,
we use repair templates that not just regenerate complete lines but also mask
out part of the lines. The model just has to regenerate the partial code
within the line. Single-line masking is only trained on generating the
complete line and thus does not perform well when used with repair templates.
Additionally, when compared with AST masking, template masking is able to fix
more unique bugs compared with the base CodeT5 since it directly leverages the
inference repair templates to create training samples. While AST masking makes
use of the structure information, it does not fully emulate the inference
setting of cloze-style APR. Furthermore, the two baselines both resulted in a
lot of patches with high compilation error rates compared with our proposed
template masking strategy. Template masking is able to directly learn the
types of repair templates that the final repair task will use as input for the
model, resulting in fewer compilation failures. By using template masking for
Repair-Oriented fine-tuning, we can train a model that is optimized for the
repair task in order to generate more bug fixes to compliment the base CodeT5
model, which shows that fine-tuning model with training strategies that
assemble the underlying repair techniques is able to further boost its bug-
fixing performance.

####  5.2.3. Impacts of Relevant-Identifier prompting

Table 6. Configurations of Relevant-Identifier prompting

Configuration |  #Corr. / #Plaus. (All) |  #Corr./#Plaus. (New)
---|---|---
Default(top-5 current-file type separate) | 25 / 39 | 3 / 3
Top-1 | 24 / 37 | 1 / 1
Top-10 | 23 / 38 | 1 / 1
Top-20 | 23 / 37 | 1 / 1
Full project | 23 / 36 | 1 / 1
No type | 24 / 37 | 2 / 2
Together | 24 / 40 | 1 / 1

We examine the different parameters of our Relevant-Identifier prompting
strategy. Table 6 shows the results of our default (Row Default) approach and
other configurations on Closure project. We first look at the effect of
varying the top $N$ identifiers and observe that when only considering the
top-1 identifier for each bug we do not generate more correct fixes since it
is unlikely that the relevant identifier to fix the bug always has the highest
ranking. On the flip side, considering a larger amount of identifiers per bug
(top-10, top-20) is also not desirable since we limit the model to sample only
5000 times per bug, and generating more prompts will decrease the number of
samples per each prompt. Next, we look at the scope of the project where we
find relevant tokens. Our default setting considers only the current file of
the bug and we compare this to when we consider the full project (i.e.,
changing Line 1 of Algorithm 1 to consider all files within the project). We
observe that the number of correct and plausible fixes decreases which
reflects a similar finding from prior work (Barr et al., 2014) where a
significant amount of correct fixing ingredients (relevant identifiers) can
already be found within the same file. Furthermore, by considering the entire
project, we could introduce more noise where potentially irrelevant
identifiers could be highly ranked. Following, we compare the effect of having
type information of the identifier in the prompt. We observe that our default
setting (with type) is able to generate more correct fixes compared to without
types, indicating the usefulness of such information in helping the model
generate the correct usage of the identifier in the patch. Finally, we examine
our default prompting method of only providing one relevant identifier at a
time. We compare this against another approach to include all the top 5
relevant tokens in the same prompt. We see that separating each relevant
identifier to its own prompt provides us with more fixes as including all
identifiers together can potentially confuse the model.

####  5.2.4. Overhead of FitRepair

As FitRepair proposes to fine-tune two separate models along with prompting
via information retrieval and static analysis, we investigate the extra
overhead of using FitRepair compared to just using the base CodeT5. Recall
that FitRepair only fine-tunes the model on the oldest version of the project
for Defects4j 1.2 (one-time cost) and uses the trained models to generate
patches for all bugs within that project. We find that on average, for each
bug in Closure, FitRepair adds 14.3 minutes (6.6 for each fine-tuning strategy
and 1.0 for prompting strategy compared with directly using the base CodeT5
model. This shows that overall, FitRepair adds a minimal amount of overhead to
the repair process (still within the 5-hour limit including overhead). For
practical use, the fine-tuning steps can be done ahead of the actual repair
task (e.g., periodically during nights or weekends), incurring no additional
time cost compared to previous LLM-based APR tools. Developers can then apply
the fine-tuned models together with the base model whenever a bug is detected.

###  5.3. RQ3: Generalizability of FitRepair

Table 7. Evaluation results of correct fixes on Defects4j 2.0

FitRepair | CodeT5$\times$4 |  Alpha Repair | SelfAPR |  Reward Repair | Recoder | TBar
---|---|---|---|---|---|---
44 | 42 | 36 | 31 | 25 | 11 | 8

Figure 9. Repair-Oriented fine-tuning unique patch

We further evaluate the generalizability of FitRepair on an additional repair
dataset of Defects4j 2.0 containing new bugs and projects. Table 7 shows the
number of correct bug fixes on single-line bugs in Defects4j 2.0. We observe
that FitRepair is able to achieve the state-of-the-art with the highest number
of correctly fixed bugs of 44 (8 more than the best-performing baseline).
Unlike other NMT-based or traditional template-based APR tools, FitRepair does
not suffer from the dataset overfitting issue of only performing well on the
base Defects4j 1.2 dataset. In fact, the relative improvement in the total
number of bugs fixed is higher on Defects4j 2.0 (22.2% increase) compared to
1.2 (20.3% increase). Furthermore, comparing against the baseline (Column
CodeT5$\times$4), FitRepair is able to improve the number of total bug fixes
from 42 to 44 and produce 3 unique bug fixes.

Figure 9 shows a bug (Codec-3) fixed by FitRepair but cannot be fixed by any
other studied APR tool. The root cause of this bug is an off-by-one error on
the second argument in the function call. While this bug looks very simple to
fix, one reason previous learning-based APR was not able to provide a correct
fix could be the unconventional values of ”4” and ”3”. During the training,
NMT-based APR can learn from bug-fixing datasets where it is common to use
swap a ”0” to a ”1” and vice-versa to fix a bug. However, changing a ”4” to a
”3” can be uncommon in the bug-fixing dataset. Cloze-style APR tool that
directly leverages LLMs can also have a hard time on this bug since the change
is very small even if a direct repair template can be applied. Since this
change is very small, the LLM should not add any additional code other than
changing ”4” to ”3”. However, during training one single mask span usually
represents multiple different tokens, which may cause the base CodeT5 model to
generate more tokens than needed. Using FitRepair and specifically the Repair-
Oriented fine-tuning strategy, the fine-tuned CodeT5 can learn such short code
generation that usually stems from repair templates such as argument
replacement. As such, FitRepair is able to generate this simple patch to fix
the underlying bug.

##  6\. Threats To Validity

Internal. Our manual examination in determining the correct patches,
semantically equivalent to reference developer patches, from plausible patches
is one internal threat to validity. Following common APR practice, the first
two authors perform a careful analysis of each plausible patch along with
multiple discussions to determine the correctness of a patch.

Another internal threat to validity comes from using the CodeT5 model which is
trained on open-source GitHub code snippets (Husain et al., 2020). This means
the training data could overlap with our evaluation repair dataset of
Defects4j. To address this, we follow prior work (Xia and Zhang, 2022) and
compute the number of patched functions by FitRepair that also exist in the
pre-training dataset. In total, out of the 89 bugs fixed on Defects4j 1.2, 13
of these fixes are part of the original pre-training dataset of CodeT5. This
shows that the majority of the correct fixes (76/89 = 85%) do not contain any
reference developer patch in the training data. Furthermore, for a fair
comparison, if we exclude the 13 bugs whose patched functions overlap with the
CodeT5 pre-training dataset following prior work (Xia and Zhang, 2022), we are
still able to achieve state-of-the-art performance on Defects4j 1.2 with 76
total fixes compared to 67 from the best-performing baseline on the remaining
bugs. This shows that FitRepair is not simply performing well on the datasets
due to the developer reference patches that the model saw during pre-training.
Similarly, on Defects4j 2.0, we found that 6 fixed bugs have their reference
patch function within the training dataset. Applying the same removal
comparison, we still achieve the state-of-the-art result of 38 compared to the
best-performing baseline of 30 on Defects4j 2.0. Additionally, we also
demonstrate that regardless of the overlap between the training and evaluation
datasets, by combining project-specific fine-tuning and prompting strategies,
we can further improve the performance of the base LLM. Future work to
completely address this threat would need to retrain the CodeT5 model from
scratch after removing the overlapping functions.

External. The major external threat to validity comes from our evaluation
dataset. The performance achieved by FitRepair may not generalize well to
other datasets. To address this, we use two different versions of Defects4j,
namely 1.2 and 2.0, and demonstrated that FitRepair is able to achieve state-
of-the-art results on both datasets. In the future, we plan on continuing to
address this threat by performing more evaluations on other repair datasets
across multiple programming languages.

##  7\. Conclusion

In this paper, we have proposed FitRepair, the first fully automated approach
to incorporate domain-specific knowledge with the insights of the _plastic
surgery hypothesis_ for boosting the performance of LLMs for APR. FitRepair
opens up a new dimension for LLM-based APR by using both fine-tuning and
prompting to combine the power of LLM with project-specific information. Our
evaluation results on the popular Defects4j 1.2 and 2.0 datasets show that
FitRepair is able to achieve the new state-of-the-art results in fixing 89 and
44 bugs (15 and 8 more than the best-performing baselines)
respectively.Different from prior APR work on plastic surgery hypothesis,
FitRepair is fully automated, effective, and general. Moreover, even
partial/imprecise information may still effectively guide LLMs for APR!

## References

  * (1)
  * Abreu et al. (2007) Rui Abreu, Peter Zoeteweij, and Arjan J.C. van Gemund. 2007\.  On the Accuracy of Spectrum-based Fault Localization. In _Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007)_. 89–98.
  * Ahmad et al. (2021) Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021.  Unified Pre-training for Program Understanding and Generation.  arXiv:2103.06333 [cs.CL]
  * Asad et al. (2019) Moumita Asad, Kishan Kumar Ganguly, and Kazi Sakib. 2019\.  Impact analysis of syntactic and semantic similarities on patch prioritization in automated program repair. In _2019 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. 328–332.
  * Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021.  Program Synthesis with Large Language Models.  https://arxiv.org/abs/2108.07732
  * Barr et al. (2014) Earl T. Barr, Yuriy Brun, Premkumar Devanbu, Mark Harman, and Federica Sarro. 2014.  The Plastic Surgery Hypothesis. In _Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering_ (Hong Kong, China) _(FSE 2014)_. Association for Computing Machinery, New York, NY, USA, 306–317.
  * Benton et al. (2020) Samuel Benton, Xia Li, Yiling Lou, and Lingming Zhang. 2020\.  On the Effectiveness of Unified Debugging: An Extensive Study on 16 Program Repair Systems. In _ASE_. 907–918.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners.  arXiv:2005.14165.
  * Chen et al. (2017) Liushan Chen, Yu Pei, and Carlo A. Furia. 2017.  Contract-based program repair without the contracts. In _2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE)_. 637–647.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021\.  Evaluating Large Language Models Trained on Code.  arXiv:2107.03374.
  * Chen et al. (2019) Zimin Chen, Steve Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and Martin Monperrus. 2019.  SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair.  _IEEE Transaction on Software Engineering_ (2019).
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022.  PaLM: Scaling Language Modeling with Pathways.  arXiv:2204.02311 [cs.CL]
  * Dallmeier and Zimmermann (2007) Valentin Dallmeier and Thomas Zimmermann. 2007.  Extraction of Bug Localization Benchmarks from History. In _Proceedings of the Twenty-Second IEEE/ACM International Conference on Automated Software Engineering_ (Atlanta, Georgia, USA) _(ASE ’07)_. Association for Computing Machinery, New York, NY, USA, 433–436.
  * DeMarco et al. (2014) Favio DeMarco, Jifeng Xuan, Daniel Le Berre, and Martin Monperrus. 2014.  Automatic Repair of Buggy If Conditions and Missing Preconditions with SMT. In _Proceedings of the 6th International Workshop on Constraints in Software Testing, Verification, and Analysis_ (Hyderabad, India) _(CSTVA 2014)_. 30–39.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  arXiv:1810.04805.
  * Feng et al. (2020) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020.  CodeBERT: A Pre-Trained Model for Programming and Natural Languages.  arXiv:2002.08155.
  * Fried et al. (2022) Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022\.  InCoder: A Generative Model for Code Infilling and Synthesis.  arXiv:2204.05999.
  * Ghanbari et al. (2019) Ali Ghanbari, Samuel Benton, and Lingming Zhang. 2019\.  Practical Program Repair via Bytecode Mutation. In _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis_ (Beijing, China) _(ISSTA 2019)_. ACM, 19–30.
  * Guo et al. (2021) Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021\.  GraphCodeBERT: Pre-training Code Representations with Data Flow.  arXiv:2009.08366 [cs.SE]
  * Hua et al. (2018) Jinru Hua, Mengshi Zhang, Kaiyuan Wang, and Sarfraz Khurshid. 2018\.  SketchFix: A Tool for Automated Program Repair Approach Using Lazy Candidate Generation. In _Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Lake Buena Vista, FL, USA) _(ESEC/FSE 2018)_. ACM, 888–891.
  * HuggingFace (2022) HuggingFace 2022.  Hugging Face.  https://huggingface.co.
  * Husain et al. (2020) Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2020.  CodeSearchNet Challenge: Evaluating the State of Semantic Code Search.  arXiv:1909.09436.
  * Jaro (1989) Matthew A Jaro. 1989\.  Advances in record-linkage methodology as applied to matching the 1985 census of Tampa, Florida.  _J. Amer. Statist. Assoc._ 84, 406 (1989), 414–420.
  * JavaParser (2023) JavaParser 2023.  JavaParser.  https://javaparser.org.
  * Jiang et al. (2019) Jiajun Jiang, Luyao Ren, Yingfei Xiong, and Lingming Zhang. 2019\.  Inferring Program Transformations From Singular Examples via Big Code. In _2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. 255–266.
  * Jiang et al. (2018) Jiajun Jiang, Yingfei Xiong, Hongyu Zhang, Qing Gao, and Xiangqun Chen. 2018.  Shaping program repair space with existing patches and similar code. In _Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2018, Amsterdam, The Netherlands, July 16-21, 2018_ , Frank Tip and Eric Bodden (Eds.). ACM, 298–309.
  * Jiang et al. (2021) Nan Jiang, Thibaud Lutellier, and Lin Tan. 2021\.  CURE: Code-Aware Neural Machine Translation for Automatic Program Repair.  _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_ (May 2021).
  * Just et al. (2014) René Just, Darioush Jalali, and Michael D. Ernst. 2014\.  Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs _(ISSTA 2014)_. Association for Computing Machinery, New York, NY, USA, 437–440.
  * Kanade et al. (2020) Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020.  Learning and Evaluating Contextual Embedding of Source Code.
  * Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba. 2014.  Adam: A Method for Stochastic Optimization.  arXiv:1412.6980 [cs.CL]
  * Kolak et al. (2022) Sophia D Kolak, Ruben Martins, Claire Le Goues, and Vincent Josua Hellendoorn. 2022\.  Patch Generation with Language Models: Feasibility and Scaling Behavior. In _Deep Learning for Code Workshop_.
  * Koyuncu et al. (2020) Anil Koyuncu, Kui Liu, Tegawendé F. Bissyandé, Dongsun Kim, Jacques Klein, Martin Monperrus, and Yves Le Traon. 2020\.  FixMiner: Mining relevant fix patterns for automated program repair.  _Empir. Softw. Eng._ 25, 3 (2020), 1980–2024.
  * Kuznia et al. (2022) Kirby Kuznia, Swaroop Mishra, Mihir Parmar, and Chitta Baral. 2022.  Less is More: Summary of Long Instructions is Better for Program Synthesis.  arXiv:2203.08597.
  * Le et al. (2017) Xuan-Bach D Le, Duc-Hiep Chu, David Lo, Claire Le Goues, and Willem Visser. 2017.  S3: syntax-and semantic-guided repair synthesis via programming by examples. In _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering_. 593–604.
  * Le et al. (2016) Xuan Bach D. Le, David Lo, and Claire Le Goues. 2016\.  History Driven Program Repair. In _2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER)_ , Vol. 1. 213–224.
  * Le Goues et al. (2012) Claire Le Goues, ThanhVu Nguyen, Stephanie Forrest, and Westley Weimer. 2012.  GenProg: A Generic Method for Automatic Software Repair.  _IEEE Transactions on Software Engineering_ 38, 1 (2012), 54–72.
  * Levenshtein et al. (1966) Vladimir I Levenshtein et al. 1966\.  Binary codes capable of correcting deletions, insertions, and reversals. In _Soviet physics doklady_ , Vol. 10. Soviet Union, 707–710.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019\.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.  arXiv:1910.13461.
  * Li et al. (2019) Xia Li, Wei Li, Yuqun Zhang, and Lingming Zhang. 2019\.  Deepfl: Integrating multiple fault diagnosis dimensions for deep fault localization. In _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis_. 169–180.
  * Li and Zhang (2017) Xia Li and Lingming Zhang. 2017.  Transforming programs and tests in tandem for fault localization.  _Proceedings of the ACM on Programming Languages_ 1, OOPSLA (2017), 1–30.
  * Li et al. (2020) Yi Li, Shaohua Wang, and Tien N. Nguyen. 2020.  DLFix: Context-Based Code Transformation Learning for Automated Program Repair. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ (Seoul, South Korea) _(ICSE ’20)_. Association for Computing Machinery, New York, NY, USA, 602–614.
  * Liu et al. (2019a) Kui Liu, Anil Koyuncu, Dongsun Kim, and Tegawendé F. Bissyandé. 2019a.  TBar: Revisiting Template-Based Automated Program Repair. In _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis_ _(ISSTA 2019)_. ACM, New York, NY, USA, 31–42.
  * Liu et al. (2019b) Kui Liu, Anil Koyuncu, Dongsun Kim, and Tegawendé F. Bissyandé. 2019b.  AVATAR: Fixing Semantic Bugs with Fix Patterns of Static Analysis Violations. In _Proceedings of the 26th IEEE International Conference on Software Analysis, Evolution, and Reengineering_. IEEE, 456–467.
  * Long and Rinard (2015) Fan Long and Martin Rinard. 2015.  Staged Program Repair with Condition Synthesis. In _Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering_ (Bergamo, Italy) _(ESEC/FSE 2015)_. New York, NY, USA, 166–178.
  * Long and Rinard (2016) Fan Long and Martin Rinard. 2016.  Automatic Patch Generation by Learning Correct Code.  _SIGPLAN Not._ 51, 1 (jan 2016), 298–312.
  * Lou et al. (2020) Yiling Lou, Ali Ghanbari, Xia Li, Lingming Zhang, Haotian Zhang, Dan Hao, and Lu Zhang. 2020.  Can automated program repair refine fault localization? a unified debugging approach. In _Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis_. 75–87.
  * Lu et al. (2021) Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021\.  CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation.  arXiv:2102.04664 [cs.SE]
  * Lutellier et al. (2020) Thibaud Lutellier, Hung Viet Pham, Lawrence Pang, Yitong Li, Moshi Wei, and Lin Tan. 2020.  CoCoNuT: Combining Context-Aware Neural Translation Models Using Ensemble for Program Repair. In _Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis_ (Virtual Event, USA) _(ISSTA 2020)_. Association for Computing Machinery, New York, NY, USA, 101–114.
  * Martinez et al. (2015) Matias Martinez, Thomas Durieux, Jifeng Xuan, Romain Sommerard, and Martin Monperrus. 2015.  Automatic Repair of Real Bugs: An Experience Report on the Defects4J Dataset.  arXiv:1505.07002.
  * Martinez and Monperrus (2016) Matias Martinez and Martin Monperrus. 2016.  ASTOR: A Program Repair Library for Java (Demo). In _Proceedings of the 25th International Symposium on Software Testing and Analysis_ (Saarbrücken, Germany) _(ISSTA 2016)_. Association for Computing Machinery, New York, NY, USA, 441–444.
  * Matteson (2018) Scott Matteson. 2018\.  Report: Software failure caused $1.7 trillion in financial losses in 2017.  _TechRepublic_ (2018).  https://www.techrepublic.com/article/report-software-failure-caused-1-7-trillion-in-financial-losses-in-2017/.
  * Mechtaev et al. (2016) Sergey Mechtaev, Jooyong Yi, and Abhik Roychoudhury. 2016\.  Angelix: Scalable Multiline Program Patch Synthesis via Symbolic Analysis. In _Proceedings of the 38th International Conference on Software Engineering_ (Austin, Texas) _(ICSE ’16)_. 691–701.
  * Nijkamp et al. (2022) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022\.  CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis.  arXiv:2203.13474.
  * Omri and Sinz (2020) Safa Omri and Carsten Sinz. 2020.  Deep Learning for Software Defect Prediction: A Survey. In _Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops_ (Seoul, Republic of Korea) _(ICSEW’20)_. Association for Computing Machinery, New York, NY, USA, 209–214.
  * Papadakis and Le Traon (2015) Mike Papadakis and Yves Le Traon. 2015.  Metallaxis-FL: mutation-based fault localization.  _Software Testing, Verification and Reliability_ 25, 5-7 (2015), 605–628.
  * Prenner et al. (2022) Julian Aron Prenner, Hlib Babii, and Romain Robbes. 2022\.  Can OpenAI’s Codex Fix Bugs?: An evaluation on QuixBugs. In _2022 IEEE/ACM International Workshop on Automated Program Repair (APR)_. 69–75.
  * PyTorch (2022) PyTorch 2022.  PyTorch.  http://pytorch.org.
  * Qi et al. (2015) Zichao Qi, Fan Long, Sara Achour, and Martin Rinard. 2015\.  An Analysis of Patch Plausibility and Correctness for Generate-and-Validate Patch Generation Systems. In _Proceedings of the 2015 International Symposium on Software Testing and Analysis_ (Baltimore, MD, USA) _(ISSTA 2015)_. Association for Computing Machinery, New York, NY, USA, 24–36.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019\.  Language Models are Unsupervised Multitask Learners.  (2019).
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _J. Mach. Learn. Res._ (jan 2020).
  * Ratcliff and Metzener (1988) John W Ratcliff and David E Metzener. 1988.  Pattern-matching-the gestalt approach.  _Dr Dobbs Journal_ 13, 7 (1988), 46.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_ (2019).
  * Richards (1990) Evelyn Richards. 1990\.  Software’s Dangerous Aspect.  _The Washington Post_ (1990).  https://www.washingtonpost.com/archive/politics/1990/12/09/softwares-dangerous-aspect/9b2e9243-8deb-4ac7-9e8f-968de0806e5e/.
  * Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009\.  The probabilistic relevance framework: BM25 and beyond.  _Foundations and Trends® in Information Retrieval_ 3, 4 (2009), 333–389.
  * Saha et al. (2017) Ripon K. Saha, Yingjun Lyu, Hiroaki Yoshida, and Mukul R. Prasad. 2017.  Elixir: Effective object-oriented program repair. In _2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE)_. 648–659.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014\.  Sequence to Sequence Learning with Neural Networks.  arXiv:1409.3215.
  * Tufano et al. (2018) Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. 2018.  An Empirical Investigation into Learning Bug-Fixing Patches in the Wild via Neural Machine Translation. In _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_. 832–837.
  * Tunstall et al. (2022) Lewis Tunstall, Leandro von Werra, and Thomas Wolf. 2022\.  _Natural language processing with transformers_.  ” O’Reilly Media, Inc.”.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention Is All You Need.  (2017).  arXiv:1706.03762.
  * Wang et al. (2022) Deze Wang, Zhouyang Jia, Shanshan Li, Yue Yu, Yun Xiong, Wei Dong, and Xiangke Liao. 2022\.  Bridging Pre-Trained Models and Downstream Tasks for Source Code Understanding. In _Proceedings of the 44th International Conference on Software Engineering_ (Pittsburgh, Pennsylvania) _(ICSE ’22)_. Association for Computing Machinery, New York, NY, USA, 287–298.
  * Wen et al. (2018) Ming Wen, Junjie Chen, Rongxin Wu, Dan Hao, and Shing-Chi Cheung. 2018.  Context-Aware Patch Generation for Better Automated Program Repair. In _Proceedings of the 40th International Conference on Software Engineering_ (Gothenburg, Sweden) _(ICSE ’18)_. 1–11.
  * Wettig et al. (2022) Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. 2022\.  Should You Mask 15% in Masked Language Modeling?  arXiv:2202.08005 [cs.CL]
  * Winkler (1990) William E Winkler. 1990\.  String comparator metrics and enhanced decision rules in the Fellegi-Sunter model of record linkage.  (1990).
  * Wong et al. (2016) W. Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa. 2016.  A Survey on Software Fault Localization.  _IEEE Transactions on Software Engineering_ 42, 8 (2016), 707–740.
  * Xia et al. (2023) Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023\.  Automated Program Repair in the Era of Large Pre-trained Language Models. In _Proceedings of the ACM/IEEE 45th International Conference on Software Engineering_ _(ICSE ’23)_.
  * Xia and Zhang (2022) Chunqiu Steven Xia and Lingming Zhang. 2022.  Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-Shot Learning. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ _(ESEC/FSE 2022)_.
  * Xu et al. (2022) Frank F. Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022.  A Systematic Evaluation of Large Language Models of Code. In _Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming_ (San Diego, CA, USA) _(MAPS 2022)_. Association for Computing Machinery, New York, NY, USA, 1–10.
  * Ye et al. (2022b) He Ye, Matias Martinez, Xiapu Luo, Tao Zhang, and Martin Monperrus. 2022b.  SelfAPR: Self-supervised Program Repair with Test Execution Diagnostics. In _37th IEEE/ACM International Conference on Automated Software Engineering_ _(ASE22)_. Association for Computing Machinery, Article 92, 13 pages.
  * Ye et al. (2022a) He Ye, Matias Martinez, and Martin Monperrus. 2022a.  Neural Program Repair with Execution-based Backpropagation. In _2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE)_. 1506–1518.
  * Yue Wang and Hoi (2021) Shafiq Joty Yue Wang, Weishi Wang and Steven C.H. Hoi. 2021.  CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021_.
  * Zhang et al. (2013) Lingming Zhang, Lu Zhang, and Sarfraz Khurshid. 2013.  Injecting mechanical faults to localize developer faults for evolving software.  _ACM SIGPLAN Notices_ 48, 10 (2013), 765–784.
  * Zhu et al. (2021) Qihao Zhu, Zeyu Sun, Yuan-an Xiao, Wenjie Zhang, Kang Yuan, Yingfei Xiong, and Lu Zhang. 2021.  A Syntax-Guided Edit Decoder for Neural Program Repair. In _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. ACM, New York, NY, USA, 341–353.