# T5APR: Empowering Automated Program Repair Across Languages through
Checkpoint Ensemble

Reza Gharibi, Mohammad Hadi Sadreddini, Seyed Mostafa Fakhrahmad
Department of Computer Science and Engineering and IT
School of Electrical and Computer Engineering
Shiraz University, Shiraz, Iran
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
Corresponding author

###### Abstract

Automated program repair (APR) using deep learning techniques has become an
important area of research in recent years, aiming to automatically generate
bug-fixing patches that can improve software reliability and maintainability.
However, most existing methods either target a single language or require high
computational resources to train multilingual models. In this paper, we
propose T5APR, a novel neural program repair approach that provides a unified
solution for bug fixing across multiple programming languages. T5APR leverages
CodeT5, a powerful pre-trained text-to-text transformer model, and adopts a
checkpoint ensemble strategy to improve patch recommendation. We conduct
comprehensive evaluations on six well-known benchmarks in four programming
languages (Java, Python, C, JavaScript), demonstrating T5APR’s competitiveness
against state-of-the-art techniques. T5APR correctly fixes 1,985 bugs,
including 1,442 bugs that none of the compared techniques has fixed. We
further support the effectiveness of our approach by conducting detailed
analyses, such as comparing the correct patch ranking among different
techniques. The findings of this study demonstrate the potential of T5APR for
use in real-world applications and highlight the importance of multilingual
approaches in the field of APR.

_Keywords_ Automated program repair $\cdot$ Neural program repair $\cdot$ Deep
learning $\cdot$ Transformer

##  1 Introduction

Software bugs are unavoidable in software development and can lead to security
breaches, system failures, and user dissatisfaction, making it crucial to
detect and fix them efficiently. However, manual debugging is time-consuming,
particularly when dealing with large and intricate software systems. The
growing need for high-quality and reliable software, coupled with the
increasing complexity of software systems, has led to a surge of interest in
automated program repair (APR). APR is an evolving research area that
automatically fixes software bugs to enhance software reliability and
maintainability. APR can potentially save developers time and effort, improve
software quality and maintenance, and enable faster and more frequent software
releases (Huang et al., 2023; Gao et al., 2022; Gazzola et al., 2018;
Monperrus, 2018a; Le Goues et al., 2019).

Recent advancements in machine learning have shown promise in improving APR by
adopting deep learning techniques, such as sequence-to-sequence, neural
machine translation (NMT), and graph-to-sequence models, to automatically
generate correct patches for buggy source code (Zhang et al., 2023; Zhong et
al., 2022). These techniques can learn patterns from large code repositories
and generate bug-fixing patches with state-of-the-art performance. APR tools
that use these techniques are called neural program repair tools. Neural
program repair has been mostly implemented with supervised learning on past
bug-fixing commits to generate patches as sequences of tokens or edits, given
a buggy code and its context (Chakraborty et al., 2022; Chen et al., 2019;
Ding et al., 2020; Jiang et al., 2021; Li et al., 2020; Lutellier et al.,
2020; Ye et al., 2022; Zhu et al., 2021).

However, most existing APR methods are limited by their language-specificity
and their high computational cost. They are either expensive to train for
multiple programming languages (Lutellier et al., 2020) or focus on a single
language domain. Although they may generalize to other languages, they are
rarely implemented and evaluated for other languages. This restricts their
applicability and scalability across different languages and domains of code.
For example, CoCoNuT (Lutellier et al., 2020) and CIRCLE (Yuan et al., 2022)
are two of the few approaches that are evaluated on multiple programming
languages. To achieve multilingual repair, CoCoNuT trains separate models for
each programming language, which requires large amounts of resources. CIRCLE
uses continual learning on a single model but still needs measures to prevent
catastrophic forgetting of the model and also a re-repairing post-processing
strategy to provide patches for different languages.

To address these limitations, this paper proposes T5APR, a novel multilingual
neural repair method that leverages the power of transformer sequence-to-
sequence models (Vaswani et al., 2017). T5APR is based on CodeT5 (Wang et al.,
2021), a pre-trained model for code generation and understanding. T5APR fine-
tunes CodeT5 in multitask learning style on a dataset of buggy and fixed code
snippets and uses checkpoint ensemble (Chen et al., 2017) from different
training steps to generate candidate patches. As shown in Figure 1, we train a
unified model to fix bugs from various programming languages and use language
control codes in our input prompt to distinguish between different languages.
This approach enables us to achieve efficient and scalable training with low
resource consumption. T5APR then ranks and validates the generated candidate
patches using the project’s test suite to select the most suitable one.

Figure 1: Illustration of T5APR for multilingual program repair.

We evaluate our method on six benchmarks including Defects4J (Just et al.,
2014), Bears (Madeiral et al., 2019), QuixBugs (Lin et al., 2017), Codelfaws
(Tan et al., 2017), ManyBugs (Le Goues et al., 2015), and BugAID (Hanam et
al., 2016), and compare its performance with 11 existing APR methods. Results
show that T5APR can generate correct patches for various types of bugs in
different languages, and achieves state-of-the-art performance in terms of
both repair effectiveness (i.e., correct fixes) and efficiency (i.e., ranking
of fixes). Across all the benchmarks and from 5,257 bugs, T5APR fixes 1,985 of
them; meaning the first plausible patch that it generates is semantically
equivalent to the developer’s patch.

The main contributions of this paper are as follows:

  * •

We introduce T5APR, a novel multilingual neural repair approach that offers a
solution for bug fixing across various programming languages (Section 2).

  * •

We address the challenge of resource efficiency in training without
introducing additional models for each language by modeling multilingual APR
with multitask learning (Section 2.6).

  * •

We propose a checkpoint ensemble strategy, enhancing the effectiveness of
generated patches (Section 2.7).

  * •

We evaluate T5APR on six benchmarks and compare its performance with 11 state-
of-the-art APR methods, achieving competitive performance (Sections 3 and 4).

  * •

We provide an open-source implementation of T5APR, and our results are
publicly available to foster further research and practical applications:
https://github.com/h4iku/T5APR.

We discuss related work in Section 5, and Section 6 concludes the paper and
suggests future directions.

##  2 Approach

###  2.1 Overview

Figure 2 shows the overview of T5APR’s structure. T5APR leverages CodeT5 as
its base model and combines the outputs of multiple checkpoints to achieve
improved performance in automated program repair (APR). T5APR involves two
stages: training and inference. During the training stage, the CodeT5 model is
fine-tuned on a large-scale multilingual dataset of buggy and fixed code
snippets. In the inference stage, multiple checkpoints are used to generate
candidate patches given buggy code and its context. The checkpoints are
selected from different steps of the fine-tuning process. Finally, we rank the
candidate patches based on a combination of their rank in each checkpoint and
their likelihood score, and validate them using the project’s test suite to
select the most suitable patch. APR tools usually return the highest-ranked
patch that compiles and passes the test cases, known as the first plausible
patch. The following sections will describe this process in detail.

Figure 2: Overview of T5APR.

###  2.2 Data extraction

The process of training T5APR involves the extraction of vast amounts of data
consisting of the buggy lines (source), their surrounding context to enhance
the model’s understanding of the faulty code, and their corresponding fixed
versions (target). The training data is collected from multiple open-source
projects in different programming languages, ensuring a broad representation
of real-world code scenarios.

For the buggy context, we adopt the “immediate buggy context,” which refers to
the function or method containing the buggy lines. Although other context
choices, such as the entire file content or context obtained through data flow
analysis or program slicing are possible, the immediate buggy context is often
short and can easily be obtained for a multilingual model. The bigger the
context, the more likely it is to have the needed ingredients to fix the bug
(Ding et al., 2020; Yang et al., 2021), but we have to give the model a longer
sequence, which increases the computation time and resources. Therefore, we
aim to create a balanced training environment that efficiently captures
essential information for APR without sacrificing computational resources. The
context-independence of our model allows for future incorporation of different
context types to potentially improve performance (Chen et al., 2019).

In our experiments, we train the T5APR model on the dataset collected by
CoCoNuT (Lutellier et al., 2020). This dataset follows the same criteria drawn
above and consists of tuples of buggy, context, and fixed hunks of code
extracted from the commit history of various open-source projects. A hunk is a
set of consecutive lines of code change extracted from the commit history. For
example, in Figure 3(a), the lines that start with (`-`) are buggy lines,
lines that start with (`+`) are fixed lines, and the whole `flatten` function
with the buggy lines is the context. The same applies to Figure 3(b), except
that the fixed hunk has more than one line here.

⬇

def flatten(arr):

for x in arr:

if isinstance(x, list):

for y in flatten(x):

yield y

else:

- yield flatten(x)

+ yield x

(a) Code change of QuixBugs FLATTEN bug.

⬇

private boolean isFailOnCCE() {

- return getStep().isFailOnCCE();

+ AbstractStep step = getStep();

+ if (step == null) {

+ return false;

+ }

+ return step.isFailOnCCE();

}

(b) Code change of Bears-32 bug.

Figure 3: Examples of buggy lines, buggy context, and fixed lines.

CoCoNuT uses a keyword-based heuristic to identify bug-fixing commits from
their commit messages (Ray et al., 2016; Mockus and Votta, 2000). They
manually examined random commits and confirmed the effectiveness of this
filtering process. However, we further clean the data during preprocessing to
ensure high-quality training instances.

###  2.3 Preprocessing

This section describes the steps taken to prepare the input training data for
the T5APR model. Before feeding the data to the model, we apply several
preprocessing steps to enhance data quality and reduce computational resource
requirements (Raffel et al., 2020):

  * •

Comment removal: Comments are removed from both sources and targets. This step
ensures that the model focuses solely on the functional aspects of the code.

  * •

Deduplication: We deduplicate the training data across source, context, and
target based on their string representation, disregarding whitespace
characters. This eliminates duplicate instances with the same functionality
that only differ in whitespace characters, significantly reducing the dataset
size without compromising the diversity of the code snippets.

  * •

Identical source and target removal: Instances with identical source and
target are discarded. This also includes instances with both empty source and
target, and instances where their source and target only differ in comments.
Such instances do not represent actual bug fixes, and they provide no
meaningful information for learning.

  * •

Empty target filtering: Instances with empty targets are removed from the
training data. Although this may negatively affect the model’s ability to
generate deletion operator patches, we demonstrate that the model can still
effectively generate empty patches. In cases where the model does not produce
an empty patch because it is a single operator, we manually add an empty patch
to the beginning of the patch list.

  * •

Source length filtering: We filter instances based on the after-tokenization
length of their source (excluding context). This step ensures that the code
snippets are compatible with the model’s input length constraints and only
complete patches are used.

###  2.4 Code representation and tokenization

We represent buggy lines and their associated context in a unified format
using a special delimiter token `:` for the tokenization process:

    input = prefix buggy_lines : context

where `prefix` is a language-specific control code that distinguishes
different programming languages and is added to the beginning of each example,
following previous works that use T5-based models (Berabi et al., 2021; Raffel
et al., 2020; Wang et al., 2021). In the case of multiline bugs, we
concatenate lines using whitespace and put them right after each other. This
input is then tokenized and truncated if necessary to ensure that its size
remains below the model’s maximum limit. We only use instances that their
`prefix + buggy_lines` and `target_lines` lengths after tokenization are less
than or equal to the model’s maximum input and output sizes. Therefore, the
truncation only affects the context part of each instance, if necessary.

We also tokenize the target in the same manner but independently and without a
prefix or context. To tokenize inputs and targets, we use a pre-trained
RoBERTa-based subword tokenizer, which uses byte-level byte-pair-encoding
(BPE) (Sennrich et al., 2016). We use the tokenizer that comes with CodeT5 and
is trained to be efficient in tokenizing source code. By using a tokenizer
trained specifically on code, we can reduce the number of generated tokens,
which in turn improves model’s training performance and output generation.

Subword tokenization algorithms split rare words into smaller, meaningful
pieces, while leaving common words intact. A BPE tokenizer allows us to
include rare project-specific tokens in our vocabulary by breaking them into
smaller pieces (Jiang et al., 2021). Subword tokenization gives the model a
reasonable vocabulary size while trying to minimize the out-of-vocabulary
(OOV) problem (Karampatsis et al., 2020).

The final output comprises tokenized source-context pairs and tokenized target
labels. We represent the encoded input in the following format:

$\verb|<s>prefix |b_{1}b_{2}...b_{n}\verb|:|c_{1}c_{2}...c_{m}\verb|</s>|$

where $n$ and $m$ denote the number of buggy and context tokens, respectively.
Tokens `<s>` and `</s>` mark the beginning and end of a sequence. Here are
some examples to illustrate the encoding process:

For a Python source `a = 1 + 2` with a context `b = 3`, the encoded input
would be

<s>PythonĠaĠ=Ġ1Ġ+Ġ2Ġ:ĠbĠ=Ġ3</s>

and for a Java input `int a = 1 + 2;` with a context int `b = 3;`, the encoded
input would be

<s>JavaĠintĠaĠ=Ġ1Ġ+Ġ2;Ġ:ĠintĠbĠ=Ġ3;</s>

The `Ġ` token is used to represent space character since this tokenizer has
been trained to consider spaces as part of the tokens.

After tokenizing and preparing the data for each programming language, we
concatenate data from different programming languages into a single dataset.

###  2.5 Model architecture

Our base model is CodeT5, a state-of-the-art pre-trained transformer model
that can handle both natural language and source code. CodeT5 uses the same
encoder-decoder architecture as T5 (Raffel et al., 2020) but is pre-trained on
a large-scale dataset of code snippets and natural language descriptions from
various programming languages. The authors used the CodeSearchNet dataset
(Husain et al., 2020), and another dataset they collected from BigQuery, which
contains code snippets from eight programming languages (Ruby, JavaScript, Go,
Python, Java, PHP, C, and C#) and corresponding natural language descriptions
extracted from public code repositories.

CodeT5’s architecture consists of an encoder-decoder framework with multiple
layers of self-attention mechanism. The encoder produces hidden
representations of the input sequence, while the decoder uses them to generate
the output sequence. The self-attention mechanism enables the model to focus
on different parts of the input sequence and capture complex relationships
between different parts of the source code (Wang et al., 2021).

CodeT5 has been shown to achieve state-of-the-art performance on several code-
related tasks, including code generation, code summarization, code refinement,
and code completion. This means that CodeT5 has a strong foundation for
handling code-related tasks and can potentially learn to perform program
repair effectively (Jiang et al., 2021).

CodeT5 undergoes four pre-training tasks to acquire its code-aware
capabilities:

  1. 1.

Masked span prediction that randomly selects spans of arbitrary lengths to
mask and then uses the decoder to predict these masked spans marked with some
sentinel tokens.

  2. 2.

Identifier tagging that trains the model to understand whether a code token is
an identifier or not.

  3. 3.

Masked Identifier prediction that masks all identifiers in the code and uses a
unique mask token for all occurrences of one specific identifier.

  4. 4.

Bimodal dual generation that considers the generation of natural text from
source code and source code from natural text (NL $\leftrightarrow$ PL).

All these tasks are formulated as a sequence-to-sequence task. Tasks 1, 2, and
3 are part of identifier-aware denoising pre-training that enhances the
model’s understanding of code syntax, structure, and semantics. Task 4 is
effective for conversions between text and code, like generating comments for
a code or generating code snippets based on a description like GitHub Copilot.

###  2.6 Fine-tuning CodeT5

Fine-tuning is the process of adapting a pre-trained model to a specific task
using task-specific data. In our approach, we fine-tune the CodeT5 model for
multilingual APR. This involves training a unified model on a multilingual
dataset comprising multiple programming languages; in our case Java, Python,
C, and JavaScript. Our training data contains three columns: the buggy hunk
(source), the surrounding buggy context function, and the corresponding fixed
hunk (target).

The fine-tuning process adjusts the parameters of the pre-trained CodeT5 model
to better suit the specific task of APR by minimizing a cross-entropy loss
function that measures the discrepancy between the model’s predictions and the
ground-truth target fixes. As the model encounters task-specific data, it
continues to learn and update its parameters to improve performance on the
repair task. We fine-tune all languages simultaneously in batches that contain
samples from all programming languages while using a prefix to identify each
language. This approach leverages multitask learning (i.e., considering
repairing each language as a separate task), enlarging the dataset, and
allowing knowledge transfer between bug-fixing tasks in different languages.
By combining data from different languages (Section 2.4), we facilitate
multilingual learning, where the model learns from bugs and fixes across
various programming languages. Notably, this strategy proves particularly
effective for handling bugs that are common across all languages (Berabi et
al., 2021).

We fine-tune the model for a specified number of epochs denoted as $i$ while
saving a checkpoint every $j$ steps, resulting in $k$ checkpoints as shown in
Figure 2.

###  2.7 Checkpoint ensemble

Because of the diverse nature of bugs and fixes, a single model with optimal
parameters may not generalize well (Lutellier et al., 2020). Ensemble learning
has been a popular technique in machine learning for enhancing model
performance and robustness (Dietterich, 2000). In the context of transformer
models, ensemble learning involves training multiple instances of the same
model with different initialization or hyperparameters and then combining
their results to obtain a final output.

Prior approaches in APR have utilized ensemble models, which combine multiple
distinct models to generate bug-fixing patches (Jiang et al., 2021, 2023;
Lutellier et al., 2020). While this method has shown effectiveness in fixing
more bugs, it often entails a significant computational cost due to training
multiple specialized models with different inputs, hyperparameters, and, in
some cases, even distinct architectures.

In contrast, we adopt a checkpoint ensemble approach for T5APR, which not only
improves performance but also reduces training overhead (Chen et al., 2017;
Das et al., 2020). Instead of training separate models, we exploit the diverse
capabilities of the model at different training steps by saving and utilizing
multiple checkpoints. We save $k$ checkpoints during the model training
process, where $k$ represents the number of checkpoints used in the ensemble.
The saved checkpoints have complementary abilities to generate patches for
different types of bugs and contribute to the quality of patch ranking.

###  2.8 Patch generation and ranking

Having obtained $k$ checkpoints from the trained model, we now proceed with
generating patches for each bug. We apply the same data preparation and
tokenization steps to the localized buggy hunk and its context as described in
Section 2.4, with the only difference being that we truncate all long
instances to match the model size without discarding any of them.

In some cases, the buggy hunk is not within a function. Although our training
data always contains functions, we do not discard these bugs. Instead, we let
the context be empty and try to generate patches using only the buggy lines.

Next, we generate candidate patches for a given example using the model. To
achieve this, we use beam search with a specific beam size $t$ on each
checkpoint, resulting in the generation of $t$ best patches from each
checkpoint based on the maximum likelihood estimation score of each sequence.
In total, we obtain $k\times t$ patches through the checkpoint ensemble as
shown in Figure 2.

To consolidate the generated patches from different checkpoints, we combine,
deduplicate, and rerank patches by applying the following steps:

  1. 1.

We normalize whitespace characters in the generated patches to ensure
consistency.

  2. 2.

We merge and sort the patches for each hunk according to their checkpoint
ranks, breaking ties using the sequence scores (i.e., the likelihood score of
each sequence generated by each checkpoint).

  3. 3.

We remove patches that are identical to the buggy source, as they do not
contribute to the repair process.

  4. 4.

We deduplicate the patches, keeping only the unique ones, and retain the first
patch in the list in case of duplicates.

  5. 5.

Finally, to account for the possibility of removing buggy lines, we add an
empty patch at the beginning of the list for any bug that lacks one, as an
empty patch corresponds to removing the buggy lines.

For single-hunk bugs, the generated list is ready for validation. However, for
multi-hunk bugs (bugs that require changes in more than one code location), we
undertake additional processing to reduce the search space of patches (Saha et
al., 2019). Specifically, we focus on multi-hunk patches that exhibit the same
changes across all hunks (Madeiral and Durieux, 2021). We identify identical
patches among those generated for all hunks of a bug and retain only the
patches present in all hunks. The patches are then sorted based on the maximum
of sequence score among each hunk’s patches. Consequently, we obtain a single
patch that can be applied to all hunks, significantly reducing the number of
patches to validate.

These final candidate patches undergo further validation to select the correct
fixes.

###  2.9 Patch validation

Test-suite-based APR uses test-suite as program correctness specification. In
this stage, we validate the candidate patches obtained from the previous stage
by applying them to the original source code, compiling the patched code, and
running the developer-written test suite. The goal is to filter out patches
that do not compile or fail to pass the test cases in the project’s test
suite.

To validate the candidate patches, we follow these steps:

We apply each patch to the buggy location of the source code by replacing the
buggy lines with the corresponding fixed lines from the generated patch. We
then compile the patched code and run the test suite. To make this process
faster, if possible, we first run the triggering test cases, and if all of
them pass, we proceed to run the rest of the test cases that make the buggy
version pass to avoid regression. We make sure to omit flaky tests from this
process. The patched program is considered valid if it passes all the test
cases that the buggy project passed and passes the triggering test cases that
previously failed on the buggy project. This validation approach aligns with
common practices in many APR studies (Lutellier et al., 2020).

The resulting patches that pass the validation process are referred to as
plausible patches. However, it is essential to compare these plausible patches
to the ground truth (i.e., developer-written patches) to assess whether they
correctly fix the bug or only overfit to the test cases (Qi et al., 2015;
Smith et al., 2015).

##  3 Experimental setup

###  3.1 Research questions

The research questions that we aim to answer in this paper are:

  * •

RQ1 (Effectiveness and generalizability): How does T5APR compare with state-
of-the-art APR methods in terms of repair effectiveness and generalizability?

  * •

RQ2 (Multiple plausible patches): How does the consideration of multiple
plausible patches improve T5APR’s repair effectiveness?

  * •

RQ3 (Ablation study): What is the impact of checkpoint ensemble on T5APR’s
performance?

  * •

RQ4 (Multilingual and monolingual): How does the effectiveness of T5APR’s
multilingual model compare with monolingual models for each programming
language?

###  3.2 Datasets

#### Training data

We use the same dataset provided by CoCoNuT (Lutellier et al., 2020) on
GitHub111https://github.com/lin-tan/CoCoNut-Artifact for training the T5APR
model. The dataset consists of tuples of buggy, context, and fixed hunks of
code from the commit history of various open-source projects hosted on
platforms such as GitHub, GitLab, and BitBucket. The dataset covers multiple
programming languages including Java, Python, C, and JavaScript, making it
ideal for training a multilingual APR model. Table 1 provides a summary of the
dataset statistics, including the cutoff year of collected data, the number of
projects, instances before preprocessing, and the number of instances after
preprocessing and tokenization for each programming language.

CoCoNuT finds the date of the earliest bug in each evaluation benchmark and
collects commits that were made before that date and discards instances
committed after that to avoid overlapping train and evaluation data. The
cutoff year in Table 1 is the year that the data is collected until that year.
Other work has also used this data (Jiang et al., 2021, 2023; Ye et al., 2022;
Yuan et al., 2022).

Table 1: Summary of training data instances before and after preprocessing. Language | Cutoff year | Projects | Instances | After preprocessing | After size filtering
---|---|---|---|---|---
Java | 2006 | 45,180 | 3,241,966 | 1,125,599 | 1,009,268
Python | 2010 | 13,899 | 480,777 | 302,727 | 264,842
C | 2005 | 12,577 | 2,735,506 | 671,119 | 586,893
JavaScript | 2010 | 10,163 | 2,254,253 | 544,860 | 463,027

In our experiment, we set the maximum input length of the model to 512 tokens,
and the maximum output length to 256. Figure 4 shows the distribution of the
source and target of training data instances based on their number of tokens
after preprocessing but before size filtering. The $x$-axis indicates the
token length range, while the $y$-axis represents the count of instances.
Instances with shorter token lengths are more abundant, and as token length
increases, the count of instances gradually decreases. This figure shows that
our choice of maximum input and output length is reasonable and covers most of
the training data. Table 1 shows that from the total of 2,644,305 instances
after preprocessing, we retain 2,324,030 instances after size filtering, which
is 87.89% of the data.

Figure 4: Distribution of training data instances by their token length.

#### Bug benchmarks

We evaluate the performance of T5APR on a diverse set of benchmarks, spanning
multiple programming languages and encompassing various types of bugs
(Sobreira et al., 2018; Ye et al., 2021a). We use the following benchmarks in
our evaluation: Defects4J (Java) (Just et al., 2014), Bears (Java) (Madeiral
et al., 2019), QuixBugs (Java and Python) (Lin et al., 2017), Codeflaws (C)
(Tan et al., 2017), ManyBugs (C) (Le Goues et al., 2015), and BugAID
(JavaScript) (Hanam et al., 2016). These benchmarks collectively cover a wide
range of real-world software defects and coding challenges.

Defects4J is a database and framework of real-world bugs from 17 well-known
open-source Java projects. We follow prior work (Jiang et al., 2023; Ye et
al., 2022; Zhu et al., 2021) and separate Defects4J into two versions:
Defects4J (v1.2) and Defects4J (v2.0). Defects4J (v1.2) contains 395 bugs and
Defects4J (v2.0) contains 444 additional bugs that are only available in the
v2.0 version. Bears benchmark is a collection of bugs from 72 Java projects
hosted on GitHub and extracted using their continues integration status
history. QuixBugs contains 40 bugs from the Quixey Challenge problems in both
Java and Python. The programs are small classic algorithms in a single file.
Codeflaws is a set of bugs from Codeforces programming competition in C where
each program is a single file. ManyBugs contains bugs from large popular open-
source C projects. BugAID benchmark consists of 12 examples of common bug
patterns in JavaScript described in Hanam et al. (2016).

Table 2 provides detailed statistics for each benchmark. The table includes
the number of bugs present in each benchmark, the count of bugs that are
removed from consideration because they are either duplicates of other bugs or
their buggy and fixed version has no change, the remaining number of bugs
eligible for evaluation, and the total number of bugs that we attempt to
repair. These statistics offer insights into the scale of each benchmark and
the scope of our experimental evaluation.

Table 2: Evaluation benchmark statistics. Benchmark | Bugs | Removed | Remained | Attempted to Repair
---|---|---|---|---
Defects4J (v1.2) | 395 | 2 | 393 | 331
Defects4J (v2.0) | 444 | 0 | 444 | 357
Bears | 251 | 0 | 251 | 83
QuixBugs (Java) | 40 | 0 | 40 | 37
QuixBugs (Python) | 40 | 0 | 40 | 40
Codeflaws | 3,903 | 7 | 3,896 | 3,863
ManyBugs | 185 | 4 | 181 | 130
BugAID | 12 | 0 | 12 | 10
Total | 5,270 | 13 | 5,257 | 4,851

###  3.3 Implementation details and parameters

#### Implementation

We implement T5APR in Python and use the Hugging Face Transformers library
(Wolf et al., 2020) with PyTorch (Paszke et al., 2019) backend for training
the model. Data preparation and preprocessing are performed using the Hugging
Face Datasets library (Lhoest et al., 2021), which is based on Apache Arrow
for efficient data processing.

We use the CodeT5 checkpoints that were trained using identifier-aware
denoising pre-training objective for 100 epochs. CodeT5 has multiple variants
with different sizes and number of parameters. We fine-tune the small model
(`CodeT5-small`) that has a total of 60M parameters. Although bigger models
tend to perform better, it has been shown that the small model is also
relatively capable (Berabi et al., 2021; Wang et al., 2021). We leave using
other model sizes of CodeT5 to future work due to resource limitations.

The CodeT5 tokenizer’s vocabulary size is 32,100, of which 32,000 tokens are
obtained from the pre-training dataset with non-printable characters and low-
frequency tokens (occurring less than three times) filtered and 100 special
tokens for padding (`<pad>`), masking (`<mask>`), marking the beginning and
end of a sequence (`<s>, </s>`), and representing unknown tokens (`<unk>`).

The choice of hyperparameters, such as the learning rate, batch size, or
number of training epochs can have a significant impact on the performance of
the fine-tuned model. These hyperparameters are typically tuned using a
separate validation set, which is held out from the training data and used to
evaluate the model’s performance on unseen examples.

We employ the Optuna optimization framework (Akiba et al., 2019) and use the
AdamW optimizer (Loshchilov and Hutter, 2018) to conduct hyperparameter
search. We randomly divide our Python training dataset into a training and a
validation set, with 5,000 instances in the validation set and the rest in the
training set. This separation is only for hyperparameter tuning and later for
training, we use the entire Python dataset.

The evaluation criteria for hyperparameter tuning are the exact match and the
BLEU score (Papineni et al., 2002). Exact match is the ratio of instances
where the predicted sequence exactly matches the ground truth sequence. BLEU
score looks at how many n-grams in the model’s output match the n-grams in the
ground truth sequence and is a measure of the similarity between the model
output and the ground truth sequence, which we compute using the sacreBLEU
library (Post, 2018). We define the objective metric for hyperparameter
optimization as the sum exact match and BLEU score as follows:

$\text{objective metric}=\text{exact match}\times 100+\text{BLEU score}$ (1)

The BLEU score from sacreBLEU ranges from 0 to 100, with 100 being the best
possible score. Therefore, we multiply exact match by 100 to have the same
range values for both metrics.

The hyperparameter search space is defined as follows: The learning rate
ranges from $1e-5$ to $1e-3$, training epochs range from 1 to 5 epochs,
training batch size has a search range from 4 to 16, beam size to 5, and
learning rate scheduler type includes constant, cosine, linear, and
polynomial.

After hyperparameter tuning, we set the final hyperparameters as follows: the
train batch size is set to 8, the training epochs to 1, the learning rate to
$1e-4$, and the learning rate scheduler type to constant. We also use mixed
precision of FP16 for faster training. During training, we set $k=5$ and save
five checkpoints at each 20% steps of the training epoch.

In our decision to use five checkpoints, we draw inspiration from the previous
successful approaches of CoCoNuT, CURE, and KNOD, which also employ five to
ten models in their ensemble. This number has proven effective in related
work, and we adopt it as a reasonable starting point for our ensemble. We show
that more checkpoints result in more bug fixes, as also shown by CoCoNuT and
CURE.

For the final inference and patch generation of benchmarks, we set the beam
size to 100 to generate 100 patches from each checkpoint. A larger beam size
would improve the results (Tufano et al., 2019a; Ye et al., 2022), but due to
resource limitations, we chose a beam size of 100.

For parsing source files and extracting buggy context, we utilize the Tree-
sitter222https://tree-sitter.github.io/tree-sitter/ parsing library and the
lexers available in Pygments333https://pygments.org/. These libraries can
tokenize and parse many programming languages, making them suitable for our
multilingual approach. We also use
Unidiff444https://github.com/matiasb/python-unidiff to parse the diff of the
buggy and fixed codes and extract the location of buggy hunks.

#### Infrastructure

We train our model on a server with 4 cores of an Intel Xeon Platinum 8259CL
CPU, 16 GB RAM, and an NVIDIA T4 GPU with 16 GB VRAM. For evaluation, we use
another system with a 6-core Intel Core i7-8750H CPU, 16 GB RAM, and an NVIDIA
GeForce GTX 1060 GPU with 6 GB VRAM.

###  3.4 Patch assessment

Patches that can be compiled and pass the project’s test suite are called
plausible. However, a plausible patch may not fix the bug if the test suite is
weak and does not cover all the cases (Qi et al., 2015). This is called the
overfitting problem when the patch only works for the test cases and not for
the problem (Smith et al., 2015). Therefore, we use the following criteria to
determine if a plausible patch is correct (Ye and Monperrus, 2023):

  * •

It is identical to the developer-provided patch.

  * •

It is identical to correct patches generated by existing techniques that have
undergone public review by the community in open-source repositories.

  * •

We judge it semantically equivalent to the developer-provided patch using
rules described by Liu et al. (2020).

To reduce the potential for errors in this process, we make all generated
patches publicly available for public judgment and review.

###  3.5 Analysis procedure

We compare T5APR against recent approaches that report their results under
perfect fault localization setting. Techniques that use perfect fault
localization are given the exact location of the bug. We identify the location
of each bug with the help of human-written patches and their diff. Some
approaches use different fault localization algorithms or implementations to
find the buggy location, which makes it difficult to only compare the repair
capabilities of each approach. Recent studies suggest that perfect fault
localization is the preferred way to evaluate APR approaches, as it allows a
fair comparison of APR techniques without depending on the fault localization
method (Liu et al., 2019a, 2020).

We compare T5APR with 11 state-of-the-art tools including eight Java APR
tools: SequenceR (Chen et al., 2019), TBar (Liu et al., 2019b), DLFix (Li et
al., 2020), CURE (Jiang et al., 2021), Recoder (Zhu et al., 2021),
RewardRepair (Ye et al., 2022), and KNOD (Jiang et al., 2023). One C tool:
SOSRepair (Afzal et al., 2019). Two tools that use large language models:
Codex (Prenner et al., 2022) and ChatGPT (Sobania et al., 2023). Lastly,
CoCoNuT (Lutellier et al., 2020), which is evaluated on all four programming
languages.

We did not include CIRCLE (Yuan et al., 2022) in the evaluation since they
have not validated their candidate patches using benchmarks’ test suite and
only reported exact match results across their generated patches.

To compare with these approaches, we follow the previous works and only
consider the first plausible patch generated by T5APR that successfully
compiles and passes the test suite (Durieux et al., 2019; Liu et al., 2019b;
Lutellier et al., 2020; Zhong et al., 2023). There might be correct patches
further down the plausible patch list, but we analyze those in another
section.

We obtain results from each tool’s paper or repository. For those who do not
report results with perfect localization or for some benchmark in their paper,
we use results from other publications that evaluate said tools under perfect
localization setting (Liu et al., 2020; Zhong et al., 2023).

To compute patch ranking, compilable patch rate, and unique bugs that T5APR
can fix compared with other approaches, we obtain the list of candidate
patches and fixed bugs for each approach on each benchmark from their
respective repositories (Those that provide it).

##  4 Results and discussion

###  4.1 RQ1: Effectiveness and generalizability

Table 3 presents the results of evaluating the performance of T5APR on
multiple benchmarks and against a selection of state-of-the-art APR tools. The
table shows the name and the total number of considered bugs for each
benchmark below its name. The results are displayed as $c/p$, where $c$ is the
number of correct patches that are ranked first as the first plausible patch
by an APR technique, and $p$ is the total number of plausible patches. We also
show, in parentheses, the number of bugs that have identical patches to the
developer-written patch. A dash (-) indicates that the tool has not been
evaluated on the benchmark or does not support the programming language of the
benchmark, to the best of our knowledge. For the ManyBugs and BugAID
benchmarks, we could not validate their patches; therefore we cannot show the
number of plausible patches, and only report the number of correct patches
that we manually identified.

We highlight T5APR’s performance on these benchmarks as follows. Overall,
T5APR fixes 1,985 bugs across all the benchmarks with 1,413 of them patched
identical to the developer’s patch. Results show that T5APR outperforms or
equals all other approaches on all the evaluated benchmarks except for
Defects4J (v1.2) and ManyBugs.

For Defects4J, we observe that T5APR achieves competitive results,
particularly in Defects4J (v2.0) where it generates correct fixes for 56 bugs
and outperforms all other approaches. In Defects4J (v1.2), KNOD performs
better than T5APR by fixing 71, while T5APR fixes 67 bugs. It should be noted
that CoCoNuT, CURE, and KNOD use a substantially larger beam size of 1000
versus 100 that T5APR uses, and it has been shown that using larger beam size
leads to more correct patches (Tufano et al., 2019a; Ye et al., 2022). Notice
that if we consider all the generated plausible patches (Table 7), T5APR
reaches 72 correct bugs. This shows the need for a better patch ranking
strategy in future studies (Kang and Yoo, 2022).

In the case of Bears, T5APR demonstrates its potential by correctly repairing
24 bugs, outperforming all the compared tools. Results in these two benchmarks
indicate T5APR’s capability in addressing real-world bugs in Java programs.

In the QuixBugs benchmark, which encompasses both Java and Python programs,
T5APR shows robust performance, repairing 25 Java bugs and achieving correct
repair for 29 Python bugs. The correct to plausible patch ratio for both
versions is about 96%, where only one of the generated plausible patches is
not correct. In the QuixBugs (Java) version, T5APR fails to fix the SQRT bug,
and in the QuixBugs (Python) version, it fails to fix DEPTH_FIRST_SEARCH bugs
where it is correctly fixed further down in the patch list.

Similarly, in Codeflaws, a C programming language benchmark, T5APR showcases
its robustness by achieving a substantial repair of 1,764 bugs, outperforming
its only other contender CoCoNuT. Turning to the ManyBugs benchmark, T5APR
performance remains competitive by repairing 15 bugs, positioning itself among
the top-performing tools but outperformed by SOSRepair. In the BugAID
benchmark, T5APR achieves the highest repair rate, successfully fixing 5 out
of the 12 bugs, demonstrating its competence in addressing JavaScript bugs.

Out of the bugs fixed by T5APR for Defects4J (v1.2), Defects4J (v2.0), Bears,
Codeflaws, and ManyBugs benchmarks, 10, 4, 4, 36, and 2 of them are multi-
hunk, respectively. These are fixed using the strategy described in Section
2.8. The remaining benchmarks either do not contain multi-hunk bugs or are not
successfully fixed by T5APR.

Comparing T5APR against existing state-of-the-art methods, we consistently
observe competitive or superior performance across various benchmarks. The
overall results suggest T5APR’s effectiveness in repairing a wide range of
software defects and its effectiveness in handling bugs from different
programming languages.

Table 3: The number of correctly fixed bugs and comparison with state-of-the-art approaches. Results are from the first plausible patch by each method and are shown as correct/plausible. Values in parentheses are bugs with identical patches to the developer’s. The highest number of correct patches for each benchmark is highlighted in bold. Tool | Defects4J (v1.2) | Defects4J (v2.0) | Bears | QuixBugs (Java) | QuixBugs (Python) | Codeflaws | ManyBugs | BugAID
---|---|---|---|---|---|---|---|---
| 393 bugs | 444 bugs | 251 bugs | 40 bugs | 40 bugs | 3,896 bugs | 181 bugs | 12 bugs
SOSRepair (Afzal et al., 2019) | - | - | - | - | - | - | 16/23 | -
SequenceR (Chen et al., 2019) | 12/19 | - | 16/26 | 15/16 | - | - | - | -
TBar (Liu et al., 2019b) | 53/84 | - | - | - | - | - | - | -
DLFix (Li et al., 2020) | 39/68 | - | - | - | - | - | - | -
CoCoNuT (Lutellier et al., 2020) | 44/85 | 21/31 | 19/33 | 13/20 | 19/21 | 423/716 | 7/- | 3/-
CURE (Jiang et al., 2021) | 57/104 | 19/- | - | 25/34 | - | - | - | -
Recoder (Zhu et al., 2021) | 64/69 | - | 5/17 | 17/17 | - | - | - | -
RewardRepair (Ye et al., 2022) | 45/- | 45/- | - | 20/- | - | - | - | -
Codex (Prenner et al., 2022) | - | - | - | 14/- | 23/- | - | - | -
ChatGPT (Sobania et al., 2023) | - | - | - | - | 19/- | - | - | -
KNOD (Jiang et al., 2023) | 71/85 | 50/82 | - | 25/30 | - | - | - | -
T5APR | 67/94 (46) |  56/103 (35) |  24/33 (12) |  25/26 (18) |  29/30 (24) |  1,764/2,359 (1,259) | 15/- (14) |  5/- (5)
Total |  |  |  | 1,985 (1,413) |  |  |  |

#### Patch ranking

To provide a comprehensive understanding of the effectiveness of T5APR’s patch
ranking strategy, we analyze the ranking of correct patches generated by each
approach. Figure 5 presents the patch ranking information at different
thresholds. Each line in the plot corresponds to a different approach,
including T5APR, CURE, RewardRepair, and KNOD. The $x$-axis represents the
ranking thresholds, while the $y$-axis indicates the number of correctly fixed
bugs in each threshold. We only consider tools that we have access to their
patch ranking information and generated candidate patches.

T5APR outperforms all other approaches in correct patch ranking except for
Top-200 on QuixBugs (Java) where CURE performs better. We can also see that
although KNOD reaches better results than T5APR for Defects4J (v1.2) (71 vs.
67 bugs in Table 3), T5APR fixes more bugs up to the Top-500 generated
candidate patches. KNOD generates fixes for the rest of the bugs in ranks
higher than 500 due to using a larger beam size. Overall, 310 of the correct
patches generated by T5APR are ranked first in the candidate patch list.

Figure 5: Ranking information of correct patches.

#### Unique bug fixes

Figure 6 presents the unique and overlapped number of bugs repaired by
individual approaches from Table 3 for all the benchmarks. For benchmarks with
more than four tools, we select the three best tools for that benchmark and
combine the fixed bugs of the remaining tools under “Others”. Across all the
benchmarks, T5APR fixes 1,442 bugs that other tools do not fix. On the
Defects4J benchmark, T5APR and KNOD complement each other by fixing 21 and 25
unique bugs for v1.2 and v2.0, respectively. On the ManyBugs benchmark, T5APR
has good complementary quality and together with SOSRepair fixes 20 unique
bugs. Overall, results show that T5APR complements compared existing works on
all the evaluated benchmarks.

(a) Defects4J (v1.2)

(b) Defects4J (v2.0)

(c) Bears

(d) QuixBugs (Java)

(e) QuixBugs (Python)

(f) Codeflaws

(g) ManyBugs

(h) BugAID

Figure 6: Number of unique and overlapped bug fixes of T5APR and other tools.

We provide a few examples of the bugs that T5APR can fix. Figure 7(c) shows
the fix generated by T5APR and the ground-truth patch for Codec 5 bug from
Defects4J (v2.0) benchmark that other tools do not fix. To fix this bug, T5APR
notices that the header of the context method has `throws ParseException`,
which prompts T5APR to synthesize an exception throw statement. The only
difference between T5APR and the developer’s patch is the exception message,
where T5APR uses the string parameter of the method while the developer writes
a custom message.

public static <T> T createValue(final String str, final Class<T> clazz) throws
ParseException /⋮/ \- return null; \+ throw new ParseException(str);

(a) T5APR’s patch.

⬇

- return null;

+ throw new ParseException("Unable to handle the class: " + clazz);

(b) Developer-written patch.

(c) Fix for Cli 40 bug from Defects4J (v2.0) benchmark.

Figure 7 gives the T5APR patch for Bears-32, which is only fixed by T5APR. For
this bug, T5APR adds a null check for the returned value of ``getStep()|,
which is a common pattern in Java. The developer’s patch for this bug is shown
in Figure 3(b). T5APR’s patch is semantically equivalent to the developer’s
patch. Figure 8 shows another Bears benchmark bug patched by T5APR. This is a
complete generation patch for Bears-46 with no lines to remove and is
identical to the developer’s patch. For this bug, T5APR also generates a null
check and then based on the ``Set| return type of the context method, returns
an empty set collection. Figure 9 shows the patch for the same bug LIS from
QuixBugs benchmark in both Java (Figure 9(a)) and Python (Figure 9(b)). The
generated patch is similar for both languages, but T5APR adapts the syntax for
each programming language. Figure 10 shows the T5APR and developer’s patch for
multi-hunk bug 465-B-bug-16282461-16282524 from Codeflaws with identical
patches for both hunk locations. T5APR finds the fix lower in the context and
copies it to the buggy location. It also adds an if condition before it to
avoid changing it if it is already true.

⬇

- return getStep().isFailOnCCE();

+ return getStep()!= null && getStep().isFailOnCCE();

Figure 7: T5APR’s fix for Bears-32 bug from Bears benchmark.

⬇

public Set<String> getMetadataKeys() {

+ if (metadata == null) {

+ return Collections.EMPTY_SET;

+ }

return metadata.keySet();

}

Figure 8: T5APR’s fix for Bears-46 bug from Bears benchmark.

⬇

if (length == longest || val < arr[ends.get(length+1)]) {

ends.put(length+1, i);

- longest = length + 1;

+ if (length == longest) { longest = length + 1; }

}

(a) T5APR’s Java patch.

⬇

if length == longest or val < arr[ends[length + 1]]:

ends[length + 1] = i

- longest = length + 1

+ if length == longest: longest = length + 1

(b) T5APR’s Python patch.

⬇

- longest = length + 1;

+ longest = Math.max(longest,length + 1);

(c) Developer’s Java patch.

⬇

- longest = length + 1

+ longest = max(longest, length + 1)

(d) Developer’s Python patch.

Figure 9: Fix for LIS bug from QuixBugs benchmark.

⬇

for(i = 1; i < letters; ++i) {

current_letter = atoi(strtok(NULL, " "));

if(current_letter && !inside_letter && actions != 0) {

+ if(!inside_letter) inside_letter = true;

actions += 2;

}

else if(current_letter) {

+ if(!inside_letter) inside_letter = true;

++actions;

}

else {

inside_letter = false;

}

}

(a) T5APR’s patch.

⬇

\-

+ inside_letter = true;

(b) Developer’s patch.

Figure 10: Fix for 465-B-bug-16282461-16282524 from Codeflaws benchmark.

#### Compilable patch rate

Furthermore, we evaluate the compilable patch rate of candidate patches
generated by different approaches, which reflects the tool’s ability to
generate syntactically correct and developer-like code. Table 4 presents the
average compilation rate across different top-X values of generated candidate
patches. T5APR has a slightly lower compilation rate than RewardRepair for all
the top-X values. RewardRepair has a semantic training step that rewards
compilable patches in its model training through backpropagation, which
greatly helps it in generating more compilable candidate patches. All tools
have a decreasing compilation rate as the X value increases, which means that
tools generate more non-compilable patches as they generate more candidate
patches (Ye et al., 2022). We use an interval for the top-200 of CoCoNuT and
CURE since they only report compilation rate for top-100 and top-1000 values.
Moreover, we filtered our considered bugs to be in the same set of bugs as
RewardRepair for this comparison since we target more bugs. Like previous
approaches (Jiang et al., 2021; Ye et al., 2022), we combine Defects4J (v1.2)
and QuixBugs (Java) patches. We directly list the numbers reported by Ye et
al. (2022) for SequenceR, CoCoNuT, CURE, and RewardRepair.

Table 4: Average compilable patch rate of the Top-X candidate patches in Defects4J (v1.2) and QuixBugs (Java). Model | Top-30 | Top-100 | Top-200
---|---|---|---
SequenceR | 33% | - | -
CoCoNuT | 24% | 15% | 6%-15%
CURE | 39% | 28% | 14%-28%
RewardRepair | 45.3% | 37.5% | 33.1%
T5APR | 42.7% | 36.2% | 32.3%

Table 5 shows the average compilable patch rate of the top-X candidate patches
for each benchmark. The Codeflaws benchmark has the highest compilation rate
among all the benchmarks, followed by QuixBugs (Java). The Bears benchmark has
the lowest compilation rate among all the benchmarks, followed by Defects4J
(v2.0). This may indicate some characteristics of Codeflaws and QuixBugs
(Java) bugs that make them easier to compile than other benchmarks, and some
characteristics of Bears and Defects4J (v2.0) bugs that make them harder to
compile.

Table 5: Average compilable patch rate of the Top-X candidate patches. Benchmark | Top-30 | Top-100 | Top-200
---|---|---|---
Defects4J (v1.2) | 33.2% | 30.1% | 27.7%
Defects4J (v2.0) | 31.8% | 28.8% | 26.3%
Bears | 34.6% | 26.4% | 23.6%
QuixBugs (Java) | 49.1% | 42.6% | 37.1%
Codeflaws | 73.5% | 70.3% | 67.3%
Overall | 67.8% | 64.8% | 61.9%

#### Validation time cost

We also assess the time cost of the validation effort required to reach
correct patches. Table 6 provides an overview of the time takes to find
correct patches, the number of validated patches until the first correct patch
is found, and the maximum instances of timeouts to reach a correct patch.
Codeflaws has both the lowest and highest time to reach a correct patch
because Codeflaws programs are small and fast to compile and test. Most of the
time consumed in validating benchmarks is for bugs with timeout patches since
we have to wait for each patch for a certain time to timeout.

Table 6: Statistics for validation until correct patch. Time is in HH:MM:SS.fff format. Benchmark | Time to correct | Validated until correct | Timeouts
---|---|---|---
min | max | median | mean | min | max | median | mean | max
Defects4J (v1.2) | 00:00:06.962 | 00:37:33.473 | 00:01:23.657 | 00:03:07.288 | 1 | 234 | 8.0 | 31.22 | 2
Defects4J (v2.0) | 00:00:04.437 | 00:18:58.074 | 00:01:00.709 | 00:02:54.237 | 1 | 322 | 12.0 | 53.66 | 3
Bears | 00:00:20.088 | 01:24:41.593 | 00:03:21.152 | 00:10:33.795 | 1 | 194 | 8.5 | 32.42 | 0
QuixBugs (Java) | 00:00:07.328 | 00:29:58.495 | 00:00:48.338 | 00:04:12.138 | 1 | 250 | 14.0 | 56.64 | 13
QuixBugs (Python) | 00:00:00.630 | 00:20:09.557 | 00:00:02.718 | 00:01:27.222 | 2 | 269 | 7.0 | 58.76 | 20
Codeflaws | 00:00:00.333 | 02:33:47.090 | 00:00:04.578 | 00:00:55.622 | 1 | 358 | 9.0 | 37.74 | 150
Overall | 00:00:00.333 | 02:33:47.090 | 00:00:05.748 | 00:01:13.520 | 1 | 358 | 9.0 | 38.45 | 150

Overall in our multilingual experiment, we validated 1,172,267 patches across
all the benchmarks that took about 27 days. Training the multilingual model
for one epoch on a single GPU took about 17 hours.

###  4.2 RQ2: Multiple plausible patches

Table 7 compares the results when only the first plausible patch is considered
versus when all plausible patches are considered. The “top-X” thresholds
consider only the first X plausible patches generated. The “all” threshold
encompasses all plausible patches generated for each bug.

We find 2,309 correct patches when we consider all plausible patches, which is
an increase of 344 from the first plausible patch. This means that 344 correct
patches are not ranked as the first plausible patch by T5APR and an incorrect
patch passes the test cases due to test suite limitation. This limitation is
an issue that most test suite-based APR tools have in common. The number of
correct patches increases the most when we raise the threshold from top-1 to
top-5, which means that most of the correct patches are ranked within the
top-5 plausible patches by T5APR. This is important since according to a
recent study, 72% of developers are only willing to review up to five patches
in practice (Noller et al., 2022). The increase in the number of correct
patches is smaller when the threshold is raised from top-5 to all.

Table 7: Number of correctly fixed bugs based on plausible ranking when all plausible patches are considered. Benchmark | Top-1 | Top-5 | Top-10 | All
---|---|---|---|---
Defects4J (v1.2) | 67 | 72 | 72 | 72
Defects4J (v2.0) | 56 | 64 | 65 | 65
Bears | 24 | 25 | 25 | 26
QuixBugs (Java) | 25 | 25 | 25 | 25
QuixBugs (Python) | 29 | 30 | 30 | 30
Codeflaws | 1,764 | 1,990 | 2,017 | 2,071
ManyBugs | - | - | - | 15
BugAID | - | - | - | 5
Total | 1,965 | 2,206 | 2,234 | 2,309

###  4.3 RQ3: Ablation study

We employ an ensemble approach, combining the outputs of individual
checkpoints, to enhance the overall performance of T5APR. Table 8 shows the
performance of each checkpoint independently for different benchmarks. Each
checkpoint result includes the manually added deletion patch. Compared with
Table 3, we can see that the combination of checkpoints has better results
than each checkpoint independently. This demonstrates the effectiveness of our
ensemble approach (Dietterich, 2000). Overall, checkpoint5 fixes more bugs
than others, but considering each benchmark independently we can see that the
best performing checkpoint varies. This suggests that checkpoints complement
each other and learn different patterns for different bugs.

Table 8: Result of each checkpoint independently shown as correct/plausible. This also includes the manually added patch to each checkpoint. Benchmark | checkpoint1 | checkpoint2 | checkpoint3 | checkpoint4 | checkpoint5
---|---|---|---|---|---
Defects4J (v1.2) | 51/73 | 55/75 | 56/78 | 53/78 | 59/78
Defects4J (v2.0) | 43/76 | 45/78 | 51/85 | 49/84 | 50/83
Bears | 14/27 | 15/26 | 20/32 | 19/29 | 18/27
QuixBugs (Java) | 15/17 | 15/16 | 16/17 | 18/18 | 18/18
QuixBugs (Python) | 18/19 | 19/20 | 22/23 | 23/24 | 24/24
Codeflaws | 1,317/1,981 | 1,318/1,959 | 1,374/2,012 | 1,381/2,028 | 1,379/2,011
ManyBugs | 11/- | 12/- | 12/- | 13/- | 12/-
BugAID | 5/- | 5/- | 5/- | 5/- | 5/-
Total | 1,474/2,193 | 1,484/2,174 | 1,556/2,247 | 1,561/2,261 | 1,565/2,241

We further analyze how each checkpoint contributes to the overall pool of
correct patches. Table 9 shows the contribution of each checkpoint and the
manual empty patch. Almost all the checkpoints contribute to the final
results. The added manual deletion patch also has a positive contribution.

Table 9: How many of the patches come from each checkpoint. Results are shown as correct/plausible. Benchmark | manual | checkpoint1 | checkpoint2 | checkpoint3 | checkpoint4 | checkpoint5
---|---|---|---|---|---|---
Defects4J (v1.2) | 8/10 | 11/15 | 8/12 | 19/25 | 10/19 | 11/13
Defects4J (v2.0) | 8/11 | 4/14 | 10/16 | 14/23 | 9/16 | 11/23
Bears | 2/4 | 1/1 | 1/1 | 12/14 | 6/8 | 2/5
QuixBugs (Java) | 1/1 | 9/10 | 5/5 | 6/6 | 4/4 | 0/0
QuixBugs (Python) | 4/5 | 5/5 | 4/4 | 8/8 | 8/8 | 0/0
Codeflaws | 240/291 | 268/389 | 301/406 | 306/398 | 346/462 | 303/413
ManyBugs | 2/- | 2/- | 2/- | 3/- | 3/- | 3/-
BugAID | 0/- | 0/- | 3/- | 2/- | 0/- | 0/-
Total | 265/322 | 300/434 | 334/444 | 370/474 | 386/517 | 330/454

Figure 11 shows the result of incrementally adding the patches of each
checkpoint to the generated patches of previous checkpoints. We can see that
for most benchmarks, adding more checkpoints leads to better results both when
considering the first plausible patch and all the plausible patches. However,
for some benchmarks, such as BugAID, adding more checkpoints does not improve
the results. This confirms the similar findings of Lutellier et al. (2020).

Figure 11: Results with incremental addition of checkpoints.

###  4.4 RQ4: Multilingual and monolingual

In addition to the multilingual model, we also train models under the same
setting as the multilingual model but only using training data of a single
programming language. We then use monolingual models to generate patches for
the benchmarks in the same language. Table 10 shows the comparison of the
first plausible correct patches of multilingual and monolingual models. The
multilingual model outperforms the monolingual models for most benchmarks,
except for ManyBugs and BugAID. Note that for ManyBugs and BugAID, running the
validation step could change the results, and there might be correct patches
that we have missed. Furthermore, the multilingual model fixes 426 unique bugs
across all the benchmarks that the monolingual models do not fix, while the
monolingual models fix 120 unique bugs. This highlights the benefit of
leveraging multiple programming languages for training as it transfers bug
patterns across languages.

Table 10: Comparison of results of multilingual and monolingual models. Results are shown as correct/plausible. Benchmark | Multilingual | Monolingual
---|---|---
Defects4J (v1.2) | 67/94 | 55/93
Defects4J (v2.0) | 56/103 | 52/98
Bears | 24/33 | 21/34
QuixBugs (Java) | 25/26 | 21/24
QuixBugs (Python) | 29/30 | 26/28
Codeflaws | 1,764/2,359 | 1,482/2,357
ManyBugs | 15/- | 16/-
BugAID | 5/- | 6/-
Total | 1,985/2,645 | 1,679/2,634

###  4.5 Threats to validity

In this section, we outline possible threats that could impact the validity of
our experimental findings and discuss how we mitigated them.

A major threat to internal validity is the potential fault in manual patch
correctness assessment, which may result in misclassification or bias due to a
lack of expertise or mistakes (Ye et al., 2021b). This is a common threat for
all program repair results based on manual assessment (Ye et al., 2021a). To
alleviate this threat, we compared our patches to patches generated by
existing tools and carefully checked their semantic equivalency to reference
developer patches. For results of other tools, we use the reported performance
number in the paper of approaches and cross-checked them with patches in their
repositories. Another threat to internal validity relates to potential fault
in the implementation and hyperparameter configuration we used. We have
double-checked our implementation, and to ensure reproducibility, we used
fixed manual seed values wherever possible. To further mitigate these threats,
we make all generated patches and our source code publicly available for
verification and review by other researchers.

A third threat to internal validity comes from using CodeT5 as our base model,
which is trained on large amounts of open-source code snippets. This means
that its training data could overlap with our evaluation benchmarks. This
issue is hard to address since retraining this model would require significant
resources. However, some factors could mitigate this concern: First, the
overlapping data, if any, would be a very small fraction of the training data.
Second, both the correct and incorrect program versions would likely be
present in the training data without any labels indicating which one is
correct or incorrect since the pre-training objective of CodeT5 is different,
and it was never specifically trained for the task of program repair. The same
issue is present in approaches that use Codex or ChatGPT models (Prenner et
al., 2022; Sobania et al., 2023). Our APR training data is collected up to the
date of bugs in our evaluation benchmark to avoid any overlap.

A threat to external validity is a threat that our approach might not be
generalizable to fixing bugs outside the tested bugs benchmarks as shown by
Durieux et al. (2019) by the phenomenon of “benchmark overfitting” in program
repair. We use six different benchmarks in four different programming
languages with up to 5,257 real-world bugs to address this issue. Evaluation
on more benchmarks (e.g., Bugs.jar (Saha et al., 2018), BugsJS (Gyimesi et
al., 2019), and BugsInPy (Widyasari et al., 2020)) could be done in the
future.

##  5 Related work

Automated program repair (APR) is a rapidly evolving and diverse field with a
wide range of research and development efforts. In this section, we highlight
some of the works that closely relate to our work while acknowledging that
there are many other important contributions. For a more comprehensive survey
of APR literature, we direct readers to recent surveys in the field (Gao et
al., 2022; Huang et al., 2023; Le Goues et al., 2019; Monperrus, 2018b).

One well-established class of APR techniques is search-based methods that
involve syntactic manipulation of code by applying different edit patterns.
These techniques use a search algorithm to iteratively explore the space of
possible code changes in order to find a plausible patch. Examples of such
techniques include GenProg (Le Goues et al., 2012), SimFix (Jiang et al.,
2018), and VarFix (Wong et al., 2021), among others.

Another class of APR approaches involves semantic analysis of code. These
techniques use a set of constraint specifications to transform the program
repair problem into a constraint solver problem and identify potential fixes
that preserve the program’s intended semantics. Examples of these techniques
include SemFix (Nguyen et al., 2013), Angelix (Mechtaev et al., 2016), and
SOSRepair (Afzal et al., 2019).

Template-based methods generate repair patches by applying a predefined
program fix template to the faulty code. A fix template specifies how to
modify the code to fix a certain type of bug. Fix templates can be either
manually defined (Liu et al., 2019a) or automatically mined from code
repositories (Koyuncu et al., 2020; Liu et al., 2019b).

Recently, learning-based techniques have gained traction in automatically
fixing software bugs by learning from extensive code repositories. These
techniques mostly employ neural networks to automatically generate correct
code from buggy source code and are called neural program repair tools. These
tools use a variety of sequence-to-sequence, neural machine translation (NMT),
graph-to-sequence, and other deep learning models to generate patches as
sequences of tokens (Chen et al., 2019; Jiang et al., 2021; Lutellier et al.,
2020; Ye et al., 2022) or edits (Chakraborty et al., 2022; Ding et al., 2020;
Li et al., 2020). Thanks to the strong learning capabilities of these models,
neural program repair techniques learn to capture the relationship between
buggy and correct code, eliminating the need for manual design of fix patterns
or feature templates and have outperformed many existing approaches. These
works mostly use supervised learning on past bug-fixing commits. There are
also works that use self-supervised learning that automatically generate
training samples (Allamanis et al., 2021; Yasunaga and Liang, 2020, 2021; Ye
et al., 2023). Our work uses a sequence-to-sequence model and belongs to the
supervised learning category.

Tufano et al. (2019b) present an empirical study on using NMT to learn how to
fix bugs in Java code. The authors mine a large dataset of bug-fixing commits
from GitHub and extract method-level pairs of buggy and fixed code. They
abstract the code to reduce the vocabulary size and train an encoder-decoder
model to learn how to transform buggy code into fixed code. Chen et al. (2019)
introduce SequenceR, an end-to-end approach to program repair based on
sequence-to-sequence learning on source code that sees the APR task as a
translation from buggy to correct source code. A copy mechanism is used to
handle the large and unlimited vocabulary of code, and an abstract buggy
context is constructed to capture the relevant information for generating
patches. The model is trained on a large dataset of one-line bug-fixing
commits from GitHub and evaluated on CodRep (Chen and Monperrus, 2018) and
Defects4J (Just et al., 2014) benchmarks. Zhu et al. (2021) present Recoder,
an approach for APR that uses a syntax-guided edit decoder with a
provider/decider architecture to generate patches that are syntactically
correct and context-aware. Recoder also introduces placeholder generation to
handle project-specific identifiers and generates edits rather than modified
code. KNOD is proposed by Jiang et al. (2023), which presents a tree decoder
and a domain-rule distillation module. The tree decoder directly generates
abstract syntax trees of patches in three stages: parent selection, edge
generation, and node generation. The domain-rule distillation module enforces
syntactic and semantic rules on the decoder during both training and inference
phases.

Ye et al. (2022) propose RewardRepair, a neural repair model that incorporates
compilation and test execution information into the training objective.
RewardRepair uses a discriminative model to reward patches that compile and
pass test cases, and penalize patches that are identical to the buggy code or
introduce regression errors. The reward signal modulates the cross-entropy
loss before backpropagation, guiding the neural network to generate high-
quality and more compilable patches as shown in Table 4. RewardRepair has two
training phases, a syntactic training and a semantic one.

The work by Jiang et al. (2021) introduces CURE, a code-aware NMT technique
for APR. CURE uses three techniques to improve the search space and the search
strategy for generating patches: subword tokenization, pre-trained programming
language model, and code-aware beam search to generate more compilable
patches. CURE demonstrates the effectiveness of applying code awareness to NMT
models for the APR task. CURE differs from our approach in several aspects.
CURE pre-trains its model exclusively using Java code and a standard causal
language modeling task for pre-training a GPT model, while we use CodeT5, an
encoder-decoder model that is pre-trained on multiple programming languages
and more diverse pre-training tasks to better understand source code.
Additionally, CURE’s tokenizer is trained on their Java corpus, while ours is
trained on the multilingual corpus of CodeT5. In contrast to CURE, we use the
vanilla beam search to generate patches, which is simpler and faster than the
code-aware one that CURE uses but may be less effective. However, beam search
is an independent component, and we can incorporate the code-aware beam search
into our tool in future work.

Most of these works target Java or are only evaluated on Java language
benchmarks. Two works that are closest to our work and are evaluated on
multiple programming languages are CoCoNuT by Lutellier et al. (2020) and
CIRCLE by Yuan et al. (2022).

CoCoNuT is an APR technique that uses NMT to learn from bug fixes in open-
source repositories. CoCoNuT has three main contributions: A context-aware NMT
architecture that uses two separate encoders with fully convolutional layers
to represent the buggy line and its surrounding context; An ensemble approach
that combines different models with different levels of complexity to capture
various repair strategies; Cross-language portability that allows CoCoNuT to
be applied to four programming languages (Java, Python, C, and JavaScript).
Our approach differs from this work in several ways. First, our approach can
handle multiple programming languages with a unified model, unlike CoCoNuT,
which requires individual models for each programming language. Second, we use
a pre-trained programming language model that learns from a large software
codebase to capture code syntax and developer-like coding style. Third, we use
checkpoint ensemble for training efficiency, as opposed to CoCoNuT’s model
ensemble for each language. We combine these techniques to form a novel APR
architecture using a text-to-text transformer model for patch generation
across multiple programming languages.

Yuan et al. (2022) propose a method for APR that can handle multiple
programming languages using continual learning. The method consists of five
components: a prompt-based data representation, a T5-based model, a
difficulty-based example replay, an elastic-based parameter updating
regularization, and a re-repairing mechanism. The prompt-based data
representation converts the bug-fixing task into a fill-in-the-blank task that
is suitable for the pre-trained T5 model. The difficulty-based example replay
and the elastic-based regularization are two continual learning strategies
that prevent the model from catastrophic forgetting. The re-repairing
mechanism is a post-processing step that corrects the errors caused by
crossing languages. The major differences between CIRCLE and our work are the
following: We formulate APR as a multitask learning task, which is simpler and
allows the trained model to remain relevant for a long time (Lutellier et al.,
2020), so we do not need frequent retraining. By using a specific control
prefix for each language, we do not need re-repairing of generated patches and
we get patches in the correct syntax of the target language. We also use a
pre-trained code tokenizer and model instead of a pre-trained NLP tokenizer
and model for better performance on APR tasks. Tokenizers that are trained on
code usually generate fewer tokens for source code. Also, tokenizers that are
only trained on natural text often fail to handle code tokens well. This is
why CIRCLE needs the re-repairing step to fix unknown tokens of the generated
patches.

Several studies have explored the potential of large language models like
OpenAI’s Codex and ChatGPT for APR. Prenner et al. (2022) evaluate the
performance of Codex on the QuixBugs benchmark. Codex is a GPT-3-like model
that can generate code snippets from natural language descriptions or partial
code inputs. The authors experiment with different prompts to trigger Codex’s
bug-fixing ability, such as providing hints, docstrings, or input-output
examples. They find that Codex is competitive with state-of-the-art neural
repair techniques, especially for Python, and that the choice of prompt has a
significant impact on the repair success. Similarly, Sobania et al. (2023)
assess the automatic bug fixing performance of ChatGPT. They compare ChatGPT
with Codex, CoCoNuT, and several standard APR approaches on the QuixBugs
benchmark. The study finds that ChatGPT has similar performance to Codex and
CoCoNuT, and outperforms the standard APR approaches in terms of the number of
fixed bugs. The authors also analyze the types of responses generated by
ChatGPT and show that providing hints to ChatGPT can improve its success rate.

Although most of the APR techniques focus on single-hunk bugs, the challenge
of addressing multi-hunk bugs has attracted attention as well (Li et al.,
2022; Saha et al., 2019; Wong et al., 2021; Ye and Monperrus, 2023). T5APR
also targets a limited subset of multi-hunk bugs with the potential for
further expansion in future work.

There are also multilingual repair works that fix compilation and syntax
issues (Joshi et al., 2023; Yasunaga and Liang, 2021). However, they target a
different problem from our work, where programs compile successfully, and we
fix dynamic errors.

##  6 Conclusion

In this paper, we proposed T5APR, a novel approach for automated program
repair (APR) that leverages the power of the CodeT5 text-to-text transformer
model. Our method addresses program repair challenges across various
programming languages, offering a unified solution for bug fixing. Our
approach has several noteworthy contributions. We demonstrated the ability of
T5APR to efficiently handle multiple programming languages, fixing bugs in
Java, Python, C, and JavaScript. The checkpoint ensemble strategy further
improves the reliability and performance of our approach, providing more
robust patches for different bugs.

We conducted an extensive experimental evaluation to highlight the
effectiveness, generalizability, and competitiveness of T5APR. T5APR correctly
fixed 1,985 bugs out of 5,257 bugs across six benchmarks, including 1,442 bugs
that other compared state-of-the-art repair techniques did not fix. Moreover,
the patch ranking comparison showed the promising performance of T5APR in
terms of generating high-ranking patches.

In addition to the contributions of this research, there are several
directions for future work that can further enrich the field of APR. We can
investigate the selection of context windows and the impact of using larger
context windows beyond the immediate buggy context to find the optimal balance
between information and computational efficiency. We can use a more advanced
training and checkpoint selection process to further enhance T5APR’s
performance. Additionally, we can extend our approach to handle more complex
scenarios, such as multi-hunk bugs with different changes in multiple
locations. We can also further expand the supported languages and evaluation
benchmarks, even languages that were not part of the pre-training of CodeT5
but have similar syntax and semantics, as multilingual learning especially
helps knowledge transfer in low-resource languages (Zügner et al., 2020). As
the field progresses, the development of explainable patch generation
techniques (Kang et al., 2023; Liang et al., 2019; Monperrus, 2019) and close
collaboration with software developers could foster the usability and
trustworthiness of automated repair solutions.

## References

  * Afzal et al. (2019) Afsoon Afzal, Manish Motwani, Kathryn T. Stolee, Yuriy Brun, and Claire Le Goues.  SOSRepair: Expressive Semantic Search for Real-World Program Repair.  _IEEE Transactions on Software Engineering_ , 47(10):2162–2181, October 2019.  ISSN 1939-3520.  doi:10.1109/TSE.2019.2944914.
  * Akiba et al. (2019) Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama.  Optuna: A Next-generation Hyperparameter Optimization Framework.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, KDD ’19, pages 2623–2631, New York, NY, USA, July 2019. Association for Computing Machinery.  ISBN 978-1-4503-6201-6.  doi:10.1145/3292500.3330701.  URL https://doi.org/10.1145/3292500.3330701.
  * Allamanis et al. (2021) Miltiadis Allamanis, Henry Jackson-Flux, and Marc Brockschmidt.  Self-Supervised Bug Detection and Repair.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 27865–27876. Curran Associates, Inc., 2021.  URL https://proceedings.neurips.cc/paper_files/paper/2021/hash/ea96efc03b9a050d895110db8c4af057-Abstract.html.
  * Berabi et al. (2021) Berkay Berabi, Jingxuan He, Veselin Raychev, and Martin Vechev.  TFix: Learning to Fix Coding Errors with a Text-to-Text Transformer.  In _Proceedings of the 38th International Conference on Machine Learning_ , pages 780–791. PMLR, July 2021.  URL https://proceedings.mlr.press/v139/berabi21a.html.
  * Chakraborty et al. (2022) Saikat Chakraborty, Yangruibo Ding, Miltiadis Allamanis, and Baishakhi Ray.  CODIT: Code Editing With Tree-Based Neural Models.  _IEEE Transactions on Software Engineering_ , 48(4):1385–1399, April 2022.  ISSN 1939-3520.  doi:10.1109/TSE.2020.3020502.
  * Chen et al. (2017) Hugh Chen, Scott Lundberg, and Su-In Lee.  Checkpoint Ensembles: Ensemble Methods from a Single Training Process, October 2017.  URL http://arxiv.org/abs/1710.03282.
  * Chen and Monperrus (2018) Zimin Chen and Martin Monperrus.  The CodRep Machine Learning on Source Code Competition, November 2018.  URL http://arxiv.org/abs/1807.03200.
  * Chen et al. (2019) Zimin Chen, Steve Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and Martin Monperrus.  SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair.  _IEEE Transactions on Software Engineering_ , 47(9):1943–1959, 2019.  ISSN 1939-3520.  doi:10.1109/TSE.2019.2940179.
  * Das et al. (2020) Kaushik Amar Das, Arup Baruah, Ferdous Ahmed Barbhuiya, and Kuntal Dey.  KAFK at SemEval-2020 Task 12: Checkpoint Ensemble of Transformers for Hate Speech Classification.  In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_ , pages 2023–2029, Barcelona (online), December 2020. International Committee for Computational Linguistics.  doi:10.18653/v1/2020.semeval-1.267.  URL https://aclanthology.org/2020.semeval-1.267.
  * Dietterich (2000) Thomas G. Dietterich.  Ensemble Methods in Machine Learning.  In _Multiple Classifier Systems_ , Lecture Notes in Computer Science, pages 1–15, Berlin, Heidelberg, 2000. Springer.  ISBN 978-3-540-45014-6.  doi:10.1007/3-540-45014-9_1.
  * Ding et al. (2020) Yangruibo Ding, Baishakhi Ray, Premkumar Devanbu, and Vincent J. Hellendoorn.  Patching as Translation: The Data and the Metaphor.  In _2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE)_ , pages 275–286, September 2020.
  * Durieux et al. (2019) Thomas Durieux, Fernanda Madeiral, Matias Martinez, and Rui Abreu.  Empirical review of Java program repair tools: A large-scale experiment on 2,141 bugs and 23,551 repair attempts.  In _Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , ESEC/FSE 2019, pages 302–313, New York, NY, USA, August 2019. Association for Computing Machinery.  ISBN 978-1-4503-5572-8.  doi:10.1145/3338906.3338911.  URL https://doi.org/10.1145/3338906.3338911.
  * Gao et al. (2022) Xiang Gao, Yannic Noller, and Abhik Roychoudhury.  Program Repair, November 2022.  URL http://arxiv.org/abs/2211.12787.
  * Gazzola et al. (2018) Luca Gazzola, Daniela Micucci, and Leonardo Mariani.  Automatic software repair: A survey.  In _Proceedings of the 40th International Conference on Software Engineering_ , ICSE ’18, page 1219, New York, NY, USA, May 2018. Association for Computing Machinery.  ISBN 978-1-4503-5638-1.  doi:10.1145/3180155.3182526.  URL https://doi.org/10.1145/3180155.3182526.
  * Gyimesi et al. (2019) Péter Gyimesi, Béla Vancsics, Andrea Stocco, Davood Mazinanian, Árpád Beszédes, Rudolf Ferenc, and Ali Mesbah.  BugsJS: A Benchmark of JavaScript Bugs.  In _2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST)_ , pages 90–101, April 2019.  doi:10.1109/ICST.2019.00019.
  * Hanam et al. (2016) Quinn Hanam, Fernando S. de M. Brito, and Ali Mesbah.  Discovering bug patterns in JavaScript.  In _Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering_ , FSE 2016, pages 144–156, New York, NY, USA, November 2016. Association for Computing Machinery.  ISBN 978-1-4503-4218-6.  doi:10.1145/2950290.2950308.  URL https://doi.org/10.1145/2950290.2950308.
  * Huang et al. (2023) Kai Huang, Zhengzi Xu, Su Yang, Hongyu Sun, Xuejun Li, Zheng Yan, and Yuqing Zhang.  A Survey on Automated Program Repair Techniques, May 2023.  URL http://arxiv.org/abs/2303.18184.
  * Husain et al. (2020) Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt.  CodeSearchNet Challenge: Evaluating the State of Semantic Code Search, June 2020.  URL http://arxiv.org/abs/1909.09436.
  * Jiang et al. (2018) Jiajun Jiang, Yingfei Xiong, Hongyu Zhang, Qing Gao, and Xiangqun Chen.  Shaping program repair space with existing patches and similar code.  In _Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis_ , ISSTA 2018, pages 298–309, New York, NY, USA, July 2018. Association for Computing Machinery.  ISBN 978-1-4503-5699-2.  doi:10.1145/3213846.3213871.  URL https://doi.org/10.1145/3213846.3213871.
  * Jiang et al. (2021) Nan Jiang, Thibaud Lutellier, and Lin Tan.  CURE: Code-Aware Neural Machine Translation for Automatic Program Repair.  In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_ , pages 1161–1173, May 2021.  doi:10.1109/ICSE43902.2021.00107.
  * Jiang et al. (2023) Nan Jiang, Thibaud Lutellier, Yiling Lou, Lin Tan, Dan Goldwasser, and Xiangyu Zhang.  KNOD: Domain Knowledge Distilled Tree Decoder for Automated Program Repair.  In _Proceedings of the 45th International Conference on Software Engineering (ICSE 2023)_. Association for Computing Machinery, April 2023.  doi:10.48550/arXiv.2302.01857.  URL http://arxiv.org/abs/2302.01857.
  * Joshi et al. (2023) Harshit Joshi, José Cambronero Sanchez, Sumit Gulwani, Vu Le, Gust Verbruggen, and Ivan Radiček.  Repair Is Nearly Generation: Multilingual Program Repair with LLMs.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 37(4):5131–5140, June 2023.  ISSN 2374-3468.  doi:10.1609/aaai.v37i4.25642.  URL https://ojs.aaai.org/index.php/AAAI/article/view/25642.
  * Just et al. (2014) René Just, Darioush Jalali, and Michael D. Ernst.  Defects4J: A database of existing faults to enable controlled testing studies for Java programs.  In _Proceedings of the 2014 International Symposium on Software Testing and Analysis_ , ISSTA 2014, pages 437–440, New York, NY, USA, July 2014. Association for Computing Machinery.  ISBN 978-1-4503-2645-2.  doi:10.1145/2610384.2628055.  URL https://doi.org/10.1145/2610384.2628055.
  * Kang and Yoo (2022) Sungmin Kang and Shin Yoo.  Language models can prioritize patches for practical program patching.  In _Proceedings of the Third International Workshop on Automated Program Repair_ , APR ’22, pages 8–15, New York, NY, USA, October 2022. Association for Computing Machinery.  ISBN 978-1-4503-9285-3.  doi:10.1145/3524459.3527343.  URL https://doi.org/10.1145/3524459.3527343.
  * Kang et al. (2023) Sungmin Kang, Bei Chen, Shin Yoo, and Jian-Guang Lou.  Explainable Automated Debugging via Large Language Model-driven Scientific Debugging, April 2023.  URL http://arxiv.org/abs/2304.02195.
  * Karampatsis et al. (2020) Rafael-Michael Karampatsis, Hlib Babii, Romain Robbes, Charles Sutton, and Andrea Janes.  Big code != big vocabulary: Open-vocabulary models for source code.  In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ , ICSE ’20, pages 1073–1085, New York, NY, USA, October 2020. Association for Computing Machinery.  ISBN 978-1-4503-7121-6.  doi:10.1145/3377811.3380342.  URL https://dl.acm.org/doi/10.1145/3377811.3380342.
  * Koyuncu et al. (2020) Anil Koyuncu, Kui Liu, Tegawendé F. Bissyandé, Dongsun Kim, Jacques Klein, Martin Monperrus, and Yves Le Traon.  FixMiner: Mining relevant fix patterns for automated program repair.  _Empirical Software Engineering_ , 25(3):1980–2024, May 2020.  ISSN 1573-7616.  doi:10.1007/s10664-019-09780-z.  URL https://doi.org/10.1007/s10664-019-09780-z.
  * Le Goues et al. (2012) Claire Le Goues, ThanhVu Nguyen, Stephanie Forrest, and Westley Weimer.  GenProg: A Generic Method for Automatic Software Repair.  _IEEE Transactions on Software Engineering_ , 38(1):54–72, January 2012.  ISSN 1939-3520.  doi:10.1109/TSE.2011.104.
  * Le Goues et al. (2015) Claire Le Goues, Neal Holtschulte, Edward K. Smith, Yuriy Brun, Premkumar Devanbu, Stephanie Forrest, and Westley Weimer.  The ManyBugs and IntroClass Benchmarks for Automated Repair of C Programs.  _IEEE Transactions on Software Engineering_ , 41(12):1236–1256, December 2015.  ISSN 1939-3520.  doi:10.1109/TSE.2015.2454513.
  * Le Goues et al. (2019) Claire Le Goues, Michael Pradel, and Abhik Roychoudhury.  Automated program repair.  _Communications of the ACM_ , 62(12):56–65, November 2019.  ISSN 0001-0782.  doi:10.1145/3318162.  URL https://doi.org/10.1145/3318162.
  * Lhoest et al. (2021) Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf.  Datasets: A Community Library for Natural Language Processing.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 175–184, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi:10.18653/v1/2021.emnlp-demo.21.  URL https://aclanthology.org/2021.emnlp-demo.21.
  * Li et al. (2020) Yi Li, Shaohua Wang, and Tien N. Nguyen.  DLFix: Context-based code transformation learning for automated program repair.  In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ , ICSE ’20, pages 602–614, New York, NY, USA, October 2020. Association for Computing Machinery.  ISBN 978-1-4503-7121-6.  doi:10.1145/3377811.3380345.  URL https://doi.org/10.1145/3377811.3380345.
  * Li et al. (2022) Yi Li, Shaohua Wang, and Tien N. Nguyen.  DEAR: A novel deep learning-based approach for automated program repair.  In _Proceedings of the 44th International Conference on Software Engineering_ , ICSE ’22, pages 511–523, New York, NY, USA, July 2022. Association for Computing Machinery.  ISBN 978-1-4503-9221-1.  doi:10.1145/3510003.3510177.  URL https://dl.acm.org/doi/10.1145/3510003.3510177.
  * Liang et al. (2019) Jingjing Liang, Yaozong Hou, Shurui Zhou, Junjie Chen, Yingfei Xiong, and Gang Huang.  How to Explain a Patch: An Empirical Study of Patch Explanations in Open Source Projects.  In _2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE)_ , pages 58–69, October 2019.  doi:10.1109/ISSRE.2019.00016.
  * Lin et al. (2017) Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama.  QuixBugs: A multi-lingual program repair benchmark set based on the quixey challenge.  In _Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity_ , SPLASH Companion 2017, pages 55–56, New York, NY, USA, October 2017. Association for Computing Machinery.  ISBN 978-1-4503-5514-8.  doi:10.1145/3135932.3135941.  URL https://dl.acm.org/doi/10.1145/3135932.3135941.
  * Liu et al. (2019a) Kui Liu, Anil Koyuncu, Tegawendé F. Bissyandé, Dongsun Kim, Jacques Klein, and Yves Le Traon.  You Cannot Fix What You Cannot Find! An Investigation of Fault Localization Bias in Benchmarking Automated Program Repair Systems.  In _2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST)_ , pages 102–113, April 2019a.  doi:10.1109/ICST.2019.00020.
  * Liu et al. (2019b) Kui Liu, Anil Koyuncu, Dongsun Kim, and Tegawendé F. Bissyandé.  TBar: Revisiting template-based automated program repair.  In _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis_ , ISSTA 2019, pages 31–42, New York, NY, USA, July 2019b. Association for Computing Machinery.  ISBN 978-1-4503-6224-5.  doi:10.1145/3293882.3330577.  URL https://doi.org/10.1145/3293882.3330577.
  * Liu et al. (2020) Kui Liu, Shangwen Wang, Anil Koyuncu, Kisub Kim, Tegawendé F. Bissyandé, Dongsun Kim, Peng Wu, Jacques Klein, Xiaoguang Mao, and Yves Le Traon.  On the efficiency of test suite based program repair: A Systematic Assessment of 16 Automated Repair Systems for Java Programs.  In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ , ICSE ’20, pages 615–627, New York, NY, USA, October 2020. Association for Computing Machinery.  ISBN 978-1-4503-7121-6.  doi:10.1145/3377811.3380338.  URL https://doi.org/10.1145/3377811.3380338.
  * Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter.  Decoupled Weight Decay Regularization.  In _International Conference on Learning Representations_ , September 2018.  URL https://openreview.net/forum?id=Bkg6RiCqY7.
  * Lutellier et al. (2020) Thibaud Lutellier, Hung Viet Pham, Lawrence Pang, Yitong Li, Moshi Wei, and Lin Tan.  CoCoNuT: Combining context-aware neural translation models using ensemble for program repair.  In _Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis_ , ISSTA 2020, pages 101–114, New York, NY, USA, July 2020. Association for Computing Machinery.  ISBN 978-1-4503-8008-9.  doi:10.1145/3395363.3397369.  URL https://doi.org/10.1145/3395363.3397369.
  * Madeiral and Durieux (2021) Fernanda Madeiral and Thomas Durieux.  A large-scale study on human-cloned changes for automated program repair.  In _2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)_ , pages 510–514, May 2021.  doi:10.1109/MSR52588.2021.00064.
  * Madeiral et al. (2019) Fernanda Madeiral, Simon Urli, Marcelo Maia, and Martin Monperrus.  BEARS: An Extensible Java Bug Benchmark for Automatic Program Repair Studies.  In _2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER)_ , pages 468–478, February 2019.  doi:10.1109/SANER.2019.8667991.
  * Mechtaev et al. (2016) Sergey Mechtaev, Jooyong Yi, and Abhik Roychoudhury.  Angelix: Scalable multiline program patch synthesis via symbolic analysis.  In _Proceedings of the 38th International Conference on Software Engineering_ , ICSE ’16, pages 691–701, New York, NY, USA, May 2016. Association for Computing Machinery.  ISBN 978-1-4503-3900-1.  doi:10.1145/2884781.2884807.  URL https://doi.org/10.1145/2884781.2884807.
  * Mockus and Votta (2000) Mockus and Votta.  Identifying reasons for software changes using historic databases.  In _Proceedings 2000 International Conference on Software Maintenance_ , pages 120–130, October 2000.  doi:10.1109/ICSM.2000.883028.
  * Monperrus (2018a) Martin Monperrus.  Automatic Software Repair: A Bibliography.  _ACM Computing Surveys_ , 51(1):17:1–17:24, January 2018a.  ISSN 0360-0300.  doi:10.1145/3105906.  URL https://doi.org/10.1145/3105906.
  * Monperrus (2018b) Martin Monperrus.  The Living Review on Automated Program Repair.  Technical Report hal-01956501, HAL Archives Ouvertes, 2018b.  URL https://hal.archives-ouvertes.fr/hal-01956501.
  * Monperrus (2019) Martin Monperrus.  Explainable Software Bot Contributions: Case Study of Automated Bug Fixes.  In _2019 IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE)_ , pages 12–15, May 2019.  doi:10.1109/BotSE.2019.00010.
  * Nguyen et al. (2013) Hoang Duong Thien Nguyen, Dawei Qi, Abhik Roychoudhury, and Satish Chandra.  SemFix: Program repair via semantic analysis.  In _2013 35th International Conference on Software Engineering (ICSE)_ , pages 772–781, May 2013.  doi:10.1109/ICSE.2013.6606623.
  * Noller et al. (2022) Yannic Noller, Ridwan Shariffdeen, Xiang Gao, and Abhik Roychoudhury.  Trust enhancement issues in program repair.  In _Proceedings of the 44th International Conference on Software Engineering_ , ICSE ’22, pages 2228–2240, New York, NY, USA, July 2022. Association for Computing Machinery.  ISBN 978-1-4503-9221-1.  doi:10.1145/3510003.3510040.  URL https://dl.acm.org/doi/10.1145/3510003.3510040.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: A Method for Automatic Evaluation of Machine Translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.  doi:10.3115/1073083.1073135.  URL https://aclanthology.org/P02-1040.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper_files/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html.
  * Post (2018) Matt Post.  A Call for Clarity in Reporting BLEU Scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium, October 2018. Association for Computational Linguistics.  doi:10.18653/v1/W18-6319.  URL https://aclanthology.org/W18-6319.
  * Prenner et al. (2022) Julian Aron Prenner, Hlib Babii, and Romain Robbes.  Can OpenAI’s codex fix bugs? an evaluation on QuixBugs.  In _Proceedings of the Third International Workshop on Automated Program Repair_ , APR ’22, pages 69–75, New York, NY, USA, October 2022. Association for Computing Machinery.  ISBN 978-1-4503-9285-3.  doi:10.1145/3524459.3527351.  URL https://doi.org/10.1145/3524459.3527351.
  * Qi et al. (2015) Zichao Qi, Fan Long, Sara Achour, and Martin Rinard.  An analysis of patch plausibility and correctness for generate-and-validate patch generation systems.  In _Proceedings of the 2015 International Symposium on Software Testing and Analysis_ , ISSTA 2015, pages 24–36, New York, NY, USA, July 2015. Association for Computing Machinery.  ISBN 978-1-4503-3620-8.  doi:10.1145/2771783.2771791.  URL https://doi.org/10.1145/2771783.2771791.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):140:5485–140:5551, January 2020.  ISSN 1532-4435.
  * Ray et al. (2016) Baishakhi Ray, Vincent Hellendoorn, Saheel Godhane, Zhaopeng Tu, Alberto Bacchelli, and Premkumar Devanbu.  On the "Naturalness" of Buggy Code.  In _2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE)_ , pages 428–439, May 2016.  doi:10.1145/2884781.2884848.
  * Saha et al. (2018) Ripon Saha, Yingjun Lyu, Wing Lam, Hiroaki Yoshida, and Mukul Prasad.  Bugs.jar: A Large-Scale, Diverse Dataset of Real-World Java Bugs.  In _2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR)_ , pages 10–13, May 2018.
  * Saha et al. (2019) Seemanta Saha, Ripon k. Saha, and Mukul r. Prasad.  Harnessing Evolution for Multi-Hunk Program Repair.  In _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_ , pages 13–24, May 2019.  doi:10.1109/ICSE.2019.00020.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch.  Neural Machine Translation of Rare Words with Subword Units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_ , pages 1715–1725. Association for Computational Linguistics (ACL), August 2016.  doi:10.18653/v1/P16-1162.  URL https://www.research.ed.ac.uk/en/publications/neural-machine-translation-of-rare-words-with-subword-units.
  * Smith et al. (2015) Edward K. Smith, Earl T. Barr, Claire Le Goues, and Yuriy Brun.  Is the cure worse than the disease? overfitting in automated program repair.  In _Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering_ , ESEC/FSE 2015, pages 532–543, New York, NY, USA, August 2015. Association for Computing Machinery.  ISBN 978-1-4503-3675-8.  doi:10.1145/2786805.2786825.  URL https://doi.org/10.1145/2786805.2786825.
  * Sobania et al. (2023) Dominik Sobania, Martin Briesch, Carol Hanna, and Justyna Petke.  An Analysis of the Automatic Bug Fixing Performance of ChatGPT.  In _2023 IEEE/ACM International Workshop on Automated Program Repair (APR)_ , pages 23–30, May 2023.  doi:10.1109/APR59189.2023.00012.
  * Sobreira et al. (2018) Victor Sobreira, Thomas Durieux, Fernanda Madeiral, Martin Monperrus, and Marcelo de Almeida Maia.  Dissection of a bug dataset: Anatomy of 395 patches from Defects4J.  In _2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER)_ , pages 130–140, March 2018.  doi:10.1109/SANER.2018.8330203.
  * Tan et al. (2017) Shin Hwei Tan, Jooyong Yi, Yulis, Sergey Mechtaev, and Abhik Roychoudhury.  Codeflaws: A programming competition benchmark for evaluating automated program repair tools.  In _2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C)_ , pages 180–182, May 2017.  doi:10.1109/ICSE-C.2017.76.
  * Tufano et al. (2019a) Michele Tufano, Jevgenija Pantiuchina, Cody Watson, Gabriele Bavota, and Denys Poshyvanyk.  On learning meaningful code changes via neural machine translation.  In _Proceedings of the 41st International Conference on Software Engineering_ , ICSE ’19, pages 25–36, Montreal, Quebec, Canada, May 2019a. IEEE Press.  doi:10.1109/ICSE.2019.00021.  URL https://doi.org/10.1109/ICSE.2019.00021.
  * Tufano et al. (2019b) Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk.  An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation.  _ACM Transactions on Software Engineering and Methodology_ , 28(4):19:1–19:29, September 2019b.  ISSN 1049-331X.  doi:10.1145/3340544.  URL https://dl.acm.org/doi/10.1145/3340544.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is All you Need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
  * Wang et al. (2021) Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi.  CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 8696–8708, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi:10.18653/v1/2021.emnlp-main.685.  URL https://aclanthology.org/2021.emnlp-main.685.
  * Widyasari et al. (2020) Ratnadira Widyasari, Sheng Qin Sim, Camellia Lok, Haodi Qi, Jack Phan, Qijin Tay, Constance Tan, Fiona Wee, Jodie Ethelda Tan, Yuheng Yieh, Brian Goh, Ferdian Thung, Hong Jin Kang, Thong Hoang, David Lo, and Eng Lieh Ouh.  BugsInPy: A database of existing bugs in Python programs to enable controlled testing and debugging studies.  In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , ESEC/FSE 2020, pages 1556–1560, New York, NY, USA, November 2020. Association for Computing Machinery.  ISBN 978-1-4503-7043-1.  doi:10.1145/3368089.3417943.  URL https://doi.org/10.1145/3368089.3417943.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush.  Transformers: State-of-the-Art Natural Language Processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online, October 2020. Association for Computational Linguistics.  doi:10.18653/v1/2020.emnlp-demos.6.  URL https://aclanthology.org/2020.emnlp-demos.6.
  * Wong et al. (2021) Chu-Pan Wong, Priscila Santiesteban, Christian Kästner, and Claire Le Goues.  VarFix: Balancing edit expressiveness and search effectiveness in automated program repair.  In _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , ESEC/FSE 2021, pages 354–366, New York, NY, USA, August 2021. Association for Computing Machinery.  ISBN 978-1-4503-8562-6.  doi:10.1145/3468264.3468600.  URL https://dl.acm.org/doi/10.1145/3468264.3468600.
  * Yang et al. (2021) Deheng Yang, Kui Liu, Dongsun Kim, Anil Koyuncu, Kisub Kim, Haoye Tian, Yan Lei, Xiaoguang Mao, Jacques Klein, and Tegawendé F. Bissyandé.  Where were the repair ingredients for Defects4j bugs?  _Empirical Software Engineering_ , 26(6):122, September 2021.  ISSN 1573-7616.  doi:10.1007/s10664-021-10003-7.  URL https://doi.org/10.1007/s10664-021-10003-7.
  * Yasunaga and Liang (2020) Michihiro Yasunaga and Percy Liang.  Graph-based, Self-Supervised Program Repair from Diagnostic Feedback.  In _Proceedings of the 37th International Conference on Machine Learning_ , pages 10799–10808. PMLR, November 2020.  URL https://proceedings.mlr.press/v119/yasunaga20a.html.
  * Yasunaga and Liang (2021) Michihiro Yasunaga and Percy Liang.  Break-It-Fix-It: Unsupervised Learning for Program Repair.  In _Proceedings of the 38th International Conference on Machine Learning_ , pages 11941–11952. PMLR, July 2021.  URL https://proceedings.mlr.press/v139/yasunaga21a.html.
  * Ye and Monperrus (2023) He Ye and Martin Monperrus.  ITER: Iterative Neural Repair for Multi-Location Patches, April 2023.  URL http://arxiv.org/abs/2304.12015.
  * Ye et al. (2021a) He Ye, Matias Martinez, Thomas Durieux, and Martin Monperrus.  A comprehensive study of automatic program repair on the QuixBugs benchmark.  _Journal of Systems and Software_ , 171:110825, January 2021a.  ISSN 0164-1212.  doi:10.1016/j.jss.2020.110825.  URL https://www.sciencedirect.com/science/article/pii/S0164121220302193.
  * Ye et al. (2021b) He Ye, Matias Martinez, and Martin Monperrus.  Automated patch assessment for program repair at scale.  _Empirical Software Engineering_ , 26(2):20, February 2021b.  ISSN 1573-7616.  doi:10.1007/s10664-020-09920-w.  URL https://doi.org/10.1007/s10664-020-09920-w.
  * Ye et al. (2022) He Ye, Matias Martinez, and Martin Monperrus.  Neural program repair with execution-based backpropagation.  In _Proceedings of the 44th International Conference on Software Engineering_ , ICSE ’22, pages 1506–1518, New York, NY, USA, July 2022. Association for Computing Machinery.  ISBN 978-1-4503-9221-1.  doi:10.1145/3510003.3510222.  URL https://doi.org/10.1145/3510003.3510222.
  * Ye et al. (2023) He Ye, Matias Martinez, Xiapu Luo, Tao Zhang, and Martin Monperrus.  SelfAPR: Self-supervised Program Repair with Test Execution Diagnostics.  In _Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering_ , ASE ’22, pages 1–13, New York, NY, USA, January 2023. Association for Computing Machinery.  ISBN 978-1-4503-9475-8.  doi:10.1145/3551349.3556926.  URL https://dl.acm.org/doi/10.1145/3551349.3556926.
  * Yuan et al. (2022) Wei Yuan, Quanjun Zhang, Tieke He, Chunrong Fang, Nguyen Quoc Viet Hung, Xiaodong Hao, and Hongzhi Yin.  CIRCLE: Continual repair across programming languages.  In _Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis_ , ISSTA 2022, pages 678–690, New York, NY, USA, July 2022. Association for Computing Machinery.  ISBN 978-1-4503-9379-9.  doi:10.1145/3533767.3534219.  URL https://doi.org/10.1145/3533767.3534219.
  * Zhang et al. (2023) Quanjun Zhang, Chunrong Fang, Yuxiang Ma, Weisong Sun, and Zhenyu Chen.  A Survey of Learning-based Automated Program Repair, January 2023.  URL http://arxiv.org/abs/2301.03270.
  * Zhong et al. (2022) Wenkang Zhong, Chuanyi Li, Jidong Ge, and Bin Luo.  Neural Program Repair : Systems, Challenges and Solutions.  In _Proceedings of the 13th Asia-Pacific Symposium on Internetware_ , Internetware ’22, pages 96–106, New York, NY, USA, September 2022. Association for Computing Machinery.  ISBN 978-1-4503-9780-3.  doi:10.1145/3545258.3545268.  URL https://doi.org/10.1145/3545258.3545268.
  * Zhong et al. (2023) Wenkang Zhong, Hongliang Ge, Hongfei Ai, Chuanyi Li, Kui Liu, Jidong Ge, and Bin Luo.  StandUp4NPR: Standardizing SetUp for Empirically Comparing Neural Program Repair Systems.  In _Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering_ , ASE ’22, pages 1–13, New York, NY, USA, January 2023. Association for Computing Machinery.  ISBN 978-1-4503-9475-8.  doi:10.1145/3551349.3556943.  URL https://doi.org/10.1145/3551349.3556943.
  * Zhu et al. (2021) Qihao Zhu, Zeyu Sun, Yuan-an Xiao, Wenjie Zhang, Kang Yuan, Yingfei Xiong, and Lu Zhang.  A syntax-guided edit decoder for neural program repair.  In _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , ESEC/FSE 2021, pages 341–353, New York, NY, USA, August 2021. Association for Computing Machinery.  ISBN 978-1-4503-8562-6.  doi:10.1145/3468264.3468544.  URL https://doi.org/10.1145/3468264.3468544.
  * Zügner et al. (2020) Daniel Zügner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, and Stephan Günnemann.  Language-Agnostic Representation Learning of Source Code from Structure and Context.  In _International Conference on Learning Representations_ , October 2020.  URL https://openreview.net/forum?id=Xh5eMZVONGF.