numbersnone#1none:
left:
right:

# Can we learn from developer mistakes?
Learning to localize and repair real bugs from real bug fixes

Cedric Richter  University of OldenburgAmmerländer Heerstraße
114-118OldenburgGermany26129<EMAIL_ADDRESS>and  Heike
Wehrheim  University of OldenburgAmmerländer Heerstraße
114-118OldenburgGermany26129<EMAIL_ADDRESS>
(2018)

###### Abstract.

Real bug fixes found in open source repositories seem to be the perfect source
for learning to localize and repair real bugs. However, the absence of large
scale bug fix collections has made it difficult to effectively exploit real
bug fixes in the training of larger neural models in the past. In contrast,
artificial bugs – produced by mutating existing source code – can be easily
obtained at a sufficient scale and are therefore often preferred in the
training of existing approaches. Still, localization and repair models that
are trained on artificial bugs usually underperform when faced with real bugs.
This raises the question whether bug localization and repair models trained on
real bug fixes are more effective in localizing and repairing real bugs.

We address this question by introducing RealiT, a pre-train-and-fine-tune
approach for effectively learning to localize and repair real bugs from real
bug fixes. RealiT is first pre-trained on a large number of artificial bugs
produced by traditional mutation operators and then fine-tuned on a smaller
set of real bug fixes. Fine-tuning does not require any modifications of the
learning algorithm and hence can be easily adopted in various training
scenarios for bug localization or repair (even when real training data is
scarce). In addition, we found that training on real bug fixes with RealiT is
empirically powerful by nearly doubling the localization performance of an
existing model on real bugs while maintaining or even improving the repair
performance.

program repair, bug detection, bug fixes, learn to debug

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: ; June 2022; Woodstock, NY

##  1\. Introduction

Figure 1. Training on real bug fixes improves localization and repair of real
bugs.

Finding and fixing software bugs are one of the most common challenges in
software engineering (Alaboudi and LaToza, 2021). Developers are often faced
with these tasks while spending a considerable amount of time in fixing
software bugs. Still, some bugs find their way in open software repositories
which then has to be fixed by a bug fixing code change. This raises the
question whether we can relieve the developer from the debugging process by
learning from common developer mistakes and their fixes found in open source
projects.

Previous work (Pradel and Sen, 2018; Hellendoorn et al., 2019; Vasic et al.,
2019; Allamanis et al., 2017; Richter and Wehrheim, 2022a; Allamanis et al.,
2021) addressed this question by designing automatic learning-based methods
for bug localization and repair. However, to obtain the necessary amount of
data needed for training, they often employed code mutants instead of real bug
fixes. Mutants are generated by automatically injecting small code changes
into existing code. As this process is automated, mutants can be easily
obtained at large masses which is necessary for training effective learning-
based models. However, a mutant may not represent a real bug which could
ultimately bottleneck the performance of learning based bug localization
repair.

In contrast in this work, we aim to explore the effect of real bug fixes
obtained from open source repositories on the training of learning-based bug
localization and repair methods. For this, we employ a novel dataset of 33k
real-world bug fixes obtained from public Python projects. Since this dataset
is still comparably small to the datasets typically used for training
localization and repair models (Allamanis et al., 2021; Hellendoorn et al.,
2019) (which oftentimes contain millionth of artificial bugs), we propose
RealiT (pronounced “reality”), a novel training scheme for learning to Repair
and localize with Transformers. RealiT is designed to combine the strengths of
both training on mutants and real bug fixes by first pre-training on a high
number of mutants and then fine-tuning on a smaller set of real bug fixes.
This design allows us to not only evaluate the impact of real bug fixes and
mutants together on the training process, but also individually (by skipping
either pre-training or fine-tuning phase).

To evaluate the impact of RealiT’s training on the localization and repair of
real bugs, we implement RealiT for fixing a variety of single token bugs in
Python. Our implementation considers four common types of single token bugs –
that can be fixed by changing a single program token. We evaluate RealiT
together with several baselines on over 2000 real world bugs collected in the
PyPIBugs benchmark (Allamanis et al., 2021).

By integrating real bug fixes in the training process with RealiT, we observe
significant performance gains over a training that solely focuses on mutants
(as in previous works). In fact, training with real bug fixes allows to nearly
double the number of successfully localized real bugs (x-axis in Figure 1)
while also maintaining or improving the repair performance.

Our main contributions can be summarized as follows:

  * •

For investigating the effect of real bug fixes on the training of neural bug
localization and repair models, we propose a simple pre-train-and-fine-tune
approach. We find that training both on mutants and real bugs with our method
significantly improves the performance over models solely trained on either
mutants or real bugs when evaluated on real single token bugs in Python.

  * •

We show that data quality and quantity has a significant impact on neural bug
localization and repair models. By pre-training on a large number of mutants
(up to 20x larger than in previous work), RealiT already significantly
improves localization and repair performance both on mutants and real bugs.
Combined with fine-tuning on real bug fixes, RealiT is the first model to
repair a significant portion of a real bugs benchmark.

  * •

For adopting RealiT into future projects, we show that even training on
smaller subsets of real bug fixes can yield performance improvement for
localization and repair of real bugs. However, more bug fixes are also more
beneficial.

We plan to release all trained models, pre-training and fine-tuning
code111https://github.com/cedricrupb/nbfbaselines.

Table 1. Examples of single token bug types taken from PyPIBugs (Allamanis et al., 2021) Example | Description
---|---
⬇ 1# VarMisuse: applied instead of patch 2applied = self.db.applied_patches() 3for patch in applied: 4 if patch in patches: 5 patches.remove(applied) |  All applied patches should be removed from the patches list. However, the developer mistakenly tries to remove applied instead of a single patch. Fix: replace applied in Line 5 by patch defined in Line 3.
⬇ 1# BinOp: != instead of == 2def updateRefractionParameters(self): 3 ... 4 if self.ui.checkRefracNone.isChecked(): 5 return False 6 if self.checkRefracNoTrack.isChecked(): 7 if self.app.mount.status != 0: 8 return False 9 ... |  The function updateRefractionParameters performs an update and returns true if the update was successful. Prior to the update the function checks some preconditions and the function should abort if the mount is not ready. Therefore, we can conventionally expect that we abort if the status is zero. However, we check whether the status is not zero. Fix: replace != in Line 7 by ==.
⬇ 1# Negation: namespace instead of not namespace 2if namespace: 3 self.namespacesFilter = [ "prymatex", "user" ] 4else: 5 self.namespacesFilter = namespace.split() |  A default namespacesFilter should be used if no namespace is given. However, the condition checks the inverse. Fix: replace namespace in Line 2 by not namespace.

##  2\. Background

In this section, we introduce the necessary background for our approach. To
begin with, we start by describing the single token localization and repair
task tackled by RealiT and how previous techniques addressed this task by
predicting token replacements and learning from mutants.

###  2.1. Single token bug localization and repair

In this work, we focus on the localization and repair of single token bugs.
Single token bugs are bugs that can be repaired by replacing only a single
program token (e.g. a variable or binary operator). For this reason, they are
often easy to repair – as only a single token has to be changed – but hard to
identify. Examples for single token bugs are given in Table 1. Interestingly,
single token bug localization and repair has previously only been addressed
through training with mutants (Pradel and Sen, 2018; Hellendoorn et al., 2019;
Vasic et al., 2019; Allamanis et al., 2017; Richter and Wehrheim, 2022a;
Allamanis et al., 2021). Nevertheless, real bug fixes for single token bugs –
which can be employed for training or testing – are available in bug fix
collections such as ManySStuBs4J (Karampatsis and Sutton, 2020) or TSSB-3M
(Richter and Wehrheim, 2022b).

Task description. Throughout this work, we view source code as a sequence of
tokens $\mathcal{T}=t_{0},t_{1},t_{2},\dots,t_{n}$. A single token bug can
then be fixed by replacing a single token $t_{l}$ with another token $r$ in
the same scope ($r=t_{l^{\prime}}$) or coming from an external vocabulary
($r\in V$). To effectively localize and repair a single token bug, the
following three tasks have to be performed: (1) the program $\mathcal{T}$ has
to be classified to contain a bug, (2) the bug location $t_{l}$ has to be
localized and then (3) the correct repair $r$ has to be identified. In
practice, these three tasks are often modeled as token replacement operations.
Let $\mathcal{T}$ be a program containing a single token bug and
$\mathcal{T}^{\prime}$ be the corrected bug-free version, then the
localization and repair model is trained to perform the following operations:

(1) $\mathcal{T}\xrightarrow{\text{replace}(t_{l},r)}\mathcal{T}^{\prime}$ (2)
$\mathcal{T}^{\prime}\xrightarrow{\text{noop}()}\mathcal{T}^{\prime}$

Here, we fix the buggy program $\mathcal{T}$ by replacing $t_{l}$ with $r$ and
therefore translating it into $\mathcal{T}^{\prime}$. Since
$\mathcal{T}^{\prime}$ is bug-free, a change is not required (noop).

In practice, we train models to estimate the likelihood of each possible token
replacement and select the most likely replacement to fix $\mathcal{T}$.

###  2.2. Mutation

Motivated by the general absence of real bug fixes at a sufficient scale,
previous learning based localization and repair approaches (Hellendoorn et
al., 2019; Vasic et al., 2019; Richter and Wehrheim, 2022a; Allamanis et al.,
2021) mainly focused on training on mutants. Mutants are artificially
generated (pseudo-)bugs that are introduced into a correct program via a
mutation operator. For single token bugs, the mutation operator can be seen as
a token replacement operator which can be inverted by a localization and
repair model:

(3)
$\mathcal{T}\xrightarrow{\text{mutate}(t_{l},r)}\mathcal{T}^{\prime}\xrightarrow{\text{replace}(t_{l},r^{-1})}\mathcal{T}$

For a dataset of bug-free programs (e.g. mined from open source projects), the
mutation operator first introduces a token mutation by replacing a random
token with a random other token. The token types are often specified (e.g.
binary operators) such that the programs remains interpretable after the
transformation. Afterwards, the localization and repair model is trained to
invert the mutation process to obtain the original program.

While traditionally mutation operator are designed as a random process,
previous work also tried to design more realistic mutation operators by
learning from real bug fixes (Patra and Pradel, 2021), by training an
adversary to the repair model (Allamanis et al., 2021) or by finding
replacements that naturally fit the context (Richter and Wehrheim, 2022a).

###  2.3. Real bug fixes

Real bug fixes are often obtained by scraping the commit history of public
open source projects. During this process, commits are often classified as bug
fixing based on certain keywords in the commit message (Karampatsis and
Sutton, 2020). Even though this process cannot guarantee that every collected
commit is a real bug fix, it has been empirically shown (Karampatsis and
Sutton, 2020) that the process is highly precise (e.g. over 90% of all
collected code changes were real bug fixes). In this work, we are interested
in special types of single token bug fixes. Here, a bug is fixed by replacing
only a single token:

(4) $\mathcal{T}_{i}\xrightarrow{\text{replace}(t_{l},r)}\mathcal{T}_{i+1}$

Note that a (bug fixing) commit only represents a snapshot of the project at
time $i$. Therefore, while it is highly likely $\mathcal{T}_{i}$ contains a
single token bug which can be fixed by $\text{replace}(t_{l},r)$, we cannot
guarantee that the bug fix is complete and $\mathcal{T}_{i+1}$ is bug-free.

##  3\. Methodology

In this section, we introduce RealiT as an effective training technique for
bug localization and repair with Transformers. We start by giving a general
overview of the training process for transferring and improving the
performance of a localization and repair model trained solely on mutants.
Afterwards, we discuss the Transformer-based architecture used during training
and the inference strategy we apply for localizing and repairing single token
bugs in more detail.

Github CodeReal Bug FixesDatasetsCorrectsupersampleMutantspre-train(1) Train
on mutantsmutateCorrectsupersampleBugssupersamplefine-tune(2) Fine-tune on
real bugsReal Test
Setpatches1.2remove3(4applied5)6Localizationpatches1.2remove3(4applied5)6Repairpatches1.2remove3(4applied5)6(3)
Evaluate Figure 2. Overview over the RealiT training and evaluation process

###  3.1. RealiT: Training on mutants and real bugs

To facilitate both mutants and real bug fixes, we design RealiT as a pre-
train-and-fine-tune approach. Therefore, we perform the training in two phases
(as shown in Figure 2). In the first pre-training phase, we train RealiT on
artificially generated mutants introduced into source code obtained by mining
public open source repositories. Afterwards, in the second fine-tuning phase,
we employ the pre-trained version of RealiT to further train it on real bug
fixes.

Pre-training with code mutants. During pre-training, we train our model
similar to the way current localization and repair models are trained
(Hellendoorn et al., 2019). Here, the training objective is not to identify
real bugs but rather to identify and transform mutated code snippets back into
the real code snippets. For this task, we naturally start with a general
corpus of Github code snippets (e.g. function implementations). This corpus
can often easily be obtained by mining the recent version of popular open
source projects. Since bugs are scarce in open source projects, we can safely
assume that most of the code snippets are likely correct. For training, we
mutate each code snippet in our corpus at max $k$ times which produces a
dataset of at max $k$ unique mutants per code snippet222The number of code
rewrites applicable to a code snippet and hence the number of unique mutants
per code snippet is limited by design and might be lower than $k$. We never
introduce mutation duplicates.. During our experiments, we decided to employ
an unusually large number of mutants per code snippet ($k$ = 100) since we
observed that this improves the performance after fine-tuning. We employ the
original code corpus as training examples of unmutated correct code. Based on
the two datasets, RealiT is then trained to (1) distinguish mutants from real
code, (2) identify the mutated location (if any) and (3) find the original
replaced token. Since the dataset of mutants is up to $k$ times larger than
the set of correct code snippet (by construction), we additionally
supersample333During training, mutants and correct programs are sampled at the
same frequency. As the set of mutants is up to $k$ times larger, we uniformly
(super-)sample correct programs multiple times to match the number of mutants
seen during training. each correct code snippet such that RealiT is trained on
correct and mutated code snippets at the same frequency. This avoids that the
model is biased towards detecting mutants.

Learning to fix real bugs with bug fixes. In the second phase, we aim to
further optimize the performance of RealiT for localization and repair of real
bugs by fine-tuning on real bug fixes. For the fine-tuning process, we adopt a
pre-trained version of RealiT obtained from the previous phase. Then, we
continue the training now on realistic buggy and bug-free code. As examples
for realistic buggy code, we employ the code related to real bug fix before
the fix is applied. Bug-free code is again obtained by using the original
Github corpus. During training, RealiT is now fine-tuned to (1) distinguish
real buggy code from bug-free code, (2) identify the bug location (if any) and
(3) imitate the original bug fix. Since now, the code corpus is usually much
larger than the set of bug fixes, we supersample the buggy programs to match
the correct programs in their frequency.

We believe that pre-training and fine-tuning can have an orthogonal effect on
the localization and repair model (which we aim to explore during our
evaluation). Due to the mutation process, the pre-training phase is more
tailored towards identifying correct programs and deviations from them. In
contrast, the fine-tuning phase aims to teach the model the difference between
real correct programs and real buggy programs (and how they can be fixed).

###  3.2. Model architecture

In the section, we discuss the neural model employed by RealiT to learn
localization and repair of single token bugs. Since our main focus is to study
the effect of mutants and real bug fixes on the training process, we employ a
fairly standard Transformer-based model (Hellendoorn et al., 2019) for
learning bug localization and repair.

Probabilistic model. For the task of single token localization and repair,
programs are represented as a sequence of tokens
$\mathcal{T}=t_{0},t_{1},t_{2},\dots,t_{n}$ where each token $t_{l}$
represents a potential bug location. Single token bugs are fixed only by
replacing single tokens $t_{l}$ with another token $r$
($\text{replace}(t_{l},r)$). In the following, we model localization and
repair as a joint probability distribution over all potential bug locations
and repairs $\\{\langle l,r\rangle\mid
t_{l}\in\mathcal{T}\cup\\{\mathtt{NOOP}\\}\text{ and }r\in\mathcal{T}\cup
V\\}$:

(5) $p(\langle
l,r\rangle\mid\mathcal{T})=p_{\text{loc}}(l\mid\mathcal{T})\cdot
p_{\text{repair}}(r\mid l,\mathcal{T})$

Here, localization and repair is factorized into first localizing a bug
location ($p_{\text{loc}}(l\mid\mathcal{T})$) and then finding a repair
dependent on the bug location ($p_{\text{repair}}(r\mid l,\mathcal{T})$). For
localization, we include a special NOOP location that indicates that
$\mathcal{T}$ is bug-free and no repair is necessary. In practice, we
implement the probability distributions similar to pointer networks (Gu et
al., 2016) (with the addition of an external vocabulary for repair).

Neural code representation. To learn a neural code representation, we learn a
neural encoding function $\textbf{e}(t_{i})$ that maps each token $t_{i}$ to a
vector representation. For this, we employ a BPE subtoken encoder (Sennrich et
al., 2016) (with a vocabulary of 10K subtokens) to obtain an initial token
embedding by averaging the embedding of its subtokens. Afterwards, we encode
the sequence of token embeddings via a Transformer encoder (Devlin et al.,
2019) (with relative position encodings (Shaw et al., 2018)) to obtain a
contextualized vector representation $\mathbf{r}_{i}=\textbf{e}(t_{i})$.

Localization & Repair module. To finally compute the probability distribution
over bug locations and repairs, we employ individual modules for localization
and repair based on the computed vector representation $\mathbf{r}_{i}$. The
localization module is a multilayer perceptron that computes a bugginess score
for each potential bug location based on the vector representation
$\mathbf{r}_{i}$ and the original token embedding. The objective of the
localization module is to learn how likely the token $t_{i}$ does not fit its
surrounding context (represented by $\mathbf{r}_{i}$). The localization
probability is computed as a softmax distribution over all potential bug
locations. The repair module is designed similar to CopyNet (Gu et al., 2016).
Given the vector representation $\mathbf{r}_{i}$ of a potential bug location,
the repair module computes a repairing score between the bug location and each
repair candidate at token $t_{j}$ (represented by $\mathbf{r}_{j}$). In
addition, a similar score is obtained based on token embeddings of an external
vocabulary $V$ (e.g. other binary operators). The repair probability score is
then computed as softmax distribution between all repair candidates.

###  3.3. Finding and repairing real bugs

After the successful training process, the localization and repair models are
typically confronted with new unseen programs with the objective to identify a
potential bug and repair it. This is typically done (Vasic et al., 2019) by
finding the most likely repair for the most likely bug location (according to
the model). However, the most likely repair at the most likely bug location
might not always be meaningful. For example, while the model might be
confident that a bug is located at a certain location, there might not be a
suitable repair candidate that can actually fix the bug. For this reason, we
propose an alternative strategy: Instead of taking the most likely repair for
the most likely bug location, we search for the most likely meaningful
combination of bug location and its repair (and thus ignoring bug
localizations that cannot be fixed by the model).

Beam search decoding. RealiT implements this search strategy via a beam search
decoding (Kalchbrenner and Blunsom, 2013). Here, we iterate the top-$k$ bug
locations according to the model and for each bug location we again search for
the top-$k$ token repairs. During this process, we only store pairs that are
assigned the highest likelihood together. Afterwards, we filter the candidate
pairs to only meaningful repairs: If the model predicts a bug location then
this should be fixed by a different token of the same type. Combinations of
the special NOOP location and repair are always meaningful (since nothing will
be changed). Finally, as a result, RealiT computes the most likely meaningful
repair operation:

(6) $\text{replace}(t_{l^{\prime}},r^{\prime})\text{ with }\langle
l^{\prime},r^{\prime}\rangle=\text{argmax }p(\langle
l,r\rangle\mid\mathcal{T})$

###  3.4. Implementation

To effectively measure the impact of mutants and real bug fixes on the
training process and to exclude variances due to implementation details, we
implemented RealiT together with several baselines considered during our
evaluation in a unified
framework444https://github.com/cedricrupb/nbfbaselines. In particular, we
followed the design of Hellendoorn et al. (Hellendoorn et al., 2019) by
implementing a common localization and repair framework with exchangeable
components (e.g. token encoder, localization and/or repair modules). In the
process, we reimplemented or reused state-of-the-art components for all
employed subcomponents. For example, RealiT and all Transformer-based
baselines are built upon the official BERT implementation from the transformer
library (Wolf et al., 2020). The localization and repair modules together with
the graph-based baselines are implemented closely to the implementation of the
PyBugsLab model (Allamanis et al., 2021). In addition, we reimplemented the
code preprocessing pipelines for tokenization (Hellendoorn et al., 2019) and
graph construction (Allamanis et al., 2021) (used for the graph-based
baselines) in independent libraries to facilitate reuse. Finally, we plan to
release all trained checkpoints of RealiT and all evaluated models. We think
that these are not only valuable for reproducing our results but also provide
easy access to effective models in neural bug localization and repair.

##  4\. Evaluation

We evaluate RealiT on localization and repair of single token bugs in Python.
To guide our evaluation, we specifically designed individual experiments to
address the following research questions:

RQ1:

Can RealiT improve the single token bug localization and repair performance in
comparison to techniques purely trained on mutants?

RQ2:

Is pre-training on mutants necessary for achieving a high localization and
repair performance?

RQ3:

Can training with mutants alone be sufficient for achieving a high
performance?

RQ4:

Are real bug fixes still helpful if the number of real bug fixes available for
training is further limited?

In RQ1, we compare RealiT with various techniques purely trained on mutants.
RQ2 and RQ3 are designed to explore the effect of mutation and real bug fixes
on the training process. Especially, since real bugs are hard to obtain, we
are interested whether they are really necessary for training effective bug
localizer and repairer. Finally, in RQ4, we explore how many real bug fixes
are necessary in practice to improve the localization and repair performance.

###  4.1. Bug types

To facilitate both mutants and real bug fixes, we require bug types that can
be introduced by existing mutation operators and where examples of real bug
fixes are available. For this reason, we focus on the four single token bug
types in Python that can be generated by existing mutation operators
(Derezińska and Hałas, 2014) and for which we can obtain real bug fixes for
training and evaluation (Richter and Wehrheim, 2022b). In the following, we
describe the bug types together with the employed mutation operator in more
detail.

Variable Misuse. As the main carrier of the program state, variables are
abundant in source code. Therefore, variable misuses easily occur when a
developer accidentally uses the wrong variable name instead of the intended
one. As specified by Allamanis et al. (Allamanis et al., 2017), the usage of a
wrong variable is considered as a variable misuse if the wrong usage refers to
a local variable which can be fixed by replacing it with another locally
defined variable.

Mutator: For generating variable misuses, the mutator replaces a usage of a
locally defined variable with another random variable defined in the same
context.

Wrong Binary Operator. As a traditional example for a mutation type in
mutation testing (Derezińska and Hałas, 2014), wrong binary operator bugs
appear when a binary operator is corrupted with a type-equivalent operator
(e.g. == is replaced by !=, but not by <<). For training RealiT, we consider
all types of binary operators including Boolean, arithmetic, comparison and
bitvector operators.

Mutator: Wrong binary operator mutants are generated by replacing an binary
operator with another random binary operator of the same type.

Wrong Unary Operator. In addition to binary operators, we also consider two
types of wrong unary operator bugs: logical and arithmetic negation. In
contrast to binary operators, wrong unary operators are often not replaced but
primarily occur when a unary operator is missing or accidentally added. This
includes for example a forgotten logical negation in a condition or an
accidentally added arithmetic inversion during a calculation.

Mutator: Wrong unary operator mutants are generated by randomly dropping or
inserting negation operators (either arithmetic or logical) in front of an
identifier. To ensure that inserted negations are semantically meaningful,
negations are inserted dependent on the context (e.g. logical negations in
conditions and arithmetic negations in arithmetic expressions).

Wrong Literal. Another common bug type produced by mutation operators are
literal replacements. Similar to mutation testing (Derezińska and Hałas,
2014), we are limited to literal replacements from finite sets of common
literal types. This naturally includes Boolean literal replacement by
replacing True with False and vice versa, but also integer replacements from
the set -2, -1, 0, 1, 2.

Mutator: Mutants are generated by replacing literals from a set of predefined
literals with another random literal of the same set and type.

###  4.2. Datasets

For training and evaluating RealiT, we require two types of datasets: a
general Github corpus of code snippets and a dataset of real bug fixes. To
achieve comparable results, we employ existing datasets (if available). For
the same reason, we also decided to focus on bugs in Python function
implementations.

Github code corpus. As a general corpus of Python code, we employ the ETH
Py150k dataset (Raychev et al., 2016) containing over 150k program files from
popular Python projects. The dataset is split into 100k files for training and
50k files for testing. During our evaluation, we employ the same split and,
hence, train only on Python functions obtained from train split. The test
split is used for evaluating the performance on mutants. We extract all top
level functions and deduplicate the datasets such that the Python functions
used for training only occur once in the training dataset and do not occur in
the test split. In total, our training corpus contains more than 360k Python
function implementations (after filtering).

Real bug fixes. For obtaining real bug fixes at a sufficient scale, we employ
the SSB-9M dataset (Richter and Wehrheim, 2022b) of over 9M general single
statement bug fixes in Python. The dataset does not include the necessary
implementation code itself but references the original commits in the
repositories in addition to other useful metadata. This includes information
about the code change as a Unix diff and whether the code change appears
inside a function. Based on this information, we first pre-filtered the
dataset for bug fixes that likely fall into one of our bug categories. After
the filtering process, we then mined the function code from the original
repositories. Since not all bug types can be identified purely on the code
difference (e.g. a variable misuse requires that all variables are defined in
scope), we filtered and deduplicated the resulting dataset of buggy Python
functions for a second time. This process has lead to around 35k examples of
real bug fixes that match at least one of our bug types. Finally, we use 33k
examples for training and hold out around 2k examples as a validation set used
during training.

Test benchmarks. We employ two test benchmarks to evaluate the performance on
the localization and repair task. To evaluate the localization and repair
performance on real bugs, we employ the PyPIBugs benchmark (Allamanis et al.,
2021). The benchmark is a dataset of 2374 real-world single statement bugs and
their fix derived from open source projects. The benchmark is hand-filtered
and therefore it is likely that each included bug represents a real world bug.
We only considered single token bugs (which excludes argument swaps) in
functions where the implementation is still publicly accessible555The
benchmark consists of references to bug fixing commits. We found that not all
bug fixing commits were publicly accessible at the time of writing.. This
produced a real world test benchmark of 2028 real-world bugs. To avoid an
overlap between train and test set, we excluded all near duplicates
(Allamanis, 2019) from our training datasets. Additionally, we also employ the
test portion of the Github corpus as a mutant benchmark. For this, we extend
the corpus of correct code snippets with up to 9 mutants per snippet.

Table 2. Evaluation results for bug detection and repair on mutants and real bug fixes |  |  FPR% | Real Bugs (PyPIBugs) |  | Mutants
---|---|---|---|---|---
|  | Joint | Loc. | Repair |  | Joint | Loc. | Repair
RNN (Vasic et al., 2019) |  | 33.92 | 9.47 | 13.36 | 47.88 |  | 52.39 | 61.49 | 80.06
Transformer (Hellendoorn et al., 2019) |  | 25.59 | 18.98 | 23.02 | 59.52 |  | 74.04 | 81.26 | 88.74
GNN (Allamanis et al., 2021) |  | 25.29 | 18.24 | 23.82 | 53.74 |  | 66.11 | 75.08 | 84.59
GREAT (Hellendoorn et al., 2019) |  | 29.98 | 19.03 | 23.62 | 56.31 |  | 70.76 | 78.84 | 86.99
RealiT (ours) |  | 29.53 | 39.00 | 44.23 | 73.52 |  | 67.09 | 75.95 | 85.66
RealiT - without beam search decoding |  | 22.96 | 36.69 | 41.86 | 73.52 |  | 65.27 | 74.08 | 85.66
RealiT - without pre-training on mutants |  | 19.32 | 12.67 | 16.07 | 40.38 |  | 2.37 | 8.58 | 31.34
RealiT - without fine-tuning on real bug fixes |  | 33.41 | 25.10 | 30.92 | 65.53 |  | 77.41 | 84.49 | 90.18
RealiT - with fine-tuning on postfix mutants |  | 27.72 | 27.66 | 32.64 | 67.40 |  | 78.95 | 85.12 | 90.55
RealiT - with reduced mutant frequency (5x) |  | 31.75 | 33.28 | 38.56 | 68.59 |  | 65.32 | 74.74 | 83.90

##  5\. Result

In this section, we discuss our evaluation results with the ultimate goal of
answering our research questions.

###  5.1. RQ1: RealiT in comparison?

For answering the first research question, we evaluate whether RealiT improves
the single token bug localization and repair performance by training with real
bug fixes. Since we are interested in the impact of real bug fixes on the
training process, we compare our results with several baseline algorithms
trained purely on mutants. For the comparison, we consider bug localization
and repair models based on recursive neural networks (RNN) (Vasic et al.,
2019), transformers (absolute positions) (Hellendoorn et al., 2019), graph
neural networks (GNN666Our evaluation setup differs slightly from (Allamanis
et al., 2021) in that only individual function implementations are considered.
Therefore, graph level information that would require access to the
implementation context cannot be computed. ) (Allamanis et al., 2021) and
GREAT (Hellendoorn et al., 2019). All baseline models are trained in a
supervised setting purely on mutants. The training dataset is constructed
similar to the pre-training dataset used for RealiT (with $k$ = 5 mutants
injected). The baselines are trained for 300 epochs (a 200k examples per
epoch) with early-stopping on our validation set. For RealiT, we skip the last
epoch and instead fine-tune on real bug fixes.

Real-world performance. To begin with, we start by considering the performance
of RealiT on our real-world benchmark. Table 2 provides an overview over our
evaluation results. We measured the joint accuracy of localizing and repairing
real bugs (Joint) in addition to the localization accuracy (Loc.) of finding
the bug location and the repair accuracy (Repair) of finding the real bug fix
given the bug location. In this section, we focus on the upper part of the
table and we consider the results on our real-world benchmark.

We observe that RealiT significantly outperforms all baseline algorithms
trained purely on mutants both in localization and repair of real single token
bugs. Interestingly enough, we find that the highest relative gain obtained
from fine-tuning on real bug fixes can be achieved for the localization
performance (with a nearly 2x improvement). This indicates that for effective
localization of human made bugs we actually need to learn from human made bug
fixes. Still, the bug localization remains harder than bug repair as RealiT
can fix more than 73% of all bugs when the bug location is given. Therefore,
it is important to investigate into better strategies for bug localization
(potentially by integrating techniques from static analysis).

Finally, the fact that significant performance improvements are observable in
both localization and repair suggests that real bug fixes exhibit exploitable
statistics for localization and repair. We will further explore this in RQ2
and RQ3.

Localization and repair of mutants. While our ultimate goal is to find and fix
real bugs, we also measured the localization and repair accuracy of RealiT for
artificial mutants. Surprisingly, we observe that RealiT performs worse than
most baseline models both in localization and repair after fine-tuning on real
bug fixes. Interestingly, this is not a limitation of the RealiT model as the
version of RealiT trained purely on mutants performs competitively or even
better than all baselines in localizing and repairing mutants. Therefore, the
fine-tuning on real bugs encourages RealiT to “forget” some (potentially
spurious) patterns that were used to detect mutants but do not help for
identifying real bugs. In addition, this provides further evidence that there
might exist mutants that either do not represent real bugs or represent bugs
that are highly unlikely to appear in reality. Finally, this observation is
also interesting for the evaluation of localization and repair models. As
there is clearly no correlation between the real world performance and the
performance on mutants when the model is fine-tuned on real bug fixes,
performance gains on mutants independent from real world performance become
difficult to interpret.

False positives. We also measured the false positive rate (FPR) of RealiT on
bug-free code snippets. Here, we employ the original test set of our Github
corpus. Our results are also summarized in Table 2. We observe that RealiT has
a false positive rate comparable to the other baselines – only outperformed by
the Transformer and GNN. However, we believe that an increase of 3% more false
positives is still acceptable as RealiT localizes and fixes nearly twice as
many real bugs. In addition, we also evaluate a version of RealiT without beam
search decoding (i.e. using the repair with the highest likelihood). The
results are also shown in the lower part of Table 2. We observe that while
beam search decoding improves the localization performance by up to 3%, it
also induces a worse false positive rate compared to the model without beam
search decoding. This is a common trade off between a higher true localization
performance with a worse false positive rate.

###  5.2. RQ2: Are mutants necessary?

As we have seen in RQ1, fine-tuning on real bug fixes does improve
localization and repair of real bugs. This raises the question whether mutants
are necessary for the performance gain or if the same performance can be
achieved with real bugs alone. To answer this question and therefore RQ2, we
trained two additional versions of RealiT: (1) a version of RealiT that is not
pre-trained on mutants and (2) a version of RealiT that is not fine-tuned on
real bug fixes. We evaluate both versions again on all benchmarks.

Mutants vs real bugs. We start by comparing the two new versions of RealiT.
Our evaluation results are summarized with all other results in Table 2. We
observe that training on mutants outperforms training on real bug fixes only.
It is likely that RealiT overfits the smaller training dataset of real bug
fixes and therefore fails to generalize to complete new unseen bugs. In
contrast, the version purely trained on mutants has learned from a variety of
mutants during training (some of which are likely similar to real bugs).
However, when evaluated on bug-free code snippets only, we see that RealiT
trained only on real bug fixes clearly outperforms all other techniques in
terms of false positive rate. This could again indicate that some mutants in
the training dataset are not bug inducing (e.g. a mutation that replaces <=
with != without changing the function behavior) which guides the model to
detect these structures in bug-free code.

Training with mutants and real bugs. We now compare the two variants of RealiT
with our original RealiT model. We find that fine-tuning on real bug fixes
significantly improves the performance of RealiT over the already strong
baseline of training on mutants alone. Interestingly enough, this does not
only hold for localization and repair of real bug fixes but also on the false
positive rate on bug-free code. This shows that pre-training on mutants and
fine-tuning on real bugs combines the strengths of both the high localization
and repair performance (by training on mutants) and the bug detection accuracy
(by training on real bugs). Therefore, we see that mutants are necessary to
achieve the high performance of RealiT but fine-tuning on real bugs provides
additional improvements.

Effect on individual bug types. Since the effect of pre-training and fine-
tuning seems to be complementary, we are also interested in how the training
affects the performance on individual bug types. Table 3 summarizes our
results on the real-world test benchmark divided into single token bug types.
First of all, we again find that training on both mutants and real bugs does
improve performance in both localization and repair on all bug types. However,
the margin of the performance gain is dependent on the bug type. For example,
we see the highest improvement for Wrong Binary Op where training on real bugs
alone already yields high performance. To answer our research question also
for individual bug types, pre-training on mutants can also be crucial for the
performance on individual bug types (where we observe a significant
improvement for at least three bug types Wrong Assign Op, Wrong Literal and
Variable Misuse).

Table 3. Evaluation results for bug detection and repair on different bug types Bug type |  | RealiT |  | Mutants only |  | No Mutants
---|---|---|---|---|---|---
| Joint | Loc. | Repair |  | Joint | Loc. | Repair |  | Joint | Loc. | Repair
Wrong Assign Op |  | 20.45 | 29.54 | 70.45 |  | 9.10 | 13.64 | 52.27 |  | 2.27 | 2.27 | 65.91
Wrong Binary Op |  | 56.34 | 59.15 | 84.51 |  | 14.08 | 28.17 | 39.44 |  | 30.99 | 35.21 | 70.42
Wrong Boolean Op |  | 42.31 | 42.31 | 95.05 |  | 23.08 | 24.18 | 93.41 |  | 21.43 | 21.97 | 81.87
Wrong Comparison Op |  | 36.95 | 51.47 | 67.00 |  | 19.70 | 35.22 |  57.64 |  | 23.40 | 33.74 | 57.14
Wrong Literal |  | 24.42 | 32.56 | 76.74 |  | 19.77 | 22.09 | 77.91 |  | 9.30 | 12.79 | 46.51
Variable Misuse |  | 39.87 | 42.62 | 71.75 |  | 28.73 | 31.88 | 65.13 |  | 7.43 | 9.04 | 25.75

###  5.3. RQ3: Are mutants sufficient?

Our evaluation for RQ2 has shown that training on mutants is crucial for
obtaining high performing RealiT models. Still, it is not clear whether
mutants on its own can be sufficient for training RealiT. In other words,
there might exist a mutation configuration that achieves the same performance
trained on mutants based on the same base datasets. To answer RQ3, we designed
several experiments.

Mutation frequency. We trained several versions of RealiT by varying the
mutation frequency (up to 1x, 3x, 5x, 10x, 100x and 1000x unique mutants per
code snippet). For the comparison, we measured the performance of each trained
model before and after fine-tuning on real bug fixes. The models are evaluated
on our real bugs validation set. Figure 3 gives an overview of our results for
bug localization and repair accuracy independently. The configuration 0x
represents a version of RealiT only trained on real bug fixes. First of all,
in contrast to common believe (Hellendoorn et al., 2019), we observe that
increasing the number of mutants up to 100x generated mutants per code snippet
leads to a performance improvement for both localization and repair777We
observe the same trend for joint localization and repair which is not shown
here for brevity.. This is surprising as the number of unique mutants per code
snippet is limited (with an average of 85 unique mutants per code snippet)
and, henceforth, buggy programs with more mutant candidates are oversampled.
Still, we found that increasing the limit of mutants beyond 100x (and thereby
oversampling code snippets in our dataset that provide up to 200k unique
mutants) actually decreases the localization and repair performance.

Now, when also considering the performance on the validation set after fine-
tuning, we find that fine-tuning always provides a significant boost over the
models trained solely on mutants for both localization and repair accuracy.
However, we still observe that the performance gain for localization is higher
than for repair (especially as we increase the number of mutants).
Surprisingly, we also observe that the gap between the model performances
before and after fine-tuning on real bug fixes shrinks as we increase the
number of mutants generated per code snippet (up to 100x). While this could
indicate that simply scaling the number of mutants can be sufficient for
achieving a high repair accuracy, the gap actually starts increasing again
after scaling beyond 100x mutants per code snippet. Therefore, we can conclude
that while mutants alone can significantly improve the performance they are
not sufficient in our training setup for achieving the same high performing
localization and repair models as obtained by fine-tuning on real bug fixes
with RealiT.

(a) Localization accuracy

(b) Repair accuracy

Figure 3. Effect of mutant frequency during training on the real world
validation performance. The gray dotted line represents the average number of
unique mutants that can be generated per code snippet.

Training with postfix mutants. While it seems that mutants alone introduced in
arbitrary code from our code corpus is not sufficient for closing the
performance graph to fine-tuned models, it is unclear whether mutants can be
sufficient when introduced in an implementation context that is more typical
for a real bug. To test this, we designed an additional experiment where we
fine-tuned RealiT on postfix mutants (i.e. mutants that are produced by first
applying the real bug fix and then reintroducing a bug with a mutation
operator). Our results are also shown in Table 2. We observe that even fine-
tuning on postfix mutants provides a slight boost in performance both on
localization and repair of mutants and real bugs. Surprisingly, the boost is
slightly higher for real bugs than for mutants even though we only increased
the number of mutants. Still, we find that a model trained on real bug fixes
clearly outperforms a model trained on postfix mutants when evaluated on real
bug fixes. Since the performance of detecting mutants does not decrease when
training on postfix mutants (similar to scaling the number of mutants
generated), we actually conclude that the performance gain is likely a scaling
effect and can not necessarily be attributed to the mutation context.

In total, we find that training on mutants alone is not sufficient for
achieving a high performing RealiT model. In addition, our results show that
training on real bug fixes is especially helpful for the localization of real
bugs which is hard to obtain by training on mutants alone.

###  5.4. RQ4: How many bug fixes are necessary?

Obtaining a dataset of multiple thousand real bug fixes can be challenging
especially if the considered bug type occurs less frequent in open source
projects. For this reason, we aim to explore how much the size of the fine-
tuning dataset (the number of real bug fixes) influence the final performance
of RealiT. Therefore, we evaluate three variants of RealiT pre-trained on 1x,
5x and 100x mutants per code snippet which we then fine-tune on several
subsamples of our real bug fix datasets. We consider subsamples of 1% (334),
3% (996), 5% (1.658), 10% (3.314), 30% (9.936), 50% (16.559), 70% (23.180),
90% (29.802) of all bug fixes. To obtain more stable results, we fine-tune our
models on three subsamples per sample size and evaluate fine-tuned models on
our validation set. Averaged results for all three RealiT variants fine-tuned
on the generated subsamples are reported in Figure 4.

Impact of the real bug fix dataset size. We can observe a clear trend that
more real bug fixes lead to an improved performance across all RealiT
variants. This holds true even for small fine-tuning datasets of around 1000
(less than 5% of the original dataset size) bug fixes. As reported by the
authors of ManySStuBs4J (Karampatsis and Sutton, 2020) or PySStuBs (Kamienski
et al., 2021), real bug fix collections of this size can be obtained by mining
the top 1000 most popular Github projects for the respective language.

Scaling real bug fixes vs. scaling mutants. Although we have seen that fine-
tuning on more real bug fixes increases the performance, it is actually
difficult to scale up the number of real bug fixes (as the number of publicly
accessible projects to mine real bug fixes from is limited). In contrast,
generating more mutants per code snippet is more cost effective. For example,
to achieve the same performance gain obtained from scaling the number of
mutants generated from 5x to 100x, we have to fine-tune on at least 10% of our
real bug fix dataset (3314 bugs). Still, although scaling the number of
mutants is preferred, the scaling effect is however limited, as we have seen
in RQ3.

Figure 4. Effect of real bug fixes on the fine-tuning performance on the
validation set. The x-axis is the percentage of the bug fix dataset used for
fine-tuning. Gray dotted lines mark datasets that exceed 1k and 10k examples
respectively.

##  6\. Threats to validity

Although a variety of learning-based bug localization and repair models have
been developed in recent years, there does not exist a unique setup for
training and benchmarking these models which is universally accepted.
Therefore, even though we implemented our baselines close to the reference
implementation, the resulting trained models might behave differently than in
the setup they were originally developed for. To still achieve comparable
results, we designed our evaluation to replicate prior studies (Hellendoorn et
al., 2019) on neural bug localization and repair models as close as possible.
For example, we adopted the same publicly accessible Github corpus ETH Py150k,
a similar architectural design and similar baselines as employed by
Hellendoorn et al. (Hellendoorn et al., 2019). To further support a wider
range of bug types which allowed us to exploit the hand validated benchmark
PyPIBugs, we adjusted the architecture and mutation process similar to
Allamanis et al. (Allamanis et al., 2021). Still, our evaluation results for
the baseline algorithms (in Table 2) are slightly different than the results
of prior studies. For example, we found that while graph-based models such as
GNNs and GREAT still perform better in localizing and repairing real bugs,
Transformers show a surprisingly strong performance on mutants. Note however
that Hellendoorn et al. (Hellendoorn et al., 2019) anticipated this result
(even though they only evaluated on variable misuses) when the models are
trained for a longer duration – which we did by training on approximately 2.4x
more training examples. In contrast to the results of Allamanis et al.
(Allamanis et al., 2021), we observe that graph-based models underperform in
our evaluation setup which we attribute to two main differences: (1) for a
fair comparison, all models only have access to the function implementation
without the implementation context which prohibits the computation of type
related or call structure related information exploited by the graph-based
models and (2) we trained all models on a different (potentially smaller)
dataset. Although integrating the type of information and training on a larger
dataset would potentially benefit all baselines, the performance ranking
between architectures might differ. However, since our experiments showed that
the performance gain due to training on real bug fixes is unique and the
effect could not be replicated by training on mutants, we expect that adapting
our evaluation setup has little to no influence on our evaluation outcome.

##  7\. Related Work

We discuss the most related previous or concurrent work that (1) tackle single
token bug localization and repair with alternative training strategies, (2)
exploit real bug fixes for automatic program repair or code mutations and (3)
consider alternative pre-train-and-fine-tune techniques.

Single token bug localization and repair. The detection and repair of single
token bugs have been explored in previous work (Allamanis et al., 2017; Pradel
and Sen, 2018; Vasic et al., 2019; Hellendoorn et al., 2019; Richter and
Wehrheim, 2022a; Patra and Pradel, 2021; Allamanis et al., 2021). Allamanis et
al. (Allamanis et al., 2017) addressed the detection and repair of variable
misuse bugs (which we also considered in this work) by representing programs
as graphs. Vasic et al. (Vasic et al., 2019) proposed a joint model for same
task and Hellendoorn et al. (Hellendoorn et al., 2019) explored alternative
program representations. These techniques all have in common that they do not
learn from real bug fixes but from artificially mutated code. In contrast,
while RealiT employs a similar Transformer-based architecture as discussed by
Hellendoorn et al. (Hellendoorn et al., 2019), we showed that integrating real
bug fixes in the training process is crucial for the localization and repair
of real bugs. More recent work (Richter and Wehrheim, 2022a; Patra and Pradel,
2021; Allamanis et al., 2021) also showed that the quality of training data is
important for effective bug localization and repair. For example, employing a
more realistic mutator (Richter and Wehrheim, 2022a; Patra and Pradel, 2021)
(i.e. a mutator that is more likely to reproduce a real bug) or learning to
inject hard to find bugs (Allamanis et al., 2021) can both improve the
localization and repair performance. However, the integration of these
approaches often increases the complexity by requiring to learn a mutation
operator either prior and concurrent to the training process. With RealiT, we
showed that integrating real bug fixes, while relying on simpler and easier to
implement mutation operators, can be sufficient to obtain a significant
improvement in real bug localization and repair performance. Interestingly
enough, a concurrent work (He et al., 2022) also explored whether real bug
fixes have an impact on the performance of learning-based bug detectors.
Similar to RealiT, their model is pre-trained on mutants and then fine-tuned
real bug fixes. Surprisingly, while the authors found that fine-tuning on real
bug fixes improves precision (i.e. the number of correct programs classified
as buggy), the recall (i.e. the number of real bugs detected and repaired)
actually suffers. In contrast, we find that RealiT improves the number of bugs
detected and repaired significantly while training on real bug fixes can also
decrease the false positive rate. We attribute the difference in our findings
to significant differences to the RealiT training process: (1) the number of
real bug fixes we fine-tune on is several magnitudes larger, (2) the number of
mutants generated per code snippet is significantly higher and (3) the
distribution of buggy and bug-free programs is balanced both during pre-
training and fine-tuning. We believe that especially (3) is key to success of
RealiT. Training on an unbalanced dataset (with significant more bug-free than
buggy code) risks that the model defaults to not detecting a bug (which would
result in a higher precision and lower recall by design).

Learning from real bug fixes. Real bug fixes are not only a valuable resource
for learning to localize and repair single token bugs but they can also be
effectively exploited for automatic program repair (Tufano et al., 2019b; Chen
et al., 2019; Li et al., 2020; Lutellier et al., 2020; Bader et al., 2019) or
code mutations (Tufano et al., 2019a; Patra and Pradel, 2021). SequenceR (Chen
et al., 2019), for example, learns from thousands of bug fixes to predict one-
line bug patches. Dlfix (Li et al., 2020) and CoCoNuT (Lutellier et al., 2020)
improved the repair performance by proposing more effective learning
strategies. In contrast to RealiT, however, these techniques are designed to
only repair a given program location and, hence, whether a program is buggy
and where the bug has to be fixed has to be known beforehand. In addition,
these techniques are often trained on real bug fixes only without considering
mutants for the training process. We showed that learning from mutants is
actually crucial to achieve high performing models. This observation is also
supported by DrRepair (Yasunaga and Liang, 2020) which showed that pre-
training a repair models on artificial errors improved the repair performance
on syntactic errors. Still, their approach rely on a compiler to detect this
type of bugs. The type of single token bugs which we considered in this work
are typically missed by a compiler.

Code mutation addresses the inverse problem of injecting a bug into a correct
program. Tufano et al. (Tufano et al., 2019a) and Patra and Pradel (Patra and
Pradel, 2021) showed that bug fixes can be effectively leverage to learn code
mutations by learning to replicate the original bug. Interestingly, Yasunaga
et al. (Yasunaga and Liang, 2021) showed that repeatedly training a breaker
and fixer that initially learn from real bug fixes but then provide training
data for each other actually improves the performance of the fixer to repair
syntactic bugs. While our work showed that real bug fixes are also crucial for
bug detection, we believe that exploiting real bug fixes in the mutation
process for training bug detection and repair models can be a promising
direction for future work.

Pre-training and fine-tuning. Pre-training on large corpora of fuzzy data and
then fine-tuning on a specific task with a smaller dataset has been shown to
be highly successful in domains such as natural language processing (Devlin et
al., 2019; Raffel et al., 2020), image processing (Kolesnikov et al., 2020)
and most recently programming language processing (Feng et al., 2020; Kanade
et al., 2020). In contrast to RealiT, these techniques are often pre-trained
on a generic unrelated task where data is available before fine-tuning them on
a specific task. RealiT, however, is trained and fine-tuned with same
architecture with largely the same objective of identifying and repairing
buggy (or mutated) code.

CuBERT (Kanade et al., 2020) showed that pre-training on a generic corpus of
Python code can improve the detection performance on variable misuses.
However, the authors employed mutants instead of real bug fixes in the fine-
tuning phase. In contrast, RealiT is pre-trained on mutants and then fine-
tuned on real bug fixes. A combination of these two approaches by applying
RealiT on top of a pre-trained model would be interesting and we leave this
open for future work.

##  8\. Conclusion

In this work, we explore the effect of training on real bug fixes and mutants
on the performance of bug localization and repair models. For this, we propose
RealiT, a novel pre-train-and-fine-tune approach for learning to localize and
repair bugs with Transformers. RealiT can effectively utilize both mutants and
real bug fixes during training by first pre-training on mutants and then fine-
tuning on real bug fixes. Our evaluation on thousands of real bugs obtained
from real Python projects showed that RealiT can significantly improve the
localization and repair of real bugs in contrast to models solely trained on
mutants. In addition, our experiments showed (1) that pre-training on mutants
plays an important role for achieving the performance level, (2) that mutants
alone are however not sufficient to unlock the potential of RealiT and (3)
that a high number of real bug fixes is actually necessary for achieving a
high performing model.

Based on these observations, we see as future work the integration of more
realistic data in the training process of neural bug localization and repair
models. For example, training on more realistic mutants could boost the
performance even before fine-tuning on real bug fixes. In addition, it might
also be interesting to explore the effect of other – even unrelated – types of
bug fixes on the training process of neural bug localization and repair
approaches. Integrating more supported bug types also allows us to exploit
more real bug fixes found in open source projects.

Finally, to conclude, RealiT demonstrates that neural bug localization and
repair models can effectively learn from developer mistakes, in form of real
bug fixes, to localize and repair real bugs.

## References

  * (1)
  * Alaboudi and LaToza (2021) Abdulaziz Alaboudi and Thomas D. LaToza. 2021.  An Exploratory Study of Debugging Episodes.  _CoRR_ abs/2105.02162 (2021).  https://arxiv.org/abs/2105.02162
  * Allamanis (2019) Miltiadis Allamanis. 2019\.  The adverse effects of code duplication in machine learning models of code. In _Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software_. 143–153.
  * Allamanis et al. (2017) Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. 2017\.  Learning to represent programs with graphs.  _arXiv preprint arXiv:1711.00740_ (2017).
  * Allamanis et al. (2021) Miltiadis Allamanis, Henry Jackson-Flux, and Marc Brockschmidt. 2021\.  Self-supervised bug detection and repair.  _Advances in Neural Information Processing Systems_ 34 (2021), 27865–27876.
  * Bader et al. (2019) Johannes Bader, Andrew Scott, Michael Pradel, and Satish Chandra. 2019.  Getafix: Learning to fix bugs automatically.  _Proceedings of the ACM on Programming Languages_ 3, OOPSLA (2019), 1–27.
  * Chen et al. (2019) Zimin Chen, Steve Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and Martin Monperrus. 2019.  Sequencer: Sequence-to-sequence learning for end-to-end program repair.  _IEEE Transactions on Software Engineering_ 47, 9 (2019), 1943–1959.
  * Derezińska and Hałas (2014) Anna Derezińska and Konrad Hałas. 2014.  Analysis of mutation operators for the python language. In _Proceedings of the Ninth International Conference on Dependability and Complex Systems DepCoS-RELCOMEX. June 30–July 4, 2014, Brunów, Poland_. Springer, 155–164.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.  https://doi.org/10.18653/v1/N19-1423
  * Feng et al. (2020) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020\.  Codebert: A pre-trained model for programming and natural languages.  _arXiv preprint arXiv:2002.08155_ (2020).
  * Gu et al. (2016) Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK Li. 2016\.  Incorporating copying mechanism in sequence-to-sequence learning.  _arXiv preprint arXiv:1603.06393_ (2016).
  * He et al. (2022) Jingxuan He, Luca Beurer-Kellner, and Martin Vechev. 2022\.  On Distribution Shift in Learning-based Bug Detectors.  _arXiv preprint arXiv:2204.10049_ (2022).
  * Hellendoorn et al. (2019) Vincent J Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. 2019\.  Global relational models of source code. In _International conference on learning representations_.
  * Kalchbrenner and Blunsom (2013) Nal Kalchbrenner and Phil Blunsom. 2013.  Recurrent convolutional neural networks for discourse compositionality.  _arXiv preprint arXiv:1306.3584_ (2013).
  * Kamienski et al. (2021) Arthur V Kamienski, Luisa Palechor, Cor-Paul Bezemer, and Abram Hindle. 2021.  Pysstubs: Characterizing single-statement bugs in popular open-source python projects. In _2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)_. IEEE, 520–524.
  * Kanade et al. (2020) Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020.  Pre-trained Contextual Embedding of Source Code.  _CoRR_ abs/2001.00059 (2020).  arXiv:2001.00059 http://arxiv.org/abs/2001.00059
  * Karampatsis and Sutton (2020) Rafael-Michael Karampatsis and Charles Sutton. 2020.  How Often Do Single-Statement Bugs Occur?: The ManySStuBs4J Dataset. In _MSR_. ACM, 573–577.  https://doi.org/10.1145/3379597.3387491
  * Kolesnikov et al. (2020) Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. 2020\.  Big transfer (bit): General visual representation learning. In _European conference on computer vision_. Springer, 491–507.
  * Li et al. (2020) Yi Li, Shaohua Wang, and Tien N Nguyen. 2020.  Dlfix: Context-based code transformation learning for automated program repair. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_. 602–614.
  * Lutellier et al. (2020) Thibaud Lutellier, Hung Viet Pham, Lawrence Pang, Yitong Li, Moshi Wei, and Lin Tan. 2020.  Coconut: combining context-aware neural translation models using ensemble for program repair. In _Proceedings of the 29th ACM SIGSOFT international symposium on software testing and analysis_. 101–114.
  * Merity et al. (2016) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016.  Pointer sentinel mixture models.  _arXiv preprint arXiv:1609.07843_ (2016).
  * Patra and Pradel (2021) Jibesh Patra and Michael Pradel. 2021.  Semantic bug seeding: a learning-based approach for creating realistic bugs. In _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 906–918.
  * Pradel and Sen (2018) Michael Pradel and Koushik Sen. 2018.  Deepbugs: A learning approach to name-based bug detection.  _Proceedings of the ACM on Programming Languages_ 2, OOPSLA (2018), 1–25.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020\.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ 21, 140 (2020), 1–67.
  * Raychev et al. (2016) Veselin Raychev, Pavol Bielik, and Martin Vechev. 2016\.  Probabilistic model for code with decision trees.  _ACM SIGPLAN Notices_ 51, 10 (2016), 731–747.
  * Richter and Wehrheim (2022a) Cedric Richter and Heike Wehrheim. 2022a.  Learning Realistic Mutations: Bug Creation for Neural Bug Detectors. In _2022 IEEE Conference on Software Testing, Verification and Validation (ICST)_. IEEE, 162–173.
  * Richter and Wehrheim (2022b) Cedric Richter and Heike Wehrheim. 2022b.  TSSB-3M: Mining single statement bugs at massive scale.  _arXiv preprint arXiv:2201.12046_ (2022).
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016\.  Neural Machine Translation of Rare Words with Subword Units. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Berlin, Germany, 1715–1725.  https://doi.org/10.18653/v1/P16-1162
  * Shaw et al. (2018) Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018\.  Self-attention with relative position representations.  _arXiv preprint arXiv:1803.02155_ (2018).
  * Tufano et al. (2019a) Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. 2019a.  Learning how to mutate source code from bug-fixes. In _2019 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE, 301–312.
  * Tufano et al. (2019b) Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. 2019b.  An empirical study on learning bug-fixing patches in the wild via neural machine translation.  _ACM Transactions on Software Engineering and Methodology (TOSEM)_ 28, 4 (2019), 1–29.
  * Vasic et al. (2019) Marko Vasic, Aditya Kanade, Petros Maniatis, David Bieber, and Rishabh Singh. 2019.  Neural program repair by jointly learning to localize and repair.  _arXiv preprint arXiv:1904.01720_ (2019).
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020\.  Transformers: State-of-the-Art Natural Language Processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_. Association for Computational Linguistics, Online, 38–45.  https://www.aclweb.org/anthology/2020.emnlp-demos.6
  * Yasunaga and Liang (2020) Michihiro Yasunaga and Percy Liang. 2020.  Graph-based, self-supervised program repair from diagnostic feedback. In _International Conference on Machine Learning_. PMLR, 10799–10808.
  * Yasunaga and Liang (2021) Michihiro Yasunaga and Percy Liang. 2021.  Break-it-fix-it: Unsupervised learning for program repair. In _International Conference on Machine Learning_. PMLR, 11941–11952.

##  Appendix A Model Architectures

For our evaluation, we implemented our baselines in a common code base. All
neural network modules are implemented in PyTorch. In the following, we
discuss the general architecture used for neural bug localization and repair
and the design and hyperparameters individually for all baseline models.

General. All our models follow the general structure proposed by Hellendoorn
et al. (Hellendoorn et al., 2019). The architecture consists of an input
module (for mapping tokens to vectors), a central encoding model and
localization and repair head. For constructing our baselines, we change the
central encoding model. The remaining structure remains the same (if not
specified otherwise). For the input module, we use a BPE subtoken encoder with
a vocabulary of 10k subtokens and embed each token by averaging its subtoken
representation.

Similar to Allamanis et al. (Allamanis et al., 2021), we employ dedicated
heads for localization and repair.

For localization, we use an architecture similar to pointer networks (Merity
et al., 2016). Given a program $\mathcal{T}=t_{0},t_{1},\dots,t_{n}$ and let
$t_{l}$ be a potential bug location, we then compute the initial token
embedding $e_{l}$ and the contextual vector representation $\mathbf{r}_{l}$
coming from the encoding model. Based on these representations, we compute a
buggyness score for each potential bug location with a simple MLP:

$s_{l}=\mathbf{W}_{2}\sigma(\mathbf{W}_{1}(\mathbf{r}_{l}||e_{l}||\mathbf{r}_{l}-e_{l}))$

Here, $\mathbf{W}_{2}\in\mathbb{R}^{1\times d}$,
$\mathbf{W}_{1}\in\mathbb{R}^{d\times 3d}$ are learnable projections of the
MLP. The intuition here is that the MLP should learn the correct token
representation $\mathbf{r}_{l}$ which then would disagree with the initial
token embedding $e_{l}$ if $t_{l}$ is buggy. We model the distribution
$p_{loc}$ by a softmax over all buggyness scores.

Based on the same intuition used for localization, we designed our repair
module. Given a potential bug location represented by $\mathbf{r}_{l}$, the
repair module computes a repair score for all other tokens (represented by
$\mathbf{r}_{j}$) similar to an attention mechanism:

$rep_{lj}=\frac{\mathbf{W}_{q}(\mathbf{r}_{l})(\mathbf{W}_{k}(\mathbf{r}_{j}))^{T}}{\sqrt{d}}$

Here, $\mathbf{W}_{q}\in\mathbb{R}^{d\times d}$,
$\mathbf{W}_{k}\in\mathbb{R}^{d\times d}$ are learnable projections. To
include an external vocabulary $V$, we represent each vocabulary entry a
learnable vector $v_{j}\in\mathbb{R}^{d}$ and compute a repair score in a
similar way:

$rep_{lj}=\frac{\mathbf{W}_{q}(\mathbf{r}_{l})(v_{j})^{T}}{\sqrt{d}}$

Finally, $p_{repair}$ is computed by a softmax over all repair scores (token
based and vocabulary based together).

We train all models using the Adam optimizer with learning rate $1e-4$ and a
linear warm-up of 800 steps, additionally clipping gradient norms at 1.0 (0.5
for the GNN). Models are trained with weight decay of 0.1 for regularization.
During training, we consider function implementations with up to 1024 tokens
(1536 nodes for the GNN) and trained with minibatch sizes of up to 12.5K
tokens (nodes).

RealiT. We follow the same architectural design for RealiT. As an encoding
model, we employ a 6-layer Transformer encoder (Devlin et al., 2019), a hidden
size of 512, an intermediate size of 2048 and 8 attention heads. During
training, we use a dropout regularization of 0.1. For encoding the positions
of tokens, we employ relative position encoding (Shaw et al., 2018) as we
found that this performed better. A comparison of Transformer with and without
relative position encoding can be found in Table 2 (”RealiT - without fine-
tuning on real bug fixes” vs ”Transformer”).

GNN. For the graph neural network baseline, we followed the design of
Allamanis et al. (Allamanis et al., 2021) as close as possible. We
reimplemented the GNN based on the reference implementation provided by the
authors. The GNN consists of 8 message propagation layers with a skip
connection between the first and the fourth layer and between the fourth and
the eights layer. The node hidden size is set to 256.

In addition, we also adapted the general architecture to match the reference
implementation. Instead of averaging the subtoken embeddings, we employ max
pooling as the authors found that this performed better. In addition, we also
reimplemented the same localization and repair head.

Remaining baselines. The remaining baselines employ the same hyperparameters
as specified by Hellendoorn et al. (Hellendoorn et al., 2019). The Transformer
is a 6-layer encoder with absolute position embeddings (512 hidden size, 2048
intermediate size, 8 attention heads). GREAT uses a similar 6-layer
architecture with the addition of an edge bias. The RNN is a 2-layer
bidirectional recursive neural network with hidden size of 512.