# Boosting Automated Patch Correctness Prediction via Pre-trained Language
Model

Quanjun Zhang, Chunrong Fang*, Weisong Sun, Yan Liu, Tieke He*, Xiaodong Hao,
Zhenyu Chen  Quanjun Zhang, Chunrong Fang, Weisong Sun, Yan Liu, Tieke He,
Xiaodong Hao and Zhenyu Chen are with the State Key Laboratory for Novel
Software Technology, Nanjing University, China.
E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>*Chunrong Fang and Tieke He are
the corresponding authors. Manuscript received xxx xxx, 2022; revised xxx xxx,
2022.

###### Abstract

Automated program repair (APR) aims to fix software bugs automatically without
human debugging efforts and plays a crucial role in software development and
maintenance. Despite the recent significant progress in the number of fixed
bugs, APR is still challenged by a long-standing overfitting problem (i.e.,
the generated patch is plausible but overfitting). Various techniques have
thus been proposed to address the overfitting problem. Among them, leveraging
deep learning approaches to predict patch correctness automatically is
emerging along with the available large-scale patch benchmarks recently.
However, existing learning-based techniques mainly rely on manually-designed
code features, which can be extremely costly and challenging to construct in
practice. In this paper, we propose APPT, a pre-trained model-based automated
patch correctness assessment technique, which treats the source code as a
sequence of tokens without extra overhead to design a mass of features from
different perspectives. In particular, APPT adopts a pre-trained model as the
encoder stack, followed by an LSTM stack and a deep learning classifier.
Although our idea is general and can be built on various existing pre-trained
models, we have implemented APPT based on the BERT model. We conduct an
extensive experiment on 1,183 Defects4J patches and the experimental results
show that APPT achieves prediction accuracy of 79.0% and recall of 81.3%,
outperforming the state-of-the-art technique CACHE by 3.6% and 4.8%. Our
additional investigation on 49,694 real-world patches shows that APPT achieves
the optimum performance (exceeding 99% in five common metrics for assessing
patch classification techniques) compared with existing representation
learning techniques. We also prove that adopting advanced pre-trained models
can further provide substantial advancement (e.g., GraphCodeBERT-based APPT
improves BERT-based APPT by 3.0% and 2.6% in precision and recall,
respectively), highlighting the generalizability of APPT.

###### Index Terms:

Automated Program Repair, Patch Correctness, Pre-trained Model

##  1 Introduction

Software bugs are inevitable in modern software systems and result in fatal
consequences, such as costing trillions of dollars in financial loss and
affecting billions of people around the world [1, 2]. It is incredibly time-
consuming and labor-intensive for developers to fix such bugs due to the
increasing size and complexity of modern software systems [3]. Automated
program repair (APR) aims to fix revealed software bugs without human
intervention automatically and has attracted massive attention from both
academia and industry in the past decades [4, 5]. Despite an emerging research
area, a variety of APR techniques have been proposed and continuously achieved
promising results in terms of the number of fixed bugs in the literature [6,
7].

However, it is fundamentally difficult to achieve high precision for generated
patches due to the weak program specifications [8]. Existing APR techniques
usually leverage the developer-written test cases as the criteria to assess
the correctness of the generated patches. In fact, a generated patch passing
the available test cases may not generalize to other potential test cases,
leading to a long-standing challenge of APR (i.e., the overfitting issue) [8].
For example, when a bug is detected in functionality, a patch can be simply
generated by deleting the functionality and the available test cases usually
fail to exercise the deleted functionality [9]. In this case, developers need
to consume tremendous time and effort to filter the overfitting patches,
resulting in a negative debugging performance when APR techniques are applied
in practice [10, 11].

Thus, various automated patch correctness assessment (APCA) techniques have
been proposed to determine whether a generated patch is indeed correct or not
[12]. According to extracted features, the traditional APCA techniques can be
categorized into two groups: static and dynamic ones[13]. Static techniques
tend to analyze the code changed patterns or code similarity based on the
syntactic and semantic features. For example, Tan et al. [14] define a set of
generic forbidden transformations (e.g., the above-mentioned functionality
deleting) for the buggy program. In contrast, dynamic techniques usually
execute the plausible patches against extra test cases generated by automated
test generation tools (e.g., Evosuite [15] and Randoop [16]). For example,
Xiong et al. [17] generate new test cases and determine patch correctness
based on the behavior similarity of the test case executions. However, the
static techniques may suffer from prediction precision problems, while it is
pretty time-consuming for dynamic techniques to generate additional test cases
and execute all patched programs [13].

Recently, inspired by large-scale patch benchmarks being released [7, 6], some
learning-based APCA techniques have been proposed to assess patch correctness
by embedding buggy and patched code snippets [18, 12, 19]. For example, Lin et
al. [20] leverage the abstract syntax tree (AST) path to represent the patch
and build a deep learning classifier to predict the correctness of the patch.
Similarly, He et al. [18] extract code features at the AST level statically
and train a probabilistic model to perform patch prediction. However, despite
outstanding prediction results, existing learning-based APCA techniques mainly
employ complex code-aware features (e.g., AST path in [20]) or manually-
designed code features (e.g., 202 code features in [18]), which are costly to
conduct and extract in practice.

In this work, we propose, APPT, the first Automated Pre-trained model-based
Patch correcTness assessment technique, which employs the pre-training and
fine-tuning to address the above limitation of prior work. We first adopt the
large pre-trained model as the encoder stack to extract code representations.
We then employ bidirectional LSTM layers to capture rich dependency
information between the buggy and patched code snippets. Finally, we build a
deep learning classifier to predict whether the patch is overfitting or not.
APPT treats only the source code tokens as the input and automatically
extracts code features using a well-trained encoder stack, getting rid of the
need for code-aware features and manually-designed features. Although APPT is
conceptually general and can be built on various pre-trained models, we have
implemented APPT as a practical APCA tool based on the BERT model. Our
experimental results on 1,183 Defects4J patches indicate that APPT improves
the state-of-the-art technique CACHE by 3.6% accuracy, 1.2% precision, 4.8%
recall, 2.9% F1-score and 3.1% AUC. We conduct an additional investigation on
49,694 real-world patches from five different patch benchmarks and the results
show that APPT exceeds 99% in accuracy, precision, recall, F1-score and AUC
metrics, outperforming the existing representation learning techniques. We
also adopt different pre-trained models to further investigate the
generalization ability of APPT. The results demonstrate that APPT with
advanced pre-trained models can enhance the prediction performance. For
example, precision and recall of APPT can be improved by 3.0% and 2.6% when
equipped with GraphCodeBERT, which are 4.2% and 7.2% higher than the state-of-
the-art technique CACHE.

To sum up, we make the following major contributions:

  * •

New Direction. This paper opens a new direction for patch correctness
prediction to directly utilize large pre-trained models by pre-training and
fine-tuning. Compared with existing learning-based APCA techniques, our
approach does not need any additional efforts to design and extract complex
code features.

  * •

Novel Technique. We propose APPT, a BERT-based APCA technique that leverages
the pre-training and classifier to predict patch correctness. To the best of
our knowledge, we are the first to exploit fine-tuning the pre-trained model
for assessing patch correctness.

  * •

Extensive Study. We conduct various empirical studies to investigate and
evaluate APPT on diverse patch benchmarks. The results show that APPT achieves
significantly better overall performance than existing learning-based and
traditional APCA techniques.

  * •

Available Artifacts. We release the relevant materials (including source code,
patches and results) used in the experiments for replication and future
research111All artifacts relevant to this work can be found at
anonymouswebsite, accessed August 2022..

##  2 Background

###  2.1 Automated Program Repair

Figure 1: Overview of APR

APR techniques’ primary objective is to identify and fix program bugs
automatically. Fig. 1 illustrates the workflow of the typical APR technique,
which is usually composed of three steps: (1) the localization phrase utilizes
off-the-shelf fault localization techniques to recognize the suspicious code
elements (e.g., statements or methods) [21, 22]; (2) the repair phrase then
modifies these elements based on a set of transformation rules to generate
various new program variants, also called candidate patches; (3) the
verification phrase adopts the original test cases as the oracle to check
whether candidate patches execute as expected or not. Specifically, a
candidate patch passing the original test cases is called a _plausible_ patch.
A plausible patch that is semantically equivalent to the developer patch
denotes a _correct_ patch; otherwise, it is an _overfitting_ patch.

It is fundamentally challenging to ensure the correctness of the plausible
patches due to the weak specification of the program behavior in practice.
Existing studies have demonstrated that manually identifying the overfitting
patches is time-consuming and may harm the debugging performance of developers
[10, 23]. Thus, various techniques have been proposed to validate patch
correctness automatically. According to whether the dynamic execution or
machine learning is required [13], we categorize them into three main
categories: static-based techniques, dynamic-based techniques and learning-
based techniques.

_$\bullet$ Static-based APCA techniques._ These techniques aim to prioritize
correct patches over overfitting ones by static code features, such as code-
deleting program transformations.

_$\bullet$ Dynamic-based APCA techniques._ These techniques aim to filter out
overfitting patches by executing extra test cases, which are generated based
on fixed or patched programs. According to whether the correct patches are
required, these techniques can be further categorized into _dynamic with
oracle-based ones_ and _dynamic without oracle-based ones_.

_$\bullet$ Learning-based APCA techniques._ These techniques aim to predict
the correctness of plausible patches enhanced by machine learning techniques.
They usually extract the manually-designed code features and then adopt a
classifier to perform patch prediction [18]. Some techniques are proposed to
adopt code embedding techniques to extract code features automatically [20],
which are also denoted as _representation learning-based APCA techniques_.

Recently, an increasing number of research efforts have attempted to use
machine learning techniques to learn from existing patch benchmarks for
predicting potential patch correctness, achieving promising results. In this
work, we adopt the large pre-trained model (i.e., BERT) to encode plausible
patches and train a deep learning classifier to predict patch correctness.
Compared to existing techniques, our paper is the first work to predict patch
correctness by pre-training and fine-tuning the pre-trained model.

###  2.2 Pre-trained Model

Recently, Pre-trained language models (e.g., BERT) have significantly improved
performance across a wide range of natural language processing (NLP) tasks,
such as machine translation and text classification [24, 25, 26]. Typically,
the models are pre-trained to derive generic language representations by self-
supervised training on large-scale unlabeled data and then are transferred to
benefit multiple downstream tasks by fine-tuning on limited data annotation.

Existing pre-trained models usually adopt the encoder-decoder architectures,
where an encoder encodes an input sequence as a fixed-length vector
representation, and a decoder generates an output sequence based on the input
representation. Encoder-only models (e.g., BERT [24]) usually pre-train a
bidirectional transformer in which each token can attend to each other.
Encoder-only models are good at understanding tasks (e.g., code search), but
their bidirectionality nature requires an additional decoder for generation
tasks, where this decoder initializes from scratch and cannot benefit from the
pre-training tasks. Decoder-only models (e.g., GPT [25]) are pre-trained using
unidirectional language modeling that only allows tokens to attend to the
previous tokens and itself to predict the next token. Decoder-only models are
good at auto-regressive tasks like code completion, but the unidirectional
framework is sub-optimal for understanding tasks. Encoder-decoder models
(e.g., T5 [26]) often make use of denoising pre-training objectives that
corrupt the source input and require the decoder to recover them. Compared to
encoder-only and decoder-only models that favor understanding and auto-
regressive tasks, encoder-decoder models can support generation tasks like
code summarization. In this work, we treat the patch correctness assessment as
a binary classification task and we consider encoder-only models to get
embeddings of code snippets according to existing work [27].

Inspired by the success of pre-trained models in NLP, many recent attempts
have been adopted to boost numerous code-related tasks (e.g., code
summarization and code search) with pre-trained models (e.g., GraphCodeBERT)
[28, 29]. Despite the promising results, little work aims to explore the
capabilities of pre-trained models in supporting patch correctness assessment.
In this work, BERT is selected to exploit pre-trained models for automated
patch correctness assessment, as it has been widely adopted in various code-
related tasks and is quite effective for classification tasks [28, 29]. Two
advanced BERT-style models (i.e., CodeBERT and GraphCodeBERT) are also
selected to investigate the generalization ability of APPT.

##  3 Approach

Figure 2: Overview of APPT

Fig. 2 presents the overall framework of our approach. Generally, APPT accepts
a buggy program and a plausible patch that passes the available test cases as
inputs. APPT extracts the buggy code snippet and its corresponding patched
code snippet, and adopts four strategies to truncate the code tokens. APPT
then uses the pre-trained BERT model for embedding the truncated tokens. After
obtaining the representations for the buggy and patched code snippets, APPT
uses four pre-defined functions for integrating the representations. Finally,
APPT adopts a deep learning classifier to return the final result (i.e.,
correct or overfitting).

###  3.1 Code Extraction

Given a buggy program, existing APR tools may return a plausible patch $p$ (if
it exists) that passes all available test cases. Code extraction phrase aims
to take the returned patch and the buggy program as the inputs, and output the
corresponding buggy and patched code tokens (shown in Fig. 2(a)).

Specifically, we get the buggy and patched code snippets (i.e., $C_{b}$ and
$C_{p}$) by parsing the patch file. Firstly, we select removed and added lines
as the buggy and patched lines, marked with “+” and ‘-’, respectively.
Secondly, to keep the context information about the plausible patch, we keep
unchanged lines (i.e., without +” and ‘- in the beginning) as part of each
code snippet. Finally, the buggy (or patched) code snippet are made up by the
buggy (patched) lines and common context part.

We treat the buggy (or patched) code snippet as sequences of tokens and
utilize a subword tokenization method to address out-of-vocabulary (OOV)
problem by further breakdowning identifiers into their subtokens [30] when
tokenizing the code snippet. In this work, we keep the original tokenization
vocabulary instead of building a new vocabulary using byte pair encoding (BPE)
algorithm as we want APPT to inherit the natural language understanding
ability and start learning prediction from a good initial point.

After the buggy (or patched) code tokens are extracted, we attempt to take
them as the inputs into the token embedding phrase. However, pre-trained
models are usually limited to a particular input length. For example, BERT can
only take input sequences up to 512 tokens in length. We further truncate the
inputs whose length is longer than 512 after tokenization. Following existing
work [31], we use different methods to truncate the method pair.

  * •

head-only: keep the first 512 tokens in $C_{b}$ and $C_{p}$.

  * •

tail-only: keep the last 512 tokens in $C_{b}$ and $C_{p}$.

  * •

mid-only: select 512 tokens in the middle of in $C_{b}$ and $C_{p}$.

  * •

hybrid: select the first 256 and the last 256 tokens in $C_{b}$ and $C_{p}$.

In our experiment, we use the head-only method to truncate the code tokens by
default. We also discuss the impact of different truncation methods in Section
5.3.2. Finally, the buggy and patched code tokens (i.e., $T_{b}$ and $T_{p}$)
are extracted based on $C_{b}$ and $T_{p}$ to fit the maximum length limit of
BERT.

###  3.2 Token Embedding

Token Embedding phrase takes the buggy (or patched) code tokens (i.e., $T_{b}$
or $T_{p}$) as input and embeds it into the buggy (or patched) vector (i.e.,
$E_{b}$ or $E_{p}$) as output (shown in Fig. 2(b)). APPT implements a stack of
twelve layers of encoder blocks to extract the hidden state of the code
snippet. Each encoder block consists of three components. The first part is a
multi-head self-attention layer to learn long-range dependencies in the input
code tokens. The second part is a simple, position-wise fully connected feed-
forward neural network, which can linearly transform the token embedding for
better feature extraction. The third part is a residual connection around each
component, followed by a layer normalization to ensure the stability of code
token embeddings distribution.

In particular, the self-attention mechanism computes the representation of
each code token by considering the position relationship between the code
tokens. The self-attention mechanism mainly relies on three main vectors,
query $Q$, key $K$, and value $V$, by mapping a query and a set of key-value
pairs to an output vector. We employ a scaled dot-product self-attention to
calculate the attention scores of each token by taking the dot product between
all of the query vectors and key vectors. The attention scores are then
normalized to probabilities using the softmax function to get the attention
weights. Finally, the value vectors can be updated by taking a dot product
between the value vectors and the attention weight vectors. The self-attention
operation is computed using three matrices $Q$, $K$ and $V$ as follows:

$\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V$
(1)

To capture richer semantic meanings of the input code tokens, we further use a
multi-head mechanism to realize the self-attention, which allows the model to
jointly attend the information from different code representation subspaces at
different positions. For $d$-dimension $Q$, $K$, and $V$, we split those
vectors into $h$ heads where each head has $d/h$-dimension. After all of the
self-attention operations, each head will then be concatenated back again to
feed into a fully-connected feed-forward neural network including two linear
transformations with a ReLU activation in between. The multi-head mechanism
can be summarized by the following equation:

$\small\operatorname{MultiHead}(Q,K,V)=\operatorname{Concat}\left(\operatorname{head}_{1},\ldots,\operatorname{head}_{h}\right)W^{O}$
(2)

where $head_{i}=Attention(QW^{Q}_{i},KW^{Q}_{i},VW^{Q}_{i})$ and $W^{O}$ is
used to linearly project to the expected dimension after concatenation.
Therefore, the encoder stack can take an input code snippet and output a real-
valued vector for each code token within the code snippet based on the
context.

###  3.3 Patch Classification

After the embedding vectors of the buggy and patched code snippets (i.e.,
$E_{b}$ and $E_{p}$) are extracted by the encoder stack, patch classification
phrase first integrates the two vectors into a single input vector (i.e.,
$E_{con}$) and then adopts a deep learning classifier to predict the patch
correctness automatically (shown in Fig. 2(c)).

####  3.3.1 Representations Integration

Given two vectors $E_{b}$ and $E_{p}$ with $n$ dimensions representing the
buggy and patched code snippets, respectively, we integrate the two vectors
into one code changed vector for patch classification. In detail, we leverage
different approaches to integrate them to characterize the differences between
$E_{b}$ and $E_{p}$ from diverse aspects, such as an vector-wise concatenation
operation $E_{con}$, element-wise addition operation $E_{add}$ , element-wise
subtraction operation $E_{sub}$, Hadamard product $E_{pro}$. We also attempt
to capture crossed features between the two vectors by concatenating the above
integrated vectors $E_{mix}$. The integration approaches are selected due to
their promising results in previous studies [32, 12], which are listed as
follows:

  1. (1)

$E_{con}$ is a concatenation operation between $E_{b}$ and $E_{p}$ on vector-
wise level with $2n$ dimension (i.e., $E_{con}=E_{b}\bigoplus E_{p}$).

  2. (2)

$E_{add}$ is an addition operation between $E_{b}$ and $E_{p}$ on element-wise
level with $n$ dimensions (i.e., $E_{add}=E_{b}+E_{p}$).

  3. (3)

$E_{sub}$ is a subtraction operation between $E_{b}$ and $E_{p}$ on element-
wise level with $n$ dimensions (i.e., $E_{sub}=E_{b}-E_{p}$).

  4. (4)

$E_{pro}$ is a Hadamard product operation between $E_{b}$ and $E_{p}$ on
element-wise level with $n$ dimensions (i.e., $E_{sub}=E_{b}\odot E_{p}$).

  5. (5)

$E_{mix}$ is a concatenation over $E_{con}$, $E_{add}$, $E_{sub}$ and
$E_{pro}$ on vector-wise level with $5n$ dimension (i.e.,
$E_{mix}=E_{con}\bigoplus E_{add}\bigoplus E_{sub}\bigoplus E_{sub}$).

####  3.3.2 LSTM Stack

After the embedding vector (e.g., $E_{con}$) of the changed code tokens is
extracted, APPT aims to determine the given patch’s correctness based on a
deep learning classifier. To extract more hidden code change features, we
further feed the code changed vector into a Long Short-Term Memory (LSTM)
stack. The LSTM stack has two bidirectional LSTM layers, the output of which
is a new state generated by concatenating the hidden states from both
directions at a time. LSTM is a specialized recurrent neural network (RNN) for
modeling long-term dependencies of sequences. A common LSTM gate unit is
composed of a cell, an input gate, an output gate and a forget gate. Thanks to
the gated mechanism, LSTM is well-suited to extract the contextual semantic
features containing token sequential dependencies and has been widely used in
various kinds of tasks, such as vulnerability detection [33], fault
localization [34], and automated program repair [35].

In APPT, the LSTM stack computes a mapping from an input code changed vector
$x=(x_{1},...,x_{T})$ (e.g., $E_{con}$) to an output vector
$z=(z_{1},...,z_{T})$ by calculating the network gate unit activations. We
implement the gated mechanism by leveraging the input gates and forget gates
to control the propagation of cell states. Specifically, when updating the
cell state, the input gates decide what new information from the current input
to be included in the cell states (i.e., Equation 3), and forget gates decide
what information to be excluded from the cell states (i.e., Equation 4). Based
on new and forgetting information, cell states as the memory of the LSTM unit
can be updated (i.e., Equation 5). The output gate then determines the value
for the next hidden state by point-wise multiplication of the output gate
(i.e., Equation 6). Finally, the value of the current cell state passed
through tanh function (i.e., Equation 7), by which the output of LSTM stack is
calculated (i.e., Equation 8).

$i_{t}=\operatorname{sigmoid}\left(W_{ix}x_{t}+W_{ih}h_{t-1}+b_{i}\right)\\\ $
(3)

$f_{t}=\operatorname{sigmoid}\left(W_{fx}x_{t}+W_{fh}h_{t-1}+b_{f}\right)\\\ $
(4)

$c_{t}=f_{t}\odot
c_{t-1}+i_{t}\odot\tanh\left(W_{gx}x_{t}+W_{gh}h_{t-1}+b_{g}\right)\\\ $ (5)

$o_{t}=\operatorname{sigmoid}\left(W_{ox}x_{t}+W_{oh}h_{t-1}+b_{o}\right)\\\ $
(6)

$h_{t}=o_{t}\odot\tanh\left(c_{t}\right)$ (7)

$z_{t}=W_{zh}h_{t}+b_{z}\\\ $ (8)

where the $W$ terms denote weight matrices (e.g., $W_{ix}$ is the matrix of
weights from the input gate to the input), the $b$ terms denote bias vectors
(e.g., $b_{i}$ is the input gate bias vector) and $\odot$ denotes element-wise
multiplication of the vectors.

####  3.3.3 Classifier

After the computation of all LSTM iterations, the embedding vectors of changed
code tokens are further fed to a designed deep learning classifier to predict
the patch correctness. The classifier is composed of two fully connected
layers followed by a binary predictor. In APPT, we apply a standard softmax
function to obtain the probability distribution over correctness. A patch is
labeled as correct if its probability of being correct is larger than that of
being incorrect; otherwise, it is considered overfitting.

In particular, for patch $p$, $z$ denotes its output of the last iteration in
the LSTM stack, which is further linearly transformed into a real number as
Equation 9, where $W\in\mathbb{R}^{d\times 1}$, $b\in\mathbb{R}$, and $n$
denotes the number of class (i.e., correct and overfitting). We then leverage
softmax function to normalize the output of patch $p$ as Equation 10, where
$s$ denotes the correct or overfitting probability of patch $p$ predicted by
the model .

$y_{i}=Wz_{i}+b_{i}\quad\forall\mathrm{i}\in 1\ldots\mathrm{n}$ (9)

$s\left(y_{i}\right)=\frac{\exp\left\\{y_{i}\right\\}}{\sum_{i=1}^{n}\exp\left\\{y_{j}\right\\}}$
(10)

###  3.4 Training

To train the network, we calculate the loss to update the neural weights based
on its predicted result and ground truth. We use the cross-entropy loss, which
has been widely used in some classification tasks and patch prediction studies
[20, 36]. In particular, $g_{i}\in\\{0,1\\}$ denotes whether the $i$-th patch
is correct or overfitting. The cross-entropy loss compares a target $g_{i}$
with a prediction $s$ in a logarithmic and hence exponential fashion. The
objective function is computed in Equation 11, which is minimized constantly
in the training to update the parameters in our model.

$L=\sum_{i}-[g_{i}\cdot\log(s)+(1-g_{i})\cdot\log(1-s)]$ (11)

We employ the dropout technique to improve the robustness of APPT and the Adam
approach [37] to optimize the objective function.

##  4 Experiment

###  4.1 Research Questions

The empirical study is conducted to answer the following research questions.

RQ1:

How does APPT perform compared with existing state-of-the-art representation
learning-based APCA techniques?

RQ2:

How does APPT perform compared with existing state-of-the-art traditional and
learning-based APCA techniques?

RQ3:

To what extent do the different choices affect the overall effectiveness of
APPT?

RQ3.1:

To what extent do the token truncation choices affect the overall
effectiveness of APPT?

RQ3.2:

To what extent do the vector concatenation choices affect the overall
effectiveness of APPT?

RQ3.3:

To what extent do the pre-trained model choices affect the overall
effectiveness of APPT?

RQ1 aims to compare APPT with 16 representation learning techniques to explore
to what extent APPT outperforms these techniques, including three classifiers
multiplied (decision tree, logistic regression, and naive Bayes) by five
representation methods (BERT, code2vec, code2seq, Doc2Vec, and CC2Vec) from
Tian et al. [12], and the most recent technique CACHE from Lin et al. [20].
RQ2 is designed to investigate the effectiveness of APPT by comparing it with
both dynamic and static techniques. The latest learning-based APCA technique,
ODS, is also evaluated in our study. RQ3 focuses on impact analysis of APPT,
which is further refined into three sub-RQs. In detail, RQ3.1 explores how the
four token truncation choices affect the effectiveness of APPT. RQ3.2 explores
how the five vector concatenation methods affect the effectiveness of APPT.
RQ3.3 replaces BERT with advanced CodeBERT and GraphCodeBERT to investigate
the impact of the pre-trained models on the effectiveness of APPT.

###  4.2 Dataset

With the rapid development of APR research in the last decades, a broad range
of repair techniques has been proposed [38, 39, 40], resulting in a growing
number of patches across many benchmarks being released [7, 13]. The large-
scale patch benchmarks enable deep learning-based prediction techniques to
learn the distribution of correct and overfitting patches for patch
correctness assessment. In this study, we adopt two patch datasets based on
the recent studies [12, 13, 20], a small one containing 1,183 Defects4J
labeled patches and a large one containing 50,794 real-world labeled patches.

TABLE I: APR tools in small benchmark Category | APR Tools
---|---
Heuristic-based | jGenProg [41], jKali [41], jMutRepair [41], SimFix [42], ARJA [38], GenProg-A [38], Kali-A [38], RSRepair-A [38], CapGen [43].
Constraint-based | DynaMoth [44], Nopol [45], ACS [46], Cardumen [47], JAID [48], SketchFix [49].
Template-based | kPAR [50], FixMiner [51], AVATAR [52], TBar [5], SOFix [53], HDRepair [54].
Learning-based | SequenceR [55].

On the small dataset, we mainly focus on the released patches from Defects4J
[56], which is the most widely-adopted benchmark in APR research [7]. We
select the benchmarks released by two recent large-scale studies, i.e., Wang
et al. [13] and Tian et al. [12]. Specifically, the first benchmark [13]
includes the labeled patches provided by Liu et al. [7], Xiong et al. [17] and
Defects4J developers [56]. The second benchmark [12] includes the labeled
patches from Liu et al. [7] and also considers the patches generated by some
well-known APR tools that are not included in Liu et al. [7] to better explore
the overfitting problem, i.e., JAID [48], SketchFix [49], CapGen [43], SOFix
[53] and SequenceR [55]. To avoid the data leakage issue in the two
benchmarks, a filtering process is also conducted to discard duplicate
patches. In particular, given a patch whose all the blank spaces are removed,
the left text information is compared with that from the other patches. If two
patches are identical concerning their text information, they are considered
duplicates, resulting in 1,183 patches in our small dataset. The patches are
generated by 22 distinct APR tools, which can be divided into four categories,
i.e., heuristic-based, constraint-based, template-based, and learning-based
techniques. The detailed information on these covered APR tools is presented
in Table I, where the first column lists the four repair technique categories
and the second column list the corresponding repair techniques.

TABLE II: Datasets used in our experiment Datasets | Subjects | # Correct | # Overfitting | Total
---|---|---|---|---
Small | Tian et al. [12] | 468 | 532 | 1,000
Wang et al. [13] | 248 | 654 | 902
Our Study | 532 | 648 | 1,183
Large | ManySStuBs4J [57] | 51,433 | 0 | 51,433
RepairThemAll [6] | 900 | 63,393 | 64,293
Our Study | 25,589 | 24,105 | 49,694

On the large dataset, we further consider a variety of patches generated from
other benchmarks, to evaluate the generality of APPT. Recently, existing
studies demonstrate that APR techniques may overfit Defects4J in terms of
repairability [6, 11]. Thus, some other benchmarks have been conducted to
evaluate the performance of APR techniques, such as Bugs.jar [58],
IntroclassJava [59], BEARS [60] and QuixBugs [61], providing substantial
patches on the large dataset. In this work, we consider a large patch dataset
released by a recent study [20] to investigate the generality of APPT. The
large patch dataset includes the labeled patches provided from RepairThemAll
framework [6] and ManySStuBs4J [57]. In particular, RepairThemAll framework
[6] contains 64,293 patches using 11 Java test-suite-based repair tools and
2,141 bugs from five diverse benchmarks. However, there exists an imbalanced
dataset issue as over 98.6%222The RepairThemAll Framework.
https://github.com/program-repair/RepairThemAll, accessed August 2022
(63,393/64,293) of generated patches are actually labeled as incorrect. Recent
studies have revealed that a well-balanced dataset is essential when
investigating deep learning-based prediction techniques [12, 18]. To
compensate the lack of correct patches, the large patch dataset then includes
ManySStuBs4J [57], which provides simple bug-fix changes mined from 1,000
popular open-source Java projects. The bug-fix changes are correct fix
attempts of real-world bugs and thus are considered correct patches in our
experiment. Finally, a large balanced patch dataset is built from the
RepairThemAll framework and ManySStuBs4J by discarding duplicate patches and
filtering the ones from small student-written programming assignments (e.g.,
IntroClassJava). The dataset involves all available patches generated on
RepairThemAll framework and ManySStuBs4J, resulting in 49,694 patches after
deduplication.

Statistics on the two datasets are shown in Table II. Table II has two main
rows representing the two datasets, each of which has three sub-rows. The
first and second sub-rows list the two sources in the corresponding dataset.
The third column lists the filtered patches used in our experiment from the
two sources. We also present the number of correct, overfitting and total
patches in the last three columns.

###  4.3 Baselines

TABLE III: Compared APCA techniques in our experiment. | with Oracle Required | without Oracle Required
---|---|---
Dynamic-based | Evosuite [15], Randoop [16], DiffTGen [62], Daikon [63] | PATCH-SIM [17], E-PATCH-SIM [17], R-Opad [63], E-Opad [63]
Static-based | $\bigotimes$ | ssFix [64], CapGen [43], Anti-patterns [14], S3 [65]
Learning-based | $\bigotimes$ | ODS [18], Random Forest [12],
Embedding learning [12], CACHE [20], Our proposed APPT
denotes the representation learning techniques.

Various APCA techniques have been proposed in the literature to validate patch
correctness. Following existing studies [20, 17], we attempt to select state-
of-the-art techniques designed for Java language as Java is the most targeted
language in APR community [7] and the existing patches of real-world bugs are
usually available in Java language [12]. We first consider the recent
empirical study by Wang et al. [13] to identify existing APCA techniques. We
then select recent advanced studies [20, 12] that are not included in Wang et
al. [13].

In general, following existing work [20, 13], the existing APCA techniques can
be categorized into static, dynamic and learning-based APCA techniques
according to whether test execution is needed or deep learning techniques are
adopted (mentioned in Section 2). Meanwhile, according to whether the ground-
truth patch is required, they can be further categorized into two categories
(i.e., with or without oracle). Particularly, similar to our proposed method
APPT, CHCHE and embedding learning techniques adopt representation models to
embed changed code and a deep learning classier to predict patch correctness.
Such techniques can be further considered as representation learning APCA
techniques.

The details of the selected APCA techniques are illustrated in Table
LABEL:tab:baselines. The first column lists three APCA categories. The second
and third columns list whether the oracle information is equipped. We also
list the representation learning techniques (e.g., APPT) in the light gray
box. We summarize the selected techniques as follows.

####  4.3.1 Dynamic-based APCA Techniques

Dynamic-based techniques are designed to distinguish correct patches from
overfitting patches based on the outcome or the execution traces of the
original or generated test cases.

Simple Test Generation. The overfitting issue is prevalent in the repair
process due to the weak adequacy of existing test cases. Thus, researchers use
test case generation tools to generate extra test cases based on the fixed
program and check whether or not the generated patches that pass the original
test cases can pass the extra test cases [66, 23]. In this work, we adopt
Evosuite [15] and Randoop [16] as the test case generation tools, as they have
been widely investigated in previous studies.

DiffTGen. Xin et al. [62] identify overfitting patches by executing test cases
generated by an external test generator (i.e., Evosuite). Different from
simple test generation generating test cases randomly, DiffTGen generates test
cases to uncover the syntactic differences between the patched and buggy
program. A plausible patch is regarded as overfitting if the output of the
patched program is not the same as that of the correct program. DiffTGen needs
a human-written patch as a reference and requires providing human-amenable
testing information for the developers to provide oracles the generated test
cases.

Daikon. Daikon is a dynamic-based technique based on the program invariant
with oracle information. Yang et al. [63] adopt the program invariant to
explore the differences between an overfitting and a correct patch. A patch is
considered correct if its inferred invariant is identical to that of the
ground-truth. If there exists a different comparison, the patch is considered
overfitting.

PATCH-SIM. Xiong et al. [17] consider the execution traces of the passing
tests on the buggy and patched programs are likely to be similar, while the
execution traces of failing tests on the buggy and patched programs are likely
different. Based on the concept, they approximate the correctness of a patch
based on the execution trace without the oracle information. PATCH-SIM adopts
Randoop to generate additional test cases to collect dynamic execution
information. In this work, we also replace Randoop with Evosuite to
comprehensively explore the impact of test generation techniques (denoted as
E-PATCH-SIM).

Opad. Yang et al. [67] adopt fuzzing testing to generate new test cases and
employ two test oracles (crash and memory-safety) to enhance the validity
checking of patches. The original implementation of Opad is not designed for
Java language and uses American Fuzz Lop (AFL) as the fuzzing technique. In
this work, following recent studies [13, 20], we replace AFL with Randoop and
Evosuite to generate new test cases on the Java programs and denote them as
R-Opad and E-Opad, respectively.

####  4.3.2 Static-based APCA Techniques

Static-based techniques usually adopt static analysis tools to extract some
designed static features and then check patch correctness based on such
features.

ssFix. ssFix [64] is a static-based technique that utilizes token-based syntax
representation to generate patches with a higher probability of correctness.
ssFix first performs a syntactic code search to find code snippets from a
codebase that is syntax-related to the context of a bug to generate correct
patches, and then prioritizes the patches based on the modification types and
the modification sizes.

CapGen. Wen et al. [43] propose three aspects of context information (i.e.,
genealogy contexts, variable contexts and dependency contexts) embedded in an
AST node and its surrounding codes to prioritize correct patches over
overfitting ones. In this work, following recent studies [13, 20], we extract
the three context information as static features to investigate patch
correctness assessment.

Anti-patterns. Tan et al. [14] define a set of rules that essentially capture
disallowed modifications to the buggy program, and a patch is overfitting if
it falls into the rules. A recent study [13] has shown that the manually-
defined anti-patterns may have false positives for correct patches, resulting
in destructive effects in patch correctness prediction.

S3. Le et al. [65] assume that a correct patch is often syntactically and
semantically close to a buggy code snippet. Thus, they adopt six syntactic
features (i.e., AST differencing, cosine similarity and locality of variables
and constants) and semantic features (i.e., model counting, output coverage
and anti-patterns) to measure the distance between a candidate patch and the
buggy code snippet.

####  4.3.3 Learning-based APCA Techniques

Learning-based techniques can predict whether a plausible patch is correct or
not based on machine learning techniques.

ODS. Ye et al. [18] first extract 202 code features at the abstract syntax
tree level and then use supervised learning to learn a probabilistic model
automatically. The results show that ODS can achieve better prediction
performance than the dynamic-based technique PATCH-SIM with a faster speed.

CACHE. Lin et al. [20] propose a context-aware APCA technique CACHE by taking
both the changed code snippet and the correlated unchanged code snippet into
consideration. CACHE first parses the patched code snippet into AST
representation and then utilizes the AST path technique to capture the
structure information.

Random Forest. Wang et al. [13] investigate the effectiveness of adopting deep
learning models to predict patch correctness based on eight static features
(two from ssFix, three from S3, and three from CapGen). To integrate the
static features, six widely-used classification models (including Random
Forest, Decision Table, J48, Naive Bayes, Logistic Regression, and SMO) are
adopted. The results demonstrate that Random Forest can achieve both superior
precision and recall performance. In this work, following existing work [20],
we also adopt Random Forest to predict the patch correctness based on the
integrated static features.

Embedding Learning. Tian et al. [12] propose to leverage representation
learning techniques to produce embedding for buggy and patched code snippets
and then adopt supervised learning classifies to predict patch correctness. In
particular, nine representation learning APCA techniques are evaluated,
involving three embedding techniques (i.e., CC2vec, BERT and Doc2Vec) and
three classifiers (logistic regression, decision tree and naive bayes).

###  4.4 Model Selection

To the best of our knowledge, APPT is the first automated patch correctness
prediction technique by fine-tuning the existing pre-trained model. In this
paper, we adopt BERT as the encoder stack due to its powerful performance in
previous work [24].

Specifically, BERT is pre-trained on large amounts of text data with two self-
supervised goals, i.e., masked language modeling (MLM) and next sentence
prediction (NSP). MLM aims to let the model predict the masked words by
masking 15% of words in each sentence randomly. NSP aims to further improve
the model’s ability to understand the relationship between two sentences by
letting the model predict whether the given sentence pair is continuous. The
model then can be fine-tuned to adapt to some specific downstream tasks and
has achieved remarkable state-of-the-art results on a variety of natural
language processing tasks, such as question answering and language inference.

There exist two model architectures at different sizes, i.e., BERTbase and
BERTlarge [24]. The former has 12 layers and 12 attention heads, and the
embedding size is 768, while the latter has a double layer number and 16
attention heads, and the embedding size is changed to 1024. In this paper, we
do not modify the vocabulary size and use the pre-trained BERTbase as the
fine-tuning starting point instead of starting from scratch.

In this paper, APPT is conceptually and practically generalizable to various
pre-trained models. We also select CodeBERT and GraphCodeBERT as the encoder
stack to evaluate the scalability of APPT. CodeBERT and GraphCodeBERT share
the same model architecture as BERT, while utilizing paired natural language
and programming language to pre-train the model to support code-related tasks
(mentioned in Section 8.2.2).

###  4.5 Evaluation Metrics

We evaluate the prediction performance of various APCA approaches by accuracy,
precision, recall, F1-score and AUC metrics, which have been widely adopted in
patch correctness assessment research and other classification tasks [20, 12].
Given the number of true positives (TPs, a TP refers to an overfitting patch
that is identified as overfitting), false positives (FPs, a FP refers to a
correct patch that is identified as overfitting), false negatives (FNs, a FN
refers to an overfitting patch is identified as correct) and true negatives
(TNs, a TN refers to a correct patch that is identified as correct), the
metrics are defined as follows:

_$\bullet$ Accuracy:_ the proportion of correctly reported (whether the patch
is correct or not) patches. Accuracy measures the probability that the
prediction of APCA techniques is correct.

$Accuracy=\frac{TP+TN}{TP+FP+FN+TN}\\\ $ (12)

_$\bullet$ Precision:_ the proportion of real overfitting patches over the
reported overfitting patches. Precision measures how much we can trust the
APCA techniques when it predicts a patch as overfitting.

$Precision=\frac{TP}{TP+FP}\\\ $ (13)

_$\bullet$ Recall:_ the proportion of reported overfitting patches over all
the real overfitting patches. Recall measures the ability of the APCA
techniques to find all the overfitting patches in the dataset.

$Recall=\frac{TP}{TP+FN}$ (14)

_$\bullet$ F1-score:_ twice the multiplication of precision and recall divided
by the sum of them. F1-score measures the trade-off between precision and
recall by taking their harmonic mean.

$F1\text{-}score=2*\frac{Precision*Recall}{Precision+Recall}$ (15)

_$\bullet$ AUC:_ the entire two-dimensional area underneath the entire
receiver operating characteristic curve. AUC measures the probability that the
classifier will rank a randomly chosen overfitting patch higher than that of a
randomly chosen correct patch. The higher the AUC, the better the APCA
techniques is at predicting real overfitting patches as overfitting and real
correct patches as correct.

$\begin{gathered}AUC=\frac{\sum I\left(P_{\text{overfitting
}},P_{\text{correct }}\right)}{M\times N}\\\
I\left(P_{\text{overfitting}},P_{\text{correct}}\right)=\left\\{\begin{array}[]{l}1,P_{\text{overfitting}}>P_{\text{correct}}\\\
0.5,P_{\text{overfitting}}=P_{\text{correct}}\\\
0,P_{\text{overfitting}}<P_{\text{correct}}\end{array}\right.\end{gathered}$
(16)

where $M$ and $N$ denote the number of overfitting and correct patches, while
$P_{\text{overfitting}}$ and $P_{\text{correct}}$ denote the prediction
probability for the overfitting and correct patches.

###  4.6 Implementation Details

All of our approaches are built based on PyTorch framework333PyTorch.
https://pytorch.org/, accessed August 2022. We use the Hugging Face444Hugging
Face. https://huggingface.co/, accessed August 2022 implementation version of
BERT in our work. Considering previous work recommendation [26, 40], we
utilize “bert-base-uncased” (refer to BERTbase) as the initial point, as the
base version is quite lightweight to employ in practice with comparable
effectiveness compared against the large version. There exist $12$ layers of
transformer blocks and $12$ self-attention heads in the “bert-base-uncased”
model. The optimizer is Adam [37] with $5e-5$ learning rate. The batch size is
$16$ and dropout rate is $0.5$. We train for most $50$ epochs and the max
length of the input is set to $512$ due to model limitation.

All the training and evaluation of our methods are conducted on one Ubuntu
18.04.3 server with two Tesla V100-SXM2 GPUs.

##  5 Results and Analysis

###  5.1 RQ1: Comparing with Representation Learning-based APCA Techniques

TABLE IV: Effectiveness of APPT compared with representation learning-based APCA techniques on the small dataset Classifier | Embedding | Accuracy | Precision | Recall | F1-score | AUC
---|---|---|---|---|---|---
Decision Tree | BERT | 63.5% | 65.3% | 70.9% | 67.9% | 63.7%
CC2vec | 66.1% | 69.4% | 68.0% | 68.7% | 66.5%
code2vec | 65.1% | 68.1% | 68.3% | 68.1% | 64.4%
code2seq | 60.1% | 63.5% | 64.0% | 63.7% | 60.0%
Doc2Vec | 61.2% | 64.5% | 65.3% | 64.8% | 60.8%
Logistic Regression | BERT | 64.8% | 66.5% | 72.4% | 69.2% | 68.7%
CC2vec | 64.9% | 62.4% | 90.1% | 73.7% | 68.6%
code2vec | 66.8% | 68.6% | 72.9% | 70.6% | 70.2%
code2seq | 60.7% | 63.3% | 67.6% | 65.3% | 63.1%
Doc2Vec | 63.7% | 65.7% | 70.8% | 68.0% | 68.9%
Naïve Bayes | BERT | 61.6% | 64.8% | 65.7% | 65.0% | 64.7%
CC2vec | 60.0% | 58.3% | 94.6% | 72.2% | 58.1%
code2vec | 57.7% | 58.1% | 81.5% | 67.8% | 55.6%
code2seq | 57.0% | 59.0% | 70.5% | 64.2% | 60.6%
Doc2Vec | 64.1% | 65.8% | 72.4% | 68.7% | 67.0%
CACHE | 75.4% | 79.5% | 76.5% | 78.0% | 80.3%
APPT | 79.0% | 80.7% | 81.3% | 80.9% | 83.4%

####  5.1.1 Experimental Design

As discussed in Section 4.3, APPT, CACHE and embedding learning techniques
(i.e., techniques within the light gray box in Table LABEL:tab:baselines) can
be categorized as representation learning APCA techniques. In this section, we
aim to explore the performance of APPT when compared with these representation
learning techniques. In particular, embedding learning techniques [12] mainly
adopt embedding models (i.e., BERT, Doc2Vec, and CC2Vec) to embed buggy and
patched code fragments, and then train classification models (i.e., Decision
Tree, Logistic Regression, and Naive Bayes) to predict patch correctness.
Following previous study [20], we also consider two additional embedding
models (i.e., code2vec and code2seq) in the experiment. Meanwhile, CACHE can
also be considered as a representation learning technique, which incorporates
the context information in embedding code changes, and trains a deep learning
classifier to predict the patch correctness.

In total, 16 representation learning techniques are considered in our
experiment, involving five embedding techniques multiplied by three
classification models, and one context-aware representation learning technique
CACHE. Following the previous study [12], we perform a 5-fold cross-validation
on both the small and large datasets for comparison.

####  5.1.2 Results

Comparison results against the existing representation learning techniques are
presented in Table IV to Table V for the both small and large dataset. The
first column lists the three classifiers and the second column lists the five
embedding approaches. The remaining columns list the detailed values of
accuracy, precision, recall, F1-score and AUC metrics, respectively. We
present the most recent representation learning work CACHE and our APPT in the
bottom part of Table IV and Table V. It can be observed that APPT achieves the
best performance under each experimental setting.

On the small dataset, APPT is around 3.6%, 1.2%, 4.8%, 2.9% and 3.1% higher
than the state-of-the-art technique CACHE in terms of all metrics (i.e., 79.0%
vs. 75.4% for accuracy, 80.7% vs. 79.5% for precision, 81.3% vs. 76.5% for
recall, 80.9% vs. 78.0% for F1-score, and 83.4% vs. 80.3% for AUC). Compared
with all representation learning techniques, APPT achieves the best
performance in terms of accuracy, precision, F1-score and AUC metrics. In
particular, the values of APPT on the accuracy and precision metrics are 79.0%
and 80.7%, respectively, while the optimal values of all other techniques are
75.4% and 79.5%. This suggests that APPT can generally achieve the most
accurate predictions, and the patches identified as overfitting by APPT are of
high confidence to be overfitting. Regarding recall, the values of CC2vec and
code2vec can sometimes exceed those of APPT since they tend to classify most
patches as overfitting (e.g., CC2vec with Naive Bayes classifies 1,051 out of
1,183 patches as overfitting and thus achieves a high recall of 94.6%).
However, these techniques achieve relatively low precision (e.g., CC2vec with
Naive Bayes classifier has only 72.2% for recall). On the contrary, APPT can
achieve a high recall exceeding 81% while maintaining a high precision of
79.5%.

TABLE V: Effectiveness of APPT compared with representation learning techniques on the large dataset Classifier | Embedding | Accuracy | Precision | Recall | F1-score | AUC
---|---|---|---|---|---|---
Decision Tree | BERT | 95.7% | 93.9% | 97.4% | 95.6% | 95.9%
CC2vec | 95.6% | 95.4% | 95.7% | 95.5% | 95.7%
code2vec | 95.0% | 93.2% | 96.6% | 94.9% | 95.4%
code2seq | 92.2% | 91.0% | 93.2% | 92.3% | 92.4%
Doc2Vec | 85.1% | 84.2% | 85.3% | 84.7% | 85.3%
Logistic Regression | BERT | 82.4% | 83.6% | 79.4% | 81.4% | 91.0%
CC2vec | 91.2% | 96.1% | 85.4% | 90.4% | 95.0%
code2vec | 89.6% | 88.6% | 90.2% | 89.4% | 95.0%
code2seq | 91.5% | 90.5% | 92.2% | 91.4% | 96.0%
Doc2Vec | 90.4% | 91.9% | 88.0% | 89.9% | 96.1%
Naïve Bayes | BERT | 68.2% | 80.3% | 45.7% | 58.2% | 74.6%
CC2vec | 78.4% | 94.8% | 58.6% | 72.5% | 92.4%
code2vec | 61.4% | 68.7% | 37.4% | 48.4% | 69.3%
code2seq | 70.3% | 76.8% | 55.5% | 64.5% | 78.9%
Doc2Vec | 81.2% | 86.4% | 75.5% | 78.9% | 88.9%
CACHE | 98.6% | 98.9% | 98.2% | 98.6% | 98.9%
APPT | 99.1% | 99.1% | 99.1% | 99.1% | 99.9%

On the large dataset, we can find APPT achieves over 99% for the five metrics,
outperforming all existing approaches. For example, APPT reaches 99.9% in
terms of AUC, which is 1.0% higher than the second highest value obtained from
the most recent technique CACHE (i.e., 98.9%). This suggests that APPT is more
capable of distinguishing correct and overfitting patches than CACHE. Besides,
the improvement against CACHE for accuracy, precision, recall and F1-score
metrics achieves 0.5%, 0.3%, 0.9% and 0.5%, respectively. We also find that
the performance achieved on the large dataset is commonly higher than that
achieved on the small dataset. For example, the average value among the five
metrics increases from 81.06% to 99.26%, resulting in a 22.5% improvement
rate. Based on our analysis on the two datasets, the possible reason for this
improvement is that bugs on the large dataset are usually simple. We observe
that all ManySStuBs4J patches on the large dataset are single-line operations,
while patches on the small dataset usually cross multiple lines (e.g., more
than 40% of Defects4J developer patches are multiple line patches [20]). It is
easy for the neural networks to learn the correctness distribution of such
simple code changes. Meanwhile, the difference in patch scale between the two
datasets may be the second reason. We find there exist 49,694 patches on the
large dataset, which is 42 times larger than that of the small dataset. The
amount of training data is often the single most dominant factor that
determines the performance of the neural networks [68]. More available patches
benefit the neural networks to learn diverse code changes better.

Answer to RQ1: Overall, our analysis on representation learning techniques
reveals that (1) APPT can outperform a state-of-the-art representation
learning technique CACHE under all metrics and datasets. (2) on the small
dataset, APPT achieves 79.0% for accuracy and 83.4% for AUC, which surpass
CACHE by 3.6% and 3.1%. (3) on the large dataset, APPT exceeds 99% on all
metrics, yet none of existing representation learning techniques achieves
that.

###  5.2 RQ2: Comparing with Traditional and Learning-based APCA Techniques

####  5.2.1 Experimental Design

In this section, we aim to further compare the proposed method APPT with the
existing APCA techniques. We select the remaining techniques mentioned in
Section 4.3 (except representation learning techniques discussed in RQ1). In
total, 14 APCA techniques are considered in the experiment, involving four
static techniques (Anti-patterns, ssFix, CapGen and S3), eight dynamic
techniques (Evosuite, Randoop, DiffTGen, Daikon, R-Opad, E-Opad, PATCH-SIM and
E-PATCH-SIM) and two learning techniques (Random Forest and ODS).

As it is time-consuming to run all the techniques (especially for dynamic and
learning ones), following the existing work [20], we reuse the released
results from the recent work [13, 18, 20]. We collect the detailed results of
all selected APCA techniques from Lin et al. [20], which are concluded based
on 902 patches (i.e., Wang et al. [13] in Table II) and a 10-fold cross-
validation. To fairly compare with all the state-of-the-art techniques, we
perform our experiment in the same experimental setting.

####  5.2.2 Results

TABLE VI: Effectiveness of APPT compared with the traditional and learning-based APCA technique Category | APCA | Accuracy | Precision | Recall. | F1-score
---|---|---|---|---|---
Dynamic-based |  w-oracle | Evosuite | 65.9% | 99.1% | 53.5% | 69.5%
Randoop | 51.3% | 97.4% | 33.8% | 50.2%
DiffTGen | 49.6% | 97.4% | 30.6% | 46.6%
Daikon | 76.1% | 89.9% | 73.7% | 81.0%
wo-oracle | R-Opad | 34.9% | 100.0% | 10.2% | 18.5%
E-Opad | 37.7% | 100.0% | 14.7% | 25.6%
PATCH-SIM | 49.5% | 83.0% | 38.9% | 53.0%
E-PATCH-SIM | 41.7% | 82.1% | 25.8% | 39.3%
Static-based | Anti-patterns | 47.6% | 85.5% | 33.5% | 48.1%
S3 | 69.7% | 79.3% | 78.9% | 79.0%
ssFix | 69.2% | 78.9% | 78.8% | 78.8%
CapGen | 68.0% | 78.3% | 77.4% | 77.8%
Learning-based | Random Forest | 72.5% | 87.0% | 89.1% | 88.0%
ODS | 88.9% | 90.4% | 94.8% | 92.5%
APPT | 90.4% | 91.5% | 96.0% | 93.6%

The experiment results are listed in Table VI. The first two columns list the
selected techniques and their corresponding categories. The remaining columns
list the detailed values of accuracy, precision, recall and F1-score metrics.

Compared with traditional dynamic-based and static-based APCA techniques, we
can find that APPT reaches 90.4%, 96.0% and 93.6% in terms of accuracy, recall
and F1-score, respectively. Specifically, APPT achieves the best overall
performance with the three metrics, and none of the previous techniques
exceeds 90%. As for precision, more than 91% of patches reported by APPT are
indeed overfitting patches, which is better than all static-based techniques
and three dynamic-based techniques (i.e., Daikon, PATCH-SIM, and E-PATCH-SIM).
Although some dynamic ones have higher precision values, it is time-consuming
to generate additional test cases and collect run-time information. More
importantly, the recall of these techniques is usually low (e.g., 10.3% for
R-Opad), or the ground-truth oracle is needed (e.g., Evosuite and Randoop
techniques), limiting the application of such techniques in practice.

Compared with learning-based techniques, we find that APPT still performs
better than a state-of-the-art technique ODS with respect to all four metrics
(90.4% vs. 88.9% for accuracy, 91.5% vs. 90.4% for precision, 96.0% vs. 94.8%
for recall, 93.6% vs. 92.5% for F1-score, respectively). Overall, the
improvement against Random Forest and ODS reaches 4.5%$\sim$17.9% and
1.1%$\sim$1.5%. Considering that it is expensive for ODS to extract hundreds
of manually-designed code features at AST level, our approach simply adopting
the pre-trained model to encode a sequence of tokens is even more promising.
We also highlight this direction of integrating code-aware features (e.g.,
code edits and AST representation) with pre-trained models for patch
correctness assessment.

Answer to RQ2: Overall, our comparison results reveal that, (1) APPT can
achieve remarkable performance compared to exiting static-based techniques
with a high recall reaching 96.0%. (2) APPT can achieve higher precision than
a state-of-the-art dynamic-based technique PATCH-SIM by 8.5%. (3) compared
with existing learning-based techniques, APPT can achieve the best performance
among all metrics.

###  5.3 RQ3: The Impact Analysis

####  5.3.1 Experimental Design

To further explore how different fine-tuning choices affect the prediction
performance of pre-trained models, we first consider and replace the head-only
token truncation with other truncation methods, such as hybrid, mid-only and
tail-only token truncation. We then adopt different methods to merge the buggy
method vector and patched method vector, such as concatenate, additional,
subtraction, and product operation. We also mix the above-mentioned merged
vectors as an additional concatenation method. Recently, following the BERT
model architecture, researchers use some code-related pre-trained tasks to
capture the semantic connection between natural language and programming
language, so as to further adapt these pre-training models for programming
language. Thus, we replace the BERT with two advanced models pre-trained with
the programming language, i.e., CodeBERT [28] and GraphCodeBERT [29].

TABLE VII: Effectiveness of APPT with different truncation choices. Dataset | Truncation | Accuracy | Precision | Recall | F1-score | AUC
---|---|---|---|---|---|---
small | APPThybrid | 79.04% | 80.67% | 81.34% | 80.92% | 83.43%
APPThead | 79.72% | 80.84% | 83.17% | 81.76% | 82.55%
APPTmid | 75.48% | 78.27% | 78.41% | 77.85% | 81.34%
APPTtail | 73.20% | 76.00% | 76.40% | 75.38% | 78.45%
large | APPThybrid | 99.13% | 99.09% | 99.13% | 99.11% | 99.86%
APPThead | 99.04% | 99.17% | 98.86% | 99.01% | 99.54%
APPTmid | 97.36% | 96.62% | 98.17% | 97.35% | 98.18%
APPTtail | 97.85% | 98.28% | 97.30% | 97.77% | 99.49%

####  5.3.2 RQ3.1 Results: The Impact of Token Truncation Choice

Table VII presents the prediction results under different truncation choices.
The first column lists the two datasets. The second column lists the four
truncation choices, i.e., head-only, mid-only, tail-only and hybrid. The
remaining columns list the detailed values of accuracy, precision, recall and
F1-score and AUC metrics.

On the small dataset, we can find that the head-only approach achieves the
optimum performance for accuracy (79.72%), precision (80.84%), recall (80.84%)
and F1-score (81.76%), while the hybrid approach achieves the optimum AUC
score (83.43%). The mid-only approach, considering the middle tokens in the
buggy and patched methods, achieves the third-best performance for all
metrics, followed by the tail-only approach. Similar performance can be
observed on the large dataset. For example, the head-only and hybrid
approaches have the best performance in all metrics, while the mid-only and
tail-only ones are the following. The results demonstrate that the head-only
approach extracting the beginning code tokens is effective in distinguishing
the buggy and patched code snippets for the pre-trained model.

####  5.3.3 RQ3.2 Results: The Impact of The Vector Concatenation Choice

TABLE VIII: Effectiveness of APPT with different concatenation choices. Dataset | Truncation | Accuracy | Precision | Recall | F1-score | AUC
---|---|---|---|---|---|---
small | APPTconcat | 79.04% | 80.67% | 81.34% | 80.92% | 83.43%
APPTaddition | 69.83% | 70.24% | 80.12% | 73.83% | 75.44%
APPTsubtraction | 71.38% | 72.42% | 77.27% | 74.72% | 75.59%
APPTproduct | 63.27% | 62.37% | 96.32% | 74.81% | 66.46%
APPTmix | 80.90% | 82.21% | 83.18% | 82.64% | 83.46
large | APPTconcat | 99.13% | 99.09% | 99.13% | 99.11% | 99.86%
APPTaddition | 98.96% | 98.80% | 99.07% | 98.93% | 99.81%
APPTsubtraction | 97.31% | 99.14% | 95.29% | 97.17% | 99.46%
APPTproduct | 98.82% | 98.88% | 98.69% | 98.78% | 99.78%
APPTmix | 99.10% | 98.99% | 99.17% | 99.08% | 99.79%

Table VIII presents the prediction results under different concatenation
choices. The first column lists the two datasets. The second column lists the
five concatenation choices, i.e., concat, addition, subtraction, product and
mix. The remaining columns list the detailed values of accuracy, precision,
recall and F1-score and AUC metrics.

On the small dataset, although conceptually simple, APPTconcat can obtain
79.04%, 80.67%, 81.34%, 80.92% and 83.43% for accuracy, precision, recall,
F1-score and AUC metrics, four of which are highest among all investigated
concatenation methods. APPTproduct has the highest recall score (96.32%),
while it performs worse than APPTconcat by 15.77%, 18.30%, 6.11% and 16.97%
for the other four metrics. APPTaddition and APPTsubtraction perform the
addition and subtraction operation for buggy and patched vectors, and have
similar performance for all metrics. Meanwhile, a mixed method APPTmix that
applies these different comparison functions to represent the changed
embedding vector can achieve better results than APPTconcat, which is also
consistent with the existing study results [32, 12]. Such results indicate
that the pre-trained model can better capture the code change information by
integrating different concatenation ways. On the large dataset, APPTconcat
achieves the best performance in accuracy, F1-score and AUC metrics, while
APPTsubtraction and APPTmix perform best in precision and recall respectively.
The difference in performance is similar as the methods have relatively high
metric values. For example, all metric values are higher than 99% for
APPTconcat and APPTmix.

####  5.3.4 RQ3.3 Results: The Impact of Pre-trained Model Choice

TABLE IX: Effectiveness of APPT with different pre-trained models. Dataset | Model | Accuracy | Precision | Recall | F1-score | AUC
---|---|---|---|---|---|---
small | APPTbert | 79.04% ($\uparrow$ 3.6) | 80.67% ($\uparrow$ 1.2) | 81.34% ($\uparrow$ 4.8) | 80.92% ($\uparrow$ 2.9) | 83.34% ($\uparrow$ 3.1)
APPTcodebert | 81.49% ($\uparrow$ 6.1) | 82.10% ($\uparrow$ 2.6) | 84.73% ($\uparrow$ 8.2) | 83.35% ($\uparrow$ 5.3) | 85.32% ($\uparrow$ 5.0)
APPTgraphcodebert | 81.83% ($\uparrow$ 6.4) | 83.68% ($\uparrow$ 4.2) | 83.63% ($\uparrow$ 7.2) | 83.47% ($\uparrow$ 5.5) | 85.79% ($\uparrow$ 5.5)
large | APPTbert | 99.13% ($\uparrow$ 0.5) | 99.09% ($\uparrow$ 0.2) | 99.13% ($\uparrow$ 0.9) | 99.11% ($\uparrow$ 0.5) | 99.86% ($\uparrow$ 1.0)
APPTcodebert | 99.57% ($\uparrow$ 1.0) | 99.71% ($\uparrow$ 0.8) | 99.40% ($\uparrow$ 1.2) | 99.55% ($\uparrow$ 1.0) | 99.89% ($\uparrow$ 1.0)
APPTgraphcodebert | 99.61% ($\uparrow$ 1.0) | 99.61% ($\uparrow$ 0.7) | 99.59% ($\uparrow$ 1.4) | 99.60% ($\uparrow$ 1.0) | 99.90% ($\uparrow$ 1.0)
$\uparrow$ denotes performance improvement against state-of-the-art technique
CACHE.

Table IX demonstrates the predicted performance of three pre-trained models.
The first column lists the two datasets. The second column lists the three
models , i.e., BERT, CodeBERT, and GraphCodeBERT. The remaining columns list
the detailed values of accuracy, precision, recall and F1-score and AUC
metrics.

Generally speaking, all of the adopted models achieve a higher performance
than state-of-the-art technique CACHE on all metrics. For example, on the
small dataset, BERT, CodeBERT and GraphCodeBERT reach 80.9%, 83.3%, and 83.5%
with respect to the F1-score, which is 2.9%, 5.3%, and 5.5% higher than CACHE,
respectively. A similar improvement can also be observed on the large dataset.
This demonstrates the model choice may not impact the performance
dramatically, and pre-trained models can consistently achieve state-of-the-art
performance.

Specifically, to compare the performance of different pre-trained models, we
can observe that both CodeBERT and GraphCodeBert achieve a better value for
all metrics on the small dataset. This superior performance also generalizes
to large datasets, where CodeBERT and GraphCodeBert have better or competitive
(e.g., AUC) performance on the metrics. One possible explanation for this is
that BERT is designed for natural language processing tasks, while CodeBERT
and GraphCodeBERT regard a source code as a sequence of tokens or graph
representation and then pre-train models on source code to support code-
related tasks. This indicates that although pre-trained models in NLP can
achieve state-of-the-art performance for assessing patch correctness, the
adoption of pre-trained models targeting source code can further boost the
improvement.

Answer to RQ3: The performance under different choices demonstrates that: (1)
the beginning code tokens can represent the buggy and patched code snippets
well for the pre-trained model; (2) the concat of buggy and patched vectors is
better than other methods to distinguish the changed code snippets, while the
integration of different concatenation ways can achieve optimum results. (3)
advanced pre-trained models can provide a stable even better performance.

##  6 Discussion

###  6.1 Threats to Validity

To facilitate the replication and verification of our experiments, we have
made the relevant materials (including source code, trained models, and patch
data) available. Despite that, our study still faces some threats to validity,
listed as follows.

The first threat to validity lies in the patch benchmark. We focus on the
Defects4J database with reproducible real faults and collect 1,183 patches
generated by existing APR tools. However, the patch benchmark may not consider
all available APR tools. To address this, following the latest work [20], we
include the 22 APR tools covering four categories. It should be worth noting
that although the learning-based category contains only SequenceR, it contains
73 patches, which is the largest number for a single APR tool [20]. We also
mitigate the potential bias by using multiple evaluation metrics to
exhaustively assess the APCA techniques. Further, we adopt another large
benchmark containing 49,694 real-world patches to evaluate the generalization
ability of the studied techniques. Overall, to the best of our knowledge, the
used patch benchmarks are the largest set explored in the literature on patch
correctness assessment.

The second threat to validity is that the performance of APPT may not
generalize to other pre-trained models. We select BERT in our experiment due
to its powerful performance in recent code-related works. However, it is
unclear whether the conclusions in our experiment (discussed in Section 5) can
be maintained when using other pre-trained models. We have mitigated the
potential threat by using CodeBERT and GraphCodeBERT to demonstrate the
performance of APPT under different pre-trained models. The investigated pre-
trained models include both code-related ones (e.g., CodeBERT) and natural
language-specific ones (e.g., BERT). We also rely on two diverse patch
benchmarks to ensure the generality of the experimental conclusions.

The last threat to validity is the implementation of the baselines. In our
work, we compare APPT against a wide range of APCA techniques with different
categories. Implementing these baselines may introduce a potential threat to
the internal validity. To mitigate this threat, following the recent work
[20], we conduct the experiment under the same setting and reuse the released
results from the original work [20, 12, 13]. Further, we carefully check the
reused results and publicly release all our materials for further
verification.

###  6.2 Comparison with BATS

In our work, following some recent APCA work [12, 13], 30 related APCA
techniques with different categories (i.e., 16 representation learning-based
ones, 9 dynamic-based ones, 4 static-based ones and 2 learning-based ones) are
compared in our experiment (discussed in Section 5). To the best of our
knowledge, the selected baselines are the largest set on patch correctness
prediction in the literature. However, there may exist other possible
techniques that could have been used. For example, the recent BATS [19]
predicts patch correctness based on the similarity of failing test cases,
which can be complementary to the state-of-the-art APCA techniques. We do not
include BATS in our experiment (discussed in Section 5) because it requires
historical test cases as the search space for searching similar cases, which
are not available in our dataset.

We then perform an additional evaluation by assessing APPT on the dataset
provided in BATS. However, BATS fails to assess some plausible patches as it
considers only historical test cases with the similarity which are higher than
a threshold. For example, BATS with 0.8 threshold value is able to predict
only 8.9% (114/1278) of the plausible patches. Thus, we compare APPT against
BATS with 0.0 threshold value, which can perform prediction for all patches.
We also compare APPT against BATS with 0.8 threshold value, as it achieves the
best recall, F1-score and AUC performance among all threshold values. The
results are presented in Table X. The first column lists APPT and BATS (with
0.0 and 0.8 threshold values, respectively). The second column lists the
number of predicted patches. Each cell is represented as $x(y)$, where $x$ is
the number of patches predicted by APPT and BATS and $y$ is the total number
of patches in the dataset. The remaining columns list the detailed performance
under the metrics. We can find APPT achieves 83.39%$\sim$85.05%, improving the
metrics by 21.56%$\sim$34.58% when compared with BATS (threshold is set to
0.0). When the threshold of BATS is set to 0.8, APPT can still improve the
metrics by 12.40% on average while predicting 91.1% more plausible patches.
Overall, the results demonstrate that APPT performs better than BATS in terms
of the number of predicted patches and the prediction metrics.

TABLE X: Comparison with a state-of-the-art learning-based APCA technique BATS. APCA | #Patch | Accurancy | Preciosn | Recall | F1-score
---|---|---|---|---|---
BATS (0.0) | 1278 (1278) | 52.50% | 48.81% | 62.82% | 54.94%
BATS (0.8) | 114 (1278) | 67.54% | 63.16% | 84.21% | 72.18%
APPA | 1278 (1278) | 85.05% | 83.39% | 84.38% | 83.88%

##  7 Implication and Guideline

Based on the observations in our experiment, we can summarize the following
essential practical guidelines for future patch correctness assessment
studies.

Simple features can work. Our study demonstrates that APPT, representing
source code as a sequence of tokens, performs even better than the existing
learning techniques (e.g., CACHE) considering complex code-aware
characteristics (e.g., abstract syntax tree). Also, the token sequences can
already outperform manually-designed static features (e.g., the line number)
and time-consuming dynamic features (e.g., code coverage) in this work. Such
observations indicate that simple features, such as code sequences, should not
be just ignored and a systematic study to explore the impact of different code
representations is needed in the future. In fact, they should be considered
and even integrated with different features (e.g., data flow graph) to design
more advanced patch correctness assessment techniques.

The quality of the training dataset is important. We can find that APPT
achieves 91.5% precision in Table IV while the precision is decreased by 10.8%
in Table VI. Similar performance can also be observed in Lin et al. [20]. The
results show that more training data cannot always lead to better performance
for patch correctness assessment. It is crucial to automatically select the
most informative training set that represents the whole patch benchmarks to
optimize the prediction accuracy. For example, it is interesting to explore
how the number of patches is distributed across fix patterns and how to select
balanced patches for each fix pattern. Future work can also be conducted to
investigate training data selection approaches targeting specific bug
benchmarks under prediction or even specific bug types under prediction.

Pre-trained model-based APCA techniques require more attention. Our results
show that the BERT-based APPT performs even better than the state-of-the-art
APCA techniques. Also, the CodeBERT-based and GraphCodeBERT-based APPT can
further enhance the prediction effectiveness. Such observation motivates
future researchers to investigate more advanced APCA techniques by employing
different pre-trained models. For example, it is interesting to propose
domain-specific pre-trained models by designing repair-related pre-training
tasks. Meanwhile, thorough evaluations are recommended to explore how
different features, such as bug types and fix patterns, influence the
performance of pre-trained models in patch correctness prediction.

##  8 Related Work

In this paper, we adopt pre-trained language models to predict patch
correctness generated by off-the-shelf automated program repair tools. Our
work is related to automated program repair, patch correctness assessment and
pre-trained models. We have introduced the existing work about patch
correctness assessment in Section 4.3. Thus, in this section, we focus on and
discuss the existing work on automated program repair techniques (Section 8.1)
and pre-trained models (Section 8.2).

###  8.1 Automated Program Repair

Over the past decade, researchers have proposed a variety of techniques to
generate patches based on different hypotheses [1, 69]. Following recent work
[11, 2, 7], we categorize them into four main categories: heuristic-based [70,
41, 38], constraint-based [44, 45, 71], template-based [51, 52, 5] and
learning-based repair techniques [35, 72, 39, 40].

_$\bullet$ Heuristic-based repair techniques._ These techniques usually use a
heuristic algorithm to find a valid patch by iteratively exploring a search
space of syntactic program modifications [70, 41, 38]. Among them, GenProg
[70] proposed in the early days has been considered a seminal work in this
field, which uses genetic programming to search for correct repairs. GenProg
represents candidate repairs as sequences of edits to source code and evaluate
them by the execution results of test cases. Those candidates that pass more
test cases are considered to have a higher fitness and are iteratively applied
to produce new candidates based on mutation and crossover operations. The
recent SimFix technique [42] utilizes code change operations from existing
patches across different projects and similar code snippets within the buggy
project to build two search spaces. Then, the intersection of the above two
search spaces is further used to search the final patch using basic
heuristics.

_$\bullet$ Constraint-based repair techniques._ These techniques mainly focus
on repairing conditional statements, which can repair more than half of the
bugs repaired by existing APR approaches [47, 44, 45]. In detail, these
techniques transform the patch generation into a constraint-solving problem,
and use a solver to obtain a feasible solution. For example, Nopol [45] relies
on an SMT solver to solve the condition synthesis problem after identifying
potential locations of patches by angelic fix localization and collecting test
execution traces of the program. Among them, ACS [46] refining the ranking of
ingredients for condition synthesis is considered one of the most advanced
constraint-based repair techniques [7].

_$\bullet$ Template-based repair techniques._ These techniques generate
patches by designing pre-defined fix patterns to mutate buggy code snippets
with the retrieved donor code [51, 5, 52]. For example, Liu et al. [5] revisit
the repair performance of repair patterns using a systematic study that
evaluates the effectiveness of a variety of fix patterns summarized from the
literature. Among them, the recent PraPR technique [73] is able to generate
plausible and correct patches for 148 and 43 real bugs, respectively, which is
the largest number of bugs reported as fixed for Defects4J when published.

_$\bullet$ Learning-based repair techniques._ These techniques attempt to fix
bugs enhanced by machine learning techniques [74, 75, 76, 39, 35, 30] and are
getting increasing attention recently. For example, Tufano et al. [75]
extensively evaluate the ability of neural machine translation techniques to
generate patches from bug-fixes commits in the wild. Li et al. [35] adopt a
tree-based RNN encoder-decoder model (i.e., DLFix) to learn code contexts and
transformations from previous bug fixes. Lutellier et al. [39] propose a new
context-aware NMT architecture (i.e., CoCoNut) that represents the buggy
source code and its surrounding context separately, to automatically fix bugs
in multiple programming languages.

In our experiment, we select 22 representative APR tools (e.g., SimFix, ACS,
and SEQUENCER) from the four categories, representing state-of-the-art
techniques in the corresponding category. Then we evaluate APPT on the
plausible patches (i.e., passing the original test cases) generated by these
APR techniques.

###  8.2 Pre-trained Model

Our approach is inspired by the application of pre-trained models in NLP and
code-related tasks. In this section, we first introduce the existing studies
about pre-trained models in NLP (Section 8.2.1) and SE (Section 8.2.2). We
then discuss the application of pre-trained models to some code-related tasks
in SE (Section 8.2.3).

####  8.2.1 Pre-trained Model in NLP

Recent work has demonstrated substantial gains on many NLP tasks and
benchmarks by pre-training on a large corpus of text followed by fine-tuning
on a specific task. For example, Devlin et al. [24] propose a new language
representation model BERT to pre-train deep bidirectional representations from
the unlabeled text by jointly conditioning on both left and right contexts in
all layers. To explore the landscape of transfer learning techniques for NLP,
Raffel et al. [26] propose a text-to-text transfer transformer T5 by
introducing a unified framework that converts all text-based language problems
into a text-to-text format. Brown et al. [25] propose an autoregressive
language model GPT-3 without any gradient updates or fine-tuning, with tasks
and few-shot demonstrations specified purely via text interaction with the
model.

In this work, we choose BERT to encode a given plausible patch into a fixed-
length representation vector as the input of the deep learning classifier, due
to the powerful performance of BERT in previous work [77].

####  8.2.2 Pre-trained Model in SE

Inspired by the application of pre-trained models in NLP, many researchers
apply the pre-trained model to code-related tasks. Instead of designing new
network architectures, SE researchers usually adopt existing architectures in
NLP and design some code-aware pre-training tasks (e.g., code-AST prediction
and bimodal dual generation) to learn representations of the source code. Then
the pre-trained models are further fine-tuned to some diversified code-related
tasks such as code-code (clone detection, defect detection, cloze test, code
completion, code refinement, and code-to-code translation), text-code (natural
language code search, text-to-code generation), and code-text (code
summarization) scenarios.

For example, Feng et al. [28] present a bimodal pre-trained model (CodeBERT)
for natural language and programming languages by masked language modeling and
replaced token detection to support code search and code documentation
generation tasks. Guo et al. [29] present the first pre-trained model
(GraphCodeBERT) that leverages code structure to learn code representation to
improve code understanding tasks (i.e., code search, clone detection, code
translation, and code refinement). Guo et al. [27] present UniXcoder, a
unified cross-modal pre-trained model for programming language. UniXcoder
utilizes mask attention matrices with prefix adapters to control the behavior
of the model and leverages cross-modal contents such as AST and code comment
to enhance code representation. In contrast to most studies pre-training a
large-scale model from scratch costly, we attempt to boost patch correctness
assessment on top of the existing pre-trained language model fine-tuning
paradigm.

In this work, to further explore the generalization ability of APPT, we select
other BERT-like models (i.e., CodeBERT and GraphCodeBERT) as the encoder stack
due to their powerful performance in the code-related tasks.

####  8.2.3 Applications of Pre-trained Model in SE

In addition to the above-mentioned typical code-related tasks (e.g., automatic
bug-fixing, injection of code mutants, generation of asserts in tests and code
summarization in [78]), researchers have also applied pre-trained models to
some other domains (e.g., code completion, and program repair) in SE.

For example, Cinisell et al. [77] evaluate the performance of the BERT model
in the task of code completion at different granularity levels, including
single tokens, one or multiple entire statements. The results show that the
model achieves promising results superior to state-of-the-art n-gram models,
and the model learns better on some specific datasets (e.g., Android) when
code abstraction is used. Ciborowska et al. [79] apply BERT to the bug
localization problem with the goal of improved retrieval quality, especially
on bug reports where straightforward textual similarity would not suffice.
Recently, Salza et al. [80] investigate how transfer learning can be applied
to code search by pre-training and fine-tuning a BERT-based model on
combinations of natural language and source code. Mashhadi et al. [81] propose
a novel pre-trained model-based APR technique by fine-tuning CodeBERT on the
ManySStuBs4J benchmark and find the approach generates fix codes for different
types of bugs with comparable effectiveness and efficacy compared with state-
of-the-art APR techniques.

Although there exist some SE tasks (e.g., code review and bug localization)
benefitting from pre-trained models, in this work, we perform the first
application of pre-trained models to predict the generated patch correctness
in automated program repair.

##  9 Conclusion

In this work, we present APPT, a novel automated patch correctness prediction
technique based on the pre-training model and classifier. We first adopt the
off-the-shelf pre-trained model as the encoder stack and LSTM stack to enhance
the dependency relationships among the buggy and patched code snippets. Then
we build a deep learning classifier by two fully connected layers and a
standard softmax function to predict whether the patch is overfitting or not.
We conduct experiments on both patch datasets and show that APPT significantly
outperforms state-of-the-art learning-based and traditional APCA techniques.
We further demonstrate that APPT is generalizable to various pre-trained
models. Based on these observations, some implications and guidelines on
improving the existing learning-based techniques (e.g., the usage of simple
features and pre-trained models) are provided. We highlight the direction of
applying pre-trained models to predict patch correctness automatically.

## Acknowledgments

This work is supported partially by the National Key Research and Development
Program of China (2021YFB1715600), the National Natural Science Foundation of
China (61932012, 62141215), and the Program B for Outstanding PhD Candidate of
Nanjing University (202201B054).

## References

  * [1] L. Gazzola, D. Micucci, and L. Mariani, “Automatic software repair: A survey,” _IEEE Transactions on Software Engineering (TSE’17)_ , vol. 45, no. 1, pp. 34–67, 2017.
  * [2] S. Benton, X. Li, Y. Lou, and L. Zhang, “Evaluating and improving unified debugging,” _IEEE Transactions on Software Engineering (TSE’21)_ , no. 01, pp. 1–1, 2021.
  * [3] J. Aranda and G. Venolia, “The secret life of bugs: Going past the errors and omissions in software repositories,” in _Proceedings of the 31st IEEE International Conference on Software Engineering (ICSE’09)_. IEEE, 2009, pp. 298–308.
  * [4] C. L. Goues, M. Pradel, and A. Roychoudhury, “Automated program repair,” _Communications of the ACM (CACM’19)_ , vol. 62, no. 12, pp. 56–65, 2019\.
  * [5] K. Liu, A. Koyuncu, D. Kim, and T. F. Bissyandé, “Tbar: Revisiting template-based automated program repair,” in _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’19)_ , 2019, pp. 31–42.
  * [6] T. Durieux, F. Madeiral, M. Martinez, and R. Abreu, “Empirical review of java program repair tools: a large-scale experiment on 2,141 bugs and 23,551 repair attempts,” in _Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’19)_ , 2019, pp. 302–313.
  * [7] K. Liu, S. Wang, A. Koyuncu, K. Kim, T. F. Bissyandé, D. Kim, P. Wu, J. Klein, X. Mao, and Y. L. Traon, “On the efficiency of test suite based program repair: A systematic assessment of 16 automated repair systems for java programs,” in _Proceedings of the 42nd ACM/IEEE International Conference on Software Engineering (ICSE’20)_ , 2020, pp. 615–627.
  * [8] X.-B. D. Le, L. Bao, D. Lo, X. Xia, S. Li, and C. Pasareanu, “On reliability of patch correctness assessment,” in _Proceedings of the 41st IEEE/ACM International Conference on Software Engineering (ICSE’19)_. IEEE, 2019, pp. 524–535.
  * [9] F. Long and M. Rinard, “An analysis of the search spaces for generate and validate patch generation systems,” in _Proceedings of the 38th IEEE/ACM International Conference on Software Engineering (ICSE’16)_ , 2016, pp. 702–713.
  * [10] Y. Tao, J. Kim, S. Kim, and C. Xu, “Automatically generated patches as debugging aids: a human study,” in _Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE’14)_ , 2014, pp. 64–74.
  * [11] Q. Zhang, Y. Zhao, W. Sun, C. Fang, Z. Wang, and L. Zhang, “Program repair: Automated vs. manual,” _arXiv preprint arXiv:2203.05166_ , 2022.
  * [12] H. Tian, K. Liu, A. K. Kaboré, A. Koyuncu, L. Li, J. Klein, and T. F. Bissyandé, “Evaluating representation learning of code changes for predicting patch correctness in program repair,” in _Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering (ASE’20)_ , 2020, pp. 981–992.
  * [13] S. Wang, M. Wen, B. Lin, H. Wu, Y. Qin, D. Zou, X. Mao, and H. Jin, “Automated patch correctness assessment: How far are we?” in _Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering (ASE’20)_ , 2020, pp. 968–980.
  * [14] S. H. Tan, H. Yoshida, M. R. Prasad, and A. Roychoudhury, “Anti-patterns in search-based program repair,” in _Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE’16)_ , 2016, pp. 727–738.
  * [15] G. Fraser and A. Arcuri, “Evosuite: automatic test suite generation for object-oriented software,” in _Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering (FSE’11)_ , 2011, pp. 416–419.
  * [16] C. Pacheco and M. D. Ernst, “Randoop: feedback-directed random testing for java,” in _Proceedings of the Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion (OOPSLA ’07)_ , 2007, pp. 815–816.
  * [17] Y. Xiong, X. Liu, M. Zeng, L. Zhang, and G. Huang, “Identifying patch correctness in test-based program repair,” in _Proceedings of the 40th IEEE/ACM International Conference on Software Engineering (ICSE’18)_ , 2018, pp. 789–799.
  * [18] H. Ye, J. Gu, M. Martinez, T. Durieux, and M. Monperrus, “Automated classification of overfitting patches with statically extracted code features,” _IEEE Transactions on Software Engineering (TSE’22)_ , vol. 48, no. 8, pp. 2920–2938, 2022.
  * [19] H. Tian, Y. Li, W. Pian, A. K. Kabore, K. Liu, A. Habib, J. Klein, and T. F. Bissyandé, “Predicting patch correctness based on the similarity of failing test cases,” _ACM Transactions on Software Engineering and Methodology (TOSEM’22)_ , vol. 31, no. 4, pp. 1–30, 2022.
  * [20] B. Lin, S. Wang, M. Wen, and X. Mao, “Context-aware code change embedding for better patch correctness assessment,” _ACM Transactions on Software Engineering and Methodology (TOSEM’22)_ , vol. 31, no. 3, pp. 1–29, 2022.
  * [21] W. E. Wong, R. Gao, Y. Li, R. Abreu, and F. Wotawa, “A survey on software fault localization,” _IEEE Transactions on Software Engineering (TSE’16)_ , vol. 42, no. 8, pp. 707–740, Aug 2016.
  * [22] Y. Lou, A. Ghanbari, X. Li, L. Zhang, H. Zhang, D. Hao, and L. Zhang, “Can automated program repair refine fault localization? a unified debugging approach,” in _Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’20)_ , 2020, pp. 75–87.
  * [23] E. K. Smith, E. T. Barr, C. Le Goues, and Y. Brun, “Is the cure worse than the disease? overfitting in automated program repair,” in _Proceedings of the 10th Joint Meeting of the European Software Engineering Conference and ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE’15)_ , 2015, pp. 532–543.
  * [24] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [25] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” in _Proceedings of the Advances in Neural Information Processing Systems (NeurIPS’20)_ , vol. 33, 2020, pp. 1877–1901.
  * [26] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” _Journal of Machine Learning Research (JMLR’20)_ , vol. 21, pp. 1–67, 2020.
  * [27] D. Guo, S. Lu, N. Duan, Y. Wang, M. Zhou, and J. Yin, “Unixcoder: Unified cross-modal pre-training for code representation,” in _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL’22)_ , 2022, pp. 7212–7225.
  * [28] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang _et al._ , “Codebert: A pre-trained model for programming and natural languages,” in _Findings of the Association for Computational Linguistics (EMNLP’20)_ , 2020, pp. 1536–1547.
  * [29] D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu _et al._ , “Graphcodebert: Pre-training code representations with data flow,” in _Proceedings of the 9th International Conference on Learning Representations (ICLR’21)_ , 2021, pp. 1–18.
  * [30] N. Jiang, T. Lutellier, and L. Tan, “Cure: Code-aware neural machine translation for automatic program repair,” in _Proceedings of the 43rd IEEE/ACM International Conference on Software Engineering (ICSE’21)_ , 2021, pp. 1161–1173.
  * [31] J. Ganhotra and S. Joshi, “Does dialog length matter for next response selection task? an empirical study,” _arXiv preprint arXiv:2101.09647_ , 2021\.
  * [32] T. Hoang, H. J. Kang, D. Lo, and J. Lawall, “Cc2vec: Distributed representations of code changes,” in _Proceedings of the 42nd ACM/IEEE International Conference on Software Engineering (ICSE’20)_ , 2020, pp. 518–529.
  * [33] Z. Li, D. Zou, S. Xu, X. Ou, H. Jin, S. Wang, Z. Deng, and Y. Zhong, “Vuldeepecker: A deep learning-based system for vulnerability detection,” in _Proceedings of the 25th Annual Network and Distributed System Security Symposium (NDSS’18)_ , 2018, pp. 1–15.
  * [34] X. Meng, X. Wang, H. Zhang, H. Sun, and X. Liu, “Improving fault localization and program repair with deep semantic features and transferred knowledge,” in _Proceedings of the 44th IEEE/ACM International Conference on Software Engineering (ICSE’22)_ , 2022, pp. 1169–1180.
  * [35] Y. Li, S. Wang, and T. N. Nguyen, “Dlfix: Context-based code transformation learning for automated program repair,” in _Proceedings of the 42nd ACM/IEEE International Conference on Software Engineering (ICSE’20)_ , 2020, pp. 602–614.
  * [36] U. Alon, M. Zilberstein, O. Levy, and E. Yahav, “code2vec: Learning distributed representations of code,” _Proceedings of the ACM on Programming Languages (POPL’19)_ , vol. 3, no. POPL, pp. 1–29, 2019.
  * [37] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _Proceedings of the 3rd International Conference on Learning Representations (ICLR Poster’15)_ , 2015, pp. 1–15.
  * [38] Y. Yuan and W. Banzhaf, “Arja: Automated repair of java programs via multi-objective genetic programming,” _IEEE Transactions on Software Engineering (TSE’18)_ , vol. 46, no. 10, pp. 1040–1067, 2018.
  * [39] T. Lutellier, H. V. Pham, L. Pang, Y. Li, M. Wei, and L. Tan, “Coconut: combining context-aware neural translation models using ensemble for program repair,” in _Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’20)_ , 2020, pp. 101–114.
  * [40] Y. Wei, Z. Quanjun, H. Tieke, F. Chunrong, H. Nguyen Quoc Viet, H. Xiaodong, and Y. Hongzhi, “Circle: Continual repair across programming languages,” in _Proceedings of the 31th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’22)_ , 2022, pp. 427–438.
  * [41] M. Martinez and M. Monperrus, “Astor: A program repair library for java,” in _Proceedings of the 25th International Symposium on Software Testing and Analysis (ISSTA’16)_ , 2016, pp. 441–444.
  * [42] J. Jiang, Y. Xiong, H. Zhang, Q. Gao, and X. Chen, “Shaping program repair space with existing patches and similar code,” in _Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’18)_ , 2018, pp. 298–309.
  * [43] M. Wen, J. Chen, R. Wu, D. Hao, and S.-C. Cheung, “Context-aware patch generation for better automated program repair,” in _Proceedings of the 40th IEEE/ACM International Conference on Software Engineering (ICSE’18)_ , 2018, pp. 1–11.
  * [44] T. Durieux and M. Monperrus, “Dynamoth: dynamic code synthesis for automatic program repair,” in _Proceedings of the 11th International Workshop on Automation of Software Test (AST’16)_ , 2016, pp. 85–91.
  * [45] J. Xuan, M. Martinez, F. Demarco, M. Clement, S. L. Marcote, T. Durieux, D. Le Berre, and M. Monperrus, “Nopol: Automatic repair of conditional statement bugs in java programs,” _IEEE Transactions on Software Engineering (TSE’16)_ , vol. 43, no. 1, pp. 34–55, 2016.
  * [46] Y. Xiong, J. Wang, R. Yan, J. Zhang, S. Han, G. Huang, and L. Zhang, “Precise condition synthesis for program repair,” in _Proceedings of the 39th IEEE/ACM International Conference on Software Engineering (ICSE’17)_. IEEE, 2017, pp. 416–426.
  * [47] M. Martinez and M. Monperrus, “Ultra-large repair search space with automatically mined templates: The cardumen mode of astor,” in _Proceedings of the International Symposium on Search Based Software Engineering (SSBSE’18)_. Springer, 2018, pp. 65–86.
  * [48] L. Chen, Y. Pei, and C. A. Furia, “Contract-based program repair without the contracts,” in _Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE’17)_ , 2017, pp. 637–647.
  * [49] J. Hua, M. Zhang, K. Wang, and S. Khurshid, “Towards practical program repair with on-demand candidate generation,” in _Proceedings of the 40th international conference on software engineering (ICSE’18)_ , 2018, pp. 12–23.
  * [50] K. Liu, A. Koyuncu, T. F. Bissyandé, D. Kim, J. Klein, and Y. Le Traon, “You cannot fix what you cannot find! an investigation of fault localization bias in benchmarking automated program repair systems,” in _Proceedings of the 12th IEEE conference on software testing, validation and verification (ICST’19)_ , 2019, pp. 102–113.
  * [51] A. Koyuncu, K. Liu, T. F. Bissyandé, D. Kim, J. Klein, M. Monperrus, and Y. Le Traon, “Fixminer: Mining relevant fix patterns for automated program repair,” _Empirical Software Engineering (ESE’20)_ , vol. 25, no. 3, pp. 1980–2024, 2020.
  * [52] K. Liu, A. Koyuncu, D. Kim, and T. F. Bissyandé, “Avatar: Fixing semantic bugs with fix patterns of static analysis violations,” in _Proceedings of the 26th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER’19)_ , 2019, pp. 1–12.
  * [53] X. Liu and H. Zhong, “Mining stackoverflow for program repair,” in _Proceedings of the 25th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER’18)_ , 2018, pp. 118–129.
  * [54] X. B. D. Le, D. Lo, and C. Le Goues, “History driven program repair,” in _Proceedings of the 23rd IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER’16)_ , vol. 1. IEEE, 2016, pp. 213–224.
  * [55] Z. Chen, S. Kommrusch, M. Tufano, L.-N. Pouchet, D. Poshyvanyk, and M. Monperrus, “Sequencer: Sequence-to-sequence learning for end-to-end program repair,” _IEEE Transactions on Software Engineering (TSE’19)_ , vol. 47, no. 9, pp. 1943–1959, 2019.
  * [56] R. Just, D. Jalali, and M. D. Ernst, “Defects4j: A database of existing faults to enable controlled testing studies for java programs,” in _Proceedings of the 23rd International Symposium on Software Testing and Analysis (ISSTA’14)_ , 2014, pp. 437–440.
  * [57] R.-M. Karampatsis and C. Sutton, “How often do single-statement bugs occur? the manysstubs4j dataset,” in _Proceedings of the 17th International Conference on Mining Software Repositories (MSR’20)_ , 2020, pp. 573–577.
  * [58] R. K. Saha, Y. Lyu, W. Lam, H. Yoshida, and M. R. Prasad, “Bugs. jar: a large-scale, diverse dataset of real-world java bugs,” in _Proceedings of the 15th International Conference on Mining Software Repositories (MSR’18)_ , 2018, pp. 10–13.
  * [59] T. Durieux and M. Monperrus, “IntroClassJava: A Benchmark of 297 Small and Buggy Java Programs,” Universite Lille 1, Tech. Rep. hal-01272126, 2016.
  * [60] F. Madeiral, S. Urli, M. Maia, and M. Monperrus, “Bears: An extensible java bug benchmark for automatic program repair studies,” in _Proceedings of the 26th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER’19)_ , 2019, pp. 468–478.
  * [61] D. Lin, J. Koppel, A. Chen, and A. Solar-Lezama, “Quixbugs: A multi-lingual program repair benchmark set based on the quixey challenge,” in _Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity (SPLASH Companion’17)_ , 2017, pp. 55–56.
  * [62] Q. Xin and S. P. Reiss, “Identifying test-suite-overfitted patches through test case generation,” in _Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’17)_ , 2017, pp. 226–236.
  * [63] B. Yang and J. Yang, “Exploring the differences between plausible and correct patches at fine-grained level,” in _Proceedings of the 2nd IEEE International Workshop on Intelligent Bug Fixing (IBF’20)_. IEEE, 2020, pp. 1–8.
  * [64] Q. Xin and S. P. Reiss, “Leveraging syntax-related code for automated program repair,” in _Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE’17)_ , 2017, pp. 660–670.
  * [65] X.-B. D. Le, D.-H. Chu, D. Lo, C. Le Goues, and W. Visser, “S3: syntax-and semantic-guided repair synthesis via programming by examples,” in _Proceedings of the 11th Joint Meeting on European Software Engineering Conference and ACM SIGSOFT Symposium on Foundations of Software Engineering (ESEC/FSE’17)_ , 2017, pp. 593–604.
  * [66] H. Ye, M. Martinez, and M. Monperrus, “Automated patch assessment for program repair at scale,” _Empirical Software Engineering (ESE’21)_ , vol. 26, no. 2, pp. 1–38, 2021.
  * [67] J. Yang, A. Zhikhartsev, Y. Liu, and L. Tan, “Better test cases for better automated program repair,” in _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE’17)_ , 2017, pp. 831–841.
  * [68] H. Ye, M. Martinez, and M. Monperrus, “Neural program repair with execution-based backpropagation,” in _Proceedings of the 44th IEEE/ACM International Conference on Software Engineering (ICSE’22)_ , 2022, pp. 1506–1518.
  * [69] M. Monperrus, “The living review on automated program repair,” HAL/archives-ouvertes.fr, Tech. Rep. hal-01956501, 2022.
  * [70] C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer, “Genprog: A generic method for automatic software repair,” _IEEE Transactions on Software Engineering (TSE’12)_ , vol. 38, no. 01, pp. 54–72, 2012.
  * [71] S. Mechtaev, J. Yi, and A. Roychoudhury, “Angelix: Scalable multiline program patch synthesis via symbolic analysis,” in _Proceedings of the 38th international conference on software engineering (ICSE’16)_ , 2016, pp. 691–701.
  * [72] Q. Zhu, Z. Sun, Y.-a. Xiao, W. Zhang, K. Yuan, Y. Xiong, and L. Zhang, “A syntax-guided edit decoder for neural program repair,” in _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’21)_ , 2021, pp. 341–353.
  * [73] A. Ghanbari, S. Benton, and L. Zhang, “Practical program repair via bytecode mutation,” in _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’19)_ , 2019, pp. 19–30.
  * [74] M. White, M. Tufano, M. Martinez, M. Monperrus, and D. Poshyvanyk, “Sorting and transforming program repair ingredients via deep learning code similarities,” in _Proceedings of the 26th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER’19)_ , 2019, pp. 479–490.
  * [75] M. Tufano, C. Watson, G. Bavota, M. D. Penta, M. White, and D. Poshyvanyk, “An empirical study on learning bug-fixing patches in the wild via neural machine translation,” _ACM Transactions on Software Engineering and Methodology (TOSEM’19)_ , vol. 28, no. 4, pp. 1–29, 2019.
  * [76] R. Gupta, S. Pal, A. Kanade, and S. Shevade, “Deepfix: fixing common c language errors by deep learning,” in _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI’17)_ , 2017, pp. 1345–1351.
  * [77] M. Ciniselli, N. Cooper, L. Pascarella, D. Poshyvanyk, M. Di Penta, and G. Bavota, “An empirical study on the usage of bert models for code completion,” in _Proceedings of the 18th IEEE/ACM International Conference on Mining Software Repositories (MSR’21)_. IEEE, 2021, pp. 108–119.
  * [78] A. Mastropaolo, S. Scalabrino, N. Cooper, D. N. Palacio, D. Poshyvanyk, R. Oliveto, and G. Bavota, “Studying the usage of text-to-text transfer transformer to support code-related tasks,” in _Proceedings of the 43rd IEEE/ACM International Conference on Software Engineering (ICSE’21)_ , 2021, pp. 336–347.
  * [79] A. Ciborowska and K. Damevski, “Fast changeset-based bug localization with bert,” in _Proceedings of the 44th International Conference on Software Engineering (ICSE’22)_ , 2022, pp. 946–957.
  * [80] P. Salza, C. Schwizer, J. Gu, and H. C. Gall, “On the effectiveness of transfer learning for code search,” _IEEE Transactions on Software Engineering (TSE’22)_ , vol. 1, no. 1, pp. 1–1, 2022.
  * [81] E. Mashhadi and H. Hemmati, “Applying codebert for automated program repair of java simple bugs,” in _Proceedings Companion of the 18th IEEE/ACM International Conference on Mining Software Repositories (MSR’21)_ , 2021, pp. 505–509.