A Systematic Literature Review on Large Language Models for Automated Program Repair

Quanjun Zhang

Chunrong Fang

Yang Xie

Yuxiang Ma

State Key Laboratory for Novel Software Technology, Nanjing University

Weisong Sun

School of Computer Science and Engineering, Nanyang Technological University
50 Nanyang Avenue

Yun Yang

Department of Computing Technologies, Swinburne University of Technology

Zhenyu Chen

State Key Laboratory for Novel Software Technology, Nanjing University

Automated Program Repair (APR) attempts to patch software bugs and reduce manual debugging efforts.
Very recently, with the advances in Large Language Models (LLMs), a rapidly increasing number of APR techniques have been proposed, significantly facilitating software development and maintenance and demonstrating remarkable performance.
However, due to ongoing explorations in the LLM-based APR field, it is challenging for researchers to understand the current achievements, challenges, and potential opportunities.
This work provides the first systematic literature review to summarize the applications of LLMs in APR between 2020 and 2024.
We analyze relevant papers from LLMs, APR and their integration perspectives.
First, we categorize existing popular LLMs that are applied to support APR and outline three types of utilization strategies for their deployment.
Besides, we detail some specific repair scenarios that benefit from LLMs, e.g., semantic bugs and security vulnerabilities.
Furthermore, we discuss several critical aspects of integrating LLMs into APR research, e.g., input forms and open science.
Finally, we highlight a set of challenges remaining to be investigated and the potential guidelines for future research.
Overall, our paper provides a systematic overview of the research landscape to the APR community, helping researchers gain a comprehensive understanding of achievements and promote future research.
Our artifacts are publicly available at the GitHub repository: <https://github.com/iSEngLab/AwesomeLLM4APR>.

<concept_desc>Software and its engineering Software testing and debugging</concept_desc>

[500]Software and its engineering Software testing and debugging

§ INTRODUCTION

Software bugs are recognized as inevitable and destructive, posing safety issues for users worldwide and costing billions of dollars in financial losses annually <cit.>.
It is non-trivial and time-consuming for developers to fix detected software bugs manually <cit.>.
Automated Program Repair (APR) plays a crucial role in software development and maintenance with the aim of fixing software bugs without human intervention.
Following the foundational work GenProg <cit.> in 2009, APR has been extensively investigated over the past decades <cit.>, and researchers have proposed a variety of APR techniques, including heuristic-based <cit.>, constraint-based <cit.>, and pattern-based <cit.> ones.
Recently, inspired by the advances of Deep Learning (DL), an increasing number of learning-based APR techniques have been proposed that utilize neural network models to automatically learn bug-fixing patterns <cit.>.
Thanks to the powerful ability of DL models to learn hidden repair patterns from massive code corpora, learning-based APR has achieved remarkable performance in the last couple of years <cit.>, attracting considerable attention from both academia and industry <cit.>.

Very recently, Large Language Models (LLMs) have been successfully applied to a broad range of source code-related tasks <cit.>, such as code generation <cit.>, code summarization <cit.>, and test generation <cit.>.
Benefiting from massive model parameters and vast training data, LLMs have demonstrated impressive performance and fundamentally revolutionized the research paradigm in the Software Engineering (SE) community.
In the domain of APR, beginning with pioneering studies, e.g., TFix <cit.>, CIRCLE <cit.> and AlphaRepair <cit.>, the community has witnessed an explosion of repair studies utilizing LLMs, already achieving considerable advantages and further indicating significant potential for future research.
However, the integration of LLMs within APR is a considerably complex undertaking, making it difficult for interested researchers to understand existing work.
For example, existing LLM-based APR studies encompass different research perspectives (e.g., empirical <cit.>, technical <cit.> and benchmark studies <cit.>), repair phases (e.g., patch generation <cit.> and correctness assessment <cit.>), repair scenarios (e.g., static warnings <cit.> and syntax errors <cit.>), mode architectures (e.g., encoder-only <cit.> and decoder-only <cit.>) and model utilization paradigms (e.g., fine-tuning <cit.>, few-shot <cit.> and zero-shot <cit.>).
Despite ongoing explorations in the field, the literature currently lacks a detailed and systematic review of the applications of LLMs in APR, making it challenging for researchers to understand the multitudinous design choices of existing work and conduct follow-up research.

This Paper.
To bridge this gap, our work provides the first systematic literature review on the deployment of rapidly emerging LLM-based APR studies.
Based on this, the community can gain a comprehensive understanding of the strengths, weaknesses, and gaps in existing LLM-based APR techniques.
We discuss what LLMs are widely adopted in state-of-the-art APR research and how they are integrated into the repair workflow.
We collect  relevant papers and perform a systematic analysis from LLMs, APR, and integration perspectives.
From our analysis, we reveal the current challenges and point out possible future directions for LLM-based APR research.
Overall, this work offers a thorough overview of the ongoing progress within the LLM-based APR community, aiding researchers in navigating this burgeoning field and advancing toward innovative practices.

To sum up, this work makes the following contributions:

* Survey Methodology.
We conduct the first systematic literature review with  high-quality APR papers that utilize recent LLMs to address repair challenges from 2020 to April 2024.

* Trend Analysis.
We perform a detailed analysis of selected APR studies in terms of publication trends, distribution of publication venues, and types of contributions.

* LLMs Perspective.
We summarize  LLMs utilized to support program repair and provide a summary of the typical usage and trends of different LLM categories in the APR domain.

* APR Perspective.
We describe common repair scenarios that LLMs are applied to, encompassing  bug types, such as security vulnerabilities and programming problems.

* Integration Perspective.
We discuss some key factors, including datasets, input representations and open science, that impact the performance of integrating LLMs into APR.

* Challenges and Opportunities.
We summarize some crucial challenges of applying LLMs in the APR field, and pinpoint some potential guidelines for future LLM-based APR research.

Paper Organization.
Section <ref> introduces some basic concepts about APR and LLMs.
Then, according to the contributions listed above, Section <ref> lists our research questions (RQs) and the research methodology to collect papers related to our work.
Section <ref> investigates the trend and distribution of LLM-based APR studies.
Section <ref> summarizes LLMs that are used by existing APR studies.
Section <ref> illustrates the primary repair scenarios that LLMs are applied to and provides a brief description of each work.
Section <ref> discusses some crucial factors during the integration of LLMs and APR, including datasets, input representation, patch correctness, and open science.
Section <ref> discusses some challenges and practical guidelines.
Section <ref> draws the conclusions.

§ BACKGROUND AND RELATED WORK

§.§ Automated Program Repair

§.§.§ Problem Description

APR, one of the most challenging activities in the SE field, aims to fix software bugs without human intervention and is regarded as a fundamental aspect of software automation.
For example, in the software development and maintenance process, developers usually attempt to implement a designed functionality according to requirement specifications and write test suites to validate its correctness.
If all test cases pass, the functionality is deemed correctly implemented; otherwise, developers must analyze the symptoms of failures and make necessary modifications to the buggy code snippets to ensure all test cases pass.
Formally, given a buggy program $P$ and its corresponding specification $S$ that $P$ fails to satisfy, APR is defined to find a minimal transformation $P'$ of $P$ that satisfies $S$.
In practice, test cases are usually utilized as the specification, and in such a typical test-driven repair scenario, APR aims to find a minimal program variant that passes the available test suite, which contains at least one test case making the original buggy program $P$ does not satisfy.

The basic pipeline of automated program repair.

§.§.§ Repair Workflow

Fig. <ref> illustrates the typical generate-and-validate workflow of APR, which is usually composed of three parts.
Specifically, for a detected bug, (1) the fault localization phase identifies suspicious code elements that need to be fixed based on off-the-shelf localization techniques <cit.>;
(2) the patch generation phase generates program variants (i.e., candidate patches) via transformation rules <cit.>;
(3) the patch validation phase utilizes available test suites as the oracle to identify correct patches by dynamic execution <cit.>.
It is important to note that a candidate patch that successfully passes the available test suite is termed a plausible patch.
However, if such a plausible patch fails to generalize to additional test cases, it is considered an overfitting patch.
Conversely, a plausible patch that remains effective across broader test cases is recognized as a correct patch, one that is semantically in alignment with patches written manually by developers.

§.§.§ Repair Techniques

The literature has introduced a multitude of APR techniques to generate correct patches from various perspectives.
Such APR techniques can be categorized into four categories:
(1) heuristic-based APR utilizes genetic programming to explore the search space of the correct patch, such as GenProg <cit.>, Astor <cit.>, ARJA <cit.>, and SimFix <cit.>;
(2) constraint-based APR usually focuses on the condition synthesis problem by treating program repair as a constraint-solving task, such as Nopol <cit.>,  Cardumen <cit.>, ACS <cit.>, and Dynamoth <cit.>;
(3) pattern-based APR utilizes pre-defined repair templates that are usually hand-crafted by experts to transform buggy code snippets into correct ones, such as TBar <cit.>, FixMiner <cit.> and Avatar <cit.>;
(4) learning-based APR treats patch generation as a neural machine translation task with the advance of DL models, such as Tufano et al. <cit.>, SequenceR <cit.>, CoCoNut <cit.>, DLFix <cit.>, CURE <cit.>, Recoder <cit.>, DEAR <cit.>, SelfAPR <cit.>, RewardAPR <cit.>, and Tare <cit.>.

Among the four types of APR techniques, learning-based APR has achieved remarkable performance by learning hidden bug-fixing patterns automatically from extensive source code databases <cit.>.
Recently, inspired by the success of LLMs in NLP and SE tasks, researchers have increasingly been utilizing advanced LLMs to address software bugs <cit.>.
Compared with learning-based APR, such LLM-based APR techniques have demonstrated significantly better performance and have received growing attention in the community, which is the focus of our work.

§.§ Large Language Models

§.§.§ Preliminaries

LLMs refer to advanced Artificial Intelligence (AI) models that undergo extensive pre-training on large text corpora to enhance their capabilities of natural language understanding, generation, and interpretation in a human-like manner <cit.>.
Typically, LLMs feature an enormous number of parameters, far exceeding the scale of traditional DL models, thus enabling them to assist in a wide range of tasks, such as question answering and machine translation.

In 1986, Rumelhart et al. <cit.> introduce Recurrent Neural Networks (RNNs), opening up the possibility of processing sequential data by maintaining a memory of previous inputs in their internal state.
In 1997, Hochreiter et al. <cit.> introduce an extension of RNNs, namely Long Short-Term Memory Networks (LSTMs), to address the long-term dependency problem by a cell state and three types of gates.
In 2017, Vaswani et al. <cit.> introduce Transformers to weigh the importance of different parts of the input data with the self-attention mechanism, laying the foundation of LLMs.
The evolution from RNNs to LSTMs and then to Transformers represents a trajectory towards more efficiently handling longer sequences and more complex patterns in data, especially for tasks involving natural language processing.
Particularly, Transformers, with their ability to manage long-range dependencies more effectively and their suitability for parallel computation, have become the dominant model architecture in many areas of AI research and application.

LLMs are typically constructed on top of the Transformer architecture mentioned above, employing a pre-training-and-fine-tuning paradigm.
Specifically, these models undergo pre-training to acquire generic language representations through self-supervised learning on extensive unlabeled data.
Subsequently, they are adapted to various downstream tasks through supervised fine-tuning using a limited amount of labeled data.
Thanks to the advanced model architecture and training paradigm, LLMs have led to breakthroughs in various NLP fields, notably with models like BERT <cit.>, GPT <cit.>, T5 <cit.>, PaLM <cit.>, LLaMA <cit.>, LLaMA2 <cit.>, LLaMA3 <cit.>.
Recently, research efforts have been rapidly growing in the domain of source code, leading to significant advancements with models like CodeBERT <cit.>, CodeGPT <cit.>, CodeT5 <cit.>, InCoder <cit.>, CodeGen <cit.>, CodeLlama <cit.>, StarCoder <cit.>) and StarCoder2 <cit.>.

§.§.§ Model Categories

The literature has seen a variety of LLMs supporting NLP and SE research, which can be categorized into three main categories based on their model architectures.
(1) Encoder-only LLMs, such as CodeBERT <cit.>, GraphCodeBERT <cit.>, train the encoder part of the Transformer to generate a fixed-dimensional bidirectional representation with Masked Language Modeling (MLM) and Next Sentence Prediction (NSP).
MLM aims to predict the original tokens that have been randomly masked out, and NSP predicts whether two given sentences actually follow each other in a text.
(2) Decoder-only LLMs, such as CodeGPT <cit.>, train the decoder part of the Transformer to support auto-regressive tasks with Causal Language Modeling (CLM), which aims to predict new tokens in a sequence based on previous tokens.
(3) Encoder-decoder LLMs, such as CodeT5 <cit.>, train both encoder and decoder parts of the Transformer to support sequence-to-sequence generation tasks with denoising objectives.
We will summarize existing LLMs and how they are leveraged to support program repair in Section <ref>.

§.§ Related Work

In this survey, we systematically collect, summarize, and analyze APR studies empowered with advanced LLMs.
Although our primary focus is the integration of LLMs with APR, given its growing prominence, there exists some related studies that explore similar aspects of either LLMs or APR.
We clarify the fundamental differences between our work and prior work below.

The first group of related surveys attempts to analyze the applications of LLM more generally in SE.
Zhang et al. <cit.> provide a detailed summarization of the LLM-based SE research from two intuitive perspectives: LLMs and their applications in SE.
They introduce representative work across 30 LLMs, 15 pre-training tasks, 16 downstream tasks, and 43 specific code-related tasks.
Besides, Wang et al. <cit.> provide a review of LLMs in software testing; Fan et al. <cit.> discuss the achievements and challenges of LLMs in SE;
Hou et al. <cit.> conduct a systematic literature review on LLM4SE.
However, these works target the whole software engineering/testing workflow rather than our specific program repair task.
They only provide a bird's-eye view of a limited number of LLM-based APR papers, while our work provides an in-depth analysis from various perspectives.
Their work indicates that program repair is the most popular scenario in which LLMs are applied within software engineering.
Given the complexity and significance of program repair, we are motivated to conduct a more in-depth review of APR achievements involving LLMs.

The second group of related surveys attempts to review the achievements of APR.
Gazzola et al. <cit.> present a survey to organize repair studies, and Monperrus et al. <cit.> present a bibliography of behavioral and state repair studies.
Both works collect publications up to 2017, hence there is no overlap with our research as the first LLM-based APR work is published in 2020.
The most highly related work to this paper is Zhang et al. <cit.>, which provides a systematic survey to summarize the state-of-the-art in the learning-based APR community.
However, they mainly focus on DL models and devote merely a single section to discussing LLM-based APR studies (only five papers), failing to include the surge of works that emerge during the last two years.
In contrast, our work solely focuses on the technology of LLM applications in the APR community until April 2024, involving the trend of utilized LLMs, optimization techniques, repair scenarios, and input forms.

§ SURVEY METHODOLOGY

In this section, guided by the principles outlined by Petersen et al. <cit.> and Kitchenham et al. <cit.>, we present details of our systematic literature review methodology.

§.§ Research Questions

We attempt to provide a comprehensive overview of recent LLMs' applications to APR by summarizing the relevant studies and further providing guidelines on the follow-up research.
To achieve this, this systematic literature review answers the following Research Questions (RQs):

* RQ1: What is the trend of APR studies that utilize LLMs?

* RQ2: Which popular LLMs have been applied to support APR?

* RQ3: What repair scenarios have been facilitated by LLMs?

* RQ4: What key factors contribute to the integration of LLMs for APR?

§.§ Search Strategy

We search and identify primary papers following the “Quasi-Gold Standard” (QGS) <cit.> method, which is a common practice to construct a set of known studies for refining search strings by combining manual and automated search processes.
We first conduct a manual search to identify a set of relevant studies and derive a search string from them, as detailed in Section <ref>.
We then utilize the search string to perform an automated search and employ a series of relatively strict filtering steps to obtain the most relevant studies, as detailed in Section <ref>.
We finally use a snowballing search to supplement the search results further, as detailed in Section <ref>.
Given the abundance of relevant papers from different research communities, such as SE and AI, this search strategy allows us to capture the most pertinent papers while achieving greater efficiency than a purely manual search with a relatively rigorous process.
Particularly, we undertake the following five phases to search for and identify relevant studies.

* Search Sources.
This phase selects high-quality publication venues for the initial manual search and well-known digital databases for the subsequent automated search.

* QGS Establishment.
This phase inspects all papers identified in the manual search and filters them by inclusion/exclusion criteria.

* Search Items.
This phase definins search items with domain knowledge and established QGS.

* Paper Collection and Selection.
This phase conducts an automated search using the above search items and filters the collected papers by inclusion/exclusion criteria.

* Snowballing Search.
This phase conducts a snowballing search to complement the final search results.

§.§ Search Items

To perform the manual search and establish QGS, we select four top-tier SE conferences (ICSE, ESEC/FSE, ASE, ISSTA) and two journals (TOSEM and TSE) and search for relevant papers involving both program repair and LLMs.
We manually inspect and identify 25 papers from 2020 to 2024 that meet our criteria, which then form the basis for constructing QGS.
Following prior surveys <cit.>, we divide the search items used for searching papers automatically into two groups: (1) an APR-related group containing some commonly-used keywords related to program repair;
and (2) an LLM-related group containing some popular keywords related to large language models or pre-trained models.
Besides, as the APR community benefits from well-maintained forums and real-time updated works, we further refine our search items by concluding some frequent keywords from three sources: a community-driven website[<http://program-repair.org/bibliography.html>], a living review of APR by Monperrus <cit.> and the most recent learning-based APR survey <cit.>.
Finally, we identify a search string that includes several LLM-related and APR-related keywords frequently appearing in program repair studies that use LLMs.
The complete set of search keywords is as follows:

(“program repair” OR “software repair” OR “automatic repair” OR “code repair” OR “bug repair” OR “bug fix” OR “code fix” OR “automatic fix” OR “patch generation” OR “patch correctness” OR “patch validation” OR “fix generation” OR “code transformation” OR “code edit” OR “fix error”)
AND (“LLM(s)” OR “Large Language Model(s)” OR “Pre-trained” OR “Pretrained” OR “Pre-training” OR “Pretraining” OR “PLM(s)” OR  “(Code)BERT” OR “(Code)T5” OR “(Code)GPT” OR “Codex” OR “ChatGPT” OR “(Code)Llama” OR “GPT-*” OR “neural” OR “machine” OR “deep” OR “learning” OR “transformer/transformers” OR “model/models” OR “transfer” OR “supervised”)

It is noteworthy that we also include the list of keywords related to machine/deep learning from Zhang et al. <cit.>, so as to avoid missing any papers related to our work as much as possible.

§.§ Study Collection and Selection

Inclusion criteria and exclusion criteria.

2lInclusion criteria

172     The paper utilizes an LLM in its framework.

173     The paper addresses the task of program repair.

174     The paper is accessible with full text.

175     The paper is only written in English.

2lExclusion criteria

182     The paper is fewer than seven pages.

183     The paper is an old conference version extended to journals with the same authors.

184     The paper uses repair methods to contribute to LLMs.

185     The paper is published as an SLR, review, or survey only mentioning the development of LLMs and APR.

186     The paper is published in a workshop or a doctoral symposium.

187     The paper is a grey publication, e.g., a technical report or thesis.

188     The paper falls into the category of short papers, tool demonstrations, and editorials.

Checklist of Quality Assessment Criteria (QAC) for LLM-based APR studies.

ID     Quality Assessment Criteria

QAC1     Is the LLM adopted as the research subject rather than only as a baseline method?

QAC2     Is the impact of the paper on the APR community clearly stated?

QAC3     Is the contribution of the paper clearly stated?

QAC4     Is the paper published in a high-repute venue?

QAC5     Is the relevant artifact made open source, e.g., datasets?

QAC6     Is there a clear motivation for the paper?

QAC7     Is the implementation of the proposed approach described clearly?

QAC8     Is the experimental setup described in detail, e.g., hyper-parameters and environments?

QAC9     Are the findings clearly supported by experimental results?

QAC10     Are the key contributions and limitations of the study discussed?

To perform the automated search, we collect potentially relevant papers by primarily searching the Google Scholar repository, ACM Digital Library, and IEEE Explorer Digital Library at the end of April 2024.
Once we gather the studies based on our automated search strategy, we proceed with a filtering and deduplication phase to eliminate papers that do not align with the study objectives.
First, we aim to filter out papers published before 2017, considering that the Transformer architecture <cit.>, which forms the foundation for LLMs, is introduced in 2017.
Second, we remove duplicate papers, as the same paper might be indexed by multiple databases.
Thirdly, we exclude papers with fewer than seven pages <cit.> (Exclusion criterion 182), resulting in a total of 283 papers.
Finally, we manually inspect the titles, abstracts, keywords, and venues of the papers to include relevant papers, according to our inclusion/exclusion criteria in Table <ref>, reducing the number of papers to 241.
It is worth noting that, about Exclusion criterion 186, we took into account International Automated Program Repair Workshop, considering its relevance to the APR field and high-quality publications.
Finally, we perform a full-text inspection to scrutinize the remaining papers and decide whether they are relevant and high-quality to the LLM-based APR field.
Following Wang et al. <cit.>, we answer ten questions to assess the relevance and quality of included papers, shown in Table <ref>.
Regarding QAC4, given the nascent nature of this field and the fact that many works, particularly those involving recently released LLMs, have not completed the peer review process, we consider papers from arXiv and select high-quality papers with the quality assessment process, to make our survey more comprehensive and up-to-date.
We obtain 110 papers that are related to our work.

§.§ Snowballing Search

To ensure maximal comprehensiveness of the collected papers, we additionally employ the snowballing search  <cit.> to manually incorporate any potential papers that are overlooked during our previous search process.
The snowballing search is a common practice to examine the references (i.e., backward snowballing) or citations (i.e., forward snowballing) within a list of papers to discover additional relevant works.
Specifically, we scrutinize every reference and citation within the collected papers to assess their relevance to our research objectives.
This meticulous manual analysis of all potential papers through the entire study selection process ultimately leads to the inclusion of  papers in our survey.

§ RQ1: WHAT IS THE TREND OF APR STUDIES THAT UTILIZE LLMS?

§.§ What are the trends of publication over time?

[Number of publications per year.]

[Cumulative number of publications per year.]

Publication trends of LLM-based APR studies.

We analyze the publication trends of APR studies empowered with LLMs between 2020 and April 2024.
Although the Transformer architecture <cit.> has been proposed in 2017, and the groundbreaking LLM, BERT <cit.>, has been introduced in 2018, we do not find any studies using LLMs to fix bugs before 2020.
Fig. <ref> illustrates the number of relevant studies, and we find that the number of publications from 2020 to 2023 shows a significant increase, with the number reaching 65 papers in 2023.
It should be noted that we collect papers as of April 2024; thus the number of relevant studies (i.e., 42 papers) in 2024 cannot reveal the overall trend of LLM-based APR in 2024.
We fit the number of publications as a power function to predict the publication trend in the last four years, and find that the slope of the curve fitting the distribution increases substantially between 2020 and 2023.
Particularly, the coefficient of determination attains the peak value ($R^2 = 0.973$), estimating that there may be over 90 relevant publications using various LLMs to solve bug-fixing during 2024.
We also calculate the cumulative number of publications as shown in Fig. <ref>.
We perform the same power function fit ($R^2 = 0.985$) and estimate more than 140 papers by the end of 2024, which indicates that the number of relevant studies using LLMs in APR intends to experience a strong rise in the future.
Overall, utilizing LLMs to address bug-fixing has become a prevalent trend since 2020, and huge numbers of studies will adopt LLMs to address the challenges of APR in the future.

Publication venues with LLM-based APR studies.

Acronym     Venues     Papers     Proportion

arXiv     N.A.      55        43.31%

ICSE      International Conference on Software Engineering     13        10.24%

ESEC/FSE     European Software Engineering Conference and International Symposium on Foundations of Software Engineering     9         7.09%

APR       International Automated Program Repair Workshop     6         4.72%

ICLR      International Conference on earning Representations     5         3.94%

ASE       International Conference on Automated Software Engineering     6         4.72%

TOSEM     Transactions on Software Engineering Methodology     4         3.15%

TSE       Transactions on Software Engineering     3         2.36%

ISSTA     International Symposium on Software Testing and Analysis     2         1.57%

AAAI      AAAI Conference on Artificial Intelligence     1         0.79%

ACL       The Association for Computational Linguistics     1         0.79%

COMPSAC     International Computer Software and Applications Conference     1         0.79%

GECCO     Genetic and Evolutionary Computation Conference     1         0.79%

ICSME     The International Conference on Software Maintenance and Evolution     1         0.79%

MSR       International Conference on Mining Software Repositories     1         0.79%

NeurIPS     Neural Information Processing Systems     1         0.79%

OOPSLA     Conference on Object-Oriented Programming Systems, Languages,and Applications     1         0.79%

PLDI      ACM SIGPLAN Conference on Programming Language Design\& Implementation     1         0.79%

PMLR      Proceedings of Machine Learning Research     1         0.79%

S&P      IEEE Symposium on Security and Privacy     1         0.79%

SIGCSE     The ACM Special lnterest Group on Computer Science Education     1         0.79%

SLE       ACM SIGPLAN International Conference on Software Language Engineering     1         0.79%

TDSC      IEEE Transactions on Dependable and Secure Computing     1         0.79%

TIFS      IEEE Transactions on Information Forensics and Security     1         0.79%

WWW       Proceedings of the ACM Web Conference     1         0.79%

Others     N.A.      8         6.30%

Total     N.A.      127       100.00%

§.§ What is the distribution of the publication venues?

We analyze  studies published in various publication venues.
Table <ref> lists the number of relevant papers published in each publication venue.
We find that 56.69% of papers are published in peer-reviewed venues, and ICSE and ESEC/FSE are the most popular venues that contain 13 papers, followed by APR (six papers), ICLR (five papers), and ASE (four papers).
Besides, the top popular venues are top-tier conferences and symposiums, and only four journals have at least one relevant publication, i.e., TOSEM (four papers), TSE (three papers), TDSC (one paper), and TIFS (one paper).
This finding indicates a preference for presenting the latest work at conferences due to the timeliness of conference proceedings.
These collected papers are published in different research fields, including SE (e.g., TOSEM and ICSE), AI (e.g., NeurIPS, AAAI and ICLR), NLP (e.g., ACL), Programming Language (e.g., OOPSLA and PLDI) and Security (e.g., S&P, TDSC and TIFS).
We also find that the remaining 43.31% of papers are non-peer-reviewed and published on arXiv.
This phenomenon can likely be attributed to the rapid emergence of relevant studies in a short development time.
Considering the varying quality levels of such non-peer-reviewed papers, we conducted a rigorous evaluation to ensure the inclusion of high-quality relevant papers in this work.

Distribution of LLM-based studies across different programming languages.

§.§ What is the distribution of the programming languages?

Fig. <ref> presents the distribution of different programming languages targeted by existing LLM-based APR techniques.
We find that Java is the most widely utilized programming language, accounting for 37% of all collected papers, followed by Python (24%) and C (11%).
The possible reason lies in the availability of mature datasets for these languages, which serve as recognized benchmarks for evaluating repair techniques, such as Defects4J <cit.>,
QuixBugs <cit.>, BFP <cit.>, CVEfixes <cit.>, Big-Vul <cit.>, and TFix <cit.>.
We also find that LLM-based APR encompasses a broader range of programming languages compared to traditional APR.
For example, our collected papers involve 18 different programming languages in total, whereas learning-based APR techniques are typically limited to only five languages <cit.>.
Importantly, we notice that some rare languages, previously overlooked in the APR community, are now being addressed.
This broader language adaptability of LLM-based APR might stem from the inherent capabilities of LLMs to encapsulate general programming knowledge that can be transferred across multiple languages by fine-tuning <cit.>.
Besides, LLMs' robust natural language understanding capabilities facilitate the few-shot or zero-shot repair settings with limited learning samples, which is a significant advantage over DL models <cit.> that typically require extensive repair corpora for training.
Consequently, LLMs can efficiently handle lesser-known programming languages, like Verilog <cit.> and Rust <cit.>, which are often underrepresented in previous APR research <cit.>.
This phenomenon highlights the promising prospects and scalability of APR brought about by recent LLMs.

§.§ What are the types of the publication contributions?

We categorize collected papers according to their main contributions into four categories: new technique or methodology, empirical study, benchmark, and human study, as illustrated in Fig. <ref>.
We find that 78 relevant papers are published, with the aim of proposing a novel repair approach or framework with LLMs to address various issues in the APR community.
Besides, 38 papers concentrate on conducting empirical studies to explore the actual benefits of LLMs in fixing various bugs, such as the potential of fine-tuning LLMs in vulnerability repair <cit.>.
We further notice nine relevant studies constructing new benchmarks to evaluate the performance of LLMs and two papers <cit.> administering a survey to offer insights into how practitioners or developers think about and employ LLMs to fix software bugs in practice.

(1) LLMs have shown a booming trend in fixing software bugs, with  papers between 2020 and 2024.
(2) The number of conference papers employing LLMs for APR significantly exceeds that of journal papers, with ICSE and TOSEM being the most popular conference and journal venues, respectively.
(3) LLM-based APR papers are published in different research fields, including SE, AI, and Security.
(4) There are  programming languages that LLM-based APR has been applied to, with Java, Python, C, and C++ being the most frequently targeted.
(5) LLMs have been applied to some underrepresented programming languages, such as Verilog and Rust.
(6) The vast majority of collected studies primarily focus on introducing new techniques and conducting empirical research, while two papers perform user studies to understand practitioners' attitudes and experiences regarding leveraging various LLMs for solving bug-fixing tasks.

§ RQ2: WHICH POPULAR LLMS HAVE BEEN APPLIED TO SUPPORT APR?

Distribution of existing LLMs utilized in APR.

Distribution of three ways to utilize LLMs for APR.

§.§ What is the distribution of involved LLMs?

Plenty of LLMs, with millions to billions (even more) of parameters, have been proposed and adapted to automatically fix software bugs.
We analyze all collected papers and summarize 46 relevant LLMs that are already utilized in existing APR studies.
Fig. <ref> illustrates the variety of different LLMs and lists the number of times these models have been utilized by prior APR studies.
We only include those LLMs with more than one time in collected papers due to page limitations.
As can be seen from Fig. <ref>, ChatGPT (37) and GPT-4 (25) from OpenAI are the two most popular LLMs, followed by CodeT5 (23) and Codex (21).
We categorize these LLMs into three groups according to their architectures: encoder-only, encoder-decoder, and decoder-only LLMs.

§.§.§ Encoder-only LLMs

Encoder-only LLMs denote a category of LLMs that only utilizes the encoder stack of the Transformer architecture.
Typically, these models are pre-trained on a massive corpus using a Masked Language Modeling (MLM) task, which is used to learn to predict the identity of masked words based on their context.
In the APR community, prominent models like CodeBERT (11) <cit.> and GraphCodeBERT (6) <cit.> have been investigated in semantic bugs <cit.> and security vulnerabilities <cit.>.
As such LLMs only contain an encoder component that is capable of generating context-aware representations for inputs, they are particularly suited for code understanding tasks, such as code search, while not directly applicable to code generation tasks, such as program repair.
Thus, in the APR community, as Zhang et al. <cit.> mention, researchers may need to integrate a new decoder that initializes from scratch to the pre-trained encoder to construct an encoder-decoder architecture for patch generation.
We also notice such encoder-only LLMs are utilized to identify patch correctness <cit.>, as discussed in Section <ref>.

§.§.§ Enocoder-decoder LLMs

Encoder-decoder LLMs denote a category of LLMs that utilize both the encoder and decoder stacks of the Transformer architecture, thus inherently suitable for transforming one sequence into another.
Particularly, the encoder takes one sequence as input and encodes it into a fixed-size hidden state, which effectively captures the semantics and meaning of the input sequence.
Then the decoder processes the hidden state and produces the corresponding output sequence using attention mechanisms to refer back to parts of the input sequence as needed.
Thanks to the encoder-decoder architecture, such LLMs are particularly suited for code generation tasks in a sequence-to-sequence learning setting, such as program repair.
In the APR community, a mass of encoder-decoder LLMs have been widely adopted, such as CodeT5 (23) <cit.>, PLBART (10) <cit.>, UniXcoder (4) <cit.> and T5 (3) <cit.>.
Similar to traditional learning-based APR <cit.>, such studies usually treat APR as a natural machine translation (NMT) task by supervised sequence-to-sequence learning, such as CIRCLE <cit.>, TFix <cit.>, VulRepair <cit.> and RAP-Gen <cit.>.

§.§.§ Decoder-only LLMs

Decoder-only LLMs denote a category of LLMs that utilize only the decoder stack of the Transformer architecture.
Decoder-only LLMs are typically pre-trained using a causal language modeling (CLM) objective, learning to predict the next word in a sentence based on the preceding words.
These models are specifically designed for generating sequences of text autoregressively, i.e., producing one token at a time and using what has been generated so far as context for subsequent tokens.
In the APR community, decoder-only LLMs are the most popular and widely used group, compared with encoder-only and encoder-decoder LLMs.
Notable repair applications of decoder-only LLMs include GPT-series models (e.g., GPT-1 <cit.>, GPT-2 <cit.>, GPT3 <cit.>, GPT-3.5 <cit.>, ChatGPT <cit.>, and GPT-4 <cit.>), some open-sourced models, (e.g., CodeGPT <cit.>, GPT-Neo <cit.>, GPT-NeoX <cit.>, GPT-J <cit.>, InCoder <cit.>, CodeGen <cit.>, CodeLLaMA <cit.> and StarCoder <cit.>), as well as some closed-source models (e.g., Codex <cit.> in Fan et al. <cit.> and CEDAR <cit.>).
The emergence of decoder-only LLM-based APR studies is primarily due to two reasons.
The first reason is that these models can naturally perform program repair from a few examples or simple instructions without any fine-tuning.
The second reason is the recent surge in decoder-only LLMs, marked by the introduction of commercial products by leading Internet companies, such as ChatGPT and GPT-4 by OpenAI.

§.§ What approaches are employed to optimize LLMs for program repair?

LLMs typically acquire general knowledge from extensive datasets.
Thus, a fundamental research issue arises when integrating off-the-shelf LLMs with APR: how to adapt general-propose LLMs to the specific program repair task.
Fig. <ref> displays the prevalence of three common adaptation strategies in LLM-based APR research: fine-tuning, few-shot learning, and zero-shot learning.
Our findings indicate that zero-shot learning, employed in 48% of the studies, is the most popular approach, suggesting a trend towards using LLMs as-is for program repair tasks.
Meanwhile, fine-tuning is utilized in 37% of the cases, followed by few-shot learning at 15%.

§.§.§ Fine-tuning

Fine-tuning refers to a process where LLMs are further trained on a smaller, task-specific dataset.
This is an intuitive way to allow LLMs to adjust their weights and biases through supervised learning, enabling them to perform as expected on new tasks that are similar to, but not exactly the same as, those they were initially trained on.
In the APR community, fine-tuning is widely utilized during the early emergence of LLMs with millions of parameters, such as T5 and CodeT5, as it can significantly improve performance on the target program repair task without the need to train an LLM from scratch.

We summarize the existing APR studies via fine-tuning LLMs into three stages.
First, researchers directly regard program repair as a downstream task by specific datasets, such as fine-tuning T5 <cit.>, CodeT5 <cit.>, CodeBERT <cit.>, and GPT-2 <cit.>.
Second, researchers utilize more advanced fine-tuning strategies for better performance.
For example, CIRCLE utilizes continual learning to repair multiple languages with a single model, and RAP-Gen <cit.> utilizes retrieval-augmented generation to guide patch search space.
Recently, Zirak et al. <cit.> empirically explore the domain shift problem in APR with two LLMs, i.e., TFix and CodeBERT, and three fine-tuning methods, i.e., Full-Fine-Tuning, Tuning-With-Light-Weight-Adapter-Layers, and Curriculum-Learning.
Third, researchers conduct empirical studies to explore the actual fix capabilities of various LLMs in different repair scenarios.
For example, Zhang et al. <cit.> fine-tune five LLMs to repair C/C++ security vulnerabilities, Wu et al. <cit.> involve four LLMs in Java vulnerabilities, and Jiang et al. <cit.> consider four LLMs on Java semantic bugs.

§.§.§ Few-shot Learning

Few-shot learning refers to the ability of LLMs to learn or adapt to new tasks with a very limited amount of data—often only a few examples.
This is an effective way to use examples to help LLMs understand the targeted task and generate appropriate responses without any explicit retraining or fine-tuning.
In the APR community, few-shot learning is usually utilized and impressive in LLMs, particularly with billions of parameters, as it requires LLMs' powerful ability to generalize from very limited data.
Researchers typically provide LLMs with a small number of repair examples directly in the input prompt and require LLMs to generate correct patches.
For example, Nashid et al. <cit.> construct effective prompts by retrieving similar repair demonstrations for CodeX, and Xia et al. <cit.> provide LLMs with examples from the same buggy project to learn the coding style.

§.§.§ Zero-shot Learning

Zero-shot learning takes the concept of few-shot learning even further by requiring LLMs to perform program repair without any explicit examples.
This is a recently popular way to query LLMs to perform a variety of unseen tasks, where LLMs are given a task description and must use their pre-existing knowledge and understanding to generate a response or solution.
In the APR community, zero-shot rapidly emerges following the advent of LLMs with record-breaking parameters, exemplified by ChatGPT, as it requires a powerful foundation model to perform human-like chats.

There are two typical development routes that utilize LLMs for program repair in a zero-shot learning setting.
The first one is cloze-style repair, i.e., reframing program repair as a cloze-style task, and then invoking LLMs to predict partially correct code with the help of repair patterns, such as AlphaRepair <cit.>, GAMMA <cit.>, FitRepair <cit.> and Repilot <cit.>.
The second one is conversational-based repair, i.e., constructing complex prompts with various valuable information (e.g., buggy code, failure diagnostics, even execution feedback), and then chatting with LLMs to generate correct patches, such as Pearce et al. <cit.>, TypeFix <cit.>, RustAssistant <cit.>, Zhang et al. <cit.>, Prenner et al. <cit.>, Sobania et al. <cit.>, and Napoli et al. <cit.>.
Such repair routes usually require LLMs capable of processing long-text prompts and human-like conversations, thus predominantly employing powerful LLMs with billion-level parameters, like ChatGPT and GPT-4.
Besides, zero-shot gets rid of training datasets, thus generalizing to various repair scenarios where gathering training data is challenging or impossible, such as hardware bugs <cit.>, DL programs <cit.> and crash bugs <cit.>.

Overall, fine-tuning, which uses supervised learning to adapt LLMs more closely to the specifics of program repair, becomes popular with the emergence of early million-parameter-level models like CodeT5.
Few-shot learning demonstrates LLMs' capability to generalize from a few examples, popular with the emergence of subsequent billion-parameter-level models like Codex.
Zero-shot learning showcases LLMs' capability to tackle the program repair task without any prior direct exposure to training or examples, particularly with the emergence of recent models with tens or hundreds of billions of parameters like ChatGPT and GPT-4.

(1) We summarize different LLMs already utilized to fix bugs, and these LLMs can be classified into three categories based on model architectures, i.e., encoder-only, encoder-decoder, and decoder-only.
(2) Decoder-only LLMs are the most frequently utilized model architecture, and four of the top popular LLMs are decoder-only models.
(3) ChatGPT, GPT-4, CodeT5, and Codex are the most popular LLMs in existing LLM-based APR studies, utilized 37, 25, 23, and 21 times, respectively.
(4) We summarize three typical ways of leveraging the vast knowledge encapsulated in LLMs for the specific program repair task, i.e., fine-tuning, few-shot, and zero-shot.

Distribution of LLM-based APR papers across different bug types.

§ RQ3: WHAT REPAIR SCENARIOS HAVE BEEN FACILITATED BY LLMS?

In this section, we conduct a comprehensive analysis of the utilization of LLMs in various repair scenarios.
We categorize  existing repair scenarios addressed with LLMs, following the taxonomy from Monperrus et al. <cit.> and Zhang et al. <cit.>, including semantic bugs, syntax bugs, and static warnings.
Fig. <ref> presents the distribution of repair studies across different repair scenarios that LLMs are applied to.
We find that semantic bugs triggered by test suites have attracted the highest attention from the LLM-based APR research, constituting approximately 48% of the total research volume.
This phenomenon is also observed in traditional learning-based APR <cit.>, and is mainly driven by well-known benchmarks, such as Defects4J <cit.>.
Security vulnerabilities account for about 14% of the research proportion, highlighting the significance of LLMs in addressing cybersecurity challenges.
The third highest number of studies is observed in the programming problems domain, constituting approximately 9% of the total research volume.
We also find a growing interest in rare bug types that are usually ignored by prior work, such as static warnings (7%), syntax errors (6%), and hardware bugs (3%).
This underscores that, thanks to LLMs' general knowledge gleaned from vast amounts of data, researchers have begun to explore repair scenarios not previously addressed in prior works.

§.§ Semantic Bugs

Semantic bugs refer to logical errors in a syntactically correct program when the code does not do what the programmer intends, even though it may compile and run without syntax errors.
Considering that the majority of LLM-based APR studies are concentrated in the semantic bug field, we summarize these representative studies based on three ways of using LLMs.

§.§.§ Repair with Fine-tuning

As early as 2021, Drain et al. <cit.> introduce DeepDebug, which learns to fix bugs in Java methods mined from real-world GitHub repositories through fine-tuning BART on the BFP <cit.> dataset.
In 2022, Yuan et al. <cit.> propose CIRCLE, an LLM-based multi-lingual APR framework by fine-tuning a pre-trained T5 with continual learning.
Particularly, CIRCLE utilizes a difficulty-based rehearsal strategy to achieve lifelong learning without access to all historical data and elastic regularization to resolve catastrophic forgetting.
CIRCLE is the first approach to repair multiple programming language bugs with a single repair model in the continual learning setting.
Hao et al. <cit.> propose APRFiT by fine-tuning CodeT5 and GraphCodeBERT with a curricular learning framework.
In 2023, RAP-Gen <cit.> further improves the repair performance of fine-tuning LLMs via a retrieval-augmented generation framework.
RAP-Gen first utilizes a hybrid patch retriever to search for a relevant fix pattern from an external codebase, and utilizes Codet5 as the foundation model to synthesize candidate patches with the augmented inputs, i.e., the retrieved fix pattern and original buggy code.
Recently, Silva <cit.> escalate the parameters of fine-tuned LLMs to the billion level, and propose RepairLLaMA based on CodeLlama-7B.
RepairLLaMA utilizes a parameter-efficient fine-tuning technique called LoRA to train repair adapters with appropriate code representations.

§.§.§ Repair with Few-shot Learning

In few-shot learning, LLMs take a prompt as input , which contains natural language instructions, a few examples of task demonstration, and a query, and generate an output.
In 2023, Nashid et al. <cit.> propose a CEDAR, retrieval-based prompt selection approach for several code generation tasks, including program repair.
CEDAR first automatically retrieves bug-fixing demonstrations relevant to the buggy code based on embedding or frequency similarity, and creates prompts to query Codex to generate correct patches in a few-shot learning manner.
Meanwhile, Do et al. <cit.> conduct a preliminary study with ChatGPT and  Galpaca-30b (a variant of LLaMA) by using a few-shot example generation pipeline.

§.§.§ Repair with Zero-shot Learning

Unlike the above fine-tuning-based APR work, which relies on high-quality bug-fixing code pairs, some approaches are proposed to directly leverage off-the-shelf LLMs without any training.
Such approaches attempt to generate patches in a zero-shot setting and can be categorized into two types, namely cloze-style APR and conversation-style APR.

Cloze-style APR leverages the pre-training objective (e.g., model language modeling) of LLMs to predict correct code tokens with the help of repair patterns.
In 2022, Xia et al. <cit.> introduce AlphaRepair, a cloze-style APR tool to directly query LLMs to generate patches in a zero-shot learning setting.
AlphaRepair first replaces the buggy line with a mask line and queries CodeBERT to fill the mask line to generate candidate patches.
Furthermore, FitRepair <cit.> is a more advanced cloze-style APR approach based on CodeT5 and the plastic surgery hypothesis.
FitRepair utilizes knowledge-intensified fine-tuning and repair-oriented fine-tuning to help CodeT5 learn the buggy project-specific knowledge and the cloze-style task knowledge, respectively.
FitRepair then retrieves relevant identifiers with static analysis, which are fed into fine-tuned CodeT5 to generate candidate patches.
Similarly, Repilot <cit.> improves the cloze-style APR AlphaRepair with a completion engine.
Repilot builds an interaction between LLMs and a completion engine to generate more valid patches by first pruning away infeasible tokens suggested by LLMs and then completing the token based on the suggestions provided by the completion engine.
GAMMA <cit.> further explores the potential of using LLMs to generate patches in a zero-shot learning scenario with a list of well-summarized fix templates.
Particularly, GAMMA attempts to address the donor code retrieval issue of traditional template-based APR (e.g., TBar <cit.>) and regards patch generation as a fill-in-the-blank task by querying LLMs to predict the correct code for masked tokens in a pre-defined fix pattern.
Unlike the cloze-style APR, Ribeiro et al. <cit.> frame the APR problem as a code completion task and apply CodeGPT to fix bugs from ManySStuBs4J <cit.>.

Conversation-style APR leverages the powerful natural language and programming language understanding capabilities of LLMs to generate patches iteratively using test failure information.
In 2023, Xia et al. <cit.> propose ChatRepair, the first conversation-driven APR approach that interleaves patch generation with instant feedback to perform APR in a dialogue manner.
ChatRepair constructs prompts with test failure information and queries ChatGPT to generate correct patches from previously incorrect and plausible patches.
However, ChatRepair mainly relies on negative feedback (i.e., failure information derived from failing tests) to guide the conversations, which may not always offer specific and adequate prompts for an effective repair.
Thus,  Kong et al. <cit.> introduce ContrastRepair, which includes positive feedback from passing tests to supplement the negative feedback.
Given a buggy program and a failing test case, ContrastRepair generates a similar passing test case by making minimal modifications to the failing test case.
ContrastRepair then constructs a contrastive pair to LLMs, allowing them to better pinpoint the root cause of the bug and generate accurate patches.
In the above conversation-style APR scenarios, LLMs take a prompt containing some tokens about buggy code as the input and infer the following tokens about patches as the output.
During the conversations, all tokens in the input prompt and output answer consume the computational and financial costs, such as $0.06 per 1k input tokens and $0.03 per 1k generated tokens for GPT-4.
To reduce the computational cost of ChatRepair, Hidvegi et al. <cit.> propose CigaR, a token-efficiency LLM-based APR approach that concentrates on token cost minimization of ChatGPT.
CigaR designs three prompts to help ChatGPT minimize the overall token cost with previous responses, including (1) an initiation prompt to initialize the repair process, (2) an improvement prompt to refine partial patches, avoiding discarding potentially valuable patches, and (3) a multiplication prompt builds upon the already generated plausible patches to synthesize more plausible patches with diversity maximization.
Unlike previous work relying on a fixed prompt template, RepairAgent <cit.> treats ChatGPT as an agent capable of autonomously planning and executing actions to generate patches by using dynamic prompts and a state machine to select suitable tools.

§.§.§ Empirical Study

In addition to the above novel techniques, researchers have conducted numerous empirical studies to explore the capabilities of LLMs in fixing semantic bugs.
As early as 2021, Mashhadi et al. <cit.> preliminarily evaluate the performance of fine-tuning CodeBERT for fixing Java bugs from ManySStuBs4J.
Lajko et al. <cit.> empirically fine-tune GPT-2 to automatically generate candidate patches for JavaScript bugs.
Prenner et al. <cit.> investigate CodeX in fixing  QuixBugs with a zero-shot setting and Sobania et al. <cit.> utilizes a more powerful LLM ChatGPT with prompt engineering to generate patches for QuixBugs.
In 2023, Horvath et al. <cit.> explore the impact of model architectures and program representations, involving two popular programming languages, i.e., Java and JavaScript, three different code representations, i.e., raw text, command sequences, and ASTs, and four LLMs, i.e., T5, CodeT5, RoBERTa, and GPTNeo.
Meanwhile, Zhao et al. <cit.> conduct a comprehensive investigation into the effective utilization of LLMs for repairing code review bugs, involving two LLMs (i.e., ChatGPT and GPT-4) in a zero-shot learning manner, and six LLMs (InCoder, CodeT5+, CodeFuse, LLaMA, CodeGen-2, CodeLLaMA) in a fine-tuning manner.
Recently, inspired by the phenomenon that grammatical errors in natural language can be fixed by round-trip translation, i.e., translating sentences to another intermediate language and then back, Ruiz et al. <cit.> investigate to what extent the RTT pipeline can fix bugs in code in a zero-shot fashion.
The comprehensive study involves eight LLMs: PLBART, CodeT5, TransCoder, SantaCoder, InCoder, StarCoderBase, GPT-3.5, and GPT-4, and four APR benchmarks: Defects4J-v1.2, Defects4J-v2.0, QuixBugs, and HumanEval-Java.

Furthermore, more comprehensive empirical studies are published in top-tier SE conferences.
Xia et al. <cit.> conduct an comprehensive study to explore the performance of LLMs in program repair, involving nine LLMs from two categories (i.e., infilling and generative models), and five datasets across three programming languages.
They explore three ways to use LLMs for patch generation: complete function generation, correct code infilling, and single-line generation.
Meanwhile, Jiang et al. <cit.> empirically explore the fixing capabilities of ten variants from four LLMs under zero-shot and fine-tuning settings, involving four Java benchmarks.
They also construct a new benchmark, HumanEval-Java, which none of the LLMs has seen during training to address the data leakage issue.
Huang et al. <cit.> conduct an empirical study on fixing capabilities of LLMs in the fine-tuning paradigm, involving five LLMs, three programming languages, and three repair scenarios.

§.§ Security Vulnerabilities

A software vulnerability refers to a flaw, bug, or weakness in the design, implementation, operation, or management of software.
When exploited, vulnerabilities can lead to unauthorized access, manipulation, or disruption of the software's intended functionality.
In 2022, Fu et al. <cit.> propose VulRepair, an LLM-based automated program repair approach by fine-tuning CodeT5 with vulnerability datasets.
Furthermore, in 2023, considering that a vulnerable function only contains a few core elements that need to be fixed, Fu et al. <cit.> propose VQM, an LLM-based automated vulnerability repair approach that leverages VIT-based approaches for object detection to help CodeT5 focus more on vulnerable code areas during patch generation.
Inspired by the relationship between semantic bugs and security vulnerabilities, Zhang et al. <cit.> propose an enhanced LLM-based vulnerability framework based on transfer learning, demonstrating superior performance against VulRepair.
In 2024, Zhou et al. <cit.> propose VulMaster, a CodeT5-based approach to address several limitations of VulRepair, i.e., (1) handling lengthy vulnerable code by the Fusion-in-Decoder framework; (2) capturing the structure of the vulnerable code by AST; and (3) understanding the expert vulnerability knowledge by the CWE system.
De et al. <cit.> leverage Quantized Low-Rank Adaptation (QLoRA) <cit.> to fine-tune two advanced 7-billion-parameter LLMs, i.e., CodeLlama <cit.> and Mistral <cit.>, to fix C vulnerabilities.

Empirical Study.
Pearce et al. <cit.> evaluate the performance of LLMs in repair vulnerabilities in a zero-shot manner, involving five commercially available black-box and two open-source LLMs.
They particularly investigate the potential of designing prompts that query LLMs to patch vulnerable code.
Furthermore, Zhang et al. <cit.> conduct an empirical study investigating the performance of fine-tuning LLMs for vulnerability repair, involving five LLMs from three categories, two C vulnerability datasets and more than 100 trained LLMs, which is the largest set to date.
They also explore the potential of ChatGPT in repairing security vulnerabilities in a zero-shot manner.
Similarly, Wu et al. <cit.> explore the fixing capabilities of five LLMs in a zero-shot manner and four LLMs in a fine-tuning manner on two real-world Java vulnerability datasets.
Recently, Le et al. <cit.> conduct a preliminary study of ChatGPT and Bard in detecting and fixing security vulnerabilities in JavaScript programs.

§.§ Static Warnings

Static warnings refer to automated alerts generated by static tools that examine code to identify potential errors, inefficiencies, or security vulnerabilities without executing the program.
In 2021, Berabi et al. <cit.> introduce TFix, which is considered a pioneer in incorporating LLMs into APR.
Inspired by T5, TFix formulates the program repair problem as a text-to-text task on code sequence, i.e., given a coding error as text, predicting a new text as the patch.
TFix is particularly fine-tuned from T5 with a high-quality dataset consisting of  100k bug-fixing pairs across 52 error types detected by a static analyzer, ESLint.
Similar to RAP-Gen <cit.>, InferFix <cit.> employs a retrieval-augmented generation manner to fine-tune LLMs for program repair, i.e., (1) utilizing a retriever to search for semantically equivalent bugs and corresponding fixes from an external codebase, and (2) fine-tuning LLMs on supervised bug-fixing datasets with prompts augmented via adding retrieved similar fixes.
However, RAP-Gen targets semantic bugs while InferFix repairs static errors detected by the static analyzer Infer; and RAP-Gen utilizes CodeT5 while InferFix has a significantly larger model Codex with 12 billion parameters.
Recently, Wadhwa et al. <cit.> present CORE to resolve code quality issues flagged by static analysis tools.
CORE utilizes ChatGPT as the proposer LLM to generate candidate code revisions and GPT-4 as the ranker to rank the candidate revisions before presenting them to the developer.
Alrashedy et al. <cit.> introduce FDSP to resolve repair security vulnerabilities detected by a static code analysis tool, Bandit <cit.>, in a self-debugging manner.

In addition to the above technical papers, Kim et al. <cit.> empirically investigate the performance of TFix in fixing errors from industrial Samsung Kotlin projects detected by a static analysis tool SonarQube.
Mohajer et al. <cit.> conduct a more comprehensive study of LLMs in the static code analysis domain, and propose SkipAnalyzer, an LLM-based powered tool to perform three related tasks: detecting bugs, filtering false positive warnings, and patching the detected bugs.

§.§ Syntax Errors

Syntax errors refer to parsing mistakes that occur when code does not follow the rules or grammar of the programming language, such as invalid statements and expressions.
These errors are detected by the compiler or interpreter during the parsing phase before the program is executed.
Ahmed et al. <cit.> propose SYNSHINE to fix syntax errors by pre-training and fine-tuning RoBERTa with compiler diagnostics.
RustAssistant <cit.> utilizes GPT-3.5 and GPT-4 to fix Rust compilation errors with prompt engineering.
RING <cit.> is a multilingual syntax error repair approach powered by Codex and few-shot learning and is proven promising in six different languages.
PCR-Chain <cit.> attempts to resolve fully-qualified names and fix last-mile syntax errors based on ChatGPT and Chain-of-Thought Prompting.

§.§ Type Errors

Type errors occur when an operation or function is applied to an object of an inappropriate type.
This indicates that the operation expects a certain data type but receives another, thus leading to an exception.
These errors are prevalent in dynamically typed languages such as Python and JavaScript, which do not allow operations that are inherently impossible with the given data types.
In 2024, Chow et al. <cit.> construct PyTyDefects, a dataset consisting of 2,766 type error-fix pairs from 176 GitHub repositories.
They also introduce PyTy, a T5-based repair technique that fine-tunes the off-the-shelf LLM TFix with PyTyDefects for fixing type errors in Python.
Unlike the fine-tuning-based approach PyTy, TypeFix <cit.> is a prompt-based repair approach for Python type errors in a zero-shot manner.
Similar to GAMMA, TypeFix queries LLMs to generate patches by filling the masks in code prompts using fix templates.
However, TypeFix distinguishes itself by automatically mining these templates through a hierarchical clustering algorithm, rather than relying on predefined ones in GAMMA.
Furthermore, Ribeiro et al. <cit.> present Mentat, a type error repair technique for OCaml programs powered by GPT-3.
Mentat first analyzes the source code to generate contextually relevant prompts, then exploits GPT-3’s advanced language understanding and generation capabilities to produce potential patches.

§.§ Programming Problems

This type of bug refers to incorrect solutions to programming problems from competition platforms, such as LeetCode.
In 2022, Zhang et al. <cit.> propose MMAPR to repair both syntactic and semantic mistakes from introductory Python programming through Codex and few-shot learning.
Fan et al. <cit.> systematically evaluate whether repair techniques can fix the incorrect solutions produced by LLMs in LeetCode contests.
Similarly, Liu et al. <cit.> analyze the correctness of ChatGPT-generated code in Java and Python for 2,033 programming tasks from LeetCode and query ChatGPT to mitigate incorrect code with different prompts.
Recently, Zhang et al. <cit.> empirically evaluate the performance of ChatGPT in fixing incorrect submissions from programming problems.
Similar studies are conducted by Haque et al. <cit.> and Tian et al. <cit.>.
Ishizue et al. <cit.> combine two LLMs (i.e., ChatGPT and GPT-4) with Refactory <cit.> to fix programming assignment problems.
They leverage LLMs to repair programs with syntax errors and those that Refactory fails to repair.
Recently, Zhao et al. <cit.> explore how LLMs, including ChatGPT, StarCoder, and CodeLlama, perform in repairing student assignments from higher-level programming courses.
They construct a student assignment dataset named Defects4DS, which contains 682 submissions from 4 programming assignments, and introduce a repair framework named PaR, which generates patches by selecting peer solutions to create prompts.

§.§ Performance Bugs

Performance bugs refer to inefficient code snippets that do not interfere with functionality but lead to unnecessary time and resource consumption.
Garg et al. <cit.> propose DeepDev-PERF, the first LLM-based approach to repair performance bugs for C# applications by fine-tuning the BART model.
Building upon DeepDev-PERF, RAPGen <cit.> further enhances the repair process through prompt engineering, which demonstrates superior efficiency over traditional intensive fine-tuning methods.
RAPGen retrieves relevant instructions from a knowledge base, constructs an input prompt tailored to the specific bug, and then uses Codex to generate the optimized code.
This approach is similar to CEDAR <cit.> yet uniquely leverages prompt engineering to achieve more effective and resource-efficient solutions.

§.§ Hardware Bugs

APR is well-explored in the software domain, while the research is less mature for hardware description languages, such as Verilog.
Ahmad et al. <cit.> empirically explore the potential of LLMs to repair security-relevant bugs in hardware designs.
They leverage prompt engineering to query Codex, ChatGPT, GPT-4, and CodeGen for 15 security-related bugs from 10 CWEs.
Inspired by the retrieval-augmented generation, Tsai et al. <cit.> introduce RTLFixer, a framework to fix Verilog syntax errors with LLMs and iterative prompting.
RTLFixer formulates an input prompt to query GPT-4 to generate the correct code.
RTLFixer then utilizes error logs from the compiler and human guidance from a retrieval database as feedback to perform an interactive debugging loop until all errors are resolved.
Yao et al. <cit.> from Huawei focus on the domain of chip design and propose HDLdebugger, an LLM-assisted hardware description language debugging framework.
This framework encompasses data generation through reverse engineering, a search engine to enhance the retrieval-augmented generation, and a fine-tuning approach to train retrieval-augmented LLMs.
Similarly, Fu et al. <cit.> focus on the hardware design iteration process and introduce LLM4SECHW, an LLM-based hardware debugging framework.
LLM4SECHW constructs a hardware debugging-oriented dataset from open-source hardware projects and fine-tunes a suite of hardware domain-specific LLMs capable of automatically reading hardware designs and fixing bugs.

§.§ Smart Contracts

Smart contracts are self-executing contracts on a blockchain network with the terms of the agreement directly written into code.
In 2023, Napoli et al. <cit.> conduct a preliminary study to explore the capabilities of ChatGPT in fixing vulnerable smart contracts with 143 vulnerable Solidity codes.
In 2024, Zhang et al. <cit.> introduce ACFIX, a GPT-based approach to repair access control vulnerabilities in smart contracts based on static code slicing and Chain-of-Thought Prompting.
ACFIX mines common AC practices for major categories of code functionality.
These practices form a knowledge base that guides LLMs in repairing code with analogous functionalities.

§.§ Misc Repair Types

We summarize other repair scenarios that are covered in fewer than two papers as follows.

Crash Bugs.
Crash bugs are considered critical issues as they might
cause unexpected program behaviors and termination.
Du et al. <cit.> conduct the first investigation into ChatGPT's capability to resolve real-world crash bugs, particularly code-related and environment-related ones.
They design different prompts and propose IntDiagSolver, an interaction approach that involves continuous interaction with LLMs.

API Misuse.
Zhang et al. <cit.> conduct an empirical study to explore the performance of learning-based APR techniques on API misuse repair, involving nine LLMs with millions of parameters under a fine-tuning setting, i.e., CodeBERT, GraphCodeBERT, CodeGPT, PolyCoder-160M, PolyCoder-0.4B, CodeTrans, PLBART, CodeT5 and UniXcoder.

Web UI.
Xu et al. <cit.> conduct the first feasibility study on integrating traditional Web UI test repair techniques with ChatGPT for enhancing Web UI test repair.
They first utilize traditional repair approaches to generate a preliminary list of candidate-matched elements and employ ChatGPT to execute a global matching to select the best-matched element further.

Translation Bugs.
Code translation refers to the process of converting source code from a source language into another language without changing the original program's functionality or logic.
Pan et al. <cit.> present a large-scale empirical study to investigate the ability of LLMs for code translation across five different languages, including C, C++, Go, Java, and Python.
They provide a bug taxonomy of unsuccessful translations from 15 categories and five groups, and design an iterative translation bug repair approach to fix unsuccessful translations from LLMs with a set of heuristics for prompt crafting.

Test Repair.
Yaraghi et al. <cit.> introduce TARGET, an LLM-based approach to repair broken test cases by treating test repair as a language translation task.
They first construct TARBENCH, a comprehensive benchmark containing 45,373 broken test repairs across 59 open-source projects.
They then fine-tune off-the-shelf LLMs (i.e., PLBART, CodeGen and CodeT5+) to generate correct test code with essential code context.

Motion Planning Algorithm.
In the automated vehicles scenario, notion planners refer to algorithms that are responsible for computing a viable path for the vehicle to travel from an initial state to a designated goal region within a predefined time frame.

Lin et al. <cit.> introduce DrPlanner, the first automated framework that utilizes GPT-4 to diagnose and repair motion planners.
DrPlanner first designs a structured prompt to describe the planners using both natural and programming languages.
DrPlanner then queries GPT-4 to repair generate algorithms with continuous diagnostic feedback in a closed-loop manner.

Software Formal Proof.
Formal software verification aims to validate the correctness of software properties.
First et al. <cit.> introduce Baldur, an LLM-based approach to automate the generation and repair of whole formal proofs.
Baldur leverages two versions of Minerva <cit.>, which is pre-trained on a mathematics corpus based on the PaLM <cit.>: one with eight billion parameters and another with 62 billion parameters.
Baldur first fine-tunes a generation model to synthesize whole proofs for theorems on a proof dataset, and then fine-tunes a repair model based on the proof assistant's error messages to repair incorrectly generated proofs.

Github Issue.
Unlike most prior work <cit.>, which evaluates LLMs in fixing self-contained problems, such as programming problems, Jimenez et al. <cit.> explore the potential of LLMs to resolve GitHub issues in a realistic software engineering setting.
Given an issue (such as a bug report or a feature request) submitted to popular GitHub Python repositories, they fine-tune CodeLlama-7B and CodeLlama-13B to generate a patch that passes the unit and system tests.

Code Review Refinement.
Guo et al. <cit.> conduct the first empirical study to explore the potential of ChatGPT in code review, specifically focusing on automated code refinement based on existing code reviews.
They leverage prompt engineering to compare ChatGPT with CodeReviewer <cit.> using two datasets: an established one named CodeReview <cit.> and a newly introduced one named CodeReview-New.
They also design several strategies to improve the performance of ChatGPT, such as using more advanced models.

Overall, we observe that LLMs have been applied in a wide array of repair scenarios in the literature, involving  bug types.
In some common scenarios dominated by traditional APR, such as semantic bugs, researchers continue to invest substantial efforts in investigating the application of LLMs.
Besides, thanks to LLMs' general knowledge learned from all possible Internet data, LLM-based APR has been extended to some rare scenarios that are previously unexplored, such as hardware bugs and Web UI.

§ RQ4: WHAT KEY FACTORS CONTRIBUTE TO THE INTEGRATION OF LLMS FOR APR?

Distribution of publications across datasets.

Distribution of publications across input forms.

§.§ What sources of datasets are used to evaluate LLM APR studies?

Benchmarking is crucial for evaluating the performance of APR techniques, fundamentally shaping the direction of development within the research community.
For example, Defects4J <cit.> has been the common practice over the last decade, facilitating a standard comparison scenario to understand the strengths and weaknesses of proposed approaches and guiding researchers towards addressing the most pressing challenges in the field.
We identify a total of  benchmarks from all collected studies and present the benchmarks utilized more than once in Fig. <ref>.

We find that Defects4J <cit.> (28 papers) is the most popular benchmark, followed by QuixBugs <cit.> (20 papers) and BFP <cit.> (12 papers), all of which are existing benchmarks from previous APR studies.
This phenomenon is reasonable, as existing benchmarks are usually well-constructed and have been inspected and utilized by the community, making them the preferred choice for evaluating new APR techniques.
For example, a mass of LLM-based APR techniques are evaluated with existing popular benchmarks, such as CIRCLE <cit.>, AlphaRepair <cit.>, and GAMMA <cit.>.

Besides, we notice some newly constructed benchmarks in LLM-based APR, such as TFix <cit.> (six papers), HumanEval-Java <cit.> (five papers), and Pearce et al. <cit.> (two papers).
We summarize these new benchmarks into three categories.
The first category dataset is tailored for rare repair scenarios.
As discussed in Section <ref>, LLMs have been used in a variety of scenarios, some of which have not been considered by previous work, leading to a gap in relevant benchmarks.
As a result, with the advent of LLM-based APR techniques, researchers have also developed corresponding new datasets, such as TFix <cit.> for static warnings, DeepDev-PERF <cit.> for performance bugs, Du et al. <cit.> for crash bugs, and Zhang et. al <cit.> for API misuse.
The second category dataset attempts to address the limitations of previous benchmarks.
For example, considering BFP <cit.> lacking test suites, FixEval <cit.> offers a collection of unit tests for a large-scale of competitive programming problems and is evaluated with PLBART and CodeT5.
The third category of datasets is designed to address the issues unique to LLMs, particularly the data leakage problem.
For example, Jiang et al. <cit.> create a new evaluation benchmark, HumanEval-Java, that has not been seen by LLMs during pre-training.
Zhang et al. <cit.> extensively explore the data leakage issue of ChatGPT in the APR domain and introduce EvalGPTFix, a new benchmark from competitive programming problems after the training cutoff point of ChatGPT.
DebugBench <cit.> is a follow-up of EvalGPTFix with a larger scale and more diverse types of bugs.
DebugBench contains 4,253 buggy programs from the
LeetCode community, covering four major bug categories and 18 minor types in C++, Java, and Python.
Similarly, ConDefects <cit.> contains 1,254 Java faulty programs and 1,625 Python faulty programs from the online competition platform AtCoder.
These collected programs are produced between October 2021 and September 2023 to address the data leakage issue for LLM-based APR approaches.
Different from the aforementioned benchmarks derived from programming problems<cit.>, Silva et al. <cit.> introduce GitBug-Java, a reproducible benchmark comprising 199 recent Java bugs. These bugs are extracted from the 2023 commit history of 55 notable open-source repositories to mitigate the risk of data leakage.

§.§ What input forms are software bugs transformed into when utilizing LLMs?

Thanks to the powerful natural language understanding capabilities of LLMs, the inputs of LLM-based APR contain rich information, thus more complex than that of traditional APR techniques <cit.>.
We summarize various input forms into five categories according to their data types.
As illustrated in Fig. <ref>, we find 52% of collected papers leverage prompt engineering to feed LLMs with bug-fixing information, and 18% utilize a conversational-style representation to provide dynamic information.
We also find only 18% of LLM-based APR studies adopt raw bug-fixing inputs in a manner similar to traditional DL/ML models <cit.>.
We will discuss the five input representations utilized by LLMs as follows.

182Raw Bug-fixing Input.
Similar to most traditional learning-based APR, this type of input regards APR as an NMT task, which translates a sentence from one source language (i.e., buggy code) to another target language (i.e., fixed code).
Such representation directly feeds LLMs with the buggy code snippet and has been typically employed to train LLMs with supervised learning in semantic bugs <cit.> security veulnerabilities <cit.>, and static warnings <cit.>.
For example, Zhang et al. <cit.> investigate the performance of three bug-fixing representations (i.e., context, abstraction, and tokenization) to fine-tune five LLMs for vulnerability repair.

183Prompt Input.
This type of input incorporates more information to the buggy code.
The prompt concatenates different input components with some prefixed prompt, thus effectively bridging the gap between pre-trained tasks and the APR downstream task.
For example, CIRCLE <cit.> utilizes manually-designed prompt template to convent buggy code and corresponding context into a unified fill-in-the-blank format.
Particularly, they utilize “Buggy line:” and “Context:” to denote the buggy and contextual code, and they utilize “The fixed code is:” to query a T5-based model to generate candidate patches according to the previous input.
Besides, TFix <cit.>, Zirak et al. <cit.> and Kim et al. <cit.> represent all valuable information about the bug as a single piece of text, including bug type, bug message, bug line, and bug context.
Furthermore, InferFix <cit.> and RAP-Gen <cit.> construct prompts by retrieving relevant repair examples from an external codebase.

184Mask Input.
This type of input masks the buggy code and queries LLMs to fill the masks with the correct code tokens.
Unlike the above input forms, the mask input reformulates the APR problem as a cloze-style task and directly leverages LLMs' pre-training objectives in a zero-shot setting.
AlphaRepair <cit.> is considered the first work to demonstrate the potential of mask inputs, and researchers have proposed various follow-ups to better perform mask prediction for patch generation, such as GAMMA <cit.> with well-summarized repair patterns, FitRepair <cit.> with the plastic surgery hypothesis, Repilot <cit.> with a completion engine, as well as empirical studies <cit.>.

185Conversation-Style Representation
This type of input further extends the prompt input with feedback-driven chats like humans.
Conversation-style representation contains more complex information, such as dynamic execution results, while iteratively improving generated patches through multiple rounds of dialogue.
For example, Sobania et al. <cit.> conduct an early exploration into the feasibility of leveraging ChatGPT's conversational capabilities for program repair, motivating some follow-ups <cit.>.

186Structure-Aware Input.
This type of input represents source code as syntactic structures, such as Abstract Syntax Trees (ASTs).
For example, Horvath et al. <cit.> utilizes RoBERTa and GPTNeo to encode ASTs for program repair.
Besides, VulMaster <cit.>  utilizes the AST as part of its input to capture the structural aspects of the vulnerable code.

§.§ How are LLMs used to support patch correctness?

Test-overfitting has recently been the focus of APR research due to the mainstream test-driven generate-and-validate repair workflow in the community.
Particularly, after candidate patches are generated, the developer-written test suites are utilized as the oracle to validate the correctness of patches, and the passing patches are returned to developers.
However, as an incomplete specification, test suites only describe a part of the program's behavioral space, resulting in plausible yet overfitting patches without fixing bugs.
Thus, developers need to spend enormous effort filtering out overfitting patches manually, even resulting in a negative debugging performance <cit.>.
Researchers have proposed various automated patch correctness assessment (APCA) techniques to identify whether a plausible patch is overfitting or not, so as to improve the quality of returned patches.
For example, PATCH-SIM <cit.> assesses the correctness of patches by calculating the similarity of dynamic execution traces.
PATCH-SIM is acknowledged as a foundational work in the APCA field, providing crucial guidance for the development of follow-up works, particularly learning-based or LLM-based ones <cit.>.

Table <ref> presents existing APCA studies involving LLMs.
We summarize them into three stages.

182LLMs as Feature Extractor.
In 2020, Tian et al. <cit.> empirically explore the performance of code embeddings via representation learning models in reasoning about patch correctness.
Following the similarity-based pipeline from PATCH-SIM, they first calculate the similarities of patched and buggy code snippets based on code embeddings, and then predict patch correctness with a binary classifier.
They consider four embedding models, including re-trained (i.e., Doc2vec, code2vec and CC2vec) and pre-trained models (i.e., BERT), which is the first APCA study empowered with LLMs.
Recently, Le et al. <cit.> propose Invalidator, to assess the correctness of patches via semantic and syntactic reasoning.
Similar to Tian et al. <cit.>, they utilize CodeBERT to extract code features and train a classifier for prediction.
Unlike the above studies calculating similarities of patches, Tian et al. <cit.> formulate APCA as a question-answering (QA) problem and propose Quatrain.
Quatrain first utilizes CodeBERT to encode bug reports and patch descriptions and trains a QA model for prediction.

183Fine-tuning LLM-based APCA.
In 2024, Zhang et al. <cit.> propose APPT, equipped with BERT as the encoder stack, followed by an LSTM stack and a deep learning classifier.
Unlike previous studies <cit.> limiting BERT to extract features without benefiting from training, APPT further fine-tunes LLMs in conjunction with other components as a whole pipeline to fully adapt it specifically for reasoning about patch correctness.
APPT is implemented with BERT by default and is also proven generalizable to other advanced LLMs, such as CodeBERT and GraphCodeBERT.

184Zero-shot LLM-based APCA.
In 2023, Zhou et al. <cit.> propose PatchZero to explore the feasibility of LLMs in predicting patch correctness with a zero-shot setting.
PatchZero directly queries LLMs to generate the next token about patch correctness (i.e., a token either “correct” or “overfitting”) based on previous tokens, which is similar to LLMs' original pre-training objective.

§.§ How are LLMs utilized to facilitate both code generation and program repair?

Compared with traditional APR approaches <cit.> that usually employ heuristics or neural networks to generate a multitude of patches in one go, LLMs can iteratively refine the generated patches based on the outcomes of dynamic execution.
As mentioned in Section <ref>, the iterative patch generation capabilities brought by LLMs have facilitated the emergence of conversational-based repair techniques <cit.>.
In addition to conversational-based repair, we have identified some efforts that improve code generation performance by repairing LLM-generated code with feedback, referred to as self-repair.
Different from conversational-based repair, which utilizes feedback for patch generation, self-repair approaches leverage LLMs to identify code errors implemented by themselves via investigating execution results and explaining the generated code.
Such self-repair approaches integrate code generation and program repair, which can be considered a further step toward automated programming.

A summary of existing studies that use APR to boost code generation.

Year     Study     LLMs     Repository

2023      AgentCoder <cit.>     GPT-3.5,GPT-4,Claude,PaLM     N.A.

2023      Self-Edit <cit.>     InCoder,CodeGen,Codex     <https://github.com/zkcpku/Self-Edit>

2024      Self-Debugging <cit.>     ChatGPT,GPT-4,Codex,StarCoder     N.A.

2024      Zheng et al. <cit.>     CodeLLaMA,DeepseekCoder     <https://github.com/OpenCodeInterpreter/OpenCodeInterpreter>

2024      CYCLE <cit.>     CodeGen,StarCoder     <https://github.com/ARiSE-Lab/CYCLE_OOPSLA_24>

2024      LDB <cit.>     StarCoder,CodeLLaMA,GPT-3.5     <https://github.com/FloridSleeves/LLMDebugger>

2024      Olausson et al. <cit.>     GPT-3.5,GPT-4,CodeLLaMA     <https://github.com/theoxo/self-repair>

2024      Hu et al. <cit.>     GPT-4     N.A.

Table <ref> presents some LLM-based studies that leverage program repair to boost code generation.
Self-Edit <cit.> represents the first attempt to adopt a neural code editor that takes both the generated code and error messages as inputs to improve the code quality on the competitive programming task.
Self-Edit is evaluated with both fine-tuned models (i.e., PyCodeGPT, GPT-Neo, CodeGen, GPT-Neo, InCoder, GPT-J) and prompt-based LLMs (i.e., InCoder, CodeGen, Codex).
Chen et al. <cit.> from DeepMind propose Self-Debugging to teach LLMs to debug their own predicted code via few-shot prompting, including Codex, ChatGPT, GPT-4 and StarCoder.
There also exist some similar self-repair studies, including OpenCodeInterpreter <cit.>, Cycle<cit.>, LDB <cit.>, SelfEvolve <cit.>, Self-Refine <cit.>, Hu et al. <cit.>, AgentCoder <cit.>.
Recently, Olausson et al. <cit.> conduct an empirical study to investigate the ability of CodeLlama, GPT-3.5, and GPT-4 to perform self-repair in code generation.
They find that self-repair is not a panacea for code generation challenges, as existing LLMs often fail to provide reliable, accurate, and valuable feedback on why the code is incorrect.

§.§ How often do the collected LLM-based APR papers provide publicly available artifacts?

Open science plays a crucial role in advancing scientific progress through principles of transparency, reproducibility, and applicability.
Given the benefits, the SE community has been actively promoting open science principles and encouraging all researchers to share their artifacts, thereby bolstering the reliability of research findings.
In this section, we investigate the extent to which the analyzed papers make their artifacts publicly accessible.

We find that 80 studies provide the replication packages in their papers, accounting for 62.99% (80/) of all collected studies.
Among 78 studies that propose novel LLM-based APR approaches, which is the largest contribution type in Table <ref>, we find that 53.85% (42/78) of them fail to make their artifacts publicly available.
This makes it difficult for researchers to validate experimental findings, conduct quantitative comparisons with existing studies, and build follow-ups instead of reinventing the wheels.
Considering that some papers have not been published, we then focus on top-tier SE venues, i.e., ICSE, ASE, FSE, ISSTA, TSE and TOSEM, and identify 86.84% of papers (33/38) make related artifacts publicly open, indicating a high-quality commitment to reproducibility in high-quality papers.
Besides, some studies only provide datasets or train models without source code or essential instructions.
Overall, open science is a critical challenge in advancing LLM-based APR research development because many factors, such as datasets, data pre-processing methods, source code, hyper-parameters, and documents, may lead to the reproducibility of studies.
Therefore, we hope that researchers in the LLM-based APR community can provide high-quality open-source artifacts for convenient reproduction.

(1) We summarize  different datasets that are utilized to benchmark LLMs in fixing bugs.
(2) Defects4J, QuixBugs, BFP, CVEfixes, and Big-Vul are most frequently adopted in the LLM-based APR.
(3) We categorize the input forms within all collected papers into five groups: raw bug-fixing input, prompt input, mask input, conversation-style input, and structure-aware input.
(4) Prompt input is the most frequently used form in applying LLMs to program repair, indicating that designing effective prompts is particularly important for leveraging LLMs' natural language processing capabilities.
(5) We summarize some studies that leverage LLMs to predict patch correctness.
(6) 62.99% of all collected papers have made their artifacts open source, and the ratio increases to 86.84% for top-tier SE publications.

§ CHALLENGES & OPPORTUNITIES OF LLM-BASED APR

Our study reveals the following crucial challenges and practical guidelines for future LLM-based APR work.

APR as a Part of Fully Autonomous Programming.
As mentioned in Section <ref>, existing APR approaches operate under a “near-correct assumption”, which posits that experienced programmers are capable of writing programs that are almost correct, requiring only minimal modifications to rectify bugs and ensure compliance with all test cases.
This assumption has long served as the foundation for APR research development.
However, the evolution of LLMs and their application in programming suggests a future where APR can transcend its traditional boundaries, moving towards a more holistic approach in conjunction with fully autonomous programming.
In this context, APR can be reimagined not only just as a tool for correcting minor coding errors, but as an integral component of a self-correcting, self-improving system that iteratively enhances the quality of automatically generated code.
Fortunately, we have observed initial explorations into combining repair with programming, yet these efforts are far from sufficient.
For example, Fan et al. <cit.> employ LLMs to fix buggy solutions generated by themselves, and recent studies <cit.> iteratively refine auto-generated code with dynamic execution.

In the future, the opportunities presented by integrating APR with fully autonomous programming are vast <cit.>.
First, it is flexible to implement collaborative Human-AI Programming tools <cit.>, where the initial code is written by developers and continuously optimized and repaired by LLMs.
Particularly for complex problem-solving, LLMs suggest innovative solutions that had not been considered by human programmers.
This partnership could accelerate development cycles, reduce the burden of debugging on human developers, and lead to more creative and effective solutions.
Second, the general knowledge inherent in LLMs endows them with the capability to support multiple downstream tasks, which opens up the possibility of bridging the gap between code generation, testing, debugging, and fixing.
For example, fault localization is a precondition of patch generation, while patch validation can reflect the accuracy of fault localization, making these takes interconnected.
Thus, it is promising to explore the capabilities of LLMs in these connected tasks using real-time feedback within a unified framework.

More Attention about Repair Costs of LLMs..
As mentioned in Section <ref>, the literature tends to employ the growing size of LLMs to achieve better performance, such as from T5-250M in CIRCLE to Codex-12B in InferFix.
This trend is reasonable as Xia et al. <cit.> demonstrate that larger models typically generate more correct patches for software bugs.
However, APR, being a task highly relevant to humans, the millions, billions and even trillions of parameters pose significant challenges in the development workflow.
First, training LLMs with billions (or even more) of parameters is a highly time-consuming and resource-intensive process.
The GPU resources required for such training are often prohibitively expensive, making them inaccessible for many researchers in both academic and industrial settings.
For example, in Section <ref>, we observe that most fine-tuning-LLM-based APR studies utilize CodeT5/T5 or similar-sized models, except for InferFix from Microsoft, which fine-tunes Codex.
However, InferFix is proposed by the world-leading technology company Microsoft and trained with industrial-grade hardware, i.e., 64 32-GB-V100-GPUs.
Second, despite the increased likelihood of generating correct patches with larger models, the patch generation time cost also increases.
For example, Jiang et al. <cit.> demonstrate PLBART takes 0.70–0.89 seconds on average to generate a correct patch, CodeGen, although fixing more bugs than PLBART with more parameters, requires 3.64–13.88 seconds on average to generate a correct patch.
Third, the increase in patch generation time further compresses the time available for patch validation, as developers need to spend more time waiting for the model to complete its inference.
For example, Shi et al. <cit.> demonstrate that the vast storage and runtime memory consumption, coupled with high inference latency, make these LLMs prohibitive for integration in modern IDEs, especially on resource-constrained or real-time terminal devices.

In the future, to address the first challenge, it is promising to explore the potential of parameter-efficient fine-tuning approaches on APR, such as prefix-tuning and low-rank adaptation <cit.>.
To address the second challenge, researchers can optimize the size of LLMs without significantly compromising their performance, such as model pruning, quantization, and knowledge distillation <cit.>.
To address the third challenge, we recommend boosting patch validation with advanced strategies, such as mutation testing <cit.>, or utilizing LLMs to rank candidate patches before validation.

Human Study with LLMs.
As summarized in Section <ref>, with the introduction of LLMs, APR has achieved groundbreaking progress in terms of the number of correctly fixed bugs in popular benchmarks, exemplified by the advancements on Defects4J-1.2 from 64 bugs by CIRCLE, to 74 bugs by AlphaRepair, and 82 bugs by GAMMA.
The progress prompts us to consider whether LLMs have indeed facilitated improvements in real-world debugging and how LLM-based APR has broader implications for developers' daily activities.
Previous research <cit.> highlights the potential pitfalls of developing APR tools without adequate feedback from developers, which could compromise their effectiveness in real-world deployment.
However, there remains a significant gap in our understanding of how software engineers tackle software problems in practical settings, including their use of dedicated debugging tools and their expertise in debugging techniques.

Thus, in the future, researchers should conduct human studies to gain deeper insights into the maturity and reliability of LLM-based APR tools in terms of human factors.
Possible directions are to investigate whether LLMs can assist developers in reducing the debugging cost, such as fixing more bugs, accelerating the bug-fixing process, and handling more complex bugs.
Besides, it would be valuable to investigate developers' perceptions and interactions with LLMs based on their practical experiences and established debugging practices.

Exploring More and Rare Repair Scenarions.
As summarized in Section <ref>, we observe that most existing LLM-based APR studies are concentrated on a limited number of bug types, particularly semantic bugs.
However, there exist some rare repair scenarios that benefit less from LLMs, such as hardware bugs (one paper) and concurrency bugs (zero paper).
The key challenge lies in insufficient training data from which LLMs can learn.

We suggest that future work concentrates on three possible directions to broaden the scope of LLM applications for more repair scenarios, such as software requirement <cit.> and fuzzing <cit.>.
First, transfer learning is an effective training approach for rare scenarios.
We can first fine-tune LLMs with abundant data in a source scenario and then utilize a small amount of data to transfer the acquired knowledge to a target scenario.
The source and target scenarios should have similar data distributions.
For example, Zhang et al. <cit.> demonstrate that transferring learning from bug-fixing can improve the vulnerability repair performance of five LLMs by 9.40% on average.
Second, it is promising to utilize the cloze-style APR with repair patterns to generate patches for rare scenarios.
Unlike fine-tuning, which requires a substantial amount of labeled data, this approach directly leverages expert domain knowledge (such as pre-defined patterns) to guide the general pre-trained knowledge of LLMs.
Besides, for scenarios not previously encountered by LLMs, researchers can employ unlabeled data in the target scenario to learn the data distribution of the project or language under test in an unsupervised learning setting.
Third, in-context learning and prompt engineering are feasible solutions for directly querying billion-parameter LLMs to generate correct code in a target scenario, given that these models are immensely large and encompass virtually all data available on the Internet.

Integration with Off-the-Shelf APR.
As mentioned in Section <ref>, researchers typically utilize LLMs as core backbones to design novel repair approaches in an end-to-end setting, such as sequence-to-sequence learning <cit.>.
Parallelly, the community has seen some explorations treating LLMs as components integrated into existing repair workflow.
These studies attempt to boost the capabilities of off-the-shelf APR approaches instead of proposing new techniques.
For example, CURE <cit.> combines GPT and CoCoNut <cit.> to capture code syntax for the APR task.
Built on top of DLFix <cit.>, DEAR <cit.> attempts to fix multi-hunk, multi-statement bugs by fine-tuning BERT to learn fixing-together relationships among statements, i.e., identifying if two statements are needed to be fixed together or not.
Recently, GAMMA <cit.> integrates LLMs into the traditional template-based APR TBar by querying LLMs to generate masked code tokens instead of retrieving donor code from local files.
These efforts demonstrate the potential of integrating LLMs with off-the-shelf APR techniques, yet there is currently a lack of more in-depth work in this area.

In the future, researchers could attempt to combine LLMs with more traditional APR techniques.
For example, it is promising to utilize LLMs to help SMT solvers generate patches for constraint-based APR, or feed search algorithms with LLM-generated potential patches to build search space for heuristic-based APR.
Besides, domain-specific repair techniques can benefit from the powerful code-understanding capabilities of LLMs, thus extending to a broader range of repair scenarios.
For example, we can design fix templates for specific scenarios, such as static warnings, and then utilize the general knowledge contained in LLMs to generate correct patches.

Data Leakage Issue.
As highlighted in Section <ref>, Zhang et al. <cit.> identify that existing repair benchmarks have been inadvertently included in the pre-training data of popular LLMs, such as ChatGPT, through web scraping and other methods.
For example, ChatGPT is able to enumerate all projects within Defects4J <cit.>, one of the most popular APR benchmarks.
Researchers <cit.> can ascertain the exposure for open-source LLMs by inspecting pre-training data against benchmarks.
It is significantly more challenging with more powerful black-box LLMs due to a lack of training details.
However, as preliminary explorations, they are mainly limited in the quantity of involved bugs and the variety of bug types.
For example, all of them are created from programming problems with only small-scale or medium-scale buggy solutions, without delving into large-scale, real-world projects that encompass complex API calls, such as Defecst4J <cit.>.
More importantly, data leakage issues in other repair scenarios, such as security vulnerabilities and API misuse, continue to be overlooked.
There may be overlaps among datasets across different repair scenarios, such as Defects4J, which also serves as a source for a subset of the API misuse dataset <cit.>.
Overall, the risk of data leakage introduces bias into the benchmarking of existing work, necessitating urgent efforts from researchers to mitigate it.

We recommend that future work can be conducted in the following two directions.
First, it is crucial to construct a large-scale benchmark free from data leakage that contains real-world projects so as to evaluate the actual fix capabilities of LLMs in a more practical debugging scenario.
Commercial closed-source software, real-time updated programming websites, or manually written programs may serve as potential data sources.
Second, considering the variety of bug types that LLMs have been applied to, researchers need to consider and attempt to address the data leakage risk when conducting related studies.

§ CONCLUSION

Automated Program Repair (APR) tackles the long-standing challenge of fixing software bugs automatically, thus facilitating software testing, validation, and debugging practices.
Very recently, Large Language Models (LLMs) have brought significant changes to the APR domain, already yielding impressive progress and further demonstrating a promising future in follow-up research.
In this paper, we provide a systematic literature review of existing LLM-based APR techniques from LLMs, APR and their integration perspectives
We summarize popular LLMs, typical utilization strategies, and repair scenarios.
We also discuss some crucial factors, such as input forms and self-debug metrics, within the LLM-based APR community.
Finally, we outline several challenges, such as data leakage issues, and suggest potential directions for future research.

§ ACKNOWLEDGMENTS

This work is supported partially by the National Natural Science Foundation of China (61932012, 62141215, 62372228).

#1     #1#1#1     #1     #1     #1     #1#1          #1#1

[Ahmad et al(2024)]
authorpersonBaleegh Ahmad, personShailja Thakur, personBenjamin Tan, personRamesh Karri, and personHammond Pearce. year2024.
On Hardware Security Bug Code Fixes By Prompting Large Language Models.
journalIEEE Transactions on Information Forensics and Security (year2024).
Early Access, DOI: <10.1109/TIFS.2024.3374558>.

[Ahmad et al(2021)]
authorpersonWasi Uddin Ahmad, personSaikat Chakraborty, personBaishakhi Ray, and personKai-Wei Chang. year2021.
Unified Pre-Training for Program Understanding and Generation. In booktitleProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. publisherAssociation for Computational Linguistics, pages2655–2668.

[Ahmed et al(2022)]
authorpersonToufique Ahmed, personNoah Rose Ledesma, and personPremkumar Devanbu. year2022.
SynShine: Improved Fixing of Syntax Errors.
journalIEEE Transactions on Software Engineering volume49, number4 (year2022), pages2169–2181.

[Alagarsamy et al(2023)]
authorpersonSaranya Alagarsamy, personChakkrit Tantithamthavorn, and personAldeida Aleti. year2023.
A3test: Assertion-Augmented Automated Test Case Generation.
journalarXiv preprint arXiv:2302.10352 (year2023).

[Alrashedy and Aljasser(2023)]
authorpersonKamel Alrashedy and personAbdullah Aljasser. year2023.
Can LLMs Patch Security Issues?
journalarXiv preprint arXiv:2312.00024 (year2023).

authorpersonBandit. year2024.
titleA Static tool to Find Common Security Issues in Python Code.
howpublishedURL: <https://github.com/PyCQA/bandit>.
Lasted accessed: 2024-04-01.

[Berabi et al(2021)]
authorpersonBerkay Berabi, personJingxuan He, personVeselin Raychev, and personMartin Vechev. year2021.
TFix: Learning to Fix Coding Errors with a Text-to-Text Transformer. In booktitleInternational Conference on Machine Learning. PMLR, pages780–791.

[Bhandari et al(2021)]
authorpersonGuru Bhandari, personAmara Naseer, and personLeon Moonen. year2021.
CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-source Software. In booktitleProceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering. pages30–39.

[Black et al(2022)]
authorpersonSidney Black, personStella Biderman, personEric Hallahan, personQuentin Anthony, personLeo Gao, personLaurence Golding, personHorace He, personConnor Leahy, personKyle McDonell, personJason Phang, et al year2022.
GPT-Neox-20b: An Open-Source Autoregressive Language Model. In booktitleProceedings of BigScience Episode# 5–Workshop on Challenges & Perspectives in Creating Large Language Models. pages95–136.

[Black et al(2021)]
authorpersonSid Black, personLeo Gao, personPhil Wang, personConnor Leahy, and personStella Biderman. year2021.
titleGPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow.
howpublishedURL: <https://doi.org/10.5281/zenodo.5297715>.
Lasted accessed: 2024-04-01.

authorpersonCO Boulder. year2013.
titleFailure to Adopt Reverse Debugging Costs Global Economy $41 Billion Annually.

Lasted accessed: 2024-04-01.

[Bouzenia et al(2024)]
authorpersonIslem Bouzenia, personPremkumar Devanbu, and personMichael Pradel. year2024.
RepairAgent: An Autonomous, LLM-Based Agent for Program Repair.
journalarXiv preprint arXiv:2403.17134 (year2024).

[Britton et al(2013)]
authorpersonTom Britton, personLisa Jeng, personGraham Carver, personPaul Cheak, and personTomer Katzenellenbogen. year2013.
Reversible Debugging Software: Quantify the Time and Cost Saved Using Reversible Debuggers.
journalJudge Bus. School, Univ. Cambridge, Cambridge, UK, Tech. Rep  volume229 (year2013).

[Brown et al(2020)]
authorpersonTom Brown, personBenjamin Mann, personNick Ryder, personMelanie Subbiah, personJared D Kaplan, personPrafulla Dhariwal, personArvind Neelakantan, personPranav Shyam, personGirish Sastry, personAmanda Askell, et al year2020.
Language Models Are Few-Shot Learners. In booktitleAdvances in Neural Information Processing Systems, Vol. volume33. pages1877–1901.

[Cao et al(2023)]
authorpersonJialun Cao, personMeiziniu Li, personMing Wen, and personShing-chi Cheung. year2023.
A Study on Prompt Design, Advantages and Limitations of ChatGPT for Deep Learning Program Repair.
journalarXiv preprint arXiv:2304.08191 (year2023).

[Chen et al(2021)]
authorpersonMark Chen, personJerry Tworek, personHeewoo Jun, personQiming Yuan, personHenrique Pondé de Oliveira Pinto, personJared Kaplan, personHarrison Edwards, personYuri Burda, personNicholas Joseph, personGreg Brockman, personAlex Ray, personRaul Puri, personGretchen Krueger, personMichael Petrov, personHeidy Khlaaf, personGirish Sastry, personPamela Mishkin, personBrooke Chan, personScott Gray, personNick Ryder, personMikhail Pavlov, personAlethea Power, personLukasz Kaiser, personMohammad Bavarian, personClemens Winter, personPhilippe Tillet, personFelipe Petroski Such, personDave Cummings, personMatthias Plappert, personFotios
Chantzis, personElizabeth Barnes, personAriel Herbert-Voss, personWilliam Hebgen Guss, personAlex Nichol, personAlex Paino, personNikolas Tezak, personJie Tang, personIgor Babuschkin, personSuchir Balaji, personShantanu Jain, personWilliam Saunders, personChristopher Hesse, personAndrew N. Carr, personJan Leike, personJoshua Achiam, personVedant Misra, personEvan Morikawa, personAlec Radford, personMatthew Knight, personMiles Brundage, personMira Murati, personKatie Mayer, personPeter Welinder, personBob McGrew, personDario Amodei, personSam McCandlish, personIlya Sutskever, and personWojciech Zaremba. year2021.
Evaluating Large Language Models Trained on Code.
journalarXiv preprint arXiv:2107.03374 (year2021).

[Chen et al(2024)]
authorpersonXinyun Chen, personMaxwell Lin, personNathanael Schärli, and personDenny Zhou. year2024.
Teaching Large Language Models to Self-Debug. In booktitleThe Twelfth International Conference on Learning Representations.

[Chen et al(2019)]
authorpersonZimin Chen, personSteve Kommrusch, personMichele Tufano, personLouis-Noël Pouchet, personDenys Poshyvanyk, and personMartin Monperrus. year2019.
Sequencer: Sequence-To-Sequence Learning for End-To-End Program Repair.
journalIEEE Transactions on Software Engineering volume47, number9 (year2019), pages1943–1959.

[Chow et al(2024)]
authorpersonYiu Wai Chow, personLuca Di Grazia, and personMichael Pradel. year2024.
PyTy: Repairing Static Type Errors in Python. In booktitleProceedings of the 46th International Conference on Software Engineering. pages871–871.

[Chowdhery et al(2023)]
authorpersonAakanksha Chowdhery, personSharan Narang, personJacob Devlin, personMaarten Bosma, personGaurav Mishra, personAdam Roberts, personPaul Barham, personHyung Won Chung, personCharles Sutton, personSebastian Gehrmann, personParker Schuh, personKensen Shi, personSasha Tsvyashchenko, personJoshua Maynez, personAbhishek Rao, personParker Barnes, personYi Tay, personNoam Shazeer, personVinodkumar Prabhakaran, personEmily Reif, personNan Du, personBen Hutchinson, personReiner Pope, personJames Bradbury, personJacob Austin, personMichael Isard, personGuy Gur-Ari, personPengcheng Yin, personToju Duke, personAnselm Levskaya,
personSanjay Ghemawat, personSunipa Dev, personHenryk Michalewski, personXavier Garcia, personVedant Misra, personKevin Robinson, personLiam Fedus, personDenny Zhou, personDaphne Ippolito, personDavid Luan, personHyeontaek Lim, personBarret Zoph, personAlexander Spiridonov, personRyan Sepassi, personDavid Dohan, personShivani Agrawal, personMark Omernick, personAndrew M. Dai, personThanumalayan Sankaranarayana Pillai, personMarie Pellat, personAitor Lewkowycz, personErica Moreira, personRewon Child, personOleksandr Polozov, personKatherine Lee, personZongwei Zhou, personXuezhi Wang, personBrennan Saeta, personMark Diaz, personOrhan Firat,
personMichele Catasta, personJason Wei, personKathy Meier-Hellstern, personDouglas Eck, personJeff Dean, personSlav Petrov, and personNoah Fiedel. year2023.
PaLM: Scaling Language Modeling with Pathways.
journalJournal of Machine Learning Research volume24, number240 (year2023), pages1–113.

[de Fitero-Dominguez et al(2024)]
authorpersonDavid de Fitero-Dominguez, personEva Garcia-Lopez, personAntonio Garcia-Cabot, and personJose-Javier Martinez-Herraiz. year2024.
Enhanced Automated Code Vulnerability Repair Using Large Language Models.
journalarXiv preprint arXiv:2401.03741 (year2024).

[Deligiannis et al(2023)]
authorpersonPantazis Deligiannis, personAkash Lal, personNikita Mehrotra, and personAseem Rastogi. year2023.
Fixing Rust Compilation Errors Using Llms.
journalarXiv preprint arXiv:2308.05177 (year2023).

[Devlin et al(2019)]
authorpersonJacob Devlin, personMing-Wei Chang, personKenton Lee, and personKristina Toutanova. year2019.
BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. In booktitleProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT'19). publisherAssociation for Computational Linguistics, pages4171–4186.

[Dinella et al(2022)]
authorpersonElizabeth Dinella, personGabriel Ryan, personTodd Mytkowicz, and personShuvendu K Lahiri. year2022.
TOGA: A Neural Method for Test Oracle Generation. In booktitleProceedings of the 44th International Conference on Software Engineering. publisherACM, pages2130–2141.

[Ding et al(2023)]
authorpersonNing Ding, personYujia Qin, personGuang Yang, personFuchao Wei, personZonghan Yang, personYusheng Su, personShengding Hu, personYulin Chen, personChi-Min Chan, personWeize Chen, et al year2023.
Parameter-Efficient Fine-Tuning of Large-Scale Pre-Trained Language Models.
journalNature Machine Intelligence volume5, number3 (year2023), pages220–235.

[Ding et al(2024)]
authorpersonYangruibo Ding, personMarcus J Min, personGail Kaiser, and personBaishakhi Ray. year2024.
CYCLE: Learning to Self-Refine the Code Generation.
journalarXiv preprint arXiv:2403.18746 (year2024).

[Do Viet and Markov(2023)]
authorpersonTung Do Viet and personKonstantin Markov. year2023.
Using Large Language Models for Bug Localization and Fixing. In booktitle2023 12th International Conference on Awareness Science and Technology. IEEE, pages192–197.

[Drain et al(2021a)]
authorpersonDawn Drain, personColin B Clement, personGuillermo Serrato, and personNeel Sundaresan. year2021a.
DeepDebug: Fixing Python Bugs Using Stack Traces, Backtranslation, and Code Skeletons.
journalarXiv preprint arXiv:2105.09352 (year2021).

[Drain et al(2021b)]
authorpersonDawn Drain, personChen Wu, personAlexey Svyatkovskiy, and personNeel Sundaresan. year2021b.
Generating Bug-Fixes Using Pretrained Transformers. In booktitleProceedings of the 5th ACM SIGPLAN International Symposium on Machine Programming. pages1–8.

[Du et al(2023)]
authorpersonXueying Du, personMingwei Liu, personJuntao Li, personHanlin Wang, personXin Peng, and personYiling Lou. year2023.
Resolving Crash Bugs Via Large Language Models: An Empirical Study.
journalarXiv preprint arXiv:2312.10448 (year2023).

[Durieux and Monperrus(2016)]
authorpersonThomas Durieux and personMartin Monperrus. year2016.
DynaMoth: Dynamic Code Synthesis for Automatic Program Repair. In booktitleProceedings of the 11th International Workshop on Automation of Software Test. pages85–91.

[Fan et al(2023b)]
authorpersonAngela Fan, personBeliz Gokkaya, personMark Harman, personMitya Lyubarskiy, personShubho Sengupta, personShin Yoo, and personJie M Zhang. year2023b.
Large Language Models for Software Engineering: Survey and Open Problems.
journalarXiv preprint arXiv:2310.03533 (year2023).

[Fan et al(2020)]
authorpersonJiahao Fan, personYi Li, personShaohua Wang, and personTien N Nguyen. year2020.
A C/C++ Code Vulnerability Dataset with Code Changes and CVE Summaries. In booktitleProceedings of the 17th International Conference on Mining Software Repositories. pages508–512.

[Fan et al(2023a)]
authorpersonZhiyu Fan, personXiang Gao, personMartin Mirchev, personAbhik Roychoudhury, and personShin Hwei Tan. year2023a.
Automated Repair of Programs from Large Language Models. In booktitle2023 IEEE/ACM 45th International Conference on Software Engineering. IEEE, pages1469–1481.

[Feng et al(2020)]
authorpersonZhangyin Feng, personDaya Guo, personDuyu Tang, personNan Duan, personXiaocheng Feng, personMing Gong, personLinjun Shou, personBing Qin, personTing Liu, personDaxin Jiang, and personMing Zhou. year2020.
CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In booktitleFindings of the Association for Computational Linguistics. pages1536–1547.

[First et al(2023)]
authorpersonEmily First, personMarkus Rabe, personTalia Ringer, and personYuriy Brun. year2023.
Baldur: Whole-proof generation and repair with large language models. In booktitleProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. pages1229–1241.

[Fried et al(2023)]
authorpersonDaniel Fried, personArmen Aghajanyan, personJessy Lin, personSida Wang, personEric Wallace, personFreda Shi, personRuiqi Zhong, personScott Yih, personLuke Zettlemoyer, and personMike Lewis. year2023.
InCoder: A Generative Model for Code Infilling and Synthesis. In booktitleThe Eleventh International Conference on Learning Representations.

[Fu et al(2023a)]
authorpersonMichael Fu, personVan Nguyen, personChakkrit Tantithamthavorn, personDinh Phung, and personTrung Le. year2023a.
Vision Transformer-Inspired Automated Vulnerability Repair.
journalACM Transactions on Software Engineering and Methodology (year2023).

[Fu et al(2022)]
authorpersonMichael Fu, personChakkrit Tantithamthavorn, personTrung Le, personVan Nguyen, and personPhung Dinh. year2022.
VulRepair: A T5-Based Automated Software Vulnerability Repair. In booktitlethe ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. publisherACM, pages935–947.

[Fu et al(2023b)]
authorpersonWeimin Fu, personKaichen Yang, personRaj Gautam Dutta, personXiaolong Guo, and personGang Qu. year2023b.
LLM4SecHW: Leveraging domain-specific large language model for hardware debugging. In booktitle2023 Asian Hardware Oriented Security and Trust Symposium (AsianHOST). IEEE, pages1–6.

[Garg et al(2022)]
authorpersonSpandan Garg, personRoshanak Zilouchian Moghaddam, personColin B Clement, personNeel Sundaresan, and personChen Wu. year2022.
DeepDev-PERF: A Deep Learning-Based Approach for Improving Software Performance. In booktitleProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. pages948–958.

[Garg et al(2023)]
authorpersonSpandan Garg, personRoshanak Zilouchian Moghaddam, and personNeel Sundaresan. year2023.
Rapgen: An Approach for Fixing Code Inefficiencies in Zero-Shot.
journalarXiv preprint arXiv:2306.17077 (year2023).

[Gazzola et al(2019)]
authorpersonLuca Gazzola, personDaniela Micucci, and personLeonardo Mariani. year2019.
Automatic Software Repair: A Survey.
journalIEEE Transactions on Software Engineering volume45, number1 (year2019), pages34–67.

[Ge and Wu(2023)]
authorpersonHaotong Ge and personYuemeng Wu. year2023.
An Empirical Study of Adoption of ChatGPT for Bug Fixing among Professional Developers.
journalInnovation & Technology Advances volume1, number1 (dateJun. year2023), pages21–29.

[Guo et al(2022)]
authorpersonDaya Guo, personShuai Lu, personNan Duan, personYanlin Wang, personMing Zhou, and personJian Yin. year2022.
UniXcoder: Unified Cross-Modal Pre-training for Code Representation. In booktitleProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages7212–7225.

[Guo et al(2021)]
authorpersonDaya Guo, personShuo Ren, personShuai Lu, personZhangyin Feng, personDuyu Tang, personShujie Liu, personLong Zhou, personNan Duan, personAlexey Svyatkovskiy, personShengyu Fu, personMichele Tufano, personShao Kun Deng, personColin B. Clement, personDawn Drain, personNeel Sundaresan, personJian Yin, personDaxin Jiang, and personMing Zhou. year2021.
GraphCodeBERT: Pre-training Code Representations with Data Flow. In booktitleProceedings of the 9th International Conference on Learning Representations. pages1–18.

[Guo et al(2024)]
authorpersonJiawei Guo, personZiming Li, personXueling Liu, personKaijing Ma, personTianyu Zheng, personZhouliang Yu, personDing Pan, personYizhi Li, personRuibo Liu, personYue Wang, et al year2024.
CodeEditorBench: Evaluating Code Editing Capability of Large Language Models.
journalarXiv preprint arXiv:2404.03543 (year2024).

[Guo et al(2023)]
authorpersonQi Guo, personJunming Cao, personXiaofei Xie, personShangqing Liu, personXiaohong Li, personBihuan Chen, and personXin Peng. year2023.
Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study. In booktitle2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE). IEEE Computer Society, pages379–391.

[Hao et al(2024)]
authorpersonSichong Hao, personXianjun Shi, and personHongwei Liu. year2024.
Exploring the Potential of Pre-Trained Language Models of Code for Automated Program Repair.
journalElectronics volume13, number7 (year2024), pages1200.

[Hao et al(2023)]
authorpersonSichong Hao, personXianjun Shi, personHongwei Liu, and personYanjun Shu. year2023.
Enhancing Code Language Models for Program Repair by Curricular Fine-tuning Framework. In booktitle2023 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, pages136–146.

[Haque et al(2023)]
authorpersonMd Mahim Anjum Haque, personWasi Uddin Ahmad, personIsmini Lourentzou, and personChris Brown. year2023.
FixEval: Execution-Based Evaluation of Program Fixes for Programming Problems. In booktitle2023 IEEE/ACM International Workshop on Automated Program Repair. IEEE, pages11–18.

[Hidvégi et al(2024)]
authorpersonDávid Hidvégi, personKhashayar Etemadi, personSofia Bobadilla, and personMartin Monperrus. year2024.
CigaR: Cost-efficient Program Repair with LLMs.
journalarXiv preprint arXiv:2402.06598 (year2024).

[Hochreiter et al(1997)]
authorpersonSepp Hochreiter, personJ urgen Schmidhuber, and personCorso Elvezia. year1997.
Long Short-Term Memory.
journalNeural Computation volume9, number8 (year1997), pages1735–1780.

[Horváth et al(2023)]
authorpersonDániel Horváth, personViktor Csuvik, personTibor Gyimóthy, and personLászló Vidács. year2023.
An Extensive Study on Model Architecture and Program Representation in the Domain of Learning-Based Automated Program Repair. In booktitle2023 IEEE/ACM International Workshop on Automated Program Repair (APR). pages31–38.

[Hou et al(2023)]
authorpersonXinyi Hou, personYanjie Zhao, personYue Liu, personZhou Yang, personKailong Wang, personLi Li, personXiapu Luo, personDavid Lo, personJohn Grundy, and personHaoyu Wang. year2023.
Large Language Models for Software Engineering: A Systematic Literature Review.
journalarXiv preprint arXiv:2308.10620 (year2023).

[Hu et al(2024)]
authorpersonXueyu Hu, personKun Kuang, personJiankai Sun, personHongxia Yang, and personFei Wu. year2024.
Leveraging Print Debugging to Improve Code Generation in Large Language Models.
journalarXiv preprint arXiv:2401.05319 (year2024).

[Hu et al(2023)]
authorpersonXing Hu, personZhuang Liu, personXin Xia, personZhongxin Liu, personTongtong Xu, and personXiaohu Yang. year2023.
Identify and Update Test Cases When Production Code Changes: A Transformer-Based Approach. In booktitle2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, pages1111–1122.

[Hu et al(2019)]
authorpersonYang Hu, personUmair Z Ahmed, personSergey Mechtaev, personBen Leong, and personAbhik Roychoudhury. year2019.
Re-Factoring Based Program Repair Applied to Programming Assignments. In booktitle2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE Computer Society, pages388–398.

[Hu et al(2022)]
authorpersonYaojie Hu, personXingjian Shi, personQiang Zhou, and personLee Pike. year2022.
Fix Bugs with Transformer through a Neural-Symbolic Edit Grammar. In booktitleDeep Learning for Code Workshop.

[Huang et al(2023a)]
authorpersonDong Huang, personQingwen Bu, personJie M Zhang, personMichael Luck, and personHeming Cui. year2023a.
AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation.
journalarXiv preprint arXiv:2312.13010 (year2023).

[Huang et al(2023b)]
authorpersonKai Huang, personXiangxin Meng, personJian Zhang, personYang Liu, personWenjie Wang, personShuhao Li, and personYuqing Zhang. year2023b.
An Empirical Study on Fine-Tuning Large Language Models of Code for Automated Program Repair. In booktitle2023 38th IEEE/ACM International Conference on Automated Software Engineering. IEEE, pages1162–1174.

[Huang et al(2023c)]
authorpersonQing Huang, personJiahui Zhu, personZhenchang Xing, personHuan Jin, personChangjing Wang, and personXiwei Xu. year2023c.
A Chain of AI-Based Solutions for Resolving Fqns and Fixing Syntax Errors in Partial Code.
journalarXiv preprint arXiv:2306.11981 (year2023).

[Ishizue et al(2024)]
authorpersonRyosuke Ishizue, personKazunori Sakamoto, personHironori Washizaki, and personYoshiaki Fukazawa. year2024.
Improved Program Repair Methods using Refactoring with GPT Models. In booktitleProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1. pages569–575.

[Jiang et al(2018)]
authorpersonJiajun Jiang, personYingfei Xiong, personHongyu Zhang, personQing Gao, and personXiangqun Chen. year2018.
Shaping Program Repair Space with Existing Patches and Similar Code. In booktitleProceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis. pages298–309.

[Jiang et al(2023a)]
authorpersonNan Jiang, personKevin Liu, personThibaud Lutellier, and personLin Tan. year2023a.
Impact of Code Language Models on Automated Program Repair. In booktitleProceedings of the 45th International Conference on Software Engineering. pages1430–1442.

[Jiang et al(2021)]
authorpersonNan Jiang, personThibaud Lutellier, and personLin Tan. year2021.
CURE: Code-Aware Neural Machine Translation for Automatic Program Repair. In booktitleProceedings of the 43rd IEEE/ACM International Conference on Software Engineering. pages1161–1173.

[Jiang et al(2023b)]
authorpersonShuyang Jiang, personYuhao Wang, and personYu Wang. year2023b.
SelfEvolve: A Code Evolution Framework via Large Language Models.