# An Exploratory Study on Using Large Language Models for Mutation Testing

Bo Wang  0000-0001-7944-9182 Beijing Jiaotong UniversityBeijingChina
<EMAIL_ADDRESS>,  Mingda Chen  0009-0002-6283-1291 Beijing Jiaotong
UniversityBeijingChina<EMAIL_ADDRESS>,  Youfang Lin  Beijing Jiaotong
UniversityBeijingChina<EMAIL_ADDRESS>,  Mike Papadakis  University of
LuxembourgLuxembourg<EMAIL_ADDRESS>and  Jie M. Zhang  King’s
College LondonLondonUK<EMAIL_ADDRESS>
(2024)

###### Abstract.

The question of how to generate high-utility mutations, to be used for testing
purposes, forms a key challenge in mutation testing literature. Large Language
Models (LLMs) have shown great potential in code-related tasks but their
utility in mutation testing remains unexplored. To this end, we systematically
investigate the performance of LLMs in generating effective mutations w.r.t.
to their usability, fault detection potential, and relationship with real
bugs. In particular, we perform a large-scale empirical study involving 4
LLMs, including both open- and closed-source models, and 440 real bugs on two
Java benchmarks. We find that compared to existing approaches, LLMs generate
more diverse mutations that are behaviorally closer to real bugs, which leads
to approximately 18% higher fault detection than current approaches (i.e., 87%
vs. 69%) in a newly collected set of bugs, purposely selected for evaluating
learning-based approaches, i.e., mitigating potential data leakage concerns.
Additionally, we explore alternative prompt engineering strategies and the
root causes of uncompilable mutations, produced by the LLMs, and provide
valuable insights for the use of LLMs in the context of mutation testing.

Mutation Testing, Mutation Generation, Large Language Models

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††journal:
JACM††journalvolume: 37††journalnumber: 4††article: 111††publicationmonth:
8††ccs: Software and its engineering Software testing and debugging

##  1\. Introduction

Mutation testing is one of the most effective software testing techniques
(Hamlet, 1977; DeMillo et al., 1978, 1979; Jia and Harman, 2010; Papadakis et
al., 2019). It is performed by producing program variants by making simple
syntactic changes (i.e., mutations) in the source code of the origin program
(i.e., mutants). These variants form the objectives of testing, in a sense
requesting tests to trigger different behaviors on the original and the mutant
programs. A mutant is called killed or live depending on whether the test
execution leads to a different program output (observable behavior) from that
of the original program or not. The proportion of the killed mutants over the
entire set of mutants forms a test adequacy metric that is called mutation
score.

Since mutations are the objectives of testing, the effectiveness of the method
strongly depends on the set of mutants that one is using. Traditional
techniques use simple syntactic transformations (rules), called mutation
operators, that introduce syntactic changes in every code location that they
apply (Jia and Harman, 2010; Papadakis et al., 2019). We call these approaches
as rule-based approaches.

Unfortunately, rule-based approaches produce large numbers of redundant
mutants, i.e., mutants that do not contribute to the testing process
(Papadakis et al., 2016; Ammann et al., 2014) and thus exacerbate the method’s
computational overheads as well as the human effort involved in examining live
mutants. To address this issue, researchers leverage deep learning approaches
to form mutation operators based on the project’s history (Tufano et al.,
2019; Patra and Pradel, 2021; Tian et al., 2022), with the hope that these
will produce few but effective mutations. While interesting, these approaches
fail to produce effective mutations (Ojdanic et al., 2023b), perhaps due to
the fact that they are trained on a few data instances.

Nevertheless, the question of which mutants should be used when performing
mutation testing remains open due to the following challenges: (1) Forming
effective mutations. To be effective, mutations need to be syntactically
correct, which is not the case of most of the existing learning-based methods,
and be likely fault revealing (Chekam et al., 2020; Wen et al., 2019), i.e.,
have a good potential to couple with real faults. While existing methods are
effective, there is still significant room for improvement (Degiovanni and
Papadakis, 2022). (2) Forming natural mutations. Mutations need to align with
natural coding patterns and practices so that they align with developer
expectations, i.e., making them feel that they are worth being addressed
(Beller et al., 2021). It is a challenge to ensure that the mutated code is
not only syntactically correct but also aligns well with the realistic, common
coding patterns and practices observed among developers. (3) Forming diverse
and killable mutations. Determining whether a mutation is equivalent is
undecidable (Budd and Angluin, 1982), making it challenging to avoid the
generation of equivalent mutants, leading to poor scalability and waste of
resources (Zhang et al., 2013; Papadakis et al., 2015; Zhang et al., 2016;
Wang et al., 2021). At the same time, mutations need to be diverse and to
cover the entire spectrum of code and program behaviours.

To deal with these challenges, we aim at using Large-Language Models (LLMs),
which have been trained on big code and can produce human-like code (Chen et
al., 2021). Following existing studies (Guo et al., 2024; Ma et al., 2024;
Schäfer et al., 2023), we designed a default prompt that contains task
instructions, the code element to be mutated, the corresponding method code,
and a few-shot examples sampled from real-world bugs. We carried out a large-
scale evaluation involving 4 LLMs, 2 closed-source commercial and 2 open-
source ones, 440 real-world bugs from two Java benchmarks, i.e., Defects4J
(Just et al., 2014a) and ConDefects (Wu et al., 2023b), and compared with
existing techniques and tools.

Our evaluation includes both quantitative and qualitative analysis for the
mutations generated by LLMs, comparing them with existing approaches to assess
their cost and effectiveness. Additionally, we study how different prompt
engineering strategies and LLMs (i.e., GPT-3.5-Turbo (Achiam et al., 2023),
GPT-4-Turbo, CodeLlama-13b-Instruct (Roziere et al., 2023), and
StarChat-$\beta$-16b (Li et al., 2023a)) affect the effectiveness of the task.
Moreover, we analyze the root causes and error types for the non-compilable
mutants generated by LLMs.

Our results reveal that the best model for mutant generation is GPT-4 as it
outperforms the other LLMs in all metrics used, i.e., produces less equivalent
mutants, mutants with higher fault detection potential, as well as higher
coupling and semantic similarity with real faults. GPT-3.5 is the second best
in terms of fault detection potential, leading to a detection of 96.7% of bugs
in Defects4J and 86.7% in ConDefects. From the rest of the techniques, Major
is the best, with the potential to reveal 91.6% of bugs in Defects4J and 68.9%
in ConDefects, indicating an advantage of approximately 18% for the GPT on
ConDefects which is a newly and unseen dataset.

Another interesting finding of our analysis regards the diversity of the
mutants (measured in terms of newly introduced AST node types). In particular,
our results show that GPTs exhibit the greatest diversity, newly introducing
45 different AST node types, in contrast to just 2 used by traditional
approaches such as Major, and subsuming all existing approaches, indicating a
much better contextualization of the LLM-based approach.

We also analyzed the mutations generated by GPT that fail to compile, a common
issue with learning-based mutations, and found that they fall in 9 compilation
error types with the Usage of Unknown Methods and Code Structural Destruction
types being the most prevalent ones. This finding indicates that future
approaches could focus on directing the LLMs in choosing the right method
invocations when mutating.

We wrapped up our method into a mutation tool called Kumo, an LLM-based
mutation generation tool for Java. Based on the experimental data, we have
also built a dataset that contains high-quality LLM-generated mutants that can
be used not only in mutation testing but also in other bug-seeding
applications, such as fault localization and fault prediction. The
implementation of Kumo and the experimental data are available for open
science (kum, 2024).

In summary, our paper makes the following main contributions:

  * •

We investigate the applicability of LLMs in mutation testing. We perform
extensive and detailed comparative experiments to evaluate LLMs against
existing tools/methods. Our findings indicate that GPT models excel in
generating mutations that closely mimic real bugs.

  * •

We compare different prompts, and find that few-shot learning with suitable
code context archives best performance.

  * •

We analyze the error types of non-compilable mutations and figure out that
method invocation and member assessment are more likely to lead LLMs to
generate non-compilable mutations.

  * •

We build a high-quality dataset of Java mutations that includes manually
analysed equivalent mutants.

##  2\. Background and Related Work

Mutation Testing and Mutation Generation.  Mutation testing is a systematic
testing approach that aims to guide the creation of high-utility test suites.
It works by introducing syntactic modifications (i.e., mutations) into the
source code of the programs under test. By observing if test suites can detect
these changes (Hamlet, 1977; DeMillo et al., 1978) one can identify weaknesses
of their test suites, i.e., mutants that are not killed. Besides testing,
mutation analysis is a tool that can be used to support multiple applications
in the field of software engineering (Shi et al., 2019; Gligoric et al., 2013;
Linares-Vásquez et al., 2017; Wang et al., 2022; Qian et al., 2022; Kazerouni
et al., 2021; Groce et al., 2021; Wu et al., 2023a). Most commonly, mutations
are employed as substitutes for real-world bugs (Just et al., 2014b; Andrews
et al., 2005; Daran and Thévenod-Fosse, 1996; Namin and Kakarla, 2011), to
guide fault localization (Moon et al., 2014; Papadakis and Le Traon, 2015),
test prioritization (Shin et al., 2019) and program repair (Xiao et al., 2023,
2024).

All these tasks require high-utility mutations. This is because it is required
to compile and execute every mutation with tests, which is computationally
expensive. Therefore, generating redundant or non-compilable mutations, as
done by most of the existing learning-based approaches, increases the
computational cost involved (Zhang et al., 2018; Wang et al., 2017, 2021; Garg
et al., 2022). Additionally, when mutations are used as substitutes for real
defects, the mutations must be as syntactically close to the real bugs as
possible, making the mutations more natural (Hindle et al., 2016; Jimenez et
al., 2018). Moreover, when performing tasks such as fault localization (Moon
et al., 2014; Papadakis and Le Traon, 2015) and program repair (Le Goues et
al., 2011; Zhu et al., 2023; Jiang et al., 2018), the behavior of the
mutations needs to couple with real faults/bugs to allow these techniques to
be effective.

Traditional mutation testing approaches generate mutations by pre-defined
program transformation rules (i.e., mutation operators), which are referred to
rule-based. For example, one popular mutation operator is to change one
arithmetic binary operator with another one (e.g., $a+b\mapsto a-b$). The
majority of existing mutation testing approaches are rule-based, such as PIT
(Coles et al., 2016), Major (Just et al., 2011), JavaLanch (Schuler and
Zeller, 2009), Proteum (Delamaro et al., 1996), Milu (Jia and Harman, 2008),
AccMut (Wang et al., 2017), WinMut (Wang et al., 2021), etc. Then researchers
proposed using deep learning techniques to generate mutations. For instance,
DeepMutation (Tufano et al., 2019) utilizes machine sequence-to-sequence
neural translation methods, while LEAM (Tian et al., 2022) employs learning-
guided rule-based expansion. However, these approaches suffer from the
challenges in generating syntactic correct mutations and mutations with
diverse forms and often fail to generate syntactically valid mutations.
Perhaps the closest to our work is that of Degiovanni et al. who proposed
$\mu$Bert (Degiovanni and Papadakis, 2022). $\mu$Bert generates mutation by
masking the code elements to be mutated and then predicting the mutation via
Bert (Devlin et al., 2018).

Several recent studies have also explored the use of LLMs for mutation
generation (Tip et al., 2024; DEB et al., 2024). However, they have not been
thoroughly evaluated, as they only count the number of mutations they
introduce and compute the mutation scores achieved by some test suites. In
contrast, we thoroughly assess the utility of the mutations we are using
(through both a quantitative and qualitative approach), contrast it with the
key previous baseline methods, and address all experimental parameters and
concerns related to the use of LLMs, i.e., data leakage, cost, and added
value.

LLMs for Code.  Recently, Large Language Models (LLMs) have shown remarkable
potential in a wide range of natural language processing (NLP) (Brown et al.,
2020) and coding tasks (Wang et al., 2024; Fan et al., 2023). Existing LLMs
are mostly based on Transformer (Vaswani et al., 2017) and are trained by a
large corpus of data, including texts and source codes. LLMs are driven by
instructions in natural languages, i.e., prompts, which guide the models to
perform target tasks and are crucial to performance. The design of prompts
plays a crucial role in obtaining accurate and useful responses from these
models.

Closed-source LLMs are proprietary models developed and maintained by
companies or organizations, often offering advanced features and requiring a
subscription or payment for access, while open-source LLMs are freely
available models that allow users to view and modify them. For the models used
in our study, GPT-3.5-Turbo and GPT-4-Turbo are closed-source LLMs released by
OpenAI (Achiam et al., 2023), while CodeLlama and StarChat are open-source. An
open-source LLM is usually fine-tuned against a general LLM for specified
tasks. For example, CodeLlama (Roziere et al., 2023) is derived from Llama
(Touvron et al., 2023) by fine-tuning with extensive code datasets, while
StarChat is built on StarCoder (Li et al., 2023a) to enhance conversational
abilities.

##  3\. Study Design

###  3.1. Overview and Research Questions

Our paper intends to thoroughly investigate the capability of existing LLMs in
mutation generation. We design research questions covering the evaluation of
the default setting of our study, the impact of prompt engineering strategies
and models, and root cause analysis for underperforming aspects. Specifically,
we aim to answer the following five research questions.

  * •

RQ1: How do the LLMs perform in generating mutations in terms of cost and
usability?

  * •

RQ2: How do the LLMs perform in generating mutations in terms of behavior
similarity with real bugs?

  * •

RQ3: How do different prompts affect the performance in generating mutations?

  * •

RQ4: How do different LLMs affect the performance in generating mutations?

  * •

RQ5: What are the root causes and compilation error types for the generation
of non-compilable mutations?

###  3.2. Datasets

Table 1. Real Bugs Used in Our Experiment Dataset | Project | # of Bugs | Time Span
---|---|---|---
Defects4J | Math | 106 | 2006/06/05 - 2013/08/31
Lang | 65 | 2006/07/16 - 2013/07/07
Chart | 26 | 2007/07/06 - 2010/02/09
Time | 27 | 2010/10/27 - 2013/12/02
Closure | 133 | 2009/11/12 - 2013/10/23
Mockito | 38 | 2009/06/20 - 2015/05/20
ConDefects | — | 45 | 2023/09/01 - 2023/09/30
Total | — | 440 | 2006/07/16 - 2023/09/30

We intend to evaluate our approach with real bugs, and thus we need to use bug
datasets with the following properties:

  * •

The datasets should comprise Java programs, as existing methods are primarily
based on Java, and we need to compare with them.

  * •

The bugs of the datasets should be real-world bugs so that we can compare the
difference between mutations and real bugs.

  * •

Every bug in datasets has the correctly fixed version provided by developers
so that we can mutate the fixed version and compare them with the
corresponding real bugs.

  * •

Every bug is accompanied by at least one bug-triggering test because we need
to measure whether the mutations affect the execution of bug-triggering tests.

To this end, we employ the Defects4J v1.2.0 (Just et al., 2014a) and
ConDefects (Wu et al., 2023b) to evaluate the mutation generation approaches,
shown as Table 1. In total, we conduct the experiments on 440 bugs.

Defects4J is a widely used benchmark in the field of mutation testing (Ojdanic
et al., 2023a; Just et al., 2014b; Kaufman et al., 2022; Degiovanni and
Papadakis, 2022; Tian et al., 2022; Kim et al., 2022), which contains history
bugs from 6 open-source projects of diverse domains, ensuring a broad
representation of real-world bugs. In total, these 6 projects contain 395 real
bugs. However, from Table 2 and Table 1, we observe the time spans of the
Defects4J bugs are earlier than the LLMs’ training time, which may introduce
data leakage. Therefore, we supplement another dataset, ConDefects (Wu et al.,
2023b), designed to address the data leakage concerns. ConDefects consists of
tasks from AtCoder (atc, 2024) programming contest. To prevent data leakage,
we exclusively use bugs reported after the LLMs’ release date, specifically
those identified on or after August 31, 2023, and in total we collect 45 Java
programs from ConDefects.

###  3.3. Mutation Generation via LLMs

Given the location of a Java program, we extract essential information (e.g.,
the context method or the corresponding unit tests) to formulate prompts
instructing mutation generation, then feed prompts to the selected LLM. Once
the LLM returns its responses, we filter out mutations that are non-
compilable, redundant, or identical to the original code, and return the
generated mutations. As shown in Section 3.4, our study supports a
comprehensive set of metrics for evaluating mutations.

####  3.3.1. Models

We aim to comprehensively compare the existing LLMs in their ability to
generate mutations from various perspectives. This includes evaluating the
capabilities of models that are fine-tuned on different base models and
contrasting the performance of commercial closed-source models with open-
source alternatives. In our study, we select LLMs based on the following
criteria. (1) The LLM must have coding capabilities, i.e., understanding,
generating, and transforming code. (2) The LLM must support interaction in
chat mode, allowing for conversational code transformation. (3) The LLM must
offer accessible APIs for integration.

Finally, we select the following models: GPT-3.5-Turbo (Achiam et al., 2023),
GPT-4-Turbo (Achiam et al., 2023), CodeLlama-13b-Instruct (Roziere et al.,
2023), and StarChat-$\beta$-16b (Li et al., 2023a)). The details of these
models are shown as Table 2. These LLMs cover open-source (i.e., StarChat and
CodeLlama) and closed-source (i.e., GPT-3.5-Turbo and GPT-4-Turbo), and
various kinds of base models.

Table 2. Studied LLMs Model Type | Model |  Training Data Time |  Release Time | Size
---|---|---|---|---
Closed | GPT-3.5-turbo | 2021/09 | 2023/03 | 20B
GPT-4-turbo | 2023/04 | 2023/06 | 110B
Open | CodeLlama-13b-Instruct | — | 2023/08 | 13B
StarChat-$\beta$-16b | — | 2023/06 | 16B

####  3.3.2. Prompts

Prompts are crucial for LLMs because they guide the models’ responses and
determine the direction of the generated output. A well-crafted prompt can
significantly influence the utility, relevance, and accuracy of the code
produced by an LLM. This subsection introduces how we design the prompts.

To design effective prompts for mutation generation, we follow the best
practices suggesting prompts should comprise four aspects, namely,
Instruction, Context, Input Data, and Output Indicator (Guo et al., 2024; Ma
et al., 2024). In the Instructions, we direct the LLMs to generate mutants for
the target code element. In the Context, we clearly state that mutant is the
concept of mutation testing, and additionally provide various sources of
information to observe the impact on the performance of LLMs, including the
whole Java method surrounding the target code element and few-shot examples
sampled from real-world bugs from another benchmark. In the Input Data, we
specify the code element to be mutated and the number of mutants to generate.
In the Output Indicator, we specify the JSON file format for mutation outputs,
facilitating further experimentation. Figure 1 shows the default prompt
template in our study.

In addition, our prompts need few-shot examples, which should originate from
real bugs enabling LLMs to learn how to mutate code from real bugs. This
allows the model to understand the context and logic behind the changes,
improving its ability to generate relevant and effective mutations.

To avoid the few-shot examples leaking information, beyond Defects4J and
ConDefects, we employ another benchmark QuixBugs (Lin et al., 2017), which
comprises 40 real-world Java bugs and is commonly used by the testing and
debugging community (Zhu et al., 2021, 2023; Jiang et al., 2023; Xia and
Zhang, 2022; Li et al., 2023b). Following the existing study on few-shot
examples (Deng et al., 2024), we sample 6 of the bugs from QuixBugs.

To guarantee the diversity of the examples, we randomly select one bug from
the dataset and assess whether its modification pattern is similar to the
examples already collected. If it is different, we add it to our collection
and continue, until we have collected all 6 examples. Table 3 shows these
examples.

Table 3. Default Few-Shot Examples from QuixBugs Correct Version | Buggy Version
---|---
n = (n & (n - 1)); | n = (n ^ (n - 1));
while (!queue.isEmpty()) | while (true)
return depth==0; | return true;
ArrayList r = new ArrayList();
r.add(first).addll(subset);
to_add(r); | to_add.addAll(subset);
c = bin_op.apply(b,a); | c = bin_op.apply(a,b);
while(Math.abs(x-approx*approx)>epsilon) | while(Math.abs(x-approx)>epsilon)
Figure 1. The Default Prompt Template

###  3.4. Evaluation Metrics

To assess the utility of the LLM-generated mutations, we should define a set
of metrics that comprehensively evaluate them. Therefore, we designed several
categories of metrics capturing different aspects as follows: Cost Metrics,
Usability Metrics, and Behavior Metrics. Besides these metrics, we also
additionally report the Mutation Count and Mutation Score. Note that both of
them cannot directly reflect the quality of generated mutations. For example,
a higher mutation score does not indicate a higher quality, which may also be
caused by generating a large proportion of mutants that are easily killed.

To better understand the metrics, we first illustrate the categories of
mutations. Let all generated mutations be the set $A$, where LLMs may generate
syntactically incorrect mutations that can not pass compilation. Thus we
denote the compilable mutations as the set $C$. Note that the set $C$ contains
two types of mutations that should be removed, the useless mutations (denoted
as $U$) and equivalent mutations (denoted as $E$). The useless mutations refer
to those that are identical to the original code or other mutations. In
mutation testing, the ideal set of mutations to employ is from the set
$(C-E-U)$, excluding equivalent mutations that have no contributions to
discovering the weakness of tests. However, determining the equivalent
mutations set $E$ is undecidable, so we follow the common practice and use the
set $(C-U)$ as a practical approximation to perform mutation testing (Diniz et
al., 2021; Tian et al., 2022; Zhang et al., 2022).

####  3.4.1. Cost Metrics

The Cost Metrics indicate time and economic costs in mutation generation,
which contains the following two metrics.

Average Generation Time:  the average duration needed to produce each
mutation, measuring the efficiency of an approach.

Cost per 1K Mutations:  the average money cost in US dollars to generate 1k
mutations. For GPT-3.5 and GPT-4, we calculate the average cost by API usage,
while for the open-source models, we employ the costs of renting cloud
servers.

####  3.4.2. Usability Metrics

The Usability Metrics serve as key indicators of the practical utility of
generated mutations.

Compilability Rate:  the proportion of mutations that successfully compile,
computed as: ${|C|}/{|A|}$. Because LLMs cannot guarantee the generation of
entirely correct code and the compilation process can be time-consuming, we
prefer the approach with a higher comparability rate.

Useless Mutation Rate:  the proportion of useless mutations, computed as:
$|U|/|A|$. LLMs sometimes generate duplicate mutations or mutations that are
exactly the same as the original code. While traditional approaches rarely
generate useless mutations, LLMs suffer from issues of misunderstanding and
repetition, leading to generating these useless mutations.

Equivalent Mutation Rate:  the proportion of equivalent mutation, computed as:
$|E|/|A|$. Equivalent mutations are semantically equivalent to the original
program, which impacts the accuracy of the mutation score. Due to the
undecidable nature of identifying whether a mutation is equivalent, we sample
significant portions of all mutations and judge them manually. More
specifically, given the population to be the set $A$, we set the confidence
level to be 95% and the margin of error to be 5%, to get a subset of mutations
to be manually inspected.

AST Node Diversity:  We prefer the set of mutations with diverse forms,
therefore, we parse the code before and after mutation, and check the
diversity of newly introduced AST nodes. The approach with a greater variety
of AST node types is considered superior.

####  3.4.3. Behavior Similarity with Real Bugs

Behavior similarity refers to the degree of similarity in functionality and
execution between a mutation and its corresponding real bug. Because capturing
all program behaviors is undecidable, following existing studies, we thus
employ the following metrics as a proxy to measure behavior similarity
(Ojdanic et al., 2023a; Papadakis et al., 2018; Khanfir et al., 2023).

Real Bug Detectability:  Given the location of a real bug (which may occupy
multiple lines), let a mutation generation approach generate a set of
mutations, if the tests that kill these mutations can also kill this real bug,
we say that the mutation generation approach can “detect” this real bug. Given
a set of real bugs and a mutation generation approach, Real Bug Detectability
quantifies the number of bugs detected by this mutation approach.

Coupling Rate:  Mutation testing is based on the fault coupling effect, i.e.,
mutations though are simple syntactic changes, coupled to more complex bugs
(Offutt, 1992). To describe the extent to which a mutation couples to its
corresponding real bug, we adopt the Coupling Rate following the existing
study (Just et al., 2014b). Given a real bug and a corresponding mutant
generated at the same location, the mutant is deemed a coupled mutant if the
set of the test cases killing the mutant overlaps with the set of the bug-
triggering test cases. The Coupling Rate refers to the proportion of coupled
mutants within $(C-U)$.

Ochiai Coefficient:  approximates the semantic similarity by passing and
failing test cases (Offutt and Untch, 2001), which is also widely used in the
field of fault localization (Zou et al., 2019). Following the existing study
(Ojdanic et al., 2023a), we employ the Ochiai Coefficient to calculate how a
mutation resembles its corresponding real bug. More concretely, let $M$ and
$B$ be a mutation and its corresponding real bug, and let $fT_{M}$ and
$fT_{B}$ be the set of their failing test cases respectively. $M$’ coefficient
is computed by the following formula:

(1) $Ochiai(M,B)=\frac{|fT_{M}\cap fT_{B}|}{\sqrt{|fT_{M}|\times|fT_{B}|}}$

Therefore, if $M$’s Ochiai Coefficient is $1$, it is semantically equal to its
corresponding bug $B$.

###  3.5. Experiment Settings

####  3.5.1. Settings for Mutation Generation

One major target of our study is to investigate the similarity between LLM-
generated mutations and real bugs. Therefore, we generate mutations within the
context of real bugs to enable such a comparison. To determine the number of
mutations generated in a prompt, we adopt the setting of PIT (Coles et al.,
2016). We use PIT-generated mutations with the full set of operators and find
that PIT generates 1.15 mutations per line of code. Therefore, based on the
given code context, we generate one mutation per line. We explore the
influence of context length on mutation generation in Section 5.

####  3.5.2. Settings for RQ1-RQ2

In the first part of our study, we intend to qualitatively and comparatively
evaluate the capability of LLMs in generating mutations. We adopt the default
settings, which contain two models (i.e., GPT-3.5-Turbo and CodeLlama), and
use the default prompt template, generating mutations on the universal set of
the 405 bugs in Section 3.2.

To understand the performance of the LLMs, we adopt LEAM (Tian et al., 2022),
$\mu$Bert (Degiovanni and Papadakis, 2022), PIT (Coles et al., 2016), and
Major (Just et al., 2011) as baselines. Among these approaches, LEAM is the
state-of-the-art mutation generation approach which employs a model trained
from 13 million real Java bugs. While $\mu$Bert is based on Bert (Devlin et
al., 2018), a masked pre-trained language model that can not conversational
respond to human-like text and code.PIT (Coles et al., 2016) and Major (Just
et al., 2011) are the popular traditional mutation testing tools with human-
predefined simple mutation operators. Note that for PIT and Major, we employ
all mutation operators.

####  3.5.3. Settings for RQ3-RQ4

In the second part of our study, we intend to explore how different prompts
and different models impact the performance of mutation generation. Due to
budget limitations, we sample a subset of 105 bugs from our dataset, which
consists of 10 sampled bugs from each Defects4J project and all ConDefects
bugs.

In RQ3, for the prompts exploration, we use the GPT-3.5-Turbo model and switch
different prompt templates. More specifically, we modify the default prompt
template by adding and removing different sources of information in the
Context, and we finally get 4 types of prompts, shown as follows:

  1. (1)

Prompt 1 (P1): The Default Prompt. P1 is the default prompt, shown as Figure
1.

  2. (2)

Prompt 2 (P2): P1 without Few-Shot Examples. P2 is based on P1 by removing all
the examples.

  3. (3)

Prompt 3 (P3): P2 without Whole Java Method. P3 is based on P2 by removing the
surrounding Java method code, only keeping the target code element to be
mutated.

  4. (4)

Prompt 4 (P4): P1 with Unit Tests. Based on the default prompt P1, P4 further
adds the source code of corresponding unit tests of the target method into
Context.

In RQ4, for the LLMs exploration, we use the default prompt and investigate
the performance of GPT-3.5-Turbo, GPT-4-Turbo, CodeLlama-13b-Instruct, and
StarChat-$\beta$-16b.

####  3.5.4. Settings for RQ5

Generating non-compilable mutations incurs additional meaningless costs, so in
the third part of our study, we study the root causes behind LLM generating
non-compilable mutations. Specifically, we sample 384 non-compilable mutations
of each mutation generation approach and analyze their reason rejected by the
Java compiler and the features of their surrounding code context.

##  4\. Evaluation Results

We performed our experimental analysis on two cloud servers, each of which is
equipped with 2 NVIDIA GeForce RTX 3090 Ti, 168GB memory, and a 64 Core CPU
Intel(R) Xeon(R) Platinum 8350C.

Table 4. Overall Performance of All the Mutation Generation Techniques Metric Category | Metric | GPT 3.5 | CodeLlama-13b | LEAM |  $\mu$BERT | PIT | Major
---|---|---|---|---|---|---|---
D4J | CD | D4J | CD | D4J | CD | D4J | CD | D4J | CD | D4J | CD
— | Mut. Score | 0.731 | 0.734 | 0.716 | 0.698 | 0.529 | 0.682
Mut. Count | 351,557 | 303,166 | 671,737 | 141,031 | 1,032,392 | 314,694
Cost | Avg. Gen. Time | 1.79 | 9.06 | 3.06 | 2.23 | 0.017 | 0.083
$ Cost per 1K Mut. | 0.288 | 0.577 | 0.195 | — | — | —
Usability | Compilability Rate | 61.7% | 79.6% | 75.8% | 73.3% | 33.8% | 50.9% | 21.9% | 26.6% | — | — | 98.3% | 92.8%
Useless Mut. Ratio | 11.6% | 9.2% | 41.0% | 9.3% | 0.5% | 9.6% | 1.8% | 4.6% | — | — | 0.0% | 0.0%
Eq. Mut. Rate | 2.3% | 2.1% | 1.3% | 0.7% | 1.6% | 1.0% | 2.1% | 3.1% | — | — | 0.5% | 0.8%
Behavior | Real Bug Detectability | 0.967 | 0.867 | 0.906 | 0.667 | 0.767 | 0.489 | 0.785 | 0.378 | 0.511 | 0.489 | 0.916 | 0.689
Coupling Rate | 0.416 | 0.625 | 0.398 | 0.612 | 0.267 | 0.318 | 0.409 | 0.581 | 0.133 | 0.593 | 0.342 | 0.479
Ochiai Coefficient | 0.638 | 0.689 | 0.390 | 0.378 | 0.286 | 0.400 | 0.230 | 0.578 | 0.271 | 0.489 | 0.519 | 0.600

###  4.1. RQ1: Performance on Cost and Usability

####  4.1.1. Cost

For each technique, we record the generation duration and the cost for API
usage or rending GPU servers, shown as the Cost lines of Table 4. In addition,
we also report the mutation count and mutation score, although they do not
fully reflect the overall quality of the mutation set.

From the table, we can find that GPT-3.5 and CodeLlama-30b-Instruct generate
351,557 and 303,166 mutations111 Although GPT-3.5 and CodeLlama-13b are
instructed to generate the same number of mutations, they do not consistently
follow the instructions, thereby yielding different numbers of mutations.,
with a mutation score of 0.731 and 0.734, respectively. Traditional approaches
PIT and Major yield the lowest mutation score among all the approaches.

In terms of cost, GPT-3.5 and CodeLlama-13b cost 1.79s and 9.06s to generate
mutations, respectively. While LEAM, $\mu$Bert, PIT, and Major cost 3.06s,
2.23s, 0.017s, and 0.083s, respectively, indicating that previous approaches
are faster than LLM-based approaches in generating mutations. In terms of
economic cost, we only record the API usage cost of GPT-3.5 and the cloud
server cost for running CodeLlama and LEAM, because the remaining approaches
barely need GPU resources. The result indicates that renting cloud servers to
run the open-source models costs more than using API for GPT-3.5.

Finding 1: Previous approaches cost less than LLM-based approaches in
generating mutations. For example, in terms of mutation generation time,
GPT-3.5 takes 1.79s to generate one mutation, CodeLlama-13b takes 9.06s, while
PIT and Major cost 0.017s and 0.083s, respectively. Regarding economy cost,
traditional approaches barely need GPU resources, but GPT-3.5 costs 0.288$ to
generate 1,000 mutations using API, and running CodeLlama-13b on a cloud
server costs 0.577$.

####  4.1.2. Mutation Usability

The usability of mutations (i.e., whether the mutations can be used to
calculate mutation score) is affected by the capabilities of LLMs in correctly
understanding the task and generating compilable, non-duplicate, and non-
equivalent mutations. The results are shown as the Usability rows in Table 4.
Due to the potential threat of data leakage, we represent the data of
Defects4J and Condefects separately. Note that the mutations of PIT are in the
form of Java bytecode and cannot be transformed back to the source level, so
we skip PIT when answering this RQ.

Compilability Rate: We observe that GPT-3.5 achieves a compilability rate of
61.7% for Defects4J and 79.6% for ConDefects datasets. While CodeLlama-13b
achieves a higher compilability rate of 75.8% for Defects4J and 73.3% for
ConDefects. The rule-based approach Major achieves the highest compilabity
rate (i.e., 98.3% on Defects4J and 92.8% on ConDefects). Note that though
Major operates under simple syntax rules, it can still generate non-compilable
mutants. For example, the Java compiler’s analyzer rejects constructs like
while(true)

Useless Mutation Rate: We also observe that LLMs generate a significant
portion of duplicate and useless mutations. For example, for Defects4J,
GPT-3.5 generates 11.6% useless mutations, accounting for 18.8% of all
compilable mutations (i.e., 11.6%/61.7%). In contrast, LEAM, $\mu$Bert, and
Major seldom generate duplicate mutations, which is likely due to these
approaches being governed by strict grammar rules that limit the production of
redundant mutations. For example, Major iteratively applies different mutation
operators producing no redundant mutants.

Table 5. Sample Results for Equivalent Mutations Dataset | Type | GPT-3.5 | CL-13b | LEAM |  $\mu$Bert | Major
---|---|---|---|---|---|---
Defecs4J | # All | 351,332 | 302,941 | 671,342 | 140,753 | 314,527
# Sampled | 384 | 384 | 384 | 384 | 384
# Eq. Mut. | 9 | 5 | 6 | 8 | 2
ConDefects | # All | 225 | 225 | 395 | 278 | 167
# Sampled | 143 | 143 | 196 | 162 | 123
# Eq. Mut. | 3 | 1 | 2 | 5 | 1

Equivalent Mutation Rate: Equivalent mutations change the source code
syntactically without affecting the program’s functionality, thereby affecting
the accuracy of mutation score calculations. Due to the undecidable nature of
identifying equivalent mutations, it is not feasible to design an algorithm
that can completely detect them. Therefore, we sample a subset of mutations
for each technique and manually assess whether they are equivalent. To
maintain a 95% confidence level and a 5% margin of error, we sampled a fixed
number of mutations from each approach’s output, to calculate the Equivalent
Mutation Rate. Following the protocol of existing studies (Islam et al., 2019;
Guo et al., 2024), two authors, each with over five years of Java programming
experience, first received training on the equivalent mutants, and then
independently labeled the mutations. The final Cohen’s Kappa coefficient score
(Viera et al., 2005) is more than 90%, indicating a high level of agreement.
Table 5 shows the number of sampled mutations and the number of identified
equivalent mutations.

We again observe that Major performs the best in generating non-equivalent
mutants. In particular, as shown in the bottom row of Usability section in
Table 4, for GPT-3.5, 2.3% of mutations generated on Defects4J are equivalent,
while the rate is 2.1% on ConDefects. CodeLlama-13b shows lower rates, with
1.3% and 0.7% on Defects4J and ConDefects, respectively. LEAM generates
equivalent mutations at rates of 1.6% on Defects4J and 1.0% on ConDefects.
$\mu$Bert produces 2.1% on Defects4J and 3.1% on ConDefects. Major achieves
the lowest rates, with only 0.5% on Defects4J and 0.8% on ConDefects.

When comparing the two datasets Defects4J and ConDefects for the three
usability aspects introduced above, we observe that the metrics vary between
the Defects4J and ConDefects datasets for each mutation approach. This
variation may be attributed to differences in the complexity and structure of
bugs present in each dataset. Mutations in Defects4J may involve APIs and
global variables that are not explicitly mentioned in prompts, making the
context more complex. This complexity results in both metrics being inferior
on Defects4J compared to ConDefects.

Finding 2: Major outperforms LLM-based and other learning-based methods in
terms of compilability rate, useless mutation rate, and equivalent mutation
rate. In particular, GPT-3.5 and CodeLlama-13b exhibit compilability rates of
61.67% and 75.86%, useless mutation rates of 11.60% and 40.96%, and 2.28% and
1.14% equivalent mutations, respectively. In contrast, Major excels with a
98.33% compilability rate, 0% useless mutations, and 0.59% equivalent mutants.

Figure 2. Number of types and examples of newly introduced AST Nodes from
different mutation approaches.

####  4.1.3. AST Node Dirversity

To assess the diversity of AST node types, we first determine if a mutation
involves deletion (i.e., removing a code element). For the non-deletion
mutations, we parse the code before and after mutation by Javalang (jav,
2024), to observe which new AST node types are introduced. A larger number of
newly introduced AST node types indicates richer diversity.

Figure 2 shows the results, which reveal that LLM-based mutation approaches
introduce more types of new AST nodes than other approaches. GPT-3.5 exhibits
the greatest diversity, introducing 45 new AST node types, and subsuming all
other approaches. Traditional approach Major introduces only two new types of
AST nodes, which is not surprising as the mutation operators are a fixed set
of simple operations.

We also check the AST node diversity of the examples we provided in the LLM
prompts (see Table 3), as shown by the “Few-shot Example” in Figure 2.
Interestingly, we find that these examples introduce only one new type of AST
node. This indicates that the LLMs are creative in generating mutations,
producing significantly more diverse mutations than the examples provided to
them.

Table 6. Ratio of Deletion Mutations and the Top-3 Newly Introduced AST Node
Types for Each Approach.

Top K | GPT-3.5 | CodeLlama-13b | LEAM |  $\mu$BERT | Major
---|---|---|---|---|---
Del. | 5.1% | 0.3% | 49.7% | 0.2% | 17.6%
1st | LT (47.6%) | LT (49.5%) | LT (23.6%) | LT (53.8%) | ST (59.9%)
2nd | BO (24.7%) | BO (35.5%) | MI (17.6%) | MR (22.7%) | LT (40.1%)
3rd | MI (8.2%) | MI (4.1%) | MR (16.3%) | MI (12.5%) |  —

ST: Statement, BO: BinaryOperation, LT: Literal, MI: MethodInvocation, MR:
MemberReference

We further analyzed the distribution of deletions and AST Node types across
different approaches, as shown in Table 6, which highlights the top-3 most
frequent AST node types. For the proportion of deletions, GPT-3.5 and
CodeLlama-13b generate 5.1% and 0.3% deletions, respectively. Nearly half of
the mutations of LEAM are deletions, which may be the reason for generating
many non-compilable mutations. Major generates deletions on restricted
locations and results in 17.6% deletions. $\mu$Bert replaces the masked code
elements with BERT, thereby rarely having deletions.

Regarding AST Node types, the most frequent new AST node type of all the
learning-based approaches is Literal (e.g., $a>b\mapsto\text{true}$). However,
for the rule-based approach Major, the most frequent node type is Statement
(e.g., replacing a statement with an empty statement). Note that although
these mutations are not with the Deletion operator, there exists a significant
portion of producing empty statements, leading to a large proportion.

Finding 3: LLM-based mutation approaches introduce more types of new AST nodes
than other approaches. The mutations generated by GPT-3.5 introduce 45 new
types of AST nodes, demonstrating the highest level of diversity.

###  4.2. RQ2: Behavior Similarity

The bottom three rows of Table 4 present a comparative evaluation of Behavior
Metrics for mutation generation approaches.

####  4.2.1. Real Bug Detectability

GPT-3.5 detects 382 bugs of the 395 Defects4J bugs and 39 bugs of all 45
ConDefects bugs, i.e., 96.7% Defects4J bugs and 86.7% ConDefects bugs could be
revealed by these mutations, achieving the best performance. CodeLlama-13b
detects 358 bugs from Defects4J (i.e., 90.6%) and 30 bugs from ConDefects
(i.e., 66.7%), respectively. Major achieves the second-best performance by
detecting 362 Defects4J bugs (i.e., accounting for 91.6%) and 31 ConDefects
bugs (i.e., accounting for 68.9%).

Finding 4: GPT-3.5 has the best performance in Real Bug Detectability,
identifying 96.7% of bugs in Defects4J and 86.7% in ConDefects. Among the
other approaches, Major performs best, detecting 91.6% of bugs in Defects4J
and 68.9% in ConDefects.

####  4.2.2. Coupling Rate

The Coupling Rate measures the degree of coupling between the generated
mutations and their corresponding real bugs. GPT-3.5 exhibits coupling rates
of 0.416 on Defects4J and 0.625 on ConDefects, respectively, achieving the
best performance on both datasets. While CodeLlama-13b achieves coupling rates
of 0.398 and 0.612, respectively.

Finding 5: GPT-3.5 leads in Coupling Rate with scores of 0.416 on Defects4J
and 0.625 on ConDefects. $\mu$Bert follows closely with 0.409 on Defects4J and
0.581 on ConDefects.

####  4.2.3. Ochiai Coefficient

The Ochiai Coefficient measures the semantic similarity between mutations and
real bugs. GPT-3.5 leads with coefficients of 0.638 on Defects4J and 0.689 on
ConDefects, outperforming CodeLlama-13b, which scores 0.39 and 0.378 on the
respective datasets. Despite the notable performance gap between the two,
their results are consistent across datasets. Major ranks second with
coefficients of 0.519 on Defects4J and 0.6 on ConDefects.

Finding 6: GPT-3.5 leads in Ochiai Coefficient performance with scores of
0.519 on Defects4J and 0.6 on ConDefects. Major closely follows, matching
these scores and securing the second-best ranking among the approaches.

###  4.3. RQ3: Impacts of Different Prompts

Table 7. Comparison Results of Different Prompts and LLMs Metric Category | Metric | GPT-3.5 | Closed Source LLM | Open Source LLM
---|---|---|---|---
P1 | P2 | P3 | P4 | GPT-3.5 | GPT-4 | StarChat-16b | CodeLlama-13b
— | Mutation Count | 47,666 | 47,905 | 41,548 | 39,638 | 47,666 | 23,526 | 25,496 | 41,808
Mutation Score | 0.705 | 0.703 | 0.731 | 0.782 | 0.705 | 0.702 | 0.538 | 0.734
Cost | $ Cost per 1K Mutations | 0.288 | 0.288 | 0.287 | 0.289 | 0.288 | 18.701 | 0.876 | 0.577
Average Generation Time | 1.79 | 1.78 | 1.81 | 1.88 | 1.79 | 42.25 | 13.76 | 9.06
Usability | Compilability Rate | 65.7% | 64.1% | 61.9% | 60.8% | 65.7% | 79.9% | 58.2% | 74.1%
Useless Mutation Ratio | 13.7% | 11.3% | 22.3% | 15.0% | 13.7% | 7.2% | 38.3% | 39.2%
Equivalent Mutation Rate | 1.5% | 1.2% | 1.2% | 1.5% | 1.5% | 0.9% | 2.1% | 1.0%
Behavior | Real Bug Detectability | 0.895 | 0.857 | 0.714 | 0.629 | 0.895 | 0.905 | 0.324 | 0.724
Coupling Rate | 0.140 | 0.136 | 0.096 | 0.096 | 0.140 | 0.143 | 0.101 | 0.110
Ochiai Coefficient | 0.590 | 0.571 | 0.438 | 0.362 | 0.590 | 0.619 | 0.152 | 0.286

The left half of Table 7 presents the comparative results of GPT-3.5 via
different prompts listed in Section 3.5.3. Prompts P1 to P3 progressively
simplify, each containing less information than its predecessor, while P4 is
the most complex, enhancing P1 with test suite codes.

Overall, P1, the default prompt, excels in Compilability Rate and all Behavior
Metrics. P2, created by removing few-shot examples from P1, leads in Average
Generation Time, Useless Mutation Ratio, and Equivalent Mutation Rate,
suggesting improved quality in compilable mutations. P3, provided only with
the code element to be mutated, achieves the lowest cost for using the least
tokens. Conversely, P4, which extends P1 with test suite codes, shows the
lowest performance across all metrics, suggesting that GPT-3.5 cannot
effectively utilize test suite data to enhance mutation quality.

Finding 7: The prompt with the whole method and few-shot examples as context
(i.e., P1) archives the best performance across all behavior metrics, whereas
adding the code of test suites (i.e., P4) decreases the performance.

###  4.4. RQ4: Impacts of Different LLMs

To answer this RQ, we add two extra LLMs, GPT-4 and StarChat-16b, and compare
their results with the two default LLMs, GPT-3.5 and Code Llama-13b. The right
half of Table 7 shows comparative results across models using the default
prompt. We observe that closed-source LLMs generally outperform others in most
metrics. GPT-3.5 excels in Mutation Count, Generation Cost per 1K Mutations,
and Average Generation Time, ideal for rapidly generating numerous mutations.
GPT-4 leads in all Usability Metrics, and Behavior Similarity Metrics,
demonstrating its effectiveness in code-related tasks, although its
enhancements over GPT-3.5 in Behavior Metrics are trivial. Between the two
open-source LLMs, despite StarChat-$\beta$-16b having more parameters,
CodeLlama-13b outperforms in all metrics. This suggests that model
architecture and training data quality significantly impact performance beyond
just the number of parameters.

Finding 8: Among the four LLMs we study, GPT-3.5 excels in generating a large
number of mutations with lower costs, while GPT-4 outperforms in all the
Usability Metrics, and Behavior Similarity Metrics.

###  4.5. RQ5: Root Causes and Error Types of Non-Compilable Mutations

Non-compilable mutations require a compilation step to filter out, which
results in wasted computational resources. As mentioned in Section 4.1, LLMs
generate a significant number of non-compilable mutations. This RQ analyzes
the types of errors and potential root causes of these non-compilable
mutations. Following the setting of the previous steps, we first sample 384
non-compilation mutations from the outputs of GPT-3.5, ensuring the confidence
level is 95% and the margin of error is 5%. From the manual analysis of these
non-compilable mutations, we identified 9 distinct error types, as shown in
Table 8.

Table 8. Error Types of Non-Compilable Mutations ID | Error Type | % | ID | Error Type | %
---|---|---|---|---|---
1 | Usage of Unknown Methods | 27.3% | 6 | Type Mismatch | 7.6%
2 | Code Structural Destruction | 22.9% | 7 | Incorrect Initialization | 3.1%
3 | Incorrect Method Parameters | 12.8% | 8 | Incorrect Location | 3.1%
4 | Usage of Unknown Variables | 11.2% | 9 | Incorrect Exceptions | 2.3%
5 | Usage of Unknown Types | 9.6% | — | — | —

Shown as Table 8, the most common error type, Usage of Unknown Methods,
accounts for 27.34% of the total errors, revealing the hallucination issue of
generative models (Huang et al., 2023). Code Structural Destruction is the
second most common error, accounting for 22.92%, indicating that ensuring
generated codes are syntactically correct remains a challenge for current
LLMs. This result suggests that there is still significant room for
improvement in current LLMs.

Finding 9: The mutations generated by GPT-3.5 have 9 main compilation error
types, with Usage of Unknown Methods and Code Structural Destruction being the
top-2 most prevalent types.

(a) GPT-3.5

(b) CodeLlama-13b

(c) LEAM

(d) $\mu$Bert

(e) Major

Figure 3. Distribution of the AST Node Types of Origin Code for Non-Compilable
Mutations

To analyze which types of code are prone to causing LLMs to generate non-
compilable mutations, we examined the code locations of all non-compilable
mutations generated by GPT-3.5, CodeLlama, LEAM, and $\mu$Bert in Section 4.1,
as shown in Figure 3. For all approaches, the code locations with
MethodInvocation and MemberReference are the top-2 prevent AST Node types. In
particular, there are more than 30% of non-compilable mutations occur at the
location with MethodInvocation, and 20% occur at the location with
MemberReference. This is potentially caused by the inherent complexity of
these operations, which often involve multiple dependencies and references. If
any required method or member is missing or incorrectly specified, it can
easily lead to non-compilable mutations. The errors highlight a need for
better context-aware mutation generation, ensuring that method calls and
member references align with the intended program structure. Additionally, we
inspect the deletion mutations rejected by the compiler and find that for
GPT-3.5, CodeLlama, LEAM, $\mu$Bert, and Major, these mutations account for
7.1%, 0.2%, 45.3%, 0.14%, and 14.4% of all their non-compilable mutations,
respectively. Thus for LLMs, deletion is not the major reason for non-
compilation.

Finding 10: Method invocations and member references in the code are the most
likely to trigger LLMs to generate non-compilable mutations.

##  5\. Discussion

###  5.1. Sensitivity to Chosen Experiment Settings

####  5.1.1. Experiment Setup of the Context Length

Table 9. Performance of Different Context Lengths Metric Category | Metric |  1-Line Context |  2-Line Context |  3-Line Context
---|---|---|---|---
Usability | Compilability Rate | 0.610 | 0.617 | 0.657
Useless Mutation Ratio | 0.119 | 0.128 | 0.137
Equivalent Mutation Rate | 1.6% | 1.1% | 1.3%
Behavior | Real Bug Detectability | 0.933 | 0.917 | 0.917
Coupling Rate | 0.141 | 0.121 | 0.136
Ochiai Coefficient | 0.517 | 0.417 | 0.467
Table 10. Performance Similarity of Different Context Lengths Settings | Spea. Coef. |  $p$-value | Pear. Coef. |  $p$-value
---|---|---|---|---
1-Line v.s. 2-Line | 0.9762 | 3.31E-05 | 0.9980 | 2.12E-08
2-Line v.s. 3-Line | 0.9762 | 3.31E-05 | 0.9997 | 1.04E-10
1-Line v.s. 3-Line | 0.9524 | 2.60E-04 | 0.9978 | 2.69E-08

To compare the similarity to real defects, we deliberately select the context
around the bug locations for generating mutations for all the approaches. To
check the impact of context selection on performance, we conduct experiments
on 60 bugs from Defects4J, comparing how different context lengths affect
performance. In our previous experiments, the context length is set to three
lines around the bug location, so we conduct experiments to compare the
effects of two-line and one-line contexts, as shown in Table 9. To measure
their similarity in performance, we use the Spearman coefficient and Pearson
correlation, as shown in Table 10. We can see that their mutual similarity
exceeds 0.95, which is above the typical threshold of 0.85 (Siami Namin et
al., 2008; Motwani and Brun, 2023; Kaufman et al., 2022), and the $p$-values
are all below 0.05. Different context lengths perform similarly, thus
validating the setup of our experiment.

####  5.1.2. Impact of Different Few-Shot Examples

Table 11. Impact on Different Few-Shot Examples

Metric | QB-6a (Default) | QB-3 | QB-6b | QB-9 | D4J-6
---|---|---|---|---|---
Comp. Rate | 65.7% | 62.7% | 63.9% | 61.8% | 62.1%
Useless Mut. Ratio | 13.7% | 12.8% | 12.8% | 13.8% | 12.5%
Equivalent Mutation Rate | 1.3% | 1.6% | 1.3% | 1.1% | 1.6%
Real Bug Det. | 0.917 | 0.933 | 0.917 | 0.917 | 0.900
Coupling Rate | 0.136 | 0.128 | 0.132 | 0.121 | 0.139
Ochiai Coefficient | 0.467 | 0.450 | 0.434 | 0.434 | 0.367

QB-N: Select N Examples from QuixBug. QB-6a is the set of default examples,
while QB-6b involves 6 other examples.

Table 12. Performance Similarity of Different Few-shot Examples Settings | Spea. Coef. |  $p$-value | Pear. Coef |  $p$-value
---|---|---|---|---
QB-6a (Default) v.s. QB-3 | 0.9701 | 6.55E-05 | 0.9996 | 1.70E-10
QB-6a (Default) v.s. QB-3b | 0.9762 | 3.31E-05 | 0.9984 | 1.00E-08
QB-6a (Default) v.s QB-9 | 0.9762 | 3.31E-05 | 0.9985 | 7.98E-09
QB-6a (Default) v.s D4J-6 | 0.9524 | 2.60E-04 | 0.9980 | 2.13E-08

To measure the impact of different few-shot examples on the results, we
conduct comparative experiments with various examples. First, we randomly
select 3, 6, and 9 additional examples from QuixBugs, and ensure there is no
overlap with the default 6 examples. Additionally, despite the risk of data
leakage, we select 6 examples from the Defects4J dataset. The results are
presented in Table 11.

We further compute their similarity, shown as Table 12. We can find that their
similarity exceeds 0.95, significantly exceeding the typical threshold of
0.85, and the $p$-values are all below 0.05. Therefore, we can conclude the
performance of different prompts is very similar.

####  5.1.3. Performance of Using the Same Number of Mutations

In Section 4, we did not restrict the number of mutations generated by each
approach. To study the performance of each approach under fixed quantity
conditions, we followed the setting of existing study (Tian et al., 2022;
Hariri et al., 2019) and limited the number of mutations generated by all
methods to the minimum produced by $\mu$Bert (Degiovanni and Papadakis, 2022),
which is 16,785. For counts exceeding this number, we randomly selected the
specified number of mutations for analysis. We conducted 10 rounds of random
sampling comparisons, and the results are presented in Table 13. Therefore,
under fixed quantity conditions, the mutations generated by GPT-3.5-Turbo are
still the closest to real bugs in behavior.

Table 13. Performance Under the Same Number of Mutations Metric | GPT-3.5 | CL-13b | LEAM |  $\mu$BERT | Major | PIT
---|---|---|---|---|---|---
Comp. Rate | 62.9% | 74.2% | 38.9% | 18.5% | 96.5% | —
Useless Mut. Ratio | 11.0% | 39.0% | 1.4% | 1.4% | 0.0% | —
Real Bug Det. | 0.892 | 0.735 | 0.675 | 0.550 | 0.833 | 0.437
Coupling Rate | 0.133 | 0.105 | 0.132 | 0.132 | 0.128 | 0.024
Ochiai Coefficient | 0.358 | 0.178 | 0.167 | 0.150 | 0.341 | 0.192

###  5.2. Implications

Based on our findings, we discuss the pros and cons of using LLMs for
generating mutations and suggest potential improvements.

First, our study shows that existing LLMs have the potential to generate high-
quality mutations for mutation testing with acceptable costs. The greatest
advantage of LLMs is their ability to generate mutations that more closely
mimic real bugs in behavior. For example, Listing 1 shows two compilable
mutations generated by GPT-3.5 in Chart-1, which are theoretically beyond the
capabilities of existing approaches. In Mutation-1, the LLM inferences there
must exist leftBlock because of the occurrence of rightBlock, and replaces the
variable correspondingly. This mutation is not killed by the bug-triggering
test. Without natural language understanding capabilities (i.e., understanding
the correspondence between “right” and “left”), it is impossible to create
such mutations. In Mutation-2, the LLM modifies the else branch to else if,
employing an integer variable seriesCount from context to synthesize the
condition, showing the capabilities of code structural changing that existing
methods cannot achieve. Also, this mutation is alive and can guide test
enhancement.

⬇

// Mutation-1

\- Size2D size = this.rightBlock.arrange(g2, c4);

\+ Size2D size = this.leftBlock.arrange(g2, c4);

// Mutation-2

\- else {

\- g2.setStroke(getItemOutlineStroke(row, column, selected));

\- }

\+ else if (seriesCount == 1) {

\+ g2.setStroke(getItemOutlineStroke(row, column, selected));

\+ }

Listing 1: Two Example Mutations of GPT-3.5 on Chart-1

Second, our study explores how different prompts, models, contexts, and few-
shot examples impact mutation generation performance. Our study highlights the
importance of using proper prompts and models with stronger coding
capabilities. In particular, designing prompts requires providing essential
context information (such as the whole method), but including too much
information (such as the target unit test) may reduce effectiveness.

Third, although LLMs achieve promising results in mutation testing, our study
indicates that there is still significant room for improvement. Specifically,
current LLMs generate a substantial number of mutations that are not
compilable. Therefore, we analyze the types of mutations that lead to non-
compilable code and identify some root causes of the LLMs. We advocate for
more research efforts in program analysis and repair approaches to address
error-prone code elements for mutation such as method invocations.

###  5.3. Threats to Validity

The selected LLMs, programming language, datasets, and baseline approaches
could be a threat to the validity of our results. To mitigate this threat, we
adopt the most widely studied models (i.e., GPT and CodeLlama), the most
popular language (i.e., Java), and the most popular dataset Defects4J. We also
employ state-of-the-art mutation testing approaches as baselines, including
learning-based (i.e., $\mu$Bert and LEAM) and rule-based (i.e., PIT and
Major).

Another validity threat may be due to data leakage, i.e., the fact that the
data in Defects4J (Just et al., 2014a) may be covered in the training set of
the studied LLMs. To mitigate this threat, we employed another dataset
ConDefects (Wu et al., 2023b) which includes programs and faults that were
made after the release time of the LLMs we use and thus have limited data
leakage risk. Additionally, to increase confidence in our results, we also
checked whether the tools can introduce exact matches (syntactically) with the
studied faults. We hypothesize that in case the tools have been tuned based on
specific fault instances, the tools would introduce at least one mutation that
is an exact match with the faults we investigate. Our results are: GPT,
CodeLlama, Major, LEAM, and $\mu$Bert, 282, 77, 67, 386, 39 on the Defects4J
dataset while on the ConDefects are 7, 9, 13, 8, 1, respectively, and indicate
that on Defects4J GPT and LEAM, approaches tend to produce significantly more
exact matches than the other approaches. Interestingly, Major produces a
similar number of exact matches with Codellama. $\mu$Bert wields significantly
the least number of exact matches, indicating a minimal or no advantage for
all these approaches (except from GPT and LEAM) due to exact matches (in the
case of Defects4J). Perhaps more interesting, in the ConDefects dataset, which
has not been seen by any of the tools, Major has the majority of the exact
matches, indicating a minor influence of any data leakage on the reported
results. Nevertheless, the LLMs we studied exhibit the same trend on the two
datasets, achieving the Spearman coefficient of 0.943 and the Pearson
correlation of 0.944, both with $p$-value less than 0.05, indicating their
performance is similar on the two datasets.

The different experimental settings may also threaten the validity of our
results. To address this threat, we elaborately explore the impacts of
prompts, context length, few-shot examples, and mutation numbers on the
performance of LLMs. The results show that different settings are highly
similar.

The subjective nature of human decisions in labeling for equivalent mutations
and non-compilation errors is another potential threat. To mitigate this
threat, we follow a rigorous annotation process where two co-authors
independently annotated each mutation. The final Cohen’s Kappa coefficient
indicates a relatively high level of agreement between the two annotators.

##  6\. Conclusion

In this paper, we systematically investigate the performance of LLMs in
generating mutations. We evaluate their utility from several aspects and find
LLMs have the advantage of generating diverse mutations that mimic the
behaviour of real bugs. We also analyze and discuss some limitations of LLMs
and point out further directions. We advocate research efforts to promote LLM
for mutation testing in the future.

## References

  * (1)
  * kum (2024) 2024\.  _Anon. Repo of Kumo_.  https://anonymous.4open.science/r/kumo-01D1/ Accessed: June 1, 2024.
  * atc (2024) 2024\.  _Atcoder_.  https://atcoder.jp Accessed: June 1, 2024.
  * jav (2024) 2024\.  _Javalang Parser_.  https://pypi.org/project/javalang Accessed: June 1, 2024.
  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023\.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ (2023).
  * Ammann et al. (2014) Paul Ammann, Márcio Eduardo Delamaro, and Jeff Offutt. 2014.  Establishing Theoretical Minimal Sets of Mutants. In _Seventh IEEE International Conference on Software Testing, Verification and Validation, ICST 2014, March 31 2014-April 4, 2014, Cleveland, Ohio, USA_. IEEE Computer Society, 21–30.
  * Andrews et al. (2005) James H Andrews, Lionel C Briand, and Yvan Labiche. 2005\.  Is mutation an appropriate tool for testing experiments?. In _Proceedings of the 27th international conference on Software engineering_. 402–411.
  * Beller et al. (2021) Moritz Beller, Chu-Pan Wong, Johannes Bader, Andrew Scott, Mateusz Machalica, Satish Chandra, and Erik Meijer. 2021.  What it would take to use mutation testing in industry—a study at facebook. In _2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)_. IEEE, 268–277.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Budd and Angluin (1982) Timothy A Budd and Dana Angluin. 1982.  Two notions of correctness and their relation to testing.  _Acta informatica_ 18, 1 (1982), 31–45.
  * Chekam et al. (2020) Thierry Titcheu Chekam, Mike Papadakis, Tegawendé F. Bissyandé, Yves Le Traon, and Koushik Sen. 2020.  Selecting fault revealing mutants.  _Empir. Softw. Eng._ 25, 1 (2020), 434–487.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021\.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_ (2021).
  * Coles et al. (2016) Henry Coles, Thomas Laurent, Christopher Henard, Mike Papadakis, and Anthony Ventresque. 2016\.  Pit: a practical mutation testing tool for java. In _Proceedings of the 25th international symposium on software testing and analysis_. 449–452.
  * Daran and Thévenod-Fosse (1996) Murial Daran and Pascale Thévenod-Fosse. 1996.  Software error analysis: A real case study involving real faults and mutations.  _ACM SIGSOFT Software Engineering Notes_ 21, 3 (1996), 158–171.
  * DEB et al. (2024) SOURAV DEB, KUSH JAIN, RIJNARD VAN TONDER, CLAIRE LE GOUES, and ALEX GROCE. 2024.  Syntax Is All You Need: A Universal-Language Approach to Mutant Generation.  (2024).
  * Degiovanni and Papadakis (2022) Renzo Degiovanni and Mike Papadakis. 2022.  $\mu$bert: Mutation testing using pre-trained language models. In _2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)_. IEEE, 160–169.
  * Delamaro et al. (1996) Márcio Eduardo Delamaro, José Carlos Maldonado, and A Mathur. 1996.  Proteum-a tool for the assessment of test adequacy for c programs user’s guide. In _PCS_ , Vol. 96. 79–95.
  * DeMillo et al. (1978) Richard A DeMillo, Richard J Lipton, and Frederick G Sayward. 1978.  Hints on test data selection: Help for the practicing programmer.  _Computer_ 11, 4 (1978), 34–41.
  * DeMillo et al. (1979) Richard A DeMillo, Richard J Lipton, and Frederick G Sayward. 1979.  Program mutation: A new approach to program testing.  _Infotech State of the Art Report, Software Testing_ 2, 1979 (1979), 107–126.
  * Deng et al. (2024) Yinlin Deng, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shujing Yang, and Lingming Zhang. 2024.  Large language models are edge-case generators: Crafting unusual programs for fuzzing deep learning libraries. In _Proceedings of the 46th IEEE/ACM International Conference on Software Engineering_. 1–13.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Diniz et al. (2021) João P Diniz, Chu-Pan Wong, Christian Kästner, and Eduardo Figueiredo. 2021.  Dissecting strongly subsuming second-order mutants. In _2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST)_. IEEE, 171–181.
  * Fan et al. (2023) Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. 2023.  Large Language Models for Software Engineering: Survey and Open Problems. In _2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE)_. IEEE, Los Alamitos, CA, USA, 31–53.
  * Garg et al. (2022) Aayush Garg, Milos Ojdanic, Renzo Degiovanni, Thierry Titcheu Chekam, Mike Papadakis, and Yves Le Traon. 2022.  Cerebro: Static subsuming mutant selection.  _IEEE Transactions on Software Engineering_ 49, 1 (2022), 24–43.
  * Gligoric et al. (2013) Milos Gligoric, Vilas Jagannath, Qingzhou Luo, and Darko Marinov. 2013.  Efficient mutation testing of multithreaded code.  _Software Testing, Verification and Reliability_ 23, 5 (2013), 375–403.
  * Groce et al. (2021) Alex Groce, Iftekhar Ahmed, Josselin Feist, Gustavo Grieco, Jiri Gesi, Mehran Meidani, and Qihong Chen. 2021.  Evaluating and improving static analysis tools via differential mutation analysis. In _2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS)_. IEEE, 207–218.
  * Guo et al. (2024) Qi Guo, Junming Cao, Xiaofei Xie, Shangqing Liu, Xiaohong Li, Bihuan Chen, and Xin Peng. 2024.  Exploring the potential of chatgpt in automated code refinement: An empirical study. In _Proceedings of the 46th IEEE/ACM International Conference on Software Engineering_. 1–13.
  * Hamlet (1977) Richard G. Hamlet. 1977\.  Testing programs with the aid of a compiler.  _IEEE transactions on software engineering_ 4 (1977), 279–290.
  * Hariri et al. (2019) Farah Hariri, August Shi, Vimuth Fernando, Suleman Mahmood, and Darko Marinov. 2019.  Comparing mutation testing at the levels of source code and compiler intermediate representation. In _2019 12th IEEE conference on software testing, validation and verification (ICST)_. IEEE, 114–124.
  * Hindle et al. (2016) Abram Hindle, Earl T Barr, Mark Gabel, Zhendong Su, and Premkumar Devanbu. 2016.  On the naturalness of software.  _Commun. ACM_ 59, 5 (2016), 122–131.
  * Huang et al. (2023) Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023\.  A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.  _arXiv preprint arXiv:2311.05232_ (2023).
  * Islam et al. (2019) Md Johirul Islam, Giang Nguyen, Rangeet Pan, and Hridesh Rajan. 2019.  A comprehensive study on deep learning bug characteristics. In _Proceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering_. 510–520.
  * Jia and Harman (2008) Yue Jia and Mark Harman. 2008.  MILU: A customizable, runtime-optimized higher order mutation testing tool for the full C language. In _Testing: Academic & Industrial Conference-Practice and Research Techniques (taic part 2008)_. IEEE, 94–98.
  * Jia and Harman (2010) Yue Jia and Mark Harman. 2010.  An analysis and survey of the development of mutation testing.  _IEEE transactions on software engineering_ 37, 5 (2010), 649–678.
  * Jiang et al. (2018) Jiajun Jiang, Yingfei Xiong, Hongyu Zhang, Qing Gao, and Xiangqun Chen. 2018.  Shaping program repair space with existing patches and similar code. In _Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis_. 298–309.
  * Jiang et al. (2023) Nan Jiang, Thibaud Lutellier, Yiling Lou, Lin Tan, Dan Goldwasser, and Xiangyu Zhang. 2023\.  Knod: Domain knowledge distilled tree decoder for automated program repair. In _2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)_. IEEE, 1251–1263.
  * Jimenez et al. (2018) Matthieu Jimenez, Thiery Titcheu Checkam, Maxime Cordy, Mike Papadakis, Marinos Kintis, Yves Le Traon, and Mark Harman. 2018\.  Are mutants really natural? a study on how” naturalness” helps mutant selection. In _Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement_. 1–10.
  * Just et al. (2014a) René Just, Darioush Jalali, and Michael D Ernst. 2014a.  Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In _Proceedings of the 2014 International Symposium on Software Testing and Analysis_. 437–440.
  * Just et al. (2014b) René Just, Darioush Jalali, Laura Inozemtseva, Michael D Ernst, Reid Holmes, and Gordon Fraser. 2014b.  Are mutants a valid substitute for real faults in software testing?. In _Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering_. 654–665.
  * Just et al. (2011) Rene Just, Franz Schweiggert, and Gregory M Kapfhammer. 2011\.  MAJOR: An efficient and extensible tool for mutation analysis in a Java compiler. In _ASE_. 612–615.
  * Kaufman et al. (2022) Samuel J Kaufman, Ryan Featherman, Justin Alvin, Bob Kurtz, Paul Ammann, and René Just. 2022\.  Prioritizing mutants to guide mutation testing. In _Proceedings of the 44th International Conference on Software Engineering_. 1743–1754.
  * Kazerouni et al. (2021) Ayaan M Kazerouni, James C Davis, Arinjoy Basak, Clifford A Shaffer, Francisco Servant, and Stephen H Edwards. 2021.  Fast and accurate incremental feedback for students’ software tests using selective mutation analysis.  _Journal of Systems and Software_ 175 (2021), 110905.
  * Khanfir et al. (2023) Ahmed Khanfir, Anil Koyuncu, Mike Papadakis, Maxime Cordy, Tegawendé F Bissyandé, Jacques Klein, and Yves Le Traon. 2023\.  IBiR: Bug-report-driven fault injection.  _ACM Transactions on Software Engineering and Methodology_ 32, 2 (2023), 1–31.
  * Kim et al. (2022) Jinhan Kim, Juyoung Jeon, Shin Hong, and Shin Yoo. 2022\.  Predictive mutation analysis via the natural language channel in source code.  _ACM Transactions on Software Engineering and Methodology (TOSEM)_ 31, 4 (2022), 1–27.
  * Le Goues et al. (2011) Claire Le Goues, ThanhVu Nguyen, Stephanie Forrest, and Westley Weimer. 2011.  Genprog: A generic method for automatic software repair.  _Ieee transactions on software engineering_ 38, 1 (2011), 54–72.
  * Li et al. (2023a) Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023a.  Starcoder: may the source be with you!  _arXiv preprint arXiv:2305.06161_ (2023).
  * Li et al. (2023b) Tsz-On Li, Wenxi Zong, Yibo Wang, Haoye Tian, Ying Wang, Shing-Chi Cheung, and Jeff Kramer. 2023b.  Nuances are the key: Unlocking chatgpt to find failure-inducing tests with differential prompting. In _2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 14–26.
  * Lin et al. (2017) Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. 2017.  QuixBugs: A multi-lingual program repair benchmark set based on the Quixey Challenge. In _Proceedings Companion of the 2017 ACM SIGPLAN international conference on systems, programming, languages, and applications: software for humanity_. 55–56.
  * Linares-Vásquez et al. (2017) Mario Linares-Vásquez, Gabriele Bavota, Michele Tufano, Kevin Moran, Massimiliano Di Penta, Christopher Vendome, Carlos Bernal-Cárdenas, and Denys Poshyvanyk. 2017\.  Enabling mutation testing for android apps. In _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering_. 233–244.
  * Ma et al. (2024) Zeyang Ma, An Ran Chen, Dong Jae Kim, Tse-Hsun Chen, and Shaowei Wang. 2024.  LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing. In _2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE)_. IEEE Computer Society, 883–883.
  * Moon et al. (2014) Seokhyeon Moon, Yunho Kim, Moonzoo Kim, and Shin Yoo. 2014\.  Ask the mutants: Mutating faulty programs for fault localization. In _2014 IEEE Seventh International Conference on Software Testing, Verification and Validation_. IEEE, 153–162.
  * Motwani and Brun (2023) Manish Motwani and Yuriy Brun. 2023.  Better automatic program repair by using bug reports and tests together. In _2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)_. IEEE, 1225–1237.
  * Namin and Kakarla (2011) Akbar Siami Namin and Sahitya Kakarla. 2011.  The use of mutation in testing experiments and its sensitivity to external threats. In _Proceedings of the 2011 International Symposium on Software Testing and Analysis_. 342–352.
  * Offutt (1992) A Jefferson Offutt. 1992\.  Investigations of the software testing coupling effect.  _ACM Transactions on Software Engineering and Methodology (TOSEM)_ 1, 1 (1992), 5–20.
  * Offutt and Untch (2001) A Jefferson Offutt and Roland H Untch. 2001.  Mutation 2000: Uniting the orthogonal.  _Mutation testing for the new century_ (2001), 34–44.
  * Ojdanic et al. (2023a) Milos Ojdanic, Aayush Garg, Ahmed Khanfir, Renzo Degiovanni, Mike Papadakis, and Yves Le Traon. 2023a.  Syntactic vs. semantic similarity of artificial and real faults in mutation testing studies.  _IEEE Transactions on Software Engineering_ (2023).
  * Ojdanic et al. (2023b) Milos Ojdanic, Ahmed Khanfir, Aayush Garg, Renzo Degiovanni, Mike Papadakis, and Yves Le Traon. 2023b.  On Comparing Mutation Testing Tools through Learning-based Mutant Selection. In _IEEE/ACM International Conference on Automation of Software Test, AST 2023, Melbourne, Australia, May 15-16, 2023_. IEEE, 35–46.
  * Papadakis et al. (2016) Mike Papadakis, Christopher Henard, Mark Harman, Yue Jia, and Yves Le Traon. 2016\.  Threats to the validity of mutation-based test assessment. In _Proceedings of the 25th International Symposium on Software Testing and Analysis_. 354–365.
  * Papadakis et al. (2015) Mike Papadakis, Yue Jia, Mark Harman, and Yves Le Traon. 2015\.  Trivial compiler equivalence: A large scale empirical study of a simple, fast and effective equivalent mutant detection technique. In _ICSE_. 936–946.
  * Papadakis et al. (2019) Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019\.  Mutation testing advances: an analysis and survey.  In _Advances in Computers_. Vol. 112. Elsevier, 275–378.
  * Papadakis and Le Traon (2015) Mike Papadakis and Yves Le Traon. 2015.  Metallaxis-FL: mutation-based fault localization.  _Software Testing, Verification and Reliability_ 25, 5-7 (2015), 605–628.
  * Papadakis et al. (2018) Mike Papadakis, Donghwan Shin, Shin Yoo, and Doo-Hwan Bae. 2018\.  Are mutation scores correlated with real fault detection? a large scale empirical study on the relationship between mutants and real faults. In _Proceedings of the 40th international conference on software engineering_. 537–548.
  * Patra and Pradel (2021) Jibesh Patra and Michael Pradel. 2021.  Semantic bug seeding: a learning-based approach for creating realistic bugs. In _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 906–918.
  * Qian et al. (2022) Ruixiang Qian, Quanjun Zhang, Chunrong Fang, and Lihua Guo. 2022\.  Investigating coverage guided fuzzing with mutation testing. In _Proceedings of the 13th Asia-Pacific Symposium on Internetware_. 272–281.
  * Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023\.  Code llama: Open foundation models for code.  _arXiv preprint arXiv:2308.12950_ (2023).
  * Schäfer et al. (2023) Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023\.  An empirical evaluation of using large language models for automated unit test generation.  _IEEE Transactions on Software Engineering_ (2023).
  * Schuler and Zeller (2009) David Schuler and Andreas Zeller. 2009.  Javalanche: efficient mutation testing for Java. In _ESEC/FSE_. 297–298.
  * Shi et al. (2019) August Shi, Jonathan Bell, and Darko Marinov. 2019\.  Mitigating the effects of flaky tests on mutation testing. In _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis_. 112–122.
  * Shin et al. (2019) Donghwan Shin, Shin Yoo, Mike Papadakis, and Doo-Hwan Bae. 2019\.  Empirical evaluation of mutation-based test case prioritization techniques.  _Software Testing, Verification and Reliability_ 29, 1-2 (2019), e1695.
  * Siami Namin et al. (2008) Akbar Siami Namin, James H Andrews, and Duncan J Murdoch. 2008\.  Sufficient mutation operators for measuring test effectiveness. In _Proceedings of the 30th international conference on Software engineering_. 351–360.
  * Tian et al. (2022) Zhao Tian, Junjie Chen, Qihao Zhu, Junjie Yang, and Lingming Zhang. 2022.  Learning to construct better mutation faults. In _Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering_. 1–13.
  * Tip et al. (2024) Frank Tip, Jonathan Bell, and Max Schäfer. 2024.  LLMorpheus: Mutation Testing using Large Language Models.  _arXiv preprint arXiv:2404.09952_ (2024).
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023\.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ (2023).
  * Tufano et al. (2019) Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. 2019.  Learning how to mutate source code from bug-fixes. In _2019 IEEE International conference on software maintenance and evolution (ICSME)_. IEEE, 301–312.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Viera et al. (2005) Anthony J Viera, Joanne M Garrett, et al. 2005\.  Understanding interobserver agreement: the kappa statistic.  _Fam med_ 37, 5 (2005), 360–363.
  * Wang et al. (2021) Bo Wang, Sirui Lu, Yingfei Xiong, and Feng Liu. 2021\.  Faster mutation analysis with fewer processes and smaller overheads. In _2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 381–393.
  * Wang et al. (2017) Bo Wang, Yingfei Xiong, Yangqingwei Shi, Lu Zhang, and Dan Hao. 2017.  Faster mutation analysis via equivalence modulo states. In _Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis_. 295–306.
  * Wang et al. (2024) Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024\.  Software testing with large language models: Survey, landscape, and vision.  _IEEE Transactions on Software Engineering_ (2024).
  * Wang et al. (2022) Xinyi Wang, Tongxuan Yu, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2022.  Mutation-based test generation for quantum programs with multi-objective search. In _Proceedings of the Genetic and Evolutionary Computation Conference_. 1345–1353.
  * Wen et al. (2019) Ming Wen, Yepang Liu, Rongxin Wu, Xuan Xie, Shing-Chi Cheung, and Zhendong Su. 2019\.  Exposing library API misuses via mutation analysis. In _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. IEEE, 866–877.
  * Wu et al. (2023a) Jiang Wu, Yan Lei, Zhuo Zhang, Xiankai Meng, Deheng Yang, Pan Li, Jiayu He, and Xiaoguang Mao. 2023a.  Mantra: Mutation Testing of Hardware Design Code Based on Real Bugs. In _2023 60th ACM/IEEE Design Automation Conference (DAC)_. IEEE, 1–6.
  * Wu et al. (2023b) Yonghao Wu, Zheng Li, Jie M Zhang, and Yong Liu. 2023b.  Condefects: A new dataset to address the data leakage concern for llm-based fault localization and program repair.  _arXiv preprint arXiv:2310.16253_ (2023).
  * Xia and Zhang (2022) Chunqiu Steven Xia and Lingming Zhang. 2022.  Less training, more repairing please: revisiting automated program repair via zero-shot learning. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 959–971.
  * Xiao et al. (2023) Yuan-An Xiao, Chenyang Yang, Bo Wang, and Yingfei Xiong. 2023\.  ExpressAPR: Efficient patch validation for java automated program repair systems. In _2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 2038–2041.
  * Xiao et al. (2024) Yuan-An Xiao, Chenyang Yang, Bo Wang, and Yingfei Xiong. 2024\.  Accelerating patch validation for program repair with interception-based execution scheduling.  _IEEE Transactions on Software Engineering_ (2024).
  * Zhang et al. (2016) Jie Zhang, Ziyi Wang, Lingming Zhang, Dan Hao, Lei Zang, Shiyang Cheng, and Lu Zhang. 2016.  Predictive mutation testing. In _Proceedings of the 25th international symposium on software testing and analysis_. 342–353.
  * Zhang et al. (2018) J Zhang, L Zhang, M Harman, D Hao, and Y Jia. 2018.  Predictive Mutation Testing.  _IEEE Transactions on Software Engineering_ (2018).
  * Zhang et al. (2013) Lingming Zhang, Darko Marinov, and Sarfraz Khurshid. 2013\.  Faster mutation testing inspired by test prioritization and reduction. In _Proceedings of the 2013 International Symposium on Software Testing and Analysis_. 235–245.
  * Zhang et al. (2022) Peng Zhang, Yang Wang, Xutong Liu, Yanhui Li, Yibiao Yang, Ziyuan Wang, Xiaoyu Zhou, Lin Chen, and Yuming Zhou. 2022.  Mutant reduction evaluation: what is there and what is missing?  _ACM Transactions on Software Engineering and Methodology (TOSEM)_ 31, 4 (2022), 1–46.
  * Zhu et al. (2021) Qihao Zhu, Zeyu Sun, Yuan-an Xiao, Wenjie Zhang, Kang Yuan, Yingfei Xiong, and Lu Zhang. 2021.  A syntax-guided edit decoder for neural program repair. In _Proceedings of the 29th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering_. 341–353.
  * Zhu et al. (2023) Qihao Zhu, Zeyu Sun, Wenjie Zhang, Yingfei Xiong, and Lu Zhang. 2023.  Tare: Type-aware neural program repair. In _2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)_. IEEE, 1443–1455.
  * Zou et al. (2019) Daming Zou, Jingjing Liang, Yingfei Xiong, Michael D Ernst, and Lu Zhang. 2019.  An empirical study of fault localization families and their combinations.  _IEEE Transactions on Software Engineering_ 47, 2 (2019), 332–347.