# Efficient Mutation Testing via Pre-Trained Language Models

Ahmed Khanfir ,  Renzo Degiovanni  Mike Papadakis and  Yves Le Traon
A. Khanfir, R. Degiovanni, M. Papadakis, Y. Le Traon are with the University
of Luxembourg, Luxembourg. SnT, University of Luxembourg, Luxembourg

###### Abstract

Mutation testing is an established fault-based testing technique. It operates
by seeding faults into the programs under test and asking developers to write
tests that reveal these faults. These tests have the potential to reveal a
large number of faults – those that couple with the seeded ones – and thus are
deemed important. To this end, mutation testing should seed faults that are
both “natural” in a sense easily understood by developers and strong (have
high chances to reveal faults). To achieve this we propose using pre-trained
generative language models (i.e. CodeBERT) that have the ability to produce
developer-like code that operates similarly, but not exactly, as the target
code. This means that the models have the ability to seed natural faults,
thereby offering opportunities to perform mutation testing. We realise this
idea by implementing $\mu$Bert, a mutation testing technique that performs
mutation testing using CodeBert and empirically evaluated it using  689 faulty
program versions. Our results show that the fault revelation ability of
$\mu$Bert is higher than that of a state-of-the-art mutation testing (PiTest),
yielding tests that have up to  17% higher fault detection potential than that
of PiTest. Moreover, we observe that $\mu$Bert can complement PiTest, being
able to detect  47 bugs missed by PiTest, while at the same time, PiTest can
find  13 bugs missed by $\mu$Bert.

###### Index Terms:

Fault Injection, Mutation Testing, Pre-Trained Language Models

##  1 Introduction

Mutation testing aims at seeding faults using simple syntactic transformations
[19]. These transformations, also known as mutation operators are typically
constructed based on syntactic rules crafted based on the grammar of the
target programming language [8], i.e. replacing an arithmetic operator with
another such as a + by a -. Unfortunately, such techniques generate mutants
(seeded faults), many of which are “unatural”, i.e., non-conforming to the way
developers code, thereby perceived as unrealistic by developers [11]. At the
same time, the syntactic-based fault seeding fails to capture the semantics of
the code snippets that they apply, leading to numerous trivial or low utility
faults [46].

To deal with the above issue we propose forming natural mutations by using big
code. Thus, we aim at introducing modifications that follow the implicit
rules, norms and coding conventions followed by programmers, by leveraging the
capabilities of pre-trained language models to capture the underlying
distribution of code and its writing, as learned by the pre-training process
on big code.

To this end, we rely on CodeBERT [22], an NL-PL bimodal language model that
has been trained on over 6.4 million programs. More precisely, we use its
Masking Modelling Language (MLM) functionality, which given a code sequence
with a masked token, predicts alternative replacements to that token, that is
best matching the sequence context. This is important, since the predictions
do not follow fixed predefined patterns as is the case of conventional
mutation testing, but are instead adapted to fit best the target code. For
instance, given a sequence int a = 1;, we pass a masked version of it as int a
= <mask>;, then CodeBERT by default proposes 5 predictions sorted by
likelihood score: 0, 1, b, 2, and 10. Being the most likely fitting tokens to
the code context, our intuition is that replacing the masked token with these
predictions would induce “natural” mutants.

Precisely, we introduce $\mu$Bert, a mutation testing approach that uses a
pre-trained language model (CodeBERT) to generate mutants by masking and
replacing tokens with the aim of forming natural mutants. $\mu$Bert iterates
through the program statements and modifies their token. In particular,
$\mu$Bert proceeds as follows: (1) it selects and masks one token at a time;
(2) feeds CodeBERT with the masked sequence and obtains the predictions; (3)
creates mutants by replacing the masked token with the predicted ones; and (4)
discards non-compilable, duplicate and equivalent mutants (mutants
syntactically equal to original code).

Recent research [32] has shown that some real faults are only captured by
using complex patterns, i.e. patterns that require more than one token
mutation. To account for such cases, $\mu$Bert is equipped with additive
mutations, i.e., mutations that add code (instead of deleting or altering).
For example, consider a boolean expression $\texttt{e}_{1}$ (typically present
in if, do, while and return statements), which is mutated by $\mu$Bert by
adding a new condition $\texttt{e}_{2}$, thereby generating a new condition
$\texttt{e}_{1}\texttt{||}\texttt{e}_{2}$ (or
$\texttt{e}_{1}\texttt{\&\&}\texttt{e}_{2}$), which is then masked and
completed by CodeBERT. For instance, given a condition if(a == b), $\mu$Bert
produces a new condition if(a == b || a > 0) that is masked and produces if(a
== b || b > 0).

We implement $\mu$Bert, and evaluate its ability to serve the main purposes of
mutation testing, i.e. guiding the testing towards finding faults. We thus,
evaluate it using  689 faults from Defects4J and asses $\mu$Bert effectiveness
and cost-efficiency to reveal111Tests are written/generated to kill (reveal)
the mutants. A bug is revealed by a mutation testing approach, if the written
tests to kill its mutants also reveal the bug. them. Our results show that
$\mu$Bert is very effective in terms of fault revelation, finding on average
84% of the faults. This implies that $\mu$Bert mutants cover efficiently
faulty behaviours caused by real bugs. More importantly, the approach is
noticeably more effective and cost-efficient than a traditional mutation
testing technique, namely PiTest [17], that we use as a baseline in our
evaluation. Precisely, we consider three different configurations for PiTest
that uses different sets of mutation operators (DEFAULT, ALL and RV). In fact,
test suites that kill all mutants of $\mu$Bert find on average between  5.5%
to 33% more faults than those generated to kill all mutants introduced by
PiTest. Moreover, even when analysing the same number of mutants, $\mu$Bert
induces test suites that find on average  6% to 16% more faults than PiTest.
These results are promising and endorse the usage of $\mu$Bert over the
considered mutation testing technique, as a test generation and assessment
criterion.

We also study the impact of the condition-seeding-based mutations in the fault
detection capability of $\mu$Bert. We observe that test-suites designed to
kill both kinds of $\mu$Bert mutants – induced by 1) direct CodeBERT
predictions and 2) a combination of conditions-seeding with CodeBERT
predictions – find on average over  9% more bugs than the ones designed to
kill direct CodeBERT prediction mutants only (1).

Overall, our main contributions are:

  * •

We introduce $\mu$Bert, the first mutation testing approach that uses pre-
trained language models. It leverages the model’s code knowledge captured
during its pretraining on large code corpora and its ability to capture the
program context, to produce “natural” mutants.

  * •

We propose new additive mutations which operate by seeding new conditions in
the existing conditional expressions of the target code, then masking and
replacing their tokens with the model predictions.

  * •

We provide empirical evidence that $\mu$Bert mutants can guide testing towards
higher fault detection capabilities, outperforming those achieved by SOA
techniques (i.e. PiTest), in terms of effectiveness and cost-efficiency. In
our empirical study, we validate also the advantage of employing the new
additive mutation patterns, w.r.t improving the effectiveness and cost-
efficiency in writing test suites with higher fault revelation capability.

##  2 Background

Figure 1: $\mu$Bert Workflow: (1) it parses the Java code given as input, and
extracts the expressions to mutate; (2) it creates simple-replacement mutants
by masking the tokens of interest and invoking CodeBERT; (3) it generates the
mutants by replacing the masked token with CodeBERT predictions; (4) it
generates complex mutants via a) conditions-seeding, b) tokens masking then c)
replacing by CodeBERT predictions; and finally, (5) it discards not compiling
and syntactically identical mutants.

###  2.1 Mutation Testing

Mutation analysis [47] is a test adequacy criterion representing test
requirements by the mean of mutants, which are obtained by performing slight
syntactic modifications to the original program. For instance, an expression
like x > 0 can be mutated to x < 0 by replacing the relational operator > with
<. These mutants are then used to assess the effectiveness and thoroughness of
a test suite in detecting their corresponding code modification.

A test case detects a mutant if it is capable of producing distinguishable
observable outputs between the mutant and the original program. A mutant is
said to be _killed_ if it is detected by a test case or a test suite;
otherwise, it is called _live_ or _survived_. Some mutants cannot be killed as
they are functionally _equivalent_ to the original program. The _mutation
score_ measures the test suite adequacy and is computed as the ratio of killed
mutants over the total number of generated mutants.

###  2.2 Generative Language Models

Advances in deep learning approaches gave birth to new language models for
code generation [4, 15, 1, 22]. These models are trained on large corpora
counting multiple projects, thereby acquiring a decent knowledge of code,
enabling them to predict accurately source code to developers. Among these
pre-trained models, CodeBERT [22], a language model that has been recently
introduced and made openly accessible for researchers by Microsoft.

CodeBERT is an NL-PL bimodal pre-trained language model (Natural Language
Programming Language) that supports multiple applications such as code search,
code documentation generation, etc. Same as most large pre-trained models,
i.e. BERT [20], CodeBERT’s developing adopts a Multilayer Transformer [55]
architecture. It has been trained on a large corpus collected from over 6.4
million projects available on GitHub, counting 6 different programming
languages, including Java. The model was trained in a cross-modal fashion,
through bimodal NL-PL data, where the input data is formed by pairs of source
code and its related documentation, as well-as unimodal data, including either
natural language or programming language sequences per input. This way, it
enables the model to offer both – PL and NL-PL – functionalities. The training
targets a hybrid objective function, that is based on replaced token
detection.

$\mu$Bert incorporates the Masked Language Modeling (MLM) functionality [2] of
CodeBERT in its workflow, to generate “natural” mutants. The CodeBERT MLM
pipeline takes as input a code sequence of maximum 512 tokens, including among
them one masked as <mask>, whose value will be predicted by the model based on
the context captured from the remaining tokens. CodeBERT provides by default 5
predictions per token, among which we use the inaccurate and compilable
predicted codes as mutants.

##  3 Approach

We propose $\mu$Bert, a generative language-model-based mutation testing
approach, which is described step by step in Figure 1. Given an input source
code, $\mu$Bert leverages CodeBERT’s knowledge of code and its capability in
capturing the program’s context to produce “natural” mutations, i.e. that are
similar to eventual developer mistakes.To do so, $\mu$Bert proceeds as follows
in six steps:

  1. 1.

First, it extracts relevant locations (AST 222AST: Abstract Syntax Tree.
nodes) where to mutate

  2. 2.

Second, it masks the identified node-tokens, creating one masked version per
selected token.

  3. 3.

Then, it invokes CodeBERT to predict replacements for these masked tokens.

  4. 4.

In addition to the mutants produced in Step (3), $\mu$Bert also implements
some condition-seeding additive mutations that modify more than one token.
Precisely, it modifies the conditional expressions in the control flow
(typically present in if, do, while and return statements) by extending the
original condition with a new one, combined with the logical operator && or
||. Then, the new conditional expression is mutated by following the same
steps (2) and (3) – masking and replacing the masked tokens by the CodeBERT
predictions.

  5. 5.

Finally, the approach discards duplicate predictions or those inducing similar
code to the original one, or not compiling, and outputs the remaining ones as
mutants, from diverse locations of the target code. More precisely, it
iterates through the statements in random order and outputs in every iteration
one mutant per line, until achieving the desired number of mutants or all
mutants are outputted.

###  3.1 AST Nodes Selection

$\mu$Bert parses the AST of the input source code and selects the lines that
are more likely to carry the program’s specification implementation, excluding
the import statements and the declaration ones, e.g. the statements declaring
a class, a method, an attribute, etc. This way, the approach focuses the
mutation on the business-logic portion of the program and excludes the lines
that are probably of lower impact on the program behaviour. It proceeds then,
by selecting from each of these statements, the relevant nodes to mutate, i.e.
the operators, the operands, the method calls and variables, etc., and
excluding the language-specific ones, like the separators and the flow
controls, i.e. semicolons, brackets, if, else, etc. Table I summarises the
type of targeted AST nodes by $\mu$Bert, with corresponding example
expressions and induced mutants. We refer to these as the conventional
mutations provided by $\mu$Bert, denoted by $\mu$Bertconv in our evaluation,
previously introduced in the preliminary version of the approach [18].

###  3.2 Token Masking

In this step, we mask the selected nodes one by one, producing a masked
version from the original source code for each node of interest. This means
that every masked version contains the original code with one missing node,
replaced by the placeholder <mask>.

This way, $\mu$Bert can generate several mutants in the same program location.
For instance, for an assignment expression like res = a + b, $\mu$Bert will
create (potentially 25) mutants from the following masked sequences:

  * •

$\textbf{{<mask>}}=\texttt{a + b}$

  * •

res <mask>= a + b

  * •

$\texttt{res}=\texttt{{<mask>} + b}$

  * •

$\texttt{res}=\texttt{a {<mask>} b}$

  * •

$\texttt{res}=\texttt{a + {<mask>}}$

TABLE I: Example of $\mu$Bert conventional mutations, available in the preliminary version of the approach [18], denoted by $\mu$Bertconv. Ast node | Expression | Masked Expression | Mutant Example
---|---|---|---
literals | res + 10 | res + <mask> | res + 0
identifiers | res + 10 | <mask> \+ 10 | a \+ 10
binary expressions | a && b | a <mask> b | a || b
unary expressions | \--a | <mask>a | ++a
assignments | sum += current | sum <mask>= current | sum -= current
object fields | node.next | node.<mask> | node.prev
method calls | list.add(node) | list.<mask>(node) | list.push(node)
array access | arr[index + 1] | arr[<mask>] | arr[index]
static type references | Math.random() * 10 | <mask>.random() * 10 | Random.random() * 10

###  3.3 CodeBERT-MLM prediction

$\mu$Bert invokes CodeBERT to predict replacements for the masked nodes. To do
so, it tokenizes every masked version into a tokens vector then crops it to a
subset one that fits the maximum size allowed by the model (512) and counts
the masked token with the surrounding code-tokens. Next, our approach feeds
these vectors to CodeBERT MLM to predict the most probable replacements of the
masked token. Our intuition is that the larger the code portion accompanying
the mask placeholder, the better CodeBERT would be able to capture the code
context, and consequently, the more meaningful its predictions would be. This
step ends with the generation of five predictions per masked token.

###  3.4 Condition seeding

$\mu$Bert generates second-order mutants by combining condition seeding with
CodeBERT prediction capabilities. To do so, our approach modifies the
conditions in control flow and return statements, including if, do, while and
return conditional expressions. For every one of these statements, it starts
by extending the original condition by a new one, separated with the logical
operator && or $||$, in both orders (original condition first or the other way
around) and with or without negation (!).

Next, all substitute conditions are put one by one in place in the original
code, forming multiple condition-seeded code versions, that we pass as input
to Step (2), in which their tokens are masked and then (3) passed each to
CodeBERT to predict the best substitute of their corresponding masked tokens.

The seeded conditions are created in two ways:

####  3.4.1 Using existing conditions in the same class

To mutate a given condition – if, do, while and return conditional expressions
–, we collect all other conditions existing in the same class, then combine
each one of them with the target condition, using logical operators.

Precisely, let $Exp_{t}$ a conditional expression to mutate and
$S_{E}=\\{Exp_{0},...,Exp_{n}\\}$ the set of other conditional expressions
appearing in the same class, excluding the null-check ones (i.e. var == null).
The alternative replacement conditions generated for $Exp_{t}$ are the
combinations of:

  * •

$Exp_{t}$ op neg $Exp_{i}$ and

  * •

$Exp_{i}$ op neg $Exp_{t}$,

where op is a binary logical operator taking the values in {&&,||}, neg is
either the negation operator ! or nothing and $Exp_{i}$ is a condition from
$S_{E}$.

####  3.4.2 Using existing variables in the same class

When the target if conditional expression to mutate contains variables
(including fields), we create new additional conditions by combining these
variables with others of the same type from the same class. Then we combine
each one of the newly created conditions with the original one, using logical
operators.

Precisely, let $Exp_{t}$ be a conditional expression to mutate containing a
set of variables $S_{vt}$. For every variable $var_{t}$ in $S_{vt}$, we load
$S_{v}=\\{var_{0},...,var_{n}\\}$ the set of other variables appearing in the
same class and of the same type $T$ as $var_{t}$, then we generate the
following new conditions:

  * •

$Exp_{t}$ op ($var_{t}$ $rel_{op}$ $var_{i}$) and

  * •

($var_{t}$ $rel_{op}$ $var_{i}$) op $Exp_{t}$,

where op is a binary logical operator taking the values in {&&,||}, $rel_{op}$
is a relational operator applicable on the type $T$ and $var_{i}$ is a
variable from $S_{v}$.

###  3.5 Mutant filtering

In this step, our approach starts by discarding accurate and duplicate
predictions; the redundant predictions and the ones that are exactly the same
as the original code. Then, it iterates through the statements and selects in
every iteration one compilable prediction by line, while discarding not
compilable ones. Once all first-order mutants are selected (issued by one
single token replacement), our approach proceeds by selecting second-order
ones (issued by the combination of condition seeding and one token
replacement) in the same iterative manner. $\mu$Bert continues iterating until
achieving the desired number of mutants or all mutants are outputted.

##  4 Research Questions

We start our analysis by investigating the advantage brought by the additive
mutations (a.k.a. conditions seeding ones) w.r.t. the fault detection
capabilities of test suites designed to kill $\mu$Bert’s mutants. Thus, we
ask:

RQ1

_( $\mu$Bert Additive mutations)_ What is the added value of the additive
mutations on the fault detection capabilities of test suites designed to kill
$\mu$Bert’s mutants?

To answer this question, we generate two sets of mutants using $\mu$Bert: 1)
the first set using all possible mutations that we denote as $\mu$Bert and 2)
a second one using only the conventional $\mu$Bert’ mutations – part of our
preliminary implementation [18], excluding the additive ones – that we denote
as $\mu$Bertconv. Then we evaluate the fault detection ability of test suites
selected to kill the mutants from each set.

The answer of this question provides evidence that the additive mutations
increase the fault detection capability of $\mu$Bert. Yet, to assess its
general performance we compare it to state-of-the-art (SOA) mutation testing,
particularly PiTest [17], and thus, we ask:

RQ2

_(Fault detection)_ How does $\mu$Bert compare with state-of-the-art mutation
testing, in terms of fault detection?

To answer this question we generate mutants using the latest version of PiTest
[17], on the same target projects as for RQ1. As we are interested in
comparing the approaches and not the implementations of the tools, we exclude
the subjects on which PiTest did not run correctly or did not generate any
mutant. This way we ensure having a fair base of comparison by counting
exactly the same study subjects for both approaches (further details are given
in Section 5). Then, we compare the fault detection capability of test suites
selected to kill the same number of mutants produced by each approach.

Finally, we qualitatively analyse some of the mutants generated with $\mu$Bert
and ask:

RQ3

_(Qualitative analysis)_ Does $\mu$Bert generate different mutants than
traditional mutation testing operators?

To answer this question, we showcase the mutants generated by $\mu$Bert that
help in detecting faults not found by PiTest. Additionally, we discuss the
program-context-capturing importance in $\mu$Bert’s functioning, by rerunning
it with a reduced size of the masked codes passed to the model, and comparing
examples of yielded mutants with those obtained in our original setup.

##  5 Experimental Setup

###  5.1 Dataset & Benchmark

To evaluate $\mu$Bert’s fault detection, we use real bugs from a popular
dataset in the software engineering research area – Defects4J [29] v2.0.0. In
this benchmark, every subject bug is provided with a buggy version of the
source code, its corresponding fixed version, and equipped with a test suite
that passes on the fixed version and fails with at least one test on the buggy
one. The dataset includes over 800 bugs from which, we exclude the ones
presenting issues, i.e. with wrong revision ids, not compiling or with
execution issues, or having failing tests on the fixed version, at the
reporting time. Next, we run $\mu$Bert and PiTest on the corresponding classes
impacted by the bug from the fixed versions of the remaining bugs and exclude
the ones where no tool generated any mutant, ending up with  689 bugs covered
by $\mu$Bert and  457 covered by PitTest. As we’re interested in comparing the
approaches and not the tools’ implementations, and to exclude eventual threats
related to the environment (i.e. supported java and juint versions by each
technique, etc.) or the limitations and shortages of the dataset, we establish
every comparison study on a dataset counting only bugs covered by all
considered approaches:  689 bugs to answer RQ1 and  457 to answer RQ2 and RQ3.

###  5.2 Experimental Procedure

To assess the complementary and added value in terms of fault revelation of
the condition-seeding-based mutations (answer to RQ1), we run our approach
with and without those additional mutations – that we name respectively
$\mu$Bert and $\mu$Bertconv–, and thus, generating all possible mutants on our
dataset programs’ fixed versions. Next, we compare the average effectiveness
of the test suites generated to kill the mutants of each set; induced by
$\mu$Bert and $\mu$Bertconv.

Once the added value of the proposed condition-seeding-based mutations is
validated, we compare its performance to S.O.A. mutation testing (answer to
RQ2 and RQ3). We use PiTest [17], a stable and mature Java mutation testing
tool, because it has been more effective at finding faults than other tools
[33] and it is among the most commonly used by researchers and practitioners
[47, 52], as of today. The tool proposes different configurations to adapt the
produced mutations and their general cost to the target users, by excluding or
including mutators. Among these configurations we used the three following:

  * •

Pit-all (ALL) which counts all available mutation operators available in the
current version333Version 1.9.4 available in PitTest’s [6] GitHub repository
(branch=master, repo=https://github.com/hcoles/pitest.git, rev-id=17e1eecf).

  * •

Pit-default (DEFAULTS) whose mutators are selected to form a stable and cost-
efficient subset of operators by producing less but more relevant mutants.

  * •

Pit-rv-all (ALL) which is a version444Version 1.7.4 available in PitTest’s [6]
GitHub repository (branch=master, repo=https://github.com/hcoles/pitest.git,
rev-id=2ec1178a) that includes the mutators of Pit-all and extra experimental
[7] ones that are made available for research studies.

To compare the different approaches, we evaluate their effectiveness and cost-
efficiency in achieving one of the main purposes of mutation testing, i.e., to
guide the testing towards higher fault detection capabilities. For this
reason, we simulate a mutation testing use-case scenario, where a
developer/tester selects mutants and writes tests to kill them [13, 34].

We run every approach on the fixed versions and test suites provided by
Defects4J, then collect the mutants and their test execution results; whether
the mutant is killed (breaks at least one test of the test suite) and if yes
by which tests. Next, we suppose that the not killed mutants are equivalent or
irrelevant, explaining why no tests have been written to kill them by the
developers. Then, we simulate the scenario of a developer testing the fixed
version, in a state where 1) it did not have any test 2) thus all mutants did
not have killing tests and 3) the developer had no knowledge of which mutants
are equivalent or not. This way, we can reproduce the developer flow of

  1. 1.

selecting and analysing one mutant,

  2. 2.

to either (a) discard it from the mutant set if it is equivalent (not killed
in the actual test suite) or (b) write a test to kill it (by selecting one of
the actual killing tests of the mutant),

  3. 3.

then discarding all killed mutants by that test and

  4. 4.

iterating similarly over the remaining mutants until all of them are analysed.

We say that a bug is found by a mutation testing technique if the resulting
test suite – formed by the written (selected) tests by the developer –
contains at least one test that reveals it; a test that breaks when executed
on the buggy version.

We express the testing cost in terms of mutants analysed, and hence, we
consider the effort required to find a bug as the number of mutants analysed
until the first bug-revealing test is written. To set a common basis of
comparison between the approaches, accounting for the different number of
generated mutants, we run the simulations until the same maximum effort is
reached (maximum number of mutants to analyse), which we set to the least cost
required to kill all the mutants by one of the compared approaches. During our
evaluation study, we use the same mutation selection strategy for all compared
approaches, iterating through the lines in random order and selecting 1
arbitrary mutant per line per iteration. To reduce the process randomness
impact on our results (in the selection of mutants and tests), we run every
simulation 100 times, then average their results for every target-bug and
considered approach. Finally, we aggregate these averages computed on all
target bugs and normalise them as global percentages of achieved fault
detection by spent effort, in terms of mutants analysed.

Finally, to answer RQ3, we select example mutants that enabled $\mu$Bert to
find bugs exclusively (not found by any of PiTest versions), from the results
of RQ2. Then we discuss the added value of $\mu$Bert mutations through the
analysis of the mutants’ behavioural difference from the fixed version and
similarity with the buggy one.

###  5.3 Implementation

We implemented $\mu$Bert’s approach as described in Section 3: we have used
Spoon [51] and Jdt [21] libraries to parse and extract the business logic
related AST nodes and apply condition-seeding mutators. To predict the masked
tokens we have used the implementation proposed by CodeBERT-nt [31, 3], using
CodeBERT Masked Language Modeling (MLM) functionality [22, 2].

We provide the implementation of our approach and the reproduction package of
its evaluation at https://github.com/Ahmedfir/mBERTa.

##  6 Experimental Results

###  6.1 RQ1: $\mu$Bert Additive mutations

(a) Effectiveness: mean fault-detection per subject.

(b) Cost-efficiency: fault detection by the number of mutants analysed.

Figure 2: Fault-detection performance improvement when using additive
patterns. Comparison between $\mu$Bert and $\mu$Bertconv, w.r.t. the fault-
detection of test suites written to kill all generated mutants.

To answer this question we compare the fault detection effectiveness of test
suites written to kill mutants generated by $\mu$Bert with and without
additive mutations, noted respectively $\mu$Bert and $\mu$Bertconv. Figure 2
depicts the fault detection improvement when extending $\mu$Bert mutations by
the additive ones. In fact, $\mu$Bert fault detection increased on average by
over  9% compared to the one achieved by $\mu$Bertconv, achieving  84.64% on
average. We can also see that besides outliers, the majority of bugs are found
in  100% of the times. Moreover, when examining the bugs separately, we find
that $\mu$Bert finds  20 more bugs than $\mu$Bertconv (with fault detection
$>0\%$), and  70 more when considering bugs found with fault detection
percentages above 90%. This confirms that the additive patterns induce
relevant mutants ensuring the detection of some bugs always or in most of the
cases, as well as representing better new types of faults, which were not
detectable otherwise.

To check the significance of the fault detection advantage brought by the
additive patterns, we performed a statistical test (Wilcoxon paired test) on
the data of Figure 2(a) to validate the hypothesis ”the fault detection
yielded by $\mu$Bert is greater than the one by $\mu$Bertconv ”. The very
small obtained p-values of  5.92e-21 ($\ll$ 0.05) showed that the differences
are significant, indicating the low probability of this fault detection
amelioration to be happening by chance. The difference size confirms also the
same advantage, with $\hat{\text{A}}_{12}$ values of  0.5827 ($>$ 0.5),
indicating that $\mu$Bert induces test-suites with higher fault detection
capability in the majority of the cases.

(a) Effectiveness: mean fault-detection per subject.

(b) Cost-efficiency: fault detection by the number of mutants analysed.

Figure 3: Fault-detection comparison between $\mu$Bert and $\mu$Bertconv, with
the same effort: where the maximum effort is limited to the minimum effort
required to analyse all mutants of any of them, which is $\mu$Bertconv in most
of the cases.

Next, we compare the fault detection performance of $\mu$Bert and
$\mu$Bertconv when analysing the same number of mutants, and illustrate in
Figure 3 their average fault detection effectiveness and cost-efficiency in
terms of analysed mutants. The box-plots of the Subfigure 3(a) show that even
when spending the same effort as $\mu$Bertconv, $\mu$Bert keeps a similar
advantage of on average  6.05% higher fault detection, achieving a maximum of
81.35%. From the line-plots of the Subfigure 3(b), we can see that both
approaches achieve a comparable fault detection  ($\approx 70\%$) at
($\leq\approx 40\%$) of the maximum costs. At higher costs, $\mu$Bertconv’s
curve increases slowly until achieving a plateau at  $\approx 60\%$ of the
effort, whereas $\mu$Bert’s curve keeps increasing towards higher fault
detection ratios even when achieving the  $\approx 100\%$ of the fixed maximum
effort.

To validate these findings we re-conducted the same statistical tests on the
data of Subfigure 3(a) and found that $\mu$Bert outperforms significantly
$\mu$Bertconv with negligible p-values of  1.15e-19 and $\hat{\text{A}}_{12}$
values of  0.5711.

###  6.2 RQ2: Fault Detection comparison with PiTest

(a) Effectiveness: mean fault-detection per subject.

(b) Cost-efficiency: fault detection by the number of mutants analysed.

Figure 4: Fault-detection comparison between $\mu$Bert and PiTest, with the
same effort: where the maximum effort is limited to the minimum effort
required to analyse all mutants of any of them, which is Pit-default in most
of the cases.

To answer this research question we reduce our dataset to the bugs covered by
$\mu$Bert and the 3 considered versions of PitTest approaches: ”Pit-default”
which contains the default mutation operators of PiTest, ”Pit-all” containing
all PiTest operators including the default ones and ”Pit-rv-all” which
contains experimental operators [7] in addition to the ”Pit-all” ones. Then,
we perform the same study as in RQ1, where we compare the considered
approaches’ effectiveness and cost-efficiency based on the fault detection
capability of test suites written to kill their generated mutants. To have a
fair base of comparison, we compare the approaches under the same effort in
analysing mutants, which is equal to the least average effort required to kill
all mutants of one of the approaches (which is the one of Pit-default in the
majority of the cases). As we are interested in comparing the mutation testing
approaches and not mutant selection strategies, we run the simulation with the
same one-mutant-per-line random sampling of mutants for all techniques (see
Subsection 5.2).

Figure 4(b) shows that with small effort  ($\leq\approx 5\%$) all approaches
yield comparable fault detection scores  ($\approx 10\%$). However, the
difference becomes more noticeable when spending more effort, with $\mu$Bert
outperforming all versions of PiTest; achieving on average  16.53% higher
fault detection scores than Pit-default,  10.10% higher than Pit-rv-all and
5.56% higher than Pit-all (see Figure 4(a)).

TABLE II: Paired (per subject bug) statistical tests of the average fault
detection of test suites written to kill the same number of mutants generated
by each approach (data of Figure 4(a)).

(a) Wilcoxon paired test p-values computed on every dataset subject, comparing each approach (A1) from the first column to the other approaches (A2). p-values smaller than 0.05 indicate that (A1) yields an average fault detection significantly higher than that of (A2). p-values | Pit-rv-all | Pit-default | Pit-all
---|---|---|---
$\mu$BERT | 7.78e-11 | 1.18e-12 | 3.32e-02
Pit-all | 1.54e-22 | 8.87e-06 | –
Pit-default | 9.55e-01 | – | –

(b) Vargha and Delaney $\hat{\text{A}}_{12}$ values computed on every dataset subject, comparing each approach (A1) from the first column to the other approaches (A2). $\hat{\text{A}}_{12}$ values higher than 0.5 indicate that (A1) yields an average fault detection higher than that of (A2) in the majority of the cases. $\hat{\text{A}}_{12}$ | Pit-rv-all | Pit-default | Pit-all
---|---|---|---
$\mu$BERT | 0.6488 | 0.5514 | 0.5066
Pit-all | 0.7210 | 0.4956 | –
Pit-default | 0.5449 | – | –

To validate these results, we performed the same statistical tests as in RQ1,
checking the hypothesis that ”$\mu$Bert yields better fault detection
capabilities than the other approaches”. We illustrate in the first row of
Tables II(a) and II(b) the corresponding computed Wilcoxon paired test
p-values and Vargha and Delaney $\hat{\text{A}}_{12}$ values. Our results show
that $\mu$Bert has a significant advantage over the considered SOA approaches
with p-values under  0.05. Additionally, $\mu$Bert scores
$\hat{\text{A}}_{12}$ values above  0.5 which confirms that guiding the
testing by $\mu$Bert mutants instead of those generated by SOA techniques
yields comparable or higher fault detection ratios, in the majority of the
cases. Indeed, the $\hat{\text{A}}_{12}$ difference between Pit-all and
$\mu$Bert is small ( 0.5066), indicating that both approaches perform
similarly or better on some studied subjects and worst on others.

(a) Effectiveness: mean fault-detection per subject.

(b) Cost-efficiency: fault detection by the number of mutants analysed.

Figure 5: Comparison between $\mu$Bert and PiTest, relative to the fault-
detection of test suites written to kill all generated mutants.

We notice also from the sub-figure 4(b) that Pit-default achieves a plateau at
around  60% of the effort while the other tools keep increasing, showing that
they are able to achieve higher fault detection capabilities, at a higher
cost. This is very noticeable when we compare the sub-figures (a) and (b) of
Figure 4 with the figure 2, where the average fault detection of $\mu$Bert is
way lower than what it achieves in RQ1 – around  66% instead of  84%. This is
a direct consequence of the fact that Pit default produces fewer mutants than
the other approaches, limiting considerably the maximum effort of the mutation
campaigns and thus the fault detection ratios, in the majority of the cases.
Indeed, as illustrated in Figure 5, all approaches score higher fault
detection percentages when spending more effort, achieving on average
$\approx$65% for Pit-all,  $\approx$66% for Pit-rv-all and  $\approx$83% for
$\mu$Bert. We explain the small decrease of  1.72% in the mean fault detection
achieved by $\mu$Bert in comparison with RQ1 ( 82,92% in RQ2 instead of
84.64% in RQ1) by the difference in the considered dataset for each RQ.

TABLE III: Paired (per subject bug) statistical tests of the average fault
detection of test suites written to kill all the mutants generated by each
approach (data of Figure 5(a)).

(a) Wilcoxon paired test p-values computed on every dataset subject, comparing each approach (A1) from the first column to the other approaches (A2). p-values smaller than 0.05 indicate that (A1) yields an average fault detection significantly higher than that of (A2). p-values | Pit-rv-all | Pit-default | Pit-all
---|---|---|---
$\mu$BERT | 2.49e-13 | 2.14e-33 | 1.47e-14
Pit-all | 4.71e-01 | 2.76e-23 | –
Pit-default | 1.00e+00 | – | –

(b) Vargha and Delaney $\hat{\text{A}}_{12}$ values computed on every dataset subject, comparing each approach (A1) from the first column to the other approaches (A2). $\hat{\text{A}}_{12}$ values higher than 0.5 indicate that (A1) yields an average fault detection higher than that of (A2) in the majority of the cases. $\hat{\text{A}}_{12}$ | Pit-rv-all | Pit-default | Pit-all
---|---|---|---
$\mu$BERT | 0.6028 | 0.7123 | 0.6061
Pit-all | 0.5077 | 0.6400 | –
Pit-default | 0.3676 | – | –

In Table III, we illustrate the $\hat{\text{A}}_{12}$ and p-values computed on
data of the boxplots in Sub-figure 5(a). The results confirm that $\mu$Bert
outperforms significantly SOA mutation testing w.r.t the fault detection
capability of test suites written to all kill mutants generated by each
approach.

(a) Faults discovered at least once per 100 runs
(Fault detection $>0\%$).

(b) Faults discovered in over 90% of the runs
(Fault detection$\geq 90\%$).

Figure 6: Number of faults discovered by test-suites written to kill mutants
generated by $\mu$Bert and PiTest versions when analysing the same number of
mutants (same effort).

Next, we turned our interest to the set of particular bugs that every approach
can and cannot reveal when spending the same effort. Hence, we map each bug
with its revealing tool, from the simulation results of Figure 4(a) and
illustrate their corresponding Venn diagrams in Figure 6.

From the disjoint sets in Sub-figure 6(a), we notice a clear advantage in
using $\mu$Bert over the considered SOA baselines, as it finds most of the
bugs they find in addition to finding exclusively  47 bugs when spending the
same effort. More precisely, $\mu$Bert finds  52,  77 and  52 more bugs than
Pit-all, Pit-default and Pit-rv-all, respectively, whereas they find each  13,
10 and  13 bugs that $\mu$Bert missed.

This endorses the fact that $\mu$Bert introduces mutants that represent more
real bugs than SOA mutation techniques, and encourages the investigation of
the eventual complementary between the approaches. This observation is more
noticeable when considering the overlapping between bugs found by each
approach in at least 90% of the simulations (Sub-figure 6(b)). We notice that
the approaches perform comparably, with a particular distinction of Pit-all
and Pit-default results which find exclusively  19 and  21 bugs with these
high fault detection percentages instead of  0, as observed in Sub-figure
6(a). Nevertheless, $\mu$Bert conserves the same advantage over the considered
baselines in this regard, finding exclusively  42 bugs more. It finds also
50,  63 and  69 more bugs than respectively Pit-all, Pit-default and Pit-rv-
all, whereas they find each  59,  58 and  27 bugs that $\mu$Bert missed.

###  6.3 RQ3: Qualitative Analysis of $\mu$Bert Mutants

To answer this research question we investigate the mutants generated by
$\mu$Bert, which induced test suites able to find bugs that were not detected
otherwise, i.e. by the considered SOA approaches (see Figure 6). Meaning that
the mutants break similar tests as the target real buggy version.

TABLE IV: Example of mutants generated by $\mu$Bert that helped find the bug
Lang-49 from Defects4J. Mutant 1: replacing binary operator
---
⬇ @@ org.apache.commons.lang.math.Fraction : 466 @@ - if (numerator == 0) { +
if (numerator > 0) {
Mutant 2: replacing literal implementation
⬇ @@ org.apache.commons.lang.math.Fraction : 466 @@ - if (numerator == 0) { +
if (numerator == 1) {
Mutant 3: adding a condition to an if statement
⬇ @@ org.apache.commons.lang.math.Fraction : 466 @@ - if (numerator == 0) { +
if ((numerator == 0) + || !(numerator==Integer.MIN_VALUE)) {
Mutant 4: replacing a condition
⬇ @@ org.apache.commons.lang.math.Fraction : 467 @@ - return equals(ZERO) ?
this: ZERO; + return this;
Mutant 5: replacing this access by another object
⬇ @@ org.apache.commons.lang.math.Fraction : 467 @@ - return equals(ZERO) ?
this: ZERO; + return equals(ZERO) ? ONE: ZERO;
Mutant 6: replacing method argument
⬇ @@ org.apache.commons.lang.math.Fraction : 469 @@ int gcd =
greatestCommonDivisor( - Math.abs(numerator), denominator); +
Math.abs(numerator), 1);
Mutant 7: replacing a variable
⬇ @@ org.apache.commons.lang.math.Fraction : 473 @@ - return
Fraction.getFraction(numerator / gcd, + return Fraction.getFraction(numerator
/ 3, denominator / gcd);
Mutant 8: adding a condition to a return statement
⬇ @@ org.apache.commons.lang.math.Fraction : 840 @@ return (getNumerator() ==
other.getNumerator() - && getDenominator() == other.getDenominator()); + &&
getDenominator() == other.getDenominator())) + || (numerator ==
other.numerator);

As a simple bug example (requiring only one change to fix it), we consider
Lang-49 from Defects4J and we investigate mutants that have been generated by
$\mu$Bert and helped in generating tests that reveal it. This bug impacts the
results of the method reduce() from the class
org.apache.commons.lang.math.Fraction, which returns a new reduced fraction
instance, if possible, or the same instance, otherwise. The bug is caused by a
miss-implementation of a specific corner case, which consists of calling the
method on a fraction instance that has $0$ as numerator. In Table IV, we
illustrate example mutants generated by $\mu$Bert that helped in revealing
this bug. Every mutant is represented by a diff between the fixed and the
mutated version by $\mu$Bert.

As can be seen, $\mu$Bert can generate mutants that can be induced by applying
conventional pattern-based mutations, i.e., Mutant 1 replaces a relational
operator (==) with another (>) and Mutant 2 replaces an integer operand (0)
with another one (1).

In addition, it proposes more complex mutations that are unlikely achievable
without any knowledge of either the AST or the context of the considered
program. For instance, it can generate Mutant 4 by changing a conditional
return statement with (this) the current instance of Fraction, which matches
the return type of the method. Similarly, to generate Mutant 5, it replaces
(this) the current instance of the class Fraction by an existent instance of
the same type (ONE), making the statement returning either the object ONE or
the object ZERO.

To produce more complex mutants, $\mu$Bert applies a condition seeding
followed by token-masking and CodeBERT prediction, such as adding ||
(numerator == other.numerator) to the original condition of a return
statement, inducing Mutant 8, or adding || !(numerator == Integer.MIN_VALUE)
to the original condition of an if statement, inducing Mutant 3.

TABLE V: Example of mutants generated by $\mu$Bert that helped in finding bugs
from Defects4J and could not be generated when limiting the maximum number of
surrounding tokens to 10. Mutant 1 (JacksonCore-4) : replacing a method call
---
⬇ @@ com.fasterxml.jackson.core.util.TextBuffer : 515 @@ - unshare(1); +
expand(1);
Mutant 2 (Closure-26) : replacing an object
⬇ @@ com.google.javascript.jscomp.ProcessCommonJSModules : 89 @@ -
.replaceAll(Pattern.quote(File.separator), MODULE_NAME_SEPARATOR) +
.replaceAll(Pattern.quote(filename), MODULE_NAME_SEPARATOR)
Mutant 3 (Closure-35) : replacing a method call
⬇ @@ com.google.javascript.jscomp.TypeInference : 1092 @@ - scope =
traverseChildren(n, scope); + scope = traverse(n, scope);
Mutant 4 (Lang-27) : replacing a method call
⬇ @@ org.apache.commons.lang3.math.NumberUtils : 526 @@ - if (!(f.isInfinite()
|| (f.floatValue() == 0.0F && !allZeros))) { + if (!(f.isInfinite() ||
(f.round() == 0.0F && !allZeros))) { // also ”f.floatValue()” to ”f.scale()”
Mutant 5 (Math-64) : replacing an object
⬇ @@ org.apache.commons.lang.math.Fraction : 852 @@ - for (int j = k; j <
jacobian.length; ++j) { + for (int j = k; j < beta.length; ++j) {
Mutant 6 (Lang-27) : replacing an object
⬇ @@ org.apache.commons.lang3.math.NumberUtils : 526 @@ - if (!(f.isInfinite()
|| (f.floatValue() == 0.0F && !allZeros))) { + if (!(f.isInfinite() ||
(f.round() == 0.0F && !zero))) {

To investigate further the impact of the code context captured by the model on
the generated mutants, we have rerun $\mu$Bert on 5 subjects from our dataset,
with a maximum number of surrounding tokens equal to 10 (instead of 512).
Then, we compared manually the induced mutants with those generated by our
default setup, in the same locations. From our results, we observed a
noticeable decrease in the number of compilable predictions, indicating the
general performance decrease of the model when it lacks information about the
code context. Particularly, we notice that it is not able to produce program-
specific mutants, i.e. by changing an object by another or a method call with
another. In Table V, we illustrate some example mutants that helped find each
of the studied subjects (breaking same tests as the original bug), which
$\mu$Bert failed to generate when the maximum number of surrounding tokens is
limited to 10.

##  7 Threats to Validity

One external threat to validity concerns the generalisation of our findings
and results in the empirical evaluation. To reduce this threat, we used a
large number of real bugs from popular open-source projects with their
associated developer test-suites, provided by an established and independently
built benchmark (i.e. Defects4J [29]). Nevertheless, we acknowledge that the
results may be different considering projects in different domains.

Other threats may arise from our way of assessing the fault detection
capability of mutation testing approaches, based on their capability of
guiding the testing via a developer/tester simulation in which we assume that
the current test suites are complete and the not killed mutants are
equivalent. Although we acknowledge that this may not be the case in real-
world scenarios, we believe that this process is sufficient to evaluate our
approach, particularly considering the fact the test suites provided by
Defects4J are relatively strong. Additionally, to mitigate any comparison
threat between the considered approaches, we use consistently and similarly
the same test-suites, setups and simulation assumptions in all our study.

The choice of our comparison baseline may form other threats to the validity
of our findings. While different fault-seeding approaches have been proposed
recently, PiTest remains among the most mature and stable mutation testing
tools for Java programs, thus, forming an appropriate comparison baseline to
evaluate our work. Furthermore, we compared our results with those obtained by
mutants from different configurations proposed by PiTest, enlarging our study
to the different audiences targeted by this latter. We acknowledge however
that the results may change when considering other techniques and consider the
evaluation of the effectiveness and cost-efficiency of different mutation
testing techniques as out of the scope of this paper.

Other construct threats may arise from considering the number of mutants
analysed as metric to measure the effort required to find a fault. In addition
to the fact that this metric has been widely used by the literature [47, 9,
34], we believe that it is intuitive and representative of the global manual
effort of the tester in analysing the mutants, discarding them or writing
tests to kill them. While being the standard in the literature, we acknowledge
that this measure does not account for the cost difference between mutants,
attributing the same cost to all mutants. This is simply because we do not
know the specific effort required to analyse every specific mutant or to write
every specific test. Additionally, our cost-efficiency results may be impacted
by costs that are not captured with this metric, such as the execution or the
developing effort of either CodeBERT, the key component of $\mu$Bert, or the
set of patterns and execution enhancements over the different releases of
PiTest. Nevertheless, we tried to mitigate any major threats that can be
induced by the implementation of the different tools, i.e. we reduce the
dataset subjects to those on which every approach generated at least one
mutant and consider any implementation difference between the approaches as
out of the current scope.

##  8 Related Work

Since the 1970s, mutation testing has been the main focus of multiple research
works [57]. Their findings have proven that artificial faults can be useful in
multiple software engineering applications, such as testing [47], debugging
[48, 37], assessing fault tolerance [42], risk analysis [16, 56] and
dependability evaluation [10].

Despite this long history of research, the generation of relevant mutants
remains an open question. Most of the related research has focused on the
design of fault patterns (mutation operators) which are usually defined based
on the target language grammar [8, 47] then refined through empirical studies
[44, 40, 33] aiming at reducing the redundancy and noise among their generated
mutants. The continuous advances in this sense were followed by a constant
emergence of pattern-based mutation testing tools and releases [39, 35, 17],
among which some are becoming popular and widely adopted by researchers and
practitioners, such as PiTest [17], from which we consider three
configurations as our comparison baseline.

Recent research has focused their interest on improving the representativeness
of artificial faults aiming at reducing the mutation space to real-like
faults. For instance, instead of basing the mutation operators’ design on the
programming language grammar, Brown et al. [12] proposed inferring them from
real bug fixes. Similarly, Tufano et al. [54] proposed a neural machine
translation technique that learns how to inject faults from real bug fixes.
Along the same line, Patra et al. [50] proposed a semantic-aware learning
approach, that learns and then adapts fault patterns to the project of
interest. Their results are promising, however, the fact that these techniques
depend on the availability of numerous, diverse, comprehensive and untangled
fix commits [27] of not coupled faults [43], which is often hard to fulfil in
practice, may hinder their performance. Acknowledging for the injection
location [42, 13], Khanfir et al. [32] combined the usage of information
retrieved from bug reports with inverted automated-program-repair patterns to
replicate real faults fixable by the original fix-patterns. Their results
showed that they can generate faults that mimic real ones, however, their
approach remains dependent and limited to the presence of good bug reports.
Overall, designing the mutation operators based on the known faults space
yields more diverse mutants that represent more fault types. However, these
extended operator sets tend to increase the number of generated mutants and
consequently the general cost of the mutation campaign i.e. the fault patterns
proposed by Brown et al. and Khanfir et al. counted also most of the
conventional mutators in addition to new ones. Unlike these techniques,
$\mu$Bert leverages pre-trained models to introduce mutants based on code
knowledge instead of the faults one. As code is more available than faults, it
offers a more flexible and complete knowledge base than faults, i.e. it perms
to overcome the limitations and efforts required 1) to collect clean bug-
fixing commits, 2) to capture the faulty behaviour and 3) design fault
patterns, be it manually or via machine learning techniques.

Aiming at reducing the number of generated mutants, researchers have proposed
different strategies to generate relevant mutants. For instance, studies that
show that mutant strength resides in not only its inducing pattern but also
where it is injected [42, 13], motivated the importance of selecting relevant
locations to mutate. In this regard, Sun et al. [53] suggest mutating multiple
places within diverse program execution paths. Gong et al. [26] also propose
the mutation in diverse locations of the program extracted from graph
analysis. Similarly, Mirshokraie et al. [41] compute complexity metrics from
program executions to extract locations with good observability to mutate.
Other approaches restrict the fault injection on specific locations of the
program, such as the code impacted by the last commits [58, 38] for better
usability in continuous integration, or targeting locations related to a given
bug-report [32] to target a specific feature or behaviour, etc. More recent
advances have resulted in powerful techniques for cost-effectively selecting
mutants, i.e., by avoiding the analysis of redundant mutants (basically,
equivalent and subsumed ones) [24, 28, 25]. In particular, the work of Garg et
al. [24] utilises the knowledge of mutants’ surrounding context, embedded into
the vector space, to predict whether a mutant is likely subsuming or not. In
this work, we do not target any specific code part or any narrow use case, but
instead, perform fault injection in a brute-force way similarly to mutation
testing, by iterating every program statement and masking every involved
token.

Multiple studies have been focused on the relationship between artificial and
real faults [47]. The results of the studies conducted by Ojdanic et al. [45],
Papadakis et al. [49], Just et al. [30] and Andrews et al. [9] showed that
there is a correlation between tests broken by a bug and tests killing
mutants. Meaning that artificial faults can be used as alternatives to real
faults in controlled studies. Moreover, the findings of Chekam et al. [14],
Frankl et al. [23] and Li et al. [36] show that guiding testing by mutants
leads to significantly higher fault revelation capability than the ones of
other test adequacy criteria. Based on these findings, we assess our approach
based on the relation between the injected and real faults, in terms of
breaking tests. More precisely, we conduct a fault detection effectiveness and
cost-efficiency study to evaluate our approach’s mutants in guiding testing
and compare it to state-of-the-art techniques. Furthermore, we discuss the
diversity and readability of $\mu$Bert mutants through real examples.

The closest related work is a preliminary implementation of $\mu$Bert that was
recently presented in the 2022 mutation workshop [18]. This implementation,
denoted as $\mu$Bertconv in our evaluation, includes the conventional
mutations (to mask and replace tokens by the model predictiosn), but it does
not include the condition-seeding additive mutations that provide major
benefits for fault detection. Moreover, $\mu$Bertconv was evaluated only on 40
bugs from Defects4J, and compared only to an early version of PiTest (similar
to Pit-rv-all). In this work, we perform an extensive experimental evaluation
including 689 bugs from Defects4J and compare $\mu$Bert effectiveness with
three different configurations from PiTest. Moreover, we show that $\mu$Bert
finds on average more bugs than $\mu$Bertconv without requiring more effort.

##  9 Conclusion

We presented $\mu$Bert; a pre-trained language model based fault injection
approach. $\mu$Bert provides researchers and practitioners with easy-to-
understand “natural” mutantsto help them in writing tests of higher fault
revelation capabilities.

Unlike state-of-the-art approaches, it does neither require nor depend on any
kind of faults knowledge or language grammar but instead on the actual code
definition and distribution, as written by developers in numerous projects.
This facilitates its developing, maintainability, integration and extension to
different programming languages. In fact, it reduces the overhead of learning
how to mutate, be it via creating and selecting patterns or collecting good
bug-fixes and learning from their patches.

In a nutshell, $\mu$Bert takes as input a given program and replaces different
pieces of its code base with predictions made by a pretrained generative
language model, producing multiple likely-to-occur mutations. The approach
targets diverse business code locations and injects either simple one-token
replacement mutants or more complex ones by extending the control-flow
conditions. This provides probable developer-like faults impacting different
functionalities of the program with higher relevance and lower cost to
developers. This is further endorsed by our results where $\mu$Bert induces
high fault detection test suites at low effort, outperforming state-of-the-art
techniques (PiTest), in this regard.

We have made our implementation and results available [5] to enable
reproducibility and support future research.

## Acknowledgment

This work was supported by the Luxembourg National Research Fund (FNR)
projects C20/IS/14761415/TestFlakes and TestFast, ref. 12630949.

## References

  * [1] Amazon codewhisperer.  https://aws.amazon.com/codewhisperer/.
  * [2] Codebert.  https://github.com/microsoft/CodeBERT.
  * [3] Codebert-nt.  https://github.com/Ahmedfir/CodeBERT-nt.
  * [4] Github copilot.  https://github.com/features/copilot.
  * [5] mberta.  https://github.com/Ahmedfir/mBERTa.
  * [6] Pitest.  http://pitest.org/.
  * [7] Pitest-rv-plugin.  https://github.com/pitest/pitest-rv-plugin.
  * [8] Paul Ammann and Jeff Offutt.  Introduction to Software Testing.  Cambridge University Press, 2008.
  * [9] James H. Andrews, Lionel C. Briand, Yvan Labiche, and Akbar Siami Namin.  Using mutation analysis for assessing and comparing testing coverage criteria.  IEEE Trans. Software Eng., 32(8):608–624, 2006.
  * [10] Jean Arlat, Alain Costes, Yves Crouzet, Jean-Claude Laprie, and David Powell.  Fault injection and dependability evaluation of fault-tolerant systems.  IEEE Trans. Computers, 42(8):913–923, 1993.
  * [11] Moritz Beller, Chu-Pan Wong, Johannes Bader, Andrew Scott, Mateusz Machalica, Satish Chandra, and Erik Meijer.  What it would take to use mutation testing in industry - A study at facebook.  In 43rd IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, ICSE (SEIP), pages 268–277. IEEE, 2021.
  * [12] David Bingham Brown, Michael Vaughn, Ben Liblit, and Thomas W. Reps.  The care and feeding of wild-caught mutants.  In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE, pages 511–522. ACM, 2017.
  * [13] Thierry Titcheu Chekam, Mike Papadakis, Tegawendé F. Bissyandé, Yves Le Traon, and Koushik Sen.  Selecting fault revealing mutants.  Empirical Software Engineering, 25(1):434–487, 2020.
  * [14] Thierry Titcheu Chekam, Mike Papadakis, Yves Le Traon, and Mark Harman.  An empirical study on mutation, statement and branch coverage fault revelation that avoids the unreliable clean program assumption.  In Proceedings of the 39th International Conference on Software Engineering, ICSE 2017, Buenos Aires, Argentina, May 20-28, 2017, pages 597–608, 2017.
  * [15] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, J Kaplan, H Edwards, Y Burda, N Joseph, G Brockman, et al.  Evaluating large language models trained on code.(2021).  arXiv preprint arXiv:2107.03374, 2021.
  * [16] Jörgen Christmansson and Ram Chillarege.  Generation of error set that emulates software faults based on field data.  In Digest of Papers: FTCS-26, The Twenty-Sixth Annual International Symposium on Fault-Tolerant Computing, 1996, pages 304–313. IEEE Computer Society, 1996.
  * [17] Henry Coles, Thomas Laurent, Christopher Henard, Mike Papadakis, and Anthony Ventresque.  Pit: A practical mutation testing tool for java (demo).  In Proceedings of the 25th International Symposium on Software Testing and Analysis, ISSTA 2016, page 449–452, New York, NY, USA, 2016. Association for Computing Machinery.
  * [18] Renzo Degiovanni and Mike Papadakis.  $\mathrm{\mu}$bert: Mutation testing using pre-trained language models.  In 15th IEEE International Conference on Software Testing, Verification and Validation Workshops ICST Workshops 2022, Valencia, Spain, April 4-13, 2022, pages 160–169. IEEE, 2022.
  * [19] Richard A. DeMillo, Richard J. Lipton, and Frederick G. Sayward.  Hints on test data selection: Help for the practicing programmer.  IEEE Computer, 11(4):34–41, 1978.
  * [20] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [21] JDT Eclipse.  Eclipse java development tools (jdt), 2013.
  * [22] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou.  Codebert: A pre-trained model for programming and natural languages.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP, volume EMNLP 2020 of Findings of ACL, pages 1536–1547. Association for Computational Linguistics, 2020.
  * [23] Phyllis G. Frankl, Stewart N. Weiss, and Cang Hu.  All-uses vs mutation testing: An experimental comparison of effectiveness.  J. Syst. Softw., 38(3):235–253, 1997.
  * [24] Aayush Garg, Milos Ojdanic, Renzo Degiovanni, Thierry Titcheu Chekam, Mike Papadakis, and Yves Le Traon.  Cerebro: Static subsuming mutant selection.  IEEE Trans. Software Eng.
  * [25] Rohit Gheyi, Márcio Ribeiro, Beatriz Souza, Marcio Augusto Guimarães, Leo Fernandes, Marcelo d’Amorim, Vander Alves, Leopoldo Teixeira, and Baldoino Fonseca.  Identifying method-level mutation subsumption relations using Z3.  Inf. Softw. Technol., 132:106496, 2021.
  * [26] Dunwei Gong, Gongjie Zhang, Xiangjuan Yao, and Fanlin Meng.  Mutant reduction based on dominance relation for weak mutation testing.  Information & Software Technology, 81:82–96, 2017.
  * [27] Kim Herzig and Andreas Zeller.  Untangling changes.  Unpublished manuscript, September, 37:38–40, 2011.
  * [28] Claudinei Brito Junior, Vinicius H. S. Durelli, Rafael Serapilha Durelli, Simone R. S. Souza, Auri M. R. Vincenzi, and Márcio Eduardo Delamaro.  A preliminary investigation into using machine learning algorithms to identify minimal and equivalent mutants.  In 13th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW, pages 304–313. IEEE, 2020\.
  * [29] René Just, Darioush Jalali, and Michael D. Ernst.  Defects4j: A database of existing faults to enable controlled testing studies for java programs.  In Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, page 437–440, New York, NY, USA, 2014. Association for Computing Machinery.
  * [30] René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser.  Are mutants a valid substitute for real faults in software testing?  In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, (FSE-22), Hong Kong, China, November 16 - 22, 2014, pages 654–665. ACM, 2014.
  * [31] Ahmed Khanfir, Matthieu Jimenez, Mike Papadakis, and Yves Le Traon.  Codebert-nt: code naturalness via codebert.  arXiv preprint arXiv:2208.06042, 2022.
  * [32] Ahmed Khanfir, Anil Koyuncu, Mike Papadakis, Maxime Cordy, Tegawende F. Bissyandé, Jacques Klein, and Yves Le Traon.  Ibir: Bug report driven fault injection.  ACM Trans. Softw. Eng. Methodol., may 2022.
  * [33] Marinos Kintis, Mike Papadakis, Andreas Papadopoulos, Evangelos Valvis, Nicos Malevris, and Yves Le Traon.  How effective are mutation testing tools? an empirical analysis of java mutation testing tools with manual analysis and real faults.  Empir. Softw. Eng., 23(4):2426–2463, 2018.
  * [34] Bob Kurtz, Paul Ammann, Jeff Offutt, Márcio Eduardo Delamaro, Mariet Kurtz, and Nida Gökçe.  Analyzing the validity of selective mutation with dominator mutants.  In Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016, Seattle, WA, USA, November 13-18, 2016, pages 571–582, 2016.
  * [35] T. Laurent, M. Papadakis, M. Kintis, C. Henard, Y. L. Traon, and A. Ventresque.  Assessing and improving the mutation testing practice of pit.  In 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST), pages 430–435, March 2017.
  * [36] Nan Li, Upsorn Praphamontripong, and Jeff Offutt.  An experimental comparison of four unit test criteria: Mutation, edge-pair, all-uses and prime path coverage.  In Second International Conference on Software Testing Verification and Validation, ICST, 2009, Workshops Proceedings, pages 220–229. IEEE Computer Society, 2009.
  * [37] Yiling Lou, Ali Ghanbari, Xia Li, Lingming Zhang, Haotian Zhang, Dan Hao, and Lu Zhang.  Can automated program repair refine fault localization? a unified debugging approach.  In ISSTA ’20: 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, USA, July 18-22, 2020, pages 75–87. ACM, 2020.
  * [38] Wei Ma, Thomas Laurent, Milos Ojdanic, Thierry Titcheu Chekam, Anthony Ventresque, and Mike Papadakis.  Commit-aware mutation testing.  In Proceedings of the 36th IEEE International Conference on Software Maintenance and Evolution, ICSME, 2020.
  * [39] Yu-Seung Ma, Jeff Offutt, and Yong Rae Kwon.  Mujava: an automated class mutation system.  Softw. Test. Verification Reliab., 15(2):97–133, 2005.
  * [40] Michaël Marcozzi, Sébastien Bardin, Nikolai Kosmatov, Mike Papadakis, Virgile Prevosto, and Loïc Correnson.  Time to clean your test objectives.  In Michel Chaudron, Ivica Crnkovic, Marsha Chechik, and Mark Harman, editors, Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018, pages 456–467. ACM, 2018.
  * [41] Shabnam Mirshokraie, Ali Mesbah, and Karthik Pattabiraman.  Guided mutation testing for javascript web applications.  IEEE Trans. Software Eng., 41(5):429–444, 2015.
  * [42] Roberto Natella, Domenico Cotroneo, João Durães, and Henrique Madeira.  On fault representativeness of software fault injection.  IEEE Trans. Software Eng., 39(1):80–96, 2013.
  * [43] A. Jefferson Offutt.  Investigations of the software testing coupling effect.  ACM Trans. Softw. Eng. Methodol., 1(1):5–20, 1992.
  * [44] A. Jefferson Offutt, Ammei Lee, Gregg Rothermel, Roland H. Untch, and Christian Zapf.  An experimental determination of sufficient mutant operators.  ACM Trans. Softw. Eng. Methodol., 5(2):99–118, 1996.
  * [45] Milos Ojdanic, Aayush Garg, Ahmed Khanfir, Renzo Degiovanni, Mike Papadakis, and Yves Le Traon.  Syntactic vs. semantic similarity of artificial and real faults in mutation testing studies.  arXiv preprint arXiv:2112.14508, 2021.
  * [46] Mike Papadakis, Thierry Titcheu Chekam, and Yves Le Traon.  Mutant quality indicators.  In 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops, pages 32–39. IEEE Computer Society, 2018.
  * [47] Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman.  Chapter six - mutation testing advances: An analysis and survey.  Advances in Computers, 112:275–378, 2019.
  * [48] Mike Papadakis and Yves Le Traon.  Metallaxis-fl: mutation-based fault localization.  Software Testing, Verification and Reliability, 25(5-7):605–628, 2015.
  * [49] Mike Papadakis, Donghwan Shin, Shin Yoo, and Doo-Hwan Bae.  Are mutation scores correlated with real fault detection?: a large scale empirical study on the relationship between mutants and real faults.  In Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018, pages 537–548, 2018.
  * [50] Jibesh Patra and Michael Pradel.  Semantic bug seeding: A learning-based approach for creating realistic bugs.  ESEC/FSE 2021, page 906–918, New York, NY, USA, 2021. Association for Computing Machinery.
  * [51] Renaud Pawlak, Martin Monperrus, Nicolas Petitprez, Carlos Noguera, and Lionel Seinturier.  Spoon: A Library for Implementing Analyses and Transformations of Java Source Code.  Software: Practice and Experience, 46:1155–1179, 2015.
  * [52] Ana Belén Sánchez, Pedro Delgado-Pérez, Inmaculada Medina-Bulo, and Sergio Segura.  Mutation testing in the wild: findings from github.  Empir. Softw. Eng., 27(6):132, 2022.
  * [53] Chang-ai Sun, Feifei Xue, Huai Liu, and Xiangyu Zhang.  A path-aware approach to mutant reduction in mutation testing.  Information & Software Technology, 81:65–81, 2017.
  * [54] Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk.  Learning how to mutate source code from bug-fixes.  In 2019 IEEE International Conference on Software Maintenance and Evolution, ICSME 2019, pages 301–312. IEEE, 2019.
  * [55] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [56] Jeffrey M. Voas, Frank Charron, Gary McGraw, Keith W. Miller, and Michael Friedman.  Predicting how badly ”good” software can behave.  IEEE Softw., 14(4):73–83, 1997.
  * [57] Jeffrey M. Voas and Gary McGraw.  Software Fault Injection: Inoculating Programs against Errors.  John Wiley & Sons, Inc., USA, 1997.
  * [58] Lingming Zhang, Darko Marinov, Lu Zhang, and Sarfraz Khurshid.  Regression mutation testing.  ISSTA 2012, page 331–341, New York, NY, USA, 2012. Association for Computing Machinery.