#  TOGLL: Correct and Strong Test Oracle Generation with LLMs

Soneya Binta Hossain  Department of Computer Science
University of Virginia
<EMAIL_ADDRESS>Matthew B. Dwyer  Department of Computer Science
University of Virginia
<EMAIL_ADDRESS>
###### Abstract

Test oracles play a crucial role in software testing, enabling effective bug
detection. Despite initial promise, neural-based methods for automated test
oracle generation often result in a large number of false positives and weaker
test oracles. While LLMs have demonstrated impressive effectiveness in various
software engineering tasks, including code generation, test case creation, and
bug fixing, there remains a notable absence of large-scale studies exploring
their effectiveness in test oracle generation. The question of whether LLMs
can address the challenges in effective oracle generation is both compelling
and requires thorough investigation.

In this research, we present the first comprehensive study to investigate the
capabilities of LLMs in generating correct, diverse, and strong test oracles
capable of effectively identifying a large number of unique bugs. To this end,
we fine-tuned seven code LLMs using six distinct prompts on the SF110 dataset.
Utilizing the most effective fine-tuned LLM and prompt pair, we introduce
TOGLL, a novel LLM-based method for test oracle generation. To investigate the
generalizability of TOGLL, we conduct studies on 25 large-scale Java projects.
Besides assessing the correctness, we also assess the diversity and strength
of the generated oracles. We compare the results against EvoSuite and the
state-of-the-art neural method, TOGA. Our findings reveal that TOGLL can
produce 3.8 times more correct assertion oracles and 4.9 times more exception
oracles. Moreover, our findings demonstrate that TOGLL is capable of
generating significantly diverse test oracles. It can detect 1,023 unique bugs
that EvoSuite cannot, which is ten times more than what the previous SOTA
neural-based method, TOGA, can detect.

##  I Introduction

Software dominates almost every aspect of our lives, including safety-critical
domains such as healthcare and autonomous transportation. Even seemingly minor
software bugs can cause large-scale system outages, security breaches, and
loss of lives [45, 30, 37]. Thus, ensuring software reliability through
rigorous testing and bug detection is paramount. Test oracles, the foundation
of effective testing, play a pivotal role in the early detection of software
bugs [15, 16, 41, 51].

Typically, a test suite consists of a set of test cases, where each test is
composed of a test prefix that exercises a certain part of a program, and a
test oracle verifies whether the executed behavior matches the expected
behavior [28]. Test oracles can be categorized into two main types: assertion
oracles, which judge the correctness of the output state of the program, and
exception oracles, which judge whether erroneous input states are detected by
the SUT.

The efficacy of test oracles is determined by their ability to detect bugs.
For effective bug detection, test oracles should be correct and strong. A test
oracle is considered correct if it aligns with the expected program behavior,
thereby avoiding false alarms. Additionally, a test oracle is deemed strong if
it can detect deviations from intended program behavior. Just because a test
oracle is correct does not mean it is strong, as demonstrated in [16]. Even
though developer-written test oracles are proven more stronger than automated
test oracles [51], manual generation of test oracles is both costly and time-
consuming [18, 3]. To address this, researchers have explored methods to
automate oracle generation.

Figure 1: Overview of our approach to explore LLM-based oracle generation and
to evaluate TOGLL.

Researchers have investigated the application of natural language processing
(NLP) and pattern matching techniques for generating test oracles from code
comments and natural language documentation [5, 14, 6, 33, 46]. These
approaches can generate both assertion oracles, which verify that a program’s
actual output matches its expected output, and exception oracle, which capture
the program’s anticipated exceptional behaviors during testing. Recent
advancements have seen the application of neural techniques for generating
test oracles, utilizing transformer-based models that learn from the method
under test (MUT) and developer-written test cases [49, 48, 47]. Building upon
these methods, the development of TOGA [10]—a neural-based test oracle
generator—outperformed prior work in detecting real faults. However, a recent
study shows that this state-of-the-art neural-based test oracle generation
method, when given a test prefix, the MUT and its docstring, generates
assertion oracles only 38% of the time. Moreover, when it does generate
assertion oracles 47% are false positives, and for exception oracles the false
positive rate is 81%. Despite significant research focused on generating test
oracles, automated generation of correct and strong test oracles has proven
challenging and the state of the art suffers from: generalizability issues,
high false positive rates and limited bug detection effectiveness [16]. These
challenges emphasize the ongoing need for further investigation, marking it as
a significant and open research problem.

Large language models (LLMs) have shown promise in software engineering tasks
such as code completion [25], program comprehension [29], automated repair
[19], and test generation [42, 24, 43]. Most prior work on ML-based test
generation has focused on the generation of test prefixes – inputs to the
system under test — with the goal of yielding substantial test coverage.
CodaMosa [24] uses LLMs to escape coverage plateaus in search based test input
generation and find it can improve coverage on small to moderate Python
programs. A recent attempt to use LLMs to generate test prefixes for large-
scale Java programs was not very successful [43]; it achieved only 2%
statement coverage on large-scale Java projects compared to over 90% for
EvoSuite’s [12] search-based approach. The LLM-based TESTPILOT [42] approach
generates both test prefixes and test oracles for JavaScript. It produces
better coverage than state-of-the-art search-based JavaScript test generation
techniques on a variety of small programs. With regard to test oracles,
TESTPILOT was shown to generate significant numbers of non-trivial test
assertions that are distinct from those generated by other methods, but this
work did not evaluate the strength of the generated test oracles for fault
detection.

In this paper, rather than task the LLM with generating both test prefix and
test oracle, as TESTPILOT did, we decouple these problems. We utilize state-
of-the-art test generation methods, such as EvoSuite, to create test prefixes
and concentrate on leveraging LLMs for test oracle generation. Initially, we
fine-tune seven code LLMs, with sizes ranging from 110M to 2.7B parameters,
employing six prompt formats that vary in the information they incorporate
about the method under test (MUT) and the test prefix. Employing the most
effective model and prompt pair, we introduce our LLM-based method, TOGLL. To
evaluate TOGLL’s generalizability on unseen projects, we conduct a
comprehensive study on 25 real-world Java projects and compare the correctness
of the generated oracles with those produced by the state-of-the-art neural
method TOGA [10] and EvoSuite. Subsequently, we assess the strength of the
generated oracles through a large-scale bug detection study, highlighting the
unique advantages of LLM-generated oracles over alternative methods. To the
best of our knowledge, this represents the first extensive study to explore
the application of LLMs in test oracle generation, examining various code
LLMs, prompt formats, and two large datasets, while evaluating multiple
aspects of the generated oracles, including correctness, diversity, and
effectiveness in bug detection.

In summary, the main contributions of the paper lie in:

  * •

Constructing a large dataset from 110 large-scale Java projects to fine-tune
code LLMs, aiming to investigate their performance in generating effective
test oracles.

  * •

Evaluating the generalizability of fine-tuned LLMs in generating effective
test oracles for unseen projects and compare them with SOTA methods.

  * •

Evaluating the impact of different contextual information on oracle generation
accuracy through fix specially designed prompts.

  * •

Demonstrating that well-tuned and prompted LLMs can generate correct, strong,
and diverse test oracles, with much lower false positive rates than SOTA-
methods and can detect large number of unique bugs neither detected by
EvoSuite nor SOTA neural method.

  * •

Releasing all datasets, fine-tuned models, and code to enable replication of
our study and the community to build on the work.

This work offers a promising starting point for LLM-based test oracle
generation, identifies directions for future improvement of such methods, and
thereby has the potential to lead to impactful new capabilities for software
engineers.

##  II Approach

Figure 1 presents an overview of our research approach. Step 1 of Figure 1
depicts fine-tuning a collection of LLMs with different prompting approaches
for test oracle generation task; we discuss this in Section II-A. Step 2 of
Figure 1 selects the best performing fine-tuned model and prompt from step 1
to define TOGLL and to evaluate its performance on unseen data. Step 3
assesses the correctness of TOGLL generated oracles through test execution; we
discuss this in Section II-B1. Finally, step 4 assesses the strength of TOGLL
generated oracles using mutation testing; we discuss this in Section II-C.

###  II-A Supervised Fine-tuning

For supervised fine-tuning, we use the SF110 dataset, construct six different
prompts, and fine-tune seven different code LLMs.

####  II-A1 SF110 Training Data

We construct the training dataset using the SF110 EvoSuite benchmark [13],
which comprises 110 open-source Java projects with over 23,000 Java classes
sourced from the SourceForge repository. This benchmark is widely recognized
for unit test generation. To construct our dataset, we utilize the test cases
generated by EvoSuite for each project. Given that a single test case can
contain multiple assertion oracles or even a combination of assertion and
exception oracles, we processed each test case. This involved decomposing them
into individual test cases, ensuring that each contained only a single
assertion or exception oracle. For each decomposed test case, we extracted the
method under test along with its associated documentation to prepare the
dataset. The dataset is comprised of tuples $((p_{i},m_{i},d_{i}),o_{i})$,
where, $p_{i}$ is the test prefix, $m_{i}$ is the method under test, and
$d_{i}$ is the docstring (if available) and $o_{i}$ is the ground truth test
oracle, either assertion or exception. The dataset has been partitioned into
three distinct subsets: 80% for training, 10% for testing, and another 10% for
validation.

####  II-A2 Prompt Designing

Even though LLMs are powerful, they are not specifically designed and pre-
trained for test oracle generation. Thus, fine-tuning with effective prompts
is necessary for optimal performance. We have designed six different prompts
that incorporate increasing amounts of context to investigate their
effectiveness in oracle generation.

  * •

Prompt 1 (P1) consists of only the test prefix (‘prefix’). This prompt is
suitable when addition details on the MUT or documents are not available.

  * •

Prompt 2 (P2) additionally includes the method documentation strings (‘prefix
+ [sep] + doc’) . The ‘[sep]’ separator token is different based on different
LLM and their respective tokenizer.

  * •

Prompt 3 (P3) instead of docstrings, includes the method signature (‘prefix +
[sep] + mutsig’). This prompt is suitable when neither the entire method
implementation nor docstring available. Our experimental study in RQ1 shows
that simply adding method signature can significantly improve accuracy.

  * •

Prompt 4 (P4) includes both method signature and docstrings in addition to
prefix (‘prefix + [sep] + doc +[sep] + mutsig’).

  * •

Prompt 5 (P5) includes the code for the entire method under test (‘prefix +
[sep] + mut‘).

  * •

Prompt 6 (P6) adds the docstring to P5 (‘prefix + [sep] + doc + [sep] + mut‘).

####  II-A3 Code LLMs

In our study, we use pre-trained auto-regressive decoder models that are
widely used, available on Hugging Face, and compatible with our GPU resources.

CodeGPT [27]: It is a GPT-style language model with 110M parameters. We choose
to fine-tune this model for two main reasons. Firstly, both our training and
unseen inference datasets came from Java projects, and this model is also pre-
trained on Java programming language. Secondly, this model is pre-trained on
the code completion task, which aligns with our designed prompts. We fine-tune
the microsoft/CodeGPT-small-java model from Hugging Face.

CodeParrot [7]: In our study, we fine-tune the 110M parameters model, which
was pre-trained for the code generation task on nine different languages:
Java, JavaScript, PHP, Python, C#, C++, GO, Ruby, and TypeScript. We have
fine-tuned the codeparrot/codeparrot-small-multi model from Hugging Face. Our
experimental study suggests that despite its small size, the fine-tuned
CodeParrot model generalizes strongly on unseen data and can outperform larger
models with billions of parameters.

CodeGen [31]: CodeGen is a family of autoregressive language models with four
different trainable parameter sizes: 350M, 2B, 6B, 16B. In this paper, we have
fine-tuned 350M and 2B Multi variant models that are pre-trained on dataset
that encompasses six programming languages: C, C++, Go, Java, JavaScript, and
Python. Because of their great capability in comprehending and generating
codes, we fine-tune them for test oracle synthesis task.

PolyCoder [50]: PolyCoder is a family of large language models with three
trainable parameter sizes: .4B, 2.7B, and 16B. This model was trained on 249
GB of code across 12 programming languages. In this paper we have fine-tuned
.4B and 2.7B model for the test oracle generation task.

Phi-1 [36]: Phi-1 is a transformer language model with 1.3 billion parameters,
initially pre-trained on Python code from a variety of data sources for the
task of Python code generation. Despite its primary training on Python, our
study observed that it performs on par with other models when fine-tuned on
Java code for generating test oracles.

###  II-B Unseen Inference

####  II-B1 Unseen Inference Data

To investigate the generalizability of the fine-tuned model, we utilized the
dataset described in [16], comprising 25 large-scale industrial standard Java
projects. Of these, 17 originate from the Apache Commons Proper [38], while
the remainder are sourced from GitHub. These latter projects are characterized
by large codebases with multiple modules, with up to 10k active users per
project, and are frequently employed in software engineering research for the
evaluation of test cases and test oracles [20, 41, 51]. Given their broad
coverage of 21 domains and significant variation in developer metrics, the 25
artifacts selected for this study provide a robust framework for assessing
fine-tuned model’s ability to generalize across a spectrum of real-world
projects. Overall, the dataset consists of 271K source lines of code (SLOC),
331K lines of JavaDoc, 1214K test suite SLOC and 223.5K test cases. Similar to
the SF110 dataset, each input sample of this dataset contains the test prefix,
MUT and docstring.

####  II-B2 TOGLL

Based on the fine-tuning results, we selected the top two performing models
(CodeGen-350M, CodeParrot-110M) based on their accuracy and also chose the
three best-performing prompts (numbers P4, P5, and P6). We then conducted a
small-scale study, performing unseen inference with these two models and three
prompts on a few projects from the unseen dataset discussed in Section II-B1.
Our results suggest that even though CodeGen-350M is three times larger than
CodeParrot-110M, CodeParrot-110M fine-tuned with P6 performs better on the
unseen project than CodeGen on several metrics: total input processed,
compilation error, and false positive count. Furthermore, the inference time
with CodeGen-350M is longer than CodeParrot. Consequently, we define TOGLL as
a fine-tuned CodeParrot-110M operating on P6. We perform rest of the study
with TOGLL on the unseen dataset II-B1.

####  II-B3 Accuracy Computation Through Test Validation

With TOGLL, we generate test oracles for all 25 projects from the unseen
dataset. This dataset framework includes all necessary artifacts for executing
the generated test oracles when integrated with the test prefixes.
Consequently, we integrate the generated oracles into the test cases and
execute the complete test suites. We calculate accuracy as the success rate,
which denotes the percentage of non-empty test oracles that were successfully
passed during test execution, indicating their alignment with expected program
behavior. We compute the success rate using Equation 1

$\textrm{SuccessRate}(A)=\frac{T-(T_{ce}+T_{fp}+T_{em})}{T}$ (1)

where $T$ is the number of total test prefixes in the artifact $A$, $T_{ce}$
is the number of compilation errors, $T_{fp}$ is the number false positive
tests, and $T_{em}$ the number of empty oracles. We assess the success rates
for both assertion and exception oracle predicted by TOGLL and compare these
rates to those of our baseline, TOGA, as discussed in Section II-B4.

####  II-B4 Baseline (TOGA)

TOGA is a neural method for both assertion and exception oracle generation
[10]. TOGA utilizes CodeBERT [11] model as backbone and fine-tuned for the
oracle generation task. TOGA takes the test prefix generated by EvoSuite, its
associated MUT and docstring (if available) and generates either an assertion
oracle or predicts an exception oracle. An example of EvoSuite generated test
cases with assertion and exception oracle is shown in Figure 2. The prefix
part is marked with yellow color. The first test prefix involves pushing the
number 10 onto the stack, followed by popping this value from the stack. For
this scenario, an assertion oracle is anticipated to validate that the value
retrieved is indeed 10. The second test attempts to pop from an empty stack,
expecting the MUT to throw an exception. If no exception is thrown, the test
should fail.

We selected TOGA as our baseline method for several reasons: 1) TOGA
represents the SOTA neural method, having surpassed specification, search, and
neural-based techniques by detecting 57 bugs in Defects4J [20]; 2) TOGA
utilizes the same set of information as P6, which is our most comprehensive
prompt, thereby providing a fair basis for comparison; 3) TOGA has been
extensively evaluated on the same unseen dataset that we also employ in our
study, ensuring consistency in our comparative analysis [16].

Figure 2: EvoSuite-Generated Test Cases with Assertion and Exception Oracles.
The prefix part is marked with yellow color.

###  II-C Mutation Testing

To assess the strength of the generated oracles, we perform mutation testing
that introduces small changes, or mutations, to the source code to create
numerous slightly altered versions of the software, called mutants[8, 35].
Studies showed that there is a statistically significant correlation between
mutant detection and real fault detection [21, 1, 34]. A test suite’s ability
to detect these mutants can be a direct measure of its bug detection
effectiveness [51, 41]. We conduct mutation testing on the cases where the
assertion generated by TOGLL could be executed, meaning the assertions were
neither syntactically incorrect nor false positives. From the unseen dataset,
we consider the same set of test cases with their test oracles to enable
comparison. For each of the 25 unseen artifacts, we perform mutation testing
and record the total number of mutants detected by each approach.
Additionally, we record the total number of unique mutants which indicates the
unique capability of each oracle generation method. Besides comparing with
EvoSuite, we also compare the results with TOGA.

##  III Experimental Study

To assess the efficacy of LLM-generated test oracles, this study investigates
several key areas: effectiveness of different code LLMs in generating test
oracles, the impact of different prompts on the oracle generation accuracy,
the ability of fine-tuned models to generalize to unseen data, comparative
accuracy among different oracle types, diversity of the generated oracle and
the overall efficacy of these oracles in detecting software bugs. To this end,
we answer the following research questions.

RQ1: What LLM and prompting approaches are effective for test oracle
generation? Increasing context in the prompt improves accuracy of oracle
generation up to 79% for a range of LLMs, but smaller LLMs meet or exceed the
performance of larger LLMs when fine tuned.

RQ2: How well does TOGLL, i.e., fine-tuned model generates correct test
oracles for unseen projects? TOGLL generates up to 4.9 times the number of
correct test oracles compared to prior neural based approach.

RQ3: How diverse are LLM-generated assertions relative to those generated by
EvoSuite? TOGLL-generated oracles vary substantially in terms of the specific
assert statements used as well as in the variables and expressions references
in those oracles compared to those generated by EvoSuite.

RQ4: How strong are LLM-generated assertions at identifying unique bugs? TOGLL
generated oracles kill nearly 10 times more unique mutants than prior SOTA
method and fall only 17% short of EvoSuite’s bug detection effectiveness.

TABLE I: Test Oracle Generation Performance of Code LLMs for Different
Prompts. Top two values for each prompt are shown in bold font. The lower row
shows prompt-wise average.

Code LLM |  P1 |  P2 |  P3 |  P4 |  P5 |  P6 |  |  | Prompt Details
---|---|---|---|---|---|---|---|---|---
CodeGPT-110M | 54.4 | 63.5 | 72.7 | 73 | 75 | 74.1 |  |  |  P1: prefix
CodeParrot-110M | 56.4 | 65.6 | 76.1 | 76.2 | 77.7 | 77.7 |  |  |  P2: prefix + [sep] + doc.
CodeGen-350M | 56.7 | 66.3 | 77.4 | 77.5 | 79.3 | 79 |  |  |  P3: prefix + [sep] + mutsig.
PolyCoder-.4B | 56.4 | 64.3 | 74.5 | 74.9 | 76.8 | 76.8 |  |  |  P4: prefix + [sep] + doc. + [sep] + mutsig
Phi-1(1.3B) | 53.42 | 61.5 | 75.5 | 73.4 | 77.6 | 74.3 |  |  |  P5: prefix + [sep] + mut
CodeGen-2B | 55.7 | 65.5 | 75 | 74.7 | 76.5 | 76.8 |  |  |  P6: prefix + [sep] + doc. + [sep] + mut
PolyCoder-2.7B | 55 | 64 | 74.8 | 74.7 | 76.6 | 76.9 |  |  |
Avg: | 55.4 | 64.4 | 75 | 75 | 77 | 76.5 |  |  |

###  III-A RQ1: Selecting LLMs and prompts for oracle generation

In this research question, we fine-tune various large language models, as
outlined in Section II-A3, utilizing six unique prompt formats discussed in
Section II-A2. The purpose of this study is to determine the most effective
combination of model and prompt format for the test oracle generation task.
This approach allows us to understand the impact of prompt engineering on
model efficacy and guides the selection of the most effective model-prompt
pair for a deeper investigation.

####  III-A1 Experimental Setup

For this study, we utilized the SF110 training dataset discussed in Section
II-A1, which includes 159,073 input samples. Each sample contains a unique
identifier, the method under test (MUT), a test prefix, and a docstring, from
which we constructed six prompts as outlined in Section II-A2. For the fine-
tuning process of the code LLMs, we employed a GPU setup consisting of 4 A100
GPUs, each with 40GB of memory. To facilitate parallel processing and
accommodate larger models such as CodeGen-2B and PolyCoder-2.7B within the
40GB memory limit, we utilized PyTorch Accelerator and DeepSpeed [40]. The
models were fine-tuned using a learning rate of 2.5e-5 and the AdamW
optimizer. Each LLM was fine-tuned across all six prompts, and accuracy was
computed on the validation dataset using the ‘exact match’ metric. This metric
may underestimate the overall accuracy, as a generated oracle may not be an
exact match but can still be correct, as illustrated in 4.

####  III-A2 Results

Table I displays the specifics of each prompt—detailing the information used
in each prompt—on the right side. P1 utilizes only the test prefix as input,
prompting the model to generate either an assertion or an exception oracle. P2
adds a docstring in addition to the test prefix, while P3 includes a method
signature. P4 includes both docstring and method signature after the test
prefix. P5 includes the entire MUT code with the test prefix, whereas, P6
contains both the docstring and entire MUT code with the test prefix. On the
left side of Table I, we present the performance of seven fine-tuned code
models in generating test oracles for all six prompt types.

The average accuracy of the fine-tuned models on P1 is around 55.4%. If we add
the available docstring in the prompt, the accuracy is increased by around
10%. Instead of adding the docstring, if we add the method signature to the
prompt, the accuracy is increased by 20%, resulting in on average 75% accuracy
for P3. P4 includes both docstring and method signature with test prefix,
which only see a marginal improvement for few models. Most accuracy
improvement was observed when we add the entire code of the MUT in P5 and P6,
with P5 achieving highest accuracy of 77%. We also observe that adding more
context can also decrease the accuracy, such as for P5 and P6. Overall, we see
a consistent accuracy improvement as we go from prompt P1 to P5.

We observe that including method signature in the prompt works significantly
better than including the docstring. Even when including both method signature
and docstring – P4 – only marginal improvement was observed. This observation
also true for P5 and P6, adding the docstring did not improve much accuracy
when either method signature or entire method code is available. However, from
prompt P1 to P2, accuracy is improved by 10%, indicating the value of
docstrings. The quality of the docstrings is not considered in this study.
This is a promising future direction as we believe that documentation that
combines natural language description of requirements with references to API
elements might yield better results.

Among the seven code models evaluated, CodeGen-350M emerged as the top
performer, closely followed by CodeParrot-110M, both highlighted in bold. In
our RQ2 analysis for unseen inference, we identified these models as leading
candidate LLMs.

RQ1 Findings: LLMs are capable of accuracy levels above 79% in generating test
oracles. Prompts containing increased contextual information enable higher
accuracy, but perhaps surprisingly larger LLMs when fine-tuned do not offer
significant advantages.

###  III-B RQ2 : Assessing test oracles correctness

RQ1 investigates various large language models’ test oracle generation
performance when they are fine-tuned on the training portion of the SF110
dataset and their accuracy is evaluated on the validation portion of the same
dataset. However, a fine-tuned model may not generalize when applied to unseen
data, a common challenge in LLM applicability [17]. To address this concern,
we designed this study to generate test oracles for 25 unseen large-scale Java
projects discussed in Section II-B1. A test oracle is correct if it captures
the expected behavior of a software system. Generated test oracles that do not
compile, that disagree with software behavior, or that are vacuous are
considered incorrect. We study the rate of success in generating correct
oracles for both TOGLL and TOGA.

Figure 3: Comparison of correct test oracle generation performance of TOGLL
vs. TOGA

####  III-B1 Experimental Setup

Test Oracle Generation. We generate test oracles for the unseen dataset with
TOGLL, which is a fine-tuned CodeParrot-110M model operating on P6, which
includes the test prefix, docstrings (if available), and the MUT. The unseen
dataset consist of a total of 202,434 assertion prefixes requiring TOGLL to
generate an assertion oracle for each input sample and 21,123 exception
prefixes requiring TOGLL to generate an exception oracle. We have used a
single A100 GPU with 10GB of memory to perform the inference task. The
inference dataset keeps a record of detailed metadata so that the generated
test oracles can be mapped back to the test class and the test prefix,
enabling test validation to be performed later.

Oracle Correctness Validation. Unlike RQ1, which assessed accuracy using the
‘exact match’ metric, RQ2 involves executing the integrated test
cases—combining the test prefix with the generated test oracles—to collect
detailed metrics for each type of oracle. For both assertion and exception
oracles, we compute success rates across all 25 artifacts using Equation 1,
and compare these rates to those achieved by TOGA.

####  III-B2 Results

In Figure 3, we present the performance of correct test oracle generation by
our LLM-based method, TOGLL, and the deep-learning-based method, TOGA. The
Y-axis displays success rate as a percentage. The X-axis enumerates the names
of 25 artifacts. Each artifact is represented by two sets of vertical bars:
blue for TOGLL and red for TOGA. Solid blue bars show TOGLL’s success rate in
generating assertion oracles, while blue bars with diagonal lines illustrate
its success in generating exception oracles. Similarly, solid red bars
represent TOGA’s assertion oracle success rates, and red bars with diagonal
lines show its performance in generating exception oracles.

Figure 3 shows that TOGLL achieved significantly higher success rate across
all projects for both assertion and exception oracle generation. For example,
for apache commons-number, TOGA could achieve a success rate of only 1%,
whereas, TOGLL achieve significantly higher accuracy of 74.7%. Similarly, for
http-request, TOGA achieved 14% where as TOGLL achieved 83% success rate.
Statistical analysis reveals a significant difference in the average success
rates, with TOGLL achieving a mean of 63% compared to TOGA’s mean of 16.4%. A
t-test yielded a t-statistic of 12 and a p-value of 1.1e-11, indicating a
statistically significant difference in performance between the two methods,
with TOGLL outperforming TOGA in terms of correct test oracle generation.

For exception oracle, the performance different is even higher. TOGA’s average
success rate is 18.86%, while LLM achieves a significantly higher average of
93.4%. A t-test further supports this distinction with a t-statistic of 24.8
and a p-value of 1.3e-18, highlighting TOGLL’s superior performance in
generating exception oracles compared to TOGA.

RQ2 Findings: TOGLL generates significantly more correct test oracles than
TOGA; bettering it by 3.8 times and 4.9 times for assertion oracles and
exception oracles, respectively.

###  III-C RQ3: Investigating diversity in oracle generation

In RQ2, we present the overall performance of TOGLL in generating correct test
oracles and compare it against SOTA deep-learning method. RQ3 delves into
understanding the diversity of the generated test assertions. More
specifically, we investigate whether during unseen inference TOGLL follows the
assertion distribution observed during training and also to what extent the
generated assertions exactly match the ground truth assertions. These metrics
can provide insights into the diversity of the generated assertions. Diversity
is an important aspect of test oracle generation, allowing TOGLL to complement
developer written oracles or those generated by other tools and thereby enable
detection of faults that would otherwise be undetected. In RQ4, we investigate
the ability of TOGLL to detect unique faults.

####  III-C1 Experimental Setup

To set up this experimental study, we utilize the training dataset and results
from RQ2. We divided the assertions into seven categories, consisting of the
most frequently used JUnit assertions investigated in previous studies [51,
10, 16]. We count the total number of assertions in each category and
calculate the distribution relative to the total number of assertions in the
training data. We follow a similar process to compute the categorical
distribution of TOGLL generated assertions during unseen inference.
Furthermore, we compute the percentage of TOGLL generated assertions that are
an exact match for ground truth assertions in our dataset.

####  III-C2 Results

Table II, Column 1 lists the assertion categories, Column 2 shows the
distribution of different types of assertions within the Training dataset.
Column 3 presents the breakdown of the oracles generated by TOGLL for the
unseen dataset. Column 4 presents the percentage of exact match assertions for
each category, reflecting the instances where the TOGLL-generated assertions
exactly match the ground truth assertions.

There is a clear evidence that the distribution of oracles generated by TOGLL
is quite different from the training data. The training data has a sharp peak
for assertEquals, whereas the inference data has lower peaks for two
categories. Moreover the tails of the distributions have a much higher share
of samples indicating that generation of assertions is much less concentrated
than training data. With regard to assertion category TOGLL is more diverse in
generating oracle assertions.

We see that out of the 194k generated assertions, only 18k are exact matches,
indicating that the generated assertion are diverse relative to provided
assertions in the dataset. We observe this diversity can come multiple ways.
For example, the generated assertion may come from a different category, like
the first and third examples in Figure 4, or may vary the assertion arguments,
like the second example in Figure 4. We observe that ground truth oracles do
not check all variables within the prefix which gives TOGLL an opportunity to
generate diverse assertion oracle. We see this in the first example where the
TOGLL generated assertion is requiring that sort yield the same array, whereas
the EvoSuite assertions just checks the length of the array. Similarly the
second example shows a TOGLL generated assertion for the q1 variable, whereas
EvoSuite’s assertion targets the q2 variable. For the third example, TOGLL
generates a different type of oracle. Not only are these TOGLL generated
assertions diverse, but they potential add bug detection power which we
explore in the next RQ.

TABLE II: Distribution of Assertion Oracles Across Different Categories

Assertion Category | Assertion Distribution
---|---
Training |  Inference |  Exact Match
assertNotNull |  18,033 (16.1%) |  79,476 (40.8%) |  9203 (11.6%)
assertEquals |  81,766 (72.8%) |  79,680 (40.9%) |  7217 (9%)
assertNull |  10,484 (9.3%) |  6,542 (3.4%) |  1660 (25.3%)
assertSame |  636 (0.6%) |  5,791 (3%) |  406 (7%)
assertFalse |  651 (0.6%) |  12,575 (6.5%) |  43 (0.3%)
assertNotSame |  334 (0.3%) |  3,210 (1.6%) |  89 (2.8%)
assertTrue |  349 (0.3%) |  3,268 (1.7%) |  12 (0.37%)
syntx. incorrect | - |  4,329 (2.2%) | -
Total: | 112,253 | 194,871 | 18,630

Figure 4: Diverse yet correct test oracles generated by TOGLL

RQ3 Finding: TOGLL can generate assertions that are diverse with respect to
both the assertion statement used and the variables and expressions targeted
for observation in those assertions.

###  III-D RQ4 : Assessing test oracle strength

TABLE III: Bug-Detection Performance of TOGLL vs. EvoSuite Generated
Assertions.

Artifact | Bug Detected by
---|---
EvoSuite Assertion |  EvoSuite Unique |  TOGLL Assertion |  TOGLL Unique
http | 64 | 38 | 29 | 3
json | 328 | 110 | 249 | 31
beanutils | 551 | 46 | 516 | 11
collections4 | 248 | 104 | 162 | 18
dbutils | 108 | 14 | 98 | 4
jsoup | 598 | 177 | 509 | 88
imaging | 1,869 | 629 | 1,315 | 75
lang3 | 2,301 | 397 | 2,063 | 159
configuration | 271 | 51 | 266 | 46
jexl3 | 697 | 73 | 638 | 14
joda-time | 1,545 | 446 | 1,323 | 224
net | 747 | 175 | 592 | 20
pool2 | 155 | 8 | 150 | 3
spark | 407 | 63 | 355 | 11
validator | 602 | 52 | 560 | 10
scribejava | 172 | 31 | 143 | 2
bcel | 1,122 | 467 | 698 | 43
numbers | 871 | 168 | 726 | 23
springside4 | 1,160 | 243 | 938 | 21
vfs2 | 339 | 37 | 310 | 8
rng | 554 | 158 | 527 | 131
jcs3 | 621 | 81 | 548 | 8
async-http | 69 | 10 | 59 | 0
weaver | 20 | 2 | 18 | 0
Total: | 15,985 | 3,690 | 13,318 | 1,023
|  | TOGA: | 6,893 | 105

Correct test oracles are essential for aligning with program requirements, yet
correctness alone may not suffice to detect diverse bugs. The strength of test
oracles is equally crucial. In RQ4, we explore the strength of generated test
assertions and comparing TOGLL to the SOTA EvoSuite and TOGA.

####  III-D1 Experimental Setup

To set up this experiment, we generated three distinct test suites for each of
the 25 unseen artifacts: TTOGLL, TES, TIO. TTOGLL comprises test cases with
TOGLL-generated assertions, excluding all test cases with non-compiling and
incorrect (i.e., false positive) assertions produced by TOGLL. We recorded the
set of final test cases, including the test class and the unique test
identifier. For constructing TES, we use the same set of tests, but the test
assertions were generated by EvoSuite. For TIO, the same set of test cases was
retained; however, this time, the test cases did not contain any test
assertions. The three test suites share identical counts of test cases, test
case prefixes, and assertion totals. The unique aspect distinguishing them is
the method used for generating assertions.

We utilize mutation testing to introduce a wide array of bugs into the source
code of 25 artifacts. Mutation testing is a widely used method for evaluating
the effectiveness of test suites in detecting bugs. Unlike bug benchmarks such
as Defects4J [20] or QuixBugs [26], which are limited in both the number and
diversity of faults, mutation testing enables thousands of diverse faults to
be injected. Recent studies have also pointed out that Defects4J might inflate
the perceived bug detection capabilities of test suites, further supporting
the preference for mutation testing [16]. We employ PIT [8], a state-of-the-
art mutation testing tool for Java programs. PIT offers several advantages
over other tools, as it generates meaningful mutants and fewer equivalent
mutants [22, 23, 39]. We have used PIT-1.9.8 to generate a total of 69,793
strong mutants. For each mutated artifact, we run all three test suites to
detect mutants. Mutants detected by TIO are those that can be detected through
implicit assertions, meaning test prefixes alone suffice for their detection
without the need for explicit assertions. Computing TIO allows our study to
follow the recommendations of [16] by removing mutants killed by implicit
oracles from study results. We calculate the total number of mutants detected
by EvoSuite and TOGLL assertions as well as the unique mutants killed by
oracles generated by each technique. Additionally, we also provide the total
and unique mutants killed by TOGA.

####  III-D2 Results

In Table III, the first Column shows the names of the 25 artifacts. The second
and third Columns present the total and unique mutants detected by EvoSuite-
generated assertions, respectively. The fourth and fifth columns present the
total and unique mutants detected by TOGLL assertions, respectively. Row 27
shows the total mutants detected across all artifacts by each method, whereas
row 28 presents the performance of TOGA, in detecting total and unique
mutants.

Out of the 69,793 mutants generated and covered by test cases, EvoSuite
detected a total of 15,985 mutants, including 3,690 unique mutants not
detected by TOGLL assertions. Conversely, TOGLL assertions identified 13,318
mutants, with 1,023 being unique mutants.

For all 25 artifacts, test cases were generated by EvoSuite, as discussed in
Section II-B1. EvoSuite, a search-based dynamic approach, generates assertions
based on executed program behavior and actual result values. Thus, it is
expected to produce strong assertions capable of detecting a large number of
mutants. TOGLL, in contrast, is a static approach that utilizes the test
prefixes generated by EvoSuite, but lacks knowledge of any values computed
during test execution. Despite this disadvantage, TOGLL-generated assertions
detected 83% of the EvoSuite-detected mutants. This significantly outperformed
TOGA, which detected only 43% of the EvoSuite-detected mutants under the same
conditions. A t-test yielded a t-statistic of 4.12 and a p-value of 0.0003,
indicating a statistically significant difference in performance between the
TOGLL and TOGA in terms of total bug detection.

Figure 5: Examples illustrating that TOGLL generated assertions can detect
unique mutants that EvoSuite generated assertions (ES) can not.

TOGLL generated assertions were able detect 1,023 mutants that EvoSuite
missed, using the same set of input prefixes. This is a significant
improvement over TOGA, which only identified 105 unique mutants. In total,
TOGLL killed nearly twice the number of mutants as TOGA, but when viewed
relative to TOGA as a baseline TOGLL killed 9.7 times the number of unique
mutants. This demonstrates that LLM-based techniques can generate a
significant numbers of strong test assertions.

We performed a qualitative study of the TOGLL assertions that could kill
unique mutants. This study was opportunistic in that we look into the 1023
unique mutants, their corresponding mutated MUTs from smallest to largest and
stopped when we found a handful of examples that illustrated oracle diversity.
Figure 5 shows two examples from the joda-time and Apache commons-angle
package. In the first example, test327 calls the calculatePrintedLength method
which was mutated by ‘replacing long division with multiplication’, TOGLL
checks the return variable int0 and was able to kill the mutant. EvoSuite
checks a value that is independent of the execution of the MUT and has no
chance to kill the mutant. The second example, TOGLL check that angle_Rad1
should not be null and thus was able to detect the mutant which replaced the
return value with null. Here again the EvoSuite generated assertion checks a
value that is independent of the MUT. While anecdotal these examples
illustrate how TOGLL leverages the EvoSuite generated test prefix and produces
valuable test oracles that EvoSuite can not. We plan to conduct a more
systematic study of the qualitative characteristics of both test oracles that
are unique to EvoSuite and unique to TOGLL to help understand their strengths
and guide technique refinement.

RQ4 Finding: TOGLL generated test oracles kill 17% fewer mutants than EvoSuite
tests, despite having less information about test behavior. Relative to TOGA,
TOGLL yields nearly a 10-fold increase in mutant kills - thereby establishing
a new SOTA in neural oracle generation.

###  III-E Threats to Validity

In our large-scale study, we have used open-source and widely recognized
dataset SF110 [13]. Yet, our results may not generalize across other datasets.
To address this threat and ensure generalizability, we further evaluated the
performance of our fine-tuned model on the unseen dataset from [16].

In our comprehensive study, we created several tools and scripts which may
contain bugs. We used publicly available libraries to mitigate the risk and
conducted extensive validity tests and repeated our experiments to ensure
consistent results.

In our investigation, during fine-tuning we utilized the ‘exact match’ metric
to compute oracle generation accuracy. This is a widely used metric by the
research community. For unseen inference, we relied on test validation. This
methodology, we assert, offers a reliable accuracy metric. Additionally, for
measuring test oracle strength, we used mutant kill score, a well established
metric used in many research studies [51, 16, 15].

##  IV Limitation And Future Direction

In this work, we have demonstrated the potential of LLMs in generating
correct, diverse and strong test oracles that can effectively detect large
number of unique bugs. However, there are challenges which require further
research efforts.

###  IV-A Compilation Error

Our in-depth experimental results show that TOGLL generates assertions that
are non-compiling about 5% of the time. Among them we have seen assertions
that are ‘garbage’ strings and one’s that have minor syntax or type errors. In
future work, we want to fine-tune LLMs to generate syntax and type correct
assertion using grammar and type based constraints similar to [10].
Additionally, incorporating compilation errors into a refinement prompt to
improve the the generated oracle is another strategy worth pursuing [9].

###  IV-B Token Length Limits

We have seen that LLMs can not process input samples that exceeds the maximum
token length capacity of the LLMs. In our study, a total of 3% of the samples
exceeded the threshold of 600 tokens when using prompt 6 that includes MUT,
docstring and prefix. Rather than increasing the token threshold, our study
suggests that in such cases we could shift to a more concise prompt, such as
prompt 3 which replaces the MUT with the method signature without
significantly compromising performance.

###  IV-C False Positives

One important and long standing issue with all type of learning based oracle
generation method is that they can generate false positives, i.e., oracles
that fail on a correct program. TOGLL exhibited a 7% false positive rate for
exception oracles and a 25% rate for assertion oracles which is a significant
improvement over the respective rates for TOGA – 81% and 47%. We have studied
a sample of false positive assertions and observed that many of them include
numeric literals which is the source of the false positive. Based on these
observations, we conjecture that assertions relating program variables, or
expressions involving variables, may be less susceptible to false positives.
In future research, we explore the benefits of using high quality
documentation, which may capture such relations, and develop prompting
strategies that encourage the generation of assertions that are not as
dependent on numeric literals.

##  V Related Work

EvoSuite [12] is a state of the practice automated test oracle generation that
uses a search-based method. EvoSuite generates oracles by observing test
execution and capturing values that it includes in assertions. This creates
regression oracles that are valuable for detecting undesirable changes in
program behavior. Importantly for our study, by construction these oracles
capture the behavior of the system under test making them a useful baseline
for judging correct test oracles.

Randoop [32] is feedback-directed random testing for Java. Research has
explored the generation of both assertion [5] and exception oracles [14] for
Randoop tests using NLP approaches. Subsequent NLP-based methods, such as
ATLAS [49] and AthenaTest [47], have been shown to outperform these methods
and more recently TOGA [10] established a new state of the art by a large
margin. This is why we use TOGA as baseline in our study.

In the past year, techniques for test generation using LLMs have emerged. For
example, TESTPILOT [42], an LLM-based test generation tool for JavaScript that
automatically generates unit tests for JavaScript. TESTPILOT achieved 70%
statement coverage and 52.8% branch coverage, which outperformed the state-of-
the feedback-directed JavaScript test generation technique, Nessie [2]. A
recent study [44] that utilized LLMa for generating test for the SF110 large-
scale real world Java projects showed that LLMa could only achieve 2%
coverage, whereas EvoSuite can achieve more than 90% coverage. This suggests
that LLMs are not very effective in generating test for real world Java
program, which is why in our study instead of focusing on both test input and
test oracle generation, we focus on a single task test oracle generation with
LLM.

##  VI Conclusion

We conduct the first large-scale investigation of the ability of large
language models to automatically generating correct and strong test oracles.
To this end, we fine-tune seven code LLMs that are pre-trained to perform code
related tasks. We fine-tune CodeGPT-110M, CodeParrot-110M, CodeGen-350M,
PolyCoder-.4B, Phi-1(1.3B), CodeGen-2B and PolyCoder-2.7B on six different
prompts with varying level of information. Our findings reveal that effective
prompt design can improve the accuracy of the models significantly and that
with effective prompts smaller size model can also perform equally or even
better than larger models. As fine-tuning and inference with large models is
environmentally and financially costly [4], small models are most preferred.

With the best performing model and prompt we develop TOGLL, our LLM-based
automated test oracle generation method. We extensively evaluate TOGLL through
large-scale study to investigate its generalizability and capability to
generate correct, diverse and strong test oracles. Our experimental study
answers critical research questions, demonstrating that TOGLL can generate
3.8x more correct assertion and 4.9x more exception oracles than the previous
SOTA neural method, TOGA. Not only does TOGLL significantly reduce the
generation of false positive oracles, but it generates oracles 97% of the time
in comparison to TOGA which only generates oracles 38% of the time. Generating
correct oracle does not necessarily mean that the generated assertions are
diverse and strong enough to detect faults. Our results indicate that TOGLL is
capable of generating diverse test oracle where only 9.5% are an exact match
with the unseen dataset. Furthermore, TOGLL’s diverse assertions can detect
1,023 unique bugs that EvoSuite can not detect with it own generated
assertions, which is 10x improvement over prior ML oracle generation
techniques. TOGLL establishes a new state-of-the-art for ML-based oracle
generation.

We describe remaining challenges in using LLMs for test oracle generation and
discuss several actionable future directions to mitigate these limitation. We
believe our large-scale study and in-depth investigation lay down the
groundwork for further advancements in automated test oracle generation.

## References

  * [1] J. Andrews, L. Briand, and Y. Labiche.  Is mutation an appropriate tool for testing experiments? [software testing].  In Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005., pages 402–411, 2005.
  * [2] E. Arteca, S. Harner, M. Pradel, and F. Tip.  Nessie: Automatically testing javascript apis with asynchronous callbacks.  In Proceedings of the 44th International Conference on Software Engineering, pages 1494–1505, 2022.
  * [3] E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo.  The oracle problem in software testing: A survey.  IEEE Transactions on Software Engineering, 41(5):507–525, 2015.
  * [4] E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell.  On the dangers of stochastic parrots: Can language models be too big?  In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623, 2021.
  * [5] A. Blasi, A. Goffi, K. Kuznetsov, A. Gorla, M. D. Ernst, M. Pezzè, and S. D. Castellanos.  Translating code comments to procedure specifications.  In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2018, page 242–253, New York, NY, USA, 2018. Association for Computing Machinery.
  * [6] A. Blasi, A. Gorla, M. D. Ernst, M. Pezzè, and A. Carzaniga.  Memo: Automatically identifying metamorphic relations in javadoc comments for test automation.  Journal of Systems and Software, 181:111041, 2021.
  * [7] CodeParrot.  https://huggingface.co/codeparrot/codeparrot-small-multi.
  * [8] H. Coles, T. Laurent, C. Henard, M. Papadakis, and A. Ventresque.  Pit: a practical mutation testing tool for java.  In Proceedings of the 25th international symposium on software testing and analysis, pages 449–452, 2016.
  * [9] P. Deligiannis, A. Lal, N. Mehrotra, and A. Rastogi.  Fixing rust compilation errors using llms.  arXiv preprint arXiv:2308.05177, 2023.
  * [10] E. Dinella, G. Ryan, T. Mytkowicz, and S. K. Lahiri.  Toga: A neural method for test oracle generation.  In Proceedings of the 44th International Conference on Software Engineering, ICSE ’22, page 2130–2141, New York, NY, USA, 2022. Association for Computing Machinery.
  * [11] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, et al.  Codebert: A pre-trained model for programming and natural languages.  arXiv preprint arXiv:2002.08155, 2020.
  * [12] G. Fraser and A. Arcuri.  Evosuite: automatic test suite generation for object-oriented software.  In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering, pages 416–419, 2011.
  * [13] G. Fraser and A. Arcuri.  A large-scale evaluation of automated unit test generation using evosuite.  ACM Trans. Softw. Eng. Methodol., 24(2), dec 2014.
  * [14] A. Goffi, A. Gorla, M. D. Ernst, and M. Pezzè.  Automatic generation of oracles for exceptional behaviors.  In Proceedings of the 25th International Symposium on Software Testing and Analysis, ISSTA 2016, page 213–224, New York, NY, USA, 2016. Association for Computing Machinery.
  * [15] S. B. Hossain, M. B. Dwyer, S. Elbaum, and A. Nguyen-Tuong.  Measuring and mitigating gaps in structural testing.  In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1712–1723. IEEE, 2023.
  * [16] S. B. Hossain, A. Filieri, M. B. Dwyer, S. Elbaum, and W. Visser.  Neural-based test oracle generation: A large-scale evaluation and lessons learned.  In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, page 120–132, NY, USA, 2023. Association for Computing Machinery.
  * [17] X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang.  Large language models for software engineering: A systematic literature review, 2024.
  * [18] G. Jahangirova, D. Clark, M. Harman, and P. Tonella.  Test oracle assessment and improvement.  In Proceedings of the 25th International Symposium on Software Testing and Analysis, pages 247–258, 2016.
  * [19] M. Jin, S. Shahriar, M. Tufano, X. Shi, S. Lu, N. Sundaresan, and A. Svyatkovskiy.  Inferfix: End-to-end program repair with llms.  In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 1646–1656, 2023.
  * [20] R. Just, D. Jalali, and M. D. Ernst.  Defects4j: A database of existing faults to enable controlled testing studies for java programs.  In Proceedings of the 2014 International Symposium on Software Testing and Analysis, pages 437–440, 2014.
  * [21] R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser.  Are mutants a valid substitute for real faults in software testing?  In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2014, page 654–665, New York, NY, USA, 2014. Association for Computing Machinery.
  * [22] M. Kintis, M. Papadakis, A. Papadopoulos, E. Valvis, N. Malevris, and Y. Le Traon.  How effective are mutation testing tools? an empirical analysis of java mutation testing tools with manual analysis and real faults.  Empirical Software Engineering, 23(4):2426–2463, 2018.
  * [23] T. Laurent, M. Papadakis, M. Kintis, C. Henard, Y. L. Traon, and A. Ventresque.  Assessing and improving the mutation testing practice of pit.  In 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST), pages 430–435, 2017.
  * [24] C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen.  Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models.  In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 919–931. IEEE, 2023.
  * [25] Z. Li, C. Wang, Z. Liu, H. Wang, D. Chen, S. Wang, and C. Gao.  Cctest: Testing and repairing code completion systems.  In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1238–1250. IEEE, 2023.
  * [26] D. Lin, J. Koppel, A. Chen, and A. Solar-Lezama.  Quixbugs: a multi-lingual program repair benchmark set based on the quixey challenge.  In Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity, SPLASH Companion 2017, page 55–56, New York, NY, USA, 2017. Association for Computing Machinery.
  * [27] S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, et al.  Codexglue: A machine learning benchmark dataset for code understanding and generation.  arXiv preprint arXiv:2102.04664, 2021.
  * [28] G. J. Myers, C. Sandler, and T. Badgett.  The art of software testing.  John Wiley & Sons, 2011.
  * [29] D. Nam, A. Macvean, V. Hellendoorn, B. Vasilescu, and B. Myers.  Using an llm to help with code understanding.  In 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE), pages 881–881. IEEE Computer Society, 2024.
  * [30] H. News.  Twitter outage report, 2016.  https://news.ycombinator.com/item?id=8810157.
  * [31] E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong.  Codegen: An open large language model for code with multi-turn program synthesis, 2023.
  * [32] C. Pacheco and M. D. Ernst.  Randoop: feedback-directed random testing for java.  In Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion, pages 815–816, 2007.
  * [33] R. Pandita, X. Xiao, H. Zhong, T. Xie, S. Oney, and A. Paradkar.  Inferring method specifications from natural language api descriptions.  In 2012 34th International Conference on Software Engineering (ICSE), pages 815–825, 2012.
  * [34] G. Petrović, M. Ivanković, G. Fraser, and R. Just.  Does mutation testing improve testing practices?  In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pages 910–921. IEEE, 2021.
  * [35] G. Petrovic, M. Ivankovic, G. Fraser, and R. Just.  Practical mutation testing at scale: A view from google.  IEEE Transactions on Software Engineering, 2021.
  * [36] Phi-1.  https://huggingface.co/microsoft/phi-1.
  * [37] A. M. Porrello.  Death and denial: The failure of the therac-25, a medical linear accelerator.  Death and Denial: The Failure of the THERAC-25, AMedical Linear Accelerator, 2012.
  * [38] A. C. Proper.  Apache commons proper – a repository of reusable java components, 2022.  https://commons.apache.org/components.html, Last accessed on 2022-10-11.
  * [39] S. Rani, B. Suri, and S. K. Khatri.  Experimental comparison of automated mutation testing tools for java.  In 2015 4th International Conference on Reliability, Infocom Technologies and Optimization (ICRITO) (Trends and Future Directions), pages 1–6, 2015.
  * [40] J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He.  Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506, 2020.
  * [41] D. Schuler and A. Zeller.  Checked coverage: an indicator for oracle quality.  Software testing, verification and reliability, 23(7):531–551, 2013.
  * [42] M. Schäfer, S. Nadi, A. Eghbali, and F. Tip.  An empirical evaluation of using large language models for automated unit test generation.  IEEE Transactions on Software Engineering, 50(1):85–105, 2024.
  * [43] M. L. Siddiq, J. Santos, R. H. Tanvir, N. Ulfat, F. A. Rifat, and V. C. Lopes.  Exploring the effectiveness of large language models in generating unit tests.  arXiv preprint arXiv:2305.00418, 2023.
  * [44] M. L. Siddiq, J. Santos, R. H. Tanvir, N. Ulfat, F. A. Rifat, and V. C. Lopes.  Exploring the effectiveness of large language models in generating unit tests.  arXiv preprint arXiv:2305.00418, 2023.
  * [45] Synopsys Editorial Team.  Coverity report on the ‘goto fail’ bug.  Blog post, Synopsys, Mountain View, CA, Feb. 25, 2014; http://security.coverity.com/blog/2014/Feb/a-quick-post-on-apple-security-55471-aka-goto-fail.html.
  * [46] S. H. Tan, D. Marinov, L. Tan, and G. T. Leavens.  @tcomment: Testing javadoc comments to detect comment-code inconsistencies.  In 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation, pages 260–269, 2012.
  * [47] M. Tufano, D. Drain, A. Svyatkovskiy, S. K. Deng, and N. Sundaresan.  Unit test case generation with transformers and focal context.  arXiv preprint arXiv:2009.05617, 2020.
  * [48] M. Tufano, D. Drain, A. Svyatkovskiy, and N. Sundaresan.  Generating accurate assert statements for unit test cases using pretrained transformers.  In Proceedings of the 3rd ACM/IEEE International Conference on Automation of Software Test, pages 54–64, 2022.
  * [49] C. Watson, M. Tufano, K. Moran, G. Bavota, and D. Poshyvanyk.  On learning meaningful assert statements for unit test cases.  In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pages 1398–1409, 2020.
  * [50] F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn.  A systematic evaluation of large language models of code, 2022.
  * [51] Y. Zhang and A. Mesbah.  Assertions are strongly correlated with test suite effectiveness.  In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, pages 214–224, 2015.