# FuzzAug: Exploring Fuzzing as Data Augmentation for Neural Test Generation

Yifeng He<EMAIL_ADDRESS>0000-0002-5389-7128 University of California,
DavisUSA ,  Jicheng Wang<EMAIL_ADDRESS>University of California,
DavisUSA ,  Yuyang Rong<EMAIL_ADDRESS>University of California,
DavisUSA and  Hao Chen<EMAIL_ADDRESS>University of California, DavisUSA

(2018; 20 February 2007; 12 March 2009; 5 June 2009)

###### Abstract.

Testing is an essential part of modern software engineering to build reliable
programs. As testing the software is important but expensive, automatic test
case generation methods have become popular in software development. Unlike
traditional search-based coverage-guided test generation like fuzzing, neural
test generation backed by large language models can write tests that are
semantically meaningful and can be understood by other maintainers. However,
compared to regular code corpus, unit tests in the datasets are limited in
amount and diversity. In this paper, we present a novel data augmentation
technique FuzzAug, that combines the advantages of fuzzing and large language
models. FuzzAug not only keeps valid program semantics in the augmented data,
but also provides more diverse inputs to the function under test, helping the
model to associate correct inputs embedded with the function’s dynamic
behaviors with the function under test. We evaluate FuzzAug’s benefits by
using it on a neural test generation dataset to train state-of-the-art code
generation models. By augmenting the training set, our model generates test
cases with $11\%$ accuracy increases. Models trained with FuzzAug generate
unit test functions with double the branch coverage compared to those without
it. FuzzAug can be used across various datasets to train advanced code
generation models, enhancing their utility in automated software testing. Our
work shows the benefits of using dynamic analysis results to enhance neural
test generation. Code and data will be publicly available.

Software Testing, Data Augmentation, Fuzzing, Large Language Models, Unit
Testing, Test Generation

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††conference:
the 39th IEEE/ACM International Conference on Automated Software Engineering;
27 October - 1 November 2024; Sacramento, California, United States††isbn:
978-1-4503-XXXX-X/18/06††ccs: Software and its engineering Software testing
and debugging††ccs: Computing methodologies Natural language processing††ccs:
Computing methodologies Learning latent representations††ccs: Security and
privacy Software security engineering

Figure 1. An illustration of previous training approaches to neural test
generation with language models and where our data augmentation method
(FuzzAug) integrates into the training pipeline. Previous work collected code
and unit tests directly from open-source repositories, where the quantity of
unit test data is insufficient. Our proposed data augmentation method utilizes
fuzzing to generate more test data that have valid syntax and semantics as
regular unit test data. The model trained with FuzzAug archives higher test
case accuracy and branch coverage, which makes the generated unit tests more
practical in real software engineering and testing.

Process of FuzzAug.

##  1\. Introduction

Testing is one of the important processes in software engineering, ensuring
the quality and health of large software applications. A unit test is a self-
assessment test written and executed by the developer that demonstrates the
components in the software satisfies the requirements set in the design
specification (Runeson, 2006). However, despite its importance, developers do
not always contribute new tests due to the difficulty of identifying which
code to test, isolating them as units, and finding relevant inputs (Daka and
Fraser, 2014). Automatic unit test generation (Fraser and Arcuri, 2011) is a
workaround to the first two issues by treading all functions as units under
test (focal function), but the resulting tests are unsatisfactory in terms of
readability and correctness of relevant input-output pairs (Panichella et al.,
2020). As a result, automatic randomized testing has diverged into fuzz
testing (Fioraldi et al., 2020; Serebryany, 2016) that ignores readability and
focuses on finding new program behaviors via coverage monitoring, and
property-based testing (Claessen and Hughes, 2000) that lets developers write
formal specifications and only generate random inputs. However, finding
semantic meaningful test cases correctly and effectively is still a remaining
problem.

More recently, people have attempted to ease these issues by leveraging the
power of machine learning models (Nie et al., 2023; Rao et al., 2023; He et
al., 2024). Large language models (LLMs) trained on large code corpus can
write meaningful programs given text descriptions (Chen et al., 2021; Li et
al., 2022; Rozière et al., 2023; Li et al., 2023; Allal et al., 2023).
Therefore, with sufficiently large code and test corpus, LLMs should generate
high-quality unit tests to assist human software engineers. However, testing
code typically occupies a minor fraction of a software repository, compared to
functional code. CAT-LM (Rao et al., 2023) finds that in popular Python and
Java repositories, test code files comprise less than 20% of all code files.
This shortcoming in training data limits LLM’s ability to write usable tests
in production for two reasons: imbalanced training data makes the model not
attend to details in units under test, and the insufficient amount of testing
code leads to a disadvantage in learning the representations of unit tests.
Code-test alignment is an effective approach to solve the first issue, where
CAT-LM introduced file-level alignment and UniTSyn (He et al., 2024) improved
the alignment to more fine-grained function level. UniTSyn also tackles the
second challenge by designing an extendable test collection framework to
gather testing code from repositories in various languages to enhance the
diversity and quantity of test cases. Although this approach can build large-
scale datasets for test generation, the size of unit test functions in the
dataset is still limited since they have to be created by human developers.

A more promising approach to achieve the maximum likelihood of testing corpus
is data augmentation (DA) (Rubin, 1987; Rebuffi et al., 2021). In computer
vision (CV), data augmentation is usually done by applying randomized
geometric or color _transformations_ on the images, or injecting _random
noise_ to the images. Compared to CV, data augmentation is harder to apply to
language processing (Shorten et al., 2021). Common augmentation techniques for
natural language data include deleting, misspelling, or randomly replacing
words. These techniques cannot be applied in program understating and
generation tasks (and hence, test generation) since programming languages
(PLs) are defined in formal grammar with strict semantics. Therefore, we find
that a valid DA method for test generation should contain semantically
meaningful PL templates with randomized but valid testing inputs for the focal
functions.

To this end, we propose FuzzAug as a direct yet efficient data augmentation
technique using fuzzing data to enhance test generation LLM. Fuzzing is a
robust dynamic program analysis method to discover vulnerabilities in software
by randomly generating inputs that trigger new paths in the software. These
inputs and their corresponding outputs embed the program’s runtime behaviors
and thus can improve language in code understanding (Zhao et al., 2023; Huang
et al., 2023). To formulate fuzzing data as augmentation for training data, we
apply _code transformation_ to fuzz targets in libFuzzer (Serebryany, 2016) to
generate new unit testing functions. FuzzAug not only eases the limitation on
the amount of testing code corpus in the training data, but also provides more
correct and diverse relevant inputs to the focal functions. LLM-based test
generation trained with FuzzAug addresses the aforementioned issues that stop
developers from writing unit tests by automatically generating test functions
for any possible focal functions with inputs whose correctness is verified by
code execution. Therefore, FuzzAug is a novel training approach to build
practical artificial intelligence (AI) unit test assistance for improving
software robustness.

To explore the effectiveness of FuzzAug, we further trained two different test
generation models. We use UniTSyn (He et al., 2024) to train the baseline
UnitCoder, and augmented UniTSyn using FuzzAug to train FuzzCoder for
comparison. We conduct our experiments based on the state-of-the-art code
generation models SantaCoder (Allal et al., 2023) to build FuzzCoder. Both
models trained with FuzzAug achieved non-negligible performance advantages in
generating accurate test cases (assertions) and complete test functions with
high code coverage. These results demonstrate the viability of our FuzzAug to
overcome the limitation on testing code datasets.

In summary, our contributions are

  1. (1)

We propose a novel data augmentation method (FuzzAug) designed for neural test
generation models. This data augmentation method solves the problem of
insufficient testing code in repositories and the lack of precise and diverse
inputs to the function under test. To the best of our knowledge, FuzzAug is
the first DA method for neural test generation.

  2. (2)

We build and release a new Rust function dataset that contains functional-
level code-test pairs to train a test generation model for Rust programs. We
also apply FuzzAug on this dataset and release the resulting augmented
dataset.

  3. (3)

We validated the quality of FuzzAug by training autoregressive language models
on a dataset augmented by it. The improvement of model performance after
adding FuzzAug demonstrates the necessity and benefits of adding fuzzing-
augmented testing functions to the training corpus.

##  2\. Design of FuzzAug

Figure 2. An illustration of data augmentation for unit test dataset based on
fuzzing. The original pre-defined fuzz target is human-written and provided in
the repositories. We instrument the fuzz target to (1) collect fuzzing input
data, and transform it into (2) to align the data representations of fuzzing
data with unit tests.

Process of FuzzAug.

###  2.1. Challenges

Generating meaningful test functions as training data for neural test
generation models is a complex and critical challenge. To introduce randomly
modified data for training test generation models, a data augmentation method
must satisfy the following requirements:

  1. (1)

The randomly generated data must be meaningful and valid to the software
testing context.

  2. (2)

The augmentation modification must keep the syntax and semantic integrity of
test functions.

First, the randomly generated data must be inherently meaningful to software
testing. This means that the data should not merely be random or arbitrary.
For example in data augmentation for image classification, randomly rotating
an image of a cat or injecting random noise into it does not change the fact
that it is a cat. Similarly, although there will be no label, data
augmentation for test generations requires the randomly generated data to be
meaningful for testing the program. The augmented data should help the model
learn to handle diverse and complex testing situations, thus improving its
ability to generate useful and effective test cases.

Second, the modifications must rigorously preserve the program’s syntax and
testing semantics. Since we are training a language model to generate code,
the tokens in training data must follow the exact syntax of the language for
the test function to be valid. Furthermore, we require this data augmentation
method to generate code with testing semantics. This is because regular
functioning code and testing code have purpose, and thus have different
distribution of embedding. We want the model to generate better test cases and
unit test functions, so it is necessary to overcome this challenge to keep the
semantic integrity of test functions.

Therefore, designing data augmentation to train test generation models
involves creating a sophisticated balance. On one hand, introducing sufficient
variability to train the models under diverse conditions is essential to
generate high-quantity test cases. On the other hand, maintaining the syntax
and semantics integrity of augmented test functions is crucial to ensure the
validity of training data. This makes the development of FuzzAug not only
challenging but also vital for advancing the capabilities of neural test
generation with language models.

###  2.2. Fuzzing for Random Input

The first requirement ensures that the randomly generated data is beneficial
to model training. High-quality test cases are expected to reflect the
behavior of the programs, which is hard to achieve by data augmentation for
natural language data. In other words, in order to improve the model’s ability
to generate useful test cases, the data augmentation method needs to be aware
of the program’s structure and behavior.

Figure 3. Fuzzing loop for dynamic program testing. This loop shows the
process of the collection of randomized generated data for augmentation.

Process of fuzzing.

Fuzzing is a widely used software testing method, which can generate or mutate
inputs randomly and keep the inputs that explore more program paths by
behavior monitor (Fioraldi et al., 2020). Coverage-guided fuzzing can be
summarized as a four-stage loop as shown in Figure 3. Since fuzzers select
input seeds by executing the programs, these inputs embed the program’s
dynamic behavior and are thus able to discover bugs and vulnerabilities in the
program. Previous studies (Zhao et al., 2023; Huang et al., 2023) show that
fuzzing input-output pairs are helpful for language models to understand
programs. Therefore, we argue that random inputs generated by fuzzers are also
suitable to contribute to randomized mutation for testing function data
augmentation. Thus, this first requirement is satisfied by engaging fuzzing in
the data augmentation process.

Fuzzing can be conducted on different levels of the software. AFL++ (Fioraldi
et al., 2020) feeds inputs to whole programs and tests the program entirely.
or on function level with user-defined fuzz targets using LibFuzzer
(Serebryany, 2016), on the other hand, allows users to define customized fuzz
targets to select the most important functions as the entry point. We select
libFuzzer for its function-level fuzzing feature, which ensures syntax
correctness on invoking the corresponding focal function. If we can compile
and run the fuzz target successfully, we are confident that the testing code
is valid training data for the language model. Therefore, the validity of
FuzzAug is guaranteed.

Furthermore, fuzzing is a programming language agnostic testing approach.
LibFuzzer is a part of the LLVM compiler infrastructure (Lattner and Adve,
2004), which supports any language if a compiler frontend is implemented. We
choose Rust (Matsakis and Klock, 2014) to conduct our study to take advantage
of its powerful build tool cargo 111https://doc.rust-lang.org/cargo/. Cargo-
fuzz (The rust-fuzz Project Developers, 2024) allows software developers to
define their fuzz targets inside the repository, making it easier for us to
execute the fuzz targets to apply our data augmentation. In principle, our
method can be generalized to all libFuzzer-supported languages, and their
corresponding fuzz targets can be found in OSS-Fuzz (Serebryany, 2017b).

To collect inputs with the program’s dynamic behavior from the fuzzing loop,
we instrument a reporter to each fuzz target as shown in Figure 2 top row. We
use reporter to denote the reporting process in the figure for simplicity.
After all the fuzz targets in the project are instrumented, we start the
fuzzing loop for each target and save the reported inputs as a randomly
generated portion of our data augmentation process.

###  2.3. Transformation for Code Representation

⬇

1 language=Rust,

2 caption={Fuzz target collected from OSS.},

3 label=lst:rust:fuzz_target,

4 ]

5#![no_main]

6#[macro_use] extern crate libfuzzer_sys;

7extern crate base64;

8use base64::*;

9mod utils;

10

11fuzz_target!(|data: &[u8]| {

12 let engine = utils::random_engine(data);

13 let _ = engine.decode(data);

14});

⬇

1#[test]

2fn test_1() {

3 let data = [...];

4 let engine = utils::random_engine(data);

5 let _ = engine.decode(data);

6}

Listing 1: Unit test augmentation transformed from LABEL:lst:rust:fuzz_target.

Figure 4. Example fuzz target and the transformed unit test in crate base64 in
our collected Rust dataset. We remove import statements in 1 for simplicity.

Fuzz targets

Code generation with casual language modeling requires the training data to be
valid and complete, with appropriate semantics within the tokens. Therefore,
we cannot add fuzzing data to training data directly due to the distinct
representations between raw fuzzing inputs and meaningful unit test functions.
Fuzzers treat all inputs as bytes and apply byte-level random mutations, for
example, bit-flip. Previous work on using fuzzing data for code understanding
tasks decodes the raw inputs into strings and attaches the inputs to the
program (Zhao et al., 2023) or uses different language modeling loss functions
for two kinds of data (Huang et al., 2023). However, these approaches are not
applicable to generative models, so we need to design a different
representation for fuzzing data as augmentation for test generation.

Since we aim to teach language models to write better unit tests, the
augmentation must be in valid syntax and carry test semantics. The validity of
syntax is guaranteed by executing the fuzz targets. To run the fuzzer
successfully, one must make sure the fuzz target contains no syntax error. In
addition, the implementation of the fuzz targets naturally embeds the testing
semantics. Fuzz targets, as shown in LABEL:lst:rust:fuzz_target, are perfect
examples of testing the focal functions under given input data. Therefore, we
can transform the fuzz targets into unit test templates that keep both syntax
validity and semantic integrity of test functions.

To obtain valid new test functions to provide testing semantics for fuzzing
data, we implement the syntax transformation in the compiler frontend for the
fuzz targets. We use Rust compiler’s proc_macro (David Tolnay and Alex
Crichton, 2024) crate for tokenization and syn (David Tolnay, 2024) to parse
the tokens into an abstract syntax tree (AST). Then we rewrite the macro for
fuzz targets into valid function definitions with the #[test] attribute on top
to help test discovery. We call the result of the syntax transformation step
_test template_. Finally, we fill in the test template with data reported
during fuzzing to generate valid new unit test functions with diverse inputs
as data augmentation.

Algorithm 1 Fuzzing as Data Augmentation

1:function ReporterInstrumentation($fuzz\\_target$)

2: $AST*\leftarrow\textsc{Parse}(fuzz\\_target)$

3:$\triangleright$ Pointer to AST for mutation

4: $entry*\leftarrow\textsc{GetBegin}(AST*)$ $\triangleright$ Pointer to the
entry point

5: $data\leftarrow\textsc{GetParam}(AST*)[0]$

6: AddInst($entry*$, Report($data$)) $\triangleright$ Add reporter to AST

7: $fuzz\\_target_{new}\leftarrow\textsc{Dump}(AST*)$

8: return $fuzz\\_target_{new}$

9:

10:function SyntaxTransformation($fuzz\\_target$)

11: $AST*\leftarrow\textsc{Parse}(fuzz\\_target)$

12: $body\leftarrow\textsc{ExtractBodyNode}(AST)$

13: $test\\_header\leftarrow...$ $\triangleright$ Language-specific header

14: $data\\_template\leftarrow...$ $\triangleright$ Declaring data variable

15: $test\\_ending\leftarrow...$ $\triangleright$ Closing this test definition

16: return $test\\_header+data\\_template+body+test\\_ending$

17:

18:function FuzzAug($repo$, $N$, $L$, $timeout$)

19:$\triangleright$ $repo=$ repository to apply FuzzAug.

20:$\triangleright$ $N=$ number of training examples to generate.

21:$\triangleright$ $L=$ maximum input length for collection.

22:$\triangleright$ $timeout=$ maximum allowed fuzzing time.

23: $dataset_{aug}\leftarrow[]$

24: for all $t\in\textsc{GetFuzzTarget}(repo)$ do

25: $t^{\prime}\leftarrow\textsc{ReporterInstrumentation}(t)$

26: $xs\leftarrow\textsc{Fuzz}(t^{\prime},timeout)$ $\triangleright$ Collect
raw fuzzing inputs

27: $selected\leftarrow\textsc{Sample}(xs)$

28: $templates\leftarrow\textsc{SyntaxTransformation}(t)\times N$

29: $dataset_{aug}\leftarrow dataset_{aug}+\textsc{Fill}(templates,selected)$

30: return $dataser_{aug}$

Syntax transformation from fuzz targets to unit test templates differs for
languages. However, the general framework can be defined in a language-
agnostic manner. UniTSyn (He et al., 2024) designed a multi-lingual framework
to collect unit test functions based on tree-sitter (tre, [n. d.]), and we
believe it can be extended to syntax transformation. If further research finds
a native compiler as a more desirable tool, an approach similar to ours can
also be applied using ast222https://docs.python.org/3/library/ast.html module
for Python datasets. We discuss the application of FuzzAug to other languages
in detail in subsection 5.1.

##  3\. Experimental Setup

To fully evaluate FuzzAug, we build two test generation models UnitCoder and
FuzzCoder. UnitCoder is trained on a Rust dataset containing focal functions
and unit test functions, and FuzzCoder is trained on the same dataset but
augmented by FuzzAug. UnitCoder serves as a baseline comparison to demonstrate
the effects of FuzzAug. To explore the quality of test cases generated by the
language models, we adopt the popular HumanEval-X (Zheng et al., 2023)
benchmark. In this section, we explain the details of our experimental setup.

###  3.1. Data Collection

Table 1. Dataset Statistics. We gathered Unit Tests and Fuzzing test cases separately. Fuzz(100) is the dataset we transformed from fuzz targets with the 100 recorded input data. Out of the 249 repositories we downloaded, 217 of them can be compiled for fuzzing.  Type of Test Dataset |  | # Repo w/ Tests |  | # Focal Calls |  | # Focal-Test Pairs |  | # Tokens
---|---|---|---|---|---|---|---|---
Unit Tests |  | 249 |  | 12452 |  | 6481 |  | 6.2M
Fuzz(100) |  | 217 |  | 52127 |  | 20504 |  | 96.4M
Combined Augmented Dataset |  | 249 |  | 64579 |  | 26985 |  | 102.6M

We chose Rust language to conduct this research for three reasons. First, Rust
projects are highly structured with \src, \tests, and \fuzz directories on the
top level. With the cargo package manager, we can build and run the project
without solving dependency issues. Second, the Rust compiler has built-in
support for unit testing (Klabnik and Nichols, 2018) and fuzzing (The rust-
fuzz Project Developers, 2024), so collecting unit tests and fuzzing data is
straightforward. Third, Rust’s syntax for libFuzzer passes a closure to a
predefined macro, so we can easily transform fuzzers to unit test format. This
eases the problem that the model needs to learn two different representations
for the two kinds of testing data.

##### Repository and Package Mining

We follow previous work (Husain et al., 2019; Rao et al., 2023; He et al.,
2024) to collect open-source repositories from GitHub. Specifically, we follow
UniTSyn (He et al., 2024) to select Rust repositories based on stars and the
date of the last commit. We require the repositories to be popular (with more
than ten stars) and under active development (with new commits after January
1st, 2020). We also de-duplicate the dataset by filtering out repositories
that are archived, forked, or mirrored. In addition to the requirements on the
project’s quality as described above, we also require the repository to
contain pre-defined unit tests and fuzzers in order to apply FuzzAug. For this
purpose, we check the presence of \tests and \fuzz directory in the root of
the project. In total, we find 249 suitable repositories to build our training
dataset.

##### Unit Test Collection

Different from previous work training on file-level code-test pairs (Rao et
al., 2023), we follow (Nie et al., 2023; He et al., 2024) to collect our
training data as function-level pairs since it suits our data augmentation
method. We implement the Rust hook for the UniTSyn (He et al., 2024) based on
the #[test] attribute on top of the Rust unit test functions. To find the call
to the focal function, since assertion in Rust is a macro instead keyword or
function as in UniTSyn, we extend the framework to handle this marco special
case. From the downloaded repositories, we found 12452 calls to the focal
functions in the unit tests, and collected 6481 focal-test pairs as training
data.

##### Augmented Test Collection

We chose LLVM libFuzzer (Serebryany, 2016) for it’s function-level fuzzing
feature. For Rust, libFuzzer is supported as cargo-fuzz mentioned in the
previous sections. For each fuzz target in the repository, we instrumented it
to report the input fuzzing data. We used the body of the fuzz target macro to
generate an equivalent unit test template, as depicted in Figure 2. We fuzzed
all targets for 1 minute on a server with dual 20-core 40-thread x86_64 CPUs
and 692 GB of RAM, then substituted the reported data back to the generated
unit test function template for training. Out of the 249 repositories we
downloaded, 217 of them can be compiled successfully for fuzzing.

After data augmentation, we collect the focal-test pairs for the generated
test functions using UniTSyn, which supports data collection in various PLs
via the language server protocol. However, the language server rust-analyzer
333https://github.com/rust-lang/rust-analyzer is not functioning on fuzz
targets. As a result, we construct a path-based focal function finder to
collect augmented focal-test pairs as an extension to UniTSyn. We collected in
total of 20,504 additional focal-test pairs generated by FuzzAug. We summarize
the statistics of the dataset in Table 1.

###  3.2. Model Training

We select SantaCoder (Allal et al., 2023) to be our base model. SantaCoder is
a state-of-the-art open-source code generation model with 1.1B parameters,
which is powerful enough to generate valid code and efficient enough to
evaluate the augmented data. We follow the previous work (Radford et al.,
2019; Rozière et al., 2023; Nijkamp et al., 2023b; He et al., 2024) to use an
autoregressive signal for continual training of the pre-trained base model. We
follow UniTSyn for the basic training configuration. Specifically, each
training example is the concatenation of the focal function and the unit test
function, joined by a \n new line symbol. We set the maximum sequence length
to 2048 for each training example with a batch size of 16. We use a $5e-5$
learning rate for our training, with 50 warm-up steps and cosine annealing
learning rate decay for each batch (Loshchilov and Hutter, 2016). Following
Kirkpatrick et al., we use 0.05 weight decay to make the trained model robust
to catastrophic forgetting. We train the model UnitCoder with only unit tests
for 300 steps until the loss converges and the FuzzCoder with the augmented
dataset for 800 steps until convergence and is proportional to the size of the
training data.

Our training approach is to apply causal language modeling (CLM) on the pairs
of focal function and unit test function (focal-test-pair). With CLM, the
language model can learn as an unsupervised distribution estimation from a set
of focal-test-pairs examples. With the focal function comes first in each
training example, the language model can understand the causal relationship
between the tokens in the focal function and the tokens in the unit test
function, and therefore can generate desired unit test functions given the
focal function as prompts. This process is also called generative pre-training
by earlier work (Radford et al., 2018, 2019). Let
$\mathcal{F}=\\{f_{1},\ldots,f_{p}\\}$ denote the tokens for the focal
function and $\mathcal{T}=\\{t_{1},\ldots,t_{q}\\}$ denote the tokens for the
corresponding unit test, a training example (focal-test-pair) in our training
corpus is represented as

(1) $\mathcal{U}=\mathcal{F}\oplus\\{\texttt{\textbackslash
n}\\}\oplus\mathcal{T}=\\{u_{1},\ldots,u_{r}\\}$

where $r=p+q+1$ is the length of the token sequence and $\oplus$ is vector
concatenation. Following (Radford et al., 2018), our training objective is
defined as maximizing the log-likelihood

$L(\mathcal{U})=\sum_{i}\log P(u_{i}|u_{i-k},\ldots,u_{i-1},\theta)$

where $k=2048$ is the maximum sequence length (context window size) and
$\theta$ is the parameters of SantaCoder (Allal et al., 2023). In simpler
terms, our objective is to predict the next token in a token sequence given
all the previous tokens. We train $\theta$ parameters of all models using the
Adam optimizer (Kingma and Ba, 2014). SantaCoder is a decoder-only
architecture using Transformer (Vaswani et al., 2017), where each Transformer
Block (TB) uses multi-headed self-attention. With this training object and the
aforementioned hyperparameter settings, we convert the pre-trained SantaCoder
model to a unit test generation model to aid automatic software testing. The
training process of FuzzCoder takes around 12 hours on a single Nvidia A100
GPU.

###  3.3. Research Questions

To evaluate our data augmentation method, we structure our experiments around
the following two research questions on the quality of generated unit tests:

##### RQ.1. Can FuzzAug improve the accuracy of generated test cases?

Software testing aims to discover hidden bugs in the code. The prerequisite of
this aim is to have _accurate_ test cases, where the generated input and
output to the focal function match with the ground truth. Therefore, measuring
the accuracy of generated test cases is essentially to evaluate if the
language model is useful in software testing. Generating the correct input-
output of the focal function requires the model to learn both the semantics
and runtime behavior of the focal function. Predicting the correct inputs and
corresponding outputs is challenging for language models (Gu et al., 2024), so
it is an important direction for models to improve on. We follow previous work
(Chen et al., 2022; He et al., 2024) to parse all the generated test cases and
execute them against the ground truth focal function independently.

##### RQ.2. Can FuzzAug improve the usefulness of generated unit tests?

Accurate assertions are essential for valid unit test functions, while
comprehensive coverage on the focal functions _enhances their quality_.
Therefore, we use the compile rate, pass rate, and branch coverage on the
focal functions to check the completeness and usefulness of the generated unit
test functions. To evaluate the coverage fairly, we apply some basic post-
process (details in subsection 3.4) on the generation results, run the entire
test function, and record the branch coverage using grcov (Marco Castelluccio,
2024). Using branch coverage as a metric for neural test coverage allows us to
better evaluate the usefulness of the test generation models. and thus
demonstrates the necessity of FuzzAug.

##### RQ.3. Can FuzzAug generalize to other models?

Data augmentation methods only operate on the training set, thus applying them
should improve all the models in the same task. The same augmentation method
should produce similar performance improvement trends on different models. To
evaluate the generalizability of FuzzAug, we apply the same training and
evaluation setting to continue train CodeLlama (Rozière et al., 2023) and
report the effects of FuzzAug.

###  3.4. Evaluation Setup

####  3.4.1. Evaluation datasets

We evaluate all the models fairly on HumanEval-X (Zheng et al., 2023).
HumanEval-X is a hand-crafted multilingual benchmark for code generation
tasks, containing problems in popular languages including Rust. HumanEval-X
has 164 different problems, where each of them is composed of description
prompt in natural language, function declaration (header), canonical solution
(ground truth implementation), and unit test function. HumanEval-X was
designed for code generation and code translation in different languages, but
is also useful for test generation. In order to turn the Rust subset of
HumanEval-X into a test generation benchmark, we follow UniTSyn (He et al.,
2024) to concatenate the function declaration and canonical solution as the
focal function, and use the focal function to prompt the model to generate the
corresponding unit test.

We choose to evaluate the test generation models on HumanEval-X for three
considerations. First, we try to eliminate data leakage. The training dataset
of the state-of-the-art code generation models is likely to contain the
popular repositories on GitHub, including the ones we used to build our
dataset. Therefore, evaluating the models on the repositories we collected
from GitHub might be unfair due to data leakage. Although comparing with other
state-of-the-art models is not our goal, the effect of data leakage on
evaluating FuzzAug is unknown. To minimize such effects, we choose to use a
hand-crafted benchmark, and HumanEval-X is the only hand-crafted benchmark we
found that contains Rust problems.

Second, we want to minimize the negative impacts of incorrect project setup.
Generating unit tests in large open source software (OSS) requires special
setups for each project. These setups for defect testing are hard to construct
and require human domain knowledge (Zhu and Rubio-González, 2023). Therefore,
choosing to evaluate test generation on OSS introduces additional bias in the
results, which is another thing we want to eliminate.

Third, Rust programs are hard to compile in a complex codebase. Different from
other popular languages in code generation, for example, Python and JavaScript
with dynamic types and C/C++ with weak types, Rust’s type system is both
static and strong, making code generation in Rust hard to pass the type
checking and therefore failed to compile. In addition to static and strong
typing, Rust introduced linear logic (Girard, 1987) (linear types (Wadler,
1990)) into its type system to manage the lifetime and ownership of pointers,
adding additional difficulty in generating type-correct code in the complex
codebase. To this end, we found HumanEval-X to be a more suitable benchmark
since the problems are simple enough without any over-complex types, yet are
challenging enough that the focal functions contain multiple branches to
evaluate code coverage. As a result, we select HumanEval-X to evaluate FuzzAug
instead of using the OSS projects.

####  3.4.2. Prompts

For all our experiments, we use the same prompting method as in CodeT (Chen et
al., 2022) to guide the language models to generate assertions. As shown in
LABEL:lst:prompt, we use natural language “Check the correctness of
`function_name`” in comments to instruct the model to complete the test
function. We guide the generation of assertions by providing the language-
specific assert keyword and the incomplete invocation of the focal function as
shown on Line 15. We allow the model to predict at most 1024 new tokens for
the synthesized assertions for all models. We set the generation temperature
to 1 for all the models to encourage output diversity. We concatenate the
import statements, the focal function implementation, the natural language
instruction in the comment, and the test header together as the import prompt
to the language model.

⬇

1 belowskip=-2\baselineskip,

2 float,

3 language=Rust,

4 caption={Example prompt used for test generation. Import statements are
removed for simplicity.},

5 label=lst:prompt,

6]

7fn has_close_elements(numbers: Vec<f32>, threshold: f32) -> bool { ... }

8// Check the correctness of ‘has_close_elements‘

9#[cfg(test)]

10mod tests {

11 use super::*;

12

13 #[test]

14 fn test_has_close_elements() {

15 assert_eq!(has_close_elements(

####  3.4.3. Post-processing

We avoid overly intricate processing of the generated test functions to keep
our evaluation results faithful. We only apply the most basic post-process to
make the generated test function complete. We first count the number of the
curly brackets. If the number of opening brackets matches the number of
closing brackets, we consider the generated test function as complete.
Otherwise, we check if the last generated line ended with a semicolon to see
if the last line is complete. If not, we remove that line. Then we add the
missing closing curly brackets to complete the generated test function.

##  4\. Results

###  4.1. Test Case Accuracy

Table 2. Evaluations of the accuracy of generated test cases. “#Accurate per
Task” is the average number of accurately generated assertions per task for
the 164 tasks in HumanEval-X. “Compile Rate” measures the number of generated
assertions that compiles without error over the total number of generated
assertions. “Accuracy” is the number of accurate assertions over the total
number of generated assertions.

Model | #Accurate per Task | Compile Rate | Accuracy
---|---|---|---
UnitCoder | 2.32 | 0.66 | 0.19
FuzzCoder | 4.25 | 0.68 | 0.21

We follow CodeT (Chen et al., 2022) to guide the language models to generate
independent test cases (assertions). Since the assertions are independent, we
can parse them and evaluate each one of them individually. We present the
accuracy of the baseline UnitCoder and the competitor FuzzCoder in Table 2.
Our model trained on the augmented dataset outperforms the baseline trained on
only unit tests on all three metrics. In particular, given the same maximum
token length for generation, our model can generate nearly twice the number of
accurate test cases than the baseline model. In addition to the number of
accurate assertions per task, FuzzCoder achieved an 11% relative margin
compared to the baseline on overall accuracy. FuzzCoder also generates test
cases with a higher compile rate, meaning that it learned the way to invoke
the focal function correctly. Rust is a strong and static typed language, so
the improvement in compile rate indicates that FuzzCoder can generate input-
output pairs that are type valid to the focal function and thus pass the type
checking. This advancement is due to the additional focal function invocations
with valid and executable inputs collected from fuzzing, which the model can
learn the relationship between the desired input type and the type signature
of the focal function.

RQ.1. Can FuzzAug improve the accuracy of generated test cases? FuzzAug can
improve the neural test generation model on all aspects of test case accuracy.
FuzzCoder trained with FuzzAug shows a higher assertion compiled rate than
UnitCoder trained without FuzzAug. FuzzCoder generated twice accuracy test
cases on average for each focal function and achieved an 11% relative
improvement on overall test case accuracy.

###  4.2. Unit Test Completeness and Usefulness

Table 3. Evaluations of branch coverage on whole unit test functions generated
by LLM. “Compile Rate” is the ratio of generated whole unit tests that compile
without error. “Pass Rate” is the ratio of generated unit tests that was
passed. “Branch Coverage” is the average branch coverage of generated unit
tests on the focal functions.

Model | Compile Rate | Pass Rate | Branch Coverage
---|---|---|---
UnitCoder | 0.68 | 0.06 | 4.88
FuzzCoder | 0.63 | 0.17 | 9.31

To evaluate if FuzzAug can help the model generate complete and useful unit
test functions, we evaluate the generated unit test functions entirely without
extracting the individual assertions. Results for this experiment are shown in
Table 3. While maintaining similar compile rates, FuzzCoder outperforms
UnitCoder by a large margin in both accuracy and branch coverage. With no
specific guiding, the accuracy of UnitCoder dropped conspicuously, whereas
FuzzCoder achieved similar results as with guided prompt. This indicates the
outstanding performance of FuzzCoder in real-world SE applications, where the
unit test functions are not simply asserting the input-output pairs. FuzzCoder
also achieved more than twice the average branch coverage than UnitCoder. This
outshines in branch coverage suggest that FuzzAug can help the model to
understand the structure of the focal function and generate more useful unit
test functions to discover potential bugs and vulnerabilities in the software.
In practice, evaluating the generated unit tests as a whole function is more
reasonable than individual assertions for their better readability and
maintainability with richer context.

RQ.2. Can FuzzAug improve the usefulness of generated unit tests? In a real-
world software engineering setting, FuzzCoder not only generates more passed
unit tests with correct setups, but also reaches higher branch coverage on the
focal functions than UnitCoder. Our results indicate that models trained with
FuzzAug can be more practical in writing useful unit tests for focal
functions.

###  4.3. Generalizability of FuzzAug

Table 4. Evaluations of the usefulness of generated unit tests by CodeLlama.
The columns are the same as in Table 3.

Model | Compile Rate | Pass Rate | Branch Coverage
---|---|---|---
UnitLlama | 0.69 | 0.23 | 17.73
FuzzLlama | 0.74 | 0.23 | 18.00

A good data augmentation method should work on different models. We also
trained UnitLlama and FuzzLlama based on CodeLlama (Rozière et al., 2023) to
evaluate the generalizability of FuzzAug. We chose the 7B parameter version of
CodeLlama and applied the same basic training setting as described in
subsection 3.2. Since CodeLlama-7B is a larger model, we have to reduce the
maximum context length from 2048 to 1024. To make sure the training process
can continue without error, we also applied LoRA (Hu et al., 2021) with the
rank $r=16$, $\alpha=16$, dropout $=0.05$. As shown in Table 4, FuzzLlama’s
performance surpasses UnitLlama’s performance with a similar trend as the
experiments on SantaCoder. While maintaining the same pass rate, FuzzLlama
achieves a higher compile rate and branch coverage than UnitLlama trained
without FuzzAug.

RQ.3. Can FuzzAug generalize to other models? FuzzAug is a general data
augmentation method for the neural test generation task. Training dataset
augmented by FuzzAug can be used to train any test generation model and gain
performance improvement in branch coverage on the target focal function, thus
improving the usefulness in real software testing scenarios.

##  5\. Discussion

###  5.1. Applying FuzzAug to Different Languages

As briefly introduced in subsection 2.3, FuzzAug can be applied to any
language if it is supported by OSS-Fuzz (Serebryany, 2017b) and libFuzzer
(Serebryany, 2016). In this section, we dive into the details of syntax
transformation for other popular PLs including C/C++, Java, and Python, and
demonstrate the method to apply FuzzAug.

⬇

1 language=c++,

2 caption={Fuzz target for C/C++ using
libFuzzer~\cite{serebryany2016libfuzzer}.},

3 label=lst:cpp:fuzz_target,

4 ]

5extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {

6 if (size > 0 && data[0] == ’H’)

7 if (size > 1 && data[1] == ’I’)

8 if (size > 2 && data[2] == ’!’)

9 __builtin_trap();

10 return 0;

11}

⬇

1 language=java,

2 caption={Fuzz target for Java using Jazzer~\cite{jazzer}.},

3 label=lst:java:fuzz_target,

4 ]

5public static void fuzzerTestOneInput(FuzzedDataProvider data) {

6 String input = data.consumeRemainingAsString();

7 long random = new SecureRandom().nextLong();

8 if (input.startsWith("magicstring" + random)

9 && input.length() > 30

10 && input.charAt(25) == ’C’) {

11 mustNeverBeCalled();

12 }

13}

⬇

1import atheris

2

3with atheris.instrument_imports():

4 import reverse

5

6def TestOneInput(data):

7 int_data = list(map(int.from_byte, data))

8 reverse(int_data)

Listing 2: Fuzz taget for Python using atheris (Google, [n. d.]).

Figure 5. Example fuzz target in other mainstream languages. The code in each
example are either taken from the cited source’s official example or made up.

Fuzz targets

For languages natively supported by LLVM (Lattner and Adve, 2004) like C/C++,
libFuzzer is supported natively as well. We demonstrate a fuzz target instance
in LABEL:lst:cpp:fuzz_target collect from the boost library, where the syntax
of fuzz targets are functions that take constant data and the size of the
data. The reporter instrumentation for C/C++ projects is implemented using
LLVM’s analysis and transform passes, which can inject code for the reporter
at the optimization stage in the compiling process. This kind of transform
pass also works for other LLVM-supported languages since it operates on the
intermediate representation. Similar to our design of syntax transformation
described in subsection 2.3, our data augmentation framework is extendable to
C/C++ using libclang. We transformed the fuzz target to any valid unit testing
library. Our FuzzAug framework can be applied after these steps are defined
for projects in C/C++.

FuzzAug is also compatible with languages that run on the Java Virtual Machine
(JVM). Although JVM languages are memory-safe, fuzzing can reveal unintended
behaviors in programs written in these languages. Fuzzing for JVM languages is
supported by Jazzer (Intelligence, 2024), which offers the syntax for fuzz
targets similar to libFuzzer, as shown in LABEL:lst:java:fuzz_target. JVM
supports the javaagent tool for reporter instrumentation and syntax
modifications, enhancing support for FuzzAug on Java dataset. Given Java’s
extensive use in LLM code generation (Allal et al., 2023; Rozière et al.,
2023; Zheng et al., 2023) and test generation (Nie et al., 2023; Rao et al.,
2023; He et al., 2024), it is crucial to apply FuzzAug to Java unit test
datasets.

Python is another popular programming language with extensive usage in code
generation. Based on libFuzzer, Google developed atheris (Google, [n. d.]) for
fuzzing Python software. This enables the extension of FuzzAug to Python
datasets. Reporter instrumentation is supported by decorators (Smith et al.,
2003), and syntax transformation is based on the built-in ast. We display the
syntax transformation for Python fuzz targets in 2. With support for C/C++,
Java, and Python, FuzzAug enhances data augmentation across the majority of
widely used PLs. This broad compatibility underscores our contribution to the
advancement of deep language models for program testing.

###  5.2. Applying FuzzAug to Different Datasets

We followed TeCo (Nie et al., 2023) and UniTSyn (He et al., 2024) to construct
our dataset on function-level focal-test pairs. In this case, each training
example in Equation 1 is a focal function and a unit test function
concatenated by a new line symbol. Although this approach is efficient on the
token length and performs well, the file-level pairing approach described in
CAT-LM (Rao et al., 2023) also has the benefit of providing more relevant
contexts. This approach is beneficial when the program lacks modularity, which
is common when the software gets complex. Therefore, a valid data augment
method should also work on file-level data instead of only on function-level
data. Indeed, FuzzAug satisfies the file-level data pairing requirement.
LibFuzzer requires the developer of the project to create different fuzzer
targets in different files. Therefore, FuzzAug fits in the training language
model on file-level datasets too. After syntax transformation and fuzz data
collection, FuzzAug can put the augmented unit test functions back in the
original fuzz target’s file, and then use CAT-LM’s strategy for pairing. Thus,
the value of FuzzAug also shines in its ability to improve and augment all
kinds of unit test datasets.

###  5.3. Function Specification as Test Template

⬇

1from hypothesis import given

2from hypothesis.strategies import integers, lists

3

4@given(integers())

5def prop_unit(x):

6 assert [x] == reverse([x])

7

8@given(xs=lists(integers()),

9 ys=lists(integers()))

10def prop_app(xs, ys):

11 assert reverse(xs + ys) == reverse(ys) + reverse(xs)

12

13@given(xs=lists(integers()), ys=lists(integers()))

14def prop_rev(xs, ys):

15 assert reverse(reverse (xs)) == xs

Listing 3: Property-based testing in Python.

Property-based testing (PBT) (Claessen and Hughes, 2000) is a hybrid approach
between self-assessment testing like unit tests and randomized testing like
fuzzing. In PBT, the developers identify a set of properties that the function
should satisfy and define the input domain for each of the properties. The
framework generates suitable inputs based on the instruction to test these
properties randomly. The set of properties defined for a function is called
its _specification_. PBT is also commonly used in software engineering to find
unexpected behaviors in the program, especially in functional programming
where each function is pure and specializes in one purpose, making the
specification easier to define. We demonstrate PBT in 3 with an example of the
reverse function and its three specifications. PBT verifies the correctness of
the focal function by checking if the specifications hold for all possible
inputs.

⬇

1 belowskip=-2\baselineskip,

2 float,

3 language=Rust,

4 caption={Fuzzer targets augmented by \fullname in \texttt{base64} that
employs coverage-guided property-based fuzzing.},

5 label=lst:fuzz_PBT,

6]

7fuzz_target!(|data: &[u8]| {

8 let encoded = STANDARD.encode(data);

9 let decoded = STANDARD.decode(&encoded).unwrap();

10 assert_eq!(data, decoded.as_slice());

11});

Since PBT also generates random inputs to test the program, we can use these
specifications as our test template. Recent studies have combined the
coverage-guided input generation feature of fuzzing and PBT as coverage-guided
property-based fuzzing (Lampropoulos et al., 2019; Padhye et al., 2019).
Combining the advantages of fuzzing and PBT, this approach is widely accepted
in testing memory-safe languages. In the augmented dataset we used to train
FuzzCoder, some of the augmented unit test functions are converted from
property-testing fuzz targets as shown in LABEL:lst:fuzz_PBT. Through our
experiments, we find that adding the knowledge of function specification in
the training set contributes to the completeness and coverage of the generated
unit test function. The feature of introducing specifications and properties
in data augmentation further enhance our contribution to neural test
generation by designing and building FuzzAug.

##  6\. Related Works

###  6.1. Fuzzing

Fuzz testing (Zeller et al., 2019), or fuzzing, is a popular execution-based
dynamic testing technique with randomized inputs, which aims to generate a set
of inputs based on the provided set of seeds to achieve high code coverage.
The fuzzing process can be described as a coverage-guided input modification
method. The fuzzer uses behavior monitoring to find inputs with high branch
coverage and favor those inputs for future input generation. Behavior
monitoring enables the fuzzer to focus on exploring new program branches that
are not yet tested to detect unseen security issues like memory errors. AFL++
(Fioraldi et al., 2020) and libFuzzer (Serebryany, 2016) are two of the most
popular fuzzers that are widely adopted in testing real-world software.
Libfuzzer is integrated into the LLVM compiler infrastructure (Lattner and
Adve, 2004) to test projects in LLVM-compiled languages, like C/C++, Rust,
Swift, and Julia. One of the advantages of libFuzzer is to discover memory
errors via the memory sanitizer, but it is also useful to find other
unexpected behaviors and crashes in memory-safe language with garbage
collection like JVM-based languages (Intelligence, 2024) and Python
(Serebryany, 2017a). Fuzzing is a robust method to find bugs in various
software domains (Rong et al., 2020; Chen et al., 2023; Rong et al., 2024a).

Coverage-guided grey-box fuzzing verifies the software by repeatedly
generating inputs for the software to execute and prioritize the inputs that
reach new paths in the program. One of the key components is to determine if
the input seed triggers new and interesting behaviors. Several related work
have proposed methods to improve fuzzing in this direction, including using
deterministic (Rong et al., 2024b) and gradient-based (Chen and Chen, 2018;
She et al., 2019) techniques. This coverage-guided input generation feature
allows the selected and generated input seeds to carry information on the
program’s dynamic runtime behavior, which is shown successfully and useful in
training machine learning models for code understanding (Zhao et al., 2023;
Huang et al., 2023).

Fuzzing was adopted as a data augmentation tool to improve the robustness of
neural networks. SenSei (Gao et al., 2020) introduced fuzzing to improve the
robust generalization of DNNs in training. Their key insight is that guided
search like fuzzing and genetic programming can be more effective in finding
optimal variants compared to other DA-produced random variants as described by
Krizhevsky et al.. The authors also pointed out that SenSei can help the
training process by skipping training data that is considered “not
interesting” by fuzzing. This work mainly focuses on improving the robustness
of neural networks, and is not a work to improve generative language models
and neural test generation. Although both incorporated fuzzing to mutate the
training dataset, SenSei and our work FuzzAug are fundamentally different in
approach and application.

###  6.2. Test Generation via Large Language Models

Using deep neural networks, especially LLMs, to generate test cases is a new
trend in automatic software testing. This method is referred to as _neural
test generation_. Some approach toward neural test generation is to instruct
pre-trained code generation LLMs like CodeT (Chen et al., 2022), CodeT5 (Wang
et al., 2021), CodeGen (Nijkamp et al., 2023a), and CodeLlama (Rozière et al.,
2023). Similarly, large foundation models like GPT4 (Achiam et al., 2023) can
also do this work. The other approach is to train test-specific models that
are specialized in generating test cases or unit test functions. This category
of neural test generation includes ATLAS (Watson et al., 2020), AthenaTest
(Tufano et al., 2021), TOGA (Dinella et al., 2022), A3Test (Alagarsamy et al.,
2023), TeCo (Nie et al., 2023), CAT-LM (Rao et al., 2023), and UniTSyn (He et
al., 2024). The more recent work (Nie et al., 2023; Rao et al., 2023; He et
al., 2024) proposed to train the test generation model on _aligned_ data that
includes the correspondence between the unit test and the function under test
(also referred to as _focal function_). TeCo and UniTSyn are in the direction
of function-level alignment to provide fine-grained correspondence, whereas
CAT-LM adopted file-level alignment for the efficiency of pairing and richer
contexts.

At inference time, previous research can be divided into only generate test
cases (input-output pair in assertions) (Watson et al., 2020; Dinella et al.,
2022; Alagarsamy et al., 2023; Nie et al., 2023) and generate complete test
files (Rao et al., 2023) or functions (He et al., 2024). Generating only the
test cases might not be useful in real-world software engineering as they are
difficult to set up for execution as part of continuous integration (CI).
Writing the complete test file or test function is more promising in practice.
However, generating a whole file at once has limitations. First, it demands a
long context window and more tokens, which is costly in practice for large
software repositories. Second, the LLM might be distracted by the long
context, causing the generated unit tests to reach insufficient coverage on
each of the focal functions. Therefore, generating each unit test function for
every single focal function is preferable.

###  6.3. Data Augmentation for Language Processing

Data augmentation was first developed for computer vision (CV) and later
adopted by natural language processing (NLP). With the advancement of LLMs,
data augmentation techniques gain additional interest from low-resource
domains and new tasks due to the scaling law of large models (Feng et al.,
2021). The challenge of data augmentation for natural languages is the
discreteness of features, where methods like injecting random noise, cropping,
and rotating for images introduced by (Krizhevsky et al., 2012; Simonyan and
Zisserman, 2014) cannot be directly applied to NLP. DAs for NLP are mainly
rule-based (Hariharan and Girshick, 2017; Schwartz et al., 2018; Wei and Zou,
2019; Xie et al., 2020) or model-based (Sennrich et al., 2015; Kobayashi,
2018; Gao et al., 2019; Nie et al., 2020; Feng et al., 2020). Rule-based DA
applies techniques like random insertion, deletion, and swap to text at the
token level. Model-based DA utilizes language models to translate or rewrite
existing text, or generate new text as new data. These methods are effective,
but still lack quality control since the augmented data might contain
unnatural or grammatically incorrect text, leading to models learning
incorrect patterns.

DA for programming languages is still insufficiently studied compared to DA
for NL due to the stricter grammar. Yu et al. proposed to augment code data by
source-to-source program transformation. This approach is rule-based and
operates on the program based on the denotational semantics. Similar to
program transformation techniques used to generate simple program for compiler
testing (Yang et al., 2011; Le et al., 2014; Livinskii et al., 2020; Rong et
al., 2024a), Yu et al. designed a set of grammar transformation rules to alter
small parts of the program to produce a new valid program. This approach can
generate a large amount of programs efficiently, but only works on simple
programs whose complexity cannot compare with real-world software. Therefore,
this method is not ideal for training large generative models. Up until we
wrote this paper, there existed no DA for unit test datasets and neural test
generation like our method FuzzAug.

##  7\. Conclusion

We developed FuzzAug, a data augmentation method for unit test function
generation. FuzzAug combines the advantages of coverage-guided fuzzing and
generative large language model to generate tests that are not only
semantically meaningful but also strategically comprehensive. We applied
FuzzAug for the continual training of SantaCoder and CodeLlama and
demonstrated its effectiveness. Although our approach is efficient and can be
automated, it requires the crate’s authors and contributors to define the
fuzzers manually. Therefore, we conduct our experiments on Rust crates that
have pre-defined fuzzers. This method can be generalized to all languages that
are supported by OSS-Fuzz with slight modifications. Our results show the
effectiveness of employing dynamic program analysis to generate high-quality
inputs to argument code corpus in training language models. We believe FuzzAug
can spur the development of unit test generation by large language models and
contribute to the field of AI for software engineering and testing.

## Limitation

Although FuzzCoder achieved extensively higher branch coverage and accuracy
than the baseline UnitCoder, it got a slightly lower compile rate in Table 3.
This is because some of the unit test functions generated by FuzzCoder have no
valid assertions. We notice that some test cases generated by FuzzCoder are
extremely long, consuming all the generation context without finishing with
any valid test cases. The cause of this behavior is the coverage-guided nature
of fuzzing. When the input type is a vector or long number (i64, f64, etc),
since very long inputs are likely to trigger edge cases, the input queue would
favor these long inputs for higher coverage. However, for generative models,
the acceptable token length is much smaller than random access memory, so
FuzzCoder may generate test cases whose inputs consume all the token limits,
causing the generated test function to fail to compile. This happens scarcely,
so it will not harm the performance of neural test generation in practice.

## References

  * (1)
  * tre ([n. d.]) [n. d.].  _Tree-sitter Introduction_.  https://tree-sitter.github.io/tree-sitter/
  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023\.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ (2023).
  * Alagarsamy et al. (2023) Saranya Alagarsamy, Chakkrit Tantithamthavorn, and Aldeida Aleti. 2023.  A3Test: Assertion-Augmented Automated Test Case Generation.  _arXiv preprint arXiv:2302.10352_ (2023).
  * Allal et al. (2023) Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. 2023\.  SantaCoder: don’t reach for the stars!  _arXiv preprint arXiv:2301.03988_ (2023).
  * Chen et al. (2022) Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022.  CodeT: Code Generation with Generated Tests.  arXiv:2207.10397 [cs.CL]
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021.  Evaluating Large Language Models Trained on Code.  arXiv:2107.03374 [cs.LG]
  * Chen and Chen (2018) Peng Chen and Hao Chen. 2018.  Angora: Efficient fuzzing by principled search. In _2018 IEEE Symposium on Security and Privacy (SP)_. IEEE, 711–725.
  * Chen et al. (2023) Peng Chen, Yuxuan Xie, Yunlong Lyu, Yuxiao Wang, and Hao Chen. 2023.  Hopper: Interpretative fuzzing for libraries. In _Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security_. 1600–1614.
  * Claessen and Hughes (2000) Koen Claessen and John Hughes. 2000.  QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs.  _SIGPLAN Not._ 35, 9 (sep 2000), 268–279.  https://doi.org/10.1145/357766.351266
  * Daka and Fraser (2014) Ermira Daka and Gordon Fraser. 2014.  A survey on unit testing practices and problems. In _2014 IEEE 25th International Symposium on Software Reliability Engineering_. IEEE, 201–211.
  * David Tolnay (2024) David Tolnay. 2024.  syn: Parser for Rust source code.  {https://github.com/dtolnay/syn}
  * David Tolnay and Alex Crichton (2024) David Tolnay and Alex Crichton. 2024.  proc-macro2: A substitute implementation of the compiler’s ‘proc_macro‘ API to decouple token-based libraries from the procedural macro use case.  {https://github.com/dtolnay/proc-macro2}
  * Dinella et al. (2022) Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu K. Lahiri. 2022.  TOGA: A Neural Method for Test Oracle Generation. In _Proceedings of the 44th International Conference on Software Engineering_ (Pittsburgh, Pennsylvania) _(ICSE ’22)_. Association for Computing Machinery, New York, NY, USA, 2130–2141.  https://doi.org/10.1145/3510003.3510141
  * Feng et al. (2020) Steven Y Feng, Varun Gangal, Dongyeop Kang, Teruko Mitamura, and Eduard Hovy. 2020.  Genaug: Data augmentation for finetuning text generators.  _arXiv preprint arXiv:2010.01794_ (2020).
  * Feng et al. (2021) Steven Y Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021.  A survey of data augmentation approaches for NLP.  _arXiv preprint arXiv:2105.03075_ (2021).
  * Fioraldi et al. (2020) Andrea Fioraldi, Dominik Maier, Heiko Eißfeldt, and Marc Heuse. 2020.  AFL++ combining incremental steps of fuzzing research. In _Proceedings of the 14th USENIX Conference on Offensive Technologies_. 10–10.
  * Fraser and Arcuri (2011) Gordon Fraser and Andrea Arcuri. 2011.  Evosuite: automatic test suite generation for object-oriented software. In _Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering_. 416–419.
  * Gao et al. (2019) Fei Gao, Jinhua Zhu, Lijun Wu, Yingce Xia, Tao Qin, Xueqi Cheng, Wengang Zhou, and Tie-Yan Liu. 2019.  Soft contextual data augmentation for neural machine translation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. 5539–5544.
  * Gao et al. (2020) Xiang Gao, Ripon K Saha, Mukul R Prasad, and Abhik Roychoudhury. 2020.  Fuzz testing based data augmentation to improve robustness of deep neural networks. In _Proceedings of the acm/ieee 42nd international conference on software engineering_. 1147–1158.
  * Girard (1987) Jean-Yves Girard. 1987.  Linear logic.  _Theoretical computer science_ 50, 1 (1987), 1–101.
  * Google ([n. d.]) Google. [n. d.].  Atheris: A Coverage-Guided, Native Python Fuzzer.  https://github.com/google/atheris.
  * Gu et al. (2024) Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. 2024.  Cruxeval: A benchmark for code reasoning, understanding and execution.  _arXiv preprint arXiv:2401.03065_ (2024).
  * Hariharan and Girshick (2017) Bharath Hariharan and Ross Girshick. 2017.  Low-shot visual recognition by shrinking and hallucinating features. In _Proceedings of the IEEE international conference on computer vision_. 3018–3027.
  * He et al. (2024) Yifeng He, Jiabo Huang, Yuyang Rong, Yiwen Guo, Ethan Wang, and Hao Chen. 2024.  UniTSyn: A Large-Scale Dataset Capable of Enhancing the Prowess of Large Language Models for Program Testing.  _arXiv preprint arXiv:2402.03396_ (2024).
  * Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  LoRA: Low-Rank Adaptation of Large Language Models.  arXiv:2106.09685 [cs.CL]
  * Huang et al. (2023) Jiabo Huang, Jianyu Zhao, Yuyang Rong, Yiwen Guo, Yifeng He, and Hao Chen. 2023.  Code Representation Pre-training with Complements from Program Executions.  arXiv:2309.09980 [cs.SE]
  * Husain et al. (2019) Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019.  Codesearchnet challenge: Evaluating the state of semantic code search.  _arXiv preprint arXiv:1909.09436_ (2019).
  * Intelligence (2024) Code Intelligence. 2024.  jazzer: About Coverage-guided, in-process fuzzing for the JVM.  https://github.com/CodeIntelligenceTesting/jazzer.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017\.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ 114, 13 (2017), 3521–3526.
  * Klabnik and Nichols (2018) Steve Klabnik and Carol Nichols. 2018.  _The Rust Programming Language_.  No Starch Press, USA.
  * Kobayashi (2018) Sosuke Kobayashi. 2018.  Contextual augmentation: Data augmentation by words with paradigmatic relations.  _arXiv preprint arXiv:1805.06201_ (2018).
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ 25 (2012).
  * Lampropoulos et al. (2019) Leonidas Lampropoulos, Michael Hicks, and Benjamin C Pierce. 2019.  Coverage guided, property based testing.  _Proceedings of the ACM on Programming Languages_ 3, OOPSLA (2019), 1–29.
  * Lattner and Adve (2004) Chris Lattner and Vikram Adve. 2004.  LLVM: A compilation framework for lifelong program analysis & transformation. In _International symposium on code generation and optimization, 2004. CGO 2004._ IEEE, 75–86.
  * Le et al. (2014) Vu Le, Mehrdad Afshari, and Zhendong Su. 2014.  Compiler validation via equivalence modulo inputs.  _ACM Sigplan Notices_ 49, 6 (2014), 216–226.
  * Li et al. (2023) Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2023.  StarCoder: may the source be with you!  arXiv:2305.06161 [cs.CL]
  * Li et al. (2022) Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022.  Competition-level code generation with AlphaCode.  _Science_ 378, 6624 (2022), 1092–1097.  https://doi.org/10.1126/science.abq1158 arXiv:https://www.science.org/doi/pdf/10.1126/science.abq1158
  * Livinskii et al. (2020) Vsevolod Livinskii, Dmitry Babokin, and John Regehr. 2020.  Random testing for C and C++ compilers with YARPGen.  _Proceedings of the ACM on Programming Languages_ 4, OOPSLA (2020), 1–25.
  * Loshchilov and Hutter (2016) Ilya Loshchilov and Frank Hutter. 2016.  Sgdr: Stochastic gradient descent with warm restarts.  _arXiv preprint arXiv:1608.03983_ (2016).
  * Marco Castelluccio (2024) Marco Castelluccio. 2024.  grcov: Rust tool to collect and aggregate code coverage data for multiple source files.  {https://github.com/mozilla/grcov}
  * Matsakis and Klock (2014) Nicholas D Matsakis and Felix S Klock. 2014.  The rust language.  _ACM SIGAda Ada Letters_ 34, 3 (2014), 103–104.
  * Nie et al. (2023) Pengyu Nie, Rahul Banerjee, Junyi Jessy Li, Raymond J. Mooney, and Milos Gligoric. 2023.  Learning Deep Semantics for Test Completion.  arXiv:2302.10166 [cs.SE]
  * Nie et al. (2020) Yuyang Nie, Yuanhe Tian, Xiang Wan, Yan Song, and Bo Dai. 2020.  Named entity recognition for social media texts with semantic augmentation.  _arXiv preprint arXiv:2010.15458_ (2020).
  * Nijkamp et al. (2023a) Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, and Yingbo Zhou. 2023a.  CodeGen2: Lessons for Training LLMs on Programming and Natural Languages.  _ICLR_ (2023).
  * Nijkamp et al. (2023b) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023b.  CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis.  _ICLR_ (2023).
  * Padhye et al. (2019) Rohan Padhye, Caroline Lemieux, and Koushik Sen. 2019.  Jqf: Coverage-guided property-based testing in java. In _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis_. 398–401.
  * Panichella et al. (2020) Annibale Panichella, Sebastiano Panichella, Gordon Fraser, Anand Ashok Sawant, and Vincent J Hellendoorn. 2020.  Revisiting test smells in automatically generated tests: limitations, pitfalls, and opportunities. In _2020 IEEE international conference on software maintenance and evolution (ICSME)_. IEEE, 523–533.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018\.  Improving language understanding by generative pre-training.  (2018).
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019\.  Language models are unsupervised multitask learners.  _OpenAI blog_ 1, 8 (2019), 9.
  * Rao et al. (2023) Nikitha Rao, Kush Jain, Uri Alon, Claire Le Goues, and Vincent J. Hellendoorn. 2023.  CAT-LM: Training Language Models on Aligned Code And Tests.  arXiv:2310.01602 [cs.SE]
  * Rebuffi et al. (2021) Sylvestre-Alvise Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy A Mann. 2021.  Data augmentation can improve robustness.  _Advances in Neural Information Processing Systems_ 34 (2021), 29935–29948.
  * Rong et al. (2020) Yuyang Rong, Peng Chen, and Hao Chen. 2020.  Int egrity: Finding Integer Errors by Targeted Fuzzing. In _Security and Privacy in Communication Networks: 16th EAI International Conference, SecureComm 2020, Washington, DC, USA, October 21-23, 2020, Proceedings, Part I 16_. Springer, 360–380.
  * Rong et al. (2024a) Yuyang Rong, Zhanghan Yu, Zhenkai Weng, Stephen Neuendorffer, and Hao Chen. 2024a.  IRFuzzer: Specialized Fuzzing for LLVM Backend Code Generation.  arXiv:2402.05256 [cs.SE]
  * Rong et al. (2024b) Yuyang Rong, Chibin Zhang, Jianzhong Liu, and Hao Chen. 2024b.  Valkyrie: Improving fuzzing performance through deterministic techniques.  _Journal of Systems and Software_ 209 (2024), 111886.
  * Rozière et al. (2023) Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023.  Code Llama: Open Foundation Models for Code.  arXiv:2308.12950 [cs.CL]
  * Rubin (1987) Donald B. Rubin. 1987.  The Calculation of Posterior Distributions by Data Augmentation: Comment: A Noniterative Sampling/Importance Resampling Alternative to the Data Augmentation Algorithm for Creating a Few Imputations When Fractions of Missing Information Are Modest: The SIR Algorithm.  _J. Amer. Statist. Assoc._ 82, 398 (1987), 543–546.  http://www.jstor.org/stable/2289460
  * Runeson (2006) Per Runeson. 2006.  A survey of unit testing practices.  _IEEE software_ 23, 4 (2006), 22–29.
  * Schwartz et al. (2018) Eli Schwartz, Leonid Karlinsky, Joseph Shtok, Sivan Harary, Mattias Marder, Abhishek Kumar, Rogerio Feris, Raja Giryes, and Alex Bronstein. 2018.  Delta-encoder: an effective sample synthesis method for few-shot object recognition.  _Advances in neural information processing systems_ 31 (2018).
  * Sennrich et al. (2015) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015.  Improving neural machine translation models with monolingual data.  _arXiv preprint arXiv:1511.06709_ (2015).
  * Serebryany (2016) Kosta Serebryany. 2016.  Continuous fuzzing with libfuzzer and addresssanitizer. In _2016 IEEE Cybersecurity Development (SecDev)_. IEEE, 157–157.
  * Serebryany (2017a) Kostya Serebryany. 2017a.  OSS-Fuzz - Google’s continuous fuzzing service for open source software. USENIX Association, Vancouver, BC.
  * Serebryany (2017b) Kostya Serebryany. 2017b.  $\\{$OSS-Fuzz$\\}$-Google’s continuous fuzzing service for open source software.  (2017).
  * She et al. (2019) Dongdong She, Kexin Pei, Dave Epstein, Junfeng Yang, Baishakhi Ray, and Suman Jana. 2019.  Neuzz: Efficient fuzzing with neural program smoothing. In _2019 IEEE Symposium on Security and Privacy (SP)_. IEEE, 803–817.
  * Shorten et al. (2021) Connor Shorten, Taghi M Khoshgoftaar, and Borko Furht. 2021.  Text data augmentation for deep learning.  _Journal of big Data_ 8, 1 (2021), 101.
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ (2014).
  * Smith et al. (2003) Kevin D. Smith, Jim J. Jewett, Skip Montanaro, and Anthony Baxter. 2003.  _Decorators for Functions and Methods_.  PEP 318.  https://peps.python.org/pep-0318/
  * The rust-fuzz Project Developers (2024) The rust-fuzz Project Developers. 2024.  cargo-fuzz: A ‘cargo‘ subcommand for fuzzing with ‘libFuzzer‘! Easy to use!  {https://github.com/rust-fuzz/cargo-fuzz/}
  * Tufano et al. (2021) Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2021.  Unit Test Case Generation with Transformers and Focal Context.  arXiv:2009.05617 [cs.SE]
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Wadler (1990) Philip Wadler. 1990.  Linear types can change the world!. In _Programming concepts and methods_ , Vol. 3. Citeseer, 5.
  * Wang et al. (2021) Yue Wang, Weishi Wang, Shafiq Joty, and Steven C. H. Hoi. 2021.  CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation.  arXiv:2109.00859 [cs.CL]
  * Watson et al. (2020) Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, and Denys Poshyvanyk. 2020.  On learning meaningful assert statements for unit test cases. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_. ACM.  https://doi.org/10.1145/3377811.3380429
  * Wei and Zou (2019) Jason Wei and Kai Zou. 2019.  Eda: Easy data augmentation techniques for boosting performance on text classification tasks.  _arXiv preprint arXiv:1901.11196_ (2019).
  * Xie et al. (2020) Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. 2020.  Unsupervised data augmentation for consistency training.  _Advances in neural information processing systems_ 33 (2020), 6256–6268.
  * Yang et al. (2011) Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011.  Finding and understanding bugs in C compilers. In _Proceedings of the 32nd ACM SIGPLAN conference on Programming language design and implementation_. 283–294.
  * Yu et al. (2022) Shiwen Yu, Ting Wang, and Ji Wang. 2022.  Data augmentation by program transformation.  _Journal of Systems and Software_ 190 (2022), 111304.
  * Zeller et al. (2019) Andreas Zeller, Rahul Gopinath, Marcel Böhme, Gordon Fraser, and Christian Holler. 2019.  The fuzzing book.
  * Zhao et al. (2023) Jianyu Zhao, Yuyang Rong, Yiwen Guo, Yifeng He, and Hao Chen. 2023.  Understanding Programs by Exploiting (Fuzzing) Test Cases.  arXiv:2305.13592 [cs.LG]
  * Zheng et al. (2023) Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. 2023.  CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X.  arXiv:2303.17568 [cs.LG]
  * Zhu and Rubio-González (2023) Hao-Nan Zhu and Cindy Rubio-González. 2023.  On the reproducibility of software defect datasets. In _2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)_. IEEE, 2324–2335.