# Explainable Fuzzer Evaluation

Dylan Wolff1, Marcel Böhme2, Abhik Roychoudhury1  1National University of
Singapore. _{<EMAIL_ADDRESS>2MPI-SP & Monash University
<EMAIL_ADDRESS>
###### Abstract

While the aim of fuzzer evaluation is to establish fuzzer performance _in
general_ , an evaluation is always conducted on a _specific_ benchmark. In
this paper, we investigate the degree to which the benchmarking result depends
on the properties of the benchmark and propose a methodology to quantify the
impact of benchmark properties on the benchmarking result _in relation to_ the
impact of the choice of fuzzer. We found that the measured performance and
ranking of a fuzzer substantially depends on properties of the programs and
the seed corpora used during evaluation. For instance, if the benchmark
contained larger programs or seed corpora with a higher initial coverage,
AFL’s ranking would improve while LibFuzzer’s ranking would worsen. We
describe our methodology as _explainable fuzzer evaluation_ because it
explains _why_ the specific evaluation setup yields the observed superiority
or ranking of the fuzzers and how it might change for different benchmarks. We
envision that our analysis can be used to assess the degree to which
evaluation results are overfitted to the benchmark and to identify the
_specific conditions_ under which different fuzzers performs better than
others.

##  I Introduction

_Fuzzing_ [6] is a well-known automated software testing method for finding
bugs in computer programs. It generates invalid or unexpected inputs which
crash the program, thereby finding bugs and vulnerabilities.Fuzzing has become
one of the most successful automatic bug finding techniques in practice [24];
yet, which fuzzer performs best and when? Recently, the fuzzing community has
identified _sound fuzzer evaluation_ as a Top-3 most important research
challenge [6].

In 2021, 23 fuzzing papers were published in the Big Four Security conferences
[15]. To demonstrate improvement over the state-of-the-art many fuzzer
benchmarks have been introduced. For instance, the MAGMA benchmark [26]
provides a set of 138 real bugs in 9 programs. ProFuzzbench [40] offers access
to 11 protocol implementations for network-enabled fuzzers. The FuzzBench
benchmark [39] offers access to over 650 open-source programs. FuzzBench is
officially developed by Google demonstrating a substantial practical interest
in sound fuzzer evaluation.

However, the evaluation methodology recommended today [5, 33, 11] provides no
deeper insights into the conditions under which one fuzzer performs better
than another. Today, we consider a new fuzzer or fuzzer modification to be
successful if it performs better (covers more code) than the baseline on a
number of benchmark programs. Arcuri and Briand [5] introduced non-parametric
measures of _effect size_ and _statistical significance_ to assess (i) the
magnitude of the difference in performance between the proposed fuzzer and the
baseline and (ii) the degree to which this difference is due to randomness,
respectively. However, nothing is mentioned about the specificity of the
evaluation results to the properties of the given benchmark.

Using the traditional evaluation methodology— _on the average_ —most fuzzers
perform similarly while for each specific program there are often clear
winners. In a recent FuzzBench experiment involving 23 programs and 11
fuzzers, we can see that the average rank for the majority of fuzzers is
$5.5\pm 1$.111https://www.fuzzbench.com/reports/2022-04-19/index.html On
average, the majority of fuzzers rank approximately the same. In contrast, the
program-specific rankings are very diverse. Particularly, some fuzzers appear
to perform better (cover more branches than others) for the larger programs.
Can we report the conditions under which a fuzzer’s ranking changes?

The measure of effect size reported by benchmarking tools can become a target
during fuzzer development. However, when a measure becomes a target, it ceases
to be a good measure (Godhart’s Law); when a fuzzer is being developed against
a benchmark, there is a risk that the fuzzer’s heuristics are tuned to
specific properties of the benchmark. In general, the fuzzer may perform very
differently. For instance, FuzzBench has been instrumental in the rapid
improvement of Honggfuzz [46] and AFL++ [20]. In the past two years, more than
100 FuzzBench experiments have been executed to benchmark AFL++ [22]. However,
the focus on fuzzer ranking raised serious concerns about over-fitting the
performance of AFL++ to FuzzBench [19, 21]. To prevent this, we ask: Can we
report the impact on the evaluation results if properties of the benchmark
were changed? For instance, in addition to the concrete fuzzer ranking, can we
report how this ranking was affected if different programs were used?

In this paper, we study the impact of various properties of the benchmark on
the individual and relative performance of a fuzzer and propose a methodology
for _explainable fuzzer evaluation_ that measures the contribution of the
evaluated fuzzers to the benchmarking outcome _relative_ to the contributions
of various benchmark properties.

Impact on performance. We observe a _strong association_ between various
properties of the benchmark and the performance of a fuzzer (Section IV).
Given two benchmarks $A$ and $B$, such that the coverage of the initial
corpora used in $A$ is generally greater than those used in $B$, the same
fuzzer may perform better on $A$ than on $B$. The same holds if there are
differences between benchmarks $A$ and $B$ with respect to the number or
average size of the initial seeds or the size of the program. Other benchmark
properties that might reasonably be expected to influence fuzzer performance
did not turn out to do so. For instance, the proportion of (in)equalities in a
program have a negligible impact on fuzzer performance. Wherever an impact is
identified, researchers and benchmark builders should be careful to create a
benchmark that is sufficiently diverse with respect to these properties.

Impact on ranking. More importantly, we find a significant impact on the
benchmarking result, i.e., how fuzzers are ranked in terms of their coverage.
For instance, given two benchmarks $A$ and $B$, such that $A$ has generally
smaller programs than $B$, LibFuzzer’s rank is better on $A$ than on $B$ while
AFL++’s rank is better on $B$ than on $A$. In fact, among the four fuzzers we
evaluated, LibFuzzer’s ranking drastically degrades as any of the measured
corpus properties increases. AFL++’s ranking is positively impacted by an
increased execution time or seed size of the initial corpus. AFL’s ranking is
positively impacted mostly by an increased coverage of the initial corpus.
Entropic’s ranking is most robust. It does not seem to be as strongly affected
by any of the measured benchmark properties. Consequently, given the relevance
of these covariates, it is insufficient to report only the outcome of an
evaluation. Researchers should also assess the degree to which the specific
benchmark properties contribute to the evaluation result.

Explainable Evaluation. To evaluate the degree to which the evaluation result
is explained by differences of the fuzzers _in contrast_ to the properties of
the benchmark, we introduce _explainable fuzzer evaluation_. Our methodology
is based on a multiple linear regression of fuzzer ranks against the benchmark
properties where the choice of fuzzer is used as interaction term. The
multiple regression allows us to quantify the change in fuzzer rank depending
on the choice of fuzzer, of the specific benchmark properties (such as program
size, seed corpus side and initial coverage of seed corpus), and their
interaction. For example, we already know that LibFuzzer’s ranking is
negatively impacted by an increase in program size. Suppose, on the current
benchmark LibFuzzer is ranked first and AFL second. Our model can predict the
degree to which the benchmark must change (e.g., by choosing larger programs),
such that AFL’s rank improves to replace LibFuzzer. Using this technique, we
allow benchmark maintainers and consumers to both understand why some fuzzers
perform better or worse in certain contexts, and predict in which
circumstances one might deviate from the overall ranking of fuzzers across all
programs and corpora.

Contributions. This paper makes the following contributions.

  * •

We develop _explainable fuzzer evaluation_ , a methodology based on non-
parametric regression analysis to evaluate the degree to which the result of
an evaluation is actually explained by the choice of fuzzer rather than the
specific choice of benchmark.

  * •

We study the impact of the specific choice of benchmark and identify several
benchmark properties that significantly influence the performance of a fuzzer
and the ranking of several fuzzers in terms of code coverage.

  * •

We make concrete recommendations for contextual choices of fuzzer based off of
our findings.

Open Science. We publish the experimental infrastructure, the generated data,
and our analysis of that data at:

> https://figshare.com/s/2a06a4849ffc188f6a81

Upon acceptance, we will also publish an R library that implements our
analysis, for usage by the community.

##  II Challenges of Atomistic Evaluations

We describe an evaluation methodology as _atomistic_ if it merely quantifies
the superiority (or ranking) of each fuzzer w.r.t. one or more other fuzzers.
We describe an evaluation methodology as _holistic_ if it gives insights as to
_why_ one fuzzer performs better than another. The result of a holistic
evaluation can be considered _explainable_.

| _AFL_ | _AFL++_ | _EnLF_ | _LF_
---|---|---|---|---
_AFL_ |  | 0.27 | 0.41 | 0.85
_AFL++_ | 0.73 |  | 0.67 | 0.94
_EnLF_ | 0.59 | 0.33 |  | 0.94
_LF_ | 0.15 | 0.06 | 0.06 |
_Ranking_
---
1. | AFL++
2. | Entropic (EnLF)
3. | AFL
4. | LibFuzzer (LF)

(a) Outcome if started on LibFuzzer-generated seeds. _Ranking_ 1. Entropic
(EnLF) 2. LibFuzzer (LF) 3. AFL++ 4. AFL (b) Outcome if started on AFL-
generated seeds.

| _AFL_ | _AFL++_ | _EnLF_ | _LF_
---|---|---|---|---
_Effect Size_ ($\hat{A}_{12}$) | 0.0429 | 0.0800 | 0.1464 | 0.1800
_Statist. Sign._ ($U$) | 2.3e-09 | 3.7e-07 | 1.9e-05 |

(c) Fuzzer-specific performance differences if the evaluation was started with
AFL-generated seed corpora instead of LibFuzzer-generated ones.

Figure 1: Benchmarking outcome based on seed origin. Bold values indicate
statistical significance at $p<0.05$.

###  II-A The Perils of Benchmark Specificity

Suppose, we want to compare the performance of four fuzzers on a single
program222AFL [1], AFL++ [20], LibFuzzer [2], Entropic [8] on
lcms:https://github.com/google/fuzzbench/tree/master/benchmarks/lcms-2017-03-21.
Typically, we would run each fuzzer on the program for one day, starting with
an initial seed corpus, and measure the coverage achieved or the number of
bugs found. To tackle the high variance of the outcome of a fuzzing campaign,
we would repeat each experiment 25 times.

Section II.a shows the outcome of such an experiment. On this program, AFL++
performed best and LibFuzzer worst. The table shows the pair-wise effect size
measured using Vargha-Delaney’s $\hat{A}_{12}$ [5]. For instance, the
probability that a random AFL run outperforms a random LibFuzzer run is
$\hat{A}_{12}=85\%$. According to the Mann-Whitney $U$ test [5], this
difference is statistically significant.

However, the benchmarking outcome would be very different if we changed one
aspect of the setup. The initial seeds in the corpus were sampled from
testcases generated by LibFuzzer over the course of many previous fuzzing
campaigns.333A greybox fuzzer adds generated, cov.-increasing inputs to the
seed corpus. We hypothesize that using these LibFuzzer-generated seeds during
our evaluation might provide a disadvantage to LibFuzzer-based fuzzers
(including Entropic).

Section II.b shows the benchmarking outcome if AFL-generated seeds were
provided during the evaluation instead. Now Entropic performs best and AFL
performs worst and the probability that an arbitrary AFL run outperforms an
arbitrary LibFuzzer run is $\hat{A}_{12}<1\%$.

The choice of seed origin has a substantial impact on the benchmarking
outcome. Section II.c demonstrates this impact of LF-generated versus AFL-
generated initial seeds. All fuzzers perform worse on AFL-generated seeds
($\hat{A}_{12}<18\%$). Yet, the impact on AFL/AFL++ is greatest
($\hat{A}_{12}<8\%$): AFL-based fuzzers perform worse if provided with AFL-
generated seeds.

This motivating example illustrates the pitfalls of an atomistic fuzzer
evaluation. Quantitative measures on their own do not explain the
contributions of specific choices made in the benchmarking setup. Instead, one
should always consider the specific properties of the benchmark to explain the
results. In this paper, we aim to develop a general explainable evaluation
methodology that provides insight into the specificity of the result to the
concrete properties of the benchmark, where benchmark properties need not be
binary as in this example.

###  II-B Threats to Validity of Atomistic Evaluations

Overfitting is a _threat to external validity_ , the extent to which the
results of an evaluation can be generalized to other situations. If a fuzzer
is developed against performance measures on the same “stationary” benchmark,
the fuzzer is unintentionally optimized against the _specific_ properties of
the benchmark. Such _overfitting_ violates the objective of sound fuzzer
evaluation, i.e., to provide an evaluation of fuzzer performance _in general_.

_“[Atomistic evaluation] leads to gamification of a problem which largely
leads to people optimizing against a benchmark, not reality. This can lead to
people overfitting to problems, or making tooling which is so focused on a
single use case”
— _Brandon Falk on FuzzBench[19]_ _

Atomistic evaluation also a poses a _threat to internal validity_ , i.e., the
degree of confidence that our conclusion is not influenced by other factors or
variables. Consider the following blog-post [48] about the evaluation of the
recent proposal of gradient-descent-based fuzzing [13].

_“Since 32-bit magic byte values comprise all of the bugs artificially
injected into the LAVA-M benchmark, it does not surprise that [magic byte
extraction] dominates the performance instead of Gradient Descent. Of course,
this raises the question of why [gradient-decent-based fuzzer] is so
successful. Magic byte extraction is not a new technique, but in the case of
this benchmark [..] leads to spectacular results on LAVA-M. However, Gradient
Descent contributed little to that success.”
– _‘When Results Are All That Matters’; Zeller et al.[48]__

The authors of [13] hypothesize that gradient-descent-based search improves
fuzzer performance. They implement a gradient-descent-based search together
with related changes (e.g., a magic byte extraction) into a baseline fuzzer
and run both fuzzers on a well-known benchmark, LAVA-M [16]. However, [48]
argues, as above, that the conclusions drawn from such evaluation can be
problematic. Our explainable evaluation methodology can mitigate these issues.

##  III Background

Due the non-deterministic nature of fuzzing, each fuzzing campaign can perform
better or worse. A campaign of one fuzzer may outperform a campaign of another
fuzzer simply due to this randomness. To mitigate and quantify the impact of
randomness on the benchmarking result, several statistical tools and
experimental methodologies have been introduced in the literature [5, 11, 33].
In this section, we introduce the pertinent concepts of experimental design
and analysis.

Types of Variables. In empirical studies, we distinguish different kinds of
variables. A _response variable_ is the outcome of the experiment. In our
case, the response variables are (i) the individual fuzzer performance
measured in terms of the number of branches covered and (ii) the overall
fuzzer ranking w.r.t. fuzzer performance. An _explanatory variable_ is what we
vary (i.e., the treatment) while a _control variable_ is what is held
constant. In other words, changes in an explanatory variable explain changes
in the response variable while control variables can be excluded as causes of
a change in the response variable. In classical fuzzer benchmarking, the
proposed fuzzer heuristics are considered explanatory variables. A _covariate
variable_ may have an effect on the response variable but is not considered of
interest. A _confounding variable_ is an extra variable that is not measured
but has a hidden effect on the response variable. For the purposes this paper,
we turn specific benchmark properties from covariate or confounding variables
into explanatory variables and hope that our approach allows to study many
other covariates and confounders in the future.

Correlation measures the strength and the direction of the association between
two variables. For instance, a strong positive correlation between explanatory
and response variables $X$ and $Y$, respectively, indicates that an increase
in $X$ very likely results in an increase in $Y$. _Spearman’s rank
correlation_ is a concrete measure of the _ordinal association_ between two
variables. Measuring correlation between variable _ranks_ reduces the impact
of outliers and of monotonic non-linearity.

Effect size is the magnitude of the difference between two groups. For
instance, _Vargha-Delaney_ $\hat{A}_{12}$ is a non-parametric measure of
effect size that estimates the probability that randomly selected values $X$
from one population are greater than randomly selected values $Y$ from another
population. In our context, $X$ and $Y$ fuzzer performance values. Values
$\hat{A}_{12}>0.5$ identify Fuzzer $X$ as better. Values $\hat{A}_{12}\geq
0.71$ are considered a _large effect_ while values $\hat{A}_{12}\geq 0.64$ and
$\hat{A}_{12}\geq 0.56$ are considered _medium_ and _small effects_ ,
respectively. Vice versa for values $\hat{A}_{12}\leq 0.5$. As non-parametric
measure of effect size, Vargha-Delaney is the preferred method to evaluate the
effect size in fuzzer evaluation [5].

Hypothesis testing checks whether an observed difference is statistically
significant, i.e., whether the null-hypothesis can be rejected. The _null-
hypothesis_ is the assertion that there is _no difference_ between the two
populations $X$ and $Y$. To conduct a hypothesis test, the experimenter
specifies a _significance level_ $\alpha$, i.e., the probability of rejecting
the null-hypothesis when the null-hypothesis is true.

Statistical significance quantifies whether or not an observed difference is
due to chance. The _p-value_ is empirically determined as a measure of
significance, given the sample. For example, the _non-parametric Mann-Whitney_
$U$ test measures the probability (i.e., p-value) that for randomly selected
values $X$ and $Y$ from two populations, the probability of $X$ being greater
than $Y$ is equal to the probability of $Y$ being greater than $X$ [38, 5]. If
the p-value is below the specified significance level $\alpha$, the null-
hypothesis is _rejected_ , i.e., the result is statistically significant. A
$(1-\alpha)$-_confidence interval_ provides an interval within which the true
value lies with probability $1-\alpha$. An empirical confidence interval for
any statistic can be computed by _non-parametric bootstrapping_ , i.e., the
repeated re-sampling from the sample data [18]. The non-parametric bootstrap
can be used as an alternative approach to assess the statistical significance
of effects. If the $(1-\alpha)$-confidence interval does not contain the zero-
effect, the null-hypothesis is _rejected_ , i.e., the result is considered
statistically significant.

Experimental design is the process of carrying out research in an objective
and controlled fashion so that precision is maximized and specific conclusions
can be drawn regarding a hypothesis statement. In fuzzer evaluation, we are
interested in the effect of choosing one fuzzer instead of another (i.e., the
explanatory variable) on our measure of (relative) fuzzer performance (i.e.,
the response variable). In this paper, we are additionally concerned with the
effects of other factors (originating in the benchmarking setup) on the
evaluation result. The prototypical experimental design is to vary the fuzzer
(the proposed fuzzer versus the baseline fuzzer) while keeping everything else
constant.

##  IV Benchmark Impact on Fuzzer Performance

Program | Description
---|---
freetype2 | Font Renderer
harfbuzz | Text Shaping
libjpeg-turbo | JPEG Image Codec
libpcap | Network Packet Capture
(fuzz_both) |
libxslt (xpath) | XML Transformation
libpng | PNG Image Codec

Program | Description
---|---
mbedtls (dtlsclient) | Cryptographic
| Primitives
sqlite3 (ossfuzz) | Embedded Database
vorbis | Audio and Music
| Encoding
woff2 | Web Fonts
zlib (uncompress) | Decompression

Figure 2: Benchmark Programs

(a) Corpus properties.

(b) Program properties.

Figure 3: Impact of Benchmark Properties on our Measure of Fuzzer Performance.

###  IV-A Methodology

Benchmark. A _benchmark_ consists of a set of programs, a set of fuzzers, and
a measure of fuzzer performance. Any existing framework can be used [26, 40,
39]. We chose the FuzzBench [39] which uses _branch coverage_ as measure of
fuzzer performance. As _fuzzers_ , we chose AFL [1], AFL++ [20], LibFuzzer
[2], and Entropic [8] (which is based on LibFuzzer), representing the state-
of-the-art. As _programs_ , we chose all programs in FuzzBench (i) that were
already available in commit 8858be7, (ii) that could be compiled and run
within the local setup, and (iii) that had an OSS-Fuzz corpus available in
Fuzzbench. Our benchmark consists of 11 programs listed in Figure 2. For every
program, all experiments were conducted as FuzzBench local experiments on an
AWS c6i.metal instance.

Benchmark properties. To investigate the impact of covariates on the response
variables, we suggest to record properties that are suspected of affecting
fuzzer performance. For instance, Herrera et al. [27] identified several
properties of the initial seed corpus that can substantially impact fuzzer
performance. As _corpus properties_ , we measure the number of seeds, the
initial branch coverage, the mean execution time in nanoseconds, and the mean
size of the seeds in bytes. As _program properties_ , we measure the size of
the program, the proportion of equality and inequalities in comparison
instructions, and the proportion of shared library calls.444To measure the
number of shared library calls, we use objdump to count all calls to functions
marked as extern. As program size, we report the size of the text segment in
the program binary.555In addition, we measure _other_ benchmark properties
(cf. Section VII), including different properties of the distributions of the
benchmark properties like the quantiles of the execution times and size of
seeds in the initial corpus. However, we found a strong correlation of these
properties with other properties that we measure and include in our analysis.
To avoid problems with multicolinearity and overfitting, we excluded these
properties from the analysis and the multiple regression model.

Sampling configurations. Unlike in classic evaluation, we suggest to vary
properties of the benchmark for _each repetition_ of the experiment (i.e.,
trial). For every {fuzzer, program}-pair, we conducted 24 trials of 24 hours.
For each trial, we generate the initial seed corpus by randomly sampling from
a saturated seed corpus that was generated over many years by the OSS-Fuzz
continuous fuzzing platform [3].666We use the FuzzBench option \--oss-fuzz-
corpus. Thus on the $x$-trial ($x:1\leq x\leq 24$) of each program all fuzzers
start from the same initial corpus. Given a saturated corpus, we do _not_
recommend to choose sample size (#seeds) uniformly. Böhme and Falk [7]
observed that a linear increase in coverage requires an exponential increase
in the number of inputs. Hence, the average corpus, if sampled uniformly,
might still be saturated. Instead, we choose the sample size from an
exponential distribution centered at 20% of the original corpus.

Rank transformation. Upon completion, we rank each trial in terms of its
benchmark properties and performance measure. There are several reasons for
studying the _ordinal association_ rather than the numeric association between
benchmark property and performance measure. Firstly, we do not need to assume
that the distribution of the data itself is normal or that the relationship
between our explanatory and response variables is linear. In fuzzing we often
observe extreme outliers or exponential effects which provide undue influence
on a measure of the strength of the association (i.e., correlation). Secondly,
we do not require value domains to substantially overlap across programs. For
instance, we observe that a low initial coverage for one program can be a high
initial coverage for another. Hence, we report the rank of each trial _within_
a program (e.g., Figure 3.a) or across all programs (e.g., Figure 3.b) w.r.t.
benchmark property and fuzzer performance.

Rank transformations are used in non-parametric methods when values vary
widely in scale [34, 29, 38, 28, 14]—like in automatic software testing. For
instance, the Friedman test [29] describes a similar regression analysis
against the ranks. When value ranks are tied, we assign the average rank.

###  IV-B Presentation and Analysis

Analysis. To visually assess the relationship between benchmark property and
fuzzer performance, we propose a scatter plot of the ranks. To quantify the
direction and strength of this relationship, we suggest to compute Spearman’s
rank correlation ($\rho$). Figure 3.a shows the ordinal relationship between
the rank with respect to a corpus property of each program-specific trial on
the X-axis and the rank w.r.t. fuzzer performance of that trial on the y-axis.
The black dots show _average_ rank of a trial w.r.t. branch coverage across
programs and fuzzers for every rank of the corpus property. The grey dots show
the entire data for all fuzzers and programs. Figure 3.b shows the ordinal
relationship between each of the four program properties and the fuzzer
performance. Since we are interested in program properties, we rank across all
$24\times 11=264$ trials.

Results. For the corpus properties, we see a strong positive and linear
relationship for the ordinal association between the number of seeds as well
as the coverage ranks of the initial corpus and the final branch coverage that
a fuzzer achieves in one day ($\rho\in\\{0.74,0.78\\}$, resp.). As one would
expect, as the initial branch coverage of the initial seeds increases, fuzzer
performances increases, as well. Between the average size of a seed in the
initial corpus and the final coverage there exists an ordinal association.
However, it does not appear to be linear. Instead, we observe that the average
seed in the initial corpus should not be too large and not too small for a
fuzzer to achieve maximum branch coverage. This could be explained by fuzzer-
specific thresholds on the maximum seed size or by interactions with other
strong covariates, such as the initial coverage. Lastly, between the average
execution time of a seed in the initial corpus and fuzzer performance—if an
ordinal association exists, it is not very obvious.

For the program properties, we can see if there is a strong positive and
linear relationship for the ordinal association between each program property
and the fuzzer performance (cf. Figure 3.b). As the size of the program
increases, the final coverage increases, as well ($\rho=0.74$). It seems that
with more code available to be covered, fuzzers are able to reach more code
regions in a fixed amount of time. If regions range in their difficulty to
cover, having more regions available in the program means more regions that
are more easily reachable, assuming a fixed distribution of difficulty. From
the random scatter and the nearly horizontal linear regression lines we can
tell that there is no significant impact from the remaining program
properties, i.e., the proportion of (in)equalities in the program or the
proportion of calls to a shared library.

_There is a strong association between various benchmark properties on fuzzer
performance. Given two benchmarks $A$ and $B$, such that the coverage of the
initial corpora used in $A$ is generally greater than those used in $B$, the
same fuzzer may perform better on $A$ than on $B$. In addition to coverage, we
identify the number and average size of the seeds in the initial corpus as
critical variables to account for during fuzzer evaluation, as well as program
size._

Consequence. In general, our first methodology can be used to assess the
impact of numeric or ordinal benchmark properties on the _individual
performance_ of each tool or algorithm. Wherever an impact is identified,
researchers should ensure that their benchmark is sufficiently diverse w.r.t.
these properties. Our first methodology also helps quantify the difference
between the performance of a fuzzer on the benchmark and the performance of
that fuzzer in a specific use case (e.g., given a certain program and initial
corpus).

(a) Corpus properties.

(b) Program properties.

(c) Regression Slopes. (95%-CIs bootstrapped)

Figure 4: Linear regression between fuzzer ranks and trials ranks w.r.t.
benchmark properties.

##  V Benchmark Impact on Benchmarking Outcome

The _benchmarking outcome_ is the average ranking across all programs of all
fuzzers in terms of their performance. If we only had two fuzzers, we would
report the average superiority as outcome. We compute the ranking of each
fuzzer relative to each other fuzzer per trial (i.e. when started on the same
initial corpus for the same program). By regressing against the ranking of a
fuzzer for each trial, we are able to assess the impact of each property on
this ranking or superiority.

Analysis. To visually assess the relationship between benchmark property and
fuzzer ranking, we propose a simple scatter plot of the ranks but clearly
differentiate each fuzzer. Quantitatively, we propose to compute (i)
Spearman’s rank correlation ($\rho$) to assess the direction and strength of
the relationship, and (ii) the coefficients of the regression slopes to assess
the magnitude of the impact. A large regression slope indicates a large change
in ranking. To assess the variance of the regression coefficients, we suggest
bootstrapping the coefficient to estimate the 95%-confidence intervals around
the slope coefficient. Figures 4.a and 4.b show the impact of the four
measured corpus and program properties, respectively, on the ranking of each
fuzzer. Figure 4.c gives the regression coefficients for a linear regression
of the ranks of each property value against the ranking of each fuzzer.

Results. As we can see in Figure 4, the individual fuzzer rankings are visibly
impacted by the benchmark properties. For instance, LibFuzzer’s ranking
worsens as the coverage, average size and average execution time of the seeds
in the initial corpus increases, or as the size of the program increases.
Meanwhile, AFL’s ranking improves as coverage and average size of seeds in the
initial corpus increases. Similarly, AFL++’s ranking improves on benchmarks
with larger programs or where the initial seed corpus executes faster. For
Entropic, we cannot identify a statistically significant impact of any
benchmark property on its ranking. In Figure 4.c, we consider the effect as
statistically significant whenever the 95%-confidence intervals do no include
the origin.

_As program size increases, LibFuzzer’s ranking worsens while AFL++’s
improves. A ten-rank increase in program size (a difference in ten positions
when the programs are sorted from smallest to largest) boosts AFL++’s relative
performance by one rank but degrades LibFuzzer’s by one rank. LibFuzzer’s
ranking drastically degrades as any of the measured corpus properties
increases. AFL++’s ranking is positively impacted by an increase in execution
time or seed size. AFL’s ranking is positively impacted by the coverage of
initial corpus. Entropic’s ranking is relatively robust to changes in
properties of the benchmark._

Consequence. The outcome of a fuzzer evaluation (in terms of effect size)
_cannot_ only be explained by the ground-truth, general, underlying relative
performance of each fuzzer. The outcome is also partially explained by the
specific choice of benchmark. This means, it is insufficient to report only
the outcome of an evaluation, as in an atomistic evaluation, even if the
specific properties of a benchmark are mentioned. Researchers should also
assess the degree to which the specific benchmark properties contribute to the
evaluation result.

$R^{2}=0.65(0.65Adj.)$, _p-val._ $<0.001$
Median Residuals: $0.045$

Figure 5: Multiple Linear Regression with LibFuzzer as reference level (Fuzzer
Ranks $\sim$ Fuzzer $\times$ Benchmark Properties).

##  VI Combined Impact on Benchmarking Outcome

Our first methodology explored the impact of a benchmark property on the
individual fuzzer performance. Our second methodology explored the impact of a
benchmark property on the relative performance of a specific fuzzer among a
group of fuzzers. The following integrates the two analyses into a combined
analysis. In particular, we propose to model the combined impact of the
pertinent benchmark properties and of the fuzzers on the outcome of the
evaluation.

###  VI-A Analysis

Multiple Linear Regression (MLR). To quantify the combined effect of benchmark
properties and fuzzers on benchmarking outcome, we proprose to use multiple
linear regression where the set of fuzzers is added as categorical interaction
term. A _multiple linear regression_ is a statistical technique that uses
several explanatory variables to predict the outcome of a response variable.
We present our model below:

$\displaystyle R$ $\displaystyle=\alpha+\left[\sum_{p\in
P}\beta_{p}X_{p}\right]+\left[\sum_{f\in
F}\gamma_{f}Y_{f}\right]+\left[\sum_{p\in P}\sum_{f\in
F}\omega_{p,f}X_{p}Y_{f}\right]$

where $R$ is the fuzzer ranking, $\alpha$ is the intercept, $P$ and $F$ are
the benchmark properties and the set of fuzzers, respectively, $X_{p}$ are the
benchmark property ranks relative to the reference level, $Y_{f}\in\\{0,1\\}$
are indicator variables such that $Y_{f}=1$ if fuzzer $f$ was used instead of
the reference fuzzer, and $\beta_{j}$, $\omega_{pf}$, and $\gamma_{f}$ are the
coefficients (and subject to estimation).

Interpretation. A _regression coefficient_ can be considered as the degree to
which the corresponding explanatory variable contributes to the response
variable. For instance, if $\beta_{p}>0$, we say that an increase in $X_{p}$
by one unit _from a reference level_ gives an increase in fuzzer rank by
$\beta_{p}$. The _reference level_ is a specific benchmark configuration and
selection of fuzzer. In our case, the reference levels for all corpus
properties are set to the lowest rank while the reference level for program
size is set to the median rank. As reference fuzzer, we choose LibFuzzer. The
_intercept_ $\alpha$ is precisely the rank of the reference fuzzer at the
reference level across all benchmark properties. A coefficient $\omega_{pf}$
of an interaction term describes the contribution of the benchmark property
$X_{p}$ if the fuzzer $f$ was used instead of the reference fuzzer (here,
LibFuzzer).

Model diagnostics. The _goodness of fit_ of the regression model describes how
well it fits the set of observations. We can measure goodness of fit using the
coefficient of determination $R^{2}$, i.e., the proportion of the variation in
the fuzzer ranking that is predictable from the benchmark properties and their
interaction with the fuzzers. To assess the _statistical significance_ of the
effects captured by the regression model, we suggest to compute the 95%
confidence intervals around the regression coefficients using non-parametric
bootstrapping.

###  VI-B Results

Presentation. Figure 5 shows the coefficients as point estimates and
95%-confidence intervals (CIs). If the CI includes the origin (0), there may
be no impact from the corresponding benchmark property relative to the
reference levels. Figure 5.left shows the change in fuzzer rank if a different
fuzzer is chosen and all other covariates are held constant at the reference
level. The first four whiskers in Figure 5.right show the change in fuzzer
rank if one of the four considered properties increase by one rank and the
fuzzer is held constant (LibFuzzer). The remaining whiskers show the
_additional_ change in fuzzer rank if both fuzzer and benchmark property are
changed simultaneously (as interaction terms).

Absolute effect. Confirming earlier results from Figure 4, LibFuzzer (our
reference level) is negatively impacted by the coverage and mean execution
time of the seeds in the initial corpus and by size of the program. Within the
MLR model, the impact of mean seed size seems negligible. As expected, the
choice of fuzzer has a substantial impact on the fuzzer ranking: Keeping all
else constant, choosing AFL would worsen the ranking while choosing AFL++ or
Entropic would improve it.

Relative effect. Keeping all other covariates constant at their reference
levels, a switch from LibFuzzer to AFL is roughly equivalent to increasing the
benchmark’s program size by five ranks.777Increasing program size by five
ranks means to choose the program five positions to the right if programs are
sorted from smallest to largest. From Figure 4, we already know that LibFuzzer
is substantially negatively impacted by program size. However, our explainable
fuzzer evaluation methodology allows us to put the negative impact of a
benchmark property _in relation_ to the impact of the choice of fuzzer on
benchmarking outcome.

Interaction. From Figure 5, we can also observe a substantial positive impact
if we chose AFL++ _and_ larger programs. For an increase in program size by
$X$ ranks, the fuzzer ranking would change by $-1.1+0.12X-0.22X$. Referencing
the regression model based equation for fuzzer ranking we presented, the
coefficients correspond to $\gamma_{f}$ (choice of AFL++), $\beta_{p}$
(increase in program size), and $\omega_{f,p}$ (increase in program size for
AFL++, _specifically_).

_Our explainable fuzzer evaluation methodology facilitates objective
assessment of absolute, relative, and interaction effects among benchmark
properties and the choice of fuzzer. Our multiple linear regression model can
predict the best fuzzer for a given setup by modelling the interaction effect
of choosing a different fuzzer and varying the considered property._

Figure 6: Correlation matrix (Spearman’s $\rho$).

Joint effects. Another advantage of our model is the ability to quantify joint
effects among the explanatory variables. For instance, the mean seed size (red
whiskers) seems to have no interactive effect between each of the other
fuzzers and LibFuzzer. Thus, the correlation between the mean seed size and
the response variables that we have observed in Sections IV and V cannot be
separated from the effects of execution time and initial coverage. In other
words, there is a significant joint effect from the seed size or the execution
time together, but no significant individual effect can be separated from
either in our experiments.

Per-benchmark interactions and model fit. We also observed that the effects of
each of our covariates are significantly different across benchmarks. For
example, AFL++ has a significantly improved rank relative to Entropic as
initial coverage increases on libjpg, but a significantly worse relative rank
as it increases on libpcap. Indeed, by changing our model described in Section
VI-A, to include each variable as an interaction with benchmark (i.e.
Benchmark $\times$ Fuzzer $\times$ Properties ), we are able to improve model
fit considerably to an $R^{2}$ of 0.811. In other words, the coefficients
depicted in Figure 5 give us a sense of the net effect of the corpus
properties across programs, but individual effects vary significantly. Thus it
is crucial for benchmarking frameworks to maintain a large, representative
sample of target programs to avoid systematic bias with respect to program and
corpus properties.

In summary, we still see significant relative effects from the initial
coverage and program size, as well as jointly from the mean seed size and
execution times. However, the corpus-property effects are relatively small.
Still, large variances in the seed size, execution time, and initial coverage
appear to be enough to change fuzzer rankings in practice. In terms of the
benchmark itself, the program size is associated with a much larger effect on
fuzzer rankings. In light of these findings, we can begin to make some general
recommendations about which fuzzers to use under which circumstances. We
recommend the use of AFL++ and Entropic and AFL over LibFuzzer for medium and
larger programs. However, on smaller programs with lower initial coverage and
smaller or faster seeds, this gap in performance (coverage achieved) narrows
significantly, and Entropic performs best, followed by AFL++ and LibFuzzer.

###  VI-C Assumptions

To apply multiple linear regression, we require five assumptions to be met:
(i) a linear association between explanatory and response variables
[_linearity_], (ii) no high correlation among explanatory variables [_no
multicolinearity_], (iii) a constant variance of residuals
[_homoscedasticity_], (iv) normality of the residuals [_normality_] and (v)
independence of observations [_independence_].

Figure 7: Model Diagnostics.

Linearity. For a multiple linear regression model to be accurate, the
relationship between the explanatory and response variables must be correctly
modeled with a linear formula. If the relationships between the model and the
data are different, this can lead to a poorly fit model that does not
generalize well to new data. When operating on data from fuzzing runs, we
often see non-linearity [7], however, by using the rank transformation, we
eliminate monotonic non-linearity from the model. Linearity can be visually
assessed by plotting the individual predictors against the response variable.
From Figure 4, we observe that all of our ranked predictors appear to be
linear with respect to the fuzzer ranking. So, we consider the linearity
assumption to be satisfied.

Multicolinearity. In applied statistics, we always prefer simple models over
more complex models if the explanatory predictive power can be maintained. If
some explanatory variables are highly correlated, it becomes difficult to
interpret the model as the join effect can be arbitrarily spread among
correlated predictors. In Figure 6, we can only see one association that is
not weak or negligible, i.e., a very strong correlation between the size and
coverage of the initial seed corpus. As a result, we decided to exclude corpus
size from our model. For reasons of model parsimony, we also excluded the
proportion of equalities, inequalities, and calls to shared library from the
multiple linear regression model. These program properties have no discernible
independent impact on individual or relative performance of our fuzzers (cf.
Sections IV & V).

Normality. To determine statistical significance of regression coefficients
analytically, the residual errors of a linear regression model must be
approximately normally distributed. Using Figure 7.top-right, we can visually
inspect the quantiles of the residuals of our data plotted against the
quantiles of a normal distribution. We can see that the plots are nearly
completely aligned, indicating that our data is, indeed approximately normally
distributed. However, as noted by Arcuri and Briand [5], the normality
assumption is often violated in software engineering contexts because the data
itself is not normally distributed. This can be addressed by bootstrapping as
a method for computing non-parametric confidence intervals in fuzzing and
other software engineering contexts. By repeatedly regressing against sub-
sampled data from our initial sample, we are able to obtain confidence
intervals computationally, rather than analytically (avoiding the normality
assumption).

Homoscedacity. Linear regression models assume that the variance of the
residuals is constant over the predicted values of the response variable. We
can assess homoscedacity by examining the residual plot or the scale-location
plot of the square root of the standardized residuals against the fitted
values. In the latter, we should see a flat line if our data is homoscedastic.
However, Figure 7.left shows that this might not be the case for our data.
There seems to be greater variance around the middle ranks, likely due to ties
in fuzzer performance (which we break by assigning the average rank). We
therefore use the _wild bootstrap_ to reduce the impact of heteroscedasticity
on our confidence intervals [47] [36]. Intuitively, the wild bootstrap
repeatedly resamples the residuals of a linear model fitted to the original
and scales them by a random variable to give additional data-points without
making assumptions that the residuals are similarly distributed.

Independence. Regression models also assume that the individual samples of
data are not dependent on each other. We use Fuzzbench for our testing setup,
which should not introduce any dependence between trials. We validate the
independence of the residuals of our data with the Durbin-Watts test [17],
obtaining a p-value of $0.794$.

##  VII Threats to Validity

Like other empirical studies, ours has several potential threats to validity
which we have attempted to mitigate.

Threats to internal validity are aspects of our study that may introduce
systemic bias. We minimize experimenter and confirmation bias in our
experiment design by utilizing Fuzzbench, an existing benchmarking tool
prepared by independent practitioners and researchers, for our experimental
setup. We reduced the chance of selection bias in our choice of fuzzers by
only choosing from state-of-the-art fuzzers with widespread adoption.
Additionally, because we are not attempting to make claims about the
performance of any particular fuzzer but rather potential covariates in
benchmarking runs, impact of selection bias is reduced. Another possible
source of bias was our use of AFL++’s corpus minimization tool to reduce our
seed pool for each experiment. For example, based on differences in
instrumentation, AFL-based corpus minimization tools might retain files that
other instrumentations are completely blind to, advantaging these other
implementations. The converse could also be true, with AFL-based minimization
removing files that other tools might keep, disadvantaging AFL-based fuzzers.
We mitigate this risk by only reducing with respect to edge-coverage, which we
believe to be an uncontroversial criteria for corpus reduction across grey-box
instrumentation tools. We also use a recent version of the AFL++ coverage
instrumentation, which is collision-free to avoid any bias with respect to
AFL-based minimization tools being blind to hash-colliding inputs. Alternative
tools using LLVM coverage data were orders of magnitude slower, to the point
that they are not practical for fuzzing practitioners. Additionally, prior
work [27] has outlined the importance of corpus reduction for fuzzing
effectiveness, and thus running experiments without minimized corpora would
have threatened the external validity of our study.

Threats to external validity could harm the generality of these results beyond
the scope of our study. We attempt to minimize this risk with respect to our
benchmark programs by utilizing programs from the Fuzzbench benchmark suite,
which are among the most broadly representative of open source software
available to the fuzzing community [39]. Additionally, we follow the
recommendation of [11], that $\geq$10 benchmarks should be sufficient for
coverage-based benchmarking, spending over 1000 CPU-hours gathering data for
our analysis. We also make our data and experimental setup available for
replication. We attempt to minimize external validity risk with respect to our
fuzzers by choosing what we believe to be a representative set of state-of-
the-art fuzzers. AFL and LibFuzzer are the baseline implementations for 10/11
state-of-the-art fuzzers supported in Fuzzbench’s main experiment. AFL++ and
Entropic are among the latest and highest performing variants of our two
baseline implementations in recent benchmarking runs.

Threats to construct validity concern whether or not the data collected in our
study measure what we claim. In this study, we are examining the impact of
various variables on “fuzzer effectiveness”. Our metric for fuzzer
effectiveness is code coverage, which is the default measure of performance in
Fuzzbench. While code-coverage has long been used as proxy for fuzzer
performance, there is a risk that our results on coverage based benchmarking
do not translate to bug-finding effectiveness. However [11] found that
coverage based benchmarking is correlated with bug finding ability.
Additionally, because of the sparsity of bugs, using even the largest bug-
based benchmarks available (around 200 bugs) as our metric for fuzzer
effectiveness would threaten external validity of our study.

Threats to conclusion validity may lead to misinterpretations of our data. We
mitigate the risk by checking the assumptions of each technique used. In the
case of our regression models, we also use the non-parametric wild bootstrap
[47] to obtain confidence intervals, rather than analytical methods dependent
on the distribution of the residuals. Another potential threat to conclusion
validity is our use of the rank transformation. We did so to reduce the impact
of outliers, scale the data for each benchmark such that it is comparable, and
eliminate the impact of any monotonic non-linearity on our regression
analysis. Indeed, without the rank transformation, our data appears to be non-
linear with no obvious higher order pattern, in addition to having several
strong outliers, and thus would not be amenable to regression analysis. While
the rank transformation is a common statistical technique for non-parametric
analysis [34, 29, 38, 28, 14], this transformation introduces a layer of
indirection to our analysis; where we report correlations and trends in the
ranked data, and not the underlying raw data.

##  VIII Related Work

In fuzzing and in other disciplines of computer science, the impact of the
particular evaluation setup on the evaluation outcome has previously been
noted, but—to the best of our knowledge—not systematically studied. We could
not find other methodologies that _account_ for or quantify the impact of the
benchmark on the benchmarking outcome _during_ the evaluation. For instance,
in machine learning (ML), Japkowicz [31] criticizes that the sole focus on
performance measures in ML research might obscure important behaviors of the
hypotheses or algorithms under consideration. In computer architecture, Panda
et al. [41] study the characteristics of the the SPEC2017 CPU benchmark suite
primarily to identify a _subset_ of benchmark programs and inputs that can
approximate the results of the _entire_ benchmark suite. In automatic software
testing, Herrera et al. [27] found that the properties of the seed corpus had
a substantial impact on the performance of a fuzzer. While all of these works
acknowledge an impact of the choice of benchmark or analysis methodology on
the outcome of the evaluation, there is no guidance on how to account for the
observed effects in an evaluation in practice.

In terms of guidelines for the evaluation of automatic software testing tools
(i.e., fuzzers), most recommendations are concerned with the sound statistical
analysis of effect size and statistical significance. Arcuri and Briand [4]
[5] provide an in-depth discussion of statistical, non-parametric measures
that are particularly applicable in the context of automatic software testing.
Klees et al. [33] later drew on these guidelines to identify specific issues
with experimental design for fuzzer evaluations, such as lack of repetition,
too few target programs, and lack of consideration for the impact of initial
corpora. However, none of the previous guidelines provide clear steps for an
evaluation methodology that incorporates these covariates, as presented in
this paper.

Several works have emerged establishing benchmarks for fuzzing tools both
generally, as in the case of Fuzzbench [39] or UNIFUZZ [35], and for specific
classes of applications such ProFuzzBench [40]. Magma [26] focuses on a suite
of real-world bugs. Other projects have gathered data-sets of real-world bugs
for the evaluation of various testing and debugging strategies [32] [25]. All
of these works use methodology similar to those described by Klees et al. [33]
or Arcuri and Briand [5], however, neglecting the influence of additional
variables on testing outcomes. Synthetic benchmarks [16] [43] [44] [45]
provide an alternative path for more explainable benchmarking. For example,
Zhu et al. [49], propose creating artificial target programs with large
numbers of features known or suspected to be problematic for fuzzers. They
compare relative performance of two fuzzers on one such program in terms of
artificial “bugs” found, observing a difference in behavior between AFL and
AFLFast [9]. This approach, however, cannot account for different effects from
the seed corpora for such artifical programs. Additionally, there remain
doubts as to whether results from such synthetic benchmarks are representative
of behavior in typical real-world programs [12] [23] [10]. Recent work by Lyu
et al. [37] suggests program senstive energy allocation procedure to determine
the power schedule of a fuzzer.

While our study focused on coverage as the primary metric for fuzzer
performance, we expect program and corpus properties to have impacts beyond
increased coverage. Another metric commonly used to evaluate testing tools is
mutant injection [42]. The ground truth for testing effectiveness is bug-
finding ability. Inozemtseva et al [30] first investigated the correlation
between code coverage and bugs found by Java unit testing suites. Böhme et al
[11] more recently found coverage to be a good objective function for fuzzers,
given the sparsity of real-world bugs. We believe there are future research
opportunities in assessing the impact of covariates with respect to these
alternative metrics.

##  IX Perspective

In this paper, we proposed explainable fuzzer evaluation, as opposed to
evaluation on specific programs and seed corpus. We introduced several
methodologies for observing trends on the ordinal effects of covariates giving
a holistic, rather than atomistic, perspective on fuzzer performance. Our
study of program and corpus properties found several examples of statistically
significant effects not accounted for in current evaluations. The seed origin,
seed size and execution time, initial coverage, and program size all impact
ranking of a specific fuzzer in terms of coverage. Based on the results from
our study, we share the following view-points.

  1. 1.

For entire fuzzing community: Sample starting corpora from a large pool of
seeds with varied characteristics and origins, or create initial corpora with
varying levels of coverage, including extremes (e.g. empty, valid, saturated
etc.). This reduces the chance that the benchmark will be biased by reporting
results for only a single corpus.

  2. 2.

For fuzzing researchers and fuzzer builders: Use explainable benchmarking
technique as presented in this paper to account for the effects of covariates
during benchmarking. Additionally, benchmarks should be diverse with respect
to the properties observed to impact fuzzer rankings.

  3. 3.

For practitioners or fuzzer users: There is a dichotomy between LibFuzzer and
the other fuzzers we used in our experiments, with LibFuzzer’s coverage
degrading for slower and larger seeds, as well as for larger programs. This
can inspire future research on improving LibFuzzer, so that the subtle bug-
detection via built-in support for sanitizers that users can get out of
LibFuzzer, does not need to come at the cost of reduced behavioral coverage.

## References

  * [1] “American fuzzy lop (AFL),” https://github.com/google/AFL.
  * [2] “libfuzzer,” https://llvm.org/docs/LibFuzzer.html.
  * [3] _OSS-Fuzz: Google’s Continuous Fuzzing Service for Open-Source Software_. Vancouver, BC: USENIX Association, Aug 2017.
  * [4] A. Arcuri and L. Briand, “A practical guide for using statistical tests to assess randomized algorithms in software engineering,” in _2011 33rd international conference on software engineering (ICSE)_. IEEE, 2011, pp. 1–10.
  * [5] A. Arcuri and L. Briand, “A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering,” _Software Testing, Verification and Reliability_ , vol. 24, no. 3, pp. 219–250, 2014. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/stvr.1486
  * [6] M. Böhme, C. Cadar, and A. Roychoudhury, “Fuzzing: Challenges and reflections,” _IEEE Software_ , vol. 38, no. 3, 2021.
  * [7] M. Böhme and B. Falk, “Fuzzing: On the exponential cost of vulnerability discovery,” in _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2020, pp. 713–724.
  * [8] M. Böhme, V. J. Manès, and S. K. Cha, “Boosting fuzzer efficiency: An information theoretic perspective,” in _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2020, pp. 678–689.
  * [9] M. Böhme, V.-T. Pham, and A. Roychoudhury, “Coverage-based greybox fuzzing as markov chain,” in _ACM SIGSAC Conference on Computer and Communications Security (CCS)_ , 2016.
  * [10] M. Böhme and A. Roychoudhury, “Corebench: Studying complexity of regression errors,” in _Proceedings of the 2014 International Symposium on Software Testing and Analysis_ , ser. ISSTA 2014. New York, NY, USA: Association for Computing Machinery, 2014, p. 105–115. [Online]. Available: https://doi.org/10.1145/2610384.2628058
  * [11] M. Böhme, L. Szekeres, and J. Metzman, “On the reliability of coverage-based fuzzer benchmarking,” in _Proceedings of the 44th International Conference on Software Engineering (ICSE)_ , 2022.
  * [12] J. Bundt, A. Fasano, B. Dolan-Gavitt, W. Robertson, and T. Leek, “Evaluating synthetic bugs,” in _Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security (AsiaCCS)_. Association for Computing Machinery, 2021, p. 716–730.
  * [13] P. Chen and H. Chen, “Angora: Efficient fuzzing by principled search,” in _IEEE Symposium on Security and Privacy (S & P)_, 2018.
  * [14] W. J. Conover and R. L. Iman, “Rank transformations as a bridge between parametric and nonparametric statistics,” _The American Statistician_ , vol. 35, no. 3, pp. 124–129, 1981.
  * [15] CPUU, “Awesome fuzzing,” https://github.com/cpuu/awesome-fuzzing, accessed: 2022-08-12.
  * [16] B. Dolan-Gavitt, P. Hulin, E. Kirda, T. Leek, A. Mambretti, W. Robertson, F. Ulrich, and R. Whelan, “Lava: Large-scale automated vulnerability addition,” in _2016 IEEE Symposium on Security and Privacy (S &P)_, 2016, pp. 110–121.
  * [17] J. Durbin and G. S. Watson, “Testing for serial correlation in least squares regression. iii,” _Biometrika_ , vol. 58, no. 1, pp. 1–19, 1971.
  * [18] B. Efron, “Better bootstrap confidence intervals,” _Journal of the American Statistical Association_ , vol. 82, no. 397, pp. 171–185, 1987.
  * [19] B. Falk, “Let’s talk a bit about fuzzer benchmarking,” https://twitter.com/gamozolabs/status/1292324287324381184, accessed: 2022-08-12.
  * [20] A. Fioraldi, D. Maier, H. Eißfeldt, and M. Heuse, “Afl++: Combining incremental steps of fuzzing research,” in _Proceedings of the 14th USENIX Conference on Offensive Technologies_ , ser. WOOT’20. USA: USENIX Association, 2020.
  * [21] Fuzzbench, “Github issue discussing overfitting in fuzzbench,” https://github.com/google/fuzzbench/issues/654, accessed: 2022-08-12.
  * [22] Fuzzbench, “History of fuzzbench experiments requested,” https://github.com/google/fuzzbench/blob/master/service/experiment-requests.yaml, accessed: 2022-08-12.
  * [23] S. Geng, Y. Li, Y. Du, J. Xu, Y. Liu, and B. Mao, “An empirical study on benchmarks of artificial software vulnerabilities,” _CoRR_ , vol. abs/2003.09561, 2020. [Online]. Available: https://arxiv.org/abs/2003.09561
  * [24] Google, “Oss-fuzz: Continuous fuzzing for open source software,” https://github.com/google/oss-fuzz#trophies, accessed: 2022-08-12.
  * [25] P. t. Gyimesi, B. Vancsics, A. Stocco, D. Mazinanian, Á. Beszédes, R. Ferenc, and A. Mesbah, “Bugsjs: a benchmark of javascript bugs,” in _2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST)_ , 2019, pp. 90–101.
  * [26] A. Hazimeh, A. Herrera, and M. Payer, “Magma: A ground-truth fuzzing benchmark,” _Proceedings of the ACM on Measurement and Analysis of Computing Systems_ , vol. 4, no. 3, pp. 1–29, 2020.
  * [27] A. Herrera, H. Gunadi, S. Magrath, M. Norrish, M. Payer, and A. L. Hosking, “Seed selection for successful fuzzing,” in _Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA)_. Association for Computing Machinery, 2021.
  * [28] T. Hettmansperger and J. McKean, _Robust Nonparametric Statistical Methods_. CRC Press, 2010. [Online]. Available: https://doi.org/10.1201/b10451
  * [29] R. L. Iman and W. J. Conover, “The use of the rank transform in regression,” _Technometrics_ , vol. 21, no. 4, pp. 499–509, 1979.
  * [30] L. Inozemtseva and R. Holmes, “Coverage is not strongly correlated with test suite effectiveness,” in _Proceedings of the 36th international conference on software engineering_ , 2014, pp. 435–445.
  * [31] N. Japkowicz, “Why question machine learning evaluation methods,” in _Proceedings of the 2006 AAAI (Workshop in Evaluation Methods for Machine Learning)_ , 2006.
  * [32] R. Just, D. Jalali, and M. D. Ernst, “Defects4j: A database of existing faults to enable controlled testing studies for java programs,” in _Proceedings of the 2014 International Symposium on Software Testing and Analysis_ , 2014, pp. 437–440.
  * [33] G. Klees, A. Ruef, B. Cooper, S. Wei, and M. Hicks, “Evaluating fuzz testing,” in _Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security_ , 2018, pp. 2123–2138.
  * [34] W. H. Kruskal and W. A. Wallis, “Use of ranks in one-criterion variance analysis,” _Journal of the American Statistical Association_ , vol. 47, no. 260, pp. 583–621, 1952.
  * [35] Y. Li, S. Ji, Y. Chen, S. Liang, W.-H. Lee, Y. Chen, C. Lyu, C. Wu, R. Beyah, P. Cheng _et al._ , “Unifuzz: A holistic and pragmatic metrics-driven platform for evaluating fuzzers.” in _USENIX Security Symposium_ , 2021, pp. 2777–2794.
  * [36] R. Y. Liu, “Bootstrap procedures under some non-iid models,” _The annals of statistics_ , vol. 16, no. 4, pp. 1696–1708, 1988.
  * [37] C. Lyu, H. Liang, S. Ji, X. Zhang, B. Zhao, M. Han, Y. Li, Z. Wang, W. Wang, and R. Beyah, “Slime: Program-sensitive energy allocation for fuzzing,” in _31st ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA)_ , 2022.
  * [38] H. B. Mann and D. R. Whitney, “On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other,” _The Annals of Mathematical Statistics_ , vol. 18, no. 1, pp. 50 – 60, 1947.
  * [39] J. Metzman, L. Szekeres, L. Simon, R. Sprabery, and A. Arya, “Fuzzbench: an open fuzzer benchmarking platform and service,” in _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2021, pp. 1393–1403.
  * [40] R. Natella and V.-T. Pham, “Profuzzbench: A benchmark for stateful protocol fuzzing,” in _Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis_ , 2021.
  * [41] R. Panda, S. Song, J. Dean, and L. K. John, “Wait of a decade: Did spec cpu 2017 broaden the performance horizon?” in _2018 IEEE International Symposium on High Performance Computer Architecture (HPCA)_. IEEE, 2018, pp. 271–282.
  * [42] M. Papadakis, M. Kintis, J. Zhang, Y. Jia, Y. Le Traon, and M. Harman, “Mutation testing advances: an analysis and survey,” in _Advances in Computers_. Elsevier, 2019, vol. 112, pp. 275–378.
  * [43] J. Patra and M. Pradel, “Semantic bug seeding: A learning-based approach for creating realistic bugs,” in _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC-FSE)_. New York, NY, USA: Association for Computing Machinery, 2021.
  * [44] J. Pewny and T. Holz, “Evilcoder: Automated bug insertion,” in _Proceedings of the 32nd Annual Conference on Computer Security Applications (ACSAC)_. New York, NY, USA: Association for Computing Machinery, 2016, p. 214–225. [Online]. Available: https://doi.org/10.1145/2991079.2991103
  * [45] S. Roy, A. Pandey, B. Dolan-Gavitt, and Y. Hu, “Bug synthesis: Challenging bug-finding tools with deep faults,” in _Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , ser. ESEC/FSE 2018. New York, NY, USA: Association for Computing Machinery, 2018, p. 224–234. [Online]. Available: https://doi.org/10.1145/3236024.3236084
  * [46] R. Swiecki, “Improving honggfuzz for fuzzbench, part 1,” https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/fuzzing-discuss/9bzAZs_BMUA, accessed: 2022-08-12.
  * [47] C.-F. J. Wu, “Jackknife, bootstrap and other resampling methods in regression analysis,” _the Annals of Statistics_ , vol. 14, no. 4, pp. 1261–1295, 1986\.
  * [48] A. Zeller, S. Just, and K. Greshake, “When results are all that matters,” https://andreas-zeller.info/2019/10/10/when-results-are-all-that-matters-case.html, accessed: 2022-08-12.
  * [49] X. Zhu, X. Feng, T. Jiao, S. Wen, Y. Xiang, S. Camtepe, and J. Xue, “A feature-oriented corpus for understanding, evaluating and improving fuzz testing,” in _Proceedings of the 2019 ACM Asia Conference on Computer and Communications Security_ , 2019, pp. 658–663.