# White-box Compiler Fuzzing
Empowered by Large Language Models

Chenyuan Yang<EMAIL_ADDRESS>University of Illinois Urbana-ChampaignUSA ,
Yinlin Deng<EMAIL_ADDRESS>University of Illinois Urbana-ChampaignUSA
,  Runyu Lu<EMAIL_ADDRESS>Huazhong University of Science and
TechnologyChina ,  Jiayi Yao<EMAIL_ADDRESS>The Chinese University
of Hong Kong, ShenzhenChina ,  Jiawei Liu<EMAIL_ADDRESS>University of
Illinois Urbana-ChampaignUSA ,  Reyhaneh Jabbarvand<EMAIL_ADDRESS>University of Illinois Urbana-ChampaignUSA and  Lingming Zhang
<EMAIL_ADDRESS>University of Illinois Urbana-ChampaignUSA

††copyright: none††journalyear: 2023

### Abstract

Compiler correctness is crucial, as miscompilation falsifying the program
behaviors can lead to serious consequences over the software supply chain. In
the literature, fuzzing has been extensively studied to uncover compiler
defects. However, compiler fuzzing remains challenging: Existing arts focus on
black- and grey-box fuzzing, which generates test programs without sufficient
understanding of internal compiler behaviors. As such, they oftentimes fail to
construct test programs to exercise important conditions of intricate
optimizations. Meanwhile, traditional white-box techniques, such as symbolic
execution, are computationally inapplicable to the giant codebase of compiler
systems. Recent advances demonstrate that Large Language Models (LLMs) excel
in code generation/understanding tasks and even have achieved state-of-the-art
performance in black-box fuzzing. Nonetheless, prompting LLMs with compiler
source-code information remains a missing piece of research in compiler
testing.

To this end, we propose WhiteFox, the first white-box compiler fuzzer using
LLMs with source-code information to test compiler optimization. WhiteFox
adopts a dual-model framework: _(i)_ an analysis LLM examines the low-level
optimization source code and produces requirements on the high-level test
programs that can trigger the optimization; _(ii)_ a generation LLM produces
test programs based on the summarized requirements. Additionally,
optimization-triggering tests are also used as feedback to further enhance the
test generation prompt on the fly. Our evaluation on four popular compilers
from various domains shows that WhiteFox can generate high-quality test
programs to exercise deep optimizations requiring intricate conditions,
practicing up to 8$\times$ more optimizations than state-of-the-art fuzzers.
To date, WhiteFox has found in total 96 bugs for the compilers under test,
with 80 confirmed as previously unknown and 51 already fixed. Beyond compiler
testing, WhiteFox can also be adapted for white-box fuzzing of other complex,
real-world software systems in general.

##  1\. Introduction

Modern compilers (llvm, ; gcc, ; opengl, ; pytorch2, ; Tensorflow, ;
chen2018tvm, ) play a critical role in translating high-level programming
languages into efficient machine code (_e.g.,_ X86 instructions). However,
incorrect or misapplied optimizations can lead to subtle, hard-to-detect bugs,
or even incur undesired program behaviors. Given the ubiquity of compilers in
software development, it is vital to ensure the correctness of compiler
optimizations. To this end, fuzzing (or fuzz testing) (sutton2007fuzzing, ;
zeller2019fuzzing, ) has been applied to automatically generate a large number
of test inputs, aiming to expose potential compiler defects (sun2016toward, ).
To date, a large body of fuzzing tools has been tailored for different
languages and compilers (csmith, ; yarpgen, ; nnsmith, ; grayc, ; polyglot, ),
highlighting their success by finding a large number of bugs in these real-
world compilers.

In the literature, researchers have proposed various fuzzing techniques to
incorporate the knowledge about the system under test (SUT) during test
generation (grayc, ; polyglot, ). They can be mainly classified into three
categories according to the extent of SUT knowledge visible to the fuzzer:
black-box (nnsmith, ; csmith, ; yarpgen, ), grey-box (libfuzzer, ; grayc, ;
polyglot, ), and white-box fuzzing (symbolic, ; sen2007concolic, ). Black-box
fuzzing has zero knowledge about the internal workings of the SUT and simply
considers system input/interface information. Consequently, the inputs are
generated without complying with the intended structures or behaviors of the
SUT. In contrast, white-box fuzzing techniques, by inspecting the source code
of the SUT, aim to synthesize test cases to exhaustively explore all possible
code paths. Grey-box fuzzing lies between black- and white-box fuzzing. By
leveraging partial program information of the SUT (_e.g.,_ code coverage),
grey-box fuzzers attempt to efficiently produce tests that are more likely to
exercise new program behaviors. While in theory, we can apply all such
approaches for compiler fuzzing, each approach grapples with its own
challenges and limitations owing to the immense complexity and scale of modern
compiler systems. For instance, the widely-used C/C++ compiler, LLVM (llvm, ),
is implemented with 14M lines of code, while the popular DL compiler,
TensorFlow (Tensorflow, ), has 3.5M lines.

Challenges. Black-box fuzzing, without knowledge of the internal workings,
struggles due to the intricate conditions required to trigger optimizations.
Simply generating random inputs, without any guidance, oftentimes proves
impractical for reaching the deep corners of optimization logic. For example,
a recent study (polyglot, ) shows that the black-box fuzzer for C compilers,
Csmith (csmith, ), which produces test-cases through grammar-based generation,
is significantly less effective than coverage-guided fuzzing. Grey-box
fuzzing, though better informed by source code instrumentation and often
achieving higher code coverage than its black-box counterpart, still faces
challenges reminiscent of black-box techniques. It frequently falls short of
fully understanding the nuanced criteria required to trigger particular
optimizations. This shortcoming stems from the fact that compiler
optimizations typically hinge on meeting precise and strict conditions. A
straightforward coverage-driven strategy might not navigate these specific
states effectively. Moreover, grey-box compiler fuzzing (polyglot, ; grayc, )
even faces the challenge of generating semantically correct inputs, leading to
the discovery of mostly front-end crash bugs. On the other hand, traditional
white-box fuzzing, which relies on analyzing the internal behaviors of the
target, becomes daunting with modern compilers. The sheer complexity of modern
optimization techniques, combined with the vast landscape of programming
paradigms and hardware platforms, makes modeling all behaviors an uphill task.
For instance, in symbolic execution (symbolic, ), a program is executed by
using symbolic variables in place of concrete values, enabling the systematic
exploration of every potential execution path determined by these symbolic
representations. However, when applied to compiler systems, it becomes
infeasible to designate every variable as symbolic. Even if such a feat were
achievable, the vast scale of the implementation—spanning millions of lines of
code—inevitably leads to path explosion, rendering the approach highly
challenging.

Furthermore, traditional compiler fuzzing techniques are typically tailored to
specific languages/compilers. Yet, designing and implementing a fuzzing
framework for a new compiler is both time-intensive and laborious. For
instance, Csmith (csmith, ) is comprised of over 80K lines of code through
years of development. Given the unique characteristics of each target
language/compiler, reusing the efforts of one fuzzing implementation for a
different input language/compiler often presents significant challenges.

Figure 1. Motivating example.

Motivation. Figure 1 presents a motivating example of the
permute_linear_fusion optimization in PyTorch Inductor (pytorch2, ). This
optimization fuses the permute and linear operators when the permute method is
invoked on an input tensor with more than two dimensions, specifically
swapping the last two dimensions. On the left side of the figure, we see its
source code implementation. Here, the constraints required to trigger this
optimization are explicitly detailed with nested if condition statements and a
helper function check_permute. However, when applying fuzzing techniques to
test this optimization, black/grey-box fuzzing struggles to generate models
that align the permute and linear operators with these constraints. For
instance, consider a scenario where a grey-box compiler fuzzer produces a
model with the linear operator and covers the first if-check (Line 3-6, Figure
1) in this optimization. Even if a black/grey-box fuzzer repeatedly selects
this test as a seed for mutation, it is challenging to successfully mutate the
model to solely invoke the permute method on a tensor—specifically, to swap
its last two dimensions—where the output should then serve as the input for
linear. This is because both black-box and grey-box fuzzing are unaware that
the models should include the permute and linear operators in this specific
way, due to the absence of guidance from the source code implementation. As
there are thousands of operators in PyTorch, it is very likely that such
fuzzers will choose a different operator than permute or apply permute in many
other ways. As a result, the generated models often fail validation checks and
are unable to trigger the target optimization, let alone discover deep bugs
(Line 16, Figure 1). On the other hand, though the white-box techniques have
the potential to trigger this optimization theoretically, it is impractical to
apply traditional white-box fuzzing to extract constraints from the
implementation source code since there are too many complicated data
structures involved in the optimization, such as torch.fx.GraphModule and
torch.fx.Node. These structures are further composed of several other
intricate classes and have diverse attributes (_e.g.,_ args, shape, and rank).
Additionally, the intricate constraints are often expressed in hierarchical
conditions (_e.g.,_ nested if checks) and are even further complicated with
checker functions. Therefore, it is extremely challenging, if not impossible,
to accurately extract and express these constraints symbolically for
constraint solvers, let alone apply any formal method to solve them.

Insight. Can we scale white-box fuzzing to fully test optimizations for any
compilers? We address this question based on the insight that Large Language
Models (LLMs) (openai2023gpt4, ; chatgpt, ; starcoder, ; codex, ;
wang2021codet5, ; feng2020codebert, ) are pre-trained on a vast array of code
spanning various programming languages. This broad foundation enables them to
excel in comprehending and generating code across diverse languages
(bubeck2023sparks, ). Therefore, for the permute_linear_fusion optimization,
different from typical white-box fuzzing, we can leverage LLMs to summarize
the requirements for the models that could trigger it based on the source code
information, as highlighted in the yellow text box of Figure 1. Subsequently,
we can leverage the generated requirement description to further prompt LLMs
to create the corresponding inputs, which are PyTorch models in this case. In
our experiments, the generated tests indeed triggered the
permute_linear_fusion optimization and even detected a previously unknown bug
in it! This bug was confirmed by developers and labeled as high-priority.

Our work. We present WhiteFox, the first white-box compiler fuzzing approach
via Large Language Models (LLMs) to fully test the core optimization modules
in various modern compilers. As discussed above, existing approaches to white-
box testing cannot scale to model the behavioral information of complex
compiler systems. Consequently, the key idea of WhiteFox is to leverage LLMs
to automatically infer the requirements on test programs that could trigger
the optimizations in compilers based on its source code implementation. LLMs,
having been pre-trained on an extensive corpus comprising natural languages
and a variety of programming languages, possess the ability to comprehend and
succinctly summarize optimization source code. In WhiteFox, the input to the
testing framework is the implementation source code of optimizations. First,
an analysis LLM is deployed to automatically analyze and summarize the testing
requirements for triggering optimizations. Subsequently, a generation LLM is
used to produce numerous meaningful test programs guided by the generated
requirements. To generate test programs that can directly exercise
corresponding optimizations, WhiteFox further employs a feedback-loop
mechanism which uses optimization-triggering tests as few-shot examples to
guide future test generation.

Summary. This work makes the following contributions:

  * •

Novelty. We introduce a new dimension of white-box compiler fuzzing by
leveraging LLMs as optimization source code analyzers and test input
generators. To our best knowledge, this work is the first to demonstrate that
LLMs can transform the low-level optimization implementation information into
the corresponding high-level test programs, making it practical to employ the
white-fox fuzzing techniques to test the compilers from various domains.
Furthermore, beyond compiler testing, WhiteFox can also be adapted for white-
box fuzzing of other complex, real-world software systems in general, and
inspire future work in this promising direction.

  * •

Approach. While our approach is general and can be applied to various compiler
systems, we implement WhiteFox as a practical fuzzer for four SUTs, including
three widely-used deep learning (DL) compilers and a popular C/C++ compiler
LLVM. It consists of an analysis LLM (GPT4) to summarize the requirements
based on the source code, and a generation LLM (StarCoder) to create numerous
test inputs based on the requirements. The artifact of WhiteFox is available
at https://github.com/ise-uiuc/WhiteFox.

  * •

Study. We conduct an extensive study for the comparison of WhiteFox with
state-of-the-art fuzzers on the target compilers. Our result shows that
WhiteFox can practice 2.5x more optimizations than the baselines. By now,
WhiteFox detects 96 bugs, with 80 already confirmed as previously unknown and
51 already fixed.

##  2\. Background and Related Work

Fuzzing has been extensively studied for testing both traditional compilers
and the emerging deep learning (DL) compilers. In this section, we first
review related work for fuzzing traditional and DL compilers without LLMs. We
then discuss the recent advances of LLM-based testing.

###  2.1. Compiler Fuzzing

The main challenge to compiler fuzzing is to synthesize and diversify
syntactically and semantically valid programs as the input to the compiler. In
the literature of traditional compiler fuzzing, _grammar-based_ techniques aim
to synthesize syntactically valid random programs via generation rules that
comply with the language grammar. Such techniques have been widely used for
fuzzing compilers of programming languages including C/C++ (_e.g.,_ Csmith
(csmith, ) and YARPGen (yarpgen, )) and JavaScript (_e.g.,_ LangFuzz
(holler2012fuzzing, ) and jsfunfuzz (jsfunfuzz, )). However, grammar-based
approaches oftentimes require massive engineering efforts to implement rules
that ensure the validity of the generated programs and may still fail to
synthesize realistic yet complicated test programs. Therefore, _mutation-
based_ techniques (le2014compiler, ; donaldson2017automated, ; le2015finding,
; zhang2017skeletal, ) propose to mutate valid seed programs for generating
new input programs that can exercise deeper code paths in the compiler.
Besides traditional compiler fuzzing, various fuzzing techniques have also
been proposed for the emerging DL compilers. DL compilers aim to optimize DL
models, whose generation requires valid compositions of tensor operations. In
early work, DL models are either directly curated from existing open-source
models (cradle, ) or are constructed using simple shape-preserving operators
(lemon, ; audee, ). To diversify operators used in model generation, recent
work has been focusing on explicitly defining constraints for operator
constructions either manually (nnsmith, ; muffin, ) or automatically (neuri,
).

Though comprehensive, most of the work discussed above is still black-box
fuzzers. Consequently, their generated test programs oftentimes fail to
practice the internal nuances of intricate compiler behaviors, especially in
the important compiler optimization passes. Therefore, in recent years, grey-
box compiler fuzzers (polyglot, ; tzer, ; grayc, ) have been proposed to
integrate code coverage guidance to discover interesting input programs that
can explore deeper compiler behaviors. PolyGlot (polyglot, ) proposes to
integrate coverage feedback into its fuzzing loop to test compilers at the
intermediate representation (IR) level. Tzer (tzer, ) proposes a coverage-
directed fuzzing approach that jointly mutates both IR files and optimization
passes, specifically targeting the TVM DL compiler (chen2018tvm, ). More
recently, GrayC (grayc, ) combines coverage feedback with mutation operators
to test C compilers.

To achieve more explicit test generation, white-box testers by theory can
tailor test programs to specifically exercise certain optimization by
inspecting the compiler logic from the source code. Yet, to our best
knowledge, no practical white-box compiler technique exists, largely due to
the unmanageable scale of compiler systems. Consequently, the analysis
(_e.g.,_ solving symbolic path constraints (symbolic, )) for such a scale of a
codebase can be computationally infeasible.

###  2.2. LLM-based Testing

With the recent advancements in Large Language Models (LLMs), there has been a
surge in efforts to leverage LLMs for unit test generation. For example, TeCo
(teco, ), built upon a fine-tuned CodeT5 (wang2021codet5, ) model, has been
introduced to aid developers in completing unit tests for the given code and
context. CodaMosa (codamosa, ) employs the capability of LLMs to help with the
search-based software testing (SBST) (fraser2011evosuite, ) by generating unit
tests for uncovered methods. Furthermore, TestPilot (testpilot, ) and
ChatTester (DBLP:journals/corr/abs-2305-04207, ) leverage Codex (codex, ) and
ChatGPT (chatgpt, ), respectively, for unit test generation.

While LLM-based unit test generation techniques incorporate source code, they
are specifically designed for particular modules (_e.g.,_ functions/classes)
of the projects under test. In contrast, our approach targets the important
problem of compiler testing at the entire system level (a totally different
problem from unit testing), and relies solely on the source code
implementation of optimizations as guidance for fuzzing. Also, unit test
generation demands detailed and comprehensive information about the module
being tested, including all associated data structures, while our guidance can
be imprecise, partial, or even incomplete. In addition, we are the first to
leverage LLMs to bridge the gap between the low-level optimization
implementation and the high-level input programs for compilers. Lastly, LLM-
based unit test generation can only assist developers and requires further
manual refinement, because generating reliable unit-test oracles remains
notably difficult or infeasible for even the current best LLMs (palm, ;
openai2023gpt4, ). In contrast, WhiteFox directly leverages various automated
oracles for traditional compiler fuzzing, _e.g.,_ differential testing
(mckeeman1998differential, ).

Beyond unit test generation, leveraging LLMs for fuzzing software systems is
another emerging trend in LLM-based testing (xia2023universal, ;
deng2023large, ; titanfuzz, ; sunsmt, ). For example, TitanFuzz (titanfuzz, ),
demonstrates for the first time that modern LLMs can be directly leveraged to
perform both grammar- and mutation-based fuzzing of real-world systems. Very
recently, researchers have also demonstrated that LLMs can serve as the
universal fuzzers for various types of software systems (xia2023universal, ).
Meanwhile, existing techniques mainly focus on prompting LLMs to generate
valid/diverse input programs accepted by the target systems. In contrast,
WhiteFox is the first white-box compiler fuzzing approach, which leverages
low-level optimization implementations for guided fuzzing, and has been
demonstrated to outperform the recent TitanFuzz (§ 6.1).

##  3\. Approach

Figure 2. Overview of WhiteFox.

Figure 2 depicts the overview of WhiteFox, consisting of three main
components: Requirement Summarization, Test Generation, and Feedback Loop.
Overall, WhiteFox takes the source code of optimization passes from the tested
compiler as inputs and generates high-quality test programs via LLMs. To that
end, during the Requirement Summarization phase, an analysis LLM is used to
summarize the requirements of the test programs to trigger the optimization by
examining its source-code implementation (§ 3.1). Next, the analyzed
requirements are used to prompt a generation LLM, which synthesizes test
programs to specifically practice corresponding optimizations (§ 3.2). The
generated test programs are then compiled and executed, and WhiteFox observes
whether they can indeed activate the corresponding optimizations via
instrumentation. If a test program triggers any optimization, it will be
incorporated into WhiteFox’s feedback loop as an example to guide the
generation LLM towards more optimization-targeted generation in subsequent
iterations (§ 3.3). To detect compiler bugs, every test program is executed
with test oracles including result inconsistency, as well as compile- and
execution-time crash (§ 3.4).

Notably, WhiteFox employs a dual-model framework: _(i)_ an analysis LLM which
is prompted to infer the conditions for triggering optimizations by inspecting
the implementation code; and _(ii)_ a generation LLM which is prompted to
create a large number of meaningful test programs. This design allows us to
balance the trade-off between the costs and benefits that different LLMs
provide. For example, we can let the analysis LLM be one with broad knowledge
and reasoning ability (in both natural language and code) and let the
generation LLM be one that is specialized for efficient program generation.

###  3.1. Requirement Summarization

While it is possible to directly generate tests from the optimization source
code with LLMs, it does not perform well (as shown in our evaluation § 6.2.1).
This is because the optimization source code has the following two undesirable
features: _(i)_ The implementation source code for the optimization is often
lengthy. It often contains redundant information like detailed comments
explaining the code behavior. Furthermore, it may contain code that is
unrelated to the characteristics of triggering input, such as implementation
details (e.g., using data structures like std::<vector> to gather operands),
the IR transformation after the optimization is triggered, and error logging.
Such long source code not only makes it time-consuming for LLMs to process but
also adds to the difficulty for LLMs to understand, as LLMs are known to
struggle with long context (liu2023lost, ). Additionally, some source code may
exceed the limited context window available to LLMs. _(ii)_ The implementation
code is written at a low level, involving large quantities of domain-specific
modules, IRs, and helper functions. For example, from Figure 2, we can observe
that the optimization source code relies on TensorFlow Lite-specific module
TFL, which contains various low-level operators (_e.g.,_ TFL::ConcatenationOp
and TFL::UnpackOp) and functions (_e.g.,_. dyn_cast_or_null). These IRs are
translated from the input programs written at a higher level (_e.g.,_ “Test
Inputs” in the figure). They are meant for internal use within a compiler, and
none of them are visible at the high level.

To address the above challenges, we propose to leverage an analysis LLM to
provide a clear, concise, and comprehensible summary of how the optimization
can be triggered, which will assist in subsequent test generation. We
considered two types of summary formats: natural language and pseudo-code.
Natural language description can be more concise than the original
optimization source code, as it only summarizes the key aspects of
requirements and ignores redundant or unrelated information. Furthermore, it
is much more comprehensible to LLMs, since LLMs have been trained on enormous
natural language data. On the other hand, pseudo-code can concisely describe
certain patterns (especially when there are long sequences of function calls)
and is closer to the structure of tests that we want to generate ultimately.
Additionally, we map the low-level implementation to a high-level
summarization using natural language or pseudo code resembling user code. For
instance, TFL::ConcatenationOp is a low-level TensorFlow Lite IR used in the
implementation source code of TensorFlow Lite optimizations (in Figure 2),
which corresponds to the high-level TensorFlow public API tf.concat, and
semantically means join data from multiple input tensors. In our natural
language description/pseudo code, instead of using the low-level
TFL::ConcatenationOp, we directly use “concatenation operator”/tf.concat to
summarize.

While both natural language and pseudo-code can help to shorten the context
and remove low-level information to avoid confusing the LLM, they both have
their own unique strengths. For example, the permute_linear_fusion
optimization from PyTorch Inductor requires that “the tensor method permute is
invoked first, and then the torch.nn.functional.linear function is invoked on
the permuted tensor”. Such a NL description is not as straightforward as the
pseudo-code format for this case. Specifically, “the tensor method permute”
could be more succinctly represented as input_tensor.permute(...) in pseudo-
code. However, for the description “it swaps the last two dimensions of this
tensor”, it is pretty challenging to leverage pseudo-code to describe it
clearly and briefly. To combine and maximize their strengths, we adopt a
hybrid format that blends NL and pseudo-code to describe the requirements for
triggering optimization, rather than relying solely on either format. This
mixed format provides the analysis LLM with greater flexibility to utilize NL
and/or pseudo-code as needed for each component of requirements, resulting in
a more expressive and higher-quality summarization. Our experimental findings
(§ 6.2.1) also support that the mix of NL and pseudo-code achieves the best
performance in this task. Despite all these aforementioned benefits of a high-
quality optimization summary, it is worth noting that the requirement
summarization is a very comprehensive and challenging task, even for domain
experts. First, it requires understanding the logic of the implementation code
and rephrasing it with semantic-preserving natural language or pseudo-code.
Second, the mapping from low-level implementation code to high-level
information necessitates a broad background knowledge of the corresponding
programming language and compiler. This ranges from understanding general
technical terminology (_e.g.,_ “variable arguments” and “tail calls”), to in-
depth domain-specific knowledge (_e.g.,_ from low-level LLVM IRs to high-level
C++ grammar). As demonstrated in our evaluation (§ 6.2.1), the most powerful
LLM to date (namely GPT4) is capable of performing this challenging analysis
process (while the current open-source models, such as StarCoder, still lag
far behind). This capability stems from its extensive training on vast
datasets, enabling it to gain a broad knowledge and implicit understanding of
various programming languages and systems. Additionally, its proficiency in
performing these domain-specific analyses in our work likely results from its
training on the source code of these open-source compiler systems.

Therefore, WhiteFox first leverages the analysis LLM to infer the requirement
of high-level inputs that could trigger the optimization, utilizing its
implementation written in the low-level source code. More specifically, for
each optimization, we use few-shot in-context learning (fewshot, ) to prompt
the analysis LLM to generate its trigger requirements for the inputs in the
mixed format of NL and pseudo-code. Figure 3(a) presents the general few-shot
prompt template used to summarize requirements for optimizations in target
compilers. This prompt template starts with the instruction “Please describe
the [TARGET INPUT] that can trigger the [OPTIMIZATION NAME] optimization…”,
where [TARGET INPUT] is the input format specific to the target compiler. Then
it is followed by the source code of the optimization implementation and
concludes with the description of requirements that the input should fulfill.
Note that the description is in the mixed format of NL and pseudo-code,
consisting of [PSEUDO CODE] and [NL DESCRIPTION]. The target optimization has
the same structure as the few-shot examples, but its requirements field is
left empty, awaiting generation by the LLM. Take the prompt of requirement
summarization in PyTorch Inductor as an example, whose few-shot prompt is
shown in Figure 3(b). The expected input format for PyTorch Inductor is a
PyTorch model; therefore, [TARGET INPUT] is populated with “PyTorch model”.
Following this, the source code for the example optimization,
permute_linear_fusion, is provided. Finally, example descriptions are given in
the mix of pseudo-code and NL formats to outline the constraints necessary to
trigger the example optimization. Note that such few-shot examples can guide
the analysis LLM in generating the desired output format. For instance, the
example description (emphasized with a yellow box in Figure 3(b)) provides the
LLM with a clearer illustration of the expected format. Furthermore, they
facilitate the learning process of analysis LLM by providing the example
mappings from low-level optimization implementation to high-level input
program requirements.

More formally, let $\mathcal{P}_{A}$ be the analysis LLM that models the
probability of token sequences. It takes the following types of information as
inputs: _(i)_ $I_{i}$, the instruction on summarizing an optimization $O_{i}$;
_(ii)_ $C_{i}$, the source code implementation of $O_{i}$; _(iii)_ $R_{i}$,
the summarized trigger pattern or requirement of the optimization $O_{i}$. Let
$E_{K}$ be the few-shot prompt prefix consisting of $K$ example optimizations:
$E_{K}=(I_{1},C_{1},R_{1})\circ(I_{2},C_{2},R_{2})\circ\ldots\circ(I_{K},C_{K},R_{K})$.
Let $O_{t}$ be the target optimization. The probability distribution of the
generated requirement $R_{t}$ can be defined as $\mathcal{P}_{A}(R_{t}\mid
E_{K},I_{t},C_{t})$.

(a) Prompt template for requirement summarization.

(b) Requirement summarization prompt in PyTorch Inductor.

Figure 3. Requirement summarization prompt in WhiteFox.

###  3.2. Test Generation

(a) Prompt template for test generation

(b) Test generation prompt in PyTorch Inductor.

Figure 4. Test generation prompt in WhiteFox.

By utilizing the requirement description of the optimization, WhiteFox employs
the power of LLM to generate test inputs that can effectively trigger the
corresponding optimization for bug detection. Similar to requirement
summarization, we leverage few-shot in-context learning to generate the test
inputs specific to each optimization based on the requirements. Figure 4(a)
shows the general prompt template used in WhiteFox for test generation. In the
template, the structure of few-shot examples comprise an instruction,
specifically: “Please generate a valid [TARGET INPUT] example with [INPUT
SPECIFICATION] meets the specified requirements.” Then, it details the
requirements for optimization activation and concludes with an illustrative
input that practices the _example_ optimization. Following the few-shot
examples, the target optimization has a similar structure while the test input
to it is _to be generated_ by the LLM. Figure 4(b) shows the test generation
prompt used in PyTorch Inductor. The format of test input to the PyTorch
Inductor is a PyTorch model ([TARGET INPUT]) with public PyTorch APIs ([INPUT
SPECIFICATION]). Next, we specify the requirements for the test inputs to
activate the optimization permute_linear_fusion. This is complemented by an
illustrative model that can trigger this optimization. The provided few-shot
examples aid the LLM in generating the test in the desired format, such as a
torch.nn.Module composing public PyTorch APIs. Furthermore, the example could
help the LLM to learn the relationship between the requirement description of
the optimization and the corresponding test input that can trigger it. In
Figure 4(b), the model input highlighted in a blue box (# Model) contains the
code x1.permute(0, 2, 1). This corresponds to the requirement: ”The permute
method is invoked on an input tensor with more than 2 dimensions, and it swaps
the last two dimensions of this tensor”. Such examples elucidate for the LLM:
_(i)_ the meaning of ”permute method is invoked on an input tensor”—which
should not be torch.permute(x1, ...), and _(ii)_ the implication of ”swaps the
last two dimensions of this tensor”-which is permute(0, 2, 1) for the tensor
with three dimensions.

Again, let $\mathcal{P}_{G}$ be the generation LLM that models the probability
of token sequences. Recall that $I$ represents the summarization instruction
and $R$ denotes the requirement for triggering an optimization. Let $E_{K}$ be
the few-shot prompt prefix consisting of $K$ example optimizations:
$E_{K}=(I_{1},R_{1},T_{1})\circ(I_{2},R_{2},T_{2})\circ\ldots\circ(I_{K},R_{K},T_{K})$,
where $T_{i}$ is a valid test input that triggers the optimization $O_{i}$.
Let $O_{t}$ be the target optimization. The probability distribution of the
generated test $T_{t}$ can be defined as $\mathcal{P}_{G}(T_{t}\mid
E_{K},I_{t},R_{t})$.

###  3.3. Feedback Loop

The aim of utilizing the optimization implementation to guide the LLM in test
generation (§ 3.2) is to produce diverse tests that can effectively exercise
the optimization, with the ultimate aim of uncovering potential bugs in the
optimization. While the prompt in Figure 4 contains example optimizations from
the same compiler under test (and their trigger tests), these few-shot
examples lack specific guidance for the target optimization.

Figure 5. Prompting for test generation with feedback.

The motivation behind incorporating a feedback loop in WhiteFox is that the
generated tests themselves are valuable feedback from the SUT, and can serve
as guidance for fuzzing a target optimization in future iterations. In each
iteration, when newly generated tests are discovered for the target
optimization, we collect them as candidates for few-shot examples for future
test generation prompts. By incorporating such successful triggering tests
into the prompt, we enhance a targeted guidance to the generation LLM,
enabling it to produce more inputs that trigger the target optimization.

Based on the inspiration, WhiteFox incorporates tests that have successfully
triggered the corresponding optimization as supplementary examples during
iterative test generation. The target compiler is instrumented to record which
optimizations are triggered for a test input. If the test input successfully
practices the corresponding optimization, it will be incorporated as a
candidate example in the subsequent iteration of test generation for that
specific optimization. More specially, in each iteration, if there are
triggering inputs for the current optimization, WhiteFox selects a number of
triggering inputs as examples (defaulting to 3). These example triggering
inputs will be plugged into the prompt shown in Figure 5, along with the
instruction and requirement summary of the target optimization (same
instruction and summary as the previous prompt), and will be used to generate
the next batch of test inputs. This feedback design aids the LLM in generating
tests that have a higher likelihood of triggering the targeted pattern, which
is demonstrated in our ablation study (§ 6.2.2). In the case where there are
no triggering inputs for a particular optimization, WhiteFox continues to use
the initial prompt (Figure 4) in subsequent iterations until it finds an input
capable of triggering that optimization.

Not all triggering examples are equally effective in guiding the LLM to
generate new valuable triggering tests. One useful signal for assessing their
effectiveness is the triggering rate of the newly generated tests when we
employ them as few-shot examples. To effectively select triggering examples
with an evolving knowledge of example effectiveness, it is crucial to find a
balance between exploration and exploitation. Exploration is critical because
it not only allows us to evaluate under-explored options but also helps to
produce a diverse set of tests for fuzzing. On the other hand, some level of
exploitation is desirable, as it enables us to fully harness the potential of
effective examples. To address this, WhiteFox adopts an (adapted) Multi-Armed
Bandit (MAB) algorithm, Thompson Sampling (thompson1933likelihood, ), as the
selection strategy for triggering examples to balance the exploiting and
exploration trade-off. Each triggering example is conceptualized as an arm in
the MAB framework. The main assumption here is that each triggering example is
associated with a probability representing the triggering rate, which
quantifies the proportion of generated tests capable of triggering the
optimization when utilizing the given example. During the fuzzing loop, our
objective is to estimate the probability associated with each triggering
example, with the aim of using the most effective example to achieve more
valuable triggering tests. More specifically, following the classical Thompson
Sampling algorithm (thompson1933likelihood, ), when we do not have any prior
information about an arm, we choose standard beta distribution
(mcdonald1995generalization, ) $B(1,1)$ (or equivalently Uniform$(0,1)$) for
its prior distribution. The beta distribution is parameterized by two shape
parameters $\alpha>0,\beta>0$, which represents the number of successes and
failures in historical trials. The probability density function of beta
distribution can be formally written as follows:

$f_{\alpha,\beta}(x)=\frac{1}{B(\alpha,\beta)}x^{\alpha-1}(1-x)^{\beta-1}$

, where $B(\alpha,\beta)$ is a constant for normalization. After drawing a new
sample and observing the reward (in our case, $1$ if a generated test
successfully triggers the targeted optimization and $0$ otherwise), the
posterior probability can be conveniently updated by increasing $\alpha$ or
$\beta$ by one, depending on whether the sample was a success or failure. More
formally, if our prior belief about $X$ is represented by $f_{\alpha,\beta}$,
the posterior distribution of $X$ will be updated to $f_{\alpha+1,\beta}$ or
$f_{\alpha,\beta+1}$ after we observe $X=1$ or $X=0$.

Algorithm 1 shows the example test selection process. Firstly, we sample
$\theta_{t}$ from each of the posterior distributions (Line 1-1).
Subsequently, we opt for top-$N$ arms with the highest sampled value, which
are the chosen example tests (ExampleTests) in this iteration (Line 1). Unlike
conventional scenarios where a single arm is chosen, we simultaneously select
multiple test examples at a time to construct a single few-shot prompt for
generating a batch of new tests. Consequently, when we observe the number of
triggering tests among all newly generated tests, we use this information to
update the posterior of each of ExampleTests (Line 1-1). Following this, we
initialize the new trigger tests (NewTriggerTests) using the mean values of
$\alpha$ and $\beta$ of the distribution of the ExampleTests (Line 1-1), to
reduce the overhead of re-exploring the distribution of NewTriggerTests from
scratch. This is based on our assumption that the effectiveness of the new
test is strongly correlated with those few-shot examples, as the new test is
generated by the LLM based on those specific examples, potentially inheriting
valuable code ingredients/patterns from such “parent” examples (fewshot, ). In
the end, we update the pool of existing trigger tests with newly found tests
(Line 1).

1

2 Function  _Select(_TriggerTests , N_)_:

3 for  _$t\in\textnormal{{TriggerTests}}$_ do

4 Sample $\theta_{t}\sim$ Beta($\alpha_{t},\beta_{t}$)

5

6 $\textnormal{{ExampleTests}}\leftarrow argmax_{t}(\theta,N)$

7 return _ExampleTests_

8

9

10 Function  _Update(_ExampleTests , NewTriggerTests, NumTrigger,
NumNotTrigger_)_:

11 for  _$t_{example}\in\textnormal{{ExampleTests}}$_ do

12 $\alpha_{example}\leftarrow\alpha_{example}+\textnormal{{NumTrigger}}$

13 $\beta_{example}\leftarrow\beta_{example}+\textnormal{{NumNotTrigger}}$

14

15 $\alpha_{avg}\leftarrow
avg(\\{\alpha_{example}|t\in\textnormal{{ExampleTests}}\\}])$

16 $\beta_{avg}\leftarrow
avg(\\{\beta_{example}|t\in\textnormal{{ExampleTests}}\\}])$

17 for  _$t_{new}\in\textnormal{{NewTriggerTests}}$_ do

18 $\alpha_{new}\leftarrow\alpha_{avg}$

19 $\beta_{new}\leftarrow\beta_{avg}$

20

21
$\textnormal{{TriggerTests}}\leftarrow\textnormal{{TriggerTests}}\cup\textnormal{{NewTriggerTests}}$

22

Algorithm 1 Modified Thompson Sampling

###  3.4. Test Oracle

WhiteFox uses the following symptoms as the lens to manifest compiler bugs.

Result inconsistency. During compilation, programs are iteratively transformed
to semantically equivalent yet more efficient code through an array of
optimization passes. However, miscompilation can silently happen due to
logical defects in the pass implementation, leading to undesired semantic
inconsistency in the produced machine code. Such semantic inconsistency can be
manifested through differential testing, as is commonly used in prior compiler
testing work (nnsmith, ; tzer, ; le2014compiler, ; le2015finding, ; yarpgen,
). Specifically, for each test program that is both compilable and executable,
given the same set of inputs to the program (if required), we cross-check the
produced outputs over the optimized and non-optimized (or minimally optimized)
versions of the program.

Crash. Following prior work (titanfuzz, ; nnsmith, ; csmith, ; grayc, ;
le2015finding, ; eagle, ; polyglot, ; tzer, ), it is undesirable to let the
compiler and the compiled executable to unexpectedly crash. Consequently,
WhiteFox actively captures crashing signals at both the compile- and
execution-time for the test program, including process aborts, segmentation
faults, and forbidden internal exceptions (_e.g.,_ INTERNAL_ASSERT_FAILED in
PyTorch).

##  4\. Implementation

Optimization collection. We start to gather the compiler source code that is
specifically for optimization by specifying the relevant directories. For
example, the source code of optimization passes for PyTorch Inductor are
managed under the torch/_inductor directory, and that for LLVM is mainly
placed under llvm/lib/Transforms. We next identify code fragments (_e.g.,_
functions) that perform optimizations through simple keyword pattern matching.
For example, operator fusion is an important optimization in DL compilers and
we collect the relevant functions by searching “fusion” or “fuse” in PyTorch
Inductor and TensorFlow-XLA. In addition, auxiliary functions invoked by the
main optimizations are also collected since they may unveil essential
conditions for activating the optimizers. Curating and identifying
optimization-relevant code fragments is required to drive WhiteFox; however,
it shall be easy for compiler developers who have a deep understanding of the
code being maintained.

Instrumentation. To gather the triggering information for the feedback loop (§
3.3), we instrument each collected optimization function by inserting a
logging statement at the function entry. As such, when compiling a test
program, from the logs a sequence of activated optimization passes can be
obtained.

Analysis and generation LLMs. While our approach is general and thus agnostic
to the LLMs being employed, our tool WhiteFox is built on the state-of-the-art
GPT4 (openai2023gpt4, ) and StarCoder (starcoder, ). Specifically, we utilize
GPT4 (openai2023gpt4, ) as the analysis LLM for its recognized excellence in
code comprehension and proficiency in natural language processing tasks
(bubeck2023sparks, ). For each optimization, we let the analysis LLM generate
one requirement description with the temperature set to zero via the OpenAI
APIs. Meanwhile, we choose StarCoder (starcoder, ) to be the generative LLM,
which is an open-source model with 15.5B parameters and a context length of
8K. In each iteration, we let StarCoder generate a batch of ten test inputs
with the temperature set to one through the HuggingFace APIs
(HuggingFaceWebPage, ). The model choices allow us to balance the trade-off
between the costs and benefits that different LLMs provide: _(i)_ GPT4 is a
powerful unified LLM (_i.e.,_ with broad knowledge and reasoning ability over
both natural language and code) but costly, making it suitable as the analysis
LLM where its use is a one-time effort; _(ii)_ StarCoder is an affordable LLM
specialized for code and is thus suitable for efficient continuous test
generation.

Few-shot prompting. For the requirement summarization and initial test
generation few-shot prompt specific to each target compiler, we opt for one-
shot prompting, aiming to minimize the manual labor involved in prompt
construction and fit into the LLM context size. To accomplish this, we select
an optimization from each target compiler. Subsequently, we manually write the
requirement description and a demonstrative input test capable of triggering
the optimization. This serves as the one-shot example in the prompts for both
requirement summarization and test generation. One exception is that PyTorch
Inductor has two distinct types of optimizations (7 utilizes a conventional
optimization check function, and 54 involves a pattern matcher). Therefore, we
separately design two prompts for each type and choose the corresponding
prompt for each optimization based on its type. For the feedback prompt, the
requirement is produced by the analysis LLM, and the sample tests are created
by the generation LLM. We use three-shot as our default setting in the
feedback prompt. On the whole, the effort required to assemble the prompt for
each target compiler remains manageable.

##  5\. Evaluation

###  5.1. Research Questions

We investigate the following research questions in our experiments:

  * •

RQ1: How does WhiteFox compare against existing compiler fuzzers?

  * •

RQ2: How do the key components of WhiteFox contribute to its effectiveness?

  * •

RQ3: Is WhiteFox able to detect real-world bugs?

###  5.2. Experimental Setup

Compilers under test. We perform the evaluation on four SUTs, including three
emerging deep DL compilers and the popular C/C++ compiler LLVM. Specifically,
we focus on three DL compiler backends: PyTorch Inductor (pytorch2, ),
TensorFlow Lite (tflite, ) and TensorFlow-XLA (tfxla, ), within PyTorch
(pytorch, ) and TensorFlow (Tensorflow, ), two of the most widely used DL
frameworks. We tried our best to collect all possible optimizations in each of
the four compilers. The only exception is for TensorFlow-XLA, which tends to
have lengthy optimization implementations: In order to make a fair comparison
against all baselines, including a WhiteFox variant where we provide the full
implementation source code to the LLMs (i.e., WF-Impl), we only choose the
TensorFlow-XLA optimizations that consist of less than 400 lines due to the
limit of LLM context window size. Table 1 lists the overview of the tested
compilers.

Table 1. Details of target compilers. | # Optimizations | Language of source | Language of tests | Version
---|---|---|---|---
PyTorch Inductor | 61 | Python | Python | nightly-20230509
TensorFlow Lite | 13 | C++ | Python | nightly-20230507
TensorFlow-XLA | 49 | C++ | Python | nightly-20230507
LLVM | 52 | C++ | C++ | nightly-20230818

Baselines. We compare WhiteFox with state-of-the-art DL library fuzzers,
including LLM-based TitanFuzz (titanfuzz, ) and symbolic rule-based NNSmith
(nnsmith, ). Since most optimizations are triggered by a sequence of
operators, we do not include the comparison with API-level DL library fuzzers
(freefuzz, ; deeprel, ; nablafuzz, ; docter, ), which are not intended for
testing optimizations. We run each tool with its default configuration on
PyTorch Inductor, TensorFlow Lite, and TensorFlow-XLA. For LLVM, our baselines
include YARPGen (yarpgen, ), a recent fuzzing tool to generate C/C++ test
inputs with strategies to trigger different optimizations in compilers, and
GrayC (grayc, ), state-of-the-art grey-box fuzzer using coverage feedback to
generate test programs in C. Each of them is run with its default setting.

Multiple variants of WhiteFox are evaluated in our ablation study. Considering
that PyTorch Inductor boasts the most optimizations, we conduct our ablation
study exclusively to PyTorch Inductor. For requirement generation, we consider
four variants: WF-Mix (the default setting of WhiteFox), WF-NL, WF-Code, and
WF-Impl. For each, we leverage the generation LLM to generate 100 test inputs,
guided by different requirement formats. The requirements of our default
setting, WF-Mix, are in the mixed format of natural language and pseudo-code
generated by the analysis LLM. By contrast, the requirements used in WF-NL
(resp. WF-Code) are the natural language (resp. pseudo-code) description
extracted from the mixed format, for a fair comparison. Besides, we also
evaluate the performance when we directly feed the generation LLM with the
implementation source code, as seen in the WF-Impl variant. Regarding the
feedback loop, in addition to our default configuration, which uses feedback
with Thompson Sampling, we contemplate two alternative variants: one without
any feedback (WF-No-Feedback) and another that incorporates feedback but
employs a random selection (WF-Rand). We further explore the decision to use
GPT4 as the analysis LLM by introducing an additional variant, WF-Starcoder.
In this variant, we employ StarCoder as the analysis LLM to generate
requirement descriptions from the implementation source code.

Environment. Our evaluation test-bed is a 64-core workstation with 256 GB RAM
and running Ubuntu 20.04.5 LTS with 4 NVIDIA RTX A6000 GPUs.

Fuzzing budget. Our default setting is generating a total of 1000 tests for
each optimization in 100 iterations. In each iteration, WhiteFox generates
tests based on the triggering feedback from the last iterations with default
batch size 10. If the optimization has triggering inputs from previous
iterations, WhiteFox randomly picks three trigger inputs as the examples in
the prompt for the following iterations with Thompson Sampling. Otherwise,
WhiteFox re-uses the default few-shot prompt to generate tests for this
optimization in this iteration.

###  5.3. Metrics

Following prior work (nnsmith, ; titanfuzz, ; yarpgen, ; grayc, ; polyglot, ;
tzer, ; donaldson2021test, ), we use the number of detected bugs as our
metric, which essentially reflects the goal of fuzzing – finding more bugs.
Moreover, the primary goal of our approach is to effectively test the
optimizations within compilers. As such, we also let the number of triggered
optimizations and the number of (optimization-)triggering tests be our
principal metrics. Specifically, an optimization is deemed “triggered” if the
corresponding optimization function (§ 4) logs its presence during the
execution of fuzz tests. Meanwhile, a test qualifies as a “triggering test”
only if during its execution, any of the optimizations are triggered.

##  6\. Result Analysis

###  6.1. Comparison with Prior Work

Table 2 presents the comparison results of WhiteFox against the baselines on
the four target compilers. Note that for a fair comparison with NNSmith, which
has a shorter execution time than our default setting, we also present results
from WhiteFox-Mini. This variant produces 100 tests for each optimization, as
opposed to the default 1000 tests.

In terms of optimization triggering, we observe that WhiteFox outperforms the
baselines significantly in PyTorch Inductor, TensorFlow Lite, and LLVM.
Overall, among the tested compilers, WhiteFox outperforms existing testers by
up to 8.2x in terms of the number of triggered optimizations. For example, out
of the 61 optimizations in PyTorch Inductor, WhiteFox is able to trigger 41
optimizations, while the baseline approaches can trigger at most 5
optimizations, which is a subset of optimizations covered by WhiteFox.
Regarding the time cost, WhiteFox consumes less time than all other techniques
except NNSmith. Meanwhile, given the inferior performance of NNSmith,
WhiteFox-Mini can still trigger more optimizations than NNSmith using less
time.

WhiteFox outperforms all baselines on compilers except for TensorFlow-XLA,
with two fewer optimizations being triggered compared to TitanFuzz. One
possible reason is that the targeted optimizations in TensorFlow-XLA are
relatively simple per our optimization filtering for fair comparison with
baselines (§ 5.2). Upon inspection, many of these optimizations represent
common model patterns that are widely used in practice. Therefore, they can be
effectively triggered by TitanFuzz since TitanFuzz leverages LLMs to generate
human-like programs by resembling the distribution of training data.
Nevertheless, despite generating slightly fewer total tests compared to
TitanFuzz, WhiteFox demonstrates its own edge by triggering four unique
optimizations which TitanFuzz cannot. In addition, we note that NNSmith has
much more triggering tests than WhiteFox and TitanFuzz over TensorFlow-XLA.
This is largely due to the implementation choice of NNSmith, which always
outputs the model with redundant reshapes. Thus, the vast majority of test
inputs from NNSmith can trigger the IdentityReshapeRemoving optimization
(117,006/117,381).

Table 2. Comparison with baselines. |  | # Optim. | # Triggered optim. | # Triggering tests | # Tests | Time (hour)
---|---|---|---|---|---|---
PyTorch Inductor | WhiteFox | 61 | 41 | 21,469 | 61,000 | 41.1
WhiteFox-Mini | 39 | 1,737 | 6,100 | 4.2
TitanFuzz | 4 | 5,519 | 521,251 | 76.6
NNSmith | 5 | 47 | 12,084 | 4.9
TensorFlow Lite | WhiteFox | 13 | 12 | 2,801 | 12,000 | 18.1
WhiteFox-Mini | 10 | 305 | 1,200 | 1.1
TitanFuzz | 8 | 571 | 243,288 | 59.0
NNSmith | 7 | 4,666 | 117,381 | 6.8
TensorFlow-XLA | WhiteFox | 49 | 20 | 12,990 | 49,000 | 59.7
WhiteFox-Mini | 19 | 1,307 | 4,900 | 5.3
TitanFuzz | 22 | 45,762 | 243,288 | 63.2
NNSmith | 16 | 117,006 | 117,381 | 6.0
LLVM | WhiteFox | 52 | 26 | 25,322 | 52,000 | 30.9
YARPGen | 3 | 3,352 | 6,948 | 28.1
GrayC | 4 | 8,353 | 107,234 | 30.6

###  6.2. Ablation Study

Given that PyTorch Inductor has the highest number of optimizations, our
ablation study is solely focused on PyTorch Inductor.

####  6.2.1. Requirement Generation & Test Generation

We first study the effectiveness of the requirement description and the
multiple choices for the format, which is shown in Table 3. The goal of the
requirement generation is to assist the generation LLM in producing more tests
that can trigger additional optimizations within the compiler. Thus, our main
points of comparison are the number of triggered optimizations (Column “#
Triggered optim.”) and the number of tests that can trigger the optimization
(Column “# Triggered tests”).

Effectiveness of requirement description. Compared with WF-Impl, which feeds
the implementation source code directly with the generation LLM, all three
variants that use requirements (WF-Mix, WF-NL, and WF-Code) demonstrate
superior performance in generating triggering tests. Notably, our default
setting WF-Mix is able to generate 1.74x more triggering tests than directly
using implementation code. Besides, WF-Mix can also trigger 7 more
optimizations than WF-Impl, emphasizing the importance of using requirement
description. This aligns with our statement in the Approach section that
optimization source code is not the most effective guide for the generation
LLM due to its redundant, unrelated information, and low-level format.

Effectiveness of mixed format. As shown in Table 3, WF-Mix has the best
performance in terms of the number of triggered optimizations and triggering
tests, underlining the effectiveness of combining NL and pseudo-code for
requirement description. Concurrently, while WF-NL triggers more optimizations
than WF-Code, it results in fewer triggering tests. This is because NL usually
contains additional information than pseudo-code, ensuring vital triggering
requirements are not missed during conversion from the implementation source
code. Conversely, it is more straightforward for the generation LLM to
correlate requirements formatted in pseudo-code with the respective test
programs, leading to a higher number of triggering tests.

Analysis LLM. When employing requirement descriptions generated by StarCoder,
WF-Starcoder not only results in fewer triggered optimizations but also a
reduced number of triggering tests compared to our default setting, which
utilizes GPT4 to summarize the implementation source code. This discrepancy is
anticipated, given that GPT4 is recognized as the cutting-edge LLM in tasks
related to code comprehension and natural language generation
(bubeck2023sparks, ). Essentially, GPT4 exhibits a superior ability in
translating intricate source code details into high-level input requirements
compared to StarCoder. This observation underscores our rationale for choosing
GPT4 as the analytical LLM. Interestingly, even WF-Starcoder generates a
higher number of triggering tests than WF-Impl, which creates the input
straight from the implementation source code. Such a discovery confirms that a
dual-model infrastructure might be better aligned for white-box compiler
fuzzing than directly utilizing the implementation source code, emphasizing
the value of having a distinct phase dedicated to requirement generation.

Table 3. Impact of requirement description formats on PyTorch Inductor. | # Triggered optim. | % Total optim. | # Triggering tests | % Total tests
---|---|---|---|---
WF-Mix | 39 | 60.9% | 1,113 | 17.4%
WF-NL | 37 | 57.8% | 940 | 14.7%
WF-Code | 32 | 50.0% | 1,055 | 16.5%
WF-Impl | 32 | 50.0% | 638 | 10.0%
WF-Starcoder | 32 | 50.0% | 745 | 11.6%

####  6.2.2. Feedback Loop

Figure 6. Impact of the feedback loop and Thompson Sampling on PyTorch
Inductor.

Next, we examine the effectiveness of our feedback loop and the Thompson
Sampling algorithm. The primary aim of the feedback loop is to enhance the
likelihood of generating additional test inputs that activate already-
triggered optimizations. Therefore, in this ablation study, we focus on
comparing the number of triggering tests. Figure 6 showcases a bar chart that
details the number of triggering tests for each triggered optimization,
spanning the range of variants explored in this ablation study.

Effectiveness of feedback loop. WhiteFox and WF-Rand, incorporating the
feedback loop, produce significantly more triggering tests than the variant
without it (WF-No-Feedback), with increases of 2.6x and 2.1x respectively.
This elevated performance not only emphasizes the effectiveness of the
feedback loop but also indicates that its guidance is more attuned to
triggering the target optimization than relying on few-shot examples for other
different optimizations.

Effectiveness of Thompson Sampling. In our default configuration, WhiteFox
leverages the Thompson Sampling algorithm for selecting triggering examples
and achieves a remarkable 1.3x more triggering tests compared to WF-Rand,
which relies on random test selection. As shown in Figure 6, WhiteFox
outperforms the rest over 32 out of the 41 triggered optimizations in terms of
the number of trigger tests. Overall, the experimental results show the
effectiveness of our MAB-based triggering example selection.

###  6.3. Bug Finding

Table 4 presents the bug-finding results of WhiteFox over the tested
compilers, namely PyTorch, TensorFlow Lite, TensorFlow-XLA, and LLVM. To date,
WhiteFox has detected 96 bugs. Of these, 80 bugs have been confirmed as
previously unknown bugs. Notably, among the 65 PyTorch Inductor bugs, 5 have
been labeled with high priority.

Table 4. Bugs detected by WhiteFox. | Total | Confirmed | New | Won’t fix | Pending
---|---|---|---|---|---
PyTorch | 65 | 62 | 60 | 3 | 0
TensorFlow Lite | 11 | 8 | 8 | 2 | 1
TensorFlow-XLA | 12 | 10 | 10 | 0 | 2
LLVM | 8 | 2 | 2 | 3 | 3
Total | 96 | 82 | 80 | 8 | 6

Bug analysis. We conduct a comprehensive analysis over the PyTorch Inductor
bugs as it is the main source of the bugs WhiteFox found (65/96, 67.7 %), and
the majority of them have been fixed (51/65, 78.5 %). Out of these 65 bugs,
only 11 (16.9%) can be covered by the state-of-the-art DL compiler fuzzers,
with 10 detectable by TitanFuzz and 3 by NNSmith.

Regarding the 51 fixed bugs, we further explore the root cause of the bugs by
inspecting their corresponding developer fixes. Impressively, 32 (62.7%) of
the fixed bugs are repaired in the optimization code of PyTorch Inductor. This
demonstrates the effectiveness of WhiteFox for detecting optimization bugs,
which is the primary aim of our approach. Specifically, only 2 of these 32
optimization bugs can be covered by TitanFuzz and NNSmith, highlighting the
significant edge of WhiteFox in testing compiler optimizations. One
interesting observation is that certain optimizations appear to be more
erroneous than the others; however, such erroneous optimizations may instead
turn out to be harder to discover. For example, _WhiteFox detects _4_ bugs in
the optimization for the important attention modules (vaswani2017attention, ),
which are the fundamental building blocks to LLMs_. While it is undesirable
for critical components to have multiple bugs uncaught by developer-crafted
tests, it comes from the difficulty of constructing specific and intricate
model patterns for manifesting such deeply hidden bugs. By exposing such
critical bugs, WhiteFox underscores the effectiveness of performing white-box
fuzzing with LLMs.

Figure 7. Example bugs detected by WhiteFox.

Bug examples. Figure 7 exemplifies confirmed bugs found by WhiteFox over
target compilers.

Figure 7(a) illustrates a misoptimization bug of PyTorch Inductor, manifested
when compiling attention modules (vaswani2017attention, ). The faulty
optimization through pattern matching identifies self-attentions and fuses
their sub-operators into a compact and efficient implementation. However, the
optimized attention module, given an input tensor attn_mask whose values are
fully False booleans, erroneously yields outputs that are entirely NaNs, which
are different from the correct results computed by the non-optimized module.
This is due to that a fusion strategy designed for float attn_mask is mis-
dispatched to compute the boolean attn_mask. Given the prevalence of attention
modules and LLMs, developers quickly fixed the bugs three days after our
report.

Figure 7(b) presents a _high-priority_ bug detected in PyTorch Inductor. The
binary_unary_fusion optimization is able to fuse the Linear (_i.e.,_ binary)
and ReLU (_i.e.,_ unary) operators into a compact and thus efficient
operation. However, after compilation, such exemplified module returns
impermissible negative outputs since its final layer is ReLU whose output is
always non-negative (agarap2018deep, ). Because such Linear-ReLU structures
are incorporated in many fundamental architectures such as ResNet (Residual
Network) (he2016identity, ), this bug, uniquely found by WhiteFox, is labeled
as _high-priority_ and was fixed immediately after our report.

Figure 7(c) depicts a bug manifested in the FuseUnpackAndConcatToReshape
function of TensorFlow Lite. Specifically, this optimization aims to simplify
unpack-concat operation pairs into a single reshape operation, when the unpack
and concat operation is inverse to each other. This requires the unpacked
dimension to be the same as the concatenated dimension to the original input
tensor of the unpack-concat operation pair. The example listed in Figure 7(c)
violates the assumption and by theory cannot be simplified to a reshaping
logic. However, the FuseUnpackAndConcatToReshape function still erroneously
transforms it into a wrong reshaping operation. Notably, this bug is
exclusively detected by WhiteFox since it hinges on generating valid tests
that can trigger this particular optimization, a task that other techniques
consistently struggle to accomplish.

Figure 7(d) shows a confirmed TensorFlow-XLA bug exclusively detected by
WhiteFox. The bug-triggering model contains consecutive fully connected layers
followed by two constant addition operations, which causes result
inconsistency between the optimized and non-optimized versions.

Figure 7(e) presents a confirmed bug detected in LLVM. This bug is revealed by
a C++ program that involves indexing a large integer array, leading to a crash
in LLVM post-optimization. Interestingly, this bug is only triggered when
initializing a large index array and accessing its element with a sufficiently
large index.

##  7\. Conclusion

In this paper, we present WhiteFox, the first white-box compiler fuzzer to
comprehensively test compiler optimizations. Specifically, WhiteFox adopts a
dual-model design: An analysis LLM reads through the implementation code of
compiler optimizations and summarizes desired patterns of test programs. A
generation LLM is then prompted with the summarized patterns to efficiently
and continuously synthesize meaningful test programs in order to exercise
corresponding optimizations. Our evaluation shows that our WhiteFox approach
is promising for testing both traditional C/C++ compilers as well as the
emerging DL compilers. To date, WhiteFox has found in total 96 bugs, with 80
confirmed as previously unknown and 51 already fixed.

## References

  * (1) Agarap, A. F. Deep learning using rectified linear units (relu).  arXiv preprint arXiv:1803.08375 (2018).
  * (2) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners.  Advances in neural information processing systems 33 (2020), 1877–1901.
  * (3) Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. Sparks of artificial general intelligence: Early experiments with gpt-4.  arXiv preprint arXiv:2303.12712 (2023).
  * (4) Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code.  arXiv preprint arXiv:2107.03374 (2021).
  * (5) Chen, T., Moreau, T., Jiang, Z., Zheng, L., Yan, E., Shen, H., Cowan, M., Wang, L., Hu, Y., Ceze, L., et al. $\\{$TVM$\\}$: An automated $\\{$End-to-End$\\}$ optimizing compiler for deep learning.  In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18) (2018), pp. 578–594.
  * (6) Chen, Y., Zhong, R., Hu, H., Zhang, H., Yang, Y., Wu, D., and Lee, W. One engine to fuzz’em all: Generic language processor testing with semantic validation.  In 2021 IEEE Symposium on Security and Privacy (SP) (2021), IEEE, pp. 642–658.
  * (7) Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways.  arXiv preprint arXiv:2204.02311 (2022).
  * (8) Deng, Y., Xia, C. S., Peng, H., Yang, C., and Zhang, L. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models.  In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (2023), ISSTA 2023.
  * (9) Deng, Y., Xia, C. S., Yang, C., Zhang, S. D., Yang, S., and Zhang, L. Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt.  arXiv preprint arXiv:2304.02014 (2023).
  * (10) Deng, Y., Yang, C., Wei, A., and Zhang, L. Fuzzing deep-learning libraries via automated relational api inference.  In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (New York, NY, USA, 2022), ESEC/FSE 2022, Association for Computing Machinery, p. 44–56.
  * (11) Donaldson, A. F., Evrard, H., Lascu, A., and Thomson, P. Automated testing of graphics shader compilers.  Proceedings of the ACM on Programming Languages 1, OOPSLA (2017), 1–29.
  * (12) Donaldson, A. F., Thomson, P., Teliman, V., Milizia, S., Maselco, A. P., and Karpiński, A. Test-case reduction and deduplication almost for free with transformation-based compiler testing.  In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation (2021), pp. 1017–1032.
  * (13) Even-Mendoza, K., Sharma, A., Donaldson, A. F., and Cadar, C. Grayc: Greybox fuzzing of compilers and analysers for c.
  * (14) Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. Codebert: A pre-trained model for programming and natural languages.  arXiv preprint arXiv:2002.08155 (2020).
  * (15) Fraser, G., and Arcuri, A. Evosuite: automatic test suite generation for object-oriented software.  In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering (2011), pp. 416–419.
  * (16) Gcc, 2023.  https://gcc.gnu.org/.
  * (17) Gu, J., Luo, X., Zhou, Y., and Wang, X. Muffin: Testing deep learning libraries via neural architecture fuzzing.  In 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE) (Los Alamitos, CA, USA, may 2022), IEEE Computer Society, pp. 1418–1430.
  * (18) Guo, Q., Xie, X., Li, Y., Zhang, X., Liu, Y., Li, X., and Shen, C. Audee: Automated testing for deep learning frameworks.  In 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE) (2020), IEEE, pp. 486–498.
  * (19) He, K., Zhang, X., Ren, S., and Sun, J. Identity mappings in deep residual networks.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14 (2016), Springer, pp. 630–645.
  * (20) Holler, C., Herzig, K., and Zeller, A. Fuzzing with code fragments.  In 21st USENIX Security Symposium (USENIX Security 12) (2012), pp. 445–458.
  * (21) Hugging face, 2023.  https://huggingface.co.
  * (22) King, J. C. Symbolic execution and program testing.  Communications of the ACM 19, 7 (1976), 385–394.
  * (23) Lattner, C., and Adve, V. Llvm: A compilation framework for lifelong program analysis & transformation.  In International symposium on code generation and optimization, 2004. CGO 2004. (2004), IEEE, pp. 75–86.
  * (24) Le, V., Afshari, M., and Su, Z. Compiler validation via equivalence modulo inputs.  ACM Sigplan Notices 49, 6 (2014), 216–226.
  * (25) Le, V., Sun, C., and Su, Z. Finding deep compiler bugs via guided stochastic program mutation.  ACM SIGPLAN Notices 50, 10 (2015), 386–399.
  * (26) Lemieux, C., Inala, J. P., Lahiri, S. K., and Sen, S. Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models.  In International conference on software engineering (ICSE) (2023).
  * (27) Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. Starcoder: may the source be with you!  arXiv preprint arXiv:2305.06161 (2023).
  * (28) libfuzzer – a library for coverage-guided fuzz testing., 2023.  https://llvm.org/docs/LibFuzzer.html.
  * (29) Liu, J., Lin, J., Ruffy, F., Tan, C., Li, J., Panda, A., and Zhang, L. Nnsmith: Generating diverse and valid test cases for deep learning compilers.  In ASPLOS (2023), p. 530–543.
  * (30) Liu, J., Peng, J., Wang, Y., and Zhang, L. Neuri: Diversifying dnn generation via inductive rule inference.  arXiv preprint arXiv:2302.02261 (2023).
  * (31) Liu, J., Wei, Y., Yang, S., Deng, Y., and Zhang, L. Coverage-guided tensor compiler fuzzing with joint ir-pass mutation.  Proc. ACM Program. Lang. 6, OOPSLA1 (apr 2022).
  * (32) Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts.  arXiv preprint arXiv:2307.03172 (2023).
  * (33) Livinskii, V., Babokin, D., and Regehr, J. Random testing for c and c++ compilers with yarpgen.  Proceedings of the ACM on Programming Languages 4, OOPSLA (2020), 1–25.
  * (34) McDonald, J. B., and Xu, Y. J. A generalization of the beta distribution with applications.  Journal of Econometrics 66, 1-2 (1995), 133–152.
  * (35) McKeeman, W. M. Differential testing for software.  Digital Technical Journal 10, 1 (1998), 100–107.
  * (36) Nie, P., Banerjee, R., Li, J. J., Mooney, R. J., and Gligoric, M. Learning deep semantics for test completion.  arXiv preprint arXiv:2302.10166 (2023).
  * (37) OpenAI.  Chatgpt.  https://openai.com/blog/chatgpt.
  * (38) OpenAI.  Gpt-4 technical report, 2023.
  * (39) Pham, H. V., Lutellier, T., Qi, W., and Tan, L. CRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries.  In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) (2019), pp. 1027–1038.
  * (40) Pytorch, 2023.  http://pytorch.org.
  * (41) Pytorch 2.0, 2023.  https://pytorch.org/get-started/pytorch-2.0.
  * (42) Schäfer, M., Nadi, S., Eghbali, A., and Tip, F. Adaptive test generation using a large language model.  arXiv preprint arXiv:2302.06527 (2023).
  * (43) Security, M. jsfunfuzz.  https://github.com/MozillaSecurity/funfuzz, 2007.
  * (44) Sen, K. Concolic testing.  In Proceedings of the 22nd IEEE/ACM international conference on Automated software engineering (2007), pp. 571–572.
  * (45) Shreiner, D., et al. OpenGL programming guide: the official guide to learning OpenGL, versions 3.0 and 3.1.  Pearson Education, 2009.
  * (46) Sun, C., Le, V., Zhang, Q., and Su, Z. Toward understanding compiler bugs in gcc and llvm.  In Proceedings of the 25th international symposium on software testing and analysis (2016), pp. 294–305.
  * (47) Sun, M., Yang, Y., Wang, Y., Wen, M., Jia, H., and Zhou, Y. Smt solver validation empowered by large pre-trained language models.  In ASE (2023).
  * (48) Sutton, M., Greene, A., and Amini, P. Fuzzing: brute force vulnerability discovery.  Pearson Education, 2007.
  * (49) Tensorflow, 2023.  https://www.tensorflow.org.
  * (50) Tensorflow lite, 2023.  https://www.tensorflow.org/lite.
  * (51) Tensorflow xla, 2023.  https://www.tensorflow.org/xla.
  * (52) Thompson, W. R. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples.  Biometrika 25, 3-4 (1933), 285–294.
  * (53) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need.  Advances in neural information processing systems 30 (2017).
  * (54) Wang, J., Lutellier, T., Qian, S., Pham, H. V., and Tan, L. Eagle: Creating equivalent graphs to test deep learning libraries.
  * (55) Wang, Z., Yan, M., Chen, J., Liu, S., and Zhang, D. Deep learning library testing via effective model generation.  In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (2020), pp. 788–799.
  * (56) Wei, A., Deng, Y., Yang, C., and Zhang, L. Free lunch for testing: Fuzzing deep-learning libraries from open source.  In 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE) (2022), pp. 995–1007.
  * (57) Xia, C. S., Paltenghi, M., Tian, J. L., Pradel, M., and Zhang, L. Universal fuzzing via large language models.  arXiv preprint arXiv:2308.04748 (2023).
  * (58) Xie, D., Li, Y., Kim, M., Pham, H. V., Tan, L., Zhang, X., and Godfrey, M. W. Docter: documentation-guided fuzzing for testing deep learning api functions.  In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (2022), pp. 176–188.
  * (59) Yang, C., Deng, Y., Yao, J., Tu, Y., Li, H., and Zhang, L. Fuzzing automatic differentiation in deep-learning libraries.  In Proceedings of the 45th International Conference on Software Engineering (2023), ICSE ’23, IEEE Press, p. 1174–1186.
  * (60) Yang, X., Chen, Y., Eide, E., and Regehr, J. Finding and understanding bugs in c compilers.  In Proceedings of the 32nd ACM SIGPLAN conference on Programming language design and implementation (2011), pp. 283–294.
  * (61) Yuan, Z., Lou, Y., Liu, M., Ding, S., Wang, K., Chen, Y., and Peng, X. No more manual tests? evaluating and improving chatgpt for unit test generation.  CoRR abs/2305.04207 (2023).
  * (62) Yue Wang, Weishi Wang, S. J., and Hoi, S. C. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation.  In EMNLP 2021 (2021).
  * (63) Zeller, A., Gopinath, R., Böhme, M., Fraser, G., and Holler, C. The fuzzing book, 2019.
  * (64) Zhang, Q., Sun, C., and Su, Z. Skeletal program enumeration for rigorous compiler testing.  In Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation (2017), pp. 347–361.