# COMET: Coverage-guided Model Generation For Deep Learning Library Testing

Meiziniu Li  0000-0001-5947-4030 The Hong Kong University of Science and
TechnologyChina<EMAIL_ADDRESS>,  Jialun Cao  0000-0003-4892-6294 The Hong
Kong University of Science and TechnologyChina Guangzhou HKUST Fok Ying Tung
Research InstituteChina<EMAIL_ADDRESS>,  Yongqiang Tian
0000-0003-1644-2965 University of WaterlooCanada The Hong Kong University of
Science and TechnologyChina<EMAIL_ADDRESS>,  Tsz On Li
0000-0001-8031-4947 The Hong Kong University of Science and TechnologyChina
<EMAIL_ADDRESS>,  Ming Wen  0000-0001-5588-9618 Huazhong University of
Science and TechnologyChina<EMAIL_ADDRESS>and  Shing-Chi Cheung
0000-0002-3508-7172 The Hong Kong University of Science and TechnologyChina
Guangzhou HKUST Fok Ying Tung Research InstituteChina<EMAIL_ADDRESS>
(2023)

###### Abstract.

Recent deep learning (DL) applications are mostly built on top of DL
libraries. The quality assurance of these libraries is critical to the
dependable deployment of DL applications. Techniques have been proposed to
generate various DL models and apply them to test these libraries. However,
their test effectiveness is constrained by the diversity of layer API calls in
their generated DL models. Our study reveals that these techniques can cover
at most 34.1% layer inputs, 25.9% layer parameter values, and 15.6% layer
sequences. As a result, we find that many bugs arising from specific layer API
calls (i.e., specific layer inputs, parameter values, or layer sequences) can
be missed by existing techniques.

Because of this limitation, we propose COMET to effectively generate DL models
with diverse layer API calls for DL library testing. COMET: (1) designs a set
of mutation operators and a coverage-based search algorithm to diversify layer
inputs, layer parameter values, and layer sequences in DL models. (2) proposes
a model synthesis method to boost the test efficiency without compromising the
layer API call diversity. Our evaluation result shows that COMET outperforms
baselines by covering twice as many layer inputs (69.7% vs. 34.1%), layer
parameter values (50.2% vs. 25.9%), and layer sequences (39.0% vs. 15.6%) as
those by the state-of-the-art. Moreover, COMET covers 3.4% more library
branches than those by existing techniques. Finally, COMET detects 32 new bugs
in the latest version of eight popular DL libraries, including TensorFlow and
MXNet, with 21 of them confirmed by DL library developers and 7 of those
confirmed bugs have been fixed by developers.

Deep Learning Testing, Library Testing, Model Generation, Model Diversity

* Corresponding author.

††copyright: acmcopyright††journalyear: 2023††doi: XXXXXXX.XXXXXXX††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††journal: TOSEM††journalvolume:
1††journalnumber: 1††article: 1††publicationmonth: 1††ccs: Software and its
engineering Software testing and debugging††ccs: Software and its engineering
Software libraries and repositories††ccs: Computing methodologies Neural
networks

##  1\. Introduction

Deep learning (DL) has been increasingly adopted for mission-critical
applications such as authentication (Das et al., 2018; Ferdowsi and Saad,
2019), medical treatment (Litjens et al., 2017), pandemic control (Ardakani et
al., 2020; Bhattacharya et al., 2020), and autonomous driving (Chen et al.,
2015b; Sallab et al., 2017; Shalev-Shwartz et al., 2016). Currently, many
recent works focus on assuring the quality of DL applications, e.g., testing
the trained DL models (Pei et al., 2019; Odena et al., 2019; Zhang et al.,
2018) or testing the DL program written by DL application developers (Zhang et
al., 2020; Wardat et al., 2021). However, there are only a few research works
on testing DL libraries, such as TensorFlow (Abadi et al., 2016), PyTorch
(Paszke et al., 2019), MXNet (Chen et al., 2015a), and ONNX (Onnx, 2022a).
These DL libraries are the basis during the development and deployment of
current DL applications. Nevertheless, recent studies (Jia et al., 2021b; Chen
et al., 2022) found that these libraries suffer from coding bugs frequently.
Early detection of these bugs is crucial.

Prior works (Pham et al., 2019; Wang et al., 2020; Guo et al., 2020; Gu et
al., 2022; Luo et al., 2021) mainly follow the same testing paradigm, i.e.,
feeding various DL models (called test inputs) to exercise specific modules
(e.g., model loading, construction, and inference modules) in DL libraries. To
obtain sufficient test inputs (i.e., DL models), existing works either
generate DL models by mutating the network structures (Wang et al., 2020; Guo
et al., 2020; Luo et al., 2021), parameters (Wang et al., 2020; Guo et al.,
2020), or input value (Pham et al., 2019; Wang et al., 2020; Guo et al., 2020;
Luo et al., 2021) of published models (e.g., ResNet (He et al., 2016) and
InceptionNet (Szegedy et al., 2015)) or build DL models from scratch (Gu et
al., 2022) based on predefined model structure templates (e.g., chain
structure (Gu et al., 2022)).

Figure 1. An Example of A Layer API Call In DL Model

However, the test adequacy of existing works is far from satisfactory (e.g.,
achieving low test coverage (Chen et al., 2022)). According to our
investigation (see §2.3), only 16% of branches in TensorFlow’s model
construction and model execution modules could be covered even using the
latest testing technique. The unsatisfactory result reveals the inadequacy of
existing techniques. Taking a closer look, we observed that a main reason is
the lack of diversity of test inputs (i.e., DL models). Specifically, existing
techniques target generating DL models to either enlarge the prediction
inconsistency in differential testing (Wang et al., 2020; Guo et al., 2020),
increase the coverage of layer APIs (Gu et al., 2022) or increase the graph
structure diversity (Luo et al., 2021). However, they are weak in generating
DL models to expose bugs that reside in some specific layer API calls. Take
Figure 1 as an example. The diversity of layer API calls inside a DL model is
attributed to three finer properties: (1) layer inputs (e.g., input datatype
and number of input dimensions, abbreviated to dims, highlighted in orange),
(2) layer parameter values (e.g., values of filters and kernel_size
highlighted in blue), and (3) layer sequences (e.g., the connection between
Conv2D layer and BatchNormalization layer highlighted in green). However,
neither covering more types of layer APIs (Gu et al., 2022) nor diversifying
the graph structure (Luo et al., 2021) could encourage generating layer API
calls with diverse layer parameter values or layer inputs. Indeed, we found
that models generated by existing techniques (Pham et al., 2019; Wang et al.,
2020; Gu et al., 2022) do not cover most layer API calls, i.e., at most 34.1%
of layer inputs, 25.9% of layer parameter values, and 15.6% of layer sequences
could be covered (see §2.3).

Consequently, existing techniques cannot effectively detect bugs residing at
specific layer API calls. Figure 2 shows a real bug detected by us that
illustrates the weaknesses of these
techniques.111https://github.com/microsoft/onnxruntime/issues/11024 It is a
crash bug triggered by a variant of the ResNet model, leading to a core dump
in ONNXRuntime, a famous DL library for model inference and training with over
6.5K stars in GitHub. The bug can only be triggered when models with a
specific layer sequence (i.e., Dense$\rightarrow$Dot) and layer input (as
shown on the left with the layer input of Dot layer to has zero shapes) are
generated. Moreover, we found out that 18 out of 21 confirmed new bugs
detected by our tools are manifested by some specific layer API calls. In
other words, testing techniques designed for diversifying layer API calls are
desired for testing DL libraries.

Figure 2. A Crash Bug of ONNXRuntime

To this end, we propose COMET, a COverage-guided Model genEration tool for DL
library Testing. COMET is designed to generate DL models with diverse layer
API calls. Similar to LEMON (Wang et al., 2020) and Audee (Guo et al., 2020),
COMET constructs the initial seeds based on published DL models. It then
generates new DL models by iteratively mutating them. The challenges of
designing COMET are two-fold. First, it is hard to generate DL models with
diverse layer API calls due to the huge search space. For instance,
considering a widely-used network, InceptionResnetV2, with 782 layers and
30,400 layer parameters, by simply considering each possible layer parameter
once, it will take more than 30 thousand times to test, let alone considering
the model structure. Therefore, random mutation (e.g., randomly choosing a
layer into a random position inside a DL model), as is used by existing
techniques (Wang et al., 2020; Guo et al., 2020; Luo et al., 2021)), may
derive a layer API call that previously generated models has already covered,
thus is of no benefit for diversifying layer API calls. All in all, it is
impossible to exhaustively search for all possible cases in the huge search
space with respect to a model. Second, the runtime overhead when generating
and executing a DL model is expensive. As reported by an earlier work (Wei et
al., 2022), testing techniques based on model generation and execution can be
cost-prohibitive. It can take over 10 minutes to process a new DL model based
on InceptionResNetV2 on an Intel GPU server, which poses a significant
challenge to test efficiency. Iteratively mutating these published models,
mostly large, incurs non-trivial runtime overhead in model generation.

The insight of COMET to address these two challenges is based on the
observation: generating DL models with new layer inputs, parameter values, or
sequences is likely to invoke different layer API calls, thus achieving more
comprehensive testing on DL libraries. Driven by the insight, we propose
COMET, which mainly contains three novelties. First, we design a set of
mutation operators222Different from mutation testing works (Ma et al., 2018;
Odena et al., 2019; Humbatova et al., 2021; Wang et al., 2019) that insert
error into DL models and consider them as subject under test. Our work uses
mutation operators to generate new DL models and we further uses the generated
DL models as test inputs for DL library testing. aiming to target diversifying
layer API calls by mutating the layer inputs, layer parameter values, and the
layer sequence inside the DL model. Instead of random mutation, our mutation
operators search for the optimal mutation option to target mutating DL models
with diverse layer API calls. Second, since not all mutation operators are
equally effective in diversifying DL models and increasing test coverage, we
employ the Monte Carlo Markov Chain (MCMC) algorithm with the coverage-based
fitness function to iteratively mutate seed models to achieve higher layer API
call diversity and test coverage. Third, we design a model synthesis method
for published models to reduce their size while preserving the diversity of
layer inputs, parameters, and sequences. Benefiting from the model synthesis
method, COMET can reduce the runtime overhead during model generation and
execution.

We evaluate the performance of COMET on eight popular DL libraries: Keras
(Chollet et al., 2015), TensorFlow (Abadi et al., 2016), PyTorch (Paszke et
al., 2019), MXNet (Chen et al., 2015a), ONNXRuntime (Runtime, 2022), Keras-
MXNet (Mxnet, 2022), TF2ONNX (Onnx, 2022b), ONNX2PyTorch (Onnx2pytorch, 2022).
In particular, we adopt differential testing for bug detection, i.e., the
inconsistencies are compared between DL libraries. While the DL model’s
implementation varies from library to library, a model conversion step is thus
needed to enable comparison. Therefore, the first five libraries (i.e., Keras,
TensorFlow, PyTorch, MXNet, and ONNXRuntime) are tested using differential
testing, exercising their model construction and inference modules. While the
rest libraries (i.e., Keras-MXNet, TF2ONNX, ONNX2PyTorch) are used for model
conversion, thus their conversion modules are tested. We compare COMET with
three state-of-the-art DL library testing techniques as baselines: CRADLE
(Pham et al., 2019), LEMON (Wang et al., 2020), and the latest technique:
Muffin (Gu et al., 2022). For GraphFuzz (Luo et al., 2021) and Audee (Guo et
al., 2020), since no executables or source code are available, we do not
include them as baselines. The evaluation results show that COMET’s mutation
operators and search algorithm can significantly increase the diversity of
layer API calls in the generated models, outperforming baselines by covering
35.6% more layer inputs, 24.3% more parameter values, and 23.4% more
sequences. Moreover, COMET can cover at least 3.4% more branches than
baselines in TensorFlow’s model construction and model execution modules.
COMET successfully finds 32 new DL library bugs. So far, 21 of them have been
confirmed by DL library developers, and 7 of the confirmed ones have already
been fixed by developers in the latest version, the remaining 11 unconfirmed
bugs are pending review.

In summary, our work makes the following major contributions:

  * •

Originality. We propose COMET, a diversity-driven model generation framework
for DL library testing, which can improve the test coverage of existing
techniques by increasing the diversity of layer API calls inside DL models.
Specifically, we propose three coverage criteria for layer API call diversity.
We further propose a set of mutation operators and a coverage-based MCMC
search algorithm to diversify layer API calls. In addition, we design a model
synthesis method based on our coverage criteria to boost the efficiency of
COMET significantly.

  * •

Effectiveness. Our evaluation shows that COMET outperforms the state-of-the-
art by covering 35.6% more layer inputs, 24.3% more layer parameters, 23.4%
more layer sequences, and 3.4% more library branches.

  * •

Usefulness. We applied COMET to eight popular DL libraries. It successfully
detected 32 new bugs that caused crashes, NaN, and inconsistent outputs. 21
out of these 32 new bugs have been confirmed by the developers. 7 of those
confirmed bugs have been fixed. Our implementation of COMET, the details of
detected bugs together with experiment results are publicly available at our
project site: https://github.com/maybeLee/COMET.

##  2\. Background and Motivation

###  2.1. Testing DL Libraries by DL Models

Prior testing techniques commonly use DL models as test input to test DL
libraries. As illustrated in Figure 3, feeding these models to DL libraries
for making inferences will execute code in the loading and inferencing
modules. For test oracle design, they adopt differential testing to apply the
same DL model on multiple DL libraries with model conversion tools or shared
high-level APIs. Inconsistent behavior (i.e., inconsistent outputs or
inconsistent symptoms between DL libraries) indicates potential library bugs.
They further design a set of mutation operators or generation rules to obtain
sufficient test inputs (i.e., DL models). These techniques complement the unit
test suites of a DL library by invoking a sequence of library APIs for the
end-to-end completion of machine learning.

Figure 3. Overview of Testing DL Library by DL Models.

CRADLE (Pham et al., 2019) is the earliest technique proposed to test DL
libraries by collecting publicly-available models and datasets to detect
inconsistent outputs across DL libraries. It successfully detects
implementation differences of Keras across different library backends. Based
on these existing DL models, two model generation techniques, LEMON (Wang et
al., 2020) and Audee (Guo et al., 2020), were later proposed to generate DL
models driven by output inconsistency or NaN symptoms. These two techniques
present a set of mutation operators, such as mutating the model’s weights,
architectures, and layer parameters to explore library code. For guiding model
mutation, they either use the inconsistency of generated models’ outputs over
different DL libraries as the guidance to expose the incorrect computation or
use a heuristic-based fitness function to encourage mutating DL models that
tend to output outliers (i.e., too large or small values) to expose NaN
values. GraphFuzz (Luo et al., 2021) is further proposed to mutate the
computation graph in the DL model for DL library testing. It introduces an
operator-level coverage to quantify the model diversity and further designs a
set of mutation operators to explore combinations of model structures and
parameters. Muffin (Gu et al., 2022) is the most recent DL library testing
work that generates DL models driven by layer diversity. In particular, Muffin
abstracts the DL model as a Directed Acyclic Graph (DAG) and defines two types
of commonly used graph structures: chain structure and cell-based structure to
generate the structure information (i.e., the topology of how layers are
connected); it further generates DL models towards increasing the coverage of
layer APIs.

###  2.2. Model Generation Strategies

Existing techniques, as mentioned in §2.1, adopt different strategies in model
generation. For example, LEMON generates models to enlarge the output
inconsistency across DL libraries. Besides output inconsistency, Audee also
favors generating models that can trigger the Not-A-Number (NaN) value in DL
libraries.333Bugs detected by Audee are available at the site:
https://sites.google.com/view/audee/bug-details. GraphFuzz designs an
operator-level coverage defined as a sum of five coverage criteria (i.e.,
operator type coverage, input degree coverage, output degree coverage, single
edge coverage, shape&parameter coverage) derived from graph theory. It further
iteratively mutates models towards a higher operator-level coverage
result.444Exceptions detected by GraphFuzz are available at the site:
https://github.com/gbftdlie/Graph-based-fuzz-testing. Muffin adopts the
strategy of maximizing layer type coverage (i.e., the percentage of invoked
layer APIs in all the pre-defined layer APIs) in model generation.

When testing DL libraries using DL models, the code executed in DL libraries
is related to the layer API calls (as is exemplified in Figure 1) used to
define the DL models. First, diversifying layer input can help better test DL
libraries. Specifically, changing the datatype of layer input can test those
precision-specific codes in DL libraries (Wang et al., 2016). For instance, in
TensorFlow, changing the input datatype of BatchNormalization ( or
LayerNormalization) from float32 to float16 (or non-float32) will trigger new
branches handling numeric instability (or layer fusion) issues. Changing the
number of dimensions for a tensor will change the data formats it represents
(e.g., 3D tensors usually refer to text data, and 4D tensors represent image
data), thus may test new code in DL libraries. For example, changing the input
of BatchNormalization layer from a 4D tensor to a 5D tensor will trigger new
functions related to operator fusion during the model construction. Changing
the shape of layer input may also affect library code, such as codes related
to weights initialization and tensor padding during the model construction. An
example of this observation is that different tensor input shapes may affect
how the tensor is padded (Strategy, 2022). Besides, our experiment also found
the manifestation of some bugs required a specific layer input shape during
the model conversion (Padding, 2022). Second, diversifying layer parameter
values can also help increase the test coverage. For instance, changing the
value of activation from relu to sigmoid will test a different activation
function in Conv2D. Changing the value of strides and dilation_rate may result
in the incorrect padding logic inside the Conv2D layer, which is a confirmed
bug (DepthwiseConv2D, 2022) we detected. Setting units of the Dense layer to 0
will generate an empty-shape tensor which may result in a direct core dump if
this empty-shape tensor is directed to a Conv3DTranspose layer
(Conv3DTranspose, 2022). Changing the value of kernel_size will also influence
how Conv2D does the padding when constructing the model (Keras, 2022). Third,
testing DL libraries with different layer sequences can also help to explore
library code. On the one hand, introducing a layer sequence with new layer
types can test more layer APIs (Gu et al., 2022). On the other hand, covering
some specific layer sequences may also cover new codes related to model
construction and model computation inside DL libraries. For instance, the
layer sequence Conv2D$\rightarrow$ReLU will trigger the operator fusion for
Conv2D and Mul operator (Conv2D and Operators, 2022) and the
FindContractionWithBias function in TensorFlow.

However, existing model generation strategies are weak in diversifying layer
API calls in DL models. First, enlarging output inconsistency has little help
in diversifying layer API calls. Second, only focusing on layer type coverage
can offer no guidance for diversifying other properties, i.e., layer parameter
values, layer inputs, and layer sequences. Although the operator-level
coverage proposed by GraphFuzz provides a more optimal granularity in guiding
model generation to increase layer sequence diversity, they are still weak in
diversifying layer parameters and layer inputs. Therefore, some library code
related to some specific layer API calls cannot be covered by existing
techniques. We observed that missing specific layer API calls may fail to
detect some library bugs. This observation motivates us to define more optimal
criteria for measuring the layer API call diversities, including layer
sequence diversity adapted from existing works (Gu et al., 2022; Luo et al.,
2021) and two newly proposed ones: layer input diversity and layer parameter
diversity. Specifically, layer input diversity includes the variety of input’s
datatype (e.g., float16), input dimension (e.g., 4D), and input shape (e.g.,
(batch_size,24,24,3)); layer parameter diversity includes the diversities of
parameter values for each type of layer; layer sequence diversity comprises
the flow of tensor from one layer API to another. All these three diversities
collectively contribute to the diversity of layer API calls. Accordingly, the
core objective of our paper is to better test DL libraries by increasing the
diversity of layer inputs, parameters, and sequences in the generated models;
thus, we can achieve higher test coverage.

###  2.3. Limitations of Existing Techniques

To explore the limitations of baselines (Pham et al., 2019; Wang et al., 2020;
Gu et al., 2022), we experimented with measuring their performance in
diversifying layer API calls. In particular, we use our proposed three
coverage criteria: layer input coverage, layer parameter coverage, and layer
sequence coverage (see §3.2) as the metrics. Additionally, we record the
branch coverage on TensorFlow’s model construction and model execution modules
achieved by existing works as the representative to reveal their inadequate
testing effectiveness. We run each technique with the default hyperparameters
for six hours and then analyze the coverage results. The experiment results
show low coverage with respect to all four coverage criteria, suggesting the
limited effectiveness of existing works in diversifying layer API calls and
achieving satisfactory test coverage on DL libraries. Specifically, only at
most 34.1% of layer inputs (261/766), 15.6% of layer sequences (338/2170), and
25.9% of layer parameter values (530/2049) have been covered by existing
techniques. As a result, only at most 16.0% of library branches are covered.
Moreover, we found that bugs (like Figure 2) residing at some specific layer
API calls and library branches are not detected by existing techniques.
However, these bugs may lead to severe consequences (e.g., a core dump), which
may affect the reliability of DL libraries. Motivated by this finding, we
propose COMET to generate diverse models to increase the test coverage for DL
library testing.

Table 1. Various Coverages by Existing Techniques

Technique | Layer Input | Layer Parameter | Layer Sequence | Branch
---|---|---|---|---
Coverage | Coverage | Coverage | Coverage
MUFFIN | 34.1% | 25.9% | 15.6% | 16.0%
LEMON | 14.2% | 6.4% | 2.4% | 12.5%
CRADLE | 10.4% | 5.9% | 1.2% | 12.0%

##  3\. Methodology

###  3.1. Overview

Our work, COMET, is designed to generate DL models with diverse layer API
calls effectively. As defined earlier, a layer API call consists of three
properties: layer input, parameter values, and sequences. We first propose
three coverage criteria (in §3.2) for the layer API call to measure its
diversity. Based on our coverage criteria, we design the workflow of COMET
into three steps (Figure 4): (1) Initial Model Synthesis, (2) Diverse Model
Generation, and (3) Library Testing. We first apply our model synthesis (§3.4)
method on published DL models to synthesize the initial seeds with small size
while retaining the published DL models’ layer API call diversity. We further
apply our mutation operators (§3.3) and MCMC algorithm (§3.5) to iteratively
generate models toward higher layer API call diversity and higher test
coverage in DL libraries. Finally, we use these generated models as test input
for library testing (§3.6).

Figure 4. Overview of COMET

###  3.2. Coverage Criteria For layer API calls

Motivated by Figure 1, we focus on three properties (i.e., layer input, layer
parameter values, and layer sequences) to measure the diversity of layer API
calls. The space of these properties is summarized in Table 2. Specifically, a
diverse layer API call is found if a DL model contains new layer inputs,
parameter values, or sequences that previously generated models have not
covered. Based on these three properties, we further introduce three coverage
criteria (i.e., layer input coverage, layer parameter coverage, and layer
sequence coverage) to measure the parts of layer API calls exercised by the
generated models.

Table 2. Components In Layer API Calls

Property | Element | Space
---|---|---
Layer Input | Datatype |  | Layer A/B/…:
---
int16 — int32 — float16 — …
Shape |  | Layer A/B/…:
---
shape 1 — shape 2 — shape 3 — …
Number of Dimensions |  | Layer A/B/…:
---
2D — 3D — 4D — 5D — …
Layer Parameter | Parameter Value |  | Layer A:
---
Param A1: v1 — v2 — …
Param A2: v1 — v2 — …
…
Layer B:
Param B1: v1 — v2 — …
Param B2: v1 — v2 — …
…
…
Layer Sequence | Pair of Layers |  | Layer A $\rightarrow$ Layer B
---
Layer B $\rightarrow$ Layer A
Layer A $\rightarrow$ Layer A
…

####  3.2.1. Layer Input Coverage

A layer input is a tensor (i.e., a multi-dimensional array). The datatype,
number of dimensions, and shape (i.e., the number of elements in each
dimension) are three essential properties that determine an array. As
exemplified by the tensor x in Figure 1, the datatype determines the floating
point format stored in the computer (e.g., float32 means x is stored in the
single-precision format). The number of dimensions (abbreviated to “dims”) of
x is 4D means that x contains four dimensions. Usually, dimensions represent
different meanings in DL libraries (e.g., in TensorFlow, the four dimensions
in x refer to the batch size, height, width, and the number of channels,
respectively). The shape of x determines the number of elements in each
dimension (e.g., x is a three-channel tensor with the length of width and
height to be 8). Motivated in § 2.2, mutating the datatype, number of
dimensions, or shape for a layer’s input may trigger different usage of
library code. Therefore, for an input of a layer, we capture these three
properties: input datatype, input dimension, and input shape. However, the
total number of possible values for each property is already huge (e.g., the
state-of-the-art (Gu et al., 2022) can cover only 10.7% input datatype, 41%
input dimension, and 34.1% input shape). Exhaustive testing that also
considers the interaction between these properties will significantly boost
the test space (Kuhn et al., 2004), and it may be expensive to find bugs
related to only a specific value of input datatype, input dimension, or input
shape. Therefore, our layer input coverage encourages covering all possible
values for each property rather than testing all combinations. Accordingly, we
define our layer input coverage for each layer type $l$ using Equation 1.
Given the number of covered datatypes ($cov_{type}$), dimensions
($cov_{dim}$), shapes ($cov_{shape}$), and the total number of possible
datatypes ($n_{type}$), dimensions ($n_{dim}$), and shapes ($n_{shape}$), the
layer input coverage $Cov_{I}$ measures the portion of values covered for each
property.

(1) $\displaystyle
Cov_{I}(l)=\frac{cov_{type}(l)+cov_{dim}(l)+cov_{shape}(l)}{n_{type}(l)+n_{dim}(l)+n_{shape}(l)}$

####  3.2.2. Layer Parameter Diversity

To measure the layer parameter diversity, for each layer type $l$, we record
the possible parameter values $v$ for each parameter $p$ inside $l$. Layer
parameters can be briefly grouped into two categories based on the type of the
parameter value: numeric parameter and non-numeric parameter. Numeric
parameter refers to those that require a number or a list of numbers as
parameter values, e.g., the rate for the Dropout layer, kernel_size for the
Conv2D layer). Non-numeric parameter refers to those that require non-numeric
values such as string or boolean as parameter values, e.g., padding for the
Conv2D layer and activation for the LSTM layer. Potential parameter values for
non-numeric parameters are usually categorical for each layer and can be
listed following the library documentation. In contrast, the numeric parameter
for each layer often has a huge value space. Since enumerating all possible
values for a numeric parameter is impossible and may not help to test DL
libraries, we approximate the size of the value space using a finite integer
$\sigma$. Accordingly, for layer API $l$, given the number of covered
parameter values $cov_{pi}$ and the total number of possible parameter values
$n_{pi}$ (for numeric parameters, $n_{pi}=\sigma$) for each parameter $p_{i}$
inside a layer API $l$, we define the layer parameter coverage $Cov_{P}$ for
layer $l$ in Equation 2, where $N$ is the total number of mutable parameters
in $l$. In our experiment, we set $\sigma=5$.

(2) $\displaystyle
Cov_{P}(l)=\frac{\sum^{N}_{i=1}cov_{pi}}{\sum^{N}_{i=1}n_{pi}}$

####  3.2.3. Layer Sequence Diversity

Layer sequence diversity defines the execution sequence of layer APIs inside
the DL model. Following GraphFuzz (Luo et al., 2021), we denote the layer pair
as ¡$layer_{i}$, $layer_{j}$¿, indicating the tensor is first sent to
$layer_{i}$, and then the output of $layer_{j}$ is sent into $layer_{j}$. In
other words, the space of layer API pairs is all possible permutations between
all layer APIs. Note that not all layer sequences are valid. For instance, the
output of the Conv3D layer is always a 5D tensor, and this tensor cannot be
sent to the Conv2D layer, which only accepts a 4D tensor. We refer to this
constraint as the dimension constraint (see Figure 5). In other words, any
layer sequence that violates this dimension constraint is invalid. To
construct the valid layer sequence, we manually collect the set of the
possible number of input dimensions (denoted as $s_{idim}(l)$) and output
dimensions (denoted as $s_{odim}(l)$) for each layer $l$. Finally, given the
total number of layer types is $N_{layer}$ and the total number of covered
layer sequences $cov_{seq}$ (note that all covered layer sequences are valid),
we define the layer sequence coverage $Cov_{S}$ as below:

(3) $\displaystyle
Cov_{S}=\frac{cov_{seq}}{\sum_{i\in[1,N_{layer}]}{\sum_{j\in[1,N_{layer}]}{\text{sign}(|s_{odim(layer_{i})}\cap
s_{idim(layer_{j})}|)}}}$

Specifically, we permute all the possible layer sequences and check if they
are valid. For each layer sequence ($layer_{i}$, $layer_{j}$), we consider it
to be valid if $layer_{i}$ has at least one output tensor whose number of
dimensions is accepted by $layer_{j}$ (i.e., $s_{odim(layer_{i})}\cap
s_{idim(layer_{j})}\neq\varnothing$).

###  3.3. Diversity-Driven Mutation

Existing model mutation operators (Wang et al., 2020; Ma et al., 2018; Guo et
al., 2020; Luo et al., 2021) are mainly designed to generate mutants by
changing the structure or weights of seed DL models. The design, however, does
not consider the diversity of layer API calls inside DL models. Examples of
these operators are inverting the activation state of neurons and layer
duplication. However, these mutation operators fail to introduce the new layer
API call (i.e., new layer inputs, layer parameters, or layer sequences).
Therefore, we propose eight mutation operators in this section to generate
diverse mutants for the three coverage criteria. As shown in Table 3, the
mutation operators could be categorized into three levels, i.e., input-level
mutation, parameter-level mutation, and structure-level mutation. In addition,
we include one anomaly mutation for robustness testing.

Our mutation operators are designed to increase the diversity of layer API
calls by introducing new layer inputs, parameter values, and layer sequences.
Unfortunately, arbitrarily mutating DL models can easily cause model failure
(Gu et al., 2022; Wang et al., 2020; Luo et al., 2021). In particular, model
mutations must comply with some tensor constraints (i.e., rules) to avoid
breaking validity checks in DL libraries. Specifically, we summarize four
general tensor constraints (as exemplified in Figure 5), including three
followed by existing works (Wang et al., 2020; Gu et al., 2022; Luo et al.,
2021): input degree constraint, dimension constraint, and shape constraint,
and an additional one required by us when mutating the layer input datatype:
datatype constraint. Input degree constraint refers to the required number of
input tensors for each layer, e.g., the Conv2D layer can only receive one
tensor while the Add layer requires more than one tensor. Dimension constraint
refers to the number of tensor dimensions required for each layer, e.g., the
Conv2D layer can only accept the four-dimensional (4D) tensor, while
BatchNormalization can accept 3D, 4D, or 5D tensor. We refer the shape
constraint to the case where merging layers (e.g., Add layer) require a
specific tensor shape (e.g., the Add layer requires the input tensors to have
the same shape). Additionally, inconsistent computation precision (i.e.,
datatype constraint), determined by the datatype property, will also cause
model failure. Specifically, DL libraries will directly raise an exception if
the precision used for layer initialization is inconsistent with the layer
input. For instance, a Conv2D layer initialized in 32 bits floating point
(float32) will reject a float16 input tensor. The model mutation that violates
either of these four constraints will result in an invalid DL model, which
cannot be correctly built and loaded by DL libraries, thus, are ineffective in
testing DL libraries. Therefore, we design our mutation operators to increase
layer API calls’ diversity while not violating four tensor constraints
mentioned above.

Figure 5. Example of Invalid Model Violating Tensor Constraints Table 3.
Mutation Operators

Type | Names | Input | Description
---|---|---|---
Input-Level Mutation | Mutate Datatype (MDtype) | a model; a list of layers | Change the datatype of input for layers in the list.
Mutate Dimension (MDims) | a model; a list of layers | Change the number of input dimensions for layers in the list.
Mutate Shape (MShape) | a model; a list of layers | Change the shape of input for layers in the list.
Parameter-Level Mutation | Mutate Parameter (MParam) | a model; a list of layers | Change the value of layer parameters for layers in the list.
Structure-Level Mutation | Insert Layers (IL) | a model; a list of layers | Insert a list of layers into model.
Merge Layers (ML) | a model; a merging layer | merge two layers’ output using the selected merging layer.
Connect Layers (CL) | a model; a list of layer pairs | connect layers according to the layer pair in the list
Anomaly Mutation | Special Input (SpecialI) | a model; a layer | Change the input of a selected layer to infinity or NaN

####  3.3.1. Input-Level Mutation

Figure 6. Example of Input-Level Mutation (Left Three) and Parameter-Level
Mutation (Right One)

The design of input-level mutation aims to mutate the layer input’s properties
(i.e., datatype, number of dimensions, and shape) inside DL models. However,
because of the huge search space in the layer input, randomly mutating these
properties is weak in exploring new datatype, number of dimensions, and shape.
Intuitively, for the target layer input to be mutated, if previously generated
models have not covered a specific option (i.e., a particular input datatype,
a particular number of dimensions, or a particular shape), our input-level
mutation should have a higher chance to generate mutants with these options.
Moreover, to increase the mutation ratio, Instead of randomly choosing one
layer to mutate, our input-level mutation will randomly choose $n$ layers
(denoted as $\\{layer_{1},layer_{2},...,layer_{n}\\}$), where $n$ is a random
integer between 1 and 10. Based on these intuitions, we design our input-level
mutation, i.e., MDtype, MDims, and MShape, as follows:

  * •

Mutate Datatype (MDtype): Given a seed model and a list of layers
$\\{layer_{1},...,layer_{n}\\}$, MDtype searches through all generated models
and chooses the datatype $dtype_{m}$ ($m\in[1,n_{type}(layer_{i})]$) that have
not been covered by inputs of $layer_{i}$ ($i\in[1,n]$). If all datatypes have
been covered, MDtype randomly chooses one instead. Finally, for each
$layer_{i}$, MDtype changes its input datatype to $dtype_{m}$ by type casting
(e.g., cast the datatype $layer_{i}$’s input from float16 to float64).

  * •

Mutate Dimensions (MDims): Given a seed model and a list of layers
$\\{layer_{1},...,layer_{n}\\}$, MDims searches through all generated models
and chooses the possible number of dimensions $ndim_{i}$ that can be accepted
by $layer_{i}$ and have not been covered by inputs of $layer_{i}$. If all
numbers of dimensions have been covered, MDims randomly choose one instead.
Finally, for each $layer_{i}$, MDims changes the number of its input
dimensions to $ndim_{i}$ by inserting dimensions of length 1 (e.g., expand
$layer_{i}$’s input shape from {10,3,3} to {10,3,3,1}) or randomly dropping
dimensions (e.g., mutate $layer_{i}$’s input shape from {10,3,3} to {10,3}).

  * •

Mutate Shape (MShape): Given a seed model and a list of layers
$\\{layer_{1},...,layer_{n}\\}$, MShape searches through all generated models
and checks if the number of different input shapes of $layer_{i}$ is larger
than $n_{shape}$ (i.e., five as defined in §3.2). If true, MShape generates a
new input shape $shape_{i}$ that preserves the same number of dimensions.
Otherwise, it randomly generates one shape. Finally, for each layer
$layer_{i}$, MShape changes the shape of its input to $shape_{i}$ by padding
with a constant value (e.g., pad the shape of $layer_{i}$ from {10,3,3} to
{10,5,5} with a constant value) or cropping (e.g., crop the shape of
$layer_{i}$ from {10,3,3} to {10,1,1}).

Note that mutating layer inputs may result in improper layer output that
cannot be sent to the subsequent layers. Therefore, we design our input-level
mutation operators as layer-wise mutations, which means we only target
mutating the input of a list of selected layers while not affecting the
subsequent layers. Specifically, when applying MDtype to the layer under
mutated (denoted as $layer_{i}$), we first record the original output datatype
of $layer_{i}$, then cast the $layer_{i}$’s output datatype to the original
one after each mutation. Similarly, when applying MShape and MDims, we also
record $layer_{i}$’s output shape (denoted as $shape^{o}_{i}$). After each
mutation, we check whether the layer output shape after mutation (denoted as
$shape^{m}_{i}$) is the same as $shape^{o}_{i}$. If not, we reshape
$shape^{m}_{i}$ to $shape^{o}_{i}$. Specifically, we first expand
$shape^{m}_{i}$’s dimensions or randomly drop some dimensions to make the
number of dimensions in $shape^{m}_{i}$ consistent with $shape^{o}_{i}$. Then,
for each dimension, we apply padding with random constant values or cropping
to make the total number of elements in each dimension in $shape^{m}_{i}$ is
consistent with $shape^{o}_{i}$.

Three examples on the left of Figure 6 are three mutants generated by input-
level mutations. MDtype changes the datatype of Conv2D’s input from float32 to
float16 so we can test precision-specific code in this layer. MDims mutates
BatchNormalization’s input from a 4D tensor to a 3D tensor; this change can
test library codes designed for the tensor with a specific number of
dimensions. MShape changes the tensor shape of Conv2D from (5,8,8,3) to
(5,8,5,3); thus can test library codes related to the specific tensor shape.
After mutating the layer inputs, all three mutation operators will cast or
reshape the layer’s output to that before mutation.

####  3.3.2. Parameter-Level Mutation.

The design of parameter-level mutation targets mutating the layer parameter
values. Similar to input-level mutation, the parameter-level mutation also
prioritizes the uncovered mutation option (i.e., parameter value) to diversify
layer parameters effectively. Moreover, like input-level mutation, our
parameter-level mutation will also randomly choose $n$ layers to mutate. We
design our parameter-level mutation, i.e., MParam, as follows:

  * •

Mutate Parameter (MParam): Given a seed model and a list of layers
$\\{layer_{1},...,layer_{n}\\}$, MParam searches through all generated models
and chooses the parameter value ($value_{i}$) of a specific parameter
($param_{i}$) that have not been covered by parameters of $layer_{i}$.
Finally, for each $layer_{i}$, MParam changes the $param_{i}$ of it to
$value_{i}$.

The example on the right of Figure 6 is a mutant generated by parameter-level
mutation. In this example, parameter-level mutation randomly chooses numeric
parameters (e.g., strides and dilation_rate) and non-numeric parameters (e.g.,
activation to mutate. By changing the value of activation parameter from relu
to sigmoid, this mutant can test new activation functions in the Conv2D.
Besides, changing the value of strides and dilation_rate will influence the
padding logic in the Conv2D. Note that though both our COMET and a prior work
(Guo et al., 2020) mutate parameters, prior work randomly mutates parameter
values, while COMET considers the uncovered parameter values. By doing so,
COMET could be more effective in exploring library code.

####  3.3.3. Structure-Level Mutation

Figure 7. Example of Structure-Level Mutation

The structure-level mutation targets diversifying the structure of a seed
model. In particular, it intends to explore the possible layer sequences
within the search space. The most straightforward way is to insert some layers
into the seed model. However, because of the huge search space when mutating a
model’s structure, randomly inserting layers (Wang et al., 2020; Ma et al.,
2018) is weak in generating an effective mutant that contains new layer
sequences. Therefore, our structure-level mutation adopts a similar strategy
as input-level and parameter-level mutation. We choose the mutation option
that can generate the mutant with new layer sequences in each mutation.
Moreover, since some structures cannot be covered by simply inserting layers
into the seed model, we also provide some other mutation operators to generate
them. Specifically, we design our structure-level mutation as follows:

  * •

Insert Layers (IL): Given a seed model and the number of layers to be inserted
(denoted as $n$), IL chooses $n$ layers ($\\{layer_{1},...,layer_{n}\\}$) from
all possible types of layers which previously generated models have not
covered. If IL cannot find enough uncovered layers, it randomly chooses one
instead. For each $layer_{i}$ ($i\in[0,n]$), IL searches for a
$layer_{i}^{\prime}$ in the seed model so that the layer pair
$\left<layer_{i}^{\prime},layer_{i}\right>$ not covered by previously
generated models can be inserted into this seed model. If no possible
$layer_{i}^{\prime}$ can be found, IL randomly chooses one layer inside the
seed model instead. Finally, for each $layer_{i}$, IL inserts $layer_{i}$
after $layer_{i}^{\prime}$ and directs the output of $layer_{i}$ to the
original subsequent layer of $layer_{i}^{\prime}$ so the $layer_{i}$ can be
inserted into the seed model. To ensure that the insertion of $layer_{i}$
follows the dimension constraint and shape constraint required by its
subsequent layer, IL further applies the same reshape strategy described in
§3.3.1 to reshape the $layer_{i}$’s output shape to the original input shape
of its subsequent layer.

  * •

Merge Layers (ML): Given a seed model, ML chooses an uncovered merging layer
$layer_{m}$ which previously generated models have not covered. If previously
generated DL models have covered all merging layers, ML randomly chooses one
instead. As described in Figure 5, ML needs to follow the shape constraint,
i.e., input tensors of the inserted merging layer should have the same shape.
To do so, ML scans the seed model and randomly chooses two layers (denoted as
$layer_{i}$ and $layer_{i}^{\prime}$) whose tensor output is the same, then
merges them using $layer_{m}$. If no $layer_{i}$ and $layer_{i}^{\prime}$ can
be found, ML will directly raise an exception, and COMET will continue the
next mutation. ML further directs the output of $layer_{m}$ to the subsequent
layer of either $layer_{i}$ or $layer_{i}^{\prime}$ so $layer_{m}$ can be
inserted into the seed model.

  * •

Connect Layers (CL): Given a seed model and the number of layer sequences to
be inserted (denoted as $n$), CL chooses $n$ layer pairs
($\left<layer_{i},layer_{i}^{\prime}\right>$, $i\in[0,n]$) inside the seed
model that previously generated models have not covered. If CL cannot find
enough layer pairs, it randomly chooses two layers inside the seed model to
form a layer pair instead. For each layer pair
$\left<layer_{i},layer_{i}^{\prime}\right>$, CL then connects $layer_{i}$ and
$layer_{i}^{\prime}$ by sending the output of $layer_{i}$ to
$layer_{i}^{\prime}$. Note that directly connecting $layer_{i}$ and
$layer_{i}^{\prime}$ will break the original connection between
$layer_{i}^{\prime}$ and its preceding layer (denoted as $layer_{p}$), leaving
$layer_{p}$ detached from the seed model. To avoid this happens, CL first
makes a copy of $layer_{i}^{\prime}$ to keep the original connection between
$layer_{i}^{\prime}$ and $layer_{p}$, then it directs the output of
$layer_{i}$ to the copied $layer_{i}^{\prime}$. Since the output shape of the
copied $layer_{i}^{\prime}$ can be different from that of the original
$layer_{i}^{\prime}$, directing the output of copied $layer_{i}^{\prime}$ to
the subsequent layers may result in an invalid model that violates shape
constraint or dimension constraint. Therefore, CL appends the output tensor of
the copied $layer_{i}^{\prime}$ to the final output lists of the mutant, so
the mutant is still valid.

Figure 7 exemplifies three mutants generated by structure-level mutations. IL
inserts the
GlobalAveragePooling after Conv2D. Since the output shape of
GlobalAveragePooling is (5,2), which is inconsistent with the original output
shape (5,6,6,2) and cannot be accepted by BatchNormalization, IL further
reshapes the output shape to the original one. Since the output shape of the
Conv2D layer and the BatchNormalization are the same, ML can merge these two
outputs using the Add layer. CL introduces the new layer sequence Conv2D
$\rightarrow$ BatchNormalization by connecting these two layers. It further
appends the output tensor of the copied Conv2D to the final output lists of
the mutant.

####  3.3.4. Anomaly Mutation

Besides diversifying layer inputs, parameter values, or sequences, we noticed
that DL libraries lack consideration for some particular values, such as NaN.
Indeed, we observed that the manifestation of some bugs requires these special
values. Following these observations, we design a SpecialI mutation operator
to test the layer API using these values. Specifically, given a seed model,
SpecialI randomly chooses a layer (denoted as $layer_{i}$) and changes the
input value of $layer_{i}$ to NaN or Inf.

Note that our mutation operators are designed to apply legitimate modification
on the seed model to pass validity checks in DL libraries. However, there are
still a few invalid mutants generated. For instance, the mutant may require
too large GPU memory that directly raises an out-of-memory issue in the DL
library; the shape of layer input may be too small, and thus some layers with
typical configurations cannot process it. These mutants cannot be successfully
constructed by DL libraries. For these invalid models, we follow existing
works (Wang et al., 2020; Gu et al., 2022) to ignore them and continue the
next round of mutation.

###  3.4. Model Synthesis

This section proposes a method to synthesize initial seed models based on the
collected published DL models (denoted as original models). Our model
synthesis method aims to reduce the original model’s size while preserving its
diversity. Specifically, for an original model, our model synthesis algorithm
constructs a new model which covers the layer API calls inside the original
model, and this new model is used as the initial seed model for further
mutation. However, randomly generating a model does not work well for the
following reasons. On the one hand, randomly building a model from scratch
likely generates invalid models, which DL libraries cannot load, and thus
cannot be used for testing. Taking Figure 5 as an example, layer Add requires
a specific shape constraint (i.e., the shape of two inputs is the same), and
layer Conv2D requires the input to have four dimensions. Arbitrary removal of
duplicated layers may break such input constraints and unable to generate a
valid model. On the other hand, randomly generating DL models cannot
effectively achieve the goal of our model synthesis method, i.e., reduce the
size of original models while retaining their layer inputs, layer parameters,
and layer sequences.

We synthesize a new model using our mutation operators, as introduced in §3.3.
Specifically, we regard the model synthesis as an iterative mutation process
using our mutation operators until all layer API calls (i.e., layer inputs,
layer parameter values, and layer sequences) in the original model are
covered. Given a published model as the original model, we first extract the
covered layer types and their parameter values to instantiate a list of
candidate layers to be inserted. We further use IL operator to insert these
layers so the synthesized model can cover all layer APIs and layer parameter
values inside the original model. MDtype, MDims, and MShape are further used
to retrieve the original model’s layer input into the synthesized model.
Finally, we use CL and ML to cover the remaining layer sequences inside the
original model.

Algorithm 1 shows the logic of our model synthesis method. Given a published
DL model, we first analyze the layer input ($\mathcal{D}_{i}$), parameter
($\mathcal{D}_{p}$), and sequences ($\mathcal{D}_{s}$) covered by the
published model. Based on the collected diversities, we then use IL operator
to insert layer_class with target layer_param to the synthesized model
$\mathcal{M^{\prime}}$ (lines 7-10). In particular, the layer is appended to
the end of $\mathcal{M^{\prime}}$, so we can insert it without breaking any
layer sequence diversity. After all $\mathcal{D}_{p}$ are covered, MDtype,
MDims, and MShape will be applied to the synthesized model to cover
$\mathcal{D}_{i}$ (lines 11-18). Finally, if some layer sequences remain
uncovered, the CL and ML operator is further used to cover these layer
sequences (lines 19-22). The model synthesis algorithm stops when the
synthesized model covers all layer inputs, layer parameters, and layer
sequences inside the original one or the given time budget (i.e., 5 minutes)
is reached.

Input: $\mathcal{M}$: original model

Output: $\mathcal{M^{\prime}}$: synthesized model

1 def  _ModelSynthesis( $\mathcal{M}$)_:

2 $\mathcal{D}_{i}$, $\mathcal{D}_{p}$, $\mathcal{D}_{s}$ $\leftarrow$
CollectDiversity($\mathcal{M}$)

3 $\mathcal{M^{\prime}}$ $\leftarrow\phi$

4 $\mathcal{D}^{\prime}_{i}$, $\mathcal{D}^{\prime}_{p}$,
$\mathcal{D}^{\prime}_{s}$ $\leftarrow$
CollectDiversity($\mathcal{M^{\prime}}$)

5 $dis$ $\leftarrow$ $\mathcal{D}_{i}\setminus\mathcal{D}^{\prime}_{i}$ $\cup$
$\mathcal{D}_{p}\setminus\mathcal{D}^{\prime}_{p}$ $\cup$
$\mathcal{D}_{s}\setminus\mathcal{D}^{\prime}_{s}$

6 while  _$dis\neq\phi\land\neg$ timeout()_ do

7 if  _$\mathcal{D}_{p}\setminus\mathcal{D}^{\prime}_{p}$ $\neq\phi$_ then

8 layer_class, layer_param $\leftarrow$
rand.Choice($\mathcal{D}_{p}\setminus\mathcal{D}^{\prime}_{p}$)

9 $\mathcal{M^{\prime}}$ $\leftarrow$ IL(layer_class, layer_param)

10 continue

11

12 if  _$\mathcal{D}_{i}\setminus\mathcal{D}^{\prime}_{i}$ $\neq\phi$_ then

13 layer_class, dtype, ndims,shape $\leftarrow$
rand.Choice($\mathcal{D}_{i}\setminus\mathcal{D}^{\prime}_{i}$)

14 if  _dtype $\neq\phi$_ then

15 $\mathcal{M^{\prime}}$ $\leftarrow$ MDtype($\mathcal{M^{\prime}}$,
$\left<\textit{layer\\_class},\textit{dtype}\right>$)

16 continue

17

18 if  _ndims $\neq\phi$_ then

19 $\mathcal{M^{\prime}}$ $\leftarrow$ MDims($\mathcal{M^{\prime}}$,
$\left<\textit{layer\\_class},\textit{ndims}\right>$)

20 continue

21

22 if  _shape $\neq\phi$_ then

23 $\mathcal{M^{\prime}}$ $\leftarrow$ MShape($\mathcal{M^{\prime}}$,
$\left<\textit{layer\\_class},\textit{shape}\right>$)

24 continue

25

26

27 if  _$\mathcal{D}_{s}\setminus\mathcal{D}^{\prime}_{s}$ $\neq\phi$_ then

28 layer_sequence $\leftarrow$
rand.Choice($\mathcal{D}_{s}\setminus\mathcal{D}^{\prime}_{s}$)

29 $\mathcal{M^{\prime}}$ $\leftarrow$ CL($\mathcal{M^{\prime}}$,
layer_sequence)

30 continue

31

32 $\mathcal{D}^{\prime}_{i}$, $\mathcal{D}^{\prime}_{p}$,
$\mathcal{D}^{\prime}_{s}$ $\leftarrow$
CollectDiversity($\mathcal{M^{\prime}}$)

33 $dis$ $\leftarrow$ $\mathcal{D}_{i}\setminus\mathcal{D}^{\prime}_{i}$
$\cup$ $\mathcal{D}_{p}\setminus\mathcal{D}^{\prime}_{p}$ $\cup$
$\mathcal{D}_{s}\setminus\mathcal{D}^{\prime}_{s}$

34

35 return $\mathcal{M^{\prime}}$

Algorithm 1 Model Synthesis Figure 8. Illustration of Model Synthesis Based on
InceptionResNetV2

Figure 8 shows a model synthesized from InceptionResNetV2. The original model
contains many duplicated layers (e.g., the Conv-BN-ReLU block), which cannot
contribute to the diversity of layer API calls. By extracting the layer
inputs, layer parameter values, and layer sequences covered by the original
model, we can use our model synthesis algorithm to synthesize a more
lightweight model while preserving the layer API calls covered by the original
one. Note that the synthesized model is generated from scratch and only
targets recovering the layer inputs, layer parameter values, and layer
sequences covered by the original model. Its topology may be different from
the original. As a result, we can reduce the total number of layers in the
original model from 782 to 41 (i.e., reduce 94.8% number of layers) and the
total number of weights from 55 million to 466 thousand (i.e., reduce 99.2%
weights). Moreover, the runtime overhead for TensorFlow to load and make
inferences on the synthesized model can be lessened from 65 seconds to six
seconds while the layer API calls are preserved.

###  3.5. Diversity-Driven Model Generation

Since not all mutation operators are equally effective, random selection may
be ineffective in searching for diverse models in the huge search space. Note
that our main objective is to diversify the generated model. Therefore, we
should generate DL models for increasing the layer input coverage, layer
parameter coverage, and layer sequence coverage. Besides, we need to cover
more branches inside the DL library to achieve a higher test coverage.
Following these objectives, we design our fitness function based on these
coverage criteria to measure the potential effectiveness of mutation
operators. However, only selecting the mutation operators with the highest
fitness score is not optimal since the fitness score is based on the
historical iteration and cannot totally represent future results. Intuitively,
mutation operators should have a certain possibility to be selected, and those
are more likely to diversify DL models and test library code should be more
likely to be chosen. This intuition motivates our adoption of the Metropolis-
Hasting algorithm, an MCMC (Markov Chain Monte Carlo) sampling method that has
been popularly used (Chen and Su, 2015; Wang et al., 2020) for mutation
selection. In each iteration of model generation, COMET prioritizes the
mutation-selection effort to prefer those mutation operators that are more
effective in increasing our fitness score.

Fitness Function. Following existing works (Wang et al., 2020; Chen and Su,
2015), we define the mutation operator ($op$) as the seed in the MCMC sampling
process. After selecting a mutation operator $op$, COMET can generate a new
model $m^{\prime}=op(m)$ by applying the chosen mutation operator $op$ to a
randomly selected seed model $m$. For each $op$, if applying $op$ is more
likely to generate a model that either increases the layer API call diversity
or covers new branches inside DL libraries (i.e., increases branch coverage),
we should prefer it for future model generation. Therefore, the fitness score
of $op$ depends on the quality of $m^{\prime}$ generated by it. We first
introduce how we measure the quality of each generated model $m^{\prime}$
below:

(4) $\displaystyle score(m^{\prime})=\lambda\cdot
diverse(m^{\prime})+(1-\lambda)\cdot branch(m^{\prime})$

Based on our intuition, we consider two factors when designing
$score(m^{\prime})$: whether it is a diverse model (denoted as
$diverse(m^{\prime})$) and whether it can cover new branches (denoted as
$branch(m^{\prime})$). The higher the $score(m^{\prime})$ is, the more
influential the model $m^{\prime}$ is in increasing the model diversity and
covering new branches in DL libraries. We set $\lambda=\frac{1}{2}$ so that
COMET prefers model diversity and branch coverage equally. To design
$diverse(m^{\prime})$, we compare the layer inputs ($\mathcal{D}^{\prime}_{i}$
and $\mathcal{D}_{i}$), parameters ($\mathcal{D}^{\prime}_{p}$ and
$\mathcal{D}_{p}$), and sequences ($\mathcal{D}^{\prime}_{s}$ and
$\mathcal{D}_{s}$) covered by $m^{\prime}$ and $M$ where $M$ is the set of
previously generated models. If $m^{\prime}$ contains new layer types, layer
pairs, or layer parameters, $diverse(m^{\prime})$ will return one, indicating
that a diverse model is found; otherwise, $diverse(m^{\prime})$ returns zero.

(5) $\displaystyle diverse(m^{\prime})$
$\displaystyle=\begin{cases}1,&(\mathcal{D}^{\prime}_{i}\setminus\mathcal{D}_{i})\cup(\mathcal{D}^{\prime}_{p}\setminus\mathcal{D}^{\prime}_{p})\cup(\mathcal{D}_{s}\setminus\mathcal{D}^{\prime}_{s})\neq\phi\\\
0,&else\end{cases}$

To design $branch(m^{\prime})$, we follow existing strategies (Chen and Su,
2015) to use Equation 6. Suppose the total branches covered by previously
generated models is $\mathcal{B}_{M}$ and the branches covered by $m^{\prime}$
is $\mathcal{B}_{m^{\prime}}$, if $\mathcal{B}_{m^{\prime}}$ contains branches
that $M$ has not covered, $branch(m^{\prime})$ will return one; otherwise,
$branch(m^{\prime})$ returns zero.

(6) $\displaystyle branch(m^{\prime})$
$\displaystyle=\begin{cases}1,&\mathcal{B}_{M}\setminus\mathcal{B}_{m^{\prime}}\neq\phi\\\
0,&else\end{cases}$

Finally, given a list of models (i.e., $M_{op}$) generated by mutation
operator $op$, we can define the fitness function for $op$ using Equation 7
where $N_{op}$ denotes the total number of models in $M_{op}$.

(7) $\displaystyle
fitness(op)=\frac{\sum^{M_{op}}_{m^{\prime}}{score(m^{\prime})}}{N_{op}}$

Seed Selection. COMET adopts the MH algorithm, a Markov Chain Monte Carlo
(MCMC) algorithm commonly used by testing techniques to guide the seed
selection (Chen et al., 2016a; Wang et al., 2020). The MH algorithm is
designed for obtaining random samples from a probability distribution. It
works by generating a sequence of samples whose distribution closely
approximates the planned distribution. Samples are produced iteratively, where
the acceptance of the subsequent sampling (say $s_{2}$) depends only on the
current one (say $s_{1}$). In our setting, a sample corresponds to a seed,
i.e., a mutation operator.

Like existing works (Chen et al., 2016b; Wang et al., 2020), we use the
geometric distribution as the desired distribution. The geometric distribution
is the probability distribution of the number X of Bernoulli trials needed to
obtain one success. Specifically, if the probability of success on each trial
is $p$, the likelihood that the $k^{th}$ trial is the first success is given
by $Pr(X=k)=(1-p)^{k-1}p$, where $p\in\left(0,1\right)$ is a hyperparameter
related to the total number of seeds (Chen et al., 2016b). We sort all seeds
based on their fitness scores computed by Equation 7 from the highest to the
lowest. By mapping the sorted seeds into the geometric distribution, the
sampling process can prefer those with high fitness scores to those with low
fitness scores. Specifically, suppose the last selected seed is $s_{1}$, the
probability of accepting seed $s_{2}$ is calculated by Equation 8. In
particular, if the $k_{2}$ (the index of $s_{2}$ after we sort all seeds) is
smaller than $k_{1}$ (the index of $s_{1}$), i.e., the fitness score of
$s_{2}$ is larger than that of $s_{1}$, $P(s_{1}\rightarrow s_{2})=1$ and we
directly accept $s_{2}$. Otherwise, we accept $s_{2}$ with a certain
probability $(1-p)^{k_{2}-k_{1}}$.

(8) $\displaystyle P(s_{1}\rightarrow
s_{2})=min\left(1,\frac{Pr(s_{2})}{Pr(s_{1})}\right)=min\left(1,(1-p)^{k_{2}-k_{1}}\right)$

Table 4 is an example demonstrating the seed selection, it lists each seed’s
fitness score (row 3) in one iteration. Based on these fitness scores, COMET
first sorts all seeds from the highest fitness score to the lowest (row 4),
and then it assigns the index $k$ for each seed (row 5) from 1 to 8. The
selection of the next seed $s_{2}$ relies on the hyperparameter $p$ and the
index of the last chosen seed $s_{1}$. Specifically, COMET randomly selects a
seed $s_{2}$ and accepts this selection based on acceptance probability (i.e.,
$P(s_{1}\rightarrow s_{2})$) calculated by Equation 8 (in this example,
$p=0.4$, and $s_{1}$ is MDtype with the index to be 3). In particular, when
COMET selects the seed whose index is smaller than or equal to 3 (i.e.,
MParam, ML, or MDtype), the acceptance probability is 1, which means that
COMET will directly accept this selection. If COMET selects the seed whose
index is larger than 3 (i.e., IL, MDims, MShape, CL, or Special), COMET will
accept this selection based on the acceptance probability. The higher the
selection’s acceptance probability is, the more likely COMET will accept this
selection. In this example, among seeds whose indexes are larger than 3,
selecting IL is most likely to be accepted, while selecting SpecialI is least
likely to be accepted. COMET repeats this seed selection until a selection is
accepted.

Table 4. Example of Seed Selection Unsorted Seeds
---
| MDtype (s1) | MDims | MShape | MParam | IL | ML | CL | SpecialI
Fitness | 0.60 | 0.53 | 0.44 | 0.68 | 0.57 | 0.62 | 0.42 | 0.38
Sorted Seeds
| MParam | ML | MDtype (s1) | IL | MDims | MShape | CL | SpecialI
k | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
$P(s_{1}\rightarrow s_{2})$ | 1 | 1 | 1 | 0.6 | 0.36 | 0.22 | 0.13 | 0.08

Model Pool Update. As illustrated earlier, in each iteration, COMET randomly
selects a model from the model pool to generate a mutant. In the beginning,
the model pool contains all the initial synthesized models. During the model
generation, COMET adds mutants, which can introduce new layer inputs,
parameters, or sequences into the model pool to increase the diversity of the
seed model. If the total number of seed models in the model pool is larger
than a pre-defined size (i.e., 50 following existing work (Wang et al.,
2020)), COMET will randomly remove a seed model from the model pool.

Algorithm 2 shows the detailed process for COMET to generate models. In each
iteration, COMET uses the MH algorithm to select an operator $op_{i}$ (in
function SeedSelection) and randomly select a seed model $m_{i}$ from the
model pool (line 4). A mutant $m^{\prime}_{i}$ is thereby generated by
applying $op_{i}$ to $m_{i}$ (line 5). If the mutant $m^{\prime}_{i}$ can
cover new layer API calls or cover new branches inside the DL library, COMET
updates the mutant to the model pool $M$ (lines 6-7). Otherwise, COMET
continues the next iteration. The diverse model generation algorithm ends when
the given time budget is reached.

Input: $\mathbf{M}$: a model pool; $\mathbf{Op}$: a mutation operator pool

Output: $\mathbf{M^{\prime}}$: All generated models

1 def  _ModelGeneration( $P$, $M$, $Op$)_:

2 while  _$\neg$ timeout()_ do

3 ${op}_{i}$ $\leftarrow$ SeedSelection ($S$)

4 $m_{i}$ $\leftarrow$ rand.Choice($M$) // randomly choose $m$

5 $m^{\prime}_{i}\leftarrow op_{i}(m)$

6 if  _score( $m^{\prime}_{i}$) ¿ 0_ then

7 $M\leftarrow$ UpdateModelPool($M$, $m^{\prime}_{i}$)

8

9

10 return $M^{\prime}$

11

12def  _SeedSelection( $S$)_:

13 // select candidate based on MH algorithm

14 $S_{sorted}\leftarrow$ SortSeed($S$) // Using Fitness Function (Eq. 7)

15 $prob\leftarrow 0$

16 $k_{1}\leftarrow$ get the index of last selected seed

17 while  _New Rand() $\geq prob$ _ do

18 $s_{i}\leftarrow$ random.Choice($S_{sorted}$)

19 $k_{2}\leftarrow S_{sorted}.index(s_{i})$

20 $prob\leftarrow(1-p)^{k_{2}-k_{1}}$

21

22 return $s_{i}$

23def  _UpdateModelPool( $M$, $m^{\prime}_{i}$)_:

24 if  _sizeof( $M$)¿=pool_size_ then

25 $idx\leftarrow$ rand.randInt(sizeof($M$))

26 $M^{\prime}$.pop(idx)

27

28 $M^{\prime}$.add($m^{\prime}_{i}$)

29 return $M^{\prime}$

Algorithm 2 Diverse Model Generation Algorithm

###  3.6. Bug Detection

We further use the generated models as test inputs for DL library testing in
each iteration. For bug detection, we follow existing works (Pham et al.,
2019; Guo et al., 2020; Wang et al., 2020; Luo et al., 2021; Gu et al., 2022)
to adopt differential testing as the test oracle. Overall, COMET targets three
types of bugs: crash, Not-A-Number (NaN), and inconsistency bugs. For crash
bugs and NaN bugs, we consider a bug detected if some DL libraries crash or
output NaN value when loading and executing the model while some other
libraries do not. Inconsistency bug refers to the difference between the model
execution results of different libraries. Since a DL model initialized with
the same weights should have the same or close model prediction result on
different DL libraries, inconsistent prediction results may indicate the
inconsistent algorithm implementation of different DL libraries, which is
likely to be a bug (Pham et al., 2019; Guo et al., 2020; Wang et al., 2020).
However, because of the computation randomness during the DL library, two
libraries’ outputs on the same model and the same input are not the same.
Therefore, we adopt a well-defined inconsistency metric, i.e., $D\\_MAD$,
proposed by CRADLE (Pham et al., 2019) and has been used by existing DL
library testing works (Wang et al., 2020; Pham et al., 2019) to measure the
inconsistency degree across different DL libraries for inconsistency bug
detection. In particular, $D\\_MAD$ calculates the inconsistency distance by
comparing the relative distance of model outputs from the ground-truth labels
(Pham et al., 2019). Specifically, given two predicted vectors $Y$,
$Y^{\prime}$, and the ground truth $O$, $\delta_{Y,O}$ and
$\delta_{Y^{\prime},O}$ are calculated to measure the distance between the
predicted vectors and the ground truth using Equation 9, and the distance
between two predicted vectors are then measured by Equation 10. An
inconsistency bug is detected when the $D\\_MAD$ score is larger than a pre-
defined threshold $T$. Note that DL models may have multiple outputs (as
exemplified on the right of Figure 7), which is inapplicable to the $D\\_MAD$
metrics. Therefore, for DL model with multiple outputs, we reshape its outputs
to the same shape and add them together so we can still use $D\\_MAD$ for
inconsistency bug detection.

(9) $\delta_{Y,O}=\frac{1}{N}\sum^{N}_{i=1}|Y_{i}-O_{i}|$ (10)
$D\\_MAD_{O,Y,Y^{\prime}}=\frac{|\delta_{Y,O}-\delta_{Y^{\prime},O}|}{\delta_{Y,O}+\delta_{Y^{\prime},O}}$

##  4\. Evaluation

We evaluated the performance of COMET from two perspectives: the effectiveness
of diverse model generation and bug detection. In particular, we studied four
research questions (RQs):

  * •

RQ1: Can COMET outperform existing techniques? We first demonstrate the
advantage of our diversity-driven approach by comparing COMET against existing
techniques in terms of our proposed coverage criteria (i.e., layer input
coverage, layer parameter coverage, and layer sequence coverage) and their
branch coverage, line coverage on a representative DL library.

  * •

RQ2: Can COMET detect real bugs? To further demonstrate the usefulness of
COMET, we present its bug-revealing capability. We evaluate COMET on the
latest release of eight popular DL libraries: TensorFlow, PyTorch, MXNet,
ONNXRuntime, Keras-MXNet, TF2ONNX, and ONNX2PyTorch.

  * •

RQ3: Can test efficiency be boosted by the synthesized initial models? We also
conducted the experiment to show the impact of the synthesized initial models
on the test efficiency by illustrating the trends of coverage growth achieved
with and without model synthesis.

  * •

RQ4: To what extent do the proposed diversity-driven mutation operators and
the search algorithm contribute to the performance of COMET? Finally, to
further dissect the secrets of COMET, we conducted an ablation study to
analyze two key components (i.e., diversity-driven model mutation operators
and the MCMC-based search algorithm) in COMET, examining their usefulness in
generating diverse models and exercising various computations.

###  4.1. Experiment Setup

All the experiments were conducted on a machine powered by Intel Core i9 with
one Titan RTX and two 2080Ti GPU cards.

Baselines. We considered three state-of-the-art DL testing techniques as
baselines in our experiment: CRADLE (Pham et al., 2019), LEMON (Wang et al.,
2020), and Muffin (Gu et al., 2022). Since no executables or source codes of
GraphFuzz (Luo et al., 2021) and Audee (Guo et al., 2020) are available, we
did not include them as baselines. We consider the layer input coverage, layer
parameter coverage, layer sequence coverage, branch coverage, and line
coverage as the metrics to evaluate their overall performance. In particular,
the first three coverage criteria are used to measure the layer API call
diversity; the branch and line coverage are used to measure their test
coverage on DL libraries. The three baselines are designed in different ways.
In particular, CRADLE focuses on detecting inconsistency issues exposed by
collected DL models and datasets. Since it does not involve the model
generation, we collect the total layer inputs, layer parameters, and layer
sequences in the initial models. We use our target DL libraries to load their
generated models and make inferences on their collected datasets to collect
their branch and line coverage. For LEMON and Muffin, we used their default
settings to generate models. Also, since other baselines do not consider the
training phase during testing, for Muffin, which targets both training and
inference phases, we only considered the inference phase for the fairness of
the comparison.

Benchmarks and Implementations.

Table 5. Statistics of Original Models

Model Name | Total # Layers (Weights) | Layer Inputs | Layer Parameters | Layer Sequences
---|---|---|---|---
AlexNet | 17 (21.592M) | 22 | 35 | 5
LSTM (SineWave) | 5 (0.071M) | 9 | 22 | 2
LeNet | 9 (0.062) | 14 | 20 | 3
DenseNet121 | 429 (8.162M) | 29 | 45 | 6
InceptionResNetV2 | 782 (55.973M) | 26 | 46 | 4
InceptionV3 | 313 (23.951) | 30 | 45 | 4
ResNet50 | 177 (25.736M) | 24 | 40 | 4
Xception | 134 (23.010M) | 30 | 54 | 3
LSTM (StockPrice) | 3 (0.026M) | 6 | 21 | 1
MobileNetV2 | 156 (3.639M) | 37 | 54 | 9

We selected 10 popular DL models with different structures as the original
models. They cover diverse application domains across both image data and
sequential data. The three baselines have also used these models in the
evaluation. In particular, eight DL models (i.e., LeNet, AlexNet, DenseNet121,
InceptionResNetV2, InceptionV3, MobileNetV2, ResNet50, Xception) are image
classification models trained on MNIST (Lecun et al., 1998), CIFAR-10
(Krizhevsky, 2009), and ImageNet (Krizhevsky et al., 2012). The remaining two
models are LSTM-based models trained on Sine-Wave dataset and Stock-Price
dataset by existing work (Wang et al., 2020). All DL models are online-
available on our project site.555https://github.com/maybeLee/COMET The number
of layers, weights, and the diversity we collected for each original model are
listed in Table 5. The setting of hyperparameter $p$ in our algorithm depends
on the size of seeds in the MH algorithm (Chen et al., 2016b). Since the total
size of initial seeds (i.e., mutation operators) is 8, following the setting
recommended by existing work (Chen et al., 2016b), the range of $p$ should be
set within [0.313, 0.598]. In our implementation, we set $p$ to 0.4 without
fine-tuning in our experiment. For inconsistency detection, we set $T$ to 0.4,
which is the value used by an existing technique (Wang et al., 2020).

Layer API call Diversity. We also use Keras (Chollet et al., 2015) to generate
DL models for a fair comparison with LEMON (Wang et al., 2020) and Muffin (Gu
et al., 2022). Following existing works (Gu et al., 2022; Guo et al., 2020),
we take the Keras documentation666https://keras.io/api/layers/ as the
reference to define the mutation space. In particular, we collect the
information of all 59 Keras layer APIs (i.e., $N_{layer}=59$), including their
possible datatypes, required input dimensions, and possible parameter values.
Specifically, we set all possible datatypes for each layer API to be
‘bfloat16’, ‘double’, ‘float16’, ‘float32’, ‘float64’, ‘half’ (i.e.,
$n_{type}(l)=6,\forall l\in LayerList$). We further manually collect the
possible parameter values, possible number of input dimensions, and possible
number of output dimensions for each layer API. We set $n_{shape}$ to be 5 to
measure the coverage of layer input shape. Following §3.2, the total number of
possible layer sequences is 2170. Since the total number of layer APIs is few
and the documentation has identified the possible numbers of input dimensions
and possible parameter values, the manual effort in collecting these spaces is
trivial.

Like GraphFuzz (Luo et al., 2021), when implementing our mutation operators,
we inevitably introduce some utility operators into the generated models.
Specifically, we add the tf.cast operator into the model to cast the original
input’s datatype to our required one; we use padding, cropping, and reshaping
operators to diversify layer input shape and input dimension. Since models
generated by other works do not have these operators, we do not consider them
when measuring the diversity of our generated models.

DL Libraries Under Test. We evaluated the bug detection performance of COMET
on the latest release of eight popular DL libraries: Keras 2.8.0, TensorFlow
2.9.0, PyTorch 1.10.0, MXNet 1.9.0, ONNXRuntime 1.10.0, Keras-MXNet 2.2.4.3,
TF2ONNX 1.9.3, and ONNX2Pytorch 0.4.1. Following earlier practice (Pham et
al., 2019; Wang et al., 2020; Gu et al., 2022), COMET leverages Keras APIs to
generate models uniformly for testing different DL libraries. The generated
models are converted for MXNet, ONNX, and PyTorch using three libraries,
Keras-MXNet, TF2ONNX, and ONNX2Pytorch. The setup helps to control the
variation of models used for different libraries.

Unique Bugs Identification. Following the practice of existing works (Guo et
al., 2020; Pham et al., 2019; Wang et al., 2020), we manually analyzed each
detected bug and identified the unique ones. In particular, we consider a
crash bug unique if it has a distinct stack trace. We manually located the
layer that triggered the bugs for NaN and inconsistency bugs and analyzed the
root cause. If the exact root cause causes multiple bugs, we consider them as
duplicated bugs and only consider one of them in the subsequent analysis.

###  4.2. RQ1: Comparison with Baselines

This section compares COMET with three baselines: CRADLE (Pham et al., 2019),
LEMON (Wang et al., 2020), and Muffin (Gu et al., 2022). Specifically, we
evaluated their performance by running them on the same 10 published DL models
(see §4.1) for six hours. Note that CRADLE tests DL libraries by directly
collecting publicly-available DL models. We, therefore, referred to the
coverage results of initial DL models to capture CRADLE’s performance. Since
Muffin does not require initial DL models for model generation, we ran it for
six hours using its default setting to capture the coverage result. Although
Muffin does not require these DL models for model generation, we can easily
obtain these 10 publicly-available DL models used in our experiment online
without additional training. Therefore, we argue that using these initial
models will not affect fairness when we compare Muffin with COMET and LEMON.

Table 6 presents the coverage results achieved by the baselines. Since
collecting DL library coverage is costly and testing practices and bugs inside
different DL libraries share a significant commonality (Jia et al., 2021a;
Nejadgholi and Yang, 2020; Yang et al., 2022; Chen et al., 2022), we used the
branch and line coverage on TensorFlow’s model construction and model
execution modules as the representative for evaluating each technique’s test
coverage. According to Table 6, COMET outperforms the baselines on all five
coverage criteria. In particular, the layer input coverage achieved by COMET
(69.7%) is 35.6% higher than the best result among the baselines (34.1%); and
the layer parameter coverage achieved by COMET (50.2%) is 24.3% higher than
the best result achieved by the baseline (25.9%). COMET can also cover over
twice as many layer sequences as the state-of-the-art (39.0% vs 15.6%).

Besides the improvement of layer API call diversity, the results suggest a
correlation between the test coverage and the diversity of the generated
models. Specifically, COMET outperforms the baselines by covering at least
3.4% more branches and 4.5% more lines in TensorFlow’s model construction and
model execution modules. Nevertheless, the issue of relatively low branch
coverage is common to fuzzing-based techniques. We further performed an in-
depth analysis towards the uncovered covered code and discussed it in §5.1.

Overall, COMET outperforms the state-of-the-art approaches. In particular,
compared with the optimum performance achieved by baselines, COMET improves
35.6%, 24.3%, and 23.4% on layer input coverage, layer parameter coverage, and
layer sequence coverage. In addition, COMET can outperform the baselines by
covering at least 3.4% more branches and 4.5% more lines in TensorFlow’s model
construction and model execution modules.

Table 6. Coverage by COMET and Baselines Technique | Branch | Line | Layer Input | Layer Parameter | Layer Sequence
---|---|---|---|---|---
Coverage | Coverage | Coverage | Coverage | Coverage
COMET | 19.4% | 34.9% | 69.7% | 50.2% | 39.0%
MUFFIN | 16.0% | 30.4% | 34.1% | 25.9% | 15.6%
LEMON | 12.5% | 26.6% | 14.2% | 6.4% | 2.4%
CRADLE | 12.0% | 25.9% | 10.4% | 5.9% | 1.2%

###  4.3. RQ2: Effectiveness of Bug Detection

Table 7. Performance on Bug Detection

| DL Libraries
---|---
| ONNXRuntime | MXNet | Keras-MXNet | TF2ONNX | ONNX2PyTorch | Keras | TensorFlow | PyTorch
Crash | 2 (2) | 4 (0) | 1 (0) | 4 (3) | 4 (1) | 8 (7) | 0 (0) | 0 (0)
NaN | 2 (2) | 1 (1) | 0 (0) | 1 (1) | 0 (0) | 4 (3) | 0 (0) | 0 (0)
Inconsistency | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 1 (1) | 0 (0)
Subtotal | 4 (4) | 5 (1) | 1 (0) | 5 (4) | 4 (1) | 12 (10) | 1 (1) | 0 (0)

  * •

The number of confirmed bugs is parenthesized.

Table 7 demonstrates the new bugs detected by COMET across various DL
libraries, categorized by their bug symptoms. In total, COMET detected 32 new
bugs in the latest release version of evaluated DL libraries. After we
reported them, 21 of them have been confirmed by DL library developers and 7
out of those confirmed bugs were fixed. The remaining 11 bugs have been
reported and are waiting for developers’ confirmation. The detected bugs can
be categorized according to three different symptoms. We further conduct an
in-depth analysis of these bugs to answer two questions: 1) Are diversifying
layer API calls practical in generating DL models to detect bugs? 2) Are there
any other diversities that can help detect bugs? The first question examines
the usefulness of our insight; if the manifestation of the majority of bugs
requires a specific layer API call (i.e., a particular layer input, parameter
value, or sequence), diversifying layer API calls can help detect DL library
bugs. The second question aims to evaluate the completeness of our
diversities. If the majority of the bugs are detected by other diversities,
diversifying layer inputs, layer parameters, and layer sequences may be
insufficient.

Crash Bugs. Among the bugs detected, 23 are crash bugs. Developers have
confirmed 13 bugs and fixed five out of these confirmed bugs. We further
analyzed the 13 confirmed crash bugs. Four occurred during model conversion
(three in TF2ONNX library and one in ONNX2PyTorch library). The remaining
eight crash bugs occurred during model execution (two in ONNXRuntime and six
in Keras). We noticed that only one crash bug is caused by a single layer with
its default parameter values. 11 of the 13 crash bugs were manifested by
either a specific layer input, a specific layer parameter, or a specific layer
sequence. For instance, sending a tensor with bfloat16 datatype to Conv2D will
cause Keras crashes; failing to set either strides or dilation_ rate in
SeparableConv2D to 1 can result in the wrong shape inference, causing Keras to
crash; layer sequence between two ThresholdReLU layers will cause TF2ONNX
crashes. The experiment result affirms the importance of generating models
with diverse layer inputs, parameters, and sequences. Moreover, COMET detected
two severe crash bugs that directly led to core dump issues in Keras and
ONNXRuntime. Among these two bugs, one is caused by a particular layer
sequence (i.e., Dense-Dot) and a particular layer parameter (i.e., units=0);
the other one is caused by a specific layer input shape (i.e., tensor with an
empty shape for Conv3DTranspose layer). For these two issues, developers
instantly created pull requests to fix them after our reporting.

NaN Bugs. Besides crash bugs, we also detected eight Not-A-Number (NaN) bugs,
seven out of which have been confirmed by developers, including two fixed
ones. We further investigated the fault-triggering conditions for these
confirmed NaN bugs. We found that four out of seven NaN bugs are caused by a
particular layer API call, while the remaining three NaN bugs are caused by a
particular value: NaN. Specifically, two confirmed NaN bugs were manifested by
particular layer parameters, such as setting the max_value in the ReLU layer
to a concrete value instead of “None” (the default option). One NaN bug was
caused by a specific layer pair (i.e., Conv2D+Multiply). One NaN bug was
caused by a specific layer input shape (i.e., setting the shape of ConvLSTM2D
input’s first dimension to “1”). We noticed that the remaining three NaN bugs
were exposed when we passed a NaN value to a layer with its default parameter.
Although the manifestation of these three NaN bugs does not require a specific
layer input, layer parameter, or layer sequence, it still requires a special
value: NaN, indicating the usefulness of our mutation operator SpecialI in
detecting bugs.

Inconsistency Bugs. One confirmed bug is the inconsistency bug between the
TensorFlow library and the ONNXRuntime library. We find that this bug lies in
the model conversion phase in TensorFlow. Specifically, when TensorFlow
converts the model from the Keras format to TensorFlow Protobuf format (a
private format of DL model inside the TensorFlow library), the model will
trigger the bug if the axis parameter in the Softmax layer is set to “-2”.

Table 8. Analysis On Fault-Triggering Conditions of Bugs Detected By COMET

Library | Issue ID | Layer Input | Layer Parameter | Layer Sequence | Single Layer W/ Default Parameter | Other
---|---|---|---|---|---|---
ONNXRuntime | 11173 |  |  | ✓ |  | ✓
ONNXRuntime | 11024 | ✓ |  |  |  |
ONNXRuntime | 11010 |  |  |  |  | ✓
ONNXRuntime | 11241 |  |  | ✓ |  |
MXNet | 20947 |  | ✓ |  |  | ✓
Keras | 16348 |  | ✓ |  |  |
Keras | 16273 |  | ✓ | ✓ |  |
Keras | 16158 |  |  |  |  | ✓
Keras | 16289 |  |  |  |  | ✓
Keras | 16314 |  | ✓ |  |  |
TF2ONNX | 1875 |  | ✓ |  |  | ✓
TF2ONNX | 1807 |  |  |  | ✓ |
ONNX2PyTorch | 33/34 |  |  | ✓ |  |
TF2ONNX | 1811 |  |  | ✓ |  |
Keras | 16314 |  | ✓ |  |  |
Keras | 16492 | ✓ |  |  |  | ✓
Keras | 16927 | ✓ |  |  |  |
Keras | 16933 | ✓ |  |  |  |
Keras | 16970 | ✓ | ✓ |  |  |
Keras | 17044 | ✓ |  |  |  |
TF2ONNX | 1903 |  |  | ✓ |  |
Count |  | 6 | 7 | 6 | 1 | 7

To better understand the characteristics of DL library bugs detected by us, we
summarized the fault-triggering condition of our confirmed bugs. As shown in
Table 8, most bugs (i.e., 18 out of 21) detected require a specific layer
input, layer parameter value, or layer sequences. In contrast, only one bug is
manifested by a single layer API with its default parameter. Therefore, only
covering the layer API is not enough to detect DL library bugs.

In conclusion, our experiment shows that COMET effectively detects bugs in DL
libraries. More importantly, we found that 18 out of the 21 bugs confirmed by
developers can only be triggered by deliberate layer inputs, layer parameters,
and layer sequences. The remaining three bugs are manifested by directly
calling a layer API with a particular value: NaN. The finding further supports
the insight of COMET, i.e., models with diverse layer API calls help detect DL
library bugs.

###  4.4. RQ3: Performance of Model Synthesis

In the above RQs, we evaluate the effectiveness and significance of COMET by
assessing the diversity of the generated models, the achieved test coverage,
and the real new bugs detected by COMET. In the following, we further
investigate the effectiveness of the major components in COMET. Specifically,
we aim to examine the effectiveness of model synthesis in this RQ by answering
the following two questions: 1) Can our model synthesis method successfully
reduce the size of original models while preserving their layer API call
diversity? 2) Can our model synthesis method boost the efficiency of DL
library testing?

Model Synthesis Results. We first study the effectiveness of our model
synthesis method. Specifically, we investigate the results of our model
synthesis method, including the size of synthesized models and the diversity
of synthesized models. To comprehensively evaluate the synthesis results, we
apply our model synthesis method to the original DL models 20 times and
present the results in Table 9. As shown in Table 9, the 10 published models
(denoted as original models) used by existing techniques contain 2025 layers
and over 300 million weights. However, these original models can only cover
116 unique layer inputs, 122 unique layer parameter values, and 25 unique
layer sequences. Compared with these original models, models synthesized by
our method contain an average of 318 layers in total. At the same time, they
can cover 123.75 layer inputs, 153.85 layer parameters, and 45.25 layer
sequences on average, including all layer inputs, parameters, and sequences
covered by the original models. Moreover, our synthesized models can provide
an even higher diversity of layer parameters and layer sequences. Such results
reflect that our model synthesis method can not only significantly reduce the
original models’ size, but also can preserve and even increase their
diversities.

We further investigate the efficiency of our synthesis method. As is shown in
Figure 9, for most original models, our model synthesis method can synthesize
a new one within one minute (the average synthesis time for most DL models is
less than 20 seconds). We notice that the synthesis time of the
InceptionResNetV2 model is comparably larger than the others. The reason is
that InceptionResNetV2 is carefully crafted with more complex structures
(i.e., it contains 782 layers). It is not easy to analyze and generate a valid
DL model that can cover all layer inputs, layer parameters, and layer
sequences inside it. Therefore, our model synthesis algorithm needs to spend
more time searching until a valid model is found. Nevertheless, since the
model synthesis will only be applied once to the original model and the
maximum synthesis time for each model is 4 minutes, our model synthesis method
is feasible and affordable.

Table 9. Performance of Model Synthesis Algorithm

Initial Seed Models | Total # Layers (Weights) | Layer Inputs | Layer Parameters | Layer Sequences
---|---|---|---|---
Original Models | 2025 (305.28M) | 116 | 122 | 25
Synthesized Models | 317.8$\pm$3.9( 104.16K/$\pm$46.44K) | 123.75$\pm$1.89 | 153.85$\pm$1.53 | 45.25$\pm$2.23

Figure 9. Model Synthesis Time For Each Original Model

Comparison of Efficiency Between Synthesized Models and Original Models. To
investigate how our synthesized models can boost the efficiency of our diverse
model generation, we prepared two sets of initial seed models based on the
original DL models collected in §4.1 and the models synthesized from these
original models. We run our diverse model generation algorithm on each set of
initial seed models for six hours. We observed a significant improvement in
terms of efficiency. In particular, when we used the original models as
initial seeds, the average model generation time overhead is 38.10 seconds. In
contrast, the average model generation time overhead is reduced to 24.35
seconds when we used the synthesized models as initial seeds. As a result,
under the same time budget, using synthesized models as initial seeds can
generate 888 models, 56.33% larger than the total number of models generated
when using the original models (i.e., 568).

We further record the layer input coverage, layer parameter coverage, and
layer sequence coverage during the model generation. Figure 10 presents the
coverage results when using the synthesized models as the initial seed model
(in red) or the original models as the initial seed model (in blue). The
coverage result demonstrates a clear improvement after using the synthesized
models as the initial seed models. Specifically, the layer input coverage
rises from 58.6% to 69.7%, layer parameter coverage increases from 39.0% to
50.2%, and layer sequence coverage improves from 25.3% to 39.0%.

Figure 10. Coverage by using Original Models and Synthesized Model

In summary, compared with the original models, our model synthesis method can
efficiently synthesize models with much smaller sizes while retaining the
original models’ diversity. Using these synthesized models as the initial seed
models for further model generation, we can significantly improve the layer
input coverage, layer parameter coverage, and layer sequence coverage compared
with those using the original models as the initial seed models.

###  4.5. RQ4: Ablation Study on Mutation Operators and Search Algorithm

We further evaluate the effectiveness of our diversity-driven mutation
operators and MCMC-based search algorithm. The diversity-driven mutation
operators aim to target mutating with diverse layer API calls. The MCMC-based
search algorithm is designed to guide model generation towards a higher layer
API call diversity on DL libraries.

To conduct an ablation study of the proposed mutation operators, we
constructed another baseline named COMETo. Instead of using our proposed
mutation operators, COMETo adopts the existing mutation operators designed by
LEMON (Wang et al., 2020), DeepMutation (Ma et al., 2018), and Audee (Guo et
al., 2020). These mutation operators are designed to randomly mutate the
model’s architecture and parameters. If COMET can outperform COMETo, our
proposed mutation operators would be more helpful in diversifying layer API
calls. To conduct an ablation study of the proposed MCMC-based search
algorithm, we constructed a baseline named COMETr. COMETr is the same as
COMET, except that COMETr adopts a random mutation strategy instead of the
MCMC-based search algorithm. In other words, COMETr randomly selects a
mutation operator in each iteration for model generation. If COMET can
outperform COMETr, our MCMC-based search algorithm would help guide model
generation towards higher diversity and test coverage.

We ran COMET, COMETo, and COMETr for six hours. Similar to the setting in
§4.4, we use layer input coverage, layer parameter coverage, and layer
sequence coverage to evaluate their effectiveness. We present our results in
Figure 11. Our result shows that, compared to COMETo and COMETr, COMET can
effectively increase the layer API call diversity of the generated models on
TensorFlow, and the improvement is significant. In particular, compared with
COMETo, COMET can cover 50.4% more layer inputs, increase the layer parameter
coverage by 39.6%, and improve the layer sequence coverage by 30.5%. The
improvement by using our proposed search algorithm is also noticeable. Besides
improving layer API call diversity, we observe that COMET’s MCMC-based search
algorithm also contributes to the test coverage. Specifically, COMET can
increase the branch coverage of COMETo and COMETr by 6.40% and 1.09%,
respectively.

The result of the ablation study shows that both our mutation operators and
search algorithm contribute to generating diverse DL models.

Figure 11. Comparison of COMET, COMETo, and COMETr

##  5\. Discussions

###  5.1. Analysis Of Uncovered Branches

As is shown in §4.2, all existing techniques and COMET can cover only a
portion of branches (at most 19.4%) and lines (at most 34.9%) inside
TensorFlow. Following the Reachability, Infection, Propagation (RIP) model
(Ammann and Offutt, 2008), a program failure requires the location of the
program that contains the fault must be reached. In this section, we collected
and analyzed some library branches that COMET cannot cover. Through this
study, we hope to better understand why these branches are not reached and how
we can reach them in the future. In particular, we collected uncovered
branches of two representative modules: keras/convolution.py and
ops/array_ops.py. The former module implements the Keras’ layer APIs, and the
latter supports implementing low-level tensor operators. Specifically, COMET
covers 173 out of 356 branches in keras/convolution.py and 133 out of 462
branches in ops/array_ops.py.

We observed two main reasons COMET fails to cover branches in our analyzed
modules. The first reason is caused by the implementation. In line with
existing works, we use Keras to generate DL models. However, Keras only
provides APIs to construct high-level layers, while using Keras’ APIs cannot
directly access many low-level operators. As a result, these low-level
operators’ input diversity is limited. For instance, we found that a specific
parameter (i.e., axis) of the low-level operator tf.stack is always 0 when
using Keras to generate models, resulting in around half of the branches
inside this operator’s implementation being uncovered. Nevertheless, COMET can
still improve the branch coverage of the state-of-the-art technique (i.e.,
Muffin) from 16.0% to 19.4%, regardless of this implementation restriction.
The testing paradigm causes another reason. We observe that in our analyzed
modules, there exist some utility functions that cannot be invoked through a
DL model. For instance, some private methods in Convolution class are designed
for developers, and these methods cannot be invoked through public APIs
provided by TensorFlow developers.

###  5.2. Effect Of the Hyperparameter $\sigma$ on Numeric Parameter Mutation

On top of the remarkable results achieved by COMET, we explore a further
question: whether the setting of the hyperparameter $\sigma$ defined in §3.2.2
will affect the effectiveness of COMET? Specifically, $\sigma$ is used to
define the size of value space for numeric parameters so COMET will not
exhaust all possible values. In our experiment, we set $\sigma=5$ based on our
experience. We further discuss whether changing the value of $\sigma$ will
influence the model diversity and test coverage on the DL library achieved by
COMET. To answer this question, we set $\sigma$ from 5 to 55 with the step of
5, and examined the branch coverage, line coverage, layer input coverage, and
layer sequence coverage. We also fix the value range of each numeric parameter
for different values of $\sigma$. 777See our implementation at
https://github.com/maybeLee/COMET for details. Since changing the value of
$\sigma$ will influence the size of test space for numeric parameters,
different $\sigma$ will output different layer parameter coverage results on
the same DL model. Therefore, we did not compare the layer parameter coverage
in this discussion. We can see from Figure 12 that branch, line, and input
coverage are relatively insensitive to the value of $\sigma$, while sequence
coverage may vary a bit but not significantly when we chose different values
of $\sigma$. For instance, the branch coverage achieved by COMET with
different $\sigma$ fluctuates from 18.8% to 19.4%, the layer sequence coverage
fluctuates from 29.3% to 39.5%.

Figure 12. Effect Of Hyperparameter $\sigma$

###  5.3. Threats to Validity

Overall, there are three threats that may affect the experimental results. We
ran experiments with a time budget of six hours for each baseline on the same
machine in turn. Yet, the amount of available system resources for the
experiments may vary across 24 hours. To alleviate the influence caused by
system computing performance, we set the total number of threads/GPU cards
used in each experiment to be the same. The second threat is the number of new
bugs detected by COMET. For those unconfirmed crash bugs, following existing
work (Wang et al., 2020), we used the stack trace to distinguish them.
However, it is possible that one crash bug can have different stack traces. To
alleviate this problem, we manually checked the root cause of each unconfirmed
crash bug to reduce false positives. The third threat is introduced by our
test coverage measurement. The fact that coverage is measured in only one DL
library may affect the evaluation result. To alleviate the bias of
measurement, our criteria for selecting the representative library to collect
code coverage follow the practice of one recent JVM testing work (Chen et al.,
2016a). Specifically, we choose TensorFlow as the representative because 1) it
is the DL library with the highest GitHub stars and is the most popular one
with the richest implementations, 2) all existing DL library testing
techniques consider TensorFlow as their subject under test, and 3) the testing
practices and bugs inside DL libraries share a significant commonality (Jia et
al., 2021a; Nejadgholi and Yang, 2020; Yang et al., 2022; Chen et al., 2022).
Therefore, the code coverage in TensorFlow can reflect DL library testing
techniques’ effectiveness in terms of test coverage. To collect the branch
coverage and line coverage on TensorFlow’s model construction and execution
modules, we iterated all modules inside TensorFlow’s source code and chose the
related ones. Indeed, this may introduce some bias (e.g., some modules may be
missed by us). To alleviate this problem, we carefully read TensorFlow’s
documentation and the description in each module. We publish our collected
module names on our project site.

##  6\. Related Works

###  6.1. Testing of DL Library

Besides testing DL libraries by DL models (Pham et al., 2019; Guo et al.,
2020; Wang et al., 2020; Luo et al., 2021; Gu et al., 2022), there are other
studies on a related testing problem with different focuses. DocTer (Xie et
al., 2022) extracts the argument constraint of DL library functions from the
documentation of DL libraries. To analyze the free-form API documentation,
DocTer first preprocesses and parses the documentation into the dependency
parse tree and employs a learning algorithm to extract constraints from this
documentation. Based on these constraints, it randomly generates two types of
test inputs: valid inputs and invalid inputs. FreeFuzz (Wei et al., 2022) also
targets collecting DL library functions constraint. Instead of statically
collecting the constraints as DocTer, it instruments the library functions and
obtains their input constraints by running code/model from open source (i.e.,
library documentation, unit test suite written by developers, and DL models in
the wild). After collecting input constraints, it randomly generates test
inputs following them for DL library testing. Predoo (Zhang et al., 2021)
proposes a fuzzing technique to test the operators in the DL library. It aims
to estimate and detect individual DL operator’s precision errors.
Specifically, Predoo collects publicly available test inputs as the seeds and
generates variants by iteratively adding perturbations such as Gaussian noise
to the initial seeds to maximize output precision errors. Furthermore, TVMFuzz
(Shen et al., 2021) focuses on the testing of the DL compilers. The major
difference between our works and these works is that COMET generates DL models
as test inputs to test DL library modules related to a DL model’s execution.

###  6.2. Empirical Studies for DL Libraries

To date, researchers also conducted some empirical studies on DL libraries. We
briefly group them into two categories: studies on DL framework bugs and
studies on DL library testing practice. Jia et al. (Jia et al., 2020) report
an empirical study to understand the characteristics of the bugs in
TensorFlow. They further study the symptoms, causes and repair patterns of
TensorFlow bugs (Jia et al., 2021b). Their studies reveal that the root causes
of a large portion of reported bugs reside in the algorithm implementations or
the interfaces (i.e., API) provided by TensorFlow. Tambon et al. (Tambon et
al., 2021) study silent bugs such as performance or accuracy issues inside
TensorFlow. Silent bugs will lead to incorrect behavior, but they do not cause
system crashes or hang, nor show an error message to users. This study
indicates that inaccurate calculation result is the most common symptom of
silent bugs. Chen et al. (Chen et al., 2022) conduct a comprehensive study to
facilitate a sufficient understanding of DL library bugs. Compared with
previous studies, they took not only TensorFlow but also PyTorch, MXNet, and
Deeplearning4j as the study subjects. Based on their analysis, they point out
that components implemented for deep learning algorithms are the buggiest
ones, and over 53.75% of bugs are observed at the model training stage, which
could lead to lengthy testing and debugging.

Several studies have been proposed to study the DL library testing practice.
Nejadgholi et al. (Nejadgholi and Yang, 2020) conduct the first study on the
design and evolution of oracle approximation(OA) assertions inside the DL
libraries’ test suite. They point out that a non-negligible portion (25%) of
OA assertions in DL libraries and developers frequently change OA assertions.
Wang et al. (Wang et al., 2021) conduct an empirical study on five machine
learning libraries with two popular unit test case generation tools, i.e.,
EVOSUITE (Fraser and Arcuri, 2011) and Randoop (Pacheco and Ernst, 2007). They
find out that most machine learning libraries do not maintain a high-quality
test suite to achieve satisfactory code coverage; although existing unit test
case generation tools can help improve test suite quality, the improvement is
limited. Jia et al. (Jia et al., 2021a) evaluate the quality of unit test
suites in DL frameworks using mutation analysis. They find out that existing
test cases are more effective in detecting bugs inside incorrect control flow
logic, while they perform worse when wrong values cause the bug. Instead of
focusing on the root causes and symptoms of library bugs or analyzing the
testing practice adopted by DL library developers, we focus on the generation
of diverse DL models in terms of their layer API calls.

###  6.3. Mutation Testing for Deep Learning

There are also some mutation testing works proposed to modify DL models.
DeepMutation (Ma et al., 2018) proposes a set of source-level mutations to
inject faults into the training data and training program, and model-level
mutations to inject the faults into the trained DL model by mutating the value
of neurons or adding/removing some layers. By doing so, DeepMutation can
inject faults into the DL application and use the mutated DL application to
evaluate the quality of DL application’s test suite (i.e., testing data).
DeepCrime (Humbatova et al., 2021) defines 35 source-level mutation operators
based on real DL program faults to inject coding errors into the training
programs and labeling errors into the training data. Same as DeepMutation,
DeepCrime is also designed to inject faults into the DL application to
evaluate the quality of test suites. Wang et al. (Wang et al., 2019) apply
model-level mutation to inject the faults into the trained DL model by
mutating the value of neurons. They further guide adversarial test input
generation using the mutated DL model based on the insight that adversarial
test input would be more sensitive to mutations on the DL model. Different
from these mutation testing works, our work designed mutation operators to
generate test input instead of generating the mutants of the subject under
test. In other words, we use the mutation operator to apply changes to the
seed DL model rather than injecting faults to the subject under test.

##  7\. Conclusion

In this paper, we propose a novel technique named COMET to test DL libraries
by generating diverse layer API calls as test inputs. Motivated by the model
generation script of Keras, COMET defines three coverage criteria to measure
the layer API call diversity: layer input coverage, layer parameter coverage,
and layer sequence coverage. Driven by these criteria, COMET proposes a set of
novel mutation operators and a coverage-guided search algorithm to search for
DL models with diverse layer API calls to achieve more comprehensive testing
on DL libraries. It also proposes a model synthesis method to significantly
boost the efficiency of diverse model generation by addressing the challenge
of runtime overhead incurred by large models. Our evaluation of popular DL
libraries shows that COMET significantly outperforms the state-of-the-art DL
library testing techniques in the effectiveness of diverse model generation.
In total, 32 new bugs were detected by COMET, among which, 21 of them were
confirmed by developers and 7 of these confirmed bugs have been fixed by
developers.

## Acknowledgements

We would like to thank the anonymous reviewers for their comments and
suggestions. We would also like to thank DL library developers for analyzing
our reported issues. This work was supported by the National Natural Science
Foundation of China (Grant Nos. 61932021, 62002125), the National Key Research
and Development Program of China (Grant No. 2019YFE0198100), the Hong Kong
RGC/GRF (Grant No. 16205722), the Hong Kong ITF (Grant No. MHP/055/19), and
the MSRA Collaborative Research Grant.

## References

  * (1)
  * Abadi et al. (2016) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016\.  TensorFlow: A System for Large-Scale Machine Learning. In _Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation_ (Savannah, GA, USA) _(OSDI’16)_. USENIX Association, USA, 265–283.
  * Ammann and Offutt (2008) Paul Ammann and Jeff Offutt. 2008.  _Introduction to Software Testing_ (1 ed.).  Cambridge University Press, USA.
  * Ardakani et al. (2020) A. Ardakani, Alireza Rajabzadeh Kanafi, U. Acharya, Nazanin Khadem, and A. Mohammadi. 2020\.  Application of deep learning technique to manage COVID-19 in routine clinical practice using CT images: Results of 10 convolutional neural networks.  _Computers in Biology and Medicine_ 121 (2020), 103795 – 103795\.
  * Bhattacharya et al. (2020) S. Bhattacharya, Praveen Kumar Reddy Maddikunta, Quoc-Viet Pham, T. Gadekallu, S SivaRamaKrishnan, C. L. Chowdhary, M. Alazab, and Md. Jalil Piran. 2020.  Deep learning and medical image processing for coronavirus (COVID-19) pandemic: A survey.  _Sustainable Cities and Society_ 65 (2020), 102589 – 102589\.
  * Chen et al. (2015b) Chenyi Chen, Ari Seff, Alain Kornhauser, and Jianxiong Xiao. 2015b.  DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving. In _2015 IEEE International Conference on Computer Vision (ICCV)_. IEEE, Santiago, Chile, 2722–2730.  https://doi.org/10.1109/ICCV.2015.312
  * Chen et al. (2022) Junjie Chen, Yihua Liang, Qingchao Shen, and Jiajun Jiang. 2022\.  Toward Understanding Deep Learning Framework Bugs.  https://doi.org/10.48550/ARXIV.2203.04026
  * Chen et al. (2015a) Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. 2015a.  MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems.  arXiv:1512.01274 [cs.DC]
  * Chen et al. (2016a) Yuting Chen, Ting Su, Chengnian Sun, Zhendong Su, and Jianjun Zhao. 2016a.  Coverage-Directed Differential Testing of JVM Implementations.  _SIGPLAN Not._ 51, 6 (June 2016), 85–99.  https://doi.org/10.1145/2980983.2908095
  * Chen et al. (2016b) Yuting Chen, Ting Su, Chengnian Sun, Zhendong Su, and Jianjun Zhao. 2016b.  Coverage-Directed Differential Testing of JVM Implementations. In _Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation_ (Santa Barbara, CA, USA) _(PLDI ’16)_. Association for Computing Machinery, New York, NY, USA, 85–99.  https://doi.org/10.1145/2908080.2908095
  * Chen and Su (2015) Yuting Chen and Zhendong Su. 2015.  Guided Differential Testing of Certificate Validation in SSL/TLS Implementations. In _Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering_ (Bergamo, Italy) _(ESEC/FSE 2015)_. Association for Computing Machinery, New York, NY, USA, 793–804.  https://doi.org/10.1145/2786805.2786835
  * Chollet et al. (2015) François Chollet et al. 2015\.  keras.
  * Conv2D and Operators (2022) Operator Fusion For Conv2D and Mul Operators. Accessed: 2022.  https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/core/optimizer/conv_mul_fusion.cc.
  * Conv3DTranspose (2022) Bug Report For Conv3DTranspose. Accessed: 2022.  https://github.com/keras-team/keras/issues/16933.
  * Das et al. (2018) Rajshekhar Das, Akshay Gadre, Shanghang Zhang, Swarun Kumar, and Jose M. F. Moura. 2018.  A Deep Learning Approach to IoT Authentication. In _2018 IEEE International Conference on Communications (ICC)_. IEEE, Kansas City, MO, USA, 1–6.  https://doi.org/10.1109/ICC.2018.8422832
  * DepthwiseConv2D (2022) Bug Report For DepthwiseConv2D. Accessed: 2022.  https://github.com/keras-team/keras/issues/16314.
  * Ferdowsi and Saad (2019) Aidin Ferdowsi and W. Saad. 2019.  Deep Learning for Signal Authentication and Security in Massive Internet-of-Things Systems.  _IEEE Transactions on Communications_ 67 (2019), 1371–1387.
  * Fraser and Arcuri (2011) Gordon Fraser and Andrea Arcuri. 2011.  EvoSuite: Automatic Test Suite Generation for Object-Oriented Software. In _Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering_ (Szeged, Hungary) _(ESEC/FSE ’11)_. Association for Computing Machinery, New York, NY, USA, 416–419.  https://doi.org/10.1145/2025113.2025179
  * Gu et al. (2022) Jiazhen Gu, Xuchuan Luo, Yangfan Zhou, and Xin Wang. 2022\.  Muffin: Testing Deep Learning Libraries via Neural Architecture Fuzzing. In _Proceedings of the 44th International Conference on Software Engineering_ (Pittsburgh, Pennsylvania) _(ICSE ’22)_. Association for Computing Machinery, New York, NY, USA, 1418–1430.  https://doi.org/10.1145/3510003.3510092
  * Guo et al. (2020) Qianyu Guo, Xiaofei Xie, Yi Li, Xiaoyu Zhang, Yang Liu, Xiaohong Li, and Chao Shen. 2020.  Audee: Automated Testing for Deep Learning Frameworks. In _2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, Melbourne, VIC, Australia, 486–498.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Identity Mappings in Deep Residual Networks. In _Computer Vision – ECCV 2016_ , Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (Eds.). Springer International Publishing, Cham, 630–645.
  * Humbatova et al. (2021) Nargiz Humbatova, Gunel Jahangirova, and Paolo Tonella. 2021\.  DeepCrime: Mutation Testing of Deep Learning Systems Based on Real Faults. In _Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis_ (Virtual, Denmark) _(ISSTA 2021)_. Association for Computing Machinery, New York, NY, USA, 67–78.  https://doi.org/10.1145/3460319.3464825
  * Jia et al. (2021a) Li Jia, Hao Zhong, and Linpeng Huang. 2021a.  The Unit Test Quality of Deep Learning Libraries: A Mutation Analysis. In _2021 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE, Luxembourg, 47–57.  https://doi.org/10.1109/ICSME52107.2021.00011
  * Jia et al. (2020) Li Jia, Hao Zhong, Xiaoyin Wang, Linpeng Huang, and Xuansheng Lu. 2020.  An Empirical Study on Bugs Inside TensorFlow. In _Database Systems for Advanced Applications_ , Yunmook Nah, Bin Cui, Sang-Won Lee, Jeffrey Xu Yu, Yang-Sae Moon, and Steven Euijong Whang (Eds.). Springer International Publishing, Cham, 604–620.
  * Jia et al. (2021b) Li Jia, Hao Zhong, Xiaoyin Wang, Linpeng Huang, and Xuansheng Lu. 2021b.  The symptoms, causes, and repairs of bugs inside a deep learning library.  _Journal of Systems and Software_ 177 (2021), 110935.  https://doi.org/10.1016/j.jss.2021.110935
  * Keras (2022) How Kernel Size Influences Padding In Keras. Accessed: 2022.  https://github.com/keras-team/keras/blob/db667716ac6be59a8495b58e0d2e246e9c031599/keras/layers/convolutional/base_conv.py#L389.
  * Krizhevsky (2009) Alex Krizhevsky. 2009\.  _Learning Multiple Layers of Features from Tiny Images_.  https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012\.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ 25 (2012), 1097–1105.
  * Kuhn et al. (2004) D.R. Kuhn, D.R. Wallace, and A.M. Gallo. 2004.  Software fault interactions and implications for software testing.  _IEEE Transactions on Software Engineering_ 30, 6 (2004), 418–421.  https://doi.org/10.1109/TSE.2004.24
  * Lecun et al. (1998) Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. 1998\.  Gradient-based learning applied to document recognition.  _Proc. IEEE_ 86, 11 (1998), 2278–2324.  https://doi.org/10.1109/5.726791
  * Litjens et al. (2017) Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara I Sánchez. 2017.  A survey on deep learning in medical image analysis.  _Medical image analysis_ 42 (2017), 60–88.
  * Luo et al. (2021) Weisi Luo, Dong Chai, Xiaoyue Run, Jiang Wang, Chunrong Fang, and Zhenyu Chen. 2021\.  Graph-Based Fuzz Testing for Deep Learning Inference Engines. In _Proceedings of the 43rd International Conference on Software Engineering_ _(ICSE ’21)_. IEEE Press, Madrid, Spain, 288–299.  https://doi.org/10.1109/ICSE43902.2021.00037
  * Ma et al. (2018) Lei Ma, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Felix Juefei-Xu, Chao Xie, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang. 2018\.  DeepMutation: Mutation Testing of Deep Learning Systems. In _2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE)_. IEEE, Memphis, 100–111.  https://doi.org/10.1109/ISSRE.2018.00021
  * Mxnet (2022) Keras Apache Mxnet. Accessed: 2022.  keras-apache-mxnet.  https://github.com/awslabs/keras-apache-mxnet.
  * Nejadgholi and Yang (2020) Mahdi Nejadgholi and Jinqiu Yang. 2020.  A Study of Oracle Approximations in Testing Deep Learning Libraries. In _Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering_ _(ASE ’19)_. IEEE Press, San Diego, California, 785–796.  https://doi.org/10.1109/ASE.2019.00078
  * Odena et al. (2019) Augustus Odena, Catherine Olsson, David Andersen, and Ian Goodfellow. 2019.  TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing. In _Proceedings of the 36th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 97)_ , Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, Long Beach Convention Center, Long Beach, 4901–4911.  https://proceedings.mlr.press/v97/odena19a.html
  * Onnx (2022a) Onnx. Accessed: 2022a.  Open standard for machine learning interoperability.  https://github.com/onnx/onnx.
  * Onnx (2022b) Tensorflow Onnx. Accessed: 2022b.  tf2onnx - Convert TensorFlow, Keras, Tensorflow.js and Tflite models to ONNX.  https://github.com/onnx/tensorflow-onnx.
  * Onnx2pytorch (2022) Onnx2pytorch. Accessed: 2022\.  ONNX to PyTorch.  https://github.com/ToriML/onnx2pytorch.
  * Pacheco and Ernst (2007) Carlos Pacheco and Michael D. Ernst. 2007.  Randoop: Feedback-Directed Random Testing for Java. In _Companion to the 22nd ACM SIGPLAN Conference on Object-Oriented Programming Systems and Applications Companion_ (Montreal, Quebec, Canada) _(OOPSLA ’07)_. Association for Computing Machinery, New York, NY, USA, 815–816.  https://doi.org/10.1145/1297846.1297902
  * Padding (2022) Bug Report For Inconsistent Padding. Accessed: 2022.  https://github.com/ToriML/onnx2pytorch/pull/41.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  _PyTorch: An Imperative Style, High-Performance Deep Learning Library_.  Curran Associates Inc., Red Hook, NY, USA.
  * Pei et al. (2019) Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2019\.  DeepXplore: Automated Whitebox Testing of Deep Learning Systems.  _Commun. ACM_ 62, 11 (oct 2019), 137–145.  https://doi.org/10.1145/3361566
  * Pham et al. (2019) Hung Viet Pham, Thibaud Lutellier, Weizhen Qi, and Lin Tan. 2019\.  CRADLE: cross-backend validation to detect and localize bugs in deep learning libraries. In _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. IEEE, IEEE, Montreal, QC, Canada, 1027–1038.
  * Runtime (2022) ONNX Runtime. Accessed: 2022\.  Optimize and Accelerate Machine Learning Inferencing and Training.  https://onnxruntime.ai/.
  * Sallab et al. (2017) Ahmad El Sallab, Mohammed Abdou, E. Perot, and S. Yogamani. 2017\.  Deep Reinforcement Learning framework for Autonomous Driving.
  * Shalev-Shwartz et al. (2016) S. Shalev-Shwartz, Shaked Shammah, and A. Shashua. 2016\.  Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving.
  * Shen et al. (2021) Qingchao Shen, Haoyang Ma, Junjie Chen, Yongqiang Tian, Shing-Chi Cheung, and Xiang Chen. 2021\.  A Comprehensive Study of Deep Learning Compiler Bugs. In _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Athens, Greece) _(ESEC/FSE 2021)_. Association for Computing Machinery, New York, NY, USA, 968–980.  https://doi.org/10.1145/3468264.3468591
  * Strategy (2022) TensorFlow Padding Strategy. Accessed: 2022.  https://www.tensorflow.org/api_docs/python/tf/nn.
  * Szegedy et al. (2015) Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015.  Going deeper with convolutions. In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, Boston, MA, 1–9.  https://doi.org/10.1109/CVPR.2015.7298594
  * Tambon et al. (2021) Florian Tambon, Amin Nikanjam, Le An, Foutse Khomh, and Giuliano Antoniol. 2021.  Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlow.  https://doi.org/10.48550/ARXIV.2112.13314
  * Wang et al. (2019) Jingyi Wang, Guoliang Dong, Jun Sun, Xinyu Wang, and Peixin Zhang. 2019.  Adversarial Sample Detection for Deep Neural Network through Model Mutation Testing. In _Proceedings of the 41st International Conference on Software Engineering_ _(ICSE ’19)_. IEEE Press, Montreal, Quebec, Canada, 1245–1256.  https://doi.org/10.1109/ICSE.2019.00126
  * Wang et al. (2016) Ran Wang, Daming Zou, Xinrui He, Yingfei Xiong, Lu Zhang, and Gang Huang. 2016\.  Detecting and Fixing Precision-Specific Operations for Measuring Floating-Point Errors. In _Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering_ (Seattle, WA, USA) _(FSE 2016)_. Association for Computing Machinery, New York, NY, USA, 619–630.  https://doi.org/10.1145/2950290.2950355
  * Wang et al. (2021) Song Wang, Nishtha Shrestha, Abarna Kucheri Subburaman, Junjie Wang, Moshi Wei, and Nachiappan Nagappan. 2021.  Automatic Unit Test Generation for Machine Learning Libraries: How Far Are We?. In _Proceedings of the 43rd International Conference on Software Engineering_ _(ICSE ’21)_. IEEE Press, Madrid, Spain, 1548–1560.  https://doi.org/10.1109/ICSE43902.2021.00138
  * Wang et al. (2020) Zan Wang, Ming Yan, Junjie Chen, Shuang Liu, and Dongdi Zhang. 2020.  Deep Learning Library Testing via Effective Model Generation. In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Virtual Event, USA) _(ESEC/FSE 2020)_. Association for Computing Machinery, New York, NY, USA, 788–799.  https://doi.org/10.1145/3368089.3409761
  * Wardat et al. (2021) Mohammad Wardat, Wei Le, and Hridesh Rajan. 2021.  DeepLocalize: Fault Localization for Deep Neural Networks. In _Proceedings of the 43rd International Conference on Software Engineering_ _(ICSE ’21)_. IEEE Press, Madrid, Spain, 251–262.  https://doi.org/10.1109/ICSE43902.2021.00034
  * Wei et al. (2022) Anjiang Wei, Yinlin Deng, Chenyuan Yang, and Lingming Zhang. 2022\.  Free Lunch for Testing: Fuzzing Deep-Learning Libraries from Open Source. In _Proceedings of the 44th International Conference on Software Engineering_ (Pittsburgh, Pennsylvania) _(ICSE ’22)_. Association for Computing Machinery, New York, NY, USA, 995–1007.  https://doi.org/10.1145/3510003.3510041
  * Xie et al. (2022) Danning Xie, Yitong Li, Mijung Kim, Hung Viet Pham, Lin Tan, Xiangyu Zhang, and Michael W. Godfrey. 2022.  DocTer: Documentation-Guided Fuzzing for Testing Deep Learning API Functions. In _Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis_ (Virtual, South Korea) _(ISSTA 2022)_. Association for Computing Machinery, New York, NY, USA, 176–188.  https://doi.org/10.1145/3533767.3534220
  * Yang et al. (2022) Yilin Yang, Tianxing He, Zhilong Xia, and Yang Feng. 2022\.  A comprehensive empirical study on bug characteristics of deep learning frameworks.  _Information and Software Technology_ 151 (2022), 107004.  https://doi.org/10.1016/j.infsof.2022.107004
  * Zhang et al. (2018) Mengshi Zhang, Yuqun Zhang, Lingming Zhang, Cong Liu, and Sarfraz Khurshid. 2018.  DeepRoad: GAN-Based Metamorphic Testing and Input Validation Framework for Autonomous Driving Systems. In _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_ (Montpellier, France) _(ASE 2018)_. Association for Computing Machinery, New York, NY, USA, 132–142.  https://doi.org/10.1145/3238147.3238187
  * Zhang et al. (2021) Xufan Zhang, Ning Sun, Chunrong Fang, Jiawei Liu, Jia Liu, Dong Chai, Jiang Wang, and Zhenyu Chen. 2021\.  _Predoo: Precision Testing of Deep Learning Operators_.  Association for Computing Machinery, New York, NY, USA, 400–412.  https://doi.org/10.1145/3460319.3464843
  * Zhang et al. (2020) Yuhao Zhang, Luyao Ren, Liqian Chen, Yingfei Xiong, Shing-Chi Cheung, and Tao Xie. 2020\.  Detecting Numerical Bugs in Neural Network Architectures. In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Virtual Event, USA) _(ESEC/FSE 2020)_. Association for Computing Machinery, New York, NY, USA, 826–837.  https://doi.org/10.1145/3368089.3409720