#  DLLens: Testing Deep Learning Libraries via LLM-aided Synthesis

Meiziniu Li, Dongze Li, Jianmeng Liu, Jialun Cao, Yongqiang Tian, Shing-Chi
Cheung  The Hong Kong University of Science and Technology, Hong Kong, China
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Testing is a major approach to ensuring the quality of deep learning (DL)
libraries. Existing testing techniques commonly adopt differential testing to
relieve the need for test oracle construction. However, these techniques are
limited in finding implementations that offer the same functionality and
generating diverse test inputs for differential testing. This paper introduces
DLLens, a novel differential testing technique for DL library testing. Our
insight is that APIs in different DL libraries are commonly designed to
accomplish various computations for the same set of published DL algorithms.
Although the mapping of these APIs is not often one-to-one, we observe that
their computations can be mutually simulated after proper composition and
adaptation. The use of these simulation counterparts facilitates differential
testing for the detection of functional DL library bugs. Leveraging the
insight, we propose DLLens as a novel mechanism that utilizes a large language
model (LLM) to synthesize valid counterparts of DL library APIs. To generate
diverse test inputs, DLLens incorporates a static analysis method aided by
LLMs to extract path constraints from all execution paths in each API and its
counterpart’s implementations. These path constraints are then used to guide
the generation of diverse test inputs. We evaluate DLLens on two popular DL
libraries, TensorFlow and PyTorch. Our evaluation shows that DLLens can
synthesize counterparts for more than twice as many APIs found by state-of-
the-art techniques on these libraries. Moreover, DLLens can extract 26.7% more
constraints and detect 2.5 times as many bugs as state-of-the-art techniques.
DLLens has successfully found 56bugs in recent TensorFlow and PyTorch
libraries. Among them, 41are previously unknown, 39of which have been
confirmed by developers after reporting, and 19 of those confirmed bugs have
been fixed by developers.

##  I Introduction

Deep learning (DL) has been increasingly adopted for mission-critical
applications such as authentication [1, 2], medical treatment [3], and
autonomous driving [4, 5, 6]. Current DL applications are mostly developed on
top of popular DL libraries such as PyTorch [7] and TensorFlow [8]. However,
the presence of functional incorrectness, commonly referred to as functional
bugs, in DL libraries poses a significant threat to the reliability of these
applications [9, 10, 11]. More than 30% of bugs reported to PyTorch developers
are categorized as functional bugs [12]. Hence, functional bug detection is
critical to assuring the quality of DL libraries.

A functional bug occurs when the library API’s behavior deviates from its
specified requirements [13]. Currently, differential testing is commonly
adopted [14, 15, 16, 17, 18] to detect these functional bugs (e.g., the
incorrect computation results or intermediate states [12]). A general workflow
of existing differential testing techniques for DL libraries is to (1) collect
the different implementations that offer the same functionality in/across DL
libraries, and (2) apply test inputs to detect output inconsistency. However,
existing works are limited in both steps, which may hinder their effectiveness
in bug detection.

(1) Ineffectiveness in finding counterparts for differential testing. Two
paradigms are proposed to find counterparts of DL library APIs for
differential testing. The first one [14, 15, 16, 17, 18] takes advantage of
model conversion libraries (e.g., TF2ONNX [19]), which support the conversion
of DL models between libraries. However, the effectiveness of this line of
research is limited by the insufficient API coverage of conversion libraries.
Take TF2ONNX [19] as an example, it only supports the conversion of 8%
(279/3,316) TensorFlow APIs [18], leaving 90%+ of TensorFlow APIs uncovered.
Another paradigm takes advantage of different computational modes, such as
backends [20, 21, 22] (e.g., CPU and GPU), execution modes [23] (e.g.,
different gradient calculation modes), and similar APIs [24] (e.g., tf.argmax
and tf.math.argmax) in one library. Similar APIs or executions of an API under
these different computational modes could also serve as counterparts of each
other. However, these counterparts often have nearly identical
implementations, limiting their effectiveness for differential testing.
Listing 1 shows a real functional bug that cannot be triggered by either of
the existing paradigms, since none of the conversion libraries support the API
(tf.math.is_non_decreasing), and the concerned computational modes deliver
consistent outputs [25].

⬇

1

2|\underline{\textbf{Bug Triggering Input}}|:

3x = tf.constant([10,9], dtype=’uint32’)

4|\underline{\textbf{Buggy API}}|:

5tf.math.is_non_decreasing

6|\underline{\textbf{Actual Result (Expected Result)}}|:

7|\textbf{\textcolor{bgRed}{True}}| (|\textbf{\textcolor{bgGreen}{False}}|)

8|\underline{\textbf{Developer’s Fix}}|:

9|\textbf{\textcolor{bgRed}{- diff =
\\_get\\_diff\\_for\\_monotonic\\_comparison(x)}}|

10|\textbf{\textcolor{bgRed}{- zero = ops.convert\\_to\\_tensor(0,
dtype=diff.dtype)}}|

11|\textbf{\textcolor{bgRed}{- return
math\\_ops.reduce\\_all(math\\_ops.less\\_equal(zero, diff))}}|

12|\textbf{\textcolor{bgGreen}{+ diff =
\\_get\\_results\\_for\\_monotonic\\_comparison(x, greater\\_equal)}}|

13|\textbf{\textcolor{bgGreen}{+ return math\\_ops.reduce\\_all(diff)}}|’

Figure 1: A real bug detected by our tool that leads to incorrect computation
result.

(2) Ineffectiveness in generating diverse test inputs. Another major cause is
the ineffectiveness of generating test inputs that induce inconsistent outputs
across different implementations. Existing DL library testing approaches
commonly generate test inputs towards covering more APIs/statements/branches
[21, 17, 26]. However, generating test inputs to trigger bugs inside these
buggy APIs/statements/branches may be more complicated. Take Listing 2 as an
example, the buggy line (line 6) leads to an integer overflow [27]. The bug
can only be triggered when satisfying the following two conditions: (1) (delta
¿ 0 && start ¿ limit)——delta ¡ 0 && start ¡ limit, and (2) limit is a large
negative integer && start is a large positive integer. Indeed, coverage-guided
approaches could generate inputs that satisfy condition (1), yet fail to
explore different values of start and limit to meet condition 2. In other
words, these approaches are ineffective in exploring inconsistency-inducing
test inputs along the execution paths that contain faulty statements.

⬇

1

2|\underline{\textbf{Buggy Code}}|

3...

41: if (delta > 0) {

52: // Error if start <= limit

63: } else {

74: // Error if start >= limit

85: }

9...

106: Eigen::numext::ceil(Eigen::numext::abs(|\textbf{\textcolor{bgRed}{(limit
- start) / delta}}|)); // |\textcolor{bgRed}{\textbf{Buggy Line}}|

11|\underline{\textbf{Our Proposed Fix Confirmed By a Developer}}|

126: Eigen::numext::ceil(Eigen::numext::abs(|\textbf{\textcolor{bgGreen}{limit
/ delta - start / delta}}|));

Figure 2: A real bug detected by DLLens. It requires a specific value to
trigger.

Solution. In this paper, we propose DLLens, a novel differential testing
technique for DL library API testing. DLLens can 1) effectively find
counterparts for DL library APIs, and 2) generate diverse inputs to explore
execution paths extracted from implementations of DL library APIs and their
counterparts. We observe that APIs are commonly designed to accomplish various
computations for the same set of published DL algorithms. Although the mapping
of these APIs is not often one-to-one, we observe that their computations can
be simulated mutually after proper composition and adaptation. Based on this
insight, DLLens synthesizes for each DL library API a function, referred to as
the API’s counterpart, by leveraging a large language model (LLM). The
counterpart simulates the API’s computation by composing and adapting various
APIs from a different library. To generate diverse inputs, DLLens incorporates
a novel static analysis method to extract path constraints for each execution
path in the implementations of the API and its counterpart. Our static
analysis method leverages an LLM to extract solvable path constraints from
execution paths that involve external functions with unknown behaviors to
existing solvers. Finally, DLLens applies a test input generation method based
on the path constraints extracted from the implementation of both the
concerned API and its counterpart, differential testing is further applied by
checking output inconsistency between the API and its counterpart for bug
detection. Taking the bug in Listing 1 as an example, the buggy API is
designed to check if a given tensor is following the non-decreasing order
(i.e., for tensor $[x[0],...],x[i]\leq x[i+1]$), which is a commonly DL
functionality [28]. DLLens detected this bug by synthesizing its counterpart,
which is a function built on top of PyTorch APIs (see Listing 3), and
generating a diverse test input that can expose this bug.

⬇

1

2|\underline{\textbf{TensorFlow API}}|

3tf.math.is_non_decreasing(x)

4|\underline{\textbf{Function Using PyTorch’s APIs}}|

5def pytorch_call(x):

6return torch.all(torch.eq(x, torch.sort(x)[0]))’

Figure 3: A function using PyTorch’s APIs can implement the specified
computation of tf.math.is_non_decreasing.

We evaluate the effectiveness of DLLens on two popular DL libraries:
TensorFlow [8] and PyTorch [7]. We compare DLLens against existing DL library
testing techniques on the effectiveness of counterpart synthesis, path
constraint extraction, code coverage, and bug detection. The evaluation
results show that DLLens’s counterpart synthesis method can double the number
of API counterparts (e.g., 739 v.s. 304 for TensorFlow APIs) collected by the
existing work [18] that relies on developer-constructed rules. On average,
DLLens can extract 21.5 and 41.87 path constraints for each TensorFlow API and
PyTorch API, respectively. Our experiment on 200 sampled APIs shows that
DLLens can cover at least 3.2% branches and detect at least 2.5 times as many
bugs as the state-of-the-art techniques. In total, DLLens detected 56bugs
inside TensorFlow and PyTorch, including 15bugs already fixed or confirmed by
developers and 41new bugs. So far, 39of our reported new bugs have been
confirmed by developers, including 19 fixed and 2 reported bugs pending
confirmation.

This work makes the following contributions:

  * •

We propose DLLens, a novel DL library testing tool that can effectively
synthesize API counterparts and extract path constraints for diverse test
input generation.

  * •

DLLens presents a new approach to counterpart synthesis using LLMs,
alleviating the test oracle problems for the testing of DL library APIs.

  * •

This paper proposes a novel path constraint extraction method that
incorporates static analysis with LLMs to effectively extract path constraints
inside DL libraries.

  * •

We applied DLLens to two popular DL libraries, TensorFlow and PyTorch. DLLens
successfully detected 56bugs. Most (41) are new bugs, and 39of them have been
confirmed by developers after being reported. So far, 19 of our reported new
bugs have been fixed by developers.

##  II Motivation and Related Works

###  II-A DL Library Testing

Existing testing approaches for DL libraries focus on either the model level
or API level. The former [14, 29, 15, 26, 16, 17] takes various DL models as
test inputs and tries to exercise specific modules (_e.g._ , model
construction, model training, and inference) in DL libraries. However, a
recent study has revealed that these model-level testing approaches are
ineffective in test adequacy [20]. One potential explanation is that these
approaches can operate on and manipulate only layer APIs within DL libraries,
leaving the majority of library APIs unexplored. Particularly, these model-
level testing approaches can cover at most 2.4% of the total deep learning
library APIs, as reported by previous studies [20, 21].

API-level testing, which improves the coverage of DL library APIs by directly
executing them, provides an alternative approach to testing DL libraries.
Several approaches [20, 30, 31] have been introduced to generate input
arguments (i.e., test inputs) to test these APIs. These approaches employ
various strategies, including collecting input arguments from open source
projects [20], extracting constraints from documentation [30], and utilizing
API’s input validation code [31]. More recently, TitanFuzz [21] and FuzzGPT
[32] leverage large language models (LLMs) like ChatGPT for generating input
arguments for DL library APIs. These approaches demonstrate the feasibility of
providing examples and instructions to LLMs, allowing them to generate input
arguments.

###  II-B Functional Bug Detection In DL Libraries

Existing approaches, as mentioned in Section II-A, mostly adopt differential
testing to check output consistency across implementations offering the same
functionality to detect functional bugs in DL libraries. There are two widely
used paradigms to find implementations offering the same functionality. The
first paradigm [14, 18] leverages model conversion tools such as TF2ONNX [19]
to compare outputs across multiple DL libraries. For instance, given a DL
library API, TensorScope [18], parsed the conversion rules in TF2ONNX [19] to
extract the counterpart of this API, which is implemented in another DL
library. However, existing model conversion tools can only support
counterparts for a limited number of DL library APIs. For example, TF2ONNX
[19] only supports counterparts for only 279 out of 3,316 TensorFlow APIs.
Since the counterparts of many DL library APIs cannot be found in existing
model conversion tools, functional bugs inside these DL library APIs may be
missed (e.g., the bug in Listing 1)

The second paradigm applies differential testing by comparing computation
results of implementations within a single DL libraries. For instance, several
works [20, 23, 22, 21, 32] detect functional bugs via checking if an API can
have consistent outputs under different computation modes, such as backends
(e.g., CPU and GPU), precision (e.g., float32, float16), and execution modes
(e.g., different gradient calculation modes). Nevertheless, these
implementations may yield identical computation results, making checking
result inconsistency for bug detection ineffective. DeepRel [24] mined
multiple groups of similar APIs within the same DL libraries based on API
signature and document similarity. As for the similar APIs found by DeepRel,
existing work [18] reveals that many of these APIs serve as callers, callees,
or aliases of other APIs within the same group. For instance, tf.argmax and
tf.math.argmax found by DeepReal are indeed aliases, suggesting that they are
likely to have the same outputs. Indeed, when applying these differential
testing strategies on one of our detected bugs (i.e., List 1), we find that
the buggy API has consistent output under different computation modes.
Besides, this buggy API’s similar APIs detected by DeepRel produced identical
output as the buggy API’s. Thus these approaches fail to detect this bug.

###  II-C Constraint Extraction for DL Library Testing

Some approaches have proposed constraint extraction strategies [30, 31, 18] to
guide valid/invalid input generation. DocTer [30] collected constraints from
DL library documentation and generated inputs that follow these constraints
(denoted as valid inputs) and inputs violating these constraints. TensorScope
[18] parsed the input validation check statements to generate valid inputs.
However, both TensorScope and DocTer [30] do not provide constraints required
by different execution paths inside the implementation of a DL library API.

ACETest [31] proposed a static analysis tool that extracted path constraints
from the validation code inside DL library APIs, so input following these path
constraints can pass the validation code to reach the core logic inside these
APIs. However, due to the large size of DL libraries, ACETest [31] could not
perform static analysis on the entire code base, leaving some extracted path
constraints incomplete. Indeed, we observe that many path constraints
extracted from the source code of DL library APIs contain functions from
external libraries (e.g., Eigen) or external functions. Path constraints
containing these external functions may not be solved properly since their
behaviors are unknown to existing solvers, and their source code are not
considered by ACETest in static analysis.

##  III Methodology

Figure 4: Workflow of DLLens with three steps: counterpart synthesis (Section
III-A), path constraint extraction (Section III-B, and input generation
(Section III-C).

Figure 4 shows the workflow of DLLens, which consists of three steps, _i.e._ ,
(1) Counterpart Synthesis, (2) Path Constraint Extraction, and (3) Test Input
Generation and Bug Detection. For each DL library API under test $f$, DLLens
first employs the LLM with dynamic validation to synthesize a function
$f^{\prime}$ that simulates the intended computation of the API by making use
of other library APIs. The function $f^{\prime}$ is referred to as a
counterpart of $f$. Second, DLLens extracts the path constraints from the
implementations of $f$ and $f^{\prime}$. Finally, DLLens generates the input
arguments for $f$ and $f^{\prime}$ following the extracted path constraints
and applies differential testing to check the output consistency between $f$
and $f^{\prime}$ on these input arguments for bug detection.

###  III-A Counterpart Synthesis

Criterion of counterparts. A counterpart of an API $f$ is a function
$f^{\prime}$ built on top of other DL library APIs simulating the intended
computation of $f$.111We are interested in only those counterparts that have a
different implementation from $f$, since counterparts such as aliases with the
same implementation as $f$ are not useful for bug detection using differential
testing. In practice, it is non-trivial to find a counterpart of an API. For
example, the state-of-the-art technique [18] can only find counterparts for a
limited number of TensorFlow APIs (i.e., 304/3,316) relying on the rules
specified by developers. In general, the chance that a counterpart of interest
is coincidentally available as another library API is not high. Moreover, due
to the large input space of $f$, it is hard to enumerate all input values for
the output consistency check between the API $f$ and $f^{\prime}$.

To address the above challenges, DLLens adopts a two-phase approach (as shown
in Figure 4 Step I). For each library API under test $f$, DLLens first
leverages an LLM to generate a non-empty set of valid inputs $\mathcal{X}$,
for which $f$ successfully performs computation and produces outputs. DLLens
then provides the LLM with the information of $f$ and $X$ to synthesize
counterpart candidates. our insight is that an API’s computation may be
alternatively simulated by a function (e.g., the code snippet in Listing 3)
composed using other DL library APIs. The set of valid inputs $\mathcal{X}$
will then be used to validate the synthesized counterpart candidates. Finally,
we use Equation 1 to check if a counterpart candidate simulates the intended
computation of API $f$.

$\forall x\in\mathcal{X},|f(x)-f^{\prime}(x)|\leq\epsilon$ (1)

where $\epsilon$ is a small positive value (e.g., 0.1). In the following, let
us visit the two phases in more detail.

####  III-A1 Valid Input Generation

DLLens adapts the direct prompting strategy proposed by TitanFuzz [21] and
asks the LLM to generate valid inputs for each DL library API. More
specifically, the prompt includes the target library (e.g., TensorFlow) and
the API signature, it further describes the task of valid input generation
using a step-by-step instruction (see Listing 5). However, such direct
prompting may not generate valid inputs satisfying the input constraint
required by the DL library API. For instance, the LLM (i.e., gpt-turbo-3.5)
fails to generate input argument image that satisfies the required shape
constraint of tf.image.pad_to_bounding, resulting in no valid inputs found for
this API.

To improve the performance of direct prompting in generating valid API inputs,
DLLens dynamically executes each generated input with the API and back prompts
the LLM with the error message for repair. The intuition is that DL libraries
commonly raise meaningful invalid argument error messages [18] when receiving
an invalid input, which can provide useful repair guidance. Listing 5
exemplifies the input generation workflow. While the LLM initially generates
an invalid input, it successfully generates a valid one after back prompting
using the error message.

Since APIs across DL libraries are built to implement the different parts of
the same set of published deep learning computations, the search space of APIs
relevant to the computation intended by $f$ is limited. So, a few valid inputs
can effectively identify the appropriate choice of APIs for the counterpart
synthesis. For a given API, DLLens obtains multiple (i.e., three) inputs from
the LLM and guides the LLM to repair the invalid ones. If no valid inputs can
be obtained for an API, DLLens skips the testing of this API because no
counterpart can be reliably synthesized under equation 1.

⬇

1

2|\underline{\textbf{Prompt}}|:

3Task 1: Import TensorFlow 2.10.0

4Task 2: Generate valid parameter ...

5Task 3: Call the function: tf.image.pad_to_bounding_box(image, ...)

6|\underline{\textbf{LLM}}:|

7image=tf.constant([[1,2],[3,4]],dtype=tf.float32)
|\textbf{\textcolor{bgRed}{(this input is invalid!)}}|

8|\underline{\textbf{Follow-up Prompt}}|:

9[Error Message]

10’image’ (shape(2,2)) must have 3 or 4 dimensions.

11|\underline{\textbf{LLM}}|: Here is the repaired input

12image=tf.constant([[[1,2],[3,4]]],dtype=tf.float32)
|\textbf{\textcolor{bgGreen}{(the repaired input is valid.)}}|

Figure 5: Example Prompt for Valid Input Generation.

####  III-A2 Counterpart Synthesis and Validation

With the API under test $f$ and the input set $\mathbf{X}$, in this stage,
DLLens aims to synthesize a counterpart. One challenge in the counterpart
synthesis lies in the alignment of the signatures and usage contexts between
an API and the counterparts thus synthesized. For instance, the same
functionality is provided by tf.keras.metrics.mean_squared_error and
torch.nn.MSELoss APIs, but they have different signatures and usage contexts.
Specifically, the former supports direct function calls, whereas the latter
requires an object construction before processing the input. As a result, they
cannot accept the same input. To address this alignment problem, DLLens
includes valid inputs into the prompt that guides the LLM to synthesize
functions with the same signature and usage context as the given API, as shown
in Figure 6.

Listing 6 illustrates the synthesis of a counterpart for API tf.scatter_nd. By
providing valid inputs and the API’s signature, DLLens instructs the LLM to
search for an equivalent function. DLLens explicitly asks the LLM to
synthesize a function using the APIs from a different DL library (e.g.,
PyTorch). The rationale behind this is to prevent the LLM from searching for
the given API’s caller, callee, or alias within the same DL library. If a
synthesized function is invalid, i.e., unable to process valid inputs, DLLens
feeds the error message to LLM for repair. Like the input generation workflow,
DLLens synthesizes multiple counterpart candidates for a given API. A
counterpart is successfully synthesized if it satisfies the output consistency
criterion mentioned above.

⬇

1

2|\underline{\textbf{Prompt}}|:

3Given these inputs, generate a function using PyTorch APIs that can perform
the same functionality as the given TensorFlow API.

4|\textit{[Input Set]}|

5indices = tf.constant([[0], [2]])...

6|\textit{[TensorFlow API]}|

7tf.scatter_nd(indices,updates,shape,name=None)

8|\textit{[Function using PyTorch APIs]}|

9def pytorch_call(indices,updates,shape):

10|\underline{\textbf{LLM}}:|

11def pytorch_call(indices,updates,shape):

12...

13tensor_shape=torch.Size(shape)

14output_tensor=torch.zeros(tensor_shape)

15|\textbf{\textcolor{bgRed}{output\_tensor.scatter\_(0,
tensor\_indices,tensor\_updates)}}| |\textbf{\textcolor{bgRed}{(this line is
buggy!)}}|

16return output_tensor

17|\underline{\textbf{Follow-up Prompt}}|:

18[Error Message]

19|\underline{Index}| tensor must have the same number of dimentions

20as |\underline{self}| tensor...

21|\underline{\textbf{LLM}}|:

22def pytorch_call(...):

23...

24|\textbf{\textcolor{bgGreen}{output\_tensor.scatter\_(dim=0,
index=tensor\_indices.squeeze(), src=tensor\_updates)}}|
|\textbf{\textcolor{bgGreen}{(the repaired line is correct)}}|

25return output_tensor

Figure 6: Example Prompt for Counterpart Synthesis.

###  III-B Path Constraint Extraction

For a DL library API and its counterpart synthesized in Sec III-A, DLLens
extracts path constraints from the control flow graphs of their
implementations. Specifically, DLLens extracts a path constraint for each
execution path in the control flow graph. Each path constraint specifies the
input conditions for the execution of the concerned path [33], where each
input condition is a condition on input arguments of the DL library API and
its counterpart. For each loop in the concerned path, DLLens follows the
existing work [31] to unroll the loop once and ignore the loop condition.

Essentially, each extracted path constraint is a conjunction of the input
conditions extracted from the sanity check statements and the control flow
statements (e.g., if statement) reached by the concerned execution path.
Sanity check statements include built-in assertion statements and validation
check function calls (e.g., the TORCH_CHECK function call) written by DL
library developers. Following the existing work [31], DLLens considers two
specific function calls, i.e., OP_REQUIRES in TensorFlow and TORCH_CHECK in
PyTorch. In specific, for each validation check function call, DLLens targets
its predicate expression, i.e., the second argument of the OP_REQUIRES and the
first argument of the TORCH_CHECK). For each execution path, DLLens adds the
True condition of the predicate expression in each reached validation check
function call and the True or False condition assumed by each reached control
flow statements to its path constraint.

The extracted path constraints from DL libraries usually involve operations of
tensors, which are complex data structures not supported by SMT solvers such
as Z3 [34]. DLLens adopts a constraint model (see Table I) proposed by
existing study [31], which observed that most conditions for a tensor input
argument only constrain specific properties (e.g., shape) while the value of
the tensor is not constrained in most cases [31, 30]. Specifically, DLLens
follows existing work to reduce a few tensor operations, which are unknown by
existing SMT solvers. For instance, given a constraint “len(¡tensor¿.shape) ¡
1 is True” with an unknown operation len() that calculates the length of a
tensor’s shape. Since the length of a tensor’s shape, by definition, is the
number of dimensions (ndims) held by the tensor; therefore, DLLens will reduce
it to “¡tensor¿.ndims¡1 is True”.

TABLE I: Tensor Property Model

Property | Description | Variable Type
---|---|---
ndims | number of dimensions | integer
shape | number of elements in each dimension | list of integers
dtype | data type (e.g., int32, float64) | string
num_element | number of elements | integer

However, solving path constraints can be challenging. Due to the large size of
DL libraries and their upstream libraries, it would be too expensive to
perform the static analysis on the entire code base [31]. Therefore, DLLens
follows existing work [31] to perform the static analysis on code within the
same module. However, we notice that input conditions in path constraints
usually involve functions written in external libraries (e.g., Eigen, a
popular C++ library for linear algebra) or external modules (e.g.,
TensorUtils). These external functions complicate the solving of path
constraints since their behaviors are unknown to existing solvers. Taking the
implementation in Figure 8 as an example. The input condition extracted from
the If statement (i.e., Stmt 4) includes an external function named
TensorShapeUtils::IsVector; thus, it is invalid (i.e., contains external
functions unknown to existing solvers). Consequently, path constraints,
including this input condition, cannot be solved. To handle external
functions, existing works [31, 18] either simply skipped them (i.e., do not
include invalid input conditions with these external functions into the path
constraint) or manually modeled the behavior only on a very limited set of
these functions. Both ways lead to incomplete path constraints, as is reported
by the existing work [31].

DLLens leverages an LLM to handle this challenge. The insight is that LLMs
have learned domain-specific knowledge of DL libraries and their upstream
libraries. Given a control flow statement or sanity check statement consisting
of external functions, LLMs are capable of “modeling” the behavior of these
external functions and outputting a valid input condition solvable by existing
solvers. A straightforward way is to simply provide the whole source code of
an API and its counterpart to an LLM and let it extract valid input
conditions. However, this is ineffective since the source code of each library
API and its counterpart usually have complex control flow graphs, and we find
that LLMs are ineffective in handling such complicated contexts [35, 36, 37].

Therefore, instead of giving the LLM the whole source code. DLLens handles
each control flow statement and sanity check statement individually by only
providing the execution trace from the input argument to this statement. Based
on this execution trace, DLLens further asks the LLM to extract a valid input
condition. Taking Figure 8 as an example, to extract the input condition of
argument k from a If statement mycode Stmt 2 which contains an external
function TensorShapeUtils::IsVector, DLLens will first build the control flow
graph and extract the execution trace from k to Stmt 2. Then DLLens constructs
a prompt (see Figure 7) formatted based on the following three parts:

  1. 1.

the execution trace from the input argument k to the concerned statement Stmt
2.

  2. 2.

the argument type of input argument k.

  3. 3.

the ‘property’ (see Table I) of ‘k’ if ‘k’ is a tensor. Otherwise, this part
will not be provided to LLMs.

As a result, DLLens successfully guides the LLM to extract the valid input
condition (i.e., k.ndims==1) from Stmt 2. Specifically, DLLens constructs the
execution trace from k to Stmt 2 by including the Stmt 1 since k’s value is
assigned to diag_index, then used in Stmt 2. DLLens further uses an SMT solver
to checks whether the returned input condition from the LLM is valid. For
valid input conditions returned by LLM, DLLens will add them to the path
constraint.

⬇

1

2|\underline{\textbf{Prompt}}|:

3Analyze the following execution path, summarize the

4NECESSARY constraint on the ‘properties‘ of input

5arguments ‘k‘ to satisfy the condition at the end of

6the path.

7|\textbf{[Execution Path]}|

8diag_index = k

9condition: TensorShapeUtils::IsVector(diag_index.shape())

10|\textbf{[Argument Type]}|

11{’k’: ’tensor’}

12|\textbf{[‘property‘ for ‘k‘]}|

13.ndims: int, number of dimensions of tensor

14...

15The constraint should only consider symbols |\underline{‘k‘}|, ...

16...

17|\underline{\textbf{LLM}}:|

18len(k.shape) == 1

Figure 7: Example prompt of DLLens to extract input conditions from control
flow statements with external functions. Figure 8: A running example of DLLens
that uses the LLM to extract valid input conditions from a control flow
statement with external functions.

The detailed algorithm for path constraint extraction is shown in Algorithm 1.
Given the set of input arguments and an execution path extracted from the
implementation of a DL library API or its counterpart, DLLens traverses all
the statements along this path and extracts conditions on input arguments from
sanity check and control flow statements (lines 3-6). Note that the condition
value of some statements may not be controlled by input arguments, e.g., a
condition value may be controlled by some global variables. In this case,
conditions from these statements will not be added to the path constraint of
the execution path (lines 6-7) since changing the value of input arguments
will not influence these condition values. For the remaining statements,
DLLens first reduces tensor operators (line 8) and checks if there are
external functions whose source code is not considered during the static
analysis (line 9). If any external function is found, DLLens formats the
prompt based on the template of Figure 7 to let the LLM extract a valid input
condition from the concerned statement (line 10). The input condition returned
by the LLM will be added to the path constraint if it is solvable (i.e., it
can be solved by an SMT solver) (lines 11-12). Otherwise, the input condition
of the concerned statement will not be added to the path constraint. For each
DL library API, DLLens applies Alg 1 to each execution path inside
implementations of this API and its counterpart. Finally, the total path
constraints of each API are formed by combining the path constraints extracted
from its own and its counterpart’s implementations.

Input: $p$, the path whose constrain to be extracted;

Input: $I$, the input arguments of path $p$

1

Output: $PC$, the extracted path Constraint

2 def  _PathConstraintExtraction ($p$, $I$)_:

3 $PC$ $\leftarrow\varnothing$ for  _stmt in $p$_ do

4 if  _stmt.type is SanityCheckStmt or ControlFlowStmt_ then

5 cond $\leftarrow$ $\textnormal{{Extract}}(\textnormal{{stmt}})$

6 related_input $\leftarrow$
$\textnormal{{ExtractInput}}(\textnormal{{cond}})$

7 if  _related_input $\cap$ $I$ is $\varnothing$_ then

8 continue

9 cond’ $\leftarrow$ $\textnormal{{Reduce}}(\textnormal{{cond}})$

10 if  _not $\textnormal{{External}}(\textnormal{{cond'}})$_ then

11 cond’ $\leftarrow$ $\textnormal{{LLM}}(\textnormal{{cond}})$

12 if  _$\textnormal{{Solvable}}(\textnormal{{cond'}})$_ then

13 $PC\leftarrow PC\cup\textnormal{{cond'}}$

14

15

16 return $\mathbf{PC}$;

17

Algorithm 1 Path Constraint Extraction

###  III-C Test Input Generation and Bug Detection

For each API, DLLens uses its total path constraint set for test input
generation. For each path constraint inside this total path constraint set,
DLLens iteratively generates input arguments to fulfill such constraint. To
avoid duplication during the iterative generation, when generating input
arguments for each path constraint, we follow existing works to add an
additional input constraint, preventing the constraint solver from generating
duplicate input tensors whose property values have been covered in the
previous iteration, back to the path constraint. Besides the path constraint
extracted in Sec.III-B, following existing techniques [18], DLLens also
incorporates natural constraints and error-feedback constraints. The former
refers to pre-defined constraints based on the input argument type (see Table
II. For instance, we require the tensor’s data type to be a string among a
list of options such as float32 and double. The latter refers to the
constraints extracted from error messages in previous iterations. For
instance, the error message formatted in Dimension out of range (expected to
be in range of [#1, #2], but got #3) indicates the ‘ndims’ of a tensor input
argument should be within the range of [#1, #2] while the actual ndims of this
tensor input argument is #3. For this error message, we use a regular
expression to extract the #1, #2, and #3, and further check if there is a
tensor input argument whose ndims is #3. If such an input argument is found,
we add a constraint that requires its ndims to be within the range of [#1, #2]
to the original path constraint.

Finally, a fuzzing driver is used to instantiate the concrete value of
generated input tensors for testing. Inspired by existing work [38]
demonstrating that special values (i.e., Inf and NaN) can expose output
inconsistency in scientific libraries, we also randomly generate special
values for input arguments.

Test Oracles. DLLens uses two test oracles to detect both crash bugs and
functional bugs. Crash bugs refer to system crashes, including aborts,
segmentation faults, floating point exception raised, and
INTERNAL_ASSERT_FAILED [32, 18]. If DLLens finds any such issues in testing,
DLLens labels them as crash bugs for manual investigation. For functional
bugs, DLLens focuses on the inconsistency between a DL library API and its
counterpart. In particular, DLLens captures two types of inconsistency: 1)
inconsistent computation results, _i.e._ , output inconsistency larger
$\epsilon$, where $\epsilon$ is a pre-defined threshold. 2) inconsistent
execution status between the DL library API and its counterpart.

##  IV Evaluation

We evaluated DLLens from three perspectives: effectiveness of counterpart
synthesis, path constraint extraction, and bug detection. We studied four
research questions (RQs):

  * •

RQ1: How effective is DLLens in finding counterparts compared with existing
techniques? We demonstrated DLLens’s effectiveness by comparing with existing
techniques on the total number of APIs found with counterpart.

  * •

RQ2: How effective is DLLens in path constraint extraction? We measured the
number of solvable path constraints extracted by DLLens.

  * •

RQ3: Can DLLens detect real bugs? We presented DLLens’s bug-revealing
capability to demonstrate its usefulness. We evaluated DLLens on two popular
DL libraries: TensorFlow and PyTorch.

  * •

RQ4: Are DLLens more effective than existing works? We compared DLLens with
existing techniques in terms of the branch coverage and the number of bugs
detected on 200 randomly sampled DL library APIs.

###  IV-A Implementations and Experiment Setup

Implementations We used gpt-turbo-3.5 for counterpart synthesis and path
constraint extraction. We set the temperature to 1, the default value
specified by the documentation [39]. For path constraint extraction, we used
the Python standard AST package [40] and tree-sitter [41] to parse an
implementation into an AST tree. We further used z3py [34] to solve the
extracted path constraint. For the natural constraint used by test input
generation, we manually crafted the valid input space based on the argument
type (see Table II). The experiments were conducted using a 32-core server
with a 3090Ti GPU.

TABLE II: Natural Constraints Used by DLLens

Argument Type | Argument Properties | Value Space
---|---|---
Tensor | ndims | [0, 5]
shape[i] | [0, 5]
dtype |  | [‘uint(8-64)’, ‘float(16-64)’,
---
‘complex(64-128)’, ‘bool’, ‘string’]
num_element | [0, $\infty$]
Bool | value | {True, False}
Integer | value | [-100, 100]
Float | value | [-100, 100]
String | value | Any

DL libraries under test. We evaluated DLLens on PyTorch (v2.1.0) [7] and
TensorFlow (v2.10.0) [8]. To collect API under test, we do not include the DL
library API in our experiment if 1) it has an incomplete signature; 2) it
belongs to special packages (e.g., torch.utils and tf.raw_ops.Experimental).
In total, we collected 1,864 TensorFlow APIs and 1,184 PyTorch APIs, which
take up 56% and 74.3% of the total APIs collected by existing works [21, 32].

API source code collection. To collect the source code of DL library APIs, we
used PyCG [42] and Joern [43]/tree-sitter [41] to construct call graphs for
Python and C++, respectively. Note that TensorFlow and PyTorch have their
front-end APIs implemented in Python, and their core logic functions are in
C++. To establish the call relationship between the Python and C++ code, we
observed that TensorFlow employs the Python interface _execute.execute to
access C++ operators, while PyTorch utilizes native_functions.yaml to link the
Python API with its C++ implementations. Leveraging these characteristics, we
developed a program analysis tool to bind function calls between Python and
C++.

Starting from the entry point of each API, we conducted a breadth-first search
in the constructed call graph, collecting the source code of all functions
within a depth of five levels.

Baselines. Overall, we compared DLLens with four existing techniques,
TitanFuzz [21], TensorScope [18], FreeFuzz [20], and DocTer [30]. Since
neither executables nor source codes of ACETest [31], and FuzzGPT [32] are
available, we did not include them in evaluation.222Although the source code
of TensorScope is available, we tried our best but failed to execute their
input generation and constraint extraction modules, thus we do not include
TensorScope in RQ2 and RQ4. For RQ1 (evaluation of counterpart synthesis), we
compared DLLens with TensorScope [18], which is the state-of-the-art work that
can find counterparts across DL libraries for DL library APIs. In RQ2
(evaluation of path constraint extraction), we compared DLLens with DocTer
[30], which mines input constraints from DL library API documentation. In RQ4
(evaluation on code coverage and bug detection), we compared DLLens with two
state-of-the-art DL library testing techniques: FreeFuzz [20] and TitanFuzz
[21] on a randomly sampled API set.

Functional Bug Detection. For output inconsistencies checking between the DL
library API and its’ counterpart, we set the absolute threshold $\epsilon$ to
0.1. For execution status inconsistency, we first removed clear syntax errors
(e.g., SyntaxError) and invalid argument messages such as TypeError and
RuntimeError. For the remaining status inconsistencies, we manually checked
the documentation of related DL library APIs to determine whether it was a
real bug.

###  IV-B RQ1: Effectiveness of Counterpart Synthesis

We compared DLLens with TensorScope [18], the state-of-the-art techniques
finding counterparts across different DL libraries. We evaluated both tools on
the API counterpart coverage, i.e., the number of TensorFlow and PyTorch APIs
that these tools could find counterparts for. To be more specific, we only
evaluated whether a valid DL library API’s counterpart is available while did
not consider the number of different counterparts found for one single API.
Therefore, if a DL library API has multiple counterparts, we consider all of
them as one when evaluating the API counterpart coverage.

For TensorScope, we followed its methodology and collected counterparts of
TensorFlow APIs and PyTorch APIs via parsing the developer-constructed rules
in model conversion libraries. Since these rules are manually written by
library developers, we consider all counterparts extracted by TensorScope to
be valid (i.e., satisfies the Equation 1). For DLLens, we applied it on each
DL library API to synthesize this API’s valid counterpart that satisfies
Equation 1 based on a non-empty valid input set (with at most three valid
inputs). Due to the random nature of the LLM, the synthesized counterpart from
the LLM may not always be valid. To make better use of the LLM, we applied our
counterpart synthesis workflow for five rounds. In each round, we asked the
LLM to synthesize counterparts for APIs whose valid counterparts had not been
correctly synthesized in previous rounds. Table III shows the number of valid
API counterparts synthesized by DLLens for in each round. We noticed that most
counterparts could be found in the first round, and the following rounds could
also find a non-negligible number of API counterparts.

Figure 9 shows the comparison between DLLens and TensorScope on the API
counterpart coverage. A higher API counterpart coverage indicates the tool can
facilitate differential testing to detect functional bugs for more APIs.
Overall, DLLens can synthesize valid counterparts for 1,481 APIs, including
739 from TensorFlow and 742 from PyTorch, which stand for 41.47% and 62.67% of
the total APIs evaluated, respectively. Compared with TensorScope, DLLens can
collect 143.1% (739 v.s. 304) more valid counterparts for TensorFlow APIs and
65.3% (742 v.s. 449) more for PyTorch APIs. We also noticed that TensorScope
found 68 and 159 counterparts for TensorFlow and PyTorch APIs beyond our
recorded APIs. This is because some APIs recorded in developers’ conversion
rules are related to special packages such as torch.utils or special usages
such as torch.float64 for representing the data type. These APIs are not
targeted by DLLens since we try to focus on APIs developed for DL algorithms,
which may be commonly used by DL library users.

TABLE III: The Number of Counterparts Found by DLLens

Library Name | Round 1 | Round 2 | Round 3 | Round 4 | Round 5 | Total
---|---|---|---|---|---|---
TensorFlow | 619 | 53 | 30 | 20 | 17 | 739
PyTorch | 624 | 61 | 30 | 15 | 12 | 742

Figure 9: Comparison on API Counterpart Coverage

###  IV-C RQ2: Effectiveness of Path Constraint Extraction

To measure the effectiveness of DLLens in path constraint extraction, we first
evaluated the number of path constraints that can be extracted by DLLens.
Specifically, we applied DLLens to extract path constraints for all APIs
(i.e., 739 TensorFlow APIs and 742 PyTorch APIs) collected in RQ1. For each
API, its path constraints are formed by combining the path constraints
extracted from both its own implementation and the implementation of its
counterpart. Table IV demonstrates the performance of DLLens’s path constraint
(i.e., P.C.) extraction capability by showing the number of path constraints
extracted per API. On average, DLLens extracts 21.50 path constraints for each
TensorFlow API and 41.87 for each PyTorch API. Each path constraint in
TensorFlow and PyTorch contains 3.35 and 3.84 input conditions, respectively.
We also noticed that incorporating our LLM-aided static analysis method led to
a significant increase in the extraction of path constraints compared to our
static analysis method that doesn’t use the LLM (referred to as ‘DLLens w/o
LLM’).

TABLE IV: Performance on Path Constraint (P.C.) Extraction  Tool | # P.C./API | # Cons./P.C.
---|---|---
DLLens (TensorFlow) | 21.50$\pm$58.76 | 3.35$\pm$2.53
DLLens w/o LLM (TensorFlow) | 2.12$\pm$5.53 | 1.83$\pm$1.04
DLLens (PyTorch) | 41.87$\pm$91.54 | 3.84$\pm$3.02
DLLens w/o LLM (PyTorch) | 3.15$\pm$6.52 | 1.95$\pm$1.10

Comparison with DocTer. We further compared DLLens with DocTer [30] in terms
of the constraint extraction capability. Although DocTer utilizes documents to
extract API constraints and does not extract path constraints, we employed
another metric (i.e., property constraints) utilized by existing works [30,
18] including DocTer, to evaluate the constraint extraction performance.
Property constraint refers to the total number of input properties considered
by extracted constraints. For each API input argument, we followed existing
work and considered all valid options for one property as one property
constraint [30]. Among all APIs for which DLLens synthesized counterparts in
RQ1, we found that DocTer could extract constraints for 271 TensorFlow APIs
and 160 PyTorch APIs. For a fair comparison, we further compared the property
constraint extraction performance with DocTer on these APIs.

TABLE V: Comparison on Property Constraint Extraction

Tool | TensorFlow | PyTorch
---|---|---
DLLens | 6.21$\pm$4.11 | 6.64$\pm$4.25
DocTer | 4.24$\pm$3.38 | 5.24$\pm$4.21

Table V shows the average number of property constraints extracted for each
API. DLLens extracts more property constraints than DocTer, _i.e._ , 6.21 v.s.
4.24 for TensorFlow and 6.64 v.s., 5.24 for PyTorch. We further analyzed the
constraint extraction performance for each property, as depicted in Figure 10.
Our findings indicate that DLLens outperforms DocTer in extracting constraints
related to shape, value, and structure (‘STR.’ in Figure 10), while it
extracts less data type constraints (’DType’ in Figure 10) compared to DocTer.

Taking TensorFlow APIs as an example, on average, DLLens is more effective in
extracting shape (2.03 vs. 0.82), value (0.53 vs. 0.14), and structure (2.37
vs. 1.66) constraints per API. As for data type constraints, averagely DLLens
extracts 1.28 constraints per API, whereas DocTer extracts 1.82 per API. This
difference could be due to the common practice of documenting constraints
related to data types, making DocTer more effective in this aspect. In
contrast, constraints on shape, value, and structure are less frequently
documented, limiting DocTer’s performance in these properties.

Figure 10: Comparison on the Number of Constraints Extracted for Each Property

###  IV-D RQ3: Effectiveness of Bug Detection

We use DLLens to detect bugs in TensorFlow v2.10 and PyTorch v2.1. In total,
DLLens detected 56bugs in TensorFlow and PyTorch, including 15bugs already
fixed or confirmed by developers and 41new ones. After we reported those newly
detected bugs, developers confirmed 39, including 19 fixed. The remaining 2
bugs are pending confirmation.

Table VI demonstrates the newly detected bugs in TensorFlow and PyTorch,
categorized by their bug symptoms. The predominant type of bug detected by
DLLens is the incorrect result bug (’Incorrect’ in Table VI), constituting
43.9% (18 out of 41) of the total reported bugs. Among these, 16 were
confirmed by developers, and 9 have been successfully addressed with fixes. An
example is Listing 1, which is caused by an internal overflow during the
computation, resulting in an incorrect output of API
tf.math.is_non_decreasing. Detection of these incorrect result bugs requires
the test oracle, which can be effectively addressed by our synthesized
counterparts. Following the ‘Incorrect’ bugs, the second most prevalent
category is the bug that API incorrectly rejected valid inputs (’Inc.Rej.’ in
Table VI), accounting for 12 out of the total 41 reported bugs. These bugs
were also detected when execution inconsistency occurs between a DL library
API and its counterpart, which also demonstrates the usefulness of synthesized
counterparts. We notice that DLLens has also detected many crash bugs (Crash
in Table VI), which take up to 10 out of 41. Although a counterpart is not
required to detect these bugs, we found that path constraints extracted by
DLLens could guide generating test inputs triggering these bugs. Taking the
bug in Listing IV-D as an example, reaching the buggy code requires passing
two sanity checks which assert the ndims of input argument indices and updates
to be no less than 1 [44]. Extracting path constraints, including these two
sanity checks, can guide generating input reaching the buggy code, increasing
the probability of triggering this bug.

—Bug Triggering Input—: indices = tf.constant([[[1]]]) updates =
tf.constant([16]) … —Buggy API—: tf.tensor_scatter_nd_update —Buggy Code—: for
(int i = 0; i ¡ outer_dims; ++i) … —Two Validation Checks Before Reaching The
Buggy Code—: OP_REQUIRES(.., —indices.shape().dims() ¿= 1—,..);
OP_REQUIRES(.., —updates.shape().dims() ¿= 1—,..);

Figure 11: A Crash Bug and Two Sanity Checks before Reaching the Buggy Line

###  IV-E RQ4: Comparison On Bug Detection and Code Coverage

We compared DLLens with two state-of-the-art DL library testing techniques:
FreeFuzz [20] and TitanFuzz [21]. Since neither executables nor source codes
of FuzzGPT [32], ACETest [31], and TensorScope [18] are available, we did not
include them as baselines. 333Although the source code of TensorScope is
available, we tried our best but failed to execute their tool. Specifically,
we randomly sampled 100 TensorFlow APIs and 100 PyTorch APIs from the total
739 TensorFlow APIs and 742 PyTorch APIs collected in RQ1. For each sampled
API, we used each testing technique to generate 300 test inputs for testing.
We considered the branch coverages of these APIs and the bug detection
performances as metrics to evaluate their overall performance. Pycoverage [45]
and lcov [46] are used to collect branch coverages for Python and C++ code,
respectively. When measuring the code coverage, we excluded additional
coverages added by APIs that are not under test for a fair comparison. In bug
detection, we manually analyzed reported bugs to filter out false positives.

Table VII shows the branch coverage and bug detection results. In specific,
DLLens outperforms the state-of-the-art (i.e., TitanFuzz) by covering 3.3%
more branches (7,896 v.s. 7,647) in TensorFlow and 13.8% more (6,433 v.s.
5,652) in PyTorch. We also notice that DLLens can detect more bugs, i.e.,
DLLens can detect at least 2.5 times as many bugs as baselines. Moreover, we
found that most (9 out of 10) bugs detected by DLLens are either incorrect
result bugs or incorrectly rejected bugs; detecting these bugs demonstrated
the usefulness of counterpart synthesized by DLLens.

TABLE VI: Effectiveness of Bug Detection

Library Name | Inc. Rej. | Incorrect | Imp.Err.Msg | Crash | Subtotal
---|---|---|---|---|---
TensorFlow | 11 (4/9) | 15 (8/15) | 1 (1/1) | 3 (1/3) | 30 (14/28)
PyTorch | 1 (0/1) | 3 (1/3) | 0 (0/0) | 7 (4/7) | 11 (5/11)

  * 1

Numbers of confirmed and fixed bugs are parenthesized. The number of confirmed
bugs is the denominator in parentheses; the number of fixed bug is the
numerator in parentheses.

TABLE VII: Code Coverage and Bug Detection Baseline | TensorFlow | PyTorch | Total Bugs
---|---|---|---
Branch | Bug | Branch | Bug
DLLens | 7,896 | 7 | 6,433 | 3 | 10
TitanFuzz | 7,647 | 2 | 5,652 | 1 | 3
FreeFuzz | 6,822 | 4 | 5,382 | 0 | 4

##  V Discussions

False Positives of Bug Detection. We summarized two types of false positives:
First, some false positives are introduced by invalid counterparts synthesized
by DLLens. This is because we only use a non-empty valid input set with at
most three valid inputs to validate the synthesized counterpart and such a
limited size of valid input may introduce some false positives, i.e., some
synthesized counterparts passing the validation process may only be valid
under specific input conditions (e.g., some counterparts are valid when the
input is a vector while invalid when input is a matrix). As a result, the
output inconsistencies w.r.t these inputs between APIs under test and their
counterparts may be false positives. To understand the influence of these
false positives, we manually analyze all inconsistencies reported by DLLens in
RQ4, which includes 200 APIs and their counterparts. According to our
analysis, only 27 false positives inconsistencies reported by DLLens were
caused by invalid counterparts, which cost ~1.5 hours to filter out.
Considering the effectiveness of DLLens in detecting real bugs, we believe
such overhead introduced by these invalid counterparts is affordable.

Another type of false positive is introduced by inconsistent behavior between
libraries when handling special values such as NaN [47]. Since inconsistencies
caused by these special values are not many (e.g., less than 10 in RQ4), we
manually check the documentation of the API under test with all APIs in its
counterpart and filter out false positives.

Threat to Validity. In RQ2 and RQ4, we compared DLLens with state-of-the-art
techniques on all DL library APIs that DLLens could identify counterparts for.
It should be noted that these APIs constitute only a subset of the total DL
library APIs, potentially limiting the comprehensiveness of our performance
evaluation. To address this potential limitation, we conducted a measurement
of the number of APIs considered in RQ2 and RQ4, which accounted for nearly
half of the total DL library APIs. Based on this analysis, we make the
assumption that the APIs covered in our study adequately represent the
performance of DL library testing techniques.

##  VI Conclusion

In this paper, we propose a novel technique named DLLens to test DL libraries
DLLens facilitates differential testing by using a novel counterpart synthesis
method and generates diverse test inputs to explore execution paths via an
effective path constraint extraction method. DLLens can synthesize
counterparts for 41.47% TensorFlow APIs and 62.67% PyTorch APIs, which double
the result of the state-of-the-art approaches. Benefiting from the synthesized
counterparts and effective path constraint extraction, DLLens can outperform
state-of-the-art approaches in terms of constraint extraction, bug detection,
and branch coverage. In total, DLLens detects 56including 41new bugs. So far,
39of these new bugs have been confirmed and 19 of them are fixed by
developers.

## References

  * [1] R. Das, A. Gadre, S. Zhang, S. Kumar, and J. M. F. Moura, “A deep learning approach to iot authentication,” in _2018 IEEE International Conference on Communications (ICC)_. Kansas City, MO, USA: IEEE, 2018, pp. 1–6.
  * [2] A. Ferdowsi and W. Saad, “Deep learning for signal authentication and security in massive internet-of-things systems,” _IEEE Transactions on Communications_ , vol. 67, pp. 1371–1387, 2019.
  * [3] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. Van Der Laak, B. Van Ginneken, and C. I. Sánchez, “A survey on deep learning in medical image analysis,” _Medical image analysis_ , vol. 42, pp. 60–88, 2017.
  * [4] C. Chen, A. Seff, A. Kornhauser, and J. Xiao, “Deepdriving: Learning affordance for direct perception in autonomous driving,” in _2015 IEEE International Conference on Computer Vision (ICCV)_. Santiago, Chile: IEEE, 2015, pp. 2722–2730.
  * [5] A. E. Sallab, M. Abdou, E. Perot, and S. Yogamani, “Deep reinforcement learning framework for autonomous driving,” 2017.
  * [6] S. Shalev-Shwartz, S. Shammah, and A. Shashua, “Safe, multi-agent, reinforcement learning for autonomous driving,” 2016.
  * [7] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, _PyTorch: An Imperative Style, High-Performance Deep Learning Library_. Red Hook, NY, USA: Curran Associates Inc., 2019.
  * [8] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, and X. Zheng, “Tensorflow: A system for large-scale machine learning,” in _Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation_ , ser. OSDI’16. USA: USENIX Association, 2016, p. 265–283.
  * [9] F. Tambon, A. Nikanjam, L. An, F. Khomh, and G. Antoniol, “Silent bugs in deep learning frameworks: An empirical study of keras and tensorflow,” 2021. [Online]. Available: https://arxiv.org/abs/2112.13314
  * [10] Y. Yang, T. He, Z. Xia, and Y. Feng, “A comprehensive empirical study on bug characteristics of deep learning frameworks,” _Information and Software Technology_ , vol. 151, p. 107004, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0950584922001306
  * [11] Y. Zhang, Y. Chen, S.-C. Cheung, Y. Xiong, and L. Zhang, “An empirical study on tensorflow program bugs,” in _Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis_ , ser. ISSTA 2018\. New York, NY, USA: Association for Computing Machinery, 2018, p. 129–140. [Online]. Available: https://doi.org/10.1145/3213846.3213866
  * [12] J. Chen, Y. Liang, Q. Shen, and J. Jiang, “Toward understanding deep learning framework bugs,” 2022. [Online]. Available: https://arxiv.org/abs/2203.04026
  * [13] Y. Xiong, M. Xu, T. Su, J. Sun, J. Wang, H. Wen, G. Pu, J. He, and Z. Su, “An empirical study of functional bugs in android apps,” in _Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis_ , ser. ISSTA 2023. New York, NY, USA: Association for Computing Machinery, 2023, p. 1319–1331. [Online]. Available: https://doi.org/10.1145/3597926.3598138
  * [14] H. V. Pham, T. Lutellier, W. Qi, and L. Tan, “Cradle: cross-backend validation to detect and localize bugs in deep learning libraries,” in _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_ , IEEE. Montreal, QC, Canada: IEEE, 2019, pp. 1027–1038.
  * [15] Z. Wang, M. Yan, J. Chen, S. Liu, and D. Zhang, “Deep learning library testing via effective model generation,” in _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , ser. ESEC/FSE 2020. New York, NY, USA: Association for Computing Machinery, 2020, p. 788–799. [Online]. Available: https://doi.org/10.1145/3368089.3409761
  * [16] J. Gu, X. Luo, Y. Zhou, and X. Wang, “Muffin: Testing deep learning libraries via neural architecture fuzzing,” in _Proceedings of the 44th International Conference on Software Engineering_ , ser. ICSE ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 1418–1430. [Online]. Available: https://doi.org/10.1145/3510003.3510092
  * [17] M. Li, J. Cao, Y. Tian, T. O. Li, M. Wen, and S.-C. Cheung, “Comet: Coverage-guided model generation for deep learning library testing,” _ACM Trans. Softw. Eng. Methodol._ , vol. 32, no. 5, jul 2023. [Online]. Available: https://doi.org/10.1145/3583566
  * [18] Z. Deng, G. Meng, K. Chen, T. Liu, L. Xiang, and C. Chen, “Differential testing of cross deep learning framework APIs: Revealing inconsistencies and vulnerabilities,” in _32nd USENIX Security Symposium (USENIX Security 23)_. Anaheim, CA: USENIX Association, Aug. 2023, pp. 7393–7410. [Online]. Available: https://www.usenix.org/conference/usenixsecurity23/presentation/deng-zizhuang
  * [19] T. Onnx, “tf2onnx - convert tensorflow, keras, tensorflow.js and tflite models to onnx.” https://github.com/onnx/tensorflow-onnx, Accessed: 2022.
  * [20] A. Wei, Y. Deng, C. Yang, and L. Zhang, “Free lunch for testing: Fuzzing deep-learning libraries from open source,” in _Proceedings of the 44th International Conference on Software Engineering_ , ser. ICSE ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 995–1007. [Online]. Available: https://doi.org/10.1145/3510003.3510041
  * [21] Y. Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang, “Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models,” in _Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis_ , ser. ISSTA 2023. New York, NY, USA: Association for Computing Machinery, 2023, p. 423–435. [Online]. Available: https://doi.org/10.1145/3597926.3598067
  * [22] J. Wang, T. Lutellier, S. Qian, H. V. Pham, and L. Tan, “Eagle: Creating equivalent graphs to test deep learning libraries,” in _2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE)_. Pittsburgh, PA, USA: IEEE, 2022, pp. 798–810.
  * [23] C. Yang, Y. Deng, J. Yao, Y. Tu, H. Li, and L. Zhang, “Fuzzing automatic differentiation in deep-learning libraries,” in _Proceedings of the 45th International Conference on Software Engineering_ , ser. ICSE ’23. Melbourne, Australia: IEEE Press, 2023, p. 1174–1186. [Online]. Available: https://doi.org/10.1109/ICSE48619.2023.00105
  * [24] Y. Deng, C. Yang, A. Wei, and L. Zhang, “Fuzzing deep-learning libraries via automated relational api inference,” in _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , ser. ESEC/FSE 2022. New York, NY, USA: Association for Computing Machinery, 2022, p. 44–56. [Online]. Available: https://doi.org/10.1145/3540250.3549085
  * [25] TensorFlow, “tf.math.is_non_decreasing outputs incorrect result when input is an uint tensor,” https://github.com/tensorflow/tensorflow/issues/62072, Accessed: March, 2024\.
  * [26] W. Luo, D. Chai, X. Run, J. Wang, C. Fang, and Z. Chen, “Graph-based fuzz testing for deep learning inference engines,” in _Proceedings of the 43rd International Conference on Software Engineering_ , ser. ICSE ’21. Madrid, Spain: IEEE Press, 2021, p. 288–299. [Online]. Available: https://doi.org/10.1109/ICSE43902.2021.00037
  * [27] TensorFlow, “slient overflow occurs in tf.range leading to incorrect result, here is a possible fix,” https://github.com/tensorflow/tensorflow/issues/64081, Accessed: March, 2024\.
  * [28] stack overflow, “How can i know if a list is decreasing? (python),” https://stackoverflow.com/questions/69576011/how-can-i-know-if-a-list-is-decreasing-python, Accessed: March, 2024.
  * [29] Q. Guo, X. Xie, Y. Li, X. Zhang, Y. Liu, X. Li, and C. Shen, “Audee: Automated testing for deep learning frameworks,” in _2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. Melbourne, VIC, Australia: IEEE, 2020, pp. 486–498.
  * [30] D. Xie, Y. Li, M. Kim, H. V. Pham, L. Tan, X. Zhang, and M. W. Godfrey, “Docter: Documentation-guided fuzzing for testing deep learning api functions,” in _Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis_ , ser. ISSTA 2022. New York, NY, USA: Association for Computing Machinery, 2022, p. 176–188. [Online]. Available: https://doi.org/10.1145/3533767.3534220
  * [31] J. Shi, Y. Xiao, Y. Li, Y. Li, D. Yu, C. Yu, H. Su, Y. Chen, and W. Huo, “Acetest: Automated constraint extraction for testing deep learning operators,” in _Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis_ , ser. ISSTA 2023. New York, NY, USA: Association for Computing Machinery, 2023, p. 690–702. [Online]. Available: https://doi.org/10.1145/3597926.3598088
  * [32] Y. Deng, C. S. Xia, C. Yang, S. D. Zhang, S. Yang, and L. Zhang, “Large language models are edge-case generators: Crafting unusual programs for fuzzing deep learning libraries,” in _Proceedings of the 46th IEEE/ACM International Conference on Software Engineering_ , ser. ICSE ’24. New York, NY, USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi.org/10.1145/3597503.3623343
  * [33] R. Baldoni, E. Coppa, D. C. D’elia, C. Demetrescu, and I. Finocchi, “A survey of symbolic execution techniques,” _ACM Comput. Surv._ , vol. 51, no. 3, may 2018. [Online]. Available: https://doi.org/10.1145/3182657
  * [34] L. de Moura and N. Bjørner, “Z3: An efficient smt solver,” in _Tools and Algorithms for the Construction and Analysis of Systems_ , C. R. Ramakrishnan and J. Rehof, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008, pp. 337–340.
  * [35] A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Survey and open problems,” 2023.
  * [36] W. Song, S. Oh, S. Mo, J. Kim, S. Yun, J.-W. Ha, and J. Shin, “Hierarchical context merging: Better long context understanding for pre-trained LLMs,” in _The Twelfth International Conference on Learning Representations_ , 2024\. [Online]. Available: https://openreview.net/forum?id=ulaUJFd96G
  * [37] I. Ozkaya, “Application of large language models to software engineering tasks: Opportunities, risks, and implications,” _IEEE Software_ , vol. 40, no. 3, pp. 4–8, 2023.
  * [38] J. Vanover, X. Deng, and C. Rubio-González, “Discovering discrepancies in numerical libraries,” in _Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis_ , ser. ISSTA 2020\. New York, NY, USA: Association for Computing Machinery, 2020, p. 488–501. [Online]. Available: https://doi.org/10.1145/3395363.3397380
  * [39] OpenAI, “Openai chat completion api reference,” https://platform.openai.com/docs/api-reference/chat/create, Accessed: March, 2024.
  * [40] Python, “Python ast package,” https://docs.python.org/3/library/ast.html, Accessed: March, 2024.
  * [41] M. Brunsfeld _et al._ , “Tree-sitter: A parser generator tool and an incremental parsing library,” 2024. [Online]. Available: https://github.com/tree-sitter/tree-sitter
  * [42] V. Salis, T. Sotiropoulos, P. Louridas, D. Spinellis, and D. Mitropoulos, “Pycg: Practical call graph generation in python,” in _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_ , 2021, pp. 1646–1657.
  * [43] Joern, “The bug hunter’s workbench,” https://joern.io, Accessed: March, 2024\.
  * [44] TensorFlow, “tf.tensor_scatter_nd_update lead to a program abortion when receiving a 3d indices,” https://github.com/tensorflow/tensorflow/issues/63575, Accessed: March, 2024\.
  * [45] “coverage.py,” 2021. [Online]. Available: https://coverage.readthedocs.io/en/coverage-5.5/
  * [46] P. Oberparleiter _et al._ , “lcov-a graphical gcov front-end,” _URL: https://linux. die. net/man/1/lcov, last checked on_ , pp. 08–03, 2019.
  * [47] Example, “Example of false positive,” https://github.com/pytorch/pytorch/issues/122426/.


# A Hybrid SFANC-FxNLMS Algorithm for Active Noise Control based on Deep
Learning

Zhengding Luo, Dongyuan Shi,
and Woon-Seng Gan (Corresponding author: Dongyuan Shi.)The authors are with
the School of Electrical and Electronic Engineering, Nanyang Technological
University, Singapore 639798, (e-mail: LUOZ0021@ e.ntu.edu.sg;
<EMAIL_ADDRESS>ewsgan@ntu.edu.sg). This paper has supplementary
downloadable material available at https://github.com/Luo-Zhengding/SFANC-
FxNLMS-ANC-Algorithm-based-on-Deep-Learning

###### Abstract

The selective fixed-filter active noise control (SFANC) method selecting the
best pre-trained control filters for various types of noise can achieve a fast
response time. However, it may lead to large steady-state errors due to
inaccurate filter selection and the lack of adaptability. In comparison, the
filtered-X normalized least-mean-square (FxNLMS) algorithm can obtain lower
steady-state errors through adaptive optimization. Nonetheless, its slow
convergence has a detrimental effect on dynamic noise attenuation. Therefore,
this paper proposes a hybrid SFANC-FxNLMS approach to overcome the adaptive
algorithm’s slow convergence and provide a better noise reduction level than
the SFANC method. A lightweight one-dimensional convolutional neural network
(1D CNN) is designed to automatically select the most suitable pre-trained
control filter for each frame of the primary noise. Meanwhile, the FxNLMS
algorithm continues to update the coefficients of the chosen pre-trained
control filter at the sampling rate. Owing to the effective combination of the
two algorithms, experimental results show that the hybrid SFANC-FxNLMS
algorithm can achieve a rapid response time, a low noise reduction error, and
a high degree of robustness.

###### Index Terms:

Active noise control, selective fixed-filter ANC, deep learning, hearables.

##  I Introduction

Active noise control (ANC) involves the generation of an anti-noise of equal
amplitude and opposite phase to suppress the unwanted noise [1, 2, 3, 4, 5, 6,
7]. Owing to its compact size and high efficiency in attenuating low frequency
noises, ANC has been widely used in industrial operations and consumer
products such as headphones [8, 9, 10, 11, 12, 13, 14]. Traditional ANC
systems typically employ adaptive algorithms to attenuate noises [15, 16, 17].
Filtered-X normalized least-mean-square (FxNLMS), being an adaptive ANC
algorithm, not only can achieve low noise reduction errors through adaptive
optimization but also does not require prior knowledge of the input data [18,
19, 20]. However, due to the least mean square (LMS) based algorithms’
inherent slow convergence and poor tracking ability [21], FxNLMS is less
capable of dealing with rapidly varying or non-stationary noises. Its slow
response to these noises may impact customers’ perceptions of noise reduction
effect [22].

In industrial applications and mature ANC products, fixed-filter ANC methods
[23, 24] are adopted to tackle the problems of adaptive algorithms, where the
control filter coefficients are fixed rather than adaptively updated to avoid
slow convergence and instability. However, the coefficients of the fixed-
filter algorithm are pre-trained for a specific noise type, resulting in the
degradation of noise reduction performance for other types of noises [25].
Therefore, for flexible selection of different pre-trained control filters in
response to different noise types, Shi et al. [26] proposed a selective fixed-
filter active noise control (SFANC) method based on the frequency band
matching. Nevertheless, several critical parameters of the method can only be
determined through trials and errors. Also, its filter-selection module
consumes extra computations in the real-time processor [27].

Thereby, automatic learning of the SFANC algorithm’s critical parameters is a
significant issue that would promote its uses in real-world scenarios and
industrial products. Under this consideration, deep learning techniques,
particularly convolutional neural networks (CNNs), achieving significant
success in classification [28, 29, 30, 31] appear to be an excellent approach
for improving SFANC. By replacing the filter-selection module with a CNN
model, the SFANC algorithm can automatically learn its parameters from noise
datasets and select the best control filter given different noise types
without resorting to extra-human efforts [32]. Additionally, a CNN model
implemented on a co-processor can decouple the computational load from the
real-time noise controller [32]. Though its superiority on response speed has
been demonstrated, the CNN-based SFANC algorithm is limited by some wrong
noise classifications and the lack of adaptive optimization, which may result
in large steady-state errors.

As discussed above, using either SFANC or FxNLMS is difficult to constantly
obtain satisfactory noise reduction performance for various noises. The SFANC
algorithm and FxNLMS algorithm can be effectively combined to overcome the
limitations of the individual ANC algorithms. Therefore, a hybrid SFANC-FxNLMS
algorithm is proposed in this paper. In the proposed algorithm, a lightweight
one-dimensional CNN (1D CNN) implemented on a co-processor can automatically
select the most suitable control filter for different noise types. For each
frame of the noise waveform, the control filter coefficients are decided by
the 1D CNN and continue to be updated adaptively via the FxNLMS algorithm at
the sampling rate. Simulation results show that the proposed algorithm not
only achieves faster noise reduction responses and lower steady-state errors
but also exhibits good tracking and robustness. Thus, it can be used for
attenuating dynamic noises such as traffic noises and urban noises etc.

The remainder of this paper is organized as follows: Section II introduces the
hybrid SFANC-FxNLMS algorithm in detail. Section III presents a series of
numerical simulations to evaluate the effectiveness of the proposed algorithm.
Finally, the conclusion is given in Section IV.

Figure 1: Block diagram of the hybrid SFANC-FxNLMS algorithm. For each frame,
the most suitable pre-trained control filter is selected by the 1D CNN
operated in a co-processor. Subsequently, FxNLMS continues to update the
coefficients of the chosen pre-trained control filter at the sampling rate.

##  II Hybrid SFANC-FxNLMS Algorithm

The overall framework of the hybrid SFANC-FxNLMS algorithm is presented in
Fig. 1. Although learning acoustic models directly from the raw waveform data
is challenging [33], the input of the proposed ANC system is raw waveform
instead of frequency spectrogram.

###  II-A A concise explanation of SFANC

An ANC progress can be abstracted as a first-order Markov chain [34, 35], as
shown in Fig. 2, where $\mathbf{w}_{o}(n)$ represents the optimal control
filter to deal with the disturbance $d(n)$. If we assumed that the optimal
control filter has $C$ discrete states as $\\{\mathbf{w}_{i}\\}^{C}_{i=1}$,
the control filter selection process can be considered as determining the best
control filter from a pre-trained filter set $\\{\mathbf{w}_{i}\\}^{C}_{i=1}$.
Hence, the SFANC strategy can be summarized as follows:

$\mathbf{w}_{o}=\underset{\mathbf{w}\in\\{\mathbf{w}_{i}\\}^{C}_{i=1}}{\mathrm{argmin}}~{}\mathbb{E}\bigg{\\{}\left[d(n)-\mathbf{x}^{\mathrm{T}}(n)\mathbf{w}(n)\ast
s(n)\right]^{2}\bigg{\\}},$ (1)

where $\text{argmin}(\cdot)$ operator returns the input value for minimum
output; $\ast$ , $\mathbf{x}(n)$, and $\mathbf{s}(n)$ represent the linear
convolution, the reference signal, and the impulse response of the secondary
path, respectively. The reference signal is assumed to be the same as the
primary noise.

In practice, $d(n)$ is typically regarded as the linear combination of
$\mathbf{x}(n)$. Thus, (1) is equivalent to

$\mathbf{w}_{o}=\underset{\mathbf{w}\in\\{\mathbf{w}_{i}\\}^{C}_{i=1}}{\mathrm{argmax}}P\left[\mathbf{w}|d(n)\right]=\underset{\mathbf{w}\in\\{\mathbf{w}_{i}\\}^{C}_{i=1}}{\mathrm{argmax}}P\left[\mathbf{w}|\mathbf{x}(n)\right],$
(2)

which refers to the process of selecting a control filter from the pre-trained
filter set that maximizes its posterior probability in the presence of
reference signal $\mathbf{x}(n)$. Moreover, according to Bayes’ theorem [36],
the posterior probability in (2) can be replaced with a conditional
probability as

$\mathbf{w}_{o}=\underset{\mathbf{w}\in\\{\mathbf{w}_{i}\\}^{C}_{i=1}}{\mathrm{argmax}}P\left[\mathbf{x}(n)|\mathbf{w}\right],$
(3)

which predicts the most suitable control filter straight from the reference
signal $\mathbf{x}(n)$.

To achieve a satisfactory noise reduction for the SFANC algorithm, we should
find a classifier model $\hat{P}\left[\mathbf{x}(n)|\mathbf{w},\Theta\right]$
to approximate ${P}\left[\mathbf{x}(n)|\mathbf{w}\right]$ from the pre-
recorded sampling set $\\{\mathbf{x}^{j}(n),\mathbf{w}^{j}\\}^{N}_{j}$. The
$\Theta$ denotes the parameters of the classifier and can be obtained through
maximum likelihood estimation (MLE) [37] as

$\Theta=\underset{\Theta}{\mathrm{argmax}}\frac{1}{N}\sum^{N}_{j=1}\log{\hat{P}\left[\mathbf{x}^{j}(n)|\mathbf{w}^{j},\Theta\right]}.$
(4)

Hence, we can utilize the deep learning approach to extract this classifier
model from the training set $\\{\mathbf{x}^{j}(n),\mathbf{w}^{j}\\}^{N}_{j}$.

Figure 2: Markov model of the active noise control progress

Figure 3: The frequency bands of the broadband noises in pre-training stage.

###  II-B CNN-based SFANC algorithm

Motivated by the work [32], this paper develops a 1D CNN capable of dealing
with noise waveforms end-to-end. The input of the network is the normalized
noise waveform with a 1-second duration. The min-max operation is defined as:

$\hat{x}(n)=\frac{x(n)}{\max[\mathbf{x}(n)]-\min[\mathbf{x}(n)]},$ (5)

where $\max[\cdot]$ and $\min[\cdot]$ obtain the maximum and minimum value of
the input vector $\mathbf{x}(n)$. It aims to rescale the input range into
$(-1,1)$ and retain the signal’s negative part that contains phase
information.

####  II-B1 Pre-training control filters

The primary and secondary paths used in the training stage of the control
filters are band-pass filters with a frequency range of 20Hz-7,980Hz.
Broadband noises with $15$ frequency ranges shown in Fig. 3 are used to pre-
train $15$ control filters. The filtered reference least mean square (FxLMS)
algorithm is adopted to obtain the optimal control filters for these broadband
noises due to its low computational complexity [38]. Subsequently, these $15$
pre-trained control filters are saved in the control filter database.

####  II-B2 The Proposed 1D CNN

The detailed architecture of the 1D CNN is illustrated in Fig. 4. Every
residual block in the network is composed of two convolutional layers,
subsequent batch normalization [39], and ReLU non-linearity [40]. Note that a
shortcut connection is adopted to add the input with the output in each
residual block since residual architecture is demonstrated easy to be
optimized [41]. Additionally, the network uses a broad receptive field (RF) in
the first convolutional layer and narrow RFs in the rest convolutional layers
to fully exploit both global and local information.

In order to train the 1D CNN model, we generated $80,000$ broadband noise
tracks with various frequency bands, amplitudes, and background noise levels
at random. Each track has a duration of 1 second. These noise tracks are taken
as primary noises to generate disturbances. The class label of a noise track
corresponds to the index of the control filter that achieves the best noise
reduction performance on the disturbance.

Figure 4: Architecture of the 1D CNN used to select the most suitable pre-
trained control filter for each frame of the noise waveform. The configuration
of convolution layer is denoted as: (kernel size, channels, stride, padding).

###  II-C The FxNLMS algorithm

The FxNLMS algorithm is a conventional adaptive ANC algorithm [42], where the
error signal can be obtained from:

$e(n)=d(n)-\mathbf{x}^{\mathrm{T}}(n)\mathbf{w}(n)\ast s(n),$ (6)

where $\mathbf{w}(n)$, $(\cdot)^{\mathrm{T}}$ and $\ast$ represent the control
filter, the transposed operation and linear convolution, respectively. The
weight update equation of control filter is given by:

$\displaystyle\mathbf{w}(n+1)$
$\displaystyle=\mathbf{w}(n)+\mu\mathbf{r}(n)e(n)\Gamma_{k}^{-1},$ (7)
$\displaystyle\mathbf{r}(n)$
$\displaystyle=\hat{\mathbf{s}}(n)*\mathbf{x}(n),$ $\displaystyle\Gamma_{k}$
$\displaystyle=\beta+\mathbf{r}^{T}(n)\mathbf{r}(n),$

in which, $\mu$ denotes the step size. $\mathbf{r}(n)$ is the filtered
reference signal generated by passing the reference signal $\mathbf{x}(n)$
through the estimated secondary path $\hat{\mathbf{s}}(n)$. Additionally,
$\beta$ is a small constant that avoids dividing by zero in the denominator.

###  II-D Hybrid SFANC-FxNLMS Algorithm

As illustrated in Fig. 1, the primary noise is sampled at a rate of
$16,000$Hz, while each data frame is made up of $16,000$ samples ($1$ second
duration data). Throughout the control process, the co-processor (e.g., a
mobile phone) utilizes the pre-trained 1D CNN to output the index of the most
appropriate control filter for each frame and sends it to the real-time
controller. When the received filter index differs from the current index, the
real-time controller will swap the control filter. Simultaneously, the real-
time controller continues to update the control filter’s coefficients via (7)
at each sample. Notably, the co-processor operates at the frame rate, whereas
the real-time controller performs at the sampling rate.

The hybrid SFANC-FxNLMS algorithm can achieve faster responses to different
noises with the assistance of the SFANC. Meanwhile, the FxNLMS algorithm
continues to update the control filter’s coefficients in the SFANC-FxNLMS
algorithm, which contributes to obtain lower noise reduction errors.

TABLE I: Comparisons of different 1D networks Network | Test Accuracy | #Layers | #Parameters
---|---|---|---
Proposed Network | 99.35$\%$ | 6 | 0.21M
M3 Network | 99.15$\%$ | 3 | 0.22M
M5 Network | 99.10$\%$ | 5 | 0.56M
M11 Network | 99.15$\%$ | 11 | 1.79M
M18 Network | 98.50$\%$ | 18 | 3.69M
M34-res Network | 98.35$\%$ | 34 | 3.99M

##  III Experiments

In the experiments, we trained the 1D CNN model on a synthesized noise dataset
and then evaluated the hybrid SFANC-FxNLMS algorithm on real-record noises.
The primary path and secondary path are chosen as band-pass filters. Moreover,
the step size of the FxNLMS algorithm is set to $0.002$, and the control
filter length is $1,024$ taps.

###  III-A Training and testing of 1D CNN

The synthetic noise dataset was subdivided into three subsets: $80,000$ noise
tracks for training, $2,000$ noise tracks for validation, and $2,000$ noise
tracks for testing. During training, the Adam algorithm [43] was used for
optimization. The training epoch was set at $30$. To avoid exploding or
vanishing gradients, the glorot initialization [44] was used. To avoid
overfitting due to the scarcity of training data, the 1D network was built as
a light-weighted network with weights all subjected to $\ell_{2}$
regularization with a coefficient of $0.0001$.

The proposed 1D CNN is compared to several different 1D networks proposed in
[33] utilizing raw acoustic waveforms: M3, M5, M11, M18, and M34-res. Table I
summarizes the comparison results. The number of network layers includes the
number of convolutional layers and fully connected layers. Among these
networks, the proposed 1D network obtains the highest classification accuracy
of $99.35\%$ on the test dataset, which means it can provide the most suitable
pre-trained control filters for different noises. Also, the proposed 1D
network is lightweight, utilizing the fewest network parameters possible due
to its efficient architecture.

###  III-B Real-world noise cancellation

This section presents using the SFANC, FxNLMS, and hybrid SFANC-FxNLMS
algorithm to attenuate real-record noises. The real-world noises are not
included in the training datasets. We created two distinct noises by cascading
and mixing an aircraft noise with a frequency range of 50Hz-14,000Hz and a
traffic noise with a frequency range of 40Hz–1,400Hz.

####  III-B1 Cascaded noise attenuation

The varying noise in this simulation is provided by cascading aircraft noise
and traffic noise. The noise reduction results using different ANC methods on
the cascaded noise are shown in Fig. 5.

Figure 5: (a)-(c): Error signals of different ANC algorithms, (d): Averaged
noise reduction level of every 1 second, on the cascaded noise.

From the result, we can observe that the hybrid SFANC-FxNLMS algorithm can
track and respond quickly to different parts of the varying noise. During the
noise reduction process, the hybrid SFANC-FxNLMS algorithm consistently
outperforms the SFANC and FxNLMS algorithms. In particular, during
$10$s-$11$s, the averaged noise reduction level achieved by the hybrid SFANC-
FxNLMS algorithm is 6dB and 15dB more than that of SFANC and FxNLMS,
respectively. Noticeably, it is the transition period from one noise to
another. Therefore, even when the frequency band and amplitude of the noise
change, the hybrid SFANC-FxNLMS algorithm can still attenuate it.

Figure 6: (a)-(c): Error signals of different ANC algorithms, (d): Averaged
noise reduction level of every 1 second, on the mixed noise.

####  III-B2 Mixed noise attenuation

In addition to cascading, we added the aircraft noise and traffic noise
together to get a mixed noise. Fig. 6 depicts the performance of various ANC
algorithms on the mixed noise. The SFANC algorithm responds to mixed noise
much faster than the FxNLMS algorithm, as illustrated in the figure. For
example, the SFANC method achieves a noise reduction level of about $18$dB
after $1$s, whereas the FxNLMS algorithm takes more than $10$s to achieve the
same level. However, after convergence during $12$s-$13$s, the FxNLMS
algorithm achieves a higher noise reduction level than the SFANC method.

Also, it is noticeable that the SFANC algorithm is incapable of dealing with
new noise during the first $1$s, since it only updates the control filter
index after the first $1$s. However, owing to combining with FxNLMS, the
hybrid SFANC-FxNLMS algorithm can slightly attenuate the noise during the
first $1$s. Furthermore, we can see that the averaged noise reduction level of
hybrid SFANC-FxNLMS algorithm is about $10$dB higher than that of SFANC and
FxNLMS algorithms in $12$s-$13$s. Therefore, the results on the mixed noise
also confirm the superiority of the hybrid SFANC-FxNLMS algorithm over the
SFANC and FxNLMS algorithm.

Figure 7: (a)-(c): Error signals of different ANC algorithms, (d): Averaged
noise reduction level of every 1 second, when dealing with a varying primary
path.

###  III-C Noise reduction performance on the varying primary path

In this section, the robustness of different ANC algorithms are examined when
dealing with a primary path that changes every $13$ seconds. The impulse
response of the primary path is mixed with different levels of white noise to
obtain additional signal-to-noise ratios (SNR) of $30$dB and $10$dB
respectively as shown in Fig. 7. The primary noise is the traffic noise.

According to Fig. 7, in the initial period, the SFANC and hybrid SFANC-FxNLMS
algorithm have better noise reduction performances over the FxNLMS algorithm.
However, when the SNR of the primary path varies, SFANC can’t perform as well
as FxNLMS and hybrid SFANC-FxNLMS algorithm. Since the primary noise type
remains the same, the selected pre-trained control filter is fixed in the
SFANC algorithm, which may not be suitable on the varying primary path. The
results indicate that the hybrid SFANC-FxNLMS algorithm is sufficiently robust
to deal with the varying primary path.

##  IV Conclusion

The proposed hybrid SFANC-FxNLMS algorithm effectively combines SFANC and
FxNLMS to achieve faster noise reduction responses and lower steady-state
errors. A lightweight 1D CNN operating at frame rate is used to automatically
determine control filter coefficients. Also, the FxNLMS algorithm continues to
update the coefficients adaptively at the sampling rate. Simulation results
validate the good tracking capability and robustness of the hybrid SFANC-
FxNLMS algorithm when dealing with varying noises and primary paths in dynamic
noise control environments. The proposed hybrid algorithm is promising to be
further explored for combining other fixed-filter and adaptive ANC algorithms.

## References

  * [1] C. N. Hansen, _Understanding active noise cancellation_. CRC Press, 2002.
  * [2] S. Elliot and P. Nelson, “Active noise control,” _Noise News International_ , vol. 2, no. 2, pp. 75–98, 1994.
  * [3] S. M. Kuo and D. R. Morgan, “Active noise control: a tutorial review,” _Proceedings of the IEEE_ , vol. 87, no. 6, pp. 943–973, 1999.
  * [4] Y. Kajikawa, W.-S. Gan, and S. M. Kuo, “Recent advances on active noise control: open issues and innovative applications,” _APSIPA Transactions on Signal and Information Processing_ , vol. 1, 2012.
  * [5] X. Qiu, _An introduction to virtual sound barriers_. CRC Press, 2019.
  * [6] J. Zhang, “Active noise control over spatial regions,” Ph.D. dissertation, The Australian National University (Australia), 2019.
  * [7] D. Chen, L. Cheng, D. Yao, J. Li, and Y. Yan, “A secondary path-decoupled active noise control algorithm based on deep learning,” _IEEE Signal Processing Letters_ , vol. 29, pp. 234–238, 2021.
  * [8] S. Liebich, J. Fabry, P. Jax, and P. Vary, “Signal processing challenges for active noise cancellation headphones,” in _Speech Communication; 13th ITG-Symposium_. VDE, 2018, pp. 1–5.
  * [9] P. Rivera Benois, P. Nowak, and U. Zölzer, “Evaluation of a decoupled feedforward-feedback hybrid structure for active noise control headphones in a multi-source environment,” in _INTER-NOISE and NOISE-CON Congress and Conference Proceedings_ , vol. 255, no. 4. Institute of Noise Control Engineering, 2017, pp. 3691–3699.
  * [10] S. M. Kuo and D. R. Morgan, “Active noise control: a tutorial review,” _Proceedings of the IEEE_ , vol. 87, no. 6, pp. 943–973, 1999.
  * [11] C.-Y. Chang, A. Siswanto, C.-Y. Ho, T.-K. Yeh, Y.-R. Chen, and S. M. Kuo, “Listening in a noisy environment: Integration of active noise control in audio products,” _IEEE Consumer Electronics Magazine_ , vol. 5, no. 4, pp. 34–43, 2016.
  * [12] S. Liebich, C. Anemüller, P. Vary, P. Jax, D. Rüschen, and S. Leonhardt, “Active noise cancellation in headphones by digital robust feedback control,” in _2016 24th European Signal Processing Conference (EUSIPCO)_. IEEE, 2016, pp. 1843–1847.
  * [13] X. Shen, D. Shi, and W.-S. Gan, “A wireless reference active noise control headphone using coherence based selection technique,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 7983–7987.
  * [14] K. Coker and C. Shi, “A survey on virtual bass enhancement for active noise cancelling headphones,” in _2019 International Conference on Control, Automation and Information Sciences (ICCAIS)_. IEEE, 2019, pp. 1–5.
  * [15] R. M. Reddy, I. M. Panahi, and R. Briggs, “Hybrid fxrls-fxnlms adaptive algorithm for active noise control in fmri application,” _IEEE Transactions on Control Systems Technology_ , vol. 19, no. 2, pp. 474–480, 2010\.
  * [16] J. Guo, F. Yang, and J. Yang, “Convergence analysis of the conventional filtered-x affine projection algorithm for active noise control,” _Signal Processing_ , vol. 170, p. 107437, 2020.
  * [17] F. Yang, Y. Cao, M. Wu, F. Albu, and J. Yang, “Frequency-domain filtered-x lms algorithms for active noise control: A review and new insights,” _Applied Sciences_ , vol. 8, no. 11, p. 2313, 2018.
  * [18] J. C. Burgess, “Active adaptive sound control in a duct: A computer simulation,” _The Journal of the Acoustical Society of America_ , vol. 70, no. 3, pp. 715–726, 1981.
  * [19] D. Shi, W.-S. Gan, B. Lam, S. Wen, and X. Shen, “Active noise control based on the momentum multichannel normalized filtered-x least mean square algorithm,” in _INTER-NOISE and NOISE-CON Congress and Conference Proceedings_ , vol. 261, no. 6. Institute of Noise Control Engineering, 2020, pp. 709–719.
  * [20] L. Lu, K.-L. Yin, R. C. de Lamare, Z. Zheng, Y. Yu, X. Yang, and B. Chen, “A survey on active noise control techniques–part i: Linear systems,” _arXiv preprint arXiv:2110.00531_ , 2021.
  * [21] Y. Tsao, S.-H. Fang, and Y. Shiao, “Acoustic echo cancellation using a vector-space-based adaptive filtering algorithm,” _IEEE Signal Processing Letters_ , vol. 22, no. 3, pp. 351–355, 2014.
  * [22] R. Ranjan and W.-S. Gan, “Natural listening over headphones in augmented reality using adaptive filtering techniques,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 23, no. 11, pp. 1988–2002, 2015\.
  * [23] R. Ranjan, T. Murao, B. Lam, and W.-S. Gan, “Selective active noise control system for open windows using sound classification,” in _INTER-NOISE and NOISE-CON Congress and Conference Proceedings_ , vol. 253, no. 6. Institute of Noise Control Engineering, 2016, pp. 1921–1931.
  * [24] C. Shi, R. Xie, N. Jiang, H. Li, and Y. Kajikawa, “Selective virtual sensing technique for multi-channel feedforward active noise control systems,” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 8489–8493.
  * [25] S. Wen, W.-S. Gan, and D. Shi, “Using empirical wavelet transform to speed up selective filtered active noise control system,” _The Journal of the Acoustical Society of America_ , vol. 147, no. 5, pp. 3490–3501, 2020.
  * [26] D. Shi, W.-S. Gan, B. Lam, and S. Wen, “Feedforward selective fixed-filter active noise control: Algorithm and implementation,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 1479–1492, 2020.
  * [27] D. Shi, B. Lam, and W.-S. Gan, “A novel selective active noise control algorithm to overcome practical implementation issue,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 1130–1134.
  * [28] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” _nature_ , vol. 521, no. 7553, pp. 436–444, 2015.
  * [29] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1–9.
  * [30] J. Sikora, R. Wagnerová, L. Landryová, J. Šíma, and S. Wrona, “Influence of environmental noise on quality control of hvac devices based on convolutional neural network,” _Applied Sciences_ , vol. 11, no. 16, p. 7484, 2021.
  * [31] H. Zhang and D. Wang, “Deep anc: A deep learning approach to active noise control,” _Neural Networks_ , vol. 141, pp. 1–10, 2021.
  * [32] D. Shi, B. Lam, K. Ooi, X. Shen, and W.-S. Gan, “Selective fixed-filter active noise control based on convolutional neural network,” _Signal Processing_ , vol. 190, p. 108317, 2022.
  * [33] W. Dai, C. Dai, S. Qu, J. Li, and S. Das, “Very deep convolutional neural networks for raw waveforms,” in _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2017, pp. 421–425.
  * [34] P. A. Lopes and M. Piedade, “The kalman filter in active noise control,” in _INTER-NOISE and NOISE-CON Congress and Conference Proceedings_ , vol. 1999, no. 5. Institute of Noise Control Engineering, 1999, pp. 1111–1124.
  * [35] P. A. Lopes and M. S. Piedade, “A kalman filter approach to active noise control,” in _2000 10th European Signal Processing Conference_. IEEE, 2000, pp. 1–4.
  * [36] S. M. Kay, _Fundamentals of statistical signal processing: estimation theory_. Prentice-Hall, Inc., 1993.
  * [37] S. van Ophem and A. P. Berkhoff, “Multi-channel kalman filters for active noise control,” _The Journal of the Acoustical Society of America_ , vol. 133, no. 4, pp. 2105–2115, 2013.
  * [38] F. Yang, J. Guo, and J. Yang, “Stochastic analysis of the filtered-x lms algorithm for active noise control,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 2252–2266, 2020.
  * [39] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _International conference on machine learning_. PMLR, 2015, pp. 448–456.
  * [40] M. D. Zeiler, M. Ranzato, R. Monga, M. Mao, K. Yang, Q. V. Le, P. Nguyen, A. Senior, V. Vanhoucke, J. Dean _et al._ , “On rectified linear units for speech processing,” in _2013 IEEE International Conference on Acoustics, Speech and Signal Processing_. IEEE, 2013, pp. 3517–3521.
  * [41] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [42] S. C. Douglas and T.-Y. Meng, “An optimum nlms algorithm: Performance improvement over lms,” in _Acoustics, Speech, and Signal Processing, IEEE International Conference on_. IEEE Computer Society, 1991, pp. 2125–2126.
  * [43] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [44] X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in _Proceedings of the thirteenth international conference on artificial intelligence and statistics_. JMLR Workshop and Conference Proceedings, 2010, pp. 249–256.