# Retromorphic Testing

A New Approach to the Test Oracle Problem

Boxi Yu<EMAIL_ADDRESS>School of Data Science, The Chinese
University of Hong Kong, Shenzhen (CUHK-Shenzhen), China ,  Qiuyang Mang
<EMAIL_ADDRESS>School of Data Science, The Chinese University of
Hong Kong, Shenzhen (CUHK-Shenzhen), China ,  Qingshuo Guo
<EMAIL_ADDRESS>School of Data Science, The Chinese University of
Hong Kong, Shenzhen (CUHK-Shenzhen), China and  Pinjia He
<EMAIL_ADDRESS>School of Data Science, The Chinese University of Hong
Kong, Shenzhen (CUHK-Shenzhen), China

###### Abstract.

A test oracle serves as a criterion or mechanism to assess the correspondence
between software output and the anticipated behavior for a given input set. In
automated testing, black-box techniques, known for their non-intrusive nature
in test oracle construction, are widely used, including notable methodologies
like differential testing and metamorphic testing. Inspired by the
mathematical concept of inverse function, we present Retromorphic Testing, a
novel black-box testing methodology. It leverages an auxiliary program in
conjunction with the program under test, which establishes a dual-program
structure consisting of a forward program and a backward program. The input
data is first processed by the forward program and then its program output is
reversed to its original input format using the backward program. In
particular, the auxiliary program can operate as either the forward or
backward program, leading to different testing modes. The process concludes by
examining the relationship between the initial input and the transformed
output within the input domain. For example, to test the implementation of the
sine function $\sin(x)$, we can employ its inverse function, $\arcsin(x)$, and
validate the equation $x=\sin(\arcsin(x)+2k\pi),\forall k\in\mathbb{Z}$. In
addition to the high-level concept of Retromorphic Testing, this paper
presents its three testing modes with illustrative use cases across diverse
programs, including algorithms, traditional software, and AI applications.

Test oracle problem, black-box testing, automated testing

††copyright: none

##  1\. Introduction

Test oracle plays a pivotal role in software testing (Barr et al., 2015),
serving as a critical benchmarking mechanism for assessing the correctness of
software output given specific input (Howden, 1978). The significance of test
oracles becomes apparent in automated validation because they establish the
ultimate standard for the evaluation of system behaviors. However, test
oracles are difficult to obtain in practice due to many reasons, such as the
lack of clear specifications, the complexity and dynamic nature of software,
and limited resources.

To solve the test oracle problem, research in recent decades has proposed a
variety of useful testing techniques (Barr et al., 2015; Pezzè and Zhang,
2014; Segura et al., 2016; Chen et al., 2018). Among these techniques,
differential testing (McKeeman, 1998) and metamorphic testing (Chen et al.,
1998) are two typical methodologies that have been widely employed because of
their black-box nature and simple yet effective high-level ideas. In
differential testing, multiple systems or software versions are expected to
exhibit identical behaviors for the shared functionalities. Discrepancies in
output for identical inputs among these systems indicate potential bugs in one
or more systems. Metamorphic testing generates new test inputs based on
existing input-output pairs, with the output of the generated input being
predictable based on the relationship between the original input and the
generated one. The detection of deviations in output relationships can
highlight potential bugs in the target systems. In addition to these black-box
methodologies, Rigger and Su (Rigger and Su, 2022) recently introduced
intramorphic testing, a white-box automated testing methodology, which
modifies a system component and constructs a test oracle that can relate the
output of the original and modified systems given the same input.

This paper introduces Retromorphic Testing, a general black-box methodology to
tackle the challenges in test oracle construction. The core idea of
Retromorphic Testing is inspired by mathematical relationships between
functions and their inverse functions, expressed as $f^{-1}(f(x))=x$. It
employs an auxiliary program to reverse the output of the software,
transforming it back to its original input format. Different from differential
testing and metamorphic testing, which typically involve a system or systems
with equivalent functionalities, Retromorphic Testing employs a “dual-program
structure”, comprising a forward program and a backward program, where the
forward program corresponds to $f(x)$ and the backward program corresponds to
$f^{-1}(x)$. The choice of the auxiliary program, whether forward or backward,
depends on the system under test.

The dual-program structure enables three testing modes: forward mode, backward
mode, and integrated mode, which is decided by the role of the system under
test in the structure. For example, to test the $\sin(x)$ function using
Retromorphic Testing, the forward mode employs the test oracle
$\arcsin(\sin(x))=x,\frac{\pi}{2}\leq x\leq\frac{\pi}{2}$, where the program
under test (i.e., $\sin(x)$) is the forward program; while the backward mode
uses the test oracle $\sin(\arcsin(x))=x,-1\leq x\leq 1$. The integrated mode
uses the system under test as both the forward program and backward program.
To test the $\frac{1}{x}$ function, the integrated mode can construct the test
oracle $f(f(x))=x,x\neq 0$. The diverse testing modes provided by Retromorphic
Testing bring additional flexibility to test oracle construction by assigning
different roles to the programs involved.

To demonstrate the flexibility of Retromorphic Testing, we provide six
practical examples of possible implementations of Retromorphic Testing in
validating algorithms, traditional software, and AI software. We believe
Retromorphic Testing has been realized in different testing scenarios by
researchers or practitioners. However, they mainly regard it as an ad hoc
solution for testing specific software or as implementation details of part of
the test suite. This paper aims to distill the high-level concept behind and
provide a general definition of this methodology, which we hope will further
facilitate the development and discussion of more instances of Retromorphic
Testing.

In summary, this paper makes the following contributions:

  * •

It introduces Retromorphic Testing, a black-box methodology to tackle the test
oracle problem.

  * •

It discusses three testing modes enabled by the dual-program structure in
Retromorphic Testing.

  * •

It provides six instances of Retromorphic Testing.

##  2\. Preliminaries

Figure 1. Differential testing, metamorphic testing, and Retromorphic Testing
in comparison.

###  2.1. Test Oracles

The test oracle problem stands as one of the most significant challenges in
the realm of software testing. A test oracle serves as a mechanism designed to
verify the correctness of a system’s output for a set of specific inputs
(Howden, 1978). This problem encompasses the complex task of automating the
differentiation between expected and potentially erroneous system behavior,
which poses a significant bottleneck in the development of more comprehensive
testing methodologies. Ideally, a system would be accompanied by a detailed
specification of its expected behaviors, or its code would contain predefined
conditions, allowing for an automated test oracle to verify outputs without
human intervention. However, in many real-world scenarios, software developers
lack such luxuries, leading to manual verification of system behaviors by
handmade test cases, which can be extremely time-consuming and inefficient in
bug detection. To mitigate this challenge, several approaches that generate
“partial test oracles” (Barr et al., 2015) have been proposed, which can still
automatically validate system output from a part of inputs. The most
influential ones are differential testing (McKeeman, 1998) and metamorphic
Testing (Chen et al., 1998).

###  2.2. Terminology

For existing testing techniques such as differential testing (McKeeman, 1998)
and metamorphic testing (Chen et al., 1998), test oracles were defined to
validate outputs for a set of inputs (Howden, 1978). In this paper, we
continue to use this terminology in these approaches, where the target program
is denoted as $P$, the input as $I$, and the output as $O$, where $O=P(I)$.

However, we will not use the original terminology in Retromorphic Testing due
to its dual-program structure, but consider the program execution as a
transition between two modalities. In this scenario, we use
$M_{i},i\in\\{1,2\\}$ to represent the data belonging to the $i$-th modality.
We define the forward program $P$ as the program that transforms the data from
$M_{1}$ to $M_{2}$ ($M_{2}=P(M_{1})$), and the backward program $Q$ that
transforms the data from $M_{2}$ back to $M_{1}$ ($M_{1}=Q(M_{2})$).

###  2.3. Black-box Testing Techniques

Among existing test oracle approaches, black-box testing techniques are a set
of methodologies where the tester is not required to know the internal
knowledge or implementation details of the target system. Instead, the focus
is on evaluating system behaviors based on its inputs and outputs. Among the
prominent black-box testing techniques are differential testing (McKeeman,
1998) and metamorphic testing (Chen et al., 1998). Differential testing
involves comparing the outputs of two or more versions of a software
application to identify discrepancies, often used to detect inconsistencies in
versions of compilers or other tools that should produce equivalent results.
With its effectiveness, differential testing has been applied to a variety of
domains, such as Rust compilers (Sharma et al., 2023), model counters (Usman
et al., 2021), Java Virtual Machines (JVMs) (Chen et al., 2016, 2019),
database engines (Slutz, 1998; Zheng et al., 2022), and deep-learning
libraries (Yang et al., 2023; Deng et al., 2023). Metamorphic testing is a
technique where the tester identifies certain properties that should remain
invariant under specific transformations of the input. If these properties
change after the transformation, a defect might be signaled. This approach is
particularly useful in scenarios where sufficient test oracles not be
available or are too expensive to use. Metamorphic testing is also used for
testing various systems such as compilers (Le et al., 2014; Li et al., 2023),
database engines (Rigger and Su, 2020b, a; Kamm et al., 2023), SMT solvers
(Winterer et al., 2020), Android apps (Su et al., 2021; Sun et al., 2021),
quantum computing platforms (Paltenghi and Pradel, 2023; Abreu et al., 2022),
and AI systems (Wang et al., 2023; Sun et al., 2020; Yu et al., 2023b, a; Wang
and Su, 2020).

In Fig. 1 (a) and (b), we give the workflow of differential testing and
metamorphic testing, respectively. Differential testing uses a fixed input $I$
for programs $P$ and $P^{\prime}$ of the same functionality and validates
whether their outputs, i.e., $O=P(I)$ and $O^{\prime}=P^{\prime}(I)$, are
equal. When there is a discrepancy between $O$ and $O^{\prime}$, it would
indicate bugs in either $P$ or $P^{\prime}$. Unlike differential testing which
uses the same input for different programs, metamorphic testing mutates the
input $I$ to $I^{\prime}$ and feeds them into one given program. It is based
on the idea of defining metamorphic relations (MRs) that capture the expected
relationships between the outputs $O=P(I)$ and $O^{\prime}=P(I^{\prime})$ of
the program.

##  3\. Retromorphic Testing

###  3.1. Testing Principle

As illustrated in Fig. 1 (c), Retromorphic Testing adopts a dual-program
structure consisting of a forward program $P$ and a backward program $Q$. Its
testing process starts with $M_{1}$ (e.g., trigonometric values), which serves
as the generated input for the forward program $P$ (e.g., $y=\arcsin(x)$).
Using $P$, Retromorphic Testing transforms $M_{1}$ into output $M_{2}$ (e.g.,
arcs). Following this, a mutation method denoted as $Mutate$ is applied to
convert $M_{2}$ into $M_{2}^{\prime}$, where $M_{2}^{\prime}=Mutate(M_{2})$
(e.g., $\theta^{\prime}=\theta+2k\pi,k\in\mathbb{Z}$). Notably, the function
$Mutate$ could be an identity function, making $M_{2}$ and $M_{2}^{\prime}$
identical. The mutated data, $M_{2}^{\prime}$, is then input into program $Q$
(e.g., $y=\sin(x)$), resulting in $M_{1}^{\prime}$. The Retromorphic Relation
exists between $M_{1}$ and $M_{1}^{\prime}$ (e.g.,
$x=\sin(\arcsin(x)+2k\pi),k\in\mathbb{Z}$). If this Retromorphic Relation does
not hold, a bug is detected in program $P$ or $Q$.

###  3.2. Testing Modes

In the dual-program structure of Retromorphic Testing, the target program can
serve as either the forward program $P$ or the backward program $Q$. This
results in three testing modes for Retromorphic Testing, namely forward mode,
backward mode, and integrated mode. Each mode has a distinct process for input
generation and result validation, offering developers flexibility in designing
generators and validating programs.

#### Forward Mode.

In the forward mode of Retromorphic Testing, the target program serves as the
forward program $P$. Here, our task is to find a backward program $Q$, which
transforms the output produced by $P$ back into the same format as $P$’s
input. For instance, if we want to test a program like $P\colon y=\sin(x)$
(assuming it contains bugs) using the forward mode, we can select a backward
program like $Q\colon y=\arcsin(x)$ (assuming it is bug-free). We generate a
set of arc inputs within the range of $-\frac{\pi}{2}$ to $\frac{\pi}{2}$ and
pass them through $P$ to obtain the trigonometric values. After that, we send
these values to $Q$, transforming them back to the arcs (i.e.,
$\arcsin(\sin(x))=x$). By comparing the generated arcs with the arcs returned
from $Q$, we can identify potential bugs in the target program. In the forward
mode, we need to generate the data and validate the results, which are both in
the input modality of the target program (e.g., arcs in the above example).

#### Backward Mode.

In the back mode of Retromorphic Testing, the target program serves as the
backward program $Q$. Here, our task is to find a forward program $P$, where
the target program can transform the output produced by $P$ back into the same
format as $P$’s input. Similar to the forward mode, if we want to test a
program like $Q\colon y=\sin(x)$ (assuming it contains bugs) using the
backward mode, we can select a back program like $P\colon y=\arcsin(x)$
(assuming it is bug-free). We generate a set of trigonometric value inputs
within the range of $-1$ to $1$ and pass them through $P$ to obtain the arcs.
We can also mutate these arcs by adding $2k\pi$ ($k\in\mathbb{Z}$) to them.
After that, we send these mutated arcs to $Q$, transforming them back to the
trigonometric values (i.e., $\sin(\arcsin(x)+2k\pi)=x$). By comparing the
generated trigonometric values with the values returned from $Q$, we can
identify potential bugs in the target program. In the backward mode, we need
to generate the data and validate the results, which are both in the output
modality of the target program (e.g., trigonometric values in the above
example).

#### Integrated Mode.

In the integrated mode of Retromorphic Testing, the target program serves as
both the forward program $P$ and the backward program $Q$ simultaneously (if
possible), which means that we do not need to find another auxiliary program
in this mode. In particular, $P$ and $Q$ could be the same system or two
different components of the same system. For example, if we want to test a
program like $P\colon y=\frac{1}{x}$, we can simply select itself as the other
program (i.e., $Q\colon y=\frac{1}{x}$). We generate a set of non-zero number
inputs and pass them through $P$ and $Q$ sequentially (i.e.,
$x=\frac{1}{1/x}$). By comparing the generated number with the number returned
from $Q$, we can identify potential bugs in the target program. In the
integrated mode, the process of the data generation and results validation can
be in either the input or output modality of the target program (e.g., non-
zero values in the above example).

###  3.3. Examples of Different Testing Methodologies

To outline the existing techniques and our idea, let us assume a typical use
case, namely that we want to test the implementation of the Discrete Fourier
Transform (DFT). DFT is a mathematical technique that has been widely used in
practice, such as signal processing, image compression, and audio processing.
DFT transforms a sequence from the time domain (or spatial domain) to the
frequency domain. Given a time-domain sequence of $N$ numbers
$x_{0},x_{1},\ldots,x_{N-1}$, the sequence after DFT is a frequency-domain
sequence of $N$ complex numbers given by:

(1) $X_{k}=\sum_{n=0}^{N-1}x_{n}\cdot\exp(-j\frac{2\pi}{N}kn),$

for $k$ = $0$, $1$, …, $N-1$, where:

  * •

$X_{k}$ are the frequency components of the sequence.

  * •

$\exp$ is the base of the natural logarithm.

  * •

$j$ is the imaginary unit.

The inverse operation, which transforms the frequency-domain sequence back to
the time domain, is called the Inverse Discrete Fourier Transform (IDFT). IDFT
is defined as:

(2) $x_{n}=\frac{1}{N}\sum_{k=0}^{N-1}X_{k}\cdot\exp(j\frac{2\pi}{N}kn)$

Let us further assume that we made a mistake when implementing the
discrete_fourier_transform() function, as illustrated in Listing 1; The
coefficient is incorrectly typed as $-1j$ rather than $-2j$. In the subsequent
paragraphs, we discuss how the instantiations of existing techniques and an
instantiation of our proposed Retromorphic Testing, could find the bug. In
practice, we expect that Retromorphic Testing will be realized that can find
bugs that are overlooked, or difficult to find, by other testing approaches.

Listing 1: A Python implementation of Discrete Fourier Transform

⬇

import numpy as np

def discrete_fourier_transform(sequence, opt = 1):

N = len (sequence)

sequence_new = [0 for _ in range ( N )]

for k in range (N):

for n in range (N):

sequence_new [k] += sequence [n] * np.exp (opt * -1j * np.pi / N * k * n) #
Bug: should be -2j

if(opt == -1):

for i in range(len(sequence_new)):

sequence_new[i] = sequence_new[i] / N

return sequence_new

#### Manual Testing.

For manual testing, we need to generate the test cases and their corresponding
expected outputs. Listing 2 shows a test case in which we use manual testing
to detect the bug of the DFT algorithm. The algorithm under test transforms a
sequence [1, 0, 1, 0] into [2, 1, 0, 0], whereas the expected output after DFT
is [2, 0, 2, 0], thus the bug of the algorithm is uncovered. While manual
testing is effective and widely utilized, it demands considerable effort from
developers to create test cases and expected outputs.

Listing 2: A manually-written test case for Discrete Fourier Transform.

⬇

time_sequence = [2 , 0 , 1 , 0]

frequency_sequence = discrete_fourier_transform(time_sequence)

frequency_sequence = [int(_.real) for _ in frequency_sequence]

assert frequency_sequence == [2 , 0 , 2 , 0]

#### Differential Testing.

Differential testing (McKeeman, 1998) validates a set of systems that
implement the same semantics, by comparing their output for a given input.
Given that the test oracle requires no human in the loop, it can be
effectively paired with automated test generation. For example, in Listing 3,
we generate random sequences as test input. It is worth noticing that we use
an infinite loop in this example; in the practice of using differential
testing, it would be reasonable to set a timeout or run the tests for a fixed
number of iterations. For an input sequence like [1, 0, 1, 0], differential
testing reveals a discrepancy between the output of the two Fourier transform
algorithms, revealing the bug.

Listing 3: Differential testing using multiple implementations of transforming
algorithms.

⬇

eps = 1e-10

transform_algorithms = [discrete_fourier_transform, fast_fourier_transform]

while True :

time_sequence = get_random_sequence()

frequency_sequences = [alg(time_sequence.copy()) for alg in
transform_algorithms]

frequency_sequences = [[_.real for _ in frequency_sequence] for
frequency_sequence in frequency_sequences]

all_same = all(all(abs(frequency_sequence[i] - frequency_sequences[0][i]) <
eps for i in range(len(frequency_sequence))) for frequency_sequence in
frequency_sequences)

assert all_same

#### Metamorphic Testing.

To adopt metamorphic testing (Chen et al., 1998) for testing DFT, we can
deduce an MR from the definition formula of DFT. Assume
$x_{0},x_{1},\ldots,x_{N-1}$ is the input time-domain sequence, and
$X_{0},X_{1},\ldots,X_{N-1}$ is the correct output frequency-domain sequence
after DFT. When a constant $c$ is added to the first element (i.e., $x_{0}$)
to generate a new input sequence $x_{0}+c,x_{1},\ldots,x_{N-1}$, the output
frequency-domain sequence should be $X_{0}+c,X_{1}+c,\ldots,X_{N-1}+c$. We can
use the property of DFT as an MR to verify the implementation of DFT.

Using the implementation in Listing 1, when the input sequence is [1, 0, 1,
0], the output sequence is [2, 1, 0, 0]. When passing [2, 1, 0, 1] as the
input by adding a constant 1 at the first element, the output is [3, 2, 1, 1],
which matches the expected output sequence based on the original output [2, 1,
0, 0]. Therefore, metamorphic testing with this MR failed to identify this
bug.

Listing 4: Metamorphic testing by comparing whether the relative difference is
maintained for a modified polynormial

⬇

eps = 1e-10

while True :

time_sequence = get_random_sequence()

frequency_sequence = discrete_fourier_transform(time_sequence)

c = random.random()

time_sequence[0] += c

frequency_sequence_new = discrete_fourier_transform(time_sequence)

all_same = all(abs(frequency_sequence_new[i].real - frequency_sequence[i].real
- c) < eps for i in range(len(frequency_sequence)))

assert all_same

#### Retromorphic Testing

Illustrated in Listing 5, Retromorphic Testing first utilizes DFT to convert
the randomly generated time-domain input sequence,
$x_{0},x_{1},\ldots,x_{N-1}$, into a frequency-domain sequence
$X_{0},X_{1},\ldots,X_{N-1}$. Given the inherent properties of DFT and IDFT,
it is anticipated that the two time-domain sequences (i.e.,
$x_{0},x_{1},\ldots,x_{N-1}$ and
$x_{0}^{\prime},x_{1}^{\prime},\ldots,x_{N-1}^{\prime}$) hold the following
relationship:

(3) $\displaystyle x_{i}^{\prime}$ $\displaystyle=x_{i},\text{for all }0\leq
i<N.$

This interrelation can serve as the Retromorphic Relation that validates the
DFT implementation. For example, consider a time-domain sequence $x$ = [1, 0,
1, 0]. Employing DFT, we expect to acquire the corresponding frequency-domain
sequence $X$, which should ideally be [2, 0, 2, 0]. However, the
implementation shown in Listing 1 fails to yield the correct frequency-domain
sequence. Instead, it returns [2, 1, 0, 0]. The realization in Listing 5 of
Retromorphic Testing can easily detect this bug. When feeding the frequency-
domain sequence into the IDFT program, it yields a time-domain sequence
$x^{\prime}=[0.75,0.68,0.5,0.32]$. However, $x^{\prime}$ is expected to be [1,
0, 1, 0] according to the Retromorphic Relation defined in Equation 3, thereby
we have detected the bugs in the DFT realization.

In addition, we can modify the frequency-domain sequence from the DFT program
to enhance the test oracle, similar to metamorphic testing. To achieve this,
we add a constant $c$ to each element of the frequency-domain sequence to get
the modified sequence $X_{0}+c,X_{1}+c,\ldots,X_{N-1}+c$, and employ the IDFT
to revert the modified sequence $X_{0}+c,X_{1}+c,\ldots,X_{N-1}+c$ back to the
time domain, yielding the sequence
$x_{0}^{\prime}+c,x_{1}^{\prime},\ldots,x_{N-1}^{\prime}$. Therefore, the
Retromorphic Relation will be changed as:

(4) $\displaystyle x_{0}^{\prime}$ $\displaystyle=x_{0}+c,$ $\displaystyle
x_{i}^{\prime}$ $\displaystyle=x_{i},\text{for all }1\leq i<N.$

For the above example, when we add constant 1 to each element of the erroneous
frequency-domain sequence [2, 1, 0, 0] to get the modified sequence [3, 2, 1,
1] and feed the modified sequence into the IDFT program, it yields a time-
domain sequence $x^{\prime}=[1.75,0.93,0.5,0.57]$. $x^{\prime}$ is expected to
be [2, 0, 1, 0] according to the Retromorphic Relation defined in Equation 4,
thereby we have also detected the bugs in the DFT realization.

Listing 5: Retromophic testing by comparing whether the relative difference is
maintained for the inversed transformation

⬇

eps = 1e-10

while True:

time_sequence = get_random_sequence()

frequency_sequence = discrete_fourier_transform(time_sequence)

#Without Modification

time_sequence_new = discrete_fourier_transform(frequency_sequence, -1)

all_same = all(abs(time_sequence_new[i].real - time_sequence[i].real) < eps
for i in range(len(time_sequence)))

assert all_same

#With Modification

c = random.random()

for i in range(len(frequency_sequence)):

frequency_sequence[i] += c

time_sequence_new = discrete_fourier_transform(frequency_sequence, -1)

all_same = all(abs(time_sequence_new[i].real - time_sequence[i].real) < eps
for i in range(1, len(time_sequence)) and abs(time_sequence_new[0].real -
time_sequence[0].real) < eps)

assert all_same

##  4\. Testing Scopes

Table 1. Overview of six testing scopes using Retromorphic Testing

Testing Scope | Software Types | Testing Mode | Forward Program | Backward Program
---|---|---|---|---
Integer factorization | Algorithm | Forward | Pollard’s rho* | Integer multiplication
Infix, prefix, and postfix transformation | Algorithm | Integrated | Expression transformation (Prefix$\rightarrow$Postfix)* | Expression transformation (Postfix$\rightarrow$Prefix)*
Database Management System | Conventional software | Backward | Query generator | Database system*
Java decompiler | Conventional software | Backward | Java compiler | Java decompiler*
Machine translation | AI application | Integrated | Google Translate (EN$\rightarrow$ZH-CN)* | Google Translate (ZH-CN$\rightarrow$EN)*
Image captioning and generation | AI application | Backward | DALL-E2 | Azure’s image captioning*

  * •

‘*’ indicates the target system under test.

In this section, we present six instances of Retromorphic Testing techniques
across three testing scopes, with two examples provided for each testing
scope. Table 1 shows the statistics. We give one example of forward mode,
three examples of backward mode, and two examples of integrated mode.

#### Example 1: Integer factorization

Integer factorization is the process of finding the prime numbers that
multiply together to give a certain integer. This is a fundamental concept in
number theory and has various applications in both mathematics and computer
science such as cryptography, primality testing, cryptanalysis, digital
signatures, codebreaking, and computational complexity (Rivest et al., 1978;
Agrawal et al., 2004; Shor, 1994). The most straightforward way to factorize
an integer is to test all the prime numbers not greater than the square root
of it, which requires a time complexity of $O(\sqrt{N})$, where $N$ is the
input integer. However, the integer that needs to be factorized is usually
extremely large in practice, making this algorithm unacceptable. To
effectively factorize large integers, programs for practical applications are
much more complicated than straightforward algorithms.

Pollard’s rho is one of the effective algorithms for integer factorization. As
shown in Listing 6’s Python implementation, it employs a divide-and-conquer
approach by iteratively selecting a factor of $N$, denoted as $d$. Then, the
procedure is divided into two sub-processes for $\frac{N}{d}$ and $d$
respectively, continuing this division until it identifies a prime number.
However, in this program, the calculation of the selected factor $d$ is
erroneously implemented as gcd(abs(x - y), x), instead of the correct version
gcd(abs(x - y), n).

To test this program, we can utilize Retromorphic Testing based on the fact
that prime numbers multiply together to the given integer. Assume $N$ is the
input integer to be factorized, and $p_{0},p_{1},\ldots,p_{k}$ are the correct
output prime numbers after factorization. The test oracle can be formulated as
$N=\prod\limits_{i=0}^{k}p_{i}$, which is demonstrated in Listing 7. For
example, when taking $12$ as the input integer, the correct factorization
should be $[2,2,3]$. However, the implementation in Listing 6 returns
$[2,2,2]$. When multiply these numbers together, we get the result of $8$,
rather than the original input $12$, breaking the Retromorphic Relation.

Listing 6: Pollard’s rho algorithm for integer factorization

⬇

import random

def gcd(a, b):

while b:

a, b = b, a % b

return a

def pollards_rho(n):

if n == 1:

return []

if n % 2 == 0:

return [2] + pollards_rho(n // 2)

def rho(x, c):

return (x * x + c) % n

x = random.randint(1, n - 1)

y = x

c = random.randint(1, n - 1)

d = 1

while d <= 1:

x = rho(x, c)

y = rho(rho(y, c), c)

d = gcd(abs(x - y), x) # Bug: should be gcd(abs(x - y), n)

if d == n:

return [n]

else:

return pollards_rho(d) + pollards_rho(n // d)

Listing 7: Retromorphic Testing for integer factorization

⬇

while(True):

N = get_random_integer()

factors = pollards_rho(N)

result = 1

for prime in factors:

result *= prime

assert(result == N)

Listing 8: Code for transforming between prefix and postfix notation

⬇

def is_operator(char):

return char in "+-*/"

def postfix_to_prefix(postfix_expression):

stack = []

for char in postfix_expression:

if char.isalnum(): # Operand

stack.append(char)

elif is_operator(char):

operand1 = stack.pop()

operand2 = stack.pop() #swap operand1 and operand2 for correct version

prefix = f"{char}{operand1}{operand2}"

stack.append(prefix)

return stack[0]

def prefix_to_postfix(prefix_expression):

stack = []

for char in reversed(prefix_expression):

if char.isalnum(): #Operand

stack.append(char)

elif is_operator(char):

operand1 = stack.pop()

operand2 = stack.pop()

postfix = f"{operand1}{operand2}{char}"

stack.append(postfix)

return stack[0]

Listing 9: Retromophic testing for transforming between prefix and postfix
notation

⬇

postfix_expression = get_random_postifx()

assert(postfix_expression ==
prefix_to_postfix(postfix_to_prefix(postfix_expression)))

#### Example 2: Infix, prefix, and postfix transformation

The Abstract Syntax Tree (AST) is a tree-like data structure employed to
represent the syntactic information of source code within programming
languages. For example, when considering the arithmetic expression 5 + 6 * a,
the corresponding AST can be denoted by the construct Operator(‘+’,
constant(5), Operator(‘*’, constant(6), a)). The process of rendering an AST
into a comprehensible format frequently involves adopting infix, prefix, or
postfix notations. These notations dictate the positioning of operators
concerning their operands in binary operations, whether they appear between,
before, or after the operands. For example, 5 + 6 * a is the infix notation of
the arithmetic expression, \+ 5 * 6 a and 5 6 a * + are the corresponding
prefix and postfix notation.

Transformations between different notations are used by different tasks in
practice. Listing 8 shows a Python implementation of transformation between
postfix notation and prefix notation as postfix_to_prefix and
prefix_to_postfix. In its implementation of postfix_to_prefix, the positions
of operand1 and operand2 are erroneously exchanged.

Similar to Integer factorization, test oracles for this program can also be
derived from Retromorphic Testing. As shown in Listing 9, we can verify it
based on the principle that if we convert a given postfix notation expression
to its equivalent prefix notation and then convert the result back to postfix
notation, it should match the original expression. The test oracle can be
formulated as Equation 5, where $S$ denotes a postfix notation.

(5) $S=\texttt{prefix\\_to\\_postfix(postfix\\_to\\_prefix(}S\texttt{))}$

For example, if we input the postfix notation 5 6 a * + into
postfix_to_prefix, it will return a prefix notation \+ * a 6 5. When
converting this result back through prefix_to_postfix, the returned postfix
notation is a 6 * 5 +, which is different from the original expression 5 6 a *
+, breaking the Retromorphic Relation.

#### Example 3: Database Management System

Figure 2. An illustrative example about using Retromorphic Testing to test
database systems

In the context of Database Management Systems (DBMS), Rigger and Su introduced
the concept of Pivoted Query Synthesis (PQS) (Rigger and Su, 2020c) as an
effective testing approach to uncover logic bugs in these systems. Although
the original paper describes PQS as a testing technique for a specific system,
we think it can be regarded as an instance of the general Retromorphic Testing
methodology. We also discussed with one author of PQS and he agrees that PQS
should be categorized as a Retromorphic Testing technique.

As depicted in Fig.2, the target database serves as the backward program $Q$
in Retromorphic Testing. The PQS testing approach initiates with the
generation of a pivot row (e.g., [t0.c0: 3, t0.c1: TRUE, t1.c0: -5] in Fig.2),
which is in the modality of rows. Subsequently, a simple program $P$ based on
AST will be utilized to generate an SQL query based on the pivot row, ensuring
that the data of the pivot row is retrieved by the query. This query exists in
the modality of SQL queries and is then passed to the target database,
reverting the modality back into rows. The underlying Retromorphic Relation
here asserts that the result obtained from the database should encompass the
pivot row.

Figure 3. An illustrative example about using Retromorphic Testing to test
decompiler.

#### Example 4: Java decompiler

While the Java compiler is renowned for its reliability, Java decompilers
occasionally exhibit susceptibility to errors or inaccuracies during the
transformation of compiled bytecode back into human-readable source code.
These discrepancies may arise due to the intricate nature of bytecode
optimization, the diversity of Java language constructs, and the inherent
complexity of decompilation. There is an example where we adopt Retromorphic
Testing for testing the Java decompiler, as shown in Fig 3. As an
illustration, we use $Comp$ to denote the Java compiler, and $DeComp$ to
denote the Java decompiler. In this example, we use javac 17.0.8 as the Java
compiler and CFR 0.150 as the Java
decompiler.111https://www.benf.org/other/cfr/ We compile the input Java code
$M_{1}$ into binary code $M_{2}$ and decompile $M_{2}$ back to Java code
$M_{1}^{\prime}$. The test oracle between $M_{1}$ and $M_{1}^{\prime}$ is
their execution output. In the example, both Java code $M_{1}$ and
$M_{1}^{\prime}$ outputs “Sum:12”, thus the decompiler works properly in this
specific test case.

#### Example 5: Machine translation

Figure 4. An illustrative example about using Retromorphic Testing to test
Google Translation.

Machine translation software aims to fully automate the process of translating
text from a source language into a target language. Machine translation
software often provides support for multiple languages. Fig. 4 shows using
Retromorphic Testing for testing Google Translate, where $P$ is defined as the
program that translates English to Chinese, and $Q$ is the program that
translates Chinese to English. Intuitively, if we translate an English
sentence $M_{1}$ into a Chinese sentence $M_{2}$ and then translate it back to
an English sentence $M_{1}^{\prime}$, the meaning of $M_{1}$ and
$M_{1}^{\prime}$ should remain unchanged. We adopt a Retromorphic Relation
that the proper nouns of the two sentences should be identical, which is
defined below:

(6) $PN(M_{1})=PN(M_{1}^{\prime}),$

where $PN(M)$ represents the proper nouns of the sentence $M$. In the
realization of this case, we adopt NLTK (Bird and Loper, 2004), a widely
adopted NLP toolkit for extracting the proper nouns of the sentence, and get
the results that $PN(M_{1})$ is $\\{$Mike, Jack$\\}$, while
$PN(M_{1}^{\prime})$ is $\\{$Mike, Jack, EU$\\}$. Therefore, the Retromorphic
Relation is not followed and there is a translating issue in Google Translate.
A similar approach has been explored by Gao et al. (Gao et al., 2023).

#### Example 6: Image captioning and generation

Figure 5. An illustrative example about using Retromorphic Testing to test
Image generation and captioning.

Image captioning and generation play pivotal roles in computer vision and
natural language processing. Image captioning involves crafting textual
depictions for images, while image generation entails crafting novel images
based on textual descriptions. We could adopt Retromorphic Testing for testing
the image captioning system by using the image generation system as the
auxiliary program, as shown in Fig 5. We first generate a sentence $M_{1}$ as
the input for the image generation system
(DALL-E2222https://openai.com/dall-e-2/) to generate an image $M_{2}$. Then we
reduce the brightness by $20\%$ and increase the contrast ratio by $10\%$ to
obtain $M_{2}^{\prime}$, which should not change the interpretation of the
image. After that, we use an image captioning system (i.e., Microsoft Azure
Cognitive Services333https://azure.microsoft.com/en-gb/products/cognitive-
services/vision-services/) to generate a caption sentence $M_{1}^{\prime}$ for
$M_{2}^{\prime}$. We identify a Retromorphic Relation that the semantic of
$M_{1}^{\prime}$ should not deviate much from $M_{1}$, and use the text
similarity to implement the test oracle, which should be higher than a
threshold. In practice, we use SimCSE (Gao et al., 2021) for calculating the
similarity between the sentences, and set the threshold as 0.6. In this case,
$SimCSE(M_{1},M_{1}^{\prime})$ is 0.129 and lower than the threshold, which
violates the Retromorphic Relation. After manual inspection, we could see that
the “airplane” within image $M_{2}$ is recognized as “cylinder” in
$M_{1}^{\prime}$. This mismatch reflects an image captioning error.

##  5\. Challenges and Future Directions

Despite the notable advantages of Retromorphic Testing, certain challenges
need to be addressed. First, the effectiveness of the test oracle highly
depends on the auxiliary program, whose design demands a certain level of
effort and favors developers who have a comprehensive understanding of the
underlying systems. Second, the creation of effective Retromorphic Relations
is challenging. As depicted in Fig. 1 (c), Retromorphic Testing involves the
transition between two modalities, allowing for data modification in the
second modality. The effective design of test oracles requires a comprehensive
consideration of both the modality transition and the alterations occurring in
the second modality.

Future research can explore auxiliary program selection and construction,
refining their ability to accurately reverse software outputs to original
input modalities. With the advancement of AI techniques such as Large Langauge
Models, there is an increasing potential for uncovering a greater number of
untapped Retromorphic Relations.

##  6\. Conclusion

This paper introduces Retromorphic Testing, a new black-box testing
methodology for tackling the oracle problem. The core insight is to leverage
an auxiliary program, form a dual-program structure in one of the three modes
(i.e., forward, backward, integrated), use the (optionally transformed) output
of the forward program as the input of the backward program, and examine the
retromorphic relation between the initial input and the output of the backward
program in the input domain. The testing scopes of Retromorphic Testing have
been discussed in six testing instances ranging from integer factorization
algorithm, database management system, to machine translators, demonstrating
the wide applicability of the proposed concept. We believe Retromorphic
Testing is a comparable methodology to differential testing and metamorphic
testing and they complement with each other.

##  7\. Data Availability

Codes for this paper are available at an anonymous
link.444https://github.com/CUHK-Shenzhen-SE/RetromorphicTesting

## References

  * (1)
  * Abreu et al. (2022) Rui Abreu, João Paulo Fernandes, Luis Llana, and Guilherme Tavares. 2022.  Metamorphic Testing of Oracle Quantum Programs. In _2022 IEEE/ACM 3rd International Workshop on Quantum Software Engineering (Q-SE)_. 16–23.  https://doi.org/10.1145/3528230.3529189
  * Agrawal et al. (2004) Manindra Agrawal, Neeraj Kayal, and Nitin Saxena. 2004\.  PRIMES Is in P.  _Annals of Mathematics_ 160, 2 (2004), 781–793.  http://www.jstor.org/stable/3597229
  * Barr et al. (2015) Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015.  The Oracle Problem in Software Testing: A Survey.  _IEEE Transactions on Software Engineering_ 41, 5 (may 2015), 507–525.  https://doi.org/10.1109/TSE.2014.2372785
  * Bird and Loper (2004) Steven Bird and Edward Loper. 2004.  NLTK: The Natural Language Toolkit. In _Proceedings of the ACL Interactive Poster and Demonstration Sessions_. Association for Computational Linguistics, Barcelona, Spain, 214–217.  https://aclanthology.org/P04-3031
  * Chen et al. (1998) Tsong Y Chen, Shing C Cheung, and Shiu Ming Yiu. 1998\.  Metamorphic testing: a new approach for generating next test cases.  (1998).  https://arxiv.org/abs/2002.12543
  * Chen et al. (2018) Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, T. H. Tse, and Zhi Quan Zhou. 2018.  Metamorphic Testing: A Review of Challenges and Opportunities.  _ACM Comput. Surv._ 51, 1, Article 4 (jan 2018), 27 pages.  https://doi.org/10.1145/3143561
  * Chen et al. (2019) Yuting Chen, Ting Su, and Zhendong Su. 2019.  Deep Differential Testing of JVM Implementations. In _Proceedings of the 41st International Conference on Software Engineering_ (Montreal, Quebec, Canada) _(ICSE 2019)_. IEEE Press, 1257–1268.  https://doi.org/10.1109/ICSE.2019.00127
  * Chen et al. (2016) Yuting Chen, Ting Su, Chengnian Sun, Zhendong Su, and Jianjun Zhao. 2016.  Coverage-Directed Differential Testing of JVM Implementations. In _Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation_ (Santa Barbara, CA, USA) _(PLDI 2016)_. Association for Computing Machinery, New York, NY, USA, 85–99.  https://doi.org/10.1145/2908080.2908095
  * Deng et al. (2023) Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. 2023\.  Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models.  arXiv:2212.14834 [cs.SE] https://arxiv.org/abs/2212.14834
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple Contrastive Learning of Sentence Embeddings. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 6894–6910.  https://doi.org/10.18653/v1/2021.emnlp-main.552
  * Gao et al. (2023) Wentao Gao, Jiayuan He, and Van-Thuan Pham. 2023.  Metamorphic Testing of Machine Translation Models using Back Translation. In _2023 IEEE/ACM International Workshop on Deep Learning for Testing and Testing for Deep Learning (DeepTest)_. 1–8.  https://doi.org/10.1109/DeepTest59248.2023.00008
  * Howden (1978) W.E. Howden. 1978\.  Theoretical and Empirical Studies of Program Testing.  _IEEE Transactions on Software Engineering_ SE-4, 4 (1978), 293–298.  https://doi.org/10.1109/TSE.1978.231514
  * Kamm et al. (2023) Matteo Kamm, Manuel Rigger, Chengyu Zhang, and Zhendong Su. 2023.  Testing Graph Database Engines via Query Partitioning. In _Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis_ (Seattle, WA, USA) _(ISSTA 2023)_. Association for Computing Machinery, New York, NY, USA, 140–149.  https://doi.org/10.1145/3597926.3598044
  * Le et al. (2014) Vu Le, Mehrdad Afshari, and Zhendong Su. 2014.  Compiler Validation via Equivalence modulo Inputs. In _Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation_ (Edinburgh, United Kingdom) _(PLDI 2014)_. Association for Computing Machinery, New York, NY, USA, 216–226.  https://doi.org/10.1145/2594291.2594334
  * Li et al. (2023) Cong Li, Yanyan Jiang, Chang Xu, and Zhendong Su. 2023\.  Validating JIT Compilers via Compilation Space Exploration. In _Proceedings of the 29th Symposium on Operating Systems Principles_ (Koblenz, Germany) _(SOSP 2023)_. Association for Computing Machinery, New York, NY, USA, 66–79.  https://doi.org/10.1145/3600006.3613140
  * McKeeman (1998) William M. McKeeman. 1998\.  Differential Testing for Software.  _DIGITAL TECHNICAL JOURNAL_ 10, 1 (1998), 100–107.  https://api.semanticscholar.org/CorpusID:14018070
  * Paltenghi and Pradel (2023) Matteo Paltenghi and Michael Pradel. 2023.  MorphQ: Metamorphic Testing of the Qiskit Quantum Computing Platform. In _2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)_ _(ICSE 2023)_. 2413–2424.  https://doi.org/10.1109/ICSE48619.2023.00202
  * Pezzè and Zhang (2014) Mauro Pezzè and Cheng Zhang. 2014.  Chapter One - Automated Test Oracles: A Survey.  Advances in Computers, Vol. 95. Elsevier, 1–48.  https://doi.org/10.1016/B978-0-12-800160-8.00001-2
  * Rigger and Su (2020a) Manuel Rigger and Zhendong Su. 2020a.  Detecting Optimization Bugs in Database Engines via Non-Optimizing Reference Engine Construction. In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Virtual Event, USA) _(ESEC/FSE 2020)_. Association for Computing Machinery, New York, NY, USA, 1140–1152.  https://doi.org/10.1145/3368089.3409710
  * Rigger and Su (2020b) Manuel Rigger and Zhendong Su. 2020b.  Finding Bugs in Database Systems via Query Partitioning.  _Proc. ACM Program. Lang._ 4, OOPSLA, Article 211 (nov 2020), 30 pages.  https://doi.org/10.1145/3428279
  * Rigger and Su (2020c) Manuel Rigger and Zhendong Su. 2020c.  Testing Database Engines via Pivoted Query Synthesis. In _14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2020)_. USENIX Association, 667–682.  https://www.usenix.org/conference/osdi20/presentation/rigger
  * Rigger and Su (2022) Manuel Rigger and Zhendong Su. 2022.  Intramorphic Testing: A New Approach to the Test Oracle Problem. In _Proceedings of the 2022 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software_ (Auckland, New Zealand) _(Onward! 2022)_. Association for Computing Machinery, New York, NY, USA, 128–136.  https://doi.org/10.1145/3563835.3567662
  * Rivest et al. (1978) R. L. Rivest, A. Shamir, and L. Adleman. 1978.  A Method for Obtaining Digital Signatures and Public-Key Cryptosystems.  _Commun. ACM_ 21, 2 (feb 1978), 120–126.  https://doi.org/10.1145/359340.359342
  * Segura et al. (2016) Sergio Segura, Gordon Fraser, Ana B. Sanchez, and Antonio Ruiz-Cortés. 2016.  A Survey on Metamorphic Testing.  _IEEE Transactions on Software Engineering_ 42, 9 (2016), 805–824.  https://doi.org/10.1109/TSE.2016.2532875
  * Sharma et al. (2023) Mayank Sharma, Pingshi Yu, and Alastair F. Donaldson. 2023\.  RustSmith: Random Differential Compiler Testing for Rust. In _Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis_ (Seattle, WA, USA) _(ISSTA 2023)_. Association for Computing Machinery, New York, NY, USA, 1483–1486.  https://doi.org/10.1145/3597926.3604919
  * Shor (1994) P.W. Shor. 1994\.  Algorithms for quantum computation: discrete logarithms and factoring. In _Proceedings 35th Annual Symposium on Foundations of Computer Science_. 124–134.  https://doi.org/10.1109/SFCS.1994.365700
  * Slutz (1998) Donald R. Slutz. 1998\.  Massive Stochastic Testing of SQL. In _Proceedings of the 24rd International Conference on Very Large Data Bases_ _(VLDB 1998)_. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 618–622.  https://dl.acm.org/doi/10.5555/645924.671199
  * Su et al. (2021) Ting Su, Yichen Yan, Jue Wang, Jingling Sun, Yiheng Xiong, Geguang Pu, Ke Wang, and Zhendong Su. 2021\.  Fully Automated Functional Fuzzing of Android Apps for Detecting Non-Crashing Logic Bugs.  _Proc. ACM Program. Lang._ 5, OOPSLA, Article 156 (oct 2021), 31 pages.  https://doi.org/10.1145/3485533
  * Sun et al. (2021) Jingling Sun, Ting Su, Junxin Li, Zhen Dong, Geguang Pu, Tao Xie, and Zhendong Su. 2021.  Understanding and Finding System Setting-Related Defects in Android Apps. In _Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis_ (Virtual, Denmark) _(ISSTA 2021)_. Association for Computing Machinery, New York, NY, USA, 204–215.  https://doi.org/10.1145/3460319.3464806
  * Sun et al. (2020) Zeyu Sun, Jie M. Zhang, Mark Harman, Mike Papadakis, and Lu Zhang. 2020.  Automatic Testing and Improvement of Machine Translation. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ (Seoul, South Korea) _(ICSE 2020)_. Association for Computing Machinery, New York, NY, USA, 974–985.  https://doi.org/10.1145/3377811.3380420
  * Usman et al. (2021) Muhammad Usman, Wenxi Wang, and Sarfraz Khurshid. 2021\.  TestMC: Testing Model Counters Using Differential and Metamorphic Testing. In _Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering_ (Virtual Event, Australia) _(ASE 2020)_. Association for Computing Machinery, New York, NY, USA, 709–721.  https://doi.org/10.1145/3324884.3416563
  * Wang and Su (2020) Shuai Wang and Zhendong Su. 2020.  Metamorphic Object Insertion for Testing Object Detection Systems. In _Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering_ (Virtual Event, Australia) _(ASE 2020)_. Association for Computing Machinery, New York, NY, USA, 1053–1065.  https://doi.org/10.1145/3324884.3416584
  * Wang et al. (2023) Wenxuan Wang, Jingyuan Huang, Chang Chen, Jiazhen Gu, Jianping Zhang, Weibin Wu, Pinjia He, and Michael Lyu. 2023\.  Validating Multimedia Content Moderation Software via Semantic Fusion. In _Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis_ (Seattle, WA, USA) _(ISSTA 2023)_. Association for Computing Machinery, New York, NY, USA, 576–588.  https://doi.org/10.1145/3597926.3598079
  * Winterer et al. (2020) Dominik Winterer, Chengyu Zhang, and Zhendong Su. 2020\.  Validating SMT Solvers via Semantic Fusion. In _Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation_ (London, UK) _(PLDI 2020)_. Association for Computing Machinery, New York, NY, USA, 718–730.  https://doi.org/10.1145/3385412.3385985
  * Yang et al. (2023) Chenyuan Yang, Yinlin Deng, Jiayi Yao, Yuxing Tu, Hanchi Li, and Lingming Zhang. 2023\.  Fuzzing Automatic Differentiation in Deep-Learning Libraries. In _Proceedings of the 45th International Conference on Software Engineering_ (Melbourne, Victoria, Australia) _(ICSE 2023)_. IEEE Press, 1174–1186.  https://doi.org/10.1109/ICSE48619.2023.00105
  * Yu et al. (2023a) Boxi Yu, Yiyan Hu, Qiuyang Mang, Wenhan Hu, and Pinjia He. 2023a.  Automated Testing and Improvement of Named Entity Recognition Systems. In _Proceedings of the 31st ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ _(ESEC/FSE 2023)_.
  * Yu et al. (2023b) Boxi Yu, Zhiqing Zhong, Jiaqi Li, Yixing Yang, Shilin He, and Pinjia He. 2023b.  ROME: Testing Image Captioning Systems via Recursive Object Melting. In _Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis_ (Seattle, WA, USA) _(ISSTA 2023)_. Association for Computing Machinery, New York, NY, USA, 766–778.  https://doi.org/10.1145/3597926.3598094
  * Zheng et al. (2022) Yingying Zheng, Wensheng Dou, Yicheng Wang, Zheng Qin, Lei Tang, Yu Gao, Dong Wang, Wei Wang, and Jun Wei. 2022\.  Finding Bugs in Gremlin-Based Graph Database Systems via Randomized Differential Testing. In _Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis_ (Virtual, South Korea) _(ISSTA 2022)_. Association for Computing Machinery, New York, NY, USA, 302–313.  https://doi.org/10.1145/3533767.3534409