marginparsep has been altered.
topmargin has been altered.
marginparpush has been altered.
The page layout violates the ICML style. Please do not change the page layout,
or include packages like geometry, savetrees, or fullpage, which change it for
you. We’re not able to reliably undo arbitrary changes to the style. Please
remove the offending package(s), or layout-changing commands and try again.

UniTSyn: A Large-Scale Dataset Capable of Enhancing the Prowess of Large
Language Models for Program Testing

Yifeng He 1  Jiabo Huang 2  Yuyang Rong 1  Yiwen Guo 3  Ethan Wang 1  Hao Chen
1

††footnotetext:  1UC Davis 2Tencent Security Big Data Lab 3Independent
Researcher. Correspondence to: Yiwen Guo<EMAIL_ADDRESS>

###### Abstract

The remarkable capability of large language models (LLMs) in generating high-
quality code has drawn increasing attention in the software testing community.
However, existing code LLMs often demonstrate unsatisfactory capabilities in
generating accurate and complete tests since they were trained on code
snippets collected without differentiating between code for testing purposes
and other code. In this paper, we present a large-scale dataset UniTSyn, which
is capable of enhancing the prowess of LLMs for Unit Test Synthesis.
Associating tests with the tested functions is crucial for LLMs to infer the
expected behavior and the logic paths to be verified. By leveraging Language
Server Protocol, UniTSyn achieves the challenging goal of collecting focal-
test pairs without per-project execution setups or per-language heuristics
that tend to be fragile and difficult to scale. It contains 2.7 million focal-
test pairs across five mainstream programming languages, making it possible to
be utilized for enhancing the test generation ability of LLMs. The details of
UniTSyn can be found in Table 1. Our experiments demonstrate that, by building
an autoregressive model based on UniTSyn, we can achieve significant benefits
in learning and understanding unit test representations, resulting in improved
generation accuracy and code coverage across all evaluated programming
languages. Code and data will be publicly available.

##  1 Introduction

Figure 1: UniTSyn workflow overview. We download repositories via the GitHub
API and extract their unit tests. The test functions are statically analyzed
to identify the call to their focal functions. UniTSyn sends JSON remote
procedure calls (RPCs) to language servers to get the location of the focal
function definition. Finally, we store the aligned function-level focal-test
pairs as training data.

Software testing is a crucial yet labor-intensive part of the software
development process Zhao et al. (2017); Beller et al. (2015), whose importance
on early detection of program defects has been established for decades
Dijkstra (1972). Recently, machine learning models, especially large language
models (LLMs), have shown their prowess in composing high-quality code Li et
al. (2023); Rozière et al. (2023), which has further fueled the software
testing community’s interest Schäfer et al. (2023); Rao et al. (2023) in
applying LLMs to unit test generation or other program testing applications.

Test generation is a more challenging task than conventional programming
synthesis, as the model not only needs executable code snippets but also
precise predictions of input-output values. Equipping off-the-shelf code LLMs
Wang et al. (2023); Zheng et al. (2023); Nijkamp et al. (2023a) with prompts
or instructions for test generation is poorly suited, as demonstrated by Rao
et al. There have been a few recent projects Nie et al. (2023); Rao et al.
(2023) targeting training testing-specific LLMs on code corpus that is closely
related to testing. A desired training set for this aim should include a large
number of test functions paired with their tested source functions (also
called _focal functions_ by Tufano et al. and Schäfer et al.).

However, there are significant challenges surrounding the automation of
collecting focal-test pairs since real-world projects do not have to follow a
consistent structure. Existing efforts either rely on dynamic analysis or
heuristics to locate focal functions Dinella et al. (2022); Nie et al. (2023);
Alagarsamy et al. (2023); Tufano et al. (2021) or find the coarse-grained
correspondence between tests and focal functions at the file level Rao et al.
(2023). The former category is _difficult to scale_ across programming
languages, which inhibits the development of universal testing models. On the
other hand, the latter category is limited by _weak focal-test
correspondences_ , which hamper the models’ capability to properly comprehend
the expected behavior and logic paths being verified. These challenges
underscore the need for more effective, scalable, and language-agnostic
approaches to collect pairwise focal-test data in an effort to fully unleash
the potential of LLMs on software testing.

We present UniTSyn in this paper as a multilingual dataset capable of
enhancing LLMs for Unit Test Synthesis. As shown in Figure 1, we integrated
the Language Server Protocol (LSP) into the dataset-building process to
harness its language extensibility and call-definition matching ability. This
substantially eases the difficulty of implementing dependency analysis
heuristics for different languages and executing different projects for
dynamic analysis. Moreover, we designed a flexible, unified static analyzer to
find calls to focal functions from the unit tests, which decreases the
complexity of performing call analysis for each language.

Table 1: Dataset statistics. The “Framework” column indicates the static
analysis we based on for test extraction. #proj is the number of projects we
found on GitHub for each language, and #pairs is the number of focal-test
pairs we collected for each language.

Language | Framework | #proj | #pairs
---|---|---|---
Python | unittest,pytest | 43,848 | 1,218,311
Java | JUnit | 25,488 | 1,097,518
Go | testing | 38,097 | 361,075
C++ | GoogleTest | 20,090 | 25,513
JavaScript | MochaJS | 17,621 | 13,293

To explore the quality of the UniTSyn dataset, we further trained an
autoregressive model called UniTester to synthesize tests in different
programming languages. UniTSyn yielded non-negligible performance advantages
on generating accurate and complete tests over several up-to-date LLMs
intended for both code and test synthesis, which demonstrates the importance
of our proposed UniTSyn as a flexible dataset collection framework.

To sum up, we make the following contributions: (1) We present a large-scale
dataset of 2.7M focal-test function pairs across five commonly used
programming languages, which will be useful in advancing the field of software
engineering through LLM coding assistance. (2) We release our generic and
easily applicable approach for building multilingual unit test datasets with
function-level focal-test alignment. Our approach is extendable to any
language that has a mature implementation of LSP, allowing the LLM to broaden
its testing capabilities in more diverse software engineering scenarios. (3)
We validated the quality of UniTSyn by training an autoregressive language
model on it. The more accurate and complete tests generated by our model
compared to existing test and code LLMs demonstrate the necessity and benefits
of training with explicit correspondence between tests and focal functions for
multilingual test generation.

##  2 Related Work

###  2.1 Code Understanding and Generation

The application of Machine Learning (ML) in Software Engineering (SE) has
gained significant attention recently, particularly with the development of
LLMs. LLMs can help SE in various ways, including code generation Chen et al.
(2021); Li et al. (2022); Nijkamp et al. (2023b); Li et al. (2023); Rozière et
al. (2023); Wang et al. (2021), code summarization Iyer et al. (2016); LeClair
et al. (2019); Ahmad et al. (2020), and code classification Feng et al.
(2020); Guo et al. (2022); Zhao et al. (2023); Huang et al. (2023). To
facilitate the study of ML for SE, a variety of datasets have been built.
These datasets are either collected from competitive programming contests,
like POJ104 in the CodeXGlue benchmark Lu et al. (2021) and CodeNet Puri et
al. (2021), or open-source software like the CodeSearchNet challenge Husain et
al. (2020) and CoderEval Yu et al. (2023). Datasets specialized for evaluating
the code generation performance of LLMs like HumanEval Chen et al. (2021) and
HumanEval-X Zheng et al. (2023) have also been established using algorithmic
coding problems formatted just like on LeetCode. These datasets are built for
general-purpose program synthesis like CodeT5+ Wang et al. (2023), CodeGen2
Nijkamp et al. (2023a), InCoder Fried et al. (2022), and SantaCoder Allal et
al. (2023). However, as shown by Zhao et al. and Huang et al., test cases can
greatly help improve the model’s ability to understand code. With few datasets
focusing on test cases, the necessity of UniTSyn is justified.

###  2.2 Unit Testing

Unit testing is a common self-assessment testing technique where developers
use a set of inputs and outputs of their code to validate that the code is
working as expected Zhu et al. (1997). In this setting, a test case consists
of an input and the corresponding output after the code execution.

To evaluate the completeness and comprehensiveness of test cases, code
coverage is often used as a common metric Yang et al. (2006); Nagappan & Ball
(2010). Code coverage measures the percentage of the code that is executed.
Statement, line, and branch coverage are often used depending on the
coarseness of the testing requirements. Code coverage is measured because
executing a piece of code is the necessary condition for finding bugs in it
Nagappan & Ball (2010). In SE, coverage-guided software testing has also shown
its power in detecting bugs in various software domains Chen et al. (2016;
2019); Serebryany (2016); Chen & Chen (2018); Fioraldi et al. (2020); Rong et
al. (2020; 2024), highlighting the importance of this metric. In general, unit
testing is an indispensable part of the modern software development ecosystem,
ensuring the quality and reliability of systems across various domains.
Therefore, constructing test cases with high coverage is a necessity.

###  2.3 Software Testing via Machine Learning

The goal of test generation is to utilize ML models to aid software testing,
which can be achieved via prompting or instructing general-purpose code LLMs
Wang et al. (2023); Zheng et al. (2023); Nijkamp et al. (2023a), or training
testing-specific LLMs. ATLAS Watson et al. (2020), AthenaTest Tufano et al.
(2021), TOGA Dinella et al. (2022), A3Test Alagarsamy et al. (2023), TeCo Nie
et al. (2023), and CAT-LM Rao et al. (2023) are testing-specific LLMs based on
the transformer architecture. Some testing-specific LLMs are trained on large-
scale test functions and their aligned focal functions Nie et al. (2023); Rao
et al. (2023), where focal functions are the functions being tested Tufano et
al. (2021); Schäfer et al. (2023). To align test and focal functions, some
work relies on dynamic execution context or heuristics for locating focal
functions Dinella et al. (2022); Nie et al. (2023); Alagarsamy et al. (2023);
Tufano et al. (2021). The extensibility of this approach is limited, since
automating the setup of dynamic execution for different projects is
challenging even within the same language. For example, TeCo’s execution-based
data collection was only applied to 1270 Java projects, which is one of the
easiest languages for cross-platform execution thanks to the Java Virtual
Machine (JVM) and the Maven build system. The build system sets up the project
in a way that makes it easy to automate the execution. Extending this method
to other languages would likely be prone to complications not present in TeCo.
On the other hand, relaxing the alignment at file-level Rao et al. (2023) is
easier to scale up than the previous approach. However, this weak focal-test
correspondence disrupts the models’ ability to thoroughly understand the
expected behavior and logic paths in the focal function.

##  3 Design of UniTSyn Dataset

###  3.1 Challenges

As the interest in using LLMs for test generation grows, the inherent
limitations of their underlying datasets are becoming increasingly apparent.
Models like CAT-LM Rao et al. (2023), specifically designed for test
generation, suffer from a lack of flexibility to adapt to various languages
due to dataset constraints. In contrast, LLMs trained on general-purpose code
datasets like SantaCoder Allal et al. (2023) can incorporate new languages by
pulling code from platforms like GitHub. However, their training data often
lacks a crucial link between the test functions and the focal functions.
Without this correspondence, the models face challenges in deducing the
intended behavior and logic paths of focal functions when generating tests.
Consequently, this leads to tests that are not precise or thorough. we aim to
build a dataset that emphasizes unit tests and their corresponding focal
functions while being multilingual. This leads to two major challenges. First,
analyzing tests for their focal function calls requires domain knowledge and
language-specific rules since different languages have distinct grammars and
unique syntax for unit tests. Second, extracting precise focal function
definitions from the dependency graph is labor-intensive and unique to each
language. To overcome these challenges, we abstracted the differences in
languages away from the static analysis pipeline to build a generalized static
call analyzer that operates on the Abstract Syntax Tree (AST) of different
languages, and integrate the existing language servers to locate the focal
function definition via their dependency analysis. This design not only
reduces the difficulty of analyzing calls across different programming
languages but also eliminates the requirement of implementing several
dependency analyses or setting up multiple execution environments.

###  3.2 Data Collection

Our dataset contains a large assemblage of data collected from open-source
software. We used CAT-LM’s Rao et al. (2023) approach to locate repositories
on GitHub that are under active development. More specifically, we mined
repositories in Python, Java, Go, C++, and JavaScript that have more than 10
stars and new commits after Jan 1st, 2020 to extend CodeSearchNet Husain et
al. (2020). In addition, we filtered out repositories that are archived,
forked, or mirrored.

###  3.3 Dataset Construction

Providing more fine-grained data with multilingual focal-test alignment is
useful to machine learning for understanding the implementation of focal
functions Dinella et al. (2022); Nie et al. (2023); Alagarsamy et al. (2023);
Tufano et al. (2021). We designed UniTSyn to achieve this goal. The frontend
identifies potential test functions and locates the focal functions. The
backend will then retrieve the source code of the focal function from the
repository codebase. Figure 1 shows an overview of the workflow for obtaining
our dataset UniTSyn.

Figure 2: Test collection phase of UniTSyn frontend. The highlighted words are
identifiers that we use to locate testing functions.

#### Parsing.

To construct a multilingual unit test dataset, we need to generalize the
static analysis process among different languages. The first step in any
static analysis is to parse the source code into AST, which can be achieved
using each language’s compiler or interpreter. To ease the differences in
invoking different tools for different languages, we selected tree-sitter tre
as the backbone of our parsing process. Tree-sitter can parse any programming
language that has a formal syntax definition. Using the tree-sitter, we
designed an easily extendable parsing and AST interface to enable smooth
static analysis for all languages.

#### Test Function Identification.

We traverse ASTs to find the test functions. To mitigate the differences
between languages, we provide an interface to determine whether an AST
contains any test functions. This interface takes hooks as call-back functions
to check for test functions. In this work, we present two implementations. One
relies on heuristics by checking the function name. The other uses language-
specific features to determine if a function is a test. For example, in Java
JUnit, all test functions need to have the @Test modifier. Figure 2
illustrates the test collection phase and the aforementioned hooks are
summarized to the right of the figure. To extend this framework to other
languages and testing suites or to improve the success rate of test function
identification, one could provide a new, specially designed hook.

#### Focal Function Call Analysis.

Given a test function, we need to identify its focal function call. Since a
unit test function usually makes multiple function calls to set up its
environment, it is hard to identify the real call to the focal function. In
this paper, we followed TeCo’s Nie et al. (2023) practice to select the last
function call that invokes a function definition in the repository before the
first assertion. LABEL:lst:java_test_func demonstrates an example of a Java
test case, where line 13 includes the focal function call. To allow easy
extension to different languages as a unified method, we designed another
interface that the developers can implement with one extra function. The
static call analysis phase to build UniTSyn is demonstrated in Figure 3. Our
framework’s generic focal-call analysis algorithm only requires one extra
function to adapt to a new language.

⬇

1 belowskip=-2\baselineskip,

2 float,

3 language=Java,

4 caption={Example Java test function and its paired focal},

5 label=lst:java_test_func,

6 basicstyle=\ttfamily\scriptsize,

7]

8@Test

9public void testAdd() {

10 int x = 500; int offsetX = 100;

11 int y = 700; int offsetY = 200;

12 Position mp = new Position(x, y);

13 Position result = mp.add(offsetX, offsetY);

14 assertEquals(mp.x + offsetX, result.x);

15}

16public Position add(int x, int y) {

17 return new Position(this.x + x, this.y + y);

18}

Figure 3: Focal function call analysis phase of UniTSyn frontend. This figure
shows a simplified AST of the example test function in
LABEL:lst:java_test_func. The red node in the AST is the first encountered
assertion node with a postorder tree traversal. var_decl is the abbreviation
of the variable declaration nodes.

#### Focal Function Extraction via LSP.

We incorporated the LSP into our dataset-building process to utilize its
adaptability to different languages. This design helps relieve the pressure on
dependency analysis across various languages and executing multiple projects.
LSP is a language-neutral and standardized protocol for communication between
language servers and editor clients Microsoft , decoupling the programming
languages’ features and functionalities from integrated development
environments Rask et al. (2021). This separation allows the language
intelligence to run as an independent service, and hence be integrated into
our dataset construction. We developed the UniTSyn backend as a client to the
language servers, where we send requests to the language servers to locate the
definition of the focal function. The language servers send back the location
of the focal function as a response Gunasinghe & Marcus (2021). Our approach
of using LSP can easily extend to other languages as long as the language’s
language server is implemented. Our proposed method of aligning function-level
focal-test pairs using LSP is easily extendable, requiring only the commands
to start the corresponding language server.

##  4 Experiment

We built a test generation model called UniTester, which was trained on
UniTSyn and is capable of synthesizing unit tests for programs in different
languages. To investigate the quality of our testing code corpus, we evaluated
UniTester and several up-to-date code or test generation models on HumanEval-X
Zheng et al. (2023), as a popular multilingual program synthesis benchmark
that shares the same programming languages involved in UniTSyn.

###  4.1 UniTester: A Unified Test Generation Model

We constructed our model based on established code generation practices
Radford et al. (2019). We utilized an autoregression signal Radford et al.
(2019); Rozière et al. (2023); Nijkamp et al. (2023b) for continual training
of SantaCoder Allal et al. (2023), which is a powerful yet lightweight state-
of-the-art code generation model composed of 1.1B parameters. We selected this
model for its size, as this allows it to strike a balance between
effectiveness and efficiency. To build a training sample, we concatenated each
focal-test function pair with a newline symbol. We packed the training corpus
and sampled sequences of a constant length to feed into the models, thereby
avoiding padding samples of varying lengths and enhancing training efficiency.
We set the constant sequence length at $2048$ with a batch size of $32$. Our
training process employed a learning rate of $5e^{-5}$, incorporating a
logarithmic warmup for the initial 500 steps and a cosine annealing strategy
Loshchilov & Hutter (2016) for the rest. We also adopted a weight decay of
$0.05$ to refrain from overfitting and catastrophic forgetting Kirkpatrick et
al. (2017). Our UniTester was trained by an Adam optimizer Kingma & Ba (2014)
for $36,000$ steps on the collected data containing around one billion tokens.
The entire training process using eight Nvidia V100 GPUs spanned approximately
24 hours.

###  4.2 Research Questions and Evaluation Setup

To study the contributions of our collected testing code corpus, UniTSyn, we
investigated the following four questions:

  1. RQ.1

How accurate are the test cases generated by LLMs?  Considering that the
primary objective of software testing is to identify potential flaws in code,
it is essential to ensure that the generated test cases are accurate to
minimize incorrect evaluations. This requires models to not only comprehend
the general semantics of focal functions but also reason about their specific
behavior and precise input-output mappings, which is fundamentally
challenging. In this case, we followed Chen et al. to parse the assertions
from the generated tests to examine their standalone correctness and derive
conclusions without mutual influences.

  2. RQ.2

How many of the generated tests are complete? We define the completeness of
tests in terms of both their executability and the proportion of code in the
focal functions that has actually been executed by them. To evaluate how
complete the tests generated by LLMs are, we took the raw outputs of models
without intricate post-processing as individual tests and reported the
line/statement and branch coverage rates they achieved on the corresponding
focal function. This assessment allows us to better understand the
effectiveness of LLM-generated tests in covering various conditions and
scenarios within the codebase without excessive manual intervention.

  3. RQ.3

Is it necessary to train LLMs with pairwise focal and test functions? UniTSyn
is designed to identify test functions and pair them with their targets in
complex real-world software projects. Given that the composition of test
functions heavily depends on the expected behavior of their target focal
functions, we were motivated to train using test-focal function pairs. To
justify our motivation, we broke the pairwise connections and treated test and
focal functions as independent code snippets for LLMs training. The resulting
models were then compared with our UniTester that was trained with pairwise
data.

  4. RQ.4

What are the effects of training with multilingual testing code? While
previous work Feng et al. (2020); Guo et al. (2022) have shown that training
language models on data of different distributions can be beneficial, it was
unclear whether this conclusion also held for test generation. Therefore, we
compared models solely trained on Python to UniTester that was trained on five
different languages. This comparison aimed to determine whether test
generation models should be language-specific or if they can be universal
across languages.

#### Evaluation datasets.

To address the aforementioned research questions, we evaluated UniTester and
the state-of-the-art code LLMs on HumanEval-X Zheng et al. (2023), which is a
popular and multilingual code generation benchmark dataset. It comprises
natural language descriptions of $164$ coding tasks and their solutions in
Python, C++, Java, JavaScript, and Go (the same languages used in UniTSyn).
Note that we prompted the models to generate tests for the given set of
programs in HumanEval-X instead of synthesizing the programs based on their
problem descriptions.

To mitigate data leakage and facilitate a more straightforward execution
setup, we chose to experiment on an existing code generation benchmark.
Previous work Rao et al. (2023); Nie et al. (2023); Schäfer et al. (2023) on
test generation typically validate their models on public data collected by
themselves. Given that the success of the latest code LLMs is also based on
large-scale open-source coding resources, it is challenging to avoid data
leakage in such cases. Consequently, we turned to the benchmark datasets
intended for this purpose, ensuring that all focal and test functions used for
assessment had never been seen by any of the models. HumanEval-X is also well
crafted such that its test cases can be executed without excessive efforts in
setting up the execution environments and sorting out the intricate
dependencies between packages. This allows for a more straightforward and
unbiased evaluation of the models’ performance in generating accurate and
comprehensive tests for software projects.

#### Instructive prompts.

Following CodeT’s approach Chen et al. (2022), we prompted the models to
generate tests using language-specific assert keywords. Examples of prompts
for the first task in HumanEval-X are shown in Figure 4. Specially, for models
trained by instruction tuning, we concatenated the focal function and our
natural language hint (“Check the correctness of...”) as the instruction then
asked the models to complete the test functions. The maximal input length was
set to $800$ and the synthesized outputs were allowed to have another $256$
tokens at most. We set the temperature for generation to $0.2$ following Xiong
et al. (2023); Rao et al. (2023) and kept it consistent for all models, then
sampled ten outputs for each task. Subsequently, we parsed and sampled the
first ten assertions from the generated tests for accuracy computation on each
task. This ensures that different models produce a similar number of
assertions for verification. Regarding the evaluation of completeness, we
treated each output of the models as an individual test and executed it
independently. We avoided applying intricate post-processing on all of the
models’ outputs except for the Java ones, on which we found that adding
closing brackets for the test class is helpful to most models for ensuring
executability. The average coverage rates from ten trials are reported in
Table 3.

⬇

1from typing import List

2def has_close_elements(

3 numbers: List[float], threshold: float) -> bool:

4 ...

5# Check the correctness of ‘has_close_elements‘

6def test_has_close_elements():

7 assert has_close_elements(

Listing 1: Python

⬇

1func HasCloseElements(

2 numbers []float64, threshold float64) bool

3{ ... }

4// Check the correctness of ‘HasCloseElements‘

5func TestHasCloseElements(t *testing.T) {

6 assert := assert.New(t)

7 assert.Equal(HasCloseElements(

Listing 2: Go

⬇

1const hasCloseElements = (numbers, threshold) => {...}

2// Check the correctness of ‘hasCloseElements‘

3const testHasCloseElements = () => {

4 console.assert(hasCloseElements(

Listing 3: JavaScript

⬇

1bool has_close_elements(

2 vector<float> numbers, float threshold)

3{ ... }

4// Check the correctness of ‘has_close_elements‘

5#undef NDEBUG

6#include <assert.h>

7int main(){

8 assert(has_close_elements(

Listing 4: C++

⬇

1class Solution {

2 public boolean hasCloseElements(

3 List<Double> numbers, double threshold)

4 { ... }

5}

6public class Main {

7 // Check the correctness of ‘hasCloseElements‘

8 public static void main(String[] args) {

9 Solution s = new Solution();

10 assert s.hasCloseElements(

Listing 5: Java

Figure 4: Prompts used for test generation in different languages

#### Compared models.

We conducted extensive comparisons between our proposed model and both the
state-of-the-art code generation models highlighted in Xiong et al.’s paper as
well as the latest test generation model Rao et al. (2023). In our evaluation,
we considered code generation models with a similar number of parameters to
ours, encompassing both encoder-decoder (CodeT5+ Wang et al. (2023)) and
decoder-only structures (CodeGen2 Nijkamp et al. (2023a), WizardCoder Luo et
al. (2023), InCoder Fried et al. (2022) and SantaCoder Allal et al. (2023)).
For test generation, we selected CAT-LM Rao et al. (2023) as our competitor,
which was trained on unit tests from Python and Java projects that were paired
with their target functions at the file level. CAT-LM was chosen because it is
trained on a much larger number of tokens ($60$B v.s. $1$B) than our model and
has double our model size. This allowed us to demonstrate our model’s superior
performance, despite any potential advantages stemming from the amount of
training data and the model size.

###  4.3 Evaluation Results

#### RQ.1 How accurate are the test cases generated by LLMs?

Table 2:  Evaluations of the accuracy of tests generated by LLMs. The “#Params” column indicates the size of models. The best results are highlighted in bold. The models marked with $\dagger$ are intended for test generation.  Model |  | #Params |  | Py |  | C++ |  | Java |  | JS |  | Go |  | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
CodeT5p |  | 770M |  | 30.6 |  | 33.7 |  | 26.9 |  | 37.1 |  | 32.9 |  | 32.2
CodeGen2 |  | 1.0B |  | 34.0 |  | 40.7 |  | 24.1 |  | 30.5 |  | 36.1 |  | 33.1
WizardCoder |  | 1.0B |  | 36.8 |  | 43.9 |  | 28.7 |  | 31.3 |  | 47.7 |  | 37.7
InCoder |  | 1.3B |  | 34.2 |  | 33.5 |  | 22.6 |  | 24.4 |  | 31.5 |  | 29.2
SantaCoder |  | 1.1B |  | 36.2 |  | 34.7 |  | 36.5 |  | 30.6 |  | 31.5 |  | 33.9
CAT-LM† |  | 2.7B |  | 37.5 |  | 31.6 |  | 34.4 |  | 29.2 |  | 36.9 |  | 33.9
UniTester† (Ours) |  | 1.1B |  | 52.5 |  | 55.1 |  | 48.8 |  | 41.7 |  | 59.7 |  | 51.5

The accuracy of the test cases generated by different models is presented in
Table 2. Our model achieved up to a $40\%$ relative margin on Python compared
to the top competitor (CAT-LM), demonstrating remarkable advantages over both
code and test synthesis models. In terms of the code generation models,
UniTester beat the strongest competitor (WizardCoder) by $36.6\%$ on average.
These results validate the quality of the data in UniTSyn and indicate its
benefits on enhancing LLMs’ capability of in-depth code understanding and
reasoning in order to ensure the high accuracy of the generated test cases.
Our improvements over the baseline SantaCoder Allal et al. (2023) on the two
languages that it was not trained on (C++ and Go) are on par with the rest.
The advantages are at times even more significant on languages with
insufficient training data, e.g., JavaScript. This implies that it might not
be necessary to have our base models trained on every language of interest to
generate high-quality tests. In addition, we observed that our improvements
over the top competitors across different languages are somewhat dependent on
the sufficiency of the tests collected, with the least improvement being in
JavaScript and the most in Python. This observation highlights the
contribution of UniTSyn, which is capable of collecting testing data for
different languages at scale.

Table 3:  Evaluations of the completeness of LLM-generated tests. The “#Params” column indicates the size of models, and “#Pass” suggests how many tests for $164$ tasks can be executed without any errors. The “Line”, “Stmt” and “Branch” are abbreviations of line, statement and branch coverage, respectively. The models marked with $\dagger$ are intended for test generation. Limited by the coverage evaluation tools we adopted, branch coverage is not available on C++ and Go.  |  |  |  | Python |  | C++ |  | Java |  | Javascript |  | Go
---|---|---|---|---|---|---|---|---|---|---|---|---
Model |  | #Params |  | #Pass | Line | Branch |  | #Pass | Line |  | #Pass | Line | Branch |  | #Pass | Line | Branch |  | #Pass | Stmt
CodeT5p |  | 770M |  | 10.0 | 5.72 | 5.41 |  | 0.7 | 0.43 |  | 40.3 | 4.22 | 2.01 |  | 4.9 | 2.07 | 1.01 |  | 1.7 | 0.73
CodeGen2 |  | 1B |  | 4.1 | 2.41 | 2.34 |  | 11.6 | 7.07 |  | 52.3 | 5.12 | 3.29 |  | 48.5 | 27.65 | 23.87 |  | 19.2 | 10.99
WizardCoder |  | 1B |  | 16.1 | 9.39 | 8.95 |  | 3.7 | 2.24 |  | 47.7 | 5.62 | 4.09 |  | 9.2 | 5.50 | 5.32 |  | 0.7 | 0.42
InCoder |  | 1.3B |  | 3.0 | 1.76 | 1.60 |  | 0.0 | 0.00 |  | 15.0 | 1.54 | 0.91 |  | 0.5 | 0.29 | 0.26 |  | 1.3 | 0.78
SantaCoder |  | 1.1B |  | 4.5 | 2.62 | 2.59 |  | 4.9 | 2.99 |  | 50.1 | 4.74 | 1.83 |  | 5.9 | 3.53 | 3.23 |  | 0.7 | 0.43
CAT-LM† |  | 2.7B |  | 35.9 | 19.51 | 18.03 |  | 0.0 | 0.00 |  | 0.9 | 0.07 | 0.00 |  | 9.2 | 4.53 | 3.49 |  | 0.0 | 0.00
UniTester† (Ours) |  | 1.1B |  | 41.2 | 20.71 | 18.27 |  | 28.1 | 13.39 |  | 103.1 | 10.78 | 4.57 |  | 53.3 | 27.59 | 23.34 |  | 36.0 | 12.39

#### RQ.2 How many of the generated tests are complete?

We presented the average number of passing tests generated by different models
and the average coverage rates they achieved on the focal functions in Table
3. The passing rates of each model across all languages range from $2\%$ to
$32\%$. Low passing rates indicate the necessity of both accurate and
executable test functions generated by LLMs. Our UniTester demonstrated
considerable advantages in this regard, with its average passing number being
nearly twice that of the top competitor, CodeGen2 Nijkamp et al. (2023a). This
result shows the inconsistency between the distributions of focal functions
and tests, emphasizing the necessity of training models with a testing code
corpus to synthesize high-quality tests. We also observed that composing
executable tests for C++ was the most challenging among all the languages.
This is not surprising, as C++ is considered relatively weaker in readability
and harder to write code in even for human developers. Moreover, UniTester
yielded superior coverage rates, with absolute improvements of up to $6.32\%$
and an average of $2.8\%$ regarding line coverage. Such improvements
underscore the effectiveness of UniTester, which was trained on the testing
code corpus of UniTSyn, in covering various conditions in the focal functions.

#### RQ.3 Is it necessary to train LLMs with pairwise focal and test
functions?

Figure 5:  Effects of training with pairwise test and focal functions.
“Baseline” is the SantaCoder model, which didn’t train on our collected data.
“Unpaired” is the model obtained by training without associating test and
focal functions. “Paired” is UniTester trained with focal-test pairs.

In Figure 5, we constructed an “Unpaired” variant of UniTester (“Paired”) by
decoupling test functions from their targets for model training and compared
it with our models trained with pairwise focal-test data. The unpaired variant
of UniTester demonstrated its capability to generate more accurate tests than
the baseline. While SantaCoder’s paper did not explicitly state the exclusion
of testing code from their training corpus, it is likely that their model was
also trained with a certain number of test functions. However, it is shown to
be less competent for test generation than the “Unpaired” model. One possible
reason for this is the more balanced focal-test functions in UniTSyn, despite
being unpaired. Furthermore, the remarkable performance advantages of
UniTester over its unpaired counterpart indicate the importance of associating
test functions with their targets. Including focal functions in the context
when generating tests explicitly provides the models with insights into the
expected usages and behavior. Without these insights, the model can only learn
to make reasonable predictions for test functions by memorizing all possible
focal functions, rather than through reasoning. This observation highlights
the value of UniTSyn on not only collecting testing data but also matching it
with the focal functions in a language-agnostic manner.

#### RQ.4 What are the effects of training with multilingual testing code?

Figure 6:  Effects of training with multilingual testing code. “Baseline” is
the SantaCoder which didn’t train on our collected testing data. “Mono” and
“Multi“ stand for monolingual and multilingual models trained with either
solely Python data or jointly five programming languages.

The construction of all-purpose code generation models has been garnering
increased attention. Previous studies have also suggested the potential
benefits of training on data with shared semantics but different distributions
Feng et al. (2020). Our proposed UniTSyn is extendable to collect tests for
any programming language with an available language server, enabling us to
build a universal test generation model. To investigate the best practice of
utilizing a multilingual testing code corpus for training LLMs, we compared
UniTester with its monolingual variant, which was trained solely on the Python
subset of UniTSyn. We denoted UniTester and its monolingual variant by “Multi”
and “Mono” respectively in Figure 6. The figure reveals that the monolingual
model demonstrated superior capability in generating high-quality tests for
Python. This capability transferred well to other scripting languages like
JavaScript and Go, but less so to C++ and Java. These results suggest modest
transferability between syntactically similar languages but also indicates the
potential for negative impacts elsewhere. Regardless of whether the aim is to
build test generation models for specific languages or for general purposes, a
flexible and universal framework for collecting tests from different languages
is indispensable. This demonstrates the contribution of our proposed UniTSyn
in the field of software testing.

##  5 Conclusion

In this paper, we present UniTSyn, a novel, diverse, and large-scale dataset
containing function-level focal-test pairs designed to stimulate AI in
understanding and writing programs, particularly for test cases. This dataset
not only excels in size and diversity, but is also effortlessly extendable to
other programming languages for specific tasks. We further built an
autoregressive model based on UniTSyn to verify the quality of the collected
testing code corpus. This is shown by its superiority in terms of both the
accuracy and completeness of the generated tests. We hope that UniTSyn will
spur the development of AI for software testing and program understanding in
general.

## Limitations

Despite our effort to design UniTSyn to involve as little human effort as
possible, some manual settings are inevitable. For example, C++ and JavaScript
do not have a commonly used testing framework, meaning that the developer has
to implement the hook to identify test functions for each framework. Since we
only implemented the test function identification hook for the GoogleTest
suite, this accounts for our low repository number in C++. We believe one can
extract more focal-test pairs from the repositories by applying more precise
hooks to our framework. Our goal is to build an extensible multilingual
dataset instead of diving deep into different C++ and JavaScript testing
suites.

## Impact Statements

This paper presents work whose goal is to advance the field of Machine
Learning for software testing and program understanding. One major impact is
that UniTSyn provides a new dataset to train LLMs. With paired focal functions
and test functions, UniTSyn is able to advance the quality of the test cases
generated by LLMs. In the meantime, we also take the generalizability to
different programming languages into account and provide an interface where
developers can easily extend UniTSyn to other datasets. We will release
UniTSyn’s code and data upon acceptance.

## References

  * (1) Tree-sitter introduction.  URL https://tree-sitter.github.io/tree-sitter/.
  * Ahmad et al. (2020) Ahmad, W., Chakraborty, S., Ray, B., and Chang, K.-W.  A transformer-based approach for source code summarization.  In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (eds.), _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 4998–5007, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.449.  URL https://aclanthology.org/2020.acl-main.449.
  * Alagarsamy et al. (2023) Alagarsamy, S., Tantithamthavorn, C., and Aleti, A.  A3test: Assertion-augmented automated test case generation.  _arXiv preprint arXiv:2302.10352_ , 2023.
  * Allal et al. (2023) Allal, L. B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C. M., Muennighoff, N., Mishra, M., Gu, A., Dey, M., et al.  Santacoder: don’t reach for the stars!  _arXiv preprint arXiv:2301.03988_ , 2023.
  * Beller et al. (2015) Beller, M., Gousios, G., Panichella, A., and Zaidman, A.  When, how, and why developers (do not) test in their ides.  In _Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering_ , ESEC/FSE 2015, pp. 179–190, New York, NY, USA, 2015. Association for Computing Machinery.  ISBN 9781450336758.  doi: 10.1145/2786805.2786843.  URL https://doi.org/10.1145/2786805.2786843.
  * Chen et al. (2022) Chen, B., Zhang, F., Nguyen, A., Zan, D., Lin, Z., Lou, J.-G., and Chen, W.  Codet: Code generation with generated tests.  _arXiv preprint arXiv:2207.10397_ , 2022.
  * Chen et al. (2021) Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W.  Evaluating large language models trained on code, 2021.
  * Chen & Chen (2018) Chen, P. and Chen, H.  Angora: Efficient fuzzing by principled search.  In _2018 IEEE Symposium on Security and Privacy (SP)_ , pp. 711–725, 2018.  doi: 10.1109/SP.2018.00046.
  * Chen et al. (2019) Chen, P., Liu, J., and Chen, H.  Matryoshka: Fuzzing deeply nested branches.  In _Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security_ , CCS ’19, pp. 499–513, New York, NY, USA, 2019. Association for Computing Machinery.  ISBN 9781450367479.
  * Chen et al. (2016) Chen, Y., Su, T., Sun, C., Su, Z., and Zhao, J.  Coverage-directed differential testing of jvm implementations.  _SIGPLAN Not._ , 51(6):85–99, jun 2016.  ISSN 0362-1340.  doi: 10.1145/2980983.2908095.  URL https://doi.org/10.1145/2980983.2908095.
  * Dijkstra (1972) Dijkstra, E. W.  _Chapter I: Notes on Structured Programming_ , pp. 1–82.  Academic Press Ltd., GBR, 1972.  ISBN 0122005503.
  * Dinella et al. (2022) Dinella, E., Ryan, G., Mytkowicz, T., and Lahiri, S. K.  Toga: A neural method for test oracle generation.  In _Proceedings of the 44th International Conference on Software Engineering_ , ICSE ’22, pp. 2130–2141, New York, NY, USA, 2022. Association for Computing Machinery.  ISBN 9781450392211.  doi: 10.1145/3510003.3510141.  URL https://doi.org/10.1145/3510003.3510141.
  * Feng et al. (2020) Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., and Zhou, M.  CodeBERT: A pre-trained model for programming and natural languages.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pp. 1536–1547, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.findings-emnlp.139.  URL https://aclanthology.org/2020.findings-emnlp.139.
  * Fioraldi et al. (2020) Fioraldi, A., Maier, D., Eißfeldt, H., and Heuse, M.  Afl++ combining incremental steps of fuzzing research.  In _Proceedings of the 14th USENIX Conference on Offensive Technologies_ , pp. 10–10, 2020.
  * Fried et al. (2022) Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, W.-t., Zettlemoyer, L., and Lewis, M.  Incoder: A generative model for code infilling and synthesis.  _arXiv preprint arXiv:2204.05999_ , 2022.
  * Gunasinghe & Marcus (2021) Gunasinghe, N. and Marcus, N.  _Language Server Protocol and Implementation_.  Springer, 2021.
  * Guo et al. (2022) Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., and Yin, J.  Unixcoder: Unified cross-modal pre-training for code representation.  In _ACL_ , 2022.
  * Huang et al. (2023) Huang, J., Zhao, J., Rong, Y., Guo, Y., He, Y., and Chen, H.  Code representation pre-training with complements from program executions, 2023.
  * Husain et al. (2020) Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., and Brockschmidt, M.  Codesearchnet challenge: Evaluating the state of semantic code search, 2020.
  * Iyer et al. (2016) Iyer, S., Konstas, I., Cheung, A., and Zettlemoyer, L.  Summarizing source code using a neural attention model.  In Erk, K. and Smith, N. A. (eds.), _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 2073–2083, Berlin, Germany, August 2016. Association for Computational Linguistics.  doi: 10.18653/v1/P16-1195.  URL https://aclanthology.org/P16-1195.
  * Kingma & Ba (2014) Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kirkpatrick et al. (2017) Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ , 114(13):3521–3526, 2017.
  * LeClair et al. (2019) LeClair, A., Jiang, S., and McMillan, C.  A neural model for generating natural language summaries of program subroutines, 2019.
  * Li et al. (2023) Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., Liu, Q., Zheltonozhskii, E., Zhuo, T. Y., Wang, T., Dehaene, O., Davaadorj, M., Lamy-Poirier, J., Monteiro, J., Shliazhko, O., Gontier, N., Meade, N., Zebaze, A., Yee, M.-H., Umapathi, L. K., Zhu, J., Lipkin, B., Oblokulov, M., Wang, Z., Murthy, R., Stillerman, J., Patel, S. S., Abulkhanov, D., Zocca, M., Dey, M., Zhang, Z., Fahmy, N., Bhattacharyya, U., Yu, W., Singh, S., Luccioni, S., Villegas, P., Kunakov, M., Zhdanov, F., Romero, M., Lee, T., Timor, N., Ding, J., Schlesinger, C., Schoelkopf, H., Ebert, J., Dao, T., Mishra, M., Gu, A., Robinson, J., Anderson, C. J., Dolan-Gavitt, B., Contractor, D., Reddy, S., Fried, D., Bahdanau, D., Jernite, Y., Ferrandis, C. M., Hughes, S., Wolf, T., Guha, A., von Werra, L., and de Vries, H.  Starcoder: may the source be with you!, 2023.
  * Li et al. (2022) Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., Hubert, T., Choy, P., de Masson d’Autume, C., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J., Robson, E. S., Kohli, P., de Freitas, N., Kavukcuoglu, K., and Vinyals, O.  Competition-level code generation with alphacode.  _Science_ , 378(6624):1092–1097, 2022.  doi: 10.1126/science.abq1158.  URL https://www.science.org/doi/abs/10.1126/science.abq1158.
  * Loshchilov & Hutter (2016) Loshchilov, I. and Hutter, F.  SGDR: stochastic gradient descent with restarts.  _CoRR_ , abs/1608.03983, 2016.  URL http://arxiv.org/abs/1608.03983.
  * Lu et al. (2021) Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al.  Codexglue: A machine learning benchmark dataset for code understanding and generation.  _arXiv preprint arXiv:2102.04664_ , 2021.
  * Luo et al. (2023) Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., and Jiang, D.  Wizardcoder: Empowering code large language models with evol-instruct.  _arXiv preprint arXiv:2306.08568_ , 2023.
  * (29) Microsoft.  Language server protocol.  URL https://microsoft.github.io/language-server-protocol/.
  * Nagappan & Ball (2010) Nagappan, N. and Ball, T.  Evidence-based failure prediction.  In Andy Oram, G. W. (ed.), _Making Software_ , chapter 23. O’REILLY, 2010.
  * Nie et al. (2023) Nie, P., Banerjee, R., Li, J. J., Mooney, R. J., and Gligoric, M.  Learning deep semantics for test completion, 2023.
  * Nijkamp et al. (2023a) Nijkamp, E., Hayashi, H., Xiong, C., Savarese, S., and Zhou, Y.  Codegen2: Lessons for training llms on programming and natural languages.  _arXiv preprint arXiv:2305.02309_ , 2023a.
  * Nijkamp et al. (2023b) Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., and Xiong, C.  Codegen: An open large language model for code with multi-turn program synthesis.  _ICLR_ , 2023b.
  * Puri et al. (2021) Puri, R., Kung, D. S., Janssen, G., Zhang, W., Domeniconi, G., Zolotov, V., Dolby, J., Chen, J., Choudhury, M., Decker, L., Thost, V., Buratti, L., Pujar, S., Ramji, S., Finkler, U., Malaika, S., and Reiss, F.  Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks, 2021.
  * Radford et al. (2019) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Rao et al. (2023) Rao, N., Jain, K., Alon, U., Goues, C. L., and Hellendoorn, V. J.  Cat-lm: Training language models on aligned code and tests, 2023.
  * Rask et al. (2021) Rask, J. K., Madsen, F. P., Battle, N., Macedo, H. D., and Larsen, P. G.  The specification language server protocol: A proposal for standardised lsp extensions.  _arXiv preprint arXiv:2108.02961_ , 2021.
  * Rong et al. (2020) Rong, Y., Chen, P., and Chen, H.  Integrity: Finding integer errors by targeted fuzzing.  In _International Conference on Security and Privacy in Communication Systems_ , pp. 360–380. Springer, 2020.
  * Rong et al. (2024) Rong, Y., Zhang, C., Liu, J., and Chen, H.  Valkyrie: Improving fuzzing performance through deterministic techniques.  _Journal of Systems and Software_ , 209:111886, 2024.
  * Rozière et al. (2023) Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Ferrer, C. C., Grattafiori, A., Xiong, W., Défossez, A., Copet, J., Azhar, F., Touvron, H., Martin, L., Usunier, N., Scialom, T., and Synnaeve, G.  Code llama: Open foundation models for code, 2023.
  * Schäfer et al. (2023) Schäfer, M., Nadi, S., Eghbali, A., and Tip, F.  An empirical evaluation of using large language models for automated unit test generation, 2023.
  * Serebryany (2016) Serebryany, K.  Continuous fuzzing with libfuzzer and addresssanitizer.  In _2016 IEEE Cybersecurity Development (SecDev)_ , pp. 157–157, 2016.  doi: 10.1109/SecDev.2016.043.
  * Tufano et al. (2021) Tufano, M., Drain, D., Svyatkovskiy, A., Deng, S. K., and Sundaresan, N.  Unit test case generation with transformers and focal context, 2021.
  * Wang et al. (2021) Wang, Y., Wang, W., Joty, S., and Hoi, S. C. H.  Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation, 2021.
  * Wang et al. (2023) Wang, Y., Le, H., Gotmare, A. D., Bui, N. D., Li, J., and Hoi, S. C.  Codet5+: Open code large language models for code understanding and generation.  _arXiv preprint arXiv:2305.07922_ , 2023.
  * Watson et al. (2020) Watson, C., Tufano, M., Moran, K., Bavota, G., and Poshyvanyk, D.  On learning meaningful assert statements for unit test cases.  In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_. ACM, jun 2020.  doi: 10.1145/3377811.3380429.  URL https://doi.org/10.1145%2F3377811.3380429.
  * Xiong et al. (2023) Xiong, W., Guo, Y., and Chen, H.  The program testing ability of large language models for code.  _arXiv preprint arXiv:2310.05727_ , 2023.
  * Yang et al. (2006) Yang, Q., Li, J. J., and Weiss, D.  A survey of coverage based testing tools.  In _Proceedings of the 2006 International Workshop on Automation of Software Test_ , AST ’06, pp. 99–103, New York, NY, USA, 2006. Association for Computing Machinery.  ISBN 1595934081.  doi: 10.1145/1138929.1138949.  URL https://doi.org/10.1145/1138929.1138949.
  * Yu et al. (2023) Yu, H., Shen, B., Ran, D., Zhang, J., Zhang, Q., Ma, Y., Liang, G., Li, Y., Xie, T., and Wang, Q.  Codereval: A benchmark of pragmatic code generation with generative pre-trained models.  _arXiv preprint arXiv:2302.00288_ , 2023.
  * Zhao et al. (2023) Zhao, J., Rong, Y., Guo, Y., He, Y., and Chen, H.  Understanding programs by exploiting (fuzzing) test cases, 2023.
  * Zhao et al. (2017) Zhao, Y., Serebrenik, A., Zhou, Y., Filkov, V., and Vasilescu, B.  The impact of continuous integration on other software development practices: A large-scale empirical study.  In _2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE)_ , pp. 60–71, 2017.  doi: 10.1109/ASE.2017.8115619.
  * Zheng et al. (2023) Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Wang, Z., Shen, L., Wang, A., Li, Y., Su, T., Yang, Z., and Tang, J.  Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x, 2023.
  * Zhu et al. (1997) Zhu, H., Hall, P. A. V., and May, J. H. R.  Software unit test coverage and adequacy.  _ACM Comput. Surv._ , 29(4):366–427, dec 1997.  ISSN 0360-0300.  doi: 10.1145/267580.267590.  URL https://doi.org/10.1145/267580.267590.


which $\beta$ turns out to be fully symmetric in the $a_{i}$), used that
$\big{(}-A\big{)}_{A}=(-1)^{A}\,A!$ on the integers $A$, and employed the
Euler reflection formula
$\Gamma(\tfrac{1}{2}+n)\,\Gamma(\tfrac{1}{2}-n)=(-1)^{n}\,\Gamma^{2}(\tfrac{1}{2})$
for integers $n=A-a_{i}$ and $n=A+1$.

Equation (176) coincides with eq. (112), and provides an independent check of
its validity.

## References

  * (1) V. A. Smirnov, Analytic tools for Feynman integrals, Springer Tracts Mod. Phys. 250 (2012) 1.
  * (2) S. Weinzierl, Feynman integrals, Springer, 2022 [arXiv:2201.03593].
  * (3) S. Abreu, R. Britto and C. Duhr, The SAGEX Review on scattering amplitudes, Chapter 3: Mathematical structures in Feynman integrals, [arXiv:2203.13014].
  * (4) J. Blümlein and C. Schneider, The SAGEX Review on scattering amplitudes, Chapter 4: Multi-loop Feynman integrals, [arXiv:2203.13015].
  * (5) P. Lairez and P. Vanhove, Algorithms for minimal Picard-Fuchs operators of Feynman integrals, [arXiv:2209.10962].
  * (6) J. L. Bourjaily, J. Broedel, E. Chaubey, C. Duhr, H. Frellesvig, M. Hidding, R. Marzucca, A. J. McLeod, M. Spradlin and L. Tancredi, et al., Functions beyond multiple polylogarithms for precision collider physics, [arXiv:2203.07088].
  * (7) A. I. Davydychev and J. B. Tausk, Two-loop selfenergy diagrams with different masses and the momentum expansion, Nucl. Phys. B 397 (1993) 123.
  * (8) J. Fleischer and O. V. Tarasov, Calculation of Feynman diagrams from their small momentum expansion, Z. Phys. C 64 (1994), 413 [arXiv:hep-ph/9403230].
  * (9) A. I. Davydychev and J. B. Tausk, Tensor reduction of two loop vacuum diagrams and projectors for expanding three-point functions, Nucl. Phys. B 465 (1996), 507 [arXiv:hep-ph/9511261].
  * (10) S. Laporta, High precision calculation of multiloop Feynman integrals by difference equations, Int. J. Mod. Phys. A 15 (2000), 5087 [arXiv:hep-ph/0102033].
  * (11) K. G. Chetyrkin and F. V. Tkachov, Integration by parts: The algorithm to calculate Beta functions in 4 loops, Nucl. Phys. B 192 (1981), 159.
  * (12) F. V. Tkachov, A theorem on analytical calculability of four loop renormalization group functions, Phys. Lett. B 100 (1981), 65.
  * (13) S. Mizera, Status of Intersection Theory and Feynman integrals, PoS MA2019 (2019), 016 [arXiv:2002.10476].
  * (14) H. Frellesvig, F. Gasparotto, S. Laporta, M. K. Mandal, P. Mastrolia, L. Mattiazzi and S. Mizera, Decomposition of Feynman integrals by multivariate intersection numbers, JHEP 03 (2021), 027 [arXiv:2008.04823].
  * (15) S. Caron-Huot and A. Pokraka, Duals of Feynman integrals. Part II. Generalized unitarity, JHEP 04 (2022), 078 [arXiv:2112.00055].
  * (16) O. V. Tarasov, Connection between Feynman integrals having different values of the space-time dimension, Phys. Rev. D 54 (1996), 6479 [arXiv:hep-th/9606018].
  * (17) R. N. Lee, Space-time dimensionality D as complex variable: Calculating loop integrals using dimensional recurrence relation and analytical properties with respect to D, Nucl. Phys. B 830 (2010), 474 [arXiv:0911.0252].
  * (18) P. A. Baikov and K. G. Chetyrkin, Four loop massless propagators: An algebraic evaluation of all master integrals, Nucl. Phys. B 837 (2010), 186 [arXiv:1004.1153].
  * (19) R. N. Lee, A. V. Smirnov and V. A. Smirnov, Master integrals for four-loop massless propagators up to transcendentality weight twelve, Nucl. Phys. B 856 (2012), 95 [arXiv:1108.0732].
  * (20) A. A. Vladimirov, Method for computing renormalization group functions in dimensional renormalization scheme, Theor. Math. Phys. 43 (1980) 417.
  * (21) G. Källén, Elementary particle physics, Addison-Wesley, 1964.
  * (22) E. Byckling and K. Kajantie, Particle kinematics, John Wiley & Sons Ltd, 1973.
  * (23) F. A. Berends, A. I. Davydychev and V. A. Smirnov, Small threshold behavior of two loop selfenergy diagrams: Two particle thresholds, Nucl. Phys. B 478 (1996), 59 [arXiv:hep-ph/9602396].
  * (24) O. V. Tarasov, Generalized recurrence relations for two loop propagator integrals with arbitrary masses, Nucl. Phys. B 502 (1997) 455 [hep-ph/9703319].
  * (25) C. G. Bollini and J. J. Giambiagi, Dimensional renormalization: The number of dimensions as a regularizing parameter, Nuovo Cim. B 12 (1972), 20.
  * (26) G. ’t Hooft and M. J. G. Veltman, Regularization and renormalization of gauge fields, Nucl. Phys. B 44 (1972), 189.
  * (27) J. F. Ashmore, A method of gauge invariant regularization, Lett. Nuovo Cim. 4 (1972), 289.
  * (28) G. M. Cicuta and E. Montaldi, Analytic renormalization via continuous space dimension, Lett. Nuovo Cim. 4 (1972), 329.
  * (29) E. Braaten and A. Nieto, Free energy of QCD at high temperature, Phys. Rev. D 53 (1996), 3421 [arXiv:hep-ph/9510408].
  * (30) K. Kajantie, M. Laine, K. Rummukainen and Y. Schröder, The pressure of hot QCD up to $g^{6}\ln(1/g)$, Phys. Rev. D 67 (2003), 105008 [arXiv:hep-ph/0211321].
  * (31) M. Laine and A. Vuorinen, Basics of Thermal Field Theory, Lect. Notes Phys. 925 (2016), Springer, 2016, [arXiv:1701.01554].
  * (32) P. Navarrete and Y. Schröder, Tackling the infamous $g^{6}$ term of the QCD pressure, [arXiv:2207.10151].
  * (33) A. I. Davydychev and J. B. Tausk, A magic connection between massive and massless diagrams, Phys. Rev. D 53 (1996), 7381 [arXiv:hep-ph/9504431].
  * (34) E. E. Boos and A. I. Davydychev, A method of the evaluation of the vertex type Feynman integrals, Moscow Univ. Phys. Bull. 42N3 (1987), 6.
  * (35) E. E. Boos and A. I. Davydychev, A method of evaluating massive Feynman integrals, Theor. Math. Phys. 89 (1991), 1052.
  * (36) A. I. Davydychev, Recursive algorithm of evaluating vertex type Feynman integrals, J. Phys. A 25 (1992), 5587.
  * (37) R. Scharf and J. B. Tausk, Scalar two loop integrals for gauge boson selfenergy diagrams with a massless fermion loop, Nucl. Phys. B 412 (1994), 523.
  * (38) K. G. Chetyrkin, A. L. Kataev and F. V. Tkachov, New approach to evaluation of multiloop Feynman integrals: The Gegenbauer polynomial x space technique, Nucl. Phys. B 174 (1980), 345.
  * (39) A. Georgoudis, V. Goncalves, E. Panzer and R. Pereira, Five-loop massless propagator integrals, [arXiv:1802.00803].
  * (40) A. Georgoudis, V. Gonçalves, E. Panzer, R. Pereira, A. V. Smirnov and V. A. Smirnov, Glue-and-cut at five loops, JHEP 09 (2021), 098 [arXiv:2104.08272].
  * (41) A. N. Vasiliev, Y. M. Pismak and Y. R. Khonkonen, 1/$N$ expansion: Calculation of the exponents $\eta$ and $\nu$ in the order 1/$N^{2}$ for arbitrary number of dimensions, Theor. Math. Phys. 47 (1981), 465.
  * (42) N. I. Usyukina, Calculation of many loop diagrams of perturbation theory, Theor. Math. Phys. 54 (1983), 78.
  * (43) D. I. Kazakov, Calculation of Feynman integrals by the method of ‘uniqueness’, Theor. Math. Phys. 58 (1984), 223.
  * (44) J. A. Gracey, On the evaluation of massless Feynman diagrams by the method of uniqueness, Phys. Lett. B 277 (1992), 469.
  * (45) J. A. Gracey, Conformal methods for massless Feynman integrals and large $N_{f}$ methods, [arXiv:1301.7583].
  * (46) A. V. Kotikov and S. Teber, Multi-loop techniques for massless Feynman diagram calculations, Phys. Part. Nucl. 50 (2019) no.1, 1 [arXiv:1805.05109].
  * (47) C. Schneider, Modern summation methods for loop integrals in Quantum Field Theory: The packages Sigma, EvaluateMultiSums and SumProduction, J. Phys. Conf. Ser. 523 (2014), 012037 [arXiv:1310.0160].
  * (48) J. Blümlein, M. Saragnese and C. Schneider, Computer algebra and hypergeometric structures for Feynman integrals, [arXiv:2207.08524].
  * (49) B. Ruijl, T. Ueda and J. Vermaseren, The diamond rule for multi-loop Feynman diagrams, Phys. Lett. B 746 (2015), 347 [arXiv:1504.08258].
  * (50) F. V. Tkachov, An algorithm for calculating multiloop Integrals, Theor. Math. Phys. 56 (1983), 866.
  * (51) E. Mendels, Feynman diagrams without Feynman parameters, Nuovo Cim. A 45 (1978), 87.
  * (52) F. A. Berends, M. Buza, M. Böhm and R. Scharf, Closed expressions for specific massive multiloop selfenergy integrals, Z. Phys. C 63 (1994), 227.
  * (53) S. Groote, J. G. Körner and A. A. Pivovarov, On the evaluation of sunset-type Feynman diagrams, Nucl. Phys. B 542 (1999), 515 [arXiv:hep-ph/9806402].
  * (54) P. A. Baikov, A practical criterion of irreducibility of multi-loop Feynman integrals, Phys. Lett. B 634 (2006), 325 [arXiv:hep-ph/0507053].
  * (55) A. I. Davydychev, P. Navarrete and Y. Schröder, in preparation.