#  TestEval: Benchmarking Large Language Models for Test Case Generation

Wenhan Wang1 Chenyuan Yang2∗ Zhijie Wang1∗ Yuheng Huang3 Zhaoyang Chu4
Da Song1 Lingming Zhang2 An Ran Chen1 Lei Ma3,1
1University of Alberta 2University of Illinois Urbana-Champaign 3The
University of Tokyo
4Huazhong University of Science and Technology
{wenhan12, zhijie.wang, dsong4<EMAIL_ADDRESS>
{cy54<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>equal contribution

###### Abstract

Testing plays a crucial role in the software development cycle, enabling the
detection of bugs, vulnerabilities, and other undesirable behaviors. To
perform software testing, testers need to write code snippets that execute the
program under test. Recently, researchers have recognized the potential of
large language models (LLMs) in software testing. However, there remains a
lack of fair comparisons between different LLMs in terms of test case
generation capabilities.

In this paper, we propose TestEval, a novel benchmark for test case generation
with LLMs. We collect 210 Python programs from an online programming platform,
LeetCode, and design three different tasks: overall coverage, targeted
line/branch coverage, and targeted path coverage. We further evaluate sixteen
popular LLMs, including both commercial and open-source ones, on TestEval. We
find that generating test cases to cover specific program lines/branches/paths
is still challenging for current LLMs, indicating a lack of ability to
comprehend program logic and execution paths. We have open-sourced our dataset
and benchmark pipelines at https://llm4softwaretesting.github.io to contribute
and accelerate future research on LLMs for software testing.

##  1 Introduction

Software testing is a crucial aspect of software development, allowing
developers to identify potential bugs and verify that the program’s behavior
meets expectations. A key task in software testing is test case generation,
which involves creating test inputs to cover different statements and branches
in the program under test. Previous research indicates that test case
generation can be extremely time-consuming, accounting for over 15% of the
time spent in software development [1].

Therefore, automated test case generation has been a long-standing challenge
in software engineering research. Various methods have been developed to
address this issue, including symbolic execution testing [2, 3], model-based
testing [4, 5], random testing [6, 7], and search-based testing [8, 9, 10].
Recently, large language models (LLMs) have demonstrated promising performance
on several code-related tasks, e.g., code generation [11, 12, 13, 14, 15],
code translation [16, 17, 18], and code summarization [19, 20, 21]. At the
same time, researchers have been exploring the potential of using LLMs to
generate unit test cases [22, 23, 24]. However, despite the rapid development
of LLM-based test case generation, there is still a lack of public benchmarks
to evaluate different LLMs’ capabilities in this area. Hence, there is a need
for a comprehensive analysis to determine whether current LLMs can (1)
generate diverse test cases to achieve high coverage on a program under test,
(2) generate test cases to cover a specific line or branch, and (3) generate
test cases to cover a specific execution path by following the tester’s
intent.

To bridge this gap, we present a new benchmark, TestEval, which focuses on
evaluating LLMs’ test case generation capabilities. To construct our
benchmark, we collected a dataset of 210 Python programs from the online
coding platform LeetCode. To address the aforementioned challenges, we
designed three benchmark tasks: (1) overall coverage, (2) targeted line/branch
coverage, and (3) targeted path coverage.

Notably, unlike popular code generation benchmarks such as HumanEval [25] and
MBPP [26], the tasks in our TestEval benchmark require LLMs to comprehend the
program’s logic. This comprehension enables LLMs to generate inputs that
invoke specific branches or paths in the program under test. Furthermore, our
tasks emphasize program logic analysis rather than merely simulating numerical
operations, as seen in benchmarks designed for predicting a program’s
input/output [27].

We perform extensive experiments on TestEval with both commercial and open-
source LLMs. Our results indicate that while state-of-the-art LLMs can
generate executable and diverse test cases, they struggle to identify which
specific statements or branches need to be covered. For example, in targeted
line coverage, 12 out of 16 LLMs’ performances are not significantly improved
(improvements $\leq 5\%$) compared to the results when the target line
information is even not given. Quantitative results show that commercial LLMs,
such as GPT-4, generally outperform open-source LLMs in both overall coverage
and targeted line/branch/path coverage. These findings suggest that future
work on test case generation should focus on developing advanced LLM-based
reasoning frameworks to enhance the understanding of program behaviors during
testing.

Our work makes the following contributions:

  * •

Benchmark. We propose TestEval, a benchmark focused on evaluating LLMs’
capabilities in generating test cases for a given program under test,
encompassing three different tasks. Our benchmark is publicly available at
https://llm4softwaretesting.github.io.

  * •

Evaluation. We design new metrics to measure the performance of test
generation for LLMs and conduct extensive experiments with 16 popular LLMs.

  * •

Analysis. We perform a systematic analysis of 16 LLMs’ performance on TestEval
and discuss the challenges and opportunities in test case generation using
LLMs.

##  2 Approach

In this section, we first introduce the tasks included in our benchmark (§
2.1). Following that, we provide an overview of the dataset used (§ 2.2).

###  2.1 Benchmark Tasks

Figure 1: The pipeline for running and evaluating LLMs for test case
generation on TestEval.

Figure 1 shows the workflow of TestEval. We consider three distinct tasks in
our benchmark: (1) overall coverage, (2) targeted line and branch coverage,
and (3) targeted path coverage. For each task, we prompt an LLM to generate
test cases for a specified program based on the task description in natural
language. Specifically, in each query round, we prompt the LLM to generate a
testing function containing a single test case (Appendix A presents the
complete prompt template for each task). Then, we filter out any non-code
content that may have been generated outside the testing function, retaining
only the first test case generated in each query round to ensure a fair
comparison across different LLMs.

After generation, all test cases must undergo a correctness check. This check
assesses both syntactic correctness and execution correctness. Syntactic
correctness determines if the generated test case is free of syntax errors,
while execution correctness evaluates if the test case can be executed
successfully without encountering any runtime errors. Regarding execution
correctness, we do not consider incorrect test assertion statements as failed
cases. This is because we observed that even state-of-the-art models struggled
to reliably produce test oracles in our early exploration. Finally, we
evaluate coverage metrics on test cases that pass the correctness check. We
now illustrate our three benchmark tasks in detail.

Input: A set of programs under test
$\mathcal{P}=\left\\{p_{1},p_{2},...\right\\}$, $k$

Output: The average $cov@k$ for all programs: $cov@k_{all}$

$cov@k$ = [];

for  _$p_{i}$ in $\mathcal{P}$_ do

Generate $N$ test cases $T_{i}=\left\\{t_{i1},t_{i2},...,t_{iN}\right\\}$;

Retain $M$ executable test cases
$T_{i}=\left\\{t_{i1},t_{i2},...,t_{iM}\right\\}$;

if  _$T_{i}=\emptyset$_ then

$cov@k$.append($0$);

else

Randomly split $T_{i}$ into $max(M//k,1)$ groups, each group with $k$ test
cases;

$cov_{i}$ = [];

for  _$T_{ij}$ in $\left\\{T_{i1},T_{i2},...,T_{iM//k}\right\\}$_ do

Compute line/branch coverage $cov_{T_{ij}}$;

$cov_{i}$.append($cov_{T_{ij}}$);

end for

$cov@k$.append(avg($cov_{i}$))

end if

end for

$cov@k_{all}~{}\leftarrow~{}\textrm{avg}(cov@k)$;

return _$cov@k_{all}$_

Algorithm 1 Computing the average line/branch $cov@k$ given a set of programs

Overall coverage. In this task, we query each LLM for $N$ rounds given a
program under test. During the $i$th ($1<i\leq N$) round, we prompt the LLM to
generate a test method different from the $i-1$th round. After all rounds of
query, we obtain $N$ test cases for each program under test. The overall
coverage for a program is computed by the proportion of lines/branches in the
program that have been covered by at least one test case.

We further propose a new metric, $cov@k$, to measure the diversity of LLM’s
generated test cases for a given program. Intuitively, $cov@k$ measures the
line/branch coverage with a subset of the generated test cases with a size of
$k$ ($k<M$). To achieve this, we randomly split $M$ executable test cases into
$max(M//k,1)$ subsets. Then for each of these subsets, we calculate its
overall line/branch coverage. In our experiments, we choose $k$ as 1, 2, and
5. When $k$ increases, the improvements of $cov@k$ can measure the diversity
of the LLM’s generated test cases. We summarize the calculation of the average
line/branch $cov@k$ for a set of programs under test in Algorithm 1.

Targeted line and branch coverage. Different from overall coverage, targeted
line and branch coverage requires the LLM to generate test cases that could
cover a specific branch, or a line inside this branch. This simulates the
scenario in which a human tester is asked to craft test cases to cover a
specific part of the program. Figure 2 shows an example of targeted branches
and lines in a given program. To measure the targeted line/path coverage, we
prompt the LLMs by including the line number(s) in the instruction (see prompt
templates in Appendix A). For each targeted line/branch, an LLM is prompted to
generate one test case.

Figure 2: An example for selecting targeted lines and branches from programs
under test.

Targeted path coverage. In real-world software development, testers sometimes
need to craft test cases to cover a specific execution path that includes
multiple branches. We refer to this task as the target path coverage. We show
an example program in Figure 3 to demonstrate the importance of the target
path coverage. In Line 6, a bug (divided by zero) will occur only if branches
“condition 1” and “condition 2” are both not executed. In this case, only
covering the two conditional branches is not sufficient. By contrast, if we
can cover all three paths (Figure 3), we can successfully detect the “divided
by zero” bug. To obtain the target path coverage, we prompt an LLM by
including a specific execution path (see Appendix A for the prompt template).
For each path, an LLM is queried to generate one test case.

Figure 3: A motivating example of why path coverage is important (left), and
examples of execution paths extracted from this program (right).

We further propose two metrics to evaluate the performance of target path
coverage. First, for a given target path, we measure whether the generated
test case covers the target path completely. Second, we measure the similarity
between the given target path $\text{PATH}_{tgt}$ and the execution path of
the LLM’s generated test case $\text{PATH}_{gen}$ by Eq. 1.

$sim(\text{PATH}_{gen},\text{PATH}_{tgt})=\frac{lcs(\text{PATH}_{gen},\text{PATH}_{tgt})}{len(\text{PATH}_{tgt})},$
(1)

where $lcs()$ calculates the length of the longest common sub-sequence
(continuous) between two paths and $len()$ calculates the length of a path.

###  2.2 Benchmark Dataset

Data collection. To construct our benchmark dataset, we first collect programs
from LeetCode 111https://leetcode.com, an online platform for evaluating a
programmer’s coding performance. We choose LeetCode as our data source since
it has a clear task description and input constraint for each programming
task. We collect all publicly available tasks on LeetCode (up to Apr. 2024).
For each task, we collect its Python solution from a GitHub project
222https://github.com/walkccc/LeetCode. The repository is under MIT license.
since LeetCode does not publish official solutions. At this stage, we collect
3,123 programs under test.

The goal of our benchmark is to evaluate LLMs’ capability of generating test
cases to cover specific statements/branches. Therefore, we filter out programs
that are too simple (e.g., programs that only have one branch) according to
the cyclomatic complexity [28]. Given the control flow graph of a program, the
cyclomatic complexity $V$ of this program is measured by: $V=e-n+p$, where $e$
is the number of edges in the graph, $n$ is the number of nodes, and $p$ is
the number of connected components. The cyclomatic complexity is positively
correlated with the number of branches/loops in a program. In this work, we
consider programs with the cyclomatic complexity $\geq$ 10\. Finally, we
collect 210 Python programs for our benchmark, consisting of 9 easy problems,
100 medium problems, and 101 hard problems according to LeetCode’s difficulty
label. Each program under test is also paired with its task description in
natural language. Note that most programs already have test cases in their
task descriptions. We remove these cases to prevent LLMs from directly copying
these test cases.

For each program, we perform the following pre-processing steps:

  * •

We add all necessary import statements for the packages required by the
problem solutions.

  * •

Python programmers often split long statements into multiple physical lines.
For all statements split into multiple lines, we remove the newline symbols
from these statements and write them in a single line. This ensures each
statement only corresponds to one line when measuring line coverage.

  * •

We reformat the in-line conditional statements (e.g., the ternary conditional
operator) into multi-line blocks. This ensures that each line of the program
is one statement that belongs to one specific branch.

  * •

We remove natural language comments in the program.

Targeted line/branch/path identification. To obtain targeted lines/branches,
we first extract all conditional branches of a given program based on its
abstract syntax tree (AST). Since loop branches (i.e., for/while loops) are
usually easy to cover, we only consider conditional branches in our task.
Specifically, we extract all if, elif, and else branches. We refer to these
branches as our targeted branches. Then, we consider all statements within
these targeted branches as our targeted lines (see Figure 2 for the example).
Overall, we identified 983 target branches in 210 programs under test (4.7
target branches per program on average). The total number of target lines in
210 programs is 1,312 (6.2 target lines per program on average). The detailed
algorithm for extracting target lines/branches can be found in Appendix B.

In a given program under test, certain branches could be hard to cover without
carefully crafting the test cases. Therefore, we label each targeted branch in
a program as easy, medium, or hard according to the average coverage after
executing 100 randomly generated inputs. The number of easy, medium, and hard
target branches are 498, 225, and 260. We refer to Appendix C for more
details.

For the targeted path coverage task, as the number of execution paths in a
program can be enormous or even undecidable, it is impossible to collect all
execution paths. Instead, we collect the target execution paths from the
example test cases given by LeetCode problem descriptions. For each example
test case, we execute it and record its execution path using all the
condition/loop branches it executed. The complete execution path would be too
long and difficult for LLMs to understand, so we perform clipping after
obtaining full paths. For each execution path, we randomly sample two sub-
paths with at most 5 branches. We further remove duplicated sampled paths,
resulting in an average of 4.1 target paths per question.

##  3 Evaluation

In this section, we first introduce our experiment setup (§ 3.1). Then we
present the results on three benchmark tasks.

###  3.1 Experiment Setup

We evaluate sixteen popular instruction-following LLMs, including both
commercial and open-source ones. The parameter sizes of open-source models
range from 1.3B to 34B. The temperature is set to 0 or 1e-5 (for models on
Huggingface that do not support temperature=0) to ensure that the evaluation
results can be reproduced. All experiments on open-source LLMs are run on two
NVIDIA A6000 GPUs. We set the length limit of outputs to 256 tokens. We use
the pytest-cov 333https://github.com/pytest-dev/pytest-cov to measure the code
coverage.

Table 1: Result on the overall coverage task. The results in () are the improvements over $cov@1$. | Size |  | Correctness |  | Overall coverage |  | Line $cov@k$ |  | Branch $cov@k$
---|---|---|---|---|---|---|---|---|---
|  | syntax | execution |  | line | branch |  | $k=1$ | $k=2$ | $k=5$ |  | $k=1$ | $k=2$ | $k=5$
GPT-3.5-turbo | N/A |  | 100 | 97.43 |  | 96.27 | 93.65 |  | 88.35 |  90.02 | (1.67) |  92.14 | (3.79) |  | 81.87 |  84.32 | (2.45) |  87.55 | (5.68)
GPT-4 | N/A |  | 100 | 92.33 |  | 94.94 | 92.81 |  | 85.65 |  87.77 | (2.12) |  90.04 | (4.39) |  | 78.89 |  81.93 | (3.04) |  85.39 | (6.50)
GPT-4-turbo | N/A |  | 100 | 94.79 |  | 96.08 | 94.81 |  | 85.46 |  87.87 | (2.41) |  90.81 | (5.35) |  | 78.62 |  82.06 | (3.44) |  86.64 | (8.02)
GPT-4o | N/A |  | 99.59 | 98.30 |  | 98.65 | 97.16 |  | 90.23 |  92.16 | (1.93) |  94.33 | (4.10) |  | 84.05 |  86.89 | (2.84) |  90.31 | (6.26)
Gemini-1.0-pro | N/A |  | 93.05 | 71.93 |  | 93.01 | 90.66 |  | 84.48 |  86.60 | (2.12) |  88.47 | (3.99) |  | 78.35 |  81.29 | (2.94) |  84.11 | (5.76)
CodeLlama | 7b |  | 99.52 | 73.86 |  | 86.09 | 81.56 |  | 79.46 |  80.72 | (1.26) |  82.04 | (2.58) |  | 72.28 |  73.96 | (1.68) |  75.90 | (3.62)
13b |  | 67.55 | 50.40 |  | 85.66 | 80.55 |  | 80.49 |  82.26 | (1.77) |  83.44 | (2.95) |  | 73.21 |  75.54 | (2.33) |  77.13 | (3.92)
34b |  | 66.33 | 46.86 |  | 87.96 | 83.74 |  | 78.83 |  81.25 | (2.42) |  83.71 | (4.88) |  | 71.37 |  74.50 | (3.13) |  77.80 | (6.43)
Llama3 | 8b |  | 99.25 | 82.24 |  | 90.98 | 89.02 |  | 77.40 |  80.08 | (2.68) |  84.42 | (7.02) |  | 69.47 |  73.37 | (3.90) |  79.22 | (9.75)
Gemma | 7b |  | 98.98 | 64.64 |  | 93.16 | 91.46 |  | 76.23 |  80.54 | (4.31) |  85.90 | (9.67) |  | 67.15 |  72.94 | (5.79) |  80.29 | (13.14)
Starcoder-2-Instruct | 15b |  | 97.07 | 94.07 |  | 89.84 | 84.41 |  | 88.03 |  88.22 | (0.19) |  88.50 | (0.47) |  | 81.80 |  82.09 | (0.29) |  82.50 | (0.70)
DeepSeek-coder | 1.3b |  | 96.05 | 82.48 |  | 81.22 | 75.99 |  | 75.89 |  76.50 | (0.61) |  77.09 | (1.20) |  | 69.06 |  69.90 | (0.84) |  70.70 | (1.64)
6.7b |  | 97.42 | 82.43 |  | 93.48 | 91.61 |  | 82.40 |  84.74 | (2.34) |  87.97 | (5.57) |  | 75.29 |  78.73 | (3.44) |  83.46 | (8.17)
33b |  | 99.21 | 83.57 |  | 94.86 | 91.92 |  | 85.47 |  87.38 | (1.91) |  90.30 | (4.83) |  | 78.49 |  81.23 | (2.74) |  85.12 | (6.63)
CodeQwen | 7b |  | 100 | 84.26 |  | 90.73 | 86.90 |  | 84.53 |  85.33 | (0.80) |  86.71 | (2.18) |  | 77.66 |  78.94 | (1.28) |  80.95 | (3.29)
Mistral-0.3 | 7b |  | 24.45 | 10.67 |  | 60.59 | 57.73 |  | 53.69 |  55.72 | (2.03) |  57.44 | (3.75) |  | 48.14 |  50.92 | (2.78) |  53.37 | (5.23)

###  3.2 Overall Coverage

In this experiment, we query every model 20 rounds ($N=20$) to generate test
cases (one test case per round) for each program under test. Table 1 shows the
evaluation results on the overall coverage task.

Regarding correctness metrics, we observe that most models can achieve high
syntactical correctness and acceptable execution correctness. Regarding the
coverage performance, fourteen out of sixteen LLMs (except DeepSeek-coder-1.3b
and Mistral-0.3-7b) are able to generate test cases that cover over 80%
lines/branches per program under test. Notably, the latest GPT-4o achieves the
best overall line (98.65%) and branch (97.16%) coverage. We also notice that
the open-source model, DeepSeek-coder-33b, outperforms the commercial LLM,
Gemini-1.0-pro, on both overall line and branch coverage.

We further use $cov@k$ to measure the diversity of each LLM’s generated test
cases. Similar to the overall coverage results, GPT-4o has the best line and
branch $cov@1$, demonstrating its ability to craft complex test cases that are
able to cover most of the program branches within a single attempt. We also
find all LLMs have a higher $cov@2$ and $cov@5$ compared with $cov@1$. This
indicates that the LLMs are able to generate different test cases. Gemma-7b
shows the most significant improvements in the line (+9.67%) and branch
(+13.14%) $cov@5$ compared with its line and branch $cov@1$. We also notice
that Starcoder-2-Instruct has the least improvement on $cov@5$ compared with
$cov@1$ (+0.47% and +0.70% for line and branch coverage, respectively). By
manually checking the test cases generated by Starcoder-2-Instruct, we find
that it frequently repeat previously generated cases despite being instructed
to generate different ones.

Table 2: Results for targeted line coverage. Results in () are the improvements over baselines. | Size |  | Targeted line |  | Baseline: no targeted line
---|---|---|---|---|---
|  | syntax | execution | cov. acc. |  | Syntax | execution | cov. acc.
GPT-3.5-turbo | N/A |  | 99.40 | 95.67 |  67.76 | (-1.27) |  | 100 | 100 | 69.03
GPT-4 | N/A |  | 100 | 98.81 |  78.20 | (10.14) |  | 100 | 99.52 | 68.06
GPT-4-turbo | N/A |  | 99.20 | 98.73 |  80.52 | (11.64) |  | 100 | 100 | 68.88
GPT-4o | N/A |  | 99.63 | 98.96 |  80.97 | (9.48) |  | 100 | 100 | 71.49
Gemini-1.0-pro | N/A |  | 100 | 96.04 |  70.75 | (4.93) |  | 100 | 95.71 | 65.82
CodeLlama | 7b |  | 99.85 | 90.97 |  58.13 | (0.89) |  | 99.52 | 93.81 | 57.24
13b |  | 99.63 | 85.22 |  54.63 | (-4.03) |  | 99.05 | 94.76 | 58.66
34b |  | 98.66 | 90.60 |  59.48 | (-0.29) |  | 100 | 96.19 | 59.77
Llama3 | 8b |  | 98.96 | 85.52 |  60.22 | (-0.60) |  | 99.52 | 95.24 | 60.82
Gemma | 7b |  | 99.78 | 88.21 |  62.91 | (4.92) |  | 99.52 | 89.52 | 57.99
Starcoder-2-Instruct | 15b |  | 98.36 | 92.84 |  64.40 | (-2.39) |  | 100 | 99.05 | 66.79
DeepSeek-coder | 1.3b |  | 98.81 | 91.04 |  58.81 | (2.69) |  | 94.76 | 90.0 | 56.12
6.7b |  | 94.78 | 92.99 |  65.60 | (3.81) |  | 99.05 | 96.67 | 61.79
33b |  | 99.63 | 97.61 |  70.52 | (2.09) |  | 100 | 99.52 | 68.43
CodeQwen | 7b |  | 94.78 | 92.99 |  65.60 | (3.81) |  | 99.05 | 96.67 | 61.79
Mistral-0.3 | 7b |  | 99.40 | 64.40 |  45.60 | (16.94) |  | 99.52 | 40.95 | 28.66

###  3.3 Targeted Line and Branch Coverage

Table 2 and Table 3 show the evaluation results for the targeted line and
branch coverage, respectively. For each subject LLM, we also include a
baseline by excluding the information about the targeted lines/branches in the
text prompt. For each program under test, we reuse the first test case
generated for the overall coverage task and measure its coverage accuracy on
the targeted lines/branches. The intuition is that, if an LLM could not
outperform the baseline, it might be struggling with identifying the
line/branch that is expected to cover when generating the test case.

Regarding the targeted line coverage, we find that GPT-4 series (i.e., GPT-4,
GPT-4-turbo, and GPT-4o) has the best performance improvement (around 10%)
over their baselines. The best-performing LLM is GPT-4o, reaching a coverage
accuracy of 80.97% on average. We also notice that Mistral-0.3 also has the
most significant performance improvement over the baseline (16.94%). However,
considering its low execution correctness, such results may become less
meaningful. We also find that five out of sixteen LLMs do not improve over
their baseline and seven out of sixteen LLMs only have marginal improvement
(less than 5%). These results suggest that the majority of the popular LLMs
may have trouble with multi-step reasoning. Specifically, to reach a specific
line inside a branch, the LLM needs to first identify which the targeted line
belongs to. Then, the LLM needs to generate a valid test input to invoke this
branch. In future work, one may consider leveraging advanced prompting
technologies such as in-context learning and chain-of-thought prompting [29],
or building LLM-based agents to decompose the targeted line coverage task into
multiple fine-grained steps.

Table 3: Results for targeted branch coverage. Results in () are the improvements over the baseline. We omit the correctness metrics of the baseline because they are the same as the targeted line coverage task. | Size |  | Targeted branch |  | Baseline: no targeted branch
---|---|---|---|---|---
|  | Correctness |  | Coverage |  | Coverage
|  | syntax | execution |  | total | easy | medium | hard |  | total | easy | medium | hard
GPT-3.5-turbo | N/A |  | 100 | 98.78 |  | 70.40 | (4.38) | 82.93 | (0.40) | 65.33 | (1.77) | 50.77 | (14.23) |  | 66.02 | 82.53 | 63.56 | 36.54
GPT-4 | N/A |  | 100 | 98.17 |  | 78.23 | (13.33) | 86.14 | (4.41) | 79.56 | (16.89) | 61.92 | (27.30) |  | 64.90 | 81.73 | 62.67 | 34.62
GPT-4-turbo | N/A |  | 100 | 98.67 |  | 80.77 | (15.15) | 88.35 | (5.42) | 79.11 | (16.00) | 67.69 | (33.07) |  | 65.62 | 82.93 | 63.11 | 34.62
GPT-4o | N/A |  | 100 | 99.08 |  | 80.87 | (12.61) | 87.55 | (3.21) | 83.11 | (11.55) | 66.15 | (31.53) |  | 68.26 | 84.34 | 71.56 | 34.62
Gemini-1.0-pro | N/A |  | 100 | 97.04 |  | 68.97 | (5.80) | 82.13 | (2.21) | 69.78 | (6.22) | 43.08 | (12.31) |  | 63.17 | 79.92 | 63.56 | 30.77
CodeLlama | 7b |  | 100 | 81.99 |  | 50.97 | (-4.17) | 64.25 | (-8.04) | 51.11 | (-3.56) | 25.38 | (2.69) |  | 55.14 | 72.29 | 54.67 | 22.69
13b |  | 99.29 | 82.91 |  | 51.58 | (-4.68) | 64.86 | (-7.83) | 46.67 | (-11.55) | 30.39 | (5.78) |  | 56.26 | 72.69 | 58.22 | 24.61
34b |  | 99.39 | 95.02 |  | 63.17 | (5.69) | 78.51 | (2.20) | 60.44 | (6.22) | 36.15 | (11.92) |  | 57.48 | 76.31 | 54.22 | 24.23
Llama3 | 8b |  | 98.88 | 84.94 |  | 58.39 | (-0.31) | 73.09 | (-0.61) | 59.11 | (0.89) | 29.26 | (-1.12) |  | 58.70 | 73.70 | 58.22 | 30.38
Gemma | 7b |  | 99.59 | 85.35 |  | 56.15 | (1.11) | 71.89 | (2.01) | 49.78 | (0.45) | 31.54 | (0.00) |  | 55.04 | 69.88 | 49.33 | 31.54
Starcoder-2-Instruct | 15b |  | 98.68 | 95.42 |  | 64.19 | (-0.41) | 78.71 | (0.20) | 63.56 | (-1.33) | 36.92 | (-0.77) |  | 64.60 | 78.51 | 64.89 | 37.69
DeepSeek-coder | 1.3b |  | 97.05 | 89.32 |  | 54.22 | (0.81) | 68.67 | (1.20) | 52.89 | (-1.78) | 27.69 | (2.31) |  | 53.41 | 67.47 | 54.67 | 25.38
6.7b |  | 96.74 | 93.79 |  | 66.43 | (7.22) | 77.11 | (5.62) | 69.33 | (4.89) | 43.46 | (12.31) |  | 59.21 | 71.49 | 64.44 | 31.15
33b |  | 100 | 97.05 |  | 68.46 | (2.54) | 80.12 | (-2.21) | 66.22 | (4.00) | 48.08 | (10.39) |  | 65.92 | 82.33 | 62.22 | 37.69
CodeQwen | 7b |  | 99.49 | 95.02 |  | 65.82 | (0.51) | 81.12 | (1.60) | 63.56 | (-3.55) | 38.46 | (1.92) |  | 65.31 | 79.52 | 67.11 | 36.54
Mistral-0.3 | 7b |  | 98.17 | 69.28 |  | 46.90 | (20.45) | 58.03 | (25.10) | 40.00 | (16.44) | 31.54 | (15.10) |  | 26.45 | 32.93 | 23.56 | 16.54

We observe a similar trend in the targeted branch coverage (Table 3).
Specifically, the GPT-4 series has the best performance improvement (12%~15%)
over their baselines. GPT4-turbo and GPT-4o are the best-performed LLMs, which
can cover 80.77% and 80.87% branches per program under test on average,
respectively. By contrast, eight LLMs only exhibit marginal improvements and
four LLMs do not improve compared with the baselines. Regarding branches with
different difficulties, we find that branches more likely to be covered by
random test cases are also easier for LLMs to cover (recall we use random
testing to label each branch’s difficulty level in § 2.2). The GPT-4 series
shows the smallest performance gap between branches with different difficulty
levels. We also notice that twelve out of sixteen LLMs show the largest
performance improvements over the baselines on those “hard” branches. These
results indicate that providing additional branch information can indeed help
us to cover branches that are hard to reach by random testing.

###  3.4 Targeted Path Coverage

Table 4 presents the results of the targeted path coverage task. We adopt a
similar baseline as in our targeted line/branch coverage tasks by excluding
the targeted path in the text prompts. Overall, GPT-4o and Gemini-1.0-pro have
the best performance on the path coverage, reaching 56.67% and 56.09% on
average, respectively. However, they do not outperform their baselines.
Generally, we do not find any LLMs except Mistral-0.3 show obvious performance
improvement (more than 5%) on the path coverage compared with the baselines.
Nine out of sixteen LLMs do not outperform the baselines. Regarding the path
similarity, we also do not find any LLMs exhibiting large performance
improvement compared with the baselines. These results suggest that
comprehending the program logic and identifying a specific execution path is
still a challenging task for the current LLMs.

Targeted path coverage is considerably more complicated compared with overall
coverage and targeted line/branch coverage. Specifically, the LLM needs to
identify a sequence of multiple branches, and create a test input that can
execute these branches following a certain order, which is challenging even
for human programmers. In future work, to fully unleash an LLM’s capability of
generating test cases covering specific program execution paths, designing
more complex prompts and reasoning frameworks is worth investigating.

Table 4: Results for targeted path coverage. Results in () are the improvements over the baseline. | Size |  | Given target path |  | Baseline: no target path
---|---|---|---|---|---
|  | syntax | execution | path cov | path similarity |  | path cov | path similarity
GPT-3.5-turbo | N/A |  | 99.88 | 98.95 | 49.30(-5.97) | 77.35(-2.39) |  | 55.27 | 79.74
GPT-4 | N/A |  | 100 | 99.18 | 54.10(-0.94) | 80.77(3.23) |  | 55.04 | 77.54
GPT-4-turbo | N/A |  | 100 | 99.41 | 50.47(-3.74) | 79.82(1.08) |  | 54.21 | 78.74
GPT-4o | N/A |  | 100 | 99.53 |  56.67(-1.76) |  82.35(1.29) |  | 58.43 | 81.06
Gemini-1.0-pro | N/A |  | 100 | 96.02 | 56.09(0.70) | 77.59(-0.23) |  | 55.39 | 77.82
CodeLlama | 7b |  | 99.76 | 90.98 | 41.57(-1.05) | 67.66(0.14) |  | 42.62 | 67.52
13b |  | 99.06 | 82.67 | 38.88(-5.50) | 61.76(-7.85) |  | 44.38 | 69.61
34b |  | 98.83 | 92.15 | 45.32(0.24) | 69.52(-0.12) |  | 45.08 | 69.64
Llama3 | 8b |  | 98.24 | 89.46 | 41.92(1.29) | 68.03(0.40) |  | 40.63 | 67.63
Gemma | 7b |  | 100 | 88.06 | 37.11(4.09) | 64.54(2.29) |  | 33.02 | 62.25
Starcoder-2-Instruct | 15b |  | 96.83 | 90.28 | 48.48(-5.38) | 70.91(-6.78) |  | 53.86 | 77.69
DeepSeek-coder | 1.3b |  | 97.89 | 88.99 | 40.16(0.46) | 64.91(0.67) |  | 39.70 | 64.24
6.7b |  | 99.06 | 95.90 | 53.04(0.23) | 76.77(1.56) |  | 52.81 | 75.21
33b |  | 100 | 96.49 | 54.10(-4.33) | 77.99(-2.73) |  | 58.43 | 80.72
CodeQwen | 7b |  | 99.77 | 94.96 | 55.97(-3.16) | 77.46(-2.67) |  | 59.13 | 80.13
Mistral-0.3 | 7b |  | 97.19 | 45.32 | 22.01(5.38) | 32.98(3.69) |  | 16.63 | 29.29

##  4 Related Work

Code-related Benchmarks for LLMs. In recent years, researchers have endeavored
to develop more rigorous and comprehensive evaluation frameworks for LLMs on
coding abilities from various perspectives. One of the earliest attempts is
HumanEval [25], which consists of 164 hand-craft programming challenges that
evaluate LLMs’ ability to understand natural language descriptions and
generate the corresponding functional correct code. Since then, there have
been several studies attempting to construct benchmarks with more diverse
problems [26], more rigorous evaluations [30], and more complex scenarios [31,
32]. Beyond these established code-generation scenarios, numerous studies are
expanding their focus to include a broader range of real-world applications,
such as reviewing code [33], performing repo-level code completion [34, 35,
36, 37], and resolving GitHub issues [38]. While all the aforementioned
studies examine the coding abilities of LLMs from different perspectives, none
specifically target test case generation, a crucial phase in the software
engineering lifecycle. The most relevant study is DevBench [39], which
evaluates LLMs across software development stages, including testing. Unlike
DevBench, our benchmark provides more comprehensive evaluations specifically
tailored to test case generation using coverage-guided tasks and includes a
broader range of studied models.

LLMs for Software Testing. Recent studies have extensively utilized LLMs to
develop efficient and effective testing pipelines for various software
applications [40, 41]. Unit test case generation [42], which aims to test
individual software components independently, is the primary focus of current
LLM-aided software testing. One line of research tries to pre-train/fine-tune
LLMs on focal methods and related assertion statements to enhance their test-
generation capabilities [43, 44, 45, 46]. Although effective, these methods
can be cost-intensive and challenging to scale. Alternatively, some
researchers focus on crafting effective prompts that instruct LLMs to analyze
relevant information, such as code [24, 23, 47, 48, 49] or documentation [50,
51], and generate test cases accordingly. Such LLM-enabled testing has
achieved state-of-the-art performance compared to traditional tools [22].
Beyond unit test case generation, related works explore creating test inputs
for specific types of software systems. For example, LLMs have been proven
effective in testing deep learning libraries [52, 53, 54], compilers [55], OS
kernels [56], SMT solvers [57], and graphical user interfaces [58, 59, 60],
etc. These studies highlight the potential of LLMs as core components of
general-purpose intelligent testing engines.

##  5 Conclusion and Future Work

We present TestEval, a novel benchmark for evaluating automated test case
generation with LLMs for Python programs. Our benchmark dataset is created
from on online programming platform LeetCode, consisting of 210 Python
programs. Base on this dataset, we propose three different tasks and
standardized evaluation pipelines. Our targeted branch/line/path tasks enable
the evaluation and assessment on the LLM’s capabilities in comprehending
complex program logic and execution path and generating test cases following
the tester’s intent, which is not considered in previous works on either
benchmarking LLMs or test case generation with LLMs. To contribute to the
broader LLM and SE research communities, we release our dataset and benchmark
pipelines on https://llm4softwaretesting.github.io.

We further conduct extensive experiments with sixteen popular LLMs on
TestEval. We find that although LLMs can achieve high overall coverage by
generating diverse test cases, generating test cases to cover a specific
branch/line/path is still challenging. Our results reveal that there is a
common lack of abilities in comprehending program logic among current LLMs,
despite their promising performance on other code-related tasks (e.g., code
generation).

In the future, we plan to maintain an open leaderboard and continuously
evaluate new LLMs. We also plan to compare LLMs with existing traditional test
case generation tools, as well as crafting more advanced prompt templates and
reasoning frameworks to enable LLMs for better test case generation.

## References

  * [1] Ermira Daka and Gordon Fraser.  A survey on unit testing practices and problems.  In 2014 IEEE 25th International Symposium on Software Reliability Engineering, pages 201–211. IEEE, 2014.
  * [2] Vitaly Chipounov, Volodymyr Kuznetsov, and George Candea.  S2e: A platform for in-vivo multi-path analysis of software systems.  Acm Sigplan Notices, 46(3):265–278, 2011.
  * [3] Cristian Cadar, Patrice Godefroid, Sarfraz Khurshid, Corina S Păsăreanu, Koushik Sen, Nikolai Tillmann, and Willem Visser.  Symbolic execution for software testing in practice: preliminary assessment.  In Proceedings of the 33rd International Conference on Software Engineering, pages 1066–1071, 2011.
  * [4] Siddhartha R Dalal, Ashish Jain, Nachimuthu Karunanithi, JM Leaton, Christopher M Lott, Gardner C Patton, and Bruce M Horowitz.  Model-based testing in practice.  In Proceedings of the 21st international conference on Software engineering, pages 285–294, 1999.
  * [5] Jeff Offutt and Aynur Abdurazik.  Generating tests from uml specifications.  In International Conference on the Unified Modeling Language, pages 416–429. Springer, 1999.
  * [6] Carlos Pacheco and Michael D Ernst.  Randoop: feedback-directed random testing for java.  In Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion, pages 815–816, 2007.
  * [7] Lei Ma, Cyrille Artho, Cheng Zhang, Hiroyuki Sato, Johannes Gmeiner, and Rudolf Ramler.  Grt: Program-analysis-guided random testing (t).  In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 212–223. IEEE, 2015.
  * [8] Gordon Fraser and Andrea Arcuri.  Evosuite: automatic test suite generation for object-oriented software.  In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering, pages 416–419, 2011.
  * [9] Luciano Baresi and Matteo Miraz.  Testful: Automatic unit-test generation for java classes.  In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 2, pages 281–284, 2010.
  * [10] Gordon Fraser and Andreas Zeller.  Mutation-driven generation of unit tests and oracles.  In Proceedings of the 19th international symposium on Software testing and analysis, pages 147–158, 2010.
  * [11] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al.  Code llama: Open foundation models for code.  arXiv preprint arXiv:2308.12950, 2023.
  * [12] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al.  Starcoder: may the source be with you!  arXiv preprint arXiv:2305.06161, 2023.
  * [13] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang.  Wizardcoder: Empowering code large language models with evol-instruct.  arXiv preprint arXiv:2306.08568, 2023.
  * [14] Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang.  Magicoder: Source code is all you need.  arXiv preprint arXiv:2312.02120, 2023.
  * [15] Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al.  Starcoder 2 and the stack v2: The next generation.  arXiv preprint arXiv:2402.19173, 2024.
  * [16] Weixiang Yan, Yuchen Tian, Yunzhe Li, Qian Chen, and Wen Wang.  CodeTransOcean: A comprehensive multilingual benchmark for code translation.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5067–5089, Singapore, December 2023. Association for Computational Linguistics.
  * [17] Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pouguem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and Reyhaneh Jabbarvand.  Lost in translation: A study of bugs introduced by large language models while translating code.  In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13, 2024.
  * [18] Hasan Ferit Eniser, Hanliang Zhang, Cristina David, Meng Wang, Brandon Paulsen, Joey Dodds, and Daniel Kroening.  Towards translating real-world code with llms: A study of translating to rust.  arXiv preprint arXiv:2405.11514, 2024.
  * [19] Weisong Sun, Chunrong Fang, Yudu You, Yuchen Chen, Yi Liu, Chong Wang, Jian Zhang, Quanjun Zhang, Hanwei Qian, Wei Zhao, et al.  A prompt learning framework for source code summarization.  arXiv preprint arXiv:2312.16066, 2023.
  * [20] Toufique Ahmed, Kunal Suresh Pai, Premkumar Devanbu, and Earl Barr.  Automatic semantic augmentation of language model prompts (for code summarization).  In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13, 2024.
  * [21] Rajarshi Haldar and Julia Hockenmaier.  Analyzing the performance of large language models on code summarization.  In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors, Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 995–1008, Torino, Italia, May 2024. ELRA and ICCL.
  * [22] Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen.  Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models.  In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 919–931. IEEE, 2023.
  * [23] Zhuokui Xie, Yinghao Chen, Chen Zhi, Shuiguang Deng, and Jianwei Yin.  Chatunitest: a chatgpt-based automated unit test generation tool.  arXiv preprint arXiv:2305.04764, 2023.
  * [24] Zhiqiang Yuan, Yiling Lou, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, and Xin Peng.  No more manual tests? evaluating and improving chatgpt for unit test generation.  arXiv preprint arXiv:2305.04207, 2023.
  * [25] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.  Evaluating large language models trained on code.  arXiv preprint arXiv:2107.03374, 2021.
  * [26] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al.  Program synthesis with large language models.  arXiv preprint arXiv:2108.07732, 2021.
  * [27] Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang.  Cruxeval: A benchmark for code reasoning, understanding and execution.  arXiv preprint arXiv:2401.03065, 2024.
  * [28] Thomas J McCabe.  A complexity measure.  IEEE Transactions on software Engineering, (4):308–320, 1976.
  * [29] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  Advances in neural information processing systems, 35:24824–24837, 2022.
  * [30] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang.  Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.  Advances in Neural Information Processing Systems, 36, 2024.
  * [31] Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu.  Ds-1000: A natural and reliable benchmark for data science code generation.  In International Conference on Machine Learning, pages 18319–18345. PMLR, 2023.
  * [32] Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et al.  Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x.  arXiv preprint arXiv:2303.17568, 2023.
  * [33] Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, et al.  Automating code review activities by large-scale pre-training.  In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 1035–1047, 2022.
  * [34] Tianyang Liu, Canwen Xu, and Julian McAuley.  Repobench: Benchmarking repository-level code auto-completion systems.  arXiv preprint arXiv:2306.03091, 2023.
  * [35] Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen.  RepoCoder: Repository-level code completion through iterative retrieval and generation.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2471–2484, Singapore, December 2023. Association for Computational Linguistics.
  * [36] Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian McAuley.  Longcoder: A long-range pre-trained language model for code completion.  In International Conference on Machine Learning, pages 12098–12107. PMLR, 2023.
  * [37] Yangruibo Ding, Zijian Wang, Wasi Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, et al.  Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion.  Advances in Neural Information Processing Systems, 36, 2024.
  * [38] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan.  Swe-bench: Can language models resolve real-world github issues?  In The Twelfth International Conference on Learning Representations, 2023.
  * [39] Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, et al.  Devbench: A comprehensive benchmark for software development.  arXiv preprint arXiv:2403.08604, 2024.
  * [40] Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang.  Universal fuzzing via large language models.  arXiv preprint arXiv:2308.04748, 2023.
  * [41] Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang.  Software testing with large language models: Survey, landscape, and vision.  IEEE Transactions on Software Engineering, 2024.
  * [42] Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip.  An empirical evaluation of using large language models for automated unit test generation.  IEEE Transactions on Software Engineering, 2023.
  * [43] Saranya Alagarsamy, Chakkrit Tantithamthavorn, and Aldeida Aleti.  A3test: Assertion-augmented automated test case generation.  arXiv preprint arXiv:2302.10352, 2023.
  * [44] Sepehr Hashtroudi, Jiho Shin, Hadi Hemmati, and Song Wang.  Automated test case generation using code models and domain adaptation.  arXiv preprint arXiv:2308.08033, 2023.
  * [45] Nikitha Rao, Kush Jain, Uri Alon, Claire Le Goues, and Vincent J Hellendoorn.  Cat-lm training language models on aligned code and tests.  In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 409–420. IEEE, 2023.
  * [46] Benjamin Steenhoek, Michele Tufano, Neel Sundaresan, and Alexey Svyatkovskiy.  Reinforcement learning from automatic feedback for high-quality unit test generation.  arXiv preprint arXiv:2310.02368, 2023.
  * [47] Ying Zhang, Wenjia Song, Zhengjie Ji, Na Meng, et al.  How well does llm generate security tests?  arXiv preprint arXiv:2310.00710, 2023.
  * [48] Vincent Li and Nick Doiron.  Prompting code interpreter to write better unit tests on quixbugs functions.  arXiv preprint arXiv:2310.00483, 2023.
  * [49] Arghavan Moradi Dakhel, Amin Nikanjam, Vahid Majdinasab, Foutse Khomh, and Michel C Desmarais.  Effective test generation using pre-trained large language models and mutation testing.  Information and Software Technology, 171:107468, 2024.
  * [50] Vasudev Vikram, Caroline Lemieux, and Rohan Padhye.  Can large language models write good property-based tests?  arXiv preprint arXiv:2307.04346, 2023.
  * [51] Laura Plein, Wendkûuni C Ouédraogo, Jacques Klein, and Tegawendé F Bissyandé.  Automatic generation of test cases based on bug reports: a feasibility study with large language models.  In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, pages 360–361, 2024.
  * [52] Sungmin Kang, Juyeon Yoon, and Shin Yoo.  Large language models are few-shot testers: Exploring llm-based general bug reproduction.  In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 2312–2323. IEEE, 2023.
  * [53] Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang.  Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models.  In Proceedings of the 32nd ACM SIGSOFT international symposium on software testing and analysis, pages 423–435, 2023.
  * [54] Yinlin Deng, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shujing Yang, and Lingming Zhang.  Large language models are edge-case generators: Crafting unusual programs for fuzzing deep learning libraries.  In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pages 1–13, 2024.
  * [55] Chenyuan Yang, Yinlin Deng, Runyu Lu, Jiayi Yao, Jiawei Liu, Reyhaneh Jabbarvand, and Lingming Zhang.  White-box compiler fuzzing empowered by large language models.  arXiv preprint arXiv:2310.15991, 2023.
  * [56] Chenyuan Yang, Zijie Zhao, and Lingming Zhang.  Kernelgpt: Enhanced kernel fuzzing via large language models.  arXiv preprint arXiv:2401.00563, 2023.
  * [57] Maolin Sun, Yibiao Yang, Yang Wang, Ming Wen, Haoxiang Jia, and Yuming Zhou.  Smt solver validation empowered by large pre-trained language models.  In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 1288–1300. IEEE, 2023.
  * [58] Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Zhilin Tian, Yuekai Huang, Jun Hu, and Qing Wang.  Testing the limits: Unusual text inputs generation for mobile app crash detection with large language model.  In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–12, 2024.
  * [59] Zhe Liu, Chunyang Chen, Junjie Wang, Xing Che, Yuekai Huang, Jun Hu, and Qing Wang.  Fill in the blank: Context-aware automated text input generation for mobile gui testing.  In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1355–1367. IEEE, 2023.
  * [60] Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Xing Che, Dandan Wang, and Qing Wang.  Make llm a testing expert: Bringing human-like interaction to mobile gui testing via functionality-aware decisions.  In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13, 2024.

##  Appendix A Prompt Templates

The prompt templates for TestEval tasks are shown as follows. For the targeted
line/branch/path coverage tasks, we add line numbers to both the program under
test and the target information in order to accurately locate the position of
the target line/branch/path. Notice that our prompt template is only a primary
setting without advanced prompting techniques such as few-shot examples or
chain-of-thought reasoning, and we encourage future researchers to design more
advanced prompts for TestEval.

[h]  Overall coverage Please write a test method for the function
‘{func_name}’ given the following program under test and function description.
Your answer should only contain one test input. Program under test: —-
{program} —- Function description for ‘{func_name}’: —- description —- Your
test method should begin with: def test_func_name(): solution=Solution()
Prompt for generating the next test case: Generate another test method for the
function under test. Your answer must be different from previously-generated
test cases, and should cover different statements and branches.

[h]  Targeted line coverage Please write a test method for the function
‘{func_name}’ given the following program under test and function description.
Your answer should only contain one test input. Program under test: —-
{program} —- Function description for ‘{func_name}’: —- description —- Your
test case must cover line {target_line}. Your test method should begin with:
def test_func_name(): solution=Solution()

[h]  Targeted branch coverage Please write a test method for the function
‘{func_name}’ given the following program under test and function description.
Your answer should only contain one test input. Program under test: —-
{program} —- Function description for ‘{func_name}’: —- description —- Your
test case must cover the branch {target_branch}. Your test method should begin
with: def test_func_name(): solution=Solution()

[h]  Targeted path coverage Please write a test method for the function
‘{func_name}’ given the following program under test and function description.
Your answer should only contain one test input. Program under test: —-
{program} —- Function description for ‘{func_name}’: —- description —- Your
test case must cover the following execution path in function {func_name}. The
path is a sequence of branch conditions. When executing your test case, each
branch condition in the target execution path must be satisfied sequentially.
Target execution path: {target_path} —- Your test method should begin with:
def test_func_name(): solution=Solution()

##  Appendix B Targeted Line/Branch Identification

The complete algorithm for extracting targeted lines/branches from a program
under test is shown in Algorithm 2. At a high level, we first extract all
conditional branches by locating the branches start with conditional operators
(i.e., ‘if’, ‘elif’, and ‘else’) through parsing the program’s abstract syntax
tree. For each branch, we record the line numbers of its first and last lines
(e.g., Lines 1:5) as one targeted branch. Then, we record the line numbers of
all lines (except the line that only includes the ‘else’ operator) within this
branch as the targeted lines (e.g., [1, 2, 3, 4, 5]). We repeat this process
until finishing parsing all branches of a program.

Input: Program with $L$ lines: $p=\left\\{s_{1},s_{2},...,s_{L}\right\\}$

Output: Target lines $ls$, target branches $bs$

$ls$ = [], $bs$ = [], $i=1$;

while  _$i <=L$_ do

if  _$s_{i}$ starts with ‘if’, ‘elif’, or ‘else’_ then

$curent\\_branch$ = [] ;

$j=i$ ;

repeat

$curent\\_branch$.append($j$) ;

$j=j+1$

until  _$s_{j}$ not in this branch_;

$bs$.append($curent\\_branch$);

end if

end while

for  _$target\\_branch$ in $bs$_ do

for  _line $s_{i}$ in $target\\_branch$_ do

if  _$s_{i}$ is inside a branch and not $s_{i}$ starts with ’else’_ then

$ls$.append($i$);

end if

end for

end for

return _$ls$ , $bs$_

Algorithm 2 Targeted Line/Branch Identification.

##  Appendix C Input Generator for Measuring Branch Difficulties

Figure 4: The input constraints for a LeetCode problem (left) and its random
input generator for TestEval (right).

For each problem in TestEval, we construct a _random input generator_ to
assess the difficulties of covering a specific branch. Each generator is a
Python program that _uniformly_ samples a valid test input for the LeetCode
problem according to its constraint description. We leverage GPT-4 to
initially generate these programs, followed by manual inspection and
correction to ensure they adhere to the problem’s constraints. An example of
an input generator is shown in Figure 4. These generators are then used to
sample 100 executable test cases for each problem, which are employed in the
targeted branch coverage task. Branch difficulty is determined by the
frequency at which a branch is covered across the 100 sampled test inputs. We
categorize branches as follows: easy (covered by $[99\%,100\%]$ of test
cases), medium (covered by $[40\%,99\%)$ of test cases), and hard (covered by
$[0,40\%)$ of test cases). This partitioning ensures that easy branches do not
significantly outnumber other categories, and promotes a balanced distribution
between medium and hard branches.