# Visualizing Neural Network Imagination

Nevan Wichers
Google
Victor Tao
University of Toronto
Riccardo Volpato
Satalia
Fazl Barez
University of Oxford

###### Abstract

In certain situations, neural networks will represent environment states in
their hidden activations. Our goal is to visualize what environment states the
networks are representing. We experiment with a recurrent neural network (RNN)
architecture with a decoder network at the end. After training, we apply the
decoder to the intermediate representations of the network to visualize what
they represent. We define a quantitative interpretability metric and use it to
demonstrate that hidden states can be highly interpretable on a simple task.
We also develop autoencoder and adversarial techniques and show that benefit
interpretability.

##  1 Introduction

When trained to predict the end state of a given sequence, neural networks
(NNs) may learn to predict intermediate states even if not explicitly trained
to do so. For example, a network trained to predict the end state of an
environment where the easiest way is to predict it one step at a time will
most likely generate hidden representations of the intermediate steps.
Consider a NN trained to predict the 4th state of Conway’s Game of Life (GoL)
(Conway, 1970) given an initial state. GoL is a cellular automata with simple
deterministic rules to obtain the next state from the previous. The network
could learn the GoL rules and apply them to the 1st state to obtain a
representation of the 2nd state, to the 2nd state to obtain the 3rd state and
so on. The states before the 4th state would be represented only internally in
the network, since it is only trained to output the 4th state. Another case
where the network might represent environment states is when making a
decision. For example, if a reinforcement learning (RL) agent needs to decide
whether to go left or right in a maze, the NN might predict what the state of
the environment will be after going left. The network may represent this state
in its hidden activations and use it to decide how to turn. Our goal is to
visualize the environment states that a network is representing. In this case,
it is like the network is imagining the consequence of its action. When a
network is making a decision, visualizing the states it is considering can
help evaluate if the network is making a safe and correct decision and for the
right reasons. To our knowledge, this interpretability problem has not been
explored before.

In this paper we focus on visualizing the network’s intermediate
representations when predicting GoL states. We use an RNN architecture with an
encoder in the beginning and a decoder at the end. We train the network to
predict a future GoL state given the initial GoL state. After training, we run
inference with a hidden state of the RNN fed into the decoder. We hypothesize
that the decoder will produce an image similar to an intermediate GoL state.
We use the GoL because we know the ground truth of the intermediate GoL states
so we can accurately evaluate our technique. For more interesting problems we
cannot know what intermediate states the network is representing in advance.

###  1.1 Contributions

  1. 1.

We introduce the task of visualizing the intermediate environment states
represented by a neural network.

  2. 2.

We develop an adversarial training and autoencoder method for visualizing
intermediate representations.

  3. 3.

We present results to show how the network architecture and training decisions
impact interpretability.

  4. 4.

We show that the network often learns to predict intermediate states earlier
than it learns to predict the state it is trained to.

##  2 Methodology

In the following sections, we describe how our method applies to the GoL for
clarity. However, our method is general-purpose and applicable to other
domains like the maze example we described.

###  2.1 Architecture and training

The left side of figure 1 shows our architecture with 3 RNN timesteps and 3
GoL timesteps. By timestep we mean a transition from one state to the next in
the GoL or the RNN. We also experiment with different numbers of RNN and GoL
timesteps. Note that the network is only trained to predict the final game of
life state, not the intermediate ones. We generate the first GoL state
randomly111We randomly generate the 1s and 0s of the state. Sometimes we use
the rules of GoL to generate the state to produce interesting patterns in the
first state.. Then we use the rules of GoL to calculate the $k$-th state. We
use pairs of the first GoL state and the $k$-th state as training data. Each
state is an image with 1 color channel where each pixel is either 0 or 1.

Figure 1:  Our setup with 3 GoL and 3 model timesteps. Left: Our setup during
training. The RNN layer blocks share weights. The Encoder, RNN layers and
Decoder are all convolutional. Middle: Our autoencoder training setup. Right:
Our setup during inference. Our hypothesis is that the states that are
generated when applying the decoder to the intermediate timesteps are similar
to the intermediate GoL states.

###  2.2 Autoencoder training

The encoder and decoder are also trained together like an autoencoder as shown
in the middle of figure 1. We use L2 loss to train the combined encoder
decoder to recreate the 1st GoL state. The autoencoder loss is averaged with
the regular training loss. Our motivation is to make the encoder use a
representation which the decoder can decode. We hope that this will encourage
the intermediate RNN layers to also use a representation which the decoder can
decode into a valid intermediate state.

###  2.3 Adversarial decoder training

We also try to make the inferred intermediate look like GoL states by training
the decoder like a generator in a Generative Adversarial Network (GAN)
(Goodfellow et al., 2014). Specifically, we train a discriminator to
differentiate between real GoL states and states generated by the decoder. At
the same time, we train the decoder to fool the discriminator. We apply this
method with a pretrained RNN and encoder. We use the GAN loss, in addition to
the autoencoder and final prediction losses to update only the weights of the
decoder, leaving the encoder and RNN unaffected. Note that this method does
not train the decoder to output a correct intermediate GoL state, only
something that looks like a GoL state.

###  2.4 Inference with decoder

During training, the output of the intermediate RNN timesteps are only fed
into the next RNN timestep. During inference, the output of the intermediate
RNN timesteps are also fed into the decoder as shown on the right of figure 1.
We hypothesize that applying the decoder to the intermediate states like this
will generate the intermediate GoL states. We do not run inference on the
output from the encoder or the final RNN timestep because these are already
trained to match respectively the first and last GoL states.

##  3 Results

###  3.1 Metric

We define a metric to determine how closely the network represents the
intermediate GoL states. We consider a GoL sequence with $m$ states
$\\{s_{1}\ldots s_{m}\\}$ and feed $s_{1}$ into the inference network to
generate $\\{h_{1}\ldots h_{n}\\}$ outputs from the $n$ hidden states of the
network. We threshold each pixel of network outputs at .5 so they become
either 0 or 1, since GoL states have binary pixel values. Finally, we count
the number of $h_{i}$ outputs that match any $s_{j}$ GoL state and divide it
by the minimum between $n$ and $m-2$ (not $m$ since we exclude $s_{1}$ and
$s_{m}$) as shown in this equation:

$\frac{\sum_{i=1}^{n}\sum_{j=2}^{m-1}f(h_{i},s_{j})}{\min(n,m-2)}\\\ $ (1)

$f$ is a function which determines if two states match. If less than 95% of
the pixels in the two inputs of $f(h_{i},s_{j})$ are equal $f$ gives 0.
Otherwise, $f$ gives .5 if another generated state already matched the GoL
state, and 1 if it did not. This gives credit for states in any order
matching, because when the number of model and GoL timesteps are different it
is not clear which states should match. Note that the calculated metric could
be greater than 1 if $n>m-2$. Each metric value that we report is the average
over ten thousand GoL sequences for each run. See appendix B for examples of
metric values of state sequences.

###  3.2 Results with same number of model and GoL timesteps

First, we evaluate our technique with the same number of GoL and model
timesteps. For each run, we try our technique with and without adversarial
training, and report the maximum of two the metric values. We only include
trainings in this analysis where the network reached 99% training accuracy in
predicting the final GoL state. We measure the mean metric value and impact of
the following hyper-parameters in table 1:

  * •

The number of timesteps in GoL and model (Num timesteps)

  * •

Whether the network was trained with an autoencoder loss (Use Autoencoder)

  * •

Whether corresponding layers of different timesteps share the weights (Use
RNN)

Num timesteps | Use Autoencoder | Use RNN | Number of Runs | Mean Metric Value
---|---|---|---|---
2 | True | True | 6 | 1.00
3 | True | True | 42 | 0.99
4 | True | True | 33 | 0.93
3 | False | True | 22 | 0.56
3 | True | False | 20 | 0.40
Table 1:  The first 3 rows show that interpretability gets slightly worse with
more timesteps. The last two rows show that the autoencoder and RNN are
helpful. We average over Number of runs different training runs of the
network.

We also do an evaluation where we train 15 versions of the network with 3
timesteps and measure how many training steps it takes each predicted state to
reach 98% similarity with the corresponding ground truth state (the states
should correspond since the number of timesteps are equal). On average, the
3rd state reaches 98% accuracy 2773 training steps before the 4th state. The
2nd state reaches 98% accuracy 5533 training steps before the 4th state. The
network is learning to predict the intermediate states before the final one,
even though it was trained to predict the final state, but not the
intermediate ones. We think the network is learning to predict the
intermediate states first because they are necessary for predicting the final
state.

We also measure the impact of other hyperparameters. We find that having 20
channels in our CNN layers improves the metric by 0.09 over larger numbers of
channels. We also find that having 1 layer in the encoder and decoder improves
the metric by 0.14 compared to 3 layers.

###  3.3 Results with a different number of model and GoL timesteps

We also measure the results when the number of model and GoL timesteps are
different as shown in table 2. This is a more difficult problem because the
model cannot simply use each model timestep to predict one GoL timestep.

GoL timesteps | Model timesteps | Number of runs | Mean metric value
---|---|---|---
2 | 3 | 16 | 0.73
3 | 2 | 22 | 0.74
Table 2: Results where the number of GoL and model timesteps are different.
Note that the runs with 2 GoL timesteps and 3 model timesteps can have a
metric value of 1.5 if both of their intermediate model states match the
single intermediate ground truth state. These runs use the RNN and the
autoencoder.

###  3.4 Impact of adversarial training

GoL timesteps |  Model timesteps |  Use auto-encoder |  Use RNN |  Number of runs |  Mean without adversarial |  Mean with adversarial
---|---|---|---|---|---|---
2 |  3 |  True |  True |  16 |  0.58 |  0.65
3 |  2 |  True |  True |  22 |  0.15 |  0.71
3 |  3 |  True |  False |  20 |  0.11 |  0.40
3 |  3 |  False |  True |  10 |  0.44 |  0.43
Table 3: Results on a subset of experiments with and without the adversarial
training technique. The adversarial training technique improves the results in
every case except when we do not use the autoencoder.

We measure the impact of our adversarial training technique and show it in
table 3. The technique usually improves results, sometimes by about 4x.
Adversarial training helps more in the cases where the interpretability score
is low without it.

##  4 Related work

nostalgebraist (2020) also applies the decoder to intermediate layers to help
interpretability. The difference with our work is that they apply the
technique to a different architecture and do not have the goal of visualizing
intermediate environment states. They also do not use an adversarial technique
or have results showing how architecture decisions impact interpretability.
See appendix A for more related work.

##  5 Conclusion and future work

In our research, we explore a new interpretability problem of visualizing the
hidden intermediate states that a network represents. We show that in a simple
environment using an autoencoder and adversarial techniques, we can obtain
hidden states that closely match the ground truth states of the environment.
However, the idea didn’t end up working out when we tried it on a larger
neural network trained on chess, highlighting the limitations of our approach
for more complex models and domains. In future work, we plan to investigate
ways to extend our technique to handle such complex scenarios, where the
network may represent the consequences of a decision it is considering.

## References

  * Conway (1970) John Conway.  The game of life.  _Scientific American_ , 223(4):4, 1970.
  * Goodfellow et al. (2014) Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  _arXiv preprint arXiv:1406.2661_ , 2014.
  * Mordvintsev et al. (2020) Alexander Mordvintsev, Ettore Randazzo, Eyvind Niklasson, and Michael Levin.  Growing neural cellular automata.  _Distill_ , 2020.  doi: 10.23915/distill.00023.  https://distill.pub/2020/growing-ca.
  * nostalgebraist (2020) nostalgebraist.  Interpreting gpt: the logit lens.  _https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens_ , 2020\.
  * Simonyan et al. (2014) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps, 2014.
  * Szegedy et al. (2014) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks, 2014.
  * Yosinski et al. (2015) Jason Yosinski, Jeff Clune, Anh Mai Nguyen, Thomas J. Fuchs, and Hod Lipson.  Understanding neural networks through deep visualization.  _CoRR_ , abs/1506.06579, 2015.  URL http://arxiv.org/abs/1506.06579.
  * Zeiler & Fergus (2013) Matthew D. Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  _CoRR_ , abs/1311.2901, 2013.  URL http://arxiv.org/abs/1311.2901.

##  Appendix A More related work

NNs have produced unprecedented advances in recent decades in a variety of
tasks. However, their inner workings are still unclear. A large number of
previous works have focused on visualizing NNs outputs, activation function
and hidden layers. For example, Yosinski et al. (2015) provides a tool to
visualize neurons in pre-trained Convolutional NNs (CNNs). Their method allows
for deeper local understanding of neuron computations. This tool only works
for images and videos.

Others such as Szegedy et al. (2014) show that CNNs encoding maintains
photographically accurate information about the original input several layers
in the network. Their approach uses shallow and deep representation to invert
an objective function with Stochastic Gradient Descent (SGD) that allows them
to reconstruct the original input. Our method also sheds light on what the
intermediate layers of a network represent.

Another popular approach for unpacking the inner-workings of NNs has been via
visualization. More specifically, the use of saliency maps (Simonyan et al.,
2014) highlights an area (map) of the image such that the classifier (e.g.
CNN) can discriminate given a class. This method has proved to be a powerful
tool for image region segmentation.

More recently, Mordvintsev et al. (2020) investigated learning a cellular
automata update rule. They use a recurrent network with simplified CNN-like
layers, with ‘per-pixel’ dropout. They train it on the task of generating
target images. Their visualizations show that the network also learns
interpretable intermediate representations. One difference with our method is
that we use a standard CNN architecture instead of a modified one. We also use
a decoder which we apply to the intermediate representations, where they treat
the last 3 channels as the visualization instead of using a decoder. We also
apply our technique to a domain with complex intermediate states, while their
intermediate states are usually subsets of the target image. We plan to
further test our technique by applying it in the domain of generating target
images.

Finally, the authors of Zeiler & Fergus (2013) consider the task of
visualizing the intermediate feature layers of a CNN as well as the operation
used for classification. Their approach shows significant improvement on
ImageNet. Our work differs in that instead of trying to extract specific
features from the hidden states, we focus on reconstructing environment
states.

##  Appendix B Results samples

These are cherry picked examples to illustrate how our metric works.

Figure 2: An example with 3 GoL timesteps and 3 model timesteps. The metric
gets a value of .5 because the 3rd predicted state matches the ground truth,
but the 2nd predicted state does not. The 1st and 4th states are ignored
because the network is trained on them. Figure 3: The metric gets a value of
.75 because both predicted states match the same ground truth state. So the
3rd predicted state gets a half score. Figure 4: An example with 2 GoL
timesteps and 3 model timesteps. The metric gets a value of 1.0. The 2nd model
state matches the intermediate GoL state. The 3rd model state matches the 3rd
GoL state, but this is ignored because the model was trained to predict the
3rd GoL state. The denominator in the metric is 1 because $min(2,3-2)=1$
Figure 5: An example with 2 GoL timesteps and 3 model timesteps. The metric
gets a value of 1.5. The 2nd model state matches the intermediate GoL state.
The 3rd model state also matches the intermediate GoL state, so the metric
gives .5 score for this state. Note that we ignore the last GoL state and not
the last model state, to this match is a valid one


# ChatUniTest: a ChatGPT-based automated unit test generation tool

Zhuokui Xie, Yinghao Chen, Chen Zhi1, Shuiguang Deng, Jianwei Yin {xiezhuokui,
yh_ch, zjuzhichen, dengsg<EMAIL_ADDRESS>1 Corresponding author Zhejiang
University, Hangzhou, China

###### Abstract

Unit testing is a crucial, yet often tedious and time-consuming task. To
relieve developers from this burden, automated unit test generation techniques
are developed. Existing automated unit test generation tools, such as program-
analysis-based tools like EvoSuite and Randoop, lack program comprehension,
resulting in unit tests with poor readability and limited assertions.
Language-model-based tools, such as AthenaTest and A3Test, have limitations in
the generation of correct unit tests. In this paper, we introduce ChatUniTest,
a ChatGPT-based automated unit test generation tool developed under the
Generation-Validation-Repair framework. ChatUniTest generates tests by parsing
the project, extracting essential information, and creating an adaptive focal
context that includes the focal method and its dependencies within the pre-
defined maximum prompt token limit. The context is incorporated into a prompt
and subsequently submitted to ChatGPT. Once ChatGPT’s response is received,
ChatUniTest proceeds to extract the raw test from the response. It then
validates the test and employs Rule-based Repair to fix syntactic and simple
compile errors, followed by ChatGPT-based Repair to address challenging
errors. Our rigorous evaluation demonstrates that ChatUniTest outperforms
EvoSuite in branch and line coverage, surpasses AthenaTest and A3Test in focal
method coverage, and effectively generates assertions while utilizing mock
objects and reflection to achieve test objectives.

##  I Introduction

Background. In today’s world, where software is becoming more extensive and
complex, even minor defects can result in significant financial losses and
reputational damage to large companies. As a result, software testing are
playing an even more critical role in the software development life cycle,
ensuring software quality prior to product delivery. The fundamental and
essential component of the software testing[1] is unit testing, which involves
testing individual units of code. Unit testing helps developers identify
defects and errors early in the development process, reducing the overall cost
of development. Additionally, unit testing improves the maintainability and
scalability of the software application, making it easier to modify and expand
in the future. However, manually writing unit tests requires developers to put
a lot of time and efforts, resulting it often ignored by developers.

To address this issue, automated unit test generations have been developed to
relieve developers from writing simple but non-trivial unit tests. These tools
are designed to produce unit tests that invoke Testing APIs in JUnit [2] and
Mockito [3] to cover as much code under test as possible, so that most defects
in the program are caught by unit tests. Automated unit test generation tools
can be categorized into two directions: traditional unit test generation tool
and deep learning unit test generation tool. EvoSuite [4] is a traditional
unit test generation tool that uses evolutionary algorithms to generate new
test cases and a fitness function to guide the search process, achieving high
code coverage criteria like branch and line coverage. However, the unit tests
generated by EvoSuite differ in essence from human-written tests, making them
hard to read and understand. AthenaTest [5], a deep learning based tool,
pretrains a BART model on English and code corpus and fine-tunes it on
Methods2Test[6]. It attempts to directly translate source code to unit tests.
A3Test [7] improves upon AthenaTest by incorporating assertion knowledge and
verifying naming consistency and test signatures, achieving better results in
terms of correctness and method coverage. However, these two state-of-the-art
deep learning based tools have limitations, as they cannot repair a simple
error by interacting with the deep learning model, leading to most generated
tests being incorrect. The emergence of ChatGPT has the potential to transform
the situation.

Motivation. ChatGPT demonstrated its capability in various domains, including
generating unit tests. Our initial analysis revealed that ChatGPT can
sometimes generate well-explained and understandable unit tests, but it
frequently fails to complete the generation process. Furthermore, less than
20% of successful responses resulted in a correct unit test. To understand
these issues, we examined the generation process and identified two critical
limitations. Firstly, the limited token (e.g., 4096 tokens for GPT-3.5), which
includes the prompt and completion, restricts ChatGPT from processing all
relevant information required to generate effective response. Suggesting an
adaptive generation of context, tailored to the test unit, could alleviate
this limitation. Secondly, without appropriate compilers and test executors,
ChatGPT is unable to validate the generated unit tests, leading to various
errors, including “cannot find symbol” during compilation and
“AssertionFailedError” during runtime. Some of these errors can be rectified
by carrying out few rounds of consecutive validation and minor modification.
Suggesting that a validation component as well as a repair component would
significantly improve ChatGPT’s effectiveness in generating unit tests.

Approach. Motivated by our initial analysis, we developed ChatUniTest 111We
have released the artifact at https://github.com/ZJU-ACES-ISE/ChatUniTest
under the Generation-Validation-Repair framework. To address the first
limitation, we define the test unit as a method and designed an adaptive focal
context generation mechanism, which generates a context containing as much as
necessary information for the focal method, while leaving enough space for
ChatGPT to produce complete responses. To tackle the second limitation, we
implemented a validation component and a repair component. The validation
component includes a parser, compiler, and test executor. The repair component
consists of rule-based repair and ChatGPT-based repair. Rule-based repair
addresses simple errors such as syntax errors and missing import statements.
If rule-based repair fails, ChatUniTest resorts to ChatGPT-based repair. It
constructs a prompt with an error message, incorrect unit test, and adaptive
focal context to obtain the corrected unit test from ChatGPT. If a test
remains unfixed after several validation and repair rounds, it is marked as
deprecated.

Evaluation Results. In the experiment, we aimed to validate ChatUniTest’s
effectiveness, which generated correct tests in 30% of attempts. It
outperformed EvoSuite in branch coverage for 7 out of 10 projects and line
coverage for 8 out of 10 projects. ChatUniTest also demonstrated superior
performance when compared to AthenaTest, achieving a result equivalent to
A3Test in terms of the correct test percentage. Moreover, ChatUniTest produced
an overwhelmingly positive outcome in terms of focal method coverage, reaching
79.67%, when compared to both AthenaTest (43.75%) and A3Test (46.80%).
ChatUniTest also excelled in comparison to AthenaTest and A3Test on 16 out of
18 methods in the NumberUtils class in the Defects4J Lang-1-f revision,
showcasing significant advantages in terms of method line coverage as well.

Contributions. In this paper, we introduce ChatUniTest, a ChatGPT-based
automated unit test generation tool. ChatUniTest offers the following
noteworthy contributions:

  * •

ChatUniTest is the first ChatGPT-based automated unit test generation tool,
outperforming EvoSuite in terms of branch coverage and line coverage, while
also surpassing AthenaTest and A3Test in terms of focal method coverage and
method line coverage.

  * •

ChatUniTest introduces an Adaptive Focal Context Generation algorithm, capable
of adaptively generating prompts with the required context, while staying
within the maximum prompt token limit, thereby avoiding truncated responses
from ChatGPT.

  * •

ChatUniTest can effectively repairs syntactic errors and simple compile errors
in unit tests utilizing its rule-based repair component. Furthermore, it is
capable of resolving approximately 50% of challenging errors by interacting
with ChatGPT within the ChatGPT-based repair component.

##  II Related Work

### Program-analysis-based Unit Test Generation

Various program analysis techniques are used to generate unit test
automatically, such as random testing, search-based testing, and symbolic
execution. For the tools fallen in this type, EvoSuite and Randoop are the
most-widely used tools among them. EvoSuite[4] generates initial individuals
randomly and applies an evolutionary algorithm to select, mutate, and combine
test cases based on their fitness values. This leads to the evolution of test
suites towards better performance. It is likely to achieve high branch and
line coverage, but lacks the ability to generate meaningful test cases, such
as variable names, test names and assertions, resulting in generated unit
tests with poor understandability and maintainability.

Randoop[8] randomly selects and invokes methods from the class under test,
along with its dependent classes. It builds sequences of method calls,
incorporating feedback to guide the test generation process. However, it can
only produce simple unit tests with limited code coverage, cannot handle
complex dependencies, and its generated tests are not easy to read, understand
or maintain, similar to EvoSuite.

### Language-model-based Unit Test Generation

Figure 1: Overview of ChatUniTest

The introduction of Transformer[9] has been a game-changer in the field of
natural language processing (NLP) and programming language processing
(PLP)[10, 11, 12, 13, 14, 15]. Extensive research on unit test generation has
been carried out on Transformer-based language models.

AthenaTest[5] generates human-written unit tests from source code by treating
unit test generation as a translation task. It generates a focal context for
each focal method and attempts to translate it into human-written unit tests
by fine-tuning its model on the Methods2Test[6] dataset. However, less than
20% of the generated unit tests are correct. To address this problem,
A3Test[7] improves pre-training by using a PLBART model and fine-tuning it
with Methods2Test. By verifying naming consistency and ensuring that test
signatures match assertion knowledge, A3Test-generated unit tests achieve
better coverage and correctness than AthenaTest. However, both models fail to
correct fundamental errors in the generated unit tests.

TESTPILOT[16] automatically generates unit tests for JavaScript programs using
Codex[17], without additional training or few-shot learning. It uses function
signatures, implementation, and usage examples extracted from documentation.
TESTPILOT adds the test and error message to the prompt to fix failing tests.
However, the usage of documentation makes it insuitable to undocumented or
less-documented programs.

Except for the above end-to-end solutions, recent studies[18, 19] also
investigate integrate LLMs with program-analysis-based test generation,
especially for test input generation.

##  III Approach

As mentioned previously, ChatGPT encounters two limitations when generating a
unit test: the token limit and the absence of validation and repair
components. This section will outline the design of ChatUniTest and explain
how it overcomes these limitations, thereby facilitating the automated
generation of high-quality unit tests with an elevated success rate.

Figure 1 provides an overview of ChatUniTest, which utilizes the Generation-
Validation-Repair framework. To enable parallelization, ChatUniTest first
requires some preprocessing, including analyzing projects, parsing Java file,
and extracting the necessary information for use in the generation process.
The focal method needs to pass through the following components to generate a
correct unit test:

  * •

Generation During the generation phase, ChatUniTest generates an adaptive
focal context based on the focal method and pre-defined maximum prompt token
limit. It then renders the context into a prompt and obtains a response from
ChatGPT.

  * •

Validation In the validation phase, ChatUniTest extracts a test from ChatGPT’s
response and checks it for any syntactic errors, compilation errors, or
runtime errors.

  * •

Repair During the repair phase, ChatUniTest attempts to fix simple errors in
the unit test using rule-based repair. If the error persists, ChatUniTest
resorts to ChatGPT-based repair to resolve it.

### Preprocessing

In order to streamline and parallelize the entire process, preprocessing the
projects under test is necessary. ChatUniTest begins by traversing the project
folder and locating all the java files, after which it parses each class file
into an Abstract Syntax Tree (AST). While navigating the AST, ChatUniTest
collects the information at two levels: class and method. Class level
information includes the package, imports, class signature, fields, and method
signatures. Method level information includes method body, field access,
getter/setter invocation, dependent class name, and method invocation (e.g.,
max(byte, byte, byte)), which is used to identify a unique method when there
are several methods with the same name.

### Generation

TABLE I: Abbreviation Table Abbr. | Explaination
---|---
FC | focal class
FM | focal method
DC | dependent class
IM | invoked method
body | source code
sig | signatures
ms | method signature
flds | fields
gs | getter and setter signatures
ns | namespace, including package and imports
ctor | constructor signature
ctx | context
dep | dependency
hasDep | has dependency
maxPrtLen | maximum prompt token limit
prtTemp | prompt template

1

2

3

4

5

6

7Function  _AdaptiveFocalContextGeneration_ :

Input : FC, FM, DC, IMsig, hasDep, maxPrtLen, prtTemp, prtTempdep

Output : prompt

8 context $\leftarrow$ FCsig $+$ FCctor $+$ FMbody

9 if  _useFlds_ then

10 context $\leftarrow$ context \+ FCflds \+ FCgs

11

12 tokens $\leftarrow$ TokenCount (prtTemp, context)

13 if  _tokens $>$ maxPrtLen_ then

14 Throw "Error: required context not satisfied!"

15 contextNew $\leftarrow$ context $+$ FCns

16 tokens $\leftarrow$ TokenCount (prtTemp, contextNew)

17 if  _tokens $<$ maxPrtLen_ then

18 context $\leftarrow$ contextNew

19

20 else

21 prompt $\leftarrow$ GeneratePrompt (prtTemp, context)

22 return _prompt_

23 if  _hasDep_ then

24 contextNew $\leftarrow$ context $+$ DCsig $+$ DCctor $+$ IMsig

25 tokens $\leftarrow$ TokenCount (prtTempdep, contextNew)

26 if  _tokens $<$ maxPrtLen_ then

27 context $\leftarrow$ contextNew

28

29 prompt $\leftarrow$ GeneratePrompt (prtTempdep, context)

30

31 else

32 contextNew $\leftarrow$ context $+$ IMsig

33 tokens $\leftarrow$ TokenCount (prtTemp, contextNew)

34 if  _tokens $<$ maxPrtLen_ then

35 context $\leftarrow$ contextNew

36 contextNew $\leftarrow$ context $+$ FCms

37 tokens $\leftarrow$ TokenCount (prtTemp, contextNew)

38 if  _tokens $<$ maxPrtLen_ then

39 context $\leftarrow$ contextNew

40

41

42 prompt $\leftarrow$ GeneratePrompt (prtTemp, context)

43

44 return _prompt_

45

Algorithm 1 Adaptive Focal Context Generation

The purpose of the generation component is to generate an adaptive focal
context and integrate it into a prompt while ensuring that the prompt length
remains under the limit of maximum prompt tokens, allowing ChatGPT sufficient
space to generate complete responses. We propose an adaptive focal context
generation algorithm, presented in Algorithm 1, with abbreviations provided in
Table I for clarity. ChatUniTest first verifies if the required context can be
rendered into a prompt that satisfies the maximum prompt token limit. If it
fails to do so, ChatUniTest terminates the process, as missing information in
the required context would result in invalid unit tests. ChatUniTest then
attempts to add more information to the current context in order of
importance, while verifying that each new addition complies with the maximum
prompt token limit. If the method has dependencies, the dependent class
signature, dependent class constructor, and invoked method signature are added
to the context to provide a better understanding of how to instantiate a
dependent class, invoke its method without compilation errors, and provide the
return type information. If the method has no dependencies, ChatUniTest
considers adding the method signature invoked by the focal method, as it
provides parameter type and return type information. If the token limit
allows, ChatUniTest then attempts to add all method signatures inside the
focal class to the context to provide a comprehensive understanding of the
method’s role and purpose.

Figure 2: Prompt for focal method without dependency

Once the adaptive focal context is generated, ChatUniTest renders it into a
prompt, using the prompt template presented in Figure 2 if the context has no
dependent class information, and the template in Figure 3 if it does. The
successfully generated prompt complies with the maximum prompt token limit and
is sent to ChatGPT through OpenAI’s API.

Figure 3: Prompt for focal method with dependency

### Validation

After receiving a response from ChatGPT, ChatUniTest extracts and validates
the test while selecting a different repair component if any errors are
detected during the validation process.

#### Extract

In our preliminary analysis, we discovered that ChatGPT occasionally fails to
satisfy all the requirements outlined in the prompt, even when explicitly
instructed with “no explanation needed”. Consequently, we developed two
generalized approaches for extracting tests from responses. Tests are
typically provided in two formats: delimited by triple backticks or as plain
code without a delimiter. For the former format, ChatUniTest employs regular
expressions to identify strings enclosed within triple backticks and filters
out code snippets lacking the “@Test” annotation. For the latter format,
ChatUniTest locates the line containing the keyword “public *Test” in the
response and attempts to determine the test boundary by verifying that the
first character in a line is valid in Java syntax. If ChatUniTest fails to
extract any test from ChatGPT’s response, the entire process is terminated
immediately.

#### Parse

Upon successful test extraction, ChatUniTest utilizes a Java parser to verify
the test’s syntax. If a syntactic error is detected at this stage, ChatUniTest
forwards the test to the rule-based repair component for correction.

#### Compile and run

In this step, ChatUniTest compiles and executes the test. If the test fails to
compile or encounters errors during execution, it proceeds to the repair
process. A test is considered passed only if it is free from syntax errors,
compiles successfully, runs without errors.

### Repair

Figure 4: Prompt for repairing error

Throughout this process, ChatUniTest strives to fix any errors detected during
validation. The repair component is divided into two parts: rule-based repair
and ChatGPT-based repair. The rule-based repair component focuses on
correcting simple and common errors without requiring any token usage, while
the ChatGPT-based repair component targets more complex and challenging errors
that require modifications to the code structure. After successfully repairing
an error through this component, the test is sent to the validation component
for further verification of its correctness.

#### Syntactic repair

When extracting tests from ChatGPT’s response, syntactic errors may occur due
to the 4096 tokens limit being exceeded, resulting in truncated tests. In such
cases, ChatUniTest implements a syntactic repair method. Initially, it
identifies the last semicolon or closing brace and adds the necessary closing
braces to ensure that the number of opening braces is balanced. ChatUniTest
then verifies the syntactic correctness of the test. If the first approach
fails, ChatUniTest attempts to remove the last test by locating the “@Test”
annotation, truncating the code before the annotation, and then adding closing
braces to complete the test structure. If the repair process fails or there
are no tests left after the removal, the entire process is terminated.

#### Imports repair

A substantial number of raw tests obtained from ChatGPT may encounter
compilation errors such as “cannot find symbol”. To address this issue,
ChatUniTest performs import repairs by ensuring that the dependencies in the
focal class are present in the test.

#### ChatGPT repair

If rule-based repair is unable to correct an error, ChatUniTest resorts to
ChatGPT-based repair. ChatUniTest extracts information about the error type
and message, combined with focal method context, and generates a prompt by
rendering the information into a prompt template, as shown in Figure 4.
However, some error messages can be lengthy and verbose, leading to prompts
that exceed the maximum prompt token limit. In such cases, ChatUniTest
attempts to truncate the error message to ensure that the prompt is within the
limit. If the prompt still exceeds the limit or if a test cannot be fixed
after six rounds, the entire process is terminated.

TABLE II: Projects dataset Project | Abbr. | Domain | Version | Trained | # Methods
---|---|---|---|---|---
Commons-Lang [20] | Lang | Utility | 3.12.0 | yes | 3018
Commons-Cli [21] | Cli | Cmd-line interface | 1.5.0 | yes | 250
Commons-Csv [22] | Csv | Data processing | 1.10.0 | yes | 209
Gson [23] | Gson | Serialization | 2.10.1 | yes | 660
Jfreechart [24] | Chart | Visualization | 1.5.4 | yes | 8340
Ecommerce-microservice-backend-app [25] | Ecommerce | Microservices | 695a6d4 | no | 316
Datafaker [26] | Datafaker | Data generation | 1.9.0 | no | 1766
Flink-kubernetes-operator [27] | Flink-k8s-opr | Cloud computing | 1.4.0 | no | 844
Binance-connector-java [28] | Binance-conn | API wrapper | 2.0.0 | no | 474
Event-ruler [29] | Event-ruler | Event Engine | 1.2.1 | no | 485
Total |  |  |  |  | 16362

##  IV Experiment Design

In the experiment, the ChatUniTest will be configured to generate unit tests
for each focal method within six attempts, while restricting the maximum
number of rounds to six. If the prompt token limit exceeded 2700, the attempt
will be aborted. Additionally, if an attempt fails to produce a passing test
within six rounds, it will be deprecated. The temperature of the
GPT-3.5-turobo model will be set to 0.5, enabling it to generate diverse unit
tests.

To evaluate the effectiveness of ChatUniTest, we formulate four research
questions:

### RQ1: How is the quality of the generated unit test cases?

To address this question, we will employ ChatUniTest to automatically generate
unit tests for ten Java projects listed in Table II. We will measure test
coverage using Cobertura [30] and Jacoco [31]. These projects not only
encompass a wide range of domains, such as utility, middleware, and
distributed computing platforms, but also cover various Java versions,
including Java 11 and Java 17. Notably, five selected projects were excluded
from ChatGPT’s training data because they were created after December 1, 2021,
to prevent possible data leakage.

In evaluating the quality of the generated test cases, we considered several
factors, such as:

  * •

Syntactic Correctness Syntactic correctness refers to whether the test code
adheres to Java syntax rules, which we will verify using a Java parser.

  * •

Compile Correctness Compile correctness will be deemed if the test does not
produce any errors during compilation using the Java Compiler.

  * •

Passing We will use JUnit as the test executor to verify if a test is passing,
meaning that it does not produce any runtime errors during execution.

  * •

Testing APIs invocations Well-written unit test cases should verify the
expected behavior of MUT by invoking Testing APIs. For instance, test cases
should use JUnit Assert APIs (e.g. assertEqual(), assertTrue()) to check the
return value of MUT, and Mockito Framework APIs (e.g., mock(), verify(),
when()) to simulate the behavior of an external dependency.

  * •

Correct We define a test as correct if it is syntactically accurate, compiles
and runs without errors, invokes the method under test, contains assertions,
and enables the target code to pass the test.

### RQ2: How does the performance of ChatUniTest compare against EvoSuite,
AthenaTest, and A3Test?

In this research question, we compare ChatUniTest against three alternative
approaches: EvoSuite (program-analysis-based test generation), AthenaTest
(language-model-based test generation) and A3Test (language-model-based test
generation). We evaluate the comparison between ChatUniTest and EvoSuite on
ten Java projects (see Table II) based on branch and line coverage.

For the comparison among ChatUniTest, AthenaTest, A3Test, we use the Defects4J
dataset. Specifically, we first compare their method line coverage on class
NumberUtils in Defects4J Lang-1-f revision. Then, we will compare their
correct test cases pertentage and focal method coverage across entire
Defects4J dataset.

  * •

Correct Test Percentage The correct test percentage represents the percentage
of correctly formulated tests in all attempts.

  * •

Focal Method Coverage The focal method coverage denotes the percentage of
focal methods that were covered during the generation of unit tests. A focal
method is considered covered if it was correctly tested within the given
attempt.

### RQ3: What are the contributions of different components of ChatUniTest?

In this research question, we divide ChatUniTest into three components and
assess their contributions to the overall test cases separately. We evaluate
their contributions based on the ratio of correct tests and method code
coverage.

### RQ4: How much does it cost to generate unit tests?

Our objective for this research question is to gain an insight into the cost
incurred by using ChatUniTest. The experiment will evaluate the token cost per
method across different projects. We will also explore the distribution of
prompt tokens, completion tokens during generation and repair process. Note
that at present, model gpt-3.5-turbo charges $0.002 for every 1000 tokens.
[32]

TABLE III: Performance of ChatUniTest on 10 Java projects: outcomes of Test Case Generation Attempts Projects | Methods | Attempts | Aborted | SyntaxError | CompileError | RuntimeError | Passed | Correct
---|---|---|---|---|---|---|---|---
Lang | 3018 | 18108 | 66 | 1112 (6.16%) | 3952 (21.90%) | 6755 (37.44%) | 6220 (34.48%) | 6118 (33.91%)
Cli | 250 | 1500 | 0 | 11 (0.73%) | 360 (24.00%) | 521 (34.73%) | 539 (35.93%) | 520 (34.67%)
Csv | 209 | 1254 | 6 | 11 (0.88%) | 363 (29.09%) | 346 (27.72%) | 393 (31.49%) | 373 (29.89%)
Gson | 660 | 3960 | 150 | 54 (1.42%) | 1487 (39.03%) | 1065 (27.95%) | 1080 (28.35%) | 1047 (27.48%)
Chart | 8340 | 50040 | 42 | 613 (1.23%) | 17843 (35.69%) | 14100 (28.20%) | 17517 (35.04%) | 17076 (34.15%)
Ecommerce | 316 | 1896 | 6 | 40 (2.12%) | 958 (50.69%) | 315 (16.67%) | 581 (30.74%) | 574 (30.37%)
Datafaker | 1766 | 10596 | 6 | 47 (0.44%) | 7754 (73.22%) | 1562 (14.75%) | 1099 (10.38%) | 1082 (10.22%)
Flink-k8s-opr | 844 | 5064 | 12 | 212 (4.20%) | 3278 (64.89%) | 629 (12.45%) | 938 (18.57%) | 871 (17.24%)
Binance-conn | 474 | 2844 | 6 | 19 (0.67%) | 1205 (42.46%) | 419 (14.76%) | 1155 (40.70%) | 1019 (35.91%)
Event-ruler | 485 | 2910 | 0 | 17 (0.58%) | 1514 (52.03%) | 689 (23.68%) | 686 (23.57%) | 662 (22.75%)
Total | 16362 | 98172 | 294 | 2136 (2.18%) | 38714 (39.55%) | 26401 (26.97%) | 30208 (30.86%) | 29342 (29.98%)

##  V Evaluation

In this section, we present the experimental results and provide answers to
the research questions posed in this study.

### RQ1: How is the quality of the generated Test Cases?

In Table III, we present the performance of ChatUniTest on 10 Java projects,
wherein it processed a total of 16,362 methods and made 98,172 unit test
generation attempts. Out of these attempts, 294 were aborted due to exceeding
the maximum prompt token limit.

Regarding syntax errors, 2.18% of the attempts encountered this issue. The
highest percentage of syntax errors was observed in the Lang project (6.16%).
Upon manual investigation of the results, we found that the most common reason
for these errors was the generation of overly lengthy tests, which were
subsequently truncated and cannot be repaired by syntactic repair component.

Compile errors were encountered in 39.55% of the unit test generation
attempts. The highest percentage of compile errors was observed in the
Datafaker project (73.22%). The most common types of errors were “cannot find
symbol” or “incompatible types”, primarily caused by ChatUniTest generating
non-existing methods or classes, or making mistakes in their usage.

In terms of runtime errors, the Lang project had the highest percentage
(37.44%). The most common types of errors were
“org.opentest4j.AssertionFailedError: expected: … but was: …” and “cannot
invoke … because … is null”. These errors were primarily due to the misuse of
assertions or reflection.

As for passed and correct test cases, 30.86% of the attempts were passed, with
29.98% of them being correct. This demonstrates the potential of ChatUniTest
in general, as it exhibits good performance across most of the projects
regardless of their size or Java version. Additionally, ChatUniTest has the
ability to utilize mock objects or reflection to achieve the desired test
objectives. This overall performance suggests that ChatUniTest can be a
valuable asset in generating test cases for a wide range of Java projects.

Figure 5: Distribution of assertion invocations in passed tests Figure 6:
Distribution of mock invocations in passed tests

The utilization of TestAPIs is crucial for ensuring the quality of unit tests.
To investigate TestAPI usage in unit tests generated by ChatUniTest, we
examined all successfully executed test files and recorded their TestAPI
invocations. Figure 5 presents the distribution of assertions, featuring the
top 15 non-zero invoked assertion types. ChatUniTest can successfully invoke
all 15 types of assertions, with an average of 3 assertions per unit test.
Additionally, Figure 6 exhibits the distribution of mock invocations,
highlighting the top 15 out of 34 types of invoked mock APIs. The most
commonly used APIs include mock(), when(), and thenReturn(), while ChatUniTest
is also capable of successfully invoking less frequently used APIs such as
doNothing() and verifyNoMoreInteractions(). These findings emphasize that the
unit tests generated by ChatUniTest are abundant in TestAPI invocations, with
ChatUniTest effectively utilizing both assertions and mocks to improve test
quality.

Answer to RQ 1 ChatUniTest achieved a 30.86% pass rate and 29.98% correct test
cases in 97,878 successful attempts. The passed tests are rich in both
quantity and variety of assertions and mocks, averaging three assertions per
unit test. This demonstrates ChatUniTest’s capability in generating high-
quality unit tests.

### RQ2: How does the performance of ChatUniTest compare against EvoSuite,
AthenaTest, and A3Test?

TABLE IV: Branch and line coverage comparison of EvoSuite and ChatUniTest on 10 Java projects Projects | Branch Coverage | Line Coverage
---|---|---
| EvoSuite | Our | EvoSuite | Our
Lang | 84.92% | 94.70% | 84.71% | 91.94%
Cli | 90.90% | 96.36% | 90.93% | 93.52%
Csv | 75.87% | 84.09% | 70.28% | 86.61%
Gson | 59.23% | 83.88% | 61.53% | 86.80%
Chart | 87.67% | 88.27% | 85.90% | 84.03%
Ecommerce | 100% | 100% | 89.58% | 96.35%
Datafaker | 91.12% | 89.76% | 58.55% | 86.07%
Flink-k8s-opr | 89.72% | 78.76% | 87.14% | 83.02%
Binnance-conn | 98.72% | 97.17% | 87.59% | 97.87%
Event-ruler | 87.78% | 93.14% | 84.02% | 87.40%
Average | 86.59% | 90.61% | 80.02% | 89.36%
TABLE V: Method line coverage comparison, compare to AthenaTest and A3Test on class NumberUtils in Defects4J Lang-1-f  Focal Method | AthenaTest | A3Test | ChatUniTest
---|---|---|---
toInt(String, int) | 23 (6.1%) | 24 (6.4%) | 75 (20.0%)
toLong(String, long) | 20 (5.3%) | 21 (5.6%) | 31 (8.3%)
toFloat(String, float) | 22 (5.9%) | 21 (5.6%) | 25 (6.7%)
toDouble(String, double) | 20 (5.3%) | 21 (5.6%) | 26 (6.9%)
toByte(String, byte) | 23 (6.1%) | 23 (6.1%) | 25 (6.7%)
toShort(String, short) | 22 (5.9%) | 23 (6.1%) | 25 (6.7%)
createFloat(String) | 20 (5.3%) | 21 (5.6%) | 23 (6.1%)
createDouble(String) | 21 (5.6%) | 21 (5.6%) | 23 (6.1%)
createInteger(String) | 21 (5.6%) | - | 23 (6.1%)
createLong(String) | 21 (5.6%) | 23 (6.1%) | 21 (5.6%)
createBigInteger(String) | 20 (5.3%) | 28 (7.5%) | 39 (10.4%)
createBigDecimal(String) | 22 (5.9%) | 22 (5.9%) | 34 (9.1%)
min(long[]) | 22 (5.9%) | 22 (5.9%) | 37 (9.9%)
min(int, int, int) | 22 (5.9%) | 25 (6.7%) | 23 (6.1%)
max(float[]) | 22 (5.8%) | 23 (6.1%) | 33 (8.8%)
max(byte, byte, byte) | 22 (5.9%) | 22 (5.9%) | 25 (6.7%)
isDigits(String) | 23 (6.1%) | 23 (6.1%) | 26 (6.9%)
isNumber(String) | 51 (13.6%) | 33 (8.8%) | 76 (20.3%)
TABLE VI: The percentage for the correct test cases and the focal method coverage of A3Test, AthenaTest, and ChatUniTest on Defects4J dataset. Projects | Correct Test Cases (%) | Focal Method Coverage (%)
---|---|---
| A3Test | AthenaTest | ChatUniTest | A3Test | AthenaTest | ChatUniTest
Gson | 14.09% | 2.89% |  23.30% | 40.90% | 9.54% | 55.91%
Cli | 25.19% | 11.07% |  38.77% | 37.20% | 29.46% | 70.34%
Csv | 25.73% | 8.98% |  44.26% | 37.80% | 34.31% | 76.94%
Chart | 31.30% | 11.70% |  39.02% | 34.40% | 32.00% | 79.56%
Lang |  49.50% | 23.35% | 39.85% | 58.30% | 56.97% | 84.05%
Total | 40.05% | 16.21% |  40.14% | 46.80% | 43.75% | 79.67%

In Table IV, we compare the branch and line coverage of ChatUniTest and
EvoSuite on 10 different Java projects. From the table, we can observe that
ChatUniTest generally outperforms EvoSuite in terms of branch coverage for the
majority of the projects. For example, in the Cli project, ChatUniTest
achieves higher both branch and line coverage (96.36% and 93.52%,
respectively) compared to EvoSuite (90.90% and 90.93%, respectively). Similar
trends can be seen in other projects, such as Csv, Gson, and Event.

However, there are cases where EvoSuite surpasses ChatUniTest in terms of
branch or line coverage, albeit with minor advantages. For instance, in the
Datafaker project, EvoSuite achieves a higher branch coverage of 91.12%
compared to ChatUniTest’s 89.76%. Additionally, in the Chart project, EvoSuite
has a higher line coverage of 85.90% compared to ChatUniTest’s 84.03%.

Despite these exceptions, ChatUniTest outperforms EvoSuite in terms of line
coverage across most projects. On average, ChatUniTest demonstrates an
advantage in both branch and line coverage, with average branch coverage of
90.61% compared to EvoSuite’s 86.59%, and average line coverage of 89.36%
compared to EvoSuite’s 80.02%. This consistency in ChatUniTest’s results
highlights its reliability and effectiveness as a unit test generation tool.

In Table V, we present an evaluation of the method line coverage performance
of AthenaTest, A3Test, and ChatUniTest on the NumberUtils class in the
Defects4J Lang-1-f revision. The data for AthenaTest and A3Test in the table
is retrived from the results presented in A3Test[7], while the data for
ChatUniTest is obtained from our experiments. From the table, it is evident
that ChatUniTest generally outperforms both AthenaTest and A3Test in terms of
method line coverage on 15 out of 17 methods. Notably, in the toInt(String,
int) method, ChatUniTest achieves an impressive coverage of 20.0% (75 lines),
while AthenaTest and A3Test reach coverage levels of 6.1% (23 lines) and 6.4%
(24 lines), respectively. However, there are two methods, createLong(String)
and min(int, int, int), where A3Test outperforms ChatUniTest.

In Table VI, we present an evaluation of the performance of the three tools on
the Defects4J dataset, focusing on the percentage of correct test cases and
the focal method coverage. We compare the three tools on the Defects4J
dataset. The data for AthenaTest and A3Test in the table is adapted from
A3Test[7] as well. From the table, we can observe that ChatUniTest
consistently outperforms both A3Test and AthenaTest in terms of the percentage
of correct test cases for all projects except Lang. With respect to focal
method coverage, ChatUniTest outperforms both AthenaTest and A3Test across all
five projects.

The utilization of TestAPIs is crucial for ensuring the quality of unit tests.
To investigate TestAPI usage in unit tests generated by ChatUniTest, we
examined all successfully executed test files and recorded their TestAPI
invocations. Figure 5 presents the distribution of assertions, featuring the
top 15 non-zero invoked assertion types. ChatUniTest can successfully invoke
all 15 types of assertions, with an average of 3 assertions per unit test.
Additionally, Figure 6 exhibits the distribution of mock invocations,
highlighting the top 15 out of 34 types of invoked mock APIs. The most
commonly used APIs include mock(), when(), and thenReturn(), while ChatUniTest
is also capable of successfully invoking less frequently used APIs such as
doNothing() and verifyNoMoreInteractions(). These findings emphasize that the
unit tests generated by ChatUniTest are abundant in TestAPI invocations, with
ChatUniTest effectively utilizing both assertions and mocks to improve test
quality.

Answer to RQ 2 When comparing the performance of ChatUniTest to EvoSuite,
ChatUniTest consistently outperforms EvoSuite in both branch and line coverage
across 10 Java projects, achieving an average branch coverage of 90.61% and
line coverage of 89.36%. In comparison to AthenaTest and A3Test, ChatUniTest
excels in method line coverage for 15 of 17 methods in the NumberUtils class
(Lang-1-f revision, Defects4J dataset), and outperforms both tools in correct
test cases and focal method coverage. These results highlight ChatUniTest’s
robustness and superiority compared to state-of-the-art tools.

### RQ3: What are the contributions of different components of ChatUniTest?

Figure 7: Overall test distributions of different components

To investigate the individual contributions of the generation, rule-based
repair, and ChatGPT-based repair components, we conducted an analysis of the
test distribution after processing by each component. The results are
presented in Figure 7. The figure indicates that the generation component
produced fewer correct tests and several syntactically incorrect tests. After
processing by the rule-based repair component, syntactic errors were
drastically reduced by 89.19%, compile errors were significantly decreased by
10.50%, and the number of correct tests increased by 15.54% from 13,391 to
15,472, primarily due to import repair. The ChatGPT-based repair component,
represented by the last set of bars, demonstrated its effectiveness as the
most significant contributor, leading to a substantial reduction in compile
error tests and a remarkable increase in the number of passed tests, by
90.63%, from 15,846 to 30,208. It is important to note that if the syntactic
error cannot be fixed by the repair component, the attempt is terminated, thus
will not be processed by ChatGPT-based component.

Figure 8: Passed test distributions for each round

To further investigate the effectiveness of ChatGPT-based repair, we presented
the number of successfully passed tests for each round in Figure 8. Each round
indicates the number of times ChatGPT was queried, meaning that a test had a
chance to go through the Generation component once and ChatGPT-based repair
five times. In Figure 8, we observe that nearly half of the total tests were
fixed by ChatGPT-based repair. Additionally, we find that the number of
repaired tests in rounds 2 through 6 was approximately half the quantity
compared to the previous round. This suggests that the ChatGPT-based repair
component plays a crucial role in enhancing test case generation,
significantly improving the overall quality and effectiveness of ChatUniTest.

Answer to RQ 3 In the study of ChatUniTest’s components, the generation
component produced fewer correct tests and several syntactically incorrect
ones. Rule-based repair drastically reduced syntactic errors by 89.19% and
increased correct tests by 15.54%. However, ChatGPT-based repair emerged as
the most significant contributor, increasing passed tests by 90.63% and
substantially reducing compile errors. The analysis showed that ChatGPT-based
repair played a crucial role in enhancing test case generation, significantly
improving ChatUniTest’s overall quality and effectiveness.

### RQ4: How much does it cost to generate unit tests?

In this research question, we report the cost from several perspectives.

Figure 9: Comparison of cost per method in total, repair, and generation
across multiple projects

Figure 9 presents the total cost per method across various projects. We can
observe that the total cost for generating unit tests varies across the
projects, with the lowest being $0.084 in the Ecommerce project and the
highest being $0.128 in the Flink project. This indicates that the complexity
for test generation differ from across projects. Repair process accounts for
the largest portion of the overall cost, at approximately 83%. Among these
projects, Flink has the highest repair cost per method, approximately $0.109,
while Ecommerce has the lowest about $0.073. In general, generation costs are
significantly lower than repair costs for all projects.

Figure 10: Token cost relationship between prompt and completion in Generation
component

Figure 10 demonstrates the correlation between prompt and completion token
costs in the Generation component. A red regression line represents the mean
trend. The green line signifies the sum of prompt and completion token costs
equating to 4096 tokens, indicating that any point on this line has exceeded
the maximum prompt token limit, resulting in a truncated response.

The vertical dotted line marks the maximum prompt token threshold set at 2700
tokens, with 99.95% of attempts remaining below this limit. The majority of
the data points are situated within the range of 0 to 1000 tokens. The
regression line’s slope is approximately 0.077, and the correlation
coefficient between prompt tokens and completion tokens is 0.09. This low
correlation suggests that the completion token count remains relatively
stable, irrespective of the prompt token count.

Figure 11: Token cost relationship between prompt and completion in Repair
component

Figure 11 illustrates the correlation between prompt and completion token
costs within the ChatGPT Repair component. The regression line’s slope is
0.23, and the correlation coefficient is 0.44, signifying a moderate positive
correlation between prompt tokens and completion tokens during the repair
process. This relationship emerges because ChatGPT is tasked with returning a
complete, corrected unit test after receiving the incorrect version.

To address this, a more efficient approach could involve requesting ChatGPT to
provide instructions for modifying the original test without responding with
the whole test. This could streamline the repair process, reduce token count,
and improve overall efficiency, potentially optimizing the ChatGPT-based
repair component and enhancing the test generation pipeline.

Answer to RQ 4 The cost of generating unit tests varies across projects, with
repair costs accounting for approximately 83% of the total cost. The
correlation between prompt and completion token costs in the Generation
component is low (0.09), suggesting completion token count remains relatively
stable. In the Repair component, a moderate positive correlation (0.44) exists
between prompt and completion tokens. A more efficient approach could involve
requesting ChatGPT to provide instructions for modifying tests rather than
returning the entire corrected test.

##  VI Conclusion and Future Work

In this study, we have introduced ChatUniTest, a ChatGPT-based automated unit
test generation tool developed within the Generation-Validation-Repair
framework. ChatUniTest employs an adaptive focal context generation algorithm
to produce informative prompts while adhering to the maximum prompt token
limit, thus enabling ChatGPT to return untruncated responses. During the
validation process, ChatUniTest utilizes a parser to ensure the unit test’s
syntactic correctness, a compiler to confirm its compile correctness, and a
test executor to verify its runtime correctness. If an error occurs during
validation, ChatUniTest first attempts to repair it using the Rule-based
repair component. If this fails, ChatUniTest resorts to interact with ChatGPT
for error correction. Our evaluation demonstrates that ChatUniTest outperforms
EvoSuite in terms of branch and line coverage and surpasses AthenaTest and
A3Test in focal method coverage.

## References

  * [1] M. Cohn, _Succeeding with agile: software development using Scrum_. Pearson Education, 2010.
  * [2] “Junit,” https://junit.org/.
  * [3] “Mockito - java mocking framework,” https://site.mockito.org/.
  * [4] G. Fraser and A. Arcuri, “Evosuite: automatic test suite generation for object-oriented software,” in _Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering_ , 2011, pp. 416–419.
  * [5] M. Tufano, D. Drain, A. Svyatkovskiy, S. K. Deng, and N. Sundaresan, “Unit test case generation with transformers and focal context,” _arXiv preprint arXiv:2009.05617_ , 2020.
  * [6] M. Tufano, S. K. Deng, N. Sundaresan, and A. Svyatkovskiy, “Methods2test: A dataset of focal methods mapped to test cases,” in _Proceedings of the 19th International Conference on Mining Software Repositories_ , 2022, pp. 299–303.
  * [7] S. Alagarsamy, C. Tantithamthavorn, and A. Aleti, “A3test: Assertion-augmented automated test case generation,” _arXiv preprint arXiv:2302.10352_ , 2023\.
  * [8] C. Pacheco and M. D. Ernst, “Randoop: feedback-directed random testing for java,” in _Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion_ , 2007, pp. 815–816.
  * [9] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [10] M. Allamanis, M. Brockschmidt, and M. Khademi, “Learning to represent programs with graphs,” _arXiv preprint arXiv:1711.00740_ , 2017.
  * [11] D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu _et al._ , “Graphcodebert: Pre-training code representations with data flow,” _arXiv preprint arXiv:2009.08366_ , 2020\.
  * [12] D. Peng, S. Zheng, Y. Li, G. Ke, D. He, and T.-Y. Liu, “How could neural networks understand programs?” in _International Conference on Machine Learning_. PMLR, 2021, pp. 8476–8486.
  * [13] M. Nassif, A. Hernandez, A. Sridharan, and M. P. Robillard, “Generating unit tests for documentation,” _IEEE Transactions on Software Engineering_ , vol. 48, no. 9, pp. 3268–3279, 2021.
  * [14] N. Cui, Y. Jiang, X. Gu, and B. Shen, “Zero-shot program representation learning,” in _Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension_ , 2022, pp. 60–70.
  * [15] T. Long, Y. Xie, X. Chen, W. Zhang, Q. Cao, and Y. Yu, “Multi-view graph representation for programming language processing: An investigation into algorithm detection,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, no. 5, 2022, pp. 5792–5799.
  * [16] M. Schäfer, S. Nadi, A. Eghbali, and F. Tip, “Adaptive test generation using a large language model,” _arXiv preprint arXiv:2302.06527_ , 2023.
  * [17] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman _et al._ , “Evaluating large language models trained on code,” _arXiv preprint arXiv:2107.03374_ , 2021.
  * [18] C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen, “Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models,” in _Proceedings of the 45th IEEE/ACM International Conference on Software Engineering_ , 2023.
  * [19] Z. Liu, C. Chen, J. Wang, X. Che, Y. Huang, J. Hu, and Q. Wang, “Fill in the blank: Context-aware automated text input generation for mobile gui testing,” _Proceedings of the 45th IEEE/ACM International Conference on Software Engineering_ , 2023.
  * [20] “Apache Commons Lang,” https://github.com/apache/commons-lang.
  * [21] “Apache Commons CLI,” https://github.com/apache/commons-cli.
  * [22] “Apache Commons CSV,” https://github.com/apache/commons-csv.
  * [23] Google, “Gson,” https://github.com/google/gson.
  * [24] “JFreeChart,” https://github.com/jfree/jfreechart.
  * [25] S. Horri, “E-commerce Microservice Backend App,” https://github.com/SelimHorri/ecommerce-microservice-backend-app.
  * [26] DataFaker, “DataFaker,” https://github.com/datafaker-net/datafaker.
  * [27] A. S. Foundation, “Flink Kubernetes Operator,” https://github.com/apache/flink-kubernetes-operator.
  * [28] binance-connector java, https://github.com/binance/binance-connector-java.
  * [29] A. W. Services, “Event Ruler,” https://github.com/aws/event-ruler.
  * [30] “Cobertura,” http://cobertura.github.io/cobertura/.
  * [31] “Jacoco,” https://www.jacoco.org/jacoco/.
  * [32] OpenAI, “Pricing,” 2023. [Online]. Available: https://openai.com/pricing/