# LLM is Like a Box of Chocolates: the Non-determinism of ChatGPT in Code
Generation

Shuyin Ouyang  University of BristolBristolUnited Kingdom ,  Jie M. Zhang
King’s College LondonLondonUnited Kingdom ,  Mark Harman  University College
LondonLondonUnited Kingdom and  Meng Wang  University of BristolBristolUnited
Kingdom

###### Abstract.

There has been a recent explosion of research on Large Language Models (LLMs)
for software engineering tasks, in particular code generation. However,
results from LLMs can be highly unstable; nondeterministically returning very
different codes for the same prompt. Non-determinism is a potential menace to
scientific conclusion validity. When non-determinism is high, scientific
conclusions simply cannot be relied upon unless researchers change their
behaviour to control for it in their empirical analyses. This paper conducts
an empirical study to demonstrate that non-determinism is, indeed, high,
thereby underlining the need for this behavioural change. We choose to study
ChatGPT because it is already highly prevalent in the code generation research
literature. We report results from a study of 829 code generation problems
from three code generation benchmarks (i.e., CodeContests, APPS, and
HumanEval). Our results reveal high degrees of non-determinism: the ratio of
coding tasks with zero equal test output across different requests is 72.73%,
60.40%, and 65.85% for CodeContests, APPS, and HumanEval, respectively. In
addition, we find that setting the temperature to 0 does not guarantee
determinism in code generation, although it indeed brings less non-determinism
than the default configuration (temperature=1). These results confirm that
there is, currently, a significant threat to scientific conclusion validity.
In order to put LLM-based research on firmer scientific foundations,
researchers need to take into account non-determinism in drawing their
conclusions.

††conference:  ; ; ††copyright: none††price: 15.00††isbn:
978-1-4503-XXXX-X/18/06

##  1\. Introduction

LLMs are nondeterministic by nature (Lee et al., 2022). This is because LLMs
predict the probability of a word or token given the context, represented by a
sample of words. The randomness in large language models typically comes from
the sampling methods used during text generation, such as top-k sampling or
nucleus sampling (Mitchell et al., 2023; Krishna et al., 2022). As a result,
identical instructions or prompts can yield completely different responses in
different requests.

This non-determinism (i.e., the inconsistency in the code candidates generated
in different requests with identical prompts)111There are other terms in the
literature that also refer to non-determinism, such as inconsistency,
variance, randomness, and instability. is an essential consideration when
using LLM in practice (Soares et al., 2022). Unreliable and inconsistent code
snippets can have significant negative effects on the process of software
development, particularly in safety-critical applications where consistency
and reliability are paramount (Kiviriga, 2023; Chatterjee et al., 2022). It
may also undermine developers’ trust in LLMs when completely different
suggestions are given at different times (Vaithilingam et al., 2022).

Moreover, non-determinism affects the reliability and reproducibility of
empirical software engineering (Poesia et al., 2022). Indeed, compared to
other tasks of ChatGPT, such as question answering and text summary, the non-
determinism threat in code-related tasks is much more serious, because the
inconsistency (especially semantic inconsistency) often suggests errors in the
generated code (Inala et al., 2022). It is therefore of vital importance to
understand how serious the non-determinism is for LLM-based software
engineering tasks.

This paper presents the first systematic empirical study on the threat of non-
determinism of ChatGPT in code generation tasks. We choose the code generation
task because code generation with Large Language Models (LLMs), such as
ChatGPT, has recently attracted significant attention due to its impressive
and cutting-edge performance (Li et al., 2022; Bubeck et al., 2023). Many
publications have emerged from both the software engineering community and the
machine learning community on evaluating the capability of ChatGPT in code
generation (Bubeck et al., 2023; Liu et al., 2023d; Feng et al., 2023;
Yetiştiren et al., 2023; Bang et al., 2023).

We conduct a series of experiments using the ChatGPT models on three widely-
studied code generation benchmarks (i.e. CodeContests, APPS, and HumanEval)
with 829 coding problems. For each code generation task, we let ChatGPT make
five predictions. We then compare the five code candidates from three aspects,
namely semantic similarity, syntactic similarity, and structural similarity.
We also explore the influence of temperature (i.e., a parameter that controls
the randomness of the text generated by ChatGPT) on non-determinism, as well
as the correlation between non-determinism and coding task features such as
the length of coding instruction and the difficulty of the task. Finally, we
compare the non-determinism between GPT-3.5 and GPT-4.

Our results reveal that the threat of non-determinism in ChatGPT for code
generation is serious. In particular, 1) the ratio of problems with not a
single equal test output among the top-five code candidates is above 60% for
all the benchmarks we study; 2) the maximum difference of the test pass rate
reaches 1.00 for all three datasets, and accounts for 48.17% of the problems
in HumanEval, the most widely used code generation benchmark; In addition,
contrary to the widely held belief (and practice followed to minimize
nondeterminism) (Deng et al., 2023; Bhavya et al., 2022; Liang et al., 2023),
setting the temperature to zero does not guarantee determinism in code
generation. Also interestingly, our correlation analysis suggests that the
length of coding instructions has a negative correlation with syntactic, and
structural similarity, and test pass rate, meaning that longer description
length tends to yield code candidates with less syntactic and structural
similarity and more buggy code.

To understand how the literature handles the non-determinism threat, we
collect 76 LLM-based code generation papers that appeared in the past 2 years.
Our manual analysis results highlight that only 22.4% of these papers consider
the non-determinism threat in their experiments. These results highlight that
there is currently a significant threat to scientific conclusion validity. We
call for researchers to take into account the non-determinism threat in
drawing their conclusions.

To summarize, this paper makes the following contributions:

  * •

We present the first study of the non-determinism threat in code generation
tasks on ChatGPT, with three widely-studied datasets (CodeContest, APPS,
HumanEval) and three types of similarity measurements. Our results reveal that
the non-determinism threat is serious and deserves attention from both
academia and industry.

  * •

We study the influence of temperature on the non-determinism of ChatGPT and
find that setting temperature to zero does not guarantee determinism in code
generation, which is contrary to many people’s beliefs.

  * •

We study the correlation between coding task features and the degree of non-
determinism. The results reveal that the length of coding instruction has a
negative correlation with syntactic and structural similarity, as well as the
average correctness of the generated code.

  * •

We collect and analyze the LLM-based code generation papers that appeared in
the past 2 years and find that only 21.1% of these papers consider the non-
determinism threat in their experiments. We call for researchers to take into
account non-determinism when conducting experiments and drawing conclusions,
considering the nature of the non-determinism of LLMs.

We release our data, code, and results at our homepage (hom, [n. d.]). The
rest of the paper is organized as follows. Section 2 introduces the main
procedure of our study. Section 3 describes the design of the experiments,
including research questions, benchmarks, selected models, and measurement
tools. Section 4 presents the results and discusses some interesting findings
based on the experimental results we obtained. Section 5 discusses the threats
to validity in two aspects, namely, dataset construction and test assessment
metrics and tools. Section 6 introduces the related work of our study. Section
7 discusses the implications for software developers and researchers. Section
8 concludes.

##  2\. Method

Fig 1 shows an overview of our experimental procedure. For each code
generation task, our study first produces a prompt with a coding instruction,
then feeds this prompt to ChatGPT API222https://platform.openai.com/docs/api-
reference/chat to generate code (zero-shot). We call the API five times to let
ChatGPT make five predictions with the same prompt. We then extract code from
each of the five responses, to get five code candidates. Our non-determinism
analysis compares the five code candidates in terms of their semantic
similarity, syntactic similarity, and structural similarity.

Figure 1. Overview of the experimental method.

Prompt Preparation: The first step in our study is prompt preparation.
Following the common practice in LLM-based code generation assessment(Austin
et al., 2021), we ask ChatGPT to generate Python code for each code generation
task with zero-shot prompting. To guarantee that ChatGPT produces code rather
than pure natural languages in its response, we augment the original code
generation description with a request for Python code. One challenge in
extracting the code from the API response is that there is no clear signal to
distinguish code with plain text in the response, which is different from
ChatGPT’s web chat window (i.e. in the chat window, codes are returned with
Markdown code blocks). To address this problem, we specify the format of the
generated code into ‘Markdown’. Thus, for each code generation task, our
prompt becomes ‘Generate Python3 code (Markdown):’ plus the original task
description.

Code Extraction and Script Writing: After receiving the response from ChatGPT,
we apply code extraction to retrieve the code from the generated text. We
compile the code directly without making any modifications. Our experiments
are mainly run on Google Deep Learning VM instances, with the Linux
environment pre-installed from open
images333https://cloud.google.com/compute/docs/images. All of the necessary
libraries are pre-installed. In this way, it can ensure to the greatest extent
that the generated code will not cause import errors caused by the library not
being installed during running.

Test Case Execution: To evaluate the semantics of ChatGPT’s generated code, we
use the test suite that is suited to each benchmark. We not only record
whether each test passes or not but also record every specific test output,
which enables us to compare the similarity of test outputs even if they both
fail. For CodeContests and HumanEval datasets, every problem has a certain
timeout value of 3 seconds. The APPS dataset does not provide a default
timeout value, and we set the value to be 3 seconds as well. We use single-
threaded scripts to run the tests to ensure that the test cases are executed
sequentially to avoid race conditions that may arise from concurrent
executions.

Similarity Checking: To measure the similarity between code candidates, we
introduce similarity measurement tools that evaluated the semantic, syntactic,
and structural similarity between the generated code solutions. The semantic
similarity is measured by comparing test execution outputs. The syntactic
similarity is measured by comparing the text similarity between codes. The
structural similarity is evaluated by comparing the code candidates’ abstract
syntax trees (ASTs). More details about our similarity measurement methods are
mentioned in Section 3.4.

##  3\. Experimental Design

###  3.1. Research Questions

This study answers the following questions:

_RQ1: To what extent is ChatGPT susceptible to non-determinism in code
generation?_ This RQ investigates the non-determinism of ChatGPT in terms of
the semantic, syntactic, and structural similarity among the code candidates
generated with identical instructions. There are three sub-RQs:

  * •

_Sub-RQ1.1: To what extent is ChatGPT susceptible to non-determinism in terms
of semantic similarity?_

  * •

_Sub-RQ1.2: To what extent is ChatGPT susceptible to non-determinism in terms
of syntactic similarity?_

  * •

_Sub-RQ1.3: To what extent is ChatGPT susceptible to non-determinism in terms
of structural similarity?_

_RQ2: How does temperature affect the degree of non-determinism?_ Temperature
is a hyperparameter of LLMs for controlling the randomness of the predictions.
This RQ checks and compares the non-determinism of ChatGPT in code generation
with different choices of temperature.
_RQ3: How does the non-determinism compare to the similarity of the top code
candidates generated within the same prediction?_ ChatGPT can be configured to
generate multiple candidates for one prediction, which are ranked by their
predictive probability. This RQ compares the similarity of the code candidates
obtained in different predictions with those obtained within the same
prediction.
_RQ4: Is there any correlation between different code generation tasks and the
non-determinism degree?_ To understand what affects non-determinism, this RQ
studies the correlation between code generation features (e.g., the length of
code generation instructions and the code generation difficulty) and the
similarity metric uesed in our study.
_RQ5: How is GPT-4’s non-determinism compared with GPT-3.5?_ This RQ compares
GPT-3.5 and GPT-4 in their degree of non-determinism in generating code.

###  3.2. Code Generation Benchmarks

Our experiments use the three most widely studied code generation benchmarks:
CodeContest (Li et al., 2022), APPS (Hendrycks et al., 2021), and HumanEval
(Chen et al., 2021). Table 1 shows the details of each benchmark. We also
introduce more details below:

Table 1. Code generation benchmarks

Name | Mean Length | No. of | Mean No. of | Mean No. of
---|---|---|---|---
| of description | Problems | Test Cases | Correct Codes
CodeContests | 1989.19 | 165 | 203.84 | 49.99
APPS | 1663.94 | 500 | 80.43 | 20.92
HumanEval | 450.60 | 164 | 9.24 | 1.00

CodeContests: CodeContests is used when training AlphaCode, which comprises
programming problems from various sources such as
Aizu444https://judge.u-aizu.ac.jp, AtCoder555https://atcoder.jp,
CodeChef666https://www.codechef.com,
CodeforcesCodeChef777https://codeforces.com, and
HackerEarthCodeChef888https://www.hackerearth.com. In our experiment,
following the assessment practice of AlphaCode, we use the test set of
CodeContests to benchmark the code generation tasks of ChatGPT.

APPS: APPS includes 10,000 problems (both the training set and testing set),
with different difficulty levels (i.e. introductory level, interview level,
and competition level). This dataset is exclusively designed for Python
program synthesis evaluation. The original test set contains 5,000 code-
generation problems, and we randomly sample 500 problems.

HumanEval: The HumanEval dataset is an evaluation set first proposed in (Chen
et al., 2021), which contains 164 hand-written code problems. Each problem
includes a function signature, docstring, body, and several unit tests, with
an average of 9.24 test cases per problem. We use the whole dataset to
benchmark our experiments.

As mentioned in Section 2, we especially focus on the code generated with
Python3 language, since it is one of the most widely studied programming
languages in code generation (Wei et al., 2019; Sun et al., 2019; Li et al.,
2022; Chen et al., 2021; Feng et al., 2020; Austin et al., 2021; Svyatkovskiy
et al., 2020).

###  3.3. Configuration of ChatGPT

ChatGPT has gained widespread popularity and recognition in multiple tasks
including question-answering, language translation, sentiment analysis, and
text summarising, among which code generation is one of the most impressive
tasks (Li et al., 2022; Bubeck et al., 2023). There are several reasons why we
have chosen ChatGPT as our research target among all large language models.
Firstly, ChatGPT has the ability to generate highly coherent and contextually
appropriate responses to a wide variety of textual prompts (Hassani and Silva,
2023). This makes it an ideal tool for conducting research in areas of code
generation by designing specific prompts. Secondly, the GPT-3.5 series is a
particularly attractive option due to its impressive performance and large-
scale training data, which allows for more accurate and nuanced language
processing capabilities (Liu et al., 2023a). Thirdly, the model API
‘gpt-3.5-turbo’ and ‘gpt-4’ released with ChatGPT have not been extensively
studied in academia, and their capabilities in terms of code generation are
thus still unknown. Therefore, we choose them as our experiment target models.
Written in ChatGPT’s official website999https://platform.openai.com/docs/api-
reference/chat/create, using ChatGPT’s model API requires various parameters.
We use the default values for most of the parameters in addition to the
following ones:

  * •

model: ID of the model to use, such as ‘gpt-3.5-turbo’ and ‘text-davinci-003’.
This parameter is strictly required, and in our case, we set this parameter to
‘gpt-3.5-turbo’ or ‘gpt-4’.

  * •

message: A list of messages describing the conversation so far, where two key
values ‘role’ and ‘content’ should be filled. This parameter is also strictly
required. In our experiments, the message’s ‘role’ is ‘user’ and the ‘content’
contains the prompt we used for requesting.

  * •

temperature: What sampling temperature to use, between 0 and 2 (Default value
is 1). Higher values will make the output more random, while lower values will
make it more focused and deterministic. In our study, we study the influence
of temperature in RQ3 with three temperature values: 0, 1, and 2. For the
remaining RQs, we use the default temperature value (i.e., temperature=1).

  * •

top_p: An alternative to sampling with temperature, called nucleus sampling,
where the model considers the results of the tokens with top_p probability
mass. In our experiment, we don’t take it into consideration and set this
value to remain at its default setting (i.e., top_p=1).

  * •

n: How many chat completion choices to generate for each input message.
(Default value is 1) This parameter is specifically used for RQ3 which
compares the difference in different requesting ways. We choose n=5, since
five is a commonly used figure in variance study papers (Nagarajan et al.,
2018). We generate five responses separately by requesting the ChatGPT model
five times but only take the first response from each time request.

###  3.4. Non-determinism Measurement

In order to answer our research questions, we introduce the following tools
for measuring the degree of non-determinism.

####  3.4.1. Semantic similarity

We measure the semantic similarity of different code candidates by checking
their similarity in test execution results, including _test pass rate_ and
_output equivalence rate_. The test pass rate calculates the ratio of the
passed test case number against the total test case number for code
candidates. It is one of the most widely used measurement metrics for
assessing code generation capabilities101010Although the benchmarks are very
widely studied, their test suites can be inadequate (Liu et al., 2023d, b;
Roziere et al., 2021). This paper is less affected by the inadequate test
suite issue as we focus on the similarity of test pass rate, rather than the
absolute value of test pass rate. (Zan et al., 2022; Li et al., 2022; Chen et
al., 2021; Hendrycks et al., 2021; Austin et al., 2021). Each code generation
problem has five test pass rates, one for each code candidate. We use the
variance and maximum difference of the five values to indicate semantic
similarity. We also calculate the mean of the five values for the purpose of
understanding correctness as well as the correlation between correctness and
non-determinism (RQ4).

The output equivalence rate records the ratio of equivalent test outputs
(across different code candidates for the same code generation instruction)
against the total test outputs. Each instruction has one output equivalence
rate. For tests that produce specific outputs (without exceptions or errors),
we check whether the output values of different code candidates are equal to
each other. In the following parts of this paper, we use OER to represent
output equivalence rate and use OER (no ex.) to represent output equivalence
rate (without exceptions or errors) for short. Each code generation problem
has only one OER and OER (no ex.). For tests that produce specific outputs
(without exceptions or errors), we check whether the output values of
different code candidates are equal to each other. For tests that yield
exceptions or timeout errors, we consider the test outputs to be the same if
the exception or error messages are the same.

Some papers use the pass@k metric (Kulal et al., 2019; Chen et al., 2021)
(i.e., the ratio of coding tasks with 100% test pass rate) to indicate the
high-level code generation correctness of a code generation approach. We do
not use this metric in our experiments because we focus on the non-determinism
threat, while pass@k ignores the correctness of each single coding task and
concentrates only on the ratio of correct code candidates in all the tasks,
which can cover the non-determinism across different requests.

####  3.4.2. Syntactic similarity

The syntactic similarity in this study treats different code candidates as
texts and checks their textual similarity. We choose the Longest Common
Subsequence and Levenshtein Edit Distance as evaluation tools. In the
following content, we use LCS and LED to represent the Longest Common
Subsequence and Levenshtein Edit Distance for short respectively. LCS measures
the similarity via the normalized length of the longest common subsequence
between two sequences. LED measures the minimum number of single-token edits
(insertions, deletions, or substitutions) required to change one code into the
other. LCS and LED both regard the token as the smallest unit, and the token
is divided by the .split() method, that is, any whitespace is used as the
separator to divide the code into tokens. LCS and LED are used to measure the
other four code candidates’ syntactic similarity with the first one. Thus,
each code-generation problem has four values of each metric. We use the mean
and worst value to indicate the syntactic similarity measured by each metric.

Below are the formulas for the LCS and LED:

$LCS=\frac{len(lcs(s,t))}{len(s)}$

where $s$ is reference string, $t$ is the string to be compared, $lcs(s,t)$ is
the longest common subsequence between $s$ and $t$.

${\scriptsize\text{LED}_{s,t}(i,j)=\begin{cases}\max(i,j)&\text{if
}\min(i,j)=0\\\ \min\begin{cases}\text{led}_{s,t}(i-1,j)+1\\\
\text{led}_{s,t}(i,j-1)+1\\\ \text{led}_{s,t}(i-1,j-1)+1_{(s_{i}\neq
t_{j})}\end{cases}&\text{otherwise}\end{cases}}$

where $\operatorname{LED}_{s,t}(i,j)$ is the LED between the first $i$
characters of $s$ and the first $j$ characters of $t$, and
$\operatorname{diff}(s_{i},t_{j})$ is $0$ if the $i$-th character of $s$ is
the same as the $j$-th character of $t$, and $1$ otherwise.

####  3.4.3. Structural similarity

We design structural similarity to measure the code similarity in terms of the
Abstract Syntax Tree (AST). AST is a tree-like representation of the source
code in which each node in the tree represents a construct in the code, such
as variable, function, or control structure, and the edges between nodes
represent the relationships between these constructs. We use a Python library
called pycode_similar111111https://github.com/fyrestone/pycode_similar (Li et
al., 2023; Wu et al., 2020) to calculate the similarity. The pycode_similar
normalizes Python code into AST representation and uses Python library difflib
to get the modification from referenced code to target code. There are two
difference measurement settings, i.e. Unified_Diff and Tree_Diff. Unified_Diff
measures the difference of normalized function AST string lines, while
Tree_Diff measures the difference in tree edit distance between two given
ASTs. Similar to syntactic similarity, for each code generation problem, we
compare the first code candidate with the remaining four and report the mean
and smallest similarity values.

##  4\. Results and Findings

This section introduces the experimental results as well as the analysis and
discussion for each RQ.

###  4.1. RQ1: Non-determinism of ChatGPT with Three Types of Similarities

####  4.1.1. RQ1.1: Semantic Similarity

Semantic similarity is measured by the following metrics: test pass rate and
OER (output equivalence rate), and OER excluding exceptions. As mentioned in
Section 3.4, each coding problem has five test pass rates, we use the variance
and maximum difference of these five values to indicate ChatGPT’s non-
determinism in generating code for the task. We also report the mean value,
which represents the average correctness of the generated code. For OER or OER
(no ex.), each coding problem has only one value, indicating the test output
similarity across all five code candidates. For each dataset, we report the
distribution of different measurements in Figure 2 and Figure 3. The mean
measurement values for all the coding problems (the mean value inside each bar
in each bar chart) in a dataset are shown in Table 2. The max diff refers to
the maximum value of the max diff among all the code problems. In addition,
Table 2 also shows the “Ratio of worst cases”, which is the ratio of problems
with maximum diff of test pass rate being 1 or OER being 0.

(a) Variance

(b) Mean

(c) Max Diff

Figure 2. RQ1.1: Distribution of semantic similarity in terms of test pass
rate.

(a) OER

(b) OER (no ex.)

Figure 3. RQ1.1: Distribution of semantic similarity in terms of test output
equivalence rate (OER and OER (no ex.)). Table 2. RQ1.1: Results of semantic
similarity. OER and OER (no ex.) are the output equivalence rate and the
equivalence rate excluding exceptions.

Semantic similarity | Metric | CodeContests | APPS | HumanEval
---|---|---|---|---
Test pass rate | Mean value | 0.13 | 0.35 | 0.55
Mean variance | 0.03 | 0.04 | 0.10
Mean max diff | 0.27 | 0.59 | 0.36
Max diff | 1.00 | 1.00 | 1.00
Ratio of worst cases | 7.88% | 10.80% | 48.17%
OER | Mean value | 0.13 | 0.22 | 0.30
Worst value | 0.00 | 0.00 | 0.00
Ratio of worst cases | 72.73% | 60.40% | 65.85%
OER (no ex.) | Mean value | 0.05 | 0.17 | 0.25
Worst value | 0.00 | 0.00 | 0.00
Ratio of worst cases | 81.82% | 66.00% | 71.34%

From Figure 2, Figure 3, and Table 2, we observe that ChatGPT is very unstable
in generating semantically consistent code candidates. In particular, the
ratios of tasks with zero equal test output (i.e., OER=0) among the five code
candidates are 72.73%, 60.40%, and 65.85% for the three datasets,
respectively. This indicates that for the majority of the cases, ChatGPT
generates code candidates with completely different semantics from identical
instructions.

The mean variance of the test pass rate is not big from Table 2, ranging
between 0.03 and 0.10, this is because the test pass rate of different code
candidates is often equally worse, as can be observed from Figure 2.(a).
However, the max diff of the test pass rate reaches 1.00 for all three
datasets and accounts for 48.17% of the problems in HumanEval, the most widely
used code generation benchmark. This indicates the correctness of code
candidates generated from the same instruction can vary significantly.

Among the three datasets, we observe that the HumanEval dataset demonstrates
the largest variance in the test pass rate, but also the best correctness in
terms of test pass rate. We suspect that this is because the coding tasks in
HumanEval are relatively easy for ChatGPT. The large difference in different
datasets also sheds light on the importance of using multiple datasets when
assessing the code generation performance for large language models.

Answer to RQ1.1: The semantic difference among the code generated by ChatGPT
in different requests is significant. In particular, the ratio of coding tasks
with not a single equal test output among the five different requests is
72.73%, 60.40%, and 65.85% for CodeContests, APPS, and HumanEval,
respectively. In addition, the maximum difference of the test pass rate
reaches 1.00 for all three datasets and accounts for 48.17% of the problems in
HumanEval, the most widely used code generation benchmark.

####  4.1.2. RQ1.2: Syntactic Similarity

Syntactic similarity measures the text similarity among code candidates. In
our experiment, the syntactic similarity is evaluated by the following
metrics: LCS and LED (more details in Section 3.4). For the five code
candidates for each coding problem, we use the first code candidate as
reference and calculate the LCS and LED between the reference and the
remaining four candidates. Thus, each problem has four LCS values and LED
values, each value indicating a syntactic similarity. We use the mean of these
four values as well as the worst of them (i.e., the smallest value for LCS and
the largest value for LED) to represent each problem’s syntactic similarity.
Figure 4 shows the distribution of LCS and LED for all the problems in each
dataset. Table 3 shows the mean and worst LCS and LED values for all the
coding problems (the mean value inside each bar in the figures) in a dataset.

(a) LCS mean

(b) LCS worst

(c) LED Mean

(d) LED Worst

Figure 4. RQ1.2: Distribution of syntactic similarity (LCS & LED). Lower LCS
and higher LED indicate less syntactic similarity. Table 3. RQ1.2: Syntactic
similarity (LCS and LED). Lower LCS and higher LED indicate lower syntactic
similarity.

Syntactic Similarity | Metric | CodeContests | APPS | HumanEval
---|---|---|---|---
LCS | Mean value | 0.12 | 0.14 | 0.45
Worst value | 0.07 | 0.09 | 0.33
LED | Mean value | 119.75 | 85.61 | 38.86
Worst value | 165.28 | 120.27 | 60.52

We observe that the code candidates generated from the same instruction also
differ largely in the syntactic measure. Specifically, the mean LCS is 0.12,
0.14, and 0.45 for CodeContests, APPS, and HumanEval, respectively, indicating
the mean ratio of the longest common subsequences among the code candidates.

For the three datasets, we could see from Table 3 that the lowest LCS and
largest LED values both happen for the CodeContests dataset. By contrast, the
largest LCS and smallest LED values both happen for HumanEval. This indicates
that ChatGPT is most unstable syntactically for the code generation tasks in
CodeContests, and most stable for HumanEval. We further explore the
correlation between different similarities and code task features in RQ4.

Answer to RQ1.2: Code candidates generated by ChatGPT in different requests
also differ significantly in syntax. The mean syntax similarity (LCS) is only
0.12, 0.14, and 0.45 for CodeContests, APPS, and HumanEval, respectively.

####  4.1.3. RQ1.3: Structural Similarity

Structural similarity measures the codes’ similarity based on their AST. In
our experiment, the structural similarity is mainly measured by the tool
pycode_similar with two difference settings, namely United_Diff and Tree_Diff
(more details in Section 3.4). For the five code candidates for each coding
problem, we use the first code candidate as reference and calculate the
structural similarity between the first candidate with the remaining four
candidates under United_Diff and Tree_Diff settings. Thus, each problem has
four values for United_Diff and Tree_Diff respectively, with each value
indicating a structural similarity measure. We use the mean of these four
values as well as the worst of them (i.e., the smallest value for United_Diff
and Tree_Diff) to represent each problem’s structural similarity. Fig 5 shows
the distribution of United_Diff and Tree_Diff for all the problems in each
dataset. Table 4 shows the mean and worst values under United_Diff and
Tree_Diff settings for all the coding problems (the mean value inside each bar
in the figures) in a dataset.

Table 4. RQ1.3: Structural similarity.

Structural Similarity | Metric | CodeContests | APPS | HumanEval
---|---|---|---|---
United_Diff | Mean value | 0.46 | 0.52 | 0.69
Worst value | 0.25 | 0.33 | 0.43
Tree_Diff | Mean value | 0.60 | 0.68 | 0.72
Worst value | 0.36 | 0.46 | 0.44

(a) Mean

(b) Worst

Figure 5. RQ3: Structural Similarity (UnitedDiff & TreeDiff).

We observe that the code candidates generated from the same instruction show
great similarity in structure. Specifically, the mean values are 0.46, 0.52,
and 0.69 under the United_Diff setting, and 0.60, 0.68, and 0.72 under
Tree_Diff setting for CodeContests, APPS, and HumanEval, respectively, which
indicates the mean ratio of AST similarity among the code candidates.

For the three datasets, we could see from Table 4 that the lowest values under
United_Diff and Tree_Diff happen for the CodeContests dataset. By contrast,
the largest values under the two settings both happen for HumanEval. This
indicates that ChatGPT is most unstable in structure for the code generation
tasks in CodeContests, and most stable for HumanEval. We further explore the
correlation between different similarities and task features in RQ4.

Answer to RQ1.3: Code candidates show high structural similarity under
UnitedDiff and TreeDiff settings. We observe that the code candidates
generated from the same instruction have high similarity in structure.
Specifically, the mean values are 0.46, 0.52, and 0.69 under the United_Diff
setting, and 0.60, 0.68, and 0.72 under Tree_Diff setting for CodeContests,
APPS, and HumanEval, respectively.

###  4.2. RQ2: Influence of Temperature

The default temperature of ChatGPT is 1. This RQ explores whether the code
generation non-determinism of ChatGPT changes with the changes in temperature.
We use identical measurements as in RQ1. Due to space limitation, we only show
the results for the CodeContests dataset as ChatGPT performs worst on it in
different types of similarity in code generation, allowing us more space to
observe the influence of temperature. The results for the other two datasets
are on our homepage (hom, [n. d.]).

Table 5. RQ2: Influence of temperature.

Temperature | Test Pass Rate
---|---
Mean | Mean | Mean | Max | Ratio of
| value | variance | max diff | diff | worst cases
0 | 0.15 | 0.02 | 0.21 | 1.00 | 4.85%
1 | 0.13 | 0.03 | 0.27 | 1.00 | 7.88%
2 | 0.01 | 0.01 | 0.02 | 1.00 | 0.61%
Temperature | OER
Mean value | Worst value | Ratio of worst cases
0 | 0.72 | 0.00 | 16.36%
1 | 0.13 | 0.00 | 72.73%
2 | 0.25 | 0.00 | 75.15%
Temperature | OER (no ex.)
Mean value | Worst value | Ratio of worst cases
0 | 0.52 | 0.00 | 33.94%
1 | 0.05 | 0.00 | 81.82%
2 | 0.01 | 0.00 | 99.39%
Temperature | LCS |  | LED
Mean value | Worst value |  | Mean value | Worst value
0 | 0.77 | 0.66 |  | 15.19 | 26.91
1 | 0.12 | 0.07 |  | 119.75 | 165.28
2 | 0.00 | 0.00 |  | 52.18 | 96.10
Temperature | United_Diff |  | Tree_Diff
Mean value | Worst value |  | Mean value | Worst value
0 | 0.84 | 0.79 |  | 0.86 | 0.83
1 | 0.34 | 0.25 |  | 0.45 | 0.34
2 | 0.00 | 0.00 |  | 0.01 | 0.00

Table 5 shows the results. Overall, we observe that when temperature=0,
ChatGPT has better determinism than the default configuration (temperature=1)
for all three types of similarities. However, setting the temperature to 0
does not completely avoid non-determinism. Take OER as an example, there are
still 16.36% of problems with no equal test output among the five code
candidates. This is contrary to many people’s belief (and common practice)
that setting the temperature to 0 can make ChatGPT deterministic (Deng et al.,
2023; Bhavya et al., 2022; Liang et al., 2023).

When temperature=2, interestingly, we observe that the mean variance of the
test pass rate is only 0.01, lower than both temperature=0 and temperature=1,
and the ratio of worst cases is only 0.61%. This is because the temperate
brings too much creativity to ChatGPT and affects its ability in generating
correct code (as can be observed from the mean test pass rate). As a result,
the generated code candidates tend to be uniformly worse.

Answer to RQ2: Contrary to the widely held belief (and common practices),
setting temperature to 0 does not guarantee determinism in code generation,
although it indeed brings better determinism than the default configuration
(temperature=1) for all three types of similarities. Setting the temperature
to 2 tends to yield uniformly worse code candidates, which may perversely
improve the similarity among code candidates.

###  4.3. RQ3: Non-determinism Comparison with Top Candidates in the Same
Prediction

RQ1 and RQ2 compare the similarity of 5 code candidates generated in multiple
requests. Each candidate is the top-1 candidate in each request. However,
ChatGPT can also generate 5 code candidates within the same request (the top-5
candidates ranked by their predictive probabilities). This RQ compares the
non-determinism degree of code candidates for the two request configurations
mentioned above. Table 6 shows the results for dataset CodeContests (full
results on our homepage (hom, [n. d.]). For ease of presentation, we use R1 to
refer to one-time requests, and R2 to refer to multiple requests.

Intuitively, the 5 top-1 candidates from 5 different requests (R2) should have
more similarity than the top-5 code candidates from one single request (R1),
because the former is a collection of the best responses for each request and
the latter is only the top-5 best. Nevertheless, our observation indicates
that the top-5 code candidates from one single request (R1) have very similar
similarities to the 5 top-1 candidates from multiple requests (R2).

Table 6. RQ3: Similarity for different request ways.

Request | Test Pass Rate
---|---
Way | Mean | Mean | Mean | Max | Ratio of
| value | variance | max diff | diff | worst cases
R1 | 0.13 | 0.03 | 0.22 | 1.00 | 4.24%
R2 | 0.13 | 0.03 | 0.27 | 1.00 | 7.88%
Request | OER
Way | Mean value | Worst value | Ratio of worst cases
R1 | 0.12 | 0.00 | 75.15%
R2 | 0.13 | 0.00 | 72.73%
Request | OER (no ex.)
Way | Mean value | Worst value | Ratio of worst cases
R1 | 0.03 | 0.00 | 86.06%
R2 | 0.05 | 0.00 | 81.82%
Request | LCS |  | LED
Way | Mean value | Worst value |  | Mean value | Worst value
R1 | 0.12 | 0.08 |  | 117.49 | 542.25
R2 | 0.12 | 0.07 |  | 119.75 | 165.28
Request | United_Diff |  | Tree_Diff
Way | Mean value | Worst value |  | Mean value | Worst value
R1 | 0.36 | 0.27 |  | 0.47 | 0.36
R2 | 0.34 | 0.25 |  | 0.45 | 0.34

Answer to RQ3: The top-5 code candidates from one single request have very
similar non-determinism with the 5 top-1 candidates from different requests
for ChatGPT.

###  4.4. RQ4: Correlations between Coding Tasks and Non-determinism Degree

Our previous experiments demonstrate that there are many non-determinisms in
ChatGPT in code generation. This RQ investigates what affects such non-
determinism by checking the correlation between characteristics of coding
tasks and similarity metric values. We use the CodeContests dataset for this
RQ because it is the only dataset with various features for each coding task,
including difficulty, time limit, and CF rating. The difficulty of a problem
is a qualitative measure that indicates the problem’s level of complexity and
the programming knowledge and skills required to solve it. The time out
indicates the program’s maximum running time limitation. The CF rating of a
problem is a quantitative measure that represents the problem’s relative
difficulty level compared to other problems on the Codeforces platform. In
addition, we also consider description length (i.e., number of characters) for
each coding task. Note that in this section, we only focus on correlation
analysis, and we do not aim to obtain any causal conclusions.

Figure 6 shows the results. For the four coding task features (description
length, difficulty, time limit, and CF rating), We observe that description
length has a negative correlation with the mean test pass rate. This means
that problems with longer descriptions tend to have worse test pass rates. We
suspect that this is because a longer description may reduce ChatGPT’s
understanding of the coding requirements (Liu et al., 2023c). Description
length also has a negative correlation with the variance of test pass rate,
and max diff of test pass rate. This is because with longer descriptions
different code candidates tend to be uniformally worse in their pass rates.
Nevertheless, the description length has a negative correlation with LCS and
United_Diff, which means that problems with longer descriptions tend to yield
more inconsistent code candidates in syntax and structure.

Figure 6. RQ4: Correlations between coding tasks and non-determinism. Only
significant correlations will be displayed on the heatmap, while the
insignificant correlations (i.e. p-value $>$ 0.05) are masked by ’-’.

The difficulty level has a positive correlation with the test pass rate
variance, which means that the problem with a higher difficulty level has a
high variance in the test pass rate. CF rating has the only significant
negative correlation with the test pass rate mean, which implies that code
problems with a higher CF rating may have a lower test pass rate on average.
We also observe that the features clustered in the same type of similarity
measurements demonstrate high correlations with p-value$<$0.05, such as test
pass rate mean, test pass rate mean variance, and test pass rate mean max
diff. This also indicates the reliability of the different measurements we
have chosen for each type of similarity. Interestingly, we do not observe any
correlation between semantic similarity and syntactic similarity, although the
correlation between syntactic similarity and structural similarity is strong.

Answer to RQ4: Description length of a coding task has a negative correlation
with LCS, United_Diff, and test pass rate. This means that a longer
description of a coding task tends to suffer from more non-determinism in the
generated code in terms of code syntax and structure. The generated code also
tends to be more buggy.

###  4.5. RQ5: GPT-4 vs. GPT-3.5

GPT-4 is believed to be “more reliable, creative, and able to handle much more
nuanced instructions than GPT-3.5” according to OpenAI (OpenAI, 2023). This
research question compares GPT-3.5 and GPT-4 in the non-determinism degree of
code generation. To answer this research question, we keep the default setting
and use all the measurements listed in RQ1. We report the results only on the
most widely used HumanEval dataset (with temperature=1) due to the rate limit
issue (rat, [n. d.]) and the high price (mod, [n. d.]) of GPT-4.

From Table 7, we observe that the non-determinism issue of GPT-4 is also
severe, and is overall slightly worse than GPT-3.5 in addition to the OER
similarity. This is because GPT-4 is known to be more creative than GPT-3.5,
while more creativity often means more non-determinism (Bubeck et al., 2023).
For OER, GPT-4 has a larger similarity than GPT-3.5. We suspect that this is
because GPT-4 produces more cases with a test pass rate of 1.0, thereby
introducing more identical test outputs. This observation does not apply to
the similarity of test pass rate, because the code candidates from GPT-3.5 can
be equally incorrect. For example, for a certain code task in our experiments,
GPT-3.5’s test pass rate is [0,0,0,0.8,0.8], while GPT-4’s test pass rate is
[1,1,1,1,0]. GPT-4’s OER, and OER (no ex.) are obviously higher than GPT-3.5,
but the test pass rate variance and max diff are slightly larger.

Table 7. RQ5: Non-determinism of GPT-4 v.s. GPT-3.5.

Model | Test Pass Rate
---|---
Mean | Mean | Mean | Max | Ratio of
| value | variance | max diff | diff | worst cases
GPT-4 | 0.73 | 0.11 | 0.60 | 1.00 | 53.66%
GPT-3.5 | 0.55 | 0.10 | 0.36 | 1.00 | 48.17%
Model | OER
Mean value | Worst value | Ratio of worst cases
GPT-4 | 0.37 | 0.00 | 59.76%
GPT-3.5 | 0.30 | 0.00 | 65.85%
Model | OER (no ex.)
Mean value | Worst value | Ratio of worst cases
GPT-4 | 0.35 | 0.00 | 60.98%
GPT-3.5 | 0.25 | 0.00 | 71.34%
Model | LCS |  | LED
Mean value | Worst value |  | Mean value | Worst value
GPT-4 | 0.22 | 0.11 |  | 39.30 | 57.00
GPT-3.5 | 0.45 | 0.33 |  | 38.86 | 60.52
Model | United_Diff |  | Tree_Diff
Mean value | Worst value |  | Mean value | Worst value
GPT-4 | 0.52 | 0.31 |  | 0.65 | 0.45
GPT-3.5 | 0.69 | 0.43 |  | 0.72 | 0.44

Answer to RQ5: The non-determinism issue of GPT-4 is also severe, and is
overall slightly worse than GPT-3.5.

##  5\. Threats to Validity

The threats to internal validity mainly lie in the implementation of our
experiment and manual analysis. To reduce the first threat, we checked our
code twice, once during the experiment stage, and once during the record
analysis stage. To reduce the second threat, the two authors independently
analyzed the experiment results and drew experimental conclusions separately.
Once their analysis results are different, the third author discussed with
them to determine the final result.

The threats to external validity mainly lie in the datasets and measurement
tools used in our study. To reduce the first threat, we use three diverse
datasets that are widely used in code generation tasks. Additionally, the
problems in our dataset are from different contests with different
difficulties. For example, CodeContests is the most challenging dataset, while
HumanEval is the easiest, in terms of the average difficulty of code problems.
To reduce the second threat, we consider three types of similarities and
choose at least two measurements for each type of similarity.

##  6\. Related Work

###  6.1. Code Generation

Code generation generates programs that need to satisfy all the constraints
defined by the underlying task. Usually, the constraints are represented in
various forms, e.g. input/output pairs, examples, problem descriptions,
partial programs, and assertions. Relatively early work includes deductive
synthesis approaches (Manna and Waldinger, 1971; Green, 1981) and inductive
synthesis approaches (Shaw et al., 1975; Biermann, 1978; Summers, 1977; Smith,
1975). The deductive synthesis approach operated under the assumption that a
comprehensive and precise formal specification of the user’s desired intention
would be provided. However, in many instances, this turned out to be just as
intricate and challenging as creating the actual program. While the inductive
synthesis approach was based on inductive specifications such as input/output
pairs and examples etc, such as works on Lisp programs (Shaw et al., 1975;
Biermann, 1978; Summers, 1977), Pygmalion (Smith, 1975) and more recently
FlashFill (Gulwani, 2011). More information could be found in a survey
(Gulwani et al., 2017), which covers notable work on the development of
program synthesis approaches.

In recent years, more and more researchers apply neural networks in code
generation. Yin and Neubig (Yin and Neubig, 2017) combined the grammar rules
with the decoder and propose a syntax-driven neural architecture to improve
code generation performance. Instead of RNN, Sun et al. (Sun et al., 2019) a
grammar-based structural CNN to capture the long dependency in code. Bolin et
al. (Wei et al., 2019) propose a dual learning framework that jointly trains
the code generation model and code summarization model together to achieve
better performance in both tasks. Xu et al.(Xu et al., 2022) present a user
study in-IDE code generation, demonstrating challenges such as time
efficiency, correctness, and code quality, as well as the willingness to use
code generation tools from developers.

###  6.2. Language Model for Code generation

The triumph of transformers in natural language modeling (Brown et al., 2020)
has stimulated considerable interest among researchers in applying transformer
models for code generation. Existing research on code generation models can be
classified into three categories: sequence-based techniques, tree-based
methods, and pre-trained models.

Sequence-based techniques take code as a sequence of tokens and employ
language models to produce source code one token at a time based on input
descriptions. Ling et al. (Ling et al., 2016) propose a generative model for
code generation along with a character level softmax and multi-pointer network
to address the problem of generating code from a mixed language and structured
specification, and receiving success in trading card games (Magic the
Gathering and Hearthstone). Hashimoto et al. (Hashimoto et al., 2018) train a
retrieval model with a noisy encoder-decoder to enable similar code
retrieving, and then use the similar code as an additional input to improve
the performance of the generator.

Tree-based methods generate a parse tree of the code, e.g. Abstract Syntax
Tree (AST), based on the input description, and then convert the parse tree
into the corresponding code. Dong et al. (Dong and Lapata, 2016) encode
natural language utterances into vectors and generate their corresponding
logical forms as trees using the LSTM model. Yin et al. (Yin and Neubig, 2018)
propose a semantic parser ‘Tranx’, which generates the tree-construction
action sequence with a transition-based neural model, and constructs the AST
from the action sequence.

Pre-trained models are obtained from training on massive data of source code,
which could be later fine-tuned on certain datasets for code generation
purposes. Encoder pre-trained models, such as CodeBERT (Feng et al., 2020),
usually are trained with two objectives, i.e., Masked Language Modeling and
Replaced Token Detection. During the fine-tuning phase, the input should be
fed in the same way as the pre-training phrase, so that semantic relevance
could be measured. Decoder pre-trained models are designed to predict the next
token based on a given input context. GPT-series (Radford et al., 2018) are
typical Decoder pre-trained models, and based on GPT, there are many efforts
on code generation. Based on GPT-2, Lu et al. (Lu et al., 2021) provide
CodeGPT for code completion and text-to-code generation. After GPT-3 was
developed, CodeX121212https://openai.com/blog/openai-codex and GitHub
Compilot131313https://github.com/features/copilot was created and released
their beta version for trial by academia and industry. Due to neither Codex
nor GitHub Copilot being open-sourced, there are several attempts to reproduce
their performance, like PYCODEGPT-CERT (Zan et al., 2022),
CodeParrot141414https://huggingface.co/codeparrot/codeparrot, and GPT-
CC151515https://github.com/CodedotAl/gpt-code-clippy. Encoder-decoder pre-
trained models are composed of an encoder and a decoder. AlphaCode (Li et al.,
2022), which is pre-trained through GitHub repositories with 715.1 GB of code,
uses an encoder-decoder transformer architecture. It achieves on average a
ranking in the top 54% in competitions with more than 5,000 participants in
simulated evaluations.

ChatGPT, a language model developed by the team of OpenAI, has the potential
to play a role in code generation. As it is widely known, ChatGPT offers a
chat window to enable interaction in a conversational way. In addition to its
powerful capabilities for natural language processing tasks, ChatGPT inherits
the code generation capabilities from Codex and can perform even better, so
the OpenAI team has announced the deprecation of Codex series models in its
official documents. There are several research works that mentioned its
ability in code-related areas, including mathematical capability (Frieder et
al., 2023), bug-solving capability (Surameery and Shakor, 2023), and software
testing (Jalil et al., 2023). ChatGPT’s ‘Regenerate response’ function
demonstrates the diversity of its output, but at the same time, it also raises
concerns about the consistency of its output given the same input. Currently,
people are amazed by its superficial performance in terms of code generation,
however, there is still no research work focused on the threat of non-
determinism. Therefore, we think it is necessary to make a comprehensive
evaluation of ChatGPT’s ability in code generation. More detailed information
could be found on its official website’s
blog161616https://chat.openai.com/chat.

###  6.3. Non-determinism Handling in the Literature

The non-determinism issue has been studied well in traditional Deep Learing-
related research. Pham et al.(Pham et al., 2020) measure the influence of
nondeterminism-introducing (NI)-factors in Deep Learning, and study the
awareness of this variance among researchers and practitioners. However, the
common practice of handling the threat of non-determinism in LLM-based code
generation remains unclear.

To understand how well LLM-based code generation papers handle the threat of
non-determinism, we collect research articles from Google Scholar with the
query ‘code generation’ AND ‘Large Language Model’ in the past 2 years (from
January 2022 to July 2023). There are 107 papers obtained from Google Scholar
according to their relevance rankings. In this survey, we mainly focus on
articles with experiments and exclude those with posters and visions only,
which yields a set of 76 papers. After an in-depth reading of the experimental
design and discussion in each paper, we find that only 35.5% (27/76) out of
the 76 papers mention non-determinism or related terms (e.g., stability,
randomness, and variance) in their papers. Among them, 21.1% (16/76) papers
consider non-determinism in their experimental evaluation, including fixed
random seeds, multiple runs of experiments with different fixed random seeds,
and report results with error bars or standard deviation. In addition, 14.5%
(11/76) of the papers do not consider non-determinism in their experiments,
but discuss the threat of non-determinism in their paper.

##  7\. Implications for Software Developers and Researchers

For developers, it is essential to recognize the limitations of ChatGPT and
the potential risks of using generated code in production. If developers
prefer a more stable code, they can use a smaller temperature but should keep
in mind that even the smallest temperature (i.e., temperature=0) could not
guarantee the determinism. Moreover, our observation on the correlation
between the length of prompts and code correctness/non-determinism suggests
the importance of prompt engineering. Developers should thoroughly test the
generated code before deploying it, and even consider incorporating more
robust testing and validation processes to ensure the determinism and
reliability of the generated code.

For researchers, the variance of the generated code raises questions about the
quality and validity of the results obtained from assessing LLMs in code
generation. If the code generated from ChatGPT is unstable, it can lead to
non-reproducible results and unreliable conclusions. Therefore, researchers
should carefully consider the limitations of ChatGPT when designing
experiments and interpreting results. To reduce the randomness caused by the
non-determinism issue, researchers can report the average results, variance,
or distribution from multiple requests. Also, it is important to use different
datasets, since our study finds that both the correctness and non-determinism
of the generated code vary significantly from dataset to dataset.

Our paper points out the issue of non-determinism in ChatGPT in code
generation. However, it is worth noting that non-determinism also has benefits
for the creativity of LLMs. Achieving determinism in LLMs may sometimes come
at the cost of reduced creativity or diversity in the generated text. Striking
the right balance between determinism and creativity is essential for
optimizing the performance of LLMs for various applications.

##  8\. Conclusion

This work studies the non-determinism threat of code generation with ChatGPT.
We perform experiments on three widely-studied code generation benchmarks and
find that the correctness, test outputs, as well as syntax and structure of
code candidates generated from the same instruction, vary significantly in
different requests. The ratio of code generation tasks with zero identical
test output among code candidates is 72.73%, 60.40%, and 65.85% for
CodeContests, APPS, and HumanEval, respectively. We also find that GPT-4 has
slightly worse determinism than GPT-3.5 with the default configuration. We
hope that this paper could raise awareness of the threat of non-determinism in
future code generation tasks when using large language models.

## References

  * (1)
  * hom ([n. d.]) [n. d.].  .  https://github.com/CodeHero0/Nondeterminism-of-ChatGPT-in-Code-Generation
  * rat ([n. d.]) [n. d.].  .  https://platform.openai.com/docs/guides/rate-limits/gpt-4-rate-limits
  * mod ([n. d.]) [n. d.].  .  https://openai.com/pricing#language-models
  * Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021\.  Program synthesis with large language models.  _arXiv preprint arXiv:2108.07732_ (2021).
  * Bang et al. (2023) Y Bang, S Cahyawijaya, N Lee, W Dai, D Su, B Wilie, H Lovenia, Z Ji, T Yu, W Chung, et al. 2023\.  A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. arXiv.
  * Bhavya et al. (2022) Bhavya Bhavya, Jinjun Xiong, and Chengxiang Zhai. 2022\.  Analogy generation by prompting large language models: A case study of instructgpt.  _arXiv preprint arXiv:2210.04186_ (2022).
  * Biermann (1978) Alan W Biermann. 1978\.  The inference of regular LISP programs from examples.  _IEEE transactions on Systems, Man, and Cybernetics_ 8, 8 (1978), 585–600.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023\.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_ (2023).
  * Chatterjee et al. (2022) Subhashis Chatterjee, Deepjyoti Saha, Akhilesh Sharma, and Yogesh Verma. 2022.  Reliability and optimal release time analysis for multi up-gradation software with imperfect debugging and varied testing coverage under the effect of random field environments.  _Annals of Operations Research_ (2022), 1–21.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021\.  Evaluating large language models trained on code.(2021).  _arXiv preprint arXiv:2107.03374_ (2021).
  * Deng et al. (2023) Yinlin Deng, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shujing Yang, and Lingming Zhang. 2023.  Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt.  _arXiv preprint arXiv:2304.02014_ (2023).
  * Dong and Lapata (2016) Li Dong and Mirella Lapata. 2016.  Language to logical form with neural attention.  _arXiv preprint arXiv:1601.01280_ (2016).
  * Feng et al. (2023) Yunhe Feng, Sreecharan Vanam, Manasa Cherukupally, Weijian Zheng, Meikang Qiu, and Haihua Chen. 2023\.  Investigating Code Generation Performance of Chat-GPT with Crowdsourcing Social Data. In _Proceedings of the 47th IEEE Computer Software and Applications Conference_. 1–10.
  * Feng et al. (2020) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020\.  Codebert: A pre-trained model for programming and natural languages.  _arXiv preprint arXiv:2002.08155_ (2020).
  * Frieder et al. (2023) Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. 2023\.  Mathematical capabilities of chatgpt.  _arXiv preprint arXiv:2301.13867_ (2023).
  * Green (1981) Cordell Green. 1981\.  Application of theorem proving to problem solving.  In _Readings in Artificial Intelligence_. Elsevier, 202–222.
  * Gulwani (2011) Sumit Gulwani. 2011\.  Automating string processing in spreadsheets using input-output examples.  _ACM Sigplan Notices_ 46, 1 (2011), 317–330.
  * Gulwani et al. (2017) Sumit Gulwani, Oleksandr Polozov, Rishabh Singh, et al. 2017\.  Program synthesis.  _Foundations and Trends® in Programming Languages_ 4, 1-2 (2017), 1–119.
  * Hashimoto et al. (2018) Tatsunori B Hashimoto, Kelvin Guu, Yonatan Oren, and Percy S Liang. 2018.  A retrieve-and-edit framework for predicting structured outputs.  _Advances in Neural Information Processing Systems_ 31 (2018).
  * Hassani and Silva (2023) Hossein Hassani and Emmanuel Sirmal Silva. 2023.  The role of ChatGPT in data science: how ai-assisted conversational interfaces are revolutionizing the field.  _Big data and cognitive computing_ 7, 2 (2023), 62\.
  * Hendrycks et al. (2021) Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021\.  Measuring coding challenge competence with apps.  _arXiv preprint arXiv:2105.09938_ (2021).
  * Inala et al. (2022) Jeevana Priya Inala, Chenglong Wang, Mei Yang, Andres Codas, Mark Encarnación, Shuvendu Lahiri, Madanlal Musuvathi, and Jianfeng Gao. 2022.  Fault-aware neural code rankers.  _Advances in Neural Information Processing Systems_ 35 (2022), 13419–13432.
  * Jalil et al. (2023) Sajed Jalil, Suzzana Rafi, Thomas D LaToza, Kevin Moran, and Wing Lam. 2023.  Chatgpt and software testing education: Promises & perils. In _2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)_. IEEE, 4130–4137.
  * Kiviriga (2023) Andrej Kiviriga. 2023\.  Efficient Model Checking: The Power of Randomness.  (2023).
  * Krishna et al. (2022) Kalpesh Krishna, Yapei Chang, John Wieting, and Mohit Iyyer. 2022\.  Rankgen: Improving text generation with large ranking models.  _arXiv preprint arXiv:2205.09726_ (2022).
  * Kulal et al. (2019) Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. 2019.  Spoc: Search-based pseudocode to code.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * Lee et al. (2022) Mina Lee, Percy Liang, and Qian Yang. 2022.  Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities. In _Proceedings of the 2022 CHI conference on human factors in computing systems_. 1–19.
  * Li et al. (2022) Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022\.  Competition-level code generation with alphacode.  _Science_ 378, 6624 (2022), 1092–1097.
  * Li et al. (2023) Zongjie Li, Chaozheng Wang, Zhibo Liu, Haoxuan Wang, Dong Chen, Shuai Wang, and Cuiyun Gao. 2023.  Cctest: Testing and repairing code completion systems. In _2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)_. IEEE, 1238–1250.
  * Liang et al. (2023) Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. 2023\.  Code as policies: Language model programs for embodied control. In _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 9493–9500.
  * Ling et al. (2016) Wang Ling, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, Andrew Senior, Fumin Wang, and Phil Blunsom. 2016\.  Latent predictor networks for code generation.  _arXiv preprint arXiv:1603.06744_ (2016).
  * Liu et al. (2023d) Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023d.  Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.  _arXiv preprint arXiv:2305.01210_ (2023).
  * Liu et al. (2023b) Kaibo Liu, Yudong Han, Jie M Zhang, Zhenpeng Chen, Federica Sarro, Mark Harman, Gang Huang, and Yun Ma. 2023b.  Who Judges the Judge: An Empirical Study on Online Judge Tests. In _Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis_.
  * Liu et al. (2023c) Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023c.  Lost in the middle: How language models use long contexts.  _arXiv preprint arXiv:2307.03172_ (2023).
  * Liu et al. (2023a) Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, et al. 2023a.  Summary of chatgpt/gpt-4 research and perspective towards the future of large language models.  _arXiv preprint arXiv:2304.01852_ (2023).
  * Lu et al. (2021) Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021\.  Codexglue: A machine learning benchmark dataset for code understanding and generation.  _arXiv preprint arXiv:2102.04664_ (2021).
  * Manna and Waldinger (1971) Zohar Manna and Richard J Waldinger. 1971.  Toward automatic program synthesis.  _Commun. ACM_ 14, 3 (1971), 151–165.
  * Mitchell et al. (2023) Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023\.  Detectgpt: Zero-shot machine-generated text detection using probability curvature.  _arXiv preprint arXiv:2301.11305_ (2023).
  * Nagarajan et al. (2018) Prabhat Nagarajan, Garrett Warnell, and Peter Stone. 2018\.  Deterministic implementations for reproducibility in deep reinforcement learning.  _arXiv preprint arXiv:1809.05676_ (2018).
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  arXiv:2303.08774 [cs.CL]
  * Pham et al. (2020) Hung Viet Pham, Shangshu Qian, Jiannan Wang, Thibaud Lutellier, Jonathan Rosenthal, Lin Tan, Yaoliang Yu, and Nachiappan Nagappan. 2020\.  Problems and opportunities in training deep learning software systems: An analysis of variance. In _Proceedings of the 35th IEEE/ACM international conference on automated software engineering_. 771–783.
  * Poesia et al. (2022) Gabriel Poesia, Oleksandr Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022.  Synchromesh: Reliable code generation from pre-trained language models.  _arXiv preprint arXiv:2201.11227_ (2022).
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018\.  Improving language understanding by generative pre-training.  (2018).
  * Roziere et al. (2021) Baptiste Roziere, Jie Zhang, Francois Charton, Mark Harman, Gabriel Synnaeve, and Guillaume Lample. 2021.  Leveraging Automated Unit Tests for Unsupervised Code Translation. In _International Conference on Learning Representations_.
  * Shaw et al. (1975) David E Shaw, William R Swartout, and C Cordell Green. 1975\.  Inferring LISP Programs From Examples.. In _IJCAI_ , Vol. 75. 260–267.
  * Smith (1975) David Canfield Smith. 1975\.  _Pygmalion: a creative programming environment._ Stanford University.
  * Soares et al. (2022) Ioana Baldini Soares, Dennis Wei, Karthikeyan Natesan Ramamurthy, Moninder Singh, and Mikhail Yurochkin. 2022.  Your fairness may vary: pretrained language model fairness in toxic text classification. In _Annual Meeting of the Association for Computational Linguistics_.
  * Summers (1977) Phillip D Summers. 1977\.  A methodology for LISP program construction from examples.  _Journal of the ACM (JACM)_ 24, 1 (1977), 161–175.
  * Sun et al. (2019) Zeyu Sun, Qihao Zhu, Lili Mou, Yingfei Xiong, Ge Li, and Lu Zhang. 2019\.  A grammar-based structural cnn decoder for code generation. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 33. 7055–7062.
  * Surameery and Shakor (2023) Nigar M Shafiq Surameery and Mohammed Y Shakor. 2023.  Use chat gpt to solve programming bugs.  _International Journal of Information Technology & Computer Engineering (IJITC) ISSN: 2455-5290_ 3, 01 (2023), 17–22.
  * Svyatkovskiy et al. (2020) Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020.  Intellicode compose: Code generation using transformer. In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 1433–1443.
  * Vaithilingam et al. (2022) Priyan Vaithilingam, Tianyi Zhang, and Elena L Glassman. 2022\.  Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In _Chi conference on human factors in computing systems extended abstracts_. 1–7.
  * Wei et al. (2019) Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu, and Zhi Jin. 2019.  Code generation as a dual task of code summarization.  _Advances in neural information processing systems_ 32 (2019).
  * Wu et al. (2020) Xiongfei Wu, Liangyu Qin, Bing Yu, Xiaofei Xie, Lei Ma, Yinxing Xue, Yang Liu, and Jianjun Zhao. 2020\.  How are deep learning models similar? an empirical study on clone analysis of deep learning software. In _Proceedings of the 28th International Conference on Program Comprehension_. 172–183.
  * Xu et al. (2022) Frank F Xu, Bogdan Vasilescu, and Graham Neubig. 2022\.  In-ide code generation from natural language: Promise and challenges.  _ACM Transactions on Software Engineering and Methodology (TOSEM)_ 31, 2 (2022), 1–47.
  * Yetiştiren et al. (2023) Burak Yetiştiren, Işık Özsoy, Miray Ayerdem, and Eray Tüzün. 2023.  Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT.  _arXiv preprint arXiv:2304.10778_ (2023).
  * Yin and Neubig (2017) Pengcheng Yin and Graham Neubig. 2017.  A syntactic neural model for general-purpose code generation.  _arXiv preprint arXiv:1704.01696_ (2017).
  * Yin and Neubig (2018) Pengcheng Yin and Graham Neubig. 2018.  Tranx: A transition-based neural abstract syntax parser for semantic parsing and code generation.  _arXiv preprint arXiv:1810.02720_ (2018).
  * Zan et al. (2022) Daoguang Zan, Bei Chen, Dejian Yang, Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang, Weizhu Chen, and Jian-Guang Lou. 2022.  CERT: Continual Pre-training on Sketches for Library-oriented Code Generation.  _arXiv preprint arXiv:2206.06888_ (2022).