# An empirical study of ChatGPT-3.5 on question answering and code maintenance

Md Mahir Asef Kabir  Virginia TechUSA<EMAIL_ADDRESS>,  Sk Adnan Hassan
Walmart, Inc.USA<EMAIL_ADDRESS>,  Xiaoyin Wang  University of Texas
at San AntonioUSA<EMAIL_ADDRESS>,  Ying Wang  Northeastern
UniversityChina<EMAIL_ADDRESS>,  Hai Yu  Northeastern
UniversityChina<EMAIL_ADDRESS>and  Na Meng  Virginia TechUSA
<EMAIL_ADDRESS>
(2018)

###### Abstract.

Ever since the launch of ChatGPT in 2022, a rising concern is whether ChatGPT
will replace programmers and kill jobs. Motivated by this widespread concern,
we conducted an empirical study to systematically compare ChatGPT against
programmers in question-answering and software-maintaining. We reused a
dataset introduced by prior work, which includes 130 StackOverflow (SO)
discussion threads referred to by the Java developers of 357 GitHub projects.
We mainly investigated three research questions (RQs). First, how does ChatGPT
compare with programmers when answering technical questions? Second, how do
developers perceive the differences between ChatGPT’s answers and SO answers?
Third, how does ChatGPT compare with humans when revising code for maintenance
requests?

For RQ1, we provided the 130 SO questions to ChatGPT, and manually compared
ChatGPT answers with the accepted/most popular SO answers in terms of
relevance, readability, informativeness, comprehensiveness, and reusability.
For RQ2, we conducted a user study with 30 developers, asking each developer
to assess and compare 10 pairs of answers, without knowing the information
source (i.e., ChatGPT or SO). For RQ3, we distilled 48 software maintenance
tasks from 48 GitHub projects citing the studied SO threads. We queried
ChatGPT to revise a given Java file, and to incorporate the code
implementation for any prescribed maintenance requirement. Our study reveals
interesting phenomena: For the majority of SO questions (97/130), ChatGPT
provided better answers; in 203 of 300 ratings, developers preferred ChatGPT
answers to SO answers; ChatGPT revised code correctly for 22 of the 48 tasks.
Our research will expand people’s knowledge of ChatGPT capabilities, and shed
light on future adoption of ChatGPT by the software industry.

empirical study, ChatGPT, StackOverflow, Q&A, software maintenance

††copyright: acmcopyright††journalyear: 2018††doi: XXXXXXX.XXXXXXX††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Human-centered computing Empirical
studies in collaborative and social computing††ccs: Software and its
engineering Software maintenance tools††ccs: Software and its engineering
Search-based software engineering

##  1\. Introduction

ChatGPT is a large language model-based chatbot developed by OpenAI; it can
answer questions and assist users with different tasks, such as composing
emails, essays, and code (cha, 2023d). Ever since ChatGPT’s launch in November
2022, heated debates have spawned concerning its impacts on industries,
society, economy, and regulations (cha, 2023a, b; wil, 2023c; gpt, 2023; wit,
2023; wil, 2023a; pat, 2023; wil, 2023b; quo, 2023). For instance, many people
hold pessimistic attitudes (cha, 2023a, b; wil, 2023c; wit, 2023). They
believe that ChatGPT could replace white-collar workers in sectors like
education, finance, software, journalism, and graphic design. Meanwhile, some
people are optimistic about ChatGPT’s role in the software industry, treating
it as a coding assistant to help improve programmer productivity (wil, 2023a;
pat, 2023; wil, 2023b; quo, 2023). The reasons they mentioned include (1)
ChatGPT uses existing code available online to answer questions, but has no
creativity to produce new code (wil, 2023a); (2) it partially automates coding
tasks (pat, 2023); (3) it is only capable of basic coding (wil, 2023b). Albeit
the heated discussion, little systematic study was done to qualitatively and
quantitatively assess ChatGPT’s programming capability.

To demystify the capability of ChatGPT and predict its potential role in
software development, we conducted an empirical study to thoroughly compare
ChatGPT against human programmers in two major scenarios: question answering
and software maintenance. We chose these scenarios for two reasons. First,
question-and-answer (Q&A) is the main interaction mode between software
developers and ChatGPT; the provided answers are essential in shaping the art
and practices of software in the future. Thus, comparing the answers provided
by both ChatGPT and humans will help developers decide how to integrate
ChatGPT into their daily programming practices, and how much trust to give to
ChatGPT’s answers. Second, software maintenance cost can make up to 90% of the
whole software development cost (sof, 2021), which means that developers are
likely to spend the majority of their time and effort maintaining software. By
using ChatGPT, developers can get answers instantly, without waiting for
someone to notice their query on a forum. Therefore, our comparison between
ChatGPT’s responses and developers’ responses to the same maintenance requests
will demonstrate how probably ChatGPT can replace humans.

For our study, we reused a dataset introduced by prior work (Chen et al.,
2023b), which includes 130 StackOverflow (SO) discussion threads referenced by
357 GitHub projects. Each thread contains a question and at least an answer,
with identifiable accepted or most popular answer(s). Each of the threads has
URLs explicitly referenced by at least one GitHub project. Based on the
dataset, we investigated the following three research questions (RQs):

  * RQ1

_How does ChatGPT compare with programmers when answering technical
questions?_ We provided 130 SO questions to ChatGPT, and manually compared
ChatGPT answers with the accepted/most popular SO answers. The 130 samples
belong to 5 major categories (e.g., optimization and debugging), and cover 9
technical topics (e.g., data processing and testing).

  * RQ2

_How do developers consider the differences between ChatGPT answers and SO
answers?_ We did a user study with 30 developers. We gave each participant 10
SO questions, 10 accepted or most popular SO answers, and 10 ChatGPT answers.
For each question, a participant assessed and compared the two given answers,
without knowing the answer providers.

  * RQ3

_How does ChatGPT compare with humans when revising code for maintenance
requests?_ We mined the 357 GitHub repositories, for any commit that
introduces both an SO link and code revision to an existing Java file. We
formulated a prompt based on the SO thread and original Java file, asking
ChatGPT to produce a new version of that file to integrate the requested
feature implementation. We formulated in total 48 prompts to query ChatGPT.

Our work provides empirical evidence for many interesting phenomena that are
rarely mentioned by prior work. The major findings are summarized as below:

  * •

For 75% (97/130) of SO questions, ChatGPT provided better answers than the
accepted or most popular SO answers. The 97 answers respond to questions of
all the styles and topics we examined. It means that ChatGPT is strong at
answering various technical questions.

  * •

Among the 300 ratings provided by developers, 68% of ratings imply that
ChatGPT answers are better; only 32% of ratings imply SO answers to be better.
Compared with other question styles, ChatGPT answers to comprehension
questions were rated higher more often.

  * •

Given 48 maintenance tasks, ChatGPT modified Java files for all tasks.
However, only 22 of these revised files can be smoothly integrated into GitHub
repositories. ChatGPT-3.5 could not correctly maintain software in 54% of
cases. Developers need to do extra work to integrate or further improve the
code revision recommended by ChatGPT.

##  2\. Background: The GitHub$\rightarrow$SO Dataset Used in Our Study

In our research, one challenge is: _What prompts should we provide to ChatGPT,
in order to compare it against developers in a realistic and fair manner?_ To
overcome this challenge, we chose to reuse the dataset constructed by Chen et
al. (so-, 2023) for their recent investigation on how GitHub developers reuse
StackOverflow answers (Chen et al., 2023b). The dataset consists of 130
StackOverflow (SO) discussion threads, referenced by Java files in 357 GitHub
repositories. Every thread has a question post and at least one answer post;
every post is assigned with a unique URL. Each of the included Java files
references an SO thread via either the question URL, or URL of any answer
belonging to that thread.

Figure 1. The taxonomy of SO threads based on both question styles and
technical content (Chen et al., 2023b)

As shown in Fig. 1, Chen et al. classified the 130 threads into 5 categories
based on the question styles. Specifically, _Coding task_ means that askers
describe software requirements and seek for code solutions. _Optimization_
means that askers provide initial programs satisfying certain requirements,
looking for better programs that have either easier implementation, lower
runtime overheads, or less platform-specific dependency. _Optimization_ is
different from the _Coding task_ category mentioned above, as askers provide
initial code implementation. _Comprehension_ is about clarification or
comparison of concepts, terms, or APIs (e.g., StringBuilder vs. StringBuffer).
_Debugging_ means that askers present their erroneous programs, and solicit
debugging feedback. _Others_ captures the miscellaneous questions not covered
by any category mentioned above. Additionally, Chen et al. also classified the
threads into nine topics based on the technical content.

According to Fig. 1, SO threads do not distribute evenly among different
categories or topics. For example, _Coding task_ is the dominant category,
covering 102 of the 130 threads. We intentionally chose this dataset instead
of creating a balanced one, because this dataset reflects (i) the major
concerns of developers when they discuss technical issues on SO, and (ii)
developers’ practices of software maintenance under the guidance of an online
knowledge base. Both (i) and (ii) can facilitate our evaluation of ChatGPT, as
they help us identify the most important questions to ask ChatGPT, and provide
good reference answers against which we can compare ChatGPT’s outputs.

##  3\. Methodology

There are three research questions (RQs) in our study:

  * RQ1

_How does ChatGPT compare with programmers when answering technical
questions?_ This RQ examines given questions of different styles (e.g., coding
or debugging task) or covering different topics (e.g., data structure or
algorithm), whether ChatGPT answers outperform or underperform SO answers.

  * RQ2

_How do developers consider the answer differences between ChatGPT and SO?_
This RQ assesses given ChatGPT answers and SO answers, whether developers
present obvious preferences towards one answer type.

  * RQ3

_How does ChatGPT compare with humans when revising code for maintenance?_
This RQ explores ChatGPT’s capability in maintaining software, so it
complements both RQ1 and RQ2 that examine ChatGPT’s capability in answering SO
questions.

Figure 2. An example SO question post

Figure 3. The prompt we crafted for the SO question

###  3.1. The Experiment Design for RQ1

We located the most popular answer (i.e., the one with the highest vote) and
accepted answer in each discussion thread, and considered such answers as the
_best answers provided by expert developers_. Notice that not every thread has
an answer labeled as “accepted”, while in many threads, the most popular
answer is simultaneously the accepted answer. Therefore, we located 1-2 best
answers in each thread. For simplicity, this paper refers to these answers as
“SO answers”. Additionally, we formulated a prompt for each SO question by
extracting the title, technical content, and keywords. For instance, given the
SO question shown in Fig. 3, we crafted the prompt shown in Fig. 3, to ensure
that ChatGPT is exposed to the same amount of question information as humans.
After sending 130 prompts to ChatGPT, we collected all generated answers and
manually compared those answers with SO answers. To avoid human bias by
individual authors, the first two authors independently compared ChatGPT
answers with SO answers. For each group of answers under comparison, they
rated which answer was better based on the following criteria:

  1. (1)

Relevance. Does the answer directly respond to the question?

  2. (2)

Readability. Does the answer clearly explain the solution?

  3. (3)

Comprehensiveness. Is the solution comprehensive enough to cover all edge
cases?

  4. (4)

Informativeness. Does the answer contain code snippets to concretize the
explanation?

  5. (5)

Reusability. If a certain code is provided by the answer, is it easy to
(re)use for developers?

After rating all answer groups, we calculated Cohen’s kappa coefficient value
(McHugh, 2012) to measure the inter-rater reliability. The measured value is
0.746, demonstrating a substantial agreement between the two authors. For the
14 ones on which they disagreed with each other, another author was invited to
compare all answers and to lead a discussion until all three authors reached a
consensus.

###  3.2. The Experiment Design for RQ2

We successfully got an IRB approval to conduct a user study, and recruited 30
Java developers by sending email invitations in our institution or spreading
information via personal networks. Each participant accessed a Google form to
compare ChatGPT answers with SO answers for 10 SO questions, filled in his/her
assessment, and submitted the form online. For all answers, we anonymized the
sources: no participant knows which answer was generated by ChatGPT or SO.
Everyone spent 30–40 minutes to complete the survey, and got a 10-dollar gift
card for compensation.

One possible design of the user study could be asking all participants to
compare the same 10 answer pairs. However, such a design can only check for
people’s opinions on 10 answer pairs at most, and considerably limit the
generalizability of our research. Thus, we decided not to experiment in this
way. Instead, from the dataset created for RQ1, we randomly selected 32 SO
questions, the corresponding best SO answers (i.e., either the accepted or
most popular answer), and ChatGPT answers. We ensured that the selected
questions cover all five question styles mentioned in Section 2, and they are
not too long to demotivate participants.

As shown in Table 1, our selection includes 18 coding tasks, 7 optimization
questions, 4 comprehension questions, 2 debugging questions, and 1 question
belonging to the _Other_ category. We formulated 6 sets $[S_{1},S_{6}]$ with
these questions, so that each set has 10 SO questions to cover all 5
categories. We created six different google forms based on the six question
sets, and sent five copies of each Google form to five participants. To sum
up, each question set is assessed by 5 participants; the 6 sets are assessed
by 30 people and cover in total 32 questions, with some questions shared
between sets because those questions are from smaller categories (e.g.,
_Debugging_ and _Other_).

For each pair of answers under comparison, developers responded to the
following five questions:

Table 1. The SO questions and participants included in our user study Question Category | # of questions | # of participants per question
---|---|---
Coding Task | 18 | 5
Optimization | 7 | 5–10
Comprehension | 4 | 5–15
Debugging | 2 | 30
Others | 1 | 30
Total | 32 | 30

  * Q1:

Is answer #1 correct?

  * Q2:

Is answer #2 correct?

  * Q3:

Does answer #1 have better readability than answer #2 (i.e., more readable)?

  * Q4:

Does answer #1 have better informativeness than answer #2 (i.e., more
informative)?

  * Q5:

Considering all factors mentioned above, which answer do you prefer?

Q1–Q2 are about the correctness checking for ChatGPT answers and SO answers.
Both questions have three options for developers to choose from: correct,
unsure, and incorrect. Q3 and Q4 separately focus on the readability and
informativeness of answers. Participants were expected to respond to both
questions in a five-level Likert scale (Allen and Seaman, 2007). Namely, Q3
and Q4 both have five options for developers to choose from: significantly
better, slightly better, no difference, slightly worse, and significantly
worse. Q5 is about developers’ preference; it provides two options for
participants to choose from: answer #1 and answer #2.

To avoid human bias due to sequential ordering, we randomized the order of
questions and answers for each set. In scenarios where participants lacked the
technical background to assess answers, we permitted them to search for
relevant information online (e.g., articles or books). However, we asked them
to intentionally avoid the information from SO, as the retrieved SO answers
may reveal identities of anonymized answers. We also asked them to avoid
querying ChatGPT for answers, as those generated answers may introduce bias
towards ChatGPT answers. Finally, we collected and analyzed the submitted
Google forms.

###  3.3. The Experiment Design for RQ3

In the 357 GitHub repositories mentioned by Chen et al. (see Section 2), we
found 480 Java files to cite any of the 130 discussion threads. We believe
that when a Java file was modified to include an SO discussion thread and code
revision, it is very likely that developers modified the program in response
to certain maintenance need and the cited thread captures that need. Based on
this insight, we located $\langle$SO-thread, code-revision$\rangle$ pairs in
Java files, and simulated maintenance tasks accordingly. By asking ChatGPT to
fulfill those tasks and by examining the tool’s outputs, we assessed ChatGPT’s
capability in code maintenance and its potential of replacing developers.

Figure 4. The experiment procedure for RQ3

Fig. 4 shows our four-step experiment procedure. Starting from the 357
repositories, we crawled the latest versions to identify 480 Java files that
contain 501 SO references, with some files citing multiple SO references in
one version or citing distinct SO references in different versions. We then
refined the data by applying three filters. First, we checked whether any of
the Java files had the SO link added to its initial version. If so, we
discarded the files because we could not easily locate the code region
specifically relevant to the SO link. Second, for any commit that adds an SO
link to a Java file, we also checked whether any code was revised in that file
by the same commit; if not, we discarded the file. Third, for any file
modified to include both an SO reference and code revision, we checked whether
there is any semantic relevance between the two; if not, the file was
discarded.

In our study, the 3 filters separately removed 357, 29, and 10 SO references
from the original 501 references. We had 105 Java files remaining, which were
further divided into 2 groups: 13 files from compilable projects and 92 files
from uncompilable ones. Due to the time limit, we experimented with all 13
files from compilable projects and 35 sampled unique files from uncompilable
projects. To simplify presentation, we use $f_{i}(i\in[1,48])$ (i.e., 48 = 13
+ 35) to refer to each file. All these files were modified in version history
to introduce (1) references to SO discussions and (2) related code revisions.
Thus, we use $f^{o}_{i}$ and $f^{n}_{i}$ to refer to the old and new versions
of $f_{i}$.

In Step 2, we formulated a prompt for $f_{i}$. The prompt provides the content
of $f^{o}_{i}$; it describes a maintenance request (e.g., feature addition or
bug fixing) based on the SO question cited by $f^{n}_{i}$ as well as
differences between $f^{o}_{i}$ and $f^{n}_{i}$. ChatGPT is supposed to revise
the given code $f^{o}_{i}$ in response to that request. Step 3 sends all
prompts to ChatGPT. Step 4 gathers outputs from ChatGPT and validates them via
automatic build and/or manual inspection. For the first group of files (i.e.,
13 files from compilable projects), this step replaces $f^{n}_{i}$ in each
project with ChatGPT’s output $f^{c}_{i}$, checking whether the generated file
is compilable or whether it is compatible with developers’ code in other
files. For the second group of files (i.e., 35 files from uncompilable
projects), automatic build is inapplicable to validate ChatGPT’s outputs. For
all 48 files revised by ChatGPT, we manually compared $f^{c}_{i}$ with
$f^{n}_{i}$ to examine the semantic equivalence. If $f^{c}_{i}$ matches
$f^{n}_{i}$ in terms of the program logic or implementation algorithm, we
consider ChatGPT to succeed in the maintenance task.

Figure 5. An example to illustrate our procedure of asking ChatGPT to maintain
a codebase

Fig. 5 shows a concrete example to illustrate the procedure. Our dataset has
an SO question describing a coding task: to concatenate two arrays in Java
(see Fig. 5 (a)); there is also a GitHub repository with a Java file $f$
edited to cite the SO question and to add relevant code. For simplicity, Fig.
5 (b) only presents the relevant diff data between program versions before and
after edits. We inspected both the SO question and developers’ related edits,
to craft a prompt. As shown in Fig. 5 (c), the prompt includes the original
version of the Java file ($f^{o}$), and a request asking ChatGPT to modify
code and add a method for concatenating two arrays in Java. To ensure that
ChatGPT’s output can be easily integrated into the GitHub project, we crafted
the prompt to also (1) specify the method header of code-to-generate, and (2)
require generating the full code of the revised file. Due to the space limit,
Fig. 5 (c) only illustrates a brief version of the actual Java code we
included in the prompt. Afterwards, we sent ChatGPT the prompt and got the
modified file $f^{c}$ (see Fig. 5 (d)). We then compared $f^{c}$ with
developers’ revision $f^{n}$, to assess ChatGPT’s maintenance capability.

##  4\. Experiment Results

This section introduces and explains our experiment results.

###  4.1. Results for RQ1

Based on our manual analysis and comparison of all answers, we noticed that
none of the SO/ChatGPT answers is fundamentally wrong. Our answer preference
was decided mainly based on the readability, comprehensiveness, and
informativenes of descriptions. For example, Fig. 6 shows two answers to the
SO question “ _How is it possible to read/write to the Windows registry using
Java?_ ”, including one ChatGPT answer and the most popular SO answer. All
authors agreed that the ChatGPT answer is better, because it is more concise
and clear: it explains how to read and write to the Windows registry using two
simple code snippets. Meanwhile, the SO answer offers a 386-line code
implementation copied from an open-source project, without explaining the
essential internal program logic. Such answers may tempt developers to blindly
copy-and-paste code or get into copyright issues, but does not necessarily
help developers improve coding skills in the long run.

Figure 6. Answers to the SO question “How is it possible to read/write to the
Windows registry using Java?”

Among the 130 SO questions, the authors preferred ChatGPT answers for 97
questions, and preferred SO answers for 33 questions. It means that ChatGPT
answers are often better. Table 2 shows the result breakdown across five
categories. According to this table, ChatGPT provides better answers for 78%
of coding tasks, while SO answers are better for 22% of coding tasks. For the
four debugging questions, ChatGPT answers are always better. Among
optimization and other questions, ChatGPT answers are better for 50% of cases,
while SO answers are better for the remaining 50%. Among the four
comprehension questions, ChatGPT answers are better for three cases.

Table 2. Distribution of comparison results across question categories | Coding Task | Optimization | Comprehension | Debugging | Other
---|---|---|---|---|---
ChatGPT answer is better | 78% (80/102) | 50% (9/18) | 75% (3/4) | 100% (4/4) | 50% (1/2)
SO answer is better | 22% (22/102) | 50% (9/18) | 25% (1/4) | 0% (0/4) | 50% (1/2)
Table 3. Distribution of comparison results across different subcategories of _Coding Task_ and _Optimization_ Question Category |  Technical Topic | ChatGPT answer is better | SO answer is better
---|---|---|---
Coding task |  Data processing | 86% (30/35) | 14% (5/35)
Feature implementation in a certain context | 72% (21/29) | 28% (8/29)
Inspection/Manipulation of program execution at runtime | 64% (9/14) | 36% (5/14)
File processing | 89% (8/9) | 11% (1/9)
Emulation of the syntax feature from another language | 60% (3/5) | 40% (2/5)
Algorithm | 75% (3/4) | 25% (1/4)
New feature for automation | 100% (3/3) | 0% (0/3)
Data structure | 100% (2/2) | 0% (0/2)
Testing | 100% (1/1) | 0% (0/1)
Optimization |  Data processing | 62% (5/8) | 38% (3/8)
File processing | 25% (1/4) | 75% (3/4)
Algorithm | 33% (1/3) | 67% (2/3)
Data structure | 50% (1/2) | 50% (1/2)

Due to the large number of data samples in categories _Coding task_ and
_Optimization_ , we further zoomed into these categories to study how well the
two types of answers compare with each other on different technical topics. As
shown in Table 3, ChatGPT answers generally outperform SO answers in all
subcategories of _Coding task_. For some minor topics like _New feature for
automation_ (i.e., discussion on rare feature implementation), _Data
structure_ (i.e., discussion on defining customized data structures), and
_Testing_ (i.e., discussion on defining test cases), ChatGPT answers are
always better. One reason to explain our observations is that ChatGPT answers
often have better clarity, readability, and/or larger coverage of edge cases
and alternative solutions.

Interestingly, for optimization questions, ChatGPT answers only outperform SO
answers for 50% of the cases. SO answers are better for the remaining 50% of
cases mainly because (1) the SO answers have more rigorous performance
comparison between alternative code snippets, or (2) the SO answers suggest
usage of advanced libraries instead of coding from scratch.

Compared with SO answers, ChatGPT answers are often better because they (1)
include as much relevant information as possible, and (2) provide more clear
and concise explanations. However, ChatGPT answers may not outperform when
askers look for optimized solutions.

Finding 1: _For 75% (97/130) of SO questions we studied, ChatGPT answers are
better than SO answers (i.e., accepted or most popular answers). For almost
all the question styles and technical topics we examined, ChatGPT answers are
typically preferable or the dominant better answers._
---

###  4.2. Results for RQ2

Table 4 presents the years of Java programming experience of the 30
participants in our user study. As shown in the table, 20 developers have 1-2
years of experience, 8 developers have 3-5 years of experience, and 2
developers have 6-10 years of experience. $U_{1}$–$U_{6}$ denote the six user
groups we created, based on the six question sets $S_{1}$–$S_{6}$ mentioned in
Section 3.2. Namely, each group $U_{i}(i\in[1,6])$ has five participants,
assessing answers for $S_{i}$. As all survey questions anonymize the sources
of answers under discussion, our analysis mapped answers to their sources
(i.e., ChatGPT or SO) after the survey was done and before we derived the
results discussed below.

Table 4. Years of Java programming experience of the 30 participants in our user study | $U_{1}$ | $U_{2}$ | $U_{3}$ | $U_{4}$ | $U_{5}$ | $U_{6}$ | Total
---|---|---|---|---|---|---|---
1-2 years | 1 | 4 | 4 | 3 | 4 | 4 | 20
3-5 years | 2 | 1 | 1 | 2 | 1 | 1 | 8
6-10 years | 2 | 0 | 0 | 0 | 0 | 0 | 2
¿10 years | 0 | 0 | 0 | 0 | 0 | 0 | 0

####  4.2.1. Correctness Comparison

As shown in Table 5, across all groups, ChatGPT answers received more
“Correct” labels than SO answers (231 vs. 206), more “Incorrect” labels (29
vs. 25), but a lot fewer “Unsure” labels (40 vs. 69). For individual groups,
ChatGPT answers received more “Correct” labels than SO answers in five groups,
but received fewer “Correct” labels in $U_{1}$ only. ChatGPT answers received
more “Incorrect” labels in four groups, but received fewer “Incorrect” labels
in only two groups: $U_{3}$ and $U_{5}$. ChatGPT answers received fewer
“Unsure” labels in all groups.

We also clustered developers’ responses for each answer, and identified the
assessments voted for by the majority. Namely, if N (N$\geq$5) developers
assessed the correctness of one answer, we identified the label commonly
chosen by at least N/2 developers and treated it as the assessment voted for
by the majority. As shown in Table 6, among the 32 sampled ChatGPT answers,
the majority voted for 30 correct answers, 1 unsure answer, and 1 incorrect
answer. Both the incorrect and unsure answers are about coding tasks, probably
because these tasks are hard to solve or the ChatGPT answers are hard to
evaluate. Among the 32 sampled SO answers, the majority voted for 25 correct
answers and 3 unsure answers; 4 SO answers received no majority voting as
developers’ opinions diverge a lot. The three “Unsure” answers are all about
coding tasks; the “No majority” answers separately correspond to one coding
task, two debugging questions, and one comprehension question. Such phenomena
imply that SO answers are generally harder to assess than ChatGPT answers.

Our observations imply that compared with SO answers, ChatGPT answers are more
likely to be considered correct, and developers are more certain about their
correctness assessment for ChatGPT answers. This may be because ChatGPT
answers are more readable or more informative.

Table 5. Developers’ assessments for answer correctness

Answer Type | Label | $U_{1}$ | $U_{2}$ | $U_{3}$ | $U_{4}$ | $U_{5}$ | $U_{6}$ | Total
---|---|---|---|---|---|---|---|---
ChatGPT | Correct | 34 | 40 | 43 | 31 | 41 | 42 | 231
Unsure | 5 | 3 | 3 | 14 | 8 | 7 | 40
Incorrect | 11 | 7 | 4 | 5 | 1 | 1 | 29
SO | Correct | 36 | 38 | 32 | 30 | 36 | 34 | 206
Unsure | 9 | 7 | 10 | 17 | 10 | 16 | 69
Incorrect | 5 | 5 | 8 | 3 | 4 | 0 | 25

Table 6. The majority voting for answer correctness

| ChatGPT answers | SO answers
---|---|---
Correct | 30 | 25
Unsure | 1 | 3
Incorrect | 1 | 0
No majority | 0 | 4

Finding 2: _Compared with SO answers, ChatGPT answers were rated to be correct
more often (231 vs. 206); developers were more certain when rating ChatGPT
answers._
---

Table 7. Developers’ assessments for answer readability Assessment | $U_{1}$ | $U_{2}$ | $U_{3}$ | $U_{4}$ | $U_{5}$ | $U_{6}$ | Total
---|---|---|---|---|---|---|---
ChatGPT answer is significantly more readable | 18 | 13 | 17 | 16 | 15 | 8 | 87
ChatGPT answer is slightly more readable | 10 | 7 | 10 | 11 | 16 | 22 | 76
ChatGPT answer and SO answer are equally readable | 6 | 11 | 6 | 13 | 9 | 11 | 56
SO answer is slightly more readable | 11 | 10 | 10 | 7 | 7 | 4 | 49
SO answer is significantly more readable | 5 | 9 | 7 | 3 | 3 | 5 | 32
Table 8. The clustering of readability assessments based on question categories Assessment |  Coding Task | Optimization | Comprehension | Debugging | Others
---|---|---|---|---|---
ChatGPT answer is significantly more readable |  31% | 20% | 37% | 20% | 37%
ChatGPT answer is slightly more readable |  24% | 30% | 30% | 27% | 20%
ChatGPT answer and SO answer are equally readable |  17% | 28% | 12% | 13% | 27%
SO answer is slightly more readable |  14% | 13% | 17% | 22% | 13%
SO answer is significantly more readable |  13% | 8% | 5% | 18% | 3%

####  4.2.2. Readability Comparison

Table 7 presents developers’ assessments of answer readability. Among the 300
ratings developers provided, 163 ratings (54%) show that ChatGPT answers are
significantly or slightly more readable; 81 ratings (27%) show that SO answers
are more readable; and 56 ratings (19%) indicate the equivalent readability
between two types of answers. Furthermore, among the five rows in Table 7, the
first row “ _ChatGPT answer is significantly more readable_ ” has the largest
total count—87, while the last row “ _SO answer is significantly more
readable_ ” has the smallest total count—32. In each of the six groups
[$U_{1}$, $U_{6}$], there are always more ChatGPT answers rated to have better
readability than SO answers.

We also clustered developers’ readability assessments based on question
categories. As shown in Table 8, the majority of ratings (¿50%) in three
categories— _Coding Task_ , _Comprehension_ , and _Others_ —imply ChatGPT
answers to be more readable. Among all five categories, _Comprehension_
received the highest percentage of ratings for the higher readability of
ChatGPT answers—67%, while _Debugging_ received the lowest—47%.

Finding 3: _Developers tend to rate ChatGPT answers to have better readability
than SO answers. Compared with other answers, ChatGPT answers to comprehension
questions are most likely to be considered having better readability._
---

####  4.2.3. Informativeness Comparison

Tables 9–10 show developers’ assessments of answer informativeness. These
tables present similar phenomena to those reported for readability assessments
(see Section 4.2.2). For instance, as shown in Table 9, among all assessments,
the majority of ratings (i.e., 180) shows that ChatGPT answers are more
informative, fewer ratings (i.e., 80) show SO answers to be more informative,
and even fewer ratings (i.e., 40) indicate the equivalent informativeness
between two types of answers. In Table 10, the majority of ratings in all five
categories imply that ChatGPT answers are more informative. Compared with
other categories, ChatGPT answers to comprehension questions are most likely
to have better informativeness.

Table 9. The clustering of assessments for answer informativeness based on user groups Assessment | $U_{1}$ | $U_{2}$ | $U_{3}$ | $U_{4}$ | $U_{5}$ | $U_{6}$ | Total
---|---|---|---|---|---|---|---
ChatGPT answer is significantly more informative | 19 | 14 | 21 | 17 | 18 | 9 | 98
ChatGPT answer is slightly more informative | 12 | 9 | 11 | 12 | 15 | 23 | 82
ChatGPT answer and SO answer are equally informative | 3 | 7 | 5 | 10 | 7 | 8 | 40
SO answer is slightly more informative | 10 | 13 | 8 | 6 | 8 | 7 | 52
SO answer is significantly more informative | 6 | 7 | 5 | 5 | 2 | 3 | 28
Table 10. The clustering of assessments for answer informativeness based on question categories Assessment |  Coding Task | Optimization | Comprehension | Debugging | Others
---|---|---|---|---|---
ChatGPT answer is significantly more informative |  30% | 25% | 53% | 27% | 27%
ChatGPT answer is slightly more informative |  28% | 28% | 27% | 27% | 30%
ChatGPT answer and SO answer are equally informative |  13% | 17% | 10% | 13% | 13%
SO answer is slightly more informative |  18% | 20% | 8% | 23% | 17%
SO answer is significantly more informative |  11% | 10% | 2% | 10% | 13%

We calculated the Pearson correlation coefficient (Benesty et al., 2009)
between developers’ assessments for readability and informativeness, to
measure how strongly the two variables are related to each other.
Specifically, we converted the five-level Likert scale to points, by having “
_ChatGPT answer is significantly more readable (or informative)_ ” mapped to
five points and getting “ _SO answer is significantly more readable (or
informative)_ ” mapped to one point. This allows us to quantitatively
represent developers’ opinions with numbers, and to apply statistical analysis
to the two groups of numeric values. We found the Pearson correlation
coefficient to be 0.73, with the p-value smaller than 0.00001. It means that
the two variables are strongly correlated, and this correlation is
statistically significant and real. Namely, if a developer highly rates an
answer’s readability (or informativeness), he/she is likely to also highly
rate the answer’s informativeness (or readability).

Finding 4: _Developers tend to consider ChatGPT answers to be more
informative. There is a strong positive correlation between developers’
assessments for readability and informativeness._
---

Table 11. Preferences organized by groups Group | ChatGPT Chosen | SO Chosen
---|---|---
$U_{1}$ | 66% | 34%
$U_{2}$ | 54% | 46%
$U_{3}$ | 66% | 34%
$U_{4}$ | 72% | 28%
$U_{5}$ | 76% | 24%
$U_{6}$ | 72% | 28%

Table 12. Preferences organized by question categories Question Category | ChatGPT Chosen | SO Chosen
---|---|---
Coding task | 64% | 36%
Optimization | 65% | 35%
Comprehension | 83% | 17%
Debugging | 60% | 40%
Other | 67% | 33%
Total | 68% | 32%

####  4.2.4. Developers’ Overall Preferences

Tables 12 and 12 show developers’ overall answer preferences. In each user
group, more developers preferred ChatGPT answers to SO answers (see Table 12).
In 54%–76% of cases, developers preferred ChatGPT answers. According to Table
12, in each question category, ChatGPT answers were chosen more often as the
preferable ones. In particular, for comprehension questions, the highest
percentage of developers (83%) prefer ChatGPT answers over SO answers. It
means that ChatGPT is especially good at answering comprehension questions. In
total, among the 300 ratings provided by developers, 203 ratings (68%) were
about their preferences for ChatGPT answers, while only 97 ratings (32%) show
that developers preferred SO answers.

Additionally, we clustered developers’ responses for each answer pair under
comparison, and identified the preferences voted for by the majority. Namely,
if N (N$\geq$5) developers simultaneously compared an $\langle
SO,ChatGPT\rangle$ answer pair, we identified the label commonly chosen by at
least N/2 developers and treated it as the preference voted for by the
majority. As shown in Table 14, in 32 sampled answer pairs, the majority
preferred 5 SO answers and 25 ChatGPT answers. The five SO answers respond to
four coding tasks and one optimization question. In another two sampled answer
pairs, responses are divided equally between SO and ChatGPT as N is even.
These answer pairs separately respond to a debugging question and an
optimization question. Our results imply that SO answers sometimes outperform
ChatGPT answers when responding to coding tasks, debugging questions, and
optimization requests.

Table 13. The Pearson correlation analysis between answer characteristics and
developers’ preferences

Variables | Coefficient | P-value
---|---|---
ChatGPT Answer correctness vs. Preferences | 0.26 | ¡0.00001
SO Answer Correctness vs. Preferences | -0.27 | ¡0.00001
Readability vs. Preferences | 0.7 | ¡0.00001
Informativeness vs. Preferences | 0.73 | ¡0.00001

Table 14. The answer preference by the majority of developers

|  The majority voting
---|---
(1) SO chosen |  5
(2) ChatGPT chosen |  25
Both (1) and (2) |  2

To better understand developers’ answer preferences, we did statistical
analysis to see whether developers’ preferences are correlated with the
correctness, readability, or informativeness of answers. Firstly, we mapped
developers’ 300 ratings for ChatGPT answer correctness to numeric values: 3
(correct), 2 (unsure), and 1 (incorrect); we also mapped developers’ 300
preference responses to numeric values: 1 (SO answer), and 2 (ChatGPT answer).
We then applied the Pearson correlation analysis to the two groups of numeric
data. As shown in Table 13, the coefficient is 0.26, implying that developers’
preferences are weakly related to the correctness of ChatGPT answers.
Secondly, we repeated the above-mentioned process to study the correlation
between developers’ preferences and the correctness of SO answers. As shown in
Table 13, the two variables are also weakly related.

Furthermore, we adopted the two-way ANOVA test (Yates, 1934)—a statistical
method to examine the influence of two different categorical independent
variables on one continuous dependent variable. By applying this method, we
examined whether developers’ preferences depend on the correctness contrast
within the $\langle SO,ChatGPT\rangle$ answer pairs for given questions. Our
analysis shows all evaluated p-values to be greater than 0.05, which means
that the correctness contrasts between SO answers and ChatGPT answers do not
determine developers’ preferences. All these three statistical tests imply
that developers did not base their preference decisions on answer correctness.
One possible reason is that the answers-under-comparison are often correct, or
have little difference in terms of the correctness property.

Next, we applied Pearson correlation analysis to (1) developers’ ratings of
readability and preferences, and (2) developers’ ratings of informativeness
and preferences. As shown in Table 13, both tests produced high coefficient
values: 0.7 and 0.73, and low p-values (¡0.00001). The phenomena indicate that
developers expressed their preferences mainly based on the readability and
informativeness of answers.

Finding 5: _Among the 300 ratings provided by developers, 203 ratings show
that developers preferred ChatGPT answers. Our statistical analysis shows that
developers’ preferences were mainly affected by the answer readability and
informativeness, but not by the answer correctness._
---

Finally, we compared developers’ manual analysis results against ours
described in Section 4.1. Our answer preferences match developers’ preferences
in 84% (27/32) of cases. There are only five cases where our preferences do
not match developers’. One of the cases is about optimization, and we are very
confident that SO provides a more efficient code solution than ChatGPT. The
other four cases cover three coding tasks and one comprehension question; the
preference divergence is mainly due to personal styles or coding habits.

###  4.3. Results for RQ3

This section reports our experiments with the tasks separately defined for 13
compilable projects and 35 uncompilable ones.

####  4.3.1. Experiment with the 13 tasks defined for compilable projects.

As shown in Table 15, our prompts ask ChatGPT to (1) add one or more methods
to an existing Java file, (2) modify a return statement in an existing method,
or (3) revise the implementation of existing Java methods. ChatGPT was able to
generate revised Java files for all prompts. By trying to compile the files
output by ChatGPT, we found 11 out of the 13 files to compile successfully.
One file (see M1) does not compile, because ChatGPT omitted almost all details
of unchanged code in the given Java file and it majorly presented the added
code implementation. Another file (see M13) does not compile because ChatGPT
introduced the usage of a variable radius, without defining or declaring
radius first.

Table 15. The 13 maintenance tasks we created in compilable projects for ChatGPT to fulfill Id |  Program |  Maintenance task |  Does ChatGPT’s code compile? |  Does ChatGPT’s code semantically match developers’ code?
---|---|---|---|---
M1 |  LOFiles (LOF, 2023) |  Add a Java method to clone an entire document using the Java DOM. |  No |  No. ChatGPT omits details of unchanged code.
M2 |  CoreNLP (Cor, 2023) |  Add a method to concatenate two arrays in Java and return the result. |  Yes |  Yes. ChatGPT’s code also adds extra sanity checks for inputs.
M3 |  DeltaLauncher (Del, 2023) |  Add a method to concatenate two arrays in Java and return the result. |  Yes |  Yes
M4 |  jlib (jli, 2023) |  Add a method to check whether the char-typed input parameter is printable. |  Yes |  No. ChatGPT’s code considers fewer corner cases.
M5 |  Achilles (Ach, 2023) |  Add a method to programmatically determine the availability of a port in a given machine. |  Yes |  No. Divergent values are assigned to the same field; ChatGPT’s code does not use the parameter hostname.
M6 |  lanterna (lan, 2023) |  Modify a return-statement, to make sure the returned value is true when the given character is printable. |  Yes |  No. ChatGPT’s code considers fewer corner cases.
M7 |  gnikrap (gni, 2023) |  Add two methods to separately decode and encode base64 data. |  Yes |  Yes
M8 |  the-holy-braille (the, 2023) |  Add a method to count the lines of a file |  Yes |  Yes
M9 |  Aiolos (Aio, 2023) |  Modify the code to configure a logger of the type java.util.logging.Logger, to have Level.ALL. |  Yes |  No. Developers’ code contains more project-specific logic.
M10 |  CodingProblems (Cod, 2023) |  Add a method to implement an optimized algorithm of checking if an integer’s square root is an integer. |  Yes |  No. ChatGPT’s code is not an optimized solution.
M11 |  markov-test (mar, 2023) |  Revise an existing method, so that it reads the content of a specified file, creates a Java string from that content, and returns the value. |  Yes |  Yes
M12 |  opennars (ope, 2023) |  Add a method to return the exponential value of a given input, in a optimized and fast way. |  Yes |  No. ChatGPT omits details of unchanged code.
M13 |  skyroad-magnets (sky, 2023) |  Modify an existing method, to check whether the circle and rectangle intersect in 2D euclidean space. |  No |  No. The variable radius is used but not defined; the output value is calculated differently.

For only 5 out of the 13 cases, ChatGPT successfully revised given files to
satisfy maintenance needs. The prompts of all these five cases perfectly match
the program logic implemented in developers’ code, and the logic is totally
irrelevant to the surrounding program context or other methods/classes defined
in the same project. Specifically for M2, ChatGPT not only output a correct
Java file, but also added extra checks for the input parameters to avoid null-
pointer dereferences.

For 8 out of the 13 cases, ChatGPT did not revise given files as expected. In
addition to the compilation issue mentioned above, two major reasons explain
our observations. First, ChatGPT’s code handles fewer corner cases than
developers’ code (see M4 and M6). For instance, M6 requires for a method
addition to check whether a given char-typed variable is printable. ChatGPT’s
code considers limited types of non-printable characters (i.e., line
separator, paragraph separator, and unassigned characters). However,
developers’ code covers more types of non-printable characters (e.g., keyboard
characters like “Tab”). Second, ChatGPT could not generate project-specific
logic (see M5, M9, M13). For instance, Fig. 7 presents the implementations by
both developers and ChatGPT for M5. Both snippets satisfy the requirement of
determining the availability of a port in a machine. However, developers’ code
calls setReuseAddress(...) with the false parameter value, while ChatGPT’s
code calls that method with true; developers’ code uses the input hostname but
ChatGPT’s does not. Although ChatGPT’s code is reasonable; it does not fit the
program context.

Finding 6: _Among the 13 maintenance tasks defined for compilable projects,
ChatGPT only successfully fulfilled 5 tasks. This implies that ChatGPT usually
cannot maintain software independently._
---

Figure 7. ChatGPT’s code does not match developers’ code for M5, as it allows for reuse of the port  Table 16. The 35 maintenance tasks we created in uncompilable projects for ChatGPT to fulfill Id |  Does ChatGPT’s code semantically match developers’ code? | Id |  Does ChatGPT’s code semantically match developers’ code?
---|---|---|---
M14 |  Yes | M32 |  No. ChatGPT’s code is incomplete.
M15 |  No. ChatGPT’s code is not an optimized solution, as required. | M33 |  Yes.
M16 |  Yes | M34 |  Yes.
M17 |  No. ChatGPT’s code does not contain project-specific logic. | M35 |  No. ChatGPT’s code is incomplete.
M18 |  No. ChatGPT’s code is incomplete, and it also misses project-specific logic. | M36 |  No. ChatGPT’s code calls an API that can throw an exception, but the method header does not declare that exception-to-throw. Meanwhile, developers’ code does not call any API that can throw exception(s).
M19 |  Yes | M37 |  No. ChatGPT’s code can throw an exception, but the developers’ code does not throw any exception.
M20 |  No. ChatGPT’s code can throw an exception, but the developers’ code does not throw any exception. | M38 |  Yes
M21 |  No. ChatGPT’s code misses project-specific logic. | M39 |  No. ChatGPT’s code can throw an exception, developers’ code does not throw any exception.
M22 |  Yes | M40 |  Yes
M23 |  Yes | M41 |  Yes
M24 |  No. ChatGPT’s code misses project-specific logic. | M42 |  No. ChatGPT’s code covers fewer corner cases.
M25 |  No. ChatGPT’s code covers fewer corner cases. | M43 |  No. ChatGPT’s code fails to use one of the input parameters.
M26 |  Yes. ChatGPT’s code further does sanity checks for inputs. | M44 |  Yes
M27 |  No. ChatGPT’s code does not match project-specific logic. | M45 |  No. ChatGPT’s code is incomplete.
M28 |  No. ChatGPT’s code covers fewer corner cases. | M46 |  No. ChatGPT’s code is incomplete.
M29 |  Yes | M47 |  Yes
M30 |  Yes | M48 |  Yes
M31 |  Yes |  |
Figure 8. ChatGPT’s code matches developers’ code for M26, and conducts an
extra sanity check for the input

####  4.3.2. Experiment with the 35 tasks defined for uncompilable projects

As shown in Table 16, given the 35 tasks, ChatGPT always output code to
respond to our prompts. Among the responses, 17 match the program logic in
developers’ code and 18 responses do not. For 1 of the 17 matching cases—M26,
ChatGPT’s code performs an extra check on the input parameter to eliminate
potential program errors. As shown in Fig. 8, both implementations satisfy the
maintenance need of “adding a method to convert java.util.Date to
java.sql.Date”. However, ChatGPT’s version is safer, as it performs a null-
pointer check before dereferencing the object date.

We identified 4 major reasons to explain why ChatGPT’s code failed to match
developers’ code in 18 cases. First, for 10 cases, ChatGPT’s code lacks the
project-specific logic even though it satisfies the described maintenance
need. Namely, ChatGPT’s code either (1) contains throw-statements to throw
exceptions that are not thrown by developers’ code, (2) misses some if-
condition checks or other statements, or (3) fails to use all input
parameters. Second, for five cases, ChatGPT’s code omits details of the
unchanged code, and only outputs the revised code with limited surrounding
context (i.e., unchanged code). Third, for three cases, ChatGPT’s code
considers fewer corner cases. Fourth, for one case, ChatGPT’s code does not
implement any optimized algorithm to decide whether an integer is a perfect
square, although the prompts ask for such an optimization.

Finding 7: _Among the 35 maintenance tasks defined for uncompilable projects,
ChatGPT could not fulfill 18 tasks mainly because the generated code is either
incomplete, lacking project-specific logic, covering fewer corner cases, or
failing to use all input parameters._
---

##  5\. Threats to Validity

##### Threats to External Validity:

All our observations are limited to the SO discussion threads, GitHub
repositories, and developers’ responses included into our dataset. They may
not generalize well to other SO discussion threads, other repositories
(including closed-source repositories), or other developers. Our study focuses
on Java programs, as the language has been popularly used and we have more
experience and domain knowledge relevant to it; we did not apply ChatGPT to
programs written in other languages. We experimented with ChatGPT-3.5 instead
of ChatGPT-4.0, because ChatGPT-4.0 is not free to use. People may observe
slightly different phenomena when applying higher versions of ChatGPT to
programs written in other languages. In the future, to make our research
findings more representative, we will expand our dataset and explore to use
ChatGPT-4.0.

_Threats to Internal Validity:_ We experimented with the default setting of
ChatGPT-3.5, without controlling or tuning any parameter it defines. By
default, when ChatGPT is queried with the same prompt multiple times, it
generates results with randomness, i.e., it can produce different results
given the same prompt. Such randomness can impact the validity or certainty of
our observations. However, based on our experience so far, ChatGPT often
produces very similar results given multiple trials of the same prompt. We
believe that the internal randomness of ChatGPT does not significantly impact
our experiment results.

_Threats to Construct Validity:_ The manual inspection of SO answers and Java
code is subject to human bias. To mitigate the potential inaccuracy due to
human bias, for RQ1, we had two authors separately compare 130 pairs of
$\langle SO,ChatGPT\rangle$ answers; to resolve any opinion conflict between
them, another author lead a discussion until the three authors reached a
consensus. For RQ2, we recruited as many developers as possible (i.e., 30
developers), asked each to compare 10 answer pairs, and ensured each answer
pair to be inspected by at least 5 developers. For RQ3, we leveraged automatic
compilation to validate ChatGPT’s outputs whenever possible, and had three
authors inspect ChatGPT’s outputs and the optional compilation reports.

ChatGPT was trained on large collections of text data (e.g., books, articles,
and web pages) publicly available by September 2021. All data used in our
study was available before that date. Therefore, model-overfitting issues may
occur and our evaluation may overestimate ChatGPT’s capabilities. However,
considering the large volume of training data used by ChatGPT (i.e., 570 GB)
and potential interference/conflicts among that data, we feel the overfitting
issues to be insignificant. In the future, we plan to evaluate ChatGPT with
more recent data to mitigate this threat.

##  6\. Lessons Learned

Below are the lessons or actionable items we learned from this study.

##### For Developers or Potential ChatGPT Users:

_ChatGPT is a good information resource to refer to when developers have
technical questions._ We observed that ChatGPT answered SO questions with high
accuracy, regardless of the question style or technical topic. According to
ChatGPT itself, “I was pre-trained using a combination of unsupervised and
supervised learning techniques, such as language modeling, auto-encoding, and
sequence prediction.” (cha, 2023c). The pre-trained knowledge enables ChatGPT
to produce answers to SO questions with high accuracy. Another important
aspect that has worked in ChatGPT’s advantage is how quickly it responds.
Developers can get answers instantly, without waiting for someone to notice
their query on a forum. This real-time interaction allows developers to
maintain their momentum and tackle coding challenges without unnecessary
delays. Among the five question categories we studied, ChatGPT answers are
generally better than SO answers for coding tasks and comprehension questions,
but competitive for optimization tasks.

_Developers need to be very cautious when merging ChatGPT’s code into their
own projects._ Compared with developers’ code, ChatGPT’s code often does not
fit into the context of given Java files, even if it can satisfy the
maintenance needs. Such a limitation is due to the unchanged code omitted by
ChatGPT, the missing program-specific logic, or fewer corner cases covered by
ChatGPT’s code. The lack of project-specific logic may be caused by ambiguous
or narrow descriptions of prompts, or by ChatGPT’s limited capability of code
generation. To reasonably simulate real-developers’ efforts, we did not
further refine prompts to describe all project-specific requirements in
addition to the basic maintenance need. It is possible that ChatGPT can work
better when more project-specific details are specified in prompts; however,
further investigation is still needed to validate this argument.

_For SE Community and Q &A Forums:_ _When people create documentation (e.g.,
coding answers or user manuals) to provide guidance on software development or
usage, they can consider using ChatGPT to improve the readability and
informativeness of documents._ Compared with the accepted or most popular SO
answers provided by human experts (e.g., experienced developers), ChatGPT’s
answers are often more desirable. Developers typically prefer ChatGPT’s
answers due to the better readability and higher informativeness, but not
necessarily due to the correctness.

Our observations also raise a potential challenge for developer websites such
as Stack Overflow and the whole SE community. Users may use these Q&A websites
less frequently because they can get better answers from ChatGPT, and even
users of these sites may start using ChatGPT-like tools to formulate and post
their answers. In the long run, these behaviors may reduce the human-crafted
materials available on the Internet for training. It is unknown and worth
further investigation whether such a reduction trend will negatively impact
ChatGPT-like tools.

_For SE Researchers:_ _New program analysis techniques can be invented to
identify the scenarios where it is safer to merge ChatGPT’s code into
projects. Novel testing techniques can be created to specifically test the
interaction between ChatGPT’s code and developers’ code in the same project._
In our study, ChatGPT performed well on certain types of maintenance tasks,
especially when it generated elementary functions to implement independent
features and those features have little or no data dependency on other parts
of the same project. Characterizing the scenarios when ChatGPT’s code is easy
to (re)use can help researchers better assess the automation boundary of
ChatGPT, and help developers better leverage ChatGPT to improve programmer
productivity and software quality.

So far, we have not observed any obvious or dummy error in the outputs by
ChatGPT to signal significant limitations of the tool, or to facilitate our
immediate rejection of specific answers. Consequently, we always spent lots of
time reading every single line of the outputs, thought carefully, and
discussed thoroughly to identify issues lying in those outputs. If developers
are not careful enough or are under time pressure, they may get misled by
ChatGPT and blindly accept all of the tool’s outputs. To avoid the potential
hallucination problem or misinformation produced by ChatGPT, researchers need
to apply comprehensive program analysis tools or even invent new tools to
examine the tool’s outputs rigorously.

_For LLM Researchers: Research needs to be done to investigate the best user
interfaces that facilitate users to provide all necessary information for
accurate code generation or appropriate program revision._ One big challenge
developers or LLM users would face is that it is unknown (1) how much project-
specific information is sufficient, and (2) what project-specific information
will help. Due to the size of code base and intellectual property
restrictions, it is unlikely that developers can use their whole code base as
the prompt. Meanwhile, for developers, the difficulty of describing all
project-specific details can be sometimes equivalent to or even higher than
that of writing code manually. _It is also worth investigation how willing
developers are to specify all project-specific details for coding tasks. The
cost-efficiency of using ChatGPT forms another important research direction_.

##  7\. Related Work

The related work of our research includes empirical studies on SO posts, and
studies on ChatGPT.

###  7.1. Empirical Studies on StackOverflow

Researchers did various studies to characterize the crowdsourced knowledge
available on StackOverflow (Nasehi et al., 2012; Movshovitz-Attias et al.,
2013; Pinto et al., 2014; Gantayat et al., 2015; Honsel et al., 2015; Zou et
al., 2015; Fischer et al., 2017; Baltes et al., 2017; Meng et al., 2018; Zhang
et al., 2018; Chen et al., 2019; Bangash et al., 2019; Openja et al., 2020;
Zahedi et al., 2020; Firouzi et al., 2020; Zhang et al., 2021; Peruma et al.,
2021; Zhu et al., 2022; Lu et al., 2022; Al Shoaibi and Mkaouer, 2023).
Specifically, Zhang et al. (Zhang et al., 2021) explored how the knowledge in
answers becomes obsolete. Gantayat et al. (Gantayat et al., 2015) studied the
synergy between voting and acceptance of answers on SO. Zhu et al. (Zhu et
al., 2022) analyzed the question discussions on SO. Some researchers studied
the behaviors and activities of developers relevant to SO. For instance,
Movshovitz-Attias et al. (Movshovitz-Attias et al., 2013) analyzed the SO
reputation system to identify the participation patterns of high- and low-
reputation users. Baltes et al. (Baltes et al., 2017) analyzed the attributed
and unattributed usages of SO code snippets in GitHub projects. Lu et al. (Lu
et al., 2022) conducted a survey with 938 developers who participate in SO to
understand their participation motivations and incentive perceptions.

Some researchers examined SO threads related to specialized domains or
technical topics. For instance, Pinto et al. (Pinto et al., 2014) analyzed
energy consumption-related posts to explore developers’ concerns, the critical
aspects of energy consumption, and developers’ solutions to improve energy
efficiency. Shoaibi et al. (Al Shoaibi and Mkaouer, 2023) conducted an
exploratory study on the challenges developers face in resolving performance
regression. Peruma et al. (Peruma et al., 2021) analyzed refactoring
discussions on SO, and revealed five areas where developers typically require
help: Code Optimization, Tools and IDEs, Architecture and Design Patterns,
Unit Testing, and Database. Zahedi et al. (Zahedi et al., 2020) applied topic
modeling to SO posts to understand the non-functional requirements (NFRs) that
developers focus on. Firouzi et al. (Firouzi et al., 2020) studied 2,283 C#
snippets mined in SO data dump, to investigate why developers used unsafe
codes (i.e., code blocks encapsulated via the C# unsafe keyword).

Zhang et al. (Zhang et al., 2018) studied the code examples of API usage, to
reveal answers that may misuse APIs. Bangash et al. (Bangash et al., 2019)
analyzed the machine learning-related posts, to investigate developers’
understanding of machine learning. Openja et al. (Openja et al., 2020)
analyzed release engineering questions, to understand the modern release
engineering topics of interest and their difficulty. Fischer et al. (Fischer
et al., 2017) and Meng et al. (Meng et al., 2018; Chen et al., 2019) examined
SO posts related to Java security, to identify developers’ concerns on
security implementation, technical challenges, or vulnerabilities in answer
code.

Our study complements all studies mentioned above, but has a unique focus on
the comparison between SO answers and ChatGPT answers. No existing work
compares ChatGPT answers with SO answers. We mainly focused on the best SO
answers developers could provide, including accepted and most popular answers.
By comparing these answers with ChatGPT answers, we intended to reveal how
ChatGPT compares with human experts in responding to technical questions.

###  7.2. Studies on ChatGPT

Several studies were conducted on ChatGPT (Nascimento et al., 2023; Tian et
al., 2023; Jalil et al., 2023; Sobania et al., 2023; Nikolaidis et al., [n.
d.]; Chen et al., 2023a). Specifically, Nascimento et al. (Nascimento et al.,
2023) used four LeetCode questions to create prompts, and observed ChatGPT to
outperform novice developers in solving easy or medium problems. Jalil et al.
(Jalil et al., 2023) checked how well ChatGPT performs when tasked with
answering common questions in a popular curriculum, and found ChatGPT to
respond to 77.5% of questions. Sobania et al. (Sobania et al., 2023) used a
bug fixing benchmark set—QuixBugs; they found ChatGPT to fix 31 out of 40 bugs
and outperform the state-of-the-art. Tian et al. (Tian et al., 2023) assessed
ChatGPT’s capability in code generation, program repair, and code
summarization. They observed ChatGPT to outperform two large language models
in code generation; it is competitive with a state-of-the-art repair tool; it
produces consistent summaries for code with the same intention. Nikolaidis et
al. (Nikolaidis et al., [n. d.]) evaluated ChatGPT and Copilot using LeetCode
problems. They found both models to well solve easy problems. Chen et al.
(Chen et al., 2023a) created GPTutor, a ChatGPT-powered programming tool, to
provide code explanation for developers in IDE.

Our study complements all prior work, as we studied ChatGPT from unique
angles. We characterized its capability of (1) answering SO questions and (2)
maintaining or revising software in response to new software requirements. We
further examined developers’ opinions on the comparison between ChatGPT
answers and best SO answers.

##  8\. Conclusion

Motivated by the widespread concern on ChatGPT’s capability of replacing
developers and killing jobs, we explored to use ChatGPT in two typical working
scenarios in developers’ daily lives: question answering and software
maintenance. We hypothesized that ChatGPT could not provide good answers to
technical questions, or satisfy the maintenance needs in given software
projects. Surprisingly, we observed ChatGPT to work very well in answering
technical questions, and provide promising outputs to facilitate software
maintenance. Specifically, both our manual inspection and user study show that
given technical questions, ChatGPT answers are often correct and reasonable;
they often have higher quality than the most popular or accepted answers from
SO. This implies that developers can always refer to ChatGPT as a reliable
information resource when they have technical questions; answer-providers or
technical supporters can also leverage ChatGPT to polish or enhance their
original answers, to better help other developers, and to better shape the art
as well as practice of software today and in future.

Meanwhile, ChatGPT’s responses to maintenance tasks are less satisfactory; the
code included in these responses do not fit into the given program contexts in
most cases either due to (1) ChatGPT’s limited understanding of program
context, (2) its limited capability of code generation, or (3) the unclearness
or ambiguity in task-describing prompts. We do not consider ChatGPT to be able
to replace humans or work as independent software maintainers, although we do
observe ChatGPT’s great capability in generating independent functional units
(e.g., Java classes or methods) that have no or little dependency on the
surrounding program context.

To sum up, we are cautiously optimistic about ChatGPT’s role in the software
industry. By quantitatively and qualitatively measuring its capabilities in
question-answering and software-maintaining, our study characterizes the
potential technical support and automation opportunities ChatGPT brings; our
study also reveals the potential pitfalls or challenges provoked by the tool.
In the future, we will create better tools to automatically assess the quality
of ChatGPT’s outputs, or integrate ChatGPT into the existing tool chains for
test generation or bug detection.

## References

  * (1)
  * sof (2021) 2021\.  Software Maintenance Cost: What Is It and Why Is It So Important?  https://bambooagile.eu/insights/software-maintenance-costs/.
  * Ach (2023) 2023\.  Achilles.  https://github.com/doanduyhai/Achilles.
  * Aio (2023) 2023\.  Aiolos.  https://github.com/praus/Aiolos.
  * cha (2023a) 2023a.  ChatGPT could make these jobs obsolete: ‘The wolf is at the door’.  https://nypost.com/2023/01/25/chat-gpt-could-make-these-jobs-obsolete/.
  * gpt (2023) 2023\.  ChatGPT took their jobs. Now they walk dogs and fix air conditioners.  https://www.washingtonpost.com/technology/2023/06/02/ai-taking-jobs/.
  * wil (2023a) 2023a.  ChatGPT will replace All The Software Developer Jobs?  https://medium.com/geekculture/chatgpt-will-replace-all-the-software-developer-jobs-dc0af5c52202.
  * wit (2023) 2023\.  ChatGPT Will Replace Programmers Within 10 Years.  https://levelup.gitconnected.com/chatgpt-will-replace-programmers-within-10-years-91e5b3bd3676.
  * wil (2023b) 2023b.  ChatGPT Will Replace Programmers Within 10 Years .  https://news.ycombinator.com/item?id=35298205.
  * Cod (2023) 2023\.  CodingProblems.  https://github.com/sureshsajja/CodingProblems.
  * Cor (2023) 2023\.  CoreNLP.  https://github.com/codev777/CoreNLP.
  * Del (2023) 2023\.  DeltaLauncher.  https://github.com/D4Delta/DeltaLauncher.
  * pat (2023) 2023\.  Dhaval Patel’s Post.  https://www.linkedin.com/posts/dhavalsays_chatgpt-will-take-away-all-programmer-jobs-activity-7007180411197874176-cPWK.
  * gni (2023) 2023\.  gnikrap.  https://github.com/jbenech/gnikrap.
  * cha (2023b) 2023b.  How ChatGPT Will Destabilize White-Collar Work.  https://www.theatlantic.com/ideas/archive/2023/01/chatgpt-ai-economy-automation-jobs/672767/.
  * cha (2023c) 2023c.  How does ChatGPT actually work?  https://www.zdnet.com/article/how-does-chatgpt-work/.
  * jli (2023) 2023\.  jlib.  https://github.com/dejlek/jlib.
  * lan (2023) 2023\.  lanterna.  https://github.com/hbyint/lanterna.
  * LOF (2023) 2023\.  LOFiles.  https://github.com/CleyFaye/LOFiles.
  * mar (2023) 2023\.  markov-test.  https://github.com/Tamini/markov-test.
  * ope (2023) 2023\.  opennars.  https://github.com/printedheart/opennars.
  * sky (2023) 2023\.  skyroad-magnets.  https://github.com/znGames/skyroad-magnets.
  * so- (2023) 2023\.  So-git experiment.  https://figshare.com/articles/dataset/So-gitexperiment/20425839.
  * the (2023) 2023\.  the-holy-braille.  https://github.com/jchien14/the-holy-braille.
  * cha (2023d) 2023d.  What is ChatGPT and why does it matter? Here’s what you need to know.  https://www.zdnet.com/article/what-is-chatgpt-and-why-does-it-matter-heres-everything-you-need-to-know/.
  * quo (2023) 2023\.  Will ChatGPT replace programmers?  https://www.quora.com/Will-ChatGPT-replace-programmers.
  * wil (2023c) 2023c.  Will ChatGPT Replace Your Job?  https://www.forbes.com/sites/ashleystahl/2023/03/03/will-chatgpt-replace-your-job/.
  * Al Shoaibi and Mkaouer (2023) Deema Adeeb Al Shoaibi and Mohamed Wiem Mkaouer. 2023.  Understanding Software Performance Challenges an Empirical Study on Stack Overflow. In _2023 International Conference on Code Quality (ICCQ)_. 1–15.  https://doi.org/10.1109/ICCQ57276.2023.10114662
  * Allen and Seaman (2007) I Elaine Allen and Christopher A Seaman. 2007.  Likert scales and data analyses.  _Quality progress_ 40, 7 (2007), 64–65.
  * Baltes et al. (2017) Sebastian Baltes, Richard Kiefer, and Stephan Diehl. 2017.  Attribution Required: Stack Overflow Code Snippets in GitHub Projects. In _2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C)_. 161–163.  https://doi.org/10.1109/ICSE-C.2017.99
  * Bangash et al. (2019) Abdul Ali Bangash, Hareem Sahar, Shaiful Chowdhury, Alexander William Wong, Abram Hindle, and Karim Ali. 2019.  What do Developers Know About Machine Learning: A Study of ML Discussions on StackOverflow. In _2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)_. 260–264.  https://doi.org/10.1109/MSR.2019.00052
  * Benesty et al. (2009) Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009.  _Pearson Correlation Coefficient_.  Springer Berlin Heidelberg, Berlin, Heidelberg, 1–4.  https://doi.org/10.1007/978-3-642-00296-0_5
  * Chen et al. (2023a) Eason Chen, Ray Huang, Han-Shin Chen, Yuen-Hsien Tseng, and Liang-Yi Li. 2023a.  GPTutor: a ChatGPT-powered programming tool for code explanation.  arXiv:2305.01863 [cs.HC]
  * Chen et al. (2023b) Juntong Chen, Kulendra Kumar Kaushal, Rutwik Kulkarni, and Na Meng. 2023b.  How Do Java Developers Reuse StackOverflow Answers in Their GitHub Projects?  arXiv:2308.09573 [cs.SE]
  * Chen et al. (2019) M. Chen, F. Fischer, N. Meng, X. Wang, and J. Grossklags. 2019.  How Reliable is the Crowdsourced Knowledge of Security Implementation?. In _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. 536–547.  https://doi.org/10.1109/ICSE.2019.00065
  * Firouzi et al. (2020) Ehsan Firouzi, Ashkan Sami, Foutse Khomh, and Gias Uddin. 2020.  On the Use of C# Unsafe Code Context: An Empirical Study of Stack Overflow. In _Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)_ (Bari, Italy) _(ESEM ’20)_. Association for Computing Machinery, New York, NY, USA, Article 39, 6 pages.  https://doi.org/10.1145/3382494.3422165
  * Fischer et al. (2017) Felix Fischer, Konstantin Böttinger, Huang Xiao, Christian Stransky, Yasemin Acar, Michael Backes, and Sascha Fahl. 2017.  Stack Overflow Considered Harmful? The Impact of Copy&Paste on Android Application Security. In _2017 IEEE Symposium on Security and Privacy (SP)_. 121–136.  https://doi.org/10.1109/SP.2017.31
  * Gantayat et al. (2015) Neelamadhav Gantayat, Pankaj Dhoolia, Rohan Padhye, Senthil Mani, and Vibha Singhal Sinha. 2015.  The Synergy between Voting and Acceptance of Answers on StackOverflow - Or the Lack Thereof. In _2015 IEEE/ACM 12th Working Conference on Mining Software Repositories_. 406–409.  https://doi.org/10.1109/MSR.2015.50
  * Honsel et al. (2015) Verena Honsel, Steffen Herbold, and Jens Grabowski. 2015.  Intuition vs. Truth: Evaluation of Common Myths about StackOverflow Posts. In _2015 IEEE/ACM 12th Working Conference on Mining Software Repositories_. 438–441.  https://doi.org/10.1109/MSR.2015.58
  * Jalil et al. (2023) S. Jalil, S. Rafi, T. D. LaToza, K. Moran, and W. Lam. 2023.  ChatGPT and Software Testing Education: Promises & Perils. In _2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)_. IEEE Computer Society, Los Alamitos, CA, USA, 4130–4137.  https://doi.org/10.1109/ICSTW58534.2023.00078
  * Lu et al. (2022) Yao Lu, Xinjun Mao, Minghui Zhou, Yang Zhang, Zude Li, Tao Wang, Gang Yin, and Huaimin Wang. 2022.  Motivation Under Gamification: An Empirical Study of Developers’ Motivations and Contributions in Stack Overflow.  _IEEE Transactions on Software Engineering_ 48, 12 (2022), 4947–4963.  https://doi.org/10.1109/TSE.2021.3130088
  * McHugh (2012) Marry L. McHugh. 2012.  Interrater reliability: the kappa statistic.  _Biochem Med (Zagreb)_ (2012).
  * Meng et al. (2018) Na Meng, Stefan Nagy, Danfeng Yao, Wenjie Zhuang, and Gustavo Arango-Argoty. 2018.  Secure coding practices in java: Challenges and vulnerabilities. In _2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE)_. IEEE, 372–383.
  * Movshovitz-Attias et al. (2013) Dana Movshovitz-Attias, Yair Movshovitz-Attias, Peter Steenkiste, and Christos Faloutsos. 2013.  Analysis of the reputation system and user contributions on a question answering website: StackOverflow. In _2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2013)_. 886–893.  https://doi.org/10.1145/2492517.2500242
  * Nascimento et al. (2023) Nathalia Nascimento, Paulo Alencar, and Donald Cowan. 2023.  Comparing Software Developers with ChatGPT: An Empirical Investigation.  arXiv:2305.11837 [cs.SE]
  * Nasehi et al. (2012) Seyed Mehdi Nasehi, Jonathan Sillito, Frank Maurer, and Chris Burns. 2012.  What makes a good code example?: A study of programming Q amp;A in StackOverflow. In _2012 28th IEEE International Conference on Software Maintenance (ICSM)_. 25–34.  https://doi.org/10.1109/ICSM.2012.6405249
  * Nikolaidis et al. ([n. d.]) Nikolaos Nikolaidis, Karolos Flamos, Daniel Feitosa, Alexander Chatzigeorgiou, and Apostolos Ampatzoglou. [n. d.].  The End of an Era: Can Ai Subsume Software Developers? Evaluating Chatgpt and Copilot Capabilities Using Leetcode Problems.  http://dx.doi.org/10.2139/ssrn.4422122.
  * Openja et al. (2020) Moses Openja, Bram Adams, and Foutse Khomh. 2020.  Analysis of Modern Release Engineering Topics : ? A Large-Scale Study using StackOverflow ?. In _2020 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. 104–114.  https://doi.org/10.1109/ICSME46990.2020.00020
  * Peruma et al. (2021) Anthony Peruma, Steven Simmons, Eman Abdullah AlOmar, Christian D. Newman, Mohamed Wiem Mkaouer, and Ali Ouni. 2021.  How do i refactor this? An empirical study on refactoring trends and topics in Stack Overflow.  _Empirical Software Engineering_ 27, 1 (2021), 11.  https://doi.org/10.1007/s10664-021-10045-x
  * Pinto et al. (2014) Gustavo Pinto, Fernando Castor, and Yu David Liu. 2014.  Mining Questions about Software Energy Consumption. In _Proceedings of the 11th Working Conference on Mining Software Repositories_ (Hyderabad, India) _(MSR 2014)_. Association for Computing Machinery, New York, NY, USA, 22–31.  https://doi.org/10.1145/2597073.2597110
  * Sobania et al. (2023) Dominik Sobania, Martin Briesch, Carol Hanna, and Justyna Petke. 2023.  An Analysis of the Automatic Bug Fixing Performance of ChatGPT.  arXiv:2301.08653 [cs.SE]
  * Tian et al. (2023) Haoye Tian, Weiqi Lu, Tsz On Li, Xunzhu Tang, Shing-Chi Cheung, Jacques Klein, and Tegawendé F. Bissyandé. 2023.  Is ChatGPT the Ultimate Programming Assistant – How far is it?  arXiv:2304.11938 [cs.SE]
  * Yates (1934) F. Yates. 1934.  The Analysis of Multiple Classifications with Unequal Numbers in the Different Classes.  _J. Amer. Statist. Assoc._ 29, 185 (1934), 51–66.  https://doi.org/10.1080/01621459.1934.10502686 arXiv:https://www.tandfonline.com/doi/pdf/10.1080/01621459.1934.10502686
  * Zahedi et al. (2020) Mansooreh Zahedi, Roshan Namal Rajapakse, and Muhammad Ali Babar. 2020.  Mining Questions Asked about Continuous Software Engineering: A Case Study of Stack Overflow. In _Proceedings of the 24th International Conference on Evaluation and Assessment in Software Engineering_ (Trondheim, Norway) _(EASE ’20)_. Association for Computing Machinery, New York, NY, USA, 41–50.  https://doi.org/10.1145/3383219.3383224
  * Zhang et al. (2021) Haoxiang Zhang, Shaowei Wang, Tse-Hsun Chen, Ying Zou, and Ahmed E. Hassan. 2021.  An Empirical Study of Obsolete Answers on Stack Overflow.  _IEEE Transactions on Software Engineering_ 47, 4 (2021), 850–862.  https://doi.org/10.1109/TSE.2019.2906315
  * Zhang et al. (2018) Tianyi Zhang, Ganesha Upadhyaya, Anastasia Reinhardt, Hridesh Rajan, and Miryung Kim. 2018.  Are Code Examples on an Online Q&A Forum Reliable? A Study of API Misuse on Stack Overflow. In _Proceedings of the 40th International Conference on Software Engineering_ (Gothenburg, Sweden) _(ICSE ’18)_. Association for Computing Machinery, New York, NY, USA, 886–896.  https://doi.org/10.1145/3180155.3180260
  * Zhu et al. (2022) Wenhan Zhu, Haoxiang Zhang, Ahmed E. Hassan, and Michael W. Godfrey. 2022.  An empirical study of question discussions on Stack Overflow.  _Empirical Software Engineering_ 27, 6 (2022), 148.  https://doi.org/10.1007/s10664-022-10180-z
  * Zou et al. (2015) Jie Zou, Ling Xu, Weikang Guo, Meng Yan, Dan Yang, and Xiaohong Zhang. 2015.  Which Non-functional Requirements Do Developers Focus On? An Empirical Study on Stack Overflow Using Topic Analysis. In _2015 IEEE/ACM 12th Working Conference on Mining Software Repositories_. 446–449.  https://doi.org/10.1109/MSR.2015.60